> Except how do you know it's generating the right tests? Can it explain its reasoning?
In my own project I've got it generating solutions to errors, and if possible, it also generates simple unit tests to validate the fix. What I haven't implemented yet (and likely won't, because Copilot does it now) but I have tested is generating a pull request describing the fix, why it works, and the same for the tests.
> Now we have Chat-GPT4 which will give you a plausible-sounding answer that is completely wrong and makes no sense.
With limited scopes, it works quite well. For example, something fails because a DOM reference is undefined in a React component. GPT will add a condition to assert that the reference is present, then generate a simple test which mounts the component with stubbed references that are present or undefined using jsdom. The tests makes sense. A quick scan shows they're sensible, and upon running them, they do work.
I began adding a recursive feature which would automatically debug issues with its own solutions, but it can get a little weird in some cases. Likely due to bad prompting – I haven't dedicated enough time to it. But it can also make it so tests with errors are revised and corrected so they will at least run.
All of that with a coherent explanation of what was changed, why, tests, and why they assert the fix is valid.
Is it perfect? No. Could it be useful? Absolutely. I'm a little sad Copilot makes my project redundant because it was actually very exciting to build. There is real potential here. I started the project in order to learn and validate GPT, and I'm very convinced it has genuine utility and massive potential.
> CoPilot just doesn't really help here. It doesn't understand specifications and doesn't do any reasoning. It can't take a specification, generate a program, discover new abstractions that make the solution more elegant, and explain its reasoning. It can generate a heck of a lot of code though! Wow! Is it the right code? Maybe!
I think the key is limited scopes. Like with the React component example, the solution is small, easy to reason about, and tedious to resolve yourself. I understand why it doesn't work, I get the error, and spinning up an entire branch and PR to clean up the mistake is a bad use of my time. I don't want Copilot/AI to work magic, but I'm okay with it resolving minor mistakes and misuses of languages and libraries here and there.
I do think it will grow from here to do more and actually be good at it, though.
In my own project I've got it generating solutions to errors, and if possible, it also generates simple unit tests to validate the fix. What I haven't implemented yet (and likely won't, because Copilot does it now) but I have tested is generating a pull request describing the fix, why it works, and the same for the tests.
> Now we have Chat-GPT4 which will give you a plausible-sounding answer that is completely wrong and makes no sense.
With limited scopes, it works quite well. For example, something fails because a DOM reference is undefined in a React component. GPT will add a condition to assert that the reference is present, then generate a simple test which mounts the component with stubbed references that are present or undefined using jsdom. The tests makes sense. A quick scan shows they're sensible, and upon running them, they do work.
I began adding a recursive feature which would automatically debug issues with its own solutions, but it can get a little weird in some cases. Likely due to bad prompting – I haven't dedicated enough time to it. But it can also make it so tests with errors are revised and corrected so they will at least run.
All of that with a coherent explanation of what was changed, why, tests, and why they assert the fix is valid.
Is it perfect? No. Could it be useful? Absolutely. I'm a little sad Copilot makes my project redundant because it was actually very exciting to build. There is real potential here. I started the project in order to learn and validate GPT, and I'm very convinced it has genuine utility and massive potential.
> CoPilot just doesn't really help here. It doesn't understand specifications and doesn't do any reasoning. It can't take a specification, generate a program, discover new abstractions that make the solution more elegant, and explain its reasoning. It can generate a heck of a lot of code though! Wow! Is it the right code? Maybe!
I think the key is limited scopes. Like with the React component example, the solution is small, easy to reason about, and tedious to resolve yourself. I understand why it doesn't work, I get the error, and spinning up an entire branch and PR to clean up the mistake is a bad use of my time. I don't want Copilot/AI to work magic, but I'm okay with it resolving minor mistakes and misuses of languages and libraries here and there.
I do think it will grow from here to do more and actually be good at it, though.