I've been using ChatGPT to answer programming questions for weeks. Unless I've been manipulating it poorly, quite simply, ChatGPT takes a great deal of prodding and pleading to get a complete, correct response from, when a correct response is achievable at all. It's a tool, basically a jacked-up autocomplete that's wrong, say, more than half the time.
For example, here's some code it wrote for an extremely trivial programming task, finding the minimum element in an enumerable in Elixir:
def find_smallest(numbers) do
Enum.min(numbers)
end
Fine and well. I asked it to translate this code to Dart and it provided this wrong answer[1]:
To solve the problem of finding the smallest number in a list in Dart, you can use the min
function from the dart:math
library.
Here is an example of how you could implement the find_smallest
function in Dart:
import 'dart:math';
int findSmallest(List<int> numbers) {
return numbers.min();
}
You can then use the expect
function from the package:test
library to write test cases for the function.
[bunch of test cases that the function fails on]
This is the sort of plausible-looking ChatGPT-based answer you'd see flooding Stack Overflow before the ban.
I'm not a Dart programmer, so after this failed, I looked it up on SO to find the human solution. This isn't an isolated incident. It's the typical workflow using ChatGPT at the present time.
There's no doubt that it's an amazing piece of technology that's correct some of the time. It's helped me quickly solve certain problems over the past few weeks. I don't doubt that it solved the problem you mentioned for you. When successful, it's magic. The problem is, it reveals itself as unreliable once you put it through the ringer on a variety of problems. I'm not talking about 80% accuracy, from what I've seen, more like 40% accuracy.
But technology is advancing and we are becoming more and more independent of human skill to find solutions to programming challenges
Not really. Human skill is just as important as ever. Nowadays, one needs to be extremely judicious about separating fact from fiction and separating plausible-looking solutions from truly correct ones.
Unless large strides are made soon, ChatGPT is nowhere near replacing Stack Overflow or programmers in general any more than self-driving mode will be replacing human drivers.
[1]: I'm aware that asking the same question again, differently or at a later time may give the correct answer for this problem. That doesn't change the fact that it fails catastrophically like this often, on simple tasks.