-67

I have signed up for an account at OpenAI and asked ChatGPT a programming question, namely: How to list all files in an Android device recursively with Java

And in an instant, it generated an answer in less than a second and the answer was correct. Given that most users post to Stack Overflow and don't even get an answer to their question, this could be a relief to most users who experience this and would discourage anyone plagiarizing code from OpenAI and pasting it here to gain reputation. But technology is advancing and we are becoming more and more independent of human skill to find solutions to programming challenges. Some high reputation contributors ignore questions deliberately but openchat gpt came to save the day. The speed and the accuracy of OpenAI responses are what gives it an edge over Stack Overflow and then users won't get banned or question limit imposed on users based on time intervals. Your thoughts and views on this matter are welcome, could openchat GPT be the end of Stack Overflow?

33
  • 37
    We are talking about the same AI that thinks that in the time you aged 64 years, your sister aged 70 years right?
    – Thom A
    Dec 29, 2022 at 8:48
  • 21
    "and the answer was correct" .. why, because it worked? How do you know it is not the worst approach which also works? Or is that unimportant as long as does what you want it to do?
    – Tom
    Dec 29, 2022 at 8:57
  • 13
    Some of the horrendously wrong and unoptimised solutions I've seen from it would dispute that, @PubliusFlaviusTiberium .
    – Thom A
    Dec 29, 2022 at 9:01
  • 12
    @PubliusFlaviusTiberium You can believe that, yes, but expect do be disappointed.
    – Tom
    Dec 29, 2022 at 9:03
  • 4
    If it can't do basic maths, @PubliusFlaviusTiberium , I'm not going to trust it with anything. Especially as I can't read or write Java and thus have no way of telling if the words it spits out are even close to correct.
    – Thom A
    Dec 29, 2022 at 9:05
  • 10
    How about you give it a question that isn't already answered multiple times on the internet? I've played with it for quite some time and have seen it imagine things and generate faulty code, unless I give it very specific directions. With their current capability these tools are only good as a wing-man / assistant for a programmer. Dec 29, 2022 at 9:06
  • 4
    @PubliusFlaviusTiberium Not specific in the sense of the problem but specific in the sense of the code I want to be generated. I've defined the problem specifically enough to it that a good developer will be able to write code for it. If you feel that it is generating stellar code and replacing SO you haven't played around with it enough. Dec 29, 2022 at 9:18
  • 7
    Note as well, that ChatGPT was out of date even when it was launched. It was only "fed" data up to some point in 2021; it does not have any knowledge of new features and progress made since then (so it knows certainly knows nothing of updates bad to languages/products made in 2022, and we're about to enter 2023). For some technologies, this lack of "bleeding edge" fucntionality is an easy downfall for ChatGPT.
    – Thom A
    Dec 29, 2022 at 9:26
  • 24
    I am forever jealous. Why do other people find places to work where they actually need only code which an AI can produce? I always need to do things that require three weeks of work, thousands of lines of code added/changed/deleted of which one week of head banging because documentation too assumes everyone is building hello world and little more :/ Not to mention the gruelling process of writing automated tests to verify that it works because testing in production tends to be a no-no and test environments are usually warzones held together with duct tape.
    – Gimby
    Dec 29, 2022 at 10:15
  • 8
    “… and it would explain like a human being. That's where the money is.” - Trying to explain a Java algorithm like that would also make a horrible Stack Overflow answer. One I would probably downvote. As a programmer who has seen a couple dozen answers generated by OpenGPT I can tell you that OpenGPT has a pretty poor track record (literally 100% of the answers I have seen have been low quality incomplete garbage) In fact I haven’t seen worse answers to a programming question anywhere from anyone in my nearly 3 decades of programming. ChartGPT is a garbage tool to answer programming questions Dec 29, 2022 at 10:40
  • 9
    Another example of stupid arithmetic (and lack of logic) by ChatGPT: i.stack.imgur.com/Z4uFC.jpg
    – PM 2Ring
    Dec 29, 2022 at 11:21
  • 21
    That's not an arithmetic failure, @PM2Ring, it's a logic failure. Same as answering "1 month" to the question "if it takes 1 woman 9 months to have a baby, how long will it take 9 women to have a baby?"
    – Cody Gray Mod
    Dec 29, 2022 at 11:23
  • 7
    What is "OpenChat GPT"? An Android app? Do you mean OpenAI's ChatGPT? OpenChat is allegedly a chat application. Have they incorporated ChatGPT into it? Can you elaborate? Dec 29, 2022 at 18:56
  • 5
    @President James K. Polk: The Stack Overflow Q&A as a training set was already dubious (unless they have applied some filters, like a vote total threshold or a 'lateness' threshold). Many (or most?) of the "try this" answers (code dumps) are likely completely bogus and/or plagiarised code (bogus in the context). Dec 29, 2022 at 22:50
  • 5
    @CodyGray ChatGPT might be a blessing in disguise for the site, if new and very inexperienced developers can get good and fast answers to their problems, which repeatedly flood the site, this should leave more room for the more intriguing head-scratching variety, problems that require ingenuity, creativity and a wealth of experience. Or am I being unrealistic and optimistic?
    – Mari-Lou A
    Dec 30, 2022 at 10:53

10 Answers 10

3

I have had the same experience as the OP.

ChatGPT has saved me quite literally hours of searching for solutions online. While I agree that ChatGPT does not accurately solve all of my problems, it has been extremely useful in aiding me with objectives like, learning react, (finally) installing and leveraging docker, and building applications using websockets and it offers far more of a head start than browsing SO. I sense that ChatGPT is facing the same pessimism that all novel technologies face, and unjustly so. It is an incredible tool that we should all be excited about and I humbly advise my colleagues here to learn to work with ChatGPT.

3
  • 7
    "does not accurately solve" That is, like, exactly the problem. None of the other things are SO's goal. I sense that SO is facing the same pessimism that all specialised tools face, and unjustly so. It is an incredible tool that we should all be excited about and I humbly advise my colleagues here to learn to work with SO. Feb 9 at 7:56
  • 3
    nailed three technical projects with chat GPT and am pretty sure if I posted here the questions would go unanswered
    – Dong Li
    Feb 9 at 11:04
  • 2
    The point is that once an answer is generated from ChatGPT, its still on you to conform it to your project. The same is true of SO, however ChatGPT will get you to the final solution much faster and with far less effort.
    – Daniel
    Feb 9 at 16:56
60

I've yet to see an answer from ChatGPT that was accurate. Speed? Sure, the robot wins. I don't come to Stack Overflow because I need the fastest answer. I come to Stack Overflow because I need to know the correct answer, from a trustworthy expert who knows what they're doing. (And, better yet, an answer that has been vetted by other experts via our voting system.) Your mileage may vary. Do whatever works for you.

11
  • 5
    Even accuracy, I just asked it how to list all audio files in my PC with python and it generated an answer answer glob patterns and os modules, this AI is amazing. it's nothing the world has ever seen before.
    – Dong Li
    Dec 29, 2022 at 7:56
  • 23
    A bot that can copy-paste code snippets from elsewhere really doesn't amaze me all that much. You know what's really ironic? If what you say about it is true, then we don't need human programmers at all. Just have the bot do it. It won't just be Stack Overflow that is obsolete; it'll be everyone. I think we have a few years before that happens.
    – Cody Gray Mod
    Dec 29, 2022 at 7:57
  • 1
    even if its copy pasting, the answer is correct and in an instant. My post would spend days or hours without a response here on SO , I feel better now as a programmer. But it's true automation will render all of us jobless in some few years
    – Dong Li
    Dec 29, 2022 at 8:01
  • 30
    Dunno about you, but no aspect of my job involves copy-pasting code. I have to design things that work reliably.
    – Cody Gray Mod
    Dec 29, 2022 at 8:07
  • 2
    but future users come to SO to copy paste ?
    – Dong Li
    Dec 29, 2022 at 8:17
  • 31
    Then those users will have the same problem with Stack Overflow as they do with ChatGPT, @PubliusFlaviusTiberium , blindly copying code without validating and altering it for their own needs first, and then wondering why it isn't working.
    – Thom A
    Dec 29, 2022 at 8:54
  • while some people(including myself) might simplify a programmer task is to "copypaste". It's hardly the case.
    – Alvi15
    Dec 29, 2022 at 9:26
  • 9
    “It’s nothing the world has ever seen before.” - A Google engineer thought the same thing about their AI project, they thought it was alive, even Google knew better and fired that engineer. ChartGPT is garbage in my opinion. Just to be clear it’s also not AI. Dec 29, 2022 at 10:42
  • 1
    I think for alot of people it comes down to the fact that most people on SO are pretty condescending, The Voting system isnt always accurate you tend to get more upvotes just for taking a bit more time to explain or write some prettier code. I think overall most just see as a way to avoid being drenched on for asking a question that to the individual is strange and difficult but might be simplistic in the eyes of the community thus becoming a burden. I think both have their place, However im going to have to go with the AI here Its less snarky and tends to explain better Sorry SO
    – Azurry
    Feb 16 at 5:42
  • 1
    @Azurry What's "inaccurate" about getting more upvotes for taking a bit more time to explain or write prettier code? Doesn't that make the answer more useful to more people in the future? That's exactly what upvotes are for. And then you complain that SO answers don't explain as well as the AI? This is a contradiction! Yes, if you're looking for a help desk or a personalized, individual answer or someone to, um, chat with, then ChatGPT is probably better than Stack Overflow. That's fine; that's not what Stack Overflow is trying to be.
    – Cody Gray Mod
    Feb 16 at 6:28
  • 2
    now we can be stupid faster
    – user
    Mar 14 at 6:30
44

No, at least not with this technology.

A lot of people don't understand that ChatGPT is a language model. Its goal is to smash words together to create a plausible-sounding explanation, not creating a correct one. So most of the time the answer can look correct at first glance, but it is blatantly wrong upon close inspection. This can be seen as it made up a return value.

In contrast, a high-upvoted Stack Overflow answer is written by a subject-matter expert and reviewed by people, so it is correct most of the time.

5
  • 1
    did you ask it a programming question? it works
    – Dong Li
    Dec 29, 2022 at 8:04
  • 17
    @PubliusFlaviusTiberium Have you seen 99% of its' blatantly wrong answers floating around the site? 1% of the time what it came up to is right by pure chance. I could do that too. Dec 29, 2022 at 8:10
  • 1
    but you lose right at the speed?
    – Dong Li
    Dec 29, 2022 at 8:18
  • 29
    @PubliusFlaviusTiberium I don't need speedy and wrong answers. People come to Stack Overflow to copy the correct answer. Dec 29, 2022 at 8:19
  • 25
    I can calculate 8912384*12333 faster than a calculator.... the answer is 400. Being fast doesn't mean much.
    – Alvi15
    Dec 29, 2022 at 9:27
34

Well, when I entered How to list all files in an Android device recursively with Java into Google, I got links to several Stack Overflow Q&A, also in less than a second. The first one has 7 different answers, and the second one has 33 different answers. People have already voted on whether the proposed solutions were helpful to them or not; the top answer on one of them has 442 upvotes and 1 downvote, so I already know that hundreds of humans have reviewed the answer for correctness.

You got one ChatGPT answer that no one else had reviewed but that just happened to be correct (and that's not always the case); in the same amount of time, I got 40 answers to choose from (all of them reviewed by other humans before, in many cases hundreds of times).

Imagine what would've happened if ChatGPT's answer hadn't been correct; what would you do then - ask it the same question and hope for a different answer? Or just post a Stack Overflow question instead?

You can downvote a bad Stack Overflow answer or comment asking for clarification, but there's no recourse for a bad ChatGPT answer.

Also,

Some high reputation contributors ignore questions deliberately...

Given that most high-rep users got that way in the first place mostly by answering questions, this claim is close to self-defeating. Personally, I have 609 answers (vs. 61 questions), so I guess I didn't ignore at least 609 questions. Jon Skeet didn't ignore 35,534 questions. Gordon Linoff didn't ignore 85,121 questions. VonC didn't ignore 29,184 questions. I could keep going with this, but the point is the vast majority of people become high-rep by answering lots of questions.

By way of contrast, you have asked 61 questions (50 of which have at least one answer and 34 have an accepted answer) and have only written 9 answers, so there is a certain irony in you accusing us of deliberately ignoring questions.

2
  • you checked my profile lol 🤣 , I will borrow the answer attitude and try to provide more answers to questions than asking my own
    – Dong Li
    Dec 29, 2022 at 18:50
  • 2
    I do ignore a lot of questions. After closing them as duplicates, lacking information, the result of typos or similar trivial problems. Not many good, original questions, but that's just the nature of the beast. In C++ there will be a burst of good questions around the release of a new Standard revision, then a few years of mostly same-old, same-old because damn near all of the good, old questions have been asked and answered. And that's exactly what should happen. Dec 31, 2022 at 9:49
31

ChatGPT is not the end of Stack Overflow. SO isn't designed to give answers to everyone who asks. The goal is to offer quality answers so that people don't have to ask a question. We trust that people who answer are experts who check their answers. The answers have value because they explain some programming concept in an easy and correct way. Speed is irrelevant as the correct answer can be posted at any time even years later.

ChatGPT is able to produce an answer that isn't necessarily correct or best practice. It might work for you but it's not coming from a programming expert.

In the long run ChatGPT may help Stack Overflow if it can provide correct answers to simple questions. This could help avoid duplicates of common questions from being asked.

5
  • 2
    Yeah and would you encourage users to try chaptGPT first then if the solution doesn't work only then post to SO?
    – Dong Li
    Dec 29, 2022 at 8:57
  • 1
    I wouldn't wait for a whole for an answer . I would rather try the incorrect one and learn something
    – Dong Li
    Dec 29, 2022 at 8:57
  • 16
    @PubliusFlaviusTiberium so you're saying you're incapable of "Trial and Error", and you need AI to help you with that? I am not sure whether the end of SO should be your main concern...
    – Alvi15
    Dec 29, 2022 at 9:28
  • 8
    I would not encourage use of ChatGPT as it is now. Maybe a couple of years into the future. But chances are that if you can ask ChatGPT, this topic may already be covered on Stack Overflow.
    – Dharman Mod
    Dec 29, 2022 at 9:49
  • 10
    “Yeah and would you encourage users to try chaptGPT first then if the solution doesn't work only then post to SO?” - Ask it how to program that hasn’t been done a million times by freshman college Students in their Introduction to Programming class. Every example you have provided you think is amazing is something any experienced programmer can do in their sleep. Dec 29, 2022 at 10:45
27

Perhaps some future AI program will be able to reliably answer SO questions, but ChatGPT certainly cannot do it. And even if you gave it ten times as much training data, it still couldn't do it, no matter how good that training data was.

GPT-3, the large language model underlying ChatGPT, is certainly an impressive piece of language processing software. But that's all it is. ChatGPT is not intended to be an artificial intelligence, although it may give some people the initial impression that it has some intelligence.

ChatGPT generates plausible text, consistent with its training data and input, but it doesn't know what it's talking about, and it has no way of representing or evaluating the truth of its utterances. Yes, it can say true things, but it can also say complete nonsense, and it can't tell the difference.

ChatGPT does not attempt to make truthful or even logical statements. Its job is to create "completions" of the text you feed it. It's ultimately doing the same kind of tasks as the spelling-checker and word suggestion feature supplied with modern phone keyboards, except that it's working with deeper and broader linguistic structures.

It has no problems making grammatically correct utterances with no spelling errors, and those utterances are linguistically consistent with the input you feed it and its training data. But it's only operating on linguistic structure, it is oblivious to the underlying logic and truth value of the statements it makes. It may emit text which is logically valid and true, but that's just a fortunate side-effect of the linguistic structure and its training data. In extended passages, it can easily say things that are logically inconsistent. And it has no way to evaluate how relevant its utterances are to the input data, apart from the linguistic consistency. So when it says true and useful things that's just a lucky coincidence.

ChatGPT often makes stupid errors of logic and silly arithmetic errors. OTOH, we have theorem-proving AI that can handle complex logic, and sophisticated mathematical systems that can do arithmetic, algebra, and calculus. In the future, we will surely have natural language software that can handle logic and mathematics, but the current incarnation of ChatGPT won't magically gain those abilities merely by throwing more training data at it.


Stephen Wolfram has written some excellent articles about how ChatGPT works, its limitations, and the potential benefit of combining it with Wolfram | Alpha. Please see What Is ChatGPT Doing … and Why Does It Work? and Wolfram|Alpha as the Way to Bring Computational Knowledge Superpowers to ChatGPT.

4
  • yes with the example you provided in a comment. I think it's beyond perfect but with time it will be improved as the research organization that built it are constantly studying it to discover ways on making it almost perfect which when happens will make SO obsolete
    – Dong Li
    Dec 29, 2022 at 13:51
  • 1
    @PubliusFlaviusTiberium OK, but that hasn't happened yet and doesn't appear to be anywhere near happening.
    – ggorlen
    Dec 29, 2022 at 19:08
  • 2
    I'm sorry, @Publius, but I can't tell what "yes with the example you provided in a comment" is supposed to mean, and the rest of your comment isn't very clear, either. Perhaps you disagree with my claim that ChatGPT will not become capable of creating reliable SO answers merely by improving the quality and quantity of its training data, or you simply don't understand my argument regarding that claim.
    – PM 2Ring
    Dec 29, 2022 at 21:37
  • @PM2Ring, the logical fallacies of Open chat gpt, I asked it the same question. if it takes 2 cars hours to travel from DC to Washington how long will it take 4 cars and it replies with 4 hours
    – Dong Li
    Dec 30, 2022 at 5:24
15

I've been using ChatGPT to answer programming questions for weeks. Unless I've been manipulating it poorly, quite simply, ChatGPT takes a great deal of prodding and pleading to get a complete, correct response from, when a correct response is achievable at all. It's a tool, basically a jacked-up autocomplete that's wrong, say, more than half the time.

For example, here's some code it wrote for an extremely trivial programming task, finding the minimum element in an enumerable in Elixir:

def find_smallest(numbers) do
  Enum.min(numbers)
end

Fine and well. I asked it to translate this code to Dart and it provided this wrong answer[1]:

To solve the problem of finding the smallest number in a list in Dart, you can use the min function from the dart:math library.

Here is an example of how you could implement the find_smallest function in Dart:

import 'dart:math';

int findSmallest(List<int> numbers) {
  return numbers.min();
}

You can then use the expect function from the package:test library to write test cases for the function.

[bunch of test cases that the function fails on]

This is the sort of plausible-looking ChatGPT-based answer you'd see flooding Stack Overflow before the ban.

I'm not a Dart programmer, so after this failed, I looked it up on SO to find the human solution. This isn't an isolated incident. It's the typical workflow using ChatGPT at the present time.

There's no doubt that it's an amazing piece of technology that's correct some of the time. It's helped me quickly solve certain problems over the past few weeks. I don't doubt that it solved the problem you mentioned for you. When successful, it's magic. The problem is, it reveals itself as unreliable once you put it through the ringer on a variety of problems. I'm not talking about 80% accuracy, from what I've seen, more like 40% accuracy.

But technology is advancing and we are becoming more and more independent of human skill to find solutions to programming challenges

Not really. Human skill is just as important as ever. Nowadays, one needs to be extremely judicious about separating fact from fiction and separating plausible-looking solutions from truly correct ones.

Unless large strides are made soon, ChatGPT is nowhere near replacing Stack Overflow or programmers in general any more than self-driving mode will be replacing human drivers.


[1]: I'm aware that asking the same question again, differently or at a later time may give the correct answer for this problem. That doesn't change the fact that it fails catastrophically like this often, on simple tasks.

1
  • 6
    In this comment I linked to a ChatGPT answer posted by a 12k+ member (before the ban). The answer text seems to make sense, and the code looks good, but they're rubbish. The loop should use continue to skip duplicates, but instead it uses break, so the loop ends when it finds the first dupe, and hence the output array is truncated. But statistically, break is much more common than continue...
    – PM 2Ring
    Dec 29, 2022 at 22:04
11

Your thoughts and views on this matter are welcome...

Speed yes, accuracy no. One example is not enough to draw any conclusion. Please conduct a more thorough investigation.

The speed of the response generation of the service you used is indeed impressive, although I wouldn't say it's necessarily much faster than using a search engine like Google and clicking on the presented link to Stack Overflow, for example.

Accuracy is unfortunately not yet that good. In same cases it gave acceptable results, but in others not, when I tested it. I cannot recommend it yet.

Why is your and my experience so vastly different? Maybe because you drew a conclusion from only a very small amount of examples (N=1?). That cannot work. There hasn't been done enough example generation and evaluation for the claim to be supported by the evidence.

It's futile to discuss the implications of something we do not know with sufficient certainty yet. We need to study this service more. I'm quite sure that scientists do that already.

Just as a side comment. ChatGPT is competition (capable or incapable) for this platform, but for a knowledge seeker it's just another (useful or not useful) source of information. How useful? The jury is still out.

8
  • 1
    your the first person to acknowledge GPT as a competition to stackoverflow. Let's see how this will affect stackoverflow traffic. Will SO still get 5000 questions a day or will this drop?
    – Dong Li
    Dec 31, 2022 at 10:23
  • 1
    @PubliusFlaviusTiberium I'm not the first one really but also not the last one. The future is difficult to predict. I know some people who use ChatGPT for programming but they might not have asked a question here anyway. Let's just wait and see. Not only the number of questions might drop maybe also the number of visits (and this would be the real loss for this platform).
    – Trilarion
    Dec 31, 2022 at 10:58
  • but i wouldn't want SO to be knocked off its perch by that site, SO can get back at them by adding new features like animations and more themes to make the site even more exciting
    – Dong Li
    Dec 31, 2022 at 11:02
  • 3
    @PubliusFlaviusTiberium What will happen is not necessarily what you want. I don't think there is any imminent risk for SO (more for Google maybe) because SO shines with expert content and high quality content is much more difficult to create than average quality content. On the other hand if that happens I would embrace it. That's progress. I also do not want to go back to write letters instead of sending text messages just because that keeps the postman busy...
    – Trilarion
    Dec 31, 2022 at 11:19
  • @evolution. making life easier and easier with technology results in other technologies being obsolete.
    – Dong Li
    Dec 31, 2022 at 11:25
  • ("animations" may refer to the slow type out of text, like on ancient 110 baud terminals, telex terminals, and old films (typewriter-like).) Dec 31, 2022 at 15:46
  • 1
    Cathode ray tube television was really cool for a long time until say 20 years ago.
    – Trilarion
    Dec 31, 2022 at 16:28
  • I don't know why you would think that a timed display of each character on a post (aka, simulating typing) would be an important addition to the site. It's certainly enticing from a certain perspective, but it doesn't do anything aside from looking weird or fancy depending on where you stand. @PubliusFlaviusTiberium Jan 3 at 8:58
-7

I also verified today that the chat bot cannot beat human experienced responses to programming questions, here is how I verified this, I asked chat GPT a very technical question and it kept displaying the same text over and over again and the solution posted by chat GPT either compiles and refuses to work or the code does not compile. I reached to a developer on quora asked him the same question and his response worked and was straight to the point.

-7

Some dudes are shooting the messenger because they don't like the message.

Here is an answer from ChatGPT itself:

While ChatGPT is a powerful tool for answering a wide range of questions, Stack Overflow still serves a valuable purpose as a community-driven platform for technical Q&A. Rather than replacing Stack Overflow, ChatGPT could be used in conjunction with the site to provide additional support and guidance for users.

One possibility would be to integrate ChatGPT's capabilities into Stack Overflow's existing interface, allowing users to ask questions in a conversational format and receive instant responses. This could potentially help users who are struggling to formulate their questions or who are looking for guidance on how to approach a particular problem.

However, it's important to consider the potential drawbacks of such an approach, such as the risk of information overload or the possibility of inaccurate or incomplete responses. Ultimately, any changes to Stack Overflow should be carefully evaluated to ensure that they align with the site's goals of providing accurate, high-quality technical information to users

2
  • 2
    "Some dudes are shooting the messenger because they don't like the message." The downvotes here are on the answer, i.e. the message. The messenger hasn't been shot at all here. However, it might help the answer if the "messenger" would provide some reasoning to why they requested and delivered a message from an uninvolved writer for recipients who could have asked the writer themselves if they cared. Feb 14 at 15:50
  • I agree with you @MisterMiyagi. Your argument makes sense to me.
    – zell
    Feb 15 at 11:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .