-17

I have signed up for an account at OpenAI and asked ChatGPT a programming question, namely: How to list all files in an Android device recursively with Java

And in an instant, it generated an answer in less than a second and the answer was correct. Given that most users post to Stack Overflow and don't even get an answer to their question, this could be a relief to most users who experience this and would discourage anyone plagiarizing code from OpenAI and pasting it here to gain reputation. But technology is advancing and we are becoming more and more independent of human skill to find solutions to programming challenges. Some high reputation contributors ignore questions deliberately but openchat gpt came to save the day. The speed and the accuracy of OpenAI responses are what gives it an edge over Stack Overflow and then users won't get banned or question limit imposed on users based on time intervals. Your thoughts and views on this matter are welcome, could openchat GPT be the end of Stack Overflow?

42
  • 47
    We are talking about the same AI that thinks that in the time you aged 64 years, your sister aged 70 years right?
    – Thom A
    Commented Dec 29, 2022 at 8:48
  • 31
    "and the answer was correct" .. why, because it worked? How do you know it is not the worst approach which also works? Or is that unimportant as long as does what you want it to do?
    – Tom
    Commented Dec 29, 2022 at 8:57
  • 17
    Some of the horrendously wrong and unoptimised solutions I've seen from it would dispute that, @PubliusFlaviusTiberium .
    – Thom A
    Commented Dec 29, 2022 at 9:01
  • 12
    @PubliusFlaviusTiberium You can believe that, yes, but expect do be disappointed.
    – Tom
    Commented Dec 29, 2022 at 9:03
  • 15
    How about you give it a question that isn't already answered multiple times on the internet? I've played with it for quite some time and have seen it imagine things and generate faulty code, unless I give it very specific directions. With their current capability these tools are only good as a wing-man / assistant for a programmer. Commented Dec 29, 2022 at 9:06
  • 5
    @PubliusFlaviusTiberium Not specific in the sense of the problem but specific in the sense of the code I want to be generated. I've defined the problem specifically enough to it that a good developer will be able to write code for it. If you feel that it is generating stellar code and replacing SO you haven't played around with it enough. Commented Dec 29, 2022 at 9:18
  • 8
    Note as well, that ChatGPT was out of date even when it was launched. It was only "fed" data up to some point in 2021; it does not have any knowledge of new features and progress made since then (so it knows certainly knows nothing of updates bad to languages/products made in 2022, and we're about to enter 2023). For some technologies, this lack of "bleeding edge" fucntionality is an easy downfall for ChatGPT.
    – Thom A
    Commented Dec 29, 2022 at 9:26
  • 34
    I am forever jealous. Why do other people find places to work where they actually need only code which an AI can produce? I always need to do things that require three weeks of work, thousands of lines of code added/changed/deleted of which one week of head banging because documentation too assumes everyone is building hello world and little more :/ Not to mention the gruelling process of writing automated tests to verify that it works because testing in production tends to be a no-no and test environments are usually warzones held together with duct tape.
    – Gimby
    Commented Dec 29, 2022 at 10:15
  • 8
    “… and it would explain like a human being. That's where the money is.” - Trying to explain a Java algorithm like that would also make a horrible Stack Overflow answer. One I would probably downvote. As a programmer who has seen a couple dozen answers generated by OpenGPT I can tell you that OpenGPT has a pretty poor track record (literally 100% of the answers I have seen have been low quality incomplete garbage) In fact I haven’t seen worse answers to a programming question anywhere from anyone in my nearly 3 decades of programming. ChartGPT is a garbage tool to answer programming questions Commented Dec 29, 2022 at 10:40
  • 10
    Another example of stupid arithmetic (and lack of logic) by ChatGPT: i.sstatic.net/Z4uFC.jpg
    – PM 2Ring
    Commented Dec 29, 2022 at 11:21
  • 26
    That's not an arithmetic failure, @PM2Ring, it's a logic failure. Same as answering "1 month" to the question "if it takes 1 woman 9 months to have a baby, how long will it take 9 women to have a baby?"
    – Cody Gray Mod
    Commented Dec 29, 2022 at 11:23
  • 7
    What is "OpenChat GPT"? An Android app? Do you mean OpenAI's ChatGPT? OpenChat is allegedly a chat application. Have they incorporated ChatGPT into it? Can you elaborate? Commented Dec 29, 2022 at 18:56
  • 6
    @President James K. Polk: The Stack Overflow Q&A as a training set was already dubious (unless they have applied some filters, like a vote total threshold or a 'lateness' threshold). Many (or most?) of the "try this" answers (code dumps) are likely completely bogus and/or plagiarised code (bogus in the context). Commented Dec 29, 2022 at 22:50
  • 11
    @CodyGray ChatGPT might be a blessing in disguise for the site, if new and very inexperienced developers can get good and fast answers to their problems, which repeatedly flood the site, this should leave more room for the more intriguing head-scratching variety, problems that require ingenuity, creativity and a wealth of experience. Or am I being unrealistic and optimistic?
    – Mari-Lou A
    Commented Dec 30, 2022 at 10:53
  • 7
    'ChatGPT seems to be better than Stack Overflow, both in speed and accuracy' - Citation Needed
    – CPlus
    Commented Jun 19, 2023 at 21:43

17 Answers 17

59

I have had the same experience as the OP.

ChatGPT has saved me quite literally hours of searching for solutions online. While I agree that ChatGPT does not accurately solve all of my problems, it has been extremely useful in aiding me with objectives like, learning react, (finally) installing and leveraging docker, and building applications using websockets and it offers far more of a head start than browsing SO. I sense that ChatGPT is facing the same pessimism that all novel technologies face, and unjustly so. It is an incredible tool that we should all be excited about and I humbly advise my colleagues here to learn to work with ChatGPT.

12
  • 19
    "does not accurately solve" That is, like, exactly the problem. None of the other things are SO's goal. I sense that SO is facing the same pessimism that all specialised tools face, and unjustly so. It is an incredible tool that we should all be excited about and I humbly advise my colleagues here to learn to work with SO. Commented Feb 9, 2023 at 7:56
  • 13
    nailed three technical projects with chat GPT and am pretty sure if I posted here the questions would go unanswered
    – user16612111
    Commented Feb 9, 2023 at 11:04
  • 9
    The point is that once an answer is generated from ChatGPT, its still on you to conform it to your project. The same is true of SO, however ChatGPT will get you to the final solution much faster and with far less effort.
    – Daniel
    Commented Feb 9, 2023 at 16:56
  • 3
    Chat GPT has rendered most of the StackExchange platforms suddenly completely obsolete.
    – Cesare
    Commented May 7, 2023 at 21:43
  • 17
    @Cesare If you believe that ChatGPT now fullfills all your needs over SE, I’m happy for you. :) You’re free to use ChatGPT instead of SE sites if you so wish. Commented May 8, 2023 at 0:23
  • 5
    Thank you!! :-) already doing that. The main difference is that when asking a question to chat GPT you get an answer. When asking a question to SE you get replies like "please provide a minimum example, or an example of what you have done so far"
    – Cesare
    Commented May 8, 2023 at 9:44
  • 1
    agree. It points you in the right direction a lot quicker
    – Andy
    Commented May 9, 2023 at 15:08
  • 12
    @Cesare And that for a reason. The humans on SO realize that to help you, all the necessary information to get at an answer, must be provided. Else help cannot be given. ChatGPT doesn’t think. It just generates some text that suits whatever input you give it. You can’t rely on it, and if information is missing, then surely the answer it gives can’t be accurate. But if it does help you, that means somebody else already had the same question, or you could’ve found the answer through some research yourself; so your question shouldn’t be put on SO anyway. Commented May 11, 2023 at 8:55
  • 6
    I think this answer deviates from the discussion a bit. The OP asked, "Is this the end for SO?" with a not-so-subtle insinuation that ChatGPT content should be allowed here. Just so we're clear, the fact that many people find ChatGPT useful for programming is not sufficient justification for allowing such content on SO. Commented May 23, 2023 at 20:49
  • @MisterMiyagi As if SO has awnsered that many questions correctly, MOST of them get closed at the start. Commented Sep 12, 2023 at 19:31
  • 3
    @RuanCarloWeiersBritzke The volunteers on SO have answered literally millions of questions. Does it answer every question? No, that too isn’t the goal. Commented Sep 13, 2023 at 5:19
  • @MisterMiyagi it does not even try to awnser most questions, "some" doesn't even come close to it. Commented Sep 13, 2023 at 18:24
86

I've yet to see an answer from ChatGPT that was accurate. Speed? Sure, the robot wins. I don't come to Stack Overflow because I need the fastest answer. I come to Stack Overflow because I need to know the correct answer, from a trustworthy expert who knows what they're doing. (And, better yet, an answer that has been vetted by other experts via our voting system.) Your mileage may vary. Do whatever works for you.

15
  • 9
    Even accuracy, I just asked it how to list all audio files in my PC with python and it generated an answer answer glob patterns and os modules, this AI is amazing. it's nothing the world has ever seen before.
    – user16612111
    Commented Dec 29, 2022 at 7:56
  • 32
    A bot that can copy-paste code snippets from elsewhere really doesn't amaze me all that much. You know what's really ironic? If what you say about it is true, then we don't need human programmers at all. Just have the bot do it. It won't just be Stack Overflow that is obsolete; it'll be everyone. I think we have a few years before that happens.
    – Cody Gray Mod
    Commented Dec 29, 2022 at 7:57
  • 6
    even if its copy pasting, the answer is correct and in an instant. My post would spend days or hours without a response here on SO , I feel better now as a programmer. But it's true automation will render all of us jobless in some few years
    – user16612111
    Commented Dec 29, 2022 at 8:01
  • 41
    Dunno about you, but no aspect of my job involves copy-pasting code. I have to design things that work reliably.
    – Cody Gray Mod
    Commented Dec 29, 2022 at 8:07
  • 3
    but future users come to SO to copy paste ?
    – user16612111
    Commented Dec 29, 2022 at 8:17
  • 36
    Then those users will have the same problem with Stack Overflow as they do with ChatGPT, @PubliusFlaviusTiberium , blindly copying code without validating and altering it for their own needs first, and then wondering why it isn't working.
    – Thom A
    Commented Dec 29, 2022 at 8:54
  • while some people(including myself) might simplify a programmer task is to "copypaste". It's hardly the case.
    – Alvi15
    Commented Dec 29, 2022 at 9:26
  • 9
    “It’s nothing the world has ever seen before.” - A Google engineer thought the same thing about their AI project, they thought it was alive, even Google knew better and fired that engineer. ChartGPT is garbage in my opinion. Just to be clear it’s also not AI. Commented Dec 29, 2022 at 10:42
  • 2
    I think for alot of people it comes down to the fact that most people on SO are pretty condescending, The Voting system isnt always accurate you tend to get more upvotes just for taking a bit more time to explain or write some prettier code. I think overall most just see as a way to avoid being drenched on for asking a question that to the individual is strange and difficult but might be simplistic in the eyes of the community thus becoming a burden. I think both have their place, However im going to have to go with the AI here Its less snarky and tends to explain better Sorry SO
    – Azurry
    Commented Feb 16, 2023 at 5:42
  • 4
    @Azurry What's "inaccurate" about getting more upvotes for taking a bit more time to explain or write prettier code? Doesn't that make the answer more useful to more people in the future? That's exactly what upvotes are for. And then you complain that SO answers don't explain as well as the AI? This is a contradiction! Yes, if you're looking for a help desk or a personalized, individual answer or someone to, um, chat with, then ChatGPT is probably better than Stack Overflow. That's fine; that's not what Stack Overflow is trying to be.
    – Cody Gray Mod
    Commented Feb 16, 2023 at 6:28
  • 5
    now we can be stupid faster
    – starball
    Commented Mar 14, 2023 at 6:30
  • 1
    SO is about coding. What we need to help us is a code generator that works in any language and it's output always compiles and runs on the target hardware and OS. Is ChatGPT a good code generator?
    – MT1
    Commented Jul 18, 2023 at 8:30
  • Stack Overflow is a question and answer site for professional and enthusiast programmers.
    – MT1
    Commented Jul 18, 2023 at 8:35
  • 3
    "I've yet to see an answer from ChatGPT that was accurate" - now that is just a wishful thinking, you wish AI was so wrong to keep your job safe. The truth is in between, as always. It is sometimes wrong, but very often it is surprisingly right.
    – Robo Robok
    Commented Oct 25, 2023 at 8:02
  • 2
    No, it's an empirical statement of fact, and it remains true to this day, actually.
    – Cody Gray Mod
    Commented Aug 17 at 12:36
58

No, at least not with this technology.

A lot of people don't understand that ChatGPT is a language model. Its goal is to smash words together to create a plausible-sounding explanation, not creating a correct one. So most of the time the answer can look correct at first glance, but it is blatantly wrong upon close inspection. This can be seen as it made up a return value.

In contrast, a high-upvoted Stack Overflow answer is written by a subject-matter expert and reviewed by people, so it is correct most of the time.

6
  • 2
    did you ask it a programming question? it works
    – user16612111
    Commented Dec 29, 2022 at 8:04
  • 23
    @PubliusFlaviusTiberium Have you seen 99% of its' blatantly wrong answers floating around the site? 1% of the time what it came up to is right by pure chance. I could do that too. Commented Dec 29, 2022 at 8:10
  • 2
    but you lose right at the speed?
    – user16612111
    Commented Dec 29, 2022 at 8:18
  • 36
    @PubliusFlaviusTiberium I don't need speedy and wrong answers. People come to Stack Overflow to copy the correct answer. Commented Dec 29, 2022 at 8:19
  • 35
    I can calculate 8912384*12333 faster than a calculator.... the answer is 400. Being fast doesn't mean much.
    – Alvi15
    Commented Dec 29, 2022 at 9:27
  • 2
    @Yohst I just said that the technology isn't good enough to meet SO's standard. ChatGPT has many benefits, but quality is just not one of them. There are many instances where ChatGPT just blatantly gives code that straight-up broke, like the link in the example. Also, why do you assume that I'm just a "SO fanboy" protecting their kingdom? That can't be further from the truth. Commented Aug 22, 2023 at 7:51
46

Well, when I entered How to list all files in an Android device recursively with Java into Google, I got links to several Stack Overflow Q&A, also in less than a second. The first one has 7 different answers, and the second one has 33 different answers. People have already voted on whether the proposed solutions were helpful to them or not; the top answer on one of them has 442 upvotes and 1 downvote, so I already know that hundreds of humans have reviewed the answer for correctness.

You got one ChatGPT answer that no one else had reviewed but that just happened to be correct (and that's not always the case); in the same amount of time, I got 40 answers to choose from (all of them reviewed by other humans before, in many cases hundreds of times).

Imagine what would've happened if ChatGPT's answer hadn't been correct; what would you do then - ask it the same question and hope for a different answer? Or just post a Stack Overflow question instead?

You can downvote a bad Stack Overflow answer or comment asking for clarification, but there's no recourse for a bad ChatGPT answer.

Also,

Some high reputation contributors ignore questions deliberately...

Given that most high-rep users got that way in the first place mostly by answering questions, this claim is close to self-defeating. Personally, I have 609 answers (vs. 61 questions), so I guess I didn't ignore at least 609 questions. Jon Skeet didn't ignore 35,534 questions. Gordon Linoff didn't ignore 85,121 questions. VonC didn't ignore 29,184 questions. I could keep going with this, but the point is the vast majority of people become high-rep by answering lots of questions.

By way of contrast, you have asked 61 questions (50 of which have at least one answer and 34 have an accepted answer) and have only written 9 answers, so there is a certain irony in you accusing us of deliberately ignoring questions.

8
  • 2
    you checked my profile lol 🤣 , I will borrow the answer attitude and try to provide more answers to questions than asking my own
    – user16612111
    Commented Dec 29, 2022 at 18:50
  • 5
    I do ignore a lot of questions. After closing them as duplicates, lacking information, the result of typos or similar trivial problems. Not many good, original questions, but that's just the nature of the beast. In C++ there will be a burst of good questions around the release of a new Standard revision, then a few years of mostly same-old, same-old because damn near all of the good, old questions have been asked and answered. And that's exactly what should happen. Commented Dec 31, 2022 at 9:49
  • 2
    @user4581301 "After closing them as duplicates, lacking information, the result of typos or similar trivial problems". Closing question because of typos and trivial problems is why SO kinda get's labeled as toxic. Please don't close questions because of simply a typo. <3
    – brat
    Commented Apr 11, 2023 at 11:03
  • 3
    @brat It's site policy to close such questions, so it would be unethical for us not to vote to close them. They're highly unlikely to be helpful to anyone other than the OP, so they create a distraction from more substantial issues. If you disagree, feel free to write a Meta post challenging that policy. Commented Apr 11, 2023 at 13:04
  • 5
    @brat Depends on the nature of typo. The vast majority of the time a comment like, "You defined FooBar and implemented foobar" or "you mixed up i and j in the for loop[,]" is enough to get the asker back on the right path and isn't going to help anyone else. Writing a formal answer and keeping the question visible is just adding noise to the search results. Close it and let the roomba do its job. If it's something subtle and sneaky or there's some secondary value-add that's more important than the question asked, considering answer that. Commented Apr 11, 2023 at 15:30
  • 1
    @user4581301 Oh. That kind of typos. Fair enough, my bad. I thought you meant spelling mistakes like this wone. Up voting your answer for clearing that up.
    – brat
    Commented Apr 11, 2023 at 15:33
  • 3
    @brat Yeah, if it's just typos in the post itself, then it should be edited (not closed). If a post is salvageable through editing, it should be. Commented Apr 11, 2023 at 16:00
  • Googling How do i list all files in an Android device recursively with Java produces several links to SO. I asked BARD the same question and it came up with some code which I can't check as I have no rooted Android device, it also gave a link to the source of its answer... which was wrong.
    – MT1
    Commented Jul 17, 2023 at 16:23
37

ChatGPT is not the end of Stack Overflow. SO isn't designed to give answers to everyone who asks. The goal is to offer quality answers so that people don't have to ask a question. We trust that people who answer are experts who check their answers. The answers have value because they explain some programming concept in an easy and correct way. Speed is irrelevant as the correct answer can be posted at any time even years later.

ChatGPT is able to produce an answer that isn't necessarily correct or best practice. It might work for you but it's not coming from a programming expert.

In the long run ChatGPT may help Stack Overflow if it can provide correct answers to simple questions. This could help avoid duplicates of common questions from being asked.

8
  • 3
    Yeah and would you encourage users to try chaptGPT first then if the solution doesn't work only then post to SO?
    – user16612111
    Commented Dec 29, 2022 at 8:57
  • 2
    I wouldn't wait for a whole for an answer . I would rather try the incorrect one and learn something
    – user16612111
    Commented Dec 29, 2022 at 8:57
  • 17
    @PubliusFlaviusTiberium so you're saying you're incapable of "Trial and Error", and you need AI to help you with that? I am not sure whether the end of SO should be your main concern...
    – Alvi15
    Commented Dec 29, 2022 at 9:28
  • 8
    I would not encourage use of ChatGPT as it is now. Maybe a couple of years into the future. But chances are that if you can ask ChatGPT, this topic may already be covered on Stack Overflow.
    – Dharman Mod
    Commented Dec 29, 2022 at 9:49
  • 11
    “Yeah and would you encourage users to try chaptGPT first then if the solution doesn't work only then post to SO?” - Ask it how to program that hasn’t been done a million times by freshman college Students in their Introduction to Programming class. Every example you have provided you think is amazing is something any experienced programmer can do in their sleep. Commented Dec 29, 2022 at 10:45
  • 1
    I don't agree with you at all. I have not used SO since ChatGPT and have asked some serious tough questions !
    – Makky
    Commented Jul 4, 2023 at 20:18
  • At the moment I am using both. But SO comes first since it gives me a range of answers. ChatGPT and BingChat are sometimes amazing. But often the code is wrong, doesn't compile or just leads me down a rabbit hole. Commented Oct 12, 2023 at 9:06
  • Yesterday I asked ChatGPT how I could pass in a list of parameters in a JSON string to a SQL server proc and use those for the IN part of a WHERE clause. It immediately suggested using a table param. There was an issue where my json wasn't a string of id's but was a list of arrays. It gave a solution to that. Problem solved in ten minutes. Commented Oct 12, 2023 at 9:10
34

Perhaps some future AI program will be able to reliably answer SO questions, but ChatGPT certainly cannot do it. And even if you gave it ten times as much training data, it still couldn't do it, no matter how good that training data was.

GPT-3, the large language model underlying ChatGPT, is certainly an impressive piece of language processing software. But that's all it is. ChatGPT is not intended to be an artificial intelligence, although it may give some people the initial impression that it has some intelligence.

ChatGPT generates plausible text, consistent with its training data and input, but it doesn't know what it's talking about, and it has no way of representing or evaluating the truth of its utterances. Yes, it can say true things, but it can also say complete nonsense, and it can't tell the difference.

ChatGPT does not attempt to make truthful or even logical statements. Its job is to create "completions" of the text you feed it. It's ultimately doing the same kind of tasks as the spelling-checker and word suggestion feature supplied with modern phone keyboards, except that it's working with deeper and broader linguistic structures.

It has no problems making grammatically correct utterances with no spelling errors, and those utterances are linguistically consistent with the input you feed it and its training data. But it's only operating on linguistic structure, it is oblivious to the underlying logic and truth value of the statements it makes. It may emit text which is logically valid and true, but that's just a fortunate side-effect of the linguistic structure and its training data. In extended passages, it can easily say things that are logically inconsistent. And it has no way to evaluate how relevant its utterances are to the input data, apart from the linguistic consistency. So when it says true and useful things that's just a lucky coincidence.

ChatGPT often makes stupid errors of logic and silly arithmetic errors. OTOH, we have theorem-proving AI that can handle complex logic, and sophisticated mathematical systems that can do arithmetic, algebra, and calculus. In the future, we will surely have natural language software that can handle logic and mathematics, but the current incarnation of ChatGPT won't magically gain those abilities merely by throwing more training data at it.


Stephen Wolfram has written some excellent articles about how ChatGPT works, its limitations, and the potential benefit of combining it with Wolfram | Alpha. Please see What Is ChatGPT Doing … and Why Does It Work? and Wolfram|Alpha as the Way to Bring Computational Knowledge Superpowers to ChatGPT.

Update July 2024: There is now Wolfram GPT, which combines Wolfram Alpha and ChatGPT. Here is its login / signup page.

18
  • 1
    yes with the example you provided in a comment. I think it's beyond perfect but with time it will be improved as the research organization that built it are constantly studying it to discover ways on making it almost perfect which when happens will make SO obsolete
    – user16612111
    Commented Dec 29, 2022 at 13:51
  • 1
    @PubliusFlaviusTiberium OK, but that hasn't happened yet and doesn't appear to be anywhere near happening.
    – ggorlen
    Commented Dec 29, 2022 at 19:08
  • 2
    I'm sorry, @Publius, but I can't tell what "yes with the example you provided in a comment" is supposed to mean, and the rest of your comment isn't very clear, either. Perhaps you disagree with my claim that ChatGPT will not become capable of creating reliable SO answers merely by improving the quality and quantity of its training data, or you simply don't understand my argument regarding that claim.
    – PM 2Ring
    Commented Dec 29, 2022 at 21:37
  • @PM2Ring, the logical fallacies of Open chat gpt, I asked it the same question. if it takes 2 cars hours to travel from DC to Washington how long will it take 4 cars and it replies with 4 hours
    – user16612111
    Commented Dec 30, 2022 at 5:24
  • GPT fixed this bug and it is no longer susceptible to fallacy as a result of testing it with this logic. AI is evolving.
    – user16612111
    Commented Jun 18, 2023 at 9:59
  • 2
    An AI trained on SO will do just that. But anyone who wants to use ChatGPT can just use that. No need to come here to shout about it. I don't under this either SO or ChatGPT. I can use them both if I want to 24/7. Commented Jun 30, 2023 at 11:56
  • 1
    @stackoverblown, what a funny username. My post is valid and GPT is the reason why Stack Overflow is losing millions of traffic, haven't you noticed the debacle between the company and mods because of AI. They need to be concerned because the company needs it's profits
    – user16612111
    Commented Jul 5, 2023 at 16:29
  • @VibrantWaves that's a bit unclear. Are you saying mods are striking because they're concerned that the company is losing profits due to GPT? or that the company made the policy change because they're concerned with losing profits.
    – Kevin B
    Commented Jul 5, 2023 at 16:35
  • 1
    It's clear that the company made this change because they're jumping on the AI Bandwagon like most other large online businesses that own communities. They can't both say they're bringing AI and the community together, while the community has a ban in place on AI generated content. The problem that caused the strike was that this was done in a very improper way, ignoring the problems that we've already identified rather than coming up with a solution that both allows AI generated content to exist while maintaining the quality metrics this site needs to remain relevant.
    – Kevin B
    Commented Jul 5, 2023 at 16:38
  • @Kevin B, it's all connected. The company needs users. Those users posted content generated by GPT, the mods suspended them, then the users stopped visiting and the traffic went down which made the company ask mods not to over apply banning, then the mods went on strike because the company literally told the mods "hey please dont ban them too much we still need them"
    – user16612111
    Commented Jul 5, 2023 at 16:55
  • 2
    @VibrantWaves that's assuming we're losing people due to suspensions, which the data doesn't actually support... that is what the company claimed, though, so we can only assume they believed it at the time. If they had instead approached this and attempted to solve it with the moderators, rather than blindsiding them on a holiday weekend and blaming them for a policy that the company supported for 6 months, the strike would likely have not occurred.
    – Kevin B
    Commented Jul 5, 2023 at 16:58
  • 3
    The policy was meant to be temporary, it was meant to be replaced with a better solution, the problem is the company never worked with us to come up with that solution. it sat stagnant as we deleted thousands of garbage 0 effort copy paste gpt answers weekly. Then they reversed course and went all in with implementing genAI solutions directly into the platform that clearly failed to do what they claimed it would for the very reasons we said it would.
    – Kevin B
    Commented Jul 5, 2023 at 17:01
  • @KevinB, they must have consulted with some experts on the inside and made this move. Other technology companies such as Microsoft have incorporated it into their search engines. They must have been like even if a user pastes content generated by GPT and he quotes the source then there is no problem.
    – user16612111
    Commented Jul 5, 2023 at 17:06
  • 2
    I don’t necessarily disagree with that in principle, but SO isn’t going to give us the tools for that to realistically exist.
    – Kevin B
    Commented Jul 5, 2023 at 17:49
  • 1
    @GeneralGrievance Thanks, but i'm well aware, ;) If i wasn't enjoying these discussions i wouldn't join them
    – Kevin B
    Commented Jul 6, 2023 at 17:57
19

I've been using ChatGPT to answer programming questions for weeks. Unless I've been manipulating it poorly, quite simply, ChatGPT takes a great deal of prodding and pleading to get a complete, correct response from, when a correct response is achievable at all. It's a tool, basically a jacked-up autocomplete that's wrong, say, more than half the time.

For example, here's some code it wrote for an extremely trivial programming task, finding the minimum element in an enumerable in Elixir:

def find_smallest(numbers) do
  Enum.min(numbers)
end

Fine and well. I asked it to translate this code to Dart and it provided this wrong answer[1]:

To solve the problem of finding the smallest number in a list in Dart, you can use the min function from the dart:math library.

Here is an example of how you could implement the find_smallest function in Dart:

import 'dart:math';

int findSmallest(List<int> numbers) {
  return numbers.min();
}

You can then use the expect function from the package:test library to write test cases for the function.

[bunch of test cases that the function fails on]

This is the sort of plausible-looking ChatGPT-based answer you'd see flooding Stack Overflow before the ban.

I'm not a Dart programmer, so after this failed, I looked it up on SO to find the human solution. This isn't an isolated incident. It's the typical workflow using ChatGPT at the present time.

There's no doubt that it's an amazing piece of technology that's correct some of the time. It's helped me quickly solve certain problems over the past few weeks. I don't doubt that it solved the problem you mentioned for you. When successful, it's magic. The problem is, it reveals itself as unreliable once you put it through the ringer on a variety of problems. I'm not talking about 80% accuracy, from what I've seen, more like 40% accuracy.

But technology is advancing and we are becoming more and more independent of human skill to find solutions to programming challenges

Not really. Human skill is just as important as ever. Nowadays, one needs to be extremely judicious about separating fact from fiction and separating plausible-looking solutions from truly correct ones.

Unless large strides are made soon, ChatGPT is nowhere near replacing Stack Overflow or programmers in general any more than self-driving mode will be replacing human drivers.


[1]: I'm aware that asking the same question again, differently or at a later time may give the correct answer for this problem. That doesn't change the fact that it fails catastrophically like this often, on simple tasks.

12
  • 8
    In this comment I linked to a ChatGPT answer posted by a 12k+ member (before the ban). The answer text seems to make sense, and the code looks good, but they're rubbish. The loop should use continue to skip duplicates, but instead it uses break, so the loop ends when it finds the first dupe, and hence the output array is truncated. But statistically, break is much more common than continue...
    – PM 2Ring
    Commented Dec 29, 2022 at 22:04
  • 2
    @Yohst Nobody is stopping you from using GPTs, and you're entirely able to use them instead of SO if you don't like human answers (of course, ironically, GPTs do nothing more than basically cough up those same human answers probabalistically). How does dumping random GPT output on SO help you use GPTs easier or add any value to anything? It certainly degrades the value of SO, and has been shown to degrade the value of LLMs by polluting their training data. It's a lose/lose. The win/win is: use SO for human answers, use GPTs for whatever benefits you accrue from them, separately from SO.
    – ggorlen
    Commented Aug 22, 2023 at 16:13
  • 3
    @Yohst Case in point: check this thread where OP is begging answerers to stop posting GPT output. "Please.. no more ChatGPT. If I want to use ChatGPT I can use it myself :("
    – ggorlen
    Commented Aug 22, 2023 at 16:15
  • 3
    @Yohst You got me. Sorry for misreading. If you're basically agreeing, is the point of your response just to take a dig at the SO community's "attitude"? I didn't realize misreading constituted an attitude problem.
    – ggorlen
    Commented Aug 22, 2023 at 16:41
  • 2
    @Yohst By all means, go ahead and use GPTs if you think they're more valuable than SO. If you find its pandering attitude more appropriate than genuine human interaction, you're entitled to that. I'd personally rather have someone push back against my incorrect assumptions than have a "yes" machine continually tell me how great I am and apologize continually as it keeps feeding me hallucinated lies and failing code. That said, GPTs do have uses, so I'm not interested in creating a false dichotomy. I'm trying to prevent GPT output from overrunning the human resource it's trained on, that's all.
    – ggorlen
    Commented Aug 22, 2023 at 17:32
  • 4
    @Yohst We disagree that there's an attitude problem on SO. There's a meme that SO has "toxic" moderation culture, but my experience that these accusations come from users who misunderstand the culture, barge in with unanswerable, poorly-asked questions and are then shocked by closures, downvotes and comments asking for clarification from volunteers who spend their time trying to build a high-quality resource.
    – ggorlen
    Commented Aug 22, 2023 at 17:36
  • 3
    @Yohst Rarely do I see personal insults or genuine toxicity. Those are against the rules and should be flagged. Most of the time when I see such comments, they're coming from users posting low-quality content, not curators, moderators or subject matter experts. Most "SO has an attitude problem" complainers don't like or understand the site rules and want to treat it like a help desk rather than a Wikipedia-like knowledge resource. GPTs work well for them because they're trained to act like the on-demand help desk servant they expect SO to be.
    – ggorlen
    Commented Aug 22, 2023 at 17:37
  • 3
    @Yohst I agree that SO does have a problem with some answerers who are more interested in reputation than curation. They should be closing duplicate questions rather than re-answering them to gain rep. However, I'm guessing you'll find these closures to be problematic if you're asking the question (most users do, failing to communicate with the SME that closed the question). FWIW, I started using SO shortly after beginning programming and have never had to actually ask a question, because I can always find an answer on the site, all prior to GPTs. Many users don't search before asking.
    – ggorlen
    Commented Aug 22, 2023 at 17:45
  • 2
    @Yohst Again, I don't agree that SO has the attitude blind spots you suggest. The users who feel SO is so toxic probably misunderstand what we're trying to do here. The best benefit GPTs offer is redirecting people who don't look at docs or make an effort to craft high-quality, original content away from SO. GPTs are basically probabalistic doc/SO answer/GitHub code regurgitators. They're being misused as a lazier, more error-prone and environmentally damaging way to find the information that's already out there, with a dollop of saccharine over-politeness on top to make you feel good.
    – ggorlen
    Commented Aug 22, 2023 at 18:23
  • 2
    @Yohst If GPTs are better than SO at fixing most dumb mistakes and helping people access the resources they should be looking at before posting on SO, this seems like a fine scenario, although I still don't quite understand why not consult the docs and existing resources directly. It's also interesting how many people still wind up taking the time to show up on old meta threads to talk about how they've quit SO, GPTs are so much better, SO community is so awful, so glad they never need to talk to humans again, programming is solved, etc. But wait, why are they still here?
    – ggorlen
    Commented Aug 22, 2023 at 18:32
  • 1
    When a point of view is built on provably false assumptions... it's not really worth seeing.
    – Kevin B
    Commented Aug 22, 2023 at 18:45
  • 1
    @Yohst Sorry if this sounds dismissive, but I've been hearing your arguments repeatedly since ChatGPT dropped. All I can say is, great, go ahead and use LLMs. I find zero value in its artificial politeness but if you do, go for it. I know this sounds crazy, but I actually like helping people, learning and giving back to the knowledge repository that gets me unstuck programming every day. GPTs don't offer much of these opportunities for me. The points are fun but not really important. I'd happily move to Codidact if there were more questions there I could answer--I don't have any points there.
    – ggorlen
    Commented Aug 22, 2023 at 18:48
12

Your thoughts and views on this matter are welcome...

Speed yes, accuracy no. One example is not enough to draw any conclusion. Please conduct a more thorough investigation.

The speed of the response generation of the service you used is indeed impressive, although I wouldn't say it's necessarily much faster than using a search engine like Google and clicking on the presented link to Stack Overflow, for example.

Accuracy is unfortunately not yet that good. In same cases it gave acceptable results, but in others not, when I tested it. I cannot recommend it yet.

Why is your and my experience so vastly different? Maybe because you drew a conclusion from only a very small amount of examples (N=1?). That cannot work. There hasn't been done enough example generation and evaluation for the claim to be supported by the evidence.

It's futile to discuss the implications of something we do not know with sufficient certainty yet. We need to study this service more. I'm quite sure that scientists do that already.

Just as a side comment. ChatGPT is competition (capable or incapable) for this platform, but for a knowledge seeker it's just another (useful or not useful) source of information. How useful? The jury is still out.

10
  • 1
    your the first person to acknowledge GPT as a competition to stackoverflow. Let's see how this will affect stackoverflow traffic. Will SO still get 5000 questions a day or will this drop?
    – user16612111
    Commented Dec 31, 2022 at 10:23
  • 1
    @PubliusFlaviusTiberium I'm not the first one really but also not the last one. The future is difficult to predict. I know some people who use ChatGPT for programming but they might not have asked a question here anyway. Let's just wait and see. Not only the number of questions might drop maybe also the number of visits (and this would be the real loss for this platform). Commented Dec 31, 2022 at 10:58
  • but i wouldn't want SO to be knocked off its perch by that site, SO can get back at them by adding new features like animations and more themes to make the site even more exciting
    – user16612111
    Commented Dec 31, 2022 at 11:02
  • 3
    @PubliusFlaviusTiberium What will happen is not necessarily what you want. I don't think there is any imminent risk for SO (more for Google maybe) because SO shines with expert content and high quality content is much more difficult to create than average quality content. On the other hand if that happens I would embrace it. That's progress. I also do not want to go back to write letters instead of sending text messages just because that keeps the postman busy... Commented Dec 31, 2022 at 11:19
  • @evolution. making life easier and easier with technology results in other technologies being obsolete.
    – user16612111
    Commented Dec 31, 2022 at 11:25
  • ("animations" may refer to the slow type out of text, like on ancient 110 baud terminals, telex terminals, and old films (typewriter-like).) Commented Dec 31, 2022 at 15:46
  • 1
    Cathode ray tube television was really cool for a long time until say 20 years ago. Commented Dec 31, 2022 at 16:28
  • I don't know why you would think that a timed display of each character on a post (aka, simulating typing) would be an important addition to the site. It's certainly enticing from a certain perspective, but it doesn't do anything aside from looking weird or fancy depending on where you stand. @PubliusFlaviusTiberium Commented Jan 3, 2023 at 8:58
  • 1
    A scene from the film (at 02 min 14 secs). Commented May 12, 2023 at 15:33
  • And here (at 01 min 21 secs). Commented May 12, 2023 at 15:58
0

Stack Overflow is not going anywhere.

LLMs and Stack Overflow solve different problems.

I think there is this idea that LLMs should be 100% accurate which is not the expectation we have on each other. LLMs are accurate and inaccurate, just like people, but are also useful while still having this issue.

Stack Overflow tries to solve the inaccuracy in people with votes, comments, and community and having the accurate answers bubble to the top. This is needed more in a world where validation of information is lacking in these systems.

So I predict Stack Overflow is not going anywhere any time soon.

5
  • The question is about the long-term effects of ChatGPT on this platform. Can you please speak to that? And a fairer speed comparison is the time it takes Google to get to SO vs the time it takes to come up with a decent prompt for ChatGPT to answer. I have never used ChatGPT for any programming task, and I have only ever asked one question ever on here (which ChatGPT answers incorrectly). Commented Oct 9, 2023 at 14:02
  • You should try it again. Last week I was brainstorming about possible solutions to a problem and I was prompting as I was doing it. The LLM gave some interesting options I didn't think about. I really believe it is going to be a work force multiplier not a replacement. Stackoverflow can have the same path, is my prediction. We have more power but still need validation of information which is the power of stackoverflow. It is not about the number of answer here it is about the right answer bubbling up to the top. Commented Oct 9, 2023 at 15:43
  • I think there is this idea that LLM should be 100% accurate which is not the expectation we have on each other. The only reason have it for the LLM is we think it is a traditional program but it is not. It is better to think of it like a person with accurate and inaccurate information, but also useful. Commented Oct 9, 2023 at 15:45
  • Just to be clear, are you saying that people should be allowed to post ChatGPT content here? They shouldn't. There have been a lot of conversations here and on the SE network about this explaining why. However, I don't have a problem with people using ChatGPT for programming tasks. That's just not my preference. If you're trying to say "both are useful for different things, SO won't be going anywhere any time soon," then sure. That should go into your answer. Commented Oct 9, 2023 at 16:00
  • I'll update my answer as that is what I'm saying, they are useful for different parts of the solution and learning process. Commented Oct 9, 2023 at 16:04
-4

Meanwhile I read frequently in the questions statements like this: "Even ChatGPT was not able to answer my question, so I try it here on SO". Typically it applies to rather basic questions but these questions are not necessarily "bad" or "poor" questions, they are just normal.

So I don't think ChatGPT can replace StackOverflow in the near future.

I asked "What the result of 968416854 + 8694165478653436"

First answer: 8694165479621854 - which is wrong!

The proof (really hard to believe):

enter image description here

5
  • 8
    If you're bad at asking questions then nothing can help you. Asking questions isn't hard, asking good questions is. Any question can be answered, only good questions get a useful answer. That is why we have help center articles for how to ask, that is why Jon Skeet blogged about the perfect question, that is why we have the ask wizard, staging ground. The technology isn't the problem. The demand for instant gratification without any effort within 140 characters or a 15 seconds tiktok is the norm today. You can't beat laziness.
    – rene
    Commented May 15, 2023 at 7:19
  • No technology is perfect just like the human beings who built the AI. No programmer is perfect. Being bad is good for learning and progress, we learn through mistakes.
    – user16612111
    Commented May 15, 2023 at 7:59
  • Just use ChatGPT4 with the Wolfram Alpha plugin, and it will solve all your maths problems in an instant correctly... Commented Jun 18, 2023 at 8:04
  • 1
    Still not convinced, ChatGPT4 even fails on the simple question "Which weekday is today?" Commented Jun 18, 2023 at 11:13
  • Just another poor example (unlike my other questions, it was not my intention to provoke ChatGPT into making a mistake): "It occurred in 2013 and became known as the 'Miracle of Humberside.' An experienced flight instructor named John Wildey was the sole passenger in a small two-seater aircraft when the pilot suffered a sudden heart attack during the flight and passed away. Despite John Wildey himself having no experience as a pilot, he managed to safely land the aircraft, thanks to his calm demeanor and guidance from air traffic control." Commented Jun 18, 2023 at 11:40
-5

ChatGPT will be useful to StackOverflow as the number of poor questions will dramaticaly decrease as they will be redirected to ChatGPT.

The ChatGPT user will be puzzled by a non-working solution and will came here to ask a more intense and minded question.

2
  • 5
    I wish this were the case. There are still tons of zero-effort questions. I would very much welcome if newbies were sent to ChatGPT before asking real humans. Commented Jul 5, 2023 at 18:55
  • @ChristophRackwitz Your wish has been granted. Commented Aug 22 at 2:47
-9

I have used it several times to generate functions or optimise bits of code. My criteria is that I pretty much know how to do something and ChatGPT just saves me actually doing it.

It is also pretty good as answering closed questions about APIs, syntax or other things you would normally look up and read through docs to find.

I have not found it a lot of use in scenarios other that though - although it is less aggressive than a lot of the users on here which is a nice change! :-)

8
  • 1
    less aggressive than the users here, it's called the elitist attitude
    – user16612111
    Commented Jun 22, 2023 at 10:40
  • 5
    It is not OK to be aggressive in a similar fashion that it is not OK to be disrespectful. Such situations should be handled by flagging them. You may have experienced criticism, rather than plain aggressiveness, which is understandable because the site is designed to be a repository of quality Q&A. Granted, many people who are only interested in getting a solution to their problem have very little interest in being also subjected to criticism, which is the main clash of expectations to new users.
    – E_net4
    Commented Jun 22, 2023 at 14:35
  • Po-tay-toe v po-tah-to I guess. It often came across to me as pointless nit-picking rather than constructive criticism. Like people were trying to show how clever they were rather than trying to help people write better questions/answers. However, I am sure that there are examples ranging through the scale of it.
    – Stefan
    Commented Jun 23, 2023 at 12:20
  • 1
    Like here I am not sure why my answer got divebombed? I gave an example of ChatGPT being useful and explained the limitations I have found. Why would that not be useful for people?
    – Stefan
    Commented Jun 23, 2023 at 12:21
  • 2
    The question up for discussion looks to be about the consequences of the emergence of this specific technology as it relates to SO, rather than the merits/demerits of the technology. I think your answer should remark on that. Also, +2/-4 is hardly a "divebomb." The (currently) lowest ranked answer, on the other hand, reads as a propaganda piece and unlikely to be salvageable in any way. Commented Jun 26, 2023 at 17:02
  • 1
    @GeneralGrievance, divebomb as in someone downvoting the answer without leaving any kind of information about how to improve it
    – Stefan
    Commented Jun 27, 2023 at 19:05
  • 1
    @Stefan Ah, OK. I was unclear on what that meant. Thanks for explaining. Commented Jun 27, 2023 at 19:06
  • @GeneralGrievance, no worries - happy to help :-)
    – Stefan
    Commented Jun 28, 2023 at 9:20
-10

It depends with the kind of project you are into. I have discovered that ChatGPT is pretty much useless when it comes to debugging ASP Web development issues.

Like there are two issues where submitting a form resulted in HTTP error 400. ChatGPT took forever to provide responses that didn't work, but the solution was just a single similar Stack Overflow post away. The Stack Overflow post also added some other information that made me understand a view called _ViewImports and the critical role that file plays in the development of ASP MVC web systems.

Stack Overflow still retains its value as it brings together similar experience from users across the globe who also add more information on what they tried which works for some future users like me.

0
-12

Some dudes are shooting the messenger because they don't like the message.

Here is an answer from ChatGPT itself:

While ChatGPT is a powerful tool for answering a wide range of questions, Stack Overflow still serves a valuable purpose as a community-driven platform for technical Q&A. Rather than replacing Stack Overflow, ChatGPT could be used in conjunction with the site to provide additional support and guidance for users.

One possibility would be to integrate ChatGPT's capabilities into Stack Overflow's existing interface, allowing users to ask questions in a conversational format and receive instant responses. This could potentially help users who are struggling to formulate their questions or who are looking for guidance on how to approach a particular problem.

However, it's important to consider the potential drawbacks of such an approach, such as the risk of information overload or the possibility of inaccurate or incomplete responses. Ultimately, any changes to Stack Overflow should be carefully evaluated to ensure that they align with the site's goals of providing accurate, high-quality technical information to users

2
  • 9
    "Some dudes are shooting the messenger because they don't like the message." The downvotes here are on the answer, i.e. the message. The messenger hasn't been shot at all here. However, it might help the answer if the "messenger" would provide some reasoning to why they requested and delivered a message from an uninvolved writer for recipients who could have asked the writer themselves if they cared. Commented Feb 14, 2023 at 15:50
  • I agree with you @MisterMiyagi. Your argument makes sense to me.
    – zell
    Commented Feb 15, 2023 at 11:43
-15

For GPT3 or ChatGPT3.5 I would not have agreed, but for ChatGPT4 and for programming tasks (in my case mostly in R and Python) and the field I work in (biostatistics) I agree. Using ChatGPT4, the answers that come out are for me generally more or less on a par with the quality of an average SO answer, and perhaps even similar to the quality of the currently best accepted answers on Stack Overflow. Here some concrete examples. For example, it was able to answer a 5 year old highly rated question that I awarded several bonuses for, producing working OpenGL+SDL code called via the rdyncall package (which is currently not even on CRAN). Or based on a verbal description of a biological problem (that had not been solved in the academic literature before) ChatGPT4 inferred how to correctly write this as a differential equation system and it was also happy to make further modifications to put in extra biological parameters, and then fired up Wolfram Alpha to determine the long-term equilibrium. Here the question plus part of the answer. enter image description here enter image description here

The answer to this question (reproduced almost verbatim from ChatGPT4, but with output of the ChatGPT R code included & maths verified to be correct) was also better than any of the other available answers.

On Cross Validated / Stats Stackexchange only 8 out of my 20 questions I asked were ever answered (with an answer that I accepted) - 3 of which by myself. That's only a success rate of 25% to get useful help from others that solved my original problem. If I paste the unanswered questions into ChatGPT4 I would count about 83% as useful & largely or entirely correct (i.e. I would have given them a checkmark, whereas the current answers are either beating about the bush, not solving my original problem or are plain wrong & for some there is no answer at all).

Sometimes it takes a bit of prodding to get things right, but so does getting a correct answer on Stack Overflow - that also usually still requires prodding other users posting answers and telling them "no, I would like this or that instead" or "your solution is not working because xxx" etc. I would say that most of the info on ChatGPT here on SO is (1) factually incorrect, (2) highly outdated & referring to ChatGPT3.5, not ChatGPT4 and (3) extremely opinionated. For example, this article citing the poor maths capabilities of ChatGPT applies to ChatGPT3.5, not ChatGPT4, which can now use the Wolfram Alpha plugin and according to recent benchmarks in fact excels at math. And with the use of the ScholarAI & browser plugin it can also give sources. Sure, anything that comes out needs checking. But most current Stack Overflow answers could also use a thorough check & a lot of edits.

So I would see good scope for AI and humans to work together to tilt our capabilities to a new level & get good answers for more questions here on SO. But of course with appropriate guardrails in place to prevent abuse. And mainly in the context of programming related questions, where I would then require reproducible code and output or benchmarks plus citation of a credible source & disclosure of the use of ChatGPT or Github Copilot if it was used, to make sure that any answers are actually correct. I would also not accept plain copy & paste answers. Nowadays most of us will probably be at least inspired by GPT when answering any question really. For those that Google using BING GPT4 is now the default search engine...

6
  • 1
    I told you and its one of the many websites contributing to the drop in traffic on Stack Exchange. GPT can make SE obsolete, especially if they extend the cut off date for the data to late 2022.
    – user16612111
    Commented Jun 18, 2023 at 8:28
  • 2
    Yes of course - the data is clear: meta.stackexchange.com/questions/389928/… trends.google.com/trends/…. Big factor also is the user experience: Stack Overflow comes across as a very hostile platform for many users. ChatGPT4 is always friendly & helpful. Trouble is you can't even bring up this discussion here - any questions remotely supportive of GPT get immediately closed by a couple of opinionated users or moderators... Commented Jun 18, 2023 at 8:36
  • 2
    Yes, the mods down vote any post that is not music to their ears. They even down voted a post made by the VP of the Stack Exchange Community which was based on facts concluded from a research conducted by the company. I saw this a long time ago and warned that if they don't come up with something to make users prefer SE to GPT then the company will lose big time. Traffic is dropping in millions because people opt to ask chat GPT first then only when GPT fails come to SO to ask.
    – user16612111
    Commented Jun 18, 2023 at 9:15
  • 1
    @PassionateSE Yes and those difficult questions that ChatGPT4 cannot solve then will also be unsolvable for the SO community. Right now I already have my difficult questions that could never be solved by SO solved by ChatGPT4... They seem really short-sighted to me. I really hope they are not digging their own grave, because I would like SO to stay around. But with policies like this it will be hard... Commented Jun 18, 2023 at 9:20
  • 1
    Yes, I also had the same experience with a question that involved code to manipulate an image and highlight certain features, unfortunately for SO users don't like to answer hard questions so I resorted to GPT which saved my contract and made sure I got paid a lot of money for thaf software
    – user16612111
    Commented Jun 18, 2023 at 9:21
  • 2
    Isn't this a circular argument? I have no background in the subject matter of this answer. @TomWenseleers is the subject matter expert validating his own answer. ChatGPT was trained on the text of this question and answer from SO. ChatGPT is just saying what he wants to hear only the language is better, that's what it does. There is no originality in the science here.
    – MT1
    Commented Jul 18, 2023 at 7:49
-16

I also verified today that the chat bot cannot beat human experienced responses to programming questions, here is how I verified this, I asked chat GPT a very technical question and it kept displaying the same text over and over again and the solution posted by chat GPT either compiles and refuses to work or the code does not compile. I reached to a developer on quora asked him the same question and his response worked and was straight to the point.

0
-18

ChatGPT largely is Stack Overflow, albeit trimmed and polished to match the user's request. So the ironic thing is, if everyone started using ChatGPT instead of SO, a significant source of ChatGPT's source of knowledge would dry up.

What to do? Stack Overflow should include provision for people to post their questions and answers from ChatGPT. These could be commented upon, and voted up or down as with any post or its replies

Doing this would: 1) benefit the community (instead of remaining private), 2) provide corroboration, correction, and refinement for the poster, and 3) give ChatGPT more knowledge to digest. In this way, we harness the incredible power of ChatGPT, help it improve, and keep the real source of knowledge --which comes from humans-- from drying up.

14
  • 10
    It's already been clearly demonstrated that this can't work, due to what would be an immense requirement for subject-matter experts to review the content and the disparity between the volume of such evaluations and people evaluating the answers solely on if it sounds good, rather than it actually being good. As has been mentioned many times, the nature of the AI generated content and how most people perform evaluations results in the AI content often being upvoted even when disastrously wrong and/or internally inconsistent.
    – Makyen Mod
    Commented Apr 12, 2023 at 22:15
  • However many times something is mentioned has no connection to its truth..unless you're some kind of follower-no-matter-what. That out of the way, people are missing the fact that this is how intelligence itself works. It's how we humans work. The brain is a prediction machine. Check out the work of Jeff Hawkins who pioneered the neuroscience of intelligence and whose work is surely responsible for the boon in AI today. Takeaway: we have to learn to live with the machines; not fight them. So strange to see "programmers" to rebel against the greatest programming feat of humanity. Commented Apr 12, 2023 at 23:14
  • 3
    Sigh Of course the number of times something is repeated doesn't affect its accuracy (if it affects "truth" is a philosophical question). What the fact that it's been mentioned many times and that you don't address the issue in your answer (not even to say you just don't believe the people who have said such things, who very likely have a much better view of what's going on than you do) tends to indicate that you've not actually taken a detailed look at the issue and/or don't have much background in the area and are just stating your personal, not well researched, opinion.
    – Makyen Mod
    Commented Apr 13, 2023 at 0:06
  • 3
    What you need to address with such a suggestion is A) how to resolve the orders of magnitude difference between the level of participation from subject-matter experts which such a plan requires to do what you're claiming, and
    – Makyen Mod
    Commented Apr 13, 2023 at 0:15
  • 7
    B) the disparity between the much lower volume of evaluations by experts (votes/comments) vs evaluation by people who are basing their evaluation primarily upon if the text reads/sounds good, rather than if it's actually correct. As things stand, there are substantially more people evaluating things based on if it reads/sounds good (something at which current AI generated content excels), rather than if it's correct (something which current AI generated content doesn't even attempt to do).
    – Makyen Mod
    Commented Apr 13, 2023 at 0:15
  • 5
    And, while there are some people who are against the concept of AI generation of content, the primary issue here on Stack Overflow is that the current capability of AI generated content is crap from the point of view of providing accurate answers, largely because the current generation of chat bots make absolutely zero attempt at being correct. What they do is string text together that their model predicts goes well together. That's great, and a really good accomplishment, for a chat bot. It's terrible as something to consistently provide an actual, correct answer.
    – Makyen Mod
    Commented Apr 13, 2023 at 0:21
  • 3
    "ChatGPT largely is Stack Overflow" Not really. Yes, SO did form a significant part of the GPT training data on coding-related matters, but so did Github, Wikipedia, Reddit, and various other sources in the Common Crawl dataset. However, even if you train GPT from scratch on nothing but high quality curated perfectly correct data it can still produce incorrect output because it's just operating on syntax. It has no capability to perform logical analysis or to verify the validity of its utterances.
    – PM 2Ring
    Commented Apr 13, 2023 at 2:04
  • I have asked ChapGPT 3 things today. 1) Is disconnecting the negative terminal of a battery a valid way to test an alternator?, 2) How do you setup a 2-dimensional array in PHP?, and 3) How do you return 2 dimensional arrays from php to javascript? It gave me lucid. to-the-point, and (I checked) accurate answers in all 3 cases (all snark-free!). It's on me to verify, but the same is true for whatever I read here on S.O. This comment I am writing now, along with what others have said, will end up in the bowels of ChatGPT. The machine is part of our lives. There is no going back. Commented Apr 13, 2023 at 2:21
  • 4
    ChatGPT is pretty good at answering questions that it has already seen the answer to, yes (AKA: duplicate questions). That makes a generative AI like this a potentially decent way of searching for existing answers. The problem is that it doesn't know which things it doesn't know, and so it won't tell you when it's completely made things up out of thin air. Most humans at least try to only answer questions they think they know the answer to, and they can cite sources to support their answer. If you ask ChatGPT to cite sources, it makes up links that don't exist.
    – Ryan M Mod
    Commented Apr 13, 2023 at 2:29
  • Thank you, Ryan M. That is exactly what I was saying in my answer. If people foolishly abandon public forums like S.O., we will be killing the goose that lays the golden eggs that ChatGPT needs in order to work. Given that ChatGPT is so convenient, people will be tempted to do just that. And that's why I would encourage socially-minded people to not just take their GPT answers and run, but to put them up on places like S.O. for public discussion. That will be good for the posters and for the machine. It would --hopefully-- prevent the sort of "forum desertion" that could tragically occur. Commented Apr 13, 2023 at 2:36
  • I don't think you would care given your arguments, but maybe reading a related post on the subject might be interesting: meta.stackoverflow.com/a/422019/12349101 Commented Apr 13, 2023 at 2:39
  • 2
    Given that GPT is primarily good at answering questions that have already been answered (quite likely on Stack Overflow), I'm not sure I see how putting GPT answers on Stack Overflow would increase the total amount of knowledge contained here. I'd also expect that a lot of the other issues it's good at spotting are the sorts of "debug my code for me" questions that also aren't particularly useful for future users.
    – Ryan M Mod
    Commented Apr 13, 2023 at 2:40
  • Everyone is (rightly) saying that ChatGPT can spit nonsense that sounds intelligible. It's for cases like this that it would be helpful to see public correction. I'd go one step further, the people at OpenAI seem like public spirited types. Maybe Q&A's from their sessions could automatically be put out on public forums (like S.O., if it consented). We have to keep humans in the loop because if people start using ChatGPT for everything and stop publicly talking to one another, the information source will dry up. (Maybe we could get the NSA to start feeding it data...Whoa, just kidding) Commented Apr 13, 2023 at 3:03
  • 1
    "However many times something is mentioned has no connection to its truth.." That also applies to claims - implicit or explicit - that SO has the curation capacities to handle content primarily created by ChatGPT or similar models. In contrast, many of the active curators on SO have voiced their unwillingness to waste their time on such content, and on many other occasions that they are already at their limits - something they can actually accurately describe. Commented Apr 13, 2023 at 4:59