3646

Moderator Note: This question being is still the best tool we have to announce this policy sitewide. However, people have been using this for protracted debate and discussion. As such, this question is now locked. If you want to discuss this policy further, or suggest other related changes, please Ask a New Question and use the tag. Do not comment on the answers instead.

Use of ChatGPT1 generated text for content on Stack Overflow is temporarily banned.

Please see the Help Center article: Why posting GPT and ChatGPT generated answers is not currently acceptable

This is a temporary policy intended to slow down the influx of answers and other content created with ChatGPT. What the final policy will be regarding the use of this and other similar tools is something that will need to be discussed with Stack Overflow staff and, quite likely, here on Meta Stack Overflow.

Overall, because the average rate of getting correct answers from ChatGPT is too low, the posting of answers created by ChatGPT is substantially harmful to the site and to users who are asking and looking for correct answers.

The primary problem is that while the answers which ChatGPT produces have a high rate of being incorrect, they typically look like they might be good and the answers are very easy to produce. There are also many people trying out ChatGPT to create answers, without the expertise or willingness to verify that the answer is correct prior to posting. Because such answers are so easy to produce, a large number of people are posting a lot of answers. The volume of these answers (thousands) and the fact that the answers often require a detailed read by someone with at least some subject matter expertise in order to determine that the answer is actually bad has effectively swamped our volunteer-based quality curation infrastructure.

As such, we need to reduce the volume of these posts and we need to be able to deal with the ones which are posted quickly, which means dealing with users, rather than individual posts.

So, for now, the use of ChatGPT to create posts here on Stack Overflow is not permitted. If a user is believed to have used ChatGPT after the posting of this temporary policy, sanctions will be imposed to prevent them from continuing to post such content, even if the posts would otherwise be acceptable.

NOTE: While the above text focuses on answers, because that's where we're experiencing the largest volume of such content, the ban applies to all content on Stack Overflow, except each user's profile content (e.g., your "About me" text).


1. ChatGPT is an Artificial Intelligence based chat bot by OpenAI, which was announced on 2022-11-30. Use of ChatGPT is currently available to the public for free.

1
  • 1
    Comments have been moved to chat; please do not continue the discussion here. Before posting a comment below this one, please review the purposes of comments. Comments that do not request clarification or suggest improvements usually belong as an answer, or in Stack Overflow Chat. Comments continuing discussion may be removed.
    – Samuel Liew Mod
    Feb 26 at 7:28

50 Answers 50

1
2
-13

The more tricky question is: should paraphrases of GPT-whatsit-generated verbiage be banned?

2
  • 11
    Yes, just like with plagiarism, direct paraphrasing of AI generated content isn't permitted. We've detected and appropriately handled a substantial number of posts and users where there was considerable effort made to hide/obfuscate that the post content was AI generated. Is detection of such posts perfect? Of course, not. However, moderators are likely going to be less lenient for a user that has clearly demonstrated, by trying to hide/obfuscate, that they know they are doing something that's against the rules and at the same time caused moderators and users doing curation to spend more time.
    – Makyen Mod
    Mar 29 at 17:43
  • 3
    whatsit: "(chiefly UK) A thing (used in a vague way to refer to something whose name one cannot recall, or that one is embarrassed to say)" Mar 29 at 18:14
-17

This is a good decision for the future of mankind. Some knowledge on the Internet should only be created by human communities of qualified professionals on specialized services such as Stack Overflow.

AI can also be used for generation, as an assistant to each person, but not as an independent person creating content. People should be responsible for the final product of labor, not robots. Someone in society must take knowledge very seriously and produce originals. After all, AI has to learn from someone.

It would be stupid and dangerous to allow AI to publish here - we will quickly drown in the world of "secondhand" generated content. We need beacons of knowledge.

I believe the day when Stack Overflow brings robots in the generation of answers is the day the service begins to end. Robots will flood the answers, many people will leave, not knowledge carriers will remain, but AI curators, the quality of information will fall, and the site will lose its relevance. Yet Stack Overflow needs to think carefully about how to compete with services that will actively use AI...

-17

I honestly believe ChatGPT is a powerful tool, but in reality it doesn't give exactly what we want. All the answers posted are from real legends who put their hands on the code and tried the solution. I would say it would be disrespectful to mix human answers and AI answers. Even the AI is trained from human answers.

1
  • 8
    ChatGPT is insanely great and is a powerful tool, but it will also produce answers that are totally wrong and/or made up. It can not be trusted. Mar 30 at 13:33
-24

Yes, it should be banned. To the question of "how do we identify those posts?" it should be considered that this problem is not new or unique to SO. Plagiarism is a concern that spans broadly. An answer found in academia is to copy/paste answers back into ChatGPT and it will respond to it as a continuation of the conversation, flagging it as AI-generated.

This is a potential technique, and one that scales. Multiple suggestions here implying that "you can tell it by looking at it" isn't all that helpful because we can't expect people to reliably keep up with the potentially exponential flow of spam answers.

8
  • 15
    No, that's not at all accurate. ChatGPT just pretends. It can't recognize its own output.
    – Cerbrus
    Jan 18 at 1:07
  • 11
    The conversation chains in ChatGPT are separate. Every time you start a new chat, it keeps that session for that session only. I've seen nothing to imply that it persists any information cross-sessions. In fact, what I do for amusement is ask ChatGPT where my hometown is located and it gives me a new, amusing, and mostly wrong information every time. You can try to correct it but if I start a new session it goes back to the nonsensical information again.
    – VLAZ
    Jan 18 at 7:38
  • Re "...is to copy/paste answers back into ChatGPT and it will respond to it as a continuation of the conversation": Interesting. Do you have some examples and/or references for that? Jan 18 at 18:55
  • @PeterMortensen openai-openai-detector.hf.space
    – n8.
    Jan 19 at 16:40
  • Also this: nypost.com/2022/12/26/…
    – n8.
    Jan 19 at 16:42
  • My main point is that SO isn't the only one suffering from this issue, and to try and concoct a solution internally is wacky. Smart people are already very far down this road, we have search engines to identify who they are.
    – n8.
    Jan 19 at 16:43
  • 7
    this just isn't really relevant to the conversation, tbh. We aren't expecting users to go out and look for chatgpt answers to report. That's not what this is about. Mods and other involved users are already on the case and already know how to find such answers and deal with them. Obviously, report things you think are chatgpt answers, but we don't need this to be a witch hunt.
    – Kevin B
    Jan 19 at 16:51
  • 4
    @n8. If that is your main point, you might want to edit your answer to actually say so. Right now the prominent focus is on a single suggestion that doesn’t actually work as described. Jan 19 at 18:35
-24

Why not have all new questions include an automatic answer by an official Stack Overflow ChatGPT account, with a clear indication that this is the ChatGPT response? Maybe even show the user the ChatGPT answer before the question is posted, to reduce duplicate/low-quality questions.

This way, it just gets the ChatGPT controversy out of the way... ironically, by embracing it. If the answer works, then great. If it doesn't work, well now at least there is a Stack Overflow sanctioned answer written by ChatGPT to compare new answers against. But if there's already a ChatGPT answer, why would someone answer it with another ChatGPT answer?

If the problem is users abusing questions with quick, low quality answers... well that's the problem then, with those users, who will always exist.

6
  • 7
    Because, to quote the question post: Overall, because the average rate of getting correct answers from ChatGPT is too low, the posting of answers created by ChatGPT is substantially harmful to the site and to users who are asking and looking for correct answers. Mar 30 at 0:50
  • 15
    I don't get where you people are getting "the ChatGPT answer". There are multiple requests for this but they all fail at that point because ChatGPT doesn't give one answer. It generates statistically plausible text which can be different answers. Even if we assume that SO will show a ChatGPT answer at asking time and/or after posting a question, that answer need not be the same as you'd get from generating again. The claim that it would discourage others from posting a ChatGPT generated answer is basically unbacked by anything as it's not just the same text spewed out.
    – VLAZ
    Mar 30 at 4:44
  • 8
    For instance, here are two different ChatGPT responses to your suggestion. They're similar-ish, but about 20 seconds of editing could eliminate the most obvious parts. Also, an answer being posted already definitely doesn't stop people from posting the same general idea again.
    – Ryan M Mod
    Mar 30 at 4:52
  • 4
    Self-contradictory (and wrong) within the same sentence: "Is the correct spelling "StackOverflow"?". Response: "Yes, that is correct. "Stack Overflow" is a popular question and answer website for programmers, and the correct spelling is "StackOverflow."" Mar 30 at 13:40
  • And it only wants to please you (right or wrong): "Are you sure about the spelling?". Response: "I apologize for the confusion. While "StackOverflow" is a common way of writing the name of the website, the correct spelling is actually "Stack Overflow" with a space between the two words. Thank you for bringing this to my attention." Mar 30 at 14:03
  • I'd support this. Seeing a GPT generated "here is an AI's best attempt" next to user created content would be good for the site. May 3 at 19:56
-25

A Solution?

I agree to most other answers, except the "but there is no solution" part. Also, I believe not all posters here are understanding, that we're just at the beginning.

Hence, my proposal would be to attack, instead of to defend.

Why not enable a feature, that sends all questions to ChatGPT right after posting and display the result alongside the answer? It should be marked as the ChatGPT answer and users could opt to not display it.

  • This would immediately stop people from abusing ChatGPT to farm credits. The similarity would be too obvious. At least for the case, where the question is just copy-pasted. If ChatGPT users enhance the question to improve the response, they already added some value and would not be in rapid fire mode anymore.

  • It would give the benefit of the doubt, that an AI answer might actually be valuable. By rating those answers the same way as rating human answers, we can see how they rank with others.

  • Humans who write answers can refer to it and agree or disagree, if that makes any sense. They can point out, if there is only a minor mistake in the AI answer or where the answer is based on a misunderstanding or predominant misconception in the internet (as the source of information).

I think this solution would scale for some time to come, but I am not sure, how feasible that is. Will SO be charged, or can SO sell this to openai as a marketing hack? I don't know.

8
  • 13
    No. This has been suggested plenty of times already. Look at the other answers here as to why this can't work.
    – Cerbrus
    Mar 3 at 17:07
  • would you care to point me at it, because I didn't see it.
    – Ingo
    Mar 3 at 17:10
  • @Cerbrus, just found it on the next side, sorry for not being thorough in the first place. It seems indeed, that the solution could be difficult, but I am not entirely on your side. In the long run, ChatGPT will become less expensive and in the short term, it may wish to run this as advertisment the same way I can go there and ask questions to it.
    – Ingo
    Mar 3 at 17:20
  • For the point with repeated questions yield different answers: I would assume that the different answer will be somewhat suffering from the same problem. Might be difficult to compare word by word, but might be good enough to discourage abuse.
    – Ingo
    Mar 3 at 17:22
  • 12
    ChatGPT can offer completely different answers to the same question, including absolute contradictions of what it said mere seconds ago. The similarities between pairs of ChatGPT answers are structural in nature, not content based; there is no use in having a "reference answer" to spot other generated answers for the same question. Mar 3 at 17:39
  • 7
    "… in the short term, it may wish to run this as advertisment the same way I can go there and ask questions to it." That would be a rather poor advertisement. ChatGPT isn’t made nor meant for the kind of questions SO is made for. Expecting experts to waste their time trying to curate a stream of technical nonsense isn’t a winning story… (This is in essence something this very meta-question already said - there is just no capacity to manually vet all the content that ChatGPT has generated, let alone could generate, for SO.) Mar 3 at 17:46
  • 1
    gotcha, and to be honest, I didn't expect the answers would be 'contradicting' and that sounds like a general flaw to me. It would at least make sense, if ChatGPT would enhance itself based on the content it receives, but I was not able to observe any valuable learning, based on my feedback. Even in the most stupid way.
    – Ingo
    Mar 6 at 16:50
  • 9
    @Ingo ChatGPT doesn't "remember" the conversations it has. As I said here "ChatGPT generates plausible text, consistent with its training data and the prompt, but it doesn't know what it's talking about, and it has no way of representing or evaluating the truth of its utterances. Yes, it can say true things, but it can also say complete nonsense, and it can't tell the difference". It's designed to manipulate syntax, not semantics. Stephen Wolfram gives a good outline of how it works in the first of his articles linked in my answer.
    – PM 2Ring
    Mar 8 at 14:20
-27

I think Peter's answer has the right spirit, but the wrong implementation. We might need to get comfortable with this technology being around in the long term, and one way to stop the abuse it might generate is to build it into the system with the necessary precautions and abuse prevention mechanisms.

Make no mistake: the genie is out of the bottle. You can't put it back in. You can't wish it away. This is going to be a thing going forward, and it doesn't even have to be a problem.

Basically, you create some kind of system user that posts an AI-generated answer to ~every question. (Maybe ask the bot if this looks like the sort of question that belongs on Stack Overflow first, so you don't start automatically answering obviously-off-topic questions like "How is babby formed?", although that can be gamed and that's a concern.) You mark it as a bot. You surround the answer with the necessary warnings. Its answers are all community wiki, meaning the bot never gains any reputation and users are encouraged to edit the answer if it's only slightly wrong. The bot automatically deletes its own answers if they get a low enough score. Other people trying to run a ChatGPT Stack Overflow gold rush for Internet brownie points would find themselves unable to keep up with this system user, and would stop trying.

You run this for a while, determine if it's successful or not, and... if I was your CFO, I'd better hope it's not, because as other comments show, this might not be a very cheap model to run (even if you had a license from the creators to run it for $0). I can't imagine Stack Overflow staying sustainable as a commercial entity if it has to pay an AI tax on every single question on the website.

The good news is that, if this does become too expensive for Stack Overflow, it's gonna also be too expensive for random Joe's trying to gain cheap reputation this way — and that probably also means the end of the gold rush of people who just want to write "make number go up with AI" blog posts.

48
  • 10
    I'd like to point out that OpenAI probably pays big time right now for the server, GPU and memory and electricity it uses, not even considering the training costs (these typically range in the lower single-digit millions for complex language models). An OpenAI-paid-for ChatGPT answer is going to be really expensive if someone actually needs to pay for it. (We're talking spacefaring carbuilding lunatic billionaire-sized venture capital being burnt here.) Dec 5, 2022 at 11:37
  • 3
    Yes, I addressed that point towards the end. It's possible that, even if successful, this trial can't work for financial reasons. The good news is that, if this does become too expensive for Stack Overflow, it's gonna also be too expensive for random Joe's trying to gain cheap reputation this way. My understanding is that the problem, for now, is the gold rush of random Joe's trying to get a number to go up.
    – badp
    Dec 5, 2022 at 11:40
  • 5
    ChatGPT does not give a canonical answer to a certain question. If you ask the same question multiple times, you will get multiple different answers that sometimes even make conflicting statements. There seems to be a random element used when the AI generates its answer.
    – NineBerry
    Dec 5, 2022 at 11:55
  • 4
    So an automated bot dumps crap answers on questions... You're missing one glaring problem: ChatGPT doesn't return the same answer for the exact same question twice, so any user that probably also doesn't use the exact same input will absolutely get a different answer... This solution can't solve anything, other than a overflowing wallet.
    – Cerbrus
    Dec 5, 2022 at 11:55
  • 10
    "if this does become too expensive for Stack Overflow, it's gonna also be too expensive for random Joe's trying to gain cheap reputation this way" A thousand users with a thousand free / trial accounts can posts tens of thousands of generated answers... SE, a single entity, will have to fork over cash for every single answer they automatically generate. You can't compare corporate usage with individual usage.
    – Cerbrus
    Dec 5, 2022 at 12:01
  • 9
    This would defeat the purpose of Stack Overflow. It actually sounds like a completely different service. GH Copilot is doing something like that already, doesn't it?
    – Dharman Mod
    Dec 5, 2022 at 12:25
  • 9
    @badp No, I am saying that the purpose of Stack Overflow is to provide quality answers to common problems. These answers can then be found by humans searching for the same issues. A bot answering every question on Stack Overflow would go against the purpose of the site. We already have this issue with some users who add a code-only answer or repeat the same solution on multiple questions.
    – Dharman Mod
    Dec 5, 2022 at 12:32
  • 4
    @badp Because they looked for it already on Google. Asking a question on Stack Overflow is the last resort. It means that a question like that hasn't been asked by anyone else yet.
    – Dharman Mod
    Dec 5, 2022 at 12:39
  • 3
    @Dharman that's desired user behaviour, not actual user behaviour. If things were that easy, we'd never need to close questions as duplicate.
    – badp
    Dec 5, 2022 at 12:40
  • 19
    The problem this ban is meant to solve is that ChatGPT can produce answers in seconds which require minutes of multiple people's time to verify if they are worth having on the site or not, and that is a waste of time when a large proportion of such answers are not worth having on the site. If every question automatically received an answer like that, it would make the problem worse, not better, by requiring more people to spend more time verifying and voting on those answers.
    – kaya3
    Dec 5, 2022 at 13:09
  • 4
    There are many questions on Stack Overflow which the person asking cannot try out first hand and see if they are correct. Consider a question like "will a linked list be more efficient than an array in this case?" with an answer like "a linked list will be more efficient because most of the operations are at the start of the list". If the OP was able to try it out and see which was more efficient then they wouldn't have asked the question, and the answer doesn't tell them how to try it out. "What is the time complexity of this algorithm?" is another class of such questions, which ...
    – kaya3
    Dec 5, 2022 at 14:05
  • 4
    ... Stack Overflow already has a lot of users posting (non-AI-written) answers that "seem right", that the OP may be satisfied with, but are wrong in ways which cannot be demonstrated by "trying them out" (and this often makes it hard to convince the person who wrote the answer that they are wrong, too). Bad answers like that waste a lot of people's time, the site needs less of them, not more. As for "voting is integral to the site", of course it is, but the site is better off if experts spend their time writing good answers, instead of checking and downvoting rubbish answers.
    – kaya3
    Dec 5, 2022 at 14:08
  • 4
    That presupposes that there are rubbish answers which need to be downvoted in order to be removed from sight. But what you're proposing is to create those rubbish answers, automatically, en masse.
    – kaya3
    Dec 5, 2022 at 14:37
  • 4
    @badp If the answers weren't rubbish, this wouldn't have been a problem in the first place. Duh. The whole entire friggin issue here is users posting AI-generated content en masse, without checking the answers.
    – Cerbrus
    Dec 5, 2022 at 14:43
  • 5
    A super power people crave is being able to express themselves better. I think you're right in that this isn't going away, and if implemented as assistive instead of prescriptive, could result in people writing way better questions. I also see the potential to help people write answers too, but perhaps as a coach and not a co-author. I think it could do what guided "wizards" just don't do very well.
    – Tim Post
    Dec 5, 2022 at 18:41
-27

Instead of banning ChatGPT, Stack Overflow could consider adding a new button "Show AI Generated Answer(s)" to each question page. Such answers should be hidden by default, but if anybody is curious about what ChatGPT (or any other future AI tool of the day) has to say about it, they can check it out.

Also, such AI-generated answers could be compared to any "human"-generated answers with another AI tool to compute a similarity score, and if a "human"-generated answer is too similar to an AI-generated one, then such an answer should be banned (since the probability that such "human" answer has actually been produced using an AI tool would be high) and the user who posted it could be given some penalty.

So, this approach would kill two birds with one stone: people would much less likely post AI-generated answers since such answers are already there automatically, and Stack Overflow would get an automatic tool for detecting AI-generated answers and for punishing users who abuse the "no AI-generated answers" policy.

5
  • 12
    No? People aren't going to go through the rigor of ensuring that the answer they see or read is comprehensible or useful for their use case. We already have cases of people copying code from Stack Overflow and putting it into production. This doesn't make that better; it actually makes it orders of magnitude worse.
    – Makoto
    Feb 1 at 21:47
  • 9
    People already barely follow instructions like How to Ask and How to Answer. The problem we see with lots of AI-generated answers is that the people who post them do so blindly, without verifying that it answers the question or testing for bugs. I doubt asking these low-effort posters to put in more effort to self-identify (even though it's barely any effort to check a box) is going to have the effect you desire. Until such time as AI-generated answers are reliably correct, if people are interested in what the AI has to say, they can go talk to it at the AI's website. Feb 2 at 0:08
  • I don't think this is a bad idea. You cannot stop anyone from looking for AI-generated answers. The point is not to be vindictive of AI generate responses but to let others decide if those are accurate or even helpful. In my experience apart from answering basic questions, pre-2021, ChatGTP has not provided any relevance. This data can also be useful in future to train better models.
    – Rahul
    Feb 7 at 22:37
  • 4
    if people want AI generated answers, there's a place for that. (it isn't here)
    – Kevin B
    Feb 7 at 22:52
  • Not sure why this was downvoted but i can understand why when everyone are so angry and ban-happy already... This is the least-effort and also most realistic way of fixing the problem now and forever. It will always work too, because chatgpt will always produce grammatically correct answers(even if completely wrong) and a similarity algo should be feasible to implement(ie: not too complex). It is also the only answer where it saves SO users from cleaning up and saves SO from becoming more authoritarian(it already is too authoritarian for many users...)
    – n00p
    Feb 24 at 7:30
-28

I think an outright ban is silly and counterproductive. If the problem is that bots would post too many unverified answers in a short amount of time then this could easily be solved by putting a limit on how many answers users are allowed to post in a short time. If one uses ChatGPT to generate an answer and then independently verify it and correct it slightly if needed to make it run, that should definitely be allowed... One could always still add a checkbox "At least partially generated by GPT" or something like that, so that everyone is at least open about it.

Here I used GPT-3 Codex to translate a bit of R code to Rcpp, and it made the code run 15x faster (and I verified it actually worked): Faster way to calculate the Hessian / Fisher Information Matrix of a nnet::multinom multinomial regression in R using Rcpp & Kronecker products. As an Rcpp beginner this would definitely have taken me longer to do by hand...

In any case, the OP surely also checks if any given answer is the correct one and if the code given is actually working.

The proposed solution above to ban the use of ChatGPT altogether is also unenforceable. In the time it would take to find out if a given answer has been produced by ChatGPT one could verify if the given answer was actually correct and the given code was working. The latter would arguably be quite a bit more useful...

Besides, I think it is only a temporary problem that some are now using the free OpenAI ChatGPT demo to get answers easily that they can post to increase their reputation. This will surely stop as soon as they have to pay to use ChatGPT...

In any case, I am already finding GPT-3 Codex and ChatGPT almost as useful as Stack Overflow and Cross Validated. All of them are also mostly targeted to giving answers to already well worked out problems. For none of them I ever received answers to as yet unsolved problems, e.g., for problems still requiring significant theoretical work.

25
  • 2
    Rate-limiting is proposed in this other answer - perhaps you might like to join the discussion there and see what issues have been raised about that proposal.
    – kaya3
    Dec 5, 2022 at 14:55
  • 4
    "and then independently verify it and correct it slightly if needed to make it run" That is allowed. The problem is users copy-pasting CGPT answers in bulk, without any verification or testing. Also, rate limits don't fix anything, it just makes people wait, and not just the abusive users.
    – Cerbrus
    Dec 5, 2022 at 14:55
  • 5
    And blatantly ignoring the referencing requirements Dec 5, 2022 at 14:58
  • 9
    If people would have verified that their answers are correct, we wouldn't have the whole discussion. Unfortunately, they didn't. The amount of additional work caused by them (again unfortunately) outweights the advantages. It would also help if more people would participate in moderation (review queues, down-votes), but I also don't see that happening.
    – BDL
    Dec 5, 2022 at 14:59
  • 1
    Well but whoever posts the original question will verify the correctness of a given answer no? Do you even need moderators for that? Whenever I post a question I would never just check an answer for being the correct one without actually checking whether it actually works... Dec 5, 2022 at 15:02
  • 4
    Many answers cannot be verified by the person who posted the question, because determining the correctness of an answer is as difficult as writing an answer. Most questions that depend on some kind of theoretical understanding are like this.
    – kaya3
    Dec 5, 2022 at 15:04
  • 5
    And answers might "work", but use terribly outdated code or a horrible approach at the problem. How would the user asking the question know?
    – Cerbrus
    Dec 5, 2022 at 15:05
  • 4
    There's also the answers that invite people to use eval or otherwise create security problems, even though the answers verifiably "work".
    – kaya3
    Dec 5, 2022 at 15:06
  • 3
    None of this is really my experience with using GPT-3 Codex or chatgpt. Most of the answers were on a par with the answers I see on this site. Sometimes the code only works after some minor edits. But that's not dissimilar to many of the answers posted here all the time (and which then tend to get downvoted). Dec 5, 2022 at 15:08
  • 2
    @kaya3 interestingly it does indeed suggest eval on "javascript how can I rune code from a string?", but at least it also tells you eval is a bad idea... It suggests the Function constructor instead -.-
    – Cerbrus
    Dec 5, 2022 at 15:08
  • 1
    Not really. I tested it for quite a bit and was really impressed with the results. Translating from one programming language to another works very well for example. This is how it codes a Mandelbrot fractal in R: twitter.com/TWenseleers/status/1558988898688188416 & this is how it does it in Rcpp using OpenMP: twitter.com/TWenseleers/status/1559001987148038144. Not bad, is it? Dec 5, 2022 at 15:11
  • 2
    Sure - calculating a Mandelbrot fractal is a solved problem. But most of the questions asked on Stack Overflow are solved problems too - in many cases fairly trivial problems in fact. This code in any case was not just lifted from somewhere. That's not how this language model works anyway - it does not just copy and paste stuff from somewhere... It is true it is better in some languages than others. But translating across languages tends to work very well. Which is a common Stack Overflow question. In computer language X I can do this, how do I do that in computer language Y, etc... Dec 5, 2022 at 15:22
  • 7
    Okay, so you found a usecase that this AI is good at. Now to find a solution for users not using it correctly, not checking it's output, and just dumping answer after answer on SO in the grind for rep.
    – Cerbrus
    Dec 5, 2022 at 15:26
  • 12
    As previously mentioned numerous times, there are indicators that are dead giveaways ChatGPT produced it. There's numerous of them, and they're constantly being detected in a backroom to aid with detection. We then delete it and any other CGPT answers by that user, sanction the user, move on to the next user. We do not check for accuracy because, right now, it's irrelevant. it's a blanket ban while we sort out the concrete, long-term enforcible rules Dec 5, 2022 at 17:10
  • 5
    @PeterCordes normally, yes. Right now, no. It's a blanket ban with no exceptions because people didn't follow the existing rule framework for AI-generated content Dec 7, 2022 at 18:12
-31

21/03/2023 edit

As GPT-4 is available for 20$/month, shouldn't you rewrite your policy?


ChatGPT is actually training. Given the fact that it gives wrong code (sometimes), it is normal not to accept answers exclusively written by the AI.

But this is only the beginning.

You are at the same point that chess players were when computer engines entered the board and started to play chess at a human level.

Chess adapted to engines, and Stack Overflow should adapt to AI coding systems, perhaps develop its own AI system (they have the site content to train its project; they could open it as an account subject to human democracy).

A few decades later the chess engines have an Elo of 3300, while Magnus Carlsen is rated 2900, so there will come a day when users will prefer to ask their code problems to an AI rather than a human (maybe soon). This will rid the site of bad questions that are not well received and not answered. It will also help the site to evaluate its own content to detect errors (perhaps adding a warning: "The AI has detected a problem in this answer").

There are still chess players who would prefer that there weren't any engines, but by now chess engines have become friends of most chess players.

The AI coding systems should also become a friend of the community of programmers.

You are complaining about a first attempt, ChatGPT, the first project open to the public. It is important AI is available for everybody, included the programming community.

45
  • 27
    How does this actually relate to the ban? It reads mostly like a vague outlook into an entirely hypothetical future which isn't there yet and frankly quite overoptimistic. Mar 14 at 14:51
  • 16
    We don't dabble in speculation here. Come back when AI has gotten batter.
    – Cerbrus
    Mar 14 at 14:57
  • 5
    We can discuss that when that "later" actually comes and we know what those "later AI" actually are. From what has been discussed countless times already, I would wager no: If there is an AI that can just generate high-quality answers, there is no need to maintain a repository of those answers. Mar 14 at 14:57
  • 13
    Chess is a whole different problem than natural language is. Solving questions it yet another different problem. Just because AI has gotten good at chess, doesn't mean it's gonna get just as good, just as fast, at solving problems. You're way overestimating the power of a language model.
    – Cerbrus
    Mar 14 at 15:09
  • 12
    Why are we even discussing this again? All of the points in this answer have been covered already in older answers and discussions here.
    – Cerbrus
    Mar 14 at 15:10
  • 22
    "You are complaining about a first attempt..." No, definitely not, we've made a policy about ChatGPT and AI generated content as it exists today, which is nearly 60 years after Eliza bot was released. While ChatGPT is a substantial improvement, it still lacks entire categories of functionality that would be necessary for it to consistently answer questions correctly, or even without being hilariously wrong and self-contradictory. The hurtles which AI still needs to surmount in order to be truly useful for answering questions are hard and will take substantial time.
    – Makyen Mod
    Mar 14 at 15:11
  • 5
    A Spanish paywalled article? Yea, no way I'm ever gonna read that.
    – Cerbrus
    Mar 14 at 15:14
  • 10
    The current ChatGPT isn't OpenAI's first release. They were incorporated in 2015. So, look at where they were then and where they are now. From that, with hopefully some acceleration factor, you could estimate how much longer it will take, but even an estimate based on that would be very optimistic as to the time-frame for improvement. Will there be improvements? Of course there will. There nearly always are. The issue is that it's gotten to the point where it's a problem for us and still has a long way to go until it's useful for what SO does. We're dealing with it as it is now.
    – Makyen Mod
    Mar 14 at 15:21
  • 14
    @Universal_learner I recommend to read up on how such an AI works, then re-annecdotes the examples you have just given. Those are quite frankly terrifying. Mar 14 at 15:35
  • 16
    I mean, that's part of the problem. You look at it and see what you think are good answers/solutions... but you don't necessarily know what a good answer/solution is. New devs using this tool are running with answers from an AI that isn't built to provide accurate/good coding advice.
    – Kevin B
    Mar 14 at 16:20
  • 11
    @Universal_learner Better than 54% means being totally average. Average for people that actually bother to do such exercises, i.e. many, many beginners. Exercises made specifically to be self-contained bites of challenges. Challenges with a huge body of available solutions. None of these match what Stack Overflow is striving for. Mar 21 at 9:09
  • 5
    @Universal_learner the ban on ChatGPT is not analogous to a ban on Stockfish being used as a tool to write and review the lines contained in chess books. It's equivalent to a ban on AI-generated chess books, which aren't banned, but would be of no use to anyone even at a task where computers are much better than at writing code. "Solve me this tactics" doesn't really have an equivalent programming question type, but if it did, the difference would be that Stockfish actually gets the right answer pretty much all the time, while ChatGPT does not.
    – David
    Mar 21 at 10:10
  • 11
    @Universal_learner good for you. Just keep in mind that obviously wrong code is better than code that tricks you into thinking it's right but actually isn't.
    – David
    Mar 21 at 10:30
  • 14
    Why do you care about this at all? if gpt or whatever does it for you, just use that? no point trying this site to be anything else. Mar 21 at 15:24
  • 7
    "ChatGPT is actually training" to be clear, ChatGPT 3 doesn't use chats you have with it as its training data. You can't "teach" it something new and have it be able to use that new knowledge in separate conversations. Probably not an important point at this point but I thought I'd point it out. Mar 22 at 9:52
-32

I was reading this post extensively and I'm really worried by the reactions.

The first thing I remembered was cab driver's reaction when Uber came to my town. They reacted extremely angrily. They got together, persecuted Uber drivers. They used lawfare and political connections against Uber drivers, and they got to physical fights with Uber drivers to the point Uber drivers initially couldn't reveal themselves when they picked up a client because cab drivers where constantly looking for them and picking fights with them and their clients.

This scenery lasted a couple of months until they realized the inevitability of their fate. Some cab corporations even tried to educate cab drivers to give candies and treat client the best way possible. Nothing could resist Uber and today there are very few cab drivers resisting in my town.

Now when I read this thread I notice some very worrying trends:

First the level of ChatGPT answers on this matter (most upvoted answers) just shows how advanced it is. The sarcastic answer was terrifying.

Second, I saw that most people see the ban as the correct option, without having a reasonable way of distinguishing AI-generated answers from human answers. I think there are only two possible ways: letting users decide if it's an AI generated answer or having direct help from OpenAI itself. But I really don't think that humans will have the ability to tell one from the other. That leaves us with the only option of asking OpenAI for help. Has anyone contacted them yet?

Then the level of harshness with those who advocated ChatGPT integration (most downvoted answers) only reinforced the memory of cab drivers reaction. This worries me the most because disruptive technologies have to be embraced from the start or things will only get worse.

Adding to this is a very compelling pro-AI factor: the fact that some users are really fed up with aggressive answers from humans in SO and would much rather prefer to interact with a AI that treated them good. This is getting so critical that some people left SO altogether. I live in Brazil and I don't use the Portuguese SO because of extreme rudeness I got there several times. English SO is less bad, but I can assure you that if there were any other options people would embrace them in a heartbeat.

Just like I and many other people were fed up by cab drivers unethical attitudes like trying to figure out if the users knew the town so they could make a longer paths to the destination, not giving correct change, rudeness and many other things. That made Uber irresistible. As soon as it was available I never ever used a cab again. Since the first day it was on town.

Finally, remember that ChatGPT is learning and its answers will only get better and better. What are wrong or bad answers now will probably be the best answers in the future.

My advice (which will make my answer quickly get to most downvoted): If we can't get OpenAI to help, integration with ChatGPT is the only possible option. Create a clearly labeled automated answer for each post from ChatGPT and let users downvote it if it's bad as with any normal user.

This way users will have an immediate answer they know was AI generated and they will know there is a greater risk of being wrong, just like we know with automated translations of text.

I know this will be unpopular because it will make more difficult to build reputation points, especially if ChatGPT improves its answers. It's still better than losing all SO or making fruitless attempts to differentiate AI answers from human answers.

Any other option will not stand this test.

Maybe this means that some time in the near future SO will be no longer relevant because you can just ask an AI what the problem is with no need for human interaction. Well, if that is the case SO is already doomed and needs to rethink its business model from the scratch. If that's the case, it's better to embrace it as soon as possible. Humans can always help with comments and corrections at least while it still generates wrong or bad answers. But if it gets really good at it, there is no possible future for SO.

If you can't beat them, join them - a popular proverb.

Resistance is futile, you will be assimilated - Borgs.

18
  • 18
    "remember that ChatGPT is learning and it's answers will only get better and better. What are wrong or bad answers now will probably be the best answers in the future." then why don't we discuss this in the future, rather than the present. We act on what we have right now. And right now ChatGPT can generate content that is very wrong and potentially dangerous. Which is a big part of the reason why it was banned.
    – VLAZ
    Mar 16 at 17:14
  • 8
    Instead of comparing people to physically violent folks, I recommend to actually acknowledge and address the points that have been brought up for the umpteenth time already. Mar 16 at 17:15
  • 25
    Uber vs cab drivers is a flawed analogy. Using an Uber, you get to your destination, just like a cab. If you have an "answer" provided by ChatGPT or other AI generation (at the current level of capability), you don't have an actual answer. You have "eloquent bullshit" that sounds like an answer. It is, sometimes, an answer, but it's quite likely to be hilariously wrong, self-contradictory, and/or insidiously wrong such that it takes a subject matter expert to see that it's incorrect. So, it's not actually an answer and is likely to substantially mislead readers. That's not a replacement.
    – Makyen Mod
    Mar 16 at 17:21
  • 14
    I'll put it more laconically. You can ask ChatGPT whatever you want. It might even work for you. But you shouldn't be posting it here and representing it as your work. Worse, you shouldn't really look to use it in your code and represent it as your work, since depending on what you're working on, you could get bit hard by licensing.
    – Makoto
    Mar 16 at 17:22
  • 16
    @Makyen I guess it would be like hiring a cab and having the cab driver confidently take you to some other destination and then drop you off. Eventually you might realise you're in the wrong part of town, or even the wrong town altogether. Mar 16 at 17:57
  • 12
    After ~30 answers that all state that AI is the future (I agree with that) and that ChatGPT is a big step forward (also agree), there is not a single post here that states a reason why ChatGPT answers should be posted on SO. Or how we deal with the fallout of people copy-pasting AI generated answers faster than they can be reviewed without any checking.
    – BDL
    Mar 16 at 18:45
  • 6
    "But if it gets really good at it, there is no possible future for SO." Stack Overflow did not ban that AI. It banned the AI that's really lousy at it, but makes it really easy for someone who has no clue what they're talking about to post a dozen plausible-sounding, but wrong, answers a day.
    – beaker
    Mar 16 at 18:45
  • 10
    There are so many misconceptions in here... And none of them are not yet discussed in the answers here.
    – Cerbrus
    Mar 16 at 19:17
  • 5
    "I saw that most people see the ban as the correct option, without having a reasonable way of distinguishing AI-generated answers from human answers <...> But I really don't think that humans will have the ability to tell one from the other." Blatantly incorrect. There's detection software out there that's plenty accurate to detect what is and isn't AI-generated.
    – Cerbrus
    Mar 16 at 19:19
  • 9
    "the fact that some users are really fed up with aggressive answers from humans in SO and would much rather prefer to interact with a AI that treated them good." Why does that interaction need to be on SO?
    – Cerbrus
    Mar 16 at 19:21
  • 14
    "remember that ChatGPT is learning and its answers will only get better and better." Blatantly incorrect. The "P" in "GPT" stands for Pre-trained. It's not learning, and it's not getting significantly smarter. Certainly not smart enough to provide answers with any measure of consistent technical accuracy.
    – Cerbrus
    Mar 16 at 19:23
  • 18
    "If we can't get OpenAI to help, integration with ChatGPT is the only possible option." Again, why does it need to be on SO? Who is going to pay for that? What benefit is there to having SO embed the mediocre output, over users just going to ChatGPT if they want to?
    – Cerbrus
    Mar 16 at 19:24
  • 12
    "I know this will be unpopular because it will make more difficult to build reputation points" That has absolutely nothing to do with this...
    – Cerbrus
    Mar 16 at 19:25
  • 7
    "If you can't beat them, join them - a popular proverb." We aren't setting out to do the same thing as a chatbot. we are not a chatbot. See also The future role of Stack Exchange vs. emerging AIs and Could ChatGPT be a viable way to answer people's questions?- both of which I have written answers to.
    – user
    Mar 16 at 22:41
  • 14
    Why are all the ChatGPT supporting answers analogies with things that have nothing to do with the subject matter?
    – David
    Mar 21 at 10:30
-34

TL;DR: assimilate, don't exterminate!

I would like to see a separate section for AI generated answers, i.e. why not just embrace it, put all questions through one/more AIs but keep them separate from human answers?

That serves two purposes:

  1. AI can distinguish so that it doesn't feed AI answers back into itself when they no-doubt use ordered site content like this to generate answers
  2. AI answers can still be viewed and voted-on, and maybe some will even make it to accepted answer
7
  • 12
    Why create more work, which will in turn create more work? A feature like what you suggest requires time to be put into implementing it, and then once the feature is made, it will require even more time by curators to confirm the content is not invalid. What is the benefit?
    – Daedalus
    Feb 17 at 3:11
  • 14
    Specifically, following up on @Daedalus's comment, what is the benefit of integrating an AI service like ChatGPT directly into SO? People who want AI-generated responses can just ask the AI. That service already exists. People who want answers written by human experts can come to SO. We already provide that service. Why mix them? Beyond that, the reason we don't have a section for AI generated answers is the same reason we don't have a section for answers written by monkeys with typewriters: those answers are terrible. They don't meet our minimum quality standards.
    – Cody Gray Mod
    Feb 17 at 6:00
  • 3
    Who's gonna pay for that? At the rate SO is getting questions, this would get very expensive very quickly.
    – Cerbrus
    Feb 17 at 13:00
  • Who said the answers needed to be curated? And who said ChatGPT is the only AI? Come on people, this technology is only going to get better. Where's your imagination?
    – racitup
    Feb 18 at 4:36
  • 13
    We don't create policies based on our imagination. We create them based on the reality that is in front of us, that we're dealing with right now. (As for who said the answers needed to be curated: that's the whole design principle/goal of this site.)
    – Cody Gray Mod
    Feb 18 at 5:45
  • Okay, give it a year or two and let's see how SO is doing with your policy...
    – racitup
    Mar 6 at 18:56
  • 3
    And why do you care about how SO will do in a year or two, with this or any other policy? just use whatever tool that works for you. that was true in the past, it's true now, and will be true in The Future (tm). if SO dies because something else replaces it... so be it. I don't get all these answers worried about "SO should adapt, or it will be replaced by something else!!!". Mar 21 at 15:30
-36

I have an idea for allowing the productive use of ChatGPT:

  1. Users who post AI-generated content must check a box indicating that. Perhaps it could go next to the "Community wiki" checkbox. A mockup screenshot
  2. Only users with at least 1,000 reputation (or some other value) are allowed to post AI answers. New users must write answers themselves.
  3. Downvotes on AI-generated answers take 10 rep, not 2.
  4. AI-generated answers with 5 or more downvotes are automatically deleted, regardless of the number of upvotes.
  5. Users who post x five-downvote AI answers in y days are not allowed to post more AI answers for z days. Not sure what the best values for x, y, and z are.
  6. Reputation lost from downvotes on AI answers is not regained on deletion of the answer, regardless of why the answer is deleted. Reputation gained from them is lost, though.
  7. Users who break Rule 1 are blocked from answering for 30 days.

Overall, this allows experienced users to post useful AI-generated answers, while preventing inexperienced users who may not know how to evaluate the output from doing so. It also makes it "expensive" in terms of reputation to post bad AI answers. Violations of rule 1 and 2 would be caught the same way violations of this policy currently are. (Really, because the option to mark an answer as AI-generated wouldn't be shown to low-rep users, any "violation" of Rule 2 would also be a violation of Rule 1, so there's only one rule people can actually violate.)

8
  • 22
    This all depends on the user being honest, which is exactly the root of this whole issue.
    – Cerbrus
    Dec 10, 2022 at 17:47
  • @Cerbrus how are AI answers currently being caught? The same methods would work for catching users who violate Rule 1.
    – Someone
    Dec 10, 2022 at 18:07
  • 3
    How does this fix any of the problems? The issue are people who would lie about step 1 and we already apply step 5 to them. What is gained by the added complexity? Dec 10, 2022 at 18:51
  • 1
    @MisterMiyagi this allows for the posting of useful AI-generated content.
    – Someone
    Dec 10, 2022 at 18:55
  • 7
    Alright, let’s play along for a bit. How would such answers look to others? Is there going to be some indication or does it just affect voting? How does this avoid the problem of seemingly good answers - as in, how realistic is it to get -5? In my experience that only happens to absolute crap. The vote based throttling seems to be too slow for the rapid posters we see, who posts dozens of answers at once. Is there some rate limit involved regardless of votes? How do we protect against people deleting their answers at -4? Dec 10, 2022 at 19:06
  • 1
    @MisterMiyagi there would be a banner at the top of the post saying it was AI-generated. The issue of seemingly good answers could be solved by deleting at five downvotes, not a score of negative five; that way people who know about the topic and see that the answer is problematic can cause it to be deleted without their votes being cancelled by upvotes from people who think it's a good answer. If someone deletes the answer at 4 downvotes, then that's a good thing; it means the problematic content was removed sooner. Reputation lost from downvotes would not be recovered when posts are deleted.
    – Someone
    Dec 10, 2022 at 19:10
  • 10
    @Someone but there really isn't any useful generated AI content, it's all well written rubbish. Dec 10, 2022 at 22:07
  • 1
    Recommended reading on this topic: ietf.org/rfc/rfc3514.txt Dec 16, 2022 at 1:36
-36

Banning all ChatGPT answers is a good temporary move, as it opens a time window where it could be discussed thoroughly, but it, IMO, shouldn't be permanent, as it could also help if used according to its capabilities and by acknowledging its limitations.

Consider the following situation:

Someone sees a question they know the answer to.

They don't want to worry about the structure of the answer, so they use ChatGPT with a prompt that directs it towards the correct answer of the problem (for example, if the question is "How do I remove and get the last element of an array?", a possible prompt might be "Using the array.pop method, write a Stack Overflow answer to the question ..."

They then check and verify the answer to see if ChatGPT has done any mistakes, and either direct ChatGPT to correct the answer, or correct it themselves.

They then post the answer.

Is this helpful to SO? I would assume it is as the user who asked their question gets an answer that works, and the answerer spends less time formulating and explaining the answer and more time worrying about the correctness of the answer. As ChatGPT is a language model, here it would have been used correctly according to its capabilities (language and not programming - the programming knowledge comes from the answerer).

Should this be banned permanently? Permanently banning all ChatGPT answers means this should be banned as well, even though it actually is helpful to the Q&A format we have going on here.

Bad ChatGPT answers are just bad answers, and I don't think we should have another rule specifically for ChatGPT. Spamming good-looking but bad answers with AI tools and abandoning them to "see the numbers go up" should be the behavior that is banned.

11
  • 9
    The problem is that users have been proven to be untrustworthy, and have been going for quantity over quality when using the AI to generate answers. Besides, answers that are simple enough for the AI to consistently answer correctly, generally have excellent due targets on SO... TL;DR: There's nothing of value to SO, generated from CGPT.
    – Cerbrus
    Dec 13, 2022 at 16:44
  • 3
    The use-case described by this answer would be a great way to use ChatGPT. Unfortunately, it's not how it's currently used. On the long run, we have to find a way how responsible use of AI can be allowed while still having methods to prevent flooding the site with garbage.
    – BDL
    Dec 13, 2022 at 17:41
  • 1
    I had a similar thought: What if you have the skills and willingness to validate the AI-generated answer, even if you didn't come up with it? Such good-faith use of a chat AI by an actual expert would be completely indistinguishable from a human answer, and such use would be unenforceable. We're being asked here to self-enforce this ban on ourselves. If you don't want to self-enforce this ban on yourself, others may never notice, but you're crossing a different line here by publicly advocating that others not self-enforce themselves.
    – durette
    Dec 13, 2022 at 18:36
  • 10
    There really isn't much value in carving out an exception for cases where the tool is used in a way such that it won't be recognizable as chatgpt and is actually producing valuable content, people using it have already proven that they're overwhelmingly unwilling to take those steps. People who are takin those steps aren't spamming the site with nonsense posts, but they're still potentially posting plagiarized content.
    – Kevin B
    Dec 13, 2022 at 18:45
  • 1
    "Is this helpful to SO?" To answer the question in the answer: Yes it is helpful. Caveat: only a minority will actually do it like this. But otherwhise it's a valid idea.
    – Trilarion
    Dec 13, 2022 at 19:48
  • 2
    The main problem though, with this suggestion of augmented answering, is that you never know when the chat bot will randomly introduce fake or false information to support its possibly correct larger point. These nuanced false points might seem harmless if for example we are talking about resolving the argument of who won the last World Cup. The chat bot would perhaps make a comment like "Argentina won the 2022 World Cup. It was their third time matching up against France in the finals, and only the second time Messi won". That last part, unless you are truly informed, may seem real.
    – Travis J
    Dec 24, 2022 at 7:28
  • 4
    In code, it can be dangerous to use those extra parts, especially since nuance is often very important (for example a wrong conversion to let's say, oh I don't know, feet to meters #mars). Perhaps with things of a more lenient nature, these white lies don't particularly harm anything, but when people's savings, kids, or lives are at stake, we can't just lightly gloss over such glaring inaccuracy.
    – Travis J
    Dec 24, 2022 at 7:28
  • @TravisJ of course I am not suggesting people to blindly copy and paste those answers directly from ChatGPT, rather here, the tool would be used just as a glorified sentence generator, and the output it generates should then be left to the human operator to check. Dec 27, 2022 at 22:25
  • 4
    @kahveciderin - Unfortunately a very large sample size of the minority have proven they cannot be trusted to use ChartGPT to help them submit an answer. They have proven they do not have the knowledge to confirm the answer content is (correct, accurate, ect.). That sample size has been extrapolated across multiple communities. Dec 28, 2022 at 12:31
  • Main Problem with this Scenario I think, ... is that 'ChatGPT' doesn't check for DUPLICATES, oops...!
    – chivracq
    Jan 15 at 3:17
  • I am against permanent ban, but not for the reasons given here. Everyone should be allowed to learn from their mistakes. It is as simple as that. Banning for a long time - sure. It needs to hurt to sink in. Permanent banning is needlessly extreme.
    – Gimby
    Mar 3 at 14:20
-36

Prediction: SO will eventually incorporate AI trained on correct answers to clear questions.

If it doesn't, the AI of Bing, Google or Baidu will make SO obsolete due to it being too slow to get answers to questions.

Of course, what is 'correct' in the training data needs to be truly correct, and not just what the OP decided among the early answers, as later answers may be more comprehensive or nuanced.

Raw SO answers would be poor training for AI because most answers are just not good enough or misleading, let alone the masses of unhelpful comments. The problem with AI is that it needs an eclectic and massive tranche of correct and accurate data for its training so that it is not going to be polluted too much by subsequent real-life data. SO is in a very good position to have its own AI, as it would not be sweeping up irrelevant or incorrect data that search engine AI would be incorporating now.

The role of moderators would shift to being more about checking the training data and AI answers than policing question quality, as language AI would easily pick up ambiguous phrasing and respond with suggestions, correct them, or flag for human attention.

Also, the increase in SO moderators aggressively closing newbie questions or commenters making snarky comments too often may just force more to go to AI answers. Popularity is relative to the help offered. Not providing help but insults instead will hurt SO when it is up against AI that is far more forgiving regarding the 'quality' of questions. See The Tyrannical Mods of Stack Overflow for some examples, including some where even popularity didn't save them.

I know tech experts and mods on SO don't want to be customer service reps, but that really needs to be part of the role, otherwise the more forgiving AI will eat all SO's meals before it get to see them.

We have so many free programming tools these days with documentation that is often unhelpful. Not everyone is building Apollo guidance systems, so do not need to know all the ins and outs of their language, but just want to get the task at hand done. Stifling a poor question instead of helping them work through their issue is going to kill SO for them. But that is a labour-intensive process, which is where AI could significantly alleviate the load by interactively converging on a result, or at least clarify the question.

SO has the opportunity to step up and meet the challenge of AI, or close off, and become irrelevant. The window of opportunity is rapidly closing!

Response to comments

  1. Aggressively enforcing rules without taking the time to educate new users is what alienates new users.

  2. Competent AI is NOT way in the future. ChatGPT was only released less than a year ago, and few were expecting it to do what it does, so I would expect capacity and competence to increase rapidly now that such AI is being put front and centre in search engines.

  3. 'Why would we even have persistent AI answers at a future point when they can just be generated on demand as good as expert answers?' - that is exactly the existential question for SO posed by AI. The competency of AI can increase very rapidly with the right training, so unless SO adapts to the disruption posed by AI, it will be irrelevant, just because the current model is so heavily reliant upon competent human attention and a huge portfolio of loosely-structured Q&A decided by a popularity contest among those whose own competency is decided by a popularity contest. Populist 'truth' does not make for good AI processing!

  4. Yes, some user and mods may not want to have to be like customer service reps, and so might be happy to have ChatGPT et al be the first point of call, but the big question is whether there will be enough users after that for SO to be viable enough to satisfy what Prosus expected of it when they bought it for $1.8b in 2021. Too few users and they will just have to write it off like many formerly popular sites before it.

When AI is given facts to prioritise upon (like BNF statements for programming languages) and training effort is made to avoid irrelevancies, the current SO paradigm will seem slow and archaic by comparison.

I don't say AI is going to be a universal panacea, but in the realm of sifting knowledge given an intended goal, its speed will trounce everything else that purports to be authoritative but is still essentially based upon hearsay, SO included.

SO will always be behind the 8-ball in getting its data up to standard given the amount of ad-hoc queries and its maxim that popularity is a reliable indicator of truth and accuracy. There is not enough unbiased humans to do that job.

28
  • 11
    Newbie questions should not be closed here. No question is too "simple" for SO. If you see questions closed because they're "newbie", please raise a flag or otherwise let us know, because this is unacceptable. There are, however, plenty of questions that are unsuitable for this site, for example because they are off-topic, are unclear, are too broad or too opinion-based to fit into our Q&A format, etc. Also, the rate of questions getting closed hasn't increased. We've always aggressively closed off-topic questions.
    – Cody Gray Mod
    Feb 17 at 5:56
  • 5
    As far as the snarky comments, I'm of two minds about that. In one sense, the borderline snarky comments are part of the "human" factor and part of what makes it fun and interesting to participate here. AI isn't going to be able to do that, but I'm not sure that's a feature so much as a limitation. As a nod to this, ChatGPT even has a feature where it can try to emulate different writing styles if you ask it to do so (e.g., "write a snarky reply"). For snarky comments that are over the line or otherwise unappreciated, they should be flagged for removal. Mods remove lots of these every day.
    – Cody Gray Mod
    Feb 17 at 5:58
  • 18
    What does this prediction of some far away future have to do with the problems and ban we have right now? Why would people on a volunteer platform even be interested in "checking the training data and AI answers"? Why would we even have persistent AI answers at a future point when they can just be generated on demand as good as expert answers? Feb 17 at 7:39
  • 9
    Language models don't understand data. They don't comprehend factual correctness. That level of comprehension is just not something current technology can even come close to. I don't see that happening any time soon.
    – Cerbrus
    Feb 17 at 12:59
  • 13
    "Competent AI is NOT way in the future. ChatGPT was only released less than a year ago." yet is not competent.
    – VLAZ
    Feb 18 at 8:25
  • 6
    People made the same prediction about en.wikipedia.org/wiki/ELIZA. When it comes to understanding the underlying concepts as opposed to the language used to communicate it, chatGPT has not fundamentally advanced beyond eliza. Feb 18 at 21:01
  • 4
    If only people would flag these inappropriate comments instead of making YouTube videos about them...
    – Cody Gray Mod
    Feb 19 at 7:18
  • 4
    @Patanjali That basically means that ELIZA wasn’t trained automatically but "by hand". The basic idea of choosing the most likely response based on weighting the input words/grammar/patterns is surprisingly similar - the key part is that neither system actually understands its input and output based on the same semantics as a human expert or technical program like a compiler would. They’re just statistical models; ChatGPT is a more sophisticated model, but it still has very similar flaws. Feb 19 at 7:18
  • 6
    @Patanjali the problem with SO is that people come here and expect hand holding and tutoring, which is not what SO is about. Users don't have to come here for training. They can visit ChatGPT, if they wish, or many other places.
    – VLAZ
    Feb 19 at 7:39
  • 11
    I've watched the video before, yes. No, a more forgiving question policy would not solve the comment problem. If anything, it would lead to more snarky/rude comments because people have no other means of dispensing with off-topic, unclear, or otherwise inappropriate questions. This site has standards, and those standards are key to our having quality content. Like Wikipedia, you can't create an article about just anything, and it can't contain whatever you want to write in it. However, SO's quality standards are also enforced by humans, who sometimes get it wrong and/or are sometimes rude.
    – Cody Gray Mod
    Feb 19 at 8:05
  • 3
    This is where flags come in, so that moderators can deal with the inappropriate enforcement of quality standards (i.e., incorrect closures), and/or users who leave rude, inappropriate comments. If people would flag these, then we could deal with them. However, if they aren't flagged, then we can't do anything about them, and the problem persists. Users are going to be wrong and do wrong things regardless of what the policy says. Correcting that requires other users to flag it. Relaxing the site's quality standards would only turn this into a complete cesspool, rather than an imperfect site.
    – Cody Gray Mod
    Feb 19 at 8:06
  • 11
    OK, so SO becomes a resource that people only consult when the AI gorilla can't solve it for them? Sounds good to me.
    – Cody Gray Mod
    Feb 19 at 8:30
  • 8
    @Patanjali I don't think you really get it. SO isn't and doesn't need to be a tutoring service. If it takes ChatGPT to finally stop the tutoring questions coming in then that's a win for SO. Not a loss.
    – VLAZ
    Feb 19 at 8:41
  • 6
    @Patanjali you aren't hired as a consultant for how to make SO profitable. You have no insight into what the company does internally, nor what it plans to do. Your suggestions are thus 1. unneeded 2. not even relevant to the company. So, feel free to stop your financial strategy plans. Or just directly apply to SE Inc. for a position which would allow you to enact them.
    – VLAZ
    Feb 19 at 8:59
  • 9
    @Patanjali Because there isn’t such a magical future AI yet, and as far as I can tell not in the next decade. Calling for a change that is both unnecessary and premature is subverting I disagree with strongly. That we shouldn’t care about making changes doesn’t mean that we shouldn’t care about not making changes. Feb 19 at 9:20
-37

It might make sense to consider integrating ChatGPT into the site engine itself.

In this way, an answer received from it should be displayed in the least annoying way with a note that this answer is not related to SF, was received programmatically and is most likely wrong, and the possibility of drowning it in minuses.

This will reduce the motivation to post similar answers, as well as create data for training neural networks (including in the minds of site users) that recognize the generated answers.

5
  • 11
    This has already been suggested in this post, multiple times. Dec 5, 2022 at 23:58
  • Hm. In comments? Understood, please understand and forgive. :( I haven't seen similar answers. Dec 6, 2022 at 0:01
  • 2
    What is "SF"? It would normally be Server Fault here. Do you mean Stack Overflow? Dec 6, 2022 at 0:14
  • 9
    You don't exactly have to look hard for it. They're all on this page. The answers are: one, two, three, Dec 6, 2022 at 10:57
  • 1
    @ZoestandswithUkraine: There's one difference this answer has from the others that suggest basically the same thing: this one would frame the AI answer as being likely wrong, and warning people not to post it or an answer like it. The others all suggested that the AI answer might actually help people solve their problems, and would get the same "benefit" without people having to post such answers. That said, I don't think this is actually a good idea. but it is different; I nearly upvoted. (The dollars and electricity to AI-generate answers from the ask interface make this a non-starter). Dec 13, 2022 at 4:29
-40

It seems like a slippery slope here. Am I banned from using ChatGPT for doing my own research? Certainly not. So if I gain education by way of ChatGPT, am I then banned from conveying that knowledge by way of answering a SO question? I wouldn't think so, as how I came to know something should be irrelevant.

So then, I suppose the question is "If I use ChatGPT to research a topic solely so that I can answer a question on SO, is that wrong?" I can't think why it would be, so long as I'm properly curating the answer from my own knowledge.

And if that's ok, then the question becomes "How much does my answer have to differ from the ChatGPT answer that I used to inform myself so that I could answer the SO question?"

8
  • 12
    "…if I gain education by way of chatGPT…" Ah, I see this is a pure hypothetical, so we don't have to worry about the answer to it, because that's not going to happen.
    – Cody Gray Mod
    Feb 15 at 6:51
  • 13
    "so long as I'm properly curating the answer from my own knowledge" The fact that most people weren't doing this is why we're in this mess in the first place.
    – E net4
    Feb 15 at 9:28
  • 2
    How you learn isn't going to turn into you write answers in a form that will 99% of the time be wrong but very well written.
    – Kevin B
    Feb 15 at 15:28
  • 1
    Wow, 14 dislikes. Guess I hit a nerve. Funny how a dissenting opinion amidst a sea of minds that are already made up leads to downvotes with no real mention of why. I'd love to know where the flaw in my logic lies. I've already proven to myself that chatGPT makes a great research assistant. So this response came from an informed position. I could prove that statement if held to the fire. But there's no interest here in evolving opinions here it seems.
    – CryptoFool
    Feb 15 at 16:32
  • 14
    Yea, i mean, funny how people express agreement on meta with votes, and how unpopular opinions meet a lot of disagreement. Almost as if the system is working exactly as designed. Funny! ¯\_(ツ)_/¯
    – Kevin B
    Feb 15 at 16:52
  • 15
    So where exactly is the slope and why is it slippery? The announcement makes it pretty clear where the line is drawn and why it is exactly where it is. If you manually write about your own, verified knowledge then no one cares where that comes from. Feb 15 at 17:27
  • @MisterMiyagi - If the metric is "copying/pasting from the chatGPT site", then you're right, it's not a slippery slope. And by reading the notice of the ban, that's the way I read it. The slippery slope would be if one wanted to take it any further than that...to say that one can't "paraphrase chatGPT output".
    – CryptoFool
    Feb 15 at 23:34
  • 5
    The problem is with people blindly copy-pasting content in bulk, without validating the contents... If a user were to take the effort to paraphrase the content (manually, not with some kind of AI), I'd presume they'd at least check if it's correct.
    – Cerbrus
    Feb 17 at 13:02
-42

I just wanted to suggest that any AI-generated answers not be deleted altogether but just moved to a separate, still-accessible page of "Suspected AI-generated content" for consultation. This way, in case they DO contain something useful, which is going to happen more and more as it quickly improves, the poster is still helped and no time is wasted. After all, that is the main and, arguably, sole purpose of the website as it is the service it provides and that which generates visits and, therefore, revenue. More importantly there is absolute NO HARM WHATSOEVER in doing this, nor does it cost any resources.

A just compensation mechanism could then be thought of for the people who actually took the time from their busy schedules to give meaningful answers based on their hard-earned knowledge. Maybe any rewards could be split among all genuine contributors or anyone who contributed gets an increased reward for their next accepted answers. I'm sure you guys could think of something fair and fitting that could not be abused.

The inescapable reality is that there will be a drastically decreasing need for human help in the coming years not just here but everywhere. This is unavoidable and should NOT be seen as something bad at all. It's just technology replacing manual labor again, this time at a much grander scale. This is a great thing, an incredible thing, and needs to be accepted and used for good, certainly not resisted in any way.

I must say that I would find it very funny if Stack Overflow, one of the world's greatest bastions of programming, machine learning and all things computer-related, were to treat arguably the greatest fruit of its subject matter with contempt and prejudice instead of teaching the world by example as how to incorporate AI into existing solutions without there being any adverse effects for the humans that already participate in them.

13
  • 4
    The social impact of ML will be massive, and perhaps the subject matter at hand is a taste of what it is to come. But people should be allowed to be cynical about these developments without their views being labelled as "contempt and prejudice". This isn't an honest way to argue.
    – halfer
    Dec 27, 2022 at 21:12
  • They are so long as there is no direct impact on the quality of the service being offered, as far as I am concerned. For instance, I would like the right to go through AI-generated content once I've tried everything else, particularly those generated by others since I am not an expert and my prompts are of lower quality than those generated by someone who is. I don't care who typed the answer or whose knowledge it is, so long as it helps me I'm happy. If it's faster and easier for whoever did it, that's all the better. Dec 27, 2022 at 21:16
  • 7
    @PythonMillionaire: If you want answers from an AI, then why not copy-paste your question to the AI site? If the answer helped you, you are highly encouraged to self-answer your problem, as long as you reformulate the answer of the AI. As of now, most answers that get deleted are generated by exactly that way, copying the question and posting it to the AI without any editing. Then they copy back the answer, also without checking for correctness or changing anything.
    – BDL
    Dec 27, 2022 at 21:35
  • 1
    I certainly understand and agree that mere copy pasting in and of itself is pretty problematic. However, my reasons are that my prompts aren't as good and I can't fact check. AI plus intermediate-level human knowledge is far, far more useful and accurate than just AI plus beginner-level knowledge. I don't know what information is relevant and worth sharing with the AI and a lot of other things. I agree that mere copy pasting has an incomparably higher chance of producing garbage answers but humans can also do that and we can at least expect AIs to constantly improve, not so humans Dec 27, 2022 at 21:47
  • 3
    @PythonMillionaire Users aren't fact checking. They aren't applying intermediate level knowledge. You can't do that for an answer in under 5 minutes. If you do do those things, then this policy probably won't trigger because no one will notice. Its about having a response for the bad actors. Until we have a manageable process for preventing those who exploit the system, then the policies have to be broad to minimize volunteer moderator effort. Dec 28, 2022 at 2:32
  • 1
    I perfectly understand and agree! Just put them ALL in the page I mentioned. No need to delete Dec 28, 2022 at 2:40
  • 9
    “More importantly there is absolute NO HARM WHATSOEVER in doing this, nor does it cost any resources.” - I strongly disagree that there is no harm caused by users using any tool similar to ChartGPT to post answers, low quality trash generated by ChartGPT, takes very limited valuable review time away from reviewing actual potential high quality answers. Horrible low quality content generated by ChartGPT or any similar tool has absolutely no place on any Stack Exchange. Dec 28, 2022 at 12:16
  • 1
    Hi Security Hound. That's not what I said, though. I said there is no harm in moving such replies to a separate thread where they are appropriately labeled as AI content as opposed to deleting it all outright Dec 28, 2022 at 13:38
  • 6
    We don't get tools like that. In fact, that would take some serious development work to make happen. Deletion is still the best tool for the job right now. Remember, deletion is not necessarily permanent
    – Machavity Mod
    Dec 28, 2022 at 13:45
  • That's too bad. Hopefully deletion won't be permanent as we really, really cannot escape the fact that everything will drastically change very soon and we need to incorporate these wonderful developments into our society as a whole Dec 28, 2022 at 15:19
  • 5
    “I said there is no harm in moving such replies to a separate thread where they are appropriately labeled as AI content as opposed to deleting it all outright” - Except Se developer resources which are limited. I would rather see improved review tools than a feature to deal with the absolute trash content generated by ChartGPT. I detected a user who submitted 12 answers and every single one, was incomplete or incorrect. A 100% track record proves that ChartGPT generated content is 100% unreliable. ChartGPT isn’t an AI. Until such time it’s 110% inaccurate CGPT content should remain banned Dec 29, 2022 at 21:31
  • 2
    SO's temporary ban & the explanation make sense. This answer raises some good points. I understand downvotes indicate disagreement, but giving reasons wld be more helpful. AI is here to stay. Skilled programmers & learners, of all people, know better than to dismiss it. Many colleges ban the use of Wikipedia (incl correctly cited) in student essays on grounds often similar to people's objection to content fr ChatGPT. Like Wikipedia, ChatGPT can be a great learning tool, a good starting point. Who else can figure out how to harness ChatGPT's power for teaching & learning than ppl here on SO?
    – YCode
    Dec 30, 2022 at 20:57
  • 1
    wld = would. fr = from. ppl = people. Dec 31, 2022 at 11:57
-54

If this issue gets too far out of hand, one possible way to mitigate this might be to integrate ChatGPT and show the user a possible answer before the question is even posted. Something like this:

Screenshot of user asking a question about removing an element from a vector in Rust, and the correct answer given through ChatGPT

That would beat the answerers hungry for quick-and-easy rep at their own game. If you can't beat them, join them.

52
  • 8
    Aside from that, it doesn't make the thing with wrong/incomplete/potentially dangerous (?) answers better.
    – dan1st
    Dec 5, 2022 at 7:30
  • 68
    The problem with this is that it provides answers which look correct but aren't necessarily actually correct.
    – forest
    Dec 5, 2022 at 7:34
  • 5
    Does this answer your question? en.wikipedia.org/wiki/Luddite Dec 5, 2022 at 7:51
  • 66
    It's all fun and games until system("sudo rm -rf /") appears and the user blindly runs it Dec 5, 2022 at 7:52
  • 22
    If someone wants this, they should probably set up their own clone of SO and run it there. It's not like we don't have enough human generated crap to drown in already. | "If you can't beat them, join them." -- If we can't beat this, I'll go spend my time on something else.
    – Dan Mašek
    Dec 5, 2022 at 8:07
  • 5
    @bad_coder that's what this post is saying, and it has been heavily downvoted and explained why that's a bad idea. were you going for a joke on your name "bad_coder"? Dec 5, 2022 at 8:44
  • 8
    Even better, for the FGITW: "Is this your answer?"
    – bad_coder
    Dec 5, 2022 at 9:07
  • 11
    I'd rather see that money go to plagiarism checking rather than creating a questionable answer that may or may not be correct (but often isn't) Dec 5, 2022 at 10:20
  • 10
    @Spidy did you miss the part where AI writes crap answers?
    – Cerbrus
    Dec 5, 2022 at 11:48
  • 15
    In addition to the possibility of these answers being wrong, I assume the question author would, in many cases, be the least equipped to understand whether the auto-generated answer is correct – and whether it's recommending something dangerous/destructive (see Zoe's example). Subject matter experts who are posting answers and reviewing them might be able to judge the quality of the answer, but the question author usually doesn't know what the right answer is – that's usually why they're asking it in the first place.
    – V2Blast StaffMod
    Dec 5, 2022 at 15:54
  • 8
    No, @mattmc. This just add the crappy generated answers in a even more official format, without any human oversight. Also, who's gonna pay for that?
    – Cerbrus
    Dec 5, 2022 at 23:17
  • 5
    @Sgdva I have no idea what you're trying to say. We don't need to wait for a future AI, we already have tons of plagiarism detection software, and it exists as numerous SaaS systems. SO hasn't integrated it because that's how they roll. If you're proposing implementing CGPT, my resistance to that remains that, unlike plagiarism checking, a CGPT can go unfiltered to a user who assumes the answer is good without anything dangerous in it, while a plagiarism detection bot, realistically, is half automatic and half manual review, and neither of those risks giving users dangerous code Dec 6, 2022 at 15:50
  • 8
    Already now, people believe anything GPT says is gold, true, and 100% correct. As has been demonstrated countless times, that's wrong. A direct CGPT integration offering answers increases the risk it's perceived as accurate, and yes, at potential risk to unsuspecting users. Additionally, since CGPT exists, why would you need an SO integration? If it's so good, just go directly to CGPT and ask it rather than taking the roundabout way of going via Stack Overflow, when SO doesn't own nor develop OpenAI nor CGPT. Dec 6, 2022 at 15:52
  • 3
    @Ajedi32 I have read it, and that doesn't change my response. It is overwhelmingly incorrect in questions that can't be answered by google's direct answer feature. This AI isn't for providing correct answers. that's literally not it's purpose.
    – Kevin B
    Dec 7, 2022 at 17:07
  • 13
    Guys, as much as you disagree with this answer, that's not a reason to delete-vote it! Don't abuse your privileges like that.
    – Cerbrus
    Dec 8, 2022 at 8:42
-67

I get the point, but if you'll allow a lurker's five cents: I believe that ChatGPT has more to contribute to the platform than to hinder it. How about implementing the bot natively on the platform? Let it answer the questions and, if you want, put an alert saying "this is an automatic response and may contain errors". ChatGPT is helping me a lot, it's fast and practical. It may (yet) not be the right one, but it's enough to help get to the answer.

13
  • 12
    We've been over this twice already. Dec 5, 2022 at 12:03
  • 34
    You clearly didn't read the other answers here: ChatGPT writes bad answers, contradicts itself in the answers, and is extremely costly to implement on a scale SE would require.
    – Cerbrus
    Dec 5, 2022 at 12:03
  • 1
    “How about implementing the bot natively on the platform?” - No; These CGPT answers are absolutely horrible and useless. Dec 5, 2022 at 12:11
  • 3
    Aggressive responses are one of the things that discourage people from posting here, another advantage of ChatGPT. By the way, have you tried to tell him that the answer is wrong or bad? He usually fixes it. I won't insist, it's just my opinion. :) Dec 5, 2022 at 12:11
  • 27
    [1/2] The problem isn't ChatGPT itself. Feel absolutely free to use it to solve your own problems. You may even use it during your research for writing an answer here. The real problem are users who copy-paste ChatGPT answer to SO without even checking if they are correct at a high rate. We had a user yesterday who posted 20 answers in a little bit over an hour, were at least a third of the answers didn't even match the programming language of the question or were outright wrong.
    – BDL
    Dec 5, 2022 at 12:14
  • 7
    And who is going to tell the bot the answer is bad, if it's just automatically showing the author of the question (that doesn't know the answer) whatever it generated? Who's to stop the bot from giving a incorrect, or even dangerous answer? And who on earth is gonna pay for the bot?
    – Cerbrus
    Dec 5, 2022 at 12:15
  • 11
    [2/2] Unless you find a way that the person who copies the answer to SO makes sure that it's a good answer and tells the bot when he is wrong, this isn't going to scale. You can't rely on volunteers here to vote on these answers to get the signal. That's not going to scale on the size of SO.
    – BDL
    Dec 5, 2022 at 12:18
  • 3
    “ChatGPT is helping me a lot, it's fast and practical.” - But the user’s based on output generated by CGPT are absolute trash answers. Low quality answers generated by CGPT are beyond unhelpful. Feel free to use it, just Don’t Post its Output, and experienced users in the community can tell when an answer is based on useless CGPT output. Dec 5, 2022 at 19:56
  • 1
    If ChatGPT helps you, there's nothing preventing you from using it. But in doing so, you're fully aware that the answers you're getting are coming from ChatGPT, and you probably know enough to at least take them with a grain of salt. The issue for SO is people expecting relatively high quality, moderated SO answers could be getting low-quality ChatGPT answers, usually without knowing it, and that's not a benefit to anybody.
    – Caleb
    Dec 7, 2022 at 4:23
  • It's funny, but I think this is involuntarily giving the right answer. Sure, it's pricey and won't be added, but chatGPT answer as first answer would not be the answer, but a reference for any other answer to say: hey, this is an AI answer, if you're answering with this, you're an AI, and your answer will go straight to moderation (or, deleted) - (WOW, I've used the word "answer" more than anyone else here :D)
    – nnsense
    Mar 9 at 21:14
  • @nnsense "a reference for any other answer to say: hey, this is an AI answer, if you're answering with this, you're an AI" why do you think there is the AI answer here? ChatGPT can generate different answers based on how you've asked and/or based on your existing chat history in the session. Each can claim either A or B if there are two options available. It's not like any and all ChatGPT answers always choose A, for example. Yet again, the only thing ChatGPT does is generate plausible text. It doesn't take decisions on questions.
    – VLAZ
    Mar 10 at 2:04
  • The topic here is: chatGPT answers are banned. Fine, so you need something to understand that an answer is indeed a taken from chatGPT, the only way I see is to have a reference, to compare. ChatGPT answers are more or less similar when the question is exactly the same, if someone used it just as reference to answer it's fine, the point is to avoid those which are copy/pasting from it. Can't think of any other way, filtering out chatGPT answers can't be just "guessed", that would create a lot of false positive.
    – nnsense
    Mar 10 at 19:13
  • The methods in use to detect chatgpt answers isn't producing a lot of false poisitves.
    – Kevin B
    Mar 10 at 19:16
1
2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .