120

Today, I stumbled upon this answer. In that answer, there is an explanation with the code:

This code was written by Github Copilot with minimal supervision

and nothing else. I see some potential problems with this:

  • user submitting such an answer might not be able to reason about the answer at all
  • there is high rate of incorrectness with Copilot generated code (source1, source2)
  • does the user even have the rights to submit such code into Stack Overflow

What do you think, is this ok? If not, what should be done about them?

25
  • 15
    Relevant consideration: should one answer questions that do not meet our minimum standards?
    – Cody Gray Mod
    Commented Oct 30, 2021 at 8:12
  • 37
    I see users often posting only code code which doesn't quite answer the question. They seem to be unable to reason about it, either. Some times it's just copy-pasted, some times it's (presumably) an original work but not even related to the problem statement. I'm not sure if you consider these better or worse to automatically generated code which also happens to be wrong but I'd rank them the same, personally. Seems both are simply not useful.
    – VLAZ
    Commented Oct 30, 2021 at 8:15
  • 2
    @VLAZ they might or might not be useful, and certainly not very good answers. However, in this case we know the submitter does not actually create the content him/herself, so I think this case is worse by definition.
    – eis
    Commented Oct 30, 2021 at 8:17
  • 9
    I don't think there is a requirement for the contributor to submit all original work. We are happy to accept an answer that points to a well-known solution that the posted didn't create themselves. We do require proper attribution for that, of course, but the point is that a post that contains non-original work is not, by itself, a problem.
    – VLAZ
    Commented Oct 30, 2021 at 8:19
  • 1
    Yes, I agree. However in this case I think there are additional, related problems. This is non-original work where the source is also not verifiable, since copilot does not really reveal its logic why it came about this specific answer, and even with copilot attribution we can't know how that answer was reached.
    – eis
    Commented Oct 30, 2021 at 8:24
  • 11
    Disclaimer on all other answers: The code was written by me being 100% sane and not intoxicated by drugs, alcohol or other substances. ...
    – rene
    Commented Oct 30, 2021 at 8:59
  • 3
    @rene The comment was written by me being 100% high on technology Commented Oct 30, 2021 at 11:14
  • 7
    It would be fun if StackOverflow would be part of CoPilot's training set, in which case answers generated through CoPilot would be some sort of reinforced learning ... Commented Oct 30, 2021 at 11:16
  • 4
    I think if co-pilot can answer a question it must be generic enough to be already asked in so. Commented Oct 30, 2021 at 11:21
  • 28
    "This is non-original work where the source is also not verifiable, since copilot does not really reveal its logic why it came about this specific answer" I still don't really see the difference between that and somebody posting "try this <code dump>". We still don't know how the author came to the decision for the code. And I have seen some baffling examples. I accept the attribution dispute being interesting poiint but I'm not sure I buy the concern about posting the code other than that. Either the poster will explain the code or not. Doesn't matter who authored it.
    – VLAZ
    Commented Oct 30, 2021 at 11:22
  • 5
    'It would be fun if StackOverflow would be part of CoPilot's training set'...erk! Kernel panic #FEEDFOOD... Commented Oct 30, 2021 at 12:21
  • 2
    @MartinJames - What makes you think it isn't?
    – PM 77-1
    Commented Oct 30, 2021 at 19:31
  • 3
    When I read the answer this post is based upon I also had an thought of writing a meta question of my own. The title would've been "Is Copilot a future of SO?". I envisioned Copilot (or similar AI) auto-answering questions. When I started answering questions on this side I quite often would find an approach on-line (possibly on SO itself) and then adopt it to OP's needs. Not much different.
    – PM 77-1
    Commented Oct 30, 2021 at 19:37
  • 2
    @Maaz that's what they claim, yes. However with cases like this it is doubtful if they can make that claim. This has now a separate question on law.stackexchange, and fossa agrees that it's not easy to reason about.
    – eis
    Commented Nov 5, 2021 at 10:11
  • 4
    Related announcement by the mods: Temporary policy: ChatGPT is banned (as of December 5, 2022).
    – V2Blast Staff
    Commented Dec 5, 2022 at 15:22

6 Answers 6

11

In light of the GPT ban, I asked about this today here on Meta, but the question was closed as a duplicate of Are all generative AIs banned?.

Since Copilot is, as I understand it from the OpenAI Codex announcement a "descendant of GPT-3", content and output from it seem to be directly banned under the current policy.

Regardless, since my "Does Github Copilot generated code fall under the GPT ban?" question was closed as a duplicate of Are all generative AIs banned?, it appears that the answer is what @RyanM posted there:

... it violates our rules on pasting AI-generated content without validating it. We can't verify exactly which AI system generated a piece of content, but we can often tell when someone is using one.

...

This exemplifies exactly why we don't allow this sort of content: it's easy to let errors slip past, and the work required to catch them is time-consuming.

0
48

Yes, posting an answer generated by an AI is acceptable, so long as the source is attributed. It's still a code-only answer (i.e. what the code does, or how it solves the problem is unexplained), which is not great, but the fact that it was generated by an AI doesn't really matter.

To address the concerns directly

user submitting such an answer might not be able to reason about the answer at all

Sure, but that's irrelevant. Whether or not the user understands the answer they posted is not the concern of the site.

there is high rate of incorrectness with Copilot generated code (source1, source2)

Again, not a problem. The quality of the answer should be judged on its own. It's similar to how an answer is judged in isolation, without considering whether the user has a history of posting incorrect answers.

does the user even have the rights to submit such code into Stack Overflow

I think so. At least it's fine so long as the source is attributed, and there's no license on code generated by GitHub Copilot that prevents it being used on Stack Overflow.

22
  • 17
    The last point sounds sketchy to me: As long as there is no license allowing redistribution (and perhaps derivative works) of the generated code, it seems like a copyright issue. I tried to find out whether github copilot offers any license to the generated code but 5mins of googling yielded no good results. Especially there is also the issue that posting it on SO is relicensing
    – lucidbrot
    Commented Oct 30, 2021 at 18:26
  • 12
    @lucidbrot: GitHub asserts that Copilot's use of source code doesn't infringe copyright, so I'd think that by that logic one can use the outputs without restriction too. (although this depends on GH's controversial assertion about copyright on training data being correct)
    – loops
    Commented Oct 30, 2021 at 18:36
  • 13
    Also, OpenAI (which actually owns the model) says "OpenAI will not assert copyright over Content generated by the API for you or your end users"
    – loops
    Commented Oct 30, 2021 at 18:37
  • 10
    @Smitop I beg to differ Commented Oct 30, 2021 at 18:37
  • 4
    @lucidbrot I'm afraid I don't know what the policy is for using that code. It shouldn't matter from SO's end anyway, since 3rd party policies are not enforced here.
    – cigien
    Commented Oct 30, 2021 at 18:46
  • 24
    How do you even attribute the source when the source is unknown? Do you just attribute the AI? Isn't that like attributing the plagiarist?
    – Cody Gray Mod
    Commented Oct 31, 2021 at 14:36
  • 1
    @CodyGray I don't follow. If the code was generated by some AI tool, then that's the source that needs to be attributed, right? What does it matter where the tool got the code from? That's the tool's problem I would think. For example, if I were to use some code that's posted on SO, I just need to attribute SO. Am I expected to track down where the user who posted the code got it from?
    – cigien
    Commented Oct 31, 2021 at 14:52
  • 12
    I can't edit... but to quote the GitHub Copilot website: "Who owns the code GitHub Copilot helps me write? ... The suggestions GitHub Copilot generates, and the code you write with its help, belong to you, and you are responsible for it**". IANAL, but my interpretation of that is that you own the code it writes
    – cocomac
    Commented Oct 31, 2021 at 19:31
  • 1
    @hakre, even if the ML model was trained on copyrighted data that doesn't automatically mean that its output is also copyrighted, same as when a human artist looks at another's painting and draws an original work inspired by it. There are situations when Copilot recites its training data, but they are extremely rare in practice.
    – Aivean
    Commented Oct 31, 2021 at 20:27
  • 3
    Sure, I always say that when I re-encode a youtube video or two, only the process was trained / seeded with copyrighted data, it's not that the output is, this is the endless summer! Look, I only looked at that video and than made a program paint a new one!
    – hakre
    Commented Oct 31, 2021 at 20:55
  • 6
    (1) As a non-mod, I can't move this to chat, but it seems to make more sense there (2) As to the question regarding copyright of GitHub Copilot created code, I've made a Law SE question in hopes of getting a better answer, hopefully, one backed by evidence
    – cocomac
    Commented Nov 1, 2021 at 1:36
  • 6
    The quality of the answer should be judged on its own.” I think that is missing an important point. Every human user should have the benefit of doubt, as they might produce a useful answer some day. But there is no reason to allow an AI that is incapable of producing useful answers in general, as it can’t explain its copied code snippets (so it can only produce low-quality code-only answers) and require our moderators to clean up after that AI. It’s not as if there wasn’t already enough work for them without such AIs.
    – Holger
    Commented Nov 1, 2021 at 12:19
  • 2
    @Holger, "AI that is incapable of producing useful answers in general, as it can’t explain its copied code snippets" — that statement is not true. Copilot is capable of explaining code in plain English, perhaps even better, than the average human (especially non-native speaker). Sure, it makes mistakes, more often for complex tasks, but so do humans. And that's why human supervision is required for now. But Copliot (or other language models) will improve in the future, it's inevitable.
    – Aivean
    Commented Nov 1, 2021 at 19:45
  • 4
    GitHub copilot is a remarkable way of circumventing copyleft open-source licenses. It hoovers up all kinds of source code in public repos, then grinds up the ideas in them, then spits out anonymized versions of the formerly copylefted code. Still, maybe StackOverflow could license their cut/paste keyboard to github :-). At any rate, don't cut and paste code from anywhere without understanding it thoroughly. It's just sloppy and rude to your users.
    – O. Jones
    Commented Nov 1, 2021 at 23:07
  • 4
    @Aivean there’s a fundamental difference between assembling and reformatting English text and explaining something. Explaining requires understanding. Including the ability to respond to comments pointing out mistakes, asking for clarifications, or suggesting improvements.
    – Holger
    Commented Nov 2, 2021 at 7:42
26

I am the author of that "Copilot answer", and I'd like to give my thoughts on the matter.

First of all, I admit that my answer was subpar, as it didn't provide any explanation besides the code itself and can be perceived as just a mindless copy-paste, and the disclaimer about "minimal supervision" didn't help the matter. I tried to rectify that by editing the answer.

That being said, I believe it is not relevant to the question of whether Copilot can be used to answer SO questions in general.

Copilot is a tool. I think, that for all intents and purposes of Stack Overflow answers, it's no different from using Google or code completion in IDEs. The only potential issue is the legality of its output.

On that, there is no definite answer yet (as in there is no established legal precedent; probably a good question for law.stackexchange). And I don't think there can be a simple answer either. It probably would differ on a case-by-case basis (see Google v Oracle case), but the point is open to debate.

Copilot and natural language neural nets in general work similar to humans in the sense that they memorize parts of their training data and then recombine them to achieve the given goal. See this article to get a sense of how original the Copilot output is (one recitation event every 10 user weeks).

Currently, there are no restrictions on the usage of Copilot output from GitHub and Microsoft, so I think the legal responsibility to avoid copyright infringement falls on the user. Note, this is pretty much the same as with any user-posted code. You don't know how that code was produced and whether it infringes some copyright. You just trust the user by default. Controversially, with Copilot you can probably be more confident that the code is original.


To summarise, I think that using Copilot to produce parts or the whole of the Stack Overflow answer is no different from using Google. You are still responsible for your answer, in terms of quality, relevance, and legality. The usage (from the legal standpoint) of the Copilot output for any purposes is not restricted at this point.


Update: Since there are lots of questions regarding how exactly the original answer was produced, I recorded a short video, which illustrates the whole process.

28
  • 9
    Your answer here makes sense but it seems to me that it misrepresents your answer being debated. I agree that Copilot is just a tool. Using a tool doesn't inherently make a post bad. Similarly to how one can still ask good questions based on homework. But when the problematic feature becomes a defining property of the question (homework dump, Copilot dump) that's a problem. The key is human supervision. As you know even a lot of humans write terrible and harmful code, and a neural net trained on crap will generate crap. Commented Oct 31, 2021 at 14:31
  • 15
    So when you say "This code was written by Github Copilot with minimal supervision" instead of "I also used Copilot for this answer", you are suggesting that there was no human supervision. Instead of "a tool like Google or code completion in IDEs" you make the tool the author. Tools can't reason about code correctness and code quality. When you give a chimpanzee a keyboard and let it write code at least it's obvious that the result is probably useless. Copilot can write convincing garbage. You can only use it as a tool, but then the fact that you used it becomes irrelevant for the answer. Commented Oct 31, 2021 at 14:31
  • 2
    @AndrasDeak, I already said that my answer was low quality, including the wording of the disclaimer, but that was for reasons unrelated to Copilot usage. I think this is a trivial matter and is not worth discussing. I agree with you on the topic of supervision, and I believe I've stated so in this post.
    – Aivean
    Commented Oct 31, 2021 at 17:25
  • 1
    @Andras Deak: A tool can never be an author, that's not how it (computes) nor an author works. And the author did place the answer there, which requires - at least - a minimum of supervision. Even if it's only to place the cursor and then select start to end , copy and paste into a place that the answer matches with its syntax - I have not read about any complaints on the formatting for example. And then even a disclaimer has been written, which clearly shows that supervision took place, not only a minimal one. And on SO the asking person has to say whether the answer was helpful or not.
    – hakre
    Commented Oct 31, 2021 at 22:11
  • 6
    @hakre, full disclosure, the question was so trivial that I decided to use it as a test for Copilot. I prompted it with a comment, something like //remove the last word from a string. Again, the generated one-liner function was trivial. Even so, I found it doesn't handle the "string without spaces" case, so I added it. The disclaimer that I wrote was more to inform people about the existence and abilities of Copilot, not because I thought it's necessary to attribute it.
    – Aivean
    Commented Oct 31, 2021 at 22:44
  • 4
    @Avean: I was also able to see the now deleted answer and it is far more than just the output of Microsofts code suggestion engine. It perhaps got deleted because of being placed on meta (sometimes called the "meta-effect" as it creates attention). I'm sure if you would have skipped the disclaimer (the one I commented on earlier and think its great you did nevertheless) it would have never got that attention. Also you being responsive here I think is great. Thanks for your contributions. I find this highly interesting.
    – hakre
    Commented Oct 31, 2021 at 22:49
  • 1
    @AndrasDeak, "If the code block was belched up by a neural network and pasted into an answer verbatim, it was authored by the neural network." no, not legally.
    – Aivean
    Commented Oct 31, 2021 at 23:16
  • 3
    @hakre 1) at the time I created this question, the answer was just the output of the AI generation. it was expanded later on. 2) do also note that the entire question was deleted which results in answers being deleted too, so this answer in particular was not the target of a deletion.
    – eis
    Commented Nov 1, 2021 at 8:33
  • 4
    "it's no different from using Google or code completion in IDEs" I mean you have to start writing code yourself for IDEs to offer code completion, and you get to determine whether you agree or want to ignore it. That doesn't seem to be present in Copilot. "Just like google" is also an enormous stretch. The biggest thing missing from both is the human discernment factor, which should not be understated.
    – TylerH
    Commented Nov 1, 2021 at 13:22
  • 5
    "The only potential issue is the legality of its output. On that, there is no definite answer yet" Huh? The legality of its output should be considered the same as the legality of any other output, which does have a definite answer: don't violate users' copyrights. If Copilot takes code from somewhere that it doesn't have the license to use... you've violated copyright and broken Stack Overflow's rules. If Copilot takes code from somewhere and you don't know who wrote it or what license to provide under it, you can't possibly license it/attribute it acceptably for use on Stack Overflow.
    – TylerH
    Commented Nov 1, 2021 at 13:25
  • 4
    Personally I don't see how anyone can rely on Copilot-generated code without having access to the full body of training data GitHub arguably stole for its use, to know which bits are verbatim copied and need to be licensed appropriately (if the licenses even permit it). They have made some noise about some small percentage being 100% identical to existing code, but it only takes 1 function, 1 comment, 1 license fragment to make it exceedingly clear that that output is, well, stolen code.
    – TylerH
    Commented Nov 1, 2021 at 13:30
  • 2
    @TylerH, why do you think that Copilot's output is "copying verbatim"? It can be, accidentally (with low probability), or if explicitly prompted (Q_rsqrt), but that's not the issue in the general case. Don't let the overhyped examples on the internet mislead you.
    – Aivean
    Commented Nov 1, 2021 at 18:16
  • 3
    @Aivean Well, it has to copy it verbatim, first, in order to train on it. Second, maybe you didn't see the Tweet that Zoe linked under the question above: twitter.com/Tercicatrix/status/1411075466094645252 I'm sure this is not an isolated case, just a glaringly obvious one because of the content that it wrote.
    – TylerH
    Commented Nov 1, 2021 at 18:22
  • 2
    @TylerH, "it has to copy it verbatim, first, in order to train on it": copying the code openly hosted on github for the purposes of indexing it is a textbook example of fair use, and listed in GH terms of service. RE: "the Tweet that Zoe linked", I literally mentioned "Q_rsqrt" in my previous comment. In my opinion it's one of the few overhyped cherry-picked examples on the internet that discredit the Copilot's image.
    – Aivean
    Commented Nov 1, 2021 at 18:31
  • 6
    @Trilarion It's the same issue raised by Cody elsewhere here (I think, maybe another thread). If someone plagiarizes code that can't legally be copied, and you copy that plagiarized code and give attribution to the plagiarizer, that's... not really a good situation. You may not necessarily know that the code you cited was illegally obtained and published, but that doesn't excuse the citation's infraction on the same issue. What is Copilot's full corpus of work? Where is it listed? Is there a guarantee that it only scans code that itself is guaranteed published under open licenses?
    – TylerH
    Commented Nov 1, 2021 at 21:34
20

does the user even have the rights to submit such code into Stack Overflow

This I cannot answer, as I am unfamiliar with what rules and agreements there are to Copilot. However, it probably is the biggest point to discuss here - the other concerns you've raised seem like a red herring:

  • user submitting such an answer might not be able to reason about the answer at all
  • there is high rate of incorrectness with Copilot generated code (source1, source2)

We get many users submitting an answer they have not authored themselves. That act, by itself, is not an issue. Users are allowed to refer to other solutions (with proper attribution) and not all work has to be completely original. This includes mentioning well-known algorithms, or even using their implementations. It is part of an age old tradition of standing on the shoulders of giants.

With that said, whether or not a user can reason about an existing solution has a bit more relevance. We do not require this, but it often makes for a poor answer if the content amounts to "use this" without explaining how or why. It is indeed possible that a user who cannot reason about a solution misapplies it to a problem.

In a very similar way, an automatically generated solution can be not useful or misapplied. So, there is very little difference whether it was produced by a live person or an automated system. In the end, the result is the same: we should rate content. And content that is not useful should be downvoted. Exceptionally poor answers may even be deleted.

Note that if it happens that content is useful then the appropriate action would be to upvote it. Does not matter if it was automatically generated or not. Same with any other answer posted by a user.

If we ever get some sort of automated system that leaves helpful and useful answers, then please refer to the final panel of this comic:

XKCD #810

1
8

I'm worried about two things here: quality and false pretenses. It doesn't mean they necessarily have to be a problem, but here they are:

More low quality content?

Are algorithms and large amounts of data already good enough to provide good enough quality to satisfyingly answer programming questions with little to no supervision? Together with the simplicity of generating large amount of content via automated means, this could lead to a surge in low quality content.

Related is that in the comments here people do not seem to have a problem with the content creator not necessarily having to understand what he/she posts and how that works. This doesn't sound like a recipe for high quality to me. At least the content creator should have understood the content. If automatically generated content really is just a tool and carefully checked and tested and understood, I'd say it's okay, but it rather seems we would be happy with fully automated user accounts. The quality of these may vary.

Who actually creates that content?

To me it's a contract that the user name that is printed below a content contribution has a significant impact on the content and not just for example: started a program. Obviously it might be a bit of a grey zone how much a significant contribution to created content really is, but if we seem to be okay with more of less fully automated created content (where the user account is the copilot or maybe only the name giver) then I think putting a user name below that content could be seen as a kind of false pretense and it would be better to print something else there.

Conclusion

Quality is usually dealt with by voting and automatically generated content could additionally be attributed as suggested in another answer. Both might be fully satisfiable. Time will tell.

8
  • "[...] people do not seem to have a problem with the content creator not necessarily having to understand what he/she posts and how that works." Strictly speaking, this is how it works currently; content creator on Stack Overflow absolutely do not have to understand either question or answer to post content. I mean, system-wise. Obviously we humans would like to think they do. Commented Nov 1, 2021 at 20:15
  • "Related is that in the comments here people do not seem to have a problem with the content creator not necessarily having to understand what he/she posts and how that works" I "do not have a problem" with this in that we've had this over the past decade. This joke was posted in 2009 - 12 years ago. It's making fun of a trend that is no less of human not understanding posts than an AI doing the same. I don't know how we enforce posters understand the question and the answer they post. It literally shows up daily. Me "not having a problem" is because
    – VLAZ
    Commented Nov 3, 2021 at 9:21
  • I don't see it as a separate problem to solve than humans posting answers they don't understand or responding to questions they don't understand. Again, this already happens. If you want to address this - great I'd love to see less low quality submissions. I just don't see how we'd enforce it. And in the spirit of being fair to everybody we shouldn't stop it right now. Since AI answers fall in the same problem category, I don't see why we should do anything special to handle them instead of addressing the broader trend of low quality posts.
    – VLAZ
    Commented Nov 3, 2021 at 9:21
  • @VLAZ I see your point. But I argue more about what should be "people should actually understand the content they post in answers" and more quantitatively. If an AI is even worse than the average human here, then quality will go down on average and that's what I would like to avoid, even though there might be some humans even less capable than an AI. You could convince me that an AI is doing great work by showing me a large enough number of examples of answers written by an AI. That would surely reduce my first concern. But that hasn't been done so far. Commented Nov 3, 2021 at 9:47
  • "You could convince me that an AI is doing great work by showing me a large enough number of examples of answers written by an AI." I am myself not convinced we humans are doing great work.
    – VLAZ
    Commented Nov 3, 2021 at 11:30
  • @VLAZ Sure, so maybe we could check and verify the output of both, humans and AI. Nothing against it. But without I'd rather say that AIs are still very dumb and I would prefer the content produced by a human expert any day. For many of the questions on SO, when they aren't actually duplicates, you would probably require a bit of creativity to solve them. Maybe AI should first do the job of detecting duplicates. Might also be an easier task. Commented Nov 3, 2021 at 12:39
  • 2
    "Maybe AI should first do the job of detecting duplicates. Might also be an easier task." paper published on this topic and another one. I don't have the knowledge to evaluate these but just wanted to show there has been some work on this. There are also some community solutions listed here.
    – VLAZ
    Commented Nov 3, 2021 at 15:21
  • @VLAZ Many thanks for the references. What will probably happen in the future is that somebody conducts similar studies with AI written answers. Commented Nov 3, 2021 at 15:34
-8

Stack Overflow is an invaluable resource for many professionals, as well as AI researchers. The process adopted by the site combined with the volume of traffic and collaborators is what produces its awesome value. At the start, Stack Overflow was an innovation, and I think the spirit of the site from the beginning was to collaborate in our different fields to create a significant volume of knowledge. I don't think the particular process of collaboration was as critical as the end result. So long as the people behave well and get value out of the Stack Overflow platform, it should succeed. And clearly it did succeed :). Not to minimize the great execution of the Stack Overflow team.

As AI evolves there may be a day when it will be able to answer every question here. And we should be very happy about it if it comes to pass. In the meantime there will be many stumbles along the way. And I can see a very important role for Stack Overflow to play, to create that future, by allowing this platform to be another point of collaboration with that future AI. I think AI answers are not just acceptable, but should be encouraged. By design, Stack Overflow supervises all activity on the platform including, of course, AI generated answers. The only caveat could be, to not post so many answers such that Stack Overflow overflows, since Stack Overflow is still managed by humans.

I think it would be very useful to have an AI generated answer flag, to help with managing the process.

It is a very exciting time indeed!!!

1
  • 2
    Regarding "I think it would be very useful to have an AI generated answer flag", say hi to natty :) And that's not the only bot that raises flags on SO. There's a whole fleet of them.
    – cigien
    Commented Nov 2, 2021 at 1:53

Not the answer you're looking for? Browse other questions tagged .