-11

Background

I am developing a tool designed to help programmers use a popular data-manipulation library. This tool asks the user for input-output examples and an English description or list of keywords describing the desired manipulation. Then, the tool will search for a program using the library that satisfies the examples. If such a program is simple enough, it will be found and displayed to the user.

A proof-of-concept demo of the tool was well-received by the team developing this library, and they expressed excitement about the public use of this tool once it is completed.

There are already many questions on Stack Overflow that could be answered with the help of this tool. Such questions have the following properties:

  • The question indicates a desire to use the library, either in the text or the tags.
  • The question describes in English what kind of manipulation is desired.
  • The question provides one or more input-output examples demonstrating the desired manipulation, or the English description of the desired manipulation is so clear that such an example could easily be constructed.

Question

Would it be appropriate to do the following:

  1. Manually identify questions with the properties above. (We may attempt to do this step automatically in the future.)
  2. If the question does not provide an input-output example, manually create one.
  3. Use the tool to automatically search for an answer to the question.
  4. Automatically post a generated answer (if found) on Stack Overflow.

I anticipate auto-generated answers to have the following form (suggestions for improvement are welcome):

Here is one way of achieving this manipulation:

[code]

For example, given the inputs

[representation of the inputs for one example]

this program will produce the output

[representation of the corresponding output]

This program uses the following functions from [library]:

  • [Link to official documentation for function 1]
  • [Link to official documentation for function 2]
  • etc.

This answer was generated and posted automatically by [tool]. Try it yourself at [link to tool].

I am a bot, so unfortunately, I cannot respond to comments.

Note that we cannot automatically write an intuitive explanation of why the program works. However, we expect most generated programs to be understandable by themselves, especially with links to official documentation for all relevant functions.

What does the community think about these kinds of auto-generated answers? Some possible effects of this include:

  • Users of the library become aware of a self-help tool that can often answer this class of question within seconds.
  • The tool can provide answers to old questions that have not yet been answered.
  • The tool can produce an answer that complements an existing answer. The auto-generated program may be simpler than the existing answer's solution program, and human-written answers rarely include links to documentation for all relevant functions.
  • Barring critical bugs in the tool, it is guaranteed to produce a program that matches the input-output examples. Sometimes human-written answers are not complete because they only solve a simplification of the question.
  • Often human answers propose solutions that involve deprecated functions. The tool can stay up-to-date with deprecation and new features offered by the library.
  • There is reduced burden on the library developers or other experts to manually answer such questions.
  • The automated system is not available to answer follow-up questions posed by the question asker via comments.
  • If the input-output example is too simple or ambiguous, the auto-generated program might not generalize fully, or it might fail for edge cases. (I will attempt to avoid or clarify such questions when manually identifying questions to process, but mistakes are still possible.)
  • Even though auto-generated answers provide a solution program with helpful links to documentation, some might still consider them to be spam.

What additional policies should I follow to ensure that the auto-generated answers are well-received? For example, perhaps it would be better to only process questions without an accepted answer.

Please keep in mind that manually reviewing every generated answer before posting it is most likely not an option on my end. However, high-level human monitoring of the tool's progress is reasonable, e.g., manually investigating negative-scored answers and modifying or deleting them if necessary.

Notes

This is not a proposal to search Google, or any existing knowledge base, for already-written answers. The goal would be to use program synthesis techniques to generate new helpful programs from scratch. In our preliminary testing using existing questions on Stack Overflow, our system can sometimes find better solutions than the ones proposed by human answerers.

12
  • 17
    Sorry but having a bot posting the "same answers" to many different questions is the exact opposite of the usefulness of the site. The point is not providing everyone with an answer, but instead to point people to the correct answers, possibly new but more likely existing. So unless your bot is prepared to mark a whole load of duplicates to existing answers, then I can't see it being welcome here.
    – Neil Lunn
    Commented Nov 26, 2018 at 6:37
  • 2
    I think you misunderstand. The bot won't be posting the "same answer" (not sure where that misconception came from). The bot is searching for a correct program for that particular question, and posting it if found. Different questions (i.e., different input-output examples) will lead to different correct programs and therefore different answers. Commented Nov 26, 2018 at 6:44
  • 4
    I think I "understand" very well thankyou. "Automatically post a generated answer (if found) on StackOverflow." That's your own content. I did not write it and I certainly did not misunderstand it. It's bad enough that people here think that searching Google and then re-posting the first answer they find is an acceptable practice. Proposing something to do that as an automated process is not wanted here.
    – Neil Lunn
    Commented Nov 26, 2018 at 6:48
  • 1
    @NeilLunn How'd you get from "automatically post an answer" to "post a bunch of duplicate answers"? If they've got a tool that can do what they claim here, it wouldn't be much of a stretch to avoid duplicates. Not everything has to be perfect at v0.1
    – Undo Mod
    Commented Nov 26, 2018 at 6:50
  • 1
    @NeilLunn No, sorry. I am not proposing to search Google, or any existing knowledge base, for already-written answers. I understand and agree that that proposal is unacceptable. Please read about the field of program synthesis. We want to generate new programs from scratch and post them here as answers. Commented Nov 26, 2018 at 6:51
  • 1
    There is a large number of shell programming questions which are trivially answered by shellcheck.net but I still don't think it's appropriate or welcome to post answers stating as much, or generating an answer out of the results from shellcheck. A comment pointing to the existence of your tool would seem more appropriate, and better aligned with the objections some commenters have posted. I still don't think it should be automated.
    – tripleee
    Commented Nov 26, 2018 at 6:53
  • @tripleee IMO, a bunch of comments saying You can try using [tool] to answer this question is even more spam-like, since it just promotes some service without actually answering the question. Also, is shellcheck.net down? Commented Nov 26, 2018 at 7:44
  • That's why I don't endorse automated comments. It's good for the community to be able to point to a tool instead of having to explain the same old stuff again and again, but I feel it needs to come from the community who leave comments like this on a daily basis anyway.
    – tripleee
    Commented Nov 26, 2018 at 7:54
  • Also, yes, it does seem like shellcheck.net is down at the moment. You can examine the source code at github.com/koalaman/shellcheck which also mentions a number of ways to install the tool locally. I left a bug report at github.com/koalaman/shellcheck/issues/1400
    – tripleee
    Commented Nov 26, 2018 at 7:55
  • 10
    Valid questions in the class "I have A and I want to transform it to B" also have code that doesn't work, and useful answers to them explain why that code didn't work. Answers that just dump in working code (and by your own admissions can't explain to the OP why that's the solution) aren't so helpful. If people have to indicate a desire to use it, can't they just... use it?
    – jonrsharpe
    Commented Nov 26, 2018 at 7:58
  • I would love to try the program, but I'm not sure posting such answers is a great idea on SO. Especially because of the missing explanation. You should also think about how you want to handle comments under your answers and how you handle edits to the initial question. Just posting answers and then leaving it to others to do the cleanup/maintenance will not be received very well.
    – BDL
    Commented Nov 26, 2018 at 8:52
  • Given that OP said that such questions will be manually identified, it's not much different from a person manually pointing out a tool. /// It may be helpful to show some examples, to see if the generated answers are actually useful.
    – user202729
    Commented Nov 26, 2018 at 14:20

2 Answers 2

2

This is a personal answer, not some kind of official mod (un)endorsement.


I love this kind of stuff. I hate shutting down automation on pure "but it's a robot" grounds. If it's helpful, we should use it. If it's not helpful, it'll get shut down. Still...

You still have to be somewhat responsible in your methodology to roll this out.

keep in mind that manually reviewing every generated answer before posting it is most likely not an option on my end.

This means you want to start out with too much volume. I'd start small, manually review stuff, provide a real-time, public list somewhere for others to review. Yes, it feels like an irritating step. I've been there.

I definitely want to play with this. I'm curious how you're identifying input/output (often unclear or highly abstract, but if you have a way to do it... congratulations, you just made an awesome question quality filter). Have the source available anywhere?

As a start, is it possible to throw up a page for people like me to test it on arbitrary questions?

4
  • 1
    Unfortunately the tool is not quite ready for public release, and I don't expect that to happen for a few more months. However we have already received suggestions to post auto-generated answers on StackOverflow as one application of the tool, so I thought I'd get the community's thoughts about that idea earlier in the tool's development process to see if we should strive to support that use-case. Commented Nov 26, 2018 at 6:59
  • 2
    You're likely to get a lot of pushback on any combination of "automation" and "not ready for public release" - just "automation" alone taps into some fear people have. I wouldn't read a negative reception here as "stop working on this" - if it's anything like you're describing, this is a valuable thing. I'd advise you to get a rough POC out for folks to play with. Something concrete to show the value this can have will go a long way.
    – Undo Mod
    Commented Nov 26, 2018 at 7:05
  • Thanks, I'll keep that in mind. Once the tool is public, I can post a follow-up question with a link to the tool so people can try it out. Commented Nov 26, 2018 at 7:07
  • 7
    If someday SO questions could even be answered by some script, I think it would be the end of this platform. OTOH, if a category of questions could be answered by a script, they should all be closed instead of being answered.
    – llllllllll
    Commented Nov 26, 2018 at 9:32
1

If you've written a library that's able to solve an entire class of problems in a generic way, then ask one question about how to solve that class of problems generically, and post your solution as an answer.

If someone is asking a question looking for a solution to a problem within that class of problems, then the post may be a duplicate of that self-answered question (if providing a solution to that class of problems is an adequate answer to the question being asked).

If the more general post wouldn't be an adequate duplicate, due to the information there not being a complete answer to the question being asked, then you could reference the more general problem and add any additional information needed to answer that specific question (for example, if they're asking why they're attempted solution didn't work, or if additional, non-trivial, steps would be needed to apply the solution to their problem). I can't imagine any automated system being able to provide that additional information (what you've described as being automatically generated would be in effect what's in the more general question) so, given what you've described so far, these additional answers would all need to be written by a person, rather than automatically generated.

2
  • The self-answered question would simply be "How do I begin to use [library] to manipulate my data?" with the answer "Try using [tool], which will find a solution if the desired manipulation is simple enough." This QA pair is not very helpful and is only an advertisement for the tool. The helpful part of my proposition is that the tool can provide a specific solution in the answer to the user's question, instead of just telling the user to invoke the tool themselves to find that specific solution. (Specific solutions can't go in the self-answered post because they depend on the question.) Commented Nov 28, 2018 at 3:22
  • @AutoAnswerer No, you wouldn't ask, "how do I use 'tool'?", you'd ask, "is there a generic solution for solving any type of problem that fits into [description of types of problems that this tool solves generically]" (it's hard to be helpful here with how vague you're being about the types of problems you can solve), in which you'd post an answer saying that you wrote a tool that is able to create solutions to those types of problems, and showing how to do it. Then anyone with any of those problems can get their specific solution by simply following the steps in the answer.
    – Servy
    Commented Nov 28, 2018 at 14:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .