Background
I am developing a tool designed to help programmers use a popular data-manipulation library. This tool asks the user for input-output examples and an English description or list of keywords describing the desired manipulation. Then, the tool will search for a program using the library that satisfies the examples. If such a program is simple enough, it will be found and displayed to the user.
A proof-of-concept demo of the tool was well-received by the team developing this library, and they expressed excitement about the public use of this tool once it is completed.
There are already many questions on Stack Overflow that could be answered with the help of this tool. Such questions have the following properties:
- The question indicates a desire to use the library, either in the text or the tags.
- The question describes in English what kind of manipulation is desired.
- The question provides one or more input-output examples demonstrating the desired manipulation, or the English description of the desired manipulation is so clear that such an example could easily be constructed.
Question
Would it be appropriate to do the following:
- Manually identify questions with the properties above. (We may attempt to do this step automatically in the future.)
- If the question does not provide an input-output example, manually create one.
- Use the tool to automatically search for an answer to the question.
- Automatically post a generated answer (if found) on Stack Overflow.
I anticipate auto-generated answers to have the following form (suggestions for improvement are welcome):
Here is one way of achieving this manipulation:
[code]
For example, given the inputs
[representation of the inputs for one example]
this program will produce the output
[representation of the corresponding output]
This program uses the following functions from [library]:
- [Link to official documentation for function 1]
- [Link to official documentation for function 2]
- etc.
This answer was generated and posted automatically by [tool]. Try it yourself at [link to tool].
I am a bot, so unfortunately, I cannot respond to comments.
Note that we cannot automatically write an intuitive explanation of why the program works. However, we expect most generated programs to be understandable by themselves, especially with links to official documentation for all relevant functions.
What does the community think about these kinds of auto-generated answers? Some possible effects of this include:
- Users of the library become aware of a self-help tool that can often answer this class of question within seconds.
- The tool can provide answers to old questions that have not yet been answered.
- The tool can produce an answer that complements an existing answer. The auto-generated program may be simpler than the existing answer's solution program, and human-written answers rarely include links to documentation for all relevant functions.
- Barring critical bugs in the tool, it is guaranteed to produce a program that matches the input-output examples. Sometimes human-written answers are not complete because they only solve a simplification of the question.
- Often human answers propose solutions that involve deprecated functions. The tool can stay up-to-date with deprecation and new features offered by the library.
- There is reduced burden on the library developers or other experts to manually answer such questions.
- The automated system is not available to answer follow-up questions posed by the question asker via comments.
- If the input-output example is too simple or ambiguous, the auto-generated program might not generalize fully, or it might fail for edge cases. (I will attempt to avoid or clarify such questions when manually identifying questions to process, but mistakes are still possible.)
- Even though auto-generated answers provide a solution program with helpful links to documentation, some might still consider them to be spam.
What additional policies should I follow to ensure that the auto-generated answers are well-received? For example, perhaps it would be better to only process questions without an accepted answer.
Please keep in mind that manually reviewing every generated answer before posting it is most likely not an option on my end. However, high-level human monitoring of the tool's progress is reasonable, e.g., manually investigating negative-scored answers and modifying or deleting them if necessary.
Notes
This is not a proposal to search Google, or any existing knowledge base, for already-written answers. The goal would be to use program synthesis techniques to generate new helpful programs from scratch. In our preliminary testing using existing questions on Stack Overflow, our system can sometimes find better solutions than the ones proposed by human answerers.
shellcheck
. A comment pointing to the existence of your tool would seem more appropriate, and better aligned with the objections some commenters have posted. I still don't think it should be automated.You can try using [tool] to answer this question
is even more spam-like, since it just promotes some service without actually answering the question. Also, is shellcheck.net down?