205

Last month we ran an experiment to test using a template to help new question askers ask better questions. This experiment is the first in a series addressing the top voted theme for TeamDAG to work on: incoming question quality. This experiment also is the first one that uses our new question quality measure for evaluating questions.

Given that we haven't executed an experiment around question quality before, we intentionally kept things simple so that we could focus on the mechanics of running this new type of experiment. We used a template loosely based on enderland's answer to Jon's question on how question templates could help people ask better questions. There are a lot of other interesting ideas for how we can expand on this approach. We don't in any way think that this specific template is the canonical template to use for this effort. In fact, it may be that there isn't a canonical template and as such may test multiple templates in a multivariate test and/or test a wizard approach that supports multiple templates. But that is fodder for a future meta post. This one is about the results of this experiment.

So, how'd it go?

Well, the good news is that with some tweaking of our A/B testing system the experiment was a success. That means the question quality measure worked as expected and there was a modest impact on the quality of questions asked by users who encountered the template.

The A/B test ran for a little over two weeks. Anyone with a rep below 111 was a candidate for seeing the template. 50% of the total pool was exposed to the template if they asked a question.

Editor with template Screenshot of the Ask box with the following in comments: What are you trying to accomplish? (Please include sample data.) Past the part of the code that shows the problem. (Please indent 4 spaces.) What do you expect the result to be? What is the actual result you get? (Please include any errors.)

Roughly 95,000 "ask a question" click events were registered for each pool of users. Now a lot of people go to "ask a question" and don't actually do so. Generally, only about 46% of users who visit "ask a question" go on to post. For our new user pool the average was closer to 32%, which isn't too surprising. We all know that asking a question on Stack Overflow comes with a bit of fear and loathing (another topic for another day).

More interesting, there was a 3% reduction in questions asked by users who were exposed to the template. I think this is actually a really good thing. There's no way to know exactly why people who saw the template chose not to ask their question. However, my hypothesis is that people who saw the template had a better understanding of the level of investment that we expect from question askers and some of them were a bit intimidated by it. Several scenarios come to mind:

  • Potential askers decided that they needed to do their homework first and come back later to ask.
  • They decided it wasn't worth the effort and didn't ask their question at all.
  • Maybe during the process of writing a better question, they solved their problem without needing to post it.

The other strong finding was that users exposed to the template asked ~3% fewer "bad" questions. Those questions were basically evenly distributed between "neutral" and "good". From the point of view of answerers "neutral" basically equals "bad" because no one was interested in answering it. However, from the point of view of the asker it really is neutral. They asked a question. They didn't get hammered with down votes, but they didn't get an answer either. Our focus is on improving overall question quality, so the upside here is small ~1.5% improvement in good questions.

Overall, we're happy with the results of the experiment. Our testing system is now working for question quality experiments. The experiment points to some hope that some sort of template can provide a measurable improvement on question quality. We don't think this template (alone) is the right solution, but we're interested in running additional experiments to figure out what will work. We are working on a more involved wizardy approach right now. We'll share more on that as it develops.

36
  • 13
    So between the 3% reduction in those continuing and 3% increase in neutral/good questions, that implies an overall 6% reduction in "bad" questions? If so, that's not bad at all!
    – Joe C
    Commented Feb 6, 2018 at 22:15
  • 171
    I imagine a number of the users who decided not to post their question did so because their question was not a debugging style question, and the template lead them to believe that we only accept debugging style questions as the template is not qualified in any way (i.e. something along the lines of another HTML comment saying "This template is for debugging questions, if your question is not a debugging question then ignore this."). I personally don't qualify endless variations of the same useless debugging questions as "good" questions, even if they don't get downvoted or closed.
    – user4639281
    Commented Feb 6, 2018 at 22:42
  • 54
    In short, I think your measure of success is highly naive and is predicated upon preconceived notions which are neither proved nor disproved by your tests, and you have taken the fact that they were not disproved as proof.
    – user4639281
    Commented Feb 6, 2018 at 22:44
  • 30
    @TinyGiant: That's entirely possible, which is why we'll be trying different types of guidance next.
    – Jon Ericson Staff
    Commented Feb 6, 2018 at 22:52
  • 27
    Regarding the 3% reduction in questions asked, another scenario might be that the template did not fit the kind of question the user wanted to ask, and despite the question being on-topic the asker decided not to post it.
    – Bergi
    Commented Feb 7, 2018 at 0:53
  • 23
    Why did you restrict the rep of users in this experiment to 111? I see offenders all the way up to 1k and beyond. You could benefit by raising the bar a bit.
    – cs95
    Commented Feb 7, 2018 at 7:50
  • 9
    From the point of view of answerers "neutral" basically equals "bad" because no one was interested in answering it. This seems to presume that (a) answerers are only interested in answering good questions and (b) questions are only unanswered because noone is interested, neither of which is true.
    – Armali
    Commented Feb 7, 2018 at 8:15
  • 14
    @cᴏʟᴅsᴘᴇᴇᴅ they need to start somewhere but in the final version everyone can skip any templates/wizards/guidance by selecting I know what I'm doing. When you post with that option ticked, the Community user will immediately administer a down vote, even if you're Jon Skeet ... it will be awesome.
    – rene
    Commented Feb 7, 2018 at 8:43
  • 5
    Is it feasible (and could it be useful) to follow-up on any of the potential askers who clicked "Ask a question", saw the template, but didn't follow-through with a question to ask why? (E.g. did "homework" first then asked, decided it's not worth it, self-solved while getting more detail, assumed a non-debugging question wasn't appropriate, other).
    – TripeHound
    Commented Feb 7, 2018 at 11:46
  • 6
    what are the numbers behind 3% and 1.5%? Percentages are handy, but I'm wondering if the numbers are telling a slightly different story behind the descriptive statistic.
    – MattR
    Commented Feb 7, 2018 at 13:34
  • 8
    Do we know how many people simply deleted the comments and asked their question anyway? Commented Feb 7, 2018 at 14:26
  • 7
    It's nice to see that we only want user to post "fix my code" question. I find those very interesting /s
    – Gudradain
    Commented Feb 7, 2018 at 14:33
  • 12
    If you're going to use statistics, you must perform error analysis. A 3% reduction with error of .1% is very different from a 3% reduction with a 5% error.
    – jpmc26
    Commented Feb 7, 2018 at 19:18
  • 7
    In a similar vein as @jpmc26, I'm quite curious as to the simple sample size. How many questions are we talking about here? Is it 100? or 10000? More? Less? This really matters, because if we're talking small percentages, if it's only a hundred questions then 3% is only three questions and hardly representative.
    – Ajean
    Commented Feb 8, 2018 at 16:46
  • 5
    @Ajean - It specifically states that the eligible group saw this template 95,000 times. 32% of them went on to post, making that 30,400 questions posted using this template. This is all ballpark based on their expected averages, so somewhere near 30,000, meaning 3% of those would be 900 questions. Roughly.
    – Travis J
    Commented Feb 8, 2018 at 22:39

9 Answers 9

79

This is a great problem to spend time on. Any percent of improvement is important in my opinion.

I poked around a few of these by searching for the prompts you include. It seems that the prompts are being included in the post most of the time; either passively still with the <!-- comment escapes, or actively with the prompt actually verbatim or styled and then the response below it.

I must say that the questions do look in decent shape overall for new question askers, especially when compared to my experience with the questions in the mentor program a few months back.

My main takeaway from looking at this first set is how important the text of the prompts are. They clearly influence the nature of the post, so any improvement available in the text itself, or the order of the prompts, or the amount of prompts, would in my opinion yield the most value for time spent as far as going forward and could also be used as a stronger indicator for a wizard design if it goes that route.

29
  • 4
    "A wizard" meaning SO asks about the nature of your problem and you get different templates for each of: "I can't figure out why my code doesn't work," "I don't understand the behaviour in this code [that I encountered]," and other archetypical questions? Commented Feb 7, 2018 at 14:27
  • 5
    @Michael Yes. See meta.stackoverflow.com/questions/358600/… for example.
    – TylerH
    Commented Feb 7, 2018 at 15:00
  • 4
    Your first link doesn't state the actual problem until a commenter asked, and no one edited the question to include it. It's also a huge code dump. The last one is also a huge code dump that the asker expects us to hunt through looking for their problem. That's 2 out of 3 of your links that have significant problems. How can you say that they seem to be in "decent shape"? I don't think going from incomprehensible to massive code dump "find my problem for me" questions is an improvement.
    – jpmc26
    Commented Feb 7, 2018 at 22:40
  • 2
    @jpmc26 - Incomprehensible to "here is way to reproduce my problem" is not an improvement? I disagree. Look, I am not saying these questions are suddenly the gold standard, but if you go from nothing, to too much, it isn't inconceivable to get to the point where you are getting a good middle ground. This is why I noted that it would be prudent to adjust the wording to something more inline with that middle ground. That it is even possible to have that conversation is definitely success.
    – Travis J
    Commented Feb 8, 2018 at 22:03
  • 2
    @TravisJ No, it isn't because the asker is still not thinking through their problem either way. Rewarding questions with little thought put into them will beget more questions with little thought put into them. Turning SO into a free personal code fixing service is not an improvement.
    – jpmc26
    Commented Feb 8, 2018 at 23:24
  • 1
    @jpmc26 - I don't think you understood what I said. If they manage to get their code into the form of an example instead of the form of a code dump then that would be ideal. This was the point I made, and in that circumstance none of what you said holds any weight.
    – Travis J
    Commented Feb 8, 2018 at 23:34
  • 2
    @jpmc26 it is impractical to expect us to go straight from our current problem status to perfectly solved in one step. When you are dealing with a huge codebase like this and millions of users visiting every day, asking thousands of questions on each of those days, the reality is that iterative changes are the best bet. This is an iterative change in the behavior of these askers. They went in the right direction. They just went too far. Now that they know the approach works, more or less, devs can scale back to try and reach that middle ground that Travis refers to.
    – TylerH
    Commented Feb 9, 2018 at 14:56
  • 2
    @jpmc26 and yes, too much code is objectively an improvement over no code. With no code, we literally can't help them. With too much code, it is only your attitude that is blocking them from getting help.
    – TylerH
    Commented Feb 9, 2018 at 14:57
  • 1
    @jpmc26 The problem is that you need to take a step toward question askers, not tell other people to take a step back. Your POV here is entirely self-centered. That's fine, because it's your prerogative to care about what you want to care about, but the Stack Exchange network is a symbiosis between answers and questions. If there are no questions being asked, then there is no Stack Exchange. And if you don't like the state of a question, and can't be bothered to provide constructive criticism or suggestions, you're always free to just move on to another question.
    – TylerH
    Commented Feb 11, 2018 at 7:11
  • 1
    @jpmc26 - You should spend less time harassing answerers, because the experience of answerers being harassed by people who think everything is too broad or unclear is a real hassle. Don't like a question? Action it, move on. Let roomba deal with it. Want to help keep the site clean? Then look at questions more than a month old which may need cleaning. Don't like an answer because you don't understand the question? Seriously, leave the answerer alone. Drivel about answers not addressing questions you think are unclear or too broad is honestly the waste of time you are seeing.
    – Travis J
    Commented Feb 12, 2018 at 19:41
  • 1
    @TravisJ I excluded the time spent commenting on an answer from what I mentioned above. Rewarding bad questions with answers encourages more bad questions. It is a real problem. Askers don't care if their question gets closed if they get an answer anyway. The system doesn't work fast enough to prevent it, and it's only going to get worse if it isn't prevented. And moderating posts a month old? That is way too late to influence asker behavior, which means the problem just keeps getting worse in the mean time.
    – jpmc26
    Commented Feb 12, 2018 at 22:05
  • 1
    @jpmc26 - It is provenly false that berating answerers influences asking behavior. If anything, that type of behavior just causes high level contributors to leave the site, making it worse on everyone.
    – Travis J
    Commented Feb 12, 2018 at 22:11
  • 1
    @jpmc26 - If you have meta commentary to make on an answer, then make it on meta, and leave it off the main site.
    – Travis J
    Commented Feb 12, 2018 at 22:12
  • 1
    @jpmc26 - At least own up to it. Why deny something which is clearly present in your history? Your meta commentary on the main site is problematic. Meta was made for this exact purpose, to keep the main site clean from that. I will not enumerate them, they are littered all over your comment history in your profile.
    – Travis J
    Commented Feb 12, 2018 at 22:20
  • 1
    To note, this is rather off topic, I did attempt to move this to chat. That said, what specifically is causing you to spend 10 minutes determining how to vote on each question? If it takes that long, maybe you should just focus on other types of posts, such as maybe posts with more votes, or which have survived for a day, etc.
    – Travis J
    Commented Feb 12, 2018 at 22:31
56

I agree with Tiny Giant:

I imagine a number of the users who decided not to post their question did so because their question was not a debugging style question, and the template lead them to believe that we only accept debugging style questions as the template is not qualified in any way (i.e. something along the lines of another HTML comment saying "This template is for debugging questions, if your question is not a debugging question then ignore this."). I personally don't qualify endless variations of the same useless debugging questions as "good" questions, even if they don't get downvoted or closed. In short, I think your measure of success is highly naive and is predicated upon preconceived notions which are neither proved nor disproved by your tests, and you have taken the fact that they were not disproved as proof.

In other words, the metrics are not being examined carefully. They are being assumed to mean things that may or may not be what they mean. This is extremely dangerous. Misunderstood data will very rapidly lead you to the wrong conclusions, much more quickly than experience and well trained intuition will, especially in arenas where much of what's being measured is subjective.

Additionally, these percentages are tiny. Do we even have an estimate on the error bars of your measurements? I don't see one. The error needs to be much smaller than these numbers, or these results haven't proved anything. Yet it seems to be assumed that any difference is meaningful.

Furthermore, data is really only meaningful if you can examine it in light of a good model of the problem space, and the model you're embracing is suspect. Models really should only be considered reliable if they've been tested hundreds or thousands of times over many different situations and a significant amount of time and found to correctly predict the results. What results has your model successfully predicted, and over how long a period of time has it been tested? The model you introduced is only a couple of months old. It's far too soon to be basing any wide-reaching decisions on.

I do not like this trend of incautious use of data that I see SO trying to embrace.

17
  • 12
    I think I'm as skeptic about statisticians and their science as you are but I would give the team a bit more credit and if possible tips what they should measure instead. This answer only offers what I picked up as well from my own study which isn't really helpful. It might be useless what they have now, but it is a first iteration to verify if the model is sound. If it isn't, which you seem to assert, then offer what needs to be improved for the next run. We would all benefit if question quality improves. Let's offer our shared knowledge to have them not make a guess at what would work.
    – rene
    Commented Feb 8, 2018 at 8:51
  • 8
    @rene I'm not opposed to your suggestion, but my experience tells me that bad questions are not infrequently upvoted or answered and often go unclosed. I'm not sure quality can be measured; if votes don't work, then I can't think of anything that does. How can you measure research or thinking the problem through or narrowing it down? If SO wants to use a measurement based model, I think it's on them to demonstrate the correctness of the model through predictive power, but if part of the model is even just measuring quality, it's kind of a catch 22. Thanks. I'll mull over what I can edit.
    – jpmc26
    Commented Feb 8, 2018 at 9:15
  • 3
    By your estimates here, no one would have ever made anything significant due to being limited by not knowing the result before doing it. Don't let perfection get in the way of progress.
    – Travis J
    Commented Feb 8, 2018 at 22:08
  • 8
    @TravisJ That is not what I am saying. The most advanced fields of science are the ones where measuring results are clear cut (physics, chemistry). The ones where measuring results is not clear cut have more difficulty getting traction (sociology, economics, psychology). SO is different in that bad analysis can be imposed from the top down; we've seen plenty of economies crash from that sort of effect. I am not asking for perfection. I am insisting that if you're going to use data, don't make sweeping assumptions and approach it with a great deal of humility. Thanks. I'll see about clarifying.
    – jpmc26
    Commented Feb 8, 2018 at 23:21
  • 16
    We used our standard A/B testing framework and most of the results given above show very low p-values. For instance, the p-value for the reduction in bad questions was 0.0022 and 95% confidence range of -6.05% to -1.33%. So we are pretty sure it made a real difference and wasn't just a result of this particular sample. But that doesn't really matter because we haven't made any decisions except to keep doing experiments. The goal is to improve the system based on evidence and that's what we will continue doing. I think you misunderstood Joe's post or at least made some hasty assumptions.
    – Jon Ericson Staff
    Commented Feb 9, 2018 at 2:51
  • 10
    @JonEricson The decision to keep doing experiments in the manner you did this experiment is what is being questioned. This experiment, its content, and how it was conducted was questioned a multitude of times from the start to no avail. I personally spoke out multiple times against the unqualified use of the debugging template, and you proceeded without any regard to the community's concerns. I'm sure the goal is to improve the system based on evidence, but I have seen no evidence to support your claim that you are doing that now or will ever do so in the future.
    – user4639281
    Commented Feb 9, 2018 at 7:58
  • @TinyGiant I'd take a guess that they're starting out with the debugging template, not because it's necessarily the best possible question that SO receives, but because it's relatively clear-cut and easy to moderate/measure. That's probably also why the "MVCE plz" fad kind of caught on to begin with -- when there's thousands of things to review (or rate and vote on), an easy "good"/"bad" criteria might end up taking priority, whether it will help the site in the long run or not. It's probably harder to measure a good conceptual question.
    – jrh
    Commented Feb 9, 2018 at 17:28
  • @jrh there has been nothing stopping them from qualifying the template other than their own will to not do so as far as I can tell.
    – user4639281
    Commented Feb 9, 2018 at 18:57
  • 4
    I don't know if you've noticed this, @TinyGiant, but we always get questioned when we do anything these days. We use feedback as one data point, but we can't rely on anecdotal evidence when we get thousands of questions a day. At that scale, we have to use statistics too. I haven't forgotten my comment from October. We're still early on in this process and a lot of things could change before we're done.
    – Jon Ericson Staff
    Commented Feb 9, 2018 at 19:13
  • @Jon no one has informed me yet why it is absolutely impossible to qualify the templates before showing them to the user. That's something that could have been done from the start and would have reduced confusion quite a bit, but was either ignored or dismissed.
    – user4639281
    Commented Feb 9, 2018 at 19:44
  • @TinyGiant: I'm not sure I understand what you mean by "qualify the templates". Do you mean give askers a chance to choose from several types? If so, we are considering that for future experiments. If you mean running them past people on meta, that's kinda what I did back in October. But maybe you mean something else?
    – Jon Ericson Staff
    Commented Feb 9, 2018 at 19:59
  • 1
    @JonEricson at the very least qualifying the template would be to tell the user that the given template is for a specific type of question and that if their question is a different type of question then the template should be ignored. (As I elaborated in my first comment to the question here, one I'm sure you havent overlooked)
    – user4639281
    Commented Feb 9, 2018 at 20:03
  • 2
    @jon I was using the definition: limited, modified, or restricted; not absolute. E.g. A qualified statement.
    – user4639281
    Commented Feb 9, 2018 at 20:42
  • 1
    Something like "What type of question are you asking" with some options like Help debug, other could probably work
    – pfg
    Commented Feb 12, 2018 at 19:38
  • 1
    @TinyGiant FWIW I was reading 'qualified' to mean 'tested for effectiveness' or something similar, rather than 'limited'
    – TylerH
    Commented Feb 13, 2018 at 16:47
46

Generally, only about 46% of users who visit "ask a question" go on to post. For our new user pool the average was closer to 32%, which isn't too surprising. We all know that asking a question on Stack Overflow comes with a bit of fear and loathing (another topic for another day).

More interesting, there was a 3% reduction in questions asked by users who were exposed to the template. I think this is actually a really good thing. There's no way to know exactly why people who saw the template chose not the ask their question.

I want to point how that the search that powers "Questions that may already have your answer" is quite a bit stronger then the normal search bar. As an experienced user I commonly start writing a question just to get a match there. New users might be doing the same thing; with a little guidance, they have better titles or questions and it then shows a related question that solves their issue.

6
  • 16
    This is a great point and one that we thought about. We've talked about doing a test where we emphasize the "find an existing answer" step explicitly and then try to capture data for people who leave "ask a question" because they've found a solution. Also, we are working on some search improvements that will hopefully make search work better for this as well.
    – Joe Friend StaffMod
    Commented Feb 7, 2018 at 16:28
  • 2
  • 2
    It's the same for me. I don't post most of the questions I come here to ask, and about half of the time it's because I found a duplicate in that search that I didn't find with a normal search (the other half it's because I figured it out on my own while formulating the MCVE or question details). Commented Feb 7, 2018 at 22:11
  • It should also be noted that the sample was of users with fairly low reputation, which implies they are fairly inexperienced with the site. Without asking directly, we can't know what people planned to do when they click the ask button, but I'd be surprised if very many of them go there for search.
    – Jon Ericson Staff
    Commented Feb 9, 2018 at 3:03
  • 2
    @JonEricson, that point I was trying to make was that by guiding them to ask better questions, the search would be inherently more helpful, even if not the primary reason for going to that page.
    – amflare
    Commented Feb 9, 2018 at 5:27
  • I don't only use this way to find similar questions, I use it as my rubber duck way! I start writing the question, and each time I try something _ out of let's say 5 possible way to fix it _ I include it in the question. Just today I actually found a solution for a problem doing this.
    – Paul Karam
    Commented Feb 15, 2018 at 12:43
39

A sort of running summary on key take-aways from this:

One template doesn't fit all

We're off to see the Wizard. Well, something Wizard-y. Trying to predict intent while someone asks a question programmatically is simply way too error prone, so we'd do better giving new users the opportunity to use a guided interface (and strongly suggest it). Something like (e.g.)

  1. I need help debugging a problem
  2. I'm stuck on how to accomplish something
  3. I don't understand how a library or tool works
  4. ...
  5. ...
  6. Your question is probably off-topic (think of something that would actually deter people from asking instead of just ignoring the message, like turtles, or something)

... something like that. We'd have to really work on the copy here so we're much more likely than not to present the user with a clear choice, and the menu selection needs to be really short (like no more than 5 things).

Then, we load a template from it, break out text inputs instead of using commented prompts, and then concatenate everything as the final question (with a preview perhaps). Whew! That's a lot and we need to take baby steps here, or we might end up realizing it broke some time ago but not really be able to tell when.

Titles are still hard

So, good questions do get folks off to a better start (as in they don't have a horrible experience because their stuff got closed or down-voted), but we need to help them attract the right folks to give answers, too!

I firmly believe that we need a title-strength-o-meter similar to those password strength meters. Measure common words, words that are tags too often appearing, capitalization, punctuation, length, presence of verbs, etc. We also need to tell folks this is your chance to get the Jon Skeet of COBOL to actually read your question, she's only going to see the title in the tag feed she reads to find things to answer.

Will update this as a scratch pad of sorts of things to keep in mind as we go with this. Due to even trivial changes needing a few days to test, things are going to go a little slow, but they are moving and the results do look promising.

Feel free to add to this!

8
  • 2
    @rene I think such an option should trigger a magnificent, mesmerizing turtle animation, if anything. At that point, if we let their attention shift even just a little, they'll be wanting to ask what color our socks are and we just don't want to go there. Turtles. Turtles are the future.
    – user50049
    Commented Feb 8, 2018 at 17:16
  • 2
    I was under the impression by Joe Friend's reply to my comments under the question here that the next step was going to be more unqualified templates. If this answer is indicative of what is to come then I am much more hopeful. My only request would be that special attention be paid to data collection and analysis, and to the separation of unqualified preconceptions from the collection and analysis of the data. From what I gather, there wasn't much thought put into any of that in this iteration (which in my mind makes no sense as that is the entire point of testing a change).
    – user4639281
    Commented Feb 8, 2018 at 20:45
  • @TimPost "We're off to see the Wizard. Well, something Wizard-y." That does sound a little familiar... :-P
    – TylerH
    Commented Feb 9, 2018 at 15:01
  • @TinyGiant Well, there are many steps, but major steps are those in which we say we're convinced of something based on the data that we observed. We saw enough to (even through possible noise) be relatively certain that the idea is worth the work required to pursue it, which is what this initial round of tests was designed to yield. I think you're right in that we need a framework to test templates that doesn't stand a very good chance of always showing the user the wrong one, and that might just be putting together an initial 'wizard' - we're not sure yet. [1/2]
    – user50049
    Commented Feb 9, 2018 at 16:02
  • @TinyGiant We might also test some other templates just to see if we get consistency, and what happened to questions where the template did fit, just so our initial 'wizard-y' test has some templates that have at least been vetted (so we have a baseline at which point we can say "that's the wizard, not the template, we think") - in short, this is going to take a long time to do correctly and we're well aware of that. [2/2]
    – user50049
    Commented Feb 9, 2018 at 16:03
  • 1
    [3/2] And in general, with something like this, it's very hard to not set yourself up so your next step invalidates the previous one since you failed to account for stuff, and that's why it's going to take some time (I know 3/2 doesn't work, but it's Friday and a very low value of two with a very high value of three so meep.)
    – user50049
    Commented Feb 9, 2018 at 16:05
  • 1
    So more unqualified templates. Great.
    – user4639281
    Commented Feb 9, 2018 at 18:58
  • New users will first need to learn about our lord and savior Jon Skeet for the suggestion in the second-to-last paragraph to work
    – SeinopSys
    Commented Feb 12, 2018 at 15:31
17

So clearly getting code was possible through this process, but perhaps a little too much code came through. The prompts also seem to heavily lean towards code. Since they may be changed in the future, I thought I might make a suggestion, just food for thought.

<!-- What specific problem are you trying to figure out? Be descriptive. -->

<!-- What do you know, and what do you need to know? -->

<!-- Show your research, explain what didn't work. -->

<!-- If you have code, show the minimal portion relevant to the problem. -->

<!-- Show exact error messages if you have them. -->

These prompts are inspired by your prompts, as well as

I am posting this a separate answer so as to avoid making it look like the prompts suggested got as many votes as my previous answer did.

3
  • 1
    2nd link returns page not found!
    – User2321
    Commented Feb 15, 2018 at 13:17
  • @User2321 - It was deleted due to users abusing its use. Here is the reasoning given for its removal. So, in order to access that link you need to be 10k+ at MSE. Normally I wouldn't give a link like that, but everyone involved with the template project would have access to it, as they are all employees so I made an exception.
    – Travis J
    Commented Feb 15, 2018 at 19:24
  • Fair enough, thank you for the answer.
    – User2321
    Commented Feb 18, 2018 at 19:18
14

Well, color me surprised.

I still have my doubts that this is the right approach, but given that we saw fewer [bad] questions, I'm fairly happy to see this level of progress.

Using this information, a correlation between question attention (e.g. number of views, number of answers, comments, edits, etc) and the template should be looked at. I'm curious to learn if this can lead to a turnaround in positive question engagement.

2
  • 15
    The opposite would have been surprising: how can we imagine more bad questions with a template to follow? The danger is to have fewer good questions as well, when one has a good question in mind that doesn't correspond to the template, one may not post it.
    – Cœur
    Commented Feb 7, 2018 at 2:22
  • 9
    An explicit option to skip the template under the condition the poster explains their reason? "Ask a question" -> "Please use <this template> or <ditch it>" -> [if chooses the second option] "Here you go: an empty textarea AND an input to tell us what was wrong with the template". Maybe too complicated at this stage, but as straightforward as I can think of.
    – ob-ivan
    Commented Feb 7, 2018 at 8:52
8

I'm glad to see this resulted in measurable success.

I think a good next step is working to find ways to either have multiple templates or finding ways to aid in the questions that are not strictly code fix types of questions.

I'd also be interested to see some sort of textual analysis on questions which had the template to see if there is any trend in question quality to which fields got filled out (if any) etc.

2

As Tim Post notes, one size doesn't fit all. Customizing the prompts based on the tag weights could prove beneficial in reducing low-quality posts. An idea entails:

  • Weighting
  • Prompts
  • Format
  • Defaults
  • Reputation

Weighting

Reviewing jasper-reports questions, many are tagged with:

  • jasper-reports
  • java and jasper-reports
  • jasper-reports and jasperserver
  • jasper-reports and jaspersoft-studio
  • jasper-reports, jasperserver, and jaspersoft-studio

The jasper-reports tag is the predominant subject matter in all of those combinations and as such can have the heaviest weighting.

Prompts

Posts weighted most heavily for jasper-reports would have a different set of prompts than the defaults. For example, a vast majority unanswerable questions do not include problematic source code. As such, it would be helpful to prompt the user as follows:

<!-- What are you trying to do? -->

<!-- Paste JRXML. -->

<!-- What do you want to happen? -->

<!-- What actually happens (show any errors)? -->

If the post is tagged with jasperserver, appending a prompt would be useful:

<!-- What version of JasperServer? -->

Similarly, questions tagged with java could prompt for additional source code:

<!-- Paste Java code. -->

Format

One issue with the templates as comments is that they don't appear in the preview. Being able to control the template formatting for individual tags could be helpful. Such as:

# Background
<!-- What are you trying to do? -->

# JRXML Source Code
<!-- Paste JRXML. -->

# Java Source Code
<!-- Paste Java code. -->

# Expected Behaviour
<!-- What do you want to happen? -->

# Actual Behaviour
<!-- What actually happens (show any errors)? -->

Defaults

For tag combinations that are not weighted, a default question template can be provided.

Reputation

Prompts are only provided for new users, such as any account having fewer than 250 reputation points. What that magic number is (10, 100, 250, etc.) can be determined over time.

7
  • Do (new!) users regularly tag first and ask questions later? I get the impression they typically only stop to wonder what the tags are for after pressing the "Post" button. And then they accept the suggested defaults – without bothering to read their description. Leading to "C and C++? Because The System Told Me To."
    – Jongware
    Commented Apr 26, 2018 at 20:40
  • And who maintains these per-tag templates? There are a lot of tags on Stack Overflow.... Commented Apr 26, 2018 at 20:45
  • @MikeMcCaughan: They'd be maintained by the community (e.g., anyone with 20k+ reputation and people having a gold tag badges). Commented Apr 26, 2018 at 21:06
  • 1
    @usr2564301: Somewhat of a Catch-22: automatic tag suggestions are proposed by content analysis, which can only happen if there's content. It may be possible to prepend or append the prompts when there is sufficient content to propose tag suggestions, which is not nearly as nice as having the prompts before writing begins. Commented Apr 26, 2018 at 21:13
  • Would it technically be feasible to always ask for tags first?
    – Jongware
    Commented Apr 27, 2018 at 1:01
  • @usr2564301: I don't know, but given the technical nature of the site, people asking questions must know the general area of technology giving them troubles. It may be that a wizard would improve question quality. At that point, tags first would be trivial. Commented Apr 27, 2018 at 1:22
  • I am leaning towards the idea that a very strict wizard-like interface for First Time Askers would be a good thing to have. Question #1 might be a simple radio button: "Is your question directly related to programming? Y/N". Some evidence for that: How do I replace stuff using my document design software?
    – Jongware
    Commented Apr 27, 2018 at 1:27
-1

Some feedback about why people might click on Ask a question without actually posting a question:

There's no better place to see which tags that exist on the site than to type something in the tag window of "Ask a question" and see what pops up. I do this all the time, when suggesting tags or participating in meta.

Other a alternatives is to start editing a post and click the tag window there, which isn't good if you actually have no intention of changing anything in it. Or to click add favourite tag, which has a worse GUI with no mouse-over with tag excerpt.

The general search feature of the site doesn't have a tag search, or at least I never found it.

3
  • 6
    Huh? There is a tag menu item in the top bar, between questions (developer jobs) and users with a search option
    – rene
    Commented Feb 8, 2018 at 15:12
  • 3
    @rene Now how come I never noticed that before! Okay so this is probably mostly irrelevant then, unless others like me haven't been paying much attention to the top bar.
    – Lundin
    Commented Feb 8, 2018 at 16:16
  • 6
    You probably never noticed it because you're a veteran user ...
    – rene
    Commented Feb 8, 2018 at 17:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .