59

It is evident that SO tries to remove hurdles and make it easy for newcomers to post questions. While this surely reduces the "friction" that new users encounter, it has the downside of letting people post anything they want without regard of what might be considered a good, on-topic post.

I would like to focus on a very specific demographic within new users, namely those who have not earned the Informed badge. "Why does this demographic deserve special interest?" you might ask - well let's look at some statistics!

This SEDE query compares the percentage of closed questions between informed and uninformed users. While the query might not be a 100% correct (i.e. it doesn't take into account when the badge was introduced, etc.), the result is still quite interesting:

  • ~0.73% of posts were closed/deleted at the time users were informed.
  • ~6.42% of posts were closed/deleted at the time users were uninformed.

That's an order of magnitude difference!

I know that getting the informed badge is trivial, but some users won't even take a few seconds to hastily scroll through the tour page. I believe that it is time to take action and start incentivizing more strongly the education of oneself about the site's rules (at the very least - going over the tour page). Even if just a fraction of those who just scroll ends up reading - that would already help.

I can think of a couple of approaches:

  • A harsher approach, which involves disallowing uninformed users to post.
  • A lighter approach, which involves reducing the number of required close votes (e.g. from 5 to 3) or delete votes (e.g. from 4 to 2) for posts by uninformed users.

I think implementing something like this would improve the quality of the content on the site and help unclog the notoriously long Close Votes review queue.

25
  • 5
    I think the harsh approach would not work because they would just fast-scroll through the help. And while they can overcome the second one easily by just fast-scrolling, it would mean that they are somehow informed that this is a way of making it easier for them to not get closed, so arguably, they are informed then! I support the second solution Jun 13, 2018 at 15:04
  • 48
    Keep in mind that correlation != causation. Consider that the type of person to go through the tour without actually being forced to is more likely to be the type of person that does their research and takes the time needed to formulate a good question, rather than the act of taking the tour being the reason they've done those things.
    – Servy
    Jun 13, 2018 at 15:05
  • 4
    I’m definitely in favor of the lighter approach. Jun 13, 2018 at 15:05
  • 17
    A recent quote from a comment written by an asker who literally copied code from "Do things twice" and was asking how to "Do that thrice": "I've provided executable code that is supposed to work and marked as correct in the previous question. I copied it exactly. I came here hoping not to spend the next hours reading [the documentation] about [the relevant built-in function].". I don't feel an unstoppable urge to help askers like this, nor does this benefit the site in the long run. Jun 13, 2018 at 15:06
  • 1
    @AnderBiguri One can "get informed" either by conventional means, or by means of having a question closed with a big notice that says something like "your post was express-closed; read the rules next time".
    – Dev-iL
    Jun 13, 2018 at 15:07
  • @Dev-iL I think you overestimate the number of people that are going to follow that link when their question is closed.
    – Servy
    Jun 13, 2018 at 15:07
  • 4
    @Dukeling I didn't say I was opposed to adding hurdles, I'm simply indicating that forcing people to take the tour is unlikely to have as much of an effect on question quality as this proposal seems to indicate. The most probable outcome is that you just waste 20 seconds of time for the person asking bad questions, without affecting question quality.
    – Servy
    Jun 13, 2018 at 15:18
  • 3
    @Servy I don't have very big expectations about people reading docs (we have the same problem with mis-taggin questions because people fail to read the excerpts), but perhaps they don't do it "out of malice" (i.e. they don't want to or actively don't care), but because they're just unaware that there are rules should be followed. Maybe if they see some rules, they might consider reading them...
    – Dev-iL
    Jun 13, 2018 at 15:19
  • 5
    Hmm... option II does not prevent new users from asking immediately. If it's a good question, fine. If, (when:), it's bad, it's easier to close and provides a shield when the bad users pop up on meta with 'hostile, rude etc' posts. I would go further and enforce a minimum time spent on reading the tour etc, but that gets shot down every time:( Still, I don't see much downside other than the development effort required, so have an upboat:) Jun 13, 2018 at 15:46
  • 11
    @MartinJames that would be very hostile towards all the users who keep getting question-banned and come back with newer and newer accounts to...oh. Jun 13, 2018 at 15:48
  • 1
    How about recording the time actually spent on the tour as part of getting the 'Informed' badge? It would not need to be public, but mods could trot it out when new accounts get their bad questions closed quickly and they moan about it. (also, if it is noted during a fraud check that 50 accounts all spent exactly 4.00 minutes 'reading' the tour.....). Jun 13, 2018 at 15:53
  • 3
  • 2
    @BilltheLizard The stats can't account for deleted posts, so effectively what this is saying is that ~7.2% of undeleted questions are closed, and 90% of them come from people who don't have the informed badge.
    – Servy
    Jun 13, 2018 at 18:59
  • 6
    So IMO there also needs to be a cultural shift. Right now, I often hear "I downvoted, but didn't think the post was bad enough to VTC." In my opinion, it should be the other way round.. voting to close and getting some relevant advice in front of the OP in a timely fashion is just as important as getting the question off the front page. The moment I see an unclear or off-topic question I'll VTC. If I come back and there's some clarification, I retract the vote.
    – jpp
    Jun 14, 2018 at 10:44
  • 1
    @KaranDesai That's a potentially dangerous assumption, because what might be considered on-topic on one site, is categorically off-topic on another.
    – Dev-iL
    Jun 15, 2018 at 9:09

4 Answers 4

19

This data doesn't support the idea that we should make it easier to eliminate posts by new users that disregard documentation. (Note that this isn't the same as saying that we shouldn't make it easier, just that this data doesn't support it.)

Let's look first at the harsher proposal and one of your data points:

A harsher approach, which involves disallowing uninformed users to post.

~6.42% of posts were closed/deleted at the time users were uninformed.

Without knowing what percentage of posts by these users are still open, this seems too harsh. Yes, they do seem more likely to ask bad questions than Informed users, but what proportion of Uniformed users are doing just fine? You have to take open questions into account as well. (And, as was also mentioned, this is such a low hurdle to get over, it probably doesn't make much difference to someone who is really motivated to just not read the information that's right in front of them.)

That leaves:

A lighter approach, which involves reducing the number of required close votes (e.g. from 5 to 3) or delete votes (e.g. from 4 to 2) for posts by uninformed users.

I don't think the data supports this change either.

~0.73% of posts were closed/deleted at the time users were informed.

~6.42% of posts were closed/deleted at the time users were uninformed.

Those numbers do seem to show that informed users ask (marginally) better questions than uninformed users, but it also shows that the community is already closing questions by uninformed users at a much higher rate than informed users. The system is already working, without close voters having any easy indicator of who is informed and who isn't. People are voting based on question quality, as they should.

In order to show that reducing the number of close votes on uniformed users would make any difference, you'd need to show that there are a lot of currently open posts by these users that should be closed but aren't. Ideally, you'd be able to show that a lot of these posts already have 3-4 close votes, so they'd already be taken care of if your proposal was already in place. (I don't know how to do this with a SEDE query. You'd probably want to start by looking at small samples of questions to make it more manageable.)

(Also, you can't make this kind of change and then use new data to "prove" that the change worked. Of course the proportion of closed questions by uninformed users will go up if you make it easier to close them.)


Addendum: After a comment by Servy on the question, I looked at the number of questions asked by Informed vs. Uninformed users. Only about 28% of all questions are asked by Informed users. That's much higher than I expected, but it still accounts for a lot of the difference in closed questions between the two groups. Uninformed users just ask more questions.

Query: Number of questions by people with the Informed badge

5
  • Maybe a query that partitions questions by number of views? I might be convinced to support the lighter approach if you could show that a much higher proportion of low-view questions had 3-4 close votes. Jun 13, 2018 at 18:57
  • I get your point regarding the representativeness of this data, and it makes sense. Followup: close votes fade away after a while if there isn't enough of them to effect closure, no? So a query would need to test whether a question was nominated for closure but in the end wasn't... But aggregating close votes over arbitrary periods of time is hard to do correctly. To conclude, I'm afraid I currently don't have the skills to improve this on my own; I might open a question asking for a better query; at the end of the day I want people to consider whether there's a larger iceberg here.
    – Dev-iL
    Jun 13, 2018 at 19:44
  • @Dev-iL yes, you're right. Close votes do age away rather rapidly. meta.stackexchange.com/questions/136989/… That does effectively kill the idea in my comment. Jun 13, 2018 at 19:49
  • "Uninformed users just ask more questions" or are there more uninformed users asking than informed users asking, and they ask relatively the same number of questions per user regardless of whether they are informed or not?
    – user4639281
    Jun 14, 2018 at 17:58
  • @TinyGiant It doesn't really matter. A big part of the reason that uninformed users (as a group) ask more questions that get closed is that uninformed users (as a group) ask more questions. (The only way that I can see where it would make a difference is if "uninformed" were a relatively small group of people that you could reasonably expect to train to stop asking low-quality questions. It's millions of people, so I don't think that's the case.) Jun 14, 2018 at 18:21
7

A few notes (some based on the comments):

  • Correlation is not causation. These numbers could be correlated because:

  • Experienced users are more likely both to have taken the tour and to have figured out what's on topic independent of that.

    For first posts only, it looks like 4.4% vs 2.3%, which is a bit less convincing.

  • Someone who cares about asking good questions might take the tour, but that doesn't suggest that being forced to take the tour would make someone care about asking good questions.

  • These percentages are not accurate.

Firstly, what about deleted posts (which presumably make up a lot of the closed posts)? If you run your query on Posts instead of PostsWithDeleted, your 6.4% drops to 2.6% with the other percentage staying exactly the same, which requires some further investigation, but that does point to a likely problem with the accuracy (one possible reason might have to do with users getting deleted when their only post is deleted).

Secondly, note that these shouldn't really be treated as percentages - 4.4% closed doesn't mean 95.6% are fine (even ignoring deleted posts), because plenty of not-fine questions aren't closed. Figuring out what the actual percentages are would be much, much harder.

  • The tour is not great for explaining what an on-topic question looks like.

Perhaps requiring people to read the tour could help with overall question quality, but largely because the tiny amount of effort required could demotivate the lazy from asking here rather than actually doing a good job telling users what belongs here - there are better hurdles we could be adding, that would do both.

The tour covers too many aspects of the site - the main part we should be showing to askers is the "Ask about" and "Don't ask about" parts, and it shouldn't be much longer than that (short of maybe adding 1-2 items to the lists). We should perhaps also review the lists to ensure we're addressing the questions we have the biggest problems with, and make the items clickable to expand for more details.

3

A different approach (which could be implemented regardless and in parallel with any other proposal) would be to make it so that uninformed users would see proper help text with tips and examples either on top of each input box (title & body would have separate sections) or on the widget that appears on the right side when you click on the different sections.

The reasons for this being a better approach than simple deletion or muting (which I don't believe has worked for any system anywhere, ever) are as follows:

  • I believe the ultimate goal would be not to be able to rid SO of bad posts as quickly as possible, but make sure as little bad posts are made as possible and the proposals should try to aim for this goal instead of just figuring out what is the quickest way to hide the problem.
  • People who don't know how to do something (eg. post a question according to set rules and format) and don't read the rules are not necessarily people who are not going to follow them, but rather people who have a hard time admitting to themselves they don't know how to do something and thus have a hard time opening and reading external instructions. For such people, simply putting the instructions in a simplified (short, easy and fast to read in a glance) form to a place where NOT reading them takes actual effort will make them follow them better in the long run (it's called propaganda or advertisement and we know it works ;) ).
  • While easier deletion would make the problem less visible to other users, it would actually make the problem (which according to you is the percentage of questions closed made by uninformed users) WORSE, as the percentage of closed questions would jump even higher, as more questions would be closed due to the increased ease of it - the underlying problem would not go away (unless the users whose posts were closed simply didn't return to SO, which I don't believe is a desired outcome and if it is, then I would say it is not constructive nor mature).
3
  • Thanks for your input! This reminds me of an old question of mine. I'd be surprised if the SO UX designers haven't experimented with something like that already. I don't agree that easier closure would make the problem worse, because if it helps clear the review queue (or even the feed in general), less potential answerers will have to wade through low-quality posts, and get demotivated to answer - which is a problem on its own. Faster closing just speeds up the inevitable anyway.
    – Dev-iL
    Jun 14, 2018 at 7:50
  • Experiments have been made with a sort of "input form", where HTML tags like <Explain why you are doing this project> <Put your research here, like what have you already found on your favourite search engine> < please enter your code here, if you have any> etc. IIRC these helped people to ask better questions. I forgot where on meta this was put though...
    – Adriaan
    Jun 14, 2018 at 9:47
  • Very relevant (and recent) post by the staff: “Ask a question” wizard prototype. The wizard can be accessed directly through this link.
    – Dev-iL
    Jun 20, 2018 at 14:36
-4

Direction is always better than restriction since the one improves learning whilst the other stunts it; why am I restricted? What should I have done?

I fear (and I might be misreading you here) that you assume that the uninformed lack the desire to become informed and, by implication, the desire to explore beyond posting. Your first suggestion (harsh measure) has been used elsewhere, but it is worth noting why someone registers in the first place - they already have the problem!

Rarely do people register on to discussion groups or forums unless they require something, and in the case of SO, this is clearly help with faulty code. If you stop them from posting they will either find a way around this or go elsewhere.

I might fall into your category since I have not necessarily formed my questions in the right way (certainly this is true for one of my questions). I certainly read the quote given by another and, momentarily, wondered if it had been taken from one of my questions. I might very easily state I wanted to save hours of reading the documentation, but this is only due to limitations of time, the fact that I have already spent considerable amounts of time trying to resolve the issue and posting my question would be the latest attempt to find the answer from knowledge and, hopefully, more articulate sources (sorry, but too many online resources want to show how clever the author is rather than assist in learning: this is true especially for UK resources, rather than the US, in my experience).

Equally, I might have hit a problem, spent a short while on it, cannot even see a path to test and take, so post the question. Frequently in this scenario, I would suddenly realise what the solution is before I get the answer. This is less so with my postings here simply because I have more experience in programming now.

Either way, I might comment that I don't want to spend hours search simply as a throwaway remark. You certainly cannot assume that the person hit a problem, immediately posted on here and then stopped until a solution appeared.

Adriaan's response is a reasonable one since it aids you in structuring your response, though listing your research is, I feel, an unnecessary hurdle. Most people will put w3schools down until shouted at so many times they end up putting MDN for a quieter life.

I cannot close my own questions at present (is this possible later on, or am I missing something; a giant green button that says CLOSE, in the most obvious place?), which means they must remain in the feed. I cannot delete the one which has been marked as a duplicate, which I would be happy to do, though it gave me a clearer idea of how to structure my future posts.

And, sorry, but the preamble is sometimes nice. The thanking for comments is polite, and accepting that the uninformed are simply the informed with less knowledge might be a useful prompt to offer direction to move them from the former to the latter.

If I have misunderstood your point, I apologise; been a long night solving lots of problems without posting a thing. :)

4
  • I did touch upon a similar idea in one of my comments on the OP. Without doubt, there are various reasons and circumstances for joining the site and posting questions, some of which, unfortunately, give users the false impression that their "urgency" is an excuse for behavior that leads to sub-par posts. I'm a long-time believer that a template of some sort would be beneficial - there's a reason GitHub encourages adding PR and Issue templates to repos. I think there needs to be some standardization, but keep enough room for personal creativity. How to optimally achieve this is the question.
    – Dev-iL
    Jun 16, 2018 at 9:16
  • You see, this is where I disagree with your frame of concept. The implication of 'urgency' is that it is not urgent; yet it often is. Equally, that they are aware that their post is sub-par, which they probably are not. I agree with a template, or advice that pops up as you are entering your first few questions, but to perceive their question in such light is unfair, and can lead to aggressive comments that put the individual off from further involvement; to the detriment of this site, I believe. Not that I consider your comments/post aggressive; I do not. :)
    – Gazzieh
    Jun 16, 2018 at 9:49
  • 2
    The reason people think they can get answers on SO is because SO is known to have a large community of experts that may see one's question and answer it. Let us not forget that in order for the site to stay beneficial to the masses, it must not push away the experienced users. Many ideas for modifications are aimed at making the onboarding and asking experience smoother for newbies, but I don't know if this has a positive effect on keeping the experts around. Just saying that the experience of seasoned users also matters.
    – Dev-iL
    Jun 16, 2018 at 10:04
  • 1
    Could not agree more. :)
    – Gazzieh
    Jun 16, 2018 at 10:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .