378

Editor's note:

This has been implemented.

This post details the outcome of an experiment that reduced the thresholds for closing and reopening questions to 3 votes on Stack Overflow. Warning: this post is looong. There is no "tl;dr". There is no summary. I've been thinking about this crap almost every day for 7 years, and I'll be damned if I'm not gonna inflict at least a little bit of that pain on anyone else who thinks they're interested.

...if you are interested... Well, there's a lot to chew on here; you might even enjoy it.

But first...

The value of a question lifecycle

So many folks - new and old - think of closing as nothing more than a mechanism for deletion. Which, it is - but thinking of it as just that is like thinking of our trial system as a mechanism for execution: if that was all it was, we could just dispense with a lot of the process. The beauty of this system is that it's more than just one more step on the road to deletion: it's a choice, a branch, in which multiple people can collaborate to create something that wouldn't be possible without it.

The concept of closing did not originate with Stack Overflow, but we lean very heavily on it as a tool for community moderation. It's hard to know how much of this was intentional and how much was organic, but at some point that distinction becomes academic; we have to analyze the system as it exists and consider its future in terms of both its positive and negative effects today.

As we discussed recently on the blog, we're looking at the core systems that underpin Stack Overflow. I'm privileged to have been able to step back over the last couple of months and take a holistic view of closing, examining and discussing its role along with my colleagues. Based on this analysis, I would like to propose the following four-part sentence as a summary of purpose for closing - and the larger "close / edit / reopen OR delete" cycle - on Stack Overflow:

  1. The closing system exists to allow askers and answerers to collaborate on constructing high-quality questions and answers, by...
  2. ...providing a clear path for questions that either cannot or should not be answered to be identified, and...
  3. ...ensuring that they are then either sufficiently improved or...
  4. ...removed from view.

In this summary, #2 corresponds to the act of closing a question, while #3 encompasses both editing and reopening, and #4 is deletion. These are the critical actions that enable this purpose to be realized.

If we assume that this is a reasonable purpose and that close-edit-reopen/delete can successfully accomplish it... Then we're left with only one crucial question:

  • Does the current system enable individuals to trigger each critical action?

That's it. If the purpose is good, then in order for this system to work we just need each action to be triggered at the proper moment. In other words, is the system "efficacious"? If that happens, then we're golden. What does "efficacious" mean in this context?

  1. An asker need not wonder why their question isn't getting answered - the system will tell them, and help them to improve it.
  2. An asker need not wonder why their question is closed - if they've corrected the problems, it will be reopened.
  3. An answerer need not sift through piles of unanswerable questions - they will be removed.

And all of this goes on to benefit not just each asker or answerer, but the world at large: no useless search results, only answers. A virtuous cycle indeed!

If this sounds like... a much happier, less frictious website from the one we all know and love... Well, yeah. Let's talk about that.

The public perception of this experiment, and of closing

If the doors of perception were cleansed everything would appear to man as it is, infinite. For man has closed himself up till he sees all things thro’ narrow chinks of his cavern. -- Wm. Blake

The reality is, closing has been controversial - divisive even - for as long as Stack Overflow has existed; everyone sees something different in it, and wants for different things from it. We collected quite a bit of feedback on this experiment, both here on meta and over on Reddit (thanks to curiousdannii for digging that thread up!). I've grouped these into rough themes, and present them only as a tool to help us all get a bit more familiar with the perspectives of our peers:

We also now run an ongoing satisfaction survey, presented to random users on Stack Overflow. Meg took the time to analyze the feedback that we received in response to our question about what users find most frustrating about using the site. No big surprises there: a lot of people mention closing as their number one frustration. We did see a slight increase during the experiment, but not enough to be considered significant. But keeping in mind the importance of managing perception, we'll be keeping a close eye on such feedback as we proceed with changes. (foreshadowing!)


This all adds up to a lot of food for thought. We're incredibly fortunate to have so many people from so many different backgrounds willing to provide thoughtful input on these ideas; big thanks to everyone who chimed in, both here and out there on them Wild Internets. We heard hopes, excitement, concerns, and overt criticism - and all of that is important when it comes to making improvements.

As seen from this feedback, we're clearly not living in that virtuous cycle I described earlier -- or at least, that's not how our world is perceived. Closing is seen by different people as a useful and necessary tool, underused but also abused, and at times capricious. The mechanisms by which questions get closed and especially how they get reopened are poorly-understood. Folks are frustrated both by the volume and nature of questions that don't get closed and by the volume and nature that are closed. When folks think a system doesn't work, they become less willing to rely on it: perception can very much become reality.

Lowering the threshold for closing and reopening is at best a small step in the right direction here... And potentially a step in the wrong direction. Either way, we must be willing to accept the results and learn from them. So... How'd we do?

Results

Remember that "crucial question" I asked earlier?

  • Does the current system enable individuals to trigger each critical action?

Answering this question about efficacy was the primary purpose of this experiment. To that end, we chose to monitor three separate measurements to determine the results:

  1. Close efficacy. How likely is the first close vote or flag on a question to result in it being closed?
  2. Re-open efficacy. How likely is the first reopen vote on a question to result in it being reopened?
  3. Edit efficacy. How likely is a (body) edit on a closed question to trigger reopening?

Note again that #2 and #3 are both measured as being part of the same action: correction and reinstatement. I happen to know that deletion already works fairly well, so I didn't measure that here. Yes, I know there are things we can do to make deletion work even better; let's talk about that elsewhere.

The change in threshold took effect on Aug 8 at 18:20 UTC and was reverted on Sep 7 at 16:33. It was noticed within minutes of being turned on; anticipating this, we posted a public announcement. The first week of the experiment saw a noticeable spike in activity as users re-engaged with the system. On top of this novelty effect, close and reopen votes can take up to two weeks to age away after being raised, so efficacy on the edges of the experimental period is… a bit fuzzy. To minimize the impact of both problems, I decided to compare two periods of time separated by a two-week buffer:

  • Prior: July 18th to August 1st
  • Experiment: August 15th to August 29th

I also compared the two-weeks immediately after the experiment as a sanity-check:

  • Post: September 7th to September 21st

Prior to the experiment

  • Close efficacy. 36% of initial close votes or flags resulted in either the post being closed or a close review that completed with a verdict of “Leave Open”.
    • 33% of initial close votes/flags resulted in closure - there are very few “Leave Open” reviews, but Triage also filters out some flags. Also, some votes reviewed as Do Not Close were still closed.
    • There were 31502 "first close votes" during this period
  • Re-open efficacy. 89% of initial reopen votes resulted in either the post being reopened or a reopen review that completed with a verdict of “Leave Closed”.
    • 38% of initial reopen votes resulted in the question being reopened
    • 52% triggered a reopen review that completed with a verdict of “Leave Closed” (but some were still reopened)
    • There were 854 "first reopen votes" during this period
  • Edit efficacy. 65% of initial edits after closure resulted in either the post being reopened or a reopen review that completed with a verdict of “Leave Closed”.
    • 7% resulted in the question being reopened
    • 59% triggered a reopen review that completed with a verdict of “Leave Closed” (a few were still reopened)
    • There were 2266 relevant edits during this period

During the experiment

  • Close efficacy. 55% of initial close votes or flags resulted in either the post being closed or a close review that completed with a verdict of “Leave Open”. A 53% improvement over the baseline.
    • 53% of initial close votes/flags resulted in closure - there are very few “Leave Open” reviews, but Triage also filters out some flags. Also, some votes reviewed as Do Not Close were still closed.
    • There were 32837 "first close votes" during this period
  • Re-open efficacy. 94% of initial reopen votes resulted in either the post being reopened or a reopen review that completed with a verdict of “Leave Closed”. A 6% improvement over the baseline.
    • 44% of initial reopen votes resulted in the question being reopened
    • 52% triggered a reopen review that completed with a verdict of “Leave Closed” (but some were still reopened)
    • There were 1694 "first reopen votes" during this period
  • Edit efficacy. 74% of initial edits after closure resulted in either the post being reopened or a reopen review that completed with a verdict of “Leave Closed”. A 14% improvement over the baseline.
    • 12% resulted in the question being reopened
    • 63% triggered a reopen review that completed with a verdict of “Leave Closed” (a few were still reopened)
    • There were 4132 relevant edits during this period

After the experiment

I’m also including a two-week period starting on the day the experiment ended, to illustrate that the changes seen during the experiment were not part of a longer trend. This is a weak comparison relative to the baseline because a significant number of votes are still pending as I write this - however, more than two weeks have passed and we should assume that most of these votes will age away unhandled.

  • Close efficacy. 36% of initial close votes or flags resulted in either the post being closed or a close review that completed with a verdict of “Leave Open”.
    • 34% of initial close votes/flags resulted in closure - there are very few “Leave Open” reviews, but Triage also filters out some flags. Also, some votes reviewed as Do Not Close were still closed.
    • 36% were still pending as I wrote this
    • There were 32692 "first close votes" during this period
  • Re-open efficacy. 91% of initial reopen votes resulted in either the post being reopened or a reopen review that completed with a verdict of “Leave Closed”.
    • 38% of initial reopen votes resulted in the question being reopened
    • 54% triggered a reopen review that completed with a verdict of “Leave Closed” (but some were still reopened)
    • 4% were still pending
    • There were 861 "first reopen votes" during this period
  • Edit efficacy. 64% of initial edits after closure resulted in either the post being reopened or a reopen review that completed with a verdict of “Leave Closed”.
    • 6% resulted in the question being reopened
    • 58% triggered a reopen review that completed with a verdict of “Leave Closed” (a few were still reopened)
    • There were 2531 relevant edits during this period

These are very similar to the observations taken prior to the experiment, so it's unlikely the experiment results are heavily skewed by some larger trend.

Here's a nifty little table that summarizes those results:


Hat-tip to Meg for the table idea.

Wow... Everything got more effective! Also, the total quantity of questions nominated for closure stayed about the same, even as efficacy - questions actually getting closed - went up; meanwhile, reopen-voting and editing increased dramatically, while also becoming more effective. Hypothesis: an awful lot of folks weren't voting to reopen or editing because they'd lost faith that the cycle actually worked - when it began to work, more started to make use of it.

You might notice that some of these numbers are still very low: less than half of the questions nominated for reopening get reopened, barely half of the questions nominated for closure even get reviewed. Even though this is a big improvement, we still have a long way to go before we can say the system is truly efficacious. We know - and if things pan out, we'll be talking a lot about this in the coming months! As I wrote earlier, this experiment is just a small part of much more extensive efforts to bring harmony and efficacy to our question lifecycle.

Effect on smaller tags

One of the hopes raised almost immediately when this experiment began (hey Cindy!) was that it would improve the ability of folks active in smaller tags to actually... do anything. We've known for years that this cycle was particularly broken when it came to niche topics, but nothing we've ever done there has been particularly effective.

I looked at questions in a set of tags that get >= 1000 questions/month on average, and calculated the efficacy of the same three actions for the same time periods listed above in the Prior and During sections:

Bar charts - everything got better (sorry, I know this is a really shitty alt text; if you'd like me to post textual results, I can do that)

Then I looked at the same metrics for all questions without tags that get >= 1000 questions/month:

More bar charts - everything got better even for small tags! (sorry again; I'm in a hurry, but that isn't a good excuse)

Yeeeaaaaah! We got a larger improvement on smaller tags! Finally, something that actually helps.

Conclusion

I think it's safe to say at this point that, per the metrics we established at the start of this experiment, this was a resounding success! The current system, with lower close/reopen vote thresholds, better enables individuals to trigger each critical action: close, reopen, edit. It's highly likely that perception was also affected: folks edited and voted to reopen many more questions during the experiment than either before or after, mirroring the increase in efficacy for both actions.

BUT: perception. Remember all those concerns? What else happened as a result of this change, and... Does the damage outweigh the good?

Side effects

Some effects are easy to guess at: making questions easier to close and reopen should mean that more questions get closed and reopened. Others are less sure: did this encourage more people to review? To review more? To get into fights over questions? These are all hopes or concerns that we considered beforehand or which were raised during the test, and I'll look at each of them below:

Abuse: close wars

A close war is when a question is closed, reopened, then closed again… Possibly going through several cycles before being either left open or deleted. This was a HUGE concern back in 2009 - indeed, it’s probably why the threshold was raised from 3 to 5 to begin with! Stack Overflow moderator Bhargav suggested that we watch for this almost instantly; a few other people also expressed similar concerns.

To make this easy, I’ll call anything a close war if a question is closed twice.

  • In the 30 days prior to the experiment, 100 questions were closed at least twice; 83 of those involved a gold tag-badge holder using their ability to instantly close a question as a duplicate.
  • During the 30-day experiment period, 188 questions were closed at least twice, with only 118 of them involving a gold tag-badge holder.

So, there were more close wars. But also, there aren’t very many close wars to begin with, even using this fairly generous definition of “close war”. In fact, there are few enough of these wars that we could probably just raise a mod flag any time one crops up (although we’d probably want to wait for at least two cycles before doing so).

Turns out, in 2009 we let folks vote to close the same question as many times as they wanted to - so the same three people could just camp on a question they wanted to close or reopen, fighting it out all day long. We’ve long ago limited users to one successful close or reopen vote per question, ever, and as a result, close wars are mostly a thing of the past, regardless of close/reopen threshold. Note that if your votes age away you can still recast as often as they age away, but that's not really abuse.

Confusion: questions closed without a consensus

This one was raised a couple of times on meta: with only 3 votes and 5 close-reasons, it’s much easier for a question to be closed without any reason having a clear majority of voters. Sure, this can happen with 5 votes as well, but it’s not common. I’m ignoring situations where one person casts a vote and then a moderator closes for a different reason, since moderators can do this regardless of the threshold.

  • In the 30 days prior to the experiment, 49 questions were closed without a consensus reason.
  • During the 30-day experiment period, 560 questions were closed without a consensus reason.

That’s a huge increase. And it’s even more problematic because of how we determine the displayed reason: we pick the oldest vote. So if someone picks a bogus reason, that gets discussed a bit and then two other people pick two better reasons… We show the bogus one.

One possible solution here is to require at least two voters to agree on a reason before closing. More on that idea later...

Review: Close queue backlog

For the rest of these sections, I’m just going to include charts - the trends are easy enough to see, and you're probably thoroughly bored by now. You may have noticed that I've deviated from my signature "ASCII tables and SEDE line charts" presentation style in this post: Meg Risdal was kind enough to go through and replace all of those with snazzy R-generated charts. They're ruining my low-fi street cred, but I do have to admit... They're a lot easier to read.

Number of tasks in the close review queue, sampled hourly over the past year. That huge dip in August is the experiment.

Review: Number of reviews

Number of reviews per week in the close and reopen queues. Huge spike in # of reviews being done in both queues!

Review: Number of active reviewers

Number of reviewers active per week in the close and reopen queues. An initial rise in the number of close reviewers, then back to normal, then below normal after the experiment ended. More interesting is the spike in reopen reviewers - a LOT more people were looking at these questions for the entire length of the experiment!

Overall: total questions closed

Total questions closed per week. Impressive rise, almost doubling the volume of closures by the second week, then falling off gradually approaching the end of the experiment along with the size of the backlog, the number of active reviewers, and the volume of reviews being performed.

Overall: total questions reopened

Total questions reopened per week. Even bigger proportional bump than closing, with less fall-off toward the end of the experiment - perhaps reflecting the larger number of additional reviewers. Drops like a rock as soon as the experiment ends.

Trends: efficacy over time, by close reason

This is a lot to chew on, so I’d like to bring these results to a… close 🥁… by returning to the concept of efficacy. There are five primary close reasons, and - as I’ve previously reported - there is a whole lot of variation in how they’re used. So it’s reasonable to assume that there is also a lot of variation in the efficacy votes and flags when considering the reason for closing chosen by the flagger or voter…

Close flag efficacy over time, by close reason

This shows the efficacy of the first close flag raised on a question, broken down by the reason chosen by the first flagger. I should mention at this point that a question can easily be counted multiple times in both these charts and the numbers above: if someone raises a flag and that flag ages away without being looked at, that’s an ineffective flag; if someone later flags the question again and this time it gets closed (or at least reviewed), that’s an effective flag. This becomes more interesting when breaking down the results, as reviewers may choose to filter the queue by close reason: e.g. if I’m only concerned with duplicates, I won’t bother reviewing questions that are flagged as off-topic.

We can observe a few new things here:

  • That fuzziness around the edges of the experiment is clearly evident: flags raised before the start which were still pending became more effective due to the start of the experiment, while some of those raised near the end became less effective.
  • Close flags are very ineffective. Remember, “effectiveness” here doesn’t even mean “the question gets closed” - it just means the question got reviewed.
  • Duplicate flags were the most effective before the experiment but saw the smallest bump. The reason for this will soon become apparent…

Close vote efficacy over time, by close reason

As above, but now considering the reason chosen for the first vote in a given sequence of votes on a question and ignoring any flags that might’ve been active up to that point. If it’s starting to sound like you might not be able to add these values to the ones for flags above and arrive at anything that matches the numbers above… Well, that’s why I’m doing colorful charts. Look at 'em! Ain’t they pretty… Almost makes you forget how horribly complicated all this closing stuff is, don’t it?

So, there we go: duplicate votes were already extremely effective - both gold-badge holders and askers could bypass the 5-vote requirement to speed up closing of duplicates (and validate any flags). So while duplicates saw a slight increase in effectiveness, it was nothing close to the bump that all the other reasons saw. Duplicate votes are already very effective - further gains in effectiveness for them must probably be found elsewhere.

In closing: a proposal for next actions

To be honest... Next steps seem like a no-brainer here: for a very simple change, we get a huge increase in efficacy across the board. Everything just works better. It's the perfect drug!

Oh, right... That no-consensus thing went up by a thousand percent. We should, uh, probably fix that.

Ok, here's my proposal:

  1. We implement a second, "consensus" threshold: n close votes have to agree before a question gets closed.
  2. We add a bit of text to the close dialog (and hover text on the "close" link under questions) that explains how many votes are still needed to close the question, with a special "n agreeing votes are needed" output when the close threshold has been hit but the consensus threshold hasn't.
  3. We lower the close threshold to 3, with the consensus threshold set to 2.

I think this both buys us the advantages and removes the potential harm that was seen during the experiment. If no one can think of a good reason not to do this, I'd like to roll out the proposed changes as soon as we're done testing / iterating on the new post notice system that's currently under development.

This is something that we very much shouldn't be changing during testing of the post notice system, because it can impact our ability to analyze the results of that test: it will skew them in a bunch of different ways, and potentially cause us to overlook problems. Waiting on that test (and any work that arises from it) gives us probably a month or two to chew on and discuss this, as well as build and test the changes necessary to support a consensus threshold.

Thanks again to everyone who participated in this, both in discussion and especially in the actual work of closing, editing, and reopening questions on Stack Overflow. We must never forget that this machine is built on people and that they deserve respect and appreciation. And special thanks today to Meg Risdal and Cesar M, who contributed extensively to the creation and editing of this report - if you think this is rambling and confusing now, you should've seen my first draft!

Excited to hear your thoughts

56
  • 39
    I think your proposal might be in danger of merely causing a change in close vote behavior where folks simply vote for the same reason as the first close vote in order to close it more expediently. I'm not sure if it's feasible but I wonder if it's worth considering hiding the prior votes...? (I guess you'd probably keep duplicate votes and migrations, but hide vote counts for the other reasons?)
    – joran
    Commented Oct 3, 2019 at 18:44
  • 15
    That's exactly the kind of edge case we want to be thoughtful of -- thanks @joran! :)
    – Meg Risdal StaffMod
    Commented Oct 3, 2019 at 18:49
  • 13
    "Re-open efficacy. 94% [...]. A 6% improvement over the baseline [89%]." Another way of looking at those numbers is that you're 50% less likely to not have your post reviewed for reopening (6% vs 11%). That's huge! Commented Oct 3, 2019 at 18:55
  • 7
    @Shog9 I'd love to read this, but it's enormous and with all the other things to read I don't think I'll have time for a bit. Could you give a one- or two-sentence summary? Maybe edit it into the post?
    – Kevin
    Commented Oct 3, 2019 at 18:57
  • 7
    Huh! It was so interesting I didn't even notice how long it is - until I started scrollng up on my tablet to see the upvotes :-) Thank you (and Meg and Cesar) for the time and effort and the great share! (And, yes, I saw my name <oops>) Commented Oct 3, 2019 at 19:00
  • 22
    so what happened to "more power to silver & gold badge owners" ? Commented Oct 3, 2019 at 19:31
  • 8
    It was my pleasure! I had a lot of fun collaborating with @Shog9 on this one. And all of those data points come from folks like yourself. So thank you, too. <3 Really looking forward to the thoughts and feedback we hear from those who take the time to read (it is a lot to dig through!).
    – Meg Risdal StaffMod
    Commented Oct 3, 2019 at 20:36
  • 24
    Requiring vote consensus is a problem--close reasons overlap & there may be more than one reason to close. If it should be closed, every vote should count. Just report the reasons & their vote counts. Moreover if there is a vote consensus requirement it will reduce feedback because people will distort their vote to match others despite the collection of reasons being less informative just to get the question closed.
    – philipxy
    Commented Oct 3, 2019 at 22:36
  • 10
    Nit: Please don't call a change from 36% to 55% a "53% increase", call it a "19 percentage point increase". Talking about percentages of percentages is weird, keeping it in percentage points normalizes things. Like, the change from 36-55 is the same magnitude change as the change from 16-35 (same denominator, numerator moved up by the same amount), but the latter would now be OVER 100% :-(
    – Barry
    Commented Oct 3, 2019 at 22:46
  • 20
    Not one crudely hand drawn red circle? I call shenanigans.
    – Nkosi
    Commented Oct 4, 2019 at 0:34
  • 3
    But seriously, this was very informative. Especially as it has been of great interest to me how the experiment went.
    – Nkosi
    Commented Oct 4, 2019 at 0:35
  • 8
    This is amazing. Thanks for this! In light of the storm that's happening now, this brings me hope that at least some part of SE still cares about us.
    – Mysticial
    Commented Oct 4, 2019 at 8:14
  • 4
    So, that's why I didn't put # or % closed or reopened as key metrics here anywhere @TheGuywithTheHat. The "correctness" of a given action is subjective: the best measurement we have for the correctness of close votes is reopening, but since neither closing nor reopening is 100% effective, that is a very noisy measurement. Hence, the focus on efficacy.
    – Shog9
    Commented Oct 4, 2019 at 18:07
  • 5
    @corsiKa ... yes.
    – Meg Risdal StaffMod
    Commented Oct 7, 2019 at 23:44
  • 5
    So when do we put this in permanently? And when do we revisit it on Software Engineering? Commented Oct 14, 2019 at 2:59

26 Answers 26

116

The only real concern I can think of is that if we tell people that there's a consensus that needs to be reached, people will act towards that consensus even if it's not the right consensus.

For instance, if we have a question that's simply clear as mud to the average passer-by but it's closed because it's lacking a code example, and the consensus message displayed is that it needs one more vote in that field, then there's a very strong chance that close voters will re-tune themselves to just...close with the crowd, which isn't really what we want.

17
  • 7
    The easy fix for this is only trigger for the third voter, when the previous 2 votes aren't the same type.
    – Braiam
    Commented Oct 3, 2019 at 18:53
  • 2
    @Braiam wouldn't that mean that two cvs would get it closed then? I thought Shog9's proposal was at least three people agreeing it should be closed, and minimum two of those agreeing on why it should be closed.
    – CalvT
    Commented Oct 3, 2019 at 19:09
  • 1
    Perhaps tune the "see close votes" rep privilege to only see total close vote count, and then maybe add a privilege to see the exact close reasons at 25k or something? With the exception of duplicate close votes, where you need to see an already-suggested duplicate, of course. Though I don't like the consensus idea.
    – TylerH
    Commented Oct 3, 2019 at 19:25
  • 3
    @CalvT no; Person 1 picks reason A, Person 2 picks reason B without being shown 1's choice, and the third person gets to see both A and B (if they differ). So you still need 3 total votes minimum, and at least 2 aligned on the same reason. It's just a matter of showing person 3 what the reasons of the former two were, but not showing person 2 what the first one voted (this differs from the current situation)
    – Adriaan
    Commented Oct 3, 2019 at 19:44
  • @TylerH I'd rather put the exact-reasons privilege in with the 10k in that case. From 3k to 10k is already a large step, let alone from 3k to 25k.
    – Adriaan
    Commented Oct 3, 2019 at 19:45
  • 2
    @Adriaan Eh. As I commented under Nathan's argument - I'd rather see the exact reasons return in the publicly viewable close reasons (though I also like the opposite of what was announced a while back in a blog post. That is, hide the close voters from people under 3k (or some amount) of rep)
    – TylerH
    Commented Oct 3, 2019 at 20:48
  • 6
    In this entire discussion about the final close reason: Why not also take into account any flags that have been raised with close reasons? Commented Oct 4, 2019 at 6:36
  • 2
    I'd think about whether the fact that a question gets so many different reasons might itself be an indicator of it being a very bad question. Maybe the problem is only displaying one reason.
    – Elin
    Commented Oct 5, 2019 at 23:04
  • 1
    Well your first paragraph defines democracy.
    – Geeky Guy
    Commented Oct 6, 2019 at 2:10
  • @TylerH "That is, hide the close voters from people under 3k" those aren't votes, you get close voting privileges at 3k, but flags. And those aren't shown.
    – Braiam
    Commented Oct 7, 2019 at 1:43
  • @Braiam No, I mean the banner that shows who voted to close - hide those people from people who have less than 3k reputation so that they see the close reason but not the close voters (including the OP).
    – TylerH
    Commented Oct 7, 2019 at 1:53
  • @CindyMeister There's usually 0 - 1 flags on closed questions, so what are you thinking is unaccounted for?
    – TylerH
    Commented Oct 7, 2019 at 1:53
  • 3
    @TylerH I'm not privy to all the background statistics. What brought the idea to mind is what I see in the CV-Review: Often one or more reasons are listed at the top, but are not visible as votes in the CV Dialog. My assumption is that these must be flags. So my thought was, if there's no clear CV reason from the dialog flag votes could be used to "break the tie". Just a thought, no emotional investment in the idea :-) Commented Oct 7, 2019 at 5:38
  • @CindyMeister Gotcha; I thought you were suggesting using the flags completely in lieu of CVs.
    – TylerH
    Commented Oct 7, 2019 at 15:09
  • 1
    I'm not 100% sold on needing a single reason, though. I know it keeps it simple for answering "why did my question get closed" to a user, but there's plenty of questions which provide a buffet of valid close reasons, and I still have to choose just one.
    – Gus
    Commented Oct 9, 2019 at 11:55
92

The need for consensus is the result of a bad design.

I post a question. It gets closed as unclear. I make it clear. It gets closed as too broad. I narrow it. It gets closed as off topic.

I'd have loved to have known about all three of these problems the first time it got closed!

Why the OP doesn't see all the problems voted on is beyond me. This is a teaching moment. Cleaning up the site is nice today. Teaching the OP what we need from them makes for a better tomorrow.

I love the idea lowering the threshold. But the thing driving this consensus concern isn't because we have a good reason to need a consensus on why to close. We have a bad reason driven by a bad design that just makes the close more confusing. Most new user questions have more than one problem. Why pretend it's just one?

17
  • 11
    Unfortunately SO is portrayed as the devil largely because of misunderstanding moderation. The way things have been going in the company there is absolutely no way they'd show more negative feedback to askers, especially since one or two votes are often wrong. Showing three pending close votes would be unwelcoming. Commented Oct 5, 2019 at 10:50
  • 33
    @AndrasDeak It's unwelcoming to be told you're question isn't good enough to deserve an answer and not be told all the reasons why so you can learn and fix them. It creates a bias against new users who are still trying to come up to speed. It makes us look hostile when a good faith effort to fix the issue just leads to other "excuses" to close the question. I remember what it was like to be new here. Learning was not fun. It was needlessly painful. But it is necessary. I don't want the bar lowered. I want the path over it to be obvious and clear. Commented Oct 5, 2019 at 11:17
  • 1
    It'd be nice if they'd figure out how to measure returning OPs who ask a new and better question after people take the time to explain the problems so we'd get positive feedback for doing more than just closing. Commented Oct 5, 2019 at 11:25
  • 9
    Unfortunately my impression has been since the beginning of the welcome wagon that the primary target of optimization is appearances. Imagine an angry twitter influencer sharing a screenshot of an asker "who didn't do anything wrong" and is bombarded by "no less than three messages" that put them down for not being up to the "standards of the elitist majority". See where I'm coming from? Commented Oct 5, 2019 at 11:30
  • 5
    @AndrasDeak three messages all at once looks better that three messages after multiple good faith attempts. I've been on the receiving end of bitter complaints about a questionable close more than once. It's better to know what really happened than to imagine. Would you appreciate getting pulled over and given a huge fine and not being told all the reasons up front? Sometimes the cops are wrong. Sometimes you can show a charge to be wrong. It's exhausting if you have to learn to fix them one at a time. It's costing us new askers. Commented Oct 5, 2019 at 11:47
  • 1
    I didn't say your suggestion was wrong. It's not, and it would help. But considering appearances, which often seems to be the motivating driving force behind policy these days, it's a show-stopper. Commented Oct 5, 2019 at 11:50
  • 2
    @AndrasDeak I don't see why. Appearances aren't made better when you let people imagine the worst. Is anyone dealing with police wishing they'd make the charges harder to understand? Is anyone dealing with politicians wishing they'd keep their activities secret? It's called transparency. Sunlight is a great disinfectant. Hell I vote to close and wish more people could see what I'm doing so I'd know if I'm getting it wrong. To the point that I sometimes copy and paste my close reasons in a comment. It makes cleaning the front page a royal pain but it's the best I can do with the tools I have. Commented Oct 5, 2019 at 12:01
  • 9
    This is a really reasonable point. @AndrasDeak it's not an easy problem for us to solve and there are a lot of constraints/complexities, but I'm open to hearing this feedback and considering approaches like the one candied_orange proposes. It's on us to think about and test the best way to deliver this feedback and I welcome suggestions from people who understand the system best. :) Anyway, just jumping in as the PM for public Q&A to say I'd be happy to see this conversation continue.
    – Meg Risdal StaffMod
    Commented Oct 7, 2019 at 5:52
  • 2
    @MeganRisdal I could post my own feature request if it would help. Won't stop this feature from being impacted by it though. Something to consider is what I call the gnat effect. He is a highly visible close voter because he almost always posts a comment or two with links to meta posts that describe his issues with the question. Because of this he sometimes takes a little crap when he's wrong. He's also the one who taught me the most about closing and posting because I could tell what he was thinking. Wish more voted this way. Commented Oct 7, 2019 at 11:46
  • Isn't this exactly the same as my answer?
    – Luuklag
    Commented Oct 7, 2019 at 12:12
  • 1
    @Luuklag after reading your answer I think we're beating the same drum. I'm trying to show what the current design is costing us. Commented Oct 7, 2019 at 12:25
  • 1
    Here's a direct link to gnats comments. The guy's amazing. Commented Oct 7, 2019 at 12:35
  • @candied_orange if you want to highlight a specific comment, use the comment link in the timestamp of the comment. The link you provided is dynamic content, and isn't expected to always show the same thing for different people.
    – Braiam
    Commented Oct 7, 2019 at 17:15
  • 1
    @Braiam I think there is no problem in this case, as the comments are plenty obvious which are meant to be examples. Only thing is Gnat could turn into a less friendlier version today and this list will be less helpfull tomorrow.
    – Luuklag
    Commented Oct 7, 2019 at 20:25
  • 1
    If we're going to display multiple closure reasons (as we should), then we should also allow to vote to close with multiple reasons. I think that will just help the poster make all improvements necessary for reopening, rather than giving one reason, poster fixing that one aspect, and seeing their question not being reopened without further reasons or help. Commented Oct 11, 2019 at 22:14
70

I wonder if having two close reasons served to the OP could help the consensus issue. A decent number of the question I close can be closed for a couple different close reasons and if we show them two of the three votes then it should help clarify what some of the issues the post has.

My main use case is the proverbial wall-o-text question. Lacks MRE, Too Broad, Unclear can all be valid reasons for that. Instead of requiring more votes to close that because people are using all three, showing any two of those reasons to the OP should be able to point them in a direction to work on getting the question opened.

14
  • 13
    Yes, I think we should simply show multiple close vote reasons again like we used to, rather than needing consensus to avoid the problem. It's kind of like sweeping items under a rug, identifying that that's a problem, and then "fixing" it by simply saying "we no longer sweep items under a rug; instead if multiple people agree to sweep it under the bed or dresser then we'll sweep it under that", and then saying "See, no more problem of sweeping dirt under the rug and saying the floor is clean!"
    – TylerH
    Commented Oct 3, 2019 at 19:28
  • 1
    Fat fingers will always be fat fingers, but it is better to present the OP with more information about what to improve than less IMO.
    – Adriaan
    Commented Oct 3, 2019 at 19:48
  • 13
    I agree. The problem isn't that there's no consensus as to why the post should be closed, the issue is that it's not communicated to the question OP all of the ways which the question needs improvement. For the diligent users who actually try to fix their question, the problem is that they will (try to) fix the thing that the close banner says is the issue, but won't necessarily address any of the other reasons that the question should be closed, because there's nothing telling them what else needs to be fixed.
    – Makyen Mod
    Commented Oct 3, 2019 at 20:01
  • 11
    Well... Part of the problem is that we have three close reasons that are all kinda interchangeable on an awful lot of questions.
    – Shog9
    Commented Oct 4, 2019 at 0:08
  • @Shog9 Do you think this would be reasonable if the close reasons didn't overlap (or not as much)? I feel like it must be simpler to do than having to write the code to handle all the edge case suggestions people are putting up.
    – Krupip
    Commented Oct 4, 2019 at 0:28
  • 10
    Here's the thing, @opa: years ago, there was a close reason that was called "Not a real question". It was great for stuff like this - a big catch-all for all sorts of "what the hell am I even looking at?" questions. Of course... It was a pretty crap experience for askers - the name was pretty insulting, and there wasn't good guidance. All these other reasons came out of analyzing questions closed for that reason (some by a rather circuitous path, but I digress) Seems to me that a nicer form of that would probably suffice here.
    – Shog9
    Commented Oct 4, 2019 at 0:30
  • @Shog9 I wasn't suggesting a combined reason, but rather reasons that were more separate, I just had no idea how that was going to be accomplished, I'll provide a separate answer outlining possible ways of distinguishing the 3 ambiguous close vote reasons.
    – Krupip
    Commented Oct 4, 2019 at 0:35
  • @Shog9 I replied before your edit, I'm confused by what is said here Seems to me that a nicer form of that would probably suffice here a nicer form of what?
    – Krupip
    Commented Oct 4, 2019 at 0:37
  • A nicer form of the original "Not a real question", @opa. Sorry for the confusion; I'm a bit tired. Looking forward to reading your suggestions!
    – Shog9
    Commented Oct 4, 2019 at 1:31
  • 3
    @Shog9 I'm not sure how good my suggestions are, but I finally finished my answer.
    – Krupip
    Commented Oct 4, 2019 at 3:45
  • 14
    If we're going to diaplay multiple close reasons, maybe we should make the individual reasons collapsed by default in the close notice. So they would see something like "your post was closed as unclear (2 votes), too broad (1 vote)" where clicking on either reason would diaplay the more detailed description of the close reason. Might help prevent multiple close reasons from making the close notice a wall of text. Commented Oct 4, 2019 at 14:37
  • @DavisBroda I really like that. Commented Oct 4, 2019 at 14:48
  • 1
    Yes, this. Especially if we're expecting OP to edit based on the stated close reason, the "shadow reasons" will still be there, but OP won't have been warned they need to get fixed too
    – Gus
    Commented Oct 9, 2019 at 12:00
  • Yes, @Shog9, overlapping close reasons is indeed a large part of the problem with the consensus aspect of the proposal. But instead of insisting on a consensus (sort of), how about just presenting all the selected close reasons rather than choosing one to distinguish? Wouldn't that give the OP more information to work with in judging how to improve the question? Wouldn't that be a good thing? And if three different people have three different reasons for closure, how is it sensible to interpret that as a sign that maybe the question shouldn't be closed after all? Commented Oct 16, 2019 at 1:42
60

I'd like to propose an alternative to the consensus system:

Instead of the current practice of using the oldest vote as a tiebreaker, use the tag scores of the closevoters, i.e., if no consensus is reached through the number of votes alone, the reason used by the closevoter with the highest tag score is chosen.

One thing we'd need to figure out is what to do when the question has more than one tag. My first idea was using the tag with the most questions (which will usually be the language tag), but Shog9 raised some valid concerns about this approach.

Suggestions for an alternative metric are welcome, but I'll also keep thinking about it.

26
  • 1
    Just to clarify, do you mean "use the vote of the user with the highest score on <most relevant tag> as a tie breaker"?
    – VLAZ
    Commented Oct 3, 2019 at 20:30
  • @VLAZ Exactly. Feel free to edit, this might be a language problem on my part.
    – Baum mit Augen Mod
    Commented Oct 3, 2019 at 20:32
  • @BaummitAugen How would one quantify "most relevant" automatically?
    – Krupip
    Commented Oct 3, 2019 at 20:34
  • 1
    @opa My first suggestion would be the tag with the most questions, which I think should usually be the language tag.
    – Baum mit Augen Mod
    Commented Oct 3, 2019 at 20:37
  • 4
    Is there any technical reason why only one close reason can be displayed? If there is a tie, why not just display all of the reasons (each attributed to the corresponding users)?
    – ahiijny
    Commented Oct 3, 2019 at 20:57
  • 2
    @ahiijny This is pretty much Nathan's proposal here, which certainly is a valid solution too if SE can make it work from a UX perspective.
    – Baum mit Augen Mod
    Commented Oct 3, 2019 at 20:59
  • 33
    So, the problem with this is that a question tagged [javascript] [reactjs] [redux] [ecmascript-6] that gets an "unclear" vote from someone with a high [javascript] score, a "too broad" vote from someone with a high [c++] score, and a "duplicate" vote from someone with a high [ecmascript-6] score... Gets closed as "unclear", even if the only person with relevant expertise is the last voter. Which is likely, given they had the highest score in the most specific tag among voters.
    – Shog9
    Commented Oct 3, 2019 at 23:45
  • 2
    @Shog9 Hm yes, good point. So the key question is whether coming up with a better metric for "most relevant tag" is easier or harder than making the consensus alternative work. I don't have a better idea off the top of my head, but I'll keep thinking about it.
    – Baum mit Augen Mod
    Commented Oct 4, 2019 at 13:03
  • Perhaps kind of standardized weight of tag scores for all tags that applied to the question? (With caveat like dupe's hammer: editing tags will forfeit the weight) Using Shog9's example, [ecmascript-6] expertise would probably also have some tag scores on [javascript], so they have more weight for a single vote? Though... another problem is the mistagged question...
    – Andrew T.
    Commented Oct 4, 2019 at 15:16
  • 1
    @shog9 the 6 [ecmascript-6] gold badgers all have a [js] gold badge too ... Sure there might be some other edgecases were that would apply .... but voting is supposed to work for the general case, right? Commented Oct 4, 2019 at 18:38
  • @andrew everyone that could act as a "tiebreaker" has probably enough reputation to have the edit privilege ... Commented Oct 4, 2019 at 18:40
  • 2
    @JonasWilms what I meant is for dupehammer, the system disables your dupehammer power if you edit the question to add a tag which you have a gold badge for it. So for this case, the weight is only counted on the current tags, but if you edit the tag, then you forfeit the weight for it.
    – Andrew T.
    Commented Oct 4, 2019 at 19:14
  • @andrew interesting, I actually didn't know that. Seems that I've never had that case ... Commented Oct 4, 2019 at 19:18
  • 1
    Well, one thing you can at least do is look for "irrelevant" tags. Something tagged [C++] [for-loop], a [C++] gold tag holder is like more likely to know what's up than a [for-loop] gold tag holder because while for loops are common to almost every language, having a knowledge of C++ is more likely to serve you well for a question like than having knowledge of for-loops, in case, for instance, the problem has nothing to do with for-loops and is actually C++ specific.
    – user10957435
    Commented Oct 7, 2019 at 4:05
  • 1
    @PatrickRoberts The point is that there are a lot of tags with both a description but very few questions. While each single such tag is used rarely, the number of questions with at least one "niche", but not very meaningful, tag on them is probably not small.
    – Baum mit Augen Mod
    Commented Oct 13, 2019 at 21:45
45

Instead of having to reach consensus can't you (not you personally, but the company), while you are working on post notices, make it that we can display up-to three close reasons:

A bit like this:

put on hold by UserX, userY, UserZ

The users who voted to close gave these reasons:

  • Off-topic; recommend a book
  • Too-broad
  • Off-topic; No MCVE

A user could then simply deduce that his question, that asked how his code should be improved to accomplish task X, was not asking for recommendations.

Too-broad and no MCVE are then likely to be the "correct" close reasons seeing that he posted a giant wall of code, being far from minimal.

2
  • Indeed, one of the biggest problems I think is questions that are asking "what's the best library for X" or "what product do you recommend for Y". These can be closed as off-topic - resource request, or as POB. If one gets listed but the other doesn't when both are used (I usually use the one that hasn't been used yet to cover the bases), OP may think one is OK and edit their question saying "see, it's not POB now", or "see, it's not a resource request now" while still falling under the other close reason.
    – TylerH
    Commented Oct 4, 2019 at 15:17
  • 3
    FWIW, all 3 close-reasons can be valid, just that some are more problematic or instead tangential than the other. I can imagine a wall of code with the 1st question asking for fixing the issue and 2nd question asking for what books to read for avoiding this kind of issue... (which is basically Too Broad for asking multiple questions, with 1st question offending the MCVE and 2nd question offending the recommendation request). The 2nd problem is usually easy to fix, but the 1st problem is the bigger problem...
    – Andrew T.
    Commented Oct 4, 2019 at 15:36
32

Wow, that's a very nice analysis (true a bit long), but great work Shog9. Let me contribute some to side-effects related to low traffic tags that you already touched some.

I'm a contributor to a low-traffic tag, under my Oak tree, where I moderate and answer incoming questions.

Of course there are both incoming questions, but also community bumped questions that can not be answered because of one of our close reasons.

Which tools do I have to deal with this?

  1. Close vote. Well, as we know, that does not really work in low-traffic tags if not dupe and hammer or I ask for help from SOCVR. This tool at 5 close votes is useless, and my vote will most probably just age away.

  2. Downvote. Yes, this works great. I don't comment, but I downvote to get the -1 and just keep my patience waiting for the roomba.

However

I love the people asking, and I would love if the OP could improve/clarify the question so I can answer it. It's no fun for me downvoting and leaving the OP clueless about what they did wrong.

During the test, and as you have shown, 3 close votes made it possible for me to change strategy and to finally do and have the right thing happen. Close vote, have post closed, and have the "diligent users" edit the question, so it gets answerable and useful for future users.

All this to say that my idea have always been that the OP feels more demotivated about the downvote (I never seen them edit the -1 question with no comment), than the close banner.

Close voting is more constructive and if Stack Exchange likes this to be effective in low-traffic tags you need to give us the possibility to use it reducing close vote needed to 3 or implementing a system with weighted close votes, for example, Gold badge counts 3 and Silver 2.

1
  • 7
    Really glad to hear how this worked out for you, Petter! Hope we can do even better in the future.
    – Shog9
    Commented Oct 4, 2019 at 0:29
25

One thing I noticed during the experiment was that there were more closed question with reopen votes on them. When I came across them (mostly due to their edit activity), I found many of them in a still horrible shape and wanted to counter the displayed reopen votes.

Unfortunately that is not possible on the question UI itself, only the review queue gives you the option to Leave Closed. On the question itself, you can only click Reopen (n) and (after a confirm prompt) you have cast the reopen vote. Please improve this experience:

  • by having the click open a modal dialog, where you can choose from all the review options - similar to how the close vote dialogue does it.
  • Or by having the button simply be a link that leads to the review queue entry.
5
  • You can narrow down the tags on the question with a filter and then click "Leave Closed".
    – S.S. Anne
    Commented Oct 3, 2019 at 23:06
  • 2
    This is already possible, but only to a very limited extent, by scraping review history to find the associated review task link. A less cumbersome method would be great. Commented Oct 3, 2019 at 23:11
  • 2
    @JL2210 Yes, I could go to the review queue and try to find the question there, but that's not very ergonomic - so I disengaged instead
    – Bergi
    Commented Oct 3, 2019 at 23:12
  • 10
    This exact thing in reverse has been requested for years: a "leave open" button for questions with close votes on them. Thing is: it's entirely possible for a question to already have been reviewed, as leave open / leave closed, and still have pending votes: the votes will take a day or more to age away after review has completed. This might all just be a little bit over-complicated.
    – Shog9
    Commented Oct 4, 2019 at 0:26
  • 1
    This was once possible, using a userscript, see meta.stackoverflow.com/questions/319682/…, @Shog9 has an answer also there Commented Oct 5, 2019 at 13:44
25

About consensus: why can't people vote for several reasons? Instead of being your preferred reason, it becomes a vote for "all reasons that apply", like the moderator elections, just without the STV aspect. Then use the reason most used. It's simpler and prevents the voter from having to manipulate the system in a way that their preferred "close reason" is forced.

The close votes becomes a "I believe this isn't an answerable question" and the reasons becomes a basic selection. This also solves the problem where there are multiple reasons that apply and stops close voters from bickering about what is the "most appropriated close reason".

6
  • 1
    This sounds horribly complex...
    – Shog9
    Commented Oct 5, 2019 at 1:43
  • 6
    While I agree that allowing people to vote for all reasons which apply could be helpful (or maybe just their top 3 reasons), I disagree that we should boil those down to a single close reason displayed on the question/to the OP. The intent of displaying close reasons is to give the OP an understanding of what is wrong with the question, so that they can edit it to make it on-topic (or just let them and others know why it's not a fit for SO). If people closing the question feel it has multiple problems, then the OP should be informed of all of those problems, not just the top one.
    – Makyen Mod
    Commented Oct 5, 2019 at 2:14
  • 2
    Well @Shog9, I'm just saying that that's a easy solution which normal people can understand.
    – Braiam
    Commented Oct 5, 2019 at 3:06
  • What is "STV"? Commented Oct 6, 2019 at 3:52
  • 1
    @PeterMortensen Single Transferable Vote.
    – Braiam
    Commented Oct 6, 2019 at 18:23
  • I agree with @Makyen. We should vote for all reasons that apply and the user should be shown all of the reasons along with how many votes each reason got. Commented May 22, 2020 at 22:45
23

That no-consensus thing went up by a thousand percent. We should, uh, probably fix that.

Or should we?

"Close" verdict with no concensus means that N people still agree that it should be closed, just don't agree why exactly.

So close it without further ado! Just show all the reasons specified under appropriate wording. E.g.: "The reasons specified were:"

It's not illogical or unexpected that something can have multiple problems thus there are multiple reasons why it's inappropriate. So that won't come as something confusing if the framing wording clearly hints that these are the reasons those people picked (so what the OP is expected to do is some amalgamation of them).

7
  • 1
    I think the requirement to have n consensus votes is a brilliant compromise. The usability problem for the OP when there is no clear indication why their question was closed is tremendous.
    – tripleee
    Commented Oct 7, 2019 at 12:06
  • Isnt this answer the same as the ones by me and buly candied_orange?
    – Luuklag
    Commented Oct 7, 2019 at 12:36
  • 2
    @tripleee the two last paragraphs rebut the clain that there would be a usability problem. Besides, Shog's graphs show that even with the lower threshold, the review process still can't handle everything and every extra vote costs about 15-20% of handled reviews -- so any "compromise" here would be a major setback. Commented Oct 7, 2019 at 13:38
  • @Luuklag Neither of those gave the critical part, the justification. Or suggested a different framing to eliminate any possible resulting usability problem. Commented Oct 7, 2019 at 13:43
  • @ivan_pozdeev speaking of posts that appear to be leaking through close review, I consider making a feature request to have a moderator review for these (similar to how it happens in LQ queue). I preliminary drafted it as a part of this answer "raise a system flag for mod attention if a question... gets stuck in review queue for longer than a day or two, to let them finish it with a binding vote..."
    – gnat
    Commented Oct 8, 2019 at 18:37
  • @gnat I foresee mods meeting it at dagger point saying that it's not their job and they have enough load already. IMO this idea shifts blame to them from system deficiencies so I'd agree. Suggestions refereced at meta.stackoverflow.com/questions/385023/… look like a better way to go. Commented Oct 9, 2019 at 18:38
  • 1
    If we're going to display multiple closure reasons (as we should), then we should also allow to vote to close with multiple reasons. I think that will just help the poster make all improvements necessary for reopening, rather than giving one reason, poster fixing that one aspect, and seeing their question not being reopened without further reasons or help. Commented Oct 11, 2019 at 22:13
19

Please fix this bug in the closing process:

...and implement this improvement (which would probably fix the bug above):

...at the same time as these changes.

6
  • 1
    Very valid point; it would have been a great feature as current mechanism suggest all voters had same reason for close vote. This is actually not fair to voters who get their name on the record for reasons they didn't. Commented Oct 7, 2019 at 8:24
  • 1
    It's pretty incredible that this was reported in 2015 and three years later it still exists.
    – kockburn
    Commented Oct 7, 2019 at 8:46
  • I would say it's incorrect to call this a "bug" because it's rather a deliberate design choice. It presents an (false) image of unity and firmness of the decision to the OP which is probably supposed to intimidate them into following it rather than look for excuses and waste everyone's time. Commented Oct 8, 2019 at 20:56
  • 1
    @ivan_pozdeev - Then it's a design bug. Either way, lying about what someone's done is a bug. :-) Commented Oct 9, 2019 at 6:04
  • "Bug" is by definition an unintended effect. They likely think it's a "white lie" (whether such a thing exists is a matter or argument and is usually justified by claiming that the receiving party is incapable of handling the truth). Commented Oct 9, 2019 at 16:46
  • @ivan_pozdeev - No, a bug is something doing something unintended or incorrect. In any case, this isn't a discussion I'm interested in having. Take "bug" above for emphasis if you feel the need to dispute the literal meaning. Commented Oct 9, 2019 at 16:58
14

Close efficacy. How likely is the first close vote or flag on a question to result in it being closed?

How many of these votes are at the same time the last close vote, i.e. when a gold-badge holder or moderator closed the question unilaterally? Can you subtract those from the numbers, please? I'd like to see another nifty little table.

Some quick stats: 36% of dup-closures are closed unilaterally by gold badge-holders; 49% of dup-closures involve normal close-voters assisted by gold badge-holders (and so are closed with between 2 and 5 votes).

Additionally, 9% of duplicates are closed when the author of the question itself confirms the question as a duplicate after it has been voted or flagged.

Scroll up and look at the chart under the heading "Close vote efficacy over time, by close reason". Now have a look at this chart, generated from the same data but excluding the closures involving badge-holders and askers:

a chart showing the efficacy of close votes by week and reason: duplicates are the least effective by far, at under 4% except during the experiment when the efficacy rose briefly to 15%

-- Shog9

2
  • 4
    @Shog9 Thanks! Now how can I upvote your answer?
    – Bergi
    Commented Oct 4, 2019 at 0:27
  • 5
    Upvoted despite the SEDE chart (jk jk 😉).
    – Meg Risdal StaffMod
    Commented Oct 4, 2019 at 0:39
14

There are many things that I believe should be addressed in question closure

Tag score should matter

I believe the tag score of the caster should affect the weight of both close and reopen votes.

Why does glibc's strlen need to be so complicated to run quickly? was one of these close wars. It was closed first as "too broad". I cast a reopen vote to be able to answer the question.

It then did get to HNQ and was subject to debate even outside the network. Subsequently it was closed with "primarily opinion based". It did also undergo drastic edits by participants so that the question now barely resembles the original one, and therefore my own answer in itself barely answers that which is being asked in the current form of the question.

Even though going through drastic edits, as a gold holder I consider it any of those revisions a very good on-topic C question. And so did many other gold badge holders, including one moderator. Now seeing who did cast close votes, many of them seem to be only drive by answerers in the compared to the many gold badge users that frequent the tag and are seen casting close votes on the umpteeth typo. To avoid close wars, perhaps a gold badge holder could be allowed to cast a "keep open" vote that would cause the remaining vote count to be increased by one, i.e. something like

  • a normal close quota is 5 votes
  • a gold badge holder can give 2 close votes
  • a gold badge holder can give 2 reopen votes
  • a gold badge holder can cast a "keep open vote" that would increase quota by one, and then not be eligible to subsequently cast a reopen vote

Notify the OP immediately

The OP should be notified immediately of receiving close votes and informing them of the suggested reason for closure. The reason why majority of posts are closed in the first place and remain closed is due to the OP's negligence in reading the help center or understanding the site rules. Especially sometimes when browsing casually on mobile phone - and while it is easy to cast a close vote, it is a tad bit harder to write a long explanation on what's wrong with the post using a mobile phone and adding the relevant help center links...

About consensus

Why require consensus! Just list all the close reasons given. Then in reviews all close reasons should be considered. If one of them still holds then and only then the question should be reopened. The worst questions are of the kind that do receive edits and then require another close reason. For example the first revision of a question might read:

error: expected '=', ',', ';', 'asm' or '__attribute__' before 'beer'

I have got the error expected '=', ',', ';', 'asm' or '__attribute__' before 'beer' in my code. How do I fix it?

Tags:

The question naturally receives close votes and comments for not having a sscce mcve minreprex mre

The OP amends the question adding the code

#include <complex.h>
complex pint beer;

and someone notes that it should probably use int, not pint. And now it becomes "off-topic - typo" - yet the closure now again requires 2 more votes?! Of course, if there were 2 prior votes of the no mre kind, all prior to the edit, the 3rd close vote with "typo, unlikely to help future readers" is the correct one for the current revision of the question, even though no mre has the majority.

I do not think chameleon questions should be the most difficult ones to close.

2
  • 2
    I guess giving gold badge holders more power than normal users will only work in tags where a sufficiently large number of gold badge holders exist. Especially for smaller tags, there is often only one or even non. And especially in those tags it's now very hard to get anything closed (because there are no 5 users who see a question).
    – BDL
    Commented Oct 7, 2019 at 6:39
  • 1
    What about silver badge holders? Antti's point is that drive-by voters with minimal score were able to have an equal impact to him. Commented May 16, 2020 at 23:43
12

Disclaimer: some of the examples aren't the best, its just the best I could do with out spending days on this. The names and descriptions of the new close topic suggestions are just strawmen that I quickly came up with. The specific names and descriptions don't matter as much as the overall ideas

Problem Context

I like the idea of just displaying all the reasons as NathanOliver suggested, however there may be issues due to the current close reasons names and descriptions. This answer is about addressing those issues, and facilitating NathanOliver's suggestion, I'm not creating an independent method to deal with the wrong close vote reason issue

from my discussion with Shog here

in response to simply displaying all chosen flag reasons:

Shog states:

Well... Part of the problem is that we have three close reasons that are all kinda interchangeable on an awful lot of questions.

The current close reasons are (source):

  • Duplicate: Questions that have already been asked and answered in the past should be closed. These serve as signposts to their previously answered duplicates. See How should duplicate questions be handled?

  • Off-topic: Questions are expected to be on-topic for the site to which they were posted, within the scope defined by the community.

    The “off-topic” option provides further guidance. Closers can either select a predefined message which will show in the close reason box, or type in a custom explanation which will be posted as a comment.

  • Unclear what you're asking: The way the question is currently written, it's hard to tell exactly what is being asked. The explanations in the question are not clear, or it is not clear what kind of answer is expected, or there is not enough information to solve the problem. The question may be reopened if it is clarified or if the missing information is provided.

  • Too broad: The question must be edited to limit it to a specific problem with enough detail to identify an adequate answer, and not ask multiple distinct questions at once.

  • Primarily opinion-based: While many good questions generate some degree of opinion based on expert experience, answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise.

I'm assuming Shog means

  • Unclear what you're asking
  • Too broad
  • Primarily opinion based

as the three amorphous close reasons.

The simple solution would be to simply combine all three reasons, however Shog goes on to provide context of a time where that was essentially the case:

Here's the thing, years ago, there was a close reason that was called "Not a real question". It was great for stuff like this - a big catch-all for all sorts of "what the hell am I even looking at?" questions. Of course... It was a pretty crap experience for askers - the name was pretty insulting, and there wasn't good guidance. All these other reasons came out of analyzing questions closed for that reason (some by a rather circuitous path, but I digress) Seems to me that a nicer form of [Not a real question] would probably suffice here.

Essentially, if you're not careful combining all three can result in vague insulting close reason.

As far as I see, to facilitate providing multiple reasons instead of picking one, we need to either:

  • disambiguate close reasons, so they overlap much less
  • combine all terms in a way that we don't repeat the mistakes of the past.

Ideally close reasons would specifically address the problems a asker has in their posts, so I think combining all three again, even with nicer language, would still fail to benefit the asker.

Instead I would propose some of the following terms could fill the same void as the current split and remove ambiguity:

  • Can't find a question
  • Asking more than one question
  • No objective answer

Each case is now mutually exclusive. However this essentially leaves out "too broad" where there's only one question, but it is not within the scope of what a person can reasonably answer on SO, and leaves out Unclear what you're asking, when some one asks a question, but the question doesn't make sense.

Using this query, and changing Comment comparison to 103 (unclear what you're asking), 104 (too broad) and 105 (primarily opinion based) we can get a sample of closed questions which we can see common patterns with these troublesome close reasons to maybe get more reasons that remain exclusive and clear and thus helpful to the asker.

Too broad

One thing I noticed is that questions like:

What is a good way to convert a software with single hosting one organisation use to single hosting multiple organisation use?

while technically too broad would work, cannot realistically be re-worked in any fashion where they would fit in with the site, only an entirely new question about a specific part of what they need might work. In which case, maybe these types of questions should just fall under off topic, so that OP knows they can't ask that specific question. The pitfall here is that it probably will still be a programming related question, which may be confusing to people posting.

In a scenario like this:

how to build voice recognition system for Kurdish language or other languages?

The user may want to ask a question "how do I use a library x in programming language y to do specific aspect I need to do in order to do voice recognition" but it is so far removed from the actual question asked, that the question can't be revived.

If we take, for example, these questions, this pattern doesn't necessarily work:

How to replace a single sentence that repeats constantly on all pages of a pdf with python

Write a script that calculates the sum of the numbers passed in parameter, checking that there is at least 2 numbers in unix?

They are still too broad, but the entire question may not need to be completely reworked in order to still count, or at the very least, we can provide better feed back to what is specifically wrong with the question.

What we have here is something I would like to call a "compound question". While there is a single question, there are several parts that need to be decomposed first to even start. In this case, the user could conceivably change their question to "How do I do this much smaller part of my question in specific language x with specific library y?" This happens often with homework problems.

In these scenarios, I would vouch for a new close reason:

  • Compound Question

A question that implicitly requires many other un-asked questions to be answered, but is not asking multiple questions. Consider breaking down your problem into simpler steps, and creating a question on the first step you have issue with.

I think using off-topic, Compound Question, and Multiple Question would cover to broad and give way better specific information to OP.

Unclear what you're asking

What is left is dealing with Unclear what you're asking

In cases like this:

How to resolve SyntaxError caused by x=input().split()

the "Can't find a question" probably works for this one. but for something like this:

Access form loops through subform

The issue is that there isn't any context for an question asker to understand what is happening. Basically when ever OP doesn't include code, instead of telling them "too broad", "unclear" etc... why not actually just tell them exactly what they are missing?

  • Not enough context

Your question does not include the required code or other relevant contextual information to answer with your question

With this, OP doesn't have to rely on people being nice enough to comment on their question to tell them what is wrong, and don't have to read the minds of people who've close voted to figure out what the deal is.

With those two there, there is still one last case:

  • some one asked a question
  • but no one can tell what the heck they are asking.

In one case, it may be that they asked in a foreign language, people use "unclear what you're asking" for foreign language questions. These should be Off-Topic instead (The community wiki would need to be updated to accommodate this).

In the other case, the question is in english, but no one knows what the user is asking, telling them "I don't know what you mean" (ie unclear what you're asking) is probably not going to help them, if they was going to help, why didn't they "make it more clear" in the first place? But telling them "Can you provide more context" will probably make them provide more information, which will allow people to understand what is happening even when OP is not able to convey that.

So "Unclear what you're asking" gets split into:

  • Can't find a question
  • Not enough Context
  • Off topic (foreign language posts).

Primarily opinion based

The last thing I haven't touched on yet is No objective answer. This would be used in all the circumstances as Primarily opinion based. The issue with Primarily opinion based is the wording. It opens people to subjectively evaluating whether something is "primarily" opinion based, and this ambiguity is my hypothesis why sometimes you would see flags with this used where it should have been one of the other two. Changing this to No objective answer changes peoples mindset into looking if there is an objective answer, which is also much more objective than primarily being opinion based.

Another way to put this: It requires them to find a way for an objective answer to be there rather than make a face value opinion if the question is primarily based on opinion.

Final thoughts/TLDR:

To recap:

  • Unclear what you're asking
  • Too broad
  • Primarily opinion based

turns into:

  • Can't find a question (there's not a question actually being asked)
  • Not enough Context (there's no code when there should be, and also useful when the question is incomprehensible)
  • Asking more than one question (Literally more than one question asked)
  • Compound Question (The question implies other questions need to be answered first ie "How to add 2 to a column in a database using python" implies the user needs to understand SQL in python first, then how to add 2 to a column in a database)
  • No objective answer ( Same as Primarily opinion based, but changed to make it less subjective)
  • and using Off-Topic when english isn't used or answer would have been too broad, but asks one non-compound question.

In conclusion all, none, or some of these categories could be used or some could be combined. I think on a whole they better help the asker get feedback from the reviewer, don't have nearly as much cross over as the previous 3 reasons, and reduce the need to comment from close voters.

4
  • 2
    With less-than-stellar English, it's entirely possible for a post to be hovering between "can't find a question" and "asking more than one question" (e.g., as in sentences that lead to the response "are you asking us or telling us"). "Unclear what you're asking" fits perfectly, though
    – muru
    Commented Oct 4, 2019 at 16:54
  • 1
    @muru Unclear what your asking is a non starter because it so vague. I do see your point though. Would you mind finding specific examples? If such a situation could arise it would pose ambiguity, though I'm wondering if there is some perfect being the enemy of good going on here. I will say, in situations where the asker is literally just telling things (I think I link to one) it would be the reviewers fault for flagging it as "multiple questions". If you don't see a question mark, there isn't a question, don't fill in the blanks, edit the question, or let the asker edit.
    – Krupip
    Commented Oct 4, 2019 at 17:13
  • I'll try to dig up specific examples, but a common case that comes to mind is Ask Ubuntu's "bug report" close reason. Often enough, some users post the description of a bug, without actually asking for anything (when what they actually want is either the status of the bug or help working around the bug). This happened often enough that they added a specific reason for it
    – muru
    Commented Oct 4, 2019 at 17:24
  • @muru IMO in those situations (no literal question asked, even if we can infer they are asking about the bug report) it should be "Can't find a question" (if it were to be asked here). I think these kinds of close reasons are going to need more objective literal reasoning to be used, but I think that's also required if we are going to have more logically separate close reasons.
    – Krupip
    Commented Oct 4, 2019 at 17:41
8

One suggestion I have for you to combat the close reason disputes is this:

  • If by the time the question has three close votes, check if they come to a consensus of two or more.

  • If not, require another vote for it to be closed.

  • Repeat this process until the question has a consensus of two or more.

I think we should not show a message like "n agreeing votes are needed". This may prompt users to vote in agreement with the (possibly incorrect) already-present close reasons. Instead, just don't close the question until enough agreeing votes accumulate. People should be able to understand this quickly enough, and converge on an appropriate close reason.

I suggest pulling up a chat room with links to those posts and discussing them. One reason that non-matching close reasons might not have been so prevalent before is that there are 5 close reasons and 5 close votes to close a question. With this change, the ratio of close reasons to close votes has increased, which probably resulted in the increased numbers you're seeing.

There are also multiple appropriate close reasons for many questions, which could've been exaggerated when the VTC count was reduced.

I've removed my other answer as it wasn't contributing much to this.

5
  • 3
    So... I'm kinda on the fence here, because we do currently display outstanding close vote counts, along with a message (hover text) detailing how many additional votes are needed. Maybe we could... just not do that, and side-step the whole issue. But as long as we do, we're gonna need to explain those numbers somehow, especially when they end up being inaccurate.
    – Shog9
    Commented Oct 4, 2019 at 0:28
  • 1
    @Shog9 See the edits to my answer.
    – S.S. Anne
    Commented Oct 4, 2019 at 0:39
  • @Shog9 Losing the information about how many outstanding close votes there are would negatively impact SOCVR and burnination/cleanup efforts, IMO, in addition to general users.
    – TylerH
    Commented Oct 4, 2019 at 15:20
  • 1
    The more I think about it, the less I think this will be a problem. We only need a "tie breaker" message in situations where there are already three votes but no consensus, a scenario that's still pretty rare - the chance that the 4th voter would have chosen a 4th unique close reason but for the message is vanishingly small; if such a message encourages them to pick from among the current three that isn't altogether a bad thing.
    – Shog9
    Commented Oct 4, 2019 at 15:24
  • @Shog9 any reason we couldn't simply show all the close reasons when we close? Commented Oct 5, 2019 at 12:23
8

I think there is a case to be made for changing the language and voting around Duplicates.

  1. Language: we might take some of the angst out of Dup closures if we don't call them closures.
    After all closing as Dup is actually saying "here's your answer", not "we don't (yet) want this question" as the other reasons do.
    I'm not sure what words to replace "Closed as Duplicate" with, but something that conveys the possitive message that the Q is (already) answered.
  2. Voting: Dup votes are saying "It's a 'good' Q, and it's already answered over there". Other votes are saying "we don't want this Q in it's current form - fix it or it may be removed"
    I suggest we seperate the threasholds, ie Dup votes and other votes are counted seperately, if 3 Dup votes (or Gold vote) are made before 3 close votes, it's "closed" (or whatevere we choose to call it) as a Dup. If 3 close votes come first, it's just closed.
2
  • "Resolved"? But if you want to change terminology, you better post a separate feature request, so it could be discussed without going off-topic here.
    – anatolyg
    Commented Oct 9, 2019 at 7:05
  • 1
    I like the idea of changing the wording on duplicate closures. It'd be good to remove the stigma many new users feels is attached to a dup-closure. I'm not sure about separating out the thresholds between dup-closure and just closure. There've been some times where I've seen dup questions closed for reasons other than duplicate. This has happened mostly when there have been close-votes for other reasons prior to someone finding an appropriate duplicate-target. But, it's relatively rare. I'm not sure it's worth increasing the overall number of CV required to close in order to solve that.
    – Makyen Mod
    Commented Oct 9, 2019 at 22:37
7

The closing system exists to allow askers and answerers to collaborate on constructing high-quality questions and answers, by... ...providing a clear path for questions that either cannot or should not be answered to be identified, and... ...ensuring that they are then either sufficiently improved or... ...removed from view.

I agree with the outlook here.

However, #2 is vague, and is also the sole pain point in this whole system. "a clear path" is not clear at all, and with tens of millions of questions being thrown at the system, it makes absolutely no sense to limit the reasons that a question may enter this path to only essentially a handful of reasons.

Does the current system enable individuals to trigger each critical action?

To Excess. This is evidenced by wasted votes and actions, and it goes it both directions; both legitimate and non beneficial votes are wasted from time to time.

Is the system "efficacious"? What does "efficacious" mean in this context?

Critically, your consideration here does not address close voters. As the users who cast these votes primarily interpret the system to determine the topicality of the site, they are an important yet ignored population in the analysis. Clearly from the results, and the distance between close vote reasons chosen, more work needs to be done to analyze the close vote user. Specifically, with regards to how the current minimalistic set of close reasons directs their efforts and dictates the site's topicality.

  1. We implement a second, "consensus" threshold: n close votes have to agree before a question gets closed.
  2. We add a bit of text to the close dialog (and hover text on the "close" link under questions) that explains how many votes are still needed to close the question, with a special "n agreeing votes are needed" output when the close threshold has been hit but the consensus threshold hasn't.
  3. We lower the close threshold to 3, with the consensus threshold set to 2.

I don't really disagree with doing this.

It is a decent bandaid approach to try to address the main issue, which I have continuously raised here (and, well, elsewhere): we need to refine the close reasons. There needs to be more of them, they need to be more specific, and there needs to be no grouping.

7

Here's my proposal: We lower the close threshold to 3

What happened to the well-received idea to instead give more power to (silver) badge holders?

I would propose to have votes by users who hold a silver badge in one of the question's tags count with double weight. Keeping the threshold at 5, this should lead to about the same result: it takes two badge-holders and one other user to close or reopen a question. This has been suggested before.

Admittedly, we probably would also need some special case for low-traffic tags with few badge holders. Closing their questions more efficiently had been another positive effect during the experiment, and we don't want to loose that. Maybe have a dynamic threshold: if fewer than X users would be eligible for double-votes on this question, lower the threshold to 4 or 3?

@Shog9: Please run the numbers for above proposal. How many questions would have been closed/reopened during the experiment with the above rules? How does this compare to the simpler rule of threshold=3? Would consensus improve?

5
  • 6
    What is the benefit? This just complicated things to account for small tags. Things worked very well with three votes to close. Commented Oct 4, 2019 at 0:48
  • @CrisLuengo It mitigates the cited fear of abuse and lack of expertise - even if unnecessary. Also it provides another privilege in the incentive system.
    – Bergi
    Commented Oct 4, 2019 at 1:10
  • 2
    Not sure what good an additional privilege in the incentive system does for low-traffic tags... Low-traffic means it's difficult to impossible to achieve a bronze, to say nothing of a silver badge. And gold...? Commented Oct 4, 2019 at 6:32
  • @bergi "the cited fear of abuse" ... "even if unneccessary". Yup, why do we need facts (188 vote wars on ~16.000 closures) if we have emotions ... Commented Oct 4, 2019 at 19:00
  • @JonasWilms Yes. I presume it should work just the same as with a threshold of 3 and no special badge-holder treatment, but (some) people would be happier with it.
    – Bergi
    Commented Oct 4, 2019 at 21:56
5

I would suggest simply displaying all the close voters' reasons for closing initially and then collecting data on the effect of doing so. Why spend effort building a complex system without understanding how multiple close reasons affects users (or doesn't)? Giving the post author more information seems like the obvious choice as it gives them more information about how their question was perceived to edit accordingly.

You could also collect data on whether there are correlations between multiple close reasons. This could reveal patterns that justify a reorganization of close reasons.

5

... meanwhile, reopen-voting and editing increased dramatically, while also becoming more effective. Hypothesis: an awful lot of folks weren't voting to reopen or editing because they'd lost faith that the cycle actually worked - when it began to work, more started to make use of it.

Alternate hypothesis: edit & re-open rate are a constant function of closure rate. We closed roughly double the number of questions, and as a result, edited roughly double, and then re-opened roughly double. (67% increase in raw closures, 82% increase in raw edits, 98% increase in raw re-opens)

Per your numbers:

Closures

  • Baseline: 33% of 31,502 = 10,395.66
  • Experiment: 53% of 32,837 = 17,403.61

Edits per closure

  • Baseline: 2,266 / 10,395.66 = 21.8%
  • Experiment: 4,132 / 17,403.61 = 23.8%

Re-opens per closure

  • Baseline: 854 / 10,395.66 = 8.2%
  • Experiment: 1,694 / 17,403.61 = 9.7%

I will take some time to digest these numbers a little more before I comment on whether the uplifts seen there are significant enough to be considered a shift in people's attitudes.

Open question: Is an increase in the re-open rate a repudiation of the idea that closures were more successful? Are we just closing more good questions which have to be re-opened?

5
  • 1
    The persistent problem with using reopening as the basis for measuring close error is that... So few closed questions get reopened. I'm a big fan of closing in general, but it seems unlikely to me that we're that perfect.
    – Shog9
    Commented Oct 10, 2019 at 3:16
  • 1
    @Shog9 - Agreed. I didn't mean to suggest that the all incorrectly closed questions get re-opened, more that a roughly constant (small) percentage of incorrectly closed questions get re-opened. If that was the case, then an increase in the number of incorrectly closed questions would manifest as an increase in the number of re-opened questions. But I agree that it's a poor direct measure, which is why I left that as an open question.
    – Stobor
    Commented Oct 10, 2019 at 5:48
  • 3
    "closed roughly double the number of questions" - worth keeping in mind that many (most?) of these additional closures go to questions that otherwise would be auto-deleted by 30- and 365-days roomba scripts - without any reopens (@Shog9 - can we estimate that?). So it looks reasonable to assume that reopen rate really increased if the amount of questions closed during experiment is roughly the same as amount of closed/deleted ones prior to it (it could be bit lower due to improved throttling of askers during experiment but probably not too much)
    – gnat
    Commented Oct 10, 2019 at 7:32
  • @gnat Actually I wonder if we really need to review questions that would otherwise be auto-deleted by roomba scripts anyway. Can't we concentrate on those questions that aren't so much in the red first. Commented Oct 13, 2019 at 20:51
  • 1
    @Trilarion I learned recently about stats (here) telling that vast majority questions entering close queue eventually end up closed or deleted anyway (even when expired from queue). That made me change my mind and now I am curious about smaller things, like if we can help educate askers by quicker closing instead of keeping them oblivious for many days or weeks before system takes care of the question. This is why I am thinking about mod review for questions that otherwise would hang longer
    – gnat
    Commented Oct 14, 2019 at 7:07
4

So many folks - new and old - think of closing as nothing more than a mechanism for deletion. Which, it is

WAT!

NOOOOOOoooooooooooooooo!!

Closing as duplicate is not a reason for deletion!!! Duplicates lead people to answers because search sucks! Deleting duplicates is not a win.

It sounds like there are some wrong assumptions in the entire analysis. Deleting duplicates does not help; it hurts. People are less likely to find their answers, because there are fewer things to make the 1000s of ways to ask something for which the answer ends up being the same.

A suggestion: Provide 2 "close as duplicate" options

  1. Close as duplicate and delete

  2. Close as duplicate and retain

It seems like it would be good to have documented consensus on whether or not certain closed questions should stay or go. As it is it's just left to the random whims of whoever has delete power whether or not they are familiar with the topic.

It seems like in a perfect world there would be very few duplicates. A user would either search for an answer and always find it, or else they'd type a question and before allowing them to post it the UX would always find the correct duplicate.

I realize that's impossible, but it's arguably reasonable goal and as such often a duplicate is a signal that the site failed. It should really only fail when the question is a duplicate, but is worded such that only a human could figure that out, but, such a duplicate is clearly not the type of close we want deleted. Rather we want it as more food for the search engine monster.

6
  • 14
    Duplicates do that only when they are good duplicates. When they contain an alternative search term or some such. And coincidentally, my experience shows that good duplicates are upvoted and retained. So there is no contradiction. Bad duplicates should be deleted, because they add nothing of value, they don't make reaching the answer easier. Commented Oct 4, 2019 at 9:48
  • I don't disagree with you but the post at the top does. It says closing is for deletion full stop. No mention of duplicates that help search
    – user128511
    Commented Oct 4, 2019 at 9:56
  • 8
    I think Shog was just innocently over-generalizing. Kinda like this answer does when it bundles all duplicates, good and bad, together ;) Commented Oct 4, 2019 at 10:00
  • I don't think Shog was and even quoted the answer showing it meant "no exceptions". It's not the only part of the answer that makes it clear the thinking is "closing = delete" which is emphatically wrong and if ignored will lead to a worse S.O.
    – user128511
    Commented Oct 4, 2019 at 10:28
  • 1
    I wouldn't tie dupes and deletion together like this. You'll only drive dissention between the never-deletes ("They're a signpost!") and the too-many-dupes ("It's over 9000!!!1"). Leave deletion out of it
    – Machavity Mod
    Commented Oct 4, 2019 at 13:01
  • 16
    Upvoted duplicates get kept; downvotes duplicates get deleted. We kinda have these options already...
    – Shog9
    Commented Oct 4, 2019 at 14:04
4

Another interesting statistics that not has been shown in the summary of the experiment is the mean time until closure or reopening. It's surely also connected to efficacy.

The activity during the experiment was a bit higher than usual. One could expect that if the threshold is changed to three votes permanently, that some kind normalization takes place which would probably reduce the efficacy in the future somewhat. And there is also a long term trend of slowly declining review activity.

With three instead of five votes approximately 40% more questions could be reviewed. This is a huge effect and if the error rate remained low it would also have been an indication that for many years quite a large number of votes have been wasted (not only because they aged away but also because they were redundant).

Three voters are bound to err more often than five. And while there is no gold standard and the absolute error rate can never be really estimated, one could have made a manual audit afterwards. Something like: according to Shog's judgement there were X% more errors during the experiment. I don't like much to fly blind, i.e. not knowing much about the error rate.

Error is here defined as a decision by the three or five voters that is different to the consensus decision, i.e. the decision that would have been taken if many more people would have voted.

Maybe sometimes put freshly closed questions directly into the reopening queue or make audits or perform close reviews with three and five votes and compare their outcomes. The audits could be freshly reopened questions with the original close reasons or just additional reviews on just closed questions.

Part of the unwelcomness is closing of questions and especially if the close was undeserved we should probably make sure it doesn't happen too often.

On the other hand, if the error rate was no concern, we could also reduce the close threshold to one, which would maximize efficacy. This summary by Shog does not show that the error rate remains reasonably low for a threshold of three.

Much easier is to check the consistency of the given close reasons. Surely one could take the past data with five close votes and draw three out of them randomly and compare the resulting close reasons. Basically replay the proposed new rules on old data and then showing a NxN matrix for all the N close reasons comparing the old and the new, proposed way. This does not seem to have been done.

Otherwise I'm happy that the experiment was finally made and that the results are so promising. I hope more experiments will be done.

3
  • "not knowing much about the error rate" what is an "error"? We all have disagreements on whenever a question is acceptable for SO.
    – Braiam
    Commented Oct 9, 2019 at 23:41
  • @Braiam Thanks for the comment, I added a definition of error rate above. Basically it is an outcome of the sampling of five or three votes that is different from the consensus outcome. The way to deal with subjectivity is to average over everyone. Otherwise, if the error rate would be meaningless we could also reduce the threshold to one right away and have 100% efficacy. We are bold with the proposed change to three but not that bold. Commented Oct 11, 2019 at 9:24
  • Actually we do sometimes reduce the threshold to one. See dupe hammers or moderator votes. But that are rather exceptions. Commented Oct 11, 2019 at 9:25
3

I think there's really only one good way to see if this proposal works: test it.

I think on the outside, it looks like an excellent idea. It's just that I think the best way to know for sure would be to run another experiment. Same length of time, with a sanity check at the end again. If that experiment has no harmful effects, then I'd say go for it. The results you have taken the time to post have been excellent so far, so barring any harmful effects from the proposed system, I say give it a go.

3
  • To "test it", they need to invest time and money to implement it (which would be wasted if the experiment fails). So it'd be better if we determine if it's a good idea before actually trying it out in the wild. Commented Oct 7, 2019 at 0:33
  • Unlike something simple like a single number change, this is a whole new mechanic. Commented Oct 7, 2019 at 0:34
  • That is a fair critique.
    – user10957435
    Commented Oct 7, 2019 at 3:40
2

A few thoughts on this:

I participate in moderation of a couple of small-ish tags, so this is good news indeed for the tags which we desperately need the answers for, but without needless clutter.

  1. Why wait? This needs to have the flavor of continuous improvement, not continuous hand-wringing. It was clear to most of us before the detailed analysis that this was a good thing. Frankly, it should have been done long ago. The beautiful thing about complex systems (and this is one) is that there is no way to anticipate all the side effects. Act now, solve new problems later. Don't suffer analysis paralysis.

  2. The proposed changes need to be paired with a user experience update. As the linked post mentions, feedback to users is delivered in a manner that is almost designed to be off-putting. This also is something that should have been addressed long ago. It's very clearly a problem, and has been for a long time. Don't delay on that either.

  3. Begin displaying notices immediately upon receiving close votes. On a small tag, it shouldn't need to take 1-2 weeks before we tell a user their question needs help. I usually try to leave a comment, but I don't think those comments are always read or acted upon. An official-looking note from the system may result in the users taking action where they otherwise aren't, and if they don't, then the (totally avoidable) close action happens.

  4. Perhaps down-vote should be accompanied by a quick and optional "why?" dialog. Similar to the previous reason, this could assist other users in moderating the site, providing helpful feedback to users without requiring them to take the time to type out a 100-character comment to the effect of the communication necessary.

-2

I think a big part of the perception problem is linked to the name "close". I think a better name would be one that is more positive sounding about the possibility to come back. Maybe something like "sent to the question clinic" or "still fledgling", but shorter?

Another thing is to try and open up more the discussion between closer and author. Not completely sure how to do that, but maybe edits should be sent back to the closers to decide whether to re-open (not sure if that's already the case, but I have the impression that it isn't, probably to avoid giving them too much power over the question).

3
  • I agree. I like the connotations of your suggestions much better. Interested to hear other thoughts!
    – Meg Risdal StaffMod
    Commented Oct 13, 2019 at 20:49
  • 8
    "Closed" was already renamed to "on hold" several years back. I never really saw any evidence of that making a difference. If you wanted to change the phrasing, I'd say the best choice is "pending revisions". But I really don't think the problem lies in the wording. People want an answer, and having the system prevent them from getting an answer is going to cause friction, no matter what badge you put on it.
    – Cody Gray Mod
    Commented Oct 13, 2019 at 23:32
  • 2
    I suspect both are important, @Cody: we introduced "on hold" to indicate the goal of achieving fast closure, editing to fix, followed by fast reopening. But we didn't do much else to support those goals. Closing is too slow, waaay too many fixable questions never get closed at all, authors are never notified in most situations, and those that do get fixed then take too much time and work to get reopened. We did the messaging but didn't do the work. Also... Anecdotal, but IME almost no one knows what "on hold" means or how it relates to "reopen"; "closed" at least we didn't invent.
    – Shog9
    Commented Oct 14, 2019 at 19:49
-6

Based on the evidence provided and goals stated, here's what I think should be done:

  1. Closure should require consensus.
  2. Number of votes for each close reason should not be displayed until the question is actually closed.
  3. Once the question is closed, all votes should be displayed with all reasons. That is, each reason that was voted at least once should be shown (highest vote count first), with the names of each user to voted for that reason listed with the reason.

    Alternatively: only the highest vote count reason is shown, but list the other people's names separately, as in "other reasons were chosen by…"
  4. The minimum number of votes in agreement should be three. If combined with the same minimum for closure, this would mean that if the first three people agree on the reason, the question can be closed with just three votes. But when there is some disagreement, there will always be at least one closure reason that got three votes, providing some strongly useful feedback to the author of the question.
  5. There should be a reopen prompt similar to the closure prompt. It is not sufficient for a person to just vote to reopen a post. They should be required, just as people who vote to close are, to provide a valid reason for reopening. And the same rules for consensus would apply.


Supporting discussion for the above:

With respect to consensus voting…

I agree that at least one good thing to come out of this experiment is support for requiring consensus on close votes.

In fact, in addition to requiring consensus, I would reduce the feedback available on still-unclosed posts, so that the dialog shows only the number of close votes cast so far (i.e. the same information already available on the "close" link itself). IMHO, if a person isn't sure enough about why they are closing the post to figure out the reason themselves, they shouldn't be voting to close at all.

Even absent the question of consensus, displaying the actual reasons provided by previous voters lowers the bar for would-be close-voters, making each subsequent close vote just a little easier then the previous. And of course, once you throw consensus into the mix, you have the problem noted by others already, that people may be swayed to use an inappropriate reason, because they prioritize closure over accurate reasoning. This gives more weight to early votes.

It's my opinion that each close vote ought to be considered independently by the person casting the vote, and should carry equal weight. Reducing the information available to people before the question is actually closed can help ensure this is the case.


As far as the overall question of the experiment goes…

per the metrics we established at the start of this experiment, this was a resounding success!

My problem with this is that, your metric is not independent of the thing you're changing. For example: "Close efficacy. How likely is the first close vote or flag on a question to result in it being closed?"

This can be rephrased as "given that a post received at least one close vote, how likely is it that the post will be eventually closed?"

IMHO it is intuitively obvious (granted, a risky approach when it comes to stats, but I think reasonably applicable here) that if you reduce the number of votes it takes to close a post, then the probability of a post receiving at least one vote eventually being closed will in fact increase.

After all, it certainly increases the probability of any post being closed, and so it naturally also increases the probability of some subset of all posts (i.e. those with at least one close vote).

It doesn't take an experiment evaluated with statistical analysis to demonstrate this. Even if you just assumed people act at random, you can demonstrate this just by computing the relative probabilities. It is natural that when the threshold for closure is lowered, more posts will be closed.

In this respect, the experiment was designed from the outset to support the idea of lowering the threshold, because the metric used is inherently going to be higher under the conditions of the experiment.

This means we must tread carefully, and not think that the experiment has provided much in the way of strong evidence in favor of reducing the vote threshold. It has provided other useful information, but the experiment was predestined to demonstrate that a reduced threshold is a positive outcome.


With respect to the exact number of votes required…

I don't think there's anything inherently wrong with making it easier to close and reopen questions. But it will of course increase churn, and it seems to me that there's some asymmetry with respect to closure vs. reopening. In particular, to close a question you have to pick from some relatively small set of options. Some people will pick randomly, but I think most will make an honest effort to make sure they are picking an option that actually applies, providing some friction to ensure quality.

On the other hand, a person can vote to reopen for any reason at all. This encourages sympathy voting, in which people just feel bad for a person (because too many people take personally what ought to be a strictly objective process). People who want to reopen a post ought to be presented with a similar set of options, to make them think twice about whether a question really should be reopened.

This will help offset the detrimental effects that lowering the threshold for voting will cause.


Bottom line for me: it seems to me that one of the biggest challenges that Stack Overflow faces today is the relative deluge of poorly presented, poorly researched questions that flows onto the site every day. Reducing the threshold for voting will obviously make it easier to close such questions, but it will also make it easier to reopen them. Due to asymmetry in the close/reopen processes, churn favors keeping posts open.

It would be better to focus on improving the quality of the feedback to the author of the question. If we must reduce the threshold for voting, this must be done in parallel with improvements to ensure that the author of the question still gets high quality feedback (i.e. at least three votes in agreement) and to ensure that a question does not get reopened unless it's done so with an actual valid and applicable reason.

9
  • 2
    I like the general feel of this suggestion, but I find the 5th point rather unmanageable: "There should be a reopen prompt similar to the closure prompt." The criteria for reopening is "Is this question closed when it should be open?" I can see the value in an comment like "This question was unclear but the author has since edited it into a clear on-topic question" to help future voters see that the close reason no longer applies and use their reopen votes, but the only reason that anyone should give for casting a reopen vote is "This question is on topic, clear and specific."
    – Davy M
    Commented Oct 4, 2019 at 19:11
  • @DavyM: thanks. I think that "unmanageable" is in the eye of the beholder, but I really appreciate the concerns you raise. Still... There are at least two obvious reasons to reopen: "I disagree the post was ever in need of closing" and "the post was legitimately closed but now has been updated". The latter would be available only if the post was indeed updated. This approach can be broadened, such that a reopen prompt would require specific counters to specific close reasons. ... Commented Oct 4, 2019 at 19:19
  • ...I admit that this is not a fully-fleshed out idea. But I think it still has merit. More to the point, something should be done to balance out the current asymmetry between close/reopen voting. Otherwise, a lowering of threshold for vote requirements will simply increase churn, favoring the reopening side. The net effect being the opposite a broader goal of improving the overall quality of questions on Stack Overflow. Commented Oct 4, 2019 at 19:19
  • 2
    "the metric used is inherently going to be higher under the conditions of the experiment." That's intuitive, but there's a trap to it: people change their behavior. Note that the quantity of questions getting reopen votes went up a lot, while the quantity of questions getting close votes/flags did not... But it easily could have. There is no shortage of questions that could be closed, nor questions that could be reopened, nor people who could vote. I can calculate what is possible without a test; I can't know what will actually occur.
    – Shog9
    Commented Oct 5, 2019 at 1:39
  • @Shog9: "the quantity of questions getting reopen votes went up a lot, while the quantity of questions getting close votes/flags did not" -- sorry, I didn't glean that from your post above. Could you elaborate? That said, given that data, it's not clear to me how you demonstrate that's due to anything other than the fact that requiring fewer close votes leads to, uh, fewer close votes, while requiring fewer close votes leads to, uh...more closed questions, thus more opportunities for reopen votes, thus more reopen votes. Commented Oct 5, 2019 at 1:48
  • @Shog9: and note that, at least in my experience, as questions age, they gain less attention, and so once a question is reopened, it's much less likely to be re-closed. The fact is, while the number of users is large, the number who participate in moderation is much smaller. The close/reopen churn has diminishing returns for both of these reasons. Commented Oct 5, 2019 at 1:51
  • 3
    I disagree with this. That users are picking multiple close reasons can mean a variety of things. One of those is that the question is just really bad in multiple ways. We shouldn't make those some of the hardest to close. They should be some of the easiest to close, or at least no harder than average.
    – Makyen Mod
    Commented Oct 5, 2019 at 1:58
  • @Makyen: you make a good point. However, my suggestion is based on my concern that lowering vote thresholds makes it harder for the user to be provided useful, actionable information about what needs to be fixed in their post. Fact is, I think the current threshold now is fine, and lowering it has some potentially harmful results (e.g. lower-quality feedback to author, and easier reopening of posts that should've remained closed). So if the threshold is going to be lowered, we need something to balance those negative effects. Commented Oct 5, 2019 at 2:04
  • 2
    Re-reading my prior comment & this answer, I realized I really should have been more nuanced in what I wrote. I actually agree with a good portion of what's said in this answer. I do, however, disagree with a requirement for agreement of CV reasons, which you prominently suggest here. I don't feel that requirement flows logically from the rest of your analysis. The primary purpose others have put forward for requiring CV reason agreement is so the single best close reason is shown to the OP. That's irrelevant if all CV reasons are shown, as you also suggest here, and with which I agree.
    – Makyen Mod
    Commented Oct 5, 2019 at 2:57
-12

We lower the close threshold to 3, with the consensus threshold set to 2.

2 seems low.

Perhaps 3 (whether consensus or not), or 2 from gold badge holders in the question's tags? That also does away with the consensus concern.

Alternately: 3 (whether consensus or not), or 2 from gold badge holders in the question's tags if they're in consensus with each other (which doesn't do away with the concern, but perhaps mitigates it somewhat).

6
  • 1
    Many close reason are actually consensus, often a question can be both too broad, unclear, missing mcve etc Commented Oct 5, 2019 at 14:49
  • @PetterFriberg - Indeed, but I don't think that's how Shog9 was defining "consensus" in his proposal. I'm referring to that definition. Commented Oct 5, 2019 at 15:22
  • 1
    As far as I can understand that's his definition, people select difference reasons. Personally to me it's not a problem in itself and there is no need to have consensus on the reason, there is only a need to determine which close reason to display, 1 or many? Commented Oct 5, 2019 at 19:47
  • @PetterFriberg - It's not ambiguous. From the post: "Confusion: questions closed without a consensus ...it’s much easier for a question to be closed without any reason having a clear majority of voters." "too broad," "unclear," and "missing mcve" are different close reasons in the current system. Again, I'm not saying that those aren't sides of the same coin, I'm just using the definition of consensus that Shog9 quite clearly stated. Commented Oct 7, 2019 at 8:36
  • 1
    without any reason having a clear majority of voters, according to me Shog9 is indicating, that there is no majority that have selected same close reasons, hence with 3 voters this means that they all have select different reason, for example 1 too broad, 1 unclear and 1 mcve. Many other answer on this question address how this issue can be solved (highest tag score, showing multiple reasons etc). Overall my personal idea is that there is no need to have this kind of consensus, it's only a need to figure out what to show to OP. Commented Oct 7, 2019 at 9:15
  • @PetterFriberg - Re your final sentence: I agree, I'm not supporting a consensus criterion above. And separately, I'm supporting communicating votes properly to the OP. Commented Oct 7, 2019 at 10:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .