This post details the outcome of an experiment that reduced the thresholds for closing and reopening questions to 3 votes on Stack Overflow. Warning: this post is looong. There is no "tl;dr". There is no summary. I've been thinking about this crap almost every day for 7 years, and I'll be damned if I'm not gonna inflict at least a little bit of that pain on anyone else who thinks they're interested.
...if you are interested... Well, there's a lot to chew on here; you might even enjoy it.
But first...
The value of a question lifecycle
So many folks - new and old - think of closing as nothing more than a mechanism for deletion. Which, it is - but thinking of it as just that is like thinking of our trial system as a mechanism for execution: if that was all it was, we could just dispense with a lot of the process. The beauty of this system is that it's more than just one more step on the road to deletion: it's a choice, a branch, in which multiple people can collaborate to create something that wouldn't be possible without it.
The concept of closing did not originate with Stack Overflow, but we lean very heavily on it as a tool for community moderation. It's hard to know how much of this was intentional and how much was organic, but at some point that distinction becomes academic; we have to analyze the system as it exists and consider its future in terms of both its positive and negative effects today.
As we discussed recently on the blog, we're looking at the core systems that underpin Stack Overflow. I'm privileged to have been able to step back over the last couple of months and take a holistic view of closing, examining and discussing its role along with my colleagues. Based on this analysis, I would like to propose the following four-part sentence as a summary of purpose for closing - and the larger "close / edit / reopen OR delete" cycle - on Stack Overflow:
- The closing system exists to allow askers and answerers to collaborate on constructing high-quality questions and answers, by...
- ...providing a clear path for questions that either cannot or should not be answered to be identified, and...
- ...ensuring that they are then either sufficiently improved or...
- ...removed from view.
In this summary, #2 corresponds to the act of closing a question, while #3 encompasses both editing and reopening, and #4 is deletion. These are the critical actions that enable this purpose to be realized.
If we assume that this is a reasonable purpose and that close-edit-reopen/delete can successfully accomplish it... Then we're left with only one crucial question:
- Does the current system enable individuals to trigger each critical action?
That's it. If the purpose is good, then in order for this system to work we just need each action to be triggered at the proper moment. In other words, is the system "efficacious"? If that happens, then we're golden. What does "efficacious" mean in this context?
- An asker need not wonder why their question isn't getting answered - the system will tell them, and help them to improve it.
- An asker need not wonder why their question is closed - if they've corrected the problems, it will be reopened.
- An answerer need not sift through piles of unanswerable questions - they will be removed.
And all of this goes on to benefit not just each asker or answerer, but the world at large: no useless search results, only answers. A virtuous cycle indeed!
If this sounds like... a much happier, less frictious website from the one we all know and love... Well, yeah. Let's talk about that.
The public perception of this experiment, and of closing
If the doors of perception were cleansed everything would appear to man as it is, infinite. For man has closed himself up till he sees all things thro’ narrow chinks of his cavern. -- Wm. Blake
The reality is, closing has been controversial - divisive even - for as long as Stack Overflow has existed; everyone sees something different in it, and wants for different things from it. We collected quite a bit of feedback on this experiment, both here on meta and over on Reddit (thanks to curiousdannii for digging that thread up!). I've grouped these into rough themes, and present them only as a tool to help us all get a bit more familiar with the perspectives of our peers:
Optimism
Motivated
Abuse
Side-effects
Non-English posts
Closing vs answering
Old, closed questions with answers
-
Lack of expertise for close-voters
Reopening
Low-traffic tags
Duplicates
We also now run an ongoing satisfaction survey, presented to random users on Stack Overflow. Meg took the time to analyze the feedback that we received in response to our question about what users find most frustrating about using the site. No big surprises there: a lot of people mention closing as their number one frustration. We did see a slight increase during the experiment, but not enough to be considered significant. But keeping in mind the importance of managing perception, we'll be keeping a close eye on such feedback as we proceed with changes. (foreshadowing!)
This all adds up to a lot of food for thought. We're incredibly fortunate to have so many people from so many different backgrounds willing to provide thoughtful input on these ideas; big thanks to everyone who chimed in, both here and out there on them Wild Internets. We heard hopes, excitement, concerns, and overt criticism - and all of that is important when it comes to making improvements.
As seen from this feedback, we're clearly not living in that virtuous cycle I described earlier -- or at least, that's not how our world is perceived. Closing is seen by different people as a useful and necessary tool, underused but also abused, and at times capricious. The mechanisms by which questions get closed and especially how they get reopened are poorly-understood. Folks are frustrated both by the volume and nature of questions that don't get closed and by the volume and nature that are closed. When folks think a system doesn't work, they become less willing to rely on it: perception can very much become reality.
Lowering the threshold for closing and reopening is at best a small step in the right direction here... And potentially a step in the wrong direction. Either way, we must be willing to accept the results and learn from them. So... How'd we do?
Results
Remember that "crucial question" I asked earlier?
- Does the current system enable individuals to trigger each critical action?
Answering this question about efficacy was the primary purpose of this experiment. To that end, we chose to monitor three separate measurements to determine the results:
- Close efficacy. How likely is the first close vote or flag on a question to result in it being closed?
- Re-open efficacy. How likely is the first reopen vote on a question to result in it being reopened?
- Edit efficacy. How likely is a (body) edit on a closed question to trigger reopening?
Note again that #2 and #3 are both measured as being part of the same action: correction and reinstatement. I happen to know that deletion already works fairly well, so I didn't measure that here. Yes, I know there are things we can do to make deletion work even better; let's talk about that elsewhere.
The change in threshold took effect on Aug 8 at 18:20 UTC and was reverted on Sep 7 at 16:33. It was noticed within minutes of being turned on; anticipating this, we posted a public announcement. The first week of the experiment saw a noticeable spike in activity as users re-engaged with the system. On top of this novelty effect, close and reopen votes can take up to two weeks to age away after being raised, so efficacy on the edges of the experimental period is… a bit fuzzy. To minimize the impact of both problems, I decided to compare two periods of time separated by a two-week buffer:
- Prior: July 18th to August 1st
- Experiment: August 15th to August 29th
I also compared the two-weeks immediately after the experiment as a sanity-check:
- Post: September 7th to September 21st
Prior to the experiment
- Close efficacy. 36% of initial close votes or flags resulted in either the post being closed or a close review that completed with a verdict of “Leave Open”.
- 33% of initial close votes/flags resulted in closure - there are very few “Leave Open” reviews, but Triage also filters out some flags. Also, some votes reviewed as Do Not Close were still closed.
- There were 31502 "first close votes" during this period
- Re-open efficacy. 89% of initial reopen votes resulted in either the post being reopened or a reopen review that completed with a verdict of “Leave Closed”.
- 38% of initial reopen votes resulted in the question being reopened
- 52% triggered a reopen review that completed with a verdict of “Leave Closed” (but some were still reopened)
- There were 854 "first reopen votes" during this period
- Edit efficacy. 65% of initial edits after closure resulted in either the post being reopened or a reopen review that completed with a verdict of “Leave Closed”.
- 7% resulted in the question being reopened
- 59% triggered a reopen review that completed with a verdict of “Leave Closed” (a few were still reopened)
- There were 2266 relevant edits during this period
During the experiment
- Close efficacy. 55% of initial close votes or flags resulted in either the post being closed or a close review that completed with a verdict of “Leave Open”. A 53% improvement over the baseline.
- 53% of initial close votes/flags resulted in closure - there are very few “Leave Open” reviews, but Triage also filters out some flags. Also, some votes reviewed as Do Not Close were still closed.
- There were 32837 "first close votes" during this period
- Re-open efficacy. 94% of initial reopen votes resulted in either the post being reopened or a reopen review that completed with a verdict of “Leave Closed”. A 6% improvement over the baseline.
- 44% of initial reopen votes resulted in the question being reopened
- 52% triggered a reopen review that completed with a verdict of “Leave Closed” (but some were still reopened)
- There were 1694 "first reopen votes" during this period
- Edit efficacy. 74% of initial edits after closure resulted in either the post being reopened or a reopen review that completed with a verdict of “Leave Closed”. A 14% improvement over the baseline.
- 12% resulted in the question being reopened
- 63% triggered a reopen review that completed with a verdict of “Leave Closed” (a few were still reopened)
- There were 4132 relevant edits during this period
After the experiment
I’m also including a two-week period starting on the day the experiment ended, to illustrate that the changes seen during the experiment were not part of a longer trend. This is a weak comparison relative to the baseline because a significant number of votes are still pending as I write this - however, more than two weeks have passed and we should assume that most of these votes will age away unhandled.
- Close efficacy. 36% of initial close votes or flags resulted in either the post being closed or a close review that completed with a verdict of “Leave Open”.
- 34% of initial close votes/flags resulted in closure - there are very few “Leave Open” reviews, but Triage also filters out some flags. Also, some votes reviewed as Do Not Close were still closed.
- 36% were still pending as I wrote this
- There were 32692 "first close votes" during this period
- Re-open efficacy. 91% of initial reopen votes resulted in either the post being reopened or a reopen review that completed with a verdict of “Leave Closed”.
- 38% of initial reopen votes resulted in the question being reopened
- 54% triggered a reopen review that completed with a verdict of “Leave Closed” (but some were still reopened)
- 4% were still pending
- There were 861 "first reopen votes" during this period
- Edit efficacy. 64% of initial edits after closure resulted in either the post being reopened or a reopen review that completed with a verdict of “Leave Closed”.
- 6% resulted in the question being reopened
- 58% triggered a reopen review that completed with a verdict of “Leave Closed” (a few were still reopened)
- There were 2531 relevant edits during this period
These are very similar to the observations taken prior to the experiment, so it's unlikely the experiment results are heavily skewed by some larger trend.
Here's a nifty little table that summarizes those results:
Hat-tip to Meg for the table idea.
Wow... Everything got more effective! Also, the total quantity of questions nominated for closure stayed about the same, even as efficacy - questions actually getting closed - went up; meanwhile, reopen-voting and editing increased dramatically, while also becoming more effective. Hypothesis: an awful lot of folks weren't voting to reopen or editing because they'd lost faith that the cycle actually worked - when it began to work, more started to make use of it.
You might notice that some of these numbers are still very low: less than half of the questions nominated for reopening get reopened, barely half of the questions nominated for closure even get reviewed. Even though this is a big improvement, we still have a long way to go before we can say the system is truly efficacious. We know - and if things pan out, we'll be talking a lot about this in the coming months! As I wrote earlier, this experiment is just a small part of much more extensive efforts to bring harmony and efficacy to our question lifecycle.
Effect on smaller tags
One of the hopes raised almost immediately when this experiment began (hey Cindy!) was that it would improve the ability of folks active in smaller tags to actually... do anything. We've known for years that this cycle was particularly broken when it came to niche topics, but nothing we've ever done there has been particularly effective.
I looked at questions in a set of tags that get >= 1000 questions/month on average, and calculated the efficacy of the same three actions for the same time periods listed above in the Prior and During sections:
Then I looked at the same metrics for all questions without tags that get >= 1000 questions/month:
Yeeeaaaaah! We got a larger improvement on smaller tags! Finally, something that actually helps.
Conclusion
I think it's safe to say at this point that, per the metrics we established at the start of this experiment, this was a resounding success! The current system, with lower close/reopen vote thresholds, better enables individuals to trigger each critical action: close, reopen, edit. It's highly likely that perception was also affected: folks edited and voted to reopen many more questions during the experiment than either before or after, mirroring the increase in efficacy for both actions.
BUT: perception. Remember all those concerns? What else happened as a result of this change, and... Does the damage outweigh the good?
Side effects
Some effects are easy to guess at: making questions easier to close and reopen should mean that more questions get closed and reopened. Others are less sure: did this encourage more people to review? To review more? To get into fights over questions? These are all hopes or concerns that we considered beforehand or which were raised during the test, and I'll look at each of them below:
Abuse: close wars
A close war is when a question is closed, reopened, then closed again… Possibly going through several cycles before being either left open or deleted. This was a HUGE concern back in 2009 - indeed, it’s probably why the threshold was raised from 3 to 5 to begin with! Stack Overflow moderator Bhargav suggested that we watch for this almost instantly; a few other people also expressed similar concerns.
To make this easy, I’ll call anything a close war if a question is closed twice.
- In the 30 days prior to the experiment, 100 questions were closed at least twice; 83 of those involved a gold tag-badge holder using their ability to instantly close a question as a duplicate.
- During the 30-day experiment period, 188 questions were closed at least twice, with only 118 of them involving a gold tag-badge holder.
So, there were more close wars. But also, there aren’t very many close wars to begin with, even using this fairly generous definition of “close war”. In fact, there are few enough of these wars that we could probably just raise a mod flag any time one crops up (although we’d probably want to wait for at least two cycles before doing so).
Turns out, in 2009 we let folks vote to close the same question as many times as they wanted to - so the same three people could just camp on a question they wanted to close or reopen, fighting it out all day long. We’ve long ago limited users to one close or reopen vote per question, ever, and as a result, close wars are mostly a thing of the past, regardless of close/reopen threshold.
Confusion: questions closed without a consensus
This one was raised a couple of times on meta: with only 3 votes and 5 close-reasons, it’s much easier for a question to be closed without any reason having a clear majority of voters. Sure, this can happen with 5 votes as well, but it’s not common. I’m ignoring situations where one person casts a vote and then a moderator closes for a different reason, since moderators can do this regardless of the threshold.
- In the 30 days prior to the experiment, 49 questions were closed without a consensus reason.
- During the 30-day experiment period, 560 questions were closed without a consensus reason.
That’s a huge increase. And it’s even more problematic because of how we determine the displayed reason: we pick the oldest vote. So if someone picks a bogus reason, that gets discussed a bit and then two other people pick two better reasons… We show the bogus one.
One possible solution here is to require at least two voters to agree on a reason before closing. More on that idea later...
Review: Close queue backlog
For the rest of these sections, I’m just going to include charts - the trends are easy enough to see, and you're probably thoroughly bored by now. You may have noticed that I've deviated from my signature "ASCII tables and SEDE line charts" presentation style in this post: Meg Risdal was kind enough to go through and replace all of those with snazzy R-generated charts. They're ruining my low-fi street cred, but I do have to admit... They're a lot easier to read.
Number of tasks in the close review queue, sampled hourly over the past year. That huge dip in August is the experiment.
Review: Number of reviews
Number of reviews per week in the close and reopen queues. Huge spike in # of reviews being done in both queues!
Review: Number of active reviewers
Number of reviewers active per week in the close and reopen queues. An initial rise in the number of close reviewers, then back to normal, then below normal after the experiment ended. More interesting is the spike in reopen reviewers - a LOT more people were looking at these questions for the entire length of the experiment!
Overall: total questions closed
Total questions closed per week. Impressive rise, almost doubling the volume of closures by the second week, then falling off gradually approaching the end of the experiment along with the size of the backlog, the number of active reviewers, and the volume of reviews being performed.
Overall: total questions reopened
Total questions reopened per week. Even bigger proportional bump than closing, with less fall-off toward the end of the experiment - perhaps reflecting the larger number of additional reviewers. Drops like a rock as soon as the experiment ends.
Trends: efficacy over time, by close reason
This is a lot to chew on, so I’d like to bring these results to a… close 🥁… by returning to the concept of efficacy. There are five primary close reasons, and - as I’ve previously reported - there is a whole lot of variation in how they’re used. So it’s reasonable to assume that there is also a lot of variation in the efficacy votes and flags when considering the reason for closing chosen by the flagger or voter…
Close flag efficacy over time, by close reason
This shows the efficacy of the first close flag raised on a question, broken down by the reason chosen by the first flagger. I should mention at this point that a question can easily be counted multiple times in both these charts and the numbers above: if someone raises a flag and that flag ages away without being looked at, that’s an ineffective flag; if someone later flags the question again and this time it gets closed (or at least reviewed), that’s an effective flag. This becomes more interesting when breaking down the results, as reviewers may choose to filter the queue by close reason: e.g. if I’m only concerned with duplicates, I won’t bother reviewing questions that are flagged as off-topic.
We can observe a few new things here:
- That fuzziness around the edges of the experiment is clearly evident: flags raised before the start which were still pending became more effective due to the start of the experiment, while some of those raised near the end became less effective.
- Close flags are very ineffective. Remember, “effectiveness” here doesn’t even mean “the question gets closed” - it just means the question got reviewed.
- Duplicate flags were the most effective before the experiment but saw the smallest bump. The reason for this will soon become apparent…
Close vote efficacy over time, by close reason
As above, but now considering the reason chosen for the first vote in a given sequence of votes on a question and ignoring any flags that might’ve been active up to that point. If it’s starting to sound like you might not be able to add these values to the ones for flags above and arrive at anything that matches the numbers above… Well, that’s why I’m doing colorful charts. Look at 'em! Ain’t they pretty… Almost makes you forget how horribly complicated all this closing stuff is, don’t it?
So, there we go: duplicate votes were already extremely effective - both gold-badge holders and askers could bypass the 5-vote requirement to speed up closing of duplicates (and validate any flags). So while duplicates saw a slight increase in effectiveness, it was nothing close to the bump that all the other reasons saw. Duplicate votes are already very effective - further gains in effectiveness for them must probably be found elsewhere.
In closing: a proposal for next actions
To be honest... Next steps seem like a no-brainer here: for a very simple change, we get a huge increase in efficacy across the board. Everything just works better. It's the perfect drug!
Oh, right... That no-consensus thing went up by a thousand percent. We should, uh, probably fix that.
Ok, here's my proposal:
- We implement a second, "consensus" threshold: n close votes have to agree before a question gets closed.
- We add a bit of text to the close dialog (and hover text on the "close" link under questions) that explains how many votes are still needed to close the question, with a special "n agreeing votes are needed" output when the close threshold has been hit but the consensus threshold hasn't.
- We lower the close threshold to 3, with the consensus threshold set to 2.
I think this both buys us the advantages and removes the potential harm that was seen during the experiment. If no one can think of a good reason not to do this, I'd like to roll out the proposed changes as soon as we're done testing / iterating on the new post notice system that's currently under development.
This is something that we very much shouldn't be changing during testing of the post notice system, because it can impact our ability to analyze the results of that test: it will skew them in a bunch of different ways, and potentially cause us to overlook problems. Waiting on that test (and any work that arises from it) gives us probably a month or two to chew on and discuss this, as well as build and test the changes necessary to support a consensus threshold.
Thanks again to everyone who participated in this, both in discussion and especially in the actual work of closing, editing, and reopening questions on Stack Overflow. We must never forget that this machine is built on people and that they deserve respect and appreciation. And special thanks today to Meg Risdal and Cesar M, who contributed extensively to the creation and editing of this report - if you think this is rambling and confusing now, you should've seen my first draft!
Excited to hear your thoughts