This experiment has concluded
and the affected settings have been restored to their previous values.
Results are available here and here.-- Shog9, 2019-09-07
For the next 30 days, we'll be running an experiment on Stack Overflow: the threshold for closing or reopening a question will be set at 3 votes instead of 5. The primary goal here is to determine what effects this has on folks' ability to successfully close or reopen questions; we're also hoping to observe the larger effects - if any - such a change might have on the behavior of voters, flaggers and editors.
This is an initial foray into research on how we might make the venerable closing system more effective and less frustrating for all parties involved.
Background
Close voting was introduced as a concept at the very end of 2008, with the original threshold set to 3 votes to close, 3 to reopen. The explicit goal was to make closing more deliberate by restricting the influence that any individual could exert on it. The secondary effects - on reopening, on editing, on duplicate identification, on participation in smaller tags - were not apparent until much later. The closing system worked very differently back then; some notable differences included the ability of any one voter to vote to close the same question multiple times, a lack of any review system, and a much less guided process for closing duplicates. A few weeks after the voting system's introduction, the threshold was raised to 5... And has stayed there ever since. As a result, we have very little idea of what different threshold values might mean on today's Stack Overflow - hence the purpose of this experiment.
Numerous people have suggested such a change over the years, far too many for me to give credit to them all. So here are a few highlights:
- EJoshuaS wrote a very well-researched proposal for reducing this threshold (including a reference to a prescient observation from tvanfosson in 2009!)
- Andre Silva wrote another well-researched proposal for conditionally reducing the threshold based on the order of reviews
- Yvette Colomb has been gently prodding us for months to reduce the threshold in some scenario just to reduce frustration for folks doing close reviews
- Jon Ericson tested this once before on Software Engineering - from which we learned the importance of nailing down useful, plausible goals ahead of time.
Also related: How high an invalidation rate do reviews currently have?
Tangential effects
Currently, migrating a question to another site requires that 4 of the 5 votes all agree that the question should be migrated to a specific site.
As migration would be blocked otherwise, I'm also temporarily lowering the threshold for migration from 4 votes to 3 - essentially making a migration require unanimous agreement from close voters.
Also... There are, of course, a LOT of questions with three close votes currently pending. These will not be instantly closed - they will instead close upon receiving another vote (so if they are closed then they'll be closed with four votes, or possibly three if one or more pending votes age away before a fourth is received).
Concrete goals
Obviously, more questions will get closed and more closed questions will be reopened when the thresholds are lower; that's not what we're testing. Rather, we're looking to improve the efficacy of individual actions: if you, a member of Stack Overflow, vote or flag a question for closure, vote to reopen a question, or edit a closed question... How likely is that action to have any tangible effect on the outcome?
Next steps
If you observe any serious bugs, please report them here.
If you observe bad behavior, please flag it or open a discussion here on meta. In both cases, please focus on the behavior, not the people involved; let's keep this positive and constructive!
If you observe anything interesting, write it up!
At the end of 30 days, we'll restore the threshold to 5 (and the migration threshold to 4). Then spend some time analyzing the results. Depending on the outcome, we may adjust the threshold further in the future, leave it alone, or... Try something else.
Community Alternative Future Experiments (Suggestions from comments)
- Lower the threshold to
4
votes but give gold tag badge holders two votes instead of one (provided they also have the standard moderation badges). – Pearly Spencer