This is the third post in our series of regular (roughly weekly) updates on the Documentation Beta. See also the previous post in the series. Yes, it’s been three weeks since the last update, but at Stack Overflow all numbers less than 6 are basically 0.
The big news is that we've got a lot numbers about reputation, and have hashed a new model we'll be implementing in the coming weeks.
But first...
Shipped Changes
Improvement Requests
We’ve been making improvements to improvement requests:
- improvement requests now require 2 votes to dismiss.
- you can now handle improvement requests submitted by others on a topic or an example even if you also put in a request of your own.
Other changes currently being considered include:
- only allowing one improvement request of a given type (e.g. "missing example") - additional requests of the same type would be converted into upvotes on the original request.
- improving the UI to make it clear that when you pick a reason for a downvote on an example, that reason is actually converted into an improvement request.
Example Limits
There are now two rules in place to keep the number of examples on a topic in check.
- Users with less than 2,000 reputation cannot propose creating a 7th or greater example
- No users can propose creating a 13th or greater example
These limits do not apply when proposing a roll back, and existing topics retain however many examples they had before these changes.
The intent of this change is to encourage very large topics to be broken up, and to discourage overly specific examples. We don’t think this change by itself is sufficient to accomplish all that, but it’s a step in the right direction.
Planned Changes
Proposed Changes Review Queue
It’s still coming. Adapting review to handle Documentation is pretty complicated it turns out.
“Focus” or “Introduction” Section For Topics
Also still coming. We won’t be introducing this until the review queue is shipped, since it has significant implications there.
Topic Outline
Based on feedback, we're going to be making some changes focused on making it easier to scan and navigate individual topics.
We’ll be adding a topic outline in the sidebar, here’s an early mock-up:
As part of this, we'll also be removing example collapsing, and the side-by-side view.
Both of these features were meant to make it easier to see more of a topic at a glance, but ultimately proved too confusing (in the case of example collapsing) or saw little use (in the case of the side-by-side view).
Reputation
Now for what most of the last few weeks have been spent producing, a planned update to Documentation’s rep system.
In the previous update, we laid out some things that the rep system was supposed to encourage:
- Creation of missing Documentation
- Creation of useful Documentation
- Good editor behavior
- Good requester behavior
While digging through the data, we’ve added some more things we think are important to encourage:
- Correct choice between creating and editing
- A user should be fairly rewarded for making the right choice, whether that is creating new examples or improving existing ones
- Review independent of potential rep gain
- “Trivial changes” shouldn’t be rejected because they’ll earn rep and don’t deserve it
- Predictability
- The rules should have an obvious logic to them, and behave “as expected” once learned
The System
A few notes to start off.
Anything below that counts characters or diff lengths ignores whitespace and formatting. Nothing in the system is going to treat "foo", " foo ", or "foo" differently, for obvious reasons.
All reputation figures are given as if the daily rep cap did not exist. This let us focus on the potential reputation from Documentation, but naturally the daily rep cap is not being removed.
We’re not making any changes to rep from citation in upvoted answers at this time. Right now there isn’t enough data to justify changes.
Changes to what earns +2 when it is reviewed
Overall this part of the system is mostly working, but we’ve got a couple of changes to address some issues we found.
- If a change rolls a topic back to a previous version, that change will not earn +2 upon approval
- Rollbacks should be proposed and judged independent of rep rewards
- If the user who proposed the change to a topic was also the last editor of that topic, that change will not earn +2 upon approval
- While it is acceptable to make several changes over time, there should be no incentive to break up an edit you could have made in a single change
Changes to who earns rep when an example is upvoted (ie. who is a “substantive” contributor)
This is the largest source of rep right now, and is seeing the most changes accordingly.
- Example creators will continue to earn +5 reputation per upvote
- It makes sense that the first contribution is “substantive”
- A new example must stand on its own, and we see that in how much larger (3x to 4x) they are the the typical edit
- Changes that remove more characters than they introduce will never be considered substantive
- Sometimes examples and topics are improved by removing content, but it both feels unfair to reward it and incentivizes some destructive behaviors
- Changes that add fewer than 20 characters will not be considered substantive
- Most of these changes are minor copy edits, grammar or punctuation fixes, etc. that don’t merit recurring rep rewards
- These changes will still receive a one-time +2 upon approval per the previous section
- Changes of more than 20 characters will be considered substantive, but only earn the editor +1 reputation per upvote (instead of the +5 they get today)
- Around 20 characters, changes start making a real difference in a typical example
- However, a 20-character change doesn't deserve as much reputation as the creator the vast majority of the time
- This does not stack. Making two 20-character edits to an example does not mean you’ll earn +2 from subsequent upvotes
- If a user ends up contributing more than 350 characters to a single example, they will earn +5 reputation per upvote
- Edits this large are almost always major, impactful contributions
- 350 characters is close to (about 3/4ths) the same size of the typical example's first revision
- These characters can be split across multiple changes (though see below)
- This does not stack, no matter how many or what size edits are made to an example; a user can never earn more than +5 per upvote
- For both the +1 and +5 levels...
- Edits smaller than 20 characters are ignored when summing "total characters contributed": you can't change enough punctuation to become "substantive" logically.
- Moving up a level is not retroactive: editors are judged on their contributions at time of upvote, not their total contributions to date
Removing reputation
- In keeping with the current system, deletion of a topic or example will remove all rep awarded from it
- Rollbacks will remove the +2 awarded on approval from the changes that were undone
- Rolled back changes will be removed from “substantive” calculations, and upvote rep adjusted accordingly
As with Q&A, if reputation is live on the site for 60 days then deletions (and rollbacks) will not remove it. This doesn't apply to changes to the rep system, so the re-calc that accompanies this new system will affect all Documentation sourced rep no matter how old.
Notes
- This system will remove about 50% of all Documentation sourced reputation
- The most impacted examples are the giant “Java Arrays”-ish ones, which end up awarding about 70% less rep
- The daily rep cap remains in place for example upvotes
- The +2 rep on approval will not be subject to the daily cap
- We selected the various character limits based on aggregates and hand auditing. Some additional statistics:
- About 20% of all additive edits add less than 20 characters (and will earn no recurring rep)
- About 80% of all additive edits add less than 350 characters (so ~60% will earn +1 recurring rep, and ~20% will earn +5)
It’ll probably take us a week or two to get this system implemented; expect an announcement when the rollout begins.