As part of the Documentation Beta, we’re going to be doing a series of meta posts (and blog posts in all likelihood) with what we’ve learned, and what we’re planning to change accordingly. See the next post in the series.
These posts will normally be about once a week, but since we just launched last Thursday we’ve already got lots to share.
What we’ve learned
Ho-boy, lots of contributions. We got more activity in 1 day on Stack Overflow than we did in the entirety of the private beta. Lots and lots of bugs that didn’t come up in our (months of) private testing. Accordingly, since Thursday we’ve been focused on bugs.
Reputation needs work. Adam already posted some tweaks, and we’ll have more in the future. Again we tested in the private beta, and did some additional offline modeling. The basic model (rep for example upvotes, approved changes, and upvoting answers citing docs) doesn’t seem terribly broken - but the limits (like who gets rep for examples, and how much) definitely need work. Expect more reputation tweaks in the coming weeks.
There’s a backlog of about 2,000 proposed changes (vs. something like 22,000 total changes reviewed). Some of this is too much content from over-enthusiastic users, some of it is presentation in the UX having too much friction, and some of it is the system limiting trusted users more than it should. There are planned changes below to address this.
Some topics have way too many examples. Part of this is because many folks are just trying it out, and adding examples is easy. Part of it is that the “expected number” of examples isn’t well indicated. There’s a planned change below to address this. We will also consider capping examples per topic if this continues to be an issue.
We haven’t quite figured out how to allocate numbers of outstanding changes a user should be allowed to have, and number of reviews a user should be allowed to do. For now we have simple daily caps. We’ll keep an eye on rejection rates, and what gets approved, as the “launch bounce” dies down and tweak accordingly.
Planned changes
- A review queue for Proposed Changes. We were hesitant to do this while Documentation hadn’t been merged into Stack Overflow, and uncertain that such a queue would be necessary. A few days of public beta have shown us otherwise, so this is now on the roadmap.
Here’s a (very early, very rough) mockup of the new queue.
Tag merges and aliases for Documentation. We will be adding a mapping from related tags to “master” tags (like python-2.7 and python-3.x are related to the “master” python). Such a mapping will migrate existing topics to the master tag and prevent re-opening the aliased tag for Documentation. Initially we (Stack Overflow employees) will be doing these by hand; it may be opened to moderators and high rep users up in the future.
A short, optional, “Focus” section at the top of each topic - sort of like what was suggested in this question. This section will be used to define the scope of a Topic and should guide contributors with respect to what sorts of examples belong on a topic. It is explicitly not a summary of the topic, but it will help readers get the gist of what they’ll find.