46

I was looking at recent changes in the C documentation, and came across the review https://stackoverflow.com/documentation/review/changes/113746?filter-tags=c.

The change was made by someone with a reputation less than 100, and approved by two with reputations in the range 101..200 and one with a reputation in the range 2000..3000. The person submitting the change has created two answers, one with the tag but with zero score; the three people reviewing and approving the change have not got the tag at all.

The change is less than stellar; I plan to reverse it because (to the extent it is comprehensible) I think it is confusing o/s-level memory management with application-level memory management.

My question, though, is:

  • Should there be a way of checking that those approving edits have demonstrated at least some relevant knowledge?

For example, should reviewers have a net positive score (not zero, not negative) in the tag? That would be a simple starting point for the C documentation; I suspect it would work sanely with a majority of other documentation subjects. Is it doable? Does it mean that there should be an identified tag which controls whether you're eligible to review documentation in a particular topic?

13
  • 41
    Edit review is pretty drastically borken, everybody knows it and nobody knows what to do about it. But hey, you saw it anyway and can do something about it. Hallelujah. Go forth. Commented Dec 7, 2016 at 0:18
  • 2
    Closely related: meta.stackoverflow.com/q/258285/1079354
    – Makoto
    Commented Dec 7, 2016 at 0:23
  • @Makoto: That question (as you know) is about reviewing tag wiki edits. It is certainly in the same general area of discussion, but different in detail. It is seeking a higher bar than I'm suggesting — but the different scenarios probably warrant that. (I'm not averse to a higher bar for approving documentation changes; I'm trying to keep it simple.) Commented Dec 7, 2016 at 0:28
  • 2
    I'd say no - there's few enough people going to answer, and in the specific case you raise, while I agree with your revert, I can also see what prompted the edit, I.e., the mention of NULL as a memory address without the clarification that it's an invalid one. Of course, you've no way of judging my creds for this comment as I've not done all that much in the c tag on SO either...
    – Gwyn Evans
    Commented Dec 7, 2016 at 0:48
  • @GwynEvans: You've got a net 12 up-votes in the c tag on SO; I'd be fine with you reviewing C documentation changes (or submitting them). I was fine with the person submitting the change — though I'd've preferred a better change, and would have rejected or improved it had I been one of the reviewers. It was the reviewers who caused me concern. Yes, sometimes someone with no reputation in a tag can do a good review, but I think it was more like robo-reviewing this time. Commented Dec 7, 2016 at 1:04
  • 4
    I've seen questions getting approved in Documentation :-/ Somehow I suspect that the problems with Documentation's reviews are deeper than just a lack of specific domain knowledge... Commented Dec 7, 2016 at 1:55
  • 10
    Part of the reason, @Carpetsmoker, is that we let just about anyone approve Documentation edits. This is in stark contrast to how we handle edit approval elsewhere on the site, and I cannot for the life of me understand the rationale for it. It seems to be either (A) the team wanted to maximize the amount of participation in Documentation to spur interest in a new subsection of the site they were afraid might fail because it never reached critical mass, or (B) they were afraid there would be an insufficient number of high-reputation users to handle the required reviewing. Both are sad.
    – Cody Gray Mod
    Commented Dec 7, 2016 at 8:28
  • @CodyGray: It's sorta strange that you'd assume fear was the motivating factor. We set the privilege level to match editing Community Wiki posts. (This was before we added Proposed Changes to the global review queue.) We'd certainly be open to hearing arguments for changing privilege levels. But I'm not very sympathetic to the "everything is terrible so let's close the city gates" arguments we've been hearing. This is probably unfair of me, but from where I sit these sound an awful lot like FUD.
    – Jon Ericson Staff
    Commented Dec 8, 2016 at 14:27
  • A more fundamental question is, "Should people who've never asked or answered a question for (tag) be allowed to ask or answer a question for (tag)?" With the quality of most 1-rep questions and answers, the answer to that question might appear to be "no," but in fact it is "yes." Commented Dec 9, 2016 at 1:28
  • 1
    @JonEricson "everything is terrible so let's close the city gates" but everything is terrible, and unless you close the gates, how will it get fixed? Making sure that the reviewer is able to do a sensible review is the first step. I proposed this. How about that?
    – Braiam
    Commented Dec 9, 2016 at 17:46
  • @Braiam: Huh. I think the pointer topic is looking pretty good. Yeah, it's kinda annoying that meaningless edit was approved. But that change has long been reverted. That said, I think something like your proposal would be helpful. If we can match reviewers to reviews a bit better, it's bound to increase the efficiency of the queue.
    – Jon Ericson Staff
    Commented Dec 9, 2016 at 18:00
  • @JonEricson my quip was directed at the average quality rather than specific examples, it's too low that even stellar examples don't help (and in some cases those examples were copied from elsewhere, which simply adds an insult to the injury).
    – Braiam
    Commented Dec 10, 2016 at 0:07
  • @JonEricson btw, you'd probably need some virgins for the blood sacrifice, Shog couldn't do it.
    – Braiam
    Commented Dec 14, 2016 at 19:44

3 Answers 3

17

During launch, before the Documentation review queue was implemented, only people who visited or watched the tag were able to find proposed changes. While there was no enforced restriction, generally only the people who had some interest in a tag reviewed changes. The problem was people were waiting hours (or longer) to see their changes approved.

There are several ways to solve this problem:

  1. Exempt high-reputation users from review as we do on Q&A. That would make it easier for many folks to get their changes through, but would do little to change the situation for new users. In fact, it might make things worse as the review system would be out of mind for users who weren't subject to it.

  2. Add proposed changes to the site wide review system. This fixes the problem of changes not getting made, but does mean reviewers could be unfamiliar with the language.

  3. Remove the review requirement for proposed changes. This would be the full wiki model where the site would be self-healing since anyone can fix mistakes.

We more or less picked #2, but we added a little of #1 in the form of expedited reviews for presumed experts:

Proposed changes require 4 votes to be approved, and 4 votes to be rejected. However, some reviewers are given more weight; that is, their reviews count as 2, 3, or 4 votes. This weight is based on reviewers' reputation and tag-badges corresponding to the tag the topic resides in (or one of its aliases):

  • >= 100 reputation: 1 vote
  • >= 1,000 reputation: 2 votes
  • >= 10,000 reputation: 3 votes
  • gold or silver tag-badge: 4 votes

Additionally, changes proposed by editors with applicable gold and silver tag badges are immediately approved without further review, unless the change modifies versions or changes which example is pinned.

So while anyone can review changes, not all reviews are treated equally. If you happen to have a gold or silver badge in a tag, you can immediately reject bad edits. Perhaps more to the point, if you spot a bad edit, you can immediately rollback the change or fix problems with a subsequent edit.

In summary, proposed changes are more quickly reviewed when anyone with 100 or more reputation can approve or reject them. Meanwhile, if some bad changes do make it through review, tag experts can quickly correct the problem.


The assumption in the question is that people familiar with C would reject the edit and that people who don't know C would accept out of ignorance. A further assumption is that answering a question in C is a good proxy for being a competent reviewer of C documentation. I think both of those assumptions are wrong. For one thing, I have a few C answers and I'd probably approve the edit in question. (Or at least I would not question the technical correctness. I would have some questions about the clarity of the addition, but that would be the same whether I understood C pointers or not.)

One of the philosophies that makes Stack Overflow work is user empowerment. If you are a new user and you see an unanswered (or poorly answered) question, you have the power to fix that by answering. If you have a question that isn't represented on the site, you have the power to ask. As you make positive contributions (as judged by other users) you'll earn more privileges to control more aspects of the site. At every level, we want people to feel they have some ability to make things better. If people don't have that feeling of empowerment, they won't bother participating.

Sometimes people abuse privileges and we have to take them away. But the basic assumption is that people will do their best to do the right thing. So when things go wrong on the site, our instinct is to find some way to empower trusted users rather than restrict new users. When it comes to review, I'm not going to spend a lot of time verifying every detail. If an edit seems plausible (and this one is to me), I'll assume the editor knows what's what.

15
  • 4
    Thanks for the explanation and auxilliary information. I don't doubt it is a tricky problem; if it weren't, a solution would've been implemented. It's a difficult balancing act. I guess it is incumbent on those with privileges who care about a tag to keep an eye on the changes. Fortunately, the rate of change on the C documentation has decreased to a point where periodic checking (less than daily) can keep up with the changes. Commented Dec 7, 2016 at 1:15
  • 4
    Re: your addition. Yes, I'm making an assumption that someone has passing knowledge of C (because that's the language in the question; pick your poison for the general case) if they've earned a net positive ranking in the C tag. That's a very poor surrogate for an answer to "does the reviewer understand C and what was written", but it might be better than what we have now. In this case, the Use of English in the change needed help too. I don't suppose anything much will happen, but I live vaguely in hope. I don't know if other reviews by the reviewers show similar signs of robo-reviewing. Commented Dec 7, 2016 at 6:32
  • 2
    With that said, empowered users will just have to keep their eyes open. It might be nice if that vigilance was not as necessary. Commented Dec 7, 2016 at 6:34
  • 12
    "If an edit seems plausible (and this one is to me), I'll assume the editor knows what's what." This seems like the most dangerous assumption that you could possibly make. There is a big difference between assuming "good faith" and assuming competence. I agree with assuming good faith, but I don't agree with assuming absolute accuracy. I'm sure the person who made this edit did so in good faith, as I'm sure that most editors do. The problem is, if we care at all about the usefulness and accuracy of our content (and I think it's a given that we do), we cannot use this simplistic of a metric.
    – Cody Gray Mod
    Commented Dec 7, 2016 at 8:25
  • 2
    @CodyGray: I basically gave up on docs because people were filling up the x86 docs with close-to-accurate stuff faster than I wanted to keep up with fixing it. A lot of the x86 topics omit important points. Apparently the goal of some contributors was "rough idea for total beginners", but even for that goal there are many things that could have been stated better (to avoid being technically wrong without going into whatever detail). Commented Dec 7, 2016 at 21:26
  • 3
    Requiring qualified reviewers would help put a higher quality bar on docs than on answers. Also good for tag wikis. I just really don't understand the SO system of having people review stuff for tags they might not know anything about, except for cases where there's nobody available for those tags. Score in a tag means a lot more than total rep, especially for reviewing docs where changes to other people's posts are not intended to be limited to "non-intrusive" edits for grammar / formatting / minor corrections. Commented Dec 7, 2016 at 21:27
  • The user-empowerment argument is a good one, so probably it's still good to allow doc contributions from users without much score in a tag. But do we need to let such users review, too? I guess maybe the qualified reviewers would get overloaded with work carefully reviewing too many low-quality edits, though, so there's no easy solution if we want to allow users to contribute to docs even for high-traffic tags without having spent some time answering / posting questions first. Commented Dec 7, 2016 at 21:36
  • 4
    @PeterCordes: Reading your two comments back-to-back neatly illustrates the problem. People who are best suited to write documentation tend not to have the time (or, dare I say, patience) to do it. There demonstrably were not enough reviewers for most tags (even the popular ones) before we rolled out global reviews. But if you look at the long game (as I do) there's plenty of opportunity to fix subpar content right now. The x86 tag is not so active that an interested party couldn't keep up with it. Your gold badge will help too.
    – Jon Ericson Staff
    Commented Dec 7, 2016 at 21:36
  • 3
    For me it's definitely an issue of the total scope of what should get written being overwhelming in scope, unlike writing an in-depth answer to a specific question. There's so much to do that one afternoon of work would barely make a dent. Also, I don't know if deleting a lot of "examples" and rewriting them from scratch would be a good thing or not. Commented Dec 7, 2016 at 21:44
  • 2
    Also, the x86 tag wiki already links to a lot of my answers, because I'm the only one who edits it much, and I wrote the answers to be canonical, and because I knew where to find them more easily than other people's answers. If I started "taking over" x86 docs, I worry that people would think I was doing it for self-promotion. (Which is not the case. I'd be happy to link to other people's answers, I just have a much easier time remembering my own answers and adding them to the tag wiki when they've come up a few times as dup targets.) Commented Dec 7, 2016 at 21:49
  • 6
    @PeterCordes Many of your answers are what the docs should aspire to be. I gave up on docs as well after seeing the c-tag brutalized for too long.
    – user3185968
    Commented Dec 7, 2016 at 21:51
  • 1
    @PeterCordes "... I don't know if deleting a lot of "examples" and rewriting them from scratch would be a good thing or not." Well, if the existing examples are bad (the judgement on this may be subjective) then why not, why shouldn't it be a good thing? In order to keep changes in Documentation documented I would do it one by one, but if I could and would be interested in doing it and had the time, I would definitely do it. We shouldn't be shy to delete bad examples. What good would they do if they stayed in? Commented Dec 9, 2016 at 8:04
  • "Perhaps more to the point, if you spot a bad edit, you can immediately rollback the change or fix problems with a subsequent edit.[...] if some bad changes do make it through review, tag experts can quickly correct the problem." I don't know about "quickly" since there aren't great tools for spotting bad edits. The actions/activity section of the tag dashboard doesn't have a "watch" option; and some changes are hidden from there, too meta.stackoverflow.com/q/339055
    – Frank
    Commented Dec 9, 2016 at 17:29
  • 1
    "The problem was people were waiting hours (or longer) to see their changes approved." considering the flip side, I prefer this to the alternative.
    – Braiam
    Commented Dec 9, 2016 at 17:47
  • 100 reputation is way too low for approving. IMO even 1000 rep is too low for approving changes to Documentation. More strictness is required here.
    – Dalija Prasnikar Mod
    Commented Dec 14, 2016 at 20:25
8

The main notion of Jon Ericson's answer was that

..proposed changes are more quickly reviewed when anyone with 100 or more reputation can approve or reject them. Meanwhile, if some bad changes do make it through review, tag experts can quickly correct the problem.

However I wonder if it cannot be further optimized? It might be better to let bad content not even enter Documentation and wasting the effort of experts on cleaning up when at the same time the experts could concentrate on polishing good content instead.

Key performance indicators are:

  • Response time of a review
  • Correctness of a review

Probably both depends on the rep limit (with opposing trends). The 100 rep limit is probably chosen quite arbitrarily.

Proposal: Measure the response time. Adapt the limit dynamically (with a bit of slack to avoid overshooting) to keep it as high as possible (best correctness) while keeping an acceptable response time. This might be better than the arbitrarily chosen fixed 100 limit. Maybe even adapt this to each tag.

What do we think is an acceptable response time? Is an hour okay or should it be mere minutes? It may depend on how much better the reviews get.

From the question:

Is it doable?

I think with sufficient effort it is doable to either require a certain tag score or a dynamic rep limit or even a combination of both. Question is more, if the benefit would be large enough to warrant the effort.

Unlike Q&A where no expert knowledge is required for reviewing (see recent The Community vs. The Domain Expert), for Documentation which is a collaborative effort, expert knowledge is important when reviewing (and we don't really require it currently). And the further Documentation progresses the higher the chance that a change actually decreases quality of the content (see example in question), so we should become more strict over time, either by reviewing more strictly or cleaning up more thoroughly. Whatever way, the onus will be on us.

Summary: We can choose if we want to review more strictly or clean up later but it's not clear if the current 100 rep limit for reviewing is optimal. Let's experiment with the limit and let us adapt it even dynamically (with some damping) to get the best out of it. This require agreeing on an acceptable response time for a review. This means gathering statistics and measuring the response time. We could even experiment with required minimal tag scores as long as the response time doesn't worsen significantly.

9
  • 3
    I was thinking about this very approach after sleeping on the question. I'm really not too concerned with response time as much as proposed changes getting piled up in review. Now that we have audits I can probably test the assumption that higher-reputation users and people who know a tag are more careful. Some of this data is public, but not all of what we'd need. Internally, we have a lot more data, so I'll see what I can come up with.
    – Jon Ericson Staff
    Commented Dec 8, 2016 at 15:04
  • @JonEricson Good to hear that you had the same idea and that you will gather data and work on something along these lines. For example you could try to mutually correlate rep (tag score) of editor, average rep (tag score) of reviewers (weighted by their votes), rejection rate, rollback rate, and voting to extract a possible dependence of review quality on rep or tag score of editor/reviewer. If you can establish that review quality is positively correlated with rep/tag score it's probably easy to get some kind of dynamically adaptive threshold to ensure proposed changes do not get piled up. Commented Dec 9, 2016 at 8:11
  • 3
    I think tag-score is a lot more relevant than total rep. If the review system wanted me to review some changes to C# documentation, my choices would be to skip it, or just check the spelling and grammar and mostly assume that the code was good. Obviously I would skip it, and probably be presented with a Javascript review; another language I know little about and have never used. This is why I don't use the review system; my time is much better spent manually "reviewing" recently-active asm / SIMD / C/C++ / etc. questions than on trying to decide anything about stuff I don't know about. Commented Dec 9, 2016 at 14:21
  • 3
    @Jon One thing I'd worry about with data from Docs review queue is truncation. I can't get a review in edgewise in the R tag since the tag-agnostic queue-sitters move too fast, so people like me won't be represented in the data and might be forgotten when weighing changes to the review system. Anyway, just a thought.
    – Frank
    Commented Dec 9, 2016 at 17:42
  • 1
    @Frank: We don't have enough data for the Docs queue in particular. However, I think we can use Suggested Edit reviews to get at the same sort of insights. I'm looking at that data now. I hope to report some analysis next week.
    – Jon Ericson Staff
    Commented Dec 9, 2016 at 17:48
  • "Unlike Q&A where no expert knowledge is required for reviewing" ugh... and then we complain when they accept seemly small but critical edits that fundamentally change the answer.
    – Braiam
    Commented Dec 9, 2016 at 17:56
  • @Braiam It definitely helps but if you look into the recent discussion and the many discussions linked therein you see that the community consensus seems to be that for reviewing changes in Q&A content knowledge is not strictly required because every change to the content is forbidden anyway. So I guess the requirement is to recognize when content is changed. Not so trivial sometimes. Also, Documentation is a completely different field. There, because it's much more collaborative, content knowledge would be quite important, but we don't really require it with the low rep threshold. Hmmm... Commented Dec 9, 2016 at 22:48
  • @PeterCordes There is some tag filtering in the review queues. The filter function right to the review queue name let's you specify three tags of your choice and Braiam also proposed to automatically propose useful tags for the filter. Commented Dec 9, 2016 at 22:57
  • I wouldn't trust the so called "community consensus" Trilarion. First, it goes against a core principle of SE: collaborative editing. It's so ingrained in the system, that SE basically tell you to "get off" if you don't like your posts being edited. Second, is too volatile. Not even a year ago everyone was agreeing with updating someone else answer. The only dissenting opinion was not popular (not even one upvote)
    – Braiam
    Commented Dec 9, 2016 at 23:54
5

Is this a joke?

So the problem at hand is

waiting hours

gosh .... whole hours. No!

Jon Ericson - OF COURSE you should have to be an expert in your field to review (for hell's sake!) changes on the - definitive! curated! expert! - Documentation system.

When I discuss extremely expert issues on SO, it's utterly normally that the discussion takes days going back and fore, until the QA settles down to a consensus.

Naturally, on the - definitive! curated! expert! - Documentation system, it would take days or weeks for an edit to be reviewed.

This is total madness.

2
  • 1
    You shouldn't have to be an "expert in your field" to review something. Not every piece of code requires an expert's level of experience to understand. It doesn't take an expert to find problems in code or to explain things better. Or to tell if someone doesn't know what they're talking about. Also, nobody ever claimed anything about "definitive! curated! expert!" with regard to Docs.SO. Commented Feb 9, 2017 at 16:32
  • 4
    Indeed. While there are legitimate reasons for worrying about reviewer pools being too small, an unwillingness to wait several hours for your changes to be approved isn't one of them.
    – duplode
    Commented Feb 9, 2017 at 22:37

Not the answer you're looking for? Browse other questions tagged .