70

TL;DR: We need measures against mass-robo-reviewing since the Stack Overflow Documentation Review queue™ has been introduced.

This is a follow-up to What can we do about robo-reviewing on documentation?

General issues

Introduction

I hadn't really been able to track this on a global scale (i.e. not only in a specific tag), but yesterday the review queues have been introduced, causing a huge surge in reviewers and probably also robo-reviewers. It also provides (for 10k+ users currently; users with less reputation will only see their own reviews there) a history which easily shows many reviews by a single user in a very short time frame, making it easy to spot trivial click-through robo-reviewing.

There have been hundreds of reviews alone yesterday which were blatantly robo-reviewing.

I flagged nine people yesterday, just looking at the review history from 19:00 to 24:00 UTC. They all showed only one to six seconds delay between their reviews - I do look at these changes, they're not only typo fixes where I could believe a six second delay, but full-blown edits.

There also have been people with longer delays, but approving very large code additions within 20 seconds, e.g. [just a random example spotted in recent review history - I do not know whether Approve would have been the correct choice or not - point is, the review must have been superficial]. Could you really review, check for correctness and plagiarism in these 20 seconds? I notice these reviewers sometimes rejecting the change proposals then usually very obviously bad/defacing the example or such.

Extrapolating this to the full review history, they probably make up 25%-40% of all the recent reviews since review queue addition, invariably leading to many change proposals being only reviewed by robo-reviewers(!!!). That's totally defeating the whole purpose of a review queue, which is preventing bad content from entering and then going unnoticed.

Example: This change has been reviewed by three people who were all doing their reviews in something less than eight seconds between their reviews (as obvious from their Documentation tab) - the lone rejecting person was quickly overruled. Good that the tag is more active and there are people looking at the approved and rejected history and it could be quickly rolled back but many tags are less active and robo-reviewed changes will definitely slip through, unnoticed!

User-moderating issues

Flagging people which we notice robo-reviewing is quite clumsy to have to first open a question/answer of them, flag it, copying a boilerplate like:

Using {question/answer} to flag irregular behavior of user: https://stackoverflow.com/users/{id}/?tab=documentation shows the user being serially reviewing larger change requests in documentation, with sometimes only {X} seconds between each review. (e.g. {Y} sec for stackoverflow.com/documentation/proposed/changes/{proposal-id}) Could you please remind him that this isn't acceptable and defeating the purpose of review?

This is really making it relatively time-consuming to report people - each offender takes me two to three minutes from noticing to submitting the flag.

Additionally, even if they are caught, I cannot, due to lack of domain-specific knowledge, re-review all their reviewed submissions and assess the actual quality of the submissions (except for blatant issues). These robo-reviewed change proposals need to be re-evaluated for their quality after it's obvious that the reviewers haven't done their job correctly.

Banning issues

For repeated offenders, apart from developers (?), moderators do not have the ability to give doc-review bans yet, only full-site suspensions. While this may be appropriate in some cases, it probably usually isn't.

Seems to be already possible as per a comment from Brad Larson. [Another moderator I've asked yesterday seemed not to be aware of it being possible.]

Proposals

Automated review audits

Use automated audits like other review queues do, which can be seeded from unilaterally approved or rejected change proposals by at least two people [gold-badge holders also may be sometimes wrong and as they only require a single review, better don't use singlehandedly reviewed proposals as audit source], and will cover a large range of rejection reasons - from blatant mistakes and plagiarism to domain-specific issues. Just skip those which have been rejected as duplicates since the duplicate may subsequently have been removed.

Yes, domain-specific issues also - it is not shameful to skip and that shall be encouraged!

Due to Documentation reviewers not being perfect and it being sometimes possible, I suggest a relatively large margin of failure [like allowing one failure per three to four audits or so, in general] (but not too infrequent audits, like on average every few submissions; for people shown usually to be correct, audit count can be reduced) before banning people automatically.

Give moderators the ability to doc-review ban people

In case the audits don't catch everyone or people circumventing them with directly going to the tag review queue, they shall be able to issue temporary bans for documentation reviewing.

Also, in case some people have very bad luck with their audits and just getting bad audits which really should have received the opposite handling, there shall be a possibility for them to reset audit bans.

Putting change proposals reviewed by banned people back to queue

Now, if a person has been Doc-review banned, there now probably are a bunch of change proposals which never should have been approved or rejected.

Put them back into the queue.

For rejected changes (e.g. people spamming "too specific" - the most trivial rejection reason not needing further explanation) just re-propose the change automatically, if there would be no conflict.

For approved changes, in case there is a conflict-free rollback (ideally after applying a three-way merge - if it's not possible to be that fine grained, perhaps resort to using improvement requests by the Community user? - for example, have the Community user propose that rollback). If the approval was fine, people will just reject that rollback.

Perhaps, to avoid unnecessary spam in the review queue, only put the changes back to review if there was only zero or one further reviewer with the same action, who has not been Doc-banned sometime after the approval and the change proposal ended up with the opposite handling than the now banned reviewer did. [E.g. request rejected by the robo-reviewer and approved by a gold-badge holder - no action. Request approved by the robo-reviewer and one person with 10k rep - put back into queue.]

Flagging reviews directly (by user)

Users should be able more easily to report irregular reviewing behavior by just having e.g. a flag symbol inside the change proposal views next to the usernames of the reviewers - moderators then shall be able to mark the flags as helpful, ask for further clarification or just decline them.

Conclusion

Status quo is not acceptable and absolutely needs to be acted on.

I have a few proposals above; I do not see it as critical to include them the exact way I proposed, but some variation of what I proposed would be appreciated in order to ensure a high quality standard for Documentation - it is annoying when answers are incorrect, but one should be largely able to trust correctness of Documentation; sure not everything can be correct, but we shall strive to ensuring as much correctness as possible.

22
  • 6
    I've been reviewing my rejects since the start of documentation. I'm glad when I noticed last night that the queue was out I started checking more often, since I knew there were two regex reviews that should be rejected. Knowing inappropriate changes were less likely to be approved would help me ease off my checks for sure.
    – Kendra
    Commented Sep 12, 2016 at 15:56
  • 1
    @Kendra I do too - not specifically my own rejects, but the latest approvals in my tag (php) in general. And maybe you noticed, I've ironically chosen exactly one of your regex rejections which ended up approved as an example here ;-)
    – bwoebi
    Commented Sep 12, 2016 at 16:00
  • Oh, hey, that's me in the second link (first example). I actually regret making some of my choices. Won't be doing Doc suggestions anymore, not for me. Commented Sep 12, 2016 at 16:40
  • 1
    @EdvinTenovimas If you learned from the mention here, great. :-) But not everyone does. I recommend you check your review history out and look whether you've done some bad reviews and eventually propose a rollback then.
    – bwoebi
    Commented Sep 12, 2016 at 16:44
  • 1
    @EdvinTenovimas click on the title of the topic, click the linked "edited" on the bottom of the topic page, click view topic on the previous revision and then "rollback to this version". — Yes, it isn't. It's not the suggested edits queue. It's large edits with much content which may have more hidden issues. It's documentation.
    – bwoebi
    Commented Sep 12, 2016 at 16:49
  • 5
    Per "moderators do not have the ability to give doc-review bans yet", undo just tested this and it looks like applying a manual review ban now also bans people from reviewing in the new Documentation queue. That might make it a little easier for us to deal with this when we see it, but audits, etc. are still needed. Commented Sep 12, 2016 at 17:07
  • 8
    A lot of these 'reviews' are garbage; people are approving duplicate sections, poor-quality examples and don't seem to notice if there are unaddressed comments on the review. The new queue succeeded in getting them through fast, but I really think more careful reviews from subject experts would be better.
    – Aurora0001
    Commented Sep 12, 2016 at 17:17
  • 3
    @Aurora0001 didn't you found people approving blatant plagiarisms and stuff that isn't documentation?
    – Braiam
    Commented Sep 12, 2016 at 17:27
  • 2
    @Braiam, I've not seen that before. Incredible that people don't spot the question being added. Spotting plagiarism is often a bit harder but reviewers could at least try a Google search with a quote of it. Just shows how bad these robo-reviewers are (a robot would do a better job!)
    – Aurora0001
    Commented Sep 12, 2016 at 17:31
  • 4
    @Braiam - ...and they're banned. Anyone approves stuff like this, flag us down and we'll ban them from review now that these bans apply to Documentation. Commented Sep 12, 2016 at 18:01
  • 1
    @Braiam That's why I've flagged a random question of them (on Q&A) and add a link to it (like I've described in the question text here). The mods then can definitely see and act on them.
    – bwoebi
    Commented Sep 12, 2016 at 18:10
  • 1
    @Braiam - I mean flags on the main site. People have been using those to identify plagiarists in Documentation, and if you find truly terrible reviews (like approving questions asked in topics) a main site flag on one of their posts is probably warranted. So far, each of the bad reviewers I've found on Documentation has been banned from review multiple times on the main site, so bad reviewers are taking advantage of the lack of audits to approve everything blindly in Documentation. We need to stop this somehow. Commented Sep 12, 2016 at 18:14
  • 2
    @BradLarson Longer review bans. You say they've all been review-banned multiple times, so maybe we need 6-month review bans like we have 6 month question bans. Also, we need a way to get robo-review flags handled faster. I know the mods are working as hard as they can, but custom mod flags about robo-reviewing can linger for days or even weeks. Commented Sep 13, 2016 at 10:12
  • 4
    @S.L.Barth - Longer review ban durations for repeat offenders (several of the Documentation reviewers I've banned have been banned over 10 times on SO) would be nice, wouldn't they? As for the flags, they take a while to handle because it takes a lot of effort for a moderator to dig into review histories, the manual banning workflow is difficult, and these flags haven't had the greatest history of being correct. When triaging flags, there are often many other issues that are easier to act on. Commented Sep 13, 2016 at 14:31
  • 1
    Audit idea: add a topic/example that is from a SO question (i.e. asking questions in Docs).
    – Jed Fox
    Commented Sep 15, 2016 at 15:32

3 Answers 3

13

The proposals in the OP all address "Who will review the reviewers?" A complementary question is "What should the reviewers review?"

Here's a related post:

My conclusion is that the documentation review queue is encouraging people to review edits outside their domain of competence, and this may lead to a quality decrease, especially in topics which have lower participation. -- Zimm i48, Documentation review may be encouraging wrong behavior

My suggestion is:

When an edit is submitted, the editor can uncheck a box "domain-specific." If the box is checked, then it only appears in the queue for reviewers using that tag's filter.

Essentially, the editor can opt into the fast-lane, with reviewers who don't know about filters or don't use them. This fast lane may be appropriate if they're just fixing a typo or something else superficial. Of course, only experienced editors would realize that they could do this, so most edits by new users (superficial or not) would land in the "domain-specific" part of the queue, hopefully getting reviewed primarily by people with an interest or expertise in the tag.

4
  • 7
    Wouldn't be appropriate, just for documentation, that an user is not allowed to review content on which he has no tag? It already happens about suggesting a new topic, so maybe the content would be reviewed by somebody acquainted with the documentation context
    – abarisone
    Commented Sep 13, 2016 at 7:39
  • Maybe expand it to having a score in the tag or a reasonably high rep (10/20k)? Some tags are rather dead if you only count those with score in the tag (but then again, do they need docs?)
    – Bart
    Commented Sep 13, 2016 at 8:00
  • 1
    @abarisone Yeah, it's clear we do not need reviewers like those we're getting now. Someone handled my improvement request and got it approved within five minutes (by reviewers who don't use the tag at all). Slowing that down and improving the quality of the review would be fine by me. I don't want to cut off competent people who want to contribute to Docs but aren't interested in Q&A, though, which is where my suggestion is coming from. If we can just hide the edits from the robos, maybe it will be enough. And the robos can be left to handle the mundane stuff (like typos).
    – Frank
    Commented Sep 13, 2016 at 11:52
  • 1
    @abarisone eh, kinda. the problem then comes when there's noone (or very few) with said tag that are interested in reviewing, which is what the review queue aimed to solve (there were cases of pending changes being in limbo for a month or more)
    – Kevin B
    Commented Sep 13, 2016 at 21:02
8

Another idea: The Approving Edits Workflow[TM]

Reviewing Docs content has multiple aspects like defined in this answer:

  • Correctness
  • Completeness
  • Plagiarism (make sure this isn't a thing)
  • Spelling/Grammar
  • Topicality (is a part of the topic)

From this, Spelling/Grammar, and Plagiarism could be checked by almost anyone, while Correctness, Completeness and Topicality are more advanced aspects.

So my proposal is that we should have multiple queues for the Documentation edits, chained one after the other:

1. Basic queue

  • Lower requirements, e.g. anyone with at least 500 rep (access to review queues) could review them
  • Goal: prescreening, lowering the amount of posts requiring professional approach
  • Specific question to be voted for: is the quality (spelling, grammar, formatting) of this proposed edit OK on a basic level, and is this not plagiarism?
  • If clearly accepted: -> queue #2
  • If clearly rejected: edit rejected
  • Both Accept and Reject votes: -> queue #3

2. Advanced queue

  • Higher, but moderate requirements, e.g. anyone with at least 1000 rep (established users) could review them
  • Goal: check plagiarism and above
  • If clearly accepted in queue #1 and #2 -> ACCEPT
  • If clearly rejected on this level: edit rejected
  • Mixed bag: go to queue #3

3. Controversial queue

  • High requirements, e.g. tag silver, or "tag bronze and 5k+" users
  • Goal: professional review of problematic cases
  • Not all edits would make it till here

Important: somebody who reviewed a change in any of the queues, should not be able to review it in any other queue.

So the workflow would involve either queues 1 and 2 if all is straight, and would involve queue 3 if there was any controversy.

This is a significantly different answer than my first one, this is why I didn't refine the first one.

4

Could this work?

If a proposed change is controversial (e.g. has both accept and reject vote(s) too, or has a discussion in the comments) then that change should be visible only for eligible people based on some kind of benchmark:

Tag score

This is problematic for low activity tags: there might not even exist such a person to move the request it forward.

General score

Problematic too - a change might seem decent to someone with a high rep in a different domain, but be factually incorrect.

Maybe the best would be some kind of a mixed logic:

  • Tag gold OR
  • (Tag silver AND 5k+) OR
  • (Tag bronze AND 10k+) OR
  • Jon Skeet

Stats for these thresholds:

  • Jon Skeet tags: 462
  • Dead tags (<5 people): 2898
  • Barely alive tags (5-10 people): 396
  • Live tags (>10 people): 598

You can check which tags would be in trouble here on Stackexchange query Here is the aggregation query feel free to play with it.

EDIT:

With the current review queue, most changes are reviewed in less than 10 minutes, before any domain expert even has a chance to see it and disagree...

Probably there should be a delay for the non-eligible folks to enable experts to see and vote posts earlier - like 1 hour, or even a day... This way, if no expert rejected the post, the non-heavyweight users can decide if it stays or goes. (I agree, now that I wrote it, it is less appealing than it was to me before...)

(My evil twin living inside me has thought of something that is so evil that I can't even... If someone reviews too quickly (e.g. <5s and the like), we should not stop the wrongdoer: just serve the next review up the queue without warning - but the vote should not be saved. It would take the perp quite some time to realize the situation, and to see that the efforts being put into wrongdoing don't bring their reward. The sad thing is that this would create an endless stream of Meta questions about "I clicked 1000 times and didn't get a badge" so this won't be implemented I think)

4
  • 5
    The largest issue here is getting the change marked as controversial in the first place. With the current review queue, most changes are reviewed in less than 10 minutes, before any domain expert even has a chance to see it and disagree...
    – bwoebi
    Commented Sep 13, 2016 at 20:49
  • @bwoebi Yep, I agree, but there were a lot of cases when there was a discussion below the change (added this to post), or an already present reject vote... Somehow a delay would be appreciable in the review process (like people who are not eligible by the above criteria could only see an edit after an hour?), but I have no strong opinion on how...
    – ppeterka
    Commented Sep 13, 2016 at 20:57
  • 2
    That would have been helpful a while ago, before review queue… With the review queue I had it twice that the change was approved while I was typing out my comment. That was frustrating…
    – bwoebi
    Commented Sep 13, 2016 at 21:22
  • Oh, didn't see the edit… yeah, could work then…
    – bwoebi
    Commented Sep 13, 2016 at 21:46

Not the answer you're looking for? Browse other questions tagged .