70

So if you look at the Docs review history you'll see lots of the following

I'm not judging the reviewer or his reviews (his block was on one page). I only offer it as an example, since there are dozens doing this. 8 minutes. 30 reviews. 15 different (and fairly disparate) Docs Topics. In other review queues this is common, but Docs is a different beast altogether.

Speaking from experience, I've only felt sure enough to approve a handful of things in the Docs queue (I don't know a lot about, say, C++), but there's no shortage of low-rep people who are clearing the queue in one fell swoop. I have a strong hunch there's some serious robo-reviewing going on. A lower limit, based on reputation, might help curtail this.

8
  • 26
    I count 9 (nine) approves in the 18th minute segment. Were all of those such minor edits that they'd take no more than 7 seconds to read, judge, and approve?
    – Jongware
    Sep 19, 2016 at 20:57
  • 17
    An alternate idea would be to limit review votes to users within enough rep in a given tag. For example, I should not ever be approving a proposed change to a haskell topic, since I haven't written a line of haskell in my life and have 0 points in the tag. The more rep you have in a topic, the more votes you are granted.
    – DJMcMayhem
    Sep 19, 2016 at 22:11
  • 1
    Strongly related: meta.stackoverflow.com/questions/334387/…
    – bwoebi
    Sep 19, 2016 at 22:50
  • 2
    @Machavity I've just looked at these specific reviews of this user. The only questionable one is stackoverflow.com/documentation/proposed/changes/… — Apart from that there only was one single change proposal which wasn't a typo or version tag fix. There were like two users mass-proposing change proposals only editing the version. Which caused this to happen today. — The issue exists, but you've chosen a really bad example to illustrate though.
    – bwoebi
    Sep 19, 2016 at 22:56
  • 17
    There's tons of robo-reviewing going on in the docs queue, though. Just the other day I had to rollback the SQL Server Join topic, and one user approved the offending edit and my rollback. It's a bit ridiculous.
    – cteski
    Sep 20, 2016 at 16:48
  • @cteski If anyway everything is accepted, just edit Docs to your liking. As long as the good (knowledgeable) guys contribute the content everything should be fine. Sep 21, 2016 at 8:16
  • 2
    @Trilarion Unfortunately, entropy always increases. Bad content is easier to write than good content. Sep 22, 2016 at 1:21
  • @DJMcMayhem Doesn't quite work out. The only thing I know about JS is that its syntax is similar to C. But since I know 7th grade math, it is my strongest tag. Sep 22, 2016 at 1:23

2 Answers 2

38

I agree this is a problem.

Some of the design aspects were highlighted in other posts as well, with one suggesting that not only the review aspect, but the collaborative edit aspect, was also a factor in this problem.

The official stance is that content creation should be available to anyone and any barrier to that would be a detraction to the feature. I would prefer that contribution be limited to users who hold at least one bronze tag badge (in any tag), but that stance is not shared by the team.

In the near term, this problem is probably not going away. The current plan is to implement review audits (which is said to be coming soon), however until that point I fear we may see more of this type of behavior. What makes it worse is that fixing these inappropriate approvals is not exactly easy.

Perhaps only until review audits are implemented (and not long term - Nicol makes a good point why this is bad long term), the review queue could be limited to users who hold one bronze badge in any tag just to hopefully limit (there may be some correlation, but this is not a silver bullet) the breadth of these robo reviews.

21
  • 5
    If a bad edit is approved, roll it back. That removes the author as a contributor, and nullifies the rep. Folks have been asking for this for suggested edits for years, and now it exists - if it doesn't actually work, it'd be nice to know that.
    – Shog9
    Sep 20, 2016 at 2:32
  • 3
    @Shog9 I'm guessing there isn't (at least currently) anything that will punish either the user who submitted the edit or the reviewer when they are rolled back multiple times? For example, banning from reviews or submitting further edits.
    – DavidG
    Sep 20, 2016 at 16:05
  • 6
    @Shog9 Rollbacks are useful, but surely it would be better to stop the bad approvals before they happen, rather than hoping a knowledgeable user looks through the history and takes action? Hopefully, audits will fix this - it's certainly preferable to have the system reviewing the reviewers, rather than asking for another layer of expert volunteers.
    – user812786
    Sep 20, 2016 at 16:47
  • 13
    @Shog9 - It would be nice if I was alerted every time one of these needs to be rolled back. Unfortunately that isn't happening for some reason, and the only way to see if there was robo reviewed content edited into docs is on accident for the most part. Worst case scenario: someone contributed a worthwhile improvement after a robo review introduces inaccuracy or mangles the example. As a result of this situation, I do not find it easy to fix these edits because of the difficulty involved in both noticing them, and also if there were complicating factors.
    – Travis J
    Sep 20, 2016 at 18:06
  • 4
    @Shog9 "If a bad edit is approved, roll it back." This could potentially result in a lot of rolling forth and back. Quality assurance on the suggested edit approval would be an important step to avoid that. Sep 20, 2016 at 20:34
  • 1
    In an ideal world, no one ever suggests edits that aren't universally loved, @Trilarion. This is not that world, hence the very existence of reviewers. In a nearly-ideal world, reviewers never approve anything that isn't universally acclaimed... This is not a nearly-ideal world either, hence the existence of rollback. Let us learn from past experience here: we're dealing with a wiki, and it is a given that wikis will be edited incorrectly... Therefore, if we wish to have a wiki we must design tools that allow corrections to be made efficiently.
    – Shog9
    Sep 20, 2016 at 20:57
  • Let's figure out why notifications aren't working first then, eh @Travis?
    – Shog9
    Sep 20, 2016 at 21:00
  • 2
    @Shog9 - Sorry, I think I unintentionally misled you. I was basically trying to say that "there is no way to know when a poor edit was approved automatically", and if there were the system would probably just prevent it in the first place. In other words, the only way that I can determine an edit that should not have been approved got approved would be manual inspection. The manual inspection requirement, paired with the difficulty of any changes that had transpired afterwards, is what makes it hard to detect and fix these.
    – Travis J
    Sep 20, 2016 at 21:06
  • You can opt into immediate notification of proposed changes if you wish, @Travis. That feature exists, I thought you were saying it was unreliable. Granted, I wouldn't recommend subscribing to these for every tag in the system, but then again I hardly think you're qualified to pass judgement on edits in all tags either. So, pick the tag(s) you care deeply about and are able to make good decisions in, and monitor edits there. Wanna play gatekeeper for jquery? That's your choice.
    – Shog9
    Sep 20, 2016 at 21:08
  • 4
    Yeah, I was not trying to say that the notification system was unreliable. Just that the system where users review this type of junk and approve it is unreliable, and needs some sort of stop gap. It is disheartening to see, and it takes precious time away from creating content to fix.
    – Travis J
    Sep 20, 2016 at 21:35
  • 2
    @Shog9 - I have subscribed to "immediately" for the proposed changes to JavaScript to see how that works. However, I cannot shake the feeling that I (and others in this role) am now somehow tasked with reviewing the review queue. Personally, I would rather that the review queue have something in place to help with quality control prior to the audits (which I have heard are going to be complicated to generate at first).
    – Travis J
    Sep 20, 2016 at 21:37
  • 2
    Yeah, I'd say some non-trivial amount of such approvals should remove your ability to review for a while, @Travis.
    – Shog9
    Sep 20, 2016 at 22:06
  • 1
    That particular user has rather a lot of history, @Travis.
    – Shog9
    Sep 20, 2016 at 22:15
  • 3
    @JF - Thanks for the correction. Yikes! Perhaps 100 is a little low.
    – Travis J
    Sep 20, 2016 at 23:46
  • 2
    @JF Maybe the low reputation threshold of 100 for reviewing Docs is the reason low rep users review particularly enthusiastic and also make more errors. Having then to roll the edits back and forth by experts to undo all these errors might be a big waste of effort. But for the time the idea seems to be that quantity trumps quality. Apart from this I'm all for more efficient tools for rolling back and forth edits. Sep 21, 2016 at 8:15
6

Agreed. Tag score and tag badges are supposed to be indicative of a user's knowledge in that category, so review power should be weighted based on them, with new privileges unlocked at higher levels.

In this regard, I would like to suggest that each person has their review power for a given documentation category determined based on a combination of their tag badges and tag score for that category, weighted towards tag badges.

  • If your tag score is halfway towards the next tag badge, then you get 1 point. [If you have a gold tag badge, this point is instead awarded based on a theoretical "platinum" badge, which has requirements equal to 10x the requirements for a silver badge (paralleling how a gold badge's requirements are 10x a bronze badge's requirement).] [I'm unsure whether this should take the number of answers posted by the user into account as well; if so, it most likely needs tweaking.]
    • No tag badges: This point is awarded for having a tag score of 50+.
    • Bronze tag badge: This point is awarded for having a tag score of 200+.
    • Silver tag badge: This point is awarded for having a tag score of 500+.
    • Gold tag badge: This point is awarded for having a tag score of 2000+.
  • Each tag badge is worth 2 points, cumulatively.
    • Bronze badge: 2 points.
    • Silver badge: 4 points.
    • Gold badge: 6 points.

Following this system, each user would have a review power in the range of 0..7, representing their knowledge of the category in question. Furthermore, to parallel gold badges allowing a user to single-handedly close questions, an edit would require a total review score of +7 to approve and -7 to reject.

In addition to being used to determine if an edit should be approved, review power in a category would also grant a user privileges for that category. The ones that come to mind are:

  • 0: No privileges, cannot vote on edits.
  • 1: Can vote on edits; able to cast 10? votes per day.
  • 4: Voting limit removed.
  • 7: Can single-handedly approve or reject edits.

Examples:

  • I would have a review power of 1 in c++ documentation (tag score 64 [+1], as of this post), and 0 in other documentation categories (second-highest language tag score is 9, for c).
  • The question's OP, Machavity, would have a review power of 6 in php documentation (gold badge [+2 x 3], tag score <2000), and 3 in mysql (bronze badge [+2], tag score 250 [+1]), going by their top tags.

I'm not sure if this is a good idea or not, but it makes sense to me. Each user's ability to review should be based on the knowledge they have demonstrated in that category, as should the value placed on their reviews.

Not the answer you're looking for? Browse other questions tagged .