25

There are quite a few users that are able to review really large volumes of posts. Generally, if I go to Review, the First Posts and Late Answers stacks are empty. If you do get a post to review, you actually need to read through it and potentially perform some actions on it. I don't think that it is possible to achieve the number of volumes that some of the users are able to achieve through this manual process.

Basically, I'm wondering if there are some methods for bulk reviews or if those guys are using robo reviews (bots). Also, are bots allowed?

21
  • 11
    I don't do much reviews in Late Answers, but per my recollection, these are much easier than First Posts. This is because lots of LA are not-an-answer or spam: spot-flag-done cycle is simple and fast
    – gnat
    Jun 30, 2014 at 9:44
  • And bots probably would be alloeed if they are really intelligent. But an usual bot just clicks everytime the same to get the badge easy, so no, it´s not allowed.
    – deviantfan
    Jun 30, 2014 at 10:58
  • 7
    How do you know how many posts they're reviewing?
    – jwodder
    Jun 30, 2014 at 16:06
  • 9
    If someone stays on SO some hours a day and check often the queue it's quite easy to do tens of reviews a day... times 5 years = tons of reviews. If this isn't what you meant then you should probably provide some example of fishy statistics.
    – Bakuriu
    Jun 30, 2014 at 16:11
  • 5
    If you want to find some fishy accounts then wait till midnight (00:00 UTC) where reviewing limits are reset, wait some time and check who first will get its daily limit (some people are able to do it in less than minute which for 20 post limit per day give us speed or reviewing 60sec/20post = 3sec/post)
    – Pshemo
    Jun 30, 2014 at 17:26
  • 4
    Flag a post of theirs with a custom flag and detail the situation clearly, if you've genuinely identified a robo reviewer.
    – Brad Koch
    Jul 1, 2014 at 0:29
  • 3
    Shouldn't those 'no action needed' guys get detected by the little tests from time to time?
    – Theolodis
    Jul 1, 2014 at 8:24
  • @Theolodis Sometimes the audits are so obvious that even they get passed.
    – Sam
    Jul 1, 2014 at 8:34
  • 1
    @BradKoch It seems highly awkward to flag some random post of a user just to accuse them of being a robo-reviewer. The post and all their other contributions could be quite good and are not really related. There doesn't seem to be a way to actually flag a user itself, but maybe there should be. Until then, I won't just be flagging random posts
    – neelsg
    Jul 1, 2014 at 9:55
  • 2
    For newer users to SO, I'd like to add that the current review limits per day are fairly new (circa summer 2012) given the age of the site. I am also a fairly prolific reviewer, and before the current limits were implemented there was no limit on the number of posts you could review per day. This is why there are many users with thousands and thousands of reviews.
    – Luke
    Jul 1, 2014 at 9:58
  • Ok I edited the 'these' link in Petr R's comment. It simply had to be done.
    – ouflak
    Jul 1, 2014 at 10:07
  • 1
    @neelsg: It won't suspend their account just reviewing privileges.. Jul 1, 2014 at 12:44
  • 1
    @PetrR. really need to resolve this. Just an hour back I voted one of those as Unclear, thank you for pointing,I am wondering how to Flag a User instead Post for Moderator attention?
    – swiftBoy
    Jul 1, 2014 at 13:12
  • 2
    @RDC current recommendation at MSE is to flag a post that was abused in review, and explain in the flag message that there seems to be a pattern of a particular reviewer, see Can we have a system to submit potential robo-reviewers for moderators attention?
    – gnat
    Jul 1, 2014 at 13:53
  • 3
    I removed the comments which called out specific users, because we don't want to publicly shame them. If you have problems with specific reviews or a pattern of reviews by users, and you think it's bad enough that we need to step in, flag one of the posts in question and we'll take a look at it. It does take us a while to go through review histories, though, so it may take a bit for us to act on this.
    – Brad Larson Mod
    Jul 1, 2014 at 15:15

3 Answers 3

5

Particularly persistent users might review up to 140 posts per day (40 cv + 20 rv + 20 se + 20 lq + 20 la + 20 fp). Once upon a time there was no limit.

There's not a problem with this unless the user is consistently making poor decisions. If that is the case, flag one of that user's posts with a custom flag, and detail the situation. Example:

Large volume of poor reviews by user_x:
Example 1
Example 2
Example 3
Example 4
Example 5

The likely outcome is that user will get a short review ban; moderators have been using them more frequently these days.

I doubt "flag user" will ever be an option; SE is designed to focus on the content, not the individual.

1
4

I focus more on what could be done against robo reviewing. Possible remedies include:

  1. Increase grace period between two consecutive reviews. If we can do it for voting it should be done for reviewing too because there it is even more important to take your time. 5-10 seconds waiting time sounds like a good idea (no idea how much it is now?). Also one could apply it only to 2/3 questions in a row (meaning a minimal time for reviewing X reviews in a row) allowing a bit of fast clicking but not too much.

  2. Put all with a 99% of the cases "no action needed" (or 5 sigma away from the average "no action needed" rate) on a special robo review suspicous list where moderators check them if they have time.

  3. Even better automatically re-submit reviews of people who voted "no action needed" in more than XX percent for the last YY reviews. Just take some criteria that catches the highly unusual robo reviewers and give these reviews at least a chance of getting a proper review.

  4. Check how often a "no action needed" question/answer is subject to edits/flags/closes afterwards and put the reviewers that have a high percentage of reviews where action was needed but they didn't do an action on a robo review suspicous list where moderators check them if they have time.

Of all these I think the second-reviewing chance for potentially robo-reviewed reviews is probably the best idea.

4
  • There is already a grace period of a few seconds. (The buttons are unavailable for a little while, after a review.)
    – Cerbrus
    Jul 1, 2014 at 11:58
  • 1
    Like @Cerbrus mentioned, there already is a few seconds. The reason people like to review, do stuff on SE, is because they allow multi-taskers, and very computer literate people to work at their own FAST SPEED. Anything that slows them down beyond 5 seconds would cause a very poor user experience and would lead to people not reviewing altogether since people don't like to wait when they already know the action they are going to take.
    – CRABOLO
    Jul 1, 2014 at 13:01
  • 1
    @VotetoClose I really need at least 5 seconds for reading and understanding the post, not alone thinking what needs to be done if anything. This comment suggest that the lowest answer rate is 1 review per 3 sec which is really to fast, even for very computer literate people. But as a compromise one could set the grace period on the average time of several consecutive reviews. Nobody is already knowing the action to all reviews they are going to take. Jul 1, 2014 at 13:42
  • I see many things are already there but obviously seeing the examples above still can be improved. Jul 1, 2014 at 21:26
1

StackExchange has an API that allows you to write back to the network. Using this, a bot could be written to automate several actions on the site. It is not explicitly denied, but the documentation does give very clear warning of what is not allowed from anything automated:

Stack Exchange is very protective of the quality of content on its sites, and will deal harshly with harmful behavior. While it's impossible to list all forbidden behavior, a good rule of thumb is "if a user would be flagged or banned for doing something, your app will as well".

Some examples of write abuse that will be punished:

Spam, of any sort.
"Auto-commenting" based on post heuristics.
Abusively "following and pestering" another user.
Automating flagging or closing based on dumb heuristics.

With that in mind, there are several precedents of bots running various chat rooms, with varying levels of controversy. I don't have examples of automated bots running on the Stack Exchange Q&A portion of the network, but it is entirely possible to do. There is even a suggested-edits API call to see suggested edits in the system. However, I'm not sure if you can approve/reject an edit via the API.

As for how to review so many posts - it just takes time (and accuracy). If you visit the review queue a few times during your visit to Stack Overflow, you'll find the queues constantly filling and emptying. Jump in when there are tasks to do, review a couple items and then return to the site to complete other activities (answering/asking questions). You'll find that you can use your allotment of reviews fairly quickly if you review just a few at a time several times during the day.

1
  • What classifies as "dumb heuristics"?
    – S.S. Anne
    Aug 27, 2019 at 22:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .