36

Moderator elections are near again.

Moderators need to do a lot of reviewing. We require moderator candidates to have a minimum of rep and several badges, but we don't seem to have requirements on a moderator's reviewing history.

We could require the moderator candidates to have at least some "Reviewer" or "Steward" badges. But these badges only tell us something about the quantity of their reviewing, not about the quality.

We do get a link to the review history of each candidate. But that doesn't prevent sloppy/inactive reviewers from nominating themselves. It only helps us to assess their reviewing work. And to do this assessment, we have to dig through a lot of data.

I'd like to see moderator candidates who, in the recent past, have shown outstanding review work both in quality and quantity.

I don't know which criteria we should have precisely. I imagine something like "having at least 500 reviews over the past year", and not being review-banned.
However, if the candidate was review-banned due to bad audits or a moderator mistake, that should not be held against them. (I know one particularly good reviewer who was accidentally review-banned due to a mod mistake; when this came to light, the moderator lifted the ban.)

So the question is, please require moderator candidates to have a recent history of active, high-quality reviewing.
And, secondary, what criteria could we use for this?

27
  • 30
    This may exclude moderator candidates who are "conscientious objectors" to the review queues in their current state. Is that a hidden benefit for you, or a disadvantage?
    – Cody Gray Mod
    Commented Jul 11, 2017 at 11:12
  • 7
    @CodyGray Hm, good point (as usual). On the one hand, moderators don't make the policy, they have to work with the tools given. So this principled stance could be a hindrance in their work. On the other hand, this hypothethical candidate understands the issues surrounding the review queues. And moderation isn't just about reviewing. I'm inclined to think it's a hidden benefit, but it comes at a price. Commented Jul 11, 2017 at 11:28
  • Why not extend this requirement to flags too? IIRC, there is no hard requirement for flagging for candidates.
    – Magisch
    Commented Jul 11, 2017 at 11:34
  • @Magisch Last year, they were required to have the Deputy badge; so to have raised at least 80 helpful flags. We should definitely keep that requirement. Commented Jul 11, 2017 at 11:36
  • 2
    @S.L.Barth Right. But last year we also had candidates with less then 100 flags raised on this site and less then 50 reviews that year. And they still got a sizeable chunk of votes. So maybe you're not representing the electorate here in caring about this at all.
    – Magisch
    Commented Jul 11, 2017 at 11:38
  • If I had to make requirements I'd say at least 500 flags raised in the last 365 days, 95% of which need to have been helpful, at least 500 reviews in the last 365 days and never in these days been manually review banned for roboing, but the regular meta users are a small part of the electorate, so our opinions here might not actually be that pertinent.
    – Magisch
    Commented Jul 11, 2017 at 11:39
  • 1
    @Magisch I'm going to be blunt here. My impression last year was that this "sizeable chunk of votes" came from people who treated the moderator election as a popularity contest. Commented Jul 11, 2017 at 11:46
  • 12
    @S.L.Barth So, do you think that's disallowed? Elections are always popularity contests to a big extent. That is not unintended. If there are more people voting for whose avatar they like best, then these people get to fill the positions. That is the beauty (and sadness) of direct democracy.
    – Magisch
    Commented Jul 11, 2017 at 11:47
  • @S.L.Barth Remember that people who care about the nitty gritty of the site and stuff like review stats probably make out VERY little of the electorate. What influences people is (in my observations) in that order: reputation, candidate score, catchy summary in the nominating post.
    – Magisch
    Commented Jul 11, 2017 at 11:49
  • 2
    @Magisch The process isn't entirely democratic. It's a mix of democracy and meritocracy. We have already established that candidates need certain requirements - badges, points, not being suspended in the last year. And we don't always require the same badges, IIRC.Nor can every member vote, that too requires a certain amount of points. So, making demands on our candidates is in line with the things we already do. Commented Jul 11, 2017 at 11:51
  • 9
    There is a candidate score which does include a few review badges in order to reach the max score, so we are already including review details for the candidates.
    – Taryn
    Commented Jul 11, 2017 at 12:17
  • 15
    I'd like to see moderator candidates who, in the recent past, have shown outstanding review work both in quality and quantity. then check out the candidates as they nominate themselves and leave comments highlighting their track records in that field. Leave it up to people to decide how important this is to them rather than add yet another requirement that not every great moderator candidate may meet.
    – Pekka
    Commented Jul 11, 2017 at 12:34
  • 3
    In my eyes, a moderator is somebody who has good judgement. While queue work is good, we can't place too much emphasis on it. A good moderator does more than just the queue, they handle flags and situations (sometimes controversial). They need to be people people who value one GOOD handled situation over 100 meh handled situations. Commented Jul 11, 2017 at 14:46
  • 2
    as of now amount of reviews seems to more reflect whether user managed to discover carefully hidden productivity features or not (skip and filtering). Unfortunately.
    – gnat
    Commented Jul 11, 2017 at 15:58
  • 1
    @bluefeet - I believe that all pertinent badges are quantitative while OP is definitely looking for some guarantee of review quality.
    – PM 77-1
    Commented Jul 12, 2017 at 13:52

2 Answers 2

19

When you're listed as a candidate there's ample links and data summed up about you. You can see this on the 2016 voting page

Last year we had a candidate who had very few reviews and people took notice. Anyone can use the reviews link to find that out. I noted that person got very few votes overall.

In other words, you can do your own homework on this. Their record is there. Someone who does few reviews will stand out.

I'd favor replacing reputation as a part of the candidate score (reputation isn't the greatest metric for who will make a good mod, case in point), with a simple review stats metric (X reviews total, Y in the past 30 days). A major part of this job is just showing up and reviewing.

13
  • 8
    The candidate you were referring to, pushed another user out of the primary. That other user was a very active reviewer, and also had a much better grasp of the English language, and a far more active Meta participation. I daresay that other user was a far more suitable candidate. I can (and will) do my own homework on this - but a number of voters will not. Commented Jul 11, 2017 at 12:58
  • @S.L.Barth There's limits to how much work we can do for the voter. And, yes, there were people pushed out whom were better reviewers but had less reputation. Some people will not vote for someone with <20k, which is sad, but the info block skews that way
    – Machavity Mod
    Commented Jul 11, 2017 at 13:04
  • Low Reputation's a reasonable proxy for users who're new or have relatively low activity rates on the site. While I don't use an arbitrary cutoff (and "low rep" varies from site to site); it is a flag I use when filtering candidates who IMO probably shouldn't win. I generally see it as secondary to moderation activity though, since working the review queues and flagging bad posts are the closest approximations to moderator activity available to normal users. Commented Jul 11, 2017 at 15:03
  • 4
    @DanNeely While I can understand that, it also is a bit arbitrary. For instance, ArtOfCode ran last year but his rep is deceptively low, yet his network profile is impressive
    – Machavity Mod
    Commented Jul 11, 2017 at 15:17
  • @Machavity Why bother to blackout the user info but not other uniquely identifiable info and then also point us to where you found it? :-P
    – TylerH
    Commented Jul 11, 2017 at 18:21
  • 2
    @TylerH If you want to use your Mystery Machine skills to figure out who that is I won't stop your meddling :P
    – Machavity Mod
    Commented Jul 11, 2017 at 18:49
  • @Machavity Oh, I already did, no worries!
    – TylerH
    Commented Jul 11, 2017 at 18:50
  • I'm a little confused about your link to Undo's profile. He has a high reputation. Or are you calling him a bad mod? Commented Jul 11, 2017 at 23:13
  • 3
    @DavidB Quite the opposite. Undo has the least reputation of any currently serving mod, but you'd be hard pressed to call him a bad mod
    – Machavity Mod
    Commented Jul 11, 2017 at 23:34
  • 2
    still it's above 20K, so not really a useful example, lets elect someone with 4K and see how they manage, all in the name of science. Commented Jul 11, 2017 at 23:36
  • @PetterFriberg IIRC Undo was at 16K when they ran for mod. Commented Jul 12, 2017 at 7:39
  • @S.L.Barth Yep, that's why he fell down to the 5th spot, else he'd have been at the top. :( Commented Jul 12, 2017 at 10:02
  • I like the idea of replacing reputation with a review stats metric, that is proposed in this answer. My only concern is that it would make robo-reviewers stand out positively. Commented Jul 12, 2017 at 13:43
12

I think we can all agree that we should elect moderators who do a good job reviewing. However, I think that finding an objective criteria for that is going to be exceptionally difficult and maybe not worth the effort.

How do you measure someone's review performance? Certainly not by the number of reviews, as then quality isn't a factor at all. Audits? Good to rule out people, but to rule them in? Doubtful. Anyone paying even a little bit of attention can spot them.

If you want to get an idea of how a candidate reviews, you'll have to look for yourself. Let's not make perfect the enemy of good.

8
  • 1
    Well, if you start looking at how other people voted in each review, then you can get a bit of an idea of the quality. Somebody consistently in the minority could (could) be not paying attention. Clearly, they could also just be always overruled by robo-reviewers. Another idea is to look at timestamps to see if there is very little time between consecutive reviews. Commented Jul 11, 2017 at 13:08
  • 2
    maybe not worth the effort Having a good measure of it is worth rather a lot of effort (not so much for moderator elections, but simply for review banning people that don't do a good job). The problem is that even with rather considerable effort spent, there still aren't particularly good ways (short of having good reviewers manually look over every single review a person has performed) of determining if someone is really a good reviewer.
    – Servy
    Commented Jul 11, 2017 at 13:12
  • @AndrewMyers Timestamps help to find robo-reviewers. In themselves, they are not proof of robo-reviewing. But I've known a few reviewers who, judging by the timestamps, would click a button as soon as it loaded. It wasn't too difficult to find some tremendously bad reviews in their history and flag them. Time-consuming, but not difficult. Commented Jul 11, 2017 at 13:12
  • 3
    @S.L.Barth That's just it, it's often possible to find people flagrantly abusing the system and not even reviewing at all, particularly given that they're likely to fail a fair number of audits, but there's not really any good ways to differentiate a really good reviewer who's reviewing posts correctly and adding a lot of value from one that's not abusive and is at least trying, but simply doesn't understand the site well enough to actually do much (or do things correctly, reliably). Finding the 0% success rates are easy enough, differentiating the 50% success rates from the 90% isn't.
    – Servy
    Commented Jul 11, 2017 at 13:16
  • @Servy has clarified my point. We have some tools that serve as a litmus test to eliminate bad reviewers. It's the equivalent of doing FizzBuzz in a technical interview. You definitely can't tell if they're a good developer, but if they're a bad one, they'll stick out. That's not especially useful for electing moderators.
    – Athena
    Commented Jul 11, 2017 at 13:20
  • @Ares Honestly, what we have is not even good enough to determine if we even want to let them keep reviewing.
    – Servy
    Commented Jul 11, 2017 at 13:21
  • If they can't even do the review equivalent of a FizzBuzz, then IMO they're not moderator material. Handling flags is a large part of the job. Another part is resolving disputes, but that does not require diamond privileges. (Although I'll grant that the authority of a diamond does help). Commented Jul 11, 2017 at 13:31
  • 1
    @S.L.Barth I think that's what is being agreed on here anyway. If you can't pass FizzBuzz then you probably shouldn't get the job, but just because you can doesn't mean you can get it. That effectively makes it a meaningless test other than it helps to filter out the really bad ones.
    – DavidG
    Commented Jul 11, 2017 at 13:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .