NOTE: There are going to be some who think this is about one particular candidate. I have thought about this for several election cycles, based on my own election experience and those I have watched run. Do not make this about one candidate please.
I've wanted to address what has been a fairly arbitrary process of helping people decide who should be elected as a diamond moderator. Let's talk a bit about the nuts and bolts first. In order to run on SO, you must meet the minimum requirements
- 3000 reputation
- Civic Duty, Strunk & White, Deputy, Convention badges
- 18+ years old
- Not suspended anywhere on the SE network in the last year
Breaking down the four required badges
- Civic Duty - 300 up/down votes
- Strunk & White - 80 posts edited
- Deputy - 80 helpful flags
- Convention - 10 Meta SO posts with +2 score or better
I think we need to acknowledge something important about voters: most people never bother with the "meta" of a candidate. The election page gives you a candidate score, and for most, that is all many voters will ever really base their votes on. The current candidate score isn't bad, but it's really dated. There are 40 points total, and half of those points are made up by reputation, with the remainder being various badges you can earn by being a "well-rounded community member".
Reputation
20,000 reputation is not uncommon. As of this writing, we have over 10,000 users above that level (nearly 90,000 users are eligible under the 3k limit). 3k will get you most of the "core" earned non-diamond moderation privileges
- Edit with no approval needed
- Close/reopen votes
- Access to all review queues
The chances a 3k user makes it very far as a candidate under the current system are... pretty low. The lowest reputation moderator I know of would be Undo (didn't win, but was called up due to high demand in the second 2015 election). The one person I would have liked to have won with low reputation was ArtOfCode. He came in 10th, and I strongly suspect it was largely due to his reputation (around 5k). In other words he was missing a whopping 15 points of candidate score (about 38%). We'll come back to why Art would have made an excellent mod in a later section.
We often debate the efficacy of reputation and votes, but all it really tells us is
- This person has posted
- A fair number of people found said posts useful
We're missing any real measurement of a moderator. There needs to be a minimum reputation to run (you have to actually use the site you're about to moderate, after all), but the way candidate score works, that minimum might as well be 20k.
Badges
Badges are more useful than reputation here, but still lacking overall. I have no idea what Sportsmanship has to do with being a diamond moderator, yet it's a point on the scoreboard (and I have no idea why Shog9 classified it as a "moderation" badge at that. It's 100% a participation badge).
Some badges are highly useful in that list (i.e. Steward, Quorum, Marshal) but some are not (i.e. Enthusiast, Yearling, Investor). I think badges have a role to play, (and provide a better picture of a candidate as a potential moderator), but still need work. 20 was a nice round number, so the number of the badges was 20 (19 was right out).
We need engaged candidates
The real problem here is that we're only kind-of sort-of measuring moderation engagement, in a very loose sense. The system works well enough on smaller SE sites, but SO gets all the flags. An SO moderator can handle more flags in a day than most smaller SE sites see in a year (I am not exaggerating). If you're not very engaged before you run, how engaged will you be after? That's part of why we have the Meta questionnaire now as part of the election system (and not a clunky set of Meta posts the candidates have to self-link to).
That's why the participation badges are useful. They just don't give a very good picture of how engaged you already are. The system links to a candidate's Meta posts and reviews, but we leave it to the voter to know why that's important, let alone what they're looking at. Unless we tell the voters these metrics directly, they are unlikely to look.
Coming back to ArtOfCode, his candidate score did not reflect anything he did for the community at large. What does he do? Charcoal. It's a community-run anti-spam project. Did you know SO has been hammered of late with fake phone support number spam? If you didn't, Charcoal was a big reason for that and Art was integral in helping to set that up. With that territory came lots of helpful flags and a decent amount of review queue work. That's not to say the people he ran against were slouches (Jean-François Fabre, who came in 9th in that same election certainly wasn't), but it's hard to be a highly engaged user and come in behind someone who has been hardly engaged at all.
TL;DR What do you propose?
I propose for future candidate scores should be out of 40 points:
Editing - 5 points
We already require 80 edits. Above 80 edits, you can earn 1 point per edit not done on your own post for every 40 edits. That means a 5/5 will need 280 edits. Moderators often suspend bad reviewers in this realm. You need to know how to make good edits.
Flagging - 10 points
We already require 80 helpful flags. As with editing, you can earn another 1 point for every 20 helpful flags. That means a 10/10 will need 280 helpful flags. We do need to break this down a bit (comment flags can easily run that total up), so no more than half of this score will be for comment flags. You can earn all 10 in post flags, however.
Meta Stack Overflow Engagement - 5 points
We already require 10 +2 Meta posts. You can earn 1 point by having 10 non-deleted comments in the last 90 days, another 1 for at least 10 post votes in the last 90 days, and 1 point for having two additional +2 meta posts (at six +2 meta posts you would gain 3 points, or 16 posts total). Voters wouldn't see a breakdown of this, but candidates would.
Reviews - 20 points
We have no shortage of reviews to be done. Moderators handle many types of flag. It's not ALL diamond moderators do, but the queues are a lot of our work (Community Managers will notice if the queue gets too large). The breakdown here would be as follows
- 10 points - 1 point per 100 reviews (up to 1000) in the last 90 days. That's an average of 12 or so reviews per day. This counts across all queues, so you can do them all in one queue, or mix it up.
- 5 points - 1 point per Steward badge (1000 reviews), max of 5. Again, you can do them all in one queue or in different queues
- 5 points - 1 point per 100 Low Quality Answers review, max of 5. We get the most of these (not counting comments and moderator flags). If you've done the 500, you should have a decent idea of what an answer should and shouldn't be. These reviews can count toward the other totals. Remember, moderator deletion can't be undone, except by another diamond.
These are all preliminary ideas. I'm hoping to spark some conversations that will lead to a better rating system for SO candidates. Do we need more points? Fewer? Other criteria not considered?