We considered a personal 'review efficacy' page for each user. It seemed appealing to have something that encouraged people to spend more time in queues that were most intuitive for them by showing them how effective they were being.
As we started considering possible designs and implementations, we realized how much of a bad idea it was turning out to be. You're essentially saying:
- You're really great at this
- You really stink at this
- You're not really any different than anyone else at this
for motivational purposes, only the first one was really shiny, the other two were more brutally honest in most cases than anything else.
But, depending on the tags you visit (as others mentioned), being really great was rather subjective; and we just wouldn't have a whole lot to show folks that weren't at least somewhat active in the tags that would get you a Generalist badge at some point in the site's evolution.
You have to go with a median from the tag group, and then a number representing the efficacy of each reviewer. So, being wrong 6/10 times in Cobol would look a lot different than being right 3/10 times in PHP, compared to respective medians. So if your goals are:
- We want graphs where people can realistically work to make them go up and to the right pretty consistently, and,
- We only occasionally want the lack of progress to be a bit discouraging to people that just chronically get it wrong
... it just became crazy to fathom the individual math cases, much less maintain them, and then design around that.
I still think there's opportunity to discretely tell folks hey, you're really good at this particular thing, we'd love it if you did more, and even the subtle reverse of that, but when we looked at it with numbers and graphs in our hands, they turned out to be the wrong tools.
Always open to suggestions, I'd love more enticement into (and potentially out of!) the queues, but it's a tricky problem and really easy to inadvertently hurt people's feelings if you don't nail it just right.