I am a complete outsider and pretty new to the systems, but I feel that SO is making to little use of the potential power of the data at its hand.
Why do you need any criteria? Why not (summary:)
- define a loss function (combination of "emptying the queue" and "errors in the process", that is, false positives and false negatives)
- fix a time sample of all the data
- try to predict "mistakes quota" of people based on any observables you have. Obviously, you can focus on the regions where you want to be more permissive: No need to include "has gold medal" as a regressor if these guys already access the queue. See caveat below
- fit in your regression results into the defined loss function and minimize it
- Voilà, you have the optimal (w.r.t. your loss function) access criteria for the close queue (or whatever the object is)
Finally, since you restricted on the time sample, you can even simulate "what if these people were allowed to access the queue" using the remainder of time in order to get an idea of how good the idea would work.
Caveat
Importantly, There is no real data on how well did people do that do not satisfy the "close queue". That's the whole point, right? How good would they be. Say, there is this user A who
- has been registered for 100 days
- has posted 5 posts with average 3 points
- has edited 8 questions and has all of the edits confirmed.
No one knows how good this user would be at the close queue, right? But, you have data on user B and user C, who both looked like A 1 year ago, and one of them is doing really well now in the queue and one is not. Essentially, the question you would answer is
How likely is it that this A turns out to become like B, not C, and what are the costs if he actually turned out to be C?
Since you can always exclude someone from the queue if he turns out to be bad, I'd think that the costs are not too high.
Downside
While the idea of implementation is very clean, it may be difficult to transmit to new users ("A likelihood estimator predicted me to be a good user, wtf?").
Also, it may make maintenance more difficult: It will not be immediately clear any more "why exactly" someone has specific rights.
Potential Downside
I don't believe this, but just for completeness: It is a step away from "fairness" towards "efficiency". People do not get rights because they have earned them, but because the system expects them to be useful with these rights.