This post is following up on some final topics that we want to cover during the product discovery and early development stage of the Staging Ground. We are building here off of the framework that we have already described in the following posts (be sure to read at least the first two before proceeding here):
Workflow 2: Listings, Filters, Quality Control, and Notifications
How will the Staging Ground & the new Ask Wizard work on the network? (MSE)
A word of introduction: All of the topics being raised below are forward-looking, anticipated issues that we will have to contend with if the initial testing of the Staging Ground is successful enough to allow us to proceed for planning for feature graduation and for a general release on Stack Overflow (and other Stack Exchange sites that are interested in using it). We are sharing these here to let everyone know that these are areas that we are already thinking about, as well as to get your opinions on the magnitude of these issues, on our current direction, and on other possible solutions that you can think of.
Reviewer Motivation
Even though the Staging Ground is not a review queue, both sections will draw on the same pool of users to serve as Reviewers. For review queues, we have a very direct correlation between the number of Reviewers who are active on a queue and success of the queue (in terms of percentage of items reviewed, and timeliness of reviews).
For example, taking the First questions queue for example (as it is the closest in terms of purpose to the Staging Ground), since it launched in August 2021 we have had anywhere between ~1300 and ~2600 unique reviewers in the queue in a given month (with average actions per reviewer/month varying between 18-27). A drop in a single month of even a few hundred reviewers has spelled the difference between the review being at manageable levels of a few thousand posts (with most new posts being seen within a few days or even hours) and a vast increase of queue size (with a concomitant delay in the time needed to review for new items).
With the Staging Ground, we are asking more from Reviewers. While some posts will be Good to go at the outset with no changes needed, on other posts we will be asking Reviewers to advise on improvements needed from the Author before the question can be published on the site, and then to potentially have continued interactions with the Author, providing additional guidance where necessary. Additionally, since a user’s first questions won't be seen on the public site until approved through the Staging Ground, we will place a higher priority on getting items seen by Reviewers as expeditiously as possible.
Though we won't really know what the amount of time a Reviewer will need to spend per item (and how much this will differ for different quality questions that are coming in), and how many items a typical Reviewer will handle per day, it is fairly obvious that we will need at least as many, if not more, Reviewers (perhaps by a significant degree) to be active in the Staging Ground than we have had in Review queues, in order to ensure the success of the section.
We will be ensuring much increased visibility for new Staging Ground questions by interspersing them in some of the questions listings on the site, which should directly give us a larger number of eligible users trying out reviewing in the Staging Ground at least once. But we also need to consider what would motivate Reviewers to continue to help in the Staging Ground once they are there, so that we can increase the percentage of Reviewers who will return multiple times.
Badges
We are planning to offer a silver participation badge for any user that performs review actions on 10 Staging Ground questions during the initial testing period.
If the Staging Ground graduates, we plan on a badge system similar to Review queues: bronze, silver, and gold badges based on the aggregate number of questions reviewed, with the ability to earn multiple gold badges (exact numbers required for each badge are still to be determined).
Reputation Rewards
An idea that we have been playing around with is to offer a reputation incentive for Reviewers. This could be offered specifically on questions where the following sequence occurs:
A Reviewer interaction on a question in the Staging Ground of marking it as requiring Major changes or putting it in one of the Closed conditions and leaving a comment…
leads to edits by the question Author…
which then leads to approval of the question in the Staging Ground and subsequent publishing of the question publicly…
and finally to the question meeting some criteria for success on the site, at which point the Reviewer would be rewarded with some amount of reputation for their effort (most likely 10 or 15 points).
Success is tentatively defined as the question score or aggregate answer score over two (excluding votes by any Reviewers of the question in the Staging Ground), an answer being accepted, or the question reaching a certain number of views.
The extra rep could potentially be awarded above the daily rep cap. It could be unlimited or subject to a daily limit of its own.
The goal here would be to reward the extra time invested in guiding the new user to successfully transforming their question from one that was initially not up to site standards into one that performs well on the site (and hopefully, in the process of doing so, also helping to improve the skills of the new user as well and increasing the chances of them becoming a successful repeat contributor and curator in the future).
We know that rewarding site activity with reputation is potentially very controversial. So we want to tread carefully here (and are definitely interested in your feedback). Here are some things that we are aware of that could evoke some debate.
Rep Points for non-content:
Reputation points are given on the site mainly for contributing content (questions and answers) that is on-topic and helps others. So awarding rep from participating in the Staging Ground is potentially disruptive to the rep ecosystem and could be too big of a change from site norms.
On the other side of this argument, we already award rep for users (below a certain rep cap) who have their suggested edits approved. While this is still related to content, it is a little bit further removed from the direct authorship of content that question and answer authors receive their rep for. So this could be seen as something of a precedent for rewarding experienced users who are able to guide others in the process of creating good content.
I would argue that the return on investment of new rep into the overall site ecosystem is much higher here than it is for a vote or two received elsewhere, in that help provided to new users can have a magnifying effect for all of that user’s future contributions.
Potential for abuse:
Adding in a rep incentive could also draw in more users who will Review in the Staging Ground in bad faith in search of rep. These users could intentionally reject good posts from new users and leave comments with the goal of subsequently approving the post (which as a good post would have a good chance of meeting the success criteria).
This could lead to Reviewers favoring Major edits instead of Minor edits as a selected action on the question, even for posts where the latter option would be more appropriate. Thus, in this scenario, a rep reward could cause the opposite of what we want.
On the other side of this argument, we already have plenty of avenues for abuse on the site, and for these we have developed counter-measures and means of detection. This is no different. If folks start to abuse this, we can add additional schemes for detecting this (or at the very least calling mod attention to these users).
We are interested in your feedback on this idea.
Additional forms of recognition
We could potentially add a stat relating to the number of new users helped in a user’s profile.
If you have any other ideas for ways to encourage Reviewer participation and/or grant more recognition, we’d love to hear them below.
Scaling
We are aware that there are potential scaling issues with the Staging Ground. It sets up a system wherein a one-to-one interaction is needed for every new user question. Even with good publicity for the content within the site and an adequate reward system, it is unknown if (and for how long) the Staging Ground can be self-sufficient in terms of getting all or most questions reviewed in a timely manner.
For the initial test release, we will not be building in anything to address this. The purpose of the initial release will be to determine if this setup can work in its purest form, and if so, to iterate on its features and details to bring it to a place that maximizes its chances of success in a wider release. While we are thinking about ways to scale in the future, we are not going to start planning on anything related to that until we see how the Staging Ground itself performs with real users and content, and how much of a need there is for additional scaling. Thus, for the initial release we'll aim for a limited launch that mitigates the risks of scale and we will use the limited launch data to help us better understand those risks. This is a topic that we are taking seriously but deem it premature to address in a complete manner before we have concluded the initial test of the Staging Ground.
Taking this context into account, we are happy to hear your thoughts related to scaling this section in the future.
Anything Else?
This is your place to raise any other questions or thoughts that you have had about the Staging Ground for which there wasn’t a place to raise them yet. As a reminder, we are planning on the first test release of this section in a few months, which will only be open to a subset of users. From there, we will be monitoring, measuring, and iterating until we can make a determination on a final release, and on what additional features and changes would need to be included.
In addition to the topics discussed above, there are still open questions around items like qualification for Reviewers (rep-based or earned through actions), daily review limits (should we have them and how would they work?), audits (do we need to have them?), automatic suspensions based on suspicious behavior detection (how would these work?), just to name a few. All of these are fair game for discussion below. While we cannot promise decisions at this point on anything, we will try our best to respond to all constructive ideas, questions, and feedback.