This is the second workflow post describing in detail the different aspects of the Staging Ground. It is written assuming that you have read the first workflow post (Staging Ground Workflow: Question Details & Actions), which gives a general overview of the project and its goals, and describes the question details view, commenting, and Actions (among other things). If you have not read that post, please go and do so now, as this post will not reintroduce concepts or terms that were introduced in the first post. Much of this post will be written from the perspective of the Reviewer (you, hopefully), rather than that of the new Author.
If you would like more background on the project, please also read the first introduction to the Staging Ground, the overview of user research that has gone into the project, and the introduction to the Ask Wizard component (which will serve as the entry point to the Staging Ground for new question Authors).
The Staging Ground Listing
When loading the main page of the Staging Ground, a Reviewer will see a listing of posts that are in the Staging Ground:
The listing is where Reviewers will go to find new questions to review, or to revisit questions that they had previously reviewed or interacted with that are ready for another look.
Filters
The Staging Ground listing will include a number of filters, which are designed to provide Reviewers with easy ways to identify questions that are familiar topics. New Authors who have posts in the Staging Ground will not see any of the filters, as the only posts that they will have access to are their own. Reviewers are supported both in discovering new items to review as well as following-up on previous reviews that require attention.
The design highlights a main differentiator from review queues, where Reviewers are served posts one at a time and are given minimal filtering options. Here, they can pick and choose which questions they want to interact on, and are given a number of data points related to questions and their stage in the Staging Ground for making this decision.
The following filters will be offered:
Status: This allows the Reviewer to filter by Staging Ground statuses.
Active statuses is the default. This includes all items that are pending Reviewer attention or that are pending Author attention that have been active recently.
Inactive will show questions that are pending Author attention that have not been active recently (we’ll start this at 36 hours). This includes questions that need Major changes where the Author has yet to respond in this interval. These will be hidden in the default view for Reviewers (Authors will always see the items that need their attention).
Moderators will additionally be able to filter by Flagged (which can apply to questions in any status).
Personalized filters will offer additional options:
None: No filter applied. Default if none of the following apply.
My reviews: This will show questions in the Staging Ground that the Reviewer is following. This is the view that Reviewers should use when looking to follow up on previously-reviewed questions that need further attention or when they want to review questions on which they have interacted in the past. This will be the default filter if the Reviewer is following any questions that await further review.
My tag badges: This new filter will apply any tags where the user has earned a badge. If we need to limit the number of tags for performance reasons, we will prefer gold tag badges over silver over bronze when making this selection. If there are no followed questions that require attention and the Reviewer has any tag badges, this will be the default filter applied.
My watched tags: This can be enabled if the user is watching any tags on the site. If selected, this filter will apply those tags to the questions being shown. This will be used by default if the previous filters do not apply and the user has any watched tags.
Tagged with allows the Reviewer to add a number of tags for filtering posts in the queue. These values will persist between page loads (at least on the same browser, potentially across browsers).
When showing questions with the New status (i.e., has not yet received any review), the order of questions shown within this status will be randomized so as to reduce the likelihood of multiple Reviewers opening one question at the same time (see later in this section for more on Active Reviews).
We are planning to include a notification on the listing to enable loading new items as they come in (and potentially toggling the In review status as well) without requiring a full page load, but we do not yet know if this will make it into our initial MVP (Minimum Viable Product) test release.
Columns and Status
Clicking on the question title in any row of the listing will load the detail page for that question.
The Last activity column will show the time of the last activity on the question, the user who performed the activity, and a description of the category of activity performed. For example, a Reviewer will be able to distinguish between questions where the last activity was the Author editing 10 minutes ago versus questions where the last activity was a Reviewer performing an Action 12 hours ago.
Rows will also display an appropriate label to indicate additional question states: Deleted or Closed (only shown to users who can view deleted and closed questions), Pending close (has a close vote), Flagged (shown only to mods), or In review (see the next subsection).
Sorting by status (the default) will consider the statuses in the order of priority for review, such that sorting in ascending order will show items that are New and Re-review at the top, and sorting in descending order will show items that are pending a close vote at the top.
Active Reviews
In review queues, the system is designed so that only one user can load a ReviewTask
at a given time. This is done by putting a temporary lock on that task when a user loads it. The lock expires when the user performs an Action on the task or when a number of minutes elapse (at which point we assume that the user is no longer active on the task). This is transparent to the end user, since in review queues, the Reviewer is not able to select which task they want to review — they have to accept whatever the system gives them (and the system skips over active tasks when selecting a task for a user to review).
On the Staging Ground, we will have a similar locking system in place, but it will work slightly differently, given that we are displaying questions in a listing and not just serving them one at a time. When a Reviewer loads a question, the system will note that the question is being actively reviewed, and will display this status in the listing and on the detail page itself for any other Reviewer while the In review state is there. During this time, other Reviewers will still be able to do anything on the post (like edit, comment, vote to close, or flag) other than one of the primary Staging Ground Actions.
We are planning on alerting a user viewing the question details page when the question has been updated in any way (and allowing for a quick refresh without having to reload the page). In the case where, despite these precautions, a Reviewer (R1) attempts to submit an Action on a question where another Reviewer (R2) has recently submitted an Action (since R1 last updated the page), R1 will need to update the page before they can submit their Action.
User Profile Activity Tab
A new tab (available for all users with Staging Ground access) will be included in the Activity area of the user profile that will highlight Actions performed by and comments left by that user in the Staging Ground.
Quality Control: Closing & Abuse Prevention
As much as we would like to help every new Author turn their question into a masterpiece, the reality is that there will always be some questions where this is not possible, and where other steps are needed. While we are designing the workflow to give Reviewers every option to provide guidance for new Authors in improving their posts, for posts where this is not possible or where content that violates the rules of the site is present, Closing and Flagging will be presented as options.
Close/Reopen Workflow
The normal criteria for closing posts will apply in the Staging Ground, and the Close button will be shown in the question menu below the content of the question. For the initial MVP release, this will provide the same close options as we have elsewhere in the site. However, there will be some other differences:
The plan right now is to start with two or three close votes being enough to close a question within the Staging Ground.
The thinking behind potentially using the lower value of two is that there will be fewer users seeing these posts than on the public site, and thus we may need to amplify the effect of a single vote.
Encouraging an over-abundance of question closing, and a pattern of closing salvageable questions, runs counter to the goals of this project. Therefore, we will closely monitor how this affects closing behaviors during our initial testing, and this value is subject to increase if the need arises.
When a single close vote has been applied to a post, a Pending close label will be appended to the question in the listing view to indicate that the question is at risk of being closed.
A proposal for reopening questions:
Closed questions in the Staging Ground will be reopened after any Staging Ground Action is performed by a Reviewer, or after any edit is made by the Author.
The goal is to allow a path for reopening without requiring any queue. While this does mean that a single reopen Action leads to it being reopened, the question will still be in the Staging Ground and will still have to be explicitly approved before graduating to a published state. (In other words, posts that were reopened will not be auto-graduated; they will only graduate following an Action of Good to go or Requires minor edits from a Reviewer.)
There will not be any initial limits on the number of close/reopen cycles a question in the Staging Ground can go through — but, like on the public portion of the site, users who voted once to close a question will not be able to apply another close vote on that Staging Ground question in the future. And likewise, questions that were re-closed after being reopened through the Actions of a specific Reviewer will not be re-openable by the same Reviewer.
Like all other
PostHistory
details, the close/reopen history and voting records of Staging Ground posts will not be transferred to published posts. So a post that was closed/reopened in the Staging Ground could be closed again on the main site (and nothing would block the same Reviewers who voted to close it in the Staging Ground from voting again on the published post).
We are interested in hearing your feedback about the proposed close/reopen workflow described here.
Abuse Prevention: Flagging, Spam, Deletions, and Suspensions
Flagging
Inappropriate behavior by users is an unfortunate reality that we need to address on the site, and the Staging Ground will not be an exception to this. As such, flagging will be an option on questions and comments, using the same workflow and options that exist today. These will be integrated into the moderator dashboards related to flags (we will be opening up a separate discussion with moderators to find out the most effective way for this to work for them).
Spam Detection
Community-run spam detection provides a valuable service for the community, and we want to ensure that this will be able to continue in the Staging Ground. To this end, I have already consulted with the admins of the Charcoal project to assess what type of access they will need in order to allow their services like SmokeDetector to run on the Staging Ground. The plans right now are to ensure that they will have the required access on day one of the test (this will involve some combination of Web Sockets and API routes; the exact details are still to be locked down).
Deletions
Authors will always be able to delete their own posts. However, for the initial test release, we propose removing the ability from non-moderator Reviewers to delete (or vote to delete) questions in the Staging Ground.
For cases of inappropriate content, spam, etc, flagging will always be available (which can lead to locking and deleting the post based on the normal rules for this, or can bring moderator attention to make a judgment call).
Beyond that, for any case where the question quality is not up to par, closing the question will be enough to prevent them from graduating to the public site. It will effectively take them out of circulation on the Staging Ground, while still giving guidance to the Author on the types of changes that would be needed to make the question a viable candidate for publishing on the public site.
In cases where the account of an Author of an active Staging Ground post is deleted, their post will be deleted as well.
Suspensions
In the case of Reviewers whose efforts are not aligned with the goals of the Staging Ground, or who just auto-review and consistently perform inappropriate Actions, a suspension will be appropriate.
The plan for the initial test is to use review queue suspensions for this: when a user is suspended from the review queue, they will also be suspended from the Staging Ground (and vice versa). Implementing a separate suspension system is not in the scope of our test release, and it seems fair to say that a user who needs a suspension from one section on the site could also use one on the other. If there is a need for a separate suspension system for the Staging Ground in the future, this can be addressed at that time.
Auto-Graduation
The status quo on Stack Overflow is that any new user can post a question and have it available for public perusal in a matter of minutes. This has always represented a valuable opportunity to new users for getting immediate visibility on their problem and answers to their questions. Every on-topic question posed on Stack Overflow — including questions on all points of the quality scale — represents a problem that someone out there had that they needed help with. And the answers to many of these questions will be of help to others.
The issues that we intend for the Staging Ground to help address — related to question quality from new users and to improve new user onboarding — also are directly related to the ease of publishing and visibility that has always been a main feature on Stack Overflow. Our attempt to fix these issues necessitates changing this status quo in a fundamental way: questions from new users will no longer be immediately available publicly, and some of them will never leave the Staging Ground at all.
While it is by design that the Staging Ground process will prevent the publication of unrectifiable low-quality questions, we also want to be sensitive to users whose posts — for whatever reason — are not attended to by Reviewers in the Staging Ground in a timely manner.
This leads us to the topic of how and when Staging Ground questions will be automatically published:
Questions that receive a Good to go or a Suggest minor edits Action (plus subsequent Author edit) from a Reviewer will be published immediately and will skip the First questions queue.
After a given period without Reviewer activity, questions that have the New or Minor edits status will automatically be published as publicly accessible questions.
Questions published in this way will enter the First questions review queue directly following publication.
The inactivity period is still to be decided, and will most likely be either 24 or 48 hours.
This includes questions that have received Reviewer comments without any Reviewer Action having been taken.
Certain questions that may not be fit for the public site will not be published unless a Good to go or Suggest minor edits Action is taken. This includes questions in the following states:
Major change status
Re-review state (includes Staging Ground questions that were reopened following an Author edit after having been closed)
Closed or having a close vote
Having an unhandled flag
Reviewer Recruitment
A link to the Staging Ground will be shown to eligible Reviewers and to Authors of posts that are currently active in the Staging Ground.
This link will appear in the left sidebar, and will display a blue “notification” icon when there are questions in the Staging Ground that require attention from the current user:
For new Authors, this will be shown when a Reviewer has performed a new Action or a comment has been left a comment on one of the Author’s questions in the Staging Ground.
For Reviewers, this will be shown when an Author has edited or commented on a question in response to an Action or comment left by the Reviewer.
A new feature popover will highlight this section to users who are eligible to Review when they gain this privilege.
Inclusion in /questions listings
As we have seen from the often overflowing First questions queue (and review queues in general) on Stack Overflow, it is very easy for the volume of questions from new users to be greater than the supply of Reviewers. And if that is true in review queues, then it is fair to say that it will be even more of an issue in the Staging Ground, where the demands being put on Reviewers in terms of time and energy commitment is even higher. If the Staging Ground gets the same number of Reviewers that the First questions queue gets (around 1800 ± 200 unique Reviewers per month), it will be very hard for this initiative to be successful long-term in a sustainable way. Thus, we are trying to think of new ways to promote the Staging Ground to eligible Reviewers.
To this end, users who are able to Review in the Staging Ground will see Staging Ground questions interspersed in a number of question listings on the site.
The image above demonstrates how Staging Ground questions can be interwoven with public questions on the site. Clicking to load this question will load the question details for the Reviewer inside the Staging Ground.
More details on how we propose that this will work:
Staging Ground questions will be included in listings on the homepage, the main /questions page, and tag listings. Search results and custom filters will be included at a later date, but will most likely not be included in the initial test release.
Only questions with the New status in the Staging Ground will be included.
We will limit the number of Staging Ground posts that will show up per page (probably starting with no more than five items per page of questions shown in a listing, though this value will be easily modifiable).
Questions will only show up in contexts where they would show up anyway if they had been public posts.
Users with Reviewer privileges in the Staging Ground will be able to opt out through an option presented in the User Profile and in the Staging Ground itself. The opt-out will most likely be from seeing Staging Ground questions in listings on the site (and not for hiding the Staging Ground sidebar link).
The “Staging Ground question” popover (shown above) will show if the user hovers over the “Staging Ground” label that appears to the left of the post title.
One of the factors leading to low participation levels in review queues is the lack of attention drawn to them on the site. A user is informed of review queues when they first gain the privilege to access them; after that, there is just a small icon in the top bar. We are designing this new integration with the goal of providing a way for Reviewers to organically discover relevant Staging Ground content in a way that is not overly disruptive to regular site activities. Your feedback and reactions to this plan are welcome.
(We plan on discussing more topics related to the overall scalability issues that the Staging Ground represents in another post that will be coming soon.)
Emails and Notifications
In the best case scenario, a new user authors a question that is posted to the Staging Ground, and it is then promptly reviewed by an experienced user who indicates that some changes are required and leaves a comment with guidance. It is important that we find ways to keep both the Author and the Reviewer informed of changes in order to prevent the review from stagnating and to expedite potential changes by the Author and subsequent publishing of the question. Here are the types of notifications that Authors and Reviewers will receive:
For Authors:
The Author will receive an inbox notification highlighting the Action and comment.
If the inbox notification is not read very promptly (within five minutes), an email will be sent to the Author informing them of the Action that was taken.
The email will not be in the format of the “1 new item in your Stack Exchange inbox” email that many of us are familiar with. Instead, it will more closely resemble the Call to Action notice that is shown on the top of the question detail page for the Author, explaining what was done and giving guidance to the user on how to proceed.
When a question is published through the Actions of a Reviewer (Good to go) or through auto-publishing, the Author will be notified via inbox and via email with a congratulatory message. This message will link to the new question and give the user guidance on what to expect and how best to tend to their new question.
For Reviewers:
A Reviewer who performs an Action on a question in the Staging Ground will automatically follow the post (they can unfollow it if they so choose). Other Reviewers will also be able to manually follow the question, even if they do not perform an Action on it.
Reviewers who follow the question will receive an inbox notification following any Action, comment, or major event (close/delete/publish).
We have not finalized any of the exact copy for these notifications.
In Closing
As was noted in the previous post, we are still in the early stages of this project and are several months away from being able to commence with testing. We are happy to receive questions and feedback below about the content of this post. Answers are preferred to comments.
Also as noted, we have two more planned posts: one to talk about issues relating to Reviewer motivation, scaling concerns, and other open questions, and one (to be posted on MSE) to introduce and open a discussion on how the Staging Ground can benefit other sites on the Stack Exchange network. If you have questions or thoughts relating to those areas, we request that you save them for the relevant post instead.