26

This is the second workflow post describing in detail the different aspects of the Staging Ground. It is written assuming that you have read the first workflow post (Staging Ground Workflow: Question Details & Actions), which gives a general overview of the project and its goals, and describes the question details view, commenting, and Actions (among other things). If you have not read that post, please go and do so now, as this post will not reintroduce concepts or terms that were introduced in the first post. Much of this post will be written from the perspective of the Reviewer (you, hopefully), rather than that of the new Author.

If you would like more background on the project, please also read the first introduction to the Staging Ground, the overview of user research that has gone into the project, and the introduction to the Ask Wizard component (which will serve as the entry point to the Staging Ground for new question Authors).

The Staging Ground Listing

When loading the main page of the Staging Ground, a Reviewer will see a listing of posts that are in the Staging Ground:

Staging ground listing view for Reviewers

The listing is where Reviewers will go to find new questions to review, or to revisit questions that they had previously reviewed or interacted with that are ready for another look.

Filters

The Staging Ground listing will include a number of filters, which are designed to provide Reviewers with easy ways to identify questions that are familiar topics. New Authors who have posts in the Staging Ground will not see any of the filters, as the only posts that they will have access to are their own. Reviewers are supported both in discovering new items to review as well as following-up on previous reviews that require attention.

The design highlights a main differentiator from review queues, where Reviewers are served posts one at a time and are given minimal filtering options. Here, they can pick and choose which questions they want to interact on, and are given a number of data points related to questions and their stage in the Staging Ground for making this decision.

Filters being offered. Includes Status filter shown: All with subcategories of New, Re-review, Major changes, and Minor edits. Followed by Inactive, Published, Closed, and Deleted. Personalized filters are: None, My reviews, My tag badges, My watched tags

The following filters will be offered:

  • Status: This allows the Reviewer to filter by Staging Ground statuses.

    • Active statuses is the default. This includes all items that are pending Reviewer attention or that are pending Author attention that have been active recently.

    • Inactive will show questions that are pending Author attention that have not been active recently (we’ll start this at 36 hours). This includes questions that need Major changes where the Author has yet to respond in this interval. These will be hidden in the default view for Reviewers (Authors will always see the items that need their attention).

    • Moderators will additionally be able to filter by Flagged (which can apply to questions in any status).

  • Personalized filters will offer additional options:

    • None: No filter applied. Default if none of the following apply.

    • My reviews: This will show questions in the Staging Ground that the Reviewer is following. This is the view that Reviewers should use when looking to follow up on previously-reviewed questions that need further attention or when they want to review questions on which they have interacted in the past. This will be the default filter if the Reviewer is following any questions that await further review.

    • My tag badges: This new filter will apply any tags where the user has earned a badge. If we need to limit the number of tags for performance reasons, we will prefer gold tag badges over silver over bronze when making this selection. If there are no followed questions that require attention and the Reviewer has any tag badges, this will be the default filter applied.

    • My watched tags: This can be enabled if the user is watching any tags on the site. If selected, this filter will apply those tags to the questions being shown. This will be used by default if the previous filters do not apply and the user has any watched tags.

  • Tagged with allows the Reviewer to add a number of tags for filtering posts in the queue. These values will persist between page loads (at least on the same browser, potentially across browsers).

When showing questions with the New status (i.e., has not yet received any review), the order of questions shown within this status will be randomized so as to reduce the likelihood of multiple Reviewers opening one question at the same time (see later in this section for more on Active Reviews).

We are planning to include a notification on the listing to enable loading new items as they come in (and potentially toggling the In review status as well) without requiring a full page load, but we do not yet know if this will make it into our initial MVP (Minimum Viable Product) test release.

Columns and Status

Clicking on the question title in any row of the listing will load the detail page for that question.

The Last activity column will show the time of the last activity on the question, the user who performed the activity, and a description of the category of activity performed. For example, a Reviewer will be able to distinguish between questions where the last activity was the Author editing 10 minutes ago versus questions where the last activity was a Reviewer performing an Action 12 hours ago.

Rows will also display an appropriate label to indicate additional question states: Deleted or Closed (only shown to users who can view deleted and closed questions), Pending close (has a close vote), Flagged (shown only to mods), or In review (see the next subsection).

Sorting by status (the default) will consider the statuses in the order of priority for review, such that sorting in ascending order will show items that are New and Re-review at the top, and sorting in descending order will show items that are pending a close vote at the top.

Active Reviews

In review queues, the system is designed so that only one user can load a ReviewTask at a given time. This is done by putting a temporary lock on that task when a user loads it. The lock expires when the user performs an Action on the task or when a number of minutes elapse (at which point we assume that the user is no longer active on the task). This is transparent to the end user, since in review queues, the Reviewer is not able to select which task they want to review — they have to accept whatever the system gives them (and the system skips over active tasks when selecting a task for a user to review).

On the Staging Ground, we will have a similar locking system in place, but it will work slightly differently, given that we are displaying questions in a listing and not just serving them one at a time. When a Reviewer loads a question, the system will note that the question is being actively reviewed, and will display this status in the listing and on the detail page itself for any other Reviewer while the In review state is there. During this time, other Reviewers will still be able to do anything on the post (like edit, comment, vote to close, or flag) other than one of the primary Staging Ground Actions.

We are planning on alerting a user viewing the question details page when the question has been updated in any way (and allowing for a quick refresh without having to reload the page). In the case where, despite these precautions, a Reviewer (R1) attempts to submit an Action on a question where another Reviewer (R2) has recently submitted an Action (since R1 last updated the page), R1 will need to update the page before they can submit their Action.

User Profile Activity Tab

A new tab (available for all users with Staging Ground access) will be included in the Activity area of the user profile that will highlight Actions performed by and comments left by that user in the Staging Ground.

Quality Control: Closing & Abuse Prevention

As much as we would like to help every new Author turn their question into a masterpiece, the reality is that there will always be some questions where this is not possible, and where other steps are needed. While we are designing the workflow to give Reviewers every option to provide guidance for new Authors in improving their posts, for posts where this is not possible or where content that violates the rules of the site is present, Closing and Flagging will be presented as options.

Close/Reopen Workflow

The normal criteria for closing posts will apply in the Staging Ground, and the Close button will be shown in the question menu below the content of the question. For the initial MVP release, this will provide the same close options as we have elsewhere in the site. However, there will be some other differences:

  • The plan right now is to start with two or three close votes being enough to close a question within the Staging Ground.

    • The thinking behind potentially using the lower value of two is that there will be fewer users seeing these posts than on the public site, and thus we may need to amplify the effect of a single vote.

    • Encouraging an over-abundance of question closing, and a pattern of closing salvageable questions, runs counter to the goals of this project. Therefore, we will closely monitor how this affects closing behaviors during our initial testing, and this value is subject to increase if the need arises.

  • When a single close vote has been applied to a post, a Pending close label will be appended to the question in the listing view to indicate that the question is at risk of being closed.

  • A proposal for reopening questions:

    • Closed questions in the Staging Ground will be reopened after any Staging Ground Action is performed by a Reviewer, or after any edit is made by the Author.

    • The goal is to allow a path for reopening without requiring any queue. While this does mean that a single reopen Action leads to it being reopened, the question will still be in the Staging Ground and will still have to be explicitly approved before graduating to a published state. (In other words, posts that were reopened will not be auto-graduated; they will only graduate following an Action of Good to go or Requires minor edits from a Reviewer.)

  • There will not be any initial limits on the number of close/reopen cycles a question in the Staging Ground can go through — but, like on the public portion of the site, users who voted once to close a question will not be able to apply another close vote on that Staging Ground question in the future. And likewise, questions that were re-closed after being reopened through the Actions of a specific Reviewer will not be re-openable by the same Reviewer.

  • Like all other PostHistory details, the close/reopen history and voting records of Staging Ground posts will not be transferred to published posts. So a post that was closed/reopened in the Staging Ground could be closed again on the main site (and nothing would block the same Reviewers who voted to close it in the Staging Ground from voting again on the published post).

We are interested in hearing your feedback about the proposed close/reopen workflow described here.

Abuse Prevention: Flagging, Spam, Deletions, and Suspensions

Flagging

Inappropriate behavior by users is an unfortunate reality that we need to address on the site, and the Staging Ground will not be an exception to this. As such, flagging will be an option on questions and comments, using the same workflow and options that exist today. These will be integrated into the moderator dashboards related to flags (we will be opening up a separate discussion with moderators to find out the most effective way for this to work for them).

Spam Detection

Community-run spam detection provides a valuable service for the community, and we want to ensure that this will be able to continue in the Staging Ground. To this end, I have already consulted with the admins of the Charcoal project to assess what type of access they will need in order to allow their services like SmokeDetector to run on the Staging Ground. The plans right now are to ensure that they will have the required access on day one of the test (this will involve some combination of Web Sockets and API routes; the exact details are still to be locked down).

Deletions

Authors will always be able to delete their own posts. However, for the initial test release, we propose removing the ability from non-moderator Reviewers to delete (or vote to delete) questions in the Staging Ground.

For cases of inappropriate content, spam, etc, flagging will always be available (which can lead to locking and deleting the post based on the normal rules for this, or can bring moderator attention to make a judgment call).

Beyond that, for any case where the question quality is not up to par, closing the question will be enough to prevent them from graduating to the public site. It will effectively take them out of circulation on the Staging Ground, while still giving guidance to the Author on the types of changes that would be needed to make the question a viable candidate for publishing on the public site.

In cases where the account of an Author of an active Staging Ground post is deleted, their post will be deleted as well.

Suspensions

In the case of Reviewers whose efforts are not aligned with the goals of the Staging Ground, or who just auto-review and consistently perform inappropriate Actions, a suspension will be appropriate.

The plan for the initial test is to use review queue suspensions for this: when a user is suspended from the review queue, they will also be suspended from the Staging Ground (and vice versa). Implementing a separate suspension system is not in the scope of our test release, and it seems fair to say that a user who needs a suspension from one section on the site could also use one on the other. If there is a need for a separate suspension system for the Staging Ground in the future, this can be addressed at that time.

Auto-Graduation

The status quo on Stack Overflow is that any new user can post a question and have it available for public perusal in a matter of minutes. This has always represented a valuable opportunity to new users for getting immediate visibility on their problem and answers to their questions. Every on-topic question posed on Stack Overflow — including questions on all points of the quality scale — represents a problem that someone out there had that they needed help with. And the answers to many of these questions will be of help to others.

The issues that we intend for the Staging Ground to help address — related to question quality from new users and to improve new user onboarding — also are directly related to the ease of publishing and visibility that has always been a main feature on Stack Overflow. Our attempt to fix these issues necessitates changing this status quo in a fundamental way: questions from new users will no longer be immediately available publicly, and some of them will never leave the Staging Ground at all.

While it is by design that the Staging Ground process will prevent the publication of unrectifiable low-quality questions, we also want to be sensitive to users whose posts — for whatever reason — are not attended to by Reviewers in the Staging Ground in a timely manner.

This leads us to the topic of how and when Staging Ground questions will be automatically published:

  1. Questions that receive a Good to go or a Suggest minor edits Action (plus subsequent Author edit) from a Reviewer will be published immediately and will skip the First questions queue.

  2. After a given period without Reviewer activity, questions that have the New or Minor edits status will automatically be published as publicly accessible questions.

    • Questions published in this way will enter the First questions review queue directly following publication.

    • The inactivity period is still to be decided, and will most likely be either 24 or 48 hours.

    • This includes questions that have received Reviewer comments without any Reviewer Action having been taken.

  3. Certain questions that may not be fit for the public site will not be published unless a Good to go or Suggest minor edits Action is taken. This includes questions in the following states:

    • Major change status

    • Re-review state (includes Staging Ground questions that were reopened following an Author edit after having been closed)

    • Closed or having a close vote

    • Having an unhandled flag

Reviewer Recruitment

A link to the Staging Ground will be shown to eligible Reviewers and to Authors of posts that are currently active in the Staging Ground.

Sidebar link to the Staging Ground

This link will appear in the left sidebar, and will display a blue “notification” icon when there are questions in the Staging Ground that require attention from the current user:

  • For new Authors, this will be shown when a Reviewer has performed a new Action or a comment has been left a comment on one of the Author’s questions in the Staging Ground.

  • For Reviewers, this will be shown when an Author has edited or commented on a question in response to an Action or comment left by the Reviewer.

A new feature popover will highlight this section to users who are eligible to Review when they gain this privilege.

Inclusion in /questions listings

As we have seen from the often overflowing First questions queue (and review queues in general) on Stack Overflow, it is very easy for the volume of questions from new users to be greater than the supply of Reviewers. And if that is true in review queues, then it is fair to say that it will be even more of an issue in the Staging Ground, where the demands being put on Reviewers in terms of time and energy commitment is even higher. If the Staging Ground gets the same number of Reviewers that the First questions queue gets (around 1800 ± 200 unique Reviewers per month), it will be very hard for this initiative to be successful long-term in a sustainable way. Thus, we are trying to think of new ways to promote the Staging Ground to eligible Reviewers.

To this end, users who are able to Review in the Staging Ground will see Staging Ground questions interspersed in a number of question listings on the site.

Staging Ground questions interspersed in a questions listing

The image above demonstrates how Staging Ground questions can be interwoven with public questions on the site. Clicking to load this question will load the question details for the Reviewer inside the Staging Ground.

More details on how we propose that this will work:

  • Staging Ground questions will be included in listings on the homepage, the main /questions page, and tag listings. Search results and custom filters will be included at a later date, but will most likely not be included in the initial test release.

  • Only questions with the New status in the Staging Ground will be included.

  • We will limit the number of Staging Ground posts that will show up per page (probably starting with no more than five items per page of questions shown in a listing, though this value will be easily modifiable).

  • Questions will only show up in contexts where they would show up anyway if they had been public posts.

  • Users with Reviewer privileges in the Staging Ground will be able to opt out through an option presented in the User Profile and in the Staging Ground itself. The opt-out will most likely be from seeing Staging Ground questions in listings on the site (and not for hiding the Staging Ground sidebar link).

  • The “Staging Ground question” popover (shown above) will show if the user hovers over the “Staging Ground” label that appears to the left of the post title.

One of the factors leading to low participation levels in review queues is the lack of attention drawn to them on the site. A user is informed of review queues when they first gain the privilege to access them; after that, there is just a small icon in the top bar. We are designing this new integration with the goal of providing a way for Reviewers to organically discover relevant Staging Ground content in a way that is not overly disruptive to regular site activities. Your feedback and reactions to this plan are welcome.

(We plan on discussing more topics related to the overall scalability issues that the Staging Ground represents in another post that will be coming soon.)

Emails and Notifications

In the best case scenario, a new user authors a question that is posted to the Staging Ground, and it is then promptly reviewed by an experienced user who indicates that some changes are required and leaves a comment with guidance. It is important that we find ways to keep both the Author and the Reviewer informed of changes in order to prevent the review from stagnating and to expedite potential changes by the Author and subsequent publishing of the question. Here are the types of notifications that Authors and Reviewers will receive:

For Authors:

  • The Author will receive an inbox notification highlighting the Action and comment.

  • If the inbox notification is not read very promptly (within five minutes), an email will be sent to the Author informing them of the Action that was taken.

  • The email will not be in the format of the “1 new item in your Stack Exchange inbox” email that many of us are familiar with. Instead, it will more closely resemble the Call to Action notice that is shown on the top of the question detail page for the Author, explaining what was done and giving guidance to the user on how to proceed.

  • When a question is published through the Actions of a Reviewer (Good to go) or through auto-publishing, the Author will be notified via inbox and via email with a congratulatory message. This message will link to the new question and give the user guidance on what to expect and how best to tend to their new question.

For Reviewers:

  • A Reviewer who performs an Action on a question in the Staging Ground will automatically follow the post (they can unfollow it if they so choose). Other Reviewers will also be able to manually follow the question, even if they do not perform an Action on it.

  • Reviewers who follow the question will receive an inbox notification following any Action, comment, or major event (close/delete/publish).

We have not finalized any of the exact copy for these notifications.


In Closing

As was noted in the previous post, we are still in the early stages of this project and are several months away from being able to commence with testing. We are happy to receive questions and feedback below about the content of this post. Answers are preferred to comments.

Also as noted, we have two more planned posts: one to talk about issues relating to Reviewer motivation, scaling concerns, and other open questions, and one (to be posted on MSE) to introduce and open a discussion on how the Staging Ground can benefit other sites on the Stack Exchange network. If you have questions or thoughts relating to those areas, we request that you save them for the relevant post instead.

11
  • 5
    "to close a question within the Staging Ground" It's not clear what closing within the staging ground means. The most practical impact of closing is not being able to answer, but with this definition all questions in the staging ground are closed by default. Maybe the post could give a few more details what closed (and especially not closed) mean in this environment. The reopening also seems to work differently with no additional voting. Maybe closing/reopening don't make much sense in the SG and should be left out altogether. Commented Apr 6, 2022 at 19:13
  • 4
    "As much as we would like to help every new Author turn their question into a masterpiece" Sorry for asking about the goals of the staging ground here, it surely is detailed in the previous posts about it and I read them but cannot remember. Is the goal of the staging ground to produce answerable, ontopic questions or great, high quality questions? And has this been specified before? I think that the height of the bar is a variable that can be set differently here and should ideally result in more (answerable but with lower quality) or less (but masterpieces they will be) questions as output. Commented Apr 6, 2022 at 19:42
  • 26
    This series is the highest quality text I have ever seen from the company (especially at this length), and a lot of work has clearly been put into it. However, a few diagrams wouldn't go amiss. For example, a state diagram for the questions, including the conditions for the transition between states. Commented Apr 6, 2022 at 20:36
  • Will there be a limit to the number of questions a Staging Ground reviewer can interact with? (I'm asking this because there is a similar limit for review queues)
    – pppery
    Commented Apr 6, 2022 at 20:54
  • How do you intend to deal with reviews that incorrectly suggest a Q is off topic? Eg, in a tag I follow, it's quite common that How To Q's are closed because they don't include any code. (FWIW there is a Hugh difference between a good How To and a work request) Commented Apr 6, 2022 at 22:18
  • 3
    @Trilarion closing within the staging ground means that the question is blocked from graduating and will not be shown in default listing views. Major changes is also a form of closing - but in that case the Reviewer will be giving guidance on how the question can be brought up to standard. For off-topic or deleted, a comment will not be required, and the general close messaging will be used (but still a possibility to reopen). We are going to explore more about the close process and canned feedback messages in a future post.
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 6:53
  • 3
    @Trilarion the project goals are included at the top of the tldr in the first workflow doc. TLDR: lower close/deletion rates of first questions, increase quality. Take pressure off of FQ queue. Improve onboarding experience for new users, lower-pressure environment for implementing feedback, improve likelihood and quality of future contributions on the site. It is not all-or-none. We would be happy with incremental improvements (even small ones) in all of these areas that we could continue to iterate on in the future. (Happy to delete these comments and move to your answer below for followup).
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 7:16
  • 6
    @PeterMortensen thank you very much for your feedback. Definitely appreciated. I spent most of the last few weeks on these last few posts and they have been reviewed by a whole team of folks internally (with literally hundreds of comments, suggestions, and revisions given). Was definitely a team effort.
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 7:18
  • 6
    @PeterMortensen as far as state diagrams, I think that that is a great idea, and will include it in our planning as something that we can hopefully address in a future post.
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 7:19
  • 2
    @pppery "Will there be a limit to the number of questions a Staging Ground reviewer can interact with" can you please ask this as an "answer" below? happy to address it there.
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 7:19
  • 2
    @chrisneilsen "How do you intend to deal with reviews that incorrectly suggest a Q is off topic?" can you please ask this as an "answer" below? happy to address it there.
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 7:20

17 Answers 17

47

Authors should not be able to invalidate my close vote by making a trivial edit.

Closed questions in the Staging Ground will be reopened after ... any edit is made by the Author.

I only get 50 close votes, but an author can trivially edit their question an extremely large number of times. Worse yet, I'm not even allowed to vote again after their trivial edit failed to address the issue.

I propose that the invalidated votes should be returned to the user and they should be allowed to vote again on the same question. If I've invested the effort into providing useful feedback and the author is able to trivially invalidate this feedback, I should be allowed to provide it again.

18
  • 4
    We'll discuss this more internally, including returning invalidated close votes and planning for potential limits on the number of times that Authors can reopen their posts. We could potentially have a different close vote limit in SG (or for the initial test just not limit the close votes, and see what patterns form and whether or not this is abused).
    – Yaakov Ellis StaffMod
    Commented Apr 6, 2022 at 18:44
  • 1
    Also, if anyone has an idea for allowing an avenue for reopening questions in the Staging Ground that does not include adding Staging Ground questions to the regular Reopen queue, we'd be happy to consider them as well.
    – Yaakov Ellis StaffMod
    Commented Apr 6, 2022 at 18:45
  • 35
    I'd go further than that. I do not understand why actions taken by the author are valued over those of experienced users - if the question gets closed in the SG, that likely means there are issues with the post. Allowing any action to reopen the post makes the whole point of closure moot, at this point there might as well be no closure (in which case we are back to square one as many questions are not a good fit in any shape or form). One possible solution is to allow the closure to start expiring once the author takes an action on the post with the ability for the reviewers...
    – 0Valt
    Commented Apr 6, 2022 at 19:01
  • 4
    ...who VTC'd the post in the first place to confirm or retract their close votes. Upon confirmation, the post would stay closed.
    – 0Valt
    Commented Apr 6, 2022 at 19:02
  • 9
    I would, personally, say that if someone sent a trivial edit as a way to get the question unclosed then it is abuse of the system and should be flagged (and have consequences). The fact that users already tick the box to state their edit "addressed the closure reason" when changing a single word is abuse enough; allowing them to completely bypass a review and go straight to open is not good UX for anyone other than the OP. Stack Overflow is a community for the masses, not for the individual.
    – Thom A
    Commented Apr 6, 2022 at 20:44
  • 2
    Reopening a question on the SG is not the same as on the public site. Reopening on public makes the question public again. When reopening a question on the SG the question would revert to Re-review status, and would still be completely blocked from going back to the public site until a Reviewer approved it. So it is still blocked from public. The only difference is that the "Closed" label is no longer there, and the Close Notice no longer shows.
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 7:12
  • 8
    If it makes a difference to folks, we could keep the close notice and close label in the SG after an edit (while still moving to Re-review) until a Reviewer approves the post or casts a Reopen vote. But it is important that there be a path to get the question back to the attention of Reviewers (without using a review queue). So leaving as "Closed" and not moving to Re-review isn't an option.
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 7:14
  • 1
    I, personally, think it would be beneficial for "re-reviewers" to see that the post was was/is closed, in the SG, @YaakovEllis . This would help those who are reviewing immediately see why the post was closed, and so know what to look for to see if the problem was resolved. Even though the closure reasons are quite broad, they still have enough meaning to know where the post failed before, and if those failures are still there, the reviewer can see that. It'll enable them to make a more informed decision on if the problems asked to be addressed have been.
    – Thom A
    Commented Apr 7, 2022 at 10:32
  • 8
    Now that I think about it, for duplicates, knowing what the question was closed as a duplicate as it very important. (In my experience) some users don't close as duplicates, even for frequently asked questions. If a question was closed as a duplicate and then when made for re-review with no details that it was closed as a duplicate, it could easily be approved because the question looks "good", but is still a duplicate. For such closure we expect the OP to explain well why the duplicate candidate isn't a useful, and an edit like" That question doesn't answer it" edits don't do that.
    – Thom A
    Commented Apr 7, 2022 at 10:47
  • @YaakovEllis I would like a checkbox a la what we have on the main site, that changes the status from close to reviewed or something in the status. BTW, I didn't find any definition on what the active/inactive sub filter mean?
    – Braiam
    Commented Apr 7, 2022 at 20:45
  • 1
    @Braiam I am not sure what checkbox you are referring to. Perhaps you should post this in a new answer?
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 21:16
  • 4
    @Larnu good suggestion about keeping the closed state even when it goes to re-review. We are considering it for inclusion.
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 21:16
  • 1
    @l4mpi I would rather at the outset keep this section to have lower friction and higher throughput then to add on additional voting restrictions. This is in order that what actual abuse vectors exist and then find the best way to plug them up. For example, if there are auto-reviewers approving things that shouldnt go through, I would rather have an automatic suspension mechanism (perhaps if a certain percentage of your recent Good-to-go reviews end up closed, deleted, or negative-score on the main site) than add extra friction and lower throughput here.
    – Yaakov Ellis StaffMod
    Commented Apr 10, 2022 at 11:50
  • 1
    @l4mpi I would be very satisfied with launching a limited test that gives us modest but significant improvements on new user question quality and close rates, and gives us insights into Reviewer behavior that we need to design for/prevent/police, that we can then iterate on. This is not a project where we are working on it, calling it a test, releasing, and then forgetting it. We are deliberately aiming to keep things on the simpler side initially, and then are prepared for multiple rounds of iteration and expanded testing to keep improving things for both authors and experienced users.
    – Yaakov Ellis StaffMod
    Commented Apr 11, 2022 at 13:38
  • 3
    @YaakovEllis you might be setting yourself up for failure that way. If the project starts out in a way that frustrates quality-focused reviewers, they will stop reviewing and getting them back on board at a later date will be significantly harder, no matter how much you improve it in the meantime. And if you don't have quality-focused reviewers, you might as well not bother with the complex SG workflow and just polish the triage queue a bit...
    – l4mpi
    Commented Apr 11, 2022 at 14:31
23

Treat closed questions differently based on whether editing can possibly fix the reason it was closed

It seems to me that there are two categories of closeable questions, with two different ways that they should be handled. Let's call these categories "salvageable" and "unsalvageable."

Salvageable questions have a problem that can be solved by editing: for example, they're unclear (clarify it), too broad (reduce the scope), or lacking debugging details (add them).

Unsalvageable questions have a problem that will not be solved by any amount of editing: they're off-topic, and they cannot be turned into a suitable question without changing it into an entirely different question. Examples include general computing questions, book recommendations, and questions that belong on Meta.

I understand your desire to make reopening easier/faster, but it's not appropriate in all circumstances. Consider differentiating between these categories of question closure, and making it harder to reopen questions closed for reasons in the "unsalvageable" category.

8
  • 6
    We are trying to do this. "Salvageable" (as you are calling them) should get a Major changes action, accompanied my comment giving guidance (canned comments to be provided to help get this started). This covers all of the "not good quality" close reasons". The types of "traditional" close options that we will provide are for the off-topic or duplicate (your "unsalvageable" category).
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 6:55
  • 2
    And we do want to make reopening manageable in either case. Happy to put safeguards in place, but not interested in recreating a reopen queue here. And dont want all of the onus on re-review to be on the original reviewer. This is the type of thing that we will definitely be evaluating in testing to see how the interactions go.
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 6:56
  • 2
    @YaakovEllis That seems totally reasonable, though it seems a bit at odds with "Closed questions in the Staging Ground will be reopened...after any edit is made by the Author."
    – Ryan M Mod
    Commented Apr 7, 2022 at 7:00
  • 3
    Perhaps you could relieve the burden by allowing other staging ground participants to reopen questions in the "unsalvageable" category more easily, rather than allowing an edit from the author to do so unilaterally. I would not generally trust a new user to understand the scope of the site sufficiently to overrule 2+ experienced users on their own, but I could support reopening after, say, an edit from the author plus a single reopen vote from any reviewer in the Staging Ground.
    – Ryan M Mod
    Commented Apr 7, 2022 at 7:00
  • 3
    That is an interesting idea @RyanM. So we kind of are already planning that right now - if a question is closed and then the user edits, it will lose the Closed state, and go back to Re-review state. This is a blocking state that will prevent graduation of the question unless there is Reviewer approval. So in reality - the question still would have a pseudo-close on it - no going public unless given approval (which is exactly what a close on the public site does). We would just be removing the "Closed" notice from the top of the question. That is it.
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 7:05
  • 3
    We could of course keep the Closed notice on after an edit (and still move it to Re-review status), and then a Reviewer would be able to "Reopen" it (all this would do it remove the close notice) and reapprove it. The only different here is the presence of the Close notice after edits. How important is that?
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 7:06
  • 2
    @YaakovEllis I think a pseudo-close fits the bill just as well from my perspective, and might feel a little "nicer" to new users while having the same effect from a site-mechanics perspective (that is, it won't go live without a review from an experienced user).
    – Ryan M Mod
    Commented Apr 7, 2022 at 7:28
  • 5
    Thanks, that is what we are aiming for. I think that we are aligned in intent here, and the "Close" label is the confusing part here (since reopening on the main site and reopening in SG have totally different repercussions). We need to do a better job in explaining the difference here.
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 7:29
19

Could the Staging Ground also be used for users who are in, or nearing a question ban?

Those of us that visit meta frequently will probably see a "Why can't I ask questions" post, or a question that should have been posted on Main at least once a day, and the reason for these posts is because the user is question banned due to low quality questions.

If a user is nearing a question ban, or is question banned and is asking their once in 6 month question, then I think it would be beneficial that such questions go to the Staging Ground. There we can try and get the user to improve their question in a constructive manner, before they send it straight the site.

Despite the warnings that many users get that they are close to a ban, they don't heed them and go ahead and continue to ask low quality questions, and then complain that they can no longer ask. If we can get them on the right path in the Staging Ground we could end up with a good question and the question asker might well be able to get themselves out of the question ban.

How these questions are weighted if they don't get out of the Staging Ground by the algorithm, I don't know. If someone who is banned fails to improve their question after being given advise and having it closed (in the staging ground) then that should be their question for the 6 month period though; if they won't make the effort with support, then they don't get another immediate chance, as that would just defeat the point of the ban.

6
  • 11
    I think that this is an excellent idea (we had not thought of this). Could make leaving a question ban be contingent not on time spent in the ban, but on # of questions that need to pass muster in the Staging Ground. Thanks for the idea, noted for the future (this wont be included in the first round of testing).
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 12:18
  • 1
    Happy to contribute ideas how questions in the SG would contribute to the algorithm when the time comes, @YaakovEllis . I appreciate that SO will want to keep the actual rules tight to their chest, but I'm not against a little more leniency for those who've so far asked low quality questions in an environment where it's designed to get the user to improve the quality of their content.
    – Thom A
    Commented Apr 7, 2022 at 12:24
  • 2
    Maybe such users would have more strict requirements to get the question out of the SG; such as it must be approved rather than it automatically after some time and they can't ask another question till it is or is fully rejected. Appreciate that this would likely be "hashed out" much further down the line, so I'll keep an eye out on posts in the future when the time comes.
    – Thom A
    Commented Apr 7, 2022 at 12:25
  • 1
    Yeah, all good considerations. We are a ways off from that discussion though (first we need to make sure that the concept is good, and at least ensure enough Reviewers to handle the load of first questions).
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 12:26
  • 3
    @YaakovEllis I would actually point out that if you go back to our first planning meeting to come up with ideas, this was actually one of the major benefits of this feature. We recognize that the question ban system is lacking and this might be an avenue to allow people with negative question records to get out of or avoid a question ban. 😁
    – Catija Staff
    Commented Apr 7, 2022 at 18:48
  • 1
    @Catija I am not sure if I was in that meeting or not. Either way, my apologies for the lack of accuracy and credit.
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 21:15
19

I said in my answer to the last SG post that this feature risks becoming worse than useless. With the info you've shared in this post, I'm now pretty much certain it will be exactly that. You're detailing a complicated but easily defeated process for question improvement, which seems to be influenced by a view that the vast majority of posts only needs a little bit of help to become a great SO question.

Here's the major issues I see with the current proposal:

Auto-Graduation

Just don't. You're arguing from the perspective that you don't want to raise the barrier to SO access too much, but for any question that auto-graduates, it would have been better for everyone if the question was never in the staging ground to begin with and instead was directly posted to main. Besides wasting everybody's time, here's the main problems with this approach:

  • OP is probably not browsing SO while their question auto-graduates and thus would be unable to react to any feedback once the question hits main, vastly increasing the time for any feedback loops for comments/answers. You propose a notification mail, but even in the best case that would add a few minutes of latency, and in the worst case it will never be read at all.
  • Abandoned questions that slipped through the SG gaps are pushed to main instead of being deleted.

Instead, I propose that you never auto-graduate SG questions, but instead turn auto-graduation into an explicit action which needs to be performed by OP. So after 48 hours or whatever with no activity, the question would not automatically be pushed to main, it would simply unlock a button which OP can press to do so. This guarantees that OP is present when their post hits main, and that they are still interested in an answer to the question.

Graduation Through User Review

As I already said in my last answer, it's highly problematic that it's very easy to push a question from SG onto main, but way harder to curate it or perform quality control actions. The process is heavily balanced towards accepting questions, as a single reviewer is enough to do so with a single button click. Closing a question is way harder as it takes more than one vote (and requires choosing a suitable close reason, which can be complicated e.g. for trash questions with code dumps that would have been "too localized" a long time ago), and is also easily defeatable with a small edit in your current proposal. Communicating with OP to get them to resolve major issues is again way harder, with the only saving grace that the lock might help prevent other reviewers from sabotaging the process in the meantime (but even that only applies if OP is responsive and the improvement doesn't take longer than your timeout).

You're proposing to use review queue suspensions for SG suspension. But a user could push a lot of trash onto SO main before being suspended. And even though review suspensions exist, we still have issues with reviewers in the existing queues such as first posts and editing approving way too much (some out of ignorance or not knowing the site rules, others just clicking whatever to farm badges). The problem is big enough that we have the audit system to catch the worst offenders in the review queues - which however doesn't seem to be planned for SG (and I'm not sure if it would be possible to create a good SG audit system). You're also aware that you probably don't have enough reviewers right now and might need to incentivize reviews with rewards, which exacerbates the issue. Combine this with the fact that we have tons of users who have no problem with writing bad answers to worse questions, and imagine these users as SG reviewers - you would end up with "good to go" on a lot of garbage questions (the example is from earlier today, I could provide a lot more, e.g. this one from yesterday - the answer is nonsense if that isn't obvious).

Instead, you should make it harder to get a question published - or, make it significantly easier to stop bad questions from ever reaching main. I would suggest setting the "good to go" threshold at three votes instead of one. A dynamic system where you require multiple "good to go" votes after the question received a close vote would be a step in the right direction, but would probably still result in a lot of garbage reaching main unfiltered.

Including SG Questions In Listings

Obsolete - I missed the fact that there's already an opt-out planned for this feature. That's good, as from my perspective the staging ground is there to filter out or at least improve the worst questions before I get to see them. Showing those posts on main would be worse than useless as I would still see the trash but couldn't even downvote it, so I'll be opting out.

10
  • 2
    "Including SG Questions In Listings" A toggle would be ideal, IMO, that way answerers who happen to have the rep for SG can see both in one place if they want to.
    – Kevin B
    Commented Apr 7, 2022 at 15:42
  • 2
    The post already says "Users with Reviewer privileges in the Staging Ground will be able to opt out" of seeing SG questions in the main question listing.
    – Tim
    Commented Apr 7, 2022 at 16:36
  • 14
    I like your idea about auto graduation. I am taking that back to the team to discuss internally. Forcing them to come back and graduate their post gives us another opportunity to put more resources in front of them and provide them with the option to close them if they don't need them any longer. I think, especially with new programmers, 48 hours can make a huge difference in trying to solve your problem.
    – SpencerG StaffMod
    Commented Apr 7, 2022 at 16:36
  • 1
    In principle I like the idea of the author having to manually publish, rather than auto publish. I'm not sure how that would work though - if they were notified their question was ready to be published a reviewer might request changes before they get round to publishing it, making the notification incorrect. If they weren't notified, it seems very likely that they won't know they can now publish the question.
    – Tim
    Commented Apr 7, 2022 at 16:37
  • 9
    @Tim I am speaking purely on a hypothetical possibility, but I think we can make it so reviewer recommendations supersede the graduation timer or figure out some way to make sure this doesn't come up.
    – SpencerG StaffMod
    Commented Apr 7, 2022 at 16:50
  • 1
    @Tim thanks, not sure how I missed the bullet point describing the opt out. Edited.
    – l4mpi
    Commented Apr 8, 2022 at 9:22
  • 5
    I strongly agree with the "auto-graduation must be a manual action" half of this answer, but cannot agree that three votes should be required for a question to leave: we don't have nearly enough reviewers to make that feasible, for one thing.
    – Ryan M Mod
    Commented Apr 9, 2022 at 2:22
  • 1
    @RyanM then you should also have a problem with three reviewers being required to close a question in the SG, right? If we don't have enough reviewers for three people to click "ok" on a question, how will we ever get three people to click "close" on a question before some drive-by badge farmer sends it to main?
    – l4mpi
    Commented Apr 11, 2022 at 7:50
  • @l4mpi I agree; I would personally lean toward two reviewers to close rather than three. However, it is somewhat less problematic if we don't have enough close reviewers than if we needed three people for a question to be posted: at least it wouldn't turn the Staging Ground into an ever-expanding backlog of unposted questions.
    – Ryan M Mod
    Commented Apr 11, 2022 at 22:03
  • 1
    @RyanM I prefer that than the ever-expanding backlog of posted questions to review, close and delete.
    – Braiam
    Commented Apr 16, 2022 at 13:55
13

Some concerns about participation:

If the Staging Ground gets the same number of Reviewers that the First questions queue gets (around 1800 ± 200 unique Reviewers per month), it will be very hard for this initiative to be successful long-term in a sustainable way. Thus, we are trying to think of new ways to promote the Staging Ground to eligible Reviewers.

I'm reading this as we would need more than "1800 ± 200 unique Reviewers per month" to make this long-term successful.

My question: is there any existing review queue that currently has enough participation to make this successful (is there evidence that we have enough reviewers to make this work)? or are we relying on making this a feature that has more participation than any existing review task?


I'm a bit confused by the phrasing of:

  • The plan right now is to start with two or three close votes being enough to close a question within the Staging Ground.
    • The thinking behind this lower value is that there will be fewer users seeing these posts than on the public site, and thus we need to amplify the effect of a single vote.

Specifically the "or three" being a "lower value" than the public site. Three is already the number of votes to close questions.

5
  • 1
    I fixed the language around close votes numbers to make it less confusing.
    – Yaakov Ellis StaffMod
    Commented Apr 6, 2022 at 18:16
  • 4
    We are predicting that we will need more than that number to make this long-term successful. First answers has slightly more unique reviewers per month, and all of the other queues have less. Hopefully there will be some migration and overlap between FQ reviewers and SG (since the goal is very similar, and the FQ workload will be significantly reduced once this goes live).
    – Yaakov Ellis StaffMod
    Commented Apr 6, 2022 at 18:18
  • 5
    But we are hoping for more participation, both based on the additional publicity discussed above (interspersing SG questions in regular /questions listings), as well as trying to offer other types of rewards beyond badges (we will be discussing some potential ideas around this in the next post, which should be coming in a few weeks).
    – Yaakov Ellis StaffMod
    Commented Apr 6, 2022 at 18:19
  • 1
    Okay, the new wording is much more clear to me. The consideration is maybe to allow 2 vote closures instead of 3. For the second part, it sounds like the answer to my question is yes we're hoping to get more participation here than we currently have in any other review queue. The hope being that integrating some SG questions into the main site will make reviews more accessible and attract higher participation. This is in addition to freeing up the workload on FQQ and potential incentives that are currently being discussed will attract the regular reviewers from the existing queues.
    – Henry Ecker Mod
    Commented Apr 6, 2022 at 18:31
  • 1
    I look forward to learning more about it in the coming weeks, but am glad to hear that this has been considered and a plan is in the works to get enough participation to make this a success.
    – Henry Ecker Mod
    Commented Apr 6, 2022 at 18:33
13

To this end, users who are able to Review in the Staging Ground will see Staging Ground questions interspersed in a number of question listings on the site.

I understand the motivation behind this: i.e., trying to bring more potential reviewers on board. However, depending on the qualifications required for, "users who are able to Review in the Staging Ground" (see my answer to the earlier post), this may turn out to be (at least partially) counter-productive, if this privilege is purely reputation-based.

There are frequent posts on Meta.SO (complaints, grumbles and other expressions of frustration from curation-minded users) about high-rep users who habitually post answers to obvious duplicates and other low-quality questions. So, assuming these users aren't going to mend their ways, they will just be able to open such SG posts, cast the single "Good to go" vote and proceed with their answer.

What we need, here, is a system where only those who are prepared to invest time in site curation get to see (and potentially approve) questions in the Staging Ground; hopefully, that way, duplicates (and other low-quality questions) will be closed before anyone gets a shot at answering them (answered questions are harder to curate; the Roomba won't touch them and no automatic redirection to the duplicate happens for anonymous visitors).

1
  • 7
    I agree, this may turn out well, or this may invite folks to act in a way that goes against what we are trying to do in the section. What I have described is the first iteration of the test. We will be monitoring how folks behave as Reviewers, and if there is a difference in the quality of reviews from those who entered through the sidebar or through a question link. If need be, we can add additional gatekeeping.
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 21:14
12

How should typographical and non-reproducible errors be handled in the Staging Ground? These, although share a closure reason, are actually quite different and the former would be very easy to spot in the SG, but as for reproducibility, I suspect that users in the SG should not be attempted to answer the question in the SG (and therefore copying code into their Sandbox for testing).

A question that is a typographical error isn't going to be helpful for future readers, and I agree they should be closed, but also closing the question in the Staging Ground feels "wrong"; when closing as a typographical error they are almost always followed/preceeded by a comment telling the OP "You missed a comma here", or "You wrote Cleint instead of Client". Are these comments acceptable in the SG, even though they aren't actually helping the user make what (might) be a well formed question get to the actual site?

As for non-reproducible; if we see a question that is good, but isn't reproducible, should these be resolved before being accepted? A non-reproducible problem is very likely to get closed, so the approval seems "wrong", but at the same time having code that the OP says "doesn't work" but for others does doesn't have anything to do with what is likely the quality that is being judged. Also, unlike a typographical error, is much more likely to not be spotted by those are who not an SME. Getting the OP to get the problem to be reproducible feels more like answering the question, not getting the question to be good (though it does also have that effect).

7
  • 1
    For the typo, if the question is written in a proper manner, I would expect that it would get approved, and then answered on the main site. An answer of "you misspelled that word" is a proper answer. In practice though, folks might use the comment and then the question would be abandoned. This is something that we will need to look out for in testing.
    – Yaakov Ellis StaffMod
    Commented Apr 10, 2022 at 6:44
  • 4
    Non-reproducible: set the Major changes action and let them know.
    – Yaakov Ellis StaffMod
    Commented Apr 10, 2022 at 6:45
  • 2
    @YaakovEllis I am confused why it matters whether a typo question is written in a specific manner. If the answer would be "you misspelled that word", then the question is off-topic and should be closed instead of answered, shouldn't it? Commented Apr 10, 2022 at 7:24
  • 2
    If the site standard is to close these types of questions as off-topic, then that is what should be done
    – Yaakov Ellis StaffMod
    Commented Apr 10, 2022 at 7:56
  • 2
    @YaakovEllis is "that is what should be done" implying it should be closed in SG or published with the intent of being closed immediately? Commented Apr 13, 2022 at 11:49
  • 1
    I would hope that they mean close in SG @AncientSwordRage . Saying a question "looks ok" in the SG to then immediately vote to close it seems counter productive, and sends an odd UX to the user.
    – Thom A
    Commented Apr 13, 2022 at 11:51
  • @Larnu fully agree with Commented Apr 13, 2022 at 12:15
12

Under the "For Authors" section, you note:

When a question is published through the Actions of a Reviewer (Good to go) or through auto-publishing, the Author will be notified via inbox and via email with a congratulatory message

Is a "congratulatory message" appropriate for the auto-publishing path?

My concern is that by treating it with a "congratulatory message" we're rewarding the quality standard of a question that's received no such validation.

I mean, the question might be fine! But it might have just run out the clock due to e.g. too few reviewers following the tags, an unusually high number of questions from new contributors, being published on the weekend, or something like that.

Consider using more neutral messaging

Obviously, contributors should be notified when their question is published—even if that process happens automatically. But I'd consider a less congratulatory tone in the latter case.

In fact, it may even be worth acknowledging that it's been automatically published, so expectations are set accordingly; e.g.,

Unfortunately, the community didn't have a chance to review your question. Don't worry, though: To ensure you still have the opportunity to get help in a timely manner, your question has been published to the site, and is now eligible to receive answers.

(Obviously, you're still working on the messaging, and I'm not a copy writer; the goal here is to convey the sentiment, not suggest exact wording.)

9

I'm not sure I understand the point of the close/reopen workflow.

  • What is the difference between an open and closed question while it's in staging? What would be the benefit of closing questions if they can be reopened by editing? On the main site it's clearly the lack of ability to answer, but that's not an issue in staging.

  • Why not simply remove the close/reopen funtionality from the staging ground entirely?

4
  • 2
    Some questions in SG will be closed because they are off-topic. For these, there is a slim chance that the question could be refocused through an edit to make it on-topic. Others will be closed as duplicates. For these, the user could edit to differentiate it from the dupe target, and make it worthy of reconsideration. Either way, if we are allowing closing (which we need to for these two cases), then a path to reopen is needed, even if it wont be used often. And note: once reopened, the question will not auto-graduate to published. Will need to be explicitly approved.
    – Yaakov Ellis StaffMod
    Commented Apr 6, 2022 at 17:27
  • 3
    Also, the third class of closed questions - where it is or might be on topic, but the quality is below our threshold - should get the Major changes needed status. This will also block it from graduating, and will prompt the user to put in edits (after which it will go to Re-review). And it will have the same effect as a close on a public question - block it from wider distribution/visibility until sufficient fixes are made.
    – Yaakov Ellis StaffMod
    Commented Apr 6, 2022 at 17:28
  • 11
    @YaakovEllis So what is there to protect from users repeatedly reopening their SG posts by performing superfluous edits? Sure, their posts won't end up graduating, does that just mean they'll sit in limbo as people stop reviewing them? Commented Apr 6, 2022 at 17:32
  • 3
    Nothing will prevent this. This is the top or if thing that we will need to keep an eye on in testing. It is a possible avenue for abuse, and if this manifests, then we'll need to consider how to address it.
    – Yaakov Ellis StaffMod
    Commented Apr 6, 2022 at 18:01
9

There's a lot in this post, so apologies if I've missed it, but given that Staging Ground posts are not really on the main site yet, per se, (and if closed while in SG, never become part of the main site, as I understand it), does that mean reviewers will have a separate pile of close votes to use?

Or will we have to now ration our already limited close votes if we want to participate in SG and still do things like existing review queues?

1
  • 13
    The plan for the initial test is that reviewers will either have unlimited close votes and review actions, or if there is a limit, it will not be shared with close votes on the main site.
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 21:18
8

A new moderator-only tab will be included in the Activity area of the user profile that will highlight Actions performed by and comments left by that user in the Staging Ground.

Will this information be available to non-moderator users via the "All Actions" profile tab? If not, it will put unnecessary burden on moderators to be the only ones to be able to monitor abuse, as well as reduce the visibility of such actions overall, leading to higher potential for abuse as there is less scrutiny.

All actions that can be performed by Reviewers are no different than those of other users: comments, suggested edits, edits, and reviews, all of which are available for public inspection — there is no point in hiding them for the Staging Ground specifically.

1
  • 6
    Thanks, you are right here. I changed the language to "A new tab (available for all users with Staging Ground access)…".
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 6:58
8

Its not really clear if you can close questions as duplicates since this would appear to conflict with the SG goals by providing an answer to the question.

And if you can close a question as a duplicate, do gold badges still work the same way?

2
  • 9
    You can close questions as duplicates. A gold badge would close right away.
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 8:21
  • 1
    Would a question closed as a duplicate enter the 'normal' close question state (i.e., could be reopened), or will it disappear into the void? Commented Apr 7, 2022 at 12:24
8

Will there be a limit to the number of questions a Staging Ground reviewer can interact with? (since there is such a limit in the review queues)

(originally posted as a comment, but I was asked to re-post as an answer, so doing so)

1
  • 3
    I think Yaakov's comment on TylerH's post (about whether the Staging Ground will share the close vote limit with SO proper for a user) at least partly addresses this: "The plan for the initial test is that reviewers will either have unlimited close votes and review actions, or if there is a limit, it will not be shared with close votes on the main site."
    – V2Blast Staff
    Commented Apr 8, 2022 at 17:49
6

The blue dot here would be entirely useless at best, and very annoying at worst on a site like SO (just like the review icon's red dot):

Blue Dot Staging Ground in menu

Please avoid indefinite notification dots. I think it’d be better if it had a blue dot if you haven’t ever visited it before, and then it go away once you do.

7
  • 17
    The dot will not be indefinite. It will only show when there are items that you have already reviewed where the author recently responded and the changes are pending your attention to review again.
    – Yaakov Ellis StaffMod
    Commented Apr 6, 2022 at 18:00
  • 2
    @YaakovEllis Will hovering over the dot show a tooltip with that very useful information? If that's not currently planned, can it please be considered?
    – TylerH
    Commented Apr 7, 2022 at 20:55
  • 3
    @TylerH yes, it can be considered. What type of info would be useful? The number of posts that need the attention of the reviewer?
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 21:17
  • 3
    @YaakovEllis The info you mentioned in your first comment is what would be useful since it is different info than what I would expect that dot to convey. If I see a dot on a sidebar or topbar on SO it usually tells me "this entire queue/Team has lots of activity and you need to visit it to view that active content". It doesn't tell me "things you have personally interacted with have been updated and require your review".
    – TylerH
    Commented Apr 8, 2022 at 14:56
  • 1
    @TylerH well, the presence of the blue dot here will always mean: "things that you personally interacted with recently had activity that probably requires follow-up from you". It wont be the same as the review queues "we are full, please come and do things" notification.
    – Yaakov Ellis StaffMod
    Commented Apr 10, 2022 at 6:21
  • 3
    @YaakovEllis Right, my point is that you have colored dots that indicate 'attention needed' at multiple places on the site, and they mean/refer to different things. We have already learned (sort of) what the dot over the review queue means, now this one will confuse us... tooltips are good because they explain what stuff means, like alt text for those who need that (and it should not be overlooked that they can serve as an instant reminder, days/weeks/months/years later, of what the dot means, in case anyone forgets and can't be bothered to come find this comment exchange).
    – TylerH
    Commented Apr 11, 2022 at 14:24
  • 2
    @TylerH got it. We'll keep it in mind for possibly varying the design or adding more context in a tooltip.
    – Yaakov Ellis StaffMod
    Commented Apr 11, 2022 at 14:27
6

I don't understand why closing and reopening is needed.

Closing a question on the main site means that something is wrong and it cannot and should not be answered. But questions in the staging ground cannot be answered anyway and closed questions cannot leave the staging ground, I guess.

Indeed we strongly expect that there will be plenty of things wrong with them. That's why we would probably tell askers all these sub-optimal aspects (not only one of them, which is traditionally sufficient to close a question).

And then we wait until the question creator corrects all these things and produces a great question (or at least answerable on-topic question). If that is the case, we green-light it and the question is ready to go live and be answered.

Closing/reopening is not needed for this workflow and would probably result in wasted effort (unless I'm missing something). I think of the staging ground like creating questions auto-closed and trying to work towards (re-)opening them for the first time. Am I mistaken there?

Consider removing close/reopening from the staging ground (or maybe motivate it a bit better in the post above).

8
  • 3
    Closing is still needed for duplicates and for off topic. Of course, these could be included in the area of needs major changes, but we are planning on keeping them separate at this point as we envision that action to be about improving question quality (unlike off topic and dupes where that doesn't really apply). This could change after we see how people respond to this during initial testing. Thanks for the feedback!
    – Yaakov Ellis StaffMod
    Commented Apr 6, 2022 at 19:28
  • @YaakovEllis So only the really unsalvageable cases are closed (but can easily be reopened). But then dupes could for example be salvaged by better delimiting from the dupe target. I think that if something is marked as a dupe or completely off-topic and the question asker agrees then ideally the asker would simply retract/delete his question at this point in the staging ground. Or would we like to keep these questions closed on the main site? Commented Apr 6, 2022 at 19:36
  • 1
    Re "I think of the staging ground like creating questions auto-closed and trying to work towards (re-)opening them for the first time.": That is not how I read it (though there is some ambiguity) - more like delayed publication by default (prompt publication will now be a privilege and will be awarded to high-quality questions). "Auto-Graduation ... how and when ... will be automatically published ... After a given period without Reviewer activity, questions that have the New or Minor edits status will automatically be published". If that is true, it will be a very good incentive. Commented Apr 6, 2022 at 20:28
  • 1
    cont' - "Want to ask your urgent question? Make it high quality!" Commented Apr 6, 2022 at 20:28
  • 1
    @PeterMortensen I saw the auto-graduation as an emergency valve. Some sort of: sorry we could not look over your new question in time, so at least you get the traditional treatment. But your view makes sense too. In this way the staging ground is some kind of nudging strategy. Something like: you better improve your question or we'll make you wait (and if we want we can make you wait even longer). Also makes sense. Let's see how it will play out. Commented Apr 6, 2022 at 20:49
  • 2
    I think that Peter's view is very in alignment with ours. And the auto-graduation also serves as a safety valve. Like I wrote above, denying public visibility on a question is a big break from the way that we have always operated. So if we cant deliver a Reviewer to look it over in a reasonable amount of time, then we cant justify keeping the question locked up indefinitely (and will still be the FQ queue to cover those as well).
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 6:49
  • @YaakovEllis One effect I would then expect is less questions asked overall. After all the hurdle to post a question will increase. Commented Apr 7, 2022 at 6:54
  • 3
    @Trilarion we'll just have to see. The questions here are going to be coming from New Users, who in almost no cases will not know about the existence of the Staging Ground, so not clear if its presence will be a deterrent for them from posting. But if we required a bigger hurdle to leave the Staging Ground for new questions (such that a user who keeps asking poor quality posts will be stuck in the SG for their new ones) then it could have that effect (and maybe we would welcome that?).
    – Yaakov Ellis StaffMod
    Commented Apr 7, 2022 at 7:01
-3

I'm seeing a whole wall of things that you would like reviewers to do along with some reasonably good info about how the queue1 will work, and next to nothing about how reviewers would be rewarded for those efforts. I'm having a very challenging time engaging in a positive way on suggestions for a queue like this without seeing what's in it for me as a reviewer.

I think overall the intent of all of this is good, but because I don't believe that there's a whole lot in it for me, I can't provide constructive criticism or point out deficiencies in this approach. I'm just...thoroughly disinterested in the whole process.

1: Yes, I'm calling back this old chestnut. Yes, I do want to understand and see what's in it for us.

24
  • 8
    This is planned to be covered in the next post. I know that it is hard to spot, but at the top of the previous post we mention the plans for "Staging Ground Workflow: Open questions for the test and the future". Exploring Reviewer motivation is the first topic to be discussed there. I am in the middle of writing this post right now, though it might be a few weeks before it goes public. I'll appreciate hearing your thoughts there.
    – Yaakov Ellis StaffMod
    Commented Apr 6, 2022 at 17:19
  • 6
    Fair enough. I suppose one thing to call explicit and overarching attention to is that you really don't want what you're offering to fall flat. There are some people out there who will happily volunteer no matter what carrot you dangle in front of them. There are others who have literal decades of experience on the site who may require a lot more to be bothered to share that expertise for the paltry offering you provide. And there are some in the middle. If you want this proposal to have a snowballs' chance in Hell, do everything you can to make this worth our while.
    – Makoto
    Commented Apr 6, 2022 at 17:22
  • 7
    You are correct. We don't want this to fail. And we understand that offering different ways to motivate Reviewers will be a key to success.
    – Yaakov Ellis StaffMod
    Commented Apr 6, 2022 at 17:23
  • 8
    Getting bad content refused or improved, and allow only good content to the site isn't reward enough? :)
    – Thom A
    Commented Apr 6, 2022 at 18:02
  • 7
    @Larnu: No. That "mutual benefit" ship sailed a very long time ago when it became overt and apparent that only some of our feedback mattered.
    – Makoto
    Commented Apr 6, 2022 at 18:14
  • 12
    Larnu's comment might have been a bit tongue-in-cheek, but in all honesty, I'm struggling to imagine what rewards aside from the one that comes from contributing to the improvement of the quality of content on the site that you would want to exist. I'm not sure I am comfortable with any other rewards that might perversely incentivize this. Can you make some suggestions? Aside from that, I am a bit confused about your feedback in general. You've bolded the suggestion to ensure this is made worth our while; Yaakov confirms they don't want it to fail. I mean…duh? How is any of that actionable?
    – Cody Gray Mod
    Commented Apr 6, 2022 at 18:21
  • 1
    @CodyGray: If they want "actionable" then they should really see what it'd take for someone who has the expertise in what they're looking for. For instance, I have copious hours and badges from doing the very thing they'd want to happen in this new workflow. Am I going to just volunteer? Probably not. Have I been asked what it'd take me to want to do this? No. What would I want? Depends on what's on offer. I know I have a valuable skillset and I want to see how they value it.
    – Makoto
    Commented Apr 6, 2022 at 18:28
  • 10
    @Makoto helping the Staging Ground will not be for everyone. As I said, we are going to be talking about other potential rewards. But we are not going to be paying folks for participating (which you have previously stated is the only type of compensation that you are looking for).
    – Yaakov Ellis StaffMod
    Commented Apr 6, 2022 at 18:34
  • 7
    @Makoto This is not the first time you have brought this up, and this is not the first time that I have responded to you in the same way. So your concerns are noted, and are off-topic for this question. I won't be responding to this thread anymore.
    – Yaakov Ellis StaffMod
    Commented Apr 6, 2022 at 18:35
  • 5
    "what's in it for me as a reviewer" What's in it for a reviewer of any review queue? Nothing but the satisfaction of contributing to something useful. I guess the reviewer motivation solely depends on how one sees SO and maybe also the struggling and effort that the askers put in (or do not put in). Yaakov simply tries to provide better tools for this, but fundamentally it's simply how much you like what SO currently is (a library of knowledge, a free helpdesk, something in between). My guess: there won't be more reviewers available unless you start paying them. But I may be wrong. Commented Apr 6, 2022 at 19:30
  • 6
    @Trilarion: I think you're seeing what I'm trying to aggressively highlight. The pool for reviewers is shrinking, not expanding. Even I remember the initial gold rush of the review queues to get the badges, then...immediately losing interest since it never felt like the actual satisfaction of helping was overruled by the wave of questions in the queue. Honestly I don't know how the pool of reviewers could possibly expand without something big. It's clear that there's going to need to be more people reviewing things than what we have today. I just don't see how this could scale.
    – Makoto
    Commented Apr 6, 2022 at 19:47
  • 7
    That said, I still don't want anyone to be getting paid for participating here, or for there to be any participation incentives created that would incentivize behavior that would run counter to our ultimate goals. Already, the review queues are too gamified for their own good.
    – Cody Gray Mod
    Commented Apr 7, 2022 at 6:23
  • 6
    Before anyone misunderstands, I agree that this probably does not scale given the current review situation. However, I also think that the feature is doomed unless people honestly want to engage with SG questions out of the goodness of their heart (or alternatively, to block the mountains of trash being deposited on the site every day). Any great incentive you could offer is doomed to result in a huge influx in reviewers who care only about the reward and neither about the specific post under review nor the site overall.
    – l4mpi
    Commented Apr 7, 2022 at 12:37
  • 3
    I understand that concern, but to me that seems like a problem we can resolve on meta. If crap is getting approved at such rates and people aren't being kicked out of the review process for doing it, the system is failing and a knob needs to be turned.
    – Kevin B
    Commented Apr 7, 2022 at 16:11
  • 2
    Effectively, Yes, obviously, people using the system opens the system up to abuse. More people, more possibility of abuse. compare that to what we have now, and what we have now is the same possibility of abuse but there's nothing we can do about it. At least with SG there will be the ability for abusers to be review banned.
    – Kevin B
    Commented Apr 7, 2022 at 16:17
-3

I noticed some conversation stating that this system will require more reviewer participation than what currently exists. I thought I would throw in my perspective about why I personally don't use the review queues.

Take, for example, this quote from this very thread.

There are frequent posts on Meta.SO (complaints, grumbles and other expressions of frustration from curation-minded users) about high-rep users who habitually post answers to obvious duplicates and other low-quality questions.

Well, that describes me.

When I was a new programmer I was mostly self-taught, wrestling with any bugs and issues that cropped up on my own. Google was my best friend, helping me get through most of my issues, but sometimes I needed a little more help, so I would turn to Stack Overflow (I used a different account at the time). At the time I had a poor understanding of the underlying concepts behind programming, but I still did the best I could to articulate my question clearly. In the end, those questions were probably as clear as mud, because I really had no idea what I was talking about. My questions would often get downvoted. I don't recall them ever getting closed, but I wouldn't be surprised if that happened. These sorts of reactions to my question did not feel that great - I had given my best attempt, and yet the community was punishing me, as if Stack Overflow was intended to be a place to go once you've passed some level of basic programming knowledge, and until then you're shunned if you ever try to use it. Those who were casting the downvotes perhaps didn't intend for me to take it this way, and actually meant the downvotes in the friendliest way you can, but from my end, with my limited perspective, it just felt like I was getting my hand slapped for not being smart enough to properly articulate myself.

However, someone would usually leave a pity answer, explaining that they weren't completely sure what I was trying to do, but would do the best to explain what they understood. These sorts of answers went a long way in helping me out and gave me the traction to get unstuck. It's thanks to people like these that I would keep coming back for Stack Overflow, despite the fact that my questions would constantly be shunned by most people.

It's been a long time since then, and now I'm an active contributor to Stack Overflow. Most of my contributions are simply through occasionally jumping in and answer people's questions. And, yes, I find that most questions are extremely unclear, and are often asking things that have been answered 100 times over in 100 different ways. But I answer them anyways. Unless it's an obvious homework question, I give them the benefit of the doubt and assume they tried their hardest to articulate the question and to research with the limited vocabulary and skills that they have. So yes, I'm "that person". That "[relatively] high-rep user who habitually post answers to obvious duplicates and other low quality questions" kind of person. And I do it, because that's the type of person that helped me when I was struggling and had no idea how to ask a clear question. I contribute to Stack Overflow because I want to help people, like my past self, or those who are struggling even worse than my past self ever did at grasping these concepts. I've seen some people frown upon this sort of behavior in the meta, sometimes it's because they feel like it's encouraging these users to continue asking more bad questions. And maybe it is. But, if it weren't for these types of people, I'm not sure I'd be here contributing to Stack Overflow, it's possible I would have just left empty-handed and turned to some other, more helpful community to work through my problems.

Eventually, I earned enough reputation to use the review queue, and I've opted to mostly dismiss it. In large part, this is because I run with the attitude of trying to salvage someone's attempt at a question by whatever means necessary, to help them out. And if it is likely a duplicate question, I'm ok pointing them to the duplicate (though I usually let others do that), but I might still want to show them a rough idea of how to do it for their specific problem as an answer or in the comments (because I know it can be difficult for newer programmers to figure out how to apply a generic principle to a specific problem). But I recognize my perspective is not the "correct" perspective in the eyes of Stack Overflow. In the review queue I'm expected to liberally flag questions as unclear or poor quality, give downvotes, and I'm not supposed to bother giving answers to duplicates, etc. Basically, I'm supposed to do the very thing that made me feel shunned when I was first asking questions on Stack Overflow. I know I can't bring myself to live up to this sort of expectation, so I choose to instead ignore the review queue.

Perhaps this staging environment will be different though. Perhaps, with the ability to ask clarifying questions it would become easier to help people salvage their questions through the conversational tools being added to it. And then I wouldn't have to feel guilty for using it. That's my hopes anyways. Depending on how this tool gets developed, and the standards the community chooses to hold the reviewers to, this may or may not be the case.

Anyways, I'm mostly posting this to give some perspective as to why I personally don't use the review queue, and a rough idea on how this staging area will need to be different than the current review system for me to be comfortable contributing. I don't know if other people fall into the same boat as me or if I'm pretty unique in this regard, but I'm betting I'm not alone.

Edit: I see people don't like this post at all. That's fine, but I would like to know what it is that people don't like.

  • Is it the way I contribute to Stack Overflow? I've never understood why these sorts of contributions are bad, nor have I seen Stack Overflow itself say that these sorts of contributions are bad, which is why I do still do it. But, if someone wants to point me to resources regarding this sort of conduct, that would be appreciated (I won't be argumentative about it).
  • Do people feel like I'm asking the staging grounds to be moved in a direction they don't want it to be moved in? Please note I'm not including any specific call-to-action within this answer. I'm merely trying to share why I don't feel comfortable using the review queue, and an idea of how the staging ground will need to feel different for me to be comfortable contributing. I'm not asking for it to be different in these ways, just expressing that if it were, I would be more comfortable using it.
  • Is there something else?
5
  • 7
    It should be pretty self-explanatory why people dislike a post that goes "Well, that describes me" to behaviour that causes "complaints, grumbles and other expressions of frustration from curation-minded users". That points out being aware that the behaviour is incorrect and doing it anyway. That points out that precisely the behaviour of getting pity answer was motivating to keep on posting bad questions. If you are wondering "why these sorts of contributions are bad", your answer right here is saying why. Commented May 5, 2022 at 14:05
  • 7
    Either way, a major point of closure is to prevent answers. If you know a question should be closed, that implies knowing that it should not be answered. Answering close-worthy questions anyway defeats much of the point of closure: it removes a major motivation of improving questions, and can actively prevent properly improving questions when that would invalidate premature answers. Commented May 5, 2022 at 14:09
  • Thank you @MisterMiyagi. And I hope what I said didn't sound too snarky - I guess I've always answered questions this way. I would run into the occasional meta post that grumbles about it, but I never fully understood why, nor did I ever see Stack Overflow itself discourage this behavior. so I just continued what I was doing, because I knew it was helpful to the askers, and I couldn't see how closing their posts was all that helpful to anyone - I don't know for sure, but I would assume most people take closure as rejection, not as "please edit and reopen". Commented May 5, 2022 at 16:02
  • 2
    Anyways, I'm willing to stop doing this if this is not how Stack Overflow wishes to run, and from the link you shared, it indeed sounds like this is the case. It just seems so unfortunate to "reject" so many questions for clarity reasons and let these people leave empty handed. But, perhaps that's where the staging area can come in - it will give people a chance to clarify their questions without feeling like their questions are being rejected. Commented May 5, 2022 at 16:02
  • (It's easier for me to get on board with not answering duplicates - I generally don't answer those out-right, sometimes I might just leave a tip in the comments) Commented May 5, 2022 at 16:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .