**Update** (2023-03-08 15:15 UTC): The Beta 2 round is now open for business. We are still open for new reviewers to participate. Please see below for instructions on how to join. --- In December, we [announced a beta test for the Staging Ground](https://meta.stackoverflow.com/questions/421718/help-needed-a-call-for-volunteer-reviewers-for-the-staging-ground-beta-test) and opened up registration for reviewers to participate. The beta test (Beta 1) ran from December 8, 2022 through December 22, 2022 and included the participation of 50 reviewers. In this post, I will go over a recap of the beta test (including some stats and numbers), and talk about some changes we will be introducing based on the feedback given by reviewers. We are also happy to announce a **second round of beta testing** (Beta 2) which will be open for reviewers to participate (with the opportunity to earn a silver badge), **starting on Wednesday, March 8**. If you have at least 500 rep and are interested in being included as a reviewer, please ping @yaakov in the [beta test signup](https://chat.stackoverflow.com/rooms/249956/staging-ground-beta-test-signup) chat room. See the bottom of this post for more details. # Some stats from Beta 1 The goal of the first beta round was to test the workflow of reviewer and author-related processes within the Staging Ground, as well as to get feedback on things like onboarding and on-screen instructions. As it had only a small number of participants, it was not meant to be a test of scale, nor was it meant to lead to any conclusions about how effective Staging Ground will be when it is being used by many more reviewers and authors. With that said, an examination of some of the stats related to Staging Ground operation during this test can be instructive, both as a way of indicating the direction in which it is able to affect the quality of first questions on the site, as well as a milestone to which we can make comparisons in the future (especially when it comes to areas in which we still have room for improvement). ## What happened to the posts? 898 posts went through the Staging Ground in Beta 1. Here is what happened to them: | Outcome | Total | Percent | Avg Steps | Avg Total Min | Secs / review | Avg # Comments | | --- | --- | --- | --- | --- | --- | --- | | Inactive | 304 | 33.85% | 1.8 | 4.6 | 168.8 | 2.4 | | Graduated (Approved) | 280 | 31.18% | 1.9 | 5.8 | 166.6 | 1.9 | | Deleted | 224 | 24.61% | 2.5 | 6.1 | 134.8 | 2.5 | | Graduated (Auto) | 93 | 10.36% | | | | | The Inactive state covers anything that was set to Major Changes, or given a duplicate or off-topic vote, and was eventually abandoned by the author. In this case, the post is not deleted, but will disappear from the default listing in the Staging Ground after 36 hours (in theory, it can be “revived” by the author at any point after that, though it will cease to be shown to reviewers loading the default view). Some observations: - Over half of the posts that entered the Staging Ground did not leave, either because they were deleted (almost all posts were deleted by the author) or because the author decided to abandon the post. - Some of these posts might have been destined for a deletion or closing on the public site, while some of them could potentially have had their quality improved. We will be looking at how these numbers shift in the future as we add more tools that can help to streamline conversations. - Reviewers spent an average of 5-6 total minutes on each post that was given attention, averaging around 2 review steps per post. While we admire the commitment of our test reviewers, this is quite a high number and one that we are hoping to help to bring down in the future through features related to improving reviewer onboarding, comment templates, and more workflow options that can help to address areas of need. ## Did the Staging Ground help to improve the quality of graduated posts? One of the main goals of the Staging Ground is to help improve the quality of first posts that are published on the site. Here is how things are looking in this area from Beta 1. The following charts compare what happened to posts that were approved and graduated from the Staging Ground (not including auto-graduated) to first questions on the public site that did not go through the Staging Ground (the “Baseline”). The sample size of approved questions (280) is relatively tiny compared to the total number of first questions in the baseline group (25,156), and we are looking to increase the amount of posts in the Staging Ground Beta 2 to have more certainty in these results. ### Approved: Open Rate | Group | Total | # Open | Conversion Rate | Confidence | | --- | --- | --- | --- | --- | | Baseline | 25,156 | 16,559 | 65.83% | \[65.24%, 66.41%\] | | SG Approved | 280 | 235 | 83.93% | \[79.63%, 88.23%\] | When looking at open rates, we base this on what happened to posts in the first 3 days following publishing. We see here that there was an improvement in the percentage of posts that stayed open (not deleted or closed), going up from 65.8% to 83.9%, with a p-value of 0.006. While this result is expected, given the large number of posts that did not graduate from the Staging Ground, it is good to see this result in the actual numbers. ### Approved: Success Rate | Group | Total | # Success | Conversion Rate | Confidence | | --- | --- | --- | --- | --- | | Baseline | 25,156 | 3,784 | 15.04% | \[14.6%, 15.48%\] | | SG Approved | 280 | 78 | 27.86% | \[22.61%, 33.11%\] | For this stat, we considered a post successful if the question has a score of at least two, the total of all answer scores is at least two, or has an accepted answer. In this case, we see a significant improvement in the rate going from 15% in the baseline to 27.9% in the group of posts approved from the Staging Ground, with a p-value of less than 0.001. This is a very nice number to see, an early affirmation that the Staging Ground can prove an effective tool. ### Approved & Auto-graduated: Deletion Rate | Group | Total | # Deleted | Conversion Rate | Confidence | | --- | --- | --- | --- | --- | | Baseline | 25,156 | 3,947 | 15.69% | \[15.24%, 16.14%\] | | SG Approved + AutoGrad | 373 | 22 | 5.90% | \[3.51%, 8.29%\] | We see a significant drop in the percent of posts deleted within 3 days of publishing, for both posts that were approved in the Staging Ground, and for posts that auto-graduated (received no reviews within one day of being created), with a p-value of less than 0.001. That this affects auto-graduated posts is another indicator that in addition to helping increase the quality of approved posts and accelerate the onboarding of new askers, it also serves as a filter for the lowest quality questions (which are destined to be deleted). ## Feedback and reviewer participation in Beta 1 We had a really great time working with the volunteer beta reviewers from the community during Beta 1. There were 52 users accepted as reviewers (all Stack Overflow mods had access as well). 50 non-staff users performed a review activity (average 33, median 18), 34 earned a [beta silver badge](https://stackoverflow.com/help/badges/11548/staging-ground-beta) by performing at least one review and leaving at least one piece of feedback on the Stack Overflow for Teams (SOT) instance that we are using to collect feedback. On the SOT instance, reviewers left over 120 questions and many answers and comments, including bug reports (small and large), clarifications, and discussions. Special mention for the 10 users who left detailed (and in some cases very lengthy) experience reports that covered overall impressions relating to both general and specific points of their experience: [user](https://stackoverflow.com/users/11107541/user), [dan1st](https://stackoverflow.com/users/10871900/dan1st), [Panda](https://stackoverflow.com/users/5022249/panda), [ouflak](https://stackoverflow.com/users/446477/ouflak), [Henry Ecker](https://stackoverflow.com/users/15497888/henry-ecker), [Joundill](https://stackoverflow.com/users/6632744/joundill), [janw](https://stackoverflow.com/users/8528014/janw), [Chris](https://stackoverflow.com/users/354577/chris), [Ethan](https://stackoverflow.com/users/17493431/ethan), and [cocomac](https://stackoverflow.com/users/16886597/cocomac). Feedback from reviewers have led to over 80 new JIRA tickets, and we were able to respond to many of these with a quick turnaround speed during the course of the Beta test period. Following Beta 1, we conducted a survey of reviewers, receiving 27 responses. Here are some highlights from this survey: ### **Reviewer Experience** - 42% of Reviewers indicated that it was somewhat easy or very easy to provide a review in the Staging Ground. - The item that would make it easier for Reviewers to participate more in the Staging Ground was “Being able to view my aggregate impact and contributions” at 65% \[note: we are planning adding features to address this\] ### **Question Quality** - 42% of Reviewers indicated that they were either very satisfied or satisfied with the question quality - 89% of Reviewers, a larger majority, felt that the question quality increased slightly to significantly because of the Staging Ground. ### **Author Interactions** - 58% of Reviewers were very satisfied or somewhat satisfied with their interactions with Authors in the Staging Ground - 66% of Reviewers indicated that Authors were somewhat interested or very interested in improving their question. We also sent out a survey to new authors whose questions went to the Staging Ground (sending to all who had indicated they were ok being contacted for research purposes). Unfortunately, the response rate was too low for us to get enough results to report on here. We are aiming for an increased pool of authors in the next beta round such that we will also be able to use their feedback in a meaningful way. We’d like to express the gratitude of the entire team towards our beta reviewers. We all felt a real sense of collaboration and enjoyed the different forms of communication that we had. We were able to make some tangible progress on the areas that we wanted to focus on (overall workflow), and it was gratifying to hear from many of you about the positive and negative experiences that you had with Staging Ground. In particular, a number of you expressed the sentiment that you were feeling (and enjoying) a sense of investment in the success of new users and their questions - this is something that is wonderful to see, and that we hope to build around moving forward. # New features and changes coming out of Beta 1 We addressed dozens of issues over the course and following Beta 1, many of which were smaller bugs or minor enhancements. Here are some of the more major improvements that we are making (some of which are in response to reviewer feedback, and some of which are additional planned items from our roadmap): ## **Save without review** The original workflow would only allow a reviewer to submit an edit to a question in the Staging Ground if they submitted a review as part of the process (indicating if the question was approved, needed minor edits, major changes, or was a duplicate or off-topic). The intention was to try to enforce a sense of deliberateness for reviewers, that there is always a next potential step that can be taken with a post, and to force this to happen through the workflow. What ended up happening is that there were reviewers who just wanted to make cosmetic improvements to questions (fixing spelling, formatting, typos, etc), and did not want to take the responsibility of giving an official review. After reviewing feedback on this, we now see the value in allowing editing without reviewing, and will therefore be allowing this as an option, while still leaving all review options available to be used (based on the judgment of the reviewer who is editing). ## Comment templates [![An image of the new comment templates selection; a set of radio buttons above the Add Comment textbox, below a header "Add feedback or comments". The options are: Elaborate on what you expected, Provide background research, Convert images to text or code, Be specific about your problem, Include an MRE, Proofread for minor edits](https://i.sstatic.net/Boq1Q.png)](https://i.sstatic.net/Boq1Q.png) We are wrapping up work on Comment Templates (formerly referred to as [Canned Comments](https://meta.stackoverflow.com/questions/420897/staging-ground-workflow-canned-comments). The comments being used are very close to [our latest revisions](https://meta.stackoverflow.com/questions/421896/staging-ground-workflow-canned-comments-revised), with the change that we have made some of them more concise. This was done based on research relating to the use of similar functionality on other platforms, where it was found that longer comments (that appeared to be more general, and not related directly to the individual case) ended up having a negative effect on engagement. It is important to note that this is the first version of these comments, and we are definitely looking to iterate on these (both on the language, functionality, and customization options) based on usage and feedback. ## New comments UI and nested comments [![An image of the new comments user interface, featuring a mix of top-level and nested comments](https://i.sstatic.net/qNxVu.png)](https://i.sstatic.net/qNxVu.png) 1. **New layout**: The comment layout has been redesigned to aid readability. Each comment now comprises at a minimum of 3 lines, the first containing the user avatar and name, the second with the comment, and the third with the relative timestamp and additional actions (on hover). Actions are also available through a menu on the right margin. Top-level comments include a border along the left margin. 2. **Related review action**: For comments that were left related to specific review actions, a review action label is included to make the relevance of the feedback clearer to the author and other reviewers. 3. **Threaded Comments**: We are happy to introduce nested comments, which are intended to help conversations related to specific areas of feedback remain more focused. For now, comments will have a single layer of nesting, but we may try multiple depths in the future. ## Avoiding perpetual reviews A new feature will be added to provide a new reviewer action called "Decline re-evaluation" that a reviewer can choose when an author has put the post into "Re-eval" state to indicate that the author hasn't followed instructions in their edits. [![A composite image showing three visual components from the avoid-perpetual-reviews workflow. 1) A notice shown to the author after re-eval has been declined informing them of this and directing them to edit their question to address previous feedback. 2) The reviewer options for a post in Re-eval status, including a "Decline re-evaluation" option with accompanying text: "The author has not addressed existing feedback. This question will revert to off-topic status. 3) A confirmation shown to authors when submitting an edit for re-eval after a previous re-eval has been declined with the header "Have you addressed all previous feedback" and accompanying text: "Double-check that you've addressed all previous feedback from community members so that your question can be successfully re-evaluated".](https://i.sstatic.net/Cmbvo.png)](https://i.sstatic.net/Cmbvo.png) When this is selected, the author will receive an email and see an on-screen notice giving more details on why the re-eval request was declined, will see a similar notice on the post, and will get an additional "are you sure" modal the next time that they try to submit the post for re-eval. If this repeats itself 3 times, then the author will receive a 12 hour re-eval suspension, during which they will not be able to resubmit for re-eval and will not be able to ask new questions (and will receive another email detailing this). Any subsequent “Declined re-eval” action (without an intervening change in question state) will renew the re-eval suspension. This new option is aimed at fixing workflow issues we had in the first beta round where authors would continue to put posts in "re-eval" with trivial edits, ignoring reviewer comments and requests. This will now give clearer instructions to users about the requirement for re-eval (introduce edits that address feedback) while giving a safety valve (blocking re-eval and new questions) to prevent abuse. ## New reviewer guidelines: Following some extended discussions with beta test reviewers, we have come up with a new list of reviewer guidelines that seek to clear up some questions that came up during testing and make it clearer to reviewers what the expectations and standards of the section are. They are [now available in the help center](https://stackoverflow.com/help/staging-ground-reviewer-guidelines). # Beta 2 plans: New reviewers are welcome We are planning on opening up for a second round of Beta testing on **Wednesday, March 8, 2023**. As we will continue to only include reviewers who volunteered to participate, this will still be a closed test. However, we will open up the beta test to any qualified reviewer (over 500 rep) who wants to be included (regardless of experience on the first questions queue or any other review queue). Any reviewer who performs at least one review and leaves one piece of feedback (through the Stack Overflow Team or survey) will get a silver beta badge. 34 reviewers who already earned a badge in the first beta will be able to earn a second one during Beta 2. If you are interested in being included as a reviewer, please ping @yaakov in the [beta test signup](https://chat.stackoverflow.com/rooms/249956/staging-ground-beta-test-signup) chat room. If you applied for the first round and were not extended an invitation, there is no need to apply again, you will be automatically included. The second beta round will extend indefinitely, ending when we get to the MVP release. The MVP release will mark the introduction of features to support inviting new reviewers to the Staging Ground (and with more reviewers, more questions as well, which will allow us to begin to test features relating to scaling). We do not have a date yet for the MVP release. ## Questions? Please leave questions or comments about the Beta 1 round and related stats, new features, or Beta 2 below as new questions (not comments). ## Need more context? For more context on Staging Ground, please visit the following links: - [Staging Ground Help Center][1] - [Staging Ground Workflow: Question Lifecycle - Meta Stack Overflow][2] - [Staging Ground Workflow: Question Details & Actions - Meta Stack Overflow][3] - [Staging Ground Workflow: Listings, Filters, Quality Control, and Notifications - Meta Stack Overflow][4] - [Staging Ground: Reviewer Motivation, Scaling, and Open Questions - Meta Stack Overflow][5] [1]: https://stackoverflow.com/help/staging-ground [2]: https://meta.stackoverflow.com/questions/419746/staging-ground-workflow-question-lifecycle [3]: https://meta.stackoverflow.com/questions/416851/staging-ground-workflow-question-details-actions [4]: https://meta.stackoverflow.com/questions/417271/staging-ground-workflow-listings-filters-quality-control-and-notifications [5]: https://meta.stackoverflow.com/questions/417932/staging-ground-reviewer-motivation-scaling-and-open-questions