63

Update (2023-03-08 15:15 UTC): The Beta 2 round is now open for business. We are still open for new reviewers to participate. Please see below for instructions on how to join.


In December, we announced a beta test for the Staging Ground and opened up registration for reviewers to participate. The beta test (Beta 1) ran from December 8, 2022 through December 22, 2022 and included the participation of 50 reviewers. In this post, I will go over a recap of the beta test (including some stats and numbers), and talk about some changes we will be introducing based on the feedback given by reviewers.

We are also happy to announce a second round of beta testing (Beta 2) which will be open for reviewers to participate (with the opportunity to earn a silver badge), starting on Wednesday, March 8. If you have at least 500 rep and are interested in being included as a reviewer, please ping @yaakov in the beta test signup chat room. See the bottom of this post for more details.

Some stats from Beta 1

The goal of the first beta round was to test the workflow of reviewer and author-related processes within the Staging Ground, as well as to get feedback on things like onboarding and on-screen instructions. As it had only a small number of participants, it was not meant to be a test of scale, nor was it meant to lead to any conclusions about how effective Staging Ground will be when it is being used by many more reviewers and authors. With that said, an examination of some of the stats related to Staging Ground operation during this test can be instructive, both as a way of indicating the direction in which it is able to affect the quality of first questions on the site, as well as a milestone to which we can make comparisons in the future (especially when it comes to areas in which we still have room for improvement).

What happened to the posts?

898 posts went through the Staging Ground in Beta 1. Here is what happened to them:

Outcome Total Percent Avg Steps Avg Total Min Secs / review Avg # Comments
Inactive 304 33.85% 1.8 4.6 168.8 2.4
Graduated (Approved) 280 31.18% 1.9 5.8 166.6 1.9
Deleted 224 24.61% 2.5 6.1 134.8 2.5
Graduated (Auto) 93 10.36%

The Inactive state covers anything that was set to Major Changes, or given a duplicate or off-topic vote, and was eventually abandoned by the author. In this case, the post is not deleted, but will disappear from the default listing in the Staging Ground after 36 hours (in theory, it can be “revived” by the author at any point after that, though it will cease to be shown to reviewers loading the default view).

Some observations:

  • Over half of the posts that entered the Staging Ground did not leave, either because they were deleted (almost all posts were deleted by the author) or because the author decided to abandon the post.

  • Some of these posts might have been destined for a deletion or closing on the public site, while some of them could potentially have had their quality improved. We will be looking at how these numbers shift in the future as we add more tools that can help to streamline conversations.

  • Reviewers spent an average of 5-6 total minutes on each post that was given attention, averaging around 2 review steps per post. While we admire the commitment of our test reviewers, this is quite a high number and one that we are hoping to help to bring down in the future through features related to improving reviewer onboarding, comment templates, and more workflow options that can help to address areas of need.

Did the Staging Ground help to improve the quality of graduated posts?

One of the main goals of the Staging Ground is to help improve the quality of first posts that are published on the site. Here is how things are looking in this area from Beta 1.

The following charts compare what happened to posts that were approved and graduated from the Staging Ground (not including auto-graduated) to first questions on the public site that did not go through the Staging Ground (the “Baseline”). The sample size of approved questions (280) is relatively tiny compared to the total number of first questions in the baseline group (25,156), and we are looking to increase the amount of posts in the Staging Ground Beta 2 to have more certainty in these results.

Approved: Open Rate

Group Total # Open Conversion Rate Confidence
Baseline 25,156 16,559 65.83% [65.24%, 66.41%]
SG Approved 280 235 83.93% [79.63%, 88.23%]

When looking at open rates, we base this on what happened to posts in the first 3 days following publishing. We see here that there was an improvement in the percentage of posts that stayed open (not deleted or closed), going up from 65.8% to 83.9%, with a p-value of 0.006. While this result is expected, given the large number of posts that did not graduate from the Staging Ground, it is good to see this result in the actual numbers.

Approved: Success Rate

Group Total # Success Conversion Rate Confidence
Baseline 25,156 3,784 15.04% [14.6%, 15.48%]
SG Approved 280 78 27.86% [22.61%, 33.11%]

For this stat, we considered a post successful if the question has a score of at least two, the total of all answer scores is at least two, or has an accepted answer. In this case, we see a significant improvement in the rate going from 15% in the baseline to 27.9% in the group of posts approved from the Staging Ground, with a p-value of less than 0.001. This is a very nice number to see, an early affirmation that the Staging Ground can prove an effective tool.

Approved & Auto-graduated: Deletion Rate

Group Total # Deleted Conversion Rate Confidence
Baseline 25,156 3,947 15.69% [15.24%, 16.14%]
SG Approved + AutoGrad 373 22 5.90% [3.51%, 8.29%]

We see a significant drop in the percent of posts deleted within 3 days of publishing, for both posts that were approved in the Staging Ground, and for posts that auto-graduated (received no reviews within one day of being created), with a p-value of less than 0.001. That this affects auto-graduated posts is another indicator that in addition to helping increase the quality of approved posts and accelerate the onboarding of new askers, it also serves as a filter for the lowest quality questions (which are destined to be deleted).

Feedback and reviewer participation in Beta 1

We had a really great time working with the volunteer beta reviewers from the community during Beta 1. There were 52 users accepted as reviewers (all Stack Overflow mods had access as well). 50 non-staff users performed a review activity (average 33, median 18), 34 earned a beta silver badge by performing at least one review and leaving at least one piece of feedback on the Stack Overflow for Teams (SOT) instance that we are using to collect feedback. On the SOT instance, reviewers left over 120 questions and many answers and comments, including bug reports (small and large), clarifications, and discussions. Special mention for the 10 users who left detailed (and in some cases very lengthy) experience reports that covered overall impressions relating to both general and specific points of their experience: user, dan1st, Panda, ouflak, Henry Ecker, Joundill, janw, Chris, Ethan, and cocomac. Feedback from reviewers have led to over 80 new JIRA tickets, and we were able to respond to many of these with a quick turnaround speed during the course of the Beta test period.

Following Beta 1, we conducted a survey of reviewers, receiving 27 responses. Here are some highlights from this survey:

Reviewer Experience

  • 42% of Reviewers indicated that it was somewhat easy or very easy to provide a review in the Staging Ground.

  • The item that would make it easier for Reviewers to participate more in the Staging Ground was “Being able to view my aggregate impact and contributions” at 65% [note: we are planning adding features to address this]

Question Quality

  • 42% of Reviewers indicated that they were either very satisfied or satisfied with the question quality

  • 89% of Reviewers, a larger majority, felt that the question quality increased slightly to significantly because of the Staging Ground.

Author Interactions

  • 58% of Reviewers were very satisfied or somewhat satisfied with their interactions with Authors in the Staging Ground

  • 66% of Reviewers indicated that Authors were somewhat interested or very interested in improving their question.

We also sent out a survey to new authors whose questions went to the Staging Ground (sending to all who had indicated they were ok being contacted for research purposes). Unfortunately, the response rate was too low for us to get enough results to report on here. We are aiming for an increased pool of authors in the next beta round such that we will also be able to use their feedback in a meaningful way.

We’d like to express the gratitude of the entire team towards our beta reviewers. We all felt a real sense of collaboration and enjoyed the different forms of communication that we had. We were able to make some tangible progress on the areas that we wanted to focus on (overall workflow), and it was gratifying to hear from many of you about the positive and negative experiences that you had with Staging Ground. In particular, a number of you expressed the sentiment that you were feeling (and enjoying) a sense of investment in the success of new users and their questions - this is something that is wonderful to see, and that we hope to build around moving forward.

New features and changes coming out of Beta 1

We addressed dozens of issues over the course and following Beta 1, many of which were smaller bugs or minor enhancements. Here are some of the more major improvements that we are making (some of which are in response to reviewer feedback, and some of which are additional planned items from our roadmap):

Save without review

The original workflow would only allow a reviewer to submit an edit to a question in the Staging Ground if they submitted a review as part of the process (indicating if the question was approved, needed minor edits, major changes, or was a duplicate or off-topic). The intention was to try to enforce a sense of deliberateness for reviewers, that there is always a next potential step that can be taken with a post, and to force this to happen through the workflow.

What ended up happening is that there were reviewers who just wanted to make cosmetic improvements to questions (fixing spelling, formatting, typos, etc), and did not want to take the responsibility of giving an official review.

After reviewing feedback on this, we now see the value in allowing editing without reviewing, and will therefore be allowing this as an option, while still leaving all review options available to be used (based on the judgment of the reviewer who is editing).

Comment templates

An image of the new comment templates selection; a set of radio buttons above the Add Comment textbox, below a header "Add feedback or comments". The options are: Elaborate on what you expected, Provide background research, Convert images to text or code, Be specific about your problem, Include an MRE, Proofread for minor edits

We are wrapping up work on Comment Templates (formerly referred to as Canned Comments. The comments being used are very close to our latest revisions, with the change that we have made some of them more concise. This was done based on research relating to the use of similar functionality on other platforms, where it was found that longer comments (that appeared to be more general, and not related directly to the individual case) ended up having a negative effect on engagement. It is important to note that this is the first version of these comments, and we are definitely looking to iterate on these (both on the language, functionality, and customization options) based on usage and feedback.

New comments UI and nested comments

An image of the new comments user interface, featuring a mix of top-level and nested comments

  1. New layout: The comment layout has been redesigned to aid readability. Each comment now comprises at a minimum of 3 lines, the first containing the user avatar and name, the second with the comment, and the third with the relative timestamp and additional actions (on hover). Actions are also available through a menu on the right margin. Top-level comments include a border along the left margin.

  2. Related review action: For comments that were left related to specific review actions, a review action label is included to make the relevance of the feedback clearer to the author and other reviewers.

  3. Threaded Comments: We are happy to introduce nested comments, which are intended to help conversations related to specific areas of feedback remain more focused. For now, comments will have a single layer of nesting, but we may try multiple depths in the future.

Avoiding perpetual reviews

A new feature will be added to provide a new reviewer action called "Decline re-evaluation" that a reviewer can choose when an author has put the post into "Re-eval" state to indicate that the author hasn't followed instructions in their edits.

A composite image showing three visual components from the avoid-perpetual-reviews workflow. 1) A notice shown to the author after re-eval has been declined informing them of this and directing them to edit their question to address previous feedback. 2) The reviewer options for a post in Re-eval status, including a "Decline re-evaluation" option with accompanying text: "The author has not addressed existing feedback. This question will revert to off-topic status. 3) A confirmation shown to authors when submitting an edit for re-eval after a previous re-eval has been declined with the header "Have you addressed all previous feedback" and accompanying text: "Double-check that you've addressed all previous feedback from community members so that your question can be successfully re-evaluated".

When this is selected, the author will receive an email and see an on-screen notice giving more details on why the re-eval request was declined, will see a similar notice on the post, and will get an additional "are you sure" modal the next time that they try to submit the post for re-eval.

If this repeats itself 3 times, then the author will receive a 12 hour re-eval suspension, during which they will not be able to resubmit for re-eval and will not be able to ask new questions (and will receive another email detailing this). Any subsequent “Declined re-eval” action (without an intervening change in question state) will renew the re-eval suspension.

This new option is aimed at fixing workflow issues we had in the first beta round where authors would continue to put posts in "re-eval" with trivial edits, ignoring reviewer comments and requests. This will now give clearer instructions to users about the requirement for re-eval (introduce edits that address feedback) while giving a safety valve (blocking re-eval and new questions) to prevent abuse.

New reviewer guidelines:

Following some extended discussions with beta test reviewers, we have come up with a new list of reviewer guidelines that seek to clear up some questions that came up during testing and make it clearer to reviewers what the expectations and standards of the section are. They are now available in the help center.

Beta 2 plans: New reviewers are welcome

We are planning on opening up for a second round of Beta testing on Wednesday, March 8, 2023. As we will continue to only include reviewers who volunteered to participate, this will still be a closed test. However, we will open up the beta test to any qualified reviewer (over 500 rep) who wants to be included (regardless of experience on the first questions queue or any other review queue). Any reviewer who performs at least one review and leaves one piece of feedback (through the Stack Overflow Team or survey) will get a silver beta badge. 34 reviewers who already earned a badge in the first beta will be able to earn a second one during Beta 2.

If you are interested in being included as a reviewer, please ping @yaakov in the beta test signup chat room. If you applied for the first round and were not extended an invitation, there is no need to apply again, you will be automatically included.

The second beta round will extend indefinitely, ending when we get to the MVP release. The MVP release will mark the introduction of features to support inviting new reviewers to the Staging Ground (and with more reviewers, more questions as well, which will allow us to begin to test features relating to scaling). We do not have a date yet for the MVP release.

Questions?

Please leave questions or comments about the Beta 1 round and related stats, new features, or Beta 2 below as new questions (not comments).

Need more context?

For more context on Staging Ground, please visit the following links:

9
  • 10
    Are stage 1 reviewers automatically enrolled for stage 2? Commented Mar 2, 2023 at 18:19
  • 8
    @WhatsThePoint Yes, stage 1 reviewers will all automatically continue. They can earn a second badge if they choose to participate in the second round.
    – Yaakov Ellis StaffMod
    Commented Mar 2, 2023 at 19:05
  • 1
    stackoverflow.com/help/staging-ground link borked?
    – starball
    Commented Mar 20, 2023 at 23:54
  • 1
    @user not borked anymore. that link now leads to a new SG section in the help center, and the old article that was at that url is now at stackoverflow.com/help/what-is-staging-ground (links in the app were updated).
    – Yaakov Ellis StaffMod
    Commented Mar 21, 2023 at 9:23
  • thanks! in /help/what-is-staging-ground: "To make sure the process is efficient for you, your question will be published automatically if you do not receive reviewer feedback in a timely manner." would it not be an improvement to state when auto-publish happens? It happens after two days if no review action happens, right?
    – starball
    Commented Mar 21, 2023 at 17:10
  • 1
    Will feedback from this ever be publicly available?
    – Kevin B
    Commented Apr 5, 2023 at 14:44
  • 1
    @KevinB if you are asking if the comments left in SG will be available publicly - currently no plans for them to be public beyond users qualified to be reviewers. We have discussed longer term making SG read-only for all, or at a lower rep level, but nothing beyond general discussion at this point.
    – Yaakov Ellis StaffMod
    Commented Apr 13, 2023 at 10:57
  • 2
    @YaakovEllis I think this is about feedback from beta testers.
    – dan1st
    Commented Apr 13, 2023 at 10:58
  • 1
    Yea, i think it'd be rather useful to be able to point to a given thread where it was discussed why a feature will be the way it is in the future rather than all this work that was put into making SG what it is being locked behind a private SOT.
    – Kevin B
    Commented Apr 13, 2023 at 14:51

10 Answers 10

43

Thank you for listening to feedback!

I really appreciate how feedback was listened to and implemented.

I hope that it will continue that way, not only in the second beta but also when it goes public.

I see great potential in the staging ground and I hope it can improve asked question and help reducing the perceived "toxicity" of Stack Overflow by educating users on how to write better questions.

As someone who had the privilege to experience it in the first beta, I want to mention that I'm looking forward to the second beta and going public some time after it.

Thank you.

12

Are there potential plans for a compact comment view? I'm a big fan of the threaded comments and the action tags but the example discussion shown takes up a fair bit of vertical space.
enter image description here
Yes, the reply button has terrible placement in my edit, but I feel like less space is wasted

4
  • 11
    The goal here was to sacrifice a little space for the sake of making things clearer and less condensed. Of course everything has tradeoffs. This hasn't even launched publicly yet, so we don't yet have plans for changing things up. Always a potential for the future based on feedback (once people are actually using it).
    – Yaakov Ellis StaffMod
    Commented Mar 2, 2023 at 19:36
  • 2
    I feel like the lack of spacing above/below the image in your answer sort of demonstrates the issue – if there's too little space in between multiple lines of text, it can become a bit harder to read. (That doesn't rule out the possibility of improvements, as Yaakov said.)
    – V2Blast Staff
    Commented Mar 2, 2023 at 20:09
  • 1
    @V2Blast I agree that my mockup is too compact, but that third line with only a timestamp and link bugs me! IMO these UI updates are very welcome though 😄️ and I appreciate all the communication around them
    – GammaGames
    Commented Mar 2, 2023 at 21:39
  • 1
    Will there be any future improvements when viewing or editing SG on mobile? Half of my time spent is on my phone, today I tried editing and it’s not working. Bottom bar actions panel just overlaps the cursor and can’t see it. Commented Mar 9, 2023 at 14:03
11

As requested by @Yaakov Ellis, I've moved this comment to an answer.

"What ended up happening is that there were reviewers who just wanted to make cosmetic improvements to questions (fixing spelling, formatting, typos, etc), and did not want to take the responsibility of giving an official review."

Just wanted to elaborate a bit on this part, as I know there are others who have a similar/identical philosophy and approach. The way this has been worded suggests that we might be trying to dodge doing any review work. But for my style of reviewing, deciding whether a post needs editing is the first step in the review. That is the initial statement to the OP that there are some minimal expectations of quality on this site. Further, I want to get the post in a state where I, or if I do decide to step away - then someone else, can progress the review based on its content and relevance. If the post has quality formatting, grammar, clarity and structure, but it's obvious that only a subject matter familiar/expert can review it properly, then letting that particular someone perform the task is the clearly the responsible thing to do.

I'm glad that this is addressed and look forward to taking a more active part in the next phase.

9
  • 2
    Sounds good, thanks for the clarification. We hope that the introduction of this will encourage overall higher quality on posts (even those that are only edited and then auto-graduated, with no actual reviews), and will allow reviewers to find their respective comfort zones when it comes to their preferred modes of interactions.
    – Yaakov Ellis StaffMod
    Commented Mar 5, 2023 at 12:33
  • 5
    One note of clarification though. You wrote "but it's obvious that only a subject matter familiar/expert can review it properly" - please see the new reviewer guidelines that were referenced above, which include: "You don’t have to be a subject matter expert on the question in order to review it. If it looks like it has the elements of a good question, trust your gut and go with it, even if you are not an expert on the subject being discussed."
    – Yaakov Ellis StaffMod
    Commented Mar 5, 2023 at 12:35
  • 5
    Well, a bit of subject-matter knowledge might be required in order to differentiate code and a cat running over a keyboard in some cases.
    – dan1st
    Commented Mar 5, 2023 at 15:29
  • 1
    @YaakovEllis: You do need expertise to identify duplicates, though. Commented Mar 6, 2023 at 14:41
  • 6
    @MatthieuM. true. But the threshold to approve something is not "I am absolutely sure that there are no duplicates for this". It is "Well formed question, and I don't see anything wrong with it". We much prefer items to go quicker through the SG, with less effort, at the cost of some of them being closed as dupes on public (and in this case, the duplicate question is at least well written, and can therefore be a sign-post for others).
    – Yaakov Ellis StaffMod
    Commented Mar 6, 2023 at 15:21
  • @YaakovEllis "but it's obvious that only a subject matter familiar/expert can review it properly" The implication being that SG is not sufficient review.
    – philipxy
    Commented Mar 21, 2023 at 0:18
  • 1
    @philipxy I think that SG should be a sufficient review that the question is properly formed. And if an SME can review it there, even better. But better to graduate the question if it is properly formed (even if not an SME) and risk chance of closure on main site, than hold everything up for SME in SG.
    – Yaakov Ellis StaffMod
    Commented Mar 21, 2023 at 9:24
  • @YaakovEllis That isn't relevant to my comment. (I addressed the point of another comment, where you ignored what they meant by "review it properly".) PS SG isn't making it clear to posters just what it is trying to achieve / what has been achieved upon "graduation", that that is merely "properly formed". Instead there is the usual salespeak.
    – philipxy
    Commented Mar 21, 2023 at 9:43
  • 2
    @philipxy yes, we have noted the lack of clarity to askers about SG purpose and significance of graduation, I have brought it up for internal discussion. FWIW this is mentioned several times on the banners and instructions shown to authors, but obviously there is room for improvement.
    – Yaakov Ellis StaffMod
    Commented Mar 21, 2023 at 10:09
9

31.18% graduation (approved) rate seems rather bad (not for the staging ground but for asking on SO in general). If you then spend 5-6 minutes on every question, this would make already 15-20 minutes per graduated (approved) question. I mean that in a total time invested overall for total amount of good questions output kind of thinking.

Edit: And this is not even the total amount of curation needed for a single surviving good question. Additional costs can occur later outside of the SG stage. See this comment

... the threshold to approve something is not "I am absolutely sure that there are no duplicates for this". It is "Well formed question, and I don't see anything wrong with it". We much prefer items to go quicker through the SG, with less effort, at the cost of some of them being closed as dupes on public ...

6
  • 2
    i'd find it a bit concerning that for a given question the reviewer would be expected to spend more than 30 seconds on a given question, much less than the 5-6 minutes that has been reported as the average. Hopefully this is just evidence that the process hasn't been ironed out yet, rather than an acceptable avg amount of time. At the current rate i'd expect most questions to "graduate" without review at all if this were opened up to all questions.
    – Kevin B
    Commented Mar 6, 2023 at 20:49
  • 2
    @KevinB The "market" will decide what is an acceptable time investment per graduated (approved) question. If I understood it correctly roughly 1% of incoming questions were handled by 50 reviewers in beta-1, so roughly 5000 reviewers in total would be sufficient for all questions if they will be able to deliver the amount of time they invested in this test for a long duration. Otherwise most questions will simply be auto-graduating. Commented Mar 6, 2023 at 21:32
  • 10
    5-6 minutes is the sum of all interactions, over initial read, and 2-3 comments (on average). I would also like to caution taking these numbers as anything other than a sign of "what happened in Beta 1". We are definitely working on adding features (like decline re-eval, and comment templates) that can lower the amount of time that reviewers spend per interaction. And I am definitely expecting that Beta 1 reviewers were spending more time than folks would after a wider release, simply due to their enthusiasm for exploring the new features and spending more time figuring things out.
    – Yaakov Ellis StaffMod
    Commented Mar 6, 2023 at 22:19
  • 2
    Also, I don't know if you can extrapolate even from these numbers that there are 15-20 minutes spent per approved question. Posts that auto-graduated got almost no comments and review time. And of the deleted posts, most of them got significantly less reviewer time. Also, posts that were ok to be approved with no major changes (which are part of Approved) take much less time. The posts that take 5-6 minutes are posts that start off with a blocking review, and end up either approved, or abandoned (after a few feedback rounds).
    – Yaakov Ellis StaffMod
    Commented Mar 6, 2023 at 22:20
  • @YaakovEllis Sure there are variances but didn't you report average durations? I simply divide an average by a percentage. On average it really should be 15-20 minutes on total effort to produce one graduated (approved) question. Just think of the whole system as a blackbox with an input (reviewer time) and an output (approved questions) and relate input to output on average. I don't see where I would go wrong. Commented Mar 7, 2023 at 6:38
  • @KevinB "spend more than 30 seconds" 30 second really might be too optimistic. After all even reading alone of a question takes time, not to mention thinking about it and commenting on it. A couple of minutes sounds actually realistic especially if you need to come back and do multiple rounds of review per question. Maybe 3-4 minutes instead of 5-6 would be possible, but only if you are super efficient and focused. Commented Mar 7, 2023 at 6:43
3

No more "vote as off-topic"?

I see that the "Vote as off-topic" radio button option is grayed out in one of the screenshots of reviewer actions. Is that going to stop being part of the workflow? What will we do for blatantly off-topic posts?

1
  • 10
    You can still vote posts as off-topic (though if the post is just low quality, Major changes is the appropriate vote). The screenshot has it grayed out because it was taken from a workflow where the current user had already voted off-topic (so they can't vote again).
    – Yaakov Ellis StaffMod
    Commented Mar 6, 2023 at 13:52
3

I would like to give a feedback from the "newcomer" point of view.

I think some other newcomers will share my impression: I don't understand this feature, or at least the point in forcing random people to use it. To tell my experience, I created my Stack Overflow account five months ago, and the site was more and more pleasant to me. I found it was a good place to exchange between experts, and to help beginners. The only little oddity was the need to have "reputation points" to do some actions that seem to be harmless, like adding a little comment under posts, but that was not a serious problem (though it may be awkward to have to write a loud post just to say a little thing when other members have the right to access a tinier zone).

But now I cannot post questions anymore, because I am trapped in this Staging Ground against my will. Contrary to members having high scores, I am considered as somebody not able to take part in the site's activity without a guide, which is very frustrating. And there is not any clue about when my questions will be reviewed, and if somebody is treating them. I have no visibility on what is going on.

Please, get esteem for everybody, including the newcomers. Allow us to opt out from this beta testing. I might try to help people here sometimes, but honestly I mainly came here to ask for help in a professional context, so if you block my questions with this new feature, Stack Overflow becomes useless for me...

Don't take my words as a rage against the site; I really like Stack Overflow, and this is why I think it is good to give a feedback about it. I wish I see it evolving and becoming better day after day.

10
  • "I am considered as somebody not able to take part in the site's activity without a guide, which is very frustrating" we can't have nice things because so many people don't read our guides. "And there is not any clue about when my questions will be reviewed" If nobody reviews your post within a certain period of you posting it, it will be auto-published to the main site.
    – starball
    Commented Mar 20, 2023 at 23:50
  • Ah, okay, so this is not so harsh. Do you have an idea about this period duration?
    – Morgana
    Commented Mar 20, 2023 at 23:52
  • 2
    If you write a good question as defined in How to Ask, your posts will very likely be approved in Staging Grounds very quickly- as soon as a reviewer looks at it. So aside from having to wait for someone to look at it, I don't see what else the problem is.
    – starball
    Commented Mar 20, 2023 at 23:53
  • Well, maybe problem is a huge word. Let's say it makes me feel like facing a regression, since now when I posted a question I had an answer quickly and there was really no problem. But now I don't even have the guarantee my question will be published. This makes me uneasy with the site.
    – Morgana
    Commented Mar 20, 2023 at 23:56
  • 2
    If you read the question post here carefully, you'll see it: "posts that auto-graduated (received no reviews within two days of being created)"
    – starball
    Commented Mar 20, 2023 at 23:56
  • Ok I may have read too quickly the description, I didn't get the equivalence graduated = published. Thanks for your explanations.
    – Morgana
    Commented Mar 20, 2023 at 23:58
  • 2
    "I posted a question I had an answer quickly and there was really no problem" again, we can't have nice things because people don't read our guidelines. "But now I don't even have the guarantee my question will be published" If your question follows the guidelines in How to Ask, it should be published. It should be virtually guaranteed that any question following the guidelines in How to Ask will be published.
    – starball
    Commented Mar 20, 2023 at 23:58
  • What do you mean by "other members have the right to access a tinier zone"? Can you elaborate? Commented Mar 31, 2023 at 8:46
  • Hi Peter Mortensen, I was talking about the "Add comment" button that always works for our own posts and in the topics we created, but not on other people's post in other topics until we have 50 points. For example, I just wanted to add an extra info to the accepted answer here, but this was not possible for me, so I had to create a whole new post : stackoverflow.com/questions/949196/…
    – Morgana
    Commented Mar 31, 2023 at 13:24
  • 1
    I gave you a +1 as it's good to hear from users on the other side. You're probably in a unique position of somebody who came in just as the staging ground started. It looks like some of your complaint is about commenting, which isn't directly related to this topic, but in both cases posting a new answer with additional information is the correct behaviour. Commented Apr 5, 2023 at 11:34
3

Did you consider tags?

I may have missed something in this or the previous posts, but has the fact some tags get little traffic been taken into account?

Either in the definition of success (it's a success, but no-one on the main site votes up or answers it as it's niche) or auto-graduation (it's not properly reviewed as it's too niche a topic for issues to be spotted)?

7
  • 1
    I mean, it's certainly a known problem, and people do tend to still review within their own set of tags that they're comfortable with. Questions auto-graduate after 24 hours of no activity if they haven't been marked as needs major improvement... so this shouldn't really be a problem. Worst case scenario a user waits 24 hours for their post to show up, only for it to be closed after that when someone finally reviews it outside of SG.
    – Kevin B
    Commented Apr 5, 2023 at 14:45
  • 1
    Loosely related: /help/staging-ground-reviewer-guidelines: "If you know what a good question looks like, you can review anything. You don’t have to be a subject matter expert on the question in order to review it. If it looks like it has the elements of a good question, trust your gut and go with it, even if you are not an expert on the subject being discussed."
    – starball
    Commented Apr 5, 2023 at 16:42
  • @KevinB but that skews the stats towards Auto-Grad for some tags and not others. Which maybe isn't a problem, but it's something to consider Commented Apr 5, 2023 at 20:47
  • 1
    yeah i adamantly disagree with that guideline, but it's meant to allow a smaller review base to handle more incoming questions with less gaps. You can't realistically expect reviewers reviewing something they know nothing about to be able to identify that it's probably a dupe and go looking for it.
    – Kevin B
    Commented Apr 5, 2023 at 20:51
  • @KevinB maybe some niche tags shouldn't be put into the SG? Commented Apr 5, 2023 at 21:11
  • I would prefer... the SG to have a sort of... "niche tags" filter so that those who'd like to assist in that area can just filter to any question not tagged with a popular tag, as there's still plenty of reasons outside of "duplicate" that non SME's can easily identify. the Duplicate problem is probably far less of a concern for low traffic tags.
    – Kevin B
    Commented Apr 5, 2023 at 21:14
  • 1
    or even a... "These things are about to be published and haven't been viewed yet" filter, that'd handle all cases more effectively without having to decide what is and isn't a popular tag. I'd rather have someone look at it than noone, even if not an SME.
    – Kevin B
    Commented Apr 5, 2023 at 21:34
0

The post provides a comparison for questions that are Approved in the Staging Ground versus a baseline of first questions using the regular workflow. The choice of using Approved posts fits with the goal of the Staging Ground "to help improve the quality of first posts that are published on the site" (emphasis mine). The statistics show that the quality is indeed higher by quite a large margin.

I think it would also be useful to look at the value of the Staging Ground as a curation mechanism, i.e., as a way to improve quality of posts that are submitted to the site. Currently these are handled by comments asking for clarification, voting to close and reopen, deleting through Roomba etc. The value of the Staging Ground would be much more apparent if it manages to increase the (average) quality of all posts that enter Stack Overflow, because the personalized feedback system is more effective, even if this comes at the expense of the amount of time spent for reviewing/curating being larger.

Conversely, if a similar number of posts is "approved" through the Staging Ground as through the existing workflow, i.e., by leaving a post open, upvoting and answering, then the Staging Ground does not have added value (and may be detrimental because of the required time investment).

Here is an attempt to calculate some statistics on all posts submitted to the Staging Ground (Inactive + Approved + Auto Graduated), from the numbers in the post. I left the deleted posts out of the comparison because they are not needed to make the point of this answer, except for the Deletion table.

Group Total # Open Conversion Rate
Baseline 25,156 16,559 65.83%
SG I+A+AG 677 235 34.71%
Group Total # Success Conversion Rate
Baseline 25,156 3,784 15.04%
SG I+A+AG 677 78 11.51%
Group Total # Deleted Conversion Rate
Baseline 25,156 3,947 15.69%
SG all 901 246 27.30%

Judging from these numbers the conclusion would be that the Staging Ground does not work as a curation mechanism. Moreover, the quality of the Approved posts may be higher that in the regular workflow but you could with some arbitrary definition of "approved" also select the same or a higher percentage of regular posts that are approved and open/successful. Any thoughts?

14
  • why not compare the baseline with just manually approved SG items? inactive posts usually aren't good (the asker doesn't care to improve), and automatically approved ones are... automatically approved (my understanding is that they're basically kicked out of the SG system with an artificial "thumbs up" to keep wait-time low for the asker)
    – starball
    Commented Mar 4, 2023 at 19:58
  • 1
    @user that is the whole point, the inactive, deleted and automatically approved posts enter the Staging Ground but the SG system is not able to turn them into good posts. Therefore, the goal of improving the quality of the posts has failed for these three categories. I think it is more informative to compare the baseline to the total number of posts in the Staging Ground to see if the overall quality of posts actually improves, not just if the good-quality (i.e., approved) posts are good quality.
    – Marijn
    Commented Mar 4, 2023 at 21:57
  • 6
    Auto-graduated posts go into the First questions queue (approved posts do not)
    – Yaakov Ellis StaffMod
    Commented Mar 4, 2023 at 21:58
  • And inactive posts don't get published from SG (SG "gatekeeps" them from reaching the main site) if I understand correctly. @YaakovEllis is that correct?
    – starball
    Commented Mar 4, 2023 at 22:00
  • 2
    @user I do see some value in the gatekeeping, because the posts are not able to be answered, which may educate the users to ask better questions and prevent setting bad examples for future askers and answerers. However, the current workflow can also handle this by downvoting and deleting. It would be better (and more convincing to show the value of the Staging Ground) if those posts actually got improved instead of remaining inactive, more so than posts in the regular workflow.
    – Marijn
    Commented Mar 4, 2023 at 22:04
  • 2
    If a post is not yet answerable (i.e. has problems other than cosmetic things, such as missing a MRE or other clarifications), then whether it gets improved is up the the owner of the post (the asker/author). SG cannot change the nature of people. Nothing can make people who don't want to put effort into asking an answerable question magically do that. So I don't understand why that should be counted against SG. My understanding was that SG is there to provide a "safe" space for posts to be improved before being shown to people who can answer.
    – starball
    Commented Mar 4, 2023 at 22:09
  • 2
    Ex. No more long comment sections on the main site with back and forth asking for more details that are no longer needed after corresponding edits are made, and no more needing manual cleanup by contributors or mods. Also less feelings of frustration due to downvotes and focus on specifically what needs to be improved.
    – starball
    Commented Mar 4, 2023 at 22:11
  • The "safe space" part was one aspect, but increasing overall question quality also was a goal (I'll see if I can find a quote on that). Because the feedback is personal with a reviewer assigned to you specifically, the idea was that people would engage more in improving their question, so indeed to change the behavior/nature of people.
    – Marijn
    Commented Mar 4, 2023 at 22:12
  • @user here is a quote from Yaakov on the original announcement: "a change to the overall onboarding experience for new users on the site - proactively trying to help with that experience (and quality)". My answer was written from the "and quality" angle. However, you could also argue that if so many posts are inactive/deleted/auto graduated then the "experience" part also doesn't work that well.
    – Marijn
    Commented Mar 4, 2023 at 22:20
  • I stand by the rest of what I said in the same comment I made about "safe space". The experience is a joint effort. If the asker refuses to participate in improving their post, then any perception of poor experience is at least partially contributed to by themselves. I think it's fair to count inactive posts out of the calculation of how quality improves for posts going to the main site.
    – starball
    Commented Mar 4, 2023 at 22:23
  • @user I agree with you that the safe space can be helpful for both experience and quality. However, if the regular workflow actually produces the same or a higher number of good questions in the end then it seems the helpfulness is limited. Also the post by Yaakov today specifically emphasized the quality aspect for approved posts, which I thought could/should be extended to the quality of all posts, regardless of the user experience aspect.
    – Marijn
    Commented Mar 4, 2023 at 22:32
  • 4
    "if the regular workflow actually produces the same or a higher number of good questions in the end then it seems the helpfulness is limited" - I think that part of the point is what @user is saying - the natural gatekeeping that occurs when authors elect to abandon or delete their posts (and they never leave the SG) means that not only are the questions being published of higher quality, but the number of high quality posts are a significantly higher proportion of all new posts. Much higher signal-to-noise. To me, this is a big improvement.
    – Yaakov Ellis StaffMod
    Commented Mar 5, 2023 at 8:15
  • @YaakovEllis except that the noise is still there, it is just left to the SG reviewers to handle it instead of the regular reviewers in the First Question (etc.) queue.
    – Marijn
    Commented Mar 5, 2023 at 12:24
  • 3
    @Marijn correct - the noise is there, but it is only shown to reviewers, not on all of /questions listings (where first questions show, even if they are in the FQ). And reviewers are given tools that can hopefully help expedite the process of approval/feedback. Changing the quality of incoming questions is not something that we are attempting to fix here in a major way (though we do plan to increment on the Ask Wizard to help nudge things into a better place). But overall QOL for the main site, combined with higher quality first questions, is something that we think can be achieved.
    – Yaakov Ellis StaffMod
    Commented Mar 5, 2023 at 12:31
-1

How did you decide on 'Success'

You said:

we considered a post successful if the question has a score of at least two, the total of all answer scores is at least two, or has an accepted answer.

And while that does sound successful, it's also arbitrary and the sort of thing that could sway statistics.

At the same time a post could be a success if the asker finds a help duplicate or a review comment points them to a useful resource only for the question to be abandoned.

I also wouldn't be surprised if there're some 'successful' questions where the two answer criteria is met, but still doesn't solve the question (but it's well received by the community, so maybe that's successful enough?).


In the same vein, who is this 'Success' metric aimed at? Is it enough that it's well received by the community, or should we look at how many Asker's in the SG go on to ask later questions that either don't need SG review, or auto-graduate?

It's an interesting topic, but I'd like more detail on the details if possible?

2
  • Success is purely looking at: are there good signals that the question was a good contribution to the site. So we are looking at votes on the question, whether the question elicited answers that got votes, multiple answers, or were accepted. We know that these are not objective standards, and that it is easy to find counter-examples (people are helped by closed questions, badly phrased questions can still have good answers, etc). And we also know that for a question to be successful requires actions beyond the control of a SG reviewer.
    – Yaakov Ellis StaffMod
    Commented Apr 13, 2023 at 12:05
  • 1
    "it's well received by the community, so maybe that's successful enough" is a good summary. These are strong signals, and while there is not a direct path of causation (and there are other factors at play), there is a very high correlation between questions of higher quality successful questions.
    – Yaakov Ellis StaffMod
    Commented Apr 13, 2023 at 12:07
-5

When will the "New comments UI and nested comments" feature be released to the public community? It looks nice.

4
  • 15
    Currently, this is only for the Staging Ground. I doubt it will ever be released to the public site. I, for one, feel strongly that it should not. There is already too much focus on comments on the public site, to the point where the majority of new users aren't even able to differentiate between comments and answers. This is extremely harmful for our mission of being a Q&A site. The design mockup does look nice, and has some advantages when comments are being used as a means for having a discussion, but that's not what comments are for on public Q&A, and it takes up way too much room.
    – Cody Gray Mod
    Commented Mar 5, 2023 at 7:19
  • 11
    I agree with what @CodyGray writes inasmuch as if/when some iteration of the new comments ui and/or nested comments ends up on the public site, it will need to be wholly reconsidered based on the needs and challenges of the public sites, which overlap — but in some cases differ significantly — with the needs of the SG. I think that there are some scenarios (especially things like meta discussions) where nested can come in handy. But there are many things that need to be considered before this. (1/2)
    – Yaakov Ellis StaffMod
    Commented Mar 5, 2023 at 8:18
  • 13
    I can say that we are building this new component in such a way that it could potentially be adapted and extended for use on the public site sometime in the future. But we are definitely not going to just turn it on for public, just because it is available. It will need its own discovery, (re)design process, and testing process. That said, after having used it in SG, my personal opinion is that it has the potential to be very impactful on public, if done properly. (2/2)
    – Yaakov Ellis StaffMod
    Commented Mar 5, 2023 at 8:20
  • And if it is really necessary, it is probably a good idea to move comments to chat/create separate chatrooms for this anyways
    – dan1st
    Commented Mar 5, 2023 at 9:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .