33

TL;DR: In this post, we will recap the launch of Staging Ground and share updates, bug fixes, and early metrics. We will also discuss the topic of question quality, where we want to head, and how you can help.


Several weeks ago, we launched Staging Ground, the second generation of our sheltered contribution environment, to general availability for new askers and reviewers. Thank you to everyone who has participated and contributed feedback to further improve the feature. Without your efforts, we would not be able to experiment with targeted experiences like Staging Ground to improve the new asker’s experience.

A short recap of the launch

Here are some early metrics that we are monitoring:

Metric Definition Result (June 4 to July 2)
Staging Ground questions Questions submitted 8,498
Graduation rate Percentage of Staging Ground questions posted on main site 51% (4,318)
Number of Staging Ground askers Unique number of askers who submitted questions 8,143
Number of Staging Ground reviewers Reviewers who completed at least 1 review action 1,523
Question survival rate Percentage of graduated Staging Ground questions that remain open within the first 7 days of creation (Baseline: Non-Staging Ground questions that remain open within 7 days of creation for questions created Jan to Apr 2024) 83.5% (Baseline: 71%)
Question success rate Percentage of graduated Staging Ground questions with an answer or post score of at least 2 (Baseline: Non-Staging Ground questions with an answer or post score of at least 2 for questions created Jan to Apr 2024) 39.4% (Baseline: 35%)

We’ve also deployed a number of updates and bug fixes:

  • Updated reviewer entry criteria to eligible users with at least 500 reputation

  • Updated bronze “Guide” badge to include progress tracking

  • Fixed logic for displaying Staging Ground questions on homepage and question list pages

  • Fixed review history filter not preserving when paginating

  • Fixed accessibility issues on timestamp, activity notice, asker/new contributor labels, and reviews completed progress bar

  • Fixed cannot reply to comment if parent comment deleted

  • Hide edit link for reviewers when post review locked

  • Cleared validation message for review actions requiring comments after submitting comment

  • Prevent reviewers from making edits during post locks

  • Update primary call-to-action for askers where post is already in re-evaluate status

The importance of question quality

In the announcement post, we mentioned taking a deeper look at what makes a “good” quality question. This question is fundamental to nearly everyone who uses Stack Overflow.

From the existing guidelines on how to ask a good question, a good question improves the chances of getting an answer which we can infer that the question quality impacts its likelihood. With Staging Ground, we are also looking to achieve that outcome as a result of improving the new asker’s experience.

Many community contributors have intrinsic motivation, a natural desire to help others on the platform. They want to help askers write better questions in order for them to receive better answers. As such, the desire to maintain quality and add another artifact to the library is important to them. However, how can we say quality has been improved if we do not have defined objective measures?

Defining and measuring question quality

In order to achieve our desired outcome of understanding, measuring, and improving question quality, our near-term goal is to define and have a baseline for what question quality means on the platform.

We acknowledge that defining and measuring question quality is difficult and nuanced because it can be subjective in nature. It is something we have been trying to do for many years in order to have a better pulse on the platform’s content health. There have been previous attempts made in discussions such as “How is question quality measured in A/B tests?” where a simple question grade was used to measure quality based on the question’s score and whether or not the post was closed. Based on the previous learnings, it is nuanced for questions that were graded neutral as they don’t give you any quality signal because users did not or could not vote on them.

Key questions we aim to answer with the help of research:

  • Why are certain questions on the site considered good and well-written?

  • What are the characteristics/components of a question?

  • What criteria should be used to evaluate whether a draft is likely to be successful?

  • What do reviewers prioritize when evaluating new questions?

  • How do reviewers decide which questions need improvement?

More than a decade ago, there was a highly popular discussion about how to write a good question with a checklist of questions to evaluate. We found similar results from a research study we conducted recently with review queue and Staging Ground beta participants to understand the specific tasks they perform to evaluate questions, learn their processes, and understand where the most time was being spent. We found similarities to their approaches in evaluating questions so here’s our first attempt at abstracting them into three broad categories:

Category Definition / components
Context and background 1) Provides sufficient detail to understand the problem 2) Presents background research 3) Incorporates a minimum reproducible example (MRE) in order for others to understand and replicate the issue
Expected outcome 1) Provides detail on the outcome, results or intended goals to achieve 2) Documents previous attempts to solve the problem
Formatting and readability 1) Provides formatting to ensure clarity and ease of understanding the problem 2) Contains proper grammar and spelling 3) When appropriate, provided code blocks are formatted properly and easy to understand

Where we are heading with question quality

You might ask where we are heading once question quality has been defined and what this means for Staging Ground. We will continue to learn from and work with reviewers and askers to improve the new asker’s experience and the overall question quality.

In the context of Staging Ground posts, the near-term outcome of this work is as follows:

Clearly defined objective measures of question quality signals. In the long term, we want to extend this to all questions on Stack Overflow. Explore in-product mechanisms to collect signals from knowledge experts in the community. Future roadmap plans for personalization and question ranking efforts to further assist contributors and knowledge seekers.

How you can help and thank you in advance

With the recent research and our initial attempt to abstract question quality into the aforementioned categories, are these the right ones? Are there any missing that you feel are more important and why? What does question quality mean to you?

In the coming weeks, we will be conducting additional research from community members – who really are the best placed to judge this – to learn how they would evaluate and rate questions using the above categories. This will give us a baseline to learn from and understand the similarities or differences in reviewer behaviors when presented with a diverse set of questions.

Participate in research studies

If question quality is important to you and you have participated in review queues or Staging Ground, we would love to hear from you. Reply as a comment to this post with “Sign me up for research” which allows us to prioritize reaching out to you on upcoming research on this topic.

Thank you again for your efforts and dedication to fostering collaboration.

16
  • 5
    Not related to what you ask, hence a comment, but are there any plans to extent the SG to have 100% of posts that would be eligible go to it? I, at least, see a non-small amount of posts going straight to the site from users that could have been sent to the SG, and a large portion of these don't meet quality expectations (5/7 of my close votes in the last 24h were on posts for users that could have gone to the SG, 6/7 are closed and the remaining post is at 2/3 VTCs).
    – Thom A
    Commented Jul 8 at 15:28
  • 4
    Also, as daft as it sounds, the Discussions feature feels like it could sorely do with the SG feature; far too much of the content there by 1 rep users should never had been posted there.
    – Thom A
    Commented Jul 8 at 15:29
  • "Percentage of Staging Ground questions posted on main site"; Is there any insight on how much questions are still open/available today? I've seen questions getting posted from SG to main site and then still got closed/deleted afterwards.
    – 0stone0
    Commented Jul 8 at 16:03
  • 24
    In fact, can you add stats of question that didn't go to the SG that could have, and the success rate of those questions? I feel that's important context.
    – Thom A
    Commented Jul 8 at 16:09
  • 5
    Is the baseline for both "question survival rate" and "question success rate" supposed to be "Non-Staging Ground questions closed/deleted within 7 days from 2024 to date"? For the former, it seems like it's backwards — one would think it should be the percentage not closed/deleted — and for the latter, it just seems like a copy/paste error. Unless I'm missing something? Commented Jul 8 at 16:17
  • 1
    Sign me up for research
    – pierpy
    Commented Jul 8 at 17:01
  • 1
    Sign me up for research - I'd be happy to help when it comes to improving question quality.
    – dan1st
    Commented Jul 8 at 18:09
  • @ThomA "the Discussions feature feels like it could sorely do with the SG feature" - Do you really think these posts can be improved? To be hones, I think the Staging Ground is more of an area for improving questions than a filter.
    – dan1st
    Commented Jul 8 at 18:11
  • 1
    @dan1st I'm resigned to the fact that Discussions is here to stay. There's posts that go to the SG as well that shouldn't even get the the main site. Maybe we could have a process to redirect SG Discussions to SG questions when they might be on topic. Currently the tooling is ****, the experience to both established and new users is ****, and the entire thing is... Yeah, you get the idea... I'm just throwing things at the wall and seeing what sticks.
    – Thom A
    Commented Jul 8 at 18:21
  • I just think discussions are not at all worth the effort.
    – dan1st
    Commented Jul 8 at 18:24
  • 1
    If you think you can come up with an answer for objectively measuring how well a piece of text communicates the underlying issue, I'm sure their is a lot of money to be made there, both in natural language and code readability tooling. Commented Jul 8 at 18:56
  • It's only natural that a baseline of "all posts are published" vs "only good posts are published" will result in a higher "success" rate for "only good posts are published", if success means posts aren't deleted in 30-60 days, but i don't think that's really saying much in terms of how effective the SG really is at improving the outcomes of questions. If you instead compared how successful the successful posts in each group are, and similarly compared their quantity in a given period, you'd have a more clear picture of how good (or bad) the result of SG really is.
    – Kevin B
    Commented Jul 8 at 19:00
  • Is the baseline also computed over new users (fitting the criteria for SG)?
    – Passer By
    Commented Jul 9 at 6:59
  • 2
    @ThomA Replying to some of your questions. Yes, there are plans to work towards being able to route all eligible Questions, for eligible new askers, through Staging Ground in the future. The baseline metrics shown in the table (35% success, and 71% survival) provide a snapshot of those stats for questions that were asked directly on the main site (in other words, ones that did not go through Staging Ground).
    – Sasha StaffMod
    Commented Jul 18 at 17:27
  • 3
    @0stone0 Out of the 5,800+ graduated posts to date so far (June 4 to July 16), about 13% (730) were closed post-approval onto the main site. The close reason for the majority of them was either off-topic or duplicate. We have upcoming plans to improve the duplicate detection feature to help alleviate some of the burden on Staging Ground reviewers, so they can focus primarily on providing feedback instead of searching for duplicates.
    – Sasha StaffMod
    Commented Jul 18 at 17:30

2 Answers 2

14

Please include a rigorous and clear definition of what baseline is and where it's taken from.

(Sorry to be nitpicking, but way too many authors write technical papers without defining what they're talking about. Peer reviewers are to blame in no small part.)


After looking at it again, it seems the baseline is:

(Baseline: Non-Staging Ground questions closed/deleted within 7 days from 2024 to date)

So what this means is questions that got past the SG reviewers (with no obvious closure/issues going uncorrected at that stage) and after being posted on main the community/SMEs then closed/downvoted/deleted the Q in the 1st week (with roomba having no part)?

I fail to see how this measure of insuccess is then correlated with two measures of success:

  • Without SG 71% get deleted/closed in 7 days but with SG 83.5% make it past the 1st week? (This conflates the 49% that get filtered at SG?!)
  • Without SG 35% get deleted/closed in 7 days but with SG 39.4% get answered/+2 upvotes. (Someone would have to explain how this last one is correlated.)
4
  • 2
    In the stats of Beta 1, the baseline was "first questions on the public site that did not go through the Staging Ground".
    – dan1st
    Commented Jul 8 at 18:16
  • 2
    @dan1st I expect the dev to confirm as much and edit the post to include a link. Readers shouldn't be expected to go searching for a definition in a 16 month old post somewhere among a myriad of related announcements.
    – bad_coder
    Commented Jul 8 at 18:35
  • 1
    I am not saying it shouldn't be included. I just added the comment as I think that's the most likely possibility until a staff member maybe decides to respond (or not).
    – dan1st
    Commented Jul 8 at 19:47
  • 4
    There were some mischaracterizations in the original post’s explanation of the baselines, which are responsible for the confusion here. Sorry about that, they have now been corrected. The baseline for the Question survival rate is non-Staging Ground questions that remain open within 7 days of creation (in Jan to Apr 2024). The baseline for the Question success rate is non-Staging Ground questions with an answer or post score of at least 2 (in Jan to Apr 2024). So we see that running questions through Staging Ground increases their chances of both survival and success, by these measures.
    – Sasha StaffMod
    Commented Jul 9 at 15:45
9

However, how can we say quality has been improved if we do not have defined objective measures?

/help/how-to-ask is kind of that.

There's still a lot of less obvious stuff there that comes from experience knowing how to ask good questions, but I wager that a lot of that isn't something you can compute/calculate, which it sounds like you are looking for some way to do. I personally am doubtful how much headway you can make in that direction (for that less obvious stuff).

Why are certain questions on the site considered good and well-written?

On-topic, well-scoped, clear, concise, searchable.

What are the characteristics/components of a question?

If you mean useful components, it depends on the question.

  1. Documents previous attempts to solve the problem

Note: Problem-solving effort is not a requirement for how-to questions. I feel compelled to point this out because too many people evaluating questions don't know this. It can help improve the quality of the Q&A, but if there is sufficient detail without it, it can just be noise. Whether there's sufficient detail without info about failed attempts often takes domain-knowledge to evaluate.

Future roadmap plans for personalization and question ranking efforts to further assist contributors and knowledge seekers.

Not clear to me what this means. Can you elaborate?

3
  • 2
    In regards to your last question: Within the first few weeks of reviewing the data, we noticed that there was a high number of skip actions taken by reviewers. From previous research we learned that some reviewers only feel comfortable reviewing questions relating to topics/technologies that they are familiar with. We are interested in validating whether personalization efforts, such as routing relevant questions to reviewers based on what we know about them from past contributions and activity on the site, can positively impact reviewer engagement.
    – Sasha StaffMod
    Commented Jul 18 at 18:11
  • @Sasha We already have review filters. How do your proposed personalization efforts differ from that?
    – wizzwizz4
    Commented Jul 19 at 8:19
  • 1
    @wizzwizz4 On the staging Ground, there are already a more powerful filtering machanism but many users don't get to the Staging Ground via the entry page I guess (instead using the question list). I think Sasha talked about more personalized questions in the automatic queue by default/automatically.
    – dan1st
    Commented Jul 19 at 13:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .