TL;DR: In this post, we will recap the launch of Staging Ground and share updates, bug fixes, and early metrics. We will also discuss the topic of question quality, where we want to head, and how you can help.
Several weeks ago, we launched Staging Ground, the second generation of our sheltered contribution environment, to general availability for new askers and reviewers. Thank you to everyone who has participated and contributed feedback to further improve the feature. Without your efforts, we would not be able to experiment with targeted experiences like Staging Ground to improve the new asker’s experience.
A short recap of the launch
Here are some early metrics that we are monitoring:
Metric | Definition | Result (June 4 to July 2) |
---|---|---|
Staging Ground questions | Questions submitted | 8,498 |
Graduation rate | Percentage of Staging Ground questions posted on main site | 51% (4,318) |
Number of Staging Ground askers | Unique number of askers who submitted questions | 8,143 |
Number of Staging Ground reviewers | Reviewers who completed at least 1 review action | 1,523 |
Question survival rate | Percentage of graduated Staging Ground questions that remain open within the first 7 days of creation (Baseline: Non-Staging Ground questions that remain open within 7 days of creation for questions created Jan to Apr 2024) | 83.5% (Baseline: 71%) |
Question success rate | Percentage of graduated Staging Ground questions with an answer or post score of at least 2 (Baseline: Non-Staging Ground questions with an answer or post score of at least 2 for questions created Jan to Apr 2024) | 39.4% (Baseline: 35%) |
We’ve also deployed a number of updates and bug fixes:
Updated reviewer entry criteria to eligible users with at least 500 reputation
Updated bronze “Guide” badge to include progress tracking
Fixed logic for displaying Staging Ground questions on homepage and question list pages
Fixed review history filter not preserving when paginating
Fixed accessibility issues on timestamp, activity notice, asker/new contributor labels, and reviews completed progress bar
Fixed cannot reply to comment if parent comment deleted
Hide edit link for reviewers when post review locked
Cleared validation message for review actions requiring comments after submitting comment
Prevent reviewers from making edits during post locks
Update primary call-to-action for askers where post is already in re-evaluate status
The importance of question quality
In the announcement post, we mentioned taking a deeper look at what makes a “good” quality question. This question is fundamental to nearly everyone who uses Stack Overflow.
From the existing guidelines on how to ask a good question, a good question improves the chances of getting an answer which we can infer that the question quality impacts its likelihood. With Staging Ground, we are also looking to achieve that outcome as a result of improving the new asker’s experience.
Many community contributors have intrinsic motivation, a natural desire to help others on the platform. They want to help askers write better questions in order for them to receive better answers. As such, the desire to maintain quality and add another artifact to the library is important to them. However, how can we say quality has been improved if we do not have defined objective measures?
Defining and measuring question quality
In order to achieve our desired outcome of understanding, measuring, and improving question quality, our near-term goal is to define and have a baseline for what question quality means on the platform.
We acknowledge that defining and measuring question quality is difficult and nuanced because it can be subjective in nature. It is something we have been trying to do for many years in order to have a better pulse on the platform’s content health. There have been previous attempts made in discussions such as “How is question quality measured in A/B tests?” where a simple question grade was used to measure quality based on the question’s score and whether or not the post was closed. Based on the previous learnings, it is nuanced for questions that were graded neutral as they don’t give you any quality signal because users did not or could not vote on them.
Key questions we aim to answer with the help of research:
Why are certain questions on the site considered good and well-written?
What are the characteristics/components of a question?
What criteria should be used to evaluate whether a draft is likely to be successful?
What do reviewers prioritize when evaluating new questions?
How do reviewers decide which questions need improvement?
More than a decade ago, there was a highly popular discussion about how to write a good question with a checklist of questions to evaluate. We found similar results from a research study we conducted recently with review queue and Staging Ground beta participants to understand the specific tasks they perform to evaluate questions, learn their processes, and understand where the most time was being spent. We found similarities to their approaches in evaluating questions so here’s our first attempt at abstracting them into three broad categories:
Category | Definition / components |
---|---|
Context and background | 1) Provides sufficient detail to understand the problem 2) Presents background research 3) Incorporates a minimum reproducible example (MRE) in order for others to understand and replicate the issue |
Expected outcome | 1) Provides detail on the outcome, results or intended goals to achieve 2) Documents previous attempts to solve the problem |
Formatting and readability | 1) Provides formatting to ensure clarity and ease of understanding the problem 2) Contains proper grammar and spelling 3) When appropriate, provided code blocks are formatted properly and easy to understand |
Where we are heading with question quality
You might ask where we are heading once question quality has been defined and what this means for Staging Ground. We will continue to learn from and work with reviewers and askers to improve the new asker’s experience and the overall question quality.
In the context of Staging Ground posts, the near-term outcome of this work is as follows:
Clearly defined objective measures of question quality signals. In the long term, we want to extend this to all questions on Stack Overflow. Explore in-product mechanisms to collect signals from knowledge experts in the community. Future roadmap plans for personalization and question ranking efforts to further assist contributors and knowledge seekers.
How you can help and thank you in advance
With the recent research and our initial attempt to abstract question quality into the aforementioned categories, are these the right ones? Are there any missing that you feel are more important and why? What does question quality mean to you?
In the coming weeks, we will be conducting additional research from community members – who really are the best placed to judge this – to learn how they would evaluate and rate questions using the above categories. This will give us a baseline to learn from and understand the similarities or differences in reviewer behaviors when presented with a diverse set of questions.
Participate in research studies
If question quality is important to you and you have participated in review queues or Staging Ground, we would love to hear from you. Reply as a comment to this post with “Sign me up for research” which allows us to prioritize reaching out to you on upcoming research on this topic.
Thank you again for your efforts and dedication to fostering collaboration.