30

This post is following up on some final topics that we want to cover during the product discovery and early development stage of the Staging Ground. We are building here off of the framework that we have already described in the following posts (be sure to read at least the first two before proceeding here):

  1. Workflow 1: Question Details & Actions

  2. Workflow 2: Listings, Filters, Quality Control, and Notifications

  3. New User Experience: Deep Dive into our Research

  4. General introduction

  5. How will the Staging Ground & the new Ask Wizard work on the network? (MSE)

A word of introduction: All of the topics being raised below are forward-looking, anticipated issues that we will have to contend with if the initial testing of the Staging Ground is successful enough to allow us to proceed for planning for feature graduation and for a general release on Stack Overflow (and other Stack Exchange sites that are interested in using it). We are sharing these here to let everyone know that these are areas that we are already thinking about, as well as to get your opinions on the magnitude of these issues, on our current direction, and on other possible solutions that you can think of.

Reviewer Motivation

Even though the Staging Ground is not a review queue, both sections will draw on the same pool of users to serve as Reviewers. For review queues, we have a very direct correlation between the number of Reviewers who are active on a queue and success of the queue (in terms of percentage of items reviewed, and timeliness of reviews).

For example, taking the First questions queue for example (as it is the closest in terms of purpose to the Staging Ground), since it launched in August 2021 we have had anywhere between ~1300 and ~2600 unique reviewers in the queue in a given month (with average actions per reviewer/month varying between 18-27). A drop in a single month of even a few hundred reviewers has spelled the difference between the review being at manageable levels of a few thousand posts (with most new posts being seen within a few days or even hours) and a vast increase of queue size (with a concomitant delay in the time needed to review for new items).

With the Staging Ground, we are asking more from Reviewers. While some posts will be Good to go at the outset with no changes needed, on other posts we will be asking Reviewers to advise on improvements needed from the Author before the question can be published on the site, and then to potentially have continued interactions with the Author, providing additional guidance where necessary. Additionally, since a user’s first questions won't be seen on the public site until approved through the Staging Ground, we will place a higher priority on getting items seen by Reviewers as expeditiously as possible.

Though we won't really know what the amount of time a Reviewer will need to spend per item (and how much this will differ for different quality questions that are coming in), and how many items a typical Reviewer will handle per day, it is fairly obvious that we will need at least as many, if not more, Reviewers (perhaps by a significant degree) to be active in the Staging Ground than we have had in Review queues, in order to ensure the success of the section.

We will be ensuring much increased visibility for new Staging Ground questions by interspersing them in some of the questions listings on the site, which should directly give us a larger number of eligible users trying out reviewing in the Staging Ground at least once. But we also need to consider what would motivate Reviewers to continue to help in the Staging Ground once they are there, so that we can increase the percentage of Reviewers who will return multiple times.

Badges

We are planning to offer a silver participation badge for any user that performs review actions on 10 Staging Ground questions during the initial testing period.

If the Staging Ground graduates, we plan on a badge system similar to Review queues: bronze, silver, and gold badges based on the aggregate number of questions reviewed, with the ability to earn multiple gold badges (exact numbers required for each badge are still to be determined).

Reputation Rewards

An idea that we have been playing around with is to offer a reputation incentive for Reviewers. This could be offered specifically on questions where the following sequence occurs:

  1. A Reviewer interaction on a question in the Staging Ground of marking it as requiring Major changes or putting it in one of the Closed conditions and leaving a comment…

  2. leads to edits by the question Author…

  3. which then leads to approval of the question in the Staging Ground and subsequent publishing of the question publicly…

  4. and finally to the question meeting some criteria for success on the site, at which point the Reviewer would be rewarded with some amount of reputation for their effort (most likely 10 or 15 points).

    1. Success is tentatively defined as the question score or aggregate answer score over two (excluding votes by any Reviewers of the question in the Staging Ground), an answer being accepted, or the question reaching a certain number of views.

    2. The extra rep could potentially be awarded above the daily rep cap. It could be unlimited or subject to a daily limit of its own.

The goal here would be to reward the extra time invested in guiding the new user to successfully transforming their question from one that was initially not up to site standards into one that performs well on the site (and hopefully, in the process of doing so, also helping to improve the skills of the new user as well and increasing the chances of them becoming a successful repeat contributor and curator in the future).

We know that rewarding site activity with reputation is potentially very controversial. So we want to tread carefully here (and are definitely interested in your feedback). Here are some things that we are aware of that could evoke some debate.

Rep Points for non-content:

  • Reputation points are given on the site mainly for contributing content (questions and answers) that is on-topic and helps others. So awarding rep from participating in the Staging Ground is potentially disruptive to the rep ecosystem and could be too big of a change from site norms.

  • On the other side of this argument, we already award rep for users (below a certain rep cap) who have their suggested edits approved. While this is still related to content, it is a little bit further removed from the direct authorship of content that question and answer authors receive their rep for. So this could be seen as something of a precedent for rewarding experienced users who are able to guide others in the process of creating good content.

  • I would argue that the return on investment of new rep into the overall site ecosystem is much higher here than it is for a vote or two received elsewhere, in that help provided to new users can have a magnifying effect for all of that user’s future contributions.

Potential for abuse:

  • Adding in a rep incentive could also draw in more users who will Review in the Staging Ground in bad faith in search of rep. These users could intentionally reject good posts from new users and leave comments with the goal of subsequently approving the post (which as a good post would have a good chance of meeting the success criteria).

  • This could lead to Reviewers favoring Major edits instead of Minor edits as a selected action on the question, even for posts where the latter option would be more appropriate. Thus, in this scenario, a rep reward could cause the opposite of what we want.

  • On the other side of this argument, we already have plenty of avenues for abuse on the site, and for these we have developed counter-measures and means of detection. This is no different. If folks start to abuse this, we can add additional schemes for detecting this (or at the very least calling mod attention to these users).

We are interested in your feedback on this idea.

Additional forms of recognition

We could potentially add a stat relating to the number of new users helped in a user’s profile.

If you have any other ideas for ways to encourage Reviewer participation and/or grant more recognition, we’d love to hear them below.

Scaling

We are aware that there are potential scaling issues with the Staging Ground. It sets up a system wherein a one-to-one interaction is needed for every new user question. Even with good publicity for the content within the site and an adequate reward system, it is unknown if (and for how long) the Staging Ground can be self-sufficient in terms of getting all or most questions reviewed in a timely manner.

For the initial test release, we will not be building in anything to address this. The purpose of the initial release will be to determine if this setup can work in its purest form, and if so, to iterate on its features and details to bring it to a place that maximizes its chances of success in a wider release. While we are thinking about ways to scale in the future, we are not going to start planning on anything related to that until we see how the Staging Ground itself performs with real users and content, and how much of a need there is for additional scaling. Thus, for the initial release we'll aim for a limited launch that mitigates the risks of scale and we will use the limited launch data to help us better understand those risks. This is a topic that we are taking seriously but deem it premature to address in a complete manner before we have concluded the initial test of the Staging Ground.

Taking this context into account, we are happy to hear your thoughts related to scaling this section in the future.

Anything Else?

This is your place to raise any other questions or thoughts that you have had about the Staging Ground for which there wasn’t a place to raise them yet. As a reminder, we are planning on the first test release of this section in a few months, which will only be open to a subset of users. From there, we will be monitoring, measuring, and iterating until we can make a determination on a final release, and on what additional features and changes would need to be included.

In addition to the topics discussed above, there are still open questions around items like qualification for Reviewers (rep-based or earned through actions), daily review limits (should we have them and how would they work?), audits (do we need to have them?), automatic suspensions based on suspicious behavior detection (how would these work?), just to name a few. All of these are fair game for discussion below. While we cannot promise decisions at this point on anything, we will try our best to respond to all constructive ideas, questions, and feedback.

19
  • 26
    If you go down the reputation route, it might be worth considering being able to more easily report bad reviewers from the review queues themselves. Probably the best allies you have is people who review in good faith; they've already got the eagle vision required to do reviews, so they may well spot people repeatedly making poor choices. The topic of audits should probably be reviewed as well, but maybe as a separate post as it's a... ehm... volatile topic.
    – Gimby
    Commented May 9, 2022 at 14:38
  • 4
    @Gimby could you please post your thoughts as an answer below?
    – Yaakov Ellis StaffMod
    Commented May 9, 2022 at 15:46
  • 24
    "We are aware that there are potential scaling issues with the Staging Ground" - they're not potential. Based on these posts, those scaling issues are going to happen. Introducing rep is most definitely one of the issues that's going to make the staging ground scale horribly. What about bad reviews? Review suspensions? Vote fraud tools? Current vote fraud tools are horrible. Are tools like that even a concern for the first release? How many months or years are we going to have to wait for staging ground-enabled fraud detection tools if they aren't? Articles had months for elementary mod tool Commented May 9, 2022 at 16:16
  • 8
    support, and those systems weren't nearly as complex as vote fraud tools are. If rep is introduced here, you won't just have Documentation 2.0 (articles) running, you'll have it running in parallel with Documentation 3.0 (staging ground). The entire project so far is set up for large-scale failure in just about every piece of the site it touches; review, quality, fraud, and reviewer management. Even if you still insist on not calling it a queue, it's still a form of a review queue Commented May 9, 2022 at 16:19
  • 13
    @ZoestandswithUkraine please write an answer with this, not comments. And to quote myself from above, "we will try our best to respond to all constructive ideas, questions, and feedback".
    – Yaakov Ellis StaffMod
    Commented May 9, 2022 at 16:19
  • 15
    Please, please do not do reputation for reviews - I've been cautiously optimistic about the project, but this kills any faith I had in it. The project needs to be sustainable without blatant external motivation, especially one so broken and controversial as reputation - if it is the only thing that you think will keep reviewers participating (as opposed to the need to help onboard others). Please do not do this to us.
    – 0Valt
    Commented May 9, 2022 at 17:55
  • 3
    @OlegValteriswithUkraine Thanks for your feedback. This would be easier to interact with in an answer below.
    – Yaakov Ellis StaffMod
    Commented May 9, 2022 at 17:57
  • 1
    Yes, I know, @YaakovEllis, just some preliminary thoughts while I am trying to collect my wits after the initial shock of seeing that my fear of "extrinsic motivation" including reputation turned out to be a reality.
    – 0Valt
    Commented May 9, 2022 at 18:00
  • 11
    @OlegValteriswithUkraine please note the message that I tried to make clear above - this is an exploratory post. None of this is scheduled or coded yet, nor are we committed to any of it. Your feedback is what we are looking for here.
    – Yaakov Ellis StaffMod
    Commented May 9, 2022 at 18:02
  • 5
    @OlegValteriswithUkraine I hear. We are entertaining lots of different options, but this is not something that we would implement on a whim, or just as an experiment. Part of the reason that we are socializing it here in this way is specifically to see what folks think about it. And if the consensus among users here is predominantly agreeing with your reaction, then I don't see us moving forward with this.
    – Yaakov Ellis StaffMod
    Commented May 9, 2022 at 18:18
  • 9
    @YaakovEllis speaking of interaction with feedback: you already got tons of feedback in the previous posts, including various suggestions for changes in the proposed workflow. I've not seen much in the way of official responses to those yet - there's comments from various staff here and there but mostly using a noncommittal "we'll take a closer look at this" language. Is taking the past feedback into account still a WIP, or did you change the workflow accordingly, or did you decide to discard the feedback?
    – l4mpi
    Commented May 10, 2022 at 10:14
  • 8
    @l4mpi any place where we said "we'll take a closer look" or "we'll talk about this internally", we actually have. This has resulted in some workflow changes. We will try to relate those back in a future post or on the original post. This post is the last of a series of initial posts, and we have others now planned. And lots of balls up in the air right now. So apologies for not being on top of updates for those. Will try to revisit that soon.
    – Yaakov Ellis StaffMod
    Commented May 10, 2022 at 14:04
  • 7
    "which will only be open to a subset of users." – can one opt in?
    – user17242583
    Commented May 10, 2022 at 14:36
  • 2
    @richardec: FYI: @Panda elevated that question to an answer. Commented May 18, 2022 at 20:02
  • 2
    Small rep or badge reward counter increment for a triage (suggestion major vs minor revision, etc.). If rep is being rewarded, only reward if the triage consensus is achieved, and the question has satisfied an "in good standing" requirement.
    – jxh
    Commented May 24, 2022 at 17:55

14 Answers 14

42

Please do not add reputation as an incentive

  1. The cons described in your post substantially outweigh the benefits. Of particular concern is the motivation users will have to select Major Revisions. This may even influence users with the best of intentions subconsciously.

  2. Similar to Zoe's concerns mentioned in her comments, I am very concerned this system will result in substantial voting fraud. The fraud detection tools available to community moderators of Stack Exchange will require considerable improvements to be able to police this type of fraud effectively.

  3. Finally, the rules you have described for how reputation would be awarded are very complicated. Many users struggle to understand some simple, fundamental features, such as commenting and @-notification. It seems likely that complicated rules (which are required to prevent even more blatant abuse), will hamper potential users from understanding why they are not receiving their expected reputation gains.

3
  • 2
    If fraud produces high quality Q&A pairs...
    – Kevin B
    Commented May 9, 2022 at 18:49
  • 3
    @KevinB "an answer being accepted, or". I'm not sure that's a mark of high quality. Commented May 9, 2022 at 18:50
  • 18
    I'm not really concerned it will, I know it will. It's a thing people can use to cheat the system, and they will use it for that. My concern is that we're not going to get proper tools and systems to deal with it automatically and manually Commented May 9, 2022 at 19:24
28

A(n alternative) suggestion for rewarding successful reviewers

After some number of successful engagements (based on, for instance, similar criteria as defined under Reputation Rewards), provide a badge next to the contributor signature, similar to [♦ Mod]; e.g.,

Jeremy Caney [♥ Mentor]

(Contributors and icon TBD)

Benefit

This would give recognition to people who have demonstrated a commitment to helping new contributors learn how to successfully engage with Stack Overflow, and just might help give them more clout when engaging with other contributors in e.g. the normal review queues.

At minimum, it’s a highly visible acknowledgment that would likely motivate engaged contributors to participate in the not-a-queue—at least up until being awarded the badge—since it provides recognition beyond their engagement with individual posts.

Sustainability

As a variation on this, the badge could be dependent on ongoing engagement—e.g., maintaining the success criteria over [e.g., the last six months]. That way, there’s an incentive to continue participating in the not-a-queue, instead of getting the badge and then drafting off that participation.

Concerns

My main concern with this is that it might give these “mentors” too much clout, by offering a badge similar to moderators. Since we’re working with new contributors, the differentiation may not be noticeable. I don’t think this is too big of a deal.

But, certainly, any accepted flags for Rude/Abusive should remove the status, and disqualify the person from regaining it for [e.g., six months]. The [♥ Mentor] badge should be seen as a symbol of hospitality and support.

Note: I've removed the suggestion of disqualifying the [♥ Mentor] badge upon an accepted Rude/Abusive flag. See @Zoe's comment below.

Caveats

This could be in addition to or substitution of the reputation reward, which I remain agnostic on.

6
  • 1
    This would IMO be better than a rep reward. But "any accepted flags for Rude/Abusive should remove the status" - as "rude" also encompasses "non-welcoming", it might be used (and accepted) for very slight infractions (mods, any insight?). Removing the badge for a single small misstep seems excessive, and could be abusable (e.g. search and flag old comments). Holding the owners to a "much higher standard" also does not seem realistic for something that's intended to increase the number of reviewers. Thus the removal process would probably need to be more complex or be a manual mod action.
    – l4mpi
    Commented May 10, 2022 at 11:31
  • 10
    "any accepted flags for Rude/Abusive should remove the status" - eh, if we go down this path, removing them on any accepted flags isn't a good idea. We some times have invalid R/A comment flags on comments with valid NLN flags. Because comment tools are trash, and the comment has to be deleted, we don't have a choice but to mark the R/A flag helpful, even though it isn't. Considering SE doesn't give any attention to comment moderation systems, it's unlikely that this is fixed in time for auto-removal to make sense. Also worth noting that this would need to work with an invalidation Commented May 10, 2022 at 12:19
  • 2
    system, that also doesn't currently exist for the same reason as fine-grained comment flag handling, for when a mod inevitably makes a mistake. Auto-removal places substantially higher requirements on comment moderation tools, which isn't going to happen any time soon, and consequently, auto-removal cannot work in practice. Commented May 10, 2022 at 12:20
  • 21
    [collective icon] [mentor] bob the builder [mod] All these labels just further draw us away from recognizing users for all the other positive ways to contribute. If only there was a badge-like system, with three tiers that we could display for the user to recognize their accomplishments
    – Kevin B
    Commented May 10, 2022 at 14:20
  • 3
    @ZoestandswithUkraine: Thank you for that insight into the moderation tools and process. That's really useful, both in general, and with specific regard to this feature. Based on that feedback, I've removed that suggestion from my proposal. Commented May 10, 2022 at 22:37
  • 16
    @KevinB: I hear your concern. I think it's reasonable, at minimum, to treat [♦ Mod] and [♥ Mentor] exclusively. That said, I'd actually like to see more statuses contextually elevated from the normal badges. E.g., If you answer a question tagged with javascript, the fact that you have a gold badge in JavaScript is far more relevant to readers than you having 15 gold badges; that should be promoted. (Conversely, I have 34 gold badges, but not a single one is relevant to my answers.) Of course, there is limited space available, and difficult design tradeoffs in how to use it effectively. Commented May 10, 2022 at 22:46
18

I wanted to think about this a bit more constructively, so I delayed a response to this.

In thinking about it, I came to two conclusions.

  • We require an incentive for people to participate in what would be more labor-intense than a review queue.
  • The only things on offer are reputation or badges.

Therefore, my train of thought is that there isn't enough motivation to go around to get people interested in doing this based on what's on offer.

Here's why.

Let's look back at the overall objective of this - to improve the new user experience with onboarding guidance or review. New users are a perennial PR sore point with some of them having a misunderstanding of what Stack Overflow does or how it works, and some may walk away disincentivized to engage with the site once they run afoul of our curation policies. However, this features seems to say that we're not intending to change our policies (which is good!), but we want to ease someone into how to use the site with policies intact.

From reading and re-reading I didn't see anywhere in which we address the obvious weak point of all of this - someone who just doesn't care. In my anecdotal experience on Stack Overflow over ten years, the people who don't care about this site and just want their question answered outnumber the people who can be reached and educated by a factor of 10,000 to 1.

This disinterest can be extended into the actual review queue (yes, I know you're not trying to call it that, but try to see this from our point of view) in that, given specific incentives, a person may or may not be motivated to participate beyond a certain threshold or may look to abuse the system for their own gain.

This is why people oppose the incentive for reputation. There's too much disinterest in the system for people to respect it.

This is what led to the whole kerfuffle with Documentation in that people who participated in it just went and took the super easy low-hanging fruit on technologies that were super well established and described how to do bang-easy stuff in them. Like, for real. No one needed to know how to do arrays in Java for the umpteenth time. Compounding this was Stack Overflow's seeming equivalence in indifference and complacency in that the feedback to address this obvious concern went "ignored". I have to put that in quotes since I can't prove it was "ignored"; it could have been discussed internally and batted around, but from what I saw, no action came of it until it was shut down.

So now we're thinking about offering this again, but we haven't addressed this mutual disinterest yet.

I'll say it like this - personally speaking I don't see how many other levers you have left to pull to get people to care about this, so your best shot would be to offer reputation.

However, you have to be prepared to deal with the issues of doing so.

You need to provide the moderators with the tools they need to curb and limit abuse in the system, unlike with what's going on today. You must commit and double-down on making that a priority to allow the moderators to take action where and when they can.

You need to be aggressive when it comes to people abusing this. Don't offer any mercy for the fact that someone's using this to gain reputation illegitimately - curb stomp that abuse into the ground and don't apologize.

You must be engaged with the volunteers that actually choose to do something like this. Burnout is a very real thing, and if we don't feel like we're getting value out of this, we're not going to volunteer to do it as much or anymore, which leaves the root problem unsolved with a awkward amount of tech debt/low traffic pages in its wake.

If this queue were to come to fruition, and any of the shortcomings pointed out were only half-solved or not even done for the sake of rolling this out MVP style, this will fail spectacularly, and it wouldn't be for lack of feedback or information about what the concerns were.

2
  • "We require an incentive for people to participate in what would be more labor-intense than a review queue." I don't understand why it would be more labour-intensive. Commented Aug 9, 2022 at 0:41
  • 2
    @KarlKnechtel: I've spent my fair share of time in the review queues with a couple of gold badges to show for it. I imagine that it'll be like that except with more than double the amount of time required to establish context and how good the question is, and I also have full-time hobbies now (when I did the queues I didn't). There's also no guarantee that this would actually help people ask better questions. Fundamentally people just want to get their question answered and they don't really care about our site or what the point is when we moderate content.
    – Makoto
    Commented Aug 10, 2022 at 15:16
16

I'm very skeptical that any number of extra editorial passes, with or without a wizard and pizza-tracker, will address this fundamental problem:

There are a lot of posters who will settle only for pasting their code into a box along with the word "help."

This is true for very many reasons, some of which are utterly beyond the control of SE/SO and its community.

Very many of these people do not want and would not participate in a more-involved process, even if it credibly promises to get them results, because their own predetermined "effort budget" for seeking help is effectively zero. These are people who want a process that requires not more engagement from them, but even less, which is probably impossible.

I am skeptical of the conclusions drawn from the user research that suggest novice users want this, because that group almost by definition would not include users who are looking for a zero-effort engagement.


Consequences

Any system that redirects zero-effort questions to a pre-drafting gauntlet will result in the majority of them being abandoned and then re-asked in slightly different form to re-roll the dice with reviewers. Some askers will conclude that SE is no longer a zero-cost answer box, and will look elsewhere. (If this does decrease the number of zero-effort questions on SE, that goal will be accomplished precisely by creating "participation barriers on the site.")

As far as the effects on involved community members, I worry that this asks the community of dedicated reviewers to invest a lot more effort in the questions that are overwhelmingly likely to contribute the least value to the site.

Again, a lot of zero-effort questions are not truly novel: they appear on SE not because the problem is unprecedented, but because the asker wants SE to bring the answer to them rather than using Google to bring themselves to the answer. Do you really find it to be the case that developers struggling with interesting or intractable problems are the ones who resort to vomiting their code into a box and clicking Submit?

SE may gain a new user, but then all SE has really accomplished is to effectively conscript its most dedicated community members into a new-user outreach program. I don't see that it's in the interests of the members of the SE community to expand the SE user base to include every last human being on Earth: we care about the breadth, depth, and quality of the content base, not the size of SE's user base. This is a case where your interests and ours diverge.

I am not impressed with the ROI of having a valued community member spend tens of minutes and potentially a half-dozen exchanges with someone whose question boils down to not knowing how to use Array.map. I would not want you to incentivize that kind of interaction, because I don't want that valued community member to spend their time that way.

15

Can anyone who wants to be a reviewer opt-in during the “initial testing period”? Or is the “initial testing period” only open to those invited?

Is there anywhere for one to register their interest to participate in the initial testing?

5
  • 5
    As an active reviewer interested in this program, I like the idea of being able to opt-in. That said, doing so would potentially skew the experiment by introducing a selection bias in reviewers, who may not be representative of the average Stack Overflow reviewer population. Commented May 18, 2022 at 20:01
  • 2
    @SEStaff I very much would like to be part of the initial testing, FWIW.
    – user17242583
    Commented May 18, 2022 at 20:12
  • 5
    We are going to try to make this happen, but cant commit to it yet. The selection bias thing is real, and some one of the things that are very important for us to figure out is how much more organic exposure SG will get from inclusion in /questions listings, not to mention how many "active users with access to SG" we need in order to get folks looking at the incoming items. And allowing wide-scale opt-in at the very early stages might skew this too much.
    – Yaakov Ellis StaffMod
    Commented May 19, 2022 at 7:09
  • 5
    That said, the feedback of users who are enthusiastic about testing a section like this at its earliest stages and helping to give us their constructive thoughts on workflow and UX issues that could use some refinement is extremely valuable. So, again - we hope to be able to accommodate this request in some way, but dont know yet how/if it an happen (or if it wont happen at the very first testing round, but could happen for subsequent rounds). If we do open things up for opt-in, we will list this clearly in an MSO announcement related to the SG test release.
    – Yaakov Ellis StaffMod
    Commented May 19, 2022 at 7:14
  • 3
    @YaakovEllis It’s great to hear that you are considering letting users join in the early stages. I’m very much interested to be part of the initial testing! Looking forward to future announcements.
    – Panda
    Commented May 19, 2022 at 11:33
15

With the Staging Ground, we are asking more from Reviewers.

Fully agreed. The more intense guidance will surely mean more work. The question is how much more work? Twice, thrice, ... 10 times? The experiment will show that. And I suggest to really measure the time that people spend approximately doing the reviewing. For example, if reviewers can do 18-27 actions per month in the first questions queue, then maybe they can only do 5-10 actions per months in the staging ground (or more?, or less?). The experiments need to determine this. The idea would be that reviewers will likely spent an approximately equal amount of total time doing reviewing, so you might be able to estimate what to expect from the full release.

..we will need at least as many, if not more, Reviewers (...) to be active in the Staging Ground

This might not be possible. I would not count on that at the very least. Reviewers doing work for free are surely a very scarce resource. I guess your only chance is to motivate them more and what motivates reviewers is the feeling of contributing to something meaningful. So if the whole thing is designed so and works in a way that is perceived as particularly meaningful then maybe (or maybe not) more reviewers will turn out. Make the experience as pleasantly as possible for the reviewers. Maybe it's time that reviewers are put more into the center and receive more attention (as there seems to be less shortage of askers in comparison).

And wait. Instead of asking for more reviewers (which might be difficult or impossible to achieve), why not asking for fewer askers in the staging ground? Wouldn't that also be a possible solution to match reviewing supply to asking demand? One way could be to manage expectations and communicate clearly what is expected of a good question before it is asked. Another way is to present the reviewing on the staging ground not as a free service, but rather as a privilege. And one should pay back that honor of having their own questions improved before publication by investing time and doing lots of (re-)search and clear formulations. I feel this is the right way to look at the staging ground.

Badges ... Reputation rewards ...

I'm personally so past badges and reputation rewards, but in general both have worked for Stack Overflow in the past. One could expect that they would work here too. Gamification increases motivation at least in the short term and rewards guide behavior. So, make sure that you actually reward the actions that you would like to see.

I'm not per se against giving rep to actions that influence content not directly, but I feel the impact should be rather limited. So what about giving a comparable rep reward to suggesting edits (+2) for suggesting improvements on the staging ground (only once per question and only if the asker approves or reviewers approve or the question is successful later)?

I would not give 10-15 rep points, which might be too much. But then again, if you really want to attract lots of reviewers and fear you cannot get them otherwise, why not going in big. But beware, you might attract the wrong kind of people and going back from a big reward later would likely kill the feature. I vote for a modest reward size with the option to increase it later if needed. Make it at most a small extra source of rep, but nothing more.

We could potentially add a stat relating to the number of new users helped in a user’s profile.

Oh yes, that is a great idea. Please add stats for this. Basically display contributions as a vector of numbers: number of questions, answers, edits, comments, review actions, and staging ground suggestions at a prominent place on the profile, maybe somewhere in the summary.

... a one-to-one interaction is needed for every new user question ...

I'm very suspicious about 1:1 scalings and see them as rather unfavorable. And I don't see how you can fundamentally change this scaling behavior later on (you say you are already thinking about it) without for example let askers review other questions as part of the asking process. But maybe you'll have a good idea there. I'm very curious.

In the meantime I think it is paramount that the pre-factor of the scaling is kept as small as possible. Askers need to be aware that they get a very luxury treatment in the staging ground and need to likewise invest lots of time and effort of their own, not just asking because that was the cheapest option. It's not cheap. Reviewing is expensive. People must understand this. If reviewers are the only quality control, then the staging ground likely won't scale and won't work out. Sorry, to be rather pessimistic there. I'm only writing this in the hope that the right measures will be taken in time to make the staging ground a success.

... we will be monitoring, measuring, and iterating

Always interested to read results of the monitoring and measuring. It would be nice, if the results of that can be shared for discussion.

...qualification for Reviewers (rep-based or earned through actions), daily review limits (should we have them and how would they work?), audits (do we need to have them?), automatic suspensions based on suspicious behavior detection (how would these work?)

Earned through actions is better than rep, because rep mingles everything together. Review limits have proven their usefulness to avoid burn out, but I prefer weekly limits to daily ones. Audits: no, but review the work of reviewers somehow. Automatic suspensions: depends on what bad behaviors there might be and how easily they are detectable. What about manual suspensions and flagging for the moment? All actions on the staging ground are flagable for moderator attention, aren't they? If not, they should be.

Summary: This is a nice framework for high quality tutoring where sufficiently many reviewers are available. But even the best piece of software is useless if there are not enough users, in this case reviewers, available. The planned experiment with a limited number of askers can artificially increase the ratio of askers and reviewers and see if it works in principle. If it also scales and works in practice, only time will tell.

12

I already wrote a lengthy answer here, so I will not add anything to it, but rather concentrate here solely on the scaling aspect (even though you say that isn't the concern right now).

The knowledge library aspect of Stack Overflow scales favorably. One person asks one new question, another person answers it and all visitors can profit from the knowledge forever. Very favorable scaling.

However, not all people manage to ask answerable, new question. So there are lots of unanswerable or already answered questions created all the time. These require lots of reviewing effort, which is done by experts and they are very expensive. This is very unfavorable scaling requiring a number of experts on the order of non-experts. It should be avoided if possible.

Let's look at the most extreme case possible: If there was only one expert in the world and everyone else wanted to ask questions. What would we do?

  1. We could tell the expert to work overtime and check (and subsequently answer) all the questions there are. Problem: This will not work, our only expert will die from exhaustion.

  2. We tell this one expert to teach another person how to ask a good (answerable, new) question, then ask them both to teach other 2 persons how to search for existing question and also how to ask good questions in return, and so on. Finally the expert stops doing that and concentrates on answering these questions, while all others teach themselves to search better and to ask better. That might again result in a more favorable scaling (especially if people don't see it only as a free service to throw all their problems at and see what comes out.). So that should be the way to go.

My conclusions are:

  • The staging ground must teach people how to ask good questions. After a few trials they must have learned it.
  • Many of those that have learned it, must return and teach others how to search for answers and ask good questions. (Maybe that can be measured. Maybe an incentive could be better placement of their own questions instead of just rep, which isn't much more than a number displayed somewhere.)
  • Pupils learning how to ask good questions must invest much effort of their own.

Then asking on Stack Overflow might work, but:

  • The more knowledge there is contained the higher the chance for duplicates. So apart from asking a clear, focused (answerable) question, the search for possible duplicates is getting more and more important over time. Any advance in search technology and automatized duplicate detection would be highly welcome there.
5
  • 3
    "The staging ground must teach people how to ask good questions. After a few trials they must have learned it." - this is one of the main things that we are after.
    – Yaakov Ellis StaffMod
    Commented May 11, 2022 at 10:52
  • 3
    "Pupils learning how to ask good questions must invest much effort of their own." - we are also incorporating this. The SG is not aiming to have the experienced users rewrite the on-topic-but-poorly-written questions. We want to give guidance, but ultimately it will be up to the new authors to improve the quality to the level where it is ready for the main site.
    – Yaakov Ellis StaffMod
    Commented May 11, 2022 at 10:53
  • 5
    "Any advance in [duplicate] search technology and automatized duplicate detection would be highly welcome there." - we all agree with you on that one, and that is on the roadmap as well, though it will be a separate initiative with its own timeline (and own staffing requirements).
    – Yaakov Ellis StaffMod
    Commented May 11, 2022 at 10:55
  • 3
    "Many of those that have learned it, must return and teach others how to ask good questions." - while we enforce this as an imperative, we are also looking to find ways to keep new users engaged here, help them to level-up, and to get to the place where they can also help others (through RQ, SG, editing, comments, etc).
    – Yaakov Ellis StaffMod
    Commented May 11, 2022 at 10:56
  • 5
    @YaakovEllis I really want the staging ground to succeed, because I think it's much better than anything existing, but also I'm sure I will only occasionally contribute, because there is so much else to do. Maybe there will be enough volunteers but probably not. I'm quite sure that in the long run people will have to teach themselves to ask good questions somehow. Maybe it's helpful to write another meta post, where far away from implementation details the problem and the solution is discussed. Like: what is really the problem here, what is the solution and why will it work. Might be helpful. Commented May 11, 2022 at 16:09
11

Since this question is linked to from the MSE question (Announcement): "How might the Staging Ground & the new Ask Wizard work on the Stack Exchange network?" I'll provide an answer here on MSO.

I'll only make two points, and leave any others to someone else.

1.

"A drop in a single month of even a few hundred reviewers has spelled the difference between the review being at manageable levels of ...":

Yes, and on some other sites the moderators and a couple of users do all the reviews; as you are aware it can be a challenge for a moderator to do some reviews (in queues with a quorum), because rather than having a consensus they become the sole reviewer.

It is also unfair to the moderators, for them to do the user's work.

Even sites that are 9 years old have only 3 users who have earned the silver Reviewer badge in a single queue - some places have absolutely minimal review participation.

  • Maybe we need a right column (above HNQ) to remind people that they have the necessary reputation on a site where they are a member, and that there are reviews that have languished for hours, even days; sometimes blocking further edits or aging away.

2.

"An idea that we have been playing around with is to offer a reputation incentive for Reviewers. This could be offered specifically on questions where the following sequence occurs:

  1. A Reviewer interaction on a question in the Staging Ground of marking it as requiring Major changes or putting it in one of the Closed conditions and leaving a comment…
  2. leads to edits by the question Author…
  3. which then leads to approval of the question in the Staging Ground and subsequent publishing of the question publicly…
  4. and finally to the question meeting ...":
  • Steps 1 and 2 should not be a requirement for reviewer rewards. If a reviewer believes that a question is great (needs no improvement) and so do the other reviewers, and the voters, the reviewer has done their job correctly; they should still be rewarded their virtual Internet award.

    Why not recognize correctly offered praise instead of focusing on obtaining criticism, possibly nitpicking, when everything has gone smoothly.

Let's not create the worry:

"Adding in a rep incentive could also draw in more users who will Review in the Staging Ground in bad faith in search of rep. These users could intentionally reject good posts from new users and leave comments with the goal of subsequently approving the post (which as a good post would have a good chance of meeting the success criteria).

  • Show people that there are sites where they have the reputation, and that there's reviews that need doing.

  • If you already have enough reputation to review you'll only gain a small additional percentage by helping out.

  • Don't force the nitpicking, or clarification requests, if the OP writes an adequately good question (capable of receiving an even better answer); if it's great let us recognize and say just that.

11
  • 8
    on sites where there's always something to review, a notification that there's something to review that never goes away just becomes an annoyance that you can only deal with via adblock. An introduction to users who haven't used it yet or in a long time would be more useful
    – Kevin B
    Commented May 9, 2022 at 15:05
  • 3
    @KevinB, that's the difference between being a volunteer and helping improve a site that offers you free services, versus being a taker who expects others to do their share as well as help them on demand. --- The right column reminder is one way of implementing a notification to people who want to help, how do you propose an "introduction to users" would be implemented (write an answer, instead of a comment) if people won't do their share, leave it to others, and actively block pleas to help. --- Some people already don't read the FAQ, tour, and review guidance; will they read your intro?
    – Rob
    Commented May 9, 2022 at 15:41
  • 2
    Someone would be far more likely to read a single notification that goes away, than a red dot that sits there, menacingly reminding you about this portion of the site that you decided not to interact with.
    – Kevin B
    Commented May 9, 2022 at 15:45
  • 3
    When I looked at the stats for each queue, Freelancers.SE has had less than 2500 total review tasks. The reason so few badges have been awarded there doesn't appear to be lack of participation in the review process by active users; it's that there's nothing going into the reviews queue themselves. That in turn appears to be due to the very low question volume on the site - less than 2000 non-deleted questions at present. Commented May 9, 2022 at 18:53
  • @Dan, it's all aspects of participation on that site; that was one previously discussed example. Look at law.stackexchange.com/review there are queues where people have left lots to do, I'm not a member there; I don't know the details.--- The point is that sometimes the few do the most, and some of the others do the least; not really worth spending efforts to make shirking easier (as a different comment suggested). --- We should consider all sites when designing the way forward, rather than implementing for one sites and patching for the others. Thanks for your observation.
    – Rob
    Commented May 9, 2022 at 19:10
  • 1
    The problem with the last bullet-point is that 1 single char out of place can make a question not MRE. So on a programming site you have to nitpick a question into shape for it to be usable.
    – bad_coder
    Commented May 9, 2022 at 21:16
  • 1
    @bad_coder, was the "if the OP writes an adequately good question (capable of receiving an even better answer); if it's great ..." portion unclear? Sometimes it's the one character out of place that creates the bug that the OP is asking about, then some helpful person offers a suggested edit correcting it; instead of an answer pointing out the problem, and additional shortcomings to the approach. --- Yaakov will have to read all the feedback and discuss how best to implement what is needed, if your answer doesn't offer such a suggestion.
    – Rob
    Commented May 10, 2022 at 7:30
  • 1
    "that's the difference between being a volunteer and helping improve a site that offers you free services, versus being a taker who expects others to do their share as well as help them on demand." I just recently found out a colleague of mine is active on this site. He's got over a 100 high quality answers, a half-dozen high quality questions, and is over 5k rep. I took a look at his profile and he doesn't have a single review badge. Nor does he obviously have any interest in reviewing. Not everybody who contributes positively to this site contributes the way you want them to.
    – ouflak
    Commented May 10, 2022 at 8:36
  • @ouflak, read the question (Announcement) title: "Staging Ground: Reviewer Motivation, Scaling, and Open Questions", this isn't about people who don't, it's about people who do. It's not my question, not what I asked for; we have been asked for Feedback on what has been proposed. --- If you want your opinion to count don't hide it in a comment, post your own answer and share your thoughts with everyone.
    – Rob
    Commented May 10, 2022 at 9:39
  • 2
    @Rob Your long argument is out-of-touch with the reality of reviewing on SO (you're right, you don't have a single steward badge on this site). It's been proven in this post 66% of new posts won't get into shape (the percentage is actually higher, but curators only catch most of the remainder in the long run). The easiest kind of review is "looks ok" compared to duplicate etc which involve work (and especially when you're interacting with the OP towards corrections) so you're wrong in your assessment.
    – bad_coder
    Commented May 10, 2022 at 16:43
  • 2
    @bad, that's explained in the first sentence and expanded on in the fourth paragraph - this feature isn't t only apply going to a single site.
    – Rob
    Commented May 10, 2022 at 19:08
10

It's entirely possible this staging ground will get plenty of engagement without bribing users with reputation. We should proceed without any reputation incentives until we've identified a problem (lack of engagement) exists, and maybe not even then. But to provide reputation at the outset seems to be looking for a solution without proving there's a problem.

10
  • 3
    Agreed. Even if they would ever happen, we are definitely not planning on any reputation incentives during the initial test period. As I tried to make clear above, this whole post is forward-looking, and trying to anticipate what might end up being problematic, and then exploring ways that we might want to address these issues.
    – Yaakov Ellis StaffMod
    Commented May 10, 2022 at 19:54
  • 2
    I admit my attention was focused on just the reputation section and missed that detail. Thanks for clarifying!
    – Kirk Woll
    Commented May 10, 2022 at 19:56
  • 3
    Can we change the frame of reference a bit here, @YaakovEllis? Unless there is significant support for reputation incentive (which I don't see so far), can we be assured you are not going to proceed with it rather than us begging not to do that and then you might address it?
    – 0Valt
    Commented May 10, 2022 at 20:06
  • 2
    @OlegValteriswithUkraine I'd make the counter-argument -as Yaakov did- rep has so many avenues for abuse that we don't need more Nth dups and off-topic questions being answered; what we need is users willing to clean those messes up! I'm also convinced hardly anyone is willing to speak their mind in favor of rep rewards (out of fear of downvotes), which is to everyone's loss because Yaakov called for ideas and we aren't going to see any creative posts exploring those possibilities.
    – bad_coder
    Commented May 11, 2022 at 15:09
  • 1
    Yea, sure, maybe the "unpopular opinion" has majority support from users who are just afraid to come forward, that's gotta be it
    – Kevin B
    Commented May 11, 2022 at 15:15
  • @bad_coder if users do not speak up in the fear of downvotes (which is, IMO, a flawed argument since there is no way to verify the claim), then they are doing it wrong since there is no penalty on MSO for being downvoted. It (the claim if it is true) also indirectly implies that there is an understanding that the idea is quite unpopular, which is exactly my point. Anyways, if SG requires rep to keep users participating, then it is a failure from the get-go and deserves to scrapped. There is never a need for extrinsic motivation if the cause is worth it: mods don't get anything, yet [1/2]
    – 0Valt
    Commented May 11, 2022 at 15:16
  • [2/2] people apply. Close voters (including me and you) don't get anything, yet they vote. Flaggers don't too, yet they flag. Editors 2K+ don't either, yet they edit. Reviewers don't as well, yet they review. What is the common thing about all of these users? They think that the quality of the site deserves the effort. Why are there only a handful? Not because of lack of extrinsic motivation but due to processes being underthought, tedious, lack basic usability, and, also, ironically, rep-gated. What I tried to convey to Yaakov - either make it worthwhile without reward or don't do it at all.
    – 0Valt
    Commented May 11, 2022 at 15:19
  • 3
    @YaakovEllis I don't know if the MSO crowd will contribute any creative discussion towards the possibility of rep rewards - although that is: what you called for. Not that the possibility isn't interesting and very worth of being discussed in its pros&cons but the simplistic Nay answers that don't address, solve or add anything are much easier. I'd be willing to write (as are likely many) for the interesting speculative exercise of exploring the upsides - but dissuasion and status-quo will get the better of this discussion, perhaps by high reps who see no gains and denegate any problems.
    – bad_coder
    Commented May 11, 2022 at 15:24
  • 4
    @bad_coder "I'd be willing to write (as are likely many) for the interesting speculative exercise of exploring the upsides" - I'd be eager to hear your thoughts and respond to them.
    – Yaakov Ellis StaffMod
    Commented May 11, 2022 at 15:28
  • 3
    I will give this an upvote, but while one buttock is still on the fence (and boy does that sting). I really like the idea that reviewing well gives you reputation. Reputation should reflect your "site experience" and it is kind of lopsided that this now comes from posting questions and providing answers only, reviewing well is a sign of Stack Overflow maturity which should reflect on you in some way. So should editing, for that matter...
    – Gimby
    Commented May 16, 2022 at 11:25
7

On the question of awarding reputation, I very much share the fears that other answerers and commentators have expressed about the huge vector for voting fraud this creates; but I am perhaps not as vehemently opposed to it as Zoe and Oleg seem to be – I would align more with the lower/limited reputation gains suggested by Trilarion. I also share the concerns that this will almost certainly lead to more (deliberately) bad reviews.

I would also add a note of 'sympathy' with the moderators who (rightly) hand out review suspensions – the pressure on them and unpleasant backlash would both surely increase, if that suspension removed a reliable source of Precious Unicorn Points.

However, you may not need this incentive – and it is clear, from your Question and in comments, that you would not introduce it unless or until such a need was indicated.

I think there may be significantly more users interested in engaging in the Staging Ground than some may predict. On first questions I see posted in the tags I follow (mainly C and C++), there are often comments added by tag regulars/experts, asking for particular clarification or suggesting other ways to improve the post. Sometimes, those suggested changes are made, and sometimes close votes are retracted or reversed.

Presented properly to such users (many/most of whom are not especially active reviewers in the existing queues), this could be seen as a 'more formal implementation' of such 1:1 interaction. The 'motivation' to so engage would be the chance to get "First Shot" in posting an answer (C and C++ get a lot of FGITW activity).

Could the interspersing of S/G questions with the normal list(s) be linked somehow to Gold Tag-Badge users? Or maybe even show such users all suitably-tagged S/G questions, by default? (Could we even make a gold tag-badge a requirement for S/G Reviewers?1)


1 This is clearly not feasible, as it would be far too restrictive on the number of potential reviewers.

11
  • 7
    I'm not too concerned about backlash. Review suspensions are anonymous (though often obvious who did it because we tend to reverse whatever incorrect reviews we can) and we haven't gotten much actual backlash about them. I'm more concerned about the potential for low-quality reviews slipping by and granting undeserved reputation. There's not a lot of supervision over the review queues beyond automated audits, it's mostly a couple moderators working on it as a side project with help from a few users over in chat.
    – Ryan M Mod
    Commented May 10, 2022 at 22:21
  • 4
    Thanks for your feedback. And for your close reading of what I wrote above - that this is not something that we would introduce unless/until such a need was indicated. And as I tried to lay out above - I have lots of reservations about rep as well and the negative aspects that it could introduce.
    – Yaakov Ellis StaffMod
    Commented May 11, 2022 at 7:14
  • 1
    "Could the interspersing of S/G questions with the normal list(s) be linked somehow to Gold Tag-Badge users?" - not sure what you mean here. Linked in which way?
    – Yaakov Ellis StaffMod
    Commented May 11, 2022 at 7:15
  • @YaakovEllis For example, those gold tag-badge holders with the appropriate review credentials would see all S/G questions in their tags. Commented May 11, 2022 at 7:39
  • 3
    @AdrianMole that is something that we can consider for later. For the initial test we'll probably stick with a low and consistent level of the number of S/G posts mixed in. Want to be cautious about not going too far in showing this content to users in way that they would be resent. Would be nice if we could later on give users (or just silver/gold tag users) the control to be able to increase the number of posts they see from S/G mixed in /questions listings.
    – Yaakov Ellis StaffMod
    Commented May 11, 2022 at 8:02
  • 1
    Engagement of tag regulars might be driven exactly by the desire to prevent FGITW and close duplicates more effectively.
    – blackgreen Mod
    Commented May 11, 2022 at 9:35
  • 1
    @blackgreen Yes, indeed. But that engagement could also be used by those (minority?) tag regulars/experts that persist in answering dupes, etc. However, should they, at some point, have their reviewing rights suspended, then they would no longer get the "sneak previews", so it might work well. Commented May 11, 2022 at 10:04
  • 1
    ... such a suspension could come about after having "reviewed as OK" a number of posts that were subsequently closed as duplicates (or for other reasons). Commented May 11, 2022 at 10:07
  • 6
    @AdrianMole I didnt get into it in this post (more for the future), but one thing that I have been playing around with is the idea of automated suspensions. These could be triggered for a user on the SG when a certain percentage of their recent posts that they approved in the SG subsequently were closed/deleted/neg-score after being published. And if we see that there are a significant number of reviewers who approve posts in order to answer them right away, definitely could consider a short timeout on answering for the approving user.
    – Yaakov Ellis StaffMod
    Commented May 11, 2022 at 11:07
  • 4
    Note: Earlier this evening, myself and another C++ gold-badger, engaged with a first questioner in a manner that would be very similar (I think) to the S/G review process. It took two of us (plus the OP) over 40 minutes to resolve. After clarification, some editing and other discussions about how to answer (it was a potentially nice question), I went looking for - and found - a duplicate. That's quite a workload if we're looking to handle thousands of Qs per day. Commented May 13, 2022 at 16:52
  • 3
    You tongue in cheek call it unicorn points. Rightfully so. I think that is a little bit of a wrench in the machine. We can't take reputation points too seriously on the site currently, they don't represent actual site experience. They only imply you're good at collecting reputation somehow, that somehow not necessarily being a lofty accolade. It'd be so nice if we could just start giving reputation points for doing good things and expect this to go to good people. But the entire system is simply not mature enough for it, and maybe never will be.
    – Gimby
    Commented May 18, 2022 at 13:15
6

Incentives aren't a bad idea, but the unlimited, after-daily-rep-cap nature defined perhaps is.

Edits, by relation, are capped. You even used them in your explanation here.

Just do the same with this. Cap the rep gained from the staging ground to only being awarded prior to the bronze badge. Once the bronze badge is awarded from the defined sequence, rep is no longer awarded. Or, use some other similarly available metric. Perhaps somewhere in the +500 rep range.

6
  • 4
    I really like this post, it adds a tangible and interesting scenario that expands on the question. I think this thread needs more posts like this!
    – bad_coder
    Commented May 12, 2022 at 3:55
  • 1
    I personally think it should be 5k-7.5k (10-15 rep per review) since you get a rep for the first 500 edits. So you should get rep for that many reviews instead of 33-50 reviews which could probably be done in 2 days.
    – Ethan
    Commented May 14, 2022 at 19:08
  • 1
    Let me get this straight. 1/2 or 3/4 of curation privileges earned by simply participating in a single queue? No, thanks, please. If we are to have rep incentives for this, I'd take Travis's proposal any time over this.
    – 0Valt
    Commented May 14, 2022 at 19:30
  • 2
    @OlegValteriswithUkraine I don't see why it's necessarily a bad thing to gain curation privileges with this idea... when this process (at least in theory) awards people for helping curate the site.
    – Isaiah
    Commented May 15, 2022 at 22:02
  • 3
    It's not a bad idea to get privileges through this, @Isaiah, and, actually, that's what I've been campaigning for lately - unlocking privileges by taking meaningful actions and issuing progression-based rewards in general as opposed to passive accumulation of points. Using rep to measure experience also seemed like a good idea in theory, but in practice it is a total disaster. I am only against digging the grave further by tying in yet another reputation source to the system.
    – 0Valt
    Commented May 15, 2022 at 22:25
  • 3
    Thanks for this feedback. I like the idea, and agree that unlimited after-cap rep is probably too much. The bronze badge is awarded at the first review on review queues, and we might do something similar here, so I dont know if it is feasible to limit it to folks who haven't yet earned that badge. But I do like the idea of (if we were to have any rep incentives) to have some combination of: a lifetime and daily awarded rep cap, to only award it when the user is below a certain rep, doesn't have a silver badge yet, and maybe significantly less rep per award (SE does 2 rep points).
    – Yaakov Ellis StaffMod
    Commented May 19, 2022 at 7:18
6

Reputation rewards should be based on votes by humans - not on heuristics

Reputation rewards for review are not inherently evil, but the problem with your proposal is that you're proposing handing them out without a single human being ever directly judging that the review was useful. Instead, you're proposing letting the system make that judgement based upon a heuristic - namely that if a reviewer takes an action that expresses that a question is very bad, and then it gets edited a bit, and then it turns out that the community thinks that the question is very good, then it probably means the reviewer did a great job at getting a terrible question improved.

The trouble is that this heuristic is not at all guaranteed to be accurate. It could equally be that the question was great to begin with, the reviewer was harshly critical of it for no good reason, and then they end up getting a share of the credit for the original author's good work simply because they chose to unreasonably criticise it.

As you note, this kind of imperfect heuristic is vulnerable to exploitation by bad-faith users, but even in the absence of exploitation, it hands out rewards kind of arbitrarily, to users who may have provided no value or done harm, without their peers ever having passed positive judgement on their work. This devalues the reward, in much the same way as it would devalue question and answer votes if there were a bot that voted on posts based on some machine learning algorithm. Part of what makes rep a valuable thing that we feel warm and fuzzy about getting is that it reflects the positive judgement of our work by our peers - but your proposal loses that.

If you really want to have rep rewards for staging ground reviews, put a human in the loop. Prompt the question asker to vote on whether the review was helpful, a few days after they post their question, sort of like how companies ask you to rate their customer support staff. Then the rep rewards are being given out by humans - just like with voting - rather than machines.

1
  • 4
    I agree with your misgivings and for the inaccuracies introduced by heuristics like this. I do like the idea of querying the author to see if the help in the SG was good. Thanks for the feedback.
    – Yaakov Ellis StaffMod
    Commented May 25, 2022 at 12:42
4

We will be ensuring much increased visibility for new Staging Ground questions by interspersing them in some of the questions listings on the site

Will the intermingled SG questions have a different background color or visual indication?

2
  • 7
    They will have a visual indication. Go to the previous post and search for "Inclusion in /questions listings" to find more discussion and a screenshot.
    – Yaakov Ellis StaffMod
    Commented May 10, 2022 at 6:57
  • 1
    @YaakovEllis I had read through the series of posts and somehow didn't process that one section...
    – bad_coder
    Commented May 10, 2022 at 12:16
0

I always thought that having reputation both from answers and questions together was not a good indicator of how a user has contributed in SO.

Now with this change that you plan, I think it is the perfect moment for splitting the total reputation into:

  • Answerer Reputation
  • Questioner Reputation
  • No content Reputation

Please do not keep going down the spiral which you have started by sunsetting the Developer Story and Jobs. Many people actively contributing in the Site complain that somehow changes are done by seeing the tree and not the forest. You may earn what you want today, but you may as well have burned the future.

3
  • 2
    What is "no content reputation"? Why should users earn reputation for doing/contributing nothing? And, more broadly, what is the advantage of splitting reputation out into different categories when all reputation really measures is the extent of one's engagement with the site? Why is it materially different that that engagement has been in the form of contributing questions versus contributing answers?
    – Cody Gray Mod
    Commented May 25, 2022 at 8:24
  • 1
    @Cody Gray It could be called Site Support Reputation or Reviewer Reputation. I am sure it can be found an appropriate name which will be good for everyone. If it is fair I think that most people would agree. However giving points of reputation for no content contribution could be seen as an insult to existing users who have contributed in the past. Commented May 25, 2022 at 8:28
  • 4
    The most interesting proposal in this thread yet regarding possible changes to the reputation system.
    – bad_coder
    Commented May 26, 2022 at 6:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .