Moderator note: - This experiment has been postponed. Please see the duplicate post above for the explanation and discussion.
It's been a while since we last talked about the one-reputation voting change. We are still pursuing this because stagnating participation on the network is a concern for all of us, and we want to think about ways to grow the active community on the network. We have, by design, utilized rep as a threshold to award privileges and prevent bad behavior. While this has been effective in creating the current status quo, it has made participating on the network in some of the most basic ways difficult.
By pursuing this experiment, we can better understand what happens when we remove those barriers and what, if anything, we need to address to ensure that doing so doesn't cause harm. We want to find a healthier middle ground where we can find ways to open up the gates to participation a bit while also addressing issues like voter fraud more efficiently.
Last we communicated, we asked for volunteer sites and proposed how we wanted to run the experiment. We have had some time to reflect on the feedback we received. We returned to the drawing board to see what changes we could make to make this a smoother experience for everyone involved. We are now coming to you with additional changes and a new direction for the experiment. This plan was presented to the moderator team to gather their insights and identify additional risks and necessary steps to improve the experiment. It's important to note that while we received valuable feedback, there were varying perspectives, which included dissenting views. We took these into account to further refine the experiment.
Many of these changes are a result of your feedback. We intend to roll this experiment out to get the data we need to determine if reducing the barrier to voting would create a positive or negative impact, and what resources we would need to invest if we identify positive signals to pursue.
Defined Terms
- Established Users: These are users that have already earned their upvote (15 rep) and/or downvote (125 rep) privileges.
- Test Users: These users are given voting privileges as a part of the test they otherwise would not have.
Major Changes
One Testing Site
Previously, we asked for volunteer sites; and while that route may have worked, we really would prefer not to take it and then find that we didn't collect enough data to learn anything, and then have to go through another period of testing. To that end, we will be testing this exclusively on Stack Overflow. We plan to run this experiment for around four to six weeks.
Shared Data Analysis
For this particular test, we understand that some may have concerns about how voters will engage and the level of fraud that could occur. We will keep this in mind as we analyze the data, and we will share the data analysis we utilize to make decisions with the community. We will also share anonymized data with the moderators to review if they feel inclined. Please note this is not something we're able to do for every test that we run, but we feel that it is necessary for this experiment to have this level of transparency.
Two-phase testing approach
We had several internal conversations about concerns raised from moderators. Stack Exchange agreed that we needed to have a better handle on how many users were gaining the privilege so that we could better keep tabs on their behavior. To meet that requirement, we made the following changes by making the voter privilege test into a two-phase test.
Phase 1: Logged-in users can vote
Hypothesis: Validate that allowing more users to vote will not increase fraud disproportionately.
This is styled as a 'do no harm' test rather than an A/B test. We will start with a small subset of users, thirty thousand, and scale up from there. To start, we won't be letting existing users know that they can vote, but if we do not see enough activity to get results one way or another, we will turn a prompt on to make that clear. We could be considering this a phase 1.5 if voting activity doesn't change much at all. As we keep tabs on the activity, we will scale up the proportion of logged-in users who get the voting requirement privilege turned off. The first couple of days we will add more users to the test group to make sure it's at a good level and that nothing is going terribly wrong. We won't be doing this on Thursday or Friday, so we have better insight and that spikes in activity will be less likely to happen over the weekend. After the first week, we will only make changes to the amount of users who have access on a weekly basis.
After we have concluded phase 1, we will pause to do the analysis on vote fraud, engagement, and what kind of sock-puppet voting we are seeing and share data with the moderators. We will then share the outcomes a little bit later with everyone on Meta Stack Overflow. We are estimating that phase 1 will take somewhere between four to six weeks. Should things start to look poorly before that, we will shut the experiment down. Should we successfully run the experiment and decide we need to make changes to phase 2, that also will be communicated.
Phase 2: A/B test the upvote prompt to anonymous users
Hypothesis: Telling users they can vote will encourage them to sign up without disproportionately increasing fraud.
This will be an A/B test where we prompt users to sign-up to vote and measure if any voting activity happens. New users will be shown a voting prompt that you can see below in the mockups section. Assuming phase 1 is complete, this experiment becomes about whether prompting users to vote will encourage them to sign up, assuming that logged-in users have the ability to vote. Similar to the above, we will increase the frequency of the voting prompt to anonymous users to let them know the option exists. We will monitor this group separately from the others to see the behavior differences.
We anticipate that there will be some form of voting fraud here. It's very important we see that to understand how this change impacts things. That being said, if things go off the rails, we will shut down the experiment. We don't need to run this phase nearly as long and could get the data we need as quickly as a week, but for clarity, we think it could take anywhere from one to four weeks.
These changes will allow us to get the signup and engagement data while reducing the risk of individuals attempting to take advantage of it as bad actors during the experiment. Because we are approaching this more conservatively, we may not see as much voter fraud or bad actor attempts otherwise. This requires us to operate under the assumption that it will be there to some extent. Should we get to a point where we believe this should be a permanent change, we will work with moderators to determine what tools are needed to prevent vote fraud.
Association Bonus
For the duration of the experiment, the association bonus will not be given out to any users who would've earned it from actions taken on Stack Overflow. We will backfill them where appropriate after the experiment has concluded. Whether Test Users earn them by receiving reputation from Established Users or not, we are just pausing this bonus while the experiment is ongoing. After the experiment's conclusion, Test Users who earned them by receiving a reputation from Established Users will have this awarded to them.
We wanted to turn this off because of the possibility of abusing this test to gain privileges on other sites on the network. We are interested in reviewing the association bonus as a whole but won't be revisiting it as a part of this test.
Votes cast during the experiment
We want our test plan to include the option to reverse the voting behavior of the users in the test group and the possibility to remove their privileges altogether. Ultimately, we can't really know what will happen and how user behavior might change due to this test. We need the data to understand its pros and cons. We don't want to assume that there will be more bad votes than good, but we also don't want to assume there won't be any bad votes at all. To that end, privileges will be reverted back to normal at its conclusion. Additionally, we have some scenarios in mind to reverse some or all of the voting behavior by the test group.
Downvoting Friction & Voting Prompts
Any experiment user who downvotes will get this prompt for the duration of the experiment starting on day one of phase 1. The other option will provide a text box to share their reasoning. This gives us the opportunity to collect some data from users on why they voted the way they did. The outcomes are not shared with the author of the question or answer.
For questions, there will just be an open text field that looks like this:
Users who are not signed in will also be shown this prompt in some cases to encourage them to sign up and to upvote content they find helpful.
Possible Scenarios
We've mapped out a few scenarios for how this experiment may go when launched. While there are certainly many different ways for how this all plays out (and perhaps a combination of a few listed below), we’ve simplified this into four different scenarios:
Scenario A: Everything is Great
We ran the experiment and found that fraud is minimal to non-existent, new users usually vote, and no issues are raised. We leave things as is but keep an eye out for any concerns that might later arise. Rollback to previous privilege thresholds, and we will prepare to make this a permanent privilege change on Stack Overflow.
Scenario B: Good things, but a few undesirable things too
We ran the experiment, and a few issues arose, but minimal fraud existed. We leave things as is but keep an eye out for any new or greater concerns that might arise. We will prioritize some work around additional moderator tooling and consider additional product changes to prevent issues. Rollback to previous privilege thresholds, and we will prepare to make this a permanent privilege change on Stack Overflow.
Scenario C: Some good signals, but just as many undesirable signals.
Lots of known issues and concerns. A disproportionate increase in voter fraud. Rollback to previous privilege thresholds. We ran the experiment, and many known issues and concerns were raised. We see a substantial increase in voter fraud. We will roll back to the previous privilege thresholds and reverse problematic votes. We will then prioritize work to address the concerns and communicate next steps.
Scenario D: The house is on fire. This is fine
Completely unforeseen issues arise, and significant concerns need to be addressed. Rollback to previous privilege thresholds. We will revert some, if not all, votes in the test user group and return to the drawing board to see if different avenues incur fewer issues. If things go genuinely off the rails, we can stop the experiment early.
Note: Users in the Test User group would keep their voting privileges if they earned them through legitimate upvotes from Established Users. Any users who lose the privilege after the conclusion of the test will have a specific communication explaining what has happened. Both of these are meant to preserve the user experience as best as possible if we have to roll back the privilege due to issues.
Rationale
Ultimately, we don't know what might happen: we can guess, but these will be just hunches at the end of the day. We need to run a test to determine what is true and what isn't to see if this is a worthwhile change or a terrible one. That being said, I did want to draw attention to the concerns around inadequate tooling to handle the potential increase in voter fraud. We are launching this experiment with some new fraud monitoring and alerting.
These tools, as well as others that we are not sharing at this junction, will better inform us of any malicious activity of this test as it occurs.
This test could result in potentially unforeseen positives. Voter privilege expansion could identify threads that are worth pursuing. I won't rehash the contents of past posts on new user experiences, but for example, what if this experiment shows that voting helps user retention? Suppose users who can vote have a faster time asking/answering questions or just generally hang around more. In that case, we know we have a thread to pursue to help user activation, which could lead to initiatives around improving those experiences, and how to take more actions on the network, such as onboarding.
We don't want anyone to come away from reading this feeling their specific concerns are being ignored. Truthfully, many, if not all, of the concerns raised through various communications of this test are possibilities that could occur. We have, however, found that the easiest and quickest way to figure out what should or shouldn't be done to help address the outcomes that may arise is for us to collect some data to work with — instead of going on pontificating about which ones are more likely. So, while we understand that the changes we have presented here may not adequately address your concerns, we would ask for the opportunity to allow us to find out what exactly happens. Once armed with the necessary data, we can then determine if this initiative needs to be abandoned or if it's just a matter of implementing appropriate tools or safeguards to prevent undesirable behaviors due to this privilege change.
Next Steps
We are intentionally not announcing a start date for this experiment. The reason behind this is that we don't want to signal to test users that they are part of an experiment. To that end, this post will be unfeatured a week after today, April 10th, 2024. The intent of this is not to hide the experiment from the community but instead to preserve the quality of it as best we can so as not to signal to test users that they are part of an experiment.
The first phase of the experiment will run for four to six weeks. Phase 2 will run between one and four weeks. If necessary, we are prepared to end it prematurely. At the conclusion of phase 1, we will share the dataset with moderators and then allow time for discussion on the outcomes we have reached and what we think the data indicates. We will then share a general summary of where things are at before starting phase 2 and repeating the process. After that, we will share a report with the wider community to share what we have learned and what direction we plan to take from there.