59

Judging from my overall experience on Stack Overflow and what I've seen happening with others, it might be that users' votes are confounded based on the reputation of the post author, or based on whether or not they are a new user.

However, this is unproven. To help (dis)prove this, I have a proposal for an experiment: show half of users who can cast votes what is currently shown, and hide the reputation and badges on posts to the other half (for all posts). In my mind, this would look something like this:

No rep or badges visible

While users in the control group would see this:

Normal user card visible

Of course, all users should be able to see the full characteristics by clicking on the username.

Let's run this for a while, and see if users in the control group are more likely to downvote or not upvote posts because they were posted by a new user.

What do you think?

20
  • 24
    Without a placebo rep, the users would all complain to meta that the rep display was borked and 'why am I being targeted? That's very unwelome'. Commented May 11, 2018 at 6:59
  • 3
    I like the concept of this test, but if half users are shown hidden rep while the rest are shown all, how do we conclude the test since voting is still anonymous? Instead, shouldn't it be that half of new posts are always hidden?
    – Andrew T.
    Commented May 11, 2018 at 7:32
  • 4
    @AndrewT. Staff still have access to personal voting data. They can calculate the statistics. Also, I don't want to hide it only on new user posts; others will know that it's by a new user if it's not visible.
    – gparyani
    Commented May 11, 2018 at 7:33
  • 2
    I seem to recall discussions on the uber meta regarding the "halo effect"... Commented May 11, 2018 at 7:36
  • I guess you could ask for a volunteer panel, and those volunteers could be split into two groups, one that got unmodified data, as usual, and the other group got the correct user name, but rep chosen at random from the set of rep available in the panel. Commented May 11, 2018 at 8:05
  • 25
    @MartinJames No, we don't want volunteers. It would introduce bias, and the result wouldn't be useful.
    – gparyani
    Commented May 11, 2018 at 8:06
  • 4
    The assumption is that new users are unfairly downvoted. There is evidence of the opposite effect: if you post a good post (which high rep poster's are likely to do more often) that gains upvotes, after a while the probability of a reader of the post upvoting it drops. Visitors seem to tend to believe that once a post gains many upvotes it does not "deserve" any more.
    – Raedwald
    Commented May 11, 2018 at 8:21
  • 4
    @Raedwald In that case, the data from the experiment will show that
    – gparyani
    Commented May 11, 2018 at 8:22
  • I find the lack of experimentation in SO very surprising to be honest, most data I've seen is observational. They have very good data scientists but I guess they focus more on the SO Jobs part.
    – user2285236
    Commented May 11, 2018 at 8:57
  • 3
    @user2285236 here is an inherent problem with experimenting on people without their permission. Commented May 11, 2018 at 9:35
  • 3
    @MartinJames is something like A/B testing considered as experimenting on people? Commented May 11, 2018 at 9:36
  • 2
    @IncreasinglyIdiotic in general, it can be acceptable, especially where the set polled already have an implied or explicit business relationship with a vendor. In the case of SO volunteer contributors, however, I have a feeling that such trials would not go down well:( Commented May 11, 2018 at 9:45
  • @MartinJames I don't see a problem with asking permissions. You don't have to reveal the specific manipulation that you are going to do as long as participants are aware they might be manipulated (This is of course a little different then an A/B testing for a basic UI element but not as serious as the evil Facebook experiment.)
    – user2285236
    Commented May 11, 2018 at 10:42
  • 4
    SO has always been about quality content. Why do we need to show who asked the question, at all? At least so prominently? Why not get rid of that thing entirely and only add it to questions/answers once they are say, 3-5 days old?
    – Polygnome
    Commented May 12, 2018 at 10:05
  • 1
    Userscript+stylesheet .user-info {display:none}. Been there. Tried that. In my experience it does not make a quantifiable difference in how you vote. Albeit it might depend on the tags you frequent.
    – mario
    Commented May 12, 2018 at 23:51

2 Answers 2

12

I like the idea of doing some sort of experiment like this; I actually suggested it right before the start of the welcoming civil war.

A few extra thoughts, none of which should be taken to mean that it'd be a bad idea to run this experiment exactly as you've designed:

  1. You've suggested that there might be discrimination in how a poster is received based upon their rep, and proposed a methodology to try to confirm this. But there are clearly other things that might prejudice a reader. Readers could plausibly discriminate on the basis of sex, race, nationality, (all of which may be revealed by a user's name or profile picture) and style of username or profile picture. It'd be nice to test these factors too.

  2. Your approach sort-of-hides the asker's rep, but not very well; if I'm in the group that can't see rep, I'll presumably still be able to click through to the answer's profile.

  3. Readers in the group that have a whole load of information suddenly stripped from them probably won't be too happy about this.

  4. Your approach requires work from the staff, which isn't ideal.

  5. You propose splitting readers into two groups with different access to information. There's another way to perform a trial like this, which is for high-rep askers to randomly be assigned an account to ask their question under. The flow would look something like this:

    • A few dozen >10k users, like me, agree that we want to participate in an experiment to find out if question reception is biased by rep/name/gender/race/nationality
    • One of us hacks together an off-site webpage and logins for all of us.
    • Whenever any of us wants to ask a question (or maybe post an answer, too?), first we write it up in full, then we go to that web page and paste the question text into a form. That form records the question (pre-registration!) and randomly chooses whether we should ask the question under our own account or under the account of a new <100 rep user. In the latter case, it randomly selects a gender and race and nationality for us, and then randomly selects either a gender-and-race-and-nationality-appropriate name or a tells us to go by userNNNNNNN for some random NNNNNNN, and then either selects a random gender-and-race-and-nationality-appropriate profile picture for us or tells us to use a gravatar image.
    • If necessary, we create a new account with the details we were randomly assigned. (This might not be necessary if the form told us to use our own account, or if we already have a fake account with the right demographics.)
    • We ask our question and paste a link to it into the form to confirm that we asked it.
    • Over time, we gather a bunch of data about the reception of these questions, which were written in advance by established users but then randomly assigned to users with a variety of demographics. Eventually, we will have enough data to confidently detect bias, if it exists.
    • At the end of the process, we each send the staff a giant account merge request asking to merge all our fake accounts back into our real one.

    This approach would have the advantage of not requiring any development work on the company's side; the community could independently carry out this randomised trial of the effect of a user's profile on their question's reception and all the staff would have to do is merge some accounts at the end.

4
  • 7
    1. While there could be other factors, the intent of this test is to see if this specific thing is a factor or not. 2. Yes, but people usually tend to make snap decisions; they aren't usually bothered to go click their profile. We could also record if a user clicked to see the author's profile. 3. The information isn't stripped, it's still viewable. Plus, in evaluating a question, there's no use for this info. 4. The staff, in recent times, has been willing to conduct experiments, such as changing the "comment" button to "suggest improvements" on IPS. (continued)
    – gparyani
    Commented May 11, 2018 at 8:46
  • (continuing) I checked the page source; to implement this, it's just a matter of removing one div. (That's what I did to produce the screenshot.) 5. The team is likely to object to users running experiments on their own; I once tried running a small survey on Meta Stack Exchange and received a stern warning from a staff member.
    – gparyani
    Commented May 11, 2018 at 8:49
  • 12
    @gparyani "to implement this, it's just a matter of removing one div" - no, it isn't. They've also got to store an extra flag about every user, maybe change their caching layers if there are any caches that currently include that div you mentioned, answer Meta questions about why people can't see anyone's rep any more, and do the statistical analysis of the results themselves (since ordinary users can't see each other's votes). You're asking them to do man-days of work here, at least; calling it "a matter of removing one div" underestimates the workload by multiple orders of magnitude.
    – Mark Amery
    Commented May 11, 2018 at 14:45
  • 2
    A few dozen >10k users, like me, agree that we want to participate in an experiment - A fundamental limitation of an experiment designed this way, is that it doesn't necessarily reflect behavior of a new user (type of problem, familiarity with site features, and so on) so I suspect that the results would have limited value.
    – zero323
    Commented May 13, 2018 at 16:32
-9

I don't know if the current proposal really digs into the root of the assumed bias problem due to the OP's stats/information.

I would extend this idea to always:

  1. Hide the username (if not hide then at least non-linked to profile and remove the detail reveal on-hover)
  2. Hide avatar
  3. Hide rep

of the OP and universally reveal it 15 minutes after it's been posted.

I suggest 15 minutes because I find that this is the period of greatest activity and visits so any comments and votes should be based strictly on the content of the post.

8
  • 3
    How would you effectively communicate with the OP if you can't even @mention him?
    – Cerbrus
    Commented May 12, 2018 at 8:25
  • @1835379 maybe by supporting mention of the userid?
    – Cœur
    Commented May 12, 2018 at 9:44
  • 7
    @Cerbrus, OP gets a notification for all the comments on their post. If it's a matter of addressing the OP without seeing their name, we could use @OP. Commented May 12, 2018 at 13:58
  • @Cerbrus @mention is only supported for @mentioning people who have commented on a post and OP is notified of all comments on their post anyways. In order to handle commenting, OP's username should be shown but not click-able so that long comment thread can properly target a given user. If you really want to see OP's username from the get-go then maybe it can be shown on their post but non-clickable.
    – MonkeyZeus
    Commented May 12, 2018 at 17:54
  • @Cerbrus Seriouly?? This is good idea why go against this also.
    – Suraj Jain
    Commented May 13, 2018 at 16:51
  • @MonkeyZeus I think your idea is wonderful, I also noticed in first 30 minutes or so downvotes and any activity on post is maximum, and I have noticed usually all activity is based on reputation of user rather than actual content of question, after a while it gets settled down, and often good questions are downvoted.
    – Suraj Jain
    Commented May 13, 2018 at 16:53
  • 2
    ^ Context, Suraj posted a answer that violated Google's terms of service. After linking the answer in chat, he got a few downvotes. He claims he was downvoted because of his low reputation.
    – Cerbrus
    Commented May 14, 2018 at 7:09
  • @Cerbrus I was not pointing to that instance brother.
    – Suraj Jain
    Commented May 14, 2018 at 11:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .