40

In a couple of recent threads regarding how users of higher reputations behave there is a lot of speculation going on. Could we get some summarized statistics to help in these discussions?

I think some information on the following would be helpful in these discussions:

  • Voting patterns (up vs. down) of users at various reputation levels across the entire site

    • Less than 125 (users can only upvote)
    • Between 125 and 500 (Users can down vote and access review queues)
    • Between 500 and 3,000 (users get access to all review queues and can edit without peer review)
    • Between 3,000 and 10k (No man's land?)
    • 10k to 20k (Moderator tool access)
    • 20k+ (Trusted users)

    This would be helpful in seeing if higher reputation levels are less selective in using their downvote. My hypothesis is that higher levels downvote more often. The -1 stings less for someone with 10k than to someone with 127. My guess is that this flip occurs most commonly around the 10k mark, when users have access to the moderation tools and can more easily see the lower quality stuff on the site.

  • Voting patterns (up vs. down) of users at various reputation levels on questions that are closed within 24 hours of being posted and remaining closed for more than a week (and for what reason it was closed).

    I think this information will help to answer the question of who is voting for questions that ultimately end up remaining closed because of off topic-ness. It may also help to pin point what groups need assistance in utilizing Stack Overflow to its fullest extent. If we want to promote high quality questions, we should encourage our user base to vote for high quality questions.

  • Voting patterns (up vs. down) of users at various reputation levels on answers that are removed within 24 hours of being posted and remaining closed for more than a week (and for what reason it was removed).

I think this will help show which groups see posts that others are flagging as low quality as worthy of being up voted. Again, this can be used to help educate users in what is and is not valuable and how we can keep the site high quality.

  • Reputation level at which users start utilizing their voting powers. For the sake of this, let's say a user is truly "using" their voting powers after they have cast 10 votes of a certain type. Do users immediately start upvoting when they hit 15 reputation or are they waiting until they have more rep? Do users immediately start down voting when they hit 125 rep or do they wait until they have more rep?

Unfortunately, a down vote costs a user 1 rep, so I'm not entirely sure how to measure the down vote until they repeatedly hit 125

  • Voting ratio by reputation toward users of various reputations? Are users of higher reputation levels truly skewing the votes toward their posts vs. someone of lower reputation levels?

I think this type of data would be most useful to see on posts where someone of 20k+ and someone of <100 answer the same question. Does the user with 20k receive more up vs. downvotes? This will be a straight numbers thing, with the giant disclaimer that one answer may be truly better than the other. A query comparing scores doesn't easily show quality between posts.

These are a few points that I've noticed are being discussed and assumptions are being thrown around. With some of this data, we could have a more informed discussion.

It'd also help, I think, if we could get some interpretation from the community manager(s) that post this data. Trends that we see may or may not be valid. What do you see that we don't?

Related Threads

12
  • Complex SEDE based on user-id perhaps? Commented Jul 22, 2015 at 19:46
  • 9
    Voting is anonymous in SEDE. Only a community manager could do several of these requests.
    – Andy Mod
    Commented Jul 22, 2015 at 19:47
  • 1
    So, as you know, what you're actually asking about? Disclosing voting behaviour? Commented Jul 22, 2015 at 19:50
  • 4
    I'm asking for summarized voting patterns. I'm not asking for specific users, but groups of users in larger reputation ranges. I don't care how you vote, but I'm interested in how 20K+ utilize their available votes and comparing that to how users with less than 100 rep utilize their votes.
    – Andy Mod
    Commented Jul 22, 2015 at 19:52
  • 37
    No one ever just asks for one easily-queryable number. Always gotta be 7 different perspectives with weird little conditions that take ages to run. grumble grumble
    – Shog9
    Commented Jul 22, 2015 at 19:56
  • 1
    Well, unless the detailed information isn't disclosed from the available data model, and there are no anonymous statistics gathered from these by the engine internally, there's no way to get that information. Are you actually asking to extend the SE engine to gather such statistics? Commented Jul 22, 2015 at 19:59
  • 3
    How do you think this would help the site? Commented Jul 22, 2015 at 21:05
  • @pnuts And there's no anonymous statistical data tracked and publicly visible. Commented Jul 22, 2015 at 21:09
  • 3
    @DisplayNameismissing It'd allow a conversation to occur with actual data instead of sweeping generalizations that new users do X or high rep users do Y
    – Andy Mod
    Commented Jul 22, 2015 at 21:35
  • 4
    @πάνταῥεῖ There are plenty of anonymous, aggregate statistics available. This is a request for one more so that we can have a conversation without generalizations.
    – Andy Mod
    Commented Jul 22, 2015 at 21:36
  • 6
    @Andy We haven't forgotten about this request, it's just a bit slow. I'm working on it.
    – Taryn StaffMod
    Commented Jul 24, 2015 at 16:05
  • is there new data on this topic? It would be interesting to see if behavior is changing over time... This is an interesting data set.
    – BenPen
    Commented Oct 26, 2016 at 21:02

3 Answers 3

43

Well, it only took 6-8ish days to gather the data for this, but a few of us have pulled together most of stats that you requested. We queried the voting history over the past 90 days, and looked at votes grouped by the reputation of the voter at the time the vote was cast.

Voting patterns (Up vs Down) of users at various reputation levels

Rep range Vote type Median reputation Median votes per user Users Votes
< 125 Up 50 3 88069 584,813
< 125 Down 112 2 27 79
125 - 500 Up 308 4 60643 702,743
125 - 500 Down 300 1 17261 59,094
500 - 3k Up 1046 6 63502 1,300,139
500 - 3k Down 1092 2 18715 157,326
3k - 10k Up 4743 9 16118 454,965
3k - 10k Down 4948 2 6152 115,755
10k - 20k Up 13430 13 3649 140,620
10k - 20k Down 13698 3 1760 60,799
> 20k Up 33268 22 2875 191,217
> 20k Down 35619 7 1832 132,856

Observations

  • There are a lot of low rep users, but the majority sit in the middle of the privilege chart
  • Very high rep users upvote more than they downvote
  • Very high rep users who vote do so more than low-rep users

Voting patterns for fast-closed questions

These are the voting patterns (up vs down) of users at various reputation levels on questions closed within 24 hours of being posted which remained closed for more than a week. They're broken out by close reason.

Up/down voting details on closed questions for each of the reputation levels:

Close Reason                                              Vote type < 125 Votes 125 - 500 Votes 500 - 3k Votes 3k - 10k Votes 10k - 20k Votes > 20k Votes 
--------------------------------------------------------- --------- ----------- --------------- -------------- -------------- --------------- -----------     
duplicate                                                 Up        958         1280            3628           1641           643             1287        
duplicate                                                 Down      1           1220            3230           3587           1599            5032        
off-topic Other...                                        Up        68          59              119            71             21              63          
off-topic Other...                                        Down      0           377             1110           896            586             1092        
off-topic general computing hardware and software         Up        46          51              90             39             9               20          
off-topic general computing hardware and software         Down      0           208             782            666            529             814         
off-topic recommend or find a book, tool, software        Up        211         263             457            159            41              94          
off-topic recommend or find a book, tool, software        Down      1           1272            3744           2889           1945            3313        
off-topic professional server- or networking-related      Up        13          12              38             9              3               10          
off-topic professional server- or networking-related      Down      0           18              125            122            148             116         
off-topic debugging help                                  Up        248         304             747            258            57              140         
off-topic debugging help                                  Down      4           1380            3802           3529           2043            4557        
off-topic belongs on another site in the Stack Exchange   Up        26          23              76             29             9               18          
off-topic belongs on another site in the Stack Exchange   Down      0           38              144            95             60              133         
off-topic out of scope                                    Up        3           2               2              1              1               3           
off-topic out of scope                                    Down      0           66              198            67             49              118         
off-topic typo                                            Up        71          103             326            109            53              83          
off-topic typo                                            Down      0           224             666            651            455             1086        
primarily opinion-based                                   Up        204         277             622            256            111             161         
primarily opinion-based                                   Down      1           634             1778           1480           1030            1839        
too broad                                                 Up        551         599             1108           407            139             253         
too broad                                                 Down      5           3117            9251           6753           4402            7673        
unclear what you're asking                                Up        385         399             947            307            100             216         
unclear what you're asking                                Down      3           2750            8305           6380           3756            7027        
Total                                                     Up        2784        3372            8160           3286           1187            2348        
Total                                                     Down      15          11304           33135          27115          16602           32800  

The number of distinct users voting on these closed questions:

Close Reason                                              Vote type < 125 Users 125 - 500 Users 500 - 3k Users 3k - 10k Users 10k - 20k Users > 20k Users 
--------------------------------------------------------- --------- ----------- --------------- -------------- -------------- --------------- ----------- 
duplicate                                                 Up        736         885             1918           816            321             495         
duplicate                                                 Down      1           754             1316           713            310             515         
off-topic Other...                                        Up        65          58              111            65             21              56          
off-topic Other...                                        Down      0           313             714            406            176             314         
off-topic general computing hardware and software         Up        43          46              82             33             9               17          
off-topic general computing hardware and software         Down      0           181             468            244            101             194         
off-topic recommend or find a book, tool, software        Up        189         214             384            137            36              76          
off-topic recommend or find a book, tool, software        Down      1           834             1451           682            280             453         
off-topic professional server- or networking-related      Up        12          12              37             9              3               8           
off-topic professional server- or networking-related      Down      0           17              106            53             27              58          
off-topic debugging help                                  Up        213         255             552            180            48              102         
off-topic debugging help                                  Down      3           923             1577           839            330             536         
off-topic belongs on another site in the Stack Exchange   Up        25          23              68             27             9               18          
off-topic belongs on another site in the Stack Exchange   Down      0           38              123            70             33              75          
off-topic out of scope                                    Up        3           2               2              1              1               3           
off-topic out of scope                                    Down      0           64              162            50             29              66          
off-topic typo                                            Up        65          98              277            84             48              61          
off-topic typo                                            Down      0           195             458            282            148             282         
primarily opinion-based                                   Up        187         252             547            202            84              116         
primarily opinion-based                                   Down      1           489             910            506            224             353         
too broad                                                 Up        449         443             831            292            108             167         
too broad                                                 Down      3           1654            2611           1152           464             657         
unclear what you're asking                                Up        314         323             667            205            80              141         
unclear what you're asking                                Down      2           1556            2602           1185           454             699         
Total                                                     Up        1822        1876            3392           1333           500             713         
Total                                                     Down      7           3939            5151           2099           737             1020        

Voting patterns on fast-deleted answers

These are the voting patterns (up vs down) of users at various reputation levels on answers that were deleted within 24 hours of being posted and remained deleted for more than a week. They're broken out by flag type (when a flag was involved in the deletion).

Voting stats on deleted answers, including those flagged NAA, VLQ, or no flag:

Flag                 Vote type < 125 Votes 125 - 500 Votes 500 - 3k Votes 3k - 10k Votes 10k - 20k Votes > 20k Votes 
-------------------- --------- ----------- --------------- -------------- -------------- --------------- ----------- 
Answer Not An Answer Up        287         284             452            179            54              102         
Answer Not An Answer Down      2           807             6786           4228           2923            5804            
Post Low Quality     Up        51          40              61             32             10              12          
Post Low Quality     Down      1           170             914            602            488             745         
No Flag              Up        2273        1919            3482           1860           694             1559        
No Flag              Down      2           2338            5382           4020           2363            7207   
Total                Up        2611        2243            3995           2071           758             1673        
Total                Down      5           3315            13082          8850           5774            13756    

The number of distinct users voting on the deleted answers:

Flag                 Vote type < 125 Users 125 - 500 Users 500 - 3k Users 3k - 10k Users 10k - 20k Users > 20k Users 
-------------------- --------- ----------- --------------- -------------- -------------- --------------- ----------- 
Answer Not An Answer Up        170         168             240            90             34              46          
Answer Not An Answer Down      1           378             1522           565            237             448         
Post Low Quality     Up        47          31              47             23             10              11          
Post Low Quality     Down      1           133             492            231            118             207         
No Flag              Up        2072        1649            2488           1117           390             630         
No Flag              Down      2           1801            2875           1357           534             906         
Total                Up        2248        1779            2645           1172           406             646         
Total                Down      3           2083            3750           1588           599             980    

Notes on methodology

  • Some downvotes are recorded for users < 125, this is due to vote invalidations, users being deleted, and other rep oddities that can happen. While we've done our best to estimate each voter's reputation at the time they voted, doing so perfectly is prohibitively expensive—rather than hiding these results, we've left them in to give you a rough idea of the overall level of accuracy.

  • Because reputation was tracked along with the vote, users could end up in more than one "bucket" during the 90-day timeframe. For instance, if a user had < 125 at the start of the timeframe but crossed the rep threshold, then their votes at the start will be counted in the < 125 bucket, while the votes after will be counted in the 125 - 500 bucket.

11
  • 2
    Ouch! Food for thought... Can we get access to the queries so we can run these on other sites?
    – Fabby
    Commented Jul 30, 2015 at 20:59
  • 1
    @Fabby Voting data isn't available in Data.SE so you wouldn't be able to run this for other sites, it'd have to be run by a SE employee.
    – Taryn StaffMod
    Commented Jul 30, 2015 at 21:01
  • But, but... you saved them, right? RIGHT?!
    – Braiam
    Commented Jul 30, 2015 at 21:06
  • 11
    @Braiam Nope, I trashed them as soon as I posted this.
    – Taryn StaffMod
    Commented Jul 30, 2015 at 21:08
  • @bluefeet: What OS are you running? I've got a sleuth of undelete utilities... ;-)
    – Fabby
    Commented Jul 30, 2015 at 21:42
  • So the 500-3K rep users bracket casts 15 thousand upvotes a day. Interesting. Thanks for doing this, we'll see how this data will be (ab)used in future discussions. :)
    – CodeCaster
    Commented Jul 30, 2015 at 21:43
  • What would trigger no flag deletions of answers in the Voting patterns on fast-deleted answers section? Are all of those no flags self deletes?
    – Andy Mod
    Commented Jul 31, 2015 at 2:20
  • @andy I'll check and let you know to be sure but that'd be my guess.
    – Taryn StaffMod
    Commented Jul 31, 2015 at 2:23
  • 1
    See meta.stackoverflow.com/questions/296608/… for the myriad reasons why answers get deleted, @andy
    – Shog9
    Commented Jul 31, 2015 at 2:51
  • wonder how many votes on average users cast in their first day at the site - in other words, what could be the impact of getting rid of first day voting (I guess it would be negligible, since not many users get +15 in their first day here)
    – gnat
    Commented Aug 3, 2015 at 5:58
  • This is amazing data. The conclusion that I draw is that high rep users are STILL voting up questions that are closed for cause or answers deleted for cause. I guess the N of the number of closed questions is missing, but still, that's quite a tail of people seeing value in ultimately discarded answers/questions.
    – BenPen
    Commented Oct 26, 2016 at 20:50
18

A question came up internally regarding the increase in downvoting observed as reputation increases... The conventional wisdom here is that downvoting is initially rare because of the reputation cost—indeed, this was the rationale for having downvotes cost rep, and much experience since then has confirmed the notion that this cost does indeed discourage downvoting…

However, we can test this another way: How do folks vote when they're new to the site, when they don't even have an account yet?

Vote type Median votes per user Users Votes
Up 1 590,534 873,225
Down 1 425,229 580,843

Those are the statistics for the past 90 days worth of anonymous feedback. These "votes" don't count toward post scores, and "users" are identified only by IP address, so take that column with a grain of salt. What's important here is the breakdown: nearly 40% of all feedback is downvotes, only a hair under the percentage for 20k+ users—in other words, the voting breakdown is quite similar for everyone who doesn't have any reason to worry about the rep-cost, regardless of how much time has been spent on the site.

You can play with this data yourself in SEDE.

11
  • Breaking this down by votes on questions (which don't cost rep for registered downvoters) and answers (which do) is left as an exercise for the reader...
    – Shog9
    Commented Jul 31, 2015 at 21:12
  • 6
    FWIW I have had to tell people many times that downvoting questions doesn't cost rep. A lot of people don't realize it and never downvote to even find out.
    – durron597
    Commented Jul 31, 2015 at 21:18
  • 1
    I suspect most of the negative anonymous feedback means "frustratingly, this post is not what I was looking for". This isn't a common reason for actual downvotes.
    – user3717023
    Commented Jul 31, 2015 at 21:19
  • 1
    Sussing out the actual reason for downvotes is another interesting exercise, @NormalHuman. One might make a bit of hay by identifying votes cast on posts a significant amount of time after they were posted.
    – Shog9
    Commented Jul 31, 2015 at 21:21
  • 2
    The reason for making them cost rep was skewing feedback? I thought it was more along the lines of ensuring they are used responsibly, instead of for fighting the competition, griefing (especially over moderation actions) and other such unwelcome acts. Anyway, it's interesting that anonymous users downvote as much as high-rep-users, even without knowing whether those votes are roughly for the same reasons... Commented Jul 31, 2015 at 21:22
  • You don't consider those skew, @Deduplicator? I mean, it's not like folks don't upvote frivolously...
    – Shog9
    Commented Jul 31, 2015 at 21:28
  • @Shog9: Well, the first is skewing the ratio between up/down. The latter is just degrading the signal in either class. Both are bad, if one wants the best possible evaluation of a posts worth, imperfect as it invariably is. Commented Jul 31, 2015 at 21:31
  • The assumption is that if folks are reluctant to use them, they'll only use them when they feel strongly, @Deduplicator. But again, there's no way to guarantee that.
    – Shog9
    Commented Jul 31, 2015 at 21:33
  • Yes, I get that. Which means there's a bias to upvoting, as there's no deterrent for that. Which might not be so bad, I would have less rep otherwise ;-). The interesting questions are 1. how good the 20K+ and unregistered voting-patterns align (which needs detail, and is probably quite hard to measure with any accuracy due to limited voting-populations), 2. whether the decrease in downvote-amount is actually counterbalanced by a corresponding increase in accuracy if there's a difference, and 3. whether downvoting-signal by members would digrade much if downvotes were free on answers. Commented Jul 31, 2015 at 21:45
  • 1
    If you've never read them, these two blog posts should give you a good understanding of the rationale here, @Deduplicator.
    – Shog9
    Commented Jul 31, 2015 at 22:10
  • Had not read the second one before. And I think the at most 2/3 downvotes rule was abolished long ago... Thanks. Commented Jul 31, 2015 at 22:41
6

I've converted a few of the tables from @bluefeet's answer to visuals.

Number of up vs down votes per reputation level

The biggest thing that I'm seeing in this particular data set is that the ratio of up to down votes decreases as the reputation levels increase.

I also find it interesting that the 10k to 20k group votes less than the > 20k group. I suspect that is because the 20k group is unbounded, thus has more users, but it is still interesting to see.


Votes per close reason per reputation level

This chart is busy looking, but the biggest thing that I'm noticing is that questions that are closed quickly receive a much higher number of down votes than upvotes. I think this is to be expected and encouraged. If a question is quickly closed and remains closed, it isn't high quality. It's nice to see the data showing this trend.

The exception to the above appears to be duplicate questions that are closed. At lower reputation levels, duplicates that get closed quickly are receiving more up votes than down votes. It isn't until the 3k mark (when users can see the close review queue) that this trend turns around and duplicates start getting more closes than opens.

Through all levels that can down vote, it seems that "too broad" and "primarily opinion based" attract the most down votes.

Once again, the 10k to 20k group is casting less votes than the over 20k users.


Total up vs down votes on quickly closed questions

This is showing the totals of the busier per close reason chart from above.


I think it's interesting to see ratio of up to down votes, compared groups that can see all the review queues to those that can not. Once users get access to those, the ratio gets closer and closer to 1:1. It is worth noting that the number of users in each reputation level also drops, so comparison of absolute numbers can be misleading.

Users don't appear to be stingy with their down votes on off topic questions. In all reputation levels that can down vote, the quickly closed questions receive many more down votes than upvotes.

1
  • 12
    Be very careful comparing raw numbers between groups - these are not at all equal-sized groups. The data is there if you want to normalize these. To say that such comparisons can be misleading is a gross understatement...
    – Shog9
    Commented Jul 31, 2015 at 2:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .