93

I study voting on Stack Overflow and found a sudden change in negative/positive voting ratio on questions in April 2014. I would imagine that such a drastic change will result from a change in Stack Overflow policy, features or UI, but I found no evidence to such a change in the features documentation. The following SEDE query result shows the change.

Enter image description here

I found a previous post from June 2014 about the topic which was not answered. Some of the comments suggested a temporary issue/backlog that would have been corrected long ago if this was the case. Any ideas? Directions?

26
  • 2
    This could be a result of changes to the review queues in March. Commented Aug 31, 2015 at 19:07
  • 32
    The planet got dumber on April 2014 and it seems it's getting dummier again this past months! We're doomed obviously. Commented Aug 31, 2015 at 19:33
  • 1
    Wow, 12% of votes where down up to 22% in 2 months. And no obvious inflection point. It isn't even September. Working out May 2014 might also be worthwhile. Wait, there was also something about a spike in access from a new region recently? Commented Aug 31, 2015 at 19:35
  • 38
    @JonathanDrapeau "it's getting dummier again this past months!" If you are referring to the spike at the end of the graph, I'm pretty sure that spike is a side-effect of auto-deletion of questions. The downvoted questions from older months have reached the deadline for auto-deletion and are not longer visible in SEDE. The newer questions that will eventually be deleted are still visible, however.
    – Louis
    Commented Aug 31, 2015 at 19:39
  • 8
    @Yakk It's been September since at least 2009. Commented Aug 31, 2015 at 19:49
  • 24
    @BilltheLizard 1993 was when September began, silly. Commented Aug 31, 2015 at 19:50
  • 6
    I can't tell exactly which dates the huge spikes are, but it looks like sometime around the beginning of August. Just a thought, but at least in the US, school starts around August, which usually leads to a whole batch of college students that want someone to do their homework for them (and asking questions that have been asked a million times before).
    – Becuzz
    Commented Aug 31, 2015 at 19:52
  • 14
    @PeterDuniho The most heavily downvoted questions from earlier months have already been deleted, but not from recent months. This causes a spike to appear at the end of the graph, but it won't be there once some of those questions are deleted. (It will just move to a later time.) Commented Aug 31, 2015 at 19:57
  • 7
    looks like an effect of big burn down of close queue. More questions getting closed faster, without getting answers (and without desperate accepts) => more questions get auto-deleted => votes down on deleted question disappear from stats. Related: Can some metadata about deleted posts be included in Data.SE?
    – gnat
    Commented Aug 31, 2015 at 20:07
  • 2
    Does anyone have any guesses for what happened in April/May of this year? The downvote surge from burning down the close queue appears to've dropped significantly in size around then; but a quick check on the review page shows that the close queue is still full of stuff that needs looked at. Commented Aug 31, 2015 at 20:25
  • 3
    @DanNeely Triaged questions stuck in limbo instead of getting closed could have something to do with this
    – gnat
    Commented Aug 31, 2015 at 20:29
  • 3
    You can see a spike in December 2014, probably from the Winter Bash game. Commented Sep 1, 2015 at 3:25
  • 1
    @CodyPiersall, you're right. The spike runs from Dec 15th to Jan 5th -- which corresponds exactly with Winter Bash. It looks like there was also a modest bump during Winter Bash 2013. ... It's kinda sad that Winter Bash reduces the Christmas cheer! Maybe it should be renamed "Newbie Bash" and all of the hats would be replaced with skulls. :) Commented Sep 1, 2015 at 5:56
  • 2
    @louis you may be interested to know that this is also an issue encountered in infectious disease study: when a new disease emerges it's hard to tell what the survival rate is because of this sort of effect. We have lots of identified cases, but we don't know the final outcome for them. So if we measure, say, symptom severity at arriving at hospital, we can estimate how many severely ill people who arrived months ago survived, but we don't know about the recent arrivals. If the disease is new, we can't see what the plateau looks like.
    – Joel
    Commented Sep 1, 2015 at 12:05
  • 3
    SO's April Fools joke was just that good.
    – Stryner
    Commented Sep 1, 2015 at 13:13

3 Answers 3

56

As others have observed, the graph is skewed by the absence of deleted posts, since older downvoted posts are typically deleted. Here's what the graph looks like when deleted posts are included:

enter image description here

Two things to note:

  1. The jump in the last month is gone. This isn't surprising. Recent downvoted, closed posts are often deleted, both automatically and by the community, so more downvoted posts are visible in the last few weeks than in the more distant past.
  2. The jump in April 2014 is still there. Whatever caused this jump, it was not an artifact in question-deletion.

If we zoom in a bit on April-May 2014, we can see where the jump occurred.

enter image description here

If we had to pick a day where the jump occurred, it would be Saturday, April 26, 2014, shown in red. (Fun fact: the downvote/upvote ratio spikes on weekends. This occurs in other months as well).

I've done a little preliminary work to eliminate hypotheses:

  • This isn't statistical chance. The pattern occurs across many days, to an extent far more consistent than random noise. (The confidence ribbons shown above are from a binomial proportion confidence interval).
  • This isn't a few "bad apples." The voting is not being driven by a small number of people (nor could it, really, due to per-person, per-day voting limits). Note that each of these days include at least thousands of downvotes.

Based on this, and on a few brief discussions with others at the company, I'd advance a tentative hypothesis that this is related to the Meta Stack Overflow/Meta Stack Exchange split, which occurred on April 17. This split led to increased attention to Meta, including highly-discussed questions such as Why is Stack Overflow so negative of late? (April 23), as well as concerns about problems with quality (this question was April 28, but there were other discussions around the same time). It's possible that this kind of introspection led people to downvote more as part of their normal activity. While the timing is very suggestive, I don't have solid evidence of this yet. I'm open to your ideas for further analyses!

Finally, I'd note that if you look at the first graph above, the downvote/upvote ratio is dropping again- indeed, it's now right around the levels it was before April 2014. (This effect was mostly masked in the OP by the effect of deleted posts). Some preliminary analyses indicate that that while the April 2014 jump was due to an increase in downvotes, this recent change is from an increase in upvotes.

6
  • 3
    Very insightful
    – Hogan
    Commented Sep 2, 2015 at 21:15
  • What confidence interval did you use? It seems either pretty tight, or none of the above variation is "chance-like". Commented Sep 3, 2015 at 17:43
  • @Yakk 95%. Note that this covers "chance-like" variation if we assumed votes are all independent (like individual coin flips), which is a very non-conservative assumption. (For instance, the votes cast by a single person in a day are certainly not independent. If one person signed on and cast only downvotes, that would break these assumptions- though as stated above this wasn't just a "few people"). You could think of it as a lower bound on our uncertainty
    – David Robinson Staff
    Commented Sep 3, 2015 at 17:50
  • I had marked this question as favorite only to see your answer. Cheers. Commented Sep 3, 2015 at 17:57
  • 10x david for looking into it so thoroughly. I have no further directions at this point in time. I plan to study it more in the near future. What about system user process -1 which is responsible for over half a million downvotes? Is this some automatic process that votes in specific cases? Can it be the cause?
    – Benny
    Commented Sep 9, 2015 at 18:31
  • @Benny the Community user owns downvotes on deleted questions: stackoverflow.com/users/-1/community (it means people get back their spent reputation point once the question is deleted, but the votes remain)
    – Dave
    Commented Sep 9, 2015 at 21:20
12

Questionable truths are usually categorized as lies, damn lies and statistics. Your graph fits the 3rd category. It is much too sensitive to changes in the number of downvotes. The auto-scaling makes it even more excessive.

Your query calculates downvotes/upvotes. That's misleading, a numerical example can show that. Say in March of 2014, there was 1 downvote and 99 upvotes. You calculate ~0.01. And April had 2 downvotes and 98 upvotes. You calculate ~0.02. Omg it got 100% worse. Well, no, it did not. The proportion of downvotes only increased by ~1%.

A decent query that shows voting patterns is this one.

But yes, there was a modest uptick in the number of downvotes in April of 2014. And downtick in the number of upvotes as well, multiplying the effect in your graph. A memorable month, that's when SO got its own meta. And a bright yellow box appeared on the front page in a prime location.

Participation was very heavy, a fair amount of dirty laundry got hung out to dry. Probably fair to say that it didn't do much to inspire SO users.

9
  • 2
    I'm skeptical of "that's when students are finishing their end-of-semester assignments" as being a post-hoc explanation. a) If it had happened in January or September, we could have said "That's when new students are starting"- it's easy to fit this kind of explanation to any data. b) The trend continues for nearly a year, rather than dropping right after. c) This April/May actually saw a mild decrease in downvote/upvote ratio.
    – David Robinson Staff
    Commented Sep 1, 2015 at 14:51
  • Well, you're the data scientist, please share the facts you know. Commented Sep 1, 2015 at 14:57
  • 1
    @HansPassant, The daily voting numbers in SO are such that this small numbers example that you gave doesn't hold. The ratio changed from being 1 downvote to every 10 upvotes to almost 1 downvote for every 7 upvotes. I see it as more than a modest uptick. When I examined downvotes from the users table I saw that the system user process -1 is responsible for over half a million downvotes. Can it be the case that it kicked in at this point in time?
    – Benny
    Commented Sep 1, 2015 at 15:54
  • 1
    The numerical example is simply there to show that the methodology is flawed, not to be representative of actual numbers. Commented Sep 1, 2015 at 16:11
  • 1
    YES! A graph with auto-scaling (and/or without units) is almost always misleading. Commented Sep 1, 2015 at 17:53
  • 1
    @HansPassant Your numerical example demonstrated that the methodology (division of two terms) is flawed if the values have high variance with respect to their magnitude. This is not the case here. Commented Sep 3, 2015 at 17:40
  • @BrockAdams The vertical axis of the OP is legitimately unitless; it is a ratio between two things which have the same units. Commented Sep 3, 2015 at 17:40
  • @Yakk, hence the "almost always" in my comment. ;) This would be one of the rare exceptions that prove the rule. Commented Sep 3, 2015 at 17:44
  • Your graph is interesting. It appears most of the change is driven through spikes in upvotes; downvotes seem to be an a uniform increasing slope. Two nearly baseless hyphothesi: new users tend to mainly upvote, and those spikes represent new users arriving in clumps. Or, there is a serious bot-voting problem, and each of those spikes is a bot-voting technique that has succeeded. They get caught over time. The splitting of mso from mse screwed up a population of such upvote bots, leading to a decreased number of upvotes after that weekend. Both are unlikely. Commented Sep 3, 2015 at 17:46
6

April 2014 had a lot of meta discussion on dropping quality:

One of the answers even specifically called out high-rep users who don't downvote.
I can't speak for others, but I know I started downvoting more questions. I didn't like downvoting, but I realized I wasn't doing much good by not down-voting poor questions.

2
  • 1
    So you are saying, April 2014 is your fault? Commented Sep 1, 2015 at 13:36
  • 1
    A drop in the ocean.
    – Kobi
    Commented Sep 1, 2015 at 13:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .