192

Has anyone looked into specific numbers of how much reputation is being given out on documentation? It's like a hyperinflation, crashing any meaning to reputation as a measure of trust.

If you take any top up-voted documentation, such as this at 423 up-votes:

enter image description here

Then, look to the 72 users who've edited that post:

enter image description here

This pool of users is now collectively reaping the benefit:

Unnamed user 1, now top 0.54% this month:

enter image description here

Unnamed user 2, now top 0.63% this month:

enter image description here

Another way of looking at this, from three other users:

enter image description here

How much reputation is this documentation example giving per day? 60,000+ reputation points?

While I understand documentation is different than Q&A, where more edits and encouraged to refine documentation posts, at a certain point you'd think these would convert to community wiki.

23
  • 22
    CW isn't meant as a tool to deny rep. If rep-denial is the intended effect here, it's better to fix it at the source - i.e. documentation itself.
    – Mysticial
    Commented Jul 25, 2016 at 16:26
  • 184
    Let's make Documentation not give out any points, the serious folks who want people to build tools correctly don't care for the rep but they care about their docs.
    – JonH
    Commented Jul 25, 2016 at 16:37
  • 6
    Or we could make documentation give some benefit, but reduce it comparatively. Commented Jul 25, 2016 at 19:24
  • 61
    Quit your day job for a couple of months....edit everything you can find...then 2 months later when rep is in 1/4 million range go on job hunt in careers
    – charlietfl
    Commented Jul 26, 2016 at 2:09
  • 41
    By that time, @charlie, employers will have surely caught on to the fact that Documentation has led to a massive reputation inflation for Stack Overflow users, making it even less valuable as a hiring metric. So much for the people who have put in lots of effort composing high-quality answers to difficult programming problems. It's back to proving our competence with FizzBuzz at interviews.
    – Cody Gray Mod
    Commented Jul 26, 2016 at 11:58
  • 7
    This is mine: screenshot. Yeah, some form of limiting needs to happen there.
    – Magisch
    Commented Jul 26, 2016 at 12:33
  • 4
    Damn, 400+ already? C++ doesn't even have anything that hit 100 yet. Come on, C++ voters, get it together.
    – Barry
    Commented Jul 26, 2016 at 12:44
  • 1
    @Magisch I agree, it'd probably cause a steady income of rep. perhaps after a certain number of rep you shouldn't receive +5 per upvote, but only +4..3..2..1 sometime. … But yes, if you have your hands on that many posts, you end up with a good 10k rep nevertheless after a while. But what do you want to do about that?
    – bwoebi
    Commented Jul 26, 2016 at 12:49
  • 1
    @Magisch Similarly it is quite unfair to the late-comers, most regulars already have upvoted and won't give you any +5 rep anymore... Definitely needs a rebalance, but we'll see.
    – bwoebi
    Commented Jul 26, 2016 at 12:51
  • 1
    Log in to SO, +5 (SE inbox, please dont fill up my cap)...buzz (from phone)...another **** notification (computer)...5 mins later...+5 (SE inbox, I JUST EMPTIED IT)...buzz (from phone)...another **** notification (really)...5 mins later...+5 (writing a userstyle to remove it from the SE toolbar)...buzz (my leg is gonna fall off)...another **** notification (turns on do not disturb)
    – geek1011
    Commented Jul 26, 2016 at 13:07
  • 2
    "Hyperinflation is ended with drastic remedies, such as imposing the shock therapy of slashing government expenditures or altering the currency basis. ....Thus it is not surprising that there exist at least seven historical cases in which the good (foreign) money did fully drive out the use of the inflating currency. In the end the government had to legalize the former, for otherwise its revenues would have fallen to zero." from Wikipedia. What does that sound like to you?
    – geek1011
    Commented Jul 26, 2016 at 13:10
  • 8
    @CodyGray "people who have put in lots of effort composing high-quality answers to difficult programming problems" You mean people who are lucky enough to have been on SO up to 2011 when getting rep was easy? Commented Jul 26, 2016 at 15:06
  • 4
    @Barry Wait... are there people who up-vote C++ stuff? I thought the vote counter was capped at 0 for C++ questions. Commented Jul 27, 2016 at 12:27
  • 3
    Since you posted this about +20 users edited more to get into the rep truck Commented Jul 27, 2016 at 15:43
  • 5
    I'm totally for cutting all rep RIGHT NOW until there is some consensus on how rep should be rewarded. Allowing this to go on doesn't help the inflation. What if in the end we decide to just scrap the rep? By that time, who knows the effects of the inflation. Commented Jul 27, 2016 at 16:57

4 Answers 4

59

Has anyone looked into specific numbers of how much reputation is being given out on documentation?

Yup. I'd wager a few of us probably have. So here's a quick breakdown - keep in mind, this is really new and we're still adjusting the fundamentals, so extrapolating from it is sketchy. In particular, I would expect link-vote rep to increase as time goes on. I've included non-Docs reputation figures where appropriate for a sense of scale, since big numbers can be kinda hard to interpret otherwise.

Note: new documentation rep rules have just gone into effect! - the numbers below now reflect the current state of the world.

What follows reflects reputation earned on docs and Q&A between July 21st and September 14th:

  • Net reputation earned via Docs since the start of the public beta: 424,721 points (for comparison, 22,620,852 points were earned outside of docs during this time)
  • Median rep earned by Docs participants who've earned rep since the start of the public beta: 6 points / user (for comparison, 10 points / user was the median earned outside of Docs in this time)
  • Number of users who've earned reputation via Docs in public beta: 9,623 (581,323 users earned reputation outside of docs during this time)
  • Most reputation earned by a single user from Docs in public beta thus far: 3,868 points (max earned by a single user outside of docs over the same period: 19,056 points)
  • Most reputation earned (by all users) for a single example in public beta: 36,025 points (yup, that Array Initialization thing) (max rep for a single post: 5,804 points, for our old friend the branch prediction answer)

And, a raw breakdown of all docs reputation earned or lost, by type:

Name                     Rep Change 
------------------------ ---------- 
Doc Rep Source Removed   -20398     
Example Unupvoted        -16486     
Doc Link Unupvoted       -80        
Doc Link Upvoted         15975      
Proposed Change Approved 53840      
Example Upvoted          391870     

Oh, and here's a fun one: who is earning what portion of that reputation, grouped by the top privilege level attained on the site before the start of the docs beta:

# of Users Privilege level before docs beta % Docs Rep Earned % non-Docs Rep Earned 
---------- -------------------------------- ----------------- --------------------- 
3946401    new user / none                  4.02 %            5.65 %                
342213     Newbie                           0.62 %            1.02 %                
611330     VoteUpMod                        3.24 %            3.88 %                
283855     PostCommenting                   5.23 %            5.12 %                
312992     VoteDownMod                      27.94 %           25.33 %               
13286      EstablishedUser                  1.93 %            2.52 %                
17891      PostEditing                      8.20 %            5.72 %                
23226      CloseQuestion                    25.11 %           15.83 %               
3321       ModerationTools                  7.85 %            6.16 %                
7532       TrustedUser                      3.17 %            6.66 %                
3255       ViewSiteAnalytics                11.72 %           20.83 %               

(% values may not add up to 100% because I've overlooked something important)

A histogram showing how many people are earning various ranges of reputation (most common value by far is 2 points):

a histogram with irregular bucket sizes

And a histogram showing how many examples have generated various ranges of reputation for their editors:

Another histogram with irregular bucket sizes

37
  • 6
    Modal (rather than mean) single-user reputation would be valuable as well.
    – jscs
    Commented Jul 27, 2016 at 2:41
  • 1
    Yeah, maybe later @josh. Super-early numbers, mostly looking for big scary anomalies.
    – Shog9
    Commented Jul 27, 2016 at 2:46
  • 1
    Just curious, are these numbers available in the SEDE or are they internal SO data? Thanks!
    – Panda
    Commented Jul 27, 2016 at 2:52
  • 4
    Would it be convenient to include Most reputation earned by a single user for the Top 5 instead of 1, thanks!
    – Panda
    Commented Jul 27, 2016 at 3:36
  • 3
    Computing from these numbers, we also get a mean rep earned per rep-earning user of 51 for Docs, compared to 16 for Q/A. Commented Jul 27, 2016 at 16:06
  • 5
    @user2357112 Huh, almost like the data bears out that rep inflation everyone sees. Another metric: Doc rep was 14.5% of SO rep (329798 vs 2272898) but it has 4.5% of the users (6459 vs 141918) Commented Jul 27, 2016 at 17:33
  • 1
    wonder if you consider an option of rep recalc if it turns out that things are way off (like it happened in March 2010)
    – gnat
    Commented Jul 27, 2016 at 18:07
  • 2
    Already happened once, @gnat - will almost certainly happen again.
    – Shog9
    Commented Jul 27, 2016 at 18:11
  • 1
    I see, thanks. In that case, idea of Docs rep being shared with that of SO doesn't look as scary. Though it might take a while to find the right balance
    – gnat
    Commented Jul 27, 2016 at 18:14
  • 4
    I believe it's also interesting to compare royalty rep: max rep for a user based on a single edit (Docs) vs. from a single answer (SO main).
    – PM 77-1
    Commented Jul 27, 2016 at 18:23
  • 1
    Went with a histogram instead, @josh - because I like histograms.
    – Shog9
    Commented Jul 27, 2016 at 20:17
  • 3
    Before you get too invested in the idea of "rep inflation", @Two-BitAlchemist, consider that average rep per user has been falling for Q&A for some time now, and is currently at an all-time low. Here's a fun graph: i.sstatic.net/aFPnR.png
    – Shog9
    Commented Jul 27, 2016 at 21:07
  • 1
    Privilege thresholds haven't really changed in years, @two. Commenting still requires 50; full edit rights are still 2k. If earning those privileges is more difficult now than in, say, 2014 then we're dealing with a deflationary system. That really shouldn't surprise anyone. But it presents a challenge when introducing new sources of reputation, because you probably don't want to chase that trend; you want someone documenting, say, React to be earning roughly par with someone answering React questions, not someone still trying to post answers to, say, PHP dups.
    – Shog9
    Commented Jul 28, 2016 at 3:01
  • 4
    @Two-BitAlchemist When you have a highly skewed distribution of data, as is the case here, medians provide a more informative comparison, not means.
    – zarak
    Commented Jul 28, 2016 at 10:47
  • 1
    My goal here is to provide concrete data, @samgak. Ideally, that helps all of us identify the critical problems and frame solutions in a way that'll actually help. Folks have widely varying opinions on the quality of the examples being edited, the usefulness of the edits themselves, and the quality of equivalent answers in Q&A... And as you can see from this answer, the reputation being earned varies widely as well. How - or whether - those correlate is something you'll have to decide for yourself.
    – Shog9
    Commented Aug 1, 2016 at 16:36
151

Disclaimer: Of course I am not implying that everyone currently involved in the Docs is only in it for the rep - that would be absurd.


It seems that SO central understands quite well (maybe too well) the values of gamification. Proof of that is the success of the Q&A SO. What SO Central does not seem to understand is what happens when you experiment too much with your currency.

And what the OP reports is exactly what happens when you merge the Q&A rep system (which has well-known downsides but, to a degree, still holds true to its meaning) and the Docs rep system which has flaws deeply nested in its concept and implementation. What we have here is failure to gamify (aka how not to solve potential participation issues).

(Addressing SO Central): If you fear that participation will be low then progressively convey your intentions to a larger but still qualified userbase. Do not use the Q&A rep as bait. If you do this (well, you did) 2 things are more or less bound to happen:

  • First, you attract en-masse a certain type of users to Docs (viz rep-whores), while leaving other well-meaning potential Docs users/contributors disheartened.
  • Second, you undermine the single most important motive for the Q&A section: a rep system that can be trusted; enough so, as to be used as an (albeit rough) index of personal programming skills. And this will backfire on you. Maybe not now, but in the long run when all the fuss about docs settles down and users ponder on what their reputation means after all.

Maybe what SO Central thought was that after the initial big-whoop influx of low quality content, the more "serious" part of the community would take over and clean the mess. Really? Hand out rep like there's no tomorrow and then expect who exactly to handle the chore? People who care about the docs? But they did not need this kind of rep awards to get involved. And now, instead of having them in the first place create the backbone of the Docs, and make it easier for other users to add quality content upon it, we see the reverse phenomenon: Less qualified users creating the backbone and caring users trying to clean up the mess.

Not to mention that the mere fact that you gain rep for every example upvote is preposterous. And still getting rep even when your contributions are edited out is absurd. And still getting rep when you only contributed at an earlier stage of an example while the upvotes come for the later more substantial version is wrong. I can't stress these enough. This is gamification gone way out of line. In that spirit, why not award rep each time we login? Or when we turn off the ad-blocker? Or when we vote (as other sites do and has rendered their rep system practically meaningless)?

I believe that we keep trying too hard to assert each other that rep is just "worthless internet points". NO. When carefully curated, the Q&A rep system has merits that transcend SO's boundaries: From a purely professional standpoint, that gets translated into better job opportunities (regardless where you stand on the employer / employee spectrum), to the more personal gains an individual gets when trying to better oneself in the process of asking/answering questions. And the common denominator of these beneficial aspects is a permanent and rigorous quality assurance of the rep system itself.

Breeding the new batch of rep-whores

Now, SO Central says that the Docs will eventually level-out and will not offer as much rep. This is like getting folks addicted to crack and then say "that's ok, in the future there will be limited supply of it". Once you addict people into getting massive amounts of rep (even with the rep cap on) for virtually no effort at all, then, when you finally 'level out', you will have created a new model army of rep whores (as if the existing one isn't big enough) rushing into more meaningless edit suggestions and answering even more crappy yet "popular" questions.

The Quality per involvement ratio

Also, Creating arrays in Java doesn't exactly qualify as rocket science or the CS equivalent. It's not a bad example/post but still... The only reason I see it got so many upvotes is because there are people out there hunting badges (like Backer) and this example happens to be the first one in the most popular docs tag (which ofc is shown first). And I am really amazed that only 72 users edited that example given all it has to offer in terms of rep.

Epilogue

I know that StackExchange/SO Central is a company. As such it has to employ techniques that help in establishing and maintaining a leading position in the respective market. But SO's success is only partially a result of successful marketing. Primarily (IMO), it is the result of the formation of a successful community of users around it. And rep is its currency. One would expect SO Central to be more careful when dealing with it. Is this the case here? (rhetorical question)

24
  • 54
    I can't help but think that documentation is a genius way to suck up the repwhores so they have less time to feed the help-vampire/repwhore cycle on SO.
    – Mysticial
    Commented Jul 25, 2016 at 20:32
  • 26
    Once the rep flow in docs tightens I fear they will return more hungry than ever...
    – sokin
    Commented Jul 25, 2016 at 20:36
  • 2
    @Mysticial Rincewind incarnated
    – Braiam
    Commented Jul 25, 2016 at 20:37
  • 4
    @Mysticial Docs feels like camp 5 (camp 5 = (camp 3)^2). I think you need to make a new diagram :]
    – Rizier123
    Commented Jul 25, 2016 at 20:49
  • 66
    Well said! I like rep, but I also like the idea I deserved it. This SE Docs rep system feels like a slap in the face. Really, I feel personally offended to see given away for (almost) free now what took me five years of conscientiously answering questions. Of course this attracts rep whores like crazy. Worse, It deters users who like to write good documentation and see their work vandalized by others who seem to see their carefully composed example only as a free ride to more rep. Commented Jul 25, 2016 at 21:41
  • 3
    It just gets annoying when users misuse rep to censor others and even close questions they dont like after getting it cheaply
    – Dr Deo
    Commented Jul 26, 2016 at 7:16
  • 8
    To be fair, I don't think that Creating arrays in Java has so many upvotes because of badge hunters - instead, it is likely to have those votes because it is helpful to the (very large) number of novice programmers. Commented Jul 26, 2016 at 12:02
  • 6
    Do we have to prepare for rep inflation? Seems like there will be a Black Rep Friday. None of us would like to see unicorns jumping out from windows.
    – totymedli
    Commented Jul 26, 2016 at 12:53
  • 2
    Log in to SO, +5 (SE inbox, please dont fill up my cap)...buzz (from phone)...another **** notification (computer)...5 mins later...+5 (SE inbox, I JUST EMPTIED IT)...buzz (from phone)...another **** notification (really)...5 mins later...+5 (writing a userstyle to remove it from the SE toolbar)...buzz (my leg is gonna fall off)...another **** notification (turns on do not disturb)
    – geek1011
    Commented Jul 26, 2016 at 13:14
  • 4
    "Hyperinflation is ended with drastic remedies, such as imposing the shock therapy of slashing government expenditures or altering the currency basis. ....Thus it is not surprising that there exist at least seven historical cases in which the good (foreign) money did fully drive out the use of the inflating currency. In the end the government had to legalize the former, for otherwise its revenues would have fallen to zero." from Wikipedia. What does that sound like to you?
    – geek1011
    Commented Jul 26, 2016 at 13:15
  • 16
    @geek1011: that's exactly the problem, and that's why their statement about "still adjusting the thresholds" also sucks. Perhaps they should have posted their rep algorithm on code review before going live, because it's clear to most people reading this that the algorithm is ridiculously gameable.
    – vgru
    Commented Jul 26, 2016 at 13:20
  • 3
    @Groo Yes. Or at least they should have made doc worth nothing, then changed it after the first month, suprising everyone who contributed and not attracting rep whores
    – geek1011
    Commented Jul 26, 2016 at 13:21
  • 1
    It's humorous to me that this is a problem to everyone now, but in the past on the q&a site, the restrictions on questions were much more lax and many more "easy" questions being asked to be answered, that still get upvotes to this day, since anything similar is closed as a duplicate, meaning it was much easier to get rep in the past. People talk about "gamification" but demand that their SO rep function as a real life currency, or evaluation of their worth. Please, get rid of all rep entirely, if only to stop the whining.
    – Kik
    Commented Jul 26, 2016 at 21:02
  • 7
    @kik -You seem to be confused about what gamification is. The fact that it is strategically or tactically deployed in order to achieve mobilization via incentives does not necessarily mean that the outcome index (rep in our case) lacks any ties to "reality". On the contrary, when the index producing system is balanced and curated, it can "escape" its virtual origin and have actual impact in the "real" world. And (as is true with every currency) in the end, it means what we want it as a community to mean. Whether it is far from or close to reality is the issue that we are all discussing here.
    – sokin
    Commented Jul 26, 2016 at 22:55
  • 4
    I wonder if it would be appropriate to start suggesting "Consider placing a bounty XXX for your question since answering it would require more than copy-paste kind of effort" where XXX would be in the range of 300-500 due to this recent inflation ;) Commented Jul 27, 2016 at 11:15
25

I'd like to propose an alternative solution:

Split the rep system.

The main problem here seems to be that Docs reputation is not "equivalent" to Q/A reputation, at least in terms of how hard it is to earn. Thus, by making Docs rep separate from Q/A rep, we can maintain the gamification the reputation system provides and encourage people to contribute to Docs, without potentially "devaluing" people's hard-earned Q/A reputation.

1
2

I think instead of giving out +5 points to each contributor (and in this way bloating up the number of given rep points for each topic), the points should be divided between the contributors and not handed out additionally.
E.g. there are 5 contributors for one topic, then each contributor will get +1 reputation for an upvote. If there are 10 contributors, you'll get +.5.

Since broad topics with a lot of upvotes will most likely have most contributors, this system would make a lot of sense. It would be more 'valuable' to post a doc for a very particular topic that only one specialist knows about since there will likely be less contributors.

The only downside I see is that you end up with fractions of points. Instead of awarding them right away, I would combine them and only award full points when they are reached.
E.g. You will only be award 1 rep point when two users upvote a topic that you created with 9 other contributors.

1

Not the answer you're looking for? Browse other questions tagged .