155

I've gotten over 350 rep points* since last week for this here Documentation edit. I think it made the example better; I'm pleased to have been able to contribute.

On the other hand, rewriting a couple of sentences has gained me more reputation -- even at +5 an upvote instead of +10 -- than any but the top 10 of my Q&A posts (out of over a thousand). That's honestly kind of embarassing. It devalues my answering in my own mind. (Also.) I'm one of nearly 40 editors of that Example as of this moment. There are 22 editors after me. I didn't even create the dang thing: I just expanded it a bit. I don't really want this rep. If it was money, I would donate it to save the whales or something.**

This has been mentioned here and there, but I'm making a concrete proposal that editors of Doc items should not be rewarded the same amount in perpetuity for every upvote. If this is collaboration, where we share the work, let us also share the reward, not multiply it.

Specifically, the value of one's contribution to an Example should diminish proportionate to one of

  • The amount of the current version which you are actually responsible for (Ideal, hard)
    • Possibly according to a blame line count (Straightforward, slightly game-able)
  • The number of editors who have come after you (Easier, still somewhat game-able)
  • The amount of time that has passed since your edit (Simplest but least appealing)

Yes, all of these are subject to gaming. But everything is; any of them would be better than the current structure.

I might suggest further that the editors of items be ranked by one of these criteria, and only the top N get rep, in fixed amounts: 12, 10, 8, 6, 5, 4, 3,... (or whatever). Another good option would be simple proportional division of some fixed amount per vote.


*And counting... I swear the achievements box lights up every third time I navigate to a new page.

**Yes, I will use it to post a bounty. (But that also doesn't take it out of the rep-conomy.)

34
  • 8
    I know the feeling but as of writing this comment I have no idea how "sharing the rewards" would work in the practice that doesn't introduce problems. I prefer docs to be an pure and simple altruistic work.
    – Braiam
    Commented Jul 25, 2016 at 18:57
  • 63
    If you take the time to go through garbage and make recommendations then approve edits based on recommendations you get zilch yet those that created the garbage will reap benefits when someone else makes something real out of what they started. None of it makes sense
    – charlietfl
    Commented Jul 25, 2016 at 18:58
  • 42
    Fractional reputation? Let's do this properly and let reputation be a complex number, with Q&A for the real part and documentation for imaginary. With the new silver "imaginary friends" badge for giving more up-votes on docs than on Q&A and hey come back, I have more…
    – null
    Commented Jul 25, 2016 at 19:17
  • 11
    I've already got my 3-d glasses on, ready to view the new reputation graph, @null!
    – jscs
    Commented Jul 25, 2016 at 19:25
  • 47
    As someone who has answered 62 questions and asked a handful, this seems ridiculous. All that effort in those answers and questions brought me to around 1,000 rep. I'm not a power user, but I try to help out when I can. Getting +350 rep for that makes me not want to answer questions. Or ask them. This is not griefing. This is the system being a confused deputy, dispensing illegitimate reputation where none is due. And just like any other exploit in a game (because that's what this is) people will continue to make use of it until it's fixed.
    – Athena
    Commented Jul 25, 2016 at 19:26
  • 11
    But this also brings up the fact that it's early on. The same way all those users from 2008 who asked/answered really basic, "classic" questions that have thousands of upvotes... that's gonna happen again here. It's just the nature of a competitive market; the early adopters harvest all the low-hanging fruit.
    – TylerH
    Commented Jul 25, 2016 at 19:28
  • 3
    The low-hanging fruit from Q&A didn't have the Gemino curse cast on it, @TylerH. Loaves and fishes are one thing; I don't know why bananas need to clone themselves too.
    – jscs
    Commented Jul 25, 2016 at 19:33
  • 4
    @TylerH but those answers at least had to be correct. Some of this low lying fruit has been atrocious...but can be fixed by others
    – charlietfl
    Commented Jul 25, 2016 at 19:35
  • 16
    @it's more like every adopter harvested every fruit, even the high hanging ones, because you only have to fix some spelling in an example that you don't necessarily have to understand yourself. The problem is: as of now, Homer Simpson could gain a lot reputation in the nuclear tag, if there was one.
    – null
    Commented Jul 25, 2016 at 19:37
  • 21
    My opinion is it would be far more beneficial to let documentation build organically with no rep involved as a side-effect of Q/A . I can think of lots of things I do repeatedly during answers that could be spun into documentation examples and save time on similar answers in the future
    – charlietfl
    Commented Jul 25, 2016 at 19:47
  • 9
    @Ares: "Give rep when users mention it or edit it into questions." Give rep to whom? Which of the people who edited the docs are worthy of rep for the citation? That's the problem this question (and this other one) brings up: a few small edits is enough to earn you the same rep that someone who did a major change gets. Commented Jul 26, 2016 at 1:28
  • 6
    woow!, they just killed the rep system (or what it was left of it), won't be surprised if the engagement numbers of SODocs are high Commented Jul 26, 2016 at 2:54
  • 14
    @JoshCaswell: I agree with your points but you've at least done a reasonable edit. I know a user who did an edit which would be worth at most 2 points on the Q & A site but has 725+ rep points to date just for that one edit. Imagine these "newbies" starting to review, edit without review etc :( The whole rep system in Docs is wrong imo.
    – Harry
    Commented Jul 26, 2016 at 10:25
  • 15
    I made one edit to a topic in the docs to see how it works, and now someone else has since gone and edited that part out completely. So as it stands I'm down to get reputation for eternity for literally having nothing in the documentation. People have been asking for years for a "SO for newbies" and it seems like that is what is currently being created. The only way I can see that stopping is if reputation is removed completely from docs.
    – Sayse
    Commented Jul 26, 2016 at 14:25
  • 7
    I performed a very simple test- I moved a segment that should have been its own example into its own example. Took about 30 seconds. +70 rep in a few hours. ?!?!!?! I got that much rep for my very best answer to a very difficult question. What gives?
    – Athena
    Commented Jul 27, 2016 at 15:10

3 Answers 3

50

Looking at this Doc example:

https://stackoverflow.com/documentation/java/99/arrays/404/creating-and-initializing-arrays#t=201607270051539205912:

125 edits so far, mostly distinct editors, roughly 100. Now an upvote is worth 100 x 5 = 500 rep points, while an upvote on SO is worth 10 rep points. And the gap keeps growing. There have been imperfections in the rep system, but this error is so huge and obvious, it's an insult to the intelligence and a mockery of your trusted users.

Reputation matters and you know it. I feed my little daughter with the money I make from contracts with people hiring me after they see my work on SO. They find me not least because of my hard earned reputation. I am not even advertising any contact details. They find me anyway.

I am puzzled you would resort to such unfair tricks just to push your new product. I used to be one of your greatest fans and most prolific unpaid contributors. For years now. If you keep this up much longer, both will end. You have certainly tainted any sympathy I might have had for your new product.

I have said something similar in kinder words before:

7
  • 1
    "100 x 5 = 500 rep points, while an upvote on SO is worth 10 rep points": On SO, a single user is responsible for the content you're upvoting. On docs, multiple users are responsible. To turn this around, why would an (possibly significant) edit on docs be worth less rep than an answer on SO? Why should the fact that 49 other users edited an example mean that the first user gets less rep for what he wrote?
    – Cerbrus
    Commented Jul 27, 2016 at 6:31
  • 24
    @Cerbrus: If my example needed another 100 people to improve it I would raise the question why this should be rewarded any reputation. Or to turn this around once more: A completely broken rep system seems to be the actual motivation for a 100 people to edit the example within a few days. Not the urge to make it better. I fail to see how you can even argue for dealing out 500 rep points for a single upvote. It's insane. And it devalues the work on SO (docs is currently not SO for me, starting with the completely incompatible rep system). Commented Jul 27, 2016 at 13:53
  • 8
    if at least the docs rep were kept totally separate from the SO rep, then it wouldn't be so bothersome.
    – Will Ness
    Commented Jul 28, 2016 at 7:42
  • 4
    @WillNess: A separate rep system can have it's own dynamic. Like a separate currency. That would solve the main problem for me. (I still don't think the system is smart, but the damage would be confined.) Commented Jul 28, 2016 at 12:47
  • The value of a vote isn't the total rep it results in. The value of a vote is the rep it gives per user. The total amount of rep a single vote results in is completely irrelevant. I can agree that it's too easy to get rep at this moment, but the problem is not the total rep a single vote gives. It doesn't make a "vote" more "valuable". The problem is more that insignificant edits get rewarded just as much as the person that wrote 95% of the example / topic.
    – Cerbrus
    Commented Jul 29, 2016 at 22:04
  • 1
    @Cerbrus: The value of a vote is exactly the total rep it results in. We have a maximum of 40 votes per day on SO to keep possible abuse in this area at bay, which limits a single user to 400 rep points from votes per day. The broken system on documentation can generate more than that with a single vote, which drastically devalues SO reputation. Of course, this is not the only problem. There are others like you mentioned. But this one is at the cost of innocent bystanders, which makes it particularly objectionable. Commented Jul 30, 2016 at 0:35
  • I have linked to your answer from my feature request to remove rep rewards for upvotes. meta.stackoverflow.com/questions/330171/…
    – TerraPass
    Commented Jul 30, 2016 at 14:28
17

To start, your edit really was helpful. But it does seem like that topic (and a handful of others) seem unbalanced. We're still in the early stages of analysis of the reputation system, but I think we know enough to address this particular edge case.

When designing reputation, we adopted the principle that reputation must be easy to understand. +5 for a question upvote and +10 for an answer upvote are really straightforward. (On the surface and for the common case, at least. There are many provisions and provisos lurking beneath.) +5 for an example upvote is equally easy to explain. (Though the shifting ground of what counts as a "significant event" is not.) If at all possible, let's keep the system that simple for the majority of cases.

The example you edited is hardly typical:

Score examples contributors (avg) 
----- -------- ------------------
    0     7777                  1  
    1     3038                  1
    2     1437                  1    
    3      655                  2     
    4      363                  2     
    5      239                  3     
    6      173                  3    
    7       95                  3      
    8       75                  4      
    9       61                  3      
   10       45                  3      
      . . . 
  106        1                 30
      . . . 
  217        1                 59       
  267        1                 44       
  306        1                 52       
  494        1                112       

(At the time of writing, the List Comprehensions example you edited a score of 106. Since it's the only example with that score, the average number of contributors (30) equals the actual number.)

As with Q&A posts, most examples have a score of 0 and there's a really long tail of examples with larger scores. That tail will extend further as more examples are created and more people vote. However, unlike Q&A, the examples with the highest scores also have the most editors and those editors potentially earn reputation. It's a clear balance problem. To be clear, it's not quite so big as that table makes it appear because contributors don't earn votes from before their edit. But it does suggest a degenerate strategy of contributing edits to the top-scored examples rather than focusing where you are most interested.

Other than the broken motivation, there's also the problem that the 30th edit to an example is likely to be less useful than the first. My guess is that, like usability testers, after 10 or so editors the major problems with an example have been fixed. Certainly, it's likely having a greater variety of examples will be more helpful than continuing to polish an already reviewed example.

My suggestion:

  1. Once 5 people have contributed to an example, a 200 reputation per user cap for upvotes is established.

  2. The cap is not retroactively applied. If the first editor has already earned 300 reputation, they won't lose 100 but won't earn any more either. The 5th and subsequent editors can only ever earn 200 reputation from upvotes on that example.

(Obviously, those numbers are from my gut. I haven't actually looked at how this system would play out in the data.)

The main advantage is that this only applies to a minority of examples. Also, it's still possible to write a (near) perfect example that gets universal acclaim (AKA, hit a homerun) and earn reputation from that accomplishment. There's still an incentive to fix problems on popular topics and it won't put people in the position of rep-capping regularly from a minor change. On the downside, it's possible that people will purposely use the cap as a rep-denial mechanism when there are already 4 contributors.

I could also see a hard cap at 10 contributors (or somesuch). Once that cap is reached, upvotes have no effect on contributor reputation. I'm not happy suggesting that, but it does seem like something should be done to encourage more examples rather than (potentially) excessive editing. let's prefer to start with the more generous cap first.

11
  • 6
    "The cap is not retroactively applied. If the first editor has already earned 300 reputation, they won't lose 100 but won't earn any more either. " That has the potential of making very valueable contributors very angry. Imagine for a second, you write a very good (near perfect) example, and while it gets into the upvote storm on the first day, 5 people decide to edit minor things in it to get in on that. Suddenly, you wont get anymore then 200 for your perfect example, despite deserving more. Fair much? No. And people will get super pissed about that in these cases.
    – Magisch
    Commented Jul 27, 2016 at 22:24
  • 2
    "I could also see a hard cap at 10 contributors (or somesuch). Once that cap is reached, upvotes have no effect on contributor reputation" This would effectively mean that once 10 contributors are reached, users are heavily disincentivised compared to new examples to edit these. Not sure if you would want that. I can see users voting to delete old and heavily contributed to examples in favor of new ones because the 10 editor cap has been reached.
    – Magisch
    Commented Jul 27, 2016 at 22:29
  • 5
    Also, many people here bring up that they're not comfortable with people repcapping from a one-time action. Yet thats exactly what is possible and regularly happens for high rep users on Q/A
    – Magisch
    Commented Jul 27, 2016 at 22:31
  • @Magisch What if the cap did not apply at all to the first five editors? Changes the problem to a fastest-gun variety kinda like we have in Q&A, but at least then there is no rep denial to the original contributors. There's also still a little incentive to keep it up to date from new editors, as they can earn up to the cap, but not as wildly as now.
    – Troyen
    Commented Jul 27, 2016 at 23:03
  • 11
    I don't know what's wrong with the simple "you can't get more than X rep from a single example" rule? It's certainly easy to understand and it makes sure that people get rep for contributing, no matter how small, but they don't get infinity rep. Thus encouraging them to contribute more in different places. Commented Jul 28, 2016 at 0:09
  • 1
    That distribution is very interesting. I think it might come from the fact that when you browse the documentation, topics are sorted by votes by default. That's going to lead to an obvious "rich get richer" long-tail scenario. They should be sorted either by some curated logical ordering (not sure how that would work), or by some hybrid mix of popularity/recency like how reddit works.
    – samgak
    Commented Jul 28, 2016 at 5:13
  • @Magisch Remember - we had this same theoretical problem with Q&A in the early beta because of automatic conversion to community wiki. A small group of people earned a ton of rep very quickly in the first few months leading to public beta, and even more after that. While it did cause a few grumbles, folks were pretty okay with it being necessary for balance. I'm not saying Jon's (gut) numbers are the best case for balance, but where he's headed does seem pretty spot on to me.
    – user50049
    Commented Jul 28, 2016 at 6:49
  • @TimPost Remember also that you later got rid of that conversion because it was unfair and misusing the concept.
    – Magisch
    Commented Jul 28, 2016 at 7:00
  • @Magisch Yup. And we've gotten a lot better at being more willing to ditch-and-rework stuff based on that experience, it was kinda nuts how many conversations needed to take place before we lifted it. When you're doing balance, though - even though you can forecast pretty well, it just takes time to see how the use of something changes based on your tweaks, and then quite a few iterative adjustments. tl;dr; Jon has a really good start to that here, and it excites me because we've gotten pretty great at not getting in our own way.
    – user50049
    Commented Jul 28, 2016 at 7:21
  • It's good to hear that this steady drip-drip-drip of +5s is not quite common. Thanks for posting numbers!
    – jscs
    Commented Jul 28, 2016 at 16:53
  • @JoshCaswell: I've hit the rep cap 5 times in the last 7 days. However, I do notice the rep coming in a little slower now, than a few days ago. I have a feeling that that was just the explosive start of docs.
    – Cerbrus
    Commented Jul 29, 2016 at 22:39
5

There are simpler rules that will fight these types of gold rush. For example:

Don't give reputation from any upvotes to the example when that example reached certain upvotes count (e.g. 10 or 20).

This will encourage early edits (usually coming with good improvements) and discourage edits "for a rep boost" of the basic examples which get regular upvotes.

Alternate also simple approach suggested by @Nicol Bolas:

You cannot gain more than X rep from any single example

3
  • 20
    What if the example is at +10, but I make the kind of amazing edit that causes tears to stream down your face in pure joy, and that edit brings the example up to +500. Should I not get rep for that?
    – Barry
    Commented Jul 26, 2016 at 14:14
  • 3
    Rep isn't money in the end. If you know such awesome things and you are concerned about getting a clear recognition in some form then probably blogging or writing a book on topic or an article somewhere like SitePoint is better for you. Documentation is more or less "authorless" like Wikipedia. Anyways, the changes history is public - it will not be a big problem for people to say "Thank you", I believe, as well as for you to reference it in your profile.
    – Ruslan Bes
    Commented Jul 26, 2016 at 14:40
  • 13
    I think a much simpler rule is this: you cannot gain more than 10 rep from any one example. Commented Jul 26, 2016 at 14:50

Not the answer you're looking for? Browse other questions tagged .