42

So, I have a concern regarding reviews and I'm just probing the community for feedback.

The idea is that with enough eyeballs on a review, the community will arrive at a reasonable consensus. Now, that only works out if reviewers will reliably:

  1. Recognize immediate, obvious problems like spelling, formatting, vandalism, i.e.: normal stuff
  2. Judge the edit on its technical merit and compatibility with the original material
  3. Refrain from voting when #1 passes and they don't have the expertise to perform #2

I don't know that #3 is happening enough. It seems like the community is okay voting on content they don't understand. I don't want to see quality edits rejected because they "changed too much" when the reviewer didn't have the domain knowledge to judge whether the edit was an improvement.

Do you guys believe this is a real problem? I can cite examples if necessary but I'm more concerned with your current opinion, given your personal experience up to this point.

33
  • 43
    This is exactly why edits are there to improve the presentation of the author's content, and not to add or change the underlying content of the post. Such changes generally shouldn't require domain expertise. If you're trying to use edits to make factual changes to a post, you're using the wrong site feature.
    – Servy
    Commented Dec 6, 2016 at 19:24
  • 30
    Your proposal of encouraging edits to make factual changes, but asking reviewers to simply not judge edits that make factual changes they don't feel qualified to judge, will result in (if it were actually successful) the queue simply filling up with really niche topics that none of the reviewers are qualified to judge, effectively removing the suggested edit feature entirely. The system only works when domain expertise isn't required.
    – Servy
    Commented Dec 6, 2016 at 19:25
  • 7
    If you don't have domain knowledge of the subject you shouldn't be reviewing something that requires said knowledge. Saying these edits shouldn't happen and reality are two very different things.
    – Kevin B
    Commented Dec 6, 2016 at 20:27
  • 2
    Ideally these edits should be done by users who can bypass this queue to avoid the problem entirely, but how do we deal with cases where something does need to be improved, but the person who notices it doesnt have said rep? do they instead ask someone else to do it, or do they put it into the review queue. neither option seems ideal, though asking the answerer or asker to do it is likely the lesser of two evils. At least then maybe a passerby with the rep could do it too.
    – Kevin B
    Commented Dec 6, 2016 at 20:35
  • 5
    @KevinB If you don't have domain knowledge of the subject you shouldn't be reviewing something that requires said knowledge. But that's just it; suggested edits were specifically set up to not require domain knowledge to evaluate.
    – Servy
    Commented Dec 6, 2016 at 20:49
  • 8
    @Servy and yet, here we are.
    – Kevin B
    Commented Dec 6, 2016 at 20:49
  • 2
    Are you suggesting it's working as intended, and all edits that aren't fixing formatting/grammatical problems should be rejected? i mean, that's a valid option, it isn't too damaging when it comes to suggested edits to answers/questions. Certainly wouldn't work for documentation, but that's another topic.
    – Kevin B
    Commented Dec 6, 2016 at 20:55
  • 4
    @KevinB Yes, with a largely functional editing and reviewing system, that works on the premise that anyone can review just about any edit, and so edits are able to be reviewed promptly, regardless of domain. It simply means that suggested edits cannot be used to change the content of another user's post; if you want to add your own original ideas to a question, you need to do so with your own answer, an entirely workable solution to the problem.
    – Servy
    Commented Dec 6, 2016 at 20:56
  • 4
    @KevinB well, that line of thought goes against SE own guidance with edits: If you see something that needs improvement, click edit! For me, content curation is critically important for SO right now, and its importance will just continue to rise over the time.
    – Braiam
    Commented Dec 6, 2016 at 21:10
  • 27
    @KevinB documentation terrifies me. I've actively avoided it.
    – canon
    Commented Dec 6, 2016 at 21:14
  • 5
    See Should people who've never asked or answered a question fo C be allowed to review C documentation changes? for a similar problem, and that has a link to the problem with wiki edits, …. There's an element of "it's an endemic problem around here". Granted, not all edits and reviews always need domain expertise, but I think there are (lots of) times when people don't skip when they should. Commented Dec 7, 2016 at 6:39
  • 2
    This problem, is real, but IMO applies more to close votes. People often vote to close a question because they don't have the expertise to understand what the questioner is trying to ask, when in fact the question could be easily improved. The questions never get opened again. Commented Dec 7, 2016 at 8:24
  • 11
    @SList No, they don't "often" do this. In fact, in the tags I follow, they rarely do this. Rather, this is the assumption that the asker often erroneously makes, out of frustration that their question got closed and the reality that the person asking the question thinks that it is sufficiently clear because it is clear in their head. If the question being asked is insufficiently clear or obvious that 5 reasonably experienced answerers think it should be closed, then it absolutely should be closed until it can be improved. You don't wait for the improvement to happen before voting to close.
    – Cody Gray Mod
    Commented Dec 7, 2016 at 8:31
  • 2
    @CodyGray "You don't wait for the improvement to happen before voting to close." That's maybe a bit of a problem. I wait with my close vote if I have the feeling the question has the certain something that gives me confidence it might be salvaged (difficult to define better). As for "are close decisions always right" - well if they would we wouldn't have reopen votes. My experience is that there is a bit of herd instinct going on with close votes but not very strong. Basically some people trust that the guys before them got it right. It's not a big issue though. Commented Dec 7, 2016 at 9:25
  • 8
    @tril Strong community consensus that you should not wait to close (see, e.g., 1, 2, 3). Obviously not everyone follows this rule, but it creates problems when you don't, like questions that could be improved by editing not getting improved by editing and then not being closed. If you think the question is salvageable and don't want to close, then you should salvage it. Also, I didn't mean to imply "always right"; rather, that closure isn't for lack of expertise.
    – Cody Gray Mod
    Commented Dec 7, 2016 at 9:28

5 Answers 5

37

It takes no domain expertise to edit for grammar or spelling. I won't deny that it's helpful to have the domain knowledge, but it absolutely isn't required.

You have to understand the difference between editing for presentation and editing to improve its factual accuracy. The vast majority of editors are equipped enough to tackle the former, but not necessarily equipped to tackle the latter. The key misunderstanding with many is that they believe that these two edit types are the same, which isn't the case at all. In fact, some editors may attempt to conflate the two edits together, which leads to someone rejecting it for changing too much.

My genuine belief is thus:

  • If a user wants to make grammar changes, then they should exclusively make grammar changes.
  • If a user wants to make changes to improve the accuracy of the post, then there's two cases to consider:
    • Do they have knowledge of the OP's situation, gleaned from comments? In that case, it's fine; just include, "The OP said this in comments" or something analogous in your edit summary.
    • Do they only have domain knowledge of the problem and believe it to be an issue with how the problem is presented? Do not edit! Answer instead! Editing the question in this scenario hides the fact that the OP didn't know any better, and could lead to answers which the OP has no means to apply to their problem.
30
  • 8
    @canon You shouldn't be editing something like that into someone else's answer. If you want to add your own original explanation to an answer, post your own answer.
    – Servy
    Commented Dec 6, 2016 at 20:09
  • 2
    @canon: A new answer is still preferable in that situation. If the answerer is wrong, then vote and comment accordingly.
    – Makoto
    Commented Dec 6, 2016 at 20:11
  • @canon: If you're talking about an answer being plagiarized, then it should be deleted. Using a custom flag on the answer would be best rather than a ham fisted attempt at an edit.
    – Makoto
    Commented Dec 6, 2016 at 20:18
  • 3
    @canon Because posting your own answer is what you do when you have your own answer. You don't just edit your answer into someone else's answer. That's how answering works. Different people post different answers when they each feel that they have something to contribute.
    – Servy
    Commented Dec 6, 2016 at 20:19
  • 7
    there is a gray area, believe it or not, where an answer may have a typo or minor inaccuracy that you as an editor can fix. These very often get rejected by people who blindly reject anything that touches code. (fortunately i have the rep to bypass this queue)
    – Kevin B
    Commented Dec 6, 2016 at 20:22
  • 4
    @canon That's not what edits are for. Edits are there to improve the existing answer's content, not to add your own original content. To do that, post your own answer, even if it happens to also include a citation for another answer in it.
    – Servy
    Commented Dec 6, 2016 at 20:46
  • 25
    Whatever happened to the logic that, in an ideal world, a question would have a single, highly-upvoted, community-audited, well-maintained, canonical answer that would be easy to find? If every time someone wants to add something non-trivial, they must post a new answer, then we've become indistinguishable from a traditional web forum, which we all roundly agree sucks. And in practice, no one will actually do this. The best you can hope for is they'll post a comment to the existing mostly-good answer, which is also strongly discouraged. This answer selfishness is destroying the site.
    – Cody Gray Mod
    Commented Dec 7, 2016 at 8:34
  • 3
    @CodyGray That was never the premise of the site in the first place. You seem to be confusing the site with Wikipedia. The premise of the site is different people each post what they feel is the best answer, and the best answer rises to the top through voting, not through different people editing someone else's answer into what they think the best answer is. The primary way the site was designed to distinguish itself from more traditional forums was through the voting pushing the best answer(s) to the top and separating out answers from clarifying questions and other meta information.
    – Servy
    Commented Dec 7, 2016 at 14:19
  • 13
    A second answer should be posted if you have something substantially different to say. You know, a completely different answer. But if you just want to expand on an existing answer that is otherwise perfectly adequate, you don't fork it and put your own name on it. Your recollection of the purpose of the site is rather different than mine. Stack Overflow didn't innovate the idea of voting on stuff on the web. There are plenty of other online forums that had that even before 2008.
    – Cody Gray Mod
    Commented Dec 7, 2016 at 14:23
  • 4
    I think that both Cody Grays's argument that adding another answer for every content update is bad as well as Servy's argument that editing one arbitrary chosen answer with updates is bad are both justified. That leaves me with the conclusion that a trade-off is required somehow, but I fail to imagine what it could be. Commented Dec 7, 2016 at 14:43
  • 7
    @Servy "You should be posting your answer if you want to contribute your own original content." That's very nice and clear, but it also means that potentially the number of answers can be be very large with lots of redundancy between them because everytime somebody wants to add even a tiny bit of content he would have to post a new answer. I could easily spam the whole site by taking all the top voted answers and publish another answer combining them and the best of their comments. Probably half (or a significant amount) of the knowledge on SO Q&A is contained in the comments. Intended? Commented Dec 7, 2016 at 14:59
  • 1
    @Servy In general I agree, although for some questions there is a large number of answers and there is some redundancy between and there is also important information mostly contained in the comments. It's good as it is, no doubt. It may be better with a different approach, who knows. We may never find out. Doesn't make our system perfect. Commented Dec 7, 2016 at 15:06
  • 1
    @canon If someone has a code-only answer and you want it to be improved, comment on the answer requesting some more information, even if you personally know enough to write that information yourself. If the OP responds or adds more info via a comment, then you can edit that content into the question or answer (while being sure to mention in the edit description that you're just copying OP's own words from the comment to the post).
    – TylerH
    Commented Dec 7, 2016 at 20:44
  • 1
    @CodyGray re: one answer per question; one of the criteria for a beta site being healthy is that it has an average of 2.5 answers per question. It is baked into the whole engine that there should usually be more than one answer.
    – user4639281
    Commented Dec 8, 2016 at 20:36
  • 2
    @TinyGiant yes, 2.5 answer each with their own point of view. Not the same answer vomited, re-vomited and regurgitated a bit more answers (btw, since some years now, CM don't rely too much on quantitative metrics)
    – Braiam
    Commented Dec 9, 2016 at 0:36
16

This appears to refer to suggested edit reviews. I don't believe people doing reviews without technical expertise is much of a problem, for the following reasons:

Stack Overflow is not Wikipedia. A post should not be edited to radically change the technical content (with a few exceptions listed further below).

If a question is missing a lot of important details, then it should obviously be edited by the OP. There is no way anyone but the OP can fix it. Close vote such questions as off topic -> lack details to reproduce. In case the OP left a lot of code in comments, then it is fine to edit the question and include this, though make a comment to the edit reviewers "pasted from comments by OP" or similar.

If an answer is missing a lot of important details or is simply incorrect, it is not a good answer. It should not be "fixed", it should possibly be down-voted with a comment explaining why it is not a good answer. Then the radical changes needed to correctly answer the question should be posted as a new, separate answer.

The only time an answer should be edited to change the technical content is either when the OP of the answer made some simple mistakes that can be fixed without changing their intent or drastically change the answer. Typos, simple syntax errors, formatting etc.

Note that minor syntax errors should never get fixed in a question, as they may be the actual cause of the problem described!


Given the above, suggested edit reviews therefore rarely require domain knowledge. Most such edits are actually invalid and should be rejected. Apart from the following exceptions:

  • Minor technical edits to an answer as described above.
  • Edits that change tags of the question. This often requires review by a domain expert.
  • The post is a community wiki.
  • The post is a tag wiki.

I suppose tag wiki edit reviews by people without technical expertise might be a valid concern. But most often such reviews is only about ensuring that the text isn't some copy-pasta from wikipedia or other sources.


A much greater concern is the "robo-reviews", which is a persistent problem on the site, and has been discussed on plenty of other meta threads.

-3

I don't want to see quality edits rejected because they "changed too much" when the reviewer didn't have the domain knowledge to judge whether the edit was an improvement.

Users get access to the suggested edit review queue as part of the edit questions and answers privilege. You seem to be worried that people who don't have sufficient domain knowledge are judging other people's edits, but these very same people have reached a reputation level that lets them edit any question or answer in the system without having their own edits reviewed at all!

It seems to me that if the community trusts someone enough to give them unconstrained edit privileges, there's little reason to be concerned about their ability to judge the quality of less trusted users' edits. Furthermore, rejecting or accepting a suggested change requires consensus among several reviewers, all of whom have edit privileges, making it even less likely that a good edit will be rejected or a bad one will be accepted.

Do you guys believe this is a real problem?

No. You say that you can cite examples, and it would be surprising if there weren't some examples of edits being judged badly. But a handful of examples aren't sufficient to show that this is a problem that happens often enough to worry about. There are many thousands of posts and at least hundreds of suggested edits on any given day, and I think we'd see a lot more complaining in meta if good edits were frequently being rejected. There are also measures in place, such as test questions, that are designed to improve the quality of reviews.

If you have some data that shows that there's a substantial problem with the review process please post it. Then we can have an informed discussion about the size of the problem and what to do about it. Absent that data, your concern seems more like well-intentioned speculation.

1
-5

I don't know that #3 is happening enough.

I don't know either. Therefore, is the community as a whole really okay voting on content they don't understand? For example, couldn't it be also a side effect of robo-reviewing?

We first need to find out if this is a problem (not just believe in it or not believing in it). How to find that out?

Review the reviews

If that turns out to be a problem you have the choice of:

  • more controls (audits) with potentially harder sanctions
  • require more votes before a decision is made
  • more warning signs (example: Are you sure, you understand the topic?)
  • stronger limiting reviewers by rep
  • limiting reviewers by topic (you need to have at least XX badge in at least YY of the tags to be able to review)

As it is currently, it's the job of the audits to wed out any wrong review activities including this. Audits may not work good enough. Maybe they can be improved.

General remark: The better the content is already the higher the chance that an edit of a human which can err is lowering the quality. Therefore being more and more critical is probably desirable. On the other hand you still want to be able to update content and improve it, so there is a trade-off. In the end, a certain amount of newly introduced noise is unavoidable. You can improve the situation by doing one or more of the above mentioned actions.

Btw. my guess is that even the un-supervised edits for >2k rep users introduce new errors from time to time.

My personal experience: ~80% of edits improve the content and ~20% make it worse (numbers are just gut feeling)

5
  • 5
    You have made the assumption, along with the OP, that edits that are making changes to the actual underlying technical content of the answer are okay in the first place. They are not. Pretty much any edit that is changing the technical content of the answer merits rejection because that's not what edits are there for, so to assume that people are rejecting edits that change the author's consent giving a rejection reason that they're changing the author's content are reviewing correctly. It's people that are approving edits that change someone else's post that are the incorrect reviews.
    – Servy
    Commented Dec 7, 2016 at 14:17
  • @Servy I see. Interestingly the proposed actions (more audits, more required votes, more warnings) mostly stay the same. What drops is the requirement for the reviewers to actually know anything about the topic. Commented Dec 7, 2016 at 14:36
  • 1
    I don't see how more audits are going to help. Either the audits are catching people actually reviewing improperly, or they're not. More of them would mean it would take maybe 2 days instead of 3 for them to get banned, not out of the question, but if there really were a major underlying problem that wouldn't fix it. Same with more reviewers. If the common consensus was simply wrong, requiring more reviewers wouldn't change the outcome. There's also not much correlation between rep and reviewer quality past a certain point, so increasing it is unlikely to help a ton.
    – Servy
    Commented Dec 7, 2016 at 14:42
  • 1
    But again, before implementing any of those you'd need to demonstrate a problem that they'd be there to solve in the first place. As of yet, that hasn't been done here.
    – Servy
    Commented Dec 7, 2016 at 14:42
  • 1
    @Servy "If the common consensus was simply wrong, requiring more reviewers wouldn't change the outcome." That depends. If for example only a minority gets it wrong, you get more decision power the more votes you have. But I fully agree that we would have to first demonstrate the problem. Commented Dec 7, 2016 at 14:52
-13

There are no easy things in this life: working, studying, sports, programming, designing, etc. Therefore reviewing is, also, hard (or not easy). Even the most obvious problems require domain knowledge of something: the language (in this particular case, English), the markdown syntax, the concepts you are reading about, etc.

Considering this, reviewers are required (in theory) to know about whatever they are reviewing. Since they are reviewing, we expect that they are able to critically evaluate if the actions being taken conforms with guidance.

Expecting a reviewer that is basically a monkey with a keyboard mouse to give a sensible evaluation of what is being reviewed is as likely as the same said monkey writing something similar to Shakespeare.

I prefer having a sensible review by an expert, even if that means having the queue to grow infinitely. I doubt this will be a problem since there is at least one user, the very author, that is notified of any edits on their own post. If they aren't available in a sensible amount of time, we can simply invalidate the edit and hope for the best.

16
  • 1
    I agree, but, i think we may disagree on how we decide whether or not a reviewer has said knowledge (or if we even should.) I'm more included to say this should be a guidance, rather than a hard enforced rule via badges/rep/whatever
    – Kevin B
    Commented Dec 6, 2016 at 20:46
  • 9
    The queue has a max size of 200 items. When there are 200 items in it edits are not allowed to be made at all. Additionally, there are lots of posts out there where the author isn't actively monitoring the site, and either won't ever be able to review the edit, or won't be able to do so in anything resembling a reasonable amount of time.
    – Servy
    Commented Dec 6, 2016 at 20:47
  • @KevinB of course there's no perfect solution, I doubt we will reach it. But something sane enough that strikes balance between the ugly and the good should be workable.
    – Braiam
    Commented Dec 6, 2016 at 20:48
  • @Servy do you know that that limit is totally arbitrary and can be modified by SE (remember, only triage and suggested edit queue has any sort of limit)? If we can build a good case to allow the queue grow, SE is likely to comply, since they are worried about this. If we solve both problems, do you believe that SE won't be flexible about it?
    – Braiam
    Commented Dec 6, 2016 at 20:50
  • 1
    @Braiam Then you end up with many, many posts out there locked for editing, because there's a pending change that there aren't qualified reviewers for, and you end up with a review queue that becomes functionally unusable. When someone needs to skip through hundreds of posts to find a single edit that they can actually review (because there's a backlog of thousands of edits in niche topics) then people will either stop reviewing edits, or make decisions that they don't have the expertise for (I guarantee most will do the latter). You just create more and more problems.
    – Servy
    Commented Dec 6, 2016 at 20:54
  • 7
    @Braiam You haven't found a way to solve any of the problems, you've just suggested ignoring all of the problems.
    – Servy
    Commented Dec 6, 2016 at 20:55
  • 2
    @Servy Care to share your own solution instead? I'm saying what should be the ideal, or you could say say "if-this-was-Canada-and-not-real-life" solution. For me, the problem to solve is: how to allow sensible edits on the system. I don't know what problem you think I am solving?
    – Braiam
    Commented Dec 6, 2016 at 20:57
  • 8
    @Braiam The current implementation is far superior to your suggestion, which is functionally equivalent to just removing suggested edits entirely. Of course, if we want to talk about entirely unrealistic ideals, ideally no post would ever have problems meriting editing; they'd all just be perfect right from the start. Of course, discussing entirely impractical idealistic situations isn't really productive.
    – Servy
    Commented Dec 6, 2016 at 20:59
  • @canon ewww..... one of the reasons I invested time on SE is that "the content is clean, neat, and professional looking." You know how painful is consuming content that is not like that?
    – Braiam
    Commented Dec 6, 2016 at 20:59
  • 1
    @canon you think that stale or unexplained content is presented correctly? A book that can't be understood by its reader, nobody will want to read it.
    – Braiam
    Commented Dec 6, 2016 at 21:05
  • 3
    @Braiam: "ideally, there shouldn't be hunger in the world, it's impractical to reach that goal? Maybe. It's worth to pursue it? Absolutely." But you can't solve world hunger by denying people food. If you come up with a solution that makes the problem worse, that's less effective than doing nothing. Commented Dec 6, 2016 at 21:50
  • 12
    @Braiam If you think a post is a bad post, because it lack an explanation in entirety, then there are lots of tools for dealing with a bad post. You can downvote, you can comment, to explain what's wrong with it, and you can post an answer that's *not bad. The solution isn't to say that anytime you don't like an answer you should edit what you would have posted into that answer instead. If you want to post your own answer you can always post your own answer. If I think your answer here is a bad answer should I edit it to correct it and explain how edits should actually be reviewed?
    – Servy
    Commented Dec 6, 2016 at 22:10
  • 1
    @Servy yeah, right, that worked on this case? But lest go towards a more serious topic: what is to you collaborative editing?
    – Braiam
    Commented Dec 7, 2016 at 4:27
  • 3
    @canon Making edits that require domain expertise to judge is fine, but when the user has <2k those edits should be discussed with other users who have domain knowledge as well as the editing privilege. In a perfect world (which it isn't), the suggested edit queue would not just be a bandaid (which it is). With the tooling that we have in the edit queue, it is just not possible to ensure that domain experts will ever see the edits that require their expertise to review. If you want to make a request to change the tooling to support the reviewing you would like, I wish you luck.
    – user4639281
    Commented Dec 7, 2016 at 7:53
  • 1
    Let alone if, as mentioned before, reviewers have to wade through 30+ technical edits to posts in niche technologies in order to find an edit that they can review, they will probably stop reviewing. So you end up with more edits in the queue, staying there for longer, while discouraging reviewers, causing less review items to be reviewed, resulting in an ever growing queue that becomes harder and harder to manage.
    – user4639281
    Commented Dec 7, 2016 at 7:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .