188

Reviewing (together with interlinking existing content) is one of the key tasks for the SO community now that the site is past its initial growth stage. And the infamous overload of review queues and other community feedback shows that the current mechanism for that task is not efficient enough.

There is a number of pending feature requests to increase its efficiency including one of mine.

By giving this area low priority (if any at all, there was no feedback from the team to make any judgement calls), I feel that the SE team does not value my volunteer time spent reviewing, and that time is not being spent effectively.

As such, I've decided (a few months ago) that it's time to stop donating it until reviewing efficiency is improved to once again be worth my time.
(For myself personally, I set the bar on implementing Increase close and reopen vote weight for tag badge holders and Close and Triage queues need a downvote option. No, really since these are the only crystallized breakthough ideas in this area atm AFAICS.)

49
  • 12
    related: Close Votes review: I'm going on a strike! (six years ago this way worked, hope it could help this time as well) cc @Persijn
    – gnat
    May 15, 2019 at 10:10
  • 13
    @Persijn Since words clearly didn't seem to have any effect, I feel it's time to take an affirmative action to make a statement. May 15, 2019 at 10:10
  • 82
    That said, I'm pretty sure many people already did this out of frustration, just didn't make this fact known. May 15, 2019 at 10:11
  • 28
    The frustration is understandable, but the act of protest will not have meaningful consequences. You aren’t donating your time reviewing to the team that runs Stack Overflow. You’re donating it to the community of people who use Stack Overflow, to ask and answer questions. The people who started using it and want to keep using it because it avoids their having to waste time slogging through piles of crap. Refusing to review will just mean more crap for you, and I, and everyone else. It won’t make the review queues get fixed. Fewer reviewers means either more work for fewer, or more crap. May 16, 2019 at 5:31
  • 76
    @CodyGray "More crap" is also negative marketing and degrading business value for the company since the knowledge base and the community is their primary asset that drives their entire business. So if enough people support me, they will be forced to do something about it. The company has become complacent and started neglecting the force that drives their success. This is going to become their undoing if kept unchecked for too long and I honestly hope it doesn't come to that. So I'm actually helping them by pushing them to realize and fix the problem sooner rather than later. May 16, 2019 at 5:53
  • 11
    @CodyGray what you are describing as just more incentive for others to go on strike and/or for conditions to be improved. That is, assuming the SO staff is interested in having less crap.
    – VLAZ
    May 16, 2019 at 7:21
  • 13
    @CodyGray I've been following CV review discussions very closely in the times of the first strike and per my observations noticeable improvements started occuring only after it. Based on that past experience your claim that it won't help sounds a bit premature.
    – gnat
    May 16, 2019 at 8:25
  • 58
    @CodyGray I've stopped formally reviewing a few years ago, out of frustration. Fewer reviews does mean more crap, overall. However, as an individual I am certainly not exposed to crap as much as I was when I was formally reviewing. And I avoid tons of frustration: the immediate frustration (for instance) of Qs not getting closed fast enough, and the long-term frustration from dealing with management unwilling to provide experienced SO users with better moderation tools. I still vote or flag when I happen to run into cases that warrant it, but I just don't go out of my way like I used to.
    – Louis
    May 16, 2019 at 13:21
  • 57
    I gave up doing reviews years ago. Lots of effort, awful UI, bad audits, Sisyphean levels of work and most of all, almost no reward.
    – DavidG
    May 17, 2019 at 14:10
  • 21
    I also can relate. As soon as I got the reputation for it, I tried working on some of the review queues. I gave up in a couple of days, and never tried it again ever since. It's a lot of work for basically no reward. Let's be honest: Unless you intend on running for moderator, those badges aren't really worth the effort. May 17, 2019 at 14:33
  • 21
    People trying to close this as "does not appear to seek input and discussion" highlights the problem, seeking discussion about this topic (for years now) has been nearly completely fruitless. May 17, 2019 at 14:48
  • 16
    @TylerH This post is a call to action. It's inherently actionable... both by the community, in either strike participation or discussion, or by SE staff. Regardless, I rather welcome an attempt to strong-arm SE rather than slinking away, quietly defeated, as have countless others before us. It's been demonstrated on more than one occasion how little power we truly have here. Our participation is really our only card to play. A coordinated strike might actually raise the needle a bit...
    – canon
    May 17, 2019 at 19:47
  • 17
    @TylerH The problem is the way things are supposed to be done here have proven to be futile. Sure, Ivan can come on and say "hey SO, this is a problem, and it needs to be fixed" - but then, it gets closed as a duplicate, because there has been years of discussion revolving around this, and no action, even with a ton of community support. "One (or a few) people threatening to ?stop reviewing? is not ammunition" - That's the purpose of this post. It's meant to draw in more people to participate in the strike. Frankly, the "correct" way doesn't work, it's time to push the envelope. May 17, 2019 at 20:00
  • 9
    @TylerH Are you seriously saying that adding "who's with me?" to the end of the question completely changes this question to something you wouldn't close? Ivan is saying he's going on strike, this is obviously to garner support of others to join him, even without explicitly saying so. May 17, 2019 at 20:18
  • 26
    "Refusing to review will just mean more crap for you ...", logical fallacy. It is ZERO crap for me, because not participating in the site much less the queues anymore I do not see any of the crap I used to see when I had 40K. Therein lies the rub, those that complain that too much gets closed to fast (usually their own questions), rarely if ever, actually have the rep to see the torrent of crap posted to the site every hour, therefore it does not exist to them and is not a problem. The only problem they see is their crap getting closed or down voted and feeling unwelcome because of it.
    – user10677470
    May 20, 2019 at 2:05

9 Answers 9

76

Update: SE is looking at feature requests now! It's hopefully just a matter of time before review/moderation-related ones start being worked on.

Background

On MSE, there are ~700 unimplemented, review-related feature requests. In general, there's over 13k feature requests without any [status-*] tags on MSE. On SO, there's nearly 4000 overall, with roughly 200 aimed at review. Those are just using the generic tags.

The main problem, as far as I see, is that there is a ton of stuff that needs to be moderated, but way too few votes and active reviewers to do so. There's a lot of incorrect reviews, and not to forget various "fine prints" on editing and the common plagiarism pitfall with tag wikis that cause additional work and reviewing reviewers.

Duplicate CVs are specifically tricky because of the domain expertise some of them require. Flags aging away is also a problem, but one that is specifically visible to users who flag. Close votes expiring isn't as visible.

The problem is better demonstrated in this post - the queue size fluctuates, but never goes specifically low.

The current situation

The burn of the close vote review queue shows we have the technical amount of people required to actually pull it off again. But what one of the answers unfortunately shows is a drop in reviewers after a review spike. At its peak, there were 12000-14000 reviews per day. So far today, there has been 658. To handle the 8152 pending questions in the queue right now, over 40000 reviews are required, or roughly 800 reviewers with all 50 votes.

There is, however, one problem with this calculation: the number isn't real. SE has an algorithm (that I don't know the exact details of, and therefore can't reverse) which reduces the number of questions that appear to be in the queue.. The real number of reviews required to keep the queue down is likely exponentially higher, further increasing the need for review enhancements.

However, the queue isn't sitting there static, and the site isn't blocking users from asking new questions. If a full-scale strike is put into action, the amount of posts in review will spike significantly.

The problem

We don't have enough reviews.

And this is a three-part problem:

  • The reviewers
  • The review system
  • The posters

This question asks for a strike in light of the review system. However, if we don't have reviewers, we still won't get enough reviews. If we lower the close votes required to close a question to three, the amount of reviews drops to 24000, or nearly cut in half.

Which brings us to the last problem: The posters. This is being worked on through things such as the wizard, but this only covers questions. Answers are more complicated, but bots such as Natty are helping flaggers with that. The major difference between VLQ review and close vote review queue is 1000 reputation.

Talking in user count, 0.9% of users have access to VLQ, but only 0.65% have access to CV review. (source). How many of those are active though? That's a very different question, and one I can't answer accurately through SEDE. But given the size differences, even though there are volume differences in questions and answers, the answer review queues are very close to empty.

Based on today's (17.05.19 at 19:55, +0 GMT) 658 reviews, and for the sake of math and missing information I will have to assume all the reviewers have burned through their reviews. Doing so gives 658 / 40, which means 16 reviewers, and one who used about 20 reviews would be able to keep the exact count listed earlier. And, again, there were near 800 during the CV queue burn.

At the same time, the queues are filling up, and it will get worse when there are fewer close voters. The strike is barely starting up, so to what extent it affects the review queues is yet to be seen.

However, I do feel I need to give a shoutout to the moderators, and especially the one I've seen most of, Bhargav Rao, who've made a noticeable impact on the VLQ review (and indirectly on other answer-related review queues) by handling NAA flags. Without the moderators, the VLQ queue would likely contain more posts than currently does.

And if they, as Yvette mentioned, are trying to tackle the close vote queue as well, that's on one hand good news (because of binding votes - which means more effective reviews), but on the other hand, additional proof the CV queue has fundamental design flaws when it comes to efficiency. While it's most noticeable in the CV queue, it wouldn't surprise me if this affects other review queues some day, or with the strike, soon.

TL;DR: We don't have enough reviews per day.

Secondary problem

Aside the lack of reviews compared to what's required for an efficient system, there's one pretty big problem affecting the queues.

As listed in the start, there are ~900 unimplemented review feature requests on MSO and MSE combined (possibly with some overlap). These not being implemented likely contributed to this.

I've talked to some people who have stopped reviewing because of missing features, or otherwise have issues with the review system - and in my opinion, with good reason.

Statistics

Update: These numbers are wrong. Without access to the close votes themselves, I wrote the query under the (wrong) assumption posts closed outside the queue completed the review task. According to Shog, the number is closer to 42%, which I still think is too high. On the bright side, it wasn't as bad as I thought.

Update: These stats cannot be corrected without Community Managers or other people with database access. Accurate statistics will eventually be available here

Update: The accurate numbers are between 10-20%. See this answer.

The original state for historical purposes:


Note: I am not sure how the fuzzied numbers affect SEDE

I get that it's hard to see the problem. I've created a SEDE query (fork-ish off this query) to better demonstrate this (thanks to rene for helping me with sanity-checking the query):

Graph showing review tasks over time

The question and answer post stats is just to demonstrate the volume. On a weekly basis, there's (on average) close to 50000-60000 of each post type posted. Every week, there's about 40000 reviews done, of which, only 8000 are on close vote review.

5k are on answers (Very Low Quality and Late Answer. First Posts is excluded because it also includes questions).

Now, this is where it gets tricky. Each post has one ReviewTask for each time it enters the queue. Which means the 8000 posts on CV Review Tasks In Week are all posts pushed into the queue for that given week. The line right under, CV Review Tasks Invalidated in Week, shows how many of the review tasks created that week that have been invalidated (which, according to both observation of an invalidated close vote review, and rene, includes expired close votes and flags). And the last line, CV Review Tasks Completed in Week, shows how many of the tasks created that week that have been completed.

Note that one ReviewTask does not signify how many user reviews were made on that task - it only shows how many, in this case, questions were in the queue with a creation date for a given week. Each of those can have any (sensible) number of reviews. To find out how many user reviews have been wasted, that requires a different SEDE query that I don't have available at the moment.

The vast majority of questions in the CV review queue aren't handled. The completed review items are those that have either been left open or have been closed in CV review. On some of the worst weeks, ~94% of CV reviews were made for nothing. I have no statistics on how many flags and close votes were essentially lost on this, but it's probably significantly more than one per review task.

The past few weeks show fewer invalidations, among other things because they haven't gotten old enough to expire, but also because the amount of reviews has spiked since about mid-April. This might be because of more reviewers, but it could also be because more questions now are worth closing than earlier (this is at least something I've observed on some tags).

Either way, those thousands of expiring close votes is a huge problem because they may contain questions that should be closed. And they can also contain the questions we actually want on the site, but that were handled poorly/incorrectly.

For those of you who aren't familiar with SEDE, completion type 3 on reviews (named "invalidated") means natural removal from the queue, and includes expired close votes/flags. This is verified in this SEDE query, as outlined in the SEDE structure post on MSE.

Note that this is an overview of what happens to review items by the review item creation date, and not necessarily the date it was handled. Because of expiration, it's fair to assume most of them were handled (or expired) relatively quickly

Personally observed situations

I don't review in the CV queue at all. I haven't for a long time because of one issue: 50 close votes doesn't let me cover the tags I watch, and review at the same time. I've seen several questions that should be closed getting answered rather than closed (and this might be a side-effect of the massive queue size for all I know).

For me, it's a question of whether review is worth it, compared to finding and tackling low-quality questions head-on as they emerge. Ironically, that doesn't work properly without review (there's relatively few active closers on the tags I'm watching, from what I can tell).

The CV queue doesn't work without people flagging or voting to close on the questions in the first place, but posts getting closed doesn't work effectively without review.

The partial lack of a coordinated effort (looking aside SOCVR) likely contributes to a lot of flags aging away. A few hundred reviews per day barely helps tackling backlog of closed questions, with new questions piling on at the same rate as review handles questions, if not faster. If I remember correctly, the CV queue was at 9000 relatively recently. I'm not sure whether the reduction is because of the fuzzy algorithm, if it's because votes and flags are aging away faster than review can handle it, or if it's because SO in general is getting fewer questions worth closing.

Ways to handle it

I can't use this answer to propose all the possible solutions, because there's far too many good ones that could be implemented. One that would massively help the active reviewers, is the one that enables regenerative close votes. I personally think we should let willing reviewers review as long as they want to as well - and if it helps on gamification-related problems, remove badge progressing for extra reviews per day. And the one Yvette mentioned: fewer close votes to close.

But the CV queue isn't alone. Help and Improvement gets a lot of posts from other queues that actually belong in the close vote queue. VLQ flags have helped, but instead of reviewing and actually helping, it's skipping and flagging that's (at least my) most used action in that specific queue.

There's essentially three ways to handle it:

The first, and by far most complicated one, is reducing the amount of closeable posts being posted. The wizard has likely had an effect (considering the CV queue is down to 8000 now), but there's still answers, although they fortunately haven't spiraled out of control in review to the same extent as the CV queue.

The second option is getting more reviewers. This has been done before with the CV queue burn, but getting more long-term reviewers is hard when the unimplemented queue improvements keep pushing reviewers away.

The third, and the one I've focused on: let the active reviewers review! Give the reviewers regenerative close votes and/or require fewer close votes to close questions, or implement something else to make sure those who review actually have a chance of making an impact, rather than browsing through a queue with 40 reviews per day.

And if the issue is a concern for abuse, why not let gold badge holder votes have more weight?

If the issue is with "zombie reviewers" clicking buttons for a badge, why not remove progress towards review badges after 40 reviews (or another number for different queues)?

Final notes

Please implement one or more feature requests that make review efficient. Assuming this comment is correct, last time there was a strike, the only action taken was reducing the visible number. While this may help motivating reviewers, it is useless alone. You can hide the truth all you want, but if nothing is done to tackle it, it's essentially like sitting in a burning house and ignoring it.

I mentioned the review burn earlier, but there is one thing I haven't mentioned yet, and that is this answer. The post itself isn't actually the main point, but before I go on, look at the numbers: all of them, including CV review, says 0.

However, under two hours later, it was back at 14000 questions. I can't verify the legitimacy of these images (given they weren't created by me, and it was 5 years ago), but if they are correct, that also means one thing for the review queue right now: there are guaranteed more than 8000 questions in the review queue.

I mentioned later the review queue size is "obfuscated" by an algorithm that makes it appear smaller. The comment I mentioned in the previous paragraph was likely made when the algorithm wasn't as aggressive. When it spikes by 14k questions in under two hours, and with activity only increasing since, it serves as further proof 8000 isn't a realistic number.

TL;DR:

If no review enhancements are made, we will not be able to moderate content efficiently given the current amount of votes, available reviews per user, and reviewers. We currently aren't either, given the massive volume of posts SO is getting.

With, on some of the worst weeks, up to 94% of review tasks expiring, this is a problem obfuscating the numbers further cannot fix. However, in light of a recent post here on meta, it has come to my attention SE actually isn't working on features for core members. That may make the purpose behind striking pointless if this push for change isn't enough (and in my opinion, based on previous posts, this probably isn't enough unless there's commitment to it). These past few days, there have been several feature requests with status changes. Some declined, some completed, some being worked on - this is progress into the massive pile of seemingly ignored feature requests. I'm not sure how long it takes until moderation-related feature requests start being implemented, but what SE stated earlier is no longer the case (fortunately). Just the fact that feature requests are being implemented is a good sign after a long time of nearly no implementations.

Related

35
  • 12
    The number of unimplemented feature requests isn't really a useful stat because the vast majority of them won't be good ideas. It's just not a good indication of the problem at all. Next, changing how many posts get reviewed (whether up or down) won't change the queue size meaningfully. If there are more reviews done then fewer posts will age out, and the queue wills stay about the same size, if fewer posts are reviewed more will age out, and the queue will stay about the same size. The actual queue is numerous orders of magnitude larger than what's shown.
    – Servy
    May 17, 2019 at 20:09
  • 31
    @Servy The size of the unimplemented feature request is one thing, but any of them could have a status-denied added if it isn't implementable. No response what so ever, at least to me, makes it seem like most of the feature requests are plain out ignored. The queue size itself is honestly confusing because of the fuzzy numbers, and you're completely right about that. But the only way to get somewhere is if the stuff that's in there is actually reviewed.
    – Zoe is on strike Mod
    May 17, 2019 at 20:12
  • 4
    @Zoe Along with that, the only thing we can do is try something, because what we are doing now obviously isn't working. Seems like several people just want to sit back and continue on the path we are already on which has been failing us for some time now. May 17, 2019 at 20:30
  • 2
    @YvetteColomb yeah, I figured you meant those ^^" Also, you and the other mods deserve credit just for trying - closing stuff requires a lot of attention and energy, and coordinated efforts are even harder (even though SOCVR exists). The binding votes do make an impact on the review efficiency. The only way to actually fix it at this point is making sure there are efficiency improvements so we avoid burning out users, whether moderators or 3k users, while making sure the queues (and especially CV review) are designed in a way that both avoids that, at the same time as it enables more reviews.
    – Zoe is on strike Mod
    May 18, 2019 at 11:28
  • 4
    "~94% of CV reviews were made for nothing" (on some weeks) -- is it possible to obtain similar stats for older reviews, to compare to how it was, say, 4-5 years ago?
    – gnat
    May 19, 2019 at 15:37
  • 9
    I see, thanks. Looks like it wasn't too good back then and it has gotten only worse over time and at this point it is just awfully bad. And this is despite community permanently complaining and suggesting improvements. Given what I learned so far 94% reviews going to waste bin make a pretty good reason to stop close voting at all
    – gnat
    May 19, 2019 at 16:27
  • 11
    @gnat honestly, that's where I'm at after seeing the numbers. I knew it was bad, but not that bad. Unless SE takes action to handle these issues rather than hiding them, we might as well stop. The decidedly worst part is that some of the other high-volume sites have the same problems. You can change the site to run the query on. Take a look at for an instance SU or Ask Ubuntu (the two top sites after Stack Overflow in traffic) - neither of those have specifically good rates either (last week, Ask Ubuntu was at 43% invalidation), but it's noticeably better than SO.
    – Zoe is on strike Mod
    May 19, 2019 at 16:33
  • 5
    Taking a look at some lower volume sites (i.e. Code Review, with more users than visits per day), they have more completed tasks than invalidated ones. I don't think the review system is currently designed to handle the extreme volume of posts on some of the bigger sites in the network.
    – Zoe is on strike Mod
    May 19, 2019 at 16:35
  • 3
    Interesting. When I review, I tend to filter to select only duplicates for which I have a gold badge, so I can instantly close the close worthy questions.
    – Raedwald
    May 23, 2019 at 6:09
  • 6
    @Raedwald after doing the same thing for a couple of years, I realized that was no more effective than pissing in a volcano in and effort to put it out. The number of votes you get vs the number of things needing closure is so fractional it is not statistically significant. The only measurable effect it has was to increase the amount of on and off site personal harassment I got from people that had their question closed. So I just gave up and stopped all together.
    – user10677470
    May 30, 2019 at 18:46
  • 9
    One critical flaw in your Statistics section: votes aging out will invalidate a review, and review tasks themselves will age out and invalidate... But they'll also be invalidated if the post is closed from outside review or deleted. So for example, an asker confirming a duplicate vote immediately closes the question and invalidates the review - but the vote itself was still helpful. Unfortunately, this stuff is fiendishly complex; actual portion of aged votes/flags is about 42% of all raised, but even that is misleading: some of those aged away because review completed… with "Do not close".
    – Shog9
    Jun 25, 2019 at 16:16
  • 3
    Sounds about right, @gnat. Essentially, try to match the review result type with the outcome of the post. PostsWithDeleted may help here, Zoe
    – Shog9
    Jun 25, 2019 at 16:41
  • 4
    Oh... One other little wrinkle: audits are included in this data, but not indicated. They shouldn't be invalidated very often, but may skew results otherwise if you're just comparing against post state.
    – Shog9
    Jun 25, 2019 at 16:48
  • 3
    I have a sneaking suspicion we're not pushing all review data to SEDE here; it's been a long time since that was written and I don't recall what we did anymore, but the original intent was to allow analysis of pending reviews... So all bets are off for completed ones. We might be dropping everything pertaining to deleted posts or something, gonna have to find time to ask bluefeet about the migration.
    – Shog9
    Jun 27, 2019 at 13:54
  • 4
    As Shog noted, @Zoe, the data on the public SEDE does not give a full view of the situation here — mind posting a separate question, asking for accurate numbers on this, so we can answer it properly? :) Doing it in the comment section here is hard, and as a separate answer seems awkward.
    – JNat StaffMod
    Jun 27, 2019 at 16:35
58

I see the frustration here and totally understand it. I'm hoping that this post will encourage the network to trial 3 close votes for a close vote queue > 2000. 4 close votes for close vote queue > 1000, and 5 close votes <1000.

The same would need to apply to the reopen queue, as it could blow out if the close vote queue is rapidly sorted.

With all the accumulated feedback and the traction this question itself has, I wish I had a magic switch and could implement this.

It seems the network is working hard on trying to improve the UX and tackle question quality from this angle. However, in the meantime we are still battling with an unbeatable queue. Even with unlimited close votes and the mod power of one vote to close each question, the close queue is too hard for me. A couple of mods have discussed attacking it in a coordinated fashion, but that's really not a solution. It's a good way to exhaust people.

Please, please trial less close votes.

5
  • 21
    I see the frustration here and totally understand it. I'm hoping that this post will encourage the network to trial 3 close votes for a close vote queue > 2000. 4 close votes for close vote queue > 1000, and 5 close votes <1000. I admire the enthusiasm but I don't think anything will happen or this will even get an acknowledgement
    – Magisch
    May 17, 2019 at 17:00
  • 3
    That has been trialed in the past. softwareengineering.meta.stackexchange.com/q/7723/42934 . It only highlighted the fact that there are a lot of users inappropriately closing posts because of perceived problems instead of actual ones. What it did do was accelerate reopening. There is no problem caused by a post which only 30 people per year view, which is often automatically deleted. There is a problem when a pervasive view of what could be potentially problematic prevents the creation of content. 3 close votes would only amplify that issue.
    – Travis J
    May 30, 2019 at 18:08
  • 1
    Whether that is on the community, is an issue of communication versus intent, or is reflective of an overall need to revisit close reasons (I think its this one); is another discussion entirely.
    – Travis J
    May 30, 2019 at 18:08
  • 1
    The Boeing solution: reduce the number of controls when under pressure! Jun 30, 2019 at 13:58
  • 4
    This has now happened!
    – Zoe is on strike Mod
    Aug 8, 2019 at 18:48
39

A good deal of the people participating on this site are donating their time. This is true of people asking good questions, of people answering those questions, and of the people who go out of their way to moderate contents.

No one is entitled to this donation of time. Not the other users of the site, not SE, Inc.... no one.

The people who donate their time to the site usually are choosing to do this at the expense of something else. When someone decides to donate time here, that time is not available for donating somewhere else. So why donate here, by opposition to donating somewhere else? Stack Overflow is competing for the bucket of time people can donate away. If the people using the site end up frustrated, they're going to donate elsewhere.

That's what happened with me. There are so many things I can do other than endure the endless frustration of going into the moderation queues that I've decided my time would be better spent elsewhere.

Some reply with an appeal to self-interest:

But if you don't take out the crap, you're going to run into more crap. That's self-defeating!

Hmm... no. Before I gave up formal moderation I would, multiple times a day, check new questions in the niche tags I care about and vote, vote-to-close, vote-to-reopen, flag, or answer the posts I ran into. I would also go into the review queues to deal with the crap. If I saw strange voting patterns or cases of plagiarism, I took the time to gather the evidence and raise a flag. I also participated in the SOCVR. If you think you run into crap in the queues, participate in the SOCVR. You'll be exposed to more crap there.

When I gave up formal moderation, I also gave up my former routine. My former routine has been replaced with subscriptions to some bounty feeds. It does wonders to improve the signal to noise ratio. It is rare that bad questions end up with a bounty (though some do, and I flag them), and those questions that get a bounty often are more interesting than the rest of the lot (not always, but often). The net result is that I see much less crap now.

Another objection:

But if people stop moderating, it will mean more work for the people still moderating.

Yeah, and if I do moderate SO instead of donating my time elsewhere, then the people who benefit from my time elsewhere will suffer. Why should I prioritize SO over those other areas where I can donate time?

All else being equal, I'm going to donate time to an activity in which I feel I'm making a difference, and in which I feel the least amount of frustration. Ultimately, spending time formally moderating on SO is just not competitive compared to the other opportunities where I can donate my time.

2
  • I totally agree that answering interesting questions for example is much more fun than reviewing not very interesting questions. However to spell out the consequence of this answer: reviews are doomed and we will have to live with unreviewed crap. This basically shifts the problem to how to find the interesting questions between all the crappy ones? If that is not possible, stackoverflow might die at some point. May 19, 2019 at 6:55
  • 14
    @Trilarion You could consider the site "dead" at that point in time at which the signal-to-noise ratio (SNR) becomes too low for you to use the site. I definitely think as we speak the SNR is on a downward trajectory. The lack of improvement in moderation mechanisms brings "the death of SO" closer to the present. However, there could be improvements elsewhere (e.g. substantially better search) that have a strong enough effect to counter the current neglect. Though I'm skeptical that better search alone is enough. Crap posts that manage to avoid auto-deletion have an ongoing maintenance cost.
    – Louis
    May 19, 2019 at 17:59
22

On one hand, I applaud the rallying cry. I sincerely hope your strike works where everything else has failed.

On the other hand, experience has made me cynical. Investment in curation tools has not scaled with site growth, despite constant pleas over the years. So, my gut tells me that your strike, at best, may rally a few participants, garner some moderator support, and potentially inspire Shog or Jon to broach the subject during some meeting. That's where it will die.

The only people who care are the people who already sympathize; they're in the trenches with you. They don't call the shots. They don't have the power to tell development, "Fix this." Your effort isn't important to the people who actually have that power. Those people have built an empire on the labor and expertise of users they take for granted.

Look, we've all had the mistaken impression, at one point or another, that our individual participation is important to SE. It's not. So, stop working for free with frustrating tools. Don't strike; just stop. Use the site, or not, on your own terms.

I know you've invested a lot of time and effort but sometimes you have to let bad investments go.

4
  • 9
    I did just what you are suggesting. After almost 10 years of contributions. I had them reset my account last year to 0 from 40+K completely divested now. It became unignorable how we were not just unappreciated, but held in contempt by the "powers that be". Just having popcorn and watching what I predicted actually happening, like a train wreck in slow motion. I lost nothing and I gained everything.
    – user10677470
    May 18, 2019 at 2:50
  • 1
    "Investment in curation tools has not scaled with site growth at all." -- that's nothing abnormal and thus nothing to get jaded about. A solution to a problem can only be started being thought of after that problem arises, not before. Who knows how much time will pass and how many ideas fall through before someone finally comes up with a real solution. May 18, 2019 at 11:35
  • 4
    @JarrodRoberson I don't see what you "gained" other than emptiness. May 18, 2019 at 11:35
  • 4
    @canon I just figured I'd mention that for like 5 years SE just ignored the site, and while I didn't love it, I could deal with it. The problem is in the past few years they've been actively harming the site's mission, and doing things that make it harder, rather than just spending all their time making new products that I can safely ignore.
    – Servy
    Jun 21, 2019 at 19:24
20

I think the call to action is implicitly clear, stop voting; period.

  • Stop reviewing organically as well.
  • Stop close voting.
  • Stop down voting.
  • Stop voting period.

Just to reinforce a sentiment in a comment on the main question, "I see much less garbage with < 150 rep than I did when I had 40+K rep."

If you do not think that a large percentage of the community moderators just taking a week or two off doing any moderation at all will not get some attention and maybe some appreciation and less likely action you are just fooling yourself.

SOCVR was a direct response to organic reviews and the review queues failing to be effective because of lack of participation by those with the rep to actually take action. When it first started up and I discovered it, I was excited to see there were people just a interested in keeping the site clean as possible.

I wanted to protect the reputation I had gained on the site because I foolishly believed it was valuable. When I realized that rep indicated nothing at best or at worst just the opposite, is when I stopped doing things that gained it, basically I stopped answering questions. I just moderated the limited amount I could by cleaning up the most obvious trash. When it was clear that we were just held in contempt for being free janitors for the owners of the site, I just quit completely. After about 6 months, when it was obvious, they were just doubling down on the attitude that the OG users were the reason the site was un-welcoming. I just had them delete my original account, just like I had them remove my name from all my contributions to Documentation when I saw what I had contributed edited into completely incorrect garbage with my name still on it.

I lurk in SOCVR every once in a while now, when I have a couple minutes to waste and it is a wasteland compared to just a year ago, I only recognize a couple names now, and where there were almost two dozen people in there at any given time I normally participated, there is only a handful during the same times of the day now.

Unfortunately, the room was too good at its stated purpose. It does concentrate those people with rep and that care about keeping the site clean to focus their limited votes on the most egregious and actionable crap on the site. But it exposes in the harshest terms what a waste of time it ultimately is over a very short amount of time.

Anyone that participates in the room for more than a trivial amount of time will just get burned out completely because they see the futility of the Sisyphean task that is Close Voting. Especially when someone with a gold hammer can un-hammer and re-open things that should stay closed just because they have a belief that nothing should be closed, that took 5X as many people to close and 5X as many people to re-close; that was the straw that broke my back.

I figure two weeks of 40% of the people with rep to close vote just not doing it will overwhelm the main page with so much crap content it would be almost impossible to recover from it for months.

A full month of the people currently bearing the load moderating the site going on strike would most likely doom it completely from the overload of garbage that it would probably not be able to recover from.

25
  • 21
    I must admit my enthusiasm for the site has waned. I also spent time in the socvr and shared the same initial excitement for hope and then realised it was impossible. I'm so frustrated. Begging, asking, threatening, it doesn't seem to matter what we do as a community it is a low priority. Fixing the UX to assist in preventing low quality posts will never stop the problem, no UI is fool proof and that's more or less what we're dealing with. I get energy, make a push, send off emails and nothing changes. What's doing the same thing over and over and expecting a different result?
    – user3956566
    May 18, 2019 at 11:10
  • 13
    SOCVR does feel like an exercise in futility, sometimes. It's part of why I don't frequent anymore, I just get reminded that the stream of trash is neverending.
    – Magisch
    May 18, 2019 at 11:20
  • 4
    The only solution to the "neverending stream of trash" is an arms race. Tools to fight it ever more efficiently. (I mean "tools" in the broad sense. E.g. "making right things easy, making wrong things hard" is just as good a tool.) May 18, 2019 at 12:03
  • 12
    @Magisch - neverending and increasing. Entropy will completely overwhelm the site eventually if they do not do something drastic and immediately. I predicted that Documentation would be a flop for exactly the reason it failed and the main site is headed in the same direction for the exact same reason quality is no longer part of the Charter. Just a slower pace the last few years since the quality requirement was removed from the charter of the site. Since that has no longer been the goal of the site, the entropy has been increasing each year more and more.
    – user10677470
    May 18, 2019 at 20:53
  • 14
    SOCVR was never meant as "an solution". That some / most users want to see it that way is fine with me. From our site The SO Close Vote Review Chat Room is a group of like-minded people who want to make a difference in the CV queue, reviewing and clean-up efforts. It is on purpose that it didn't define any clearer what that difference exactly would mean and I'm not planning on making it any more exact. The room is meant to meet-up with other reviewers, learn a bit about moderation, share experiences, have some fun. That is all there is in it, really.
    – rene
    May 19, 2019 at 16:29
  • 1
    @rene - you are the only person to use the words "an solution" on the page so not sure where you are "quoting" that from. I said, "SOCVR was a direct response to organic reviews and the review queues failing to be effective because of lack of participation by those with the rep to actually take action." which is not inaccurate. Nobody but you claims anywhere on the page that it was "an solution" that I an see.
    – user10677470
    May 19, 2019 at 19:38
  • 4
    @JarrodRoberson SOCVR was not "a direct response" (I air-quoted "an solution" to make that connection) as you describe. I would appreciate if you stop portraying SOCVR as something it never was, had been or will be (assuming I have enough life left to prevent non-sense from happening).
    – rene
    May 19, 2019 at 19:50
  • 1
    "The SO Close Vote Review Chat Room is a group of like-minded people who want to make a difference in the CV queue, reviewing and clean-up efforts." sounds like a response to the failure of the CV queue and organic reviews ineffectiveness because people were not reviewing directed at making those that were voting at making a more of a difference. But whatever, SOCVR was like drinking from a firehose at the height of participation with a dozen or more of us of use burning all our CV every day. Now, it is even more futile with the less than a handful of people I see when I visit.
    – user10677470
    May 20, 2019 at 1:28
  • 3
    @rene - I only mentioned SOCVR because it most likely what the most effective tool possible would look like and it is ineffective in the long run. If it is ineffective then the problem is not solvable given the current structure of the site. The fact that those that can and do CV get crapped on constantly because CV are not anonymous. Good Luck, but the participation, or lack thereof, in your room should be a wake up call to the powers that be. It is a good coal mine canary indicator for engagement in the rest of the site for sure.
    – user10677470
    May 20, 2019 at 1:43
  • 1
    Organizing is an entirely separate need, I think, @JarrodRoberson. Even if there were 1000 people maxing out their reviews every day and closing things organically (and a feature-rich review queue), there would still be folks who would want to organize. After all, we are a bunch of programmers, whose shared curse is that we always look at a situation and ask "how can we improve on this". Sorry to hear you are burnt out, but luckily I think there are plenty of us still there who are not and are willing to continue contributing curation in the face of things like OP's tantrum.
    – TylerH
    May 20, 2019 at 1:45
  • 11
    @TylerH - Using the word (tantrum) is just an ad hominem attack on the OP because you do not like their opinion. It is unkind at best and in my opinion passive aggressive abusive. But way to make your point and get them/us back on your side. Good Luck with that!
    – user10677470
    May 20, 2019 at 1:55
  • 1
    @JarrodRoberson I don't understand this answer; was the praise of volunteer reviewers in this blog post not enough for you?
    – Rob Grant
    May 20, 2019 at 8:51
  • 12
    @RobertGrant I don't understand your comment, does it mean that if they post a blog article once a year / month with words like "hats off to volunteer moderators, we need to do more to help them" and don't do anything else, that should keep community happy?
    – gnat
    May 20, 2019 at 11:27
  • 1
    @JarrodRoberson ad hominem is an attack on someone's person. Calling the post above a tantrum is a criticism of form and behavior. The two are different, and criticism is not inherently abusive. Have a nice day.
    – TylerH
    May 20, 2019 at 15:19
  • 5
    It's going to take a long time of no/little new quality content added to the site for the site to actually die. Even if the site went into read only mode today and stopped allowing new contributions, the site would be a major player in it's niche for years before it lacked enough important information or updates to not be a reliable source.
    – Servy
    May 20, 2019 at 15:27
20

If nobody voted to close anymore, would SE care? Recall that in his infamous blog post about SO being "unwelcoming", Jaydles said that downvoting lazy duplicate questions makes him "sad". Presumably he was not alone in considering curation to be a problem rather than an asset. If so, going on strike would be seen as improving things; it would be a worthless threat (although perhaps still worthwhile for your own peace of mind).

5
  • 1
    I think one of the goals of expressing to SE our frustration with moderation activities is to make (a perhaps futile) attempt at getting a better reading on the degree to which SE cares about curation. When I read the level of care from SE regarding curation with my Care-O-Meter, it shows a needle pointing more towards "don't care" than "care".
    – Louis
    May 22, 2019 at 13:55
  • 11
    from perspective of that blog post there is nothing to improve because at stats showing 94% reviews wasted they can consider mission already accomplished. Those joining the strike will simply spare themselves from wasting time on useless activity
    – gnat
    May 22, 2019 at 14:28
  • He didn't say "downvoting lazy duplicate questions makes him "sad"." You added the word "lazy" in there. Just saying. May 23, 2019 at 13:44
  • 11
    @AlexHarvey He said that downvoting duplicates makes him sad, without qualification. That means, downvoting for any reason makes him sad. Including lazy questions.
    – Raedwald
    May 23, 2019 at 14:56
  • 4
    @Raedwald, in context, he wrote: "It makes me sad when someone gets downvoted for posting a duplicate. We should better surface them in the posting flow, but it’s not reasonable to expect askers to find dupes consistently. Users aren’t “too lazy” to search; searching takes less work than posting." So, actually, he has said that he doesn't believe that duplicates are, in general, lazy. What you have attributed to him is quite inconsistent with what he actually said. May 23, 2019 at 15:57
-29

It seems to me that if you find the review system frustrating to the point where you don't enjoy doing reviews, you should stop doing reviews. As Cody Gray points out:

The frustration is understandable, but the act of protest will not have meaningful consequences. You aren’t donating your time reviewing to the team that runs Stack Overflow. You’re donating it to the community of people who use Stack Overflow, to ask and answer questions. The people who started using it and want to keep using it because it avoids their having to waste time slogging through piles of crap. Refusing to review will just mean more crap for you, and I, and everyone else. It won’t make the review queues get fixed. Fewer reviewers means either more work for fewer, or more crap.

Mechanically, review queues are a system designed to spread the load of curation. A lot of that curation happens organically anyway, but review is designed to bring to light posts that might have gone unnoticed. Without review, those posts will likely continue to go unnoticed and deleted after a year.

Unfortunately, our development efforts are pointed elsewhere at the moment. Between custom question lists, the Ask Question Wizard and a duplicate questions display experiment, we're looking at shaving some questions off the top of the funnel and making it easier for answerers to find the questions they want to address. (There are ways to use the custom question list feature to assist in curation as well.) There is more work to be done on these projects so that they accomplish those goals.

Now there are a few existing site settings community managers could use to change the way the review system works. (I've had, um, adventures playing with these settings.) Whether or not twiddling thresholds will help make the queue more enjoyable to use is hard to know. The one setting I do know can help is to lower the review indicator threshold for the close queue. It's currently set at 10k and the review queue is at 8.6k. If we lowered the threshold to something closer to that number, the review queue indicator will light up more often. That will bring more reviewers into the queue, which lightens the load.


After writing the above answer and before posting, I was in a meeting with Meg, our PM. I mentioned this question and Shog reminded us of a setting that we tested a few years ago. It personalized reviews so that people would be more likely to see posts in tags they had interest in. Unfortunately, that setting broke the site. Our Architecture team has more pressing issues right at the moment, but when things settle down, Meg would like to look revisit it. We also talked about a few other ideas around review since people feeling miserable on the site is a thing we want to avoid.

22
  • 36
    "If fewer people use the review system, it means more of the load is on people viewing posts organically" - The problem with this argument is that it's more about making those refusing to review feel guilty, without actually doing anything to resolve the issue. In a workplace, that's a pretty decent sign of bad management. There was a discussion about this in the Tavern starting here.
    – fbueckert
    May 17, 2019 at 22:12
  • 3
    @fbueckert: I suppose making the review queue indicator light up would make people who don't review feel guilty, but I'm not sure why you quoted the bit about people viewing the posts organically. In any case, I'd much rather people not review than feel miserable about review. Personally, I think the work we are doing on the front end of the bad question funnel is more pressing than work on the back end of it. May 17, 2019 at 22:16
  • 6
    I wonder what would happen if everyone just stopped reviewing... not with an agenda or anything. I just wonder if it would really be all that bad if everyone just stopped.
    – canon
    May 17, 2019 at 22:22
  • 6
    @canon: I also wonder that. I suspect many questions get more views in the process of being reviewed than from people stumbling across them or via search. (I suppose I could get some data to back up that guess. Maybe next week.) If a bad question is asked and nobody sees it does it matter? May 17, 2019 at 22:25
  • 6
    If a bad question falls on the site and nobody hears it, does it still make a sound?
    – user3956566
    May 18, 2019 at 1:37
  • 3
    I think some back end work would alleviate some issues while the site is working on the front end work. Not sure how difficult some of these ideas would be to implement, but they would take up less time than all the front end work. I honestly wish you'd throw the community a bone.
    – user3956566
    May 18, 2019 at 1:39
  • 11
    @canon - lots of high rep people have just stopped reviewing for all practical intents and purposes, a long time ago, it is futile and more importantly unappreciated. I had my account reset from 40+K to 0 specifically so I would be unable to do anything on the site. I accidently got some rep that I am trying to shed right now so I can do even less, like not even being able to leave comments would keep me from wasting the little time I do spend watching the car crash in slow motion that SO is now.
    – user10677470
    May 18, 2019 at 2:27
  • 5
    @JonEricson If a bad question is asked and nobody sees it does it matter? It matters to the poster, I guess. If you think getting your question closed upsets people, what do you think happens when they start seeing nothing, like you would on a shadowban? That will drive people away more then getting questions closed, imo.
    – Magisch
    May 20, 2019 at 8:18
  • 7
    Jon, spreading the load was the original intent of review but I am not sure that it holds now. Stats in another answer suggest that up to 94% reviews end in nothing - that sounds more like a wasted effort to me
    – gnat
    May 20, 2019 at 8:25
  • 3
    "I wonder what would happen if everyone just stopped reviewing... not with an agenda or anything. I just wonder if it would really be all that bad if everyone just stopped." The easiest way to see what that would end up like is simply looking at other similar Q/A sites that don't have the same quality standards and moderation tools that SO has. If SO's standards aren't enforced, and their tools aren't utilized, it will continue to look more an more like its competitors, the traditional forums.
    – Servy
    May 20, 2019 at 15:18
  • 3
    @Servy Are reviews what's keeping us from becoming a forum? What about the Q&A format itself? Organic curation/voting? Are the queues really what protects us from the void?
    – canon
    May 20, 2019 at 16:06
  • 3
    @Louis: It won't help the underlying problem of questions needing to be closed, but it will prevent people from being pulled into Sisyphean work. Perhaps it's time to rethink closing. May 20, 2019 at 20:30
  • 5
    @JonEricson Because not closing anything and thus letting crap live on the site is going to make things better?
    – Louis
    May 20, 2019 at 20:49
  • 3
    @Louis If 94% of CV tasks are expiring without closing the question, then removing the queue would be pretty similar to the status-quo, minus wasting a bunch of reviewers' time.
    – Troyen
    May 20, 2019 at 21:14
  • 3
    I know this seems heavily downvoted, but I for one am happy to hear this outlook. Addressing content quality where it is produced, and allowing users easier paths to connecting with quality content will only benefit the site. The main issue with closure was to make sure users didn't waste time answering a question which was not topical, it makes absolutely no sense to spend development time making better mops when we can just prevent leaks.
    – Travis J
    May 30, 2019 at 17:56
-38

Voting is a democratic process. Giving some people more votes than others is not very democratic.

There are questions that should obviously remain open, should obviously be closed, and then there's a large grey area. Some people talk about "quality" like some sort of measurable objective thing but it's not (or at least not completely). There's a decent amount of judgement involved.

Voting on Stack Overflow is already a skewed democracy; I don't think that the people who are very active in the review queues are representative of the community at large, and instead of some form of consensus system (e.g. majority vote), it's an absolute system: 5 people out of the total community have to agree. Imagine parliament working like that: find 200 people who find a law a good idea and it's passed. Find 12 people who think someone's guilty and off to prison they go!

Many people feel questions are closed too quickly. I suspect that your suggestion would make this worse. There are trade-offs to be made.

The team is also working on reducing the inflow of bad questions with the question wizard and such, which is essentially a different approach to solve the same problem (too many questions that need closing). That doesn't mean I don't think there can be improvements to the review system, but there's more ways to skin a cat and "I will strike until this specific suggestion gets implemented" doesn't strike me as a constructive way to have a conversation about it. Quite frankly, it looks like a tantrum.


A bit of an aside on improvements to Stack Overflow in general: like many people I have a lot of ideas how Stack Overflow could potentially be improved, but I don't know they will actually make things better; I just think they are based on my experience, which is incomplete and biased. I can't rule out unexpected effects.

Much can be said about Stack Overflow and how it works, but people sometimes seem to forget that in the end, it really is a spectacular success. Even small changes can have unexpected effects down the line that may not be obvious. I think a healthy dose of conservatism is appropriate.

17
  • 11
    en.wikipedia.org/wiki/Meritocracy is grounds for giving more voting power to some who have proven themselves May 18, 2019 at 11:09
  • If you're trying to suggest that people with more rep have more "merit" to close questions, then that is completely besides the point @ivan_pozdeev, and isn't really a reply to anything I've said in my answer (dumping a link is probably not the best way to have a constructive conversation). May 18, 2019 at 11:17
  • 10
    to call it a tantrum is unfair. People are tired and burnt out. It's a sign of how desperate people are that the question is upvoted and gaining attention. Aside from this I like many of your points. Although I think the question wizard will only fix so much
    – user3956566
    May 18, 2019 at 11:17
  • I don't know what to think of the question wizard as such @YvetteColomb, I haven't looked at it closely. It's unlikely a panacea, but it does show that action is being taken. As for tantrum, perhaps my chief issue with the post (and the way it's written) is that it's completely ignoring there are many people who disagree with how many questions are closed in the first place and is attempting to force the SE team's hand by "striking". I don't think this will be effective at all, but I find this kind of attitude to be ... not great. May 18, 2019 at 11:24
  • 2
    I edited my last comment. On your other statements, the review system is designed to produce a concensus. This is a different (and evidently more progressive) decision-making process than a majority vote. Wikipedia is built upon it and it's there where it proved its merit. May 18, 2019 at 11:55
  • 17
    This isn't a democracy @MartinTournoij. Nobody says it is, the sites guidelines do not imply it is, the way it's moderated do not imply it is. As for the people who dislike seeing questions being closed, they sure as hell don't care about the other side's viewpoint, so why should others do? May 18, 2019 at 12:04
  • the review system is designed to produce a concensus Yes, exactly @ivan_pozdeev, but if you empower a (small) group with strong opinions then you'll get further removed from consensus, especially since people who feel a question should remain opened can't do anything about it other than not casting a close vote. In cases of dispute, it's already the close-vote side who will have the upper hand. May 18, 2019 at 12:18
  • I never claimed Stack Overflow as a whole is democratic @FélixGagnon-Grenier, I said that voting is and that there is an element of judgement involved. Applying the law also isn't democratic (you either broke it or you didn't), yet there are judges and jurors and all that to determine if someone broke the law. May 18, 2019 at 12:19
  • I do agree about the overall spectacular success and small changes having possible unexpected effects, which is why requests from user base are evaluated by the folks that make the site first (: May 18, 2019 at 12:20
  • 4
    "if you empower a (small) group with strong opinions then you'll get further removed from consensus" -- not if that group is more likely to produce concensual decisions than the general public. May 18, 2019 at 12:22
  • 1
    Wikipedia is built upon it and it's there where it proved its merit Wikipedia is an interesting example @ivan_pozdeev, and I would argue it's very very far removed from a meritocracy. Instead, it's an "activeocracy": whoever is the most active wins. I can be 100% correct but if I don't feel like battling that one toxic Wikipedia editor then my changes are not going to stick. I'm not saying Wikipedia is all bad (like SO, it's hard to argue with its success), but IMHO it does have serious problems. I stopped contributing a long time ago as it was 25% writing articles and 75% bikesheds. May 18, 2019 at 12:23
  • 5
    Ok, maybe I misunderstand the spirit of your answer @Martin, but the statement "Voting is a democratic process. Giving some people more votes than others is not very democratic." (emphasis mine) clearly imply (to me) that you believe it should be a more democratic process, wherever you place the boundaries of that democratic endeavour. It should not. May 18, 2019 at 12:24
  • Yes, but I don't think they will @ivan_pozdeev, as I mentioned the entire system is already very biased towards closing questions rather than keeping them open, and it's already hard to prevent a question from getting closed. This will only make that worse. Half the questions I see in the close vote review queue should IMHO very obviously remain open. The reason I'm tired of reviewing is because of that. May 18, 2019 at 12:26
  • 7
    @MartinTournoij The bias is intentional, to close borderline cases as substandard, on the assumption that there's enough influx of new questions to eventually get a better version of a closed one if it really is worth keeping around. May 18, 2019 at 13:42
  • 6
    "if you empower a (small) group with strong opinions then you'll get further removed from consensus" this is a ridiculous logical fallacy. You get a concensus of those that are able to vote. Claiming it is not a consensus because there are those that are not allowed to vote because they have not earned the right to vote is illogical. It requires a consensus of how every many people that have earned the right to vote, plain and simple. If you want a consensus of the every question is sacred crowd, enjoy the same results as you get in the link.
    – user10677470
    May 19, 2019 at 19:49
-42

It is surprising that so many people are spending their volunteer hours manually reviewing close vote queues, reopen vote queues, and down votes, when the whole thing could be automated.

Simple, just:

  • Remove the close, and down vote features completely.
  • Use machine learning to identify low value questions based on views per week, upvotes and stars. And perhaps other metrics. I'm sure SO has some great data scientists.
  • Auto close questions that are of low value.
  • Allow closed questions to be challenged if someone notices and cares and disagrees with the bots.

A simple system that is designed for the year 2019, that would be fairer, completely remove the maintenance burden from senior developers who should be writing code, not wasting their time doing menial work. It would be fairer, and there would be little, if anything, to argue about.

SO's unwelcoming culture problem would be solved at the same time.

20
  • 12
    "Use machine learning to identify low value questions based on views per week, upvotes and stars." -- most questions get too little human feedback to reliably judge their value. E.g. average question score is getting ever closer to zero. May 23, 2019 at 9:56
  • Auto close questions that are of low value. huh? A warning must be enough. May 23, 2019 at 10:28
  • 10
    this reminds the way how Quora tries to curate its content. Given that quora turns out almost completely useless when I search for solutions to my coding problems (as opposed to SO) I would rather prefer that we keep the old way here... at least until I notice quora getting any better in helping my coding
    – gnat
    May 23, 2019 at 10:44
  • @ivan_pozdeev, to your points: 1/ most questions and content really is of low value, and if rules were being enforced equally and consistently, then most questions would be closed. 2/ aside from that, the new system would change voting behaviour. If users lost the ability to downvote, they would probably become nicer, and they would upvote more, understanding that no vote = an opinion that the question is of low quality. May 23, 2019 at 10:50
  • @gnat, Quora's quality is low because everyone wants to ask and answer over here at SO. May 23, 2019 at 10:51
  • In general, obviously, the data is there. Every post could be measured for, the number of answers; the number of interactions; the number of views; its ranking on Google; the number of stars; the number of comments; the quality of the writing; and no doubt much more. Obviously, it could be done. May 23, 2019 at 10:54
  • As can be seen, @Jeremy, the baseline to compare the proposed system with is the real one. Not the idealised one that doesn't exist, never has existed, and never will exist. The real one. Where the resources required to make it work are about 1% of what we actually have, Where people are going on strike because there's just not enough people to even manage the close vote queues. Imagine if all questions that should be closed were also flagged. Google's ranking algorithms already do just fine, and that's actually all we need. May 23, 2019 at 12:29
  • 2
    We could close a lot of bad questions ("flip a coin" is a heuristic that would achieve > 50% accuracy), but I fear this is a situation where the ratio of good:bad content is so skewed that it would be very hard to obtain a sufficiently low false-positive rate to avoid closing a large fraction of good questions as well. (There's probably a technical term for that.)
    – Jeremy
    May 23, 2019 at 12:40
  • 9
    This again? Your continuing narrative of curation being unwelcoming is getting rather tiring. If you want to make a point, it'd help if it didn't all hit the same old agenda.
    – fbueckert
    May 23, 2019 at 15:01
  • 16
    If you think it's so easy to write an automated system to determine question quality, why don't you? If you're able to accurately determine whether a question is clear, well researched, on topic, reproducible, useful to others, etc. entirely automatically, then great. I can't imagine it's possible, but by all means, prove me wrong. If you succeed in doing that I'm sure you'll have no problem selling your solution for many millions of dollars. You sure would put me, and other users who've invested a lot of time into manual curation, into our place.
    – Servy
    May 23, 2019 at 16:02
  • 1
    As I said above, @Servy, the baseline is the real system, not an idealised system that doesn't exist, has never existed, and never will exist. The real system is dysfunctional. It motivates the majority of users by rep to pay lip service to the rules while answering questions for rep, and a persistent minority meanwhile enforces the rules, each according to their own opinions about them. The end result is that most questions that do stay open are not "clear, well researched, reproducible, useful to others"- that bar is way too high to clear. A different, automated system could do better. May 23, 2019 at 17:01
  • 15
    @AlexHarvey So then let's see you write such a system. I'm skeptical, but by all means, prove me wrong. Just saying that it can be done doesn't really mean anything. No one has been able to manage to do it thus far. If you can provide one that's better than the current system, even if it's not the perfect ideal system, then you're right, that'd still be great. So do you have an implementation of such a system?
    – Servy
    May 23, 2019 at 17:05
  • 18
    @AlexHarvey Sure, people try automating moderation tools on SO all the time. The problem is that they generally have an extremely low accuracy, generally much lower than people. Certain particularly easy to notice problems are somewhat easier to automate, and as such, have been. What you see on the site now (as bad as it is) is everything that's able to get past the (numerous, but mostly narrow) automated roadblocks. Criticizing the system when you are failing to provide a better alternative is obviously going to be shut down. If/when you can come up with something better...
    – Servy
    May 24, 2019 at 13:21
  • 2
    -1 for I'm sure SO has some great data scientists.. Is this sarcasm? Does not seem to be based on your comments. If not cite some evidence, because it appears completely the opposite, they have none and just make decisions on feelings because "we know this because you have told us".
    – user10677470
    May 30, 2019 at 18:41
  • 1
    @Alex Harvey: I think (I could be wrong) that the triage system uses such a system to rate question quality. From Help us test question triage!: "Behind the scenes, a "quality score" is calculated for each question based on an automated analysis of the content. Those that score well are sent immediately to the homepage; those that score poorly will now be sent to Triage." Jul 12, 2019 at 10:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .