57

๐Ÿ‘‹ Welcome to June's installment of data science research updates for Meta! You can check out the most recent installment here, or last month's UX research update here.

This month, I'm cross-posting with our blog about recent work focused on a very common badge, the Tumbleweed badge. ๐ŸŒ€๐ŸŒพ

Tumbleweed and its impact

The Tumbleweed badge is awarded to questions with zero score, no answers, no comments, and low views for a week. It's an incredibly common badge, the 9th most common badge awarded over the past year.

Although this badge was originally intended to be somewhat humorous and as a consolation prize, we know that many users over the years have experienced earning this badge negatively. ๐Ÿ˜” We can use data to explore what the impact of this badge is, compared to a similar interaction. Jon Ericson suggested that we compare a set of questions that were similar in that each caused their asker to receive exactly one interaction. We compared questions that got

  • only an upvote
  • only a comment
  • only an answer
  • only a downvote
  • the Tumbleweed badge

How many questions did users ask after these single-interaction questions?

enter image description here

We don't see evidence that the Tumbleweed badge is an effective consolation prize, even compared to getting a downvote!

What can we learn about the questions that users ask after these interactions?

enter image description here

For users who do ask questions after these low-interaction questions of various kinds, those who earned the Tumbleweed badge don't fare differently from those who got an answer or comment. We don't see evidence that, as a population overall, the Tumbleweed earners ask particularly bad questions, just that they are less likely to ask again.

Why worry about all these Tumbleweeds rolling along?

So the Tumbleweed badge isn't helping, and it may be hurting. This is the opposite of what our incentive systems are intended to do, so we have decided to retire the Tumbleweed badge, starting today. We are also introducing two shiny new badges, focused on reversing the score of a negative question; read more about it in Jon's blog post.

Thanks so much for reading! ๐Ÿ™Œ What have been your experiences with badges on Stack Overflow? Has earning any particular badge stood out to you, either positively or negatively?

48
  • 132
    You're attributing the lower activity to a badge, not to someone putting a lot of effort into a question and not receiving any response whatsoever. I think the badge is 100% irrelevant here, it's more likely that the fact that a question got ignored that makes people unlikely to ask again. Any feedback is probably better than no feedback, because they know that their question got read at least.
    – Erik A
    Commented Jun 18, 2019 at 16:34
  • 44
    @ErikA We have enough user feedback to know that the badge isn't 100% irrelevant (many people feel negatively when they get this badge), and this analysis shows that the badge isn't helping compared to other kinds of interactions. That's the point we're making, not that the badge causes lower activity.
    – Julia Silge StaffMod
    Commented Jun 18, 2019 at 16:44
  • 34
    That may or may not be true, but you're not providing any evidence that can separate between the effect of earning the badge and the effect of posting something that qualifies for the badge. If people tell you they feel bad/ashamed, then that's a good reason to remove it, of course.
    – Erik A
    Commented Jun 18, 2019 at 16:53
  • 16
    @ErikA Yep, since this badge has been awarded on Stack Overflow for ~10 years now, we can't use a gold standard like an A/B test to understand its effects. This is why we chose to look at data we do have to get some insight, comparing to other single interactions.
    – Julia Silge StaffMod
    Commented Jun 18, 2019 at 17:04
  • 28
    I appreciate that, but I'd like to see something about the direct feedback in the question to justify this change. That's the most relevant piece of info advocating for removal of the badge imo, the numbers are just an illustration that mainly shows that way more questions than I expected get ignored. And that's a very real problem that should get our attention, but removing a badge won't help for that.
    – Erik A
    Commented Jun 18, 2019 at 17:13
  • 6
    @gmauch These are box and whisker plots: en.wikipedia.org/wiki/Box_plot
    – Julia Silge StaffMod
    Commented Jun 18, 2019 at 18:16
  • 47
    Agreed with @ErikA - "people who get a Tumbleweed badge are much less likely to ask a question ever again." seems to say that the badge itself makes you stop asking questions. But I don't expect this trend to change when the badge is removed. I understand the point, but that data is not, in itself, a reason to remove the badge. Even though it is presented as such.
    – KevinG
    Commented Jun 18, 2019 at 18:26
  • 31
    Also, and I might be nitpicking here, I don't think the emoji are appropriate, given that they'd be edited out for being noisy on any other post.
    – grooveplex
    Commented Jun 18, 2019 at 18:44
  • 6
    @grooveplex, it seems to be the norm for the usual update posts... meta.stackoverflow.com/q/384981/4099593 I don't think we've ever had a discussion on whether emojis are acceptable on these posts. (I personally don't mind either way)
    – Bhargav Rao Mod
    Commented Jun 18, 2019 at 19:11
  • 8
    @ErikA: It's very likely that the lack of response when asking a question (whether it was good or bad) is the driving reason people don't ask more after the Tumbleweed. Some people aren't coming back after a week to even see it. Being ignored is a prime reason to stop asking. But at least one reason for having the badge is to be an encouragement. "Get back on that horse, pardner" sort of thing. The data suggests it utterly failed at this mission, so why keep a useless badge?
    – Jon Ericson Staff
    Commented Jun 18, 2019 at 19:53
  • 16
    @Jon I wouldn't know if that's true. It might turn out that the badge did work, and that ditching it makes the problem considerably worse. Imo the data is uninterpretable with regards to the efficacy of the badge. It only shows we have a large proportion of questions that get ignored, and that this is highly discouraging to the users asking them. The badges just distract from the real discussion to be had on how to address that problem (currently no ideas on that though).
    – Erik A
    Commented Jun 18, 2019 at 19:58
  • 14
    This question is being discussed on meta.
    – Script47
    Commented Jun 19, 2019 at 12:37
  • 31
    Seems like a classic case of mistaking correlation for causation. Commented Jun 19, 2019 at 17:36
  • 16
    So removing a badge (that's not even problematic imo) and adding more badges is apparently more important than addressing the backlog of bugs and highly-requested features that have been sitting around for 10+ years... sigh. seems like a classic case of "solving" problems that aren't even problems to begin with. Or providing solutions that aren't solving anything. One of those two.
    – pushkin
    Commented Jun 20, 2019 at 14:36
  • 8
    It's fashionable nowadays to talk about being inclusive and hope that people just stop thinking and believe your reasoning. However, your reasoning is oversimplified. Only if you had provided statistical evidence of tumbleweed-worthy questions where that feature is disabled, and shown how they differ from questions that actually got the badge, you could have established an argument supporting the purported causality. Unfortunately, 90% of people will not notice. Good job. Do us a favor, collect some data about tumbleweed-worthy questions once the badge is gone and present them in the same way. Commented Jun 24, 2019 at 6:37

10 Answers 10

144

I've spent the last few years just looking at Tumbleweed badges. The badge was useful to me for few reasons. A couple of reasons why they became tumbleweed out of the many others (like narrow topics, non availability of experts, and so on) were:

  1. Many of the tumbleweed questions were ignored just because the users hadn't tagged them properly. The Tumbleweed badge page listed these ones.

  2. Many of the tumbleweed questions were ignored because they were asked at the wrong time, say on a Saturday or Sunday.

Once I found such a question, I used to either:

  • edit them into shape and add the right tag, so that they received proper answers.
  • vote to close them, if they were completely off-topic for the site.

This gave me a small chance to review before they were roombad as "AbandonedQuestions"

I had mentioned this even in Teacher's Lounge (I can't link it here) :

Tumbleweed: to find bad questions that escaped the communities eyes because of wrong tags.

I can somewhat understand that the Tumbleweed wasn't a positive badge, but I'd certainly not term it as "Unwelcoming" or "Unfriendly". It seems to be a stretch to call it that.

All that said, there were a few advantages that Tumbleweed had, which had become a part of my workflow on the site. Would it be possible to get a 10k tools page that lists the questions that would have received the badge?

14
  • 42
    "I can somewhat understand that the Tumbleweed wasn't a positive badge, but I'd certainly not term it as "Unwelcoming" or "Unfriendly". It seems to be a stretch to call it that." Think of "tumbleweed" as the equivalent of "Achievement unlocked! You Were Ignored!", and it shouldn't be too hard to see why someone might take it as a negative. Commented Jun 18, 2019 at 18:29
  • 41
    It's more like "you asked a question which we can't answer, because we don't know" or "your question was ignored because you didn't use the proper tags" ... The issue is that people don't exactly understand the positive/neutral aspect of the badge.
    – Bhargav Rao Mod
    Commented Jun 18, 2019 at 18:34
  • 8
    "The issue is that people don't exactly understand the positive/neutral aspect of the badge." Um, yes. That's why it's unfriendly; because the negative aspect of it is more easily seen and more obvious to the reader than the positive interpretation. The fact that something could be interpreted in a neutral or positive way does not mean that the negative interpretation is not the more obvious or prevalent one. If people keep "mis"interpreting something, at some point, you have to decide whether it's a problem with them or a problem with the "something". Commented Jun 18, 2019 at 18:46
  • 24
    I still don't think that it merits the term "unfriendly" @Nicol. Unfriendly is when you're going out of the limits to say something impolite or that which is unpleasant according professional standards. Informing a user that their question didn't receive any answers or comments is more like an update. The user themselves can use it in a positive way to either 1. Edit the post and clarify the issues 2. Add a bounty to get more views 3. (If they have 20 rep) Contact users in chat and find out why they are not getting answers.
    – Bhargav Rao Mod
    Commented Jun 18, 2019 at 19:09
  • 10
    I appreciate the utility of the badge to easily identify questions that might be mistagged and such. I'm not sure this is the feature I'd design for that particular activity, however. In particular, 7 days is really too long a time. 3 is more like it. I was able to construct a search that gets all of the criteria except comments. I wonder if this would be a job for the review queues?
    – Jon Ericson Staff
    Commented Jun 18, 2019 at 21:03
  • 7
    A review queue wouldn't be a great idea, @Jon. The usage isn't of that great importance as compared to the other queues (H&I excluded). A 10k took should be fine, something like the New Answers To Old questions tool.
    – Bhargav Rao Mod
    Commented Jun 18, 2019 at 22:15
  • 3
    I used to have a query a little bit like this, more than a few days old, no answers or comments. I like those because you're not likely to have someone else working on it at the same time you are.
    – Elin
    Commented Jun 18, 2019 at 22:33
  • 18
    This answer is a far better comment on Tumbleweed than the official SO response. Commented Jun 19, 2019 at 6:45
  • 3
    I actually would have liked a tumbleweed badge (although obviously I'm not going to craft a question specifically to get it). So I'm a bit miffed that I won't get it now :(
    – user5940189
    Commented Jun 19, 2019 at 8:33
  • 7
    @Orangesandlemons Now that they're discontinued the value will skyrocket. Especially if left in the original packaging. Commented Jun 19, 2019 at 13:10
  • 9
    @NicolBolas I find the badge name mildly entertaining and actually endearing (imagining a dry tumbleweed tumbling through the desert). My .02 is that people are too sensitive if they're being upset by it. Commented Jun 19, 2019 at 16:25
  • 2
    @BhargavRao Nicol means "unfriendly" in the same sense SO means "not welcoming." That standard is far beyond simply being professional and polite. It means everything you say must be perceived as positive and encouraging (unless the recipient has been deemed "unwelcoming"). I agree with you that the badge is not problematic, but the value system of SO has changed dramatically as a result of the Welcoming push.
    – jpmc26
    Commented Jun 21, 2019 at 1:18
  • 9
    That exactly is what I'm saying too, @jpmc26. Stack Overflow terming it "unfriendly" is incorrect. It is not "not welcoming" either. Stack Overflow has termed it as such just to show to others that they're working on the welcoming initiative, without realizing that they've effectively caused harm to people who were using it to curate Stack Overflow's content. (This certainly is one of those straws that can break a camel's back)
    – Bhargav Rao Mod
    Commented Jun 21, 2019 at 1:28
  • The name "tumbleweed" is very US cultural centric. How is a Mandarin or Swahili speaking user supposed to know what a tumbleweed is, or what its cultural implications are? [cut to shot of tumbleweeds blowing down an empty street in a 1950s Hollywood Western] This badge is another instance of Stack Overflow being unwelcoming.
    – markgz
    Commented Jun 21, 2019 at 18:15
46

As someone who works mostly in a mature technology (ASP.NET and C#), I'm lucky enough to find that the vast majority of high-impact general questions have already been asked and very answered well.

Therefore, when I ask a question, it tends to be very narrow and very specific, often in a fairly niche area with interest to a small audience. Those questions naturally get low views, low upvotes and therefore don't earn many badges.

It seems to me that a question's 'worth' isn't necessarily proportional to its viewing figures, but many of the badges and the whole upvoting system somewhat contradict that(?).

You mention new users in the article and I often think it must be much harder for your average new programmer to start getting rep and badges nowadays than it was when there was less content on SO. Not many new developers are working on cutting-edge technology or really pushing the coding envelope, so their opportunities to ask high-scoring questions, or find easier unanswered questions to answer are low.

Some badges that reward new users for using the site or participating in other ways could be useful I think. Plus maybe a badge for asking x number or questions that followed the template, formatted the code properly, etc. and generally asked questions that were readable, just to help new people get started.

Actually, thinking about this, I wonder if the review queues could be used here, maybe by allowing reviewers to vote for a user to receive a 'well asked question' badge or by looking at the stats for how many of a new user's questions require edits in review, or something like that.

12
  • 2
    Plus maybe a badge for asking x numer of questions that followed the template, formatted the code properly etc - in other words, well-received questions? Sounds like Curious, Inquisitive and Socratic badges. Commented Jun 18, 2019 at 18:21
  • @MathieuGuindon that's interesting, what's the actual criteria for the 'well-received' bit of those badges?
    – tomRedox
    Commented Jun 18, 2019 at 18:45
  • No idea. Presumably, a non-zero positive net score. Commented Jun 18, 2019 at 18:48
  • 3
    @MathieuGuindon, so the problem I see there then is that if say 1/100 people actually upvote, then the person's question potentially needs a 100 views to get that vote. If the question is niche and isn't answered then there are no points earned there either and could easily take a year to get that many views, so the user doesn't the quick reinforcement of a question well asked that the badge theoretically rewards
    – tomRedox
    Commented Jun 18, 2019 at 20:09
  • 12
    If a question needs 100 views to get a single upvote, maybe the problem is with the community, rather than the question? Why aren't any of these viewers voting if the question is well-formed and adheres to all posting guidelines? Commented Jun 18, 2019 at 20:13
  • 11
    @MathieuGuindon I definitely agree with that, i never understand why good questions don't get more upvotes. I think the Electorate badge is valuable in that respect. Perhaps that's another badge area that needs investigating, some way to incentivise people to upvote good questions!
    – tomRedox
    Commented Jun 18, 2019 at 20:18
  • 2
    @MathieuGuindon: Correct. The idea behind these badges is to acknowledge people who are skilled at asking. So the criteria is pretty rigorous. We don't want to accidently reward unskilled askers!
    – Jon Ericson Staff
    Commented Jun 18, 2019 at 22:01
  • 1
    @JonEricson Ah, that's what the full criteria are! Is there any way to get that linked from the badge description pages so people can easily get a bit more background on these badges if they are interested?
    – tomRedox
    Commented Jun 18, 2019 at 22:05
  • @MathieuGuindon Because voting up or down is a somehwat significant action for me. I'll upvote a question if the topic resonates with me, perhaps forces me to hack at an interesting problem. "Clearly written" is insufficient for me. Frequently on other sites on the network I would upvote a question only after writing an answer. Sure, maybe some people can vote as if they're liking friends' posts on facebook, but I guess there aren't that many such people.
    – LLlAMnYP
    Commented Jun 19, 2019 at 9:48
  • @JonEricson Consider this a +1 request to link your MSE post somewhere for the badge description pages.
    – TylerH
    Commented Jun 21, 2019 at 14:12
  • Khan Academy flashy colors/shapes are more enticing badges than just text-badges with boring, monochromatic circles imho.
    – Dave Liu
    Commented Jun 24, 2019 at 19:45
  • I think this is what bounties are supposed to counter-act; if it's important to you, you can burn 100 points to boost the question and hopefully get an answer.
    – user9903
    Commented Jun 25, 2019 at 15:10
39

While I'm somewhat happy to see Tumbleweed gone, I have two issues with the findings here:

  • The root cause of why the question wasn't seen for so long, or why there was such minimal activity on the question isn't being addressed; we're only rewarding those who actually put effort into questions which might be salvageable rather than helping those questions get in front of more experts.

  • This is yet another backfill towards the Welcoming initiative, and even if it doesn't seem to be the case that it was meant to be, I find it hard to convince myself otherwise.

It's good that we're actively rewarding those who go around fixing questions. Problem is, nothing presented - not even the badges - motivate a solution for questions that got themselves into that state in the first place. Anecdotally, more people wanting this badge implies more experts looking at the question, but that's highly unlikely, especially if you're talking about a niche technology or a niche subject matter. The Tumbleweed badge should be retired, but let's not for an instant think that we've Solved the Problemโ„ข by any stretch of the imagination.

In the blog post (but definitely not here), I noticed that the Reversal badge has also been retired. While the data is somewhat in favor of this, I don't agree with this removal since it was never geared towards the OP of a question, but more to the answerer (e.g. identifying the pearl in the middle of sand). Not a fan.

11
  • 1
    This is certainly not a solution. The new badges are more about recognition of a rare feat than actually motivational. Getting eyes on ignored, but worthy questions is a tricky problem unless you don't mind exposing more ignored and unworthy questions along the way. As Bhargav points out, mistagging your question is a good way to get it ignored. Perhaps encouraging people who have asked and not heard anything in a few days to consider editing their question would help.
    – Jon Ericson Staff
    Commented Jun 18, 2019 at 21:52
  • 5
    The main reason for removing Reversal is not the asker at all. Instead it's a sign the badge isn't rewarding useful behavior when so many of the answers that get it are deleted. If the community doesn't think a question is worth keeping for the sake of one of the answers, that's a good sign we shouldn't be handing out a badge for it. It hasn't made sense for quite some time.
    – Jon Ericson Staff
    Commented Jun 18, 2019 at 22:06
  • 2
    "It's good that we're actively rewarding those who go around fixing questions" hahahaha, poor Makoto. No, they aren't rewarding someone for fixing questions. Editing isn't required at all to get this badge. There was a hat that did that (Wokong I think) but that's not what this one is about. This one is about someone having a negative scored question, getting an answer and then get positive score.
    – Braiam
    Commented Jun 18, 2019 at 22:37
  • 6
    @Braiam: The badge set for fixing questions and answering them is Explainer โ†’Refiner โ†’ Illuminator. These new badges can be earned without editing the question, but you are much more likely to get them if you take some time to fix any problems with the question too. Unlike Reversal, which we retired, most of these badges aren't going to answers that get deleted later on.
    – Jon Ericson Staff
    Commented Jun 19, 2019 at 0:35
  • 1
    @JonEricson I am corecting Makoto misguided impression that this badges foment fixing questions. It doesn't. Proof of that is that these two already surpassed the silver/gold of the trio.
    – Braiam
    Commented Jun 19, 2019 at 1:05
  • 3
    @Braiam: The new badges are earned by getting lucky on one question. (Though it helps to write a great answer too.) Refiner and Illuminator are earned by purposefully going out of your way to edit 50 and 500 questions. The more useful comparison is Explainer and there are about 7 times as many of those. To the extent badges motivate people at all, it seems likely this prize motivates editing.
    – Jon Ericson Staff
    Commented Jun 19, 2019 at 2:08
  • 2
    @JonEricson: I think with your justification on removing Reversal, you mฬถaฬถyฬถ ฬถhฬถaฬถvฬถeฬถ ฬถiฬถnฬถaฬถdฬถvฬถeฬถrฬถtฬถeฬถnฬถtฬถlฬถyฬถ struck a nerve. I'm satisfied with the explanation provided, but the inconsistency honestly hurts.
    – Makoto
    Commented Jun 19, 2019 at 2:15
  • 1
    @Braiam: I'd kinda argue that if a question which has been downvoted significantly enough (e.g. some question which was beyond -3 or -5) would have to have some kind of editing work done to actually bring it to standards. No, the badge doesn't actually indicate that the awardee do anything more than answer it. I'd hope that the system would encourage said candidate to actually care.
    – Makoto
    Commented Jun 19, 2019 at 2:17
  • @Makoto: Do you mean arguing for not rewarding a good answer to a bad question when it comes to Reversal while arguing against deleting a question because of a great answer? I can see that. I think the underlying consistency is that I tend to trust voters to evaluate content.
    – Jon Ericson Staff
    Commented Jun 19, 2019 at 2:28
  • 6
    @JonEricson "Perhaps encouraging people who have asked and not heard anything in a few days to consider editing their question would help." That's exactly what I thought when reading the blog post. If a badge as consolation prize is bad, we should ask ourselves what would be good. Maybe an automated message, maybe an additional bump to the questions queue. Commented Jun 19, 2019 at 7:44
  • 2
    Sadly, Makoto, when a question gets an answer, most people shy away from editing. Now, I wonder why was that? BTW, I've said before that having a stellar answer with a lackluster question is a disrespect towards the answer, because that knowledge is very likely to not serve the future readers due the question being either difficult to find or difficult to understand the specific context. It also tends to allow other low quality answers to be posted.
    – Braiam
    Commented Jun 19, 2019 at 9:05
33

It seems to me like the analysis fell for the correlation-implies-causation fallacy.

we know that many users over the years have experienced earning this badge negatively. ๐Ÿ˜”

I'm not sure we can know that, unless they've explicitly stated it as such, and if so, then this was not presented in the post here. It seems to me that the only thing we can really know is that the question got no interaction, and while I think it's reasonable to infer that people wouldn't be too happy about their question being ignored, some people may not necessarily care... but then again, it seems like this would be the case regardless of the badge.

Also, how did you filter out people who create one-time accounts for quick questions, with no plans of ever coming back afterwards?

We can use data to explore what the impact of this badge is, compared to a similar interaction

I think this is where the correlation-implies-causation fallacy is made clear.

While I can clearly see there's a correlation between some questions getting the badge and a decreased likelihood of posting another question, suggesting that the reason some people are less likely to ask more questions is because of the badge is where I think the logical fallacy is.

I'd think it's likely there will be little to no change by removing the badge alone. I think it's more likely that the users don't follow up due to a lack of feedback and/or interaction from the community; no one likes to get ignored, after all. (And that's if they planned to return at all.)

The badge is more likely to simply be identifying those cases, rather than causing them in the first place, so, if the only "fix" here is removing the badge, I'm not sure what I'd be expecting to change, if anything changes at all.

My thought is that, if it were possible to compare those who got the badge with others that would've otherwise qualified for the badge, but don't have it, the results would probably have been the same...

That's the point we're making, not that the badge causes lower activity

My understanding from the wording in the original post seems to suggest otherwise, hence my reply above. Perhaps it should be reworded to clarify what you included in the comments section, but that seems to be the underlying assumption in the original post, IMHO.

40
  • 2
    So, pretty sure nobody thinks Tumbleweed caused questions to be unanswered. The big problem is that it doesn't seem to help much either. Years ago, Bill suggested we feed them into a review queue so that we could do something useful with this information... But as Jon & Julie both noted, this falls far away from the original goal of the badge. In 2009, it was actually really hard to post a question on SO and not get some sort of response - but that's neither true nor useful now.
    – Shog9
    Commented Jun 20, 2019 at 15:33
  • 13
    @Shog9 "So, pretty sure nobody thinks Tumbleweed caused questions to be unanswered." That isn't what this post says. It says that there's no evidence to support that the Tumbleweed badge contributes to not asking more questions. In other words, it's debunking the claim that this data serves as evidence that the badge is "unfriendly," which the blog claims in its title. Stop misrepresenting what people write. Your claim that "the big problem is that it doesn't seem to help much either" is wrong, too; the blog post explicitly says it's being removed because of concerns that it's "unwelcoming."
    – jpmc26
    Commented Jun 20, 2019 at 16:49
  • 3
    That's some irony right there, @jpmc26. Lemme spell it out: we've known for years that getting no response is a huge deterrent. The stated rationale for Tumbleweed was that it was a consolation prize - effectively a response from the system when no one else would respond. But, Julia's data suggests there's no evidence that this is true, no sign that askers are in any tangible way "consoled" by the badge. To fall victim to the stated fallacy, we'd have to be assuming somewhere that the badge itself triggered the lack of response, but as I said: nobody is making that argument.
    – Shog9
    Commented Jun 20, 2019 at 20:17
  • 4
    @Shog9 Hi. I understand what you're saying in your comment, but that seems very different from the wording used in the quote from the OP, i.e. "We can use data to explore what the impact of this badge is, compared to a similar interaction" (bold emphasis mine). It definitely sounds like the OP is attributing those things to the badge.
    – code_dredd
    Commented Jun 20, 2019 at 20:30
  • 3
    @Shog9 This answer is explicit about what behavior the blog post is claiming the badge induces: "...suggesting that the reason some people are less likely to ask more questions is because of the badge is where I think the logical fallacy is." Your comments are talking about something different. You are either extremely confused or willfully ignoring what the answer says. I can't say which, but your responses here aren't in good faith either way because you're not re-examining your understanding.
    – jpmc26
    Commented Jun 20, 2019 at 20:32
  • 5
    @Shog9 No, you might want to. It says, "So the Tumbleweed badge isn't helping, and it may be hurting." It also says, "...we know that many users over the years have experienced earning this badge negatively." Furthermore, the blog post's title explicitly calls it "unfriendly" and thus implicitly blames it for decreased participation (since the basis of the "welcoming" push is that "unwelcoming" things decrease participation). dredd's entire point is that these claims about the badge are supported by only a logical fallacy.
    – jpmc26
    Commented Jun 20, 2019 at 20:44
  • 2
    Go read my last comment to code_dredd, @jpmc26: we've had lots of complaints about Tumbleweed over the years; that's not the news here, go search MSE if you don't believe it. The saving grace of the badge was supposed to be that it was better than nothing - but it turns out that's not true either. So we know there's negatives and can't find positives: do the math.
    – Shog9
    Commented Jun 20, 2019 at 21:33
  • 15
    @Shog9 If you have evidence that so many users are upset by receiving it, then present it. The blog post asserts that the badge is actively harmful. It doesn't assert that it's unhelpful and you don't want to bother confirming whether it's harmful or not. Those are two different things. I don't really care about the Tumbleweed badge; it's not a big deal. I care about invalid reasoning and data analysis related to the "Welcoming" push and social justice ideas. So defenses of removing the badge don't matter. The question is whether you all analyzed correctly, and the evidence shows you didn't.
    – jpmc26
    Commented Jun 20, 2019 at 22:28
  • 5
    @Shog9 Did you bother checking the results? Because all but 4 of those (by my manual count) have the tag. So how would that show a bunch of posts I missed? Also the "mark of shame" one I linked above doesn't have the tag, so you can see I did some checking outside it. "Which was the point of this analysis." I don't care what the point of the analysis originally was. I care what's being claimed based on it now that it's done. It does matter if data analysis is going to be misinterpreted and abused. Testing assumptions is fine; using data to feed confirmation bias is not.
    – jpmc26
    Commented Jun 21, 2019 at 0:58
  • 5
    @Shog9 I'm certainly interested in reading more into it. But you haven't shown me anything I haven't already been looking at. If you're seeing something I missed, tell me what I should be looking at. Are there some specific comments that demonstrated your point I might have missed on those questions? Are there specific answers you want me to read? I've already sunk significant time looking for evidence of your claim. I care that the case has not been made. It's not reasonable to ask me to build your entire case for you. You should care because you have not earned the trust of your users.
    – jpmc26
    Commented Jun 21, 2019 at 1:05
  • 4
    @Shog9 (1/3): "goal here was to figure out what effect the badge actually had - we already knew why some folks disliked it" If "you already knew some folks disliked it", then why was there a "goal to figure out what effect the badge actually had" in the first place? How did you already know before this? What sample size of the actual SO population was claiming to not like it (i.e. was it even statistically significant)? This is starting to come across as a potential example of confirmation bias, in addition to my other comments.
    – code_dredd
    Commented Jun 21, 2019 at 1:24
  • 5
    @Shog9 (2/3): "we didn't know if there was any measurable impact; turns out, there wasn't." So, if there was no "measurable impact" regarding the badge, then on what empirical and logical basis is it really asserted that the badge is being taken "negatively" by those who receive it and that this is the reason why these users don't ask more questions, as opposed to it being the lack of interaction/feedback the badge merely points out?
    – code_dredd
    Commented Jun 21, 2019 at 1:24
  • 6
    @Shog9 (3/3): Lastly, why is users not asking questions assumed to be a bad thing? Says who? There're a lot of half-assed questions on the site that get deleted, so I'm not sure trying to prioritize quantity over quality is a good set of priorities to apply here in SO; we literally have the rest of the Internet at large for that, right?
    – code_dredd
    Commented Jun 21, 2019 at 1:24
  • 5
    @Shog9 "Your entire hangup here seems to be centered on the idea that there's data demonstrating unfriendliness..." My hang up is on the process by which a decision was made. I am interested in reading more into this particular issue's data only because it would demonstrate how you arrived at the conclusion. There's currently no evidence that anyone did justify the premise, and the blog post asserts that the data presented justifies it, in the quote I've posted here in these comments and in my answer. If you're going to justify things based on data, you need to do it properly.
    – jpmc26
    Commented Jun 21, 2019 at 3:32
  • 5
    @Shog9 This question is titled "data science." Telling you that you need to verify and justify your assumptions before you can make a scientific claim is hardly a tangent. It's just good science. Just assuming it and saying "you have to grant us our unjustified assumption" would never pass academic review or anything even close to it. Calling it rude to point that out and say that the information needs to be added is nothing more than deflection from an important problem with how you're using data.
    – jpmc26
    Commented Jun 21, 2019 at 18:01
30

The Tumbleweed badge is awarded to questions with zero score, no answers, no comments, and low views for a week. It's an incredibly common badge, the 9th most common badge awarded over the past year.

Better to have neither downvotes nor snarky comments, and earn a badge, than the alternatives; but low views isn't good. The badge tells me "here's a second chance to improve your question, bump it, and you just got a badge". It's a useful reminder, and a small bonus.

It doesn't tell me to dry up and blow away.

Although this badge was originally intended to be somewhat humorous and as a consolation prize, we know that many users over the years have experienced earning this badge negatively.

Yes, well sometimes the fault is on the other end.

Experience life positively or negatively - but learn from it!

If one often experiences things that aren't sugar coated in a negative manner and learns nothing, or to go away to the next negative reinforcement, that's on them. Over coddling is just as bad as seeming unwelcoming - the catch-all phrase for everyone with energy to negative-blog but none to positive-learn.

You write this up like it's a going away present, or go away sign. How many were active participants and upon receiving this so-called indignity proceeded to rage-quit; writing a "You're all doomed" Meta-goodbye, zero has been my observation.

Established users don't take the badge that way, and newcomers ought to read up on what the badge means and improve their question (or even delete it, since they've figured it out by now).

I'll miss you Tumbleweed, now we shall swelter in the Rain Forest.

26

So the Tumbleweed badge seems to be a worse consolation prize than if the system just gave out a downvote.

I don't see how presented data lead to that conclusion. There are multiple potential reasons:

  • Maybe user seeing lack of interest in his question didn't consider Stack Overflow as a good place for him (with or without badge).
  • Maybe user realized that his question is really specific (pointed earlier in https://meta.stackoverflow.com/a/386190/5226328) and should look for answers somewhere else.
  • Maybe user just had one question that was quite complicated and no one were able to help.

My 'Tumbleweed' question was exactly like reason 3. I had to test something about BT on Mac and asked this question (https://stackoverflow.com/questions/49147464/bluetooth-explorer-cbcentralmanager-state-unsupported - now deleted). I didn't have further questions in the matter at the time. I wasn't expecting answers after 4-5 days and Tumbleweed was a nice surprise in my 'Inbox' - I didn't blame badge or SO that question didn't get attention.

I don't want to argue retirement of Tumbleweed badge (I don't have a strong opinion). But I don't agree with this interpretation. I think that Tumbleweed didn't have a major impact on future questions.

Maybe to test it further it would be worth to look at time form asking first to asking a second question. Tumbleweed was awarded after 7 days. Were users posting more questions before getting this badge (in respect of other groups)? Were there a point in time, when probability of second question dropped (at maybe was it about 7 days after asking non popular question)?

There is also another important question: How does Tumbleweed badge affected question with 2 or more actions. Were users that got Tumbleweed and answer were posting less questions, then users that got only answer? How it interacted with upvotes, comments or downvotes? Maybe giving badge after 7 days (when new users usually need help ASAP) doesn't matter in the big picture.

25

I'm going to respond to a comment below the post because I think it demonstrates all the reasons I'm frustrated.

We have enough user feedback to know that the badge isn't 100% irrelevant (many people feel negatively when they get this badge),

You have shown zero data to support that the badge is perceived negatively by recipients. Proper data analysis does not hide information and then demand that others trust your findings, nor does it accept premises that are both unverified and plausibly false.

...and this analysis shows that the badge isn't helping compared to other kinds of interactions.

Users asking fewer questions later isn't an indicator that there's a problem. (You use of the word "helping" implies that there's a problem.) There may be a selection bias with Tumbleweed, where questions that earn them tend to be more difficult than the average question. This could be merely an indication that these askers are able to find answers to more basic questions without posting a new one. That would be an indicator of success, and it would be an indicator of failure that other users were unable to find an answer to trivial questions that almost anyone could answer. You've demonstrated no analysis to rule out other hypotheses like these. It's not better if we experience a greater influx of questions that are trivial and add no long term value to the site, yet you have deemed it a problem if people ask fewer questions.

That's the point we're making, not that the badge causes lower activity.

I'm sorry, but this is just either complete lack of awareness of the contents of the blog post or a bold faced a lie. The post explicitly contradicts this claim:

In summary, the Tumbleweed needlessly discourages askers who might very well be asking good and interesting questions.

And this doesn't even get into the implications of labeling the badge "unfriendly" in the title.

Making a statement this blatantly wrong about a short blog post makes trusting your unpresented analysis of user feedback difficult for me.

I don't really care about the Tumbleweed badge; it's an incredibly minor issue as far as I'm concerned. I do care about decisions being made based on half-baked data analysis and about whether SO employees can accurately represent each others' statements. You're not the only employee getting this wrong. As a user, when the employees are telling me that no one is saying the thing that's right there in the blog post and even trying to argue against it being there, I feel like I'm being gaslighted. How am I supposed to trust you or any other SO employee? Step up your game: be honest, accurate, and thorough. That's the only way users are going to trust you and your analyses.

12

This post needs to answer many questions before reaching this kind of conclusions:

  • Did the people that retrieved the tumbleweed badge actually visited the site to see that they got that badge?
  • If the tumbleweed badge is considered "negative feedback", it's still considered feedback. How does that play with the knowledge that this kind feedback also promote further interactions with the site? (also pointed out in this very post)
  • How was established the causal link between getting the badge and not further interactions with the site (this ties up with the first question, but this is looking at the methodology)?
  • If the badge is a consolation prize, how is the user made aware that they got it?
  • After getting the tumbleweed badge, how likely are users to get feedback/votes vs those that got feedback earlier that made them ineligible? (as pointed out on this answer, this questions may get another look just because they got the badge)
2
  • 3
    On the third bullet point. It's apparently not so much that it caused user attrition, but that it utterly failed at the intended goal of user retention. Not so much causing harm as failing to do it's job.
    – user4639281
    Commented Jun 23, 2019 at 1:16
  • @TinyGiant still, there's no evidence that it causes any negative outcomes, and at least we know about one positive one (last bullet point).
    – Braiam
    Commented Jun 23, 2019 at 1:40
10

I very much doubt that the badge alone has such a negative effect. It would be very interesting to look at the difference between people who had asked a none-Tumbleweed question before receiving the badge and those whose first question was a Tumbleweed. I personally did receive the badge after quite a few other well-received questions, so for me it was rather a positive experience. However, I do see that it could be received negatively if you receive it for the first question you ever ask.

I do agree that any feedback is better than none. Yet, after having received feedback on at least one question before, it should be clear to any user that it is not that they generally ask such stupid questions that no one wants to help them, but that this specific question should probably improved in some way. Consequently, you could reward the badge only to users who have received any kind of feedback on another question before.

Apart from that, I agree with the sentiment in most of the other answers, in that it is really a minor issue, if one at all.

1

The tumbleweed badge

Although this badge was originally intended to be somewhat humorous and as a consolation prize, we know that many users over the years have experienced earning this badge negatively

This is correct.

I also didn't understand what it meant, but soon figured it out by reading the tag wiki and googling tumbleweeds, despite growing up on John Wayne movies and other black and white westerns that did indeed feature tumbleweeds blowing across the screen.

Has earning any particular badge stood out to you, either positively or negatively?

Now people can take this whatever way they want, but it was my experience and this is how I felt. I struggled immensely with trying to understand the culture of this site. Call me a sensitive nelly and yes I am sensitive.

When I first joined Stack Overflow, on an account prior to this one. I was awarded the tumble weed badge multiple times. It was depressing. As a new programmer floundering in programming and on the site it was like being told "your question is worthless". It didn't help my confidence at all.

I actually posted on meta (before the split) and complained about it. I can't find the post, it must be deleted and wasn't re-associated with my new account. We're not sure which meta site it is on.If we can find it, I'll link it in.

I later earned a tumbleweed badge on this account, but by this time I had a lot of successful posts behind me, so it didn't bother me.

Other badges

What have been your experiences with badges on Stack Overflow?

I must confess, I enjoy collecting badges. I have gone out of my way at times to aim for and collect a particular badge. The latest one on meta is the gold discussion tag badge. That one takes a while as it requires answering so many questions.

Moving forward

Honestly, a badge to encourage people is better for new people. For old timers it can be a "hey what the heck are you doing with this backsliding question" badge (that is totally tongue in cheek). People who have been using the site successfully tend to enjoy the culture and humour of the site more than someone flailing around trying to fit in and figuring out why their questions are not working.

Given the initial failing user retention it's understandable the business needs to shore up the parts of the site that discourage people from staying. Whether we like it or not it's a business and they need to make money. It's changed the site and it's something that is difficult to accept for many people who have been here a long time, myself included. Some changes have been good, but still it's change and it used to be our site and now it's less so.

It's one of those things and it's certainly not the fault of the employees. They're between a rock and hard place. They're given a job description and have to come on here and report to the community. Let's try and be supportive of them. We don't have to agree with the implementation, but we can at least acknowledge, behind the keyboard is a human being, going about their business, doing their job.

4
  • 1
    almost there with the discussion gold badge!
    – Magisch
    Commented Jun 25, 2019 at 13:11
  • @Magisch ikr! how are you? Ping me if you need anything :)
    – user3956566
    Commented Jun 25, 2019 at 17:15
  • I am not surprised this question isn't upvoted. It actually confirms what the organisation says - I am one consumer who doesn't like the badge and yet people don't seem tolerant of that. Or is it me asking people to be nice to the employees.
    – user3956566
    Commented Jun 26, 2019 at 5:50
  • I reckon it's a reflection of the current state of trust on meta. Anything an employee posts will get extra scrutiny, and so will anything that defends the conclusions of an employee's posts. The Post itself has -43 / +100 which is uncharacteristic for data science posts, especially on something as minor as a badge retiring.
    – Magisch
    Commented Jun 26, 2019 at 6:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .