I've had some spare time on my hands recently, so I've been looking at the Tumbleweed badge and wondering if there's anything we can do about this growing list of mostly unanswered questions. The badge has been awarded well over half a million times now, which is a pretty significant portion of the questions asked in Stack Overflow's history. It's a lot of questions that just get ignored.
Shog9 and Jarrod Dixon were kind enough to send me a list of all of the questions that had been awarded the Tumbleweed badge (up to about a week and a half ago). The data includes QuestionId, Title, Awarded, UserId, DisplayName, Score, ViewCount, Tags, AnswerCount, LastEditDate, CreationDate, ClosedDate, DeletionDate, and FirstAnswerDate. I spent some time poking and prodding at the list, and wanted to share the results and see if we could come up with some ways to reduce the number of these questions that go unanswered.
Proportion of questions with no answers: 73.8% (390677 / 529639)
Proportion of questions that did get at least one answer: 28.2% (149479 / 529639)
If you're wondering why those numbers don't add up to 100%, it's because there are two ways of looking at whether a question is answered or not. Count the number of answers, or look at when the first answer was posted. I used both methods to check for consistency, and of course they weren't. The discrepancy is due to a little over 10,000 questions that did receive an answer that was later deleted. Either way you look at it, the answered rate for Tumbleweeds is well below the site average of 74%.
Of those that did get answered, 75.8% (113292) received only one answer.
- Proportion of closed questions: 1.6% (8320 / 529639)
That's not a typo. Only 1.6% of these questions get closed. That was shocking until I considered that this is the set of questions we ignore. It makes sense that they don't get enough views to be closed by the community.
- Proportion of deleted questions: 47.9% (253540 / 529639)
That's not a typo, either. Far more of these questions get deleted than closed because there's a script that runs that deletes questions with zero score, no answers, and low views after one year. It makes sense that a lot of these questions would fit that criteria.
So far we've identified a problem and a partial solution. These questions aren't getting enough attention to be answered or closed by the community, so a script is deleting them after they sit around for a year. Can we do better than that? Let's take a closer look at the questions that do get at least one answer to see if we can identify what sets them apart.
First, I want to look at how long it's taking (in days) for Tumbleweeds to get an answer.
> quantile(time_to_answer, c(.1, .5, .90, .95))
10% 50% 90% 95%
7.442454 48.278021 281.156600 357.092005
At the low end of the scale we can see that only about 10% of these questions are getting answers by the time they get the Tumbleweed badge a week later. Half of them are getting answered in 48 days, and 95% of those that do get answered do so in just under a year. (This should not be surprising, since we already saw that most of the ones that don't get answered are automatically getting deleted after a year. This pretty much guarantees that the majority that are answered will be answered within one year. It's almost a definition.)
Next, I want to look at how many questions are getting edited, to see if that makes a difference in whether or not they get answered.
- Proportion of all Tumbleweeds that get edited: 35.8% (189825 / 529639)
That was a bit of a surprise, considering only 1.6% of them get closed. However, it does take five votes to close a question, and the author can edit it themselves.
- Proportion of answered Tumbleweeds that were edited: 42.4% (63399 / 149479)
- Proportion of unanswered Tumbleweeds that were edited: 33.3% (126426 / 380160)
9% isn't as big a difference as I would expect, but at least it's in the right direction. There does appear to be a positive correlation between a question getting edited and it getting answered. This makes me wonder what effect there is based on when a post is edited. I already have the edited posts split into those that were answered and those that weren't, so I can look at the differences between when the questions were posted and when they were last edited to see what difference time of edit makes.
> quantile(post_to_edit_ans, c(.25, .5, .75, .95))
25% 50% 75% 95%
0.0252662 1.5239352 96.1937789 725.6703183
> quantile(post_to_edit_unans, c(.25, .5, .75, .95))
25% 50% 75% 95%
0.01089120 0.06153935 1.26274016 183.80472512
Here we see a huge difference. Posts that were answered (first group) were edited over a much longer span of time than those that weren't answered. 50% of the answered questions were last edited 1.5 days after being posted, while 50% of the unanswered questions were last edited just 0.06 days (88 minutes) after being posted.
Questions that are edited within minutes of being posted aren't likely to get more attention than posts that aren't edited at all. It's better to edit them later, presumably so they get bumped to the Active page at a different time of day, or on a different day altogether.
Tags
Andy suggested a few more things to look for in the comments, including a couple of questions about how Tumbleweed questions are tagged. I looked at how many tags are used, and of those questions with only one tag, whether or not one of the most popular 36 tags was used. Here are those results.
The proportion of answered questions doesn't deviate much from the overall Tumbleweed answer rate (28.2%) that we saw above, no matter how many tags are used. It's between 25 - 30% for each category. Number of tags used doesn't seem to make much difference.
I also found that among those questions with only one tag, 16% of the answered questions were tagged with one of the 36 most popular tags, while 21% of the unanswered questions were tagged with one of the 36 most popular tags (20% if you look at the combined set). Those numbers aren't very far apart, but they are very low. (9% of all questions on Stack Overflow are tagged java alone.) This seems to confirm what many of us have long suspected, which is that questions that are not tagged with one of the more popular tags are more likely to be ignored.
Title and Body lengths
Andy also asked for a rough breakdown of the lengths of both answered and unanswered posts. I thought it might be a good idea to look at titles as well, since that's what people first see and use to decide whether to view a question or not.
> summary(title_lengths_ans)
Min. 1st Qu. Median Mean 3rd Qu. Max.
12.00 39.00 51.00 53.54 65.00 150.00
> summary(title_lengths_unans)
Min. 1st Qu. Median Mean 3rd Qu. Max.
10.00 39.00 51.00 54.08 66.00 180.00
I have the titles for all 500k+ Tumbleweeds, so these summaries are for the entire data set. As you can see, these numbers are nearly identical. Title length apparently doesn't play a part in whether a Tumbleweed gets answered or not.
Body lengths were a little bit more difficult to obtain. I don't have the full body in my data set, so I took a random sample of 1000 QuestionIds to the Stack Exchange Data Explorer. I only drew from the set of non-deleted Tumbleweeds, since SEDE doesn't seem to have deleted posts to return. If anyone wants to run their own tests, you can find those Ids in the query Tumbleweed Random Sample (not deleted). (Note that you currently only get 984 results. I guess a few of those randomly selected posts were deleted in the past couple of weeks.) Here are the body length summaries:
> summary(nchar(answered_sample$Body))
Min. 1st Qu. Median Mean 3rd Qu. Max.
51 459 810 1311 1525 15360
> summary(nchar(unanswered_sample$Body))
Min. 1st Qu. Median Mean 3rd Qu. Max.
77.0 498.5 891.0 1608.0 1668.0 25770.0
Unanswered questions are definitely longer, but only by about 300 characters (50-60 words) on average. I guess the lesson to be learned here is that if you're going to try to edit a Tumbleweed into shape, help the author get to the point and make the post more concise. (Yes, I can see the length of this post, and I get the irony in that recommendation.)
User Reputation and Activity
Several people asked about the activity of users who received the Tumbleweed badge. I didn't have any information about users in my data set other than their IDs, so I had to go to the Data Explorer with a random sample to make some estimates. You can see the query Random Sample of Tumbleweed Awardees if you want to dig deeper than I do below. The results are mixed.
First I just looked at the reputation of users who were awarded the badge.
> quantile(randUserQueryResults$Reputation, c(.25, .5, .75))
25% 50% 75%
1 18 113
We can ignore the mean because there are a handful of very high-reputation users that skew it high. The quantiles tell the real story. At least one-fourth of the users who have been awarded the Tumbleweed badge only have 1 reputation. (That's their reputation today, not when the badge was awarded.) Half of them have a reputation of 18 or less, and three-fourths have a reputation of 113 or less. This confirms any suspicions people might have had that most of the people who ask Tumbleweed questions are new or at least low-reputation users.
The next thing that I looked at was a little bit more encouraging. I wanted to get an idea about whether or not people who asked Tumbleweed questions were even returning to the site after posting the question. I looked at the date the Tumbleweed was awarded and compared it to the user's last activity on the site. Out of 1000 users, only 86 of them never returned to the site after the Tumbleweed badge was awarded.
Here's a summary of the number of days between when the badge was awarded and the user's last access:
> quantile(awarded_to_last_active, c(.25, .5, .75))
25% 50% 75%
62.93141 293.73100 730.20633
This shows that only 25% of users who ask a Tumbleweed question are active on the site for about 60 days or less after the badge is awarded. This is good news, since it means that a good number of users would be back at least once to see a system message or comment asking them if they're still interested in an answer to their question if that message was left at the time the Tumbleweed was awarded. If they don't take some action within that time (edit the question or delete it themselves), it's probably safe to delete it if it still has low views and no answers or upvotes.
Suggestion for improved handling of Tumbleweeds
In addition to deleting questions with low views, no answers, and zero score after one year, I think it would help get some more questions answered if one of the following actions were taken:
Create a new review queue out of the Tumbleweed badge. Basically, just add a question to the queue as soon as the badge is awarded. Older Tumbleweeds could also slowly be introduced to the queue.
Add questions that are awarded the Tumbleweed badge to one of the existing review queues. I think Triage would be the most appropriate, but Help and Improvement might also work.
Either one of these actions would get more views for questions that have largely been ignored, and allow the community the chance to either vote to close these questions earlier, or edit them into shape so that they can be answered.
Further Analysis
If anyone can think of any other ways to identify what's different about Tumbleweed questions, I'm open to suggestions. I have ids for over half a million of them, so it would be far easier now to query the Stack Exchange API or Data Explorer with a random subset of ids to get even more information.
I also already have several fields that I didn't use yet (tags, titles, score, view count, etc.) for the whole data set. If you can think of ways that these pieces of information might be used to identify how the questions can be improved, or how the system might use that information to suggest improvements to post authors or reviewers, let me know in a comment or an answer. I'd be happy to run more tests on this data.