79

People frequently rant about how Stack Overflow's draconian rules stink, too many questions get closed, etc.

Regardless of where one stands on that argument, those people often complain that they ended up at a closed question through Google. Recent example

That seems a valid point.

If we don't allow resource requests or list questions, why is...

This is bound to lead tons of people (290,000 views between the last two examples alone) to a question that likely contains stale content, or none at all, and can no longer be answered.

That is actively making the web a slightly worse place. No?

Is hiding closed questions* from the Google bot technically feasible?

If it is: why is SO not doing it?

* = There'd have to be the obvious exceptions: Questions closed as duplicates, and questions with a historical lock.

31
  • 2
    Related: meta.stackexchange.com/questions/209905/…
    – Pekka
    Commented Feb 15, 2016 at 7:36
  • 2
    What happened to on MSE? o_O Commented Feb 15, 2016 at 7:49
  • 1
    @BhargavRao I have the Metal Umlaut there!
    – Pekka
    Commented Feb 15, 2016 at 7:50
  • 28
    Closure isn't necessarily a permanent state depending on the question and the close reason, and I don't imagine it's as simple as choosing whether to include or exclude a page from SERPs. And whether a closed question is useful or not useful depends on the content as well. Shopping/list questions tend to go stale, but only those questions do. An old, subjective question might contain a wealth of information that remains relevant even today. (Another old, subjective question might contain nothing worth preserving at all, in which case it should be voted off the site altogether.)
    – BoltClock Mod
    Commented Feb 15, 2016 at 8:19
  • 3
    @BoltClock when a question has been closed for five years, though, like in the second example, the likelihood is very strong that it's going to remain closed. Excluding pages from the index could, in theory, be done through a robots.txt but I have no idea whether a robots.txt with five million entries is feasible. Probably not.
    – Pekka
    Commented Feb 15, 2016 at 8:23
  • @BoltClock isn't the historical lock for protecting old, subjective questions? We could take that into account
    – Pekka
    Commented Feb 15, 2016 at 8:24
  • 41
    Some "dupes" have much better answers than the "original" posts. Commented Feb 15, 2016 at 8:26
  • 1
    There is some merit to your proposal. But I don't like the ultimate consequence. What? You did not know that SO users close questions?? Get used to it already!!! Commented Feb 15, 2016 at 9:00
  • 2
    Not sure that I agree with the premise here. Why is that a valid point? What's wrong with ending up on a closed question? Presumably, the questions are closed and not deleted because they contain some possibly useful information or may be salvageable through editing. Why would we want to hide these from a spider, effectively sweeping them under the rug forever? Just delete 'em.
    – Cody Gray Mod
    Commented Feb 15, 2016 at 11:02
  • 2
    The middle one isn't stale. I have referred to it twice within the past week. Once to help someone else out on Stack Overflow, and a second time for myself. :-) But not to focus on coincidences, the questions are still visible because they might be useful. So why should we banish them from search engines? I assume your thinking along the "broken window" lines? People's first impression of Stack Overflow will be that there are all of these closed questions? So what?
    – Cody Gray Mod
    Commented Feb 15, 2016 at 11:16
  • 1
    @CodyGray The middle one isn't stale fair enough - although I don't like the fact that it shows up for visual c# download. The reasoning is that SO has huge search engine juice, and closed worthless questions push others (which are actually useful) off the results page. In an ideal world, we'd delete all the non-useful closed questions, but I don't think we're even remotely there.
    – Pekka
    Commented Feb 15, 2016 at 11:21
  • 2
    I agree, the "close" policy feels way too strict, and they should probably just be re opened, but that's just me.
    – rogerdpack
    Commented Feb 15, 2016 at 13:20
  • 1
    I think there are at least two faulty assumptions in the OP: 1) Closed questions aren't valuable, and 2) It's technically possible for SO to control what appears in Google's search index in a fine-grained, dynamic way. Both are false.
    – aroth
    Commented Feb 15, 2016 at 13:26
  • 2
    because Ad Impressions ...
    – user177800
    Commented Feb 15, 2016 at 14:42
  • 2
    @Nappy I don't really care whether they're relevant to us as a community or not - I care whether they're good search results for the rest of the world. Way too often they're not. They are locked down on SO in a state that heavily discourages further editing, so they're bound to become stale. It's insane to close questions that we as a community no longer want to maintain, but to still signal to the world that they're resources worth searching for and checking out.
    – Pekka
    Commented Feb 15, 2016 at 21:40

7 Answers 7

149

I disagree with this, and I think that closed questions should show up on Google.

Making the web a slightly worse place. No?

No. In my experience:

  • If I'm searching for something, I usually find that an SO link is of high quality, and it will usually answer my question, even if it's closed.- in contrast, if that SO link wasn't there, I'd have to look elsewhere, and random blogs, well, are pretty random in whether they have what I'm looking for.

    I find that SO questions, even if they're closed, are of higher quality than anything else - and if they weren't there in the Google results, I would be very unhappy.

  • As BoltClock said, question closure isn't permanent, and this means that questions closed in error or which get reopened for some other reason will not be in Google - potentially for a very long time after it gets reopened, since recrawling things is a low priority for Google.

Some examples:

  • How do I prevent site scraping Closed as too broad, but there's a lot of useful information there and most of it isn't obsolete or stale. In fact, the information in all of those answers combined is more useful than many of the thin blog posts around the net, and it shows, because that's the top hit when googling for "How to prevent scraping". (I'm biased, since one of the top answers is mine though..)

  • Is there a way to get the source code from an APK file Closed as external resource recommendation, but again, there's a lot of information here, of fairly high quality. Some might argue that it was closed in error, since it doesn't ask for an external resource. (Now reopened)

  • Creating a memory leak with Java, closed as too broad, still very educational and otherwise useful. (Now reopened also)

  • ...

I agree with you - tool recommendation or list questions which were closed years ago may be stale or obsolete - but the solution is not to hide them from Google. The solution is to edit the answers so they're not obsolete (in fact many answers to closed questions are being actively maintained, and are not obsolete), or just to delete it.

If a Q & A contains stale content, or none at all, why not just delete it, and problem solved?

People complain that they ended at a closed question through Google

If people are thinking that they can ask similar questions because they found a closed question through Google, we need to tell them that "closed" means "this should not be asked here", if it's not clear already. If people are complaining about closed questions in general, educate them on why those questions are getting closed, and why those "draconian rules" exist. The solution is not to remove closed questions from Google. People will find them anyway.

Technically feasible?

Yes: just serve a 404 response on the page, and the Googlebot won't crawl it. (This may have unwanted side effects, but there's other ways too. It's not difficult.)

Why is SO not doing it?

  • Very often, closed questions are useful. In fact, Google wouldn't have these questions ranked so highly in search results if they weren't useful.

  • Ad dollars.

11
  • 1
    Well, deletion of oudated content would be nice - but it just doesn't seem to be happening (plus it's a super contentious issue that frequently leads to almost-civil war.) Look at my examples #2 and #3. They're 5 and 6 years old, respectively. If deletion were a thing, wouldn't they be deleted already?
    – Pekka
    Commented Feb 15, 2016 at 10:09
  • 1
    How do I prevent site scraping - Closed as too broad, but there's a lot of useful information there and most of it isn't obsolete or stale. at the moment, maybe. But what about in 5 or 10 years? Who will edit the answers?... that said, it'd certainly be possible to add more exemptions for certain types of closed questions to the rule.
    – Pekka
    Commented Feb 15, 2016 at 10:10
  • 1
    just serve a 404 response on the page, and the googebot won't crawl it. possible, but probably too dangerous; Google is said to frown upon behaviour that is custom-tailored to their bot. Other than that I can't think of a way, do you know one? I'm genuinely interested.
    – Pekka
    Commented Feb 15, 2016 at 10:16
  • 2
    @Pekka웃 webmasters.stackexchange.com/questions/16090/… Commented Feb 15, 2016 at 10:38
  • @Jeffrey ah, so there is a meta tag and header. Nice!
    – Pekka
    Commented Feb 15, 2016 at 10:41
  • 9
    The "ad dollars" talking point isn't new; the counterargument is: "Even if you added up ALL the closed questions this popular, they wouldn't pay for Joel's gourmet coffee habit". More likely, nobody's interested in investing developer time into an absurd system where some content is both hosted and hidden at the same time. (Also, this is against Google's guidelines, which require humans to see what bots see.)
    – user3717023
    Commented Feb 15, 2016 at 13:00
  • 2
    @Sally this could be solved with a simple Meta tag. No "system" required. As it stands, SO is polluting search results with plenty of content that is no longer actively maintained.
    – Pekka
    Commented Feb 15, 2016 at 13:34
  • 2
    Just to add, another benefit is that they often link to a non-closed question that didn't come up on the Google search. Commented Feb 15, 2016 at 14:50
  • @Pekka웃: In the case of your examples 2 & 3: I think they're still relevant and useful - they're being edited and maintained, ans they give visitors useful information, probably the information they're looking for. I don't think things should be deleted unless they're harmful - and these still seem to be pretty useful to many people. This is the beauty of SO - If i'm no longer around in 10 years to maintain my answer, someone else can - and if it's completely obsolete and useless (perhaps to the extent of being dangerous) it can be edited with a warning - or outright deleted. Commented Feb 15, 2016 at 21:04
  • 1
    I agree with you, but I am not so sure about the "stale content" needing editing or deletion. The unfortunate truth is that some enterprises use old and outdated stuff, so answers for old and outdated stuff may still be useful to people. Commented Feb 16, 2016 at 11:22
  • @Mark, Yes that's true - but if they're edited to reflect current practices, they're useful to many more people. I see this a lot in PHP tags: old, potentially insecure stuff (mysqli_) is getting edited to reflect newer standards, or is getting a "this is a bad practice now" warning added, or is being outright deleted. In this case, there's no real value in keeping old stuff around. However, I agree with you: a lot of old, obsolete things are still useful to some. Commented Feb 16, 2016 at 14:05
48

We want to show Google the same thing we show everyone else. Since closed questions are still visible on the site, they're visible to Google. Deleted questions are hidden for most users (only visible at 10k+ rep) and are therefore not indexed by Google.

3
  • 11
    Of course, this raises the question of whether we should be deleting all those old closed questions, but that seems like it should be a separate question.
    – David Fullerton Mod
    Commented Feb 15, 2016 at 15:12
  • 1
    Visible to Google doesn't necessarily mean eligible for inclusion. See <meta name="ROBOTS">. Commented Feb 16, 2016 at 0:22
  • I agree most with this sentiment. And I'd like to suggest that it be up to the user. best c# ide -"closed as off-topic" removes the offensive link. (Though interestingly it does bring up a similar question from 2008 that was never closed.) Perhaps a universal and relatively distinct tag for Google to read such as seClosedQ within the HTML of all closed questions, so one could just add -seClosedQ to any Google query to ignore those topics. On the tangent of deletion, I would not want anything deleted unless it sunk to the level of spam. As with others, I've found closed Qs useful.
    – s.co.tt
    Commented Feb 16, 2016 at 16:44
23

It's throwing the baby out with the bathwater in quite a few instances. When I really put the screws to myself to get comfortable with JavaScript without the use of frameworks, I happened upon many closed questions that actually helped me. Now these questions should have been closed in their current form - many of them were pretty vague (or duplicates), but they had just enough detail in the answers for me to find them, or the canonical questions they led to.

What made me feel kind of bad at the time was I didn't have enough time to edit all of these questions. I knew what the problem was because I was experiencing it also, and the answers they got really helped me - So I could have made some edits and reopened them.

A lot of folks just don't get the chance to do that as well, so I think it's good that we leave things pretty accessible because they do help, and we have an off chance of someone polishing them up.

The stuff we don't want people tripping over are poorly-worded questions that didn't get answers - and that's why we have the roombas running as well as lots of users pretty active in taking out the bad stuff. Deleted questions return a 404, so they're very quickly dropped by search engines.

Thinking out loud a bit - it might not be a bad idea to identify stuff that's closed as anything but a duplicate, yet keeps getting a bunch of views, and pipe it through the helper queue. These things generally have answers that explain the question pretty well and could really use some polishing to become lasting fixtures.

6
  • 1
    That might actually result in more posts in the helper-queue which can actually be fixed. Might make it more rewarding... Commented Feb 16, 2016 at 15:54
  • 1
    it might not be a bad idea to identify stuff that's closed as anything but a duplicate yet keeps getting a bunch of views, and pipe it through the helper queue Here's a couple ideas on how to weed out the chaff: meta.stackexchange.com/questions/209905/…
    – Pekka
    Commented Feb 16, 2016 at 16:02
  • @Deduplicator I was thinking that. There'd probably be quite a bit of it early on, then a pretty constant trickle - just enough to break up the monotony a bit. We're talking about it now.
    – user50049
    Commented Feb 16, 2016 at 16:33
  • 3
    I like the idea of reviewing high-visibility posts in general. There are many open questions that get tons of views that could use some care.
    – David Fullerton Mod
    Commented Feb 16, 2016 at 16:53
  • 3
    Even duplicates might be interesting since it sometimes makes sense to reverse the direction of the duplicate. This would be especially true if the closed question has a better answer than the (theoretically) canonical question.
    – Jon Ericson Staff
    Commented Feb 16, 2016 at 16:54
  • 1
    @JonEricson That's interesting when the dup gets more views on average than the canonical. Need to run some numbers.
    – user50049
    Commented Feb 16, 2016 at 19:14
17

Some of the closed questions (or marked as a duplicate) have great answers. Why should we voluntarily give up an important part of this site?

Here is an example: How can I create ActiveX using C#?. The answer is better than in the "original" question.

9
  • Because most of them aren't actively maintained any more, and are bound to become stale, if they aren't already.
    – Pekka
    Commented Feb 15, 2016 at 15:16
  • 9
    @Pekka웃 Worthwhile closed questions should be re-opened. The close policies are too strict. Commented Feb 15, 2016 at 15:52
  • @ChadKillingsworth Useful on-topic questions, sure. If they only fail one of the criteria which we adopt to avoid too many useless answers, but already got an outstanding answer anyway. Commented Feb 15, 2016 at 17:07
  • 1
    Why not merge the questions? I'm sure SO supported merging questions at one time. Commented Feb 15, 2016 at 17:52
  • @DisgruntledGoat Merging currently requires moderator attention, and the mods are reluctant to merge because merges are irreversible. As I commented on that answer, "As long as merging is irreversible, it can't be done by the community; as long is merging is both irreversible and moderator-exclusive, it won't be done often; as long as merging is rare, developers can't justify making it reversible or community-driven." Commented Feb 15, 2016 at 22:26
  • 1
    If the closed questions are useful, why are we keeping them closed? They're "stale" because no one can maintain them. Commented Feb 16, 2016 at 0:20
  • The question deliberately excludes duplicates. Not sure what this answer brings.
    – Ben
    Commented Feb 16, 2016 at 3:07
  • Flagged them for merging. Seems straight-forward enough even if the processing mod is not a C# expert. Still, the question explicitly and deliberately excludes duplicates. Commented Feb 16, 2016 at 15:59
  • I've found some helpful answers from closed questions as well, after googling.
    – Sharp Edge
    Commented Feb 18, 2016 at 6:11
8

Stack Overflow already tells Google the pages that are “best” using “priority” in the Sitemap based on the number of votes a questions and related answers have got. Note that this Sitemap file is protected so that it can only be accessed by Google.

Sorry I can’t find the meta question(s) that confirm the above.

Therefore a question with lots of down votes, will tend to be shown later in the Google result, most “bad” closed questions also have lots of down votes.

So therefore the issue is when the “priority” should be lower in the sitemap file due to a question being closed…..

Personally I think all questions should show up in Google, so that I can find them based on what I remember about I question I have read in the past – this does not mean they can’t be “discouraged” by giving them a low “priority”, so there are not on the first page of results.

So I think my message is, if the question is “bad” down vote it as well as voting to close.

5

A number of duplicate/opinionated questions have some value, even if they are considered off-topic.

However, I would like to propose that we should actively discourage search engines from indexing questions closed with the "cannot reproduce/typographical error reason". These questions have virtually no value to future users with accepted answers like the following preventing the question from getting cleaned up by the Roomba:

  • Your code works fine to me
  • You forgot a semicolon/curly brace
  • You misspelled your class/function/whatever

We may even want to consider adjusting the Roomba to remove such questions, even if they don't normally qualify for automatic deletion (positive score, accepted answer, etc.).

3
  • This seems obvious given the close reason states "this [question] was resolved in a manner unlikely to help future readers". Commented Feb 15, 2016 at 22:22
  • I agree that questions like this should probably be deleted. Perhaps through the Roomba, perhaps through a queue, perhaps through a gang of ravenous Meta wolves. But that means we don't have to do anything special with regards to search engine crawling. They won't crawl stuff that's deleted, and until it's deleted, what's wrong with it being indexed? It is, after all, really there.
    – Cody Gray Mod
    Commented Feb 16, 2016 at 11:29
  • 1
    Useless questions can and do push out useful questions in search results. Users tend not to curate crappy questions that should be closed, so we end up with questions that have just the right keywords to end up ranked higher than they should be.
    – cimmanon
    Commented Feb 16, 2016 at 12:10
3

A lot of questions are closed for being a duplicate of another. Those questions are still useful for many reasons:

  • The wording of each question is different. Your search terms may get you to one question over the other. Sometimes it may be quite impossible to find the answer without this serendipity, for example if you don't know the standard term that applies to your question.
  • Often the closed question will have a good answer or two, possibly better than the one marked as a duplicate.
  • The closed question will have a prominent link to one that is considered better. I don't know if links like that have an effect on SEO, but it can't hurt.
6
  • 2
    If the duplicate has better answers, we should merge them (or copy only the worthy answer(s)). Visitors who are not logged in are transparently redirected from the duplicate to the target, so the good answers are effectively hidden from them. (Anyway, duplicates are already noted as an exception in the question.) Commented Feb 16, 2016 at 0:38
  • 1
    @Deduplicator Oh, you're right, only for duplicates with no answers. My point stands that we should merge more, if we feel the problem with duplication is that "Answers to duplicate questions scatter information in a way that makes finding it time-consuming and error-prone for those who need it." Commented Feb 16, 2016 at 0:59
  • @JeffreyBosboom: If there is actually at least one answer on the dupe which adds something to the master, sure, flag for merging, linking to that answer and giving at least one good point. That's quite rare, but it does happen. Commented Feb 16, 2016 at 1:03
  • @Deduplicator Unfortunately the mods are (understandably) reluctant to merge because merging is irreversible. But yes, I know the procedure. Commented Feb 16, 2016 at 1:05
  • @JeffreyBosboom my main point was actually the first one, which merging utterly destroys. And I've never heard of this automatic redirection before. Commented Feb 16, 2016 at 1:15
  • 3
    The question deliberately excludes duplicates.
    – Ben
    Commented Feb 16, 2016 at 3:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .