81

I maintain a reasonably popular Open Source Project. In my ideal world, all of our users would come to the project Discourse for help. But Stack Overflow exists, and I have come to accept the additional burden that it represents. However, I am hoping in this case that SO can help me, help it.

After nearly half a decade, we will be restructuring the directory layout of our examples in the project repo. I nearly always provide a link back to our documentation or examples in order that users may learn that these valuable resources exist for them to use. This restructuring will certainly result in broken links in a substantial fraction of my previous 800+ answers.

Updating lots of my old answers is not a task I relish, but one I will do for the sake of the project. Stack Overflow goes on about not wanting stale answers. In this we are aligned. So, great! So what would help enormously in that regard? This:

A way to search and find all of my answers that have broken links, or more generally, a way to search for any [tag] answers that have broken links.

Is this possible?

15
  • 13
    I'd imagine it would involve (1) getting a list of all your answer IDs which have links, accomplishable with SEDE (filter by only links to the github project, if you want.) (2) extracting the text from each of those answers, accomplishable with the API (3) going through the links and checking the response status code, filtering for 404 (or other errors), accomplishable with pretty much any language. If just for your Github, you might simplify with a whitelist of working project links. Not entirely trivial, but pretty doable Commented Jun 10, 2019 at 5:35
  • 11
    Asking SO to do the Network check is a bit too much IMO. If you have an URL pattern, then SO search can actually already provide a bit of an help.
    – Kaiido
    Commented Jun 10, 2019 at 5:51
  • 11
    Since you don't want broken links from Google or any other site that links to your documentation, you want to setup redirects either way. So broken links on SO doesn't matter, as long as a click will lead to the correct page it can't be considered stale. Commented Jun 10, 2019 at 9:54
  • 1
    @DennisGermundal There is no way to make redirects for GitHiub content, AFAIK.
    – bigreddot
    Commented Jun 10, 2019 at 13:14
  • 8
    @Kaiido If SO cares about shepherding the quality of their content, they would want to make it easy for authors to update their old answers. I would go so far as to suggest they should automate this reporting, since it can be automated, and pester authors to fix things.
    – bigreddot
    Commented Jun 10, 2019 at 13:15
  • @CertainPerformance What is SEDE? GIS turns up nothing helpful (or too much unhelpful, to be specific)
    – bigreddot
    Commented Jun 10, 2019 at 13:17
  • 4
    @bigreddot I remember I asked for an automation correction of 6,600 links a year ago, and I poke a Moderator to get help from a Community Manager about it, and ... and well, a year later, only 2% links got manually fixed, the many others are still broken.
    – Cœur
    Commented Jun 10, 2019 at 14:39
  • 2
    SEDE: data.stackexchange.com Commented Jun 10, 2019 at 21:37
  • @CertainPerformance Thank you!
    – bigreddot
    Commented Jun 10, 2019 at 21:45
  • 3
    Future-proof tip: always replace master in a Github file URL by its commit hash. Commented Jun 10, 2019 at 22:49
  • 1
    @mgarciaisaia My job, as an OSS project maintainer, is to funnel our users towards the most up to date information, so I am obliged to link to master. This is somewhat at odds with SO's "answers encased in amber" state of being, which ignores the realities of OSS evolving over time. This is not a case of "link-only answers go out of date". because ALL answers can go out of date. At least broken links can be searched for in an automated fashion, unlike dated code in answer code blocks. Another reason SO is more often a burden on OSS maintainers, than a boon.
    – bigreddot
    Commented Jun 10, 2019 at 23:58
  • 5
    @bigreddot: I disagree about always redirecting to master. A user having a specific version of the project needs documentation for this version, not a former or later version. As such, I think it can be beneficial to create tags for each release and use an URL based off the tag, rather than master, with a specific disclaimer at the top of each page for which version it is documenting. Commented Jun 11, 2019 at 6:54
  • 5
    @MatthieuM Realistically speaking, most SO users are never bothered to state version information. I have had to ask for it on innumerable occasions, and even then often fail to receive it. In that climate, I will do whatever I can to point people at the latest for the sake of my own sanity. You are certainly welcome to disagree all you like, but it will not affect my own practices.
    – bigreddot
    Commented Jun 11, 2019 at 7:14
  • 4
  • Maybe archive.org/web can be useful plus auto check broken links.
    – Jroger
    Commented Jun 12, 2019 at 22:20

1 Answer 1

61

A Stack Overflow search with url:myPathToTheBrokenUrl will result in all the posts with such broken URL. Example: if you search for url:imageshack.us/i, you'll get all the broken urls with imageshack.us/i in the URL (may not work if the path you're searching is too long, has too many digits or too many dots: in such cases, try cutting the end of the path you're searching until you get results). Add user:me to the search to limit it to your own posts.

If you're searching for an advanced pattern, you may go to https://data.stackexchange.com and use the Transact-SQL LIKE logical operator which supports the set pattern [] and negative set pattern [^], but that's about the best it can do: you can't search for anything with a variable length (?, {}, +, ...) for instance. Usage example: https://data.stackexchange.com/stackoverflow/query/1055712/tag-wikis-with-referral-links

If you need a regex search, you may try to download the internet Stack Overflow and perform the search locally.

Finally, if you're not even sure which URLs are actually broken, and which ones are not, then it's a tricky situation, potentially requiring a crawler to check for HTTP 4xx responses. Well, someone did it in I estimate 10% of the links posted here are dead. How do we deal with them?, but it may be too much work to review 150,000+ known broken links for the few people willing to help.

Note that you shouldn't just fix your own posts, especially when you got more than 2000 reputation: it's best to fix everybody's posts when you're aware of the new correct URL for a given broken one. I know it's time consuming (I've been fixing most of the deimos.apple.com, deimos3.apple.com and devworld.apple.com lately, despite being unaffiliated with Apple) but you can ask for help on meta with the meta tag and the little mice will help at night.

4
  • 1
    "if you're not even sure which URLs are actually broken, and which ones are not" Somehow I think that the "if" can be safely removed from that sentence. The way how URLs become broken usually means you don't get a warning. What one could do is searching for broken links and then concentrate on the domains that appear to be more often out of order. Some kind of hybrid approach. Commented Jun 11, 2019 at 12:47
  • 9
    @Trilarion from the first sentence of the question, it's about updating links for a project that you're maintaining. So for instance when you rename README.txt to README.md, you know precisely what you've broken.
    – Cœur
    Commented Jun 11, 2019 at 14:22
  • Super! I just did a user:me url:jsfiddle.net - a revelation
    – mplungjan
    Commented Jun 12, 2019 at 9:13
  • That's cute, as if 24PB were enough to store the Internet. Commented Jun 12, 2019 at 12:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .