97

Could we have URLs to bit.ly and t.co (etc.) auto-expanded to their final homes?

Someone posted here (10K only) a bit.ly URL that went to LMGTFY. I had no idea what it was from the comment.

I'm not saying all URLs should be checked, but just the top 10 shorteners.

11
  • 46
    How about just flag comments with short URLs immediately. Don't even bother following the links. The character limit on comments is incredibly generous - there is almost no reason to post a comment consisting solely or primarily of short URL(s) regardless of where it might lead to.
    – BoltClock
    Jun 1, 2014 at 16:27
  • @BoltClock i did flag it. perhaps url only (or significantly weighted) comments should be banned in that case. Jun 1, 2014 at 16:28
  • @BoltClock there are good reasons to use goo.gl for example - it does view tracking which is useful if you link it to a specific github page for instance. Also, it does help save a few characters. Jun 1, 2014 at 16:31
  • 53
  • 2
    Honestly, seeing URL shorteners in questions is worse. I've seen it used a lot to avoid the "you can't link to jsfiddle without also pasting code" rule.
    – cimmanon
    Jun 1, 2014 at 22:05
  • 2
    @cimmanon Look on the bright side, it tells us who to -1, CV (pick anything), and VLQ immediately.
    – bjb568
    Jun 2, 2014 at 19:14
  • I use a shortener for stackoverflow.com/questions/5963269/… ( tinyurl.com/reproducible-000 ) because it saves me typing and/or looking up the link again ... is there a better way to handle that?
    – Ben Bolker
    Jun 2, 2014 at 19:19
  • @BenBolker why not use a snipit tool? Jun 2, 2014 at 20:49
  • 2
    @BenBolker Even better, use this to post entire, perfectly-formed comments. Jun 3, 2014 at 7:38
  • I have sometimes used shorteners when the link contained certain URL-escaped characters that kept breaking when trying to link normally (some characters would be replaced by spaces).
    – harold
    Jun 8, 2014 at 16:11
  • Except for an outright ban, I'm not sure how a link-follower/link-verifyer would work to stop spam. SO doesn't have the bandwidth to constantly scan every single link in their database. What prevents a spammer from posting a single-redirect legitimate link, and then 5 minutes later after the verifier has run, change where the link points to. SO would never be able to detect that in a cost-effective manner.
    – Sam Axe
    Jun 8, 2014 at 21:15

7 Answers 7

150

To be honest, I'd support completely blacklisting all of the common URL shorteners at this point. I've been campaigning for this internally for a while, and I think the abuse of these URL shorteners is getting worse. I do not believe Jeff's earlier answer about this reflects the state of the site at present.

Spammers have turned to circumventing blacklists on their spam URLs using shorteners. Almost every comment I see using a bit.ly link leads to LMGTFY or something equally insulting.

A particularly egregious abuse of this came up last week, when a troll suggested a series of otherwise reasonable-looking edits that slipped in shortened URLs to a hardcore porn site. Two examples of these edits are here and here (do not go to the link they added there), and if the actual target URL had been shown to the reviewers they certainly would not have approved this. It required me clicking through those (and going to a hardcore porn site at work) to see what the problem was with the edits and why someone had flagged them. This was not the first such spam or trolling link to make it through edit approval aided by URL shortener.

I'm not alone in my displeasure with link shorteners, as both Shog9 and Tim Post have had choice words for them on Twitter recently.

There are currently 1331 deleted posts (not counting comments) on Stack Overflow that have a bit.ly link in them, and the vast majority appear to have been deleted for good reason.

However, there are also 3568 live posts on SO with bit.ly links, so there are many honest users who have used shortened links. That said, there is no limit to the length of a post on SO (and I've very rarely seen URL length be the limiting factor in comments), so that's not a good reason for their usage here. Benjamin brings up statistic tracking with goo.gl links in the comments above, but I don't see that one limited advantage outweighing the overall problems with these links.

I believe we'd be better off with them blocked entirely. We now have the ability to provide custom explanations when someone inputs blacklisted content, and for URL shorteners we can describe why we prefer full links here to educate otherwise well-meaning posters.

14
  • 10
    i'm fine with this nuclear option as well. it will provide some trust to the links. Jun 1, 2014 at 17:55
  • 3
    Can you please whitelist git.io then? I see it used for legitimate reasons all the time. TBH it's mostly bit.ly and its friends that are abused. Jun 1, 2014 at 17:56
  • @DanielA.White no it doesn't. Try it :) Jun 1, 2014 at 17:59
  • oh cool. i'm for that as one to skip in the white list then. Jun 1, 2014 at 18:01
  • 1
    @BenjaminGruenbaum - Yeah, bit.ly is top of the list, but goo.gl might come in second among spammers like these: stackoverflow.com/questions/23314796/… stackoverflow.com/questions/7295873/… . tinyurl.com gets used frequently, but I don't often see spammers or trolls relying on it, mostly people linking to images.
    – Brad Larson Mod
    Jun 1, 2014 at 18:02
  • 1
    On the topic of blocking: what about the review-blocking those three who approved that first edit?
    – Arjan
    Jun 1, 2014 at 19:26
  • 11
    @Arjan - To be fair, there were some really good reviewers who approved several of these edits (people you'd be surprised at) and in general they were sensible edits without the links. This was a very subtle troll who knew what they were doing, so I'm not as likely to hold it against reviewers in this case.
    – Brad Larson Mod
    Jun 1, 2014 at 19:32
  • 8
    I've very rarely seen URL length be the limiting factor in comments - I hit that limit quite often any time I add a couple of links to github and/or an existing question/answer/comment. The only "solution" to the question as asked would be to follow all links checking they don't 301 to something that's banned - otherwise users would just hop from one shortner service to another, or create their own (it's not like it's hard to do). Would probably add up to significant overhead though.
    – AD7six
    Jun 2, 2014 at 9:11
  • 5
    @AD7six an existing question/answer/comment - Use the "share" URL for questions/answers at least. Those are almost as short as URL shorteners anyway.
    – Izkata
    Jun 2, 2014 at 19:23
  • I am OK with the idea of flagging but not so much of blocking entirely as your research showed 72% of posts with bit.ly links are still live. While it doesn't mean 72% are good links, clearly people have found use for it. Would it be useful to do a UI change (change colour of the link) to let everyone know that the link is short? I'm thinking specifically for reviewers of posts but could also be used for links in comments? Another idea, how about having a stackexchange short URL service? Allows for users to get what they want (shorter URLs, link analytics) as well as allows more control.
    – Turnerj
    Jun 2, 2014 at 23:58
  • What if the links were one-boxed like on Discourse or Facebook? That way we would have some sort of preview of where we were heading? Jun 4, 2014 at 16:11
  • 1
    since there is no good reason to use URL-shorteners on SO, I think it would be a good idea to block at least the top 25 most used URL-shorteners. If an honest user inadvertedly uses an URL-shortener, he should just get an error-message if he tries to post it. also, if its a policy to disallow URL-shorteners, I could just flag them and don't have to follow them to find (surprise...) that it is bogus, before I flag them.
    – Michael
    Jun 4, 2014 at 22:35
  • I have had people unshorten my links a few times recently for example here and for godbolt links which can be very long this is a little annoying. Within an answer it is not a show-stopper but I would not be able to fit one of those unshortened links in a comment. Dec 8, 2014 at 4:34
  • "do not go to the link they added there" - Here is porn, do not look at it. Yeah, good luck with that. ;-) Feb 29, 2016 at 18:09
18

While I'd like to support Brad Larson♦'s blacklist unreservedly, the matter is complicated by the fact that some people have asked questions about URL shorteners, as noted in this old Meta discussion. I would hate for Stack Exchange to create another filter which is right in 99% of cases but cripplingly annoying in the 1% of edge cases.

Can't we autoflag posts containing blacklisted instead of blocking them entirely? In an ideal world, we would have a 'shortened URLs' queue that offered manual editing, automatic substitution for the expanded URL, flagging as spam, or taking no action. If a dedicated queue is more development work than the problem is worth, though, then as a poor man's solution you could just shove them all into the Low Quality review queue:

Mocked up image showing tiny urls in the LQ queue

1
  • 2
    As a side note, I just came across a 'proper' use of a URL shortener in a post - the content was on someone's dropbox account, and the 'shortened' URL just pointed it, meaning that if they move the content, the URL can be re-pointed. Jun 8, 2014 at 16:45
5

Create a out-link wiki.

Links out of stack overflow to "unapproved" websites bounce to the out-link wiki, with a comment about the link destination not being validated, and the option to up/down vote and flag the out-link website.

Using an out-link in your post prompts you to enter information into the out-link wiki about the website. This does not cause it to be approved.

Out-links can be blacklisted though this process (really low rankings, or manually by moderators), at which point existing out-links break, and new posts using those domains are told the website is blacklisted.

A mixture of blacklists (of common url shorteners, which will include advice to users) and whitelisting (to known "good" targets, which don't bounce you through the link-wiki), and greylisting (to other targets, where you are first sent to the link-wiki and warned that the website is unknown) should result in most links working just like they do today, common url-shorteners resulting in posters being told "just post the real link, please", and other links at least providing warnings to users that the destination website may not be validated (and a request to mark it up if it was good).

This forces spammers to use greylisted websites, and honest users who use a url redirector get told the proper behavior.

3

Perhaps what you are asking for is some link verification rather than auto-expansion?

As others have noted auto-expanding shortened URLs is impractical due to the huge number of URL shorteners available and the ease of creating entirely custom ones.

Link verification could be a useful feature that would do a number of things:

  1. Verify that the link is actually valid i.e. check that it doesn't 404
  2. Verify that the link does not resolve to blacklisted sites
  3. Attempt to verify that the link is relevant

Whether SO would actually want to implement any of this is unclear since all the ideas here risk turning SO into a DDoS pathway since you require the platform to make some number of HTTP requests to every link posted. Obviously there are technological means to get around this such as request throttling, caching link verification results for some period of time etc. but I still have concerns about the feasibility of these techniques on a heavy traffic site like SO.

However I think the ideas at least warrant some discussion so I have attempted to detail what I mean by each and in vague terms how they would be achieved technologically.

1 - Link Validity

This is relatively easy to do, you simply send a HEAD request (or OPTIONS/GET) to the given URL and see what HTTP response code comes back (likely ignoring the body where you have to use GET). If you get a 2xx code then you assume the link is valid.

Most of the time you want to use HEAD but some servers won't allow HEAD for whatever reason so you may need to use OPTIONS or GET as a fallback/double check that an error code a HEAD request produced was indeed a genuine error. Of course getting a 406 Method Not Allowed response to the HEAD request is a clear sign you need to try a different verb

Now obviously this won't be perfect because some poorly configured sites will produce error pages with a 2xx code but there isn't much you can do about this. Also you can encounter transient errors with a site being temporarily down when you try and check it. Plus it will incorrectly flag links to content that the poster doesn't realise is behind paywalls/login gateways or as I've done in the past in this answer post despite this with a note to that effect.

Links that are checked and shown to return an error code should likely result in some visible notification to users that they've entered an apparently invalid link.

As noted in the comments there are a class of questions that are predominantly about URLs returning error codes e.g. REST, HTTP debugging etc hence why for this part of link verification I am suggesting that any errors are simply presented as a warning rather than being flagged for review in any way.

2 - Checking blacklist

Part of the point of the OP appears to be that people are using link shorteners to obscure links to undesirable content that should probably be on some blacklist e.g. lmgtfy.com

Here you probably want to check the raw link URL itself and also the response URL. Any good HTTP client will allow you to auto-follow 3xx redirects and provide you the means to retrieve the actual URL at the end of the redirect chain (because a really sneaky user could hide stuff behind multiple link shorteners). So you want to get and check both the raw and resolve URLs against the blacklist.

Links that appear in the blacklist should either be auto-removed from the post (likely with notification to the user) or cause posts to go into the low quality post review queue.

3 - Link Relevance

This is perhaps the hardest to implement and maybe not something that SO would actually want as a result since I suspect the false positive and negative rates would be too high.

What I am thinking here is that if you GET the content at the end of the link you can apply standard term frequency analysis techniques to it and see if the terms mentioned in the linked content have some reasonable degree of correlation with terms in the question (and associated answers). The actual technical details of how you would decide what constitutes correlation are somewhat vague but I think this is at least in principal technically feasible.

Linked content with poor/zero correlation is likely to be spam though of course there are some legitimate cases where this might not be the case e.g. links to framed javadoc where the raw HTML returned at the root of the javadoc would likely have little term correlation to the question. Also posts with HTML/CSS/JS fragments would likely have high correlation with arbitrary content if you don't ignore the HTML tags in the retrieved content.

Posts where multiple links are flagged as potentially irrelevant should drop into the low quality post review queue

2
  • 1
    As far as the link validity check goes, it's worth noting that we shouldn't automatically block or remove links that return error codes when followed because such links may actually be useful when dealing with questions about public web APIs. The link in a question like "When I hit example.com/api/user/1123185 with a GET request I receive a 400 code - why?" is useful despite (indeed, because of) the link returning an error.
    – Mark Amery
    Jun 4, 2014 at 18:02
  • +1 I totally agree, I wasn't advocating blocking such links rather just warning users of the error to allow them to fix accidental typos. Obviously there are whole classes of questions where bad links are the topic of the question
    – RobV
    Jun 5, 2014 at 9:16
-2

We can't do it. It'll destroy too many legitimate questions about using the shorteners, and locking out classes of otherwise legitimate questions is against everything the site is for. I believe that such collateral damage isn't really acceptable here.

HOWEVER, one thing we could do is to notice use of things like bit.ly (let's just tage the top 10 or 20 shorteners) in a post and suggest somehow that they expand the link (perhaps a box just above the 'post' button?). I'm unclear on the UI, and I don't think it's reasonable to prevent immediate posting, but if we could somehow warn people before they post, perhaps we could cut down inappropriate use of these things.

-2

How about tying URL shortening into the reputation system.

If a user under, say, 1000 rep tries to use a URL shortener they are summarily denied.
Users with rep between, say, 1000 and 2999 would be allowed to use shorteners, but they would be auto-expanded. Users with rep 3000+ would be allowed to use shorteners but the post would be auto-flagged for moderator review. And users with 10k+ rep could use shorteners without any interference . For the 10K+ users we'd rely in the community to flag inappropriate posts.

SO would then need back-end software to determine if a link was shortened. It would have to scan all links in all posts.

I would suggest starting by limiting the number of links in any post to no more than 5. Seems plenty to me.

Then begin link scanning. If there's a single redirect then mark the link good. If there is more than one redirect then stop following the link and mark the link bad.

-12

Automatically flag links that result in more than one redirect.

This allows a shortener to do what it was intended for. Shorten URLs. It also encourages linking to original sources if legitimate analytics are either desired or unavoidable and forgoing the shortener. Further, if a resource is legitimately relocated, the permanently moved resource may still be accessible via the SO content until it can be updated and pointed to its new permanent home.

Automatically crawl all shortened URLs, and verify that redirect count doesn't exceed one. When more than one redirect occurs in established content, mask the link and flag for review.

2
  • 1
    Why do you want to allow one additional critical link in the chain, which might break at any time? Just so you get statistics? Jun 2, 2014 at 23:48
  • Any link, chain or not, can break at any time. Most shorteners and analytics services will probably outlast most of the content they point to for that matter. More to the point, genuine analysis and measuring should be generally encouraged, at least until we know the measurement is without relevant consequence.
    – JustinC
    Jun 2, 2014 at 23:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .