77

(near-duplicate of Common online compiler blacklisted. This Q was meant to be about posts, where length limits aren't a serious problem and posting full URLs is a valid answer. This has become the go-to question for all discussion, including links in comments (where shortening is also blocked), and of course generalized from http://gcc.godbolt.org/ to include other code-posting sites.)


http://gcc.godbolt.org/ is the only online-compiler site I know of that shows asm output (other than hacks like having your program execl("objdump", argv[0], NULL) or copying MSVC's asm output to stdout). It even strips out the noise and formats it nicely, so it's great for seeing how something compiles with different gcc/clang versions, or even for ARM. Most of my answers have links to code on godbolt, since I spend a lot of time answering performance or SIMD-vectorization questions.

Godbolt has a "permalink" button which gives a goo.gl-shortened link to be copied-and-pasted. The unshortened links have all the compiler options, and the source code mime-encoded. All of that is urlencoded. I don't mind having links that are 1300 characters long for a 25 lines of code (with long comments), but it seems a bit excessive. It might make some of my answers run into the 30k char limit.

Should we just manually expand these links ourselves? SO could expand them for us. This would be especially convenient when editing old answers that have shortened links, to avoid having to manually go and un-shorten them.

In this particular case, Matt Godbolt (the site owner) can add a non-shortened textbox to copy from, since he still actively maintains/improves it. So long term, the extra step of pasting the URL into a new tab and hitting ctrl-L ctrl-C should go away, leaving us with just the length of the URLs.

The godbolt problem was already identified by a comment when the original proposal was posted, but there were no replies to that comment.

My point is undermined by the fact that the shortened godbolt URL I was going to use as an example (from Fastest way to do horizontal float vector sum on x86) no longer works. That's the first time I've seen a goo.gl-shortened godbolt link go dead.


That example godbolt link is from What is the efficient way to count set bits at a position or lower?, where I have two godbolt links to the same code with different compiler options. That's not uncommon. Sometimes the code is even longer, making the URL that much longer, if I have some extra experimentation in the code on godbolt that I leave out of my answer. (For readability, and because it's only interesting when you're looking at the asm output.)

I've only ever run into the 30k char limit once, on a question that caught my interest and kept me coming back to add more stuff I found (and did only a mediocre job of editing to keep it concise). I usually manage to keep things shorter than that, probably by enough that expanding godbolt links wouldn't have been a problem.

Most of the time I don't actually post separate godbolt links for different compile options, I just mention the effect and leave it up to readers to modify the compile options after following one of the godbolt links I do post. Partly that's for maintainability of the answer: if I change something in the code, I don't want to have to update all the links with different compile options. So again, that's a factor that keeps the amount of godbolt links in check, reducing the chance I'd want to include so many that the 30k char limit would matter.

19
  • 7
    Ah, I knew this would come up. See also: meta.stackoverflow.com/questions/295084/…
    – BoltClock Mod
    Commented Mar 23, 2016 at 5:17
  • 1
    We had another post about this, focusing on comments which are much more constrained than answers. Commented Mar 23, 2016 at 5:23
  • 2
    @BoltClock: Hmm, when I see a goo.gl link on SO, I assume it's to godbolt (from other context). In my answers / comments, I usual say it's a godbolt link, or that it's to "how the code compiles". But I think I understand more clearly the concern that the extra level of indirection disguises the link, so it could be anything (malicious, something the reader has already read, a Rickroll, ...). Ideally godbolt.org would have its own URL-shortening, so the links would still be godbolt links, but then he'd need a persistent database with backups to avoid losing all that state. Commented Mar 23, 2016 at 5:26
  • What would be the length impact of including the code and assembly output in the post instead of the link? (Yes, that would force an interested user to copy it themselves if they wanted to play with it.) Commented Mar 23, 2016 at 5:27
  • 4
    That's interesting that you can safely assume where a goo.gl link will take you. I suppose it depends on the tags you participate in. Every single goo.gl link I've come across on SO has either led to a spam site or a rickroll.
    – BoltClock Mod
    Commented Mar 23, 2016 at 5:28
  • 6
    @JeffreyBosboom: shortened URLs aren't allowed in comments?????? For the love of science, that's a huge problem. I leave godbolt links in comments all the time. Commented Mar 23, 2016 at 5:28
  • 1
    @JeffreyBosboom: All my answers do include the code and asm output I'm discussing (e.g. just the inner loop, when the asm for the whole function is bloated). It would be irresponsible to do otherwise. The godbolt link is there to let readers play with compile options / different compiler versions, and to demonstrate that it actually compiles. Sometimes I leave other example functions in the godbolt link, as an aside for readers interested enough to follow the link, but SO answers should never depend on the continued functioning of godbolt.org. Commented Mar 23, 2016 at 5:34
  • 9
    I am not sure what can be done about this. It's a very legitimate problem since we can't make these third-party services generate URLs of "reasonable length" instead of shortlinks just to comply with our policies. (It would be super nice if the developer of godbolt implemented that, but he doesn't have to just for our sake.) But the malicious use of shortlinks is so rampant that, save for evaluating every single shortlink before processing answers/comments (which would be unnecessarily expensive), we can't exactly start allowing them back in again.
    – BoltClock Mod
    Commented Mar 23, 2016 at 5:39
  • 4
    @BoltClock: the most obvious way for godbolt to implement it would be to redirect URLs like http://godbolt.org/shortened/XYZABC to http://goo.gl/XYZABC. Then godbolt doesn't have to worry about backing up the shortening map, and can still leave that to google. However, this doesn't help anyone, because spammers will just start using godbolt! The way godbolt currently works, it doesn't have to save user state at all. Using URLs that don't contain the full information content of the code and options would require storing it persistently, which is a major change. Commented Mar 23, 2016 at 5:46
  • 15
    Hi there, happy to add whatever needed to make this easy. For manageability and privacy reasons I decided not to store state on godbolt.org; so having a canonical URL with all options "in" it seemed more sensible. Then of course they got very long and unwieldy. I've toyed with using GitHub gists as the "backing store", but that's more work and it's not clear the godbolt.org URLs would be that much better; nor that they wouldn't also be blocked. Commented Mar 23, 2016 at 12:45
  • 2
    @BoltClock I understand the underlying rationale but it is not like the SO team did not know this would be an issue. I am disappointed that it seems like no reach out was done to see if there were work-around possible before just blocking. I also wish some more consideration would be put into making this a privilege, it would not be a total solution but would probably ease the pain a lot. Commented Mar 23, 2016 at 12:55
  • 7
    Aside: yet another irritation caused by the use of shorteners is that I can't really get objective data on how often this service is used!
    – Shog9
    Commented Mar 23, 2016 at 16:34
  • 1
    Related: some commenters think godbolt url looks "sketchy"
    – Krease
    Commented Mar 24, 2016 at 16:42
  • 1
    @Chris: That question had it in a code block, not formatted as a link. It was a terrible sketchy-looking question until the edit. But interesting point; I wonder if some people would mouseover and be scared to visit it. Commented Mar 24, 2016 at 20:55
  • 5
    The big problem on that question was that there was no code in the question. Should probably add godbolt to the same check we use for jsfiddle and kin, disallowing links to them unless there's a code block in the question.
    – Shog9
    Commented Mar 25, 2016 at 0:31

7 Answers 7

48

The big, big advantage of this particular service is that it contains all of the data necessary to work in the URL (compressed and encoded). I can't overstate how great that is: an awful lot of similar sites store the code in a database somewhere, indexed by a short string in the URL, meaning the link becomes useless if the database is lost or the service decides to age out entries.

This, coupled with the fact that Matt made this all open source, means that links to http://gcc.godbolt.org/ could easily remain useful even if that site drops off the face of the 'Net forever! All the information necessary to make this happen is contained in the posts that link to the site, in the links, rather than the site being linked to. This is the same philosophy that drove the creation of Stack Snippets.

...And using URL shorteners completely destroys this advantage. Now you're back to depending on an opaque key into some 3rd-party's database; if they ever drop the associated entry, you're screwed.

Please, just put the full URLs in the posts you're writing. Sure, they look huge, but they're not really all that long; for any non-trivial amount of code, the URL will be shorter than posting the code itself. And ten years from now, someone reading your Stack Overflow posts from a data-dump via a holographic display while on Mars could still make those links work if need-be...

See also: stats on the use of shorteners in comments

6
  • 7
    I already bought my mars tickets, so exited of reading SO on mars :D
    – Braiam
    Commented Mar 23, 2016 at 16:32
  • 30
    This answer makes a great case for reviving the suggestion of not counting URLs inside [text](url) in comments, where the char-limit is a real thing. I completely agree that having the entire information content right there in the URL is ideal. I think it clearly follows that allowing this to happen naturally in comments is a good idea, but that simply raising the total markdown char-limit to 2000 is not a good idea. Unlike Griwes, you will find many of my comments containing godbolt links. Every goo.gl link in comments I've posted is a godbolt link. I never use shorteners otherwise. Commented Mar 23, 2016 at 21:51
  • 4
    Yeah, I've been through your comments, @Peter - you're definitely making good use of the space there, which makes the shorteners all the more troublesome (not like anyone can go back and expand them even if they wanted to). A change to comment length probably isn't an easy fix here, but it would be a more elegant solution.
    – Shog9
    Commented Mar 23, 2016 at 22:17
  • Thanks for the upvote on "full URLs"; they're just really really giant despite best attempts to keep them small. If people really want them I should make the "permalink" optionally use the full URL (or at least to update the window.location with the actual current full URL). Thoughts welcomed (feel free to file issues on github) Commented Mar 24, 2016 at 4:19
  • I opened an issue for discussion of what godbolt.org can do to make things easy. Anyone else with ideas should contribute them there. Commented Mar 24, 2016 at 6:01
  • 20
    @Shog9 "massive support for banning these things" I got the impression there was massive support for banning these links in posts, not in comments. The original feature request had several upvoted answers stating agreement with the general idea, with the addendum that they should not be banned in comments specifically because of services like godbolt. (Even the top-voted answer suggested that they should not be banned in comments.)
    – Ajedi32
    Commented Mar 24, 2016 at 18:53
28

Maybe this issue could lead to a new feature on Stack Overflow. Godbolt could end up providing its own URL shortening feature, but this would not fix similar troubles for other popular developer tools sites which would not have such a feature themselves, while requiring longs URIs.

Stack Overflow could provide a feature named something like "external site URL shortener", which would generate and handle shortened URIs for external sites, but limited to a well-known external sites list.

This list could be expanded from the user's requests, with some voting system on those requests. Popular ones would trigger a moderator review for them to decide whether to accept the external site or not.

5
  • 11
    probably SO could shorten the URLs themselves, or collaborate with some service like they do to images with imgur. That way users would have an option to shorten their links by SO
    – gnat
    Commented Mar 23, 2016 at 9:27
  • Hi; that was my first thought! The alternative is making a more long-winded approach via a github gist or similar. Commented Mar 23, 2016 at 12:44
  • 11
    A really cool feature here would be something to take a block of code on SO and automatically generate a gcc.godbolt link from it. Maybe triggered by a special comment in the way Stack Snippets work.
    – Shog9
    Commented Mar 23, 2016 at 17:28
  • SO could automatically (or per setting) convert long links to an SO shorturl format when they are submitted in the wysiwyg editor.
    – Knossos
    Commented Mar 29, 2016 at 14:25
  • 2
    @Knossos: And that buys you the worst of all worlds. Commented Mar 29, 2016 at 16:18
19

For short links, Compiler Explorer (gcc.godbolt.org and related sites) now uses its own storage (backed by goo.gl's link storage). Links are now of the form: https://godbolt.org/g/MnTzHI which hopefully gives a better intention of the final destination of the link.

2
  • 2
    If such a link cannot ever go anywhere but to godbolt, well done. Commented Mar 29, 2016 at 16:20
  • 4
    Thanks; the links can indeed only go to godbolt.org (source is on GitHub if you would like to take a look!) Commented Mar 30, 2016 at 2:39
14

This isn't just a problem for Godbolt. As pointed out in some of the answers to the original feature request, this change creates problems for a number of online compiler type services.

Here's a list of just some of the services negatively impacted by this change:

Also I'd like to point out that while this change does pose some problems for links to longer code snippets in answers and questions, the problem is much worse for comments. Even relatively short code snippets produce URLs that are much too long for the 600 character limit for comments, meaning links to godbolt, play.rust-lang.org, etc are now effectively banned in comments. This is a pretty significant issue, especially in communities where these services are frequently utilized.

Note that I discovered the above list of services by running a Stack Exchange data query for comments with goo.gl links in them, parsing the first goo.gl link in each comment, following the redirects, then filtering by links over 600 characters in length. (The maximum comment length.) In other words, every single one of the services in the above list has had at least one StackOverflow user use goo.gl to get arround the comment length restriction when posting a link to it.

I'm not really sure what the best solution to this problem is for questions and answers (Maybe raising the character limit would help?), but for comments I think the best solution is to just not block URL shorteners there in the first place. Comments already require 50 rep to post so there's a significantly less chance of spam; and comments are meant to be more ephemeral than posts anyway so the chances of broken links becoming a problem are also lessened.

Alternately, Stack Overflow could start its own URL shortening service, or partner with an existing one (like it did for images with Imgur). This would solve the problem of broken links, and would allow SO to continue to moderate links for spam. Links could even be expanded when rendered, so hovering over the link actually displays the real URL to users instead of the shortened one.

7
  • 4
    I disagree: allowing long urls in comments, and not having them count as part of the 600, is by far the best solution. URL shorteners are apparently a real problem, and they are subject to link-rot if their database loses the key-value mapping. Shog9's answer on this question makes this point. However, good point that this isn't just godbolt. Commented Mar 24, 2016 at 21:00
  • 2
    Also, can't 1 rep users still comment on their own questions? (and answers?) Still, re-allowing shorteners in comments until SO implements long URLs in comments somehow would be a good stop-gap. Commented Mar 24, 2016 at 21:09
  • They can, and it's not uncommon to see spammers try to make use of this (putting links in comments on their own answers), @Peter. So we don't want to ignore this. At the same time, hurting folks using the site correctly for the sake of spammers isn't great either.
    – Shog9
    Commented Mar 25, 2016 at 0:29
  • @Shog9 I guess the straightforward solution to would be to not allow shortened links from users under 50 rep. I do kinda agree though that not counting links towards the character limit in comments would be a better solution, assuming it's feasible to implement.
    – Ajedi32
    Commented Mar 25, 2016 at 0:40
  • 1
    @Shog9: I did say "stopgap", i.e. until a long-term viable solution can be designed an implemented. Would re-allowing shortened URLs in comments but not posts keep the spam workload down enough to give you guys enough extra time to work on a long-term solution? I'm still strongly in favour of not counting [](url) character as part of the 600, but an SO built-in link shortener that would allow short URLs in markup while still rendering full URLs in HTML for people to mouseover would be fine, too. The current situation is a big problem, and a short-term workaround would be appreciated. Commented Mar 25, 2016 at 2:08
  • 1
    For godbolt specifically, plans are underway to allow URLs like http://gcc.godbolt.org/s/F02h38, using the google API to expand the goo.gl link and check that it's actually to godbolt.org. So that will work as a stopgap for godbolt, but doesn't fix the comment problem for other sites, and still leaves URLs subject to link-rot if goo.gl drops the entry. Commented Mar 25, 2016 at 2:26
  • 1
    I've done some more detailed analysis on the use of shorteners in comments. It's mildly depressing.
    – Shog9
    Commented Mar 29, 2016 at 22:33
7

Boltclock commented that

malicious use of shortlinks is so rampant that, save for evaluating every single shortlink before processing answers/comments (which would be unnecessarily expensive), we can't exactly start allowing them back in again.

So that leaves us with no choice but to use non-shortened links.

SO could expand them for us on the fly, since sites like godbolt often make it easier to copy/paste a shortened URL.

The best defence against link-rot due to losing the mapping from short to full URL is for the url to encode the full state. This will inherently be long, but in the case of godbolt, still short enough to easily copy/paste. It might be possible for godbolt to use a somewhat shorter encoding scheme, but without introducing a database on their end (which currently the site doesn't need at all), it's not possible for the URLs to be very short. The Kolmogorov complexity of the code + compiler options is a lower bound, and it has to be ASCII/urlencoded.

The "save as" button on godbolt works via cookies, IIRC, so the key-value store is client-side.


The 30k char limit really only needs to apply to the rendered output, not the markdown. However, length-counting the rendered output is presumably non-trivial. I propose that URLs in [text](url) and footnote [text][1] / [1]: URL form not count against the 30k char limit.

To avoid abuse (e.g. backing up encrypted & Base64ed bulk data into an un-displayed URL in a post), something like a 60k total markdown char limit should be imposed (including URLs).

To avoid server-side performance overhead from this, check post.markdown_length() < 30k before finding URL text and subtracting its length. Almost all posts will still have a markdown length shorter than 30k, so it doesn't matter if the text parsing needed to distinguish URL characters vs. non-URL characters takes some CPU time. The vast majority of posts will never need to run it.


This idea has already been suggested for comments to address the godbolt issue, and before that multiple other times.

Again, a total length is needed to avoid abuse (either DOS / disk-fill attacks or mis-use for data storage). The same fast-path check of the markdown length applies, but a modification to the as-you-type character count would also be good.

6
  • 6
    30k is, AFAIK, rather arbitrary; if folks are legitimately hitting that limit on a regular basis, it could just be extended.
    – Shog9
    Commented Mar 23, 2016 at 6:58
  • @Shog9: I think the only way I'd hit it would be if one of my long and rambling answers has many godbolt links with a lot of code on godbolt. I don't expect it to be a problem for me in practice. I mostly went into detail about it because implementing don't-count-URL-chars idea for comments is now necessary. Commented Mar 23, 2016 at 7:03
  • 1
    I'm not convinced about the rationale in the first quote. Non-shortened URLs may be malicious or spam too, and it's easy enough to make the URL look innocent at first glance. What was the reason for not using reputation ? (i.e. high rep users can post any link; low-rep users can only post links to whitelisted sites).
    – M.M
    Commented Mar 23, 2016 at 13:11
  • I don't really understand Boltclock's comment. If evaluating the shortlink is the only way to allow genuine links while blocking malicious ones, it's not "unnecessarily expensive", it's necessarily expensive.
    – Dan Hulme
    Commented Mar 23, 2016 at 13:17
  • 1
    @DanHulme: I assumed he meant "evaluating by a human" (reviewer or moderator), not just hitting the shortening service to get the redirection. But that is a good point. If non-shortened URLs are still ok, then why not just expand them programmatically? Commented Mar 23, 2016 at 13:25
  • 2
    Trivia: we already invested a significant amount of time trying to make link validation work in the past (a task which also involved trying to resolve every link posted). It was not a successful project. If rewriting links was feasible, we'd have done that long ago in preference to blacklisting.
    – Shog9
    Commented Mar 23, 2016 at 14:54
2

I think the best solution would be to ask the people behind Godbolt if they could do the URL shortening on their side, e.g. https://godbo.lt/asdfxyz123. If that was only available for Godbolt links and not a general-purpose URL shortener it would probably not get blacklisted.


To avoid the need for storage, why not integrate it with Gist? It has an API so the site could create a secret Gist containing the code and setting and then generate a link containing the Gist ID. When opening that link it'd make an API call to retrieve the file.

15
  • 17
    This would require a massive change to the architecture. Currently, Godbolt is completely stateless, all information is contained in the URI: the complete source code to be compiled, the compiler version, compiler options, etc. Everything. There is no server-side storage at all. Commented Mar 23, 2016 at 8:47
  • Why? As far as I can tell, this only requires a minor addition to the architecture; everything can stay as it is, except there should be an extra service that expands links. It could even live on a different machine! @JörgWMittag
    – 11684
    Commented Mar 23, 2016 at 8:58
  • 26
    @11684: At the moment, there is no data to lose, since there is literally no data. It doesn't matter whether you put the service on the same machine or on a different machine, something has to persistently store a mapping from short URIs to long URIs. Now, all of a sudden, you are storing dynamic user-supplied data of unbounded size, and users get very cranky if their data is lost. Which means that you have to introduce backups. You also have to maintain, update, upgrade, administer, and secure some form of database. You increase the attack surface of your site. And for what? Just because … Commented Mar 23, 2016 at 9:07
  • 22
    … some random third-party site imposes silly restrictions on its own users, they have to expend manpower, resources, and money to implement a feature they neither need nor want? Commented Mar 23, 2016 at 9:08
  • Wow. I hadn't thought of It that way. Thanks for the insight!
    – 11684
    Commented Mar 23, 2016 at 9:20
  • Updated my post with a possible way to avoid the need for storage on the site while still not requiring an URL shortener.
    – ThiefMaster Mod
    Commented Mar 23, 2016 at 10:18
  • 11
    Bless you for thinking there's "people" behind godbolt.org :) Yes, GCC Explorer is currently stateless for exactly the reasons Jörg says; but it could be changed. It's open source: github.com/mattgodbolt/gcc-explorer - I'm definitely considering gists; but then the URLs to godbolt.org must encode a gist ID (so would probably be of the form gcc.godbo.org/g/mattgodbolt/45cf779ccd32bbaae5ca ... who's to say SO doesn't block that?) Commented Mar 23, 2016 at 12:49
  • You don't need the username to reference a Gist - at least for normal HTTP access the IDs is enough (you are redirected to the URL containing the gist's owner), so that extra url segment wouldn't be needed. And I don't see why it'd be blocked - it's not an URL shortener and I don't see any other way to abuse it
    – ThiefMaster Mod
    Commented Mar 23, 2016 at 13:08
  • 5
    Really impressed with your work, @MattG - I think the URL scheme here shows an unusual amount of consideration for the often-unreliable nature of the Internet. I'd hate to see that advantage squandered.
    – Shog9
    Commented Mar 23, 2016 at 16:25
  • Thanks! Working on a gist version now; hopefully that'll get around the issues. @Thiefmaster you're right about the username too. Commented Mar 24, 2016 at 2:17
  • 1
    Have implemented gist version; but not pushed. Mainly as GH rate limits IPs to 60 requests/hour, which might not be enough :( Thoughts welcomed! Commented Mar 24, 2016 at 4:17
  • Wouldn't it be possible to do the requests client-side? That way the 60 reqs/hour would most likely not be an issue (and I guess when using a proper token the limit is much higher anyway?)
    – ThiefMaster Mod
    Commented Mar 24, 2016 at 8:18
  • Doing the requests client-side with XMLHttpRequest or similar would throw a security exception because Gist responses lack an Access-Control-Allow-Origin header, unless by "client-side" you recommend to download and install a native application. Commented Mar 24, 2016 at 16:30
  • developer.github.com/v3/#cross-origin-resource-sharing - CORS should be possible
    – ThiefMaster Mod
    Commented Mar 24, 2016 at 18:03
  • 1
    The requests are done client-side. 60reqs/hour would definitely not be enough for one client as the request is used to look up a URL too and I know I click more than 60 an hour :) There's no way to get a proper token without every GitHub user logging in themselves : for client applications one can't use tokens without revealing those tokens to the whole world (and thus making them pretty much useless)! Commented Mar 25, 2016 at 14:20
1

Since we already attempt to exhaustively blacklist all known URL shorteners, why can't we whitelist some known shorteners that have a preview feature? Just well-known ones should suffice.

Take goo.gl for instance. Just adding a + sign at the end of shortened URL turns it into preview link. So goo.gl can be used when it ends with the + sign (or even add a + sign automatically to links from goo.gl).

This way people will see the original URL just like what is shown in the status bar. If the preview link is nested with another URL shortener? Well, that should raise some flag and people will choose to follow the link or not by themselves.

3
  • 1
    That doesn't seem to be how that actually works. [foo](http://goo.gl/ac6GdX+) foo makes a link to the google-analytics page for the shortener. You can edit that into an answer even if you can't submit it. In any case, the HTML on a SO page would have to include the title, which means the SO server has to resolve it (probably on save, not on every view), so unless I'm missing something, this isn't really different from my proposal for SO to expand shortened URLs for us. It could only count the shortened URL length. Commented Mar 23, 2016 at 9:38
  • @PeterCordes try that on the main site. This blacklist doesn't affect meta.
    – ryanyuyu
    Commented Mar 24, 2016 at 16:47
  • @ryanyuyu: I mean you can have a goo.gl link in a post while editing, so you can click on it. You can't save a post that way, but you don't need to because it's fully formatted. Commented Mar 24, 2016 at 20:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .