100

I've seen a lot of spam lately that looks like this deleted post. Essentially, it's a flat-out spam, without any attempt at presenting it as something else.

Although post like that get deleted within minutes, it looks like they get indexed by Google before deletion. I was surprised to see the post linked above within the top ten search results for the keywords it's trying to promote!

Does Google re-crawl the site on its own, or does SO use an API to tell Google about new pages? If SO drives the discovery of new pages, is there a way to delay indexing of new questions until after they get viewed by the community? For example, delaying by five minutes, and excluding deleted questions, would make the site virtually useless to spammers of this kind, because their message would get zero visibility.

18
  • 47
    How about we block crawlers from a question for the first 5 minutes.
    – Mysticial
    Commented Jun 3, 2015 at 17:15
  • 71
    These particular hotline spammers have been attacking the site for months, using an incredibly large botnet. Stack Exchange is one of the few places actually cleaning up their mess: google.com/… . I'm not sure they even care how long their spam stays up, they're just contracted to spam it in X places. Also, I don't think SE does any direct feeding of information to Google, the site is just popular enough to be crawled constantly.
    – Brad Larson Mod
    Commented Jun 3, 2015 at 17:46
  • 6
    The premise that spammers actually care how effective their spam is seems somewhat questionable. Often they really don't seem to take even fairly basic steps to improve effectiveness and keep things around longer, so they are either incompetent enough not to notice that their posts are never being crawled, or somehow it's beneficial anyway. SO already nofollows those links, so there's no googlejuice being transferred. Commented Jun 3, 2015 at 18:24
  • 5
    they're just contracted to spam it in X places That's probably true, but I can't imagine what the company that pays these spammers actually gets out of it. I can't imagine there are many people who pick a printer repair company because they saw what is clearly bot-generated spam.
    – BSMP
    Commented Jun 3, 2015 at 19:30
  • 13
    @BSMP - What, you're not going to call them up after seeing this: i.sstatic.net/namZJ.png ? (Also note all the variants that they use to work around various content blacklists, filters, etc.)
    – Brad Larson Mod
    Commented Jun 3, 2015 at 19:56
  • 3
    @BradLarson - No, I'm afraid that stretching out the s-word just isn't a compelling argument for getting my printer repair business. That might have worked better if the number was for a plumber.
    – BSMP
    Commented Jun 3, 2015 at 20:01
  • 2
    Also, YouTube has thousands of them, see the Google search Brad linked. Seems like Google doesn't care about scammers abusing multiple of their products. Commented Jun 3, 2015 at 20:29
  • 5
    It is ridiculously easy to create an account at SO and post a question. By spammers and question-banned users alike. Why SE won't address this is beyond my comprehension. Commented Jun 3, 2015 at 21:33
  • 33
    @HansPassant - For spammers, there are significant hurdles to creating accounts. Captchas and the like keep out almost all true bots, and intelligent IP blocks, etc. do an incredibly effective job at blocking human spammers from creating new accounts: meta.stackexchange.com/questions/228043/… . My screenshot above was every single instance of spam that Stack Overflow received in three hours. Only 10 spam questions making it through in three hours is pretty impressive for a site this size.
    – Brad Larson Mod
    Commented Jun 3, 2015 at 22:22
  • 2
    @Brad - that doesn't make me feel the least bit better. It only tells me that this isn't a bot network. And that SE would have a shot at identifying the source from the IP address but doesn't. Do you have any idea how many of these illicit accounts you never detect? Well, it is unknowable, isn't it? Commented Jun 3, 2015 at 22:48
  • 1
    @BradLarson You should remove the referenced link to google as this gives out a number that bills the living hell out of people - and we all know people are always very curious creatures.
    – JonH
    Commented Jun 4, 2015 at 16:36
  • 2
    Hmya, SE professes to care, that's why they ban users that contribute poor content. And I care. A lot. Moderating content at SO is not a fun job, it turns into a pointless never-ending Sisyphean job if it doesn't actually accomplish anything. Commented Jun 4, 2015 at 16:55
  • 16
    The problem isn't that we can't identify the source, or even that we don't block them - it's that IP addresses are quickly becoming an unreliable tool for this, @Hans. Large (sometimes country-wide) NATs and widely available anonymizing proxies both aid spammers and other ne'er-do-wells in avoiding these blocks and penalize others who get caught in the crossfire. We still use this information as best we can, and currently block or throttle constantly based on it - but it isn't and cannot be 100% reliable. Where signal is weak, human-review is essential in getting this stuff off the site ASAP
    – Shog9
    Commented Jun 4, 2015 at 21:45
  • 1
    @Shog9 what about requiring an account and email verification for anyone that uses a large NAT or anonymizing proxie Commented Jun 9, 2015 at 7:52
  • 1
    That'd work if these spammers didn't apparently have all the time in the world to burn: the bulk are already registering accounts with email verification. @Ian
    – Shog9
    Commented Jun 9, 2015 at 16:16

5 Answers 5

48

Seeing these things turn up for their target keyword is coincidence, an artifact of their efforts on other sites, it doesn't last long enough on our sites to be indexed meaningfully.

Google (and others) do frequently crawl our sites, and given that these are deleted within minutes (sometimes even seconds), subsequent requests by crawlers will almost always yield a 404. We show the content of deleted posts to users that have the privilege of seeing it, but the HTTP response on them is a 404 (not found), crawlers immediately stop caring about whatever else is on that page once they see the response.

tl;dr - by the time they hit our sites, they're already indexed, because most of the Internet doesn't do as good of a job as we do when it comes to handling spam. We're doing more things to shut down these enormous 'snow shoe' spam rings that use giant botnets. Generally speaking, by the time you see spam, the origins have already been blocked, cleanup just remains.

We're definitely not helping them in any way, but we surely wouldn't mind seeing less artifacts of it, so we're working on some improvements to how we treat it on a network level.

11
  • A recurring theme in these recent ones seems to contain the phrase "Support phone number" repeated a dozen times. Maybe some metric of that occurring could be used.
    – Travis J
    Commented Jun 4, 2015 at 17:14
  • @TravisJ We don't currently do as good of a job as I'd like when it comes to analyzing stuff as we block it. I've been opposed to regex blocks because they're so easy to get around. Learning more, on the other hand .. I have some ideas.
    – user50049
    Commented Jun 4, 2015 at 18:45
  • 5
    (Keep in mind, Stack Overflow's scale is massive, and performance is a feature)
    – user50049
    Commented Jun 4, 2015 at 18:49
  • 2
    Yeah, I do appreciate the performance as a feature. I just figured that there could be ways to screen the content in areas where there was some signal (first spam flag on a post for example could trigger the the examination or whatnot). I agree just blindly regexing every question posted is probably a bad idea. On the other hand, there should be some signal to check, right? Only check the question using some metric if it was from a 1 reputation user whose account is less than 1 hour old? Or is that even too often of an occurrence? Anyway, I think you guys do a fantastic job with spam.
    – Travis J
    Commented Jun 4, 2015 at 19:19
  • While we are on the topic, what is (are) most effective action(s) for us to take to kill those as soon as possible and disable the accounts used? Downvote, flag as spam, something else?
    – PM 77-1
    Commented Jun 4, 2015 at 22:47
  • @PM77-1 Flagging as spam is all it takes. If you're talking in a chatroom when this happens, pointing spam out to others may also help.
    – user3717023
    Commented Jun 5, 2015 at 4:03
  • We show the content of deleted posts to users that have the privilege of seeing it, but the HTTP response on them is a 404 (not found) : I just clicked the link to that deleted post and the HTTP response was 200 (OK).
    – Adam Arold
    Commented Jun 5, 2015 at 13:42
  • You could use this wonderful captcha system to block spammers.
    – jkd
    Commented Jun 5, 2015 at 14:52
  • 4
    @AdamArold That's because you're logged in with high rep. Google won't be - load it in a private browsing session, and you get a 404 with no trace of the original content.
    – IMSoP
    Commented Jun 5, 2015 at 15:21
  • 4
    your tl;dr; is almost as long as the rest of your response :)
    – Fortega
    Commented Jun 5, 2015 at 19:56
  • 2
    Spam may be deleted "in seconds" on Stack Overflow, but I have watched spam posts persist for many hours on lesser-trafficked SE sites. (¡The Horror!). Commented Jun 5, 2015 at 20:45
25

Keeping Google away from brand new questions (especially those that trip enough spam heuristics) for a few minutes, to let the community review them first, sounds like a perfect win-win scenario:

  • We become a less tempting spam target.

  • Google sees a cleaner, more high-quality question feed, and so can serve better results to users.

The down side seems pretty negligible — basically, we may lose a few visits from people who just happen to Google for a question minutes after it was asked. Most sites live with that anyway, just because Google's crawling is not instantaneous; it's only because Google loves us so much that we do often get a pretty good approximation to instant indexing.

The tricky part is implementing this effectively. Basically, the problem is that we need to make these questions visible to active users, so that they can spot the spam and flag it (and potentially fix other issues, e.g. with legitimate but poorly written questions), while somehow temporarily hiding them from Google, and doing this in a way that doesn't prevent the real questions from being properly indexed later.

Now, I don't actually work at either SE or Google (although, you know, I am currently between jobs...), so what follows is necessarily based on indirect information and reasoned conjecture, but as far as I know, there are three main ways Google uses to discover new posts on SO:

  1. They use the XML sitemap. According to our robots.txt file, SO does have one, even though visiting that URL yourself just gives a 404 error; apparently, it's set up to be only served to Google, or something.

  2. They follow links from the main page (and other pages they've crawled), just like a normal user browsing the site would.

  3. They follow the recent questions feed.

Now, keeping new (and spammy-looking) questions out of the XML sitemap for a few minutes is pretty trivial (i.e. just a simple matter of programming); the XML sitemap is only meant for search engines, so changes there have minimal effect on normal users.

The main page issue is also solvable — in fact, AFAIK we already have such a delay implemented there: new questions will first show up only in review, and on the new questions list, until they're about five minutes old or so. We might want to tweak our robots.txt and/or meta tagging a bit, to keep Google from crawling that new questions page, but that should also not be a major problem.

The Atom feed is a problem, though. Again, I'll stress that I'm not an insider in either company, but as far as I know, Google does make heavy use of such feeds for discovering new content, and so, for the proposed 5-minute "embargo" to be effective, we'd really need to implement it for the feeds as well.

But those same feeds are also used by normal users. Indeed, while I don't really use them myself, and so have no first-hand experience to report, I do suspect that we probably have active users who use those feeds to find new questions to review and improve, and whose user experience could be significantly hampered by hiding the very newest questions from the feed. (Or, then again, maybe not. Perhaps this calls for some real UX testing from SE's part?)

That said, one potential solution could be to only show very new questions in the feed for logged-in users. Presumably, Google doesn't have an account here, while most (all?) people actively looking for brand new questions to review and answer do. The only potential issue is that some of those users might be e.g. using some kind of an external RSS/Atom reader gizmo whose requests don't actually carry the cookies necessary to tell that the user is logged in.

Anyway, to summarize, I'd say that:

  • this is a good idea, even if it obviously won't stop all spam from getting on Google;
  • SE is already doing something like this, for the front page;
  • it would probably be useful to implement such delays more widely;
  • alas, some of the places we'd need to implement it for, like the recent questions feed, require some careful thought to prevent unwanted user experience side effects.

(Ps. One thing we almost certainly don't want to do is serve a 404 status, or a meta noindex tag, for new posts. While this would keep them out of Google's index, and while Google likely would recrawl them eventually, that kind of recrawling is not typically a high priority for Googlebot, and so some of those questions might take weeks or months to get properly indexed. Basically, while we may want to delay Googlebot's discovery of new questions by a few minutes, we should probably not be denying it access to any questions it does discover.)

1
  • "new questions will first show up only in review" -- as far as I know, this happens only to questions that are automatically recognized by system for triage. If system is more or less good at automatically recognizing potential spam, it probably matters less that the rest is shown immediately
    – gnat
    Commented Jun 5, 2015 at 6:41
11

I have been seeing these too. Especially on Meta Stack Exchange at night.

enter image description here

Some of these questions remained for 10 to 30 minutes because it was "after hours" so to speak. I think that Stack Exchange could use some signal here that is present.

If a user has had two spam flags accepted in the past hour, and they flag a new question from a new user (made in the past day) of 1 reputation as spam then it should only take one other flag to remove the question immediately.

I think this would basically deputize helpful users to quickly dispatch spam if it begins to accumulate and would only kick in under duress.

6
  • 11
    I'm not a big fan of giving elevated spam flag powers to people. Too many times I've seen people start spam-flagging well, then start to use spam flags on innocent posts. This can get out of hand quickly, and I've seen a bunch of spam and offensive flag abuse lately. Meta.SE is a bit of a special case, because it doesn't have the same kind of moderator crew that other sites and Metas do.
    – Brad Larson Mod
    Commented Jun 3, 2015 at 19:46
  • 3
    @BradLarson - Well you would definitely have a better insight for that than I do. I guess all I had to go on was my own experience so that is where the suggestion came from :)
    – Travis J
    Commented Jun 3, 2015 at 19:47
  • 1
    Or perhaps something less extreme: if the user has had many helpful spam flags, one spam flag immediately takes it off the front page, (just like 3 flags do right now), and hides it from Google's crawlers. Commented Jun 4, 2015 at 6:12
  • 9
    @JonasCz: Taking it off the front page is what we don't want, since that slows down or completely stops the spam-flagging process.
    – Ben Voigt
    Commented Jun 4, 2015 at 13:40
  • 1
    @BenVoigt, Maybe take it off the front page only for unregistered users and users who don't have the 15 rep required to flag. Commented Jun 6, 2015 at 8:39
  • @BenVoigt But hiding it from crawlers makes sense, at least temporarily. At the same time, a voting on spam could be started, and if someone with a non-negligible reputation votes "non-spam", the normal behavior could be restored, until it gets resolved definitively.
    – maaartinus
    Commented Jun 9, 2015 at 14:26
5

With over 250 helpful spam flags on SO, I'll just throw the experience card and say that these spam waves are definitely not only lately. In fact, one of the first major meta posts featuring the spam deluge dates back to 2013.

Now to address your concern - While Google's webcache does cache these spam posts in time before deletion (As well as normally deleted posts, that's how we get around the 10k rep limit to see deleted posts until we reach that privilege) and index them, the impact is negligible because Google's algorithm has been constantly tweaked to not take links as important for SEO.

Additionally, massive amounts of spam are already blocked before they get through. As a tip of the iceberg, communities can flag them on their own to prevent them from lasting.

1

I think the way to deal with this is to ask what legitimate role phone numbers have in actual posts (and not, say, user profile pages).

Basically, some database / text data processing questions will involve data that includes phone number columns, and may involve sample data / test cases that include numbers structured as phone numbers. These usages don't depend on the data containing any particular phone number, just a certain structure.

It seems like it would be safe, then, to randomize numeric data structured as phone numbers, thus reducing the conversion rate of spam posted to Stack Overflow to exactly 0.0% (this was what the question wanted, right?).

In case there are some legitimate scenarios I haven't thought of, apply randomization only to posts by users who are either unregistered or have less than 1000 reputation points.

This is, of course, only an escalation of the arms race, and we'll certainly see phone numbers obfuscated as part of the text much the same way personals ads on e.g. Craigslist try to sneak phone numbers past filters, but that obfuscation eliminates browser recognition of phone numbers also, and without the ability of users to click-dial, the conversion rate is limited to users dumb enough to not recognize a scam spam when they see it yet smart enough to deobfuscate the number.

Or, when toll-free (callee pays) numbers are found in posts, they could be auto-dialed, costing the spammers real-world money. But this could too easily be used to harass legitimate businesses, so forget it. On the other hand, a list of phone numbers found in posts closed as spam probably has value to e.g. blog anti-spam engines.

That's probably the ideal situation -- if you post spam to Stack Overflow, thousands of spam blog comments you successfully snuck in to other sites over the previous week immediately disappear.

8
  • 3
    So how long until some greyhat writes a bot to spam SO with numbers in order to get those posts removed from the wider web?
    – jscs
    Commented Jun 4, 2015 at 4:31
  • "click-dial"!!! That's the best explanation of the reason to post such an obvious spam, thank you very much! Commented Jun 4, 2015 at 12:10
  • 2
    "Or, when toll-free (callee pays) numbers are found in posts, they could be auto-dialed" - This may not such a good idea: consumer.ftc.gov/blog/… , as several of these numbers look like toll-free numbers but will charge you a huge sum when you call. That's part of the scam these hotline folks are running.
    – Brad Larson Mod
    Commented Jun 4, 2015 at 14:25
  • @BradLarson: Your link has nothing to do with toll-free numbers.
    – Ben Voigt
    Commented Jun 4, 2015 at 14:27
  • 4
    Several of the hotline spammer numbers look like valid toll-free numbers (8XX area codes), but are in fact pay-toll numbers. You'd need to be very careful as to which numbers you'd call. I think that's a large part of what's driving their particular spam. I remember someone said they called one of these out of curiosity and racked up a huge phone bill.
    – Brad Larson Mod
    Commented Jun 4, 2015 at 14:35
  • 1
    @Brad: 8XX numbers really are toll-free. Do you mean 8XY?
    – Ben Voigt
    Commented Jun 4, 2015 at 15:01
  • 2
    @Ben: what, you mean when it says "See Page XX" in a book, they knew the page number once typeset was going to be a small multiple of 11, but didn't know which one? ;-) Commented Jun 4, 2015 at 22:52
  • 1
    I've tried that. Blocking phone numbers using regex. They change the format. They replace numbers with characters. They introduce whitespace. These are not bots - these are people trained at regex avoidance. Ended up backfiring - meta.stackexchange.com/questions/258317/…
    – Oded StaffMod
    Commented Jun 9, 2015 at 22:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .