69

After reading bluefeet's answer on this question, specifically the last line stating:

And trust me when I say that SE is blocking a lot more than we actually see.

I am curious about how many posts are being blocked daily. Is there such a count that's keeping track or is there a way that I could find that out on my own? I'm just very curious about that :P.

5

2 Answers 2

100

So here it is, the year in SPAM across the network:

Lovely year it was

Orange is how many flags it took from users across all sites to keep [blue] pieces of spam from appearing in the system. On most days, it's well under a thousand flags, keeping up to nearly 50,000 pieces of junk from ever gracing our eyes.

There are other things, too - such as honeypots. Posts that receive an abnormal amount of edits rejected as spam become traps - any new hosts caught doing anything bad to them get an instant blocking blow.

It's not perfect and no system is - you'll still see some Indian psychics (coming from a new mobile network each time) letting you know their services are available, and some sites are frequent targets of 'snow shoe' rings (or, spam cartels that have an army of millions of infected PCs and phones working for them).

But, we keep out orders of magnitude more than what comes in, without having to ask volunteers to do any more work, and that makes us feel pretty damn good. It should make you feel good, too, for doing even a small part of it.

Every flag does count :) So does every spammer your moderators destroy.

14
  • I flag about 1-3 times daily. Commented Feb 21, 2015 at 14:49
  • 32
    But, if you block all these spammers I'm going to miss out on the thrill of destroying their accounts ;)
    – ChrisF Mod
    Commented Feb 21, 2015 at 15:31
  • 8
    Tim, I might just be slow, but I don't quite understand your numbers. For the blue numbers, are these posts that are blocked by an automated system, before ever appearing on the site? Or posts that are deleted after being flagged as spam by users? Particularly the part "[..] flags it took from users [..] to keep pieces of spam from appearing [..]" sounds confusing to me. If something is flagged by a user, by definition it has appeared on the site. Otherwise it couldn't have been flagged. Commented Feb 21, 2015 at 19:39
  • 1
    Huh. @Reto is right -- how come it does not take more flags to mark more spam? In that case the orange line should rougly run parallel with the blue one.
    – Jongware
    Commented Feb 21, 2015 at 22:47
  • @RetoKoradi posts that are prevented before even posting (on some cases, they are shown captchas).
    – Braiam
    Commented Feb 22, 2015 at 2:16
  • 1
    @RetoKoradi The idea is that the information supplied by flaggers ("this is spam") is used by the system to block future attempts by the spammers, even if they create new accounts.
    – user3717023
    Commented Feb 22, 2015 at 6:19
  • 3
    Just curious, are there any statistics on which tags are the most common targets?
    – Radiodef
    Commented Feb 22, 2015 at 6:25
  • 6
    Any theories on the two spikes of spam? One in August and one in November?
    – Laurence
    Commented Feb 22, 2015 at 12:56
  • 5
    @RetoKoradi Blue means spam that was blocked due to something the system learned from flags. Depending on what has sights on us, 600 - 1k flags can be enough to block 10k, or up to 50k spam posts. On average, no more than 1k or less flags (network wide) are needed to handle the brunt of even large attacks, as that's more than sufficient to adequately train the system on what's coming in.
    – user50049
    Commented Feb 22, 2015 at 15:07
  • 1
    @TheShiftExchange Those are generally aggressive, but not really sophisticated bot nets that try to brute force their way in by switching networks and such. We deal with that pretty effectively, a secret in plain view is that the spam layer is shared among all sites - what you do on Ask Ubuntu also blocks you on SO, for instance. Spammers seldom figure that out.
    – user50049
    Commented Feb 22, 2015 at 15:10
  • (This would all be much easier to figure out on your own had I posted the logarithmic graph, but that's even less easy to see with a one year sampling).
    – user50049
    Commented Feb 22, 2015 at 15:12
  • 11
    @TimPost Working on a system that successfully blocks such huge attacks and so much stuff, in a learning manner, against attacks by large systems must feel like being a true programming super hero. That has got to feel good looking at this graph for you and your team.
    – user2076675
    Commented Feb 23, 2015 at 5:08
  • It's nice to see that the spams are being taken care of properly :)
    – John Odom
    Commented Feb 23, 2015 at 17:56
  • 1
    @MediaWebDev Well, maybe. Although I'd be surprised if the SO team were stupid enough to waste resources trying to roll their own learning anti-spam system. Since those already exist for free. So, probably, it feels more like being a true system administrating super hero. Could be wrong, though Commented Jul 7, 2015 at 20:24
2

Tim's graph and internal look at the spam to flag relation is very nice.

Just for extra consideration from an averages standpoint you can look at the current question primary key, the current amount of total questions, and then apply that percent to the daily questions asked for an idea of how many questions are removed or prevented. This would include non spam posts as well so it is not the same pinpoint accuracy.

id at time of posting:   2868228
total questions:         8,904,341
percent removed:         69%
questions per day:       7.8k

average questions
removed per day:         5.3k

While this is an overall average from the past 6 years, it is still pretty accurate if you look at Tim's graph on average. The peaks and the path are still interesting.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .