37

I recently came across some suspicious (plain nonsense) answers sharing the same characteristics:

The pattern

  • all answers have arbitrary code snippets – remotely matching the topic: e.g SVG, JS, HTML etc
  • no explanation/description
  • there is no response to comments
  • all these responses were posted at short intervals

Why this strategy is quite "clever" or efficient

  • answers with snippets appear unsuspicious for members reviewing flagged posts in a review chain
  • the text length is OK so the community bot won't find an obvious issue
  • seriously: do we always thoroughly test every posted code? As commented by @TylerH we should always pay attention to details when reviewing and should't be tricked by at first-glance "plausible" code

Flagging

I already flagged these via custom In need of moderator intervention option.
It's perfectly clear that we must be very cautious before assuming a user may be a Bot/AI or Spammer with malicious intententions. It's also clear, moderators can't always evaluate if an answer is

  • "just not great/very helpful" but not harmful – the community should decide via downvotes
  • very-low-quality (imho, the most quirky/ambigious flagging category)
  • not an answer (usually quite obvious, if the answer raises a new question or doesn't provide a new solution)
  • Spam (actually not always easy to distinguish from lackluster "link-only" answers)

But I'm 100% sure we're facing a new (or maybe not that new) Bot/AI strategy based on circumventing the current SO heuristics which are mainly focused on missing code examples, too short text content or just indications of spam.

Problems with the current flagging system

  • report text length limit is too low: It is quite tricky, to report a more complex issue precisely within the current limits. Here is my report pretty cramped into the allowed text length:

Bot: The (nonsense) answers by user:26995695 never include any description/explanation but always includes a "remotely" plausible snippet (see also previously rejected flag for user:20786021). These snippets completely ignore present OP's code examples – even snippets – you would usually refer to as a human community member. It's a new bot strategy to circumvent common heuristics (too short description, no code)! Please investigate on it.

While keeping reports as concise as possible is obviously a good idea – it makes reporting rather complex issues overly complicated – especially if you're not a native speaker.

Suggestions

  • tweak the publishing requirement: When posting a new answer, the system may require at least some portions of non-code text. If an answer is code-only the form may simply reject submission. This may also have the benefit to better detect Bot/AI generated content due to characteristic wording patterns (e.g we can pretty easily detect chatGPT answers this way)
  • increase text limit for custom flags (as explained before)
  • snippet-IDs Maybe SO snippets could introduce something like a snippet ID/hash. This meta-data ID would only be appended to a snippet markdown if the snippet was inserted via the snippet editor. If a newer (2024) post without any description doesn't contain this metadata the community bot could at least return some alert
  • Extra flag for suspected AI/Bot content This could add more focus on generative content spam ideally in the review chain. So if a post is flagged as "Potential AI/Bot-content" community members revising this (hypothetical new) review chain could pay more attention to the characteristics of automated content generation

Incentives for community based moderation?

This may deserve its own post, but there is a lot of talk about incentives for new users to post a question. Perhaps SO/Stack Exchange should consider "rewarding" members for spotting and removing bad content.

Referring to @gnat's comment downvote penalties should be removed:
AFAIK, when a post I previously downvoted gets deleted I receive a "payback" as described here "https://meta.stackoverflow.com/questions/261651/does-deleting-questions-with-negative-vote-undo-the-negative-rep-lost"
However, we may also reward all

  • users contributing to the deletion of harmful/misleading post may be rewarded with 1+ reputation

Admittedly, this may also introduce new misuse scenarios.

While I'm not an AI hater – it is quite obvious we are increasingly spammed by Bot/AI generated "content"

Examples

Admittedly the described answers are not harmful by means of spreading spam. What do you think?

24
  • 5
    Some of the posts that lead to blogs can be flagged as spam. Those that only generate svg images and don't answer their respective questions may be flagged with not an answer flag. I think the system suspends the user from posting if there are too many flags in their posts. Commented Aug 25 at 3:51
  • 1
    Those are all terrible very low quality answers. Maybe they were made using a bot, but that's not really relevant.
    – PM 2Ring
    Commented Aug 25 at 4:01
  • It's a bit weird that the "You TV" account is over a year old, but has no (undeleted) activity apart from those recent rubbish answers.
    – PM 2Ring
    Commented Aug 25 at 4:04
  • 14
    @AugustoVasques The "Not An Answer" flag should not be used for posts that look like they might be answers but which require the mod to evaluate the technical correctness of the content.
    – PM 2Ring
    Commented Aug 25 at 4:08
  • 2
    @Augusto If the account is fairly new, there will be a "Very low quality" flag option. I'm not sure what flag to use in other cases. I'm hesitant to use a custom flag.
    – PM 2Ring
    Commented Aug 25 at 4:30
  • 32
    Thanks for raising this question! It's something that needs more careful consideration. However, I want to say very clearly that the transgressions you lay out here (code snippets seeming to match the topic, no explanation/description, and no response to comments) are not, in general, reasons for a mod to delete an answer as either VLQ or NAA. Therefore, you should not raise generic VLQ or NAA flags on these answers (unless it's very obvious even to someone who doesn't know the language/tools asked about that the code snippet in the answer is completely irrelevant). Use custom flags.
    – Cody Gray Mod
    Commented Aug 25 at 7:37
  • 3
    Why would a bot be designed to post such answers, what benefit would be gained? Maybe to edit in spam at a later date? Commented Aug 25 at 11:31
  • 9
    @PresidentJamesK.Polk Or maybe to farm reputation from the occasional drive-by upvoter, to get off the "new account" radar for later spam posts
    – Bergi
    Commented Aug 25 at 15:26
  • 3
    @Cody Gray: Thanks for your and the community's quick response - most of the posts have now been deleted. This definitely proves that the community moderation concept can work! I completely agree and have added more detail - But I actually used the custom flags. In short, the current character limit is a pretty ridiculous problem when it comes to flagging a more complex case. Hopefully my clumsy ideas will be helpful for some potential SO tweaks. Commented Aug 26 at 0:43
  • 3
    In addition to Mark's answer, code-only answers on SO should be flagged and held in a review queue, or until some text explanation is provided. Code-only answer are not-helpful, and by holding them in a moderation queue until approved (I can see where heavily commented code could provide needed text-as-code), or until the submitter remedies the no text problem would be perfectly reasonable. Commented Aug 26 at 7:00
  • 1
    "answer downvotes should be free"
    – gnat
    Commented Aug 26 at 10:02
  • 2
    "answers with snippets appear unsuspicious for members reviewing flagged posts in a review chain" I disagree. The existence of a snippet does not modify my level of suspicion about the quality or provenance of a post in anyway, at least for me. I find snippets misused on a significant basis when reviewing posts... probably 25% of the time.
    – TylerH
    Commented Aug 26 at 15:12
  • 3
    @gnat - downvotes should not be free regardless. Having now been on SO for a decade, I've watched and complained about downvote activity. The 1-2 rep cost is completely reasonable and discourages the roving bands of "downvote weenies" that downvote every answer on a page that isn't theirs. There are always pros and cons, but the downvote cost pros far outweigh its cons, even in circumstances such as this. Commented Aug 26 at 21:07
  • 2
    @herrstrietzel First, let me say that I'm aware you used custom flags. My comment wasn't necessarily directed to you, but to the others who were reading and commenting on this, especially as it hit "Hot Meta Posts". Regarding serial downvotes, yes, technically the system would notice and flag as suspicious if you downvoted a large number of answers (like 6) by the same user. Those votes might even get automatically invalidated by the system (which, as you said, tries to be "well-balanced", as well as a relatively dumb system can be). ...
    – Cody Gray Mod
    Commented Aug 28 at 9:14
  • 3
    ... But moderators weren't born yesterday, and we would not punish or otherwise sanction you for downvoting answers where there is objective basis for assessing them as low quality, especially if you flagged them with such a rationale. In my opinion, anything you feel strongly should be deleted should probably also be downvoted. And, I feel you about the character limits, as you can well see. The character limits that mods get when marking a flag helpful/declined are even more restrictive, and they are the perpetual bane of my existence.
    – Cody Gray Mod
    Commented Aug 28 at 9:15

2 Answers 2

15

Code-only answers without an explanation have always been discouraged on Stack Overflow. I've even gotten a few downvotes on some of my own. If the purpose of the bots is to farm reputation, downvotes would seem to be an effective method of getting rid of them.

5
  • Thanks! I agree, but the point is also whether even e.g a simplistic markdown-analysis may filter some of the crappy content. For instance I can't post an answer if the markdown content is too long. I'm also nagged when I try to link e.g an external codepen/jsfiddle example – I immediately see a warning preventing me from posting the answer. While this perimeter can certainly be hacked/circumvented as well - extending the "form" validations may be helpful. Commented Aug 27 at 1:00
  • 6
    The penalty for downvotes is negligible compared to the award for upvotes.
    – Didier L
    Commented Aug 27 at 9:32
  • 1
    I can't see the examples because they're deleted so I can't tell for sure, but if my understanding is correct the code is complete nonsense that nobody would ever upvote anyway, so this seems like a rather pointless attempt at farming reputation if that's the purpose. Commented Aug 27 at 9:55
  • @DidierL that's true, but answers with a negative score don't attract upvotes as much. Commented Aug 27 at 11:59
  • And I've seen folks with thousands of rep manage to tip the scales into answer and question bans. Usually Q-bans, though. Upvotes are worth more rep but it's the scoring of the posts that allows you to keep writing questions and answers. Commented Aug 28 at 16:01
6

There's different considerations here:

  • Code only answers related to the question and in the same programming language are on-topic. They may be poor answers that should be down-voted.

  • Code only answers unrelated to the question and/or in a different programming language than the question are off-topic. Essentially someone just posted random code. This could be flagged as very low quality or delete-voted.

  • If the answer clearly is self-promotional, containing links to something unrelated to the question, flag as spam.

  • If you merely suspect malicious or self-promotional intent for whatever the reason ("spam seeds" etc), use a custom flag and explaining your reasoning. You don't need to write a novel, just type something like:

    User x is repeatedly posting code snippets irrelevant to the question like the one I'm flagging or like <link>. I suspect that these are spam seeds.

    Alternatively you could ask for advise in the SO Close Vote Review Chat. In case you do feel a need to post a verbose explanation of the issue, that would be a good place to do so as well. You could then link to the chat message when raising a flag - chats are logged and saved.

    (Also, lots of mods do lurk in said chat. And while it isn't appropriate to use that chat to chase down individual moderators, one of them whom consider themselves on-duty at the moment may respond and take it from there.)

7
  • 2
    Inspector Gadget is ALWAYS on duty. Commented Aug 26 at 22:00
  • 2
    Could the system not simply just forbid code only answers? I mean, it would probably block this particular spam only for a short time, but also in general it's what we want from a good answer. Commented Aug 26 at 22:41
  • 1
    Thanks! the close vote chat is definitely a way I haven't considered before Commented Aug 27 at 1:12
  • How can an answer be a spam seed? Commented Aug 27 at 10:01
  • 2
    @DonaldDuck You post an answer that looks legit, then later edit it to include spam?
    – Lundin
    Commented Aug 27 at 10:04
  • @DonaldDuck It can't, only questions can be. Answers can only be spam.
    – TylerH
    Commented Aug 27 at 13:37
  • 1
    @NoDataDumpNoContribution You'd have to get SO devs to actually make good design changes to the site for that to happen. But it is surprising it wasn't included in the beginning of the site, or near to it.
    – TylerH
    Commented Aug 27 at 13:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .