66

I just run into some suggested edits by TaggerBot. It's a bot running from a Python script via the Stack Exchange API made to suggest tag edits to questions.

E.g.:

Tags are quite generic, but not necessarily wrong.

On one hand it could be a nice idea to automatize post tagging. On the other hand this will keep busy reviewers for generic (I'm not saying irrelevant, nor wrong) tags.

I'd like to know:

  • Does Stack Overflow encourage such use of its API? Is against the rule or something? Should I flag for moderators?
  • Is this kind of bot welcome?

I'm just curious about SO policy and community thoughts. Personally I haven't still decided whether TaggerBot is a good or a bad idea, or if will be useful to SO.

15
  • 12
    My bot. Will add an answer in a moment. Current discussion: see chat.meta.stackexchange.com/transcript/message/4151532#4151532
    – ArtOfCode
    Commented Nov 4, 2015 at 18:55
  • 1
    @ArtOfCode thank you. Hope you don't mind that I brought the discussion on meta (I wasn't aware of the chat). This seems a brand new bot. It will improve suggested tags quality over time?
    – Miki
    Commented Nov 4, 2015 at 18:59
  • 1
    I cannot locate the bemused discussion from a couple of years ago (it was not entirely amusing), but wasn't there a bot once running rampage around the site, prompting concerned citizens to give it nicks such as HAL and SkyNet?
    – Jongware
    Commented Nov 4, 2015 at 18:59
  • 6
    Might be a pretty cool thing if it ran some auto-corrections on recent, problematic posts (trademark capitalization, spelling, noise, ...). As it is right now, nope nope nope. it is not appropriate to do tag-only edits on a large scale out of a community effort.
    – Kyll
    Commented Nov 4, 2015 at 19:03
  • 29
    Shouldn't this bot have been discussed on Meta before being unleashed in the wild? We discourage people from doing burninations via suggested edits, but we're going to let this one add tags? Commented Nov 4, 2015 at 19:04
  • It looks like TaggerBot is on its way, if it hasn't already hit one, to a suggestion ban. I just counted 9/17 suggestions as rejected, with a few still pending.
    – Kendra
    Commented Nov 4, 2015 at 19:05
  • 1
    @MikeMcCaughan Difference: Everything this does requires a review from three real people. The damage with unauthorized mass-burninations is when a 2ker goes nuts and adds a bunch of stuff without review.
    – Undo Mod
    Commented Nov 4, 2015 at 19:05
  • 21
    According to his profile, TaggerBot is Informed so at least aware of the general rules.
    – Jongware
    Commented Nov 4, 2015 at 19:06
  • 2
    @Undo, I was talking about <2kers going crazy on a burnination, as indicated by "via suggested edits". Especially as these suggested edits are not fixed everything that's wrong with these posts. Commented Nov 4, 2015 at 19:08
  • 2
    @MikeMcCaughan Burninations are discouraged in suggesteds because tags require rep to restore. This bot is simply working with existing tags.
    – ArtOfCode
    Commented Nov 4, 2015 at 19:08
  • 4
    I just find it fascinating that this flood is okay, but others are not. Commented Nov 4, 2015 at 19:27
  • 21
    @Mike that's my major problem here. Undo says "treat it like a human" but if a human was doing this I'd tell them to knock it off. Commented Nov 4, 2015 at 19:36
  • 9
    Repeatedly submitting tag edit suggestions constitutes abuse for both humans and bots. It's not helpful and all of these edits should be rejected as no improvement whatsoever. Suggested edits must substantially improve multiple aspects of a post, as explained by the editing help.
    – bjb568
    Commented Nov 4, 2015 at 19:45
  • 1
    I regret not having the foresight to get a screen cap, but TaggerBot was the placeholder user for this audit. Besides the usual suggested edit audits making no sense, I also noticed that TaggerBot was editing stuff other than tags. Thanks TaggerBot for even extra help with my audit.
    – ryanyuyu
    Commented Nov 5, 2015 at 23:11
  • that is as much meta I can take for one day Commented Nov 7, 2015 at 9:06

4 Answers 4

63

The API is there to be utilized. There are write capabilities via the API. The API is being used in many applications. It even has end points to edit questions, see suggested edits and issue flags.

The question on whether or not this is welcome is more complicated. Beginning with the ever relevant XKCD, if a bot can improve the site, then it should be welcome.

And what about all the people who won't be able to join the community because they're terrible at making helpful and constructive co-- ... oh. -XKCD

I know of one instance where this type of automated activity has been welcomed by the community, because I wrote the bot: Can a machine be taught to flag comments automatically?

Regarding this specific tagging bot, I think it needs some improvements. Of its recent suggestions, there are 8 rejected edits, 6 approved, and a handful of pending. This isn't horrible, but it can be improved.

  • The bot should focus on less general tags. is a very broad tag. Does this edit, really help at all? Same with this one, this one and this one
  • The bot needs to pull relevant tags. To my untrained eye, this edit doesn't involve at all.
  • Focus on "good" posts. The first several that I clicked through on the bot's profile have edited tags on 0 or negatively scored questions. Several have been closed. Focus on questions that have good scores or positively scoring answers. I realize the idea is to improve the bad stuff, but we get so many questions here, let's make our "ok" questions better.
  • Fix more than just tags. Fixing only tags, when there are other things that can be fixed in a post (spelling, grammar, etc.) is missing a lot of ways to improve even the bad questions.

I do think this type of automation can be useful, but it needs to be done well and it needs to focus on the entire post, not just potentially related tags.

9
  • To clarify: "related" tags are also pulled from the API, which I have no control over. They're done by SE's algorithms. I deliberately assigned less score to them because I didn't want to rely too heavily on them.
    – ArtOfCode
    Commented Nov 4, 2015 at 19:16
  • "Of it's recent suggestions, there are 8 rejected edits, 6 approved, and a handful of pending." Actually, there are (and were when this was posted) 9 rejects, 5 accepts, and 3 pending. Gets kinda hard to count them, but I had already a few times before I saw your answer.
    – Kendra
    Commented Nov 4, 2015 at 19:23
  • 1
    @Kendra if you open any of the suggested edits and click "more" you can see statistics for the suggester and reviewers Commented Nov 4, 2015 at 19:57
  • 11
    @mikeTheLiar I stopped looking at the statistics ages ago because they were often wrong. (I think they were recently fixed, but old habits die hard. (And there were only 17 edits to count anyway.))
    – Kendra
    Commented Nov 4, 2015 at 20:04
  • oh, maybe that's why my profile says I reject something like three times the edits I approve...! @Kendra Commented Nov 7, 2015 at 9:14
  • I lurk design-patterns and so often that tag wrongly appears with regex. It's a no-brainer for a bot to fix. Have you written TaggerBot to work with rule-based logic? I.e., if (design-patterns and regex and user.rep<200) then removeTag(design-patterns). Here's a search to give you an idea: stackoverflow.com/questions/tagged/design-patterns+regex Commented Jul 21, 2017 at 13:34
  • Another idea would be to classify the simplest approved edits, to perhaps extract rules like the one I just gave. Ironically, I corrected "it's" to "its" in this answer, and that is a very common error (made it many times myself). I suspect there are a lot of cases that could be mined from edit histories. Commented Jul 21, 2017 at 13:42
  • @Fuhrmanator TaggerBot is mine, not Andy's. It's also been shelved since I didn't have enough time to put into it to make it worthwhile.
    – ArtOfCode
    Commented Jul 21, 2017 at 21:40
  • @ArtOfCode Sorry for the noise. I didn't realize from its profile that it was shelved. Commented Jul 21, 2017 at 22:23
42

TaggerBot is not ready to be let loose full speed, but you should keep iterating on the code until it can achieve > 60% approval rate. You've picked one of the hardest problems in computer science today, success at this would be a contribution to the field of computer science and machine learning.

Concerns:

Consider this edit: https://stackoverflow.com/review/suggested-edits/10095827

The 1-reputation user asked a terrible question and pasted a mountain of wrongly formatted code that doesn't run, and says: "I don't know what to do" or some variation on that. The question should be deleted, closed, or have it's mountain of code trimmed down to 8 lines with a SSCCE.

Instead, Taggerbot examines the question, and sees an invocation of "listView", sure, listview is used in their code, but that's not the main theme of the question, it's a not-important side-show story.

So Taggerbot did damage by claiming that "the most important concept of this question is listView". It's not! Maybe the user wanted a shoulder to cry on or wanted to have the fact that he's missing a semicolon pointed out. The added tags do damage, in my mind, by adding noise in the form of wrongly asserted precision.

TaggerBot is a noble idea, and you should keep building it out and improving it, but keep it on a tight leash, and at current approval levels < 40%, you're hurting stackoverflow by consuming the good will of editors who greatly improve the site.

Conclusion:

Keep iterating on it, have it do at most 2 or 5 edit suggestions per day, and when you can demonstrate 80% or more approval rate, with positive reviews, then after another performance review, then the highest rep users give it the green light and increase throttle to full.

1
  • 17
    Reviewers love to approve bad edits. Spammers have in the past had a >60% approval rating posting pure spam. You need the bar to be a lot higher than that for such a bot to actually be helpful. The analysis of the one example edit is well explained though, the problem of course is that if you propose that edit a bunch times it'll get approved a lot of the time, even though it should be rejected.
    – Servy
    Commented Nov 5, 2015 at 18:31
23

My take on it: treat this thing like it's a human. Is the edit good? Approve it. Bad? Reject it.

There's a good chance that this becomes useful - but that can't happen if we shut it down. Let people experiment, as long as they aren't breaking things. We have a guy who figured out how to automatically mass-flag comments very accurately, and if we can do the same thing with edits, starting with tags... I think we win the Internet.

This thing isn't hurting anything. If it makes really bad edits, they'll be rejected* and enough of them will impose an automatic edit ban. If it makes good edits... then it's working, and we should be proud of it.

We're a community of programmers. When one of us wants to try something innovative... let's not try to kill that. Especially when there's so little risk and such a potentially high reward.


*right now, I'm considering using this as a way to find bad reviewers. For every bad reviewer we find, that's far more benefit to the site than the harm of a few dozen bad suggestions.

8
  • 23
    Great, now we'll have robo-reviewers reviewing robo-editors ;). Commented Nov 4, 2015 at 19:28
  • 1
    @MikeMcCaughan Next thing you know, we'll have robo-programmers writing the robo-reviewers. Next stop: singularity!
    – user
    Commented Nov 4, 2015 at 19:51
  • 2
    @MichaelKjörling Nah, that'll happen when robo-answerers start answering robo-questioners' questions ;) Commented Nov 4, 2015 at 19:54
  • 2
    @MikeMcCaughan Ah, you're right. Robo-answerers asking robo-questioners' questions that the robo-programmers run into writing the robots?
    – user
    Commented Nov 4, 2015 at 19:55
  • I hope Stack Exchange Inc has the budget to buy some more disk space and bandwidth... we're all going to need it once that happens.
    – user
    Commented Nov 4, 2015 at 19:56
  • But by our standards, isn't adding a marginally better tag and ignoring all other aspects of an edit considered flooding the review queue, and hence breaking things?
    – Teepeemm
    Commented Nov 6, 2015 at 15:35
  • 1
    @Teepeemm My thoughts on that are here: "SO has never had issues clearing the suggested edit queue. Until that happens, the 'don't flood' argument is moot in my opinion".
    – Undo Mod
    Commented Nov 6, 2015 at 16:01
  • 2
    @Teepeemm in the old days, flooding the review queue meant people could not suggest edits because the queue had a cap. But since then, the system has changed enough that it will never happen. In current times, "flooding the review queue" has come to mean "I am tired of seeing the same user editing posts while I am reviewing" or "I don't like someone earning too much rep from repeated targeted editing" (seeking a specific problem in all posts and fixing only that problem). The size of the queue itself is a minor issue on Stack Overflow Commented Nov 6, 2015 at 23:00
12

I created the bot, so I'll put in some details and my viewpoint.

TaggerBot is a pretty simple Python script (see the code). It makes use of the API to grab a few things:

  • popular tags
  • related tags
  • questions

It starts of by compiling a list of the top 20 pages of popular tags (though I think I'll reduce this to 10 or even 5) [As of rev 20B, only the top 8 pages are used]. It then fetches a set of questions - either the top 20 pages (default API sort, which is activity, IIRC) and runs them through this algorithm:

for each tag:
    is the tag in the question body?
        if so, add 2 to the tag's score
    find related tags
    for each related tag:
        is the related tag in the question body?
            if so, add 3 to the tag's score
            if not, add 1 to the tag's score
order the list of tags by score
pick the top (as many as can be added to the question) where score >= 6 
suggest these as an edit

Here is the exact algorithm. It's designed to require a number of factors both in the current tags and the question body before it suggests a tag.


I created it, quite clearly, to add related tags to questions. I did this because tagging is important: it's the main mode of navigation in the new nav UI, and is designed to allow some pretty complex queries on tags. The more questions are tagged with tags that are relevant to the question, the easier it will be to find them.

I will happily admit that some of its edits aren't quite as accurate as I hoped, and I'm working on improving the algorithm. Suggestions there would be welcome. However, I would like to see the bot being treated as any other edit - I'm deliberately doing this on a low-rep account not my own (which could apply edits instantly) so that they can be reviewed. Good? Approve. Bad? Reject.

Undo's answer sums up pretty nicely my own attitude to using bots to try to improve the site.


Please note: the above answer was written very soon after the bot was started for the first time. The algorithm detailed in the post is revision 18A. The bot is being updated, and its algorithm version is noted in each edit message.

  • The current revision is 23B
  • The last tested revision is 21B, which showed an accuracy of 60% approved edits on an admittedly small sample.
16
  • 42
    A requirement of a robot is that it doesn't waste resources. In this case, the edit review queue is a finite resource of programmer's time and good will. When you submit hundreds of tag edits adding things like "camera" when tags such as "android-camera" already exist in the thread, you are burdening a person with good will who wants to edit the site, and filling his monitor full of irrelevance, scaring away that potential good-editor from the site. Unless you can get your approval rate for edits consistently over 60%. Right now it looks like it's under 40% approval rate. Commented Nov 4, 2015 at 19:21
  • 4
    @EricLeschinski I'd like to know where this "requirements of robots" is published... However, I can't and won't dispute that there is improvement required; this is early stages and a lot of an experiment. But if we can treat it as an experiment, there's a high chance it can get better and actually be useful. If we kill it, there's no way.
    – ArtOfCode
    Commented Nov 4, 2015 at 19:59
  • 19
    If it only add tags, it's kinda pointless. I hate edits that miss obvious spelling just for a damn tag. Commented Nov 4, 2015 at 20:07
  • 4
    @JonathanDrapeau I contest that it's pointless; tags are there to help an unanswered question be found by the people who can answer it. That's important, as I see it. OK, it's not correcting everything about the post, but it's helping to achieve that goal.
    – ArtOfCode
    Commented Nov 4, 2015 at 20:11
  • 4
  • @JonathanDrapeau I also contest: a good retag is not trivial. With some improvements, a bot can achieve that.
    – ArtOfCode
    Commented Nov 4, 2015 at 20:15
  • 1
    You're contesting the own writings of SO's help on edits... well... good luck. Commented Nov 4, 2015 at 20:16
  • 3
    @JonathanDrapeau Not at all. Trivial edits should be discouraged. A good retag is not trivial. And yes, you should correct everything, but that's a lofty goal even for humans. Something that goes some way to achieving that goal is worthwhile.
    – ArtOfCode
    Commented Nov 4, 2015 at 20:19
  • 14
    Keep it down to a few edit suggestions per day. If you get tired of the project and abandon it, don't leave it running, instead, turn it off, something like this needs constant monitoring and constant improving. At the current level of performance it's just turd polishing. Which is ok I guess, shiny turds are better than dull ones. Try to get it up to a reasonable level of performance, if you can get it so it's reliably adds intuitive tags which represent the spirit of the question, that would be amazing, and such a project should be forked and improved further. Commented Nov 4, 2015 at 20:19
  • @Eric good points well made, will do.
    – ArtOfCode
    Commented Nov 4, 2015 at 20:24
  • 1
    @ArtOfCode Have a look at the Stack-Exchange-Editor-Toolkit originally initiated AstroCB and now improved by Tiny Giant. It improves all kinds of stuff as a user script (JavaScript). Maybe you can accommodate (rewrite) it into your bot so that it produces more useful edits.
    – Artjom B.
    Commented Nov 4, 2015 at 20:40
  • @ArtjomB. Thanks for the link, I'll take a look.
    – ArtOfCode
    Commented Nov 4, 2015 at 20:41
  • 3
    @ArtOfCode - There's also the data dump which has a large chunk of real data that you could use to try the algorithm out.
    – theB
    Commented Nov 4, 2015 at 21:15
  • 1
    @ShawnMehan At least one of those tags was more specific than the existing tags, which is a good thing. However, as I say, there are plenty of improvements to be made, and your notes there will help.
    – ArtOfCode
    Commented Nov 5, 2015 at 17:33
  • 2
    Well, there are a couple of things about that. One is if we are expecting full utility to (only) come from accurate, precise tagging of posts by humans, we are already screwed. So another way to fix it would be to apply semantically related or fuzzy search capabilities at the UI end, to accommodate that. Secondly, I don't know that the mvc-4 tag is more useful. I don't know from the post that it's not actually just a more general mvc issue, and applies over several versions. I think the poster just put in 4 as they dumped their characters into the body. Commented Nov 5, 2015 at 17:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .