33

Recently there was a Stack Exchange podcast which discussed the new "quality-score" algorithm that has been developed by the SE team (discussed from roughly 30min mark onwards).

The "quality-score" algorithm attempts to quantitatively score the quality of questions that are asked on SE sites by using machine learning (ML). The idea behind ML and the algorithm is to take questions from the past and train a model using certain characteristics so that predictions can be made for questions in the future on whether they are low quality or not (and then presumably do something if they are low quality, but we're not really interested in that part).

We (members of the Python SO Chat community) have been planning/working towards a similar open-source idea called Project Nidaba where we would use machine learning to predict low-quality/duplicate questions that we could then close-vote/delete-vote/edit as needed. We've recently re-built our sopython website (using a Python framework, obviously) and we're now looking towards working on Nidaba itself (making the announcement of the SE work in this podcast very timely).

I was wondering whether any of the developers who worked on the quality-score algorithm would be willing to share any of their insights into what they've done (more than is done so in the podcast itself). In particular any problems or particularly interesting parameters they used would be very useful to us. I understand that Vowpal Wabbit was used to construct your model whilst we'll probably use scikit-learn but any help/advice/information you could provide would be very useful.

Essentially we're looking for any info/advice to help us avoid re-inventing the wheel. All the work we produce for sopython is licensed using the BSD 3-clause licence and we'd love it if someday it could also benefit other communities within SO/SE.

7
  • 1
    I have posted this on meta.SO as the podcast says "This is primarily a Stack Overflow thing, so Meta Stack Overflow is the best place to discuss it. Have at it!" but I'm open to meta.SE being a better place for it.
    – Ffisegydd
    Commented Aug 5, 2014 at 14:57
  • 1
    Maybe you could also leverage the work of the winners from the Stack Exchange Machine Learning Contest of a couple of years ago: blog.stackoverflow.com/2012/08/… . I wonder if some if this latest work comes out of that.
    – Brad Larson Mod
    Commented Aug 5, 2014 at 15:15
  • 4
    One big problem I had with that contest, and that still troubles me about machine learning for determining bad posts, is that the absolute worst posts tend to be deleted, so they appear in none of the data dumps or other data sets. Therefore, any training done on public data will already be skewed, and will most likely miss the most important stuff we want to identify first.
    – Brad Larson Mod
    Commented Aug 5, 2014 at 15:18
  • 1
    @BradLarson yes I agree it's something we've thought about ourselves. One of our ideas is to keep a track of questions with respect to time (as we're only doing this for Python data on sopython we only have ~330,000 questions) and then trying to work out questions that have been deleted. This won't be perfect (some questions may be self-deleted for example) but it's the best we can do with the public data dumps/api.
    – Ffisegydd
    Commented Aug 5, 2014 at 15:21
  • 2
    @BradLarson while we have the latest data dumps as a base, we also plan to use the live web sockets to keep track of questions in their early stages - looking at voting activity, comments and edits (even within grace periods). We then periodically (absolute rubbish mostly gets closed/deleted very quickly - so some time window...) query the main API to identify question IDs that no longer exist and to check its close state. It's not perhaps the most perfect of solutions, will need some thought and some experimentation, but probably the best we can do to cover the case you mention Commented Aug 5, 2014 at 15:27
  • 7
    As the PM of the project - answer forthcoming, but probably in about 12 hours (1) because I'm headed out for the day and (2) I have to give a bit of a think to how much secret sauce I can spill. (seekrit only so this doesn't become a series of carrots on sticks)
    – user50049
    Commented Aug 5, 2014 at 15:40
  • 1
    The model I would be interested in is the close queue; we still have a large backlog. If a model could predict the questions with 1 close votes or flag in a given tag that should be looked at by an expert in that tag, then it could speed up the duplicate close process. Also a model that predicts if a questions is likely to be a duplicate of the 30 most common questions for a tag would be a great way to direct the effect of gold tag badge holders. Commented Aug 5, 2014 at 19:49

2 Answers 2

29

I think the most obvious insight (to me) might be that our approach to this and yours might be different. Our goal is to try to nip low-quality stuff in the bud to keep it from repelling our best contributors, which means we want to feed low-quality questions to the review queue as they come in, rather than letting them occupy valuable real estate in place of better questions. Obviously we have access to the system in a way that you don't, and that can affect our respective models. I'm curious to see what you come up with, though. We'd be happy to take a close look at anything that would help us accomplish our goals.

Where we have spent most of our time thus far is modeling, although now we're actually writing code. At the beginning, we tried doing semi-manual training using the Accord Framework (which is fairly a complete statistical/ML framework for .NET). We then discovered that for purely modeling purposes, vowpal wabbit is much more productive. You still have to do your own feature extraction, but vowpal makes it very easy to train once you have your data. Most of our modeling has been with one-against-all classification with the default squared-error loss function, although we've tried out models with binary logistic regression as well. See Vowpal Wabbit inverted_hash option produces empty output, but why? for some OAA vs binary results that I thought a bit unexpected.

We are only looking at information contained in a new post to do our scoring. That means we are not taking any kind of user information into account - for one, we don't want to bias for or against a particular type of user. To paraphrase a famous line, "We want the scoring to based not on the color of the user's avatar, but on the content of their question." Anyway, this is more restrictive than it sounds at first. Brand-spanking-new posts have only a title, the body, and tags. There is no post history, there are no comments, and we don't consider who the user is.

I suggest that you try to make small models (e.g., < 500 features). We found that after a few hundred features, accuracy didn't go up much with more features (I have a model with < 100 features that performs only 2% worse than other models with thousands on certain test sets). Small models are much easier to work with, so you won't loose as much time to dataslinging.

The only other things I can think of at the moment are just general ML advice: clean your data, make sure your test/validation sets make sense and aren't overlapping, etc.

Good luck! I'll add to this if I think of anything else.

1
  • 3
    Thanks for taking the time to answer this question Ben. When we finish the project I will definitely put some sort of report together and share it, if it ends up helping your efforts as well then all the better.
    – Ffisegydd
    Commented Aug 6, 2014 at 8:46
13

What will be perceived opaquely from the outside as our 'quality score' algorithm is actually quite a few moving pieces going behind the scenes. The best analogy that I can provide is a large-scale automated conveyor system, with some humans tending to it in various places:

You could totally build that in Minecraft

The ML stuff is a major and integral part of the system that we're coming up with, but there's also a whole lot of other things being taken into consideration. Everything from what brand new users did on the site prior to asking a question, question history (if applicable), quality we've seen from those sharing the same network, - all kinds of tiny bits of signal that while unhelpful alone, give us a great indication of where the question should land when combined and paired with comprehensive ML.

Every one of these factors program about two dozen sort of 'dip switches', which the system reads in order to know where your question needs to go.

  • Most questions will just go to the front page, or a triage where a couple of people saying 'that looks fine' will send them on their way with minimal delay.

  • Some questions land firmly on 'not bad, but not good', these go to a special queue for folks that like to edit these gems and polish them up.

  • Some questions will head straight for a close queue and probably see no other visibility

Of course, triage also exists to slam the breaks on everything that automated processing could gather - in case it's another incarnation of our favorite programming cartoons. If you haven't realized it yet, this is all about sorting.

I can't name all of the checks that we've come up with, because that's making it a carrot on a stick rather than something in place to help folks genuinely get better at asking questions. Were some of the recent votes on your questions deemed suspicious by the system or invalidated? Alone, that's okay - but if other triggers fire, there's a good chance you're asking crap questions and getting your co-workers to up-vote them. That all ends.

The best advice I can give is do not waste a shred of signal that could potentially be useful, when confirmed by other sources of signal. Suspicious votes alone won't alter your path, but if it's been just 30 minutes since your last question in addition to that, someone might need to sign-off on it before we put it on the front page (just as a wild example).

So, tl;dr - Don't restrict your analysis to simply what the user submits. Use everything you know about the user as well, to the extent that it's deterministic. Hence, if you're building something new, be very watchful of places where you can conveniently capture this kind of signal, and capture it. Every pattern is interesting until you can technically discount it.

I wish I could share every single thing we're checking, because I'm quite proud of what we were able to come up with. Getting into this I was doubtful, because the mine for good signals to indicate question quality had been picked pretty well clean over the years.

Also, if your system doesn't try to teach users, it won't know if they're improving, or to what extent.

1
  • 3
    Thanks for the answer Tim and especially thanks for the link to the meta.SE posts you're putting together. It's a shame that you can't share more details of what work you're doing as I'm sure it's very interesting but as I said to Ben, we'll produce some sort of report ourselves from our findings and hopefully it may even help some of your efforts.
    – Ffisegydd
    Commented Aug 6, 2014 at 8:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .