Skip to main content
replaced http://meta.stackexchange.com/ with https://meta.stackexchange.com/
Source Link

What will be perceived opaquely from the outside as our 'quality score' algorithm is actually quite a few moving pieces going behind the scenes. The best analogy that I can provide is a large-scale automated conveyor system, with some humans tending to it in various places:

You could totally build that in MinecraftYou could totally build that in Minecraft

The ML stuff is a major and integral part of the system that we're coming up with, but there's also a whole lot of other things being taken into consideration. Everything from what brand new users did on the site prior to asking a question, question history (if applicable), quality we've seen from those sharing the same network, - all kinds of tiny bits of signal that while unhelpful alone, give us a great indication of where the question should land when combined and paired with comprehensive ML.

Every one of these factors program about two dozen sort of 'dip switches', which the system reads in order to know where your question needs to go.

  • Most questions will just go to the front page, or a triage where a couple of people saying 'that looks fine' will send them on their way with minimal delay.

  • Some questions land firmly on 'not bad, but not good', these go to a special queue for folks that like to edit these gems and polish them up.

  • Some questions will head straight for a close queue and probably see no other visibility

Of course, triage also exists to slam the breaks on everything that automated processing could gather - in case it's another incarnation of our favorite programming cartoons. If you haven't realized it yet, this is all about sorting.

I can't name all of the checks that we've come up with, because that's making it a carrot on a stick rather than something in place to help folks genuinely get better at asking questions. Were some of the recent votes on your questions deemed suspicious by the system or invalidated? Alone, that's okay - but if other triggers fire, there's a good chance you're asking crap questions and getting your co-workers to up-vote them. That all ends.

The best advice I can give is do not waste a shred of signal that could potentially be useful, when confirmed by other sources of signal. Suspicious votes alone won't alter your path, but if it's been just 30 minutes since your last question in addition to that, someone might need to sign-off on it before we put it on the front page (just as a wild example).

So, tl;dr - Don't restrict your analysis to simply what the user submits. Use everything you know about the user as well, to the extent that it's deterministic. Hence, if you're building something new, be very watchful of places where you can conveniently capture this kind of signal, and capture it. Every pattern is interesting until you can technically discount it.

I wish I could share every single thing we're checking, because I'm quite proud of what we were able to come up with. Getting into this I was doubtful, because the mine for good signals to indicate question quality had been picked pretty well clean over the years.

Also, if your system doesn't try to teach users, it won't know if they're improving, or to what extentif your system doesn't try to teach users, it won't know if they're improving, or to what extent.

What will be perceived opaquely from the outside as our 'quality score' algorithm is actually quite a few moving pieces going behind the scenes. The best analogy that I can provide is a large-scale automated conveyor system, with some humans tending to it in various places:

You could totally build that in Minecraft

The ML stuff is a major and integral part of the system that we're coming up with, but there's also a whole lot of other things being taken into consideration. Everything from what brand new users did on the site prior to asking a question, question history (if applicable), quality we've seen from those sharing the same network, - all kinds of tiny bits of signal that while unhelpful alone, give us a great indication of where the question should land when combined and paired with comprehensive ML.

Every one of these factors program about two dozen sort of 'dip switches', which the system reads in order to know where your question needs to go.

  • Most questions will just go to the front page, or a triage where a couple of people saying 'that looks fine' will send them on their way with minimal delay.

  • Some questions land firmly on 'not bad, but not good', these go to a special queue for folks that like to edit these gems and polish them up.

  • Some questions will head straight for a close queue and probably see no other visibility

Of course, triage also exists to slam the breaks on everything that automated processing could gather - in case it's another incarnation of our favorite programming cartoons. If you haven't realized it yet, this is all about sorting.

I can't name all of the checks that we've come up with, because that's making it a carrot on a stick rather than something in place to help folks genuinely get better at asking questions. Were some of the recent votes on your questions deemed suspicious by the system or invalidated? Alone, that's okay - but if other triggers fire, there's a good chance you're asking crap questions and getting your co-workers to up-vote them. That all ends.

The best advice I can give is do not waste a shred of signal that could potentially be useful, when confirmed by other sources of signal. Suspicious votes alone won't alter your path, but if it's been just 30 minutes since your last question in addition to that, someone might need to sign-off on it before we put it on the front page (just as a wild example).

So, tl;dr - Don't restrict your analysis to simply what the user submits. Use everything you know about the user as well, to the extent that it's deterministic. Hence, if you're building something new, be very watchful of places where you can conveniently capture this kind of signal, and capture it. Every pattern is interesting until you can technically discount it.

I wish I could share every single thing we're checking, because I'm quite proud of what we were able to come up with. Getting into this I was doubtful, because the mine for good signals to indicate question quality had been picked pretty well clean over the years.

Also, if your system doesn't try to teach users, it won't know if they're improving, or to what extent.

What will be perceived opaquely from the outside as our 'quality score' algorithm is actually quite a few moving pieces going behind the scenes. The best analogy that I can provide is a large-scale automated conveyor system, with some humans tending to it in various places:

You could totally build that in Minecraft

The ML stuff is a major and integral part of the system that we're coming up with, but there's also a whole lot of other things being taken into consideration. Everything from what brand new users did on the site prior to asking a question, question history (if applicable), quality we've seen from those sharing the same network, - all kinds of tiny bits of signal that while unhelpful alone, give us a great indication of where the question should land when combined and paired with comprehensive ML.

Every one of these factors program about two dozen sort of 'dip switches', which the system reads in order to know where your question needs to go.

  • Most questions will just go to the front page, or a triage where a couple of people saying 'that looks fine' will send them on their way with minimal delay.

  • Some questions land firmly on 'not bad, but not good', these go to a special queue for folks that like to edit these gems and polish them up.

  • Some questions will head straight for a close queue and probably see no other visibility

Of course, triage also exists to slam the breaks on everything that automated processing could gather - in case it's another incarnation of our favorite programming cartoons. If you haven't realized it yet, this is all about sorting.

I can't name all of the checks that we've come up with, because that's making it a carrot on a stick rather than something in place to help folks genuinely get better at asking questions. Were some of the recent votes on your questions deemed suspicious by the system or invalidated? Alone, that's okay - but if other triggers fire, there's a good chance you're asking crap questions and getting your co-workers to up-vote them. That all ends.

The best advice I can give is do not waste a shred of signal that could potentially be useful, when confirmed by other sources of signal. Suspicious votes alone won't alter your path, but if it's been just 30 minutes since your last question in addition to that, someone might need to sign-off on it before we put it on the front page (just as a wild example).

So, tl;dr - Don't restrict your analysis to simply what the user submits. Use everything you know about the user as well, to the extent that it's deterministic. Hence, if you're building something new, be very watchful of places where you can conveniently capture this kind of signal, and capture it. Every pattern is interesting until you can technically discount it.

I wish I could share every single thing we're checking, because I'm quite proud of what we were able to come up with. Getting into this I was doubtful, because the mine for good signals to indicate question quality had been picked pretty well clean over the years.

Also, if your system doesn't try to teach users, it won't know if they're improving, or to what extent.

Source Link
user50049
user50049

What will be perceived opaquely from the outside as our 'quality score' algorithm is actually quite a few moving pieces going behind the scenes. The best analogy that I can provide is a large-scale automated conveyor system, with some humans tending to it in various places:

You could totally build that in Minecraft

The ML stuff is a major and integral part of the system that we're coming up with, but there's also a whole lot of other things being taken into consideration. Everything from what brand new users did on the site prior to asking a question, question history (if applicable), quality we've seen from those sharing the same network, - all kinds of tiny bits of signal that while unhelpful alone, give us a great indication of where the question should land when combined and paired with comprehensive ML.

Every one of these factors program about two dozen sort of 'dip switches', which the system reads in order to know where your question needs to go.

  • Most questions will just go to the front page, or a triage where a couple of people saying 'that looks fine' will send them on their way with minimal delay.

  • Some questions land firmly on 'not bad, but not good', these go to a special queue for folks that like to edit these gems and polish them up.

  • Some questions will head straight for a close queue and probably see no other visibility

Of course, triage also exists to slam the breaks on everything that automated processing could gather - in case it's another incarnation of our favorite programming cartoons. If you haven't realized it yet, this is all about sorting.

I can't name all of the checks that we've come up with, because that's making it a carrot on a stick rather than something in place to help folks genuinely get better at asking questions. Were some of the recent votes on your questions deemed suspicious by the system or invalidated? Alone, that's okay - but if other triggers fire, there's a good chance you're asking crap questions and getting your co-workers to up-vote them. That all ends.

The best advice I can give is do not waste a shred of signal that could potentially be useful, when confirmed by other sources of signal. Suspicious votes alone won't alter your path, but if it's been just 30 minutes since your last question in addition to that, someone might need to sign-off on it before we put it on the front page (just as a wild example).

So, tl;dr - Don't restrict your analysis to simply what the user submits. Use everything you know about the user as well, to the extent that it's deterministic. Hence, if you're building something new, be very watchful of places where you can conveniently capture this kind of signal, and capture it. Every pattern is interesting until you can technically discount it.

I wish I could share every single thing we're checking, because I'm quite proud of what we were able to come up with. Getting into this I was doubtful, because the mine for good signals to indicate question quality had been picked pretty well clean over the years.

Also, if your system doesn't try to teach users, it won't know if they're improving, or to what extent.