54

I'm curious if the teams behind jobs and Stack Overflow / Stack Exchange in general unit test. Is there a quality assurance team who run regular tests before things go into beta? I ask because I am a bit surprised how many bugs are found even out of beta.

If unit testing / QA is not done why isn't it done?

14
  • 17
    That question had to come :) After like 20 bug report questions.
    – Rizier123
    Commented Dec 16, 2015 at 16:06
  • It's a serious question. I know a lot of teams are trying to get away from hiring additional quality support staff but developers don't have the time to do everything. As good of programmers the team at so/se are they really need at least a couple of dba's and a team of qa people. It is just needed period.
    – JonH
    Commented Dec 16, 2015 at 16:07
  • 5
    You are giving error reports like a conveyor belt. [I didn't expected from SO that there are so many bugs]. And now this question was just the logical follow-up questions after like 20 bug reports. Now I expect an answer from you with a TDD / unit-test tutorial :] (Or you can rename your question to: You suck at unit testing with JonH)
    – Rizier123
    Commented Dec 16, 2015 at 16:13
  • 3
    @Rizier123 - I think the programmers at so/se are awesome, hell look at Mark Gravell's profile. I am stressing it to their management team (not sure if they even read stuff like this)..those guys need some help. For the amount of traffic they get they need some dbas and some qa folks if they don't have them - and clearly they dont have them.
    – JonH
    Commented Dec 16, 2015 at 16:19
  • 15
    This is a good question; I'm gonna defer to the developers involved, but there is internal testing (and even unit testing) and has been for years... Nothing beats real folks actually trying to use a system though: we've known (and benefited!) from this on Stack Overflow proper for years, but Careers has been kinda deprived due to the divergent culture and smaller user-base.
    – Shog9
    Commented Dec 16, 2015 at 16:28
  • 1
    @Shog9 I assume you would say the same about docs, that's got quite a few bugs that (under normal SO structure) would not have made it to a real user test phase?
    – DavidG
    Commented Dec 16, 2015 at 16:29
  • 2
    @Shog9 - As much as I respect that I feel its kind of not fair to the current programmers as well as the community. Nothing will ever beat real folks using it but I just feel that doesn't attack the problem. Why don't you guys think about a qa team?
    – JonH
    Commented Dec 16, 2015 at 16:33
  • 15
    Looks like someone wants a QA job at SO :D
    – ken2k
    Commented Dec 16, 2015 at 17:01
  • I can't wait for the blog post that's going to come out of this question.
    – kojiro
    Commented Dec 16, 2015 at 17:17
  • @ken2k - I'm not smart enough for that!
    – JonH
    Commented Dec 16, 2015 at 17:22
  • 4
    @JonH: You'd be surprised, actually... ;)
    – Makoto
    Commented Dec 16, 2015 at 17:26
  • You do want a job! :) meta.stackexchange.com/questions/270890/…
    – user2535467
    Commented Dec 16, 2015 at 18:11
  • @CaptJak - That was a silly joke, although a remote job would be awesome.
    – JonH
    Commented Dec 16, 2015 at 18:12
  • 8
    What's "unit testing"?
    – Nick Craver Mod
    Commented Dec 16, 2015 at 20:16

2 Answers 2

23

We do not have any formalized testing only roles (a "Quality Assurance" team). We do a lot of human based testing, but it's mostly done by members of other teams.

We do use unit tests and integration tests, but the coverage / importance of them differs greatly across teams. Normally, anything that touches money is unit tested by default. A lot of other things also have unit tests, for example on Careers we have unit tests for loading critical pages, completing a job checkout, and fulfilling sold but not activated inventory items.

When it comes to pushing new code on Careers, this is what our flow is like:

TeamCity build setup

Every time new code is pushed (we normally don't work on branches, we all push to the master branch and wrap most of our code in feature flags) the Dev tier there is automatically compiled and a lot of unit tests run. If the unit build succeeds, the new changes are uploaded onto a website on our dev tier automatically. The "Dev - Integration Tests" task starts running at the same time, and if any of the integration tests fail we physically can't push the new changes onto the production tier until that's resolved.

For a lot of things we work on, I think the integration tests are a lot more important than having wide coverage on unit tests (there are a lot of people at the company who disagree with me on this) -- In most cases, all we're doing is consuming some sort of CRUD action, I would rather test to make sure all the actions combined have the outcome we want, than test every individual one of those just to make sure that the C in CRUD really stands for "Create".

To your point about the need for a real QA team, I don't think even if we had anyone doing that as their full time job it'd make a big difference. Our biggest issue right now isn't that we're just not testing, it's that we're moving really really quickly so in a lot of cases there's nothing actually written down to say how something is supposed to function. We're consistently changing what we're working on as we're working on it, which makes it difficult to do something like TDD or having test plans.

Note that the above is really mostly about the Careers product (and somewhat about Mobile) but not about the Core Q&A team nor any of our other internal teams. Everyone kind of does it a little differently.

One thing that both the Core and Careers teams rely on however is the continuous feedback we get through our meta channels like this. I think this is honestly the best part of my job, I love that people care enough about our product to help us make it better, and that we can rely on this being consistent time over time.

We're currently beta testing Docs, Teams, the Jobs tab, and have beta tiers of our iOS and Android applications, the fact that we continue to get great feedback on all of these channels makes me really really happy. Thank you (sincerely) for being one of our awesome bug reporters :)

12
  • 4
    Nothing written down about how something works. That is really scary don't you think? There needs to be some definition of how all of this works. I've probably posted over 10 or 20 simple bug fixes. You guys need mockups with proper process flows. Not knowing this information and just guessing is going to give you guys grief.
    – JonH
    Commented Dec 16, 2015 at 18:43
  • 1
    I'm scared of a lot of things, including that, yes. It will (and is) giving us grief. We write down what things do at a macro level (every month we write down what goals/features we want to get out and what success looks like) but nothing at the micro (it SHALL do X, it SHALL NOT do Y) level.
    – Kasra Rahjerdi Staff
    Commented Dec 16, 2015 at 18:46
  • I really recommend that you guys totally have a project manager on a feature. As much as it sounds hierarchical it is totally needed to avoid all of these issues. Good luck.
    – JonH
    Commented Dec 16, 2015 at 18:47
  • 2
    We have PMs on every project (but we're actively hiring more!) -- The issue that we're talking about comes into play when the PM is the only one that has all the knowledge of how something is supposed to work in their head.
    – Kasra Rahjerdi Staff
    Commented Dec 16, 2015 at 18:48
  • Okay makes sense. Thanks for the details.
    – JonH
    Commented Dec 16, 2015 at 18:49
  • 3
    Hey! This answer's great, but product managers do write out specs about how stuff is supposed to work and not work for much of this stuff. :P Not always, and sometimes the specs become out of date as we're iterating internally or publicly. (We're trying to get better at writing stuff down, but as Kasra already pointed out, we're also trying to balance that with moving quickly.)
    – Laura Staff
    Commented Dec 16, 2015 at 19:47
  • 3
    @Laura this would be a great time for me to learn what you actually do here
    – Kasra Rahjerdi Staff
    Commented Dec 16, 2015 at 19:49
  • 2
    In defense of the PMs, I'd say it's at least 50/50 between "not well documented" and "well documented, but the devs never read it".
    – David Fullerton Mod
    Commented Dec 16, 2015 at 19:54
  • @Laura - Instead of rushing and pushing stuff out - I don't see the need of some of the stuff you guys push out so quickly, why isn't there a reflection period and testing period before you guys put it out in beta. I know you guys are looking for some meta discussions but I can almost guarantee if you guys didn't rush this stuff out this many bugs would not be reported.
    – JonH
    Commented Dec 16, 2015 at 20:20
  • @JonH as I mentioned in the post, we do have different tiers of our webservers running for that purpose. We test out things for weeks (sometimes months) before actually pushing them out, but we're never going to be able to find all the edge cases and bugs that actual users can find.
    – Kasra Rahjerdi Staff
    Commented Dec 16, 2015 at 20:21
  • @JonH we had a few projects that dragged on much longer than they should have. We now are trying out a "strict deadline, negotiable scope" model in an attempt to fail/succeed faster and not waste months on the wrong project or trying to perfect something before shipping. We hope this means more betas, and a quicker turnaround from beta to a mature product.
    – Greg Bray Staff
    Commented Dec 17, 2015 at 4:01
  • And we honestly do have some of the best PMs I have ever had the pleasure of working with. SREs often rely on their docs to understand new projects and get answers to our questions.
    – Greg Bray Staff
    Commented Dec 17, 2015 at 4:04
8

[Closed captioning for the humor impaired (aka, "Meta regular"): it's a joke son.]

Yes, SO uses unit testing.

To be more specific, the unit of testing is known as the "guinea pig" (but often shortened to just "user"). Current test logs show that the average line of code has been subjected an average of 256'347 units of testing, plus or minus 317'132.

In case you're wondering: yes, that does imply negative units of testing. These negative units of testing are also known as "whiners". They are easily detected by searching for: [bug] and/or [feature-request] on meta.SO.

[We now return you to your regularly scheduled flaming, already in progress.]

1
  • 1
    s\whiners\winners ;) Commented Dec 16, 2015 at 17:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .