22

So far we have been able to enjoy (and I truly enjoy reading them) three "UX research time" posts, which have had various receptions, as has been discussed here to great lengths, and which I do not mean to discuss further in this post.

However, I have been wondering about how big the sample size is used for these researches. The only information on this I was able to extract from the last post is

We conducted interviews with people sourced from our social networks and the community. Participants included people with less than 3 years of programming experience, a college Python instructor, Stack Overflow moderators and answerers, and leaders of various technology communities

The post from May 2019 didn't seem to specify a sample size either

This research was motivated by earlier qualitative research that found many users consider the reputation system to be a barrier to participation [...] We conducted 1:1 user interviews with recently active Stack Overflow users with <=100 reputation, sourced from our research email list.

And the first installment didn't give much insight into sample size either

We gathered insights through a Meta.SE survey, UX teardowns of other sites, data analysis, and 1:1 interviews with people of varying levels of engagement with Stack Overflow, Stack Exchange, Teams, and/or Jobs. The target users of our research included [...]

Now I'm not a data analyst or anything in that ballpark, so this may be my lack of understanding, or downplaying the impact a single individual can have even in a big sample size (and please do correct me if this is the case). But I found being able to single out "A Python instructor" coming across as a sign of a rather small sample size, which taking the current amount of users (10747446 taken from the query explorer running Select Count(*) From Users) made me wonder about the sample size used for this kind of research.

Is there a specific reason why this stat isn't included? If so, what is this reason, and if not, can we get this in future installments?

2
  • 3
    The first quote here points out interviews were part of the methodology. That being so, there is nothing unusual in singling out "a Python instructor". If you have carried out interviews, you clearly should be able to single out individual answers -- that is part of the point of carrying out interviews in the first place.
    – duplode
    Jul 22, 2019 at 22:02
  • @duplode I hadnt thought about it that way yet, and definitely makes a lot of sense. Thanks for pointing that out
    – Remy
    Jul 23, 2019 at 5:37

2 Answers 2

11

Thanks for your question, and I'm happy to hear you've enjoyed the posts so far!

To build on Julia's points, UX research uses smaller samples than quant. research. The general distinction is that qualitative research helps identify the why, while quantitative research helps identify how much.

For usability testing, the general rule of thumb is to interview 5 users. For more exploratory or riskier endeavors (e.g. exploring new features/product lines, changes that affect core user journeys or would be expensive to build/maintain) we interview a larger set of people (generally 10+) and/or rely on mixed methods research (some combo of qualitative, competitive, and quantitative research). We also tend to conduct research iteratively - so 3 rounds of 5 users, as opposed to one massive round of 15 users. When choosing a sample size, a researcher might ask questions like:

  • How much risk is involved?
  • How unique is the problem? What can we learn from other sites (and therefore reduce the cost of our own research)?
  • How much do we already know?

Regarding the specific research posts you mentioned, here's more detail on our sampling/methods:

  • Profile and settings: mixed method research including a Meta survey (28 answers), 1:1 user interviews (6), UX teardowns, and quant. data analysis.
  • Reputation: 5 1:1 user interviews. Note that we wouldn't overhaul the reputation system based on such a small research project. Instead, we'd use this small project to help us ID initial trends and future directions of research.
  • Learning and teaching code: 8 1:1 user interviews. Also note that we wouldn't launch a new product line based on this research, which explored many different directions, some of which are more feasible than others. Rather, we'd conduct multiple rounds of research as we narrow in on idea, and supplement it with other research methods.
2
  • 2
    Thank you for your additions to Julia's points. This really helped me understand the entire process better, instead of "just" seeing the results. Looking forward to future installments!
    – Remy
    Jul 24, 2019 at 17:39
  • I wonder, statistically speaking, couldn't it happen that one misses a significant amount of "why"s with a small sample size? To test it, one could maybe once increase the sample size and then select only a smaller subset and ask the question how many "why"s on average were missed in any of these subsets. Jul 24, 2019 at 21:43
21

I hope that Donna gets a chance to chime in with an answer, since this is her area of expertise, but I wanted to share my perspective as

  • someone who collaborates with Donna and other UX researchers here at Stack Overflow
  • someone who works with statistics, modeling, machine learning, and so forth

Qualitative and quantitative research are really different, but are both valuable.

As a data scientist, I do almost all quantitative research, with big sample sizes (thousands to millions) for data sets that I analyze with statistical techniques.

As our UX design lead, Donna does almost all qualitative research, with smaller samples of data collected through participant observation and interviews, focused on interpretation and themes. It is normal (industry standard) for sample sizes to be smaller in qualitative research, and the sample size is not one of the most important aspects of such a project. Researchers like Donna go deeper with individuals, rather than analyzing massive data sets using computational statistics.

I ❤️ my work as a data scientist, but it's not the only, or always the best, way to understand an issue. Using both qualitative and quantitative methods to make decisions is the way to go! Qualitative researchers do need to be thoughtful and aware of the limitations of how they collect data and who they are hearing from, but I am happy to report that I always see Donna and other researchers here do that. And frankly, the same concern is true of data scientists and quantitative researchers, although it plays out in a different way.

3
  • 2
    Thank you for these insights, Julia. I hadn't fully thought about the aspect that UX needs more qualitative research than quantitative, as it is focused on interpretation aswel as just raw data. Do you happen to know how people are selected for this kind of research, and what attempts are made to ensure people from all kinds of backgrounds (in the broadest sense) are heard? Aside from years of experience/occupation. (I would like to emphasise that i'm not trying to say you guys are doing a bad job, I think it is great, just curious as to what happens behind the scenes for these researches)
    – Remy
    Jul 23, 2019 at 7:14
  • 5
    Most of the UX research that happens at SO happens with folks who have opted in to our research list, which you can do here. There are positives and negatives to that. For example, it is only registered users, which means we can automatically screen for account age, curation/moderation activity, etc. At the same time, it is... only registered users, which are a minority of our actual users. We look to other venues if we need a broader group of users. We also ask screener questions when inviting folks to particular projects. Jul 23, 2019 at 14:54
  • 1
    Very interesting to know that, thank you for your time and answers!
    – Remy
    Jul 23, 2019 at 15:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .