This is long, even after I tried to shorten it. You can skip to the bottom bullets for some key take-aways, but if this really interests you, I suggest reading the whole thing. Sorry about the book.
The new user growth rate doesn't have to be fast, it has to be commensurate. And by that, I mean on any random 90-day sampling, you see more users staying and climbing over 250 rep than leaving in aggregate.
Those that leave fall into two major buckets:
- Got to at least 500 rep with every sign that they could continue to contribute positively. In there, you've got:
- Made it to 3k+
- Made it to 10k+
- Became a trusted user
- Didn't get to at least 500 rep, in there you've got:
- Didn't do anything but create an account, favorite and subscribe to a few tags, shared a few posts, etc. Positive but read-only existence, and then vanished.
- Didn't unlock even basic creature comforts. Maybe 1 or 2 posts.
- Got to between 50 and 250 rep and just vanished, no signs of contributions not being welcome (by votes / flags / etc).
- Got between 1 - 50 rep, but obviously struggled (we can usually pinpoint how / why).
So, if you get past 250 rep, you at least (statistically) stand a chance of having another shot at leaving when you hit 500 rep, and then again at regular intervals as you climb the ladder. The bucket between 0 - 250 becomes pretty important because it feeds every other bucket.
People leave for many reasons. Trying to explain everything is foolish.
But, you should always try and figure out what you can learn.
Those that left past getting 500 rep are actually the easiest to explain. We've looked at it over a dozen (literally) ways and no matter how we scope the phenomenon, we can pretty clearly see that:
It wasn't us. Maybe they lost interest, changed jobs, passed away (morbid, but common), lost the ability to post publicly about work, etc. In essence, stuff we maybe could capture better and account for? But, if we're honest with ourselves, we probably can't intervene much here, life happens.
It was us, and we can clearly see why. A false positive on voting reversal scripts, folks losing rep because someone else removed their account, folks losing rep because something got deleted, or a thousand other little ways where if we look hard enough, one or several events stand out as a probable culprit or at least catalyst. DAG works on mining and running tests around this stuff to try and take advantage of what our own data is telling us.
This is an interesting journey because you often don't realize that you were looking at things completely wrong until you've... well, you've been there, ever had to confront an early decision that proved fatal near the end of a project? That's why we now have a full time data team and UX researchers (I had no idea that was a thing until we hired them) and increasingly better models.
We have historically sucked at parsing outside feedback.
Even considering how much criticism is in the air right now, we open ourselves to more criticism than any mature company that I've ever encountered in the technology sector. In fact for every single site we have, there's an accompanying place where people are encouraged to speak what's on their mind. We call this inside feedback because:
It's a dynamic where we hold all of the power because it's our platform (which is why we try our best to be as liberal as possible and allow as much room as we can for passion as long as an attempt at respect is maintained).
It tends to come mostly from people that know how things work, which is why they can make great arguments for changing things, even if we don't agree with them. The small rep barrier for child meta sites alone pretty much says if you want to talk about how we do stuff, you need some prior participation. Meta means murder and at our scale, the barrier to entry does need to be slightly higher than "I think I might have an opinion about [thing]".
That means, outside feedback generally comes from e-mail, Twitter, Facebook, Reddit, Hacker News, Blogs, Conferences, etc. The only commonality in these sources are that they are extremely difficult to quantify and parse in aggregate. Going back to the people that leave - we can use the same software that monitors our network connectivity to alert us of a sudden change in users we have.
We failed to capture the entirety of users we just weren't getting (which suddenly makes that 0-250 rep bucket a lot bigger), and why, and how much of that was related to the perceived hostility of the site. We knew it was a problem (I've personally lost sleep over it for the last 4 years as I saw trends but had only anecdotal evidence). Until we finally hired and on-boarded people that dream in data, we didn't know how bad it was.
The good news is, it's not that bad yet.
At around 30 - 35% attrition rate on average (that's all I'm going to share, and it's only for discussion), we're not doing too bad. In fact, right now, we're doing pretty darn good. And we absolutely, positively, most decidedly do not want that number to reach anywhere near 50%, because that's the point where stuff gets real and Joel takes away our standing desks.
It's difficult to speculate on prospects for a community of this size because frankly, we're writing the history academics are using to make models. But I (and lots of other folks much smarter than I am) believe that we have crossed an event horizon of sorts, and while new users should never be commoditized as "a dime a dozen", I don't think we'll ever reach a point where we've burned through the market faster than it grows.
But this isn't about $$$. It's about everything.
I guess money is a part of everything, and we're not going to pretend that we're not a business, but stopping there is short-sighted.
Until a few years ago, Stack Overflow was kind of elusive when it came to community metrics. The thinking we established based on what we learned from hundreds of other sites just broke horribly when we applied it to SO. What we knew from the network made what we were seeing on SO make even less sense in many cases; we just didn't have the tools and people we needed to really study it, and we didn't know we lacked those. We just attributed it to 'scale'.
Yes, more users sometimes eventually translates into more revenue for us. In some cases we can charge a little more for ads, in other cases it might inspire more creative partnerships with companies using new technologies as new sub-communities form around those tags. But broadly, more users will not translate into more revenue unless we can find ways to provide enough value to those users that they positively engage.
But had we been able to understand and better parse the different kinds of signals we were getting two years ago or even sooner, you'd have seen us prioritize this then. This is the kind of thing you want to nip in the early stages, so to say, and it's not just Stack Overflow that we're trying to help.
So putting that all together:
Our success depends on finding innovative ways of delighting people that tend to come from a very technical audience. Sheer numbers just occupy a database unless we can inspire those folks to do something.
We absolutely must course-correct on outright ignoring and failing to act on outside feedback. That doesn't make innovating and delighting folks that stick around less important, it's just a different part of the machine that happens to need urgent attention (largely, because we ignored it, because we didn't see the forest for the trees). All feedback needs sunlight to disinfect and we have to find a way to embrace perspectives of non-participants. We're getting better.
We're still a growing and relatively small company with limited resources. While we don't struggle and thrash around in the name of profitability, we're still going to struggle and thrash around as we encounter even scarier problems that come with becoming more established. We're way more public and open than anyone else and as you can see, it has been a heck of a ride. We wish we could do everything, right now, and make everyone happy. We'll get there.
We can't say money isn't part of it, we're a business, that's like saying oxygen isn't a part of being alive. And I think it's perfectly normal to connect knee-jerk-ish behavior to the worst possible cause, but we really wish you wouldn't do that, we can't always react to stuff and get sufficiently ahead of it in discussions here in the amount of time we feel that we need to act. We act on strategy, your input is a big part of that strategy, but there are other components.
Speaking of strategy ...
I've pushed my comfort level in the volume of what I've shared, and I'm not going to dive further into strategy with discussions about methodology, how I get my numbers, what we look at, how we plot certain things, etc. This is meta, you're welcome to accept our narrative or create your own, as long as you do it respectfully.
But there's really not much room for anything to hide, there's no boogeymen. It's not greed, it's us being stewards of what we've built.
We will (hopefully, very very soon) return focus to more tangible things that say "We <3 our core users!" instead of offering promises that we're working on it. This will probably resume after we finish annoying the heck out of a bunch more sites getting the unified theme so we can finally have one codebase where new features can just flow without 10 years of code smell.
And, frankly, paying down debt is the focus - everything else (including the sudden need to devote resources to new user experiences and preconceptions) are things that we wish could put off until we finish what we're doing. But when you realize OH CRAP EVERYTHING IS ON FIRE, ignoring or (worse) acquiescing to those circumstances is what leads to failure.
I'm happy to take questions but please understand that I've already over-shared to an extent. After almost 10 years of most of my time sinking into this beast we've built every single day, you can bet I'd be outta here if it was just about the money.