Skip to main content
38 events
when toggle format what by license comment
Jun 3, 2020 at 15:29 history edited CommunityBot
Commonmark migration
Sep 18, 2018 at 15:27 comment added Shog9 I think what Tim's trying to do here is apply some systems-level thinking to all of these problems, @jpmc26. You've been part of all this for a long time; I'd wager you've probably seen the direction that both this and similar communities tend to take... So you know then that folks tend to not stay in tidy little boxes: they grow, they change, they regress, they move on; there is an emergent behavior to the group that differs from that of any individual. The metric is important: it tells you something is wrong with the system... But you must understand the system as a whole to fix it.
Sep 18, 2018 at 13:47 comment added user50049 [2/2] .. has come to mean. We're not by any means done, this wasn't really even ready for scrutiny, and .. you see what I'm getting at here? I don't yet have all the data, but people are demanding a response, and I owe you an apology for doing my best to communicate? Meh.
Sep 18, 2018 at 13:47 comment added user50049 @jpmc26 We've put ourselves in the position to being open to questioning about essentially anything at any time. If we're reticent to answer, we're hiding something. If we give what we've got before we're ready, we're not thinking it through enough. If we say "you'll have to wait until we're ready to talk about it", then we're just kicking the can. I feel like, sometimes, our accessibility puts us in a position where we simply can't possibly win. This answer is already more than I was comfortable communicating, but I put what I had out there because that's what transparency [1/2]
Sep 18, 2018 at 2:14 comment added jpmc26 ...post high quality content due to the flood of poor quality questions. Furthermore, if this was all really just about, "We have this data and think it means this and are going to act on it," you owe everyone an apology for not just presenting it that way from day one. Instead, SO has cast inexperienced users as mere victims and done nothing to even acknowledge the quality problems that we all know contribute to frustration with low quality posts and the users behind them. That approach has dealt a serious blow to the community's trust SO worked so hard to earn.
Sep 18, 2018 at 2:11 comment added jpmc26 @TimPost I've thought about this for a long time, and I think I understand why the stat and the response bother me so much. They hide too much. If you look at only the aggregate, then you can't say anything about the underlying causes. One thing that's especially concerning is that SO has always considered an Eternal September effect, where knowledgeable users withdraw from the community, to be a serious risk that needed to be mitigated. This push is exactly the opposite, but the data you're presenting doesn't tell us anything about whether SO is having trouble just retaining users who...
Aug 30, 2018 at 17:37 vote accept kabanus
Aug 30, 2018 at 17:37 comment added kabanus Thanks for all the feedback. I see the discussion has died out, so I'm accepting, I hope this answer and comments will be a good go to for everyone.
Aug 30, 2018 at 12:37 comment added jpp @TimPost, It's tough, but we have to face the facts. Both a content perspective (most theoretical / ideological canonicals have been asked & answered, so we are left mostly with application) and user perspective (more schools / workplaces incorporate programming as a necessity where previously irrelevant) point towards an adjustment. Technologies and uses do change, so good Q&A will never disappear completely. But we also have to adjust to demands faced in the real world and strike a balance, e.g. the new question template is definitely a good step.
Aug 30, 2018 at 12:31 comment added user50049 @jpp We've experimented with surveys previously and they yield .. something, how useful it is remains to be seen. We've run A/B tests where requests to take short surveys appear in modal dialogs after certain events fire, and get feedback about that particular experience (e.g asking questions). The problem with asking if professional or enthusiast is people tend to significantly exaggerate or underrate their experience. And the wording matters, too - e.g "are you solving a problem for work" doesn't make you a programmer, it could just mean you got stuck with an access database.
Aug 29, 2018 at 18:21 comment added jpp It's interesting this post doesn't mention the quality of users, by this I mean do they all count under "professional or enthusiast"; would a survey be able to discern anything? This point is the most upvoted comment on a related post.
Aug 29, 2018 at 8:09 comment added Magisch It is rather interesting that you categorize these buckets by rep, when in reality rep is quite a poor classification since how much you get depends vastly on the tag you're in and the time you've joined, so the real measure of participation is constantly shifting.
Aug 28, 2018 at 20:59 comment added kabanus Tim do you mind commenting on @jfs's comment? in particular this? That seems more trouble than completely new users bouncing off.
Aug 28, 2018 at 17:27 comment added user50049 @jpmc26 Also: If that weren't complicated enough, it changes for ... who knows why ... every few months (even outside of usual seasonal stuff). That's even more head-scratching and digging we're doing, and partly why we're just not yet ready to present it as a whole, we only have an outline of a narrative. It's substantially complete now, but still lots of questions. You could get lost for years studying this stuff.
Aug 28, 2018 at 17:25 comment added user50049 @jpmc26 We do. I'm presenting as aggregate for the purposes of discussion, we're developing very different models for understanding the difference between burnout / fatigue and just people's natural tendency to wander off once they've done all they came to do, for instance. This also cross correlates with effects new users that 'bounce' quickly might have on veteran departures (correlation or causation) and a lot of other mining. As I note, I'm reticent to discuss it deeply, but we are looking at a lot of different things.
Aug 28, 2018 at 15:49 comment added Nicol Bolas @jpmc26: It does represent something meaningful: the flow of users away from the site. Those who were on it who decided not to be anymore, even if they were on it for mere minutes. But don't assume that the number by itself is all they know; that's merely the number Tim is presenting. The aggregate number is one they want to shrink, but from Tim's post (heh), it's clear that they're not ignoring the distinctions of why some users stop using the site.
Aug 28, 2018 at 15:42 comment added jpmc26 @TimPost I don't understand why you would combine users not assimilating with active users leaving. As far as I can tell, doing so mixes separate concerns and doesn't represent any sort of meaningful quantity. The more immediately meaningful quantities are the total user base growth/shrink rate and new user retention, and perhaps some analysis of how they interact. Statistics can be extremely useful, but it is incredibly easy to paint a false picture by focusing on the wrong numbers. You get a more accurate picture by considering a range of them, rather than focusing on one kind.
Aug 28, 2018 at 13:28 comment added user50049 (2) it's a little too close to strategy at this point thus guarded mostly because we don't fully understand it yet, so we don't know if it's a hairy spider or that ugly holiday sweater we bought last year hanging in the closet. We may come up with a way to make this metric more public (maybe even just for moderators), I think it would be useful, I just don't know if it's possible. [2/2]
Aug 28, 2018 at 13:28 comment added user50049 @jpmc26 Aggregate (and I hope you understand that I can't be much more specific). It's a combination of "veteran" users that go dark, along with the sum of non-veteran users that essentially just bounce off of us like hitting a wall. I will say there are two reasons why I don't elaborate more. (1) I'm sure the number isn't entirely off-base, but we're still refining how we derive it, which makes explaining it suboptimal at the moment and [1/2]
Aug 28, 2018 at 13:22 comment added user50049 While we measure this stuff in a variety of ways and we're still making lots of mistakes, we do not go by IP addresses because they simply aren't reliable. We've got offices where thousands of people use the same IP (Hello Samsung! Hello Microsoft! IBM! etc) so it's just noise for the most part. I don't think we'll implement any more advanced tracking metrics post-GDPR even though we could justify a business case for it just out of the potential pain in the butt factor. [3/3]
Aug 28, 2018 at 13:22 comment added user50049 There are also more absolute models, e.g. "Did they do anything this period? They were present! else, no!" but those paint a very jagged picture. [2/3]
Aug 28, 2018 at 13:22 comment added user50049 @TylerH We go more by just rep, and we do make light use of cookies, but we don't use 'sessions' in any sense one might see as conventional (perf and privacy reasons). This limits what we track even when folks are logged-in. We track a visit, but beyond that, you kind a need to do something in order to show as engaged (registration is one of those somethings). So we won't count you as missing if all you did was vote for 90 days, but if that keeps up, you might be 3/4ths missing at 180 days out, based on prior activity, depending on the model used [1/3]
Aug 27, 2018 at 21:23 comment added jpmc26 "At around 30 - 35% attrition rate on average..." I'm unclear what this means. Are you saying the user base is shrinking by 30% to 35% over... some unspecified time period? Or is this a decrease in the growth rate? Or what?
Aug 27, 2018 at 20:23 comment added TylerH @TimPost Haven't read the full post yet, but my gut reaction is a little surprise that SO measure user retention based on rep-gain (250 and 500 metrics), rather than repeat visits from the same IP or registrations. Or do you separate users and visitors into two different buckets? Or should I just read the whole answer? :-P
Aug 27, 2018 at 18:07 comment added jscs I appreciate this explanation, and your recent one on MSE (that Yvette linked from yesterday's question here. I need to study that one a bit more.). You folks, and you especially, are insanely awesome at being open. Thank you for taking the time to write this.
Aug 27, 2018 at 18:05 comment added kabanus That's a fair reply. I think this post, maybe with the q&a in the comments, should have come way earlier in this new process. Perhaps instead of another blog post... But no use crying over spilled milk, at least for me this post was helpful and even comforting.
Aug 27, 2018 at 17:52 comment added user50049 @PaulWhite If we go by the size of debt in terms of touches to effectiveness of helping the sites thrive, mod tools are probably at the top of the list of things we need to overhaul. That's kinda serendipitous, as in order to give mods better tools to handle comments in general, we kinda need to fix all of that. I hope to have some more news on that in the coming weeks.
Aug 27, 2018 at 17:50 comment added user440595 Some indication of what "We <3 our core users!" will translate to in terms of long-requested features or tools would be welcome.
Aug 27, 2018 at 17:50 comment added user50049 @kabanus I can say that market growth rates are very hard to track, and this is compounded by our annual survey being such a big influence in how the market is tracked, but "yes-ish" is the best answer I can give. We know we need to get better at tracking it, but are a little further along than some others, if that helps.
Aug 27, 2018 at 17:47 comment added user50049 @Dukeling It also depends on what other experiments or building we've been up to during that period. Massive changes to logging in and signup really skew things, and introducing spam protection really skewed anon participation metrics, we've historically done a very bad job of noting potential influencers in a way that's easily correlated (yay, historically, we're bad at history!). We're thankfully getting a lot better at that in recent months.
Aug 27, 2018 at 17:44 comment added user50049 @Dukeling For your second question, 90 days of pretty much no activity based on what you normally did is a good indicator of 'leaving', but if someone becomes active again, they just get counted on the positive side of a different sampling. On the other hand, 45 days seems to make sense in a lot of models that apply more uniformly to all sites, so .. at least 45 days, 90 days is a good indicator, but certain patterns of use require specialized scoping, especially for read-only types. I'm sorry if that's even more confusing.
Aug 27, 2018 at 17:41 comment added user50049 @Dukeling There's a bit of unknown there, and when you don't know what you're not competent at doing, it's difficult to assert that you're competent. However, the vast majority of outside feedback points at problems separate from interestingness / captivating content for entry barriers, although we're certain it's a problem, it's difficult to say how much of one. We can pretty comfortably assert that it's not nearly as much of a problem as what was slipping by in comments and such. But, again - disclaimer: we don't know what we don't know, if that makes sense?
Aug 27, 2018 at 17:39 comment added Bernhard Barker And do you define "leaving" as abandoning their account completely, or do you look at people whose contributions simply dropped down to a near-negligible point?
Aug 27, 2018 at 17:39 comment added kabanus I do not think I could have asked for a better answer. I appreciate your candor and forthrightness. I guess I was under the assumption most programmers (say 99.99%) were familiar with SO only when searching for a question (and not through reddit etc.), so they were not lost in the sense they wanted to join but were put off. It's cool you guys can check that. Do you mind commenting on whether you do compare growth rates to market growth rates? Please do not divulge any secrets, I'm must wondering if it's on the data dream team's mind.
Aug 27, 2018 at 17:35 comment added Bernhard Barker The question is: can you differentiate users losing interest, or making this a much lower priority, due to them simply being over the whole idea of helping build a Q&A, versus because they consider too much of the content to be low quality and that it's honestly just too draining to push past to find something actually worth reading, never mind answering.
Aug 27, 2018 at 17:00 history edited user50049 CC BY-SA 4.0
Stop saying 'so' more than a bad pearl jam song.
Aug 27, 2018 at 16:48 history edited user50049 CC BY-SA 4.0
added 93 characters in body
Aug 27, 2018 at 16:42 history answered user50049 CC BY-SA 4.0