140

Welcome to February's installment of the regular, bite-size, data-focused updates I am sharing with Meta! You can check out previous posts if you like. We just wrapped up the fielding period for the 2019 Developer Survey, so for this month, I want to share one more analysis from 2018's data, before I buckle down with my teammates to the task of the new batch of survey responses.

In 2018, one of the questions we asked respondents was:

How would you describe the Stack Overflow community, as a whole?

You can see the most common words that respondents used to answer this question in the main analysis/report. Respondents said that the Stack Overflow community is helpful, good/great/the best, focused on questions and answers, and so on.

We have more information about the survey respondents who gave us these answers, though, so we can dig deeper into these kinds of opinions about the Stack Overflow community. For example, how do the words people use to describe Stack Overflow change with years of coding experience?

We can answer this question with the following modeling approach, similar to what I lay out in this blog post:

  • Count words used by users of each experience level
  • Model these word counts as a function of years of experience
  • Find the words that change (either up or down) significantly, in the sense of p-value and effect size

enter image description here

These words are the ones that change the most with years of experience. They are not the most common for each level of experience (those are mostly the same as the most common overall) but instead are the ones that demonstrate a change in how they are used by less and more experienced developers.

The most experienced developers are more likely to see the Stack Overflow community as a group of focused geeks, and are more likely to use the word "diverse" to describe the community. I was curious what these users meant by that, so I looked at these responses in their entirety. It turns out these responses are addressing diversity of technical background, interest, and nationality.

Less experienced developers are more likely to talk about Stack Overflow as a community for learning.

  • "best place to learn"
  • "source to learn and share knowledge"

More experienced developers, by contrast, are more likely to frame their understanding of the Stack Overflow community in terms of a resource.

  • "very effective and valuable resource"
  • "an invaluable resource for any programmer"

Sometimes we as a community experience these understandings of site identity in tension, and we can see these differences in the very words our respondents use to describe who we are as a community.

It's unlikely we'll resolve these deep tensions in this single post, but what are your thoughts or reflections? Do you have topic ideas for future data science explorations?

22
  • 23
    It would be interesting to see these analyses performed on the 2019 data. I'd like to see how the "welcoming wagon" has affected these responses.
    – cs95
    Commented Feb 14, 2019 at 20:15
  • 17
    I really like these analytics. Keep it up!
    – roberrrt-s
    Commented Feb 14, 2019 at 20:15
  • 12
    Wow, that's a very interesting slice of information! Nice presentation choice!
    – jscs
    Commented Feb 14, 2019 at 20:38
  • 4
    It seems really interesting to me that "learn" will go down with experience, than up with very high experience. Can we infer that those very experienced people think about learning from SO again, or are they simply talking about other people learning? Commented Feb 14, 2019 at 21:01
  • 144
    I wonder how much this correlation is due to experience, and how much of it is just due to age. Different generations have characteristically different ways of expressing themselves and of looking at the world. I suspect that may, as much or more than experience, affect why respondents spoke of a “resource” and “geeks”, rather than “platform” and “developer”. I have no idea how you untangle programming experience and just plain age, but they are different.
    – Cody Gray Mod
    Commented Feb 14, 2019 at 21:09
  • 1
    I really like these analysis, where I can get complete design of implementation? Commented Feb 15, 2019 at 7:09
  • 9
    Another interpretation is that certain groups of people (e.g. people saying "geeks") are more likely to stay active on SO. >50% of the respondents have less than 10 years of experience, so survival bias applies here. Commented Feb 15, 2019 at 10:01
  • 6
    Why did you not exclude the obvious outliers from your trendlines, or use simpler curves? Most of the curves are pretty ridiculous. What's the sample size on them?
    – OrangeDog
    Commented Feb 15, 2019 at 13:06
  • 3
    Can you elaborate on the "deep tension" bit? I'm not sure it follows from the statistics that there's any tension about what stack overflow means. Maybe variance, but not tension.
    – kettlecrab
    Commented Feb 15, 2019 at 14:39
  • 1
    Are these levels approximately the same size? What percentage of respondents are novice, intermediate, experienced, etc?
    – M Waz
    Commented Feb 15, 2019 at 15:34
  • 1
    @CodyGray you untangle programming experience and age with a sensitivity analysis looking at both. They have both variables. It would be a good idea for them to do this. Good suggestion :)
    – De Novo
    Commented Feb 15, 2019 at 17:43
  • 2
    @OrangeDog The curves shown here aren't the fitted models but for visualization purposes; the approach I took, linked to in the blog post, is logistic regression modeling since this can be considered "success" data (was the word mentioned? yes/no?). In that context, removing outliers would be quite a bad idea. There were over 5000 responses to this question overall.
    – Julia Silge StaffMod
    Commented Feb 15, 2019 at 22:00
  • 9
    @CodyGray Great point! We did ask about age last year, so I could fit the model with both predictors and see which is more important, for which words. Nice idea!
    – Julia Silge StaffMod
    Commented Feb 16, 2019 at 1:11
  • 1
    Is the experience in languages themselves, development experience, other? Is it only taken from those who call themselves developers? I did not, for instance, but still use SO because I develop as part of my career as a data engineer and as a professor. I also know there are others careers, physics & data, which use SO heavily but are not developers. Who can have very different views and goals for SO then full-time developers. Granted, I still don't know if I'm even suppose to take the "developer survey" since I'm not a developer anymore (no dev jobs here) - and its not called SO Survey. Commented Feb 17, 2019 at 14:56
  • 3
    <1% incidence? How do we know that we see more than noise?
    – Raphael
    Commented Feb 25, 2019 at 11:11

5 Answers 5

18

I wonder if the differences of the age factor are due to the amount of experience we have or due to how we learned to program computers in the first place?

For example, I have > 25 years experience, and I learned through books. By the time I was writing code that delt with the internet, I already had 5 years of professional experience, had graduated from college, and was already using O'Reilly's "... in a nutshell" books for reasources, and was the sole developer at the company so I had to learn debugging and such stuff by myself.

Thus, I have no idea how to respond to beginner questions that really need debugging help or basic concepts because I never got to ask them myself, and I know I'm no good with them, so I don't try.

I wonder if the lack of patience with beginners is really a matter of years of experience, or if it is because "we all learned the hard way, so you should too"?

Maybe there is a way to look for that in the data?

4
  • 8
    It took me 6 months after I threw my Java for beginners book frustrated in a corner before I grasped the concept of variables before it suddenly clicked in my head. I had nobody to ask, and not a clue where to find help. Ah, the olden days where you had to figure it out yourself, and the frustrating path for every concept learned. It were different times. Commented Feb 18, 2019 at 18:56
  • 9
    I learned to code in the pre-Stack-Overflow era myself (HEY THERE FELLOW 90s FOLKS) so this resonates a lot with me. I think people learning to code now are having a qualitatively different experience than we did.
    – Julia Silge StaffMod
    Commented Feb 19, 2019 at 19:05
  • If I were to suggest to a newbie to read a book on their chosen tech first, it wouldn't be because "they should learn the hard way, as I did". It would be because, imho, a well written book is still the best way to understand a technology. Such a tome should cover not just the hows but the whys and answer questions you wouldn't have even thought of asking. 2 of the best tech books I ever read, 20 years ago+, planted seeds which remain with me today even though I no longer use the languages in question. Tl;dr: newbies - don't discount books! EBooks are fine too! :) Commented Feb 27, 2019 at 13:07
  • 1
    @Tschallacka I had most of the early works of the Linux Documentation Project printed out on dot matrix, and spent countless hours reading man pages and bash help and using apropos, just to be able to write bash scripts. I wouldn't have the patience for that any more - I Google for the answer like everyone else :) Commented Feb 27, 2019 at 13:12
12

Why not both?
I personally don't see a conflict between the identities of learning and as a resource. The problem shouldn't have to be a problem when the mindset of a lot of people are willing to change from the stark attitudes they have.

When I look at the new questions posted when the schools have started again and people are learning new concepts and the homework questions come, the usual response is a barrage of close and downvotes. A couple of comments when the person is lucky where to look for more information. I personally feel like there is little reflection on the intent of the poster. Some questions just scream: I don't have the slightest of clues on how to get started please help me. I personally try to help those people either via comments where they can look for answers or deepening their understanding. When I feel that an answer might help others with a similar struggle and I find the risk of close votes minimal, I might invest in an answer to educate the question poster and possible others who might stumble upon it.

That doesn't mean we should give free roaming access for help vampires with gimme the codez questions. But I feel like a little time should be invested in trying to discern the intent of OP. Does OP want us to do homework for him or does OP actually wish to learn? The biggest difficulty in learning is knowing what to look for, how to apply different concepts into one solution, and we should keep the quality of Stack Overflow high.

What I propose is that when a question is closed as too broad, typing error or whatever close reason was picked, and people who have sufficient rep and get satisfaction from helping people and not per se for the points can open a chat room with the user, sort of "mentor chat" where things can be clarified, spoken out and solved within chat, away from the main site. Others could join the chat to either help or watch when not enough reputation. When the issue is resolved or OP has learned enough to make it on her/himself the OP could mark it as resolved and the mentor(s) could get 5 points for the work they invested. I would say for mentor chat privileges one should have sufficient rep and familiarity with the site, something like 500 points and more than one year membership.

These are just my musings on how to keep Stack Overflow a high quality resource for looking up things, whilst also enabling the offering of help to newbies by those so inclined.

6

It is great Stack Exchange has a Data Science team, I am sure you are doing awesome things behind the scenes. However, I miss posts showing results/patterns you have discovered through data which lead to improvements on how the site works. In other words, I miss more applied analysis.

Do you have topic ideas for future data science explorations?

I'd like to see real data as an answer to my question in Close a question when it gets 3 (or 4) leading 'Close' votes?. I went the most far I could using data from SEDE.

If my queries are correct, I have an hypothesis that when a question receives 3 or 4 close votes without them being intervened by an edit or 'Leave Open' vote, it gets closed anyway (I believe real data would make this even more evident; it is explained why in that post). This means we are wasting 1 or 2 close votes that could be cast to decrease the Close Vote review queue backlog.

By making such adjustment to the source code, we could decrease the number of review votes that age away without changing the pattern on how closing works, i.e., without being less or more rigorous than what we already are. While other proposals about closing are analysed (for example, being more rigorous by reducing the total close vote count threshold no matter what; or giving high-rep users extra weight on close votes) this could be a change with immediate impact with no cost (except the one adjusting the code).

It would be great if the Data Science team could take a look on this. Again, if the queries are correct, it is like at least 50% of the job is already done.

5

I probably use this website in the way very similar to how inexperienced folks do: when I bump into a coding problem in the topic where I have little experience or no experience at all, I search the web for solution and often find an answer at Stack Overflow.

You see - despite having quite a bit of programming experience in general, when it comes to using SO I am usually just as inexperienced in the topic of interest as your next student doing their homework. Okay, I will drill deeper in the way how I use the site and try to guess what could be different between me and them.


One thing that springs to mind is what happens when answerers explain how they arrived at the solution. To me this part is rarely interesting because I am well aware of typical ways to solve coding problems; if I wouldn't find the answer I would probably solve it myself using the same approach.

For less experienced developers things could easily differ here. "Oh so this kind of problems can be solved by studying specification! (...by applying bit mask, using profiler, etc etc)". I think that besides providing concrete solution to particular problem SO answers can often teach newbies various ways of doing things - that's where they learn while I am not.

You can probably try to check if there is such a difference by asking respondents questions like, how much they learned about approaches to solving programming problems from answers at Stack Overflow. If this part of my analysis is sensible then responses from experienced programmers will tend to be mostly like "not much" while inexperienced ones will be likely tell that they learned a lot.


Another thing we may differ about is expectations of what happens after we discovered needed solution. I learned from experience that this doesn't help me memorize it. There are few things that I remember having repeated but very infrequent need and every time I need these things I have to repeat search for solution again, I can't really learn these things without sufficient practice.

Inexperienced developers who haven't broken their noses like that may fall into believing that once they found solution at SO they have learned it forever without any additional practice. They may perceive this as learning things (while my experience suggests that it really isn't).

Speaking of survey, it seems hard to properly spell a question that would explore if there is such a difference indeed.

Maybe there should be something like asking respondents how likely it is that some time later (if they don't practice in the topic) they will be able to repeat solution discovered at SO without searching for it again, something like that. If my analysis holds, such a question will probably get more optimistic (and less realistic) answers from inexperienced developers.

2
  • 6
    How often do you find your answer, then see you had already upvoted it, so must have had that same problem before. Happens to me every time trying to use Git. I can't remember the syntax, and I can't remember I've had the same problem before. :D Commented Feb 22, 2019 at 6:34
  • 4
    @CrisLuengo The best is when the answer that helps me is one that I posted... proof that I never really learn. Commented Feb 25, 2019 at 22:40
-17

Maybe it's time to consider a new site: Stackoverflow for begginer?

We have a rift between 2 groups in here and it's not going to change. We have 2 communities in the same site that want 2 different things.

9
  • 2
    Related: Stack Overflow for dummies.
    – Wai Ha Lee
    Commented Feb 19, 2019 at 16:51
  • 27
    "We have 2 communities in the same site that want 2 different things." No, we don't. We have one group who all wants exactly the same thing: high quality answers to their questions provided by experts. A Stack Overflow for Beginners would become a vast wasteland; who would provide these answers, and more importantly, who would audit and maintain their quality? Congratulations, you've just reinvented Yahoo! Answers.
    – Cody Gray Mod
    Commented Feb 20, 2019 at 6:50
  • @CodyGray convert reputation to a cryptocurrency, bounties award crypto from asker to answerer(smart contract to ensure payout if answer is deemed "valid" by upvotes by enough peers),you earn crypto at certain tiers of reputation ("5,000" onwards). we have the technology to motivate semi-experts(1-2 years of experience) to help beginners.ofcourse the features that worked could be migrated over to the main stack site Commented Feb 20, 2019 at 7:02
  • 9
    Who funds this currency, @kkarakk, crypto or otherwise? Where does the money come from? Reputation is free because it's not tied to anything in the real world.
    – Cody Gray Mod
    Commented Feb 20, 2019 at 7:05
  • @CodyGray stack would have to pay initially ofc(maybe voluntary mining by community,maybe sponsored donations by companies), you can already see sites that award crypto for good articles/q&a (musing). the crypto doesn't have to be a terribly impressive amount, people are motivated even by pennies and crypto appreciates over time anyways(in the well designed coins). the goal is to get journeymen to help beginners not fuel the crypto-craze Commented Feb 20, 2019 at 7:12
  • 2
    @kkarakk How...would that help? Cryptocurrency is a solution in search of a problem it can solve. What happens when the currency bottoms out? Crypto sounds like a simple solution, but it doesn't sound like it's been thought through very well.
    – fbueckert
    Commented Feb 28, 2019 at 22:17
  • currencies backed by someone never bottom out - again why does everyone have such a hate boner for crypto? i'm saying that having some form of payout as incentive, it doesn't have to be crypto. crypto just seems(in my mind) the easiest way to ensure payouts globally. Commented Mar 1, 2019 at 6:01
  • @kkarakk meta.stackoverflow.com/questions/380790/…
    – fbueckert
    Commented Mar 1, 2019 at 14:03
  • @fbueckert linking a heavily downvoted question without answers just shows that people in stackoverflow's community hate crypto, it doesn't prove anything else. remember the days when everyone hated javascript/jquery? edit: also using the bad question sledgehammer in the comments didn't make it any more likely for that question to be answered Commented Mar 2, 2019 at 5:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .