6

As Stack Overflow gains more users, questions and answers on this site, or network of sites increase exponentially too.

On the other hand, the amount of human knowledge does not increase nearly as fast. So in theory, any new question will provide decreasing amount of value to the knowledge database on SO.

Just out of curiosity, will SO eventually reach a near saturation point where on average any new question will provide near zero new information? (zero value is quite impossible due to newer version of OS, programming languages and their improved features & new technologies).

Whether making new interfaces every few years and forcing people to spend weeks to re-learn everything is for or against productivity is, of course, a different issue.

3
  • I find it telling that the sentiments that we'd already reached this point were expressed three years ago, and we're still going. Commented Apr 7, 2014 at 11:13
  • Questions and answers are not exactly the same as information and knowledge. For example there are two questions: "2 x 3 = ?" & "5 x 6 = ?" two different questions will have two different answers, but the core knowledge required is the same: simple multiplication.
    – TelKitty
    Commented Apr 7, 2014 at 11:32
  • Sure, and we get an awful lot of both of those every day. And it is, statistically, the average and the overwhelming majority. But we still get new stuff. Commented Apr 7, 2014 at 11:34

2 Answers 2

11

In the dystopian future in which all questions have been asked and answered on Stack Overflow, every new question will be closed as an exact duplicate of an existing question. (This assumes, of course, that we figure out a way to incentivize that behavior.) So we can estimate how close we are to saturation by measuring the ratio of otherwise good questions that are closed as duplicates. Our existential risk has increased each of the last 7 years and we are well on our way to yet another increase:

year questions dupes dupe_rate 
---- --------- ----- --------- 
2008     55347     410   0.74%      
2009    337729    2431   0.72%      
2010    691701    6636   0.96%      
2011   1192282   16063   1.35%      
2012   1621485   26562   1.64%      
2013   2006560   41549   2.07%      
2014   2229369   48971   2.20%      
2015   1665499   40157   2.41%

This doesn't quite add up to 10 million because this view doesn't include questions closed for reasons other than being a duplicate. It's also quite likely that the duplication rate will increase in the future even for questions before 2015 as people discover more duplication. However, while we might be getting more than 2.4% of our questions closed as duplicates, we aren't in any real danger of hitting 100%.

However, some tags have matured more than others:

tag            N    dupes dupe% avg_score deleted deleted% closed closed% 
-----------   ---- ------ ----- --------- ------- -------- ------ ------- 
sass          2110    131  6.21      1.01     475    22.51    212   10.05       
c           181169   7461  4.12      1.12   29258    16.15  25993   14.35       
c++         445750  17788  3.99      1.61   73827    16.56  61354   13.76       
r           114421   4223  3.69      1.35   18919    16.53  12665   11.07       
php         956392  30834  3.22      0.41  204317    21.36 143504   15          
jsf          17888    576  3.22      0.8     3629    20.29   1009    5.64        
c++11         1694     53  3.13      1.07     284    16.77    129    7.62        
java       1125545  33909  3.01      1.06  217949    19.36 143140   12.72       
python      526526  15245  2.9       1.65   78528    14.91  51593    9.8         
javafx        2970     83  2.79      0.43     564    18.99    163    5.49        
swift        14097    391  2.77      0.97    2108    14.95    932    6.61        
javascript 1060086  28983  2.73      1.13  187398    17.68  97863    9.23        
bash         32961    885  2.68      2.61    3272     9.93   2712    8.23        
matlab       50808   1288  2.54      0.5    11712    23.05   5875   11.56       
objective-c 113674   2874  2.53      1.56   19969    17.57   8350    7.35        
delphi       28911    723  2.5       1.95    3870    13.39   2772    9.59        
c#         1007001  25046  2.49      1.35  172802    17.16 101813   10.11       
git          53232   1285  2.41      7.24    5892    11.07   3170    5.96        

With the exception of C++11 and Swift, these technologies were introduced before Stack Overflow launched. (Though plenty of people had questions about C++0x from nearly the beginning of the site.) Newer technologies tend to have lower duplication rates:

rust          1753     38  2.17      2.42      69     3.94    55     3.14
coffeescript  1342     23  1.71      2.19     187    13.93    59      4.4 
go            5572     94  1.69      2.19     589    10.57   330     5.92
mongodb      23142    346  1.5       1.33    3550    15.34  1303     5.63
hadoop       16087     36  0.22      0.6     3237    20.12   750     4.66
docker        2685      4  0.15      2.89     231     8.6     66     2.46

So one faint glimmer of hope is that new technologies must be learnt. As you imply, the downside consists in each technology generation needing to build up a corpus of the same basic questions using new syntax. Once someone invents the Esperanto of programming languages and we stop playing with the rest, perhaps we'll finally see the writing on the wall.

If you want to look up your favorite tag's duplicate doomsday number, I've published a Google spreadsheet. For the purposes of these statistics, I'm only looking at the first, or primary, tag. So questions with both and will be counted as C++ questions, since there are more of those. Not fair? I agree, but it's a pretty good way to eliminate secondary tags such as . (It's also expedient in simplifying my query.)

1
  • @pnuts: I'd thought about it. It doesn't surprise me that bash and git have a lot of duplication because they aren't as complicated as C++ (but what is?) and so the questions will tend to cluster around common problems. Quantifying the complexity of a tag seems difficult, however. I'm not sure what to make of SASS's high duplication rate compared to total questions. Presumably there's some aspect of the language or it's users (or the culture of the tag on SO) that causes more questions than you'd expect to be duplicates.
    – Jon Ericson Staff
    Commented Sep 7, 2015 at 19:06
5

Just out of curiosity, will SO eventually reach a near saturation point where on average any new question will provide near zero new information?

We already are in this state, which makes a lot of Stack Overflow regulars fairly grumpy: on average, any new question is about someone getting a null reference exception for the first time, or doing something else silly that has been asked about many times before.

However, to quote PHP: a fractal of bad design's opening paragraph:

programming is a hilariously young discipline, and none of us have the slightest clue what we’re doing.

New technologies and languages are always being developed. The programming world's knowledge is always growing, and new best practices for what to do with existing technologies are always evolving.

Our discipline is growing far too quickly for us to hit the barrier of every possible question that could possibly be asked about it anytime soon.

Even sites like Physics or Mathematics which cover ancient disciplines show no signs of hitting such a barrier. There's simply too much stuff to ask about: if both have 10,000 concepts, you'd have to have just short of 100,000,000 questions (10000 × 10000) before you have one question about each thing in combination with one other thing. And you're going to have more than one question about each of those combinations, and that doesn't cover combinations of three or four things together, or using a thing several times with itself.

Physics.SE has only recently passed 100,000 questions, let alone a hundred million.

So in total: no. We're very unlikely to reach a saturation point anytime soon.

2
  • 1
    What's the deal with all these sites trying to take on google from different angles - facebook from social networking, SE from questions & answers (different ways to gather & retrieving knowledge). I love information technology - we spend decades gathering, sorting and re-sorting information, but still don't know what to do with them ...
    – TelKitty
    Commented Apr 7, 2014 at 14:56
  • 2
    If I'm understanding you correctly, they're not actually taking on Google, since this isn't Google's turf. Stack Exchange is actually taking on forums, which is what people used to do Q&A in, but they suck at Q&A. Take this thread as an example, note that on page 10 there's a thank you! to someone for posting a solution but it'll take some digging around just to find it. For social networking: Google Plus didn't launch until 2011; seven years after Facebook launched in 2004. Commented Apr 7, 2014 at 22:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .