3

Based on this monthly average questions per day query and this questions per day query on SEDE, Stack Overflow is getting significantly over 2000 questions per day.

Currently, the RSS feed for newest questions (/feeds/newest) holds the latest 30 questions. It would need to be refreshed and retrieved by an RSS reader nearly 80 times a day. That's every 18 minutes. And that's assuming a steady flow of questions and not bursty behavior. The Stack Overflow Newest feed alone would need to be requested at least every 5-10 minutes.

On top of all of this, page request rate limits (which appear to apply to all feeds as well) are network wide. Although the details aren't public, there are 173 sites on the network. Each site has RSS feeds for recent, featured, hot, hot weekly, and hot monthly. This is in addition to a large number of tag feeds (that support AND and OR operations, optional sorting), user feeds, question feeds. Based on personal experience, both local and cloud-hosted RSS readers can very quickly run into rate limiting issues. Once Stack Overflow becomes involved, not only are you running into rate limiting issues, but that means missing posts.

I don't think that the tag-based question feeds or the various hot feeds are the solution if the desire is to be notified of new questions in a huge range of tags. If the question is improperly tagged or the tags in the query aren't maintained, questions that should be retrieved won't be. In the case where a cloud-based aggregator is being used, many people with their own feeds also prevents the aggregator from implementing 1 request for a feed and would increase the risk of rate limiting not only SO feeds, but feeds from across the network.

It feels like the solution is on Stack Overflow's side. I'm not sure what this looks like, though. Increasing the number of posts in the feed from 30 could be part of it. Slowing down the rate at which the feed is updated to reduce the questions in the feed could allow closed and deleted questions to not filtered out. Changing the page request rate limit for feeds could also be a good change, but would have to be carefully thought out.

9
  • 8
    I can't think of any good reason to have a desire to "be notified of new questions in a huge range of tags". It would serve little purpose beyond noise. Commented Jan 3 at 0:56
  • 1
    @Nickistired Unless you're interested in questions in a huge range of tags. I'm interested in questions in javascript, python, jquery, css, sql, mysql, json, ruby-on-rails, ruby, linux, aws, git, docker, pandas, numpy, sqlite, unit-testing, github, security, visual-studio-code, selenium, devcontainer. Given how cloud-based readers work and the inclusion of not only tags, but also titles and body, it would be more efficient for such readers to read the full question feed and filter on the reader side for users of the reader - it would be reading one feed instead of per-user feeds. Commented Jan 3 at 3:42
  • 1
    If you are interested in that many, why not subscribe to several feeds with more concisely related tags.
    – Thom A
    Commented Jan 3 at 10:55
  • @ThomA I explained that in the post. If I organize the tags I'm interested in into groups, I'd probably end up with 4-6 groups of tags, if not more. I would also need to manually maintain those feed subscriptions as tags change. Not only is it more work for me, but my feed reader would have to make 4-6+ requests every handful of minutes. Since some of the tags are frequently used tags, it would be even more likely to hit rate limits. And those rate limits would not only impact my ability to get my SO feeds, but all the other network feeds as well. Commented Jan 3 at 12:26
  • 2
    @ThomA Also, once you have python and javascript on the list of tags, those tags alone are a very large proportion of the overall questions, so its not exactly helpful to the problem. Python regularity gets 700 questions a day, and Javascript over 400. Commented Jan 3 at 15:01
  • This is not unique to Stack Overflow. How do other sites handle a high rate of 'posts'? Or doesn't RSS really scale? Even Dave Winer's feed is only 5-10 per day. Commented Jan 3 at 15:37
  • I agree that watching individual tags is not enough. Here is my main custom filter for JS questions* it includes quite a few tags because often people don't tag JS questions with the JS tag. They might put down Node.js or ES6 or jQuery when they are after some basic JavaScript syntax and/or grammar and/or functionality. I don't understand the pushback for this feature - it's undeniably an improvement. Or the pushback wouldn't be "use a more limited workaround". * Well mostly. There are few other tags but it's mostly JS-based
    – VLAZ
    Commented Jan 3 at 15:39
  • @PeterMortensen I don't understand your question. The maximum number of "posts" in a feed depends on the version of the specification used. RSS 1.0, 2.0, and Atom (which SE sites use) do not have an upper bound of the limit of posts. A feed could easily include 100 posts, 1000 posts, all posts from the last 8 hours, all undeleted posts from the last 8 hours. But some sites only include 10, 15, 25, or 30 posts in their feed. It's up to the generator to decide how many posts to include, especially in the newer specs that don't have limits enforced. Commented Jan 3 at 16:28
  • 2
    @PeterMortensen I can't think of many other sites where a single feed has the frequency of Stack Overflow. The issue isn't so much the number of posts in the feed, but the limited size of the feed coupled with network-wide request rate limits. You very quickly run into the issue where you can't request feeds fast enough to not miss content. Commented Jan 3 at 16:30

0

You must log in to answer this question.

Browse other questions tagged .