3

I have a newsletter where the total number of monthly SO questions is a data point for technology popularity. Here are JVM languages and their competitors, for example. I also query the total number of monthly SO questions for context. I use the Data Explorer for that. The query is below and goes up to Oct 31, 2023.

I ran the query on Nov 28 last year (Git timestamp) and stored the results. Then I ran the same query unchanged today: November 2022 now has 10% fewer questions, December 2022 and January 2023 19%, February 2023 is off by nearly 8%, and October 2023 is off by almost 5%.

I can see why the number fluctuates a bit. Emphasis on "bit": From January 2009 through October 2022, the average difference between my November 2023 and February 2024 results was 0.05%. Even from March to September 2023, it was just 0.45%.

You can find all the data in this Google sheet, problem area highlighted.

What's wrong with my query to get such huge differences?

DECLARE @StartDate DATE = '2009-01-01';
DECLARE @EndDate DATE = '2023-10-31';

WITH Questions AS (
  SELECT
    Id,
    CreationDate,
    1 AS AQuestion
  FROM Posts
  WHERE
    PostTypeId = 1 AND -- 1 for questions
    CreationDate >= @StartDate AND
    CreationDate <= @EndDate
),
MonthlyCounts AS (
  SELECT
    DATEADD(month, DATEDIFF(month, 0, CreationDate), 0) AS Month,
    SUM(AQuestion) AS Questions
  FROM Questions
  GROUP BY DATEADD(month, DATEDIFF(month, 0, CreationDate), 0)
)
SELECT
  Month,
  Questions
FROM MonthlyCounts
ORDER BY Month;
7
  • 4
    Isn't this just deleted questions? Have you checked the PostsWithDeleted table for more questions there? Most of the data is stripped from the deleted questions but metadata should be there - tags and creation dates, for example.
    – VLAZ
    Commented Feb 13 at 20:07
  • Do I understand you correctly? You say that between Nov 28, 2023, and today, people deleted 19% of their questions from December 2022 and January 2023 but only 0.22% of all the October 2022 questions. Correct? Commented Feb 13 at 20:12
  • 4
    Yes, that seems pretty likely, due to the fact that questions that have received little to no response are automatically deleted 365 days after posting (see the "If the question is more than 365 days old, and ..." section). This results in a large spike in the deletion rate of questions when they hit that age. You could probably verify that this is correct by checking the DeletionDate column in PostsWithDeleted.
    – Ryan M Mod
    Commented Feb 13 at 20:35
  • Further evidence that this is the cause is that the period between queries spans parts of November and February (it's not exactly proportional, but recall that SEDE is only updated once a week), which have medium differences, and all of December and January, which have large differences.
    – Ryan M Mod
    Commented Feb 13 at 20:44
  • Thank you for the explanation! In short, questions that are less than one year old are not reliable because they still may be deleted? Commented Feb 13 at 21:57
  • It depends what you're looking for. If you want statistics on what's being asked, questions less than 1 year old are more reliable than older questions if you're relying on the Posts table (with additional cliffs at 30 and ~10 days). Though if that's what you want, you should use PostsWithDeleted if it's possible for your analysis (it has fewer columns present).
    – Ryan M Mod
    Commented Feb 13 at 22:13
  • I have an index for technology popularity. Commented Feb 19 at 5:56

1 Answer 1

4

The root cause is that questions without answers get deleted after a year. I didn't know that.

When I queried in November 2023, the unanswered October 2022 questions were already deleted. That's why that number only changed by 0.22% between November 2023 and February 2024. But then some unanswered questions from November 2022 (9.9% difference) and all unanswered questions from December 2022 (19.5%) and January 2023 (19%) were deleted before I queried again in February 2024. That's why I saw the differences.

For me, the number of questions is a data point for technology popularity. Whether it gets answered or not doesn't matter then. That's why I'll modify my queries to hit the "all questions, including deleted ones" table called PostsWithDeleted. Below's the query, now going up to January 2024:

DECLARE @StartDate DATE = '2009-01-01';
DECLARE @EndDate DATE = '2024-01-31';

WITH Questions AS (
  SELECT
    Id,
    CreationDate,
    1 AS AQuestion
  FROM PostsWithDeleted
  WHERE
    PostTypeId = 1 AND
    CreationDate >= @StartDate AND
    CreationDate <= @EndDate
),
MonthlyCounts AS (
  SELECT
    DATEADD(month, DATEDIFF(month, 0, CreationDate), 0) AS Month,
    SUM(AQuestion) AS Questions
  FROM Questions
  GROUP BY DATEADD(month, DATEDIFF(month, 0, CreationDate), 0)
)
SELECT
  Month,
  Questions
FROM MonthlyCounts
ORDER BY Month;
2
  • You can, to some extent, know why a post was deleted: meta.stackoverflow.com/a/403095
    – rene
    Commented Feb 14 at 15:41
  • @rene Thank you for pointing this out! I won't need that information for now. Commented Feb 15 at 19:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .