8

It looks like search results have not been showing recent posts since sometime this morning. For example, if I do a search for "OpenGL", the newest post that shows up is about 8 hours old:

https://stackoverflow.com/search?tab=newest&q=opengl

If I search for one of the related tags instead, there are plenty of newer hits:

https://stackoverflow.com/questions/tagged/opengl

Are others seeing this as well? Is it a known issue? I looked through the recent topics here as well as on MSE, and did not see it mentioned.

4
  • We had this earlier meta.stackexchange.com/questions/233740/… but Anna says it is solved now...
    – rene
    Jun 24, 2014 at 21:07
  • Yes, there were search issues before, definitely after one of the previous site maintenance events. But this one only started today, as far as I can tell. Jun 24, 2014 at 21:11
  • Do notice that there was an outage yesterday. This might be related to that.
    – rene
    Jun 24, 2014 at 21:27
  • 3
    Well, crap. This is my fault, a full re-index is working right now. I'll write a more detailed post when the reindex is complete. Jun 24, 2014 at 21:32

1 Answer 1

12

TL;DR: Fixed, enjoy!

The way we index the posts table is pretty simple. We use a field on Posts called LastChange which is a rowversion column, basically an always incrementing number that updates when anything on the row changes, even though it presents as a TIMESTAMP.

We use that rowversion by simply casting that column to a bigint and remembering the last thing we indexed as we go. At the end of the job we record that value in Elasticsearch. The next time we index (30 second interval in the Stack Overflow case), we grab the value and get all posts with LastChange > PrevValue (plus the parent or children, for title and tag changes, etc.).

Normally, that works great.

However, this morning that strategy conflicted with my need to update 16 million rows on the Posts table in preparation of making IsAnswered a non-nullable BIT (currently nullable). After the updates, it resulted in a slightly larger than normal indexing pass. It was simply creating a timeout on the batch due to the wait on the SQL side, a delta of that size is simply brutal.

We handle delta indexes differently than full indexes (which are just a "select everything"), and that's what was needed here to restore some sanity. Luckily that's an online operation and only takes a few hours (we throttle it) - it was completed a few hours ago.

Note: this was written around 10:00 PM EST, but evidently I forgot to hit the post button before bed, oops.

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .