20

I was reading this infographic about Stack Exchange performance and topology and I came across this part where it says that there are "3 TAG ENGINE SERVERS" that are "a unique custom, high performance index of Stack Exchange questions".

Does anyone have any more information on how this is implemented?

3
  • 18
    We have no idea how the damn thing works. Jun 4, 2015 at 15:51
  • 17
    Each week, we sacrifice the blood of a unicorn to the Tag Engine in a dark ritual. In return, it continues to serve our websites. Past that, it's a black box.
    – Haney StaffMod
    Jun 4, 2015 at 16:05
  • 4
    Until Marc Gravell answers this, read this and this and watch this youtube video
    – rene
    Jun 4, 2015 at 17:08

2 Answers 2

21

Basically it is just a custom server that keeps an in-memory set of pre-indexed data (all questions from all sites) in a way that is particularly amenable to most "show me a set of related questions" searches; it also has a query parser for a DSL and a few optimized strategies for applying our most common searches, and a fallback strategy for the rest. Not as flexible as a sql server or lucene etc, but pretty good at doing one thing well. It also obviously has to be able to pull in all changes (add / edit / delete / etc) to all questions in a fairly timely manner, and have some caching and cache-invalidation.

The main other thing it contains is a self-updating core so that we can leave it running as a windows service (think: daemon), and it will update itself with the latest code without ever seeing a disruption in service (the outer layer spins up a second copy of the the new code, gets it all working, switches between the two, then decommissions the old).

2
  • 1
    I have also read your blog posts about it and technical debt. Very helpful. The hot deploy with 2 versions of code sounds very complicated to implement by yourself. Is this also custom implementation or is did you use a server feature like tomcat v7 parallel deployment (tomcat.apache.org/tomcat-7.0-doc/config/…)
    – zpontikas
    Jun 8, 2015 at 9:42
  • 1
    @zpontikas nope; outer service shell deals with network/IO; inner core is hosted app-domain instances in isolated folders Jun 8, 2015 at 10:16
6

If you want to know a bit more see these posts (warning, I don't actually know how it's implemented, these posts are my attempt at finding out!)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .