It is just a hazard of participating in a niche tag. There are just not that many people that read the posts, understand your answer and vote. I participate in several such niche tags (I've written answers to [half the questions in the zodb tag) and the voting patterns are anaemic compared to popular tags. That's just how it is.
If all you do is participate in the niche tags, there is a badge to encourage you to keep contributing. The Tenacious and Unsung Hero badges are aimed squarely at supporting those that answer posts with low-frequency tags. If your accepted answers don't get upvotes, at least you get a badge!