I can think of two possible reasons, but I can't personally confirm either.
After you downvoted the OP's two answers and (I assume) the question, the serial voting detection mechanism detected that as serial voting, and reversed the votes. Not saying the votes were unwarranted by any means, just that the mechanism recognized a pattern and took action.
Seems like it'd take more votes than that (did you maybe dv his only other question too?), but then I have no idea what the threshold or various factors are that play into the automated reversal mechanism.
Another (highly unlikely) reason might be that an employee got involved. I only mention it because Anna Lear had made mention of it awhile back, in a comment under this answerthis answer:
Moderators can't reverse anyone's votes. We (employees) can, but we don't make a habit of it and it's not exactly an easy button beside every post.
However, there's nothing in this instance that suggests they stepped in. The more likely scenario of the two (assuming it's not a bug) would be the serial-voting-reversal mechanism.