Is there some advantage to having these points that I'm not aware of?
Sure. The more reputation you have, the more you can participate in and moderate the site.
Reputation is directly tied in to the privilegesprivileges one can gain on the site.
I feel I must be missing something here
Given that downvotes indicate something wrong/bad in the post, yes, it makes sense for people to worry about them. They are a strong signal that something isn't right. They are tied to negative reputation so people actually pay attention.
Now, if you are asking why do people care about imaginary Internet points, I can't help with that. People care about imaginary points everywhere - playing games and more.