We have both automated systems in place that detect vote fraud based on certain heuristics, and moderators investigate specific instances that are brought to our attention by user flags. Also, once you've been a moderator long enough, you just start to be able to smell sockpuppetry (dirty socks—you know the smell), and then you go looking for it.
As far as what constitutes sockpuppetry, the definitive explanation can be found here on Meta Stack Exchange (the global Meta site for the entire Stack Exchange network). It can be summed up as:
…if the second account allows you to do something on the site that your normal account would be prevented from doing, it is abuse.
Or, an even simpler rule to follow is, if your second account is interacting in any way with your first account, then it's a sockpuppet and breaking the rules.
How we actually prove that an account is a sockpuppet, well, that's a trade secret. We intentionally do not reveal details of these investigations or even the overall process, lest it give users ammunition to evade detection.
But, I can tell you how we handle it. As that answer I linked above indicates, when a sockpuppet account is found, it is deleted, removing it from the system. This removes all traces of its votes. A moderator also warns the operator of the sockpuppet via a private message, which may be accompanied by a suspension at the discretion of the moderator. A second offense always comes with a suspension. The duration of the suspension increases with repeated offenses.
Taking voting fraud seriously is in everyone's interest. It means you can trust the scores of posts on the site, and it means that other people aren't getting an unfair advantage.