There'd be no path short of moderator intervention for getting rid of popular distractionspopular distractions.
With downvotes now the primary means of moderating questions, the number of "moderators" would increase by more than 10x. With downvotes now the only means of marking problematic questions, their use - along with flags - would likely see a marked increase, along with "lord of the flies" comparisons such as our dear "love-hate" author's. Note that Stack Overflow current sees more close votes + flags on questions than it does downvotes on questions and answers combined.
We - that is, the community here on meta, the elected moderators, and the folks like you and me working for Stack Exchange - would lose a powerful tool for influencing moderation: predefined close reasons.
Along the same lines, there'd no doubt be considerably more pressure to require downvoters to specify a reason for downvoting, along with calls for meta-moderation features that could invalidate downvotes that were seen as groundless by some segment of the population.
Is this still a collaborative effort, a great leap of faith predicated on trusting your fellow programmersa great leap of faith predicated on trusting your fellow programmers? Is Stack Overflow you? Or are you just canon fodder, a digital sharecropper to be milked for all you're worth and then cast aside.