I think a solution would be if you had to type a reason for why you downvoted, but remain anonymous. The author of the submission would be able to see the reason, and can report it if the reason isn't good (something like "this project sucks"), and a moderator can look at it and decide whether or not to remove it. That way trolls can't downvote without reason, and people aren't afraid to give downvotes if they actually have constructive criticism.
