There are multiple reasons why this may not be a good idea. People may be less willing to vote (especially downvote) when they know their vote is visible. We do not want to discourage downvoting as it is a part of this website and the scoring/reputation system that needs to remain healthy. We would also want to avoid situations where users start downvoting someone, just because they have received a downvote from them. Moreover, users that downvoted could be chased on this site (or even in the game) with questions/threats to retract their vote.
We think it is a good practice for users that downvoted to post a comment on what could be improved, but we don't want to enforce anything.