There should be a downvote or dislike emoji and they should be public.
Someone could be "confused" with your logic, but not be downvoting it. Although i'm sure that's not whats happening here.
There's another site I frequent and they have public downvotes and even a buttfumble emoji. It's pretty funny.