ActivityPub votes are public by design, Lemmy just hides the voters from users, while Mbin displays them. Anyone can also spin up their own instance and get access to the data.
PieFed recently added a feature to pseudonymize votes. When enabled, your votes are labeled with the name of a shadow account linked to yours, but only the administrators of your instance know that it belongs to you.
No defense, then? Lemmy is just intended as a delivery vector for shitty politics, shackled to the rotting corpse of the USSR?