It might be more effective to penalize instances with open sign ups that the bots are using, to pressure such instances to use bot-filtering tactics on their sign up pages.

Rate limiting, blocking DMs, or defederating (perhaps graduated after a warning) from minor instances that can’t control bots sounds like a reasonable admin action.

I think the most effective would be to add a step so you could only write a message to people when both parties have consented or something like that.