As a first measure I think we could make it so that once you removed yourself as a collaborator from a project, it would not be possible for the person to re-add you. I realize that's not ideal but would be much more trivial to implement than UI / additional server state for confirmation before being added.
That'd be a good moderate solution. It'd keep the simplicity of your collaborator UI intact, but let me avoid this kind of abuse. You might want to make it a "remove" and a "remove and block" so that people don't get into a bad state on accident.
That seems a bit stateful. For every project you'd have to remember the set of people who've removed themselves. What if a person later removes themselves and then changes their minds?
Maybe a simpler solution is: you can't add people, you can just send them an invite. The person can then choose to accept the invite or not.
