Well, we do actually get to decide. That's what society does -- decides who can do what to whom, under what circumstances and to what degree, as well as what the consequences will be for breaching those perimeters.. That's part of why discussion around this issue has been so rich.

American society currently allows for the use of physical violence with various levels of impunity in a number of circumstances. But if you are a staunch pacifist and it is your view that violence is never acceptable, full stop, that's fine. You just may not get far expecting everyone else to agree with you, or accusing people of being hypocrites for taking nuance and circumstance into consideration, just because you don't.