They literally give it a small knock with one or two fingers as they are passing by, as if using the table as a pivot point.
Am I being over sensitive ? How do you prevent people from doing that ?
Yes, you are probably being overly sensitive.
But if you want to try to prevent it, put a floor planplant, lamp, or additional chair at the end of the table so folks have to swing wider and won't be able to knock.