Its called being guilty by association. You don't have to actually do anything wrong. People like to say/pretend you are innocent until proven guilty but just look at anyone who is ever accused of rape, molesting or steroid use. You are considered guilty even after you have been found innocent.
There are a lot of people who will bow out understandably