...would you like to be fired from a management job because you are rumoured to have an extra-marital affair?
How does that impair your decision making?
It can and does happen. More to the point though, public figures have chosen that life and hence must live with the consequences of such. Cases like this show how public figures want all the perks that come with the lifestyle they chose but none of the drawbacks. That really is too bad for them.
If a person has shown poor judgment in their personal life, then it is fair to expect that they will show equally poor judgment in their professional life.