The problem comes in not that you are holding the phone, it is that you are trying to concentrate one two tasks simultaneously.
Most people are too bloody retarded to just drive! Now they must focus on receiving information via the phone, processing it and formulating a reply, all the while constantly receiving information from the environment around them while driving, processing that and responding accordingly while driving.
Simple solution to a simple problem. You should not be allowed to use a phone/hands free in a car, period.