Gravity does not prevent droplets coming out of your mouth from staying airborne. A mask does this by blocking the path with a material that the droplets stick to.
The very reason that "quickly the inside of the mask becomes an uncomfortably hot, wet environment" is because it is filtering out the moisture (droplets) in your breath. If there was no difference then that wouldn't happen.
Obviously the looser the fit, the less effective. The whole purpose of the mask is to filter your breath. Though even if you wear it loosely it will at least direct the vapour away from the person you are talking to.
Laser scans showed that about 2,600 small droplets are produced per second while talking and can float in the air for 8 to 14 minutes. Even a loose mask will act as a shield and prevent much of those reaching the person you are talking to.
I really don't see how you would come to the conclusion that the difference between a mask and no mask "is tiny".
Without a mask your breath is ejected at force and will continue to travel in that direction until air resistance brings it to a halt. That distance can be several meters (or a lot more if there is wind).
A mask will stop that almost immediately and the remaining directional force will be scattered, resulting in a very short travel distance.
So a mask will filter out a large amount of particles and stop the directional force of the remaining ones. That's a huge difference.