Yes, yes. Possible technically and proven in a real world (as you read this thread from the beginning).
Fact that ping time has increased gives me a thought that internal antenna is communicating with two towers simultaneously in 2x2 MIMO downstream mode (one data stream from one tower and another from some other one). When using directional antenna you have streamlined your operation to a single tower. Now it can be still working in 2x2 MIMO or it could be even reduced to 2x1 MIMO downstream mode (depends of software configuration). Slower download speeds indicate the later one. However
the average ping time is always quicker when two different towers respond to your ping requests. This is from the mathematical equation.
Download speeds can be cheated by various 2x2 MIMO modes, but upload speeds are clearly indicative of higher radio link quality as your router is unable to negotiate 2x2 MIMO upstream transfer. So I think in the absence of error rate indicator you should be looking for optimising upload speeds when positioning your external antenna.
You have mentioned that with external antenna your connection is more suitable for playing games, but is not neccessary because of faster upload speed. Upload speed is probably still fast enough for gaming requirements, it is a stability of the connection which matters. Stable response time in absence of lost packets.
This is a classic (not obscured) scenario, you can see that a similiar case had been reported on the beginning of this thread and repeated regularly afterwards. Typical explanation was given that a tower you are connecting has a slower capacity. I see you had been already given such explanation.

I am more and more convinced that it is not true and the only correct explanation is given in this post.
Conguested towers could add complication to the above picture in future, but I think we don't need to take into consideration as speak.
If you want to take advantage connecting to multiple towers simultaneously, dual omnidirectional antennas would do a trick. Unless you have a reason for dual cross-polarized highly directional antennas for your LTE connection, I think this type of antenna should be considered in the first place.
My opinion, but make it sticky.