Could be the cell tower, phone drops down to 2g for voice? Thereafter it's Apple internal algorithms trying to ensure a connection on the most stable link (time based and also considering 2g stable during your call) ; basically to avoid the constant forward and backward switching.
So option is to possibly log a bug / feature request.