The TCP protocol, which most Internet traffic uses, uses a "sliding window" scheme to provide reliability. This means it will only send new contents (slide the window) when it receives the ACK messages indicating that part or whole of the current window has been received completely. So it requires frequent ACK messages to be send in the uplink for your download traffic to go through smoothly If the ACK messages are delayed or lost, the sender may slow down transmission and/or retransmit packets that you have already received.
By default, packets are transmitted by the OS in the order they are generated by the applications, so the ACK packets are interleaved with your upload traffic randomly. Your ISP (or your network device) controls your bandwidth use by delaying packets when your transmission speed exceeds the limit (aka throttling). This delay can happen on any packet including the delay-sensitive ACK packets.
To solve this problem, you'll need some kind of priority system to make sure the ACK packets are not throttled. There are paid software solutions for this. You can also achieve a similar effect if you limit the upload speed to a value below your allocated upload bandwidth. Alternatively, you can use applications that transmit files through UDP which does not require ACK messages.