Latency does increase but mildly. It goes up by 1-4ms.You know what really surprised me about the whole Octotel issue is that clearly it's congestion somewhere by the speeds you are getting but no increased latency? Why. Congestion when it gets bad latency suffers but not with you guys on Octotel. Very strange.
Honestly though I have no idea whats wrong, @TheRoDent says its packet loss on my line which goes up to ~3% in the evenings. At first I didn't think 3% packet loss could cripple a 100mb line with the speeds I've been seeing but then he showed some proof with a tcp throughput calculator.
Try it yourself. 100mb line, RTT 160, packet loss 3%, MTU 1480 (if I remember correctly) and you'll end up with an expected throughput of 0.72mb which is more or less what I get.