Hahaha. Exactly. It's making me ****ing nuts. Luckily you've found the cause. I've ruled out router, and PC, and they've ruled out exchange. So what now? The only thing I can think of, is the physical line between me and the exchange. But apparently my line stats are perfect. So I dunno any more man. I've tested every scenario I can think of.
2nd time in 4 years (different homes each time, same exchange) that it was diagnosed by Telkom as congestion, and it turns out not to be. First time was actually how I got to know Charl, via Twitter. He was again the one that identified the real problem. That time, however, it looked and smelled even more like congestion, as I had ridiculous ping - yet it wasn't that, and within days of him identifying it, it was fixed.
I wish I can tell Telkom to **** off, but I don't have that luxury. I tried Level 7, but I cannot afford their installation costs. Besides, chances are they're likely running through Telkom anyway.