Your previous test was done at night when residential areas have their peak traffic. I had the same problem where my latencies were sky high between 7pm and 12pm. During the day everything was fine though.
Telkom moved me to a recently upgraded exchange and that solved everything. I suggest you contact them to do the same for you.
You may well be right, but then why did switching to another ISP account not suffer the same fate?