Happened to me about 2.5 month ago, luckily got a very clever techy here and he sorted everything out in 40 minutes, got me moved to a less congested DSLAM, been fine since, "touch wood"
If I restart my router, which I must stress is a few days old, when 'congestion' is very bad (over 500ms first hop), the first hop immediately goes to 9 to 20ms. What on earth is this madness?