Response times to the modem don't necessarily mean that the modem or router is functioning perfectly on all protocols. Nor does it mean it is routing efficiently on all protocols. It just means that ICMP is routing fine. TCP and UDP traffic operate a little differently, so always best to test thoroughly when this sort of thing happens. Pinging bras is fine but it doesn't show the step before, and in addition an ICMP response time may be different to TCP packets or UDP packets traversing the same route (majority of the data you consume is TCP and UDP protocol). So I'd suggest, in the efforts of going the extra mile here to rule out all other possibilities, to test with another modem with the splitter off the line, and see if you can replicate. If you can, it sounds to me like rather odd exchange congestion or a line fault. If the results are not replicated, you can very easily identify the modem/router as the culprit...
Good advice here. Thanks, DJ