It appears to be your line, but unfortunately I cannot venture to say that solidly without any proof.
Dealing with performance issues, and packet loss on a line with any of the FNO's is an almighty task. It's actually simpler if a line is simply down.
We use TCP accelerators to try and mitigate some of the effects of lossy Fibre Provider's but it's just a crutch.
Getting another line installed in some cases is actually a good solution, new fibre pair, new ONT or new splitter possibly, but an ISP should be able to get an FNO to do at least basic optic tests and throughput on a diagnostics VLAN, or a CPE swapout before having to resort to an additional installation. Frogfoot does have metro-ethernet certified testing kit that they will install at the ISP and at your premises under *heavy* duress.
If however, your entire area's backhaul is shoddy, this takes months and months of arguments with the FNO to resolve, especially if the problem only occurs during peak times.