Cool Ideas not so Cool anymore

[TheMightyQuin]

Well that's just the shitty feeling I have now of "Which relatively crappy ISP to I go to next?"

Sounds like the best for you would be to form your own ISP and get a direct pipe to Blizzard.

Then you don't have to contend with all these real life issues.

You seem to be one of those people that know just enough about something, to be dangerous and excruciatingly irritating.

Paul's detailed explanations of what was done to alleviate your personal connectivity requirements, went way above and beyond the industry standard...

 

[EvilSurd]

Looking at the ticket this was due to stale routes on Blizzards network that was resolved? This was after the COD update.

The initial delay in response regarding your issue, I totally agree with and the respective staff member has been spoken to so apologies for that.

This is why we make ourselves available for escalation should a case fall through the cracks.

Alex apologized and that was accepted and I tried to give him his due(No, not on Wifi, yes updated firmware, yes, screenshots of tests plugged directly into CPE, AV off, yada yada) to get through the initial due diligence and trust he would realise it needed to be booted up the line to be properly looked at.
Things fall through cracks and I gave leeway not knowing staffing %'s with Covid etc etc and that was a very small agitation overall, especially as Alex was super polite

Then being asked by the next guy if I actually know which region I play a game I have played for 15 years on??
Followed by:
That bieng said the fibre line tests to vumatel you have done are good, so we know the fibre is not causing this.

What devices are you gaming on ?
How are these devices connecting to your Router ?
Has the firmware of the router been updated ?
Is this happening with other online games ?
Is this happening with your internet connection in general when doing other activities other than gaming?
Is this happening at specific times of day?

Sure, the call has been open a week, you have the complete ticket history and this is the approach.

 

[EvilSurd]

You seem to be one of those people that know just enough about something, to be dangerous and excruciatingly irritating.

Exactly that guy

Paul's detailed explanations of what was done to alleviate your personal connectivity requirements, went way above and beyond the industry standard...

Alleviate my requirements?
Something changed, no one knows what or why nor can they put it back

 

[PBCool]

Exactly that guy


Alleviate my requirements?
Something changed, no one knows what or why nor can they put it back

I just explained why and what and we can go back but it's an experimental thing, you can't easily policy route an IP range to test. We are happy to test directly with you in turning up the peer again for a short period and happy to deal directly with you in doing so.

 

[EvilSurd]

It is impossible to policy route different ranges effectively to test this kind of thing.

I hear you, and it's unmanageable and can affect performance and impact an otherwise 5 9's network that is servicing all the other customers adequately.
Look at the struggle to try get to someone to have a conversation of what CAN realistically be done to eliminate the issue and return to what was before. If you feel you're having a considered informed conversation with someone trying to resolve the problem, there is far less pain and resentment and you end up far more eager to put the effort in and stay a dedicated customer.

The changes Paul ultimately made, made my connection usable again. It didn't return it to what it was before and no root cause on that.

I get that my situation is weird and the manhours required to track this down aren't worth it, especially as it's extremely unlikely the resolution could even be used on other customers going forward to add value.

It sucked is all I am essentially saying, and the reason I felt so bleak was coming off of the previous years service and support always being high and being a proponent of CISP to others.

 

[s0lar]

Gets fragged, blames lag. Players laugh, posts a mean “hellopeter” thread on myBB. Repeat. So glad I am not an engineer in the ISP space anymore.

 

[PBCool]

It is impossible to policy route different ranges effectively to test this kind of thing.

I hear you, and it's unmanageable and can affect performance and impact an otherwise 5 9's network that is servicing all the other customers adequately.
Look at the struggle to try get to someone to have a conversation of what CAN realistically be done to eliminate the issue and return to what was before. If you feel you're having a considered informed conversation with someone trying to resolve the problem, there is far less pain and resentment and you end up far more eager to put the effort in and stay a dedicated customer.

The changes Paul ultimately made, made my connection usable again. It didn't return it to what it was before and no root cause on that.

I get that my situation is weird and the manhours required to track this down aren't worth it, especially as it's extremely unlikely the resolution could even be used on other customers going forward to add value.

It sucked is all I am essentially saying, and the reason I felt so bleak was coming off of the previous years service and support always being high and being a proponent of CISP to others.

Hence why we have this thread as a gateway to do exactly that.

 

[EvilSurd]

Original route 230ms, the way life used to be, then became the issue that we wanted to resolve.
MTR:
Start: Mon Jun 8 14:22:18 2020 Blizzard 1.|-- Blizzard 0.0% 10 0.3 0.4 0.3 0.8 0.0
2.|-- 37.244.23.131 0.0% 10 0.5 0.7 0.5 1.5 0.0
3.|-- 24.105.62.152 0.0% 10 1.9 1.3 0.9 2.8 0.3
4.|-- 137.221.66.10 0.0% 10 2.2 2.1 2.0 2.3 0.0
5.|-- 137.221.69.52 0.0% 10 22.5 32.1 22.4 69.7 16.9
6.|-- 137.221.65.9 0.0% 10 22.4 22.5 22.3 22.9 0.0
7.|-- 137.221.71.32 0.0% 10 22.5 23.2 22.4 25.9 1.1
8.|-- nyiixv4_1.cisp.co.za 0.0% 10 211.6 211.7 211.6 211.9 0.0
9.|-- uwx-cust.coolideas.co.za 10.0% 10 212.1 212.5 212.1 213.9 0.5
10.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
11.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
12.|-- u13a-cust.coolideas.co.za 0.0% 10 229.0 227.9 227.6 229.0 0.3
13.|-- uzq-cust.coolideas.co.za 40.0% 10 229.9 230.4 229.3 233.1 1.2
14.|-- uo3-cust.coolideas.co.za 0.0% 10 230.7 231.0 230.6 232.0 0.0
15.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0

Workaround route 270ms via London
Start: Wed Jun 10 19:30:06 2020 Blizzard 1.|-- Blizzard 0.0% 10 0.3 0.3 0.3 0.5 0.0
2.|-- 37.244.23.131 0.0% 10 0.5 0.7 0.5 1.1 0.0
3.|-- 24.105.62.152 0.0% 10 1.0 1.4 0.9 2.6 0.5
4.|-- 137.221.66.10 0.0% 10 2.2 8.2 2.1 62.3 19.0
5.|-- 137.221.69.48 0.0% 10 2.1 9.8 2.1 20.8 5.8
6.|-- 137.221.69.32 0.0% 10 2.6 8.6 2.6 43.4 13.0
7.|-- 7-1-4.ear7.Chicago2.Level3.net 0.0% 10 2.1 2.3 2.1 2.8 0.0
8.|-- ae-1-3501.edge4.Chicago3.Level3.net 0.0% 10 2.0 2.1 2.0 2.7 0.0
9.|-- IPTRIPLEPLA.edge4.Chicago3.Level3.net 0.0% 10 2.5 2.8 2.4 4.8 0.7
10.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
11.|-- u10e-cust.coolideas.co.za 0.0% 10 92.0 91.9 91.7 92.6 0.0
12.|-- cisp.demarc.cogentco.com 0.0% 10 88.6 88.7 88.6 89.1 0.0
13.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
14.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
15.|-- u13a-cust.coolideas.co.za 0.0% 10 249.3 252.1 249.3 254.4 2.3
16.|-- uzq-cust.coolideas.co.za 0.0% 10 263.9 263.0 255.1 270.5 5.4
17.|-- uo3-cust.coolideas.co.za 0.0% 10 257.7 257.8 255.4 266.8 3.3
18.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0

I'm not asking for special routing to improve only my experience with increased overheads on your end.
Asking what changed, can we fix it and go back, or is this the new world

 

[PBCool]

Original route 230ms, the way life used to be, then became the issue that we wanted to resolve.
MTR:
Start: Mon Jun 8 14:22:18 2020 Blizzard 1.|-- Blizzard 0.0% 10 0.3 0.4 0.3 0.8 0.0
2.|-- 37.244.23.131 0.0% 10 0.5 0.7 0.5 1.5 0.0
3.|-- 24.105.62.152 0.0% 10 1.9 1.3 0.9 2.8 0.3
4.|-- 137.221.66.10 0.0% 10 2.2 2.1 2.0 2.3 0.0
5.|-- 137.221.69.52 0.0% 10 22.5 32.1 22.4 69.7 16.9
6.|-- 137.221.65.9 0.0% 10 22.4 22.5 22.3 22.9 0.0
7.|-- 137.221.71.32 0.0% 10 22.5 23.2 22.4 25.9 1.1
8.|-- nyiixv4_1.cisp.co.za 0.0% 10 211.6 211.7 211.6 211.9 0.0
9.|-- uwx-cust.coolideas.co.za 10.0% 10 212.1 212.5 212.1 213.9 0.5
10.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
11.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
12.|-- u13a-cust.coolideas.co.za 0.0% 10 229.0 227.9 227.6 229.0 0.3
13.|-- uzq-cust.coolideas.co.za 40.0% 10 229.9 230.4 229.3 233.1 1.2
14.|-- uo3-cust.coolideas.co.za 0.0% 10 230.7 231.0 230.6 232.0 0.0
15.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0

Workaround route 270ms via London
Start: Wed Jun 10 19:30:06 2020 Blizzard 1.|-- Blizzard 0.0% 10 0.3 0.3 0.3 0.5 0.0
2.|-- 37.244.23.131 0.0% 10 0.5 0.7 0.5 1.1 0.0
3.|-- 24.105.62.152 0.0% 10 1.0 1.4 0.9 2.6 0.5
4.|-- 137.221.66.10 0.0% 10 2.2 8.2 2.1 62.3 19.0
5.|-- 137.221.69.48 0.0% 10 2.1 9.8 2.1 20.8 5.8
6.|-- 137.221.69.32 0.0% 10 2.6 8.6 2.6 43.4 13.0
7.|-- 7-1-4.ear7.Chicago2.Level3.net 0.0% 10 2.1 2.3 2.1 2.8 0.0
8.|-- ae-1-3501.edge4.Chicago3.Level3.net 0.0% 10 2.0 2.1 2.0 2.7 0.0
9.|-- IPTRIPLEPLA.edge4.Chicago3.Level3.net 0.0% 10 2.5 2.8 2.4 4.8 0.7
10.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
11.|-- u10e-cust.coolideas.co.za 0.0% 10 92.0 91.9 91.7 92.6 0.0
12.|-- cisp.demarc.cogentco.com 0.0% 10 88.6 88.7 88.6 89.1 0.0
13.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
14.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
15.|-- u13a-cust.coolideas.co.za 0.0% 10 249.3 252.1 249.3 254.4 2.3
16.|-- uzq-cust.coolideas.co.za 0.0% 10 263.9 263.0 255.1 270.5 5.4
17.|-- uo3-cust.coolideas.co.za 0.0% 10 257.7 257.8 255.4 266.8 3.3
18.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0

I'm not asking for special routing to improve only my experience with increased overheads on your end.
Asking what changed, can we fix it and go back, or is this the new world

I mentioned what changed, we peer with Blizzard at NYIIX via the SACS and Monet cable systems. Hence the increased latencies.

When the COD update happened we had some strange loss and inconsistent latencies. Hence we dropped that peer.

So we are happy to test with you again but to limit this to a specific block is basically impossible, so we would need to do it in short bursts.

Then once we do and shows no packet loss but you still get rubber banding in game makes it very difficult to diagnose where the issue may be.

 

[EvilSurd]

When was the COD update?

 

[Johnatan56]

Your issue was packet loss which is what we established with you if you recall

But on a pure technical level throughput anywhere can't be guaranteed in a WAN environment where you have no control over external factors. Hence why when talking about bandwidth availability it is done on a contention level, not on a throughput level.

Yes, and we both already knew that there was packet loss between ONT and your network, with me stating that it was packet loss causing it, yet it took months with nothing happening. Not going to get into it now.

International not guaranteed doesn't mean 2% of line speed, you'd have flipped out at such service as well.

 

[PBCool]

Yes, and we both already knew that there was packet loss between ONT and your network, with me stating that it was packet loss causing it, yet it took months with nothing happening. Not going to get into it now.

International not guaranteed doesn't mean 2% of line speed, you'd have flipped out at such service as well.

Same discussion as before we went through the process to resolve with the FNO, they upgraded their backhaul and everything was resolved.

 

[EvilSurd]

Sounds like the best for you would be to form your own ISP and get a direct pipe to Blizzard.

It's almost like this guy knows the origin story of the likes of AH, Openweb, CISP.
Just because the trolls like to be fed.


Good work on the dirty dangle upvotes boys, way to snipe mad emoji celly boys, boost ranks ferda boys boys.
Forget where I mentioned frags or blaming lag

 

[CrypticZA]

When was the COD update?

I believe he is referring to the one that broke Blizzards network it was like 30GB like a week ago. Whenever COD pushes a patch its huge and Blizzards network strains, people come here looking for help because there is packet loss going via SACS or this or that. I can remember this issue because in Apex i was getting packet loss to the Brazil server too.

Issues happen and things need to be changed for periods of time to alleviate a problem, not long ago Blizzard transit was suffering so Paul changed the routing so Telia carried us to the server, it was an increase of about 20ms to the one server but hey no packet loss, after a while we did some investigation and Blizzard transit issues were fixed, and the peer was brought up.

"nyiixv4_1.cisp.co.za 0.0% 10 211.6 211.7 211.6 211.9" would have shown it went via SACS to the New York exchange, you could probably VPN to New York to force the route from going to London, 210 to New York with NordVPN.

 

[EvilSurd]

Thanks, good info.
So to keep piling onto the ishtpile of facts/evidence, this started first week of April.
I didn't log a ticket then as we had 2x cable breaks, double speed upgrades, and Covid forcing everyone to do everything remotely. Throughput was never an issues, and with the majority base wanting fat pipes for streaming and other content, I left it, as it felt a small issue in a time of lots of changes. Played a bunch of SP games and left it alone.
But that also means the COD and changes made attributed to this don't gel with me, further clouding the issue.

Have stayed away from the VPN testing side thus far, but it's a valid approach. If I find a needs specific VPN that gets me down to base pipe latency, it's further evidence to an issue of sorts with the way we route. If I'm told that that's just the way we route now by default, then that is what it is. It just takes us back a few steps to all comparing notes in community to find the best combination of ISP, Unshaped Option over and above, Best VPN to attach.
My service was great before by default, hate to want to admit taking that step back

 

[CrypticZA]

Thanks, good info.
So to keep piling onto the ishtpile of facts/evidence, this started first week of April.
I didn't log a ticket then as we had 2x cable breaks, double speed upgrades, and Covid forcing everyone to do everything remotely. Throughput was never an issues, and with the majority base wanting fat pipes for streaming and other content, I left it, as it felt a small issue in a time of lots of changes. Played a bunch of SP games and left it alone.
But that also means the COD and changes made attributed to this don't gel with me, further clouding the issue.

Have stayed away from the VPN testing side thus far, but it's a valid approach. If I find a needs specific VPN that gets me down to base pipe latency, it's further evidence to an issue of sorts with the way we route. If I'm told that that's just the way we route now by default, then that is what it is. It just takes us back a few steps to all comparing notes in community to find the best combination of ISP, Unshaped Option over and above, Best VPN to attach.
My service was great before by default, hate to want to admit taking that step back

There has been regular CoD patches and Blizzard themselves have said at times people will experience slow download speeds as well as packet loss - you can find this information on their forums. Cod receives regular patches that are huge so my comment above was regarding the latest huge patch.

As for routing VPN's are pretty good, i wanted to get better pings to Asia and am getting 220ms to South Korean OW servers via VPN now.

Speak to Paul (PBCool) he has done numerous favours for me which i am very grateful for. Sent him a few IP's to try and get a connection to Apex Brazil servers and he made it happen within 24 hours. Myself and the community have approached him and Rodent with numerous issues and they all have been resolved and they always have gone out their way to try accommodate us to the best of their abilities and network abilities.

He has mentioned some testing needing to be done, give it a go and see how things go..

 

[EvilSurd]

In my head though, the story runs along the lines of:

My traffic used to go this way, life great.
My traffic goes same way, life not great.
Arguments follow, actually traffic can go this other way now, life better, but not great.
Question: Was the way my traffic went before the default for all customers, or special? (This answer involves a fair bit potentially based on potential basic infrastructure up to dynamic provisioning of service delivery to customers spinning up additional routes and bandwidth spread across multiple peer providers and the associated higher costs, but ensuring a consistent customer experience) It may have been completely unnoticed by me, which would be a great tech solution, as I experienced the same rock solid 230ms for years. It may just have been that good, but I feel that the default pipe I was going through just worked.
Paul gets onto the ticket, my traffic now goes through this pipe, stable but with a higher ms.
Question: Is this the way my traffic was mean't to go before by default for all customers, or special? Not special I feel, as worse performance.
The current targeted reasons for all this are fairly recent, so I can't commit to them as being a COD update and no other explanations like a conscious change of peering providers, contention etc.

When the COD update happened we had some strange loss and inconsistent latencies. Hence we dropped that peer.
Paul still managed to flip me back to that route very recently that I had the issues on, so don't feel that it is completely dropped.
Do I request to go back to that quicker imo base route and add a super VPN on top, or just deal with what I have now and no real answers, or pull the trigger on the next potential ISP and come back with an apology or an AHA, look how good it is now feedback?

 

[EvilSurd]

Speak to Paul (PBCool) he has done numerous favours for me which i am very grateful for.

He has mentioned some testing needing to be done, give it a go and see how things go..

I admit to getting very old man Karengrumpy on this journey and not wanting to always have to 'know the right guy'.
It can't be maintained and it's big strain on their end.
He has also come into this whole situation very late and been the best part of it. I woulda told me to f'off and arranged maintenance on my line yesterday already

 

[CrypticZA]

In my head though, the story runs along the lines of:

My traffic used to go this way, life great.
My traffic goes same way, life not great.
Arguments follow, actually traffic can go this other way now, life better, but not great.
Question: Was the way my traffic went before the default for all customers, or special? (This answer involves a fair bit potentially based on potential basic infrastructure up to dynamic provisioning of service delivery to customers spinning up additional routes and bandwidth spread across multiple peer providers and the associated higher costs, but ensuring a consistent customer experience) It may have been completely unnoticed by me, which would be a great tech solution, as I experienced the same rock solid 230ms for years. It may just have been that good, but I feel that the default pipe I was going through just worked.
Paul gets onto the ticket, my traffic now goes through this pipe, stable but with a higher ms.
Question: Is this the way my traffic was mean't to go before by default for all customers, or special? Not special I feel, as worse performance.
The current targeted reasons for all this are fairly recent, so I can't commit to them as being a COD update and no other explanations like a conscious change of peering providers, contention etc.

When the COD update happened we had some strange loss and inconsistent latencies. Hence we dropped that peer.
Paul still managed to flip me back to that route very recently that I had the issues on, so don't feel that it is completely dropped.
Do I request to go back to that quicker imo base route and add a super VPN on top, or just deal with what I have now and no real answers, or pull the trigger on the next potential ISP and come back with an apology or an AHA, look how good it is now feedback?

When they drop a peer it means they de-peer, its still there they just not choosing to hand their traffic off at that point and to take it on a more stable route so currently its going via London instead of via SACS.

The Americas	24.105.30.129 (US West) 24.105.62.129 (US Central)

These are the two test IP's for Blizzard America, both are going via London, once you do some testing with Paul, he will be able to see whether it is feasible and stable to run traffic via SACS to USA.

And an FYI you get such an amazing ping to USA because of the SACS cable and capacity they have on it, only a handful of ISP's have this so its an added benefit, most other ISP's would be happy with what you getting now.

 

[EvilSurd]

24.105.62.129 is the one I initially grabbed from the Blizz support forums and stuck with to try keep some consistency in testing. A large challenge was finding a friend with a similar setup(Vuma/CISP/NorthJHB/IPPool) to compare and present tests. The closest I got was a mate using MTR to that IP.
I found one, his IP assigned in the 102.132.139.x pool, mine has remained the same in the 102.132.136.x pool. Vuma/CISP/5km's of each other. The below two MTR's done within an hour of each other, and frustratingly weird and different indicators.