Web Squad ISP

Seeyou

Expert Member
Joined
May 1, 2007
Messages
2,358
Its Apex Servers, i have tried CISP, Cloudflare and Exitlag and been suffering with all of them. Friend on Mweb was suffering as well and my other friend on WebAfrica was also struggling, we all in different areas, different FNO's.

During the day servers are fine but at night they been going to hell... I have been knocked so many times sliding behind cover, my friend died behind a corner last night, we just closed the game because it was so ****.

People are ddosing servers again, especially ranked, seen multiple streamers have their games ddosed again...

Ah, that explains it. I tried VPNGame on Singapore, so perhaps they are less affected. It's been shockingly bad. The number of times I've been knocked because of "front-loaded" damage caused by these delays is infuriating.
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,456

This looks like an issue on unitas network (we hand over locally at hop 5 (that’s a peering IP at NAP - ixp.jobug)) and seems to be on their transit leg between JHB and London (perhaps a primary route failure). They will need to withdraw advertisements in JHB for our transit to take over this traffic (similar to what you see Cloudflare and other more established networks do when they have an issue).
 

Seeyou

Expert Member
Joined
May 1, 2007
Messages
2,358
We are working on this with the affected upstream- their DDOS mitigation tool flags the high packet volumes as DOS traffic, and kicks in.. we had a good middle ground that worked well until the accelerator kicked in and messed with the packet ratios (the accelerator helps boost packet volumes many times over.. unfortunately the DOS tool being used (and this protection remains critical for our network) isn’t able to differentiate at the disproportionately high packet levels we see with this kind of traffic. An additional upstream is being added to the mix in the next two weeks- which will help balance this out. Additional tools are also being tested and more feedback to follow.

Wondering if this is still on track @websquadza ?
 

Seeyou

Expert Member
Joined
May 1, 2007
Messages
2,358
Cross connect is ordered for the new upstream. Waiting for installation, patching and configs... I'd say latest end of next week.

How's this going to work btw? Are you going to route certain traffic types over the new upstream? (which presumably doesn't have the same DDOS triggers)
 

tehan

Active Member
Joined
Jul 5, 2017
Messages
72
@websquadza Any updates for me from Vumatel regarding the OLT? Last I heard you would be seeing the area manager again, and give me feedback, but have heard nothing since...
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,456
@websquadza Any updates for me from Vumatel regarding the OLT? Last I heard you would be seeing the area manager again, and give me feedback, but have heard nothing since...
Vumatel are still “investigating”. Personally I think they’re waiting for DFA to upgrade the backhaul to the OLT feeding your estate. You’re in a unique situation because of the legacy SADV hub-spoke network design that deployed OLTs on the back of individual backhauls, and 1G of backhaul was probably all that was ever provisioned. We’ll obviously never get an admission to this and we have asked directly if the OLT has more than 1G feeding it- no one could answer. This explains why you get a peak of 800 at off peak times and 500 during peak hours.. Our team has been out twice now to meet with the guys, with very little momentum. We will continue to follow up.
 

HowTo

Expert Member
Joined
Aug 7, 2006
Messages
3,209
@websquadza
I'm back my ticket info [Ticket ID: 613525] IPv6 settings info.
Received mail on the 9th, did reply today.

Tnx in advance.
 

PsyWulf

Honorary Master
Joined
Nov 22, 2006
Messages
12,433
Just to double check the Apex servers are a lagfest for everyone today? Otherwise something is HECTICALLY wrong with my line.
Depends on the region,but new patch is proving quite the potatofest
 

Seeyou

Expert Member
Joined
May 1, 2007
Messages
2,358
Depends on the region,but new patch is proving quite the potatofest

I've tried EU and Singapore. Every game is a slow-mo action fight until about 10 squads die, but still laggy. That's if I didn't get a Code:Net or Leaf trying to get into a game.
 

Seeyou

Expert Member
Joined
May 1, 2007
Messages
2,358
I've tried EU and Singapore. Every game is a slow-mo action fight until about 10 squads die, but still laggy. That's if I didn't get a Code:Net or Leaf trying to get into a game.
Shiv is currently lagging to crap on stream quite hilariously, so that answers that.
 

PsyWulf

Honorary Master
Joined
Nov 22, 2006
Messages
12,433
Right now the general process is join game,1-2 people time out,wait for everyone to get ingame,proceed to rotate,pray for nobody else to kill the DCs
 

Seeyou

Expert Member
Joined
May 1, 2007
Messages
2,358
Gave up honestly. Not even worth trying, and it's just getting worse as people in EU get home and start logging on.
 

tehan

Active Member
Joined
Jul 5, 2017
Messages
72
Vumatel are still “investigating”. Personally I think they’re waiting for DFA to upgrade the backhaul to the OLT feeding your estate. You’re in a unique situation because of the legacy SADV hub-spoke network design that deployed OLTs on the back of individual backhauls, and 1G of backhaul was probably all that was ever provisioned. We’ll obviously never get an admission to this and we have asked directly if the OLT has more than 1G feeding it- no one could answer. This explains why you get a peak of 800 at off peak times and 500 during peak hours.. Our team has been out twice now to meet with the guys, with very little momentum. We will continue to follow up.
Thought it would be something like that. The Network Engineer they had here looked like he definitely thought it might be capacity related if it wasn't an ONT issue, but obviously, Vumatel won't give up that easily. Thanks for keeping on them, seems we're in it for a long haul.
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,456
How's this going to work btw? Are you going to route certain traffic types over the new upstream? (which presumably doesn't have the same DDOS triggers)
With routing, inbound and outbound are handled completely separately. We can dictate outbound routing by favouring certain paths or adjusting various factors with BGP filters. Just remember, what you see as a traceroute going to a server is not always the route that network chooses to return traffic on (one of the caveats of BGP). Inbound is a little more complex, we only have two tools; communities (where we explicitly tell an upstream what to do) and AS path length (making a specific path look longer or shorter based on AS hops). We have a strategy for managing DDOS with this new provider in tow. With regards to inbound traffic, our strategy in the first few days once we go live will be to advertise equally across all upstreams in the region (CPT) to see how traffic is shifted around and what users are seeing. We'll bring up a bunch of probes globally and run some tests inbound to CPT prefixes as well. Once we have a clear picture of how the rest of the world sees us behind this new upstream, we can start tweaking; so user feedback during this week or so soak period will be vital. We'll keep you posted along the way.
 
Top