-
Join the MyBroadband community
South Africa’s biggest forum. Discuss, discover, and connect with thousands of members.
-
Question of the Day: What's the most interesting small town you've visited in South Africa?
Domains down again ?
- Thread starter hitech247
- Start date
It's everyone, hetzner is also intermittent - there is a big DDoS attack going on.
Anthro
Expert Member
Support Ticket Opened
Good Day
I trust you are well
We are currently being targeted in a DDoS attack again. Our upstream providers and engineers are working on mitigating the attack.
These are the same DDoS attacks which have made the news recently and affected other providers like RSAWEB, Cool Ideas, Bitco, Liquid Telecom, Web Africa, Afrihost, etc.
The attack is causing intermittent and slow connectivity to Cloud and VPS servers.
Extra Information:
These attacks are relentless, very complex, sometimes last for days at a time and are very difficult to completely mitigate currently.
The majority of the attack traffic is coming from unsecured Microsoft Azure instances which are being used to amplify the attack traffic to levels previous unseen in local South African traffic.
We have reached out to Microsoft to ask for assistance, but as yet they are not willing to help and simply put, just don't care that their infrastructure is being used as a weapon to attack local network providers.
In the past, attacks like this have been fairly easy to mitigate, as 90% of the traffic came over international links, which made it easy to block outside the country and to take advantage of international DDoS scrubbing services.
Since Microsoft, Amazon, China Telecom etc have started plugging into the local South African exchanges, the attack traffic is now coming over all the local exchanges and peering links.
Internationally we have Terabits of DDoS mitigation through multiple DDoS scrubbing clouds, however scrubbing clouds and services like this do not exist in South Africa as they were never really needed.
Among the local providers, the general consensus is that these attacks are targeted at specific providers and are being funded by someone or some company who stands to gain from these attacks.
We however, are not sitting around and doing nothing about these attacks, we have a number of solutions that we are implementing and going to be implementing to better mitigating these attacks, however the time of year makes implementation and ordering tricky.
We cannot disclose what we are implementing yet, but we can assure you that we are bringing in the big guns to fight these attacks, and to restore your confidence in us as your chosen service provider.
Kind Regards,
Jurie Wessels
Good Day
I trust you are well
We are currently being targeted in a DDoS attack again. Our upstream providers and engineers are working on mitigating the attack.
These are the same DDoS attacks which have made the news recently and affected other providers like RSAWEB, Cool Ideas, Bitco, Liquid Telecom, Web Africa, Afrihost, etc.
The attack is causing intermittent and slow connectivity to Cloud and VPS servers.
Extra Information:
These attacks are relentless, very complex, sometimes last for days at a time and are very difficult to completely mitigate currently.
The majority of the attack traffic is coming from unsecured Microsoft Azure instances which are being used to amplify the attack traffic to levels previous unseen in local South African traffic.
We have reached out to Microsoft to ask for assistance, but as yet they are not willing to help and simply put, just don't care that their infrastructure is being used as a weapon to attack local network providers.
In the past, attacks like this have been fairly easy to mitigate, as 90% of the traffic came over international links, which made it easy to block outside the country and to take advantage of international DDoS scrubbing services.
Since Microsoft, Amazon, China Telecom etc have started plugging into the local South African exchanges, the attack traffic is now coming over all the local exchanges and peering links.
Internationally we have Terabits of DDoS mitigation through multiple DDoS scrubbing clouds, however scrubbing clouds and services like this do not exist in South Africa as they were never really needed.
Among the local providers, the general consensus is that these attacks are targeted at specific providers and are being funded by someone or some company who stands to gain from these attacks.
We however, are not sitting around and doing nothing about these attacks, we have a number of solutions that we are implementing and going to be implementing to better mitigating these attacks, however the time of year makes implementation and ordering tricky.
We cannot disclose what we are implementing yet, but we can assure you that we are bringing in the big guns to fight these attacks, and to restore your confidence in us as your chosen service provider.
Kind Regards,
Jurie Wessels
Anthro
Expert Member
To quote a friend
Now THAT is a proper reply from @Domains.co.za (Wayne)
Unlike the CISP stuff
"**** their zombies"
lol
Now THAT is a proper reply from @Domains.co.za (Wayne)
Unlike the CISP stuff
"**** their zombies"
lol
Unless latency and POPI is an issue.
I dont really care about proper replies Antrho
I care about my customers loosing orders during peak holiday season.
I also worry that this is not the first time that this has happened and the problem was not fixed.
Surely some red flag must have popped up to say "we need to fix this NOW" after the first attack ?
I care about my customers loosing orders during peak holiday season.
I also worry that this is not the first time that this has happened and the problem was not fixed.
Surely some red flag must have popped up to say "we need to fix this NOW" after the first attack ?
Great Idea.
Murmaider
Expert Member
Hi,
We did, it's not as easy as you think to rip up your entire network, restructure it and just drop in new equipment .
New Hardware in the millions of rands can take up to 6 - 8 weeks and we are waiting on some upstream changes on our IPT providers. If there were local cloud scrubbing providers then it would be easy to simply drop in a solution. There is a gap in the market here for someone.
Dave @ Domains.co.za
Hi Dave,
Thanks for your reply. I do understand the problem (no matter what Thor thinks
) and I understand that you guys are trying to solve the problem caused by idiots that launch these DDOS attacks. I have gone through the forums and no one really seems to know why all these attacks occur and why they are being launched. I cant really see anyone gaining anything by it.Only thing that seems to happen is that the customers get angry. lol. And then we ( yes they shout at me as well) have to deal with it.
Murmaider
Expert Member
Hi Dave,
Thanks for your reply. I do understand the problem (no matter what Thor thinks
Only thing that seems to happen is that the customers get angry. lol. And then we ( yes they shout at me as well) have to deal with it.
Completely understand and it's expected, reasonable and extremely frustrating for both customers and our staff.
My team of awesome guys and girls honestly do try on a daily basis to set the bar for customer service. They do take these attacks to heart and absorb a lot of abuse for something which is not even in their direct control and I'm sure you can relate with that.
We have always taken the stance that we will be open and honest with customers, no matter what. I might not be able to disclose certain information publicly on a forum, but you more than welcome to PM me and I will give you some more info on what we doing.
One thing to keep mind, local providers, no matter who they are, what they say, were never really prepared for the big cloud guys like aws, google cloud and microsoft azure to plug into the local exchanges with their multiple 100Gbit connections.
These are the 3 largest cloud providers.. think about that for a moment, the 3 of them combined make up the largest internet infrastructure in the world, and there is a hell of a lot of unsecured instances on these massive infrastructures.
local providers who only ever had to deal with local attack traffic had equipment in place, we did and it has worked great for the last 5 years, however when the largest infrastructures in the world gets weaponized to attack a single target over the local exchanges, it becomes a whole new attack vector.
Anyone who isn't running 100Gbit equipment (which is crazy expensive) will battle with mitigating these attacks.
For example, if a provide has for example, 20Gbits or even say 40Gbits capacity on a local exchange is going to have their links completely saturated in these attacks, no matter what DDoS equipment they have in place, the physical connections still have to be able to handle the volume of traffic. Traffic can only be scrubbed once it has already gone down the physical connection, if it's saturated, no piece of equipment in the world can help.
This is like trying to squeeze the vaal dam through a hosepipe, it's never going to work. The hosepipe needs to be upgraded to something way bigger that allows for the free flow and still has capacity left over for more traffic if need be.
For example, if you have a 100Gbit link and the attack traffic is 60Gbit, you still have 40Gbits available for clean normal traffic to come in, this is the ideal situation.
Local providers are now having to invest heavily in 100Gbit+ routers so their links aren't saturated and upgrade their scrubbing hardware to be able to handle 100Gbits+ of bad traffic and only allow the clean traffic through and do this with as little latency as possible.
This is a very expensive exercise and escalates very quickly into the 10's of millions in CAPEX and OPEX costs.
This is also not equipment that providers can be money back on, its a flat cost to the company with no increase in revenue. The amount of money being spent by local providers on solutions dwarfs the loss of revenue from customers leaving and switching to other providers, who will also eventually get attacked once the current targeted companies get their new solutions in place.
Even a small provider who might only do say 500Mbps of normal traffic during the day and are connected to the local exchanges, need to now "consider" investing in 100Gbit+ capable infrastructure... This means he they have to overspec their required normal capacity by more than 200 times (20 000%) to the point that they only use 0.05% of their capacity for normal everyday traffic.
The other 99.95% of the capacity remains unused until they get attacked. Imagine if Engineers had to overspec buildings, cars, planes, etc by 20 000%... what would the cost of these things be?
Upgrades like this take time, careful planning needs to be done before just blindly spending millions on new equipment. New contracts need to be drawn up and signed, finance needs to be in place, new network links need to be run across the DC's. Importing of 100Gbits+ equipment takes time as it's not readily available in SA, there has been very little demand for it in the past, so no one keeps stock of it.
Internet Solutions, with all their capacity, got attacked the other day, (provider downstream), and they battled to mitigate it on their new shiny Arbor ddos system - this is the best and most expensive scrubbing system in the world - unfortunately, saturated links don't care what ddos protection you have inside your network.
These are not small attacks and in no way a true reflection on the knowledge, ability or will of the teams of people at these companies who are being targeted in these attacks, I have spoken with guys and girls at a lot of these targeted providers, the sleepless nights are real. None of the targeted providers I have spoken to are sitting there blindly hoping that these attacks will stop.
Even the local exchanges are trying to help with this by implementing a form of Remote Trigger Black Holing (RTBH) to help in trying to mitigate these attacks, they are well aware of the scale of these attacks.
On the flip side, just as someone or some company is throwing money to fund these attacks, money is also being thrown at cybercrime investigators who specialize in finding out who is behind attacks like this. they will get to the bottom of these attacks.
Dave @ Domains.co.za
Last edited:
Adenoid Hynkel
Executive Member
AWS DNS was recently hit and broken for more than 8 hours.
Also where do you get the info from Hetzner is also being targeted?
Hummercellc
Expert Member
I had to move my instance to Xneelo last night, as it carries voice traffic, I am sorry I had to leave domains, and yes Xneelo costs me more because I am now using a dedicated server, rather than a virtual server.
But Xneelo even when experiencing DDoS attacks, at least functions, although services are degraded.
Domains goes down completely.
It is a sad state of affairs, it truly is, if only the culprits of these attacks could be pin pointed.
But Xneelo even when experiencing DDoS attacks, at least functions, although services are degraded.
Domains goes down completely.
It is a sad state of affairs, it truly is, if only the culprits of these attacks could be pin pointed.
Murmaider
Expert Member
Hi,
It's always sad to see a customer go, but it's also completely understandable.
Hopefully we can win you back once all our network upgrades are complete.
Dave @ Domains.co.za