The massive DDoS attack that won’t stop – Cool Ideas’ nightmare
At 13:30 on Saturday, 21 September, Cool Ideas’ networking team started receiving alerts. They were under attack and their DDoS mitigation had kicked in. Their network had been brought to its knees by a similar attack just ten days ago, so they watched closely to make sure their new defences held.
Less than half an hour into the attack, they knew they had problems. This attack was four times larger than the one on 11 September and subscribers had started to report connectivity issues.
Cool Ideas posted a notice to its website at 14:00 on Saturday to inform clients that it was being hit with another distributed denial of service attack (DDoS). It screenshotted the notice and posted it to Facebook and Twitter at around 09:20 the following morning.
The notice is posted below:
We are currently experiencing a distributed denial of service attack that is affecting all customers on our network’s international traffic.
Customers will experience intermittent connectivity loss and degraded performance to any international service/site during this time.
Our engineers are currently working to mitigate the attack and hope to restore connectivity soon. Currently no ETR.
We apologise for any inconvenience caused.
There was no further official communication from Cool Ideas on Saturday, although one of its staff and a respected member of the MyBroadband community began providing updates on the MyBroadband forum from Saturday afternoon.
Customers were unhappy, and when there was still no improvement or communication on Sunday, a few threatened to leave Cool Ideas and migrate to a different Internet service provider.
Cool Ideas posted updates to its website at 08:30 and 15:00 on Sunday, and at around 14:40 and 21:00 on social media. The afternoon update on Sunday read as follows: “We have made some changes and seen an improvement. Thank you for your patience during this time.”
Thank you for your patience and kind messages of support pic.twitter.com/5bKrkWNSyN
— Cool Ideas (@coolideas_ZA) September 22, 2019
This angered subscribers who had not experienced any improvement at all. Several subscribers were quick to jump onto Twitter and Facebook to tell the ISP that they were still experiencing problems.
The DDOS attack has been mitigated throughout the day and we are seeing traffic levels normalize via alternative routes, we will update our social platforms should we see any change, the team is monitoring the situation. Thank you for your patience and apologies
— Cool Ideas (@coolideas_ZA) September 22, 2019
MyBroadband asked Cool Ideas about the attack, its response, and its communication with clients. Co-owner Paul Butschi provided feedback to our questions.
DNS Amplification attack
“It was originally a classic DNS Amplification attack,” Butschi said.
“There usually isn’t much chance of identifying the source of the attack due to the nature of how DDoS attacks are initiated.”
Butschi explained that poorly configured ‘zombie’ devices on the Internet were used to send a massive amount of ‘spoofed’ traffic to their network.
“Identifying the zombie orchestrator is practically impossible. There was no specific target and the attack ranged across our whole IP space. The attack changed over time to involve other ports and protocols but the vast majority was DNS amplification.”
DNS Amplification attacks, also referred to as DNS reflection, use improperly configured Domain Name System (DNS) servers to flood computers with network traffic. If the flood of bogus traffic is able to overwhelm the computer, it can’t respond to legitimate requests and appears offline to anyone on the Internet trying to reach it.
When the target computer is a web server or critical network infrastructure, such a DDoS attack causes an outage.
Reflection attacks work by requesting information from a server on the Internet, but then tricking it to send its response to the target computer the attacker wants to flood.
DNS servers are a popular choice for such attacks because they are critical Internet infrastructure designed to field millions of requests per second. They are also usually connected to high-bandwidth links to enable them to deal with large amounts of traffic.
Most importantly, attackers can often cause a DNS server to generate a response that is several times larger than their spoofed request. In other words, attackers use DNS servers to amplify their attack bandwidth. Hence the term DNS Amplification.
Why was the attack so effective?
Butschi said that the sheer size and distribution of the attack made it as effective as it was.
“There are small DDoS attacks happening all the time which are mitigated easily,” he said.
“This was a very much a targeted attack on our network. The volume of the attack was a magnitude of four times larger than before.”
Who attacked you?
Cool Ideas was not the only ISP under attack this weekend. Atomic Access contacted MyBroadband and let us know that they too had felt the sting of an extended DDoS attack on their network.
This raises several questions, chiefly: Who attacked these ISPs and what were their motivations for doing so?
Butschi cautioned that any attempt to answer such a question would be pure speculation.
“There are many conspiracy theories around this,” he said. “It could be competitors, it could be the DDoS mitigators themselves.”
It was pointed out in the Cool Ideas forum discussion on MyBroadband that Cloudflare began routing MyBroadband’s website traffic via the UK since 11 September — the same day Cool Ideas was first attacked.
This could be for several reasons, one of which is that there is a large-scale DDoS attack happening across South African networks.
The following traceroute performed from Vox’s FTTH network confirms that the rerouting is still in place on 23 September.
Attackers watching for announcements from Cool Ideas
After Cool Ideas’ announcement at 14:40 on Sunday that it was seeing performance improvements on the network, Butschi said that the attack increased in intensity.
“We integrated some mitigation with our Internet Solutions capacity, which helped,” Butschi said. “The attack then increased and it took quite a bit of time to tune filter profiles.”
Ultimately, the attack was mitigated with the increased capacity and filtering, but was still ongoing, he said.
“This can result in increased latency and sometimes packet loss for specific ranges that are under attack. Mitigation mechanisms take some time to take effect on a targeted range. Scrubbing reduces the volume of attack for a targeted IP range, but some bad traffic could still affect an end-user.”
Butschi said that it is important to note that only the portions of the network under attack would “feel” it.
Lack of communication
Butschi said there was not a lack of communication from Cool Ideas regarding the attack.
“We did announce the incident on our website announcements section and there was lots of social media interaction,” he said.
“We weren’t as busy as usual on MyBroadband, but we did post occasional updates there as well. After several replies and posts obviously the feedback gets lost in the thread, and it might seem like we aren’t communicating.”
That said, Butschi explained that communicating during an incident like this is difficult.
“Most people obviously want to know when things will be resolved, and back to normal, which considering that it’s a moving target is nigh impossible to do.”
Butschi said that the last thing they want to do is make false promises, so they only updated their main announcement page once they had final confirmation of resolution.
“This does not mean that we weren’t doing things in-between to improve the situation and that it didn’t improve when we made the 14:40 announcement,” he said.
“We certainly weren’t trying to create a Streisand Effect, but we believe that factual updates are more useful than streams of messages saying that ‘we’re working on it'”.