Linode's crippling cyber-siege enters day four

Hamish McPanji

Hamish McPanji

Honorary Master
Joined
Oct 29, 2009
Messages
42,240
Reaction score
6,696
Location
Jhb
And punters aren't happy about the downtime


Virtual server host Linode has been on and offline since Christmas Day as it weathers an ongoing denial-of-service attack. Four days in, its customers are getting grumpy.

"We are currently aware of a DoS attack that is affecting the Linode Manager/Website and our Dallas datacenter. This post will be updated as soon as we have more information to provide," the biz*said*in the wee small hours of Christmas Day.


www.theregister.co.uk/2015/12/29/day_four_of_linode_data_center_attacks/
 
Series of DDoS attacks plague Linode data centers, infrastructure


Cloud hosting*company Linode reported that a set of distributed denial of service (DDoS) attacks have caused service interruptions at DNS infrastructure and data center locations in the U.S. and the U.K., including Dallas, London, Atlanta, Frankfurt, Newark, N.J., Tokyo, Singapore and Fremont, Calif.

While some have been resolved, new attacks have continued to emerge and some disruptions remain ongoing.

The disruptions began on Christmas Day when the company discovered and resolved connectivity issues affecting the Linode Manager and Website, just days after it completed scheduled maintenance on Xen Linode host servers in the wake of receiving “several Xen Security Advisories (XSAs).”

www.scmagazine.com/cloud-hosting-co...nterruptions-for-ddos-attacks/article/462535/
 
A cautionary tale. Our telemetry servers are down, and because of the DOS, we can't even bring our latest data out to our Amazon or Cloudflare servers
 
imranpanji said:
A cautionary tale. Our telemetry servers are down, and because of the DOS, we can't even bring our latest data out to our Amazon or Cloudflare servers
Click to expand...

Incidents like this highlight the necessity for distributed infrastructure. Pretty poor the way they are handling this though...
 
Ipwn 4 said:
Incidents like this highlight the necessity for distributed infrastructure. Pretty poor the way they are handling this though...
Click to expand...
What would be the correct way to handle it?

How much do solutions like cloudflare cost to nullify ddos at this scale of attack?
 
airborne said:
What would be the correct way to handle it?

How much do solutions like cloudflare cost to nullify ddos at this scale of attack?
Click to expand...

from an end user perspective: distribute your infrastructure.
from a service provider perspective: ensure that your network providers have measures in place to assist with this sort of thing.

At this stage Atlanta seems to be their weakest link.
 
from their IRC, by the looks of things they are changing upstream to Level3

10:46 <@aforster> waiting for level3 to call me and make our xconnect unroutable slash turn on ddos mitigation Source: IRC:linode#OFTC
Click to expand...
 
Ipwn 4 said:
from an end user perspective: distribute your infrastructure.
from a service provider perspective: ensure that your network providers have measures in place to assist with this sort of thing.

At this stage Atlanta seems to be their weakest link.
Click to expand...

I agree. Although we have servers with other companies, and we are mid migration to Amazon at the moment......most of our core infrastructure is on linode.

Considering our servers recieve tens of thousands of time sensitive updates every minute, from thousands of devices, it's difficult to mitigate this kind of scenario. You end up having parts of data on different servers, some of which are inaccessible....and so complete data cannot be assembled

The problem is when a particular company is targeted, despite them having servers worldwide, this type of attack will go after all of them.
 
Is someone holding them to ransom?

Moonfruit that sells hosted/diy websites got taken down for days on end a week or 2 ago, they were held to ddos ransom. Apparently if you pay they just ask for more and then ddos anyway once they've dried you out to squeeze out what little is left.
 
imranpanji said:
I agree. Although we have servers with other companies, and we are mid migration to Amazon at the moment......most of our core infrastructure is on linode.

Considering our servers recieve tens of thousands of time sensitive updates every minute, from thousands of devices, it's difficult to mitigate this kind of scenario. You end up having parts of data on different servers, some of which are inaccessible....and so complete data cannot be assembled

The problem is when a particular company is targeted, despite them having servers worldwide, this type of attack will go after all of them.
Click to expand...

a month ago I would have said this is not likely but this is a massive wake up call, one provider, no matter how great they are, is becoming a flaw in systems design.

unfortunately there is still much that cannot be made redundant with ease... based on the info you provided it sounds like your app will require a redesign from the ground up which simply isn't feasible for most. We are in the same boat with some critical services which simply cannot be protected (against something like this) without massive financial implications.

The one good thing that comes from attacks like this and software like randsomeware is that people are becoming more and more aware of the threats out there which should hopefully result in better solutions in the future. The general consensus right now is that many guys are going to stay with Linode as this will only make them stronger.

airborne said:
Is someone holding them to ransom?

Moonfruit that sells hosted/diy websites got taken down for days on end a week or 2 ago, they were held to ddos ransom. Apparently if you pay they just ask for more and then ddos anyway once they've dried you out to squeeze out what little is left.
Click to expand...

yeah that seems to be what the internet is speculating. With the increases in bandwidth world wide and the rapid adoption of cloud more and more infrastructure is being hosted publicly, when the services are not secured they quickly become a member of these "botnets" used to carry out attacks on this scale.
 
adrianx said:
Some useless information (for many, I'm sure)...

I see that Taylor Otwell (Laravel creator) has said that he is going to move everything over to Digital Ocean.
Click to expand...

DigitalOcean dont do DDOS mitigation, they will null the IP getting DDOS'd and if it continues they will after some effort move you to a new IP after some time has passed, and if it continues they will Null the new IP, give you a IP to move your data off and be done with you. This is from experience. We have been the target of DDOS attacks in the past.

As with many web hosting companies, CPANEL is used to host the websites. Cpanel has NO clustering service other than DNS.
Cpanel have feature requests to introduce redundant failover of all hosting services, being Apache, FTP, Email and Mysql, but this is far from ready. This is why its hard on hosting companies to be resilient against DDOS attacks.

Proper DDOS hardened hosting is expensive, customers who want to pay the lowest possible price expect 100% uptime with the best service. The very great hosting companies will give you the very best service at any price you pay for it, but DDOS protection is not something you can contend with. I have personally been to Afrihost during a DDOS attack on one of our servers, which they turned off (not even Nulled, they pulled the plug) and you know what they told me ? "The only thing you can do during a DDOS attack is to hide, you have to take the target offline and wait for the attack to stop". Which is true when you have zero DDOS migitation hardware in place and no experienced engineers with DDOS hardware skills to deal with these things. Case and point, I know of very few DDOS hardened data centers in the world. Blacklotus and Limestone to name two of the largest. OVH also comes to mind.

Now someone is going to ask "what on earth are you hosting that irritated someone enough to be DDOSed", the answer is simple. Anything from a Church website that posted a blog about Muhammad , to a competitive B&B who wants to keep your website down over easter weekend can order a DDOS online, there are cheap services online you can use to get a server switched off or nulled in South Africa. (which I wont mention here)

None of the information I posted here is thumb sucking, sadly this is from experience.

I quote here words from a network security tech that works for us, "A online security company can never state that its unhackable, as that only invites hackers to prove you wrong. You can only make the effort as much as possible and hope to make it more effort than the hack is worth"

Customers who have data that needs to be online all the time, should host at several different companies, use CDN, use DNS failover in the event a site goes down, and look at using cron jobs to make sure all the sites you host at contains the most recent data possible.
The larger the site the more likely it is that you will have dataloss. Its a bit of a migraine but with good planning your website will remain online during a DDOS attack. Hosting at several places at once means you factor out server failure, IP being targeted by a DDOS attack, even Cloudflare going down (and yes, I can tell you from experience that even CloudFlare doesnt have 100% up time)

Its a jungle out there, keep safe.

Happy new year to everyone !
 
Last edited:
How secured/Safe is AWS and their cloud services, wouldn't that be a natural target, seeing how influential and popular they are?
 
airborne said:
How secured/Safe is AWS and their cloud services, wouldn't that be a natural target, seeing how influential and popular they are?
Click to expand...

Id say its as good as the skill of the hacker whos targeting you. Ive seen forum posts on other forums about sites going offline on AWS for varied reasons, ways DDOS attacks still manage to take services offline.

Now if you host a single website with a few databases its better to spread the site around locations and look at DNS failover.

With that said, a DDOS attacker who really wants to take you offline, will keep attacks going to each and every site until all of them are attacked.

This is not cat and mouse, the mouse has a chance of getting away. Someone who wants to DDOS you and has the money,the time and the skill for it, will take you down. Setting up at several sites makes it more expensive, and if you have sites at DDOS hardened facilities like BlackLotus or Limestone they will have to change attack vectors depending on how good the mitigation is for a specific kind of attack. It makes it more trouble than its worth. But its not impossible, if someone really wants you offline and has the resources for it, they can make it happen.
 
The Web Space Bar said:
Id say its as good as the skill of the hacker whos targeting you. Ive seen forum posts on other forums about sites going offline on AWS for varied reasons, ways DDOS attacks still manage to take services offline.

Now if you host a single website with a few databases its better to spread the site around locations and look at DNS failover.

With that said, a DDOS attacker who really wants to take you offline, will keep attacks going to each and every site until all of them are attacked.
Click to expand...

Found sites that fully protected infrastructure using Cloudclare to prevent DDoS. Got Servers IP from mail I triggered from server, where there is a will there is a way.
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X