to give you some background: we have extensive infrastructure for various services that we offer hosted all over the show, from collocation to dedicated servers and cloud services mixed in our business occupies:
6 data centers in South Africa
3 continents
13 ISPs
if we add in the infrastructure which we manage on behalf of clients I cannot think of a ISP/datacenter which is not used which gives a pretty good idea of what the service looks like from the various companies.
In the past 2 weeks we experienced an identical issue on 2 dedicated servers hosted with 2 different local hosts. In both cases we had a drive fail inside a RAID 1 config resulting in the server not booting, this is how things went down:
Host 1: they have HOST in their name
@5pm we noticed a drop in performance to the point where we could not access the server so we initiated a reboot, the server never came back up. At that point we opened a ticket requesting that someone go to the rack and investigate what is happening (their 1st line do not have access to iDrac/console access) so the wait began. @11pm after much shouting we find out that the server is not booting as a result of a grub issue...
at that point we asses our situation and see that our last backup is 2 days old which is not ideal so we opt to try and save the server. This is when host 1 informs us that they no longer have anyone available to join us at the datacenter (after they took 6 hours to read an error message from the screen). After more shouting they make a plan to have someone go onsite which one of our engineers.
Onsite the drive failure is confirmed though can only be addressed the next morning when someone else is available to look at the issue. That means a min of 14 hours downtime and at that point someone else will only start to look into the issue, pretty devastating if this server hosts something at the core of your business (looking at the many guys on here who have shared hosting businesses on the sideline).
Host 2
We receive a notification @6pm (this happened 1 week prior to host 1's failure) that a host has gone down, all of the servers from this host include console access so someone hops on and finds Windows stuck on a startup repair. Within minutes the server is online in safe mode and the tech finds that a drive has failed. As a result he logs a ticket with the host who dispatches someone to the DC, he is there in <30min and proceeds to swop out the drive and connect an external drive (one of their own) to the server in case we need to do a quick backup.
within <2hours the drive is replaced and our work is done to get the server back online.
Also, we ended up migrating data from host 1 to host 2 after their blunder, host 2 arranged access to the datacenter with ease to allowing us to migrate the data from the failed server with host 1.
What should you learn from this?
- make sure you have console access to asses a non booting server.
- make sure you have access to your host's datacenter 24 hours a day (even if it comes at a cost) as migrating large volumes of data is still quicker with a HDD.
- make sure your host keeps spares onsite, at some point you will need a RAM module/Hard drive in the early hours of the morning/on a public holiday when ordering a replacement part is not an option.
- if the redundancies below sound like french to you you really need to a get someone involved who can assist with the management of your servers. Many of these things don't cost that much and there are cases where smarter design of your infrastructure with redundancy can be cheaper...
Finally and I cannot stress this enough!!
Make sure you have a documented DR plan/redundancies in place for everything that is even mildly important. This makes any outage a breeze to deal with as there is a documented process of what needs to be done. The server we had with host 1 is a mail server, our redundancies include the following:
- secondary servers to catch incoming mails, these kick in the moment the primary server goes down.
- offsite backups (though not up to date) allow us to restore usernames and passwords for email accounts to another server, once smtp.domain.com is pointing to the fail over server users can send mails again. They won't be receiving anything but at this point they probably won't notice as mails are going out.
- once we decided to fail over we had functioning backups ready to go. We still needed to recover the data which was created after the backup so keeping the gaps between your backups as small as possible is key. The mails received from the moment the server failed were delivery back from the secondary servers.
Host 2 hosts our monitoring system which is replicated to another DC, fail over when you have a replica 15min behind is a breeze. 2 clicks and you are back online.
6 data centers in South Africa
3 continents
13 ISPs
if we add in the infrastructure which we manage on behalf of clients I cannot think of a ISP/datacenter which is not used which gives a pretty good idea of what the service looks like from the various companies.
In the past 2 weeks we experienced an identical issue on 2 dedicated servers hosted with 2 different local hosts. In both cases we had a drive fail inside a RAID 1 config resulting in the server not booting, this is how things went down:
Host 1: they have HOST in their name
@5pm we noticed a drop in performance to the point where we could not access the server so we initiated a reboot, the server never came back up. At that point we opened a ticket requesting that someone go to the rack and investigate what is happening (their 1st line do not have access to iDrac/console access) so the wait began. @11pm after much shouting we find out that the server is not booting as a result of a grub issue...
at that point we asses our situation and see that our last backup is 2 days old which is not ideal so we opt to try and save the server. This is when host 1 informs us that they no longer have anyone available to join us at the datacenter (after they took 6 hours to read an error message from the screen). After more shouting they make a plan to have someone go onsite which one of our engineers.
Onsite the drive failure is confirmed though can only be addressed the next morning when someone else is available to look at the issue. That means a min of 14 hours downtime and at that point someone else will only start to look into the issue, pretty devastating if this server hosts something at the core of your business (looking at the many guys on here who have shared hosting businesses on the sideline).
Host 2
We receive a notification @6pm (this happened 1 week prior to host 1's failure) that a host has gone down, all of the servers from this host include console access so someone hops on and finds Windows stuck on a startup repair. Within minutes the server is online in safe mode and the tech finds that a drive has failed. As a result he logs a ticket with the host who dispatches someone to the DC, he is there in <30min and proceeds to swop out the drive and connect an external drive (one of their own) to the server in case we need to do a quick backup.
within <2hours the drive is replaced and our work is done to get the server back online.
Also, we ended up migrating data from host 1 to host 2 after their blunder, host 2 arranged access to the datacenter with ease to allowing us to migrate the data from the failed server with host 1.
What should you learn from this?
- make sure you have console access to asses a non booting server.
- make sure you have access to your host's datacenter 24 hours a day (even if it comes at a cost) as migrating large volumes of data is still quicker with a HDD.
- make sure your host keeps spares onsite, at some point you will need a RAM module/Hard drive in the early hours of the morning/on a public holiday when ordering a replacement part is not an option.
- if the redundancies below sound like french to you you really need to a get someone involved who can assist with the management of your servers. Many of these things don't cost that much and there are cases where smarter design of your infrastructure with redundancy can be cheaper...
Finally and I cannot stress this enough!!
Make sure you have a documented DR plan/redundancies in place for everything that is even mildly important. This makes any outage a breeze to deal with as there is a documented process of what needs to be done. The server we had with host 1 is a mail server, our redundancies include the following:
- secondary servers to catch incoming mails, these kick in the moment the primary server goes down.
- offsite backups (though not up to date) allow us to restore usernames and passwords for email accounts to another server, once smtp.domain.com is pointing to the fail over server users can send mails again. They won't be receiving anything but at this point they probably won't notice as mails are going out.
- once we decided to fail over we had functioning backups ready to go. We still needed to recover the data which was created after the backup so keeping the gaps between your backups as small as possible is key. The mails received from the moment the server failed were delivery back from the secondary servers.
Host 2 hosts our monitoring system which is replicated to another DC, fail over when you have a replica 15min behind is a breeze. 2 clicks and you are back online.
