Afrihost cloud down again, and again, and again, and again.

[AfriMan]

I copied Gian into an email sent to my account manager this morning. Our server went down yesterday at around 04:30 in the morning and is still inaccessible. Hoping to get a response in terms of when the server will be back up and / or how we can get access to our backups to relocate our hosting.

We are truly sorry for the downtime. As you can see from this thread, there are quite a few clients affected by this issue, and I'm sure many of them are mailing Gian directly.

I can promise that we are aware of every server still experiencing downtime and we are addressing each one individually to ensure we get everyone back up as quickly and thoroughly as possible.

We know it's a lot to ask in light of recent hosting issues, but please bear with us. I can assure you that everything that can be done is being done and no expense will be spared in getting you back up

 

[AfriMan]

Haha there was an instance last year where there website resolved to mweb during an outage. I nearly cried laughing at what had taken place.

Not our finest moment - to be sure

 

[AfriMan]

Afrihost, Afrihost, O Afrihost.

If you have so much believe in your "Cloud Servers", why don't you host your own domain on a "Cloud Server" with the rest of us?

Our website and internal management servers are all hosted on cloud servers. However we have redundancies setup with failovers at multiple MTN data centres, and we use clustered servers so that if one fails, traffic can be diverted to other servers which are spread out for maximum disaster recovery.

 

[AfriMan]

He must know about this; and it's probably his doing anyway if you think about it. And it's not just the VPS that is a stuff up; the entire Gallo Manor MTN data centre is also a stuff up. Moving from IS was a big mistake in my opinion. Over the last 6 months the VPS issues have been between the virtual environment and the physical data centre. I know that data centre, I had a Hetzner server there and I remember having issues not related to Hetzner, but that data centre. Since Hetnzer moved to their own data centre, I have not had a single instance of ****.

At the end of the day Afrihost chose a cheap thing and that cheap thing is breaking. I think they should go and beg IS for some space and say sorry.

Gian is definitely aware of what is going on, and I can promise you that there is no shortage of whips being cracked and fires being lit to ensure that we get this sorted out as quickly as possible.

This issue is specific to us, and is not an MTN DC outage. It an issue with data storage in one of our cloud environments

 

[AfriMan]

I wonder if we should maybe email Gian and let him know about these threads and problems. He surely can't be happy with his staff. They used to be so great when they were small. It's saddening to see them like this.

I know that Gian has full confidence in our team. There will definitely be a full review of what went wrong and what could have been done better. I don't believe that any individual is at fault here, but obviously as a company we are responsible and are doing everything possible to ensure we address this quickly

 

[AfriMan]

I wouldn't call this cloud...
This is a crappy Vps solution on cheap hardware branded as something it isn't close to being. Where is the redundancy? Where is the DR plan? How bad is your config for this many nodes to fail at once?

I see a repeat of the Web Africa VPS situation taking place. The sooner they admit that they are in over their heads the better.

We don't believe that this is the case. We use the same VMware as is used by major hosting companies overseas and even the New York Stock exchange. We believe the paltform is stable and we do use industry standard hardware, at a state of the art data centre.

We were actually implementing stronger DR mesaures, which related to data storage, and we believe that the issue may be related to that. We are working with MTN and the data storage vendors to bring services back and then review what changes need to be made in the infrastructure.

 

[AfriMan]

Our host provider moved our sites to SA in Nov/Dec last year via AfriHost and ever since then we have had excessive downtime...today we have been down since around 7am and not end in sight.

Given that we are in the middle of some R 140m in transactions, this downtime places our company at huge risk.

Needless to say, our host provider is going to lose passive income on 4 websites for life thanks to AfriHost's problems...I am sure there are many many others that are of same mind...he choose to use AfriHost...it backfired so he must pay as we move our domains back to Godaddy.com...

We know that the impact on affected clients has been huge, and we promise that we don't take this lightly. Every effort and resource is being employed to get things up and running in the most efficient manner possible. Our team have worked through the night and will continue to do so until the issue is resolved.

We are truly sorry for the negative impact this has had on you and your company

 

[Shred]

I feel for you Afriman, I know it's not your fault and must be pretty hard to deal with all these complaints.

But you know, as clients this isn't the first time this has happened. It's not even the 2nd 3rd or 4th time. It's happened countless times in the last 6 months; maybe not this particular issue but other issues. We are paying money for this and now yes after this I am going to move my servers but we have still paid 100% of the money value but did not get the quality of service we paid for. So effectively Afrihost's clients take a knock because of this. How fair do you think this is?

 

[ezras]

At last a compliment to Afrihost team. It seems like someone is reading these post and actually doing something about it.

-Network status was updated back to High on their website.
-Seems like they update the status continuously, keeping us posted on whats happening.

Just two questions I'm sure everyone wants to know:
1) Do you have any estimated "down time" - wont hold you to it if its not, but will at least help with planning and communication to our clients. - another day, two days....
2) I know that a couple of people is moving their entire server away like me. Is there any alternative option to access the backups or data - copy it out directly to the new server as we not really interested in getting the server up and running if it make sense.

-----------------
High Impact
Opened: 07:02 Wed, January 22, 2014
Update:11h00
Our team is working as fast as possible to move affected clients off the affected data store


Update: 09h00

Our team continues to migrate affected clients off the affected data store.

Our sincere apologies for the inconvenience caused. We hope to resolve the issue as fast as possible.


Update: 20h00

A critical problem has been identified on a data store.
Our engineers are migrating affected clients off the affected data store.

Opened: 07:02 Wed, January 22, 2014
We are reporting a problem at the MTN Gallo Manor Data Center that is affecting our Cloud Hosting environment. Affected clients may experience issues connecting to their servers. The cause is unknown but our team is addressing the issue as top priority. Apologies for any inconvenience caused during this time, we hope to resolve the issue as soon as possible.

 

[AfriMan]

I feel for you Afriman, I know it's not your fault and must be pretty hard to deal with all these complaints.

But you know, as clients this isn't the first time this has happened. It's not even the 2nd 3rd or 4th time. It's happened countless times in the last 6 months; maybe not this particular issue but other issues. We are paying money for this and now yes after this I am going to move my servers but we have still paid 100% of the money value but did not get the quality of service we paid for. So effectively Afrihost's clients take a knock because of this. How fair do you think this is?

Absolutely, I think we (and our clients) have been pretty unlucky with the string of incidents of late, and we are definitely aware of the impact this has on our brand and consumer perception. I can promise that if this was due to negligence or poor planning, we would spend whatever it took to correct these issues. I do believe we have some of the best people working on this. I know the server admin manager works all hours (I've email him at midnight and got an immediate reply) making sure that we're always running the best and latest tech for our environment.

I have confidence in our team and I know they'll figure these issues out. It is after all the same core team that built our reputation over the last 14 years as the best web hosting provider. I know they get this done right

 

[AfriMan]

At last a compliment to Afrihost team. It seems like someone is reading these post and actually doing something about it.

-Network status was updated back to High on their website.
-Seems like they update the status continuously, keeping us posted on whats happening.

Just two questions I'm sure everyone wants to know:
1) Do you have any estimated "down time" - wont hold you to it if its not, but will at least help with planning and communication to our clients. - another day, two days....
2) I know that a couple of people is moving their entire server away like me. Is there any alternative option to access the backups or data - copy it out directly to the new server as we not really interested in getting the server up and running if it make sense.

-----------------
High Impact
Opened: 07:02 Wed, January 22, 2014
Update:11h00
Our team is working as fast as possible to move affected clients off the affected data store


Update: 09h00

Our team continues to migrate affected clients off the affected data store.

Our sincere apologies for the inconvenience caused. We hope to resolve the issue as fast as possible.


Update: 20h00

A critical problem has been identified on a data store.
Our engineers are migrating affected clients off the affected data store.

Opened: 07:02 Wed, January 22, 2014
We are reporting a problem at the MTN Gallo Manor Data Center that is affecting our Cloud Hosting environment. Affected clients may experience issues connecting to their servers. The cause is unknown but our team is addressing the issue as top priority. Apologies for any inconvenience caused during this time, we hope to resolve the issue as soon as possible.

I don't have an indication of the ETA just yet, we are moving and rebooting servers individually. Some could come up shortly, but other may still be several hours.

I think it would be best to let us get the servers up and running for now, making backups to move servers will delay the process for other affected clients

 

[AfriMan]

View attachment 95127

This is the message you receive when you login to their Client Zone.

Did Afrihost offer anyone their money back so far?

Then the bragging banner on their website with "ISP of the year 2013"
http://www.afrihost.com/#3

Anyway, just praying that the server will come on long enough that we can copy the data over to the new provider. I eventually had to switch my cellphone off. Clients swearing you like a........

Anyone is welcome to claim against our guarantee if their product is within the eligible time period. That is a guarantee that we absolutely stand by.

I do believe we'll have this issue sorted as quickly as possible though

 

[simonbee]

how do we claim against the guarantee?

 

[Superspeed]

Well I dont need to add to this - I see everyone is having the same amount of issues. Our business is at a standstill. We have voip, crm, emails and website running through Afrihost. BIG MISTAKE! I'm sending my guys home - 15 staff! Let me quickly work out what that claim amount is going to be against Afrihost.....

 

[Pantheon]

We changed the impact becuase overall services are not down, and the data centre and our hosting network are not down completely. This is a data storage issue affecting a specific group of clients, at present less than 300 clients in total, and the bulk of our client son shared, dedicated and cloud hosting are not affected. This is also specific to one environment within our Gallo Manor DC. Our other DC's have not been affected either.

It would be better if the status page then showed more granular information, for example, what is working and what isn't on the various hosting services. Seeing a medium impact warning when our server have been down for more than 24 hours isn't right.

Less than 300 clients doesn't sound like much but considering some of us provide hosting services the overall impact of this outage is significant. We alone have thousands of people directly being affected by this issue.

 

[AfriMan]

how do we claim against the guarantee?

You'll see it on our website in the About Tab > Our Guarantee. You can submit a claim for review there

 

[AfriMan]

Well I dont need to add to this - I see everyone is having the same amount of issues. Our business is at a standstill. We have voip, crm, emails and website running through Afrihost. BIG MISTAKE! I'm sending my guys home - 15 staff! Let me quickly work out what that claim amount is going to be against Afrihost.....

We only pay you back what you paid us (or double that) on the 60 day guarantee. It's not really designed for a claim for costs or expenses.

I do understand that the impact is pretty huge (and horrible) to clients affected by this. I know our management do as well

 

[AfriMan]

It would be better if the status page then showed more granular information, for example, what is working and what isn't on the various hosting services. Seeing a medium impact warning when our server have been down for more than 24 hours isn't right.

Less than 300 clients doesn't sound like much but considering some of us provide hosting services the overall impact of this outage is significant. We alone have thousands of people directly being affected by this issue.

We do understand, and we know that there are secondary effects for our clients who are servicing their own clients, who have businesses with their own clients.

We are truly aware of how badly this affects people. I can only say that we have the very best people working on this, with the help of MTN's top guys, and we're doing absolutely everything that we can to get services back up quickly

 

[connorlong]

Moved from the USA to SA

It would be better if the status page then showed more granular information, for example, what is working and what isn't on the various hosting services. Seeing a medium impact warning when our server have been down for more than 24 hours isn't right.

Less than 300 clients doesn't sound like much but considering some of us provide hosting services the overall impact of this outage is significant. We alone have thousands of people directly being affected by this issue.

I moved servers at the end of December. Nothing but hassles. More downtime in with Afrihost in the first few days than I have experienced in the last 10 years. I have 200 clients/companies who are down at the moment.
I have lost half a dozen clients already. The final amount I will know in the coming days.

 

[ezras]

Ok, so we not that big, but I guess every single cent helps.

Many thanks to Dave from domains.co.za (who is currently my hero) who managed to copy out 10 sites last week Friday night between 7 & 11pm.

WE ALL KNOW HOW THE DEVIL IS WORKING! Our training domain was one of the lucky ones that he copied out.

We had this week alone 36 positive helpdesk queries, 18 confirm bookings at a cost of average R3500 per head. Last question on the enrollment ask how they find us, and every single one was "google".

We not that big and R63 000 may not be much to everyone else, but Dave just saved me a lot of money this week. Just image he did not copy this domain out last week? Just image the 36 also book!

On the other side we lost now 13 clients with web hosting. Been called anything and everything you can think of by people.

What can you do..... now we must fill in a form to claim for something we had to pay for in advance......Don't Afrihost bill us in advance?