Hi All,
I was away on holiday and as murphy would have it, we ran into server issues. One server has suffered severe hardware failure (we have over 5 in a cluster), but due to the shared storage between then, when one server fails, the others are affected as well.
I managed to restart the failed server via Cell Phone and then a few hours later it crashed again. It was then taken out of service completely (and still is).
I realise this isn't an excuse, but unfortunately litterally the only device that I had with me to work with was an Cell phone. Just to become aware of the problem took way to long.
As indicated in my emails, we will be compensating users affected for the downtime (this will include trial users who didn't have time to test the service), and we will drastically work on increasing our monitoring and the like. I will also be looking for another individual to look after the well being of the servers with me, so that the single point of failure (i.e. me) can be removed.
All I can say is sorry for the issues - everything is back online and stable and running smoothly...
I litterally just got home - I will go through the logs and what not during the course of tonight / today and e-mail account holders individually in terms of said compensation. Users on time based subscriptions will more than likely receive 3 or 4 days free access, whilst volume based users will receive a couple of GB free of charge on their accounts. I will know a bit more once I established just how severe the impact was.
Once again - my sincerest appologies.