Hi All,
Ok first of all, I'm not going to make any excuses here, and will openly admit that mirror.ac.za has slipped up a coupla times in the last few months. A lot of this was due to a MAJOR system failure a few months back that we never totally recovered from (in which we lost almost 8 terabytes of data)
However, its not acceptable that a service as public and as utilized as this one has slipped and hence we are taking some proactive steps to resolve this issue.
Firstly, we have brought in a third person (He uses the nick Ambo on here) to help admin the system, and we are confident more human capacity will help reduce system problems in the future and allow us to respond to issues faster.
Secondly, to avoid another major system failure, the largest of the arrays has been configured to raid-6 with hot spare, from the old raid-5 config. This gives us additional disk redundancy.
Thirdly, we are looking at bringing in another front end system to front the backend disk systems, to ensure that on front end system failure there is redundancy there. We are also looking at bringing in another 3 SAN's which will be migrated from another setup we used to run and these will run in a form of software raid across the SAN's, again, protecting us from system failures.
Fourth, we are currently reviewing every sync script and going through all the sync logs to see if we can see if there are any major synchronization problems and modify the uplink sources as needed. I've also asked Ambo to improve the systems functionality that alerts us on synchronization failures so we pick these up faster.
Lastly, on the specific issues mentioned. I've rectified the FreeBSD problems and the arch linux issues that were seen are currently being rectified by a resynch that is running as we speak.
If you guys do have problems with the mirror.ac.za service though, please, do not hesitate to drop me or ambo a note and we will take a look at rectifying the problem as fast as possible.