TENET's mirror service has been decommissioned

After many mirror sync issues recently, I now found out it's TENET's mirror discontinuation. I thought Mybroadband would have reported on it, but beyond your post I've seen nothing.
 
Yeah been seeing that. They were a really good mirror. Rain also hosts a public highspeed mirror which is very reliable.

 
Thought I'd share for some awareness.
View attachment 1848768

View attachment 1848766

Seems it completely came out of the blue?

Edit: I am disappointed in the South Africa higher education and research community right now

I wouldn't blame the community that quick. It usually takes huge amounts of free time to convince cfo's who couldn't care less to keep funding these projects.
 
MyBroadband asked TENET for comment about this last week. Since we've not heard back from them, and they don't seem to have done anything with that, here's a verbatim copy of the response I sent:

We're a bit reluctant to respond to this, in part because the matter is a lot more nuanced than it might appear. We had an internal debate about whether any press reporting of it would adequately reflect that, and whether we would be better served by simply publishing something on our web page. Ultimately, I won the argument on the basis that your audience aligns with the people who're asking about this, and responding might reduce the number of times we get asked the same questions. I'm hoping I don't regret that.

With that disclaimer, here are the answers to your questions:

When was mirror.ac.za officially decommissioned?

The official end date was 10 September 2025. However, in reality, it was shut down a bit after that.

Well before the planned decommissioning date, we tried to make contact with every project for which we were listed as an official mirror and requested that they remove us from their mirror list. We only decommissioned after we'd received confirmation from a project. Unfortunately, with some hindsight, some projects confirmed they had removed us (or would before the end date), but haven't actually cleaned up properly. For example, we contacted the FreeBSD Project during August and received confirmation from them on 19 August. However, as of a few minutes ago, ftp.za.freebsd.org still points to our non-existent mirror.

It's for that reason that we've created placeholder websites for every single decommissioned mirror. We deliberately return an HTTP 410 Gone response so that search engines and other crawlers will de-list the page, and end users will receive a valid error response. In most cases, the error message will refer you to the Project's official mirror list. And where we know of viable local alternatives, we've listed those too (see, for example, https://freebsd.mirror.ac.za/). We'll maintain those placeholders for at least six months, although the non *.mirror.ac.za ones will get certificate warnings once their existing SSL certificates expire.

Why was it decided to decommission TENET's mirror service?

The short answer is that we no longer have a mandate to continue.

To properly explain that, we need a bit of history and an explanation of how TENET works.

TENET is a non-profit company that was set up specifically for the benefit of South African (public) universities and associated research institutions. That's the community we serve, who funds us, and who we're ultimately accountable to. We serve at their behest, and that's where we get our mandate.

The mirror service was created in about 2012, at a time of relative bandwidth scarcity and complex interconnection requirements. At the time, the motivation for creating it revolved around bandwidth and peering arrangements. But importantly, being a community-led organisation, the service was governed by an advisory group drawn from our client institutions, who made decisions about what projects should be mirrored and why. One of the decisions at the time was that the service should be generally accessible and, wherever possible, we should be an official mirror for any project we hosted. That led to users of the service well beyond the community that TENET itself actually serves.

Fast forward, and South Africa has a lot more bandwidth. Additionally, there are more submarine cables, resulting in a more reliable Internet in general. Over the last few years, we've seen substantial changes in the usage patterns of the mirror service. For many of the projects we mirror, we had reached the point where the bandwidth expended in maintaining a mirror exceeded that of the clients. In other words, rather than saving bandwidth (its original justification), the service now had a net cost. The five-minute average of client traffic just before we decommissioned was less than 100Mbps (whereas the server has multiple 10Gbps links and used to average gigabits per second).

Where mirrors were seeing significant client traffic, the majority of that was not from our own community or even South Africa. The only mirrors where South African clients dominated were those where we were an official mirror behind a load balancer that did proper geolocation. For all the others, with one exception, African countries never appear among the top users. For example, we saw more USA and Dutch users of the Alpine, docker, and Ubuntu cloud & release mirrors than South African; Macports was more popular in Brazil; CPAN was used in Singapore and France; Raspbian was used in Cameroon and the Netherlands more than South Africa; FreeBSD is popular in Brazil (with ZA in 16th place); etc.

However, as telling as those numbers are, TENET has lots of bandwidth, and bandwidth is not the reason this went away (if it were, we would have shut it down several years ago). Maintaining a large mirror service takes staff time, and it requires hardware, power, and other resources. So, at the point at which that hardware (and simultaneously, the operating system) reached end-of-life and we needed to spend money on replacing it, we began to consider whether this service was still adding value. And importantly, whether the time and resources would be better spent on other projects.

Ultimately, we do what the institutions we serve want (and are willing to pay for). If they wanted us to maintain a mirror service, we would. We didn't arbitrarily decide to shut the service down — we asked the community that pays our bills for input. However, that call was met with deafening silence. Indeed, all the queries we've received about shutting it down have come from outside the community that funds us. And perhaps most tellingly, the last active institutional member of the advisory group responsible for the service left in 2017, with several subsequent attempts to resurrect it having failed. What all of this means is we don't have a mandate from the institutions we serve to expend further resources on this.

And yes, we understand arguments about digital sovereignty, African hosting of international projects, etc. But as a small non-profit, we also have to be pragmatic. So what we're doing now is redirecting the resources we might have expended on this into digital sovereignty exercises that directly impact higher education, particularly around the FAIR principles and safeguarding South African research output. That's far more directly aligned with the mission of an NREN (national research and education network) than running a mirror service.

Ultimately, MyBroadband is not a forum TENET uses or wants to use to engage with stakeholders. And I'll warrant that the majority of people who read this probably aren't directly part of the community TENET serves. For that reason, I'm not going to follow up on any further comments.

If you are from an institution TENET serves and you feel this was the wrong decision, you should take that up through your institutional representation (typically through your central IT and/or library). Given TENET has already consulted several times over a number of years, it is difficult, but not impossible, for this to be reversed. However, that needs to be done with all of the above in mind, and with the understanding that reinstating this comes at a cost -- either directly or indirectly in planned projects that get sidelined.
 
Last edited:
MyBroadband asked TENET for comment about this last week. Since we've not heard back from them, and they don't seem to have done anything with that, here's a verbatim copy of the response I sent:



Ultimately, MyBroadband is not a forum we use or want to use to engage with our stakeholders. And I'll warrant that the majority of people who read this probably aren't directly part of the community we serve. For that reason, I'm not going to follow up on any further comments.

If you are from an institution we serve and you feel this was the wrong decision, you should take that up through your institutional representation (typically through your central IT and/or library). Given we've already consulted several times over a number of years, it is difficult, but not impossible, for this to be reversed. However, that needs to be done with all of the above in mind, and with the understanding that reinstating this comes at a cost -- either directly or indirectly in planned projects that get sidelined.
"Ultimately, MyBroadband is not a forum we use or want to use to engage with our stakeholders"
ANY forum should be a platform as your stakeholders were anyone using your mirror? Not just those funding your projects. It was not a private mirror lol.
Sounds a bit like a "we're too special to handle general requests for people who used our services all the time"

I miss Vortexia.
 
"Ultimately, MyBroadband is not a forum we use or want to use to engage with our stakeholders"
ANY forum should be a platform as your stakeholders were anyone using your mirror? Not just those funding your projects. It was not a private mirror lol.
Sounds a bit like a "we're too special to handle general requests for people who used our services all the time"

I miss Vortexia.
Im sure Andrew is still around :D I got some great stories with that guy :D
 
MyBroadband asked TENET for comment about this last week. Since we've not heard back from them, and they don't seem to have done anything with that, here's a verbatim copy of the response I sent:



Ultimately, MyBroadband is not a forum TENET uses or wants to use to engage with stakeholders. And I'll warrant that the majority of people who read this probably aren't directly part of the community TENET serves. For that reason, I'm not going to follow up on any further comments.

If you are from an institution TENET serves and you feel this was the wrong decision, you should take that up through your institutional representation (typically through your central IT and/or library). Given TENET has already consulted several times over a number of years, it is difficult, but not impossible, for this to be reversed. However, that needs to be done with all of the above in mind, and with the understanding that reinstating this comes at a cost -- either directly or indirectly in planned projects that get sidelined.
If you create a public mirrors that serve as official sources that gets included in every single ubuntu distribution, then those users become stakeholders.
1760437916042.png


There is something to be said for just showing a little bit of consideration for people using their service. Would have cost them absolutely nothing but an email or a post on twitter.


On the plus side, since they have explicitly said they won't engage here, everyone has the green light to call them a bunch of inconsiderate pricks.
 
"Ultimately, MyBroadband is not a forum we use or want to use to engage with our stakeholders"
ANY forum should be a platform as your stakeholders were anyone using your mirror? Not just those funding your projects. It was not a private mirror lol.
Sounds a bit like a "we're too special to handle general requests for people who used our services all the time"

If you create a public mirrors that serve as official sources that gets included in every single ubuntu distribution, then those users become stakeholders.

Let's be very clear about this. That people used the service didn't make them a stakeholder in decisions about its future or governance. It made them users of a "free" service that someone else was paying for.

Maybe, just maybe, you should check your sense of entitlement?

On the plus side, since they have explicitly said they won't engage here, everyone has the green light to call them a bunch of inconsiderate pricks.

And "they" is only me, one of the two people who's actually been doing the work of maintaining those mirrors for the last 7+ years, despite it not actually being part of my day job.

It's precisely comments like this that made me reluctant to respond to this in the first place. It's clear you didn't actually want answers; you wanted to pick a fight.

But since I'm an "inconsiderate prick", maybe I should just delete my post and go back to doing the things that the community I am accountable to wants me to do rather than wasting my time responding to trolls.
 
Last edited:
Let's be very clear about this. That people used the service didn't make them a stakeholder in decisions about its future or governance. It made them users of a "free" service that someone else was paying for.

Maybe, just maybe, you should check your sense of entitlement?



And "they" is only me, one of the two people who's actually been doing the work of maintaining those mirrors for the last 7+ years, despite it not actually being part of my day job.

It's precisely comments like this that made me reluctant to respond to this in the first place. It's clear you didn't actually want answers; you wanted to pick a fight.

But since I'm an "inconsiderate prick", maybe I should just delete my post and go back to doing the things that the community I am accountable to wants me to do rather than wasting my time responding to trolls.
Personally Im grateful for the mirror while it lasted, but I guess it made more sense when data was more expensive. Eventually the cost of maintaining those servers might eclipse savings the provider might have by hosting the mirror locally. Thank you to everyone who maintained them over the years. We used them.
 
Last edited:
Let's be very clear about this. That people used the service didn't make them a stakeholder in decisions about its future or governance. It made them users of a "free" service that someone else was paying for.

Maybe, just maybe, you should check your sense of entitlement?



And "they" is only me, one of the two people who's actually been doing the work of maintaining those mirrors for the last 7+ years, despite it not actually being part of my day job.

It's precisely comments like this that made me reluctant to respond to this in the first place. It's clear you didn't actually want answers; you wanted to pick a fight.

But since I'm an "inconsiderate prick", maybe I should just delete my post and go back to doing the things that the community I am accountable to wants me to do rather than wasting my time responding to trolls.

As an occasional user of the mirrors over the years, they were always a valuable thing when I needed them, and I'm grateful for TENET for their existence and for people like you who maintained them.

But the decision to discontinue them makes all the sense in the world when the academic/research institutions who fund TENET have no real interest or need for them anymore.
 
Let's be very clear about this. That people used the service didn't make them a stakeholder in decisions about its future or governance. It made them users of a "free" service that someone else was paying for.

Maybe, just maybe, you should check your sense of entitlement?
You used the word stakeholder, it means anyone who has a stake in its operations. As opposed to shareholder which refers to ownership.

And "they" is only me, one of the two people who's actually been doing the work of maintaining those mirrors for the last 7+ years, despite it not actually being part of my day job.

It's precisely comments like this that made me reluctant to respond to this in the first place. It's clear you didn't actually want answers; you wanted to pick a fight.

But since I'm an "inconsiderate prick", maybe I should just delete my post and go back to doing the things that the community I am accountable to wants me to do rather than wasting my time responding to trolls.

And yet here you are responding....

I know this type of work is thankless, but your organisation applied for and made itself an official public mirror, which meant other people were entitled to become dependant on it.

I am completely sympathetic for not wanting to do it anymore, but that doesn't mean that giving notice for discontinuing the service is something that people should feel bad for being entitled to. Just a bit of responsibility and consideration.

From the Ubuntu guidelines for mirrors
To keep your mirror up to date and working, follow these guidelines:

Be committed to being a mirror
Obviously, each offer to be an Ubuntu mirror is great and users appreciate it. However, it’s in nobody’s interest to go and change the <span>sources.list</span> every time a mirror disappears. So if you’re not planning on creating a mirror for the long run, don’t register it. People might get disappointed in you and/or Ubuntu.

https://documentation.ubuntu.com/project/release-team/mirrors/
 
Last edited:
MyBroadband asked TENET for comment about this last week. Since we've not heard back from them, and they don't seem to have done anything with that, here's a verbatim copy of the response I sent:



Ultimately, MyBroadband is not a forum TENET uses or wants to use to engage with stakeholders. And I'll warrant that the majority of people who read this probably aren't directly part of the community TENET serves. For that reason, I'm not going to follow up on any further comments.

If you are from an institution TENET serves and you feel this was the wrong decision, you should take that up through your institutional representation (typically through your central IT and/or library). Given TENET has already consulted several times over a number of years, it is difficult, but not impossible, for this to be reversed. However, that needs to be done with all of the above in mind, and with the understanding that reinstating this comes at a cost -- either directly or indirectly in planned projects that get sidelined.
Thank you for your service.
 
They always had the Ubuntu ISO’s as torrents and I wonder why they didn’t just bake that into sources as well.

Or do what Windows does where you can simply peer share updates/patches and in that way no mirrors are really required and anyone can just opt in.
 
They always had the Ubuntu ISO’s as torrents and I wonder why they didn’t just bake that into sources as well.

Or do what Windows does where you can simply peer share updates/patches and in that way no mirrors are really required and anyone can just opt in.
With modern CDNs, mirrors are an outdated concept anyway. They were very much needed in the days of very slow expensive international bandwidth, but nowadays that's not an issue at all.


Thanks Tenet for the years of service. It's much appreciated by those who benefited.
 
Top
Sign up to the MyBroadband newsletter
X