SauRoNZA
Honorary Master
My opinion based on my own experience in this space:
These distros like Proxmox, TrueNAS (especially TrueNAS), pfsense (most professional of these bunch) are on a whole different level.
Their release philosophy and their standards are incredible low.
This is part of the reason the large cloud providers have become so popular.
Breaking releases are waved away like "well it's free".
That is fundamentally the problem with their approach to software development.
They are actually doing this as a day job and they are actually getting paid.
But because they are "giving it away for free", they feel entitled to a very low quality bar.
If I said some of the things these guys say publicly I'd lose my job.
The professionalism is just not there.
FWIW I work for a fairly large cloud provider and we moved to humans not allowed in terms of anything that is production quite a few years ago.
Our deployments have been hands off now for years.
The servers are constantly rotated (automatically) and software redeployed (automatically). (ie. the entire VM is replaced and moves between machines, etc.)
It exercises the path pretty well and avoids nasty surprises from "special servers" suddenly dying and being a mission to replace.
To make sure we can unblock people from fixing things if it goes really bad we have a break glass mechanism.
You can ssh to a host by breaking glass which sets off a lot of alarms and logs a transcript of what you are doing.
Systems with higher confidentially levels (ie. stores customer data like CC info, and so on) don't typically have a break-glass or has several additional restrictions for break glass for obvious reasons.
But overall in our world where we have millions of servers, the data has shown again and again that having people do things on servers is a bad thing, no matter how accomplished, professional, well intentioned or educated the individual believes they are. It doesn't work.
Not if you care about your customers and providing them with a stable product.
My opinion again but:
What's good for the Adobe installer on their company issued laptop, is good for an "IT admin", "dev ops eng.", etc.
We've actually gone completely the opposite. Everyone has admin rights on their computers (except if it is shared like call centers). Because it drastically reduces friction for "getting it done". Instead we assume zero trust. Treat each individual laptop/desktop/etc. as compromised and zero trust.
The opposite is building a walled garden which is inherently incredible fragile.
LastPass has gone through some epic hack attempts and it shows the fragility IMO of this walled garden approach.
They allowed some of the "super engineers" access to incredible sensitive databases and that person became a target of a long game.
Compromising that single individual brought down their walled garden.
This is true of any system or individual that is protected by a walled garden, at some point the wall is going to get compromised. A good engineer recognizes that it can never be completely secure, a bad "engineer" insists on more restrictions and lockdowns. It is the same as a dictator attempting to hold on to their power. Eventually it comes crashing down.
Ofc. the zero trust model is not perfect either, but it is inherently built on the concept of free will = bad stuff gonna happen. Which more accurately reflects our reality.
If humans really do need access to insane things like a database with every users passwords database, I suspect the CIA is onto something there with SCIF approach.
I suspect if we had a coffee table conversation off the record on the day to day operations and what and why we’d largely be in agreement on all things.
Our environments are mostly Terraform with pipelined deployments for easy replication and disaster recovery etc.
There’s no break glass required because we alert and monitor by default and for the same reason there’s no reason to take the keys away from people.
On the desktop front we actually come from a model of giving everyone open Admin access and now we are clawing back on that because it’s just too much to work with and manage in the space we operate regulation wise.
But we are going to change it to model where you can happily use the MacOS AppStore and request other apps with minimal justification, so not entirely locked down.


