Wake Me Up, Before You GoGo
Ahhh, the '80s, when we were young and had long hairs... or for somebody, we simply had hairs. And the best thing of that time, was the music.
Alan Parsons, Pink Floyd, Police, Doors, Van Halen, Duran Duran... Say what you want, but if you turn on the Radio (if you still have one, I mean) you'll hear some song of that time. And this must mean something I think.
The only thing that wasn't very good in that time, were the computers. The problem was that, after the explosion of the PCs, everybody, I mean every-fricking-body, had begun to put together and sell PCs. And the result was that a lot of garbage was sold and used in places that could have used something a lot better. And the biggest problem of using such garbage instead of decent machines, was that they tended to crash at random.
This produced the "save at every change" effect that was typical of whoever was working on PC during that age. You could see that in the ability of the person to find the 'save' key without missing a beat in the typing, so the final effect was something akin to "make a change - save - make a change - save - make a chan - save - make a - save - make - save - ma - save - save - SAVE - SAVEFORFUCKSACKESAVE!!!"
And since hard disks were the most subject to failure, to that you had to add the "floppy-disk-jokeying" too.
Luckily for us, time passes and technology improves. Computers got better and that lowered the number of crashes, that let us work with less save-happy-fingers and go on for more than an hour without triggering a panic-save.
Then... The "cloud" arrived...
And as usual, it arrived promising the heaven and the heart. Server and Services that are impervious to hardware or software problems, servers that can be upgraded or downgraded at will with just a few clicks of the mouse, low cost, infinite bandwidth and so on. And, as usual, since everybody is still looking for "something for nothing" and they even think they can get it, forgetting that "you get what you pay for", everybody jumped on it.
And immediately the first problem start to show off. Sure, your services are maybe still running, but if the gateway is off line they are still unreachable, and for a customer, an unreachable system looks a lot like a dead one. Sure, you can upgrade a system with a click of the mouse, but good luck in downgrading it without having to trash it and rebuild it from scratch. And sure, the initial cost are low, just like those packaged deal that promises "holiday in $expensiveplace starting from X$", for X$ they show you the flier, if you want something more, like getting to $expensiveplace and being able to stay there for a while, that costs extra.
And then... You discover that the "server" is a fantasy, it exists only as long as you look at it, if you decide to turn it off because you want to save in costs (those costs that looked very low at the beginning), then you discover that when you turn it on again, everything in it is gone like a bad dream, all your data have gone in that empty white room where data go when nobody wants them anymore.
And it is at this point that the SAVESAVESAVE effect came back with vengeance.
And after this introduction we talk about... nobody special.
We manage a series of "cloud" environment, from the 'home-made' one, to stuff hosted on Amazon and Azure, not a big difference except that with Amazon and Azure you can pick several location and it should be more "reliable". In theory at least.
What passes for the "infrastructure" department, put together an enormous tool built with Python and Ansible, that is supposed to automate the creation and maintenance of environment on all the "cloud". This thing is built in such way that, if you create a server (or more than one), all the info ends up into a database from which you should also extract invoice data for the customer. I say "you should" because that bit apparently was never finalized.
Everything fine as long as you use that tool. If you don't, you're basically guaranteed that something, somewhere sooner or later is gonna go south. In an unexpected way and without details on how it got broken or why and what to do to fix it without trashing everything and starting from scratch.
Then there is the problem of "persistence".
Yes, because initialize the machines is easy, but once that is done, the machine doesn't do anything if you do not configure it for what it needs to do. That involve configuring the services, adding stuff like firewalls, gateways, relays and maybe also putting together special scripts or adjust others to do... whatever that thing needs to do.
Now, there is the idea, somewhere in the head of the guy that designed all this, that everybody should always and only use Ansible to do... everything. Idea that I think is crackpot. Let's be clear here, the fact that you could, in the event of a catastrophe, rebuild everything by simply running a script is fantastic, BUT... I'm old enough to know that this is based on the fact that EVERYTHING is in the script, and this is not going to happen.
Everybody (everybody that works in sysadmin I mean) during normal operation, first of all try to see what the problem is and then we normally try different thing, looking for the best approach, this require some trial and error and multiple changes. So the idea of "change the script - try - change again", turn something that maybe takes 5 minutes into a several hour of script(s) debugging, and when multiple peoles are trying to maintain a single script, things are going to spiral down fast.
What is going to happens in the end, is that you do the changes, check if it works, repeat and then... you leave it like it is. If you remember it later (yeah, right) you're going to add it to the script, otherwise... well, this is what backups are for.
And this (the backup I mean), is ESSENTIAL. Yes, because you can have a script that install the system, but the USER'S DATA are not in it. The data have to be saved and restored anyway.
So, the idea of having "everything in the script" is broken anyway. After the script, you need a restore. So why bother so much with the script?
Moreover, if you're like me, you'll end up with a lot of stuff that is not strictly essential, but as sure as hell is very handy to have around. That script that analyze the logs and generate a report that is so handy, for the 2 times a year they ask for it. But if somebody trash the script, you can bet your ass they are going to ask it right away. The other script that runs in cron and does... something and you've never backed it up or copied it anywhere, you're personal configuration file for vi with all your macros... All that stuff that is guaranteed isn't in the backup, and is not essential or you cannot rebuild it, but when it happens is a pain in the ass to rebuild them and you keep thinking "what else did I have on this thing?".
So... why am I writing this? You ask. Simple, I say, because it happened that we were working on a bunch of servers for a customer, and to apply some upgrades to them, somebody from the infrastructure decided that a nice reinstall was easy. And he did it without informing anyone. Again, this is not critical, the environment wasn't production so who cares, but the next day I was puzzled because some stuff I had and was testing were gone. Now, I managed to rebuild or recover them, but it toke me a good chunk of the day and I'm still unconvinced that always is back where it was. And I'll know only when I need something and isn't there.
Now, the aforementioned dude insists that is my fault because I didn't "ansibilized" that stuff, and we could discuss this, but my point is that nothing of that stuff was "production", there were not supposed to be there forever, once the system goes into production they will probably be disposed of. Still, I'd rather not have spent a full day rebuilding them.
Somebody suggested to have a "permanent" partition where to save all these stuff, nice, I say, but let's go back to the original problem. Why do I drive myself nuts to save stuff that shouldn't be deleted without warning? Also, if we do that, we'll end up putting everything in the "permanent" partition, that in the end will explode and you still need to back it up. Now, if it is a natural disaster, like a Tyrannosaurus Rex lose in the datacenter, ok, but this is a problem that could be avoided with a simple mail to the team to inform of the impeding datadoom.
Like... Wake me up before you make the FS gogo...
Davide
20/07/2020 10:09
