Tales from the Machine Room
Like that joke said: everybody has a Test environment, somebody is so lucky to have a completely different environment to run production on.
In fact, like many other things in this universe, the usefulnes of a TEST environment is discovered only AFTER the production one has been completely fucked up by some operation, wrong or not. And even AFTER that, mostly, the Test environment is perceived as a useless waste of resources.
Personally, I think that the hundreds of "hot-standby" production environments that I saw during my years of activities, are a lot more wasteful than the least used test environment, also because those "hot-standby" are seldom allined with the 'real' production, lowering their utility to less than zero.
But when a Test environment do exists, then there is this drive, manicala sometimes, to use it in an obsessive-compulsive way, the result is a plethora of releases, deployments and tests, even for things that are obviously not goint to be used on production EVER. Until even the developers themselves don't know anymore what has been released and how (or if) to transfer it to production, making the 'test' unique as a Stradivari.
And after this confused beginning, let's start.
Is a cold and dark late-autumn morning, temperatures are already winter-y while the rain is still practically autumn-al, and the two things mixes into a perfect cocktail that provoke that "crap" effect and make you think that stay in bed is the logical choice.
In this mood I am cursing and trying to make one of our customers happy by producing some "graphs" over the use and abuse of his system, and to do so I need to get some data out of the database (that is a mess) and then feed them to Cacti in order to produce something watchable.
Yes, I'm definitively having fun.
While I am messing around with SNMP's histerical behavour, one of my collegue (CL) turns up at my desk with a puzzled face:
CL - Can you have a look at ticket 79845910-ADF7631-8476710 ?
ME - Right now I am busy, what's the problem?
CL - Is a request from Customer about a release...
ME - And where is the problem? Releases for Customer have been basically fully automated by now [a perl script of which I'm particolarly proud]
CL - Yeah, but this is different...
ME - What do you mean?
CL - It has... things...
ME - What? Is that time of the month again?
CL doesn't get it, and since doesn't look like he wants to go somewhere else, I leave SNMP for a while and look that ticket up.
Oh nice, the developers finally decided to toss the towel and admit that they don't know how or what has been released on Test and decided to start with a clean slate with a restore on Test of the Production stuff. Can't agree more. To check the differences they also want to have a dump of the database structure and a copy of the application from test before the restore from Production.
ME - So, where is the problem?
CL - Database... yada yada yada... Backup ... yada yada yada ... copy ... yada yada yada...
ME - The procedure for 'refresh' is documented in the wiki, go read it, the backup procedure also. The only thing is that TEST and PRODUCTION are using two different databases but both are on the same db Server.
CL keeps whining, evidently what he wants is for me to take care of the thing so he can go back at doing bloody fucking nothing, but since I'm not willing to comply to his plan, he ends up going to whine with DumBoss, who shows up with the same questions.
I must point out that the customer so far hasn't requsted anything strange, the majority of the activities involved are documented or should in the capacity of any sysadmin, even Junior and we're talking about a TEST environment, so even breaking it shouldn't be too bad.
DB - Can you do it?
ME - This stuff [pointing to my screen] has been in my list for a month, and since out of 5 days I am busy 4 doing customers' requests, I do it today or it will stay in my list for another month. So the answer is NO.
DB withdraw, CL looks dissatisfied and I go on with my Cacti work. But not for long, after a while CL is back for some more questions about the db backup.
ME - The procedure for the dump is in the wiki, there should be even a link to the documentation's page of the customer since we did it multiple time already.
CL - Yes but I can't understand a thing about it.
I get up and walk to his desk, followed by him.
ME - Ok, show me what ain't clear.
CL sit down and turn the screen-saver off, obviously on his screen there is everything but the documentation, he rummage a bit to look it up, after a couple of minutes I tell him that I'm going to get a coffee and when he manage to find and read it he can call me back.
When I'm back from my coffe CL has found the documentation.
ME - Ok, so what's unclear?
CL scroll the documentation up and down with a few scroll of the mouse without answering.
ME - Did you read this stuff?
CL scroll the documentation up and down with a few scroll of the mouse without answering...
ME - Good, when you've read it let me know.
And go back to my work, until about 15.30, at that moment the continous beeping of the system monitor manage to displace the Pink Floyd in my earphones and make me rise my head and try to see what is going on. What is going on is Customer's production environemnt (yes, the same Customer) that is returning an Error 500. And that's bad.
At the question if "somebody is doing something about it" only silence respond, so I log into the system and check it out. The application is running, but the log is filling up with "no data found", since I saw CL still logged into the database I go directly to him and ask The Question...
ME - What the heck are you doing with Customer's production db?
CL - What the asked... a dump and restore of the data structure, like the doc says...
ME - They asked a dump of the structure of TEST and a restore of PRODUCTION on TEST.
CL - Boh, I can't understand a thing in this.
Before I can say anything (like compare the fact that "he doesn't understand" with Cercopiteco's IQ levels), DB shows up.
DB - [weaving his mobile] What is happening with Customer?
ME - I was asking the same.
CL - I am following the documentation.
DB - So?
CL - Here it says to do a restore this way.
ME - You have to do the restore on TEST.
CL - It doesn't say it there...
ME - Yes it does. On the above line, why don't you read it. All of it?
CL scroll up and down the documentation with the speed of a rocket with a few stroke of the mouse's wheel, until I grab the mouse and highlight the part of the documentation that says (more or less):
"1. Take a database dump of PRODUCTION [example commands sequence]
2. Make a restore on TEST database [example commands sequence]"
ME - You made a dump of the data structure of PRODUCTION and restored it on PRODUCTION right?
CL - I'm following the documentation...
IO - No, you're not.
CL - This documentation ain't clear...
IO - The documentation is not made to do a cut & paste no, it is made so you can UNDERSTAND what you're supposed to do and then use YOUR BRAIN.
DB - Can we solve the problem now that the customer is at the phone?
I notice "en-passant" that apparently nobody is paying any attention to the system monitor...
ME - [to CL] get the last backup of the production db and restore it.
DB - But this way we're gonna lose the whole day!
ME - Do you have a better idea?
CL - And how do I do a restore?
ME - The same way you just did, but with the CORRRECT DUMP.
How it is or how it ain't... I do the restore and more or less everything goes back to working condition, I leave DB and MarketingDude to the fight with the Customer that wants to now WTF and go back to my work, that I hope to finish for today, but that idea is quickly disappearing in the sad darnkess outside.
The peace is short-lived, because after a bit the System Monitor begins to beeps again. Without even looking I go directly to CL's desk.
ME - What the heck is it now?
CL - I'm following the documentation...
No this time is even better: now he dumped the data structure of TEST and restored it on PRODUCTION!
Now, obviously, we are going to have a long, intense discussion over the documentation that "ain't clear" and the objective responsability of whoever wrote the documentation, completely avoiding the much more useful discussion about the objective (in)ability of whoever is reading such documentation to use his own brain and the fact that before doing something, with or without documentation, you should UNDERSTAND what the heck you're going to do.
And I understand that this work isn't gonna be done for today.
Comments are added when and more important if I have the time to review them and after removing Spam, Crap, Phishing and the like. So don't hold your breath. And if your comment doesn't appear, is probably becuase it wasn't worth it.
Storie come sempre da far rizzare il pelo, ma in questa rinnovata "serie" trovo piu' un senso di sconfitta che di ridicolo.
E la cosa mi fa un po' paura, debbo ammettere. Ora capisco bene il titolo "Anni perduti".
By Francesco - posted 30/01/2017 11:45
Scusa D ma sto genio di CL era parente di chi per essere ancora li dopo una cazzata di questa portata?
By argaar - posted 30/01/2017 13:32
spero bene che sia stato adeguatamente castigato..
passi il non capire (oh mica possono esse tutti compententi in tutto), passi il leggere male (anche se sei superficiale troppo)...ma far finta de niente accusando altro invece della tua incapacità proprio no!
PS: D, la password non me la riconosce, e il reset password non porta ad altro che alla stessa schermata per loggarsi o registrarsi, se provo a registrarmi mi torna il popup con un semplice "Error", se vuoi faccio da cavia per l'eventuale bug via mail o facebook o scegli te
By Davide Bianchi - posted 31/01/2017 05:58
PS: D, la password non me la riconosce,
Vedo che un "aargar" ha fatto login alle 30/01/2017 15:44, quindi se non eri tu...
By Guido - posted 31/01/2017 08:16
<i>IO - La documentazione non e' fatta per fare un cut & paste, no, devi metterci un pelo di cervello.</i>
Per usare il cervello devi avercelo, e se lo avesse e lo usasse non sarebbe CL. No?
who uses Debian learns Debian but who uses Slackware learns Linux
By Mario G. - posted 31/01/2017 09:56
Leggendo m'e' tornata in mente una cosa che diceva sempre mia nonna la traduco dal milanese, "roba da chiodi con la testa in legno" che indicava la totale follia.
Non ho davvero parole
By Yet Another Anonymous coward - posted 31/01/2017 21:51
Impressionante ... E gliene veniva ancora a lui ... Ma come è possibile avere questa faccia tosta ? E' un imbecille ignorante e pure dava la colpa a te ?
Capisco che poi alla fine te ne sei andato ...
Yet Another Anonymous coward
By Marcuxx - posted 01/02/2017 09:55
Già, si capisce perché alla fine hai mollato quella faccenda - con un CL che era lì (penso io) per raccomandazione e un boss che appioppava a te tutti i lavori di escremento e tutti i rimproveri che sarebbero spettati al CL, ecco, io al posto tuo non avrei resistito cinque lunghi, eterni, cosmici anni...
By mima85 - posted 14/07/2017 17:30
Non è che CL era un ex dipendente di $brancodipaguri?
This site is made by me with blood, sweat and gunpowder, if you want to republish or redistribute any part of it, please drop me (or the author of the article if is not me) a mail.
This site was composed with VIM, now is composed with VIM and the (in)famous CMS FdT.
This site isn't optimized for vision with any specific browser, nor
it requires special fonts or resolution.
You're free to see it as you wish.