And Then... Disaster!
This is not an "old" tale, it is, in fact, quite new. Just to show that nothing really ever changes.
I think I've already mentioned $wehostyourstuff, that are the ones that... herr... hosts our stuff. Those are germans. And I've always thought of the germans as intelligent and highly organized peoples. However, my faith in humankind have reached rock bottom and, helped by the interaction with these peoples, start digging to even greater depth.
For several reasons that I'm not going to explain because they are way too complex, OUR stuff is hosted by them, but the contract is under the name of our "mother company". This put us in the uneasy situation that if we need them to do something with our stuff, we need to ask our mother-company to tell them to do it.
Of course our mother-company doesn't know shit about how things are organized and 9 times out of 10 doesn't understand what we want, so before "passing along" the request they asks a lot of silly questions, until they decide to open a "ticket" with $wehosts, and the ticket is, usually, lacking a lot of the information we passed the first time. So $wehost start asking the very same questions to $mothercompany that turn around and asks us AGAIN .... over and over. It is clear that this is not an "efficient" way of working, and as such, every simple stupid thing takes days, if not weeks.
And "emergency" operations are even worse.
In theory, we should be able to send an "emergency" request directly to the hosting and then follow up with a phone call to be sure they proceed immediately with that. The reality, the only 3 times we actually had to activate such procedure, was that we got bounced forth and back between the hosting and the help-desk of our mother company several times, and it took a lot of phone calls to the boss, the boss' boss, the boss' boss boss, the VP, the VVP and so on and so forth before somebody finally did something.
Is no surprise to anyone that in the "annual plan" now there is a voice "move to Azure/Google".
The result is quite funny, since we moved to a "generic" hosting to a "specialized" hosting to get a better service but... Wathever, until the end of the year (and probably longer) we're stuck with them.
And now that you know the basic, let's go on with the actual tale. That began with an innocuos-looking mail in an innocuos-looking day.
The mail from $wehost was directed to our "operation team" (that's me) and refer to the TEST environment (that is not monitored by anyone) and was written in German. Because $wehosts are in Germany and they only write or talk in German. Doesn't matter if we are not German and we don't speak German (as we already tolde them multiple times). They are German and they only communicate in German.
After visiting Google Translate the content was sort of:
From: $WeHostYourStuff
To: OperationsTeam
Subject: Storage database test
Grulfgrulf (google translate failed here), resulting that the storage in server e9x852Y2G25a47353225Z73R69W7yRX7E54U ($WeHost has strange ideas about servers names) is 3% empty. Please up-clean. Informed duty.
That, after having processed a bit, I translated as: the server what-is-that-thing has a disk that is 97% full and we should clean it up.
After a research in their "dynamic documentation", that is the size of the Britannic and all the comments are (should I say?) in German, I assume that the problem is that one of the disks in the test database server is almost full. And that's the problem.
For performances we decided that the database servers (all of them, prod and test) are physical machines and not virtual, with real disks attached to a raid controller and not virtual storages. This allow for low latency, but also caused some trouble, like when the raid controller of the production db communicated that the battery was defective and the techie decided to power down the server and replace the battery on the spot. At 11.00Am. On a work day. Without informing anyone. A lot of good cursing was wasted that day.
But one of the problems with this is that... we have no access to the database server. We can USE the database but we can't login in the server or other. And the disk reported seems to be a "shared" drive that was added by $wehost itself to perform data transfer from the production system to the test system (to do tests). So after a few hours of searching around, I put together an answer:
From: OperationsTeam
To: $WeHostYourStuff
Subject: Re: Storage database test.
Dears, that machine is completely managed by your Infrastructure Group. The disk is used by the very same group to transfer data from the Production Environment. We can't access the machine or the disk and as such we can't clean it up or say anything about it. Please contact your Infra group.
Clear right? But those, I reapeat, are Germans. And the wrong type of Germans because the day after, I get another German mail that, again require the translate treatment.
From: $WeHost
To: OperationsTeam
Subject: Re: Re: Storage database test
GullaPulla, resulting the storage van server e9x852Y2G25a47353225Z73R69W7yRX7E54U empty 1% is, please empty procedure with very high celerity immediate priority. Due information.
And this start pissing me off a bit.
From: OperationsTeam
To: $WeHost
Subject: Re: Storage database test.
As already explained, that machine and that disk are managed by your Infrastructure group, please ask them what to do.
After that, I start searching in my agenda unti I found the mail and phone number of the guy that managed the thing ad infrastructure level and try to contact him, but the mail bounce with "no such address" and the phone doesn't go anywhere. As usual. So I send a mail to the 'generic' address and ask who is supposed to handle that kind of problem an wait.
In the afternoon, one of the developer came to complaint that he as requested a "data refresh" from production to test and it failed, I suppose the problem is that that disk has never been cleaned and it is now full of junk but again, can't do anything about it except inform $wehost of the thing.
The next morning, yet another mail awaits for Translate to do its thing.
From: $noihostiamo
To: OperationsTeam
Subject: Re: Re: Re: Re: Storage database test
The storage van server e9x852Y2G25a47353225Z73R69W7yRX7E54U no more empty is, empty must with maximum celerity be.
I start with replying to the mail:
From: OperationsTeam
To: $WeHost
Subject: Re: Storage database test.
Re-read my fucking mail from yesterday. THAT FUCKING SERVER IS MANAGED BY YOUR INFRA GROUP, we have NO ACCESS to the syste so ASK THEM WHAT THE FUCK TO DO.
After that, I begin the 'escalation' procedure, that consist in sending mail to EVERYONE that could be even remotely connected or interested to the thing and start phoning every single phone number I can lay my hands on. Around 3pm I've finally got somebody that has a vague idea of what I am talking about and start informing somebody that could maybe do something about it.
Around 7pm I've spoken with at least 10 people between $wehost and our mother company, none of them has any idea of who in the Infra group is responsible of the thing or who has the authorization to do something. Almost everybody agree that we shouldn't be contacted for that but nobody seems to be willing to step up and take action.
During the night, protected by the darkness, somebody does clean-up that fucking disk, because the next morning I receive yet another fucking german mail that tells me that the problem has been solved.
The "incident" (in quotes), has left a trail of mails between us, $mothercompany and $wehosts because at this point we started annoying everybody to try to figure out who is responsible of what and what should be the correct procedure, if any, on how to handle this kind of problems and why those assholes insists in asking US about stuff that THEY should manage and we can't say shit, but they keep performing semi-destructive operation without informing anybody when a mail would be nice.
Now, I've no idea how things works in that place and how is it possible that that peoples, despite the problems, keeps repeating the very same mistakes over and over, but I've the impression that they are not really oriented towards "improvement". Well, Google and Azure are still an option.
Davide
09/10/2018 12:43
e no non era italiano era proprio tedesco!
