Tales from the Machine Room


Home Page | Comments | Articles | Faq | Documents | Search | Archive | Tales from the Machine Room | Contribute | Login/Register

And Then... Disaster!

This is not an "old" tale, it is, in fact, quite new. Just to show that nothing really ever changes.

I think I've already mentioned $wehostyourstuff, that are the ones that... herr... hosts our stuff. Those are germans. And I've always thought of the germans as intelligent and highly organized peoples. However, my faith in humankind have reached rock bottom and, helped by the interaction with these peoples, start digging to even greater depth.

For several reasons that I'm not going to explain because they are way too complex, OUR stuff is hosted by them, but the contract is under the name of our "mother company". This put us in the uneasy situation that if we need them to do something with our stuff, we need to ask our mother-company to tell them to do it.

Of course our mother-company doesn't know shit about how things are organized and 9 times out of 10 doesn't understand what we want, so before "passing along" the request they asks a lot of silly questions, until they decide to open a "ticket" with $wehosts, and the ticket is, usually, lacking a lot of the information we passed the first time. So $wehost start asking the very same questions to $mothercompany that turn around and asks us AGAIN .... over and over. It is clear that this is not an "efficient" way of working, and as such, every simple stupid thing takes days, if not weeks.

And "emergency" operations are even worse.

In theory, we should be able to send an "emergency" request directly to the hosting and then follow up with a phone call to be sure they proceed immediately with that. The reality, the only 3 times we actually had to activate such procedure, was that we got bounced forth and back between the hosting and the help-desk of our mother company several times, and it took a lot of phone calls to the boss, the boss' boss, the boss' boss boss, the VP, the VVP and so on and so forth before somebody finally did something.

Is no surprise to anyone that in the "annual plan" now there is a voice "move to Azure/Google".

The result is quite funny, since we moved to a "generic" hosting to a "specialized" hosting to get a better service but... Wathever, until the end of the year (and probably longer) we're stuck with them.

And now that you know the basic, let's go on with the actual tale. That began with an innocuos-looking mail in an innocuos-looking day.

The mail from $wehost was directed to our "operation team" (that's me) and refer to the TEST environment (that is not monitored by anyone) and was written in German. Because $wehosts are in Germany and they only write or talk in German. Doesn't matter if we are not German and we don't speak German (as we already tolde them multiple times). They are German and they only communicate in German.

After visiting Google Translate the content was sort of:

From: $WeHostYourStuff
To: OperationsTeam
Subject: Storage database test

Grulfgrulf (google translate failed here), resulting that the storage in server e9x852Y2G25a47353225Z73R69W7yRX7E54U ($WeHost has strange ideas about servers names) is 3% empty. Please up-clean. Informed duty.

That, after having processed a bit, I translated as: the server what-is-that-thing has a disk that is 97% full and we should clean it up.

After a research in their "dynamic documentation", that is the size of the Britannic and all the comments are (should I say?) in German, I assume that the problem is that one of the disks in the test database server is almost full. And that's the problem.

For performances we decided that the database servers (all of them, prod and test) are physical machines and not virtual, with real disks attached to a raid controller and not virtual storages. This allow for low latency, but also caused some trouble, like when the raid controller of the production db communicated that the battery was defective and the techie decided to power down the server and replace the battery on the spot. At 11.00Am. On a work day. Without informing anyone. A lot of good cursing was wasted that day.
But one of the problems with this is that... we have no access to the database server. We can USE the database but we can't login in the server or other. And the disk reported seems to be a "shared" drive that was added by $wehost itself to perform data transfer from the production system to the test system (to do tests). So after a few hours of searching around, I put together an answer:

From: OperationsTeam
To: $WeHostYourStuff
Subject: Re: Storage database test.

Dears, that machine is completely managed by your Infrastructure Group. The disk is used by the very same group to transfer data from the Production Environment. We can't access the machine or the disk and as such we can't clean it up or say anything about it. Please contact your Infra group.

Clear right? But those, I reapeat, are Germans. And the wrong type of Germans because the day after, I get another German mail that, again require the translate treatment.

From: $WeHost
To: OperationsTeam
Subject: Re: Re: Storage database test

GullaPulla, resulting the storage van server e9x852Y2G25a47353225Z73R69W7yRX7E54U empty 1% is, please empty procedure with very high celerity immediate priority. Due information.

And this start pissing me off a bit.

From: OperationsTeam
To: $WeHost
Subject: Re: Storage database test.

As already explained, that machine and that disk are managed by your Infrastructure group, please ask them what to do.

After that, I start searching in my agenda unti I found the mail and phone number of the guy that managed the thing ad infrastructure level and try to contact him, but the mail bounce with "no such address" and the phone doesn't go anywhere. As usual. So I send a mail to the 'generic' address and ask who is supposed to handle that kind of problem an wait.

In the afternoon, one of the developer came to complaint that he as requested a "data refresh" from production to test and it failed, I suppose the problem is that that disk has never been cleaned and it is now full of junk but again, can't do anything about it except inform $wehost of the thing.

The next morning, yet another mail awaits for Translate to do its thing.

From: $noihostiamo
To: OperationsTeam
Subject: Re: Re: Re: Re: Storage database test

The storage van server e9x852Y2G25a47353225Z73R69W7yRX7E54U no more empty is, empty must with maximum celerity be.

I start with replying to the mail:

From: OperationsTeam
To: $WeHost
Subject: Re: Storage database test.

Re-read my fucking mail from yesterday. THAT FUCKING SERVER IS MANAGED BY YOUR INFRA GROUP, we have NO ACCESS to the syste so ASK THEM WHAT THE FUCK TO DO.

After that, I begin the 'escalation' procedure, that consist in sending mail to EVERYONE that could be even remotely connected or interested to the thing and start phoning every single phone number I can lay my hands on. Around 3pm I've finally got somebody that has a vague idea of what I am talking about and start informing somebody that could maybe do something about it.

Around 7pm I've spoken with at least 10 people between $wehost and our mother company, none of them has any idea of who in the Infra group is responsible of the thing or who has the authorization to do something. Almost everybody agree that we shouldn't be contacted for that but nobody seems to be willing to step up and take action.

During the night, protected by the darkness, somebody does clean-up that fucking disk, because the next morning I receive yet another fucking german mail that tells me that the problem has been solved.

The "incident" (in quotes), has left a trail of mails between us, $mothercompany and $wehosts because at this point we started annoying everybody to try to figure out who is responsible of what and what should be the correct procedure, if any, on how to handle this kind of problems and why those assholes insists in asking US about stuff that THEY should manage and we can't say shit, but they keep performing semi-destructive operation without informing anybody when a mail would be nice.

Now, I've no idea how things works in that place and how is it possible that that peoples, despite the problems, keeps repeating the very same mistakes over and over, but I've the impression that they are not really oriented towards "improvement". Well, Google and Azure are still an option.

Davide
09/10/2018 12:43

Previous Next

Comments are added when and more important if I have the time to review them and after removing Spam, Crap, Phishing and the like. So don't hold your breath. And if your comment doesn't appear, is probably becuase it wasn't worth it.

11 messages this document does not accept new posts
Guido By Guido - posted 03/12/2018 09:30

Loro sono tedeschi e parlano solo tedesco.

A me invece e' capitato l'opposto - ho scritto ad un'astinenza tecnica tedesca (in inglese) - loro mi hanno risposto io pensando che fosse un qualche traduttore automatico (per altro molto bravo a dire il vero) ho proseguito in inglese - al che l' "astinente" mi ha risposto tra il piccato ed il sorpreso come mai scrivessi in inglese e non nella mia madrelingua. Al che e' partito un OT pazzesco ma sorvoliamo :\) e no non era italiano era proprio tedesco!

--
who uses Debian learns Debian but who uses Slackware learns Linux


Anonymous coward By Anonymous coward - posted 03/12/2018 11:20

I tedeschi NON sono intelligenti (lasciatemi generalizzare un po', su). Sono organizzati proprio perché non sanno che cappero fare appena la situazione deraglia dallo standard. Sono flessibili come il granito, ma il problema non è mai loro: sei tu. Sempre.

--
Anonymous coward


Mirko Schiavolin@ Anonymous coward By Mirko Schiavolin - posted 03/12/2018 14:14

Io lavoro per una azienda tedesca: non posso che confermare (amaramente...)

I tedeschi NON sono intelligenti (lasciatemi generalizzare un po', su). Sono organizzati proprio perché non sanno che cappero fare appena la situazione deraglia dallo standard. Sono flessibili come il granito, ma il problema non è mai loro: sei tu. Sempre.

 

 

--
Mirko Schiavolin


Wrann By Wrann - posted 03/12/2018 11:40

Mi sa tanto che Google o Azure (o Amazon, o qualche altro grosso servizio di hosting) sono ormai ben più che un'opzione.

--
Wrann


Anonymous coward By Anonymous coward - posted 03/12/2018 14:30

Un mio collega tedesco usa dire: "per i tedeschi ci sono 2 modi di vedere le cose: il loro, e quello sbagliato..."

--
Anonymous coward


Anonymous coward By Anonymous coward - posted 03/12/2018 14:30

Informatica: scienza del trattamento automatico dell'informazione*

* ad esclusione di: "chi fa cosa"

--
Anonymous coward


Luca Bertoncello By Luca Bertoncello - posted 04/12/2018 08:08

Mi sento comunque di prendere le difese dei tedeschi. Non siamo poi cosi' inflessibili. I coglioni ci sono dappertutto...

Tu sei tedesco quanto io sono olandese.


Luca Bertoncello@ Luca Bertoncello By Luca Bertoncello - posted 05/12/2018 07:38

Mi sento comunque di prendere le difese dei tedeschi. Non siamo poi cosi' inflessibili. I coglioni ci sono dappertutto...

Tu sei tedesco quanto io sono olandese.

?!? Non so te, ma io son 14 anni che ho la cittadinanza tedesca... ;\)

E anche quand'ero in Italia mi dicevano tutti che come testa ero piu' un tedesco che altro...

--
Luca Bertoncello


Davide Bianchi@ Luca Bertoncello By Davide Bianchi - posted 05/12/2018 09:40

?!? Non so te, ma io son 14 anni che ho la cittadinanza tedesca... ;\)

La cittadinanza non significa niente.

E anche quand'ero in Italia mi dicevano tutti che come testa ero piu' un tedesco che altro.

Ecco, questo significa qualche cosa.

 

--
Davide Bianchi


Anonymous coward@ Luca Bertoncello By Anonymous coward - posted 05/12/2018 09:43

 

Mi sento comunque di prendere le difese dei tedeschi. Non siamo poi cosi' inflessibili. I coglioni ci sono dappertutto...

Un tizio di nome Luca bertoncello tedesco? Ma va a da via i ciapp, Luca :\). Ancora mi ricordo le storie (poi cancellate) di quando eri ai ferri corti col kapo' crucco. "Salgo io sulla scala... No io la tengo, salici tu", " col cazzo che salgo","e io ti licenzio","e io ti denuncio hai sindacati". Tze, Tedeschi: gente che inseguiva nuda le marmotte quando noi già si accoltellava un Giulio Cesare". (Semicit)

--
Anonymous coward


Anonymous coward By Anonymous coward - posted 05/12/2018 14:17

Anche qui abbiamo sedi (e relativi colleghi) in Germania. E uso dire che a loro piace stabilire le regole, ma per farle rispettare agli altri. In quanto alla loro osservanza...

--
Anonymous coward


11 messages this document does not accept new posts

Previous Next


This site is made by me with blood, sweat and gunpowder, if you want to republish or redistribute any part of it, please drop me (or the author of the article if is not me) a mail.


This site was composed with VIM, now is composed with VIM and the (in)famous CMS FdT.

This site isn't optimized for vision with any specific browser, nor it requires special fonts or resolution.
You're free to see it as you wish.

Web Interoperability Pleadge Support This Project
Powered By Gojira