Tales from the Machine Room


Home Page | Comments | Articles | Faq | Documents | Search | Archive | Tales from the Machine Room | Contribute | Login/Register

Dev-OOPS!

If you've followed my (mis)adventures in the course of history, you should have realized by now that I love scripting and every time I can automate or simplify something, I jump at it.

However, we should note that the "automatic procedure" is not always the "best" procedure, sometimes the best thing to do is to proceed manually. This should always be a backup option. There are situation where the automatic way fails and it is correct that it fails, and the best thing to do is to go at it by hand, one line at a time. Moreover, the manual system should always be documented with the automatic way, to know what the "automatic" thing do, so when it fails miserabily (and note I said "when" and not "if")  you can proceed with the debugging and check what went wrong.

But, as usual, there are peoples out there that doesn't like to "RTFM" if this is longer than a single Ikea instruction sheet. What they want is to do a copy & paste of the whole thing into a terminal window, hit "enter" and then go for a smoke (or whatever they have in their pockets that looks like a fag).

This type of peoples is the type that get surprised when they discover that the "autopilot" of their brand new car can't "autopilot" jack squat and is nothing more than a glorified "cruise control" with some more bell and whistle, or they get stuck yelling vocal commands to their phones that keep ordering the same pizza (or worse).

Another sticky point is that I tend to make scripts when the operations are many and always the same, make a script to automate ONE single operation that has to be executed at worse twice on a system is not "saving time", is just a pain in the ass. But the evangelist of 'dev-ops' always and everywhere insists that if dev-ops has to be, then dev-ops must be all the time, doesn't matter the context.

And after this confused introduction, let's talk about $confused, they were a small hosting company that had the bad luck of entering the sight of my company in the moment they were looking around for more small companies they wanted to buy. The result was that $confused was now another "division" that was supposed to take care of "their" customers while the manglement figured out what to do with them.

Now, after the general "run-away-till-you-can" that followed the buy-out, the "personnel" of $confused that remained and "help out" (against their will, when they really can't get out of it) to manage the stuff, is reduced to 3~4 peoples. From what I understood the smart ones decided to jump ship and the ones that remained were the too lazy or didn't got a new contract fast enough. What is important, is that they sort-of "discovered" dev-ops and deciced that it was the best thing after pre-sliced bread.

So, one nice morning, I got a phone call from CL, complaining that something was amiss in their system.

For some reason that I never managed to figure out, our "general" number was the standard default-to number, so whenever somebody was calling we were getting the hot potato no matter what. Even if the problem wasn't one of "ours". And the various companies we bought up weren't really in any kind of hurry to write down the documentation in our "wiki", in fact they were doing everything but documenting, the result was that figuring out what was what and who did what was kinda complicated.
After a vain search to understand who the fuck was CL, which was the "system" he was talking about, who installed it and who was supposed to maintain it, I decided to "escalate" the problem to DB and good luck with that. Because you can nag us that we should be "competent", but if you don't provide any information I can't be competent.

After a couple of days (yes, we were at the point that toke DAYS to figure out who was supposed to do something), turns out that the CL's company has a few machines that were maintained by $confused, so I direct a request to the survivors to handle the ticket. The request remained completely unaswered. In the mean time, of course, CL had called another 2 times, and I can understand that.

Also because his request (CL I mean) wasn't anything special: change the timeout of a webserver. Something that, knowing which machine, where, and how to connect to it, it is done in about 5 minutes. But after 2 days we were still searching for who had installed that stuff.

After a week (!) we managed to catch one of the few poor survivors and, after an hard questioning session, we managed to find out that the stuff was in Azure.

Now, I don't have any problem with Azure, besides the fact that their web-interface is horrible.

At this point we run into a Tactical problem: who should manage the thing? "We" or $confused's survivor? After a number of discussion, the manglement decided that $confused was supposed to take care of that stuff and all their ex-customers while they were documenting the whole shebang and prepare to move everything into our datacenter and under our management. I noted that the longer they toke, the greater was the possibility that the customers decided to become somebody else customers but...

Anyhow, I dropped the ticket to one of the penguin and that was it. After a few days, we began to get phone calls from several ex-customers of $confused that lamented that weird things were happening, as per instructions, we simply "routed" the things to the same penguins, but that didn't solved the problem.

Until I begin to dig a bit deeper and managed to get one of the penguin at the phone, let's call him CL1.

CL1 - ...because we always use dev-ops to do this kind of stuff, because dev-ops is the best way to do this kind of thing...
Me - Yeah, sure. So why do we have all those peoples that reports problems?
CL1 - I don't know, because everything is done dev-ops then...
Me - Let me understand, WHAT is done 'devops'?
CL1 - Everything.
Me - ... going into details?
CL1 - Everything. If we need to do something we do it dev-ops.
Me - ...for example, if a customer ask to increase the timeout of his web-server... what do you do?
CL1 - Devops.
Me - ... and the meaning of that is?
CL1 - It means that we make a script and run it.
Me - Nice, what kind of script and how do you run it and where?
CL1 - We use puppet.
Me - Ok, and how do you run it and how?
CL1 - Are you stupid? We let it run by puppet obviously.
Me - Yes, I know that, but where?
CL1 - Everywhere obviously.
Me - ... but I told you that ONE customer want to change the timeout... so you should limit the scope of that script right?
CL1 - No, what the ... wait a minute... what do you mean with ONE customer?
Me - Go look at that  ticket that I sent you last month.

Well yeah. The penguins, dev-opsed around and run the same script on ALL the systems, no matter what kind of system, if it had or not the software, what was the old configuration or the like.

And just like that, dev-ops has become dev-UOPS!

Well, they only need to document it, right?

Davide
11/06/2018 16:53

Previous Next

Comments are added when and more important if I have the time to review them and after removing Spam, Crap, Phishing and the like. So don't hold your breath. And if your comment doesn't appear, is probably becuase it wasn't worth it.

3 messages this document does not accept new posts
Guido By Guido - posted 06/08/2018 09:23

Qui in $ENTE abbiamo degli script in perl (made by me) che prendono dei file-oni  (gb e gb di roba) e li scaricano su qualche tabella (per motivi che francamente non mi interessano). 

Primo script: prendere un file csv e buttarlo sulla tabella tal dei tali  (i cui campi sono esattamente quelli del file) - fatto a mano

Secondo script: uguale al primo cambiano solo tabella e i campi - fatto a mano.

Terzo script: uguale identico ai primi due salvo per le colonne e il nome della tabella. 'spetta un attimo. Fatto script in perl che prende in input un po' di parametri (un array per i nomi colonne, il nome tabella, i dati di connessione al db) e con quello genera lo script "ad hoc" - l'alternativa era creare uno script unico ma flessibile (cosa non fattibile pero' perche' questi caricamenti li vogliono schedulare in momenti diversi).

--
who uses Debian learns Debian but who uses Slackware learns Linux


Thomas By Thomas - posted 07/08/2018 09:51

Quello che vogliono loro e' fare un bel copia-incolla delle istruzioni nel loro terminale, schissare invio e poi andare a fumarsi una sigaretta (o quello che hanno in tasca che passa per "sigaretta").





Il bello (?) è che la frase si può applicare tanto all'Olanda (dove è ancora ancora "capibile") quanto all'Italia...

--
Thomas


Jepessen By Jepessen - posted 09/08/2018 14:03

Sarò io antico, ma tutto quello che vedo e sento su devops, su quando e come si applica l'ho sempre ridotto al caro e vecchio buonsenso, senza markettamenti vari...

--
Jepessen


Previous Next


This site is made by me with blood, sweat and gunpowder, if you want to republish or redistribute any part of it, please drop me (or the author of the article if is not me) a mail.


This site was composed with VIM, now is composed with VIM and the (in)famous CMS FdT.

This site isn't optimized for vision with any specific browser, nor it requires special fonts or resolution.
You're free to see it as you wish.

Web Interoperability Pleadge Support This Project
Powered By Gojira