Tales from the Machine Room

Home Page | Comments | Articles | Faq | Documents | Search | Archive | Tales from the Machine Room | Contribute | Set language to:en it | Login/Register

A Lot of Things are just like Nothing

Have you ever heard of "change management"?

No, is not when you decide to change your management and go on a killing spree with a machine gun and then put out fliers with "looking for new managers: the old ones are all dead" around the office. Because that would be a fantastic idea, even if highly illegal.

What is usually referred as "change management" is a very confused set of things that can, but are not required to, involves things like a fixed procedure to issue "changes" to a set of systems.

The basic idea behind it is that, if you have a set of systems, issuing changes in bulk is a lot faster, and better, than go one by one manually. That makes sense. In context.

The major stumbling block, to move from the idea to the reality, is that you need to have a SET of systems. That is, all your "target" have to be sort-of similar. The problem is when the similarity starts and ends with "they are computers and are hosted by us".

What do I mean? Let's get into the details.

The lovely place I ofter refer as ShittyHostingProvider, was created as a platform by SUSL when he wanted to launch his Web Development company, because he realized that having a Web Development agency, without a way to HOST the resulting products would be stupid. So, he teamed up with an old school buddy of his and created the "sister" company that handled the hosting. Of course, the hosting part couldn't survive only with the products of the web development company, so both went, more or less, to work on their own.

This means that, ShittyHostingProvider managed to scrounge up a lot of different customers, that wanted to host stuff on the internet without having to hire a sysadmin or learn all that shit on their own.

All nice and well. The problem was that, all those different customers, had their own ideas about how the system should have worked and what should have done, and the only way to get the customer on board was, more or less, do the things their way and not "your" way... Especially when you don't have "your" way yet.

ShittyHostingProvider ended up with a large collections of systems that were installed, configured and operated differently from each other. You had your PHP server, your Tomcat system, your WebSphere oddity, your Oracle behemoth, you Db2 juggernaut and then there was the Windows Collective, and none of them was even remotely similar to each other.

I mean, even looking at 2 theoretically similar systems, like 2 tomcat servers, you had differences. One had Tomcat installed in /var/lib/tomcat (why?) and the other was installed in /opt/tomcat (WHY?), even going into more general stuff there were differences. SSL Certificates? This server has them in /etc/pki/tls/certs, that one into /etc/httpd/ssl (whyyyyyy???).

And since the customers themselves didn't know the details, and weren't even interested in knowing (that's why they were paying ShittyHostingProvider after all), whoever was installing a system was, more or less, in charge of choosing whichever way he thought was the best. And since the boss never ever felt the need to sit down and say "hold on a minute, this is shit, let's bring some order into this chaos"... well, Chaos had its way.

When it was time to tackle the "deployment" of an application, of course, each one was a unique snowflake. So, for each system, there was a way to do the deployment. That could be a simple way, of just copy the bloody fucking thing in the right directory and then restart the application server, to manually adjust a bunch of configuration files and recompile things, and then restart the application server. So, a "deployment" could be anything from a few minutes, to several hours.

And you can go on with that method almost forever, sure, every system is a one-off, but who cares? As long as they works and you keep an almost-decent documentation about how to perform the task for each system.

In the meantime, SUSL decide to "grow" the business, by buying and merging together a bunch of other companies, the result of this "merging" looks a lot like some kind of B-movie "blob" monster, with a bunch of peoples that don't know each other and suddenly are supposed to cooperate and "share" their systems. And each one of them, is a custom-grown system, that has no relation or similitudes with anything else.

Of course, everyone has perfect knowledge about the system THEY designed and built, but has no information (and in fact, doesn't give a flying fuck) about any OTHER system that is now part of this hodge-podge.

And then... Somebody got the idea: Hey! Let's get this-or-that certification! And one of the point of the certification is: devise and employ a standardized system to handle changes in your environment. And then the bomb goes off, because you don't have "an" environment. You have an hundred different environments rolled into one.

To tackle the problem, the management decided to hire somebody else and create a "group" to devise the management system.

The initial person that was hired for the task, let's call him CL for short, had special "qualifications": he created and then run into the ground his own hosting company. Then, in the 5 years after that, he worked for about 5 different companies in different positions, from "office manager" to "dispatch operator" (fuck if I know what that means). So, he was perfectly competent in devising a fully integrated change management procedure.

Now, when I met this guy the first time, my brain immediately snapped back to a similar guy I met many, many, many years ago, that is the only guy that I know that when told by his project manager "this thing, we must proceed like on eggs", he replied "what does that mean?".

Anyhow, he immediately jumped into setting up a bunch of meetings with this-or-that guy in the company. This-or-that guy that happened to be permanently absent, in the wrong office, just too busy, on holiday or in various state of alcoholic stupor and so couldn't participate to the meetings. Yes, I do realize that having a meeting with yourself doesn't exactly qualify as "having a meeting", but is the perfect way to justify a total lack of progression in the project.

Since he was unable to proceed, SUSL decided to temporarily attach him to our "unit" so he could "see first hand what is the actual procedure". The "actual procedure", I pointed out, "is simply 'do the fucking thing'". But that wasn't enough for SUSL, so he got a desk and had some fun looking at the various tickets that we were handling. Of course he was very careful in not doing absolutely anything.

Until our Boss decided to drop one ticket on him so he had something to do.

The problem was with the system of a customer, they, or better, their developers, decided that instead of uploading pictures ready to be published, they wanted to upload "raw" pictures then have an automatic procedure to "manipulate" the images scaling, cropping etc. before publication. To do so, they had written some sort of interface on top of ImageMagik.

Now, nothing odd, except the fact that halfway through, there was a nice update to ImageMagik that BROKE EVERYTHING. In specific, a bunch of parameter they were using to 'pilot' ImageMagik were "deprecated" and the new version of ImageMagik was choking on them. Since nobody had checked in the non-existent test environment, when we applied the upgrade to the production system, that one went belly up and stopped working.

The customer didn't realized immediately, since the developers weren't interested in any message returned by ImageMagik and were simply trashing the return code, and when they reported the problem and the Boss dropped the ticket on CL he had no clue about what the problem could be.

To be fair, he looked at the script for about 45 minutes before declaring that he couldn't see anything wrong with it (because there was nothing wrong with it), so he started poking left and right with file rights and directory structures and what not.

After a couple of days spent mucking around without achieving anything, he started setting up meetings with other peoples again with the "change management" project and basically disappeared from the office for about a month. Since he was still "owner" of the ticket, nobody saw any updates that was applied to it. Updates that were mostly done by the customer on the line of "what the fuck is going on with this?".

During the month, there were a number of "team meeting" that he was supposed to attend and during which he was either absent or gave one-line update on the line of "lots of stuff happening..."

After about a month, he sent a mail that he had an accident in the morning and was in the hospital. About a day later, one of our project managers bumped in him, while he was doing some shopping, and he didn't looked like somebody that just had an accident at all. The gig was off. Two days later he was requested to return the office's key and then was shown the door.

Now, what can we say about this sad story? That if you give somebody a goal without any sort of measurable milestones, you can be sure that things won't go the way you'd like.

22/01/2017 13:06

Previous Next

Comments are added when and more important if I have the time to review them and after removing Spam, Crap, Phishing and the like. So don't hold your breath. And if your comment doesn't appear, is probably becuase it wasn't worth it.

7 messages this document does not accept new posts

Daniele Lupo

By Daniele Lupo posted 22/05/2017 09:08

Bene, dato che proprio oggi comincio un lavoro da un cliente che non sa cosa vuole e come capire se sto facendo il lavoro bene o male, spero di non fare la sua fine... -- Daniele Lupo


By Nik posted 22/05/2017 17:12

Comunque, per me "dispatch operator" è "fattorino"

-- Chronicles of a Broken Heart


By Thomas posted 23/05/2017 00:48

A me "dispatch operator" sembra un bel modo per dire "centralinista"...

-- Thomas


@ Thomas By Boso posted 23/05/2017 09:56


A me "dispatch operator" sembra un bel modo per dire "centralinista"...


Switchboard Operator, centralinista si dice Switchboard Operator... con le maiuscole eh!

-- Boso


By Guido posted 23/05/2017 07:54

Piu' porti avanti una cosa fatta male e piu' difficile sara' sistemarla...

-- who uses Debian learns Debian but who uses Slackware learns Linux

Solito codardo anonimo

By Solito codardo anonimo posted 23/05/2017 15:18

Chi s'è poi preso la gatta da pelare? Uno a caso, vero?

Del resto cosa potevi aspettarti da qualcuno che vantava tra le sue qualifiche il "dispatch operator", che sarebbe semplicemente l'operatore da call center (almeno credo)?

-- Solito codardo anonimo


By Matteo posted 26/05/2017 09:27

Un po' come il nostro Warehouse & Logistic Manager (magazziniere)...

-- Matteo

7 messages this document does not accept new posts

Previous Next

This site is made by me with blood, sweat and gunpowder, if you want to republish or redistribute any part of it, please drop me (or the author of the article if is not me) a mail.

This site was composed with VIM, now is composed with VIM and the (in)famous CMS FdT.

This site isn't optimized for vision with any specific browser, nor it requires special fonts or resolution.
You're free to see it as you wish.

Web Interoperability Pleadge Support This Project
Powered By Gojira