Tales from the Machine Room
Everything that can go wrong, will. This is Murphy's Law, that is not really a "law" but is the simple result of the fact that things tend to break or malfunction with time and use, and if something is very important, it is very much used and so it has more chance to break than something that isn't used. Of course when "x" breaks, everything that is built on top of it will also break. From then on, things follows the cascading effect with perfectly understandable results, with a final effect that is some kind of spectacular.
That kind of spectacular that you can't avoid to watch, because is mesmerizing but also because, in theory, you learn a lot more from a disaster than from a success. In theory. In practice, everybody is too fucking busy pretending they have nothing to do with the subject and covering their asses in various ridiculous way to learn anything.
Let's talk, for example, about $companyX, of which I've already talked about some times ago, as said, those peoples had a fundamentally simple system, but it was based on a bunch of stuff that were a) obsolete, b) unmaintained and c) undocumented.
The first draft of their "update" plan did not considered small details like "testing" and "informing the users", and it was more on the way of "toss everything there, turn it on and hope for the best". Of course they immediately met the aforementioned law, and after a very quick withdrawal decided that, MAYBE they needed a better plan.
One of the problems, was the (in)famous "billing", that, in human terms, means "getting paid".
If you remember the story, they were managing a B2B messaging system. Now, turn it around as you like, but this is basically a glorified mail server that send and receive mail from and to different machines. So what is the billing? Well, how many messages are sent, received, kept and so on. At the end of the month, all those data had to be summed up and voila', the amount of money the customers need to pay.
It sounds easy and simple, but obviously, when you add an UL (or ten) to the mixture, it gets nothing but. In specific, the "billing system" looked like had been designed by some very bored Torquemada.
First of all, a bunch of scripts collected all the logs from the various mail servers and did some sort of pre-processing, building and feeding data into not one but 3 databases, then another bunch of scripts computed the total amount of messages in the mailboxes dividing them in "groups" by the size (less than 1Kb, between 1 and 2 Kb...). No, I don't know why and has no sense. Anyhow, after all this, at the end of the month another bunch of scripts proceeded to analyze the database and produced a bunch of .csv files that then were re-re-processed (again) to build "reports" that had to be sent to each customers and then another gigantic .csv was produced and sent to $companyX so they could send out the real invoices.
But wait! It gets better!
Because, not happy enough with all this, $companyX, in the person of the "customer manager", had decided that the best thing to do was to keep changing things around, so every month a bunch of manual changes had to be applied to the various reports. Things like "this customer this month has a fixed fee of X$", so change the report, toss away all the content and just write down the number. Or "this customer this month has an extra discount of X%", so, again, change the report to add the discount. And of course the changes were different each month, so no sense in trying to automate them, and you had to change the reports, the CSV files (but not all of them... no, I don't know why)... EVERY FUCKING MONTH!
And since these were manual changes, the chance of making a mess was very high.
So, when the "upgrade" thing popped up, I thought they were going also to re-do the whole billing thing in a better way, incorporating all these changes at once.
Yeah, I'm not young anymore but I'm still a dumbass.
$companyX immediately reported that "there is no budget to renew the billing process, so it is absolutely necessary that all this will stay intact and functioning in the new environment".
Me - Hold on... you want to change the whole infrastructure, especially the part that produce the source DATA of this billing process, but the process in itself has to stay the same? And how do you plan to obtain that?
UL - Well, the billing process uses the data from the database, so if the database stay the same it shouldn't even otice.
Me - Yeah, right. The only way to keep the database the same is to understand what all those crappy scripts do and then see when can we get the same data, assuming we get the same data.
UL - No, we don't look at the crappy scripts.
Me - ...say again...
UL - There is no budget to look at the script.
Me - ...so?
UL - So we only check the database.
Me - ...so you want to try to figure out how to fill in the database without looking at the procedure that does that?
UL - Exactly!
Me - ... and how do you plan to test if it does it correctly?
UL - Well, if the old and the new database are the same is correct.
Me - The only way to compare the new and the old is if you can feed the same data to both system, that is difficult to do since only one of the environment can be active at one time.
Yes, because you either process the messages in the old environment or in the new environment, if you try to do both, you'll end up with duplicated data somewhere.
Anyhow, a software house was "hired" to produce this "feeding procedure". After several MONTHS of work, they had produced something that was supposed to read the logs of the new environment and fill the old database with the same info as the old one. Of course, since the new environment wasn't in use, the "test" that could be done were... limited.
Now, I repeat, we're talking about messages here. Mail. That can be "weird" if you want but how much does is take to build a mail system? Well, it seems that the required time is way beyond one year. Or better, the whole infrastructure was completed and "functional" in about a month, but there were some "problems" left over. One was the fucking billing of course, because it couldn't be tested in any functional way, and the second is connected to the first. That is, whoever had built this fucking "billing" process had decided that cron wasn't enough. So they had their own "enhanced" version of cron.
Now, having very little to do, while waiting that somebody decided how to proceed with this thing, I decided to have a look at that process. I quickly realized that the simplest way to test the whole thing was to get the logs from the old system and "translate" them into something that looks like a Postfix log, then have the scripts made by the developers to ingest them and see what pops out the other side.
After some messing around (and a lot of cursing) I had my "pseudo-log" so I could start the "real" deal.
The scripts built by the developers run... didn't report any error... but didn't add anything to the database either.
Ok, I'll call this "a failure".
I reported this to UL that looked a bit irritated.
UL - What do you mean "they don't work"?
Me - As said: they do not report any error but they do not add any data.
UL - And what did the developers said about this?
Me - To me nothing, you're the one that is keeping in touch with those peoples.
Yes, because UL managed to be the only line of communication with them, so everything had to go through him.
After a bit I get a few mail in CC with some discussion about how the software should work and the fact that it doesn't. The discussions goes on for a while and my line remains as "look at the original log and to the database that is produced". And yes, I do realize that looking at the original script could be a lot easier but UL is still of the idea that is not something we need to do.
Anyhow, we reach the point when $softwarehouse decide that what we need is a good ANALYSIS of the procedure (that makes sens IMHO), while, according to them, they have been asked to simply WRITE the code and the analysis should've been done by us. And if we now want an analysis, that is a different task and they want more money (this also makes sense IMHO).
Now, since all the contacts with those peoples have been kept by UL, I simply look at the thing from far away and wave, especially during the various meeting when we keep repeating that no, the procedure still doesn't work.
UL - Did you spoke with the developers?
Me - To do what? To repeat that the procedure doesn't work? They know about it.
UL - But they should've done the analysis, I was very clear about it when we began ... hemm... six months ago.
Me - (tending my hand) Fantastic, let's see the paper.
UL - ...what paper?
Me - Those peoples have been contacted by you to do some sort of work, there should be a piece of paper, normally referred as 'contract', that specify what kind of work, for how long, and how and it should be signed by both party. If that piece of paper say they were supposed to deliver an analysis, then why were they writing code? Why did you accepted the code and even requested that it was written following some sort of "format"? So let's see this contract.
UL - But I told them...
Me - Did you also WROTE IT DOWN?
UL - Wrote?
Me - Yes. Onto that famous piece of paper I just explain about.
UL - ...I'm not sure...
Me - You didn't wrote shit, didn't you?
UL - But I told them...
Me - And they are telling something else, and since they are 2 and you one, there is very little to discuss about. And switching argument, how about that "scheduler"?
UL - What?
Me - That sort-of-cron-that-isn't-cron that is required for the whole thing to work.
UL - Ah, yes, we have the source code.
Me - Yes, you told me this six months ago. Where is this source code?
UL - Hemmm... I'm not sure...
Me - And did you try to compile those sources? On something that isn't Solaris 8? On something modern? Like, the past 5 years? And does it compile? What libraries does it need to compile? And once compiled, does it work? That is, does it do what it is supposed to do insead of something else, like deleting sectors from the disk at random?
UL - Hmmm...
Me - Because since we have some time while we wait maybe we should start looking at this instead of twiddling our fingers.
UL - Yeah... I wanted to do some test but... I forgot...
Me - To be somebody that has always a pen in his hand you write very little down eh?
Something make me think that this "project" will be a very long one indeed.
Comments are added when and more important if I have the time to review them and after removing Spam, Crap, Phishing and the like. So don't hold your breath. And if your comment doesn't appear, is probably becuase it wasn't worth it.
Si, nonostante non sia piu' giovane continuo ed essere un coglione.
Benvenuto nel club! Il bagno è in fondo a destra e la sala riunioni dedicata all'autoflagellazione è questa, devo prenotare almeno 24 ore prima nel planning online e pulire le macchie di sangue quando hai finito. Mi raccomando la mascherina. Vedrai che ti troverai bene, siamo un grande gruppo!
By Davide Bianchi - posted 14/09/2020 10:17
Benvenuto nel club!
...forse non hai notato, ma io sono il Presidente...
By Messer Franz - posted 14/09/2020 12:01
Sarò io che tendo ad esagerare le cose, ma nell'ultima riunione descritta l'adrenalina scorreva a fiumi... e UL era concentratissimo a pensare come non averti più davanti senza qualcuno su cui scaricare la colpa dei suoi errori mentre lui si confonde con la parete alle sue spalle...
ps: quanto manca alla pensione? Perchè se quando cambi lavoro (tipo l'immortale fuga da Branco Di Paguri) sei... "rilassato" nella gestione dei rapporti gerarchici, io voglio vedere come ti comporti quando stai per andare in pensione...
By Anonymous coward - posted 15/09/2020 16:55
Si, nonostante non sia piu' giovane continuo ed essere un coglione.
Ecco, io non lo volevo dire esplicitamente, ma...
By gabrielAnonymous coward - posted 15/09/2020 18:21
Cervello_di_ul.exe: Impossibile eseguire il programma.
Windows: "il tizio è troppo deficente e non trovo una soluzione online sul sito della microsoft, errore 666"
notare il numero del diavolo.
Ma veramente, ma come puoi far fare un lavoro ha dei programmatori senza fare neanche un contratto?
non c'è limite alla deficenza.
This site is made by me with blood, sweat and gunpowder, if you want to republish or redistribute any part of it, please drop me (or the author of the article if is not me) a mail.
This site was composed with VIM, now is composed with VIM and the (in)famous CMS FdT.
This site isn't optimized for vision with any specific browser, nor
it requires special fonts or resolution.
You're free to see it as you wish.