Tales from the Machine Room


Home Page | Comments | Articles | Faq | Documents | Search | Archive | Tales from the Machine Room | Contribute | Login/Register

The Hammer

When the only tool you have is an hammer, everything is a nail.

You probably have heard this sentence some times already, it refers to the attitude of some peoples that have some specific knowledge or a preferred "tool", and they tend to apply that knowledge or tool always and everywhere, no matter what. Even when the tool or the knowledge has absolutely nothing to do with the situation at hand.

The problem with this kind of one-size-fit-all approach is obvious: sometimes that way is the worst possible way.

And now we're gonna talk about $processesandso, a company that does... different things.

Their main tools were Mule and a bunch of script and "applications" custom mades in python and php. Over a Postgre database. Nothing exciting then. The major problem of these people was that they were trying to solve all the problems with a "quick and dirty" approach. That works 99% of the time but is not really an approach that is durable or produces great results in the long run.

And here comes THE problem. One of their customers had some "processes" that were ... doing stuff... and send mails. The problem was that sometimes those mails weren't delivered correctly. Mostly because the addresses were fumbled and sometimes because the process went sideways.

And the major problem wasn't that the thing couldn't send mail (I'm always in favour of things that DO NOT send mail, less spam to treat) but the "queue" that was the source for the e-mails was getting bigger and bigger and after a while the process that handled it was also getting stuck and then, like a domino effect, the whole shebang was going to be rebooted.

After a number of nightly and daily alarm (because that thing tended to crash in a very random way), I ended up discussing the issues with one of the developers (CL) to try to figure out what the problem actually was and to put it to rest.

Me - ...so if that thing breaks there should be a way to notice it before something else start to break.
CL - And we agree on that, the problem is that we don't really know  how to do it.
Me - ...what do you mean you don't know? Didn't you wrote that thing?
CL - Well, yes, but is not that simple.
Me - Hmmm... that thing read a 'queue' and send a mail, then read the next entry. Where is the problem in understanding if it's broken or not?
CL - Eh... that each run it extract an entry from the queue, but we aren't sure if the extraction has generated a mail or not.
Me - ...what do you mean "each run"?
CL - Yes, each run.
Me - Why? Does it run sometimes?
CL - Every minute, basically all the time.
Me - ...every minute? What do you mean?
CL - It's in cron.
Me - ...cron... why cron?

At this point I started asking difficult question like who the fuck decided to have that piece of shit run from cron every minute?

If you aren't used to the Unixverse, probably you have no idea about what Cron is. Cron is a watch, basically, what it does is, every minute it checks a table of "operations to be performed" if there is one operation to be executed at that time (hour, minute, day of the week) and if it is, it runs it.

Is an easy and simple way to run "one shot" things that needs to run every now and then.

The problem is when somebody decide to abuse or misuse it. For example, what is the bloody point in having an operation run every minute? An operation should be run once. If it has to run continuosly, then it's not a "one shot" thing, it is (or should be) a DAEMON, a process that keeps running forever.

But building a daemon is not the same as building a one-shot thing, everybody with a minimun (or none) of knowledge can build a one-shot script, but running a one-shot all the time, can cause problems.

And I understand that misusing cron is a simple thing to do, I fell into the same trap sometimes ago and Karma the crocodile came back to bite my ass, so everything as expected.

Me - ...and that's why it would be a lot better to make a daemon instead of using cron.
CL - A daemon? And how the hell do we make a daemon? That's way to difficult.
Me - Difficult? Are you developers or not?
CL - But how do you make a daemon?
Me - You call fork in the code.
CL - ... and how do you do that in Python?

Now, I'm partial, but I can't stand Python. I like to add spaces, tabs and empty lines without fucking up the logic of the whole script, that's why I use either bash or perl.

Me - I don't know in Python, I'm almost sure that it can be done, but I've no idea how. If you want I can send an example in perl.
CL - No, we don't use perl...
Me - Wathever, but you have to fix this thing because this way is unmanageable.

A couple of days later, we're back to the deathbed of that thing, and the diagnosis is that the various CL haven't followed my suggestion (whatdoyathink) and instead added MANY MORE scripts in cron that runs every minute, and every now and then they run into each other.

CL - ...so when the second script start but the first one hasn't finished...
Me - I told you already, that thing is fucked inside, you have to make a daemon, then is going to run whenever he wants and nobody is gonna bother him.
CL - But that's not they way we work.
Me - That's not my problem. You're asking for a solution and that is a solution.
CL - There is also another problem.
Me - Really?
CL - Yes, because we also added this script that runs each minute in cron and get the last 300 lines from the database and...
Me - And it also got stuck, right?
CL - eh...

Yes, because when you're used to hit everything with an hammer, when you get a rubber tire and the hammer bounces against your face isn't easy to accept.

Davide
31/08/2018 15:56

Previous Next

Comments are added when and more important if I have the time to review them and after removing Spam, Crap, Phishing and the like. So don't hold your breath. And if your comment doesn't appear, is probably becuase it wasn't worth it.

7 messages this document does not accept new posts
Thor By Thor - posted 24/09/2018 09:21

Chiamami quando vuoi: sono pronto a usare il mio fedele Mjollnir come strumento omnirisolutivo (tipo, per esempio, per risolvere il problema dei CL piantandoli nel pavimento come chiodi).

:D

--
Thor


Guido By Guido - posted 24/09/2018 12:16

'spetta non sono sicuro di aver capito bene: tu hai rilevato che il problema era uno script che girava ogni minuto e gli hai consigliato di "demonizzarlo" - loro come applicazione della tua soluzione hanno creato altri script che girano ogni minuto?

Geniale...

--
who uses Debian learns Debian but who uses Slackware learns Linux


Riccardo Cagnasso By Riccardo Cagnasso - posted 24/09/2018 17:32

Beh ma al giorno d'oggi ha senso fare un demone nel senso di una roba che fa doppio fork? In fondo systemd gestisce come demone praticamente qualunque cosa. Ovviamente per fare task periodiche o cose del genere ci sono tool come Celery su python, ma se devi semplicemente leggere da una coda...

--
Riccardo Cagnasso


Akart72 By Akart72 - posted 25/09/2018 08:18

Idiocracy era un film profetico

--
Akart72


Lazza By Lazza - posted 26/09/2018 01:04

Personalmente uso script in Python presso clienti e funzionano bene, chiaro che bisogna saperli scrivere e purtroppo la facilità di Python attira anche programmatori poco avvezzi. Ad ogni modo con systemd il loro bel daemon che esegue a ripetizione lo farebbero con uno unit file di 3 righe.

Spero che questo commento non venga letto da qualche Devuanista. ;\)

--
Lazza


Jenson@ Lazza By Jenson - posted 26/09/2018 10:18

Spero che questo commento non venga letto da qualche Devuanista. ;\)

Mai capito tutto il bric-à-brac su systemd. Fa il suo lavoro come qualunque init di sistema, no? Ma l'hanno attaccato da tutte le parti con le motivazioni più assurde, tipo il fatto che il nome "non segue le Sacre Linee Guida della Santissima Bibbia della Nomenclatura di Linux" (non in termini così espliciti, ma il senso di profanazione dei testi sacri si intuisce). Ma che cazzo.

--
Jenson


Messer Franz By Messer Franz - posted 27/09/2018 16:48

Si potrebbe fare un demone che gira ogni 30 secondi e controlla se il cron ha croneggiato, e poi un altro cron per controllare se il demone ha funzionato e così via...se lo fai abbastanza incasinato poi ti assumono alla microsoft che lo propone come nuovo sistema operativo...

--
Messer Franz


7 messages this document does not accept new posts

Previous Next


This site is made by me with blood, sweat and gunpowder, if you want to republish or redistribute any part of it, please drop me (or the author of the article if is not me) a mail.


This site was composed with VIM, now is composed with VIM and the (in)famous CMS FdT.

This site isn't optimized for vision with any specific browser, nor it requires special fonts or resolution.
You're free to see it as you wish.

Web Interoperability Pleadge Support This Project
Powered By Gojira