The Hammer
When the only tool you have is an hammer, everything is a nail.
You probably have heard this sentence some times already, it refers to the attitude of some peoples that have some specific knowledge or a preferred "tool", and they tend to apply that knowledge or tool always and everywhere, no matter what. Even when the tool or the knowledge has absolutely nothing to do with the situation at hand.
The problem with this kind of one-size-fit-all approach is obvious: sometimes that way is the worst possible way.
And now we're gonna talk about $processesandso, a company that does... different things.
Their main tools were Mule and a bunch of script and "applications" custom mades in python and php. Over a Postgre database. Nothing exciting then. The major problem of these people was that they were trying to solve all the problems with a "quick and dirty" approach. That works 99% of the time but is not really an approach that is durable or produces great results in the long run.
And here comes THE problem. One of their customers had some "processes" that were ... doing stuff... and send mails. The problem was that sometimes those mails weren't delivered correctly. Mostly because the addresses were fumbled and sometimes because the process went sideways.
And the major problem wasn't that the thing couldn't send mail (I'm always in favour of things that DO NOT send mail, less spam to treat) but the "queue" that was the source for the e-mails was getting bigger and bigger and after a while the process that handled it was also getting stuck and then, like a domino effect, the whole shebang was going to be rebooted.
After a number of nightly and daily alarm (because that thing tended to crash in a very random way), I ended up discussing the issues with one of the developers (CL) to try to figure out what the problem actually was and to put it to rest.
Me - ...so if that thing breaks there should be a way to notice it before something else start to break.
CL - And we agree on that, the problem is that we don't really know how to do it.
Me - ...what do you mean you don't know? Didn't you wrote that thing?
CL - Well, yes, but is not that simple.
Me - Hmmm... that thing read a 'queue' and send a mail, then read the next entry. Where is the problem in understanding if it's broken or not?
CL - Eh... that each run it extract an entry from the queue, but we aren't sure if the extraction has generated a mail or not.
Me - ...what do you mean "each run"?
CL - Yes, each run.
Me - Why? Does it run sometimes?
CL - Every minute, basically all the time.
Me - ...every minute? What do you mean?
CL - It's in cron.
Me - ...cron... why cron?
At this point I started asking difficult question like who the fuck decided to have that piece of shit run from cron every minute?
If you aren't used to the Unixverse, probably you have no idea about what Cron is. Cron is a watch, basically, what it does is, every minute it checks a table of "operations to be performed" if there is one operation to be executed at that time (hour, minute, day of the week) and if it is, it runs it.
Is an easy and simple way to run "one shot" things that needs to run every now and then.
The problem is when somebody decide to abuse or misuse it. For example, what is the bloody point in having an operation run every minute? An operation should be run once. If it has to run continuosly, then it's not a "one shot" thing, it is (or should be) a DAEMON, a process that keeps running forever.
But building a daemon is not the same as building a one-shot thing, everybody with a minimun (or none) of knowledge can build a one-shot script, but running a one-shot all the time, can cause problems.
And I understand that misusing cron is a simple thing to do, I fell into the same trap sometimes ago and Karma the crocodile came back to bite my ass, so everything as expected.
Me - ...and that's why it would be a lot better to make a daemon instead of using cron.
CL - A daemon? And how the hell do we make a daemon? That's way to difficult.
Me - Difficult? Are you developers or not?
CL - But how do you make a daemon?
Me - You call fork in the code.
CL - ... and how do you do that in Python?
Now, I'm partial, but I can't stand Python. I like to add spaces, tabs and empty lines without fucking up the logic of the whole script, that's why I use either bash or perl.
Me - I don't know in Python, I'm almost sure that it can be done, but I've no idea how. If you want I can send an example in perl.
CL - No, we don't use perl...
Me - Wathever, but you have to fix this thing because this way is unmanageable.
A couple of days later, we're back to the deathbed of that thing, and the diagnosis is that the various CL haven't followed my suggestion (whatdoyathink) and instead added MANY MORE scripts in cron that runs every minute, and every now and then they run into each other.
CL - ...so when the second script start but the first one hasn't finished...
Me - I told you already, that thing is fucked inside, you have to make a daemon, then is going to run whenever he wants and nobody is gonna bother him.
CL - But that's not they way we work.
Me - That's not my problem. You're asking for a solution and that is a solution.
CL - There is also another problem.
Me - Really?
CL - Yes, because we also added this script that runs each minute in cron and get the last 300 lines from the database and...
Me - And it also got stuck, right?
CL - eh...
Yes, because when you're used to hit everything with an hammer, when you get a rubber tire and the hammer bounces against your face isn't easy to accept.
Davide
31/08/2018 15:56
