Hack Hack Hack Hack Hack Hack
What is an "hack"? Is a quick and dirty solution, more often than not put together without too much thinking, to solve or circumvent an occasional problem. Usually, you "hack" together something when you encounter a problem and you don't have the time, the budget or the requirements to search and implement a better, more "permanent" solution.
When you end up with a flat tire, you slap a patch on it and go. And then, later, with time, you replace the tire or fix it for real. But everyone knows that the patched tire remains until it fails.
And it usually works. Those are also known as "temporary solutions". And let's ignore the fact that most of the time those are "forever solution". I've already said multiple time what I think about it but I also understand that sometimes there aren't many other solutions. A "forever" solution is either impossible or really really expensive, so... you leave the hack and go on with your life.
The problem is that, sometimes, or better, always, hacks... aren't perfect... they... Leaks. And you end continously pumping your tire. And then the patch fails catastrofically. Usually at night, on a very steep descent, with wet road.
And after this hacked up introduction, we go back talking about UL and his "relay" of which I already told sometimes ago.
And now we need to know some detail.
A looooooong time ago, somebody thought that it could be useful to send mail messages in a different way. Because, apparently, nobody has anything better to do than re-invent the wheel every freaking day. So, instead of using something that already exists, is a standard and works quite well, like SMTP for example, they wrote a new spec to use (drum roll) HTTP. Yes. Why? Fuck if I know. The thing became yet another standard, that since ain't compatible with anything else, is normally used in that niche that is 'B2B', so it's machine talking to other machines.
The basic difference between this thing (called AS2) and SMTP, is that AS2 use encryption all the way and as such it only works if the 2 "partner" (the one that send a message and the one that receive it) have exchanged certificates to be able to read each other's messages.
And if you're going to ask how is this different from a normal mail encrypted with Pgp, the answer is "none".
Anyhow, the idea is that you can send messages using this thing. The problem is that the "protocol" only talks about sending or receiving messages, it has no provision about "creating" or "storing" messages. That is, there is no "client" defined or "message store".
Now, the thing that UL had built, was basically a bridge between AS2 and normal SMTP/IMAP. It was based on an open source (and no longer maintained) implementation of the AS2 part, and then the part that had been developed that toke the messages from an IMAP mailbox, reformatted them and then fed them to AS2, or toke AS2 messages, reformatted them and then fed them to Postfix to send them 'the normal' way.
At least, this was the idea.
The problem was that, as explained some times ago, the "development" was done in a not-very-professional way and the end result was lacking some fundamental characteristics, like "being able to do its job" for example.
But, since there was a customer (or something like that), there was an interest in having this thing to work. So UL got the various "suggestions" that I sent him and, after boasting "years of development experience", he decided to tackle the problem himself.
After a couple of week of sweath and hard work (or so he said), he showed up the "final product" and he personally reassured the customer that all was well and truly fixed. I had my doubt.
After a month, I found a (not so) nice mail in my mailbox. Apparently the "customer" was asking info about a number of messages that were supposed to have been sent but were nevere received. Ok, is time to put the 'debug' hat on and dive into this thing... again.
The open source part is the only bit that I can kinda trust, also because there is a log that tells me that message so-and-so has been received. The problem is after this.
The message is received and the open source bit write it as a file on disk, then the whole contraption takes over, and it should read this message, figure out what to do with it and do it.
UL (or the original developers) decided to split the thing in 6.
Phase 1: there is a script that every 5 minutes check if there are new messages in the directory, it expect a message as a file with a specific name. If there is one, he picks it up, create an "header" with source, destinationa and some other stuff, and then stuff it into a queue.
Problem number 1: the file name that the script expects has a specific format, but the open source bit that create the file uses some bits of the message content as name of the file and as such it that can be overwritten by the message sender. So the system can fail right at the start.
Phase 2: once queued, the messagre is picked up by a second script that check the header and, depending on the sender/recipient combo, move it to a different queue.
Problem number 2: the sender/recipient combo is case-sensitive, but the list is read from a file and is not checked for consistency, so if one of the name does not match exactly, the message remain unprocessed in the queue and none is the wiser.
Phase 3: at this point a message is in the 'destination' queue, and, yes, ANOTHER script picks it up. Actually, two scripts, one for smtp and one for imap. They looks waaaaay too similar to be a coincidence. Each script picks the message from its queue and then move it to its destination.
Problem number 3: there is no control over the existence of the mailbox or the final address. If the mailbox or the address are non-existant, the message is removed from the queue and then disappear in the void.
Problem number 4: when the message is received by AS2, an ID is created, but when the first script read the message and put it into the queue, the ID is ignored, so there is no longer a connection between which message has been read and which one as been processed.
Problem number 5: not one, I say, NOT ONE of those fucking scripts checks for any errors whatsoever! If something goes wrong the options are: the script crashes or the message is left somewhere half-processed or both.
Problem number 6 (corollary of the one before): none of the script does any kind of decent "logging". So there is no way to follow a message along the chain, I can see from the log that 20 messages have been received from AS2, I can see 10 messages posted by postfix. Which 10 are those? What happened to the other 10? Who knows!
After several hours lost into this mess I also discover other problems, that shows up only when you start looking at things closer. For example, if a messages has a file name that isn't what the script in Phase 1 expects, the script doesn't pick it up, but it doesn't do anything also. So you can end up with thousands of messages left in the directory, and the script gets slower and slower because it has to check all the files in there.
Also, the script in phase 3 pick a message from the queue and create files on disk. This file has a name that is a combination of the sender and the timestamp. This means that more than one message per second in sent by the same sender, files can be overwritten and then "lost" without a trace.
As said before, there is no error controls, so messages could be removed from the queue and then the script could crash, again vanishing the message.
If the message cannot be processed, it will be left in the queue, that means that thousands of messages could be left in the queue and, again, the script will spend long time checking and failing every message.
Or the message can be read, having a wrong address or mailbox destination, be removed from the queue and then trashed.
And, as said before, nothing logs anything! So how do we know which messages have been processed? How do we know what happend to that message?
After spending a good day and an half looking at this thing and remembering why I hate Python (and UL), I came out with a list of bugs and improvement that should be implemented in this thing.
I've also reached the conclusion that it is much better if our "customer" realize that re-sending the missing messages is the best way to proceed, AFTER we've implemented all the fixes of course.
At this point however, the "customer" was cursing loudly, that I could understand and relate with, in principle, but not really. Because nowhere, in the contract that he DIDN'T signed there was any mention that we had any responsibility about the good processing of any messages. And this is also what I pointed out in the discussion that followed.
I also pointed out however, that this entire thing, at the state it was, couldn't be considered a "production" product. It was an Hack. Something that kinda work if it is used always in "baby sitter mode", but it is incapable of stable functioning for a while on its own. It doesn't provide any information for debugging and is definitively not ready for production, it doesn't matter how many beers you had the night before.
And UL? Well, he was claiming that since the system is now production, he doesn't want to touch the code (that doesn't work) if there isn't a development environment. Obviously.
Davide
13/11/2020 12:56
