Silence! We shouldn't disturb the users...
Silence is golde, they said. But is not really clear why in fact, sure, annoying everybody at every hour day or night isn't really a good idea, but sometimes, TALKING is a good thing.
Especially when you have important and useful information to share. Like somebody is towing your car away for example or the fucking building is on fire and you should really start walking towards the nearest exit.
But there are other situations when not only nobody talks, but somebody even actively try to prevent the exchange of any kind of information, even when the result is clearly wrong.
And after this quick and informational introduction, we talk about $screamandyell.
These peoples had decided that their mailing list management system was a tad old and required waaaay too much maintenance, therefore, having listened too much to MarketingMan talk about this magical thing called "the cloud", that always works, it never fails, is invisible, omnipresent and basically Jesus Christ, they began salivating like a bunch of famished lions looking at a fat tourist.
Before you could say "hold on..." they had already signed to move the whole thing onto some jimagathingy installed in AWS. Why AWS? WhathefuckdoIknow.
No, seriously, there was no reason to have this stuff in AWS, it was there because it was there.
Anyhow, after several days spent trying to figure out how all this crap had to be installed and how it worked, we ended up with 4 mail servers equipped with Mailnan with a PostgreSQL backend to manage all the lists.
I won't talk about the problems to "migrate" all the lists from the old system (home-grown) into the new one without fucking it up completely and other stuff.
And after all this, we're at the point to decide how to put this thing into production. And at this point the shit hit the spinning paddle.
The problem is that, to put everything in production we need to distribute new passowrd to the various "maintainers" and "moderators" of the various mailing lists, and tell them about the new interface and the like. Moreover, between the moment the old system goes into retirement and the new one is fully functional there will be some delay, at minimum, the DNS need to update.
And this is the problem.
It seems that the UL that is at the helm of the thing is convinced that the best way to do this kind of operation is without communicating anything to anyone, all hush-hush and in secret. Ironic, considering the company is specialized in mass-communication.
So his idea would be to do the switch in the night and not say anything at all.
I interjected that the new passwords need to be communicate, and the new management interface needs to be opened up to the various maintaners etc. and there will probably be question about the various procedures and the like.
UL - Noooooo!!! We shouldn't say anything to the users, otherwise they begin to whine and if there are problems they will never finish it.
Me - ...if there are problems maybe they have their reasons, and this is why we should ask them to do some TEST on the new environment before going into production. So they see how it works and we also do some real testing.
UL - NoNoNoNoNoNo. What we need to do is to put it into service quietly, without disturbing anyone, nobody has to notice anything.
Me - That is impossible to do, since the whole infrastructure is completely different, required several changes on the system and the various mailing list and when it goes into service there will be some deadtime.
UL - We need to just put it into service, who uses the interface will figure out that it is changed and when they can't login they will request a new password to the help-desk, who doesn't will never notice.
Me - I repeat that the best thing to do is to contact ALL the users, because they aren't idiots (ok, not that much) and they will eventually figure out, and if there are problems at the start is better if they know.
The discussion (let's call it like this), went on for a while, then UL decided to go in production at 7pm without informing anyone.
OFCOURSE, since nobody had done any real life test (the kind that involve the customers) nobody ever discovered that some customers had automated processing for the messages enabled, and now those messages were getting rejected because the message format had changed. The there were problems because lots of customers were using the e-mail version of the management interface and that was completely different.
It also caused an effect of "multiplication of e-mails" that had ... interesting effects.
The final result was that after several hours at the receiving end of screaming and yelling, UL had to order a withdrawl and put back the old system, grab all the "information" received and go back to the drawing board.
The most interesting part (if you can call it like that) is that the majoirity of the comments was "if you had told us..."
But no, we can't disturb the users. Not even when you're about to pull the carpet from under their feet.
Davide
16/06/2020 11:36
