Tales from the Machine Room

Home Page | Comments | Articles | Faq | Documents | Search | Archive | Tales from the Machine Room | Contribute | Set language to:en it | Login/Register

Silence! We shouldn't disturb the users...

Silence is golde, they said. But is not really clear why in fact, sure, annoying everybody at every hour day or night isn't really a good idea, but sometimes, TALKING is a good thing.

Especially when you have important and useful information to share. Like somebody is towing your car away for example or the fucking building is on fire and you should really start walking towards the nearest exit.

But there are other situations when not only nobody talks, but somebody even actively try to prevent the exchange of any kind of information, even when the result is clearly wrong.

And after this quick and informational introduction, we talk about $screamandyell.

These peoples had decided that their mailing list management system was a tad old and required waaaay too much maintenance, therefore, having listened too much to MarketingMan talk about this magical thing called "the cloud", that always works, it never fails, is invisible, omnipresent and basically Jesus Christ, they began salivating like a bunch of famished lions looking at a fat tourist.

Before you could say "hold on..." they had already signed to move the whole thing onto some jimagathingy installed in AWS. Why AWS? WhathefuckdoIknow.

No, seriously, there was no reason to have this stuff in AWS, it was there because it was there.

Anyhow, after several days spent trying to figure out how all this crap had to be installed and how it worked, we ended up with 4 mail servers equipped with Mailnan with a PostgreSQL backend to manage all the lists.

I won't talk about the problems to "migrate" all the lists from the old system (home-grown) into the new one without fucking it up completely and other stuff.

And after all this, we're at the point to decide how to put this thing into production. And at this point the shit hit the spinning paddle.

The problem is that, to put everything in production we need to distribute new passowrd to the various "maintainers" and "moderators" of the various mailing lists, and tell them about the new interface and the like. Moreover, between the moment the old system goes into retirement and the new one is fully functional there will be some delay, at minimum, the DNS need to update.

And this is the problem.

It seems that the UL that is at the helm of the thing is convinced that the best way to do this kind of operation is without communicating anything to anyone, all hush-hush and in secret. Ironic, considering the company is specialized in mass-communication.

So his idea would be to do the switch in the night and not say anything at all.

I interjected that the new passwords need to be communicate, and the new management interface needs to be opened up to the various maintaners etc. and there will probably be question about the various procedures and the like.

UL - Noooooo!!! We shouldn't say anything to the users, otherwise they begin to whine and if there are problems they will never finish it.
Me - ...if there are problems maybe they have their reasons, and this is why we should ask them to do some TEST on the new environment before going into production. So they see how it works and we also do some real testing.
UL - NoNoNoNoNoNo. What we need to do is to put it into service quietly, without disturbing anyone, nobody has to notice anything.
Me - That is impossible to do, since the whole infrastructure is completely different, required several changes on the system and the various mailing list and when it goes into service there will be some deadtime.
UL - We need to just put it into service, who uses the interface will figure out that it is changed and when they can't login they will request a new password to the help-desk, who doesn't will never notice.
Me - I repeat that the best thing to do is to contact ALL the users, because they aren't idiots (ok, not that much) and they will eventually figure out, and if there are problems at the start is better if they know.

The discussion (let's call it like this), went on for a while, then UL decided to go in production at 7pm without informing anyone.

OFCOURSE, since nobody had done any real life test (the kind that involve the customers) nobody ever discovered that some customers had automated processing for the messages enabled, and now those messages were getting rejected because the message format had changed. The there were problems because lots of customers were using the e-mail version of the management interface and that was completely different.
It also caused an effect of "multiplication of e-mails" that had ... interesting effects.

The final result was that after several hours at the receiving end of screaming and yelling, UL had to order a withdrawl and put back the old system, grab all the "information" received and go back to the drawing board.

The most interesting part (if you can call it like that) is that the majoirity of the comments was "if you had told us..."

But no, we can't disturb the users. Not even when you're about to pull the carpet from under their feet.

16/06/2020 11:36

Previous Next

Comments are added when and more important if I have the time to review them and after removing Spam, Crap, Phishing and the like. So don't hold your breath. And if your comment doesn't appear, is probably becuase it wasn't worth it.

9 messages this document does not accept new posts

Anonymous coward

By Anonymous coward posted 06/07/2020 09:18

Sbaglio, o manca la parte dove UL riceve il bonus e viene promosso? :D

-- Anonymous coward

Davide Bianchi

@ Anonymous coward By Davide Bianchi posted 06/07/2020 11:06

Sbaglio, o manca la parte dove UL riceve il bonus e viene promosso? :D

Quella viene dopo ovviamente. Con lodi per la migrazione compiuta con successo.


-- Davide Bianchi

Messer Franz

@ Davide Bianchi By Messer Franz posted 06/07/2020 19:43


Sbaglio, o manca la parte dove UL riceve il bonus e viene promosso? :D

Quella viene dopo ovviamente. Con lodi per la migrazione compiuta con successo.

>Nonostante DB remasse contro...non dimenticarlo!

Povero UL, quanti scogli sulla sua strada ! (citazione talmente vecchia che non la capirà nessuno, ma volevo farla...)



-- Messer Franz

Anonymous coward

@ Anonymous coward By Anonymous coward posted 27/07/2020 00:31

>Sbaglio, o manca la parte dove UL riceve il bonus e viene promosso? :D

Stavo per dire "machecazzostaidicendo" ma poi mi sono ricordato dove lavoravo e.. Ah gia, dove e' la parte di promozione + bonus + ufficionuovoalpianodeidirigenti + autoaziendale?



-- Anonymous coward


By Osiride posted 06/07/2020 10:29

questo typo "[...] comunicazioni di messa [...]", ammesso che sia un typo, mi ha fatto molto ridere XD

-- Osiride


By Thomas posted 06/07/2020 14:40

Un po' lo capisco: quando hai a che fare col management (aka Coloro Che Tutto Possono Ma Nulla Pensano) la strategia "meglio chiedere perdono che permesso" spesso e' la migliore.

Purche' pero' dopo i cambiamenti tutto funzioni. E soprattutto, purche' la strategia venga rivolta solo verso il management, non verso gli utenti (aka Coloro Che Fanno Le Cose).

Altrimenti sei un idiota e meriti l'orda inferocita.

-- Thomas


By gabriel posted 07/07/2020 03:20

una sola parola...., idiota!

ul, non d ovviamente!

ma come si può solamente pensare o far passare dall'anticamera del cervello una cosa del genere.

d, poi come è finita la storia?

hanno messo in produzione il coso, avvertendo gli utenti spero o utilizzano ancora il vecchio sistema?

-- gabriel

Anonymous coward

By Anonymous coward posted 08/07/2020 09:46

Il classico mantra "deve essere trasparente per l'utente, sono sempre stati abituati così e devono cambiare meno possibile". Ah, e "il servizio deve sempre essere garantito".

-- Anonymous coward


By trekfan1 posted 09/08/2020 19:50

Spero che nessuno lo avverta prima che apprenda la notizia di essere stato "licenziato"

-- trekfan1

9 messages this document does not accept new posts

Previous Next

This site is made by me with blood, sweat and gunpowder, if you want to republish or redistribute any part of it, please drop me (or the author of the article if is not me) a mail.

This site was composed with VIM, now is composed with VIM and the (in)famous CMS FdT.

This site isn't optimized for vision with any specific browser, nor it requires special fonts or resolution.
You're free to see it as you wish.

Web Interoperability Pleadge Support This Project
Powered By Gojira