Tales from the Machine Room

Home Page | Comments | Articles | Faq | Documents | Search | Archive | Tales from the Machine Room | Contribute | Set language to:en it | Login/Register

No, YOU test it!

Testing! Or not. And looking at the news in the past few... years.. I think that the general consensus on the matter is that "testing" for bugs is just a waste of time and money. Is not the first time that I lament the fact that lots of the software that should be sent to "production" is to say the least "not well tested" and to say the truth, not tested ad all, so when it does go to production the results are what *I* expect.

But if we look outside the IT environment, we can clearly see that the idea is taking roots.

Some years ago there was a big scandal when a known german motorcar company (Cough! CoughVOLKSWAGEN! Cough...) "cheated" the economy and ecology tests by adjusting the electronics of their cars in such a way that the engine was working perfectly to pass the tests but when used "normally" the result were... a lot less eco-logical or eco-nomical. In short, they built special conditions into their system specifically to pass the tests. Lately, a manufcturer of civilian airplanes got itself with hundreds of airplanes grounded because a certain bit of the on-board electronic wasn't correctly tested.

If you read technical publication, you should be used to see security bugs reported in many products that should've been discovered by a casual observer, but remained unpatched for years. Some of them are so well-established, that fixing them breaks lots of other things.

From everywhere you can hear screams and yells, "why this thing hasn't been tested?"...

The answer is, in many cases, because testing stuff COSTS MONEY AND TIME. And, as discovered in many cases, testing most of the time doesn't fixes the problem.

What we see, more often than not, is that manufacturers (especially of software), decide to go with "pre-releases", call them "alpha" or "early access" or wathever. The point is that the product ain't ready, but if you really want, you can get it right-fucking-now and have a go at it.

This is also known as "using your customers as testers", that is named and shamed by many peoples around, but is more or less normality for many things... and from my point of view... is perfectly fine.

Ok, put away the pitchforks and the torches please, the insurance said that they don't cover lynch mobs damages.

What I mean is that, in many circumstances, the only functional way to discover some problems is to emply somebody that has NO FUCKING CLUE of what he is doing, try the product and then see what the heck does he do. Yes, I know, there are professionals for that, peoples that are very good at doing it, but usually they wants money, and the better they are, the more money they want. That is fine, but in some cases the company manufacturing ... wathever that thing is, has no money whatsoever to pay them, or no money whatsoever -POINT-. So the way of "early access" is the only one. No, most of the time is not the best one, and yes, sometimes the company is not really interested in fixing the bugs or improving the product, and in many cases the company COULD employ professional testers (Microsoft), and sometimes they even do that and then still go the "early access" way.

And in some cases, the "testers"... aren't told that they are actually "testing" the thing... Like some airlines discovered on their skin, or the skin of their customers. I mean, if $carmanufacturer decided to test a new type of "active braking system", I'd like to know about the possibility that pressing the brakes could have the car speed up instead of... the normal way reaction.

And now, let's talk about fact.

Once upon a time there was, and still there is, a company, let's call it $companyX, that provided services. One of these services was a "B2B messaging system", and if you don't know what that means, is basically a mail system that is used only by computers to exchange messages between each others.
Basically a computer send a message to another computer that after receiving send back a reply that says "got it".

If you have used once a mailing list, you know what is like. Yes, what is today "sold" as "AI" or "ChatBots"... that regularly doesn't work is sucks and that worked fine about 20 or 30 years ago.

To get a system that was very flexible, $companyX had picked as "backed" for the whole thing a product developed by a german software house, software that was very flexible but also very expensive.

And it came the day that $companyX decied the system required replacement because it was quite old. And it was also no longer maintained, especially in some areas, like antispam. To avoid paying a very high price for a simple upgrade, the company decided to retire the old system and use instead a new system mostly based around postfix.

Nice, but since it was used for B2B, replace the "middle" could very well mean to "break" everything that was connected to it.

In fact, during the update of another system (let's call it "brother") there had been a complete disaster, that provoked a couple of hundreds tickets for "communication problems" in a couple of hours and required a roll-back and full restore of several things in the wee hours of the night.

Obviously, the UL of $companyX wasn't in the mood for a repeat of that performance and so was pressing for a "better management" of the transition.

And here is the point of discontent, where I got against the "project manager" of the thing. Or better the project UNmanager.

Allow me to clarify.

There were 2 guys involved. One of the two had a good knowledge of the technical part of the thing, but he wasn't officially a "manager", the second one had NO FUCKING CLUE about the technical part, or the operative part, or the administrative part... in fact, he had no fucking clue at all, but he was the official Project Manager.

Now, if you ask me, the logical move would have been to send the second guy to fuck himself and promote the first one to 'project manager'. The problem was that the first one had ZERO INTEREST in becoming a project manager because otherwise he also had to take responsabilities. In fact, that first guy was the one that toke the decision to send the other product I said before to production, knowing that nobody had tested it and without informing anyone about the thing. Basically he had turned the switch and when the shit had hit the fun, he started backpedalling at warp speed.

So we were in discussion about putting the new "thing" into production. Me this UL that actually wasn't an UL and the other one (UL2).

Me - ...all the users have been created in the new environment and the messages have been reported, however, the messages that have been reported are all marked as "new" messages, there is no way to know which messages are not new.
UL - Why no?
Me - Because the format of the database of the other product is unknown, and reading one message at a time from IMAP takes way too long.
UL2 - And what does it means?
Me - That we need to tell the users. That they have to check their messages or the way they process the messages.
UL - But... then we have to tell the users!
Me - Obviously we have to. In fact, the sooner we tell the user, the better so they can test things before we go to production.
UL2 - Why?
Me - What do you mean "why"? Becaue we are not talking about you reading your mail, these are messages that are sent between computers, if something doesn't work is very possible that they have to do some changes in their processing flow and since we have no idea what the other product did or how, the only way to know is for the customers to do the testing.
UL - But we should do all the test and be sure that the new system does exactly the same thing as the old one.
Me - Yes but we can't. Because we have NO IDEA what the customer do or check or look at of the messages, we can only grab the messages and throw them into a mailbox, what happen next we don't know. And anyway... you wanted to go in production on saturday, so when did you wanted to do those tests?
UL - I wanted to go on production but then there was the problem with the billing...
Me - Yes, and that problem is still there, but the point stays. If you want to go on production on day X, when are you going to do the 'tests'? X-5 minutes?
UL - I'm kinda busy...
Me - That's your problem. You sent around that "plan" where you specified that before the "go" you want to perform tests. So?
UL - Do you have any suggestion about the tests?
Me - Sure. Let's contact a few of the customers and let THEM do the tests.
UL and UL2 together - WHAT???
Me - Is the best way. They know what should happens before, during and after. If they test and say everything works, it means it works. If not, then we have work to do.
UL - And what happens if they start using the system before we've adjusted for the billing?
Me - That's a completely different problem and has nothing to do with the testing. Anyway, you still wanted to go on production on saturday, so what was your idea about the billing?
UL2 - But what should we tell the customers? Who should sent the mesages? And how? And When?
Me - And why?
UL2 - What?
Me - Since you were asking all the questions...
UL2 - What?
Me - (massaging my nose) ... We need to communicate that we are doing a big upgrade to the system to a new platform, that is the time since it's fucking 20XX already, and there are going to be lots of changes, new interface, new administrative functions and so on, so would be nice if they could test the system. We give the the IPs and let them do their stuff. It shouldn't take too much.
UL - But... if they found problems?
Me - That's why we do the tests. Then we have to see what problems and what to do about them.
UL2 - What kind of problems could be?
Me - I don't know, that's why I say that the customers are the best peoples to do the tests.

Yes, because in many circumstances, you try yourself o you let whoever should use it to try it. And when you have no idea how is going to be used, there is only one way.

02/02/2020 11:52

Previous Next

Comments are added when and more important if I have the time to review them and after removing Spam, Crap, Phishing and the like. So don't hold your breath. And if your comment doesn't appear, is probably becuase it wasn't worth it.

15 messages this document does not accept new posts

Antonio Pennino

By Antonio Pennino posted 03/02/2020 11:48

come e' finita? Il mio radicato pessimismo mi dice "MALE" :\)

-- Antonio Pennino

Messer Franz

By Messer Franz posted 03/02/2020 12:38

Mi è venuta in mente una vecchia storia...anni 90, la Microsoft era vista come un'azienda innovatrice e foriera di speranze per il futuro, e buona parte del testing era dal neonato web, da gente/aziende che collaboravano per avere un sistema funzionante.

Ad un tratto chiamano la microsoft e gli dicono "avete un bug nell'Explorer (mi pare fosse l'explorer), se uno fa una certa cosa col vostro ip vi spegne il pc"


Tizio:"salvate tutto e fatemi un fischio quando avete fatto"

salvano, avvertono.

PUF! PC spenti in tutta la Microsoft.

Penso se succedesse oggi: "avete un bug nell'Explorer!" "E' falso! Se lo dice in giro si becca una denuncia per diffamazione e l'arrestiamo come hacker! Però ci dica qual'è...."

-- Messer Franz

Davide Bianchi

@ Messer Franz By Davide Bianchi posted 03/02/2020 15:10

Penso se succedesse oggi: "avete un bug nell'Explorer!"

"uno solo?"



-- Davide Bianchi

Massi Miliano

@ Davide Bianchi By Massi Miliano posted 05/02/2020 21:50


Penso se succedesse oggi: "avete un bug nell'Explorer!"

"uno solo?"

Sì, uno solo: "Explorer". Ma ora l'hanno risolto con una patch: "Edge"... Non so se ridere o piangere...



-- Massi Miliano


@ Massi Miliano By Budrum posted 06/02/2020 15:40

Sì, uno solo: "Explorer". Ma ora l'hanno risolto con una patch: "Edge"... Non so se ridere o piangere...

Edge: l'unico browser al mondo in grado di farti rimpiangere quella ciofeca di IE.

-- Budrum


@ Messer Franz By Guido posted 06/02/2020 08:17

avete un bug nell'Explorer

Mi hai ricordato una cosa "carina" di IE (6). Sostanzialmente se chiamavi un link sul desktop come un url (tipo www.quaccheccosa.it) e da IE digitavi quell'url ti apriva il link invece dell'url...

-- who uses Debian learns Debian but who uses Slackware learns Linux


By Peoh posted 03/02/2020 14:37

Finale del film: il mannagger - doppie volute - ha deciso di scaraventare in produzione l'accrocchiosaurus maleficus senza l'ombra di un test e tutto s'è schiantato il giorno dopo.

-- Peoh

Davide Bianchi

@ Peoh By Davide Bianchi posted 03/02/2020 15:10

Finale del film: il mannagger - doppie volute - ha deciso di scaraventare in produzione l'accrocchiosaurus maleficus senza l'ombra di un test e tutto s'è schiantato il giorno dopo.

Quella sarebbe la normalita'...



-- Davide Bianchi


By gabriel posted 03/02/2020 17:22

mi sono accorto anche io che il testing è andato a farsi benedire.

io essendo un ragazzo non vedente utilizzo uno screen reader.

questo screen reader(di cui non faccio il nome perchè fa schifo) è stato passato da piton 2 a piton 3, con la prospettiva di pubblicare l'aggiornamento a dicembre.

da quanto è stato testato male e da come sono stati gestiti i vari componenti(lo screen reader supporta l'installazione dei componenti aggiuntivi o estensioni) è ancora in testing, ma da noi utenti non dai paguri australiani o da dove vengono quelli.

a e i componenti aggiuntivi per la maggiorparte sono compatibili con piton 3 e la versione dello screen reader, ma qualcosa mi dice che non sono stati mica testati affondo e che schianteranno non appena si cercherà di farli funzionare.

aspetto che schianti tutto quando verrà rilasciato e poi mi metterò a girare intondo ridendo molto forte e molto malvagiamente.

-- gabriel


By SistemistaDisperato posted 04/02/2020 00:12

Il mio responsabile, dopo essersi finalmente reso conto che avere il server MDaemon in DMZ su di un Windows Server 2008 senza più aggiornamenti non sia proprio una bella cosa (tradotto: era da gennaio 2019 che provavo a spiegarglielo senza successo, ma io sono troppo allarmista, a cosa servirà mai la sicurezza informatica ad un'azienda da 2000 dipendenti e 300 negozi, gli sarà arrivata qualche newsletter più attendibile di me che lo gestisco il server e cambio a mano le password degli indirizzi bucati...), ha deciso di migrarlo stanotte su di una fiammante macchina virtuale Windows Server 2012 nascosta finalmente dietro il firewall ed addirittura in versione X64. Pagando profumatamente un sistemista esterno poiché io non sono laureato e non ho certificazionia al riguardo = mica l'ho configurato e fatto funzionare negli ultimi 8 anni, si è autogestito, autoriparato ed autoconfigurato, altro che IA. E indovinate domattina chi si smazzerà tutti i problemi? Ma tanto il vecchio sarà ancora lí che ci saluta con la manina dalla console del Nutanix, se va male basterà che lo rimetta in piedi e via altra fatturazione al sistemista per la seconda notte, e la terza, e così via fino alla "soluzione". A cosa mai serviranno i test? Robbba sopravvaluta. 

-- SistemistaDisperato


By Thomas posted 04/02/2020 02:17

Sulla questione "early access" sono d'accordo solo in parte.

Nel senso che mi sta benissimo il principio base e lo trovo anche io molto valido, mi può star bene il fare da beta tester aggratis, ma quando iniziano a chiedere soldi per il "privilegio" di essere un beta tester... mavafangul.

A meno che questo "chiedere soldi" non si traduca in: dacci X ora, facci da beta tester, in cambio ti daremo una licenza per il prodotto finito (licenza che normalmente costerebbe non meno di 10X). In tal caso può starmi bene, se il prodotto mi interessa.

-- Thomas


By Guido posted 04/02/2020 07:21

Leggevo da qualche parte che e' questo il rischio dei test automatizzati (perche' nessuno ha voglia di fare test o addirittura di pagare un gruppo di test che non siano gli sviluppatori*) -- ovvero che si scriva sw con il solo scopo di passare i test...


*far testare sw (solo) a chi lo scrive e' male ma non mi far cominciare...

-- who uses Debian learns Debian but who uses Slackware learns Linux


@ Guido By Gianluca posted 05/02/2020 11:29

@Guido : 

"Leggevo da qualche parte che e' questo il rischio dei test automatizzati (perche' nessuno ha voglia di fare test o addirittura di pagare un gruppo di test che non siano gli sviluppatori*) -- ovvero che si scriva sw con il solo scopo di passare i test...


Non è un rischio, è già realtà : https://it.wikipedia.org/wiki/Test_driven_development




-- Gianluca


By Jack posted 09/02/2020 18:14

@Gianluca: Dopo aver letto la pagina che hai indicato e alcune da lì linkate, capisco perché i software recenti sembrano scritti con il deretano rispetto a quelli più datati...

Forse l'unica indicazione sensata è il pair programming, che però raddoppia i costi (o dimezza gli stipendi)....

-- Jack

Anonymous coward

By Anonymous coward posted 24/02/2020 09:33

Perfettamente d'accordo con te sugli early adopters, ma soprattutto nell'hardware è ormai una pratica selvaggia, soprattutto da $quellichevendonofrutticarianewton.

Portatili da 2k che escono con a corredo già 2 programmi di richiamo (display e tastiera al day1) e poi dopo 1 mese altri 2 programmi di richiamo (ssd e scheda logica)...

-- Anonymous coward

15 messages this document does not accept new posts

Previous Next

This site is made by me with blood, sweat and gunpowder, if you want to republish or redistribute any part of it, please drop me (or the author of the article if is not me) a mail.

This site was composed with VIM, now is composed with VIM and the (in)famous CMS FdT.

This site isn't optimized for vision with any specific browser, nor it requires special fonts or resolution.
You're free to see it as you wish.

Web Interoperability Pleadge Support This Project
Powered By Gojira