Testing, RoboTesting or NoTesting
Testing, what a nice word. In fact no, ain't nice at all, but everybody knows what it means because, for bad or good, all of us have seen some crap that wasn't working as supposed and thought "how the heck did this passed testing"? Well, easy, there was no testing.
When we talk about "testing" we mean to "try" something to be sure that it does what it's supposed to do and the way it's supposed to do and not some other way. In the case of software (since we're here), it's a matter of check if the software that we need to use or we're developing does what it need to do and not some other shit. The "test phase" is normally a very long procedure, boring and very complicated. Because you can't just try what you know that it works and check that it did (and this is already a complicated thing), you also need to check that every "unexptected" thing doesn't provoke weird results. And every time there is a new version or a change in the application, you should redo the whole thing to be sure that nothing got broken in the meantime.
Long time ago, that is about 10 years ago, there were peoples that were specifically employed to do "product testing" or "software testing", also known are "quality controller" because they were supposed to check the "quality" of the products... By brealing them, or trying to. The problem in employing people to let them break your products (and to see how long and what it takes to do so) is that it costs a lot of money. In several way.
First of all, you need to pay the people, that is already a pain. Then you need to give them the thing to break and then you have to collect and analyze the results that is also a complicated and long process and lastly, you need to fix the problem you've found. Or decide that is not worth fixing them. And then repeat everything again.
Years ago there were discussions about the best way to do software testing and the consensus was that if you want the best results you need to grab peoples that never ever saw your software before, only that way you're sure that the guy in front of the keyboard will do everything that he can think of, and not just what YOU think is possible. Because the problem in testing the software you know is that you try thing that you know should work, while the 'real' user is capable of being very very "creative" and can find things that the developers look in disbelief muttering things like "howthefuckdidhedothat" or similar.
Then... still about a decade ago, searching for way to save money in every possible way, somebody came up with this idea: computers are very fast and kinda cheap, why don't let THEM test the stuff? That is a very good idea but... computers are very fast and cheap ans STUPID. The only thing they can do is to follow programs. Sure, you can write a very good one that "emulate" a user and let it interact with your software and check the result but this has 2 problems: first you need to be sure that your 'test' program works, so you need to ... hemmm... TEST the 'test' program. And second, every time you change your software you need to change the test software and test it again. And then there is the other problem that is specific of computerized tests: it does only what we told him to do. It doesn't came up with weird ideas. It doesn't think "let me see what happens if I put this-or-that in this-or-that field".
And with this, we're gonna talk about $bingobongo, a company that ... well, was doing several things for which they've developed a few webapplicascion and a mobile app. And as expected, they were receiving a huge amount of complaints because their applications were full of holes.
A nice morning just to say one, they decided to ask us why they kept receiving reports from their customers that they couldn't get the various "newsletter". And after a quick check we found a couple of thousands of e-mails like '@gmial.com' and some more like '@gmail.co'. Answer: because you don't check that the addresses are valid evidently. Their answer: our system test if the mail address is a good mail address. A "good" mail address yes, but not if it "does work".
Another time they had problems with some stuff that was computing the distribution of peoples by geographical area. And after another check we found a lot of wrongly spelled addresses.
And so on and so forth.
The best was when, after a particularly confused release, confused because after the release they discovered they had left some piece of code in there so everybody was able to login using the default password, because their automated testing system wasn't using a "real" account they had a "special" one and used a switch in the code to disable password authentication. And that made me thin that their testing process was bugged since the beginning. Anyhow, after all that messing around (rollback? nope!), they also found that one of the application was crashing every now and then and after some check they found out that one of the database field was supposed to be 'numeric' but the data were ALPHAnumeric.
And when asked "how the fuck did it passed your test", the answer was that the value is provided by a third party, so their automated testing system spit out a random number. A random NUMBER not a randomly generated string. And why a number? Because in the previous version the third party was always providing a number, but it looks like they also changed something.
And then there was the Big one. When turned out that a lot of their users got 'hacked'. And we found out that to login you only needed to send a numerical ID as a POST to the site. And since the IDs where sequential, it wasn't too difficult to pick a good one, and if we want to add the cherry on the cake, the ID 0 was the site admin.
And obviously their "robot" didn't found out because it wasn't programmet to think.
But fear not! $bingobongo thought the thought! And they decided that the best way to test the applications is (drum roll) give access to the TEST system to "selected individuals" that will be able to "preview" the new versions and check if everything works correctly and submit "quality" bug reports. Basically, they want to use their users for testing but calling it "early access" or "Xperimental" that is cool because it has an X in the name.
Oh, is not that Microsoft is doing any better lately eh...
Davide
22/11/2018 16:05
(scherzo eh!)
