Tinkering Away
I'm surrounded by tinkerers!
What is a "tinkerer" and why is it a bad thing, you're probably asking. Well, sit down and buckle up, you're going for a ride...
Once upon a time, there were no "engineers" or "scientists" because we still had to invent "engineering" and "science", so everybody was a "tinkerer". A tinkerer (lowercase) is somebody that like to glue, bolt, tie, mix or somehow jam together different things and see what happens. Just for the fun of it. And no, in most cases the tinkerer doesn't know how the 'things' in question works or what could be the effect of the tinkering.
Now, that is not that bad, per-se. We started like that as I just noted. Somebody tinkered together a rock and piece of wood and the axe was invented, somebody else tried a different shape of rock and tested it on the ground and you have the hoe and then somebody else came along with a cow and you have farming.
The problem is that things have began to spiral down pretty fast in the past few centuries. Don't you believe me? Open the window and listen. What do you ear? Engine noise. All over the place. And all the time, 24 hours a day, there is always an engine buzzing, humming or screaming around. And all because about 100 years ago, a couple of guys were tinkering with metal buckets, pistons and some dangerous explosive chemicals, and they managed to cobble together a contraption that worked sometimes, when it was in the mood, and then, while they were thinking "what the fuck do we do with this thing now", somebody else came along and bolted it onto an old horse carriage (after removing the horse luckily) and the "automobile" was invented. That was a total piece of garbage of course. But then, a lot of other tinkerers came along and started bolting or glueing other stuff to it. Until we had autos coming out of our ears.
The biggest problem with autos is that they tend to crash into each other and into other things. To try to stem the flood of incidents, governments realized that they had to do something, so they tried legislation, that is what governments do when they feel they need to do something but are unsure what and can't declare war to somebody else.
The problem was immediately identified and analyzed: peoples can't fucking drive. The answer was... try to specify a shitload of "rules of engagement" that the drivers were supposed to follow to minimize the chances that two autos were going to cross each other paths.
When I took my driving exam, fucking long time ago, a good 60% of the 'rules' dealt with exactly one problem: what to do when you get to a crossing with other autos that also wants to cross that road.
That didn't really improved the situation, especially because even more autos were going to be on the road, so government tried other "solution", like install "traffic lights" (tinkering with the roads) to regulate the crossings, and when even that turned out to be ineffective they tried removing the crossing altogether and Rondabout where invented (more tinkering with the roads).
And nowadays, since the governments have sort-of run out of tinkering they can do, somebody else is tinkering with the autos, bolting on more junk like "sensor" and "advanced AIs" to solve the "problem"... Problem that still is "people can't fucking drive!". And since "autos" have become a massive gigantic industry that move around billions of dosh every day, lots of tinkerers are dipping their hands into the jar.
"What is the point?" you ask. Well, "tinkering" doesn't SOLVE the problem, it doesn't even really address it. In most cases (as shown by my example) the only effect is to glue more junk together in an effort to circumvent or mitigate the problem. The real "problem" is ignored or conveniently swept under the carpet while the tinkerer wave his lates tinkering effort in our faces.
Aaaaaannnddd.. now we get back to earth. Let's talk about $atomicmushrooms, that is a company that ... well, I'm not really sure what they do, but they are hosting stuff with us.
As incredible as it can be, they actually had a TEST environment and a PRODUCTION environment. They also had some sort of 'automated' sistem to perform deployment, system that was basically a giant script that grabbed the code from GIT, copied it onto the machine in a temporary directory and then run a number of 'tests' to see if the code could work and, if everything looked okey-dokey, copied the whole thing in the right place and then restarted all the services.
That's all fine and dandy, as long as your code works correctly. The problem is when the code doesn't. And of course, when that happens, the problem is not the code but the person that wrote it. Enters CL, our tinkerer.
CL - I've started the auto-deployment in test and it has returned and error.
Me - Good, did you fixed it?
CL - No you have to fix it.
Me - But is the script that returns an error or is your code that doesn't check out?
CL - I don't know.
Me - What does the error says?
CL - I have no time, need to fix a bug in production, we need the deploy before this evening.
So I start looking at this thing, that basically download a bunch of stuff and then run a script that is also downloaded to perform the full deployment. It seems the download works fine, and the environment is created correctly, then it checks/updates the packages required by the application and then runs a number of tests... and then it says "failed" and that's about it.
I strongly suspect that the problem is in CL's code, so I change the script to have more debugging and try again.
Well, what do you know?
"Error in SomePieceOfCode.php line 397..."
There is your problem mister. I report the thing to CL, that doesn't sound thrilled.
CL - That is not our stuff.
Me - It came from your repository, so it is your stuff.
CL - No, that is part of the framework $whattheheck that is used by a library we use to access some data from another framework...
Me - Yeah, sure. It is part of your code in any event and that is the problem.
CL - Did you fixed it?
Me - Which part of 'is your code so is your problem' wasn't clear?
CL - But is not my code, I don't know how that thing work.
Me - So who put it into your repository?
CL - Ok, that was me, but I still don't know how that thing works.
Me - ...how do you use it then?
CL - I copied an example that did more or less what I needed to do and that was it.
Me - You're telling me that you don't actually write your code but simply string together bits and pieces that you scavenge on-line?
CL - Well, not all the time... Sometimes... I didn't had the time to figure out all the details on how to use that framework and that was the only functionality I needed so...
Me - If it was just one functionality why didn't you tried to just write that bit instead of putting the whole framework in?
CL - Listen, this is not the point, we need to have the deployment functional for this evening.
Me - The problem is your framework, either something else you added conflict with it or maybe that example you copied wasn't really a good one, fix that.
Of course he doesn't really like the idea of having to "fix" his code. Fixing require understanding and that takes time and effort, while grabbing code from the 'net and slapping it together is a lot faster and, apparently, a lot "cooler". Anyhow, since is very clear that the problem lies in his code and not in the server, I can just drop the problem on his lap and let him deal with it.
A few days later, we get an alarm from the production environment of $atomicmushrooms that is apparently comatose. Somebody checks out and the site seems to be choking on something. Since the instructions are "when in doubt, restore", we do exactly that and rollback to the previous version of the application and everything goes back to "functional".
However, the "automatic deploy" does also store in a convenienently non-tamperable log file all the operations, and guess what? There have been about 10 "hot fix" in the last 2 days, all initiated by CL and none of them can be found on the TEST environment. So whatever CL was trying, he was doing it directly in production. An RCA is immediately sent in the general direction of $atomicmushrooms and we go ahead with our things, until $atomic, in the person of UL is at the phone.
UL - What happened to our system?
Me - Didn't you read the RCA?
UL - Yes, but what does that mean?
Me - It means that CL started an hotfix deployment on the system and that broke the site, and we rolled back to the old version.
UL - How is that hotfix broke the site?
Me - You should ask this to CL, not to us. We are not aware of which changes are done to the site, only that changes are done.
UL - Why does it work on Test?
Me - Again, you should ask to CL, but if I check the status of Test, no deployment has been done on Test in the past week, and the last 3 failed to pass the flight check, so probably this stuff hasn't been tested either.
UL - What?
Me - You heard me.
Well, after some more messing around we found out that CL was quite pissed about us "refusing" to fix his framework, so he decided to glue even more junk to the system to circumvent the problem, that however backfired badly, so to "fix" the bug, he used the 'hotfix' ability to simply upload some stuff in the system without running a full check and that broke the Test environment. Since he couldn't revert his changes, he left the Test environment broken and performed hot fix on production until that one broke too.
And that's another problem with tinkerer. They tinker until it's broken. And then they start tinkering something else.
Davide
09/05/2018 15:52
