Loggings
...BEEEP BEEEP BEEEEP BEEEP... The proximity alarm keeps screaming, while the fragmentation missile whizzes 300Km away, its trajectory changed by the gravity of the gas giant "planet" the "HaveYouTriedTurningOffAndOn?" is orbiting.
The missile will probably blow up in before reaching the atmosphere, in about 25 minutes. But this is the first one, there are 2 more and something make me think that whoever is sitting in the Control chair on the enemy cruiser is right now sending updates to the AI of the other 2 missiles.
The computer informs me that both missiles are now acting trajectory corrections, the thrusters are invisible in normal light, but very visible in the infrared against the 2 Kelvin of the surrounding space. Both missiles are now travelling at cruise speed of about 30.000 mt/s, with an accelleration of about 8gs to correct... The computer spit out the data: the first one will zip by at less than 20 Km. Waay too close...
My orbit is changing since about 10 minutes, and should bring me close enough to the second moon to mess up the radar reading, the enemy cruiser is about 600.000 Km away so the next course change will reach the missile's computer in about 2 seconds...
BEEEP BEEEP BEEEP... What the ... ? Another signature is on the scanner now, this one is close. Way too close. What the heck is he doing 20 Km away?
BEEEP BEEEEP BEEEEP Oh Fuck... Ok, Ok, I'm awake now! Fuck...
Rummage around searching for the pager that is doing its best to wake up the whole block (and is very good at it) and stop the alarm, while with the other hand I'm trying to turn the light on. I manage to find the cat, that doesn't light up but only meows. After turning on the light, I think that if I grab the glasses I'll have better chance to read it.
Allarm: disk space $machinename
Oh, the joy of being "on call".
Turn on the PC and login on the machine. Or better, I try to. But I only get "access denied". Something make me think that the jackass that configured this thing, DID NOT made /var its own partition. As it specified in the "best practice" and is told and repeated all the time by me. At least since a year and half. The result is that some fucking process filled up /var/log, and hence it filled up / and now you can't login anymore.
It's time to go to the console. After messing around for about a quarter of an hour to figure out which one of the thousands of "default" password was used, I manage to login and my suspects are confirmed: /var/log is 16 Gb on a 20 Gb disk.
I quickly found the culprit in a nice "data_processing_debug.log" file and zap it, cleaning up about 7 Gb at once. After a while the file begin to grow immediately like some sort of alien monster.
Hmmmm... Maybe it's me, but the content of this file looks like a huge pile of useless crap. Anyhow, I have 2 choices here: I stop wathever is writing in this thing and make it write somewhere else (like /dev/null) or the whole thing will be full again in about an hour. Ok, /dev/null it is.
After a while it looks like everything is working and nothing gets logged. Nice. I write down the report with the number of hours spent rummaging around and go back to sleep. Of course the next day the developer (CL) came aloung to complain about the mistreatement of his log file.
CL - Why the log file has been deleted? We need that file for debugging!
Me - First of all, debugging should be done on the test environment, not the production, second, we had 2 choices: we remove log files that grew waay too fast or leave the whole system stuck.
CL - But the log file doesn't lock the whole system.
Me - It does if it fills up the whole / partition that is where everything else is. No space there means no space where to write, including the database. If the database can't be written, everything stop working.
After repeating the whole thing for about 96 times, I manage to lose CL but at this point is DB turn.
DB - What happened with CL server?
Me - It happened that some idiot installed the machine without reading the documentation, so with everything in /, and some other idiots that performed the "quality control" did it with the same care, so none wathsoever. With the expected result. Or better, expected by anyone that can use his brain.
DB - And why hasn't been noted by the check?
Me - Why don't you ask whoever was supposed to check? Maybe the same that should have been checking in the last... I don't know.. 3 days?
DB - And why isn't in the graphs?
Me - Because the same idiot didn't added the server to the graphs. Now have you finished asked stupid questions or shall we continue?
DB - I'm just trying to understand why this incident happened...
Me - This ain't an incident, this is the result of incompetent administration. And is incompetent because the procedures that exists are not followed. And are not followed because whoever should check doesn't. And guess who is that "whoever"?
DB - I have a meeting with the megaboss now, we'll talk later.
Yep... It's always "later".
Anyhow, I didn't see him for the rest of the day.
Fast forward of about 12 hours, that is, about 2 AM the next morning, when the pager started again. And yes, it is again the same fucking log file. I repeat the very same process and re-mark the very same time spent.
The next day I go directly to DB.
DB - Don't tell me... Again CL server.
Me - Of course.
DB - It should've been fixed...
Me - And obviously wasn't done. At this point I'm asking what are we supposed to do with this thing.
DB - Well... Nothing...
Me - "Nothing" is not an acceptable answer. If there is a problem and in this case obviously there is one, that problem have to be confronted and fixed.
DB - What do you suggest?
Me - Phase one: disable logging from that thing.
DB - We can't remove the logging they are using it for debugging.
Me - Phase two: install test environment for debugging.
DB - And whose going to pay for it?
Me - Them. Or we suspend the monitoring and if the application crashes, and it will crash, it stay crashed.
DB - That can't be.
Me - Good (I drop the pager on the table) This is yours then, because I'm not getting up tonight.
What followed was a very long and boring discussion, that was waaaay too long to be reported, in any case, my point of view was, and still is, that the monitoring should cover INCIDENTS, that are impredictable and unavoidable events, while a PROBLEM is predictable and avoidable and should be treated in a complete different way. Always act as in "emergency response" condition doesn't demonstrate any special capability or efficiency. Quite the opposite in fact.
And if the problem is caused by a larger Structural or Organizational problem, then it has to be solved in the same way.
DB - ...what do you mean?
Me - That we have procedures, that should be followed because they are part of all the fucking "certification" that everybody seems to adore, and if the procedures are not followed, then we shouldn't be too surprised when things don't work as expected.
DB - Hmmm...
Me - And to be sure that the procedures are followed there should be corresponding checks and administrative tasks that should also be followed. For example one of these specify that when a system is installed, before it gets a stamp of "approved", somebody should verify if the installation has been completed and all the box ticked. Now, this test has not been done on CL's, the system however was marked as "production". Why?
DB - Well, it was an urgent request...
Me - It is ALWAYS an Urgent request, how is that we are the ones that have always to scramble to fix somebody's else shit? Well, now you fix it.
And from that moment, the log file of CL has been symlined to /dev/null.
Davide
24/05/2018 13:53
