What you DO NOT know...
We keep talking about "Business Continuity", that, as explained before, is that thing that nobody understand until it stops working, and then everybody start screaming and running around like headless chickens.
Let's start by repeating my personal opinion that if "something" is an integral part of the chain of processes that in the end put money in your pocket, you should take care of it and not ignore it until it crashes, catch fire or show you the middle finger and walk out the door. This last one in case the "something" is actually a "somebody".
Of course my opinion is just my opinion, and everybody is entitles to his own wrong opinions, but I prefer that, once your wrong opinions are proven wrong, multiple times, with scientific details, you should stop bothering me.
And now we're back talking about $shitandstuff, of which I've already written some times ago, as you remember, those peoples were making money by buying junk and selling it on the internet. Nothing incredible in this, there are lots of companies that made lots of money reselling junk on the internet, and you only need to look at things like e-bay to realize that selling junk is a very lucrative activity with lots of participants.
In the specific case of $shitandstuff, those peoples had made that their religion and belief, to the point they had a system specifically developed to manage order and invoices.
This "thing" was based on some very old stuff, so old that it was running on Solari 8. Let's ignore why or what, in any event, it was hosted on a dedicated machine somewhere in $shitandstuff's office.
What I know is that that thing was in our monitoring, and I know that because one day somebody, let's call him Bob, phoned up to inform us he was going to do some maintenance on it and if we could please put it in downtime. And I remember that because normally nobody, I mean NOBODY, ever informed us before doing maintenance on their system and when we tried to get the irresponsible one, we got insults because we made him "miss the goal".
So, when that thing suddenly went "down" in the monitoring, I showed some surprise. After sending a message and receiving no answer (what were you thinking?) I simply acknowledged the error and that was it.
Days passes, about a week, and then a phone call arrive:
SL - I am SL of $shitandstuff, we have problems with the order processing!
Me - Hummm... I see your ERP software is still down, maybe that is the problem.
SL - Oh, good, put it back up then! Right away!
Me - No, you should put it back up.
SL - What do you mean "it's us"?
Me - That machine is somewhere in your office, we do not maintain it, we only do some monitoring on it.
SL - In our office?
Me - Yes. Don't know where, why or how, but is in your office.
SL - And if it is monitored why didn't you informed us?
Me - We sent a mail a week ago and never received any answer, since there is no established escalation policy and that thing is in your office there wasn't much we could do.
SL - Are you sure is in our office?
Me - Well, the IP address is from your office so...
SL - And now what am I supposed to do? We need that thing back right now!
Me - How about going to talk with your IT? Somebody should manage that thing.
After a bit of "discussion", SL, that apparently was the "Finance Director" goes on an hunt for whomever in IT is managing that thing. Several hours passes without any change on the status of the server.
At 17, I start putting up helmet and gloves and walking towards the door, when the phone start ringing, I ignored it and kept going. Of course once at home I noticed 3 missed calls on my cellphone (no, I don't answer the phone while I am riding, or driving for that matter). And I refrained to call back the caller because I do 24x7 support when it's my turn, not when it's somebody else turn.
Of course the next day DB came to visit when he gets to the office (I was in the office by a couple of hours already).
DB - Yesterday SL from $shitandstuff called.
Me - For that server they lost probably.
DB - They lost it?
Me - Seems like it, did they found it?
DB - They said doesn't start anymore.
Me - And why are they talking with us? Isn't our server.
DB - They asked if we can help in starting it up, we could get the job of "clouderizing" it.
Me - That thing is Solaris, and we have no idea what runs on it, I seriously doubt we can do something with it.
DB - Right, you have some experience with Solaris, right?
... I don't have to go on right? The result was that I found myself in $shitandstuff office, office that was on 2 floors on a building in some desolate zone of town. There I get to know SL1 (IT Manager) and SL2 (Finance). SL2 stresses the importance of restarting immediately the process that is "central to the business"... curious that they discovered that is "central" only now that is not working. SL1 just stresses. Let's go see this server then.
The server turns out to be a Sun Ultra 450, and it also turns out to be on top of a table with no network or power connected to it. Maybe that explains why is not working. I also meet CL that is the 'IT' techie.
Me - Ok, let's plug it in and turn it on.
CL - Yes but... How do we plug in the monitor, mouse and keyboard?
Me - The monitor can be any CRT monitor, keyboard and mouse should be PS2.
CL - CRwhat? PSWhat?
I look at him and realize that, with all probability, this guy has never seen a CRT monitor or a keyboard that was not a USB or wireless one in his entire life.
Me - Where was this thing before being on top of this table?
CL - Ehhhh.. it was under that desk, but the rest of the stuff we don't have them anymore.
Me - ...what do you mean "we don't have them anymore"?
Then I have a doubt and open the front panel. Nope. No disks inside it. Empty.
Me - And what happened to the disks that were in this thing?
CL - Ah, the disks are here.
And pull out a plastic box with a few disks inside. Disks that looks like somebody tossed them out of the window and then went to pick them up again.
At this point I already knew that the server wouldn't have restarted ever again. I also understood that the "IT experience" of CL could be summarized in "he saw Ubuntu Retarder Romba once", and the experience of SL1 is probably even lower. But at that point I am curious, so I sat down and asked The Question.
Me - Ok, what the fuck happened here?
After a long winded story, the Truth was revealed.
Once upon a time, there was a guy called Bob, that, during the years, had put together some sort of ERP thingy that was used by $shitandstuff to run their system. That thing was an incredible hack, it worked, but Bob was the only one that knew how. And it lived in an old Sun Ultra 450 that lived under Bob's desk.
Then, one day, somebody had hired a new "IT Manager" whom decided that the IT needed to get "younger", and since he knew nothing about what Bob did, managed to piss him off. Now, I can't and won't discuss if Bob's behavior was correct or not, but it was clear that they pissed him off enough that he flipped them the bird and walked out.
His "software" remained there, churning peacefully in its server for another month or so, until the IT Manager asked CL what that thing was for and, since CL had no idea, they decided to turn it off and see if somebody was complaining. And CL, that had no idea how to properly turn it off, pulled the plug. Since the process was using that thing only in burst and not continuously, nobody noticed right away. So after a week, SL told CL to throw away the machine. And he did.
That was a day or two before SL2 started to receive notices that they couldn't process order or invoices anymore. At that point CL was quickly sent to chase the garbage truck down the road.
And this, should demonstrate, again, like it was still required, that everything you DO NOT KNOW, could came back and bite you in the ass, and if what you DO NOT KNOW is an integral part of your job, maybe you should change job.
Davide
22/04/2021 12:03
