Experts
Set: Day - Interior IT office - The sysadmin sits at his table and is happily typing on the pc. Suddenly te door opens and CL
enters brandishing a not-so-pristine shoe.
CL - I need a shoe! With a very sturdy heel!
Me - Hemmm... this is the IT office, I'm not a shoemaker...
CL - (weaving the shoe) Shoe! Heel! Sturdy!
Me - Ohi! Calm the fuck down. What the heck are you doing?
CL - I need a show with a sturdy heel! I told you already!
Me - And I told you that I'm not a shoemaker! What are you doing with a shoe?
CL - (show a huge nail) I need to put this into the wall!
Me - ...and why don't you use an hammer for that?
CL - Hammer? What the fuck.. SHOE! HEEL!
The scene vanishes with wibbidly-wobbidly effect.
Ok, this hasn't happened (yet), but if it does, I wouldn't be too surprised.
Very often there are individual that get "specialized" in doing specific things in a specific way, this is what is normally referred as "getting competent" and there is nothing wrong with it. The problem is when the acquired "competence" is the only one and the person in question see only one solution for every problem, doesn't matter what kind of problem.
The situation is aggravated when the "specialist" in question has actually acquired fragmented or just plain wrong informations. In that case, the person tend to apply repeatedly the same process that, of course, produces results that are not the expected. This in turn produces more attempts at the 'solution' that doesn't work.
And after this theatrical introduction, let's talk about $weknoweverything, a company that... did several things.
They had acquired some sort of specialization in a very narrow field and were selling such "experience" at a price (it is called "consultancy" I think). The problem wasn't that they were specialized in a field, but that they had convinced themselves that such "specialization" could be extended to every other field no matter how disconnected from the original it was.
For example, they had decided to build some sort of "database" with information about their expertise and provide some carefully selected customers (aka: everybody that could pay) access to the information, and since they were "competent" (they had read some article somewhere) they've decided the whole IT infrastructure, then they've mailed their ideas to MarketingMan whom had talked with somebody else and that was how the whole "architecture" was designed.
Fast forward a couple of months, when their database apparently went in a coma and obviously $weknow started to scream a lot.
That day I had the misfortune to look at the problem.
The database seemed to work correctly, at least, I couldn't see any problem, and the application for the access was also working fine, so the problem was somewhere else.
However, I noticed that repeating the same request, sometimes I got an "error 500", but that error didn't show in the application log. So... somewhere there was a snag.
After a bit spent poking and prodding, I realized that the idea of $weknow of a "redundant system" was to have 2 frontend server, in 2 different hosting and both IPs were in the DNS. Now the frontend that was hosted by the other ISP apparently was returning errors.
I could ping it and I could send him requests, but all I got back was errors. At this point my hypotesis was that the server or the connection with the database was broken, but I couldn't check.
Every time the DNS returned the "wrong" IP, I got an error in response. And obviously the DNS wasn't under our control.
After composing a mail to explain the problem and basically saying "you're barking to the wrong tree", I declared the problem closed on my side and went to do something else. But of course this didn't went well with $weknow, about a quarter later, we got CL on the phone.
CL - What do you mean you're not the one that manage that stuff?
Me - The server that returns errors is managed by $otherISP, therefore you have to ask them what the problem is, I can't do anything about it
CL - But your server should respond.
Me - And it does, the errors are from the other one.
CL - But if your server answers we shouldn't get any error!
Me - Hummm... Maybe I wasn't clear, the errors are coming from the other server, the one hosted by $otherISP that you should contact for...
CL - I got that, we're EXPERT you know? If one of the server responds, it should always respond so why isn't doing it?
Me - ...Ok, now I'm confused, how can the server respond to requests that are not sent to it without a connection between the two?
CL - Of course there is a connection they're both in the DNS right?
Me - ...And what is the point of that?
CL - What do you mean? Aren't you expert?
Me - Maybe I didn't understood.
CL - That both systems are in the DNS, so if one doesn't respond the other one should.
Me - Hemmm... No, the DNS doesn't work like this.
CL - What are you talking about?
Me - Maybe you're talking about a load balancer, but I don't see any load balancer in your structure...
CL - No, what load balancer, those things are useless, we have the DNS and that should answer with the good server all the time.
Me - No, the DNS doesn't work like that. The DNS simply translate an hostname into an IP address and that's it. The DNS doesn't check if the server exists, answers or anything.
CL - What the heck are you talking about? Experts...
The discussion went on for a bit, then the boss was called upon and the boss' boss... and at the end, we were all invited at a big meeting. And several peoples explained to them that NO, the DNS DOESN'T WORK LIKE THAT. And what you need is a LOAD BALANCER.
CL - yes, and if the load balancer breaks then?
Now, I want to open a little parenthesis here, this story of the "server that breaks" is by now a bit of a fairy tale. Long (ok, not really that long) time ago, we had PHYSICAL machines, that were expensive and as such everybody was trying to get along by saving every possible penny, using machines way past their lifetime or recycling all kind of garbage. That the hardware at some point were breaking was not so uncommon. But we're in 201X now! It was sometimes ago that VMWare blew everybody showing how they could run a "server" as a piece of software, today there is good chance that the 'server' is nothing more than software run on a cluster, that is a lot of machines and if one of them goes down you barely notice. And disks are also "slices" of a very large "volume". So the "server breaks" is getting less and less a possibility and even when IT DOES happens, you can spin an identical one in minutes. So... No, that's not the problem.
Everybody happy then? In that case you wouldn't be reading this. Because as said, the problem of an "expert" is that he doesn't accept so easily that his "expertise" is in doubt. So $weknow decided to get not one, but TWO load balancer, in different hosting. Each one connected with ONE backend. And both IPs for the LBs were in the DNS. In case of a malfunction, somebody had to login in the remaining lb and reconfigure it. And obviously we had no access to the other lb and the other way around.
Davide
14/01/2019 12:56
