Are You Anonymized?
Let's talk about data. Lots of data. The kind of data you get when you run the same system for a few years without ever cleaning up your data structure. What you do with all that bunch of stuff? Well, mostly nothing. You keep it in there, make backups and keep on going. The problem is that every now and then, you need to test stuff. And if you have a TEST environment, you should have the same stuff as in production, so a very nice, big copy of all that junk.
But then you have 2 problems: 1. you have lots of data that you actually don't need and 2. you shouldn't keep real personal data in TEST because there is no need for and bad things (tm) could happen.
What to do?
Simple. Instead of keeping all the data in there permanently, every now and then you do a "data refresh", that basically means you make a copy of the production data, copy it into test and then zap all the data older than 1 month (for example) and replace all the names and addresses with random junk. This way your database is a lot cleaner and leaner and still the data are coherent and you don't have "personal" data in there that could cause problems.
Now, notice that I marked this a 'simple', not 'easy', because the process to go from a full-sized-database to a leaner-meaner-database is everything but easy. First of all, you need to get the whole database. That means A LOT OF DATA, potentially. And everytime you run it, it's more data than the last time. And then you need to figure out a way to DELETE stuff without breaking the data-coherency.
That is not easy and that is why it costs a shitload of time to do right.
Enters $hurryupitslate, a company that... I don't know what they do but they are... sorta... always late for something apparently.
Their normal procedure is to phone requesting something, when you ask for a support ticket to be opened they yell across the room for somebody else to enter the ticket while they keep you at the phone and as soon as the other guy say "it's done" they keep asking if you got the ticket and when you say "yes" they start asking why you didn't do wathever they asked already.
Is not such a wonder that when the phone rings and it's their number nobody wants to pick up the phone and suddenly everybody is really busy in doing... something...
Anyhow, sometimes ago they did their "shtick" to get a copy of their production database moved to the test environment. Unfortunately for them they got ME. And this means that A) I punctually informed them that no database could be copied to the 'test' enviroment since this wasn't existing and I wasn't going to create one without a signed contract for which I tossed them to the wolf, I mean MarketingMan.
When they finished with that, I also informed them that the size of database server they picked for the test environment was a bit on the small size and as such, only a very small set of data could be maintained in there. Also, there was no established procedure for the anonymization of the data and we couldn't devise one because we had no idea of the data structure.
A lot of discussion later, their developers provided a script that was supposed to do the whole delete-old-stuff-and-anonymize gig, it is only a matter of testing it.
And now, let's introduce the star of the day: CL. That was the "on call" person for the day and got the not-so-nice phone call from $hurryupitslate at 9 PM to test the whole thing at once.
CL wasn't exactly thrilled about doing the whole thing, in fact he was more counting on a quiet evening without having to do anything at all, but as you can understand, $hurryup wasn't going to take "no" for an answer. And before somebody mention it: no, that shouldn't have been a task to do during the evening unless it was arranged before hand and concorded with the person, but $hurryup did what they were used to do: called the "emergency number" and ignored the fact that it wasn't a fucking emergency at all.
Anyhow, about half an hour into the process, CL discovered a tragic mismatch between the "test" environment and the production one: the disk size. In specific, the disk size of the database server. However, $hurryup told him, basically, to not bother because the "script should take care of that". So CL simply proceeded.
The problem was that the process required the entire database to be imported before being reduced in size. And when he attempted to do so, the process failed because the disk was too small to host the whole production database. The result was that the process failed to import the database, leaving the TEST database basically an empty slate.
And here cames the problem.
The correct procedure at this point would have been to refuse to proceed with anything and refer the whole matter to DB & MarketingMan the next day, but of course $hurryup wasn't pleased with the solution so they pressed with a different solution. Solution that didn't existed. And this made CL confused... And when CL is confused, CL does stupid thing. What he thought was "hey, how about making a copy of the db on production and running the script on the copy? Then there should be enough space to make a dump of the reduced copy and move that to Test!".
That is a nice idea, I have to admit. But... It requires to run a destructive script on a PRODUCTION environment. And with an already noisy customer on the phone pressing for "speedy solution" and at a time when your brain is no longer in "high attention mode". You know what happened of course.
It happened that CL made a new database on production. And then run the script without making a copy first. The result was a nicely shrunk and anonymized database. On production. And a full test of the restore procedure for the production database and the whole set of data for the day lost.
What did we learned that day? First: do not give the 'emergency phone number' to customer and Second: the Emergency phone number is for EMERGENCY that is "something in production does not work", anything else is NOT an emergency. And third thing: $hurryup can go fuck themselves.
Davide
31/05/2018 16:30
