Last Friday, I packed my stuff, jumped in the car and hit the road for 3 deserved weeks away, leaving Greg with everything he needed to run Botify platform, package our software and deploy new virtual machines just like I do everyday.
That’s what I thought.
I could not see anything wrong. My Ansible roles were smoothly running out Puppet modules on a daily basis. Packaging was working like a charm, and deploying a new version of our software was a question of minutes, to a point I could ship it on staging 5 or 6 times a day if needed.
It didn’t work as expected. In fact, it didn’t work at all.
Many things didn’t work for Greg, including our Ansible roles not being correct after a bogus merge, some missing points in my doc, AWS Autoscaling Tools not being the same on Ubuntu and Amazon Website, my doc not mentioning how not signing new packages after generating the repository could lead to an inconsistent state and many more deviating from this.
Indeed, I never had any of these problems since it works for me.
It took Greg almost a full day to make things work as expected, and every time something went wrong, I was staring bluntly at my screen, 600 kilometers from my office thinking but… it works for me.
I was devastated.
For the past 9 months, a large part of my job has been to make deployments simpler, faster, the platform more reliable so we can sleep at night, and ensure such a thing would never happen. When I left for this 3 weeks vacation, I was pretty confident as it works for me, 5 days a week, up to 6 times a day if needed.
Indeed it would have been faster that I took over the deployment, but I’m in vacation, and most of my job is to make the platform run without me. From a job security point of view, it probably sounds suicidal. From a company perspective, that’s exactly the opposite.
The graveyards are full of indispensable men.
No company should have indispensable employees. Even Apple showed it can exist without Steve Jobs.
I did three mistakes here.
The biggest one was not taking the time to sit down behind Greg and watch him deploy the whole thing reading my doc. I had too much to do, he also had lot to do and I didn’t want to disturb him. After all, it works for me.
I worked on automatizing the whole deployment thing so it’s easier, faster and more reliable by making most places it used to break stronger, but I didn’t automatize the deployment environment setup. The documentation is not enough until you mention every software version, linked library and even where
/bin/sh links to as two people won’t have the same setup. But once again, it works for me.
The third one is about pride. I was too confident in both my documentation and the fact I’m working with some of the smartest guys I know, so nothing could go wrong. The documentation was a cut / paste of a full deployment, so it could not fail since it just worked for me.
Since I’ve been in the tech industry, the it works for me answer I often get from some engineers or lousy support guys has been a source of deep frustration. Every time I hear it, I feel like smashing them in te face with a haddock until they understand how stupid it is or until they die, whichever comes first.
For too long, it works for me has been an excuse for laziness and poorly executed work. It considers the problem is somewhere between the keyboard and the chair, raising the tech savvy engineer to a kind of superior being and avoids the effort of understanding what’s actually going on.
From a user perspective, it works for me is not an acceptable answer, and if you except the telcos, most Web companies are now understanding this.
Competition being just one click away and the rise of social media where anyone can shout their poor experience with tech support to the world have greatly changed how the job is done. A great support, whatever you call it from « customer heroes » to « happiness squad » has become a sales argument, and companies with great support stats are now publishing their figures, so it works for me is – in most cases – a souvenir from an older, tougher time.
But the final truth is: if it works (only) for you, consider it works for nobody.