Be useful in emergencies

Before the years of compounding technical experience, before accumulating an intricate understanding of corporate bureaucracy, before I started my own company — I was an intern. An intern who thought he knew more than any other intern on the planet. I miss the confidence I had before I learned more about the world. I really thought that I could do no wrong.

As with most forms of arrogance, it didn’t take long to catch up to me. This time, in the form of a P0, company-wide alarm. When you’re 19, few things prepare you for a situation like this. The closest I had come to dealing with an emergency was a mandatory CPR class I took while working the front desk at a YMCA as a teenager. This, though, was different.

The team that I worked on was in charge of datacenter failovers. This is the process of migrating traffic from one datacenter to another in case of an emergency. One day, there was some sort of cooling issue in one of our datacenters on the East Coast. Servers were getting overheated and we had to act fast to redirect traffic to the West Coast. It was my team’s time to shine.

Code that is only executed during emergencies tends to collect cobwebs. In these cobwebs, there lived a bug. A bug that I had put there. The same one that rendered this emergency lever of ours completely useless. The only way around it was to do the failover by hand — a long, painful process involving dozens of people across the company.

The embarrassment I felt when it was discovered that I had caused the issue was… something else. Thankfully, I had a great manager who made sure the attention was kept off of me until he had a chance to talk to me privately after this whole thing was resolved.

At the end of the day, when all affected customers were migrated to the healthy datacenter and everyone was happy, my manager told me something I have never forgotten. The best piece of advice I’ve been given in my career:

Things like this will happen again in your career. Things break all the time. Sometimes, it will be your doing and other times someone else’s. Regardless, you need to learn how to deal with situations like these. Learn to be useful during emergencies.

Since then, I’ve tried to put myself in positions where I’d play a role in resolving major issues. Some caused by me; most (thankfully) not. Emergencies are scary and most people will try to weasel their way out of playing a major role in them, so this opens up space for those of us willing to take on the challenge. Take full advantage.

The way that you deal with an emergency is a whole separate, complex topic. You should read Sources of Power by Gary Klein if you’re interested in learning about how professionals make decisions in high-stakes situations. For now, I’ll leave you with this one tip: Present as the calmest person in the room. Slow down your heart rate. Write things down so you don’t forget about all the moving pieces as they come in. Keep track of who is supposed to be doing what. Speak more slowly. Breathe.