When you hear about DevOps, you often hear about all of the ways it can help your code be better. That’s true. But there’s a whole host of other benefits. This week highlighted how a dev team can be stronger with a solid DevOps practice.
The Storm Cloud
Last Monday started normally. We did remote co-working, reviewed PRs, and chatted about what we did over the weekend. It seemed like a cloudy, but pleasant and normal start to the week. And then the derecho hit.
Most of my team is in St. Paul, but I’m down in Des Moines. And in the 90 minutes between my first and last message above, my neighborhood went from your generic midcentury housing tract to a literal disaster zone.
Power lines, trees, and stranded cars were unfortunately common this past week.
I took the afternoon off, letting my team know I’d be outside helping clean up anything we safely could and waiting for the power to come back on. The waiting lasted three days. I did my best to find power and stable internet for as long as I could each day, but I was effectively cut off from my team.
The Silver Lining
Luckily, we’ve been remote-ready from the start. And we have been working hard to build a DevOps culture in the team. And it showed this week. The team was able to pick up tasks where they’d been left off because statuses were communicated to the whole team before trouble hit. They knew where to find the information and the processes were already known. We have several ways to communicate, so we could still get in touch with each other to ask questions or for me to let them know I was fine, but still waiting for the lights to come back on. In addition to an enabled team, many processes are automated, so if I only had flaky LTE for a few minutes, I could still kick off builds or PRs and know that things would still complete even when my connection dropped.
Of course, we didn’t get as much done this week as a “normal” week. But we got a lot more done while dealing with an emergency situation than we would have had we not prepared ahead of time for trouble to happen and to keep any one of us from becoming a bottleneck.
How do I make my team resilient?
There are plenty of consultants out there that will happily sell you a program in one of many detailed systems. Here we believe the important part is that you have a system. It doesn’t matter which one, as long as you don’t lack a system. In our DevOps culture, we have come to value good communication, daily growth and learning, and team empowerment.
To get where we are, we’ve been working on:
- If you feel like you’re over-communicating, you’re probably communicating the right amount.
- Show your work. Keep written logs of where your tasks stand. The more detailed the better.
- Automate the things that are easy to mess up first, even before the boring stuff.
- Daily learning. Share our mistakes, articles or hints we found valuable.
- Give frequent mini-talks about topics or technology that we’re working on or learning about.
- Build the tools to work effectively remotely.
