Great Teams Break Stuff, Too

Facebook’s internal motto for a few years was famously, “Move fast and break things.” It was founder Mark Zuckerberg’s way of saying to his developers, don’t let perfect be the enemy of good. Digital innovation is often about rapid iteration, trying things that may not work along the path to ultimate success. If you’re not making mistakes, the thinking goes, you’re not really exploring new territory or being aggressive enough.

While we take an iterative approach to development at DSGa, prioritizing speed over accuracy and efficacy is not how we roll. We are careful when we make updates. We’re not managing just any old websites; we take the weight of our civic responsibility seriously. Before releasing new code, we test and tweak thoroughly in a development environment that’s a close-but-not-exact copy of the live website. And yet … from time to time we unintentionally break things. This happened recently when we tried to upgrade site Search.

Full transparency, there were warning signs that something wasn’t quite right. We noticed some weirdness in our development environment but couldn’t isolate the problem. We assumed it was because of the environment rather than the code, being optimistic that it would all work fine once it got to production. If it didn’t, we figured we’d have better success identifying the root cause in the live production environment. This was a calculated risk to move forward with the code release but planned for the (hopefully unlikely) possibility it could fail. The good news is that we were ready for what happened next. Our code release broke site Search.

Creating a safe space to make mistakes

It’s not great that we broke Search. We’re sorry for the inconvenience it caused. Our preferred way to learn is not by screwing up. That said, we’re human and mistakes happen. Fortunately, we’d thought through different scenarios before going live.

We know from analytics that only about 3% of GovHub visitors land on a Search Results page. Of course we want everything to work for 100% of them, but the impact of failure would be relatively minor.

By going ahead with the release, we’d get much more data and support help from our hosting provider to find the bugs, if they truly existed. Just in case, we had a Google Search results link ready to go, so constituents still had a way to find what they were looking for on the website. We didn’t really expect to use it, but thank goodness it was ready!

Learn, grow, and move on

Even though we were prepared for what happened, we still had a hard time fixing it. It was a lot of trial and error, and at times our approach was chaotic. We can see that now in hindsight, and we learned from it.

After the issue with Search was resolved and everything was working again, we hashed out what went wrong. This was a hard conversation, but our team has worked together for a long time and we trust each other. Our desire is to learn from our mistakes and improve processes where necessary.

It’s not like we won’t make mistakes again, but hopefully we won’t repeat the same ones! Sometimes, we break stuff. We learn. We grow. We move on, getting better as we go.

Now about that Search upgrade

Why were we messing around with Search in the first place? Well, it all has to do with the GovHub update to Drupal 9, which has required us to update several internal elements like modules and design themes to ensure compatibility. The Search indexing tool took a giant leap forward, going from its version 4 to version 7. Drupal is the open source content management system upon which GovHub is built. The upgrade to Drupal 9 ensures that our sites are fast, secure, and running on the latest supported technology.

Related to: 

• GovHub
• User Testing