F*ck it, ship it.

September 18, 2024

A control panel with a rocket taking off into the night sky in the background. — ChatGPT's witty interpretation of a dark launch.

When we're building a SaaS product, we have to walk a tight rope when we want to put updates to the product out there. We want to fix bugs and release new features but we also don't want to upset our customers and throw big launch parties with tons of new features. So how do we balance them?

Slow and steady

Let's face it, people hate change, especially the big in-your-face kind of change of new features and designs. If the user interface changes very often, people are going to get frustrated because they won't find the feature they're looking for. The visual cues they're used to are suddenly gone. They have to change their workflows because someone decided to replace an old feature with a "better" one. The customer success team will have a hard time keeping track of all changes and the documentation will go out of date really quick. Nobody will find what they're looking for in the product. They will get annoyed with the product and the team behind it. And if the product is an essential part of the company operations, some customers might not even want any updates at all and are perfectly content to stay on the very first version that they ever used.

The marketing team loves it when new changes roll out to customers once a quarter though. They work with the product team to make as much noise as possible. A press release, fancy new marketing materials, a nice booth at a top-tier conference. The customers and fans of the company will love it, the press will feature it prominently. A major release is the perfect marketing event, but all those activities quickly loose their magic when executed many times a quarter.

Move fast and break things

Product and engineering teams, on the other hand, hate this slow release cycle. The product teams doesn't have a way to get early feedback from customers before the release. They can show designs and mockups, but they can't put it in the hands of some designated customers to know if works as intended with their workflows and data. They risk shipping something that will do what customers ask for, on the surface, but has way too many rough edges to be actually useful. After the new release, they have to wait another quarter to implement even the smallest feedback and improvements.

The main reason engineering teams hate quarterly releases, is a reason they share with the customers: bugs. Nobody wants bugs, but if a full quarter of code piles up to be released to the customers all at once, there will be bugs. Nobody still remembers all the details of the code that was committed inn the first days of the quarter. The bugs will be in scenario's that nobody thought the customer would ever use and the QA team didn't add to their test matrix. They'll be in early, old and forgotten features that should have been sunset long ago, but that are being kept alive for that one big account. And once the release goes out and the big customer account discovers the bug, they will escalate it as high as possible to get it fixed as soon as possible. With quarterly release, the post-release days are usually a flurry of bug fixing, hot-fixing and late night fire-fighting. Even if all the feature do what they are supposed to do, there is a big chance that some performance issues are lurking below the surface, waiting to strike as customers discover and explore them. Big quarterly releases push the feature from 0 to 100 faster than most engineering teams account for during development, running into simple but highly visible performance issues like missing database indexes.

Both the slow and fast camps have perfectly valid reasons for their preferred release cadence, so how can we satisfy both camps?

It's a feature, not a bug

The solution to the problem is quite simple and staring us right in the eye. It's a process used all across the software industry to get new versions of operating systems and desktop applications into customers hands: pre-releases. Before the new version of the software is officially released, one or more "preview" versions can be given to a select set of customers to test and validate new features, and find potential bugs, which makes product and engineering happy. The customer can choose whether or not they want to be part of this evaluation period and the marketing team can schedule a big marketing push around the final public release.

But the beauty of SaaS is also its achilles heel in this story: all customers run on the same shared servers with the same code against the same databases, reducing the cost of serving all customers. But how can new updates be handed to some customers but not to others? With a the technique called a "dark launch".

In a "dark launch", all the finished code is deployed to production as soon as it is ready. Code that should not be accessible is hidden behind a concept called a "feature flag". A feature flag is just a boolean condition that indicates whether the protected code is available to a certain customer.

This approach has a couple of benefits. First of all, if code is deployed when it is ready, it means the deployed changes will be a lot smaller. Small changes tend to have a lot of properties which will lead to less bugs overall:

Less chance of a merge conflict hell. If a developer wants to merge in just 5 lines, the chances are relatively small that someone also changed those lines in that same short time frame. Big feature branches that live for weeks or months tend to rack up a lot of changes that are bound to conflict with new features, bug fixes or refactors by other teams.
A lot easier to review by multiple developers. There's a joke in the programmer world that a 5-line pull request will get 10 change requests, while a 500-line pull request gets a quick and dirty LGTM.
A lot faster to QA. A 5 line change only has impact on a very limited set of functionality, allowing for a very narrowly scoped manual QA run.
A lot easier to debug when a bug does get through. The symptoms of the bug together with the 5 changed lines of code will make the issue obvious very quickly, compared to searching through thousands of changed lines of code and hidden side-effects.
A lot easier to roll back if a new bug turns out to fix right away. Rolling back one day worth of changes will have a lot less impact than rolling back a month or quarter of changes.

Small code changes lead to a much more stable system with happy engineers and customers.

As a second benefit, customers can get early access to certain features without opening the features up to everyone. With the early feedback from these key partners, the product team will know whether the feature is good or if it may need some quick adjustments. If there is an internal team drinking their own product champagne, early access for those teams and their feedback can even save the company from a public embarrassment when the feature is finally open to everyone. A happy product team and happy a customer success team that can prepare docs and training materials before the new features go live.

In the end, deploying your code on a daily basis allows you to release your features on a much slower cycle, in a controlled, stress free way.

If you want to get started with feature flags, your engineering team could build their own, or you can build on existing products like Unleash, Flagsmith and Launch Darkly.