Many are in the dark about Dark Launching
It was a wild year. I did a large AI Readiness Assessments on dozens of portfolio companies for a private equity firm along several different axes. Part of the thesis for the project is that companies that are good at delivering software, will be good at software delivery with AI. The DORA research bears this out. We ask a lot of questions about the SDLC and two really jump out:
- Do you use feature flags?
- Do you practice dark launching?
Astute readers will notice that these two capabilities are related.
Because AI tools allow us to develop much more code, much faster, and because it’s essential that we are able to quickly get feedback on the things we develop, these two techniques are now crucial parts of the SDLC.
Most people understand feature flags, or the ability to turn features on and off for a percentage of traffic, or a defined group of customers. Some people get this confused with licensing (which is a totally different thing), some try to do this in the database (bad idea), and some think it’s only about features (instead of risk, debt, and defects as well).
But almost no one understands Dark Launching. Most people think it’s having an alpha or beta group of testers or some other kind of early release mechanism for certain customers. It is not.
To help clear the air, below are three iconic examples of dark launching. By the time you finish this essay, you will know exactly what dark launching is and is not, and how best to leverage it for AI and other capabilities at your portco.
From Ruby to Java
The portfolio company was doing a lot of traffic. For many years they had been using the same webservers to ingest the data and route it to the proper application layer for processing. As the company grew, the webserver tier got bigger and bigger. The Ruby code that had been doing the routing had done a great job, but the costs were rising with success.
They decided to rewrite the Ruby tier in Java. The processing savings meant they would be able to reduce the size of the Ruby tier by 80%! But the software had been running for years and edge case after edge case in the data had been accounted for. Almost every data packet was processed smoothly and without hesitation.
As part of the rewrite they tee’d off a copy of the data stream to run through the Java software. The switch was controlled by a feature flag so they could turn this stream on and off at will. What did they do with the data processed by the Java code? Nothing, it was dropped on the floor.
For months, they watched the output of the Java code and compared it with that of the Ruby code. They tweaked, they compared, they optimized. When they were finally satisfied, they shifted a portion of the traffic to the Java code to actually be used. If there were problems, they shifted it back.
When they were finally satisfied with the results, they turned off the entire Ruby tier. They had successfully, seamlessly launched a new smaller tier using dark launching.
Facebook Messenger
Facebook had plans for a big new feature for the platform that took a lot of planning. Often even planning is not enough. Just thinking longer and harder about a complex problem is unlikely to give you the best answer. The only way to really know is to collect data, analyze it, and adjust and learn. Then repeat the process.
Facebook wanted to launch a Messenger feature to allow their hundreds of millions of users to communicate in real-time over the Internet. But how could they be sure that if they launched such a feature it would work? Sure, they could try with a small group of people to be the beta testers. But beta testers would never have enough scale for the volume of messages and communication required for a real test.
So they embedded some code in the browser of every Facebook user. The code exchanged asynchronous messages with other browsers. Soon browsers all over the Internet were exchanging messages. The users had no idea that these synthetic messages were being sent. They were in the dark.
After a year of testing, tweaking, and improving, they were finally able to turn on the new feature. Other than perhaps some minor hiccups, the product was launched successfully. They had successfully, seamlessly launched a brand new capability using dark launching.
Storage Savings
The portfolio company had been running successfully for years using an open source database to store their customer data. Then they were purchased by private equity. The pressure on EBITDA was real and the storage costs amounted to the vast majority of their costs.
A paper was published that described a new way of storing that type of data that was compelling. If the claims in the paper were true, they could reduce costs by more than 50%, which would have a tremendous effect on their EBITDA number. But there was no source code included in the paper, and no open source libraries. Just a description of how it all worked.
For the next 18 months, the portco started building the new storage system piece by piece. Some pieces were to validate the claims in the paper. Some pieces were to add their own additions and creations. Caches were added. New infrastructure layers were deployed.
Throughout all of this, the data continued to be stored in the open source database just as it always had been. Customers had absolutely no idea that the storage paradigm was being iterated upon. In fact, because the storage layer was hidden from the front end behind a data abstraction layer, most of the developers didn’t know the storage paradigm was being iterated upon!
Over time, both the reading and writing functionality was tested. Developers would come into work and flip on the feature flag that allowed the data to flow to and from the new system, and when they would leave for the day, they would flip it off.
Day after day, until finally 18 months later, the entire system had been replaced. Conservative estimates are that they had lowered their costs by 70%! They had successfully, seamlessly launched a brand new storage backend using dark launching.
The Reality
The reality is that dark launching is not something a company uses every day. It’s not like feature flagging where we run many experiments simultaneously and at the end, they are retired. But dark launching can be used in conjunction with feature flags as a specialized technique for special circumstances.
As part of the AI readiness assessment, we were looking for how well positioned companies were to take advantage of AI in development and in the product. Because AI gives us the ability to develop faster, and experiment more, there are much more ambitious projects that can be imagined. Maybe something as ambitious as rewriting an entire storage backend, rolling out a new capability to millions or billions of uses, or even gaining efficiency in an old tried and true process. But if you’re going to go after those ambitious projects, it helps to have the right tools in your toolkit.
How have you used dark launching?