What your data team should be hiding from you - the database
The team spent 18 months replacing the database where all user data was stored. In doing so, they lowered their operating costs by 70+%. Most engineers who retrieved that data during the migration had no idea that anything was changing, or had changed. How is this possible?
Many portfolio companies face challenges around their data store. Because of the 1st rule of SaaS (don’t lose customer data), most engineers are hesitant to make major changes to the database. However, whether it’s to reduce the number of database technologies or to enable better response to the market by moving from monoliths to microservices, database abstraction can help.
When we abstract the database, engineers are still able to make calls to retrieve data, but don’t have to know as many details about the underlying database technology. That means they don’t have to know when to index, or about stored procedures. The database experts shift their expertise “left” and handle all the complexity behind the data retrieval API. In this way we don’t need to hire many database experts.
Variation Reduction
This is useful in a number of ways. Many portcos acquire a significant number of add-ons during the holding period. In the course they can acquire many different database technologies. While there may be good reasons for each add-on to have chosen that particular data storage technology, in aggregate, it becomes a major challenge.
Database experts are left trying to support many different technologies, often with only a passing familiarity with each of them and possibly some deep knowledge in one or two. In many cases, there is only one person in the organization who can support a particular technology and the advantages of team expertise are lost as they are spread too thin.
When the database is abstracted away from the users making the calls, it’s possible to start consolidating the number of technologies necessary to be supported without having a major impact on the overall development velocity. As the number of technologies consolidates down to a well supported few, the monitoring improves and performance increases, and the database-focused team has more time to render assistance to their various consumers without being run ragged.
Fracturing the Monolith
The same can be true when breaking apart a monolith. What often happens during these activities is the software is broken apart into services which all remain talking to a common database. In other words, a distributed monolith. For teams to truly go fast in a service oriented architecture, they need to be able to move quickly while maintaining contractual obligations to other services and a shared database can prevent this.
When we abstract the database behind an API itself, the team responsible for the service can satisfy all requests for their data. This can come from the original database. As the team learns more about the usage patterns and boundaries of the original database, they are more able to understand the best ways to break that piece off into something that they can manage, while still allowing the rest of the services to operate unimpeded. At the end of the migration, the other services will not be aware that they are now making requests serviced by a completely different data store.
Architecture Matters, Technology Doesn’t
Being able to service the data needs of the organization are paramount at software companies that provide data as a part of their service to their customers. However, neither the customers nor the engineers retrieving that data need to know what the underlying technology is that services those requests. When we are able to abstract away the data store, we enable a powerful architectural pattern that enables portfolio companies to successfully manage their challenges.