Show me the tests
When working with portfolio companies, the question of how and when to do testing comes up often. How to handle code that wasn’t written to be tested? How do we know if a release is ready for production? How do we know if a product meets the specification?
If we want to deliver software with speed and quality, automated testing is essential.
Release
I was sitting in a Go/No Go meeting for the release of a piece of software. That sentence should already tell you that at the time, this company was not releasing often. Every release was a “big bang” affair. The different groups involved (development, QA, Operations, etc.) were each given a chance to give a thumbs up or down to the release. When it was each leader’s turn, they would give a short explanation to explain how they felt about the release, and then cast their vote. Like a jury.
I’ve coached leaders on how to ask their employees about what makes them frustrated and what challenges they enjoy. I’ve encouraged team members to be open and honest with each other. For each of them to talk about their feelings. I think feelings are very important.
I think feelings are very important, unless we’re talking about whether or not to release software. In that case, show me the tests. Did the tests pass? Great. Do the deployment. Did the tests not all pass? Then we need to go back to the software and fix it until they do. There is little room for feelings in software release. Either the tests pass, or they don’t. If we don’t have tests, then that is the problem to solve.
There can be unit tests, integration tests, security tests, performance tests. At Salesforce we had literally hundreds of thousands of tests that ran against the software. We had teams dedicated to keeping the farms of testing infrastructure up and running. The more testing you do, the more confidence you will have in your software when it is released.
Part of the suite of tests to accept that the software is suitable for release are Acceptance tests…
Product Specifications
I’ve talked to, or worked with, a number of portfolio companies who are involved in financial calculations (taxes, payroll), etc. One of their concerns has been that they might release software that does these types of calculations incorrectly.
These are perfect candidates for tests.
Pre-Production In the development of the software, we know what we are trying to calculate. If we were to practice Test Driven Development (TDD), we would write the tests first and write the software until the tests pass.
If I need to calculate a 5% tax rate on $100, the software had better calculate $5 as the tax. If it does not, then that software is not ready for production. The tax, or payroll, or whatever is necessary in that feature are part of the product specification. Part of the product specification can be the acceptance tests as discussed above. If the acceptance tests are not accepted, then that software is not ready for release.
The developers need to write these tests as part of the process of development. They cannot wait until someone from QA has a chance to write them or execute them. Part of being a developer is writing tests. As my friend Jonathan Hall says “If you can’t write code, and verify that it’s doing what you expect it to do… you’re not a programmer. You’re a guesser.”
Production
If there is an especially tricky calculation that’s required, we can also “dark launch” the code into production as a way of testing. The code is released but not used for anything. The inputs and outputs of the calculation are written to some datastore that is not read by the primary application. This is to account for strange things that the user may input that we did not take into account in our acceptance tests.
We can run this dark launched code in production until we are satisfied and confident that the calculation performs as expected. Only then can we permit that code to be utilized for our customers.
It’s the only way
Releasing code without being able to prove that it works as intended is about feelings and guesswork. When delivering software quickly and safely, testing is the only way that we can feel confident. If the tests fail, the software needs to be continually improved until we can prove that the software works as advertised. It’s the only way to be sure.