DevOps Across the Enterprise: Moving Past Dev and Ops
This post was originally published on the Librato Blog on March 24, 2015.
Last year, when talking to Patrick Debois about what the organizers of DevOpsDays Belgium were looking for in a talk, he told me he wanted to have conversations that move the movement forward. Having just finished a stint at a major enterprise corporation, Dave Zwieback suggested I talk about what it’s like to try and do DevOps at scale. This is my attempt to combine those two ideas.
Enterprise DevOps
I gave a talk last year with Reena Matthew at Gene Kim’s DevOps Enterprise Summit (#DOES14) called On the journey of an Enterprise transformation, Quality is still Job 1. I thought back to all the different speakers I’d seen at the Summit and what they were trying to accomplish. There seemed to be a notion that because DevOps practices were not pervasive throughout the entire organization, but maybe only in one silo or two, that they were doing something slightly different, they were doing Enterprise DevOps. Soon after there was an amazing blog post by Dave Roberts called “Enterprise DevOps” Doesn’t Make Sense. His post really spoke to me, especially this part:
“Enterprise DevOps doesn’t make sense because it confuses the forest for the trees. Those advocating for Enterprise DevOps make the mistake of focusing on specific tools and solutions (Jenkins, Kanban, etc.), then find them wanting for various (extremely valid) reasons in an enterprise context, and then reject DevOps in favor of Enterprise DevOps, which they suggest is somehow different.
But DevOps is about flow and continuous improvement, not about specific solutions. Flow and continuous improvement are equally applicable to a large enterprise as they are to an agile web startup. And if you miss that, you’re lost, regardless of the tools and solutions you choose.” - Dave Roberts
Dave was right: we are trying to accomplish the same goals, it’s still the same DevOps. My own view of DevOps in the enterprise (as opposed to Enterprise DevOps) has to do with differences in scale. If you were to work for a company in operations and they asked you to setup a monitoring solution, you would make very different choices if you were to monitor 60 cloud instances in AWS vs. 30,000 Dell servers spread across 6 data centers on 3 continents. In both cases, you are still setting up “monitoring”, but in each case, the infrastructure required to ensure a successful deployment differ radically. So it is, with DevOps across the enterprise.
Systems Thinking and Transparency
I believe there are two main criteria that are required when talking about the success of DevOps across the enterprise: systems thinking and transparency. I would argue that most of the things that we recognize at DevOps can likely be represented as manifestations of these two main ideas.
From systems thinking we get things like continuous integration pipelines, where one is trying to optimize the flow of software artifacts through the entire system. Configuration management is another instance where we are not tinkering with the configuration of an individual node anymore, but are making changes that can have global effects from a single code change. Even the DevOps emphasis on MTTR speaks to looking at ways to design our systems so that recovery is as close to instantaneous as possible, at least from the perspective of the end user.
Similarly, we can see examples of a focus on transparency in many DevOps principles. Shared revision control allows anyone with access to the code base to not only see any of the code running in production, but even see how it’s changed over time. All agile methodologies have their foundations in transparency, which allows us to gain insight into both bottlenecks and blockers and work to clear them as expeditiously as possible. Almost anything that fits under sharing in the CAMS model by Edwards and Willis, can once again be classified as transparency.
The 1st Way
Some may argue that we’ve not even solved the collaboration issues between Dev and Ops yet, but I think it’s time we start taking our DevOps discussion wider, to encompass the entire system. We need to build upon what we’ve already been able to establish to date.
Most students of DevOps are familiar with Gene Kim’s The Three Ways: The Principles Underpinning DevOps. I hold Gene in the highest regard, and I think we can do even better.
The traditional representation of the 1st way looks like this:
While I agree with Gene that the 1st way should focus on Systems Thinking, I feel the diagram only focuses on a narrow part of the system. The first way shows the business as being represented by Dev and the Customer represented by Ops. I think a more accurate representation of the system would look more like this:
Our system actually starts with the customer and ends with the customer. The purpose of a business is to determine what a customers wants, or what a customer would want, and to optimize the delivery of that product so that the customer will buy the product. That’s what we aim to accomplish in DevOps. It is just as important to make the lives of a million IT professionals better, a cause to which Gene Kim has dedicated a large part of his life. We know that burnt out, bitter, professionals do not perform at their peak, which means that making those lives better are an essential part of our system. Ultimately, we’re trying to enable our businesses to be successful, to turn a profit. That’s why we’re paid to show up to work every day. That’s why the systems thinking that we need to apply must encompass the entire system. Our systems do not merely start with Dev and end with Ops.
Traditionally when we’ve talked about DevOps, the typical cast of characters that are open for discussion are Dev, Ops, QE, Security, etc. - the technical disciplines. But, that is selling our new model short, because there are other departments in an enterprise that are just as, if not more critical for its success. I think it’s time to ignore the words Dev and Ops in DevOps. The term DevOps has again begun to acquire secondary meaning (see: I’m not a DevOps…Are you an Agile? - urandom Mangot ideas). We are at the point of moving to a post-DevOps world, a world where DevOps principles include other elements who are required to deliver in our system, groups like Sales and Product.
The System of Selling
Most people who have had even the slightest contact with a sales organization are familiar with the concept of the sales funnel. The idea is that a great many leads come in at the top of the funnel and as you move down, only the bona fide leads come out the end as actual sales.
I believe that the sales funnel is the top of the system that we’re trying to optimize with DevOps. At the top we are collecting all the consumers of our software product, or all the features that our customers might want. As they move through the system, they will encounter product who will try and make sense of all the things entering the system, but ultimately, sales is feeding the system from its inception. Without sales, we have no reason to build and maintain our complex distributed systems, and likewise, without our systems, they have nothing to sell. We are all part of the same system, trying to create or maintain a successful business.
Perhaps the best example I can think of for this is our classic DevOps favorite, Toyota Motor Systems. If you’ve read Toyota Kata by Mike Rother, you know that Toyota is famous for delivering a high quality product using Lean principles while trying to optimize the system for the specific challenges they face. But you also know that Toyota is well known for just in time manufacturing in that they don’t keep lots of inventory arourate a lean system.
Good salespeople are a key to the success of the entire system because they understand the customer. I remember a presentation from Flowcon 2013 where a product manager sat down and met with a number of his customers and when asked about it afterwards, replied “I could always tell you what they wanted, but I could never tell you why.” A good salesperson will always understand why. I have a good friend in sales and he always laughs at me when he hears me talk about “that DevOps stuff”. The notion that you wouldn’t automatically have people trying to communicate their needs and others trying to make those things a reality is a foreign concept. He told me once that within a minute of calling one of his clients on a Monday morning, he can tell based on the description of how the client’s child did during the past weekend’s Little League baseball game whether or not it is a good day to try and sell that client something. A good salesperson knows their clients well, and spends a lot of time trying to make sure they are successful with those clients, that is why they are an ideal head to our system.
That is not to say that in a large enterprise, there aren’t misunderstandings between the sales staff and the technical staff. Many of us have experienced the sales staff promising all varieties of “vaporware” promised by the said staff without truly consulting with the technical departments. What often winds up happening is some kind of awful “death march” where the IT professionals are working night after night and weekends just to meet some insane deadline and give the customer something that at least resembles what was promised. To me, that feels a lot like the “throwing it over the wall” from development to operations that we try and fight so hard in our DevOps initiatives. That kind of situation is not even close to optimizing our entire system, and therefore, is not DevOps.
A Story
You may be saying to yourself that what I described so far doesn’t sound like something that is applicable only in an enterprise, that these ideas could be applied to a startup as well. That is true, but remember, we’re talking about differences in scale. Hopefully you’ll see that the concepts of transparency and systems thinking that we discussed at the beginning, are crucial to the success of our DevOps implementation at scale. To further illustrate, here is a fictional story.
In our story we have a sales guy, Bern. For the purposes of our story, Bern is good at his job. He is very responsive to his customers and tries very hard to give them what they want. He would also never throw something over the wall to his technical staff. Bern has been hearing from a lot of his customers that they would like feature X in his SaaS product. Bern talks to product and finds out that many other customers besides his own are interested in feature X as well. So Bern and a member of the product team contact the head of Engineering and ask her how long it would take for her teams to develop that feature.
Our head of engineering, Tess, has a number of teams that practice agile methodologies. They estimate that they can deliver feature X in two sprints, a two weeks each, which means one month until it’s in the hands of the customers. Everyone agrees, and Bern, being very responsive, explains to the customer that they should be able to purchase or use this new feature in about a month.
One of Tess’ teams starts on the new feature at the beginning of their next iteration. Like all good scrum teams, at the end of their first iteration, they hold the sprint retrospective where they demo what they have so far. During the course of the sprint, they have also been moving the stories that are part of the feature across the wall. During this time, Bern has had the ability to track the progress of the stories through the system because of this transparency. On the day of the sprint retro, Tess’ team records a short (< 5 minute) demo of each of the features they’ve been developing, and posts the demos on the wiki page assigned to their team. Then they go home for the weekend.
Posting the demos for all to see accomplishes a number of things:
- Despite the fact that Bern is located on the other side of the planet from this dev team, he can view the demo and comment on it, make suggestions, etc., despite the geographical disparity
- As a matter of fact, anyone in the organization - sales, product, or otherwise - can view the demo
- By posting the demos, even if those stories are not complete for the features, the Dev team is able to get fast feedback (fail fast anyone?) so they do not hand over a feature at the end that was not what the customer wanted
- Bern can go to his customers and tell them that he’s seen an early version of the feature and that it looks great ( and that he perhaps made some suggestions for further improvement), so they understand the situation exactly
When Tess’ team is finally able to deliver the feature, it is exactly what the customer wanted. This is not just because her team, along with Operations, have spent a lot of time in the past few years implementing all kinds of optimizations to their system so they can do things like continuous delivery and feature flagging to enable the successful rollout of feature X. It is also because they required the correct input at the head of the system, to know what they should spend their time on building, in order to enable the business to satisfy the customer.
DevOps Across the Enterprise
DevOps across the enterprise does not throw away the work we’ve done to optimize delivery systems in Dev and Ops. Rather, it expands that work so that Dev and Ops are not working in isolation, nor are they the only part of the system that has been optimized. The systems thinking is a requirement in order to allow the business to determine what are the customers actual needs. The transparency is a requirement that allows the business to deliver solutions to those needs at scale.
It’s time to move beyond simply the Dev and the Ops of DevOps. It’s time to embrace the performance of the entire system with transparency like Toyota does in their manufacture of automobiles. Only then can we achieve our true DevOps goals.