Puppet recently published its annual State of DevOps report. Previously, its 2014 report revealed that DevOps was becoming widely accepted in the enterprise. The 2015 report explained that as enterprises become more mature with DevOps, they become higher performing and release software more frequently with better quality. This year’s report has expanded the research to include employee loyalty, security, and lean product management.
Key Findings
- High-performing organizations are decisively outperforming their lower-performing peers in terms of throughput
- High performers have better employee loyalty, as measured by employee Net Promoter Score (eNPS)
- Improving quality is everyone’s job
- High performers spend fifty percent less time remediating security issues than low performers
- Taking an experimental approach to product development can improve your IT and organizational performance
- Undertaking a technology transformation initiative can produce sizeable returns for any organization
My Take
I have recapped the last three State of DevOps reports. Since the first of those reports, I have been working on DevOps initiatives with several large enterprises. Each year, the DevOps report goes deeper with its findings because enterprises are getting more mature with DevOps. I see the same thing in the consulting business. Customers in 2014 were asking “What is DevOps”? In 2015, they were asking for help with continuous integration (CI) and continuous delivery (CD). In 2016, I am seeing clients ask about DevOps transformation. They have come to understand that DevOps is bigger than automation and that they must evaluate their legacy processes and organization structures.
What has changed since last year? In my opinion, there are two major forces that are causing enterprises to look at the bigger picture.
C-Level Adoption of DevOps
Over the last few years, most DevOps initiatives have been grassroots driven. Development has been looking for a better and faster way to ship software, while operations has been looking for more reliability and less firefighting. DevOps has provided the opportunity to satisfy the needs of both groups. Grassroots initiatives have sprung up everywhere. Small teams have started drastically improving their overall performance, which has driven up interest across other teams. In many cases, enough momentum and results have been achieved that the C-level suite has taken notice.
As much as the C-level loves the results these teams produce, they have become concerned with security and lack of consistency. One of my clients had four different teams implement CI/CD, each in its own specific way and with its own suite of tools. Sure, it was great that they were getting code out the door quickly, but nobody knew if it was secure, compliant, or auditable. The other concern the C-level had was scalability. How could they scale DevOps across thousands of developers if there were four different ways to do it?
Grassroots initiatives are great for innovation and bringing DevOps into an organization, but in order to scale, DevOps must be driven from the top. The grassroots initiatives mostly focused on software builds and automating infrastructure. But to truly move the needle, enterprises must look beyond automation and improve the entire SDLC from project inception to customer service. DevOps is bigger than tools.
Need for Speed
The other big driver for the C-level is agility. Every industry is being disrupted by fast-moving companies, often startups, that are chipping away at market share with modern cloud architectures and agile delivery capabilities. C-levels are looking to cloud computing and DevOps as the way to become more agile to compete in the new age of computing.
Large enterprises are burdened with more constraints than young startups. They are buried in legacy technologies, processes, and cultures. Many enterprises are highly regulated and have auditors, Wall Street, and the press breathing down their necks each day. To make DevOps work in the enterprise is far more challenging than in a startup or SMB.
In order to get to market quickly, enterprises must find a way to reduce the massive numbers of meetings, reviews, processes, and bureaucracy that have evolved over time. I have seen a number of clients become really good at automating their build and deployment processes only to find out that it still takes forever to get code out the door. The reason for this is that they worked within their silo to improve their processes, but are still constrained by all the waste in the processes that come before and after their build. It is great that they can turn a business request around in a week, but if it still takes another two to four weeks to make it to production, there is little value to the business.
The DevOps report addresses these issues head on. Let’s discuss a few of these findings.
Security and Testing Must Shift Left
Two findings support the shift-left initiative. The first one is “everyone owns quality.” In the old days, the dev team threw code over the wall to the QA team, who got blamed for any and all defects that made it into production. This failed model must die. With DevOps, quality must be built in from the start. In order to automate the build and deployment process, testing must be automated as well. Test automation is a major component of CI. One of the guiding principles of CI is that defects should be caught in the build process and not be allowed to flow downstream.
Quality is not only a software issue. One of the biggest quality issues is the lack of consistent environments. That is why infrastructure automation and infrastructure as code becomes so important. It doesn’t matter how good the test automation or testing resources are if the production environment is configured differently than the development and testing environments.
Another key finding was “high performers spend fifty percent less time remediating security issues than low performers.” For high performers, security is an integral part of CD. High performers include security from sprint 0 as opposed to holding security reviews late in the lifecycle. Security controls and policies are also baked into the infrastructure, and security scans are run during the build process. Just as quality is everyone’s job, so is security.
Lean and Culture
The report also covered lean product management and culture transformation. To me, these two areas are what make and break a true DevOps initiative. Companies that think DevOps means automating the CI/CD pipeline are missing the boat. CI and CD are definitely valuable, but by themselves, they do not deliver the business value that DevOps promises.
If you have ever read Goldratt’s The Goal, you will be familiar with his take on bottlenecks. He recommends that after you remove a bottleneck, you find the next bottleneck and continually optimize the overall system. CI/CD addresses two bottlenecks: the build and deployment processes. Unfortunately, companies often have several other bottlenecks that are not technology related that never get addressed. For example, I have many clients that have anywhere from two weeks to three months of legacy processes that must be performed to initiate a new project. Tasks like getting project IDs created and entered into tracking systems, gaining approvals and credentials, architecture and security reviews, and others must be completed before a lick of code can be written.
Once the code is completed and ready to ship, another series of (usually manual) processes such as review gates, additional testing, DR planning, compliance checks, etc. add more wait time to the project.
This is where lean and culture change come into play. Many of these processes were put into place because releases were historically performed quarterly or biannually within large, monolithic applications. The risk of change in a large change set over an extended period of time is very high. The new model is to perform smaller changes more frequently. It is easy for a development team to embrace the new model, but if the rest of the organization is not on board, the dev team is still constrained by the legacy processes.
For a grassroots DevOps initiative, it makes sense to focus on the area of build and deploy. To become an agile, high-performing company, DevOps must be driven from the top down. The scope needs to expand to addressing the people, process, and technology bottlenecks across the entire SDLC and business. At scale, DevOps is bigger than dev and ops.
Summary
The State of DevOps report is the most comprehensive and quantitative research about DevOps on the web. If you review each of the last three annual reports, you will be able to see how enterprises have matured greatly over time. I believe a majority of enterprises have reached the understanding that DevOps is not just IT automation. DevOps is all about enabling an enterprise to become high performing and competitive in a world in which speed wins.