Puppet Labs has published its annual State of DevOps report, and it is loaded with interesting information as always. Last year’s report brought home the point that DevOps was becoming widely accepted in the enterprise. This year’s report further validates that point and provides us with some interesting insights from surveying a wide variety of companies in different phases of their DevOps journey.

Key Findings

  • High-performing IT organizations deploy 30 times more frequently, with 200 times shorter lead times
  • At the same time, these organizations have 60 times fewer failures and recover 168 times faster
  • DevOps works whether the apps are greenfield, brownfield, or legacy
  • IT managers play a critical role in any DevOps transformation
  • Deployment pain can tell you a lot about your IT performance
  • DevOps (done right) can help prevent burnout

My Take

You can read the full report to get the researchers’ reasoning and statistics behind these findings. I would like to share my support of these findings both from leading software development teams as a CTO (2008–2012) and as a consultant working with a dozen or so Fortune 500 companies (2013–current) over the past seven years. All of my experience with DevOps has been in a cloud environment, but note that DevOps works in non-cloud environments also. I have seen examples of teams successfully using DevOps within a mainframe environment. I have also read recently of DevOps principles being applied in non-IT business settings. The fact is, working collaboratively with the drive to continuously improve is a formula that can work anywhere in any business.

High Performance

The first two bullets show what DevOps done right can look like. Deploying smaller batch sizes more frequently reduces risks and gives the business features faster. Ever since I read the 2014 report, I have stopped using the word “DevOps” when engaging with clients (as much as I can) and have focused instead on the goal, which is to become a high-performing culture. A lot of companies that I visit tend to define DevOps as automation, and that is where they go wrong. Automation is something that you do, but by itself it is not enough to make you become a high-performing organization. The companies that achieve faster time to market with better quality and reliability do so by making changes in the areas of people, process, and technology. Automation is just one piece of the technology equation. If a company is automating things without making the appropriate people and process changes, it is usually automating waste and not removing the necessary bottlenecks to achieve the desired goals.

I have seen teams that work around their bottlenecks. For example, several companies have a painful manual review process that is required to move code to production. On several occasions, I have seen companies automate the build pipeline so that they can get from code to stage in a press of a button, only to wait for a manual review gate in order to get to production. Then they reluctantly are forced to attend the review gate meetings. If the deployment is approved, another team must hit the button to push it to production. I have seen the time between the request for approval to the actual button being pressed take as much as two weeks. That’s two weeks for something that takes a few seconds. Even worse, I have seen instances where the deployment was rejected because certain requirements were not met. Then the team had to go back and fix the items that were called out and go through this long process again. So yes, automation is great, but collaboration and process optimization needs to be a part of it or you are just moving bottlenecks around.

IT Managers Play a Huge Role

I am very pleased to see this point being highlighted. This is true for any kind of transformation. I experienced this firsthand in a large SOA initiative years back. The most important thing for any transformation is that the middle managers, the ones whom the hands-on-keyboard resources report to, are totally brought into the vision and are all marching to the same drumbeat. When a middle manager is not on board, it is almost guaranteed that the team this person manages will be negatively influenced by their manager, thus creating resistance to change. Even if the middle managers are all on board, if they all have their own definition of DevOps and a different view of what the future state will look like, there is almost no way that the work each team does will align and produce an optimal solution in the long run.

The first things I ask my clients are “What problem are you trying to solve?” and “What does the desired future state look like?” You would be amazed at how challenging it is to get a group of people to agree on the answers to those questions. Once the team comes to an agreement on what they are trying to accomplish and what the organization will look like when it grows up, I then have them focus on creating some messaging around these answers and put a communication plan in place to constantly reinforce these messages.

The other key thing the middle managers must do is remove roadblocks. These roadblock removals can be things like breaking down communication barriers between teams, providing tools, reorganizing teams, changing processes, settling disputes, making decisions, and anything else that moves the ball forward.

Measuring Success

The report points out that you can tell how your DevOps initiative is progressing by looking at the amount of pain you feel doing deployments. I think that is an excellent indicator of how well the team has applied technology and process changes to the problem. I think an additional indicator is measuring the pain the customers are feeling. It is one thing to get better at deploying software. It is another thing to deploy software that meets the customers’ needs. I recommend establishing a baseline set of metrics to measure the customers’ expectations and satisfaction rate and then constantly collaborate with those customers to ensure that the progress being made has a positive impact on them. After all, the reason we are getting paid is to provide goods and services for customers to consume.

A true DevOps culture is one that is metrics-driven and seeks continuous improvement. To continuously improve, one must  capture relevant metrics and constantly review how any decisions or changes to people, process, and technology impact those metrics. This requires a culture that is transparent and encourages experimentation. Teams must feel empowered to try different approaches, measure each approach’s impact, and quickly adjust based on the findings. A metrics-driven organization designs with metrics in mind. These organizations think about what type of metrics the applications should generate in support of their continuous improvement efforts.

DevOps and Morale

The report points out that in high-performance companies, DevOps can prevent burnout. I totally support this statement. In fact, there is a huge risk for companies today if they are not moving toward high-performing DevOps culture. I have seen talented engineers leave companies because things like long provisioning times or painful manual stop gates have become a huge source of frustration. These engineers realize that many companies have solved these age-old problems and that job satisfaction and the feeling of accomplishment can be found at organizations that have embraced DevOps. People just want to get work done and make a difference. Waiting four weeks for a VM in the year 2015 is insane. Waiting two weeks for another team with no knowledge of your code to push a button to deploy it does not create job satisfaction for anybody.

The biggest boost to morale is that when DevOps is done right, there is less firefighting and more innovation. Engineers like to create. When we aren’t running around at 4 am fixing down systems we can create some pretty awesome stuff. High-performing companies spend more time creating and less time fixing. In fact, many can deploy at almost any hour of the day. How much happier would a developer or admin be if they didn’t have to deploy on nights and weekends all the time?

The downside is that DevOps done wrong can be even a bigger hit to morale. People jump on the DevOps bandwagon with such high hopes, but all too often it turns into just another failed attempt at chasing the next buzzword. I have seen far too many “DevOps teams” loaded with “DevOps engineers” who accomplish nothing more than creating yet another silo and writing a ton of automation scripts that help nobody other than themselves. I have sat in meetings with “DevOps teams” made up of only admins. When I ask them where the developers are, they say they don’t talk to them. I have sat in meetings with “DevOps teams” made up of only developers. When I ask them where the admins are, they say they don’t need them anymore. Folks, that is not DevOps, and DevOps != automation.

Summary

DevOps has come a long way in the four years during which Puppet Labs has produced this annual report. I encourage you to look back at the previous reports to see how far we have come. But we still have a long, long way to go. The good news is I believe DevOps has moved far past the hype phase, which means enterprises are gradually getting better at putting it into practice. I look forward to next year’s report. DevOps is a long journey, not a destination. It will be exciting to see the level of maturity that the early adopters reach by next year and the business benefits that the advanced levels of maturity produce.