I have seen many DevOps initiatives at various levels of maturity. One common pattern in successful ones is that they have made progress in automating the build process or the provisioning of environments. Both of these accomplishments are necessary and worthy of focus, but the efforts don’t stop there. The following is a list of things to consider that can take your DevOps initiative to the next level.
Add Inspection to the Build Process
Once the build process is automated, start looking for more opportunities to replace manual review processes with automation. I have seen development teams quickly turn business requirements into workable code, but review gates for things like code quality, security, and governance require setting up meetings for human code review to ensure that standards are met. These meetings are time-consuming, hard to schedule, and often single-threaded, resulting in huge delays in getting code out the door. A lot of tools are available, both open source and commercial, that allow the development team to set up rules and add automated code inspection capabilities to the build process. The output of these tools consist of a score (usually from 1 to 100) and feedback about the code broken out by severity code. The build process can be set up to fail if the score does not meet company standards. The entire scan can be archived as an artifact and used for auditing and post-mortem reviews.
In many companies, getting the security and governance teams on board can be a major challenge. In companies where I have seen these tools implemented, the security and governance teams have been very engaged. They bought in because they like the fact that they can implement rules that can be easily enforced in the build process. In fact, one security team I met with loved this process so much that they started demanding the same functionality from all development teams, which drove up adoption of DevOps overall.
Create Standards for Logging
By now, most IT professionals understand the importance of a centralized logging system. In the cloud, where servers come and go as systems scale up and down, it has become standard practice to send all logs to a centralized location. Unfortunately, often very little thought is put into creating standards to ensure that consistent and actionable log data is created. I learned this the hard way several years ago. We had a big Logstash implementation that we wrote logs to from every component of our architecture. Unfortunately, we didn’t think about standardizing severity codes, formats, or error messages, and we got a classic “garbage in, garbage out” scenario. For example, when an auditor asked me to prove that we could detect unauthorized access, it wasn’t as simple as searching for a single error message. We didn’t have any standards, so we had to run a variety of queries looking for strings like “access,” “unauthorized,” and “denied,” until we knew what all of the different permutations were.
In order to make log data valuable, I strongly recommend establishing logging standards. Start with adopting a standard set of error codes (RFC 5424), create a standard log format that can be enforced via an API, and consider adding metadata to the virtual infrastructure that can be automatically appended to log files. Examples of metadata include environment name (dev, test, stage, prod), business unit, customer, blueprint name/version, and build number. With the addition of meaningful technical and business metadata, you can dice and slice log data into many different views and learn a great deal about the behavior of the overall system.
Change Incentives to Drive Desired Behaviors
Why are most companies embracing DevOps? Most want to improve speed to market, quality, and reliability. DevOps also promises to improve customer satisfaction, productivity, and profitability. Whatever the drivers are for your company, make sure that people doing DevOps are incented to support the outcomes matching those drivers.
If your company is still counting function points and lines of code, or simply relying on an annual review process that goes through the motions but drives no real behavior, it is time to change. Many people have written about the difficulty of changing culture. One reason why culture change is so hard is because it requires people to think and perform in a different manner than before. If they are still incented to think and perform the old way, how do you expect them to adapt to the new way? Work with the leadership team and your human resources department to adjust the incentives so that the desired behaviors are encouraged, rewarded, and recognized.
Evaluate Your KPIs
Many companies track key performance indicators (KPIs) to measure how both people and systems are performing. KPIs are often designed within silos. Development teams may have one set of KPIs, QA another, operations theirs, security a different set, and so on. Often the KPIs across silos conflict with each other. The problem with silo-based KPIs is that they serve each silo’s own goals, but they often don’t align with the goals of the product. For example, speed to market is rarely a goal found in a security or governance silo. Those silos are usually more focused on minimizing risks, driving standardization, managing licensing, and so on. In the DevOps model, those KPIs are still relevant, but they should not occur at the expense of agility and customer satisfaction.
In legacy development shops, KPIs often focused on metrics like lines of code, story points, or velocity to track how productive development teams are. These metrics may be nice to know for a scrum master trying to tweak their Agile process, but they are meaningless in terms of driving automation, agility, quality, and customer satisfaction. Look at KPIs that tell you how your automation processes are improving the overall delivery capabilities (frequency of builds, build success rates, build time, etc.) Metrics like MTTR (mean time to repair) are important and measure your responsiveness to issues, problems, and incidents. I could go on here, but you get the point. Track metrics that look at the systems as a whole across the entire team instead of tracking silo-based KPIs that create friction between teams.
Promote Transparency
One of the goals of DevOps is to promote a culture of continuous improvement. One way to encourage this behavior is to create an open and honest environment in which people are not afraid to make mistakes and welcome experimentation. We talked about incentives and KPIs. The tracking of this information should be available for everyone to see. I have worked in environments where leadership berated team members when problems occurred. This created a culture in which team members would minimize communications in an effort to resolve issues before they were known to others. When this occurs, nobody shares lessons learned, and nobody helps each other. In fact, finger-pointing becomes the norm because nobody wants to be a part of the public “flogging” that takes place when things go bad. With DevOps, it is all about teamwork and collaboration. We are all in this together with a common mission (hopefully driven by the right incentives and KPIs). Transparent organizations share information, good and bad, to both internal and external customers. This creates a level of trust. Trust leads teamwork, which improves problem-solving. Successful teams hold blameless post-mortems. People discuss openly what went right and what went wrong, and they put plans in place to make improvements for the next release.
Evangelize
It is one thing for a small team to embrace DevOps, but it is an entirely different challenge to adopt DevOps at scale across the entire organization. To desire change, people must know WIIFM—“what’s in it for me?” To scale DevOps, you must always be selling. It is important to have a communication plan (some call it a marketing plan) to spread the word across the organization. It is almost like running for public office. You need to target your message to match the audience you are trying to reach. Management does not respond well to buzzwords like DevOps. Talk to them about the business value, such as speed to market and increased agility and productivity, and back it up with metrics that show how it impacts the bottom line. Business partners and product owners want to see how DevOps helps them get more features into the market, deliver more frequently, improve customer satisfaction, etc. Developers, admins, security, governance, and operations people all see the world differently. Show them how DevOps improves their world in terms they can relate to.
Use multiple mediums to evangelize as well. Write a company blog, contribute to the company newsletter, hold brown bags, hackathons, meetups, and demo days. Post statistics all over the company walls. Celebrate successful implementations, and have management hand out gift cards, dinners, or whatever token of appreciation makes sense. Evangelize externally, too. Speak at conferences, attend meetups and webinars. Tell the world what you are doing. It will get you feedback and is a great way to recruit good talent. People want to work at a company that is doing good things.
Summary
Over the last year, I have seen a huge increase in the number of companies embracing DevOps. Many companies are now moving from talking about DevOps to embracing DevOps. DevOps is a journey, not a destination. Many companies are early in their journey and are focusing mostly on automation of builds and infrastructure. DevOps is much more than automation. I have listed a few areas to focus on that can make a big difference in the long run. What other things have you seen and done that have made an impact? Please share.