As companies embrace the DevOps movement, they rely heavily on automation to improve the time to market for new features and services. DevOps is a long, never-ending journey with a goal of continuously improving the software delivery process, resulting in better products and services and, ultimately, happier customers. At the beginning of their DevOps journies, many companies focus on continuous integration (CI), in which they automate the build process. Automated testing is implemented so that builds will fail if any changes fail the baseline tests. The idea here is to never move bugs forward, catching them early in the process.

Once companies get good at implementing CI, continuous delivery (CD) is the logical next step. The idea with CD is to be able to deliver a clean, consistent environment along with the automated build. One of the biggest bottlenecks I see with clients is inconsistent environments. How often do we hear “it worked on my laptop” when a build fails in a testing environment? Too much time is wasted fixing environment issues, leading to lost productivity and a decrease in overall quality. CD aims at fixing all of this by ensuring that no matter what environment a build is deployed to, the configuration of that environment is always the same.

Companies that have implemented CD are usually in a position to deliver to production frequently, possibly even multiple times a day if necessary. This presents a challenge to the legacy methods of inspecting for security vulnerabilities. In the past, manual security reviews were a common method of inspecting software to protect against introducing new vulnerabilities into the production environment. Now that companies can deploy daily, manual inspection is no longer feasible. First, manual inspection doesn’t scale. By that, I mean that if a company has ten teams that can all deploy software each day, there is likely not a big enough security team in house that can respond to the constant need for security inspection. Even if there were enough security personnel to perform these inspections, it would be a full-time task for these people, and other tasks would fall to the wayside. Second, manual inspection would get in the way of the development teams and reduce the number of times they could deploy in a day, due to the constant need to stop the automation process to hold a meeting to manually inspect software.

A New Model Is Required

In order to add value and not be the biggest bottleneck in the software delivery process, security inspection needs to be embedded in the automation of the CI/CD process. Many of my clients use an open-source product called Black Duck, which inspects code for security vulnerabilities. Tools like Black Duck are added to the CI process to give the code a security vulnerability score. If the code does not meet or beat the desired security score set by the security team, the build fails. The security team also has the ability to configure the tool to update security rules and policies to keep up with the ever-changing world of security. Tools like Black Duck solve a whole host of problems. First, they eliminate the need for manual intervention by the security team. Development teams can now self-police to ensure that they are meeting the established security requirements while still meeting the speed to market requirements. Second, it makes the developers better at building secure software by providing fast feedback. Developers quickly learn how to build more secure code, because their builds fail whenever they create a new vulnerability. Third, this solves the scalability issue. With automation, security best practices can be applied across numerous teams at the pace at which these teams are able to push code.

Code Scanning Is Not Enough

Scanning code for security vulnerabilities is a necessity, but code scanning alone is not enough. What about the environments? The VMs that are being deployed are made up of a number of different technology stacks. There may be numerous flavors and versions of Linux and Windows, application servers (Apache, Tomcat, NGINX, etc.), database servers (MySQL, Oracle, MSSQL, etc.), NoSQL (MongoDB, Cassandra, Hadoop, etc.), application stacks (Java, .Net, Python, etc.), and many others. All of these servers need to be locked down, patched regularly, and continuously inspected for vulnerabilities. To make things even more challenging, some applications in the production environment are built to autoscale. That means that VMs come and go in an automated fashion, making it impossible to perform manual inspections.

The old methods of manually inspecting monitors and alerts just doesn’t cut it in today’s environments. Catching a vulnerability hours or days after it is introduced is extremely risky. Today’s attacks are very sophisticated and automated. As soon as a vulnerability is discovered, malicious code can quickly penetrate a system, run whatever scripts it needs to run, complete its malicious activity (steal data, destroy data, install agents, etc.), and then clean up and leave without being discovered if security checks are not continuously running on the systems. It only takes a few minutes or even seconds for a lot of damage to be done.

Companies can write their own software to continuously inspect their environments, but that comes with costs. First, keeping up with the latest security threats is a full-time job. Unless the company’s core competency is security, it is probably not feasible to dedicate a team of security experts to maintaining the software, tracking the latest security trends, and proactively analyzing petabytes of log data to try to stay one step ahead of the bad guys. Recently, a number of new companies have emerged to tackle this problem on your behalf. There are many advantages to using these new third-party continuous security solutions:

  1. Leverage the knowledge of industry experts in security
  2. Automatic updates to combat the latest security threats
  3. Take advantage of crowd-sourced knowledge (these solutions gather data across numerous enterprises and can see trends)
  4. Allow companies to focus on their core
  5. Reduce the need to hire a large team of security engineers
  6. Continuous updates of features and capabilities

Tim Prendergast is the founder and CEO of Evident.io, which offers a SaaS solution that continuously scans your AWS environments and provides deep analytics, alerting, and dashboarding capabilities. Tim turned me on to the crowd-sourcing capabilities of SaaS security solutions. Here is what he told me about the crowd-sourcing capabilities of Evident.io:

“There’s a lot of value in our customers’ building custom security controls in our platform and sharing them back for the community to leverage. The ability of our platform to interpret large security datasets and boil them down with algorithms built by our own data science team is a huge benefit to customers. We can not only look holistically at the broad customer base in an anonymized fashion to derive security insights, but also across all of a customer’s accounts and services throughout their organization. These layers of security intelligence really drive customer security excellence through thorough understanding of their cloud platform security practices.”

Another advantage of third-party solutions over building your own is that when you build your own, it takes a long time to accumulate enough features to cover a high level of threats. With third-party solutions, most known threats are already accounted for in the software, and new ones are addressed each day. In fact, some companies are able to drastically reduce their risks almost immediately after implementing these solutions.

“We often see customers reducing their threat profile by drastic amounts in their first week using the platform. A well-known retailer signed up and remediated 100% of their threats in their first thirty days, which was a huge step forward for their cloud security practice. Going from zero knowledge to total awareness plus guided remediation opens doors for progressive IT teams.”

Another interesting solution is Palerra. Palerra’s LORIC platform continuously scans various IaaS, PaaS, and SaaS cloud providers, such as AWS, Force.com, Box, Exchange, SharePoint, GitHub, and many others. It can detect suspicious patterns and raise alerts to head off malicious behavior. For example, the platform could detect that an employee is downloading a lot of information out of Salesforce.com and Box, which may mimic the behavior of an employee getting ready to leave an organization and take a lot of proprietary customer data and IP with them. This type of pattern matching is yet another example of why third-party solutions have advanced intelligence; they can analyze patterns across companies and across the globe to provide analytics that are impossible to derive from homegrown solutions.

Summary

As we move to a world in which companies are maturing with CI and CD, we need to take a fresh, new approach at security. Software and infrastructure are changing in our production environments at rates that we have never witnessed in any era before. Addressing security by manual inspection or by combing through reports is no longer an option. We need our security solutions to be as automated as our software deployments. The market is ripe for new security solutions. Companies like Evident.io and Palerra are two exciting solutions that have emerged in recent years. There are many more that are emerging as well. Keep an eye on this space, as I will try to highlight more interesting solutions as they emerge.