“To Test, or Not to Test,” that is the question. Or more to the point, do we force our customers to be beta testers without asking them, or do we do testing up front as part of our Agile Cloud Development practices? Too often, I feel like I am a beta tester and not the user of a web-scale application. Because we move so fast from coding to production, two things get hammered along the way: real testing, and testing for security! This has got to change, and there is a way.
In traditional waterfall approaches to software development, developers would unit test, QA would systems test, and somewhere between the two, many of the serious bugs would be found. In web-scale agile and DevOps movements, I often see developers doing unit tests and then an A/B switch being used so that a subset of the populace gets the new code while the rest get the old code. In essence, those users in the population with the new code end up being beta testers. Now, having been a beta tester, I know there is often some recompense, such as free software or payment; these folks, on the other hand, are being required to use possibly faulty software without even knowing it.
Yet, I have heard time and time again that we just cannot generate the load necessary to test an application outside of production. That is short-sighted commentary. Telcos have been finding ways to test things at scale for quite a while before their user base has even seen the new code. Yet, I will grant you that this can be fairly expensive. So, what change is required?
The figure to the right depicts the traditional waterfall approach to development and testing: each team gives feedback to development and architecture and then passes their work on to the next team once concerns have been satisfied. The goal is to have well-tested code before things get into production, which offers the advantage of not having your customers become your beta testers. However, because humans are involved, and they make mistakes, security problems eventually wind up in production. This was and still is a major issue with this approach, unless those employing it have security tests that are a part of each phase. If they do, then that development shop has the right attitude. Often, the problem is not a lack of testing for security problems, but that results are misunderstood, or tests are misunderstood and either fall into disuse or test the wrong things. This is where security testing tools such as Metasploit, Core Impact, and others come into their own. However, you still need skilled practitioners, and those are far and few between. Regression testing is big with this approach, involving more tests being written without dropping or even looking at older tests. This approach also often requires expensive processes, hardware, and people.
The next style of development I see is the shortened waterfall, in which most of the testing is done in production by customers after perhaps a very short regression test that does not do more than test for crashes. A crash is bad, but security issues are not part of regression. This approach moves the final QA into production, where the real load for the application can be felt for the first time. This makes the end users the beta testers, with or without their knowledge. However, it allows heavily instrumented environments using performance management tools such as AppNeta, TeamQuest, Cirba, New Relic, LogicMonitor, ExtraHop, and the like to provide incredibly useful information and to feed that information back to development. In addition, end user experience tools, such as from Aternity, help in determining whether the user experience is bad. This approach is often the first attempt at Agile Development. Yet, we lose rigorous testing in favor of speed and monitoring for issues.
In essence, the developer would write code, while those in charge of virtual and cloud environments would script deployment of their virtual assets, including networking, storage, etc., all via APIs. The security team would write its own tests and required changes to the infrastructure, while the testing team would write tests to be automated.
This approach mimics the current thoughts around large-scale web applications. When they are that large, deployment by hand is just not possible. However, deployment using scripting is possible. In effect, infrastructure or deployment of your assets becomes nothing more than code. It behooves us to also make buckets for security and testing, so that they become nothing more than code as well: Infrastructure as Code, Security as Code, and Testing as Code. Then, the automation server would deploy and test at the same time. If an egregious error occurred, development would be notified, deployment would stop, and Production A would revert back to the original production environment.