Sum of the Parts… Not equal to the Whole

Christofer Hoff (@Beaker) and I had a short discussion on twitter the other day about the VMware Cloud Director (vCD) security guidance. We both felt it was a bit lite and missed the point of Secure Multi Tenancy. However, I feel even more strongly that people will implement what is in the vCD Guidance, vBlock Security Guidance, and the vSphere Hardening Guidance, and in effect have a completely insecure cloud. These three guides look at the problem as if they were singular entities and not as a whole.
This realization tied to Chad Sakac’s recent discussion on the 9/22 VMware Communities podcast leads me to believe that ‘good enough’ is no longer ‘good enough’ from a security perspective.  Chad discussed that there need only be the vCloud Director administrator and the vSphere administrator to do the daily heavy lifting. That there would no longer be the need for a security, network, storage, and system specific administrators. In other words, OPEX savings.Any C-Level reading or listening to this will think they can just let go their high paid experts in non-virtualization or cloud technologies. I think this will lead to having an insecure environment. I recently found and discussed several attacks and virtualization security with others in the field. Out of this discussion comes the following:

• It is now possible to use side-channel attacks against non-virtualization systems to actually gain low-level access to virtualization systems using a previously known but thought to be closed attack vector
• The virtualization administrators do not have the security expertise to ensure these non-virtualization subsystems are protected.
• Following the guides will not produce Secure Multi-Tenant public and possibly private cloud deployments.

Mike Foley (@mikefoley), Brad Hedlund (@bradhedlund) and I continued this discussion over the last week end on twitter. The issue at hand is that the vCloud Director is being marketed as producing a Secure Multi-Tenant cloud and that low-level access to hardware is no longer a security concern.
While I do agree that vCloud Director is a step in the right direction, I do not believe it provides Secure Multi-Tenancy except in a single instance. Outside this narrow view of the cloud, vCloud Director is no better or worse than anything else out there. What is the narrow view of the cloud?

vCloud Directory provides Secure Multi-Tenancy for private clouds hosted within the enterprise with only a single classification level.

Let us review what is required to for Secure Multi-Tenancy:

1. One Tenant cannot see another Tenant’s data (confidentiality).
2. One Tenant cannot impact the availability of another Tenant’s data (availability).
3. One Tenant cannot modify another Tenant’s data (integrity).
4. The Cloud Administrators cannot see, modify, or impact the availability of ANY Tenant’s data.
5. The Virtualization Administrators cannot see, modify, or impact the availability of ANY Tenant’s data.
6. The Hardware Administrators cannot see, modify, or impact the availability of ANY Tenant’s data.
7. The Tenant can dial in the level of security they require.

vCloud Director provides a solution for 1-3 but not for 4-6 and 7 is not yet possible. By their very nature Cloud, Virtualization, and hardware administrators have low-level access to any hypervisor. At this time they can see, modify, take, and destroy data within the Cloud without the Tenants even knowing. It is at this time impossible to prevent them from doing this. While there are tools that will audit for these actions, there are still ways an administrator can bypass any of the existing controls provided outside of the hypervisor.
Granted tools like HyTrust, Catbird, Reflex Systems, RSA Envision, and others provide audit capability, but they are not preventatives at this time. Yes, root password vaulting is important, as is centralized access control, ala HyTrus,t and real-time continuous auditing provided by Catbird. Yet, if these systems can be bypassed we now have a security issue. These tools are improving every day and eventually they will cover all avenues. And yes I believe a combination of these tools are necessary to cloud security. They are only a step in the proper direction for solving 4-6 above. Implementing #7 will take a bit more work.
Because we do not have solid answers for 4-7, any cloud, virtualization, or hardware administrator needs to have classification level to view any of the data residing within the cloud as they can do that now. In government speak these classifications are the clearance levels and outside the government, these are the confidentiality levels. In either case, your administrators still need the classification level to see any data within the system.
This last element is why I believe vCloud Director is great for private clouds hosted within the Enterprise and not with a cloud provider. Current Cloud security boils down to how well you TRUST the administrative team and organizations TRUST those they hire more than they TRUST those they do not. Unless, there are reasons to TRUST those cloud hosting facilities such as bonded administrators and those who have the appropriate clearance levels, etc. This means that Cloud providers must provide guarantees and conditions within their agreements that protect their administrators and at the same time the organizations who use their clouds.
We apply the vCloud Director security Guidance, vSphere Hardening Guides, and others to secure the parts of the environment, but when you put it all together, we do not secure everything. The most recent spate of attacks against non-virtualization subsystems to gain access to virtualization systems definitely implies The Sum of the Parts does not imply security.  We still need our security experts to vet the systems, check them after patches are applied, and to look at all subsystems not just the virtualization ones.
While the vCloud Administrator can handle the heavy lifting and day to day administration of virtual machines, behind the scenes needs to exist a team of security, network, storage, and system administrators that continually audit, provide the service catalog, and solve problems before they become disasters.  These ‘experts’ need to work together to form a pro-active team that when the problems do occur, and they will, they can be solved quickly with the proper incident response no matter the time of day.
The security, network, storage, and system specific experts need to be continuously available to work all security issues within the environment.