Security Questions from VMworld 2013

While at VMworld 2013, I started to ask 5 security questions that have been bothering me for some time now. Some of these questions apparently have no answers currently and others only have operational answers, no technology. Security of a secure hybrid cloud is a mix of procedures, policies, operations, and technology. These questions are about various aspects of virtual and cloud environments that have been nagging at me for some time now as well as problems I have faced managing our own cloud instances. Perhaps you have questions you would like to add to the list, if so please share.

Security Question 1: Can we move security out of the way of a Virtual Storage Appliance?

Recently, I have been playing around with virtual storage appliances and noticed that I was not getting the throughput I expected. Now, the question is, if we mark a workloads as needing low latency disk access can security and other introspective features be bypassed as necessary? In the case of introspective security, the drivers are in the network and SCSI stacks by default. You still have to traverse these drivers unless we can short circuit them in some way. Storage requires low-latency access and if security tools impact that in any way, they will be ripped out and never used. So we need security tools that do not impact storage hypervisors and virtual storage appliances. Given how introspective technologies work today, they are either on (installed) or off (uninstalled). There seems to be no middle ground.
If you use a VSA, do you also use introspective security measures? If so, on the SCSI path or the network path?

Security Question 2: How can we prevent flash caching layers from caching encrypted data?

There has been a major push to improve overall storage by using flash cache and other caching layers, however, these present a unique security issue. Specifically around the caching of data that is normally encrypted at rest. Such a possibility is the encryption of PII or something else as delicate as encryption keys. Caching layers read through to the disk and put data either within a virtual machine (Condusiv, Flashsoft) or into the hypervisor or onto SSD (Proximal Data, Pernix Data). If the encryption is below these levels there is a good chance that caching software will now hold unencrypted data in a persistent location. If the encryption is above these levels then the data is encrypted even within the cache (such as use of Vormetric).
So for encrypted data, cache layers can present a store of persistent data that stores this data unencrypted which creates a new attack point. Is there a way to cache-bust based on encrypted data so it is never within the cache?

Security Question 3: How to implement location, user, and context aware security policies?

The secure hybrid cloud’s entry point is the end user computing device which not only is considered a ‘what you have’ portion of authentication but the location where you enter ‘what you know’. With the new iPhone 5S, that is rumoured to have fingerprint scanning technology, we can also pick up on an end user computing device ‘who you are’.  But that is not really sufficient as we may still not know who is really at the end of the device (depending on what is on, the application in use, etc.) we also should pick up location information so that the following example can be handled properly.

• The user is logging in on their device from China, this should give them access to a secure mailbox with only the ability to see the inbox and no other folders
• The user is logging in on their device from France, this should give them access to a mailbox plus folders located within a French data center
• The user is logging in on their device from the United States, this should give them access to their entire mail and five other applications
• The user is logging in on their device from their in-building office, this should give them access to everything

Today, we can only handle the last item properly as we currently do this by network location, but network location can be spoofed so it needs to be a combination of location and networking. In addition, we need to have better controls within applications not necessarily to an application. Location and context outside username and group is required going forward. Currently, only AFORE CypherX is looking within the application to apply security, but we still need to handle location and other context when we access our environments such as virtual desktops and the applications within the desktops. VMware NSX’s service composer may be a mechanism to pull other security contexts into a firewall type device.
How do you do context aware security now?

Security Question 4: How do we handle the delegate user problem?

There still exists a problem where we do not know who did what above the hypervisor and in general who actually issued the command that caused a problem in our virtual environment. With vSphere for example, we can look at logs from the hypervisor to determine who did an action, and if the action was issued by vCenter all we know is that vpxuser did something, not the actual user within vCenter. Furthermore, if vCenter is called from something like vCloud Director, Embotics, or other cloud portals all we know is the user that all those items delegate to within vCenter, and once more all we know at the hypervisor is that vpxuser requested an action, not the actual who. This has two issues:

• How to correlate up the stack to determine the real who that issued the action. This is currently a time based correlation, but since many actions can come in using many users in the same second, it is hard to correlate based on time unless every action is synchronized some how.
• How do we know that proper user handling has been implemented. In general, all programs that call into vCenter tend to use the same user name, when in reality each program needs its own service username, unique to that program. We still have a correlation problem but we gain the ability to at least correlate to specific programs such as vCloud Director or vCAC over both at the same time.

Security Question 5: How do you prevent direct logins to hypervisors?

Many of the seemingly esoteric encryption and other attacks require a direct login to the hypervisor to work. If we close off this avenue by limiting direct login to a hypervisor in a break glass situation, we mitigate the risk. Yet, virtualization administrators still think they need this level of access to do their jobs. I disagree, there needs to be operational changes to limit this level of access and to keep anyone from directly logging into a hypervisor.  What tasks still require direct hypervisor login capability? I have found only one, fix broken hardware. Since 5.1, and vDS rollback capability, there is no need to login to the host anymore to fix networking problems with vSphere. With Hyper-V and other hypervisors I still only think it is a break glass situation.

Concluding Thoughts

There is still many questions about virtualization, secure hybrid cloud, and cloud security out there. These questions are ones that have come up fairly recently given the technological advances being made. The real question is, however, how do we use our environments today? Why do we keep violating best practices, perhaps those also need to be adjusted. We need to apply policy, procedures, and operations changes on top of technology to effectively secure our hybrid clouds.
How do you answer these 5 questions? Are there more you need answered as well?