As a delegate for Tech Field Day 6 in Boston, I was introduced to several virtualization and performance management tools from vKernel, NetApp, Solarwinds, Embotics, and a company still in stealth mode. With all these tools and products I noticed that each was not integrated into the roles and permissions of the underlying hypervisor management servers such as VMware vCenter, Citrix XenConsole, or Microsoft System Center. This lack of integration implies that a user with one set of authorizations just needs to switch tools to gain a greater or even lesser set of authorizations. This is not a good security posture and in fact could devolve any security to non-existent.We already have issues with the base hypervisor and its many ways to access and control management functions. Now we add to it other management tools that add yet another layer and method to access and control management functions. This was addressed within my previous article, Security of Performance and Management tools within the Virtual Environment, but now there are some new wrinkles. The main premise of this previous post is that all activity that touches the management server of the hypervisor or the hypervisor directly should be proxied through something like HyTrust or some other tool. So let’s look at the new wrinkles created by the same tools.
Service Accounts
vKernel, Solarwinds, and NetApp all use service accounts to access the virtual environment details to gather data, which is pretty standard and normal. However, each of these tools uses a directory service such as Active Directory just for authentication, not for authorization. So since the service account must have minimally read-only access to the entire environment, a user who is restricted to just one set of VMs could login to these tools and see all the VMs, Networks, Storage Devices, etc. In this way these tools lack basic multi-tenant capabilities and allow for data leakage of oft-critical details about an environment.
The tools themselves are being marketed as administrator-only, but are starting to expand to provide portlet functionality so that their output could be mashed-up with other tools. Solarwinds has this feature built in, which I find to be quite handy. Even so, the authorization restrictions built into VMware vCenter and the other management tools are completely ignored.
Delegate Users
Tools such as Embotics and vKernel also offer the ability to perform tasks on your behalf through their interface, such as reducing memory and such. However, they do this using a delegate user. Because they have used a delegate user, you now have an audit trail correlation issue. While this correlation issue already existed within VMware vCenter and other management tools, it gets worse when you add yet another delegate user. Figure 1 shows the break in the audit trail when you just have one delegate user. You can audit data within vCenter but you just do not know who did what, when, where, or how on the host. The only recourse on a host is currently to correlate the logfiles between vCenter and the Host and if the time servers do not match this correlation is impossible.
Conclusion
In many cases, auditing may not be difficult, depending on the number of simultaneous actions that take place, however, when faced with a forensic study or compliance regulation that must determine who did what when where and how, such a broken audit trail makes it impossible. Tie this to the possibility of data leakage and destruction from tools having the incorrect authorizations and you end up with a system that could be heavily impacted with the possibility of no way to track back who did what when where and how. Not only would we be insecure but in violation of regulatory compliance.
I call on the tool vendors to work with the hypervisor vendors to solve this problem. There must be not only centralized authentication (which we have with directory services) but centralized authorizations in order for a multi-tenant environment to succeed. In addition, we need to be concerned about tools that span hypervisors which complexes this quite a bit. We also need audit capability all the way down the stack so that we know beyond any shadow of a doubt who did what when where and how.
Additional Info: There is at least one feature request with VMware to address their hypervisor/vCenter interactions.
Comments are closed.
Interesting read, Here recently we didnt do so well with out pen testing on the virtual environments we host. Security has always been by far the largest concern with going virtual anything. However, it seems often than not that this is an after thought. Even in my current position it’s still an after thought. People dont understand if your virtual environment is comprimised its not just one server they are getting access too. Its the entire virtual environment…..
Hello Chad,
I think that depends on the compromise but for Management and Performance Tools for the virtual environment this is definitely the case. I wrote about a method to aid with that a while back as well. I do not cover the proper defense in depth required either, which is absolutely required. There is quite a bit you can do today that you could not do even a year ago to improve overall security.
Boils down to including all this in the architecture from the beginning.
Best regards,
Edward
Check out DynamicOps. We actually take care of this piece as an operations virtualization layer which unifies all of these tools under a single, user-aware governance model.
C
Hello Chad,
Many tools claim to do that such as Embotics, but all need to speak to VMware vCenter who at this time is on of the final arbiter of who has access to what. However, if I my view is from the virtualization host to determine who did what on that host, then all the delegate users do is make that a feat of correlation which is not necessarily 100%. All these are good tools, but they are really too far up the stack to contain 100% of the audit trail as well as contain 100% of all authorizations and actions.
Track how DynamicOps works and as soon as it hits vCenter the audit trail stops. Can a user in DynamicOps see more than they would be able to see in vCenter. It is possible. For cross-hypervisor tools, the same applies to XenConsole and SCVVM.
Best regards,
Edward L. Haletky
Edward brings up some interesting observations about the implementation of proprietary RBAC by numerous virtualization and cloud management vendors and the need for the industry (as a whole) to be aware of forensic study, audit and compliance. Embotics agrees with Edward that the industry needs to evolve over time and we look forward to participating in the evolution.
It is important to highlight that most all vendors are quite frankly in the same boat, including VMware who have their own proprietary RBAC (see Feb 2011 Virtualization Security Podcast: http://www.talkshoe.com/talkshoe/web/talkCast.jsp?masterId=34217&cmd=tc)
As with most product features, ISV implementations of RBAC are routinely driven by customer requirements and ultimately what they deem important. Instead of focusing on whose RBAC system is the authoritative one to use, we thought we would shift the focus of this discussion to our customers’ requirements.
In Embotics V-Commander, we have responded to customers who wanted the ability to optionally proxy out admin tasks across multiple hypervisors and management platforms. This makes perfect operational sense, but may leave ‘security sense’ up for debate. Having reviewed the audit, logging, and authorization requirements that Edward has outlined, we do have several alternative solutions suited for a range of environments. We will actively work with interested customers and platform providers on implementing a solution that addresses both operational needs and security requirements within their environment.
It was our pleasure to meet with Edward and we appreciate his insight and guidance that he has provided the industry. We would also like to thank Edward for spending time on the phone providing clarifications to us, and we look forward to working with him again.