Security of Performance and Management tools within the Virtual Environment

One of the basic tenants of virtualization security is to protect the management components of your virtualization hosts by placing these all important components on a separate network. These components often include management servers such as SCOM, vCenter, XenCenter, VirtManager, etc. as well as the management appliances of your virtualization hosts. In essence, the use of a properly configured, firewalled, and monitored virtualization management network would be the simplest and most effective security measure that can be made to day within any virtual environment. A message shared by Citrix, VMware, myself, and many others.
The problem is that not everything is as black and white as security folks desire. If we implement performance and other management tools, we often need to expose part of our all important virtualization management network to others. But how do we do this safely, securely, with minimal impact to usability? Why do we need to this is also another question. You just have to take one look at the Virtualization ASsessment TOolkit (Vasto) to realize the importance of this security requirement. But the question still exists, how do you implement other necessary tools within your virtual environment without impacting usability? Which we discussed on the May 5th Virtualization Security Podcast.One option is to limit access to just those who are virtualization administrators but this would greatly impact usability when the information is required by a developer, the network operations center, needs to be delivered to managers, or you outsource your management as well as many other cases. The solution is to think about how the tools will be used and design something appropriate that does not make swiss-cheese of your carefully prepared virtualization administration network. As an example I would like to look at the following classifications of tools:

• Administrator only tools (VMware vCenter, Citrix XenCenter, Microsoft System Center for Virtualization)
• Application Performance Management Tools (VMware AppSpeed, New Relic RPM, AppDynamics, BlueStripe, dynaTrace, Coradiant, etc.)
• Network and Security Tools (Netwitness, any SIEM, HyTrust, Catbird, Reflex Systems vTrust, Trend Micro Deep Security, etc.)
• Network and Server Operations Tools (VMware vCenter Operations, VMTurbo, vKernel, VMware Capacity IQ, Veeam Reporter, Reflex Systems VMC, SolarWinds, NetApp (BalancePoint), Netuitive, Xangati, Zenoss, etc.)
• Cloud Tools (DynamicOps, Embotics, Abiquo, Nimbula, VMware vCloud Director, RightScale, rPath, Trend Micro Secure Cloud, etc.)

While some may argue where the tools I mentioned under Network Operations should exist as their own classification, but they represent a class of tools that work across many layers, and therefore belong together. In addition, some may argue that their product is really a cloud tool, but once more these are tools that work across many layers but also purposefully span multiple locations to provide hybrid and other cloud functionality.
The most important element is first creating a virtualization management network as depicted in Figure 1. This is by far the lowest hanging fruit of virtualization security and is by far the easiest and least expensive security option you can implement. This has been mentioned many times and was once more brought up at InfoSec World 2011’s Summit on Cloud and Virtualization Security. There are four areas to Figure 1 that should be broken out:

1. Other Network – This is your physical and perhaps virtual network where your virtualization management endpoints live, which could be the administrator’s laptop, desktop, blackberry, iPad, or nearly any smart phone.
2. Admin Network – This is the first stage of an administrative network where you place elements like VMware vCMA (to proxy your smart phone and iPad tools into the virtualization management network), Administrator Jump Machines (full blown VMs with the necessary management tools installed), your SIEM, and any tools that manage your other virtual environment security mechanisms.
3. Proxied Admin Network – This could be combined with the Admin Network for those not using VMware vSphere or some other Authentication and Authorization control mechanism such as the HyTrust appliance. My personal preference is to segregate this network from the Admin Network. This network controls access to the management server (VMware vCenter, SCVVM, XenCenter, etc.) and the actual virtualization host’s built-in management appliances (Service Console, Primary Domain, Parent Domain, or the Busybox shell for ESXi)
4. Storage Network (Transport) – This is where all your storage access goes whether it is from the hypervisor directly or via the virtualization host’s built-in management appliances or directly from a virtual machine. Since storage is often overlooked from a security perspective it is important to include it.

In Figure 1, we split the administrative network in two using the HyTrust Appliance to ensure all authentication and authorization (A&A) is the same across all access points into the management server (in this case VMware vCenter) and the built-in management appliances of the virtualization hosts (in this case ESX and ESXi).   A control for A&A is nearly always required within any secure virtualization management network as there are many ways to directly access the management components of the virtual environment, and without control your virtual environment is at serious risk. Just take a look at Vasto if you think otherwise. If you cannot use the HyTrust Appliance I would suggest you instead use a firewall that locks down access to these all important systems by IP or Mac address to only those machines within the Admin Network or the first stage of the virtualization administrative network.
In Figure 1, we already placed several security tools into the mix, and these are in the first stage of our administrative network but do they really belong there. For a small environment, I would say yes they do, but for a larger environment we may have some other concerns. But before we go further, we have to discuss the first stage administrative firewall a bit more.
This firewall currently allows only the following elements through:

• Active Directory queries
• DNS queries
• Port 443 (required for vCMA, others may apply for other tools)
• Port 80 (for VMware and other Update Managers locked down to destination)
• RDP with the highest level of security

Given the small number of required ports it would be extremely easy to also lock this firewall down to specific IP or Mac addresses of the devices that will connect to it. Port 443 and 80 could also be redirected through a proxy to ensure nothing malicious is happening over those ports. The key is to limit access to this network as much as possible and definitely do not make this particular firewall your edge firewall to the internet. It should be buried deep inside limiting who can access it.
So given that we need to limit who can access this all important network how can we allow our security, application development, networking, storage, and other teams access the network securely so that their performance management and other management tools still work.  There are several ways to do this:

1. Give everyone a jump machine that needs that level of access but limit what tools are installed on each jump machine. I.e. if you are a developer, you could jump to your specific machine and then only access tools like VMware AppSpeed.
2. Use tools that do not require direct access to the virtual environment management network such as New Relic which installs onto the virtual machine, but reports to an external server over an encrypted channel which means any development machine using this would also need to have access to the internet to send the data.
3. Use a Portal that has a Mash-Up capability to display the necessary information such as Reflex Systems VMC Portlets, and VMware vCenter Operations Enterprise Edition (the old Integrien product).
4. Use only web based tools and proxy all requests through a reverse proxy server
5. Or place them outside the virtualization management network completely

Now some of these are doable if you use the proper tool, but #5 is just not feasible as nearly every tool such as vCenter Operations, AppSpeed, VMTurbo, vKernel, vCloud Director, rPath, etc. require some level of access to the management network in order to actually work. So security of these tools must be considered while doing any security design as not only are they accessed by virtualization administrators but by others outside the virtualization administration team, and may actually send data to remote locations. #1 is not that feasible as well unless your organization is fairly small. This option may be good for a NOC however where data is displayed on large screens. This is one way I would use VMware vCenter Operations standard and advanced within the NOC. The login would be the difficult task.
There is a new class of tools from VMware that now authenticate users directly through VMware vCenter which makes vCenter the central control for A&A but at the same time requires Role Based Access Controls (RBAC) to be implemented properly within VMware vCenter (VMware Roles & Permissions), which is just not done today, so use of service accounts for such access is growing. If tools like Identity Logix and HyTrust also inspected vCenter Roles and Permissions with respect to Active Directory and other access mechanisms, we may have a handle on the RBAC problem, but at the moment such tools do not exist.
In essence, what it boils down to is picking carefully where to place the tools we use to extend the management of our virtual and cloud environments. Wherever we place these tools, we need to increase the logging and auditing of the network so that we know exactly what is happening.
So we now have a 2 stage administrative network, and possible ways of securing other traffic. Do we need any other sort of security? Yes. When a tool that extends management opens a connection to a management server such as VMware vCenter or the virtualization hosts, that is when they are the must vulnerable to MiTM attacks (once more see Vasto’s tools to see how easy this is to do). As such these initial connections need to be protected within their own secure network. But then how do we get the data out to the ones that need to use it?

The approach I like is taking a leaf from VMware’s own security book and that is to use a Reverse Proxy to further protect what is being accessed. So now we add a Proxy Network into the mix that sits between the virtualization management network firewall and the first stage of the administrative network as depicted in Figure 2. We place on this network a reverse proxy. This would allow all the tools that need to talk directly to the management servers (VMware vCenter, Microsoft SCOM, Citrix XenCenter, etc.) or the virtualization hosts directly to do so in a secure environment, but also allow external users to access the web sites of the management tools. Granted if there is no web interface for the tools, then a Jump Machine in this situation would be necessary.
So we have achieved more security by either choosing tools that do not need to talk to the virtualization management components, or by segmenting all those tools into a private area (outlined by the orange box within Figure 2) and also not allowing them to have direct access to the actual virtualization management components (virtualization management servers and virtualization hosts). The use of a reverse proxy limits how credentials are passed for accessing those all important virtualization management tools but also allows the users to access their tools as they normally did.
We have improved security, but have not impacted usability very much. Some may claim using a jump machine to perform administrative functions is an impact, but every administrator I know already knows how to use RDP and does so in their daily job.
It is extremely important to never allow the keys to the virtualization environment to be available to those that do not need them, and performance monitoring and extended virtualization management tools do not need that level of access. To implement these tools securely we need to restrict access without limiting usability, increase auditing, logging, and improve log analysis, as well as control authentication and authorization.