More and more is coming out about the attack from a MacDonald’s that left an organization crippled for a bit of time. The final tally was that the recently fired employee was able to delete 15 VMs before either being caught or he gave up. On twitter, it was commented that the administrator must not have been a powershell programmer because in the time it takes to delete 15 VMs by hand, a powershell script could have removed 100s. Or perhaps the ‘Bad Actor’ was trying to not be discovered. In either case, this has prompted discussions across the twitter-sphere, blog-sphere, and within organizations about how to secure from such attacks.
To help with these discussions I will reiterate some of the discussions I have had and how you can protect your data from such an attack in the future. Some of these will come out at the VMworld Session: SEC2284 Securing Government Virtual Environments: Part II (Twitter hashtag: #SEC2284)
- Follow proper policy and procedures and disable user accounts, VPN access, and administrative accesses before the employee is asked to leave, or perhaps immediately after. However, this can be tricky given the multitude of authentication and authorization mechanisms available within any cloud and virtual environment. Some tools to help protect are: HyTrust Appliance (use the community appliance at the very least for critical resources) and tools like Identity Logix’s SpyLogix for VMware to correlate all accesses to the virtual environment. Unfortunately, there is no mandatory access control mechanism for any virtual environment, so it has to be imposed from the outside.
- Audit your Authentication and Authorization controls. Use a tool like IdentityLogix’s SpyLogix for VMware to inspect all authorizations and authentications within your virtual environment. Ensure that a users do not have permissions to components they have been denied access to in vCenter. For VMware vCenter is starting to be the holder of RBAC for the vSphere environment, it is not 100% the case, but it is a good starting point. You have to audit your users within groups, within vCenter, ESX, ESXi, and all other management tools. Xen has its own mechanisms for authentication and authorization within XenConsole and Hyper-V makes use of Active Directory. A user, regardless of management tool, used needs to have only one set of authentication and authorization.
- Restrict access to management controls to specified jump machines within a virtualization management layer. This is the lowest hanging fruit of any virtualization security implementation. Place behind a firewall all your management tools including vCenter, XenConsole, System Center, Performance Management Tools, Backup Tools, etc. Pretty much anything that directly or indirectly touches your virtualization management tools should be within this management only network. Access to these components should only be allowed from within this management network and access to the management network should be restricted to specific administrator workstations.
- Inspect administrator’s workstations for rogue hypervisors that hide VMs behind a NAT network. This includes any type 2 hypervisor that administrators may install on their workstations that have been granted access to the virtualization management network. Any Type 2 hypervisor can provide a loop hole to bypass security, such as VMware Workstation, Player, Fusion, and Server, Oracle Virtualbox, and Microsoft Virtual Server. If you really want to disable the use of NAT generically, switch your network to an IPV6 only network for accessing the virtualization management network and within the virtualization management network, as NAT does not currently work within IPV6.
- Audit and understand your administrators. While this may be difficult for non government organizations it is still an important step as you are ultimately trusting administrators with your data and with that trust comes the ability to destroy that data. It is best to understand your administrators and what they are capable of doing compared to what they have done in the past, etc. Background checks should be required if the data is sensitive enough, but also because everything is not easily removed.
- Require Two Factor authentication to access not only the VPN, but administrator workstations, and the jump machines which can talk to any of the management tools.
- Make proper backups. It is crucial to make proper backups and to consistently check those backups. Tools like Veeam Surebackup makes this testing easy. Without proper and tested backups, recovery from disasters whether man made or not is difficult at best.
- Continually monitor your system. Employ tools that will look at your system for configuration changes and major changes every 5 minutes or less. Be sure to respond to any and all alerts. A simple monitor of whether or not a crucial node still exists and is running should be part of any such monitoring tool. There are hundreds of tools, but not all understand the virtual environment, so I would look at tools like Zenoss.
These steps are just the beginning, you need preventative and proactive steps in your security posture. As security professionals and virtualization administrators, we need to ensure the proper controls are in place at all times, while monitoring for changes per our security policy.
We need to Trust but Verify.