Virtualization/Cloud and Multi-Factor Authentication

I was posed with a question today, “I’m looking for some info on account & password management for consultants that visit a lot of customers where they have to do admin stuff.” with a secondary question of “how to manage the account if a constultant leaves?”  This was specific to the VMware vSphere but would apply to any hypervisor.
There are two issues:

• How to control who can access the administrative functions within their own datacenter and how to access administrative functions on remote systems spread all over the world.
• How to ensure when an administrator leaves there is no way they can cause any further damage.

This grows in complexity when we start to look at the differences between VMware vCenter which runs on Windows, VMware ESX with its defense in depth capability, and VMware ESXi with no defense in depth capability.
Centralized password management is one aspect of this issue, and for VMware vCenter and VMware ESX, this can be achieved using Active Directory. Granted, to do this well with VMware ESX, you need a full integration approach to active directory with a compensating control of which groups are allowed to login directly tothe host using the pam_access PAM module.
While, it is not important what provides the compensating control to limit access, it is important to realize that not many avenues for such controls exist within VMware ESXi, actually, there are none available with the default install, so you need to find an external device such as HyTrust, or an external firewall to provide access control.
My suggestion was to enable a second factor of authentication so that authentication and authorization includes what you know, password, and what you have, something like an RSA SecurID. With both factors you have some level of built-in compensating control as access to any administrative function would require both factors to allow the access.
Since password management becomes an issue with multiple consultants and administrators that could leave, part of the leaving process would be the ‘handing over; of the SecurID on leaving. Without the SecurID, it would be impossible for a consultant to perform any work which would limit the need to rush around and change perhaps 100s of passwords on systems outside this particular groups immediate control. This is also one method that can be used to provide user control within the cloud.
The issue going forward is that VMware vCenter and VMware ESX can handle multi-factor authentication either directly or by installing the appropriate modules and drivers, but ESXi can not handle this without a third party device being involved. One solution to this problem with ESXi is to create an administrative VM that allows for multi-factor authentication then limit all access to the ‘console’ to be only from this VM using a firewall device or some other compensating control.  We agile datacenters we have to worry about two console accesses, the first is via SSH and the other is via remote access cards such as the HP ILO and Dell DRAC devices. So fronting both of these with some sort of device that understands multi-factor authentication and limiting access to them provides a simple compensating control.
In a situation like this, it is very important to think about how to implement some form of compensating control. Without such controls an administrator leaving with the password could cause two issue; running around trying to change passwords for 100s to 1000s of machines, and perhaps loss of functionality if the person left on less than perfect conditions. In either case, you have issues. Simply using multi-factor authentication and requiring at least two factors to perform any administrative work, provides a compensating control. A centralized directory server such as Active Directory, will also help in these situations, but if the directory server is outside your direct control, then we enter the need to make the changes in these remote locations. Once more running around trying to change things.
This does mean however, that part of any exit, the second factor authentication mechanism must be collected or disabled somehow. This is where items like RSA SecurID become extremely useful, they are very easy to manage and disable.
Within the cloud where everything is connected by the internet, calling ‘home’ to the SecurID management server may be the best mechanism to employ. Each customer within a cloud would have their own directory server and may even have their own SecurID management server, the cloud provider would need to have its own that can be used to either allow or disallow access by its administrators and consultants.
Authentication and Authorization of ‘users’ is a well understood issue, as is authentication and authorization of ‘administrators’ known to an organization, but when it comes to clouds the cloud ‘administrator’ may also need access to hosts and VMs. Or in the case where administration is outsourced, the ‘consultant’ may also need such access. Introducing multi-factor authentication may be the solution but this also means the virtualization host and cloud infrastructure need the support for multi-factor technology, which is far from reality with out of the box solutions.