Protecting ITaaS Consoles

There has been quite a bit written about Code Spaces and how unauthorized access to its ITaaS console granted enough permissions to delete everything out of Amazon, including backups. There are lessons here not only for tenants, but also for those vendors who create ITaaS consoles, such as VMware (vCHS, vCD, vCAC, vCenter, Orchestrator, etc.), Virtustream (xStream), OpenStack, and many others. These consoles need better controls and security so that such behavior is prevented, logged, and monitored, and the proper authorities are informed. Now, we may think this is a cloud-only attack, but we use these tools within our own environments day in and day out. For anyone using virtualization, private, or hybrid cloud consoles and automation tools, it is time to take a good long look at role-based access controls (RBAC). The steps we discussed at the end of my other lessons article still apply. 
However, now it is time for some lessons the vendors can learn from this catastrophic end-of-business failure. The creators of the tools we use are also part of the solution. Some of the solutions may already exist, but we as administrators may not be using them, as we find them rather unwieldy. That is where the other lessons come into play. But for the vendors, there is another set of lessons, with some overlap.
Allow mapping of delete to a remove from interface/inventory: One of the possible solutions to the problem Code Spaces had was to disable delete functionality. But since delete is there for a reason, perhaps it would be better to allow during configuration (or some other out-of-band configuration mechanism) the modification of delete to make it a remove from inventory instead. The data is not deleted; it is just no longer visible and as such appears deleted. Then, to fully delete, a different credential from that of the person who actually asked for the delete would be required. This way, you have some sort of two-person approval process for deletes. The mapping of delete to remove from inventory and the requirement for delete to have a different user than the one who requested the delete could be added into any of the tools.  However, the ability to enable and disable this functionality should not be in the interface in which the actions are taking. I could even see third-party tools such as HyTrust CloudControl, Adallom, Skyfence, or Elastica adding in such capabilities.
Add thresholds for all destructive actions: Another possible solution is to add to the consoles the ability to limit destructive activities to a set number per period of time. In addition, if too many are occurring, just suspend the ability to perform the actions. To allow the actions would, once more, require the use of the two-person rule, and not the same user as the one committing the actions. Currently, HyTrust CloudControl can require the two-person rule for deletes, for example, but they are per delete, not based on a threshold. We can also currently use RBAC to deny capability; however, I think we need to allow, but limit, such destructive behavior in some cases.
Add a least privilege health check: I have yet to see this done effectively, but we really need a least privilege health check built into the management tools. We need to ensure that administrative access or a copy of such credentials is not given to everyone. We need to ensure that individual users within a group of users do not have higher privileges than anyone else within the group. Since authentication is performed against several different services, perhaps a federated identity service could be used to perform such functionality. In either case, however, least privilege should be proven, not assumed. There are a few tools that look solely at Active Directory, but many of these tools integrate to at least three different authentication stores, if not more, that are not Active Directory.
Add a big red disable console button: Vendors could provide an out-of-band mechanism to disable the console immediately, kicking off all users, and to stop all processing. If we had a big red button, then when something does happen that is unforeseen, we could either automatically or manually press it to stop the activity. We would also need another out-of-band mechanism to grant administrative access to clean up any user or security issues within the console.
Better monitor and logging: We need a way to determine what is happening within an environment all the way up and down the stack on a per-tenant basis. At the moment, if we are looking at logs from the bottom of the stack (the hypervisor), there is no way to know which user performed an action. This is the Delegate User Problem. It still has yet to be solved. Perhaps we need a common logging project that all vendors can use to add user and vendor IDs to the log string that is finally spit out at each layer of the stack. A single vendor can do this, but others in the ecosystem should also be allowed the opportunity. If we improve logging, then the network operations center could also be alerted to bizarre behavior and could push a big red button to shut down the console.
There are many things we can do now to protect ITaaS consoles, but in most cases, they are considered too draconian to implement. However, the vendors could help us to get things right and not be draconian when going about it. We need more from the vendors to allow us to better protect ourselves from bad actors inside our own defenses, from those coming in from outside and those already allowed in. The vendors can assist; we need their help, and we need better-thought-out solutions that provide not only agility but also protection from malicious and accidental actions.