I was reading a Reddit request for help regarding ransomware. The title was “Got hit BAD tonight.” That title describes the catastrophe simply and to the point. The ransomware in question attacked the hypervisor. Then, it proceeded to encrypt all backups and other systems connected to the hypervisor. This is the exact issue that virtualization and cloud security folks talk about daily with others. This is the ultimate in admin escape. This was not an escape-the-VM; this was an admin escape. The rule for accessing the hypervisor directly is DO NOT. The rule for using administrator credentials to do anything is DO NOT. Admin escape counts on those mistakes being made. Even so, there is a ton we can learn from this episode. I feel for the target, but it is time to quickly learn and implement better protections within your own environments. They are targets as well.
What can we learn about this catastrophe that struck another? Several things spring to mind immediately:
- Administrative users are only for break-glass failures. Keep them under lock and key.
- Monitor anything an administrative user does.
- Be proactive and only allow administrative users to do very specific things while monitoring them.
- Ensure there is an air gap between your main infrastructure and your backups. They should not be mounted to your main infrastructure as volumes. Not even temporarily.
Those are the easy-to-spot issues and solutions within some data protection and security architectures, designs, and implementations. In this case, how did the hypervisor get infected with ransomware? The answer is most likely very simple. This is a Hyper-V solution, and they most likely used the Windows system for more than a hypervisor. Perhaps they answered mail on the system, went to browse the web, or any other number of other things. Or the attacker waited for and watched for administrative access to the Hyper-V hosts. As such, we can further learn the following:
- A hypervisor is an appliance. DO NOT use it as a desktop. In other words, just do not log in directly to your hypervisor! Furthermore, if you are using Hyper-V, KVM, or even Xen, remove anything related from the desktop and trim down your distribution. For Hyper-V, I would use Server Core, not Windows. For KVM and Xen, remove all versions of Xorg, etc. Limit functionality.
- Hypervisors are not desktops!
- All management should be within a segregated network—segregated from everything else.
We still have not said anything new here. The virtualization and cloud security folks have been asking, pleading, and begging people to treat virtualization and cloud management as a security trust zone: a zone with higher security requirements than anything else. In consulting, I have even set up small doctor’s office systems where their management was behind a firewall and a restricted jump machine was in use. The management network or trust zone should have extremely limited access. Once you are in there, you should only be able to manage the system using proper interfaces, NOT directly. Never directly. Never, ever directly. This has been the same message since 2004, when VMware ESX was first looked at by many.
So, if the message has not changed, then
What has changed?
We have a proliferation of data protection, management, and security devices that are all interconnected. We have a false sense of security, thinking that everything is segregated when in fact it is not. Some things STILL need an air gap. Data-protection archive repositories are one such item. I would go so far as to say that we need to rethink our entire data-protection architecture. Perhaps it should be something like the following, where nothing is ever mounted; our archive storage does versioned writes, deduplication, etc.; and direct access to the hypervisor or even server is just not allowed. Instead proper APIs are used.
Getting data protection right is crucial with ransomware. However, limiting the attack footprint is even more important. None of this is really new, but it is eye opening that such basic mistakes can still be made. Was this problem due to an inability to spend money? Not really. Many businesses depend on IT to get it right with minimal input and minimal spending. Nothing stated here actually costs a lot of money. Yes, it is possible to spend huge sums of money, but it is not required if the skills exist. IT itself can fix many of its own issues by working on process and refusing to do the wrong thing while supporting the business via alternatives. The place to start is to look at what is out there from data protection and storage vendors such as Quantum, DellEMC, Hitachi, Western Digital, etc. They all understand the need for such an airgap. Their integrated solutions will add context to your IT discussions.
This is as much a conversation about people and process as it is about technology.
How do you do virtualization and cloud management? If administrative logins are being used, stop. Rethink this need, and change it. Admin escape is far too easy today.