During the Virtualization Security Podcast on 7/8, Vizioncore’s Thomas Bryant joined us to discuss the state of virtualization backup security and forensic use of such backups. In the world of virtualization, backups are performed mostly by 4 distinct vendors: VMware Data Recovery (VDR) and VMware Consolidated Backup (VCB), Vizioncore vRanger, Veeam, and PHD Virtual Backup for vSphere. Each of these provide the most basic of security capabilities:
- Encrypted tunnels for data movement (SSL)
- Encryption of the backup
But in the increasing global nature of businesses and the difference in privacy laws between townships, states, and the need for Secure Multi-Tenancy, backup companies fall short with their products while making it increasing harder to use backups as a source of forensically sound data.The companies that write data to tape such as Symantec and HP (Data protector) have the same issues, they currently encrypt the data in motion as well as on the media, but this is not where we need to be.
Backup security should move forward to allow the data to be restored only to certain locations, the data to move only to certain locations, and to prevent the use of the data where not allowed. In essence a combination of DLP and DRM specifically built for backups of VMs. Given how backups work within the virtual environments, which is discussed within my Backup Security post increased security is necessary. In essence, data can be placed in all sorts of places as it makes it way to tape, hot-site, off-site backup repository, etc. If we introduce VPLEX into the mix, we could be backing up data from a remote data-center.
When we discus Secure Multi-Tenancy (SMT) we are really talking about data protection. Governing who can see, manipulate, and control the data. Virtualization Backup tools do not currently have much that can do this. Yes, they have limited Role Based Access Controls (RBAC) while the traditional backup tools have intricate RBAC capabilities. For SMT, an integrated RBAC will be required for any form of backup.
Many of the current backup tools run from within VMs, which means the data in motion and at rest encryption is handled within the VM, which implies it can be broken by an administrator or anyone with administrative access. Since backups copy data around the virtual environment from storage array to array as it moves to its final resting place, there are myriad copies of the data available all encrypted using software not hardware. However, hardware based encryption would not be good for a backup, as you may end up restoring to non-specific hardware in an emergency. So what is required to move forward:
- Better RBAC integrations
- Some form of Digital Rights Management to only allow a backup to be usable under certain conditions.
- Some form of Data Loss Prevention that protects against a backup landing on an inappropriate device (USB drives come to mind)
- A secure way to encrypt all data from within a VM but decrypt anywhere
In essence, we need metadata about the backup in order to prevent its use as needed. Perhaps OVF can help us here. But we also need depth to our controls to allow us to shift critical and classified information around. Just like placing an envelope in an envelope when sending company mail around, you may have to do the same with backups. The outer envelope could contain where the data can be stored, while an inner envelope could contain the necessary controls, while the innermost envelope could contain the data.
Forensics
Traditional backups are well known and usable as a source of forensic data but modern backup tools for virtualization may not be anymore. That is because the backups are not bit for bit duplications of the data. Backup tools use data compression (which is understood), change block tracking (which limits the scope of what to backup on an incremental backup), knowledge of the Guest OS to track which blocks have been deleted or zeroed (to further reduce the size of a backup), and deduplication (to reduce the final size of a backup).
For some backup tools, the data is then rebuilt in a synthetic way to build in essence an easily restorable image of the disk. Others store the incrementals in the same file as the primary disk backup.
So a backup no longer provides a bit-by-bit duplicate of the data, which ends up impacting the use of backups as forensically sound. Research and proof needs to be developed by forensic specialists to allow these backups to be used. In other words, since they are not ‘traditional’ backups, they are suspect until the proof can be provided beyond a shadow of a doubt that the data is forensically valid including chain of custody issues.
Deduplication adds a large wrinkle into the fabric of backups as deduplication of backup data is about reducing the size of a backup job. In essence, deduplication does not take place over just one virtual machine but over all virtual machines within the backup. This reduces the overall size of the backup, which is generally a good thing, but from a forensics perspective could end up mixing data from multiple tenants and then the chain of custody could be broken, because how would we know that the data we are looking at is for the tenant under investigation. We would not know.
Secure Multi-Tenancy and Backup Testing
Thomas went on to state that SMT requires a way to safely restore a single tenants data without administrative help and that such functionality just does not exist today. Granted, a portal could be written to do this and that most if not all tools provide command line or scriptable interfaces for performing such tasks, but none exist.
In addition, earlier in the conversation, we discussed the restoration of backups and how to test this. There is a feeling that some automated tool may give a false sense of security as while the backup may restore properly, the application may still not run properly which implies the automation may work but the backup may still not be usable. Quite a bit of work is still needed on this front but tools like SureBackup from Veeam are a step in the right direction.
As Tim Pierson stated, one of his companies used to run once a month from a backup restored from Tape. In this way they tested their backups at least once a month and knew their applications worked. With virtualization, the agility is there to do something similar as long as the data is kept up to date. So I imagine this was NOT done for databases, etc.
How we would do this in an SMT environment is still a bit of a concern.
Conclusion
Backup security has a long ways still to go to be ready for SMT, but the companies are at least thinking about the requirements. Backups need to be more robust, not only encryption but secure means to encrypt data from within a VM and keeping the keys available for disasters but include a way to determine if its proper, safe, or allowed to restore a backup to a target host within a global virtual environment.