Something Is Wrong: It Must Be the Hypervisor!

Something is wrong—it must be the hypervisor!

If you work in any virtual or cloud environments, how many times have you heard that statement as soon as any kind of problem surfaces? Way back when during the twentieth century, as a problem deflection, the network would immediately be blamed. As we got into the twenty-first century, virtualization quickly became the go-to area for any and all problems. As part of the virtualization and cloud computing teams, we would have to prove that a problem was not caused by virtualization before any other teams would really dig in and troubleshoot the issue. Even after the fourteen years since the turn of the century and the mainstream acceptance of virtualization technology as a whole, I still see that kind of blame mentality today. And just when I thought I’d heard it all when it comes to virtualization blame, a news story comes out that takes this immediate blame game to a whole new level.

It has been reported that on December 29, 2013, the official website for the OpenSSL code library was compromised in an incident that caused great concern among security professionals. Although the actual code repositories were untouched, the breach left defacement on the OpenSSL home page. Upon discovering the defacement, OpenSSL immediately restored the index.html file from backup and then started the forensics, investigation, and recovery process. So far, so good, right? Well, by New Year’s Day, OpenSSL had issued an advisory stating that “the attack was made via hypervisor through the hosting provider and not via any vulnerability in the OS configuration.” This advisory, which lacked any real details, immediately raised the question of whether the attack could be exploited to target other sites that utilize the same service. Without finishing the forensic investigation, OpenSSL had jumped the gun and deflected blame to the hypervisor. Once the advisory placed the blame on the hypervisor, it did not take long before people started to realize that the hypervisor under suspicion appears to have been VMware’s ESXi server.

As a result of the advisory, the VMware Security Response Center started to actively investigate the incident in order to understand if and how any VMware products were involved and whether VMware needed to take any action to ensure customer safety.

Oh, but wait, here is where it gets really good. About twelve hours after this brief was published, OpenSSL updated the advisory to say: “The OpenSSL server is a virtual server which shares a hypervisor with other customers of the same ISP. Our investigation found that the attack was made through insecure passwords at the hosting provider, leading to control of the hypervisor management console, which then was used to manipulate our virtual server.”

So now it is not just the users who will rush to blame. It seems that companies like OpenSSL have no problem jumping to conclusions and releasing information without performing a complete assessment, because, after all, the culprit could not be anything else, right?

Although researchers have documented theoretical examples of possible avenues of attack through a hypervisor, thus far no concrete incidents have been reported in hypervisors implemented in the field. Furthermore, security experts have shown how those theoretical attacks could be thwarted, so hypervisors have not generally been viewed as sources of exposure affecting the virtual machines running underneath them.

Even though no concrete hypervisor incidents have been reported, this did not stop OpenSSL from trying to pin the blame there for the message “TurkGuvenligiTurkSec Was Here… we love openssl…,” which was placed on the compromised web page. After all, if you turn out to be the first person or group that has compromised a hypervisor in the wild, the only thing that you would do is deface a company public webpage, right? Well, OpenSSL, it appears that the problem is your own carelessness or that of your provider. Based on data returned from trace route commands, several observers have speculated that OpenSSL’s provider is IndIT Hosting.

Now, this brings up the most important point and question. Let’s say for a moment that OpenSSL had no control over the management of the infrastructure, and therefore had no say on the security and setup of the physical hosts. For all of us who are utilizing third-party infrastructure, what steps can we take to ensure that something like this does not happen to any of us? Is this another blemish that will trend toward more companies’ building and running of their own private clouds rather than taking a chance on outside providers?