I have been thinking about blades and virtualization security for some time spurred on by a conversation with Brad Hedlund six months ago. Nearly all my customers use Blades and virtualization security is a big concern to them. In my Rethinking vNetwork Security article, I touched on some of the issues in response to Brad’s comments a while back. I would like to now expand that discussion to blades.
There are three sets of blade enclosures I would like to discuss, those that use pass thru networking, those that use standard switching fabric within the enclosures, and those that use flexible interconnects such as HP Flex-10 and Cisco Palo adapters. The last is the so called physical-virtual network device.
I presented similar figures in the Rethinking vNetwork Security article that cover the case of pass thru and normal switching fabric based blades. Figure 1, on the other hand looks at blades from the perspective of the physical virtual network device. In this case, you can see that within the vmkernel or hypervisor the similar split is done, BUT instead of going to the outside, the Blade MidPlane comes into play. Figure 1, shows two blades connecting to the same Midplane and therefore the same pvSwitch control plane.
In essence the mezzanine card within the blade acts in concert with the pvSwitch to provide a set of networking devices to the vmkernel that can be attached to any number of physical virtual port groups (pvGoup). The pvSwitch Control Plane is used to manage these groups and if VLANs are assigned, the VLANs as well. If the pvGroup also uplinks to an external source, such as another blade enclosure or an external pSwitch, then the pvGroup is no longer private to the pvSwitch. From a technical perspective, pvGroups work just like VLANs, but unless there is a VLAN associated with them, the traffic may not be restricted to just that group. This depends on the type of pvSwitch in use. In the case of the HP Flex-10 device, a pvNIC (on the mezzanine card) is connected to a pvGroup that is private to the pvSwitch unless there happens to also be an uplink involved regardless of VLAN usage.
What does this all mean? It means that the logical separation achieved by using multiple vSwitch types (such as the VMware vSwitch, Distributed Virtual Switch, and Cisco Nexus 1000V or N1KV) could be lost once you use a blade, depending on the blade interconnect chosen. Why? Because everything that you logically separated within the hypervisor once more joins together within the blade. So the choice of interconnect becomes very important from a security perspective. You need to understand the impact and workings of the interconnect to properly secure your environment.
The key questions are:
- Does the pvGroup provide separation with or without the use of VLANs?
- Does the pvSwitch Control Plane provide any other inherent protections?
- Does uplinking between blade enclosures provide the necessary segregation of data from other uplinks?
- How do you achieve redundancy using pvSwitch’s and pvNICs?
- What other special security features exist?
Ask these questions and plan out your environment properly to account for flexible or converged networking within blade enclosures so that the physical-virtual components are also part of this plan. A few companies consider the physical-virtual components to be the realm of virtualization while others consider them to be the realm of the traditional networking teams. Either case, you must consider them from a virtualization security view.
Edward — Your article was promising. You identified some good questions, but essentially you say you haven’t got a clue regarding the answers to the questions even though you’ve identified the two main vendors of the flexible interconnects: HP and Cisco. I think you could have gone one step further and asked HP and Cisco to answer your questions — then your article would have been useful to security planners. — Mike
Hello Mike,
The gist of the article was two fold: 1) to make people aware that when using blade enclosures, depending on the networking components, the segregation achieved within the hypervisor becomes moot when you use blades. 2) That there is more work to do as vNetworking is pushed into the hardware completely. Typical Layer-2 controls are no longer sufficient as the vNetwork within the hypervisor has an overall better security stance when you use a vSwitch than when you use VLANs. These same protections are not available within the Layer-2 blade network. Each switch in a blade is a simple Layer-2 device. So given this, another Blade could act as a MiTM. If VMDirectPath is in use, then a VM could act in this manner as well. This depends on network configurations once more.
The physical network’s major security control is to ensure the switch configurations are correctly in place and that the configuration does not fail-open (which is the case by default) even within a blade enclosure. Hence why I posed the questions I did.
As for Cisco Palo Adapters, they send ALL traffic Northbound to the Cisco 6100s (or is it Nexus 5Ks?) which route it to the Nexus 7K/Catalyst 6150 (etc.) As for the HP FLEX-10, it is a simple Layer-2 device that only allows UPlinks on the external ports. Actually both only allow uplinks to other switches (which in itself is one form of security). But they are still fairly simple Layer-2 devices. Cisco requires you to egress the blade chassis to deliver traffic where HP FLex 10 does not.
Best regards,
Edward L. Haletky
Edward,
You are absolutely right to point out that vNetwork isolation and security at the vmkernel layer eventually meets at a Layer 2 switch where VLAN isolation is pretty much as good as it gets. For now, thats pretty much a fact of life in designs leveraging adapter and switch consolidation with 10GE. In the legacy blade server architectures, such as HP Flex-10, you have (2) configurable Layer 2 switches (pvSwitch), with VLANs (pvGroups) in every chassis.
Having said that, one of the key differences with Cisco UCS form a security perspective as it relates to your article is the fact that with UCS you do NOT have a “pvSwitch” and “pvGroups” in every blade chassis. Rather, the networking element in the UCS chassis (FEX) behaves like a pass-through device, without all of the extra cables. The FEX does not have a configurable control plane or data plane. All of the Layer 2 switching (control and data plane) is consolidated at the Fabric Interconnect, the 6100’s, the networking switch connecting up to (40) chassis.
This makes a key difference in your point: “The physical network’s major security control is to ensure the switch configurations are correctly in place”
From a security auditing perspective, with the Cisco UCS architecture you have (1) Layer 2 switch across (40) chassis to worry about. Compare that to the (80) Layer 2 switches across (40) chassis with the legacy blade server designs (HP).
Does having (1) configuration to audit, versus (80), make a difference? I think so, but curious to hear your thoughts on that.
Great article! Perhaps a topic for a podcast? 😉
Cheers,
Brad Hedlund
(Cisco Systems)