vSphere Upgrade Saga: Adding Fusion-io to an HP BL460c

I recently received an HP ioDrive Accelerator for some testing and discovered that putting the device into the blade was just not enough to get it to work with VMware vSphere. There are a few other items to consider along the way. Since the ioDrive is a mezzanine card for the HP blade, it is important to understand how mezzanine cards connect within the blade enclosure and to the interconnects within the back of the enclosure. I found this out the hard way.

Here are the steps for installing an HP IO Accelerator (designed around ioMemory) within an HP BL460c for use by vSphere ESXi v5.1. The hardware install is fairly simple:

1. If an existing mezzanine card is in the top of the two slots for mezzanine cards, unscrew the existing card and remove.
2. Place the HP ioDrive Accelerator within the previously unused slot: in my case, in the bottom slot.
3. Put the existing mezzanine card back into the slot it came from: in my case, the top slot.
4. Connect the battery to the HP ioDrive Accelerator. My battery has two cables: one for the ioDrive and one for the battery-backed cache for the built-in HP 410i.
5. Place the battery in the cradle designed for the purpose.

Now the hardware is installed. If you moved your mezzanine cards around, then on boot you will get an angry red light on your system, as the profile for the blade will not match what is expected.

Since the node is down, it is now time to determine whether you need to upgrade any firmware. Using the HP ILO, mount the HP firmware disk using the virtual media. Boot the blade and upgrade any firmware necessary. The only time I upgrade firmware is for vSphere upgrades or hardware modifications. Once this is done, reboot, and vSphere ESXi will boot normally.

Next, install the ioDrive drivers. To do this, use two HP Offline Bundles. The package to download from HP includes the two bundles, documentation, and firmware for the ioDrive. Unpack the zip file to access the bundles for import into VMware Update Manager. I created a special host extension baseline just for the Fusion-io ioDrive device so that I can control to which nodes it is attached. However, just remediating the two offline bundles is not enough to bring the ioDrive to life. I also have to upgrade the firmware on the device, which requires logging in to the management console of the ESXi host. Since you are making a hardware upgrade, this is one of the times this should be allowed; however, it should be monitored carefully.

Notice: The rest of this procedure requires you to use the vSphere ESXi management console either from the CLI or via SSH.

How did I determine that new firmware was required? By using the “fio-status” command direct from the CLI, which then allowed me to use the “fio-update-iodrive” command. All of these instructions are within the critical HP IO Accelerator for VMware ESXi 5.x release notes, which can be found under the drivers for VMware ESXi 5.0 and not the VMware vSphere 5.1 link off the HP BL460c Gen 8 drivers and support page. Since this moves around, a link is difficult to specify, but if you have ever downloaded drivers from HP, you can easily find the BL460c Gen 8 drivers page.

Once the firmware is installed, you need to reboot the host and then verify the firmware version once more using the “fio-status” command. If everything checks out, then it is time to format and attach the device.

Interrupted, Updates, and Moving Forward

I was interrupted while finishing this writeup but am now back to it, and I am at vSphere 5.5. The VSL drivers I was using are no longer valid. As a matter of fact, while lspci from the vSphere console shows the iomemory-vsl mass storage controller, “fio-status” shows there is no card available. Since I am not using the card much since my upgrade, this is not a problem. However, I do have a new use for the device, so I need to get it running. The steps for updating are as follows:

Step 1: Download the 5.5 driver from HP’s drivers and downloads page for your version of HP IO Accelerator.

Step 2: Unpack and import the patches into your VMware Update Manager (VUM) console.

I use VUM to update all my nodes. It is a convenient location in which to place everything that is necessary, and I can attach specific baselines to the entire cluster or specific nodes. This way, the cluster stays in sync and up-to-date. There are two offline bundles to import into VUM: libvsl and scsi-iomemory.

Step 3: Create a host extension baseline for the two offline bundles imported into VUM.

By creating a baseline of the two offline bundles, I can attach this bundle to the specific host and manage updates from one location.

Step 4: Attach the new baseline to any host that has a Fusion-io (HP IO Accelerator) in it.

Once you attach, you can do a scan, and you will see the two extensions needing an update.

Step 5: Remediate the host extensions.

I do this in three phases: I first stage the updates. Then, I place the node by hand into maintenance mode and complete the remediation. I do a by-hand maintenance mode due to the fact that some systems may need to be powered off by hand (such as vCNS App and other VMs pegged to a node).

Step 6: Check status, and detach ioMemory device if necessary.

Run “fio-status”. If it shows the device is attached, run “fio-detach /dev/fct0” to detach the device.

Step 7: Update the firmware on the ioMemory device.

Generally, when you upgrade drivers, you will also have to upgrade the firmware. The firmware is located in the Firmware directory of the zip file you downloaded; and it also contained the offline bundles that you used to remediate the host. Transfer this firmware to the /tmp directory of the vSphere host. (Yes, you need to enable ssh for this to work.) Then run the following command: fio-update-iodrive /tmp/ioaccelerator_3.2.6-20131003.fff.

Use the name of the firmware you downloaded; it could differ from the above. The firmware is held within an .fff file.

Per the instructions, reboot the vSphere host.

Step 8: Format the ioMemory device.

You do not want to format the entire ioMemory (HP IO Accelerator) for use by a filesystem; it is generally recommended to use only 80% of the drive. This leaves space for memory management. VMFS only uses 512 byte blocks, so that is the value that should be used. Use the following to format: fio-format -s 80% -b 512B.

Step 9: Attach the device for use.

This is simply the running of the fio-attach command, but before I attach, I generally run the commands fio-status and fio-pci-check just to ensure everything looks fine. It is best to verify everything before use. Once the check is finished, fio-attach /dev/fct0 can be run.

Step 10: Disable SSH and your vSphere console.

For security reasons, you should disable your SSH or vSphere CLI now that the ioMemory hardware is installed, formatted, and attached.

Now, the HP IO Accelerator (ioMemory) device can be used by PernixData and other vSphere caching products.

Here is an excerpt of the output of the commands described:

Click to See the vSphere Console Commands Used