vSphere Upgrade Saga: Staging Upgrade to vSphere 5

I have done all my homework, planned my upgrade to vSphere 5, ensured my licenses are available and all that, so now it is time to stage the upgrade using my spare node. Why stage the upgrade and not perform it? By staging using my spare node, I am able to test to see if firmware and other upgrades work appropriately. This is crucial to the success of the upgrade of the other nodes. In essence, all other subsystems are upgraded before I hit the production nodes, which means vCenter is also upgraded but first I backup my databases, etc. As well as the VM running vCenter.

Here are the steps I followed:

Step 0: Review any updates to firmware for my HP BladeSystem c3000 just in case something new came out since the last time I looked. There was a new upgrade for the Onboard Administrator.

Step 1: Given this upgrade was available the first step was to upgrade the Onboard Administrator firmware, which I could do directly talking to the Onboard Administrator web interface. Quite simple, and easy. Unlike previous Onboard Administrator ungrades there was no issues with loosing power or connectivity to any of the blades. While doing this, I made out a shopping list to ensure even more redundancy: Another Onboard Administrator as well as a replacement Insight Display Module.

Step 2: Upgrade the Flex10 Virtual Connect firmware. This is required to use vSphere 5. You must upgrade any Flex10/Virtual connect firmware to version 3.30 in order to use vSphere 5. Step 2 was to upgrade the secondary Flex10 first using the command line Virtual Connect Support Utility (VCSU) using the update command.

Step 3: Upgrade the primary Flex10 Virtual Connect firmware. There was an oddness with this upgrade using the VCSU, and that was the VM running the VCSU lost connectivity which seemed to cause a reboot. So something inside the VCSU failed and rebooted, which is really a horrible failure mechanism. However, I only lost network connectivity for a few seconds. Which was also much better than the last Flex10 Upgrade I did.

Step 4: Upgrade the firmware within my spare blade using the HP Firmware DVD v9.30. There was an ILO update, so to be sure all firmware was upgraded, I booted from the DVD twice just to be sure I caught everything.

Step 5: Install vSphere 5 ESXi onto the spare blade just upgraded. The trick is to ensure you pick the proper disk, in this case local disk instead of a SAN drive. I double checked this before proceeding. The install is so much easier than that of ESX.

Step 6: Configure the Management Network for the spare blade to not use DHCP but to use the provided IP Address, Netmask, and hostname.

Step 7: Decided to upgrade my existing vCenter instead of using the Linux appliance, mainly because I already have a Windows configuration for vCenter and the Database server.

Step 8: Backup your vCenter Database and SSL certificates. vCenter 5 will remind you of this as well and nicely ask you to ensure you are happy with your backup.

Step 9: Perform vCenter Upgrade

Step 10: Perform vCenter Client Upgrade

Step 11: Install vSphere Web Client for new License Reporting functionality. This required me to find out which process held port 9443 open, which turned out to be something running Java. However, what I do not know. There was a Java update to be made, once I did that, this error went a way. So I went through and removed all my third party tools from the vCenter server which eventually fixed the issue. A part of this step was to register your vCenter Server with the Web Client Server so that it can communicate.

Step 12: Upgrade vCenter Update Manager. Once more this is a polite reminder to make a backup first of your VUM database which requires you to select the I am happy with my backup checkbox. I find that a great improvement. While VUM upgraded fine, it did say it wanted to reboot my host, eventually that will happen as a matter of course. Need to test if everything comes back up on boot.

Step 13: Install all the vCenter Support Tools, Dump Collector, Syslog Collector, and Auto Deploy but not the Authentication Proxy. These are all new for vCenter 5. The key to these installs, is that if it asks, yes you want the Integrated VMware vCenter Server Installation. The Authentication proxy requires IIS which could cause issues so we hold off on that for now. Now reboot your vCenter server and make sure everything runs.

Step 14: Reconnect the ESX hosts to vCenter which deployed the new vCenter Agents onto the hosts.

Step 15: Create a new cluster so that I can enable EVC mode for Nahalem or better processors. This will help with new blades once they arrive. Somehow EVC was disabled on my other nodes.

Step 16: Configure VUM and import the vSphere 5 ESXi image for later use. The VUM the client for some reason did not enable by default which took going to VMware’s KB Article 1015223 to solve.

Step 17: Configure the newly installed spare node using the old nodes Host Profile. If you have Enterprise Plus licenses the use of Host Profiles for this type of staging is incredibly useful. Even though I am moving from ESX to ESXi, use of Host Profiles to at least get the network configuration correct is very important.  If you do not have Enterprise Plus, you can use an Evaluation Mode to do this. The gotcha here is that I am moving from ESX to ESXi so some of the Host Profile elements will not apply. So not only do I do a clone of the Host Profile, apply that profile, but I need to delete and recreate the Host Profile once the spare node is fully configured else there will be compliance issues.

Step 18: Reboot Node and enter the BIOS to set HP Power Profile to Custom and HP Power Regulator to OS Control Mode per VMware KB Article 1018206. Save and reboot. You can do this as a part of any of the node upgrade steps previously mentioned, but without this you will not be able to take advantage of the power controls within vSphere 5.

Step 19: Upgrade vSphere 5. This is a good test of your VUM install from Step 16. As of this writing there are at least 2 patches for vSphere 5. One requires a reboot.

At the moment, the system is staged for migration of VMs and upgrade of the other nodes. Staging such an upgrade is important as now we know that the firmware upgrades work, I can enable the proper EVC mode, and the other systems are still manageable.