This was a painful upgrade, but I had lots of help from various sources. The main element is to ensure you follow the vSphere Update Sequence. Update everything in that specific order: do not differentiate. This is important, as some things will not upgrade if they are out of order. This part of my vSphere Upgrade Saga upgrade was far from smooth.
Michael White over at Notes from MWhite has a very good description of how to perform the upgrade. I suggest reading it and following it, specifically around using the Migration Assistant. Since Michael went into this in gory detail, I will not repeat everything. However, I had several other issues that caused me to attempt my vCenter upgrade several times.
vSphere Upgrade Saga: vCenter Upgrade
First, we must understand that the vCenter upgrade is not an upgrade per se: it is a migration from either the Microsoft Windows version of vCenter and VMware Upgrade Manager or just the Microsoft Windows version of the VMware Upgrade Manager to the VMware vCenter Server Appliance (vCSA). For a migration, you must have your vCenter Server in good shape. Mine apparently was not. I was faced with the following problems:
- Deployment failed with the inability to migrate Auto Deploy
- Migration Assistant failed several times with respect to VUM
These issues required me to do the following fixes to make things work:
- Reboot the existing vCSA
- Reboot the existing VUM server
- Disable Auto Deploy
Surprisingly, once I did these things, the migration worked as expected. There are several good articles over at vRyan.co.uk that cover a number of Microsoft Windows–based, VUM-related issues.
In effect, the process was less than painful. I also had to upgrade vRA (and all its components), vRO, NSX Manager, vROPs, and VRLI. Those upgrades went as expected and quite well. The VDP upgrade was not an upgrade. I actually redeployed. The VM was stuck in a non-responsive mode, and since I was not really using it, I chose instead to start fresh. For those with existing VDP installations, be careful. You do not want to delete your repositories. You will want to power off the old VDP, deploy the new VDP, and then hook into the existing repositories. Then unhook the old VDP from the repository systems and remove.
If you can upgrade VDP, even better.
vSphere Upgrade Saga: Windows Helper VM
When we move to vCenter 6.5, we also leave behind many Windows-based components of vCenter. Specifically, the following can be removed from any Windows Helper or Windows jump machine VM(s):
- VMware Update Manager
- VMware Authentication Proxy
- VMware Client Integration Plug-In
- VMware Update Manager SDK for PowerCLI
Currently, you still need the Windows-based SRM components.
To clean up my Helper VM, I removed the VMware components except for SRM and VMware Tools. Then, I reinstalled what I needed from the components downloaded. I ended up replacing ten or so installs with only four:
- VMware vSphere CLI (for ESXCLI functionality)
- VMware PowerCLI (for PowerShell functionality)
- VMware OVF Tool (for import functionality)
On a Linux Helper VM (you can also use the vSphere Management Assistant), I installed:
- VMware vSphere CLI
- VMware OVF Tool
- govc (vSphere CLI built on top of govmoni or the Go SDK)
I had to either use OVF Tool or govc to import OVAs and OVFs, as that functionality is currently broken in both the Web Client and the HTML5 client. govc provides a quick way to import, as written up by William Lam. Using govc is a three-step process:
- Extract the import specification
- Edit the specification file or add your settings and remove those settings that will not work (often trial and error)
- Import the OVA/OVF using the edited spec. Iterate on Steps 2 and 3 until 3 provides no errors (usually by deleting things from the specification file)
It is important to realize that the NetworkMapping section has two components: Name and Network. Name refers to the virtual switch name, and Network refers to the portgroup name.
It is actually faster to import this way. Granted, I also kept my import specs in a repository for future imports. These are stored in JSON format. When importing an OVF, I found I had to edit the .ovf file to change or remove offending bits as well (mostly around networking) to make this approach work.
To keep all this straight and to generalize imports, I ended up coding a simple OVF and OVA importer using both govc and ovftool directly. You can find this in my Git Hub repository. The resultant script uses a configuration file with simple key value pairs and will import either by filename or every OVA and OVF in the directory from which the script is run. I also added the ability to unpack ZIP files and to mount ISOs specifically around VCSA imports.
vSphere Upgrade Saga: vCNS to NSX
Part of this upgrade to vSphere 6.5 was a migration to NSX from vCNS. For certain aspects of vCNS (Endpoint), this migration is required and available to anyone for free. It was quite simple to do this migration, but I have one caveat. My vCNS installation would often fill up its disks. The solution is to purge a number of logs. VCDX56 offers some great steps for doing so. VMware’s KB articles were not very useful, actually.
Once the logs were purged and vShield Manager was rebooted, it was time to update. I used the NSX 6.2.4 bits for my upgrade. The upgrade package will be named something like VMware-vShield-Manager-upgrade-bundle-to-NSX-6.2.4-??????.tar.gz. This is the package I used to do my upgrade using the standard vShield Manager upgrade paths.
Once that was done, a reboot left me with just NSX Manager, which could manage my existing Edge firewalls.
The upgrade to vSphere 6.5 also required an upgrade to NSX Manager 6.5. This upgrade went smoothly as well. However, after upgrading vCenter to 6.5 and vSphere to 6.5, I ran into an issue. That issue was that in order to upgrade my NSX Edge firewalls, I had to first prepare the hosts: that is, install the distributed firewall, VXLAN, and other VIBs onto my ESXi hosts. I believe this was due to the fact that the Edge OVF files were also deployed in this manner.
Not a huge hurdle, but not something I had wished to do yet.
Closing Thoughts
Always plan your upgrades carefully. Before you upgrade, be aware of what you need in order to proceed. For example, to go to vSphere 6.5, I needed to wait for Veeam Availability Suite 9.5 as well as NSX 6.3. Once those were available, the upgrade could proceed. What I did not know, due to improper documentation, was that I also needed to wait for OneView for vCenter version 8.2. I am still waiting on that.
Plan, then plan some more. Be prepared for issues. This means that before you upgrade or migrate any VM, take a snapshot, make a backup, etc. Be prepared to go backward if necessary.
