I use VMware NSX-V to provide security services for my virtual environment: edge gateway and distributed firewall. These services do not require the software-defined networking features to be enabled. At least not yet. That could change. Given my use for security, upgrading NSX-V is important to my environment. Here are the steps I follow when a new release is available:
Using NSX Manager, I select the upgrade button and upload the latest upgrade bundle. Be sure to read the README, as some upgrades require a bit more work. I.e., upgrading from older versions often requires going to a common version before applying the latest upgrade. However, I keep my upgrades in sync with releases so as to avoid that particular problem. It was an issue when I converted from vShield to NSX.
Upgrade NSX
The upgrade is very straightfoward:
- Snapshot the NSX Manager VM. We want to recover if there is an issue with the upgrade. Every time I skip this step, something seems to go wrong.
- Upload the upgrade bundle using the NSX Manager. You will have to log in as the NSX Manager administrator to perform any upgrade.
Once the upload is finished, you will receive the following:
- Now, verify the waiting version against the existing version. Ensure the upgrade is allowed.
- Begin the upgrade. It is wise to perform any pre-upgrade checks just to be sure.
- Let the upgrade finish.
- Reboot the NSX Manager.
- Clear your browser’s cache and cookies.
- Log in to vCenter HTML5 client, and now you should see NSX Edges under Networking and Security, which will show that some updates are necessary.
- Go to Installation and Upgrade. Note: there is an upgrade to the NSX Controller I have deployed available.
- Upgrade your controllers.
- Switch to the Host Preparation tab, and note the upgrade is available.
- Now, once you press Upgrade, it will become Not Ready, as you need to then enter maintenance mode on each node to complete the upgrade. Enter maintenance mode to resolve the VIB update per host. There is no need to do anything else, as the staged items will automatically be resolved.
Problem
In my case, Not Ready did not clear on a reboot of the hypervisor nodes. Some upgrades could not continue until this was fixed. When there is a problem, the RESOLVE link exists (per the previous diagram but where UPGRADE is in the diagram). The resolve showed an empty string error; i.e. Error:. Logs also showed an empty string for an error.
Often such an error implies a version mismatch between ESXi and NSX. However, this was not the case. So, back to some research.
I thought it was something to do with my latest failure. One of my nodes had a PSOD, which was attributed to a failing switch. When the switch failed, the switch failover did not work as expected.
The Solution
I did some more digging, and the solution was to also reboot vCenter. Rebooting vCenter fixed the Not Ready problem that persisted.
Back to the Upgrade
Unfortunately, during my debugging of the problem above, I had already upgraded some components. I thought perhaps there was an ordering problem. There was not. So, diagrams will be a bit sketchy from here on out.
- If you use Service Deployments such as Guest Introspection, you would upgrade those next. I do not at this time, so no upgrade was required.
- The next item to upgrade for me was my NSX Edges. I have many to separate my virtual labs. As you can see in the following diagram, my upgrades had already happened, but if an upgrade were required, directly to the right of the version number would be an UPGRADE link. The only thing you need to do is select that link, and your upgrade will proceed very quickly with no loss of rules.
Conclusion
For me, this was the end of the upgrades, as I use NSX-V for security purposes, not networking. However, if this were for networking, I would be looking at upgrading all my logical routers and any deployed services as well.
