vSphere Upgrade Saga: Update Order

I have collected quite a few updates over the last few weeks since VMworld ended, and now I should make them. However, the most important non-security update for me is the one for VMware Horizon View. My View environment is old, and though it is maintained, I am in need of several upgrades to Windows, View, and underlying bits. Given that I have nearly everything installed within my environment (except vCAC and vCD at the moment), it is time to find the proper upgrade order. VMware provides the helpful KB2057795, which offers the following supported update sequence. This sequence should be followed whether installing updates or installing major releases.

Usage is quite simple: start at the lowest number and work toward the highest number. What if some things are missing, like VSAN? Well, VSAN is actually part of ESXi, and that is listed. So, some planning is still required. The basic steps to an upgrade are as follows:

1. Back up the VM(s) that you will be updating. Remember to back up dependent VMs as well, such as a database server.
2. Snapshot the VM(s) that you will update. Remember to snapshot dependent VMs as well, such as a database server.
3. Perform the update (in the minimal time required; snapshots should not live for days or months).
4. Test the update.
5. Delete (or consolidate) the snapshot.

Now, for those involved in storage, the snapshots mentioned in the above list are unrelated to storage snapshots: they are hypervisor snapshots. I often see issues because steps 1 and 2 are not done properly. Rollbacks are only possible if the VM dependencies are also backed up and snapshots created. For example, most people who have issues have made a snapshot of the vCenter Server but not of the database being upgraded, so when a failure happens, they roll back vCenter, but the database is incompatible.

This is where backups could assist. Ask yourself one question before you proceed:

Are you satisfied with your application and data backups?

This is a crucial question to answer. So, how do you proceed?

(1) vCD

Minimally back up and snapshot the vCD server as well as the server containing the database. I do not currently have vCD installed, so I skipped this step.

(2) vCNS (formerly vShield Manager)

Minimally back up and snapshot the vCNS (formerly vShield Manager) VM. Backups should include a separate output for your firewall configuration and rules in an easy-to-import format. Do not commit the vCNS server until after you have updated all your Edge and App components later down the line.

You will not be able to make modifications to Edge and App Firewall rules until after those components are upgraded. This was due to an incorrect password stored within SSO (see below).

(3) View Composer

Minimally back up and snapshot your View Composer, View Connection Server, View Security Server, and any database servers in use. If using replica Connection Servers, you can reinstall all but the first one easily enough. While View Security Server is not listed in the chart, I would suggest it be done after the View Connection Server.

(4) View Connection Server

If you receive the dreaded “The current logged on user does not have sufficient privileges on the existing Directory Services instance” error message during your upgrade, then log in to the View Administration panel. Visit View Configuration -> Administrators and ensure the user you are logged in as in order to perform the upgrade is also listed as a full administrator per the following (Figure 1).

Next, remove the VMware vCenter Operations Manager for Horizon View server. While KB2075114 states that this is not needed for upgrades from 5.x to 6.x, I found it was necessary, as otherwise the View manager does not start. You will need to reinstall this component for 6.x.

(4) View Security Server

Since my security server is located in a DMZ, I placed the VMware-viewconnectionserver*.exe file onto an ISO image and mounted it directly to the security server. This allowed me to transfer the files without opening up a new hole in my firewall.

Then, Delete All (or commit) all snapshots for the View Security Server, View Connection Server, View Composer, and any dependent virtual machines.

Leave provisioning off until after all other updates are completed.

(4) VSA Manager

Not having the VSA Manager installed, I skipped this product. But minimally ensure no critical systems are running within the VSA such as vCenter. Then follow our default steps. Backup, snapshot, upgrade, then commit. I do have a HP StoreVirtual, so this is when I performed any of its pending upgrades.

(5) vCenter Server

Minimally back up and snapshot the vCenter Server and any database servers or helper VMs (such as for Update Manager, etc.) in use by vCenter Server. A helper VM will be in use if you are using the vCSA as VUM, and other tools must run within Windows with v5.x of vSphere. Referencing How to recover VCSA 5.5 from an expired administrator account, by William Lam, I was able to recover my password to perform an appropriate upgrade. In my case, I just had to change the date of the last password change (first number) and the expiration age (third number) and not remove the ‘x’ in the front of the password. So, the password line was something like this:

root:...:16171:0:99999:7:::

A reboot per William Lam’s article, and all was well once more. Follow the rest of the article to reset the administrator password and to fix any password expiry. This is often quite important for security policies. To fix, you would edit the following once in the administration portal (using the :5480 port for the vCSA IP address). Be sure to put in the email for expiration notification, so that you can log in and change the password as required. Or better yet, pull vCSA into an identity management platform that handles these things for you. Also be sure to change your password after you reset your login. This will be required, as once you turn back on expiry, your password will most likely expire immediately and cause the update to fail (Figure 2).

By the way, this is where the .NET client comes into play. It is very hard to manipulate the console of vCSA without something talking directly to the console of the hypervisor on which it resides.

Now you just have to apply any updates, if they exist (or check for them if you have not enabled automatic checking). This is common to all appliances generated with VMware Studio (Figure 3).

We are not finished here; with 5.5U2b, we have to upgrade the web client integration packages as well, which can be downloaded from the web client directly. In addition, since it does not say otherwise, you should upgrade all your helper tools such as the VMware Update Manager (VUM), vSphere .NET Client, and web client integration packages.

For VUM, I needed to reset the SSO password I use for the VUM service account as it expired. There is no notification of expiry, so there is nothing that can tell me what is happening. Furthermore, I use VMware SSO for all my inter-VMware product service accounts. This way, I have no management dependencies on active directory (AD). If AD is unavailable, my management systems still work. I, however, use AD for all user accounts. So during this stage I reset all my SSO users to have an administrative email address that I read. I reset the SSO user passwords at need.

Delete All (or commit) your snapshots for vCSA, vCenter, your vCenter helper VM(s), and databases.

(5) vCenter Orchestrator

vCenter Orchestrator (vCO) is not mentioned in the schedule of updates, but since it is part of vCenter, I will schedule it to be updated after I update vCenter. The steps for updating vCO are the same as those for vCSA. First, take a snapshot of the vCO server and of any system with the vCO Client installed. Update the vCO server using check updates; install updates. Then, update any clients. Lastly, Delete All (or commit) your snapshots.

However, I also ran into a password issue with vCO Appliance. This required me to use the same approach as William Lam had for vCSA, but for the vCO appliance. And instead of mounting /dev/sda3, I mounted /dev/sda1 after booting from the KNOPPIX ISO image. The file to edit is /mnt/etc/vco/configuration/passwd.properties. The VMware vSphere 5.5 Documentation Center contains the appropriate document for resetting the password to default. VMware vSphere 5.5 Documentation Center -> VMware vCenter Orchestrator 5.5.2 Documentation -> Installing and Configuring VMware vCenter Orchestrator -> Configuration Use Cases and Troubleshooting -> Revert to the Default Password for Orchestrator Configuration.

I also ran into the same problem I had with VUM with respect to SSO. My vCO SSO service account expired and needed to be reset in order to complete the configuration updates.

 (6) vSphere Replication/Site Recovery Manager

vSphere Replication appliance updates just like the vCO and all other VMware appliances created by VMware Studio. The key is remembering whether the username is root or vmware. In this case, it is root.

SRM was different. I have it installed in a helper VM, so after creating a backup and snapshot, I went and performed the upgrade and ensured I installed the vSphere Replication (VR) bits as well. I also had to update the password within my SSO repository, as it had expired without warning. It does expire every ninety days.

Delete All (or commit) all snapshots for VR and SRM helper VMs.

(6) vCenter Operations

Take a snapshot of your Analytics VM and UI VM that makes up vCenter Operations (vCops). vCops is upgraded by using the administrative interface accessed via https://ipAddress/admin and uploading the PAK file. This was seamless. Remember to Delete All (or commit) your snapshots after patching.

(6) vCenter Infrastructure Navigator

Not on the schedule directly, but this is when I upgraded the vCenter Infrastructure Navigator (VIN). This update is just like the one for the vSphere Replication appliance and any created by VMware Studio.

(6) vCenter Log Insight

Not on the schedule directly, but this is when I upgraded the vCenter Log Insight (vLI). This update was quite straightforward as well, with no issues.

(6) VMware Support Assistant Appliance

Not on the schedule directly, but this is when I deployed the VMware Support Assistant Appliance (vSA). I did not previously have this tool deployed.

(6) vSphere Management Assistant

vSphere Management Assistant (vMA) is not on the schedule, but it should be updated now as well. This upgrade is like that for other VMware Studio appliance from VMware: take a snapshot, perform the upgrade through the web-based management appliance, then commit the snapshot.

(6) VMware Data Protection

VMware Data Protection (VDP) was tricky for me, as my password had expired. As with so many of these appliances, you need to be reminded to change your passwords. There was, however, a well-documented article, KB2039507, to assist after I made a snapshot of the VM. Once I did this, my upgrade failed several times, even after reverting to the snapshot. In my case, I could live without the backups made by VDP and start over. Which is what I did. However, if you have a need to keep all that data, you can delete the appliance but not the data disks and point the new VDP to the old data disks.

Even so, it seems extremely difficult to keep VDP in time sync with vCSA. When this fails (even off by only two minutes), the VDP appliance disconnects from vCenter, and your backups do not happen. Still looking for a better solution to this problem. The issue is that vCSA uses AD as the time source, while the ESXi hosts use a router. When we go to update ESXi in Step 7, a change can be made to keep vSphere’s time source the same as vCSAs, which should help VDP.

(7) VMware vSphere

This is by far the easiest of things to do. Rolling upgrades have been working for years. But this time, I need to make a time source change to use our AD server as the time source just to keep things in time-sync. There are several steps to take when performing a rolling upgrade:

1. Choose your host that is the host-profile master
2. Enter maintenance mode
3. Update the host
4. Exit maintenance mode
5. Update the host-profile from the host
6. Check host profile compliance.

For all other hosts, do steps 2 to 4 and 6 from the list above.

(8) VMware Tools

You should update VMware Tools on all your VMs at this time if you are using vCNS Endpoint or Data Security, as they could be using agents within the tools to make these happen.

(9) vCNS Edge

You can do these updates prior to VMware Tools, which is what I did. I needed my Edge gateways updated, as I use them all the time. However, I had the SSO password problem to fix, first with vCNS Manager in order to do the updates or even manage my vCNS Edges. Not sure why I did not do this in Step (2). I should have. Upgrades of vCNS Edge are fairly seamless, except that I needed to double check that nothing critical needed to be accessed when updating the Edge that sits before my iSCSI management layer. Nothing did.

You do have a backup of your firewall rules just in case?

(9) vShield App

I do not currently have this deployed, so no need to update. You do have a backup of your firewall rules just in case?

(9) vShield Endpoint

I do not currently have this deployed, so no need to update. You do have a backup of your firewall rules just in case?

(9) View Agent

As part of my View upgrade to version 6, I will need to do this step, but since I am deploying brand new desktops, the agent is part of this deploy and will be discussed in another post.

I have completed my upgrade to fix all the current releases of VMware products that have Shellshocked and other security issues. Now, I just need to finish my VMware Tools upgrades as I find the time to do so. This process was extremely time consuming due to issues with SSO timing out my critical passwords.

I also still have one remaining issue. VDP does not sync with vCSA for SSO login, except occasionally, due to clock mismatch issues. Yet all systems use the same clock, which is a bit worrisome. I will monitor and report as necessary. Currently, it works just fine.