My latest vSphere 6.7 upgrade issues were troublesome, frustrating, and problematic. VMware Update Manager (VUM) would not update a host. I tried to do an update many ways at many times, but it always showed as out of compliance even though the version of ESXi installed was the most recent version. VUM was broken. Or was it?
After some research, I considered the following possibilities:
- The issues within vSphere Upgrade Saga: Finally ESXi 6.7
- A pending update may have been unfinished
- There may have been a duplicate or bad VIB
- The nodes may have been different install bases (HPE vs. VMware images)
This problem was vexing, as it was none of the above. I used remote esxcli commands to investigate the host. Then, I moved to looking directly at the host—specifically, at the locations used by upgrade packages. One such investigation led me to realize that /locker/packages directory was not accessible. In fact, it looked corrupt. The solution was within this VMTN Community thread.
Why did this directory become corrupt? All I can think is that my use of SD cards must have been the issue. Why I went with SD cards I documented in my vSphere Upgrade Saga: VSAN Try 2 post. SD cards have their own issues and are often corrupt. This was not the first time, but it definitely was an oddity.
Once I fixed the locker directory, I was able to remediate, which did nothing but fill out the proper directory structure. This proper directory structure was required to allow VMware Update Manager to verify compliance against the chosen baselines.
So, how do we determine if the SD card is corrupted? Is there a remote method we can use? If SSH were enabled on the host, then the directories could be scanned or a file checked as it is done in this post. However, SSH is not enabled, and it should not be enabled. Thus, we need another method to scan for corruption.
Another potential approach is to use vifs to scan a particular set of directories as long as the scan is of a filesystem or file that is not within the protected locations of the hypervisor. However, I have yet to get this method to work appropriately. It could work, but then it would have to look at a directory to which data is being written that is also local to the SD card.
I have already had one SD card go bad; this is expected for SD cards. This is why you want your temporary file space to be on another disk and not on the SD card. It is also possible to put /locker onto another location as it too is just a directory. But that does not answer the “Is the SD card corrupted?” question.
Options exist, and as always, they are impacted by security. Which approach works best is yet to be decided. I will update once I find a viable solution!
