The last major modification to my datacenter occured on friday. I physically relocated my SAN and its UPS. I have been reducing the size of my datacenter, as I documented in VMotion – Redundant Power Supplies Aid in Migration, and started to document in Upgrading My SAN. I was waiting on this move until the new, much shorter, fibre cables arrived. Which they did earlier in the week.
Since this is a production virtualization cluster my crucial servers had to stay running at all times and one of my virtual machines that used an RDM had to be powered off for the least amount of time possible. However, I have found that it is best to plan for failure. To that end I made a disaster recover backup of all my VMs, and backed up the physical RDM to another server. These backups were started on the day before the move, and completed the morning of the move.
Each of my ESX servers is built using two 146GB SAS drives in a RAID 1 configuration. There are no other drives in the server. Given my default build this left a 100GB VMFS on each of my servers. The combined 200GBs was more than enough for my crucial VMs.
After the backups using vRangerPro and rsync from within a Linux VM with an attached RDM, I reached within my Virtualization Toolbox for Andrew Kutz’s SVMotion Plugin for the Virtual Infrastructure Client. Using the SVMotion plugin I was able to quickly move my most important VMs and a single development VM to local storage within the ESX servers. Then carefully shutdown the other non-crucial VMs, specifically one with a 500GB RDM.
Once all the VMs were either relocated to local storage or powered off, the UPS, SAN, and SAN Switches were relocated, reconnected and powered back on. To do so unfortunately, the rack had to be moved to another area with slightly less cooling. The moved UPS power cord would not reach to the outlet. More on this later is this post.
The first order of business was to reconnect the fibre channel SAN to the switches and the switches to at least one host. Once that was achieved and we had light on the cable, the VMs I powered down were powered on including the VM with the large RDM.
Total down time for this VM was less than 20 minutes. Total downtime for the crucial VMs was 0 minutes.
Now the rest of the nodes were connected to the fibre channel switches. Two physical boxes were moved to the relocated UPS (which requried these systems to be powered off as they lacked redundant power supplies). Lastly, power was balanced between all the systems in the rack to which the SAN and its UPS. During this move we also went from an APC RS1500 w/XS1500 extender to an APC SUA2200RM unit. This alleviates all the overload warnings we were getting on the smaller UPS.
Once all the cabling was completed, the VMs were migrated using SVMotion from the local storage back to the SAN.
Total downtime for crucial infracture was 0 seconds. Total downtime for non-crucial infrastructure was no more than 20 minutes.
However, I was not finished. After a trip to Home Depot for a NEMA 5-20R, 5-20P, and 8 feet of 12/2 flexible cable, I was able to create an 8 foot extension cable for the UPS and move the rack back to the area that gets the best and most efficient cooling.
Things to take from this:
- Always have secondary VMFS storage: either local storage, which I prefer, or some other SAN storage available to which you can SVMotion VMs as needed.
- SVMotion is an invaluable tool for the Virtualization Toolbox
- Plan any major relocations carefully
- Always make a backup before any major relocation.
There are a few more upgrades to my infrstructure that are planned but nothing that would require such drastic measures. This was all done as a way of saving on power costs, and I have been able to do so. With the SAN relocation I was able to remove a switch, PDU, and UPS from my power mix.
We will have to see what the new savings will be once the latest bill arrives.
Your post is quite interesting and useful. I have bookmarked this site for later use to see what other great articles you post!