vSphere Upgrade Saga: Move the Data Center: Step 1

We recently moved from Boston to Austin, in doing so, we had to move our data center but we still need to maintain uptime while we physically move the data center. The obvious answer was a Cloud presence to help us with the move. While this move was planned for a bit, finding an inexpensive but good provider was difficult.

We choose to use an Infrastructure as a Service (IaaS) based cloud. Into it we planned on moving 4 carefully selected workloads. Which workloads to move to the cloud was a relatively easy decision based on how our virtual environment was already split. Our customer facing systems were located together and where very well understood due to the level of monitoring we had in our virtual environment.

We moved to the chosen cloud the following systems:

• External Web services
• Email
• Database to support external services
• A Jump machine to access customer systems (a static IP was required) as well as our other cloud systems

We chose just these 4 workloads based on which of our 100s of virtual machines were required to keep the business running while the datacenter moved. The only truly required systems were those that were externally facing used for communication with customers.

Moving these workloads to the cloud could have been accomplished by direct copy of the workloads, but only if the cloud I chose had the support and if I was already running the workloads conveniently within a similar cloud structure, but that was not the case. So instead, I had to set up everything from scratch. The steps where straightforward:

1. Within the cloud service deploy the virtual machines from a template, my four templates were 2 minimal CentOS, 1 minimal CentOS + MySQL, and 1 LAMP Stack.
2. Deploy the applications from within the cloud virtual machines, this was a simple as setting up MailScanner + Postfix + Dovecot (which I wrote about previously in my Email Upgrade Saga) within a minimal CentOS, Setup WordPress within the LAMP Stack, Setup MySQL inside the minimal CentOS + MySQL, install the necessary communication tools within the last minimal CentOS.
3. Copy my Data Over from the production systems. This stage presented the most technical issues.
4. Transfer DNS from my existing DNS services to the Cloud Service, which required the Cloud to put in a reverse name look-up.
5. Wait 24-48 hours until all internet DNS servers were updated before performing final data migration
6. Shutdown old workload virtual machines

These steps seem fairly straightforward and they were. The big choices where to use the existing WordPress installation or start from scratch, I choose to use the existing WordPress stack. While deploying everything I created a script to handle the deployment in the future, just in case I have to do it again or I migrate my workloads to another cloud provider or back to the main site. There are some key considerations that I found when using a cloud service provider that could be helpful to others. Items that are hypervisor dependent as each hypervisor handles memory over-commit differently. The cloud I chose runs Citrix Xen Server while my workloads generally run on VMware vSphere.

1. Shutdown any unnecessary processes, this includes Cron jobs. The problem one for me was cron jobs, there are a bunch that chew through all available memory to create the Whatis directory. Disable makewhatis and their ilk.
2. Tune your LAMP stack to use only what is necessary. No need to run 1000s of web processes when the server will work just fine with a lower number but be ready to add more as necessary.
3. Use an Application Performance Management tool such as New Relic to track everything your web application does and re-tune as necessary. Such tools can also email you when the site is down due to the cloud provider availability issues (if any) or system crashes from using too much memory or other reasons.
4. Be a good public cloud citizen, you are sharing resources, so tune your application for those shared resources, what may work within your datacenter may not work as expected within a cloud.

The key is to plan for all aspects of a migration to the cloud, even if it is only for a short time. In this case, migrating the application code was the easy part with data migration being the more difficult part. We had to migrate the following data:

• WordPress database records (twice for some aspects of the data). The second data migration was a limited subset based on knowledge of the data. The goal was to ensure we had the most up to date statistics data after the DNS switch over, which can take 48 hours in some places.
• Email accounts. Email account data, such as folders, and unread email, etc. had to be migrated.

A well orchestrated plan, scripting, and knowledge of your data will improve any cloud migration. But also understand not only how the application works but how the underlying operating system works. The makewhatis cron job caused our cloud services to crash with out of memory issues. It was not the application running that caused the problem, but a relatively useless cron daemon that is available on all systems.

Would we have caught that with a scripted install? Not really, each cloud is different. Now it is part of our scripted install of the OS and applications!