OpenShift, Why Won’t You Do What I Want?

I recently spent a fruitless afternoon on the public PaaS version of Cloud Foundry. In this post, I document an equally fruitless afternoon spent on Red Hat’s OpenShift. It think it is fair to say that OpenShift has some advantages over Cloud Foundry for public PaaS. OpenShift feels more comfortable, its integration of a build server introduces a lot of flexibility into its deployment, it makes it easier to know what is going on, and it seems to have more documentation and more discussion on the forums. However, once you veer away from the standard use case, it doesn’t work terribly well. Ultimately, I failed to get it to do what I wanted, but maybe it was just too hard.

To recap, I am trying to deploy the same Groovy/Grails/Tomcat/MySQL/JavaScript ISV application to various platforms, using tools provided by the vendor while maintaining elements of the existing application lifecycle. OpenShift runs applications in “gears,” which are essentially locked-down Red Hat Enterprise Linux (RHEL) environments configured with the necessary infrastructure. For Java or Java-derived applications like this one, there are three basic ways of getting your program into your gear. The first is to use a standard Maven src structure in your application workspace and push that to the OpenShift GIT. OpenShift will build it as part of the deployment. (Note that it will stop your application while it is building!) The second is to prebuild it (e.g., on a developer workstation) and drop the WAR file into the webapps directory in the workspace before pushing it to the OpenShift GIT. OpenShift will deploy that WAR file into the gear. The third is to use OpenShift’s built-in build server, Jenkins. A GIT push triggers a Jenkins build job to create a WAR file, which Jenkins uploads to the gear. It is this third approach that I am planning to take, because:

1. This approach doesn’t stop the server during the build.
2. It is the most flexible and maps the most closely onto the current build processes.
3. It explicitly creates the the WAR file, which is required for on-premises deployments of the application.

So, the first step is a standard OpenShift application creation process. It’s quite a tortuous on-ramp. You get a certain distance via the OpenShift web GUI, and you find you need some SSH security keys. The way to obtain them is to install the command line, so you have to install Ruby and GIT and then the OpenShift rhc command line tools. Make sure you check the option regarding path variables when you install Ruby on Windows; also check the option required to get GIT to work from the command line. Although both are flagged in the documentation, they are easy to miss.

At this point, you run the rhc tool for the first time, and it sets things up for you and copies your public key to the server. It won’t be entirely obvious that it has done this, and there is a delay in the process that can easily confuse you for a while. In addition, the rhc tool helpfully tells you to change the permissions on your private key file using chmod. This doesn’t necessarily work on Windows. However, all of these are minor niggles. The process does work. So, at this point, you have your application created, you can define the various services that are associated with it (MySQL, for example), and you need to start thinking about how to get your code into your gear.

In OpenShift, Jenkins is provided inside its own gear, which is completely separate from your application gear(s) and can be provisioned through the OpenShift console or via the command line. Provisioning again may take a little while, but when it is complete, you get to log in to a nice, familiar Jenkins console. The console is preconfigured with a build job that seems to contain all the magic steps required to check your application out of GIT and upload it to your gear. All you have to do is put in place any additional steps required to build the application.

In case you are not familiar with Jenkins, it is an extremely flexible build environment that checks code out of a source code repository and can then run almost anything either through a shell script (embedded in the build job) or through an enormous number of plugins. Most real-world build jobs require the use of various scripts and plugins. In OpenShift, Jenkins comes preconfigured with a number of plugins, including, to my surprise, one that allows it to operate directly against an externally hosted subversion (SVN) repository. Since I already had a hosted SVN environment, I was able to point Jenkins at this and not concern myself at this stage about using the OpenShift GIT repository.

Having successfully checked out the code into the workspace in Jenkins, I still needed to get the following three elements to work:

1. A number of Unix shell scripts that retrieve some external JavaScript libraries via curl and copy some other libraries into the correct place
2. A Groovy script, which is run for wrapping and packaging
3. The grails war command (this is actually executed using a grailsw script).

Step one, the shell scripts, was very straightforward; they can be run by Jenkins inside the Red Hat environment provided in the Jenkins gear. They manipulate the contents of the workspace, which is fine; there are no problems with doing this inside your gear.

Steps two and three proved more problematic. In the existing build server, they are executed by, respectively, the Groovy and Grails plugins to Jenkins. The Groovy plugin is installed (though, in my case, it wasn’t configured properly; see below). Grails isn’t installed.

Jenkins has an extremely easy mechanism for installing plugins through the GUI from a central repository on Jenkins.org. Unfortunately, the mechanism doesn’t seem to work on OpenShift.  Specifically, the list of available plugins appears to be empty. At first sight, this would seem to imply that Red Hat is trying to stop us from using Jenkins plugins. However, if you look at the forums carefully, this turns out not to be the case. There are number of the people with the problem, and Red Hat has engaged in a number of conversations in which they have attempted to resolve the problem, with varying degrees of success. In fact, judicious use of Firebug (the debugger inside Firefox) led me to the problem: Jenkins is installed on secure https on OpenShift. By default, the plugin repository is on insecure http. The browser won’t allow mixed active secure and insecure content in an iframe, so the GUI can’t list the plugins. If you change the repository URL in your Jenkins on OpenShift to point to the https version of the repository on jenkins.org, you get a nice long list of plugins. So I posted the answer to the forums, and at this point I realized there couldn’t be very many people in the world who had successfully installed plugins into Jenkins on OpenShift. Maybe I was veering dangerously off the well-trodden path.

Once I had my list of plugins, I was able to install the Grails plugin successfully. Then, I needed to configure the Groovy and Grails plugins. Both plugins are essentially “wrappers” and need to install executable versions of  both Grails and Groovy. They will both do it automatically if you point them at the right repositories on codehaus.org. I was slightly worried at this point—would I be allowed to install Grails and Groovy into my Jenkins gear? I never got far enough to find out. It turns out that this too requires access to an http site from an iframe, so it doesn’t work on OpenShift. In this case, you can’t work around it by pointing to an https repository, so neither plugin actually works.

At this point, I nearly gave up. However, it is actually possible to run both Groovy and Grails from shell scripts inside Jenkins without using plugins, so in the spirit of adventure, I decided to have a go. The approach that I took was to get Jenkins to download Groovy into my workspace (immediately after checking the code out from SVN) and run Groovy from there. I had to set up some environment variables using another Jenkins plugin called the Environment Injector Plugin.  This worked, and thus step two was accomplished.

For step three, I decided to execute the grailsw script directly from the command line. This, as it turns out, does not work on OpenShift. It falls over because it doesn’t have the necessary permissions to change its environment. Confronted with this issue, at this point I stopped. I’m fairly convinced that with a few more hours’ work it would have been possible to run this build on OpenShift, although it wouldn’t have been anything like the same Jenkins job as currently runs on the ISV build server. However, I’d already spent at least as long getting the application working on OpenShift as I had with my previous failed attempt on Cloud Foundry. To provide a fair comparison with Cloud Foundry, I decided to stop.

The whole exercise has nevertheless been extremely instructive. It is quite clear that the big challenge for PaaS in the enterprise in 2014 will be getting the application lifecycle to play properly with the tools provided. I’m not really doing anything complex here.