Application Performance Management – Our New Relic Details

We at The Virtualization Practice, practice what we write about, as such we have been monitoring our systems using many of our sponsors tools. One that has helped quite a bit has been New Relic RPM with its End User and Application Server monitoring capabilities as well as its inherent diagnostics to track through what is the root cause of our performance problems. Since we like to keep our website snappy, we constantly monitor for issues and ways to fix them. As such we are not only looking at the application, but our network and storage environments as well.   New Relic RPM however has surprised us quite a few times in what it can tell us. This is where performance monitoring and virtualization security overlap.
What is New Relic and Where does it Fit?
There are many application performance management tools, and this category of tools has existed for quite some time.  In “Why is Application Performance Management so Screwed Up?“, we went through why the first generation APM tools that primarily targeted large scale J2EE applications deployed on Weblogic and WebSphere were not going to make the leap into monitoring applications for dynamic environments (virtualization), private cloud (IT as a Service), or Public Cloud situations. We detailed why the APM industry is being reinvented by vendors like New Relic, AppDynamics, BlueStripe, Extrahop, Confio Software, and dynaTrace.
In “VMware Joins the APM Revolution with vFabric Application Performance Manager” we explained that VMware has now joined in the redefinition of APM with its new APM product, and that VMware intends to fully fold this new product into it longer term Service Assurance and Dynamic Operations strategies.
With that as background where does New Relic shine, and why do we run it? New Relic is an APM solution offered on a SaaS basis. The importance of this is that you do not have to install and maintain the back end monitoring system, its database, and its storage – that is all done by New Relic for you. You just have to install the New Relic agents into your applications. Now this is where things get interesting. One of the outstanding things about New Relic is the breadth of environments that it supports with its agents. New Relic is the only APM tool that provides deep code level diagnostics into Java, .NET, Ruby, PHP, and Python applications. Since we are a WordPress site, and WordPress is written in PHP, this alone made New Relic a natural fit.
The other thing about New Relic that is both a promised benefit and that has turned out to be a real benefit is that it is pretty much plug and play. You create your account on their back end, you download the agents you need and install them. Then you log onto their web based console (screen shots below) and you are up and running. This was crucial to us because, like many other people we are really busy. We do not have time to manually configure a monitoring solution to tell it what to monitor so that it catches the important stuff, but does not impose too much overhead itself. This is a huge distinguishing feature of second generation APM solutions – they just work (no consultant in the box).
So What did New Relic do for Us?
First we monitor using New Relic RPM both our production and development sites as we need to determine in development if what we are doing will ultimately improve the performance of the site. There are two interesting graphs of use to any LAMP developer.

The first is show in Figure 1, which displays render times from around the world, if you have it configured that way and how much of that rendering is network time, etc. The second is in Figure 2, which shows response times from within your application. The response time graph has been a cornerstone of our site improvements.

And as a side effect, we get extremely useful security information as well, namely what are the Web External sites being called by our own site. With the list from Figure 3, we are able to tell what is being called from our own web sites and to where. This is not only useful from a performance perspective, but if the elements listed do not show something that is even remotely expected, you may be infected with malware as most malware that makes it way into websites call home (really to a control server), and either dump sensitive information or as a means of control.

Even more interesting is the diagnostic capability of New Relic RPM, as it will show where your time is being spent within your application. It does this for Java, Ruby, and PHP web-based and other applications. In such a way that makes it easy to drill down to the offending call for further research. We were able to use this data to reduce the time it took for several bits of the code we use. So what did we do?

• Rewrote a plugin to save 10-20s every time it was called.
• Redesigned our website to only make Web External calls on request, instead of every page load.
• Determined that we are spending most of our time calling the database and for that we optimized by adding in a DB Cache functionality so that we do not need to re-query the same thing multiple times on a page.

All this from charts such as Figure 4, which show a break down of one call to the website that displays a post.

We can see that we are spending most of our time in the ‘the_content’ routine, with some close second by database queries.  To see exactly where our time is being spent, we can easily go into the trace details and note where the red marks are to hone in on the problem spots (Figure 5).

Which also shows us that the problem area is within Uninstrumented time, which is time that cannot be instrumented or determined because it is either within a binary or from our cookie-less domain. New Relic RPM just added in this Uninstrumented time as it used to just add the time onto the end, at least now I can look through ‘the_content’ functionality to see what is being called and why it takes so much time.
Given the complexity of modern web sites, tools such as New Relic RPM are incredible assets and a valuable tool to determine what is happening within your environment. While your data does go into their SaaS offering, all SQL and other statements are scrubbed.  The results of using this tool are incredible. While it takes a developer to make the code changes, the multi-user aspects of New Relic RPM even let those who are not developers understand where the performance issues exist and there is a side benefit of performing continual monitoring for some, but not all security breaches.