There is a new set of tools available for Caching up and down the stack which we covered within Caching through out the Stack, however in reality where is the best place to cache data for your application and what are the ramifications of using such a cache. Recently, we had a caching problem, actually two of them. Both caused by the same thing, a lack of full understanding about what was being cached. For any application, the best way to cache is to cache in memory as close to the application stack as possible, which in our stack could be within the application, the OS, or even a hypervisor based disk cache. However, which does your application actually use?That is the crux of the matter, if your application does not intrinsically use an in memory cache, you need to come up with some way for that to happen, which often implies using something within the operating system such as a disk cache that uses operating system memory or a disk cache below the operating system (Condusiv, SanDisk), within the hypervisor (SanDisk, VMware, PernixData, Proximal Data, etc.), which seems to be the trend these days with several companies developing and shipping products to use local SSD as a form of either write thru or read back disk cache (GreenBytes, DataCore, etc.) for those applications who do not intrinsically have the ability to make use of a cache.
But in some cases, even these disk cache mechanisms may not work, our first problem was with a disk cache which is supposed to generate lots of IOs per second (IOPs). Some disk caches are built into the underlying hypervisor or are built to present to the hypervisor as a form of storage; usually iSCSI or NFS. In the first case, everything moves at the speeds the hypervisor can attain, in the second case, data can move at the speed of the network available to the caching storage appliance.
But as we move up the stack (as seen in Figure 1), the cached data gets closest to the application. The closer the application is to the cache the faster that application could run, at least in theory. In reality, it entirely depends upon what is being cached and where that cache exists.

Figure 1: Caching Through the Stack
Figure 1: Caching Through the Stack
Take WordPress for example, is is a well used, well known application with many in application cache mechanisms. Most of these cache mechanisms end up using disk to aggregate data into a central repository. This cache can be a DB results cache, an object cache (of DB results or other items such as network query results), or a file cache (aggregation of javascript, css, etc.) All of these will improve performance or at least claim to increase performance, but if you do not know what is being cached, the sizes, and lifetime of the cache, and the limitation of the actual caching algorithms you may find that they adversely impact performance.
So what are those limits?

In App Caching: Memcache has a 1M limit depending on language used to program it. So you want to keep things small in this cache. Other in memory caches may not have such a limit. There are also caches associated with the languages used but that is often to cache compiled byte code on disk.
OS Caching: Caches mostly disk activity, so if you are not disk intensive this may be of limited use. There is also caches for network activity which could be beneficial depending on the application. Many anti-virus solutions use a local cache to alleviate the need to go over the wire every single time but that cache ends up having a short time to live.
Driver Caching: usually related once more to disk cache but if your app is network intensive this could have limited benefit. There are also network caches at this point as well but since the driver does not really understand the application these could be of limited use.

Then as we move down the stack we will end up with multiple layers of hardware caches related to network and disk. Which is often where we start to discuss storage cache mechanisms and why they are popular at the moment. It is an easy win and does not have to know the application to benefit the application. However, depending on how these storage caches are accessed may have other dependencies such as the network or fabric involved. What is boils down to is knowing how your application works and picking the best cache mechanism for that application. But it also requires knowing what parts of your data are cached where and how often they are accessed.
For example: Our application used a db cache, an object cache, and a network cache. We discovered that our db cache was trying to cache 6M of data into Memcache which could not cache more than 1M which caused the system to fail outright and would not work unless the cache worked. Well there are several issues here, but the caching algorithm in use was just working too hard it would attempt to cache, fail, server up the data and would happen all the time, which implies a cycle that would impact the application negatively.
Our solution was to find a way to understand what was being cached, where it was being stored, and the limitations of the cache mechanisms in use. Ultimately, we chose to use an entirely different cache mechanism, one that provided visibility into the actual items being cached. We also chose to use an in app cache as our application lives within a cloud where we do not have access to the underlying layers of the environment.
Ultimately, this is the deciding factor on which caching mechanism to use, while you may want to use a lower-level cache for storage, if you do not have the access, then you are limited in the choices you can make.