Abstraction Is the Future

The IT industry is abstracting everything it can, from hardware, to networking, to storage and more. “Abstract and build an API, and the world becomes better” seems to be the motto. Abstraction has become the goal of many subsystems. Many people seem to believe that if you move everything to software, everything will be better—and that if you use the API, things will move faster. This is all true, up to a point. Abstraction is a good thing, and we are now seeing levels of abstraction even within the hardware itself. This is also a good thing.

So, what is it about abstraction that bothers me so much? I use it daily myself. Abstraction may be the future—the here and now—but we can never abstract away the requirement to use some form of hardware. We cannot forget that at the base of all our abstractions is the hardware. Yes, by abstracting the hardware, we gain more efficiency. When we program to an API, we can alleviate human workflows. These themselves are worthy goals. However, even so, we cannot forget that underneath everything is hardware.
EMC Neutrino Nodes are a new breed of hardware, one where abstraction is almost baked in. This is a new style of manager that is easy to program to, and it is easier to use with some of the more modern tools. We keep moving abstraction lower and lower.
Yet, the question of how you debug a problem when all the layers are abstracted remains. Is the problem within the abstraction layers themselves? Or is the problem within the hardware? To debug the program, we need something more: knowledge, which we seem to be losing fast in favor of further abstraction. We are now looking to tools to provide us with this knowledge.
Zenoss is an infrastructure monitoring platform. It takes many of the usual day-to-day worries about hardware and brings them into an analytics package to give us insight into our data and therefore into our hardware. However, simple things seem to be missing from monitoring today, such as chipset features, failure modes of the hardware itself, and other system-level information. Oversights, I am sure.
When we abstract the network, we tend to overlook the fact that there is only so much bandwidth available. Network functions virtualization, if not properly optimized, could cause a massive amount of bandwidth to be used. Consider the use case of four billion web requests per day. The number of new sessions per minute exceeds the limits of most firewalls. As such, in many cases, we do not use firewalls. If we instead use NFV to inject the load balancer, the application delivery platform, and other checks and balances, we may end up quintupling our bandwidth requirements. At those numbers, it could saturate a 10 GbE link easily. We actually have to understand the hardware underneath to use NFV abstractions within our network. We cannot yet segregate ourselves from the wires.
We abstract storage as well, by placing before it a device that maps from block to object to file. This is also a good idea, but I often wonder how well it scales to billions of lines of logs being written per day from an application such as the one I described. Perhaps well, but front-ending an object store with a file interface to do this could be a bit of overhead. This is also where hardware helps us, such as using hardware caching mechanisms as well as flash. Once more, we are talking about hardware to discuss abstractions.
Abstraction and use of APIs is a scale choice, but the real question is how well they scale. Many administrators, operations people, and security folks may be satisfied with simple scripting methods using well-known interfaces to script their operations, such as updating the entire environment. Some may just stick to simple Puppet scripts and eschew tools such as Ansible, Vagrant, etc.
We have been through abstraction phases in the past, and we are in one now. What is really changing is not how we look at the hardware, or how we consider it; rather, we are changing how we manage that hardware. We have more tools available that allow us to bypass GUIs and other mechanisms that take time. This is not a change for those who were in the UNIX/Linux world for years; those administrators are the basis for tools such as Puppet. What has changed is that Windows administrators now have some of the same tools and capabilities available to them.
However, do not let abstraction distract you from the underlying capabilities of the hardware.