When we were first introduced to Docker containers, we were told that they should be ephemeral and stateless. It turns out that there are lot of great uses for Docker containers that are either stateful or long-lived. The stateful part is leading to a bunch of startups that have storage products to deliver persistent storage to these stateful containers. Even our old friends at VMware are in on this game. Their Photon platform is going to include VSAN to provide persistent storage for containers.
What Is Stateful?
Stateful essentially means that when we ask a question of the application, the answer can be different depending on what we previously did with the application. If we ask the application “What is the speed of light in a vacuum?” it will always give the same answer. This is stateless. If we ask the application “What is my name?” then the answer depends on what we previously told the application. This is stateful. Almost every application is stateful in some form. With container-based applications, the question becomes where the state is held.
Where Is That State?
While almost every application has state, not every component needs to have state. Microservices are about breaking the application into smaller pieces, each piece being made up of containers. Some of those containers are stateless, and others maintain their own state. Stateless containers often rely on something external to hold application state, maybe a database or an object store. A stateful container holds state inside the container, either in memory or on disk. This state inside the container can be rebuilt from an external source, like that database or object store. Essentially, the state in the container is a cache of the central state store.
Cache Rebuild at What Cost?
The next question is whether rebuilding that stateful cache is an “expensive” operation. If the state can be rebuilt rapidly without impacting the central state store, then there is little reason to persist the state. Just re-create state cache when a new container instance is started. Let the state cache be destroyed when the instance is killed. This kind of transient and disposable state is more like statelessness; there is no need to maintain the state of the container. But what if it takes five minutes to rebuild the state? Or if rebuilding the state places an unacceptable load on the database server? Then it makes sense to persist the state. Detach the stored state information from the instance before killing and reattaching it when a new instance is created.
Really Stateful
There are some applications that share state within containers rather than caching an external database or object store. A group of containers together provide the state store either for themselves or as a store for other containers. Other applications have containers that generate state from user input. Sometimes a container instance will be created for a user and will handle that user’s whole interaction with the application. As both of these types of containers are the source of some unique information, we need to treat their state with great care. We will need persistence, availability, and backups to ensure data durability.
Where Should Container State Persist?
If we do need to maintain a container’s state, where should that state be stored? Docker provides the Volumes interface, which allows a directory to be passed to a Docker container. This directory is persistent, while the container’s other file systems are not. The basic Volumes functionality suits persistence within a container host or on an NFS server. As usual, Docker has batteries included, but swappable. Storage vendors can write a driver to use the Docker Volume function with their own storage. So, where should you store state for your containers? That depends a lot on the nature of the persistent data. Caches can be rebuilt, so maybe the basic Docker volumes will work. The consequence of losing the persistent cache is simply a performance impact.
Where the data is the primary copy, not a cache, then we need more care. If there are only a small number of containers, then NFS may still suit. Make sure the NFS server is reliable and backed up. However, if you will need to store the state of thousands of containers, then you might need something more. Expect to use some sort of scale-out architecture to avoid performance bottlenecks and availability risks as you grow. This is the space where I expect VMware’s VSAN will be interesting. At the same time, there are plenty of other solutions. Using Ceph is a common solution for scale-out file and object storage. Then there are startups like StorageOS, which is building a persistent storage platform for containers that is itself run in containers.
Persistent storage for containers is a fact of life for a lot of deployments. There are multiple reasons and architectures for this persistent storage. You may be able to accommodate all the requirements with a single storage platform, or you may need to support multiple platforms for multiple requirements.