Bending Your Containers with Bind Mounts

Last time we explored how Outrigger uses environment templating to allow easy configuration of common project customizations. It would be delightful if that could cover 80% of project customization needs, but often there are less-used toggles that need tweaking. The next step on your container journey is to look at how Docker Volumes allow overriding the configuration in your Docker Image on a file-by-file basis.

A Docker Volume is akin to a floating file system or directory you can attach to one or more containers at a path of your choosing. Since you decide where the volume is attached, you can decide whether this is new data, configuration, code… or an override to files already in the container. Volumes initially seem mysterious given how they are abstracted from your file system by Docker utilities, but as you learn the tools (and learn more about file systems as a result!) they become more understandable.

An alternative type of volume called a bind mount is more concrete for those newer to the devops journey. It allows files and data that live on your host machine to be used inside your containers. The downside is Docker cannot manage that data, and your container is less self-contained in its runtime requirements.

Bind mounts allow you to modify your Docker image by overriding it with files you have lying about your machine.

As container practices go, this is a cheat! Templating your configuration has finite, testable variations. Overriding your configuration on an ad hoc basis is a backdoor approach that compromises the "immutable infrastructure" promise of containers; however, if you are not yet in a place to work with customized Docker images this is an effective middle ground. We prefer this approach over customized Docker images only for non-production environment configuration changes that would otherwise require environment-specific Docker images.

For our exploration of how we can leverage bind mounts, we'll use the example of the Outrigger Apache & PHP image. By default, this container locates Apache configuration files at /etc/httpd/, the main configuration is /etc/httpd/conf/httpd.conf, and individual vhosts and other extended configs might be placed inside /etc/httpd/conf.d/. Even though we are using an Outrigger image as an example, these techniques and considerations can be generally applied.

Use Case: Adding a Configuration File

Imagine you want to add TLS support to our Apache container* by adding a new Virtual Host on port 443. This is not supported out-of-box but can be enabled by adding a new configuration file to /etc/httpd/conf/conf.d/docker-tls-vhost.conf.

Before we continue, please note that this is not a tutorial on implementing TLS support, it is a tutorial on working with configuration customization. Code specific to TLS support is likely incomplete, and there are further steps you would need to take before this would work in practice.

(* We recommend creating a separate TLS Termination proxy to handle SSL separately from the main Apache container. This better reflects the philosophy of single-purpose containers, and incidentally is a better match for most containers-in-production environments.)

First, let's define the configuration file we want to inject into the container:

<VirtualHost 0.0.0.0:443> 
  ServerName localhost
  ## Use the default docroot. 
  DocumentRoot "/var/www/html" 
  DirectoryIndex index.php index.html 
  ## Additional overrides for the default VirtualHost directory. 
  <Directory "/var/www/html"> 
    Options Indexes FollowSymLinks MultiViews 
    AllowOverride All 
    Require all granted 
  </Directory> 
  ## Logging 
  CustomLog "|/usr/sbin/cronolog /var/log/httpd/%Y-%m-%d.access_log" combined 
  ErrorLog "|/usr/sbin/cronolog /var/log/httpd/%Y-%m-%d.error_log" 
</VirtualHost>

Now, where should this file go? In order to ensure consistency of the project, this should be committed to the codebase. For clarity, use a single directory to capture all your configuration overrides, such as env/ at the top-level of the code repository. This shouts "I am the environment configuration!" which may be a slight distraction for day-to-day use, but this configuration should be trivially discoverable by intermittently participating Ops engineers that need these details.

Within the environment directory use subdirectories to categorize and isolate configuration that is only used for specific purposes. For example, a configuration only used with a specific service in a specific container such as our Apache config might be in ./env/www, and a configuration only used in the development environment might not on production may be in in ./env/development/www. The particular directory tree you use should be consistent, explicit, and only as complex as you need it to be.

Our file might fit best in an otherwise simple repo at ./env/www/apache.vhost.tls.conf

That's not all! Now that we've created and saved our configuration file, we need to adjust the project's Docker configuration to include that file in the running container. Let's look at a stripped-down docker-compose configuration for this service:

version: '3.4' 
services: 
  www: 
    image: outrigger/apache-php:php71 
    volumes: 
      # NEW APACHE CONFIGURATION FILE 
      - ./env/www/apache.vhost.tls.conf:/etc/httpd/conf.d/docker-tls-vhost.conf 
      # Inject the operational code into the container. 
      - ./docroot:/var/www/projectname 
    environment: 
      DOCROOT: /var/www/projectname 
      # For the sake of completeness, here's how to wire up this test database to Outrigger's DNS system.
    network_mode: bridge
    labels: 
      com.dnsdock.name: www 
      com.dnsdock.image: projectname

In the bolded text of example above, we are specifying our new configuration file inside the repo should be mounted and used inside the container at the path on the right side of the colon.

In this specific case, we are changing the name of the file to clarify it is not from the default Apache distribution when viewed inside the container. In our code repository the name help clarifies what the configuration file is for. This name change isn’t required, but shows the possibilities offered by bind mounts.

Our bind mount is adding a file where none existed before, but the running container does not know the difference.

Use Case: Overriding "Normal" Configuration

When building a Docker image, the files and configuration of an operating system and the essential system packages you need are put in place. These constitute the built-in layers of the Docker image, which we are free to override with bind mounts or volumes with the same freedom as we added a new file above.

Suppose we want to override the ServerAdmin or change other Apache configuration that should be inherited by all our other config files. In this case, we probably want to replace the main configuration file at /etc/httpd/conf/httpd.conf. There are instructions for copying a file out of the container in the Modifying a Templated Configuration section below, so we will move along to how we could work with our new ./env/www/apache.httpd.conf file. Be sure to read to the end of this section before attempting to run this example.

version: '3.4' 
  services: 
    www: 
      image: outrigger/apache-php:php71 
      volumes: 
        # Apache configuration injection. 
        - ./env/www/apache.vhost.tls.conf:/etc/httpd/conf.d/docker-tls-vhost.conf 
        - ./env/www/apache.httpd.conf:/etc/httpd/conf/httpd.conf 
        # Inject the operational code into the container. 
        - ./docroot:/var/www/projectname 
      environment: 
        DOCROOT: /var/www/projectname 
      # For the sake of completeness, here's how to wire up this test database to Outrigger's DNS system.
      network_mode: bridge 
      labels: 
        com.dnsdock.name: www 
        com.dnsdock.image: projectname

Now with two single-file bind mounts, you might start to wonder if switching to a directory mount might be easier. It certainly has the advantage of fewer overrides to manage, as well as allowing these files to be writable from inside the container. When a volume or bind mount is applied, it will replace everything in the container that would otherwise exist at the path to the right of the colon with the files found at the path to the left side of the colon. This means we would need to be prepared to manage the entire /etc/httpd directory tree in our codebase to switch to a single directory mount.

Our new bind mount is replacing a file that was "baked" into the Docker image with something that could be entirely different. Similar to the "new-in-the-bind-mount" example above, the running container accepts the replaced file without any friction (unless our new config file is malformed!).

There is one caveat to keep in mind that would keep this particular example from working as expected: if you try to run it you’ll find your custom apache.httpd.conf file changes keep reverting when you start your container. This is because the Outrigger Apache & PHP image generates the httpd.conf file via confd from a templated configuration file at startup. This brings us to our final use case …

Use Case: Modifying a Templated Configuration

Many of the changes that induce Outrigger users to override configuration files require overriding files that are already managed as templated configuration. This allows Outrigger images to provide some highly targeted levers for configuration tweaking. Templated configuration cannot be overridden based on file path where that file normally lives, making this the trickiest part of a Docker container to bend with our bind mounting technique.

To illustrate a typical case of templated configuration in the Docker image, the diagram below shows how a template file in the image is processed at container start to generate a "normal" configuration file.

The confd process running as part of the container startup means all volumes and bind mounts have already overridden the filesystem, giving confd the final say over what files will be available to the processes running inside the container.

In the Outrigger Apache & PHP image, we manage the primary vhost configuration via such a  template. If you were interested in overriding this configuration, you might look in the container, see the file at /etc/httpd/conf.d/99-docker-default.conf, and think creating a bind mount to override that file in the container would do the trick. That will not work because confd will override the change!

Identifying and Managing Template Files

There's no easy way to know that a given file is templated configuration. Inside the container, you might examine /etc/confd/conf.d to review the *.toml files which define the templating process, including the destination location for every templated file. Here's a shell script which will provide a list of all the overridden configurations:

# First, run the container you wish to examine, overriding the default behavior with a BASH shell 
# Change BASH to whatever shell is available in your particular image. 
docker run --rm -it outrigger/apache-php:php71 bash 
# Adapted from https://unix.stackexchange.com/a/148289 
cat /etc/confd/conf.d/*toml | grep dest | awk -F'"' '$0=$2'

If a configuration file you want to override is in that list, you should look for the the template, copy it out to your codebase, make your changes, and then bind mount the confd template. This will preserve the existing functionality from the Docker image and ensure it does not override your customizations. (Note that doing this will prevent your codebase from using updates to the Docker image for that template.)

My favorite trick for extracting files from a Docker image is to spin up the container with an extra bind mount directory. Anything copied into the mounted directory inside the container will be available on your host after the container is stopped.

# Run the container you need to customize, specifying BASH in lieu of it's default. 
# Change BASH to whatever shell is available. 
docker run --rm -it -v $PWD:/opt/export outrigger/apache-php:php71 bash 
# Copy out the file you want to customize. 
cp /etc/confd/conf.d/99-docker-default.conf.tmpl /opt/export/

Once you exit the container, you will find 99-docker-default.conf.tmpl sitting in your working directory, ready to be copied into ./env/www.

Once wired up with a bind mount in the docker-compose.yml, the correct file override flow will work: confd will use the overridden template to generate the config file.

If you found the technical concepts in this post helpful, you may want to keep the Outrigger documentation on Working with Volumes and Changing Container Configuration handy as more concise reference material.

Bind mounts are an incredibly powerful way to bend a Docker image to your immediate needs without needing to delve into Docker image building. With the knack of overriding pieces of your Docker image, you will be ready to move into deeper customization of Docker images for your projects, moving closer to the ideal of using standalone, pre-built Docker images as production release artifacts.