Share configuration settings between docker-compose files

Docker compose supports 2 methods for sharing configuration settings:

Expand a docker compose file using multiple docker compose files
Expand each service with the school extends field (Only applicable to compose docker versions 2.1 and above)

How to use multiple docker compose files

By using multiple docker compose files, it allows you to customize the docker compose configuration settings for different environments or business streams.

Mechanism works when using multiple docker compose files

By default, docker compose reads 2 files: docker-compose.yml and docker-compose.override.yml. By convention, then docker-compose.yml will contain base configuration configurations. And the file overwrite docker-compose.override.yml will contain configurations used to override existing service configurations or even define new services.

If a service is defined in both files, the docker compose will merge the service-related configurations according to the principles described in https://docs.docker.com/compose/extends/#adding. -and-overriding-configuration

To use more files to overwrite, or to use an overwrite file with a different name than the standard, you can use additional parameters -f to define a list of files to be used. Docker compose consolidates the configurations in these files in the order they are defined at the command line. You can see more information about the parameter -f at https://docs.docker.com/compose/reference/overview/

When you use multiple configuration files, you must make sure that all the paths of these files are relative to the base compose file (the file is listed first right after). -f in the command line).

Example of using multiple docker compose files

There are two fairly common cases for using multiple docker compose files:

Compose docker compose configurations for different environments
Run the admin tasks

Configuration for different environments

One of the common cases when using multiple docker compose files is to change the docker compose configuration for an environment like production (or staging, CI). To support this, you can break down the docker compose configuration into quite a few files:

Start with a base file that defines the standard configuration for services.

docker-compose.yml

web:
  image: example/my_web_app:latest
  depends_on:
    - db
    - cache

db:
  image: postgres:latest

cache:
  image: redis:latest

In a dev environment, you need to configure to expose a few ports connected to the host, need to mount the code or build the web image

docker-compose.override.yml

web:
  build: .
  volumes:
    - '.:/code'
  ports:
    - 8883:80
  environment:
    DEBUG: 'true'

db:
  command: '-d'
  ports:
    - 5432:5432

cache:
  ports:
    - 6379:6379

When you execute run the command docker-compose up then the configuration settings will be overwritten automatically

Now, we create docker compose file for production environment. Therefore, you need to create a log file for the production environment

docker-compose.prod.yml

web:
  ports:
    - 80:80
  environment:
    PRODUCTION: 'true'

cache:
  environment:
    TTL: '500'

To deploy to this production environment, we need to run the command

docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d

This command will deploy deploy 3 services using the internal configuration docker-compose.yml and docker-compose.prod.yml

Configuration for administrative tasks

Another common case is running administrative tasks in a Compose app, such as backing up a running database.

docker-compose.yml

web:
  image: example/my_web_app:latest
  depends_on:
    - db

db:
  image: postgres:latest

docker-compose.admin.yml Implementing a new service to run database export or backup.

dbadmin:
  build: database_admin/
  depends_on:
    - db

To create a regular environment we just need to run the command docker-compose up -d . To run a database backup, we have to embed it docker-compose.admin.yml

docker-compose -f docker-compose.yml -f docker-compose.admin.yml
    run dbadmin db-backup

How to expand to each service

Key word extends Allows sharing shared configurations between different files, or even different projects. How to expand the service is really useful if you have several services that reuse a set of options configurations. Using extends You can define a set of services configurations in one place and reference it from anywhere.

Note that volumes_from and depends_on Never be shared between services using extends . These exceptions exist to avoid potential dependencies; You always have to define volumes_from inside each service. This ensures that dependencies between services are completely visible when reading in the current configuration file. This explicit definition ensures that changes in reference files will not break anything.

The mechanism of action of expanding services

When defining any service in docker-compose.yml , you can declare extended service as follows:

web:
  extends:
    file: common-services.yml
    service: webapp

This directive tells Compose to reuse the configuration webapp The service is defined internally common-services.yml file. Assuming that common-services.yml would look like this:

webapp:
  build: .
  ports:
    - "8000:8000"
  volumes:
    - "/data"

In this case, you will get the same result as you wrote in the main text docker-compose.yml with the same configuration values build port volumes right below web

Moreover, you can redefine the settings that are set in docker-compose.yml :

web:
  extends:
    file: common-services.yml
    service: webapp
  environment:
    - DEBUG=1
  cpu_shares: 5

important_web:
  extends: web
  cpu_shares: 10

You can also write other services and associate web services with them:

web:
  extends:
    file: common-services.yml
    service: webapp
  environment:
    - DEBUG=1
  cpu_shares: 5
  depends_on:
    - db
db:
  image: postgres

Examples of use cases

It is useful to extend each service by direction when you have multiple services reusing a shared configuration set. For example, below is a Compose app with 2 services: a web app and a queue worker. Both of these services use the same codebase and share many shared configurations.

in 1 common.yml We define the shared configuration:

app:
  build: .
  environment:
    CONFIG_FILE_PATH: /code/config
    API_KEY: xxxyyy
  cpu_shares: 5

in 1 docker-compose.yml We define specific services that use shared configurations:

webapp:
  extends:
    file: common.yml
    service: app
  command: /code/run_web_app
  ports:
    - 8080:8080
  depends_on:
    - queue
    - db

queue_worker:
  extends:
    file: common.yml
    service: app
  command: /code/run_worker
  depends_on:
    - queue

How to add new and override the configuration

Compose replicates the configuration from the original service to the local service. If a configuration setting is defined at both the original service and the local service, the configuration value at the local value will replace the original value.

The configuration example has a single value image command mem_limit then the new value will replace the old value:

# original service
command: python app.py

# local service
command: python otherapp.py

# result
command: python otherapp.py

For configurations with multiple type values ports expose external_links dns dns_search tmpfs , Compose will merge both value sets from original service and local service:

# original service
expose:
  - "3000"

# local service
expose:
  - "4000"
  - "5000"

# result
expose:
  - "3000"
  - "4000"
  - "5000"

In case environment labels volumes devices Compose consolidates configuration values in order of priority. Eg environment and labels

# original service
environment:
  - FOO=original
  - BAR=original

# local service
environment:
  - BAR=local
  - BAZ=local

# result
environment:
  - FOO=original
  - BAR=local
  - BAZ=local

Eg volumes and devices

# original service
volumes:
  - ./original:/foo
  - ./original:/bar

# local service
volumes:
  - ./local:/bar
  - ./local:/baz

# result
volumes:
  - ./original:/foo
  - ./local:/bar
  - ./local:/baz