Andreas Mausch

Docker Swarm: Wait for deployment

I love Docker Compose files for their simplicity and use them a lot to start local services.

However, when using them in a Docker Swarm, you always need to take precautions to validate a deployment was successful.

If a health check failed or the image could not be pulled, the docker stack deploy command just finished without any error message, because the real deployment was done asynchronously after the command already exited and returned to the shell.

Today I just found out it is finally possible to wait for the result of the deployment by using the --detach=false mode. The command will block until the state of the swarm is stable again.

This is so good. It allows automatic scripts to check whether the deployment was successful or not, without any self-written scripts checking the stdout of docker service ps commands.

The pull request was merged in March 2024 (link to issue). Make sure to use a Docker version >= 26.

Ansible also supports the flag.

1. Example with Keycloak

✔︎asciinema22:56:25✔︎asciinemadockerstackdeploy--detach=false--compose-filedocker-compose.yamlteststackCreatingnetworkteststack_defaultCreatingserviceteststack_idpoverallprogress:0outof1tasks1/1:1/1:assigned1/1:startingoverallprogress:overallprogress:1outof1tasks1/1:runningverify:Waiting5secondstoverifythattasksarestable...verify:Waiting4secondstoverifythattasksarestable...verify:Waiting3secondstoverifythattasksarestable...verify:Waiting2secondstoverifythattasksarestable...verify:Waiting1secondstoverifythattasksarestable...verify:Serviceu4oov19n5q5srpfron5pt83bkconverged✔︎asciinema22:56:56✔︎asciinemadockerstackservicesteststack22:56:56IDNAMEMODEREPLICASIMAGEPORTSu4oov19n5q5steststack_idpreplicated1/1quay.io/keycloak/keycloak:26.1.3*:8080->8080/tcp✔︎asciinema22:57:03✔︎asciinemadockerservicepsteststack_idp--filter='desired-state=running'IDNAMEIMAGENODEDESIREDSTATECURRENTSTATEERRORPORTSt48z5dvdd2hateststack_idp.1quay.io/keycloak/keycloak:26.1.3neonew-xps9310RunningRunning18secondsago✔︎asciinema22:57:09✔︎asciinemadockerstackrm--detach=falseteststack22:57:09Removingserviceteststack_idpRemovingnetworkteststack_default✔︎asciinemadocker22:57:20✔︎asciinemadockerstack22:57:20✔︎asciinemadockerstackls22:57:20NAMESERVICES✔︎asciinema22:57:23✔︎asciinemadockerstackdeploy--detach=false--compose-filedocker-compose.yalteststack1/1:new✔︎asciinemadockerstackservicesteststack22:56:56✔︎asciinemadockerservicepsteststack_idp--filter='desired-state=running'✔︎asciinemadockerstackrm--detach=falseteststack22:57:09✔︎asciinema22:57:20✔︎asciinemad22:57:20✔︎asciinemad22:57:20✔︎asciinemado22:57:20✔︎asciinemadoc22:57:20✔︎asciinemadock22:57:20✔︎asciinemadocke22:57:20✔︎asciinemadocker22:57:20✔︎asciinemadockers22:57:20✔︎asciinemadockers22:57:20✔︎asciinemadockerst22:57:20✔︎asciinemadockersta22:57:20✔︎asciinemadockerstac22:57:20✔︎asciinemadockerstackl22:57:20

For example let's have a service, in this case a Keycloak:

services:
  idp:
    image: quay.io/keycloak/keycloak:26.1.3
    command:
      - start-dev
    environment:
      KC_BOOTSTRAP_ADMIN_USERNAME: admin
      KC_BOOTSTRAP_ADMIN_PASSWORD: admin
      KC_HEALTH_ENABLED: "true"
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD-SHELL", "exec 3<>/dev/tcp/localhost/9000 && echo -e 'GET /health/ready HTTP/1.1\\r\\nHost: localhost\\r\\nConnection: close\\r\\n\\r\\n' >&3 && cat <&3 | grep -q '200 OK'"]
      start_period: 10s
      interval: 5s
      retries: 12
      timeout: 5s

Test the docker-compose.yaml without Docker Swarm:

docker compose up
docker compose rm --volumes

Now run the following commands and notice the --detach:

docker stack deploy --detach=false --compose-file docker-compose.yaml teststack
docker stack services teststack
docker service ps teststack_idp --filter='desired-state=running'
docker stack rm --detach=false teststack

You must be connected to a swarm in order to run docker stack commands.

Some more commands for debugging:

docker stack services teststack
docker stack services teststack --format '{{.Name}}'
docker service ps teststack_idp --filter='desired-state=running' --format '{{.CurrentState}}'
docker service update --image redis:7.4.1 teststack_idp --args='' --health-cmd=true
teststack_idp

Note: docker service update has --detach=false by default, so you do not need to specify it here. Second note: --args is the docker command. Don't know why they use a different wording here.