dswarm

dswarm manages Docker Swarm stacks from a gitops repo: each directory under stacks/ is a stack, secrets live in sops+age-encrypted files in git, and at deploy time dswarm injects them as versioned, immutable, native docker secrets — never .env files, never plaintext on disk. A small socat-based webhook listener (dswarm listen) closes the loop: push to the gitops repo, the changed stacks redeploy.

Requirements

Runtime target: an Ubuntu Docker Swarm manager node. Development happens on macOS; everything is bash ≥ 4 (/usr/bin/env bash).

tool	why	install
docker CLI	everything	https://docs.docker.com/engine/install/
sops ≥ 3.8	secrets decrypt/encrypt	https://github.com/getsops/sops
yq (mikefarah) ≥ v4.18	all YAML work	https://github.com/mikefarah/yq
jq	JSON (webhook payload, secrets set)	https://jqlang.github.io/jq/
openssl	HMAC, tokens	base OS
git	sync	base OS
flock	sync serialization	util-linux (base)
socat	listen only	http://www.dest-unreach.org/socat/
age-keygen	init only	https://github.com/FiloSottile/age
sha256sum or shasum	content hashes	coreutils / base

Dev only: shellcheck, bats-core (macOS: brew install shellcheck bats-core).

Install

• On the manager (to run dswarm directly): clone this repo and make install (installs to /usr/local by default; PREFIX=/opt make install). Layout: <prefix>/lib/dswarm/{bin,lib} with a <prefix>/bin/dswarm symlink.
• The listener runs as a container built from this repo (make build), deployed from your gitops repo (see Quickstart) — dogfooding.

Concepts

Gitops repo layout (see examples/gitops-repo/ for a ready skeleton):

.sops.yaml                  # age recipients + creation rules
stacks/
  <name>/                   # a stack (immediate subdirs only)
    compose.yaml            # required (compose.yaml > compose.yml >
                            #   docker-compose.yaml > docker-compose.yml)
    secrets.enc.yaml        # optional, sops-encrypted flat map
    configs/                # optional, files become docker configs

Managed vs escape hatch. A top-level secrets:/configs: entry whose value is empty (pg_password: or pg_password: {}) is dswarm-managed: dswarm sources its content (decrypted secrets file / configs/ file) and rewrites it at deploy. An entry with any content (external: true, file:, name:) is an escape hatch and left alone (relative file: paths and env_file values are absolutized against the stack dir; values containing $ are never touched).

Versioned naming. Managed objects become docker secrets/configs named <stack>_<key>_<sha8> where sha8 is the first 8 hex of the content's sha256. Rotation = new name = rolling update; after a successful deploy the superseded versions are garbage-collected (same-name = same-content, so unchanged values are reused, never recreated).

Ownership. Everything dswarm creates at deploy time carries the label dswarm.stack=<stack>. GC and dswarm rm operate strictly on that label (never name prefixes — web vs web_api would collide). One deliberate exception: the three bootstrap secrets created by dswarm init (dswarm_webhook_secret, dswarm_deploy_key, dswarm_age_key) are created unlabeled — operator-owned. GC and rm never touch them; the dogfooded dswarm stack merely mounts them as external secrets.

Secrets policy. secrets.enc.yaml must decrypt to a single YAML document that is a flat map of single-line string values. Author rule: quote every value in the source file (db_password: "hunter2"). Unquoted scalars that YAML would mangle (1.10, 0x1F, yes) are rejected at deploy time with the key named. Note sops normalizes quoting on decrypt — validation checks the decoded type, which is what matters.

Quickstart

1. Install dswarm on the manager (make install) and clone this source repo somewhere (for the image build and the examples).
2. dswarm init — one-time bootstrap: docker swarm init if needed, generates an age keypair, creates the dswarm_webhook_secret, dswarm_deploy_key, and dswarm_age_key docker secrets, and prints your age public key + deploy public key.
3. Create your gitops repo from the skeleton: cp -R examples/gitops-repo gitops && cd gitops && git init
   • .sops.yaml: replace age1...REPLACE... with your key from step 2.
   • stacks/dswarm/compose.yaml: set DSWARM_GITOPS_REMOTE to your repo.
   • Remove the demo pieces before real use: the DEMO path_regex rule in .sops.yaml, stacks/example-postgres/, and stacks/example-postgres/EXAMPLE-ONLY-age-key.txt. Commit and push.
4. Forgejo: add the deploy public key (printed by init) under repository → Settings → Deploy Keys (read access is enough).
5. Image: make build && make push (registry prefix var REGISTRY=).
6. Bootstrap the listener once, manually: dswarm deploy dswarm --repo <gitops checkout>
7. Webhook (below). From now on, pushes redeploy automatically.

Commands

dswarm [--repo DIR] [--state-dir DIR] <command>

• dswarm deploy <stack> / dswarm deploy --all — render, inject secrets/configs, docker stack deploy --prune, GC superseded versions. --all continues past per-stack failures and exits non-zero if any failed.
• dswarm rm <stack> — docker stack rm, wait for teardown, remove all label-owned secrets/configs. Idempotent; deleting a directory from git does NOT tear a stack down — only rm does.
• dswarm ls — table of stacks: in repo vs deployed.
• dswarm ps <stack> [docker args...] — docker stack ps passthrough.
• dswarm logs <stack> [-s service] [-f] [--tail N] — service logs; without -s, all services sequentially under ==> <svc> <== headers.
• dswarm diff <stack> — preview: rendered compose path, which secret/config versions are new vs reused, what would be GC'd, then docker stack deploy --dry-run (docker CLI ≥ 23.0). Changes nothing.
• dswarm secrets ls <stack> — list secret keys (never values).
• dswarm secrets edit <stack> — open secrets.enc.yaml in $EDITOR via sops (creates it encrypted first if missing); policy-validated after save.
• dswarm secrets set <stack> <key> [value] — non-interactive upsert. Value from argv (shell-history warning), or stdin (read -rs prompt on a TTY, verbatim from a pipe — the safe path). Use -- for dash-leading values.
• dswarm sync [--repo CHECKOUT] [--branch main] [--remote URL] [--since SHA] — fetch/reset the checkout and deploy the stacks changed in the diff range. Clones on first run. Deleted stack dirs are skipped, never torn down.
• dswarm listen [--addr 0.0.0.0] [--port 9000] [--repo CHECKOUT] [--branch main] [--remote URL] — webhook server. In the container, DSWARM_REPO must equal the checkout volume's mount target (the skeleton pairs them and comments it as load-bearing); without it the checkout defaults to . and sync clones into the container filesystem instead of the volume.
• dswarm init — manager bootstrap (above).
• dswarm version — version string.

Config resolution is flag > env > default: --repo/DSWARM_REPO/.; --state-dir/DSWARM_STATE_DIR//var/lib/dswarm (root) or ~/.local/share/dswarm.

The secrets workflow

dswarm secrets edit mystack          # $EDITOR on the sops file (created if missing)
dswarm secrets set mystack api_token < token.txt   # non-interactive, via stdin
dswarm secrets ls mystack            # keys only
dswarm deploy mystack                # injects + deploys

Rotate a value: dswarm secrets set mystack api_token < newtoken.txt then dswarm deploy mystack (or push and let the webhook do it). dswarm creates mystack_api_token_<newsha8>, updates the service to it, and GCs the old version after the deploy succeeds.

Forgejo webhook setup

Repo → Settings → Webhooks → Add Webhook → Forgejo:

• Target URL: http://<manager-host>:9000/hook
• HTTP Method: POST; POST Content Type: application/json
• Secret: the webhook token. docker secret inspect never returns secret bytes — reveal it through the listener container instead: docker exec $(docker ps -qf name=dswarm_dswarm) cat /run/secrets/dswarm_webhook_secret (or from wherever you recorded it at init — dswarm never prints it). The listener strips trailing CR/LF from the secret, so the value you paste into Forgejo must match the stripped value.
• Trigger: push events. The listener additionally filters ref == refs/heads/<branch> (branch mismatches get a friendly 202 "ignored"), so a single listener on main ignores other branches.

GET /healthz returns 200 for monitoring.

Bootstrap order (recap)

dswarm init → gitops repo from skeleton (+ age pubkey, push) → deploy key in Forgejo → make build && make push → dswarm deploy dswarm → create the webhook. The listener stack then manages itself: changes to stacks/dswarm/compose.yaml redeploy via the webhook.

Operations

• Rotate a secret: secrets set + deploy (above).
• Remove a stack: dswarm rm <stack> — tears down the stack AND its label-owned objects. Deleting the directory from git alone never does.
• Manual sync: dswarm sync --repo /var/lib/dswarm/gitops --branch main (what the listener runs; manual runs serialize with it on <state-dir>/sync.lock).
• Listener logs: docker service logs dswarm_dswarm (or dswarm logs dswarm -s dswarm -f).

Known limitations

• Bootstrap secret cleanup is manual: the three init-created secrets are unlabeled by design; decommission with docker secret rm dswarm_webhook_secret dswarm_deploy_key dswarm_age_key. The flip side of label ownership: if you label a hand-made external secret with dswarm.stack=<stack> yourself, that stack's GC (and dswarm rm) WILL delete it once it leaves the desired set — don't label things you don't mean dswarm to own.
• GC-vs-rollback window: superseded secret/config versions are GC'd after a successful deploy; rolling back to the previous compose after that window recreates the old-named objects fresh (values come from the repo, so this is safe, but the old versions no longer exist).
• HMAC compare is not constant-time (webhook signature check).
• The checkout is disposable: sync does git reset --hard — never edit inside the managed checkout; it also destroys untracked files there.
• Lock scope: syncs serialize on <state-dir>/sync.lock — a manual sync and the listener only serialize when they share the same state dir.
• Socket mount = root-equivalent: the listener container mounts the docker socket and runs as root; treat it as you would root on the host.
• Single-doc, flat, single-line secrets only — by design (multi-line values belong in configs/).
• dswarm deploy vs sync concurrency: both take the same <state-dir>/sync.lock, so they queue rather than overlap.
• dswarm diff in the image: the container's docker.io CLI is 20.10, so in-image diff falls back to no-dry-run (--dry-run needs docker CLI ≥ 23.0). A host-installed dswarm with a modern CLI is unaffected.

Development

• make test (or directly): shellcheck bin/dswarm lib/*.sh and bats tests/.
• Tests never touch a real daemon or real sops: tests/helpers/stubs/{docker, sops,age-keygen,ssh-keygen} go first on PATH, log every invocation to $STUB_LOG (pipe-delimited argv records), serve canned state from $STUB_STATE, and record secret create stdin as the secret's bytes. Failure injection is file-driven (fail_deploy, fail_secret_rm, stuck_stacks, sops_fail, ...). Sync tests use real temp git repos; one guarded test starts a real socat listener on 127.0.0.1.
• Layout: bin/dswarm (dispatcher) + lib/{core,stack,render,secrets,docker, deploy,manage,sync,listen,init}.sh; functions are namespaced dswarm::<module>::<name>; library functions return 1 rather than exiting; docker wrappers pass data through DSWARM_DOCKER_OUTPUT/_ERROR, never stdout.