RFC: Automated TLS Certificate Renewal via Composer

Date: 2026-07-05

On 2026-07-05 the Let's Encrypt certificate for gateway.architect.exchange expired at 01:14 UTC, taking down all HTTPS at the ax-prod edge for ~4 hours (alerts: md-pub/api-gateway/order-gateway /health probe failures, all Urgent). The backend services were healthy the whole time — the outage was entirely TLS at nginx. Recovery required a human to stop nginx and run certbot by hand.

Two root causes:

1. No renewal automation on either box. Certbot exists in compose.yml only as a manual docker compose run certbot service (co.architect.composer.run=manual). Nothing schedules certbot renew; prod's cert simply ran out its 90 days. Demo's last renewal (2026-06-29) appears to have been manual too.
2. ax-prod's renewal config uses the standalone authenticator. Certbot records the authenticator used at first issuance in /etc/letsencrypt/renewal/<name>.conf and replays it on every renew. Standalone binds its own :80 listener, which collides with nginx (both network_mode: host) — so prod renewal requires stopping nginx. ax-demo's certs were issued with --webroot, which renews fine with nginx up. This asymmetry is invisible in the config repos; it lives in host state.

The infrastructure for webroot renewal already exists in both envs: nginx and certbot share /home/ec2-user/certbot (mounted at /mnt/certbot), and nginx's :80 server block serves location /.well-known/acme-challenge/ from it.

Certificates renew unattended on both envs, nginx picks up renewed certs without dropping connections, and a renewal failure pages us with ~30 days of runway rather than at expiry.

Composer is already the scheduler on these boxes ("cron for Docker Compose"), so renewal becomes two labeled job services in compose.yml:

certbot renew twice daily is Let's Encrypt's recommended cadence; with the 30-day renewal window, the first Slack error fires ~60 runs before expiry.

nginx only reads cert files at startup and on reload. SIGHUP to the master process (nginx -s reload equivalently) is its built-in graceful reload: new workers start with the new cert; old workers stop accepting connections but finish serving existing ones — including long-lived websockets — before exiting. Established TLS sessions continue uninterrupted; new handshakes get the new cert. A HUP when nothing changed is a harmless no-op, which is why an unconditional periodic reload (≤12h after renewal, against a 30-day window) is preferred over event-driven plumbing.

Note co.architect.composer.restart is not suitable here — it does a container stop/start, which drops connections.

• Host crontab — works, but drifts from the config repo and doesn't travel with the project; avoiding exactly this is why composer exists.
• certbot --deploy-hook — the natural event-driven trigger, but the hook runs inside the certbot container, which has no docker socket or CLI, so it can't signal nginx. Bridging via a marker file on the shared volume plus a watch loop wrapped around nginx's entrypoint works but adds moving parts for no real gain given the 30-day window.
• Scheduled nginx restart — drops connections; rejected.

The nginx_reload job works but is inelegant: a full docker:cli image with the docker socket mounted, just to send one signal. Before rolling this pattern out, it's worth deciding whether composer should grow first-class support:

1. Native signal jobs — e.g. co.architect.composer.signal=HUP + cron on the target service, so no sidecar needs socket access. Composer already holds the socket; this keeps the privilege confined to it.
2. Job sequencing — the 30-minute offset between certbot_renew and nginx_reload is a convention, not a dependency. Should composer support "run B after A completes/succeeds" so the ordering is explicit?
3. Partial-failure semantics — certbot renew exits non-zero if any renewal conf fails, so one stale conf (e.g. demo's dead api-gateway.sandbox.architect.exchange, NXDOMAIN) makes every run "fail" in Slack. Do we rely on operators pruning confs (certbot delete), or should the job wrap certbot to distinguish "some certs failed" from "the cert we care about failed"?
4. Socket exposure — if we keep the docker:cli approach, the reload container has full docker API access on a box running the exchange. A composer-native signal feature (Q1) eliminates the widest-privilege container in the stack.

1. Land composer changes if we take Q1/Q2 (otherwise skip).
2. ax-demo: prune stale renewal confs, add the two jobs, restart composer, verify the 03:17 UTC run and a forced renewal end-to-end.
3. ax-prod: webroot conversion (Design §1), then same as demo.
4. Drop the deprecated http2 param from prod's listen 80 default_server http2; while touching nginx config — plaintext HTTP/2 on :80 is deprecated since nginx 1.25.1 and demo already omits it (ACME validation works on 1.30.2 regardless, via h2c auto-detection).