git tpl test¶

Run the test cases a template carries.

git tpl test

A template above a few files has conditionals, and without a suite the first person to find a broken one is the person generating a project from it. A case says given these answers, this is what comes out — and it says it in the template repository, next to the thing it describes.

test takes an optional --template PATH, defaulting to the current directory, but only ever a local checkout — never a remote source, with or without --ref. There is also no --root to override the manifest's declared render subdirectory. --template is a flag rather than a positional so a bare case name is never mistaken for a template path — see ADR-030.

A case¶

Cases are files in tests/ at the template repository root, in TOML, JSON or YAML. The file's stem is the case name.

# tests/minimal.toml
[answers]
project_name = "thing"
with_ci = false

[expect]
files = ["pyproject.toml", "src/thing/__init__.py"]
absent = [".github/workflows/ci.yml"]

[expect.contains]
"pyproject.toml" = ['name = "thing"']

[expect.lacks]
"pyproject.toml" = ["deploy"]

A path named in expect.contains or expect.lacks that the rendering never produces is a failure either way — never a pass, vacuous or otherwise. "This file does not mention deploy" is not proven by a file that never rendered.

Everything is optional. A case with only [answers] asserts that the answer set renders at all, which is a real and frequently sufficient test.

Unanswered questions take their defaults. Nothing is ever prompted for — a prompt in a test runner is a hang — so a question with no default and no answer fails the case with tpl::eval::unanswered, which is a true thing to know about the template.

An answer that names no question fails the case too, with tpl::answers::unknown_key — unconditionally, unlike render's --strict-answers, which this command has no equivalent of and never will (see Options below). A case's [answers] is hand-authored, not a recorded set that might outlive a dropped question, so an unrecognised key here is always the case's own mistake. See ADR-029.

Options¶

There is no --answer, --answers-from or --defaults. The case file is the answer set: a flag that changed the answers would change what every case asserts while every case file still said otherwise.

Failures are asserted by code, never by message¶

[expect]
error = "tpl::eval::wrong_type"

A suite that pinned error prose would make every improvement to a diagnostic a breaking change, which is how error messages stop improving. The codes are already the stable surface, so they are what a case names.

The code is matched anywhere in the failure's cause chain. tpl::render::content says a file failed to render; only the tpl::eval::expression beneath it says why, and that is usually the one worth naming. Either passes.

A case with error cannot also have files, absent, contains or lacks — there is no rendering for them to describe. Split it into two cases.

Commands¶

[commands]
before   = ["mkdir -p src", "touch src/existing.py"]
rendered = ["python -m venv .venv"]
after    = [".venv/bin/pip install -e ."]
finally  = ["rm -rf .venv"]

Four lists, each run in a fresh, empty scratch directory created just for this case and thrown away afterward:

The render is written on top of whatever before created, not into a directory cleared first. This lets a case test its output alongside files that already exist in a project, which --dirty/render alone cannot exercise. update itself builds a Git tree and never writes rendered files into the project worktree.

before sees "nothing but what it creates itself" only when the case has no git: a case that sets it finds a .git directory and one commit already there before before runs its first entry.

Each entry is a plain string, split into a program and its arguments the way a shell would parse quoting and escapes — but no shell actually runs. There is no pipe, no glob, no redirection and no $VAR expansion. A pipeline needs a script file: write it with before, then name it in rendered.

--write never runs any of the four lists. It only records what a case renders; proving what running it does is what a plain git tpl test is for. See Snapshots and ADR-032.

Environment¶

Every command inherits the full environment git tpl test itself runs under. commands.env adds to it, for every list in the case:

[commands]
env = { PDM_IGNORE_ACTIVE_VENV = "true" }
rendered = ["pdm install --plugins", "pdm install -dG:all", "pdm test"]

A single list can override commands.env for itself alone by writing it as a table with run and env instead of a bare array:

[commands]
env = { PDM_IGNORE_ACTIVE_VENV = "true" }

[commands.after]
env = { PDM_IGNORE_ACTIVE_VENV = "false" }
run = ["pdm run something-that-needs-the-active-venv"]

commands.after.env wins over commands.env for a key both set; every other list still gets commands.env alone. Neither adds $VAR expansion to a command's own text — a value only ever reaches a process as an environment variable, never as text substituted into the command line before it runs.

Every command also sees TEMPLATE_ROOT, set to the resolved template's root on disk — the working tree for --dirty, and the same working tree for a local --ref, since test never resolves a remote (see ADR-030). It is distinct from both cwd (each list's own throwaway sandbox) and from a template's declared render subdirectory: a case's script does not live in the sandbox, so a command that needs more than a line or two writes it once, commits it in the template repository, and names it directly — "{{ env.TEMPLATE_ROOT }}/tests/scripts/check.sh" — instead of synthesizing it inline via before on every run. A case's own env/commands.env may still override it for a key it deliberately sets, the same as any other default.

before, rendered and after each stop at their own first failure — they are sequential, and a later entry usually assumes an earlier one worked. A failing before, or a render that fails without expect.error naming it, skips straight to finally: there is nothing for rendered/after to run against. finally is the opposite: every entry in it runs regardless of anything failing before or within it, because it is cleanup.

There is no timeout. A hanging command hangs the run.

Every command also gets CLICOLOR_FORCE=1 and FORCE_COLOR=1 added to its environment, whenever git tpl test's own output is colourised (never under --color=never, NO_COLOR, or a non-terminal stderr). A command talking to a pipe rather than a terminal — which is what capturing its output requires — otherwise assumes nobody can see colour and silently prints in black and white, which is backwards when that output is about to be shown on a real terminal either way. commands.env (or a list's own env) still wins for either variable, so a case that wants plain output from a tool that only checks these two can still ask for it.

A CI job wants colour for the same reason, but has to ask for it explicitly: a runner's shell is not a terminal, so --color auto (the default) never colourises there on its own — the same reason a workflow already sets CARGO_TERM_COLOR=always to get colour out of cargo itself. Set FORCE_COLOR=1 or CLICOLOR_FORCE=1 (or pass --color always) in the job's own environment — either variable works, they are treated as aliases for the same signal:

env:
  FORCE_COLOR: "1"

One variable does double duty: it colourises git tpl test's own progress output below, and is what causes CLICOLOR_FORCE/FORCE_COLOR to be forwarded to [commands] children, as described above — so a case's own cargo build no longer needs CARGO_TERM_COLOR set separately for that part either. See the environment for the full colour precedence.

Templating¶

A command is rendered as a MiniJinja template — the same engine that renders every other string in the template — before it is word-split, against two names:

• env.* — exactly what that command is about to be spawned with: the inherited environment, TEMPLATE_ROOT, CLICOLOR_FORCE/FORCE_COLOR when set, and the list's own env/commands.env, all merged in the order described above. {{ env.TEMPLATE_ROOT }} always names the same path $TEMPLATE_ROOT would inside the process.
• answers.* — the case's own [answers] table, exactly as written, with no defaults or computed values applied. Available to before as well as rendered/after/finally, since it comes from the case file itself rather than from a render.

[answers]
name = "widget"

[commands]
rendered = ["{{ env.TEMPLATE_ROOT }}/tests/scripts/check.sh", "{{ answers.name }}"]

A command with no {{/{% in it is never templated, so every case written before this existed keeps behaving identically. One that references a name that does not exist — {{ answers.projct_name }}, a typo — fails that command alone, reported the same way a missing program already is, rather than silently spawning it with an empty string in place of the mistake.

This is not $VAR shell expansion: a literal $FOO in a command is still passed through unchanged. Only {{ }}/{% %} mean anything, because MiniJinja renders the string — no shell runs, and nothing about the command format above changes.

Progress¶

A case in progress shows on stderr: a spinner naming the current case and phase — rendering, or a running command — on a real terminal, or the same events as plain scrolling lines when stderr is not one (a CI log, say). Neither shows under --quiet or --json.

-v/--verbose replaces the spinner with a full scrolling log: every phase, and every command's own stdout/stderr forwarded live, exactly as it writes it — ANSI included. The final report no longer repeats a failed command's captured output in that case, since it was already shown as it happened; --json is unaffected either way, and always carries it.

Under --write, no [commands] run at all (see Snapshots), so there is nothing for -v to stream live. In its place, -v shows the same coloured unified diff a normal run's snapshot-mismatch failure already shows, once per case whose snapshot changed — and the final summary names which cases were written, updated or left unchanged. See ADR-032.

Under GitHub Actions¶

When GITHUB_ACTIONS=true is set — every job on GitHub Actions sets it, so this needs no flag and has no opt-out — git tpl test reports progress as workflow commands instead of any of the above: each case is wrapped in a ::group::/::endgroup:: pair, so a scan of the log shows one line per case with everything else folded away, expandable on demand. While a case's group is open, its commands' own stdout/stderr are forwarded live, exactly as -v would, whether or not -v was passed — a folded group has nothing left to trade that away for. A failing case gets exactly one ::error:: annotation, naming the case's own file, so it shows up on a pull request's "Files changed" view without opening the log at all. Everything that would otherwise be internal phase chatter — "rendering", "checking snapshot" — goes to ::debug:: instead, visible only once a job has step debug logging enabled.

--json is unaffected: it is checked first, and always wins. See ADR-035.

Running git tpl test is the consent¶

A case's [commands] need no confirmation: running git tpl test on a template you have in front of you is already the same act as cloning a repository and running make test in it. Commands run by default. To skip them for one invocation, pass --skip-commands; to disable them for yourself by default, set tpl.testCommands to false (git config tpl.testCommands false) — --skip-commands can only disable further, never re-enable what configuration turned off. See ADR-027 for the full reasoning, including why this does not reopen the rule that a rendered project — render, init, update — cannot execute anything.

Both also skip a case's git: disabling [commands] disables everything the sandbox exists to serve, not only the four lists themselves — there is nothing left for an isolated repository to prove if nothing runs inside it.

A case's trust rests on the same act of consent, extended to a template's declared remote data sources — see ADR-028.

Git sandbox¶

git = true

[commands]
rendered = ["pdm install", "pdm build"]

Some tooling needs Git for its own identity — pdm-backend's source = "scm", and every setuptools_scm-alike version source, read git describe before installing a single dependency. git = true makes the sandbox commands runs in an isolated Git repository rather than a plain scratch directory: one empty commit, already there before before runs its first entry, so there is something for that tooling to find.

The identity is a built-in default unless overridden:

[git]
user.name = "Someone"
user.email = "someone@example.com"

git.user.name/git.user.email are the only keys a case may set; everything else about the seed commit — commit.gpgsign off, the branch it lands on — is fixed. git = false, or no git key at all, is today's plain scratch directory, byte for byte: this is opt-in, the same choice snapshot already made for the same reason.

Isolation is by construction, not by fighting the running machine's configuration file by file. The seed commit is written through git-tpl's own Git backend, never a spawned git process, so no hook, alias or ambient identity is ever consulted for it — the chosen identity lands in the sandbox's own local .git/config before anything asks for a signature, and Git's own precedence (local always outranks global and system) makes it authoritative regardless of what either ambient level says. And because a case's own [commands] may still invoke a literal git — the whole point of seeding a repository at all — every command in a case that set git is spawned with GIT_CONFIG_NOSYSTEM=1 and GIT_CONFIG_GLOBAL pointed at a path that is never created, so a literal git sees only the sandbox's own local config: no credential helper, no alias, no commit.gpgsign the machine running the suite happens to have. commands.env/a list's own env may still override either variable for a case that deliberately wants to.

See ADR-033 for the full reasoning, and issue #155 for where the gap this closes came from.

Trust¶

# tests/remote.toml
trust = false

[expect]
error = "tpl::data::untrusted"

trust decides whether this case's render may reach the template's declared remote data sources — a [data.*] source that is remote or git. It defaults to true: a case renders for real unless it says otherwise, because the point of a suite is to prove what the template's output actually looks like, and an untested remote source is a gap in the suite, not a safety margin.

Set trust = false to assert the refused path instead — the render fails with tpl::data::untrusted before anything reaches the network, deterministically and with no host to reach.

This is not the --trust/[trust] mechanism render, init, update and backport use. Those act on a real project for a real person, so they refuse by default and ask, once, for consent; test never asks anybody, and the persistent [trust] list in ~/.config/git-tpl/config.toml is not consulted here either — a case's trust must mean the same thing on every machine that runs the suite, including CI, where there is nobody to ask and nothing pre-trusted. See ADR-028 for the reasoning.

Snapshots¶

# tests/minimal.toml
snapshot = true

git tpl test --write

records each case's rendering under tests/__snapshots__/<case>/, and every later run compares against it.

A case is written and compared only when it says snapshot = true. This is explicit for a reason: without it, --write would silently start recording (and every future run silently start comparing) a case that never asked for one. A case that says snapshot = true but has never been recorded fails outright — "record one with git tpl test --write" — rather than passing having asserted nothing.

tests/
├── minimal.toml
└── __snapshots__/
    └── minimal/
        ├── MANIFEST
        └── files/
            ├── pyproject.toml
            └── src/thing/__init__.py

The rendered files are stored verbatim, at their rendered paths. That is the whole point: a change to the template shows up as a git diff of the generated project, in the generated project's own language. Reviewing it needs no tooling and no Rust.

MANIFEST records what the files themselves cannot — the executable bit, a digest and a size, one sorted line each. A Windows checkout loses the mode on disk, and a snapshot that silently stopped asserting on chmod +x would be worse than none.

Things worth knowing:

• A case with snapshot unset (or false) is not a failure on a plain run. Snapshots are opt-in per case, and are neither written nor compared for it.
• snapshot = true with nothing recorded yet is a failure, not a pass. Run --write once to record it.
• --write clears the case's snapshot directory rather than merging into it. A template that stops producing a file has to be seen to stop.
• --write does not stage or commit anything. You review the diff and commit it.
• --write only records; it does not run a case. A case's [commands] never run, and expect is never checked, under --write — recording a broken rendering is not the same as passing it. Run a plain git tpl test afterward (or let CI do it) to prove the recording still holds. See ADR-032.
• A case with no snapshot = true is skipped entirely under --write — not rendered, not checked, not counted as run. --write's only reason to look at a case is to write its snapshot.

--write writes into the checked-out template's working tree — there is nowhere else for a snapshot to go.

Sharding¶

git tpl test --record-durations   # once, to seed tests/__timings__/durations
git tpl test --shard 1/4          # then, in each of four CI matrix jobs
git tpl test --shard 2/4
git tpl test --shard 3/4
git tpl test --shard 4/4

--shard INDEX/TOTAL (both starting at 1) runs only the cases assigned to this shard, out of whatever this invocation would otherwise run — applied after [CASE] filtering, not instead of it. Without a recorded durations file, the split is a deterministic slice by count: this never fails just because nothing has been recorded yet, so a template's first --shard run works exactly as well as its hundredth.

--record-durations runs the suite exactly as a plain invocation would — nothing about [commands], expect or snapshots changes — while timing each case, then writes tests/__timings__/durations, merged over whatever was already there. Commit that file to the template repository; a later --shard reads it back and bin-packs the split so every shard finishes in about the same time, rather than merely getting the same number of cases. A case missing from the file (new, or never measured) is weighted at the mean of the ones that are.

--record-durations conflicts with --shard — recording from a partial run would corrupt the file with timings for only some of the suite — and with --write, which does not run a case at all and so has nothing real to time.

A GitHub Actions matrix supplies both halves of --shard directly, with only the shard index varying between jobs:

strategy:
  matrix:
    shard: [1, 2, 3, 4]
steps:
  - run: git tpl test --shard ${{ matrix.shard }}/${{ strategy.job-total }}

This is unrelated to the "no matrix language" rule below: that rule is about a case file expanding into many cases combinatorially, which this project still refuses. Splitting the cases that already exist across several CI jobs is a different question, and --shard/--record-durations is this project's answer to it. See ADR-036.

Cases come from the revision, not from disk¶

git tpl test --ref v1.2.0 runs that tag's cases against that tag's template, and compares against that tag's snapshots. Without --ref, test reads the working tree instead: the point of testing is catching a broken conditional before it is committed, not after. Reading cases off the filesystem is not a special case, though it may look like one: the resolver builds a synthetic tree of the working directory the same way it always has for a --dirty render on any other command, so an edited-but-uncommitted case is picked up by the same code path, with the same .gitignore handling the rendering got. See ADR-030.

A snapshot is not subject to the project's .gitignore either, deliberately: it is data --write recorded, not a project file a render produced. An ordinary rule matching a snapshot's own filename — a bare MANIFEST, say, the Python setup.py sdist convention — does not stop it from being read back.

In CI¶

git tpl test

Exit 0 if every case passed, 1 if any did not, and 1 with a diagnostic code if the run could not start.

Under --json, note that ok is true whenever the command ran. Whether the suite passed is summary.failed:

$ git tpl --json test | jq '{ok, failed: .summary.failed}'
{"ok": true, "failed": 2}

The two are separate because a caller has to be able to tell "two cases failed" from "the template could not be resolved", and both are non-zero exits.

--json also carries summary.durationsRecorded (whether this run wrote the durations file), a top-level shard object (null unless --shard was given: {index, total, casesTotal, balanced}), and a durationMs on every case — measured on every run, whether or not --shard/--record-durations was used. See Sharding and ADR-036.

What a case cannot do¶

[commands] runs against a scratch sandbox the test harness creates and destroys for one case — never against a render, init or update, none of which can execute anything, still, unconditionally. See Commands and ADR-027 for exactly what a case's commands can and cannot see.

For everything outside that sandbox — checking a real, project-shaped rendering with the tools that understand it, actionlint, a full npm ci, a linter with its own config discovery — your own CI still does it better:

git tpl render . --dirty -o /tmp/out --answers-from tests/minimal.toml
cd /tmp/out && cargo build && actionlint .github/workflows/*.yaml

There is also no matrix language. Three files beat a combinatorial block whose expansion nobody can predict; if a template needs twelve cases, twelve files say so honestly.

A note on partials¶

A rendered file named something.jinja lands in the snapshot as tests/__snapshots__/<case>/files/something.jinja, and the renderer collects every .jinja outside the render root as an importable partial. It is keyed by its full path, so it shadows nothing and changes no rendering.