unkin-agent
04ec9e6d41
Report the accumulated warnings on stderr when the estate load fails, and fold the dropped-record tally into the error. Count records rather than warning messages in --explain, by carrying the count on the Warn hook.
276 lines
13 KiB
Markdown
276 lines
13 KiB
Markdown
# puppet-catalog-tools
|
|
|
|
CLI tooling for the Puppet estate:
|
|
|
|
- **`catalog-diff`** — compiles a host's catalog in a base environment and in
|
|
the environment built from the branch under test, then diffs the two.
|
|
- **`catalog-impact`** — works out which hosts a control repo change can
|
|
affect, and prints their hostnames.
|
|
|
|
Catalogs are compiled over the puppetserver API (`POST /puppet/v4/catalog`)
|
|
with facts read from PuppetDB. Nothing is compiled locally and nothing is
|
|
persisted on the server.
|
|
|
|
## Install
|
|
|
|
```bash
|
|
dnf install puppet-catalog-tools # from the artifactapi rpm-internal repo
|
|
# or
|
|
make build # -> dist/bin/
|
|
```
|
|
|
|
# catalog-diff
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
catalog-diff web01.example.net --branch benvin/motd-change
|
|
catalog-diff web01.example.net --environment benvin_motd_change --json
|
|
catalog-diff web01.example.net --branch benvin/motd-change --testonly
|
|
```
|
|
|
|
| Flag | Description |
|
|
|---|---|
|
|
| `-b, --branch` | branch under test (required unless `--environment` is given) |
|
|
| `--environment` | explicit environment name, bypasses branch name conversion |
|
|
| `--base` | base environment to diff against (default `develop`) |
|
|
| `--testonly` | compile the test environment only; result via exit code |
|
|
| `--json` | machine-readable output |
|
|
| `--server` | puppetserver base URL (default `https://puppet.k8s.syd1.au.unkin.net:8140`) |
|
|
| `--puppetdb` | PuppetDB base URL (default `https://pdbmux.k8s.syd1.au.unkin.net`) |
|
|
| `--cert` / `--key` / `--cacert` | client mTLS material |
|
|
| `--timeout` | per-compile timeout (default 10m; must be positive) |
|
|
| `--retry` | retries for transient compile or transport failures (default 2, range 0-20; a failed compiler-side PuppetDB query counts, a manifest error does not) |
|
|
| `--include-classes` | diff `Class` resources (ignored by default; they manage nothing) |
|
|
| `--ignore` | ignore resources matching `Type[Title]` (glob), repeatable |
|
|
| `--ignore-attr` | ignore a parameter on every resource, repeatable |
|
|
| `--display-datatype-changes` | report changes where only the datatype differs (`28` vs `"28"`) |
|
|
| `--display-detail-add` | expand the parameters of added resources |
|
|
| `--display-source-file-line` | print the manifest file and line each change came from |
|
|
|
|
Both compilations run concurrently, so a diff costs roughly one compile.
|
|
|
|
### Exit codes
|
|
|
|
| Code | Meaning |
|
|
|---|---|
|
|
| 0 | catalogs compiled, no differences |
|
|
| 1 | catalog compilation failed in one or both environments |
|
|
| 2 | catalogs compiled, differences present |
|
|
| 3 | transport, auth, environment-mismatch or usage error |
|
|
|
|
`--testonly` prints nothing on success and returns 0, 1 or 3 — never 2.
|
|
|
|
## Configuration
|
|
|
|
Precedence (lowest to highest): defaults < config file < env vars < flags.
|
|
|
|
Config file: `$XDG_CONFIG_HOME/puppet-catalog-tools/config.yaml`
|
|
(`~/.config/puppet-catalog-tools/config.yaml`). Write a default with
|
|
`catalog-diff config init`; print the active values with `catalog-diff config show`.
|
|
|
|
```yaml
|
|
server_url: https://puppet.k8s.syd1.au.unkin.net:8140
|
|
puppetdb_url: https://pdbmux.k8s.syd1.au.unkin.net
|
|
cert: /etc/puppetlabs/puppet/ssl/certs/catalog-diff.pem
|
|
key: /etc/puppetlabs/puppet/ssl/private_keys/catalog-diff.pem
|
|
cacert: /etc/puppetlabs/puppet/ssl/certs/ca.pem
|
|
base_environment: develop
|
|
```
|
|
|
|
| Env var | Config key |
|
|
|---|---|
|
|
| `CATALOG_DIFF_SERVER` | `server_url` |
|
|
| `CATALOG_DIFF_PUPPETDB` | `puppetdb_url` |
|
|
| `CATALOG_DIFF_CERT` | `cert` |
|
|
| `CATALOG_DIFF_KEY` | `key` |
|
|
| `CATALOG_DIFF_CACERT` | `cacert` |
|
|
| `CATALOG_DIFF_BASE_ENVIRONMENT` | `base_environment` |
|
|
|
|
`catalog-diff` needs a client certificate whose certname is allowed to
|
|
`POST /puppet/v4/catalog` in the puppetserver `auth.conf`. PuppetDB is queried
|
|
without credentials.
|
|
|
|
## Branch to environment name
|
|
|
|
Puppet environment names must match `\A[a-z0-9_]+\Z`, so a branch name is
|
|
converted the way g10k deploys it: every character outside `[0-9A-Za-z_]`
|
|
becomes `_` (`benvin/foo-bar` -> `benvin_foo_bar`). Case is preserved, because
|
|
g10k preserves it — a branch with capitals therefore deploys to a directory
|
|
Puppet rejects, and `catalog-diff` fails fast telling you to rename the branch
|
|
or pass `--environment`. `--environment` bypasses the conversion entirely.
|
|
|
|
## Environment override
|
|
|
|
Classification runs through an exec ENC (`encapic` -> `encapi`), and an
|
|
ENC-supplied environment outranks the requested one. Every request sets
|
|
`options.prefer_requested_environment`, and the returned catalog's own
|
|
`environment` is always compared against the one requested. A mismatch between
|
|
two named environments is a hard error (exit 3) rather than a silent diff of an
|
|
environment against itself. A response that decodes without an environment name
|
|
at all is reported as a protocol mismatch instead, because that is a client/
|
|
server problem rather than a classification one.
|
|
|
|
## What a catalog diff cannot see
|
|
|
|
- Facts come from PuppetDB's last report, not a live agent run, so changes to
|
|
custom facts are not tested.
|
|
- Provider changes and order-of-operations problems (dependency cycles) do not
|
|
appear in a catalog.
|
|
- Catalogs are fetched over the API rather than compiled from a checkout, so a
|
|
change to a static file's content — without its `source` path changing —
|
|
produces no readable text diff.
|
|
|
|
Puppet catalogs can contain unencrypted secrets. Parameters the catalog marks
|
|
as sensitive are replaced with a digest so a change is still visible without
|
|
the plaintext reaching stdout or a CI log; anything Puppet does not mark stays
|
|
in the clear.
|
|
|
|
# catalog-impact
|
|
|
|
Given a change to the `puppet-prod` control repo, `catalog-impact` works out
|
|
which files it touches and which hosts those files affect, then prints the
|
|
affected hostnames — one per line, sorted and deduplicated, ready to pipe into
|
|
a fan-out that compiles each host's catalog.
|
|
|
|
```bash
|
|
catalog-impact -r ~/src/puppet-prod # current branch vs origin/develop
|
|
catalog-impact -r ~/src/puppet-prod -b benvin/motd --explain
|
|
catalog-impact --files hieradata/common.yaml --json
|
|
git diff --name-only origin/develop... | paste -sd, | xargs catalog-impact --files
|
|
```
|
|
|
|
| Flag | Description |
|
|
|---|---|
|
|
| `-r, --repo` | path to a puppet-prod checkout (default `.`) |
|
|
| `-b, --branch` | branch under test (default: the repo's current branch) |
|
|
| `--base` | base ref to diff against (default `origin/develop`) |
|
|
| `--files` | explicit changed-file list, bypassing git entirely |
|
|
| `--puppetdb` | PuppetDB base URL (default `https://pdbmux.k8s.syd1.au.unkin.net`) |
|
|
| `--json` | structured output with per-host reasons |
|
|
| `--explain` | human-readable: for each host, which changed files hit it and why |
|
|
| `--limit` | cap the number of hosts emitted (0 = no cap) |
|
|
| `--timeout` | PuppetDB request timeout (default 1m) |
|
|
|
|
`--puppetdb` defaults to the shared configuration described under
|
|
[Configuration](#configuration), so `puppetdb_url` and `CATALOG_DIFF_PUPPETDB`
|
|
apply to both binaries. No credentials are needed.
|
|
|
|
Facts for the whole estate are fetched from PuppetDB in a single query and every
|
|
selector is evaluated in memory, so the cost is one HTTP request regardless of
|
|
how many files changed.
|
|
|
|
## How a changed file becomes a host list
|
|
|
|
Classification runs through an exec ENC and `manifests/site.pp` is empty, so a
|
|
node's `enc_role` fact is its classification. Everything below keys off that
|
|
and the facts `hiera.yaml` interpolates.
|
|
|
|
**Hieradata paths become fact selectors**, mirroring the hierarchy:
|
|
|
|
| Path | Selects |
|
|
|---|---|
|
|
| `hieradata/nodes/<certname>.yaml` | that node |
|
|
| `hieradata/roles/<t1>[/<t2>[/<t3>]]` | role-tier **prefix** — `roles/infra/puppet.yaml` hits every `roles::infra::puppet::*` host |
|
|
| `hieradata/country/<c>/region/<r>/<t1>[/...]` | country and region and role-tier prefix |
|
|
| `hieradata/country/<c>/roles/<t1>[/...]` | country and role-tier prefix |
|
|
| `hieradata/country/<c>/region/<r>` | country and region |
|
|
| `hieradata/country/<c>` | country |
|
|
| `hieradata/virtual/<v>.yaml` | `virtual` == v |
|
|
| `hieradata/os/<name>/<name><major>[.<minor>].yaml` | `os.name` and `os.release.major` (and minor when present) |
|
|
| `hieradata/os/<name>/all_releases.yaml` | every host with that `os.name` |
|
|
| `hieradata/common.{yaml,eyaml}` | every host |
|
|
|
|
**Repo-global files select every host**: `hiera.yaml`, `environment.conf`,
|
|
`manifests/site.pp`, `Puppetfile`. The Puppetfile pins external modules whose
|
|
code is not in this repo, so there is nothing to trace — mapping it to the
|
|
whole estate is the deliberate conservative choice. `lib/` under any module is
|
|
pluginsynced to every agent and is treated the same way.
|
|
|
|
**Code paths become classes**, then the classes are traced backwards:
|
|
|
|
- `site/roles/manifests/<t1>/<t2>/<t3>.pp` is class
|
|
`roles::<t1>::<t2>::<t3>`, which maps to hosts by exact `enc_role`. A role
|
|
class is a leaf — nothing includes it, the ENC assigns it.
|
|
- `site/profiles/manifests/**.pp` and `modules/<mod>/manifests/**.pp` become
|
|
their autoloaded class names (`init.pp` counts as both the namespace and
|
|
`<namespace>::init`) and need a reverse lookup.
|
|
- `modules/<mod>/{templates,files}/**` and `site/profiles/{templates,files}/**`
|
|
are attributed to the classes whose manifests reference them through
|
|
`template()`, `epp()` or `puppet:///modules/...`. An unreferenced asset falls
|
|
back to every class its module defines.
|
|
|
|
## The reverse lookup
|
|
|
|
Finding what applies a class means searching two places, not one:
|
|
|
|
1. `.pp` files under `site/` and `modules/` for `include`, `contain`,
|
|
`inherits`, `class { '...': }`, capitalised resource references,
|
|
`create_resources()`, `$class::var` reads and defined-type usage. Comments
|
|
are stripped with quote tracking and heredoc bodies are skipped, so a `#` in
|
|
a string never truncates a line and documentation in a heredoc never becomes
|
|
an edge.
|
|
2. `hieradata/**/*.{yaml,eyaml}` for the class as a list item under
|
|
`hiera_include:`. `profiles::base` declares
|
|
`($hiera_include - $hiera_exclude).include` with a `unique` merge across the
|
|
whole hierarchy, and it is included by essentially every role.
|
|
|
|
The second is not an edge case. `hieradata/roles/apps/media/jellyfin.yaml` is
|
|
the only thing that applies the `jellyfin` module — there is no `include
|
|
jellyfin` anywhere — so a manifest-only search reports it as orphaned and finds
|
|
no hosts. A `hiera_include` hit is resolved through the hieradata path rules
|
|
above: the position of the listing file in the hierarchy decides which hosts
|
|
get the class. `hiera_exclude` entries are subtracted the same way.
|
|
|
|
The walk runs to a fixed point, because profiles include profiles several
|
|
layers deep — `profiles::base` alone pulls in around twenty-five. Cycles are
|
|
visited once.
|
|
|
|
## Bias and limits
|
|
|
|
A missed host means an unvalidated catalog, so every ambiguity widens:
|
|
|
|
- a path matching no rule is reported on stderr and mapped to every host;
|
|
- an asset no manifest references widens to its whole module;
|
|
- an unrecognised hieradata path in a `hiera_include` hit widens to every host;
|
|
- a class reference that is not a literal name is reported as unresolved and
|
|
widened. A selector yields every literal name in its arms, and a bare
|
|
variable is followed through the assignments in its own file. What is left —
|
|
`include "profile::role_${role}"`, `include lookup('x')` — is recorded
|
|
against its file, and every class the reference could have named is treated
|
|
as reaching the classes that file declares. The references appear in
|
|
`--explain`, in `unresolved_references` in `--json`, on stderr in plain mode,
|
|
and set `summary.resolution_incomplete`.
|
|
|
|
A fact record that arrives with no certname belongs to no host, and one whose
|
|
value does not decode carries nothing to fold; either way it cannot enter the
|
|
estate table, and a host whose records all fail leaves the table with them. Both
|
|
are reported in `node_warnings`, in `--explain` and on stderr, counted in
|
|
`summary.dropped_fact_records`, and set `summary.resolution_incomplete` too. If
|
|
the drops empty the table altogether the run fails, and the warnings and the
|
|
tally go to stderr with the error rather than dying with it.
|
|
|
|
Paths outside the Puppet modulepath — `.woodpecker/`, `doc/`, `spec/`,
|
|
`vendor/`, `Gemfile`, `*.md` — cannot reach a catalog and are skipped.
|
|
|
|
`--explain` also lists changed files that mapped cleanly but reached no host,
|
|
which normally means the role they belong to has no members.
|
|
|
|
Two things the tool cannot see:
|
|
|
|
- A class nothing declares and no `hiera_include` lists reaches no hosts. That
|
|
is reported as an orphan on stderr rather than silently dropped.
|
|
- The index is built from the checkout at `--repo`, not from `--branch`. In CI
|
|
the checkout is the branch under test; pointing `--branch` at a different ref
|
|
than the working tree means classes the branch adds look orphaned.
|
|
|
|
|
|
## Development
|
|
|
|
```bash
|
|
make build test lint fmt pre-commit
|
|
make rpm # build + package into dist/*.rpm via nfpm
|
|
```
|
|
|
|
Golden-file tests: `go test ./internal/catalog -update` rewrites them.
|