add the catalog-impact binary
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful

Work out which hosts a puppet-prod change can affect, so a catalog test can
fan out over exactly those hosts instead of the whole estate.

- add cmd/catalog-impact and internal/impact
- map hieradata paths onto fact selectors from the hiera hierarchy
- trace manifest classes back to role classes through include, contain,
  class {} and hiera_include, to a fixed point
- fetch the estate fact table from pdbmux in one query via FactsByName
- widen to all hosts, loudly, for any path that matches no rule
- ship the binary and its completions in the RPM
This commit is contained in:
2026-09-12 16:15:00 +10:00
parent dcbd810c34
commit 8093933500
44 changed files with 2492 additions and 5 deletions
+22
View File
@@ -9,14 +9,20 @@ shared code lives in `internal/`.
- **`catalog-diff`** — compiles a host's catalog twice (base environment and
the environment built from the branch under test) over the puppetserver API
and diffs the result.
- **`catalog-impact`** — maps a control repo change onto the hosts it can
affect, by reading hieradata paths as hierarchy positions and tracing
manifest classes backwards to role classes.
## Structure
```
cmd/catalog-diff/ # the CLI (flags, exit codes, orchestration)
cmd/catalog-impact/ # the CLI (flags, output modes)
internal/catalog/ # catalog types, normalisation, diff engine, renderers,
# branch.go (BranchToEnvironment)
internal/puppetserver/ # mTLS client for POST /puppet/v4/catalog + /status/v1/simple
internal/impact/ # path -> selector rules, repo index, reverse lookup,
# estate fact table, renderers
internal/pdbmux/ # unauthenticated fact client for pdbmux
internal/config/ # XDG config file + env var loading
packaging/nfpm.yaml # RPM spec (envsubst-templated)
@@ -36,6 +42,9 @@ Makefile and to `scripts/build-rpm.sh`, and add its files to
certname must be allowed in the puppetserver `auth.conf`.
- Facts: `GET /pdb/query/v4/nodes/<certname>/facts` on pdbmux, falling back to
`/pdb/query/v4/factsets/<certname>`. No auth, no client cert.
- Estate-wide facts: one AST query against `/pdb/query/v4/facts`
(`pdbmux.Client.FactsByName`). The estate is ~150 nodes, so `catalog-impact`
pulls the whole table once and filters in Go rather than querying per host.
- The two services have different auth models and stay separate clients.
## Gotchas
@@ -52,6 +61,19 @@ Makefile and to `scripts/build-rpm.sh`, and add its files to
File titles keyed by `path`, datatype-only changes suppressed). Removing one
floods the output with noise.
## catalog-impact gotchas
- The reverse lookup must search hieradata as well as manifests. `profiles::base`
declares `($hiera_include - $hiera_exclude).include`, and some modules — the
`jellyfin` one, for instance — are applied *only* by a `hiera_include` list in
a role yaml. A manifest-only search reports them as orphans.
- `hieradata/roles/<t1>/<t2>.yaml` is a **prefix** match on the role tiers, not
an exact one. `site/roles/manifests/.../<t3>.pp` is exact.
- The bias is to over-include: an unmapped path widens to the whole estate and
says so on stderr. Never turn a widening into a silent empty result.
- The repo index comes from the `--repo` checkout, not from `--branch`. CI runs
on the branch checkout, so the two agree there.
## Testing
```bash
+1 -1
View File
@@ -1,6 +1,6 @@
# All shipped binaries. Every tool is a separate main package under cmd/, so
# adding one is a matter of adding its name here.
BINARIES := catalog-diff
BINARIES := catalog-diff catalog-impact
DIST := dist
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
GOFLAGS := -ldflags="-s -w -X main.version=$(VERSION)"
+122 -2
View File
@@ -1,9 +1,11 @@
# puppet-catalog-tools
CLI tooling for the Puppet estate. Currently one binary:
CLI tooling for the Puppet estate:
- **`catalog-diff`** — compiles a host's catalog in a base environment and in
the environment built from the branch under test, then diffs the two.
- **`catalog-impact`** — works out which hosts a control repo change can
affect, and prints their hostnames.
Catalogs are compiled over the puppetserver API (`POST /puppet/v4/catalog`)
with facts read from `pdbmux`. Nothing is compiled locally and nothing is
@@ -14,9 +16,11 @@ persisted on the server.
```bash
dnf install puppet-catalog-tools # from the artifactapi rpm-internal repo
# or
make build # -> dist/catalog-diff
make build # -> dist/
```
# catalog-diff
## Usage
```bash
@@ -119,6 +123,122 @@ as sensitive are replaced with a digest so a change is still visible without
the plaintext reaching stdout or a CI log; anything Puppet does not mark stays
in the clear.
# catalog-impact
Given a change to the `puppet-prod` control repo, `catalog-impact` works out
which files it touches and which hosts those files affect, then prints the
affected hostnames — one per line, sorted and deduplicated, ready to pipe into
a fan-out that compiles each host's catalog.
```bash
catalog-impact -r ~/src/puppet-prod # current branch vs origin/develop
catalog-impact -r ~/src/puppet-prod -b benvin/motd --explain
catalog-impact --files hieradata/common.yaml --json
git diff --name-only origin/develop... | paste -sd, | xargs catalog-impact --files
```
| Flag | Description |
|---|---|
| `-r, --repo` | path to a puppet-prod checkout (default `.`) |
| `-b, --branch` | branch under test (default: the repo's current branch) |
| `--base` | base ref to diff against (default `origin/develop`) |
| `--files` | explicit changed-file list, bypassing git entirely |
| `--pdb` | pdbmux base URL (default `https://pdbmux.k8s.syd1.au.unkin.net`) |
| `--json` | structured output with per-host reasons |
| `--explain` | human-readable: for each host, which changed files hit it and why |
| `--limit` | cap the number of hosts emitted (0 = no cap) |
| `--timeout` | pdbmux request timeout (default 1m) |
`--pdb` defaults to the shared configuration described under
[Configuration](#configuration), so `pdb_url` and `CATALOG_DIFF_PDB` apply to
both binaries. No credentials are needed.
Facts for the whole estate are fetched from pdbmux in a single query and every
selector is evaluated in memory, so the cost is one HTTP request regardless of
how many files changed.
## How a changed file becomes a host list
Classification runs through an exec ENC and `manifests/site.pp` is empty, so a
node's `enc_role` fact is its classification. Everything below keys off that
and the facts `hiera.yaml` interpolates.
**Hieradata paths become fact selectors**, mirroring the hierarchy:
| Path | Selects |
|---|---|
| `hieradata/nodes/<certname>.yaml` | that node |
| `hieradata/roles/<t1>[/<t2>[/<t3>]]` | role-tier **prefix** — `roles/infra/puppet.yaml` hits every `roles::infra::puppet::*` host |
| `hieradata/country/<c>/region/<r>/<t1>[/...]` | country and region and role-tier prefix |
| `hieradata/country/<c>/roles/<t1>[/...]` | country and role-tier prefix |
| `hieradata/country/<c>/region/<r>` | country and region |
| `hieradata/country/<c>` | country |
| `hieradata/virtual/<v>.yaml` | `virtual` == v |
| `hieradata/os/<name>/<name><major>[.<minor>].yaml` | `os.name` and `os.release.major` (and minor when present) |
| `hieradata/os/<name>/all_releases.yaml` | every host with that `os.name` |
| `hieradata/common.{yaml,eyaml}` | every host |
**Repo-global files select every host**: `hiera.yaml`, `environment.conf`,
`manifests/site.pp`, `Puppetfile`. The Puppetfile pins external modules whose
code is not in this repo, so there is nothing to trace — mapping it to the
whole estate is the deliberate conservative choice. `lib/` under any module is
pluginsynced to every agent and is treated the same way.
**Code paths become classes**, then the classes are traced backwards:
- `site/roles/manifests/<t1>/<t2>/<t3>.pp` is class
`roles::<t1>::<t2>::<t3>`, which maps to hosts by exact `enc_role`. A role
class is a leaf — nothing includes it, the ENC assigns it.
- `site/profiles/manifests/**.pp` and `modules/<mod>/manifests/**.pp` become
their autoloaded class names (`init.pp` counts as both the namespace and
`<namespace>::init`) and need a reverse lookup.
- `modules/<mod>/{templates,files}/**` and `site/profiles/{templates,files}/**`
are attributed to the classes whose manifests reference them through
`template()`, `epp()` or `puppet:///modules/...`. An unreferenced asset falls
back to every class its module defines.
## The reverse lookup
Finding what applies a class means searching two places, not one:
1. `.pp` files under `site/` and `modules/` for `include`, `contain`,
`class { '...': }`, capitalised resource references and defined-type usage.
2. `hieradata/**/*.{yaml,eyaml}` for the class as a list item under
`hiera_include:`. `profiles::base` declares
`($hiera_include - $hiera_exclude).include` with a `unique` merge across the
whole hierarchy, and it is included by essentially every role.
The second is not an edge case. `hieradata/roles/apps/media/jellyfin.yaml` is
the only thing that applies the `jellyfin` module — there is no `include
jellyfin` anywhere — so a manifest-only search reports it as orphaned and finds
no hosts. A `hiera_include` hit is resolved through the hieradata path rules
above: the position of the listing file in the hierarchy decides which hosts
get the class. `hiera_exclude` entries are subtracted the same way.
The walk runs to a fixed point, because profiles include profiles several
layers deep — `profiles::base` alone pulls in around twenty-five. Cycles are
visited once.
## Bias and limits
A missed host means an unvalidated catalog, so every ambiguity widens:
- a path matching no rule is reported on stderr and mapped to every host;
- an asset no manifest references widens to its whole module;
- an unrecognised hieradata path in a `hiera_include` hit widens to every host.
Paths outside the Puppet modulepath — `.woodpecker/`, `doc/`, `spec/`,
`vendor/`, `Gemfile`, `*.md` — cannot reach a catalog and are skipped.
Two things the tool cannot see:
- A class nothing declares and no `hiera_include` lists reaches no hosts. That
is reported as an orphan on stderr rather than silently dropped.
- The index is built from the checkout at `--repo`, not from `--branch`. In CI
the checkout is the branch under test; pointing `--branch` at a different ref
than the working tree means classes the branch adds look orphaned.
## Development
```bash
+152
View File
@@ -0,0 +1,152 @@
// Command catalog-impact maps a change in the Puppet control repo onto the
// hosts whose catalogs it can alter.
package main
import (
"context"
"fmt"
"io"
"os"
"time"
"github.com/spf13/cobra"
"git.unkin.net/unkin/puppet-catalog-tools/internal/config"
"git.unkin.net/unkin/puppet-catalog-tools/internal/impact"
"git.unkin.net/unkin/puppet-catalog-tools/internal/pdbmux"
)
var version = "dev"
// Exit codes, also documented in --help and the README.
const (
exitOK = 0 // analysis completed
exitUsage = 1 // transport, git or usage error
)
const longHelp = `Work out which hosts a change to the Puppet control repo can affect, and print
their hostnames one per line.
Changed files are mapped onto hosts two ways. A hieradata path is read as its
position in the hiera hierarchy and turned into a fact selector. A manifest,
template or file is turned into the classes it defines, and those classes are
traced backwards — through include, contain and class {} in manifests, and
through hiera_include in hieradata — until only role classes and hierarchy
positions remain.
The bias is to over-include: an unrecognised path widens to the whole estate
rather than silently returning nothing.`
type options struct {
repo string
branch string
base string
files []string
pdb string
jsonOut bool
explain bool
limit int
timeout time.Duration
}
func main() {
cfg, err := config.Load()
if err != nil {
fmt.Fprintln(os.Stderr, "config error:", err)
os.Exit(exitUsage)
}
opts := &options{}
root := &cobra.Command{
Use: "catalog-impact",
Short: "List the hosts a Puppet control repo change affects",
Long: longHelp,
Args: cobra.NoArgs,
Version: version,
SilenceUsage: true,
SilenceErrors: true,
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), cmd.OutOrStdout(), opts)
},
}
f := root.Flags()
f.StringVarP(&opts.repo, "repo", "r", ".", "path to a puppet-prod checkout")
f.StringVarP(&opts.branch, "branch", "b", "", "branch under test (default: the repo's current branch)")
f.StringVar(&opts.base, "base", "origin/develop", "base ref to diff against")
f.StringSliceVar(&opts.files, "files", nil, "explicit changed-file list, bypassing git")
f.StringVar(&opts.pdb, "pdb", cfg.PDBURL, "pdbmux base URL")
f.BoolVar(&opts.jsonOut, "json", false, "emit structured output with per-host reasons")
f.BoolVar(&opts.explain, "explain", false, "human-readable: for each host, which changed files hit it and why")
f.IntVar(&opts.limit, "limit", 0, "cap the number of hosts emitted (0 = no cap)")
f.DurationVar(&opts.timeout, "timeout", time.Minute, "pdbmux request timeout")
if err := root.Execute(); err != nil {
fmt.Fprintln(os.Stderr, "error:", err)
os.Exit(exitUsage)
}
os.Exit(exitOK)
}
func run(ctx context.Context, out io.Writer, opts *options) error {
if opts.jsonOut && opts.explain {
return fmt.Errorf("--json and --explain are mutually exclusive")
}
if opts.limit < 0 {
return fmt.Errorf("--limit must not be negative")
}
branch := opts.branch
files := opts.files
if len(files) == 0 {
var err error
if branch == "" {
if branch, err = impact.CurrentBranch(opts.repo); err != nil {
return err
}
}
if files, err = impact.ChangedFiles(opts.repo, opts.base, branch); err != nil {
return err
}
}
if branch == "" {
branch = "(explicit file list)"
}
idx, err := impact.BuildIndex(opts.repo)
if err != nil {
return fmt.Errorf("indexing %s: %w", opts.repo, err)
}
nodes, err := impact.LoadNodes(ctx, pdbmux.New(opts.pdb, opts.timeout))
if err != nil {
return err
}
analyzer := &impact.Analyzer{Index: idx, Nodes: nodes}
result := analyzer.Analyze(opts.base, branch, files, opts.limit)
switch {
case opts.jsonOut:
return impact.RenderJSON(out, result)
case opts.explain:
return impact.RenderExplain(out, result)
}
warn(result)
return impact.RenderPlain(out, result)
}
// warn surfaces the widenings and dead ends on stderr so a piped host list
// stays clean.
func warn(r *impact.Result) {
for _, file := range r.UnmappedFiles {
fmt.Fprintf(os.Stderr, "warning: %s matches no mapping rule; every host is included\n", file)
}
for _, class := range r.OrphanClasses {
fmt.Fprintf(os.Stderr, "warning: class %s is declared by nothing and listed in no hiera_include; it reaches no hosts\n", class)
}
if r.Truncated {
fmt.Fprintf(os.Stderr, "warning: %d hosts affected, output capped at %d by --limit\n", r.Summary.Hosts, len(r.Hosts))
}
}
+89
View File
@@ -0,0 +1,89 @@
package main
import (
"bytes"
"context"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
)
const estateBody = `[
{"certname":"web1.example.net","name":"enc_role","value":"roles::apps::media::jellyfin"},
{"certname":"web1.example.net","name":"enc_role_tier1","value":"apps"},
{"certname":"web1.example.net","name":"enc_role_tier2","value":"media"},
{"certname":"web1.example.net","name":"enc_role_tier3","value":"jellyfin"},
{"certname":"web1.example.net","name":"country","value":"au"},
{"certname":"web1.example.net","name":"region","value":"syd1"},
{"certname":"web1.example.net","name":"virtual","value":"lxc"},
{"certname":"web1.example.net","name":"os","value":{"name":"AlmaLinux","release":{"major":"9","minor":"7"}}},
{"certname":"k8s1.example.net","name":"enc_role","value":"roles::infra::k8s::node"},
{"certname":"k8s1.example.net","name":"enc_role_tier1","value":"infra"},
{"certname":"k8s1.example.net","name":"enc_role_tier2","value":"k8s"},
{"certname":"k8s1.example.net","name":"enc_role_tier3","value":"node"},
{"certname":"k8s1.example.net","name":"country","value":"au"},
{"certname":"k8s1.example.net","name":"region","value":"syd1"},
{"certname":"k8s1.example.net","name":"virtual","value":"physical"},
{"certname":"k8s1.example.net","name":"os","value":{"name":"AlmaLinux","release":{"major":"9","minor":"8"}}}
]`
func fakePDB(t *testing.T) string {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(estateBody))
}))
t.Cleanup(srv.Close)
return srv.URL
}
func TestRunWithExplicitFileList(t *testing.T) {
var out bytes.Buffer
opts := &options{
repo: "../../internal/impact/testdata/repo",
base: "origin/develop",
files: []string{"modules/jellyfin/manifests/install.pp"},
pdb: fakePDB(t),
timeout: 5 * time.Second,
}
if err := run(context.Background(), &out, opts); err != nil {
t.Fatalf("run: %v", err)
}
if got, want := out.String(), "web1.example.net\n"; got != want {
t.Errorf("output = %q, want %q", got, want)
}
}
func TestRunExplain(t *testing.T) {
var out bytes.Buffer
opts := &options{
repo: "../../internal/impact/testdata/repo",
base: "origin/develop",
files: []string{"hieradata/common.yaml"},
pdb: fakePDB(t),
explain: true,
timeout: 5 * time.Second,
}
if err := run(context.Background(), &out, opts); err != nil {
t.Fatalf("run: %v", err)
}
if !strings.Contains(out.String(), "the whole estate is affected") {
t.Errorf("explain output = %q", out.String())
}
}
func TestRunRejectsConflictingOutputFlags(t *testing.T) {
opts := &options{jsonOut: true, explain: true}
if err := run(context.Background(), &bytes.Buffer{}, opts); err == nil {
t.Fatal("expected an error for --json with --explain")
}
}
func TestRunRejectsNegativeLimit(t *testing.T) {
opts := &options{limit: -1}
if err := run(context.Background(), &bytes.Buffer{}, opts); err == nil {
t.Fatal("expected an error for a negative --limit")
}
}
+53
View File
@@ -0,0 +1,53 @@
package impact
import (
"bytes"
"fmt"
"os/exec"
"strings"
)
// CurrentBranch returns the branch checked out in repo.
func CurrentBranch(repo string) (string, error) {
out, err := git(repo, "rev-parse", "--abbrev-ref", "HEAD")
if err != nil {
return "", err
}
branch := strings.TrimSpace(out)
if branch == "" || branch == "HEAD" {
return "", fmt.Errorf("%s is not on a branch; pass --branch", repo)
}
return branch, nil
}
// ChangedFiles lists the paths that differ between base and branch, using the
// three-dot form so only the branch's own commits count. Renames contribute
// both the old and the new path, because a moved manifest changes the classes
// at both ends.
func ChangedFiles(repo, base, branch string) ([]string, error) {
out, err := git(repo, "diff", "--name-status", "--find-renames", base+"..."+branch)
if err != nil {
return nil, err
}
var files []string
for _, line := range strings.Split(out, "\n") {
fields := strings.Split(strings.TrimSpace(line), "\t")
if len(fields) < 2 || fields[0] == "" {
continue
}
files = append(files, fields[1:]...)
}
return uniqueSorted(files), nil
}
func git(repo string, args ...string) (string, error) {
cmd := exec.Command("git", append([]string{"-C", repo}, args...)...)
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
return "", fmt.Errorf("git %s: %w: %s", strings.Join(args, " "), err, strings.TrimSpace(stderr.String()))
}
return stdout.String(), nil
}
+89
View File
@@ -0,0 +1,89 @@
package impact
import (
"os"
"os/exec"
"path/filepath"
"reflect"
"testing"
)
func gitRepo(t *testing.T) string {
t.Helper()
if _, err := exec.LookPath("git"); err != nil {
t.Skip("git not available")
}
dir := t.TempDir()
run := func(args ...string) {
t.Helper()
cmd := exec.Command("git", append([]string{"-C", dir}, args...)...)
cmd.Env = append(os.Environ(), "GIT_CONFIG_GLOBAL=/dev/null", "GIT_CONFIG_SYSTEM=/dev/null")
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatalf("git %v: %v: %s", args, err, out)
}
}
write := func(name, body string) {
t.Helper()
p := filepath.Join(dir, name)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
}
run("init", "--initial-branch=develop")
run("config", "user.email", "test@example.net")
run("config", "user.name", "test")
write("keep.pp", "keep\n")
write("gone.pp", "gone\n")
write("old/name.pp", "some reasonably long body so rename detection fires\n")
run("add", ".")
run("commit", "-m", "base")
run("checkout", "-b", "feature")
write("added.pp", "added\n")
write("keep.pp", "changed\n")
if err := os.Remove(filepath.Join(dir, "gone.pp")); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(filepath.Join(dir, "new"), 0o755); err != nil {
t.Fatal(err)
}
run("mv", "old/name.pp", "new/name.pp")
run("add", "-A")
run("commit", "-m", "feature")
run("checkout", "develop")
return dir
}
func TestChangedFilesCoversAddDeleteAndRename(t *testing.T) {
dir := gitRepo(t)
got, err := ChangedFiles(dir, "develop", "feature")
if err != nil {
t.Fatalf("ChangedFiles: %v", err)
}
want := []string{"added.pp", "gone.pp", "keep.pp", "new/name.pp", "old/name.pp"}
if !reflect.DeepEqual(got, want) {
t.Errorf("ChangedFiles = %v, want %v", got, want)
}
}
func TestCurrentBranch(t *testing.T) {
dir := gitRepo(t)
got, err := CurrentBranch(dir)
if err != nil {
t.Fatalf("CurrentBranch: %v", err)
}
if got != "develop" {
t.Errorf("CurrentBranch = %q, want develop", got)
}
}
func TestChangedFilesReportsGitErrors(t *testing.T) {
dir := gitRepo(t)
if _, err := ChangedFiles(dir, "develop", "no-such-branch"); err == nil {
t.Fatal("expected an error for a missing ref")
}
}
+159
View File
@@ -0,0 +1,159 @@
package impact
import "sort"
// Reason explains why one changed file put one host in the result.
type Reason struct {
File string `json:"file"`
Rule string `json:"rule"`
Detail string `json:"detail"`
}
// Host is an affected node and every reason it is affected.
type Host struct {
Certname string `json:"certname"`
Reasons []Reason `json:"reasons"`
}
// Summary is the headline count.
type Summary struct {
Hosts int `json:"hosts"`
Files int `json:"files"`
AllHostsTriggered bool `json:"all_hosts_triggered"`
}
// Result is the whole analysis, and the --json document.
type Result struct {
Base string `json:"base"`
Branch string `json:"branch"`
ChangedFiles []string `json:"changed_files"`
Hosts []Host `json:"hosts"`
UnmappedFiles []string `json:"unmapped_files"`
Summary Summary `json:"summary"`
// IgnoredFiles are changed files outside the modulepath — CI, docs,
// specs — which cannot reach a catalog.
IgnoredFiles []string `json:"-"`
// OrphanClasses are classes a changed file defines that nothing declares
// and no hieradata file pulls in.
OrphanClasses []string `json:"-"`
// Truncated is set when --limit cut the host list.
Truncated bool `json:"-"`
// TotalHosts is the estate size the selectors were evaluated against.
TotalHosts int `json:"-"`
}
// Certnames returns the affected hostnames, sorted.
func (r *Result) Certnames() []string {
out := make([]string, 0, len(r.Hosts))
for _, h := range r.Hosts {
out = append(out, h.Certname)
}
return out
}
// Analyzer evaluates changed files against one estate fact table.
type Analyzer struct {
Index *Index
Nodes []Node
}
// Analyze maps changed files onto hosts. Files are processed in the order
// given; everything in the result is sorted, so output does not move between
// runs.
func (a *Analyzer) Analyze(base, branch string, files []string, limit int) *Result {
resolver := newClassResolver(a.Index, a.Nodes)
reasons := map[string][]Reason{}
orphans := map[string]bool{}
result := &Result{
Base: base,
Branch: branch,
ChangedFiles: uniqueSorted(files),
TotalHosts: len(a.Nodes),
}
for _, file := range result.ChangedFiles {
m := classifyPath(file, a.Index)
switch {
case m.ignored:
result.IgnoredFiles = append(result.IgnoredFiles, file)
continue
case m.unknown:
result.UnmappedFiles = append(result.UnmappedFiles, file)
m.targets = []target{allTarget(RuleUnmapped,
fmtDetail("%s matches no mapping rule; widened to the whole estate", file))}
}
for _, t := range m.targets {
for _, certname := range t.sel.Select(a.Nodes) {
reasons[certname] = append(reasons[certname],
Reason{File: file, Rule: t.rule, Detail: t.detail})
}
}
for _, ref := range m.classes {
res := resolver.resolve(ref.name)
if res.orphan {
orphans[ref.name] = true
continue
}
for certname, hits := range res.hosts {
for _, h := range hits {
reasons[certname] = append(reasons[certname],
Reason{File: file, Rule: h.rule, Detail: ref.why + "; " + h.detail})
}
}
}
}
for name := range orphans {
result.OrphanClasses = append(result.OrphanClasses, name)
}
sort.Strings(result.OrphanClasses)
certnames := make([]string, 0, len(reasons))
for certname := range reasons {
certnames = append(certnames, certname)
}
sort.Strings(certnames)
result.Summary = Summary{
Hosts: len(certnames),
Files: len(result.ChangedFiles),
AllHostsTriggered: len(certnames) > 0 && len(certnames) == len(a.Nodes),
}
if limit > 0 && len(certnames) > limit {
certnames = certnames[:limit]
result.Truncated = true
}
for _, certname := range certnames {
result.Hosts = append(result.Hosts, Host{Certname: certname, Reasons: dedupeReasons(reasons[certname])})
}
return result
}
func dedupeReasons(in []Reason) []Reason {
sort.Slice(in, func(i, j int) bool {
if in[i].File != in[j].File {
return in[i].File < in[j].File
}
if in[i].Rule != in[j].Rule {
return in[i].Rule < in[j].Rule
}
return in[i].Detail < in[j].Detail
})
out := in[:0]
var prev Reason
for idx, r := range in {
if idx > 0 && r == prev {
continue
}
prev = r
out = append(out, r)
}
return out
}
+256
View File
@@ -0,0 +1,256 @@
package impact
import (
"bytes"
"context"
"encoding/json"
"reflect"
"strings"
"testing"
"git.unkin.net/unkin/puppet-catalog-tools/internal/pdbmux"
)
func testAnalyzer(t *testing.T) *Analyzer {
t.Helper()
return &Analyzer{Index: testIndex(t), Nodes: fixtureNodes()}
}
func analyze(t *testing.T, files ...string) *Result {
t.Helper()
return testAnalyzer(t).Analyze("origin/develop", "benvin/test", files, 0)
}
func TestAnalyzeCommonAndHieraTriggerEveryHost(t *testing.T) {
all := []string{"bare1", "k8s1", "k8s2", "prox1", "web1"}
for _, file := range []string{"hieradata/common.yaml", "hiera.yaml", "environment.conf", "Puppetfile", "manifests/site.pp"} {
r := analyze(t, file)
if got := r.Certnames(); !reflect.DeepEqual(got, all) {
t.Errorf("%s affected %v, want every host", file, got)
}
if !r.Summary.AllHostsTriggered {
t.Errorf("%s did not set all_hosts_triggered", file)
}
}
}
func TestAnalyzeHieradataRolePrefix(t *testing.T) {
r := analyze(t, "hieradata/roles/infra.yaml")
if got, want := r.Certnames(), []string{"k8s1", "k8s2", "prox1"}; !reflect.DeepEqual(got, want) {
t.Errorf("hosts = %v, want %v", got, want)
}
if r.Summary.AllHostsTriggered {
t.Error("a role-prefix change must not report the whole estate")
}
}
func TestAnalyzeRoleManifestIsExact(t *testing.T) {
r := analyze(t, "site/roles/manifests/infra/k8s/node.pp")
if got, want := r.Certnames(), []string{"k8s1", "k8s2"}; !reflect.DeepEqual(got, want) {
t.Errorf("hosts = %v, want %v", got, want)
}
if r.Hosts[0].Reasons[0].Rule != RuleRoleClass {
t.Errorf("rule = %q, want %q", r.Hosts[0].Reasons[0].Rule, RuleRoleClass)
}
}
func TestAnalyzeTemplateAttribution(t *testing.T) {
r := analyze(t, "site/profiles/templates/leaf/x.erb")
if got, want := r.Certnames(), []string{"k8s1", "k8s2", "web1"}; !reflect.DeepEqual(got, want) {
t.Errorf("hosts = %v, want %v", got, want)
}
}
func TestAnalyzeUnattributedModuleAssetWidensToTheModule(t *testing.T) {
// Nothing references this template, so it falls back to every class the
// module defines — here, the hiera_include-only jellyfin module.
r := analyze(t, "modules/jellyfin/templates/unreferenced.erb")
if got, want := r.Certnames(), []string{"web1"}; !reflect.DeepEqual(got, want) {
t.Errorf("hosts = %v, want %v", got, want)
}
}
func TestAnalyzeUnmappedFileWidensToEveryHost(t *testing.T) {
r := analyze(t, "totally/unknown/thing.txt")
if got, want := r.UnmappedFiles, []string{"totally/unknown/thing.txt"}; !reflect.DeepEqual(got, want) {
t.Errorf("unmapped = %v, want %v", got, want)
}
if len(r.Hosts) != len(fixtureNodes()) {
t.Errorf("unmapped file affected %d hosts, want all %d", len(r.Hosts), len(fixtureNodes()))
}
if r.Hosts[0].Reasons[0].Rule != RuleUnmapped {
t.Errorf("rule = %q", r.Hosts[0].Reasons[0].Rule)
}
}
func TestAnalyzeIgnoresNonPuppetPaths(t *testing.T) {
r := analyze(t, ".woodpecker/test.yaml", "modules/orphanmod/spec/orphanmod_spec.rb", "README.md")
if len(r.Hosts) != 0 {
t.Errorf("CI and spec changes affected %v", r.Certnames())
}
if len(r.IgnoredFiles) != 3 {
t.Errorf("ignored = %v", r.IgnoredFiles)
}
}
func TestAnalyzeReportsOrphanClasses(t *testing.T) {
r := analyze(t, "modules/orphanmod/manifests/init.pp")
if got, want := r.OrphanClasses, []string{"orphanmod", "orphanmod::init"}; !reflect.DeepEqual(got, want) {
t.Errorf("orphans = %v, want %v", got, want)
}
if len(r.Hosts) != 0 {
t.Errorf("orphan class reached %v", r.Certnames())
}
}
func TestAnalyzeLimit(t *testing.T) {
r := testAnalyzer(t).Analyze("origin/develop", "benvin/test", []string{"hieradata/common.yaml"}, 2)
if got, want := r.Certnames(), []string{"bare1", "k8s1"}; !reflect.DeepEqual(got, want) {
t.Errorf("hosts = %v, want %v", got, want)
}
if !r.Truncated {
t.Error("Truncated not set")
}
if r.Summary.Hosts != len(fixtureNodes()) {
t.Errorf("summary.hosts = %d, want the untruncated count", r.Summary.Hosts)
}
}
func TestAnalyzeIsDeterministic(t *testing.T) {
files := []string{
"hieradata/roles/apps/media/jellyfin.yaml",
"site/profiles/manifests/leaf.pp",
"modules/jellyfin/manifests/install.pp",
"hieradata/virtual/physical.yaml",
"totally/unknown/thing.txt",
}
var first string
for i := range 20 {
var buf bytes.Buffer
if err := RenderJSON(&buf, analyze(t, files...)); err != nil {
t.Fatalf("RenderJSON: %v", err)
}
if i == 0 {
first = buf.String()
continue
}
if buf.String() != first {
t.Fatal("output changed between runs")
}
}
}
func TestRenderJSONShape(t *testing.T) {
var buf bytes.Buffer
if err := RenderJSON(&buf, analyze(t, "hieradata/roles/apps/media/jellyfin.yaml")); err != nil {
t.Fatalf("RenderJSON: %v", err)
}
var doc struct {
Base string `json:"base"`
Branch string `json:"branch"`
ChangedFiles []string `json:"changed_files"`
UnmappedFiles []string `json:"unmapped_files"`
Hosts []struct {
Certname string `json:"certname"`
Reasons []struct {
File string `json:"file"`
Rule string `json:"rule"`
Detail string `json:"detail"`
} `json:"reasons"`
} `json:"hosts"`
Summary struct {
Hosts int `json:"hosts"`
Files int `json:"files"`
AllHostsTriggered bool `json:"all_hosts_triggered"`
} `json:"summary"`
}
if err := json.Unmarshal(buf.Bytes(), &doc); err != nil {
t.Fatalf("unmarshal: %v", err)
}
if doc.Base != "origin/develop" || doc.Branch != "benvin/test" {
t.Errorf("base/branch = %q/%q", doc.Base, doc.Branch)
}
if len(doc.Hosts) != 1 || doc.Hosts[0].Certname != "web1" {
t.Fatalf("hosts = %#v", doc.Hosts)
}
if len(doc.Hosts[0].Reasons) != 1 || doc.Hosts[0].Reasons[0].Rule != RuleRole {
t.Errorf("reasons = %#v", doc.Hosts[0].Reasons)
}
if doc.Summary.Files != 1 || doc.Summary.Hosts != 1 || doc.Summary.AllHostsTriggered {
t.Errorf("summary = %#v", doc.Summary)
}
}
func TestRenderPlainIsOneHostPerLine(t *testing.T) {
var buf bytes.Buffer
if err := RenderPlain(&buf, analyze(t, "hieradata/roles/infra.yaml")); err != nil {
t.Fatalf("RenderPlain: %v", err)
}
if got, want := buf.String(), "k8s1\nk8s2\nprox1\n"; got != want {
t.Errorf("output = %q, want %q", got, want)
}
}
func TestRenderExplainNamesFileRuleAndHost(t *testing.T) {
var buf bytes.Buffer
if err := RenderExplain(&buf, analyze(t, "modules/jellyfin/manifests/install.pp", "bogus/path")); err != nil {
t.Fatalf("RenderExplain: %v", err)
}
out := buf.String()
for _, want := range []string{"web1", RuleHieraInclude, "hieradata/roles/apps/media/jellyfin.yaml", "bogus/path", "unmapped files"} {
if !strings.Contains(out, want) {
t.Errorf("explain output missing %q:\n%s", want, out)
}
}
}
// stubFacts serves a canned fact table the way pdbmux would.
type stubFacts struct {
records []pdbmux.FactRecord
asked []string
}
func (s *stubFacts) FactsByName(_ context.Context, names []string) ([]pdbmux.FactRecord, error) {
s.asked = names
return s.records, nil
}
func TestLoadNodesFoldsStructuredOS(t *testing.T) {
src := &stubFacts{records: []pdbmux.FactRecord{
{Certname: "b.example.net", Name: "enc_role", Value: "roles::infra::k8s::node"},
{Certname: "b.example.net", Name: "enc_role_tier1", Value: "infra"},
{Certname: "b.example.net", Name: "enc_role_tier2", Value: "k8s"},
{Certname: "b.example.net", Name: "enc_role_tier3", Value: "node"},
{Certname: "b.example.net", Name: "country", Value: "au"},
{Certname: "b.example.net", Name: "region", Value: "syd1"},
{Certname: "b.example.net", Name: "virtual", Value: "physical"},
{Certname: "b.example.net", Name: "os", Value: map[string]any{
"name": "AlmaLinux", "release": map[string]any{"major": "9", "minor": "7"},
}},
{Certname: "a.example.net", Name: "enc_role", Value: "roles::base"},
{Certname: "a.example.net", Name: "enc_role_tier1", Value: "base"},
}}
nodes, err := LoadNodes(context.Background(), src)
if err != nil {
t.Fatalf("LoadNodes: %v", err)
}
if !reflect.DeepEqual(src.asked, FactNames) {
t.Errorf("asked for %v, want %v", src.asked, FactNames)
}
if len(nodes) != 2 || nodes[0].Certname != "a.example.net" {
t.Fatalf("nodes = %#v", nodes)
}
want := Node{Certname: "b.example.net", Role: "roles::infra::k8s::node",
Tier1: "infra", Tier2: "k8s", Tier3: "node", Country: "au", Region: "syd1",
Virtual: "physical", OSName: "AlmaLinux", OSMajor: "9", OSMinor: "7"}
if nodes[1] != want {
t.Errorf("node = %#v, want %#v", nodes[1], want)
}
}
func TestLoadNodesRejectsEmptyEstate(t *testing.T) {
if _, err := LoadNodes(context.Background(), &stubFacts{}); err == nil {
t.Fatal("expected an error for an empty estate")
}
}
+279
View File
@@ -0,0 +1,279 @@
package impact
import (
"fmt"
"io/fs"
"os"
"path/filepath"
"regexp"
"sort"
"strings"
"gopkg.in/yaml.v3"
)
// manifestRoots are the modulepath entries that live in this repo.
// external_modules/ is Puppetfile-managed and not checked in.
var manifestRoots = []string{"site", "modules"}
var (
commentRe = regexp.MustCompile(`(?m)#.*$`)
includeRe = regexp.MustCompile(`\b(?:include|contain|require)\s*\(?\s*['"]?(?:::)?([a-z][a-z0-9_]*(?:::[a-z0-9_]+)*)['"]?`)
classParamRe = regexp.MustCompile(`\bclass\s*\{\s*['"](?:::)?([a-z][a-z0-9_]*(?:::[a-z0-9_]+)*)['"]`)
classRefRe = regexp.MustCompile(`\bClass\s*\[\s*['"](?:::)?([a-z][a-z0-9_]*(?:::[a-z0-9_]+)*)['"]`)
defineUseRe = regexp.MustCompile(`(?m)^\s*(?:::)?([a-z][a-z0-9_]*(?:::[a-z0-9_]+)+)\s*\{`)
resourceRefRe = regexp.MustCompile(`\b([A-Z][a-zA-Z0-9_]*(?:::[A-Z][a-zA-Z0-9_]*)+)\s*\[`)
assetRe = regexp.MustCompile(`(?:template|epp|file)\s*\(\s*['"]([a-z0-9_]+/[^'"]+)['"]|puppet:///modules/([a-z0-9_]+/[^'"\s]+)`)
hieraBlockRe = regexp.MustCompile(`(?m)^(hiera_include|hiera_exclude):\s*$((?:\n\s+-\s*\S+)*)`)
hieraItemRe = regexp.MustCompile(`(?m)^\s+-\s*['"]?([^'"\s]+)['"]?\s*$`)
)
// Index is a static view of the control repo: which file defines which class,
// which classes reference which, which hieradata files pull classes in through
// hiera_include, and which manifest owns which template or file.
type Index struct {
Root string
// includedBy maps a class name to the classes that declare it.
includedBy map[string]map[string]bool
// moduleClasses maps a module name to the classes it defines.
moduleClasses map[string][]string
// assetOwners maps "<module>/<path>" to the classes referencing it.
assetOwners map[string][]string
// hieraInclude and hieraExclude map a class name to the hieradata files
// listing it.
hieraInclude map[string][]string
hieraExclude map[string][]string
// known holds every class this repo defines.
known map[string]bool
}
// BuildIndex walks the control repo at root and indexes it.
func BuildIndex(root string) (*Index, error) {
idx := &Index{
Root: root,
includedBy: map[string]map[string]bool{},
moduleClasses: map[string][]string{},
assetOwners: map[string][]string{},
hieraInclude: map[string][]string{},
hieraExclude: map[string][]string{},
known: map[string]bool{},
}
for _, base := range manifestRoots {
dir := filepath.Join(root, base)
if _, err := os.Stat(dir); err != nil {
continue
}
if err := filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
if err != nil || d.IsDir() || !strings.HasSuffix(p, ".pp") {
return err
}
rel, relErr := filepath.Rel(root, p)
if relErr != nil {
return relErr
}
return idx.addManifest(filepath.ToSlash(rel), p)
}); err != nil {
return nil, err
}
}
hieraDir := filepath.Join(root, "hieradata")
if _, err := os.Stat(hieraDir); err == nil {
if err := filepath.WalkDir(hieraDir, func(p string, d fs.DirEntry, err error) error {
if err != nil || d.IsDir() {
return err
}
if ext := filepath.Ext(p); ext != ".yaml" && ext != ".eyaml" {
return nil
}
rel, relErr := filepath.Rel(root, p)
if relErr != nil {
return relErr
}
return idx.addHieradata(filepath.ToSlash(rel), p)
}); err != nil {
return nil, err
}
}
for module := range idx.moduleClasses {
sort.Strings(idx.moduleClasses[module])
}
return idx, nil
}
func (i *Index) addManifest(rel, full string) error {
parts := strings.Split(rel, "/")
if len(parts) < 4 || parts[2] != "manifests" {
return nil
}
module := parts[1]
declared := ManifestClasses(module, parts[3:])
for _, name := range declared {
i.known[name] = true
i.moduleClasses[module] = append(i.moduleClasses[module], name)
}
data, err := os.ReadFile(full) //nolint:gosec // paths come from walking the given repo
if err != nil {
return fmt.Errorf("reading %s: %w", rel, err)
}
body := commentRe.ReplaceAllString(string(data), "")
self := map[string]bool{}
for _, name := range declared {
self[name] = true
}
for _, ref := range references(body) {
if self[ref] {
continue
}
if i.includedBy[ref] == nil {
i.includedBy[ref] = map[string]bool{}
}
for _, name := range declared {
i.includedBy[ref][name] = true
}
}
for _, asset := range assetReferences(body) {
i.assetOwners[asset] = append(i.assetOwners[asset], declared...)
}
return nil
}
// references extracts every class or defined-type name a manifest body names,
// across all the declaration forms this repo uses.
func references(body string) []string {
seen := map[string]bool{}
add := func(name string) {
if name != "" && name != "class" && name != "define" {
seen[name] = true
}
}
for _, re := range []*regexp.Regexp{includeRe, classParamRe, classRefRe, defineUseRe} {
for _, m := range re.FindAllStringSubmatch(body, -1) {
add(m[1])
}
}
// Resource references (Profiles::Foo::Bar['x']) name a defined type in
// capitalised form.
for _, m := range resourceRefRe.FindAllStringSubmatch(body, -1) {
add(strings.ToLower(m[1]))
}
out := make([]string, 0, len(seen))
for name := range seen {
out = append(out, name)
}
sort.Strings(out)
return out
}
func assetReferences(body string) []string {
var out []string
for _, m := range assetRe.FindAllStringSubmatch(body, -1) {
for _, g := range m[1:] {
if g != "" {
out = append(out, g)
}
}
}
return out
}
func (i *Index) addHieradata(rel, full string) error {
data, err := os.ReadFile(full) //nolint:gosec // paths come from walking the given repo
if err != nil {
return fmt.Errorf("reading %s: %w", rel, err)
}
include, exclude := parseHieraLists(data)
for _, name := range include {
i.hieraInclude[name] = append(i.hieraInclude[name], rel)
}
for _, name := range exclude {
i.hieraExclude[name] = append(i.hieraExclude[name], rel)
}
return nil
}
// parseHieraLists reads hiera_include and hiera_exclude. eyaml files can carry
// block scalars a strict parser chokes on, so a regex scan backs the YAML up.
func parseHieraLists(data []byte) (include, exclude []string) {
var doc map[string]any
if err := yaml.Unmarshal(data, &doc); err == nil && doc != nil {
return stringList(doc["hiera_include"]), stringList(doc["hiera_exclude"])
}
for _, m := range hieraBlockRe.FindAllStringSubmatch(string(data), -1) {
var items []string
for _, item := range hieraItemRe.FindAllStringSubmatch(m[2], -1) {
items = append(items, item[1])
}
if m[1] == "hiera_include" {
include = append(include, items...)
} else {
exclude = append(exclude, items...)
}
}
return include, exclude
}
func stringList(v any) []string {
items, ok := v.([]any)
if !ok {
return nil
}
var out []string
for _, item := range items {
if s, ok := item.(string); ok && s != "" {
out = append(out, s)
}
}
return out
}
// IncludedBy returns the classes that declare the given class, sorted.
func (i *Index) IncludedBy(class string) []string {
names := make([]string, 0, len(i.includedBy[class]))
for name := range i.includedBy[class] {
names = append(names, name)
}
sort.Strings(names)
return names
}
// ModuleClasses returns every class the module defines, sorted.
func (i *Index) ModuleClasses(module string) []string { return i.moduleClasses[module] }
// AssetOwners returns the classes referencing a "<module>/<path>" template or
// file, sorted and deduplicated.
func (i *Index) AssetOwners(asset string) []string { return uniqueSorted(i.assetOwners[asset]) }
// HieraInclude returns the hieradata files listing the class under
// hiera_include, sorted.
func (i *Index) HieraInclude(class string) []string { return uniqueSorted(i.hieraInclude[class]) }
// HieraExclude returns the hieradata files listing the class under
// hiera_exclude, sorted.
func (i *Index) HieraExclude(class string) []string { return uniqueSorted(i.hieraExclude[class]) }
// Known reports whether the repo defines the class.
func (i *Index) Known(class string) bool { return i.known[class] }
func uniqueSorted(in []string) []string {
if len(in) == 0 {
return nil
}
seen := map[string]bool{}
out := make([]string, 0, len(in))
for _, s := range in {
if !seen[s] {
seen[s] = true
out = append(out, s)
}
}
sort.Strings(out)
return out
}
+132
View File
@@ -0,0 +1,132 @@
// Package impact maps a change in the Puppet control repo onto the hosts whose
// catalogs that change can alter.
package impact
import (
"context"
"fmt"
"sort"
"strings"
"git.unkin.net/unkin/puppet-catalog-tools/internal/pdbmux"
)
// FactNames are the facts every selector is evaluated against. They are
// fetched for the whole estate in one query.
var FactNames = []string{
"enc_role",
"enc_role_tier1",
"enc_role_tier2",
"enc_role_tier3",
"country",
"region",
"virtual",
"os",
}
// Node is the slice of a node's facts the hiera hierarchy keys off.
type Node struct {
Certname string
Role string
Tier1 string
Tier2 string
Tier3 string
Country string
Region string
Virtual string
OSName string
OSMajor string
OSMinor string
}
// Tiers returns the role tiers as a slice, truncated at the first empty one.
func (n Node) Tiers() []string {
tiers := make([]string, 0, 3)
for _, t := range []string{n.Tier1, n.Tier2, n.Tier3} {
if t == "" {
break
}
tiers = append(tiers, t)
}
return tiers
}
// FactSource supplies the estate-wide fact table.
type FactSource interface {
FactsByName(ctx context.Context, names []string) ([]pdbmux.FactRecord, error)
}
// LoadNodes fetches FactNames for every node and folds them into a sorted
// table. Nodes without an enc_role are kept: they still match country, os and
// virtual selectors.
func LoadNodes(ctx context.Context, src FactSource) ([]Node, error) {
records, err := src.FactsByName(ctx, FactNames)
if err != nil {
return nil, err
}
byName := map[string]*Node{}
for _, r := range records {
if r.Certname == "" {
continue
}
n, ok := byName[r.Certname]
if !ok {
n = &Node{Certname: r.Certname}
byName[r.Certname] = n
}
applyFact(n, r.Name, r.Value)
}
nodes := make([]Node, 0, len(byName))
for _, n := range byName {
nodes = append(nodes, *n)
}
sort.Slice(nodes, func(i, j int) bool { return nodes[i].Certname < nodes[j].Certname })
if len(nodes) == 0 {
return nil, fmt.Errorf("pdbmux returned no nodes")
}
return nodes, nil
}
func applyFact(n *Node, name string, value any) {
switch name {
case "enc_role":
n.Role = str(value)
case "enc_role_tier1":
n.Tier1 = str(value)
case "enc_role_tier2":
n.Tier2 = str(value)
case "enc_role_tier3":
n.Tier3 = str(value)
case "country":
n.Country = str(value)
case "region":
n.Region = str(value)
case "virtual":
n.Virtual = str(value)
case "os":
os, ok := value.(map[string]any)
if !ok {
return
}
n.OSName = str(os["name"])
release, ok := os["release"].(map[string]any)
if !ok {
return
}
n.OSMajor = str(release["major"])
n.OSMinor = str(release["minor"])
}
}
func str(v any) string {
switch t := v.(type) {
case string:
return t
case nil:
return ""
default:
return strings.TrimSpace(fmt.Sprint(t))
}
}
+249
View File
@@ -0,0 +1,249 @@
package impact
import (
"path"
"strings"
)
// Rules, as they appear in --json output and --explain.
const (
RuleCommon = "hieradata/common"
RuleNode = "hieradata/node"
RuleRole = "hieradata/role"
RuleCountry = "hieradata/country"
RuleCountryRegion = "hieradata/country-region"
RuleCountryRegRole = "hieradata/country-region-role"
RuleCountryRole = "hieradata/country-role"
RuleVirtual = "hieradata/virtual"
RuleOS = "hieradata/os"
RuleRepoGlobal = "repo-global"
RulePluginsync = "pluginsync"
RuleRoleClass = "role-class"
RuleHieraInclude = "hiera_include"
RuleUnmapped = "unmapped"
RuleClassReverse = "class-reverse-lookup"
)
// repoGlobalFiles change classification or module resolution for the whole
// estate. Puppetfile pins external modules whose code is not in this repo, so
// it is deliberately mapped to every host rather than guessed at.
var repoGlobalFiles = map[string]bool{
"hiera.yaml": true,
"environment.conf": true,
"Puppetfile": true,
"Puppetfile.lock": true,
}
// ignoredPrefixes hold paths outside the Puppet modulepath: CI, docs, specs
// and the bundler vendor tree. Nothing under them can reach a catalog.
var ignoredPrefixes = []string{
".woodpecker/", ".github/", ".claude/", "ci/", "doc/", "docs/", "spec/", "vendor/", "examples/",
}
var ignoredFiles = map[string]bool{
"Gemfile": true, "Gemfile.lock": true, "Rakefile": true,
".rubocop.yml": true, ".reek.yml": true, ".pre-commit-config.yaml": true,
".gitignore": true, ".fixtures.yml": true, ".ruby-version": true,
"catalog_diff.py": true, "deploy.py": true,
}
// moduleSubdirIgnored are module subtrees that are not consulted during a
// catalog compile.
var moduleSubdirIgnored = map[string]bool{"spec": true, "examples": true, "doc": true, "docs": true}
// classRef is a class a changed file defines, awaiting reverse lookup.
type classRef struct {
name string
why string
}
// mapping is what a single changed path resolves to before hosts are selected.
type mapping struct {
targets []target
classes []classRef
ignored bool
unknown bool
}
// classifyPath maps one repo-relative path onto selectors or classes. idx may
// be nil, in which case asset files fall back to their whole module.
func classifyPath(p string, idx *Index) mapping {
p = path.Clean(strings.TrimPrefix(p, "./"))
if repoGlobalFiles[p] {
return mapping{targets: []target{allTarget(RuleRepoGlobal,
fmtDetail("%s is repo-global; every catalog reads it", p))}}
}
if strings.HasPrefix(p, "manifests/") {
return mapping{targets: []target{allTarget(RuleRepoGlobal,
"the environment manifest applies to every node")}}
}
if ignoredFiles[p] || strings.HasSuffix(p, ".md") {
return mapping{ignored: true}
}
for _, prefix := range ignoredPrefixes {
if strings.HasPrefix(p, prefix) {
return mapping{ignored: true}
}
}
if rest, ok := strings.CutPrefix(p, "hieradata/"); ok {
sel, rule, ok := hieradataSelector(rest)
if !ok {
return mapping{unknown: true}
}
return mapping{targets: []target{{sel: sel, rule: rule,
detail: fmtDetail("hieradata file selects %s", sel)}}}
}
if strings.HasPrefix(p, "site/") || strings.HasPrefix(p, "modules/") {
return classifyModulePath(p, idx)
}
return mapping{unknown: true}
}
// classifyModulePath handles everything under the modulepath roots: site/ (the
// roles and profiles modules) and modules/ (in-repo modules).
func classifyModulePath(p string, idx *Index) mapping {
parts := strings.Split(p, "/")
if len(parts) < 3 {
return mapping{unknown: true}
}
module, subdir := parts[1], parts[2]
rest := parts[3:]
if moduleSubdirIgnored[subdir] {
return mapping{ignored: true}
}
if subdir == "manifests" && len(rest) > 0 && strings.HasSuffix(rest[len(rest)-1], ".pp") {
var classes []classRef
for _, name := range ManifestClasses(module, rest) {
classes = append(classes, classRef{name: name, why: fmtDetail("%s defines class %s", p, name)})
}
return mapping{classes: classes}
}
// lib/ and facts.d/ are pluginsynced to every agent, so a custom fact,
// function or type there can move any catalog.
if subdir == "lib" || subdir == "facts.d" {
return mapping{targets: []target{allTarget(RulePluginsync,
fmtDetail("%s is pluginsynced to every agent", p))}}
}
if subdir == "templates" || subdir == "files" {
asset := module + "/" + strings.Join(rest, "/")
if idx != nil {
if owners := idx.AssetOwners(asset); len(owners) > 0 {
var classes []classRef
for _, name := range owners {
classes = append(classes, classRef{name: name,
why: fmtDetail("%s is referenced by class %s", p, name)})
}
return mapping{classes: classes}
}
}
return moduleWide(p, module, idx, "no manifest references this asset")
}
return moduleWide(p, module, idx, fmtDetail("%s/ is module-wide", subdir))
}
// moduleWide attributes a file to every class its module defines. It is the
// deliberate over-include fallback for module data that cannot be traced to a
// single class.
func moduleWide(p, module string, idx *Index, why string) mapping {
if idx == nil {
return mapping{unknown: true}
}
names := idx.ModuleClasses(module)
if len(names) == 0 {
return mapping{unknown: true}
}
var classes []classRef
for _, name := range names {
classes = append(classes, classRef{name: name,
why: fmtDetail("%s: %s, attributed to class %s", p, why, name)})
}
return mapping{classes: classes}
}
// ManifestClasses returns the class names a manifest file autoloads as. An
// init.pp is reachable both as the namespace itself and as <namespace>::init,
// and this repo uses both spellings.
func ManifestClasses(module string, rest []string) []string {
segs := append([]string{module}, rest...)
segs[len(segs)-1] = strings.TrimSuffix(segs[len(segs)-1], ".pp")
full := strings.Join(segs, "::")
if segs[len(segs)-1] != "init" {
return []string{full}
}
short := strings.Join(segs[:len(segs)-1], "::")
return []string{short, full}
}
// hieradataSelector maps a path under hieradata/ onto the facts the hiera
// hierarchy interpolates into it.
func hieradataSelector(rel string) (Selector, string, bool) {
ext := path.Ext(rel)
if ext != ".yaml" && ext != ".eyaml" {
return Selector{}, "", false
}
parts := strings.Split(strings.TrimSuffix(rel, ext), "/")
switch {
case len(parts) == 1 && parts[0] == "common":
return Selector{All: true}, RuleCommon, true
case parts[0] == "nodes" && len(parts) == 2:
return Selector{Certname: parts[1]}, RuleNode, true
case parts[0] == "virtual" && len(parts) == 2:
return Selector{Virtual: parts[1]}, RuleVirtual, true
case parts[0] == "os" && len(parts) == 3:
return osSelector(parts[1], parts[2]), RuleOS, true
case parts[0] == "roles" && len(parts) >= 2 && len(parts) <= 4:
return Selector{RoleTiers: parts[1:]}, RuleRole, true
case parts[0] == "country" && len(parts) == 2:
return Selector{Country: parts[1]}, RuleCountry, true
case parts[0] == "country" && len(parts) == 4 && parts[2] == "region":
return Selector{Country: parts[1], Region: parts[3]}, RuleCountryRegion, true
case parts[0] == "country" && len(parts) >= 5 && len(parts) <= 7 && parts[2] == "region":
return Selector{Country: parts[1], Region: parts[3], RoleTiers: parts[4:]}, RuleCountryRegRole, true
case parts[0] == "country" && len(parts) >= 4 && len(parts) <= 6 && parts[2] == "roles":
return Selector{Country: parts[1], RoleTiers: parts[3:]}, RuleCountryRole, true
}
return Selector{}, "", false
}
// osSelector reads os/<name>/<name><major>[.<minor>].yaml and its
// all_releases.yaml sibling.
func osSelector(osName, file string) Selector {
sel := Selector{OSName: osName}
if file == "all_releases" {
return sel
}
version := file
if len(file) >= len(osName) && strings.EqualFold(file[:len(osName)], osName) {
version = file[len(osName):]
}
version = strings.TrimLeft(version, "-_")
if version == "" {
return sel
}
major, minor, hasMinor := strings.Cut(version, ".")
sel.OSMajor = major
if hasMinor {
sel.OSMinor = minor
}
return sel
}
+132
View File
@@ -0,0 +1,132 @@
package impact
import (
"reflect"
"testing"
)
func TestHieradataSelector(t *testing.T) {
tests := []struct {
name string
path string
rule string
want Selector
}{
{"common yaml", "common.yaml", RuleCommon, Selector{All: true}},
{"common eyaml", "common.eyaml", RuleCommon, Selector{All: true}},
{"node", "nodes/web01.example.net.yaml", RuleNode, Selector{Certname: "web01.example.net"}},
{"virtual", "virtual/physical.yaml", RuleVirtual, Selector{Virtual: "physical"}},
{"role tier1", "roles/infra.yaml", RuleRole, Selector{RoleTiers: []string{"infra"}}},
{"role tier2", "roles/infra/puppet.yaml", RuleRole, Selector{RoleTiers: []string{"infra", "puppet"}}},
{"role tier3", "roles/infra/puppet/master.yaml", RuleRole,
Selector{RoleTiers: []string{"infra", "puppet", "master"}}},
{"role eyaml", "roles/apps/media.eyaml", RuleRole, Selector{RoleTiers: []string{"apps", "media"}}},
{"country", "country/au.yaml", RuleCountry, Selector{Country: "au"}},
{"country eyaml", "country/au.eyaml", RuleCountry, Selector{Country: "au"}},
{"country region", "country/au/region/syd1.yaml", RuleCountryRegion,
Selector{Country: "au", Region: "syd1"}},
{"country region tier1", "country/au/region/syd1/infra.yaml", RuleCountryRegRole,
Selector{Country: "au", Region: "syd1", RoleTiers: []string{"infra"}}},
{"country region tier3", "country/au/region/syd1/infra/puppet/master.eyaml", RuleCountryRegRole,
Selector{Country: "au", Region: "syd1", RoleTiers: []string{"infra", "puppet", "master"}}},
{"country roles tier1", "country/au/roles/infra.yaml", RuleCountryRole,
Selector{Country: "au", RoleTiers: []string{"infra"}}},
{"country roles tier3", "country/au/roles/infra/puppet/master.yaml", RuleCountryRole,
Selector{Country: "au", RoleTiers: []string{"infra", "puppet", "master"}}},
{"os all releases", "os/AlmaLinux/all_releases.yaml", RuleOS, Selector{OSName: "AlmaLinux"}},
{"os major", "os/AlmaLinux/AlmaLinux9.yaml", RuleOS, Selector{OSName: "AlmaLinux", OSMajor: "9"}},
{"os major minor", "os/AlmaLinux/AlmaLinux9.7.yaml", RuleOS,
Selector{OSName: "AlmaLinux", OSMajor: "9", OSMinor: "7"}},
{"os debian major", "os/Debian/Debian11.yaml", RuleOS, Selector{OSName: "Debian", OSMajor: "11"}},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
got, rule, ok := hieradataSelector(tc.path)
if !ok {
t.Fatalf("hieradataSelector(%q) not recognised", tc.path)
}
if rule != tc.rule {
t.Errorf("rule = %q, want %q", rule, tc.rule)
}
if !reflect.DeepEqual(got, tc.want) {
t.Errorf("selector = %#v, want %#v", got, tc.want)
}
})
}
}
func TestHieradataSelectorRejects(t *testing.T) {
for _, p := range []string{
"README.md",
"roles/a/b/c/d.yaml",
"country/au/region/syd1/a/b/c/d.yaml",
"os/AlmaLinux/AlmaLinux9.json",
"whatever/thing.yaml",
} {
if _, _, ok := hieradataSelector(p); ok {
t.Errorf("hieradataSelector(%q) unexpectedly recognised", p)
}
}
}
func TestManifestClasses(t *testing.T) {
tests := []struct {
module string
rest []string
want []string
}{
{"profiles", []string{"base.pp"}, []string{"profiles::base"}},
{"profiles", []string{"base", "motd.pp"}, []string{"profiles::base::motd"}},
{"roles", []string{"infra", "k8s", "node.pp"}, []string{"roles::infra::k8s::node"}},
{"jellyfin", []string{"init.pp"}, []string{"jellyfin", "jellyfin::init"}},
{"profiles", []string{"cloudinit", "init.pp"}, []string{"profiles::cloudinit", "profiles::cloudinit::init"}},
}
for _, tc := range tests {
if got := ManifestClasses(tc.module, tc.rest); !reflect.DeepEqual(got, tc.want) {
t.Errorf("ManifestClasses(%q, %v) = %v, want %v", tc.module, tc.rest, got, tc.want)
}
}
}
func TestClassifyPathRepoGlobal(t *testing.T) {
for _, p := range []string{"hiera.yaml", "environment.conf", "Puppetfile", "manifests/site.pp"} {
m := classifyPath(p, nil)
if len(m.targets) != 1 || !m.targets[0].sel.All {
t.Fatalf("classifyPath(%q) = %#v, want an all-hosts target", p, m)
}
if m.targets[0].rule != RuleRepoGlobal {
t.Errorf("classifyPath(%q) rule = %q", p, m.targets[0].rule)
}
}
}
func TestClassifyPathIgnoredAndUnknown(t *testing.T) {
for _, p := range []string{".woodpecker/test.yaml", "Gemfile", "doc/notes.md", "vendor/bundle/x.rb", "README.md"} {
if m := classifyPath(p, nil); !m.ignored {
t.Errorf("classifyPath(%q) should be ignored, got %#v", p, m)
}
}
for _, p := range []string{"random/thing.txt", "hieradata/nope/deep/deeper/x.yaml"} {
if m := classifyPath(p, nil); !m.unknown {
t.Errorf("classifyPath(%q) should be unknown, got %#v", p, m)
}
}
}
func TestClassifyPathPluginsync(t *testing.T) {
m := classifyPath("modules/libs/lib/facter/netbox.rb", nil)
if len(m.targets) != 1 || !m.targets[0].sel.All || m.targets[0].rule != RulePluginsync {
t.Fatalf("pluginsync path = %#v, want an all-hosts pluginsync target", m)
}
}
func TestClassifyPathManifest(t *testing.T) {
m := classifyPath("site/profiles/manifests/base/motd.pp", nil)
if len(m.classes) != 1 || m.classes[0].name != "profiles::base::motd" {
t.Fatalf("classes = %#v", m.classes)
}
}
+102
View File
@@ -0,0 +1,102 @@
package impact
import (
"encoding/json"
"fmt"
"io"
)
// RenderPlain writes one hostname per line, ready to pipe.
func RenderPlain(w io.Writer, r *Result) error {
for _, h := range r.Hosts {
if _, err := fmt.Fprintln(w, h.Certname); err != nil {
return err
}
}
return nil
}
// RenderJSON writes the structured report.
func RenderJSON(w io.Writer, r *Result) error {
enc := json.NewEncoder(w)
enc.SetIndent("", " ")
if r.Hosts == nil {
r.Hosts = []Host{}
}
if r.ChangedFiles == nil {
r.ChangedFiles = []string{}
}
if r.UnmappedFiles == nil {
r.UnmappedFiles = []string{}
}
return enc.Encode(r)
}
// RenderExplain writes the human-readable report: what each file mapped to,
// and for each host which changed files reach it.
func RenderExplain(w io.Writer, r *Result) error {
p := func(format string, args ...any) error {
_, err := fmt.Fprintf(w, format+"\n", args...)
return err
}
if err := p("%s...%s", r.Base, r.Branch); err != nil {
return err
}
if err := p("%d changed file(s), %d of %d host(s) affected", r.Summary.Files, r.Summary.Hosts, r.TotalHosts); err != nil {
return err
}
if r.Summary.AllHostsTriggered {
if err := p("the whole estate is affected"); err != nil {
return err
}
}
if r.Truncated {
if err := p("host list truncated by --limit"); err != nil {
return err
}
}
if err := section(w, "unmapped files (widened to all hosts)", r.UnmappedFiles); err != nil {
return err
}
if err := section(w, "ignored files (outside the Puppet modulepath)", r.IgnoredFiles); err != nil {
return err
}
if err := section(w, "orphan classes (nothing declares them and no hiera_include lists them)", r.OrphanClasses); err != nil {
return err
}
if len(r.Hosts) == 0 {
return p("\nno hosts affected")
}
if err := p("\nhosts:"); err != nil {
return err
}
for _, h := range r.Hosts {
if err := p(" %s", h.Certname); err != nil {
return err
}
for _, reason := range h.Reasons {
if err := p(" %s [%s] %s", reason.File, reason.Rule, reason.Detail); err != nil {
return err
}
}
}
return nil
}
func section(w io.Writer, title string, items []string) error {
if len(items) == 0 {
return nil
}
if _, err := fmt.Fprintf(w, "\n%s:\n", title); err != nil {
return err
}
for _, item := range items {
if _, err := fmt.Fprintf(w, " %s\n", item); err != nil {
return err
}
}
return nil
}
+139
View File
@@ -0,0 +1,139 @@
package impact
import (
"sort"
"strings"
)
// hit is one reason a host is affected, before it is attached to a file.
type hit struct {
rule string
detail string
}
// classResolver walks the include graph backwards from a changed class to the
// hosts that end up with it.
type classResolver struct {
idx *Index
nodes []Node
cache map[string]*resolution
}
type resolution struct {
hosts map[string][]hit
orphan bool
}
func newClassResolver(idx *Index, nodes []Node) *classResolver {
return &classResolver{idx: idx, nodes: nodes, cache: map[string]*resolution{}}
}
// resolve returns the hosts a class reaches. The walk runs to a fixed point
// over both routes a class can be applied by — a manifest declaring it, and a
// hieradata file listing it under hiera_include — and stops at role classes,
// which the ENC assigns directly.
func (r *classResolver) resolve(class string) *resolution {
if cached, ok := r.cache[class]; ok {
return cached
}
res := &resolution{hosts: map[string][]hit{}}
r.cache[class] = res
// via records how each class was reached, so --explain can print the chain.
via := map[string]string{class: class}
seen := map[string]bool{class: true}
queue := []string{class}
reached := false
for len(queue) > 0 {
current := queue[0]
queue = queue[1:]
if strings.HasPrefix(current, "roles::") {
sel := Selector{RoleExact: current}
detail := fmtDetail("role %s carries %s (%s)", current, class, via[current])
for _, certname := range sel.Select(r.nodes) {
res.hosts[certname] = append(res.hosts[certname], hit{rule: RuleRoleClass, detail: detail})
}
reached = true
}
if hosts, detail, ok := r.viaHiera(current); ok {
reached = true
for _, certname := range hosts {
res.hosts[certname] = append(res.hosts[certname],
hit{rule: RuleHieraInclude, detail: detail + " (" + via[current] + ")"})
}
}
for _, parent := range r.idx.IncludedBy(current) {
if seen[parent] {
continue
}
seen[parent] = true
via[parent] = via[current] + " <- " + parent
queue = append(queue, parent)
}
}
res.orphan = !reached
return res
}
// viaHiera applies a class that hieradata pulls in with hiera_include. The
// hierarchy position of the listing file decides which hosts get it, and
// hiera_exclude entries are subtracted because profiles::base declares
// ($hiera_include - $hiera_exclude).
func (r *classResolver) viaHiera(class string) ([]string, string, bool) {
includes := r.idx.HieraInclude(class)
if len(includes) == 0 {
return nil, "", false
}
included := map[string]bool{}
var sources []string
for _, file := range includes {
sel := hieraFileSelector(file)
for _, certname := range sel.Select(r.nodes) {
included[certname] = true
}
sources = append(sources, fmtDetail("%s [%s]", file, sel))
}
excludes := r.idx.HieraExclude(class)
var excluded []string
for _, file := range excludes {
sel := hieraFileSelector(file)
for _, certname := range sel.Select(r.nodes) {
if included[certname] {
delete(included, certname)
excluded = append(excluded, certname)
}
}
}
hosts := make([]string, 0, len(included))
for certname := range included {
hosts = append(hosts, certname)
}
sort.Strings(hosts)
detail := fmtDetail("hiera_include %s in %s", class, strings.Join(sources, ", "))
if len(excluded) > 0 {
detail += fmtDetail("; hiera_exclude in %s drops %d host(s)",
strings.Join(excludes, ", "), len(uniqueSorted(excluded)))
}
return hosts, detail, true
}
// hieraFileSelector maps a hieradata file back onto its hierarchy position. An
// unrecognised path widens to the whole estate rather than silently selecting
// nothing.
func hieraFileSelector(file string) Selector {
rel := strings.TrimPrefix(file, "hieradata/")
sel, _, ok := hieradataSelector(rel)
if !ok {
return Selector{All: true}
}
return sel
}
+125
View File
@@ -0,0 +1,125 @@
package impact
import (
"reflect"
"testing"
)
func testIndex(t *testing.T) *Index {
t.Helper()
idx, err := BuildIndex("testdata/repo")
if err != nil {
t.Fatalf("BuildIndex: %v", err)
}
return idx
}
func resolvedHosts(t *testing.T, class string) []string {
t.Helper()
res := newClassResolver(testIndex(t), fixtureNodes()).resolve(class)
hosts := make([]string, 0, len(res.hosts))
for certname := range res.hosts {
hosts = append(hosts, certname)
}
return uniqueSorted(hosts)
}
func TestResolveThroughIncludeChain(t *testing.T) {
// profiles::leaf <- profiles::mid <- profiles::base <- two role classes.
if got, want := resolvedHosts(t, "profiles::leaf"), []string{"k8s1", "k8s2", "web1"}; !reflect.DeepEqual(got, want) {
t.Errorf("profiles::leaf = %v, want %v", got, want)
}
}
func TestResolveContainAndClassResource(t *testing.T) {
for _, class := range []string{"profiles::contained", "profiles::viaclass"} {
if got, want := resolvedHosts(t, class), []string{"k8s1", "k8s2", "web1"}; !reflect.DeepEqual(got, want) {
t.Errorf("%s = %v, want %v", class, got, want)
}
}
}
func TestResolveDefinedTypeUsage(t *testing.T) {
if got, want := resolvedHosts(t, "profiles::defined::thing"), []string{"k8s1", "k8s2", "web1"}; !reflect.DeepEqual(got, want) {
t.Errorf("profiles::defined::thing = %v, want %v", got, want)
}
}
func TestResolveTerminatesOnCycle(t *testing.T) {
// profiles::cycle::a and ::b include each other; ::a is pulled in by
// profiles::base, so the walk must reach the roles and stop.
if got, want := resolvedHosts(t, "profiles::cycle::b"), []string{"k8s1", "k8s2", "web1"}; !reflect.DeepEqual(got, want) {
t.Errorf("profiles::cycle::b = %v, want %v", got, want)
}
}
func TestResolveViaHieraIncludeOnly(t *testing.T) {
// The jellyfin module is applied by hiera_include in a role yaml and by
// nothing in any manifest — a .pp-only search would call it orphaned.
if got, want := resolvedHosts(t, "jellyfin"), []string{"web1"}; !reflect.DeepEqual(got, want) {
t.Errorf("jellyfin = %v, want %v", got, want)
}
// One hop further in: jellyfin::install is only reachable through jellyfin.
if got, want := resolvedHosts(t, "jellyfin::install"), []string{"web1"}; !reflect.DeepEqual(got, want) {
t.Errorf("jellyfin::install = %v, want %v", got, want)
}
}
func TestResolveHieraIncludeFromVirtualHierarchy(t *testing.T) {
if got, want := resolvedHosts(t, "profiles::lldpd"), []string{"k8s1", "k8s2", "prox1"}; !reflect.DeepEqual(got, want) {
t.Errorf("profiles::lldpd = %v, want %v", got, want)
}
}
func TestResolveHonoursHieraExclude(t *testing.T) {
// common.yaml pulls basemod in for everyone; roles/infra/proxmox.yaml
// excludes it again, so the proxmox node drops out.
if got, want := resolvedHosts(t, "basemod"), []string{"bare1", "k8s1", "k8s2", "web1"}; !reflect.DeepEqual(got, want) {
t.Errorf("basemod = %v, want %v", got, want)
}
}
func TestResolveOrphanClass(t *testing.T) {
res := newClassResolver(testIndex(t), fixtureNodes()).resolve("orphanmod")
if !res.orphan {
t.Errorf("orphanmod should be orphaned, hosts = %v", res.hosts)
}
if len(res.hosts) != 0 {
t.Errorf("orphanmod reached %v", res.hosts)
}
}
func TestIndexIgnoresCommentedReferences(t *testing.T) {
if got := testIndex(t).IncludedBy("profiles::commented"); len(got) != 0 {
t.Errorf("a commented-out class name was indexed as a reference: %v", got)
}
}
func TestIndexAssetOwners(t *testing.T) {
idx := testIndex(t)
for _, asset := range []string{"profiles/leaf/x.erb", "profiles/leaf/y.txt"} {
if got, want := idx.AssetOwners(asset), []string{"profiles::leaf"}; !reflect.DeepEqual(got, want) {
t.Errorf("AssetOwners(%q) = %v, want %v", asset, got, want)
}
}
}
func TestIndexModuleClasses(t *testing.T) {
got := testIndex(t).ModuleClasses("jellyfin")
want := []string{"jellyfin", "jellyfin::init", "jellyfin::install"}
if !reflect.DeepEqual(got, want) {
t.Errorf("ModuleClasses(jellyfin) = %v, want %v", got, want)
}
}
func TestParseHieraListsRegexFallback(t *testing.T) {
// eyaml payloads a strict parser rejects still have to yield their lists.
broken := []byte("---\nhiera_include:\n - alpha\n - beta\nhiera_exclude:\n - gamma\nbad: [unclosed\n")
include, exclude := parseHieraLists(broken)
if !reflect.DeepEqual(include, []string{"alpha", "beta"}) {
t.Errorf("include = %v", include)
}
if !reflect.DeepEqual(exclude, []string{"gamma"}) {
t.Errorf("exclude = %v", exclude)
}
}
+123
View File
@@ -0,0 +1,123 @@
package impact
import (
"fmt"
"strings"
)
// Selector is a conjunction of fact constraints. Every field left at its zero
// value is unconstrained, so an empty Selector with All unset matches nothing
// and is treated as a programming error by the callers that build them.
type Selector struct {
// All matches the whole estate (hieradata/common.yaml, repo-global files).
All bool
// Certname pins one host (hieradata/nodes/<certname>.yaml).
Certname string
// RoleTiers is a prefix match on enc_role_tier1..3, so a two-tier
// hieradata file covers every role nested under it.
RoleTiers []string
// RoleExact matches enc_role verbatim, which is how a role class maps to
// hosts.
RoleExact string
Country string
Region string
Virtual string
OSName string
OSMajor string
OSMinor string
}
// Match reports whether the node satisfies every constraint set on s.
func (s Selector) Match(n Node) bool {
if s.All {
return true
}
if s.Certname != "" && s.Certname != n.Certname {
return false
}
if s.RoleExact != "" && s.RoleExact != n.Role {
return false
}
if s.Country != "" && s.Country != n.Country {
return false
}
if s.Region != "" && s.Region != n.Region {
return false
}
if s.Virtual != "" && s.Virtual != n.Virtual {
return false
}
if s.OSName != "" && !strings.EqualFold(s.OSName, n.OSName) {
return false
}
if s.OSMajor != "" && s.OSMajor != n.OSMajor {
return false
}
if s.OSMinor != "" && s.OSMinor != n.OSMinor {
return false
}
if len(s.RoleTiers) > 0 {
tiers := n.Tiers()
if len(tiers) < len(s.RoleTiers) {
return false
}
for i, want := range s.RoleTiers {
if tiers[i] != want {
return false
}
}
}
return true
}
// Select returns the certnames of every matching node, sorted.
func (s Selector) Select(nodes []Node) []string {
var out []string
for _, n := range nodes {
if s.Match(n) {
out = append(out, n.Certname)
}
}
return out
}
// String renders the selector the way --explain prints it.
func (s Selector) String() string {
if s.All {
return "all hosts"
}
var parts []string
add := func(k, v string) {
if v != "" {
parts = append(parts, k+"="+v)
}
}
add("certname", s.Certname)
add("enc_role", s.RoleExact)
if len(s.RoleTiers) > 0 {
parts = append(parts, "role-prefix="+strings.Join(s.RoleTiers, "::"))
}
add("country", s.Country)
add("region", s.Region)
add("virtual", s.Virtual)
add("os.name", s.OSName)
add("os.release.major", s.OSMajor)
add("os.release.minor", s.OSMinor)
if len(parts) == 0 {
return "nothing"
}
return strings.Join(parts, " ")
}
// target is one resolved "this file reaches these hosts, for this reason".
type target struct {
sel Selector
rule string
detail string
}
func allTarget(rule, detail string) target {
return target{sel: Selector{All: true}, rule: rule, detail: detail}
}
func fmtDetail(format string, args ...any) string { return fmt.Sprintf(format, args...) }
+88
View File
@@ -0,0 +1,88 @@
package impact
import (
"reflect"
"testing"
)
// fixtureNodes mirrors the shape of the real estate: one country, two regions,
// physical and virtual hosts, and a mix of OS releases.
func fixtureNodes() []Node {
return []Node{
{Certname: "web1", Role: "roles::apps::media::jellyfin",
Tier1: "apps", Tier2: "media", Tier3: "jellyfin",
Country: "au", Region: "syd1", Virtual: "lxc",
OSName: "AlmaLinux", OSMajor: "9", OSMinor: "7"},
{Certname: "k8s1", Role: "roles::infra::k8s::node",
Tier1: "infra", Tier2: "k8s", Tier3: "node",
Country: "au", Region: "syd1", Virtual: "physical",
OSName: "AlmaLinux", OSMajor: "9", OSMinor: "8"},
{Certname: "k8s2", Role: "roles::infra::k8s::node",
Tier1: "infra", Tier2: "k8s", Tier3: "node",
Country: "au", Region: "drw1", Virtual: "physical",
OSName: "Debian", OSMajor: "12", OSMinor: "1"},
{Certname: "prox1", Role: "roles::infra::proxmox::node",
Tier1: "infra", Tier2: "proxmox", Tier3: "node",
Country: "nz", Region: "akl1", Virtual: "physical",
OSName: "AlmaLinux", OSMajor: "8", OSMinor: "10"},
{Certname: "bare1", Role: "roles::base", Tier1: "base",
Country: "au", Region: "syd1", Virtual: "lxc",
OSName: "AlmaLinux", OSMajor: "9", OSMinor: "7"},
}
}
func TestSelectorSelect(t *testing.T) {
nodes := fixtureNodes()
tests := []struct {
name string
sel Selector
want []string
}{
{"all", Selector{All: true}, []string{"web1", "k8s1", "k8s2", "prox1", "bare1"}},
{"certname", Selector{Certname: "k8s2"}, []string{"k8s2"}},
{"role exact", Selector{RoleExact: "roles::infra::k8s::node"}, []string{"k8s1", "k8s2"}},
{"tier1 prefix", Selector{RoleTiers: []string{"infra"}}, []string{"k8s1", "k8s2", "prox1"}},
{"tier2 prefix", Selector{RoleTiers: []string{"infra", "k8s"}}, []string{"k8s1", "k8s2"}},
{"tier3 exact", Selector{RoleTiers: []string{"apps", "media", "jellyfin"}}, []string{"web1"}},
{"tier1 only node", Selector{RoleTiers: []string{"base"}}, []string{"bare1"}},
{"tier prefix longer than node", Selector{RoleTiers: []string{"base", "x"}}, nil},
{"country", Selector{Country: "au"}, []string{"web1", "k8s1", "k8s2", "bare1"}},
{"country region", Selector{Country: "au", Region: "syd1"}, []string{"web1", "k8s1", "bare1"}},
{"country region role", Selector{Country: "au", Region: "syd1", RoleTiers: []string{"infra"}},
[]string{"k8s1"}},
{"country role", Selector{Country: "au", RoleTiers: []string{"infra", "k8s"}}, []string{"k8s1", "k8s2"}},
{"virtual", Selector{Virtual: "physical"}, []string{"k8s1", "k8s2", "prox1"}},
{"os name", Selector{OSName: "AlmaLinux"}, []string{"web1", "k8s1", "prox1", "bare1"}},
{"os major", Selector{OSName: "AlmaLinux", OSMajor: "9"}, []string{"web1", "k8s1", "bare1"}},
{"os major minor", Selector{OSName: "AlmaLinux", OSMajor: "9", OSMinor: "7"},
[]string{"web1", "bare1"}},
{"os name case insensitive", Selector{OSName: "almalinux", OSMajor: "8"}, []string{"prox1"}},
{"os other name", Selector{OSName: "Debian", OSMajor: "12"}, []string{"k8s2"}},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
if got := tc.sel.Select(nodes); !reflect.DeepEqual(got, tc.want) {
t.Errorf("Select() = %v, want %v", got, tc.want)
}
})
}
}
func TestSelectorString(t *testing.T) {
if got := (Selector{All: true}).String(); got != "all hosts" {
t.Errorf("all selector = %q", got)
}
got := Selector{Country: "au", Region: "syd1", RoleTiers: []string{"infra", "puppet"}}.String()
want := "role-prefix=infra::puppet country=au region=syd1"
if got != want {
t.Errorf("String() = %q, want %q", got, want)
}
}
func TestNodeTiersTruncatesAtGap(t *testing.T) {
n := Node{Tier1: "base"}
if got := n.Tiers(); !reflect.DeepEqual(got, []string{"base"}) {
t.Errorf("Tiers() = %v", got)
}
}
+3
View File
@@ -0,0 +1,3 @@
---
hiera_include:
- basemod
@@ -0,0 +1,3 @@
---
hiera_include:
- jellyfin
@@ -0,0 +1,3 @@
---
hiera_exclude:
- basemod
@@ -0,0 +1,3 @@
---
hiera_include:
- profiles::lldpd
@@ -0,0 +1,2 @@
class basemod {
}
@@ -0,0 +1,3 @@
class jellyfin {
include jellyfin::install
}
@@ -0,0 +1,2 @@
class jellyfin::install {
}
@@ -0,0 +1,2 @@
class orphanmod {
}
@@ -0,0 +1 @@
not a manifest
@@ -0,0 +1 @@
leaf file
@@ -0,0 +1,8 @@
class profiles::base {
# a comment naming profiles::commented must not count as a reference
include profiles::mid
contain profiles::contained
class { 'profiles::viaclass': }
include profiles::cycle::a
profiles::defined::thing { 'x': }
}
@@ -0,0 +1,2 @@
class profiles::contained {
}
@@ -0,0 +1,3 @@
class profiles::cycle::a {
include profiles::cycle::b
}
@@ -0,0 +1,3 @@
class profiles::cycle::b {
include profiles::cycle::a
}
@@ -0,0 +1,2 @@
define profiles::defined::thing () {
}
@@ -0,0 +1,6 @@
class profiles::leaf {
file { '/tmp/leaf':
content => template('profiles/leaf/x.erb'),
source => 'puppet:///modules/profiles/leaf/y.txt',
}
}
@@ -0,0 +1,2 @@
class profiles::lldpd {
}
@@ -0,0 +1,3 @@
class profiles::mid {
include profiles::leaf
}
@@ -0,0 +1,2 @@
class profiles::viaclass {
}
@@ -0,0 +1 @@
leaf template
@@ -0,0 +1,3 @@
class roles::apps::media::jellyfin {
include profiles::base
}
@@ -0,0 +1,3 @@
class roles::infra::k8s::node {
include profiles::base
}
+42
View File
@@ -128,3 +128,45 @@ func snippet(body []byte) string {
}
return s
}
// FactRecord is one (certname, fact) pair from an estate-wide fact query.
type FactRecord struct {
Certname string `json:"certname"`
Name string `json:"name"`
Value any `json:"value"`
}
// FactsByName returns the named facts for every node pdbmux knows about, in a
// single query. Tools that evaluate selectors over the whole estate want the
// table once rather than a request per host.
func (c *Client) FactsByName(ctx context.Context, names []string) ([]FactRecord, error) {
if len(names) == 0 {
return nil, nil
}
clauses := make([]any, 0, len(names)+1)
clauses = append(clauses, "or")
for _, n := range names {
clauses = append(clauses, []any{"=", "name", n})
}
query := []any{"extract", []string{"certname", "name", "value"}, clauses}
encoded, err := json.Marshal(query)
if err != nil {
return nil, fmt.Errorf("encoding fact query: %w", err)
}
path := "/pdb/query/v4/facts?query=" + url.QueryEscape(string(encoded))
body, status, err := c.get(ctx, path)
if err != nil {
return nil, err
}
if status != http.StatusOK {
return nil, fmt.Errorf("pdbmux returned %d for fact query: %s", status, snippet(body))
}
var records []FactRecord
if err := json.Unmarshal(body, &records); err != nil {
return nil, fmt.Errorf("parsing fact query response: %w", err)
}
return records, nil
}
+55
View File
@@ -157,3 +157,58 @@ func TestFactsContextCancelled(t *testing.T) {
t.Fatal("expected a timeout error")
}
}
func TestFactsByName(t *testing.T) {
var gotQuery string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotQuery = r.URL.Query().Get("query")
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`[
{"certname":"a.example.net","name":"country","value":"au"},
{"certname":"a.example.net","name":"os","value":{"name":"AlmaLinux","release":{"major":"9","minor":"7"}}}
]`))
}))
defer srv.Close()
records, err := New(srv.URL, 5*time.Second).FactsByName(context.Background(), []string{"country", "os"})
if err != nil {
t.Fatalf("FactsByName: %v", err)
}
if gotQuery != `["extract",["certname","name","value"],["or",["=","name","country"],["=","name","os"]]]` {
t.Errorf("query = %s", gotQuery)
}
if len(records) != 2 {
t.Fatalf("records = %#v", records)
}
if records[0].Certname != "a.example.net" || records[0].Value != "au" {
t.Errorf("first record = %#v", records[0])
}
os, ok := records[1].Value.(map[string]any)
if !ok || os["name"] != "AlmaLinux" {
t.Errorf("structured os fact lost: %#v", records[1].Value)
}
}
func TestFactsByNameEmptyNames(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {
t.Error("no request should be made for an empty fact list")
}))
defer srv.Close()
records, err := New(srv.URL, 5*time.Second).FactsByName(context.Background(), nil)
if err != nil || records != nil {
t.Errorf("FactsByName(nil) = %v, %v", records, err)
}
}
func TestFactsByNameServerError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
_, _ = w.Write([]byte("boom"))
}))
defer srv.Close()
if _, err := New(srv.URL, 5*time.Second).FactsByName(context.Background(), []string{"country"}); err == nil {
t.Fatal("expected an error for a 500")
}
}
+20
View File
@@ -30,6 +30,13 @@ contents:
owner: root
group: root
- src: dist/catalog-impact
dst: /usr/bin/catalog-impact
file_info:
mode: 0755
owner: root
group: root
# Shell completions (generated by scripts/build-rpm.sh before packaging).
- src: dist/completions/catalog-diff.bash
dst: /usr/share/bash-completion/completions/catalog-diff
@@ -43,3 +50,16 @@ contents:
dst: /usr/share/fish/vendor_completions.d/catalog-diff.fish
file_info:
mode: 0644
- src: dist/completions/catalog-impact.bash
dst: /usr/share/bash-completion/completions/catalog-impact
file_info:
mode: 0644
- src: dist/completions/_catalog-impact
dst: /usr/share/zsh/site-functions/_catalog-impact
file_info:
mode: 0644
- src: dist/completions/catalog-impact.fish
dst: /usr/share/fish/vendor_completions.d/catalog-impact.fish
file_info:
mode: 0644
+2 -2
View File
@@ -12,7 +12,7 @@ cd "${ROOT_DIR}"
VERSION="${1:-${CI_COMMIT_TAG:-0.0.0-dev}}"
VERSION="${VERSION#v}" # strip a leading v
PACKAGE="puppet-catalog-tools"
BINARIES=(catalog-diff)
BINARIES=(catalog-diff catalog-impact)
DIST="dist"
for b in "${BINARIES[@]}"; do
@@ -37,7 +37,7 @@ export PACKAGE_VERSION="${VERSION}"
export PACKAGE_RELEASE="1"
export PACKAGE_ARCH="amd64"
export PACKAGE_PLATFORM="linux"
export PACKAGE_DESCRIPTION="Puppet catalog tooling: catalog-diff compiles a host's catalog in two environments and diffs them"
export PACKAGE_DESCRIPTION="Puppet catalog tooling: catalog-diff compiles a host's catalog in two environments and diffs them; catalog-impact lists the hosts a control repo change affects"
export PACKAGE_MAINTAINER="Ben Vincent <ben@unkin.net>"
export PACKAGE_HOMEPAGE="https://git.unkin.net/unkin/puppet-catalog-tools"
export PACKAGE_LICENSE="MIT"