add the catalog-impact binary
Work out which hosts a puppet-prod change can affect, so a catalog test can
fan out over exactly those hosts instead of the whole estate.
- add cmd/catalog-impact and internal/impact
- map hieradata paths onto fact selectors from the hiera hierarchy
- trace manifest classes back to role classes through include, contain,
class {} and hiera_include, to a fixed point
- fetch the estate fact table from pdbmux in one query via FactsByName
- widen to all hosts, loudly, for any path that matches no rule
- ship the binary and its completions in the RPM
This commit is contained in:
@@ -9,14 +9,20 @@ shared code lives in `internal/`.
|
||||
- **`catalog-diff`** — compiles a host's catalog twice (base environment and
|
||||
the environment built from the branch under test) over the puppetserver API
|
||||
and diffs the result.
|
||||
- **`catalog-impact`** — maps a control repo change onto the hosts it can
|
||||
affect, by reading hieradata paths as hierarchy positions and tracing
|
||||
manifest classes backwards to role classes.
|
||||
|
||||
## Structure
|
||||
|
||||
```
|
||||
cmd/catalog-diff/ # the CLI (flags, exit codes, orchestration)
|
||||
cmd/catalog-impact/ # the CLI (flags, output modes)
|
||||
internal/catalog/ # catalog types, normalisation, diff engine, renderers,
|
||||
# branch.go (BranchToEnvironment)
|
||||
internal/puppetserver/ # mTLS client for POST /puppet/v4/catalog + /status/v1/simple
|
||||
internal/impact/ # path -> selector rules, repo index, reverse lookup,
|
||||
# estate fact table, renderers
|
||||
internal/pdbmux/ # unauthenticated fact client for pdbmux
|
||||
internal/config/ # XDG config file + env var loading
|
||||
packaging/nfpm.yaml # RPM spec (envsubst-templated)
|
||||
@@ -36,6 +42,9 @@ Makefile and to `scripts/build-rpm.sh`, and add its files to
|
||||
certname must be allowed in the puppetserver `auth.conf`.
|
||||
- Facts: `GET /pdb/query/v4/nodes/<certname>/facts` on pdbmux, falling back to
|
||||
`/pdb/query/v4/factsets/<certname>`. No auth, no client cert.
|
||||
- Estate-wide facts: one AST query against `/pdb/query/v4/facts`
|
||||
(`pdbmux.Client.FactsByName`). The estate is ~150 nodes, so `catalog-impact`
|
||||
pulls the whole table once and filters in Go rather than querying per host.
|
||||
- The two services have different auth models and stay separate clients.
|
||||
|
||||
## Gotchas
|
||||
@@ -52,6 +61,19 @@ Makefile and to `scripts/build-rpm.sh`, and add its files to
|
||||
File titles keyed by `path`, datatype-only changes suppressed). Removing one
|
||||
floods the output with noise.
|
||||
|
||||
## catalog-impact gotchas
|
||||
|
||||
- The reverse lookup must search hieradata as well as manifests. `profiles::base`
|
||||
declares `($hiera_include - $hiera_exclude).include`, and some modules — the
|
||||
`jellyfin` one, for instance — are applied *only* by a `hiera_include` list in
|
||||
a role yaml. A manifest-only search reports them as orphans.
|
||||
- `hieradata/roles/<t1>/<t2>.yaml` is a **prefix** match on the role tiers, not
|
||||
an exact one. `site/roles/manifests/.../<t3>.pp` is exact.
|
||||
- The bias is to over-include: an unmapped path widens to the whole estate and
|
||||
says so on stderr. Never turn a widening into a silent empty result.
|
||||
- The repo index comes from the `--repo` checkout, not from `--branch`. CI runs
|
||||
on the branch checkout, so the two agree there.
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# All shipped binaries. Every tool is a separate main package under cmd/, so
|
||||
# adding one is a matter of adding its name here.
|
||||
BINARIES := catalog-diff
|
||||
BINARIES := catalog-diff catalog-impact
|
||||
DIST := dist
|
||||
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
|
||||
GOFLAGS := -ldflags="-s -w -X main.version=$(VERSION)"
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
# puppet-catalog-tools
|
||||
|
||||
CLI tooling for the Puppet estate. Currently one binary:
|
||||
CLI tooling for the Puppet estate:
|
||||
|
||||
- **`catalog-diff`** — compiles a host's catalog in a base environment and in
|
||||
the environment built from the branch under test, then diffs the two.
|
||||
- **`catalog-impact`** — works out which hosts a control repo change can
|
||||
affect, and prints their hostnames.
|
||||
|
||||
Catalogs are compiled over the puppetserver API (`POST /puppet/v4/catalog`)
|
||||
with facts read from `pdbmux`. Nothing is compiled locally and nothing is
|
||||
@@ -14,9 +16,11 @@ persisted on the server.
|
||||
```bash
|
||||
dnf install puppet-catalog-tools # from the artifactapi rpm-internal repo
|
||||
# or
|
||||
make build # -> dist/catalog-diff
|
||||
make build # -> dist/
|
||||
```
|
||||
|
||||
# catalog-diff
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
@@ -119,6 +123,122 @@ as sensitive are replaced with a digest so a change is still visible without
|
||||
the plaintext reaching stdout or a CI log; anything Puppet does not mark stays
|
||||
in the clear.
|
||||
|
||||
# catalog-impact
|
||||
|
||||
Given a change to the `puppet-prod` control repo, `catalog-impact` works out
|
||||
which files it touches and which hosts those files affect, then prints the
|
||||
affected hostnames — one per line, sorted and deduplicated, ready to pipe into
|
||||
a fan-out that compiles each host's catalog.
|
||||
|
||||
```bash
|
||||
catalog-impact -r ~/src/puppet-prod # current branch vs origin/develop
|
||||
catalog-impact -r ~/src/puppet-prod -b benvin/motd --explain
|
||||
catalog-impact --files hieradata/common.yaml --json
|
||||
git diff --name-only origin/develop... | paste -sd, | xargs catalog-impact --files
|
||||
```
|
||||
|
||||
| Flag | Description |
|
||||
|---|---|
|
||||
| `-r, --repo` | path to a puppet-prod checkout (default `.`) |
|
||||
| `-b, --branch` | branch under test (default: the repo's current branch) |
|
||||
| `--base` | base ref to diff against (default `origin/develop`) |
|
||||
| `--files` | explicit changed-file list, bypassing git entirely |
|
||||
| `--pdb` | pdbmux base URL (default `https://pdbmux.k8s.syd1.au.unkin.net`) |
|
||||
| `--json` | structured output with per-host reasons |
|
||||
| `--explain` | human-readable: for each host, which changed files hit it and why |
|
||||
| `--limit` | cap the number of hosts emitted (0 = no cap) |
|
||||
| `--timeout` | pdbmux request timeout (default 1m) |
|
||||
|
||||
`--pdb` defaults to the shared configuration described under
|
||||
[Configuration](#configuration), so `pdb_url` and `CATALOG_DIFF_PDB` apply to
|
||||
both binaries. No credentials are needed.
|
||||
|
||||
Facts for the whole estate are fetched from pdbmux in a single query and every
|
||||
selector is evaluated in memory, so the cost is one HTTP request regardless of
|
||||
how many files changed.
|
||||
|
||||
## How a changed file becomes a host list
|
||||
|
||||
Classification runs through an exec ENC and `manifests/site.pp` is empty, so a
|
||||
node's `enc_role` fact is its classification. Everything below keys off that
|
||||
and the facts `hiera.yaml` interpolates.
|
||||
|
||||
**Hieradata paths become fact selectors**, mirroring the hierarchy:
|
||||
|
||||
| Path | Selects |
|
||||
|---|---|
|
||||
| `hieradata/nodes/<certname>.yaml` | that node |
|
||||
| `hieradata/roles/<t1>[/<t2>[/<t3>]]` | role-tier **prefix** — `roles/infra/puppet.yaml` hits every `roles::infra::puppet::*` host |
|
||||
| `hieradata/country/<c>/region/<r>/<t1>[/...]` | country and region and role-tier prefix |
|
||||
| `hieradata/country/<c>/roles/<t1>[/...]` | country and role-tier prefix |
|
||||
| `hieradata/country/<c>/region/<r>` | country and region |
|
||||
| `hieradata/country/<c>` | country |
|
||||
| `hieradata/virtual/<v>.yaml` | `virtual` == v |
|
||||
| `hieradata/os/<name>/<name><major>[.<minor>].yaml` | `os.name` and `os.release.major` (and minor when present) |
|
||||
| `hieradata/os/<name>/all_releases.yaml` | every host with that `os.name` |
|
||||
| `hieradata/common.{yaml,eyaml}` | every host |
|
||||
|
||||
**Repo-global files select every host**: `hiera.yaml`, `environment.conf`,
|
||||
`manifests/site.pp`, `Puppetfile`. The Puppetfile pins external modules whose
|
||||
code is not in this repo, so there is nothing to trace — mapping it to the
|
||||
whole estate is the deliberate conservative choice. `lib/` under any module is
|
||||
pluginsynced to every agent and is treated the same way.
|
||||
|
||||
**Code paths become classes**, then the classes are traced backwards:
|
||||
|
||||
- `site/roles/manifests/<t1>/<t2>/<t3>.pp` is class
|
||||
`roles::<t1>::<t2>::<t3>`, which maps to hosts by exact `enc_role`. A role
|
||||
class is a leaf — nothing includes it, the ENC assigns it.
|
||||
- `site/profiles/manifests/**.pp` and `modules/<mod>/manifests/**.pp` become
|
||||
their autoloaded class names (`init.pp` counts as both the namespace and
|
||||
`<namespace>::init`) and need a reverse lookup.
|
||||
- `modules/<mod>/{templates,files}/**` and `site/profiles/{templates,files}/**`
|
||||
are attributed to the classes whose manifests reference them through
|
||||
`template()`, `epp()` or `puppet:///modules/...`. An unreferenced asset falls
|
||||
back to every class its module defines.
|
||||
|
||||
## The reverse lookup
|
||||
|
||||
Finding what applies a class means searching two places, not one:
|
||||
|
||||
1. `.pp` files under `site/` and `modules/` for `include`, `contain`,
|
||||
`class { '...': }`, capitalised resource references and defined-type usage.
|
||||
2. `hieradata/**/*.{yaml,eyaml}` for the class as a list item under
|
||||
`hiera_include:`. `profiles::base` declares
|
||||
`($hiera_include - $hiera_exclude).include` with a `unique` merge across the
|
||||
whole hierarchy, and it is included by essentially every role.
|
||||
|
||||
The second is not an edge case. `hieradata/roles/apps/media/jellyfin.yaml` is
|
||||
the only thing that applies the `jellyfin` module — there is no `include
|
||||
jellyfin` anywhere — so a manifest-only search reports it as orphaned and finds
|
||||
no hosts. A `hiera_include` hit is resolved through the hieradata path rules
|
||||
above: the position of the listing file in the hierarchy decides which hosts
|
||||
get the class. `hiera_exclude` entries are subtracted the same way.
|
||||
|
||||
The walk runs to a fixed point, because profiles include profiles several
|
||||
layers deep — `profiles::base` alone pulls in around twenty-five. Cycles are
|
||||
visited once.
|
||||
|
||||
## Bias and limits
|
||||
|
||||
A missed host means an unvalidated catalog, so every ambiguity widens:
|
||||
|
||||
- a path matching no rule is reported on stderr and mapped to every host;
|
||||
- an asset no manifest references widens to its whole module;
|
||||
- an unrecognised hieradata path in a `hiera_include` hit widens to every host.
|
||||
|
||||
Paths outside the Puppet modulepath — `.woodpecker/`, `doc/`, `spec/`,
|
||||
`vendor/`, `Gemfile`, `*.md` — cannot reach a catalog and are skipped.
|
||||
|
||||
Two things the tool cannot see:
|
||||
|
||||
- A class nothing declares and no `hiera_include` lists reaches no hosts. That
|
||||
is reported as an orphan on stderr rather than silently dropped.
|
||||
- The index is built from the checkout at `--repo`, not from `--branch`. In CI
|
||||
the checkout is the branch under test; pointing `--branch` at a different ref
|
||||
than the working tree means classes the branch adds look orphaned.
|
||||
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
// Command catalog-impact maps a change in the Puppet control repo onto the
|
||||
// hosts whose catalogs it can alter.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"git.unkin.net/unkin/puppet-catalog-tools/internal/config"
|
||||
"git.unkin.net/unkin/puppet-catalog-tools/internal/impact"
|
||||
"git.unkin.net/unkin/puppet-catalog-tools/internal/pdbmux"
|
||||
)
|
||||
|
||||
var version = "dev"
|
||||
|
||||
// Exit codes, also documented in --help and the README.
|
||||
const (
|
||||
exitOK = 0 // analysis completed
|
||||
exitUsage = 1 // transport, git or usage error
|
||||
)
|
||||
|
||||
const longHelp = `Work out which hosts a change to the Puppet control repo can affect, and print
|
||||
their hostnames one per line.
|
||||
|
||||
Changed files are mapped onto hosts two ways. A hieradata path is read as its
|
||||
position in the hiera hierarchy and turned into a fact selector. A manifest,
|
||||
template or file is turned into the classes it defines, and those classes are
|
||||
traced backwards — through include, contain and class {} in manifests, and
|
||||
through hiera_include in hieradata — until only role classes and hierarchy
|
||||
positions remain.
|
||||
|
||||
The bias is to over-include: an unrecognised path widens to the whole estate
|
||||
rather than silently returning nothing.`
|
||||
|
||||
type options struct {
|
||||
repo string
|
||||
branch string
|
||||
base string
|
||||
files []string
|
||||
pdb string
|
||||
jsonOut bool
|
||||
explain bool
|
||||
limit int
|
||||
timeout time.Duration
|
||||
}
|
||||
|
||||
func main() {
|
||||
cfg, err := config.Load()
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "config error:", err)
|
||||
os.Exit(exitUsage)
|
||||
}
|
||||
|
||||
opts := &options{}
|
||||
root := &cobra.Command{
|
||||
Use: "catalog-impact",
|
||||
Short: "List the hosts a Puppet control repo change affects",
|
||||
Long: longHelp,
|
||||
Args: cobra.NoArgs,
|
||||
Version: version,
|
||||
SilenceUsage: true,
|
||||
SilenceErrors: true,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
return run(cmd.Context(), cmd.OutOrStdout(), opts)
|
||||
},
|
||||
}
|
||||
|
||||
f := root.Flags()
|
||||
f.StringVarP(&opts.repo, "repo", "r", ".", "path to a puppet-prod checkout")
|
||||
f.StringVarP(&opts.branch, "branch", "b", "", "branch under test (default: the repo's current branch)")
|
||||
f.StringVar(&opts.base, "base", "origin/develop", "base ref to diff against")
|
||||
f.StringSliceVar(&opts.files, "files", nil, "explicit changed-file list, bypassing git")
|
||||
f.StringVar(&opts.pdb, "pdb", cfg.PDBURL, "pdbmux base URL")
|
||||
f.BoolVar(&opts.jsonOut, "json", false, "emit structured output with per-host reasons")
|
||||
f.BoolVar(&opts.explain, "explain", false, "human-readable: for each host, which changed files hit it and why")
|
||||
f.IntVar(&opts.limit, "limit", 0, "cap the number of hosts emitted (0 = no cap)")
|
||||
f.DurationVar(&opts.timeout, "timeout", time.Minute, "pdbmux request timeout")
|
||||
|
||||
if err := root.Execute(); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "error:", err)
|
||||
os.Exit(exitUsage)
|
||||
}
|
||||
os.Exit(exitOK)
|
||||
}
|
||||
|
||||
func run(ctx context.Context, out io.Writer, opts *options) error {
|
||||
if opts.jsonOut && opts.explain {
|
||||
return fmt.Errorf("--json and --explain are mutually exclusive")
|
||||
}
|
||||
if opts.limit < 0 {
|
||||
return fmt.Errorf("--limit must not be negative")
|
||||
}
|
||||
|
||||
branch := opts.branch
|
||||
files := opts.files
|
||||
if len(files) == 0 {
|
||||
var err error
|
||||
if branch == "" {
|
||||
if branch, err = impact.CurrentBranch(opts.repo); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if files, err = impact.ChangedFiles(opts.repo, opts.base, branch); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if branch == "" {
|
||||
branch = "(explicit file list)"
|
||||
}
|
||||
|
||||
idx, err := impact.BuildIndex(opts.repo)
|
||||
if err != nil {
|
||||
return fmt.Errorf("indexing %s: %w", opts.repo, err)
|
||||
}
|
||||
|
||||
nodes, err := impact.LoadNodes(ctx, pdbmux.New(opts.pdb, opts.timeout))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
analyzer := &impact.Analyzer{Index: idx, Nodes: nodes}
|
||||
result := analyzer.Analyze(opts.base, branch, files, opts.limit)
|
||||
|
||||
switch {
|
||||
case opts.jsonOut:
|
||||
return impact.RenderJSON(out, result)
|
||||
case opts.explain:
|
||||
return impact.RenderExplain(out, result)
|
||||
}
|
||||
|
||||
warn(result)
|
||||
return impact.RenderPlain(out, result)
|
||||
}
|
||||
|
||||
// warn surfaces the widenings and dead ends on stderr so a piped host list
|
||||
// stays clean.
|
||||
func warn(r *impact.Result) {
|
||||
for _, file := range r.UnmappedFiles {
|
||||
fmt.Fprintf(os.Stderr, "warning: %s matches no mapping rule; every host is included\n", file)
|
||||
}
|
||||
for _, class := range r.OrphanClasses {
|
||||
fmt.Fprintf(os.Stderr, "warning: class %s is declared by nothing and listed in no hiera_include; it reaches no hosts\n", class)
|
||||
}
|
||||
if r.Truncated {
|
||||
fmt.Fprintf(os.Stderr, "warning: %d hosts affected, output capped at %d by --limit\n", r.Summary.Hosts, len(r.Hosts))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
const estateBody = `[
|
||||
{"certname":"web1.example.net","name":"enc_role","value":"roles::apps::media::jellyfin"},
|
||||
{"certname":"web1.example.net","name":"enc_role_tier1","value":"apps"},
|
||||
{"certname":"web1.example.net","name":"enc_role_tier2","value":"media"},
|
||||
{"certname":"web1.example.net","name":"enc_role_tier3","value":"jellyfin"},
|
||||
{"certname":"web1.example.net","name":"country","value":"au"},
|
||||
{"certname":"web1.example.net","name":"region","value":"syd1"},
|
||||
{"certname":"web1.example.net","name":"virtual","value":"lxc"},
|
||||
{"certname":"web1.example.net","name":"os","value":{"name":"AlmaLinux","release":{"major":"9","minor":"7"}}},
|
||||
{"certname":"k8s1.example.net","name":"enc_role","value":"roles::infra::k8s::node"},
|
||||
{"certname":"k8s1.example.net","name":"enc_role_tier1","value":"infra"},
|
||||
{"certname":"k8s1.example.net","name":"enc_role_tier2","value":"k8s"},
|
||||
{"certname":"k8s1.example.net","name":"enc_role_tier3","value":"node"},
|
||||
{"certname":"k8s1.example.net","name":"country","value":"au"},
|
||||
{"certname":"k8s1.example.net","name":"region","value":"syd1"},
|
||||
{"certname":"k8s1.example.net","name":"virtual","value":"physical"},
|
||||
{"certname":"k8s1.example.net","name":"os","value":{"name":"AlmaLinux","release":{"major":"9","minor":"8"}}}
|
||||
]`
|
||||
|
||||
func fakePDB(t *testing.T) string {
|
||||
t.Helper()
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(estateBody))
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return srv.URL
|
||||
}
|
||||
|
||||
func TestRunWithExplicitFileList(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
opts := &options{
|
||||
repo: "../../internal/impact/testdata/repo",
|
||||
base: "origin/develop",
|
||||
files: []string{"modules/jellyfin/manifests/install.pp"},
|
||||
pdb: fakePDB(t),
|
||||
timeout: 5 * time.Second,
|
||||
}
|
||||
if err := run(context.Background(), &out, opts); err != nil {
|
||||
t.Fatalf("run: %v", err)
|
||||
}
|
||||
if got, want := out.String(), "web1.example.net\n"; got != want {
|
||||
t.Errorf("output = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunExplain(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
opts := &options{
|
||||
repo: "../../internal/impact/testdata/repo",
|
||||
base: "origin/develop",
|
||||
files: []string{"hieradata/common.yaml"},
|
||||
pdb: fakePDB(t),
|
||||
explain: true,
|
||||
timeout: 5 * time.Second,
|
||||
}
|
||||
if err := run(context.Background(), &out, opts); err != nil {
|
||||
t.Fatalf("run: %v", err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "the whole estate is affected") {
|
||||
t.Errorf("explain output = %q", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunRejectsConflictingOutputFlags(t *testing.T) {
|
||||
opts := &options{jsonOut: true, explain: true}
|
||||
if err := run(context.Background(), &bytes.Buffer{}, opts); err == nil {
|
||||
t.Fatal("expected an error for --json with --explain")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunRejectsNegativeLimit(t *testing.T) {
|
||||
opts := &options{limit: -1}
|
||||
if err := run(context.Background(), &bytes.Buffer{}, opts); err == nil {
|
||||
t.Fatal("expected an error for a negative --limit")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
package impact
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// CurrentBranch returns the branch checked out in repo.
|
||||
func CurrentBranch(repo string) (string, error) {
|
||||
out, err := git(repo, "rev-parse", "--abbrev-ref", "HEAD")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
branch := strings.TrimSpace(out)
|
||||
if branch == "" || branch == "HEAD" {
|
||||
return "", fmt.Errorf("%s is not on a branch; pass --branch", repo)
|
||||
}
|
||||
return branch, nil
|
||||
}
|
||||
|
||||
// ChangedFiles lists the paths that differ between base and branch, using the
|
||||
// three-dot form so only the branch's own commits count. Renames contribute
|
||||
// both the old and the new path, because a moved manifest changes the classes
|
||||
// at both ends.
|
||||
func ChangedFiles(repo, base, branch string) ([]string, error) {
|
||||
out, err := git(repo, "diff", "--name-status", "--find-renames", base+"..."+branch)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var files []string
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
fields := strings.Split(strings.TrimSpace(line), "\t")
|
||||
if len(fields) < 2 || fields[0] == "" {
|
||||
continue
|
||||
}
|
||||
files = append(files, fields[1:]...)
|
||||
}
|
||||
return uniqueSorted(files), nil
|
||||
}
|
||||
|
||||
func git(repo string, args ...string) (string, error) {
|
||||
cmd := exec.Command("git", append([]string{"-C", repo}, args...)...)
|
||||
var stdout, stderr bytes.Buffer
|
||||
cmd.Stdout = &stdout
|
||||
cmd.Stderr = &stderr
|
||||
if err := cmd.Run(); err != nil {
|
||||
return "", fmt.Errorf("git %s: %w: %s", strings.Join(args, " "), err, strings.TrimSpace(stderr.String()))
|
||||
}
|
||||
return stdout.String(), nil
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
package impact
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func gitRepo(t *testing.T) string {
|
||||
t.Helper()
|
||||
if _, err := exec.LookPath("git"); err != nil {
|
||||
t.Skip("git not available")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
run := func(args ...string) {
|
||||
t.Helper()
|
||||
cmd := exec.Command("git", append([]string{"-C", dir}, args...)...)
|
||||
cmd.Env = append(os.Environ(), "GIT_CONFIG_GLOBAL=/dev/null", "GIT_CONFIG_SYSTEM=/dev/null")
|
||||
if out, err := cmd.CombinedOutput(); err != nil {
|
||||
t.Fatalf("git %v: %v: %s", args, err, out)
|
||||
}
|
||||
}
|
||||
write := func(name, body string) {
|
||||
t.Helper()
|
||||
p := filepath.Join(dir, name)
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(p, []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
run("init", "--initial-branch=develop")
|
||||
run("config", "user.email", "test@example.net")
|
||||
run("config", "user.name", "test")
|
||||
write("keep.pp", "keep\n")
|
||||
write("gone.pp", "gone\n")
|
||||
write("old/name.pp", "some reasonably long body so rename detection fires\n")
|
||||
run("add", ".")
|
||||
run("commit", "-m", "base")
|
||||
|
||||
run("checkout", "-b", "feature")
|
||||
write("added.pp", "added\n")
|
||||
write("keep.pp", "changed\n")
|
||||
if err := os.Remove(filepath.Join(dir, "gone.pp")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Join(dir, "new"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
run("mv", "old/name.pp", "new/name.pp")
|
||||
run("add", "-A")
|
||||
run("commit", "-m", "feature")
|
||||
run("checkout", "develop")
|
||||
return dir
|
||||
}
|
||||
|
||||
func TestChangedFilesCoversAddDeleteAndRename(t *testing.T) {
|
||||
dir := gitRepo(t)
|
||||
got, err := ChangedFiles(dir, "develop", "feature")
|
||||
if err != nil {
|
||||
t.Fatalf("ChangedFiles: %v", err)
|
||||
}
|
||||
want := []string{"added.pp", "gone.pp", "keep.pp", "new/name.pp", "old/name.pp"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("ChangedFiles = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCurrentBranch(t *testing.T) {
|
||||
dir := gitRepo(t)
|
||||
got, err := CurrentBranch(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("CurrentBranch: %v", err)
|
||||
}
|
||||
if got != "develop" {
|
||||
t.Errorf("CurrentBranch = %q, want develop", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestChangedFilesReportsGitErrors(t *testing.T) {
|
||||
dir := gitRepo(t)
|
||||
if _, err := ChangedFiles(dir, "develop", "no-such-branch"); err == nil {
|
||||
t.Fatal("expected an error for a missing ref")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,159 @@
|
||||
package impact
|
||||
|
||||
import "sort"
|
||||
|
||||
// Reason explains why one changed file put one host in the result.
|
||||
type Reason struct {
|
||||
File string `json:"file"`
|
||||
Rule string `json:"rule"`
|
||||
Detail string `json:"detail"`
|
||||
}
|
||||
|
||||
// Host is an affected node and every reason it is affected.
|
||||
type Host struct {
|
||||
Certname string `json:"certname"`
|
||||
Reasons []Reason `json:"reasons"`
|
||||
}
|
||||
|
||||
// Summary is the headline count.
|
||||
type Summary struct {
|
||||
Hosts int `json:"hosts"`
|
||||
Files int `json:"files"`
|
||||
AllHostsTriggered bool `json:"all_hosts_triggered"`
|
||||
}
|
||||
|
||||
// Result is the whole analysis, and the --json document.
|
||||
type Result struct {
|
||||
Base string `json:"base"`
|
||||
Branch string `json:"branch"`
|
||||
ChangedFiles []string `json:"changed_files"`
|
||||
Hosts []Host `json:"hosts"`
|
||||
UnmappedFiles []string `json:"unmapped_files"`
|
||||
Summary Summary `json:"summary"`
|
||||
|
||||
// IgnoredFiles are changed files outside the modulepath — CI, docs,
|
||||
// specs — which cannot reach a catalog.
|
||||
IgnoredFiles []string `json:"-"`
|
||||
// OrphanClasses are classes a changed file defines that nothing declares
|
||||
// and no hieradata file pulls in.
|
||||
OrphanClasses []string `json:"-"`
|
||||
// Truncated is set when --limit cut the host list.
|
||||
Truncated bool `json:"-"`
|
||||
// TotalHosts is the estate size the selectors were evaluated against.
|
||||
TotalHosts int `json:"-"`
|
||||
}
|
||||
|
||||
// Certnames returns the affected hostnames, sorted.
|
||||
func (r *Result) Certnames() []string {
|
||||
out := make([]string, 0, len(r.Hosts))
|
||||
for _, h := range r.Hosts {
|
||||
out = append(out, h.Certname)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Analyzer evaluates changed files against one estate fact table.
|
||||
type Analyzer struct {
|
||||
Index *Index
|
||||
Nodes []Node
|
||||
}
|
||||
|
||||
// Analyze maps changed files onto hosts. Files are processed in the order
|
||||
// given; everything in the result is sorted, so output does not move between
|
||||
// runs.
|
||||
func (a *Analyzer) Analyze(base, branch string, files []string, limit int) *Result {
|
||||
resolver := newClassResolver(a.Index, a.Nodes)
|
||||
|
||||
reasons := map[string][]Reason{}
|
||||
orphans := map[string]bool{}
|
||||
result := &Result{
|
||||
Base: base,
|
||||
Branch: branch,
|
||||
ChangedFiles: uniqueSorted(files),
|
||||
TotalHosts: len(a.Nodes),
|
||||
}
|
||||
|
||||
for _, file := range result.ChangedFiles {
|
||||
m := classifyPath(file, a.Index)
|
||||
|
||||
switch {
|
||||
case m.ignored:
|
||||
result.IgnoredFiles = append(result.IgnoredFiles, file)
|
||||
continue
|
||||
case m.unknown:
|
||||
result.UnmappedFiles = append(result.UnmappedFiles, file)
|
||||
m.targets = []target{allTarget(RuleUnmapped,
|
||||
fmtDetail("%s matches no mapping rule; widened to the whole estate", file))}
|
||||
}
|
||||
|
||||
for _, t := range m.targets {
|
||||
for _, certname := range t.sel.Select(a.Nodes) {
|
||||
reasons[certname] = append(reasons[certname],
|
||||
Reason{File: file, Rule: t.rule, Detail: t.detail})
|
||||
}
|
||||
}
|
||||
|
||||
for _, ref := range m.classes {
|
||||
res := resolver.resolve(ref.name)
|
||||
if res.orphan {
|
||||
orphans[ref.name] = true
|
||||
continue
|
||||
}
|
||||
for certname, hits := range res.hosts {
|
||||
for _, h := range hits {
|
||||
reasons[certname] = append(reasons[certname],
|
||||
Reason{File: file, Rule: h.rule, Detail: ref.why + "; " + h.detail})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for name := range orphans {
|
||||
result.OrphanClasses = append(result.OrphanClasses, name)
|
||||
}
|
||||
sort.Strings(result.OrphanClasses)
|
||||
|
||||
certnames := make([]string, 0, len(reasons))
|
||||
for certname := range reasons {
|
||||
certnames = append(certnames, certname)
|
||||
}
|
||||
sort.Strings(certnames)
|
||||
|
||||
result.Summary = Summary{
|
||||
Hosts: len(certnames),
|
||||
Files: len(result.ChangedFiles),
|
||||
AllHostsTriggered: len(certnames) > 0 && len(certnames) == len(a.Nodes),
|
||||
}
|
||||
|
||||
if limit > 0 && len(certnames) > limit {
|
||||
certnames = certnames[:limit]
|
||||
result.Truncated = true
|
||||
}
|
||||
|
||||
for _, certname := range certnames {
|
||||
result.Hosts = append(result.Hosts, Host{Certname: certname, Reasons: dedupeReasons(reasons[certname])})
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func dedupeReasons(in []Reason) []Reason {
|
||||
sort.Slice(in, func(i, j int) bool {
|
||||
if in[i].File != in[j].File {
|
||||
return in[i].File < in[j].File
|
||||
}
|
||||
if in[i].Rule != in[j].Rule {
|
||||
return in[i].Rule < in[j].Rule
|
||||
}
|
||||
return in[i].Detail < in[j].Detail
|
||||
})
|
||||
out := in[:0]
|
||||
var prev Reason
|
||||
for idx, r := range in {
|
||||
if idx > 0 && r == prev {
|
||||
continue
|
||||
}
|
||||
prev = r
|
||||
out = append(out, r)
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,256 @@
|
||||
package impact
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.unkin.net/unkin/puppet-catalog-tools/internal/pdbmux"
|
||||
)
|
||||
|
||||
func testAnalyzer(t *testing.T) *Analyzer {
|
||||
t.Helper()
|
||||
return &Analyzer{Index: testIndex(t), Nodes: fixtureNodes()}
|
||||
}
|
||||
|
||||
func analyze(t *testing.T, files ...string) *Result {
|
||||
t.Helper()
|
||||
return testAnalyzer(t).Analyze("origin/develop", "benvin/test", files, 0)
|
||||
}
|
||||
|
||||
func TestAnalyzeCommonAndHieraTriggerEveryHost(t *testing.T) {
|
||||
all := []string{"bare1", "k8s1", "k8s2", "prox1", "web1"}
|
||||
for _, file := range []string{"hieradata/common.yaml", "hiera.yaml", "environment.conf", "Puppetfile", "manifests/site.pp"} {
|
||||
r := analyze(t, file)
|
||||
if got := r.Certnames(); !reflect.DeepEqual(got, all) {
|
||||
t.Errorf("%s affected %v, want every host", file, got)
|
||||
}
|
||||
if !r.Summary.AllHostsTriggered {
|
||||
t.Errorf("%s did not set all_hosts_triggered", file)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeHieradataRolePrefix(t *testing.T) {
|
||||
r := analyze(t, "hieradata/roles/infra.yaml")
|
||||
if got, want := r.Certnames(), []string{"k8s1", "k8s2", "prox1"}; !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("hosts = %v, want %v", got, want)
|
||||
}
|
||||
if r.Summary.AllHostsTriggered {
|
||||
t.Error("a role-prefix change must not report the whole estate")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeRoleManifestIsExact(t *testing.T) {
|
||||
r := analyze(t, "site/roles/manifests/infra/k8s/node.pp")
|
||||
if got, want := r.Certnames(), []string{"k8s1", "k8s2"}; !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("hosts = %v, want %v", got, want)
|
||||
}
|
||||
if r.Hosts[0].Reasons[0].Rule != RuleRoleClass {
|
||||
t.Errorf("rule = %q, want %q", r.Hosts[0].Reasons[0].Rule, RuleRoleClass)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeTemplateAttribution(t *testing.T) {
|
||||
r := analyze(t, "site/profiles/templates/leaf/x.erb")
|
||||
if got, want := r.Certnames(), []string{"k8s1", "k8s2", "web1"}; !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("hosts = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeUnattributedModuleAssetWidensToTheModule(t *testing.T) {
|
||||
// Nothing references this template, so it falls back to every class the
|
||||
// module defines — here, the hiera_include-only jellyfin module.
|
||||
r := analyze(t, "modules/jellyfin/templates/unreferenced.erb")
|
||||
if got, want := r.Certnames(), []string{"web1"}; !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("hosts = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeUnmappedFileWidensToEveryHost(t *testing.T) {
|
||||
r := analyze(t, "totally/unknown/thing.txt")
|
||||
if got, want := r.UnmappedFiles, []string{"totally/unknown/thing.txt"}; !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("unmapped = %v, want %v", got, want)
|
||||
}
|
||||
if len(r.Hosts) != len(fixtureNodes()) {
|
||||
t.Errorf("unmapped file affected %d hosts, want all %d", len(r.Hosts), len(fixtureNodes()))
|
||||
}
|
||||
if r.Hosts[0].Reasons[0].Rule != RuleUnmapped {
|
||||
t.Errorf("rule = %q", r.Hosts[0].Reasons[0].Rule)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeIgnoresNonPuppetPaths(t *testing.T) {
|
||||
r := analyze(t, ".woodpecker/test.yaml", "modules/orphanmod/spec/orphanmod_spec.rb", "README.md")
|
||||
if len(r.Hosts) != 0 {
|
||||
t.Errorf("CI and spec changes affected %v", r.Certnames())
|
||||
}
|
||||
if len(r.IgnoredFiles) != 3 {
|
||||
t.Errorf("ignored = %v", r.IgnoredFiles)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeReportsOrphanClasses(t *testing.T) {
|
||||
r := analyze(t, "modules/orphanmod/manifests/init.pp")
|
||||
if got, want := r.OrphanClasses, []string{"orphanmod", "orphanmod::init"}; !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("orphans = %v, want %v", got, want)
|
||||
}
|
||||
if len(r.Hosts) != 0 {
|
||||
t.Errorf("orphan class reached %v", r.Certnames())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeLimit(t *testing.T) {
|
||||
r := testAnalyzer(t).Analyze("origin/develop", "benvin/test", []string{"hieradata/common.yaml"}, 2)
|
||||
if got, want := r.Certnames(), []string{"bare1", "k8s1"}; !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("hosts = %v, want %v", got, want)
|
||||
}
|
||||
if !r.Truncated {
|
||||
t.Error("Truncated not set")
|
||||
}
|
||||
if r.Summary.Hosts != len(fixtureNodes()) {
|
||||
t.Errorf("summary.hosts = %d, want the untruncated count", r.Summary.Hosts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeIsDeterministic(t *testing.T) {
|
||||
files := []string{
|
||||
"hieradata/roles/apps/media/jellyfin.yaml",
|
||||
"site/profiles/manifests/leaf.pp",
|
||||
"modules/jellyfin/manifests/install.pp",
|
||||
"hieradata/virtual/physical.yaml",
|
||||
"totally/unknown/thing.txt",
|
||||
}
|
||||
var first string
|
||||
for i := range 20 {
|
||||
var buf bytes.Buffer
|
||||
if err := RenderJSON(&buf, analyze(t, files...)); err != nil {
|
||||
t.Fatalf("RenderJSON: %v", err)
|
||||
}
|
||||
if i == 0 {
|
||||
first = buf.String()
|
||||
continue
|
||||
}
|
||||
if buf.String() != first {
|
||||
t.Fatal("output changed between runs")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderJSONShape(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
if err := RenderJSON(&buf, analyze(t, "hieradata/roles/apps/media/jellyfin.yaml")); err != nil {
|
||||
t.Fatalf("RenderJSON: %v", err)
|
||||
}
|
||||
var doc struct {
|
||||
Base string `json:"base"`
|
||||
Branch string `json:"branch"`
|
||||
ChangedFiles []string `json:"changed_files"`
|
||||
UnmappedFiles []string `json:"unmapped_files"`
|
||||
Hosts []struct {
|
||||
Certname string `json:"certname"`
|
||||
Reasons []struct {
|
||||
File string `json:"file"`
|
||||
Rule string `json:"rule"`
|
||||
Detail string `json:"detail"`
|
||||
} `json:"reasons"`
|
||||
} `json:"hosts"`
|
||||
Summary struct {
|
||||
Hosts int `json:"hosts"`
|
||||
Files int `json:"files"`
|
||||
AllHostsTriggered bool `json:"all_hosts_triggered"`
|
||||
} `json:"summary"`
|
||||
}
|
||||
if err := json.Unmarshal(buf.Bytes(), &doc); err != nil {
|
||||
t.Fatalf("unmarshal: %v", err)
|
||||
}
|
||||
if doc.Base != "origin/develop" || doc.Branch != "benvin/test" {
|
||||
t.Errorf("base/branch = %q/%q", doc.Base, doc.Branch)
|
||||
}
|
||||
if len(doc.Hosts) != 1 || doc.Hosts[0].Certname != "web1" {
|
||||
t.Fatalf("hosts = %#v", doc.Hosts)
|
||||
}
|
||||
if len(doc.Hosts[0].Reasons) != 1 || doc.Hosts[0].Reasons[0].Rule != RuleRole {
|
||||
t.Errorf("reasons = %#v", doc.Hosts[0].Reasons)
|
||||
}
|
||||
if doc.Summary.Files != 1 || doc.Summary.Hosts != 1 || doc.Summary.AllHostsTriggered {
|
||||
t.Errorf("summary = %#v", doc.Summary)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderPlainIsOneHostPerLine(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
if err := RenderPlain(&buf, analyze(t, "hieradata/roles/infra.yaml")); err != nil {
|
||||
t.Fatalf("RenderPlain: %v", err)
|
||||
}
|
||||
if got, want := buf.String(), "k8s1\nk8s2\nprox1\n"; got != want {
|
||||
t.Errorf("output = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderExplainNamesFileRuleAndHost(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
if err := RenderExplain(&buf, analyze(t, "modules/jellyfin/manifests/install.pp", "bogus/path")); err != nil {
|
||||
t.Fatalf("RenderExplain: %v", err)
|
||||
}
|
||||
out := buf.String()
|
||||
for _, want := range []string{"web1", RuleHieraInclude, "hieradata/roles/apps/media/jellyfin.yaml", "bogus/path", "unmapped files"} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("explain output missing %q:\n%s", want, out)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// stubFacts serves a canned fact table the way pdbmux would.
|
||||
type stubFacts struct {
|
||||
records []pdbmux.FactRecord
|
||||
asked []string
|
||||
}
|
||||
|
||||
func (s *stubFacts) FactsByName(_ context.Context, names []string) ([]pdbmux.FactRecord, error) {
|
||||
s.asked = names
|
||||
return s.records, nil
|
||||
}
|
||||
|
||||
func TestLoadNodesFoldsStructuredOS(t *testing.T) {
|
||||
src := &stubFacts{records: []pdbmux.FactRecord{
|
||||
{Certname: "b.example.net", Name: "enc_role", Value: "roles::infra::k8s::node"},
|
||||
{Certname: "b.example.net", Name: "enc_role_tier1", Value: "infra"},
|
||||
{Certname: "b.example.net", Name: "enc_role_tier2", Value: "k8s"},
|
||||
{Certname: "b.example.net", Name: "enc_role_tier3", Value: "node"},
|
||||
{Certname: "b.example.net", Name: "country", Value: "au"},
|
||||
{Certname: "b.example.net", Name: "region", Value: "syd1"},
|
||||
{Certname: "b.example.net", Name: "virtual", Value: "physical"},
|
||||
{Certname: "b.example.net", Name: "os", Value: map[string]any{
|
||||
"name": "AlmaLinux", "release": map[string]any{"major": "9", "minor": "7"},
|
||||
}},
|
||||
{Certname: "a.example.net", Name: "enc_role", Value: "roles::base"},
|
||||
{Certname: "a.example.net", Name: "enc_role_tier1", Value: "base"},
|
||||
}}
|
||||
|
||||
nodes, err := LoadNodes(context.Background(), src)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadNodes: %v", err)
|
||||
}
|
||||
if !reflect.DeepEqual(src.asked, FactNames) {
|
||||
t.Errorf("asked for %v, want %v", src.asked, FactNames)
|
||||
}
|
||||
if len(nodes) != 2 || nodes[0].Certname != "a.example.net" {
|
||||
t.Fatalf("nodes = %#v", nodes)
|
||||
}
|
||||
want := Node{Certname: "b.example.net", Role: "roles::infra::k8s::node",
|
||||
Tier1: "infra", Tier2: "k8s", Tier3: "node", Country: "au", Region: "syd1",
|
||||
Virtual: "physical", OSName: "AlmaLinux", OSMajor: "9", OSMinor: "7"}
|
||||
if nodes[1] != want {
|
||||
t.Errorf("node = %#v, want %#v", nodes[1], want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadNodesRejectsEmptyEstate(t *testing.T) {
|
||||
if _, err := LoadNodes(context.Background(), &stubFacts{}); err == nil {
|
||||
t.Fatal("expected an error for an empty estate")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,279 @@
|
||||
package impact
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
// manifestRoots are the modulepath entries that live in this repo.
|
||||
// external_modules/ is Puppetfile-managed and not checked in.
|
||||
var manifestRoots = []string{"site", "modules"}
|
||||
|
||||
var (
|
||||
commentRe = regexp.MustCompile(`(?m)#.*$`)
|
||||
includeRe = regexp.MustCompile(`\b(?:include|contain|require)\s*\(?\s*['"]?(?:::)?([a-z][a-z0-9_]*(?:::[a-z0-9_]+)*)['"]?`)
|
||||
classParamRe = regexp.MustCompile(`\bclass\s*\{\s*['"](?:::)?([a-z][a-z0-9_]*(?:::[a-z0-9_]+)*)['"]`)
|
||||
classRefRe = regexp.MustCompile(`\bClass\s*\[\s*['"](?:::)?([a-z][a-z0-9_]*(?:::[a-z0-9_]+)*)['"]`)
|
||||
defineUseRe = regexp.MustCompile(`(?m)^\s*(?:::)?([a-z][a-z0-9_]*(?:::[a-z0-9_]+)+)\s*\{`)
|
||||
resourceRefRe = regexp.MustCompile(`\b([A-Z][a-zA-Z0-9_]*(?:::[A-Z][a-zA-Z0-9_]*)+)\s*\[`)
|
||||
assetRe = regexp.MustCompile(`(?:template|epp|file)\s*\(\s*['"]([a-z0-9_]+/[^'"]+)['"]|puppet:///modules/([a-z0-9_]+/[^'"\s]+)`)
|
||||
hieraBlockRe = regexp.MustCompile(`(?m)^(hiera_include|hiera_exclude):\s*$((?:\n\s+-\s*\S+)*)`)
|
||||
hieraItemRe = regexp.MustCompile(`(?m)^\s+-\s*['"]?([^'"\s]+)['"]?\s*$`)
|
||||
)
|
||||
|
||||
// Index is a static view of the control repo: which file defines which class,
|
||||
// which classes reference which, which hieradata files pull classes in through
|
||||
// hiera_include, and which manifest owns which template or file.
|
||||
type Index struct {
|
||||
Root string
|
||||
|
||||
// includedBy maps a class name to the classes that declare it.
|
||||
includedBy map[string]map[string]bool
|
||||
// moduleClasses maps a module name to the classes it defines.
|
||||
moduleClasses map[string][]string
|
||||
// assetOwners maps "<module>/<path>" to the classes referencing it.
|
||||
assetOwners map[string][]string
|
||||
// hieraInclude and hieraExclude map a class name to the hieradata files
|
||||
// listing it.
|
||||
hieraInclude map[string][]string
|
||||
hieraExclude map[string][]string
|
||||
// known holds every class this repo defines.
|
||||
known map[string]bool
|
||||
}
|
||||
|
||||
// BuildIndex walks the control repo at root and indexes it.
|
||||
func BuildIndex(root string) (*Index, error) {
|
||||
idx := &Index{
|
||||
Root: root,
|
||||
includedBy: map[string]map[string]bool{},
|
||||
moduleClasses: map[string][]string{},
|
||||
assetOwners: map[string][]string{},
|
||||
hieraInclude: map[string][]string{},
|
||||
hieraExclude: map[string][]string{},
|
||||
known: map[string]bool{},
|
||||
}
|
||||
|
||||
for _, base := range manifestRoots {
|
||||
dir := filepath.Join(root, base)
|
||||
if _, err := os.Stat(dir); err != nil {
|
||||
continue
|
||||
}
|
||||
if err := filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
|
||||
if err != nil || d.IsDir() || !strings.HasSuffix(p, ".pp") {
|
||||
return err
|
||||
}
|
||||
rel, relErr := filepath.Rel(root, p)
|
||||
if relErr != nil {
|
||||
return relErr
|
||||
}
|
||||
return idx.addManifest(filepath.ToSlash(rel), p)
|
||||
}); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
hieraDir := filepath.Join(root, "hieradata")
|
||||
if _, err := os.Stat(hieraDir); err == nil {
|
||||
if err := filepath.WalkDir(hieraDir, func(p string, d fs.DirEntry, err error) error {
|
||||
if err != nil || d.IsDir() {
|
||||
return err
|
||||
}
|
||||
if ext := filepath.Ext(p); ext != ".yaml" && ext != ".eyaml" {
|
||||
return nil
|
||||
}
|
||||
rel, relErr := filepath.Rel(root, p)
|
||||
if relErr != nil {
|
||||
return relErr
|
||||
}
|
||||
return idx.addHieradata(filepath.ToSlash(rel), p)
|
||||
}); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
for module := range idx.moduleClasses {
|
||||
sort.Strings(idx.moduleClasses[module])
|
||||
}
|
||||
return idx, nil
|
||||
}
|
||||
|
||||
func (i *Index) addManifest(rel, full string) error {
|
||||
parts := strings.Split(rel, "/")
|
||||
if len(parts) < 4 || parts[2] != "manifests" {
|
||||
return nil
|
||||
}
|
||||
module := parts[1]
|
||||
declared := ManifestClasses(module, parts[3:])
|
||||
for _, name := range declared {
|
||||
i.known[name] = true
|
||||
i.moduleClasses[module] = append(i.moduleClasses[module], name)
|
||||
}
|
||||
|
||||
data, err := os.ReadFile(full) //nolint:gosec // paths come from walking the given repo
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading %s: %w", rel, err)
|
||||
}
|
||||
body := commentRe.ReplaceAllString(string(data), "")
|
||||
|
||||
self := map[string]bool{}
|
||||
for _, name := range declared {
|
||||
self[name] = true
|
||||
}
|
||||
for _, ref := range references(body) {
|
||||
if self[ref] {
|
||||
continue
|
||||
}
|
||||
if i.includedBy[ref] == nil {
|
||||
i.includedBy[ref] = map[string]bool{}
|
||||
}
|
||||
for _, name := range declared {
|
||||
i.includedBy[ref][name] = true
|
||||
}
|
||||
}
|
||||
|
||||
for _, asset := range assetReferences(body) {
|
||||
i.assetOwners[asset] = append(i.assetOwners[asset], declared...)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// references extracts every class or defined-type name a manifest body names,
|
||||
// across all the declaration forms this repo uses.
|
||||
func references(body string) []string {
|
||||
seen := map[string]bool{}
|
||||
add := func(name string) {
|
||||
if name != "" && name != "class" && name != "define" {
|
||||
seen[name] = true
|
||||
}
|
||||
}
|
||||
for _, re := range []*regexp.Regexp{includeRe, classParamRe, classRefRe, defineUseRe} {
|
||||
for _, m := range re.FindAllStringSubmatch(body, -1) {
|
||||
add(m[1])
|
||||
}
|
||||
}
|
||||
// Resource references (Profiles::Foo::Bar['x']) name a defined type in
|
||||
// capitalised form.
|
||||
for _, m := range resourceRefRe.FindAllStringSubmatch(body, -1) {
|
||||
add(strings.ToLower(m[1]))
|
||||
}
|
||||
|
||||
out := make([]string, 0, len(seen))
|
||||
for name := range seen {
|
||||
out = append(out, name)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
func assetReferences(body string) []string {
|
||||
var out []string
|
||||
for _, m := range assetRe.FindAllStringSubmatch(body, -1) {
|
||||
for _, g := range m[1:] {
|
||||
if g != "" {
|
||||
out = append(out, g)
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (i *Index) addHieradata(rel, full string) error {
|
||||
data, err := os.ReadFile(full) //nolint:gosec // paths come from walking the given repo
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading %s: %w", rel, err)
|
||||
}
|
||||
include, exclude := parseHieraLists(data)
|
||||
for _, name := range include {
|
||||
i.hieraInclude[name] = append(i.hieraInclude[name], rel)
|
||||
}
|
||||
for _, name := range exclude {
|
||||
i.hieraExclude[name] = append(i.hieraExclude[name], rel)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// parseHieraLists reads hiera_include and hiera_exclude. eyaml files can carry
|
||||
// block scalars a strict parser chokes on, so a regex scan backs the YAML up.
|
||||
func parseHieraLists(data []byte) (include, exclude []string) {
|
||||
var doc map[string]any
|
||||
if err := yaml.Unmarshal(data, &doc); err == nil && doc != nil {
|
||||
return stringList(doc["hiera_include"]), stringList(doc["hiera_exclude"])
|
||||
}
|
||||
for _, m := range hieraBlockRe.FindAllStringSubmatch(string(data), -1) {
|
||||
var items []string
|
||||
for _, item := range hieraItemRe.FindAllStringSubmatch(m[2], -1) {
|
||||
items = append(items, item[1])
|
||||
}
|
||||
if m[1] == "hiera_include" {
|
||||
include = append(include, items...)
|
||||
} else {
|
||||
exclude = append(exclude, items...)
|
||||
}
|
||||
}
|
||||
return include, exclude
|
||||
}
|
||||
|
||||
func stringList(v any) []string {
|
||||
items, ok := v.([]any)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
for _, item := range items {
|
||||
if s, ok := item.(string); ok && s != "" {
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// IncludedBy returns the classes that declare the given class, sorted.
|
||||
func (i *Index) IncludedBy(class string) []string {
|
||||
names := make([]string, 0, len(i.includedBy[class]))
|
||||
for name := range i.includedBy[class] {
|
||||
names = append(names, name)
|
||||
}
|
||||
sort.Strings(names)
|
||||
return names
|
||||
}
|
||||
|
||||
// ModuleClasses returns every class the module defines, sorted.
|
||||
func (i *Index) ModuleClasses(module string) []string { return i.moduleClasses[module] }
|
||||
|
||||
// AssetOwners returns the classes referencing a "<module>/<path>" template or
|
||||
// file, sorted and deduplicated.
|
||||
func (i *Index) AssetOwners(asset string) []string { return uniqueSorted(i.assetOwners[asset]) }
|
||||
|
||||
// HieraInclude returns the hieradata files listing the class under
|
||||
// hiera_include, sorted.
|
||||
func (i *Index) HieraInclude(class string) []string { return uniqueSorted(i.hieraInclude[class]) }
|
||||
|
||||
// HieraExclude returns the hieradata files listing the class under
|
||||
// hiera_exclude, sorted.
|
||||
func (i *Index) HieraExclude(class string) []string { return uniqueSorted(i.hieraExclude[class]) }
|
||||
|
||||
// Known reports whether the repo defines the class.
|
||||
func (i *Index) Known(class string) bool { return i.known[class] }
|
||||
|
||||
func uniqueSorted(in []string) []string {
|
||||
if len(in) == 0 {
|
||||
return nil
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
out := make([]string, 0, len(in))
|
||||
for _, s := range in {
|
||||
if !seen[s] {
|
||||
seen[s] = true
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
// Package impact maps a change in the Puppet control repo onto the hosts whose
|
||||
// catalogs that change can alter.
|
||||
package impact
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"git.unkin.net/unkin/puppet-catalog-tools/internal/pdbmux"
|
||||
)
|
||||
|
||||
// FactNames are the facts every selector is evaluated against. They are
|
||||
// fetched for the whole estate in one query.
|
||||
var FactNames = []string{
|
||||
"enc_role",
|
||||
"enc_role_tier1",
|
||||
"enc_role_tier2",
|
||||
"enc_role_tier3",
|
||||
"country",
|
||||
"region",
|
||||
"virtual",
|
||||
"os",
|
||||
}
|
||||
|
||||
// Node is the slice of a node's facts the hiera hierarchy keys off.
|
||||
type Node struct {
|
||||
Certname string
|
||||
Role string
|
||||
Tier1 string
|
||||
Tier2 string
|
||||
Tier3 string
|
||||
Country string
|
||||
Region string
|
||||
Virtual string
|
||||
OSName string
|
||||
OSMajor string
|
||||
OSMinor string
|
||||
}
|
||||
|
||||
// Tiers returns the role tiers as a slice, truncated at the first empty one.
|
||||
func (n Node) Tiers() []string {
|
||||
tiers := make([]string, 0, 3)
|
||||
for _, t := range []string{n.Tier1, n.Tier2, n.Tier3} {
|
||||
if t == "" {
|
||||
break
|
||||
}
|
||||
tiers = append(tiers, t)
|
||||
}
|
||||
return tiers
|
||||
}
|
||||
|
||||
// FactSource supplies the estate-wide fact table.
|
||||
type FactSource interface {
|
||||
FactsByName(ctx context.Context, names []string) ([]pdbmux.FactRecord, error)
|
||||
}
|
||||
|
||||
// LoadNodes fetches FactNames for every node and folds them into a sorted
|
||||
// table. Nodes without an enc_role are kept: they still match country, os and
|
||||
// virtual selectors.
|
||||
func LoadNodes(ctx context.Context, src FactSource) ([]Node, error) {
|
||||
records, err := src.FactsByName(ctx, FactNames)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
byName := map[string]*Node{}
|
||||
for _, r := range records {
|
||||
if r.Certname == "" {
|
||||
continue
|
||||
}
|
||||
n, ok := byName[r.Certname]
|
||||
if !ok {
|
||||
n = &Node{Certname: r.Certname}
|
||||
byName[r.Certname] = n
|
||||
}
|
||||
applyFact(n, r.Name, r.Value)
|
||||
}
|
||||
|
||||
nodes := make([]Node, 0, len(byName))
|
||||
for _, n := range byName {
|
||||
nodes = append(nodes, *n)
|
||||
}
|
||||
sort.Slice(nodes, func(i, j int) bool { return nodes[i].Certname < nodes[j].Certname })
|
||||
if len(nodes) == 0 {
|
||||
return nil, fmt.Errorf("pdbmux returned no nodes")
|
||||
}
|
||||
return nodes, nil
|
||||
}
|
||||
|
||||
func applyFact(n *Node, name string, value any) {
|
||||
switch name {
|
||||
case "enc_role":
|
||||
n.Role = str(value)
|
||||
case "enc_role_tier1":
|
||||
n.Tier1 = str(value)
|
||||
case "enc_role_tier2":
|
||||
n.Tier2 = str(value)
|
||||
case "enc_role_tier3":
|
||||
n.Tier3 = str(value)
|
||||
case "country":
|
||||
n.Country = str(value)
|
||||
case "region":
|
||||
n.Region = str(value)
|
||||
case "virtual":
|
||||
n.Virtual = str(value)
|
||||
case "os":
|
||||
os, ok := value.(map[string]any)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
n.OSName = str(os["name"])
|
||||
release, ok := os["release"].(map[string]any)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
n.OSMajor = str(release["major"])
|
||||
n.OSMinor = str(release["minor"])
|
||||
}
|
||||
}
|
||||
|
||||
func str(v any) string {
|
||||
switch t := v.(type) {
|
||||
case string:
|
||||
return t
|
||||
case nil:
|
||||
return ""
|
||||
default:
|
||||
return strings.TrimSpace(fmt.Sprint(t))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,249 @@
|
||||
package impact
|
||||
|
||||
import (
|
||||
"path"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Rules, as they appear in --json output and --explain.
|
||||
const (
|
||||
RuleCommon = "hieradata/common"
|
||||
RuleNode = "hieradata/node"
|
||||
RuleRole = "hieradata/role"
|
||||
RuleCountry = "hieradata/country"
|
||||
RuleCountryRegion = "hieradata/country-region"
|
||||
RuleCountryRegRole = "hieradata/country-region-role"
|
||||
RuleCountryRole = "hieradata/country-role"
|
||||
RuleVirtual = "hieradata/virtual"
|
||||
RuleOS = "hieradata/os"
|
||||
RuleRepoGlobal = "repo-global"
|
||||
RulePluginsync = "pluginsync"
|
||||
RuleRoleClass = "role-class"
|
||||
RuleHieraInclude = "hiera_include"
|
||||
RuleUnmapped = "unmapped"
|
||||
RuleClassReverse = "class-reverse-lookup"
|
||||
)
|
||||
|
||||
// repoGlobalFiles change classification or module resolution for the whole
|
||||
// estate. Puppetfile pins external modules whose code is not in this repo, so
|
||||
// it is deliberately mapped to every host rather than guessed at.
|
||||
var repoGlobalFiles = map[string]bool{
|
||||
"hiera.yaml": true,
|
||||
"environment.conf": true,
|
||||
"Puppetfile": true,
|
||||
"Puppetfile.lock": true,
|
||||
}
|
||||
|
||||
// ignoredPrefixes hold paths outside the Puppet modulepath: CI, docs, specs
|
||||
// and the bundler vendor tree. Nothing under them can reach a catalog.
|
||||
var ignoredPrefixes = []string{
|
||||
".woodpecker/", ".github/", ".claude/", "ci/", "doc/", "docs/", "spec/", "vendor/", "examples/",
|
||||
}
|
||||
|
||||
var ignoredFiles = map[string]bool{
|
||||
"Gemfile": true, "Gemfile.lock": true, "Rakefile": true,
|
||||
".rubocop.yml": true, ".reek.yml": true, ".pre-commit-config.yaml": true,
|
||||
".gitignore": true, ".fixtures.yml": true, ".ruby-version": true,
|
||||
"catalog_diff.py": true, "deploy.py": true,
|
||||
}
|
||||
|
||||
// moduleSubdirIgnored are module subtrees that are not consulted during a
|
||||
// catalog compile.
|
||||
var moduleSubdirIgnored = map[string]bool{"spec": true, "examples": true, "doc": true, "docs": true}
|
||||
|
||||
// classRef is a class a changed file defines, awaiting reverse lookup.
|
||||
type classRef struct {
|
||||
name string
|
||||
why string
|
||||
}
|
||||
|
||||
// mapping is what a single changed path resolves to before hosts are selected.
|
||||
type mapping struct {
|
||||
targets []target
|
||||
classes []classRef
|
||||
ignored bool
|
||||
unknown bool
|
||||
}
|
||||
|
||||
// classifyPath maps one repo-relative path onto selectors or classes. idx may
|
||||
// be nil, in which case asset files fall back to their whole module.
|
||||
func classifyPath(p string, idx *Index) mapping {
|
||||
p = path.Clean(strings.TrimPrefix(p, "./"))
|
||||
|
||||
if repoGlobalFiles[p] {
|
||||
return mapping{targets: []target{allTarget(RuleRepoGlobal,
|
||||
fmtDetail("%s is repo-global; every catalog reads it", p))}}
|
||||
}
|
||||
if strings.HasPrefix(p, "manifests/") {
|
||||
return mapping{targets: []target{allTarget(RuleRepoGlobal,
|
||||
"the environment manifest applies to every node")}}
|
||||
}
|
||||
if ignoredFiles[p] || strings.HasSuffix(p, ".md") {
|
||||
return mapping{ignored: true}
|
||||
}
|
||||
for _, prefix := range ignoredPrefixes {
|
||||
if strings.HasPrefix(p, prefix) {
|
||||
return mapping{ignored: true}
|
||||
}
|
||||
}
|
||||
|
||||
if rest, ok := strings.CutPrefix(p, "hieradata/"); ok {
|
||||
sel, rule, ok := hieradataSelector(rest)
|
||||
if !ok {
|
||||
return mapping{unknown: true}
|
||||
}
|
||||
return mapping{targets: []target{{sel: sel, rule: rule,
|
||||
detail: fmtDetail("hieradata file selects %s", sel)}}}
|
||||
}
|
||||
|
||||
if strings.HasPrefix(p, "site/") || strings.HasPrefix(p, "modules/") {
|
||||
return classifyModulePath(p, idx)
|
||||
}
|
||||
|
||||
return mapping{unknown: true}
|
||||
}
|
||||
|
||||
// classifyModulePath handles everything under the modulepath roots: site/ (the
|
||||
// roles and profiles modules) and modules/ (in-repo modules).
|
||||
func classifyModulePath(p string, idx *Index) mapping {
|
||||
parts := strings.Split(p, "/")
|
||||
if len(parts) < 3 {
|
||||
return mapping{unknown: true}
|
||||
}
|
||||
module, subdir := parts[1], parts[2]
|
||||
rest := parts[3:]
|
||||
|
||||
if moduleSubdirIgnored[subdir] {
|
||||
return mapping{ignored: true}
|
||||
}
|
||||
|
||||
if subdir == "manifests" && len(rest) > 0 && strings.HasSuffix(rest[len(rest)-1], ".pp") {
|
||||
var classes []classRef
|
||||
for _, name := range ManifestClasses(module, rest) {
|
||||
classes = append(classes, classRef{name: name, why: fmtDetail("%s defines class %s", p, name)})
|
||||
}
|
||||
return mapping{classes: classes}
|
||||
}
|
||||
|
||||
// lib/ and facts.d/ are pluginsynced to every agent, so a custom fact,
|
||||
// function or type there can move any catalog.
|
||||
if subdir == "lib" || subdir == "facts.d" {
|
||||
return mapping{targets: []target{allTarget(RulePluginsync,
|
||||
fmtDetail("%s is pluginsynced to every agent", p))}}
|
||||
}
|
||||
|
||||
if subdir == "templates" || subdir == "files" {
|
||||
asset := module + "/" + strings.Join(rest, "/")
|
||||
if idx != nil {
|
||||
if owners := idx.AssetOwners(asset); len(owners) > 0 {
|
||||
var classes []classRef
|
||||
for _, name := range owners {
|
||||
classes = append(classes, classRef{name: name,
|
||||
why: fmtDetail("%s is referenced by class %s", p, name)})
|
||||
}
|
||||
return mapping{classes: classes}
|
||||
}
|
||||
}
|
||||
return moduleWide(p, module, idx, "no manifest references this asset")
|
||||
}
|
||||
|
||||
return moduleWide(p, module, idx, fmtDetail("%s/ is module-wide", subdir))
|
||||
}
|
||||
|
||||
// moduleWide attributes a file to every class its module defines. It is the
|
||||
// deliberate over-include fallback for module data that cannot be traced to a
|
||||
// single class.
|
||||
func moduleWide(p, module string, idx *Index, why string) mapping {
|
||||
if idx == nil {
|
||||
return mapping{unknown: true}
|
||||
}
|
||||
names := idx.ModuleClasses(module)
|
||||
if len(names) == 0 {
|
||||
return mapping{unknown: true}
|
||||
}
|
||||
var classes []classRef
|
||||
for _, name := range names {
|
||||
classes = append(classes, classRef{name: name,
|
||||
why: fmtDetail("%s: %s, attributed to class %s", p, why, name)})
|
||||
}
|
||||
return mapping{classes: classes}
|
||||
}
|
||||
|
||||
// ManifestClasses returns the class names a manifest file autoloads as. An
|
||||
// init.pp is reachable both as the namespace itself and as <namespace>::init,
|
||||
// and this repo uses both spellings.
|
||||
func ManifestClasses(module string, rest []string) []string {
|
||||
segs := append([]string{module}, rest...)
|
||||
segs[len(segs)-1] = strings.TrimSuffix(segs[len(segs)-1], ".pp")
|
||||
|
||||
full := strings.Join(segs, "::")
|
||||
if segs[len(segs)-1] != "init" {
|
||||
return []string{full}
|
||||
}
|
||||
short := strings.Join(segs[:len(segs)-1], "::")
|
||||
return []string{short, full}
|
||||
}
|
||||
|
||||
// hieradataSelector maps a path under hieradata/ onto the facts the hiera
|
||||
// hierarchy interpolates into it.
|
||||
func hieradataSelector(rel string) (Selector, string, bool) {
|
||||
ext := path.Ext(rel)
|
||||
if ext != ".yaml" && ext != ".eyaml" {
|
||||
return Selector{}, "", false
|
||||
}
|
||||
parts := strings.Split(strings.TrimSuffix(rel, ext), "/")
|
||||
|
||||
switch {
|
||||
case len(parts) == 1 && parts[0] == "common":
|
||||
return Selector{All: true}, RuleCommon, true
|
||||
|
||||
case parts[0] == "nodes" && len(parts) == 2:
|
||||
return Selector{Certname: parts[1]}, RuleNode, true
|
||||
|
||||
case parts[0] == "virtual" && len(parts) == 2:
|
||||
return Selector{Virtual: parts[1]}, RuleVirtual, true
|
||||
|
||||
case parts[0] == "os" && len(parts) == 3:
|
||||
return osSelector(parts[1], parts[2]), RuleOS, true
|
||||
|
||||
case parts[0] == "roles" && len(parts) >= 2 && len(parts) <= 4:
|
||||
return Selector{RoleTiers: parts[1:]}, RuleRole, true
|
||||
|
||||
case parts[0] == "country" && len(parts) == 2:
|
||||
return Selector{Country: parts[1]}, RuleCountry, true
|
||||
|
||||
case parts[0] == "country" && len(parts) == 4 && parts[2] == "region":
|
||||
return Selector{Country: parts[1], Region: parts[3]}, RuleCountryRegion, true
|
||||
|
||||
case parts[0] == "country" && len(parts) >= 5 && len(parts) <= 7 && parts[2] == "region":
|
||||
return Selector{Country: parts[1], Region: parts[3], RoleTiers: parts[4:]}, RuleCountryRegRole, true
|
||||
|
||||
case parts[0] == "country" && len(parts) >= 4 && len(parts) <= 6 && parts[2] == "roles":
|
||||
return Selector{Country: parts[1], RoleTiers: parts[3:]}, RuleCountryRole, true
|
||||
}
|
||||
|
||||
return Selector{}, "", false
|
||||
}
|
||||
|
||||
// osSelector reads os/<name>/<name><major>[.<minor>].yaml and its
|
||||
// all_releases.yaml sibling.
|
||||
func osSelector(osName, file string) Selector {
|
||||
sel := Selector{OSName: osName}
|
||||
if file == "all_releases" {
|
||||
return sel
|
||||
}
|
||||
version := file
|
||||
if len(file) >= len(osName) && strings.EqualFold(file[:len(osName)], osName) {
|
||||
version = file[len(osName):]
|
||||
}
|
||||
version = strings.TrimLeft(version, "-_")
|
||||
if version == "" {
|
||||
return sel
|
||||
}
|
||||
major, minor, hasMinor := strings.Cut(version, ".")
|
||||
sel.OSMajor = major
|
||||
if hasMinor {
|
||||
sel.OSMinor = minor
|
||||
}
|
||||
return sel
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
package impact
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestHieradataSelector(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
path string
|
||||
rule string
|
||||
want Selector
|
||||
}{
|
||||
{"common yaml", "common.yaml", RuleCommon, Selector{All: true}},
|
||||
{"common eyaml", "common.eyaml", RuleCommon, Selector{All: true}},
|
||||
{"node", "nodes/web01.example.net.yaml", RuleNode, Selector{Certname: "web01.example.net"}},
|
||||
{"virtual", "virtual/physical.yaml", RuleVirtual, Selector{Virtual: "physical"}},
|
||||
|
||||
{"role tier1", "roles/infra.yaml", RuleRole, Selector{RoleTiers: []string{"infra"}}},
|
||||
{"role tier2", "roles/infra/puppet.yaml", RuleRole, Selector{RoleTiers: []string{"infra", "puppet"}}},
|
||||
{"role tier3", "roles/infra/puppet/master.yaml", RuleRole,
|
||||
Selector{RoleTiers: []string{"infra", "puppet", "master"}}},
|
||||
{"role eyaml", "roles/apps/media.eyaml", RuleRole, Selector{RoleTiers: []string{"apps", "media"}}},
|
||||
|
||||
{"country", "country/au.yaml", RuleCountry, Selector{Country: "au"}},
|
||||
{"country eyaml", "country/au.eyaml", RuleCountry, Selector{Country: "au"}},
|
||||
{"country region", "country/au/region/syd1.yaml", RuleCountryRegion,
|
||||
Selector{Country: "au", Region: "syd1"}},
|
||||
{"country region tier1", "country/au/region/syd1/infra.yaml", RuleCountryRegRole,
|
||||
Selector{Country: "au", Region: "syd1", RoleTiers: []string{"infra"}}},
|
||||
{"country region tier3", "country/au/region/syd1/infra/puppet/master.eyaml", RuleCountryRegRole,
|
||||
Selector{Country: "au", Region: "syd1", RoleTiers: []string{"infra", "puppet", "master"}}},
|
||||
{"country roles tier1", "country/au/roles/infra.yaml", RuleCountryRole,
|
||||
Selector{Country: "au", RoleTiers: []string{"infra"}}},
|
||||
{"country roles tier3", "country/au/roles/infra/puppet/master.yaml", RuleCountryRole,
|
||||
Selector{Country: "au", RoleTiers: []string{"infra", "puppet", "master"}}},
|
||||
|
||||
{"os all releases", "os/AlmaLinux/all_releases.yaml", RuleOS, Selector{OSName: "AlmaLinux"}},
|
||||
{"os major", "os/AlmaLinux/AlmaLinux9.yaml", RuleOS, Selector{OSName: "AlmaLinux", OSMajor: "9"}},
|
||||
{"os major minor", "os/AlmaLinux/AlmaLinux9.7.yaml", RuleOS,
|
||||
Selector{OSName: "AlmaLinux", OSMajor: "9", OSMinor: "7"}},
|
||||
{"os debian major", "os/Debian/Debian11.yaml", RuleOS, Selector{OSName: "Debian", OSMajor: "11"}},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
got, rule, ok := hieradataSelector(tc.path)
|
||||
if !ok {
|
||||
t.Fatalf("hieradataSelector(%q) not recognised", tc.path)
|
||||
}
|
||||
if rule != tc.rule {
|
||||
t.Errorf("rule = %q, want %q", rule, tc.rule)
|
||||
}
|
||||
if !reflect.DeepEqual(got, tc.want) {
|
||||
t.Errorf("selector = %#v, want %#v", got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestHieradataSelectorRejects(t *testing.T) {
|
||||
for _, p := range []string{
|
||||
"README.md",
|
||||
"roles/a/b/c/d.yaml",
|
||||
"country/au/region/syd1/a/b/c/d.yaml",
|
||||
"os/AlmaLinux/AlmaLinux9.json",
|
||||
"whatever/thing.yaml",
|
||||
} {
|
||||
if _, _, ok := hieradataSelector(p); ok {
|
||||
t.Errorf("hieradataSelector(%q) unexpectedly recognised", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestManifestClasses(t *testing.T) {
|
||||
tests := []struct {
|
||||
module string
|
||||
rest []string
|
||||
want []string
|
||||
}{
|
||||
{"profiles", []string{"base.pp"}, []string{"profiles::base"}},
|
||||
{"profiles", []string{"base", "motd.pp"}, []string{"profiles::base::motd"}},
|
||||
{"roles", []string{"infra", "k8s", "node.pp"}, []string{"roles::infra::k8s::node"}},
|
||||
{"jellyfin", []string{"init.pp"}, []string{"jellyfin", "jellyfin::init"}},
|
||||
{"profiles", []string{"cloudinit", "init.pp"}, []string{"profiles::cloudinit", "profiles::cloudinit::init"}},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
if got := ManifestClasses(tc.module, tc.rest); !reflect.DeepEqual(got, tc.want) {
|
||||
t.Errorf("ManifestClasses(%q, %v) = %v, want %v", tc.module, tc.rest, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassifyPathRepoGlobal(t *testing.T) {
|
||||
for _, p := range []string{"hiera.yaml", "environment.conf", "Puppetfile", "manifests/site.pp"} {
|
||||
m := classifyPath(p, nil)
|
||||
if len(m.targets) != 1 || !m.targets[0].sel.All {
|
||||
t.Fatalf("classifyPath(%q) = %#v, want an all-hosts target", p, m)
|
||||
}
|
||||
if m.targets[0].rule != RuleRepoGlobal {
|
||||
t.Errorf("classifyPath(%q) rule = %q", p, m.targets[0].rule)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassifyPathIgnoredAndUnknown(t *testing.T) {
|
||||
for _, p := range []string{".woodpecker/test.yaml", "Gemfile", "doc/notes.md", "vendor/bundle/x.rb", "README.md"} {
|
||||
if m := classifyPath(p, nil); !m.ignored {
|
||||
t.Errorf("classifyPath(%q) should be ignored, got %#v", p, m)
|
||||
}
|
||||
}
|
||||
for _, p := range []string{"random/thing.txt", "hieradata/nope/deep/deeper/x.yaml"} {
|
||||
if m := classifyPath(p, nil); !m.unknown {
|
||||
t.Errorf("classifyPath(%q) should be unknown, got %#v", p, m)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassifyPathPluginsync(t *testing.T) {
|
||||
m := classifyPath("modules/libs/lib/facter/netbox.rb", nil)
|
||||
if len(m.targets) != 1 || !m.targets[0].sel.All || m.targets[0].rule != RulePluginsync {
|
||||
t.Fatalf("pluginsync path = %#v, want an all-hosts pluginsync target", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassifyPathManifest(t *testing.T) {
|
||||
m := classifyPath("site/profiles/manifests/base/motd.pp", nil)
|
||||
if len(m.classes) != 1 || m.classes[0].name != "profiles::base::motd" {
|
||||
t.Fatalf("classes = %#v", m.classes)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
package impact
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
)
|
||||
|
||||
// RenderPlain writes one hostname per line, ready to pipe.
|
||||
func RenderPlain(w io.Writer, r *Result) error {
|
||||
for _, h := range r.Hosts {
|
||||
if _, err := fmt.Fprintln(w, h.Certname); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// RenderJSON writes the structured report.
|
||||
func RenderJSON(w io.Writer, r *Result) error {
|
||||
enc := json.NewEncoder(w)
|
||||
enc.SetIndent("", " ")
|
||||
if r.Hosts == nil {
|
||||
r.Hosts = []Host{}
|
||||
}
|
||||
if r.ChangedFiles == nil {
|
||||
r.ChangedFiles = []string{}
|
||||
}
|
||||
if r.UnmappedFiles == nil {
|
||||
r.UnmappedFiles = []string{}
|
||||
}
|
||||
return enc.Encode(r)
|
||||
}
|
||||
|
||||
// RenderExplain writes the human-readable report: what each file mapped to,
|
||||
// and for each host which changed files reach it.
|
||||
func RenderExplain(w io.Writer, r *Result) error {
|
||||
p := func(format string, args ...any) error {
|
||||
_, err := fmt.Fprintf(w, format+"\n", args...)
|
||||
return err
|
||||
}
|
||||
|
||||
if err := p("%s...%s", r.Base, r.Branch); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := p("%d changed file(s), %d of %d host(s) affected", r.Summary.Files, r.Summary.Hosts, r.TotalHosts); err != nil {
|
||||
return err
|
||||
}
|
||||
if r.Summary.AllHostsTriggered {
|
||||
if err := p("the whole estate is affected"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if r.Truncated {
|
||||
if err := p("host list truncated by --limit"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if err := section(w, "unmapped files (widened to all hosts)", r.UnmappedFiles); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := section(w, "ignored files (outside the Puppet modulepath)", r.IgnoredFiles); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := section(w, "orphan classes (nothing declares them and no hiera_include lists them)", r.OrphanClasses); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if len(r.Hosts) == 0 {
|
||||
return p("\nno hosts affected")
|
||||
}
|
||||
if err := p("\nhosts:"); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, h := range r.Hosts {
|
||||
if err := p(" %s", h.Certname); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, reason := range h.Reasons {
|
||||
if err := p(" %s [%s] %s", reason.File, reason.Rule, reason.Detail); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func section(w io.Writer, title string, items []string) error {
|
||||
if len(items) == 0 {
|
||||
return nil
|
||||
}
|
||||
if _, err := fmt.Fprintf(w, "\n%s:\n", title); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, item := range items {
|
||||
if _, err := fmt.Fprintf(w, " %s\n", item); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,139 @@
|
||||
package impact
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// hit is one reason a host is affected, before it is attached to a file.
|
||||
type hit struct {
|
||||
rule string
|
||||
detail string
|
||||
}
|
||||
|
||||
// classResolver walks the include graph backwards from a changed class to the
|
||||
// hosts that end up with it.
|
||||
type classResolver struct {
|
||||
idx *Index
|
||||
nodes []Node
|
||||
cache map[string]*resolution
|
||||
}
|
||||
|
||||
type resolution struct {
|
||||
hosts map[string][]hit
|
||||
orphan bool
|
||||
}
|
||||
|
||||
func newClassResolver(idx *Index, nodes []Node) *classResolver {
|
||||
return &classResolver{idx: idx, nodes: nodes, cache: map[string]*resolution{}}
|
||||
}
|
||||
|
||||
// resolve returns the hosts a class reaches. The walk runs to a fixed point
|
||||
// over both routes a class can be applied by — a manifest declaring it, and a
|
||||
// hieradata file listing it under hiera_include — and stops at role classes,
|
||||
// which the ENC assigns directly.
|
||||
func (r *classResolver) resolve(class string) *resolution {
|
||||
if cached, ok := r.cache[class]; ok {
|
||||
return cached
|
||||
}
|
||||
res := &resolution{hosts: map[string][]hit{}}
|
||||
r.cache[class] = res
|
||||
|
||||
// via records how each class was reached, so --explain can print the chain.
|
||||
via := map[string]string{class: class}
|
||||
seen := map[string]bool{class: true}
|
||||
queue := []string{class}
|
||||
reached := false
|
||||
|
||||
for len(queue) > 0 {
|
||||
current := queue[0]
|
||||
queue = queue[1:]
|
||||
|
||||
if strings.HasPrefix(current, "roles::") {
|
||||
sel := Selector{RoleExact: current}
|
||||
detail := fmtDetail("role %s carries %s (%s)", current, class, via[current])
|
||||
for _, certname := range sel.Select(r.nodes) {
|
||||
res.hosts[certname] = append(res.hosts[certname], hit{rule: RuleRoleClass, detail: detail})
|
||||
}
|
||||
reached = true
|
||||
}
|
||||
|
||||
if hosts, detail, ok := r.viaHiera(current); ok {
|
||||
reached = true
|
||||
for _, certname := range hosts {
|
||||
res.hosts[certname] = append(res.hosts[certname],
|
||||
hit{rule: RuleHieraInclude, detail: detail + " (" + via[current] + ")"})
|
||||
}
|
||||
}
|
||||
|
||||
for _, parent := range r.idx.IncludedBy(current) {
|
||||
if seen[parent] {
|
||||
continue
|
||||
}
|
||||
seen[parent] = true
|
||||
via[parent] = via[current] + " <- " + parent
|
||||
queue = append(queue, parent)
|
||||
}
|
||||
}
|
||||
|
||||
res.orphan = !reached
|
||||
return res
|
||||
}
|
||||
|
||||
// viaHiera applies a class that hieradata pulls in with hiera_include. The
|
||||
// hierarchy position of the listing file decides which hosts get it, and
|
||||
// hiera_exclude entries are subtracted because profiles::base declares
|
||||
// ($hiera_include - $hiera_exclude).
|
||||
func (r *classResolver) viaHiera(class string) ([]string, string, bool) {
|
||||
includes := r.idx.HieraInclude(class)
|
||||
if len(includes) == 0 {
|
||||
return nil, "", false
|
||||
}
|
||||
|
||||
included := map[string]bool{}
|
||||
var sources []string
|
||||
for _, file := range includes {
|
||||
sel := hieraFileSelector(file)
|
||||
for _, certname := range sel.Select(r.nodes) {
|
||||
included[certname] = true
|
||||
}
|
||||
sources = append(sources, fmtDetail("%s [%s]", file, sel))
|
||||
}
|
||||
|
||||
excludes := r.idx.HieraExclude(class)
|
||||
var excluded []string
|
||||
for _, file := range excludes {
|
||||
sel := hieraFileSelector(file)
|
||||
for _, certname := range sel.Select(r.nodes) {
|
||||
if included[certname] {
|
||||
delete(included, certname)
|
||||
excluded = append(excluded, certname)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
hosts := make([]string, 0, len(included))
|
||||
for certname := range included {
|
||||
hosts = append(hosts, certname)
|
||||
}
|
||||
sort.Strings(hosts)
|
||||
|
||||
detail := fmtDetail("hiera_include %s in %s", class, strings.Join(sources, ", "))
|
||||
if len(excluded) > 0 {
|
||||
detail += fmtDetail("; hiera_exclude in %s drops %d host(s)",
|
||||
strings.Join(excludes, ", "), len(uniqueSorted(excluded)))
|
||||
}
|
||||
return hosts, detail, true
|
||||
}
|
||||
|
||||
// hieraFileSelector maps a hieradata file back onto its hierarchy position. An
|
||||
// unrecognised path widens to the whole estate rather than silently selecting
|
||||
// nothing.
|
||||
func hieraFileSelector(file string) Selector {
|
||||
rel := strings.TrimPrefix(file, "hieradata/")
|
||||
sel, _, ok := hieradataSelector(rel)
|
||||
if !ok {
|
||||
return Selector{All: true}
|
||||
}
|
||||
return sel
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
package impact
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func testIndex(t *testing.T) *Index {
|
||||
t.Helper()
|
||||
idx, err := BuildIndex("testdata/repo")
|
||||
if err != nil {
|
||||
t.Fatalf("BuildIndex: %v", err)
|
||||
}
|
||||
return idx
|
||||
}
|
||||
|
||||
func resolvedHosts(t *testing.T, class string) []string {
|
||||
t.Helper()
|
||||
res := newClassResolver(testIndex(t), fixtureNodes()).resolve(class)
|
||||
hosts := make([]string, 0, len(res.hosts))
|
||||
for certname := range res.hosts {
|
||||
hosts = append(hosts, certname)
|
||||
}
|
||||
return uniqueSorted(hosts)
|
||||
}
|
||||
|
||||
func TestResolveThroughIncludeChain(t *testing.T) {
|
||||
// profiles::leaf <- profiles::mid <- profiles::base <- two role classes.
|
||||
if got, want := resolvedHosts(t, "profiles::leaf"), []string{"k8s1", "k8s2", "web1"}; !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("profiles::leaf = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveContainAndClassResource(t *testing.T) {
|
||||
for _, class := range []string{"profiles::contained", "profiles::viaclass"} {
|
||||
if got, want := resolvedHosts(t, class), []string{"k8s1", "k8s2", "web1"}; !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("%s = %v, want %v", class, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveDefinedTypeUsage(t *testing.T) {
|
||||
if got, want := resolvedHosts(t, "profiles::defined::thing"), []string{"k8s1", "k8s2", "web1"}; !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("profiles::defined::thing = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveTerminatesOnCycle(t *testing.T) {
|
||||
// profiles::cycle::a and ::b include each other; ::a is pulled in by
|
||||
// profiles::base, so the walk must reach the roles and stop.
|
||||
if got, want := resolvedHosts(t, "profiles::cycle::b"), []string{"k8s1", "k8s2", "web1"}; !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("profiles::cycle::b = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveViaHieraIncludeOnly(t *testing.T) {
|
||||
// The jellyfin module is applied by hiera_include in a role yaml and by
|
||||
// nothing in any manifest — a .pp-only search would call it orphaned.
|
||||
if got, want := resolvedHosts(t, "jellyfin"), []string{"web1"}; !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("jellyfin = %v, want %v", got, want)
|
||||
}
|
||||
// One hop further in: jellyfin::install is only reachable through jellyfin.
|
||||
if got, want := resolvedHosts(t, "jellyfin::install"), []string{"web1"}; !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("jellyfin::install = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveHieraIncludeFromVirtualHierarchy(t *testing.T) {
|
||||
if got, want := resolvedHosts(t, "profiles::lldpd"), []string{"k8s1", "k8s2", "prox1"}; !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("profiles::lldpd = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveHonoursHieraExclude(t *testing.T) {
|
||||
// common.yaml pulls basemod in for everyone; roles/infra/proxmox.yaml
|
||||
// excludes it again, so the proxmox node drops out.
|
||||
if got, want := resolvedHosts(t, "basemod"), []string{"bare1", "k8s1", "k8s2", "web1"}; !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("basemod = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveOrphanClass(t *testing.T) {
|
||||
res := newClassResolver(testIndex(t), fixtureNodes()).resolve("orphanmod")
|
||||
if !res.orphan {
|
||||
t.Errorf("orphanmod should be orphaned, hosts = %v", res.hosts)
|
||||
}
|
||||
if len(res.hosts) != 0 {
|
||||
t.Errorf("orphanmod reached %v", res.hosts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIndexIgnoresCommentedReferences(t *testing.T) {
|
||||
if got := testIndex(t).IncludedBy("profiles::commented"); len(got) != 0 {
|
||||
t.Errorf("a commented-out class name was indexed as a reference: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIndexAssetOwners(t *testing.T) {
|
||||
idx := testIndex(t)
|
||||
for _, asset := range []string{"profiles/leaf/x.erb", "profiles/leaf/y.txt"} {
|
||||
if got, want := idx.AssetOwners(asset), []string{"profiles::leaf"}; !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("AssetOwners(%q) = %v, want %v", asset, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestIndexModuleClasses(t *testing.T) {
|
||||
got := testIndex(t).ModuleClasses("jellyfin")
|
||||
want := []string{"jellyfin", "jellyfin::init", "jellyfin::install"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("ModuleClasses(jellyfin) = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseHieraListsRegexFallback(t *testing.T) {
|
||||
// eyaml payloads a strict parser rejects still have to yield their lists.
|
||||
broken := []byte("---\nhiera_include:\n - alpha\n - beta\nhiera_exclude:\n - gamma\nbad: [unclosed\n")
|
||||
include, exclude := parseHieraLists(broken)
|
||||
if !reflect.DeepEqual(include, []string{"alpha", "beta"}) {
|
||||
t.Errorf("include = %v", include)
|
||||
}
|
||||
if !reflect.DeepEqual(exclude, []string{"gamma"}) {
|
||||
t.Errorf("exclude = %v", exclude)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package impact
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Selector is a conjunction of fact constraints. Every field left at its zero
|
||||
// value is unconstrained, so an empty Selector with All unset matches nothing
|
||||
// and is treated as a programming error by the callers that build them.
|
||||
type Selector struct {
|
||||
// All matches the whole estate (hieradata/common.yaml, repo-global files).
|
||||
All bool
|
||||
// Certname pins one host (hieradata/nodes/<certname>.yaml).
|
||||
Certname string
|
||||
// RoleTiers is a prefix match on enc_role_tier1..3, so a two-tier
|
||||
// hieradata file covers every role nested under it.
|
||||
RoleTiers []string
|
||||
// RoleExact matches enc_role verbatim, which is how a role class maps to
|
||||
// hosts.
|
||||
RoleExact string
|
||||
Country string
|
||||
Region string
|
||||
Virtual string
|
||||
OSName string
|
||||
OSMajor string
|
||||
OSMinor string
|
||||
}
|
||||
|
||||
// Match reports whether the node satisfies every constraint set on s.
|
||||
func (s Selector) Match(n Node) bool {
|
||||
if s.All {
|
||||
return true
|
||||
}
|
||||
if s.Certname != "" && s.Certname != n.Certname {
|
||||
return false
|
||||
}
|
||||
if s.RoleExact != "" && s.RoleExact != n.Role {
|
||||
return false
|
||||
}
|
||||
if s.Country != "" && s.Country != n.Country {
|
||||
return false
|
||||
}
|
||||
if s.Region != "" && s.Region != n.Region {
|
||||
return false
|
||||
}
|
||||
if s.Virtual != "" && s.Virtual != n.Virtual {
|
||||
return false
|
||||
}
|
||||
if s.OSName != "" && !strings.EqualFold(s.OSName, n.OSName) {
|
||||
return false
|
||||
}
|
||||
if s.OSMajor != "" && s.OSMajor != n.OSMajor {
|
||||
return false
|
||||
}
|
||||
if s.OSMinor != "" && s.OSMinor != n.OSMinor {
|
||||
return false
|
||||
}
|
||||
if len(s.RoleTiers) > 0 {
|
||||
tiers := n.Tiers()
|
||||
if len(tiers) < len(s.RoleTiers) {
|
||||
return false
|
||||
}
|
||||
for i, want := range s.RoleTiers {
|
||||
if tiers[i] != want {
|
||||
return false
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// Select returns the certnames of every matching node, sorted.
|
||||
func (s Selector) Select(nodes []Node) []string {
|
||||
var out []string
|
||||
for _, n := range nodes {
|
||||
if s.Match(n) {
|
||||
out = append(out, n.Certname)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// String renders the selector the way --explain prints it.
|
||||
func (s Selector) String() string {
|
||||
if s.All {
|
||||
return "all hosts"
|
||||
}
|
||||
var parts []string
|
||||
add := func(k, v string) {
|
||||
if v != "" {
|
||||
parts = append(parts, k+"="+v)
|
||||
}
|
||||
}
|
||||
add("certname", s.Certname)
|
||||
add("enc_role", s.RoleExact)
|
||||
if len(s.RoleTiers) > 0 {
|
||||
parts = append(parts, "role-prefix="+strings.Join(s.RoleTiers, "::"))
|
||||
}
|
||||
add("country", s.Country)
|
||||
add("region", s.Region)
|
||||
add("virtual", s.Virtual)
|
||||
add("os.name", s.OSName)
|
||||
add("os.release.major", s.OSMajor)
|
||||
add("os.release.minor", s.OSMinor)
|
||||
if len(parts) == 0 {
|
||||
return "nothing"
|
||||
}
|
||||
return strings.Join(parts, " ")
|
||||
}
|
||||
|
||||
// target is one resolved "this file reaches these hosts, for this reason".
|
||||
type target struct {
|
||||
sel Selector
|
||||
rule string
|
||||
detail string
|
||||
}
|
||||
|
||||
func allTarget(rule, detail string) target {
|
||||
return target{sel: Selector{All: true}, rule: rule, detail: detail}
|
||||
}
|
||||
|
||||
func fmtDetail(format string, args ...any) string { return fmt.Sprintf(format, args...) }
|
||||
@@ -0,0 +1,88 @@
|
||||
package impact
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// fixtureNodes mirrors the shape of the real estate: one country, two regions,
|
||||
// physical and virtual hosts, and a mix of OS releases.
|
||||
func fixtureNodes() []Node {
|
||||
return []Node{
|
||||
{Certname: "web1", Role: "roles::apps::media::jellyfin",
|
||||
Tier1: "apps", Tier2: "media", Tier3: "jellyfin",
|
||||
Country: "au", Region: "syd1", Virtual: "lxc",
|
||||
OSName: "AlmaLinux", OSMajor: "9", OSMinor: "7"},
|
||||
{Certname: "k8s1", Role: "roles::infra::k8s::node",
|
||||
Tier1: "infra", Tier2: "k8s", Tier3: "node",
|
||||
Country: "au", Region: "syd1", Virtual: "physical",
|
||||
OSName: "AlmaLinux", OSMajor: "9", OSMinor: "8"},
|
||||
{Certname: "k8s2", Role: "roles::infra::k8s::node",
|
||||
Tier1: "infra", Tier2: "k8s", Tier3: "node",
|
||||
Country: "au", Region: "drw1", Virtual: "physical",
|
||||
OSName: "Debian", OSMajor: "12", OSMinor: "1"},
|
||||
{Certname: "prox1", Role: "roles::infra::proxmox::node",
|
||||
Tier1: "infra", Tier2: "proxmox", Tier3: "node",
|
||||
Country: "nz", Region: "akl1", Virtual: "physical",
|
||||
OSName: "AlmaLinux", OSMajor: "8", OSMinor: "10"},
|
||||
{Certname: "bare1", Role: "roles::base", Tier1: "base",
|
||||
Country: "au", Region: "syd1", Virtual: "lxc",
|
||||
OSName: "AlmaLinux", OSMajor: "9", OSMinor: "7"},
|
||||
}
|
||||
}
|
||||
|
||||
func TestSelectorSelect(t *testing.T) {
|
||||
nodes := fixtureNodes()
|
||||
tests := []struct {
|
||||
name string
|
||||
sel Selector
|
||||
want []string
|
||||
}{
|
||||
{"all", Selector{All: true}, []string{"web1", "k8s1", "k8s2", "prox1", "bare1"}},
|
||||
{"certname", Selector{Certname: "k8s2"}, []string{"k8s2"}},
|
||||
{"role exact", Selector{RoleExact: "roles::infra::k8s::node"}, []string{"k8s1", "k8s2"}},
|
||||
{"tier1 prefix", Selector{RoleTiers: []string{"infra"}}, []string{"k8s1", "k8s2", "prox1"}},
|
||||
{"tier2 prefix", Selector{RoleTiers: []string{"infra", "k8s"}}, []string{"k8s1", "k8s2"}},
|
||||
{"tier3 exact", Selector{RoleTiers: []string{"apps", "media", "jellyfin"}}, []string{"web1"}},
|
||||
{"tier1 only node", Selector{RoleTiers: []string{"base"}}, []string{"bare1"}},
|
||||
{"tier prefix longer than node", Selector{RoleTiers: []string{"base", "x"}}, nil},
|
||||
{"country", Selector{Country: "au"}, []string{"web1", "k8s1", "k8s2", "bare1"}},
|
||||
{"country region", Selector{Country: "au", Region: "syd1"}, []string{"web1", "k8s1", "bare1"}},
|
||||
{"country region role", Selector{Country: "au", Region: "syd1", RoleTiers: []string{"infra"}},
|
||||
[]string{"k8s1"}},
|
||||
{"country role", Selector{Country: "au", RoleTiers: []string{"infra", "k8s"}}, []string{"k8s1", "k8s2"}},
|
||||
{"virtual", Selector{Virtual: "physical"}, []string{"k8s1", "k8s2", "prox1"}},
|
||||
{"os name", Selector{OSName: "AlmaLinux"}, []string{"web1", "k8s1", "prox1", "bare1"}},
|
||||
{"os major", Selector{OSName: "AlmaLinux", OSMajor: "9"}, []string{"web1", "k8s1", "bare1"}},
|
||||
{"os major minor", Selector{OSName: "AlmaLinux", OSMajor: "9", OSMinor: "7"},
|
||||
[]string{"web1", "bare1"}},
|
||||
{"os name case insensitive", Selector{OSName: "almalinux", OSMajor: "8"}, []string{"prox1"}},
|
||||
{"os other name", Selector{OSName: "Debian", OSMajor: "12"}, []string{"k8s2"}},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := tc.sel.Select(nodes); !reflect.DeepEqual(got, tc.want) {
|
||||
t.Errorf("Select() = %v, want %v", got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestSelectorString(t *testing.T) {
|
||||
if got := (Selector{All: true}).String(); got != "all hosts" {
|
||||
t.Errorf("all selector = %q", got)
|
||||
}
|
||||
got := Selector{Country: "au", Region: "syd1", RoleTiers: []string{"infra", "puppet"}}.String()
|
||||
want := "role-prefix=infra::puppet country=au region=syd1"
|
||||
if got != want {
|
||||
t.Errorf("String() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNodeTiersTruncatesAtGap(t *testing.T) {
|
||||
n := Node{Tier1: "base"}
|
||||
if got := n.Tiers(); !reflect.DeepEqual(got, []string{"base"}) {
|
||||
t.Errorf("Tiers() = %v", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
---
|
||||
hiera_include:
|
||||
- basemod
|
||||
@@ -0,0 +1,3 @@
|
||||
---
|
||||
hiera_include:
|
||||
- jellyfin
|
||||
@@ -0,0 +1,3 @@
|
||||
---
|
||||
hiera_exclude:
|
||||
- basemod
|
||||
@@ -0,0 +1,3 @@
|
||||
---
|
||||
hiera_include:
|
||||
- profiles::lldpd
|
||||
@@ -0,0 +1,2 @@
|
||||
class basemod {
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
class jellyfin {
|
||||
include jellyfin::install
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
class jellyfin::install {
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
class orphanmod {
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
not a manifest
|
||||
@@ -0,0 +1 @@
|
||||
leaf file
|
||||
@@ -0,0 +1,8 @@
|
||||
class profiles::base {
|
||||
# a comment naming profiles::commented must not count as a reference
|
||||
include profiles::mid
|
||||
contain profiles::contained
|
||||
class { 'profiles::viaclass': }
|
||||
include profiles::cycle::a
|
||||
profiles::defined::thing { 'x': }
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
class profiles::contained {
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
class profiles::cycle::a {
|
||||
include profiles::cycle::b
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
class profiles::cycle::b {
|
||||
include profiles::cycle::a
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
define profiles::defined::thing () {
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
class profiles::leaf {
|
||||
file { '/tmp/leaf':
|
||||
content => template('profiles/leaf/x.erb'),
|
||||
source => 'puppet:///modules/profiles/leaf/y.txt',
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
class profiles::lldpd {
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
class profiles::mid {
|
||||
include profiles::leaf
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
class profiles::viaclass {
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
leaf template
|
||||
@@ -0,0 +1,3 @@
|
||||
class roles::apps::media::jellyfin {
|
||||
include profiles::base
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
class roles::infra::k8s::node {
|
||||
include profiles::base
|
||||
}
|
||||
@@ -128,3 +128,45 @@ func snippet(body []byte) string {
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// FactRecord is one (certname, fact) pair from an estate-wide fact query.
|
||||
type FactRecord struct {
|
||||
Certname string `json:"certname"`
|
||||
Name string `json:"name"`
|
||||
Value any `json:"value"`
|
||||
}
|
||||
|
||||
// FactsByName returns the named facts for every node pdbmux knows about, in a
|
||||
// single query. Tools that evaluate selectors over the whole estate want the
|
||||
// table once rather than a request per host.
|
||||
func (c *Client) FactsByName(ctx context.Context, names []string) ([]FactRecord, error) {
|
||||
if len(names) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
clauses := make([]any, 0, len(names)+1)
|
||||
clauses = append(clauses, "or")
|
||||
for _, n := range names {
|
||||
clauses = append(clauses, []any{"=", "name", n})
|
||||
}
|
||||
query := []any{"extract", []string{"certname", "name", "value"}, clauses}
|
||||
|
||||
encoded, err := json.Marshal(query)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("encoding fact query: %w", err)
|
||||
}
|
||||
path := "/pdb/query/v4/facts?query=" + url.QueryEscape(string(encoded))
|
||||
|
||||
body, status, err := c.get(ctx, path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if status != http.StatusOK {
|
||||
return nil, fmt.Errorf("pdbmux returned %d for fact query: %s", status, snippet(body))
|
||||
}
|
||||
|
||||
var records []FactRecord
|
||||
if err := json.Unmarshal(body, &records); err != nil {
|
||||
return nil, fmt.Errorf("parsing fact query response: %w", err)
|
||||
}
|
||||
return records, nil
|
||||
}
|
||||
|
||||
@@ -157,3 +157,58 @@ func TestFactsContextCancelled(t *testing.T) {
|
||||
t.Fatal("expected a timeout error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFactsByName(t *testing.T) {
|
||||
var gotQuery string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
gotQuery = r.URL.Query().Get("query")
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`[
|
||||
{"certname":"a.example.net","name":"country","value":"au"},
|
||||
{"certname":"a.example.net","name":"os","value":{"name":"AlmaLinux","release":{"major":"9","minor":"7"}}}
|
||||
]`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
records, err := New(srv.URL, 5*time.Second).FactsByName(context.Background(), []string{"country", "os"})
|
||||
if err != nil {
|
||||
t.Fatalf("FactsByName: %v", err)
|
||||
}
|
||||
if gotQuery != `["extract",["certname","name","value"],["or",["=","name","country"],["=","name","os"]]]` {
|
||||
t.Errorf("query = %s", gotQuery)
|
||||
}
|
||||
if len(records) != 2 {
|
||||
t.Fatalf("records = %#v", records)
|
||||
}
|
||||
if records[0].Certname != "a.example.net" || records[0].Value != "au" {
|
||||
t.Errorf("first record = %#v", records[0])
|
||||
}
|
||||
os, ok := records[1].Value.(map[string]any)
|
||||
if !ok || os["name"] != "AlmaLinux" {
|
||||
t.Errorf("structured os fact lost: %#v", records[1].Value)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFactsByNameEmptyNames(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {
|
||||
t.Error("no request should be made for an empty fact list")
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
records, err := New(srv.URL, 5*time.Second).FactsByName(context.Background(), nil)
|
||||
if err != nil || records != nil {
|
||||
t.Errorf("FactsByName(nil) = %v, %v", records, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFactsByNameServerError(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
_, _ = w.Write([]byte("boom"))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
if _, err := New(srv.URL, 5*time.Second).FactsByName(context.Background(), []string{"country"}); err == nil {
|
||||
t.Fatal("expected an error for a 500")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -30,6 +30,13 @@ contents:
|
||||
owner: root
|
||||
group: root
|
||||
|
||||
- src: dist/catalog-impact
|
||||
dst: /usr/bin/catalog-impact
|
||||
file_info:
|
||||
mode: 0755
|
||||
owner: root
|
||||
group: root
|
||||
|
||||
# Shell completions (generated by scripts/build-rpm.sh before packaging).
|
||||
- src: dist/completions/catalog-diff.bash
|
||||
dst: /usr/share/bash-completion/completions/catalog-diff
|
||||
@@ -43,3 +50,16 @@ contents:
|
||||
dst: /usr/share/fish/vendor_completions.d/catalog-diff.fish
|
||||
file_info:
|
||||
mode: 0644
|
||||
|
||||
- src: dist/completions/catalog-impact.bash
|
||||
dst: /usr/share/bash-completion/completions/catalog-impact
|
||||
file_info:
|
||||
mode: 0644
|
||||
- src: dist/completions/_catalog-impact
|
||||
dst: /usr/share/zsh/site-functions/_catalog-impact
|
||||
file_info:
|
||||
mode: 0644
|
||||
- src: dist/completions/catalog-impact.fish
|
||||
dst: /usr/share/fish/vendor_completions.d/catalog-impact.fish
|
||||
file_info:
|
||||
mode: 0644
|
||||
|
||||
@@ -12,7 +12,7 @@ cd "${ROOT_DIR}"
|
||||
VERSION="${1:-${CI_COMMIT_TAG:-0.0.0-dev}}"
|
||||
VERSION="${VERSION#v}" # strip a leading v
|
||||
PACKAGE="puppet-catalog-tools"
|
||||
BINARIES=(catalog-diff)
|
||||
BINARIES=(catalog-diff catalog-impact)
|
||||
DIST="dist"
|
||||
|
||||
for b in "${BINARIES[@]}"; do
|
||||
@@ -37,7 +37,7 @@ export PACKAGE_VERSION="${VERSION}"
|
||||
export PACKAGE_RELEASE="1"
|
||||
export PACKAGE_ARCH="amd64"
|
||||
export PACKAGE_PLATFORM="linux"
|
||||
export PACKAGE_DESCRIPTION="Puppet catalog tooling: catalog-diff compiles a host's catalog in two environments and diffs them"
|
||||
export PACKAGE_DESCRIPTION="Puppet catalog tooling: catalog-diff compiles a host's catalog in two environments and diffs them; catalog-impact lists the hosts a control repo change affects"
|
||||
export PACKAGE_MAINTAINER="Ben Vincent <ben@unkin.net>"
|
||||
export PACKAGE_HOMEPAGE="https://git.unkin.net/unkin/puppet-catalog-tools"
|
||||
export PACKAGE_LICENSE="MIT"
|
||||
|
||||
Reference in New Issue
Block a user