docs: strip over-commenting from README and source
This commit is contained in:
@@ -1,7 +1,4 @@
|
||||
# Build and push the pdbmux container image on a v* tag. pdbmux is a k8s-only
|
||||
# daemon (deployed via argocd-apps), so it ships as an image. Mirrors the estate
|
||||
# convention: the CA-baked plugin-docker-buildx image pushes to the
|
||||
# artifactapi local docker registry (unauthenticated in-cluster push).
|
||||
# plugin-docker-buildx is the CA-baked variant; artifactapi's cert is not in the default trust store.
|
||||
when:
|
||||
- event: tag
|
||||
ref: refs/tags/v*
|
||||
|
||||
@@ -1,6 +1,3 @@
|
||||
# Container image for pdbmux, the merging PuppetDB proxy daemon. pdbmux is a
|
||||
# k8s-only service (deployed via argocd-apps), so it ships as a distroless
|
||||
# static image rather than an RPM.
|
||||
FROM golang:1.25-alpine AS builder
|
||||
|
||||
RUN apk add --no-cache git
|
||||
|
||||
@@ -9,7 +9,6 @@ ARCH ?= $(shell go env GOARCH)
|
||||
|
||||
all: build
|
||||
|
||||
# Build the single static binary into dist/.
|
||||
build:
|
||||
CGO_ENABLED=0 GOOS=$(OS) GOARCH=$(ARCH) go build $(GOFLAGS) -o $(DIST)/$(BINARY) .
|
||||
|
||||
@@ -28,8 +27,6 @@ clean:
|
||||
install:
|
||||
go install $(GOFLAGS) .
|
||||
|
||||
# Bump helpers — read the latest semver tag and create the next one.
|
||||
# If no tag exists yet, start from v0.0.0.
|
||||
_LATEST := $(shell git tag --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$$' | head -1)
|
||||
_BASE := $(if $(_LATEST),$(_LATEST),v0.0.0)
|
||||
_MAJ := $(shell echo $(_BASE) | sed 's/^v//' | cut -d. -f1)
|
||||
|
||||
@@ -55,12 +55,10 @@ unknown fields survive untouched.
|
||||
|
||||
Precedence (lowest → highest): **defaults < config file < env vars (`PDBMUX_*`) < flags**.
|
||||
|
||||
Config file: `$XDG_CONFIG_HOME/pdbmux/config.yaml`. In Kubernetes, configuration
|
||||
is supplied entirely via `PDBMUX_*` env vars (no config file), which is the
|
||||
supported deployment path — see [Deployment](#deployment).
|
||||
Config file: `$XDG_CONFIG_HOME/pdbmux/config.yaml`. In Kubernetes there is no
|
||||
config file — everything comes from `PDBMUX_*` env vars.
|
||||
|
||||
```yaml
|
||||
# ~/.config/pdbmux/config.yaml (local dev; in k8s use PDBMUX_* env instead)
|
||||
listen: ":8080"
|
||||
backends:
|
||||
- name: old
|
||||
@@ -74,8 +72,8 @@ timeout: 10s # per-upstream request timeout
|
||||
freshness_ttl: 30s # freshness-map cache TTL (freshness merge only)
|
||||
```
|
||||
|
||||
`backends[*].url` is a **base** URL (`scheme://host[:port]`), without the
|
||||
`/pdb/query/v4/...` path — `pdbmux` appends the path per request.
|
||||
`backends[*].url` is a **base** URL (`scheme://host[:port]`); `pdbmux` appends
|
||||
the `/pdb/query/v4/...` path per request.
|
||||
|
||||
| Env var | Overrides |
|
||||
|---|---|
|
||||
@@ -91,66 +89,24 @@ Flags: `--listen`, `--primary`, `--merge`.
|
||||
|
||||
## Running
|
||||
|
||||
```bash
|
||||
pdbmux # start the proxy (serve is the default action)
|
||||
pdbmux serve # explicit
|
||||
pdbmux config init # write a default config file
|
||||
pdbmux config show # print active config after all overrides
|
||||
pdbmux version
|
||||
```
|
||||
|
||||
Point a consumer at it:
|
||||
Subcommands: `serve` (default), `config init`, `config show`, `version`. Run
|
||||
`pdbmux --help` for details.
|
||||
|
||||
```bash
|
||||
PDBMUX_BACKENDS='old=http://puppetdbapi.service.consul:8080,new=https://puppetdb.k8s.syd1.au.unkin.net' pdbmux
|
||||
node-lookup --url http://localhost:8080/pdb/query/v4/facts -R
|
||||
NODE_LOOKUP_URL=http://localhost:8080/pdb/query/v4/facts pblastreport somehost
|
||||
```
|
||||
|
||||
## Build
|
||||
|
||||
```bash
|
||||
make build # -> dist/pdbmux (CGO disabled, static)
|
||||
make test # go test -race ./...
|
||||
make lint # golangci-lint
|
||||
```
|
||||
|
||||
Requires Go 1.25+. Dependencies: `github.com/spf13/cobra` (CLI),
|
||||
`gopkg.in/yaml.v3` (config file).
|
||||
`make build` (static binary into `dist/`), `make test`, `make lint`. Requires Go 1.25+.
|
||||
|
||||
## Deployment
|
||||
|
||||
`pdbmux` runs **in Kubernetes** as a container, in line with the all-in-k8s
|
||||
estate direction — it is not shipped as a per-VM RPM/systemd service. The image
|
||||
is built and pushed on every `v*` tag (`.woodpecker/docker.yaml`) to:
|
||||
Kubernetes only — no RPM. Every `v*` tag builds and pushes
|
||||
`artifactapi.k8s.syd1.au.unkin.net/docker-internal/pdbmux:<tag>`
|
||||
(`.woodpecker/docker.yaml`); tag with `make patch` / `minor` / `major`.
|
||||
|
||||
```
|
||||
artifactapi.k8s.syd1.au.unkin.net/docker-internal/pdbmux:<tag>
|
||||
```
|
||||
|
||||
It is a minimal static (`CGO_ENABLED=0`) binary on a distroless base
|
||||
(`Dockerfile`), configured entirely via `PDBMUX_*` env vars, with a single HTTP
|
||||
listener and `/healthz` for liveness/readiness probes.
|
||||
|
||||
The Deployment/Service/Gateway manifests live in the estate's `argocd-apps` repo
|
||||
under `apps/base/pdbmux/` (namespace `pdbmux`, 2 replicas), and it is exposed to
|
||||
VM/workstation `node-lookup` consumers over HTTPS at:
|
||||
|
||||
```
|
||||
https://pdbmux.k8s.syd1.au.unkin.net
|
||||
```
|
||||
|
||||
Locally you can still run the binary directly for development:
|
||||
|
||||
```bash
|
||||
PDBMUX_BACKENDS='old=http://puppetdbapi.service.consul:8080,new=http://puppetdb.puppet.svc.cluster.local:8080' \
|
||||
pdbmux serve
|
||||
curl -s localhost:8080/healthz
|
||||
```
|
||||
|
||||
## Version bumps
|
||||
|
||||
```bash
|
||||
make patch # tag vX.Y.(Z+1) and push (triggers the docker release)
|
||||
make minor # tag vX.(Y+1).0
|
||||
make major # tag v(X+1).0.0
|
||||
```
|
||||
Manifests live in `argocd-apps` under `apps/base/pdbmux/` (namespace `pdbmux`,
|
||||
2 replicas). Use `/healthz` for liveness/readiness probes. Reachable from VMs and
|
||||
workstations at `https://pdbmux.k8s.syd1.au.unkin.net`.
|
||||
|
||||
@@ -16,49 +16,27 @@ const (
|
||||
configFileName = "config.yaml"
|
||||
envPrefix = "PDBMUX_"
|
||||
|
||||
// defaultListen is the default HTTP listen address.
|
||||
defaultListen = ":8080"
|
||||
// defaultOldURL / defaultNewURL are the two PuppetDBs merged during the
|
||||
// VM -> k8s migration. old = legacy Consul-registered puppetdbapi; new =
|
||||
// the k8s PuppetDB behind the gateway (TLS terminated there).
|
||||
defaultOldURL = "http://puppetdbapi.service.consul:8080"
|
||||
defaultNewURL = "https://puppetdb.k8s.syd1.au.unkin.net"
|
||||
// defaultPrimary is the backend name used for pass-through (non-merged)
|
||||
// /pdb/query/v4/* paths and as static precedence for merge fallback.
|
||||
defaultListen = ":8080"
|
||||
defaultOldURL = "http://puppetdbapi.service.consul:8080"
|
||||
defaultNewURL = "https://puppetdb.k8s.syd1.au.unkin.net"
|
||||
defaultPrimary = "new"
|
||||
|
||||
defaultTimeout = 10 * time.Second
|
||||
defaultFreshnessTTL = 30 * time.Second
|
||||
)
|
||||
|
||||
// Backend is one upstream PuppetDB. URL is the base URL (scheme://host[:port]),
|
||||
// without the /pdb/query/v4/... path — that is appended per request.
|
||||
type Backend struct {
|
||||
Name string `yaml:"name"`
|
||||
URL string `yaml:"url"`
|
||||
URL string `yaml:"url"` // base URL only; the query path is appended per request
|
||||
}
|
||||
|
||||
// Config holds every configurable value. Fields map 1:1 to config-file keys and
|
||||
// env vars (PDBMUX_*). See Load for precedence.
|
||||
type Config struct {
|
||||
// Listen is the HTTP listen address (host:port).
|
||||
Listen string `yaml:"listen"`
|
||||
// Backends is the ordered list of upstream PuppetDBs to fan out to.
|
||||
Backends []Backend `yaml:"backends"`
|
||||
// Primary is the backend Name used for transparent pass-through of
|
||||
// non-merged /pdb/query/v4/* paths.
|
||||
Primary string `yaml:"primary"`
|
||||
// Merge selects how /facts records are attributed to a backend when a
|
||||
// certname appears in both: "freshness" (query /nodes report_timestamp,
|
||||
// newer wins) or "static" (always prefer the Prefer backend).
|
||||
Merge string `yaml:"merge"`
|
||||
// Prefer names the backend that wins under static merge and as the
|
||||
// tie-breaker/fallback under freshness merge.
|
||||
Prefer string `yaml:"prefer"`
|
||||
// Timeout bounds each upstream request.
|
||||
Timeout time.Duration `yaml:"timeout"`
|
||||
// FreshnessTTL is how long a per-certname freshness map (from /nodes) is
|
||||
// cached under the "freshness" merge strategy.
|
||||
Listen string `yaml:"listen"`
|
||||
Backends []Backend `yaml:"backends"`
|
||||
Primary string `yaml:"primary"`
|
||||
Merge string `yaml:"merge"`
|
||||
Prefer string `yaml:"prefer"` // wins under static merge, and breaks ties under freshness merge
|
||||
Timeout time.Duration `yaml:"timeout"`
|
||||
FreshnessTTL time.Duration `yaml:"freshness_ttl"`
|
||||
}
|
||||
|
||||
@@ -67,8 +45,6 @@ const (
|
||||
mergeStatic = "static"
|
||||
)
|
||||
|
||||
// DefaultConfig returns the built-in defaults: both migration PuppetDBs,
|
||||
// freshness merge, "new" primary/preferred.
|
||||
func DefaultConfig() Config {
|
||||
return Config{
|
||||
Listen: defaultListen,
|
||||
@@ -84,7 +60,6 @@ func DefaultConfig() Config {
|
||||
}
|
||||
}
|
||||
|
||||
// ConfigDir returns the XDG_CONFIG_HOME/pdbmux directory.
|
||||
func ConfigDir() string {
|
||||
base := os.Getenv("XDG_CONFIG_HOME")
|
||||
if base == "" {
|
||||
@@ -94,15 +69,11 @@ func ConfigDir() string {
|
||||
return filepath.Join(base, appName)
|
||||
}
|
||||
|
||||
// ConfigPath returns the full path to the config file.
|
||||
func ConfigPath() string {
|
||||
return filepath.Join(ConfigDir(), configFileName)
|
||||
}
|
||||
|
||||
// Load reads the config file (if present), then applies env var overrides.
|
||||
// Precedence (lowest -> highest): defaults < config file < env vars < flags
|
||||
// (flags are applied by the caller). Backends can be overridden wholesale via
|
||||
// PDBMUX_BACKENDS ("name=url,name=url").
|
||||
// Precedence: defaults < config file < env vars < flags, and flags are applied by the caller.
|
||||
func Load() (Config, error) {
|
||||
cfg := DefaultConfig()
|
||||
|
||||
@@ -125,7 +96,6 @@ func Load() (Config, error) {
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
// applyEnv overlays PDBMUX_* env vars onto cfg. getenv is injected for testing.
|
||||
func applyEnv(cfg *Config, getenv func(string) string) {
|
||||
if v := getenv(envPrefix + "LISTEN"); v != "" {
|
||||
cfg.Listen = v
|
||||
@@ -156,8 +126,7 @@ func applyEnv(cfg *Config, getenv func(string) string) {
|
||||
}
|
||||
}
|
||||
|
||||
// parseBackends parses "name=url,name=url" into Backends. Entries without an
|
||||
// "=" are skipped. Used for the PDBMUX_BACKENDS env override.
|
||||
// Parses the PDBMUX_BACKENDS form "name=url,name=url"; entries without an "=" are skipped.
|
||||
func parseBackends(s string) []Backend {
|
||||
var out []Backend
|
||||
for _, part := range strings.Split(s, ",") {
|
||||
@@ -175,7 +144,6 @@ func parseBackends(s string) []Backend {
|
||||
return out
|
||||
}
|
||||
|
||||
// Validate checks the config is internally consistent and usable.
|
||||
func (c Config) Validate() error {
|
||||
if len(c.Backends) == 0 {
|
||||
return fmt.Errorf("no backends configured")
|
||||
@@ -207,8 +175,6 @@ func (c Config) Validate() error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// PrimaryBackend returns the backend named by Primary (guaranteed present after
|
||||
// Validate).
|
||||
func (c Config) PrimaryBackend() Backend {
|
||||
for _, b := range c.Backends {
|
||||
if b.Name == c.Primary {
|
||||
@@ -218,7 +184,6 @@ func (c Config) PrimaryBackend() Backend {
|
||||
return c.Backends[0]
|
||||
}
|
||||
|
||||
// writeDefaultConfig creates the config dir and writes a default config file.
|
||||
func writeDefaultConfig() error {
|
||||
dir := ConfigDir()
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
@@ -240,8 +205,6 @@ func writeDefaultConfig() error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// durationString renders a duration for `config show` (falls back to a plain
|
||||
// seconds count for zero to avoid "0s" ambiguity in logs).
|
||||
func durationString(d time.Duration) string {
|
||||
if d == 0 {
|
||||
return "0"
|
||||
|
||||
@@ -1,21 +1,4 @@
|
||||
// Command pdbmux is a small merging HTTP proxy over two PuppetDB backends.
|
||||
//
|
||||
// During the VM -> k8s Puppet migration there are two PuppetDBs — the legacy
|
||||
// Consul-registered one and the new k8s one — and nodes move between them as
|
||||
// they migrate. pdbmux presents a single merged PuppetDB v4 query surface so
|
||||
// node-lookup and pblastreport (and anything else) see one consistent view:
|
||||
//
|
||||
// - GET /pdb/query/v4/nodes — fan out to both backends, dedupe by certname,
|
||||
// keep the record with the newer report_timestamp.
|
||||
// - GET /pdb/query/v4/facts — fan out to both, and for a certname present in
|
||||
// both keep ALL facts from the backend holding that node's newer report
|
||||
// (freshness merge) or a static preferred backend (static merge).
|
||||
// - any other GET /pdb/query/v4/* — transparently proxied to the primary.
|
||||
// - GET /healthz — per-backend reachability.
|
||||
//
|
||||
// The query param is forwarded verbatim (PuppetDB AST JSON). If one backend
|
||||
// errors/times out, the other's results are served; only if both fail does a
|
||||
// merged endpoint return 502.
|
||||
// Command pdbmux serves one merged PuppetDB v4 query surface over two PuppetDBs.
|
||||
package main
|
||||
|
||||
import (
|
||||
@@ -119,8 +102,6 @@ func main() {
|
||||
}
|
||||
}
|
||||
|
||||
// runServer starts the HTTP server and blocks until SIGINT/SIGTERM, then
|
||||
// gracefully shuts down.
|
||||
func runServer(cfg Config) error {
|
||||
logger := log.New(os.Stderr, "pdbmux: ", log.LstdFlags)
|
||||
srv := NewServer(cfg, logger)
|
||||
@@ -155,7 +136,6 @@ func runServer(cfg Config) error {
|
||||
}
|
||||
}
|
||||
|
||||
// printConfig renders the active config for `config show`.
|
||||
func printConfig(cfg Config) {
|
||||
fmt.Printf("config file : %s\n", ConfigPath())
|
||||
fmt.Printf("listen : %s\n", cfg.Listen)
|
||||
|
||||
@@ -5,24 +5,18 @@ import (
|
||||
"time"
|
||||
)
|
||||
|
||||
// record is a single PuppetDB result element kept as raw JSON so unknown fields
|
||||
// survive the merge untouched. certname/report_timestamp are decoded only for
|
||||
// merge decisions.
|
||||
// Raw is kept verbatim so unknown PuppetDB fields survive the merge.
|
||||
type record struct {
|
||||
Raw json.RawMessage
|
||||
Certname string
|
||||
ReportTimestamp string // only populated for /nodes records
|
||||
}
|
||||
|
||||
// recordMeta is the subset we decode from any /nodes or /facts element to drive
|
||||
// merge decisions.
|
||||
type recordMeta struct {
|
||||
Certname string `json:"certname"`
|
||||
ReportTimestamp string `json:"report_timestamp"`
|
||||
}
|
||||
|
||||
// decodeRecords turns a raw PuppetDB JSON array into records, preserving each
|
||||
// element verbatim in Raw. A body that is not a JSON array yields (nil, err).
|
||||
func decodeRecords(body []byte) ([]record, error) {
|
||||
var raws []json.RawMessage
|
||||
if err := json.Unmarshal(body, &raws); err != nil {
|
||||
@@ -41,8 +35,7 @@ func decodeRecords(body []byte) ([]record, error) {
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// parseTimestamp parses a PuppetDB RFC3339(nano) timestamp. Zero time on
|
||||
// failure sorts oldest, so a backend with a well-formed newer timestamp wins.
|
||||
// An unparseable timestamp yields the zero time, which sorts oldest.
|
||||
func parseTimestamp(s string) time.Time {
|
||||
if s == "" {
|
||||
return time.Time{}
|
||||
@@ -53,10 +46,7 @@ func parseTimestamp(s string) time.Time {
|
||||
return time.Time{}
|
||||
}
|
||||
|
||||
// mergeNodes dedupes /nodes records by certname, keeping the one with the newer
|
||||
// report_timestamp. backends is the ordered list of (name, records) results;
|
||||
// when timestamps tie (or both are zero), the earlier backend in the slice
|
||||
// wins, so callers should order by precedence.
|
||||
// results must be ordered by precedence: ties keep the earlier backend's record.
|
||||
func mergeNodes(results []backendResult) []json.RawMessage {
|
||||
type pick struct {
|
||||
raw json.RawMessage
|
||||
@@ -73,7 +63,6 @@ func mergeNodes(results []backendResult) []json.RawMessage {
|
||||
order = append(order, rec.Certname)
|
||||
continue
|
||||
}
|
||||
// Strictly-newer wins; ties keep the existing (earlier-backend) pick.
|
||||
if ts.After(cur.ts) {
|
||||
best[rec.Certname] = pick{raw: rec.Raw, ts: ts}
|
||||
}
|
||||
@@ -86,12 +75,10 @@ func mergeNodes(results []backendResult) []json.RawMessage {
|
||||
return out
|
||||
}
|
||||
|
||||
// freshness maps certname -> backend name that holds that node's newest report.
|
||||
// certname -> name of the backend holding that node's newest report.
|
||||
type freshness map[string]string
|
||||
|
||||
// buildFreshness computes, per certname, which backend has the newer
|
||||
// report_timestamp. results must be ordered by precedence; on a tie the
|
||||
// earlier backend wins.
|
||||
// results must be ordered by precedence: ties keep the earlier backend.
|
||||
func buildFreshness(results []backendResult) freshness {
|
||||
type pick struct {
|
||||
backend string
|
||||
@@ -114,19 +101,9 @@ func buildFreshness(results []backendResult) freshness {
|
||||
return f
|
||||
}
|
||||
|
||||
// mergeFacts merges /facts records at node granularity: for each certname, all
|
||||
// facts from the winning backend are kept and the other backend's facts for
|
||||
// that certname are dropped.
|
||||
//
|
||||
// The winner is chosen per certname by `owner(certname)`. Callers supply owner
|
||||
// from either a freshness map (freshness merge) or a constant preferred backend
|
||||
// (static merge). When owner returns a backend that has no facts for a certname
|
||||
// (or a name not in results), records fall back to precedence order so a node
|
||||
// present in only one backend still appears.
|
||||
// owner names the winning backend per certname; when it holds no facts for that certname, precedence order wins.
|
||||
func mergeFacts(results []backendResult, owner func(certname string) string) []json.RawMessage {
|
||||
// Which backends actually returned facts for each certname, in precedence
|
||||
// order, so we can fall back if the chosen owner has none.
|
||||
present := map[string][]string{} // certname -> ordered backend names
|
||||
present := map[string][]string{} // certname -> backend names, in precedence order
|
||||
byKey := map[string][]json.RawMessage{}
|
||||
for _, res := range results {
|
||||
for _, rec := range res.records {
|
||||
@@ -154,7 +131,6 @@ func mergeFacts(results []backendResult, owner func(certname string) string) []j
|
||||
for _, cn := range order {
|
||||
backends := present[cn]
|
||||
chosen := owner(cn)
|
||||
// Fall back to precedence order if the chosen backend has no facts here.
|
||||
if !contains(backends, chosen) {
|
||||
chosen = backends[0]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user