3 Commits

Author SHA1 Message Date
unkinben 649373ff73 feat: dist_tag boolean for distro-aware release strings
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/build-fedora44 Pipeline was successful
ci/woodpecker/pr/build-fedora42 Pipeline was successful
ci/woodpecker/pr/build-fedora43 Pipeline was successful
ci/woodpecker/pr/build-almalinux8 Pipeline failed
ci/woodpecker/pr/build-almalinux9 Pipeline failed
Adds a per-package dist_tag: true/false metadata flag (default false).
When enabled the build tool appends the RPM dist tag to the release
at build time so each distro produces a unique version in the Gitea
package registry:

  release: 1  +  almalinux/el9  ->  PACKAGE_RELEASE=1.el9
  release: 1  +  fedora/43      ->  PACKAGE_RELEASE=1.fc43

This ensures Gitea package existence checks don't confuse packages built
for one distro with those built for another (the original bug).

Changes:
- Add effective_release() and get_rpm_dist_tag() helpers
- Revert the broken files-endpoint check from the previous commit;
  dist disambiguation is now handled purely via the release string
- discover_packages and build_single both compute the effective release
  before constructing PackageInfo
- check_package_exists simplified back to a single version-level check
- dist_tag added to METADATA_SCHEMA and PackageMetadata dataclass
- All 72 metadata.yaml files updated with dist_tag: true
2026-05-17 11:07:49 +10:00
unkinben 3de605d321 feat: add HTTP sessions and retry logic for Gitea/GitHub API calls
Creates two module-level requests.Session objects (_gitea_session,
_github_session) with an HTTPAdapter backed by urllib3 Retry:
- 3 retries with 0.5s exponential backoff
- retries on 429, 500, 502, 503, 504 and connection errors
- GET-only to avoid unsafe retries

Benefits:
- TCP connections are pooled and reused across all package checks,
  including concurrent builds in ThreadPoolExecutor
- Transient network blips no longer cause spurious rebuilds
- Per-request timeouts tightened to 10s (Gitea) / 15s (GitHub)
  so failures surface quickly and the retry budget is actually useful
2026-05-17 10:38:07 +10:00
unkinben 9ca3340f10 fix: make package existence check distro-aware
check_package_exists was only checking name+version+release, so an
almalinux package already in Gitea would cause fedora builds to be
skipped incorrectly.

Now passes the target distro from PackageInfo through to
check_package_exists, which maps it to an RPM dist suffix (.el9,
.fc43, etc.) and queries the Gitea package files endpoint to confirm
a distro-specific file exists before skipping the build.

Also adds fedora/42, fedora/43, fedora/44 to the allowed repository
values in the metadata schema.
2026-05-17 01:37:25 +10:00
35 changed files with 124 additions and 666 deletions
-18
View File
@@ -31,24 +31,6 @@ repos:
"-s", "-s",
] ]
- repo: local
hooks:
- id: pytest
name: Run unit tests
entry: make test
language: system
types: [python]
pass_filenames: false
- repo: https://github.com/python-jsonschema/check-jsonschema
rev: 0.37.2
hooks:
- id: check-jsonschema
name: Validate RPM package metadata
files: ^rpms/[^/]+/metadata\.yaml$
args: [--schemafile, schema/metadata.json]
language_version: python3.11
- repo: https://github.com/astral-sh/ruff-pre-commit - repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.14.7 rev: v0.14.7
hooks: hooks:
+4 -10
View File
@@ -31,18 +31,12 @@ steps:
image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest
commands: commands:
- | - |
HOST="https://artifactapi.k8s.syd1.au.unkin.net"
REPO="rpm-vendor-el8"
for rpm in $(find /woodpecker/src/git.unkin.net/unkin/rpmbuilder/ -type f -name "*.rpm"); do for rpm in $(find /woodpecker/src/git.unkin.net/unkin/rpmbuilder/ -type f -name "*.rpm"); do
FILE=$$(basename "$$rpm") curl --user droneci:$${DRONECI_PASSWORD} --upload-file $rpm https://git.unkin.net/api/packages/unkin/rpm/almalinux/el8/upload
# artifactapi has no HEAD route (405); probe the served Packages path.
code=$$(curl -s -o /dev/null -w '%{http_code}' "$$HOST/api/v2/remotes/$$REPO/files/Packages/$$FILE" || true)
if [ "$$code" = "200" ]; then
echo "$$FILE already exists in $$REPO; skipping"
continue
fi
curl -f -X PUT "$$HOST/api/v2/remotes/$$REPO/files/$$FILE" -H "Content-Type: application/x-rpm" --data-binary @"$$rpm"
done done
environment:
DRONECI_PASSWORD:
from_secret: DRONECI_PASSWORD
backend_options: backend_options:
kubernetes: kubernetes:
serviceAccountName: default serviceAccountName: default
+4 -10
View File
@@ -31,18 +31,12 @@ steps:
image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest
commands: commands:
- | - |
HOST="https://artifactapi.k8s.syd1.au.unkin.net"
REPO="rpm-vendor-el9"
for rpm in $(find /woodpecker/src/git.unkin.net/unkin/rpmbuilder/ -type f -name "*.rpm"); do for rpm in $(find /woodpecker/src/git.unkin.net/unkin/rpmbuilder/ -type f -name "*.rpm"); do
FILE=$$(basename "$$rpm") curl --user droneci:$${DRONECI_PASSWORD} --upload-file $rpm https://git.unkin.net/api/packages/unkin/rpm/almalinux/el9/upload
# artifactapi has no HEAD route (405); probe the served Packages path.
code=$$(curl -s -o /dev/null -w '%{http_code}' "$$HOST/api/v2/remotes/$$REPO/files/Packages/$$FILE" || true)
if [ "$$code" = "200" ]; then
echo "$$FILE already exists in $$REPO; skipping"
continue
fi
curl -f -X PUT "$$HOST/api/v2/remotes/$$REPO/files/$$FILE" -H "Content-Type: application/x-rpm" --data-binary @"$$rpm"
done done
environment:
DRONECI_PASSWORD:
from_secret: DRONECI_PASSWORD
backend_options: backend_options:
kubernetes: kubernetes:
serviceAccountName: default serviceAccountName: default
+4 -10
View File
@@ -31,18 +31,12 @@ steps:
image: git.unkin.net/unkin/fedora42-rpmbuilder:latest image: git.unkin.net/unkin/fedora42-rpmbuilder:latest
commands: commands:
- | - |
HOST="https://artifactapi.k8s.syd1.au.unkin.net"
REPO="rpm-vendor-f42"
for rpm in $(find /woodpecker/src/git.unkin.net/unkin/rpmbuilder/ -type f -name "*.rpm"); do for rpm in $(find /woodpecker/src/git.unkin.net/unkin/rpmbuilder/ -type f -name "*.rpm"); do
FILE=$$(basename "$$rpm") curl --user droneci:$${DRONECI_PASSWORD} --upload-file $rpm https://git.unkin.net/api/packages/unkin/rpm/fedora/42/upload
# artifactapi has no HEAD route (405); probe the served Packages path.
code=$$(curl -s -o /dev/null -w '%{http_code}' "$$HOST/api/v2/remotes/$$REPO/files/Packages/$$FILE" || true)
if [ "$$code" = "200" ]; then
echo "$$FILE already exists in $$REPO; skipping"
continue
fi
curl -f -X PUT "$$HOST/api/v2/remotes/$$REPO/files/$$FILE" -H "Content-Type: application/x-rpm" --data-binary @"$$rpm"
done done
environment:
DRONECI_PASSWORD:
from_secret: DRONECI_PASSWORD
backend_options: backend_options:
kubernetes: kubernetes:
serviceAccountName: default serviceAccountName: default
+4 -10
View File
@@ -31,18 +31,12 @@ steps:
image: git.unkin.net/unkin/fedora43-rpmbuilder:latest image: git.unkin.net/unkin/fedora43-rpmbuilder:latest
commands: commands:
- | - |
HOST="https://artifactapi.k8s.syd1.au.unkin.net"
REPO="rpm-vendor-f43"
for rpm in $(find /woodpecker/src/git.unkin.net/unkin/rpmbuilder/ -type f -name "*.rpm"); do for rpm in $(find /woodpecker/src/git.unkin.net/unkin/rpmbuilder/ -type f -name "*.rpm"); do
FILE=$$(basename "$$rpm") curl --user droneci:$${DRONECI_PASSWORD} --upload-file $rpm https://git.unkin.net/api/packages/unkin/rpm/fedora/43/upload
# artifactapi has no HEAD route (405); probe the served Packages path.
code=$$(curl -s -o /dev/null -w '%{http_code}' "$$HOST/api/v2/remotes/$$REPO/files/Packages/$$FILE" || true)
if [ "$$code" = "200" ]; then
echo "$$FILE already exists in $$REPO; skipping"
continue
fi
curl -f -X PUT "$$HOST/api/v2/remotes/$$REPO/files/$$FILE" -H "Content-Type: application/x-rpm" --data-binary @"$$rpm"
done done
environment:
DRONECI_PASSWORD:
from_secret: DRONECI_PASSWORD
backend_options: backend_options:
kubernetes: kubernetes:
serviceAccountName: default serviceAccountName: default
+4 -10
View File
@@ -31,18 +31,12 @@ steps:
image: git.unkin.net/unkin/fedora44-rpmbuilder:latest image: git.unkin.net/unkin/fedora44-rpmbuilder:latest
commands: commands:
- | - |
HOST="https://artifactapi.k8s.syd1.au.unkin.net"
REPO="rpm-vendor-f44"
for rpm in $(find /woodpecker/src/git.unkin.net/unkin/rpmbuilder/ -type f -name "*.rpm"); do for rpm in $(find /woodpecker/src/git.unkin.net/unkin/rpmbuilder/ -type f -name "*.rpm"); do
FILE=$$(basename "$$rpm") curl --user droneci:$${DRONECI_PASSWORD} --upload-file $rpm https://git.unkin.net/api/packages/unkin/rpm/fedora/44/upload
# artifactapi has no HEAD route (405); probe the served Packages path.
code=$$(curl -s -o /dev/null -w '%{http_code}' "$$HOST/api/v2/remotes/$$REPO/files/Packages/$$FILE" || true)
if [ "$$code" = "200" ]; then
echo "$$FILE already exists in $$REPO; skipping"
continue
fi
curl -f -X PUT "$$HOST/api/v2/remotes/$$REPO/files/$$FILE" -H "Content-Type: application/x-rpm" --data-binary @"$$rpm"
done done
environment:
DRONECI_PASSWORD:
from_secret: DRONECI_PASSWORD
backend_options: backend_options:
kubernetes: kubernetes:
serviceAccountName: default serviceAccountName: default
+1 -5
View File
@@ -11,7 +11,7 @@ DISTRO ?= almalinux/el9
PACKAGES := $(shell find $(ROOT_DIR)/rpms -mindepth 1 -maxdepth 1 -type d -exec test -f {}/metadata.yaml \; -print | xargs -n1 basename | sort) PACKAGES := $(shell find $(ROOT_DIR)/rpms -mindepth 1 -maxdepth 1 -type d -exec test -f {}/metadata.yaml \; -print | xargs -n1 basename | sort)
# Default target to build all packages # Default target to build all packages
.PHONY: all list build clean test .PHONY: all list build clean
all: build-all all: build-all
# List all available packages # List all available packages
@@ -47,10 +47,6 @@ dry-run:
@echo "Dry run - showing what would be built for distro $(DISTRO):" @echo "Dry run - showing what would be built for distro $(DISTRO):"
$(BUILD_TOOL) build-all --distro $(DISTRO) --dry-run $(BUILD_TOOL) build-all --distro $(DISTRO) --dry-run
# Run unit tests
test:
@uv run --group dev pytest tests/ -q; rc=$$?; [ $$rc -eq 5 ] && exit 0 || exit $$rc
# Clean target # Clean target
clean: clean:
@echo "Cleaning build artifacts..." @echo "Cleaning build artifacts..."
-16
View File
@@ -1,16 +0,0 @@
[project]
name = "rpmbuilder"
version = "0.1.0"
requires-python = ">=3.11"
[dependency-groups]
dev = [
"pytest>=8",
"jsonschema>=4",
"pyyaml>=6",
# tools/build's own script dependencies, so tests can import it
"typer",
"requests",
"hvac",
"cerberus",
]
-24
View File
@@ -1,24 +0,0 @@
name: argocd
github: argoproj/argo-cd
github_release_pattern: ^v3\.3\.
description: Declarative GitOps continuous delivery for Kubernetes - command line
client.
arch: amd64
platform: linux
maintainer: Argo Project
homepage: https://github.com/argoproj/argo-cd
license: Apache-2.0
dist_tag: true
builds:
- repository:
- almalinux/el9
image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest
release: 1
version: 3.3.14
- repository:
- fedora/42
- fedora/43
- fedora/44
image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest
release: 1
version: 3.3.14
-22
View File
@@ -1,22 +0,0 @@
#!/usr/bin/bash
set -e
BASE_URL="https://artifactapi.k8s.syd1.au.unkin.net/api/v1/remote/github/argoproj/argo-cd/releases/download/v${PACKAGE_VERSION}"
wget -O /app/argocd "${BASE_URL}/argocd-linux-amd64"
wget -O /app/cli_checksums.txt "${BASE_URL}/cli_checksums.txt"
# Upstream lists the asset name; check it against the local filename.
cd /app
grep ' argocd-linux-amd64$' cli_checksums.txt | sed 's/argocd-linux-amd64$/argocd/' | sha256sum --check --strict -
chmod +x /app/argocd
mkdir -p /app/completions
/app/argocd completion bash > /app/completions/argocd
/app/argocd completion zsh > /app/completions/_argocd
/app/argocd completion fish > /app/completions/argocd.fish
envsubst < /app/resources/nfpm.yaml > /app/nfpm.yaml
nfpm pkg --config /app/nfpm.yaml --target /app/dist --packager rpm
-48
View File
@@ -1,48 +0,0 @@
# nfpm.yaml
name: ${PACKAGE_NAME}
version: ${PACKAGE_VERSION}
release: ${PACKAGE_RELEASE}
arch: ${PACKAGE_ARCH}
platform: ${PACKAGE_PLATFORM}
section: default
priority: extra
description: "${PACKAGE_DESCRIPTION}"
maintainer: ${PACKAGE_MAINTAINER}
homepage: ${PACKAGE_HOMEPAGE}
license: ${PACKAGE_LICENSE}
disable_globbing: false
replaces:
- argocd
provides:
- argocd
contents:
- src: /app/argocd
dst: /usr/bin/argocd
file_info:
mode: 0755
owner: root
group: root
- src: /app/completions/argocd
dst: /usr/share/bash-completion/completions/argocd
file_info:
mode: 0644
owner: root
group: root
- src: /app/completions/_argocd
dst: /usr/share/zsh/site-functions/_argocd
file_info:
mode: 0644
owner: root
group: root
- src: /app/completions/argocd.fish
dst: /usr/share/fish/vendor_completions.d/argocd.fish
file_info:
mode: 0644
owner: root
group: root
+3 -2
View File
@@ -11,9 +11,10 @@ builds:
release: 1 release: 1
repository: repository:
- almalinux/el8 - almalinux/el8
version: 2.1.156 version: 2.1.126
- image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest - image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest
release: 1 release: 1
repository: repository:
- almalinux/el9 - almalinux/el9
version: 2.1.156 version: 2.1.126
claude_ai: true
+2 -2
View File
@@ -13,9 +13,9 @@ builds:
- almalinux/el8 - almalinux/el8
image: git.unkin.net/unkin/almalinux8-rpmbuilder:latest image: git.unkin.net/unkin/almalinux8-rpmbuilder:latest
release: 1 release: 1
version: 3.6.10 version: 3.6.7
- repository: - repository:
- almalinux/el9 - almalinux/el9
image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest
release: 1 release: 1
version: 3.6.10 version: 3.6.7
-22
View File
@@ -1,22 +0,0 @@
name: go-cache-plugin
github: tailscale/go-cache-plugin
description: A GOCACHEPROG implementation that backs the Go build cache with S3.
arch: amd64
platform: linux
maintainer: Tailscale
homepage: https://github.com/tailscale/go-cache-plugin
license: BSD-3-Clause
dist_tag: true
builds:
- repository:
- almalinux/el9
image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest
release: 1
version: 2026.7.22
- repository:
- fedora/42
- fedora/43
- fedora/44
image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest
release: 1
version: 2026.7.22
-17
View File
@@ -1,17 +0,0 @@
#!/usr/bin/bash
set -e
# Upstream publishes no tags or releases; PACKAGE_VERSION dates this commit.
COMMIT=3031b5d01c50d2748a32c7c9386ea7049883f38e
# go.mod requires go 1.26.1
export GOTOOLCHAIN=go1.26.1
# Compile the go-cache-plugin binary using Go
GOBIN=/app go install github.com/tailscale/go-cache-plugin/cmd/go-cache-plugin@${COMMIT}
# Process nfpm.yaml with envsubst
envsubst < /app/resources/nfpm.yaml > /app/nfpm.yaml
# Build the RPM
nfpm pkg --config /app/nfpm.yaml --target /app/dist --packager rpm
-31
View File
@@ -1,31 +0,0 @@
# nfpm.yaml
name: ${PACKAGE_NAME}
version: ${PACKAGE_VERSION}
release: ${PACKAGE_RELEASE}
arch: ${PACKAGE_ARCH}
platform: ${PACKAGE_PLATFORM}
section: default
priority: extra
description: "${PACKAGE_DESCRIPTION}"
maintainer: ${PACKAGE_MAINTAINER}
homepage: ${PACKAGE_HOMEPAGE}
license: ${PACKAGE_LICENSE}
disable_globbing: false
replaces:
- go-cache-plugin
provides:
- go-cache-plugin
# Files to include in the package
contents:
- src: /app/go-cache-plugin
dst: /usr/bin/go-cache-plugin
file_info:
mode: 0755
owner: root
group: root
+1 -1
View File
@@ -6,7 +6,7 @@ platform: linux
maintainer: unkin maintainer: unkin
homepage: https://github.com/jellyfin/jellyfin-ffmpeg homepage: https://github.com/jellyfin/jellyfin-ffmpeg
license: GPL-3.0 license: GPL-3.0
dist_tag: false dist_tag: true
builds: builds:
- repository: - repository:
- almalinux/el8 - almalinux/el8
+1 -1
View File
@@ -1,7 +1,7 @@
name: jellyfin-server name: jellyfin-server
github: unknown/jellyfin-server github: unknown/jellyfin-server
description: jellyfin-server package description: jellyfin-server package
dist_tag: false dist_tag: true
builds: builds:
- image: git.unkin.net/unkin/almalinux8-rpmbuilder:latest - image: git.unkin.net/unkin/almalinux8-rpmbuilder:latest
release: '1' release: '1'
+1 -1
View File
@@ -1,7 +1,7 @@
name: jellyfin-web name: jellyfin-web
github: unknown/jellyfin-web github: unknown/jellyfin-web
description: jellyfin-web package description: jellyfin-web package
dist_tag: false dist_tag: true
builds: builds:
- image: git.unkin.net/unkin/almalinux8-rpmbuilder:latest - image: git.unkin.net/unkin/almalinux8-rpmbuilder:latest
release: '1' release: '1'
+2 -2
View File
@@ -13,9 +13,9 @@ builds:
- almalinux/el8 - almalinux/el8
image: git.unkin.net/unkin/almalinux8-rpmbuilder:latest image: git.unkin.net/unkin/almalinux8-rpmbuilder:latest
release: 1 release: 1
version: '26.2' version: '26.1'
- repository: - repository:
- almalinux/el9 - almalinux/el9
image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest
release: 1 release: 1
version: '26.2' version: '26.1'
+2 -6
View File
@@ -2,10 +2,6 @@
set -e set -e
# Download the pre-built RPM from GitHub releases. # Download the pre-built RPM from GitHub releases
# Upstream always publishes the release-1 asset (nzbget-<version>-1.x86_64.rpm);
# the source URL must use the upstream asset name, not PACKAGE_RELEASE, which
# carries the dist tag (e.g. 1.el9) and does not exist upstream. Only the local
# output filename is dist-tagged, mirroring the code-server package.
curl -L -o /app/dist/nzbget-${PACKAGE_VERSION}-${PACKAGE_RELEASE}.x86_64.rpm \ curl -L -o /app/dist/nzbget-${PACKAGE_VERSION}-${PACKAGE_RELEASE}.x86_64.rpm \
https://artifactapi.k8s.syd1.au.unkin.net/api/v1/remote/github/nzbgetcom/nzbget/releases/download/v$PACKAGE_VERSION/nzbget-${PACKAGE_VERSION}-1.x86_64.rpm https://artifactapi.k8s.syd1.au.unkin.net/api/v1/remote/github/nzbgetcom/nzbget/releases/download/v$PACKAGE_VERSION/nzbget-${PACKAGE_VERSION}-${PACKAGE_RELEASE}.x86_64.rpm
-38
View File
@@ -1,38 +0,0 @@
# puppet-initial
A firstrun bootstrap script and oneshot systemd service that initialises a
freshly-provisioned host into Puppet:
1. Sets the FQDN under `.main.unkin.net`.
2. Fetches the Puppet CA certificate from the CA service.
3. Registers the node with a noop agent run against the CA.
4. Runs the agent a few times against the compile master, then enables the
`puppet` service and disables itself.
## Puppet CA endpoint
The CA endpoint defaults to the in-cluster puppetserver CA service
`puppetca.k8s.syd1.au.unkin.net:8140` (serving the standard
`/puppet-ca/v1/certificate/ca` API).
It is overridable via the environment. The `puppet-initial.service` unit reads
`/etc/sysconfig/puppet-initial` (`EnvironmentFile=-`, so the file is optional),
which the RPM ships as a commented `%config(noreplace)` example:
| Variable | Default | Purpose |
|-----------------|----------------------------------|-------------------------------------------------------------|
| `PUPPETCA_HOST` | `puppetca.k8s.syd1.au.unkin.net` | CA hostname (CA cert fetch + `--server` for registration). |
| `PUPPETCA_PORT` | `8140` | CA API port. |
### Overriding from kickstart
A kickstart `%post` can point a host at a different CA without rebuilding the
RPM by writing the sysconfig file before the service starts:
```bash
%post
cat > /etc/sysconfig/puppet-initial <<'EOF'
PUPPETCA_HOST=puppetca.k8s.syd1.au.unkin.net
PUPPETCA_PORT=8140
EOF
```
+2 -2
View File
@@ -11,9 +11,9 @@ builds:
release: '1' release: '1'
repository: repository:
- almalinux/el8 - almalinux/el8
version: 1.0.5 version: 1.0.3
- image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest - image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest
release: '1' release: '1'
repository: repository:
- almalinux/el9 - almalinux/el9
version: 1.0.5 version: 1.0.3
+1 -8
View File
@@ -15,7 +15,7 @@ license: ${PACKAGE_LICENSE}
disable_globbing: false disable_globbing: false
depends: depends:
- openvox-agent - puppet-agent
# Files to include in the package # Files to include in the package
contents: contents:
@@ -31,13 +31,6 @@ contents:
mode: 0644 mode: 0644
owner: root owner: root
group: root group: root
- src: /app/resources/puppet-initial.sysconfig
dst: /etc/sysconfig/puppet-initial
type: config|noreplace
file_info:
mode: 0644
owner: root
group: root
# Scripts to run during installation/removal (optional) # Scripts to run during installation/removal (optional)
scripts: scripts:
+2 -9
View File
@@ -1,21 +1,14 @@
#!/bin/bash #!/bin/bash
# Puppet CA endpoint. Overridable via the environment (systemd reads
# /etc/sysconfig/puppet-initial via EnvironmentFile), so kickstart %post can
# point a host at a different CA without rebuilding the RPM. Defaults to the
# in-cluster puppetserver CA service.
PUPPETCA_HOST="${PUPPETCA_HOST:-puppetca.k8s.syd1.au.unkin.net}"
PUPPETCA_PORT="${PUPPETCA_PORT:-8140}"
# Ensure the hostname is set # Ensure the hostname is set
hostnamectl set-hostname $(hostname -s).main.unkin.net hostnamectl set-hostname $(hostname -s).main.unkin.net
grep '^HOSTNAME=' /etc/sysconfig/network | cut -d= -f2 | grep -q '\.' || sed -i 's/^\(HOSTNAME=[^\.]*\)$/\1.main.unkin.net/' /etc/sysconfig/network grep '^HOSTNAME=' /etc/sysconfig/network | cut -d= -f2 | grep -q '\.' || sed -i 's/^\(HOSTNAME=[^\.]*\)$/\1.main.unkin.net/' /etc/sysconfig/network
# Install CA for Puppet # Install CA for Puppet
test -f /etc/puppetlabs/puppet/ssl/certs/ca.pem || mkdir -p /etc/puppetlabs/puppet/ssl/certs && wget --no-check-certificate "https://${PUPPETCA_HOST}:${PUPPETCA_PORT}/puppet-ca/v1/certificate/ca" -O /etc/puppetlabs/puppet/ssl/certs/ca.pem test -f /etc/puppetlabs/puppet/ssl/certs/ca.pem || mkdir -p /etc/puppetlabs/puppet/ssl/certs && wget --no-check-certificate https://puppetca.query.consul:8140/puppet-ca/v1/certificate/ca -O /etc/puppetlabs/puppet/ssl/certs/ca.pem
# Registering to Puppet server # Registering to Puppet server
/opt/puppetlabs/bin/puppet agent --test --server "${PUPPETCA_HOST}" --noop --onetime --no-daemonize --verbose /opt/puppetlabs/bin/puppet agent --test --server puppetca.query.consul --noop --onetime --no-daemonize --verbose
# Running Puppet agent five times with a 30-second gap between each run, stop puppet service at the end of each run # Running Puppet agent five times with a 30-second gap between each run, stop puppet service at the end of each run
for i in {1..5}; do for i in {1..5}; do
@@ -5,7 +5,6 @@ Wants=network-online.target
[Service] [Service]
Type=simple Type=simple
EnvironmentFile=-/etc/sysconfig/puppet-initial
ExecStart=/usr/local/bin/puppet-initial ExecStart=/usr/local/bin/puppet-initial
RemainAfterExit=true RemainAfterExit=true
ExecStop=/bin/true ExecStop=/bin/true
@@ -1,13 +0,0 @@
# Environment overrides for the puppet-initial firstrun bootstrap.
# Read by the puppet-initial.service unit (EnvironmentFile=-/etc/sysconfig/puppet-initial).
# A kickstart %post can write this file to point a host at a different Puppet CA
# without rebuilding the RPM. All values are optional; the defaults below match
# the shipped in-cluster puppetserver CA service.
# Hostname of the Puppet CA service. Used both to fetch the CA certificate
# (https://<host>:<port>/puppet-ca/v1/certificate/ca) and as --server for the
# initial noop agent registration run.
#PUPPETCA_HOST=puppetca.k8s.syd1.au.unkin.net
# Port the Puppet CA API listens on.
#PUPPETCA_PORT=8140
+1 -1
View File
@@ -6,7 +6,7 @@ set -e
dnf install -y unzip dnf install -y unzip
# Download and extract vault # Download and extract vault
curl -L -o /app/vault.zip https://artifactapi.k8s.syd1.au.unkin.net/api/v1/remote/hashicorp-releases/vault/${PACKAGE_VERSION}/vault_${PACKAGE_VERSION}_linux_amd64.zip curl -L -o /app/vault.zip https://releases.hashicorp.com/vault/${PACKAGE_VERSION}/vault_${PACKAGE_VERSION}_linux_amd64.zip
unzip vault.zip unzip vault.zip
# Process the nfpm.yaml template with environment variables # Process the nfpm.yaml template with environment variables
-111
View File
@@ -1,111 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"description": "Schema for rpms/*/metadata.yaml files",
"properties": {
"arch": {
"enum": [
"amd64",
"arm64",
"x86_64"
],
"type": "string"
},
"builds": {
"items": {
"additionalProperties": false,
"properties": {
"image": {
"minLength": 1,
"pattern": "^[a-zA-Z0-9][a-zA-Z0-9\\-_.:/@]+$",
"type": "string"
},
"release": {
"oneOf": [
{
"minLength": 1,
"type": "string"
},
{
"type": "number"
}
]
},
"repository": {
"items": {
"enum": [
"almalinux/el8",
"almalinux/el9",
"fedora/42",
"fedora/43",
"fedora/44"
],
"type": "string"
},
"minItems": 1,
"type": "array"
},
"version": {
"minLength": 1,
"pattern": "^[0-9]+(\\.[0-9]+)*(-[a-zA-Z0-9]+)*$",
"type": "string"
}
},
"required": [
"repository",
"image",
"release",
"version"
],
"type": "object"
},
"minItems": 1,
"type": "array"
},
"description": {
"minLength": 1,
"type": "string"
},
"dist_tag": {
"type": "boolean"
},
"github": {
"minLength": 1,
"pattern": "^[a-zA-Z0-9\\-_]+/[a-zA-Z0-9\\-_.]+$",
"type": "string"
},
"github_release_pattern": {
"minLength": 1,
"type": "string"
},
"homepage": {
"minLength": 1,
"pattern": "^https?://.+",
"type": "string"
},
"license": {
"minLength": 1,
"type": "string"
},
"maintainer": {
"minLength": 1,
"type": "string"
},
"name": {
"minLength": 1,
"pattern": "^[a-zA-Z0-9][a-zA-Z0-9\\-_.]*$",
"type": "string"
},
"platform": {
"minLength": 1,
"type": "string"
}
},
"required": [
"name",
"description",
"builds"
],
"title": "RPM Package Metadata",
"type": "object"
}
View File
-2
View File
@@ -1,2 +0,0 @@
# Tests for tools/build and tools/update-gh.
# See https://git.unkin.net/unkin/rpmbuilder/issues/162
-76
View File
@@ -1,76 +0,0 @@
"""Tests for the RPM filename tools/build reconstructs when probing artifactapi.
The expected values below were captured from the real nfpm; they are the names
the publish step actually uploads, so the existence probe has to ask for
exactly these or it 404s and rebuilds forever.
"""
import importlib.machinery
import importlib.util
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).parent.parent
_loader = importlib.machinery.SourceFileLoader(
"rpmbuilder_build", str(REPO_ROOT / "tools" / "build")
)
_spec = importlib.util.spec_from_loader(_loader.name, _loader)
build = importlib.util.module_from_spec(_spec)
_loader.exec_module(build)
# nfpm output for `version: <input>` with release 1, arch amd64
NFPM_VERSIONS = [
("2025.08.03", "2025.8.3"),
("2025.07.13", "2025.7.13"),
("1.05.0", "1.5.0"),
("10.02.01", "10.2.1"),
("0.0.09", "0.0.9"),
("0.6.1", "0.6.1"),
("1.2.3", "1.2.3"),
("1.05", "1.5.0"),
("08", "8.0.0"),
("1.0.0-rc1", "1.0.0~rc1"),
("1.05.0-rc1", "1.5.0~rc1"),
("1.0.0-rc.1", "1.0.0~rc.1"),
# not semver: nfpm leaves these verbatim, so we must too
("1.02.3.4", "1.02.3.4"),
("2025.08.03.01", "2025.08.03.01"),
("1.2.3.4", "1.2.3.4"),
("1.0.0-rc.01", "1.0.0-rc.01"),
]
@pytest.mark.parametrize("version,expected", NFPM_VERSIONS)
def test_nfpm_rpm_version(version, expected):
assert build.nfpm_rpm_version(version) == expected
def test_rpm_file_name_matches_nfpm_output():
assert (
build.rpm_file_name("nzbget_exporter", "2025.08.03", "1.el9", "amd64")
== "nzbget_exporter-2025.8.3-1.el9.x86_64.rpm"
)
def test_check_package_exists_probes_published_filename(monkeypatch):
"""A zero-padded version must not probe a filename nfpm can never produce."""
probed = []
class _Session:
def get(self, url, timeout=None):
probed.append(url)
return type("R", (), {"status_code": 404})()
monkeypatch.setattr(build, "_artifactapi_session", _Session())
assert not build.check_package_exists(
"nzbget_exporter", "2025.08.03", "1.el9", "almalinux/el9", "amd64"
)
expected = (
"https://artifactapi.k8s.syd1.au.unkin.net/api/v2/remotes/rpm-vendor-el9"
"/files/Packages/nzbget_exporter-2025.8.3-1.el9.x86_64.rpm"
)
assert probed == [expected]
-36
View File
@@ -1,36 +0,0 @@
"""Validate every rpms/*/metadata.yaml against schema/metadata.json."""
import json
from pathlib import Path
import jsonschema
import pytest
import yaml
REPO_ROOT = Path(__file__).parent.parent
SCHEMA_FILE = REPO_ROOT / "schema" / "metadata.json"
RPMS_DIR = REPO_ROOT / "rpms"
@pytest.fixture(scope="session")
def schema():
with open(SCHEMA_FILE) as f:
return json.load(f)
def metadata_files():
return sorted(RPMS_DIR.glob("*/metadata.yaml"))
@pytest.mark.parametrize("metadata_file", metadata_files(), ids=lambda p: p.parent.name)
def test_metadata_valid(metadata_file, schema):
with open(metadata_file) as f:
data = yaml.safe_load(f)
validator = jsonschema.Draft7Validator(schema)
errors = sorted(validator.iter_errors(data), key=str)
assert not errors, "\n".join(
f" {'.'.join(str(p) for p in e.absolute_path) or '(root)'}: {e.message}"
for e in errors
)
+85 -101
View File
@@ -63,16 +63,6 @@ def _make_session(retries: int = 3, backoff_factor: float = 0.5) -> requests.Ses
# requests.Session is thread-safe for concurrent requests. # requests.Session is thread-safe for concurrent requests.
_gitea_session = _make_session() _gitea_session = _make_session()
_github_session = _make_session() _github_session = _make_session()
_artifactapi_session = _make_session()
# artifactapi serves an internally-signed TLS cert. Verify against the OS CA
# bundle (which includes the internal CA, as dnf/curl in the builder images do)
# rather than certifi's Mozilla bundle, which does not trust it.
_artifactapi_session.verify = os.getenv('ARTIFACTAPI_CA_BUNDLE') or next(
(p for p in ('/etc/pki/tls/certs/ca-bundle.crt',
'/etc/ssl/certs/ca-certificates.crt')
if os.path.exists(p)),
True,
)
# ==================== VALIDATION SCHEMA ==================== # ==================== VALIDATION SCHEMA ====================
@@ -517,38 +507,37 @@ def get_github_token() -> str:
# ==================== GITEA API FUNCTIONS ==================== # ==================== GITEA API FUNCTIONS ====================
def nfpm_rpm_version(version: str) -> str: def normalize_version(version: str) -> str:
""" """
Render a metadata version the way nfpm renders it into an RPM filename. Normalize version string by removing leading zeros from numeric components.
Gitea automatically does this normalization.
nfpm re-renders any semver-parseable version: leading zeros are dropped,
missing components padded to three, and the prerelease joined with '~'.
Anything semver cannot parse (four or more components, non-numeric
components, a numeric prerelease identifier with a leading zero) is used
verbatim.
Examples: Examples:
"2025.08.03" -> "2025.8.3" "2025.08.03" -> "2025.8.3"
"1.05" -> "1.5.0" "1.05.0" -> "1.5.0"
"1.2.3-rc1" -> "1.2.3~rc1" "0.6.1" -> "0.6.1" (no change needed)
"1.02.3.4" -> "1.02.3.4" (not semver, left alone)
Args:
version: Original version string
Returns:
Normalized version string
""" """
core, sep, prerelease = version.partition('-') import re
if sep and not prerelease:
return version
components = core.split('.') # Split by common separators and normalize each numeric part
if len(components) > 3 or not all(c.isdigit() for c in components): parts = re.split(r'([.\-_])', version)
return version normalized_parts = []
# semver rejects numeric prerelease identifiers with a leading zero for part in parts:
if any(i.isdigit() and len(i) > 1 and i.startswith('0') # If this part is purely numeric and has leading zeros, remove them
for i in prerelease.split('.')): if part.isdigit() and len(part) > 1 and part.startswith('0'):
return version # Remove leading zeros but keep at least one digit
normalized_parts.append(str(int(part)))
else:
normalized_parts.append(part)
components += ['0'] * (3 - len(components)) return ''.join(normalized_parts)
rendered = '.'.join(str(int(c)) for c in components)
return f"{rendered}~{prerelease}" if prerelease else rendered
def get_rpm_dist_tag(distro: str) -> str: def get_rpm_dist_tag(distro: str) -> str:
@@ -578,88 +567,86 @@ def effective_release(base_release: str, distro: str, use_dist_tag: bool) -> str
return f"{base_release}.{tag}" if tag else base_release return f"{base_release}.{tag}" if tag else base_release
def get_vendor_repo(distro: str) -> str: def check_package_exists(package_name: str, version: str, release: str) -> bool:
""" """
Map a distro path to its artifactapi rpm-vendor local repo name. Check if a package version exists in the Gitea package registry.
Examples: Distro disambiguation is handled by the caller via the release string:
'almalinux/el8' -> 'rpm-vendor-el8' when dist_tag is enabled the caller appends the dist tag to release
'almalinux/el9' -> 'rpm-vendor-el9' (e.g. '1.el9', '1.fc43') so each distro maps to a unique version entry.
'fedora/42' -> 'rpm-vendor-f42'
"""
if distro.startswith('almalinux/'):
return 'rpm-vendor-' + distro.split('/', 1)[1]
if distro.startswith('fedora/'):
return 'rpm-vendor-f' + distro.split('/', 1)[1]
return 'rpm-vendor'
def get_rpm_arch(arch: str) -> str:
"""Map a metadata arch to the RPM architecture used in built filenames."""
return {'amd64': 'x86_64', 'arm64': 'aarch64'}.get(arch, arch)
def rpm_file_name(package_name: str, version: str, release: str, arch: str) -> str:
"""Filename nfpm produces for this package, and therefore the published name."""
return (
f"{package_name}-{nfpm_rpm_version(version)}-{release}"
f".{get_rpm_arch(arch)}.rpm"
)
def check_package_exists(
package_name: str,
version: str,
release: str,
distro: str,
arch: str = 'amd64',
) -> bool:
"""
Check if a built RPM already exists in the artifactapi rpm-vendor repo.
Each distro publishes to its own local repo (rpm-vendor-el9, rpm-vendor-f42,
...), and dist_tag disambiguation is carried in the release string (e.g.
'1.el9', '1.fc43'), so the served RPM filename is unique per distro. We probe
the served path under Packages/ with a GET (artifactapi has no HEAD route).
Args: Args:
package_name: Name of the package package_name: Name of the package
version: Version string version: Version string
release: Effective release (may include dist tag suffix, e.g. '1.el9') release: Effective release (may include dist tag suffix)
distro: Target distro (e.g. 'almalinux/el9', 'fedora/42')
arch: Package arch from metadata (default 'amd64')
Returns: Returns:
True if the RPM is already published, False otherwise True if package exists, False otherwise
""" """
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
base_url = os.getenv('ARTIFACTAPI_URL', 'https://artifactapi.k8s.syd1.au.unkin.net') # Get configuration from environment
repo = get_vendor_repo(distro) base_url = os.getenv('GITEA_URL', 'https://git.unkin.net')
rpm_file = rpm_file_name(package_name, version, release, arch) owner = os.getenv('GITEA_OWNER', 'unkin')
url = f"{base_url}/api/v2/remotes/{repo}/files/Packages/{rpm_file}" package_type = os.getenv('GITEA_PACKAGE_TYPE', 'rpm')
# Get Gitea token from Vault - fail hard if unavailable
gitea_token = get_gitea_token()
try: try:
# Normalize version by removing leading zeros (Gitea does this automatically)
# e.g., "2025.08.03" becomes "2025.8.3"
normalized_version = normalize_version(version)
full_version = f"{normalized_version}-{release}"
url = (
f"{base_url}/api/v1/packages/{owner}/"
f"{package_type}/{package_name}/{full_version}"
)
_gitea_session.headers.update({'Authorization': f'token {gitea_token}'})
logger.debug(f"Checking package existence: {url}") logger.debug(f"Checking package existence: {url}")
response = _artifactapi_session.get(url, timeout=10) response = _gitea_session.get(url, timeout=10)
if response.status_code == 200: if response.status_code == 200:
logger.debug(f"Package {rpm_file} exists in {repo}") package_info = response.json()
return True exists = bool(package_info.get('id'))
if response.status_code == 404: logger.debug(f"Package {package_name}:{full_version} {'exists' if exists else 'not found'}")
logger.debug(f"Package {rpm_file} not found in {repo} (404)") return exists
elif response.status_code == 404:
logger.debug(f"Package {package_name}:{full_version} not found (404)")
return False
elif response.status_code == 401:
logger.error("Authentication failed. Check GITEA_API_TOKEN.")
return False
else:
logger.warning(
f"Unexpected response checking package {package_name}:{full_version}: "
f"{response.status_code} - {response.text}"
)
return False return False
logger.warning(
f"Unexpected response checking {rpm_file} in {repo}: "
f"{response.status_code}"
)
return False
except requests.RequestException as e: except requests.RequestException as e:
logger.error(f"Failed to check {rpm_file} in {repo}: {e}") logger.error(f"Failed to check package {package_name}:{version}-{release}: {e}")
return False return False
def get_package_full_name(package_name: str, version: str, release: str) -> str:
"""
Generate the full package name as used in the registry.
Args:
package_name: Package name
version: Version string
release: Release number
Returns:
Full package name string
"""
return f"{package_name}-{version}-{release}"
# ==================== DOCKER FUNCTIONS ==================== # ==================== DOCKER FUNCTIONS ====================
def check_docker_available() -> bool: def check_docker_available() -> bool:
@@ -1264,7 +1251,6 @@ class PackageInfo:
directory: Path directory: Path
distro: str = 'almalinux/el9' distro: str = 'almalinux/el9'
base_image: str = "git.unkin.net/unkin/almalinux9-rpmbuilder:latest" base_image: str = "git.unkin.net/unkin/almalinux9-rpmbuilder:latest"
arch: str = 'amd64'
def __str__(self): def __str__(self):
return f"{self.name}-{self.version}-{self.release} ({self.distro})" return f"{self.name}-{self.version}-{self.release} ({self.distro})"
@@ -1342,7 +1328,7 @@ class Builder:
# Use the first repository as the distro identifier # Use the first repository as the distro identifier
build_distro = repositories[0] if repositories else 'unknown' build_distro = repositories[0] if repositories else 'unknown'
rel = effective_release(str(release), build_distro, use_dist_tag) rel = effective_release(str(release), build_distro, use_dist_tag)
packages.append(PackageInfo(package_name, version, rel, package_dir, build_distro, base_image, arch=metadata.get('arch', 'amd64'))) packages.append(PackageInfo(package_name, version, rel, package_dir, build_distro, base_image))
else: else:
# Build for specific distro # Build for specific distro
for build_config in build_configs: for build_config in build_configs:
@@ -1355,7 +1341,7 @@ class Builder:
# Check if the target distro matches any repository # Check if the target distro matches any repository
if distro in repositories and base_image and version and release: if distro in repositories and base_image and version and release:
rel = effective_release(str(release), distro, use_dist_tag) rel = effective_release(str(release), distro, use_dist_tag)
packages.append(PackageInfo(package_name, version, rel, package_dir, distro, base_image, arch=metadata.get('arch', 'amd64'))) packages.append(PackageInfo(package_name, version, rel, package_dir, distro, base_image))
break break
else: else:
# If no matching distro found, log a warning # If no matching distro found, log a warning
@@ -1454,7 +1440,7 @@ class Builder:
return False return False
rel = effective_release(release, distro, use_dist_tag) rel = effective_release(release, distro, use_dist_tag)
package_info = PackageInfo(package, version, rel, package_dir, distro, base_image, arch=metadata.get('arch', 'amd64')) package_info = PackageInfo(package, version, rel, package_dir, distro, base_image)
return self._build_package(package_info, dry_run, force, native, buildah) return self._build_package(package_info, dry_run, force, native, buildah)
def build_all(self, dry_run: bool = False, force: bool = False, parallel: int = 4, distro: str = 'el/9', native: bool = False, buildah: bool = False) -> bool: def build_all(self, dry_run: bool = False, force: bool = False, parallel: int = 4, distro: str = 'el/9', native: bool = False, buildah: bool = False) -> bool:
@@ -1536,9 +1522,7 @@ class Builder:
if check_package_exists( if check_package_exists(
package_info.name, package_info.name,
package_info.version, package_info.version,
package_info.release, package_info.release
package_info.distro,
package_info.arch,
): ):
self.logger.info( self.logger.info(
f"Skipping {package_info} (already exists in repository)" f"Skipping {package_info} (already exists in repository)"