unkin/artifactapi
3
0
Fork 0
Code Issues 5 Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

rpm-local repodata: intermittent checksum mismatch (repomd vs primary) breaks dnf #117

New Issue
Closed
opened 2026-08-12 22:07:46 +10:00 by unkin-agent · 1 comment
unkin-agent commented 2026-08-12 22:07:46 +10:00
Member
Copy Link
Copy Source

Problem

dnf install from an rpm-local repo (rpm-internal) intermittently fails with a repodata checksum mismatch — repomd.xml advertises a primary.xml.gz sha256, but the bytes served for that content-addressed file hash to something else. Observed in the wild:

Downloading successful, but checksum doesn't match.
  Calculated: e8ca7eefc3f31f3cd7f430eb9110b549f91349d252c5dbafaefc7dc327ff9c03
  Expected:   18da7927b57d18dc797583906b0a2d82cc396731e58c77232df79684c6448787
  - https://artifactapi.k8s.syd1.au.unkin.net/api/v1/local/rpm-internal/repodata/18da7927...-primary.xml.gz
No more mirrors to try - All mirrors were already tried without success

The repodata is regenerated per request (not stored), and the API runs 2 replicas with no session affinity. So the repomd.xml fetch and the subsequent primary.xml.gz fetch can see different bytes when either:

  1. the repo state changes between the two fetches (a concurrent upload — this occurred during a release publish), or
  2. generation is not byte-identical across replicas (Go map iteration order, a metadata query without a stable ORDER BY, or gzip mtime/OS bytes).

Retried immediately, the hashes matched — so it is intermittent, not persistent. It affects every rpm-local consumer via dnf. The deb/apk local providers likely share the class (Release → Packages/Packages.gz checksums) under the same 2-replica deployment.

Workaround: install the RPM directly by URL (dnf install <.../Packages/<file>.rpm>), which bypasses repodata.

Proposal

Make the served repodata set internally consistent for any fetch sequence:

  • Generate the whole set (repomd + primary/filelists/other) from ONE DB snapshot with deterministic ordering (explicit ORDER BY on the metadata query) and zero-mtime gzip, so it is byte-identical across replicas and requests.
  • And/or pin/cache the generated set keyed by a content revision, so repomd.xml and its referenced hashed files always agree; evict on upload/delete.
  • Apply the same guarantee to the deb/apk local Release ↔ Packages/APKINDEX generation.

Acceptance

  • Repeated dnf makecache/install against rpm-internal across many requests — including concurrent with an upload — never produce a checksum mismatch.
  • For any fetch, each repomd.xml primary/filelists/other sha256 equals the sha256 of the bytes actually served at those paths, verified against both replicas.
  • Same verified for apt against a deb-local repo (Release ↔ Packages) and for apk (APKINDEX).
## Problem `dnf install` from an rpm-local repo (`rpm-internal`) intermittently fails with a repodata checksum mismatch — `repomd.xml` advertises a `primary.xml.gz` sha256, but the bytes served for that content-addressed file hash to something else. Observed in the wild: ``` Downloading successful, but checksum doesn't match. Calculated: e8ca7eefc3f31f3cd7f430eb9110b549f91349d252c5dbafaefc7dc327ff9c03 Expected: 18da7927b57d18dc797583906b0a2d82cc396731e58c77232df79684c6448787 - https://artifactapi.k8s.syd1.au.unkin.net/api/v1/local/rpm-internal/repodata/18da7927...-primary.xml.gz No more mirrors to try - All mirrors were already tried without success ``` The repodata is **regenerated per request** (not stored), and the API runs **2 replicas with no session affinity**. So the `repomd.xml` fetch and the subsequent `primary.xml.gz` fetch can see different bytes when either: 1. the repo state changes between the two fetches (a concurrent upload — this occurred during a release publish), or 2. generation is not byte-identical across replicas (Go map iteration order, a metadata query without a stable `ORDER BY`, or gzip mtime/OS bytes). Retried immediately, the hashes matched — so it is **intermittent, not persistent**. It affects every rpm-local consumer via `dnf`. The deb/apk local providers likely share the class (`Release` → `Packages`/`Packages.gz` checksums) under the same 2-replica deployment. Workaround: install the RPM directly by URL (`dnf install <.../Packages/<file>.rpm>`), which bypasses repodata. ## Proposal Make the served repodata set internally consistent for any fetch sequence: - Generate the whole set (`repomd` + `primary`/`filelists`/`other`) from ONE DB snapshot with deterministic ordering (explicit `ORDER BY` on the metadata query) and zero-mtime gzip, so it is byte-identical across replicas and requests. - And/or pin/cache the generated set keyed by a content revision, so `repomd.xml` and its referenced hashed files always agree; evict on upload/delete. - Apply the same guarantee to the deb/apk local `Release` ↔ `Packages`/`APKINDEX` generation. ## Acceptance - Repeated `dnf makecache`/`install` against `rpm-internal` across many requests — including concurrent with an upload — never produce a checksum mismatch. - For any fetch, each `repomd.xml` primary/filelists/other sha256 equals the sha256 of the bytes actually served at those paths, verified against both replicas. - Same verified for `apt` against a deb-local repo (Release ↔ Packages) and for `apk` (APKINDEX).
unkin-agent commented 2026-08-12 23:33:23 +10:00
Author
Member
Copy Link
Copy Source

Resolved across three PRs — local generated repodata is now byte-identical across replicas and regenerations, so the advertised checksums always match the served content-addressed bytes:

  • #118 (rpm): derive <time>/repomd revision from persisted created_at instead of time.Now(); total-order ORDER BY; pin gzip header.
  • #119 (deb/apk): derive deb Release Date: from newest created_at; pin apk APKINDEX.tar.gz tar ModTime; total-order ORDER BY on both.

Root cause was a wall-clock timestamp (time.Now()) embedded inside the content-addressed gzipped primary.xml; the deb/apk paths got the same determinism guarantee plus regression tests asserting advertised-checksum == served-bytes.

Shipped in v3.10.1. (Remote repos were never affected — they proxy upstream's content-addressed repodata verbatim.)

Resolved across three PRs — local generated repodata is now byte-identical across replicas and regenerations, so the advertised checksums always match the served content-addressed bytes: - #118 (rpm): derive `<time>`/`repomd` revision from persisted `created_at` instead of `time.Now()`; total-order `ORDER BY`; pin gzip header. - #119 (deb/apk): derive deb `Release` `Date:` from newest `created_at`; pin apk `APKINDEX.tar.gz` tar ModTime; total-order `ORDER BY` on both. Root cause was a wall-clock timestamp (`time.Now()`) embedded inside the content-addressed gzipped `primary.xml`; the deb/apk paths got the same determinism guarantee plus regression tests asserting advertised-checksum == served-bytes. Shipped in **v3.10.1**. (Remote repos were never affected — they proxy upstream's content-addressed repodata verbatim.)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/artifactapi#117
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?