unkin/artifactapi
2
0
Fork 0
Code Issues 5 Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

feat: immutable/mutable caching patterns with conditional revalidation and stale fallback #14

Merged
unkinben merged 5 commits from benvin/immutable-mutable-patterns into master 2026-04-27 11:44:49 +10:00
Conversation 0 Commits 5 Files Changed 11
+818 -316
unkinben commented 2026-04-27 11:41:47 +10:00
Owner
Copy Link
Copy Source

Summary

  • Renames include/index pattern terminology to immutable/mutable throughout config, cache, and tests
  • Adds per-remote mutable_ttl configuration and a check_mutable_updates flag for conditional HEAD revalidation (304 → refresh TTL, no re-download)
  • Keeps expired mutable files cached when the upstream backend is unreachable (network error / timeout), refreshing the TTL so RPM repodata, Alpine indexes, and branch archives remain available during outages
  • Commits remotes.yaml as a documented example configuration
  • Extracts duplicated expired-mutable handling into handle_expired_mutable() helper
  • Updates README to current terminology and documents all new caching features

Test plan

  • pytest — all 162 tests pass
  • check_mutable_updates=true + upstream returns 304 → TTL refreshed, no re-download
  • check_mutable_updates=true + upstream returns 200 → cache evicted, re-downloaded
  • check_mutable_updates=true + upstream unreachable → stale copy kept, TTL extended
  • Plain mutable expiry + upstream unreachable → stale copy kept, TTL extended
  • Plain mutable expiry + upstream reachable → cache evicted and re-fetched normally
## Summary - Renames `include`/`index` pattern terminology to `immutable`/`mutable` throughout config, cache, and tests - Adds per-remote `mutable_ttl` configuration and a `check_mutable_updates` flag for conditional `HEAD` revalidation (304 → refresh TTL, no re-download) - Keeps expired mutable files cached when the upstream backend is unreachable (network error / timeout), refreshing the TTL so RPM repodata, Alpine indexes, and branch archives remain available during outages - Commits `remotes.yaml` as a documented example configuration - Extracts duplicated expired-mutable handling into `handle_expired_mutable()` helper - Updates README to current terminology and documents all new caching features ## Test plan - [x] `pytest` — all 162 tests pass - [x] `check_mutable_updates=true` + upstream returns 304 → TTL refreshed, no re-download - [x] `check_mutable_updates=true` + upstream returns 200 → cache evicted, re-downloaded - [x] `check_mutable_updates=true` + upstream unreachable → stale copy kept, TTL extended - [x] Plain mutable expiry + upstream unreachable → stale copy kept, TTL extended - [x] Plain mutable expiry + upstream reachable → cache evicted and re-fetched normally
unkinben added 5 commits 2026-04-27 11:41:47 +10:00
feat: rename include/index patterns to immutable/mutable with per-remote TTL ce01a94141 
Replace the include_patterns/index_patterns split with a clearer
immutable_patterns/mutable_patterns model:

- immutable_patterns: artifacts cached indefinitely (no TTL)
- mutable_patterns: artifacts that expire and are re-fetched after
  cache.mutable_ttl seconds (replaces cache.index_ttl)

_PACKAGE_INDEX_PATTERNS renamed to _PACKAGE_MUTABLE_PATTERNS; all
built-in package-type index patterns (APKINDEX, repomd, manifests, etc.)
default to the remote's mutable_ttl (default 1 hour).

cache.file_ttl renamed to cache.immutable_ttl for consistency.
Adds github-archive remote to remotes.yaml as a worked example showing
tag archives as immutable and branch archives as mutable (1-day TTL).

docker-compose.yml: fix VERSION=dev → 2.2.2.dev0 (valid PEP 440),
add :z SELinux label to volume mounts.
chore: track remotes.yaml as a documented example config 8bc9285117 
Remove remotes.yaml from .gitignore and add header comments explaining
the immutable_patterns/mutable_patterns/cache keys. Marks the file
clearly as an example to copy and adapt; warns against committing
real credentials.
feat: add check_mutable_updates flag for conditional upstream revalidation 8fe4bac2b9 
When check_mutable_updates: true is set on a remote, expired user-defined
mutable files are revalidated before re-downloading:

- On expiry a conditional HEAD is sent with If-None-Match / If-Modified-Since
- 304 Not Modified: TTL is refreshed in Redis, S3 cache is untouched
- 200 / no conditional support: cache is invalidated and file re-downloaded
- Network error: safe fallback — assume changed, re-download

ETag and Last-Modified from upstream responses are stored in Redis under
mutable:meta:<remote>:<hash> (no expiry, cleaned up on re-download or
cache flush). The flag only applies to user-configured mutable_patterns;
built-in package-type defaults (APKINDEX, repomd.xml, Docker manifests)
are always re-fetched unconditionally.

cache/flush also clears mutable:meta:* keys alongside index:* keys.
refactor: extract handle_expired_mutable helper; add redownload success test 78296dae8f 
Deduplicates the expired-mutable TTL/redownload branching logic that
was copied verbatim between get_artifact and docker_v2_proxy. Adds the
missing happy-path test for a changed mutable file that is successfully
re-fetched from upstream.
feat: keep stale mutables when upstream is unreachable; update README
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
ci/woodpecker/pr/build Pipeline was successful
Details
fe837dabf7 
When a mutable file's TTL expires and the upstream backend cannot be
contacted (network error or timeout), the cached copy is kept and its
TTL refreshed instead of being evicted. This keeps RPM repodata, Alpine
indexes, branch archives, and other mutable data available during
upstream outages.

Adds UpstreamUnreachable exception and _upstream_reachable() helper.
check_upstream_changed() now raises UpstreamUnreachable on network
errors (was silently returning True). handle_expired_mutable() catches
the exception on the check_mutable_updates path and calls
_upstream_reachable() on the plain-expiry path.

README updated to current immutable/mutable terminology and documents
all new caching features.
unkinben merged commit 70cd439961 into master 2026-04-27 11:44:49 +10:00
unkinben deleted branch benvin/immutable-mutable-patterns 2026-04-27 11:44:49 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/artifactapi#14
Delete Branch "benvin/immutable-mutable-patterns"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?