feat: cache parsed member indexes as msgpack to skip YAML re-parse on rebuild
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful

Warm rebuilds of virtual repos (member caches valid, virtual TTL expired)
previously re-parsed all member index.yaml files on every rebuild. With 19
Helm members totalling 14 MB, YAML parsing was 60% of merge time (~6.3s of
~9.6s). Parsing each member's YAML also produces msgpack and stores it in S3
alongside the raw index. Subsequent rebuilds load the compact msgpack and skip
YAML parsing entirely.

Before: warm rebuild ~9.6s (CSafeLoader baseline)
After:  warm rebuild ~5.9s (38% faster, merge=4.7s down from ~9.6s)
This commit is contained in:
2026-05-02 17:04:19 +10:00
parent 15f934cd0b
commit 1e0f4dc840
4 changed files with 284 additions and 41 deletions
+1
View File
@@ -14,6 +14,7 @@ dependencies = [
"lxml>=4.9.0",
"prometheus-client>=0.19.0",
"python-multipart>=0.0.6",
"msgpack>=1.0.0",
]
requires-python = ">=3.11"
readme = "README.md"