Virtual helm merge: use yaml.CSafeLoader/CDumper for 10-50x YAML parse/dump speedup #34

New Issue

2026-05-01T23:44:12+10:00

unkinben commented

2026-05-01 23:44:12 +10:00

Performance Issue

The _merge_helm_indexes function in artifact/virtual.py uses yaml.safe_load (pure-Python loader) and yaml.dump (pure-Python dumper). For a 19-member virtual repo merging ~14MB of index data, the merge step took 38.5 seconds in testing.

PyYAML ships C extensions (yaml.CSafeLoader, yaml.CDumper/yaml.CSafeDumper) that are 10–50x faster than their pure-Python equivalents.

Fix

Replace in _merge_helm_indexes:

yaml.safe_load(raw_data) → yaml.load(raw_data, Loader=yaml.CSafeLoader)
yaml.dump(merged, Dumper=_HelmDumper, ...) — extend _HelmDumper from yaml.CDumper instead of yaml.Dumper and use yaml.dump(merged, Dumper=_CHelmDumper, ...)

Add a graceful fallback to pure-Python loaders if the C extension is not available (importable as yaml.CSafeLoader).

Expected Impact

The 38.5s merge step should drop to under 5s for 19 members. The cache-miss latency for large virtual repos becomes acceptable for the first request.

## Performance Issue The `_merge_helm_indexes` function in `artifact/virtual.py` uses `yaml.safe_load` (pure-Python loader) and `yaml.dump` (pure-Python dumper). For a 19-member virtual repo merging ~14MB of index data, the merge step took **38.5 seconds** in testing. PyYAML ships C extensions (`yaml.CSafeLoader`, `yaml.CDumper`/`yaml.CSafeDumper`) that are 10–50x faster than their pure-Python equivalents. ## Fix Replace in `_merge_helm_indexes`: - `yaml.safe_load(raw_data)` → `yaml.load(raw_data, Loader=yaml.CSafeLoader)` - `yaml.dump(merged, Dumper=_HelmDumper, ...)` — extend `_HelmDumper` from `yaml.CDumper` instead of `yaml.Dumper` and use `yaml.dump(merged, Dumper=_CHelmDumper, ...)` Add a graceful fallback to pure-Python loaders if the C extension is not available (importable as `yaml.CSafeLoader`). ## Expected Impact The 38.5s merge step should drop to under 5s for 19 members. The cache-miss latency for large virtual repos becomes acceptable for the first request.

unkinben referenced this issue

2026-05-01 23:44:34 +10:00

Virtual helm merge: offload CPU-bound merge to thread pool to unblock async event loop #35

unkinben referenced this issue

2026-05-01 23:44:44 +10:00

Virtual helm: cache parsed member indexes (not raw bytes) to skip re-parse on rebuild #36

unkinben referenced a pull request that will close this issue

2026-05-02 01:50:39 +10:00

perf: use yaml.CSafeLoader/CDumper for 4x faster virtual index merge #39

unkinben closed this issue

2026-05-02 11:51:01 +10:00

unkinben referenced this issue from a commit