perf: use yaml.CSafeLoader/CDumper for 4x faster virtual index merge #39

Merged

unkinben merged 1 commits from benvin/issue-34-csafeloader-cdumper into master 2026-05-02 11:51:01 +10:00

Conversation 0 Commits 1 Files Changed 2

+39 -2

unkinben commented 2026-05-02 01:50:39 +10:00

Owner

Copy Link

Copy Source

Closes #34

Summary

• At module load time, a try/except selects yaml.CSafeLoader / yaml.CDumper (C extensions) when libyaml is available, otherwise falls back to yaml.SafeLoader / yaml.Dumper
• _HelmDumper inherits from whichever dumper base was selected — custom datetime/date representers are registered the same way as before
• _merge_helm_indexes uses yaml.load(raw_data, Loader=_YamlLoader) instead of yaml.safe_load
• No change to yaml.dump(...) call — it already passes Dumper=_HelmDumper, which now inherits from the C base when available
• Five new tests in TestYamlExtensionSelection cover: loader/dumper base are classes, _HelmDumper inherits from the selected base, C extensions used when available, loader can parse YAML

Measured performance gain

19-member helm-all virtual repo, real upstream data, Docker (AlmaLinux 9):

	merge= time
Before (SafeLoader + Dumper)	38,877ms
After (CSafeLoader + CDumper)	9,625ms
Speedup	4.0×

Local microbenchmark (500 charts × 10 versions × 19 members, 3 runs avg):

• Before: 40.8s → After: 6.1s (6.7× faster)

Test plan

• 283 unit tests pass (make test)
• Wheel builds cleanly (uv build --wheel)
• C extension confirmed available in AlmaLinux 9 container: yaml.CSafeLoader: <class 'yaml.cyaml.CSafeLoader'>
• Baseline Docker timing measured with pure-Python path forced: merge=38,877ms
• After Docker timing measured with C extension path: merge=9,625ms

Closes #34 ## Summary - At module load time, a `try/except` selects `yaml.CSafeLoader` / `yaml.CDumper` (C extensions) when libyaml is available, otherwise falls back to `yaml.SafeLoader` / `yaml.Dumper` - `_HelmDumper` inherits from whichever dumper base was selected — custom datetime/date representers are registered the same way as before - `_merge_helm_indexes` uses `yaml.load(raw_data, Loader=_YamlLoader)` instead of `yaml.safe_load` - No change to `yaml.dump(...)` call — it already passes `Dumper=_HelmDumper`, which now inherits from the C base when available - Five new tests in `TestYamlExtensionSelection` cover: loader/dumper base are classes, `_HelmDumper` inherits from the selected base, C extensions used when available, loader can parse YAML ## Measured performance gain 19-member `helm-all` virtual repo, real upstream data, Docker (AlmaLinux 9): | | `merge=` time | |---|---| | Before (SafeLoader + Dumper) | **38,877ms** | | After (CSafeLoader + CDumper) | **9,625ms** | | Speedup | **4.0×** | Local microbenchmark (500 charts × 10 versions × 19 members, 3 runs avg): - Before: **40.8s** → After: **6.1s** (**6.7×** faster) ## Test plan - [x] 283 unit tests pass (`make test`) - [x] Wheel builds cleanly (`uv build --wheel`) - [x] C extension confirmed available in AlmaLinux 9 container: `yaml.CSafeLoader: <class 'yaml.cyaml.CSafeLoader'>` - [x] Baseline Docker timing measured with pure-Python path forced: merge=38,877ms - [x] After Docker timing measured with C extension path: merge=9,625ms

unkinben added 1 commit 2026-05-02 01:50:39 +10:00

perf: use yaml.CSafeLoader/CDumper for 4x faster virtual index merge ... b7488fce10

Replace yaml.safe_load / yaml.Dumper with the C extension equivalents
(yaml.CSafeLoader, yaml.CDumper) for all YAML parsing and serialisation
in the virtual repo merge path. A module-level try/except falls back to
the pure-Python SafeLoader/Dumper when libyaml is not available.

Measured on 19-member helm-all (real upstream data):
  Before (SafeLoader + Dumper): merge=38,877ms
  After  (CSafeLoader + CDumper): merge=9,625ms  (4.0x faster)

Local microbenchmark (500 charts × 10 versions × 19 members):
  Before: 40.8s  After: 6.1s  (6.7x faster)

unkinben merged commit 15f934cd0b into master 2026-05-02 11:51:01 +10:00

unkinben deleted branch benvin/issue-34-csafeloader-cdumper 2026-05-02 11:51:01 +10:00

unkinben referenced this issue from a commit 2026-05-02 11:51:01 +10:00

perf: use yaml.CSafeLoader/CDumper for 4x faster virtual index merge (#39)

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

benvin (BenVincent) unkinben (Ben Vincent)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/artifactapi#39