unkinben
8a7f26b193
ci/woodpecker/tag/docker Pipeline was successful
Closes #36 ## Summary - After fetching a member's `index.yaml` (from upstream or S3), the handler now parses it and stores a compact msgpack file (`index.msgpack`) alongside the raw YAML in S3 - On subsequent virtual rebuilds (member caches valid, virtual TTL expired), the handler loads the msgpack file instead of re-parsing raw YAML — eliminating the costliest phase - `_entries_to_msgpack_safe()` converts datetime/date objects to ISO strings before packing (msgpack cannot natively serialize Python datetimes) - `_merge_helm_indexes()` accepts `list[dict | None]` as pre-parsed entries; falls back to raw YAML parse when msgpack is unavailable - `_VirtualHandler.merge()` protocol updated to pass pre-parsed entries to all future handler implementations - Broken msgpack is detected and rebuilt from raw YAML automatically ## Performance Phase breakdown (19-member helm-all virtual, 14 MB total): | Phase | Time | % | |---|---|---| | YAML parse (eliminated) | 6314 ms | 60% | | URL rewrite + dedup | 33 ms | 0.3% | | YAML dump | 4124 ms | 39% | | Scenario | Before (CSafeLoader only, #34) | After | |---|---|---| | Cold rebuild (upstream fetch) | ~21s | ~26s (+5s for msgpack build, one-time) | | **Warm rebuild (S3 hit, virtual expired)** | **~9.6s** | **~5.9s (38% faster)** | | Virtual cache hit | ~0.03s | ~0.03s | Log line confirms msgpack hits: `msgpack=19/19` ## Test plan - 297 tests pass - `TestEntriesToMsgpackSafe`: datetime/date serialization, empty input, round-trip - `TestMergeHelmIndexesWithParsed`: pre-parsed path produces identical output to raw-bytes path - `TestGetMemberIndexMsgpack`: msgpack hit, cold-build, broken msgpack fallback, upstream failure - Docker warm-rebuild measured at 5.9s vs 9.6s baseline Reviewed-on: #40
61 lines
1.2 KiB
TOML
61 lines
1.2 KiB
TOML
[project]
|
|
name = "artifactapi"
|
|
dynamic = ["version"]
|
|
description = "Generic artifact caching system with support for various package managers"
|
|
|
|
dependencies = [
|
|
"fastapi>=0.104.0",
|
|
"uvicorn[standard]>=0.24.0",
|
|
"httpx>=0.25.0",
|
|
"redis>=5.0.0",
|
|
"boto3>=1.29.0",
|
|
"psycopg2-binary>=2.9.0",
|
|
"pyyaml>=6.0",
|
|
"lxml>=4.9.0",
|
|
"prometheus-client>=0.19.0",
|
|
"python-multipart>=0.0.6",
|
|
"msgpack>=1.0.0",
|
|
]
|
|
requires-python = ">=3.11"
|
|
readme = "README.md"
|
|
license = {text = "MIT"}
|
|
|
|
[project.scripts]
|
|
artifactapi = "artifactapi.main:main"
|
|
|
|
[build-system]
|
|
requires = ["hatchling", "hatch-vcs"]
|
|
build-backend = "hatchling.build"
|
|
|
|
[tool.hatch.version]
|
|
source = "vcs"
|
|
|
|
[tool.hatch.metadata]
|
|
allow-direct-references = true
|
|
|
|
[tool.hatch.build.targets.wheel]
|
|
packages = ["src/artifactapi"]
|
|
|
|
[project.optional-dependencies]
|
|
dev = [
|
|
"pytest>=7.4.0",
|
|
"pytest-asyncio>=0.21.0",
|
|
"black>=23.9.0",
|
|
"isort>=5.12.0",
|
|
"mypy>=1.6.0",
|
|
"ruff>=0.4.0",
|
|
"tox>=4.0.0",
|
|
"pre-commit>=3.0.0",
|
|
]
|
|
|
|
[tool.pytest.ini_options]
|
|
asyncio_mode = "auto"
|
|
testpaths = ["tests"]
|
|
|
|
[tool.ruff]
|
|
line-length = 140
|
|
|
|
[tool.ruff.lint]
|
|
select = ["E", "F", "I", "UP"]
|
|
ignore = ["E501"]
|