Compare commits
10 Commits
373366e695
...
v2.7.1
| Author | SHA1 | Date | |
|---|---|---|---|
| 1656664dfa | |||
| c7baae8d0d | |||
| 4789635e87 | |||
| ba52fedd27 | |||
| 76633403b2 | |||
| cae3503ac4 | |||
| 3f098df428 | |||
| 64266f40e9 | |||
| be25fc19f7 | |||
| 3bd3ca8b74 |
@@ -5,6 +5,7 @@ FastAPI caching proxy that downloads and stores files from remote sources in S3-
|
|||||||
## Features
|
## Features
|
||||||
|
|
||||||
- Remote definitions via `remotes.yaml` — generic HTTP, Alpine APK, RPM, Docker, PyPI, npm, Helm
|
- Remote definitions via `remotes.yaml` — generic HTTP, Alpine APK, RPM, Docker, PyPI, npm, Helm
|
||||||
|
- Virtual repositories — merge multiple remotes of the same package type into a single unified index
|
||||||
- Immutable/mutable caching model with per-remote TTLs
|
- Immutable/mutable caching model with per-remote TTLs
|
||||||
- Conditional revalidation (`If-None-Match` / `If-Modified-Since`) on TTL expiry
|
- Conditional revalidation (`If-None-Match` / `If-Modified-Since`) on TTL expiry
|
||||||
- Stale-on-upstream-error: refreshes TTL when backend is unreachable rather than evicting
|
- Stale-on-upstream-error: refreshes TTL when backend is unreachable rather than evicting
|
||||||
@@ -37,6 +38,7 @@ src/artifactapi/
|
|||||||
├── docker_auth.py — backwards-compat shim → auth/docker.py
|
├── docker_auth.py — backwards-compat shim → auth/docker.py
|
||||||
├── artifact/ — route handler implementations
|
├── artifact/ — route handler implementations
|
||||||
│ ├── proxy.py — GET /api/v1/remote (remote proxy, cache, revalidation)
|
│ ├── proxy.py — GET /api/v1/remote (remote proxy, cache, revalidation)
|
||||||
|
│ ├── virtual.py — GET /api/v1/virtual (virtual repo index merging)
|
||||||
│ ├── local.py — PUT/HEAD/DELETE /api/v1/remote (local repos)
|
│ ├── local.py — PUT/HEAD/DELETE /api/v1/remote (local repos)
|
||||||
│ ├── docker.py — /v2/ Docker Registry v2 proxy
|
│ ├── docker.py — /v2/ Docker Registry v2 proxy
|
||||||
│ ├── discovery.py — /api/v1/artifacts discovery + bulk cache
|
│ ├── discovery.py — /api/v1/artifacts discovery + bulk cache
|
||||||
@@ -68,9 +70,11 @@ src/artifactapi/
|
|||||||
| Method | Path | Description |
|
| Method | Path | Description |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `GET` | `/api/v1/remote/{remote}/{path}` | Fetch artifact (auto-cache on miss) |
|
| `GET` | `/api/v1/remote/{remote}/{path}` | Fetch artifact (auto-cache on miss) |
|
||||||
| `PUT` | `/api/v1/remote/{remote}/{path}` | Upload to local remote |
|
| `GET` | `/api/v1/virtual/{virtual}/{path}` | Fetch from virtual (merged) repository |
|
||||||
| `HEAD` | `/api/v1/remote/{remote}/{path}` | Check existence (local remotes) |
|
| `GET` | `/api/v1/local/{local}/{path}` | Download from local repository |
|
||||||
| `DELETE` | `/api/v1/remote/{remote}/{path}` | Delete from local remote |
|
| `PUT` | `/api/v1/local/{local}/{path}` | Upload to local repository |
|
||||||
|
| `HEAD` | `/api/v1/local/{local}/{path}` | Check existence (local) |
|
||||||
|
| `DELETE` | `/api/v1/local/{local}/{path}` | Delete from local repository |
|
||||||
| `GET` | `/v2/{remote}/{path}` | Docker Registry v2 proxy |
|
| `GET` | `/v2/{remote}/{path}` | Docker Registry v2 proxy |
|
||||||
| `PUT` | `/cache/flush` | Flush cache entries |
|
| `PUT` | `/cache/flush` | Flush cache entries |
|
||||||
| `GET` | `/health` | Health check |
|
| `GET` | `/health` | Health check |
|
||||||
@@ -79,12 +83,13 @@ src/artifactapi/
|
|||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
Runtime settings come from environment variables; remote definitions live in `remotes.yaml`.
|
Runtime settings come from environment variables; remote definitions live in one or more YAML files pointed to by `CONFIG_PATH`.
|
||||||
|
|
||||||
### Environment Variables
|
### Environment Variables
|
||||||
|
|
||||||
| Variable | Description |
|
| Variable | Description |
|
||||||
|---|---|
|
|---|---|
|
||||||
|
| `CONFIG_PATH` | Path to a config YAML file **or** a directory of YAML files |
|
||||||
| `DBHOST`, `DBPORT`, `DBUSER`, `DBPASS`, `DBNAME` | PostgreSQL connection |
|
| `DBHOST`, `DBPORT`, `DBUSER`, `DBPASS`, `DBNAME` | PostgreSQL connection |
|
||||||
| `REDIS_URL` | Redis URL (e.g. `redis://localhost:6379`) |
|
| `REDIS_URL` | Redis URL (e.g. `redis://localhost:6379`) |
|
||||||
| `MINIO_ENDPOINT` | MinIO/S3 endpoint |
|
| `MINIO_ENDPOINT` | MinIO/S3 endpoint |
|
||||||
@@ -93,13 +98,37 @@ Runtime settings come from environment variables; remote definitions live in `re
|
|||||||
| `MINIO_BUCKET` | S3 bucket name |
|
| `MINIO_BUCKET` | S3 bucket name |
|
||||||
| `MINIO_SECURE` | Use HTTPS (`true`/`false`) |
|
| `MINIO_SECURE` | Use HTTPS (`true`/`false`) |
|
||||||
|
|
||||||
### remotes.yaml Structure
|
### Split configuration
|
||||||
|
|
||||||
|
`CONFIG_PATH` accepts three forms:
|
||||||
|
|
||||||
|
**Single file** (original behaviour):
|
||||||
|
```
|
||||||
|
CONFIG_PATH=/etc/artifactapi/remotes.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
**Directory** — all `*.yaml` / `*.yml` files in the directory are loaded and merged alphabetically. `remotes` keys are merged across files; later files win on conflict:
|
||||||
|
```
|
||||||
|
CONFIG_PATH=/etc/artifactapi/conf.d/
|
||||||
|
```
|
||||||
|
|
||||||
|
**Main file + `config_dir`** — the main file holds global settings and a `config_dir` pointer; each file in that directory contributes its own `remotes`. Relative `config_dir` paths are resolved relative to the main file:
|
||||||
|
```yaml
|
||||||
|
# /etc/artifactapi/config.yaml
|
||||||
|
config_dir: conf.d # or an absolute path
|
||||||
|
|
||||||
|
# s3/redis/database settings go here (or in env vars)
|
||||||
|
remotes: {} # optional base remotes
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration structure
|
||||||
|
|
||||||
|
Repositories are declared under three top-level keys matching their type:
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
remotes:
|
remotes: # proxy (caching) remotes
|
||||||
remote-name:
|
remote-name:
|
||||||
base_url: "https://example.com"
|
base_url: "https://example.com"
|
||||||
type: "remote" # "remote" or "local"
|
|
||||||
package: "generic" # generic, alpine, rpm, docker, pypi, npm, helm
|
package: "generic" # generic, alpine, rpm, docker, pypi, npm, helm
|
||||||
description: "..."
|
description: "..."
|
||||||
immutable_patterns: # regex — cached forever
|
immutable_patterns: # regex — cached forever
|
||||||
@@ -110,6 +139,20 @@ remotes:
|
|||||||
cache:
|
cache:
|
||||||
immutable_ttl: 0 # 0 = indefinitely
|
immutable_ttl: 0 # 0 = indefinitely
|
||||||
mutable_ttl: 3600
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
virtuals: # virtual (merged-index) repositories
|
||||||
|
virtual-name:
|
||||||
|
package: "helm"
|
||||||
|
members:
|
||||||
|
- remote-a
|
||||||
|
- remote-b
|
||||||
|
|
||||||
|
locals: # local upload repositories (no base_url)
|
||||||
|
local-name:
|
||||||
|
package: "generic"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 0
|
||||||
```
|
```
|
||||||
|
|
||||||
## Remote Types
|
## Remote Types
|
||||||
@@ -122,7 +165,6 @@ Arbitrary HTTP file servers — GitHub releases, HashiCorp, custom servers.
|
|||||||
remotes:
|
remotes:
|
||||||
github:
|
github:
|
||||||
base_url: "https://github.com"
|
base_url: "https://github.com"
|
||||||
type: "remote"
|
|
||||||
package: "generic"
|
package: "generic"
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
- "gruntwork-io/terragrunt/.*terragrunt_linux_amd64.*"
|
- "gruntwork-io/terragrunt/.*terragrunt_linux_amd64.*"
|
||||||
@@ -131,7 +173,6 @@ remotes:
|
|||||||
|
|
||||||
github-archive:
|
github-archive:
|
||||||
base_url: "https://github.com"
|
base_url: "https://github.com"
|
||||||
type: "remote"
|
|
||||||
package: "generic"
|
package: "generic"
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
- ".*/archive/refs/tags/.*\\.tar\\.gz$" # tag archives never change
|
- ".*/archive/refs/tags/.*\\.tar\\.gz$" # tag archives never change
|
||||||
@@ -151,7 +192,6 @@ Access: `GET /api/v1/remote/github/owner/repo/releases/download/v1.0/binary.tar.
|
|||||||
remotes:
|
remotes:
|
||||||
alpine:
|
alpine:
|
||||||
base_url: "https://dl-cdn.alpinelinux.org"
|
base_url: "https://dl-cdn.alpinelinux.org"
|
||||||
type: "remote"
|
|
||||||
package: "alpine"
|
package: "alpine"
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
- ".*/x86_64/.*\\.apk$"
|
- ".*/x86_64/.*\\.apk$"
|
||||||
@@ -168,7 +208,6 @@ remotes:
|
|||||||
remotes:
|
remotes:
|
||||||
almalinux:
|
almalinux:
|
||||||
base_url: "https://mirror.example.com/almalinux"
|
base_url: "https://mirror.example.com/almalinux"
|
||||||
type: "remote"
|
|
||||||
package: "rpm"
|
package: "rpm"
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
- ".*/x86_64/.*\\.rpm$"
|
- ".*/x86_64/.*\\.rpm$"
|
||||||
@@ -186,7 +225,6 @@ remotes:
|
|||||||
remotes:
|
remotes:
|
||||||
dockerhub:
|
dockerhub:
|
||||||
base_url: "https://registry-1.docker.io"
|
base_url: "https://registry-1.docker.io"
|
||||||
type: "remote"
|
|
||||||
package: "docker"
|
package: "docker"
|
||||||
# username / password optional for public images
|
# username / password optional for public images
|
||||||
cache:
|
cache:
|
||||||
@@ -195,7 +233,6 @@ remotes:
|
|||||||
|
|
||||||
ghcr:
|
ghcr:
|
||||||
base_url: "https://ghcr.io"
|
base_url: "https://ghcr.io"
|
||||||
type: "remote"
|
|
||||||
package: "docker"
|
package: "docker"
|
||||||
username: "your-github-username"
|
username: "your-github-username"
|
||||||
password: "ghp_your_pat" # read:packages scope
|
password: "ghp_your_pat" # read:packages scope
|
||||||
@@ -228,7 +265,6 @@ mirrors:
|
|||||||
remotes:
|
remotes:
|
||||||
pypi:
|
pypi:
|
||||||
base_url: "https://files.pythonhosted.org"
|
base_url: "https://files.pythonhosted.org"
|
||||||
type: "remote"
|
|
||||||
package: "pypi"
|
package: "pypi"
|
||||||
check_mutable_updates: true
|
check_mutable_updates: true
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
@@ -260,7 +296,6 @@ default = true
|
|||||||
remotes:
|
remotes:
|
||||||
npm:
|
npm:
|
||||||
base_url: "https://registry.npmjs.org"
|
base_url: "https://registry.npmjs.org"
|
||||||
type: "remote"
|
|
||||||
package: "npm"
|
package: "npm"
|
||||||
check_mutable_updates: true
|
check_mutable_updates: true
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
@@ -287,7 +322,6 @@ registry=https://artifacts.example.com/api/v1/remote/npm/
|
|||||||
remotes:
|
remotes:
|
||||||
hashicorp-helm:
|
hashicorp-helm:
|
||||||
base_url: "https://helm.releases.hashicorp.com"
|
base_url: "https://helm.releases.hashicorp.com"
|
||||||
type: "remote"
|
|
||||||
package: "helm"
|
package: "helm"
|
||||||
check_mutable_updates: true
|
check_mutable_updates: true
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
@@ -306,12 +340,72 @@ helm repo add hashicorp https://artifacts.example.com/api/v1/remote/hashicorp-he
|
|||||||
helm repo update
|
helm repo update
|
||||||
```
|
```
|
||||||
|
|
||||||
### local
|
### virtual
|
||||||
|
|
||||||
|
A virtual repository presents a single unified index built from multiple member remotes of the same package type. Clients configure one endpoint and get access to all member remotes transparently.
|
||||||
|
|
||||||
|
All members must share the same `package` type as the virtual repo. Currently supported package types: `helm`.
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
remotes:
|
remotes:
|
||||||
|
helm-hashicorp:
|
||||||
|
base_url: "https://helm.releases.hashicorp.com"
|
||||||
|
package: "helm"
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
helm-bitnami:
|
||||||
|
base_url: "https://charts.bitnami.com/bitnami"
|
||||||
|
package: "helm"
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
virtuals:
|
||||||
|
helm-all:
|
||||||
|
package: "helm"
|
||||||
|
members:
|
||||||
|
- helm-hashicorp # listed first = highest priority
|
||||||
|
- helm-bitnami
|
||||||
|
```
|
||||||
|
|
||||||
|
**How it works:**
|
||||||
|
|
||||||
|
1. A request for the package index triggers a parallel fetch of each member's index from S3 cache, falling back to upstream if not yet cached.
|
||||||
|
2. Member indexes are merged into a single index with URL rewriting so artifact download URLs continue to resolve through the individual member remote.
|
||||||
|
3. The merged index is cached in Redis with a TTL equal to the minimum `mutable_ttl` across all members.
|
||||||
|
|
||||||
|
**Priority / conflict resolution:**
|
||||||
|
|
||||||
|
When the same artifact name and version appears in more than one member, the member listed **first** in `members` wins. Subsequent members contribute only artifacts not already present.
|
||||||
|
|
||||||
|
**Partial failures:**
|
||||||
|
|
||||||
|
If a member is unreachable and has no cached index, it is skipped and a warning is logged. The merged index is still served from available members. If *no* members can be reached, the request returns `502`.
|
||||||
|
|
||||||
|
**Caching:**
|
||||||
|
|
||||||
|
The merged index is cached using `min(mutable_ttl)` across all members. Each member's raw index is cached in S3 under its own remote key by the normal proxy rules; the virtual handler reuses those copies when available.
|
||||||
|
|
||||||
|
**Helm example:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
helm repo add all https://artifacts.example.com/api/v1/virtual/helm-all
|
||||||
|
helm repo update
|
||||||
|
```
|
||||||
|
|
||||||
|
Chart tarball URLs in the merged `index.yaml` are rewritten to point at the individual member remote (e.g. `…/api/v1/remote/helm-hashicorp/vault-0.27.0.tgz`), so downloads bypass the virtual endpoint entirely.
|
||||||
|
|
||||||
|
### local
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
locals:
|
||||||
local-generic:
|
local-generic:
|
||||||
type: "local"
|
|
||||||
package: "generic"
|
package: "generic"
|
||||||
description: "Local file repository"
|
description: "Local file repository"
|
||||||
cache:
|
cache:
|
||||||
@@ -319,7 +413,7 @@ remotes:
|
|||||||
mutable_ttl: 0
|
mutable_ttl: 0
|
||||||
```
|
```
|
||||||
|
|
||||||
No `base_url`. Files are uploaded via `PUT` and served via `GET`.
|
No `base_url`. Files are uploaded via `PUT /api/v1/local/{name}/{path}` and downloaded via `GET /api/v1/local/{name}/{path}`.
|
||||||
|
|
||||||
## Caching Model
|
## Caching Model
|
||||||
|
|
||||||
@@ -352,3 +446,24 @@ Set `check_mutable_updates: true` to send `HEAD` with `If-None-Match` / `If-Modi
|
|||||||
### Stale-on-upstream-error
|
### Stale-on-upstream-error
|
||||||
|
|
||||||
When a mutable file expires and the upstream is unreachable (connection refused, DNS failure, timeout), the cached copy is kept and its TTL refreshed. HTTP error responses (4xx, 5xx) are not treated as network failures and proceed with normal expiry.
|
When a mutable file expires and the upstream is unreachable (connection refused, DNS failure, timeout), the cached copy is kept and its TTL refreshed. HTTP error responses (4xx, 5xx) are not treated as network failures and proceed with normal expiry.
|
||||||
|
|
||||||
|
### Quarantine (supply-chain protection)
|
||||||
|
|
||||||
|
Set `quarantine_new: true` and `quarantine_days: N` on a remote to block immutable artifacts published within the last N days. Requests return `404` until the quarantine period expires, giving time to detect malicious packages before they are consumed.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
remotes:
|
||||||
|
pypi:
|
||||||
|
base_url: "https://files.pythonhosted.org"
|
||||||
|
package: "pypi"
|
||||||
|
quarantine_new: true
|
||||||
|
quarantine_days: 3 # block packages published in the last 3 days
|
||||||
|
immutable_patterns:
|
||||||
|
- "packages/.*\\.whl$"
|
||||||
|
- "packages/.*\\.tar\\.gz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 600
|
||||||
|
```
|
||||||
|
|
||||||
|
The upstream `Last-Modified` response header is used as the publish date proxy. Artifacts that have no `Last-Modified` header are allowed through (fail-open). Mutable files (index pages, tag manifests) are never quarantined.
|
||||||
|
|||||||
+1
-1
@@ -10,7 +10,7 @@ services:
|
|||||||
ports:
|
ports:
|
||||||
- "8000:8000"
|
- "8000:8000"
|
||||||
volumes:
|
volumes:
|
||||||
- ./remotes.yaml:/app/remotes.yaml:ro,z
|
- ./examples/single-file/remotes.yaml:/app/remotes.yaml:ro,z
|
||||||
- ./ca-bundle.pem:/app/ca-bundle.pem:ro,z
|
- ./ca-bundle.pem:/app/ca-bundle.pem:ro,z
|
||||||
environment:
|
environment:
|
||||||
- CONFIG_PATH=/app/remotes.yaml
|
- CONFIG_PATH=/app/remotes.yaml
|
||||||
|
|||||||
@@ -0,0 +1,10 @@
|
|||||||
|
remotes:
|
||||||
|
alpine:
|
||||||
|
base_url: "https://dl-cdn.alpinelinux.org"
|
||||||
|
package: "alpine"
|
||||||
|
description: "Alpine Linux APK package repository"
|
||||||
|
immutable_patterns:
|
||||||
|
- ".*/x86_64/.*\\.apk$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 7200
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
remotes:
|
||||||
|
github:
|
||||||
|
base_url: "https://github.com"
|
||||||
|
package: "generic"
|
||||||
|
description: "GitHub releases and files"
|
||||||
|
immutable_patterns:
|
||||||
|
- "gruntwork-io/terragrunt/.*terragrunt_linux_amd64.*"
|
||||||
|
- "prometheus/node_exporter/.*/node_exporter-.*\\.linux-amd64\\.tar\\.gz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 0
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
remotes:
|
||||||
|
pypi:
|
||||||
|
base_url: "https://files.pythonhosted.org"
|
||||||
|
package: "pypi"
|
||||||
|
description: "Python Package Index"
|
||||||
|
check_mutable_updates: true
|
||||||
|
quarantine_new: true
|
||||||
|
quarantine_days: 3
|
||||||
|
immutable_patterns:
|
||||||
|
- "packages/.*\\.whl$"
|
||||||
|
- "packages/.*\\.whl\\.metadata$"
|
||||||
|
- "packages/.*\\.tar\\.gz$"
|
||||||
|
- "packages/.*\\.zip$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 600
|
||||||
@@ -9,6 +9,13 @@
|
|||||||
# immutable_ttl: TTL for immutable files (0 = forever, rarely needed to change).
|
# immutable_ttl: TTL for immutable files (0 = forever, rarely needed to change).
|
||||||
# mutable_ttl: TTL in seconds for mutable files. Omit to use the default (3600).
|
# mutable_ttl: TTL in seconds for mutable files. Omit to use the default (3600).
|
||||||
#
|
#
|
||||||
|
# quarantine_new: Set to true to block immutable artifacts published within the last
|
||||||
|
# quarantine_days days. Requests return 404 until the quarantine period
|
||||||
|
# expires. Fails open when the publish date cannot be determined.
|
||||||
|
# quarantine_days: Number of days to quarantine newly published artifacts (requires
|
||||||
|
# quarantine_new: true). The upstream Last-Modified header is used as
|
||||||
|
# the publish date.
|
||||||
|
#
|
||||||
# WARNING: this file may contain credentials — do not commit real values.
|
# WARNING: this file may contain credentials — do not commit real values.
|
||||||
#
|
#
|
||||||
# Global configuration
|
# Global configuration
|
||||||
@@ -27,9 +34,7 @@
|
|||||||
#
|
#
|
||||||
remotes:
|
remotes:
|
||||||
github:
|
github:
|
||||||
base_url: "https://github.com"
|
base_url: "https://github.com" package: "generic"
|
||||||
type: "remote"
|
|
||||||
package: "generic"
|
|
||||||
description: "GitHub releases and files"
|
description: "GitHub releases and files"
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
- "gruntwork-io/terragrunt/.*terragrunt_linux_amd64.*"
|
- "gruntwork-io/terragrunt/.*terragrunt_linux_amd64.*"
|
||||||
@@ -59,9 +64,7 @@ remotes:
|
|||||||
mutable_ttl: 0
|
mutable_ttl: 0
|
||||||
|
|
||||||
github-archive:
|
github-archive:
|
||||||
base_url: "https://github.com"
|
base_url: "https://github.com" package: "generic"
|
||||||
type: "remote"
|
|
||||||
package: "generic"
|
|
||||||
description: "GitHub repository archive tarballs"
|
description: "GitHub repository archive tarballs"
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
# Tag archives are immutable — a tag never changes
|
# Tag archives are immutable — a tag never changes
|
||||||
@@ -79,9 +82,7 @@ remotes:
|
|||||||
mutable_ttl: 86400 # Branch archives refreshed after 1 day
|
mutable_ttl: 86400 # Branch archives refreshed after 1 day
|
||||||
|
|
||||||
gitea-dl:
|
gitea-dl:
|
||||||
base_url: "https://dl.gitea.com"
|
base_url: "https://dl.gitea.com" package: "generic"
|
||||||
type: "remote"
|
|
||||||
package: "generic"
|
|
||||||
description: "Gitea download site"
|
description: "Gitea download site"
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
- "act_runner/.*/act_runner-.*-linux-amd64$"
|
- "act_runner/.*/act_runner-.*-linux-amd64$"
|
||||||
@@ -90,9 +91,7 @@ remotes:
|
|||||||
mutable_ttl: 0
|
mutable_ttl: 0
|
||||||
|
|
||||||
hashicorp-releases:
|
hashicorp-releases:
|
||||||
base_url: "https://releases.hashicorp.com"
|
base_url: "https://releases.hashicorp.com" package: "generic"
|
||||||
type: "remote"
|
|
||||||
package: "generic"
|
|
||||||
description: "HashiCorp product releases"
|
description: "HashiCorp product releases"
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
- "terraform/.*terraform_.*_linux_amd64\\.zip$"
|
- "terraform/.*terraform_.*_linux_amd64\\.zip$"
|
||||||
@@ -111,9 +110,7 @@ remotes:
|
|||||||
mutable_ttl: 0
|
mutable_ttl: 0
|
||||||
|
|
||||||
alpine:
|
alpine:
|
||||||
base_url: "https://dl-cdn.alpinelinux.org"
|
base_url: "https://dl-cdn.alpinelinux.org" package: "alpine"
|
||||||
type: "remote"
|
|
||||||
package: "alpine"
|
|
||||||
description: "Alpine Linux APK package repository"
|
description: "Alpine Linux APK package repository"
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
- ".*/x86_64/.*\\.apk$"
|
- ".*/x86_64/.*\\.apk$"
|
||||||
@@ -125,9 +122,7 @@ remotes:
|
|||||||
mutable_ttl: 7200 # Index files (APKINDEX.tar.gz) cached for 2 hours
|
mutable_ttl: 7200 # Index files (APKINDEX.tar.gz) cached for 2 hours
|
||||||
|
|
||||||
almalinux:
|
almalinux:
|
||||||
base_url: "https://gsl-syd.mm.fcix.net/almalinux"
|
base_url: "https://gsl-syd.mm.fcix.net/almalinux" package: "rpm"
|
||||||
type: "remote"
|
|
||||||
package: "rpm"
|
|
||||||
description: "AlmaLinux RPM package repository"
|
description: "AlmaLinux RPM package repository"
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
- ".*/x86_64/.*\\.rpm$"
|
- ".*/x86_64/.*\\.rpm$"
|
||||||
@@ -142,9 +137,7 @@ remotes:
|
|||||||
mutable_ttl: 7200 # Metadata files cached for 2 hours
|
mutable_ttl: 7200 # Metadata files cached for 2 hours
|
||||||
|
|
||||||
epel:
|
epel:
|
||||||
base_url: "http://mirror.aarnet.edu.au/pub/epel"
|
base_url: "http://mirror.aarnet.edu.au/pub/epel" package: "rpm"
|
||||||
type: "remote"
|
|
||||||
package: "rpm"
|
|
||||||
description: "EPEL (Extra Packages for Enterprise Linux)"
|
description: "EPEL (Extra Packages for Enterprise Linux)"
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
- "8/Everything/x86_64/.*\\.rpm$"
|
- "8/Everything/x86_64/.*\\.rpm$"
|
||||||
@@ -157,9 +150,7 @@ remotes:
|
|||||||
mutable_ttl: 7200 # Metadata files cached for 2 hours
|
mutable_ttl: 7200 # Metadata files cached for 2 hours
|
||||||
|
|
||||||
fedora:
|
fedora:
|
||||||
base_url: "https://gsl-syd.mm.fcix.net/fedora/linux"
|
base_url: "https://gsl-syd.mm.fcix.net/fedora/linux" package: "rpm"
|
||||||
type: "remote"
|
|
||||||
package: "rpm"
|
|
||||||
description: "Fedora Linux RPM package repository"
|
description: "Fedora Linux RPM package repository"
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
- "releases/.*/Everything/x86_64/.*\\.rpm$"
|
- "releases/.*/Everything/x86_64/.*\\.rpm$"
|
||||||
@@ -172,9 +163,7 @@ remotes:
|
|||||||
mutable_ttl: 300 # Metadata files cached for 5 minutes
|
mutable_ttl: 300 # Metadata files cached for 5 minutes
|
||||||
|
|
||||||
ghcr:
|
ghcr:
|
||||||
base_url: "https://ghcr.io"
|
base_url: "https://ghcr.io" package: "docker"
|
||||||
type: "remote"
|
|
||||||
package: "docker"
|
|
||||||
description: "GitHub Container Registry"
|
description: "GitHub Container Registry"
|
||||||
# username: "your-github-username"
|
# username: "your-github-username"
|
||||||
# password: "your-github-pat" # needs read:packages scope
|
# password: "your-github-pat" # needs read:packages scope
|
||||||
@@ -186,22 +175,23 @@ remotes:
|
|||||||
mutable_ttl: 300
|
mutable_ttl: 300
|
||||||
|
|
||||||
dockerhub:
|
dockerhub:
|
||||||
base_url: "https://registry-1.docker.io"
|
base_url: "https://registry-1.docker.io" package: "docker"
|
||||||
type: "remote"
|
|
||||||
package: "docker"
|
|
||||||
description: "Docker Hub registry"
|
description: "Docker Hub registry"
|
||||||
cache:
|
cache:
|
||||||
immutable_ttl: 0
|
immutable_ttl: 0
|
||||||
mutable_ttl: 300
|
mutable_ttl: 300
|
||||||
|
|
||||||
pypi:
|
pypi:
|
||||||
base_url: "https://files.pythonhosted.org"
|
base_url: "https://files.pythonhosted.org" package: "pypi"
|
||||||
type: "remote"
|
|
||||||
package: "pypi"
|
|
||||||
description: "Python Package Index — simple index and package files via a single remote"
|
description: "Python Package Index — simple index and package files via a single remote"
|
||||||
# simple/ requests are transparently fetched from pypi.org; package files come from
|
# simple/ requests are transparently fetched from pypi.org; package files come from
|
||||||
# files.pythonhosted.org (base_url). URLs in the simple index are rewritten to this remote.
|
# files.pythonhosted.org (base_url). URLs in the simple index are rewritten to this remote.
|
||||||
check_mutable_updates: true
|
check_mutable_updates: true
|
||||||
|
# Block packages published within the last 3 days (supply-chain attack mitigation).
|
||||||
|
# Immutable artifacts (wheel/sdist) newer than quarantine_days return 404 until
|
||||||
|
# the window passes. Disable by setting quarantine_new: false or removing both keys.
|
||||||
|
quarantine_new: true
|
||||||
|
quarantine_days: 3
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
- "packages/.*\\.whl$"
|
- "packages/.*\\.whl$"
|
||||||
- "packages/.*\\.whl\\.metadata$"
|
- "packages/.*\\.whl\\.metadata$"
|
||||||
@@ -213,9 +203,7 @@ remotes:
|
|||||||
mutable_ttl: 600 # Simple index pages refreshed after 10 minutes
|
mutable_ttl: 600 # Simple index pages refreshed after 10 minutes
|
||||||
|
|
||||||
pypi-gitea:
|
pypi-gitea:
|
||||||
base_url: "https://gitea.example.com/api/packages/myorg/pypi"
|
base_url: "https://gitea.example.com/api/packages/myorg/pypi" package: "pypi"
|
||||||
type: "remote"
|
|
||||||
package: "pypi"
|
|
||||||
description: "Private Gitea PyPI registry — simple index and files at the same host"
|
description: "Private Gitea PyPI registry — simple index and files at the same host"
|
||||||
# username: "your-gitea-username"
|
# username: "your-gitea-username"
|
||||||
# password: "your-personal-access-token" # needs package:read scope
|
# password: "your-personal-access-token" # needs package:read scope
|
||||||
@@ -231,9 +219,7 @@ remotes:
|
|||||||
mutable_ttl: 600
|
mutable_ttl: 600
|
||||||
|
|
||||||
npm:
|
npm:
|
||||||
base_url: "https://registry.npmjs.org"
|
base_url: "https://registry.npmjs.org" package: "npm"
|
||||||
type: "remote"
|
|
||||||
package: "npm"
|
|
||||||
description: "npm registry — package metadata with tarball URL rewriting"
|
description: "npm registry — package metadata with tarball URL rewriting"
|
||||||
check_mutable_updates: true
|
check_mutable_updates: true
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
@@ -245,9 +231,7 @@ remotes:
|
|||||||
mutable_ttl: 600 # Package metadata refreshed after 10 minutes
|
mutable_ttl: 600 # Package metadata refreshed after 10 minutes
|
||||||
|
|
||||||
hashicorp-helm:
|
hashicorp-helm:
|
||||||
base_url: "https://helm.releases.hashicorp.com"
|
base_url: "https://helm.releases.hashicorp.com" package: "helm"
|
||||||
type: "remote"
|
|
||||||
package: "helm"
|
|
||||||
description: "HashiCorp Helm chart repository (Vault, Consul, Nomad, etc.)"
|
description: "HashiCorp Helm chart repository (Vault, Consul, Nomad, etc.)"
|
||||||
check_mutable_updates: true
|
check_mutable_updates: true
|
||||||
immutable_patterns:
|
immutable_patterns:
|
||||||
@@ -256,8 +240,214 @@ remotes:
|
|||||||
immutable_ttl: 0 # Chart tarballs are versioned — cache forever
|
immutable_ttl: 0 # Chart tarballs are versioned — cache forever
|
||||||
mutable_ttl: 3600 # index.yaml refreshed after 1 hour
|
mutable_ttl: 3600 # index.yaml refreshed after 1 hour
|
||||||
|
|
||||||
|
metallb:
|
||||||
|
base_url: "https://metallb.github.io/metallb" package: "helm"
|
||||||
|
description: "MetalLB load balancer Helm charts"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
jetstack:
|
||||||
|
base_url: "https://charts.jetstack.io" package: "helm"
|
||||||
|
description: "Jetstack Helm charts (cert-manager)"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
rancher-stable:
|
||||||
|
base_url: "https://releases.rancher.com/server-charts/stable" package: "helm"
|
||||||
|
description: "Rancher stable Helm charts"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
purelb:
|
||||||
|
base_url: "https://gitlab.com/api/v4/projects/20400619/packages/helm/stable" package: "helm"
|
||||||
|
description: "PureLB load balancer Helm charts"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
istio:
|
||||||
|
base_url: "https://istio-release.storage.googleapis.com/charts" package: "helm"
|
||||||
|
description: "Istio service mesh Helm charts"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
cnpg:
|
||||||
|
base_url: "https://cloudnative-pg.github.io/charts" package: "helm"
|
||||||
|
description: "CloudNativePG operator Helm charts"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
ceph-csi:
|
||||||
|
base_url: "https://ceph.github.io/csi-charts" package: "helm"
|
||||||
|
description: "Ceph CSI driver Helm charts"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
external-dns:
|
||||||
|
base_url: "https://kubernetes-sigs.github.io/external-dns/" package: "helm"
|
||||||
|
description: "ExternalDNS Helm charts"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
intel-helm:
|
||||||
|
base_url: "https://intel.github.io/helm-charts/" package: "helm"
|
||||||
|
description: "Intel Helm charts"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
elastic:
|
||||||
|
base_url: "https://helm.elastic.co" package: "helm"
|
||||||
|
description: "Elastic stack Helm charts"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
k8up-io:
|
||||||
|
base_url: "https://k8up-io.github.io/k8up" package: "helm"
|
||||||
|
description: "K8up backup operator Helm charts"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
victoriametrics:
|
||||||
|
base_url: "https://victoriametrics.github.io/helm-charts/" package: "helm"
|
||||||
|
description: "VictoriaMetrics observability Helm charts"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
grafana:
|
||||||
|
base_url: "https://grafana.github.io/helm-charts" package: "helm"
|
||||||
|
description: "Grafana observability Helm charts"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
helm-openldap:
|
||||||
|
base_url: "https://jp-gouin.github.io/helm-openldap/" package: "helm"
|
||||||
|
description: "OpenLDAP Helm charts"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
woodpecker:
|
||||||
|
base_url: "https://woodpecker-ci.org/" package: "helm"
|
||||||
|
description: "Woodpecker CI Helm charts"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
stakater:
|
||||||
|
base_url: "https://stakater.github.io/stakater-charts" package: "helm"
|
||||||
|
description: "Stakater Helm charts"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
jfrog:
|
||||||
|
base_url: "https://charts.jfrog.io/" package: "helm"
|
||||||
|
description: "JFrog Helm charts"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
openvox:
|
||||||
|
base_url: "https://openvoxproject.github.io/openvox-helm-chart" package: "helm"
|
||||||
|
description: "OpenVox Helm charts"
|
||||||
|
check_mutable_updates: true
|
||||||
|
immutable_patterns:
|
||||||
|
- "\\.tgz$"
|
||||||
|
cache:
|
||||||
|
immutable_ttl: 0
|
||||||
|
mutable_ttl: 3600
|
||||||
|
|
||||||
|
|
||||||
|
virtuals:
|
||||||
|
helm-all:
|
||||||
|
package: "helm"
|
||||||
|
description: "Virtual repository merging all helm remotes — member order is priority order for duplicate chart+version"
|
||||||
|
members:
|
||||||
|
- hashicorp-helm
|
||||||
|
- metallb
|
||||||
|
- jetstack
|
||||||
|
- rancher-stable
|
||||||
|
- purelb
|
||||||
|
- istio
|
||||||
|
- cnpg
|
||||||
|
- ceph-csi
|
||||||
|
- external-dns
|
||||||
|
- intel-helm
|
||||||
|
- elastic
|
||||||
|
- k8up-io
|
||||||
|
- victoriametrics
|
||||||
|
- grafana
|
||||||
|
- helm-openldap
|
||||||
|
- woodpecker
|
||||||
|
- stakater
|
||||||
|
- jfrog
|
||||||
|
- openvox
|
||||||
|
|
||||||
|
locals:
|
||||||
local-generic:
|
local-generic:
|
||||||
type: "local"
|
|
||||||
package: "generic"
|
package: "generic"
|
||||||
description: "Local generic file repository"
|
description: "Local generic file repository"
|
||||||
cache:
|
cache:
|
||||||
@@ -59,6 +59,18 @@ async def proxy(request: Request, remote_name: str, path: str, storage, cache, c
|
|||||||
logger.info(f"Mutable file cached with TTL: {remote_name}/{path} (ttl: {mutable_ttl}s)")
|
logger.info(f"Mutable file cached with TTL: {remote_name}/{path} (ttl: {mutable_ttl}s)")
|
||||||
if result.get("etag") or result.get("last_modified"):
|
if result.get("etag") or result.get("last_modified"):
|
||||||
cache.store_mutable_meta(remote_name, path, result.get("etag"), result.get("last_modified"))
|
cache.store_mutable_meta(remote_name, path, result.get("etag"), result.get("last_modified"))
|
||||||
|
if not is_mutable:
|
||||||
|
published = result.get("last_modified")
|
||||||
|
if published:
|
||||||
|
cache.store_artifact_published(remote_name, path, published)
|
||||||
|
_proxy._check_quarantine(remote_name, published, config)
|
||||||
|
elif not is_mutable:
|
||||||
|
published = cache.get_artifact_published(remote_name, path)
|
||||||
|
if not published:
|
||||||
|
published = await _proxy._fetch_last_modified(remote_url, remote_config)
|
||||||
|
if published:
|
||||||
|
cache.store_artifact_published(remote_name, path, published)
|
||||||
|
_proxy._check_quarantine(remote_name, published, config)
|
||||||
|
|
||||||
artifact_data = storage.download_object(storage.get_object_key(remote_name, path))
|
artifact_data = storage.download_object(storage.get_object_key(remote_name, path))
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
import hashlib
|
import hashlib
|
||||||
import logging
|
import logging
|
||||||
|
import os
|
||||||
|
|
||||||
from fastapi import HTTPException, Response, UploadFile
|
from fastapi import HTTPException, Response, UploadFile
|
||||||
from fastapi.responses import JSONResponse
|
from fastapi.responses import JSONResponse
|
||||||
@@ -7,12 +8,23 @@ from fastapi.responses import JSONResponse
|
|||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def download(remote_name: str, path: str, storage, database, config) -> Response:
|
||||||
|
if not config.get_local_config(remote_name):
|
||||||
|
raise HTTPException(status_code=404, detail=f"Local repository '{remote_name}' not configured")
|
||||||
|
metadata = database.get_local_file_metadata(remote_name, path)
|
||||||
|
if not metadata:
|
||||||
|
raise HTTPException(status_code=404, detail="File not found")
|
||||||
|
content = storage.download_object(metadata["s3_key"])
|
||||||
|
return Response(
|
||||||
|
content=content,
|
||||||
|
media_type=metadata.get("content_type", "application/octet-stream"),
|
||||||
|
headers={"Content-Disposition": f"attachment; filename={os.path.basename(path)}"},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
async def upload(remote_name: str, path: str, file: UploadFile, storage, database, config) -> JSONResponse:
|
async def upload(remote_name: str, path: str, file: UploadFile, storage, database, config) -> JSONResponse:
|
||||||
remote_config = config.get_remote_config(remote_name)
|
if not config.get_local_config(remote_name):
|
||||||
if not remote_config:
|
raise HTTPException(status_code=404, detail=f"Local repository '{remote_name}' not configured")
|
||||||
raise HTTPException(status_code=404, detail=f"Remote '{remote_name}' not configured")
|
|
||||||
if remote_config.get("type") != "local":
|
|
||||||
raise HTTPException(status_code=400, detail="Upload only supported for local repositories")
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
content = await file.read()
|
content = await file.read()
|
||||||
@@ -59,12 +71,8 @@ async def upload(remote_name: str, path: str, file: UploadFile, storage, databas
|
|||||||
|
|
||||||
|
|
||||||
def check_exists(remote_name: str, path: str, database, config) -> Response:
|
def check_exists(remote_name: str, path: str, database, config) -> Response:
|
||||||
remote_config = config.get_remote_config(remote_name)
|
if not config.get_local_config(remote_name):
|
||||||
if not remote_config:
|
raise HTTPException(status_code=404, detail=f"Local repository '{remote_name}' not configured")
|
||||||
raise HTTPException(status_code=404, detail=f"Remote '{remote_name}' not configured")
|
|
||||||
|
|
||||||
if remote_config.get("type") != "local":
|
|
||||||
raise HTTPException(status_code=405, detail="HEAD method only supported for local repositories")
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
metadata = database.get_local_file_metadata(remote_name, path)
|
metadata = database.get_local_file_metadata(remote_name, path)
|
||||||
@@ -87,11 +95,8 @@ def check_exists(remote_name: str, path: str, database, config) -> Response:
|
|||||||
|
|
||||||
|
|
||||||
def delete(remote_name: str, path: str, storage, database, config) -> JSONResponse:
|
def delete(remote_name: str, path: str, storage, database, config) -> JSONResponse:
|
||||||
remote_config = config.get_remote_config(remote_name)
|
if not config.get_local_config(remote_name):
|
||||||
if not remote_config:
|
raise HTTPException(status_code=404, detail=f"Local repository '{remote_name}' not configured")
|
||||||
raise HTTPException(status_code=404, detail=f"Remote '{remote_name}' not configured")
|
|
||||||
if remote_config.get("type") != "local":
|
|
||||||
raise HTTPException(status_code=400, detail="Delete only supported for local repositories")
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
s3_key = database.delete_local_file(remote_name, path)
|
s3_key = database.delete_local_file(remote_name, path)
|
||||||
|
|||||||
@@ -2,6 +2,8 @@ import base64
|
|||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
from email.utils import parsedate_to_datetime
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
from fastapi import HTTPException, Request, Response
|
from fastapi import HTTPException, Request, Response
|
||||||
@@ -19,6 +21,42 @@ class UpstreamUnreachable(Exception):
|
|||||||
"""Raised when the upstream backend cannot be contacted (network or timeout error)."""
|
"""Raised when the upstream backend cannot be contacted (network or timeout error)."""
|
||||||
|
|
||||||
|
|
||||||
|
def _check_quarantine(remote_name: str, last_modified_str: str | None, config) -> None:
|
||||||
|
"""Raise HTTP 404 if the artifact is within the per-remote quarantine window.
|
||||||
|
|
||||||
|
Fails open (allows the request) when the publish date cannot be determined.
|
||||||
|
"""
|
||||||
|
enabled, days = config.get_quarantine_config(remote_name)
|
||||||
|
if not enabled or not days:
|
||||||
|
return
|
||||||
|
if not last_modified_str:
|
||||||
|
return # cannot determine age → allow
|
||||||
|
try:
|
||||||
|
publish_date = parsedate_to_datetime(last_modified_str)
|
||||||
|
except Exception:
|
||||||
|
return # unparseable → allow
|
||||||
|
cutoff = datetime.now(UTC) - timedelta(days=days)
|
||||||
|
if publish_date > cutoff:
|
||||||
|
available_on = (publish_date + timedelta(days=days)).date()
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=404,
|
||||||
|
detail=(
|
||||||
|
f"Package quarantined: published {publish_date.date()}, available after {available_on} ({days}-day new-release quarantine)"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def _fetch_last_modified(remote_url: str, remote_cfg: dict) -> str | None:
|
||||||
|
"""HEAD the upstream URL and return the Last-Modified header, or None on any failure."""
|
||||||
|
auth = _basic_auth_header(remote_cfg)
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient(follow_redirects=True) as client:
|
||||||
|
response = await client.head(remote_url, headers=auth, timeout=10.0)
|
||||||
|
return response.headers.get("Last-Modified")
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def _basic_auth_header(remote_cfg: dict) -> dict[str, str]:
|
def _basic_auth_header(remote_cfg: dict) -> dict[str, str]:
|
||||||
username = remote_cfg.get("username")
|
username = remote_cfg.get("username")
|
||||||
password = remote_cfg.get("password")
|
password = remote_cfg.get("password")
|
||||||
@@ -180,19 +218,6 @@ async def handle(request: Request, remote_name: str, path: str, storage, cache,
|
|||||||
if not remote_config:
|
if not remote_config:
|
||||||
raise HTTPException(status_code=404, detail=f"Remote '{remote_name}' not configured")
|
raise HTTPException(status_code=404, detail=f"Remote '{remote_name}' not configured")
|
||||||
|
|
||||||
if remote_config.get("type") == "local":
|
|
||||||
metadata = database.get_local_file_metadata(remote_name, path)
|
|
||||||
if not metadata:
|
|
||||||
raise HTTPException(status_code=404, detail="File not found")
|
|
||||||
content = storage.download_object(metadata["s3_key"])
|
|
||||||
if content is None:
|
|
||||||
raise HTTPException(status_code=500, detail="File not accessible")
|
|
||||||
return Response(
|
|
||||||
content=content,
|
|
||||||
media_type=metadata.get("content_type", "application/octet-stream"),
|
|
||||||
headers={"Content-Disposition": f"attachment; filename={os.path.basename(path)}"},
|
|
||||||
)
|
|
||||||
|
|
||||||
path_parts = path.split("/")
|
path_parts = path.split("/")
|
||||||
if len(path_parts) >= 2:
|
if len(path_parts) >= 2:
|
||||||
repo_path = f"{path_parts[0]}/{path_parts[1]}"
|
repo_path = f"{path_parts[0]}/{path_parts[1]}"
|
||||||
@@ -225,6 +250,14 @@ async def handle(request: Request, remote_name: str, path: str, storage, cache,
|
|||||||
cached_key = None
|
cached_key = None
|
||||||
|
|
||||||
if cached_key:
|
if cached_key:
|
||||||
|
if not is_mutable:
|
||||||
|
published = cache.get_artifact_published(remote_name, path)
|
||||||
|
if not published:
|
||||||
|
published = await _fetch_last_modified(remote_url, remote_config)
|
||||||
|
if published:
|
||||||
|
cache.store_artifact_published(remote_name, path, published)
|
||||||
|
_check_quarantine(remote_name, published, config)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
artifact_data = storage.download_object(cached_key)
|
artifact_data = storage.download_object(cached_key)
|
||||||
artifact_data, content_type = _resolve_content(artifact_data, path, filename, remote_config, request, remote_name)
|
artifact_data, content_type = _resolve_content(artifact_data, path, filename, remote_config, request, remote_name)
|
||||||
@@ -240,6 +273,8 @@ async def handle(request: Request, remote_name: str, path: str, storage, cache,
|
|||||||
"X-Artifact-Size": str(len(artifact_data)),
|
"X-Artifact-Size": str(len(artifact_data)),
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
except HTTPException:
|
||||||
|
raise
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise HTTPException(status_code=500, detail=f"Error retrieving cached artifact: {str(e)}")
|
raise HTTPException(status_code=500, detail=f"Error retrieving cached artifact: {str(e)}")
|
||||||
|
|
||||||
@@ -258,6 +293,12 @@ async def handle(request: Request, remote_name: str, path: str, storage, cache,
|
|||||||
if result.get("etag") or result.get("last_modified"):
|
if result.get("etag") or result.get("last_modified"):
|
||||||
cache.store_mutable_meta(remote_name, path, result.get("etag"), result.get("last_modified"))
|
cache.store_mutable_meta(remote_name, path, result.get("etag"), result.get("last_modified"))
|
||||||
|
|
||||||
|
if not is_mutable:
|
||||||
|
published = result.get("last_modified")
|
||||||
|
if published:
|
||||||
|
cache.store_artifact_published(remote_name, path, published)
|
||||||
|
_check_quarantine(remote_name, published, config)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
cache_key = storage.get_object_key(remote_name, path)
|
cache_key = storage.get_object_key(remote_name, path)
|
||||||
artifact_data = storage.download_object(cache_key)
|
artifact_data = storage.download_object(cache_key)
|
||||||
|
|||||||
@@ -0,0 +1,225 @@
|
|||||||
|
import asyncio
|
||||||
|
import base64
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
from datetime import UTC, date, datetime
|
||||||
|
from typing import Protocol, runtime_checkable
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
import yaml
|
||||||
|
from fastapi import HTTPException, Request, Response
|
||||||
|
|
||||||
|
from ..remote import helm as _helm
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class _HelmDumper(yaml.Dumper):
|
||||||
|
"""YAML dumper that serializes datetime/date objects back to ISO 8601 strings.
|
||||||
|
|
||||||
|
yaml.safe_load converts timestamp-shaped YAML scalars (e.g. chart `created`
|
||||||
|
fields) to Python datetime objects. Without a custom representer, yaml.dump
|
||||||
|
would render them as "2022-12-16 11:08:49+00:00" (space, not T), which
|
||||||
|
Go's YAML parser cannot unmarshal into time.Time.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _repr_datetime(dumper: yaml.Dumper, data: datetime) -> yaml.ScalarNode:
|
||||||
|
s = data.strftime("%Y-%m-%dT%H:%M:%S.%f") + ("Z" if data.tzinfo else "")
|
||||||
|
return dumper.represent_scalar("tag:yaml.org,2002:str", s)
|
||||||
|
|
||||||
|
|
||||||
|
def _repr_date(dumper: yaml.Dumper, data: date) -> yaml.ScalarNode:
|
||||||
|
return dumper.represent_scalar("tag:yaml.org,2002:str", data.isoformat())
|
||||||
|
|
||||||
|
|
||||||
|
_HelmDumper.add_representer(datetime, _repr_datetime)
|
||||||
|
_HelmDumper.add_representer(date, _repr_date)
|
||||||
|
|
||||||
|
|
||||||
|
async def _get_member_index(
|
||||||
|
member_name: str,
|
||||||
|
member_cfg: dict,
|
||||||
|
path: str,
|
||||||
|
storage,
|
||||||
|
cache,
|
||||||
|
) -> tuple[str, dict, int, bytes | None]:
|
||||||
|
"""Fetch or retrieve cached index.yaml for one member remote.
|
||||||
|
|
||||||
|
Returns (member_name, member_cfg, ttl, raw_bytes).
|
||||||
|
raw_bytes is None if the member is unreachable and not in S3.
|
||||||
|
"""
|
||||||
|
member_ttl = member_cfg.get("cache", {}).get("mutable_ttl", 3600)
|
||||||
|
s3_key = storage.get_object_key(member_name, path)
|
||||||
|
raw_data: bytes | None = None
|
||||||
|
|
||||||
|
if storage.exists(s3_key) and cache.is_index_valid(member_name, path):
|
||||||
|
try:
|
||||||
|
raw_data = storage.download_object(s3_key)
|
||||||
|
logger.info(f"Virtual: cache hit for member '{member_name}'")
|
||||||
|
except Exception:
|
||||||
|
raw_data = None
|
||||||
|
|
||||||
|
if raw_data is None:
|
||||||
|
base_url = member_cfg.get("base_url", "").rstrip("/")
|
||||||
|
upstream_url = f"{base_url}/index.yaml"
|
||||||
|
headers = {}
|
||||||
|
username = member_cfg.get("username")
|
||||||
|
password = member_cfg.get("password")
|
||||||
|
if username and password:
|
||||||
|
token = base64.b64encode(f"{username}:{password}".encode()).decode()
|
||||||
|
headers["Authorization"] = f"Basic {token}"
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient(follow_redirects=True) as client:
|
||||||
|
response = await client.get(upstream_url, headers=headers, timeout=30.0)
|
||||||
|
response.raise_for_status()
|
||||||
|
raw_data = response.content
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Virtual: failed to fetch index.yaml from member '{member_name}': {e}")
|
||||||
|
return member_name, member_cfg, member_ttl, None
|
||||||
|
try:
|
||||||
|
storage.upload(s3_key, raw_data)
|
||||||
|
cache.mark_index_cached(member_name, path, member_ttl)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Virtual: failed to cache index.yaml for member '{member_name}': {e}")
|
||||||
|
|
||||||
|
return member_name, member_cfg, member_ttl, raw_data
|
||||||
|
|
||||||
|
|
||||||
|
def _merge_helm_indexes(raw_indexes: list[bytes], member_names: list[str], member_configs: list[dict], proxy_base: str) -> bytes:
|
||||||
|
"""Merge helm index.yaml files with per-member URL rewriting.
|
||||||
|
|
||||||
|
Priority is determined by position in member_names: earlier members win
|
||||||
|
when the same chart name + version appears in multiple remotes.
|
||||||
|
"""
|
||||||
|
merged_entries: dict[str, list] = {}
|
||||||
|
|
||||||
|
for raw_data, member_name, member_cfg in zip(raw_indexes, member_names, member_configs):
|
||||||
|
base_url = member_cfg.get("base_url", "").rstrip("/")
|
||||||
|
rewritten, _ = _helm.resolve_content(raw_data, "index.yaml", "index.yaml", base_url, proxy_base, member_name)
|
||||||
|
|
||||||
|
try:
|
||||||
|
index = yaml.safe_load(rewritten)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Virtual: failed to parse index.yaml from member '{member_name}': {e}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
for chart_name, versions in (index.get("entries") or {}).items():
|
||||||
|
if chart_name not in merged_entries:
|
||||||
|
merged_entries[chart_name] = list(versions)
|
||||||
|
else:
|
||||||
|
existing = {(v.get("name"), v.get("version")) for v in merged_entries[chart_name]}
|
||||||
|
for version_entry in versions:
|
||||||
|
key = (version_entry.get("name"), version_entry.get("version"))
|
||||||
|
if key not in existing:
|
||||||
|
merged_entries[chart_name].append(version_entry)
|
||||||
|
existing.add(key)
|
||||||
|
|
||||||
|
merged = {
|
||||||
|
"apiVersion": "v1",
|
||||||
|
"entries": merged_entries,
|
||||||
|
"generated": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S.000Z"),
|
||||||
|
}
|
||||||
|
return yaml.dump(merged, Dumper=_HelmDumper, default_flow_style=False, allow_unicode=True).encode()
|
||||||
|
|
||||||
|
|
||||||
|
@runtime_checkable
|
||||||
|
class _VirtualHandler(Protocol):
|
||||||
|
def accepts_path(self, path: str) -> bool: ...
|
||||||
|
def merge(self, raw_indexes: list[bytes], member_names: list[str], member_configs: list[dict], proxy_base: str) -> bytes: ...
|
||||||
|
def path_error(self) -> str: ...
|
||||||
|
|
||||||
|
|
||||||
|
class _HelmHandler:
|
||||||
|
def accepts_path(self, path: str) -> bool:
|
||||||
|
return path == "index.yaml"
|
||||||
|
|
||||||
|
def merge(self, raw_indexes: list[bytes], member_names: list[str], member_configs: list[dict], proxy_base: str) -> bytes:
|
||||||
|
return _merge_helm_indexes(raw_indexes, member_names, member_configs, proxy_base)
|
||||||
|
|
||||||
|
def path_error(self) -> str:
|
||||||
|
return "Virtual helm repositories only serve index.yaml; chart tarballs are served directly by member remotes"
|
||||||
|
|
||||||
|
|
||||||
|
_HANDLERS: dict[str, _VirtualHandler] = {
|
||||||
|
"helm": _HelmHandler(),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
async def handle(request: Request, virtual_name: str, path: str, storage, cache, config) -> Response:
|
||||||
|
virtual_cfg = config.get_virtual_config(virtual_name)
|
||||||
|
if not virtual_cfg:
|
||||||
|
raise HTTPException(status_code=404, detail=f"Virtual repository '{virtual_name}' not configured")
|
||||||
|
|
||||||
|
package = virtual_cfg.get("package")
|
||||||
|
handler = _HANDLERS.get(package)
|
||||||
|
if handler is None:
|
||||||
|
raise HTTPException(status_code=400, detail=f"Virtual repositories with package '{package}' are not yet supported")
|
||||||
|
|
||||||
|
if not handler.accepts_path(path):
|
||||||
|
raise HTTPException(status_code=404, detail=handler.path_error())
|
||||||
|
|
||||||
|
members = virtual_cfg.get("members", [])
|
||||||
|
if not members:
|
||||||
|
raise HTTPException(status_code=500, detail=f"Virtual repository '{virtual_name}' has no members configured")
|
||||||
|
|
||||||
|
virtual_key = storage.get_object_key(virtual_name, path)
|
||||||
|
|
||||||
|
if cache.is_index_valid(virtual_name, path) and storage.exists(virtual_key):
|
||||||
|
data = storage.download_object(virtual_key)
|
||||||
|
logger.info(f"Virtual HIT: {virtual_name}/{path}")
|
||||||
|
return Response(content=data, media_type="text/yaml")
|
||||||
|
|
||||||
|
# Resolve configs first (config reads are sync/cheap)
|
||||||
|
member_entries = []
|
||||||
|
for member_name in members:
|
||||||
|
member_cfg = config.get_remote_config(member_name)
|
||||||
|
if not member_cfg:
|
||||||
|
logger.warning(f"Virtual '{virtual_name}': member '{member_name}' not found in config, skipping")
|
||||||
|
continue
|
||||||
|
member_entries.append((member_name, member_cfg))
|
||||||
|
|
||||||
|
# Fetch all member indexes in parallel; asyncio.gather preserves input order
|
||||||
|
proxy_base = str(request.base_url).rstrip("/")
|
||||||
|
t_fetch = time.perf_counter()
|
||||||
|
results = await asyncio.gather(*[_get_member_index(name, cfg, path, storage, cache) for name, cfg in member_entries])
|
||||||
|
fetch_ms = int((time.perf_counter() - t_fetch) * 1000)
|
||||||
|
|
||||||
|
raw_indexes: list[bytes] = []
|
||||||
|
used_members: list[str] = []
|
||||||
|
used_configs: list[dict] = []
|
||||||
|
min_ttl: int | None = None
|
||||||
|
|
||||||
|
for member_name, member_cfg, member_ttl, raw_data in results:
|
||||||
|
if min_ttl is None or member_ttl < min_ttl:
|
||||||
|
min_ttl = member_ttl
|
||||||
|
if raw_data is None:
|
||||||
|
logger.warning(f"Virtual '{virtual_name}': skipping unreachable member '{member_name}'")
|
||||||
|
continue
|
||||||
|
raw_indexes.append(raw_data)
|
||||||
|
used_members.append(member_name)
|
||||||
|
used_configs.append(member_cfg)
|
||||||
|
|
||||||
|
if not raw_indexes:
|
||||||
|
raise HTTPException(status_code=502, detail=f"Virtual repository '{virtual_name}': no member indices could be fetched")
|
||||||
|
|
||||||
|
if min_ttl is None:
|
||||||
|
min_ttl = 3600
|
||||||
|
|
||||||
|
t_merge = time.perf_counter()
|
||||||
|
merged = handler.merge(raw_indexes, used_members, used_configs, proxy_base)
|
||||||
|
merge_ms = int((time.perf_counter() - t_merge) * 1000)
|
||||||
|
|
||||||
|
try:
|
||||||
|
t_store = time.perf_counter()
|
||||||
|
storage.upload(virtual_key, merged)
|
||||||
|
cache.mark_index_cached(virtual_name, path, min_ttl)
|
||||||
|
store_ms = int((time.perf_counter() - t_store) * 1000)
|
||||||
|
logger.info(
|
||||||
|
f"Virtual MISS: {virtual_name}/{path} rebuilt from {used_members} "
|
||||||
|
f"(fetch={fetch_ms}ms merge={merge_ms}ms store={store_ms}ms ttl={min_ttl}s)"
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Virtual: failed to store merged index for '{virtual_name}': {e}")
|
||||||
|
|
||||||
|
return Response(content=merged, media_type="text/yaml")
|
||||||
Vendored
+21
@@ -78,6 +78,27 @@ class RedisCache:
|
|||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
def get_artifact_published_key(self, remote_name: str, path: str) -> str:
|
||||||
|
return f"pkg:published:{remote_name}:{hashlib.sha256(path.encode()).hexdigest()[:16]}"
|
||||||
|
|
||||||
|
def store_artifact_published(self, remote_name: str, path: str, last_modified: str) -> None:
|
||||||
|
"""Persist the upstream Last-Modified header for a (typically immutable) artifact."""
|
||||||
|
if not self.available:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
self.client.set(self.get_artifact_published_key(remote_name, path), last_modified)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def get_artifact_published(self, remote_name: str, path: str) -> str | None:
|
||||||
|
"""Return the stored Last-Modified string for an artifact, or None."""
|
||||||
|
if not self.available:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return self.client.get(self.get_artifact_published_key(remote_name, path))
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
def cleanup_expired_index(self, storage, remote_name: str, path: str) -> None:
|
def cleanup_expired_index(self, storage, remote_name: str, path: str) -> None:
|
||||||
if not self.available:
|
if not self.available:
|
||||||
return
|
return
|
||||||
|
|||||||
+92
-14
@@ -1,3 +1,4 @@
|
|||||||
|
import glob
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
|
||||||
@@ -30,31 +31,87 @@ _PACKAGE_MUTABLE_PATTERNS: dict[str, list[str]] = {
|
|||||||
|
|
||||||
|
|
||||||
class ConfigManager:
|
class ConfigManager:
|
||||||
def __init__(self, config_file: str = "remotes.yaml"):
|
def __init__(self, config_path: str = "remotes.yaml"):
|
||||||
self.config_file = config_file
|
self.config_path = config_path
|
||||||
self._last_modified = 0
|
self._config_dir: str | None = None
|
||||||
|
self._last_modified: float = 0.0
|
||||||
self.config = self._load_config()
|
self.config = self._load_config()
|
||||||
|
|
||||||
def _load_config(self) -> dict:
|
def _load_single_file(self, path: str) -> dict:
|
||||||
try:
|
try:
|
||||||
with open(self.config_file) as f:
|
with open(path) as f:
|
||||||
if self.config_file.endswith(".yaml") or self.config_file.endswith(".yml"):
|
if path.endswith((".yaml", ".yml")):
|
||||||
return yaml.safe_load(f)
|
return yaml.safe_load(f) or {}
|
||||||
else:
|
|
||||||
return json.load(f)
|
return json.load(f)
|
||||||
except FileNotFoundError:
|
except FileNotFoundError:
|
||||||
return {"remotes": {}}
|
return {}
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _merge(base: dict, overlay: dict) -> dict:
|
||||||
|
result = {**base}
|
||||||
|
for key, value in overlay.items():
|
||||||
|
if key in ("remotes", "virtuals", "locals") and isinstance(base.get(key), dict) and isinstance(value, dict):
|
||||||
|
result[key] = {**base.get(key, {}), **value}
|
||||||
|
else:
|
||||||
|
result[key] = value
|
||||||
|
return result
|
||||||
|
|
||||||
|
def _load_from_dir(self, dir_path: str) -> dict:
|
||||||
|
merged: dict = {}
|
||||||
|
files = sorted(glob.glob(os.path.join(dir_path, "*.yaml")) + glob.glob(os.path.join(dir_path, "*.yml")))
|
||||||
|
for path in files:
|
||||||
|
merged = self._merge(merged, self._load_single_file(path))
|
||||||
|
return merged
|
||||||
|
|
||||||
|
def _load_config(self) -> dict:
|
||||||
|
self._config_dir = None
|
||||||
|
|
||||||
|
if os.path.isdir(self.config_path):
|
||||||
|
return self._load_from_dir(self.config_path) or {"remotes": {}, "virtuals": {}, "locals": {}}
|
||||||
|
|
||||||
|
config = self._load_single_file(self.config_path)
|
||||||
|
if not config:
|
||||||
|
return {"remotes": {}, "virtuals": {}, "locals": {}}
|
||||||
|
|
||||||
|
config_dir = config.pop("config_dir", None)
|
||||||
|
if config_dir:
|
||||||
|
if not os.path.isabs(config_dir):
|
||||||
|
config_dir = os.path.join(os.path.dirname(os.path.abspath(self.config_path)), config_dir)
|
||||||
|
self._config_dir = config_dir
|
||||||
|
config = self._merge(config, self._load_from_dir(config_dir))
|
||||||
|
|
||||||
|
return config
|
||||||
|
|
||||||
|
def _file_mtimes(self) -> list[float]:
|
||||||
|
mtimes: list[float] = []
|
||||||
|
if os.path.isdir(self.config_path):
|
||||||
|
for f in glob.glob(os.path.join(self.config_path, "*.yaml")) + glob.glob(os.path.join(self.config_path, "*.yml")):
|
||||||
|
try:
|
||||||
|
mtimes.append(os.path.getmtime(f))
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
mtimes.append(os.path.getmtime(self.config_path))
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if self._config_dir and os.path.isdir(self._config_dir):
|
||||||
|
for f in glob.glob(os.path.join(self._config_dir, "*.yaml")) + glob.glob(os.path.join(self._config_dir, "*.yml")):
|
||||||
|
try:
|
||||||
|
mtimes.append(os.path.getmtime(f))
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
return mtimes
|
||||||
|
|
||||||
def _check_reload(self) -> None:
|
def _check_reload(self) -> None:
|
||||||
"""Check if config file has been modified and reload if needed"""
|
|
||||||
try:
|
try:
|
||||||
import os
|
current_modified = max(self._file_mtimes(), default=0.0)
|
||||||
|
|
||||||
current_modified = os.path.getmtime(self.config_file)
|
|
||||||
if current_modified > self._last_modified:
|
if current_modified > self._last_modified:
|
||||||
self._last_modified = current_modified
|
self._last_modified = current_modified
|
||||||
self.config = self._load_config()
|
self.config = self._load_config()
|
||||||
print(f"Config reloaded from {self.config_file}")
|
print(f"Config reloaded from {self.config_path}")
|
||||||
except OSError:
|
except OSError:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
@@ -62,6 +119,14 @@ class ConfigManager:
|
|||||||
self._check_reload()
|
self._check_reload()
|
||||||
return self.config.get("remotes", {}).get(remote_name)
|
return self.config.get("remotes", {}).get(remote_name)
|
||||||
|
|
||||||
|
def get_virtual_config(self, virtual_name: str) -> dict | None:
|
||||||
|
self._check_reload()
|
||||||
|
return self.config.get("virtuals", {}).get(virtual_name)
|
||||||
|
|
||||||
|
def get_local_config(self, local_name: str) -> dict | None:
|
||||||
|
self._check_reload()
|
||||||
|
return self.config.get("locals", {}).get(local_name)
|
||||||
|
|
||||||
def get_immutable_patterns(self, remote_name: str, repo_path: str = "") -> list[str]:
|
def get_immutable_patterns(self, remote_name: str, repo_path: str = "") -> list[str]:
|
||||||
remote_config = self.get_remote_config(remote_name)
|
remote_config = self.get_remote_config(remote_name)
|
||||||
if not remote_config:
|
if not remote_config:
|
||||||
@@ -159,3 +224,16 @@ class ConfigManager:
|
|||||||
return {}
|
return {}
|
||||||
|
|
||||||
return remote_config.get("cache", {})
|
return remote_config.get("cache", {})
|
||||||
|
|
||||||
|
def get_quarantine_config(self, remote_name: str) -> tuple[bool, int]:
|
||||||
|
"""Return (enabled, quarantine_days) for a remote.
|
||||||
|
|
||||||
|
When enabled=True and quarantine_days>0, immutable artifacts published
|
||||||
|
within the last quarantine_days days are blocked with a 404.
|
||||||
|
"""
|
||||||
|
remote_config = self.get_remote_config(remote_name)
|
||||||
|
if not remote_config:
|
||||||
|
return False, 0
|
||||||
|
enabled = bool(remote_config.get("quarantine_new", False))
|
||||||
|
days = int(remote_config.get("quarantine_days", 0))
|
||||||
|
return enabled, days
|
||||||
|
|||||||
+27
-11
@@ -13,7 +13,7 @@ try:
|
|||||||
except ImportError:
|
except ImportError:
|
||||||
__version__ = "dev"
|
__version__ = "dev"
|
||||||
|
|
||||||
from .artifact import discovery, flush, local, proxy
|
from .artifact import discovery, flush, local, proxy, virtual
|
||||||
from .artifact import docker as docker_handler
|
from .artifact import docker as docker_handler
|
||||||
from .cache import RedisCache
|
from .cache import RedisCache
|
||||||
from .config import ConfigManager
|
from .config import ConfigManager
|
||||||
@@ -49,7 +49,13 @@ class ArtifactRequest(BaseModel):
|
|||||||
@app.get("/")
|
@app.get("/")
|
||||||
def read_root():
|
def read_root():
|
||||||
config._check_reload()
|
config._check_reload()
|
||||||
return {"message": "Artifact Storage API", "version": app.version, "remotes": list(config.config.get("remotes", {}).keys())}
|
return {
|
||||||
|
"message": "Artifact Storage API",
|
||||||
|
"version": app.version,
|
||||||
|
"remotes": list(config.config.get("remotes", {}).keys()),
|
||||||
|
"virtuals": list(config.config.get("virtuals", {}).keys()),
|
||||||
|
"locals": list(config.config.get("locals", {}).keys()),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
@app.get("/health")
|
@app.get("/health")
|
||||||
@@ -89,24 +95,34 @@ async def docker_v2_proxy(request: Request, remote_name: str, path: str):
|
|||||||
return await docker_handler.proxy(request, remote_name, path, storage, cache, config, metrics)
|
return await docker_handler.proxy(request, remote_name, path, storage, cache, config, metrics)
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/api/v1/virtual/{virtual_name}/{path:path}")
|
||||||
|
async def get_virtual_artifact(request: Request, virtual_name: str, path: str):
|
||||||
|
return await virtual.handle(request, virtual_name, path, storage, cache, config)
|
||||||
|
|
||||||
|
|
||||||
@app.get("/api/v1/remote/{remote_name}/{path:path}")
|
@app.get("/api/v1/remote/{remote_name}/{path:path}")
|
||||||
async def get_artifact(request: Request, remote_name: str, path: str):
|
async def get_artifact(request: Request, remote_name: str, path: str):
|
||||||
return await proxy.handle(request, remote_name, path, storage, cache, config, database, metrics)
|
return await proxy.handle(request, remote_name, path, storage, cache, config, database, metrics)
|
||||||
|
|
||||||
|
|
||||||
@app.put("/api/v1/remote/{remote_name}/{path:path}")
|
@app.get("/api/v1/local/{local_name}/{path:path}")
|
||||||
async def upload_file(remote_name: str, path: str, file: UploadFile = File(...)):
|
def get_local_artifact(local_name: str, path: str):
|
||||||
return await local.upload(remote_name, path, file, storage, database, config)
|
return local.download(local_name, path, storage, database, config)
|
||||||
|
|
||||||
|
|
||||||
@app.head("/api/v1/remote/{remote_name}/{path:path}")
|
@app.put("/api/v1/local/{local_name}/{path:path}")
|
||||||
def check_file_exists(remote_name: str, path: str):
|
async def upload_local_file(local_name: str, path: str, file: UploadFile = File(...)):
|
||||||
return local.check_exists(remote_name, path, database, config)
|
return await local.upload(local_name, path, file, storage, database, config)
|
||||||
|
|
||||||
|
|
||||||
@app.delete("/api/v1/remote/{remote_name}/{path:path}")
|
@app.head("/api/v1/local/{local_name}/{path:path}")
|
||||||
def delete_file(remote_name: str, path: str):
|
def check_local_file_exists(local_name: str, path: str):
|
||||||
return local.delete(remote_name, path, storage, database, config)
|
return local.check_exists(local_name, path, database, config)
|
||||||
|
|
||||||
|
|
||||||
|
@app.delete("/api/v1/local/{local_name}/{path:path}")
|
||||||
|
def delete_local_file(local_name: str, path: str):
|
||||||
|
return local.delete(local_name, path, storage, database, config)
|
||||||
|
|
||||||
|
|
||||||
@app.post("/api/v1/artifacts/cache")
|
@app.post("/api/v1/artifacts/cache")
|
||||||
|
|||||||
@@ -87,9 +87,10 @@ class MetricsManager:
|
|||||||
# Get from database if available
|
# Get from database if available
|
||||||
db_sizes = self.database_manager.get_storage_by_remote()
|
db_sizes = self.database_manager.get_storage_by_remote()
|
||||||
if db_sizes:
|
if db_sizes:
|
||||||
# Initialize all configured remotes to 0
|
# Initialize all configured remotes and locals to 0
|
||||||
remote_sizes = {}
|
remote_sizes = {}
|
||||||
for remote in config_manager.config.get("remotes", {}).keys():
|
all_names = list(config_manager.config.get("remotes", {}).keys()) + list(config_manager.config.get("locals", {}).keys())
|
||||||
|
for remote in all_names:
|
||||||
remote_sizes[remote] = db_sizes.get(remote, 0)
|
remote_sizes[remote] = db_sizes.get(remote, 0)
|
||||||
|
|
||||||
# Update Prometheus gauges
|
# Update Prometheus gauges
|
||||||
@@ -101,10 +102,10 @@ class MetricsManager:
|
|||||||
# Fallback to S3 scanning if database not available
|
# Fallback to S3 scanning if database not available
|
||||||
try:
|
try:
|
||||||
remote_sizes = {}
|
remote_sizes = {}
|
||||||
remotes = config_manager.config.get("remotes", {}).keys()
|
all_names = list(config_manager.config.get("remotes", {}).keys()) + list(config_manager.config.get("locals", {}).keys())
|
||||||
|
|
||||||
# Initialize all remotes to 0
|
# Initialize all remotes and locals to 0
|
||||||
for remote in remotes:
|
for remote in all_names:
|
||||||
remote_sizes[remote] = 0
|
remote_sizes[remote] = 0
|
||||||
|
|
||||||
paginator = storage.client.get_paginator("list_objects_v2")
|
paginator = storage.client.get_paginator("list_objects_v2")
|
||||||
@@ -174,8 +175,13 @@ class MetricsManager:
|
|||||||
metrics["requests"]["cache_hit_ratio"] = cache_hits / total_requests if total_requests > 0 else 0.0
|
metrics["requests"]["cache_hit_ratio"] = cache_hits / total_requests if total_requests > 0 else 0.0
|
||||||
metrics["bandwidth"]["saved_bytes"] = bandwidth_saved
|
metrics["bandwidth"]["saved_bytes"] = bandwidth_saved
|
||||||
|
|
||||||
# Get per-remote metrics
|
# Get per-repo metrics
|
||||||
for remote in config_manager.config.get("remotes", {}).keys():
|
all_repos = {
|
||||||
|
**config_manager.config.get("remotes", {}),
|
||||||
|
**config_manager.config.get("virtuals", {}),
|
||||||
|
**config_manager.config.get("locals", {}),
|
||||||
|
}
|
||||||
|
for remote in all_repos.keys():
|
||||||
remote_cache_hits = int(self.redis_client.client.get(f"metrics:cache_hits:{remote}") or 0)
|
remote_cache_hits = int(self.redis_client.client.get(f"metrics:cache_hits:{remote}") or 0)
|
||||||
remote_cache_misses = int(self.redis_client.client.get(f"metrics:cache_misses:{remote}") or 0)
|
remote_cache_misses = int(self.redis_client.client.get(f"metrics:cache_misses:{remote}") or 0)
|
||||||
remote_total = remote_cache_hits + remote_cache_misses
|
remote_total = remote_cache_hits + remote_cache_misses
|
||||||
|
|||||||
+43
-16
@@ -20,61 +20,48 @@ TEST_REMOTES = {
|
|||||||
"remotes": {
|
"remotes": {
|
||||||
"alpine-test": {
|
"alpine-test": {
|
||||||
"base_url": "https://dl-cdn.alpinelinux.org",
|
"base_url": "https://dl-cdn.alpinelinux.org",
|
||||||
"type": "remote",
|
|
||||||
"package": "alpine",
|
"package": "alpine",
|
||||||
"immutable_patterns": [".*/x86_64/.*\\.apk$"],
|
"immutable_patterns": [".*/x86_64/.*\\.apk$"],
|
||||||
"cache": {"immutable_ttl": 0, "mutable_ttl": 3600},
|
"cache": {"immutable_ttl": 0, "mutable_ttl": 3600},
|
||||||
},
|
},
|
||||||
"rpm-test": {
|
"rpm-test": {
|
||||||
"base_url": "https://example.com/rpm",
|
"base_url": "https://example.com/rpm",
|
||||||
"type": "remote",
|
|
||||||
"package": "rpm",
|
"package": "rpm",
|
||||||
"immutable_patterns": [".*/x86_64/.*\\.rpm$", ".*/repodata/.*$"],
|
"immutable_patterns": [".*/x86_64/.*\\.rpm$", ".*/repodata/.*$"],
|
||||||
"cache": {"immutable_ttl": 0, "mutable_ttl": 3600},
|
"cache": {"immutable_ttl": 0, "mutable_ttl": 3600},
|
||||||
},
|
},
|
||||||
"docker-test": {
|
"docker-test": {
|
||||||
"base_url": "https://registry.example.com",
|
"base_url": "https://registry.example.com",
|
||||||
"type": "remote",
|
|
||||||
"package": "docker",
|
"package": "docker",
|
||||||
"cache": {"immutable_ttl": 0, "mutable_ttl": 300},
|
"cache": {"immutable_ttl": 0, "mutable_ttl": 300},
|
||||||
},
|
},
|
||||||
"docker-restricted": {
|
"docker-restricted": {
|
||||||
"base_url": "https://registry.example.com",
|
"base_url": "https://registry.example.com",
|
||||||
"type": "remote",
|
|
||||||
"package": "docker",
|
"package": "docker",
|
||||||
"immutable_patterns": ["^library/nginx"],
|
"immutable_patterns": ["^library/nginx"],
|
||||||
"cache": {"immutable_ttl": 0, "mutable_ttl": 300},
|
"cache": {"immutable_ttl": 0, "mutable_ttl": 300},
|
||||||
},
|
},
|
||||||
"generic-test": {
|
"generic-test": {
|
||||||
"base_url": "https://releases.example.com",
|
"base_url": "https://releases.example.com",
|
||||||
"type": "remote",
|
|
||||||
"package": "generic",
|
"package": "generic",
|
||||||
"immutable_patterns": [".*\\.tar\\.gz$"],
|
"immutable_patterns": [".*\\.tar\\.gz$"],
|
||||||
"cache": {"immutable_ttl": 0, "mutable_ttl": 0},
|
"cache": {"immutable_ttl": 0, "mutable_ttl": 0},
|
||||||
},
|
},
|
||||||
"custom-index-test": {
|
"custom-index-test": {
|
||||||
"base_url": "https://example.com",
|
"base_url": "https://example.com",
|
||||||
"type": "remote",
|
|
||||||
"package": "generic",
|
"package": "generic",
|
||||||
"mutable_patterns": ["metadata\\.json$"],
|
"mutable_patterns": ["metadata\\.json$"],
|
||||||
"cache": {"immutable_ttl": 0, "mutable_ttl": 600},
|
"cache": {"immutable_ttl": 0, "mutable_ttl": 600},
|
||||||
},
|
},
|
||||||
"check-mutable-test": {
|
"check-mutable-test": {
|
||||||
"base_url": "https://example.com",
|
"base_url": "https://example.com",
|
||||||
"type": "remote",
|
|
||||||
"package": "generic",
|
"package": "generic",
|
||||||
"mutable_patterns": ["metadata\\.json$"],
|
"mutable_patterns": ["metadata\\.json$"],
|
||||||
"check_mutable_updates": True,
|
"check_mutable_updates": True,
|
||||||
"cache": {"immutable_ttl": 0, "mutable_ttl": 600},
|
"cache": {"immutable_ttl": 0, "mutable_ttl": 600},
|
||||||
},
|
},
|
||||||
"local-test": {
|
|
||||||
"type": "local",
|
|
||||||
"package": "generic",
|
|
||||||
"cache": {"immutable_ttl": 0, "mutable_ttl": 0},
|
|
||||||
},
|
|
||||||
"pypi-test": {
|
"pypi-test": {
|
||||||
"base_url": "https://files.pythonhosted.org",
|
"base_url": "https://files.pythonhosted.org",
|
||||||
"type": "remote",
|
|
||||||
"package": "pypi",
|
"package": "pypi",
|
||||||
"immutable_patterns": [
|
"immutable_patterns": [
|
||||||
r"packages/.*\.whl$",
|
r"packages/.*\.whl$",
|
||||||
@@ -85,7 +72,6 @@ TEST_REMOTES = {
|
|||||||
},
|
},
|
||||||
"npm-test": {
|
"npm-test": {
|
||||||
"base_url": "https://registry.npmjs.org",
|
"base_url": "https://registry.npmjs.org",
|
||||||
"type": "remote",
|
|
||||||
"package": "npm",
|
"package": "npm",
|
||||||
"immutable_patterns": [r"\.tgz$"],
|
"immutable_patterns": [r"\.tgz$"],
|
||||||
"mutable_patterns": [r"^(?!.*\.tgz$).*"],
|
"mutable_patterns": [r"^(?!.*\.tgz$).*"],
|
||||||
@@ -93,12 +79,53 @@ TEST_REMOTES = {
|
|||||||
},
|
},
|
||||||
"helm-test": {
|
"helm-test": {
|
||||||
"base_url": "https://helm.releases.hashicorp.com",
|
"base_url": "https://helm.releases.hashicorp.com",
|
||||||
"type": "remote",
|
|
||||||
"package": "helm",
|
"package": "helm",
|
||||||
"immutable_patterns": [r"\.tgz$"],
|
"immutable_patterns": [r"\.tgz$"],
|
||||||
"cache": {"immutable_ttl": 0, "mutable_ttl": 3600},
|
"cache": {"immutable_ttl": 0, "mutable_ttl": 3600},
|
||||||
},
|
},
|
||||||
}
|
"quarantine-test": {
|
||||||
|
"base_url": "https://releases.example.com",
|
||||||
|
"package": "generic",
|
||||||
|
"immutable_patterns": [r".*\.tar\.gz$"],
|
||||||
|
"quarantine_new": True,
|
||||||
|
"quarantine_days": 3,
|
||||||
|
"cache": {"immutable_ttl": 0, "mutable_ttl": 0},
|
||||||
|
},
|
||||||
|
"quarantine-disabled": {
|
||||||
|
"base_url": "https://releases.example.com",
|
||||||
|
"package": "generic",
|
||||||
|
"immutable_patterns": [r".*\.tar\.gz$"],
|
||||||
|
"quarantine_new": False,
|
||||||
|
"quarantine_days": 3,
|
||||||
|
"cache": {"immutable_ttl": 0, "mutable_ttl": 0},
|
||||||
|
},
|
||||||
|
"helm-member-2": {
|
||||||
|
"base_url": "https://charts.example.com",
|
||||||
|
"package": "helm",
|
||||||
|
"immutable_patterns": [r"\.tgz$"],
|
||||||
|
"cache": {"immutable_ttl": 0, "mutable_ttl": 1800},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"locals": {
|
||||||
|
"local-test": {
|
||||||
|
"package": "generic",
|
||||||
|
"cache": {"immutable_ttl": 0, "mutable_ttl": 0},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"virtuals": {
|
||||||
|
"helm-virtual-test": {
|
||||||
|
"package": "helm",
|
||||||
|
"members": ["helm-test", "helm-member-2"],
|
||||||
|
},
|
||||||
|
"unsupported-virtual-test": {
|
||||||
|
"package": "rpm",
|
||||||
|
"members": ["rpm-test"],
|
||||||
|
},
|
||||||
|
"empty-virtual-test": {
|
||||||
|
"package": "helm",
|
||||||
|
"members": [],
|
||||||
|
},
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|||||||
@@ -283,3 +283,47 @@ class TestMutableMeta:
|
|||||||
|
|
||||||
def test_delete_no_op_when_unavailable(self, unavailable_cache):
|
def test_delete_no_op_when_unavailable(self, unavailable_cache):
|
||||||
unavailable_cache.delete_mutable_meta("remote", "path") # must not raise
|
unavailable_cache.delete_mutable_meta("remote", "path") # must not raise
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# artifact published date (quarantine support)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestArtifactPublished:
|
||||||
|
def test_key_format_is_deterministic(self, bare_cache):
|
||||||
|
path = "some/path/package-1.0.tar.gz"
|
||||||
|
expected_hash = hashlib.sha256(path.encode()).hexdigest()[:16]
|
||||||
|
assert bare_cache.get_artifact_published_key("myremote", path) == f"pkg:published:myremote:{expected_hash}"
|
||||||
|
|
||||||
|
def test_key_hash_is_16_chars(self, bare_cache):
|
||||||
|
key = bare_cache.get_artifact_published_key("remote", "path/to/file.whl")
|
||||||
|
assert len(key.split(":")[-1]) == 16
|
||||||
|
|
||||||
|
def test_different_paths_produce_different_keys(self, bare_cache):
|
||||||
|
k1 = bare_cache.get_artifact_published_key("remote", "pkg-1.0.tar.gz")
|
||||||
|
k2 = bare_cache.get_artifact_published_key("remote", "pkg-2.0.tar.gz")
|
||||||
|
assert k1 != k2
|
||||||
|
|
||||||
|
def test_store_calls_set_with_correct_value(self, cache_with_redis, mock_redis_client):
|
||||||
|
lm = "Mon, 01 Jan 2024 00:00:00 GMT"
|
||||||
|
cache_with_redis.store_artifact_published("remote", "path/pkg.tar.gz", lm)
|
||||||
|
expected_key = cache_with_redis.get_artifact_published_key("remote", "path/pkg.tar.gz")
|
||||||
|
mock_redis_client.set.assert_called_once_with(expected_key, lm)
|
||||||
|
|
||||||
|
def test_get_returns_stored_value(self, cache_with_redis, mock_redis_client):
|
||||||
|
lm = "Tue, 15 Mar 2022 12:00:00 GMT"
|
||||||
|
mock_redis_client.get.return_value = lm
|
||||||
|
result = cache_with_redis.get_artifact_published("remote", "path/pkg.tar.gz")
|
||||||
|
assert result == lm
|
||||||
|
|
||||||
|
def test_get_returns_none_when_not_stored(self, cache_with_redis, mock_redis_client):
|
||||||
|
mock_redis_client.get.return_value = None
|
||||||
|
result = cache_with_redis.get_artifact_published("remote", "path/pkg.tar.gz")
|
||||||
|
assert result is None
|
||||||
|
|
||||||
|
def test_store_no_op_when_unavailable(self, unavailable_cache):
|
||||||
|
unavailable_cache.store_artifact_published("remote", "path", "Mon, 01 Jan 2024 00:00:00 GMT")
|
||||||
|
|
||||||
|
def test_get_returns_none_when_unavailable(self, unavailable_cache):
|
||||||
|
assert unavailable_cache.get_artifact_published("remote", "path") is None
|
||||||
|
|||||||
+203
-16
@@ -27,24 +27,24 @@ def make_config(tmp_path):
|
|||||||
|
|
||||||
class TestGetMutablePatterns:
|
class TestGetMutablePatterns:
|
||||||
def test_alpine_returns_package_defaults(self, make_config):
|
def test_alpine_returns_package_defaults(self, make_config):
|
||||||
cfg = make_config({"r": {"type": "remote", "package": "alpine", "base_url": "https://x.com"}})
|
cfg = make_config({"r": {"package": "alpine", "base_url": "https://x.com"}})
|
||||||
patterns = cfg.get_mutable_patterns("r")
|
patterns = cfg.get_mutable_patterns("r")
|
||||||
assert r"APKINDEX\.tar\.gz$" in patterns
|
assert r"APKINDEX\.tar\.gz$" in patterns
|
||||||
|
|
||||||
def test_rpm_returns_package_defaults(self, make_config):
|
def test_rpm_returns_package_defaults(self, make_config):
|
||||||
cfg = make_config({"r": {"type": "remote", "package": "rpm", "base_url": "https://x.com"}})
|
cfg = make_config({"r": {"package": "rpm", "base_url": "https://x.com"}})
|
||||||
patterns = cfg.get_mutable_patterns("r")
|
patterns = cfg.get_mutable_patterns("r")
|
||||||
assert r"repomd\.xml$" in patterns
|
assert r"repomd\.xml$" in patterns
|
||||||
assert any("repodata" in p for p in patterns)
|
assert any("repodata" in p for p in patterns)
|
||||||
|
|
||||||
def test_docker_returns_package_defaults(self, make_config):
|
def test_docker_returns_package_defaults(self, make_config):
|
||||||
cfg = make_config({"r": {"type": "remote", "package": "docker", "base_url": "https://x.com"}})
|
cfg = make_config({"r": {"package": "docker", "base_url": "https://x.com"}})
|
||||||
patterns = cfg.get_mutable_patterns("r")
|
patterns = cfg.get_mutable_patterns("r")
|
||||||
assert any("manifests" in p for p in patterns)
|
assert any("manifests" in p for p in patterns)
|
||||||
assert any("tags/list" in p for p in patterns)
|
assert any("tags/list" in p for p in patterns)
|
||||||
|
|
||||||
def test_generic_returns_empty_list(self, make_config):
|
def test_generic_returns_empty_list(self, make_config):
|
||||||
cfg = make_config({"r": {"type": "remote", "package": "generic", "base_url": "https://x.com"}})
|
cfg = make_config({"r": {"package": "generic", "base_url": "https://x.com"}})
|
||||||
assert cfg.get_mutable_patterns("r") == []
|
assert cfg.get_mutable_patterns("r") == []
|
||||||
|
|
||||||
def test_unknown_remote_returns_empty_list(self, make_config):
|
def test_unknown_remote_returns_empty_list(self, make_config):
|
||||||
@@ -52,12 +52,12 @@ class TestGetMutablePatterns:
|
|||||||
assert cfg.get_mutable_patterns("nonexistent") == []
|
assert cfg.get_mutable_patterns("nonexistent") == []
|
||||||
|
|
||||||
def test_missing_package_field_defaults_to_generic(self, make_config):
|
def test_missing_package_field_defaults_to_generic(self, make_config):
|
||||||
cfg = make_config({"r": {"type": "remote", "base_url": "https://x.com"}})
|
cfg = make_config({"r": {"base_url": "https://x.com"}})
|
||||||
assert cfg.get_mutable_patterns("r") == []
|
assert cfg.get_mutable_patterns("r") == []
|
||||||
|
|
||||||
def test_unknown_package_type_returns_empty_list(self, make_config):
|
def test_unknown_package_type_returns_empty_list(self, make_config):
|
||||||
# A mis-spelled package type silently returns [] — this is a known footgun
|
# A mis-spelled package type silently returns [] — this is a known footgun
|
||||||
cfg = make_config({"r": {"type": "remote", "package": "deb", "base_url": "https://x.com"}})
|
cfg = make_config({"r": {"package": "deb", "base_url": "https://x.com"}})
|
||||||
assert cfg.get_mutable_patterns("r") == []
|
assert cfg.get_mutable_patterns("r") == []
|
||||||
|
|
||||||
def test_extra_patterns_appended_after_defaults(self, make_config):
|
def test_extra_patterns_appended_after_defaults(self, make_config):
|
||||||
@@ -134,7 +134,7 @@ class TestGetMutablePatterns:
|
|||||||
assert r"custom-meta\.xml$" in patterns
|
assert r"custom-meta\.xml$" in patterns
|
||||||
|
|
||||||
def test_npm_has_no_package_defaults(self, make_config):
|
def test_npm_has_no_package_defaults(self, make_config):
|
||||||
cfg = make_config({"r": {"type": "remote", "package": "npm", "base_url": "https://x.com"}})
|
cfg = make_config({"r": {"package": "npm", "base_url": "https://x.com"}})
|
||||||
assert cfg.get_mutable_patterns("r") == []
|
assert cfg.get_mutable_patterns("r") == []
|
||||||
|
|
||||||
def test_npm_explicit_mutable_pattern_matches_metadata(self, make_config):
|
def test_npm_explicit_mutable_pattern_matches_metadata(self, make_config):
|
||||||
@@ -155,14 +155,14 @@ class TestGetMutablePatterns:
|
|||||||
assert any(re.search(p, "@babel/core") for p in patterns)
|
assert any(re.search(p, "@babel/core") for p in patterns)
|
||||||
|
|
||||||
def test_helm_returns_index_yaml_as_mutable(self, make_config):
|
def test_helm_returns_index_yaml_as_mutable(self, make_config):
|
||||||
cfg = make_config({"r": {"type": "remote", "package": "helm", "base_url": "https://helm.example.com"}})
|
cfg = make_config({"r": {"package": "helm", "base_url": "https://helm.example.com"}})
|
||||||
patterns = cfg.get_mutable_patterns("r")
|
patterns = cfg.get_mutable_patterns("r")
|
||||||
assert r"index\.yaml$" in patterns
|
assert r"index\.yaml$" in patterns
|
||||||
|
|
||||||
def test_helm_chart_tarballs_not_mutable_by_default(self, make_config):
|
def test_helm_chart_tarballs_not_mutable_by_default(self, make_config):
|
||||||
import re
|
import re
|
||||||
|
|
||||||
cfg = make_config({"r": {"type": "remote", "package": "helm", "base_url": "https://helm.example.com"}})
|
cfg = make_config({"r": {"package": "helm", "base_url": "https://helm.example.com"}})
|
||||||
patterns = cfg.get_mutable_patterns("r")
|
patterns = cfg.get_mutable_patterns("r")
|
||||||
# Only index.yaml is mutable; .tgz chart tarballs are not
|
# Only index.yaml is mutable; .tgz chart tarballs are not
|
||||||
assert not any(re.search(p, "vault-0.29.1.tgz") for p in patterns)
|
assert not any(re.search(p, "vault-0.29.1.tgz") for p in patterns)
|
||||||
@@ -210,7 +210,7 @@ class TestGetImmutablePatterns:
|
|||||||
assert cfg.get_immutable_patterns("nonexistent") == []
|
assert cfg.get_immutable_patterns("nonexistent") == []
|
||||||
|
|
||||||
def test_returns_empty_when_no_patterns_configured(self, make_config):
|
def test_returns_empty_when_no_patterns_configured(self, make_config):
|
||||||
cfg = make_config({"r": {"type": "remote", "package": "generic", "base_url": "https://x.com"}})
|
cfg = make_config({"r": {"package": "generic", "base_url": "https://x.com"}})
|
||||||
assert cfg.get_immutable_patterns("r") == []
|
assert cfg.get_immutable_patterns("r") == []
|
||||||
|
|
||||||
def test_multiple_patterns_returned(self, make_config):
|
def test_multiple_patterns_returned(self, make_config):
|
||||||
@@ -281,7 +281,7 @@ class TestGetUserMutablePatterns:
|
|||||||
|
|
||||||
def test_excludes_package_defaults(self, make_config):
|
def test_excludes_package_defaults(self, make_config):
|
||||||
# Package defaults (APKINDEX etc.) must NOT appear here
|
# Package defaults (APKINDEX etc.) must NOT appear here
|
||||||
cfg = make_config({"r": {"type": "remote", "package": "alpine", "base_url": "https://x.com"}})
|
cfg = make_config({"r": {"package": "alpine", "base_url": "https://x.com"}})
|
||||||
assert cfg.get_user_mutable_patterns("r") == []
|
assert cfg.get_user_mutable_patterns("r") == []
|
||||||
|
|
||||||
def test_returns_empty_for_missing_remote(self, make_config):
|
def test_returns_empty_for_missing_remote(self, make_config):
|
||||||
@@ -289,7 +289,7 @@ class TestGetUserMutablePatterns:
|
|||||||
assert cfg.get_user_mutable_patterns("nonexistent") == []
|
assert cfg.get_user_mutable_patterns("nonexistent") == []
|
||||||
|
|
||||||
def test_returns_empty_when_key_absent(self, make_config):
|
def test_returns_empty_when_key_absent(self, make_config):
|
||||||
cfg = make_config({"r": {"type": "remote", "package": "generic", "base_url": "https://x.com"}})
|
cfg = make_config({"r": {"package": "generic", "base_url": "https://x.com"}})
|
||||||
assert cfg.get_user_mutable_patterns("r") == []
|
assert cfg.get_user_mutable_patterns("r") == []
|
||||||
|
|
||||||
|
|
||||||
@@ -317,7 +317,7 @@ class TestGetCacheConfig:
|
|||||||
assert cfg.get_cache_config("nonexistent") == {}
|
assert cfg.get_cache_config("nonexistent") == {}
|
||||||
|
|
||||||
def test_returns_empty_dict_when_no_cache_key(self, make_config):
|
def test_returns_empty_dict_when_no_cache_key(self, make_config):
|
||||||
cfg = make_config({"r": {"type": "remote", "package": "generic", "base_url": "https://x.com"}})
|
cfg = make_config({"r": {"package": "generic", "base_url": "https://x.com"}})
|
||||||
assert cfg.get_cache_config("r") == {}
|
assert cfg.get_cache_config("r") == {}
|
||||||
|
|
||||||
|
|
||||||
@@ -329,11 +329,11 @@ class TestGetCacheConfig:
|
|||||||
class TestConfigReload:
|
class TestConfigReload:
|
||||||
def test_reloads_when_file_mtime_advances(self, tmp_path):
|
def test_reloads_when_file_mtime_advances(self, tmp_path):
|
||||||
cfg_file = tmp_path / "remotes.yaml"
|
cfg_file = tmp_path / "remotes.yaml"
|
||||||
cfg_file.write_text(yaml.dump({"remotes": {"repo-a": {"type": "remote", "package": "generic", "base_url": "https://x.com"}}}))
|
cfg_file.write_text(yaml.dump({"remotes": {"repo-a": {"package": "generic", "base_url": "https://x.com"}}}))
|
||||||
cfg = ConfigManager(str(cfg_file))
|
cfg = ConfigManager(str(cfg_file))
|
||||||
assert "repo-a" in cfg.config["remotes"]
|
assert "repo-a" in cfg.config["remotes"]
|
||||||
|
|
||||||
cfg_file.write_text(yaml.dump({"remotes": {"repo-b": {"type": "remote", "package": "generic", "base_url": "https://y.com"}}}))
|
cfg_file.write_text(yaml.dump({"remotes": {"repo-b": {"package": "generic", "base_url": "https://y.com"}}}))
|
||||||
future_mtime = cfg._last_modified + 1
|
future_mtime = cfg._last_modified + 1
|
||||||
os.utime(str(cfg_file), (future_mtime, future_mtime))
|
os.utime(str(cfg_file), (future_mtime, future_mtime))
|
||||||
|
|
||||||
@@ -344,10 +344,197 @@ class TestConfigReload:
|
|||||||
|
|
||||||
def test_no_reload_when_file_unchanged(self, tmp_path):
|
def test_no_reload_when_file_unchanged(self, tmp_path):
|
||||||
cfg_file = tmp_path / "remotes.yaml"
|
cfg_file = tmp_path / "remotes.yaml"
|
||||||
cfg_file.write_text(yaml.dump({"remotes": {"repo-a": {"type": "remote", "package": "generic", "base_url": "https://x.com"}}}))
|
cfg_file.write_text(yaml.dump({"remotes": {"repo-a": {"package": "generic", "base_url": "https://x.com"}}}))
|
||||||
cfg = ConfigManager(str(cfg_file))
|
cfg = ConfigManager(str(cfg_file))
|
||||||
|
|
||||||
# Call check_reload without touching the file — should not reload
|
# Call check_reload without touching the file — should not reload
|
||||||
cfg._check_reload()
|
cfg._check_reload()
|
||||||
|
|
||||||
assert "repo-a" in cfg.config["remotes"]
|
assert "repo-a" in cfg.config["remotes"]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# get_quarantine_config
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestGetQuarantineConfig:
|
||||||
|
def test_returns_false_zero_when_not_configured(self, make_config):
|
||||||
|
cfg = make_config({"r": {"package": "generic", "base_url": "https://x.com"}})
|
||||||
|
enabled, days = cfg.get_quarantine_config("r")
|
||||||
|
assert enabled is False
|
||||||
|
assert days == 0
|
||||||
|
|
||||||
|
def test_returns_false_zero_for_missing_remote(self, make_config):
|
||||||
|
cfg = make_config({})
|
||||||
|
enabled, days = cfg.get_quarantine_config("nonexistent")
|
||||||
|
assert enabled is False
|
||||||
|
assert days == 0
|
||||||
|
|
||||||
|
def test_enabled_true_and_days_returned(self, make_config):
|
||||||
|
cfg = make_config(
|
||||||
|
{
|
||||||
|
"r": {
|
||||||
|
"type": "remote",
|
||||||
|
"package": "generic",
|
||||||
|
"base_url": "https://x.com",
|
||||||
|
"quarantine_new": True,
|
||||||
|
"quarantine_days": 7,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
enabled, days = cfg.get_quarantine_config("r")
|
||||||
|
assert enabled is True
|
||||||
|
assert days == 7
|
||||||
|
|
||||||
|
def test_quarantine_new_false_returns_disabled(self, make_config):
|
||||||
|
cfg = make_config(
|
||||||
|
{
|
||||||
|
"r": {
|
||||||
|
"type": "remote",
|
||||||
|
"package": "generic",
|
||||||
|
"base_url": "https://x.com",
|
||||||
|
"quarantine_new": False,
|
||||||
|
"quarantine_days": 7,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
enabled, days = cfg.get_quarantine_config("r")
|
||||||
|
assert enabled is False
|
||||||
|
assert days == 7
|
||||||
|
|
||||||
|
def test_enabled_with_zero_days_returns_zero(self, make_config):
|
||||||
|
cfg = make_config(
|
||||||
|
{
|
||||||
|
"r": {
|
||||||
|
"type": "remote",
|
||||||
|
"package": "generic",
|
||||||
|
"base_url": "https://x.com",
|
||||||
|
"quarantine_new": True,
|
||||||
|
"quarantine_days": 0,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
enabled, days = cfg.get_quarantine_config("r")
|
||||||
|
assert enabled is True
|
||||||
|
assert days == 0
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Directory mode (CONFIG_PATH points to a directory)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _remote(base_url: str = "https://x.com") -> dict:
|
||||||
|
return {"package": "generic", "base_url": base_url}
|
||||||
|
|
||||||
|
|
||||||
|
class TestConfigDirMode:
|
||||||
|
def test_loads_all_yaml_files(self, tmp_path):
|
||||||
|
(tmp_path / "a.yaml").write_text(yaml.dump({"remotes": {"repo-a": _remote()}}))
|
||||||
|
(tmp_path / "b.yaml").write_text(yaml.dump({"remotes": {"repo-b": _remote("https://y.com")}}))
|
||||||
|
cfg = ConfigManager(str(tmp_path))
|
||||||
|
assert "repo-a" in cfg.config["remotes"]
|
||||||
|
assert "repo-b" in cfg.config["remotes"]
|
||||||
|
|
||||||
|
def test_later_file_overrides_earlier_on_same_key(self, tmp_path):
|
||||||
|
(tmp_path / "a.yaml").write_text(yaml.dump({"remotes": {"r": _remote("https://first.com")}}))
|
||||||
|
(tmp_path / "b.yaml").write_text(yaml.dump({"remotes": {"r": _remote("https://second.com")}}))
|
||||||
|
cfg = ConfigManager(str(tmp_path))
|
||||||
|
assert cfg.config["remotes"]["r"]["base_url"] == "https://second.com"
|
||||||
|
|
||||||
|
def test_empty_directory_returns_empty_remotes(self, tmp_path):
|
||||||
|
cfg = ConfigManager(str(tmp_path))
|
||||||
|
assert cfg.config == {"remotes": {}, "virtuals": {}, "locals": {}}
|
||||||
|
|
||||||
|
def test_ignores_non_yaml_files(self, tmp_path):
|
||||||
|
(tmp_path / "notes.txt").write_text("not yaml")
|
||||||
|
(tmp_path / "a.yaml").write_text(yaml.dump({"remotes": {"repo-a": _remote()}}))
|
||||||
|
cfg = ConfigManager(str(tmp_path))
|
||||||
|
assert list(cfg.config["remotes"].keys()) == ["repo-a"]
|
||||||
|
|
||||||
|
def test_reload_picks_up_new_file(self, tmp_path):
|
||||||
|
(tmp_path / "a.yaml").write_text(yaml.dump({"remotes": {"repo-a": _remote()}}))
|
||||||
|
cfg = ConfigManager(str(tmp_path))
|
||||||
|
assert "repo-a" in cfg.config["remotes"]
|
||||||
|
assert "repo-b" not in cfg.config["remotes"]
|
||||||
|
|
||||||
|
new_file = tmp_path / "b.yaml"
|
||||||
|
new_file.write_text(yaml.dump({"remotes": {"repo-b": _remote("https://y.com")}}))
|
||||||
|
future_mtime = cfg._last_modified + 1
|
||||||
|
os.utime(str(new_file), (future_mtime, future_mtime))
|
||||||
|
|
||||||
|
cfg._check_reload()
|
||||||
|
|
||||||
|
assert "repo-a" in cfg.config["remotes"]
|
||||||
|
assert "repo-b" in cfg.config["remotes"]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# config_dir key (main file contains a config_dir pointer)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestConfigDirKey:
|
||||||
|
def test_merges_remotes_from_config_dir(self, tmp_path):
|
||||||
|
conf_d = tmp_path / "conf.d"
|
||||||
|
conf_d.mkdir()
|
||||||
|
(conf_d / "remotes.yaml").write_text(yaml.dump({"remotes": {"repo-extra": _remote("https://extra.com")}}))
|
||||||
|
main = tmp_path / "config.yaml"
|
||||||
|
main.write_text(yaml.dump({"config_dir": str(conf_d), "remotes": {"repo-main": _remote()}}))
|
||||||
|
cfg = ConfigManager(str(main))
|
||||||
|
assert "repo-main" in cfg.config["remotes"]
|
||||||
|
assert "repo-extra" in cfg.config["remotes"]
|
||||||
|
|
||||||
|
def test_relative_config_dir_resolved_from_main_file(self, tmp_path):
|
||||||
|
conf_d = tmp_path / "conf.d"
|
||||||
|
conf_d.mkdir()
|
||||||
|
(conf_d / "r.yaml").write_text(yaml.dump({"remotes": {"repo-a": _remote()}}))
|
||||||
|
main = tmp_path / "config.yaml"
|
||||||
|
main.write_text(yaml.dump({"config_dir": "conf.d", "remotes": {}}))
|
||||||
|
cfg = ConfigManager(str(main))
|
||||||
|
assert "repo-a" in cfg.config["remotes"]
|
||||||
|
|
||||||
|
def test_config_dir_key_not_present_in_loaded_config(self, tmp_path):
|
||||||
|
conf_d = tmp_path / "conf.d"
|
||||||
|
conf_d.mkdir()
|
||||||
|
main = tmp_path / "config.yaml"
|
||||||
|
main.write_text(yaml.dump({"config_dir": str(conf_d), "remotes": {}}))
|
||||||
|
cfg = ConfigManager(str(main))
|
||||||
|
assert "config_dir" not in cfg.config
|
||||||
|
|
||||||
|
def test_dir_remote_overrides_main_file_remote(self, tmp_path):
|
||||||
|
conf_d = tmp_path / "conf.d"
|
||||||
|
conf_d.mkdir()
|
||||||
|
(conf_d / "override.yaml").write_text(yaml.dump({"remotes": {"r": _remote("https://new.com")}}))
|
||||||
|
main = tmp_path / "config.yaml"
|
||||||
|
main.write_text(yaml.dump({"config_dir": str(conf_d), "remotes": {"r": _remote("https://old.com")}}))
|
||||||
|
cfg = ConfigManager(str(main))
|
||||||
|
assert cfg.config["remotes"]["r"]["base_url"] == "https://new.com"
|
||||||
|
|
||||||
|
def test_empty_config_dir_uses_main_file_only(self, tmp_path):
|
||||||
|
conf_d = tmp_path / "conf.d"
|
||||||
|
conf_d.mkdir()
|
||||||
|
main = tmp_path / "config.yaml"
|
||||||
|
main.write_text(yaml.dump({"config_dir": str(conf_d), "remotes": {"repo-main": _remote()}}))
|
||||||
|
cfg = ConfigManager(str(main))
|
||||||
|
assert list(cfg.config["remotes"].keys()) == ["repo-main"]
|
||||||
|
|
||||||
|
def test_reload_picks_up_changed_dir_file(self, tmp_path):
|
||||||
|
conf_d = tmp_path / "conf.d"
|
||||||
|
conf_d.mkdir()
|
||||||
|
dir_file = conf_d / "r.yaml"
|
||||||
|
dir_file.write_text(yaml.dump({"remotes": {"repo-v1": _remote()}}))
|
||||||
|
main = tmp_path / "config.yaml"
|
||||||
|
main.write_text(yaml.dump({"config_dir": str(conf_d), "remotes": {}}))
|
||||||
|
cfg = ConfigManager(str(main))
|
||||||
|
assert "repo-v1" in cfg.config["remotes"]
|
||||||
|
|
||||||
|
dir_file.write_text(yaml.dump({"remotes": {"repo-v2": _remote("https://v2.com")}}))
|
||||||
|
future_mtime = cfg._last_modified + 1
|
||||||
|
os.utime(str(dir_file), (future_mtime, future_mtime))
|
||||||
|
|
||||||
|
cfg._check_reload()
|
||||||
|
|
||||||
|
assert "repo-v2" in cfg.config["remotes"]
|
||||||
|
assert "repo-v1" not in cfg.config["remotes"]
|
||||||
|
|||||||
+162
-26
@@ -2,6 +2,7 @@
|
|||||||
|
|
||||||
import hashlib
|
import hashlib
|
||||||
import json
|
import json
|
||||||
|
from datetime import UTC
|
||||||
from unittest.mock import ANY, AsyncMock, MagicMock, patch
|
from unittest.mock import ANY, AsyncMock, MagicMock, patch
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
@@ -522,68 +523,53 @@ class TestGenericArtifactRoute:
|
|||||||
deps["database"].get_local_file_metadata.return_value = None
|
deps["database"].get_local_file_metadata.return_value = None
|
||||||
deps["database"].available = True
|
deps["database"].available = True
|
||||||
|
|
||||||
response = client.get("/api/v1/remote/local-test/path/to/nonexistent.bin")
|
response = client.get("/api/v1/local/local-test/path/to/nonexistent.bin")
|
||||||
assert response.status_code == 404
|
assert response.status_code == 404
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Upload route PUT /api/v1/remote/{remote}/{path}
|
# Upload route PUT /api/v1/local/{local}/{path}
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
class TestUploadRoute:
|
class TestUploadRoute:
|
||||||
def test_unknown_remote_returns_404(self, client, patched_deps):
|
def test_unknown_local_returns_404(self, client, patched_deps):
|
||||||
response = client.put(
|
response = client.put(
|
||||||
"/api/v1/remote/nonexistent/path/to/file.tar.gz",
|
"/api/v1/local/nonexistent/path/to/file.tar.gz",
|
||||||
files={"file": ("file.tar.gz", b"content", "application/octet-stream")},
|
files={"file": ("file.tar.gz", b"content", "application/octet-stream")},
|
||||||
)
|
)
|
||||||
assert response.status_code == 404
|
assert response.status_code == 404
|
||||||
|
|
||||||
def test_non_local_remote_returns_400(self, client, patched_deps):
|
|
||||||
response = client.put(
|
|
||||||
"/api/v1/remote/generic-test/path/to/file.tar.gz",
|
|
||||||
files={"file": ("file.tar.gz", b"content", "application/octet-stream")},
|
|
||||||
)
|
|
||||||
assert response.status_code == 400
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# HEAD route HEAD /api/v1/remote/{remote}/{path}
|
# HEAD route HEAD /api/v1/local/{local}/{path}
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
class TestHeadRoute:
|
class TestHeadRoute:
|
||||||
def test_non_local_remote_returns_405(self, client, patched_deps):
|
|
||||||
response = client.head("/api/v1/remote/generic-test/path/to/file.tar.gz")
|
|
||||||
assert response.status_code == 405
|
|
||||||
|
|
||||||
def test_local_repo_file_not_found_returns_404(self, client, patched_deps):
|
def test_local_repo_file_not_found_returns_404(self, client, patched_deps):
|
||||||
deps = patched_deps
|
deps = patched_deps
|
||||||
deps["database"].get_local_file_metadata.return_value = None
|
deps["database"].get_local_file_metadata.return_value = None
|
||||||
deps["database"].available = True
|
deps["database"].available = True
|
||||||
|
|
||||||
response = client.head("/api/v1/remote/local-test/path/to/nonexistent.bin")
|
response = client.head("/api/v1/local/local-test/path/to/nonexistent.bin")
|
||||||
assert response.status_code == 404
|
assert response.status_code == 404
|
||||||
|
|
||||||
def test_unknown_remote_returns_404(self, client, patched_deps):
|
def test_unknown_local_returns_404(self, client, patched_deps):
|
||||||
response = client.head("/api/v1/remote/nonexistent/path/to/file.bin")
|
response = client.head("/api/v1/local/nonexistent/path/to/file.bin")
|
||||||
assert response.status_code == 404
|
assert response.status_code == 404
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# DELETE route DELETE /api/v1/remote/{remote}/{path}
|
# DELETE route DELETE /api/v1/local/{local}/{path}
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
class TestDeleteRoute:
|
class TestDeleteRoute:
|
||||||
def test_unknown_remote_returns_404(self, client, patched_deps):
|
def test_unknown_local_returns_404(self, client, patched_deps):
|
||||||
response = client.delete("/api/v1/remote/nonexistent/path/to/file.tar.gz")
|
response = client.delete("/api/v1/local/nonexistent/path/to/file.tar.gz")
|
||||||
assert response.status_code == 404
|
assert response.status_code == 404
|
||||||
|
|
||||||
def test_non_local_remote_returns_400(self, client, patched_deps):
|
|
||||||
response = client.delete("/api/v1/remote/generic-test/path/to/file.tar.gz")
|
|
||||||
assert response.status_code == 400
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Cache flush PUT /cache/flush
|
# Cache flush PUT /cache/flush
|
||||||
@@ -924,3 +910,153 @@ class TestHelmRemote:
|
|||||||
|
|
||||||
response = client.get("/api/v1/remote/helm-test/vault.zip")
|
response = client.get("/api/v1/remote/helm-test/vault.zip")
|
||||||
assert response.status_code == 403
|
assert response.status_code == 403
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Quarantine (quarantine-test remote: quarantine_new=True, quarantine_days=3)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestQuarantine:
|
||||||
|
def _recent_date(self, days_ago=1):
|
||||||
|
"""Return an HTTP-format date string N days in the past (within quarantine window)."""
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
from email.utils import format_datetime
|
||||||
|
|
||||||
|
dt = datetime.now(UTC) - timedelta(days=days_ago)
|
||||||
|
return format_datetime(dt, usegmt=True)
|
||||||
|
|
||||||
|
def _old_date(self, days_ago=10):
|
||||||
|
"""Return an HTTP-format date string N days in the past (outside quarantine window)."""
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
from email.utils import format_datetime
|
||||||
|
|
||||||
|
dt = datetime.now(UTC) - timedelta(days=days_ago)
|
||||||
|
return format_datetime(dt, usegmt=True)
|
||||||
|
|
||||||
|
def test_cache_miss_recent_artifact_quarantined(self, client, patched_deps):
|
||||||
|
"""Cache miss: artifact published within quarantine window → 404."""
|
||||||
|
deps = patched_deps
|
||||||
|
deps["storage"].exists.return_value = False
|
||||||
|
deps["storage"].download_object.return_value = b"content"
|
||||||
|
deps["cache"].is_mutable_file.return_value = False
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"artifactapi.artifact.proxy.cache_single_artifact",
|
||||||
|
new_callable=AsyncMock,
|
||||||
|
return_value={"status": "cached", "last_modified": self._recent_date()},
|
||||||
|
):
|
||||||
|
response = client.get("/api/v1/remote/quarantine-test/some/path/package-1.0.tar.gz")
|
||||||
|
|
||||||
|
assert response.status_code == 404
|
||||||
|
assert "quarantined" in response.json()["detail"].lower()
|
||||||
|
|
||||||
|
def test_cache_miss_old_artifact_allowed(self, client, patched_deps):
|
||||||
|
"""Cache miss: artifact published outside quarantine window → 200."""
|
||||||
|
deps = patched_deps
|
||||||
|
deps["storage"].exists.return_value = False
|
||||||
|
deps["storage"].download_object.return_value = b"content"
|
||||||
|
deps["cache"].is_mutable_file.return_value = False
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"artifactapi.artifact.proxy.cache_single_artifact",
|
||||||
|
new_callable=AsyncMock,
|
||||||
|
return_value={"status": "cached", "last_modified": self._old_date()},
|
||||||
|
):
|
||||||
|
response = client.get("/api/v1/remote/quarantine-test/some/path/package-1.0.tar.gz")
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
def test_cache_miss_no_last_modified_fails_open(self, client, patched_deps):
|
||||||
|
"""Cache miss: no Last-Modified header → fail open (200, not quarantined)."""
|
||||||
|
deps = patched_deps
|
||||||
|
deps["storage"].exists.return_value = False
|
||||||
|
deps["storage"].download_object.return_value = b"content"
|
||||||
|
deps["cache"].is_mutable_file.return_value = False
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"artifactapi.artifact.proxy.cache_single_artifact",
|
||||||
|
new_callable=AsyncMock,
|
||||||
|
return_value={"status": "cached", "last_modified": None},
|
||||||
|
):
|
||||||
|
response = client.get("/api/v1/remote/quarantine-test/some/path/package-1.0.tar.gz")
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
def test_cache_hit_recent_artifact_quarantined(self, client, patched_deps):
|
||||||
|
"""Cache hit: stored publish date within quarantine window → 404."""
|
||||||
|
deps = patched_deps
|
||||||
|
deps["storage"].exists.return_value = True
|
||||||
|
deps["storage"].download_object.return_value = b"content"
|
||||||
|
deps["cache"].is_mutable_file.return_value = False
|
||||||
|
deps["cache"].get_artifact_published.return_value = self._recent_date()
|
||||||
|
|
||||||
|
response = client.get("/api/v1/remote/quarantine-test/some/path/package-1.0.tar.gz")
|
||||||
|
|
||||||
|
assert response.status_code == 404
|
||||||
|
assert "quarantined" in response.json()["detail"].lower()
|
||||||
|
|
||||||
|
def test_cache_hit_old_artifact_allowed(self, client, patched_deps):
|
||||||
|
"""Cache hit: stored publish date outside quarantine window → 200."""
|
||||||
|
deps = patched_deps
|
||||||
|
deps["storage"].exists.return_value = True
|
||||||
|
deps["storage"].download_object.return_value = b"content"
|
||||||
|
deps["cache"].is_mutable_file.return_value = False
|
||||||
|
deps["cache"].get_artifact_published.return_value = self._old_date()
|
||||||
|
|
||||||
|
response = client.get("/api/v1/remote/quarantine-test/some/path/package-1.0.tar.gz")
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
def test_cache_hit_no_stored_date_fetches_upstream(self, client, patched_deps):
|
||||||
|
"""Cache hit: no stored date → HEAD upstream to get Last-Modified."""
|
||||||
|
deps = patched_deps
|
||||||
|
deps["storage"].exists.return_value = True
|
||||||
|
deps["storage"].download_object.return_value = b"content"
|
||||||
|
deps["cache"].is_mutable_file.return_value = False
|
||||||
|
deps["cache"].get_artifact_published.return_value = None
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"artifactapi.artifact.proxy._fetch_last_modified",
|
||||||
|
new_callable=AsyncMock,
|
||||||
|
return_value=self._old_date(),
|
||||||
|
) as mock_fetch:
|
||||||
|
response = client.get("/api/v1/remote/quarantine-test/some/path/package-1.0.tar.gz")
|
||||||
|
|
||||||
|
mock_fetch.assert_called_once()
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
def test_quarantine_disabled_allows_recent_artifact(self, client, patched_deps):
|
||||||
|
"""quarantine_new=False: recent artifacts are not blocked."""
|
||||||
|
deps = patched_deps
|
||||||
|
deps["storage"].exists.return_value = False
|
||||||
|
deps["storage"].download_object.return_value = b"content"
|
||||||
|
deps["cache"].is_mutable_file.return_value = False
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"artifactapi.artifact.proxy.cache_single_artifact",
|
||||||
|
new_callable=AsyncMock,
|
||||||
|
return_value={"status": "cached", "last_modified": self._recent_date()},
|
||||||
|
):
|
||||||
|
response = client.get("/api/v1/remote/quarantine-disabled/some/path/package-1.0.tar.gz")
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
def test_quarantine_detail_includes_available_date(self, client, patched_deps):
|
||||||
|
"""The 404 detail should include the date when the artifact becomes available."""
|
||||||
|
deps = patched_deps
|
||||||
|
deps["storage"].exists.return_value = False
|
||||||
|
deps["storage"].download_object.return_value = b"content"
|
||||||
|
deps["cache"].is_mutable_file.return_value = False
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"artifactapi.artifact.proxy.cache_single_artifact",
|
||||||
|
new_callable=AsyncMock,
|
||||||
|
return_value={"status": "cached", "last_modified": self._recent_date()},
|
||||||
|
):
|
||||||
|
response = client.get("/api/v1/remote/quarantine-test/some/path/package-1.0.tar.gz")
|
||||||
|
|
||||||
|
assert response.status_code == 404
|
||||||
|
detail = response.json()["detail"]
|
||||||
|
assert "available after" in detail
|
||||||
|
assert "3-day" in detail
|
||||||
|
|||||||
@@ -0,0 +1,596 @@
|
|||||||
|
"""Unit tests for the virtual repository handler (artifact/virtual.py)."""
|
||||||
|
|
||||||
|
from datetime import UTC, date, datetime
|
||||||
|
from unittest.mock import AsyncMock, MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
from artifactapi.artifact.virtual import (
|
||||||
|
_HANDLERS,
|
||||||
|
_get_member_index,
|
||||||
|
_HelmDumper,
|
||||||
|
_HelmHandler,
|
||||||
|
_merge_helm_indexes,
|
||||||
|
_VirtualHandler,
|
||||||
|
)
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Shared sample data
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_INDEX_A = b"""\
|
||||||
|
apiVersion: v1
|
||||||
|
entries:
|
||||||
|
vault:
|
||||||
|
- name: vault
|
||||||
|
version: "0.27.0"
|
||||||
|
urls:
|
||||||
|
- https://helm.releases.hashicorp.com/vault-0.27.0.tgz
|
||||||
|
consul:
|
||||||
|
- name: consul
|
||||||
|
version: "1.2.0"
|
||||||
|
urls:
|
||||||
|
- https://helm.releases.hashicorp.com/consul-1.2.0.tgz
|
||||||
|
generated: "2023-01-01T00:00:00.000Z"
|
||||||
|
"""
|
||||||
|
|
||||||
|
_INDEX_B = b"""\
|
||||||
|
apiVersion: v1
|
||||||
|
entries:
|
||||||
|
nginx:
|
||||||
|
- name: nginx
|
||||||
|
version: "15.0.0"
|
||||||
|
urls:
|
||||||
|
- https://charts.example.com/nginx-15.0.0.tgz
|
||||||
|
vault:
|
||||||
|
- name: vault
|
||||||
|
version: "0.27.0"
|
||||||
|
urls:
|
||||||
|
- https://charts.example.com/vault-0.27.0.tgz
|
||||||
|
- name: vault
|
||||||
|
version: "0.26.0"
|
||||||
|
urls:
|
||||||
|
- https://charts.example.com/vault-0.26.0.tgz
|
||||||
|
generated: "2023-01-01T00:00:00.000Z"
|
||||||
|
"""
|
||||||
|
|
||||||
|
_INDEX_SIMPLE = b"""\
|
||||||
|
apiVersion: v1
|
||||||
|
entries:
|
||||||
|
mychart:
|
||||||
|
- name: mychart
|
||||||
|
version: "1.0.0"
|
||||||
|
urls:
|
||||||
|
- https://helm.releases.hashicorp.com/mychart-1.0.0.tgz
|
||||||
|
generated: "2023-01-01T00:00:00.000Z"
|
||||||
|
"""
|
||||||
|
|
||||||
|
_CFG_A = {"base_url": "https://helm.releases.hashicorp.com", "cache": {"mutable_ttl": 3600}}
|
||||||
|
_CFG_B = {"base_url": "https://charts.example.com", "cache": {"mutable_ttl": 1800}}
|
||||||
|
|
||||||
|
|
||||||
|
def _identity_resolve(data, *args, **kwargs):
|
||||||
|
return data, None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# _HelmDumper — datetime/date YAML serialization
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestHelmDumper:
|
||||||
|
def _dump(self, value):
|
||||||
|
return yaml.dump({"v": value}, Dumper=_HelmDumper)
|
||||||
|
|
||||||
|
def test_datetime_with_tz_includes_Z_suffix(self):
|
||||||
|
dt = datetime(2023, 6, 15, 12, 0, 0, tzinfo=UTC)
|
||||||
|
assert "Z" in self._dump(dt)
|
||||||
|
|
||||||
|
def test_datetime_without_tz_has_no_Z_suffix(self):
|
||||||
|
dt = datetime(2023, 6, 15, 12, 0, 0)
|
||||||
|
assert "Z" not in self._dump(dt)
|
||||||
|
|
||||||
|
def test_datetime_uses_T_separator_not_space(self):
|
||||||
|
dt = datetime(2023, 6, 15, 12, 30, 0, tzinfo=UTC)
|
||||||
|
assert "T12:30:00" in self._dump(dt)
|
||||||
|
|
||||||
|
def test_date_serialized_as_iso_string(self):
|
||||||
|
assert "2023-01-15" in self._dump(date(2023, 1, 15))
|
||||||
|
|
||||||
|
def test_datetime_round_trips_as_string_not_python_datetime(self):
|
||||||
|
dt = datetime(2023, 6, 15, 12, 0, 0, tzinfo=UTC)
|
||||||
|
parsed = yaml.safe_load(self._dump(dt))
|
||||||
|
# yaml.safe_load must not re-parse this as a datetime object
|
||||||
|
assert isinstance(parsed["v"], str)
|
||||||
|
|
||||||
|
def test_date_round_trips_as_string_not_python_date(self):
|
||||||
|
parsed = yaml.safe_load(self._dump(date(2023, 1, 15)))
|
||||||
|
assert isinstance(parsed["v"], str)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# _HelmHandler
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestHelmHandler:
|
||||||
|
def setup_method(self):
|
||||||
|
self.handler = _HelmHandler()
|
||||||
|
|
||||||
|
def test_accepts_index_yaml(self):
|
||||||
|
assert self.handler.accepts_path("index.yaml") is True
|
||||||
|
|
||||||
|
def test_rejects_tgz_path(self):
|
||||||
|
assert self.handler.accepts_path("vault-0.27.0.tgz") is False
|
||||||
|
|
||||||
|
def test_rejects_subdirectory_index(self):
|
||||||
|
assert self.handler.accepts_path("charts/index.yaml") is False
|
||||||
|
|
||||||
|
def test_rejects_empty_path(self):
|
||||||
|
assert self.handler.accepts_path("") is False
|
||||||
|
|
||||||
|
def test_path_error_is_non_empty_string(self):
|
||||||
|
msg = self.handler.path_error()
|
||||||
|
assert isinstance(msg, str) and len(msg) > 0
|
||||||
|
|
||||||
|
def test_merge_returns_bytes(self):
|
||||||
|
with patch("artifactapi.artifact.virtual._helm.resolve_content", side_effect=_identity_resolve):
|
||||||
|
result = self.handler.merge([_INDEX_A], ["member-a"], [_CFG_A], "http://proxy.example.com")
|
||||||
|
assert isinstance(result, bytes)
|
||||||
|
|
||||||
|
def test_merge_delegates_to_merge_helm_indexes(self):
|
||||||
|
with patch("artifactapi.artifact.virtual._merge_helm_indexes", return_value=b"merged") as mock_fn:
|
||||||
|
result = self.handler.merge([b"data"], ["m"], [{}], "http://proxy")
|
||||||
|
mock_fn.assert_called_once_with([b"data"], ["m"], [{}], "http://proxy")
|
||||||
|
assert result == b"merged"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# _HANDLERS registry
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestHandlersRegistry:
|
||||||
|
def test_helm_handler_is_registered(self):
|
||||||
|
assert "helm" in _HANDLERS
|
||||||
|
assert isinstance(_HANDLERS["helm"], _HelmHandler)
|
||||||
|
|
||||||
|
def test_helm_handler_satisfies_protocol(self):
|
||||||
|
assert isinstance(_HANDLERS["helm"], _VirtualHandler)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# _merge_helm_indexes
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestMergeHelmIndexes:
|
||||||
|
def _merge(self, raw_indexes, member_names, member_configs, proxy_base="http://proxy.example.com"):
|
||||||
|
with patch("artifactapi.artifact.virtual._helm.resolve_content", side_effect=_identity_resolve):
|
||||||
|
return _merge_helm_indexes(raw_indexes, member_names, member_configs, proxy_base)
|
||||||
|
|
||||||
|
def _parse(self, raw):
|
||||||
|
return yaml.safe_load(raw)
|
||||||
|
|
||||||
|
def test_single_member_all_charts_present(self):
|
||||||
|
index = self._parse(self._merge([_INDEX_A], ["member-a"], [_CFG_A]))
|
||||||
|
assert "vault" in index["entries"]
|
||||||
|
assert "consul" in index["entries"]
|
||||||
|
|
||||||
|
def test_two_members_non_overlapping_charts_all_present(self):
|
||||||
|
index = self._parse(self._merge([_INDEX_A, _INDEX_B], ["member-a", "member-b"], [_CFG_A, _CFG_B]))
|
||||||
|
assert "vault" in index["entries"]
|
||||||
|
assert "consul" in index["entries"]
|
||||||
|
assert "nginx" in index["entries"]
|
||||||
|
|
||||||
|
def test_first_member_wins_on_duplicate_name_and_version(self):
|
||||||
|
index = self._parse(self._merge([_INDEX_A, _INDEX_B], ["member-a", "member-b"], [_CFG_A, _CFG_B]))
|
||||||
|
v027 = next(e for e in index["entries"]["vault"] if e["version"] == "0.27.0")
|
||||||
|
assert "helm.releases.hashicorp.com" in v027["urls"][0]
|
||||||
|
|
||||||
|
def test_different_versions_of_same_chart_both_included(self):
|
||||||
|
index = self._parse(self._merge([_INDEX_A, _INDEX_B], ["member-a", "member-b"], [_CFG_A, _CFG_B]))
|
||||||
|
versions = {e["version"] for e in index["entries"]["vault"]}
|
||||||
|
assert "0.27.0" in versions
|
||||||
|
assert "0.26.0" in versions
|
||||||
|
|
||||||
|
def test_malformed_yaml_from_member_is_skipped(self):
|
||||||
|
index = self._parse(self._merge([_INDEX_A, b"{bad yaml"], ["member-a", "bad"], [_CFG_A, _CFG_B]))
|
||||||
|
assert "vault" in index["entries"]
|
||||||
|
assert "consul" in index["entries"]
|
||||||
|
|
||||||
|
def test_output_has_apiVersion_v1(self):
|
||||||
|
index = self._parse(self._merge([_INDEX_A], ["member-a"], [_CFG_A]))
|
||||||
|
assert index["apiVersion"] == "v1"
|
||||||
|
|
||||||
|
def test_output_has_generated_field(self):
|
||||||
|
index = self._parse(self._merge([_INDEX_A], ["member-a"], [_CFG_A]))
|
||||||
|
assert "generated" in index
|
||||||
|
|
||||||
|
def test_output_is_valid_yaml(self):
|
||||||
|
raw = self._merge([_INDEX_A, _INDEX_B], ["member-a", "member-b"], [_CFG_A, _CFG_B])
|
||||||
|
assert isinstance(yaml.safe_load(raw), dict)
|
||||||
|
|
||||||
|
def test_empty_index_from_member_produces_no_entries(self):
|
||||||
|
empty = b"apiVersion: v1\nentries: {}\ngenerated: '2023-01-01T00:00:00.000Z'\n"
|
||||||
|
index = self._parse(self._merge([empty], ["member-a"], [_CFG_A]))
|
||||||
|
assert index["entries"] == {}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# _get_member_index (async)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestGetMemberIndex:
|
||||||
|
@pytest.fixture
|
||||||
|
def storage(self):
|
||||||
|
m = MagicMock()
|
||||||
|
m.get_object_key.return_value = "member/key/index.yaml"
|
||||||
|
m.exists.return_value = False
|
||||||
|
m.download_object.return_value = b"cached bytes"
|
||||||
|
return m
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def cache(self):
|
||||||
|
m = MagicMock()
|
||||||
|
m.is_index_valid.return_value = False
|
||||||
|
return m
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def member_cfg(self):
|
||||||
|
return {"base_url": "https://helm.releases.hashicorp.com", "cache": {"mutable_ttl": 3600}}
|
||||||
|
|
||||||
|
def _fake_response(self, content=b"upstream bytes"):
|
||||||
|
r = MagicMock()
|
||||||
|
r.content = content
|
||||||
|
r.raise_for_status = MagicMock()
|
||||||
|
return r
|
||||||
|
|
||||||
|
def _patch_httpx(self, response):
|
||||||
|
mock_client_cls = patch("artifactapi.artifact.virtual.httpx.AsyncClient")
|
||||||
|
p = mock_client_cls.start()
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
p.return_value.__aenter__.return_value = mock_client
|
||||||
|
mock_client.get.return_value = response
|
||||||
|
return mock_client_cls, mock_client
|
||||||
|
|
||||||
|
async def test_cache_hit_returns_stored_bytes(self, storage, cache, member_cfg):
|
||||||
|
storage.exists.return_value = True
|
||||||
|
cache.is_index_valid.return_value = True
|
||||||
|
|
||||||
|
_, _, _, raw_data = await _get_member_index("m", member_cfg, "index.yaml", storage, cache)
|
||||||
|
|
||||||
|
assert raw_data == b"cached bytes"
|
||||||
|
|
||||||
|
async def test_cache_hit_does_not_fetch_upstream(self, storage, cache, member_cfg):
|
||||||
|
storage.exists.return_value = True
|
||||||
|
cache.is_index_valid.return_value = True
|
||||||
|
|
||||||
|
with patch("artifactapi.artifact.virtual.httpx.AsyncClient") as mock_cls:
|
||||||
|
await _get_member_index("m", member_cfg, "index.yaml", storage, cache)
|
||||||
|
|
||||||
|
mock_cls.assert_not_called()
|
||||||
|
|
||||||
|
async def test_cache_hit_storage_error_falls_through_to_upstream(self, storage, cache, member_cfg):
|
||||||
|
storage.exists.return_value = True
|
||||||
|
cache.is_index_valid.return_value = True
|
||||||
|
storage.download_object.side_effect = Exception("S3 read error")
|
||||||
|
|
||||||
|
with patch("artifactapi.artifact.virtual.httpx.AsyncClient") as mock_cls:
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_cls.return_value.__aenter__.return_value = mock_client
|
||||||
|
mock_client.get.return_value = self._fake_response(b"fresh bytes")
|
||||||
|
|
||||||
|
_, _, _, raw_data = await _get_member_index("m", member_cfg, "index.yaml", storage, cache)
|
||||||
|
|
||||||
|
assert raw_data == b"fresh bytes"
|
||||||
|
|
||||||
|
async def test_cache_miss_fetches_from_upstream(self, storage, cache, member_cfg):
|
||||||
|
with patch("artifactapi.artifact.virtual.httpx.AsyncClient") as mock_cls:
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_cls.return_value.__aenter__.return_value = mock_client
|
||||||
|
mock_client.get.return_value = self._fake_response()
|
||||||
|
|
||||||
|
_, _, _, raw_data = await _get_member_index("m", member_cfg, "index.yaml", storage, cache)
|
||||||
|
|
||||||
|
assert raw_data == b"upstream bytes"
|
||||||
|
|
||||||
|
async def test_cache_miss_stores_result_in_s3(self, storage, cache, member_cfg):
|
||||||
|
with patch("artifactapi.artifact.virtual.httpx.AsyncClient") as mock_cls:
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_cls.return_value.__aenter__.return_value = mock_client
|
||||||
|
mock_client.get.return_value = self._fake_response()
|
||||||
|
|
||||||
|
await _get_member_index("m", member_cfg, "index.yaml", storage, cache)
|
||||||
|
|
||||||
|
storage.upload.assert_called_once()
|
||||||
|
|
||||||
|
async def test_cache_miss_marks_cache_with_configured_ttl(self, storage, cache, member_cfg):
|
||||||
|
with patch("artifactapi.artifact.virtual.httpx.AsyncClient") as mock_cls:
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_cls.return_value.__aenter__.return_value = mock_client
|
||||||
|
mock_client.get.return_value = self._fake_response()
|
||||||
|
|
||||||
|
await _get_member_index("m", member_cfg, "index.yaml", storage, cache)
|
||||||
|
|
||||||
|
cache.mark_index_cached.assert_called_once_with("m", "index.yaml", 3600)
|
||||||
|
|
||||||
|
async def test_cache_miss_with_auth_sends_basic_auth_header(self, storage, cache):
|
||||||
|
cfg = {
|
||||||
|
"base_url": "https://private.example.com",
|
||||||
|
"username": "user",
|
||||||
|
"password": "pass",
|
||||||
|
"cache": {"mutable_ttl": 3600},
|
||||||
|
}
|
||||||
|
with patch("artifactapi.artifact.virtual.httpx.AsyncClient") as mock_cls:
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_cls.return_value.__aenter__.return_value = mock_client
|
||||||
|
mock_client.get.return_value = self._fake_response()
|
||||||
|
|
||||||
|
await _get_member_index("m", cfg, "index.yaml", storage, cache)
|
||||||
|
|
||||||
|
headers = mock_client.get.call_args.kwargs["headers"]
|
||||||
|
assert "Authorization" in headers
|
||||||
|
assert headers["Authorization"].startswith("Basic ")
|
||||||
|
|
||||||
|
async def test_no_credentials_sends_no_auth_header(self, storage, cache, member_cfg):
|
||||||
|
with patch("artifactapi.artifact.virtual.httpx.AsyncClient") as mock_cls:
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_cls.return_value.__aenter__.return_value = mock_client
|
||||||
|
mock_client.get.return_value = self._fake_response()
|
||||||
|
|
||||||
|
await _get_member_index("m", member_cfg, "index.yaml", storage, cache)
|
||||||
|
|
||||||
|
headers = mock_client.get.call_args.kwargs["headers"]
|
||||||
|
assert "Authorization" not in headers
|
||||||
|
|
||||||
|
async def test_upstream_fetch_failure_returns_none(self, storage, cache, member_cfg):
|
||||||
|
with patch("artifactapi.artifact.virtual.httpx.AsyncClient") as mock_cls:
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_cls.return_value.__aenter__.return_value = mock_client
|
||||||
|
mock_client.get.side_effect = Exception("connection refused")
|
||||||
|
|
||||||
|
_, _, _, raw_data = await _get_member_index("m", member_cfg, "index.yaml", storage, cache)
|
||||||
|
|
||||||
|
assert raw_data is None
|
||||||
|
|
||||||
|
async def test_s3_upload_failure_still_returns_data(self, storage, cache, member_cfg):
|
||||||
|
storage.upload.side_effect = Exception("S3 write error")
|
||||||
|
|
||||||
|
with patch("artifactapi.artifact.virtual.httpx.AsyncClient") as mock_cls:
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_cls.return_value.__aenter__.return_value = mock_client
|
||||||
|
mock_client.get.return_value = self._fake_response()
|
||||||
|
|
||||||
|
_, _, _, raw_data = await _get_member_index("m", member_cfg, "index.yaml", storage, cache)
|
||||||
|
|
||||||
|
assert raw_data == b"upstream bytes"
|
||||||
|
|
||||||
|
async def test_returns_ttl_from_config(self, storage, cache):
|
||||||
|
cfg = {"base_url": "https://example.com", "cache": {"mutable_ttl": 900}}
|
||||||
|
with patch("artifactapi.artifact.virtual.httpx.AsyncClient") as mock_cls:
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_cls.return_value.__aenter__.return_value = mock_client
|
||||||
|
mock_client.get.return_value = self._fake_response()
|
||||||
|
|
||||||
|
_, _, ttl, _ = await _get_member_index("m", cfg, "index.yaml", storage, cache)
|
||||||
|
|
||||||
|
assert ttl == 900
|
||||||
|
|
||||||
|
async def test_defaults_ttl_to_3600_when_not_configured(self, storage, cache):
|
||||||
|
cfg = {"base_url": "https://example.com"}
|
||||||
|
with patch("artifactapi.artifact.virtual.httpx.AsyncClient") as mock_cls:
|
||||||
|
mock_client = AsyncMock()
|
||||||
|
mock_cls.return_value.__aenter__.return_value = mock_client
|
||||||
|
mock_client.get.return_value = self._fake_response()
|
||||||
|
|
||||||
|
_, _, ttl, _ = await _get_member_index("m", cfg, "index.yaml", storage, cache)
|
||||||
|
|
||||||
|
assert ttl == 3600
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Virtual route GET /api/v1/virtual/{name}/{path}
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def mock_storage_v():
|
||||||
|
m = MagicMock()
|
||||||
|
m.get_object_key.return_value = "virtual/helm-virtual-test/index.yaml"
|
||||||
|
m.exists.return_value = False
|
||||||
|
m.download_object.return_value = b"apiVersion: v1\nentries: {}\n"
|
||||||
|
return m
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def mock_cache_v():
|
||||||
|
m = MagicMock()
|
||||||
|
m.is_index_valid.return_value = False
|
||||||
|
m.available = False
|
||||||
|
m.client = None
|
||||||
|
return m
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def patched_virtual_deps(mock_storage_v, mock_cache_v):
|
||||||
|
import artifactapi.main as main_mod
|
||||||
|
|
||||||
|
with (
|
||||||
|
patch.object(main_mod, "storage", mock_storage_v),
|
||||||
|
patch.object(main_mod, "cache", mock_cache_v),
|
||||||
|
):
|
||||||
|
yield {"storage": mock_storage_v, "cache": mock_cache_v}
|
||||||
|
|
||||||
|
|
||||||
|
class TestVirtualRoute:
|
||||||
|
def test_unknown_virtual_name_returns_404(self, client, patched_virtual_deps):
|
||||||
|
response = client.get("/api/v1/virtual/no-such-virtual/index.yaml")
|
||||||
|
assert response.status_code == 404
|
||||||
|
|
||||||
|
def test_non_virtual_name_returns_404(self, client, patched_virtual_deps):
|
||||||
|
# helm-test is in remotes, not virtuals
|
||||||
|
response = client.get("/api/v1/virtual/helm-test/index.yaml")
|
||||||
|
assert response.status_code == 404
|
||||||
|
|
||||||
|
def test_unsupported_package_returns_400(self, client, patched_virtual_deps):
|
||||||
|
# unsupported-virtual-test has package "rpm"
|
||||||
|
response = client.get("/api/v1/virtual/unsupported-virtual-test/index.yaml")
|
||||||
|
assert response.status_code == 400
|
||||||
|
|
||||||
|
def test_non_index_path_returns_404(self, client, patched_virtual_deps):
|
||||||
|
response = client.get("/api/v1/virtual/helm-virtual-test/vault-0.27.0.tgz")
|
||||||
|
assert response.status_code == 404
|
||||||
|
|
||||||
|
def test_no_members_returns_500(self, client, patched_virtual_deps):
|
||||||
|
response = client.get("/api/v1/virtual/empty-virtual-test/index.yaml")
|
||||||
|
assert response.status_code == 500
|
||||||
|
|
||||||
|
def test_virtual_cache_hit_returns_200(self, client, patched_virtual_deps):
|
||||||
|
deps = patched_virtual_deps
|
||||||
|
deps["storage"].exists.return_value = True
|
||||||
|
deps["cache"].is_index_valid.return_value = True
|
||||||
|
|
||||||
|
response = client.get("/api/v1/virtual/helm-virtual-test/index.yaml")
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
def test_virtual_cache_hit_content_type_is_yaml(self, client, patched_virtual_deps):
|
||||||
|
deps = patched_virtual_deps
|
||||||
|
deps["storage"].exists.return_value = True
|
||||||
|
deps["cache"].is_index_valid.return_value = True
|
||||||
|
|
||||||
|
response = client.get("/api/v1/virtual/helm-virtual-test/index.yaml")
|
||||||
|
assert "text/yaml" in response.headers["content-type"]
|
||||||
|
|
||||||
|
def test_virtual_cache_hit_returns_stored_content(self, client, patched_virtual_deps):
|
||||||
|
deps = patched_virtual_deps
|
||||||
|
deps["storage"].exists.return_value = True
|
||||||
|
deps["cache"].is_index_valid.return_value = True
|
||||||
|
deps["storage"].download_object.return_value = b"apiVersion: v1\nentries: {}\n"
|
||||||
|
|
||||||
|
response = client.get("/api/v1/virtual/helm-virtual-test/index.yaml")
|
||||||
|
assert response.content == b"apiVersion: v1\nentries: {}\n"
|
||||||
|
|
||||||
|
def test_virtual_cache_hit_skips_member_fetch(self, client, patched_virtual_deps):
|
||||||
|
deps = patched_virtual_deps
|
||||||
|
deps["storage"].exists.return_value = True
|
||||||
|
deps["cache"].is_index_valid.return_value = True
|
||||||
|
|
||||||
|
with patch("artifactapi.artifact.virtual._get_member_index", new_callable=AsyncMock) as mock_get:
|
||||||
|
client.get("/api/v1/virtual/helm-virtual-test/index.yaml")
|
||||||
|
|
||||||
|
mock_get.assert_not_called()
|
||||||
|
|
||||||
|
def test_cache_miss_returns_200_with_yaml_content_type(self, client, patched_virtual_deps):
|
||||||
|
with (
|
||||||
|
patch("artifactapi.artifact.virtual._get_member_index", new_callable=AsyncMock) as mock_get,
|
||||||
|
patch("artifactapi.artifact.virtual._helm.resolve_content", side_effect=_identity_resolve),
|
||||||
|
):
|
||||||
|
mock_get.return_value = ("helm-test", _CFG_A, 3600, _INDEX_SIMPLE)
|
||||||
|
response = client.get("/api/v1/virtual/helm-virtual-test/index.yaml")
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert "text/yaml" in response.headers["content-type"]
|
||||||
|
|
||||||
|
def test_cache_miss_response_contains_merged_entries(self, client, patched_virtual_deps):
|
||||||
|
with (
|
||||||
|
patch("artifactapi.artifact.virtual._get_member_index", new_callable=AsyncMock) as mock_get,
|
||||||
|
patch("artifactapi.artifact.virtual._helm.resolve_content", side_effect=_identity_resolve),
|
||||||
|
):
|
||||||
|
mock_get.return_value = ("helm-test", _CFG_A, 3600, _INDEX_SIMPLE)
|
||||||
|
response = client.get("/api/v1/virtual/helm-virtual-test/index.yaml")
|
||||||
|
|
||||||
|
index = yaml.safe_load(response.content)
|
||||||
|
assert "mychart" in index["entries"]
|
||||||
|
|
||||||
|
def test_cache_miss_stores_result_in_s3(self, client, patched_virtual_deps):
|
||||||
|
deps = patched_virtual_deps
|
||||||
|
with (
|
||||||
|
patch("artifactapi.artifact.virtual._get_member_index", new_callable=AsyncMock) as mock_get,
|
||||||
|
patch("artifactapi.artifact.virtual._helm.resolve_content", side_effect=_identity_resolve),
|
||||||
|
):
|
||||||
|
mock_get.return_value = ("helm-test", _CFG_A, 3600, _INDEX_SIMPLE)
|
||||||
|
client.get("/api/v1/virtual/helm-virtual-test/index.yaml")
|
||||||
|
|
||||||
|
deps["storage"].upload.assert_called_once()
|
||||||
|
|
||||||
|
def test_cache_miss_marks_index_cached(self, client, patched_virtual_deps):
|
||||||
|
deps = patched_virtual_deps
|
||||||
|
with (
|
||||||
|
patch("artifactapi.artifact.virtual._get_member_index", new_callable=AsyncMock) as mock_get,
|
||||||
|
patch("artifactapi.artifact.virtual._helm.resolve_content", side_effect=_identity_resolve),
|
||||||
|
):
|
||||||
|
mock_get.return_value = ("helm-test", _CFG_A, 3600, _INDEX_SIMPLE)
|
||||||
|
client.get("/api/v1/virtual/helm-virtual-test/index.yaml")
|
||||||
|
|
||||||
|
deps["cache"].mark_index_cached.assert_called_once()
|
||||||
|
|
||||||
|
def test_cache_miss_uses_min_ttl_across_members(self, client, patched_virtual_deps):
|
||||||
|
deps = patched_virtual_deps
|
||||||
|
with (
|
||||||
|
patch("artifactapi.artifact.virtual._get_member_index", new_callable=AsyncMock) as mock_get,
|
||||||
|
patch("artifactapi.artifact.virtual._helm.resolve_content", side_effect=_identity_resolve),
|
||||||
|
):
|
||||||
|
mock_get.side_effect = [
|
||||||
|
("helm-test", _CFG_A, 3600, _INDEX_SIMPLE),
|
||||||
|
("helm-member-2", _CFG_B, 1800, _INDEX_SIMPLE),
|
||||||
|
]
|
||||||
|
client.get("/api/v1/virtual/helm-virtual-test/index.yaml")
|
||||||
|
|
||||||
|
_, _, ttl = deps["cache"].mark_index_cached.call_args[0]
|
||||||
|
assert ttl == 1800
|
||||||
|
|
||||||
|
def test_all_members_unreachable_returns_502(self, client, patched_virtual_deps):
|
||||||
|
with patch("artifactapi.artifact.virtual._get_member_index", new_callable=AsyncMock) as mock_get:
|
||||||
|
mock_get.return_value = ("helm-test", _CFG_A, 3600, None)
|
||||||
|
response = client.get("/api/v1/virtual/helm-virtual-test/index.yaml")
|
||||||
|
|
||||||
|
assert response.status_code == 502
|
||||||
|
|
||||||
|
def test_one_member_unreachable_still_returns_200(self, client, patched_virtual_deps):
|
||||||
|
with (
|
||||||
|
patch("artifactapi.artifact.virtual._get_member_index", new_callable=AsyncMock) as mock_get,
|
||||||
|
patch("artifactapi.artifact.virtual._helm.resolve_content", side_effect=_identity_resolve),
|
||||||
|
):
|
||||||
|
mock_get.side_effect = [
|
||||||
|
("helm-test", _CFG_A, 3600, _INDEX_SIMPLE),
|
||||||
|
("helm-member-2", _CFG_B, 1800, None),
|
||||||
|
]
|
||||||
|
response = client.get("/api/v1/virtual/helm-virtual-test/index.yaml")
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
def test_member_not_in_config_is_skipped(self, client, patched_virtual_deps):
|
||||||
|
import artifactapi.main as main_mod
|
||||||
|
|
||||||
|
real_get = main_mod.config.get_remote_config
|
||||||
|
|
||||||
|
def patched_get(name):
|
||||||
|
return None if name == "helm-member-2" else real_get(name)
|
||||||
|
|
||||||
|
with (
|
||||||
|
patch("artifactapi.artifact.virtual._get_member_index", new_callable=AsyncMock) as mock_get,
|
||||||
|
patch("artifactapi.artifact.virtual._helm.resolve_content", side_effect=_identity_resolve),
|
||||||
|
patch.object(main_mod.config, "get_remote_config", side_effect=patched_get),
|
||||||
|
):
|
||||||
|
mock_get.return_value = ("helm-test", _CFG_A, 3600, _INDEX_SIMPLE)
|
||||||
|
response = client.get("/api/v1/virtual/helm-virtual-test/index.yaml")
|
||||||
|
|
||||||
|
# only helm-test was available — should succeed
|
||||||
|
assert response.status_code == 200
|
||||||
|
mock_get.assert_called_once()
|
||||||
|
|
||||||
|
def test_s3_store_failure_still_returns_200(self, client, patched_virtual_deps):
|
||||||
|
deps = patched_virtual_deps
|
||||||
|
deps["storage"].upload.side_effect = Exception("S3 write error")
|
||||||
|
|
||||||
|
with (
|
||||||
|
patch("artifactapi.artifact.virtual._get_member_index", new_callable=AsyncMock) as mock_get,
|
||||||
|
patch("artifactapi.artifact.virtual._helm.resolve_content", side_effect=_identity_resolve),
|
||||||
|
):
|
||||||
|
mock_get.return_value = ("helm-test", _CFG_A, 3600, _INDEX_SIMPLE)
|
||||||
|
response = client.get("/api/v1/virtual/helm-virtual-test/index.yaml")
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
Reference in New Issue
Block a user