Compare commits
4 Commits
v2.7.2
..
8fc9b179a6
| Author | SHA1 | Date | |
|---|---|---|---|
| 8fc9b179a6 | |||
| 9287cf7cf2 | |||
| ff2aefeef4 | |||
| a115904bbc |
@@ -4,13 +4,14 @@ FastAPI caching proxy that downloads and stores files from remote sources in S3-
|
||||
|
||||
## Features
|
||||
|
||||
- Remote definitions via `remotes.yaml` — generic HTTP, Alpine APK, RPM, Docker, PyPI, npm, Helm
|
||||
- Remote definitions via `remotes.yaml` — generic HTTP, Alpine APK, RPM, Docker, PyPI, npm, Helm, Puppet Forge, Terraform/OpenTofu registry
|
||||
- Virtual repositories — merge multiple remotes of the same package type into a single unified index
|
||||
- Immutable/mutable caching model with per-remote TTLs
|
||||
- Conditional revalidation (`If-None-Match` / `If-Modified-Since`) on TTL expiry
|
||||
- Stale-on-upstream-error: refreshes TTL when backend is unreachable rather than evicting
|
||||
- URL rewriting for PyPI simple index, npm metadata, and Helm `index.yaml`
|
||||
- Access control via regex patterns — unmatched paths return 403
|
||||
- Docker tag banning — block named tags (e.g. `latest`) while allowing digest pulls
|
||||
|
||||
## Architecture
|
||||
|
||||
@@ -61,8 +62,10 @@ src/artifactapi/
|
||||
├── generic.py — generic HTTP remotes
|
||||
├── helm.py — Helm index.yaml URL rewriting
|
||||
├── npm.py — npm metadata URL rewriting
|
||||
├── puppet.py — Puppet Forge JSON URL rewriting
|
||||
├── python.py — PyPI URL construction + HTML rewriting
|
||||
└── rpm.py — RPM remotes
|
||||
├── rpm.py — RPM remotes
|
||||
└── terraform.py — Terraform/OpenTofu registry URL construction + download URL rewriting
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
@@ -129,7 +132,7 @@ Repositories are declared under three top-level keys matching their type:
|
||||
remotes: # proxy (caching) remotes
|
||||
remote-name:
|
||||
base_url: "https://example.com"
|
||||
package: "generic" # generic, alpine, rpm, docker, pypi, npm, helm
|
||||
package: "generic" # generic, alpine, rpm, docker, pypi, npm, helm, puppet, terraform
|
||||
description: "..."
|
||||
immutable_patterns: # regex — cached forever
|
||||
- ".*\\.tar\\.gz$"
|
||||
@@ -243,6 +246,26 @@ remotes:
|
||||
|
||||
Tag manifests and `/tags/list` are built-in mutable patterns. Digest-addressed blobs are immutable.
|
||||
|
||||
#### Banning tags
|
||||
|
||||
Set `ban_tags_enabled: true` and list named tags in `ban_tags` to block specific tag references. Requests for a banned tag return `403`. Digest-addressed pulls (`sha256:…`) are never blocked, so images already in use can still be referenced by digest.
|
||||
|
||||
```yaml
|
||||
remotes:
|
||||
dockerhub:
|
||||
base_url: "https://registry-1.docker.io"
|
||||
package: "docker"
|
||||
ban_tags_enabled: true
|
||||
ban_tags:
|
||||
- latest # force pinned tags in CI/CD
|
||||
- edge
|
||||
cache:
|
||||
immutable_ttl: 0
|
||||
mutable_ttl: 300
|
||||
```
|
||||
|
||||
`ban_tags_enabled` defaults to `false`. Setting it to `true` with an empty `ban_tags` list has no effect.
|
||||
|
||||
For RKE2/containerd, configure `/etc/rancher/rke2/registries.yaml`:
|
||||
|
||||
```yaml
|
||||
@@ -340,6 +363,94 @@ helm repo add hashicorp https://artifacts.example.com/api/v1/remote/hashicorp-he
|
||||
helm repo update
|
||||
```
|
||||
|
||||
### puppet
|
||||
|
||||
Proxy for [Puppet Forge](https://forge.puppet.com) (forgeapi.puppet.com). Module metadata is cached as mutable; versioned module tarballs are cached as immutable.
|
||||
|
||||
```yaml
|
||||
remotes:
|
||||
puppet-forge:
|
||||
base_url: "https://forgeapi.puppet.com"
|
||||
package: "puppet"
|
||||
check_mutable_updates: true
|
||||
immutable_patterns:
|
||||
- "^v3/files/.*\\.tar\\.gz$"
|
||||
cache:
|
||||
immutable_ttl: 0 # module tarballs cached indefinitely
|
||||
mutable_ttl: 600 # module metadata refreshed after 10 minutes
|
||||
```
|
||||
|
||||
`v3/modules/` and `v3/releases` are built-in mutable patterns — module metadata pages expire after `mutable_ttl` and are re-fetched on the next request.
|
||||
|
||||
**URL rewriting**: the proxy rewrites `file_uri` fields in Forge JSON responses from relative paths (`/v3/files/…`) to absolute proxy URLs. g10k resolves download URLs with Go's `url.ResolveReference`, so an absolute `file_uri` overrides the forge base entirely — tarballs download straight from the proxy without a second hop.
|
||||
|
||||
**Client configuration — g10k**: set `forge_base_url` in the g10k config file:
|
||||
|
||||
```yaml
|
||||
# g10k.yaml
|
||||
cachedir: /tmp/g10k
|
||||
forge_base_url: https://artifacts.example.com/api/v1/remote/puppet-forge
|
||||
sources:
|
||||
control:
|
||||
remote: git@git.example.com:puppet/control.git
|
||||
basedir: /etc/puppetlabs/code/environments
|
||||
```
|
||||
|
||||
Alternatively, set the URL per-Puppetfile with the `forge.baseUrl` directive (works with `-puppetfile` mode and does not require a config file):
|
||||
|
||||
```ruby
|
||||
forge.baseUrl https://artifacts.example.com/api/v1/remote/puppet-forge
|
||||
|
||||
mod 'puppetlabs-stdlib', '9.7.0'
|
||||
mod 'puppetlabs-inifile', '6.2.0'
|
||||
```
|
||||
|
||||
### terraform
|
||||
|
||||
Proxy for [Terraform](https://registry.terraform.io) / [OpenTofu](https://opentofu.org) provider registries using the [Registry Protocol](https://developer.hashicorp.com/terraform/internals/provider-registry-protocol). Provider version listings are mutable; per-version download info is immutable.
|
||||
|
||||
Two remotes are needed: one for the registry API and one for the release CDN (where the actual `.zip` binaries live):
|
||||
|
||||
```yaml
|
||||
remotes:
|
||||
terraform-registry:
|
||||
base_url: "https://registry.terraform.io"
|
||||
package: "terraform"
|
||||
releases_remote: "hashicorp-releases" # name of the CDN remote below
|
||||
immutable_patterns:
|
||||
- "[^/]+/[^/]+/[^/]+/download/[^/]+/[^/]+$"
|
||||
cache:
|
||||
immutable_ttl: 0 # per-version download info cached indefinitely
|
||||
mutable_ttl: 300 # provider version lists refreshed after 5 minutes
|
||||
|
||||
hashicorp-releases:
|
||||
base_url: "https://releases.hashicorp.com"
|
||||
package: "generic"
|
||||
immutable_patterns:
|
||||
- ".*\\.zip$"
|
||||
- ".*SHA256SUMS(\\.sig)?$"
|
||||
cache:
|
||||
immutable_ttl: 0
|
||||
mutable_ttl: 0
|
||||
```
|
||||
|
||||
`{namespace}/{type}/versions` is a built-in mutable pattern — the version list expires after `mutable_ttl` and is re-fetched on the next request.
|
||||
|
||||
**URL rewriting**: the `download_url`, `shasums_url`, and `shasums_signature_url` fields in per-version download info JSON are rewritten from `releases.hashicorp.com` to point at the remote named by `releases_remote`, so Terraform fetches binaries through the proxy.
|
||||
|
||||
**Client configuration**: redirect Terraform's provider registry lookup via `.terraformrc` without changing any provider source addresses in your Terraform code:
|
||||
|
||||
```hcl
|
||||
# ~/.terraformrc (or /etc/terraform.rc, or TF_CLI_CONFIG_FILE)
|
||||
host "registry.terraform.io" {
|
||||
services = {
|
||||
"providers.v1" = "http://artifacts.example.com/api/v1/remote/terraform-registry/"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
With this in place, `terraform init` / `tofu init` fetches provider metadata from the proxy and downloads zips from the `hashicorp-releases` remote. No changes to `.tf` files are needed.
|
||||
|
||||
### virtual
|
||||
|
||||
A virtual repository presents a single unified index built from multiple member remotes of the same package type. Clients configure one endpoint and get access to all member remotes transparently.
|
||||
@@ -436,6 +547,8 @@ Each package type has built-in defaults that are merged with any user-defined `m
|
||||
| `docker` | Tag manifests (non-digest refs), `/tags/list` |
|
||||
| `pypi` | `simple/` (per-package and top-level index pages) |
|
||||
| `helm` | `index\.yaml$` |
|
||||
| `puppet` | `^v3/modules/`, `^v3/releases` |
|
||||
| `terraform` | `[^/]+/[^/]+/versions$` |
|
||||
| `npm` | *(none built-in — define via `mutable_patterns`)* |
|
||||
| `generic` | *(none)* |
|
||||
|
||||
|
||||
@@ -452,6 +452,51 @@ remotes:
|
||||
immutable_ttl: 0
|
||||
mutable_ttl: 3600
|
||||
|
||||
puppet-forge:
|
||||
base_url: "https://forgeapi.puppet.com"
|
||||
package: "puppet"
|
||||
description: "Puppet Forge module registry"
|
||||
# Module metadata (v3/modules/, v3/releases) is mutable by default.
|
||||
# Configure r10k / librarian-puppet with this remote as the Forge URL:
|
||||
# http://your-proxy/api/v1/remote/puppet-forge
|
||||
check_mutable_updates: true
|
||||
immutable_patterns:
|
||||
- "^v3/files/.*\\.tar\\.gz$"
|
||||
cache:
|
||||
immutable_ttl: 0 # Module tarballs cached indefinitely
|
||||
mutable_ttl: 600 # Module metadata refreshed after 10 minutes
|
||||
|
||||
terraform-registry:
|
||||
base_url: "https://registry.terraform.io"
|
||||
package: "terraform"
|
||||
description: "Terraform/OpenTofu provider registry (Registry Protocol)"
|
||||
# Provider version lists are mutable by default.
|
||||
# Point Terraform at this remote via .terraformrc:
|
||||
# host "registry.terraform.io" {
|
||||
# services = {
|
||||
# "providers.v1" = "http://your-proxy/api/v1/remote/terraform-registry/"
|
||||
# }
|
||||
# }
|
||||
# releases_remote must match the name of the hashicorp-releases remote below,
|
||||
# so download_url / shasums_url in per-version download info are rewritten.
|
||||
releases_remote: "hashicorp-releases"
|
||||
immutable_patterns:
|
||||
- "[^/]+/[^/]+/[^/]+/download/[^/]+/[^/]+$"
|
||||
cache:
|
||||
immutable_ttl: 0 # Per-version download info cached indefinitely
|
||||
mutable_ttl: 300 # Provider versions list refreshed after 5 minutes
|
||||
|
||||
hashicorp-releases:
|
||||
base_url: "https://releases.hashicorp.com"
|
||||
package: "generic"
|
||||
description: "HashiCorp releases CDN — provider zips, SHA256SUMS, and signatures"
|
||||
immutable_patterns:
|
||||
- ".*\\.zip$"
|
||||
- ".*SHA256SUMS(\\.sig)?$"
|
||||
cache:
|
||||
immutable_ttl: 0 # Release artifacts cached indefinitely
|
||||
mutable_ttl: 0
|
||||
|
||||
|
||||
virtuals:
|
||||
helm-all:
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
@@ -33,6 +34,16 @@ async def proxy(request: Request, remote_name: str, path: str, storage, cache, c
|
||||
logger.info(f"PATTERN BLOCKED: {remote_name}/{path}")
|
||||
raise HTTPException(status_code=403, detail="Image not allowed by configuration patterns")
|
||||
|
||||
if remote_config.get("ban_tags_enabled", False):
|
||||
ban_tags = remote_config.get("ban_tags", [])
|
||||
if ban_tags:
|
||||
tag_match = re.search(r"/manifests/([^/]+)$", path)
|
||||
if tag_match:
|
||||
tag = tag_match.group(1)
|
||||
if not tag.startswith("sha256:") and tag in ban_tags:
|
||||
logger.info(f"TAG BANNED: {remote_name}/{path} (tag: {tag})")
|
||||
raise HTTPException(status_code=403, detail=f"Tag '{tag}' is not permitted on this remote")
|
||||
|
||||
base_url = remote_config.get("base_url", "").rstrip("/")
|
||||
remote_url = f"{base_url}/v2/{path}"
|
||||
|
||||
@@ -47,23 +58,39 @@ async def proxy(request: Request, remote_name: str, path: str, storage, cache, c
|
||||
if not await _proxy.handle_expired_mutable(remote_name, path, remote_url, config, cache, storage):
|
||||
cached_key = None
|
||||
|
||||
lock_acquired = False
|
||||
if not cached_key:
|
||||
lock_acquired = cache.acquire_fetch_lock(remote_name, path)
|
||||
if not lock_acquired:
|
||||
# Another pod is already fetching — poll storage briefly before issuing a duplicate upstream request
|
||||
for _ in range(10):
|
||||
await asyncio.sleep(0.5)
|
||||
probe_key = storage.get_object_key(remote_name, path)
|
||||
if storage.exists(probe_key):
|
||||
cached_key = probe_key
|
||||
break
|
||||
|
||||
if not cached_key:
|
||||
logger.info(f"Cache MISS: {remote_name}/{path} - fetching from remote: {remote_url}")
|
||||
result = await _proxy.cache_single_artifact(remote_url, remote_name, path, storage, remote_config)
|
||||
if result["status"] == "error":
|
||||
raise HTTPException(status_code=502, detail=f"Failed to fetch: {result['error']}")
|
||||
if result["status"] == "cached" and is_mutable:
|
||||
cache_config = config.get_cache_config(remote_name)
|
||||
mutable_ttl = cache_config.get("mutable_ttl", 3600)
|
||||
cache.mark_index_cached(remote_name, path, mutable_ttl)
|
||||
logger.info(f"Mutable file cached with TTL: {remote_name}/{path} (ttl: {mutable_ttl}s)")
|
||||
if result.get("etag") or result.get("last_modified"):
|
||||
cache.store_mutable_meta(remote_name, path, result.get("etag"), result.get("last_modified"))
|
||||
if not is_mutable:
|
||||
published = result.get("last_modified")
|
||||
if published:
|
||||
cache.store_artifact_published(remote_name, path, published)
|
||||
_proxy._check_quarantine(remote_name, published, config)
|
||||
try:
|
||||
result = await _proxy.cache_single_artifact(remote_url, remote_name, path, storage, remote_config)
|
||||
if result["status"] == "error":
|
||||
raise HTTPException(status_code=502, detail=f"Failed to fetch: {result['error']}")
|
||||
if result["status"] == "cached" and is_mutable:
|
||||
cache_config = config.get_cache_config(remote_name)
|
||||
mutable_ttl = cache_config.get("mutable_ttl", 3600)
|
||||
cache.mark_index_cached(remote_name, path, mutable_ttl)
|
||||
logger.info(f"Mutable file cached with TTL: {remote_name}/{path} (ttl: {mutable_ttl}s)")
|
||||
if result.get("etag") or result.get("last_modified"):
|
||||
cache.store_mutable_meta(remote_name, path, result.get("etag"), result.get("last_modified"))
|
||||
if not is_mutable:
|
||||
published = result.get("last_modified")
|
||||
if published:
|
||||
cache.store_artifact_published(remote_name, path, published)
|
||||
_proxy._check_quarantine(remote_name, published, config)
|
||||
finally:
|
||||
if lock_acquired:
|
||||
cache.release_fetch_lock(remote_name, path)
|
||||
elif not is_mutable:
|
||||
published = cache.get_artifact_published(remote_name, path)
|
||||
if not published:
|
||||
@@ -90,6 +117,14 @@ async def proxy(request: Request, remote_name: str, path: str, storage, cache, c
|
||||
content_type = "application/vnd.oci.image.manifest.v1+json"
|
||||
|
||||
digest = f"sha256:{hashlib.sha256(artifact_data).hexdigest()}"
|
||||
|
||||
# Cross-link tag manifests to their sha256 digest key so digest-addressed pulls hit cache
|
||||
if is_mutable and "/manifests/" in path:
|
||||
digest_path = re.sub(r"/manifests/[^/]+$", f"/manifests/{digest}", path)
|
||||
digest_key = storage.get_object_key(remote_name, digest_path)
|
||||
if not storage.exists(digest_key):
|
||||
storage.upload(digest_key, artifact_data)
|
||||
|
||||
headers = {
|
||||
"Docker-Distribution-Api-Version": "registry/2.0",
|
||||
"Docker-Content-Digest": digest,
|
||||
|
||||
@@ -11,7 +11,9 @@ from fastapi import HTTPException, Request, Response
|
||||
from ..auth import get_docker_token_for_response
|
||||
from ..remote import helm as _helm
|
||||
from ..remote import npm as _npm
|
||||
from ..remote import puppet as _puppet
|
||||
from ..remote import python as _pypi
|
||||
from ..remote import terraform as _terraform
|
||||
from ..remote.base import get_content_type
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -84,6 +86,11 @@ def _resolve_content(
|
||||
return _npm.resolve_content(data, path, filename, remote_config.get("immutable_patterns", []), base_url, proxy_base, remote_name)
|
||||
if package == "helm":
|
||||
return _helm.resolve_content(data, path, filename, base_url, proxy_base, remote_name)
|
||||
if package == "puppet":
|
||||
return _puppet.resolve_content(data, path, filename, base_url, proxy_base, remote_name)
|
||||
if package == "terraform":
|
||||
releases_remote = remote_config.get("releases_remote")
|
||||
return _terraform.resolve_content(data, path, filename, base_url, proxy_base, remote_name, releases_remote)
|
||||
return data, get_content_type(filename)
|
||||
|
||||
|
||||
@@ -93,6 +100,8 @@ def construct_url(remote_config: dict, path: str) -> str:
|
||||
return f"{base_url}/v2/{path}"
|
||||
if remote_config.get("package") == "pypi":
|
||||
return _pypi.construct_url(base_url, path)
|
||||
if remote_config.get("package") == "terraform":
|
||||
return _terraform.construct_url(base_url, path)
|
||||
return f"{base_url}/{path}"
|
||||
|
||||
|
||||
|
||||
Vendored
+19
@@ -99,6 +99,25 @@ class RedisCache:
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def acquire_fetch_lock(self, remote_name: str, path: str, ttl: int = 30) -> bool:
|
||||
"""Try to acquire a short-lived fetch lock. Returns True if acquired, False if held by another caller."""
|
||||
if not self.available:
|
||||
return True # fail open: no Redis → behave as if we always hold the lock
|
||||
key = f"fetchlock:{remote_name}:{hashlib.sha256(path.encode()).hexdigest()[:16]}"
|
||||
try:
|
||||
return bool(self.client.set(key, 1, nx=True, ex=ttl))
|
||||
except Exception:
|
||||
return True
|
||||
|
||||
def release_fetch_lock(self, remote_name: str, path: str) -> None:
|
||||
if not self.available:
|
||||
return
|
||||
key = f"fetchlock:{remote_name}:{hashlib.sha256(path.encode()).hexdigest()[:16]}"
|
||||
try:
|
||||
self.client.delete(key)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def cleanup_expired_index(self, storage, remote_name: str, path: str) -> None:
|
||||
if not self.available:
|
||||
return
|
||||
|
||||
@@ -26,6 +26,13 @@ _PACKAGE_MUTABLE_PATTERNS: dict[str, list[str]] = {
|
||||
"helm": [
|
||||
r"index\.yaml$",
|
||||
],
|
||||
"puppet": [
|
||||
r"^v3/modules/",
|
||||
r"^v3/releases",
|
||||
],
|
||||
"terraform": [
|
||||
r"[^/]+/[^/]+/versions$",
|
||||
],
|
||||
"generic": [],
|
||||
}
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
from . import generic, helm, npm, python, rpm
|
||||
from . import generic, helm, npm, puppet, python, rpm, terraform
|
||||
from .base import get_content_type
|
||||
|
||||
__all__ = ["generic", "helm", "npm", "python", "rpm", "get_content_type"]
|
||||
__all__ = ["generic", "helm", "npm", "puppet", "python", "rpm", "terraform", "get_content_type"]
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
from .base import get_content_type
|
||||
|
||||
|
||||
def resolve_content(
|
||||
data: bytes,
|
||||
path: str,
|
||||
filename: str,
|
||||
base_url: str,
|
||||
proxy_url: str,
|
||||
remote_name: str,
|
||||
) -> tuple[bytes, str]:
|
||||
if not path.startswith("v3/files/"):
|
||||
proxy_remote_url = f"{proxy_url}/api/v1/remote/{remote_name}"
|
||||
# Rewrite any absolute forge API URLs
|
||||
data = data.replace(base_url.encode(), proxy_remote_url.encode())
|
||||
# Rewrite relative file_uri paths ("/v3/files/...") to absolute proxy URLs.
|
||||
# g10k resolves file_uri against only the forge host, so a relative path
|
||||
# would drop our /api/v1/remote/<name> prefix.
|
||||
data = data.replace(
|
||||
b'"/v3/files/',
|
||||
f'"{proxy_remote_url}/v3/files/'.encode(),
|
||||
)
|
||||
return data, "application/json"
|
||||
return data, get_content_type(filename)
|
||||
@@ -0,0 +1,36 @@
|
||||
import json
|
||||
import re
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from .base import get_content_type
|
||||
|
||||
_DOWNLOAD_PATH = re.compile(r"^[^/]+/[^/]+/[^/]+/download/[^/]+/[^/]+$")
|
||||
|
||||
|
||||
def construct_url(base_url: str, path: str) -> str:
|
||||
return f"{base_url}/v1/providers/{path}"
|
||||
|
||||
|
||||
def resolve_content(
|
||||
data: bytes,
|
||||
path: str,
|
||||
filename: str,
|
||||
_base_url: str,
|
||||
proxy_url: str,
|
||||
_remote_name: str,
|
||||
releases_remote: str | None = None,
|
||||
) -> tuple[bytes, str]:
|
||||
if filename.endswith((".zip", ".sig")):
|
||||
return data, get_content_type(filename)
|
||||
if releases_remote and _DOWNLOAD_PATH.match(path):
|
||||
releases_proxy = f"{proxy_url}/api/v1/remote/{releases_remote}"
|
||||
try:
|
||||
obj = json.loads(data)
|
||||
for field in ("download_url", "shasums_url", "shasums_signature_url"):
|
||||
if field in obj:
|
||||
parsed = urlparse(obj[field])
|
||||
obj[field] = f"{releases_proxy}{parsed.path}"
|
||||
data = json.dumps(obj).encode()
|
||||
except (json.JSONDecodeError, KeyError):
|
||||
pass
|
||||
return data, "application/json"
|
||||
@@ -41,6 +41,13 @@ TEST_REMOTES = {
|
||||
"immutable_patterns": ["^library/nginx"],
|
||||
"cache": {"immutable_ttl": 0, "mutable_ttl": 300},
|
||||
},
|
||||
"docker-bantags-test": {
|
||||
"base_url": "https://registry.example.com",
|
||||
"package": "docker",
|
||||
"ban_tags_enabled": True,
|
||||
"ban_tags": ["latest", "edge"],
|
||||
"cache": {"immutable_ttl": 0, "mutable_ttl": 300},
|
||||
},
|
||||
"generic-test": {
|
||||
"base_url": "https://releases.example.com",
|
||||
"package": "generic",
|
||||
@@ -105,6 +112,27 @@ TEST_REMOTES = {
|
||||
"immutable_patterns": [r"\.tgz$"],
|
||||
"cache": {"immutable_ttl": 0, "mutable_ttl": 1800},
|
||||
},
|
||||
"puppet-test": {
|
||||
"base_url": "https://forgeapi.puppet.com",
|
||||
"package": "puppet",
|
||||
"immutable_patterns": [r"^v3/files/.*\.tar\.gz$"],
|
||||
"cache": {"immutable_ttl": 0, "mutable_ttl": 600},
|
||||
},
|
||||
"terraform-registry-test": {
|
||||
"base_url": "https://registry.terraform.io",
|
||||
"package": "terraform",
|
||||
"immutable_patterns": [
|
||||
r"[^/]+/[^/]+/[^/]+/download/[^/]+/[^/]+$",
|
||||
],
|
||||
"releases_remote": "hashicorp-releases-test",
|
||||
"cache": {"immutable_ttl": 0, "mutable_ttl": 300},
|
||||
},
|
||||
"hashicorp-releases-test": {
|
||||
"base_url": "https://releases.hashicorp.com",
|
||||
"package": "generic",
|
||||
"immutable_patterns": [r".*\.zip$", r".*SHA256SUMS(\.sig)?$"],
|
||||
"cache": {"immutable_ttl": 0, "mutable_ttl": 0},
|
||||
},
|
||||
},
|
||||
"locals": {
|
||||
"local-test": {
|
||||
|
||||
@@ -327,3 +327,74 @@ class TestArtifactPublished:
|
||||
|
||||
def test_get_returns_none_when_unavailable(self, unavailable_cache):
|
||||
assert unavailable_cache.get_artifact_published("remote", "path") is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# fetch lock (thundering-herd deduplication)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestFetchLock:
|
||||
def test_acquire_returns_true_when_lock_obtained(self, cache_with_redis, mock_redis_client):
|
||||
mock_redis_client.set.return_value = True
|
||||
result = cache_with_redis.acquire_fetch_lock("myremote", "library/nginx/manifests/latest")
|
||||
assert result is True
|
||||
|
||||
def test_acquire_calls_set_nx_with_ttl(self, cache_with_redis, mock_redis_client):
|
||||
mock_redis_client.set.return_value = True
|
||||
cache_with_redis.acquire_fetch_lock("myremote", "library/nginx/manifests/latest", ttl=15)
|
||||
_, kwargs = mock_redis_client.set.call_args
|
||||
assert kwargs.get("nx") is True
|
||||
assert kwargs.get("ex") == 15
|
||||
|
||||
def test_acquire_returns_false_when_lock_already_held(self, cache_with_redis, mock_redis_client):
|
||||
mock_redis_client.set.return_value = None # Redis SET NX → None when key exists
|
||||
result = cache_with_redis.acquire_fetch_lock("myremote", "library/nginx/manifests/latest")
|
||||
assert result is False
|
||||
|
||||
def test_acquire_fails_open_when_unavailable(self, unavailable_cache):
|
||||
# caller must be allowed to proceed when Redis is down
|
||||
assert unavailable_cache.acquire_fetch_lock("myremote", "some/path") is True
|
||||
|
||||
def test_acquire_fails_open_on_redis_exception(self, cache_with_redis, mock_redis_client):
|
||||
mock_redis_client.set.side_effect = Exception("connection reset")
|
||||
assert cache_with_redis.acquire_fetch_lock("myremote", "some/path") is True
|
||||
|
||||
def test_lock_key_embeds_path_hash(self, cache_with_redis, mock_redis_client):
|
||||
mock_redis_client.set.return_value = True
|
||||
path = "library/nginx/manifests/latest"
|
||||
cache_with_redis.acquire_fetch_lock("myremote", path)
|
||||
args, _ = mock_redis_client.set.call_args
|
||||
expected_hash = hashlib.sha256(path.encode()).hexdigest()[:16]
|
||||
assert args[0] == f"fetchlock:myremote:{expected_hash}"
|
||||
|
||||
def test_lock_key_hash_is_16_chars(self, cache_with_redis, mock_redis_client):
|
||||
mock_redis_client.set.return_value = True
|
||||
cache_with_redis.acquire_fetch_lock("myremote", "some/long/path/file.tar.gz")
|
||||
args, _ = mock_redis_client.set.call_args
|
||||
# key format: fetchlock:<remote>:<16-char hash>
|
||||
parts = args[0].split(":")
|
||||
assert len(parts) == 3
|
||||
assert len(parts[2]) == 16
|
||||
|
||||
def test_different_paths_produce_different_lock_keys(self, cache_with_redis, mock_redis_client):
|
||||
mock_redis_client.set.return_value = True
|
||||
cache_with_redis.acquire_fetch_lock("myremote", "path/a/manifests/latest")
|
||||
key_a = mock_redis_client.set.call_args[0][0]
|
||||
mock_redis_client.set.reset_mock()
|
||||
cache_with_redis.acquire_fetch_lock("myremote", "path/b/manifests/latest")
|
||||
key_b = mock_redis_client.set.call_args[0][0]
|
||||
assert key_a != key_b
|
||||
|
||||
def test_release_deletes_correct_key(self, cache_with_redis, mock_redis_client):
|
||||
path = "library/nginx/manifests/latest"
|
||||
cache_with_redis.release_fetch_lock("myremote", path)
|
||||
expected_hash = hashlib.sha256(path.encode()).hexdigest()[:16]
|
||||
mock_redis_client.delete.assert_called_once_with(f"fetchlock:myremote:{expected_hash}")
|
||||
|
||||
def test_release_no_op_when_unavailable(self, unavailable_cache):
|
||||
unavailable_cache.release_fetch_lock("myremote", "some/path") # must not raise
|
||||
|
||||
def test_release_no_op_on_redis_exception(self, cache_with_redis, mock_redis_client):
|
||||
mock_redis_client.delete.side_effect = Exception("timeout")
|
||||
cache_with_redis.release_fetch_lock("myremote", "some/path") # must not raise
|
||||
|
||||
@@ -260,6 +260,211 @@ class TestDockerProxy:
|
||||
mock_fetch.assert_called_once()
|
||||
assert response.status_code == 200
|
||||
|
||||
# --- Issue 1: sha256 digest cross-linking ---
|
||||
|
||||
def test_tag_manifest_is_stored_under_digest_key_on_cache_hit(self, client, patched_deps):
|
||||
# When serving a cached tag manifest the handler must also write the content
|
||||
# under the sha256 digest key so subsequent sha256-addressed pulls hit cache.
|
||||
deps = patched_deps
|
||||
manifest = json.dumps({"mediaType": "application/vnd.oci.image.manifest.v1+json", "layers": []}).encode()
|
||||
# First exists call (tag manifest): hit. Second (digest key): miss → triggers upload.
|
||||
deps["storage"].exists.side_effect = [True, False]
|
||||
deps["storage"].download_object.return_value = manifest
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
deps["cache"].is_index_valid.return_value = True
|
||||
|
||||
response = client.get("/v2/docker-test/library/nginx/manifests/v1.25.3")
|
||||
|
||||
assert response.status_code == 200
|
||||
deps["storage"].upload.assert_called_once_with(deps["storage"].get_object_key.return_value, manifest)
|
||||
|
||||
def test_tag_manifest_digest_key_not_written_when_already_exists(self, client, patched_deps):
|
||||
# When the digest key already exists in storage upload must not be called.
|
||||
deps = patched_deps
|
||||
manifest = json.dumps({"mediaType": "application/vnd.oci.image.manifest.v1+json", "layers": []}).encode()
|
||||
# Both the tag key and the digest key already present.
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = manifest
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
deps["cache"].is_index_valid.return_value = True
|
||||
|
||||
client.get("/v2/docker-test/library/nginx/manifests/v1.25.3")
|
||||
|
||||
deps["storage"].upload.assert_not_called()
|
||||
|
||||
def test_sha256_manifest_request_is_not_cross_linked(self, client, patched_deps):
|
||||
# sha256-addressed manifests are immutable — the cross-link logic must not apply.
|
||||
deps = patched_deps
|
||||
manifest = json.dumps({"mediaType": "application/vnd.oci.image.manifest.v1+json", "layers": []}).encode()
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = manifest
|
||||
deps["cache"].is_mutable_file.return_value = False # sha256 manifest is immutable
|
||||
|
||||
with patch("artifactapi.artifact.proxy._fetch_last_modified", new_callable=AsyncMock, return_value=None):
|
||||
client.get("/v2/docker-test/library/nginx/manifests/sha256:" + "a" * 64)
|
||||
|
||||
deps["storage"].upload.assert_not_called()
|
||||
|
||||
# --- Issue 2: thundering herd distributed lock ---
|
||||
|
||||
def test_lock_acquired_and_released_on_upstream_fetch(self, client, patched_deps):
|
||||
deps = patched_deps
|
||||
manifest = json.dumps({"mediaType": "application/vnd.oci.image.manifest.v1+json", "layers": []}).encode()
|
||||
deps["storage"].exists.side_effect = [False, False] # initial miss; digest key also absent
|
||||
deps["storage"].download_object.return_value = manifest
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
deps["cache"].acquire_fetch_lock.return_value = True
|
||||
|
||||
with patch(
|
||||
"artifactapi.artifact.proxy.cache_single_artifact",
|
||||
new_callable=AsyncMock,
|
||||
return_value={"status": "cached"},
|
||||
):
|
||||
response = client.get("/v2/docker-test/library/nginx/manifests/latest")
|
||||
|
||||
deps["cache"].acquire_fetch_lock.assert_called_once()
|
||||
deps["cache"].release_fetch_lock.assert_called_once()
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_lock_released_even_when_fetch_returns_error(self, client, patched_deps):
|
||||
deps = patched_deps
|
||||
deps["storage"].exists.return_value = False
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
deps["cache"].acquire_fetch_lock.return_value = True
|
||||
|
||||
with patch(
|
||||
"artifactapi.artifact.proxy.cache_single_artifact",
|
||||
new_callable=AsyncMock,
|
||||
return_value={"status": "error", "error": "upstream down"},
|
||||
):
|
||||
response = client.get("/v2/docker-test/library/nginx/manifests/latest")
|
||||
|
||||
deps["cache"].release_fetch_lock.assert_called_once()
|
||||
assert response.status_code == 502
|
||||
|
||||
def test_thundering_herd_polls_storage_when_lock_not_acquired(self, client, patched_deps):
|
||||
# When the lock is held by another pod the handler must poll storage and serve
|
||||
# from cache once the competing fetch completes, without issuing its own upstream request.
|
||||
deps = patched_deps
|
||||
manifest = json.dumps({"mediaType": "application/vnd.oci.image.manifest.v1+json", "layers": []}).encode()
|
||||
# Initial cache check: miss. First poll iteration: another pod has written it.
|
||||
# Third call is for the digest cross-link check (is_mutable=True path); digest key exists.
|
||||
deps["storage"].exists.side_effect = [False, True, True]
|
||||
deps["storage"].download_object.return_value = manifest
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
deps["cache"].is_index_valid.return_value = True
|
||||
deps["cache"].acquire_fetch_lock.return_value = False # lock held by peer
|
||||
|
||||
with patch("artifactapi.artifact.docker.asyncio.sleep", new_callable=AsyncMock):
|
||||
with patch(
|
||||
"artifactapi.artifact.proxy.cache_single_artifact",
|
||||
new_callable=AsyncMock,
|
||||
) as mock_fetch:
|
||||
response = client.get("/v2/docker-test/library/nginx/manifests/latest")
|
||||
|
||||
mock_fetch.assert_not_called()
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_thundering_herd_falls_through_to_fetch_if_poll_times_out(self, client, patched_deps):
|
||||
# If the item never appears in storage during the poll window the handler must
|
||||
# still issue its own upstream fetch as a fallback.
|
||||
deps = patched_deps
|
||||
manifest = json.dumps({"mediaType": "application/vnd.oci.image.manifest.v1+json", "layers": []}).encode()
|
||||
# All exists calls return False — item never appears during polling.
|
||||
deps["storage"].exists.return_value = False
|
||||
deps["storage"].download_object.return_value = manifest
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
deps["cache"].acquire_fetch_lock.return_value = False # lock held by peer
|
||||
|
||||
with patch("artifactapi.artifact.docker.asyncio.sleep", new_callable=AsyncMock):
|
||||
with patch(
|
||||
"artifactapi.artifact.proxy.cache_single_artifact",
|
||||
new_callable=AsyncMock,
|
||||
return_value={"status": "cached"},
|
||||
) as mock_fetch:
|
||||
response = client.get("/v2/docker-test/library/nginx/manifests/latest")
|
||||
|
||||
mock_fetch.assert_called_once()
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Docker ban_tags feature
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestDockerBanTags:
|
||||
def test_banned_tag_returns_403(self, client, patched_deps):
|
||||
response = client.get("/v2/docker-bantags-test/library/nginx/manifests/latest")
|
||||
assert response.status_code == 403
|
||||
assert "latest" in response.json()["detail"]
|
||||
|
||||
def test_second_banned_tag_returns_403(self, client, patched_deps):
|
||||
response = client.get("/v2/docker-bantags-test/library/nginx/manifests/edge")
|
||||
assert response.status_code == 403
|
||||
assert "edge" in response.json()["detail"]
|
||||
|
||||
def test_allowed_tag_proceeds(self, client, patched_deps):
|
||||
deps = patched_deps
|
||||
manifest = json.dumps({"mediaType": "application/vnd.oci.image.manifest.v1+json", "layers": []}).encode()
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = manifest
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
deps["cache"].is_index_valid.return_value = True
|
||||
|
||||
response = client.get("/v2/docker-bantags-test/library/nginx/manifests/1.25.3")
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_digest_pull_bypasses_ban(self, client, patched_deps):
|
||||
# sha256-addressed pulls must never be blocked by the tag ban list
|
||||
deps = patched_deps
|
||||
manifest = json.dumps({"mediaType": "application/vnd.oci.image.manifest.v1+json", "layers": []}).encode()
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = manifest
|
||||
deps["cache"].is_mutable_file.return_value = False
|
||||
|
||||
digest = "sha256:" + "a" * 64
|
||||
with patch("artifactapi.artifact.proxy._fetch_last_modified", new_callable=AsyncMock, return_value=None):
|
||||
response = client.get(f"/v2/docker-bantags-test/library/nginx/manifests/{digest}")
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_ban_tags_disabled_by_default(self, client, patched_deps):
|
||||
# docker-test has no ban_tags_enabled — "latest" must pass through
|
||||
deps = patched_deps
|
||||
manifest = json.dumps({"mediaType": "application/vnd.oci.image.manifest.v1+json", "layers": []}).encode()
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = manifest
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
deps["cache"].is_index_valid.return_value = True
|
||||
|
||||
response = client.get("/v2/docker-test/library/nginx/manifests/latest")
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_ban_tags_enabled_but_empty_list_allows_all(self, client, patched_deps):
|
||||
# If ban_tags_enabled is true but ban_tags is empty nothing should be blocked.
|
||||
# docker-test doesn't have ban_tags_enabled, but we can verify via the
|
||||
# docker-bantags-test remote with an unlisted tag.
|
||||
deps = patched_deps
|
||||
manifest = json.dumps({"mediaType": "application/vnd.oci.image.manifest.v1+json", "layers": []}).encode()
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = manifest
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
deps["cache"].is_index_valid.return_value = True
|
||||
|
||||
response = client.get("/v2/docker-bantags-test/library/nginx/manifests/stable")
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_ban_check_does_not_apply_to_blobs(self, client, patched_deps):
|
||||
# Blob paths don't contain /manifests/ — the ban check must not interfere
|
||||
deps = patched_deps
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = b"\x00" * 100
|
||||
deps["cache"].is_mutable_file.return_value = False
|
||||
|
||||
with patch("artifactapi.artifact.proxy._fetch_last_modified", new_callable=AsyncMock, return_value=None):
|
||||
response = client.get("/v2/docker-bantags-test/library/nginx/blobs/sha256:" + "b" * 64)
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Generic artifact route /api/v1/remote/{remote}/{path}
|
||||
@@ -912,6 +1117,259 @@ class TestHelmRemote:
|
||||
assert response.status_code == 403
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Puppet Forge remote /api/v1/remote/puppet-test/...
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPuppetRemote:
|
||||
def test_module_metadata_is_mutable(self, client, patched_deps):
|
||||
"""v3/modules/ paths are detected as mutable (package-type default)."""
|
||||
deps = patched_deps
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = b'{"current_release":{"file_uri":"/v3/files/puppetlabs-stdlib-9.7.0.tar.gz"}}'
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
deps["cache"].is_index_valid.return_value = True
|
||||
|
||||
response = client.get("/api/v1/remote/puppet-test/v3/modules/puppetlabs-stdlib")
|
||||
assert response.status_code == 200
|
||||
deps["cache"].mark_index_cached.assert_not_called()
|
||||
|
||||
def test_releases_path_is_mutable(self, client, patched_deps):
|
||||
"""v3/releases paths are detected as mutable (package-type default)."""
|
||||
deps = patched_deps
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = b'{"file_uri":"/v3/files/puppetlabs-stdlib-9.7.0.tar.gz"}'
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
deps["cache"].is_index_valid.return_value = True
|
||||
|
||||
response = client.get("/api/v1/remote/puppet-test/v3/releases/puppetlabs-stdlib-9.7.0")
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_relative_file_uri_rewritten_to_absolute_proxy_url(self, client, patched_deps):
|
||||
"""Relative /v3/files/ paths in JSON responses are rewritten to absolute proxy URLs."""
|
||||
deps = patched_deps
|
||||
meta = b'{"current_release":{"file_uri":"/v3/files/puppetlabs-stdlib-9.7.0.tar.gz","version":"9.7.0"}}'
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = meta
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
deps["cache"].is_index_valid.return_value = True
|
||||
|
||||
response = client.get("/api/v1/remote/puppet-test/v3/modules/puppetlabs-stdlib")
|
||||
assert response.status_code == 200
|
||||
assert b'"/v3/files/' not in response.content
|
||||
assert b"/api/v1/remote/puppet-test/v3/files/puppetlabs-stdlib-9.7.0.tar.gz" in response.content
|
||||
|
||||
def test_absolute_forge_url_rewritten_to_proxy(self, client, patched_deps):
|
||||
"""Absolute forgeapi.puppet.com URLs in JSON are rewritten to the proxy URL."""
|
||||
deps = patched_deps
|
||||
meta = b'{"uri":"https://forgeapi.puppet.com/v3/modules/puppetlabs-stdlib"}'
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = meta
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
deps["cache"].is_index_valid.return_value = True
|
||||
|
||||
response = client.get("/api/v1/remote/puppet-test/v3/modules/puppetlabs-stdlib")
|
||||
assert response.status_code == 200
|
||||
assert b"forgeapi.puppet.com" not in response.content
|
||||
assert b"/api/v1/remote/puppet-test" in response.content
|
||||
|
||||
def test_metadata_content_type_is_json(self, client, patched_deps):
|
||||
deps = patched_deps
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = b'{"current_release":{}}'
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
deps["cache"].is_index_valid.return_value = True
|
||||
|
||||
response = client.get("/api/v1/remote/puppet-test/v3/modules/puppetlabs-concat")
|
||||
assert response.status_code == 200
|
||||
assert "application/json" in response.headers["content-type"]
|
||||
|
||||
def test_tarball_served_without_rewriting(self, client, patched_deps):
|
||||
"""Module tarballs (v3/files/*.tar.gz) are served as binary without URL rewriting."""
|
||||
deps = patched_deps
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = b"\x1f\x8b tarball bytes"
|
||||
deps["cache"].is_mutable_file.return_value = False
|
||||
|
||||
response = client.get("/api/v1/remote/puppet-test/v3/files/puppetlabs-stdlib-9.7.0.tar.gz")
|
||||
assert response.status_code == 200
|
||||
assert "application/gzip" in response.headers["content-type"]
|
||||
assert response.headers["X-Artifact-Source"] == "cache"
|
||||
|
||||
def test_tarball_not_blocked_by_immutable_pattern(self, client, patched_deps):
|
||||
"""v3/files/*.tar.gz matches the configured immutable_patterns and is allowed."""
|
||||
deps = patched_deps
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = b"\x1f\x8b tarball bytes"
|
||||
deps["cache"].is_mutable_file.return_value = False
|
||||
|
||||
response = client.get("/api/v1/remote/puppet-test/v3/files/puppetlabs-inifile-6.2.0.tar.gz")
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_unknown_path_blocked(self, client, patched_deps):
|
||||
"""Paths outside v3/modules, v3/releases, and v3/files are blocked."""
|
||||
deps = patched_deps
|
||||
deps["cache"].is_mutable_file.return_value = False
|
||||
|
||||
response = client.get("/api/v1/remote/puppet-test/v3/users/puppetlabs")
|
||||
assert response.status_code == 403
|
||||
|
||||
def test_metadata_cache_miss_fetches_upstream(self, client, patched_deps):
|
||||
deps = patched_deps
|
||||
meta = b'{"current_release":{"file_uri":"/v3/files/puppetlabs-stdlib-9.7.0.tar.gz"}}'
|
||||
deps["storage"].exists.return_value = False
|
||||
deps["storage"].download_object.return_value = meta
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
|
||||
with patch(
|
||||
"artifactapi.artifact.proxy.cache_single_artifact",
|
||||
new_callable=AsyncMock,
|
||||
return_value={"status": "cached"},
|
||||
) as mock_fetch:
|
||||
response = client.get("/api/v1/remote/puppet-test/v3/modules/puppetlabs-stdlib")
|
||||
|
||||
mock_fetch.assert_called_once()
|
||||
assert response.status_code == 200
|
||||
assert b'"/v3/files/' not in response.content
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Terraform registry remote (terraform-registry-test)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestTerraformRemote:
|
||||
def test_versions_path_is_mutable(self, client, patched_deps):
|
||||
"""Provider versions listing is detected as mutable."""
|
||||
deps = patched_deps
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = b'{"versions":[]}'
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
deps["cache"].is_index_valid.return_value = True
|
||||
|
||||
response = client.get("/api/v1/remote/terraform-registry-test/hashicorp/vault/versions")
|
||||
assert response.status_code == 200
|
||||
deps["cache"].mark_index_cached.assert_not_called()
|
||||
|
||||
def test_versions_returns_json_content_type(self, client, patched_deps):
|
||||
deps = patched_deps
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = b'{"versions":[]}'
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
deps["cache"].is_index_valid.return_value = True
|
||||
|
||||
response = client.get("/api/v1/remote/terraform-registry-test/hashicorp/vault/versions")
|
||||
assert response.status_code == 200
|
||||
assert "application/json" in response.headers["content-type"]
|
||||
|
||||
def test_download_info_download_url_rewritten(self, client, patched_deps):
|
||||
"""download_url in download-info JSON is rewritten to point to the releases proxy."""
|
||||
deps = patched_deps
|
||||
download_info = json.dumps({
|
||||
"os": "linux",
|
||||
"arch": "amd64",
|
||||
"filename": "terraform-provider-vault_0.28.0_linux_amd64.zip",
|
||||
"download_url": "https://releases.hashicorp.com/terraform-provider-vault/0.28.0/terraform-provider-vault_0.28.0_linux_amd64.zip",
|
||||
"shasums_url": "https://releases.hashicorp.com/terraform-provider-vault/0.28.0/terraform-provider-vault_0.28.0_SHA256SUMS",
|
||||
"shasums_signature_url": "https://releases.hashicorp.com/terraform-provider-vault/0.28.0/terraform-provider-vault_0.28.0_SHA256SUMS.sig",
|
||||
}).encode()
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = download_info
|
||||
deps["cache"].is_mutable_file.return_value = False
|
||||
|
||||
response = client.get("/api/v1/remote/terraform-registry-test/hashicorp/vault/0.28.0/download/linux/amd64")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "releases.hashicorp.com" not in data["download_url"]
|
||||
assert "/api/v1/remote/hashicorp-releases-test/" in data["download_url"]
|
||||
|
||||
def test_download_info_shasums_url_rewritten(self, client, patched_deps):
|
||||
"""shasums_url is also rewritten to the releases proxy."""
|
||||
deps = patched_deps
|
||||
download_info = json.dumps({
|
||||
"os": "linux",
|
||||
"arch": "amd64",
|
||||
"filename": "terraform-provider-vault_0.28.0_linux_amd64.zip",
|
||||
"download_url": "https://releases.hashicorp.com/terraform-provider-vault/0.28.0/terraform-provider-vault_0.28.0_linux_amd64.zip",
|
||||
"shasums_url": "https://releases.hashicorp.com/terraform-provider-vault/0.28.0/terraform-provider-vault_0.28.0_SHA256SUMS",
|
||||
"shasums_signature_url": "https://releases.hashicorp.com/terraform-provider-vault/0.28.0/terraform-provider-vault_0.28.0_SHA256SUMS.sig",
|
||||
}).encode()
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = download_info
|
||||
deps["cache"].is_mutable_file.return_value = False
|
||||
|
||||
response = client.get("/api/v1/remote/terraform-registry-test/hashicorp/vault/0.28.0/download/linux/amd64")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "/api/v1/remote/hashicorp-releases-test/" in data["shasums_url"]
|
||||
assert "/api/v1/remote/hashicorp-releases-test/" in data["shasums_signature_url"]
|
||||
assert "releases.hashicorp.com" not in data["shasums_url"]
|
||||
assert "releases.hashicorp.com" not in data["shasums_signature_url"]
|
||||
|
||||
def test_download_info_path_preserved(self, client, patched_deps):
|
||||
"""The path portion of the upstream URL is preserved when rewriting."""
|
||||
deps = patched_deps
|
||||
zip_path = "/terraform-provider-vault/0.28.0/terraform-provider-vault_0.28.0_linux_amd64.zip"
|
||||
download_info = json.dumps({
|
||||
"download_url": f"https://releases.hashicorp.com{zip_path}",
|
||||
"shasums_url": "https://releases.hashicorp.com/terraform-provider-vault/0.28.0/terraform-provider-vault_0.28.0_SHA256SUMS",
|
||||
"shasums_signature_url": "https://releases.hashicorp.com/terraform-provider-vault/0.28.0/terraform-provider-vault_0.28.0_SHA256SUMS.sig",
|
||||
}).encode()
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = download_info
|
||||
deps["cache"].is_mutable_file.return_value = False
|
||||
|
||||
response = client.get("/api/v1/remote/terraform-registry-test/hashicorp/vault/0.28.0/download/linux/amd64")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["download_url"].endswith(zip_path)
|
||||
|
||||
def test_zip_served_as_binary(self, client, patched_deps):
|
||||
"""Provider zip files are served as binary without JSON rewriting."""
|
||||
deps = patched_deps
|
||||
deps["storage"].exists.return_value = True
|
||||
deps["storage"].download_object.return_value = b"PK\x03\x04 zip bytes"
|
||||
deps["cache"].is_mutable_file.return_value = False
|
||||
|
||||
response = client.get("/api/v1/remote/hashicorp-releases-test/terraform-provider-vault/0.28.0/terraform-provider-vault_0.28.0_linux_amd64.zip")
|
||||
assert response.status_code == 200
|
||||
assert response.headers["X-Artifact-Source"] == "cache"
|
||||
|
||||
def test_construct_url_prepends_v1_providers(self, client, patched_deps):
|
||||
"""Upstream URL for the terraform package type prepends /v1/providers/."""
|
||||
deps = patched_deps
|
||||
deps["storage"].exists.return_value = False
|
||||
|
||||
with patch(
|
||||
"artifactapi.artifact.proxy.cache_single_artifact",
|
||||
new_callable=AsyncMock,
|
||||
return_value={"status": "cached"},
|
||||
) as mock_fetch:
|
||||
deps["storage"].download_object.return_value = b'{"versions":[]}'
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
client.get("/api/v1/remote/terraform-registry-test/hashicorp/vault/versions")
|
||||
|
||||
called_url = mock_fetch.call_args[0][0]
|
||||
assert called_url == "https://registry.terraform.io/v1/providers/hashicorp/vault/versions"
|
||||
|
||||
def test_versions_cache_miss_fetches_upstream(self, client, patched_deps):
|
||||
deps = patched_deps
|
||||
deps["storage"].exists.return_value = False
|
||||
deps["storage"].download_object.return_value = b'{"versions":[]}'
|
||||
deps["cache"].is_mutable_file.return_value = True
|
||||
|
||||
with patch(
|
||||
"artifactapi.artifact.proxy.cache_single_artifact",
|
||||
new_callable=AsyncMock,
|
||||
return_value={"status": "cached"},
|
||||
) as mock_fetch:
|
||||
response = client.get("/api/v1/remote/terraform-registry-test/hashicorp/vault/versions")
|
||||
|
||||
mock_fetch.assert_called_once()
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Quarantine (quarantine-test remote: quarantine_new=True, quarantine_days=3)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
Reference in New Issue
Block a user