Add support for remote-specific S3 object deletion using hierarchical key prefixes.
When a remote parameter is provided, only objects under that remote's hierarchy
(e.g., 'fedora/*') are deleted, preserving cached files for other remotes.
Key improvements:
- Use S3 list_objects_v2 Prefix parameter for targeted deletion
- Enhanced logging to indicate scope of cache clearing operations
- Backward compatible - existing behavior preserved when no remote specified
Resolves artifactapi-u46
This commit introduces two major improvements:
1. **Hierarchical S3 Key Structure**:
- Replace URL-based hashing with remote-name/hash(directory_path)/filename format
- Enables remote-specific cache operations and intuitive S3 organization
- Cache keys now independent of mirror URL changes
- Example: fedora/886d215f6d1a0108/eccodes-2.44.0-1.fc42.x86_64.rpm
2. **Automated Version Management**:
- Add bumpver for semantic version bumping
- Single source of truth in pyproject.toml
- FastAPI dynamically reads version from package metadata
- Eliminates manual version synchronization between files
Changes:
- storage.py: New get_object_key(remote_name, path) method with directory hashing
- main.py: Dynamic version import and updated cache key generation calls
- cache.py: Updated to use new hierarchical key structure
- pyproject.toml: Added bumpver config and dev dependency
Breaking change: S3 key format changed, existing cache will need regeneration
- Add PUT /cache/flush endpoint for selective cache management
- Fix critical cache key mismatch in cleanup_expired_index()
- Update index file detection to use full path instead of filename
- Bump version to 2.0.2
The key changes include adding a comprehensive cache flush API that supports selective flushing by cache type (index, files, metrics) and fixing a critical bug where cache keys were
inconsistent between storage and cleanup operations, preventing proper cache invalidation.
- Add support for custom CA bundle via REQUESTS_CA_BUNDLE/SSL_CERT_FILE environment variables
- Configure boto3 client with custom SSL verification to support Ceph RadosGW through nginx proxy
- Maintain boto3 checksum validation configuration for compatibility with third-party S3 providers
- Resolves XAmzContentSHA256Mismatch errors when connecting to RadosGW endpoints
Fixes#4400 compatibility issue with boto3 v1.36+ stricter checksum validation
- Replace hardcoded remotes.yaml path with CONFIG_PATH environment variable
- Update docker-compose.yml to set CONFIG_PATH=/app/remotes.yaml
- Application now requires CONFIG_PATH to be set, enabling flexible deployment
- FastAPI-based caching proxy for remote file servers
- YAML configuration for multiple remotes (GitHub, Gitea, HashiCorp, etc.)
- Direct URL API: /api/v1/remote/{remote}/{path} with auto-download and caching
- Pattern-based access control with regex filtering
- S3/MinIO backend storage with predictable paths
- Docker Compose setup with MinIO for local development
