Compare commits

..

6 Commits

Author SHA1 Message Date
unkinben 72a07663e7 feat: add local RPM repository with on-demand repodata
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
Upload RPMs to local repos. Metadata is parsed async after upload
using cavaliergopher/rpm and stored in rpm_metadata table. Repodata
(repomd.xml, primary.xml.gz, filelists.xml.gz, other.xml.gz) is
generated on-demand from the DB — nothing stored in S3.

- RPM provider implements LocalUploader (validates .rpm extension,
  stores under Packages/)
- RPM provider implements PostUploadHook (async goroutine parses RPM
  headers, extracts name/version/arch/deps/etc into rpm_metadata)
- RPM provider implements LocalIndexer (serves repodata/* paths by
  querying rpm_metadata and generating XML on the fly)
- New provider interfaces: PostUploadHook, BlobReader, MetadataStore,
  RPMMetadataReader
- New rpm_metadata table with JSONB columns for requires/provides/
  files/changelogs

Tested e2e: upload cowsay RPM → repodata generated → dnf install
from local repo
2026-06-23 23:14:24 +10:00
unkinben 3a6721c2a7 refactor: modular local provider interfaces (#52)
ci/woodpecker/tag/docker Pipeline was successful
## Summary
Move package-type-specific local repo logic out of centralized handlers into provider packages via optional Go interfaces.

**New interfaces in `provider` package:**
- \`LocalUploader\`: \`ValidateUpload(filePath) → (storagePath, contentType, error)\` + \`UploadResponse(...)\`
- \`LocalIndexer\`: \`ServeLocalIndex(w, r, files, repoName, path) → bool\` + \`GenerateLocalIndex(ctx, files, repoName, path) → ([]byte, error)\`
- \`FileStore\`: \`ListFilesByPrefix\` + \`ListPackages\` (implemented by database.DB)

**Providers implement these interfaces:**
- PyPI: upload validation (wheel/sdist naming), simple index serving + generation
- Terraform: upload validation (provider zip naming), mirror protocol serving

**Handlers simplified to generic dispatch:**
- \`local.go\`: type-asserts to \`LocalUploader\`, falls back to generic upload
- \`proxy.go\`: type-asserts to \`LocalIndexer\`, falls back to raw file serving
- \`engine.go\`: type-asserts to \`LocalIndexer\` for local virtual members

Adding a new local repo type (e.g. RPM) = implement the interfaces in its provider package. Zero handler changes.

## Test plan
- [x] Build + unit tests pass
- [x] E2E: PyPI local upload → simple index → uv pip install (smoke test after refactor)

Reviewed-on: #52
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-23 22:56:25 +10:00
unkinben 7b13644421 feat: virtual PyPI repos can merge local + remote members (#51)
ci/woodpecker/tag/docker Pipeline was successful
## Summary
- Virtual engine detects local members and generates indexes in-memory
- MemberIndex.RepoType drives correct URL prefix in merged output
- PyPI merger rewrites links to /api/v1/local/ or /api/v1/remote/ appropriately
- Includes local PyPI support (cherry-picked from #50)

## Test plan
- [x] Upload wheel to local PyPI → install from direct local URL
- [x] Create virtual with local + remote → install from virtual URL
- [x] Both paths produce correct absolute download URLs

Reviewed-on: #51
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-23 22:20:05 +10:00
unkinben de96637122 feat: add local PyPI repository support (#50)
## Summary
- Upload Python wheels/sdists to local PyPI repos with filename validation
- PEP 503 simple index computed on-demand from stored files
- Package names normalized per PEP 503 (lowercase, hyphens)
- Overwrites rejected (409 Conflict)

## Test plan
- [x] Build wheel with `uv build` → upload → verify simple index HTML → `uv pip install` from local repo
- [x] Bad filename rejection (400)
- [x] Overwrite rejection (409)
- [x] Hash integrity verification on download

Reviewed-on: #50
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-23 22:13:09 +10:00
benvin 1e91a5fb72 feat: add local repository type with repo_type field (#49)
ci/woodpecker/tag/docker Pipeline was successful
Introduces repo_type (remote/local) as a separate axis from package_type
so that any package type can be hosted locally. A terraform local repo
is package_type=terraform + repo_type=local.

- Remote model gains RepoType field (defaults to "remote")
- Database schema adds repo_type column with migration for existing DBs
- V1 proxy adds /api/v1/local/{name}/* route for serving local files
- V2 upload via PUT /api/v2/remotes/{name}/files/{ns}/{type}/{file}.zip
  validates filename matches terraform-provider-{type}_{ver}_{os}_{arch}.zip
  and returns 409 on duplicate (no overwrites)
- index.json and {version}.json are computed on-the-fly from uploaded zips
  rather than stored as separate files
- V2 create validates repo_type and requires base_url only for remotes

---------

Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #49
2026-06-22 23:52:20 +10:00
benvin a481a5c3b7 feat: tree view for cached objects, top-files stats on dashboard (#48)
- Objects page renders paths as a collapsible tree instead of flat list
  with expand/collapse all, aggregated size/hits per directory
- Dashboard gains top-files-by-hits and top-files-by-bandwidth tables
- Backend: new /api/v2/stats/top-files-by-hits and
  /api/v2/stats/top-files-by-bandwidth endpoints
- Raised per_page max to 5000 for objects listing

---------

Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #48
2026-06-22 22:49:56 +10:00
27 changed files with 1993 additions and 135 deletions
+1
View File
@@ -3,6 +3,7 @@ module git.unkin.net/unkin/artifactapi
go 1.25.9
require (
github.com/cavaliergopher/rpm v1.3.0
github.com/charmbracelet/bubbletea v1.3.10
github.com/charmbracelet/lipgloss v1.1.0
github.com/go-chi/chi/v5 v5.3.0
+2
View File
@@ -12,6 +12,8 @@ github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
github.com/cavaliergopher/rpm v1.3.0 h1:UHX46sasX8MesUXXQ+UbkFLUX4eUWTlEcX8jcnRBIgI=
github.com/cavaliergopher/rpm v1.3.0/go.mod h1:vEumo1vvtrHM1Ov86f6+k8j7zNKOxQfHDCAIcR/36ZI=
github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
+55 -2
View File
@@ -9,9 +9,11 @@ import (
"github.com/go-chi/chi/v5"
v2 "git.unkin.net/unkin/artifactapi/internal/api/v2"
"git.unkin.net/unkin/artifactapi/internal/database"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/proxy"
"git.unkin.net/unkin/artifactapi/internal/storage"
"git.unkin.net/unkin/artifactapi/internal/virtual"
)
@@ -19,15 +21,18 @@ type ProxyHandler struct {
engine *proxy.Engine
virtualEngine *virtual.Engine
db *database.DB
store *storage.S3
local *v2.LocalHandler
}
func NewProxyHandler(engine *proxy.Engine, virtualEngine *virtual.Engine, db *database.DB) *ProxyHandler {
return &ProxyHandler{engine: engine, virtualEngine: virtualEngine, db: db}
func NewProxyHandler(engine *proxy.Engine, virtualEngine *virtual.Engine, db *database.DB, store *storage.S3, local *v2.LocalHandler) *ProxyHandler {
return &ProxyHandler{engine: engine, virtualEngine: virtualEngine, db: db, store: store, local: local}
}
func (h *ProxyHandler) Routes() chi.Router {
r := chi.NewRouter()
r.Get("/remote/{remoteName}/*", h.handleProxy)
r.Get("/local/{localName}/*", h.handleLocal)
r.Get("/virtual/{virtualName}/*", h.handleVirtual)
return r
}
@@ -95,6 +100,54 @@ func (h *ProxyHandler) handleVirtual(w http.ResponseWriter, r *http.Request) {
w.Write(body)
}
func (h *ProxyHandler) handleLocal(w http.ResponseWriter, r *http.Request) {
localName := chi.URLParam(r, "localName")
path := chi.URLParam(r, "*")
remote, err := h.db.GetRemote(r.Context(), localName)
if err != nil {
http.Error(w, fmt.Sprintf("local %q not found", localName), http.StatusNotFound)
return
}
prov, _ := provider.Get(remote.PackageType)
if indexer, ok := prov.(provider.LocalIndexer); ok {
if indexer.ServeLocalIndex(w, r, h.db, remote.Name, path) {
return
}
}
h.serveLocalFile(w, r, localName, path)
}
func (h *ProxyHandler) serveLocalFile(w http.ResponseWriter, r *http.Request, repoName, path string) {
file, err := h.db.GetLocalFile(r.Context(), repoName, path)
if err != nil {
slog.Error("local file lookup failed", "repo", repoName, "path", path, "error", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
if file == nil {
http.Error(w, "not found", http.StatusNotFound)
return
}
s3Key := storage.BlobKey(file.ContentHash[len("sha256:"):])
reader, info, err := h.store.Download(r.Context(), s3Key)
if err != nil {
slog.Error("local file download failed", "repo", repoName, "path", path, "error", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
defer reader.Close()
w.Header().Set("Content-Type", info.ContentType)
w.Header().Set("Content-Length", fmt.Sprintf("%d", info.Size))
w.Header().Set("X-Artifact-Source", "local")
w.WriteHeader(http.StatusOK)
io.Copy(w, reader)
}
func scheme(r *http.Request) string {
if r.TLS != nil {
return "https"
+205
View File
@@ -0,0 +1,205 @@
package v2
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"github.com/go-chi/chi/v5"
"git.unkin.net/unkin/artifactapi/internal/database"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/storage"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
type LocalHandler struct {
db *database.DB
store *storage.S3
cas *storage.CAS
}
func NewLocalHandler(db *database.DB, store *storage.S3) *LocalHandler {
return &LocalHandler{
db: db,
store: store,
cas: storage.NewCAS(store),
}
}
func (h *LocalHandler) Routes() chi.Router {
r := chi.NewRouter()
r.Put("/*", h.upload)
r.Get("/*", h.download)
r.Delete("/*", h.remove)
return r
}
func (h *LocalHandler) upload(w http.ResponseWriter, r *http.Request) {
repoName := chi.URLParam(r, "name")
filePath := chi.URLParam(r, "*")
if filePath == "" {
http.Error(w, "file path required", http.StatusBadRequest)
return
}
remote, err := h.db.GetRemote(r.Context(), repoName)
if err != nil {
http.Error(w, fmt.Sprintf("remote %q not found", repoName), http.StatusNotFound)
return
}
if remote.RepoType != models.RepoTypeLocal {
http.Error(w, "upload only allowed for local repository types", http.StatusBadRequest)
return
}
prov, _ := provider.Get(remote.PackageType)
if uploader, ok := prov.(provider.LocalUploader); ok {
h.uploadValidated(w, r, remote, filePath, prov, uploader)
return
}
h.uploadGeneric(w, r, remote, filePath)
}
func (h *LocalHandler) uploadValidated(w http.ResponseWriter, r *http.Request, remote *models.Remote, filePath string, prov provider.Provider, uploader provider.LocalUploader) {
storagePath, contentType, err := uploader.ValidateUpload(filePath)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
existing, err := h.db.GetLocalFile(r.Context(), remote.Name, storagePath)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if existing != nil {
http.Error(w, fmt.Sprintf("file %q already exists; overwrites are not allowed", storagePath), http.StatusConflict)
return
}
result, err := h.cas.Store(r.Context(), r.Body, contentType)
if err != nil {
http.Error(w, fmt.Sprintf("store failed: %v", err), http.StatusInternalServerError)
return
}
if err := h.db.UpsertBlob(r.Context(), result.ContentHash, result.S3Key, result.SizeBytes, contentType); err != nil {
http.Error(w, fmt.Sprintf("record blob: %v", err), http.StatusInternalServerError)
return
}
if err := h.db.CreateLocalFile(r.Context(), remote.Name, storagePath, result.ContentHash); err != nil {
if errors.Is(err, database.ErrAlreadyExists) {
http.Error(w, fmt.Sprintf("file %q already exists; overwrites are not allowed", storagePath), http.StatusConflict)
return
}
http.Error(w, fmt.Sprintf("record file: %v", err), http.StatusInternalServerError)
return
}
if hook, ok := prov.(provider.PostUploadHook); ok {
go hook.AfterUpload(context.Background(), remote.Name, storagePath, result.ContentHash, h, h.db)
}
writeJSON(w, http.StatusCreated, uploader.UploadResponse(storagePath, result.ContentHash, result.SizeBytes))
}
func (h *LocalHandler) uploadGeneric(w http.ResponseWriter, r *http.Request, remote *models.Remote, filePath string) {
existing, err := h.db.GetLocalFile(r.Context(), remote.Name, filePath)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if existing != nil {
http.Error(w, fmt.Sprintf("file %q already exists; overwrites are not allowed", filePath), http.StatusConflict)
return
}
contentType := "application/octet-stream"
if ct := r.Header.Get("Content-Type"); ct != "" && ct != "application/octet-stream" {
contentType = ct
}
result, err := h.cas.Store(r.Context(), r.Body, contentType)
if err != nil {
http.Error(w, fmt.Sprintf("store failed: %v", err), http.StatusInternalServerError)
return
}
if err := h.db.UpsertBlob(r.Context(), result.ContentHash, result.S3Key, result.SizeBytes, contentType); err != nil {
http.Error(w, fmt.Sprintf("record blob: %v", err), http.StatusInternalServerError)
return
}
if err := h.db.CreateLocalFile(r.Context(), remote.Name, filePath, result.ContentHash); err != nil {
if errors.Is(err, database.ErrAlreadyExists) {
http.Error(w, fmt.Sprintf("file %q already exists; overwrites are not allowed", filePath), http.StatusConflict)
return
}
http.Error(w, fmt.Sprintf("record file: %v", err), http.StatusInternalServerError)
return
}
writeJSON(w, http.StatusCreated, map[string]any{
"path": filePath,
"content_hash": result.ContentHash,
"size_bytes": result.SizeBytes,
})
}
func (h *LocalHandler) download(w http.ResponseWriter, r *http.Request) {
repoName := chi.URLParam(r, "name")
filePath := chi.URLParam(r, "*")
file, err := h.db.GetLocalFile(r.Context(), repoName, filePath)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if file == nil {
http.Error(w, "not found", http.StatusNotFound)
return
}
s3Key := storage.BlobKey(file.ContentHash[len("sha256:"):])
reader, info, err := h.store.Download(r.Context(), s3Key)
if err != nil {
http.Error(w, fmt.Sprintf("download failed: %v", err), http.StatusInternalServerError)
return
}
defer reader.Close()
w.Header().Set("Content-Type", info.ContentType)
w.Header().Set("Content-Length", fmt.Sprintf("%d", info.Size))
w.WriteHeader(http.StatusOK)
io.Copy(w, reader)
}
func (h *LocalHandler) remove(w http.ResponseWriter, r *http.Request) {
repoName := chi.URLParam(r, "name")
filePath := chi.URLParam(r, "*")
if err := h.db.DeleteLocalFile(r.Context(), repoName, filePath); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
func (h *LocalHandler) DB() *database.DB {
return h.db
}
func (h *LocalHandler) Download(ctx context.Context, key string) (io.ReadCloser, int64, error) {
reader, info, err := h.store.Download(ctx, key)
if err != nil {
return nil, 0, err
}
return reader, info.Size, nil
}
+1 -1
View File
@@ -28,7 +28,7 @@ func (h *ObjectsHandler) Routes() chi.Router {
func (h *ObjectsHandler) list(w http.ResponseWriter, r *http.Request) {
remoteName := chi.URLParam(r, "name")
limit, _ := strconv.Atoi(r.URL.Query().Get("per_page"))
if limit <= 0 || limit > 100 {
if limit <= 0 || limit > 5000 {
limit = 50
}
page, _ := strconv.Atoi(r.URL.Query().Get("page"))
+11
View File
@@ -58,6 +58,17 @@ func (h *RemotesHandler) create(w http.ResponseWriter, r *http.Request) {
http.Error(w, fmt.Sprintf("invalid package type: %q", remote.PackageType), http.StatusBadRequest)
return
}
if remote.RepoType == "" {
remote.RepoType = models.RepoTypeRemote
}
if !remote.RepoType.Valid() {
http.Error(w, fmt.Sprintf("invalid repo type: %q", remote.RepoType), http.StatusBadRequest)
return
}
if remote.RepoType == models.RepoTypeRemote && remote.BaseURL == "" {
http.Error(w, "base_url is required for remote repositories", http.StatusBadRequest)
return
}
if err := h.db.CreateRemote(r.Context(), &remote); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
+20
View File
@@ -20,6 +20,8 @@ func (h *StatsHandler) Routes() chi.Router {
r := chi.NewRouter()
r.Get("/", h.overview)
r.Get("/top-remotes", h.topRemotes)
r.Get("/top-files-by-hits", h.topFilesByHits)
r.Get("/top-files-by-bandwidth", h.topFilesByBandwidth)
return r
}
@@ -40,3 +42,21 @@ func (h *StatsHandler) topRemotes(w http.ResponseWriter, r *http.Request) {
}
writeJSON(w, http.StatusOK, remotes)
}
func (h *StatsHandler) topFilesByHits(w http.ResponseWriter, r *http.Request) {
files, err := h.db.GetTopFilesByHits(r.Context(), 10)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
writeJSON(w, http.StatusOK, files)
}
func (h *StatsHandler) topFilesByBandwidth(w http.ResponseWriter, r *http.Request) {
files, err := h.db.GetTopFilesByBandwidth(r.Context(), 10)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
writeJSON(w, http.StatusOK, files)
}
+146
View File
@@ -0,0 +1,146 @@
package database
import (
"context"
"errors"
"fmt"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgconn"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
type LocalFile struct {
ID int64 `json:"id"`
RepoName string `json:"repo_name"`
FilePath string `json:"file_path"`
ContentHash string `json:"content_hash"`
CreatedAt time.Time `json:"created_at"`
}
var ErrAlreadyExists = fmt.Errorf("file already exists")
func (db *DB) CreateLocalFile(ctx context.Context, repoName, filePath, contentHash string) error {
_, err := db.Pool.Exec(ctx, `
INSERT INTO local_files (repo_name, file_path, content_hash)
VALUES ($1, $2, $3)
`, repoName, filePath, contentHash)
if err != nil {
var pgErr *pgconn.PgError
if errors.As(err, &pgErr) && pgErr.Code == "23505" {
return ErrAlreadyExists
}
return err
}
return nil
}
func (db *DB) GetLocalFile(ctx context.Context, repoName, filePath string) (*LocalFile, error) {
row := db.Pool.QueryRow(ctx, `
SELECT id, repo_name, file_path, content_hash, created_at
FROM local_files
WHERE repo_name = $1 AND file_path = $2
`, repoName, filePath)
var f LocalFile
if err := row.Scan(&f.ID, &f.RepoName, &f.FilePath, &f.ContentHash, &f.CreatedAt); err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return nil, nil
}
return nil, err
}
return &f, nil
}
func (db *DB) ListLocalFiles(ctx context.Context, repoName string, limit, offset int) ([]LocalFile, error) {
rows, err := db.Pool.Query(ctx, `
SELECT id, repo_name, file_path, content_hash, created_at
FROM local_files
WHERE repo_name = $1
ORDER BY file_path
LIMIT $2 OFFSET $3
`, repoName, limit, offset)
if err != nil {
return nil, err
}
defer rows.Close()
var files []LocalFile
for rows.Next() {
var f LocalFile
if err := rows.Scan(&f.ID, &f.RepoName, &f.FilePath, &f.ContentHash, &f.CreatedAt); err != nil {
return nil, err
}
files = append(files, f)
}
return files, rows.Err()
}
func (db *DB) ListLocalFilesByPrefix(ctx context.Context, repoName, prefix string) ([]LocalFile, error) {
rows, err := db.Pool.Query(ctx, `
SELECT id, repo_name, file_path, content_hash, created_at
FROM local_files
WHERE repo_name = $1 AND file_path LIKE $2
ORDER BY file_path
`, repoName, prefix+"%")
if err != nil {
return nil, err
}
defer rows.Close()
var files []LocalFile
for rows.Next() {
var f LocalFile
if err := rows.Scan(&f.ID, &f.RepoName, &f.FilePath, &f.ContentHash, &f.CreatedAt); err != nil {
return nil, err
}
files = append(files, f)
}
return files, rows.Err()
}
func (db *DB) ListLocalFilePackages(ctx context.Context, repoName string) ([]string, error) {
rows, err := db.Pool.Query(ctx, `
SELECT DISTINCT split_part(file_path, '/', 1)
FROM local_files
WHERE repo_name = $1
ORDER BY 1
`, repoName)
if err != nil {
return nil, err
}
defer rows.Close()
var packages []string
for rows.Next() {
var pkg string
if err := rows.Scan(&pkg); err != nil {
return nil, err
}
packages = append(packages, pkg)
}
return packages, rows.Err()
}
func (db *DB) ListFilesByPrefix(ctx context.Context, repoName, prefix string) ([]provider.FileEntry, error) {
files, err := db.ListLocalFilesByPrefix(ctx, repoName, prefix)
if err != nil {
return nil, err
}
result := make([]provider.FileEntry, len(files))
for i, f := range files {
result[i] = provider.FileEntry{FilePath: f.FilePath, ContentHash: f.ContentHash}
}
return result, nil
}
func (db *DB) ListPackages(ctx context.Context, repoName string) ([]string, error) {
return db.ListLocalFilePackages(ctx, repoName)
}
func (db *DB) DeleteLocalFile(ctx context.Context, repoName, filePath string) error {
_, err := db.Pool.Exec(ctx, `DELETE FROM local_files WHERE repo_name = $1 AND file_path = $2`, repoName, filePath)
return err
}
+35 -1
View File
@@ -42,7 +42,8 @@ func (db *DB) migrate() error {
CREATE TABLE IF NOT EXISTS remotes (
name TEXT PRIMARY KEY,
package_type TEXT NOT NULL,
base_url TEXT NOT NULL,
repo_type TEXT DEFAULT 'remote',
base_url TEXT NOT NULL DEFAULT '',
description TEXT DEFAULT '',
username TEXT DEFAULT '',
password TEXT DEFAULT '',
@@ -121,6 +122,39 @@ func (db *DB) migrate() error {
);
CREATE INDEX IF NOT EXISTS idx_access_log_remote_time ON access_log(remote_name, created_at);
ALTER TABLE remotes ADD COLUMN IF NOT EXISTS repo_type TEXT DEFAULT 'remote';
CREATE TABLE IF NOT EXISTS rpm_metadata (
id BIGSERIAL PRIMARY KEY,
repo_name TEXT NOT NULL,
file_path TEXT NOT NULL,
content_hash TEXT NOT NULL,
name TEXT NOT NULL,
epoch INTEGER DEFAULT 0,
version TEXT NOT NULL,
release TEXT NOT NULL,
arch TEXT NOT NULL,
summary TEXT DEFAULT '',
description TEXT DEFAULT '',
rpm_size BIGINT DEFAULT 0,
installed_size BIGINT DEFAULT 0,
license TEXT DEFAULT '',
vendor TEXT DEFAULT '',
build_group TEXT DEFAULT '',
build_host TEXT DEFAULT '',
source_rpm TEXT DEFAULT '',
url TEXT DEFAULT '',
packager TEXT DEFAULT '',
requires JSONB DEFAULT '[]',
provides JSONB DEFAULT '[]',
files JSONB DEFAULT '[]',
changelogs JSONB DEFAULT '[]',
created_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(repo_name, file_path)
);
CREATE INDEX IF NOT EXISTS idx_rpm_metadata_repo ON rpm_metadata(repo_name);
`)
return err
}
+12 -12
View File
@@ -6,7 +6,7 @@ import (
"git.unkin.net/unkin/artifactapi/pkg/models"
)
const remoteCols = `name, package_type, base_url, description, username, password,
const remoteCols = `name, package_type, repo_type, base_url, description, username, password,
immutable_ttl, mutable_ttl, check_mutable,
patterns, blocklist, mutable_patterns, immutable_patterns,
ban_tags_enabled, ban_tags,
@@ -15,7 +15,7 @@ const remoteCols = `name, package_type, base_url, description, username, passwor
func scanRemote(scanner interface{ Scan(...any) error }, r *models.Remote) error {
return scanner.Scan(
&r.Name, &r.PackageType, &r.BaseURL, &r.Description, &r.Username, &r.Password,
&r.Name, &r.PackageType, &r.RepoType, &r.BaseURL, &r.Description, &r.Username, &r.Password,
&r.ImmutableTTL, &r.MutableTTL, &r.CheckMutable,
&r.Patterns, &r.Blocklist, &r.MutablePatterns, &r.ImmutablePatterns,
&r.BanTagsEnabled, &r.BanTags,
@@ -54,15 +54,15 @@ func (db *DB) ListRemotes(ctx context.Context) ([]models.Remote, error) {
func (db *DB) CreateRemote(ctx context.Context, r *models.Remote) error {
_, err := db.Pool.Exec(ctx, `
INSERT INTO remotes (
name, package_type, base_url, description, username, password,
name, package_type, repo_type, base_url, description, username, password,
immutable_ttl, mutable_ttl, check_mutable,
patterns, blocklist, mutable_patterns, immutable_patterns,
ban_tags_enabled, ban_tags,
quarantine_enabled, quarantine_days, stale_on_error,
releases_remote, managed_by
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20)
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21)
`,
r.Name, r.PackageType, r.BaseURL, r.Description, r.Username, r.Password,
r.Name, r.PackageType, r.RepoType, r.BaseURL, r.Description, r.Username, r.Password,
r.ImmutableTTL, r.MutableTTL, r.CheckMutable,
r.Patterns, r.Blocklist, r.MutablePatterns, r.ImmutablePatterns,
r.BanTagsEnabled, r.BanTags,
@@ -75,15 +75,15 @@ func (db *DB) CreateRemote(ctx context.Context, r *models.Remote) error {
func (db *DB) UpdateRemote(ctx context.Context, r *models.Remote) error {
_, err := db.Pool.Exec(ctx, `
UPDATE remotes SET
package_type=$2, base_url=$3, description=$4, username=$5, password=$6,
immutable_ttl=$7, mutable_ttl=$8, check_mutable=$9,
patterns=$10, blocklist=$11, mutable_patterns=$12, immutable_patterns=$13,
ban_tags_enabled=$14, ban_tags=$15,
quarantine_enabled=$16, quarantine_days=$17, stale_on_error=$18,
releases_remote=$19, managed_by=$20, updated_at=NOW()
package_type=$2, repo_type=$3, base_url=$4, description=$5, username=$6, password=$7,
immutable_ttl=$8, mutable_ttl=$9, check_mutable=$10,
patterns=$11, blocklist=$12, mutable_patterns=$13, immutable_patterns=$14,
ban_tags_enabled=$15, ban_tags=$16,
quarantine_enabled=$17, quarantine_days=$18, stale_on_error=$19,
releases_remote=$20, managed_by=$21, updated_at=NOW()
WHERE name=$1
`,
r.Name, r.PackageType, r.BaseURL, r.Description, r.Username, r.Password,
r.Name, r.PackageType, r.RepoType, r.BaseURL, r.Description, r.Username, r.Password,
r.ImmutableTTL, r.MutableTTL, r.CheckMutable,
r.Patterns, r.Blocklist, r.MutablePatterns, r.ImmutablePatterns,
r.BanTagsEnabled, r.BanTags,
+129
View File
@@ -0,0 +1,129 @@
package database
import (
"context"
"encoding/json"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
func (db *DB) InsertRPMMetadata(ctx context.Context, meta *provider.RPMMetadata) error {
requiresJSON, _ := json.Marshal(meta.Requires)
providesJSON, _ := json.Marshal(meta.Provides)
filesJSON, _ := json.Marshal(meta.Files)
changelogsJSON, _ := json.Marshal(meta.Changelogs)
_, err := db.Pool.Exec(ctx, `
INSERT INTO rpm_metadata (
repo_name, file_path, content_hash,
name, epoch, version, release, arch,
summary, description, rpm_size, installed_size,
license, vendor, build_group, build_host, source_rpm, url, packager,
requires, provides, files, changelogs
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23)
ON CONFLICT (repo_name, file_path) DO NOTHING
`,
meta.RepoName, meta.FilePath, meta.ContentHash,
meta.Name, meta.Epoch, meta.Version, meta.Release, meta.Arch,
meta.Summary, meta.Description, meta.RPMSize, meta.InstalledSize,
meta.License, meta.Vendor, meta.Group, meta.BuildHost, meta.SourceRPM, meta.URL, meta.Packager,
requiresJSON, providesJSON, filesJSON, changelogsJSON,
)
return err
}
type RPMMetadataRow struct {
RepoName string
FilePath string
ContentHash string
Name string
Epoch int
Version string
Release string
Arch string
Summary string
Description string
RPMSize int64
InstalledSize int64
License string
Vendor string
Group string
BuildHost string
SourceRPM string
URL string
Packager string
Requires json.RawMessage
Provides json.RawMessage
Files json.RawMessage
Changelogs json.RawMessage
}
func (db *DB) ListRPMMetadataEntries(ctx context.Context, repoName string) ([]provider.RPMMetadata, error) {
rows, err := db.ListRPMMetadata(ctx, repoName)
if err != nil {
return nil, err
}
result := make([]provider.RPMMetadata, len(rows))
for i, r := range rows {
meta := provider.RPMMetadata{
RepoName: r.RepoName,
FilePath: r.FilePath,
ContentHash: r.ContentHash,
Name: r.Name,
Epoch: r.Epoch,
Version: r.Version,
Release: r.Release,
Arch: r.Arch,
Summary: r.Summary,
Description: r.Description,
RPMSize: r.RPMSize,
InstalledSize: r.InstalledSize,
License: r.License,
Vendor: r.Vendor,
Group: r.Group,
BuildHost: r.BuildHost,
SourceRPM: r.SourceRPM,
URL: r.URL,
Packager: r.Packager,
}
json.Unmarshal(r.Requires, &meta.Requires)
json.Unmarshal(r.Provides, &meta.Provides)
json.Unmarshal(r.Files, &meta.Files)
json.Unmarshal(r.Changelogs, &meta.Changelogs)
result[i] = meta
}
return result, nil
}
func (db *DB) ListRPMMetadata(ctx context.Context, repoName string) ([]RPMMetadataRow, error) {
rows, err := db.Pool.Query(ctx, `
SELECT repo_name, file_path, content_hash,
name, epoch, version, release, arch,
summary, description, rpm_size, installed_size,
license, vendor, build_group, build_host, source_rpm, url, packager,
requires, provides, files, changelogs
FROM rpm_metadata
WHERE repo_name = $1
ORDER BY name, epoch, version, release, arch
`, repoName)
if err != nil {
return nil, err
}
defer rows.Close()
var result []RPMMetadataRow
for rows.Next() {
var r RPMMetadataRow
if err := rows.Scan(
&r.RepoName, &r.FilePath, &r.ContentHash,
&r.Name, &r.Epoch, &r.Version, &r.Release, &r.Arch,
&r.Summary, &r.Description, &r.RPMSize, &r.InstalledSize,
&r.License, &r.Vendor, &r.Group, &r.BuildHost, &r.SourceRPM, &r.URL, &r.Packager,
&r.Requires, &r.Provides, &r.Files, &r.Changelogs,
); err != nil {
return nil, err
}
result = append(result, r)
}
return result, rows.Err()
}
+65
View File
@@ -76,3 +76,68 @@ func (db *DB) GetTopRemotes(ctx context.Context, limit int) ([]RemoteStatRow, er
}
return result, rows.Err()
}
type FileStatRow struct {
RemoteName string `json:"remote_name"`
Path string `json:"path"`
AccessCount int64 `json:"access_count"`
SizeBytes int64 `json:"size_bytes"`
}
func (db *DB) GetTopFilesByHits(ctx context.Context, limit int) ([]FileStatRow, error) {
rows, err := db.Pool.Query(ctx, `
SELECT a.remote_name, a.path, a.access_count, b.size_bytes
FROM artifacts a
JOIN blobs b ON a.content_hash = b.content_hash
ORDER BY a.access_count DESC
LIMIT $1
`, limit)
if err != nil {
return nil, err
}
defer rows.Close()
var result []FileStatRow
for rows.Next() {
var r FileStatRow
if err := rows.Scan(&r.RemoteName, &r.Path, &r.AccessCount, &r.SizeBytes); err != nil {
return nil, err
}
result = append(result, r)
}
return result, rows.Err()
}
type BandwidthStatRow struct {
RemoteName string `json:"remote_name"`
Path string `json:"path"`
Bandwidth int64 `json:"bandwidth"`
Requests int64 `json:"requests"`
}
func (db *DB) GetTopFilesByBandwidth(ctx context.Context, limit int) ([]BandwidthStatRow, error) {
rows, err := db.Pool.Query(ctx, `
SELECT remote_name, path,
COALESCE(SUM(size_bytes), 0) AS bandwidth,
COUNT(*) AS requests
FROM access_log
WHERE created_at > NOW() - INTERVAL '30 days'
GROUP BY remote_name, path
ORDER BY bandwidth DESC
LIMIT $1
`, limit)
if err != nil {
return nil, err
}
defer rows.Close()
var result []BandwidthStatRow
for rows.Next() {
var r BandwidthStatRow
if err := rows.Scan(&r.RemoteName, &r.Path, &r.Bandwidth, &r.Requests); err != nil {
return nil, err
}
result = append(result, r)
}
return result, rows.Err()
}
+82
View File
@@ -3,6 +3,7 @@ package provider
import (
"context"
"fmt"
"io"
"net/http"
"git.unkin.net/unkin/artifactapi/pkg/models"
@@ -24,6 +25,87 @@ type Provider interface {
AuthHeaders(ctx context.Context, remote models.Remote) (http.Header, error)
}
type FileEntry struct {
FilePath string
ContentHash string
}
type FileStore interface {
ListFilesByPrefix(ctx context.Context, repoName, prefix string) ([]FileEntry, error)
ListPackages(ctx context.Context, repoName string) ([]string, error)
}
type LocalUploader interface {
ValidateUpload(filePath string) (storagePath, contentType string, err error)
UploadResponse(storagePath, contentHash string, sizeBytes int64) map[string]any
}
type LocalIndexer interface {
ServeLocalIndex(w http.ResponseWriter, r *http.Request, files FileStore, repoName, path string) bool
GenerateLocalIndex(ctx context.Context, files FileStore, repoName, path string) ([]byte, error)
}
type BlobReader interface {
Download(ctx context.Context, key string) (io.ReadCloser, int64, error)
}
type PostUploadHook interface {
AfterUpload(ctx context.Context, repoName, storagePath, contentHash string, blobs BlobReader, db MetadataStore)
}
type MetadataStore interface {
InsertRPMMetadata(ctx context.Context, meta *RPMMetadata) error
}
type RPMMetadataReader interface {
ListRPMMetadataEntries(ctx context.Context, repoName string) ([]RPMMetadata, error)
}
type RPMMetadata struct {
RepoName string
FilePath string
ContentHash string
Name string
Epoch int
Version string
Release string
Arch string
Summary string
Description string
RPMSize int64
InstalledSize int64
License string
Vendor string
Group string
BuildHost string
SourceRPM string
URL string
Packager string
Requires []RPMDep
Provides []RPMDep
Files []RPMFile
Changelogs []RPMChangelog
}
type RPMDep struct {
Name string `json:"name"`
Flags string `json:"flags,omitempty"`
Epoch string `json:"epoch,omitempty"`
Version string `json:"version,omitempty"`
Release string `json:"release,omitempty"`
}
type RPMFile struct {
Path string `json:"path"`
Type string `json:"type,omitempty"`
}
type RPMChangelog struct {
Author string `json:"author"`
Date int64 `json:"date"`
Text string `json:"text"`
}
type IndexMerger interface {
MergeIndexes(members []MemberIndex, proxyBaseURL string) ([]byte, error)
}
+180
View File
@@ -2,7 +2,10 @@ package pypi
import (
"context"
"fmt"
"io"
"net/http"
"regexp"
"strings"
"git.unkin.net/unkin/artifactapi/internal/auth"
@@ -14,6 +17,9 @@ func init() {
provider.Register(&Provider{})
}
var fileRe = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._-]*\.(whl|tar\.gz|zip)$`)
var normalizeRe = regexp.MustCompile(`[-_.]+`)
type Provider struct{}
func (p *Provider) Type() models.PackageType { return models.PackagePyPI }
@@ -60,3 +66,177 @@ func (p *Provider) RewriteResponse(body []byte, remote models.Remote, proxyBaseU
func (p *Provider) AuthHeaders(_ context.Context, remote models.Remote) (http.Header, error) {
return auth.BasicHeaders(remote), nil
}
func normalize(name string) string {
return strings.ToLower(normalizeRe.ReplaceAllString(name, "-"))
}
func packageFromWheel(filename string) string {
parts := strings.SplitN(filename, "-", 3)
if len(parts) < 2 {
return ""
}
return normalize(parts[0])
}
func packageFromSdist(filename string) string {
name := filename
for _, suffix := range []string{".tar.gz", ".zip"} {
if strings.HasSuffix(name, suffix) {
name = strings.TrimSuffix(name, suffix)
break
}
}
idx := strings.LastIndex(name, "-")
if idx <= 0 {
return ""
}
return normalize(name[:idx])
}
func (p *Provider) ValidateUpload(filePath string) (storagePath, contentType string, err error) {
filename := filePath
if idx := strings.LastIndex(filePath, "/"); idx >= 0 {
filename = filePath[idx+1:]
}
if !fileRe.MatchString(filename) {
return "", "", fmt.Errorf("filename %q must be a .whl, .tar.gz, or .zip file", filename)
}
var pkgName string
if strings.HasSuffix(filename, ".whl") {
pkgName = packageFromWheel(filename)
} else {
pkgName = packageFromSdist(filename)
}
if pkgName == "" {
return "", "", fmt.Errorf("cannot parse package name from %q", filename)
}
ct := "application/zip"
if strings.HasSuffix(filename, ".tar.gz") {
ct = "application/gzip"
}
return pkgName + "/" + filename, ct, nil
}
func (p *Provider) UploadResponse(storagePath, contentHash string, sizeBytes int64) map[string]any {
parts := strings.SplitN(storagePath, "/", 2)
filename := storagePath
if len(parts) == 2 {
filename = parts[1]
}
return map[string]any{
"package": parts[0],
"filename": filename,
"content_hash": contentHash,
"size_bytes": sizeBytes,
}
}
func (p *Provider) ServeLocalIndex(w http.ResponseWriter, r *http.Request, files provider.FileStore, repoName, path string) bool {
if path == "simple" || path == "simple/" {
p.servePackageList(w, r, files, repoName)
return true
}
if strings.HasPrefix(path, "simple/") {
pkg := strings.TrimPrefix(path, "simple/")
pkg = strings.TrimSuffix(pkg, "/")
if pkg != "" && !strings.Contains(pkg, "/") {
p.servePackageFiles(w, r, files, repoName, pkg)
return true
}
}
return false
}
func (p *Provider) GenerateLocalIndex(ctx context.Context, files provider.FileStore, repoName, path string) ([]byte, error) {
if !strings.HasPrefix(path, "simple/") {
return nil, fmt.Errorf("unsupported index path: %q", path)
}
pkg := strings.TrimPrefix(path, "simple/")
pkg = strings.TrimSuffix(pkg, "/")
if pkg == "" {
return p.generatePackageListHTML(ctx, files, repoName)
}
return p.generatePackageFilesHTML(ctx, files, repoName, pkg)
}
func (p *Provider) servePackageList(w http.ResponseWriter, r *http.Request, files provider.FileStore, repoName string) {
body, err := p.generatePackageListHTML(r.Context(), files, repoName)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "text/html")
w.WriteHeader(http.StatusOK)
w.Write(body)
}
func (p *Provider) servePackageFiles(w http.ResponseWriter, r *http.Request, files provider.FileStore, repoName, packageName string) {
normalized := normalize(packageName)
prefix := normalized + "/"
entries, err := files.ListFilesByPrefix(r.Context(), repoName, prefix)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if len(entries) == 0 {
http.Error(w, "not found", http.StatusNotFound)
return
}
var b strings.Builder
b.WriteString("<!DOCTYPE html>\n<html><body>\n")
for _, f := range entries {
filename := strings.TrimPrefix(f.FilePath, normalized+"/")
hash := strings.TrimPrefix(f.ContentHash, "sha256:")
fmt.Fprintf(&b, "<a href=\"../../%s/%s#sha256=%s\">%s</a>\n",
normalized, filename, hash, filename)
}
b.WriteString("</body></html>\n")
w.Header().Set("Content-Type", "text/html")
w.WriteHeader(http.StatusOK)
io.WriteString(w, b.String())
}
func (p *Provider) generatePackageListHTML(ctx context.Context, files provider.FileStore, repoName string) ([]byte, error) {
packages, err := files.ListPackages(ctx, repoName)
if err != nil {
return nil, err
}
var b strings.Builder
b.WriteString("<!DOCTYPE html>\n<html><body>\n")
for _, pkg := range packages {
fmt.Fprintf(&b, "<a href=\"%s/\">%s</a>\n", pkg, pkg)
}
b.WriteString("</body></html>\n")
return []byte(b.String()), nil
}
func (p *Provider) generatePackageFilesHTML(ctx context.Context, files provider.FileStore, repoName, packageName string) ([]byte, error) {
normalized := normalize(packageName)
prefix := normalized + "/"
entries, err := files.ListFilesByPrefix(ctx, repoName, prefix)
if err != nil {
return nil, err
}
var b strings.Builder
b.WriteString("<!DOCTYPE html>\n<html><body>\n")
for _, f := range entries {
filename := strings.TrimPrefix(f.FilePath, normalized+"/")
hash := strings.TrimPrefix(f.ContentHash, "sha256:")
fmt.Fprintf(&b, "<a href=\"%s/%s#sha256=%s\">%s</a>\n",
normalized, filename, hash, filename)
}
b.WriteString("</body></html>\n")
return []byte(b.String()), nil
}
+387
View File
@@ -1,13 +1,24 @@
package rpm
import (
"bytes"
"compress/gzip"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/xml"
"fmt"
"log/slog"
"net/http"
"regexp"
"strings"
"time"
rpmlib "github.com/cavaliergopher/rpm"
"git.unkin.net/unkin/artifactapi/internal/auth"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/storage"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
@@ -55,3 +66,379 @@ func (p *Provider) RewriteResponse(_ []byte, _ models.Remote, _ string) ([]byte,
func (p *Provider) AuthHeaders(_ context.Context, remote models.Remote) (http.Header, error) {
return auth.BasicHeaders(remote), nil
}
func (p *Provider) ValidateUpload(filePath string) (storagePath, contentType string, err error) {
filename := filePath
if idx := strings.LastIndex(filePath, "/"); idx >= 0 {
filename = filePath[idx+1:]
}
if !strings.HasSuffix(strings.ToLower(filename), ".rpm") {
return "", "", fmt.Errorf("file must be an .rpm package")
}
return "Packages/" + filename, "application/x-rpm", nil
}
func (p *Provider) UploadResponse(storagePath, contentHash string, sizeBytes int64) map[string]any {
filename := strings.TrimPrefix(storagePath, "Packages/")
return map[string]any{
"filename": filename,
"content_hash": contentHash,
"size_bytes": sizeBytes,
}
}
func (p *Provider) AfterUpload(ctx context.Context, repoName, storagePath, contentHash string, blobs provider.BlobReader, db provider.MetadataStore) {
s3Key := storage.BlobKey(strings.TrimPrefix(contentHash, "sha256:"))
reader, blobSize, err := blobs.Download(ctx, s3Key)
if err != nil {
slog.Error("rpm metadata: download failed", "repo", repoName, "path", storagePath, "error", err)
return
}
defer reader.Close()
pkg, err := rpmlib.Read(reader)
if err != nil {
slog.Error("rpm metadata: parse failed", "repo", repoName, "path", storagePath, "error", err)
return
}
meta := &provider.RPMMetadata{
RepoName: repoName,
FilePath: storagePath,
ContentHash: contentHash,
Name: pkg.Name(),
Epoch: pkg.Epoch(),
Version: pkg.Version(),
Release: pkg.Release(),
Arch: pkg.Architecture(),
Summary: pkg.Summary(),
Description: pkg.Description(),
RPMSize: blobSize,
InstalledSize: int64(pkg.Size()),
License: pkg.License(),
Vendor: pkg.Vendor(),
Group: firstGroup(pkg.Groups()),
BuildHost: pkg.BuildHost(),
SourceRPM: pkg.SourceRPM(),
URL: pkg.URL(),
Packager: pkg.Packager(),
}
for _, req := range pkg.Requires() {
meta.Requires = append(meta.Requires, rpmDepFromEntry(req))
}
for _, prov := range pkg.Provides() {
meta.Provides = append(meta.Provides, rpmDepFromEntry(prov))
}
if meta.Requires == nil {
meta.Requires = []provider.RPMDep{}
}
if meta.Provides == nil {
meta.Provides = []provider.RPMDep{}
}
meta.Files = []provider.RPMFile{}
meta.Changelogs = []provider.RPMChangelog{}
if err := db.InsertRPMMetadata(ctx, meta); err != nil {
slog.Error("rpm metadata: insert failed", "repo", repoName, "path", storagePath, "error", err)
return
}
slog.Info("rpm metadata: parsed", "repo", repoName, "name", meta.Name, "version", meta.Version, "arch", meta.Arch)
}
func rpmDepFromEntry(e rpmlib.Dependency) provider.RPMDep {
dep := provider.RPMDep{Name: e.Name()}
if e.Flags() != 0 {
dep.Flags = rpmFlagString(e.Flags())
dep.Version = e.Version()
dep.Release = e.Release()
if e.Epoch() > 0 {
dep.Epoch = fmt.Sprintf("%d", e.Epoch())
}
}
return dep
}
func rpmFlagString(f int) string {
switch {
case f&0x08 != 0 && f&0x04 != 0:
return "GE"
case f&0x02 != 0 && f&0x04 != 0:
return "LE"
case f&0x08 != 0:
return "GT"
case f&0x02 != 0:
return "LT"
case f&0x04 != 0:
return "EQ"
default:
return ""
}
}
func firstGroup(groups []string) string {
if len(groups) > 0 {
return groups[0]
}
return "Unspecified"
}
func (p *Provider) ServeLocalIndex(w http.ResponseWriter, r *http.Request, files provider.FileStore, repoName, path string) bool {
if !strings.HasPrefix(path, "repodata/") {
return false
}
rpmReader, ok := files.(provider.RPMMetadataReader)
if !ok {
http.Error(w, "rpm metadata not available", http.StatusInternalServerError)
return true
}
tail := strings.TrimPrefix(path, "repodata/")
switch {
case tail == "repomd.xml":
p.serveRepomd(w, r, rpmReader, repoName)
case strings.HasSuffix(tail, "-primary.xml.gz"):
p.servePrimary(w, r, rpmReader, repoName)
case strings.HasSuffix(tail, "-filelists.xml.gz"):
p.serveFilelists(w, r, rpmReader, repoName)
case strings.HasSuffix(tail, "-other.xml.gz"):
p.serveOther(w, r, rpmReader, repoName)
default:
http.Error(w, "not found", http.StatusNotFound)
}
return true
}
func (p *Provider) GenerateLocalIndex(ctx context.Context, files provider.FileStore, repoName, path string) ([]byte, error) {
return nil, fmt.Errorf("rpm local index generation for virtual repos not supported")
}
func (p *Provider) serveRepomd(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) {
metas, err := reader.ListRPMMetadataEntries(r.Context(), repoName)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
primary := generatePrimaryXMLGZ(metas)
filelists := generateFilelistsXMLGZ(metas)
other := generateOtherXMLGZ(metas)
primaryHash := sha256Hex(primary)
filelistsHash := sha256Hex(filelists)
otherHash := sha256Hex(other)
repomd := generateRepomd(primaryHash, len(primary), filelistsHash, len(filelists), otherHash, len(other))
w.Header().Set("Content-Type", "application/xml")
w.WriteHeader(http.StatusOK)
w.Write(repomd)
}
func (p *Provider) servePrimary(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) {
metas, err := reader.ListRPMMetadataEntries(r.Context(), repoName)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/gzip")
w.WriteHeader(http.StatusOK)
w.Write(generatePrimaryXMLGZ(metas))
}
func (p *Provider) serveFilelists(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) {
metas, err := reader.ListRPMMetadataEntries(r.Context(), repoName)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/gzip")
w.WriteHeader(http.StatusOK)
w.Write(generateFilelistsXMLGZ(metas))
}
func (p *Provider) serveOther(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) {
metas, err := reader.ListRPMMetadataEntries(r.Context(), repoName)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/gzip")
w.WriteHeader(http.StatusOK)
w.Write(generateOtherXMLGZ(metas))
}
func generateRepomd(primaryHash string, primarySize int, filelistsHash string, filelistsSize int, otherHash string, otherSize int) []byte {
ts := fmt.Sprintf("%d", time.Now().Unix())
var b bytes.Buffer
b.WriteString(xml.Header)
b.WriteString(`<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">` + "\n")
fmt.Fprintf(&b, " <revision>%s</revision>\n", ts)
writeRepomdData(&b, "primary", primaryHash, primarySize, ts)
writeRepomdData(&b, "filelists", filelistsHash, filelistsSize, ts)
writeRepomdData(&b, "other", otherHash, otherSize, ts)
b.WriteString("</repomd>\n")
return b.Bytes()
}
func writeRepomdData(b *bytes.Buffer, dtype, hash string, size int, ts string) {
fmt.Fprintf(b, " <data type=\"%s\">\n", dtype)
fmt.Fprintf(b, " <checksum type=\"sha256\">%s</checksum>\n", hash)
fmt.Fprintf(b, " <location href=\"repodata/%s-%s.xml.gz\"/>\n", hash, dtype)
fmt.Fprintf(b, " <timestamp>%s</timestamp>\n", ts)
fmt.Fprintf(b, " <size>%d</size>\n", size)
fmt.Fprintf(b, " </data>\n")
}
func generatePrimaryXMLGZ(metas []provider.RPMMetadata) []byte {
var xmlBuf bytes.Buffer
xmlBuf.WriteString(xml.Header)
fmt.Fprintf(&xmlBuf, "<metadata xmlns=\"http://linux.duke.edu/metadata/common\" xmlns:rpm=\"http://linux.duke.edu/metadata/rpm\" packages=\"%d\">\n", len(metas))
for _, m := range metas {
pkgHash := strings.TrimPrefix(m.ContentHash, "sha256:")
fmt.Fprintf(&xmlBuf, "<package type=\"rpm\">\n")
fmt.Fprintf(&xmlBuf, " <name>%s</name>\n", xmlEscape(m.Name))
fmt.Fprintf(&xmlBuf, " <arch>%s</arch>\n", xmlEscape(m.Arch))
fmt.Fprintf(&xmlBuf, " <version epoch=\"%d\" ver=\"%s\" rel=\"%s\"/>\n", m.Epoch, xmlEscape(m.Version), xmlEscape(m.Release))
fmt.Fprintf(&xmlBuf, " <checksum type=\"sha256\" pkgid=\"YES\">%s</checksum>\n", pkgHash)
fmt.Fprintf(&xmlBuf, " <summary>%s</summary>\n", xmlEscape(m.Summary))
fmt.Fprintf(&xmlBuf, " <description>%s</description>\n", xmlEscape(m.Description))
if m.Packager != "" {
fmt.Fprintf(&xmlBuf, " <packager>%s</packager>\n", xmlEscape(m.Packager))
}
if m.URL != "" {
fmt.Fprintf(&xmlBuf, " <url>%s</url>\n", xmlEscape(m.URL))
}
fmt.Fprintf(&xmlBuf, " <time file=\"%d\" build=\"0\"/>\n", time.Now().Unix())
fmt.Fprintf(&xmlBuf, " <size package=\"%d\" installed=\"%d\" archive=\"0\"/>\n", m.RPMSize, m.InstalledSize)
fmt.Fprintf(&xmlBuf, " <location href=\"%s\"/>\n", xmlEscape(m.FilePath))
fmt.Fprintf(&xmlBuf, " <format>\n")
if m.License != "" {
fmt.Fprintf(&xmlBuf, " <rpm:license>%s</rpm:license>\n", xmlEscape(m.License))
}
if m.Vendor != "" {
fmt.Fprintf(&xmlBuf, " <rpm:vendor>%s</rpm:vendor>\n", xmlEscape(m.Vendor))
}
fmt.Fprintf(&xmlBuf, " <rpm:group>%s</rpm:group>\n", xmlEscape(m.Group))
if m.BuildHost != "" {
fmt.Fprintf(&xmlBuf, " <rpm:buildhost>%s</rpm:buildhost>\n", xmlEscape(m.BuildHost))
}
if m.SourceRPM != "" {
fmt.Fprintf(&xmlBuf, " <rpm:sourcerpm>%s</rpm:sourcerpm>\n", xmlEscape(m.SourceRPM))
}
if len(m.Provides) > 0 {
xmlBuf.WriteString(" <rpm:provides>\n")
for _, d := range m.Provides {
writeRPMEntry(&xmlBuf, d)
}
xmlBuf.WriteString(" </rpm:provides>\n")
}
if len(m.Requires) > 0 {
xmlBuf.WriteString(" <rpm:requires>\n")
for _, d := range m.Requires {
writeRPMEntry(&xmlBuf, d)
}
xmlBuf.WriteString(" </rpm:requires>\n")
}
fmt.Fprintf(&xmlBuf, " </format>\n")
fmt.Fprintf(&xmlBuf, "</package>\n")
}
xmlBuf.WriteString("</metadata>\n")
return gzipBytes(xmlBuf.Bytes())
}
func generateFilelistsXMLGZ(metas []provider.RPMMetadata) []byte {
var xmlBuf bytes.Buffer
xmlBuf.WriteString(xml.Header)
fmt.Fprintf(&xmlBuf, "<filelists xmlns=\"http://linux.duke.edu/metadata/filelists\" packages=\"%d\">\n", len(metas))
for _, m := range metas {
pkgHash := strings.TrimPrefix(m.ContentHash, "sha256:")
fmt.Fprintf(&xmlBuf, "<package pkgid=\"%s\" name=\"%s\" arch=\"%s\">\n", pkgHash, xmlEscape(m.Name), xmlEscape(m.Arch))
fmt.Fprintf(&xmlBuf, " <version epoch=\"%d\" ver=\"%s\" rel=\"%s\"/>\n", m.Epoch, xmlEscape(m.Version), xmlEscape(m.Release))
for _, f := range m.Files {
if f.Type != "" {
fmt.Fprintf(&xmlBuf, " <file type=\"%s\">%s</file>\n", f.Type, xmlEscape(f.Path))
} else {
fmt.Fprintf(&xmlBuf, " <file>%s</file>\n", xmlEscape(f.Path))
}
}
xmlBuf.WriteString("</package>\n")
}
xmlBuf.WriteString("</filelists>\n")
return gzipBytes(xmlBuf.Bytes())
}
func generateOtherXMLGZ(metas []provider.RPMMetadata) []byte {
var xmlBuf bytes.Buffer
xmlBuf.WriteString(xml.Header)
fmt.Fprintf(&xmlBuf, "<otherdata xmlns=\"http://linux.duke.edu/metadata/other\" packages=\"%d\">\n", len(metas))
for _, m := range metas {
pkgHash := strings.TrimPrefix(m.ContentHash, "sha256:")
fmt.Fprintf(&xmlBuf, "<package pkgid=\"%s\" name=\"%s\" arch=\"%s\">\n", pkgHash, xmlEscape(m.Name), xmlEscape(m.Arch))
fmt.Fprintf(&xmlBuf, " <version epoch=\"%d\" ver=\"%s\" rel=\"%s\"/>\n", m.Epoch, xmlEscape(m.Version), xmlEscape(m.Release))
for _, cl := range m.Changelogs {
fmt.Fprintf(&xmlBuf, " <changelog author=\"%s\" date=\"%d\">%s</changelog>\n",
xmlEscape(cl.Author), cl.Date, xmlEscape(cl.Text))
}
xmlBuf.WriteString("</package>\n")
}
xmlBuf.WriteString("</otherdata>\n")
return gzipBytes(xmlBuf.Bytes())
}
func writeRPMEntry(b *bytes.Buffer, d provider.RPMDep) {
if d.Flags != "" {
fmt.Fprintf(b, " <rpm:entry name=\"%s\" flags=\"%s\"", xmlEscape(d.Name), d.Flags)
if d.Epoch != "" {
fmt.Fprintf(b, " epoch=\"%s\"", d.Epoch)
}
if d.Version != "" {
fmt.Fprintf(b, " ver=\"%s\"", xmlEscape(d.Version))
}
if d.Release != "" {
fmt.Fprintf(b, " rel=\"%s\"", xmlEscape(d.Release))
}
b.WriteString("/>\n")
} else {
fmt.Fprintf(b, " <rpm:entry name=\"%s\"/>\n", xmlEscape(d.Name))
}
}
func xmlEscape(s string) string {
var b bytes.Buffer
xml.EscapeText(&b, []byte(s))
return b.String()
}
func gzipBytes(data []byte) []byte {
var buf bytes.Buffer
gz := gzip.NewWriter(&buf)
gz.Write(data)
gz.Close()
return buf.Bytes()
}
func sha256Hex(data []byte) string {
h := sha256.Sum256(data)
return hex.EncodeToString(h[:])
}
+149
View File
@@ -3,6 +3,7 @@ package terraform
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/url"
"regexp"
@@ -19,6 +20,12 @@ func init() {
var versionsRe = regexp.MustCompile(`[^/]+/[^/]+/versions$`)
var providerZipRe = regexp.MustCompile(
`^terraform-provider-([a-zA-Z0-9_-]+)_([0-9]+\.[0-9]+\.[0-9]+(?:-[a-zA-Z0-9.]+)?)_([a-z0-9]+)_([a-z0-9]+)\.zip$`,
)
var semverRe = regexp.MustCompile(`^[0-9]+\.[0-9]+\.[0-9]+(?:-[a-zA-Z0-9.]+)?$`)
type Provider struct{}
func (p *Provider) Type() models.PackageType { return models.PackageTerraform }
@@ -86,3 +93,145 @@ func rewriteDownloadURL(originalURL, releasesRemote, proxyBaseURL string) string
func (p *Provider) AuthHeaders(_ context.Context, remote models.Remote) (http.Header, error) {
return auth.BasicHeaders(remote), nil
}
func (p *Provider) ValidateUpload(filePath string) (storagePath, contentType string, err error) {
parts := strings.Split(filePath, "/")
if len(parts) != 3 {
return "", "", fmt.Errorf("path must be {namespace}/{type}/{filename}.zip")
}
namespace, typeName, filename := parts[0], parts[1], parts[2]
m := providerZipRe.FindStringSubmatch(filename)
if m == nil {
return "", "", fmt.Errorf("filename %q does not match terraform-provider-{type}_{version}_{os}_{arch}.zip", filename)
}
if m[1] != typeName {
return "", "", fmt.Errorf("provider type in filename %q does not match path type %q", m[1], typeName)
}
return fmt.Sprintf("%s/%s/%s", namespace, typeName, filename), "application/zip", nil
}
func (p *Provider) UploadResponse(storagePath, contentHash string, sizeBytes int64) map[string]any {
parts := strings.Split(storagePath, "/")
if len(parts) != 3 {
return map[string]any{"path": storagePath, "content_hash": contentHash, "size_bytes": sizeBytes}
}
m := providerZipRe.FindStringSubmatch(parts[2])
if m == nil {
return map[string]any{"path": storagePath, "content_hash": contentHash, "size_bytes": sizeBytes}
}
return map[string]any{
"namespace": parts[0],
"type": parts[1],
"version": m[2],
"os": m[3],
"arch": m[4],
"content_hash": contentHash,
"size_bytes": sizeBytes,
}
}
type terraformIndex struct {
Versions map[string]json.RawMessage `json:"versions"`
}
type terraformVersionDoc struct {
Archives map[string]terraformArchive `json:"archives"`
}
type terraformArchive struct {
URL string `json:"url"`
Hashes []string `json:"hashes,omitempty"`
}
func (p *Provider) ServeLocalIndex(w http.ResponseWriter, r *http.Request, files provider.FileStore, repoName, path string) bool {
parts := strings.Split(path, "/")
if len(parts) < 3 {
return false
}
namespace, typeName := parts[0], parts[1]
tail := parts[2]
if tail == "index.json" {
p.serveIndex(w, r, files, repoName, namespace, typeName)
return true
}
if strings.HasSuffix(tail, ".json") {
version := strings.TrimSuffix(tail, ".json")
if semverRe.MatchString(version) {
p.serveVersionDoc(w, r, files, repoName, namespace, typeName, version)
return true
}
}
return false
}
func (p *Provider) GenerateLocalIndex(ctx context.Context, files provider.FileStore, repoName, path string) ([]byte, error) {
return nil, fmt.Errorf("terraform local index generation for virtual repos not supported")
}
func (p *Provider) serveIndex(w http.ResponseWriter, r *http.Request, files provider.FileStore, repoName, namespace, typeName string) {
prefix := fmt.Sprintf("%s/%s/", namespace, typeName)
entries, err := files.ListFilesByPrefix(r.Context(), repoName, prefix)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
versions := map[string]json.RawMessage{}
for _, f := range entries {
filename := strings.TrimPrefix(f.FilePath, prefix)
m := providerZipRe.FindStringSubmatch(filename)
if m == nil {
continue
}
versions[m[2]] = json.RawMessage(`{}`)
}
if len(versions) == 0 {
http.Error(w, "not found", http.StatusNotFound)
return
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(terraformIndex{Versions: versions})
}
func (p *Provider) serveVersionDoc(w http.ResponseWriter, r *http.Request, files provider.FileStore, repoName, namespace, typeName, version string) {
prefix := fmt.Sprintf("%s/%s/terraform-provider-%s_%s_", namespace, typeName, typeName, version)
entries, err := files.ListFilesByPrefix(r.Context(), repoName, prefix)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
archives := map[string]terraformArchive{}
for _, f := range entries {
filename := strings.TrimPrefix(f.FilePath, fmt.Sprintf("%s/%s/", namespace, typeName))
m := providerZipRe.FindStringSubmatch(filename)
if m == nil || m[2] != version {
continue
}
platform := m[3] + "_" + m[4]
archive := terraformArchive{URL: filename}
if f.ContentHash != "" {
archive.Hashes = []string{"zh:" + strings.TrimPrefix(f.ContentHash, "sha256:")}
}
archives[platform] = archive
}
if len(archives) == 0 {
http.Error(w, "not found", http.StatusNotFound)
return
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(terraformVersionDoc{Archives: archives})
}
+25 -16
View File
@@ -34,14 +34,15 @@ import (
)
type Server struct {
cfg *config.Config
router chi.Router
db *database.DB
cache *cache.Redis
store *storage.S3
engine *proxy.Engine
virtEngine *virtual.Engine
gc *gc.Collector
cfg *config.Config
router chi.Router
db *database.DB
cache *cache.Redis
store *storage.S3
engine *proxy.Engine
virtEngine *virtual.Engine
localHandler *v2.LocalHandler
gc *gc.Collector
}
func New(cfg *config.Config) (*Server, error) {
@@ -61,17 +62,19 @@ func New(cfg *config.Config) (*Server, error) {
}
engine := proxy.NewEngine(db, redis, s3)
localHandler := v2.NewLocalHandler(db, s3)
virtEngine := virtual.NewEngine(db, engine)
collector := gc.New(db, s3, 1*time.Hour)
s := &Server{
cfg: cfg,
db: db,
cache: redis,
store: s3,
engine: engine,
virtEngine: virtEngine,
gc: collector,
cfg: cfg,
db: db,
cache: redis,
store: s3,
engine: engine,
virtEngine: virtEngine,
localHandler: localHandler,
gc: collector,
}
s.router = s.routes()
@@ -91,7 +94,7 @@ func (s *Server) routes() chi.Router {
r.Get("/health", s.handleHealth)
r.Get("/", s.handleRoot)
proxyHandler := v1.NewProxyHandler(s.engine, s.virtEngine, s.db)
proxyHandler := v1.NewProxyHandler(s.engine, s.virtEngine, s.db, s.store, s.localHandler)
r.Mount("/api/v1", proxyHandler.Routes())
remotesHandler := v2.NewRemotesHandler(s.db)
@@ -114,6 +117,12 @@ func (s *Server) routes() chi.Router {
r.Get("/", objHandler.Routes().ServeHTTP)
r.Delete("/*", objHandler.Routes().ServeHTTP)
})
r.Route("/remotes/{name}/files", func(r chi.Router) {
r.Put("/*", s.localHandler.Routes().ServeHTTP)
r.Get("/*", s.localHandler.Routes().ServeHTTP)
r.Delete("/*", s.localHandler.Routes().ServeHTTP)
})
})
return r
+25 -1
View File
@@ -73,6 +73,16 @@ func (e *Engine) fetchMemberIndexes(ctx context.Context, virt models.Virtual, pa
return
}
if remote.RepoType == models.RepoTypeLocal {
body, err := e.fetchLocalIndex(ctx, *remote, path)
if err != nil {
results[idx] = result{err: fmt.Errorf("local index %q: %w", name, err)}
return
}
results[idx] = result{index: MemberIndex{RemoteName: name, RepoType: remote.RepoType, Body: body}}
return
}
prov, err := provider.Get(remote.PackageType)
if err != nil {
results[idx] = result{err: fmt.Errorf("provider %q: %w", remote.PackageType, err)}
@@ -92,7 +102,7 @@ func (e *Engine) fetchMemberIndexes(ctx context.Context, virt models.Virtual, pa
return
}
results[idx] = result{index: MemberIndex{RemoteName: name, Body: body}}
results[idx] = result{index: MemberIndex{RemoteName: name, RepoType: remote.RepoType, Body: body}}
}(i, memberName)
}
@@ -109,3 +119,17 @@ func (e *Engine) fetchMemberIndexes(ctx context.Context, virt models.Virtual, pa
return members, nil
}
func (e *Engine) fetchLocalIndex(ctx context.Context, remote models.Remote, path string) ([]byte, error) {
prov, err := provider.Get(remote.PackageType)
if err != nil {
return nil, fmt.Errorf("no provider for %q: %w", remote.PackageType, err)
}
indexer, ok := prov.(provider.LocalIndexer)
if !ok {
return nil, fmt.Errorf("provider %q does not support local index generation", remote.PackageType)
}
return indexer.GenerateLocalIndex(ctx, e.db, remote.Name, path)
}
+1
View File
@@ -8,6 +8,7 @@ import (
type MemberIndex struct {
RemoteName string
RepoType models.RepoType
Body []byte
}
+6 -1
View File
@@ -36,8 +36,13 @@ func (m *PyPIMerger) MergeIndexes(members []MemberIndex, proxyBaseURL string) ([
}
if proxyBaseURL != "" && href != "" {
href = fmt.Sprintf("%s/api/v1/remote/%s/%s",
routePrefix := "remote"
if member.RepoType == "local" {
routePrefix = "local"
}
href = fmt.Sprintf("%s/api/v1/%s/%s/%s",
strings.TrimRight(proxyBaseURL, "/"),
routePrefix,
member.RemoteName,
strings.TrimLeft(href, "/"))
}
+33 -1
View File
@@ -1,10 +1,42 @@
package models
import "time"
import (
"fmt"
"time"
)
type RepoType string
const (
RepoTypeRemote RepoType = "remote"
RepoTypeLocal RepoType = "local"
)
var validRepoTypes = map[RepoType]bool{
RepoTypeRemote: true,
RepoTypeLocal: true,
}
func (r RepoType) Valid() bool {
return validRepoTypes[r]
}
func (r RepoType) String() string {
return string(r)
}
func ParseRepoType(s string) (RepoType, error) {
rt := RepoType(s)
if !rt.Valid() {
return "", fmt.Errorf("unknown repo type: %q", s)
}
return rt, nil
}
type Remote struct {
Name string `json:"name"`
PackageType PackageType `json:"package_type"`
RepoType RepoType `json:"repo_type"`
BaseURL string `json:"base_url"`
Description string `json:"description,omitempty"`
Username string `json:"-"`
+3 -1
View File
@@ -1,4 +1,4 @@
import type { Remote, Virtual, Artifact, OverviewStats, RemoteStatRow, HealthStatus, ProbeResult } from './types';
import type { Remote, Virtual, Artifact, OverviewStats, RemoteStatRow, FileStatRow, BandwidthStatRow, HealthStatus, ProbeResult } from './types';
const BASE = '';
@@ -19,6 +19,8 @@ export const api = {
health: () => fetchJSON<HealthStatus>('/api/v2/health'),
stats: () => fetchJSON<OverviewStats>('/api/v2/stats'),
topRemotes: () => fetchJSON<RemoteStatRow[]>('/api/v2/stats/top-remotes'),
topFilesByHits: () => fetchJSON<FileStatRow[]>('/api/v2/stats/top-files-by-hits'),
topFilesByBandwidth: () => fetchJSON<BandwidthStatRow[]>('/api/v2/stats/top-files-by-bandwidth'),
listRemotes: () => fetchJSON<Remote[]>('/api/v2/remotes'),
getRemote: (name: string) => fetchJSON<Remote>(`/api/v2/remotes/${name}`),
+15
View File
@@ -1,6 +1,7 @@
export interface Remote {
name: string;
package_type: string;
repo_type: string;
base_url: string;
description: string;
username?: string;
@@ -62,6 +63,20 @@ export interface RemoteStatRow {
requests_30d: number;
}
export interface FileStatRow {
remote_name: string;
path: string;
access_count: number;
size_bytes: number;
}
export interface BandwidthStatRow {
remote_name: string;
path: string;
bandwidth: number;
requests: number;
}
export interface HealthStatus {
status: string;
postgres: string;
+35
View File
@@ -45,3 +45,38 @@
font-size: 0.9em;
padding: 32px 0;
}
.top-files-grid {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 24px;
margin-top: 8px;
}
@media (max-width: 1100px) {
.top-files-grid {
grid-template-columns: 1fr;
}
}
.file-link {
display: flex;
flex-direction: column;
gap: 2px;
text-decoration: none;
color: inherit;
}
.file-link:hover .file-path {
text-decoration: underline;
}
.file-remote {
font-size: 0.75em;
color: var(--text-muted);
}
.file-path {
font-size: 0.82em;
word-break: break-all;
}
+79 -3
View File
@@ -1,7 +1,7 @@
import { useEffect, useState } from 'react';
import { Link } from 'react-router-dom';
import { api } from '../api/client';
import type { OverviewStats, RemoteStatRow, HealthStatus } from '../api/types';
import type { OverviewStats, RemoteStatRow, FileStatRow, BandwidthStatRow, HealthStatus } from '../api/types';
import { StatsCard } from '../components/StatsCard';
import { Badge } from '../components/Badge';
import { DataTable } from '../components/DataTable';
@@ -11,14 +11,24 @@ import './Dashboard.css';
export function Dashboard() {
const [stats, setStats] = useState<OverviewStats | null>(null);
const [topRemotes, setTopRemotes] = useState<RemoteStatRow[]>([]);
const [topFilesByHits, setTopFilesByHits] = useState<FileStatRow[]>([]);
const [topFilesByBW, setTopFilesByBW] = useState<BandwidthStatRow[]>([]);
const [health, setHealth] = useState<HealthStatus | null>(null);
const [error, setError] = useState<string | null>(null);
useEffect(() => {
Promise.all([api.stats(), api.topRemotes(), api.health()])
.then(([s, tr, h]) => {
Promise.all([
api.stats(),
api.topRemotes(),
api.topFilesByHits(),
api.topFilesByBandwidth(),
api.health(),
])
.then(([s, tr, tfh, tfb, h]) => {
setStats(s);
setTopRemotes(tr || []);
setTopFilesByHits(tfh || []);
setTopFilesByBW(tfb || []);
setHealth(h);
})
.catch(e => setError(e.message));
@@ -87,6 +97,72 @@ export function Dashboard() {
data={topRemotes}
emptyMessage="No remotes configured yet"
/>
<div className="top-files-grid">
<div>
<h2 className="section-title">Top Files by Hits</h2>
<DataTable
columns={[
{
key: 'path',
header: 'File',
render: (r: FileStatRow) => (
<Link to={`/remotes/${r.remote_name}/objects`} className="file-link">
<span className="file-remote">{r.remote_name}</span>
<span className="file-path mono">{r.path}</span>
</Link>
),
},
{
key: 'hits',
header: 'Hits',
render: (r: FileStatRow) => formatNumber(r.access_count),
width: '90px',
},
{
key: 'size',
header: 'Size',
render: (r: FileStatRow) => formatBytes(r.size_bytes),
width: '100px',
},
]}
data={topFilesByHits}
emptyMessage="No cached files yet"
/>
</div>
<div>
<h2 className="section-title">Top Files by Bandwidth (30d)</h2>
<DataTable
columns={[
{
key: 'path',
header: 'File',
render: (r: BandwidthStatRow) => (
<Link to={`/remotes/${r.remote_name}/objects`} className="file-link">
<span className="file-remote">{r.remote_name}</span>
<span className="file-path mono">{r.path}</span>
</Link>
),
},
{
key: 'bandwidth',
header: 'Bandwidth',
render: (r: BandwidthStatRow) => formatBytes(r.bandwidth),
width: '110px',
},
{
key: 'requests',
header: 'Requests',
render: (r: BandwidthStatRow) => formatNumber(r.requests),
width: '100px',
},
]}
data={topFilesByBW}
emptyMessage="No access data yet"
/>
</div>
</div>
</div>
);
}
+46 -15
View File
@@ -3,13 +3,58 @@
align-items: center;
gap: 12px;
margin-bottom: 16px;
flex-wrap: wrap;
}
.obj-path {
.tree-controls {
display: flex;
gap: 6px;
margin-left: auto;
}
.tree-table .tree-label {
display: inline-flex;
align-items: center;
gap: 4px;
}
.tree-toggle {
display: inline-block;
width: 16px;
font-size: 0.85em;
color: var(--text-muted);
user-select: none;
flex-shrink: 0;
}
.tree-dir-name {
font-weight: 600;
color: var(--text-bright);
}
.tree-file-name {
font-size: 0.85em;
word-break: break-all;
}
.tree-dir {
cursor: pointer;
}
.tree-dir:hover {
background: var(--bg-elevated);
}
.tree-summary {
font-size: 0.8em;
color: var(--text-muted);
}
.num-cell {
font-family: var(--font-mono);
font-size: 0.85em;
}
.hash-cell {
font-size: 0.8em;
color: var(--text-muted);
@@ -30,20 +75,6 @@
background: rgba(239, 68, 68, 0.15);
}
.pagination {
display: flex;
align-items: center;
justify-content: center;
gap: 16px;
margin-top: 16px;
}
.page-info {
font-size: 0.85em;
color: var(--text-muted);
font-family: var(--font-mono);
}
.btn-sm {
padding: 5px 12px;
font-size: 0.85em;
+245 -81
View File
@@ -1,25 +1,188 @@
import { useEffect, useState, useCallback } from 'react';
import { useEffect, useState, useCallback, useMemo } from 'react';
import { useParams, Link } from 'react-router-dom';
import { api } from '../api/client';
import type { Artifact } from '../api/types';
import { DataTable } from '../components/DataTable';
import { formatBytes, timeAgo, truncateHash } from '../components/format';
import './Objects.css';
interface TreeNode {
name: string;
path: string;
children: Map<string, TreeNode>;
artifact?: Artifact;
totalSize: number;
totalHits: number;
fileCount: number;
}
function buildTree(artifacts: Artifact[]): TreeNode {
const root: TreeNode = {
name: '',
path: '',
children: new Map(),
totalSize: 0,
totalHits: 0,
fileCount: 0,
};
for (const a of artifacts) {
const parts = a.path.split('/').filter(Boolean);
let node = root;
let currentPath = '';
for (let i = 0; i < parts.length; i++) {
currentPath += (currentPath ? '/' : '') + parts[i];
const isLeaf = i === parts.length - 1;
if (!node.children.has(parts[i])) {
node.children.set(parts[i], {
name: parts[i],
path: currentPath,
children: new Map(),
totalSize: 0,
totalHits: 0,
fileCount: 0,
});
}
const child = node.children.get(parts[i])!;
child.totalSize += a.size_bytes;
child.totalHits += a.access_count;
child.fileCount += 1;
if (isLeaf) {
child.artifact = a;
}
node = child;
}
root.totalSize += a.size_bytes;
root.totalHits += a.access_count;
root.fileCount += 1;
}
return root;
}
function filterTree(node: TreeNode, query: string): TreeNode | null {
if (!query) return node;
const lower = query.toLowerCase();
if (node.artifact) {
if (node.path.toLowerCase().includes(lower)) return node;
return null;
}
const filtered: Map<string, TreeNode> = new Map();
let totalSize = 0;
let totalHits = 0;
let fileCount = 0;
for (const [key, child] of node.children) {
const result = filterTree(child, query);
if (result) {
filtered.set(key, result);
totalSize += result.totalSize;
totalHits += result.totalHits;
fileCount += result.fileCount;
}
}
if (filtered.size === 0) return null;
return { ...node, children: filtered, totalSize, totalHits, fileCount };
}
interface TreeRowProps {
node: TreeNode;
depth: number;
expanded: Set<string>;
onToggle: (path: string) => void;
onEvict: (path: string) => void;
}
function TreeRow({ node, depth, expanded, onToggle, onEvict }: TreeRowProps) {
const isDir = node.children.size > 0 && !node.artifact;
const isExpanded = expanded.has(node.path);
const sortedChildren = useMemo(() => {
if (!isDir) return [];
return Array.from(node.children.values()).sort((a, b) => {
const aIsDir = a.children.size > 0 && !a.artifact;
const bIsDir = b.children.size > 0 && !b.artifact;
if (aIsDir !== bIsDir) return aIsDir ? -1 : 1;
return a.name.localeCompare(b.name);
});
}, [node.children, isDir]);
return (
<>
<tr className={isDir ? 'tree-dir' : 'tree-file'} onClick={isDir ? () => onToggle(node.path) : undefined}>
<td>
<span style={{ paddingLeft: depth * 20 }} className="tree-label">
{isDir && (
<span className="tree-toggle">{isExpanded ? '▾' : '▸'}</span>
)}
<span className={isDir ? 'tree-dir-name' : 'mono tree-file-name'}>
{node.name}{isDir ? '/' : ''}
</span>
</span>
</td>
<td className="num-cell">{formatBytes(node.totalSize)}</td>
{node.artifact ? (
<>
<td className="mono hash-cell" title={node.artifact.content_hash}>
{truncateHash(node.artifact.content_hash)}
</td>
<td>{timeAgo(node.artifact.last_accessed_at)}</td>
<td className="num-cell">{node.artifact.access_count}</td>
<td>
<button
className="btn-evict"
onClick={(e) => { e.stopPropagation(); onEvict(node.artifact!.path); }}
>
Evict
</button>
</td>
</>
) : (
<>
<td className="tree-summary">{node.fileCount} files</td>
<td></td>
<td className="num-cell">{node.totalHits}</td>
<td></td>
</>
)}
</tr>
{isDir && isExpanded && sortedChildren.map(child => (
<TreeRow
key={child.path}
node={child}
depth={depth + 1}
expanded={expanded}
onToggle={onToggle}
onEvict={onEvict}
/>
))}
</>
);
}
export function Objects() {
const { name } = useParams<{ name: string }>();
const [artifacts, setArtifacts] = useState<Artifact[]>([]);
const [page, setPage] = useState(1);
const [loading, setLoading] = useState(true);
const [filter, setFilter] = useState('');
const [expanded, setExpanded] = useState<Set<string>>(new Set());
const load = useCallback(() => {
if (!name) return;
setLoading(true);
api.listObjects(name, page, 50)
api.listObjects(name, 1, 5000)
.then(a => setArtifacts(a || []))
.finally(() => setLoading(false));
}, [name, page]);
}, [name]);
useEffect(() => { load(); }, [load]);
@@ -29,9 +192,43 @@ export function Objects() {
load();
};
const filtered = filter
? artifacts.filter(a => a.path.toLowerCase().includes(filter.toLowerCase()))
: artifacts;
const tree = useMemo(() => buildTree(artifacts), [artifacts]);
const filtered = useMemo(() => filterTree(tree, filter), [tree, filter]);
const toggleExpand = useCallback((path: string) => {
setExpanded(prev => {
const next = new Set(prev);
if (next.has(path)) next.delete(path);
else next.add(path);
return next;
});
}, []);
const expandAll = useCallback(() => {
const allDirs = new Set<string>();
function walk(node: TreeNode) {
if (node.children.size > 0 && !node.artifact) {
allDirs.add(node.path);
for (const child of node.children.values()) walk(child);
}
}
if (filtered) walk(filtered);
setExpanded(allDirs);
}, [filtered]);
const collapseAll = useCallback(() => {
setExpanded(new Set());
}, []);
const topChildren = useMemo(() => {
if (!filtered) return [];
return Array.from(filtered.children.values()).sort((a, b) => {
const aIsDir = a.children.size > 0 && !a.artifact;
const bIsDir = b.children.size > 0 && !b.artifact;
if (aIsDir !== bIsDir) return aIsDir ? -1 : 1;
return a.name.localeCompare(b.name);
});
}, [filtered]);
return (
<div>
@@ -47,84 +244,51 @@ export function Objects() {
value={filter}
onChange={e => setFilter(e.target.value)}
/>
<span className="result-count">{filtered.length} objects</span>
<span className="result-count">
{filtered ? filtered.fileCount : 0} objects
{filtered && filtered.fileCount > 0 && <> &middot; {formatBytes(filtered.totalSize)}</>}
</span>
<div className="tree-controls">
<button className="btn btn-sm" onClick={expandAll}>Expand All</button>
<button className="btn btn-sm" onClick={collapseAll}>Collapse All</button>
</div>
</div>
{loading ? (
<div className="loading">Loading...</div>
) : (
<>
<DataTable
columns={[
{
key: 'path',
header: 'Path',
render: (a: Artifact) => <span className="mono obj-path">{a.path}</span>,
},
{
key: 'size',
header: 'Size',
render: (a: Artifact) => formatBytes(a.size_bytes),
width: '100px',
},
{
key: 'hash',
header: 'Hash',
render: (a: Artifact) => (
<span className="mono hash-cell" title={a.content_hash}>
{truncateHash(a.content_hash)}
</span>
),
width: '160px',
},
{
key: 'accessed',
header: 'Last Accessed',
render: (a: Artifact) => timeAgo(a.last_accessed_at),
width: '120px',
},
{
key: 'hits',
header: 'Hits',
render: (a: Artifact) => a.access_count,
width: '70px',
},
{
key: 'actions',
header: '',
render: (a: Artifact) => (
<button
className="btn-evict"
onClick={(e) => { e.stopPropagation(); handleEvict(a.path); }}
>
Evict
</button>
),
width: '80px',
},
]}
data={filtered}
emptyMessage="No cached objects"
/>
<div className="pagination">
<button
className="btn btn-sm"
disabled={page === 1}
onClick={() => setPage(p => Math.max(1, p - 1))}
>
Previous
</button>
<span className="page-info">Page {page}</span>
<button
className="btn btn-sm"
disabled={artifacts.length < 50}
onClick={() => setPage(p => p + 1)}
>
Next
</button>
</div>
</>
<div className="data-table-wrap">
<table className="data-table tree-table">
<thead>
<tr>
<th>Path</th>
<th style={{ width: 100 }}>Size</th>
<th style={{ width: 160 }}>Hash</th>
<th style={{ width: 120 }}>Last Accessed</th>
<th style={{ width: 70 }}>Hits</th>
<th style={{ width: 80 }}></th>
</tr>
</thead>
<tbody>
{topChildren.length === 0 ? (
<tr>
<td colSpan={6} className="data-table-empty">No cached objects</td>
</tr>
) : (
topChildren.map(child => (
<TreeRow
key={child.path}
node={child}
depth={0}
expanded={expanded}
onToggle={toggleExpand}
onEvict={handleEvict}
/>
))
)}
</tbody>
</table>
</div>
)}
</div>
);