feat: server-level GitHub machine credential for authenticated requests (#109)
ci/woodpecker/tag/docker Pipeline was successful
ci/woodpecker/tag/docker Pipeline was successful
## Why Anonymous GitHub is capped at 60 requests/hour and cannot read private repositories. A machine credential usable by a free (non-enterprise) account is needed to lift the request budget to ~5000/hr and to read private-repo release assets. Builds on the background syncer (#108, now merged to `master`); this diff is the auth changes only. ## How - Add `internal/githubauth`: a process-wide GitHub credential delivered via env/secret, applied by default to every outbound GitHub request (releases scan, ranged asset-header GETs, and the generic-github byte proxy for private assets). - Support two modes: - **PAT** — `GITHUB_TOKEN` sent as `Authorization: Bearer <token>`. - **GitHub App** — `GITHUB_APP_ID` + `GITHUB_APP_INSTALLATION_ID` + private key (`GITHUB_APP_PRIVATE_KEY` inline PEM or `GITHUB_APP_PRIVATE_KEY_PATH`). Mint a short-lived RS256 JWT with stdlib `crypto/rsa` (no new dependency), exchange it at `POST /app/installations/{id}/access_tokens` for a ~1h installation token, cache it, and single-flight a refresh a few minutes before expiry. - Inject at the two GitHub call paths: the rpm github provider header builder (releases + ranged fetches) and the generic provider `AuthHeaders` (byte proxy, github.com hosts only; the pre-signed `objects.githubusercontent.com` redirect deliberately gets no Authorization). - Honor precedence: a remote's own `username`/`password` overrides the server credential; no credential configured stays anonymous (current behavior). - Fail closed at startup on partial App configuration (e.g. App id without a private key); a token-and-App conflict is also rejected. - Never persist the credential to the DB, return it from an API, or log it (token-exchange failures never echo the response body). - Read config via the existing `getenv` convention; document PAT vs App setup, the free-account fine-grained PAT scopes (Contents:read + Metadata:read), precedence, and the rate-limit implication. ## Rate limit Authenticated requests share the syncer's single global limiter — no second limiter is added. A token raises the effective GitHub ceiling (~5000/hr vs ~60/hr), so the limiter defaults stay safe. ## Tests `internal/githubauth` and `internal/provider/{rpm,generic}`: - PAT attaches the correct `Authorization` header to releases + asset-header requests. - App mints a valid RS256 JWT (verified against the app public key), exchanges it at a mocked endpoint, reuses the cached token without re-exchanging, refreshes near expiry, and single-flights concurrent callers. - Per-remote credential overrides the server credential (rpm + generic). - No credential → no `Authorization` header, requests still succeed anonymously. - ETag/304 flow still works with auth attached. - The credential does not appear in a remote's serialized JSON. - Config validation: no-config is anonymous; partial App config and token/App conflict both error. Verified fail-before/pass-after for the injection tests. `gofmt -l`, `go build ./...`, `go vet ./...`, `go test ./...` all clean (26 packages). Reviewed-on: #109 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net>
This commit was merged in pull request #109.
This commit is contained in:
@@ -111,6 +111,71 @@ resource "artifactapi_remote_github_rpm" "acme-tools" {
|
|||||||
The repo is multi-arch (no `$basearch` needed) — `dnf` selects matching packages
|
The repo is multi-arch (no `$basearch` needed) — `dnf` selects matching packages
|
||||||
from the synthesized metadata.
|
from the synthesized metadata.
|
||||||
|
|
||||||
|
### GitHub authentication
|
||||||
|
|
||||||
|
Anonymous GitHub is capped at **60 requests/hour** and cannot read private
|
||||||
|
repositories. Configure a **server-level GitHub credential** to raise the ceiling
|
||||||
|
to roughly **5000 requests/hour** and to read private-repo release assets. The
|
||||||
|
credential is a process-wide machine identity applied by default to *every*
|
||||||
|
outbound GitHub request — the releases scan, the ranged asset-header fetches, and
|
||||||
|
the generic-github byte proxy that streams private release assets.
|
||||||
|
|
||||||
|
The credential is read from the environment (deliver it from a Vault or
|
||||||
|
Kubernetes secret). It is **never** stored per-remote in the database, **never**
|
||||||
|
returned by any API, and **never** logged. Configure **exactly one** mode.
|
||||||
|
|
||||||
|
**Precedence.** A remote's own `username`/`password` credential still wins for
|
||||||
|
that remote's requests; the server credential is the default for everything else.
|
||||||
|
With no credential configured at all, requests stay anonymous (current behavior).
|
||||||
|
Partial configuration (e.g. an App id with no private key) is a **startup error**
|
||||||
|
— artifactapi fails closed rather than silently falling back to anonymous.
|
||||||
|
|
||||||
|
Both modes share the syncer's single global rate limiter, so a token simply
|
||||||
|
raises the effective GitHub ceiling; the default limiter settings stay safe.
|
||||||
|
|
||||||
|
#### Mode 1 — Personal Access Token (minimum viable, recommended for free accounts)
|
||||||
|
|
||||||
|
Set `GITHUB_TOKEN`. It is sent as `Authorization: Bearer <token>`.
|
||||||
|
|
||||||
|
Recommended free-account setup — a **fine-grained PAT** scoped to just the target
|
||||||
|
repositories:
|
||||||
|
|
||||||
|
1. GitHub → *Settings → Developer settings → Personal access tokens →
|
||||||
|
Fine-grained tokens → Generate new token*.
|
||||||
|
2. Limit *Repository access* to the specific repo(s) serving releases.
|
||||||
|
3. Grant repository permissions **Contents: Read-only** and **Metadata:
|
||||||
|
Read-only** (Metadata is mandatory and auto-selected).
|
||||||
|
|
||||||
|
A classic PAT with the `repo` scope also works but is broader than necessary.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
GITHUB_TOKEN=github_pat_xxxxxxxx
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Mode 2 — GitHub App installation token (proper machine identity)
|
||||||
|
|
||||||
|
A GitHub App is not tied to a personal account and can be created and installed on
|
||||||
|
free personal repos. artifactapi mints a short-lived RS256 **JWT** from the app
|
||||||
|
private key, exchanges it at `POST /app/installations/{id}/access_tokens` for a
|
||||||
|
~1-hour **installation access token**, caches that token, and refreshes it a few
|
||||||
|
minutes before expiry (thread-safe, single-flighted).
|
||||||
|
|
||||||
|
1. GitHub → *Settings → Developer settings → GitHub Apps → New GitHub App*.
|
||||||
|
2. Under *Permissions → Repository permissions* grant **Contents: Read-only**
|
||||||
|
(Metadata: Read-only is implied).
|
||||||
|
3. Generate a **private key** (downloads a PEM) and note the **App ID**.
|
||||||
|
4. *Install* the App on the account and select the target repositories, then read
|
||||||
|
the **Installation ID** from the installation URL
|
||||||
|
(`.../settings/installations/<installation-id>`).
|
||||||
|
|
||||||
|
```bash
|
||||||
|
GITHUB_APP_ID=123456
|
||||||
|
GITHUB_APP_INSTALLATION_ID=7654321
|
||||||
|
GITHUB_APP_PRIVATE_KEY_PATH=/etc/artifactapi/github-app.pem
|
||||||
|
# or inline PEM (e.g. mounted from a secret):
|
||||||
|
# GITHUB_APP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----"
|
||||||
|
```
|
||||||
|
|
||||||
## Terraform
|
## Terraform
|
||||||
|
|
||||||
Remotes and virtuals are managed by Terraform. Each package type has its own resource:
|
Remotes and virtuals are managed by Terraform. Each package type has its own resource:
|
||||||
@@ -277,6 +342,11 @@ S3 client supports MinIO, Ceph RGW, and AWS S3 (via minio-go).
|
|||||||
| `GITHUB_SYNC_BURST` | `5` | Token-bucket burst for the shared limiter |
|
| `GITHUB_SYNC_BURST` | `5` | Token-bucket burst for the shared limiter |
|
||||||
| `GITHUB_SYNC_WORKERS` | `3` | Concurrent `github_rpm` scan workers |
|
| `GITHUB_SYNC_WORKERS` | `3` | Concurrent `github_rpm` scan workers |
|
||||||
| `GITHUB_SYNC_POLL_INTERVAL` | `60` | Base scheduler tick in seconds; per-remote cadence is its `mutable_ttl`, enforced by the DB lease |
|
| `GITHUB_SYNC_POLL_INTERVAL` | `60` | Base scheduler tick in seconds; per-remote cadence is its `mutable_ttl`, enforced by the DB lease |
|
||||||
|
| `GITHUB_TOKEN` | | Server-level GitHub PAT (fine-grained or classic), sent as `Authorization: Bearer`. Applies to every GitHub request; per-remote creds override it. See [GitHub authentication](#github-authentication) |
|
||||||
|
| `GITHUB_APP_ID` | | GitHub App id (App auth mode; mutually exclusive with `GITHUB_TOKEN`) |
|
||||||
|
| `GITHUB_APP_INSTALLATION_ID` | | GitHub App installation id |
|
||||||
|
| `GITHUB_APP_PRIVATE_KEY` | | GitHub App private key, inline PEM |
|
||||||
|
| `GITHUB_APP_PRIVATE_KEY_PATH` | | GitHub App private key, file path (alternative to inline PEM) |
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
|
|||||||
@@ -44,6 +44,18 @@ type Config struct {
|
|||||||
GitHubSyncBurst int
|
GitHubSyncBurst int
|
||||||
GitHubSyncWorkers int
|
GitHubSyncWorkers int
|
||||||
GitHubSyncPollInterval int
|
GitHubSyncPollInterval int
|
||||||
|
|
||||||
|
// Server-level GitHub machine credential, applied by default to every
|
||||||
|
// outbound GitHub request (releases scan, ranged asset fetches, and the
|
||||||
|
// generic-github byte proxy for private assets). Delivered via env/secret
|
||||||
|
// only — never stored per-remote, never returned by an API, never logged.
|
||||||
|
// Configure exactly one mode: a Personal Access Token, or a GitHub App
|
||||||
|
// (id + installation id + private key). Partial App config fails at startup.
|
||||||
|
GitHubToken string
|
||||||
|
GitHubAppID string
|
||||||
|
GitHubAppInstallationID string
|
||||||
|
GitHubAppPrivateKey string
|
||||||
|
GitHubAppPrivateKeyPath string
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *Config) DatabaseDSN() string {
|
func (c *Config) DatabaseDSN() string {
|
||||||
@@ -105,6 +117,12 @@ func Load() (*Config, error) {
|
|||||||
GitHubSyncBurst: syncBurst,
|
GitHubSyncBurst: syncBurst,
|
||||||
GitHubSyncWorkers: syncWorkers,
|
GitHubSyncWorkers: syncWorkers,
|
||||||
GitHubSyncPollInterval: syncPoll,
|
GitHubSyncPollInterval: syncPoll,
|
||||||
|
|
||||||
|
GitHubToken: getenv("GITHUB_TOKEN", ""),
|
||||||
|
GitHubAppID: getenv("GITHUB_APP_ID", ""),
|
||||||
|
GitHubAppInstallationID: getenv("GITHUB_APP_INSTALLATION_ID", ""),
|
||||||
|
GitHubAppPrivateKey: getenv("GITHUB_APP_PRIVATE_KEY", ""),
|
||||||
|
GitHubAppPrivateKeyPath: getenv("GITHUB_APP_PRIVATE_KEY_PATH", ""),
|
||||||
}
|
}
|
||||||
|
|
||||||
return cfg, nil
|
return cfg, nil
|
||||||
|
|||||||
@@ -0,0 +1,199 @@
|
|||||||
|
package githubauth
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto"
|
||||||
|
"crypto/rand"
|
||||||
|
"crypto/rsa"
|
||||||
|
"crypto/sha256"
|
||||||
|
"crypto/x509"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"encoding/pem"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
defaultAPIBase = "https://api.github.com"
|
||||||
|
|
||||||
|
// jwtLifetime is how long the app JWT is valid. GitHub caps it at 10 minutes;
|
||||||
|
// 9 leaves headroom for clock skew.
|
||||||
|
jwtLifetime = 9 * time.Minute
|
||||||
|
// jwtBackdate backdates iat to tolerate the app server's clock running behind
|
||||||
|
// GitHub's, which otherwise rejects the JWT.
|
||||||
|
jwtBackdate = 60 * time.Second
|
||||||
|
// refreshSkew refreshes the installation token this long before it expires so
|
||||||
|
// a request never races an expiry.
|
||||||
|
refreshSkew = 5 * time.Minute
|
||||||
|
)
|
||||||
|
|
||||||
|
type httpDoer interface {
|
||||||
|
Do(*http.Request) (*http.Response, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// appCredential mints installation access tokens for a GitHub App. It signs a
|
||||||
|
// short-lived RS256 JWT with the app private key, exchanges it for a ~1h
|
||||||
|
// installation token, caches that token, and refreshes it shortly before expiry.
|
||||||
|
// Refreshes are single-flighted by holding the mutex across the exchange, so
|
||||||
|
// concurrent callers coalesce onto one HTTP request and reuse the cached token.
|
||||||
|
type appCredential struct {
|
||||||
|
appID string
|
||||||
|
installationID string
|
||||||
|
key *rsa.PrivateKey
|
||||||
|
apiBase string
|
||||||
|
client httpDoer
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
token string
|
||||||
|
expiry time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
func newAppCredential(opts Options) (*appCredential, error) {
|
||||||
|
if opts.AppID == "" {
|
||||||
|
return nil, errors.New("github app: GITHUB_APP_ID is required")
|
||||||
|
}
|
||||||
|
if opts.InstallationID == "" {
|
||||||
|
return nil, errors.New("github app: GITHUB_APP_INSTALLATION_ID is required")
|
||||||
|
}
|
||||||
|
pemBytes, err := loadPrivateKeyPEM(opts)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
key, err := parseRSAPrivateKey(pemBytes)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
apiBase := opts.apiBaseURL
|
||||||
|
if apiBase == "" {
|
||||||
|
apiBase = defaultAPIBase
|
||||||
|
}
|
||||||
|
client := opts.httpClient
|
||||||
|
if client == nil {
|
||||||
|
client = &http.Client{Timeout: 30 * time.Second}
|
||||||
|
}
|
||||||
|
|
||||||
|
return &appCredential{
|
||||||
|
appID: opts.AppID,
|
||||||
|
installationID: opts.InstallationID,
|
||||||
|
key: key,
|
||||||
|
apiBase: strings.TrimRight(apiBase, "/"),
|
||||||
|
client: client,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Token returns a cached installation token, refreshing it under a single-flight
|
||||||
|
// lock when it is missing or within refreshSkew of expiry.
|
||||||
|
func (a *appCredential) Token(ctx context.Context) (string, error) {
|
||||||
|
a.mu.Lock()
|
||||||
|
defer a.mu.Unlock()
|
||||||
|
if a.token != "" && time.Now().Before(a.expiry.Add(-refreshSkew)) {
|
||||||
|
return a.token, nil
|
||||||
|
}
|
||||||
|
if err := a.refreshLocked(ctx); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return a.token, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *appCredential) refreshLocked(ctx context.Context) error {
|
||||||
|
jwt, err := mintJWT(a.appID, a.key, time.Now())
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("github app: mint jwt: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
u := fmt.Sprintf("%s/app/installations/%s/access_tokens", a.apiBase, a.installationID)
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodPost, u, nil)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
req.Header.Set("Authorization", "Bearer "+jwt)
|
||||||
|
req.Header.Set("Accept", "application/vnd.github+json")
|
||||||
|
req.Header.Set("X-GitHub-Api-Version", "2022-11-28")
|
||||||
|
|
||||||
|
resp, err := a.client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("github app: token exchange: %w", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
body, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||||
|
if resp.StatusCode != http.StatusCreated && resp.StatusCode != http.StatusOK {
|
||||||
|
// Never echo the body verbatim — it can contain sensitive material.
|
||||||
|
return fmt.Errorf("github app: token exchange status %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
var out struct {
|
||||||
|
Token string `json:"token"`
|
||||||
|
ExpiresAt time.Time `json:"expires_at"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(body, &out); err != nil {
|
||||||
|
return fmt.Errorf("github app: decode token response: %w", err)
|
||||||
|
}
|
||||||
|
if out.Token == "" {
|
||||||
|
return errors.New("github app: token exchange returned an empty token")
|
||||||
|
}
|
||||||
|
a.token = out.Token
|
||||||
|
a.expiry = out.ExpiresAt
|
||||||
|
if a.expiry.IsZero() {
|
||||||
|
// Defensive: assume the documented ~1h lifetime if GitHub omits it.
|
||||||
|
a.expiry = time.Now().Add(time.Hour)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// mintJWT builds and RS256-signs a GitHub App JWT (iss=app id, backdated iat,
|
||||||
|
// ≤10m exp) using stdlib crypto — no third-party JWT dependency.
|
||||||
|
func mintJWT(appID string, key *rsa.PrivateKey, now time.Time) (string, error) {
|
||||||
|
header := map[string]string{"alg": "RS256", "typ": "JWT"}
|
||||||
|
claims := map[string]any{
|
||||||
|
"iat": now.Add(-jwtBackdate).Unix(),
|
||||||
|
"exp": now.Add(jwtLifetime).Unix(),
|
||||||
|
"iss": appID,
|
||||||
|
}
|
||||||
|
hb, err := json.Marshal(header)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
cb, err := json.Marshal(claims)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
signingInput := b64url(hb) + "." + b64url(cb)
|
||||||
|
digest := sha256.Sum256([]byte(signingInput))
|
||||||
|
sig, err := rsa.SignPKCS1v15(rand.Reader, key, crypto.SHA256, digest[:])
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return signingInput + "." + b64url(sig), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func b64url(b []byte) string {
|
||||||
|
return base64.RawURLEncoding.EncodeToString(b)
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseRSAPrivateKey accepts PKCS#1 ("RSA PRIVATE KEY") and PKCS#8 ("PRIVATE
|
||||||
|
// KEY") PEM, covering both GitHub App key export formats.
|
||||||
|
func parseRSAPrivateKey(pemBytes []byte) (*rsa.PrivateKey, error) {
|
||||||
|
block, _ := pem.Decode(pemBytes)
|
||||||
|
if block == nil {
|
||||||
|
return nil, errors.New("github app: private key is not valid PEM")
|
||||||
|
}
|
||||||
|
if key, err := x509.ParsePKCS1PrivateKey(block.Bytes); err == nil {
|
||||||
|
return key, nil
|
||||||
|
}
|
||||||
|
keyAny, err := x509.ParsePKCS8PrivateKey(block.Bytes)
|
||||||
|
if err != nil {
|
||||||
|
return nil, errors.New("github app: private key is not a supported RSA PKCS#1/PKCS#8 key")
|
||||||
|
}
|
||||||
|
rsaKey, ok := keyAny.(*rsa.PrivateKey)
|
||||||
|
if !ok {
|
||||||
|
return nil, errors.New("github app: private key is not an RSA key")
|
||||||
|
}
|
||||||
|
return rsaKey, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,207 @@
|
|||||||
|
package githubauth
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto"
|
||||||
|
"crypto/rand"
|
||||||
|
"crypto/rsa"
|
||||||
|
"crypto/sha256"
|
||||||
|
"crypto/x509"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"encoding/pem"
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func testRSAKeyPEM(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
key, err := rsa.GenerateKey(rand.Reader, 2048)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("generate key: %v", err)
|
||||||
|
}
|
||||||
|
der := x509.MarshalPKCS1PrivateKey(key)
|
||||||
|
return string(pem.EncodeToMemory(&pem.Block{Type: "RSA PRIVATE KEY", Bytes: der}))
|
||||||
|
}
|
||||||
|
|
||||||
|
// appFixture serves the installation-token exchange endpoint, records requests,
|
||||||
|
// verifies the presented JWT against the app public key, and returns tokens with
|
||||||
|
// a controllable expiry.
|
||||||
|
type appFixture struct {
|
||||||
|
srv *httptest.Server
|
||||||
|
pub *rsa.PublicKey
|
||||||
|
mu sync.Mutex
|
||||||
|
exchanges int
|
||||||
|
lastJWT string
|
||||||
|
expiresAt func() time.Time
|
||||||
|
tokenSeq int
|
||||||
|
}
|
||||||
|
|
||||||
|
func newAppFixture(t *testing.T, pemKey string) *appFixture {
|
||||||
|
t.Helper()
|
||||||
|
block, _ := pem.Decode([]byte(pemKey))
|
||||||
|
key, err := x509.ParsePKCS1PrivateKey(block.Bytes)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse test key: %v", err)
|
||||||
|
}
|
||||||
|
f := &appFixture{
|
||||||
|
pub: &key.PublicKey,
|
||||||
|
expiresAt: func() time.Time { return time.Now().Add(time.Hour) },
|
||||||
|
}
|
||||||
|
mux := http.NewServeMux()
|
||||||
|
mux.HandleFunc("/app/installations/456/access_tokens", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
auth := r.Header.Get("Authorization")
|
||||||
|
jwt := strings.TrimPrefix(auth, "Bearer ")
|
||||||
|
f.mu.Lock()
|
||||||
|
f.exchanges++
|
||||||
|
f.lastJWT = jwt
|
||||||
|
f.tokenSeq++
|
||||||
|
seq := f.tokenSeq
|
||||||
|
exp := f.expiresAt()
|
||||||
|
f.mu.Unlock()
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.WriteHeader(http.StatusCreated)
|
||||||
|
json.NewEncoder(w).Encode(map[string]any{
|
||||||
|
"token": fmt.Sprintf("ghs_installation_%d", seq),
|
||||||
|
"expires_at": exp.UTC().Format(time.RFC3339),
|
||||||
|
})
|
||||||
|
})
|
||||||
|
f.srv = httptest.NewServer(mux)
|
||||||
|
t.Cleanup(f.srv.Close)
|
||||||
|
return f
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *appFixture) verifyJWT(t *testing.T) {
|
||||||
|
t.Helper()
|
||||||
|
f.mu.Lock()
|
||||||
|
jwt := f.lastJWT
|
||||||
|
f.mu.Unlock()
|
||||||
|
parts := strings.Split(jwt, ".")
|
||||||
|
if len(parts) != 3 {
|
||||||
|
t.Fatalf("jwt not three-part: %q", jwt)
|
||||||
|
}
|
||||||
|
signingInput := parts[0] + "." + parts[1]
|
||||||
|
sig, err := base64.RawURLEncoding.DecodeString(parts[2])
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("decode sig: %v", err)
|
||||||
|
}
|
||||||
|
digest := sha256.Sum256([]byte(signingInput))
|
||||||
|
if err := rsa.VerifyPKCS1v15(f.pub, crypto.SHA256, digest[:], sig); err != nil {
|
||||||
|
t.Fatalf("jwt signature invalid: %v", err)
|
||||||
|
}
|
||||||
|
var claims struct {
|
||||||
|
Iss string `json:"iss"`
|
||||||
|
Iat int64 `json:"iat"`
|
||||||
|
Exp int64 `json:"exp"`
|
||||||
|
}
|
||||||
|
cb, _ := base64.RawURLEncoding.DecodeString(parts[1])
|
||||||
|
if err := json.Unmarshal(cb, &claims); err != nil {
|
||||||
|
t.Fatalf("decode claims: %v", err)
|
||||||
|
}
|
||||||
|
if claims.Iss != "123" {
|
||||||
|
t.Fatalf("iss = %q, want 123", claims.Iss)
|
||||||
|
}
|
||||||
|
if claims.Exp-claims.Iat > int64((10*time.Minute)/time.Second) {
|
||||||
|
t.Fatalf("jwt lifetime exceeds 10m: iat=%d exp=%d", claims.Iat, claims.Exp)
|
||||||
|
}
|
||||||
|
if claims.Iat > time.Now().Unix() {
|
||||||
|
t.Fatalf("iat not backdated: %d", claims.Iat)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newAppCred(t *testing.T, f *appFixture, pemKey string) *appCredential {
|
||||||
|
t.Helper()
|
||||||
|
c, err := newAppCredential(Options{
|
||||||
|
AppID: "123",
|
||||||
|
InstallationID: "456",
|
||||||
|
PrivateKeyPEM: pemKey,
|
||||||
|
apiBaseURL: f.srv.URL,
|
||||||
|
httpClient: f.srv.Client(),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("newAppCredential: %v", err)
|
||||||
|
}
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestApp_MintsJWTAndExchangesForInstallationToken(t *testing.T) {
|
||||||
|
pemKey := testRSAKeyPEM(t)
|
||||||
|
f := newAppFixture(t, pemKey)
|
||||||
|
c := newAppCred(t, f, pemKey)
|
||||||
|
|
||||||
|
tok, err := c.Token(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("token: %v", err)
|
||||||
|
}
|
||||||
|
if tok != "ghs_installation_1" {
|
||||||
|
t.Fatalf("token = %q, want ghs_installation_1", tok)
|
||||||
|
}
|
||||||
|
if f.exchanges != 1 {
|
||||||
|
t.Fatalf("exchanges = %d, want 1", f.exchanges)
|
||||||
|
}
|
||||||
|
f.verifyJWT(t)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestApp_CachesInstallationToken(t *testing.T) {
|
||||||
|
pemKey := testRSAKeyPEM(t)
|
||||||
|
f := newAppFixture(t, pemKey)
|
||||||
|
c := newAppCred(t, f, pemKey)
|
||||||
|
|
||||||
|
for i := 0; i < 5; i++ {
|
||||||
|
if _, err := c.Token(context.Background()); err != nil {
|
||||||
|
t.Fatalf("token: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if f.exchanges != 1 {
|
||||||
|
t.Fatalf("exchanges = %d, want 1 (token should be cached)", f.exchanges)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestApp_RefreshesNearExpiry(t *testing.T) {
|
||||||
|
pemKey := testRSAKeyPEM(t)
|
||||||
|
f := newAppFixture(t, pemKey)
|
||||||
|
// Token expires within refreshSkew, so every call must re-exchange.
|
||||||
|
f.expiresAt = func() time.Time { return time.Now().Add(2 * time.Minute) }
|
||||||
|
c := newAppCred(t, f, pemKey)
|
||||||
|
|
||||||
|
t1, err := c.Token(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("token 1: %v", err)
|
||||||
|
}
|
||||||
|
t2, err := c.Token(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("token 2: %v", err)
|
||||||
|
}
|
||||||
|
if f.exchanges != 2 {
|
||||||
|
t.Fatalf("exchanges = %d, want 2 (near-expiry token must refresh)", f.exchanges)
|
||||||
|
}
|
||||||
|
if t1 == t2 {
|
||||||
|
t.Fatalf("expected a fresh token after refresh, both = %q", t1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestApp_ConcurrentTokenSingleFlights(t *testing.T) {
|
||||||
|
pemKey := testRSAKeyPEM(t)
|
||||||
|
f := newAppFixture(t, pemKey)
|
||||||
|
c := newAppCred(t, f, pemKey)
|
||||||
|
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := 0; i < 20; i++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
if _, err := c.Token(context.Background()); err != nil {
|
||||||
|
t.Errorf("token: %v", err)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
if f.exchanges != 1 {
|
||||||
|
t.Fatalf("exchanges = %d, want 1 (concurrent calls must coalesce)", f.exchanges)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,106 @@
|
|||||||
|
// Package githubauth provides the process-wide GitHub machine credential used to
|
||||||
|
// authenticate every outbound GitHub request (releases scan, ranged asset header
|
||||||
|
// fetches, and the generic-github byte proxy for private assets). The credential
|
||||||
|
// is delivered via env/secret only — it is never stored per-remote in the DB,
|
||||||
|
// never returned by any API, and never logged.
|
||||||
|
package githubauth
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Credential yields a bearer token for GitHub requests. Token may block to mint
|
||||||
|
// or refresh (the GitHub App path); an empty string means "no auth", which only
|
||||||
|
// happens when no credential is configured.
|
||||||
|
type Credential interface {
|
||||||
|
Token(ctx context.Context) (string, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Options is the raw, env-sourced auth configuration. Exactly one mode may be
|
||||||
|
// configured: a static token, or a GitHub App (id + installation id + private
|
||||||
|
// key). Partial App configuration is an error (fail closed); no fields at all is
|
||||||
|
// fine and yields a nil credential (anonymous, current behavior).
|
||||||
|
type Options struct {
|
||||||
|
// Token is a Personal Access Token (fine-grained or classic) sent verbatim
|
||||||
|
// as "Authorization: Bearer <token>".
|
||||||
|
Token string
|
||||||
|
|
||||||
|
// GitHub App fields. PrivateKeyPEM and PrivateKeyPath are alternatives; the
|
||||||
|
// inline PEM wins when both are set.
|
||||||
|
AppID string
|
||||||
|
InstallationID string
|
||||||
|
PrivateKeyPEM string
|
||||||
|
PrivateKeyPath string
|
||||||
|
|
||||||
|
// apiBaseURL overrides https://api.github.com for tests. Empty uses the real
|
||||||
|
// endpoint. httpClient likewise overrides the default client for tests.
|
||||||
|
apiBaseURL string
|
||||||
|
httpClient httpDoer
|
||||||
|
}
|
||||||
|
|
||||||
|
// New builds the process credential from options, validating that auth is either
|
||||||
|
// fully configured or fully absent. It returns (nil, nil) when nothing is set.
|
||||||
|
func New(opts Options) (Credential, error) {
|
||||||
|
hasToken := opts.Token != ""
|
||||||
|
hasAppField := opts.AppID != "" || opts.InstallationID != "" ||
|
||||||
|
opts.PrivateKeyPEM != "" || opts.PrivateKeyPath != ""
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case !hasToken && !hasAppField:
|
||||||
|
return nil, nil // no auth configured — anonymous is fine
|
||||||
|
case hasToken && hasAppField:
|
||||||
|
return nil, errors.New("github auth: both a token and GitHub App fields are set; configure exactly one")
|
||||||
|
case hasToken:
|
||||||
|
return staticToken{token: opts.Token}, nil
|
||||||
|
default:
|
||||||
|
return newAppCredential(opts)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// staticToken is a fixed PAT credential.
|
||||||
|
type staticToken struct{ token string }
|
||||||
|
|
||||||
|
func (s staticToken) Token(context.Context) (string, error) { return s.token, nil }
|
||||||
|
|
||||||
|
// server is the process-wide credential set once at startup. A nil value means
|
||||||
|
// no server credential (anonymous). Access is guarded so a late SetServer in a
|
||||||
|
// test is race-free.
|
||||||
|
var (
|
||||||
|
serverMu sync.RWMutex
|
||||||
|
server Credential
|
||||||
|
)
|
||||||
|
|
||||||
|
// SetServer installs the process credential. Call once during startup.
|
||||||
|
func SetServer(c Credential) {
|
||||||
|
serverMu.Lock()
|
||||||
|
server = c
|
||||||
|
serverMu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Server returns the process credential, or nil if none is configured.
|
||||||
|
func Server() Credential {
|
||||||
|
serverMu.RLock()
|
||||||
|
defer serverMu.RUnlock()
|
||||||
|
return server
|
||||||
|
}
|
||||||
|
|
||||||
|
// loadPrivateKeyPEM resolves the App private key bytes from the inline PEM or a
|
||||||
|
// file path, without ever returning the key material in an error message.
|
||||||
|
func loadPrivateKeyPEM(opts Options) ([]byte, error) {
|
||||||
|
if strings.TrimSpace(opts.PrivateKeyPEM) != "" {
|
||||||
|
return []byte(opts.PrivateKeyPEM), nil
|
||||||
|
}
|
||||||
|
if opts.PrivateKeyPath != "" {
|
||||||
|
b, err := os.ReadFile(opts.PrivateKeyPath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("github app: read private key file: %w", err)
|
||||||
|
}
|
||||||
|
return b, nil
|
||||||
|
}
|
||||||
|
return nil, errors.New("github app: no private key configured")
|
||||||
|
}
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
package githubauth
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestNew_NoConfigIsAnonymous(t *testing.T) {
|
||||||
|
c, err := New(Options{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
if c != nil {
|
||||||
|
t.Fatalf("expected nil credential when nothing configured, got %T", c)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNew_TokenMode(t *testing.T) {
|
||||||
|
c, err := New(Options{Token: "ghp_example"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
tok, err := c.Token(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("token: %v", err)
|
||||||
|
}
|
||||||
|
if tok != "ghp_example" {
|
||||||
|
t.Fatalf("token = %q, want ghp_example", tok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNew_TokenAndAppConflict(t *testing.T) {
|
||||||
|
_, err := New(Options{Token: "ghp_example", AppID: "123"})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error when both token and app fields are set")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNew_PartialAppFailsClosed(t *testing.T) {
|
||||||
|
cases := map[string]Options{
|
||||||
|
"app id without key": {AppID: "123", InstallationID: "456"},
|
||||||
|
"key without app id": {InstallationID: "456", PrivateKeyPEM: testRSAKeyPEM(t)},
|
||||||
|
"app id without inst": {AppID: "123", PrivateKeyPEM: testRSAKeyPEM(t)},
|
||||||
|
}
|
||||||
|
for name, opts := range cases {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
if _, err := New(opts); err == nil {
|
||||||
|
t.Fatalf("expected fail-closed error for %q", name)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNew_AppModeParsesKey(t *testing.T) {
|
||||||
|
c, err := New(Options{
|
||||||
|
AppID: "123",
|
||||||
|
InstallationID: "456",
|
||||||
|
PrivateKeyPEM: testRSAKeyPEM(t),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
if _, ok := c.(*appCredential); !ok {
|
||||||
|
t.Fatalf("expected *appCredential, got %T", c)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNew_AppModeRejectsBadKey(t *testing.T) {
|
||||||
|
_, err := New(Options{
|
||||||
|
AppID: "123",
|
||||||
|
InstallationID: "456",
|
||||||
|
PrivateKeyPEM: "-----BEGIN RSA PRIVATE KEY-----\nnope\n-----END RSA PRIVATE KEY-----",
|
||||||
|
})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for malformed private key")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -4,9 +4,11 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/base64"
|
"encoding/base64"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"net/url"
|
||||||
"path"
|
"path"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/githubauth"
|
||||||
"git.unkin.net/unkin/artifactapi/internal/provider"
|
"git.unkin.net/unkin/artifactapi/internal/provider"
|
||||||
"git.unkin.net/unkin/artifactapi/pkg/models"
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
)
|
)
|
||||||
@@ -59,10 +61,42 @@ func (p *Provider) RewriteResponse(_ []byte, _ models.Remote, _ string) ([]byte,
|
|||||||
return nil, nil
|
return nil, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *Provider) AuthHeaders(_ context.Context, remote models.Remote) (http.Header, error) {
|
// AuthHeaders authenticates outbound requests. A per-remote username/password
|
||||||
|
// (Basic auth) takes precedence. Otherwise, when the remote points at a GitHub
|
||||||
|
// host (e.g. a releases_remote proxying private release assets), the process-wide
|
||||||
|
// GitHub credential is attached as a bearer token so private downloads work.
|
||||||
|
func (p *Provider) AuthHeaders(ctx context.Context, remote models.Remote) (http.Header, error) {
|
||||||
h := http.Header{}
|
h := http.Header{}
|
||||||
if remote.Username != "" {
|
if remote.Username != "" {
|
||||||
h.Set("Authorization", "Basic "+base64.StdEncoding.EncodeToString([]byte(remote.Username+":"+remote.Password)))
|
h.Set("Authorization", "Basic "+base64.StdEncoding.EncodeToString([]byte(remote.Username+":"+remote.Password)))
|
||||||
|
return h, nil
|
||||||
|
}
|
||||||
|
if isGitHubHost(remote.BaseURL) {
|
||||||
|
if c := githubauth.Server(); c != nil {
|
||||||
|
tok, err := c.Token(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if tok != "" {
|
||||||
|
h.Set("Authorization", "Bearer "+tok)
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return h, nil
|
return h, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// isGitHubHost reports whether rawURL targets a GitHub API/download host that
|
||||||
|
// accepts the server credential. objects.githubusercontent.com is deliberately
|
||||||
|
// excluded: release-asset downloads 302-redirect there with a pre-signed URL
|
||||||
|
// that must not carry an Authorization header.
|
||||||
|
func isGitHubHost(rawURL string) bool {
|
||||||
|
u, err := url.Parse(rawURL)
|
||||||
|
if err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
switch strings.ToLower(u.Hostname()) {
|
||||||
|
case "github.com", "www.github.com", "api.github.com", "codeload.github.com", "uploads.github.com":
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|||||||
@@ -4,11 +4,56 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/githubauth"
|
||||||
"git.unkin.net/unkin/artifactapi/internal/provider"
|
"git.unkin.net/unkin/artifactapi/internal/provider"
|
||||||
"git.unkin.net/unkin/artifactapi/internal/provider/generic"
|
"git.unkin.net/unkin/artifactapi/internal/provider/generic"
|
||||||
"git.unkin.net/unkin/artifactapi/pkg/models"
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
type staticCred string
|
||||||
|
|
||||||
|
func (s staticCred) Token(context.Context) (string, error) { return string(s), nil }
|
||||||
|
|
||||||
|
func TestProvider_AuthHeaders_GitHubServerCredential(t *testing.T) {
|
||||||
|
githubauth.SetServer(staticCred("ghs_server"))
|
||||||
|
t.Cleanup(func() { githubauth.SetServer(nil) })
|
||||||
|
|
||||||
|
p := &generic.Provider{}
|
||||||
|
h, err := p.AuthHeaders(context.Background(), models.Remote{BaseURL: "https://github.com"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("auth headers: %v", err)
|
||||||
|
}
|
||||||
|
if h.Get("Authorization") != "Bearer ghs_server" {
|
||||||
|
t.Fatalf("Authorization = %q, want Bearer ghs_server", h.Get("Authorization"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProvider_AuthHeaders_NonGitHubHostNoServerCredential(t *testing.T) {
|
||||||
|
githubauth.SetServer(staticCred("ghs_server"))
|
||||||
|
t.Cleanup(func() { githubauth.SetServer(nil) })
|
||||||
|
|
||||||
|
p := &generic.Provider{}
|
||||||
|
h, _ := p.AuthHeaders(context.Background(), models.Remote{BaseURL: "https://example.com/downloads"})
|
||||||
|
if h.Get("Authorization") != "" {
|
||||||
|
t.Fatalf("server credential must not be sent to non-github host, got %q", h.Get("Authorization"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProvider_AuthHeaders_PerRemoteOverridesServerCredential(t *testing.T) {
|
||||||
|
githubauth.SetServer(staticCred("ghs_server"))
|
||||||
|
t.Cleanup(func() { githubauth.SetServer(nil) })
|
||||||
|
|
||||||
|
p := &generic.Provider{}
|
||||||
|
h, _ := p.AuthHeaders(context.Background(), models.Remote{
|
||||||
|
BaseURL: "https://github.com",
|
||||||
|
Username: "user",
|
||||||
|
Password: "pass",
|
||||||
|
})
|
||||||
|
if got := h.Get("Authorization"); got != "Basic dXNlcjpwYXNz" {
|
||||||
|
t.Fatalf("per-remote Basic auth must win, got %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestProvider_Type(t *testing.T) {
|
func TestProvider_Type(t *testing.T) {
|
||||||
p := &generic.Provider{}
|
p := &generic.Provider{}
|
||||||
if p.Type() != models.PackageGeneric {
|
if p.Type() != models.PackageGeneric {
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ import (
|
|||||||
rpmlib "github.com/cavaliergopher/rpm"
|
rpmlib "github.com/cavaliergopher/rpm"
|
||||||
"golang.org/x/time/rate"
|
"golang.org/x/time/rate"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/githubauth"
|
||||||
"git.unkin.net/unkin/artifactapi/internal/provider"
|
"git.unkin.net/unkin/artifactapi/internal/provider"
|
||||||
"git.unkin.net/unkin/artifactapi/pkg/models"
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
)
|
)
|
||||||
@@ -76,6 +77,10 @@ type GitHubProvider struct {
|
|||||||
// the shared background work queue instead of an inline per-replica scan.
|
// the shared background work queue instead of an inline per-replica scan.
|
||||||
syncer *Syncer
|
syncer *Syncer
|
||||||
|
|
||||||
|
// serverCred overrides the process-wide GitHub credential for this provider
|
||||||
|
// instance. nil falls back to githubauth.Server(); set directly in tests.
|
||||||
|
serverCred githubauth.Credential
|
||||||
|
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
scanning map[string]bool
|
scanning map[string]bool
|
||||||
lastScan map[string]time.Time
|
lastScan map[string]time.Time
|
||||||
@@ -136,8 +141,8 @@ func (p *GitHubProvider) RewriteResponse(_ []byte, _ models.Remote, _ string) ([
|
|||||||
return nil, nil
|
return nil, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *GitHubProvider) AuthHeaders(_ context.Context, remote models.Remote) (http.Header, error) {
|
func (p *GitHubProvider) AuthHeaders(ctx context.Context, remote models.Remote) (http.Header, error) {
|
||||||
return githubHeaders(remote, false), nil
|
return p.githubHeaders(ctx, remote, false)
|
||||||
}
|
}
|
||||||
|
|
||||||
// ServeRemote answers a request against a github_rpm remote. It refreshes the
|
// ServeRemote answers a request against a github_rpm remote. It refreshes the
|
||||||
@@ -414,7 +419,11 @@ func (p *GitHubProvider) fetchReleases(ctx context.Context, remote models.Remote
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, "", false, err
|
return nil, "", false, err
|
||||||
}
|
}
|
||||||
copyHeaders(req, githubHeaders(remote, true))
|
hdr, err := p.githubHeaders(ctx, remote, true)
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", false, err
|
||||||
|
}
|
||||||
|
copyHeaders(req, hdr)
|
||||||
if page == 1 && etag != "" {
|
if page == 1 && etag != "" {
|
||||||
req.Header.Set("If-None-Match", etag)
|
req.Header.Set("If-None-Match", etag)
|
||||||
}
|
}
|
||||||
@@ -571,7 +580,11 @@ func (p *GitHubProvider) rangeGet(ctx context.Context, remote models.Remote, dow
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, false, err
|
return nil, false, err
|
||||||
}
|
}
|
||||||
copyHeaders(req, githubHeaders(remote, false))
|
hdr, err := p.githubHeaders(ctx, remote, false)
|
||||||
|
if err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
copyHeaders(req, hdr)
|
||||||
req.Header.Set("Range", fmt.Sprintf("bytes=0-%d", n-1))
|
req.Header.Set("Range", fmt.Sprintf("bytes=0-%d", n-1))
|
||||||
|
|
||||||
if err := p.limiterWait(ctx); err != nil {
|
if err := p.limiterWait(ctx); err != nil {
|
||||||
@@ -599,7 +612,11 @@ func (p *GitHubProvider) computeSHA256(ctx context.Context, remote models.Remote
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return "", err
|
return "", err
|
||||||
}
|
}
|
||||||
copyHeaders(req, githubHeaders(remote, false))
|
hdr, err := p.githubHeaders(ctx, remote, false)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
copyHeaders(req, hdr)
|
||||||
|
|
||||||
if err := p.limiterWait(ctx); err != nil {
|
if err := p.limiterWait(ctx); err != nil {
|
||||||
return "", err
|
return "", err
|
||||||
@@ -638,23 +655,48 @@ func sha256FromDigest(digest string) (string, bool) {
|
|||||||
return "", false
|
return "", false
|
||||||
}
|
}
|
||||||
|
|
||||||
func githubHeaders(remote models.Remote, api bool) http.Header {
|
// githubHeaders builds the outbound headers for a GitHub request, attaching a
|
||||||
|
// bearer credential when one is available. A per-remote credential wins; absent
|
||||||
|
// that, the process-wide server credential is used; absent both, the request is
|
||||||
|
// unauthenticated (anonymous, subject to the 60/hr cap).
|
||||||
|
func (p *GitHubProvider) githubHeaders(ctx context.Context, remote models.Remote, api bool) (http.Header, error) {
|
||||||
h := http.Header{}
|
h := http.Header{}
|
||||||
if api {
|
if api {
|
||||||
h.Set("Accept", "application/vnd.github+json")
|
h.Set("Accept", "application/vnd.github+json")
|
||||||
h.Set("X-GitHub-Api-Version", "2022-11-28")
|
h.Set("X-GitHub-Api-Version", "2022-11-28")
|
||||||
}
|
}
|
||||||
if tok := githubToken(remote); tok != "" {
|
tok, err := p.githubToken(ctx, remote)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if tok != "" {
|
||||||
h.Set("Authorization", "Bearer "+tok)
|
h.Set("Authorization", "Bearer "+tok)
|
||||||
}
|
}
|
||||||
return h
|
return h, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func githubToken(remote models.Remote) string {
|
// githubToken resolves the bearer token for a remote. Precedence: a per-remote
|
||||||
|
// credential (password, then username) overrides the server credential.
|
||||||
|
func (p *GitHubProvider) githubToken(ctx context.Context, remote models.Remote) (string, error) {
|
||||||
if remote.Password != "" {
|
if remote.Password != "" {
|
||||||
return remote.Password
|
return remote.Password, nil
|
||||||
}
|
}
|
||||||
return remote.Username
|
if remote.Username != "" {
|
||||||
|
return remote.Username, nil
|
||||||
|
}
|
||||||
|
if c := p.serverCredential(); c != nil {
|
||||||
|
return c.Token(ctx)
|
||||||
|
}
|
||||||
|
return "", nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// serverCredential returns this provider's server credential, defaulting to the
|
||||||
|
// process-wide one installed at startup.
|
||||||
|
func (p *GitHubProvider) serverCredential() githubauth.Credential {
|
||||||
|
if p.serverCred != nil {
|
||||||
|
return p.serverCred
|
||||||
|
}
|
||||||
|
return githubauth.Server()
|
||||||
}
|
}
|
||||||
|
|
||||||
func copyHeaders(req *http.Request, h http.Header) {
|
func copyHeaders(req *http.Request, h http.Header) {
|
||||||
|
|||||||
@@ -0,0 +1,134 @@
|
|||||||
|
package rpm
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/githubauth"
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
// staticCred is a test Credential yielding a fixed token.
|
||||||
|
type staticCred string
|
||||||
|
|
||||||
|
func (s staticCred) Token(context.Context) (string, error) { return string(s), nil }
|
||||||
|
|
||||||
|
func TestGitHubServerCredentialAttachedToReleasesAndAssets(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider()
|
||||||
|
p.serverCred = staticCred("ghp_server_secret")
|
||||||
|
store := newFakeStore()
|
||||||
|
|
||||||
|
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
|
||||||
|
t.Fatalf("scan: %v", err)
|
||||||
|
}
|
||||||
|
if got := fx.releaseAuth; got != "Bearer ghp_server_secret" {
|
||||||
|
t.Fatalf("releases Authorization = %q, want Bearer ghp_server_secret", got)
|
||||||
|
}
|
||||||
|
if got := fx.assetAuth; got != "Bearer ghp_server_secret" {
|
||||||
|
t.Fatalf("asset Authorization = %q, want Bearer ghp_server_secret", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGitHubPerRemoteCredentialOverridesServer(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider()
|
||||||
|
p.serverCred = staticCred("ghp_server_secret")
|
||||||
|
store := newFakeStore()
|
||||||
|
|
||||||
|
remote := fx.remote()
|
||||||
|
remote.Password = "ghp_remote_wins"
|
||||||
|
|
||||||
|
if err := p.scan(context.Background(), remote, store); err != nil {
|
||||||
|
t.Fatalf("scan: %v", err)
|
||||||
|
}
|
||||||
|
if got := fx.releaseAuth; got != "Bearer ghp_remote_wins" {
|
||||||
|
t.Fatalf("releases Authorization = %q, want per-remote token to win", got)
|
||||||
|
}
|
||||||
|
if got := fx.assetAuth; got != "Bearer ghp_remote_wins" {
|
||||||
|
t.Fatalf("asset Authorization = %q, want per-remote token to win", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGitHubNoCredentialSendsNoAuthHeader(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider() // serverCred nil, package Server() unset in unit tests
|
||||||
|
store := newFakeStore()
|
||||||
|
|
||||||
|
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
|
||||||
|
t.Fatalf("scan: %v", err)
|
||||||
|
}
|
||||||
|
if fx.releaseAuth != "" {
|
||||||
|
t.Fatalf("expected no Authorization header, got %q", fx.releaseAuth)
|
||||||
|
}
|
||||||
|
if fx.assetAuth != "" {
|
||||||
|
t.Fatalf("expected no asset Authorization header, got %q", fx.assetAuth)
|
||||||
|
}
|
||||||
|
// Requests still succeed anonymously.
|
||||||
|
if rows, _ := store.ListRPMMetadataEntries(context.Background(), "acme-rpm"); len(rows) != 1 {
|
||||||
|
t.Fatalf("anonymous scan should still derive metadata, got %d rows", len(rows))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGitHubETag304FlowWithAuth(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
fx.etag = `"v1"`
|
||||||
|
p := newTestProvider()
|
||||||
|
p.serverCred = staticCred("ghp_server_secret")
|
||||||
|
store := newFakeStore()
|
||||||
|
|
||||||
|
etag, changed, err := p.scanWithState(context.Background(), fx.remote(), store, "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("first scan: %v", err)
|
||||||
|
}
|
||||||
|
if !changed || etag != `"v1"` {
|
||||||
|
t.Fatalf("first scan changed=%v etag=%q, want true and \"v1\"", changed, etag)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Re-scan with the captured ETag: a 304 means no change and no asset fetch.
|
||||||
|
etag2, changed2, err := p.scanWithState(context.Background(), fx.remote(), store, etag)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("second scan: %v", err)
|
||||||
|
}
|
||||||
|
if changed2 {
|
||||||
|
t.Fatal("expected no change on 304")
|
||||||
|
}
|
||||||
|
if etag2 != `"v1"` {
|
||||||
|
t.Fatalf("etag = %q, want preserved \"v1\"", etag2)
|
||||||
|
}
|
||||||
|
if fx.notModHit != 1 {
|
||||||
|
t.Fatalf("expected exactly one 304 response, got %d", fx.notModHit)
|
||||||
|
}
|
||||||
|
// The conditional request still carried the credential.
|
||||||
|
if fx.releaseAuth != "Bearer ghp_server_secret" {
|
||||||
|
t.Fatalf("conditional request Authorization = %q, want the server credential", fx.releaseAuth)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGitHubCredentialAbsentFromRemoteJSON asserts the server credential never
|
||||||
|
// appears in a remote's serialized API representation, and per-remote secrets
|
||||||
|
// stay redacted by the models.Remote json:"-" tags.
|
||||||
|
func TestGitHubCredentialAbsentFromRemoteJSON(t *testing.T) {
|
||||||
|
githubauth.SetServer(staticCred("ghp_super_secret_server_token"))
|
||||||
|
t.Cleanup(func() { githubauth.SetServer(nil) })
|
||||||
|
|
||||||
|
remote := models.Remote{
|
||||||
|
Name: "acme-rpm",
|
||||||
|
PackageType: models.PackageGitHubRPM,
|
||||||
|
BaseURL: "https://api.github.com/repos/acme/tools",
|
||||||
|
Username: "per_remote_user",
|
||||||
|
Password: "per_remote_secret",
|
||||||
|
}
|
||||||
|
b, err := json.Marshal(remote)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("marshal remote: %v", err)
|
||||||
|
}
|
||||||
|
js := string(b)
|
||||||
|
for _, secret := range []string{"ghp_super_secret_server_token", "per_remote_secret", "per_remote_user"} {
|
||||||
|
if strings.Contains(js, secret) {
|
||||||
|
t.Fatalf("credential %q leaked into remote JSON: %s", secret, js)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -73,6 +73,8 @@ type githubFixture struct {
|
|||||||
etag string // when set, served as ETag; matching If-None-Match yields 304
|
etag string // when set, served as ETag; matching If-None-Match yields 304
|
||||||
releasesHit int // total releases-list requests (200 + 304)
|
releasesHit int // total releases-list requests (200 + 304)
|
||||||
notModHit int // releases-list requests answered 304
|
notModHit int // releases-list requests answered 304
|
||||||
|
releaseAuth string // Authorization header seen on the last releases request
|
||||||
|
assetAuth string // Authorization header seen on the last asset request
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -94,6 +96,7 @@ func newGitHubFixture(t *testing.T, withDigest bool) *githubFixture {
|
|||||||
}
|
}
|
||||||
f.mu.Lock()
|
f.mu.Lock()
|
||||||
f.releasesHit++
|
f.releasesHit++
|
||||||
|
f.releaseAuth = r.Header.Get("Authorization")
|
||||||
etag := f.etag
|
etag := f.etag
|
||||||
if etag != "" && r.Header.Get("If-None-Match") == etag {
|
if etag != "" && r.Header.Get("If-None-Match") == etag {
|
||||||
f.notModHit++
|
f.notModHit++
|
||||||
@@ -130,6 +133,7 @@ func newGitHubFixture(t *testing.T, withDigest bool) *githubFixture {
|
|||||||
}
|
}
|
||||||
rng := r.Header.Get("Range")
|
rng := r.Header.Get("Range")
|
||||||
f.mu.Lock()
|
f.mu.Lock()
|
||||||
|
f.assetAuth = r.Header.Get("Authorization")
|
||||||
if rng != "" {
|
if rng != "" {
|
||||||
f.rangeHit[name]++
|
f.rangeHit[name]++
|
||||||
} else {
|
} else {
|
||||||
|
|||||||
@@ -19,6 +19,7 @@ import (
|
|||||||
"git.unkin.net/unkin/artifactapi/internal/config"
|
"git.unkin.net/unkin/artifactapi/internal/config"
|
||||||
"git.unkin.net/unkin/artifactapi/internal/database"
|
"git.unkin.net/unkin/artifactapi/internal/database"
|
||||||
"git.unkin.net/unkin/artifactapi/internal/gc"
|
"git.unkin.net/unkin/artifactapi/internal/gc"
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/githubauth"
|
||||||
_ "git.unkin.net/unkin/artifactapi/internal/provider/alpine"
|
_ "git.unkin.net/unkin/artifactapi/internal/provider/alpine"
|
||||||
_ "git.unkin.net/unkin/artifactapi/internal/provider/docker"
|
_ "git.unkin.net/unkin/artifactapi/internal/provider/docker"
|
||||||
_ "git.unkin.net/unkin/artifactapi/internal/provider/generic"
|
_ "git.unkin.net/unkin/artifactapi/internal/provider/generic"
|
||||||
@@ -66,6 +67,25 @@ func New(cfg *config.Config, version string) (*Server, error) {
|
|||||||
return nil, fmt.Errorf("s3: %w", err)
|
return nil, fmt.Errorf("s3: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Install the process-wide GitHub credential before any provider makes an
|
||||||
|
// outbound call. A misconfiguration (e.g. App id without a private key) fails
|
||||||
|
// closed here rather than silently falling back to anonymous. No credential
|
||||||
|
// configured is fine — requests stay anonymous.
|
||||||
|
ghCred, err := githubauth.New(githubauth.Options{
|
||||||
|
Token: cfg.GitHubToken,
|
||||||
|
AppID: cfg.GitHubAppID,
|
||||||
|
InstallationID: cfg.GitHubAppInstallationID,
|
||||||
|
PrivateKeyPEM: cfg.GitHubAppPrivateKey,
|
||||||
|
PrivateKeyPath: cfg.GitHubAppPrivateKeyPath,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("github auth: %w", err)
|
||||||
|
}
|
||||||
|
githubauth.SetServer(ghCred)
|
||||||
|
if ghCred != nil {
|
||||||
|
slog.Info("github machine credential configured")
|
||||||
|
}
|
||||||
|
|
||||||
engine := proxy.NewEngine(db, redis, s3)
|
engine := proxy.NewEngine(db, redis, s3)
|
||||||
localHandler := v2.NewLocalHandler(db, s3)
|
localHandler := v2.NewLocalHandler(db, s3)
|
||||||
virtEngine := virtual.NewEngine(db, engine)
|
virtEngine := virtual.NewEngine(db, engine)
|
||||||
|
|||||||
Reference in New Issue
Block a user