feat: server-level GitHub machine credential for authenticated requests (#109)
ci/woodpecker/tag/docker Pipeline was successful
ci/woodpecker/tag/docker Pipeline was successful
## Why Anonymous GitHub is capped at 60 requests/hour and cannot read private repositories. A machine credential usable by a free (non-enterprise) account is needed to lift the request budget to ~5000/hr and to read private-repo release assets. Builds on the background syncer (#108, now merged to `master`); this diff is the auth changes only. ## How - Add `internal/githubauth`: a process-wide GitHub credential delivered via env/secret, applied by default to every outbound GitHub request (releases scan, ranged asset-header GETs, and the generic-github byte proxy for private assets). - Support two modes: - **PAT** — `GITHUB_TOKEN` sent as `Authorization: Bearer <token>`. - **GitHub App** — `GITHUB_APP_ID` + `GITHUB_APP_INSTALLATION_ID` + private key (`GITHUB_APP_PRIVATE_KEY` inline PEM or `GITHUB_APP_PRIVATE_KEY_PATH`). Mint a short-lived RS256 JWT with stdlib `crypto/rsa` (no new dependency), exchange it at `POST /app/installations/{id}/access_tokens` for a ~1h installation token, cache it, and single-flight a refresh a few minutes before expiry. - Inject at the two GitHub call paths: the rpm github provider header builder (releases + ranged fetches) and the generic provider `AuthHeaders` (byte proxy, github.com hosts only; the pre-signed `objects.githubusercontent.com` redirect deliberately gets no Authorization). - Honor precedence: a remote's own `username`/`password` overrides the server credential; no credential configured stays anonymous (current behavior). - Fail closed at startup on partial App configuration (e.g. App id without a private key); a token-and-App conflict is also rejected. - Never persist the credential to the DB, return it from an API, or log it (token-exchange failures never echo the response body). - Read config via the existing `getenv` convention; document PAT vs App setup, the free-account fine-grained PAT scopes (Contents:read + Metadata:read), precedence, and the rate-limit implication. ## Rate limit Authenticated requests share the syncer's single global limiter — no second limiter is added. A token raises the effective GitHub ceiling (~5000/hr vs ~60/hr), so the limiter defaults stay safe. ## Tests `internal/githubauth` and `internal/provider/{rpm,generic}`: - PAT attaches the correct `Authorization` header to releases + asset-header requests. - App mints a valid RS256 JWT (verified against the app public key), exchanges it at a mocked endpoint, reuses the cached token without re-exchanging, refreshes near expiry, and single-flights concurrent callers. - Per-remote credential overrides the server credential (rpm + generic). - No credential → no `Authorization` header, requests still succeed anonymously. - ETag/304 flow still works with auth attached. - The credential does not appear in a remote's serialized JSON. - Config validation: no-config is anonymous; partial App config and token/App conflict both error. Verified fail-before/pass-after for the injection tests. `gofmt -l`, `go build ./...`, `go vet ./...`, `go test ./...` all clean (26 packages). Reviewed-on: #109 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net>
This commit was merged in pull request #109.
This commit is contained in:
@@ -111,6 +111,71 @@ resource "artifactapi_remote_github_rpm" "acme-tools" {
|
||||
The repo is multi-arch (no `$basearch` needed) — `dnf` selects matching packages
|
||||
from the synthesized metadata.
|
||||
|
||||
### GitHub authentication
|
||||
|
||||
Anonymous GitHub is capped at **60 requests/hour** and cannot read private
|
||||
repositories. Configure a **server-level GitHub credential** to raise the ceiling
|
||||
to roughly **5000 requests/hour** and to read private-repo release assets. The
|
||||
credential is a process-wide machine identity applied by default to *every*
|
||||
outbound GitHub request — the releases scan, the ranged asset-header fetches, and
|
||||
the generic-github byte proxy that streams private release assets.
|
||||
|
||||
The credential is read from the environment (deliver it from a Vault or
|
||||
Kubernetes secret). It is **never** stored per-remote in the database, **never**
|
||||
returned by any API, and **never** logged. Configure **exactly one** mode.
|
||||
|
||||
**Precedence.** A remote's own `username`/`password` credential still wins for
|
||||
that remote's requests; the server credential is the default for everything else.
|
||||
With no credential configured at all, requests stay anonymous (current behavior).
|
||||
Partial configuration (e.g. an App id with no private key) is a **startup error**
|
||||
— artifactapi fails closed rather than silently falling back to anonymous.
|
||||
|
||||
Both modes share the syncer's single global rate limiter, so a token simply
|
||||
raises the effective GitHub ceiling; the default limiter settings stay safe.
|
||||
|
||||
#### Mode 1 — Personal Access Token (minimum viable, recommended for free accounts)
|
||||
|
||||
Set `GITHUB_TOKEN`. It is sent as `Authorization: Bearer <token>`.
|
||||
|
||||
Recommended free-account setup — a **fine-grained PAT** scoped to just the target
|
||||
repositories:
|
||||
|
||||
1. GitHub → *Settings → Developer settings → Personal access tokens →
|
||||
Fine-grained tokens → Generate new token*.
|
||||
2. Limit *Repository access* to the specific repo(s) serving releases.
|
||||
3. Grant repository permissions **Contents: Read-only** and **Metadata:
|
||||
Read-only** (Metadata is mandatory and auto-selected).
|
||||
|
||||
A classic PAT with the `repo` scope also works but is broader than necessary.
|
||||
|
||||
```bash
|
||||
GITHUB_TOKEN=github_pat_xxxxxxxx
|
||||
```
|
||||
|
||||
#### Mode 2 — GitHub App installation token (proper machine identity)
|
||||
|
||||
A GitHub App is not tied to a personal account and can be created and installed on
|
||||
free personal repos. artifactapi mints a short-lived RS256 **JWT** from the app
|
||||
private key, exchanges it at `POST /app/installations/{id}/access_tokens` for a
|
||||
~1-hour **installation access token**, caches that token, and refreshes it a few
|
||||
minutes before expiry (thread-safe, single-flighted).
|
||||
|
||||
1. GitHub → *Settings → Developer settings → GitHub Apps → New GitHub App*.
|
||||
2. Under *Permissions → Repository permissions* grant **Contents: Read-only**
|
||||
(Metadata: Read-only is implied).
|
||||
3. Generate a **private key** (downloads a PEM) and note the **App ID**.
|
||||
4. *Install* the App on the account and select the target repositories, then read
|
||||
the **Installation ID** from the installation URL
|
||||
(`.../settings/installations/<installation-id>`).
|
||||
|
||||
```bash
|
||||
GITHUB_APP_ID=123456
|
||||
GITHUB_APP_INSTALLATION_ID=7654321
|
||||
GITHUB_APP_PRIVATE_KEY_PATH=/etc/artifactapi/github-app.pem
|
||||
# or inline PEM (e.g. mounted from a secret):
|
||||
# GITHUB_APP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----"
|
||||
```
|
||||
|
||||
## Terraform
|
||||
|
||||
Remotes and virtuals are managed by Terraform. Each package type has its own resource:
|
||||
@@ -277,6 +342,11 @@ S3 client supports MinIO, Ceph RGW, and AWS S3 (via minio-go).
|
||||
| `GITHUB_SYNC_BURST` | `5` | Token-bucket burst for the shared limiter |
|
||||
| `GITHUB_SYNC_WORKERS` | `3` | Concurrent `github_rpm` scan workers |
|
||||
| `GITHUB_SYNC_POLL_INTERVAL` | `60` | Base scheduler tick in seconds; per-remote cadence is its `mutable_ttl`, enforced by the DB lease |
|
||||
| `GITHUB_TOKEN` | | Server-level GitHub PAT (fine-grained or classic), sent as `Authorization: Bearer`. Applies to every GitHub request; per-remote creds override it. See [GitHub authentication](#github-authentication) |
|
||||
| `GITHUB_APP_ID` | | GitHub App id (App auth mode; mutually exclusive with `GITHUB_TOKEN`) |
|
||||
| `GITHUB_APP_INSTALLATION_ID` | | GitHub App installation id |
|
||||
| `GITHUB_APP_PRIVATE_KEY` | | GitHub App private key, inline PEM |
|
||||
| `GITHUB_APP_PRIVATE_KEY_PATH` | | GitHub App private key, file path (alternative to inline PEM) |
|
||||
|
||||
## Development
|
||||
|
||||
|
||||
Reference in New Issue
Block a user