feat: serve local terraform repos as a provider registry #102

Merged

benvin merged 2 commits from benvin/terraform-provider-registry into master 2026-07-03 18:55:35 +10:00

Conversation 1 Commits 2 Files Changed 14

+1152 -2

unkinben commented 2026-07-03 17:47:33 +10:00

Owner

Copy Link

Copy Source

Why

Local terraform repos already served the Terraform network mirror protocol, but consuming that requires every user to add a provider_installation { network_mirror } block to ~/.terraformrc. A source = "artifactapi.k8s.../ns/type" address instead triggers the provider registry protocol (service discovery at /.well-known/terraform.json + GPG-signed SHA256SUMS), which returned 404 — hence "does not offer a provider registry."

Local repos are meant to be the real thing, so this makes a terraform local repo a first-class provider registry: terraform init installs from a bare source address with no client config.

What

• Serve /.well-known/terraform.json service discovery and the providers.v1 endpoints under /terraform/v1/providers: versions, download/{os}/{arch}, sha256sums, sha256sums.sig.
• Map the Terraform namespace segment to the artifactapi repo name; locate the provider by type. download_url points back at the existing /api/v1/local/... path.
• Generate SHA256SUMS per version and sign it with a GPG key loaded from TF_SIGNING_KEY_PATH (optional TF_SIGNING_KEY_PASSPHRASE); advertise the public key + key id in the download response. No key → registry stays disabled (endpoints 404), so behaviour is unchanged until the signing secret is present.
• New internal/tfsign (key load + detached signing, via x/crypto/openpgp) and internal/api/terraform (registry handler). Export ParseProviderZip for reuse.
• TF_PROVIDER_PROTOCOLS (default 5.0,6.0) sets the advertised plugin protocols.
• README section documenting usage.

Consumer

terraform {
  required_providers {
    artifactapi = {
      source  = "artifactapi.k8s.syd1.au.unkin.net/terraform-unkin/artifactapi"
      version = "0.1.2"
    }
  }
}

Tests

• internal/tfsign: sign + verify round-trip, disabled/missing-key paths.
• internal/api/terraform: dockerised full flow (discovery → versions → download → sha256sums → sig), verifying the signature against the advertised public key.

Follow-ups (separate PRs)

• argocd-apps: mount the signing K8s secret into the api deployment + set TF_SIGNING_KEY_PATH. The / HTTPRoute already routes /.well-known and /terraform to the API, so no gateway change is needed.
• Image/version bump once tagged.

Note

Anchored the terraform/ gitignore to the repo root (/terraform/) so it stops matching internal/*/terraform/. This surfaced internal/provider/terraform/terraform_extra_test.go, which had been silently untracked — now committed.

## Why Local terraform repos already served the Terraform **network mirror** protocol, but consuming that requires every user to add a `provider_installation { network_mirror }` block to `~/.terraformrc`. A `source = "artifactapi.k8s.../ns/type"` address instead triggers the **provider registry** protocol (service discovery at `/.well-known/terraform.json` + GPG-signed SHA256SUMS), which returned 404 — hence *"does not offer a provider registry."* Local repos are meant to be the real thing, so this makes a terraform local repo a first-class provider registry: `terraform init` installs from a bare source address with no client config. ## What - Serve `/.well-known/terraform.json` service discovery and the `providers.v1` endpoints under `/terraform/v1/providers`: `versions`, `download/{os}/{arch}`, `sha256sums`, `sha256sums.sig`. - Map the Terraform **namespace** segment to the artifactapi **repo name**; locate the provider by **type**. `download_url` points back at the existing `/api/v1/local/...` path. - Generate `SHA256SUMS` per version and sign it with a GPG key loaded from `TF_SIGNING_KEY_PATH` (optional `TF_SIGNING_KEY_PASSPHRASE`); advertise the public key + key id in the download response. **No key → registry stays disabled (endpoints 404)**, so behaviour is unchanged until the signing secret is present. - New `internal/tfsign` (key load + detached signing, via `x/crypto/openpgp`) and `internal/api/terraform` (registry handler). Export `ParseProviderZip` for reuse. - `TF_PROVIDER_PROTOCOLS` (default `5.0,6.0`) sets the advertised plugin protocols. - README section documenting usage. ## Consumer ```hcl terraform { required_providers { artifactapi = { source = "artifactapi.k8s.syd1.au.unkin.net/terraform-unkin/artifactapi" version = "0.1.2" } } } ``` ## Tests - `internal/tfsign`: sign + verify round-trip, disabled/missing-key paths. - `internal/api/terraform`: dockerised full flow (discovery → versions → download → sha256sums → sig), verifying the signature against the advertised public key. ## Follow-ups (separate PRs) - **argocd-apps**: mount the signing K8s secret into the api deployment + set `TF_SIGNING_KEY_PATH`. The `/` HTTPRoute already routes `/.well-known` and `/terraform` to the API, so no gateway change is needed. - Image/version bump once tagged. ## Note Anchored the `terraform/` gitignore to the repo root (`/terraform/`) so it stops matching `internal/*/terraform/`. This surfaced `internal/provider/terraform/terraform_extra_test.go`, which had been silently untracked — now committed.

unkinben added 1 commit 2026-07-03 17:47:34 +10:00

feat: serve local terraform repos as a provider registry ... edb6c7c0f7

Local terraform repos already spoke the network mirror protocol, which needs
per-consumer .terraformrc config. This adds the provider registry protocol so
`terraform init` installs from a bare source address
(artifactapi.k8s.../{repo}/{type}) with no client setup.

- serve /.well-known/terraform.json service discovery and the providers.v1
  versions/download endpoints under /terraform/v1/providers
- map the Terraform namespace to the artifactapi repo name and locate the
  provider by type; download_url points back at the existing local file path
- generate SHA256SUMS per version and sign it with a GPG key loaded from
  TF_SIGNING_KEY_PATH; advertise the public key + key id in the download
  response. No key configured -> registry stays disabled (endpoints 404)
- new internal/tfsign (key loading + detached signing) and
  internal/api/terraform (registry handler); export ParseProviderZip for reuse
- add TF_SIGNING_KEY_PATH/PASSPHRASE and TF_PROVIDER_PROTOCOLS config
- unit test signing + verification; dockerised test of the full flow incl.
  signature verification against the advertised key

Also anchor the terraform/ gitignore to the repo root so it stops swallowing
internal/api/terraform and internal/provider/terraform test files (the latter
had gone silently untracked).

unkinben added 1 commit 2026-07-03 18:46:23 +10:00

feat: self-generate and store the terraform registry signing key ... 97cdb9c6b5

Rather than requiring an operator to create a GPG key and a K8s secret, the
registry now provisions itself: on first start artifactapi generates a signing
keypair and persists it in a new signing_keys table, so all replicas share one
key and there is nothing to set up. TF_SIGNING_KEY_PATH still overrides with a
bring-your-own key when set.

- signing_keys table + GetSigningKey / InsertSigningKeyIfAbsent (ON CONFLICT DO
  NOTHING so a replica race converges on one key)
- tfsign.Generate, LoadArmored, and LoadOrCreate(store, purpose)
- server prefers a configured key file, else LoadOrCreate against the DB
- tests: generate/load round-trip, load-or-create generates once then reuses,
  DB insert idempotency

unkinben commented 2026-07-03 18:47:17 +10:00

Author

Owner

Copy Link

Copy Source

Updated (97cdb9c): the signing key now self-provisions. On first start artifactapi generates a GPG keypair and stores it in a new signing_keys table (INSERT ON CONFLICT DO NOTHING, so replicas converge on one key). No operator setup, no K8s secret. TF_SIGNING_KEY_PATH stays as an optional bring-your-own override.

Consequences:

• The registry is enabled by default once this ships (a valid key always exists).
• argocd-apps #217 (secret mount) is closed as no longer needed.
• Note: the private key is stored plaintext in Postgres — acceptable for an internal TOFU-trust registry (DB write access is already full compromise), but worth being aware of.

Updated (97cdb9c): the signing key now **self-provisions**. On first start artifactapi generates a GPG keypair and stores it in a new `signing_keys` table (INSERT ON CONFLICT DO NOTHING, so replicas converge on one key). No operator setup, no K8s secret. `TF_SIGNING_KEY_PATH` stays as an optional bring-your-own override. Consequences: - The registry is enabled by default once this ships (a valid key always exists). - argocd-apps #217 (secret mount) is closed as no longer needed. - Note: the private key is stored plaintext in Postgres — acceptable for an internal TOFU-trust registry (DB write access is already full compromise), but worth being aware of.

benvin merged commit 936cf8846a into master 2026-07-03 18:55:35 +10:00

benvin deleted branch benvin/terraform-provider-registry 2026-07-03 18:55:36 +10:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

benvin (BenVincent) unkinben (Ben Vincent)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/artifactapi#102