unkin/argocd-apps
3
0
Fork 0
Code Issues 7 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Replace vector-archiver with logarchiver #308

Merged
benvin merged 2 commits from benvin/logarchiver-swap into main 2026-07-29 21:07:20 +10:00
Conversation 0 Commits 2 Files Changed 9
+214 -135
unkinben commented 2026-07-29 20:34:10 +10:00
Owner
Copy Link
Copy Source

Why

The Vector archiver leg wrote gzip NDJSON to S3 with no index or encryption. logarchiver replaces it with a Go service that seals raw logs to S3 as zstd + OpenPGP objects and indexes each object in ClickHouse (logs.archive_index), acking JetStream only after the object is stored and indexed.

Changes

  • Add logarchiver Deployment (git.unkin.net/unkin/logarchiver:v0.1.0), ConfigMap, and dedicated ServiceAccount, reusing the archiver's NATS (log-consumer / durable archiver / ARCHIVE_SUBJECTS=logs.k8s.vault.>), S3 (logs-archive-s3), ClickHouse (clickhouse-credentials) and vault-ca wiring.
  • Encrypts to the logarchive gpg public key, fetched from the gpg engine via k8s auth (role logging_logarchiver, projected vault-audience token). ack_wait (5m) > batch max_age (2m) so messages aren't redelivered mid-batch.
  • Add logs.archive_index DDL to the clickhouse-schema bootstrap Job (no TTL — outlives logs.raw).
  • Remove the vector-archiver Helm release, values and pipeline ConfigMap.

Cross-repo: apply terraform-vault #106 (gpg key + role/policy) before this syncs, or the pod can't fetch the public key. Sequencing: apply after #306 (already merged).

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

## Why The Vector archiver leg wrote gzip NDJSON to S3 with no index or encryption. logarchiver replaces it with a Go service that seals raw logs to S3 as zstd + OpenPGP objects and indexes each object in ClickHouse (`logs.archive_index`), acking JetStream only after the object is stored and indexed. ## Changes - Add logarchiver Deployment (`git.unkin.net/unkin/logarchiver:v0.1.0`), ConfigMap, and dedicated ServiceAccount, reusing the archiver's NATS (`log-consumer` / durable `archiver` / `ARCHIVE_SUBJECTS=logs.k8s.vault.>`), S3 (`logs-archive-s3`), ClickHouse (`clickhouse-credentials`) and `vault-ca` wiring. - Encrypts to the `logarchive` gpg public key, fetched from the gpg engine via k8s auth (role `logging_logarchiver`, projected vault-audience token). `ack_wait` (5m) > batch `max_age` (2m) so messages aren't redelivered mid-batch. - Add `logs.archive_index` DDL to the clickhouse-schema bootstrap Job (no TTL — outlives `logs.raw`). - Remove the vector-archiver Helm release, values and pipeline ConfigMap. Cross-repo: apply **terraform-vault #106** (gpg key + role/policy) before this syncs, or the pod can't fetch the public key. Sequencing: apply after #306 (already merged). https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
unkinben added 1 commit 2026-07-29 20:34:11 +10:00
Replace vector-archiver with logarchiver
ci/woodpecker/pr/vector-test Pipeline failed
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
a96c46fd6e 
The Vector archiver leg wrote gzip NDJSON to S3 with no index or encryption.
logarchiver replaces it with a Go service that seals raw logs to S3 as zstd +
OpenPGP objects and indexes each object in ClickHouse (logs.archive_index),
acking JetStream only after the object is stored and indexed.

- Add logarchiver Deployment (image git.unkin.net/unkin/logarchiver:v0.1.0),
  ConfigMap, and dedicated ServiceAccount. Reuses the archiver's NATS
  (log-consumer / durable archiver / ARCHIVE_SUBJECTS=logs.k8s.vault.>), S3
  (logs-archive-s3 BucketAccess), ClickHouse (clickhouse-credentials) and
  vault-ca wiring. Encrypts to the logarchive gpg public key, fetched from the
  gpg engine via k8s auth (role logging_logarchiver, projected vault-audience
  token). ack_wait > batch max_age so messages are not redelivered mid-batch.
- Add logs.archive_index DDL to the clickhouse-schema bootstrap Job (no TTL).
- Remove the vector-archiver Helm release, values and pipeline ConfigMap.

Cross-repo: apply terraform-vault #106 (gpg key + role/policy) before this syncs,
or the pod cannot fetch the public key. Sequencing: apply after argocd-apps #306.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
unkinben added 1 commit 2026-07-29 20:36:37 +10:00
Pin logarchiver S3 endpoint; drop archiver from vector-test
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
130fea71a1 
- Pin the RGW endpoint/bucket in the logarchiver config to the proven
  s3.ceph.unkin.net (ignore the secret's S3_ENDPOINT/BUCKET_NAME; AWS creds still
  come from the secret) to match the Vector archiver exactly.
- Remove the deleted vector/archiver.yaml from the vector-test validate step.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
benvin merged commit 9c10b9096a into main 2026-07-29 21:07:20 +10:00
benvin deleted branch benvin/logarchiver-swap 2026-07-29 21:07:20 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/argocd-apps#308
Delete Branch "benvin/logarchiver-swap"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?