benvin
c05ccfcb5d
logarchiver replaces the plain Vector archiver leg of the centralized logging stack (argocd-apps #296) with a Go service that archives raw logs from NATS JetStream to S3 as zstd-compressed, OpenPGP-encrypted, indexed objects, plus an operator CLI to search the index and retrieve/decrypt archived logs. It adds the things that outgrew Vector: zstd compression, encryption keyed from Ben's Vault GPG secrets engine, a searchable ClickHouse index, and sink-conditional acks (a batch is acknowledged to JetStream only after the object is durably in S3 AND indexed). Service (`logarchiver run`): - Durable JetStream pull consumer (stream LOGS, durable archiver, subject filter default logs.k8s.vault.>), explicit acks, independent offsets. - Batch per subject by size/count/time -> NDJSON -> zstd -> encrypt -> S3 PUT -> ClickHouse index row -> ack. On any failure the batch is Nak'd and redelivered, so nothing is lost on a sink outage. - Encryption is a wrapped-DEK envelope (container LARC1): the bulk is AES-256-GCM framed under a random data key, and only that 32-byte key is OpenPGP-encrypted to the engine's public key. This is because the Vault GPG engine does whole-payload decrypt only; retrieval round-trips just the tiny wrapped key regardless of object size. Public key fetched from the engine or a mounted file (configurable); key fingerprint recorded per object; periodic pubkey refresh for rotation. - Prometheus metrics, structured slog, graceful drain on shutdown. CLI: - `search` queries the index (subject/host/time) and lists matching objects. - `fetch` downloads, decrypts via the Vault GPG engine, unzstds and emits NDJSON (optionally re-filtered by host/time). - `init-schema` creates/prints the ClickHouse archive_index DDL. - cobra `completion` subcommands. Config via file+env (k8s-friendly, secrets from env), boundaries (NATS/S3/ ClickHouse/Vault) behind interfaces with unit tests (config, batching, host/subject extraction, crypto roundtrip with a test key, ack-after-persist with fakes, search query building). go build/vet/test -race clean; golangci-lint v2 clean. Woodpecker CI: build/test/pre-commit on PR; on v* tag a container image plus a Gitea binary release + rpm-internal RPM. Docs per subcommand + architecture + retrieval runbook + deployment drop-in. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
79 lines
3.6 KiB
Modula-2
79 lines
3.6 KiB
Modula-2
module git.unkin.net/unkin/logarchiver
|
|
|
|
go 1.25.9
|
|
|
|
require (
|
|
github.com/ClickHouse/clickhouse-go/v2 v2.47.0
|
|
github.com/ProtonMail/go-crypto v1.4.1
|
|
github.com/aws/aws-sdk-go-v2/config v1.32.31
|
|
github.com/aws/aws-sdk-go-v2/service/s3 v1.106.0
|
|
github.com/hashicorp/vault/api v1.23.0
|
|
github.com/klauspost/compress v1.19.1
|
|
github.com/nats-io/nats.go v1.52.0
|
|
github.com/prometheus/client_golang v1.24.1
|
|
github.com/spf13/cobra v1.10.2
|
|
gopkg.in/yaml.v3 v3.0.1
|
|
)
|
|
|
|
require (
|
|
github.com/ClickHouse/ch-go v0.73.0 // indirect
|
|
github.com/andybalholm/brotli v1.2.1 // indirect
|
|
github.com/aws/aws-sdk-go-v2 v1.43.0 // indirect
|
|
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 // indirect
|
|
github.com/aws/aws-sdk-go-v2/credentials v1.19.30 // indirect
|
|
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.31 // indirect
|
|
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.31 // indirect
|
|
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.31 // indirect
|
|
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.32 // indirect
|
|
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13 // indirect
|
|
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.24 // indirect
|
|
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.31 // indirect
|
|
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.32 // indirect
|
|
github.com/aws/aws-sdk-go-v2/service/signin v1.5.0 // indirect
|
|
github.com/aws/aws-sdk-go-v2/service/sso v1.33.0 // indirect
|
|
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.0 // indirect
|
|
github.com/aws/aws-sdk-go-v2/service/sts v1.45.0 // indirect
|
|
github.com/aws/smithy-go v1.27.3 // indirect
|
|
github.com/beorn7/perks v1.0.1 // indirect
|
|
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
|
|
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
|
github.com/cloudflare/circl v1.6.2 // indirect
|
|
github.com/go-faster/city v1.0.1 // indirect
|
|
github.com/go-faster/errors v0.7.1 // indirect
|
|
github.com/go-jose/go-jose/v4 v4.1.1 // indirect
|
|
github.com/google/uuid v1.6.0 // indirect
|
|
github.com/hashicorp/errwrap v1.1.0 // indirect
|
|
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
|
|
github.com/hashicorp/go-multierror v1.1.1 // indirect
|
|
github.com/hashicorp/go-retryablehttp v0.7.8 // indirect
|
|
github.com/hashicorp/go-rootcerts v1.0.2 // indirect
|
|
github.com/hashicorp/go-secure-stdlib/parseutil v0.2.0 // indirect
|
|
github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 // indirect
|
|
github.com/hashicorp/go-sockaddr v1.0.7 // indirect
|
|
github.com/hashicorp/hcl v1.0.1-vault-7 // indirect
|
|
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
|
github.com/mitchellh/go-homedir v1.1.0 // indirect
|
|
github.com/mitchellh/mapstructure v1.5.0 // indirect
|
|
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
|
|
github.com/nats-io/nkeys v0.4.15 // indirect
|
|
github.com/nats-io/nuid v1.0.1 // indirect
|
|
github.com/paulmach/orb v0.13.0 // indirect
|
|
github.com/pierrec/lz4/v4 v4.1.27 // indirect
|
|
github.com/prometheus/client_model v0.6.2 // indirect
|
|
github.com/prometheus/common v0.70.1 // indirect
|
|
github.com/prometheus/procfs v0.21.1 // indirect
|
|
github.com/ryanuber/go-glob v1.0.0 // indirect
|
|
github.com/segmentio/asm v1.2.1 // indirect
|
|
github.com/shopspring/decimal v1.4.0 // indirect
|
|
github.com/spf13/pflag v1.0.9 // indirect
|
|
go.opentelemetry.io/otel v1.44.0 // indirect
|
|
go.opentelemetry.io/otel/trace v1.44.0 // indirect
|
|
go.yaml.in/yaml/v3 v3.0.4 // indirect
|
|
golang.org/x/crypto v0.54.0 // indirect
|
|
golang.org/x/net v0.57.0 // indirect
|
|
golang.org/x/sys v0.47.0 // indirect
|
|
golang.org/x/text v0.40.0 // indirect
|
|
golang.org/x/time v0.12.0 // indirect
|
|
google.golang.org/protobuf v1.36.11 // indirect
|
|
)
|