Expand ASN groups from the iplocate ip-to-asn database
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful

Replace the (unconfirmed) iplocate API expander with a database-backed one that
reads the iplocate ip-to-asn CSV (network,asn,...) proxied through the artifactapi
github remote. It downloads and indexes the whole DB once (ASN -> CIDRs), serves
every asn address group from the in-memory index, and rebuilds on a 24h TTL;
refresh failures keep the last-good index (fail-safe). No API key needed.

- Add IPLocateDB expander (zip + CSV parsing, ASN normalization).
- Wire it in main (TOMSWALLAPI_IPLOCATE_DB_URL overrides the default artifactapi
  URL); remove the dead API client.
- Unit tests: CSV indexing (incl. quoted org fields), zip extraction, missing
  columns, and ASN normalization.
This commit is contained in:
benvin
2026-07-21 22:36:19 +10:00
parent dc3b2ecdb9
commit 92213090fe
5 changed files with 272 additions and 106 deletions
+8 -11
View File
@@ -54,18 +54,15 @@ func main() {
slog.Warn("TOMSWALLAPI_AGENT_TOKEN is not set; agent config endpoint is disabled")
}
// Start the central ASN expander when a key is configured. Without one, asn
// address groups simply stay unexpanded (their sets render empty and inert).
if cfg.IPLocateAPIKey != "" {
refresher := &asnexpand.Refresher{
Store: store.New(db.Pool),
Expander: asnexpand.NewIPLocate(cfg.IPLocateAPIKey),
}
go refresher.Run(ctx)
slog.Info("started ASN expander")
} else {
slog.Warn("TOMSWALLAPI_IPLOCATE_API_KEY is not set; asn address groups will not be expanded")
// Start the central ASN expander, which reads the iplocate ip-to-asn database
// (proxied via artifactapi) and expands asn address groups into prefixes.
expander := asnexpand.NewIPLocateDB(cfg.IPLocateDBURL)
refresher := &asnexpand.Refresher{
Store: store.New(db.Pool),
Expander: expander,
}
go refresher.Run(ctx)
slog.Info("started ASN expander", "source", "iplocate-db", "url", expander.URL)
srv := server.New(server.Options{
DB: db,