Expand ASN groups from the iplocate ip-to-asn database #7

2026-07-21T22:36:20+10:00

unkinben commented

2026-07-21 22:36:20 +10:00

Why

ASN address groups need expanding into prefixes. iplocate's downloadable ip-to-asn database (CIDR → ASN) is a better source than the API (no key, one download serves every ASN, and the API's ASN→prefixes endpoint was never confirmed).

Changes

Add a DB-backed expander that downloads the ip-to-asn CSV (proxied via the artifactapi github remote), indexes it once (ASN → CIDRs) in memory, and serves every group from the index; rebuilds on a 24h TTL, keeps last-good on refresh failure (fail-safe). No API key.
Wire it in main via TOMSWALLAPI_IPLOCATE_DB_URL (defaults to the artifactapi github-remote URL for iplocate/ip-address-databases); remove the unused API client.
Unit tests: CSV indexing (including quoted org fields with embedded commas), zip extraction, missing-column handling, ASN normalization (AS-prefix).

Depends on

terraform-artifactapi #9 (github-remote patterns for the iplocate DB) once applied.

## Why ASN address groups need expanding into prefixes. iplocate's downloadable ip-to-asn database (CIDR → ASN) is a better source than the API (no key, one download serves every ASN, and the API's ASN→prefixes endpoint was never confirmed). ## Changes - Add a DB-backed expander that downloads the ip-to-asn CSV (proxied via the artifactapi github remote), indexes it once (ASN → CIDRs) in memory, and serves every group from the index; rebuilds on a 24h TTL, keeps last-good on refresh failure (fail-safe). No API key. - Wire it in main via `TOMSWALLAPI_IPLOCATE_DB_URL` (defaults to the artifactapi github-remote URL for iplocate/ip-address-databases); remove the unused API client. - Unit tests: CSV indexing (including quoted org fields with embedded commas), zip extraction, missing-column handling, ASN normalization (AS-prefix). ## Depends on terraform-artifactapi #9 (github-remote patterns for the iplocate DB) once applied.

unkinben added 1 commit 2026-07-21 22:36:20 +10:00

Expand ASN groups from the iplocate ip-to-asn database

ci/woodpecker/pr/build Pipeline was successful

Details

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

ci/woodpecker/pr/test Pipeline was successful

Details

92213090fe

Replace the (unconfirmed) iplocate API expander with a database-backed one that
reads the iplocate ip-to-asn CSV (network,asn,...) proxied through the artifactapi
github remote. It downloads and indexes the whole DB once (ASN -> CIDRs), serves
every asn address group from the in-memory index, and rebuilds on a 24h TTL;
refresh failures keep the last-good index (fail-safe). No API key needed.

- Add IPLocateDB expander (zip + CSV parsing, ASN normalization).
- Wire it in main (TOMSWALLAPI_IPLOCATE_DB_URL overrides the default artifactapi
  URL); remove the dead API client.
- Unit tests: CSV indexing (incl. quoted org fields), zip extraction, missing
  columns, and ASN normalization.

benvin merged commit 86876687a7 into main

2026-07-22 00:53:19 +10:00

benvin deleted branch benvin/asn-db

2026-07-22 00:53:19 +10:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/tomswallapi#7