feat: merge /reports and /events across both PuppetDBs #5
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: unkin/pdbmux#5
Reference in New Issue
Block a user
Delete Branch "benvin/reports-merge"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
Puppetboard's report views only ever saw the primary backend, so reports for a
node that had migrated between PuppetDBs went missing.
How
GET /pdb/query/v4/reportsand/eventsas the union of both backends— deduped by report
hashand by record identity respectively — instead ofpassing them through to the primary.
order_byand cut the requestedlimit/offsetwindow from it, asking each backend for the first
offset + limitrecords.X-Recordsinto one merged header when the request setsinclude_total./pdb/query/v4/reports/<hash>/{events,logs,metrics}from whicheverbackend holds that report;
404when neither does.limit,offsetororder_bywith400.extract/group_byaggregate rows from every backend rather thancollapsing the hashless rows into one.
An extract/count()/group_by query returns synthetic rows with no report hash, which reportKey fell back to keying by verbatim raw bytes. Two backends emitting a byte-identical aggregate row (e.g. {"status":"changed","count":1}) therefore collapsed into one, silently undercounting the merged result and contradicting the documented guarantee that no backend's rows are dropped. Give the mergeUnion key func an ok return: false means the record has no dedupe identity and is always kept. reportKey returns ok=false for hash-less rows; hash-keyed report dedupe and event verbatim-identity dedupe are unchanged. Add TestMergeUnion_IdenticalHashlessRowsAreNotCollapsed covering the collision case, and reword the README line to say aggregate rows pass through even when byte-identical.Fixed the hash-less row collision.
mergeUnion's key func now returns(key, ok);reportKeyreturnsok=falsefor rows with no hash, soextract/count()/group_byaggregate rows are never deduped and two backends emitting a byte-identical row both survive. Hash-keyed report dedupe and event verbatim-identity dedupe are unchanged. AddedTestMergeUnion_IdenticalHashlessRowsAreNotCollapsed(fails on the old code: 1 row instead of 2) and reworded the README line to say aggregate rows pass through even when byte-identical.