Make engine health metrics granular on chunk and result - #5147
Open
mcastorina wants to merge 1 commit into
Open
Conversation
Some metrics were emitting multiple times per chunk or result, so this adds logic to track it on a per chunk and result basis. This will help us describe the percentage of items getting filtered out in each stage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Some metrics were emitting multiple times per chunk or result, so this adds logic to track it on a per chunk and result basis. This will help us describe the percentage of items getting filtered out in each stage.
Checklist:
make test-community)?make lintthis requires golangci-lint)?Note
Low Risk
Metrics-only changes; scanning, detection, and verification behavior are unchanged.
Overview
Aligns
chunks_dropped_totalwithchunks_entered_stage_totalso drop reasons are counted once per work item at each stage, instead of firing multiple times inside loops.At the scanner,
no_matching_detectorsis emitted only when no decoded variant gets any Aho–Corasick detector match (not once per empty decoded slice). In verification overlap and detect,detector_erroris recorded at most once per chunk when any match/FromDatacall fails, rather than on every failing iteration.The
chunks_dropped_totalmetric help text now states it uses the same granularity aschunks_entered_stage_totalfor the same stage, so drop ratios per stage are meaningful.Reviewed by Cursor Bugbot for commit 6927e58. Bugbot is set up for automated code reviews on this repo. Configure here.