fix(skills): stop auto-saving exploration transcripts as procedure skills - #130
Merged
Merged
Conversation
…ills The multi-step heuristic fired on any 4+ contiguous successful shell calls, so an ordinary explore-test-commit session was saved as a "procedure-ls" skill containing the raw command transcript (full commit messages included). Two holes allowed it: - No substance bar: the quality gate only checks for template sections, which the generated body always contains. Now a sequence is rejected when any step is multi-line or longer than 200 chars (embedded commit messages, heredocs), or when read-only inspection verbs (ls, grep, cat, sed -n, echo, git status/log/diff, go test/vet, ...) make up more than half the steps — that is an exploration transcript, not a reusable procedure. - Generic recurrence fingerprints: the candidate fingerprint was just heuristic|name, so any two unrelated sessions starting with ls both produced multi-step|procedure-ls and met min_occurrences=2 within days. The fingerprint now keys on heuristic + name + a digest of the normalized command log, so only the same normalized pattern seen again counts as a recurrence. Regression tests cover transcript rejection, oversized/multi-line steps, false recurrence via generic names, and acceptance of a genuine build/tag/push sequence. The TestRunLearn_* E2E mock now drives an action verb (echo is correctly classified as inspection). LEARNING.md and AGENTS.md document the new gates.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Auto-saved skills like
procedure-lswere raw session transcripts: 19 steps ofls/grep/sed -nexploration plus fullgit commit -mmessages, with junk trigger keywords. Two holes in the pipeline let them through:DetectMultiStepProcedurehad no substance bar — any 4+ contiguous successful shell calls became a "procedure" skill. The quality gate only checks for## Overview/## Common Pitfallssections, which the generated template always contains.candidateFingerprintwas justheuristic|name, and the name isprocedure-<first command verb>. Any two unrelated sessions starting withlsboth producedmulti-step|procedure-ls, somin_occurrences: 2passed within days.Fix
buildSuggestionFromSequence: reject sequences where any step is multi-line or > 200 chars (embedded commit messages, heredocs — session payloads that can never be replayed), or where read-only inspection verbs (ls,grep,cat,head,sed -n,echo,git status/log/diff,go test/vet, …) make up more than half the steps — an exploration transcript, not a reusable procedure.heuristic|name|sha256(normalized command log)[:12]— only the same normalized command pattern seen again counts as a recurrence; generic names no longer collide across unrelated sessions.Tests
procedure-lsskillgit commit -m …) and multi-line (heredoc) stepsprocedure-lswith different command logs stay pendingTestRunLearn_*E2E mock now drives an action verb (true step N) —echois correctly classified as inspectiongo test ./internal/skills/,go vet,golangci-lintall cleanNote:
TestApproveMCPServers_ProjectServerRequiresApprovalfails locally in full-package runs on this machine, but it also fails on a clean checkout ofmain(verified viagit stash) — pre-existing local environment issue, unrelated to this change.