Skip to content

fix(skills): stop auto-saving exploration transcripts as procedure skills - #130

Merged
jkyberneees merged 1 commit into
mainfrom
fix/skill-autosave-substance-gate
Aug 10, 2026
Merged

fix(skills): stop auto-saving exploration transcripts as procedure skills#130
jkyberneees merged 1 commit into
mainfrom
fix/skill-autosave-substance-gate

Conversation

@jkyberneees

Copy link
Copy Markdown
Contributor

Problem

Auto-saved skills like procedure-ls were raw session transcripts: 19 steps of ls/grep/sed -n exploration plus full git commit -m messages, with junk trigger keywords. Two holes in the pipeline let them through:

  1. DetectMultiStepProcedure had no substance bar — any 4+ contiguous successful shell calls became a "procedure" skill. The quality gate only checks for ## Overview / ## Common Pitfalls sections, which the generated template always contains.
  2. The recurrence gate was defeated by generic fingerprintscandidateFingerprint was just heuristic|name, and the name is procedure-<first command verb>. Any two unrelated sessions starting with ls both produced multi-step|procedure-ls, so min_occurrences: 2 passed within days.

Fix

  • Substance bar in buildSuggestionFromSequence: reject sequences where any step is multi-line or > 200 chars (embedded commit messages, heredocs — session payloads that can never be replayed), or where read-only inspection verbs (ls, grep, cat, head, sed -n, echo, git status/log/diff, go test/vet, …) make up more than half the steps — an exploration transcript, not a reusable procedure.
  • Content-keyed recurrence fingerprint: heuristic|name|sha256(normalized command log)[:12] — only the same normalized command pattern seen again counts as a recurrence; generic names no longer collide across unrelated sessions.

Tests

  • Rejects an exploration transcript modeled on the reported procedure-ls skill
  • Rejects oversized (git commit -m …) and multi-line (heredoc) steps
  • Accepts a genuine build/tag/push release sequence
  • No false recurrence: two sessions sharing the name procedure-ls with different command logs stay pending
  • TestRunLearn_* E2E mock now drives an action verb (true step N) — echo is correctly classified as inspection
  • go test ./internal/skills/, go vet, golangci-lint all clean

Note: TestApproveMCPServers_ProjectServerRequiresApproval fails locally in full-package runs on this machine, but it also fails on a clean checkout of main (verified via git stash) — pre-existing local environment issue, unrelated to this change.

…ills

The multi-step heuristic fired on any 4+ contiguous successful shell
calls, so an ordinary explore-test-commit session was saved as a
"procedure-ls" skill containing the raw command transcript (full commit
messages included). Two holes allowed it:

- No substance bar: the quality gate only checks for template sections,
  which the generated body always contains. Now a sequence is rejected
  when any step is multi-line or longer than 200 chars (embedded commit
  messages, heredocs), or when read-only inspection verbs (ls, grep,
  cat, sed -n, echo, git status/log/diff, go test/vet, ...) make up
  more than half the steps — that is an exploration transcript, not a
  reusable procedure.

- Generic recurrence fingerprints: the candidate fingerprint was just
  heuristic|name, so any two unrelated sessions starting with ls both
  produced multi-step|procedure-ls and met min_occurrences=2 within
  days. The fingerprint now keys on heuristic + name + a digest of the
  normalized command log, so only the same normalized pattern seen
  again counts as a recurrence.

Regression tests cover transcript rejection, oversized/multi-line
steps, false recurrence via generic names, and acceptance of a genuine
build/tag/push sequence. The TestRunLearn_* E2E mock now drives an
action verb (echo is correctly classified as inspection). LEARNING.md
and AGENTS.md document the new gates.
@jkyberneees
jkyberneees merged commit 0a3054d into main Aug 10, 2026
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant