Fix stale doc_sourced reference and enforce provenance consistency - #27
Merged
Conversation
…stency The README still told contributors that an all-doc-sourced provider carries `doc_sourced: true` in its index.json. Nothing has done that since provenance replaced the flag — no index.json sets it and no code reads it. Replaced with what actually marks those versions, `"sourced_via": "docs"`. That leaves two records of the same fact: the per-file `source` key doc-sourced samples carry, and the version-level provenance block. They're complementary — per-file travels with the payload, version-level is what a consumer filters on without downloading every version file — but nothing stopped them drifting. `compile.ts` now fails the build when a version marked `capture` contains a file with a `source` key. That's the drift that matters: doc-sourced samples published under a claim they were captured live. The reverse isn't checked, since a `docs` version legitimately need not mark every file. Verified against the full tree: 112 providers and 1,260 doc-sourced files compile clean, and planting a `source` key in a captured version fails as intended. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019pce9oHWGjdwHsJNya1ovP
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Fail rather warn works for me |
garethx
added a commit
to garethx/webhook-samples
that referenced
this pull request
Aug 12, 2026
Real deliveries from a Vapi sandbox assistant through a Hookdeck source on 2026-08-12, captured byte-exact from the wire. `status-update` and `end-of-call-report` — both among the 11 message types the docs could NOT supply, because their examples elide the Call Object behind a placeholder comment. These carry it in full: 21 keys on end-of-call-report, with artifact, costBreakdown, performanceMetrics and the real call object. Redacted by generator/redact_capture.py in the registry repo, not by hand. Every UUID is swapped for a well-formed fake through one stable mapping shared across both files, so identifiers that correlate in a real delivery still correlate here; the two call ids differ because these are two different calls, which the mapping preserved rather than flattened. 56 values redacted, and the script then audits its own OUTPUT for surviving identifiers rather than trusting its intent. One redaction was not on the list supplied with the captures, and it was the one that mattered most: `assistant.server.url` is the LIVE INGEST ENDPOINT the webhook was delivered to. It is not personal data, so reading the payload for personal data misses it — but publishing it lets anyone POST arbitrary bodies into that source. Transcripts were emptied too; on these calls they held only "AI: Hello," because the calls errored before any customer audio, but the rule should not depend on that. Headers are the observed set with placeholder values. `x-signature` is an HMAC over the RAW body and the body here is redacted, so the real digest no longer verifies — shipping it would teach a verification that fails, which is worse than shipping none. The header NAMES are what carry the information, and they are exact: x-signature, x-timestamp, x-vapi-secret (present but empty when an HMAC credential is used), x-call-id, user-agent: axios/1.8.3. `provenance` for `latest` is now removed rather than set. The version holds 10 doc-sourced files and 2 captures, and sourced_via has no value for that: `docs` would misdescribe the captures, `capture` would misdescribe the other ten and fail the hookdeck#27 build check, and Vapi has no version scheme to split them into without inventing one. Undeclared is what this repo already does for its 8 other mixed versions, and the per-file `source` key still distinguishes them exactly. A `provenance_note` says so in place of a wrong declaration. `yarn compile` passes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two small follow-ups after #26 and #25 merged.
The README still referenced
doc_sourcedIt told contributors that a provider whose samples are all doc-sourced carries
doc_sourced: truein itsindex.json. Nothing has done that since provenance replaced the flag — noindex.jsonsets it, and no code reads it. Replaced with what actually marks those versions:The two provenance records could drift
After both PRs, the same fact is recorded twice: the per-file
sourcekey that doc-sourced samples carry (1,260 files), and the version-levelprovenanceblock (95 providers). They're complementary rather than redundant — per-file travels with the payload and records exactly where each example was read, version-level is what a consumer can filter on without downloading every version file — but nothing kept them in step.compile.tsnow fails the build when a version markedcapturecontains a file with asourcekey:That's the drift worth catching: doc-sourced samples published under a claim they were captured live. The reverse isn't checked — a
docsversion legitimately needn't mark every file.Testing
yarn compilepasses on the full tree: 112 providers, 1,260 doc-sourced files, publishingdocs: 93, unknown: 21, capture: 2. Planting asourcekey into a captured version fails as intended, and removing it passes again.One thing worth a second opinion
This is stricter than the tree it validates, so it's a new way for a build to break. If someone captures fresh samples over a version that was previously
docsand flipssourced_viatocapturewithout clearing the oldsourcekeys, compile stops rather than warns. That's deliberate — publishing a doc example as captured is the failure mode the whole provenance idea exists to prevent — but @garethx it's yoursamples-doc/workflow it would interrupt most, so say if you'd rather it warned.🤖 Generated with Claude Code
https://claude.ai/code/session_019pce9oHWGjdwHsJNya1ovP