fix(chunkers): preserve FAQ prose in docs chunks

[j15z]

What

The docs chunker's cleanContent deletes every FAQ section before embedding:

• The multiline tag regex swallows entire <FAQ items={[...]}/> blocks — it matches from <FAQ to the first >, which is often inside an answer string — and the brace strip eats any surviving { question: ..., answer: ... } items.
• Net effect: 637 Q&As across the docs never reach the search index, and mangled JSX fragments get embedded in their place.

This PR makes the chunker consume FAQ components whole (before the tag strip runs) and emit their question/answer text as plain prose. The extraction is escape-aware, so quotes and braces inside answers survive intact. Tag and brace stripping are otherwise byte-identical to today — a corpus survey confirmed FAQ props are the only place real page prose lives inside JSX syntax on searchable pages, so nothing else needed rescuing.

Why it matters

FAQ answers are the docs' most retrieval-shaped content — literal user questions paired with concrete answers ("What is the maximum file size for uploads?" → "20 MB"). Today a semantic search for exactly that question cannot match it, because the text isn't in the index. This fix applies to the existing docs_embeddings pipeline, so current production search benefits on the next index rebuild — no dependency on other in-flight work.

Testing

• New docs-chunker.test.ts: the 20 MB regression on the real page shape, nested-brace and angle-token answers, multiline item formatting, escaped quotes, unchanged scaffolding strips.
• Full-corpus validation: 637/637 FAQ questions preserved in cleaned output across apps/docs/content/docs/en, zero JSX debris.
• Full local reindex ran clean: 402 pages / 0 failures.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Aug 7, 2026 7:59pm

[cursor]

PR Summary

Low Risk
Scoped to docs chunking/cleaning for embeddings; no auth or runtime API changes, with tests locking FAQ extraction behavior.

Overview
Docs search indexing was dropping FAQ content: cleanContent ran generic tag and brace stripping on <FAQ items={[...]} /> blocks, so multiline FAQ JSX (often ending at > inside answers) and { question, answer } objects were removed instead of embedded.

This change extracts FAQ blocks first—before the existing tag/brace passes—and writes question/answer strings as plain text. Parsing is escape-aware for single/double quotes, multiline items, and optional trailing commas; emitted text strips <, >, {, } wrappers so later strips cannot eat answers that mention tokens like data:{mime} or <gmail.attachments[0]>.

Adds docs-chunker.test.ts covering regression shapes, nested tokens in answers, and unchanged behavior for imports, exports, comments, and other JSX scaffolding.

^{Reviewed by Cursor Bugbot for commit b71444b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

The PR updates docs chunk cleaning to preserve FAQ questions and answers as searchable prose while retaining the existing cleanup behavior for other JSX scaffolding.

• Extracts double- and single-quoted FAQ item strings before generic tag and brace removal.
• Handles multiline items, trailing commas, escaped quotes, braces, and angle-bracket tokens.
• Adds focused regression coverage for production FAQ shapes and unchanged scaffolding removal.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
apps/sim/lib/chunkers/docs-chunker.ts	Adds escape-aware extraction for both quote styles before existing JSX cleanup, and the previously reported single-quote omission is corrected.
apps/sim/lib/chunkers/docs-chunker.test.ts	Adds regression tests covering production FAQ syntax, difficult inline tokens, escaped quotes, and preservation of existing cleanup behavior.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Raw MDX documentation] --> B[Extract FAQ components]
  B --> C[Emit question and answer prose]
  C --> D[Strip remaining JSX scaffolding]
  D --> E[Split cleaned content into chunks]
  E --> F[Generate and persist docs embeddings]

Loading

_{Reviews (3): Last reviewed commit: "fix(chunkers): accept single-quoted FAQ ..." | Re-trigger Greptile}

[j15z]

@greptile

[j15z]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e814d46. Configure here.}

[j15z]

@greptile

[j15z]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e814d46. Configure here.}