Tool
RAG Text Chunker
Paste a document and split it into retrieval-ready chunks: sentence-aware boundaries, a token budget per chunk, and overlap so a fact straddling a cut still shows up whole. Everything runs in your browser; your text never leaves the page.
Chunks
Paste text to see chunks.
Token counts use this site's estimator (1.33 per English word, 0.6 per CJK character, 0.25 per other character) — the same numbers as the Token Counter. Treat them as planning estimates, not exact tiktoken counts: for exact counts, run the same split through your model's tokenizer before indexing.
Why chunk at all
Embeddings work best on focused passages. Embed a whole 10,000-token document as one vector and the retrieval query has to match the entire document at once; split it into 300–500 token pieces and each chunk stands for one idea, so the right passage surfaces. Chunking is the step in a RAG pipeline where most quality is won or lost — before any embedding model runs.
How this splitter works
It packs whole sentences into each chunk until the token budget is full, so cuts land on sentence boundaries instead of mid-sentence. Lines are respected first, so markdown lists, headings and code blocks stay intact where possible. A sentence longer than the budget is split by words. Overlap carries the trailing sentences of one chunk into the next, so an idea that straddles a cut still appears whole in at least one chunk — at the cost of duplicated embedding calls, which the stats above show as a percentage.
Picking a size and overlap
Start with 300–500 tokens per chunk and overlap around 10–15% of that. Smaller chunks retrieve more precisely but lose context; larger chunks keep context but dilute the match. There is no universal best size — try one setting on a real document, check whether the chunks you'd want retrieved actually come back, and adjust from there.
Next: check a chunk against the Token Counter for exact cost, or the context window table to see how many chunks fit in one prompt.