What is a Text Chunker?
Text Chunker is a tool that counts the tokens of a text and then splits it into chunks: fixed-size segments measured in tokens, with a configurable overlap between two consecutive segments.
AI models (GPT, Claude, embedding models…) do not read words but tokens, and their context window is limited. Chunking is therefore essential to prepare long content: feeding a RAG (retrieval-augmented generation) knowledge base, generating embeddings, or sending a long document to an LLM without exceeding its limit. The overlap preserves context between segments so an idea never gets cut in half.
Paste your text, pick the model (the matching tokenizer is loaded automatically), the chunk size and the overlap: the tool shows every chunk with its positions and token count, and you can export everything as JSON. You can also compare several chunk sizes at once.