Split text into chunks under a hard token cap without cutting a table, sentence or code fence
Plans chunk boundaries for text an agent must feed to a model in pieces: every chunk is at most maxTokens in the named tokenizer (o200k_base or cl100k_base, verified by counting each finished chunk), boundaries fall only between sentences, blocks, whole Markdown or HTML tables and whole code fences, headings stay with the text that follows, and the chunks tile the input exactly (concatenating the spans gives the original). Spans carry offsets in UTF-16, code points, UTF-8 bytes and graphemes. If one sentence, table or fence alone exceeds the cap it is returned alone with exceedsCap and the status is fail; it is never cut. Sentence breaks follow ICU rules and can be wrong after abbreviations. Other tokenizer names (including Claude) return unsupported.
What you get
Chunking that never splits a table or fence and never exceeds the cap. 100,000 characters per $0.005 call.
Call it
Send the JSON body below to https://agents.writerslogic.com/api/v1/chunk-boundaries. An unpaid request answers 402 with the price and how to pay; a request that fails with a 5xx status is not charged.
curl -i -X POST https://agents.writerslogic.com/api/v1/chunk-boundaries \
-H 'content-type: application/json' \
-d '{"text":"# Title\n\nFirst paragraph. It has two sentences.\n\n| a | b |\n|---|---|\n| 1 | 2 |\n\n```js\nconsole.log(1)\n```\n","maxTokens":40,"tokenizer":"o200k_base"}'
Input
| Field | Type | Required | Description |
|---|---|---|---|
text | string; length 1..100000 | yes | Text to chunk (Markdown, plain text or HTML tables). |
maxTokens | integer; 32..1000000 | yes | Hard cap per chunk. |
tokenizer | string; length 1..64 | yes | o200k_base or cl100k_base. Any other name (including Claude models) returns status unsupported for items that need counting; caller token counts are always accepted. |
includeText | boolean | no | Also return each chunk's text. Default false (slice by span). |
Example request body
{
"text": "# Title\n\nFirst paragraph. It has two sentences.\n\n| a | b |\n|---|---|\n| 1 | 2 |\n\n```js\nconsole.log(1)\n```\n",
"maxTokens": 40,
"tokenizer": "o200k_base"
}
Result
The response is { "data": ..., "_provenance": ... }. data carries these fields; _provenance is a signed W3C Verifiable Credential with a C2PA-style manifest, checked free at POST /api/v1/verify-provenance.
| Field | Type | Required | Description |
|---|---|---|---|
status | string; one of "pass", "fail", "unknown", "unsupported" | no | pass: every rule held. fail: a rule was violated (see witnesses). unknown: the input lacks the evidence to decide. unsupported: outside what this engine covers. Missing evidence is never a pass. |
chunks | array<object>; max 5000 items | no | |
totalTokens | integer | no | Token count of the whole text. |
witnesses | array<object> | no | |
versions | object | no | Every semantic dependency of this answer (engine, tables and their as-of dates, tokenizer, calendar, rounding profile). |
limits | array<string> | no | |
deterministic | boolean | no | Always true: pure code, no model. |
Pay and connect
Available through: x402 (USDC on Base), MCP. MCP clients reach it through the context-evidence-kit package.
Machine-readable
- This agent as JSON (schemas, example, OpenAPI operation)
- Free signed sample response
- OpenAPI 3.1 · Catalog · Index of every agent