ctxfold

v0.2.0

Know before you fold: the token profiler

ctxfold --profile answers two questions about any prompt payload: where do the characters go, and what would folding save? Composition is attributed exactly — the categories sum to the input size — and the savings come from actually running the encoder, so the profiler can never promise more than it delivers.

paste something above, or load a sample — then press Profile

$ ctxfold --profile users.json

[ctxfold profile]
format      JSON array — 300 records × 7 fields
size        65,614 chars ≈ 16,404 tokens (estimated; pass a tokenizer for exact)

where the characters go
  keys         27%   repeated field names (with quotes)
  syntax        7%   braces, brackets, commas, colons
  values       36%   the data itself (with string quotes)
  whitespace   30%   indentation and spacing

foldable (lossless, verified by round-trip)
  fold             -66%   direct-readable — validated 24/24 vs raw
  + --dictionary   -73%   readability tradeoff — off by default, see README

verdict: fold it — 16,404 → ~5,595 tokens (~66% fewer)

The panel above runs the real encoder in your browser — same code as the CLI, no API key, nothing uploaded. Prefer a terminal? npx ctxfold --profile yourfile.json, or node examples/profile-demo.js after cloning.

v0.3.0

One level of nesting

API responses nest. {"account":{"plan":"pro"}} now folds into account.plan columns instead of being kept verbatim — so a whole class of payload that earlier versions declined outright is foldable. The gain here isn't a bigger ratio; it's the difference between folding and not folding at all.

$ ctxfold users.json     # ctxfold 0.2.2
  no-op — records contain nested objects, kept verbatim
  54,404 chars in, 54,404 chars out

$ ctxfold users.json     # ctxfold 0.3.0
  folded — 54,404 → 7,751 chars, lossless, verified by round-trip

cols: id  account.plan  account.seats  account.active  location.city
10000  pro  8  1  Denver
10001  team  14  1  Denver
…

The dotted name is what the model reads; the decoder stores the path as an array, so a literal key named account.plan and a nested one stay distinct. Shapes the format can't express unambiguously — empty nested objects, values two levels down, a key holding an object in one record and a scalar in another — are kept verbatim rather than flattened lossily. One record in the run above hit exactly that and was passed through untouched.

benchmarks

Every claim has a measurement

Lookup accuracy is scored against exact ground truth: the same questions asked of the folded form and the raw form, across GPT-4o-mini, Claude, Llama, and Qwen. Where the measurement came in against the format, the table says so.

Table shows the original GPT-4o-mini runs; the full four-model matrix (including the one model that reads folded CSV correctly) is in the README. The CSV result is a published negative: CSV has no syntax to remove, so folding it factors the data itself — and models don't reliably reconstruct values through a header. Fold CSV for lossless transit and decompress() before the model reads it; for direct reading, send CSV raw. The full story of that measurement →

get it

Code, package, write-ups