jsonl-cc
A from-scratch, structure-aware lossless compressor for JSONL — built for LLM agent transcripts and structured logs. It ships as a single Rust binary (vsk) and every output is byte-exact by construction.
About the Project
General-purpose compressors don't see the structure of the data the LLM era produces (agent transcripts, API logs). jsonl-cc fills that gap: a codec that understands JSONL's record/column structure, with a hand-written core that uses no compression libraries.
Honest headline: on synthetic transcript data it beats general compressors by ~25%, but on my own real Claude transcripts it ends up roughly tied with brotli (+0.35%). I dug into why (real transcript text is ~99% high-entropy prose that brotli’s window already captures) and showed that the real double-digit win is on structured/log data. I’d rather show the honest number than a benchmark tuned to flatter myself.
Built from Scratch
Arithmetic coder
Carryless range coder, lpaq-style. Fully integer and deterministic.
Context mixing (PAQ-class)
Order-0..6 byte contexts, word / word-bigram models, a match model, logistic mixing and two SSE/APM refinement stages.
BWT pipeline
Burrows–Wheeler transform via suffix array, with bespoke post-BWT models (run-length factorization + order-0/1/2/3 contexts with adaptive count decay) instead of the classic MTF+RLE.
Columnar transform for JSONL
Records are transposed into typed columns (tag / int / float / string / nested JSON) with zigzag+delta varint integers, packed UUIDs, dictionary string coding and a record-shape dictionary. Numbers and text go to separate streams.
Correctness by construction
A lossless compressor is only useful if it never corrupts data; correctness is guaranteed by design (every output round-trips byte-exact).
Results
Real Claude transcripts — 6 files, ~25 MB, 0 exceptions, byte-exact:
| Total bytes | brotli-11 | bz2-9 | vsk |
|---|---|---|---|
| 25,290,419 | 9,555,952 | 11,042,413 | 9,522,899 |
→ a practical tie with brotli on real transcripts (most of the text is opaque prose).
Synthetic structured logs (where the columnar transform pays off):
| 16 MB API log | brotli-11 | xz-9e | zstd-22 | vsk |
|---|---|---|---|---|
| bytes | 2,042,364 | 1,998,400 | 2,109,336 | 1,505,939 |
→ ~26% smaller than brotli when the data is genuinely structured.
Tech Stack
Fully open source
jsonl-cc is published as open source on GitHub. You can read the source, build it and contribute.