Serialization & Parsing
JSON, MessagePack, CBOR, Protocol Buffers, FlatBuffers — round-trip encode/decode throughput on structured payloads from 64 B to 64 MB.
Open suite
Vol. 13 / No. 47 — Reference Lab
The Runtime Benchmark Lab: three architectures, raw numbers, open repository.
Every algorithm we publish runs across ARM, x86, and WASM on pinned hardware and pinned toolchains. The raw timings, scripts, and reproducibility notes are public. No curated vendor benchmarks, no hidden methodology — just the data an engineer can rerun on a Tuesday afternoon and verify before shipping.
Suite index
Twenty-three benchmark suites, all peer-reviewed, all reproducible.
The lab maintains a working catalogue of suite releases. Each suite is versioned, pinned to a specific compiler and kernel, and rerun before any article that cites it goes to print.
Lock-Free Concurrency
Mutex, RWMutex, futex, MCS lock, RCU, hazard pointers, seqlocks — throughput and tail-latency under 1 to 256 contended readers and writers.
Open suiteSIMD Sort & Select
vqsort, radix sort, IPS²Ra, std::sort, Boost spreadsort — sort throughput on 32-bit and 64-bit keys across in-register and cache-resident partitions.
Open suiteMemory Allocators
glibc malloc, jemalloc, mimalloc, tcmalloc, rpmalloc, sbrk — allocation throughput and fragmentation under mixed-size workloads and steady-state churn.
Open suite
Hardware matrix
Every result is published across three architectures. No single-vendor numbers.
A benchmark that only runs on the vendor's reference machine is marketing copy. The lab runs every published suite on three physically distinct hardware profiles and reports the raw distribution — median, p99, p99.9 — not just the headline.
- ARM AWS Graviton 4 · 96 cores · Neoverse V2 Linux 6.6 LTS, GCC 13.2, pinned mitigations off for reproducible side-channels
- x86 Intel Xeon 8488C · 96 cores · Sapphire Rapids Linux 6.6 LTS, GCC 13.2 and Clang 17, frequency governor pinned to performance
- WASM Wasmtime 18 · V8 isolate · single-threaded 32 MB heap, deterministic clock source, GC pauses captured and excluded
Methods
How the lab defends a number.
Controlled hardware
Every suite runs on machines with disabled turbo, pinned frequency, locked NUMA topology, and kernel boot parameters frozen at install. We never co-tenant. A run that touches a noisy neighbour is discarded and rerun — the audit log is in the repository.
Pinned toolchains
Compiler version, libc, allocator, and standard library are recorded per run. We do not run two different compilers in the same row of a table. When a toolchain change moves a number by more than 3%, we rerun the full suite and republish — version-bumping is honest, version-burying is not.
Statistical reporting
We report median, p99, and p99.9 across at least 31 cold runs. We report mean only when the distribution is symmetric and we have a reason to. We publish the raw CSV in every result directory; if a chart does not have an attached CSV, it does not ship.
Peer review
A second senior engineer on the 42-person editorial board must be able to rerun the suite from a clean checkout and reproduce the table within the stated error bounds before the article is signed off. The peer-reviewer mark on each piece is a real signature, not a rubber stamp.
Lab at a glance
23
benchmark suites maintained and versioned in the public repo
3
hardware architectures — ARM, x86, WASM — rerun for every published result
48,400+
GitHub stars on the members' benchmark repository (as of Oct 2024)
1,847
published articles in the archive, every claim backed by a runnable test
Published findings
Three findings the lab has shipped. Decisions, not just tables.
A benchmark is only useful if it answers a question a working engineer is actually asking. Here are three case studies from the archive.
JSON parsing: where zero-copy actually wins
A four-week cross-architecture sweep showed zero-copy JSON parsers only beat allocating parsers above a payload size that depends on the allocator, the cache, and the OS readahead window — not on the parser itself. The crossover point shifts by 14× between ARM Graviton and Xeon. The article ships the exact crossover table.
Lock contention under read-mostly workloads
For workloads above 92% reads, RCU delivers a 6.4× throughput win over a well-tuned RWMutex on ARM — and almost nothing on x86, where the architecture's store-buffer behaviour quietly closes the gap. The piece ships a decision tree, not a recommendation.
SIMD sort: the tolerance window where vqsort wins
vqsort's worst-case bound pays off only on near-sorted inputs with bounded error tolerance. Outside that window — and our sweep covers eight windows — std::sort beats it on every architecture, including WASM SIMD. The article is honest about when not to switch.