Benchmarks

	Start → first query	Fresh copy per test	Indexed SELECT	Sort 200k rows	Backend + test-process memory	Download
pgmem (Go library)	70 ms	14 ms (fork)	34 µs10 µs in-process	120 ms	110 MiBtest-process growth 110 MiB	37.9 MB (linked into the test binary)
pgmem binary (Python / Java / Node.js)	218 ms	14 ms (fork)	34 µs	120 ms	105 MiB	16.8 MB (wheel / jar / npm payload)
docker run	1.65 s	61 ms (CREATE DATABASE … TEMPLATE)	93 µs	259 ms	77 MiBcontainer usage 76 MiB, test-process growth 1 MiB, VM net change 433 MiB, VM absolute 1335 MiB	118 MB (image layers)
Testcontainers (Go)	1.09 s	21 ms (Snapshot / Restore)	102 µs	258 ms	137 MiBcontainer usage 134 MiB, test-process growth 2 MiB, VM net change 295 MiB, VM absolute 1192 MiB	120 MB (image layers + Ryuk)
devbox (nix PostgreSQL)	411 ms	83 ms (CREATE DATABASE … TEMPLATE)	28 µs	228 ms	36 MiBserver process tree	113 MB (nix closure)

Median of 5 runs on 2026-09-14: Apple M3, 8 cores, 16 GB, Darwin 27.0.0; OrbStack 2.2.3 (Docker 29.4.0); arm64. The memory column compares backend usage and measured test-process growth. Testcontainers includes PostgreSQL and Ryuk container usage. VM net change and absolute footprint are separate context; VM net change can fall when OrbStack reclaims memory.

Backend and test-process memory during the benchmark

Bars show medians; whiskers span the minimum and maximum across memory runs. Docker and Testcontainers sum container usage with benchmark-process growth; Testcontainers includes PostgreSQL and Ryuk. OrbStack VM net changes are separate because the VM can reclaim memory during a run.

What each column measures

Every run is a fresh process that starts one server, loads the same schema, and then measures. The harness is in bench/alternatives.

• Start → first query: from invoking the target’s start operation until TCP SELECT 1 succeeds. docker pull and devbox install run before timing; Devbox environment resolution is also done in the untimed setup. docker run starts a fresh container and initializes its cluster. For Devbox, the harness prepares an initialized persistent cluster and app database, then measures devbox services up -b through successful SQL readiness. Testcontainers includes the Ryuk reaper container a fresh test process starts; the pgmem binary is timed until its ready line, which the Python, Java and Node.js wrappers wait for.
• Fresh copy per test: the median of 20 isolated copies of the prepared database (1,000 users, 10,000 orders), each followed by a connect and a join. pgmem forks a snapshot. docker run and devbox use CREATE DATABASE … TEMPLATE and DROP DATABASE. Testcontainers uses its PostgreSQL module’s Snapshot and Restore, which do the same through a template database.
• Indexed SELECT: SELECT v FROM bench WHERE id = $1 with pgx, 3,000 iterations after warm-up. For pgmem the second number uses the in-process dialer (Server.Dial) instead of TCP.
• Sort 200k rows: SELECT count(*) FROM (SELECT md5(i::text) AS h FROM generate_series(1, 200000) i ORDER BY h) t, median of three runs after one warm-up.
• Memory: OrbStack’s net VM footprint can fall under load, making Testcontainers look smaller than docker run even while its containers use more memory. The chart instead compares backend usage plus measured test-process growth. pgmem uses its process’s phys_footprint increase; the binary row uses the standalone pgmem process. Docker and Testcontainers sum Docker’s usage for their containers with the Go benchmark process’s growth. Testcontainers includes both PostgreSQL and Ryuk, using the memory values reported by docker stats (the Linux CLI subtracts cache from its display). devbox sums the PostgreSQL postmaster and its child processes. The table keeps OrbStack’s net change and absolute footprint as separate host context.
• Download: what a first run fetches. pgmem as a Go library is the growth of a stripped test binary that links it. The binary row is the compressed binary that the wheel, jar and npm package carry. Images are compressed layers for the measured architecture; the nix closure is what devbox downloads from cache.nixos.org.

Reading the numbers

• Defaults differ. The servers use PostgreSQL’s defaults (shared_buffers=128MB). The Testcontainers benchmark mounts PostgreSQL 18’s data directory (/var/lib/postgresql/18/docker) as tmpfs; docker run and devbox use disk-backed storage. pgmem uses shared_buffers=32MB and an in-memory file system, so sorts that spill to temporary files never touch a disk.
• The chart compares the backend and test process. It shows medians and observed ranges after schema setup and after the query/isolation workload. Testcontainers includes both PostgreSQL and Ryuk container usage. The Linux docker stats CLI subtracts cache from the displayed usage.
• OrbStack’s net change is separate context. Each container memory run restarts the VM and records its idle footprint. The VM can reclaim memory during a run, so its net change may fall even as container usage rises. The table reports that change and the absolute VM footprint separately.
• pgmem memory grows under load. WebAssembly linear memory is not returned to the OS while a server runs, and each live fork holds its own copy. Budget roughly the idle figure per live fork and bound the number with MaxForks.
• The container rows include different setup costs. docker run uses disk-backed storage; Testcontainers uses tmpfs for PGDATA and includes Ryuk, so its result reflects both storage and orchestration choices.

Reproduce

cd bench/alternatives
RUNS=5 MEM_RUNS=3 ./run.sh

The script needs OrbStack and devbox, and restarts OrbStack for the memory pass; it refuses to do so while other containers run. It writes raw samples to results/raw.jsonl, medians to results/summary.json, and copies the summary to website/src/data/benchmarks.json, which is what the tables on this site render.