bench/modelcase
is a small but ordinary Go project: the data layer of a shop
(database/sql, five SQL migrations with a trigger and a view, 20
categories, 500 products and 1,000 users as master data), and 70 tests
against it, as tables of cases. Five read-only scenarios with ten cases
each (order lists per user, order detail, top products, a product search,
revenue by category) make 50 tests that only query; five writing scenarios
with four cases each (create, cancel and ship an order, delete a user with
its orders, apply a discount) make 20 tests that modify tables and check
the rows afterwards. Every test starts from the same 40 sample orders. The
same test functions run on both models; only the fixture behind them
changes.
pgmem: TestMain boots a server in the test process, imports the
schema and master data, and takes one snapshot. The read-only tests need
no copy: they share the template server itself, loaded with the sample
data once after the snapshot. Every writing test forks its own copy of
the snapshot and loads the sample data into it. All 70 tests call
t.Parallel(), so everything after the snapshot runs at once, as many
at a time as -parallel allows.
One shared server: the official image through testcontainers-go’s
PostgreSQL module in memory (PGDATA on tmpfs, fsync, synchronous_commit
and full_page_writes off, plus the Ryuk reaper a fresh test process
starts), native PostgreSQL 18 from devbox, or the official image
with docker run. The driver starts the server, then the test binary
imports the schema and master data into the shared database. The
read-only tests still run in parallel, on the shared database, but a
writing test cannot: it truncates the transactional tables, restores
stock and prices, loads the sample data and runs, and the next writer
waits its turn. That is what a suite on one server has to do to stay
deterministic.
The same 70 tests (50 read-only, 20 writing) on the same schema, migrations and seed data, run with go test at parallelism 8. Each timeline is the run whose total was the median of 5 runs on Apple M3 (8 cores). Bars are the nodes of each execution model; lower and to the left is faster.
Whole suite, wall clock
pgmem204 ms
196 ms–219 ms over 5 runs
Testcontainers, one shared server1.73 s
1.70 s–1.81 s over 5 runs
PostgreSQL, one shared server646 ms
633 ms–671 ms over 5 runs
docker run, one shared server2.48 s
2.26 s–3.59 s over 5 runs
Tests are drawn in the 8 slots of go test -parallel; a slot shows the tests that ran in it one after another. All timelines share one time axis.
Boot database
Import schema + seed master data
Snapshot
Fork (a writing test's own copy)
Input sample data
Reset data
Test that queries data
Test that modifies a table
pgmem204 ms totalin the test process; read tests share the template, each writing test forks the snapshot, all in parallel20 forks, one per writing test (20); the 50 read tests use the template database without a fork.Testcontainers, one shared server1.73 s totaltestcontainers-go postgres module on the official image, in memory (tmpfs, no fsync); writing tests take turns after a resetno copies: the 50 read tests and the 20 writing tests share one database.PostgreSQL, one shared server646 ms totalnative PostgreSQL (devbox); writing tests take turns after a resetno copies: the 50 read tests and the 20 writing tests share one database.docker run, one shared server2.48 s totalofficial image; writing tests take turns after a resetno copies: the 50 read tests and the 20 writing tests share one database.
Time per node
Node
pgmem
Testcontainers, one shared server
PostgreSQL, one shared server
docker run, one shared server
Boot database
39 ms
761 ms
192 ms
1.55 s
Import schema + seed master data
47 ms
108 ms
57 ms
97 ms
Snapshot
3.2 ms
—
—
—
Fork (a writing test's own copy)
218 ms
—
—
—
Input sample data
288 ms
624 ms
231 ms
570 ms
Reset data
—
140 ms
111 ms
161 ms
Test that queries data
61 ms
161 ms
105 ms
163 ms
Test that modifies a table
16 ms
21 ms
10 ms
38 ms
Wall clock (boot + test binary)
204 ms
1.73 s
646 ms
2.48 s
Sums over every copy or test; on pgmem the copies overlap in time, so the sums exceed the wall clock.
Every bar is one node of the execution model, timed inside the test binary
(MODELCASE_TRACE); the driver times the external server boot until
SELECT 1 succeeds on the app database, and the wall clock of the test
binary. The total is boot plus test binary; the binary’s own start-up and
exit are the few milliseconds the nodes do not cover. Each timeline is the
run whose total was the median; the first execution of a freshly built
binary is a warm-up and not recorded. go test runs with -parallel at
GOMAXPROCS, its default. Every timeline shares one time axis.
What to read from it:
The shared server pays per writing test. Reset plus sample data costs
about the same as a pgmem fork plus sample data, but the shared server
runs them one after another and the forks run them side by side.
pgmem’s fixed cost is the boot and the snapshot, and both are
smaller than starting a native server, let alone a container.
Forks are cheap and overlap. A fork clones the data directory
copy-on-write (the tree is copied, file contents are shared until
written) and starts a backend whose shared memory is no longer zeroed
twice, so one fork alone takes about 6 ms. With eight in flight they
still contend for page faults and take about 10 ms each, and the 20
forks with their sample data fit in about 120 ms of wall clock.
Read-only tests on one shared server serialize at transaction
boundaries (one backend session per pgmem server), so on pgmem they
are a little slower one by one than on a multi-process server. They are
short either way.
Terminal window
cdbench/modelcase
RUNS=5./run.sh# pgmem, Testcontainers, devbox and docker
TARGETS="pgmem devbox"./run.sh# without Docker
The script compiles the test binary once, runs the driver for each target
and writes results/raw.jsonl (every run with its spans),
results/summary.json and website/src/data/modelcase.json, which this
page renders. go test ./... in that directory runs the suite on pgmem the
ordinary way, without any tracing.
77 MiBcontainer usage 76 MiB, test-process growth 1 MiB, VM net change 433 MiB, VM absolute 1335 MiB
118 MB (image layers)
Testcontainers (Go)
1.09 s
21 ms (Snapshot / Restore)
102 µs
258 ms
137 MiBcontainer usage 134 MiB, test-process growth 2 MiB, VM net change 295 MiB, VM absolute 1192 MiB
120 MB (image layers + Ryuk)
devbox (nix PostgreSQL)
411 ms
83 ms (CREATE DATABASE … TEMPLATE)
28 µs
228 ms
36 MiBserver process tree
113 MB (nix closure)
Median of 5 pgmem and binary runs on 2026-09-18: Apple M3, 8 cores, 16 GB, Darwin 27.0.0; OrbStack 2.2.3 (Docker 29.4.0); arm64. The Docker, Testcontainers and devbox comparison baselines are from 2026-09-14. The memory column compares backend usage and measured test-process growth. Testcontainers includes PostgreSQL and Ryuk container usage. VM net change and absolute footprint are separate context; VM net change can fall when OrbStack reclaims memory.
Backend and test-process memory during the benchmark
Bars show medians; whiskers span the minimum and maximum across memory runs. Docker and Testcontainers sum container usage with benchmark-process growth; Testcontainers includes PostgreSQL and Ryuk. OrbStack VM net changes are separate because the VM can reclaim memory during a run.
The pgmem library and standalone binary rows were rerun on 2026-09-18 with
the export-pruned AOT build: five runs for each timing/memory sample, plus a
three-run Go benchmark. The stripped darwin/arm64 binary is 36.7 MB, its zip
payload is 16.3 MB, and linking pgmem into a stripped Go test binary adds
33.1 MB. Docker, Testcontainers and devbox were not rerun; their comparison
rows remain the 2026-09-14 baselines.
Every run is a fresh process that starts one server, loads the same schema, and then measures. The harness is in bench/alternatives.
Start → first query: from invoking the target’s start operation until TCP SELECT 1 succeeds. docker pull and devbox install run before timing; Devbox environment resolution is also done in the untimed setup. docker run starts a fresh container and initializes its cluster. For Devbox, the harness prepares an initialized persistent cluster and app database, then measures devbox services up -b through successful SQL readiness. Testcontainers includes the Ryuk reaper container a fresh test process starts; the pgmem binary is timed until its ready line, which the Python, Java and Node.js wrappers wait for.
Fresh copy per test: the median of 20 isolated copies of the prepared database (1,000 users, 10,000 orders), each followed by a connect and a join. pgmem forks a snapshot. docker run and devbox use CREATE DATABASE … TEMPLATE and DROP DATABASE. Testcontainers uses its PostgreSQL module’s Snapshot and Restore, which do the same through a template database.
Indexed SELECT: SELECT v FROM bench WHERE id = $1 with pgx, 3,000 iterations after warm-up. For pgmem the second number uses the in-process dialer (Server.Dial) instead of TCP.
Sort 200k rows: SELECT count(*) FROM (SELECT md5(i::text) AS h FROM generate_series(1, 200000) i ORDER BY h) t, median of three runs after one warm-up.
Memory: OrbStack’s net VM footprint can fall under load, making Testcontainers look smaller than docker run even while its containers use more memory. The chart instead compares backend usage plus measured test-process growth. pgmem uses its process’s phys_footprint increase; the binary row uses the standalone pgmem process. Docker and Testcontainers sum Docker’s usage for their containers with the Go benchmark process’s growth. Testcontainers includes both PostgreSQL and Ryuk, using the memory values reported by docker stats (the Linux CLI subtracts cache from its display). devbox sums the PostgreSQL postmaster and its child processes. The table keeps OrbStack’s net change and absolute footprint as separate host context.
Download: what a first run fetches. pgmem as a Go library is the growth of a stripped test binary that links it. The binary row is the compressed binary that the wheel, jar and npm package carry. Images are compressed layers for the measured architecture; the nix closure is what devbox downloads from cache.nixos.org.
Defaults differ. The servers use PostgreSQL’s defaults (shared_buffers=128MB). The Testcontainers benchmark mounts PostgreSQL 18’s data directory (/var/lib/postgresql/18/docker) as tmpfs; docker run and devbox use disk-backed storage. pgmem uses shared_buffers=32MB and an in-memory file system, so sorts that spill to temporary files never touch a disk.
The chart compares the backend and test process. It shows medians and observed ranges after schema setup and after the query/isolation workload. Testcontainers includes both PostgreSQL and Ryuk container usage. The Linux docker stats CLI subtracts cache from the displayed usage.
OrbStack’s net change is separate context. Each container memory run restarts the VM and records its idle footprint. The VM can reclaim memory during a run, so its net change may fall even as container usage rises. The table reports that change and the absolute VM footprint separately.
pgmem memory grows under load. WebAssembly linear memory is not returned to the OS while a server runs, and each live fork holds its own copy. Budget roughly the idle figure per live fork and bound the number with MaxForks.
The container rows include different setup costs. docker run uses disk-backed storage; Testcontainers uses tmpfs for PGDATA and includes Ryuk, so its result reflects both storage and orchestration choices.
The script needs OrbStack and devbox, and restarts OrbStack for the memory pass; it refuses to do so while other containers run. It writes raw samples to results/raw.jsonl, medians to results/summary.json, and copies the summary to website/src/data/benchmarks.json, which is what the tables on this site render.