Skip to content
pgmem

Architecture

pgmem is PostgreSQL 18.3 compiled to WebAssembly once, translated to Go ahead of time, and run as a real cluster (a postmaster and a process per connection, each a WebAssembly instance) against an in-memory file system. A small bridge exposes it on a loopback TCP port by default, with an optional Unix domain socket.

Runtime at a glance

Section titled “Runtime at a glance”
Query pathA query flows from a client through the bridge and PostgreSQL backend to the in-memory file system.One ServerPostgreSQL driverTCP via the DSNGo: net.Pipe (Server.Dial)pgmem bridgea backend process per connectionPostgreSQLpostmaster + processesIn-memory VFSdata + WAL
Both transports take the same path. The postmaster starts a backend process per connection; locks and notifications are PostgreSQL's own, through the shared memory.
From template to forksPrepare a template once, snapshot it, and start independent servers from a bounded pool of fork slots.Templatemigrations + seed datarunning serverSnapshotfrozen data stateMaxForksMaxForks slots for this snapshotFork 1data copyserver + URLFork 2data copyserver + URLFork Ndata copyserver + URLFork()When full, wait; Close() frees a slot
Each fork owns an independent data directory. Reset restores its snapshot in place, keeping the same port and client connections.

How language wrappers run: Go uses the library inside the application process. Python, Java and Node.js launch the cmd/pgmem binary; drivers connect to each server’s DSN. JSON Lines and Node.js control sockets manage forks, not query traffic.

Layers

Section titled “Layers”
Layer What it is
PostgreSQL source The PGlite fork (electric-sql/postgres-pglite), pinned by commit and checksum in wasm/postgres-pglite.lock, patched at build time by wasm/patches.py.
WebAssembly module wasm/build.sh links one static module with Emscripten. Loadable modules and extensions are linked in, not loaded dynamically. There is no OpenSSL, ICU or zlib.
Generated Go wasm2go translates the module into the pure-Go package internal/aot/pgaot. The .wasm file is a build intermediate; only Go ships, so there is no wasm runtime and no cgo.
Host internal/host implements the system calls, clock, memory growth and exit unwinding the module imports. Entropy, CRC-32C, hashes and pgcrypto’s ciphers run on Go’s crypto and hash packages.
File system internal/vfs is an in-memory POSIX-like file system holding the data directory, WAL and share files. Nothing is written to the host disk.
Engine internal/engine runs the postmaster and its processes. PostgreSQL is built with EXEC_BACKEND, so every process is a fresh module instance on its own goroutine; internal/host is the kernel they share: pids and signals, System V shared memory mapped at the same address into every instance, semaphores, and the sockets that carry connections to the postmaster.
Bridge pgmem.go accepts connections over loopback TCP, an opt-in Unix domain socket, or an in-process net.Pipe and hands each one to the postmaster. It remembers a session’s startup packet, prepared statements and LISTEN set so that a reset, which restarts the cluster, leaves client connections working.
Embedded data internal/pgdata/pgdata.tar.zst is the output of initdb, produced at build time. Unpacking it into the file system is why a server starts in about a tenth of a second.

A query’s path

Section titled “A query’s path”
  1. A driver connects over loopback TCP or a Unix socket using the DSN, or a Go in-process dialer connects through net.Pipe.
  2. The bridge hands the connection to the postmaster, which starts a backend process for it; the backend reads the startup packet and authenticates as a server would.
  3. The bytes flow to that backend’s socket and PostgreSQL’s main loop runs in generated Go.
  4. Locks, notifications and the other sessions are PostgreSQL’s own, through the shared memory.
  5. Page reads and WAL writes hit the in-memory file system.
  6. Responses are copied back through the same connection.

Snapshots and forks

Section titled “Snapshots and forks”

A snapshot stops a prepared server cleanly, deep-copies its in-memory data directory and starts it again. A fork copies that image again, starts a new cluster on the copy and listens on its own endpoint. A reset does the same for a running server in place: the endpoint and client connections stay, so a URL an application captured at import time remains valid. Startup replays almost no WAL, so a fork takes milliseconds, and the loaded engine code is shared by every server in the process.

Forks are full clusters. They are independent of each other, so tests that write can run in parallel, and each one holds its own buffer cache and its own view of the data directory, shared with the snapshot until a file is written (copy on write). MaxForks bounds how many exist at once; asking for another blocks until one is closed.

External processes

Section titled “External processes”

The Python, Java and Node.js packages bundle pgmem, a standalone binary built from cmd/pgmem. A wrapper spawns it, reads one JSON line with the template’s DSN, and sends snapshot, fork and close requests as JSON lines on stdin. When the parent exits, stdin closes and the binary shuts everything down, so a crashed test run cannot leave servers behind. Node.js test workers, which are separate processes, fork and reset through a loopback control socket instead; forks made on a socket connection close when that connection ends.

Go can use the same binary through pgmemprocess, a lightweight client that does not import the embedded engine. Install the binary separately and choose TCP or Unix socket connections. The fixture prepares one snapshot and exposes the same driver helpers as embedded Go tests.

Unix mode creates private socket directories on the host. Normal shutdown removes them; PostgreSQL data and WAL remain in memory. A forced kill can leave socket entries.

Every test process (a pytest-xdist worker, a forked test JVM) starts its own binary. Within it, forks behave exactly as in Go.