Reverb

---

Notes written with Claude Code (Claude Opus 5.5) from a reading discussion.

---

Reverb is the replay buffer under Acme. It is a gRPC server that sits between many actors that write experience and learners that sample it. The ideas are simple, and the paper’s vocabulary hides that a little. Here is how it works:

• An actor streams steps through a client Writer. The Writer packs K consecutive steps into a Chunk, stores each field column-wise, compresses it, and sends it to the server.
• The server keeps chunks in RAM and counts references to each one.
• A Table holds Items. An Item is a view into chunks: (chunk keys, offset, length, priority).
• Each Table has a Sampler and a Remover. These pluggable Selectors decide which item to hand out and which to evict.
• A RateLimiter blocks inserts or samples to keep the samples-per-insert ratio (SPI) inside a band.
• The learner reads through a tf.data dataset, which decompresses on the learner’s side.

None of these pieces is new. In storage terms, Reverb is a column-compressed, append-only log (like Kafka record batches or Parquet row groups) with many indexes on top, pluggable eviction, and backpressure. The contribution is assembling them into one shared service that thousands of concurrent clients can use, and that can act as ER, PER, or an on-policy queue just by configuration.

The problem: one buffer, many clients running at different speeds

In a single-process loop, a replay buffer is a Python list and the replay ratio is a for-loop. Distributed RL splits the agent into actors and learners on different machines that run at speeds nobody controls. The buffer then has to take inserts from hundreds or thousands of writers and serve samples to learners. It also has to switch between ER, PER and a FIFO queue for on-policy data without code changes, and keep the ratio of training to data collection from drifting. SEED RL covered the ER/PER/FIFO part. But in SEED only the learner inserts and samples, so the ratio is trivially controlled, and there is no way to put several tables over the same data. Reverb’s design follows from removing those two restrictions.

A chunk is a compressed row group, not a file

My first model was “chunks are files: buffer in memory, flush to disk, avoid small files.” That’s the wrong frame. Everything lives in server RAM. Disk is used only for checkpoints. The reason to chunk is compression across time. Consecutive RL steps are highly redundant (two adjacent Atari frames share most of their pixels). So the Writer stacks each field over K steps (all K observations together, all K actions together) and compresses each column. The paper reports up to 90% compression on 40-frame Atari sequences. Sending fewer, bigger gRPC messages is a secondary benefit.

Because every step in a stream has the same signature (same nested structure, shapes and dtypes), the flattened stream is effectively a 2-D table: one row per step, one column per leaf tensor. A chunk is a horizontal slice of that table.

Who pays for compression

Compression isn’t free, but Reverb puts the cost where it’s cheapest:

• The Writer on the actor builds and compresses the chunk.
• The server stores the compressed bytes and never decompresses them.
• The learner’s dataset decompresses after sampling.

So the CPU work is spread across many actor machines and the learner’s input-pipeline threads, which are mostly idle anyway. The real bottlenecks are network and RAM. The 11 GB/s ceiling in the benchmarks is the network. Spending actor CPU to send about 10× fewer bytes, and to fit about 10× more experience in server memory, is a good trade for pixels. For small low-dimensional states it may not pay off, and chunk length is a knob. (The codec, zstd with optional delta encoding, comes from memory of the codebase. The paper doesn’t name it.)

Items are views, so the stream is stored once

An Item doesn’t own any data. That’s not abstraction for its own sake. It solves a specific RL problem: samples are usually overlapping windows over a single stream (steps 0–2, 1–3, 2–4, …). Different tables may also want different window lengths or priorities over the same stream. If each Item held a copy, every step would be stored N times. With Items as views, the stream is stored once.

The paper’s example of two tables over one stream shows how small the client API is:

with client.writer(3) as writer:          # keep the last 3 steps in the client buffer
    ts, a, done = env_step(None)
    step = 0
    while not done:
        writer.append((ts, a))
        if step >= 1:
            writer.create_item(table='my_table_a', num_timesteps=2, priority=1.5)
        if step >= 2:
            writer.create_item(table='my_table_b', num_timesteps=3, priority=1.5)
        ts, a, done = env_step(a)
        step += 1

The rest of the design follows from items being views:

• Refcounting. The ChunkStore counts how many Items reference each chunk and frees the chunk when the count reaches zero. Freeing memory happens outside the mutex-protected Table operations, which matters for throughput.
• Ordering. The Writer holds a new Item locally until all of its chunks have been sent. That’s what lets many Items safely share data without sending it twice.
• Granularity cost. Sampling an Item ships all of its chunks whole. If the item length N isn’t a multiple of the chunk length K, the extra steps go over the wire unused. So pick K to divide N.
• “Decoupled from storage” is a possibility, not a feature. The paper points out that a chunk could reference data on disk instead of owning it. It wasn’t built.

Selectors decide what kind of buffer a Table is

This is where Reverb is actually used. Each Table pairs a Sampler (which item to return) with a Remover (which item to evict when the table is full). Built-in options are FIFO, LIFO, Uniform, Max/Min-Heap, and Prioritized (PER: $P(i)=p_i^C/{\sum }_k{p}_k^C$). Together with a rate limiter, the same Table becomes:

You want	Sampler	Remover	Rate limiter
Uniform ER (Acme’s D4PG)	Uniform	FIFO	MinSize(1)
PER	Prioritized	FIFO	SampleToInsertRatio
On-policy queue (IMPALA, PPO)	FIFO	FIFO	Queue(size)
Stack	LIFO	LIFO	Queue(size)
Keep the highest-priority data	any	MinHeap	—
Latest policy weights (TF-Agents SAC)	Uniform	FIFO, max_size=1	MinSize(1)

The last row is a nice trick: a one-item table works as a parameter server. The learner inserts its weights, actors sample them, and MinSize(1) makes actors wait until the first weights exist.

Selectors are deliberately blind. They can’t look at an item’s contents, only at their own state built from watching table operations (insertion order, priorities, which clients can update). That restriction is for performance. Anything content-aware has to be expressed as a priority. Items also leave a table when they hit max_times_sampled. Table extensions are hooks that run under the table’s mutex, used for things like spreading priority updates to neighboring items.

The rate limiter owns the replay ratio

SPI = items sampled / items inserted. It’s essentially the replay ratio (gradient updates per collected sample), a hyperparameter RL is quite sensitive to. In a synchronous loop you set it with a for-loop. In an async distributed system, actors and learners drift apart and the hardware ends up choosing the ratio. The rate limiter belongs in the buffer because the buffer is the only component that sees both rates.

The mechanism is backpressure. A limiter has a minimum table size, a target SPI, and an upper and lower bound around it. Think of a cursor: each insert moves it forward by the target SPI, and each sample moves it back by one. Either side blocks when its move would push the cursor outside the band:

The preset limiters are the same mechanism with different parameters:

• SampleToInsertRatio(min_size, samples_per_insert, error_buffer): the general case. A wider error_buffer means less blocking near equilibrium.
• MinSize(n): bounds set to ±∞. It only stops sampling until the table holds n items.
• Queue(size): SPI 1 with bounds [0, size]. Inserts block when the queue is full and samples block when it’s empty.

The limiter counts Items, not data. If each item references 16 frames instead of 4, the data throughput changes even though SPI stays the same.

The "deadlock" is the limiter doing its job

If actors die or the learner consumes faster than the actors can sustain, sampling simply blocks, possibly forever. Set rate_limiter_timeout_ms on the dataset. When the timeout expires, the server tells the iterator that it’s safe to end the sequence, which behaves like EOF on a file-backed dataset.

Why it’s a service, and what limits it

As a data structure, a replay buffer doesn’t need to be a service. But the distributed setting is the whole reason the paper exists: a local buffer is trivial, and a shared one under heavy concurrency is the real problem. That’s why the evaluation is about server throughput and not RL returns.

The benchmarks deliberately stress one server with incompressible random float32 data, items that don’t share chunks, and up to 200 clients on separate machines:

• Insert: about 11 GB/s or 60k items/s.
• Sample: about 11 GB/s or 600k items/s.
• Scaling: throughput scales linearly with clients until it hits one of those caps. Adding more clients after that doesn’t degrade it.

The byte ceiling is the same at every payload size, which points to the network. The 10× gap in item rate between inserting and sampling comes from Table mutex contention. Sampling had already been optimized for it and inserting hadn’t. Splitting the load across 8 tables on one server raised insert QPS about 3×.

Implementation snapshot (2021)

• Sharding means independent servers with no replication or coordination. Clients round-robin inserts through a gRPC load balancer. Samples are requested from several servers in parallel and merged into one stream, which also hides tail latency and single-server failures.
• Checkpointing is triggered by a client call and blocks every operation while the ChunkStore and Tables are written to disk. It can take minutes.
• Sampling client: each Sampler keeps long-lived gRPC streams with max_in_flight_samples_per_worker items of prefetch. Deterministic selectors like FIFO need a single stream to preserve order.
• ReverbDataset is a C++ tf.data source, which is also used from JAX.

Contrast: RLinf puts the buffer inside the trainer

Not from the paper. This is from RLinf’s docs as of October 2026. RLinf goes almost the opposite way, and it’s closer to my original “files” mental model:

• The buffer is not a service. Rollout workers publish trajectory pieces over a Channel. A collector assembles Trajectory objects, and the trainer puts them into a TrajectoryReplayBuffer that lives in its own process. RLinf splits Reverb’s two jobs, moving data and storing/sampling it, into two components.
• Data on disk, index in memory. Each trajectory is written with torch.save or pickle by a single-thread async executor, and the index is updated once the write finishes. A small FIFO cache keeps recent trajectories in memory, flattened from [T, B, …] to [T·B, …] so it can sample individual transitions.
• The selector is hardcoded. RLinf samples uniformly over all steps in the most recent sample_window_size trajectories (using bucketize to map a global index back to its trajectory) and loads each trajectory once per batch. In Reverb terms, that’s roughly a Uniform sampler with a FIFO remover.
• No rate limiter in the buffer. The only gate is min_replay_buffer_size on the prefetching dataset, which is the equivalent of MinSize. The training loop and scheduler decide how many updates to run per collected sample. I haven’t checked how tightly they bound it.
• Sharding is per trainer rank. Checkpoints are saved and loaded in shards keyed by local_rank and world_size.

The difference comes from the workloads. Reverb targets thousands of cheap CPU actors, small networks and very high insert rates from many writers, where a central concurrent server with backpressure makes sense. RLinf targets big VLA models and image observations from a few GPU rollout workers. There, inserts are rare, each trajectory is large, and the data is valuable (especially real-robot data). Persisting it to disk for resume and offline reuse, and sampling locally in the trainer, is the simpler fit.