Acme

---

Notes written with Claude Code (Claude Opus 5.5) from a reading discussion.

---

Acme is a paper about API design for RL systems, not about algorithms. The problem: “scaling a simple prototype to a full distributed system may require a complete re-implementation of the agent.” The idea: split every agent into an actor, a replay, and a learner that touch in exactly two places, a dataset iterator and a variable source. Each can be an in-memory call or an RPC without the algorithm code knowing, so the same agent runs in one process for debugging or across hundreds of machines. The rest is DeepMind writing down the discipline it had converged on through Ape-X, D4PG, R2D2 and IMPALA. (Notes are on v2, 2022, which adds builders and more offline and imitation agents.)

Overview

1. Environment loop. dm_env returns $(r_t,d_t,o_{t+1},e_{t+1})$. The loop calls select_action, steps the environment, then calls observe and update on the actor.
2. Actor evaluates the policy. An Adder writes replay items, and a VariableClient pulls fresh parameters.
3. Replay (Reverb) stores items, and a rate limiter fixes samples per insert.
4. Learner reads batches through a dataset iterator, runs step(), and serves get_variables().
5. A builder constructs each piece. run_experiment wires them in one process, and make_distributed_experiment wires them as Launchpad nodes.

Each component does one job, so it can be cloned or swapped

Actor: act, record, fetch parameters. It has four methods: select_action, observe_first, observe, update. The paper deliberately keeps it thin: “we generally prefer to limit the use of this interface to only those functions necessary for generating actions and storing the experience data”, because that is what lets it be cloned across machines. Parameters are pulled: the actor periodically polls a VariableSource (usually the learner) through a background-thread VariableClient. Most agents just supply an ActorCore of three pure functions: init (recurrent state), select_action, and get_extras. The last stores behavior-policy information, such as log-probs for V-trace or Retrace, alongside each step, because the learner can’t recover it later.

Adder + Reverb: what a sample is, and how fast it’s consumed. The Adder defines the sampleable unit: transitions, n-step transitions, sequences, or episodes. That choice is also how agents pick n-step backups. Reverb provides tables, FIFO/LIFO/prioritized sampling, and priorities that either side can update (see Reverb).

The rate limiter is part of the algorithm

The acting-to-learning ratio “can have a dramatic effect not only on the sample efficiency … but also on its long-term learning performance and stability.” In a distributed system, hardware decides it implicitly. Reverb’s RateLimiter pins it within a tolerance by blocking whichever side gets ahead. With FIFO sampling it also implements a queue for on-policy agents such as IMPALA.

Learner: step() + get_variables(), built on an iterator. Because it only sees an iterator, offline RL is free:

Quote

“Any algorithm implemented via the Learner interface can be applied in the offline setting purely by eliminating the actor/replay part of their algorithm.”

…which, the paper adds, “certainly does not mean that they will perform well”. Offline data comes from RLDS episodes.

Environment discount. $d_t=0$ at episode end means termination (don’t bootstrap). $d_t>0$ means truncation (bootstrap). This is the classic time-limit bug, prevented by the interface.

Builders separate what to run from how to run it

A builder is the full definition of an agent: make_policy, make_replay_tables, make_adder, make_actor, make_learner, make_dataset_iterator. Each takes the environment spec and the networks. They’re factories rather than objects because in a distributed run each piece is built in a different process. They also compose: imitation and LfD agents wrap a base agent’s builder.

run_experiment (Listing 2) builds everything in order: networks → policy → replay → iterator → learner → adder → actor, then runs the environment loop. The learner is passed as the actor’s variable source. _make_learning_actor only exists so a single thread also runs learner steps; it is not an actor/learner merge. Evaluation runs a separate loop with an eval=True policy that doesn’t write to replay.

make_distributed_experiment takes the same inputs and returns a Launchpad program: a graph whose nodes are the environment loops, the learner and Reverb, with RPCs as edges. It can be launched as threads, processes or machines. The rate limiter replaces the learning actor. In the paper’s words, the split “largely divorces the question of how to run (and scale) an algorithm from what to run.”

Agents differ almost only in the learner

“Their primary differences will be contained within the learner”. Data handling (n-step, priorities, queues) comes from shared machinery, and being distributed isn’t an algorithmic distinction.

• Online:
  • DQN (with Rainbow-style extras) and Munchausen DQN
  • R2D2 (recurrent, sequence replay with burn-in)
  • D4PG, TD3, SAC, MPO (plus distributional and mixture-of-Gaussians critics)
  • on-policy: PPO, and IMPALA (V-trace over a FIFO queue)
• Offline: BC, TD3+BC, CQL, CRR / one-step CRR, BVE, REM.
• Imitation: AIL on TD3, SQIL on SAC, PWIL on D4PG, each wrapping its base agent.
• Learning from demonstrations: SACfD and TD3fD, where the demo-to-interaction sampling ratio is a hyperparameter.

The experiments validate the implementations rather than rank them: one hyperparameter configuration per agent, roughly matching published results. The speed test runs the same R2D2 code with a V100 learner, then TPU-v2 1×1 (1.7× faster) and 2×2 (5.6×).

”Of course, how else?”

The decomposition feels obvious because it’s codified practice. In 2020 the alternatives were real:

• actors that send gradients (A3C, Gorila);
• one script per algorithm (Baselines);
• the textbook agent that acts and learns in one object;
• SEED RL, which moves inference onto a central accelerator.

Acme’s contribution is the constraints: the actor never learns, the learner only sees an iterator, the acting-to-learning ratio is explicit, and single-process mode is the same code.

The topology assumes small models and cheap CPU inference. The paper uses 3×256 MLPs (under 1 MB) and the Nature DQN torso (about 7 MB), and had to enlarge R2D2’s ResNet just to make the learner the bottleneck. With multi-GB policies, weight transfer dominates, and designs move to SEED-style central inference or LLM-RL’s colocated generation and training. The seams carry over; the topology doesn’t.