Dyna

An algorithm proposed by R. Sutton.

1. collect some data, consisting of transitions $(s,a,s^{\prime },r)$
2. learn model $\hat{p}(s^{\prime }|s,a)$ (and optionally, $\hat{r}(s,a)$)
3. repeat $K$ times:
   1. sample $s\sim \mathcal{B}$ from buffer
   2. choose action $a$ (from $\mathcal{B}$, from $\pi$, or random)
   3. simulate $s^{\prime }\sim \hat{p}(s^{\prime }|s,a)$ (and $r=\hat{r}(s,a)$)
   4. train on $(s,a,s^{\prime },r)$ with model-free RL
   5. (optional) take $N$ more model-based steps

The algorithm mentioned here is close to MBPO.