The real overview of basics

I’m so disappointed In the HuggingFace online course, I would start my notes from scratch with DeepMind x UCL 2021 lecture series.

The why

The reward hypothesis:

Any goal can be formalized as the outcome of maximizing a communal award reward.

I would also like to put this image here from LLM Post-Training that I find helpful:

State

As we can see from control theory, the definition of state can be crucial. There are environment state and agent state. Let’s talk about agent state here. The history is

$$H_t=O_0,A_0,R_1,O_1,\dots ,O_{t-1},A_{t-1},R_{t,}{O}_t$$

The markovian agent state $S_t$ typically is some compression of $H_t$. The full hitory is Markov (but keeps growing). In an environment with partial observability, the observations are not Markovian, making them a POMDP. The environment state may still be Markovian, but the agent does not know it.

Value function

A quick reminder: we need the notion of value / state, since the reward only have meaning in a state. It’s in a specific state that we receives reward, not with a specific action. In other words, state is how we define the state where we could get reward.

The actual value function is defined as

$$\begin{array}{rl}{v}_{\pi }(s)& =\mathbb{E}\left[G_t\mid S_t=s,\pi \right]\\ & =\mathbb{E}\left[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+...\mid S_t=s,\pi \right]\end{array}$$

$\gamma$ $0$ means we only care about the instant reward, while $1$ means it considers long term and immediate rewards equally. Personally I’m interested to see if there are alternative definitions. There’s no reason it’s defined like this.

The bellman equation is its recursive form, the Bellman equation.

$$\begin{array}{rl}{v}_{\pi }(s)& =\mathbb{E}\left[R_{t+1}+\gamma G_{t+1}\mid S_t=s,A_t\sim \pi (s)\right]\\ & =\mathbb{E}\left[R_{t+1}+\gamma v_{\pi }(S_{t+1})\mid S_t=s,A_t\sim \pi (s)\right]\end{array}$$

Now note here that $a$ is chosen by policy $\pi$ in state $s$. We can take out the dependency on a specific policy by stating that the following holds for the optimal case:

$$v_{\ast }(s)=\underset{a}{\max }\mathbb{E}\left[R_{t+1}+\gamma v_{\ast }(S_{t+1})\mid S_t=s,A_t=a\right]$$

model

A model predicts what the environment will do next. It can predict the next state, or the next (immediate reward). Note it does not directly give a policy.

$$\begin{array}{rl}P(s,a,s^{\prime })& \approx p(S_{t+1}=s^{\prime }\mid S_t=s,A_{t=a)}\\ \mathcal{R}(s,a)& \approx \mathbb{E}\left[R_{t+1}\mid S_t=s,A_t=a\right]\end{array}$$

Prediction and control

This is like the Expecation-maximization algorithm.

• Prediction: evaluate the future (for a given policy)
• Control: optimise the future (find the best policy)

Info

The rest of the note comes from 285-rl-basics.pdf

In RL, we almost always care about expectation since that makes things smooth. $r(x)$ - not smooth $E_{{\pi }_{\theta }}[r(x)]$ - smooth in $\theta$

The slide also gives good explanation to algorithm tradeoffs, why we need different kind of them, etc.