Stochastic optimization

Pick some action and let’s roll! $\begin{array}{rl}{a}_1,\dots ,a_T& =\arg \underset{a_1,\dots ,a_T}{\max }\underset{\text{don’t care what this is}}{\underbrace{J\left(a_1,\dots ,a_T\right)}}\\ A& =\arg \underset{A}{\max }J(A)\end{array}$ We can simply do random shooting (parallelize very well), or doing these:

CEM Monte Carlo tree search

Derivative based methods

$\begin{array}{rl}& \underset{{\mathbf{u}}_1,\dots ,{\mathbf{u}}_T}{\min }\sum _{t=1}^{T}c({\mathbf{x}}_t,{\mathbf{u}}_t)\text{s.t.}{\mathbf{x}}_t=f({\mathbf{x}}_{t-1},{\mathbf{u}}_{t-1})\\ & \underset{{\mathbf{u}}_1,\dots ,{\mathbf{u}}_T}{\min }c({\mathbf{x}}_1,{\mathbf{u}}_1)+c(f({\mathbf{x}}_1,{\mathbf{u}}_1),{\mathbf{u}}_2)+\cdots +c(f(f(\dots )\dots ),{\mathbf{u}}_T)\end{array}$ $2^{nd}$ order method tends to work better than first order method, as the chain can be long and it’s easy to see vanishing / exploding gradients.

Shooting vs collocation method: we can either optimize over actions only, or both action and state with constraints (conditions better) ${\min }_{{\mathbf{u}}_1,\dots ,{\mathbf{u}}_T,{\mathbf{x}}_1,\dots ,{\mathbf{x}}_T}{\sum }_{t=1}^{T}c({\mathbf{x}}_t,{\mathbf{u}}_t)\text{s.t.}{\mathbf{x}}_t=f({\mathbf{x}}_{t-1},{\mathbf{u}}_{t-1})$

In the linear case, we have a nice dynamic programming thing: LQR]