Reinforcement Learning Map: Model-Based vs. Model-Free
A beginner-oriented tutorial organizes reinforcement-learning methods by whether they use an environment model. Using a corridor-robot example, it distinguishes planning with a known or learned model from learning directly from experience. The article introduces dynamic programming, model predictive control, Monte Carlo tree search, learned models, value-based methods, policy-based methods, and actor-critic methods. It also clarifies that “model” here means environment dynamics and rewards, rather than a large language model or the use of a neural network.
The tutorial places multiple reinforcement-learning methods in a single framework and explains the distinction between an environment model and other meanings of “model,” providing a useful foundation for understanding the field.
【上期回顾】 上一篇讲了价值：状态价值 Vπ(s)V_\pi(s)Vπ(s) 是照策略 π\piπ 走的平均回报，动作价值 Qπ(s,a)Q_\pi(s,a)Qπ(s,a) 是先做动作 aaa、再照策略走的平均回报。贝尔曼方程说，这一格的价值可以从下一格的价值推出来；贝尔曼最优方程说，每格都挑最好的动作，就能得到最优策略：走廊里是一路向右，V∗=6.2, 8, 10V_*=6.2,\ 8,\ 10V∗=6.2, 8, 10。 那些方程能直接解，是因为我们知道走廊的全部规则。这一篇就从“规则”说起：知道规则怎么办，不知道又怎么办？顺着这个问题，把强化学习的各种算法放到一张地图上。
▲ 图 1 本篇例子：走廊里的扫地机器人（规则同前两篇）
【设定】走廊的规则