3.2 · Behavioral Cloning, Multimodality, and Shift
Behavioral cloning is simple and powerful, but it learns under expert-visited states and deploys under learner-visited states. Multimodal actions and compounding error require explicit treatment.
- Formulate behavioral cloning as supervised policy learning.
- Recognize when mean actions are unsafe.
- Explain covariate shift and the role of DAgger.
Behavioral cloning
The policy learns to predict expert actions from observations and goals. Loss choice should match the action distribution: mean-squared error assumes a unimodal target, while likelihood-based or generative policies can represent alternatives.
The dangerous average
When left and right paths around an obstacle are both valid, their average may collide. A low average regression loss can therefore hide a physically invalid central action.
Compounding shift
Small policy errors change the next state, creating observations that were rare in expert data. DAgger addresses this by collecting expert labels on states visited by the learner and iteratively aggregating them.
Imitation quality depends on the states the learner will visit, not only on one-step error under expert data.
