Teaching a model what a good answer looks like
RLHF (reinforcement learning from human feedback) refines a model’s responses using human judgments of better and worse. It was one of the things that made ChatGPT work, adding the helpfulness and manner that pre-training alone does not produce.
Roughly how it works
A model fresh from pre-training is knowledgeable but blunt, and will happily produce inappropriate content. RLHF addresses that:
- The model produces several candidate answers, and human raters choose which is better
- Those judgments train a reward model that stands in for human preference
- Reinforcement learning then adjusts the model to score well against that reward model
It resembles training an animal with rewards for desired behavior, applied to the quality of conversation.
What changed
RLHF is why assistants follow instructions cooperatively, respond politely, and decline dangerous requests. The approachability of current chat AI is a product of this stage, and it is the best-known practical method for alignment.
Limits and successors
It is expensive, requiring many human raters. Rater preferences transfer into the model. And optimizing for what people like can push a model toward agreeable answers over accurate ones. Research into supplements — having AI do the rating, or stating principles for the model to follow — is active. Knowing that a model’s character is shaped deliberately in a finishing stage explains a lot about how these systems behave.
The shock of ChatGPT’s launch owed more to the quality of the interaction than to raw knowledge, and RLHF is what produced it. An unglamorous name for the technique that turned AI into a tool ordinary people use.