What is RLHF in AI?

In plain English

After pre-training, a language model knows a lot but behaves oddly: it may ramble, ignore the question or say something harmful, because it is only predicting likely text. RLHF, short for reinforcement learning from human feedback, teaches it what a good answer looks like.

It works in three steps. People write some example answers to show the style wanted. Then the model produces several answers to the same question, and people rank them from best to worst. Those rankings train a second model, a “reward model”, that learns to predict which answers people would prefer. Finally, the main model is trained by reinforcement learning to produce answers the reward model scores highly.

Why it matters

RLHF is a big part of why ChatGPT felt so different from earlier models. OpenAI said it trained ChatGPT with RLHF, using methods from its 2022 InstructGPT research. Most chatbots now use RLHF or a close cousin of it, sometimes with AI feedback replacing some of the human ratings. It is how models learn to follow instructions, admit uncertainty and decline harmful requests, which makes it one of the main practical tools of AI alignment.

An example

A model is asked, “How do I get rid of a headache?” One answer gives sensible tips and suggests seeing a doctor if it persists; another lists obscure remedies with total confidence. Human raters prefer the first. After many thousands of comparisons like this, the model learns to answer in the first style.

What to watch out for

The model learns what raters like, which is not always what is true. RLHF can make models overly agreeable, flattering users or confirming their mistakes, a behaviour researchers call “sycophancy”. The values built in also depend on who wrote the rating guidelines and who the raters were.