John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI

单集文稿 ...

• 00:00

  Today I have the pleasure to speak with John Shulman,

• 00:03

  who is one of the co-founders of OpenAI and leads the post training team here.

• 00:08

  And he also led the creation of ChatGBT and is the author of many of the most important and widely cited papers in AI and RL,

• 00:16

  including PPO and many others.

• 00:18

  So John, really excited to chat with you.

• 00:20

  Thanks for coming on the podcast.

• 00:21

  Thanks for having me on the podcast.

• 00:22

  I'm a big fan.

• 00:23

  Oh, thank you.

• 00:23

  Thank you for saying that.

• 00:25

  So the first question I had is,

• 00:27

  we have these distinctions between pre-training and post-training beyond what is actually happening in terms of loss function and training regimes.

• 00:35

  I'm just curious, taking a step back conceptually, like what kind of thing is pre-training creating?

• 00:41

  What does post-training do on top of that?

• 00:44

  In pre-training,

• 00:45

  you're basically training to imitate all of the content on the internet or on the web,

• 00:50

  including websites and code and so forth.

• 00:54

  So you get a model that can basically generate content that looks like random web pages from the internet.

• 01:01

  And the model is also trained to maximize likelihood where it has to put a probability on everything.

• 01:08

  So the objective is basically predicting the next token given the previous tokens.

• 全部文稿