Francois Chollet — Why the biggest AI models can't solve simple puzzles

单集文稿 ...

• 00:00

  Okay, today I have the pleasure to speak with Francois Chollet,

• 00:04

  who is a AI researcher at Google and creator of Keras,

• 00:09

  and he's launching a prize in collaboration with Mike Canouf,

• 00:13

  the co-founder of Zapier, who we'll also be talking to in a second,

• 00:16

  a million-dollar prize to solve the ARC benchmark that he created.

• 00:20

  So, first question, what is the ARC benchmark, and why do we even need this prize?

• 00:24

  Why won't the biggest LLM we have in a year be able to just saturate it?

• 00:27

  Sure.

• 00:28

  So ARC is intended as a kind of IQ test for machine intelligence.

• 00:32

  And what makes it different from most LLM benchmarks out there is that it's designed to be resistant to memorization.

• 00:39

  So if you look at the way LLMs work, they're basically this big interpolative memory.

• 00:45

  And the way you scale up their capabilities is by trying to cram as much knowledge and patterns as possible into them.

• 00:52

  And by contrast, arc does not require a lot of knowledge at all.

• 00:58

  It's designed to only require what's known as core knowledge,

• 01:02

  which is basic knowledge about things like elementary physics,

• 01:07

  objectness, counting, that sort of thing.

• 01:10

  The sort of knowledge that any four-year-old or five-year-old possesses.

• 01:14

  But what's interesting is that each puzzle in arc is novel,

• 01:19

  is something that you've probably not encountered before,

• 01:22

  even if you've memorized the entire internet.

• 全部文稿