What is Intelligence?
A lecture in seven parts

What is
Intelligence?

Everything that follows is built from one distinction: the difference between a proof and the truth.

Those two colours keep those two jobs for the whole hour: proof, notation, syntaxtruth, meaning, semantics.

Full disclosure

This entire talk was generated by an AI.

It took me 40+ years to become someone who could prompt it into existence. I could do that today. I could not have done it yesterday.

I met people on the way vastly more intelligent than me, including some of my students. Without them this talk would not exist.

So, join me in preventing our smartphones, social media, and AI from distracting us, and help such people find you by becoming who you are and no one else. This talk is about that.

Acknowledgement

This is a collaborative effort.

They inspired me, encouraged me, and helped me leave my computer-science comfort zone — and reach out to everyone affected by AI.

Not by repeating

The booming. The dooming. Both are forecasts about a product cycle, and both expire on the next one.

But by academic grounding

Results that were already true before this technology arrived, and will still be true after whatever replaces it. That is the whole of what follows.

The question · and why we won't answer it

"Is it intelligent?" is a badly posed question.

It asks about a hidden property of a system, in a language — English — with no precise semantics for the word intelligent.

Psychology met this first and answered it best: you cannot measure a construct, only an operational definition of it — and the definition is never the thing. A century of psychometrics is a century of taking that gap seriously.

We will spend the next hour earning the right to ask a better question, one that stays valid no matter what technology arrives next.

The question of whether machines can think is about as relevant as the question of whether submarines can swim. Edsger W. Dijkstra, 1984
The short answer · stated now, earned later

Intelligence is finding the next language — which means finding truth you cannot yet prove.

  1. Intelligence develops new formal languages — or at least new properties, expressible and provable in languages we already have.
  2. That requires finding and understanding promising unproven truth. You apprehend something as true while no proof exists; the notation is built afterwards, to hold what you already saw.
  3. Nothing hands you that step. No rule, no procedure, no quantity of compute derives it — a result we will prove, not assert. So it is a challenge for a first-year student, for a Nobel laureate, and for any AI, equally.

The rest of the hour is the derivation — and the reason this is good news for everyone in this room, whatever you are studying.

Route

Seven parts.

IVastnesswhy 34 bytes beat the universe
IIInfinitywhy meanings outnumber notations
IIISelf-referencewhy prooftruth
IVCosttime, space, energy
VEverywherebiology, mind, money, music
VIMachineswhat today's AI cannot escape
VIIPracticea method with no expiry date

Four of these parts end in a theorem that sounds like bad news. All four are good news. That reversal is the point of the lecture.

Part I

Vastness

First a feeling for size — then the smallest possible distinction, and everything it builds.

The map

One line carries the whole talk: small, vast, then two sizes of endless.

log scale · then off the end of every scale

Left. From one thing to every atom there is — the whole physical world in eighty steps, each one ten times the last.

The wall. No number of steps crosses it. Infinity is not the far end of the line; it is what the line never reaches.

Right. Past the wall, endlessness comes in sizes: one counts every notation, the other every meaning — and is strictly bigger.

Anchors

A million, a billion, a trillion. Three words one syllable apart — three different worlds.

106 sa million seconds · 11½ days
109 sa billion seconds · 32 years
1012 sa trillion seconds · 32,000 years

A million seconds ago you had not yet packed for today. A billion seconds ago you were a child, or not yet born. A trillion seconds ago somebody was painting the walls of the Chauvet cave.

Each step is three zeros — the same three zeros, three times over. The words rhyme, so the mind files them together and treats the gaps as small. The gaps are not small. They are the subject.

The habit

Never accept a number you cannot picture. Find the anchor — a fortnight, a career, an ice age — or admit you are repeating the number, not reading it.

Units

Kilo, mega, giga, tera: the same three zeros, again and again.

prefixfactorbytes · how much it holdshertz · how often it happens
kilo103KB — a long paragraphkHz — the top of human hearing, 20 kHz
mega106MB — a novelMHz — the first IBM PC, 4.77 MHz, 1981
giga109GB — a thousand novelsGHz — this laptop; Wi-Fi at 2.4
tera1012TB — a million novelsTHz — infrared light

A byte is eight bits — one keystroke. A hertz is one beat per second. The left column counts things, the right counts events, and the prefixes do not care which.

Hold a nanosecond

At 1 GHz a beat lasts a billionth of a second, and light gets 30 cm in that time. Grace Hopper handed those 30 cm out as lengths of wire, so a room could feel why a signal cannot cross a desk and return inside one beat.

Two kilos

Kilo means a thousand. In memory it means 1,024. Both are in daily use.

base 10base 2gap
kiloKB = 103KiB = 210 = 1,024+2.4%
megaMB = 106MiB = 220+4.9%
gigaGB = 109GiB = 230+7.4%
teraTB = 1012TiB = 240+10.0%

Memory is addressed with bits, so its natural steps are powers of two — and 210 lands so close to 1,000 that the prefix was simply borrowed. Every further rung widens the gap.

Which one you were handed depends on the trade. Memory comes in powers of two; disks and networks in powers of ten — a "1 TB" drive holds 1012 bytes and shows up as 931 GiB. Hertz is never base two, and neither is the kilo in kilometre.

Notice what just happened

One notation, two meanings, and nothing in the notation to tell you which you were handed. Keep hold of that. The rest of the lecture is about that gap.

One bit

A bit is one distinction: this, not that.

state space = 2

Add a bit and you don't add states. You double them.

Ten bits: a thousand states. Twenty: a million. Thirty: a billion.

Doubling is the most underestimated operation in human reasoning — and the entire engine of computing.

Scale check

34 bytes of memory has more states than the observable universe has atoms.

1080atoms in the observable universe
2266states of 266 bits ≈ 1.2 × 1080
37bytes to beat every photon, ~1089
log scale · 1 → 10⁸⁰
Your pocket

A phone with 8 GB of memory has 268,719,476,736 states.

Written out in decimal, that number has about 20.7 billion digits.

At 3,000 digits a page: 6.9 million pages. Bound into 500-page books: 13,800 volumes — some 400 metres of shelf.

That shelf does not hold the phone's states. It holds the number of them, written down once.

Every app you will ever install, every photo you will ever take, and every bug you will ever hit is one point in this space.

The consequence

In a space that big, good states and bad states look alike.

tested states, lit

Test a billion states per second, starting at the Big Bang, and by now you would have checked about 288 of them.

Out of 268,719,476,736. The fraction is not small. It is indistinguishable from zero.

Testing shows the presence, not the absence of bugs.Edsger W. Dijkstra, 1969
Reframe · 1 of 5

The vastness is not the problem. It is the inventory.

What it forbids

Certainty by inspection. You will never check your system exhaustively — not with more testers, not with faster machines, not ever.

What it opens

Every symphony not yet written, every protein not yet folded, every proof not yet found, every program not yet imagined — all of them are already in there, waiting to be addressed.

The same enormity that hides your bugs is the reason novelty is inexhaustible. You cannot have one without the other.

Interlude · literature

Borges built this space in 1941 and called it a library.

The Library of Babel holds every possible book: every truth, every refutation — and overwhelmingly, gibberish.

Its inhabitants go mad: containing every truth is worthless without a way of finding one.

Modern echo: a large language model already "contains" astonishing amounts of text. Containment was never the hard part.

Borges · 1941

410 pages, 40 lines, 80 characters, 25 symbols. The shelves hold 251,312,000 books.

≈ 101,834,097 volumes
The shift

From storage to search: from having the answer somewhere to knowing it when you see it.

Part II

Infinity

Vast is still finite. Now we count things that aren't — and discover that some infinities are bigger than others.

Counting without numbers

Two sets are the same size if you can pair them up.

No counting needed — just a perfect pairing. Every left has exactly one right, and nothing is left over.

So there are as many even numbers as numbers: pair n with 2n. The part is as big as the whole.

A set pairable with 1, 2, 3, … is called countable. It can be listed.

n ↔ 2n
Notation

Everything we can write down is countable.

A program is a finite string of symbols. So is a proof, a specification, a sentence, a score, a formula, a prompt.

List all strings of length 1, then length 2, then 3 … every finite text appears at some finite position.

So: all programs that will ever exist form a countable list. Same for all proofs. Same for all sentences of English.

the enumeration of all texts
This has a name

Assigning a number to each piece of syntax is Gödelisierung — Gödel numbering. It is how a machine reads a machine, and we return to it in Part III.

Meaning

What we want to talk about is not.

Consider all infinite sequences of bits: 0110100011… forever.

Each one is a complete answer to an endless list of yes/no questions — a behaviour, a function, a real number, a fate.

Claim: no list can contain them all. Not a long list. Not an infinite list. No list.

Cantor's proof is three lines and it is the most consequential argument of the 20th century.

the diagonal
The objects

A subset of the numbers is one yes-or-no answer per number.

Take any collection of counting numbers — the evens, the primes, just {3}, none of them, all of them. Each one is a subset.

To pin one down, walk 1, 2, 3, … and answer in or out, forever. That answer sheet is an infinite bit string, and every infinite bit string is one.

All of them together make the power set, written 2 — two choices, made ℕ times.

A handful have names. Almost all are an endless coin-flip, with nothing shorter to say about them than the flips.

subsets as answer sheets · 1 = in
The proof · 1 of 2 · subsets

Hand me a list of every subset. I will build one that is not on it.

● in · ○ out

Suppose the subsets could be listed: S1, S2, S3, …, every single one of them somewhere on the list.

Go down the diagonal and ask each row about its own number. Is 1 in S1? Is 2 in S2? Is n in Sn?

Now build D by answering the opposite every time: D = { n : n is not in Sn }. Nothing exotic — a rule anyone can apply, one number at a time.

D is not S1: they disagree about 1. Not S2: they disagree about 2. Not Sn for any n whatsoever. So the list left something out — and it was any list at all.

The proof · 2 of 2 · numbers

The same move, on the numbers between 0 and 1.

Suppose you could list them: r1, r2, r3, …, each written out as an endless decimal.

Take the first digit of the first, the second digit of the second, the n-th of the n-th. The diagonal again.

Build a new number x by changing every one of those digits. Then x differs from r1 in the first place, from r2 in the second, from rn in the n-th — so x is on no row, and the list was not a list of all of them.

The one place it needs care

Change each digit to 5, or to 4 if it was already 5. That keeps you clear of the trailing nines: 0.4999… and 0.5000… are the same number written twice, and a proof that only landed on that would have proved nothing.

any list · and the number it missed
Intuition

Between any two numbers, however close, there is another.

halve, and halve again

Pick two reals as close together as you like. Their midpoint lies strictly between them. Now take those two — and do it again, forever.

There is no next real number: nothing to step from and nothing to step to. A list is nothing but firsts and nexts, which is the wrong shape for a line with no grain.

But do not let it do the work

Density is the feeling, not the reason. The fractions are dense in the same way — between any two there is another — and they can still be listed, by walking a grid of numerators and denominators corner by corner. Only the diagonal tells the two cases apart.

Theorem

The diagonal, once and for all.

Cantor · 1891
For every set S, the set of all subsets of S is strictly larger than S. In particular, the real numbers cannot be listed.
Where self-reference enters

The proof builds an object from the list that asks of each row: "do you contain yourself here?" — and then answers the opposite. The list is used against itself.

You have just seen it twice

Once on subsets, once on decimals — and it was the same proof both times, because a subset is an infinite bit string and so is a decimal expansion. Power set, real numbers, bit sequences: three costumes, one cardinality. The theorem says it for every S at once.

Russell would turn the same trick on set theory itself in 1901: the set of all sets that do not contain themselves. Same diagonal, aimed at the foundations.

The result everything hangs on

There are incomparably more meanings than notations.

countable notation · uncountable meaning

Programs: countable. Behaviours: uncountable. So almost every behaviour has no program.

Proofs: countable. Truths about numbers: uncountable. So almost every truth has no proof.

Sentences: countable. Distinctions the world admits: uncountable. So almost everything is unsaid.

"Almost every" here is exact: the expressible is a vanishing sliver, of measure zero, inside the meaningful.

Reframe · 2 of 5

Scarcity of notation is a job description.

What it forbids

A final language. No vocabulary — mathematical, legal, musical, or neural — will ever cover the space of meanings.

What it opens

An inexhaustible supply of things worth naming. Every new notation captures meaning that was previously unreachable — and there is always more left.

Calculus, double-entry bookkeeping, staff notation, the periodic table, chemical formulae, DNA sequencing, type systems: each was a raid on the uncountable. The supply of raids never runs out.

Part III

Self-Reference

Counting told us the gap exists. Self-reference walks us to the edge of it and points.

Syntax and semantics

The score is not the music.

Syntax · notation

Marks on paper. Finite, discrete, checkable, copyable. 1 + 1, a staff of crotchets, H₂O, a line of code, this sentence.

Semantics · meaning

What the marks are about. The number two. Sound in a room. A molecule that dissolves salt. A machine's behaviour over all inputs.

The map

A semantics is a function from notation to meaning. Defining that function precisely is the founding act of every exact discipline.

Frege 1892 · Sinn / Bedeutung Saussure · signifier / signified Korzybski 1931 · the map is not the territory Magritte 1929 · ceci n'est pas une pipe Peirce · sign / object / interpretant
Two kinds of language

Natural language is compression. Formal language is commitment.

"I saw the man with the telescope." Two readings, one string. English resolves it with a shared world you and I already have.

That shared world is why English is powerful — and why it cannot be a foundation. Its semantics is us, and we differ.

A formal language pays a price — narrowness, pedantry, effort — to buy one thing: a meaning that does not depend on who is reading.

naturalformal
ambiguoussingle-valued
elastic, forgivingbrittle, exact
persuadesproves
needs a mindneeds a machine
learned by livingdefined by decree

Prompting an AI in English is negotiation. Writing a test, a type, a schema, a unit, a contract is legislation.

Gödelisierung

Give every text a number, and notation becomes something you can compute with.

Gödel's device, 1931: encode formulas, proofs and programs as integers. Now arithmetic can talk about arithmetic.

Every computer you have ever used is this idea in metal: code is data. A compiler eats programs. An operating system runs programs. A model is trained on programs.

And once a system can describe systems, it can describe itself. Self-reference is not a trick; it is the price of being expressive enough to be useful.

Aside · for the computer scientists

In selfie, a 12KLOC C* system, a compiler compiles its own source and an emulator executes that code — including itself. Gödelisierung you can run in a terminal.

text → number → text
Theorem

There are truths with no proof.

Gödel · first incompleteness · 1931
Any consistent formal system rich enough to describe arithmetic contains statements that are true but not provable within it.
How

Build, by Gödelisierung, a sentence that says: "this sentence has no proof in this system." If it were provable, the system proves a falsehood. So it is unprovable — and therefore true.

The same diagonal

Cantor built a row not on the list. Gödel builds a truth not on the list of provable things. Adding it as a new axiom does not help: the construction simply runs again.

Proof is a finite object you can check. Truth is not.
Proof is syntax. Truth is semantics. They are not the same size.

Theorem

And no strong system can certify itself.

Gödel · second incompleteness · 1931
Such a system cannot prove its own consistency — unless it is inconsistent, in which case it proves everything.
Read it as engineering

A trustworthy system cannot be the source of its own trust. Confidence has to come from outside: a stronger theory, an experiment, an independent auditor, reality.

Read it as advice

When a model explains why its own answer is correct, you have received more output from the same system — fluent, plausible, and not a certificate. Introspection is not audit.

Tarski, 1936, closes the circle: no sufficiently expressive language can define truth for its own sentences. Truth always lives one level up.

Undecidability

Will this program ever stop? No program can always tell.

feed the decider its own description

Suppose H(P,x) decides, for every program and input, whether P halts on x.

Build D(P): ask H whether P halts on P — then do the opposite.

Now run D(D). It halts exactly if it doesn't. Contradiction. So H never existed.

Turing, 1936 — the same paper that defined the universal machine, and thus invented the computer. The limit and the machine arrived together.

The same move, forwards

One machine that can be any machine.

the job moved from the wiring to the tape

A machine used to be its job — to sort instead of add you built a different machine. Turing's move: put the machine's description on the tape, as data.

Then one machine U reads any description and does whatever that machine would do. That is universality: one piece of hardware, every possible behaviour, because the behaviour arrives as notation.

Your phone is not phone-shaped. It is U holding a description — which is why a new app needs no new hardware.

One sentence, read twice

Hand a decider its own description: contradiction, no H. Hand a machine any description: every program at once, one U. Turing published both in 1936.

The general case

It is not just halting. It is every interesting question about meaning.

Rice · 1953
Every non-trivial property of the behaviour of programs is undecidable. Only questions about their text are safe.
Decidable · about syntax

How long is the code? Does it parse? Does it use this library? Are the types consistent? Does it contain a loop?

Undecidable · about semantics

Is it correct? Is it equivalent to that one? Does it ever leak the key? Is it free of infinite loops? Is it safe?

So every practical tool — a type checker, a test suite, a linter, a model checker, a fuzzer, a proof assistant — is a deliberate approximation: sound but incomplete, or complete but unsound, or exact only within a bound. Choosing which to give up is the discipline.

Interlude · law

No text contains its own application.

Hart's core, and Hart's penumbra

A statute says no vehicles in the park. A car, plainly. And then the arguing starts.

Hart called the easy cases the core, the arguable ones the penumbra. No redrafting removes it — only moves it. The words are finite; the situations are not.

So law stops trying to settle meaning in the text and builds an institution instead: courts, appeals, precedent. Not a workaround — the only available design.

Gödel's other theorem

At his 1947 citizenship hearing, Einstein beside him, Gödel announced a self-referential flaw by which the Constitution could be legally turned into a dictatorship. The judge steered him off it, and nobody recorded which flaw he meant.

Pattern

One idea. Sixty years. Six theorems.

yearwhothe listthe diagonal object
1891Cantorall real numbersa number on no row
1901Russellall setsthe set of non-self-members
1931Gödelall provable sentences"I am unprovable"
1936Tarskiall definable predicates"I am false"
1936Turingall decidable questionsa program that defies its judge
1953Riceall semantic propertiesall of the above, at once

Assume a complete list. Ask each item about itself. Answer the opposite.
Learn the move once and you own the century.

Reframe · 3 of 5

No final authority means no ceiling.

What it forbids

A machine, a method, or a person that settles all questions. No complete rulebook, no self-certifying system, no automatic correctness.

What it opens

Mathematics is not a finished building but an open frontier; engineering is a craft rather than a lookup; and judgement — yours — never becomes redundant.

Beware of bugs in the above code; I have only proved it correct, not tried it. Donald E. Knuth, 1977
Part IV

Cost

Suppose a question is decidable. You still have to pay for the answer — in time, space, and energy.

Decidable ≠ doable

A question with a guaranteed answer you will never receive.

Given a logical formula over 100 yes/no variables: is there an assignment making it true? Perfectly decidable — try all 2100.

2100≈ 1.3 × 1030 assignments
1013 yearsat a billion checks per second

Chess has ~1044 legal positions; Go, ~10170. Nobody "solves" these. We navigate them — with heuristics, structure, and luck.

2ⁿ branches · one satisfying leaf
Intractability

Hard to find. Easy to check.

That asymmetry has a name: NP — problems whose solutions are quick to verify even if finding them seems to need exponential search.

Cook and Levin, 1971–73: thousands of such problems are the same problem in disguise. Crack one efficiently and you crack scheduling, routing, folding, packing, proving.

Whether that is possible — P = NP? — is the most consequential open question in the exact sciences. Most researchers bet no.

The asymmetry, everywhere

Composing a symphony versus hearing it is off. Writing a proof versus reading it. Designing a protein versus assaying it. Doing the homework versus grading it.

Remember this one

Generation is expensive; verification is cheap. Every healthy division of labour — and every safe way to use an AI — is built on that gap.

Physics

Computation is not abstract. It costs energy, by law.

joules · log scale

Landauer, 1961: erasing one bit at room temperature costs at least kT ln 2 ≈ 3 × 10-21 joules. Information is physical.

So brute-forcing our 266-bit space costs ≈ 1059 J — about 1015 times everything the Sun will radiate in its entire lifetime.

Meanwhile a human brain runs on 20 watts: a dim light bulb, doing what data centres cannot.

Time, space, energy — three currencies for one budget. Any claim about intelligence that ignores the budget is a claim about magic.

Reframe · 4 of 5

Hardness is load-bearing.

What it forbids

Brute force as a strategy. You cannot search your way to correctness, to a cure, or to a business plan.

What it opens

Every private message, every signature, every payment, every password you rely on today exists because some problems are hard. Difficulty is the raw material of security — and the reason abstraction, theory, and taste are worth more than compute.

If exhaustive search worked, there would be no cryptography, no privacy, and — since anything findable would be found — nothing left to discover.

Metrics

A language without a metric is decoration.

Notation lets you say it. Semantics fixes what it means. A metric tells you whether you are getting closer.

Running time, memory, energy, error rate, coverage, p-values, yield, latency, mortality, loss on a held-out set — every mature field runs on invented measures.

And every metric decays the moment it becomes a target.

When a measure becomes a target, it ceases to be a good measure. Goodhart's law · 1975

Teaching to the test. Optimising engagement. Publishing to the h-index. Training to the benchmark. In 1989 the CAST trial found drugs that suppressed the arrhythmia and raised the death rate. Same failure, five fields. A metric is a semantics for "better" — always approximate, and sometimes fatally so.

Part V

Everywhere

None of this is about computers. Computers are just where it was noticed first, because there the notation is forced to be exact.

Biology

Life was running a formal language long before we invented one.

Syntax: 3.2 billion base pairs, two bits each — about 800 megabytes. Smaller than a phone's memory; state space 43,200,000,000.

Semantics: the genetic code maps 64 codons onto 20 amino acids and a stop — a redundant, many-to-one interpretation function, executed by the ribosome.

Self-reference: the genome encodes the machinery that reads the genome. Von Neumann described this architecture in 1948 — five years before Watson and Crick.

Intractability · Levinthal 1969

A modest protein has ~10300 possible shapes. Sampling them all would outlast the universe; real proteins fold in milliseconds. Nature does not search — it funnels.

And where AI actually won

AlphaFold did not solve folding by exhaustion either. It learned an approximation, and its value was settled by a metric the field had already invented and by experiments it could not fake.

Biology · self-reference

A copier that never interprets, and an interpreter that prints the copier and itself!

von Neumann's architecture, as run by a cell

Copy. Polymerase walks the strand and duplicates it letter by letter, never asking what a letter means. Notation to notation.

Interpret. The ribosome walks the same strand and executes it — an interpreter in the sense a 3D printer is one: description in, object out. Notation to meaning.

Close it. Among the objects it prints are the polymerase and the ribosome — and one ribosome prints any protein, because which protein is data on the strand. That is U from Part III: universality, in chemistry.

Mathematics · physics · chemistry

The postulate nobody could prove was a door, not a wall.

one line, one point beside it · how many parallels?

Euclid's fifth: through a point beside a line there is exactly one parallel. For two thousand years everybody failed to prove it.

In the 1820s Bolyai and Lobachevsky changed the postulate instead. Many parallels, or none — nothing breaks.

Unprovable because independent: a choice, not a fact. Gödel's shape, a century early — and the way out was a new language, not more effort.

And then the notation knew first

Riemann built curved-space geometry in 1854 for nothing in particular; Einstein could not have written relativity without it. Mendeleev left holes in his table and named elements nobody had seen. Dirac's equation had a solution nobody wanted; the positron turned up in 1932.

Mind

You navigate an unimaginable space with about four slots of working memory.

Space · Miller 1956

Seven items, plus or minus two; later work says nearer four chunks. Chunking is the mind inventing notation to beat its own memory limit.

86 billion neurons · ~10¹⁴ synapses
Time · Kahneman

Fast heuristics that are cheap and sometimes wrong; slow deliberation that is expensive and sometimes right. A biological answer to a cost problem.

System 1 / System 2 · 2011
Language · Piaget, Harnad

Learning is accommodation: when the world breaks your scheme, you build a new one. And symbols only mean something if they are grounded in experience.

symbol grounding · 1990

Growing up is inventing better notation for a world too big to store — exactly the task of a discipline, run on one person.

The pattern

Every leap in every field was a new notation.

fieldnotation inventedwhat became possible
accountingdouble-entry ledger · Pacioli 1494the firm, the audit, capitalism
musicstaff notation · Guido d'Arezzo c.1025polyphony, composition at scale
paintinglinear perspective · Alberti 1435depth, and a space to compose in
physicscalculus · Newton & Leibniz 1670smotion, orbits, engineering
chemistryformulae & the periodic table · 1869prediction of unknown elements
medicinediagnostic criteria, trial protocolsevidence instead of authority
linguisticsgenerative grammar · Chomsky 1957language as a formal object
computingTuring machines · 1936all of the above, mechanised

None of these was a discovery of new facts. Each was a new language — or a new property an old language could finally express. The facts followed.

Part VI

Machines

Now the question everyone actually came with. Today's AI is remarkable — and it is subject to every single thing we have just established.

What it is

A large language model is a machine trained on notation, asked for meaning.

tokens in · a distribution over the next one out · then a sample

The name is exact. What a large language model — an LLM — is trained on is language, so the signal is syntactic: which symbol follows which. Meaning is never handed to it.

That this works as well as it does is the genuine surprise of the decade. It cost megawatts for weeks; you run on twenty watts. Neither figure changes what kind of object it is.

Billions of parameters, each many bits. Part I applies unchanged: that space cannot be inspected, so the behaviour cannot be enumerated. Only sampled.

The old gap, industrialised

A hallucination is a proof-shaped object that isn't true.

three tests · and they are not the same test

Fluency is syntax. Correctness is semantics. We built a machine of extraordinary fluency, so the gap between the two is something you now meet before breakfast.

Not a defect to be patched away: it is the proof/truth distinction at consumer scale.

So the durable response is not "trust it more" or "trust it less" but check it against something with a semantics — a compiler, a test, an experiment, a source, a colleague.

Grounding again

Harnad, 1990: symbols defined only by other symbols never touch the world. Tools and feedback plug that hole partially, never completely.

Where it is going

World models aim at meaning itself — and inherit every limit anyway.

two routes to the same world

Not more text but a model of the world — state, dynamics, consequence. Predict what happens, not what is said: an attempt at real semantics.

And a bid for efficiency — fewer examples, fewer joules. On a fixed budget efficiency is capability, so world models may well outperform LLMs outright.

And still a finite notation, sitting inside the world it models. Parts II–IV apply unchanged.

What it inherits

Counting — models are countable, behaviours are not.
Rice — "is this model right?" is semantic. Undecidable.
Gödel — no self-certificate from inside.
Cost — the state space did not shrink. Only the map did.

Self-reference · again

The loops are already closed.

systems judging, training, writing systems
  • Models are trained on text that models wrote — with measurable degradation when the loop tightens (model collapse, 2024).
  • Models grade models. The judge and the candidate share the same blind spots.
  • Models write the code that trains models, and agents invoke themselves.
  • "Is this system safe?" is a semantic property of a program. Rice says: no general decision procedure. Not hard — impossible.
  • And by Gödel's second theorem, no system this expressive can certify itself. An explanation of its own output is more output.
What actually changed

Generation got cheap. Verification did not.

Before

Producing a draft, a proof sketch, a program, an image, a translation was expensive and therefore scarce. Scarcity did our filtering for us.

Now

Production is nearly free and unbounded in volume. The filter has to be supplied deliberately — by specification, by measurement, by review.

Recall the asymmetry from Part IV: hard to find, easy to check.
That is a description of a healthy relationship with a machine.

Value migrates to the two ends AI does not occupy: deciding what should be true (specification) and establishing that it is (verification). Both are acts of meaning. Neither is automated by better generation — and the better generation gets, the more they are worth.

Part VII

Practice

A method that does not care which model is current, which company leads, or what happens next year.

The question, replaced

Stop asking whether it is intelligent.

Undecidable · unproductive

Is it intelligent? Does it truly understand? Is it conscious? Will it always behave? Each asks for a semantic verdict on a system, in a language with no semantics for the words used.

Decidable · today

What language did I state my requirement in? What are its semantics? What is my metric, and how will it be gamed? What is the budget? Who checks the result, from outside?

The right question is never about the machine's essence.
It is about your language, your semantics, your metric, your check.

This is why the method is future-proof: it never mentions a model, a vendor, or a year. The same five questions worked for the steam engine and the spreadsheet, and they will work for whatever arrives after transformers.

Take this with you

Six habits with no expiry date.

  1. Name the language. State what you want in something with a semantics — a type, a schema, a unit, a test, an acceptance criterion. English negotiates; formality commits.
  2. Attach a metric, and name its failure. If you cannot say what "better" means, you are wandering. Then write down how the metric will be gamed — it will be.
  3. Delegate generation. Own verification. Accept work you could not have produced; never work you cannot check. That is the only shape delegation has ever had.
  4. Budget time, space, and energy. Know how big your space is and how much of it you can afford to visit. Most doomed projects were doomed arithmetically on day one.
  5. Break every loop from outside. Nothing certifies itself — not a model, not a company, not you. Import an independent check: an experiment, a proof, a person allowed to say no.
  6. Trust the unproven, then formalise it. When answers get hard to check, what is missing is a language, not more effort. Take the truth you can see but not yet prove — and build the notation that proves it.
Precedent

No theorem says your bridge will hold. You crossed one this morning.

Aviation, medicine, and civil engineering never had decidability either. Their questions are semantic, their state spaces astronomical, their proofs unavailable.

They became extraordinarily safe anyway — with layered approximations: standards, redundancy, staged testing, incident reporting, licensing, liability, and the freedom to say not yet.

None of it is proof. All of it is accumulated, measured approximation. It worked.

Therefore

"We cannot prove it safe" was never a reason for paralysis and never a licence for recklessness. It is the normal condition of every mature engineering discipline — and the reason those disciplines built institutions instead of theorems.

AI is early in exactly that process. Being early is a job opening, whatever you are studying.

Back to the short answer

Every notation was invented by someone who saw a truth before it could be proved.

Fermat asserted it; Wiles proved it 358 years later. Riemann's hypothesis has been assumed true, and built upon, for 165 years. Mendeleev left blank cells for unseen elements. Darwin had no genetics.

In each case the notation came after the conviction — and the conviction was not deduced from anything.

Gödel makes this precise: no mechanical procedure yields the next axiom. Deciding what to hold true is not a step inside the system. It is a step to a new one.

Therefore, symmetrically

This step is not automatable — not because machines are weak, but because there is no algorithm there to automate. Scale does not help; it was never a compute problem.

The challenge for all of us

Machines are already formidable at proving within a language. The open frontier — for every student here and for every AI ever built — is finding promising unproven truth, and then building the language or the property that captures it.

The definition

So — what is intelligence?

Working definition
Intelligence is developing new formal languages — or at least new properties in existing ones — which requires finding and understanding promising unproven truth. New languages and properties let us ask new questions about that truth, and then answer them in proofs. Forever.
Asking the right questions

Choosing which unproven truth is worth formalising. Not deducible, not teachable by rote — this is what advanced, graduate-level study is for.

Developing the skills to answer

Proving, computing, measuring, building. Rigorous, cumulative, teachable — this is what an undergraduate programme gives you.

Not a threshold, a score, or a possession — an activity with a direction and no terminating condition. And notice where the difficulty sits: not in the proving, which machines do superbly, but in the finding, which nothing yet does reliably.

Why depth still matters

This is exactly why you study a field in depth.

Not because AI is unavailable — it is extravagantly available, and it will get better every year of your career.

But finding and understanding promising unproven truth requires having lived inside a subject long enough to feel where it is thin, where it is wrong, and where it is about to give.

No summary, no search, and no generated answer transfers that. It is built, slowly, and it is yours.

And it need not be computer science

Every field has unproven truth in front of it — history, medicine, ecology, music theory, economics, law. The method in this talk is field-independent; the depth has to be specific.

Pick the field — or fields — whose unsolved problems still make you strangely comfortable.

Reframe · 5 of 5

Truth can only be approximated. The approximating never stops.

What we lost

Completeness. Certainty. A final theory, a final language, a decision procedure for meaning, and any machine that could hand us all of it.

What we have instead

Work that cannot be finished, and therefore cannot be taken away: an unbounded frontier where every new notation makes yesterday's unreachable truths reachable, on every single day, forever.

The halting problem, applied to progress, gives the most hopeful result in science:
this computation does not terminate.

Sources · and where to go next

Further reading.

The theorems

Cantor 1891 · Russell 1901 · Gödel 1931 · Tarski 1936 · Turing 1936 · Rice 1953 · Cook 1971 · Karp 1972 · Levin 1973

Physics of computing

Shannon 1948 · Landauer 1961 · Bremermann 1962 · von Neumann, Theory of Self-Reproducing Automata 1966

Other fields

Frege 1892 · Wittgenstein 1921 / 1953 · Korzybski 1931 · Miller 1956 · Chomsky 1957 · Wigner 1960 · Levinthal 1969 · Goodhart 1975 · Harnad 1990 · Kahneman 2011 · Jumper et al. 2021

Read for pleasure

Borges, The Library of Babel 1941 · Hofstadter, Gödel, Escher, Bach 1979 · Dijkstra, EWD notes · Chaitin on randomness

Hands on

Kirsch, Elementary Computer Science: From Bits and Bytes to the Universality of Computing — and the selfie system it is built on.

github.com/cksystemsteaching/selfie
Keys

→ ← steps · ↑ ↓ slides · n notes · t clock · d theme

Close

Notation is finite.
Meaning is not.
Mind the gap — then go widen it.

You are not competing with a machine at producing notation. You are the part of the loop that decides what it should mean, and whether it is true.

Aside · for the computer scientists in the room

If you want to hold self-reference in your hands: selfie is a 12KLOC system in which a compiler compiles itself and an emulator executes itself — plus monster and rotor, which translate real machine code into logical formulae, so you can watch syntax become semantics and then run straight into NP-completeness on your own laptop. github.com/cksystemsteaching/selfie

Two slides left. One is a joke. One is the point.

Penultimate

Whatever intelligence is, humour is the only way.

For most of you, this hour has probably confirmed your conviction never to study computer science.

Or the exact opposite. And who knows — some of you may now be considering a switch into computer science. I have seen it happen. (With apologies to my colleagues in the other fields.)

Outcome A

Confirmed conviction.

Outcome B

Informed confusion.

Regardless of which one you leave with: this is about becoming who you are and no one else.

Denn es ist zuletzt doch nur der Geist, der jede Technik lebendig macht. Johann Wolfgang von Goethe

For in the end it is only the Geist that brings any technique to life.

Geist — spirit, mind, meaning. For the last hour we have been calling it truth. And it is, finally, you and me.
Technik — technique, technology, the formal machinery. Necessary. Never sufficient. Never alive on its own.

Speaker notes
Keys
→ · space
next step / slide
back
↓ · ↑
next / previous slide, skipping steps
home · end
first / last slide
n
speaker notes
t
start / pause the clock
r
reset the clock
d
toggle light / dark
? · esc
this panel