Skip to content
Multigrid

AGI, superintelligence, alignment & the long future

The arguments about where AI is going, stated at the strength their proponents state them — with the definitions pinned down, the evidence attributed, and the predictions labelled as predictions.

This is the part of the subject where confident writing is usually the least reliable. The questions are genuinely open, the people arguing about them are serious, and the two easiest pages to write — the one where it is all inevitable and the one where it is all hype — are both wrong in the same way: they report a live disagreement as a settled fact.

These pages do something narrower and, hopefully, more useful. They take each argument apart into its premises, say which premises are empirical and could be checked, which are predictions and rest on assumptions, and which are value judgements that no evidence resolves. Where a result exists, it is named and its scope is stated: what was demonstrated in a real system, what was proved under stated assumptions, and what is argued but not shown. Where two people using the same word mean different things — and in this cluster that is the single most common source of disagreement — the page says what each of them means.

You will not find a year for AGI here, or a probability, or a survey percentage. Those numbers exist and some of them are worth reading; what travels badly is the number without the method that produced it. What these pages try to give you instead is the machinery to evaluate the next such claim you encounter.

What Would Count as AGI? Definitions That Can Be Tested

Seven serious definitions of artificial general intelligence, graded on the only property that makes a definition useful in an argument: whether anything could settle it.

5 min read

The Technological Singularity, Explained

Where the idea came from, the four different things the word has been used to mean, and which of them make claims that could turn out to be false.

5 min read

Intelligence Explosion: The Argument and Its Weak Points

The argument set out as premises strong enough that its proponents would recognise it, followed by the specific objections to each premise.

5 min read

Recursive Self-Improvement: What It Would Require

Four different things the phrase is used for, and the concrete resources each would need — compute, data, evaluation signal and serial experiment time.

4 min read

AGI Timeline Predictions and Their Track Record

What a timeline prediction must contain before anyone could score it, the documented pattern in the historical record, and four checks to run on any forecast you are shown.

5 min read

Why Forecasting AI Progress Is So Hard

The specific structural reasons this domain defeats normal forecasting methods, and the errors it produces in both directions.

5 min read

Scaling Hypothesis vs Missing Ingredients

The two positions stated as their strongest proponents state them, with the specific evidence each side cites and the observations that would move each.

5 min read

The Data Wall: Are We Running Out of Text?

How an estimate of the world's usable text is actually constructed, which term in it carries the uncertainty, and what the proposed ways around the limit do and do not solve.

5 min read

Continual Learning: The Capability Models Still Lack

Why no deployed model incorporates what it saw yesterday, which part of that is a research problem and which part is a deliberate operational choice.

5 min read

World Models and Physical Understanding

Three different things the term means, an operational definition of what having one would require, and what the probing evidence has actually established.

5 min read

Embodiment: Does Intelligence Need a Body?

The embodied cognition argument stated properly, the difference between a necessity claim and a data claim, and what robotics results actually establish.

4 min read

Sample Efficiency: Humans vs Models

Why the human-versus-model data gap has no single number, five defensible ways to count it, and the research programme that fixes the comparison by construction.

5 min read

Transformative AI: Economic Definitions of the Threshold

A definition built from effects on the world rather than from capabilities, why that choice was deliberate, and the measurement problems it inherits.

4 min read

AI and Economic Growth: What Models Predict

The two families of economic model applied to AI, the specific parameters their disagreement lives in, and why two careful economists reach very different answers.

5 min read

Compute Trends: Is Exponential Growth Continuing?

Four different quantities that all get called the compute trend, how each is estimated, how wrong the estimates can be, and which constraints bind next.

5 min read

Algorithmic Efficiency: Progress Without More Compute

How researchers measure the compute needed to reach a fixed capability over time, what that measurement is sensitive to, and why the resulting halving times travel badly.

4 min read

The Alignment Problem, Explained Properly

Three technically distinct problems that the phrase covers — specification, robustness and assurance — with what is known about each and what is not.

5 min read

Instrumental Convergence and Why It Is Contested

The argument that most goals imply the same intermediate goals, the formal results that support a version of it, and four distinct critiques worth taking seriously.

5 min read

Specification Gaming: Real Examples From Real Systems

Documented cases in which systems satisfied their stated objective and violated the intent behind it, from evolutionary algorithms through game-playing agents to language model training.

5 min read

Inner vs Outer Alignment

The distinction between an objective that is wrong and a learned goal that differs from a correct objective, with the experiments that demonstrate the second.

5 min read

Mechanistic Interpretability: Reading a Model's Mind

What features, circuits and sparse autoencoders are, what has actually been reverse-engineered, and what the field still cannot do.

5 min read

Can We Detect Deception in a Model?

What counts as deception in a system with no beliefs in the ordinary sense, what has been demonstrated in the lab, and what current methods can verify.

4 min read

Constitutional AI and RLAIF

How a model is trained against a written set of principles instead of case-by-case human labels, and what that does and does not buy.

4 min read

Scalable Oversight: Supervising Systems Better Than You

Debate, iterated amplification and weak-to-strong generalisation: three proposals for evaluating work you cannot check yourself.

5 min read

Evaluations for Dangerous Capabilities

What a dangerous-capability evaluation measures, why elicitation effort is the hard part, and how threshold-linked release policies are structured.

4 min read

AI Control: Safety Without Assuming Alignment

The research programme that drops the assumption that a model is aligned and asks which safety properties survive one that is trying to subvert them.

4 min read

Corrigibility and the Off-Switch Problem

Why an agent optimising for a goal has reason not to be switched off, what the formal attempts to fix it ran into, and which parts of the problem are real today.

4 min read

The Orthogonality Thesis

The claim that intelligence and goals vary independently, what it is actually arguing against, and the strongest objections to it.

4 min read

P(doom): What the Number Means and Why It Varies

Why two people can give wildly different p(doom) figures while agreeing about the world, and how to read the surveys that get quoted.

5 min read

AI Risk Taxonomy: Misuse, Accident, Structural

Three classes of AI risk that have different causes, different evidence and — most importantly — different people who can do anything about them.

4 min read

Gradual Disempowerment vs Sudden Takeover

The quiet scenario in which humans lose influence without any system seizing anything, compared with the dramatic one that gets the coverage.

4 min read

AI Optimism: The Best Case, Stated Seriously

Four distinct optimistic positions, the arguments each actually rests on, and what would count as evidence against them.

5 min read

Post-AGI Economics: What Happens to Wages

The mechanism that sets a wage, why comparative advantage does not guarantee a liveable one, and where economists actually disagree.

4 min read

Universal Basic Income and Automation

What the cash-transfer trials were built to measure, why almost none of it transfers to the automation question, and what the financing identity actually says.

5 min read

AI Consciousness: Why the Question Is Hard

Four different questions hide behind one word, behavioural evidence cannot settle any of them, and the theory-driven approach is the only method with traction.

5 min read

Moral Status of AI Systems

What could ground moral status, why the question cannot simply be deferred until consciousness is settled, and the two symmetric ways of getting it wrong.

5 min read

Whole-Brain Emulation and Other Paths to AGI

The copy-the-brain route and the other non-LLM programmes, each with the specific capability that is currently missing.

4 min read

The Great Filter, Fermi and AI

Whether AI could be a Great Filter, why the obvious version of that argument does not work, and which assumptions the whole discussion rests on.

5 min read

Reading List: The Papers Behind the AGI Debate

An annotated route through the primary sources, organised so each entry says what it established and what it does not.

6 min read

A Timeline of AI: From Dartmouth to Now

The turning points, each named by the primary document behind it, and the recurring pattern the winters share.

5 min read

Other topics

AGI, superintelligence, alignment & the long future · Multigrid