With my summer at Optiver finally over, Iâ€™ve started catching on some of the reading Iâ€™ve been meaning to do. One of the many items on my reading list is Ajeya Cotraâ€™s draft report on Forecasting Transformative AI with Biological Anchors. Although Iâ€™ve skimmed this in the past, I have not yet tried to sit down and distill/critique it in complete depth.

FLOPS

- Measure hardware power (work per unit time) in floating point operations per seconds
- Measure total computation in floating point operations
- One FLOP is one addition, subtraction, multiplication or division of two decimnal numbers

Framework

- Probability distribution over computation required (in FLOPs) to train TAI using 2020 ML architectures/algorithms
- The likelihood of this being affordable will fall over time: algorithmic progress reduces computation required, the amount of computation per dollar is likely to fall and the amount of money an AI project is willing to spend is likely to increase over time
- Thus calculate the probability distribution for future years by updating the 2020 distribution with algorithmic progress lowering the weight on high compute scenarios
- Then calculate the total computation available for training by multiplying the computation per dollar that year with the dollars a project is willing to spend in that year
- The probability of TAI that year is the probability the total computation available exceeds the computation required

TAI

- A model that can perform a large majority of economically valuable jobs more cheaply than human workers can

Biological anchors

- Human brains are the existence proof for TAI
- Computation required to train a human brain is a good proxy for the computation required to train a TAI
- Four anchors: total computation done over evolution, total computation done over a human lifetime, the computational power of the human brain and the amount of information in the human genome
- Produce probability distributions assuming each of these four anchors were entirely correct
- Update against low levels of FLOP given we donâ€™t have TAI right now
- Combine the probability distributions by weighting each of them

Brain computation hypothesis

- Bioanchors rely on an estimate of â€śthe amount of computation performed by the human brainâ€ť in FLOP/s
- Brain FLOP/s are defined as â€śhow many floating point units which perform 1 FLOP per second would you need per neuron to get to human-level intelligence over a similar evolutionary timescale if you replaced neurons with them when neurons first emerged?â€ť
- Largely based on Joe Carlsmithâ€™s report on Brain Computation
- Leads to two biological approaches: anchoring to the total FLOP done over the course of a natural training process or to the size of a transformative model

Four different anchors

- Evolution: the amount of FLOP done over the course of evolution
- Lifetime: the amount of FLOP done over the course of a childâ€™s brain growing to be an adult
- Neural network: the amount of computation required to train a neural network which does the same number of FLOP per subjective second and is the size of the human brain
- Genome: the amount of computation required to train a neural network which does the same number of FLOP per subjective second and is the size of the number of bytes in the human genome

This is the high level. Stopped taking notes here and decided to read through the rest before starting to decompose the argument.