Contra the Moratorium


Trevor Chow


May 20, 2023

There is currently a protest outside the OpenAI headquarters, calling for a pause in the development of powerful AI (where powerful is usually proxied by size). I’ve said before that I am not a fan of this approach, and I want to explain why.

To be clear, I’m not unsympathetic to the concerns around the potential risks associated with these systems, but I think those who are arguing for such a moratorium either have substantially higher p(doom)s (and vastly different stories of how doom arises) than I do, or haven’t thought through their theory of change very carefully.

A useful piece of framing to start with is to note that an indefinite pause is simply not going to happen. The economic and political benefits of transformative artificial intelligence are far too vast for improvements in these models to be put off forever, and algorithmic improvements will lower the barrier to entry over time. This is not like the Nuclear Non-Proliferation Treaty or Asilomar, and if anything, the failure to properly regulate gain-of-function research in spite of its limited commercial value is a worrying base rate for this approach!

Given that, the question is: how we can affect the inputs to the AI production function in a way that minimises the total amount of catastrophic risk it poses across time?

Historically, we’ve seen that models have improved based on more compute and better algorithms. A moratorium right now means that models which are larger than say GPT-4 don’t get developed. Clearly, the immediate marginal effect is to reduce the catastrophic risks from AI systems, since the compute input factor doesn’t increase. I don’t think there’s much disagreement here, and if you think we are one step away from AI systems which can wreck havok on the entire world, then this is a perfectly reasonable position to take.

I do not think this.

I think we are still quite a while away from that point, and so most of the impact this policy would have is not in the here and the now, but instead in how it affects the path of AI development going forward.

Firstly, it’s not clear if a moratorium could even be enforced, and especially internationally, given we don’t yet have a good monitoring regime. Thus there are very strong incentives for evasion and defection, especially by other state actors.

Secondly, a pause would allow AI labs which are not at the frontier to have time to catch up. This would mean that after the pause is lifted, the top labs would likely face more competitive pressures and be more inclined to cut corners in the future.

Thirdly, stopping larger models from being built doesn’t stop all capabilities research. Instead, there is likely to be a compensating effort to focus on algorithmic improvements, which may not require the largest models to learn about. (Slightly unsure here since one of the plausible interpretations of the Bitter Lesson is that scaling up compute unlocks new algorithms and architectures.) Furthermore, hardware progress is unlikely to stop. Thus after the pause is lifted, you will suddenly be able to throw enormous amounts of compute with better hardware at the improved algorithms. Discontinuous improvements in AI systems are likely riskier than continuous ones due to the unpredictability of the resulting capabilities.

Fourthly, even if the research which occurs during the pause is focused on alignment rather than capabilities, the value of alignment research is probably highest when we are closest to the dangerous models.

Thus it seems that most of the value of the pause is in precedent-setting or consensus-building. Unfortunately, it is likely to do neither. Labs which were already sympathetic to these catastrophic risks probably would be willing to abide by some norms around safety regardless, but what this has done is instead pit safety against progress in a way that almost certainly alienates relevant stakeholders e.g. national security officials, less sympathetic labs, VCs etc. Recall, you don’t shift the Overton window by shouting crazy ideas in the hope that this makes the moderate ones look more reasonable!

Some alternative governance ideas which I’m more into:

Of course, pauses are part of this package, but these are deployment pauses conditional on various triggers being met and which have a clear and pre-defined endpoint. (This is driven in part by the fact that my models of AI risk are focused more on systems misbehaving after they are deployed rather than some lab leak-type scenario.)

PS: On a more aesthetic note, I have an instinctive aversion to calls for moratoriums because they tend to come from a view of policy which ignores how much the implementation details matter and because they underestimate how the endogenous societal response as soon as prosaic risks from AI start appearing is overwhelmingly far too conservative.