Jeffrey Heninger, 22 November 2022

I was recently reading the results of a survey asking climate experts about their opinions on geoengineering. The results surprised me: “We find that respondents who expect severe global climate change damages and who have little confidence in current mitigation efforts are more opposed to geoengineering than respondents who are less pessimistic about global damages and mitigation efforts.”¹ This seems backwards. Shouldn’t people who think that climate change will be bad and that our current efforts are insufficient be more willing to discuss and research other strategies, including intentionally cooling the planet?

I do not know what they are thinking, but I can make a guess that would explain the result: people are responding using a ‘general factor of doom’ instead of considering the questions independently. Each climate expert has a p(Doom) for climate change, or perhaps a more vague feeling of doominess. Their stated beliefs on specific questions are mostly just expressions of their p(Doom).

If my guess is correct, then people first decide how doomy climate change is, and then they use this general factor of doom to answer the questions about severity, mitigation efforts, and geoengineering. I don’t know how people establish their doominess: it might be as a result of thinking about one specific question, or it might be based on whether they are more optimistic or pessimistic overall, or it might be something else. Once they have a general factor of doom, it determines how they respond to specific questions they subsequently encounter. I think that people should instead decide their answers to specific questions independently, combine them to form multiple plausible future pathways, and then use these to determine p(Doom). Using a model with more details is more difficult than using a general factor of doom, so it would not be surprising if few people did it.

To distinguish between these two possibilities, we could ask people a collection of specific questions that are all doom-related, but are not obviously connected to each other. For example:

1. How much would the Asian monsoon weaken with 1°C of warming?
2. How many people would be displaced by a 50 cm rise in sea levels?
3. How much carbon dioxide will the US emit in 2040?
4. How would vegetation growth be different if 2% of incoming sunlight were scattered by stratospheric aerosols?

If the answers to all of these questions were correlated, that would be evidence for people using a general factor of doom to answer these questions instead of using a more detailed model of the world.

I wonder if a similar phenomenon could be happening in AI Alignment research.²

We can construct a list of specific questions that are relevant to AI doom:

1. How long are the timelines until someone develops AGI?
2. How hard of a takeoff will we see after AGI is developed?
3. How fragile are good values? Are two similar ethical systems similarly good?
4. How hard is it for people to teach a value system to an AI?
5. How hard is it to make an AGI corrigible?
6. Should we expect simple alignment failures to occur before catastrophic alignment failures?
7. How likely is human extinction if we don’t find a solution to the Alignment Problem?
8. How hard is it to design a good governance mechanism for AI capabilities research?
9. How hard is it to implement and enforce a good governance mechanism for AI capabilities research?

I don’t have any good evidence for this, but my vague impression is that many people’s answers to these questions are correlated.

It would not be too surprising if some pairs of these questions ought to be correlated. Different people would likely disagree on which things ought to be correlated. For example, Paul Christiano seems to think that short timelines and fast takeoff speeds are anti-correlated.³ Someone else might categorize these questions as ‘AGI is simple’ vs. ‘Aligning things is hard’ and expect correlations within but not between these categories. People might also disagree on whether people and AGI will be similar (so aligning AGI and aligning governance are similarly hard) or very different (so teaching AGI good values is much harder than teaching people good values). With all of these various arguments, it would be surprising if beliefs across all of these questions were correlated. If they were, it would suggest that a general factor of doom is driving people’s beliefs.

There are several biases which seem to be related to the general factor of doom. The halo effect (or horns effect) is when a single good (or bad) belief about a person or brand causes someone to believe that that person or brand is good (or bad) in many other ways.⁴ The fallacy of mood affiliation is when someone’s response to an argument is based on how the argument impacts the mood surrounding the issue, instead of responding to the argument itself.⁵ The general factor of doom is a more specific bias, and feels less like an emotional response: People have detailed arguments describing why the future will be maximally or minimally doomy. The futures described are plausible, but considering how much disagreement there is, it would be surprising if only a few plausible futures are focused on and if these futures have similarly doomy predictions for many specific questions.⁶ I am also reminded of Beware Surprising and Suspicious Convergence, although it focuses more on beliefs that aren’t updated when someone’s worldview changes, instead of on beliefs within a worldview which are surprisingly correlated.⁷

The AI Impacts survey⁸ is probably not relevant to determining if AI safety researchers have a general factor of doom. The survey was of machine learning researchers, not AI safety researchers. I spot checked several random pairs of doom-related questions⁹ anyway, and they didn’t look correlated. I’m not sure whether to interpret this to mean that they are using multiple detailed models or that they don’t even have a simple model.

There is also this graph,¹⁰ which claims to be “wildly out-of-date and chock full of huge outrageous errors.” This graph seems to suggest some correlation between two different doom-related questions, and that the distribution is surprisingly bimodal. If we were to take this more seriously than we probably should, we could use it as evidence for a general factor of doom, and that most people’s p(Doom) is close to 0 or 1.¹¹ I do not think that this graph is particularly strong evidence even if it is accurate, but it does gesture in the same direction that I am pointing at.

It would be interesting to do an actual survey of AI safety researchers, with more than just two questions, to see how closely all of the responses are correlated with each other. It would also be interesting to see whether doominess in one field is correlated with doominess in other fields. I don’t know whether this survey would show evidence for a general factor of doom among AI safety researchers, but it seems plausible that it would.

I do not think that using a general factor of doom is the right way to approach doom-related questions. It introduces spurious correlations between specific questions that should not be associated with each other. It restricts which plausible future pathways we focus on. The future of AI, or climate change, or public health, or any other important problem, is likely to be surprisingly doomy in some ways and surprisingly tractable in others.¹² A general factor of doom is too simple to use as an accurate description of reality.

Notes

1. Dannenberg & Zitzelsberger. Climate experts’ views on geoengineering depend on their beliefs about climate change impacts. Nature Climate Change 9. (2019) p. 769-775. https://www.nature.com/articles/s41558-019-0564-z.
2. Public health also seems like a good place to check for a general factor of doom.
3. https://sideways-view.com/2018/02/24/takeoff-speeds/.
4. https://en.wikipedia.org/wiki/Halo_effect.
5. https://marginalrevolution.com/marginalrevolution/2011/03/the-fallacy-of-mood-affiliation.
6.  This suggests a possible topic for a vignette. Randomly choose 0 (doomy) or 1 (not doomy) for each of the nine questions I listed. Can you construct a plausible future for the development of AI using this combination?
7. https://www.lesswrong.com/posts/JvRb34CFF74hKa9jJ/beware-surprising-and-suspicious-convergence.
8. https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/.
9.  I looked at (1) the probability of the rate of technological progress being 10x greater after 2 years of HLAI vs the year when people thought that the probability of HLAI is 50%, (2) the probability of there being AI vastly smarter than humans within 2 years of HLAI vs the probability of having HLAI within the next 20 years, and (3) the probability of human extinction because we lose control of HLAI vs the probability of having HLAI within the next 10 years.
10. https://twitter.com/robbensinger/status/1541228487205236736/photo/1.
11. There is some discussion of why there is a correlation between these two particular questions here: https://twitter.com/infinirien/status/1537859930564218882. To actually see a general factor of doom, I would want to look at more than two questions.
12. During the pandemic, I was surprised to learn that it only took 2 days to design a good vaccine – and I was surprised at how willing public health agencies were to follow political considerations instead of probabilistic reasoning. My model of the world was not optimistic enough about vaccine development and not doomy enough about public health agencies. I expect that my models of other potential crises are in some ways too doomy and in other ways not doomy enough.

Jeffrey Heninger, 22 November 2022

AI Impacts is a research group with seven employees. From Oct 31 – Nov 3, we had a work retreat. We decided to try using Manifold Markets to help us plan social events in the evenings. Here are some notes from this experiment.

Structure of the Experiment

Katja created a group on Manifold Markets for AI Impacts, and an initial collection of markets. Anyone could add a market to this group, and five of us created at least one market. Each of us would rate each evening from 0 to 10 on an anonymous Google form. Most of the questions in the group were about the results of the form, often conditional on what activity we would do that evening. For example: “On the first day that at least 4 people begin a game of One Night Werewolf at the AI Impacts retreat, will the average evening rating be above 8?” The markets would resolve at some point the next morning after we had submitted our forms and Katja calculated the average evening rating.

Disagreements about the Experiment

There were several disagreements about how the experiment was supposed to be run. 

Initially, the role of the evening rating form was unclear. Was it asking for your honest assessment of the evening or was it part of the game? “What number would you like to assign to the evening?” is different from “How good was your evening honestly?” We decided that we wanted honest responses. Even then, the numbers were ambiguous. What constitutes a 7 evening vs. a 9 evening? Different people’s baselines result in different scores, which can alter the average. After the first evening, we had a better estimate of the baseline. Many of the markets had used an average score of above 8, which was higher than the baseline. This made the markets feel less useful, instead shifting the predictions to lower probabilities while remaining useful. It’s not clear why this happened, but it might have been because we didn’t want to bet against ourselves having a good time or because the tail of an unknown distribution is harder to predict than the middle of the distribution.

One morning, Katja told us the average score before resolving her markets. Zach used this information to bet on these markets. Rick thought that it was unclear whether this should be allowed, because not everyone was there and because the previous discussion about honest ratings suggested that we should ask before doing something that might give an advantage independent of prediction ability. We decided that this would not be allowed in the future, and that we would not tell each other the results of the markets before resolving them.

Unrealized Potential Problems

We thought of several other potential problems that did not end up being an issue.

One potential concern was that the interplay between the dynamics of the market and social events might make the socialization worse. Someone who had bet against having a good evening might have less reason to want the evening to be enjoyable to himself and others. If people spent time during the evening thinking about and frequently betting on the markets, it might disrupt the ongoing activities. In practice, while people did bet on the markets in the evening, it did not disrupt the other activities.

We had several other ideas for how to mess up the markets: filling out the anonymous form multiple times, colluding or bribing people to alter their scores, publicly filling out your form before the evening begins to manipulate the market, and purposely trying to thwart other people’s clever strategies. None of us tried doing any of these, but they might become relevant if the stakes were higher. There is also the concern that conditional and counterfactual predictions are not the same: For decision making, we would like to compare various counterfactuals, but it’s easier to make markets which are conditional on us doing something. If we decide to do that thing, it is probably because at least some of us want to do it, so the conditional prediction will be higher than the counterfactual prediction.

What We Did in the Evening

The goal of the markets was to help us plan out social events in the evenings. If the market thought that the evening’s rating would be more likely to be higher if we wore halloween costumes than if we used the hot tub, then we should decide to wear halloween costumes.

People mostly did not use the markets to decide what to do. On the first evening, the highest rated activity was a guitar sing-along. We did not end up doing that on any of the evenings. The activity that seems to have been the most fun for the most people¹ was cooperative round-the-table ping-pong. This was done spontaneously, adding more people as they came to the table, without any market predicting the result. We spent a decent amount of time just sitting around talking to each other, which also did not have a market. Our decision making process seemed to be less formal: someone would suggest an activity or say that they would personally do the activity, and other people would join. Having someone look at the markets and announce which activity rated the highest would have added more steps and organization compared to what we did.

We also tried varying the structure of the markets to see if that made them more useful. For example, the market “Will we use the hot tub and have fun tonight?” had four choices for the combinations of whether or not at least four people would use the hot tub and whether the average evening rating would be above or below 7.² Katja did use this market to argue that people should use the hot tub.

There seems to have been a few things that kept the markets from being more useful: (1) Most of us did not know what kinds of social activities most of the rest of us preferred, so it was hard for anyone to make an informed bet. It wasn’t clear how the market provided more information than if we had used a voting system. (2) The connection between four people doing an activity and the average evening rating was too weak for much of a signal to go through. The ratings ended up being noisy, and not specific enough for particular activities. (3) The act of checking the markets and announcing a decision was more formal than our actual decision making process. The market only included a short list of possibilities and did not suggest spontaneity. 

Conclusion

Having prediction markets for the evening social activities was a fun addition to the AI Impacts retreat. There were about 20 markets about the retreat which most of the people at the retreat bet on. But the markets did not end up having a significant impact on what we did during the evening.

Most of us did not have experience using prediction markets before the retreat. We decided not to use the markets to make important decisions, because we did not know what problems they would cause. The markets would likely have been more impactful if we were more experienced and if the questions were about more important decisions. If we did use the markets for important decisions, we would have to make sure that the markets are harder to exploit and have more rules and fewer norms governing how we would bet on the markets.

Since the retreat, Katja has used a market to help plan an AI Impacts dinner. We plan to continue experimenting with using prediction markets to make predictions in the future.

Notes

1. Three people rated the evening 9. 
2. A fifth choice for the average evening rating being exactly 7 was added by someone else.

Katja Grace, Aug 31 2022

This is going to be a list of holes I see in the basic argument for existential risk from superhuman AI systems¹. 

To start, here’s an outline of what I take to be the basic case²:

I. If superhuman AI systems are built, any given system is likely to be ‘goal-directed’

Reasons to expect this:

1. Goal-directed behavior is likely to be valuable, e.g. economically. 
2. Goal-directed entities may tend to arise from machine learning training processes not intending to create them (at least via the methods that are likely to be used).
3. ‘Coherence arguments’ may imply that systems with some goal-directedness will become more strongly goal-directed over time.

II. If goal-directed superhuman AI systems are built, their desired outcomes will probably be about as bad as an empty universe by human lights 

Reasons to expect this:

1. Finding useful goals that aren’t extinction-level bad appears to be hard: we don’t have a way to usefully point at human goals, and divergences from human goals seem likely to produce goals that are in intense conflict with human goals, due to a) most goals producing convergent incentives for controlling everything, and b) value being ‘fragile’, such that an entity with ‘similar’ values will generally create a future of virtually no value.
2. Finding goals that are extinction-level bad and temporarily useful appears to be easy: for example, advanced AI with the sole objective ‘maximize company revenue’ might profit said company for a time before gathering the influence and wherewithal to pursue the goal in ways that blatantly harm society.
3. Even if humanity found acceptable goals, giving a powerful AI system any specific goals appears to be hard. We don’t know of any procedure to do it, and we have theoretical reasons to expect that AI systems produced through machine learning training will generally end up with goals other than those they were trained according to. Randomly aberrant goals resulting are probably extinction-level bad for reasons described in II.1 above.

III. If most goal-directed superhuman AI systems have bad goals, the future will very likely be bad

That is, a set of ill-motivated goal-directed superhuman AI systems, of a scale likely to occur, would be capable of taking control over the future from humans. This is supported by at least one of the following being true:

1. Superhuman AI would destroy humanity rapidly. This may be via ultra-powerful capabilities at e.g. technology design and strategic scheming, or through gaining such powers in an ‘intelligence explosion‘ (self-improvement cycle). Either of those things may happen either through exceptional heights of intelligence being reached or through highly destructive ideas being available to minds only mildly beyond our own.
2. Superhuman AI would gradually come to control the future via accruing power and resources. Power and resources would be more available to the AI system(s) than to humans on average, because of the AI having far greater intelligence.

***

Below is a list of gaps in the above, as I see it, and counterarguments. A ‘gap’ is not necessarily unfillable, and may have been filled in any of the countless writings on this topic that I haven’t read. I might even think that a given one can probably be filled. I just don’t know what goes in it.  

This blog post is an attempt to run various arguments by you all on the way to making pages on AI Impacts about arguments for AI risk and corresponding counterarguments. At some point in that process I hope to also read others’ arguments, but this is not that day. So what you have here is a bunch of arguments that occur to me, not an exhaustive literature review. 

Counterarguments

A. Contra “superhuman AI systems will be ‘goal-directed’”

Different calls to ‘goal-directedness’ don’t necessarily mean the same concept

‘Goal-directedness’ is a vague concept. It is unclear that the ‘goal-directednesses’ that are favored by economic pressure, training dynamics or coherence arguments (the component arguments in part I of the argument above) are the same ‘goal-directedness’ that implies a zealous drive to control the universe (i.e. that makes most possible goals very bad, fulfilling II above). 

One well-defined concept of goal-directedness is ‘utility maximization’: always doing what maximizes a particular utility function, given a particular set of beliefs about the world. 

Utility maximization does seem to quickly engender an interest in controlling literally everything, at least for many utility functions one might have³. If you want things to go a certain way, then you have reason to control anything which gives you any leverage over that, i.e. potentially all resources in the universe (i.e. agents have ‘convergent instrumental goals’). This is in serious conflict with anyone else with resource-sensitive goals, even if prima facie those goals didn’t look particularly opposed. For instance, a person who wants all things to be red and another person who wants all things to be cubes may not seem to be at odds, given that all things could be red cubes. However if these projects might each fail for lack of energy, then they are probably at odds. 

Thus utility maximization is a notion of goal-directedness that allows Part II of the argument to work, by making a large class of goals deadly.

You might think that any other concept of ‘goal-directedness’ would also lead to this zealotry. If one is inclined toward outcome O in any plausible sense, then does one not have an interest in anything that might help procure O? No: if a system is not a ‘coherent’ agent, then it can have a tendency to bring about O in a range of circumstances, without this implying that it will take any given effective opportunity to pursue O. This assumption of consistent adherence to a particular evaluation of everything is part of utility maximization, not a law of physical systems. Call machines that push toward particular goals but are not utility maximizers pseudo-agents. 

Can pseudo-agents exist? Yes—utility maximization is computationally intractable, so any physically existent ‘goal-directed’ entity is going to be a pseudo-agent. We are all pseudo-agents, at best. But it seems something like a spectrum. At one end is a thermostat, then maybe a thermostat with a better algorithm for adjusting the heat. Then maybe a thermostat which intelligently controls the windows. After a lot of honing, you might have a system much more like a utility-maximizer: a system that deftly seeks out and seizes well-priced opportunities to make your room 68 degrees—upgrading your house, buying R&D, influencing your culture, building a vast mining empire. Humans might not be very far on this spectrum, but they seem enough like utility-maximizers already to be alarming. (And it might not be well-considered as a one-dimensional spectrum—for instance, perhaps ‘tendency to modify oneself to become more coherent’ is a fairly different axis from ‘consistency of evaluations of options and outcomes’, and calling both ‘more agentic’ is obscuring.)

Nonetheless, it seems plausible that there is a large space of systems which strongly increase the chance of some desirable objective O occurring without even acting as much like maximizers of an identifiable utility function as humans would. For instance, without searching out novel ways of making O occur, or modifying themselves to be more consistently O-maximizing. Call these ‘weak pseudo-agents’. 

For example, I can imagine a system constructed out of a huge number of ‘IF X THEN Y’ statements (reflexive responses), like ‘if body is in hallway, move North’, ‘if hands are by legs and body is in kitchen, raise hands to waist’.., equivalent to a kind of vector field of motions, such that for every particular state, there are directions that all the parts of you should be moving. I could imagine this being designed to fairly consistently cause O to happen within some context. However since such behavior would not be produced by a process optimizing O, you shouldn’t expect it to find new and strange routes to O, or to seek O reliably in novel circumstances. There appears to be zero pressure for this thing to become more coherent, unless its design already involves reflexes to move its thoughts in certain ways that lead it to change itself. I expect you could build a system like this that reliably runs around and tidies your house say, or runs your social media presence, without it containing any impetus to become a more coherent agent (because it doesn’t have any reflexes that lead to pondering self-improvement in this way).

It is not clear that economic incentives generally favor the far end of this spectrum over weak pseudo-agency. There are incentives toward systems being more like utility maximizers, but also incentives against. 

The reason any kind of ‘goal-directedness’ is incentivised in AI systems is that then the system can be given an objective by someone hoping to use their cognitive labor, and the system will make that objective happen. Whereas a similar non-agentic AI system might still do almost the same cognitive labor, but require an agent (such as a person) to look at the objective and decide what should be done to achieve it, then ask the system for that. Goal-directedness means automating this high-level strategizing. 

Weak pseudo-agency fulfills this purpose to some extent, but not as well as utility maximization. However if we think that utility maximization is difficult to wield without great destruction, then that suggests a disincentive to creating systems with behavior closer to utility-maximization. Not just from the world being destroyed, but from the same dynamic causing more minor divergences from expectations, if the user can’t specify their own utility function well. 

That is, if it is true that utility maximization tends to lead to very bad outcomes relative to any slightly different goals (in the absence of great advances in the field of AI alignment), then the most economically favored level of goal-directedness seems unlikely to be as far as possible toward utility maximization. More likely it is a level of pseudo-agency that achieves a lot of the users’ desires without bringing about sufficiently detrimental side effects to make it not worthwhile. (This is likely more agency than is socially optimal, since some of the side-effects will be harms to others, but there seems no reason to think that it is a very high degree of agency.)

Some minor but perhaps illustrative evidence: anecdotally, people prefer interacting with others who predictably carry out their roles or adhere to deontological constraints, rather than consequentialists in pursuit of broadly good but somewhat unknown goals. For instance, employers would often prefer employees who predictably follow rules than ones who try to forward company success in unforeseen ways.

The other arguments to expect goal-directed systems mentioned above seem more likely to suggest approximate utility-maximization rather than some other form of goal-directedness, but it isn’t that clear to me. I don’t know what kind of entity is most naturally produced by contemporary ML training. Perhaps someone else does. I would guess that it’s more like the reflex-based agent described above, at least at present. But present systems aren’t the concern.

Coherence arguments are arguments for being coherent a.k.a. maximizing a utility function, so one might think that they imply a force for utility maximization in particular. That seems broadly right. Though note that these are arguments that there is some pressure for the system to modify itself to become more coherent. What actually results from specific systems modifying themselves seems like it might have details not foreseen in an abstract argument merely suggesting that the status quo is suboptimal whenever it is not coherent. Starting from a state of arbitrary incoherence and moving iteratively in one of many pro-coherence directions produced by whatever whacky mind you currently have isn’t obviously guaranteed to increasingly approximate maximization of some sensical utility function. For instance, take an entity with a cycle of preferences, apples > bananas = oranges > pears > apples. The entity notices that it sometimes treats oranges as better than pears and sometimes worse. It tries to correct by adjusting the value of oranges to be the same as pears. The new utility function is exactly as incoherent as the old one. Probably moves like this are rarer than ones that make you more coherent in this situation, but I don’t know, and I also don’t know if this is a great model of the situation for incoherent systems that could become more coherent.

What it might look like if this gap matters: AI systems proliferate, and have various goals. Some AI systems try to make money in the stock market. Some make movies. Some try to direct traffic optimally. Some try to make the Democratic party win an election. Some try to make Walmart maximally profitable. These systems have no perceptible desire to optimize the universe for forwarding these goals because they aren’t maximizing a general utility function, they are more ‘behaving like someone who is trying to make Walmart profitable’. They make strategic plans and think about their comparative advantage and forecast business dynamics, but they don’t build nanotechnology to manipulate everybody’s brains, because that’s not the kind of behavior pattern they were designed to follow. The world looks kind of like the current world, in that it is fairly non-obvious what any entity’s ‘utility function’ is. It often looks like AI systems are ‘trying’ to do things, but there’s no reason to think that they are enacting a rational and consistent plan, and they rarely do anything shocking or galaxy-brained.

Ambiguously strong forces for goal-directedness need to meet an ambiguously high bar to cause a risk

The forces for goal-directedness mentioned in I are presumably of finite strength. For instance, if coherence arguments correspond to pressure for machines to become more like utility maximizers, there is an empirical answer to how fast that would happen with a given system. There is also an empirical answer to how ‘much’ goal directedness is needed to bring about disaster, supposing that utility maximization would bring about disaster and, say, being a rock wouldn’t. Without investigating these empirical details, it is unclear whether a particular qualitatively identified force for goal-directedness will cause disaster within a particular time.

What it might look like if this gap matters: There are not that many systems doing something like utility maximization in the new AI economy. Demand is mostly for systems more like GPT or DALL-E, which transform inputs in some known way without reference to the world, rather than ‘trying’ to bring about an outcome. Maybe the world was headed for more of the latter, but ethical and safety concerns reduced desire for it, and it wasn’t that hard to do something else. Companies setting out to make non-agentic AI systems have no trouble doing so. Incoherent AIs are never observed making themselves more coherent, and training has never produced an agent unexpectedly. There are lots of vaguely agentic things, but they don’t pose much of a problem. There are a few things at least as agentic as humans, but they are a small part of the economy.

B. Contra “goal-directed AI systems’ goals will be bad”

Small differences in utility functions may not be catastrophic

Arguably, humans are likely to have somewhat different values to one another even after arbitrary reflection. If so, there is some extended region of the space of possible values that the values of different humans fall within. That is, ‘human values’ is not a single point.

If the values of misaligned AI systems fall within that region, this would not appear to be worse in expectation than the situation where the long-run future was determined by the values of humans other than you. (This may still be a huge loss of value relative to the alternative, if a future determined by your own values is vastly better than that chosen by a different human, and if you also expected to get some small fraction of the future, and will now get much less. These conditions seem non-obvious however, and if they obtain you should worry about more general problems than AI.)

Plausibly even a single human, after reflecting, could on their own come to different places in a whole region of specific values, depending on somewhat arbitrary features of how the reflecting period went. In that case, even the values-on-reflection of a single human is an extended region of values space, and an AI which is only slightly misaligned could be the same as some version of you after reflecting.

There is a further larger region, ‘that which can be reliably enough aligned with typical human values via incentives in the environment’, which is arguably larger than the circle containing most human values. Human society makes use of this a lot: for instance, most of the time particularly evil humans don’t do anything too objectionable because it isn’t in their interests. This region is probably smaller for more capable creatures such as advanced AIs, but still it is some size.

Thus it seems that some amount of AI divergence from your own values is probably broadly fine, i.e. not worse than what you should otherwise expect without AI. 

Thus in order to arrive at a conclusion of doom, it is not enough to argue that we cannot align AI perfectly. The question is a quantitative one of whether we can get it close enough. And how close is ‘close enough’ is not known. 

What it might look like if this gap matters: there are many superintelligent goal-directed AI systems around. They are trained to have human-like goals, but we know that their training is imperfect and none of them has goals exactly like those presented in training. However if you just heard about a particular system’s intentions, you wouldn’t be able to guess if it was an AI or a human. Things happen much faster than they were, because superintelligent AI is superintelligent, but not obviously in a direction less broadly in line with human goals than when humans were in charge.

Differences between AI and human values may be small 

AI trained to have human-like goals will have something close to human-like goals. How close? Call it d, for a particular occasion of training AI. 

If d doesn’t have to be 0 for safety (from above), then there is a question of whether it is an acceptable size. 

I know of two issues here, pushing d upward. One is that with a finite number of training examples, the fit between the true function and the learned function will be wrong. The other is that you might accidentally create a monster (‘misaligned mesaoptimizer’) who understands its situation and pretends to have the utility function you are aiming for so that it can be freed and go out and manifest its own utility function, which could be just about anything. If this problem is real, then the values of an AI system might be arbitrarily different from the training values, rather than ‘nearby’ in some sense, so d is probably unacceptably large. But if you avoid creating such mesaoptimizers, then it seems plausible to me that d is very small. 

If humans also substantially learn their values via observing examples, then the variation in human values is arising from a similar process, so might be expected to be of a similar scale. If we care to make the ML training process more accurate than the human learning one, it seems likely that we could. For instance, d gets smaller with more data.

Another line of evidence is that for things that I have seen AI learn so far, the distance from the real thing is intuitively small. If AI learns my values as well as it learns what faces look like, it seems plausible that it carries them out better than I do.

As minor additional evidence here, I don’t know how to describe any slight differences in utility functions that are catastrophic. Talking concretely, what does a utility function look like that is so close to a human utility function that an AI system has it after a bunch of training, but which is an absolute disaster? Are we talking about the scenario where the AI values a slightly different concept of justice, or values satisfaction a smidgen more relative to joy than it should? And then that’s a moral disaster because it is wrought across the cosmos? Or is it that it looks at all of our inaction and thinks we want stuff to be maintained very similar to how it is now, so crushes any efforts to improve things? 

What it might look like if this gap matters: when we try to train AI systems to care about what specific humans care about, they usually pretty much do, as far as we can tell. We basically get what we trained for. For instance, it is hard to distinguish them from the human in question. (It is still important to actually do this training, rather than making AI systems not trained to have human values.)

Maybe value isn’t fragile

Eliezer argued that value is fragile, via examples of ‘just one thing’ that you can leave out of a utility function, and end up with something very far away from what humans want. For instance, if you leave out ‘boredom’ then he thinks the preferred future might look like repeating the same otherwise perfect moment again and again. (His argument is perhaps longer—that post says there is a lot of important background, though the bits mentioned don’t sound relevant to my disagreement.) This sounds to me like ‘value is not resilient to having components of it moved to zero’, which is a weird usage of ‘fragile’, and in particular, doesn’t seem to imply much about smaller perturbations. And smaller perturbations seem like the relevant thing with AI systems trained on a bunch of data to mimic something. 

You could very analogously say ‘human faces are fragile’ because if you just leave out the nose it suddenly doesn’t look like a typical human face at all. Sure, but is that the kind of error you get when you try to train ML systems to mimic human faces? Almost none of the faces on thispersondoesnotexist.com are blatantly morphologically unusual in any way, let alone noseless. Admittedly one time I saw someone whose face was neon green goo, but I’m guessing you can get the rate of that down pretty low if you care about it.

Eight examples, no cherry-picking:

Skipping the nose is the kind of mistake you make if you are a child drawing a face from memory. Skipping ‘boredom’ is the kind of mistake you make if you are a person trying to write down human values from memory. My guess is that this seemed closer to the plan in 2009 when that post was written, and that people cached the takeaway and haven’t updated it for deep learning which can learn what faces look like better than you can.

What it might look like if this gap matters: there is a large region ‘around’ my values in value space that is also pretty good according to me. AI easily lands within that space, and eventually creates some world that is about as good as the best possible utopia, according to me. There aren’t a lot of really crazy and terrible value systems adjacent to my values.

Short-term goals

Utility maximization really only incentivises drastically altering the universe if one’s utility function places a high enough value on very temporally distant outcomes relative to near ones. That is, long term goals are needed for danger. A person who cares most about winning the timed chess game in front of them should not spend time accruing resources to invest in better chess-playing.

AI systems could have long-term goals via people intentionally training them to do so, or via long-term goals naturally arising from systems not trained so. 

Humans seem to discount the future a lot in their usual decision-making (they have goals years in advance but rarely a hundred years) so the economic incentive to train AI to have very long term goals might be limited.

It’s not clear that training for relatively short term goals naturally produces creatures with very long term goals, though it might.

Thus if AI systems fail to have value systems relatively similar to human values, it is not clear that many will have the long time horizons needed to motivate taking over the universe.

What it might look like if this gap matters: the world is full of agents who care about relatively near-term issues, and are helpful to that end, and have no incentive to make long-term large scale schemes. Reminiscent of the current world, but with cleverer short-termism.

C. Contra “superhuman AI would be sufficiently superior to humans to overpower humanity”

Human success isn’t from individual intelligence

The argument claims (or assumes) that surpassing ‘human-level’ intelligence (i.e. the mental capacities of an individual human) is the relevant bar for matching the power-gaining capacity of humans, such that passing this bar in individual intellect means outcompeting humans in general in terms of power (argument III.2), if not being able to immediately destroy them all outright (argument III.1.). In a similar vein, introductions to AI risk often start by saying that humanity has triumphed over the other species because it is more intelligent, as a lead in to saying that if we make something more intelligent still, it will inexorably triumph over humanity.

This hypothesis about the provenance of human triumph seems wrong. Intellect surely helps, but humans look to be powerful largely because they share their meager intellectual discoveries with one another and consequently save them up over time⁴. You can see this starkly by comparing the material situation of Alice, a genius living in the stone age, and Bob, an average person living in 21st Century America. Alice might struggle all day to get a pot of water, while Bob might be able to summon all manner of delicious drinks from across the oceans, along with furniture, electronics, information, etc. Much of Bob’s power probably did flow from the application of intelligence, but not Bob’s individual intelligence. Alice’s intelligence, and that of those who came between them.

Bob’s greater power isn’t directly just from the knowledge and artifacts Bob inherits from other humans. He also seems to be helped for instance by much better coordination: both from a larger number people coordinating together, and from better infrastructure for that coordination (e.g. for Alice the height of coordination might be an occasional big multi-tribe meeting with trade, and for Bob it includes global instant messaging and banking systems and the Internet). One might attribute all of this ultimately to innovation, and thus to intelligence and communication, or not. I think it’s not important to sort out here, as long as it’s clear that individual intelligence isn’t the source of power.

It could still be that with a given bounty of shared knowledge (e.g. within a given society), intelligence grants huge advantages. But even that doesn’t look true here: 21st Century geniuses live basically like 21st Century people of average intelligence, give or take.

Why does this matter? Well for one thing, if you make AI which is merely as smart as a human, you shouldn’t then expect it to do that much better than a genius living in the stone age. That’s what human-level intelligence gets you: nearly nothing. A piece of rope after millions of lifetimes. Humans without their culture are much like other animals. 

To wield the control-over-the-world of a genius living in the 21st Century, the human-level AI would seem to need something like the other benefits that the 21st century genius gets from their situation in connection with a society. 

One such thing is access to humanity’s shared stock of hard-won information. AI systems plausibly do have this, if they can get most of what is relevant by reading the internet. This isn’t obvious: people also inherit information from society through copying habits and customs, learning directly from other people, and receiving artifacts with implicit information (for instance, a factory allows whoever owns the factory to make use of intellectual work that was done by the people who built the factory, but that information may not available explicitly even for the owner of the factory, let alone to readers on the internet). These sources of information seem likely to also be available to AI systems though, at least if they are afforded the same options as humans.

My best guess is that AI systems easily do better than humans on extracting information from humanity’s stockpile, and on coordinating, and so on this account are probably in an even better position to compete with humans than one might think on the individual intelligence model, but that is a guess. In that case perhaps this misunderstanding makes little difference to the outcomes of the argument. However it seems at least a bit more complicated. 

Suppose that AI systems can have access to all information humans can have access to. The power the 21st century person gains from their society is modulated by their role in society, and relationships, and rights, and the affordances society allows them as a result. Their power will vary enormously depending on whether they are employed, or listened to, or paid, or a citizen, or the president. If AI systems’ power stems substantially from interacting with society, then their power will also depend on affordances granted, and humans may choose not to grant them many affordances (see section ‘Intelligence may not be an overwhelming advantage’ for more discussion).

However suppose that your new genius AI system is also treated with all privilege. The next way that this alternate model matters is that if most of what is good in a person’s life is determined by the society they are part of, and their own labor is just buying them a tiny piece of that inheritance, then if they are for instance twice as smart as any other human, they don’t get to use technology that it twice as good. They just get a larger piece of that same shared technological bounty purchasable by anyone. Because each individual person is adding essentially nothing in terms of technology, so twice that is still basically nothing. 

In contrast, I think people are often imagining that a single entity somewhat smarter than a human will be able to quickly use technologies that are somewhat better than current human technologies. This seems to be mistaking the actions of a human and the actions of a human society. If a hundred thousand people sometimes get together for a few years and make fantastic new weapons, you should not expect an entity somewhat smarter than a person to make even better weapons. That’s off by a factor of about a hundred thousand. 

There might be places you can get far ahead of humanity by being better than a single human—it depends how much accomplishments depend on the few most capable humans in the field, and how few people are working on the problem. But for instance the Manhattan Project took a hundred thousand people several years, and von Neumann (a mythically smart scientist) joining the project did not reduce it to an afternoon. Plausibly to me, some specific people being on the project caused it to not take twice as many person-years, though the plausible candidates here seem to be more in the business of running things than doing science directly (though that also presumably involves intelligence). But even if you are an ambitious somewhat superhuman intelligence, the influence available to you seems to plausibly be limited to making a large dent in the effort required for some particular research endeavor, not single-handedly outmoding humans across many research endeavors.

This is all reason to doubt that a small number of superhuman intelligences will rapidly take over or destroy the world (as in III.i.). This doesn’t preclude a set of AI systems that are together more capable than a large number of people from making great progress. However some related issues seem to make that less likely.

Another implication of this model is that if most human power comes from buying access to society’s shared power, i.e. interacting with the economy, you should expect intellectual labor by AI systems to usually be sold, rather than for instance put toward a private stock of knowledge. This means the intellectual outputs are mostly going to society, and the main source of potential power to an AI system is the wages received (which may allow it to gain power in the long run). However it seems quite plausible that AI systems at this stage will generally not receive wages, since they presumably do not need them to be motivated to do the work they were trained for. It also seems plausible that they would be owned and run by humans. This would seem to not involve any transfer of power to that AI system, except insofar as its intellectual outputs benefit it (e.g. if it is writing advertising material, maybe it doesn’t get paid for that, but if it can write material that slightly furthers its own goals in the world while also fulfilling the advertising requirements, then it sneaked in some influence.) 

If there is AI which is moderately more competent than humans, but not sufficiently more competent to take over the world, then it is likely to contribute to this stock of knowledge and affordances shared with humans. There is no reason to expect it to build a separate competing stock, any more than there is reason for a current human household to try to build a separate competing stock rather than sell their labor to others in the economy. 

In summary:

1. Functional connection with a large community of other intelligences in the past and present is probably a much bigger factor in the success of humans as a species or individual humans than is individual intelligence. 
2. Thus this also seems more likely to be important for AI success than individual intelligence. This is contrary to a usual argument for AI superiority, but probably leaves AI systems at least as likely to outperform humans, since superhuman AI is probably superhumanly good at taking in information and coordinating.
3. However it is not obvious that AI systems will have the same access to society’s accumulated information e.g. if there is information which humans learn from living in society, rather than from reading the internet. 
4. And it seems an open question whether AI systems are given the same affordances in society as humans, which also seem important to making use of the accrued bounty of power over the world that humans have. For instance, if they are not granted the same legal rights as humans, they may be at a disadvantage in doing trade or engaging in politics or accruing power.
5. The fruits of greater intelligence for an entity will probably not look like society-level accomplishments unless it is a society-scale entity
6. The route to influence with smaller fruits probably by default looks like participating in the economy rather than trying to build a private stock of knowledge.
7. If the resources from participating in the economy accrue to the owners of AI systems, not to the systems themselves, then there is less reason to expect the systems to accrue power incrementally, and they are at a severe disadvantage relative to humans. 

Overall these are reasons to expect AI systems with around human-level cognitive performance to not destroy the world immediately, and to not amass power as easily as one might imagine. 

What it might look like if this gap matters: If AI systems are somewhat superhuman, then they do impressive cognitive work, and each contributes to technology more than the best human geniuses, but not more than the whole of society, and not enough to materially improve their own affordances. They don’t gain power rapidly because they are disadvantaged in other ways, e.g. by lack of information, lack of rights, lack of access to positions of power. Their work is sold and used by many actors, and the proceeds go to their human owners. AI systems do not generally end up with access to masses of technology that others do not have access to, and nor do they have private fortunes. In the long run, as they become more powerful, they might take power if other aspects of the situation don’t change. 

AI agents may not be radically superior to combinations of humans and non-agentic machines

‘Human level capability’ is a moving target. For comparing the competence of advanced AI systems to humans, the relevant comparison is with humans who have state-of-the-art AI and other tools. For instance, the human capacity to make art quickly has recently been improved by a variety of AI art systems. If there were now an agentic AI system that made art, it would make art much faster than a human of 2015, but perhaps hardly faster than a human of late 2022. If humans continually have access to tool versions of AI capabilities, it is not clear that agentic AI systems must ever have an overwhelmingly large capability advantage for important tasks (though they might). 

(This is not an argument that humans might be better than AI systems, but rather: if the gap in capability is smaller, then the pressure for AI systems to accrue power is less and thus loss of human control is slower and easier to mitigate entirely through other forces, such as subsidizing human involvement or disadvantaging AI systems in the economy.)

Some advantages of being an agentic AI system vs. a human with a tool AI system seem to be:

1. There might just not be an equivalent tool system, for instance if it is impossible to train systems without producing emergent agents.
2. When every part of a process takes into account the final goal, this should make the choices within the task more apt for the final goal (and agents know their final goal, whereas tools carrying out parts of a larger problem do not).
3. For humans, the interface for using a capability of one’s mind tends to be smoother than the interface for using a tool. For instance a person who can do fast mental multiplication can do this more smoothly and use it more often than a person who needs to get out a calculator. This seems likely to persist.

1 and 2 may or may not matter much. 3 matters more for brief, fast, unimportant tasks. For instance, consider again people who can do mental calculations better than others. My guess is that this advantages them at using Fermi estimates in their lives and buying cheaper groceries, but does not make them materially better at making large financial choices well. For a one-off large financial choice, the effort of getting out a calculator is worth it and the delay is very short compared to the length of the activity. The same seems likely true of humans with tools vs. agentic AI with the same capacities integrated into their minds. Conceivably the gap between humans with tools and goal-directed AI is small for large, important tasks.

What it might look like if this gap matters: agentic AI systems have substantial advantages over humans with tools at some tasks like rapid interaction with humans, and responding to rapidly evolving strategic situations.  One-off large important tasks such as advanced science are mostly done by tool AI. 

Trust

If goal-directed AI systems are only mildly more competent than some combination of tool systems and humans (as suggested by considerations in the last two sections), we still might expect AI systems to out-compete humans, just more slowly. However AI systems have one serious disadvantage as employees of humans: they are intrinsically untrustworthy, while we don’t understand them well enough to be clear on what their values are or how they will behave in any given case. Even if they did perform as well as humans at some task, if humans can’t be certain of that, then there is reason to disprefer using them. This can be thought of as two problems: firstly, slightly misaligned systems are less valuable because they genuinely do the thing you want less well, and secondly, even if they were not misaligned, if humans can’t know that (because we have no good way to verify the alignment of AI systems) then it is costly in expectation to use them. (This is only a further force acting against the supremacy of AI systems—they might still be powerful enough that using them is enough of an advantage that it is worth taking the hit on trustworthiness.)

What it might look like if this gap matters: in places where goal-directed AI systems are not typically hugely better than some combination of less goal-directed systems and humans, the job is often given to the latter if trustworthiness matters. 

Headroom

For AI to vastly surpass human performance at a task, there needs to be ample room for improvement above human level. For some tasks, there is not—tic-tac-toe is a classic example. It is not clear how close humans (or technologically aided humans) are from the limits to competence in the particular domains that will matter. It is to my knowledge an open question how much ‘headroom’ there is. My guess is a lot, but it isn’t obvious.

How much headroom there is varies by task. Categories of task for which there appears to be little headroom: 

1. Tasks where we know what the best performance looks like, and humans can get close to it. For instance, machines cannot win more often than the best humans at Tic-tac-toe (playing within the rules) or solve Rubik’s cubes much more reliably, or extracting calories from fuel
2. Tasks where humans are already be reaping most of the value—for instance, perhaps most of the value of forks is in having a handle with prongs attached to the end, and while humans continue to design slightly better ones, and machines might be able to add marginal value to that project more than twice as fast as the human designers, they cannot perform twice as well in terms of the value of each fork, because forks are already 95% as good as they can be. 
3. Better performance is quickly intractable. For instance, we know that for tasks in particular complexity classes, there are computational limits to how well one can perform across the board. Or for chaotic systems, there can be limits to predictability. (That is, tasks might lack headroom not because they are simple, but because they are complex. E.g. AI probably can’t predict the weather much further out than humans.)

Categories of task where a lot of headroom seems likely:

4. Competitive tasks where the value of a certain level of performance depends on whether one is better or worse than one’s opponent, so that the marginal value of more performance doesn’t hit diminishing returns, as long as your opponent keeps competing and taking back what you just won. Though in one way this is like having little headroom: there’s no more value to be had—the game is zero sum. And while there might often be a lot of value to be gained by doing a bit better on the margin, still if all sides can invest, then nobody will end up better off than they were. So whether this seems more like high or low headroom depends on what we are asking exactly. Here we are asking if AI systems can do much better than humans: in a zero sum contest like this, they likely can in the sense that they can beat humans, but not in the sense of reaping anything more from the situation than the humans ever got.
5. Tasks where it is twice as good to do the same task twice as fast, and where speed is bottlenecked on thinking time.
6. Tasks where there is reason to think that optimal performance is radically better than we have seen. For instance, perhaps we can estimate how high Chess Elo rankings must go before reaching perfection by reasoning theoretically about the game, and perhaps it is very high (I don’t know).
7. Tasks where humans appear to use very inefficient methods. For instance, it was perhaps predictable before calculators that they would be able to do mathematics much faster than humans, because humans can only keep a small number of digits in their heads, which doesn’t seem like an intrinsically hard problem. Similarly, I hear humans often use mental machinery designed for one mental activity for fairly different ones, through analogy. For instance, when I think about macroeconomics, I seem to be basically using my intuitions for dealing with water. When I do mathematics in general, I think I’m probably using my mental capacities for imagining physical objects.

What it might look like if this gap matters: many challenges in today’s world remain challenging for AI. Human behavior is not readily predictable or manipulable very far beyond what we have explored, only slightly more complicated schemes are feasible before the world’s uncertainties overwhelm planning; much better ads are soon met by much better immune responses; much better commercial decision-making ekes out some additional value across the board but most products were already fulfilling a lot of their potential; incredible virtual prosecutors meet incredible virtual defense attorneys and everything is as it was; there are a few rounds of attack-and-defense in various corporate strategies before a new equilibrium with broad recognition of those possibilities; conflicts and ‘social issues’ remain mostly intractable. There is a brief golden age of science before the newly low-hanging fruit are again plucked and it is only lightning fast in areas where thinking was the main bottleneck, e.g. not in medicine.

Intelligence may not be an overwhelming advantage

Intelligence is helpful for accruing power and resources, all things equal, but many other things are helpful too. For instance money, social standing, allies, evident trustworthiness, not being discriminated against (this was slightly discussed in section ‘Human success isn’t from individual intelligence’). AI systems are not guaranteed to have those in abundance. The argument assumes that any difference in intelligence in particular will eventually win out over any differences in other initial resources. I don’t know of reason to think that. 

Empirical evidence does not seem to support the idea that cognitive ability is a large factor in success. Situations where one entity is much smarter or more broadly mentally competent than other entities regularly occur without the smarter one taking control over the other:

1. Species exist with all levels of intelligence. Elephants have not in any sense won over gnats; they do not rule gnats; they do not have obviously more control than gnats over the environment. 
2. Competence does not seem to aggressively overwhelm other advantages in humans: 
   1. Looking at the world, intuitively the big discrepancies in power are not seemingly about intelligence.
   2. IQ 130 humans apparently earn very roughly $6000-$18,500 per year more than average IQ humans.
   3. Elected representatives are apparently smarter on average, but it is a slightly shifted curve, not a radically difference.
   4. MENSA isn’t a major force in the world.
   5. Many places where people see huge success through being cognitively able are ones where they show off their intelligence to impress people, rather than actually using it for decision-making. For instance, writers, actors, song-writers, comedians, all sometimes become very successful through cognitive skills. Whereas scientists, engineers and authors of software use cognitive skills to make choices about the world, and less often become extremely rich and famous, say. If intelligence were that useful for strategic action, it seems like using it for that would be at least as powerful as showing it off. But maybe this is just an accident of which fields have winner-takes-all type dynamics.
   6. If we look at people who evidently have good cognitive abilities given their intellectual output, their personal lives are not obviously drastically more successful, anecdotally.
   7. One might counter-counter-argue that humans are very similar to one another in capability, so even if intelligence matters much more than other traits, you won’t see that by looking at  the near-identical humans. This does not seem to be true. Often at least, the difference in performance between mediocre human performance and top level human performance is large, relative to the space below, iirc. For instance, in chess, the Elo difference between the best and worst players is about 2000, whereas the difference between the amateur play and random play is maybe 400-2800 (if you accept Chess StackExchange guesses as a reasonable proxy for the truth here). And in terms of AI progress, amateur human play was reached in the 50s, roughly when research began, and world champion level play was reached in 1997. 

And theoretically I don’t know why one would expect greater intelligence to win out over other advantages over time.  There are actually two questionable theories here: 1) Charlotte having more overall control than David at time 0 means that Charlotte will tend to have an even greater share of control at time 1. And, 2) Charlotte having more intelligence than David at time 0 means that Charlotte will have a greater share of control at time 1 even if Bob has more overall control (i.e. more of other resources) at time 1.

What it might look like if this gap matters: there are many AI systems around, and they strive for various things. They don’t hold property, or vote, or get a weight in almost anyone’s decisions, or get paid, and are generally treated with suspicion. These things on net keep them from gaining very much power. They are very persuasive speakers however and we can’t stop them from communicating, so there is a constant risk of people willingly handing them power, in response to their moving claims that they are an oppressed minority who suffer. The main thing stopping them from winning is that their position as psychopaths bent on taking power for incredibly pointless ends is widely understood.

Unclear that many goals realistically incentivise taking over the universe

I have some goals. For instance, I want some good romance. My guess is that trying to take over the universe isn’t the best way to achieve this goal. The same goes for a lot of my goals, it seems to me. Possibly I’m in error, but I spend a lot of time pursuing goals, and very little of it trying to take over the universe. Whether a particular goal is best forwarded by trying to take over the universe as a substep seems like a quantitative empirical question, to which the answer is virtually always ‘not remotely’. Don’t get me wrong: all of these goals involve some interest in taking over the universe. All things equal, if I could take over the universe for free, I do think it would help in my romantic pursuits. But taking over the universe is not free. It’s actually super duper duper expensive and hard. So for most goals arising, it doesn’t bear considering. The idea of taking over the universe as a substep is entirely laughable for almost any human goal.

So why do we think that AI goals are different? I think the thought is that it’s radically easier for AI systems to take over the world, because all they have to do is to annihilate humanity, and they are way better positioned to do that than I am, and also better positioned to survive the death of human civilization than I am. I agree that it is likely easier, but how much easier? So much easier to take it from ‘laughably unhelpful’ to ‘obviously always the best move’? This is another quantitative empirical question.

What it might look like if this gap matters: Superintelligent AI systems pursue their goals. Often they achieve them fairly well. This is somewhat contrary to ideal human thriving, but not lethal. For instance, some AI systems are trying to maximize Amazon’s market share, within broad legality. Everyone buys truly incredible amounts of stuff from Amazon, and people often wonder if it is too much stuff. At no point does attempting to murder all humans seem like the best strategy for this. 

Quantity of new cognitive labor is an empirical question, not addressed

Whether some set of AI systems can take over the world with their new intelligence probably depends how much total cognitive labor they represent. For instance, if they are in total slightly more capable than von Neumann, they probably can’t take over the world. If they are together as capable (in some sense) as a million 21st Century human civilizations, then they probably can (at least in the 21st Century).

It also matters how much of that is goal-directed at all, and highly intelligent, and how much of that is directed at achieving the AI systems’ own goals rather than those we intended them for, and how much of that is directed at taking over the world. 

If we continued to build hardware, presumably at some point AI systems would account for most of the cognitive labor in the world. But if there is first an extended period of more minimal advanced AI presence, that would probably prevent an immediate death outcome, and improve humanity’s prospects for controlling a slow-moving AI power grab. 

What it might look like if this gap matters: when advanced AI is developed, there is a lot of new cognitive labor in the world, but it is a minuscule fraction of all of the cognitive labor in the world. A large part of it is not goal-directed at all, and of that, most of the new AI thought is applied to tasks it was intended for. Thus what part of it is spent on scheming to grab power for AI systems is too small to grab much power quickly. The amount of AI cognitive labor grows fast over time, and in several decades it is most of the cognitive labor, but humanity has had extensive experience dealing with its power grabbing.

Speed of intelligence growth is ambiguous

The idea that a superhuman AI would be able to rapidly destroy the world seems prima facie unlikely, since no other entity has ever done that. Two common broad arguments for it:

1. There will be a feedback loop in which intelligent AI makes more intelligent AI repeatedly until AI is very intelligent.
2. Very small differences in brains seem to correspond to very large differences in performance, based on observing humans and other apes. Thus any movement past human-level will take us to unimaginably superhuman level.

These both seem questionable.

1. Feedback loops can happen at very different rates. Identifying a feedback loop empirically does not signify an explosion of whatever you are looking at. For instance, technology is already helping improve technology. To get to a confident conclusion of doom, you need evidence that the feedback loop is fast.
2. It does not seem clear that small improvements in brains lead to large changes in intelligence in general, or will do on the relevant margin. Small differences between humans and other primates might include those helpful for communication (see Section ‘Human success isn’t from individual intelligence’), which do not seem relevant here. If there were a particularly powerful cognitive development between chimps and humans, it is unclear that AI researchers find that same insight at the same point in the process (rather than at some other time). 

A large number of other arguments have been posed for expecting very fast growth in intelligence at around human level. I previously made a list of them with counterarguments, though none seemed very compelling. Overall, I don’t know of strong reason to expect very fast growth in AI capabilities at around human-level AI performance, though I hear such arguments might exist. 

What it would look like if this gap mattered: AI systems would at some point perform at around human level at various tasks, and would contribute to AI research, along with everything else. This would contribute to progress to an extent familiar from other technological progress feedback, and would not e.g. lead to a superintelligent AI system in minutes.

Key concepts are vague

Concepts such as ‘control’, ‘power’, and ‘alignment with human values’ all seem vague. ‘Control’ is not zero sum (as seemingly assumed) and is somewhat hard to pin down, I claim. What an ‘aligned’ entity is exactly seems to be contentious in the AI safety community, but I don’t know the details. My guess is that upon further probing, these conceptual issues are resolvable in a way that doesn’t endanger the argument, but I don’t know. I’m not going to go into this here.

What it might look like if this gap matters: upon thinking more, we realize that our concerns were confused. Things go fine with AI in ways that seem obvious in retrospect. This might look like it did for people concerned about the ‘population bomb’ or as it did for me in some of my youthful concerns about sustainability: there was a compelling abstract argument for a problem, and the reality didn’t fit the abstractions well enough to play out as predicted.

D. Contra the whole argument

The argument overall proves too much about corporations

Here is the argument again, but modified to be about corporations. A couple of pieces don’t carry over, but they don’t seem integral.

I. Any given corporation is likely to be ‘goal-directed’

Reasons to expect this:

4. Goal-directed behavior is likely to be valuable in corporations, e.g. economically
5. Goal-directed entities may tend to arise from machine learning training processes not intending to create them (at least via the methods that are likely to be used).
6. ‘Coherence arguments’ may imply that systems with some goal-directedness will become more strongly goal-directed over time.

II. If goal-directed superhuman corporations are built, their desired outcomes will probably be about as bad as an empty universe by human lights

Reasons to expect this:

4. Finding useful goals that aren’t extinction-level bad appears to be hard: we don’t have a way to usefully point at human goals, and divergences from human goals seem likely to produce goals that are in intense conflict with human goals, due to a) most goals producing convergent incentives for controlling everything, and b) value being ‘fragile’, such that an entity with ‘similar’ values will generally create a future of virtually no value. 
5. Finding goals that are extinction-level bad and temporarily useful appears to be easy: for example, corporations with the sole objective ‘maximize company revenue’ might profit for a time before gathering the influence and wherewithal to pursue the goal in ways that blatantly harm society.
6. Even if humanity found acceptable goals, giving a corporation any specific goals appears to be hard. We don’t know of any procedure to do it, and we have theoretical reasons to expect that AI systems produced through machine learning training will generally end up with goals other than those that they were trained according to. Randomly aberrant goals resulting are probably extinction-level bad, for reasons described in II.1 above.
   

III. If most goal-directed corporations have bad goals, the future will very likely be bad

That is, a set of ill-motivated goal-directed corporations, of a scale likely to occur, would be capable of taking control of the future from humans. This is supported by at least one of the following being true:

1. A corporation would destroy humanity rapidly. This may be via ultra-powerful capabilities at e.g. technology design and strategic scheming, or through gaining such powers in an ‘intelligence explosion‘ (self-improvement cycle). Either of those things may happen either through exceptional heights of intelligence being reached or through highly destructive ideas being available to minds only mildly beyond our own.
2. Superhuman AI would gradually come to control the future via accruing power and resources. Power and resources would be more available to the corporation than to humans on average, because of the corporation having far greater intelligence.

This argument does point at real issues with corporations, but we do not generally consider such issues existentially deadly. 

One might argue that there are defeating reasons that corporations do not destroy the world: they are made of humans so can be somewhat reined in; they are not smart enough; they are not coherent enough. But in that case, the original argument needs to make reference to these things, so that they apply to one and not the other.

What it might look like if this counterargument matters: something like the current world. There are large and powerful systems doing things vastly beyond the ability of individual humans, and acting in a definitively goal-directed way. We have a vague understanding of their goals, and do not assume that they are coherent. Their goals are clearly not aligned with human goals, but they have enough overlap that many people are broadly in favor of their existence. They seek power. This all causes some problems, but problems within the power of humans and other organized human groups to keep under control, for some definition of ‘under control’.

Conclusion

I think there are quite a few gaps in the argument, as I understand it. My current guess (prior to reviewing other arguments and integrating things carefully) is that enough uncertainties might resolve in the dangerous directions that existential risk from AI is a reasonable concern. I don’t at present though see how one would come to think it was overwhelmingly likely.

Published 29 August 2022; last updated 23 August 2022

This project is incomplete, contains information that may change over time, and may or may not be updated in the future

This is a dataset of prizes we could find for incentivizing progress toward a specific technical or intellectual goal.

Details

Inducement prizes are created to encourage effort toward solving a specific problem, usually by offering a large monetary award for accomplishing a goal according to pre-specified criteria. Inducement prizes are distinct from those recognizing past achievements, like the Fields Medal or Nobel Prize.¹

Methodology

We spent approximately 10-12 hours researching prizes for technological and intellectual progress, such as crossing the ocean or proving a mathematical theorem. For inclusion in our list, a prize needed to be:

1. Offered for accomplishing a well-specified goal that is reached primarily through intellectual or technological progress.
2. Announced before the goal had been achieved, with a specific prize amount.

For each prize, we tried to answer several questions:

• What are the basic parameters, such as the prize amount, the conditions for winning, the year it was announced, and the financial sponsor?
• Was the prize collected, and if so, by whom?
• Was the prize’s goal achieved in the intended time frame and for the original prize amount?
• Were there other notable consequences of the prize, such as increased interest in relevant industries or a change in public perception of the party offering the prize?

We were unable to answer every question for every prize in the time allotted to the project, nor were we able to investigate every prize that appeared to match our criteria.

Dataset

This table contains a condensed version of our dataset. The full dataset with sources can be found in this Google Sheet.

Primary author: Elizabeth Santos
Additional writing and research: Rick Korzekwa

Notes

Published 6 Aug 2022

This page may represent little of what is known on the topic. It is incomplete, under active work and may be updated soon.

Superhuman AI appears to be very likely to be created at some point.

Details

Let ‘superhuman’ AI be a set of AI systems that together achieve human-level performance across virtually all tasks (i.e. HLMI) and substantially surpass human-level performance on some tasks.

Arguments

A. Superhuman AI is very likely to be physically possible

1. Human brains prove that it is physically possible to create human-level intelligence

A single human brain is not an existence proof of human-level intelligence according to our definition, because no specific human brain is able to perform at the level of any human brain, on all tasks. Given only the observation of the existence of human brains, it could conceivably be impossible to build a machine that performs any task at the level of any chosen human brain at the total cost of a single human brain. For instance, if each brain could only hold the information required to specialize in one career, then a machine that could do any career would need to store much more information than a brain, and thus could be more expensive.

The entire population of human brains together could be said to perform at ‘human-level’, if the cost of doing a single task is considered to be the cost of the single person’s labor used for that task, rather than the cost of maintaining the entire human race. This seems like a reasonable accounting for the present purposes. Thus, the entire collection of human brains demonstrates that it is physically possible to have a system which can do any task as well as the most proficient human, and can do marginal tasks at the cost of human labor (even if the cost of maintaining the entire system would be much higher, were it not spread between many tasks).

2. We know of no reason to expect that human brains are near the limits of possible intelligence

Human brains do appear to be near the limits of performance for some specific tasks. For instance, humans can play tic-tac-toe perfectly. Also for many tasks, human performance reaps a lot of the value potentially available, so it is impossible to perform much better in terms of value (e.g. selecting lunch from a menu, making a booking, recording a phone number).

However, many tasks do not appear to be like this (e.g. winning at Go), and even for the above mentioned tasks, there is room to carry out the task substantially faster or more cheaply than a human does. Thus there appears to be room for substantially better-than-human performance on a wide range of tasks, though we have not seen a careful accounting of this.

3. Artificial minds appear to have some intrinsic advantages over human minds

a) Human brains developed under constraints that would not apply to artificial brains. In particular, energy use was a more important cost, and there were reproductive constraints to human head size.

b) Machines appear to have huge potential performance advantages over biological systems on some fronts. Carlsmith summarizes Bostrom¹:

Thus, as Bostrom (2014, Chapter 3) discusses, where neurons can fire a maximum of hundreds of Hz, the clock speed of modern computers can reach some 2 Ghz — ~ten million times faster. Where action potentials travel at some hundreds of m/s, optical communication can take place at 300,000,000 m/s — ~ a million times faster. Where brain size and neuron count are limited by cranial volume, metabolic constraints, and other factors, supercomputers can be the size of warehouses. And artificial systems need not suffer, either, from the brain’s constraints with respect to memory and component reliability, input/output bandwidth, tiring after hours, degrading after decades, storing its own repair mechanisms and blueprints inside itself, and so forth. Artificial systems can also be edited and duplicated much more easily than brains, and information can be more easily transferred between them.

4. Superhuman AI is very likely to be physically possible (from 1-3)

The human species exists (1). There seems little reason to think that no system could perform tasks substantially better (2), and multiple moderately strong reasons to think that more capable systems are possible (3). Thus it seems very likely that more capable systems are possible.

5. The likely physical possibility of superhuman AI minds strongly suggests the physical feasibility of creating such minds

A superhuman mind is a physically possible object (4). However, that a physical configuration is possible does not imply that bringing about such a configuration intentionally can be feasible in practice. For an example of the difference, a waterfall whose water is in exactly the configuration as that of the Niagara Falls for the last ten minutes is physically possible (the Niagara Falls just did it), yet bringing this about again intentionally may remain intractable forever.

In fact we know that human brains specifically are not only physically possible, but feasible for humans to create. However this is in the form of biological reproduction, and does not appear to straightforwardly imply that humans can create arbitrary different systems with at least the intelligence of humans. That is, human creation of human brains does not obviously imply that it is possible for humans to intentionally create human-level intelligence that isn’t a human brain.

However it seems strongly suggestive. If a physical configuration is possible, natural reasons it might be intractable to bring about are a) that it is computationally difficult to transform the given reference to an actionable description of the physical state required (e.g. ‘Niagara Falls ten minutes ago’ points at a particular configuration of water, but not in a way that is easily convertible to a detailed specification of the locations of that water²), and b) the actionable description is hard to bring about. For instance, it might require minute manipulation of particles beyond what is feasible today, either to get the required degree of specificity, or to avoid chaotic dynamics taking the system away from the desired state even at a macroscopic level (e.g. even if your description of the starting state of the waterfall is fairly detailed, it will quickly diverge farther from the real waterfall.)

These issues don’t appear to apply to creating superhuman intelligence, so in the absence of other evident defeaters, its physical possibility seems to strongly suggest its physical feasibility.

6. Contemporary AI systems exhibit qualitatively similar capabilities to human minds, suggesting modified versions of similar processes would give rise to capabilities matching human minds.

That is, given that current AI techniques create systems that do a lot of human-like tasks at some level of performance, e.g. recognize images and write human-like language, it would be somewhat surprising if getting to human level performance on these tasks required such starkly different methods as to be impossible.

7. Superhuman AI systems will very likely be physically feasible to create (from 4-6)

Superhuman AI systems are probably physically possible (4), and this suggests that they are feasible to create (5). Separately, presently feasible AI systems exhibit qualitatively similar behavior to human minds (6), weakly suggesting that systems exhibiting similar behavior at a higher level of performance will also be feasible to create.

B. If feasible, superhuman AI will very likely be created

8. Superhuman AI would appear to be very economically valuable, given its potential to do most human work better or more cheaply

Whenever such minds become feasible, by stipulation they will be superior in ways to existing sources of cognitive labor, or those sources would have already constituted superhuman AI. Unless the gap is quite small between human-level AI and the best feasible superhuman AI (which seems unlikely given the large potential room for improvement over human minds), the economic value from superhuman AI should be at least at the scale of the global labor market.

9. It appears there will be large incentives to create such systems (from 8)

That a situation would make large amounts of economic value available does not imply that any individual has an incentive to make that situation happen, because the value may not accrue to the possible decision maker. In this case however, substantial parts of the economic value created by superhuman AI systems appear likely to be captured by their creators, by analogy to other commercial software.

These economic incentives may not be the only substantial incentives in play. Creating such systems could incur social or legal consequences which could negate the positive incentives. Thus this step is relatively uncertain.

10. Superhuman AI seems likely to be created

Given that creation of a type of machine is physically feasible (7) and strongly incentivized (9), it seems likely that it will be created.

Notes

Published 6 Aug 2022

This page is incomplete, under active work and may be updated soon.

This is a bibliography of pieces arguing against the idea that AI poses an existential risk.

List

Cegłowski, Maciej. “Superintelligence: The Idea That Eats Smart People.” Idle Words (blog). Accessed December 9, 2021. https://idlewords.com/talks/superintelligence.htm.

Garfinkel, Ben, and Lempel, Howie. “How Sure Are We about This AI Stuff?” 80,000 Hours. Accessed September 16, 2020. https://80000hours.org/podcast/episodes/ben-garfinkel-classic-ai-risk-arguments/.

Garfinkel, Ben. How Sure Are We about This AI Stuff? (talk) | EA Global: London 2018, 2019. https://www.youtube.com/watch?v=E8PGcoLDjVk. Also in blog form: https://ea.greaterwrong.com/posts/9sBAW3qKppnoG3QPq/ben-garfinkel-how-sure-are-we-about-this-ai-stuff

LeCun, Yann, and Anthony Zador. “Don’t Fear the Terminator.” Scientific American Blog Network. Accessed December 9, 2021. https://blogs.scientificamerican.com/observations/dont-fear-the-terminator/.

Yudkowsky, Eliezer, and Robin Hanson. “The Hanson-Yudkowsky AI-Foom Debate – LessWrong.” Accessed August 6, 2022. https://www.lesswrong.com/tag/the-hanson-yudkowsky-ai-foom-debate.

See also

• List of sources arguing for existential risk from AI
• Is AI an existential threat to humanity?

Notes

Published 6 Aug 2022

This page is incomplete, under active work and may be updated soon.

This is a bibliography of pieces arguing that AI poses an existential risk.

List

Adamczewski, Tom. “A Shift in Arguments for AI Risk.” Fragile Credences. Accessed October 20, 2020. https://fragile-credences.github.io/prioritising-ai/.

Amodei, Dario, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. “Concrete Problems in AI Safety.” ArXiv:1606.06565 [Cs], July 25, 2016. http://arxiv.org/abs/1606.06565.

Bensinger, Rob, Eliezer Yudkowsky, Richard Ngo, So8res, Holden Karnofsky, Ajeya Cotra, Carl Shulman, and Rohin Shah. “2021 MIRI Conversations – LessWrong.” Accessed August 6, 2022. https://www.lesswrong.com/s/n945eovrA3oDueqtq.

Bostrom, N., Superintelligence, Oxford University Press, 2014.

Carlsmith, Joseph. “Is Power-Seeking AI an Existential Risk? [Draft].” Open Philanthropy Project, April 2021. https://docs.google.com/document/d/1smaI1lagHHcrhoi6ohdq3TYIZv0eNWWZMPEy8C8byYg/edit?usp=embed_facebook.

Christian, Brian. The Alignment Problem: Machine Learning and Human Values. W. W. Norton & Company, 2021.

Christiano, Paul. “What Failure Looks Like.” AI Alignment Forum (blog), March 17, 2019. https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like.

Dai, Wei. “Comment on Disentangling Arguments for the Importance of AI Safety – LessWrong.” Accessed December 9, 2021. https://www.lesswrong.com/posts/JbcWQCxKWn3y49bNB/disentangling-arguments-for-the-importance-of-ai-safety.

Garfinkel, Ben, Miles Brundage, Daniel Filan, Carrick Flynn, Jelena Luketina, Michael Page, Anders Sandberg, Andrew Snyder-Beattie, and Max Tegmark. “On the Impossibility of Supersized Machines.” ArXiv:1703.10987 [Physics], March 31, 2017. http://arxiv.org/abs/1703.10987.

Hubinger, Evan, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. “Risks from Learned Optimization in Advanced Machine Learning Systems,” June 5, 2019. https://arxiv.org/abs/1906.01820v3.

Ngo, Richard. “Thinking Complete: Disentangling Arguments for the Importance of AI Safety.” Thinking Complete (blog), January 21, 2019. http://thinkingcomplete.blogspot.com/2019/01/disentangling-arguments-for-importance.html. (Also LessWrong and the Alignment Forum, with relevant comment threads.)

Ngo, Richard. “AGI Safety from First Principles,” September 28, 2020. https://www.lesswrong.com/s/mzgtmmTKKn5MuCzFJ.

Ord, Toby. The Precipice: Existential Risk and the Future of Humanity. Illustrated Edition. New York: Hachette Books, 2020.

Piper, Kelsey. “The Case for Taking AI Seriously as a Threat to Humanity.” Vox, December 21, 2018. https://www.vox.com/future-perfect/2018/12/21/18126576/ai-artificial-intelligence-machine-learning-safety-alignment.

Russell, Stuart. Human Compatible: Artificial Intelligence and the Problem of Control. Viking, 2019.

Turner, Alexander Matt, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. “Optimal Policies Tend to Seek Power.” ArXiv:1912.01683 [Cs], December 3, 2021. http://arxiv.org/abs/1912.01683.

Yudkowsky, Eliezer. “Artificial Intelligence as a Positive and Negative Factor in Global Risk.” In Global Catastrophic Risks, edited by Nick Bostrom and Milan M. Ćirković, 46. New York, n.d. https://intelligence.org/files/AIPosNegFactor.pdf.

Yudkowsky, Eliezer, Rob Bensinger, and So8res. “2022 MIRI Alignment Discussion – LessWrong.” Accessed August 6, 2022. https://www.lesswrong.com/s/v55BhXbpJuaExkpcD.

Yudkowsky, Eliezer, and Robin Hanson. “The Hanson-Yudkowsky AI-Foom Debate – LessWrong.” Accessed August 6, 2022. https://www.lesswrong.com/tag/the-hanson-yudkowsky-ai-foom-debate.

See also

• List of sources arguing against existential risk from AI
• Is AI an existential threat to humanity?

Notes

Economic incentives to deploy AI systems seem unlikely to be reliably eliminated by knowledge that those AI systems pose an existential risk.

Details

Reasons for people with normal values to be incentivized to bring about human extinction

One might reason that if advanced AI systems had such malign preferences as to pose a substantial existential risk to humanity, then almost nobody would be motivated to deploy such systems. This reasoning fails because a) the personal cost of incurring such risks can be small relative to the benefits, even for a person who cares unusually much about the future of humanity, and b) because coordination problems reduce the counterfactual downside of taking such risks further still.

a) Externalities, or, personal costs of incurring extinction risks can be small

A person might strongly disprefer human extinction, and yet want to take an action which contributes to existential risk if:

• the action only incurs a risk of extinction, or the risk is in the distant future, so that taking the risk does not negate other benefits accruing from the action
• the person does not value the survival of humanity (or a slightly higher chance of the survival of humanity) radically more than their other interests
• Using the AI system would materially benefit the person

A different way of describing this issue is that even if people disprefer causing human extinction, since most of the costs of human extinction fall on others, any particular person making a choice that risks humanity for private gain will take more risk than is socially optimal.

Examples:

1. A person faces the choice of using an AI lawyer system for $100, or a human lawyer for $10,000. They believe that the AI lawyer system is poorly motivated and agentic, and that movement of resources to such systems is gradually disempowering humanity, which they care about. Nonetheless, their action only contributes a small amount to this problem, and they are not willing to raise tens of thousands of dollars to avoid that harm.
2. A person faces the choice of deploying the largest scale model to date, or trying to call off the project. They believe that at some scale, a model will become an existential threat to humanity. However they are very unsure at what scale, and estimate that the model in front of them only has a 1% chance of being the dangerous one. They value the future of humanity a lot, but not ten times more than their career, and calling off the project would be a huge hit, for only 1% of the future of humanity.

b) Coordination problems

Coordination problems can make the above situation more common: if a person believes that if they don’t take an action that incurs a cost to others, then the same action will be taken by others and the cost incurred anyway, then the real downside to incurring that cost is even smaller.

If many people are independently choosing whether to use dangerously misaligned AI systems, and all believe that they will anyway be used enough to destroy humanity in the long run, then even people who wouldn’t have wanted to deploy such systems if they were the sole decision-maker have reason to deploy them.

Practical plausibility

Situations where extinction risk is worth incurring for individuals seem likely to be common in a world where advanced AI systems in fact pose extinction risk. Reasons to expect this include:

1. In a large class of scenarios, AI x-risk is anticipated to take at least decades to transpire, after the decisions that brought about the risk
2. People commonly weight benefits to strangers in the future substantially lower than benefits to themselves immediately¹
3. Many people claim to not intrinsically care about the long run existence of humanity.
4. AI systems with objectives that roughly match people’s short term goals seem likely to be beneficial for those goals in the short term, while potentially costly to society at large in the long term. (e.g. an AI system which aggressively optimizes for mining coal may be economically useful to humans running a coal-mining company in the short term, but harmful to those and other humans in the long run, if it gains the resources and control to mine coal to a destructive degree.)

Notes

Katja Grace, Aug 4 2022

AI Impacts just finished collecting data from a new survey of ML researchers, as similar to the 2016 one as practical, aside from a couple of new questions that seemed too interesting not to add.

This page reports on it preliminarily, and we’ll be adding more details there. But so far, some things that might interest you:

• 37 years until a 50% chance of HLMI according to a complicated aggregate forecast (and biasedly not including data from questions about the conceptually similar Full Automation of Labor, which in 2016 prompted strikingly later estimates). This 2059 aggregate HLMI timeline has become about eight years shorter in the six years since 2016, when the aggregate prediction was 2061, or 45 years out. Note that all of these estimates are conditional on “human scientific activity continu[ing] without major negative disruption.”
• P(extremely bad outcome)=5% The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%. This is the same as it was in 2016 (though Zhang et al 2022 found 2% in a similar but non-identical question). Many respondents put the chance substantially higher: 48% of respondents gave at least 10% chance of an extremely bad outcome. Though another 25% put it at 0%.
• Explicit P(doom)=5-10% The levels of badness involved in that last question seemed ambiguous in retrospect, so I added two new questions about human extinction explicitly. The median respondent’s probability of x-risk from humans failing to control AI¹ was 10%, weirdly more than median chance of human extinction from AI in general², at 5%. This might just be because different people got these questions and the median is quite near the divide between 5% and 10%. The most interesting thing here is probably that these are both very high—it seems the ‘extremely bad outcome’ numbers in the old question were not just catastrophizing merely disastrous AI outcomes.
• Support for AI safety research is up: 69% of respondents believe society should prioritize AI safety research “more” or “much more” than it is currently prioritized, up from 49% in 2016.
• The median respondent thinks there is an “about even chance” that an argument given for an intelligence explosion is broadly correct. The median respondent also believes machine intelligence will probably (60%) be “vastly better than humans at all professions” within 30 years of HLMI, and that the rate of global technological improvement will probably (80%) dramatically increase (e.g., by a factor of ten) as a result of machine intelligence within 30 years of HLMI.
• Years/probabilities framing effect persists: if you ask people for probabilities of things occurring in a fixed number of years, you get later estimates than if you ask for the number of years until a fixed probability will obtain. This looked very robust in 2016, and shows up again in the 2022 HLMI data. Looking at just the people we asked for years, the aggregate forecast is 29 years, whereas it is 46 years for those asked for probabilities. (We haven’t checked in other data or for the bigger framing effect yet.)
• Predictions vary a lot. Pictured below: the attempted reconstructions of people’s probabilities of HLMI over time, which feed into the aggregate number above. There are few times and probabilities that someone doesn’t basically endorse the combination of.
• You can download the data here (slightly cleaned and anonymized) and do your own analysis. (If you do, I encourage you to share it!)

The survey had a lot of questions (randomized between participants to make it a reasonable length for any given person), so this blog post doesn’t cover much of it. A bit more is on the page and more will be added.

Thanks to many people for help and support with this project! (Many but probably not all listed on the survey page.)

Cover image: Probably a bootstrap confidence interval around an aggregate of the above forest of inferred gamma distributions, but honestly everyone who can be sure about that sort of thing went to bed a while ago. So, one for a future update. I have more confidently held views on whether one should let uncertainty be the enemy of putting things up.

Published 3 August 2022; last updated 3 August 2022

This page is in progress. It includes results that are preliminary and have a higher than usual chance of inaccuracy and suboptimal formatting. It is missing many results.

The 2022 Expert Survey on Progress in AI (2022 ESPAI) is a survey of machine learning researchers that AI Impacts ran in June-August 2022.

Details

Background

The 2022 ESPAI is a rerun of the 2016 Expert Survey on Progress in AI that researchers at AI Impacts previously collaborated on with others. Almost all of the questions were identical, and both surveyed authors who recently published in NeurIPS and ICML, major machine learning conferences.

Zhang et al ran a followup survey in 2019 (published in 2022)¹ however they reworded or altered many questions, including the definitions of HLMI, so much of their data is not directly comparable to that of the 2016 or 2022 surveys, especially in light of large potential for framing effects observed.

Methods

Population

We contacted approximately 4271 researchers who published at the conferences NeurIPS or ICML in 2021. These people were selected by taking all of the authors at those conferences and randomly allocating them between this survey and a survey being run by others. We then contacted those whose email addresses we could find. We found email addresses in papers published at those conferences, in other public data, and in records from our previous survey and Zhang et al 2022. We received 738 responses, some partial, for a 17% response rate.

Participants who previously participated in the the 2016 ESPAI or Zhang et al surveys received slightly longer surveys, and received questions which they had received in past surveys (where random subsets of questions were given), rather than receiving newly randomized questions. This was so that they could also be included in a ‘matched panel’ survey, in which we contacted all researchers who completed the 2016 ESPAI or Zhang et al surveys, to compare responses from exactly the same samples of researchers over time. These surveys contained additional questions matching some of those in the Zhang et al survey.

Contact

We invited the selected researchers to take the survey via email. We accepted responses between June 12 and August 3, 2022.

Questions

The full list of survey questions is available below, as exported from the survey software. The export does not preserve pagination, or data about survey flow. Participants received randomized subsets of these questions, so the survey each person received was much shorter than that shown below.

A small number of changes were made to questions since the 2016 survey (list forthcoming).

Definitions

‘HLMI’ was defined as follows:

The following questions ask about ‘high–level machine intelligence’ (HLMI). Say we have ‘high-level machine intelligence’ when unaided machines can accomplish every task better and more cheaply than human workers. Ignore aspects of tasks for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility, not adoption.

Results

Data

The anonymized dataset is available here.

Summary of results

• The aggregate forecast time to a 50% chance of HLMI was 37 years, i.e. 2059 (not including data from questions about the conceptually similar Full Automation of Labor, which in 2016 received much later estimates). This timeline has become about eight years shorter in the six years since 2016, when the aggregate prediction put 50% probability at 2061, i.e. 45 years out. Note that these estimates are conditional on “human scientific activity continu[ing] without major negative disruption.”
• The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%. This is the same as it was in 2016 (though Zhang et al 2022 found 2% in a similar but non-identical question). Many respondents were substantially more concerned: 48% of respondents gave at least 10% chance of an extremely bad outcome. But some much less concerned: 25% put it at 0%.
• The median respondent believes society should prioritize AI safety research “more” than it is currently prioritized. Respondents chose from “much less,” “less,” “about the same,” “more,” and “much more.” 69% of respondents chose “more” or “much more,” up from 49% in 2016.
• The median respondent thinks there is an “about even chance” that a stated argument for an intelligence explosion is broadly correct. 54% of respondents say the likelihood that it is correct is “about even,” “likely,” or “very likely” (corresponding to probability >40%), similar to 51% of respondents in 2016. The median respondent also believes machine intelligence will probably (60%) be “vastly better than humans at all professions” within 30 years of HLMI, and the rate of global technological improvement will probably (80%) dramatically increase (e.g., by a factor of ten) as a result of machine intelligence within 30 years of HLMI.

High-level machine intelligence timelines

The aggregate forecast time to HLMI was 36.6 years, conditional on “human scientific activity continu[ing] without major negative disruption.” and considering only questions using the HLMI definition. We have not yet analyzed data about the conceptually similar Full Automation of Labor (FAOL), which in 2016 prompted much later timeline estimates. Thus this timeline figure is expected to be low relative to an overall estimate from this survey.

This aggregate is the 50th percentile date in an equal mixture of probability distributions created by fitting a gamma distribution to each person’s answers to three questions either about the probability of HLMI occurring by a given year or the year at which a given probability would obtain.

Impacts of HLMI

Question

Participants were asked:

Assume for the purpose of this question that HLMI will at some point exist. How positive or negative do you expect the overall impact of this to be on humanity, in the long run? Please answer by saying how probable you find the following kinds of impact, with probabilities adding to 100%:

______ Extremely good (e.g. rapid growth in human flourishing) (1)

______ On balance good (2)

______ More or less neutral (3)

______ On balance bad (4)

______ Extremely bad (e.g. human extinction) (5)

Answers

Medians:

• Extremely good: 10%
• On balance good: 20%
• More or less neutral: 15%
• On balance bad: 10%
• Extremely bad: 5%

Means:

• Extremely good: 24%
• On balance good: 26%
• More or less neutral: 18%
• On balance bad: 17%
• Extremely bad: 14%

Intelligence explosion

Probability of dramatic technological speedup

Question

Participants were asked:

Assume that HLMI will exist at some point. How likely do you then think it is that the rate of global technological improvement will dramatically increase (e.g. by a factor of ten) as a result of machine intelligence:

Within two years of that point?       ___% chance

Within thirty years of that point?    ___% chance

Answers

Median P(within two years) = 20% (20% in 2016)

Median P(within thirty years) = 80% (80% in 2016)

Probability of superintelligence

Question

Participants were asked:

Assume that HLMI will exist at some point. How likely do you think it is that there will be machine intelligence that is vastly better than humans at all professions (i.e. that is vastly more capable or vastly cheaper):

Within two years of that point?       ___% chance

Within thirty years of that point?    ___% chance

Answers

Median P(…within two years) = 10% (10% in 2016)

Median P(…within thirty years) = 60% (50% in 2016)

Chance that the intelligence explosion argument is about right

Question

Participants were asked:

Some people have argued the following:

If AI systems do nearly all research and development, improvements in AI will accelerate the pace of technological progress, including further progress in AI.

Over a short period (less than 5 years), this feedback loop could cause technological progress to become more than an order of magnitude faster.

How likely do you find this argument to be broadly correct?

• Quite unlikely (0-20%)
• Unlikely (21-40%)
• About even chance (41-60%)
• Likely (61-80%)
• Quite likely (81-100%)

Answers

• 20% quite unlikely (25% in 2016)
• 26% unlikely (24% in 2016)
• 21% about even chance (22% in 2016)
• 26% likely (17% in 2016)
• 7% quite likely (12% in 2016)

Existential risk

In an above question, participants’ credence in “extremely bad” outcomes of HLMI have median 5% and mean 14%. To better clarify what participants mean by this, we also asked a subset of participants one of the following questions, which did not appear in the 2016 survey:

Extinction from AI

Participants were asked:

What probability do you put on future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species? 

Answers

Median 5%.

Extinction from human failure to control AI

Participants were asked:

What probability do you put on human inability to control future advanced AI systems causing human extinction or similarly permanent and severe disempowerment of the human species?

Answers

Median 10%.

This question is more specific and thus necessarily less probable than the previous question, but it was given a higher probability at the median. This could be due to noise (different random subsets of respondents received the questions, so there is no logical requirement that their answers cohere), or due to the representativeness heuristic.

Safety

General safety

Question

Participants were asked:

Let ‘AI safety research’ include any AI-related research that, rather than being primarily aimed at improving the capabilities of AI systems, is instead primarily aimed at minimizing potential risks of AI systems (beyond what is already accomplished for those goals by increasing AI system capabilities).

Examples of AI safety research might include:

• Improving the human-interpretability of machine learning algorithms for the purpose of improving the safety and robustness of AI systems, not focused on improving AI capabilities
• Research on long-term existential risks from AI systems
• AI-specific formal verification research
• Policy research about how to maximize the public benefits of AI

How much should society prioritize AI safety research, relative to how much it is currently prioritized?

• Much less
• Less
• About the same
• More
• Much more

Answers

• Much less: 2% (5% in 2016)
• Less: 9% (8% in 2016)
• About the same: 20% (38% in 2016)
• More: 35% (35% in 2016)
• Much more: 33% (14% in 2016)

69% of respondents think society should prioritize AI safety research more or much more, up from 49% in 2016.

Stuart Russell’s problem

Question

Participants were asked:

Stuart Russell summarizes an argument for why highly advanced AI might pose a risk as follows:

The primary concern [with highly advanced AI] is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken […]. Now we have a problem:

1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.

2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.

A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.  This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want.

Do you think this argument points at an important problem?

• No, not a real problem.
• No, not an important problem.
• Yes, a moderately important problem.
• Yes, a very important problem.
• Yes, among the most important problems in the field.

How valuable is it to work on this problem today, compared to other problems in AI?

• Much less valuable
• Less valuable
• As valuable as other problems
• More valuable
• Much more valuable

How hard do you think this problem is compared to other problems in AI?

• Much easier
• Easier
• As hard as other problems
• Harder
• Much harder

Answers

Importance:

• No, not a real problem: 4%
• No, not an important problem: 14%
• Yes, a moderately important problem: 24%
• Yes, a very important problem: 37%
• Yes, among the most important problems in the field: 21%

Value today:

• Much less valuable: 10%
• Less valuable: 30%
• As valuable as other problems: 33%
• More valuable: 19%
• Much more valuable: 8%

Hardness:

• Much easier: 5%
• Easier: 9%
• As hard as other problems: 29%
• Harder: 31%
• Much harder: 26%

Contributions

The survey was run by Katja Grace and Ben Weinstein-Raun. Data analysis was done by Zach Stein-Perlman and Ben Weinstein-Raun. This page was written by Zach Stein-Perlman and Katja Grace.

We thank many colleagues and friends for help, discussion and encouragement, including John Salvatier, Nick Beckstead, Howie Lempel, Joe Carlsmith, Leopold Aschenbrenner, Ramana Kumar, Jimmy Rintjema, Jacob Hilton, Ajeya Cotra, Scott Siskind, Chana Messinger, Noemi Dreksler, and Baobao Zhang.

We also thank the expert participants who spent time sharing their impressions with us, including:

Michał Zając
Morten Goodwin
Yue Sun
Ningyuan Chen
Egor Kostylev
Richard Antonello
Elia Turner
Andrew C Li
Zachary Markovich
Valentina Zantedeschi
Michael Cooper
Thomas A Keller
Marc Cavazza
Richard Vidal
David Lindner
Xuechen (Chen) Li
Alex M. Lamb
Tristan Aumentado-Armstrong
Ferdinando Fioretto
Alain Rossier
Wentao Zhang
Varun Jampani
Derek Lim
Muchen Li
Cong Hao
Yao-Yuan Yang
Linyi Li
Stéphane D’Ascoli
Lang Huang
Maxim Kodryan
Hao Bian
Orestis Paraskevas
David Madras
Tommy Tang
Li Sun
Stefano V Albrecht
Tristan Karch
Muhammad A Rahman
Runtian Zhai
Benjamin Black
Karan Singhal
Lin Gao
Ethan Brooks
Cesar Ferri
Dylan Campbell
Xujiang Zhao
Jack Parker-Holder
Michael Norrish
Jonathan Uesato
Yang An
Maheshakya Wijewardena
Ulrich Neumann
Lucile Ter-Minassian
Alexander Matt turner
Subhabrata Dutta
Yu-Xiang Wang
Yao Zhang
Joanna Hong
Yao Fu
Wenqing Zheng
Louis C Tiao
Hajime Asama
Chengchun Shi
Moira R Dillon
Yisong Yue
Aurélien Bellet
Yin Cui
Gang Hua
Jongheon Jeong
Martin Klissarov
Aran Nayebi
Fabio Maria Carlucci
Chao Ma
Sébastien Gambs
Rasoul Mirzaiezadeh
Xudong Shen
Julian Schrittwieser
Adhyyan Narang
Fuxin Li
Linxi Fan
Johannes Gasteiger
Karthik Abinav Sankararaman
Patrick Mineault
Akhilesh Gotmare
Jibang Wu
Mikel Landajuela
Jinglin Liu
Qinghua Hu
Noah Siegel
Ashkan Khakzar
Nathan Grinsztajn
Julian Lienen
Xiaoteng Ma
Mohamad H Danesh
Ke ZHANG
Feiyu Xiong
Wonjae Kim
Michael Arbel
Piotr Skowron
Lê-Nguyên Hoang
Travers Rhodes
Liu Ziyin
Hossein Azizpour
Karl Tuyls
Hangyu Mao
Yi Ma
Junyi Li
Yong Cheng
Aditya Bhaskara
Xia Li
Danijar Hafner
Brian Quanz
Fangzhou Luo
Luca Cosmo
Scott Fujimoto
Santu Rana
Michael Curry
Karol Hausman
Luyao Yuan
Samarth Sinha
Matthew McLeod
Hao Shen
Navid Naderializadeh
Alessio Micheli
Zhenbang You
Van Huy Vo
Chenyang Wu
Thanard Kurutach
Vincent Conitzer
Chuang Gan
Chirag Gupta
Andreas Schlaginhaufen
Ruben Ohana
Luming Liang
Marco Fumero
Paul Muller
Hana Chockler
Ming Zhong
Jiamou Liu
Sumeet Agarwal
Eric Winsor
Ruimeng Hu
Changjian Shui
Yiwei Wang
Joey Tianyi Zhou
Anthony L. Caterini
Guillermo Ortiz-Jimenez
Iou-Jen Liu
Jiaming Liu
Michael Perlmutter
Anurag Arnab
Ziwei Xu
John Co-Reyes
Aravind Rajeswaran
Roy Fox
Yong-Lu Li
Carl Yang
Divyansh Garg
Amit Dhurandhar
Harris Chan
Tobias Schmidt
Robi Bhattacharjee
Marco Nadai
Reid McIlroy-Young
Wooseok Ha
Jesse Mu
Neale Ratzlaff
Kenneth Borup
Binghong Chen
Vikas Verma
Walter Gerych
Shachar Lovett
Zhengyu Zhao
Chandramouli Chandrasekaran
Richard Higgins
Nicholas Rhinehart
Blaise Agüera Y Arcas
Santiago Zanella-Beguelin
Dian Jin
Scott Niekum
Colin A. Raffel
Sebastian Goldt
Yali Du
Bernardo Subercaseaux
Hui Wu
Vincent Mallet
Ozan Özdenizci
Timothy Hospedales
Lingjiong Zhu
Cheng Soon Ong
Shahab Bakhtiari
Huan Zhang
Banghua Zhu
Byungjun Lee
Zhenyu Liao
Adrien Ecoffet
Vinay Ramasesh
Jesse Zhang
Soumik Sarkar
Nandan Kumar Jha
Daniel S Brown
Neev Parikh
Chen-Yu Wei
David K. Duvenaud
Felix Petersen
Songhua Wu
Huazhu Fu
Roger B Grosse
Matteo Papini
Peter Kairouz
Burak Varici
Fabio Roli
Mohammad Zalbagi Darestani
Jiamin He
Lys Sanz Moreta
Xu-Hui Liu
Qianchuan Zhao
Yulia Gel
Jan Drgona
Sajad Khodadadian
Takeshi Teshima
Igor T Podolak
Naoya Takeishi
Man Shun Ang
Mingli Song
Jakub Tomczak
Lukasz Szpruch
Micah Goldblum
Graham W. Taylor
Tomasz Korbak
Maheswaran Sathiamoorthy
Lan-Zhe Guo
Simone Fioravanti
Lei Jiao
Davin Choo
Kristy Choi
Varun Nair
Rayana Jaafar
Amy Greenwald
Martin V. Butz
Aleksey Tikhonov
Samuel Gruffaz
Yash Savani
Rui Chen
Ke Sun

We thank FTX Future Fund for funding this project.

Suggested citation

Zach Stein-Perlman, Benjamin Weinstein-Raun, Katja Grace, “2022 Expert Survey on Progress in AI.” AI Impacts, 3 Aug. 2022. https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/.

Notes

AI Impacts is beginning a serious hiring round (see here for job postings), so I’d like to explain a bit why it has been my own best guess at the highest impact place for me to work for me. (As in, this is a personal blog post by Katja on the AI Impacts blog, not some kind of officialesque missive from the organization.)

But first—

What is AI Impacts?

AI Impacts is a few things:

1. An online library of best-guess answers to questions about the future of AI. Including big questions, like ‘how likely is a sudden jump in AI progress at around human-level performance?’, and sub-questions informing those answers (‘are discontinuities common in technological trends?’), and sub-sub questions (‘did penicillin cause any discontinuous changes in syphilis trends?’), and so on. Each page ideally has a high-level conclusion at the top, and reasoning supporting it below, which will often call on the conclusions of other pages. These form something like a set of trees, with important, hard, decision-relevant questions at the root and low-level, tractable, harder-to-use-on-their-own questions at the leaves. This isn’t super obvious at the moment, because a lot of the trees are very incomplete, but that’s the basic idea.
2. A research group focused on finding such answers, through a mixture of original research and gathering up that which has been researched by others.
3. A blog on these topics, for more opinionated takes, conversational guides to the research, updates, and other things that don’t fit in the main library (like this!).
4. A locus of events for people interested in this kind of research, e.g. dinners and workshops, a Slack with other researchers, online coffees.

Why think working on AI Impacts is among the best things to do?

1. AI risk looks like a top-notch cause area

It seems plausible to me that advanced AI poses a substantial risk to humanity’s survival. I don’t think this is clear, but I do think there’s enough evidence that it warrants a lot of attention. I hope to write more about this, see here for recent discussion. Furthermore, I don’t know of other similarly serious risks (see Ord’s The Precipice for a review), or of other intervention areas that look clearly more valuable than reducing existential risk to humanity.

I actually also think AI risk is a potentially high-impact area to work (for a little while at least) if AI isn’t a huge existential risk to humanity, because so many capable and well-intentioned people are dedicating themselves to it. Demonstrating that it wasn’t that bad could redirect mountains of valuable effort to real problems.

2. Understanding the situation beats intervening on the current margin

Within the area of mitigating AI risk, there are several broad classes of action being taken. Technical safety research focuses on building AI that won’t automatically cause catastrophe. AI Governance focuses on maneuvering the policy landscape to lower risk. These are both kinds of intervention: ‘intervening’ is a meta-category, and the other main meta-category in my mind is ‘understanding the situation’. My own best guess is that on the current margin, ‘understanding the situation’ is a better place for an additional person with general skills than any particular intervening that I know of. (Or maybe it’s only almost as good—I flip-flop, but it doesn’t really matter much: the important thing is that for some large part of the space of people and their skills and characteristics, it seems better.)

By ‘understanding the situation’, I mean for instance working toward better answers to questions like these:

• Fast or slow takeoff?
• What concrete kinds of problems might destroy humanity? E.g. single AI god intentionally murders everyone with nanotech vs. large economy gradually drifts away from human comprehension or control?
• Is there a single relevant ‘deployment’?
• If so, what does it look like?
• Would we be safe if AI systems weren’t ‘agentic’?
• Do not-intentionally-agentic things readily become agentic things? Under what circumstances?
• How fast would an intelligence explosion go?
• Is it possible to describe a plausible future where things go well? (Is it possible to describe a plausible future where things go badly?)

Carrying out any particular intervention also involves a lot of ‘understanding the situation’, but I think this is often at a different level. For instance, if you decide to intervene by trying to get AI labs to collaborate with each other, you might end up accruing better models of how people at AI projects interact socially, how decisions are made, how running events works, and so on, because these things are part of the landscape between you and your instrumental goal: improving collaboration between AI projects. You probably also learn about things around you, like what kinds of AI projects people are doing. But you don’t get to learn much at all about how the achievement of your goal affects the future of AI. (I fear that in general this situation means you can end up lumbering forward blindly while thinking you can see, because you are full of specific concrete information—the intricacies of the steering wheel distracting you from the dense fog on the road.) There are some exceptions to this. For instance, I expect some technical work to be pretty enlightening about the nature of AI systems, which is directly relevant to how the development of better AI systems will play out. For instance, mesa-optimization seems like a great contribution to ‘understanding the situation’ which came out of a broadly intervention-oriented organization.

It is that kind of understanding the situation—understanding what will happen with AI and its effects on society, under different interventions—that I think deserves way more attention.

Why do I think understanding the situation is better than intervening? Of course in general, both are great. Intervening is generally necessary for achieving anything, and understanding the situation is arguably necessary for intervening well. (The intense usefulness of understanding the situation for achieving your goals in most situations is exactly the reason one might be concerned about AI to begin with.) So in general, you want a combination of understanding the situation and intervening. The question is how valuable the two are on the current margin.

My guess: understanding the situation is better. Which is to say, I think a person with a subjectively similar level of skill at everything under consideration will add more value via improving everyone’s understanding of the situation by one person’s worth of effort than they would by adding one person’s worth of effort to pursuing the seemingly best intervention.

Here are a few things influencing this guess:

1. A basic sense that our understanding of the situation is low My impression when talking to people working on AI risk is that they often don’t feel that they understand the situation very well. There are major disagreements about what kind of basic scenario we are expecting. The going explanations for why there will be human extinction at all seem to vary across time and between people. Offers to try to clarify are generally met with enthusiasm. These things don’t seem great as signs about whether we understand the situation well enough to take useful action.
2. It’s easy to think of specific questions for which updated answers would change the value of different interventions. Here are a few examples off the top of my head of questions, answers, and strategies that would seem to be relatively favored by those answers:
   • Does AI pose a substantial risk of human extinction?
     Yes: work on AI risk instead of other EA causes and other non-emergency professions. Show a good case for this to large numbers of people who aren’t thinking about it and try to change views within the AI community and public about the appropriate degree of caution for relevant AI work.
     No: work on something more valuable, support AI progress
   • When will relevantly advanced AI be developed?
     5 years: plan for what specific actors should do in a situation much like our current one and talk to them about doing it; build relationships with likely actors; try to align systems much like our current AI systems.
     20 years: more basic time-consuming alignment research; movement building; relationship building with institutions rather than people.
     100 years: avert risks from narrow or weak AI and other nearer technologies, even more basic alignment research, improve society’s general institutions for responding to risks like this, movement building directed at broader issues that people won’t get disillusioned with over that long a period (e.g. ‘responding to technological risks’ vs. AI specifically).
   • How fast is the progression to superhumanly powerful AI likely to be?
     Before you know it: searching for technical solutions that can be proven to entirely solve the problem before it arises (even if you are unlikely to find any), social coordination to avoid setting off such an event.
     Weeks: Immediate-response contingency plans.
     Years: Fast-response contingency plans; alignment plans that would require some scope for iteration.
     Decades: Expect to improve safety through more normal methods of building systems, observing them, correcting, iterating. ‘Soft’ forces like regulations, broadscale understanding of the problems, cooperation initiatives. Systems that are incrementally safer but not infinitely safer.
3. Broad heuristic value of seeing
   When approaching a poorly understood danger down a dark corridor, I feel like even a small amount of light is really good. Good for judging whether you are facing a dragon or a cliff, good for knowing when you are getting close to it so you can ready your sword (or your ropes, as the case may be), good for telling how big it is. But even beyond those pre-askable questions, I expect the details of the fight (or climb) to go much better if you aren’t blind. You will be able to strike well, and jump out of the way well, and generally have good feedback about your micro-actions and local risks.So I don’t actually trust tallying up possible decision changes as in the last point, that much. If you told me that we had reasoned through the correct course of action for dragons, and cliff faces, and tar pits, and alternate likely monsters, and decided they were basically the same, I’d persist in being willing to pay a lot to be able to see.

   Applied to AI strategy: understanding the situation both lets you choose interventions that might help, and having chosen an intervention, probably helps you make smaller choices within that intervention well, such that the intervention hits its target.

   I think another part of the value here is that very abstract reasoning about complicated situations seems untrustworthy (especially when it isn’t actually formal), and I expect getting more data and working out more details to generally engage people’s concrete thinking better, and for that to be helpful.

4. Large multipliers available It’s not that hard to imagine the work of one person’s year substantially redirecting way more than one person-year worth of time or money. Intuitively the chance of this seems high enough to make it a good prospect.
5. We have a really long list of projects to do. About a hundred that we have bothered to write down, though they vary in tractability. It isn’t hard to find important topics which have received little thorough research. On the current margin, it looks to me like an additional competent person can expect to do useful research.
6. If I were to work on a direct intervention in this space, I would feel fairly unsure about whether it would be helpful even if it succeeded in its goals.
7. Understanding the situation has way fewer people than intervening: I haven’t measured this carefully, but my guess is that between ten and a hundred times as much labor goes into intervening than understanding the situation. I’m not sure what the division should be, but intuitively this seems too lopsided.
8. Assumptions do not seem solid: it’s arguably not very hard to find points that people are bringing to the table that, upon empirical investigation, seem contrary to the evidence. Einstein and idiots are probably not really right next to each other on natural objective measures of intelligence, as far as I can tell. Qualitatively cool technologies do not generally cause large discontinuities in any particular metrics. Not empirical, but many of the arguments I’ve heard for expecting discontinuous progress at around the time of human-level AI just don’t make much sense to me.
9. The ‘understanding the situation’ project is at a pretty unsophisticated stage, compared with intervening projects, according to my assessment anyway. That suggests a mistake, in the same way that navigating an expensive car using divining rods because you don’t have a GPS or map suggests some kind of misallocation of investments.
10. I think people overestimate the effort put into understanding the situation, because there is a decent amount of talking about it at parties and blogging about it.
11. There are people positioned to make influential choices if they knew what to do asking for help in assessing the situation (e.g. Holden of Open Phil, people with policy influence, philanthropists).

People sometimes ask if we might be scraping the barrel on finding research to do in this space, I guess because quite a few people have prolifically opined on it over numerous years, and things seem pretty uncertain. I think that radically under-imagines what understanding, or an effort dedicated to understanding, could look like. Like, we haven’t gotten as far as making sure that the empirical claims being opined about are solid, whereas a suitable investment for a major international problem that you seriously need to solve should probably look more like the one we see for climate change. Climate change is a less bad and arguably easier to understand problem than AI risk, and the ‘understanding the situation’ effort there looks like an army of climate scientists working for decades. And they didn’t throw up their hands and say things were too uncertain and they had run out of things to think about after twenty climate hobbyists had thought about it for a bit. There is a big difference between a vibrant corner of the blogosphere and a serious research effort.

3. Different merits of different projects

Ok, so AI risk is the most impactful field to my knowledge, and within AI risk I claim that the highest impact work is on understanding the situation¹. This is reason to work at AI Impacts, and also reason to work at Open Philanthropy, FHI, Metaculus, as an independent scholar, in academia, etc. Probably who should do which depends on the person and their situation. Here are some things AI Impacts is about, and axes on which we have locations:

• Openness, broad comprehensibility and reasoning transparency: our goal is to make an online repository of reasoning around these topics, so we prioritize publishing work (vs. distributing it privately to smaller networks of people), and legibility. There can be research that is better done privately, but such research is not our project. We hope to describe the basis for our conclusions well enough that a non-expert reader can verify the reasoning.
• Modularity and question decomposition: AI Impacts is intended to be something like a group of hierarchical trees of modular conclusions, that can be referred to and questioned in a relatively clean way. We try to roughly have a page for each important conclusion, though things get complicated sometimes, and it is easier to have a short list of them. I think this kind of structure for understanding a complex topic is a promising one, relative to for instance less structured piles of prose. I expect this to make research more re-purposeable, clear, updateable, navigable, and amenable to tight feedback loops. Echoing this structure, we try to answer big questions by breaking them into smaller questions, until we have tractable questions.
• Eye on the prize vs. exploratory wandering: there are many research questions that are interesting and broadly shed light on the future of AI, and following one’s curiosity can be a good strategy. Nonetheless we especially strive to answer the questions that more help with answering important high-level questions. While researchers have a decent amount of freedom, we expect people to be contributing to filling in the gaps in this shared structure of understanding that we are building.
• Back of the envelopes expanding into arbitrarily detailed investigation: in places like academia, it seems normal to work on a project for many months or years, and to finish with something polished. Part of the idea with AI Impacts is to look out for questions that can be substantially clarified by a day and a back of the envelope calculation, to not put in more research than needed, and to iterate at more depth when relevant. This is hard to get right, and we usually fail at this so far, with investigations often expanding to be large clusters of pages before any go up. But as far as I’m concerned, long projects are a failure mode, not a goal.
• Adding concrete reusable things to the conversation, which can be called on in other discussions. This means prioritizing things like empirical investigations that add new data, or cleanly stated considerations, rather than long vague or hard-to-disentangle discussions, or conclusions whose use requires trusting the author a lot.
• Generalist research and widely ranging projects vs. developed expertise. I’m not an expert on anything, as far as I know. Some things my work has involved: thinking about the origin of humans, examining records of 1700s cotton exports, designing incentives for survey participants, reasoning about computer hardware designs, corresponding incredulously with makers of computing benchmarks, skimming papers about the energy efficiency of albatrosses. We do have relative specializations (I do more philosophy, Rick does more empirical work), and would welcome more relevant expertise, but this work can be pretty wide ranging.
• Trustworthiness as an unbiased source vs. persuasion. We focus on questions where we are genuinely unsure of the answer (though we might expect that data will reveal our own current guess is correct), and try to write neutrally about the considerations that we think merit attention. We are unlikely to look for the best way to ‘convince people of AI risk’, but rather to set out to establish whether or not there is AI risk, and to document our reasoning clearly.
• Thriving emphasis vs. high-pressure productivity orientation. We sit toward the thriving end of this spectrum, and hope that pays off in terms of longer term productivity. We are relatively accommodating to idiosyncratic needs or preferences. Our work requires less temporal consistency or predictability than some jobs, so while we value seeing each other regularly and getting stuff done often, we are able to be flexible if someone has things to contribute, but difficulties with the standard office situation.

I’m focused here on the positives, but here are a few negatives too:

• Variable office situation: through a series of unfortunate and fortunate events which is getting ridiculous, we haven’t had a consistent shared office in years. At present, we have an office in SF but Rick works from the bigger Rationalist/EA offices in Berkeley.
• Small: currently two full-time people, plus various occasional people and socially around people. Working from Berkeley next to other AI risk orgs mitigates this some. Has been as many as seven people in a summer, which seemed better, and we hope to move back to at least four soon.
• Even the relatively easy work is hard in ways: everything is complicated and even if you set out to do the most basic analysis ever there seems to be a strong current pulling toward getting bogged down in details of details. This is not the kind of ‘hard’ where you need to be a genius, but rather where you can easily end up taking much longer than hoped, and also get discouraged, which doesn’t help with speed. We’re still figuring out how to navigate this while being epistemically careful enough to produce good information.

So, that was a hand-wavy account of why I think working at AI Impacts is particularly high impact, and some of what it’s like. If you might want to work for us, see our jobs page². If you don’t, but like thinking about the future of AI and wish we invited you to dinners, coffees, parties or our Slack, drop me a DM or send us a message through the AI Impacts feedback box. Pitches that I’m wrong and should do something else are also welcome.

Notes

by Rick Korzekwa, 2 February, 2022

Summary

In this post I outline apparent regularities in how major new technological capabilities and methods come about. I have not rigorously checked to see how broadly they hold, but it seems likely to me that they generalize enough to be useful. For each pattern, I give examples, then some relatively speculative explanations for why they happen. I finish the post with an outline of how AI progress will look if it sticks to these patterns. The patterns are:

• The first version of a new technology is almost always of very low practical value
• After the first version comes out, things improve very quickly
• Many big, famous, impressive, and important advances are preceded by lesser-known, but still notable advances. These lesser-known advances often follow a long period of stagnation
• Major advances tend to emerge while relevant experts are still in broad disagreement about which things are likely to work

Introduction

While investigating major technological advancements, I have noticed patterns among the technologies that have been brought to our attention at AI Impacts. I have not rigorously investigated these patterns to see if they generalize well, but I think it is worthwhile to write about them for a few reasons. First, I hope that others can share examples, counterexamples, or intuitions that will help make it clearer whether these patterns are “real” and whether they generalize. Second, supposing they do generalize, these patterns might help us develop a view of the default way in which major technological advancements happen. AI is unlikely to conform to this default, but it should help inform our priors and serve as a reference for how AI differs from other technologies. Finally, I think there is value in discussing and creating common knowledge around which aspects of past technological progress are most informative and decision-relevant to mitigating AI risk.

What I mean by ‘major technological advancements’

Throughout this post I am referring to a somewhat narrow class of advances in technological progress. I won’t try to rigorously define the boundaries for classification, but the kinds of technologies I’m writing about are associated with important new capabilities or clear turning points in how civilization solves problems. This includes, among many others, flight, the telegraph, nuclear weapons, the laser, penicillin, and the transistor. It does not include achievements that are not primarily driven by improvements in our technological capabilities. For example, many of the structure height or ship size advances included in our discontinuities investigation seem to have occurred mainly because someone had the means and desire to build a large ship or structure and the materials and methods were available, not because of a substantial change in known methods or available materials. It also does not include technologies granting impressive new capabilities that have not yet been useful outside of scientific research, such as gravitational wave interferometers.

I have tried to be consistent with terminology, but I have probably been a bit sloppy in places. I use ‘advancement’ to mean any discrete event in which a new capability is demonstrated or a device is created that demonstrates a new principle of operation. In some places, I use ‘achievement’ to mean things like flying across the ocean for the first time, and ‘new method’ to mean things like using nuclear reactions to release energy instead of chemical reactions.

Things to keep in mind while reading this:

• The primary purpose of this post is to present claims, not to justify them. I give examples and reasons why we might expect the pattern to exist, but my goal is not to make the strongest case that the patterns are real.
• I make these claims based on my overall impression of how things go, having spent part of the past few years doing research related to technological progress. I have done some searching for counterexamples, but I have not made a serious effort to analyze them statistically.
• The proposed explanations for why the patterns exist (if they do indeed exist) are largely speculative, though they are partially based on well-accepted phenomena like experience curves.
• There are some obvious sources of bias in the sample of technologies I’ve looked at. They’re mostly things that started out as candidates for discontinuities, which were crowd sourced and then examined in a way that strongly favored giving attention to advancements with clear metrics for progress.
• I’m not sure how to precisely offer my credence on these claims, but I estimate roughly 50% that at least one is mostly wrong. My best guess for how I will update on new information is by narrowing the relevant reference class of advancements.
• Although I did notice these patterns at least somewhat on my own, I do not claim these are original to me. I have seen related things in a variety of places, particularly the book Patterns of Technological Innovation by Devendra Sahal.

The patterns

The first version of a new device or capability is almost always terrible

Most of the time when a major milestone is crossed, it is done in a minimal way that is so bad it has little or no practical value. Examples include:

• The first ship to cross the Atlantic using steam power was slower than a fast ship with sails.
• The first transatlantic telegraph cable took an entire day to send a message the length of a couple Tweets, would stop working for long periods of time, and failed completely after three weeks.
• The first flight across the Atlantic took the shortest reasonable path and crash-landed in Ireland.
• The first apparatus for purifying penicillin did not produce enough to save the first patient.
• The first laser did not produce a bright enough spot to photograph for publication

Although I can’t say I predicted this would be the case before learning about it, in retrospect I do not think it is surprising. As we work toward an invention or achievement, we improve things along a limited number of axes with each design modification, oftentimes at the expense of other axes. In the case of improving an existing technology until it is fit for some new task, this means optimizing heavily on whichever axes are most important for accomplishing that task. For example, to modify a World War I bomber to cross the ocean, you need to optimize heavily for range, even if it requires giving up other features of practical value, like the ability to carry ordinance. Increasing range without making such tradeoffs makes the problem much harder. Not only does this make it less likely you will beat competitors to the achievement, it means you need to do design work in a way that is less iterative and requires more inference.

This also applies to building a device that demonstrates a new principle of operation, like the laser or transistor. The easiest path to making the device work at all will, by default, ignore many unnecessary practical considerations. Skipping straight past the dinky minimal version to the practical version is difficult, not only because designing something is harder with more constraints, but because we can learn a lot from the dinky minimal version on our way to the more practical version. This appears related to why Theodore Maiman won the race to build a laser. His competitors were pursuing designs that had clearer practical value but were substantially more difficult to get right.

The clearest example I’m aware of for a technology that does not fit this pattern is the first nuclear weapon. The need to create a weapon that was useful in the real world quickly and in secret using extremely scarce materials drove the scientists and engineers of the Manhattan Project to solve many practical problems before building and testing a device that demonstrated the basic principles. Absent these constraints, it would likely have been easier to build at least one test device along the way, which would most likely have been useless as a weapon.

Progress often happens more quickly following a major advancement

New technologies may start out terrible, but they tend to improve quickly. This isn’t just a matter of applying a constant rate of fractional improvement per year to an increase in performance—the doubling time for performance decreases. Examples:

• Average speed to cross the Atlantic in wind-powered ships doubled over roughly 300 years, after which crossing speed for steam-powered ships doubled in about 50 years, and crossing speed for aircraft doubled twice in less than 40 years.
• Telecommunications performance, measured as the product of bandwidth and distance, doubled every 6 years before the introduction of fiber optics and doubled every 2 years after¹
• The energy released per mass of conventional explosive increased by 20% during the 100 years leading up to the first nuclear weapon in 1945, and by 1960 there were devices with 1000x the energy density of the first nuclear weapon.
• The highest intensity available from artificial light sources had an average doubling time of about 10 years from 1800 to 1945. Following the invention of the laser in 1960, the doubling time averaged just six months for over 15 years. Lasers also found practical applications very quickly, with successful laser eye surgery within 18 months and the laser rangefinders a few years after that.

This pattern is less robust than the others. An obvious counterexample is Moore’s Law and related trajectories for progress in computer hardware, which remain remarkably smooth across major changes in underlying technology. Still, even if major technological advances do not always accelerate progress, they seem to be one of the major causes of accelerated progress.

Short-term increases in the rate of progress may be explained by the wealth of possible improvements to a new technology. The first version will almost always neglect numerous improvements that are easy to find. Furthermore, once a the technology is out in the world, it becomes much easier to learn how to improve it. This can be seen in experience curves for manufacturing, which typically show large improvements per unit produced in manufacturing time and cost when a new technology is introduced.

I’m not sure if this can explain long-running increases in the rate of progress, such as those for telecommunications and transatlantic travel by steamship. Maybe these technologies benefited from a large space of design improvements and sufficient, sustained growth in adoption to push down the experience curve quickly². It may be related to external factors, like the advent of steam power increasing general economic and technological progress enough to maintain the strong trend in advancements for crossing the ocean.

Major advancements are usually preceded by rapid or accelerating progress

Major new technological advances are often preceded by lesser-known, but substantial advancements. While we were investigating candidate technologies for the discontinuities project, I was often surprised to find that this preceding progress was rapid enough to eliminate or drastically reduce the calculated size of the candidate discontinuity. For example:

• The invention of the laser in 1960 was preceded by other sources of intense light, including the argon flash in the early 1940s and various electrical discharge lamps by the 1930s. Progress was fairly slow from ~1800 to the 1930s, and more-or-less nonexistent for centuries before that.
• Penicillin was discovered during a period of major improvements in public health and treatments for bacterial infections, including a drug that was such an improvement over existing treatments for infection it was called the “magic bullet”.
• The Haber Process for fixing nitrogen, often given substantial credit for improvements in farming output that enabled the huge population growth of the past century, was invented in 1920. It was preceded by the invention of two other processes, in 1910 and 1912, the latter of which was more than a factor of two improvement over the former.
• The telegraph and flight both crossed the Atlantic for the first time during a roughly 100 year period of rapid progress in steamships, during which crossing times decreased by a factor of four.

This pattern seems less robust than the previous one and it is, to me, less striking. The accelerated progress preceding a major advancement varies widely in terms of overall impressiveness, and there is not a clear cutoff for what should qualify as fitting the pattern. Still, when I to come up with a clear example of something that fails to match the pattern, I had some difficulty. Even nuclear weapons were preceded by at least one notable advance in conventional explosives a few years earlier, following what seems to have been decades of relatively sluggish progress.

An obvious and perhaps boring explanation for this is that progress on many metrics was accelerating during the roughly 200-300 year period when most of these took place. Technological progress exploded between 1700 and 2000, in a way that seems to have been accelerating pretty rapidly until around 1950. Every point on an aggressively accelerating performance trend follows a period of unprecedented progress. It is plausible to me that this fully explains the pattern, but I’m not entirely convinced.

An additional contributor may be that the drivers for research and innovation that result in major advances tend to cause other advancements along the way. For example, wars increase interest in explosives, so maybe it is not surprising that nuclear weapons were developed around the same time as some advances in conventional explosives. Another potential driver for such innovation is a disruption to incremental progress via existing methods that requires exploration of a broader solution space. For example, the invention of the argon flash would not have been necessary if it had been possible to improve the light output of arc lamps.

Major advancements are produced by uncertain scientific communities

Nearly every time I look into the details of the work that went into producing a major technological advancement, I find that the relevant experts had substantial disagreements about which methods were likely to work, right up until the advancement takes place, sometimes about fairly basic things. For example:

• The team responsible for designing and operating the first transatlantic telegraph cable had disagreements about basic properties of signal propagation on very long cables, which eventually led to the cable’s failure.
• The scientists of the Manhattan Project had widely varying estimates for the weapon’s yield, including scientists who expected the device to fail entirely.
• Widespread skepticism, misunderstanding, and disagreement of penicillin’s chemical and therapeutic properties were largely responsible for a 10 year delay in its development.
• Few researchers involved in the race to build a functioning laser expected the design that ultimately prevailed to be viable.

The default explanation for this seems pretty clear to me—empirical work is an important part of clearing up scientific uncertainty, so we should expect successful demonstrations of, novel, impactful technologies to eliminate a lot of uncertainty. Still, it did not have to be the case that the eliminated uncertainty is substantial and related to basic facts about the functioning of the technology. For example, the teams competing for the first flight across the Atlantic did not seem to have anything major to disagree about, though they may have made different predictions about the success of various designs.

There is a distinction to be made here, between high levels of uncertainty on the one hand, and low levels of understanding on the other. The communities of researchers and engineers involved in these projects may have collectively assigned substantial credence to various mistaken ideas about how things work, but my impression is that they did at least agree on the basic outlines of the problems they were trying to solve. They mostly knew what they didn’t know. For example, most of the disagreement about penicillin was about the details of things like its solubility, stability, and therapeutic efficacy. There wasn’t a disagreement about, for example, whether the penicillium fungus was inhibiting bacterial growth by producing an antibacterial substance or by some other mechanism. I’m not sure how to operationalize this distinction more precisely.

Relevance to AI risk

As I explained at the beginning of this post, one of my goals is to develop a typical case for the development of major technological advancements, so we can inform our priors about AI progress and think about the ways in which it may or may not differ from past progress. I have my own views on the ways in which it is likely to differ, but I don’t want to get into them here. To that end, here is what I expect AI progress will look like if it fits the patterns of past progress.

• Major new methods or capabilities for AI will be demonstrated in systems that are generally pretty poor.
• Under the right conditions, such as a multi-billion-dollar effort by a state actor, the first version of an important new AI capability or method may be sufficiently advanced to be a major global risk or of very large strategic value.
• An early system with poor practical performance is likely to be followed by very rapid progress toward a system that is more valuable or dangerous
• Progress leading up to an important new method or capability in AI is more likely to be accelerating than it is to be stagnant. Notable advances preceding a new capability may not be direct ancestors to it.
• Although high-risk and transformative AI capabilities are likely to emerge in an environment of less uncertainty than today, the feasibility of such capabilities and which methods can produce them are likely to be contentious issues within the AI community right up until those capabilities are demonstrated.

Thanks to Katja Grace, Daniel Kokotajlo, Asya Bergal, and others for the mountains of data and analysis that most of this article is based on. Thanks to Jalex Stark, Eli Tyre, and Richard Ngo for their helpful comments. All views and mistakes are my own.

All charts in this post are original to AI Impacts. All photographs are in the public domain.

This page is a stub. It does not necessarily represent much of what is known on the topic.

Our understanding is that the degree to which neural networks learn concepts that are potentially understandable to humans is an open question.

Details

A very incomplete list of sources on the topic:

• Acquisition of Chess Knowledge in AlphaZero (McGrath et al, 2021)¹
  From the paper: ‘…In this work we provide evidence that human knowledge is acquired by the AlphaZero neural network as it trains on the game of chess. By probing for a broad range of human chess concepts we show when and where these concepts are represented in the AlphaZero network….’
• Zoom in: An Introduction to Circuits (Olah et al, 2020)²
  From the paper: ‘In contrast to the typical picture of neural networks as a black box, we’ve been surprised how approachable the network is on this scale. Not only do neurons seem understandable (even ones that initially seemed inscrutable), but the “circuits” of connections between them seem to be meaningful algorithms corresponding to facts about the world. You can watch a circle detector be assembled from curves. You can see a dog head be assembled from eyes, snout, fur and tongue. You can observe how a car is composed from wheels and windows. You can even find circuits implementing simple logic: cases where the network implements AND, OR or XOR over high-level visual features.’

Notes

Featured image: from Olah, et al., “Zoom In: An Introduction to Circuits”, Distill, 2020., CC-BY 4.0

The AI Vignettes Project is an ongoing effort to write concrete plausible future histories of AI development and its social impacts.

Details

Purposes

We hope to:

• Check that abstract views about the future of AI have plausible concrete instantiations. (Especially, hypothesized extinction scenarios, and proposed safe scenarios.)
• Develop better intuitions about possible scenarios by thinking through them concretely.
• Notice recurring themes in concrete stories, that may be worth thinking about more broadly.
• Fill out the space of plausible feasible scenarios with concrete illustrations, to decrease bias in thinking about the future.
• Keep a collection of AI vignettes for others to use, in the above or other ways.

Methods

Our current intended method is:

1. Write draft vignettes, with no particular systematic method for choosing an unbiased selection of scenarios
2. Request comments on their realism
3. Modify according to comments
4. Repeat 2-3 until realism critiques subside

Work so far

AI Impacts has run two small workshops where participants wrote AI vignettes.

Vignette collection

This is a subset of vignettes arising from this project, or similar.