Plan

1. Try to write plausible future histories of the world, focusing on AI-relevant features. (“Vignettes.”)
2. Read each others’ vignettes and critique the implausible bits: “Wouldn’t the US government do something at that point?” “You say twenty nations promise each other not to build agent AI–could you say more about why and how?”
3. Amend and repeat.
4. Better understand your own views about how the development of advanced AI may go down.
(5. Maybe add your vignette to our collection.)

This event will happen over two days, so you can come Friday if this counts as work for you, Saturday if it counts as play, and both if you are keen. RSVP to particular days is somewhat helpful; let us know in the comments.

Date, Time, Location

The event will happen on Friday the 25th of June and Saturday the 26th.

It’ll go from 10am (California time) until probably around 4pm both days.

It will take place online, in the LessWrong Walled Garden. Here are the links to attend:
Friday 
Saturday

FAQ

> Do I need literary merit or creativity?
No.

> Do I need to have realistic views about the future?
No, the idea is to get down what you have and improve it.

> Do I need to write stories?
Nah, you can just critique them if you want.

> What will this actually look like?
We’ll meet up online, discuss the project and answer questions, and then spend chunks of time (online or offline) writing and/or critiquing vignettes, interspersed with chatting together.

> Have you done this before? Can I see examples?
Yes, on a small scale. See here for some resulting vignettes. We thought it was fun and interesting.

This event is co-organized by Katja Grace and Daniel Kokotajlo. Thanks to everyone who participated in the trial Vignettes Day months ago. Thanks to John Salvatier for giving us the idea.

This page is an incomplete collection of fiction about the development of advanced AI, and the consequences for society. Entries are generally included if we judge that they contain enough that is plausible or correctly evocative to be worth considering, in light of AI futurism. The list includes works created for the purpose of better understanding the future of AI, as well as those intended for other purposes such as mainstream entertainment. We welcome further submissions here.

Stories intended as AI futurism

These are stories intended to clarify, explore or appreciate possible future AI scenarios.

Vignettes for iterative improvement

These stories are written by people known to us—largely AI Impacts staff and participants at our workshops—with the intention of incrementally improving their realism via comments. All are in commentable form, and we welcome criticism, especially of departures from realism.

Slow takeoff + value drift

Asya Bergal

In which slow development of robotics serves as a bottleneck to full automation of human labor, and a slow takeoff eventually leads to a future in which values substantially different from current human values are optimized.

#slow-takeoff #robotics #long-term-value-loss

Daniel’s randomly generated future

Daniel Kokotajlo

In which early automation of R&D leads to near-term AGI takeoff, but robust tools for prediction and AI interpretation lead to a not-clearly-catastrophic outcome.

Generated using Daniel’s “Generate Future” widget.

#narrow-tools #prediction #interpretability #hardware-boom #intelligence-explosion #race

Winning the Corporate Minigame

Ben Goldhaber

In which transfer learning allows game-playing AI to take over parts of middle management and eventually capture the entire economy in a way that is misaligned with human values.

#misaligned-corporations

A compelling story

Katja Grace

In which our internal narrativizing is automated. (Not well fleshed out.)

#language-processing #non-detailed

Skin Deep

Ben Goldhaber

In which the epistemic world is transformed by non-agentic persuasion AI.

#persuasion-technology #social-epistemology-disaster

Other realism-oriented stories

These are stories from elsewhere in the AI or AI risk community, e.g. LessWrong.

What Failure Looks Like

Paul Christiano

1) A slow-rolling catastrophe from getting what we measure, and 2) a rapid catastrophe from influence-seeking behavior.

#slow-takeoff

Another (outer) alignment failure story

Paul Christiano

“This is a story where the alignment problem is somewhat harder than I expect, society handles AI more competently than I expect, and the outcome is worse than I expect. It also involves inner alignment turning out to be a surprisingly small problem.”

#slow-takeoff

The production web v1a, b, c and d and Flash War, v.1

Andrew Critch

Five scenarios in which competitive pressures force automation across the economy, at a pace that does not allow for sufficient safety. Illustrative of hypothesized existential risk arising from coordination failures, regardless of individual alignment.

#multi-polar #doom #slow-takeoff

April 15, 2040

Nisan

Tax evasion is attempted in a future with AI-facilitated cooperation.

#cooperation-AI

Forward Pass

Andrej Karpathy

Consciousness arises in GPT-3.

#consciousness #GPT-3

Mainstream AI fiction

These are works of fiction not (as far as we know) designed primarily to contribute to AI futurism. We include both prominent works, and those which we think contain at least something of interest in thinking about the future of AI. This list is incomplete. We have not necessarily read or watched all of them.

Runaround

Isaac Asimov, 1942
Introduced explicitly Asimov’s influential ‘three laws of robotics‘. Subsequent spinoffs include 2004 film I, Robot.

Colossus: The Forbin Project

Universal Pictures, 1970
Superhuman defense system attempts to take global power.

#takeover #military-action #superhuman-agent #singleton

The Terminator (and franchise)

Various media and authors, 1984-present
A military AI system attempts to exterminate humanity for reasons of self-preservation. People and cyborg assassins travel through time.

Transcendence

Written by Jack Paglen, directed by Wally Pfister, 2014
A scientist uploads his mind to a computer and becomes vastly more capable than any human.

Ex Machina

Written/directed by Alex Garland, 2014
A software engineer is tasked with performing a Turing test on an artificial general intelligence with a humanoid body.

Pandora Sequence

Frank Herbert and Bill Ransom, 1965-1988
Four novels in which an AI that is created out of necessity during space travel shapes the future of a human colony on a planet called Pandora.

Her

Written and directed by Spike Jonze, 2013
A man falls in love with his virtual assistant.

This Perfect Day

Ira Levin, 1970
Residents of a future world controlled entirely by a central computer attempt to gain freedom.

Do Android Dream of Electric Sheep? (and associated works)

Philip K. Dick, Ridley Scott, Denis Villeneuve, 1968-2017
A bounty hunter is tasked with tracking down synthetic humans in a dystopian future.

Understand

Ted Chiang, 1991
An experimental drug makes a man superhumanly intelligent.

Published 8 April 2021

This is an initial page, in the process of review, which may not be comprehensive or represent the best available understanding.

Coherence arguments say that if an entity’s preferences do not adhere to the axioms of expected utility theory, then that entity is susceptible to losing things that it values.

This does not imply that advanced AI systems must adhere to these axioms (‘be coherent’), or that they must be goal-directed.

Such arguments do appear to suggest that there will be non-zero pressure for advanced AI to become more coherent, and arguably also more ‘goal-directed’, given some minimal initial level of agency.

Details

Motion toward coherence

Expected utility maximization

‘Maximizing expected utility’ is a decision-making strategy, in which you assign a value to each possible ‘outcome’, and assign a probability to each outcome conditional on your actions, then always choose the action whose resulting outcomes have the highest ‘expected value‘ (average value of outcomes, weighted by the probability of those outcomes).

Coherence arguments

‘Coherence arguments’¹ demonstrate that if one’s preferences cannot be understood as ‘maximizing expected utility’, then one can be manipulated into giving up things that one values.

For instance, one coherence argument notes that if you have ‘circular preferences’ then you will consent to series of decisions that will leave you worse off, given your preferences:

Suppose you prefer:

• apple over pear
• pear over quince
• quince over apple
• any fruit over nothing

Then there is some tiny amount of money you would pay to go from apple to quince, and quince to pear, and pear to apple. At which point, you have spent money and are back where you started. If it is also possible to buy all of these fruit for money, then losing money means you lost some of whatever fruit for nothing, and you do want all of the fruit, by assumption.

If you avoid all such predictable losses, then according to the coherence arguments, you must be maximizing expected utility (as described above).

Coherence forces

That a certain characteristic of an entity’s ‘preferences’ makes it vulnerable to manipulation does not mean that it will not have that characteristic. In order for such considerations to change the nature of an entity, ignoring outside intervention, something like the following conditions need to hold:

• The entity can detect the characteristic (which could be difficult, if it is a logical relationship between all of its ‘preferences’ which are perhaps not straightforwardly accessible or well-defined)
• The realistic chance for loss is large enough to cause the entity to prioritize the problem
• The entity is motivated to become coherent by the possibility of loss (versus for instance inferring that losing money is good, since it is equivalent to a set of exchanges that are each good)
• The entity is in a position to alter its own preferences

Similar might apply if versions of the above hold for an outside entity with power over the agent, e.g. its creators, though in that case it is less clear that ‘coherence’ is a further motivator beyond that for having the agent’s preferences align with those of the outside agent (which would presumably coincide with coherence, to the extant that the outside agent had more coherent preferences).

Thus we say there is generally an incentive for coherence, but it may or may not actually cause a an entity to change in the direction of coherence at a particular time. We can also describe this as a ‘coherence force’ or ‘coherence pressure’, pushing minds toward coherence, all things equal, but for all we know, so weakly as to be often irrelevant.

Coherence forces apply to entities with ‘preferences’

The coherence arguments only apply to creatures with ‘preferences’ that might be thwarted by their choices, so there are presumably possible entities that are not subject to any coherence forces, due to not having preferences, depending on the most apt definition of ‘preferences’.

Behavior of coherent creatures

Supposing entities are likely to become more coherent, all things equal, a natural question is how coherent entities differ from incoherent entities.

Coherence is consistent with any behavior

If we observe an agent exhibiting any history of behavior, that is consistent with coherence because the agent could have a utility function that rates that history higher than any other history. Rohin Shah discusses this.

Coherence and goal-directedness

Coherence doesn’t logically require goal-directedness

As Rohin Shah discusses, the above means that coherence does not imply ‘goal-directed’ behavior (however you choose to define that, if it doesn’t include all behavior):

1. Coherence arguments do not exclude any behavior
2. Non-goal-directed behavior is consistent with coherence arguments
3. Thus coherence arguments do not imply goal directed behavior

Increasing coherence seems likely to be associated with increased intuitive ‘goal-directedness’

The following hypotheses (quoted from this blog post) seem plausible (where goal-directedness_Rohin means something like ‘that which looks intuitively goal-directed)²:

1. Coherence-reformed entities will tend to end up looking similar to their starting point but less conflicted
For instance, if a creature starts out being indifferent to buying red balls when they cost between ten and fifteen blue balls, it is more likely to end up treating red balls as exactly 12x the value of blue balls than it is to end up very much wanting the sequence where it takes the blue ball option, then the red ball option, then blue, red, red, blue, red. Or wanting red squares. Or wanting to ride a dolphin.

(I agree that if a creature starts out valuing Tuesday-red balls at fifteen blue balls and yet all other red balls at ten blue balls, then it faces no obvious pressure from within to become ‘coherent’, since it is not incoherent.)

2. More coherent strategies are systematically less wasteful, and waste inhibits goal-direction_Rohin, which means more coherent strategies are more forcefully goal-directed_Rohin on average
In general, if you are sometimes a force for A and sometimes a force against A, then you are not moving the world with respect to A as forcefully as you would be if you picked one or the other. Two people intermittently changing who is in the driving seat, who want to go to different places, will not cover distance in any direction as effectively as either one of them. A company that cycles through three CEOs with different evaluations of everything will—even if they don’t actively scheme to thwart one another—tend to waste a lot of effort bringing in and out different policies and efforts (e.g. one week trying to expand into textiles, the next week trying to cut everything not involved in the central business).

3. Combining points 1 and 2 above, as entities become more coherent, they generally become more goal-directed_Rohin. As opposed to, for instance, becoming more goal-directed_Rohin on average, but individual agents being about as likely to become worse as better as they are reformed. Consider: a creature that values red balls at 12x blue balls is very similar to one that values them inconsistently, except a little less wasteful. So it is probably similar but more goal-directed_Rohin. Whereas it’s fairly unclear how goal-directed_Rohin a creature that wants to ride a dolphin is compared to one that wanted red balls inconsistently much. In a world with lots of balls and no possible access to dolphins, it might be much less goal-directed_Rohin, in spite of its greater coherence. 

4. Coherence-increasing processes rarely lead to non-goal-directed_Rohin agents—like the one that twitches on the ground In the abstract, few starting points and coherence-motivated reform processes will lead to an agent with the goal of carrying out a specific convoluted moment-indexed policy without regard for consequence, like Rohin’s twitching agent, or to valuing the sequence of history-action pairs that will happen anyway, or to being indifferent to everything. And these outcomes will be even less likely in practice, where AI systems with anything like preferences probably start out caring about much more normal things, such as money and points and clicks, so will probably land at a more consistent and shrewd version of that, if 1 is true. (Which is not to say that you couldn’t intentionally create such a creature.)

Thus it presently seems likely that coherence arguments correspond to a force for for entities with something like ‘preferences’ to grow increasingly coherent, and generally increasingly goal-directed (intuitively defined).

Thus, to the extent that future advanced AI has preferences of the relevant kind, there appears to be a pressure for it to become more goal-directed. However it is unclear what can be said generally about the strength of this force.

Primary author: Katja Grace

Notes

By Katja Grace, April 1 2021

Today we are sharing with our blog readers a collection of yet-to-be-published drafts, in the hope of receiving feedback. We are especially looking for methodological critique, but all comments welcome!

Human-level performance estimate (Katja Grace)

How much hardware will we need to create AGI? (Asya Bergal)

Historic trends in AI Impacts productivity (Daniel Kokotajlo)

Analysis of superbombs as a global threat (Katja Grace)

By Katja Grace, Mar 25 2021

[Epistemic status: my current view, but I haven’t read all the stuff on this topic even in the LessWrong community, let alone more broadly.]

There is a line of thought that says that advanced AI will tend to be ‘goal-directed’—that is, consistently doing whatever makes certain favored outcomes more likely—and that this is to do with the ‘coherence arguments’. Rohin Shah, and probably others¹, have argued against this. I want to argue against them.

The old argument for coherence implying (worrisome) goal-directedness

I’d reconstruct the original argument that Rohin is arguing against as something like this (making no claim about my own beliefs here):

1. ‘Whatever things you care about, you are best off assigning consistent numerical values to them and maximizing the expected sum of those values’
   ‘Coherence arguments²’ mean that if you don’t maximize ‘expected utility’ (EU)—that is, if you don’t make every choice in accordance with what gets the highest average score, given consistent preferability scores that you assign to all outcomes—then you will make strictly worse choices by your own lights than if you followed some alternate EU-maximizing strategy (at least in some situations, though they may not arise). For instance, you’ll be vulnerable to ‘money-pumping’—being predictably parted from your money for nothing.³
2. ‘Advanced AI will tend to do better things instead of worse things, by its own lights’
   Advanced AI will tend to avoid options that are predictably strictly worse by its own lights, due to being highly optimized for making good choices (by some combination of external processes that produced it, its own efforts, and the selection pressure acting on its existence).
3. ‘Therefore advanced AI will maximize EU, roughly’
   Advanced AI will tend to be fairly coherent, at least to a level of approximation where becoming more coherent isn’t worth the cost.⁴ Which will probably be fairly coherent (e.g. close enough to coherent that humans can’t anticipate the inconsistencies).
4. ‘Maximizing EU is pretty much the same as being goal-directed’
   To maximize expected utility is to pursue the goal of that which you have assigned higher utility to.⁵

And since the point of all this is to argue that advanced AI might be hard to deal with, note that we can get to that conclusion with:

5. ‘Highly intelligent goal-directed agents are dangerous’
   If AI systems exist that very competently pursue goals, they will likely be better than us at attaining their goals, and therefore to the extent there is a risk of mismatch between their goals and ours, we face a serious risk.

Rohin’s counterargument

Rohin’s counterargument begins with an observation made by others before: any behavior is consistent with maximizing expected utility, given some utility function. For instance, a creature just twitching around on the ground may have the utility function that returns 1 if the agent does whatever it in fact does in each situation (where ‘situation’ means, ‘entire history of the world so far’), and 0 otherwise. This is a creature that just wants to make the right twitch in each detailed, history-indexed situation, with no regard for further consequences. Alternately the twitching agent might care about outcomes, but just happen to want the particular holistic unfolding of the universe that is occurring, including this particular series of twitches. Or it could be indifferent between all outcomes.

The basic point is that rationality doesn’t say what ‘things’ you can want. And in particular, it doesn’t say that you have to care about particular atomic units that larger situations can be broken down into. If I try to call you out for first spending money to get to Paris, then spending money to get back from Paris, there is nothing to say you can’t just have wanted to go to Paris for a bit and then to come home. In fact, this is a common human situation. ‘Aha, I money pumped you!’ says the airline, but you aren’t worried. The twitching agent might always be like this—a creature of more refined tastes, who cares about whole delicate histories and relationships, rather than just summing up modular momentarily-defined successes. And given this freedom, any behavior might conceivably be what a creature wants. 

Then I would put the full argument, as I understand it, like this:

1. Any observable sequence of behavior is consistent with the entity doing EU maximization (see observation above)
2. Doing EU maximization doesn’t imply anything about what behavior we might observe (from 1)
3. In particular, knowing that a creature is an EU maximizer doesn’t imply that it will behave in a ‘goal-directed’ way, assuming that that concept doesn’t apply to all behavior. (from 2)

Is this just some disagreement about the meaning of the word ‘goal-directed’? No, because we can get back to a major difference in physical expectations by adding:

4.  Not all behavior in a creature implicates dire risk to humanity, so any concept of goal-directedness that is consistent with any behavior—and so might be implied by the coherence arguments—cannot imply AI risk.

So where the original argument says that the coherence arguments plus some other assumptions imply danger from AI, this counterargument says that they do not. 

(There is also at least some variety in the meaning of ‘goal-directed’. I’ll use goal-directed_Rohin to refer to what I think is Rohin’s preferred usage: roughly, that which seems intuitively goal directed to us, e.g. behaving similarly across situations, and accruing resources, and not flopping around in possible pursuit of some exact history of personal floppage, or peaceably preferring to always take the option labeled ‘A’.⁶)

My counter-counterarguments

What’s wrong with Rohin’s counterargument? It sounded tight. 

In brief, I see two problems:

1. The whole argument is in terms of logical implication. But what seems to matter is changes in probability. Coherence doesn’t need to rule out any behavior to matter, it just has to change the probabilities of behaviors. Understood in terms of probability, argument 2 is a false inference: just because any sequence of behavior is consistent with EU maximization doesn’t mean that EU maximization says nothing about what behavior we will see, probabilistically. All it says is that the probability of a behavioral sequence is never reduced to zero by considerations of coherence alone, which is hardly saying anything.

You might then think that a probabilistic version still applies: since every entity appears to be in good standing with the coherence arguments, the arguments don’t exert any force, probabilistically, on what entities we might see. But:

2. An outside observer being able to rationalize a sequence of observed behavior as coherent doesn’t mean that the behavior is actually coherent. Coherence arguments constrain combinations of external behavior and internal features—‘preferences’⁷ and beliefs. So whether an actor is coherent depends on what preferences and beliefs it actually has. And if it isn’t coherent in light of these, then coherence pressures will apply, whether or not its behavior looks coherent. And in many cases, revision of preferences due to coherence pressures will end up affecting external behavior. So 2) is not only not a sound inference from 1), but actually a wrong conclusion: if a system moves toward EU maximization, that does imply things about the behavior that we will observe (probabilistically). 

Perhaps Rohin only meant to argue about whether it is logically possible to be coherent and not goal-directed-seeming, for the purpose of arguing that humanity can construct creatures in that perhaps-unlikely-in-nature corner of mindspace, if we try hard. In which case, I agree that it is logically possible. But I think his argument is often taken to be relevant more broadly, to questions of whether advanced AI will tend to be goal-directed, or to be goal-directed in places where they were not intended to be.

I take 1) to be fairly clear. I’ll lay out 2) in more detail.

My counter-counterarguments in more detail

How might coherence arguments affect creatures?

Let us step back.

How would coherence arguments affect an AI system—or anyone—anyway? They’re not going to fly in from the platonic realm and reshape irrational creatures.

The main routes, as I see it, are via implying:

1. incentives for the agent itself to reform incoherent preferences
2. incentives for the processes giving rise to the agent (explicit design, or selection procedures directed at success) to make them more coherent
3. some advantage for coherent agents in competition with incoherent agents

To be clear, the agent, the makers, or the world are not necessarily thinking about the arguments here—the arguments correspond to incentives in the world, which these parties are responding to. So I’ll often talk about ‘incentives for coherence’ or ‘forces for coherence’ rather than ‘coherence arguments’.

I’ll talk more about 1 for simplicity, expecting 2 and 3 to be similar, though I haven’t thought them through.

Looking coherent isn’t enough: if you aren’t coherent inside, coherence forces apply

If self-adjustment is the mechanism for the coherence, this doesn’t depend on what a sequence of actions looks like from the outside, but from what it looks like from the inside.

Consider the aforementioned creature just twitching sporadically on the ground. Let’s call it Alex.

As noted earlier, there is a utility function under which Alex is maximizing expected utility: the one that assigns utility 1 to however Alex in fact acts in every specific history, and utility 0 to anything else.

But from the inside, this creature you excuse as ‘maybe just wanting that series of twitches’ has—let us suppose—actual preferences and beliefs. And if its preferences do not in fact prioritize this elaborate sequence of twitching in an unconflicted way, and it has the self-awareness and means to make corrections, then it will make corrections⁸. And having done so, its behavior will change. 

Thus excusable-as-coherent Alex is still moved by coherence arguments, even while the arguments have no complaints about its behavior per se.

For a more realistic example: suppose Assistant-Bot is observed making this sequence of actions: 

• Offers to buy gym membership for $5/week 
• Consents to upgrade to gym-pro membership for $7/week, which is like gym membership but with added morning classes
• Takes discounted ‘off-time’ deal, saving $1 per week for only using gym in evenings

This is consistent with coherence: Assistant-Bot might prefer that exact sequence of actions over all others, or might prefer incurring gym costs with a larger sum of prime factors, or might prefer talking to Gym-sales-bot over ending the conversation, or prefer agreeing to things.

But suppose that in fact, in terms of the structure of the internal motivations producing this behavior, Assistant-Bot just prefers you to have a gym membership, and prefers you to have a better membership, and prefers you to have money, but is treating these preferences with inconsistent levels of strength in the different comparisons. Then there appears to be a coherence-related force for Assistant-Bot to change. One way that that could look is that since Assistant-Bot’s overall behavioral policy currently entails giving away money for nothing, and also Assistant-Bot prefers money over nothing, that preference gives Assistant-Bot reason to alter its current overall policy, to avert the ongoing exchange of money for nothing.⁹ And if its behavioral policy is arising from something like preferences, then the natural way to alter it is via altering those preferences, and in particular, altering them in the direction of coherence.

One issue with this line of thought is that it’s not obvious in what sense there is anything inside a creature that corresponds to ‘preferences’. Often when people posit preferences, the preferences are defined in terms of behavior. Does it make sense to discuss different possible ‘internal’ preferences, distinct from behavior? I find it helpful to consider the behavior and ‘preferences’ of groups:

Suppose two cars are parked in driveways, each containing a couple. One couple are just enjoying hanging out in the car. The other couple are dealing with a conflict: one wants to climb a mountain together, and the other wants to swim in the sea together, and they aren’t moving because neither is willing to let the outing proceed as the other wants. ‘Behaviorally’, both cars are the same: stopped. But their internal parts (the partners) are importantly different. And in the long run, we expect different behavior: the car with the unconflicted couple will probably stay where it is, and the conflicted car will (hopefully) eventually resolve the conflict and drive off.

I think here it makes sense to talk about internal parts, separate from behavior, and real. And similarly in the single agent case: there are physical mechanisms producing the behavior, which can have different characteristics, and which in particular can be ‘in conflict’—in a way that motivates change—or not. I think it is also worth observing that humans find their preferences ‘in conflict’ and try to resolve them, which is suggests that they at least are better understood in terms of both behavior and underlying preferences that are separate from it. 

So we have: even if you can excuse any seizuring as consistent with coherence, coherence incentives still exert a force on creatures that are in fact incoherent, given their real internal state (or would be incoherent if created). At least if they or their creator have machinery for noticing their incoherence, caring about it, and making changes.

Or put another way, coherence doesn’t exclude overt behaviors alone, but does exclude combinations of preferences, and preferences beget behaviors. This changes how specific creatures behave, even if it doesn’t entirely rule out any behavior ever being correct for some creature, somewhere. 

That is, the coherence theorems may change what behavior is likely to appear amongst creatures with preferences. 

Reform for coherence probably makes a thing more goal-directed_Rohin

Ok, but moving toward coherence might sound totally innocuous, since, per Rohin’s argument, coherence includes all sorts of things, such as absolutely any sequence of behavior. 

But the relevant question is again whether a coherence-increasing reform process is likely to result in some kinds of behavior over others, probabilistically.

This is partly a practical question—what kind of reform process is it? Where a creature ends up depends not just on what it incoherently ‘prefers’, but on what kinds of things its so-called ‘preferences’ are at all¹⁰, and what mechanisms detect problems, and how problems are resolved.

My guess is that there are also things we can say in general. It’s is too big a topic to investigate properly here, but some initially plausible hypotheses about a wide range of coherence-reform processes:

1. Coherence-reformed entities will tend to end up looking similar to their starting point but less conflicted
   For instance, if a creature starts out being indifferent to buying red balls when they cost between ten and fifteen blue balls, it is more likely to end up treating red balls as exactly 12x the value of blue balls than it is to end up very much wanting the sequence where it takes the blue ball option, then the red ball option, then blue, red, red, blue, red. Or wanting red squares. Or wanting to ride a dolphin.
   
   (I agree that if a creature starts out valuing Tuesday-red balls at fifteen blue balls and yet all other red balls at ten blue balls, then it faces no obvious pressure from within to become ‘coherent’, since it is not incoherent.)
   
2. More coherent strategies are systematically less wasteful, and waste inhibits goal-direction_Rohin, which means more coherent strategies are more forcefully goal-directed_Rohin on average
   In general, if you are sometimes a force for A and sometimes a force against A, then you are not moving the world with respect to A as forcefully as you would be if you picked one or the other. Two people intermittently changing who is in the driving seat, who want to go to different places, will not cover distance in any direction as effectively as either one of them. A company that cycles through three CEOs with different evaluations of everything will—even if they don’t actively scheme to thwart one another—tend to waste a lot of effort bringing in and out different policies and efforts (e.g. one week trying to expand into textiles, the next week trying to cut everything not involved in the central business).

3. Combining points 1 and 2 above, as entities become more coherent, they generally become more goal-directed_Rohin. As opposed to, for instance, becoming more goal-directed_Rohin on average, but individual agents being about as likely to become worse as better as they are reformed. Consider: a creature that values red balls at 12x blue balls is very similar to one that values them inconsistently, except a little less wasteful. So it is probably similar but more goal-directed_Rohin. Whereas it’s fairly unclear how goal-directed_Rohin a creature that wants to ride a dolphin is compared to one that wanted red balls inconsistently much. In a world with lots of balls and no possible access to dolphins, it might be much less goal-directed_Rohin, in spite of its greater coherence. 

4. Coherence-increasing processes rarely lead to non-goal-directed_Rohin agents—like the one that twitches on the ground
   In the abstract, few starting points and coherence-motivated reform processes will lead to an agent with the goal of carrying out a specific convoluted moment-indexed policy without regard for consequence, like Rohin’s twitching agent, or to valuing the sequence of history-action pairs that will happen anyway, or to being indifferent to everything. And these outcomes will be even less likely in practice, where AI systems with anything like preferences probably start out caring about much more normal things, such as money and points and clicks, so will probably land at a more consistent and shrewd version of that, if 1 is true. (Which is not to say that you couldn’t intentionally create such a creature.)
   

These hypotheses suggest to me that the changes in behavior brought about by coherence forces favor moving toward goal-directedness_Rohin, and therefore at least weakly toward risk.

Does this mean advanced AI will be goal-directed_Rohin?

Together, this does not imply that advanced AI will tend to be goal-directed_Rohin. We don’t know how strong such forces are. Evidently not so strong that humans¹¹, or our other artifacts, are whipped into coherence in mere hundreds of thousands of years¹². If a creature doesn’t have anything like preferences (beyond a tendency to behave certain ways), then coherence arguments don’t obviously even apply to it (though discrepancies between the creature’s behavior and its makers’ preferences probably produce an analogous force¹³ and competitive pressures probably produce a similar force for coherence in valuing resources instrumental to survival). Coherence arguments mark out an aspect of the incentive landscape, but to say that there is an incentive for something, all things equal, is not to say that it will happen.

In sum

1) Even though any behavior could be coherent in principle, if it is not coherent in combination with an entity’s internal state, then coherence arguments point to a real force for different (more coherent) behavior.

2) My guess is that this force for coherent behavior is also a force for goal-directed behavior. This isn’t clear, but seems likely, and also isn’t undermined by Rohin’s argument, as seems commonly believed.

.

.

.

Updated Dec 21, 2020

This is a list of work done at AI Impacts published in 2020.

Major projects

Discontinuities project

Primary authors: Katja Grace, Richard Korzekwa, Asya Bergal, Daniel Kokotajlo
Main page: Discontinuous progress investigation
Blog post: Discontinuous progress in history: an update

Other pages added in 2020: Effect of AlexNet on historic trends in image recognition, Historic trends in transatlantic message speed, Historic trends in long-range military payload delivery, Historic trends in bridge span length, Historic trends in light intensity, Historic trends in book production, Historic trends in telecommunications performance, Historic trends in slow light technology, Penicillin and historic syphilis trends, Historic trends in the maximum superconducting temperature, Historic trends in chess AI, Effect of Eli Whitney’s cotton gin on historic trends in cotton ginning, Historic trends in flight airspeed records, 

Evolution engineering comparison

Primary authors: Ronny Fernandez
Main page: How energy efficient are human-engineered flight designs relative to natural ones?

Other pages added in 2020: Energy efficiency of monarch butterfly flight, Energy efficiency of wandering albatross flight, Energy efficiency of paramotors, Energy efficiency of The Spirit of Butt’s Farm, Energy efficiency of MacCready Gossamer Albatross, Energy efficiency of Boeing 747-400, Energy efficiency of Airbus A320, Energy efficiency of North American P-51 Mustang, Energy efficiency of Vickers Vimy plane, Energy efficiency of Wright model B, Energy efficiency of Wright Flyer

Time to cross human performance range

Primary authors: Richard Korzekwa
Pages added in 2020:

• Time for AI to cross the human range in English draughts
• Time for AI to cross the human range in StarCraft
• Time for AI to cross the human performance range in ImageNet image classification
• Time for AI to cross the human performance range in chess
• Time for AI to cross the human performance range in Go

Historic hardware trends

Primary authors: Asya Bergal
Pages added in 2020:

• 2019 recent trends in GPU price per FLOPS
• 2019 recent trends in Geekbench score per CPU price
• Trends in DRAM price per gigabyte

Interviews on plausibility of AI safety by default

Primary authors: Asya Bergal, Robert Long
Main page: Interviews on plausibility of AI safety by default
Blog post: Takeaways from safety by default interviews

Shallow survey of prescient actions

Primary authors: Richard Korzekwa
Main page: Preliminary survey of prescient actions

Other work

Pages added in 2020

• AI Impacts key questions of interest — Katja Grace
• Was the industrial revolution a drastic departure from historic trends? — Katja Grace
• Surveys on fractional progress towards HLAI — Asya Bergal
• Precedents for economic n-year doubling before 4n-year doubling — Daniel Kokotajlo
• Resolutions of mathematical conjectures over time — Asya Bergal

Blog posts added in 2020

• Misalignment and misuse: whose values are manifest? — Katja Grace
• Automated intelligence is not AI — Katja Grace
• Relevant pre-AGI possibilities — Daniel Kokotajlo, Asya Bergal
• Description vs simulated prediction — Richard Korzekwa
• Atari early — Katja Grace
• Three kinds of competitiveness — Daniel Kokotajlo
• AGI in a vulnerable world — Asya Bergal
• Cortes, Pizarro, and Afonso as precedents for takeover — Daniel Kokotajlo

Updated Dec 17, 2020

This is a list of questions that AI Impacts focuses on answering.

Details

We are interested in understanding how the development of advanced AI will proceed and how it may affect humanity, especially insofar as these are relevant to efforts to improve the outcomes. This is a list of questions within this topic that we currently consider particularly important to answer.

List

• AI RISK: What type and degree of risk will be posed to humanity by advanced AI systems?
• CHARACTER: What will early advanced AI systems be like?
  • ARCHITECTURE: What types of algorithms will advanced AI systems use?
  • AGENCY: Will the bulk of advanced AI systems be in the form of ‘agents’? (If so, in what sense? Will they pursue ‘goals’? Can we say anything about the nature the goals or the pursuit?)
  • PRICE: How much will the first human-level AI systems cost?
• TIMELINES: When will human-level AI be developed? (When will other important AI milestones take place?)
• TAKE-OFF SPEED: How rapid is the development of AI likely to be near human-level?
  • DISCONTINUITY: Will there be abrupt progress in AI development at around human-level performance?
  • INTELLIGENCE EXPLOSION: How much will AI development be accelerated by feedback from AI-based automation of the process?
• PRE-AI DEVELOPMENTS: What developments will take place before advanced AI is developed?
  • PATHS TO HLAI: By what methods is advanced AI likely to come about? (e.g. will human-level AI be developed via brain emulation before it is developed via machine learning? Will neuroscientific understanding play a large role in development?)
  • CONTEMPORARY EVENTS: How will the world be relevantly different at the time that advanced AI is developed?
  • WARNING SIGNS: Should we expect advance notice of disruptive change from AI? (What would it look like?)
• POST-AI SOCIETY: If advanced AI transforms the world, what will the world look like afterwards? (e.g. What will the economic impacts be? Will humans flourish? What roles will AI systems play in society?)
• ACTIONS: What can we say about the impact of contemporary choices on long-term outcomes?

Updated Dec 10, 2020

Among two animals and nine machines:

• In terms of mass⋅distance/energy, the most efficient animal was 2-8x more efficient than the most efficient machine. All entries fell within two orders of magnitude.
• In terms of distance/energy, the most efficient animal was 3,000-20,000x more efficient than the most efficient machine. Both animals were more efficient than all machines. Entries ranged over more than eight orders of magnitude.

Details

Background

This case study is part of research that intends to compare the performance of human engineers and natural evolution on problems where both have developed solutions. The goal of this is to inform our expectations about the performance of future artificial intelligence relative to biological minds.

Metrics

We consider two metrics:

1. Distance per energy used (meters / kilojoule).
2. Mass times distance per energy used (kilograms⋅meters / joule).

These operationalize the problem of flight into two more specific problems. There are many other aspects of flight performance that one could measure, such as energy efficiency of acceleration in a straight line, turning, hovering, vertical acceleration, vertical distance, landing, taking off, time flying per energy, and our same measures with fewer or further restrictions on acceptable entries. For instance, we might look at the problem of flying with flapping wings, or without the restriction that the solutions we consider are heavier than air and self powered.

We did not require that the flight of an entry be constantly powered. Solutions that spend some time gliding as well as some time using powered flight were allowed. Both albatrosses and butterflies use air currents to fly further.¹ The energy gains from these techniques were not included in the final score, and entries were not penalized for spending a larger fraction of time gliding. It seems likely that paramotor pilots use similar techniques, since paramotors are well suited to gliding (being paragliders with propeller motors strapped to the backs of their pilots). Our energy efficiency estimate for the paramotor came from a record breaking distance flight in which the quantity of available fuel was limited, and so it is likely that some gliding was used to increase the distance traveled as much as possible.

When multiple input values could have been used, such as the takeoff weight and the landing weight, or different estimates for the energetic costs of different kinds of flight for the Monarch butterfly, we generally calculated a high and a low estimate, taking the most optimistic and pessimistic inputs respectively. In all cases, the resulting best and worst estimates differed by less than a factor of ten.

Selection of case studies

We selected case studies informally, according to judgments about possible high energy efficiencies, and with an eye to exploring a wider range of case studies.

We started by looking at the Boeing 747-400 plane, the Wandering Albatross, and the Monarch Butterfly. We chose the animals for both being known for their abilities to fly long distances, and for both having fairly different body plans.

All three scored surprisingly similarly on distance times weight per energy (details below). This prompted us to look for engineered solutions that were optimized for fuel efficiency. To that end, we looked at paramotors and record breaking flying machines. In the latter category, we found the MacCready Gassomer Albatross, which was a human powered flying device that crossed the English Channel, and the Spirit of Butts’ Farm, which was a model airplane that crossed the Atlantic on one gallon of gasoline. 

For reasons that are now obscure, we also included a number of different planes.

We would have liked to include microdrones, since they are different enough from other entries that they might be unusually efficient. However we did not find data on them.

Case studies

These are the full articles calculating the efficiencies of different flying machines and animals:

• Wright Flyer
• Wright model B
• Vickers Vimy
• North American P-51 Mustang
• Paramotors
• The Spirit of Butt’s Farm
• Monarch butterfly
• MacCready Gossamer Albatross
• Airbus A-320
• Boeing 747-400
• Wandering albatross

Summary results

Results are available in Table 1 below, and in this spreadsheet. Figures 1 and 2 below illustrate the equivalent questions of how far each of these animals and machines can fly, given either the same amount of fuel energy, or fuel energy proportional to their body mass.

Table 1: Energy efficiency of flight for a variety of natural and man-made flying entities.

On mass⋅distance/energy, evolution beats engineers, but they are relatively evenly matched: the albatross (1.4-3.0 kg.m/J) and the Boeing 747-400 (0.39-0.83 kg.m/J) are the best in the natural and engineered classes respectively. Thus the best natural solution we found was roughly 2x-8x more efficient than the human-engineered one.² We found several flying machines more efficient on this metric than the monarch butterfly.

On distance/energy, the natural solutions have a much larger advantage. Both are better than all man-made solutions we considered. The best natural and engineered solutions respectively are the monarch butterfly (100,000-600,000 m/kJ) and the Spirit of Butts’ Farm (32 m/kJ), for roughly a 3,000x to 20,000x advantage to natural evolution.

Interpretation

We take this as weak evidence about the best possible distance/energy and distance.mass/energy measures achievable by human engineers or natural evolution. One reason for this is that this is a small set of examples. Another is that none of these animals or machines were optimized purely for either of these flight metrics—they all had other constraints or more complex goals. For instance, the paramotor was competing for a record in which a paramotor had to be used, specifically. For the longest human flight, the flying machine had to be capable of carrying a human. The albatross’ body has many functions. Thus it seems plausible that either engineers or natural evolution could reach solutions far better on our metrics than those recorded here if they were directly aiming for those metrics.

The measurements for distance.mass/energy covered a much narrower band than those for distance/energy: a factor of under two orders of magnitude versus around eight. Comparing best scores between evolution and engineering, the gap is also much smaller, as noted above (a factor of less than one order of magnitude versus three orders of magnitude). This seems like some evidence that that band of performance is natural for some reason, and so that more pointed efforts to do better on these metrics would not readily lead to much higher performance.

Primary author: Ronny Fernandez

Notes

Updated Nov 25, 2020

According to very rough estimates, the monarch butterfly:

• can fly around 100,000-600,000 m/kJ
• and move mass at around 0.065-0.36 kg.m/J

Details

The Monarch Butterfly is a butterfly known for its impressively long migration.¹

Mass

The average mass of a monarch butterfly prior to its annual migration has been estimated to be 600mg²

Distance per Joule

The following table gives some very rough estimates of energy expenditures, speeds and distances for several modes of flight, based on confusing information from a small number of papers (see footnotes for details).

Soaring is estimated to be potentially very energy efficient (see Table 1), since it mostly makes use of air currents for energy. It seems likely that at least a small amount of powered flight is needed for getting into the air, however monarch butterflies can apparently fly for hundreds of kilometers in a day¹⁵, so supposing that they don’t stop many times in a day, taking off seems likely a negligible part of the flight.¹⁶

This would require ideal wind conditions, and our impression is that in practice, butterflies do not often fly very long distances without using at least a small amount of powered flight.¹⁷

There is stronger evidence that monarch butterflies can realistically soar around 85% of the time, from Gibo & Pallett, who report their observations of butterflies under relatively good conditions.¹⁸ So as a high estimate, we use this fraction of the time for soaring, and suppose that the remaining time is the relatively energy-efficient cruising, and take the optimistic end of all ranges. This gives us:

One second of flight = 0.15 seconds cruising + 0.85 seconds soaring

= 0.15s * 5 m/s cruising + 0.85s * 3.6m/s soaring

= 0.75m cruising + 3.06m soaring = 3.81m total

= 0.75m / 143 m/J cruising + 3.06m / 2571 m/J soaring = 0.0064 J total

Thus distance/energy = 3.81m/0.0064 J = 595 m/J

For a low estimate of efficiency, we will assume that all of the powered flight is the most energetic flight, that powered flight is required half the time on average, and that the energy cost of gliding is twice that of resting. This gives us:

Energy efficiency = (50% * soaring distance + 50% * powered distance)/(50% * soaring energy + 50% * powered energy)

= (50% * soaring distance/time + 50% * powered distance/time)/(50% * soaring energy/time + 50% * powered energy/time)

= (0.5 * 2.5m/s + 0.5 * 13.9m/s)/(0.5 * (0.0056 * 2) J/s + 0.5 * 0.14 J/s)

= 108 m/J

Thus we have, very roughly:

distance/energy = 100,000-600,000 m/kJ

Mass.distance per Joule

As noted earlier, the average mass of a monarch butterfly prior to its annual migration has been estimated to be 600mg¹⁹

Thus we have:

mass*distance/energy = 0.0006 kg * 108 — 0.0006 kg * 595 m/J

= 0.065 — 0.36 kg.m/J

Notes

Updated Nov 24, 2020

The wandering albatross:

• can fly at around 481m/kJ
• and moved mass at around 2.8—6.1 kg.m/J

Details

The wandering albatross is a very large seabird that flies long distances on wings with the largest span of any bird.¹

Distance per Joule

Speed

In a study of wandering albatrosses flying in various wind speeds and directions, the fastest ground speed measured appears to be around 24m/s, ²

Energy expenditure

One study produced an estimate that when flying, albatrosses use 2.35 times their basal metabolic rate³ which same paper implies is around 1,833 kJ/bird.day.⁴

That gives us a flight cost for flying of 0.050 kJ/second.⁵ 

Distance per Joule calculation

This gives us a distance per energy score of:

distance/energy

= 24 m/s / 0.050 kJ/s

= 481m/kJ

Mass.distance per Joule

Albatrosses weigh 5.9 to 12.7 kg.⁶

Thus we can estimate:

mass.distance/Joule

= 5.9kg * 0.48 m/J to 12.7kg*0.48 m/J

= 2.8—6.1kg.m/J

Notes

Updated Nov 24, 2020

We estimate that a record-breaking two-person paramotor:

• covered around = 0.36 m/kJ
• and moved mass at around 0.058 – 0.10 kg.m/J

Details

Paramotors are powered parachutes that allow the operator to steer.¹

Distance per Joule

The Fédération Aéronautique Internationale (FAI) maintains records for a number of classes of paramotor contest. We look at subclass RPF2T—(Paramotors : Paraglider Control / Foot-launched / Flown with two persons / Thermal Engine)—which is appears to be the most recent paramotor record for ‘Distance in a straight line with limited fuel’.²

The record distance was 123.18 km.³ The FAI rules state that no more than 7.5 kg of fuel may be used.⁴ We will assume that in the process of breaking this record, all of the available fuel was used. We will also assume that regular gasoline was used. Gasoline has an energy density of 45 MJ/kg.⁵

Distance per energy = 123.18 km / (7.5 kg * 45 MJ/kg)

= 0.36 m/kJ

Mass.distance per Joule

The weight of an entire paramotoring apparatus appears to be the weights of the passengers plus motor plus wing plus clothing and incidentals, based on forum posts.⁶ These posts put clothing and incidentals at around 8kg, but are estimates for single person flying, whereas this record was a two person flight. We guess that two people need around 1.5x as much additional weight, for 12kg.

Wikipedia says that the weight of a paramotor varies from 18kg to 34 kg.⁷ However it is unclear whether this means the motor itself, or all of the equipment involved.

The glider used appears to be MagMax brand, a typical example of which weighs around 8kg, though this may have been different in 2013, or they may have used a different specific glider.⁸ To account for this uncertainty, we shall add the glider weight to the high estimate, and so estimate the weight of the glider and motor together at 18-42kg.

We will assume that the apparently male pilots weighed between 65 and 115 kgs each, based on normal male weights⁹. 

Thus we have:

weight = motor + wing + people + clothing and incidentals

weight (low estimate) = 18 + 65*2 + 12 = 160kg

weight (high estimate) = 42 + 115*2 + 12 = 284kg

High efficiency estimate:

284kg * 0.36 m/kJ  = 0.10 kg.m/J

Low efficiency estimate:

160kg * 0.36 m/kJ  = .058 kg.m/J

This gives us a range of 0.058 – 0.10 kg.m/J

Notes

By Katja Grace, Nov 18 2020, Crossposted from world spirit sock puppet.

AI related disasters are often categorized as involving misaligned AI, or misuse, or accident. Where:

• misuse means the bad outcomes were wanted by the people involved,
• misalignment means the bad outcomes were wanted by AI (and not by its human creators), and
• accident means that the bad outcomes were not wanted by those in power but happened anyway due to error.

In thinking about specific scenarios, these concepts seem less helpful.

I think a likely scenario leading to bad outcomes is that AI can be made which gives a set of people things they want, at the expense of future or distant resources that the relevant people do not care about or do not own.

For example, consider autonomous business strategizing AI systems that are profitable additions to many companies, but in the long run accrue resources and influence and really just want certain businesses to nominally succeed, resulting in a worthless future. Suppose Bob is considering whether to get a business strategizing AI for his business. It will make the difference between his business thriving and struggling, which will change his life. He suspects that within several hundred years, if this sort of thing continues, the AI systems will control everything. Bob probably doesn’t hesitate, in the way that businesses don’t hesitate to use gas vehicles even if the people involved genuinely think that climate change will be a massive catastrophe in hundreds of years.

When the business strategizing AI systems finally plough all of the resources in the universe into a host of thriving 21st Century businesses, was this misuse or misalignment or accident? The strange new values that were satisfied were those of the AI systems, but the entire outcome only happened because people like Bob chose it knowingly (let’s say). Bob liked it more than the long glorious human future where his business was less good. That sounds like misuse. Yet also in a system of many people, letting this decision fall to Bob may well have been an accident on the part of others, such as the technology’s makers or legislators.

Outcomes are the result of the interplay of choices, driven by different values. Thus it isn’t necessarily sensical to think of them as flowing from one entity’s values or another’s. Here, AI technology created a better option for both Bob and some newly-minted misaligned AI values that it also created—‘Bob has a great business, AI gets the future’—and that option was worse for the rest of the world. They chose it together, and the choice needed both Bob to be a misuser and the AI to be misaligned. But this isn’t a weird corner case, this is a natural way for the future to be destroyed in an economy.

Thanks to Joe Carlsmith for conversation leading to this post.

Updated Nov 18, 2020

The Spirit of Butt’s Farm:

• covered around 31.67 m/kJ
• and moved mass at around 0.16 – 0.086 kg.m/J

Details

The Spirit of Butt’s Farm was a record setting model airplane that crossed the Atlantic on one gallon of fuel.¹ Fully fueled it weighed 4.987 kg, dry it weighed 2.705 kg.²

The record setting flight used 117.1 fluid ounces of fuel.³ The straight line distance of the flight was 3,028.1 km.⁴ It was powered by 88% Coleman lantern fuel, mixed with lubricant.⁵ Coleman fuel is based on naphtha ⁶, so we can use the energy density of naphtha—31.4 MJ/L⁷—as a rough guide to its energy content, though naphtha appears to vary in its content, and it is unclear whether Coleman fuel consists entirely of naphtha.

From all this, we have:

Distance per energy = 3,028.1 km / (117.1 fl oz * 0.88 * 31.4 MJ/L)

= 31.67 m/kJ

For weight times distance per energy we will calculate a best and a worst score. To calculate the best score we will use the fully fueled weight, and to calculate the worst score we will use the dry weight. All other values are the same in both calculations. 

Best score:

Distance*mass/energy = 4.987 kg * 31.67 m/kJ

= 0.16 kg.m/J

(4.987 kg * 3,028.1km) / (117.1 US fluid ounces * 31.4MJ/litre) = 0.1389 kg*m/j

Worst score:

Distance*mass/energy = 2.705 kg * 31.67 m/kJ

= .086 kg.m/J

Notes

Photo by Ronan Coyne, licensed under the Creative Commons Attribution-Share Alike 3.0 Unported license, unaltered.

We do not have a considered view on this topic.

Details

We have not investigated this topic. This is an incomplete list of evidence that we know of:

• David Roodman’s analysis of the surprisingness of the industrial revolution under his 2020 model of economic history.¹
• Ben Garfinkel’s analysis of whether economic history suggests a singularity²
• Robin Hanson’s analysis of historic economic growth understood as a sequence of exponential modes.³
• On a log(GWP)-log(doubling time) graph, the industrial revolution appears to be almost perfectly on trend, according to our very crude analysis.

Relevance

The nature of the industrial revolution is relevant to AI forecasting in the following ways:

• If growth during the industrial revolution is a highly improbable aberration from longer term trends, it suggests that it is a consequence of specific developments at the time, most saliently new technologies. This suggests that new technologies can sometimes alone cause changes at the level of the global economy.
• ‘The impact of the industrial revolution’ is sometimes used as a measure against which to compare consequences of AI developments. Thinking here may be sharpened by clarification on the nature of the industrial revolution. This use is likely related to the point above, where ‘the scale of the industrial revolution’ is taken to be a historically plausible scale of impact for the most ambitious technologies.
• If economic history is best understood as a sequence of ‘growth modes’, per Hanson 2000,⁴ the industrial revolution being one, this changes our best extrapolation to the future. For instance, we might expect the continuation of the current mode to be slower than in a continuously super-exponential model, but may also expect to meet a new growth mode at some point, which may be substantially faster (and have other characteristics recognizable from past ‘growth mode’ changes). See Hanson 2000 for more on this.

Notes

Updated Nov 9, 2020

The MacCready Gossamer Albatross:

• covered around 2.0—4.6 m/kJ
• and moved mass at around 0.1882 —0.4577 kg.m/J

Details

The MacCready Gossamer Albatross was a human-powered flying machine that crossed the English Channel in 1979.¹ The pilot pedaled the craft, seemingly as if on a bicycle. It had a gross mass of 100kg, flying across the channel,² and flew 35.7 km in 2 hours and 49 minutes.³ The crossing was difficult however, so it seems plausible that the Gossamer Albatross could fly more efficiently in better conditions.

We do not know the pilot’s average power output, however:

• Wikipedia claims at least 300W was required to fly the craft⁴
• Chung 2006, an engineering textbook, claims that the driver, a cyclist, could produce around 200W of power.⁵
• Our impression is that 200W is a common power output over houres for amateur cycling. For instance, one of our researchers is able to achieve this for three hours.⁶

The best documented human cycling wattage that we could easily find is from professional rider Giulio Ciccone who won a stage of the Tour de France, then uploaded power data to the fitness tracking site Strava.⁷ His performance suggests around 318W is a reasonable upper bound, supposing that the pilot of the Gossamer Albatross would have had lower performance.⁸

To find the energy used by the cyclist, we divided power output by typical efficiency for a human on a bicycle, which according to Wikipedia ranges from .18 to .26.⁹

Distance per Joule

For distance per energy this gives us a highest measure of:

35.7 km / ((200W * (2 hours + 49 minutes))/0.26) = 4,577 m/MJ

And a lowest measure of:

35.7 km / ((318W * (2 hours + 49 minutes))/0.18) = 1,993 m/MJ

Mass per Joule

For weight times distance per energy this gives us a highest measure of:

(100kg * 35.7 km) / ((200W * (2 hours + 49 minutes))/0.26) = 0.4577 kg*m/j

And a lowest measure of:

(100kg * 35.7 km) / ((318W * (2 hours + 49 minutes))/0.17) =  0.1882 kg*m/j

Notes

Updated Nov 5, 2020

The Boeing 747-400:

• covers around 0.0021m/kJ.
• and moves mass at around 0.39 – 0.83 kg.m/J

Details

The Boeing 747-400 is a 1987 passenger plane.¹

Distance per Joule

The plane uses 10.77 kg/km of fuel on a medium haul flight.² We do not know what type of fuel it uses, but typical values for aviation fuel are around 44MJ/kg.³ Thus to fly a kilometer, the plane needs 10.77 kg of fuel, which is 10.77 x 44 MJ = 474 MJ of fuel. This gives us 0.0021m/kJ.

Mass.distance per Joule

According to Wikipedia, the 747’s ‘operating empty weight’ is 183,523 kg and its ‘maximum take-off weight’ is 396,893 kg.⁴ We use the range 183,523 kg—396,893 kg since we do not know at what weight in that range the relevant speeds were measured.

We have:

• Distance per kilojoule: 0.0021m/kJ
• Mass: 183,523 kg—396,893 kg

This gives us a range of 0.39 – 0.83 kg.m/J

Notes

Updated Nov 5, 2020

The Airbus A320:

• covers around 0.0078 m/kJ
• and moves mass at around 0.33 – 0.61 kg.m/J

Details

The Airbus A320 is a 1987 passenger plane.¹

Distance per Joule

The plane uses 2.91 kg of fuel per km on a medium haul flight.² We do not know what type of fuel it uses, but typical values for aviation fuel are around 44MJ/kg.³ Thus to fly a kilometer, the plane needs 2.91kg of fuel, which is 2.91 x 44 MJ = 128MJ of fuel. This gives us 0.0078 m/kJ

Mass.distance per Joule

According to modernairliners.com, the A320’s ‘operating empty weight’ is 42,600 kg and its ‘maximum take-off weight’ is 78,000 kg.⁴ We use the range 42,600—78,000 kg, since we do not know at what weight in that range the relevant speeds were measured.

We have:

• Distance per kilojoule: 0.0078 m/kJ
• Mass: 42,600—78,000 kg

This gives us a range of 0.33 – 0.61 kg.m/J

Notes

Updated Nov 5, 2020

The North American P-51 Mustang:

• flew around 0.073—0.092 m/kJ
• and moved mass at around 0.25 – 0.50 kg.m/J

Details

The North American P-51 Mustang was a 1940 US WWII fighter and fighter-bomber.¹

Mass

According to Wikipedia²:

• Empty weight: 7,635 lb (3,465 kg)
• Gross weight: 9,200 lb (4,175 kg)
• Max takeoff weight: 12,100 lb (5,488 kg)

We use the range 3,465—5,488 kg, since we do not know at what weight in that range the relevant speeds were measured.

Distance per Joule

Wikipedia tells us that cruising speed was 362 mph (162 m/s)³

A table from WWII Aircraft Performance gives combinations of flight parameters apparently for a version of the P-51, however it has no title or description, so we cannot be confident. ⁴ We extracted some data from it here. This data suggests the best combination of parameters gives a fuel economy of 6.7 miles/gallon (10.8km)

We don’t know what fuel was used, but fuel energy density seems likely to be between 31—39 MJ/L = 117—148 MJ/gallon.⁵

Thus the plane flew about 10.8km on 117—148 MJ of fuel, for 0.073—0.092 m/kJ

Mass.distance per Joule

We have:

• Distance per kilojoule: 0.073—0.092 m/kJ
• Mass: 3,465—5,488 kg

This gives us a range of 0.25 – 0.50 kg.m/J

Notes

Updated Nov 5, 2020

The Vickers Vimy:

• flew around 0.025—0.050 m/kJ
• and moved mass at around 0.081 – 0.25 kg.m/J

Details

The Vickers Vimy was a 1917 British WWI bomber.¹ It was used in the first non-stop transatlantic flight.

Mass

According to Wikipedia²:

• Empty weight: 7,104 lb (3,222 kg)
• Max takeoff weight: 10,884 lb (4,937 kg)

We use the range 3,222—4,937 kg, since we do not know at what weight in that range the relevant speeds were measured.

Energy use per second

We also have:

• Power: 360 horsepower = 270 kW³
• Efficiency of use of energy from fuel: we did not find data on this, so use an estimate of 15%-30%, based on what we know about the energy efficiency of the Wright Flyer.

From these we can calculate:

Energy use per second
= power of engine x 1/efficiency in converting energy to engine power
= 270kJ/s / .15—270kJ/s / .30
= 900—1800 kJ/s

Distance per second

Wikipedia gives us:

• Maximum speed: 100 mph (160 km/h, 87 kn)⁴

Note that the figures for power do not obviously correspond to the highest measured speed. This is a rough estimate.

Distance per Joule

We now have (from above):

• speed = 100 miles/h = 44.7m/s
• energy use = 900—1800 kJ/s

Thus, on average each second the plane flies 44.7 m and uses 900—1800 kJ, for 0.025—0.050 m/kJ.

Mass.distance per Joule

We have:

• Distance per kilojoule: 0.025—0.050 m/kJ
• Mass: 3,222—4,937 kg

This gives us a range of 0.081 – 0.25 kg.m/J

Notes

Updated Nov 5, 2020

The Wright model B:

• flew around 0.10—0.21m/kJ
• and moved mass at around 0.036 – 0.12 kg.m/J

Details

The Wright Model B was a 1910 plane developed by the Wright Brothers.¹

Mass

According to Wikipedia²:

• Empty weight: 800 lb (363 kg)
• Gross weight: 1,250 lb (567 kg)

We use the range 363—567 kg, since we do not know at what weight in that range the relevant speeds were measured.

Energy use per second

From Wikipedia, we have³:

• Power: 35 horsepower = 26kW
• Efficiency of use of energy from fuel: we could not find data on this, so use an estimate of 15%-30%, based on what we know about the energy efficiency of the Wright Flyer.

From these we can calculate:

Energy use per second
= power of engine x 1/efficiency in converting energy to engine power
= 26kJ/s / .15—26kJ/s / .30
= 86.6—173 kJ/s

Distance per second

Wikipedia gives us⁴:

• Maximum speed: 45 mph (72 km/h, 39 kn)
• Cruise speed: 40 mph (64 km/h, 35 kn)

We use the cruise speed, as it seems more likely to represent speed achieved with the energy usages reported.

Distance per Joule

We now have (from above):

• speed = 40miles/h = 17.9m/s
• energy use = 86.6—173 kJ/s

Thus, on average each second the plane flies 17.9m and uses 86.6—173 kJ, for 0.10—0.21m/kJ.

Mass.distance per Joule

We have:

• Distance per kilojoule: 0.10—0.21m/kJ
• Mass: 363—567kg

This gives us a range of 0.036 – 0.12 kg.m/J

Notes