Vulnerable world hypothesis

Bostrom illustrated the hypothesis using an urn analogy. He likened the process of technological invention to drawing balls from an urn where the color of balls represents their impact. White balls are beneficial and constitute most of the balls drawn from the urn. Some balls are gray, which represent technologies with mixed or moderate effects. Black balls represent hypothetical technologies that tend to destroy by default the civilization that invents them. According to Bostrom, it is largely due to luck that humanity has not encountered a black ball yet, rather than carefulness or wisdom.^[3]

Bostrom defined the vulnerable world hypothesis as follows: "If technological development continues then a set of capabilities will at some point be attained that make the devastation of civilization extremely likely, unless civilization sufficiently exits the semi-anarchic default condition."^[1][4] The "semi-anarchic default condition" refers here to having:^[4][6]

1. Limited capacity for preventive policing.
2. Limited capacity for global governance.
3. Actors with diverse motivations^[a]

Bostrom defines "civilizational devastation" as an event at least as bad as the death of 15 per cent of the world population or a reduction in global GDP of more than 50 per cent lasting over a decade; it need not entail human extinction. He treats the truth of the hypothesis as an open question and cautions that further analysis is needed before drawing definitive policy conclusions.^[1]

To exemplify the vulnerabilities, Bostrom proposed a classification system and gave examples of how technology could have gone wrong, and policy recommendations such as differential technological development.^[3][4] If a technology that entails such a vulnerability is developed, the proposed responses (i.e. effective global governance or preventive policing depending on the type of vulnerability) are controversial.^[3][5][7] The classification includes:^[4][1]

• Type 0 ("surprising strangelets"): a technology carries a hidden risk and inadvertently devastates the civilization.

A proposed hypothetical example of this is if nuclear bombs had been able to ignite the atmosphere. Nuclear ignition was predicted not to occur for the Trinity nuclear test in a report commissioned by Robert Oppenheimer. Bostrom cited the report's conclusion: "One may conclude that the arguments of this paper make it unreasonable to expect that the N + N reaction could propagate. An unlimited propagation is even less likely. However, the complexity of the argument and the absence of satisfactory experimental foundation make further work on the subject highly desirable."^[1]

• Type 1 ("easy nukes"): a technology gives small groups of people the ability to cause mass destruction.

The "easy nukes" thought experiment proposed by Bostrom opens the question of what would have happened if nuclear chain reactions had been easier to produce, for example by "sending an electric current through a metal object placed between two sheets of glass."^[3]

• Type 2a ("safe first strike"): a technology has the potential to devastate the civilization, and powerful actors are incentivized to use it, potentially because using it first seems to bring an advantage.^[1]

• Type 2b ("worse global warming"): a great many actors face incentives to take some slightly damaging action such that the combined effect of those actions is civilizational devastation. This is a form of the tragedy of the commons.^[1]

According to Bostrom, pausing technological progress may not be possible or desirable. An alternative would be to prioritize the technologies that are expected to have a positive impact, and to delay those that may be catastrophic, a principle called differential technological development.^[3]

The potential solutions vary depending on the type of vulnerability. Dealing with type-2 vulnerabilities may require very effective governance and international cooperation. For type-1 vulnerabilities, if mass destruction ever becomes accessible to individuals, there may be at least some small fraction of the population that would use it.^[3] In extreme cases, Bostrom argues that mass surveillance might be required to avoid the destruction of civilization, a controversial prospect that has received significant media coverage.^{[8][9][10][11][12]}

Technologies that have been proposed as potential vulnerabilities are advanced artificial intelligence, nanotechnology and synthetic biology (synthetic biology may give the ability to easily create enhanced pandemics).^{[13][2][14][15]}

Writing in Vox, Kelsey Piper questioned Bostrom's optimism about universal surveillance, arguing that it would not resolve selective law enforcement or the criminalization of legitimate conduct. She also cited economist Robin Hanson's concern that stronger global governance could introduce a single point of failure and reduce competition between political systems.^[3]