Stanislav Petrov

20 results back to index

Risk: A User's Guide by Stanley McChrystal, Anna Butrico

"Hurricane Katrina" Superdome, Abraham Maslow, activist fund / activist shareholder / activist investor, airport security, Albert Einstein, Apollo 13, banking crisis, Bernie Madoff, Boeing 737 MAX, business process, cognitive dissonance, collapse of Lehman Brothers, collateralized debt obligation, computer vision, coronavirus, corporate governance, cotton gin, COVID-19, cuban missile crisis, deep learning, disinformation, don't be evil, Dr. Strangelove, fake news, fear of failure, George Floyd, Glass-Steagall Act, global pandemic, Googley, Greta Thunberg, hindsight bias, inflight wifi, invisible hand, iterative process, late fees, lockdown, Paul Buchheit, Ponzi scheme, QWERTY keyboard, ride hailing / ride sharing, Ronald Reagan, San Francisco homelessness, School Strike for Climate, Scientific racism, Silicon Valley, Silicon Valley startup, Skype, social distancing, source of truth, Stanislav Petrov, Steve Jobs, Thomas L Friedman, too big to fail, Travis Kalanick, wikimedia commons, work culture

five Minuteman intercontinental ballistic missiles: Chan, “Stanislav Petrov.” paused in shock: Chan, “Stanislav Petrov.” They understood well: Chan, “Stanislav Petrov”; Hoffman, “ ‘I Had a Funny Feeling in My Gut.’ ” Still, he knew the time: Chan, “Stanislav Petrov.” Petrov later explained: Chan, “Stanislav Petrov”; Hoffman, “ ‘I Had a Funny Feeling in My Gut.’ ” strange for the Americans to launch: Chan, “Stanislav Petrov”; Hoffman, “ ‘I Had a Funny Feeling in My Gut.’ ” “50-50” estimate of probability: Chan, “Stanislav Petrov.” early warning satellite constellation: Chan, “Stanislav Petrov”; Hoffman, “ ‘I Had a Funny Feeling in My Gut.’ ” he received a reprimand: Chan, “Stanislav Petrov.”

…

September 26, 1983: Sewell Chan, “Stanislav Petrov, Soviet Officer Who Helped Avert Nuclear War, Is Dead at 77,” The New York Times, September 18, 2017, https://nytimes.com/2017/09/18/world/europe/stanislav-petrov-nuclear-war-dead.html. no direct authority to initiate: David Hoffman, “ ‘I Had a Funny Feeling in My Gut,’ ” The Washington Post Foreign Service, February 10, 1999, p. A19, https://washingtonpost.com/wp-srv/inatl/longterm/coldwar/soviet10.htm; Chan, “Stanislav Petrov.” single phone call of warning: Chan, “Stanislav Petrov.” five Minuteman intercontinental ballistic missiles: Chan, “Stanislav Petrov.” paused in shock: Chan, “Stanislav Petrov.”

…

—Carl Sagan, astronomy professor, science writer, host of THE PBS TELEVISION SERIES cosmos Technology raises a new question: Who or what is in control? Sitting on a Hot Frying Pan Rule breaking makes for great storytelling: American children know the tale of the Boston Tea Party and lunch counter sit-ins of the 1960s from a young age, just as Chinese citizens know the story of Chairman Mao and the Long March. But the name Stanislav Petrov is at most a historical curiosity, despite his world-changing act of disobedience. Arguably, September 26, 1983, witnessed the single most important act of insubordination in the history of human civilization. On that fateful day, Lieutenant Colonel Petrov of the Soviet Union’s Air Defense Forces was stationed at a command center outside Moscow.

Army of None: Autonomous Weapons and the Future of War by Paul Scharre

"World Economic Forum" Davos, active measures, Air France Flight 447, air gap, algorithmic trading, AlphaGo, Apollo 13, artificial general intelligence, augmented reality, automated trading system, autonomous vehicles, basic income, Black Monday: stock market crash in 1987, brain emulation, Brian Krebs, cognitive bias, computer vision, cuban missile crisis, dark matter, DARPA: Urban Challenge, data science, deep learning, DeepMind, DevOps, Dr. Strangelove, drone strike, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, facts on the ground, fail fast, fault tolerance, Flash crash, Freestyle chess, friendly fire, Herman Kahn, IFF: identification friend or foe, ImageNet competition, information security, Internet of things, Jeff Hawkins, Johann Wolfgang von Goethe, John Markoff, Kevin Kelly, Korean Air Lines Flight 007, Loebner Prize, loose coupling, Mark Zuckerberg, military-industrial complex, moral hazard, move 37, mutually assured destruction, Nate Silver, Nick Bostrom, PalmPilot, paperclip maximiser, pattern recognition, Rodney Brooks, Rubik’s Cube, self-driving car, sensor fusion, South China Sea, speech recognition, Stanislav Petrov, Stephen Hawking, Steve Ballmer, Steve Wozniak, Strategic Defense Initiative, Stuxnet, superintelligent machines, Tesla Model S, The Signal and the Noise by Nate Silver, theory of mind, Turing test, Tyler Cowen, universal basic income, Valery Gerasimov, Wall-E, warehouse robotics, William Langewiesche, Y2K, zero day

Introduction: The Power Over Life and Death 1 shot down a commercial airliner: Thom Patterson, “The downing of Flight 007: 30 years later, a Cold War tragedy still seems surreal,” CNN.com, August 31, 2013, http://www.cnn.com/2013/08/31/us/kal-fight-007-anniversary/index.html. 1 Stanislav Petrov: David Hoffman, “ ‘I Had a Funny Feeling in My Gut,’ ” Washington Post, February 10, 1999, http://www.washingtonpost.com/wp-srv/inatl/longterm/coldwar/shatter021099b.htm. 1 red backlit screen: Pavel Aksenov, “Stanislav Petrov: The Man Who May Have Saved the World,” BBC.com, September 26, 2013, http://www.bbc.com/news/world-europe-24280831. 2 five altogether: Ibid. 2 Petrov had a funny feeling: Hoffman, “I Had a Funny Feeling in My Gut.’” 2 Petrov put the odds: Aksenov, “Stanislav Petrov: The Man Who May Have Saved the World.” 5 Sixteen nations already have armed drones: The United States, United Kingdom, Israel, China, Nigeria, Iran, Iraq, Jordan, Egypt, United Arab Emirates, Saudi Arabia, Kazakhstan, Turkmenistan, Pakistan, Myanmar, Turkey.

…

Fearing retaliation, the Soviet Union was on alert. The Soviet Union deployed a satellite early warning system called Oko to watch for U.S. missile launches. Just after midnight on September 26, the system issued a grave report: the United States had launched a nuclear missile at the Soviet Union. Lieutenant Colonel Stanislav Petrov was on duty that night in bunker Serpukhov-15 outside Moscow, and it was his responsibility to report the missile launch up the chain of command to his superiors. In the bunker, sirens blared and a giant red backlit screen flashed “launch,” warning him of the detected missile, but still Petrov was uncertain.

…

Yet computers still fall far short of humans in understanding context and interpreting meaning. AI programs today can identify objects in images, but can’t draw these individual threads together to understand the big picture. Some decisions in war are straightforward. Sometimes the enemy is easily identified and the shot is clear. Some decisions, however, like the one Stanislav Petrov faced, require understanding the broader context. Some situations, like the one my sniper team encountered, require moral judgment. Sometimes doing the right thing entails breaking the rules—what’s legal and what’s right aren’t always the same. THE DEBATE Humanity faces a fundamental question: should machines be allowed to make life-and-death decisions in war?

1983: Reagan, Andropov, and a World on the Brink by Taylor Downing

Able Archer 83, active measures, anti-communist, Ayatollah Khomeini, Berlin Wall, Boeing 747, cuban missile crisis, disinformation, Donald Trump, Dr. Strangelove, fake news, Fall of the Berlin Wall, full employment, Herman Kahn, Korean Air Lines Flight 007, kremlinology, Mikhail Gorbachev, military-industrial complex, mutually assured destruction, nuclear paranoia, nuclear winter, RAND corporation, Robert Hanssen: Double agent, Ronald Reagan, Ronald Reagan: Tear down this wall, Seymour Hersh, Stanislav Petrov, Strategic Defense Initiative, Vladimir Vetrov: Farewell Dossier, Yom Kippur War

Serpukhov-15 could provide around twelve minutes’ additional warning time of an attack over the radars stationed along the northern fringes of the Soviet Union. These extra minutes would be vital in giving the leadership time in which to respond appropriately; and to get to a shelter.4 At 7 p.m. on Monday, 26 September 1983, Lieutenant-Colonel Stanislav Petrov arrived at the compound’s command centre having been asked at the last minute to take over a shift as another officer had reported in ill. At the centre he spent an hour talking with colleagues whose shift was finishing, receiving the latest reports and updates from them. Then Petrov ordered his team of twelve specialist officers to salute the Soviet flag and take up their positions in the control room.

…

By the time the first journalists got to Petrov he was living a wretched life in a ghastly tower block in a Moscow suburb. But on the black and white television set in his tiny apartment there was a small statue of a globe resting inside an open hand which Petrov proudly pointed out to visitors. It was inscribed with a few words from Kofi Annan, the Secretary General of the United Nations: ‘To Stanislav Petrov, The Man Who Saved the World’. By the autumn of 1983, the two superpowers possessed about 18,400 nuclear warheads. They were on the tips of giant intercontinental missiles held in silos spread right across North America and the Soviet Union. Submarines carrying nuclear missiles crossed every ocean, and heavy bombers fully armed with nuclear bombs were constantly on stand-by.

…

But it also made the possibility of miscalculation even greater. Only six weeks before, the Soviet early warning system had malfunctioned by interpreting reflections of the sun on clouds in the Midwest of the United States as a sign that missiles had been launched. On that occasion the cool head of Stanislav Petrov had defused the situation, but there was no guarantee that the officer on duty would respond in the same way to another early warning. The entire Soviet nuclear launch system was resting on a knife edge. Either man or machine could interpret a situation wrongly and the consequence would be nuclear war by miscalculation.

Know Thyself by Stephen M Fleming

Abraham Wald, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, AlphaGo, autism spectrum disorder, autonomous vehicles, availability heuristic, backpropagation, citation needed, computer vision, confounding variable, data science, deep learning, DeepMind, Demis Hassabis, Douglas Hofstadter, Dunning–Kruger effect, Elon Musk, Estimating the Reproducibility of Psychological Science, fake news, global pandemic, higher-order functions, index card, Jeff Bezos, l'esprit de l'escalier, Lao Tzu, lifelogging, longitudinal study, meta-analysis, mutually assured destruction, Network effects, patient HM, Pierre-Simon Laplace, power law, prediction markets, QWERTY keyboard, recommendation engine, replication crisis, self-driving car, side project, Skype, Stanislav Petrov, statistical model, theory of mind, Thomas Bayes, traumatic brain injury

Some psychologists restrict the term self-awareness to mean bodily self-awareness, or awareness of the location and appearance of the body, but here I am generally concerned with awareness of mental states. Chapter 1: How to Be Uncertain 1. Jonathan Steele, “Stanislav Petrov Obituary,” The Guardian, October 11, 2017, www.theguardian.com/world/2017/oct/11/stanislav-petrov-obituary. 2. Green and Swets (1966). 3. The seeds of Bayes’s rule were first identified by the eleventh-century Arabic mathematician Ibn al-Haytham, developed by English clergyman and mathematician Thomas Bayes in 1763, and applied to a range of scientific problems by the eighteenth-century French mathematician Pierre-Simon Laplace.

…

PART I BUILDING MINDS THAT KNOW THEMSELVES 1 HOW TO BE UNCERTAIN The other fountain [of] ideas, is the perception of the operation of our own minds within us.… And though it be not sense, as having nothing to do with external objects, yet it is very like it, and might properly enough be called internal sense. —JOHN LOCKE, Essay Concerning Human Understanding, Book II Is something there, or not? This was the decision facing Stanislav Petrov one early morning in September 1983. Petrov was a lieutenant colonel in the Soviet Air Defense Forces and in charge of monitoring early warning satellites. It was the height of the Cold War between the United States and Russia, and there was a very real threat that long-range nuclear missiles could be launched by either side.

The Alignment Problem: Machine Learning and Human Values by Brian Christian

Albert Einstein, algorithmic bias, Alignment Problem, AlphaGo, Amazon Mechanical Turk, artificial general intelligence, augmented reality, autonomous vehicles, backpropagation, butterfly effect, Cambridge Analytica, Cass Sunstein, Claude Shannon: information theory, computer vision, Computing Machinery and Intelligence, data science, deep learning, DeepMind, Donald Knuth, Douglas Hofstadter, effective altruism, Elaine Herzberg, Elon Musk, Frances Oldham Kelsey, game design, gamification, Geoffrey Hinton, Goodhart's law, Google Chrome, Google Glasses, Google X / Alphabet X, Gödel, Escher, Bach, Hans Moravec, hedonic treadmill, ImageNet competition, industrial robot, Internet Archive, John von Neumann, Joi Ito, Kenneth Arrow, language acquisition, longitudinal study, machine translation, mandatory minimum, mass incarceration, multi-armed bandit, natural language processing, Nick Bostrom, Norbert Wiener, Northpointe / Correctional Offender Management Profiling for Alternative Sanctions, OpenAI, Panopticon Jeremy Bentham, pattern recognition, Peter Singer: altruism, Peter Thiel, precautionary principle, premature optimization, RAND corporation, recommendation engine, Richard Feynman, Rodney Brooks, Saturday Night Live, selection bias, self-driving car, seminal paper, side project, Silicon Valley, Skinner box, sparse data, speech recognition, Stanislav Petrov, statistical model, Steve Jobs, strong AI, the map is not the territory, theory of mind, Tim Cook: Apple, W. E. B. Du Bois, Wayback Machine, zero-sum game

The world will be, for good or ill, full of these algorithmic two-year-olds, walking up to us, opening the doors they think we might want opened, trying, in their various ways, to help. 9 UNCERTAINTY Most of the greatest evils that man has inflicted upon man have come through people feeling quite certain about something which, in fact, was false. —BERTRAND RUSSELL1 I beseech you, in the bowels of Christ, think it possible that you may be mistaken. —OLIVER CROMWELL The spirit of liberty is the spirit which is not too sure that it is right. —LEARNED HAND It was September 26, 1983, just after midnight, and Soviet duty officer Stanislav Petrov was in a bunker outside of Moscow, monitoring the Oko early-warning satellite system. Suddenly the screen lit up and sirens began howling. There was an LGM-30 Minuteman intercontinental ballistic missile inbound, it said, from the United States. “Giant blood-red letters appeared on our main screen,” he says.

…

I marvel that this manuscript was written using typesetting software more than 40 years old, and for which none other than Arthur Samuel himself wrote the documentation. We really do stand on the shoulders of giants. Humbly, I want to acknowledge those who passed away during the writing of this book whose voices I would have loved to include, and whose ideas are nevertheless present: Derek Parfit, Kenneth Arrow, Hubert Dreyfus, Stanislav Petrov, and Ursula K. Le Guin. I want to express a particular gratitude to the University of California, Berkeley. To CITRIS, where I was honored to be a visiting scholar during the writing of this book, with very special thanks to Brandie Nonnecke and Camille Crittenden; to the Simons Institute for the Theory of Computing, in particular Kristin Kane and Richard Karp; to the Center for Human-Compatible AI, in particular Stuart Russell and Mark Nitzberg; and to the many brilliant and spirited members and visitors of the CHAI Workshop.

…

See Bateson, Nettle, and Roberts, “Cues of Being Watched Enhance Cooperation in a Real-World Setting,” and Heine et al., “Mirrors in the Head.” 58. Bentham, “Letter to Jacques Pierre Brissot de Warville.” 59. Bentham, “Preface.” CHAPTER 9. UNCERTAINTY 1. Russell, “Ideas That Have Harmed Mankind.” 2. “Another Day the World Almost Ended.” 3. Aksenov, “Stanislav Petrov.” 4. Aksenov. 5. Hoffman, “‘I Had a Funny Feeling in My Gut.’ ” 6. Nguyen, Yosinski, and Clune, “Deep Neural Networks Are Easily Fooled.” For a discussion of the confidence of predictions of neural networks, see Guo et al., “On Calibration of Modern Neural Networks.” 7. See Szegedy et al., “Intriguing Properties of Neural Networks,” and Goodfellow, Shlens, and Szegedy, “Explaining and Harnessing Adversarial Examples.”

Who Rules the World? by Noam Chomsky

Able Archer 83, Alan Greenspan, Albert Einstein, anti-communist, Ayatollah Khomeini, Berlin Wall, Bretton Woods, British Empire, capital controls, classic study, corporate governance, corporate personhood, cuban missile crisis, deindustrialization, Donald Trump, Doomsday Clock, Edward Snowden, en.wikipedia.org, facts on the ground, failed state, Fall of the Berlin Wall, Garrett Hardin, high-speed rail, Howard Zinn, illegal immigration, Intergovernmental Panel on Climate Change (IPCC), invisible hand, liberation theology, Malacca Straits, Martin Wolf, Mikhail Gorbachev, Monroe Doctrine, Nelson Mandela, nuclear winter, Occupy movement, oil shale / tar sands, one-state solution, Plutonomy: Buying Luxury, Explaining Global Imbalances, precariat, public intellectual, Ralph Waldo Emerson, Robert Solow, Ronald Reagan, South China Sea, Stanislav Petrov, Strategic Defense Initiative, structural adjustment programs, The Theory of the Leisure Class by Thorstein Veblen, The Wealth of Nations by Adam Smith, Thorstein Veblen, too big to fail, trade route, Tragedy of the Commons, union organizing, uranium enrichment, wage slave, WikiLeaks, working-age population

The exercises “almost became a prelude to a preventative nuclear strike,” according to an account in the Journal of Strategic Studies.10 It was even more dangerous than that, as we learned in the fall of 2013, when the BBC reported that right in the midst of these world-threatening developments, Russia’s early-warning systems detected an incoming missile strike from the United States, sending its nuclear system onto the highest-level alert. The protocol for the Soviet military was to retaliate with a nuclear attack of its own. Fortunately, the officer on duty, Stanislav Petrov, decided to disobey orders and not report the warnings to his superiors. He received an official reprimand. And thanks to his dereliction of duty, we’re still alive to talk about it.11 The security of the population was no more a high priority for Reagan administration planners than for their predecessors.

…

The Pentagon’s risky measures included sending U.S. strategic bombers over the North Pole to test Soviet radar, and naval exercises in wartime approaches to the USSR where U.S. warships had previously not entered. Additional secret operations simulated surprise naval attacks on Soviet targets.”11 We now know that the world was saved from likely nuclear destruction in those frightening days by the decision of a Russian officer, Stanislav Petrov, not to transmit to higher authorities the report of automated detection systems that the USSR was under missile attack. Accordingly, Petrov takes his place alongside Russian submarine commander Vasili Arkhipov, who, at a dangerous moment of the 1962 Cuban Missile Crisis, refused to authorize the launching of nuclear torpedoes when the subs were under attack by U.S. destroyers enforcing a quarantine.

…

Fischer, “A Cold War Conundrum: The 1983 Soviet War Scare,” Center for the Study of Intelligence, 7 July 2008, https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/a-cold-war-conundrum/source.htm; Dmitry Dima Adamsky, “The 1983 Nuclear Crisis—Lessons for Deterrence Theory and Practice,” Journal of Strategic Studies 36, no.1 (2013): 4–41. 11. Pavel Aksenov, “Stanislav Petrov: The Man Who May Have Saved the World,” BBC News Europe, 26 September 2013, http://www.bbc.com/news/world-europe-24280831. 12. Eric Schlosser, Command and Control: Nuclear Weapons, the Damascus Accident, and the Illusion of Safety (New York: Penguin, 2013). 13. President Bill Clinton, Speech before the UN General Assembly, 27 September 1993, http://www.state.gov/p/io/potusunga/207375.htm; Secretary of Defense William Cohen, Annual Report to the President and Congress: 1999 (Washington, DC: Department of Defense, 1999) http://history.defense.gov/Portals/70/Documents/annual_reports/1999_DoD_AR.pdf. 14.

Hello World: Being Human in the Age of Algorithms by Hannah Fry

23andMe, 3D printing, Air France Flight 447, Airbnb, airport security, algorithmic bias, algorithmic management, augmented reality, autonomous vehicles, backpropagation, Brixton riot, Cambridge Analytica, chief data officer, computer vision, crowdsourcing, DARPA: Urban Challenge, data science, deep learning, DeepMind, Douglas Hofstadter, driverless car, Elon Musk, fake news, Firefox, Geoffrey Hinton, Google Chrome, Gödel, Escher, Bach, Ignaz Semmelweis: hand washing, John Markoff, Mark Zuckerberg, meta-analysis, Northpointe / Correctional Offender Management Profiling for Alternative Sanctions, pattern recognition, Peter Thiel, RAND corporation, ransomware, recommendation engine, ride hailing / ride sharing, selection bias, self-driving car, Shai Danziger, Silicon Valley, Silicon Valley startup, Snapchat, sparse data, speech recognition, Stanislav Petrov, statistical model, Stephen Hawking, Steven Levy, systematic bias, TED Talk, Tesla Model S, The Wisdom of Crowds, Thomas Bayes, trolley problem, Watson beat the top human players on Jeopardy!, web of trust, William Langewiesche, you are the product

There will be times when we have to hand over control to the unknown, even while knowing that the algorithm is capable of making mistakes. Times when we are forced to weigh up our own judgement against that of the machine. When, if we decide to trust our instincts instead of its calculations, we’re going to need rather a lot of courage in our convictions. When to over-rule Stanislav Petrov was a Russian military officer in charge of monitoring the nuclear early warning system protecting Soviet airspace. His job was to alert his superiors immediately if the computer indicated any sign of an American attack.34 Petrov was on duty on 26 September 1983 when, shortly after midnight, the sirens began to howl.

…

Kristine Phillips, ‘The former Soviet officer who trusted his gut – and averted a global nuclear catastrophe’, Washington Post, 18 Sept. 2017, https://www.washingtonpost.com/news/retropolis/wp/2017/09/18/the-former-soviet-officer-who-trusted-his-gut-and-averted-a-global-nuclear-catastrophe/?utm_term=.6546e0f06cce. 35. Pavel Aksenov, ‘Stanislav Petrov: the man who may have saved the world’, BBC News, 26 Sept. 2013, http://www.bbc.co.uk/news/world-europe-24280831. 36. Ibid. 37. Stephen Flanagan, Re: Accident at Smiler Rollercoaster, Alton Towers, 2 June 2015: Expert’s Report, prepared at the request of the Health and Safety Executive, Oct. 2015, http://www.chiark.greenend.org.uk/~ijackson/2016/Expert%20witness%20report%20from%20Steven%20Flanagan.pdf. 38.

The Dead Hand: The Untold Story of the Cold War Arms Race and Its Dangerous Legacy by David Hoffman

Able Archer 83, active measures, anti-communist, banking crisis, Berlin Wall, Boeing 747, Chuck Templeton: OpenTable:, crony capitalism, cuban missile crisis, disinformation, failed state, guns versus butter model, It's morning again in America, joint-stock company, Kickstarter, Korean Air Lines Flight 007, launch on warning, Mikhail Gorbachev, military-industrial complex, mutually assured destruction, nuclear winter, Oklahoma City bombing, radical decentralization, Robert Hanssen: Double agent, rolodex, Ronald Reagan, Ronald Reagan: Tear down this wall, Seymour Hersh, Silicon Valley, standardized shipping container, Stanislav Petrov, Strategic Defense Initiative, Thomas L Friedman, undersea cable, uranium enrichment, Vladimir Vetrov: Farewell Dossier, warehouse robotics, zero-sum game

See Bunn, p. 116. 11 Valentin Yevstigneev, interview, Feb. 10, 2005. Yevstigneev's comment repeated the claim made in an article published May 23, 2001, in the Russian newspaper Nezavisamaya Gazeta. Stanislav Petrov, the general in charge of chemical weapons, was a coauthor. The piece claimed the Sverdlovsk anthrax outbreak was the result of "subversive activity" against the Soviet Union. Stanislav Petrov et al., "Biologicheskaya Diversia Na Urale" [Biological Sabotage in the Urals], NG, May 23, 1001. 12 The closed military facilities are: the Scientific-Research Institute of Microbiology of the Ministry of Defense of the Russian Federation, Kirov, which is the main biological weapons facility of the military; the Virology Center of the Scientific-Research Institute of Microbiology of the Ministry of Defense, Sergiev Posad; and the Department of Military Epidemiology of the Scientific Research Institute of Microbiology of the Ministry of Defense, Yekaterinburg. 13 When the United States and Russia signed the Chemical Weapons Convention in 1997 they promised to destroy stocks of chemical weapons by 2012.

…

But he also gave the pathologists a private signal to hide and protect their evidence. Nikiforov later died of a heart attack. "We are certain that he knew the truth," Grinberg said.9 But the people of the Soviet Union and the outside world did not. II. Night Watch for Nuclear War The shift change began at 7 P.M. on September 26, 1983. Stanislav Petrov, a lieutenant colonel, arrived at Serpukhov-15, south of Moscow, a top-secret missile attack early-warning station, which received signals from satellites. Petrov changed from street clothes into the soft uniform of the military space troops of the Soviet Union. Over the next hour, he and a dozen other specialists asked questions of the outgoing officers.

…

Little is known about the disposition in Russia of the thousands of tactical nuclear weapons removed from Eastern Europe and the former Soviet republics after the 1991 Bush-Gorbachev initiative. They may be in storage or deployed; they have never been covered by any treaty, nor any verification regime, and the loss of just one could be catastrophic.2 Another step would be to take the remaining strategic nuclear weapons off launch-ready alert. When Stanislav Petrov faced the false alarm in 1983, launch decisions had to be made in just minutes. Today, Russia is no longer the ideological or military threat the Soviet Union once was; nor does the United States pose such a threat to Russia. Americans invested much time and effort to assist Russia's leap to capitalism in the 1990s--should we aim our missiles now at the very stock markets in Moscow we helped design?

I, Warbot: The Dawn of Artificially Intelligent Conflict by Kenneth Payne

Abraham Maslow, AI winter, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, AlphaGo, anti-communist, Any sufficiently advanced technology is indistinguishable from magic, artificial general intelligence, Asperger Syndrome, augmented reality, Automated Insights, autonomous vehicles, backpropagation, Black Lives Matter, Bletchley Park, Boston Dynamics, classic study, combinatorial explosion, computer age, computer vision, Computing Machinery and Intelligence, coronavirus, COVID-19, CRISPR, cuban missile crisis, data science, deep learning, deepfake, DeepMind, delayed gratification, Demis Hassabis, disinformation, driverless car, drone strike, dual-use technology, Elon Musk, functional programming, Geoffrey Hinton, Google X / Alphabet X, Internet of things, job automation, John Nash: game theory, John von Neumann, Kickstarter, language acquisition, loss aversion, machine translation, military-industrial complex, move 37, mutually assured destruction, Nash equilibrium, natural language processing, Nick Bostrom, Norbert Wiener, nuclear taboo, nuclear winter, OpenAI, paperclip maximiser, pattern recognition, RAND corporation, ransomware, risk tolerance, Ronald Reagan, self-driving car, semantic web, side project, Silicon Valley, South China Sea, speech recognition, Stanislav Petrov, stem cell, Stephen Hawking, Steve Jobs, strong AI, Stuxnet, technological determinism, TED Talk, theory of mind, TikTok, Turing machine, Turing test, uranium enrichment, urban sprawl, V2 rocket, Von Neumann architecture, Wall-E, zero-sum game

The problem, it seems, was the moon: the ultra-sensitive radar was picking up an echo from its surface and attributing that to an incoming intercontinental ballistic missile. Presumably the boffins eventually managed to ratchet down the sensitivity to a more appropriate level. A similar panic happened on the other side of the Iron Curtain, this more widely known. In 1982 at a time of heightened tension in the Cold War, an unflappable Soviet Colonel, Stanislav Petrov chose to ignore the alarm from his new state-of-the-art early warning system which signalled that the US had launched intercontinental missiles. Happily, in both cases, humans were resolutely in the loop, and sufficiently sceptical about the ability of technology to interpret reality. Ellsberg features again in our discussion of strategic AI, thanks to his vivid depiction of his job as a mid-level Pentagon functionary in the early 1960s.4 In this pre-computer era, the job of the intelligence analyst involved wading through a large stack of papers.

…

In the heat of the moment, we set doubts aside and let ourselves be guided by the machine, like the captain of the USS Vincennes failing to interrogate the ship’s targeting computer. On the other hand, in some circumstances, there is often a reluctance to trust the workings of an intelligent machine. That might especially be the case if it goes against our ingrained preferences. Stanislav Petrov famously refusing to believe his faulty nuclear warning computer is the shining example here. Which one of those two instincts predominates in any centaur strategy team is a question of psychology, not just computer science. The US Army is testing it out already—at the tactical level. In a fascinating recent experiment, the Army Research Lab set up cameras pointed at truck drivers to measure their reactions as they drove a convoy.10 The humans could decide whether to engage their AI assistant, and then their responses to its decisions could be measured with facial recognition software and compared to their personality profiles.

The AI Economy: Work, Wealth and Welfare in the Robot Age by Roger Bootle

"World Economic Forum" Davos, 3D printing, agricultural Revolution, AI winter, Albert Einstein, AlphaGo, Alvin Toffler, anti-work, antiwork, autonomous vehicles, basic income, Ben Bernanke: helicopter money, Bernie Sanders, Bletchley Park, blockchain, call centre, Cambridge Analytica, Capital in the Twenty-First Century by Thomas Piketty, Carl Icahn, Chris Urmson, computer age, Computing Machinery and Intelligence, conceptual framework, corporate governance, correlation does not imply causation, creative destruction, David Ricardo: comparative advantage, deep learning, DeepMind, deindustrialization, Demis Hassabis, deskilling, Dr. Strangelove, driverless car, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, everywhere but in the productivity statistics, facts on the ground, fake news, financial intermediation, full employment, future of work, Future Shock, general purpose technology, Great Leap Forward, Hans Moravec, income inequality, income per capita, industrial robot, Internet of things, invention of the wheel, Isaac Newton, James Watt: steam engine, Jeff Bezos, Jeremy Corbyn, job automation, job satisfaction, John Markoff, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John von Neumann, Joseph Schumpeter, Kevin Kelly, license plate recognition, low interest rates, machine translation, Marc Andreessen, Mark Zuckerberg, market bubble, mega-rich, natural language processing, Network effects, new economy, Nicholas Carr, Ocado, Paul Samuelson, Peter Thiel, Phillips curve, positional goods, quantitative easing, RAND corporation, Ray Kurzweil, Richard Florida, ride hailing / ride sharing, rising living standards, road to serfdom, Robert Gordon, Robert Shiller, Robert Solow, Rutger Bregman, Second Machine Age, secular stagnation, self-driving car, seminal paper, Silicon Valley, Silicon Valley billionaire, Simon Kuznets, Skype, social intelligence, spinning jenny, Stanislav Petrov, Stephen Hawking, Steven Pinker, synthetic biology, technological singularity, The Future of Employment, The Wealth of Nations by Adam Smith, Thomas Malthus, trade route, universal basic income, US Airways Flight 1549, Vernor Vinge, warehouse automation, warehouse robotics, Watson beat the top human players on Jeopardy!, We wanted flying cars, instead we got 140 characters, wealth creators, winner-take-all economy, world market for maybe five computers, Y2K, Yogi Berra

At a time when tensions were already high because of the shooting down by the Soviet Union of a Korean aircraft, killing 269 passengers, a Soviet early-warning system reported the launch by the USA of five missiles at the Soviet Union. The Soviet officer responsible for this early-warning system, Stanislav Petrov, had minutes to decide what to do. The protocol required that he should report this as a nuclear attack. Instead, he relied on his gut instinct. If America was really launching a nuclear attack, he reasoned, why had it sent only five missiles? So, he decided that it was a false alarm and took no action.

…

It turned out that a Soviet satellite had misread the sun’s reflections off cloud tops for flares from rocket engines. It is now widely acknowledged that Petrov’s judgment saved the world from nuclear catastrophe. Despite involving the saving of the world from a real disaster, this true story has a bitter ending. Stanislav Petrov was sacked for disobeying orders and lived the rest of his life in drab obscurity. (That was the Soviet Union for you.) Could we imagine entrusting the sort of judgment that Petrov made to some form of AI? This conjures up the scenario depicted in the now very old but still rewarding film Dr.

The Glass Half-Empty: Debunking the Myth of Progress in the Twenty-First Century by Rodrigo Aguilera

"Friedman doctrine" OR "shareholder theory", "World Economic Forum" Davos, activist fund / activist shareholder / activist investor, Alan Greenspan, Anthropocene, availability heuristic, barriers to entry, basic income, benefit corporation, Berlin Wall, Bernie Madoff, Bernie Sanders, bitcoin, Boris Johnson, Branko Milanovic, Bretton Woods, Brexit referendum, Capital in the Twenty-First Century by Thomas Piketty, capitalist realism, carbon footprint, Carmen Reinhart, centre right, clean water, cognitive bias, collapse of Lehman Brothers, Colonization of Mars, computer age, Corn Laws, corporate governance, corporate raider, creative destruction, cryptocurrency, cuban missile crisis, David Graeber, David Ricardo: comparative advantage, death from overwork, decarbonisation, deindustrialization, Deng Xiaoping, Doha Development Round, don't be evil, Donald Trump, Doomsday Clock, Dunning–Kruger effect, Elon Musk, European colonialism, fake news, Fall of the Berlin Wall, first-past-the-post, Francis Fukuyama: the end of history, fundamental attribution error, gig economy, Gini coefficient, Glass-Steagall Act, Great Leap Forward, green new deal, Hans Rosling, housing crisis, income inequality, income per capita, index fund, intangible asset, Intergovernmental Panel on Climate Change (IPCC), invisible hand, Jean Tirole, Jeff Bezos, Jeremy Corbyn, Jevons paradox, job automation, job satisfaction, John Maynard Keynes: Economic Possibilities for our Grandchildren, joint-stock company, Joseph Schumpeter, karōshi / gwarosa / guolaosi, Kenneth Rogoff, Kickstarter, lake wobegon effect, land value tax, Landlord’s Game, late capitalism, liberal capitalism, long peace, loss aversion, low interest rates, Mark Zuckerberg, market fundamentalism, means of production, meta-analysis, military-industrial complex, Mont Pelerin Society, moral hazard, moral panic, neoliberal agenda, Network effects, North Sea oil, Northern Rock, offshore financial centre, opioid epidemic / opioid crisis, Overton Window, Pareto efficiency, passive investing, Peter Thiel, plutocrats, principal–agent problem, profit motive, public intellectual, purchasing power parity, race to the bottom, rent-seeking, risk tolerance, road to serfdom, Robert Shiller, Robert Solow, savings glut, Scientific racism, secular stagnation, Silicon Valley, Silicon Valley ideology, Slavoj Žižek, Social Justice Warrior, Social Responsibility of Business Is to Increase Its Profits, sovereign wealth fund, Stanislav Petrov, Steven Pinker, structural adjustment programs, surveillance capitalism, tail risk, tech bro, TED Talk, The Spirit Level, The Wealth of Nations by Adam Smith, too big to fail, trade liberalization, transatlantic slave trade, trolley problem, unbiased observer, universal basic income, Vilfredo Pareto, Washington Consensus, Winter of Discontent, Y2K, young professional, zero-sum game

A senior State Department official was quoted during this time as saying that “false alerts of this kind are not a rare occurrence” and that there is a “complacency about handling them that disturbs me”.3 And in September 1983, probably the second closest call came when another Soviet officer, Stanislav Petrov, disobeyed protocol after an early warning system detected a US missile strike which he deliberately failed to report to his superiors. It was, as he correctly assumed, a false alarm. He later admitted that he was only “50–50” sure at the time that the attack was not real.4 What is arguably more complacent is the assumption — possibly a lethal one for a large chunk of humanity — that a repeat of this situation could not take place again.

…

., “Soviets Close to Using A-Bomb in 1962 Crisis, Forum Is Told”, Boston Globe, 13 Oct. 2002, http://www.latinamericanstudies.org/cold-war/sovietsbomb.htm 2 Pinker, S., Enlightenment Now, pg. 312 3 “False Warnings of Soviet Missile Attacks during 1979-80 Led to Alert Actions for U.S. Strategic Forces”, National Security Archive, 1 Mar. 2012, https://nsarchive2.gwu.edu/nukevault/ebb371/ 4 Aksenov, P., “Stanislav Petrov: The Man Who May Have Saved the World”, BBC News, 26 Sep. 2013, https://www.bbc.co.uk/news/world-europe-24280831 5 Roser, M., “Nuclear Weapons”, Our World in Data, https://ourworldindata.org/nuclear-weapons 6 Nolen, S., “Don’t Talk to Me About Justice”, Globe and Mail, 3 Apr. 2004, https://www.theglobeandmail.com/news/world/dont-talk-to-me-about-justice/article1135480/ 7 “More Mexicans Leaving Than Coming to the U.S.”, Pew Research Center, 19 Nov. 2015, https://www.pewresearch.org/hispanic/2015/11/19/more-mexicans-leaving-than-coming-to-the-u-s/ 8 Gomez, J.M et al., “The Phylogenic Roots of Human Violence”, Nature, 538, 13 Oct. 2016, https://doi.org/10.1038/nature19758 9 Ibid. 10 Falk, D., and Hildebolt, C., “Annual War Deaths in Small-Scale versus State Societies Scale with Population Size Rather than Violence”, Current Anthropology, 58(6), Dec. 2017, https://doi.org/10.1086/694568 11 Fazal, T.M., “Dead Wrong?

Surviving AI: The Promise and Peril of Artificial Intelligence by Calum Chace

3D printing, Ada Lovelace, AI winter, Airbnb, Alvin Toffler, artificial general intelligence, augmented reality, barriers to entry, basic income, bitcoin, Bletchley Park, blockchain, brain emulation, Buckminster Fuller, Charles Babbage, cloud computing, computer age, computer vision, correlation does not imply causation, credit crunch, cryptocurrency, cuban missile crisis, deep learning, DeepMind, dematerialisation, Demis Hassabis, discovery of the americas, disintermediation, don't be evil, driverless car, Elon Musk, en.wikipedia.org, epigenetics, Erik Brynjolfsson, everywhere but in the productivity statistics, Flash crash, friendly AI, Geoffrey Hinton, Google Glasses, hedonic treadmill, hype cycle, industrial robot, Internet of things, invention of agriculture, job automation, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John von Neumann, Kevin Kelly, life extension, low skilled workers, machine translation, Mahatma Gandhi, means of production, mutually assured destruction, Neil Armstrong, Nicholas Carr, Nick Bostrom, paperclip maximiser, pattern recognition, peer-to-peer, peer-to-peer model, Peter Thiel, radical life extension, Ray Kurzweil, Robert Solow, Rodney Brooks, Second Machine Age, self-driving car, Silicon Valley, Silicon Valley ideology, Skype, South Sea Bubble, speech recognition, Stanislav Petrov, Stephen Hawking, Steve Jobs, strong AI, technological singularity, TED Talk, The future is already here, The Future of Employment, theory of mind, Turing machine, Turing test, universal basic income, Vernor Vinge, wage slave, Wall-E, zero-sum game

Most people are aware that the world came close to this annihilation during the Cuban missile crisis in 1962; fewer know that we have also come close to a similar fate another four times since then, in 1979, 1980, 1983 and 1995. (52) In 1962 and 1983 we were saved by individual Soviet military officers who decided not to follow prescribed procedure. Today, while the world hangs on every utterance of Justin Bieber and the Kardashian family, relatively few of us even know the names of Vasili Arkhipov and Stanislav Petrov, two men who quite literally saved the world. Perhaps this survival illustrates our ingenuity. There was an ingenious logic in the repellent but effective doctrine of mutually assured destruction (MAD). More likely we have simply been lucky. We have time to rise to the challenge of superintelligence – probably a few decades.

Click Here to Kill Everybody: Security and Survival in a Hyper-Connected World by Bruce Schneier

23andMe, 3D printing, air gap, algorithmic bias, autonomous vehicles, barriers to entry, Big Tech, bitcoin, blockchain, Brian Krebs, business process, Citizen Lab, cloud computing, cognitive bias, computer vision, connected car, corporate governance, crowdsourcing, cryptocurrency, cuban missile crisis, Daniel Kahneman / Amos Tversky, David Heinemeier Hansson, disinformation, Donald Trump, driverless car, drone strike, Edward Snowden, Elon Musk, end-to-end encryption, fault tolerance, Firefox, Flash crash, George Akerlof, incognito mode, industrial robot, information asymmetry, information security, Internet of things, invention of radio, job automation, job satisfaction, John Gilmore, John Markoff, Kevin Kelly, license plate recognition, loose coupling, market design, medical malpractice, Minecraft, MITM: man-in-the-middle, move fast and break things, national security letter, Network effects, Nick Bostrom, NSO Group, pattern recognition, precautionary principle, printed gun, profit maximization, Ralph Nader, RAND corporation, ransomware, real-name policy, Rodney Brooks, Ross Ulbricht, security theater, self-driving car, Seymour Hersh, Shoshana Zuboff, Silicon Valley, smart cities, smart transportation, Snapchat, sparse data, Stanislav Petrov, Stephen Hawking, Stuxnet, supply-chain attack, surveillance capitalism, The Market for Lemons, Timothy McVeigh, too big to fail, Uber for X, Unsafe at Any Speed, uranium enrichment, Valery Gerasimov, Wayback Machine, web application, WikiLeaks, Yochai Benkler, zero day

Future of Life Institute (1 Feb 2016), “Accidental nuclear war: A timeline,” https://futureoflife.org/background/nuclear-close-calls-a-timeline. 95The Cuban Missile Crisis is probably: Benjamin Schwarz (1 Jan 2013), “The real Cuban missile crisis,” Atlantic, https://www.theatlantic.com/magazine/archive/2013/01/the-real-cuban-missile-crisis/309190. 95although the 1983 false alarm is a close second: Sewell Chan (18 Sep 2017), “Stanislav Petrov, Soviet officer who helped avert nuclear war,” New York Times, https://www.nytimes.com/2017/09/18/world/europe/stanislav-petrov-nuclear-war-dead.html. 95although much less damaging than: Laura Geggel (9 Feb 2016), “The odds of dying,” Live Science, https://www.livescience.com/3780-odds-dying.html. 95But instead of regarding it as: As amazing as it seems today, immediately after 9/11, people actually believed that terrorist attacks of that magnitude would happen every few months.

What We Owe the Future: A Million-Year View by William MacAskill

Ada Lovelace, agricultural Revolution, Albert Einstein, Alignment Problem, AlphaGo, artificial general intelligence, Bartolomé de las Casas, Bletchley Park, British Empire, Brownian motion, carbon footprint, carbon tax, charter city, clean tech, coronavirus, COVID-19, cuban missile crisis, decarbonisation, deep learning, DeepMind, Deng Xiaoping, different worldview, effective altruism, endogenous growth, European colonialism, experimental subject, feminist movement, framing effect, friendly AI, global pandemic, GPT-3, hedonic treadmill, Higgs boson, income inequality, income per capita, Indoor air pollution, Intergovernmental Panel on Climate Change (IPCC), Isaac Newton, Islamic Golden Age, iterative process, Jeff Bezos, job satisfaction, lab leak, Lao Tzu, Large Hadron Collider, life extension, lockdown, long peace, low skilled workers, machine translation, Mars Rover, negative emissions, Nick Bostrom, nuclear winter, OpenAI, Peter Singer: altruism, Peter Thiel, QWERTY keyboard, Robert Gordon, Rutger Bregman, Sam Altman, seminal paper, Shenzhen special economic zone , Shenzhen was a fishing village, Silicon Valley, special economic zone, speech recognition, Stanislav Petrov, stem cell, Steven Pinker, strong AI, synthetic biology, total factor productivity, transatlantic slave trade, Tyler Cowen, William MacAskill, women in the workforce, working-age population, World Values Survey, Y Combinator

Upon further investigation, they realized a training tape designed to simulate a Soviet nuclear strike had been accidentally playing on the command centre screens. Just four years later, during a period of heightened tensions between the United States and Soviet Union, a similar false alarm took place in a Soviet command centre after a Soviet early-warning system detected five incoming nuclear missiles.52 The officer on duty, Stanislav Petrov, was sceptical that a US first strike would involve just five nuclear missiles, and he couldn’t find evidence of the missile’s vapor trails. Based on this alone, he reasoned that the warning system must have been mistaken and correctly reported the warning as a false alarm. If he had not, Soviet protocol was to launch a counterstrike, though it is unclear whether those higher in command would have believed that it was not a false alarm.

…

These changes would also have impacted the timing of further reproductive events, until at some point in the future, the identities of everyone who is born is different than they would have been. And this is all because of small decisions like which route home your parents took from work one day. I dedicated my first book, Doing Good Better, to Peter Singer, Toby Ord, and Stanislav Petrov, and I said that “without [them] this book would not have been written.” But the book also would not have been written were it not for Jesus, Hitler, or any random English peasant in the fifteenth century. In time travel stories, small actions in the past often result in radical changes in the present.

Because We Say So by Noam Chomsky

Affordable Care Act / Obamacare, American Legislative Exchange Council, Anthropocene, Chelsea Manning, cuban missile crisis, David Brooks, drone strike, Edward Snowden, Garrett Hardin, gentrification, high-speed rail, Intergovernmental Panel on Climate Change (IPCC), Julian Assange, Malacca Straits, Martin Wolf, means of production, Monroe Doctrine, Nelson Mandela, no-fly zone, Occupy movement, oil shale / tar sands, Powell Memorandum, public intellectual, Ralph Waldo Emerson, RAND corporation, Slavoj Žižek, Stanislav Petrov, Strategic Defense Initiative, Thorstein Veblen, too big to fail, Tragedy of the Commons, uranium enrichment, WikiLeaks

The NATO exercise “almost became a prelude to a preventative [Russian] nuclear strike,” according to an account last year by Dmitry Adamsky in the JOURNAL OF STRATEGIC STUDIES. Nor was this the only close call. In September 1983, Russia’s early-warning systems registered an incoming missile strike from the United States and sent the highest-level alert. The Soviet military protocol was to retaliate with a nuclear attack of its own. The Soviet officer on duty, Stanislav Petrov, intuiting a false alarm, decided not to report the warnings to his superiors. Thanks to his dereliction of duty, we’re alive to talk about the incident. Security of the population was no more a high priority for Reagan planners than for their predecessors. Such heedlessness continues to the present, even putting aside the numerous near-catastrophic accidents, reviewed in a chilling new book, COMMAND AND CONTROL: NUCLEAR WEAPONS, THE DAMASCUS ACCIDENT, AND THE ILLUSION OF SAFETY, by Eric Schlosser.

On the Future: Prospects for Humanity by Martin J. Rees

23andMe, 3D printing, air freight, Alfred Russel Wallace, AlphaGo, Anthropocene, Asilomar, autonomous vehicles, Benoit Mandelbrot, biodiversity loss, blockchain, Boston Dynamics, carbon tax, circular economy, CRISPR, cryptocurrency, cuban missile crisis, dark matter, decarbonisation, DeepMind, Demis Hassabis, demographic transition, Dennis Tito, distributed ledger, double helix, driverless car, effective altruism, Elon Musk, en.wikipedia.org, Geoffrey Hinton, global village, Great Leap Forward, Higgs boson, Hyperloop, Intergovernmental Panel on Climate Change (IPCC), Internet of things, James Webb Space Telescope, Jeff Bezos, job automation, Johannes Kepler, John Conway, Large Hadron Collider, life extension, mandelbrot fractal, mass immigration, megacity, Neil Armstrong, Nick Bostrom, nuclear winter, ocean acidification, off-the-grid, pattern recognition, precautionary principle, quantitative hedge fund, Ray Kurzweil, Recombinant DNA, Rodney Brooks, Search for Extraterrestrial Intelligence, sharing economy, Silicon Valley, smart grid, speech recognition, Stanford marshmallow experiment, Stanislav Petrov, stem cell, Stephen Hawking, Steven Pinker, Stuxnet, supervolcano, technological singularity, the scientific method, Tunguska event, uranium enrichment, Walter Mischel, William MacAskill, Yogi Berra

Arkhipov held out against such action—and thereby avoided triggering a nuclear exchange that could have escalated catastrophically. Post-Cuba assessments suggest that the annual risk of thermonuclear destruction during the Cold War was about ten thousand times higher than the mean death rate from asteroid impact. And indeed, there were other ‘near misses’ when catastrophe was only avoided by a thread. In 1983 Stanislav Petrov, a Russian Air Force officer, was monitoring a screen when an ‘alert’ indicated that five Minuteman intercontinental ballistic missiles had been launched by the United States towards the Soviet Union. Petrov’s instructions, when this happened, were to alert his superior (who could, within minutes, trigger nuclear retaliation).

The Precipice: Existential Risk and the Future of Humanity by Toby Ord

3D printing, agricultural Revolution, Albert Einstein, Alignment Problem, AlphaGo, Anthropocene, artificial general intelligence, Asilomar, Asilomar Conference on Recombinant DNA, availability heuristic, biodiversity loss, Columbian Exchange, computer vision, cosmological constant, CRISPR, cuban missile crisis, decarbonisation, deep learning, DeepMind, defense in depth, delayed gratification, Demis Hassabis, demographic transition, Doomsday Clock, Dr. Strangelove, Drosophila, effective altruism, Elon Musk, Ernest Rutherford, global pandemic, Goodhart's law, Hans Moravec, Herman Kahn, Higgs boson, Intergovernmental Panel on Climate Change (IPCC), Isaac Newton, James Watt: steam engine, Large Hadron Collider, launch on warning, Mark Zuckerberg, Mars Society, mass immigration, meta-analysis, Mikhail Gorbachev, mutually assured destruction, Nash equilibrium, Nick Bostrom, Norbert Wiener, nuclear winter, ocean acidification, OpenAI, p-value, Peter Singer: altruism, planetary scale, power law, public intellectual, race to the bottom, RAND corporation, Recombinant DNA, Ronald Reagan, self-driving car, seminal paper, social discount rate, Stanislav Petrov, Stephen Hawking, Steven Pinker, Stewart Brand, supervolcano, survivorship bias, synthetic biology, tacit knowledge, the scientific method, Tragedy of the Commons, uranium enrichment, William MacAskill

When Soviet Premier Brezhnev found out, he asked President Carter “What kind of mechanism is it which allows a possibility of such incidents?”27 Autumn Equinox Incident: September 26, 1983 Shortly after midnight, in a period of heightened tensions, the screens at the command bunker for the Soviet satellite-based early-warning system showed five ICBMs launching from the United States.28 The duty officer, Stanislav Petrov, had instructions to report any detected launch to his superiors, who had a policy of immediate nuclear retaliatory strike. For five tense minutes he considered the case, then despite his remaining uncertainty, reported it to his commanders as a false alarm. He reasoned that a US first strike with just the five missiles shown was too unlikely and noted that the missiles’ vapor trails could not be identified.

…

-E. (2003). “Modeling the Size of Wars: From Billiard Balls to Sandpiles.” The American Political Science Review, 97(1), 135–50. Challinor, A. J., et al. (2014). “A Meta-Analysis of Crop Yield under Climate Change and Adaptation.” Nature Climate Change, 4(4), 287–91. Chan, S. (September 18, 2017). “Stanislav Petrov, Soviet Officer Who Helped Avert Nuclear War, Is Dead at 77.” The New York Times. Chapman, C. R. (2004). “The Hazard of Near-Earth Asteroid Impacts on Earth.” Earth And Planetary Science Letters, 222(1), 1–15. Charney, J. G., et al. (1979). “Carbon Dioxide and Climate: A Scientific Assessment.”

On Power and Ideology by Noam Chomsky

anti-communist, Ayatollah Khomeini, Berlin Wall, British Empire, Cornelius Vanderbilt, cuban missile crisis, disinformation, feminist movement, guns versus butter model, imperial preference, land reform, launch on warning, Mikhail Gorbachev, Monroe Doctrine, RAND corporation, Ronald Reagan, Stanislav Petrov, Strategic Defense Initiative, union organizing

In September 2013, the BBC reported that during this dangerous period, Russia’s early-warning systems detected an incoming missile strike from the United States, sending the highest-level alert. The protocol for the Soviet military was to retaliate with a nuclear attack of its own. The officer on duty, Stanislav Petrov, decided to disobey orders and not report the warnings to his superiors. Thanks to his dereliction of duty, we are alive to reflect on the black swan we prefer not to see. Other studies reveal a shocking array of close calls, even apart from the “most dangerous moment in history” during the Cuban missile crisis of 1962.

Doing Good Better: How Effective Altruism Can Help You Make a Difference by William MacAskill

barriers to entry, basic income, behavioural economics, Black Swan, Branko Milanovic, Cal Newport, Capital in the Twenty-First Century by Thomas Piketty, carbon footprint, clean water, corporate social responsibility, correlation does not imply causation, Daniel Kahneman / Amos Tversky, David Brooks, Edward Jenner, effective altruism, en.wikipedia.org, end world poverty, experimental subject, follow your passion, food miles, immigration reform, income inequality, index fund, Intergovernmental Panel on Climate Change (IPCC), Isaac Newton, job automation, job satisfaction, Lean Startup, M-Pesa, mass immigration, meta-analysis, microcredit, Nate Silver, Peter Singer: altruism, power law, public intellectual, purchasing power parity, quantitative trading / quantitative finance, randomized controlled trial, self-driving car, Skype, Stanislav Petrov, Steve Jobs, Steve Wozniak, Steven Pinker, The Future of Employment, The Wealth of Nations by Adam Smith, Tyler Cowen, universal basic income, William MacAskill, women in the workforce

HM1146.M33 2015 171'.8—dc23 2015000705 While the author has made every effort to provide accurate telephone numbers, Internet addresses, and other contact information at the time of publication, neither the publisher nor the author assumes any responsibility for errors or for changes that occur after publication. Further, the publisher does not have any control over and does not assume any responsibility for author or third-party websites or their content. Version_1 To Toby Ord, Peter Singer, and Stanislav Petrov, without whom this book would not have been written CONTENTS PRAISE FOR DOING GOOD BETTER TITLE PAGE COPYRIGHT DEDICATION INTRODUCTION Worms and Water Pumps: How can you do the most good? ONE You Are the 1 Percent: Just how much can you achieve? PART ONE THE FIVE KEY QUESTIONS OF EFFECTIVE ALTRUISM TWO Hard Trade-offs: Question #1: How many people benefit, and by how much?

The Doomsday Calculation: How an Equation That Predicts the Future Is Transforming Everything We Know About Life and the Universe by William Poundstone

Albert Einstein, anthropic principle, Any sufficiently advanced technology is indistinguishable from magic, Arthur Eddington, Bayesian statistics, behavioural economics, Benoit Mandelbrot, Berlin Wall, bitcoin, Black Swan, conceptual framework, cosmic microwave background, cosmological constant, cosmological principle, CRISPR, cuban missile crisis, dark matter, DeepMind, digital map, discounted cash flows, Donald Trump, Doomsday Clock, double helix, Dr. Strangelove, Eddington experiment, Elon Musk, Geoffrey Hinton, Gerolamo Cardano, Hans Moravec, heat death of the universe, Higgs boson, if you see hoof prints, think horses—not zebras, index fund, Isaac Newton, Jaron Lanier, Jeff Bezos, John Markoff, John von Neumann, Large Hadron Collider, mandelbrot fractal, Mark Zuckerberg, Mars Rover, Neil Armstrong, Nick Bostrom, OpenAI, paperclip maximiser, Peter Thiel, Pierre-Simon Laplace, Plato's cave, probability theory / Blaise Pascal / Pierre de Fermat, RAND corporation, random walk, Richard Feynman, ride hailing / ride sharing, Rodney Brooks, Ronald Reagan, Ronald Reagan: Tear down this wall, Sam Altman, Schrödinger's Cat, Search for Extraterrestrial Intelligence, self-driving car, Silicon Valley, Skype, Stanislav Petrov, Stephen Hawking, strong AI, tech billionaire, Thomas Bayes, Thomas Malthus, time value of money, Turing test

During the height of the Cuban missile crisis one Russian submarine remained submerged and out of radio contact with Moscow. The sub’s captain assumed that war had broken out and decided to launch nuclear torpedoes. To do so he needed the authorization of two ranking officers. One agreed. The other was Vasili Arkhipov. His veto prevented World War III. The other room commemorates Stanislav Petrov, who chose not to initiate a nuclear attack when his computer screen showed five US missiles approaching the Soviet Union in 1983. Petrov reasoned that the United States would not strike Russia with a mere five missiles; therefore it must be a computer glitch. It was. The Future of Humanity Institute is an expression of a global movement.

Scary Smart: The Future of Artificial Intelligence and How You Can Save Our World by Mo Gawdat

3D printing, accounting loophole / creative accounting, AI winter, AlphaGo, anthropic principle, artificial general intelligence, autonomous vehicles, basic income, Big Tech, Black Lives Matter, Black Monday: stock market crash in 1987, butterfly effect, call centre, carbon footprint, cloud computing, computer vision, coronavirus, COVID-19, CRISPR, cryptocurrency, deep learning, deepfake, DeepMind, Demis Hassabis, digital divide, digital map, Donald Trump, Elon Musk, fake news, fulfillment center, game design, George Floyd, global pandemic, Google Glasses, Google X / Alphabet X, Law of Accelerating Returns, lockdown, microplastics / micro fibres, Nick Bostrom, off-the-grid, OpenAI, optical character recognition, out of africa, pattern recognition, Ponzi scheme, Ray Kurzweil, recommendation engine, self-driving car, Silicon Valley, smart contracts, Stanislav Petrov, Stephen Hawking, subprime mortgage crisis, superintelligent machines, TED Talk, TikTok, Turing machine, Turing test, universal basic income, Watson beat the top human players on Jeopardy!, Y2K

Soviet protocol necessitated that Russia respond decisively, launching its entire nuclear arsenal before any US missile detonations could disable their response capability. This catastrophic error could have launched a world war that would have dwarfed the damage caused by the Second World War had it not been for a duty officer, Lt Col. Stanislav Petrov, who intercepted the messages and flagged them as faulty. He reasoned that if the US was really attacking, they would launch more than five missiles.3 He said he had a funny feeling in his gut that led him to investigate further. Good catch, Stan. Will our AI machines, assuming their code is perfect, be subjected to similar external signals that can lead to errors in the future?

Robot Rules: Regulating Artificial Intelligence by Jacob Turner

"World Economic Forum" Davos, Ada Lovelace, Affordable Care Act / Obamacare, AI winter, algorithmic bias, algorithmic trading, AlphaGo, artificial general intelligence, Asilomar, Asilomar Conference on Recombinant DNA, autonomous vehicles, backpropagation, Basel III, bitcoin, Black Monday: stock market crash in 1987, blockchain, brain emulation, Brexit referendum, Cambridge Analytica, Charles Babbage, Clapham omnibus, cognitive dissonance, Computing Machinery and Intelligence, corporate governance, corporate social responsibility, correlation does not imply causation, crowdsourcing, data science, deep learning, DeepMind, Demis Hassabis, distributed ledger, don't be evil, Donald Trump, driverless car, easy for humans, difficult for computers, effective altruism, Elon Musk, financial exclusion, financial innovation, friendly fire, future of work, hallucination problem, hive mind, Internet of things, iterative process, job automation, John Markoff, John von Neumann, Loebner Prize, machine readable, machine translation, medical malpractice, Nate Silver, natural language processing, Nick Bostrom, Northpointe / Correctional Offender Management Profiling for Alternative Sanctions, nudge unit, obamacare, off grid, OpenAI, paperclip maximiser, pattern recognition, Peace of Westphalia, Philippa Foot, race to the bottom, Ray Kurzweil, Recombinant DNA, Rodney Brooks, self-driving car, Silicon Valley, Stanislav Petrov, Stephen Hawking, Steve Wozniak, strong AI, technological singularity, Tesla Model S, The Coming Technological Singularity, The Future of Employment, The Signal and the Noise by Nate Silver, trolley problem, Turing test, Vernor Vinge

These issues raise fundamental questions about humanity’s relationship with AI: Why do we harbour concerns about giving up control? Can we strike a balance between AI effectiveness and human oversight? Will fools rush in where AIs fear to tread? 4.2 Why Might We Need Laws of Limitation? In September 2017, Stanislav Petrov died alone and destitute in an unremarkable Moscow suburb. His inauspicious death belied the pivotal role he played one night in 1983 when he was the duty officer in a secret command centre tasked with detecting nuclear attacks on the USSR by America. Petrov’s computer screen showed five intercontinental ballistic missiles heading towards the USSR.

Animal Spirits by Jackson Lears

1960s counterculture, Alan Greenspan, bank run, banking crisis, behavioural economics, business cycle, buy and hold, California gold rush, clockwork universe, conceptual framework, Cornelius Vanderbilt, creative destruction, cuban missile crisis, dark matter, Doomsday Clock, double entry bookkeeping, epigenetics, escalation ladder, feminist movement, financial innovation, Frederick Winslow Taylor, George Akerlof, George Santayana, heat death of the universe, Herbert Marcuse, Herman Kahn, Ida Tarbell, invisible hand, Isaac Newton, joint-stock company, Joseph Schumpeter, Lewis Mumford, lifelogging, market bubble, market fundamentalism, Mikhail Gorbachev, moral hazard, Norman Mailer, plutocrats, prosperity theology / prosperity gospel / gospel of success, Ralph Waldo Emerson, RAND corporation, Robert Shiller, Ronald Reagan, scientific management, Scientific racism, short selling, Shoshana Zuboff, Silicon Valley, source of truth, South Sea Bubble, Stanislav Petrov, Steven Pinker, Stewart Brand, Strategic Defense Initiative, surveillance capitalism, the market place, the scientific method, The Soul of a New Machine, The Wealth of Nations by Adam Smith, transcontinental railway, W. E. B. Du Bois, Whole Earth Catalog, zero-sum game

But in 1985, Reagan began yielding to the diplomatic charms of Mikhail Gorbachev as well as the citizen diplomacy of the nuclear freeze movement and came within an ace of abolishing nuclear weapons outright. Gorbachev must have known what had happened on September 26, 1983, when a Soviet early warning system mistook the sun’s reflection off clouds for five incoming Minuteman missiles. Lt. Col. Stanislav Petrov and his staff, under unimaginable pressure, correctly concluded that the system flashing “LAUNCH” had raised a false alarm; Petrov did not report the incoming missiles to his superior officers, who would have immediately ordered a full-scale nuclear attack on the United States. This was the sort of incident (and there have been more on both sides) that would have inspired a thoughtful leader like Gorbachev to urge the abolition of nuclear weapons.

The Cold War by Robert Cowley

Able Archer 83, anti-communist, Berlin Wall, British Empire, cuban missile crisis, defense in depth, disinformation, Dissolution of the Soviet Union, Doomsday Clock, Dr. Strangelove, friendly fire, Great Leap Forward, guns versus butter model, Henry Ford's grandson gave labor union leader Walter Reuther a tour of the company’s new, automated factory…, Korean Air Lines Flight 007, launch on warning, means of production, Mikhail Gorbachev, mutually assured destruction, RAND corporation, refrigerator car, Ronald Reagan, South China Sea, Stanislav Petrov, Strategic Defense Initiative, Suez canal 1869, Suez crisis 1956, transcontinental railway

There Goes Brussels … WILLIAMSON MURRAY Let us play a counterfactual game, and suppose for a moment that the one-sided crisis of 1983 had gotten out of control. What if, for example, on the night of September 26, a Soviet officer in a bunker outside Moscow had not had doubts about what he was seeing on a computer screen—first one incoming missile, and then another, five in all. Had Lieutenant Colonel Stanislav Petrov followed regulations, he would have telephoned his superiors to warn them that the Soviet Union was only minutes away from a nuclear attack. Petrov hesitated, convinced that something had gone awry in the computer system. The minutes passed. Nothing did happen. That night one man's hunch may have averted World War III.

Superintelligence: Paths, Dangers, Strategies by Nick Bostrom

agricultural Revolution, AI winter, Albert Einstein, algorithmic trading, anthropic principle, Anthropocene, anti-communist, artificial general intelligence, autism spectrum disorder, autonomous vehicles, backpropagation, barriers to entry, Bayesian statistics, bioinformatics, brain emulation, cloud computing, combinatorial explosion, computer vision, Computing Machinery and Intelligence, cosmological constant, dark matter, DARPA: Urban Challenge, data acquisition, delayed gratification, Demis Hassabis, demographic transition, different worldview, Donald Knuth, Douglas Hofstadter, driverless car, Drosophila, Elon Musk, en.wikipedia.org, endogenous growth, epigenetics, fear of failure, Flash crash, Flynn Effect, friendly AI, general purpose technology, Geoffrey Hinton, Gödel, Escher, Bach, hallucination problem, Hans Moravec, income inequality, industrial robot, informal economy, information retrieval, interchangeable parts, iterative process, job automation, John Markoff, John von Neumann, knowledge worker, Large Hadron Collider, longitudinal study, machine translation, megaproject, Menlo Park, meta-analysis, mutually assured destruction, Nash equilibrium, Netflix Prize, new economy, Nick Bostrom, Norbert Wiener, NP-complete, nuclear winter, operational security, optical character recognition, paperclip maximiser, pattern recognition, performance metric, phenotype, prediction markets, price stability, principal–agent problem, race to the bottom, random walk, Ray Kurzweil, recommendation engine, reversible computing, search costs, social graph, speech recognition, Stanislav Petrov, statistical model, stem cell, Stephen Hawking, Strategic Defense Initiative, strong AI, superintelligent machines, supervolcano, synthetic biology, technological singularity, technoutopianism, The Coming Technological Singularity, The Nature of the Firm, Thomas Kuhn: the structure of scientific revolutions, time dilation, Tragedy of the Commons, transaction costs, trolley problem, Turing machine, Vernor Vinge, WarGames: Global Thermonuclear War, Watson beat the top human players on Jeopardy!, World Values Survey, zero-sum game

The USA made emergency retaliation preparations before data from early-warning radar systems showed that no attack had been launched (McLean and Stewart 1979). On September 26, 1983, the malfunctioning Soviet Oko nuclear early-warning system reported an incoming US missile strike. The report was correctly identified as a false alarm by the duty officer at the command center, Stanislav Petrov: a decision that has been credited with preventing thermonuclear war (Lebedev 2004). It appears that a war would probably have fallen short of causing human extinction, even if it had been fought with the combined arsenals held by all the nuclear powers at the height of the Cold War, though it would have ruined civilization and caused unimaginable death and suffering (Gaddis 1982; Parrington 1997).

The Last Empire: The Final Days of the Soviet Union by Serhii Plokhy

affirmative action, Anton Chekhov, Berlin Wall, bilateral investment treaty, Boeing 747, cuban missile crisis, Dissolution of the Soviet Union, Fall of the Berlin Wall, Francis Fukuyama: the end of history, Korean Air Lines Flight 007, land reform, language acquisition, Mikhail Gorbachev, military-industrial complex, mutually assured destruction, Potemkin village, RAND corporation, Ronald Reagan, Seymour Hersh, Sinatra Doctrine, Stanislav Petrov, Strategic Defense Initiative, Transnistria

International tensions rose, threatening for the first time since the early 1960s to turn the Cold War into a hot one.3 On September 1, 1983, near Sakhalin Island, the Soviets shot down a South Korean airliner with 269 people aboard, including a sitting member of the US Congress. They then awaited American retaliation. Later that month, Lieutenant Colonel Stanislav Petrov of the Soviet Air Defense Forces Command near Moscow saw a blip on his radar screen indicating a missile headed toward the USSR. Then he saw what appeared to be four more missiles headed in the same direction. Suspecting a computer malfunction, he did not report the image to his superiors. Had he done so, nuclear war between the two powers might well have become a reality.

An Economic History of the Twentieth Century by J. Bradford Delong

affirmative action, Alan Greenspan, Andrei Shleifer, ASML, asset-backed security, Ayatollah Khomeini, banking crisis, Bear Stearns, Bretton Woods, British Empire, business cycle, buy and hold, Capital in the Twenty-First Century by Thomas Piketty, Carmen Reinhart, centre right, collapse of Lehman Brothers, collective bargaining, colonial rule, coronavirus, cotton gin, COVID-19, creative destruction, crowdsourcing, cryptocurrency, cuban missile crisis, deindustrialization, demographic transition, Deng Xiaoping, Donald Trump, en.wikipedia.org, ending welfare as we know it, endogenous growth, Fairchild Semiconductor, fake news, financial deregulation, financial engineering, financial repression, flying shuttle, Ford Model T, Ford paid five dollars a day, Francis Fukuyama: the end of history, full employment, general purpose technology, George Gilder, German hyperinflation, global value chain, Great Leap Forward, Gunnar Myrdal, Haber-Bosch Process, Hans Rosling, hedonic treadmill, Henry Ford's grandson gave labor union leader Walter Reuther a tour of the company’s new, automated factory…, housing crisis, Hyman Minsky, income inequality, income per capita, industrial research laboratory, interchangeable parts, Internet Archive, invention of agriculture, invention of the steam engine, It's morning again in America, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, Joseph Schumpeter, Kenneth Rogoff, labor-force participation, land reform, late capitalism, Les Trente Glorieuses, liberal capitalism, liquidity trap, Long Term Capital Management, low interest rates, manufacturing employment, market bubble, means of production, megacity, Menlo Park, Mikhail Gorbachev, mortgage debt, mutually assured destruction, Neal Stephenson, occupational segregation, oil shock, open borders, open economy, Paul Samuelson, Pearl River Delta, Phillips curve, plutocrats, price stability, Productivity paradox, profit maximization, public intellectual, quantitative easing, Ralph Waldo Emerson, restrictive zoning, rising living standards, road to serfdom, Robert Gordon, Robert Solow, rolodex, Ronald Coase, Ronald Reagan, savings glut, secular stagnation, Silicon Valley, Simon Kuznets, social intelligence, Stanislav Petrov, strikebreaker, structural adjustment programs, Suez canal 1869, surveillance capitalism, The Bell Curve by Richard Herrnstein and Charles Murray, The Chicago School, The Great Moderation, The Nature of the Firm, The Rise and Fall of American Growth, too big to fail, transaction costs, transatlantic slave trade, transcontinental railway, TSMC, union organizing, vertical integration, W. E. B. Du Bois, Wayback Machine, Yom Kippur War

In 1979, the loading of a training scenario onto an operational computer led NORAD to call the White House, claiming that the USSR had launched 250 missiles against the United States, and that the president had only between three and seven minutes to decide whether to retaliate. In 1983, the Soviet Union’s Lieutenant Colonel Stanislav Petrov refused to classify an early warning system missile sighting as an attack, dismissing it (correctly) as an error, and thereby preventing a worse error. That same year, the Soviet air force mistook an off-course Korean airliner carrying 100 people for one of the United States’ RC-135 spy planes that routinely violated Russian air space and shot it down.

Command and Control: Nuclear Weapons, the Damascus Accident, and the Illusion ofSafety by Eric Schlosser

Able Archer 83, Albert Einstein, anti-communist, Berlin Wall, Boeing 747, cuban missile crisis, Dr. Strangelove, Fall of the Berlin Wall, Haight Ashbury, Herman Kahn, impulse control, interchangeable parts, Isaac Newton, launch on warning, life extension, Mikhail Gorbachev, military-industrial complex, mutually assured destruction, nuclear taboo, nuclear winter, packet switching, prompt engineering, RAND corporation, Ronald Reagan, Stanislav Petrov, Stewart Brand, Strategic Defense Initiative, tacit knowledge, technological determinism, too big to fail, two and twenty, uranium enrichment, William Langewiesche

President Reagan called the attack “an act of barbarism” and a “crime against humanity [that] must never be forgotten.” A few weeks later alarms went off in an air defense bunker south of Moscow. A Soviet early-warning satellite had detected five Minuteman missiles approaching from the United States. The commanding officer on duty, Lieutenant Colonel Stanislav Petrov, tried to make sense of the warning. An American first strike would surely involve more than five missiles—but perhaps this was merely the first wave. The Soviet general staff was alerted, and it was Petrov’s job to advise them whether the missile attack was real. Any retaliation would have to be ordered soon.

Gorbachev by William Taubman

"World Economic Forum" Davos, Able Archer 83, active measures, affirmative action, Albert Einstein, anti-communist, Berlin Wall, British Empire, card file, conceptual framework, Deng Xiaoping, disinformation, Donald Trump, Fall of the Berlin Wall, fear of failure, haute couture, indoor plumbing, Korean Air Lines Flight 007, means of production, Mikhail Gorbachev, military-industrial complex, Neil Kinnock, Potemkin village, RAND corporation, Ronald Reagan, Ronald Reagan: Tear down this wall, Saturday Night Live, Stanislav Petrov, Strategic Defense Initiative, trade liberalization, young professional

Then, on the night of September 26, 1983, in a secret underground bunker outside Moscow, alarms indicated that American missiles were speeding toward Moscow. The duty officer had seven minutes to alert Andropov, who was on a dialysis machine in a suburban sanatorium. Fortunately, Lieutenant Colonel Stanislav Petrov concluded that the alarm was false. Meanwhile, however, the Americans and British were preparing to conduct the “Able Archer 83” war game, in which NATO’s supreme commander would request permission to use nuclear weapons and receive it. When Able Archer began in early November, the chief of the Soviet General Staff took cover in his command bunker under Moscow and ordered a “heightened alert” for some land-based Soviet forces.