Therac-25

6 results back to index


Robot Futures by Illah Reza Nourbakhsh

3D printing, autonomous vehicles, Burning Man, commoditize, computer vision, Mars Rover, Menlo Park, phenotype, Skype, software as a service, stealth mode startup, strong AI, telepresence, telepresence robot, Therac-25, Turing test, Vernor Vinge

Technology tends to increase the complexity of systems along several axes—the number of people partly responsible for a newer product is ever larger; the amount of software in new products dwarfs earlier systems; the interface used by the operator to control the product becomes more intricate. All these axes of complexity make the resulting system errors less clearly understandable and less accountable, with no one ever directly or solely responsible for the behavior of a complex robotic system. Brainspotting 101 Technology ethics and design courses frequently study the tragedy of the Therac-25 to understand how much can go wrong when poor design, incorrect training, and simple errors are compounded (Leveson and Turner 1993). The Therac-25 was a radiation therapy machine that provided focused radiation to cancer victims to destroy malignant tumors by rapidly moving a high-energy radiation beam. The nurse-operator of the machine would configure the machine for a customized treatment pattern, then launch its autonomous radiation therapy mode. In the rare event that the operator entered the mode incorrectly, then backed up in the interface and corrected the entry within eight seconds, the machine would configure to an incorrect internal setting, where it would deliver one hundred times the intended dose of radiation, inducing massive pain in the patient and, eventually, killing patients through radiation sickness.

In the rare event that the operator entered the mode incorrectly, then backed up in the interface and corrected the entry within eight seconds, the machine would configure to an incorrect internal setting, where it would deliver one hundred times the intended dose of radiation, inducing massive pain in the patient and, eventually, killing patients through radiation sickness. Many aspects of the Therac-25 therapy process are partially to blame for this. The interface was poorly designed, making incorrect data entry easy. Training for the operators was lightweight, and the nurses afforded the expensive, fancy machines more authority than the machines deserved. When the patients complained of pain during the procedure, the nurses would discount this complaint because the machine indicated that all was well.

Berkeley Robotics & Human Engineering Laboratory. http://bleex.me.berkeley.edu/ (accessed May 9, 2012). Kelly, Kevin. 2010. What Technology Wants. New York: Viking Press. Kurzweil, Ray. 2006. The Singularity Is Near: When Humans Transcend Biology. New York: Penguin Group. Lerner, Steve. 2010. Sacrifice Zones. Cambridge, MA: MIT Press. Leveson, N. G., and C. S. Turner. 1993. An investigation of the Therac 25 accidents. Computer 26 (7): 18–41. Lewis, M., S. Carpin, and S. Balakirsky. 2009. “Virtual Robots RoboCupRescue Competition: Contributions to Infrastructure and Science.” In Proceedings of IJCAI Workshop on Competitions in Artificial Intelligence and Robotics. Lewis, M., and K. Sycara. 2011. “Network-Centric Control for Multirobot Teams in Urban Search and Rescue.” In Proceedings of the 44th Hawaiian International Conference on Systems Sciences.


pages: 222 words: 53,317

Overcomplicated: Technology at the Limits of Comprehension by Samuel Arbesman

Amazon: amazon.comamazon.co.ukamazon.deamazon.fr

3D printing, algorithmic trading, Anton Chekhov, Apple II, Benoit Mandelbrot, citation needed, combinatorial explosion, Danny Hillis, David Brooks, digital map, discovery of the americas, en.wikipedia.org, Erik Brynjolfsson, Flash crash, friendly AI, game design, Google X / Alphabet X, Googley, HyperCard, Inbox Zero, Isaac Newton, iterative process, Kevin Kelly, Machine translation of "The spirit is willing, but the flesh is weak." to Russian and back, mandelbrot fractal, Minecraft, Netflix Prize, Nicholas Carr, Parkinson's law, Ray Kurzweil, recommendation engine, Richard Feynman, Richard Feynman, Richard Feynman: Challenger O-ring, Second Machine Age, self-driving car, software studies, statistical model, Steve Jobs, Steve Wozniak, Steven Pinker, Stewart Brand, superintelligent machines, Therac-25, Tyler Cowen: Great Stagnation, urban planning, Watson beat the top human players on Jeopardy!, Whole Earth Catalog, Y2K

We next turn to the social and biological limits of human comprehension: the reasons why our brains—and our societies—are particularly bad at dealing with these complex systems, no matter how hard we try. Chapter 3 LOSING THE BUBBLE In 1985, a patient entered a clinic to undergo radiation treatment for cancer of the cervix. The patient was prepared for treatment, and the operator of the large radiation machine known as the Therac-25 proceeded with radiation therapy. The machine responded with an error message, as well as noting that “no dose” had been administered. The operator tried again, with the same result. The operator tried three more times, for a total of five attempts, and each time the machine returned an error and responded that no radiation dosage had been delivered. After the treatment, the patient complained of a burning sensation around her hip and was admitted to the hospital.

After the treatment, the patient complained of a burning sensation around her hip and was admitted to the hospital. Several months later, the patient died of her cancer. It was discovered that she had suffered horrible radiation overexposure—her hip would have needed to be replaced—despite the machine’s having indicated that no dose of radiation was delivered. This was not the only instance of this radiation machine malfunctioning. In the 1980s, the Therac-25 failed for six patients, irradiating them with many times the dose they should have received. Damage from the massive radiation overdoses killed some of these people. These overdoses were considered the worst failures in the history of this type of machine. Could these errors have been prevented, or at least minimized? If you look at a 1983 safety analysis of these machines by the manufacturer, one of the reasons for the failure becomes clear.

While this is a true statement, it completely ignores the fact that software is complex and can fail in many different ways. This report implies a lack of awareness on the part of its makers that software could have a deadly complexity and be responsible for a radiation overdose. Software bugs are a fact of life, and yet the safety analysis almost completely ignored the risks they present. The people responsible for ensuring the safety of the Therac-25 misunderstood technological complexity, with lethal consequences. In hindsight it’s almost easy to see where they went wrong: they downplayed the importance of whole portions of the constructed system, and the result was a catastrophic failure. However, it’s more and more difficult to diagnose these kinds of problems in new technology. No matter how hard we try to construct well-built, logical technologies, there will always be some part that is beyond our complete understanding.


pages: 394 words: 118,929

Dreaming in Code: Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software by Scott Rosenberg

Amazon: amazon.comamazon.co.ukamazon.deamazon.fr

A Pattern Language, Benevolent Dictator For Life (BDFL), Berlin Wall, c2.com, call centre, collaborative editing, conceptual framework, continuous integration, Donald Knuth, Douglas Engelbart, Douglas Engelbart, Douglas Hofstadter, Dynabook, en.wikipedia.org, Firefox, Ford paid five dollars a day, Francis Fukuyama: the end of history, George Santayana, Grace Hopper, Guido van Rossum, Gödel, Escher, Bach, Howard Rheingold, index card, Internet Archive, inventory management, Jaron Lanier, John Markoff, John von Neumann, knowledge worker, Larry Wall, life extension, Loma Prieta earthquake, Menlo Park, Merlin Mann, new economy, Nicholas Carr, Norbert Wiener, pattern recognition, Paul Graham, Potemkin village, RAND corporation, Ray Kurzweil, Richard Stallman, Ronald Reagan, Ruby on Rails, semantic web, side project, Silicon Valley, Singularitarianism, slashdot, software studies, source of truth, South of Market, San Francisco, speech recognition, stealth mode startup, stem cell, Stephen Hawking, Steve Jobs, Stewart Brand, Ted Nelson, Therac-25, thinkpad, Turing test, VA Linux, Vannevar Bush, Vernor Vinge, web application, Whole Earth Catalog, Y2K

“In all of modern technology”: From a video distributed by the Software Engineering Institute, available at http://www.sei.cmu.edu/videos/watts/DPWatts.mov. Minasi, The Software Conspiracy. The Mariner 1 bug is described at http://nssdc.gsfc.nasa.gov/nmc/tmp/MARIN1.htm. James Gleick tells the story of the Ariane 5 bug at http://www.around.com/ariane.htm. The Therac-25 bug is detailed in a paper by Nancy Leveson and Clark S. Turner in IEEE Computer, July 1993, at http://courses.cs.vt.edu/~cs3604/lib/Therac_25/ Therac_1.htm. The 1991 Patriot missile bug is well documented, for instance at http://www.cs.usyd.edu.au/~alum/patriot_bug.htm. Jon Ogg’s talk was at the Systems & Software Technology Conference, Salt Lake City, April 2004. “The amount of software the Department of Defense”: Barry Boehm at the Systems & Software Technology Conference, 2004.

A missing hyphen in its guidance control program. In June 1996, the European Space Agency’s $500 million unmanned Ariane 5 rocket exploded forty seconds after liftoff because of a bug in the software that controlled its guidance system. (It tried to convert a 64-bit variable to a 16-bit variable, but the number was too high, a buffer overflowed, and the system froze.) From 1985 to 1987 a radiation therapy machine named the Therac-25 delivered massive X-ray overdoses to a half-dozen patients because of software flaws. During the 1991 Gulf War, a battery of American Patriot missiles failed to fire against incoming Scud missiles; the enemy missile hit a U.S. barracks, leaving twenty-eight dead. Investigations found that the software’s calculations had a defect that compounded over time, and after one hundred hours of continuous use, the Patriot’s figures were so far off, it simply didn’t fire.


pages: 351 words: 123,876

Beautiful Testing: Leading Professionals Reveal How They Improve Software (Theory in Practice) by Adam Goucher, Tim Riley

Amazon: amazon.comamazon.co.ukamazon.deamazon.fr

Albert Einstein, barriers to entry, Black Swan, call centre, continuous integration, Debian, Donald Knuth, en.wikipedia.org, Firefox, Grace Hopper, index card, Isaac Newton, natural language processing, p-value, performance metric, revision control, six sigma, software as a service, software patent, the scientific method, Therac-25, Valgrind, web application

Thanks to Nigel Simpson for his detailed reviews and comments on this chapter. ‖ http://defect.opensolaris.org/bz/show_bug.cgi?id=6962 BUG MANAGEMENT AND TEST CASE EFFECTIVENESS 83 References Chernak, Y. 2001. “Validating and Improving Test-Case Effectiveness.” IEEE Software, 18(1): 81–86. Kidwell, P. A. 1998. “Stalking the Elusive Computer Bug.” Annals of the History of Computing, 20: 5–9. McPhee, N. “Therac-25 accidents,” http://www.morris.umn.edu/~mcphee/Courses/Readings/Therac _25_accidents.html. Smithsonian National Museum of American History. “Log Book With Computer Bug,” http:// americanhistory.si.edu/collections/object.cfm?key=35&objkey=30. Tzu, Sun. The Art of War. Trans. Lionel Giles. http://www.gutenberg.org/etext/132. 84 CHAPTER SIX CHAPTER SEVEN Beautiful XMPP Testing Remko Tronçon A T MY FIRST JOB INTERVIEW , ONE OF THE INTERVIEWERS ASKED ME if I knew what “unit testing” was and whether I had used it before.


pages: 239 words: 64,812

Geek Sublime: The Beauty of Code, the Code of Beauty by Vikram Chandra

Amazon: amazon.comamazon.co.ukamazon.deamazon.fr

Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, Apple II, barriers to entry, Berlin Wall, British Empire, business process, conceptual framework, create, read, update, delete, crowdsourcing, don't repeat yourself, Donald Knuth, East Village, European colonialism, finite state, Firefox, Flash crash, glass ceiling, Grace Hopper, haute couture, iterative process, Jaron Lanier, John von Neumann, land reform, London Whale, Norman Mailer, Paul Graham, pink-collar, revision control, Silicon Valley, Silicon Valley ideology, Skype, Steve Jobs, Steve Wozniak, supercomputer in your pocket, theory of mind, Therac-25, Turing machine, wikimedia commons, women in the workforce

A bug can exist for half a century despite our best efforts to exterminate it.17 That software algorithms are now running our whole world means that software faults or errors can send us down the wrong highway, injure or kill people, and cause disasters. Every programmer is familiar with the most infamous bugs: the French Ariane 5 rocket that went off course and self-destructed forty seconds after lift-off because of an error in converting between representations of number values; the Therac-25 radiation therapy machine that reacted to a combination of operator input and a “counter overflow” by delivering doses of radiation a hundred times more intense than required, resulting in the agonizing deaths of five people and injuries to many others; the “Flash Crash” of 2010, when the Dow Jones suddenly plunged a thousand points and recovered just as suddenly, apparently as a result of automatic trading programs reacting to a single large trade.


pages: 1,201 words: 233,519

Coders at Work by Peter Seibel

Amazon: amazon.comamazon.co.ukamazon.deamazon.fr

Ada Lovelace, bioinformatics, cloud computing, Conway's Game of Life, domain-specific language, don't repeat yourself, Donald Knuth, fault tolerance, Fermat's Last Theorem, Firefox, George Gilder, glass ceiling, Guido van Rossum, HyperCard, information retrieval, Larry Wall, loose coupling, Marc Andreessen, Menlo Park, Metcalfe's law, Perl 6, premature optimization, publish or perish, random walk, revision control, Richard Stallman, rolodex, Ruby on Rails, Saturday Night Live, side project, slashdot, speech recognition, the scientific method, Therac-25, Turing complete, Turing machine, Turing test, type inference, Valgrind, web application

Eich: So a blue-collar language like Java shouldn't have a crazy generic system because blue-collar people can't figure out what the hell the syntax means with covariant, contravariant type constraints. Certainly I've experienced some toe loss due to C and C++'s foot guns. Part of programming is engineering; part of engineering is working out various safety properties, which matter. Doing a browser they matter. They matter more if you're doing the Therac-25. Though that was more a thread-scheduling problem, as I recall. But even then, you talk about better languages for writing concurrent programs or exploiting hardware parallelism. We shouldn't all be using synchronized blocks—we certainly shouldn't be using mutexes or spin locks. So the kind of leverage you can get through languages may involve trade-offs where you say, “I'm going, for safety, to sacrifice some expressiveness.”