14 results back to index
airport security, availability heuristic, Benoit Mandelbrot, Berlin Wall, Bernie Madoff, big-box store, Black Swan, Broken windows theory, Carmen Reinhart, Claude Shannon: information theory, Climategate, Climatic Research Unit, cognitive dissonance, collapse of Lehman Brothers, collateralized debt obligation, complexity theory, computer age, correlation does not imply causation, Credit Default Swap, credit default swaps / collateralized debt obligations, cuban missile crisis, Daniel Kahneman / Amos Tversky, diversification, Donald Trump, Edmond Halley, Edward Lorenz: Chaos theory, en.wikipedia.org, equity premium, Eugene Fama: efficient market hypothesis, everywhere but in the productivity statistics, fear of failure, Fellow of the Royal Society, Freestyle chess, fudge factor, George Akerlof, haute cuisine, Henri Poincaré, high batting average, housing crisis, income per capita, index fund, Internet Archive, invention of the printing press, invisible hand, Isaac Newton, James Watt: steam engine, John Nash: game theory, John von Neumann, Kenneth Rogoff, knowledge economy, locking in a profit, Loma Prieta earthquake, market bubble, Mikhail Gorbachev, Moneyball by Michael Lewis explains big data, Monroe Doctrine, mortgage debt, Nate Silver, new economy, Norbert Wiener, PageRank, pattern recognition, pets.com, prediction markets, Productivity paradox, random walk, Richard Thaler, Robert Shiller, Robert Shiller, Rodney Brooks, Ronald Reagan, Saturday Night Live, savings glut, security theater, short selling, Skype, statistical model, Steven Pinker, The Great Moderation, The Market for Lemons, the scientific method, The Signal and the Noise by Nate Silver, The Wisdom of Crowds, Thomas Kuhn: the structure of scientific revolutions, too big to fail, transaction costs, transfer pricing, University of East Anglia, Watson beat the top human players on Jeopardy!, wikimedia commons
Stephen’s Green, Dublin 2, Ireland (a division of Penguin Books Ltd) • Penguin Books Australia Ltd, 250 Camberwell Road, Camberwell, Victoria 3124, Australia (a division of Pearson Australia Group Pty Ltd) • Penguin Books India Pvt Ltd, 11 Community Centre, Panchsheel Park, New Delhi – 110 017, India • Penguin Group (NZ), 67 Apollo Drive, Rosedale, Auckland 0632, New Zealand (a division of Pearson New Zealand Ltd) • Penguin Books (South Africa) (Pty) Ltd, 24 Sturdee Avenue, Rosebank, Johannesburg 2196, South Africa Penguin Books Ltd, Registered Offices: 80 Strand, London WC2R 0RL, England First published in 2012 by The Penguin Press, a member of Penguin Group (USA) Inc. Copyright © Nate Silver, 2012 All rights reserved Illustration credits Figure 4-2: Courtesy of Dr. Tim Parker, University of Oxford Figure 7-1: From “1918 Influenza: The Mother of All Pandemics” by Jeffery Taubenberger and David Morens, Emerging Infectious Disease Journal, vol. 12, no. 1, January 2006, Centers for Disease Control and Prevention Figures 9-2, 9-3A, 9-3C, 9-4, 9-5, 9-6 and 9-7: By Cburnett, Wikimedia Commons Figure 12-2: Courtesy of Dr. J. Scott Armstrong, The Wharton School, University of Pennsylvania LIBRARY OF CONGRESS CATALOGING IN PUBLICATION DATA Silver, Nate. The signal and the noise : why most predictions fail but some don’t / Nate Silver. p. cm. Includes bibliographical references and index.
“Election Results: House Big Board,” New York Times, November 2, 2010. http://elections.nytimes.com/2010/results/house/big-board. 26. Nate Silver, “A Warning on the Accuracy of Primary Polls,” FiveThirtyEight, New York Times, March 1, 2012. http://fivethirtyeight.blogs.nytimes.com/2012/03/01/a-warning-on-the-accuracy-of-primary-polls/. 27. Nate Silver, “Bill Buckner Strikes Again,” FiveThirtyEight, New York Times; September 29, 2011. http://fivethirtyeight.blogs.nytimes.com/2011/09/29/bill-buckner-strikes-again/. 28. Otherwise, you should have assigned the congressman a 100 percent chance of victory instead. 29. Matthew Dickinson, “Nate Silver Is Not a Political Scientist,” in Presidential Power: A Nonpartisan Analysis of Presidential Power, Blogs Dot Middlebury, November 1, 2010. http://blogs.middlebury.edu/presidentialpower/2010/11/01/nate-silver-is-not-a-political-scientist/. 30.
Alan Schwarz, “The Great Debate,” Baseball America, January. 7, 2005. http://www.baseballamerica.com/today/features/050107debate.html. 20. Per interview with Billy Beane. 21. Nate Silver, “What Tim Geithner Can Learn from Baseball,” Esquire, March 11, 2009. http://www.esquire.com/features/data/mlb-player-salaries-0409. 22. As a result of my original agreement in 2003 and a subsequent agreement in 2009, Baseball Prospectus now fully owns and operates PECOTA. Beginning with the 2010 season, the PECOTA forecasts reflect certain changes, improvements, and departures from my original methodology. The methods I describe herein apply to the 2003–2009 version of PECOTA specifically. 23. Nate Silver, “PECOTA Takes on the Field,” Baseball Prospectus, January 16, 2004. http://www.baseballprospectus.com/article.php?articleid=2515. 24. Nate Silver, “Lies, Damned Lies: Projection Reflection,” Baseball Prospectus, October 11, 2006. http://www.baseballprospectus.com/article.php?
Albert Einstein, algorithmic trading, Amazon Mechanical Turk, Apple's 1984 Super Bowl advert, backtesting, Black Swan, book scanning, bounce rate, business intelligence, business process, call centre, computer age, conceptual framework, correlation does not imply causation, crowdsourcing, dark matter, data is the new oil, en.wikipedia.org, Erik Brynjolfsson, experimental subject, Google Glasses, happiness index / gross national happiness, job satisfaction, Johann Wolfgang von Goethe, Machine translation of "The spirit is willing, but the flesh is weak." to Russian and back, Moneyball by Michael Lewis explains big data, Nate Silver, natural language processing, Netflix Prize, Network effects, Norbert Wiener, personalized medicine, placebo effect, prediction markets, Ray Kurzweil, recommendation engine, risk-adjusted returns, Ronald Coase, Search for Extraterrestrial Intelligence, self-driving car, sentiment analysis, software as a service, speech recognition, statistical model, Steven Levy, text mining, the scientific method, The Signal and the Noise by Nate Silver, The Wisdom of Crowds, Turing test, Watson beat the top human players on Jeopardy!, X Prize, Yogi Berra
Michael Scherer, “Inside the Secret World of the Data Crunchers Who Helped Obama Win.” TIME Magazine, November 07, 2012. http://swampland.time.com/2012/11/07/inside-the-secret-world-of-quants-and-data-crunchers-who-helped-obama-win/. Colbert Nation, www.colbertnation.com. Stephen Colbert interviews Nate Silver, New York Times blogger about his book, The Signal and the Noise: Why So Many Predictions Fail—but Some Don’t. http://www.colbertnation.com/the-colbert-report-videos/420765/november-05-2012/nate-silver. Peggy Noonan, “They’ve Lost That Lovin’ Feeling.” Wall Street Journal, July 30, 2011. http://online.wsj.com/article/SB10001424053111904800304576474620336602248.html. Jack Gillum, “Mitt Romney Uses Secretive Data Mining To Identify Wealthy Donors.” Huffington Post, August 24, 2012. www.huffingtonpost.com/2012/08/24/mitt-romney-data-mining_n_1827318.html.
The bad news is that it’s actually more than half; the good news is that PA can learn to do better. A Faulty Oracle Everyone Loves The first step toward predicting the future is admitting you can’t. —Stephen Dubner, Freakonomics Radio, March 30, 2011 The “prediction paradox”: The more humility we have about our ability to make predictions, the more successful we can be in planning for the future. —Nate Silver, The Signal and the Noise: Why So Many Predictions Fail—but Some Don’t Half of what we will teach you in medical school will, by the time you are done practicing, be proved wrong. —Dr. Mehmet Oz Your resident “oracle,” PA, tells you which customers are most likely to respond. It earmarks a quarter of the entire list and says, “These folks are three times more likely to respond than average!” So now you have a short list of 250,000 customers of which 3 percent will respond—7,500 responses.
However, they kept at it, squeezing every drop of potential out of their brainshare and data, right up until the final weeks before the big match. Confidence without Overconfidence Both experts and laypeople mistake more confident predictions for more accurate ones. But overconfidence is often the reason for failure. If our appreciation of uncertainty improves, our predictions can get better too. —Nate Silver, The Signal and the Noise: Why So Many Predictions Fail—but Some Don’t The trouble with the world is that the stupid are cocksure and the intelligent are full of doubt. —Bertrand Russell You got to know when to hold ‘em, know when to fold ‘em. —Don Schlitz, “The Gambler” (sung by Kenny Rogers) Jeopardy! wasn’t built for players with no self-doubt. —Chris Jones, Esquire Magazine Besides answering questions, there’s a second skill each Jeopardy!
Superforecasting: The Art and Science of Prediction by Philip Tetlock, Dan Gardner
Affordable Care Act / Obamacare, Any sufficiently advanced technology is indistinguishable from magic, availability heuristic, Black Swan, butterfly effect, cloud computing, cuban missile crisis, Daniel Kahneman / Amos Tversky, desegregation, Edward Lorenz: Chaos theory, forward guidance, Freestyle chess, fundamental attribution error, germ theory of disease, hindsight bias, index fund, Jane Jacobs, Jeff Bezos, Mikhail Gorbachev, Mohammed Bouazizi, Nash equilibrium, Nate Silver, obamacare, pattern recognition, performance metric, place-making, placebo effect, prediction markets, quantitative easing, random walk, randomized controlled trial, Richard Feynman, Richard Feynman, Richard Thaler, Robert Shiller, Robert Shiller, Ronald Reagan, Saturday Night Live, Silicon Valley, Skype, statistical model, stem cell, Steve Ballmer, Steve Jobs, Steven Pinker, the scientific method, The Signal and the Noise by Nate Silver, The Wisdom of Crowds, Watson beat the top human players on Jeopardy!
A mere two years after it was published the Arab Spring turned the Middle East topsy-turvy, but I can’t find it in Friedman’s book, which casts some doubt on his forecasts for the remaining ninety-eight years. Friedman is also the author of the 1991 book The Coming War with Japan—that’s the coming American war with Japan—which has yet to prove its prescience. 7. For islands of professionalism in a sea of malpractice, see the forecasting concepts and tools reviewed in Nate Silver, The Signal and the Noise: Why So Many Predictions Fail—but Some Don’t (New York: Penguin Press, 2012); J. Scott Armstrong, ed., Principles of Forecasting: A Handbook for Researchers and Practitioners (Boston: Kluwer, 2001); and Bruce Bueno de Mesquita, The Predictioneer’s Game (New York: Random House, 2009). Expanding these islands has proven hard. There is often little transfer of classroom statistical concepts, like regression toward the mean, to problems that students later encounter in life.
They have no idea how good their forecasts are in the short, medium, or long term—and no idea how good their forecasts could become. At best, they have vague hunches. That’s because the forecast-measure-revise procedure operates only within the rarefied confines of high-tech forecasting, such as the work of macroeconomists at central banks or marketing and financial professionals in big companies or opinion poll analysts like Nate Silver.7 More often forecasts are made and then … nothing. Accuracy is seldom determined after the fact and is almost never done with sufficient regularity and rigor that conclusions can be drawn. The reason? Mostly it’s a demand-side problem: The consumers of forecasting—governments, business, and the public—don’t demand evidence of accuracy. So there is no measurement. Which means no revision. And without revision, there can be no improvement.
Consider the weather in Phoenix, Arizona. Each June, it gets very hot and sunny. A forecaster who followed a mindless rule like, “always assign 100% to hot and sunny” could get a Brier score close to 0, leaving 0.2 in the dust. Here, the right test of skill would be whether a forecaster can do better than mindlessly predicting no change. This is an underappreciated point. For example, after the 2012 presidential election, Nate Silver, Princeton’s Sam Wang, and other poll aggregators were hailed for correctly predicting all fifty state outcomes, but almost no one noted that a crude, across-the-board prediction of “no change”—if a state went Democratic or Republican in 2008, it will do the same in 2012—would have scored forty-eight out of fifty, which suggests that the many excited exclamations of “He called all fifty states!”
Stuffocation by James Wallman
3D printing, Airbnb, back-to-the-land, Berlin Wall, big-box store, Black Swan, BRICs, carbon footprint, Cass Sunstein, clean water, collaborative consumption, crowdsourcing, David Brooks, Fall of the Berlin Wall, happiness index / gross national happiness, high net worth, income inequality, James Hargreaves, Joseph Schumpeter, Martin Wolf, McMansion, means of production, Nate Silver, Occupy movement, post-industrial society, Post-materialism, post-materialism, Richard Florida, Richard Thaler, sharing economy, Silicon Valley, Simon Kuznets, Skype, spinning jenny, The Signal and the Noise by Nate Silver, Thorstein Veblen, Tyler Cowen: Great Stagnation, World Values Survey, Zipcar
Using the Past to Tell the Future I am indebted to three sources for this section: Peter N Stearns, “Why Study History?”, American Historical Association, 1998; Nate Silver, The Signal and the Noise (New York: Allen Lane, 2012); and Rob Hyndman, “Why are some things easier to forecast than others?”, 18 September 2012, on his blog, Hyndsight (www.robjhyndman.com/hyndsight). “In the 1970s, the high temperature forecasts were wrong, on average, by about six degrees. Today they are only wrong by half that amount, three degrees. When hurricane forecasters predicted where a hurricane would hit land in the 1980s they were usually out by 350 miles. Today, their predictions are only wrong by 100 miles.” If you’re not ready – yet – to take on all of Nate Silver’s The Signal and the Noise, read Nate Silver, “The Weatherman Is Not a Moron”, New York Times, 7 September 2012. The Farm Where the Corn Did Not Grow Tall For Everett Rogers’s version of his life, read Everett M Rogers, The Fourteenth Paw (Singapore: Asian Media Information and Communication Centre, 2008).
3D printing, Albert Einstein, Amazon Mechanical Turk, Arthur Eddington, Benoit Mandelbrot, bioinformatics, Black Swan, Brownian motion, cellular automata, Claude Shannon: information theory, combinatorial explosion, computer vision, constrained optimization, correlation does not imply causation, crowdsourcing, Danny Hillis, data is the new oil, double helix, Douglas Hofstadter, Erik Brynjolfsson, experimental subject, Filter Bubble, future of work, global village, Google Glasses, Gödel, Escher, Bach, information retrieval, job automation, John Snow's cholera map, John von Neumann, Joseph Schumpeter, Kevin Kelly, lone genius, mandelbrot fractal, Mark Zuckerberg, Moneyball by Michael Lewis explains big data, Narrative Science, Nate Silver, natural language processing, Netflix Prize, Network effects, NP-complete, P = NP, PageRank, pattern recognition, phenotype, planetary scale, pre–internet, random walk, Ray Kurzweil, recommendation engine, Richard Feynman, Richard Feynman, Second Machine Age, self-driving car, Silicon Valley, speech recognition, statistical model, Stephen Hawking, Steven Levy, Steven Pinker, superintelligent machines, the scientific method, The Signal and the Noise by Nate Silver, theory of mind, transaction costs, Turing machine, Turing test, Vernor Vinge, Watson beat the top human players on Jeopardy!, white flight
Adam, the robot scientist, is described in “The automation of science,” by Ross King et al. (Science, 2009). Sasha Issenberg’s The Victory Lab (Broadway Books, 2012) dissects the use of data analysis in politics. “How President Obama’s campaign used big data to rally individual votes,” by the same author (MIT Technology Review, 2013), tells the story of its greatest success to date. Nate Silver’s The Signal and the Noise (Penguin Press, 2012) has a chapter on his poll aggregation method. Robot warfare is the theme of P. W. Singer’s Wired for War (Penguin, 2009). Cyber War, by Richard Clarke and Robert Knake (Ecco, 2012), sounds the alarm on cyberwar. My work on combining machine learning with game theory to defeat adversaries, which started as a class project, is described in “Adversarial classification,”* by Nilesh Dalvi et al.
David Wolpert derives his “no free lunch” theorem for induction in “The lack of a priori distinctions between learning algorithms”* (Neural Computation, 1996). I discuss the importance of prior knowledge in machine learning in “Toward knowledge-rich data mining”* (Data Mining and Knowledge Discovery, 2007) and misinterpretations of Occam’s razor in “The role of Occam’s razor in knowledge discovery”* (Data Mining and Knowledge Discovery, 1999). Overfitting is one of the main themes of The Signal and the Noise, by Nate Silver (Penguin Press, 2012), who calls it “the most important scientific problem you’ve never heard of.” “Why most published research findings are false,”* by John Ioannidis (PLoS Medicine, 2005), discusses the problem of mistaking chance findings for true ones in science. Yoav Benjamini and Yosef Hochberg propose a way to combat it in “Controlling the false discovery rate: A practical and powerful approach to multiple testing”* (Journal of the Royal Statistical Society, Series B, 1995).
In politics, as in business and war, there is nothing worse than seeing your opponent make moves that you don’t understand and don’t know what to do about until it’s too late. That’s what happened to the Romney campaign. They could see the other side buying ads in particular cable stations in particular towns but couldn’t tell why; their crystal ball was too fuzzy. In the end, Obama won every battleground state save North Carolina and by larger margins than even the most accurate pollsters had predicted. The most accurate pollsters, in turn, were the ones (like Nate Silver) who used the most sophisticated prediction techniques; they were less accurate than the Obama campaign because they had fewer resources. But they were a lot more accurate than the traditional pundits, whose predictions were based on their expertise. You might think the 2012 election was a fluke: most elections are not close enough for machine learning to be the deciding factor. But machine learning will cause more elections to be close in the future.
Big Data: A Revolution That Will Transform How We Live, Work, and Think by Viktor Mayer-Schonberger, Kenneth Cukier
23andMe, Affordable Care Act / Obamacare, airport security, AltaVista, barriers to entry, Berlin Wall, big data - Walmart - Pop Tarts, Black Swan, book scanning, business intelligence, business process, call centre, cloud computing, computer age, correlation does not imply causation, dark matter, double entry bookkeeping, Eratosthenes, Erik Brynjolfsson, game design, IBM and the Holocaust, index card, informal economy, Internet of things, invention of the printing press, Jeff Bezos, Louis Pasteur, Mark Zuckerberg, Menlo Park, Moneyball by Michael Lewis explains big data, Nate Silver, natural language processing, Netflix Prize, Network effects, obamacare, optical character recognition, PageRank, performance metric, Peter Thiel, Post-materialism, post-materialism, random walk, recommendation engine, self-driving car, sentiment analysis, Silicon Valley, Silicon Valley startup, smart grid, smart meter, social graph, speech recognition, Steve Jobs, Steven Levy, the scientific method, The Signal and the Noise by Nate Silver, The Wealth of Nations by Adam Smith, Turing test, Watson beat the top human players on Jeopardy!
Neyman’s famous paper is Jerzy Neyman, “On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection,” Journal of the Royal Statistical Society 97, no. 4 (1934), pp. 558–625. A sample of 1,100 observations is sufficient—Earl Babbie, Practice of Social Research (12th ed. 2010), pp. 204–207. [>] The cellphone effect—“Estimating the Cellphone Effect,” September 20, 2008 (http://www.fivethirtyeight.com/2008/09/estimating-cellphone-effect-22-points.html); for more on polling biases and other statistical insights see Nate Silver, The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t (Penguin, 2012). [>] Steve Jobs’s gene sequencing—Walter Isaacson, Steve Jobs (Simon and Schuster, 2011), pp. 550–551. [>] Google Flu Trends predicting to city level—Dugas et al., “Google Flu Trends.” Etzioni on temporal data—Interview by Cukier, October 2011. [>] John Kunze quotation—Jonathan Rosenthal, “Special Report: International Banking,” The Economist, May 19, 2012, pp. 7–8.
Wall Street Journal, November 19, 2010 (http://online.wsj.com/article/SB10001424052748704648604575620750998072986.html). Scott, James. Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed. Yale University Press, 1998. Seltzer, William, and Margo Anderson. “The Dark Side of Numbers: The Role of Population Data Systems in Human Rights Abuses.” Social Research 68 (2001) pp. 481–513. Silver, Nate. The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t. Penguin, 2012. Singel, Ryan. “Netflix Spilled Your Brokeback Mountain Secret, Lawsuit Claims.” Wired, December 17, 2009 (http://www.wired.com/threatlevel/2009/12/netflix-privacy-lawsuit/). Smith, Adam. The Wealth of Nations (1776). Reprinted Bantam Classics, 2003. A free electronic version is available (http://www2.hn.psu.edu/faculty/jmanis/adam-smith/Wealth-Nations.pdf).
Its accuracy depends on ensuring randomness when collecting the sample data, but achieving such randomness is tricky. Systematic biases in the way the data is collected can lead to the extrapolated results being very wrong. There are echoes of such problems in election polling using landline phones. The sample is biased against people who only use cell-phones (who are younger and more liberal), as the statistician Nate Silver has pointed out. This has resulted in incorrect election predictions. In the 2008 presidential election between Barack Obama and John McCain, the major polling organizations of Gallup, Pew, and ABC/Washington Post found differences of between one and three percentage points when they polled with and without adjusting for cellphone users—a hefty margin considering the tightness of the race. Most troublingly, random sampling doesn’t scale easily to include subcategories, as breaking the results down into smaller and smaller subgroups increases the possibility of erroneous predictions.
23andMe, 3D printing, Affordable Care Act / Obamacare, Anne Wojcicki, Atul Gawande, augmented reality, bioinformatics, call centre, Clayton Christensen, clean water, cloud computing, computer vision, conceptual framework, connected car, correlation does not imply causation, crowdsourcing, dark matter, data acquisition, disintermediation, don't be evil, Edward Snowden, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, Firefox, global village, Google Glasses, Google X / Alphabet X, Ignaz Semmelweis: hand washing, interchangeable parts, Internet of things, Isaac Newton, job automation, Joseph Schumpeter, Julian Assange, Kevin Kelly, license plate recognition, Lyft, Mark Zuckerberg, Marshall McLuhan, meta analysis, meta-analysis, microbiome, Nate Silver, natural language processing, Network effects, Nicholas Carr, obamacare, pattern recognition, personalized medicine, phenotype, placebo effect, RAND corporation, randomized controlled trial, Second Machine Age, self-driving car, Silicon Valley, Skype, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, Snapchat, social graph, speech recognition, stealth mode startup, Steve Jobs, the scientific method, The Signal and the Noise by Nate Silver, The Wealth of Nations by Adam Smith, Turing test, Uber for X, Watson beat the top human players on Jeopardy!, X Prize
Although Eisenstein stopped short of claiming that the first industrial revolution was an outgrowth of the printing press, many others have claimed this. Marshall McLuhan, in The Gutenberg Galaxy, wrote: “The invention of typography confirmed and extended the new visual stress of applied knowledge, providing the first uniformly repeatable commodity, the first assembly-line, and the first mass-production.”11 More recently, Nate Silver, in The Signal and the Noise, asserted that the industrial revolution of 1775 was sparked by the printing press, whereby the economic growth rate that was stagnant at 0.1 percent per year then grew faster than the growth rate of the population.12 But I prefer to principally assess the Gutenberg transformative effects by the specific attributes that they induced or cultivated instead of as a precursor for subsequent momentous periods in history.
Matthew, “The World’s Most Expensive Book Just Sold For Over $14 Million,” Business Insider, November 26, 2013, http://www.businessinsider.com/worlds-most-expensive-book-sells-for-14-million-2013-11. 5. Eisenstein, The Printing Press as an Agent of Change, 152. 6. Ibid., 159. 7. N. Silver, The Signal and the Noise (New York, NY: Penguin, 2012), 2. 8. N. Carr, The Shallows: What the Internet Is Doing to Our Brains (New York, NY: W.W. Norton, 2010), 69. 9. Silver, The Signal and the Noise, 12. 10. Eisenstein, The Printing Press as an Agent of Change, 41. 11. McLuhan, The Gutenberg Galaxy, 124. 12. Silver, The Signal and the Noise, 7. 13. Ibid., 3. 14. Carr, The Shallows: What the Internet Is Doing to Our Brains, 71. 15. “The Book of Jobs: Hope, Hype, and Apple’s iPad,” The Economist, January 30–February 5, 2010. 16. Eisenstein, The Printing Press as an Agent of Change, 75. 17.
Standage, “Social Networking in the 1600s,” New York Times, June 23, 2013, http://www.nytimes.com/2013/06/23/opinion/sunday/social-networking-in-the-1600s.html?pagewanted=all. 22. Eisenstein, The Printing Press as an Agent of Change, 53. 23. V. Goel, “Our Daily Cup of Facebook,” New York Times, August 13, 2013, http://bits.blogs.nytimes.com/2013/08/13/our-daily-cup-of-facebook/?ref=technology&_r=0&pagewanted=print. 24. N. Silver, The Signal and the Noise, 2. 25. “The March of Protest,” The Economist, June 29, 2013, http://www.economist.com/printedition/2013-06-29. 26. W. Ghonim, Revolution 2.0: The Power of the People Is Greater Than the People in Power (New York: Houghton Mifflin Harcourt, 2012). 27. Eisenstein, The Printing Press as an Agent of Change, 129. 28. M. B. Hall, The Scientific Renaissance 1450–1630 (New York, NY: Harper & Brothers, 1962), 130. 29.
Misbehaving: The Making of Behavioral Economics by Richard H. Thaler
Albert Einstein, Amazon Mechanical Turk, Andrei Shleifer, Apple's 1984 Super Bowl advert, Atul Gawande, Berlin Wall, Bernie Madoff, Black-Scholes formula, capital asset pricing model, Cass Sunstein, Checklist Manifesto, choice architecture, clean water, cognitive dissonance, conceptual framework, constrained optimization, Daniel Kahneman / Amos Tversky, delayed gratification, diversification, diversified portfolio, Edward Glaeser, endowment effect, equity premium, Eugene Fama: efficient market hypothesis, experimental economics, Fall of the Berlin Wall, George Akerlof, hindsight bias, Home mortgage interest deduction, impulse control, index fund, invisible hand, Jean Tirole, John Nash: game theory, John von Neumann, late fees, law of one price, libertarian paternalism, Long Term Capital Management, loss aversion, market clearing, Mason jar, mental accounting, meta analysis, meta-analysis, More Guns, Less Crime, mortgage debt, Nash equilibrium, Nate Silver, New Journalism, nudge unit, payday loans, Ponzi scheme, presumed consent, pre–internet, principal–agent problem, prisoner's dilemma, profit maximization, random walk, randomized controlled trial, Richard Thaler, Robert Shiller, Robert Shiller, Ronald Coase, Silicon Valley, South Sea Bubble, statistical model, Steve Jobs, technology bubble, The Chicago School, The Myth of the Rational Market, The Signal and the Noise by Nate Silver, The Wealth of Nations by Adam Smith, Thomas Kuhn: the structure of scientific revolutions, transaction costs, ultimatum game, Walter Mischel
So what effect has this research plus a free app had on the behavior of football coaches? Essentially none. Since Romer wrote his paper, the frequency of going for it on fourth down has marginally gone down, meaning that teams have gotten dumber! (Similarly, there has been no noticeable change in teams’ draft strategy since our paper came out.) Nate Silver, the ex–sports analytics junkie who became famous for his political forecasts and for the excellent book The Signal and the Noise, estimates that bad fourth-down decisions cost a football team an average of half a win per season. The Times analysts estimate it to be closer to two-thirds of a win per year. That may not seem like a lot, but the season is only sixteen games. A team can win an extra game every other year just by making the smart decision two or three times a game, one they can even check online if they need help.¶ Of course, coaches are Humans.
Chapter 29: Football 277 “Division of labor strongly attenuates”: Stewart (1997). 278 football paper: Massey and Thaler (2013). 280 The winner’s curse: For a review, see my “Anomalies” column on the subject (Thaler, 1988a). 280 The false consensus effect: Ross, Greene, and House (1977). 284 If a team is paying a high draft pick a lot of money: Camerer and Weber (1999). 292 teams don’t go for it: Romer (2006). 292 New York Times used his model: For an example of Brian Burke’s work, see http://www.advancedfootballanalytics.com/. 292 “New York Times 4th Down Bot”: The bot’s recommendations can be found at http://nyt4thdownbot.com/. For a comparison between coaches and the NYT Bot’s performances, see Burk and Quealy (2014). 292 The Signal and the Noise: Silver (2012). 293 Peter Principle: Peter and Hull (1969). Chapter 30: Game Shows 296 They asked me if I would like to join the team: Post et al. (2008). 299 my paper with Eric Johnson: Thaler and Johnson (1990). 300 an experiment to measure the difference between public and private decisions: Baltussen, van den Assem, and van Dolder (2015). 301 more risk averse in front of the crowd: This lines up with findings that investors take more risks online than in front of others.
. ———. 1984. “Stock Prices and Social Dynamics.” Brookings Papers on Economic Activity 2: 457–510. ———. 1986. “Comments on Miller and on Kleidon.” Journal of Business 59, no. 4, part 2: S501–5. ———. 2000. Irrational Exuberance. Princeton: Princeton University Press. Shleifer, Andrei, and Robert W. Vishny. 1997. “The Limits of Arbitrage.” Journal of Finance 52, no. 1: 35–55. Silver, Nate. 2012. The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t. New York: Penguin. Simon, Herbert A. 1957. Models of Man, Social and Rational: Mathematical Essays on Rational Human Behavior in a Social Setting. Oxford: Wiley. Sloman, Steven A. 1996. “The Empirical Case for Two Systems of Reasoning.” Psychological Bulletin 119, no. 1: 3. Slonim, Robert L., and Alvin E. Roth. 1998. “Learning in High Stakes Ultimatum Games: An Experiment in the Slovak Republic.”
3D printing, algorithmic trading, Any sufficiently advanced technology is indistinguishable from magic, augmented reality, big data - Walmart - Pop Tarts, call centre, Cass Sunstein, Clayton Christensen, computer age, death of newspapers, deferred acceptance, Edward Lorenz: Chaos theory, Erik Brynjolfsson, Filter Bubble, Flash crash, Florence Nightingale: pie chart, Frank Levy and Richard Murnane: The New Division of Labor, Google Earth, Google Glasses, High speed trading, Internet Archive, Isaac Newton, Jaron Lanier, Jeff Bezos, job automation, Kevin Kelly, Kodak vs Instagram, Marshall McLuhan, means of production, Nate Silver, natural language processing, Netflix Prize, pattern recognition, price discrimination, recommendation engine, Richard Thaler, Rosa Parks, self-driving car, sentiment analysis, Silicon Valley, Silicon Valley startup, Slavoj Žižek, social graph, speech recognition, Steve Jobs, Steven Levy, Steven Pinker, Stewart Brand, the scientific method, The Signal and the Noise by Nate Silver, upwardly mobile, Wall-E, Watson beat the top human players on Jeopardy!, Y Combinator
While I was first coming up with formulas at college, trying to mathematically determine whether we should go to the library to get some work done, deep down in the recesses of our dorky ids I think that what we were saying is that life is uncertain and we were trying to make it more certain. I’m not as disturbed by numbers providing answers as I am by the potential that there might not be answers.” What is it about the modern world that makes us demand easy answers? Is it that we are naturally pattern-seeking creatures, as the statistician Nate Silver argues in The Signal and the Noise? Or is there something about the effects of the march of technology that demands the kind of answers only an algorithm can provide? “[The algorithm does] seem to be a key metaphor for what matters now in terms of organizing the world,” acknowledges McKenzie Wark, a media theorist who has written about digital technologies for the last 20 years. “If one thinks of algorithms as processes which terminate and generate a result, there’s a moment when the process ceases and you have your answer.
Simple Rules: How to Thrive in a Complex World by Donald Sull, Kathleen M. Eisenhardt
Affordable Care Act / Obamacare, Airbnb, asset allocation, Atul Gawande, barriers to entry, Basel III, Berlin Wall, carbon footprint, Checklist Manifesto, complexity theory, Craig Reynolds: boids flock, Credit Default Swap, Daniel Kahneman / Amos Tversky, diversification, en.wikipedia.org, European colonialism, Exxon Valdez, facts on the ground, Fall of the Berlin Wall, haute cuisine, invention of the printing press, Isaac Newton, Kickstarter, late fees, Lean Startup, Louis Pasteur, Lyft, Moneyball by Michael Lewis explains big data, Nate Silver, Network effects, obamacare, Paul Graham, performance metric, price anchoring, RAND corporation, risk/return, Saturday Night Live, sharing economy, Silicon Valley, Startup school, statistical model, Steve Jobs, TaskRabbit, The Signal and the Noise by Nate Silver, transportation-network company, two-sided market, Wall-E, web application, Y Combinator, Zipcar
., “Health on Impulse: When Low Self-Control Promotes Healthy Food Choices,” Health Psychology 33, no. 2 (2013): 103–9, http://www.medscape.com/medline/abstract/2347758. [>] In contrast, people: Brian Wansink, David R. Rust, and Collin R. Payne, “Mindless Eating and Healthy Heuristics for the Irrational,” American Economic Review: Papers and Proceedings 99, no. 2 (2009): 165–69. [>] Meteorologists make: Nate Silver, The Signal and the Noise (New York: Penguin, 2012), 126–27. [>] Japanese honeybees: Atsushi Ugajin et al., “Detection of Neural Activity in the Brains of Japanese Honeybee Workers During the Formation of a ‘Hot Defensive Bee Ball,’” PLoS One 7, no. 3 (2012), available at the website of the National Center for Biotechnology Information, http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3303784/. [>] As an example of: Our account of the bees’ choice of new nest is based on the research of Thomas Seeley, especially Thomas D.
Affordable Care Act / Obamacare, Black Swan, business intelligence, Carmen Reinhart, cognitive bias, correlation does not imply causation, Daniel Kahneman / Amos Tversky, Donald Trump, en.wikipedia.org, Kenneth Rogoff, labor-force participation, lake wobegon effect, Long Term Capital Management, Mercator projection, Mercator projection distort size, especially Greenland and Africa, meta analysis, meta-analysis, Nate Silver, obamacare, p-value, PageRank, pattern recognition, randomized controlled trial, risk-adjusted returns, Ronald Reagan, statistical model, The Signal and the Noise by Nate Silver, Tim Cook: Apple, wikimedia commons, Yogi Berra
“There is no scientifically plausible way of predicting the occurrence of a particular earthquake,” they note, adding that “prediction, as people expect it, requires predicting the magnitude, timing, and location of the future earthquake, which is not currently possible.”17 We simply don’t have the data, nor do we have the technology, to accurately predict quakes at this time. That said, the USGS does describe the places “most likely to produce earthquakes in the long term.” They call this forecasting, when they estimate the likelihood of a seismic event occurring over a period of time. This brings us to the distinction—or lack thereof—between a prediction and a forecast. As Nate Silver notes in The Signal and the Noise, the terms are used differently by some (most notably seismologists, who study earthquakes) but interchangeably by others. Some would argue that predictions are binary—something will or won’t happen—while forecasts are more probabilistic—there’s an X percent chance that something will happen. (To further complicate the issue, an estimate may be used when talking about past, current, or future data.)
The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies by Erik Brynjolfsson, Andrew McAfee
2013 Report for America's Infrastructure - American Society of Civil Engineers - 19 March 2013, 3D printing, access to a mobile phone, additive manufacturing, Airbnb, Albert Einstein, Amazon Mechanical Turk, Amazon Web Services, American Society of Civil Engineers: Report Card, Any sufficiently advanced technology is indistinguishable from magic, autonomous vehicles, barriers to entry, Baxter: Rethink Robotics, British Empire, business intelligence, business process, call centre, clean water, combinatorial explosion, computer age, computer vision, congestion charging, corporate governance, crowdsourcing, David Ricardo: comparative advantage, employer provided health coverage, en.wikipedia.org, Erik Brynjolfsson, factory automation, falling living standards, Filter Bubble, first square of the chessboard / second half of the chessboard, Frank Levy and Richard Murnane: The New Division of Labor, Freestyle chess, full employment, game design, global village, happiness index / gross national happiness, illegal immigration, immigration reform, income inequality, income per capita, indoor plumbing, industrial robot, informal economy, inventory management, James Watt: steam engine, Jeff Bezos, jimmy wales, job automation, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, Joseph Schumpeter, Kevin Kelly, Khan Academy, knowledge worker, Kodak vs Instagram, law of one price, low skilled workers, Lyft, Mahatma Gandhi, manufacturing employment, Mark Zuckerberg, Mars Rover, means of production, Narrative Science, Nate Silver, natural language processing, Network effects, new economy, New Urbanism, Nicholas Carr, Occupy movement, oil shale / tar sands, oil shock, pattern recognition, payday loans, price stability, Productivity paradox, profit maximization, Ralph Nader, Ray Kurzweil, recommendation engine, Report Card for America’s Infrastructure, Robert Gordon, Rodney Brooks, Ronald Reagan, Second Machine Age, self-driving car, sharing economy, Silicon Valley, Simon Kuznets, six sigma, Skype, software patent, sovereign wealth fund, speech recognition, statistical model, Steve Jobs, Steven Pinker, Stuxnet, supply-chain management, TaskRabbit, technological singularity, telepresence, The Bell Curve by Richard Herrnstein and Charles Murray, The Signal and the Noise by Nate Silver, The Wealth of Nations by Adam Smith, total factor productivity, transaction costs, Tyler Cowen: Great Stagnation, Vernor Vinge, Watson beat the top human players on Jeopardy!, winner-take-all economy, Y2K
., “Grading Student Loans,” Liberty Street Economics blog, Federal Reserve Bank of New York, March 5, 2012, http://libertystreeteconomics.newyorkfed.org/2012/03/grading-student-loans.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+LibertyStreetEconomics+(Liberty+Street+Economics). 21. Tim Hornyak, “Towel-folding Robot Won’t Do the Dishes,” CNET, March 31, 2010, http://news.cnet.com/8301-17938_105-10471898-1.html. 22. Nate Silver, The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t, 1st ed. (New York: Penguin, 2012). Chapter 13 POLICY RECOMMENDATIONS 1. “Employment Level,” Economic Research—Federal Reserve Bank of St. Louis (U.S. Department of Labor, Bureau of Labor Statistics, August 2, 2013), http://research.stlouisfed.org/fred2/series/LNU02000000. 2. Claudia Goldin and Lawrence F. Katz, The Race Between Education and Technology (Cambridge, MA: Belknap Press of Harvard University Press, 2010). 3.
business intelligence, business process, cellular automata, Celtic Tiger, cloud computing, collateralized debt obligation, conceptual framework, congestion charging, corporate governance, correlation does not imply causation, crowdsourcing, discrete time, George Gilder, Google Earth, Infrastructure as a Service, Internet Archive, Internet of things, invisible hand, knowledge economy, late capitalism, linked data, Masdar, means of production, Nate Silver, natural language processing, openstreetmap, pattern recognition, platform as a service, recommendation engine, RFID, semantic web, sentiment analysis, slashdot, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, smart grid, smart meter, software as a service, statistical model, supply-chain management, the scientific method, The Signal and the Noise by Nate Silver, transaction costs
Sicular, S. (2013) ‘Big data is falling into the trough of disillusionment’, Gartner, 22 January, http://blogs.gartner.com/svetlana-sicular/big-data-is-falling-into-the-trough-of-disillusionment/ (last accessed 26 February 2013). Siegel, E. (2013) Predictive Analytics. Wiley, Hoboken, NJ. Sigala, M. (2005) ‘Integrating customer relationship management in hotel operations: managerial and operations implications’, International Journal of Hospitality Management, 24(3): 391–413. Silver, N. (2012) The Signal and the Noise: The Art and Science of Prediction. Penguin, London. Singer, N. (2012a) ‘You for sale: mapping, and sharing, the consumer genome’, New York Times, 17 June, http://www.nytimes.com/2012/06/17/technology/acxiom-the-quiet-giant-of-consumerdatabase-marketing.html (last accessed 11 October 2013). Singer, N. (2012b) ‘F.T.C. opens an inquiry into data brokers’, New York Times, 18 December, http://www.nytimes.com/2012/12/19/technology/ftc-opens-an-inquiry-into-data-brokers.html (last accessed 11 October 2013).
Rubinstein, I.S. (2013) ‘Big data: the end of privacy or a new beginning?’, International Data Privacy Law, online first, http://idpl.oxfordjournals.org/content/early/2013/01/24/idpl.ips036.short (last accessed 15 July 2013). Ruppert, E. (2012) ‘The governmental topologies of database devices’, Theory, Culture Society, 29: 116–36. Ruppert, E. (2013) ‘Rethinking empirical social sciences’, Dialogues in Human Geography, 3(3): 268–73. Salmon, F. (2014) ‘Why the Nate Silvers of the world don’t know everything’, Wired, 7 January, http://www.wired.com/business/2014/01/quants-dont-know-everything/ (last accessed 8 January 2014). Salus, P. (1995) Casting the Net: From Arpanet to Internet and Beyond. Addison Wesley, Reading, MA. Sawyer, S. (2008) ‘Data wealth, data poverty, science and cyberinfrastructure’, Prometheus: Critical Studies in Innovation, 26(4): 355–71. Schnapp, J. and Presner, P. (2009) Digital Humanities Manifesto 2.0. http://www.humanitiesblast.com/manifesto/Manifesto_V2.pdf (last accessed 13 March 2013).
Stress Test: Reflections on Financial Crises by Timothy F. Geithner
Affordable Care Act / Obamacare, asset-backed security, Atul Gawande, bank run, banking crisis, Basel III, Bernie Madoff, Bernie Sanders, Buckminster Fuller, Carmen Reinhart, central bank independence, collateralized debt obligation, correlation does not imply causation, credit crunch, Credit Default Swap, credit default swaps / collateralized debt obligations, currency manipulation / currency intervention, David Brooks, Doomsday Book, eurozone crisis, financial innovation, Flash crash, Goldman Sachs: Vampire Squid, housing crisis, Hyman Minsky, illegal immigration, implied volatility, London Interbank Offered Rate, Long Term Capital Management, margin call, market fundamentalism, Martin Wolf, McMansion, Mexican peso crisis / tequila crisis, moral hazard, mortgage debt, Nate Silver, Northern Rock, obamacare, paradox of thrift, pets.com, price stability, profit maximization, pushing on a string, quantitative easing, race to the bottom, RAND corporation, regulatory arbitrage, reserve currency, Saturday Night Live, savings glut, short selling, sovereign wealth fund, The Great Moderation, The Signal and the Noise by Nate Silver, Tobin tax, too big to fail, working poor
That seemed highly unlikely, so Merrill usually kept the super-seniors on its balance sheet. Their modest returns were still more than the cost of financing them, and they seemed almost bulletproof. Standard & Poor’s estimated a mere 0.12 percent chance that one of its AAA-rated CDOs would fail to pay out over five years—and super-seniors were considered safer than typical AAAs. But as Nate Silver noted in The Signal and the Noise, his excellent book about why many predictions fail, the actual default rate for AAA-rated tranches of CDOs would be 28 percent, more than two hundred times higher than S&P had predicted. Their perceived safety rested on all kinds of flawed assumptions, starting with the notion that housing prices would never fall simultaneously across the country. CDOs were often spliced together from geographically diverse piles of subprime mortgages, which was supposed to mitigate the effects of a housing slump in any one region.