22 results back to index
Mastering Pandas by Femi Anthony
Amazon Web Services, Bayesian statistics, correlation coefficient, correlation does not imply causation, Debian, en.wikipedia.org, Internet of things, natural language processing, p-value, random walk, side project, statistical model, Thomas Bayes
Index A .at operatorabout / The .iat and .at operators Active State PythonURL / Third-party Python software installation aggregate methodusing / Using the aggregate method aggregation, in Rabout / Aggregation in R aliases, for Time Series frequenciesabout / Aliases for Time Series frequencies alphaabout / The alpha and p-values alternative hypothesisabout / The null and alternative hypotheses Anacondaabout / Continuum Analytics Anaconda URL / Continuum Analytics Anaconda, Final step for all platforms, Other numeric or analytics-focused Python distributions installing / Installing Anaconda URL, for download / Installing Anaconda installing, on Linux / Linux installing, on Mac OS/X / Mac OS X installing, on Windows / Windows installing, final steps / Final step for all platforms numeric or analytics-focused Python distributions / Other numeric or analytics-focused Python distributions IPython installation / Install via Anaconda (for Linux/Mac OS X) scikit-learn, installing via / Installing via Anaconda appendusing / Using append arithmetic operationsapplying, on columns / Arithmetic operations on columns B Bayesian analysis exampleswitchpoint detection / Bayesian analysis example – Switchpoint detection Bayesiansabout / How the model is defined Bayesian statistical analysisconducting, steps / Conducting Bayesian statistical analysis Bayesian statisticsabout / Introduction to Bayesian statistics reference link / Introduction to Bayesian statistics mathematical framework / Mathematical framework for Bayesian statistics references / Mathematical framework for Bayesian statistics, Applications of Bayesian statistics, References applications / Applications of Bayesian statistics versus Frequentist statistics / Bayesian statistics versus Frequentist statistics Bayes theoryabout / Bayes theory and odds Bernoulli distributionabout / The Bernoulli distribution reference link / The Bernoulli distribution big datareferences / We live in a big data world 4V’s / 4 V's of big data about / 4 V's of big data examples / The move towards real-time analytics binomial distributionabout / The binomial distribution Boolean indexingabout / Boolean indexing any() method / The is in and any all methods isin method / The is in and any all methods all method / The is in and any all methods where() method, using / Using the where() method indexes, operations / Operations on indexes C 4-4-5 calendarreference link / pandas/tseries central limit theoremreference link / Background central limit theorem (CLT)about / The mean classes, converter.pyConverter / pandas/tseries Formatters / pandas/tseries Locators / pandas/tseries classes, offsets.pyDateOffset / pandas/tseries BusinessMixin / pandas/tseries MonthOffset / pandas/tseries MonthBegin / pandas/tseries MonthEnd / pandas/tseries BusinessMonthEnd / pandas/tseries BusinessMonthBegin / pandas/tseries YearOffset / pandas/tseries YearBegin / pandas/tseries YearEnd / pandas/tseries BYearEnd / pandas/tseries BYearBegin / pandas/tseries Week / pandas/tseries WeekDay / pandas/tseries WeekOfMonth / pandas/tseries LastWeekOfMonth / pandas/tseries QuarterOffset / pandas/tseries QuarterEnd / pandas/tseries QuarterrBegin / pandas/tseries BQuarterEnd / pandas/tseries BQuarterBegin / pandas/tseries FY5253Quarter / pandas/tseries FY5253 / pandas/tseries Easter / pandas/tseries Tick / pandas/tseries classes, parsers.pyTextFileReader / pandas/io ParserBase / pandas/io CParserWrapper / pandas/io PythonParser / pandas/io FixedWidthReader / pandas/io FixedWithFieldParser / pandas/io classes, plm.pyPanelOLS / pandas/stats MovingPanelOLS / pandas/stats NonPooledPanelOLS / pandas/stats classes, sql.pyPandasSQL / pandas/io PandasSQLAlchemy / pandas/io PandasSQLTable / pandas/io PandasSQLTableLegacy / pandas/io PandasSQLLegacy / pandas/io columnmultiple functions, applying to / Applying multiple functions column namespecifying, in R / Specifying column name in R specifying, in pandas / Specifying column name in pandas columnsarithmetic operations, applying on / Arithmetic operations on columns concat functionabout / The concat function concat function, elementsobjs function / The concat function axis function / The concat function join function / The concat function join_axes function / The concat function keys function / The concat function concat operationreference link / The join function Condadocumentation, URL / Final step for all platforms conda commandURL / Final step for all platforms Confidence (Frequentist) intervalversus Credible (Bayesian) interval / Confidence (Frequentist) versus Credible (Bayesian) intervals confidence intervalabout / Confidence intervals example / An illustrative example container types, RVector / R data types List / R data types DataFrame / R data types Matrix / R data types continuous probability distributionsabout / Continuous probability distributions continuous uniform distribution / The continuous uniform distribution exponential distribution / The exponential distribution normal distribution / The normal distribution continuous uniform distributionabout / The continuous uniform distribution Continuum AnalyticsURL / Third-party Python software installation correlationabout / Correlation and linear regression, Correlation reference link / Correlation, An illustrative example Credible (Bayesian) intervalversus Confidence (Frequentist) interval / Confidence (Frequentist) versus Credible (Bayesian) intervals cross-sections / Cross sections cut() function, pandasabout / The pandas solution cut() method, Rabout / An R example using cut() reference link / An R example using cut() Cython / What is pandas?
A Tour of Statistics – The Classical Approach Descriptive statistics versus inferential statistics Measures of central tendency and variability Measures of central tendency The mean The median The mode Computing measures of central tendency of a dataset in Python Measures of variability, dispersion, or spread Range Quartile Deviation and variance Hypothesis testing – the null and alternative hypotheses The null and alternative hypotheses The alpha and p-values Type I and Type II errors Statistical hypothesis tests Background The z-test The t-test Types of t-tests A t-test example Confidence intervals An illustrative example Correlation and linear regression Correlation Linear regression An illustrative example Summary 8. A Brief Tour of Bayesian Statistics Introduction to Bayesian statistics Mathematical framework for Bayesian statistics Bayes theory and odds Applications of Bayesian statistics Probability distributions Fitting a distribution Discrete probability distributions Discrete uniform distributions The Bernoulli distribution The binomial distribution The Poisson distribution The Geometric distribution The negative binomial distribution Continuous probability distributions The continuous uniform distribution The exponential distribution The normal distribution Bayesian statistics versus Frequentist statistics What is probability? How the model is defined Confidence (Frequentist) versus Credible (Bayesian) intervals Conducting Bayesian statistical analysis Monte Carlo estimation of the likelihood function and PyMC Bayesian analysis example – Switchpoint detection References Summary 9.
A Brief Tour of Bayesian Statistics In this chapter, we will take a brief tour of an alternative approach to statistical inference called Bayesian statistics. It is not intended to be a full primer but just serve as an introduction to the Bayesian approach. We will also explore the associated Python-related libraries, how to use pandas, and matplotlib to help with the data analysis. The various topics that will be discussed are as follows: Introduction to Bayesian statistics Mathematical framework for Bayesian statistics Probability distributions Bayesian versus Frequentist statistics Introduction to PyMC and Monte Carlo simulation Illustration of Bayesian inference – Switchpoint detection Introduction to Bayesian statistics The field of Bayesian statistics is built on the work of Reverend Thomas Bayes, an 18th century statistician, philosopher, and Presbyterian minister.
Bayesian statistics, bioinformatics, British Empire, Claude Shannon: information theory, Daniel Kahneman / Amos Tversky, double helix, Edmond Halley, Fellow of the Royal Society, full text search, Henri Poincaré, Isaac Newton, John Markoff, John Nash: game theory, John von Neumann, linear programming, meta analysis, meta-analysis, Nate Silver, p-value, Pierre-Simon Laplace, placebo effect, prediction markets, RAND corporation, recommendation engine, Renaissance Technologies, Richard Feynman, Richard Feynman, Richard Feynman: Challenger O-ring, Ronald Reagan, speech recognition, statistical model, stochastic process, Thomas Bayes, Thomas Kuhn: the structure of scientific revolutions, traveling salesman, Turing machine, Turing test, uranium enrichment, Yom Kippur War
JASA (95) 1282–86 Couzin, Jennifer. (2004) The new math of clinical trials. Science (303) 784–86. DeGroot, Morris H. (1986b) A conversation with Persi Diaconis. Statistical Science (1:3) 319–34. Diaconis P, Efron B. (1983) Computer-intensive methods in statistics. Scientific American (248) 116–30. Diaconis, Persi. (1985) Bayesian statistics as honest work. Proceedings of the Berkeley Conference in Honor of Jerzy Neyman and Jack Kiefer (1), eds., Lucien M. Le Cam and Richard A. Olshen. Wadsworth. Diaconis P, Holmes S. (1996) Are there still things to do in Bayesian statistics? Erkenntnis (45) 145–58. Diaconis P. (1998) A place for philosophy? The rise of modeling in statistical science. Quarterly of Applied Mathematics (56:4) 797–805. DuMouchel WH, Harris JE. (1983) Bayes methods for combining the results of cancer studies in humans and other species.
Today, Bayes’ rule is used everywhere from DNA de-coding to Homeland Security. Drawing on primary source material and interviews with statisticians and other scientists, The Theory That Would Not Die is the riveting account of how a seemingly simple theorem ignited one of the greatest controversies of all time”—Provided by publisher. Includes bibliographical references and index. ISBN 978-0-300-16969-0 (hardback) 1. Bayesian statistical decision theory—History. I. Title. QA279.5.M415 2011 519.5’42—dc22 2010045037 A catalogue record for this book is available from the British Library. This paper meets the requirements of ANSI/NISO Z39.48–1992 (Permanence of Paper). 10 9 8 7 6 5 4 3 2 1 When the facts change, I change my opinion. What do you do, sir? —John Maynard Keynes contents Preface and Note to Readers Acknowledgments Part I.
Bayes combined judgments based on prior hunches with probabilities based on repeatable experiments. He introduced the signature features of Bayesian methods: an initial belief modified by objective new information. He could move from observations of the world to abstractions about their probable cause. And he discovered the long-sought grail of probability, what future mathematicians would call the probability of causes, the principle of inverse probability, Bayesian statistics, or simply Bayes’ rule. Given the revered status of his work today, it is also important to recognize what Bayes did not do. He did not produce the modern version of Bayes’ rule. He did not even employ an algebraic equation; he used Newton’s old-fashioned geometric notation to calculate and add areas. Nor did he develop his theorem into a powerful mathematical method. Above all, unlike Price, he did not mention Hume, religion, or God.
3D printing, AI winter, Amazon Web Services, artificial general intelligence, Asilomar, Automated Insights, Bayesian statistics, Bernie Madoff, Bill Joy: nanobots, brain emulation, cellular automata, Chuck Templeton: OpenTable, cloud computing, cognitive bias, commoditize, computer vision, cuban missile crisis, Daniel Kahneman / Amos Tversky, Danny Hillis, data acquisition, don't be evil, drone strike, Extropian, finite state, Flash crash, friendly AI, friendly fire, Google Glasses, Google X / Alphabet X, Isaac Newton, Jaron Lanier, John Markoff, John von Neumann, Kevin Kelly, Law of Accelerating Returns, life extension, Loebner Prize, lone genius, mutually assured destruction, natural language processing, Nicholas Carr, optical character recognition, PageRank, pattern recognition, Peter Thiel, prisoner's dilemma, Ray Kurzweil, Rodney Brooks, Search for Extraterrestrial Intelligence, self-driving car, semantic web, Silicon Valley, Singularitarianism, Skype, smart grid, speech recognition, statistical model, stealth mode startup, stem cell, Stephen Hawking, Steve Jobs, Steve Wozniak, strong AI, Stuxnet, superintelligent machines, technological singularity, The Coming Technological Singularity, Thomas Bayes, traveling salesman, Turing machine, Turing test, Vernor Vinge, Watson beat the top human players on Jeopardy!, zero day
But by the time the tragedy unfolded, Holtzman told me, Good had retired. He was not in his office but at home, perhaps calculating the probability of God’s existence. According to Dr. Holtzman, sometime before he died, Good updated that probability from zero to point one. He did this because as a statistician, he was a long-term Bayesian. Named for the eighteenth-century mathematician and minister Thomas Bayes, Bayesian statistics’ main idea is that in calculating the probability of some statement, you can start with a personal belief. Then you update that belief as new evidence comes in that supports your statement or doesn’t. If Good’s original disbelief in God had remained 100 percent, no amount of data, not even God’s appearance, could change his mind. So, to be consistent with his Bayesian perspective, Good assigned a small positive probability to the existence of God to make sure he could learn from new data, if it arose.
Aboujaoude, Elias accidents AI and, see risks of artificial intelligence nuclear power plant Adaptive AI affinity analysis agent-based financial modeling “Age of Robots, The” (Moravec) Age of Spiritual Machines, The: When Computers Exceed Human Intelligence (Kurzweil) AGI, see artificial general intelligence AI, see artificial intelligence AI-Box Experiment airplane disasters Alexander, Hugh Alexander, Keith Allen, Paul Allen, Robbie Allen, Woody AM (Automatic Mathematician) Amazon Anissimov, Michael anthropomorphism apoptotic systems Apple iPad iPhone Siri Arecibo message Aristotle artificial general intelligence (AGI; human-level AI): body needed for definition of emerging from financial markets first-mover advantage in jump to ASI from; see also intelligence explosion by mind-uploading by reverse engineering human brain time and funds required to develop Turing test for artificial intelligence (AI): black box tools in definition of drives in, see drives as dual use technology emotional qualities in as entertainment examples of explosive, see intelligence explosion friendly, see Friendly AI funding for jump to AGI from Joy on risks of, see risks of artificial intelligence Singularity and, see Singularity tight coupling in utility function of virtual environments for artificial neural networks (ANNs) artificial superintelligence (ASI) anthropomorphizing gradualist view of dealing with jump from AGI to; see also intelligence explosion morality of nanotechnology and runaway Artilect War, The (de Garis) ASI, see artificial superintelligence Asilomar Guidelines ASIMO Asimov, Isaac: Three Laws of Robotics of Zeroth Law of Association for the Advancement of Artificial Intelligence (AAAI) asteroids Atkins, Brian and Sabine Automated Insights availability bias Banks, David L. Bayes, Thomas Bayesian statistics Biden, Joe biotechnology black box systems Blue Brain project Bok globules Borg, Scott Bostrom, Nick botnets Bowden, B. V. brain augmentation of, see intelligence augmentation basal ganglia in cerebral cortex in neurons in reverse engineering of synapses in uploading into computer Brautigan, Richard Brazil Brooks, Rodney Busy Child scenario Butler, Samuel CALO (Cognitive Assistant that Learns and Organizes) Carr, Nicholas cave diving Center for Applied Rationality (CFAR) Chandrashekar, Ashok chatbots chess-playing computers Deep Blue China Chinese Room Argument Cho, Seung-Hui Church, Alonso Churchill, Winston Church-Turing hypothesis Clarke, Arthur C.
4chan, Any sufficiently advanced technology is indistinguishable from magic, Bayesian statistics, Brewster Kahle, buy low sell high, corporate governance, crowdsourcing, disintermediation, don't be evil, global village, Hacker Ethic, hypertext link, index card, informal economy, information retrieval, Internet Archive, invention of movable type, invention of writing, Isaac Newton, John Markoff, Lean Startup, moral panic, Paul Buchheit, Paul Graham, profit motive, RAND corporation, Republic of Letters, Richard Stallman, selection bias, semantic web, Silicon Valley, social web, Steve Jobs, Steven Levy, Stewart Brand, strikebreaker, Vannevar Bush, Whole Earth Catalog, Y Combinator
So we need an algorithm or computer program that would encourage lots of people to identify the fights and to start the campaigns,” McLean told the Sydney Morning Herald in 2014. “We’d put the tools that we have at our disposal in their hands.”32 Swartz had actually been building tools like these for several months with his colleagues at ThoughtWorks. Victory Kit, as the project was called, was an open-source version of the expensive community-organizing software used by groups such as MoveOn. Victory Kit incorporated Bayesian statistics—an analytical method that gets smarter as it goes along by consistently incorporating new information into its estimates—to improve activists’ ability to reach and organize their bases. “In the end, a lot of what the software was about was doing quite sophisticated A/B testing of messages for advocacy,” remembered Swartz’s friend Nathan Woodhull.33 Swartz was scheduled to present Victory Kit to the group at the Holmes retreat.
Ashcroft, 137–38, 140 FBI file on, 191–92, 223 fleeing the system, 8, 145, 151, 158–59, 161, 171, 173, 193, 248, 267 and free culture movement, 3–4, 141, 152–55, 167, 223 and Harvard, 3, 205, 207, 223, 224, 229 health issues of, 9, 150, 165–66, 222 immaturity of, 8–9 and Infogami, 147, 148–51, 158 interests of, 6–7, 8–9, 204, 221 “Internet and Mass Collaboration, The,” 166–67 lawyers for, 6, 254–55 legacy of, 14–15, 268, 269–70 and Library of Congress, 139 and Malamud, 187–93, 222, 223 manifesto of, 6–7, 178–81, 189–90, 201, 228–30, 247 mass downloading of documents by, 1, 3, 188–94, 197–202, 207, 213, 215, 222, 228, 235 media stories about, 125 and MIT, 1, 3, 201, 204, 207, 213, 222, 227, 232, 249–50, 262 and money, 170–71 on morality and ethics, 205–6 and Open Library, 163, 173, 179, 223, 228 and PCCC, 202–3, 225 as private person/isolation of, 2–3, 5, 124, 127, 143, 154–55, 158–60, 166, 169, 205, 224, 227, 228, 248–49, 251 and public domain, 123 as public speaker, 213–14, 224, 243, 257 and Reddit, see Reddit The Rules broken by, 14 “saving the world” on bucket list of, 7, 8, 15, 125, 151–52, 181, 205–6, 247–48, 266, 267, 268 self-help program of, 251–53 and theinfo.org, 172–73 and US Congress, 224–25, 239–40 Swartz, Robert: and Aaron’s death, 261, 262, 264 and Aaron’s early years, 124, 127 and Aaron’s legal woes, 232, 250, 254 and MIT Media Lab, 203–4, 212, 219, 232, 250 and technology, 124, 212 Swartz, Susan, 128–29, 160, 192 Swartz’s legal case: as “the bad thing,” 3, 7–8, 234 change in defense strategy, 256–57 evidence-suppression hearing, 259–60 facts of, 11 felony charges in, 235, 253 grand jury, 232–33 indictment, 1, 5, 8, 10, 11, 233, 234, 235–37, 241, 253–54 investigation and capture, 215–17, 223, 228 JSTOR’s waning interest in, 231–32 manifesto as evidence in, 228–30 motion to suppress, 6 motives sought in, 223, 229 Norton subpoenaed in, 1–2, 227–29 ongoing, 248, 249–51 online petitions against, 236–37 original charges in, 218, 222 plea deals offered, 227, 250 possible prison sentence, 1, 2, 5, 7–8, 11, 222, 232, 235–36, 253, 260 potential harm assessed, 218, 219, 222, 235 prosecutor’s zeal in, 7–8, 11, 218, 222–24, 235–37, 253–54, 259–60, 263, 264 search and seizure in, 6, 223–24, 256–57 Symbolics, 103 systems, flawed, 265–67 T. & J. W. Johnson, 49 Tammany Hall, New York, 57 tech bubble, 146, 156 technology: Bayesian statistics in, 258–59 burgeoning, 69, 71, 84, 87–88 communication, 12, 13, 18, 87–88 computing, see computers and digital culture, 122 and digital utopia, 91, 266–67 of electronic publishing, 120 and intellectual property, 90–91 and irrational exuberance, 146 in library of the future, 81–83 as magic, 152 moving inexorably forward, 134 overreaching police action against, 233 power of metadata, 128, 130 as private property, 210 resisting change caused by, 120 saving humanity via, 101 thinking machines, 102 unknown, future, 85 and World War II, 208 telephone, invention of, 69 Templeton, Brad, 261 theinfo.org, 172–73 theme parks, 134 ThoughtWorks, 9, 248, 257, 258 “thumb drive corps,” 187, 191, 193 Toyota Motor Corporation, “lean production” of, 7, 257, 265 Trumbull, John, McFingal, 26 trust-busting, 75 Tucher, Andie, 34 Tufte, Edward, 263–64 “tuft-hunter,” use of term, 28 Tumblr, 240 Twain, Mark, 60, 62, 73 Tweed, William “Boss,” 57 Twitter, 237 Ulrich, Lars, 133 United States: Articles of Confederation, 26 copyright laws in, 26–27 economy of, 44–45, 51, 55, 56 freedom to choose in, 80, 269 industrialization, 57 literacy in, 25, 26–27, 39, 44, 48 migration to cities in, 57 national identity of, 28, 32 new social class in, 69–70 opportunity in, 58, 80 poverty in, 59 railroads, 55, 56 rustic nation of, 44–45 values of, 85 UNIVAC computer, 81, 90 Universal Studios Orlando, 134 University of Illinois at Urbana-Champaign, 94, 95–96, 112–15 Unix, 104 US Chamber of Commerce, 239 utilitarianism, 214 Valenti, Jack, 111, 132 Van Buren, Martin, 44 Van Dyke, Henry, The National Sin of Literary Piracy, 61 venture capital, 146 Viaweb, 146 Victor, O.
Albert Einstein, barriers to entry, Bayesian statistics, Berlin Wall, business intelligence, carbon-based life, Claude Shannon: information theory, complexity theory, David Heinemeier Hansson, declining real wages, deliberate practice, discrete time, double helix, Douglas Engelbart, Douglas Engelbart, Downton Abbey, Drosophila, Firefox, Frank Gehry, Google X / Alphabet X, informal economy, invention of the printing press, inventory management, John Markoff, Khan Academy, Kickstarter, low skilled workers, Lyft, Marc Andreessen, Mark Zuckerberg, meta analysis, meta-analysis, natural language processing, Network effects, open borders, pattern recognition, Peter Thiel, pez dispenser, ride hailing / ride sharing, Ronald Reagan, Ruby on Rails, Sand Hill Road, self-driving car, Silicon Valley, Silicon Valley startup, social web, South of Market, San Francisco, speech recognition, Steve Jobs, technoutopianism, transcontinental railway, Vannevar Bush
In describing how the brain reacts to surprise, Lue said that “everything is a function of risk and opportunity.” To survive and prosper in the world with limited cognitive capacity, humans filter waves of constant sensory information through neural patterns—heuristics and mental shortcuts that our minds use to weigh the odds that what we are sensing is familiar and categorizable based on our past experience. Sebastian Thrun’s self-driving car does this with Bayesian statistics built into silicon and code, while the human mind uses electrochemical processes that we still don’t fully understand. But the underlying principle is the same: Based on the pattern of lines and shapes and edges, that is probably a boulder and I should drive around it. That is probably a group of three young women eating lunch at a table near the sushi bar and I should pay them no mind. Heuristics are also critically important to the market for higher education.
., 90–91, 98 Air Force, 91 Artificial Intelligence (AI), 11, 79, 136, 153, 159, 170, 264n Adaptive Control of Thought—Rational (ACT-R) model for, 101–4 cognitive tutoring using, 103, 105, 138, 179, 210 Dartmouth conference on, 79, 101 learning pathways for, 155 personalized learning with, 5, 232 theorem prover based in, 110 Thrun’s work in, 147–50 Arum, Richard, 9, 10, 36, 85, 244 Associate’s degrees, 6, 61, 117, 141, 193, 196, 198 Atlantic magazine, 29, 65, 79, 123 AT&T, 146 Australian National University, 204 Bachelor’s degrees, 6–9, 31, 36, 60–61, 64 for graduate school admission, 30 percentage of Americans with, 8, 9, 57, 77 professional versus liberal arts, 35 required for public school teachers, 117 social mobility and, 76 time requirement for, 6, 22 value in labor market of, 58 Badges, digital, 207–12, 216–18, 233, 245, 248 Barzun, Jacques, 32–34, 44, 45, 85 Bayesian statistics, 181 Bell Labs, 123–24 Bellow, Saul, 59, 78 Berlin, University of, 26, 45-46 Bhave, Amol, 214–15 Bing, 212 Binghamton, State University of New York at, 183–84 Bishay, Shereef, 139, 140 Bloomberg, Michael, 251 Blue Ocean Strategy (Kim and Mauborgne), 130 Bologna, University of, 16–17, 21, 41 Bonn, University of, 147 Bonus Army, 51 Borders Books, 127 Boston College, 164, 175 Boston Gazette, 95 Boston Globe, 2 Boston University (BU), 59, 61–62, 64 Bowen, William G., 112–13 Bowman, John Gabbert, 74–75 Brigham Young University, 2 Brilliant, 213 British Army, 98 Brookings Institution, 54 Brooklyn College, 44 Brown v.
airport security, availability heuristic, Bayesian statistics, Benoit Mandelbrot, Berlin Wall, Bernie Madoff, big-box store, Black Swan, Broken windows theory, Carmen Reinhart, Claude Shannon: information theory, Climategate, Climatic Research Unit, cognitive dissonance, collapse of Lehman Brothers, collateralized debt obligation, complexity theory, computer age, correlation does not imply causation, Credit Default Swap, credit default swaps / collateralized debt obligations, cuban missile crisis, Daniel Kahneman / Amos Tversky, diversification, Donald Trump, Edmond Halley, Edward Lorenz: Chaos theory, en.wikipedia.org, equity premium, Eugene Fama: efficient market hypothesis, everywhere but in the productivity statistics, fear of failure, Fellow of the Royal Society, Freestyle chess, fudge factor, George Akerlof, haute cuisine, Henri Poincaré, high batting average, housing crisis, income per capita, index fund, Intergovernmental Panel on Climate Change (IPCC), Internet Archive, invention of the printing press, invisible hand, Isaac Newton, James Watt: steam engine, John Nash: game theory, John von Neumann, Kenneth Rogoff, knowledge economy, locking in a profit, Loma Prieta earthquake, market bubble, Mikhail Gorbachev, Moneyball by Michael Lewis explains big data, Monroe Doctrine, mortgage debt, Nate Silver, negative equity, new economy, Norbert Wiener, PageRank, pattern recognition, pets.com, Pierre-Simon Laplace, prediction markets, Productivity paradox, random walk, Richard Thaler, Robert Shiller, Robert Shiller, Rodney Brooks, Ronald Reagan, Saturday Night Live, savings glut, security theater, short selling, Skype, statistical model, Steven Pinker, The Great Moderation, The Market for Lemons, the scientific method, The Signal and the Noise by Nate Silver, The Wisdom of Crowds, Thomas Bayes, Thomas Kuhn: the structure of scientific revolutions, too big to fail, transaction costs, transfer pricing, University of East Anglia, Watson beat the top human players on Jeopardy!, wikimedia commons
Scott Armstrong, The Wharton School, University of Pennsylvania LIBRARY OF CONGRESS CATALOGING IN PUBLICATION DATA Silver, Nate. The signal and the noise : why most predictions fail but some don’t / Nate Silver. p. cm. Includes bibliographical references and index. ISBN 978-1-101-59595-4 1. Forecasting. 2. Forecasting—Methodology. 3. Forecasting—History. 4. Bayesian statistical decision theory. 5. Knowledge, Theory of. I. Title. CB158.S54 2012 519.5'42—dc23 2012027308 While the author has made every effort to provide accurate telephone numbers, Internet addresses, and other contact information at the time of publication, neither the publisher nor the author assumes any responsibility for errors, or for changes that occur after publication. Further, publisher does not have any control over and does not assume any responsibility for author or third-party Web sites or their content.
In essence, this player could go to work every day for a year and still lose money. This is why it is sometimes said that poker is a hard way to make an easy living. Of course, if this player really did have some way to know that he was a long-term winner, he’d have reason to persevere through his losses. In reality, there’s no sure way for him to know that. The proper way for the player to estimate his odds of being a winner, instead, is to apply Bayesian statistics,31 where he revises his belief about how good he really is, on the basis of both his results and his prior expectations. If the player is being honest with himself, he should take quite a skeptical attitude toward his own success, even if he is winning at first. The player’s prior belief should be informed by the fact that the average poker player by definition loses money, since the house takes some money out of the game in the form of the rake while the rest is passed around between the players.32 The Bayesian method described in the book The Mathematics of Poker, for instance, would suggest that a player who had made $30,000 in his first 10,000 hands at a $100/$200 limit hold ’em game was nevertheless more likely than not to be a long-term loser.
McGrayne, The Theory That Would Not Die, Kindle location 7. 61. Raymond S. Nickerson, “Null Hypothesis Significance Testing: A Review of an Old and Continuing Controversy,” Psychological Methods, 5, 2 (2000), pp. 241–301. http://126.96.36.199/richman/plogxx/gallery/17/%E9%AB%98%E7%B5%B1%E5%A0%B1%E5%91%8A.pdf. 62. Andrew Gelman and Cosma Tohilla Shalizi, “Philosophy and the Practice of Bayesian Statistics,” British Journal of Mathematical and Statistical Psychology, pp. 1–31, January 11, 2012. http://www.stat.columbia.edu/~gelman/research/published/philosophy.pdf. 63. Although there are several different formulations of the steps in the scientific method, this version is mostly drawn from “APPENDIX E: Introduction to the Scientific Method,” University of Rochester. http://teacher.pas.rochester.edu/phy_labs/appendixe/appendixe.html. 64.
Albert Einstein, banking crisis, Bayesian statistics, cognitive bias, end world poverty, endowment effect, energy security, experimental subject, framing effect, hindsight bias, impulse control, John Nash: game theory, loss aversion, meta analysis, meta-analysis, out of africa, pattern recognition, placebo effect, Ponzi scheme, Richard Feynman, Richard Feynman, risk tolerance, stem cell, Stephen Hawking, Steven Pinker, the scientific method, theory of mind, ultimatum game, World Values Survey
If we are measuring sanity in terms of sheer numbers of subscribers, then atheists and agnostics in the United States must be delusional: a diagnosis which would impugn 93 percent of the members of the National Academy of Sciences.63 There are, in fact, more people in the United States who cannot read than who doubt the existence of Yahweh.64 In twenty-first-century America, disbelief in the God of Abraham is about as fringe a phenomenon as can be named. But so is a commitment to the basic principles of scientific thinking—not to mention a detailed understanding of genetics, special relativity, or Bayesian statistics. The boundary between mental illness and respectable religious belief can be difficult to discern. This was made especially vivid in a recent court case involving a small group of very committed Christians accused of murdering an eighteen-month-old infant.65 The trouble began when the boy ceased to say “Amen” before meals. Believing that he had developed “a spirit of rebellion,” the group, which included the boy’s mother, deprived him of food and water until he died.
The ACC and the caudate display an unusual degree of connectivity, as the surgical lesioning of the ACC (a procedure known as a cingulotomy) causes atrophy of the caudate, and the disruption of this pathway is thought to be the basis of the procedure’s effect in treating conditions like obsessive-compulsive disorder (Rauch et al., 2000; Rauch et al., 2001). There are, however, different types of uncertainty. For instance, there is a difference between expected uncertainty—where one knows that one’s observations are unreliable—and unexpected uncertainty, where something in the environment indicates that things are not as they seem. The difference between these two modes of cognition has been analyzed within a Bayesian statistical framework in terms of their underlying neurophysiology. It appears that expected uncertainty is largely mediated by acetylcholine and unexpected uncertainty by norepinephrine (Yu & Dayan, 2005). Behavioral economists sometimes distinguish between “risk” and “ambiguity”: the former being a condition where probability can be assessed, as in a game of roulette, the latter being the uncertainty borne of missing information.
3D printing, Albert Einstein, Amazon Mechanical Turk, Arthur Eddington, basic income, Bayesian statistics, Benoit Mandelbrot, bioinformatics, Black Swan, Brownian motion, cellular automata, Claude Shannon: information theory, combinatorial explosion, computer vision, constrained optimization, correlation does not imply causation, creative destruction, crowdsourcing, Danny Hillis, data is the new oil, double helix, Douglas Hofstadter, Erik Brynjolfsson, experimental subject, Filter Bubble, future of work, global village, Google Glasses, Gödel, Escher, Bach, information retrieval, job automation, John Markoff, John Snow's cholera map, John von Neumann, Joseph Schumpeter, Kevin Kelly, lone genius, mandelbrot fractal, Mark Zuckerberg, Moneyball by Michael Lewis explains big data, Narrative Science, Nate Silver, natural language processing, Netflix Prize, Network effects, NP-complete, off grid, P = NP, PageRank, pattern recognition, phenotype, planetary scale, pre–internet, random walk, Ray Kurzweil, recommendation engine, Richard Feynman, Richard Feynman, Second Machine Age, self-driving car, Silicon Valley, speech recognition, statistical model, Stephen Hawking, Steven Levy, Steven Pinker, superintelligent machines, the scientific method, The Signal and the Noise by Nate Silver, theory of mind, Thomas Bayes, transaction costs, Turing machine, Turing test, Vernor Vinge, Watson beat the top human players on Jeopardy!, white flight, zero-sum game
The distinction between descriptive and normative theories was articulated by John Neville Keynes in The Scope and Method of Political Economy (Macmillan, 1891). Chapter Six Sharon Bertsch McGrayne tells the history of Bayesianism, from Bayes and Laplace to the present, in The Theory That Would Not Die (Yale University Press, 2011). A First Course in Bayesian Statistical Methods,* by Peter Hoff (Springer, 2009), is an introduction to Bayesian statistics. The Naïve Bayes algorithm is first mentioned in Pattern Classification and Scene Analysis,* by Richard Duda and Peter Hart (Wiley, 1973). Milton Friedman argues for oversimplified theories in “The methodology of positive economics,” which appears in Essays in Positive Economics (University of Chicago Press, 1966). The use of Naïve Bayes in spam filtering is described in “Stopping spam,” by Joshua Goodman, David Heckerman, and Robert Rounthwaite (Scientific American, 2005).
Albert Einstein, Bayesian statistics, Black-Scholes formula, Bretton Woods, Brownian motion, capital asset pricing model, collateralized debt obligation, correlation coefficient, Credit Default Swap, credit default swaps / collateralized debt obligations, David Ricardo: comparative advantage, discovery of penicillin, discrete time, Emanuel Derman, en.wikipedia.org, Eugene Fama: efficient market hypothesis, financial innovation, fixed income, floating exchange rates, full employment, Henri Poincaré, implied volatility, index fund, Isaac Newton, John Meriwether, John von Neumann, Joseph Schumpeter, Kenneth Arrow, Long Term Capital Management, Louis Bachelier, margin call, market clearing, martingale, means of production, moral hazard, Myron Scholes, naked short selling, Paul Samuelson, price stability, principal–agent problem, quantitative trading / quantitative ﬁnance, RAND corporation, random walk, risk tolerance, risk/return, Ronald Reagan, shareholder value, Sharpe ratio, short selling, stochastic process, The Chicago School, the scientific method, too big to fail, transaction costs, tulip mania, Works Progress Administration, yield curve
He postulated that the rational decision-maker will align his or her beliefs of unknown probabilities to the consensus bets of impartial bookmakers, a technique often called the Dutch Book. Thirty later, the great mind Leonard “Jimmie” Savage (1917–1971) elaborated his concept into an axiomatic approach to decision-making under uncertainty using arguments remarkably similar to Ramsey’s logic. The concepts of Ramsey and Savage also formed the basis for the theory of Bayesian statistics and are important in many aspects of financial decision-making. Marschak’s great insight While Ramsey created and Savage broadened the logical landscape for the inclusion of uncertainty into decision-making, it was not possible to incorporate their logic until the finance discipline could develop actual measures of uncertainty. Of course, modern financial analysis depends crucially even today on such a methodology to measure uncertainty.
Thinking, Fast and Slow by Daniel Kahneman
Albert Einstein, Atul Gawande, availability heuristic, Bayesian statistics, Black Swan, Cass Sunstein, Checklist Manifesto, choice architecture, cognitive bias, complexity theory, correlation coefficient, correlation does not imply causation, Daniel Kahneman / Amos Tversky, delayed gratification, demand response, endowment effect, experimental economics, experimental subject, Exxon Valdez, feminist movement, framing effect, hindsight bias, index card, information asymmetry, job satisfaction, John von Neumann, Kenneth Arrow, libertarian paternalism, loss aversion, medical residency, mental accounting, meta analysis, meta-analysis, nudge unit, pattern recognition, Paul Samuelson, pre–internet, price anchoring, quantitative trading / quantitative ﬁnance, random walk, Richard Thaler, risk tolerance, Robert Metcalfe, Ronald Reagan, The Chicago School, The Wisdom of Crowds, Thomas Bayes, transaction costs, union organizing, Walter Mischel, Yom Kippur War
So if you believe that there is a 40% chance plethat it will rain sometime tomorrow, you must also believe that there is a 60% chance it will not rain tomorrow, and you must not believe that there is a 50% chance that it will rain tomorrow morning. And if you believe that there is a 30% chance that candidate X will be elected president, and an 80% chance that he will be reelected if he wins the first time, then you must believe that the chances that he will be elected twice in a row are 24%. The relevant “rules” for cases such as the Tom W problem are provided by Bayesian statistics. This influential modern approach to statistics is named after an English minister of the eighteenth century, the Reverend Thomas Bayes, who is credited with the first major contribution to a large problem: the logic of how people should change their mind in the light of evidence. Bayes’s rule specifies how prior beliefs (in the examples of this chapter, base rates) should be combined with the diagnosticity of the evidence, the degree to which it favors the hypothesis over the alternative.
.); WYSIATI (what you see is all there is) and associative memory; abnormal events and; anchoring and; causality and; confirmation bias and; creativity and; and estimates of causes of death Åstebro, Thomas Atlantic, The attention; in self-control paneight="0%" width="-5%"> Attention and Effort (Kahneman) Auerbach, Red authoritarian ideas availability; affect and; and awareness of one’s biases; expectations about; media and; psychology of; risk assessment and, see risk assessment availability cascades availability entrepreneurs bad and good, distinctions between banks bank teller problem Barber, Brad Bargh, John baseball baseball cards baseline predictions base rates; in cab driver problem; causal; in helping experiment; low; statistical; in Tom W problem; in Yale exam problem basic assessments basketball basketball tickets bat-and-ball problem Baumeister, Roy Bayes, Thomas Bayesian statistics Bazerman, Max Beane, Billy Beatty, Jackson Becker, Gary “Becoming Famous Overnight” (Jacoby) behavioral economics Behavioral Insight Team “Belief in the Law of Small Numbers” (Tversky and Kahneman) beliefs: bias for; past, reconstruction of Benartzi, Shlomo Bentham, Jeremy Berlin, Isaiah Bernoulli, Daniel Bernouilli, Nicholas Beyth, Ruth bicycle messengers Black Swan, The (Taleb) blame Blink (Gladwell) Borg, Björn Borgida, Eugene “Boys Will Be Boys” (Barber and Odean) Bradlee, Ben brain; amygdala in; anterior cingulate in; buying and selling and; emotional framing and; frontal area of; pleasure and; prefrontal area of; punishment and; sugar in; threats and; and variations of probabilities British Toxicology Society broad framing Brockman, John broken-leg rule budget forecasts Built to Last (Collins and Porras) Bush, George W.
Analysis of Financial Time Series by Ruey S. Tsay
Asian financial crisis, asset allocation, Bayesian statistics, Black-Scholes formula, Brownian motion, capital asset pricing model, compound rate of return, correlation coefficient, data acquisition, discrete time, frictionless, frictionless market, implied volatility, index arbitrage, Long Term Capital Management, market microstructure, martingale, p-value, pattern recognition, random walk, risk tolerance, short selling, statistical model, stochastic process, stochastic volatility, telemarketer, transaction costs, value at risk, volatility smile, Wiener process, yield curve
In this chapter, we introduce the ideas of MCMC methods and data augmentation that are widely applicable in finance. In particular, we discuss Bayesian inference via Gibbs sampling and demonstrate various applications of MCMC methods. Rapid developments in the MCMC methodology make it impossible to cover all the new methods available in the literature. Interested readers are referred to some recent books on Bayesian and empirical Bayesian statistics (e.g., Carlin and Louis, 2000; Gelman, Carlin, Stern, and Rubin, 1995). For applications, we focus on issues related to financial econometrics. The demonstrations shown in this chapter only represent a small fraction of all possible applications of the techniques in finance. As a matter of fact, it is fair to say that Bayesian inference and the MCMC methods discussed here are applicable to most, if not all, of the studies in financial econometrics.
Such a prior distribution is called a conjugate prior distribution. For MCMC methods, use of conjugate priors means that a closed-form solution for the conditional posterior distributions is available. Random draws of the Gibbs sampler can then be obtained by using the commonly available computer routines of probability distributions. In what follows, we review some well-known conjugate priors. For more information, readers are referred to textbooks on Bayesian statistics (e.g., DeGroot, 1970, Chapter 9). Result 1: Suppose that x1 , . . . , xn form a random sample from a normal distribution with mean µ, which is unknown, and variance σ 2 , which is known and positive. Suppose that the prior distribution of µ is a normal distribution with mean µo and variance σo2 . Then the posterior distribution of µ given the data and prior is 401 BAYESIAN INFERENCE normal with mean µ∗ and variance σ∗2 given by µ∗ = σ 2 µo + nσo2 x̄ σ 2 + nσo2 and σ∗2 = σ 2 σo2 , σ 2 + nσo2 n xi /n is the sample mean. where x̄ = i=1 In Bayesian analysis, it is often convenient to use the precision parameter η = 1/σ 2 (i.e., the inverse of the variance σ 2 ).
Data Scientists at Work by Sebastian Gutierrez
Albert Einstein, algorithmic trading, Bayesian statistics, bioinformatics, bitcoin, business intelligence, chief data officer, clean water, cloud computing, commoditize, computer vision, continuous integration, correlation does not imply causation, creative destruction, crowdsourcing, data is the new oil, DevOps, domain-specific language, Donald Knuth, follow your passion, full text search, informal economy, information retrieval, Infrastructure as a Service, Intergovernmental Panel on Climate Change (IPCC), inventory management, iterative process, lifelogging, linked data, Mark Zuckerberg, microbiome, Moneyball by Michael Lewis explains big data, move fast and break things, move fast and break things, natural language processing, Network effects, nuclear winter, optical character recognition, pattern recognition, Paul Graham, personalized medicine, Peter Thiel, pre–internet, quantitative hedge fund, quantitative trading / quantitative ﬁnance, recommendation engine, Renaissance Technologies, Richard Feynman, Richard Feynman, self-driving car, side project, Silicon Valley, Skype, software as a service, speech recognition, statistical model, Steve Jobs, stochastic process, technology bubble, text mining, the scientific method, web application
Our vision was building ubiquitous machine learning. We were really inspired by companies like Heroku and Twilio and the way they had democratized access to a lot of what at the time was fairly cutting-edge technology. We felt it was crucial to preserve uncertainty—the ability to say “I don’t know the answer”—when putting this technology in the hands of normal people. At the time, that was a really radical thought. Certainly, the Bayesian statistical community had been doing this for a long time, but a lot of machine learning methods that were out there were just all about giving people an answer. It is important to ask whether you can trust this answer or not, especially since often whether or not you can trust an answer varies greatly on the question you happen to be asking. Machine learning systems will quite reliably tell you that there are two genders, male and female, but they won’t always be accurate in predicting which one you’re talking to.
., 139 UMSI, 148 Wall Street, 131 Wilson, Fred (investors), 141 Electrical engineering and computer science (EECS), 295 Embedding, 52 F www.it-ebooks.info Index G Galene, 87 Global Bay Mobile Technologies, 131 Google, 89, 91, 311 LeCun,Yann, 58 Lenaghan, Jonathan, 185, 197 Smallwood, Caitlin, 27 Tunkelang, Daniel, 89, 91 Grameen Foundation, 319 H Hadoop Distributed File System (HDFS), 195 Hadoop software, 103 Heineike, Amy advice to fresher, 254 Cambridge University, 239 career, 240 Crawford, Kate (researcher), 250 Cukier, Kenneth (big data editor), 250 data exploration, 251 data science colleagues, 257 data visualization, 239, 245 Director of Mathematics, 239, 241 economic consultancy work, 246 engineering team, 242 first data set, 247 government agencies, 239 The Guardian, 249 hedge funds, 239 hiring people, 255 machine learning, 239, 245 math and programming interest, 246 natural language processing, 239 network diagram, 251 network science, 239, 245 new data streams, 244 news sources, 244 next challenges, 249 NLP, 245 optimize user experience, 255 Ormerod, Paul (director), 247 own success measurement, 252 Quid’s product, 242 research and strategy analysis, 248 Rosewell, Bridget (director), 248 scaling up, 244 Software-as-a-Service platform, 242 software development toolkit, 249 spreadsheet/PowerPoint slide, 245 Strata conference, 250 study and analyze, 252 tools/techniques, 256 Twitter feed/posts, 249, 253 typical day, 251 users, 253 Volterra Consulting, 247 Hu,Victor advanced machine learning, 272 advice to fresher, 269 artist evolution, 267 Billboard 200, 266 book division, 264 Chief Data Scientist, (Next Big Sound), 259 college life, 260 cross-sectional analyses, 271 D3.js visualizations, 268 DataKind Data Dive, 269 data management track, 262 data team structure, 265 effective data visualization, 262 effective writing, 262 first project, 264 forecasting album sales, 263 hiring data scientists, 268 impactful feedback, 270 Java and PHP, 268 machine learning techniques, 262, 264 modeling process, 268 new skills, 262 NLP projects, 270 nonwork data sets, 270 personal believe, 272 PrestoDB, 268 product team and customer team, 266 record label deal, 264 R/Python, 268 skill acquisition, 261 social and music industry data, 263 social media, 263 SoundCloud, 263 theoretical techniques, 266 traditional industries, 260 Twitter profile, 265 two-week project cycles, 265 Yankees experience, 261 www.it-ebooks.info 337 338 Index I new BRAIN initiative, 295 nontechnical advice, 317 other people’s work expedition, 309 Palantir, 300–301 Peter Norvig philosophy, 311 Peter Thiel’s group, 294 Peter Thiel’s venture capital firm, 293 as PhD student, 303 PhD theses, 300 predictive analytics, 311 Prior Knowledge (PK), 294, 302 “Probability Theory: The Logic of Science”, 316 problem solving, 308 Python, 306 quantitative and computational science, 316 research scientist, 293 Salesforce, 302 science tool companies, 301 Sebastian Seung’s group, 296 as startup CEO, 303 study and analyze, 295, 297, 305 TechCrunch Disrupt, 294 VC money, 307 IA Ventures, 131 Institute for Data Sciences and Engineering (IDSE), 1 International Conference on Learning Representation (ICLR), 51 Invite Media, 131, 137 J Jonas, Eric advice to fresher, 313 Bayesian models, 306 Bayesian nonparametric models, 315 Bayesian statistical community, 294 Berkeley’s AMPLab, 312 biological systems, 299 Brown, Scott (CEO of Vicarious and Bob Mcgrew), 300 C++, 306 Chief Predictive Scientist, 293 C++ numerical methods, 317 computational neuroscience, 298, 311 COSYNE conference, 296, 303 DARPA, 307 data sets, 312 dating spread sheet, 310 EECS and BCS sciences, 297 experiences, 300 Founders Fund, 294 funding agencies, 298 Grande Data problems, 312 Hadoop, 307 Harvard Business Review, 311 hiring people, 314 as independent researcher, 304 Kording, Konrad (scientist), 296 LASSO-based linear regression system, 316 Levie, Aaron (Box CEO), 295 machine learning models, 293 Markov chain Monte Carlo, 315 Matplotlib, 309 Nature Reviews Genetics, 300 Nature Reviews Neuroscience, 300 Navia Systems, 293 neuroscience research, 294 neuroscience tools, 301 K Karpištšenko, André ADCP, 223 advice to starter, 235 atmosphere capturing data, 221 Brute-force analytics, 231 career, 222 challenges, 234 climate changes, 226 co-founder and research lead (Planet OS), 221 co-founder (ASA Quality Services), 221 communication patterns, 225 Cornell and Rutgers (partners), 227 create models, 237 creating and writing software, 237 daily operation decisions, 222 data mining methods, 225 data privacy and trust, 223 data science, future of, 234 engineering and calibration tools, 225 engineering, data, and technology side, 222 www.it-ebooks.info Index European environmental agency, 224 FuturICT program, 224 Gephi, 231 Greenplum, 231 hiring, 232 insurance companies, 227 investment decisions, 222 IPCC reports, 237 Kaggle data science community, 228 machine learning methods, 225 model-driven development, 224 music generation algorithms, 237 NASA, 227 National Data Buoy Center profilers, 223 NOAA, 227 ocean infrastructure, 223 oceanography, 222 OPeNDAP data exchange protocol, 228 past year project, 233 people’s work, expectation, 234 personal operational things, 229 physical world, 226 problem solving, 230 public transport ticketing system, 224 Python, 231 R, 231 Sharemind, 236 shipping, oil, and gas industries, 226 success and measure success, 229 team building, 232 team members, 223 Vowpal Wabbit, 232 Weka, 232 Keyhole Markup Language (KML), 166 Knight Foundation, 319 L LASSO-based linear regression system, 316 LeCun,Yann academic scientific disciplines, 63 AI research materials, 50 artificial neural networks with back-propagation, 45 check-reading/check-recognition system, 52 content analysis and understanding users, 57 convolutional net, 53 CVPR 2014 conference papers, 57 daily work, 54 data sets, 50 deep learning, 45, 47, 52 Director of AI Research, Facebook, 45–46 DjVu system, 54 educational courses, 64 EMNLP 2014 papers, 57 experimental work, 60 first data set, 49 future of data science, 62 grad school, 48 graduation,VLSI integrated circuit design and automatic control, 48 graphical models, 51 Hinton,Geoff, 48 hiring research scientists and exceptional engineers, 65 ICML conference, 51 Image Processing Research Department head, AT&T Bell Labs, 45–46 image recognition, 57 industry impacts deployed things, 55 intellectual impact, 54 research projects, 55 initial understanding of data science, 65 innovative and creative work, 60 inspired work, 59 interest in AI, 48 kernel methods, 51 language translation, 64 long-term goal, understanding intelligence, 47 machine learning, 51–52 mathematical virtuosity, 60 Memory Networks, 57 multi-layer neural nets learning, 48 natural language, 52, 57 NEC Research Institute, 45 opportunities for data science, 65 opportunity at Facebook, 46 organizing research lab, 55 radar techniques, 51 www.it-ebooks.info 339 340 Index LeCun,Yann (cont.) robotics, Google, 58 Sejnowski, Terry, 48 shorter-term goal, understanding content, 47 Silver Professor of Computer Science, New York University, 45–46 solving right problem, 58 supervised and unsupervised learning, 46 support vector machines, 51 team members, 46 tools Lush project, 56 Torch 7, 56 undergrad, electrical engineering, 47 unsupervised learning, 61 Lenaghan, Jonathan career, 179, 182 engineering practices, 198 financial vs. ad tech industry, 184 Google, 197 high-frequency algorithmic trading, 183 mathematical consistency, 183 overfitting models, 183 packages and libraries, 196 Physical Review in Brookhaven, 183 PlaceIQ academic literature, 185 academic research, 185 ad-request logs, 181 ad-supported apps, 189 Air Traveler audience, 192 algorithmic trading, 180 Amazon’s S3 service, 194 ambient background location app, 184 architecture and long-term planning, 186 augmented intelligence service, 190 building models, 186 campaigns, 186 C++ and Perl, 190 Clojure projects, 195 data science team, 186 data scientist, 180 data sets, 180 data volume, 183 demographic results, 193 device IDs, 194 discussions with account managers, 187 domain, 190 explosive growth, 186 final output and functionality, 187 financial services, 180 geography, 185 geospatial layer, 182 geospatial location analytics, 181 geospatial visualization, 191 Google, 185 high-QPS and low-latency environment, 196 human behavior, 185 identifier, 194 ingest, transformation, and contextualization, 182 initial prototype, 188 interview process, 195 Julia, 195 location targeting, 192 matplotlib, 191 meetup, 180 mobile ads and location intelligence, 179 munging data, 191 ontology/taxonomy, 192 potential APIs, 190 problem solving, 187, 194 product/troubleshooting meetings, 187 prototype code, 188 prototype performs, 188 Python, 190 qualitative terms, 182 query language, 192–193 real-time processing and computation affects, 196 residential/nonresidential classifiers, 193 scaling test, 188 self-critical, 195 smartphones, 189 social anthropology, 189 spatial dimension testing, 188 temporal data, 191 tile level, 192 www.it-ebooks.info Index time periods, 180 urgency, 184 quantitative fields, 197 quantitative finance industry, 183 quantum chromodynamics, 182 relational queries, 196 software engineering skills, 198 Starbucks, 197 statistical analyses, 196 tech ad space, 181 testing, 198 LinkedIn, 34, 83 Lua programming language, 56 M Magnetic, 131 MailChimp.com, 108 Mandrill, 111–112 Media6Degrees, 151 Moneyball, 260 N National Data Buoy Center profilers, 223 National Security Agency, 107 Natural language, 52, 57 Natural Language Processing (NLP), 245 Netflix Culture, 20 Network operations Center (NOC), 120 The New York Times (NYT), 1 Neural net version 1.0, 48 Neural net winter, 48 Next Big Sound, 259 O O’Reilly Strata, 95 P, Q PANDA, 57 Perlich, Claudia AAAI, 162 advertising exchanges, 154 algorithm sorts list, 167 appreciation, 168 artificial neural networks, 152 Bayesian theory, 172 career, 151 challenges, 158 chief scientist (Dstillery), 151 communication, 171 Dstillery’s history and focus, 153 engineering team, 167 first data set, 158 fraud problem, 160 Hadoop cluster, 165 hire data scientists, 173 IBM’s Watson Research Center, 152 insight translate, 158 interview, 172 Journal of Advertising Research and American Marketing Association, 161 Kaggle website, 177 KML, 166 learning lessons, 166–167 long-tailed distributions, 170 machine learning techniques, 177 managerial responsibilities, 154 marksmanship, 167 matchmaking, 169 medical field, 175–176 Melinda approach, 172 Nielsen reports, 172 nonwork data puzzles, 157 NoSQL, 165 NYU Stern Business School, 152 PhD program, 152 photo-sharing URL, 154 present and future, data science, 174 prototype building tasks, 164 Provost, Foster (PhD advisor), 153 puzzles, 156 responsibilities, 155 routine tasks, 163 single-numbered aggregates, 170 statistical measures, 170 SVMlight, 165 teach data mining, 151 team members, 154 typical day, 161 www.it-ebooks.info 341 342 Index Perlich, Claudia (cont.) typical intellectual leadership day, 162 typical modeling and analysis day, 162 Personally identifiable information (PII), 181 Porway, Jake academia/government labs, 331 Amnesty International, 323–324 artificial intelligence, 325 career, 319 communication/translation challenges, 322 computer vision course, 325 data science, future of, 330 ethical responsibility, 331 founder and executive director (DataKind), 319 global chapter network, 323 Grameen Foundation’s Community Knowledge Program, 327 “The Human Face of Big Data”, 326 hunger alleviation experts, 321 Kenyan village, 322 Kirkpatrick, Robert (UN Global Pulse), 332 “The Macroscope”, 329 measure success, 330 Netflix, 325 New York Times, 324 OkCupid, 332 personal philosophies, 332 PhD program, 326 playbook, 322 problem solving, 325 pro bono service, 324 Rees, Kim, 329 Stanford Social Innovation Review, 327 statistical background, 331 statistics department, 326 team and organization, 320 Thorp, Jer, 329 typical workday, 327 volunteer, 325 volunteer data scientists, 322 Predictive Analytics Innovation Summit Chicago 2013 conference, 262 Prior Knowledge (PK), 293 R Radinsky, Kira computer science bioinformatics, 280 causality graph, 282 causes and effects, 283 cholera outbreaks, 284 computer games, 280 earthquakes, 282–283 Google Trends, 281 history patterns, 283 iPad, 283 Mayan calendar, 281 Microsoft research, 284–285 oxygen depletion, 281–282 Russian language, 280 storylines, 283 Technion External Studies program, 281 workshops, 285 data scientists hiring data science skills, 288 data stack, 289–290 government-backed research data sets, 290 medical data sets, 290 personal philosophy, 289 problem domain, 288 problem perception, 288 self-building algorithms, 290 smartphones, 290 team building, 288 toolkits and techniques, 288 data structures, 291 genetic hardware, 291 learning resources, 291 problem solving data science, and big data, 286 engineers, 286 lectures, 286 passion, 292 SalesPredict artificial intelligence, 285 buyer persona, 279 cloud-based solution, 275 customer based challenges, 276 customer’s perception, 279 www.it-ebooks.info Index customer’s website data, 276 data distributors, 279 data-specific challenges, 277 decision making, 274 engineering task, 287 engineering team, 275 global data changes, 280 hiring people, 287–288 HR department, tackling, 274 issues, 278 Java and Scala, 286 money spending, 274 MySQL and NoSQL, 286 ontology, 280 performance, 278 pilot customer, 274 problem solving, 287 salesperson, 278 sales process, 277 senior engineer, 279 statistical model, 277 web crawlers, 276 S Shellman, Erin advice to data scientists, 81 advice to undergrads, 70 beauty replenishment project, 77 beauty stylists, 73 company-wide open-door policy, 72 Confluence, 79 cost-benefit conversation, 78 data lab structure, 68 develop ideas, 76 experiment, 73 fashion retail industry, 74 freeing data scientists, 80 HauteLook and Trunk Club, 74 internal customers, 72 kanban board, 75 Lancôme and M•A•C, 78 machine learning class, 81 measuring success, 75 NIH internship, 70 other companies, 73 pair programing, 68 people relationship, 72 predictive modeling, 80 presentation skills, 80 programming and computer science, 70 quantitative and computational skills, 71 recommendations, 71 recommendation strategy, 78 Recommendo, 71, 75 Recommendo API, 77 R programmer, 69 Segmento, 71 SKU turnover, 77 STEM subject, 81 under graduation, 69 Wickham’s, Hadley work, 79 work area, 68 SIGIR conferences, 95 Skype, 221–222, 225, 230–231 Smallwood, Caitlin A/B test, 37 algorithm, 23, 42 Amazon, 34 analytics meeting, 28 analytics pre-Internet, 20 appreciation change, 36 basic data, 42 brainstorming meeting, 27 business priorities, 32 camaraderie, 35 collaborative environment, 42 company strategies, 21 content acquiring model, 29 custom model implementation, 31 data capture, 32 data-centric organizations, 20 data culture, 21 egoless attitude, 40 experience, 39, 42 experimentation, 23, 28, 30 experimentation-heavy culture, 28 Gomez-Uribe, Carlos (colleague), 34 Google search, 27 health care data sharing, 39 HiQ Labs, 34 hunger and insatiable curiosity, 36 Hunt, Neil (manager), 20 incredibly creative and innovative, 43 interesting insights, 35 internet data products, 19 internet entertainment, 24 www.it-ebooks.info 343 344 Index Smallwood, Caitlin (cont.)
Bayesian statistics, business intelligence, business process, cellular automata, Celtic Tiger, cloud computing, collateralized debt obligation, conceptual framework, congestion charging, corporate governance, correlation does not imply causation, crowdsourcing, discrete time, George Gilder, Google Earth, Infrastructure as a Service, Internet Archive, Internet of things, invisible hand, knowledge economy, late capitalism, lifelogging, linked data, Masdar, means of production, Nate Silver, natural language processing, openstreetmap, pattern recognition, platform as a service, recommendation engine, RFID, semantic web, sentiment analysis, slashdot, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, smart grid, smart meter, software as a service, statistical model, supply-chain management, the scientific method, The Signal and the Noise by Nate Silver, transaction costs
Inferential statistics seek to explain, not simply describe, the patterns and relationships that may exist within a dataset, and to test the strength and significance of associations between variables. They include parametric statistics which are employed to assess hypotheses using interval and ratio level data, such as correlation and regression; non-parametric statistics used for testing hypotheses using nominal or ordinal-level data; and probabilistic statistics that determine the probability of a condition occurring, such as Bayesian statistics. The armoury of descriptive and inferential statistics that have traditionally been used to analyse small data are also being applied to big data, though as discussed in Chapter 9 this is not always straightforward because many of these techniques were developed to draw insights from relatively scarce rather than exhaustive data. Nonetheless, the techniques do provide a means of making sense of massive amounts of data.
Little Brother by Cory Doctorow
airport security, Bayesian statistics, Berlin Wall, citizen journalism, Firefox, game design, Golden Gate Park, Haight Ashbury, Internet Archive, Isaac Newton, Jane Jacobs, Jeff Bezos, mail merge, RFID, Sand Hill Road, Silicon Valley, slashdot, Steve Jobs, Steve Wozniak, Thomas Bayes, web of trust, zero day
They hopped from Xbox to Xbox until they found one that was connected to the Internet, then they injected their material as undecipherable, encrypted data. No one could tell which of the Internet's packets were Xnet and which ones were just plain old banking and e-commerce and other encrypted communication. You couldn't find out who was tying the Xnet, let alone who was using the Xnet. But what about Dad's "Bayesian statistics?" I'd played with Bayesian math before. Darryl and I once tried to write our own better spam filter and when you filter spam, you need Bayesian math. Thomas Bayes was an 18th century British mathematician that no one cared about until a couple hundred years after he died, when computer scientists realized that his technique for statistically analyzing mountains of data would be super-useful for the modern world's info-Himalayas.
Statistics hacks by Bruce Frey
Bayesian statistics, Berlin Wall, correlation coefficient, Daniel Kahneman / Amos Tversky, distributed generation, en.wikipedia.org, feminist movement, game design, Hacker Ethic, index card, Milgram experiment, p-value, place-making, RFID, Search for Extraterrestrial Intelligence, SETI@home, Silicon Valley, statistical model, Thomas Bayes
William Skorupski is currently an assistant professor in the School of Education at the University of Kansas, where he teaches courses in psychometrics and statistics. He earned his Bachelor's degree in educational research and psychology from Bucknell University in 2000, and his Doctorate in psychometric methods from the University of Massachusetts, Amherst in 2004. His primary research interest is in the application of mathematical models to psychometric data, including the use of Bayesian statistics for solving practical measurement problems. He also enjoys applying his knowledge of statistics and probability to everyday situations, such as playing poker against the author of this book! Acknowledgments I'd like to thank all the contributors to this book, both those who are listed in the "Contributors" section and those who helped with ideas, reviewed the manuscript, and provided suggestions of sources and resources.
Red-Blooded Risk: The Secret History of Wall Street by Aaron Brown, Eric Kim
activist fund / activist shareholder / activist investor, Albert Einstein, algorithmic trading, Asian financial crisis, Atul Gawande, backtesting, Basel III, Bayesian statistics, beat the dealer, Benoit Mandelbrot, Bernie Madoff, Black Swan, capital asset pricing model, central bank independence, Checklist Manifesto, corporate governance, creative destruction, credit crunch, Credit Default Swap, disintermediation, distributed generation, diversification, diversified portfolio, Edward Thorp, Emanuel Derman, Eugene Fama: efficient market hypothesis, experimental subject, financial innovation, illegal immigration, implied volatility, index fund, Long Term Capital Management, loss aversion, margin call, market clearing, market fundamentalism, market microstructure, money market fund, money: store of value / unit of account / medium of exchange, moral hazard, Myron Scholes, natural language processing, open economy, Pierre-Simon Laplace, pre–internet, quantitative trading / quantitative ﬁnance, random walk, Richard Thaler, risk tolerance, risk-adjusted returns, risk/return, road to serfdom, Robert Shiller, Robert Shiller, shareholder value, Sharpe ratio, special drawing rights, statistical arbitrage, stochastic volatility, The Myth of the Rational Market, Thomas Bayes, too big to fail, transaction costs, value at risk, yield curve
If you accept that your entire earthly life is the appropriate numeraire for decision making, then the rest of Pascal’s case is easy to accept. Just as Archimedes claimed that with a long enough lever he could move the earth, I claim that with a big enough numeraire, I can make any faith-based action seem reasonable. Frequentist statistics suffers from paradoxes because it doesn’t insist everything be stated in moneylike terms, without which there’s no logical connection between frequency and degree of belief. Bayesian statistics suffers from insisting on a single, universal numeraire, which is often not appropriate. One thing we know about money is that it can’t buy everything. One thing we know about people is they have multiple natures, and groups of people are even more complicated. There are many numeraires, more than there are people. Picking the right one is key to getting meaningful statistical results. The only statistical analyses that can be completely certain are ones that are pure mathematical results, and ones that refer to gamelike situations in which all outside considerations are excluded by rule and the numeraire is specified.
Superintelligence: Paths, Dangers, Strategies by Nick Bostrom
agricultural Revolution, AI winter, Albert Einstein, algorithmic trading, anthropic principle, anti-communist, artificial general intelligence, autonomous vehicles, barriers to entry, Bayesian statistics, bioinformatics, brain emulation, cloud computing, combinatorial explosion, computer vision, cosmological constant, dark matter, DARPA: Urban Challenge, data acquisition, delayed gratification, demographic transition, Donald Knuth, Douglas Hofstadter, Drosophila, Elon Musk, en.wikipedia.org, endogenous growth, epigenetics, fear of failure, Flash crash, Flynn Effect, friendly AI, Gödel, Escher, Bach, income inequality, industrial robot, informal economy, information retrieval, interchangeable parts, iterative process, job automation, John Markoff, John von Neumann, knowledge worker, Menlo Park, meta analysis, meta-analysis, mutually assured destruction, Nash equilibrium, Netflix Prize, new economy, Norbert Wiener, NP-complete, nuclear winter, optical character recognition, pattern recognition, performance metric, phenotype, prediction markets, price stability, principal–agent problem, race to the bottom, random walk, Ray Kurzweil, recommendation engine, reversible computing, social graph, speech recognition, Stanislav Petrov, statistical model, stem cell, Stephen Hawking, strong AI, superintelligent machines, supervolcano, technological singularity, technoutopianism, The Coming Technological Singularity, The Nature of the Firm, Thomas Kuhn: the structure of scientific revolutions, transaction costs, Turing machine, Vernor Vinge, Watson beat the top human players on Jeopardy!, World Values Survey, zero-sum game
They also provide important insight into the concept of causality.28 One advantage of relating learning problems from specific domains to the general problem of Bayesian inference is that new algorithms that make Bayesian inference more efficient will then yield immediate improvements across many different areas. Advances in Monte Carlo approximation techniques, for example, are directly applied in computer vision, robotics, and computational genetics. Another advantage is that it lets researchers from different disciplines more easily pool their findings. Graphical models and Bayesian statistics have become a shared focus of research in many fields, including machine learning, statistical physics, bioinformatics, combinatorial optimization, and communication theory.35 A fair amount of the recent progress in machine learning has resulted from incorporating formal results originally derived in other academic fields. (Machine learning applications have also benefitted enormously from faster computers and greater availability of large data sets
Against the Gods: The Remarkable Story of Risk by Peter L. Bernstein
Albert Einstein, Alvin Roth, Andrew Wiles, Antoine Gombaud: Chevalier de Méré, Bayesian statistics, Big bang: deregulation of the City of London, Bretton Woods, buttonwood tree, capital asset pricing model, cognitive dissonance, computerized trading, Daniel Kahneman / Amos Tversky, diversified portfolio, double entry bookkeeping, Edmond Halley, Edward Lloyd's coffeehouse, endowment effect, experimental economics, fear of failure, Fellow of the Royal Society, Fermat's Last Theorem, financial deregulation, financial innovation, full employment, index fund, invention of movable type, Isaac Newton, John Nash: game theory, John von Neumann, Kenneth Arrow, linear programming, loss aversion, Louis Bachelier, mental accounting, moral hazard, Myron Scholes, Nash equilibrium, Paul Samuelson, Philip Mirowski, probability theory / Blaise Pascal / Pierre de Fermat, random walk, Richard Thaler, Robert Shiller, Robert Shiller, spectrum auction, statistical model, The Bell Curve by Richard Herrnstein and Charles Murray, The Wealth of Nations by Adam Smith, Thomas Bayes, trade route, transaction costs, tulip mania, Vanguard fund, zero-sum game
Jahn Maynard Keynes. Vol. 1: Hopes Betrayed. New York: Viking. Slovic, Paul, Baruch Fischoff, and Sarah Lichtenstein, 1990. "Rating the Risks." In Glickman and Gough, 1990, pp. 61-75. Smith, Clifford W., Jr., 1995. "Corporate Risk Management: Theory and Practice." Journal of Derivatives, Summer, pp. 21-30. Smith, M. F. M., 1984. "Present Position and Potential Developments: Some Personal Views of Bayesian Statistics." Journal of the Royal Statistical Association, Vol. 147, Part 3, pp. 245-259. Smithson, Charles W., and Clifford W. Smith, Jr., 1995. Managing Financial Risk: A Guide to Derivative Products, Financial Engineering, and Value Maximization. New York: Irwin.* Sorensen, Eric, 1995. "The Derivative Portfolio Matrix-Combining Market Direction with Market Volatility." Institute for Quantitative Research in Finance, Spring 1995 Seminar.
Algorithms to Live By: The Computer Science of Human Decisions by Brian Christian, Tom Griffiths
4chan, Ada Lovelace, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, Albert Einstein, algorithmic trading, anthropic principle, asset allocation, autonomous vehicles, Bayesian statistics, Berlin Wall, Bill Duvall, bitcoin, Community Supported Agriculture, complexity theory, constrained optimization, cosmological principle, cryptocurrency, Danny Hillis, David Heinemeier Hansson, delayed gratification, dematerialisation, diversification, Donald Knuth, double helix, Elon Musk, fault tolerance, Fellow of the Royal Society, Firefox, first-price auction, Flash crash, Frederick Winslow Taylor, George Akerlof, global supply chain, Google Chrome, Henri Poincaré, information retrieval, Internet Archive, Jeff Bezos, John Nash: game theory, John von Neumann, knapsack problem, Lao Tzu, Leonard Kleinrock, linear programming, martingale, Nash equilibrium, natural language processing, NP-complete, P = NP, packet switching, Pierre-Simon Laplace, prediction markets, race to the bottom, RAND corporation, RFC: Request For Comment, Robert X Cringely, sealed-bid auction, second-price auction, self-driving car, Silicon Valley, Skype, sorting algorithm, spectrum auction, Steve Jobs, stochastic process, Thomas Bayes, Thomas Malthus, traveling salesman, Turing machine, urban planning, Vickrey auction, Vilfredo Pareto, Walter Mischel, Y Combinator, zero-sum game
Laplace was born in Normandy: For more details on Laplace’s life and work, see Gillispie, Pierre-Simon Laplace. distilled down to a single estimate: Laplace’s Law is derived by working through the calculation suggested by Bayes—the tricky part is the sum over all hypotheses, which involves a fun application of integration by parts. You can see a full derivation of Laplace’s Law in Griffiths, Kemp, and Tenenbaum, “Bayesian Models of Cognition.” From the perspective of modern Bayesian statistics, Laplace’s Law is the posterior mean of the binomial rate using a uniform prior. If you try only once and it works out: You may recall that in our discussion of multi-armed bandits and the explore/exploit dilemma in chapter 2, we also touched on estimates of the success rate of a process—a slot machine—based on a set of experiences. The work of Bayes and Laplace undergirds many of the algorithms we discussed in that chapter, including the Gittins index.
The Organized Mind: Thinking Straight in the Age of Information Overload by Daniel J. Levitin
airport security, Albert Einstein, Amazon Mechanical Turk, Anton Chekhov, Bayesian statistics, big-box store, business process, call centre, Claude Shannon: information theory, cloud computing, cognitive bias, complexity theory, computer vision, conceptual framework, correlation does not imply causation, crowdsourcing, cuban missile crisis, Daniel Kahneman / Amos Tversky, delayed gratification, Donald Trump, en.wikipedia.org, epigenetics, Eratosthenes, Exxon Valdez, framing effect, friendly fire, fundamental attribution error, Golden Gate Park, Google Glasses, haute cuisine, impulse control, index card, indoor plumbing, information retrieval, invention of writing, iterative process, jimmy wales, job satisfaction, Kickstarter, life extension, meta analysis, meta-analysis, more computing power than Apollo, Network effects, new economy, Nicholas Carr, optical character recognition, Pareto efficiency, pattern recognition, phenotype, placebo effect, pre–internet, profit motive, randomized controlled trial, Rubik’s Cube, Skype, Snapchat, statistical model, Steve Jobs, supply-chain management, the scientific method, The Wealth of Nations by Adam Smith, The Wisdom of Crowds, theory of mind, Thomas Bayes, Turing test, ultimatum game, zero-sum game
For every 5 people who take the treatment, 1 will be cured (because that person actually has the disease) and .25 will have the side effects. In this case, with two tests, you’re now about 4 times more likely to experience the cure than the side effects, a nice reversal of what we saw before. (If it makes you uncomfortable to talk about .25 of a person, just multiply all the numbers above by 4.) We can take Bayesian statistics a step further. Suppose a newly published study shows that if you are a woman, you’re ten times more likely to get the disease than if you’re a man. You can construct a new table to take this information into account, and to refine the estimate that you actually have the disease. The calculations of probabilities in real life have applications far beyond medical matters. I asked Steve Wynn, who owns five casinos (at his Wynn and Encore hotels in Las Vegas, and the Wynn, Encore, and Palace in Macau), “Doesn’t it hurt, just a little, to see customers walking away with large pots of your money?”
The Golden Passport: Harvard Business School, the Limits of Capitalism, and the Moral Failure of the MBA Elite by Duff McDonald
activist fund / activist shareholder / activist investor, Affordable Care Act / Obamacare, Albert Einstein, barriers to entry, Bayesian statistics, Bernie Madoff, Bob Noyce, Bonfire of the Vanities, business process, butterfly effect, capital asset pricing model, Capital in the Twenty-First Century by Thomas Piketty, Clayton Christensen, cloud computing, collateralized debt obligation, collective bargaining, commoditize, corporate governance, corporate raider, corporate social responsibility, creative destruction, deskilling, discounted cash flows, disintermediation, Donald Trump, family office, financial innovation, Frederick Winslow Taylor, full employment, George Gilder, glass ceiling, Gordon Gekko, hiring and firing, income inequality, invisible hand, Jeff Bezos, job-hopping, John von Neumann, Joseph Schumpeter, Kenneth Arrow, London Whale, Long Term Capital Management, market fundamentalism, Menlo Park, new economy, obamacare, oil shock, pattern recognition, performance metric, Peter Thiel, Plutocrats, plutocrats, profit maximization, profit motive, pushing on a string, Ralph Nader, Ralph Waldo Emerson, RAND corporation, random walk, rent-seeking, Ronald Coase, Ronald Reagan, Sand Hill Road, Saturday Night Live, shareholder value, Silicon Valley, Skype, Steve Jobs, survivorship bias, The Nature of the Firm, the scientific method, Thorstein Veblen, union organizing, urban renewal, Vilfredo Pareto, War on Poverty, William Shockley: the traitorous eight, women in the workforce, Y Combinator
Anyone who has come across a decision tree when contemplating the choices and uncertainties in business owes them a debt. In short, their work opened up just about any business problem to mathematical analysis, without necessarily sacrificing expert opinion in the process. In 1959, Schlaifer published Probability and Statistics for Business Decisions, and in 1961, Raiffa and Schlaifer coauthored Applied Statistical Decision Theory, which “set the direction of Bayesian statistics for the next two decades.”10 But this was geeky stuff, especially for the more “broad-gauged” crowd at HBS. So even if the School was trying as hard as it could to keep up with the GSIAs of the world, it still felt a need to apologize for getting too geeky with Applied Statistical Decision Theory. Calling it “a new type of publication,” Dean Teele explained that “[whereas] most reports . . . published by the Division of Research have as their intended audience informed and forward-looking business executives in general, the new series has been written primarily for specialists. . . .”11 Translation: You may not understand it, but that doesn’t mean you’re not “informed and forward-looking.”
Rationality: From AI to Zombies by Eliezer Yudkowsky
Albert Einstein, Alfred Russel Wallace, anthropic principle, anti-pattern, anti-work, Arthur Eddington, artificial general intelligence, availability heuristic, Bayesian statistics, Berlin Wall, Build a better mousetrap, Cass Sunstein, cellular automata, cognitive bias, cognitive dissonance, correlation does not imply causation, cosmological constant, creative destruction, Daniel Kahneman / Amos Tversky, dematerialisation, discovery of DNA, Douglas Hofstadter, Drosophila, effective altruism, experimental subject, Extropian, friendly AI, fundamental attribution error, Gödel, Escher, Bach, hindsight bias, index card, index fund, Isaac Newton, John Conway, John von Neumann, Long Term Capital Management, Louis Pasteur, mental accounting, meta analysis, meta-analysis, money market fund, Nash equilibrium, Necker cube, NP-complete, P = NP, pattern recognition, Paul Graham, Peter Thiel, Pierre-Simon Laplace, placebo effect, planetary scale, prediction markets, random walk, Ray Kurzweil, reversible computing, Richard Feynman, Richard Feynman, risk tolerance, Rubik’s Cube, Saturday Night Live, Schrödinger's Cat, scientific mainstream, sensible shoes, Silicon Valley, Silicon Valley startup, Singularitarianism, Solar eclipse in 1919, speech recognition, statistical model, Steven Pinker, strong AI, technological singularity, The Bell Curve by Richard Herrnstein and Charles Murray, the map is not the territory, the scientific method, Turing complete, Turing machine, ultimatum game, X Prize, Y Combinator, zero-sum game
Some frequentists criticize Bayesians for treating probabilities as subjective states of belief, rather than as objective frequencies of events. Kruschke and Yudkowsky have replied that frequentism is even more “subjective” than Bayesianism, because frequentism’s probability assignments depend on the intentions of the experimenter.10 Importantly, this philosophical disagreement shouldn’t be conflated with the distinction between Bayesian and frequentist data analysis methods, which can both be useful when employed correctly. Bayesian statistical tools have become cheaper to use since the 1980s, and their informativeness, intuitiveness, and generality have come to be more widely appreciated, resulting in “Bayesian revolutions” in many sciences. However, traditional frequentist methods remain more popular, and in some contexts they are still clearly superior to Bayesian approaches. Kruschke’s Doing Bayesian Data Analysis is a fun and accessible introduction to the topic.11 In light of evidence that training in statistics—and some other fields, such as psychology—improves reasoning skills outside the classroom, statistical literacy is directly relevant to the project of overcoming bias.
I responded—note that this was completely spontaneous—“What on Earth do you mean? You can’t avoid assigning a probability to the mathematician making one statement or another. You’re just assuming the probability is 1, and that’s unjustified.” To which the one replied, “Yes, that’s what the Bayesians say. But frequentists don’t believe that.” And I said, astounded: “How can there possibly be such a thing as non-Bayesian statistics?” That was when I discovered that I was of the type called “Bayesian.” As far as I can tell, I was born that way. My mathematical intuitions were such that everything Bayesians said seemed perfectly straightforward and simple, the obvious way I would do it myself; whereas the things frequentists said sounded like the elaborate, warped, mad blasphemy of dreaming Cthulhu. I didn’t choose to become a Bayesian any more than fishes choose to breathe water.