Estimating the Reproducibility of Psychological Science

6 results back to index


pages: 184 words: 46,395

The Choice Factory: 25 Behavioural Biases That Influence What We Buy by Richard Shotton

active measures, call centre, cashless society, cognitive dissonance, Daniel Kahneman / Amos Tversky, David Brooks, Estimating the Reproducibility of Psychological Science, Firefox, framing effect, fundamental attribution error, Google Chrome, Kickstarter, loss aversion, nudge unit, placebo effect, price anchoring, principal–agent problem, Ralph Waldo Emerson, replication crisis, Richard Feynman, Richard Thaler, Robert Shiller, Robert Shiller, Rory Sutherland, Veblen good, When a measure becomes a target, World Values Survey

, by Evan Davis, John Kay, and Jonathan Star [London Business School Review, Vol. 2, No. 3, pp. 1–23, 1991] Marketers Are from Mars, Consumers Are from New Jersey by Bob Hoffman [2015] Bias 16: The curse of knowledge Made to Stick: Why Some Ideas Survive and Others Die by Chip Heath and Dan Heath [2008] The Wiki Man by Rory Sutherland [2011] Bias 17: Goodhart’s law Long and Short of It: Balancing Short- and Long-Term Marketing Strategies by Les Binet and Peter Field [2012] Management in 10 Words by Terry Leahy [2012] Leading by Alex Ferguson and Michael Moritz [2015] Bias 18: The pratfall effect: Social Animal by Elliot Aronson [1972] The Wasp Factory by Iain Banks [1984] Bias 19: Winner’s curse The Winner’s Curse: Paradoxes and Anomalies of Economic Life by Richard Thaler [1991] Originals: How Non-Conformists Move the World by Adam Grant [2016] ‘Harnessing naturally occurring data to measure the response of spending to income’, by Michael Gelman, Shachar Kariv, Matthew Shapiro, Dan Silverman, Steven Tadelis [Science, Vol. 345, No. 6193, pp. 212–215, 2014] ‘The Psychology of Windfall Gains’, by Hal Arkes, Cynthia Joyner, Mark Pezzo, Jane Gradwohl Nash, Karen Siegel-Jacobs, Eric Stone Eric [Organizational Behaviour and Human Decision Processes, Vol. 59, No. 3, pp. 331–347, 1994] On the Fungibility of Spending and Earnings – Evidence from Rural China and Tanzania by Luc Christiaensen and Lei Pan [2012] Bias 20: The power of the group ‘Humour in Television Advertising: The Effects of Repetition and Social Setting’, by Yong Zhang and George Zinkhan [Advances In Consumer Research, Vol. 18, pp. 813–818, 1991] ‘Feeling More Together: Group Attention Intensifies Emotion’, by Garriy Shteynberg, Jacob Hirsh, Evan Apfelbaum, Jeff Larsen, Adam Galinsky, and Neal Roese [Emotion, Vol. 14, No. 6, pp. 1102–1114, 2014] Bias 21: Veblen goods ‘Commercial Features of Placebo and Therapeutic Efficacy’, by Rebecca Waber, Baba Shiv, Ziv Carmon; Dan Ariely [Journal of the American Medical Association, Vol. 299, No.9, pp. 1016–1017, 2008] Bias 22: The replicability crisis ‘Why Susie Sells Seashells by the Seashore: Implicit Egotism and Major Life Decisions’, by Brett Pelham, Matthew Mirenberg, and John Jones [Journal of Personality and Social Psychology, Vol. 82, No. 4, pp. 469–487, 2002] ‘Rich the banker? What’s not in a Name’, by Tim Harford [2016], www.timharford.com/2016/11/rich-the-banker-whats-not-in-a-name ‘Estimating the reproducibility of psychological science’, by Brian Nosek et al. [Science, Vol. 349, No. 6251, 2015] ‘Comment on “Estimating the reproducibility of psychological science”’, by Daniel Gilbert, Gary King, Stephen Pettigrew and Timothy Wilson [Science, Vol. 351, Issue 6277, p. 1037, 2016] ‘Meta-assessment of bias in science’, by Daniele Fanelli, Rodrigo Costats, and John Ioannidis [Proceedings of the National Academy of Sciences, Vol. 114, No. 14, pp. 3714–3719, 2017] ‘Evaluating replicability of laboratory experiments in economics’, by Colin F.


Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth by Stuart Ritchie

Albert Einstein, anesthesia awareness, Bayesian statistics, Carmen Reinhart, Cass Sunstein, citation needed, Climatic Research Unit, cognitive dissonance, complexity theory, coronavirus, correlation does not imply causation, Covid-19, COVID-19, crowdsourcing, deindustrialization, Donald Trump, double helix, en.wikipedia.org, epigenetics, Estimating the Reproducibility of Psychological Science, Growth in a Time of Debt, Kenneth Rogoff, l'esprit de l'escalier, meta-analysis, microbiome, Milgram experiment, mouse model, New Journalism, p-value, phenotype, placebo effect, profit motive, publication bias, publish or perish, race to the bottom, randomized controlled trial, recommendation engine, rent-seeking, replication crisis, Richard Thaler, risk tolerance, Ronald Reagan, Scientific racism, selection bias, Silicon Valley, Silicon Valley startup, Stanford prison experiment, statistical model, stem cell, Steven Pinker, Thomas Bayes, twin studies, University of East Anglia

See also Le Texier’s reply to a more recent (at the time of writing unpublished) version: Thibault Le Texier, ‘The SPE Remains Debunked: A Reply to Zimbardo and Haney (2020)’, Preprint, PsyArXiv (24 Jan. 2020); https://doi.org/10.31234/osf.io/9a2er 26.  Open Science Collaboration, ‘Estimating the Reproducibility of Psychological Science’, Science 349, no. 6251 (28 Aug. 2015): aac4716; https://doi.org/10.1126/science.aac4716 27.  77 per cent: Colin F. Camerer et al., ‘Evaluating the Replicability of Social Science Experiments in Nature and Science between 2010 and 2015’, Nature Human Behaviour 2, no. 9 (Sept. 2018): pp. 637–44; https://doi.org/10.1038/s41562-018-0399-z 28.  

I’ve been stressing the importance of robust results, but in making the case that there’s a replication crisis, I’m relying on multi-study replication attempts that weren’t representative samples of all the scientific literature. The conclusion of ‘only about half of published results replicate’ might not generalise to all science. This was a point made in a critique of one of the replication survey studies: D. T. Gilbert et al., ‘Comment on “Estimating the Reproducibility of Psychological Science”’, Science 351, no. 6277 (4 Mar. 2016): p. 1037; https://doi.org/10.1126/science.aad7243. Whereas I disagree with many of the arguments made in this rejoinder (for some reasons to be sceptical of it, see Daniël Lakens, ‘The Statistical Conclusions in Gilbert et al (2016) Are Completely Invalid’, The 20% Statistician, 6 March 2016; https://daniellakens.blogspot.com/2016/03/the-statistical-conclusions-in-gilbert.html), the criticism about representativeness was fair.


pages: 288 words: 81,253

Thinking in Bets by Annie Duke

banking crisis, Bernie Madoff, Cass Sunstein, cognitive bias, cognitive dissonance, Daniel Kahneman / Amos Tversky, delayed gratification, disinformation, Donald Trump, en.wikipedia.org, endowment effect, Estimating the Reproducibility of Psychological Science, Filter Bubble, hindsight bias, Jean Tirole, John Nash: game theory, John von Neumann, loss aversion, market design, mutually assured destruction, Nate Silver, p-value, phenotype, prediction markets, Richard Feynman, ride hailing / ride sharing, Stanford marshmallow experiment, Stephen Hawking, Steven Pinker, the scientific method, The Signal and the Noise by Nate Silver, urban planning, Walter Mischel, Yogi Berra, zero-sum game

New York: Current, 2014. Oettingen, Gabriele, and Peter Gollwitzer. “Strategies of Setting and Implementing Goals.” In Social Psychological Foundations of Clinical Psychology, edited by James Maddox and June Price Tangney, 114–35. New York: Guilford Press, 2010. Open Science Collaboration. “Estimating the Reproducibility of Psychological Science.” Science 349, no. 6251 (August 28, 2015): 943 and aac4716-1–8. Oswald, Dan. “Learn Important Lessons from Lombardi’s Eight-Hour Session.” HR Hero (blog), March 10, 2014. http://blogs.hrhero.com/oswaldletters/2014/03/10/learn-important-lessons-from-lombardis-eight-hour-session.


Know Thyself by Stephen M Fleming

Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, autonomous vehicles, availability heuristic, backpropagation, citation needed, computer vision, Douglas Hofstadter, Elon Musk, Estimating the Reproducibility of Psychological Science, global pandemic, index card, Jeff Bezos, l'esprit de l'escalier, Lao Tzu, lifelogging, longitudinal study, meta-analysis, mutually assured destruction, Network effects, patient HM, Pierre-Simon Laplace, prediction markets, QWERTY keyboard, recommendation engine, replication crisis, self-driving car, side project, Skype, Stanislav Petrov, statistical model, theory of mind, Thomas Bayes

“Birds Have Primate-Like Numbers of Neurons in the Forebrain.” Proceedings of the National Academy of Sciences 113, no. 26 (2016): 7255–7260. Onishi, Kristine H., and Renée Baillargeon. “Do 15-Month-Old Infants Understand False Beliefs?” Science 308, no. 5719 (2005): 255–258. Open Science Collaboration. “Estimating the Reproducibility of Psychological Science.” Science 349, no. 6251 (2015). Ortoleva, Pietro, and Erik Snowberg. “Overconfidence in Political Behavior.” American Economic Review 105, no. 2 (2015): 504–535. Palser, E. R., A. Fotopoulou, and J. M. Kilner. “Altering Movement Parameters Disrupts Metacognitive Accuracy.”


pages: 340 words: 94,464

Randomistas: How Radical Researchers Changed Our World by Andrew Leigh

Albert Einstein, Amazon Mechanical Turk, Anton Chekhov, Atul Gawande, basic income, Black Swan, correlation does not imply causation, crowdsourcing, David Brooks, Donald Trump, ending welfare as we know it, Estimating the Reproducibility of Psychological Science, experimental economics, Flynn Effect, germ theory of disease, Ignaz Semmelweis: hand washing, Indoor air pollution, Isaac Newton, Kickstarter, longitudinal study, loss aversion, Lyft, Marshall McLuhan, meta-analysis, microcredit, Netflix Prize, nudge unit, offshore financial centre, p-value, placebo effect, price mechanism, publication bias, RAND corporation, randomized controlled trial, recommendation engine, Richard Feynman, ride hailing / ride sharing, Robert Metcalfe, Ronald Reagan, statistical model, Steven Pinker, uber lyft, universal basic income, War on Poverty

Lalande, ‘A peculiar prevalence of p values just below .05’, Quarterly Journal of Experimental Psychology, vol. 65, no. 11, 2012, pp. 2271–9; Kewei Hou, Chen Xue & Lu Zhang, ‘Replicating anomalies’, NBER Working Paper 23394, Cambridge, MA: National Bureau of Economic Research, 2017. 46Alexander A. Aarts, Joanna E. Anderson, Christopher J. Anderson, et al., ‘Estimating the reproducibility of psychological science’, Science, vol. 349, no. 6251, 2015. 47This represented two out of eighteen papers: John P.A. Ioannidis, David B. Allison, Catherine A. Ball, et al., ‘Repeatability of published microarray gene expression analyses’, Nature Genetics, vol. 41, no. 2, 2009, pp. 149–55. 48This represented six out of fifty-three papers: C.


pages: 442 words: 94,734

The Art of Statistics: Learning From Data by David Spiegelhalter

algorithmic bias, Antoine Gombaud: Chevalier de Méré, Bayesian statistics, Carmen Reinhart, complexity theory, computer vision, correlation coefficient, correlation does not imply causation, dark matter, Edmond Halley, Estimating the Reproducibility of Psychological Science, Hans Rosling, Kenneth Rogoff, meta-analysis, Nate Silver, Netflix Prize, p-value, placebo effect, probability theory / Blaise Pascal / Pierre de Fermat, publication bias, randomized controlled trial, recommendation engine, replication crisis, self-driving car, speech recognition, statistical model, The Design of Experiments, The Signal and the Noise by Nate Silver, The Wisdom of Crowds, Thomas Bayes, Thomas Malthus

Raftery, ‘Bayes Factors’, Journal of the American Statistical Association 90 (1995), 773–95. 9. J. Cornfield, ‘Sequential Trials, Sequential Analysis and the Likelihood Principle’, American Statistician 20 (1966), 18–23. CHAPTER 12: HOW THINGS GO WRONG 1. Open Science Collaboration, ‘Estimating the Reproducibility of Psychological Science’, Science 349:6251 (28 August 2015), aac4716. 2. A. Gelman and H. Stern, ‘The Difference Between “Significant” and “Not Significant” Is Not Itself Statistically Significant’, American Statistician 60:4 (November 2006), 328–31. 3. Ronald Fisher, Presidential Address to the first Indian Statistical Congress, 1938, Sankhyā 4(1938), 14–17. 4.


pages: 404 words: 92,713

The Art of Statistics: How to Learn From Data by David Spiegelhalter

algorithmic bias, Antoine Gombaud: Chevalier de Méré, Bayesian statistics, Carmen Reinhart, complexity theory, computer vision, correlation coefficient, correlation does not imply causation, dark matter, Edmond Halley, Estimating the Reproducibility of Psychological Science, Hans Rosling, Kenneth Rogoff, meta-analysis, Nate Silver, Netflix Prize, p-value, placebo effect, probability theory / Blaise Pascal / Pierre de Fermat, publication bias, randomized controlled trial, recommendation engine, replication crisis, self-driving car, speech recognition, statistical model, The Design of Experiments, The Signal and the Noise by Nate Silver, The Wisdom of Crowds, Thomas Bayes, Thomas Malthus

Raftery, ‘Bayes Factors’, Journal of the American Statistical Association 90 (1995), 773–95. 9. J. Cornfield, ‘Sequential Trials, Sequential Analysis and the Likelihood Principle’, American Statistician 20 (1966), 18–23. CHAPTER 12: HOW THINGS GO WRONG 1. Open Science Collaboration, ‘Estimating the Reproducibility of Psychological Science’, Science 349:6251 (28 August 2015), aac4716. 2. A. Gelman and H. Stern, ‘The Difference Between “Significant” and “Not Significant” Is Not Itself Statistically Significant’, American Statistician 60:4 (November 2006), 328–31. 3. Ronald Fisher, Presidential Address to the first Indian Statistical Congress, 1938, Sankhyā 4(1938), 14–17. 4.


Calling Bullshit: The Art of Scepticism in a Data-Driven World by Jevin D. West, Carl T. Bergstrom

airport security, algorithmic bias, Amazon Mechanical Turk, Andrew Wiles, bitcoin, cloud computing, computer vision, correlation coefficient, correlation does not imply causation, crowdsourcing, cryptocurrency, delayed gratification, disinformation, Dmitri Mendeleev, Donald Trump, Elon Musk, epigenetics, Estimating the Reproducibility of Psychological Science, experimental economics, invention of the printing press, John Markoff, longitudinal study, Lyft, meta-analysis, new economy, p-value, Pluto: dwarf planet, publication bias, RAND corporation, randomized controlled trial, replication crisis, ride hailing / ride sharing, Ronald Reagan, selection bias, self-driving car, Silicon Valley, Silicon Valley startup, social graph, Socratic dialogue, Stanford marshmallow experiment, statistical model, stem cell, superintelligent machines, the scientific method, theory of mind, Tim Cook: Apple, twin studies, Uber and Lyft, Uber for X, uber lyft, When a measure becomes a target

Press release. National Aeronautics and Space Administration. January 31, 2018. https://www.nasa.gov/​feature/​nasa-twins-study-confirms-preliminary-findings. NORC General Social Survey. 2017. Data compiled by the Pew Research Center. Open Science Collaboration. “Estimating the Reproducibility of Psychological Science.” Science 349 (2015): aac4716. Pauling, L., and R. B. Corey. “A Proposed Structure for the Nucleic Acids.” Proceedings of the National Academy of Sciences 39 (1953): 84–97. Pauling, Linus. Vitamin C and the Common Cold. 1st edition. San Francisco: W. H. Freeman, 1970.


pages: 428 words: 103,544

The Data Detective: Ten Easy Rules to Make Sense of Statistics by Tim Harford

access to a mobile phone, Ada Lovelace, affirmative action, algorithmic bias, Automated Insights, banking crisis, basic income, Black Swan, Bretton Woods, British Empire, business cycle, Capital in the Twenty-First Century by Thomas Piketty, Cass Sunstein, clean water, collapse of Lehman Brothers, coronavirus, correlation does not imply causation, Covid-19, COVID-19, cuban missile crisis, Daniel Kahneman / Amos Tversky, David Attenborough, Diane Coyle, disinformation, Donald Trump, Estimating the Reproducibility of Psychological Science, experimental subject, financial innovation, Florence Nightingale: pie chart, Gini coefficient, Hans Rosling, income inequality, Isaac Newton, job automation, Kickstarter, life extension, meta-analysis, microcredit, Milgram experiment, moral panic, Netflix Prize, Paul Samuelson, publication bias, publish or perish, random walk, randomized controlled trial, recommendation engine, replication crisis, Richard Feynman, Richard Thaler, rolodex, Ronald Reagan, selection bias, sentiment analysis, Silicon Valley, sorting algorithm, statistical model, stem cell, Stephen Hawking, Steve Bannon, Steven Pinker, survivorship bias, universal basic income, When a measure becomes a target

An alternative metric is to ask how many of the replication studies produced results that passed the standard (but rather problematic) hurdle of “statistical significance.” Only thirty-six did; ninety-seven of the original studies had cleared that hurdle. Open Science Collaboration, “Estimating the Reproducibility of Psychological Science,” Science 28, no. 6251 (August 2015), 349, DOI: 10.1126/science.aac4716. 12. Brief film on YouTube: “Derren Brown—Ten Heads in a Row,” ThinkSceptically, April 8, 2012, https://www.youtube.com/watch?v=n1SJ-Tn3bcQ. 13. Planet Money, episode 677. 14. F. J. Anscombe, “Fixed-Sample-Size Analysis of Sequential Observations,” Biometrics 10, no. 1 (1954), 89–100, JSTOR, www.jstor.org/stable/3001665; and Andrew Gelman’s blog post “Statistical Inference, Modeling and Social Science,” May 2, 2018, https://statmodeling.stat.columbia.edu/2018/05/02/continuously-increased-number-animals-statistical-significance-reached-support-conclusions-think-not-bad-actually/. 15.


pages: 434 words: 117,327

Can It Happen Here?: Authoritarianism in America by Cass R. Sunstein

active measures, affirmative action, Affordable Care Act / Obamacare, airline deregulation, anti-communist, anti-globalists, availability heuristic, business cycle, Cass Sunstein, David Brooks, disinformation, Donald Trump, Edward Snowden, Estimating the Reproducibility of Psychological Science, failed state, Filter Bubble, Francis Fukuyama: the end of history, Garrett Hardin, ghettoisation, illegal immigration, immigration reform, Isaac Newton, job automation, Joseph Schumpeter, Long Term Capital Management, microaggression, Nate Silver, Network effects, New Journalism, night-watchman state, obamacare, Potemkin village, random walk, Richard Thaler, road to serfdom, Ronald Reagan, Steve Bannon, the scientific method, Tragedy of the Commons, War on Poverty, WikiLeaks, World Values Survey

Princeton, NJ: Princeton University Press, 2017. Camerer, Colin F., and Eric J. Johnson. “The Process-Performance Paradox in Expert Judgment: How Can Experts Know So Much and Predict So Badly?” Research on Judgment and Decision Making: Currents, Connections, and Controversies 342 (1997). Collaboration, Open Science. “Estimating the Reproducibility of Psychological Science.” Science 349, no. 6251 (2015): 10.1126/science.aac4716. DiPrete, Thomas A., and Gregory M. Eirich. “Cumulative Advantage as a Mechanism for Inequality: A Review of Theoretical and Empirical Developments.” Annual Review of Sociology 32, no. 1 (2006): 271–97. Dunning, Thad.


pages: 533 words: 125,495

Rationality: What It Is, Why It Seems Scarce, Why It Matters by Steven Pinker

affirmative action, Albert Einstein, autonomous vehicles, availability heuristic, Ayatollah Khomeini, backpropagation, basic income, butterfly effect, Cass Sunstein, choice architecture, clean water, coronavirus, correlation coefficient, correlation does not imply causation, Covid-19, COVID-19, crowdsourcing, cuban missile crisis, Daniel Kahneman / Amos Tversky, David Attenborough, delayed gratification, disinformation, Donald Trump, effective altruism, en.wikipedia.org, Erdős number, Estimating the Reproducibility of Psychological Science, feminist movement, framing effect, George Akerlof, germ theory of disease, high batting average, index card, Jeff Bezos, job automation, John Nash: game theory, John von Neumann, libertarian paternalism, longitudinal study, loss aversion, Mahatma Gandhi, meta-analysis, microaggression, Monty Hall problem, Nash equilibrium, New Journalism, Paul Erdős, Paul Samuelson, Peter Singer: altruism, Pierre-Simon Laplace, placebo effect, QAnon, QWERTY keyboard, Ralph Waldo Emerson, randomized controlled trial, replication crisis, Richard Thaler, scientific worldview, selection bias, Stanford marshmallow experiment, Steve Bannon, Steven Pinker, sunk-cost fallacy, the scientific method, Thomas Bayes, Tragedy of the Commons, twin studies, universal basic income, Upton Sinclair, urban planning, Walter Mischel, yellow journalism, zero-sum game

Journal of Elections, Public Opinion and Parties, 29, 222–44. https://doi.org/10.1080/17457289.2018.1465061. O’Keefe, S. M. 2020. One in three Americans would not get COVID-19 vaccine. Gallup, Aug. 7. https://news.gallup.com/poll/317018/one-three-americans-not-covid-vaccine.aspx. Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science, 349. https://doi.org/10.1126/science.aac4716. Paresky, P., Haidt, J., Strossen, N., & Pinker, S. 2020. The New York Times surrendered to an outrage mob. Journalism will suffer for it. Politico, May 14. https://www.politico.com/news/magazine/2020/05/14/bret-stephens-new-york-times-outrage-backlash-256494.


pages: 543 words: 153,550

Model Thinker: What You Need to Know to Make Data Work for You by Scott E. Page

"Robert Solow", Airbnb, Albert Einstein, Alfred Russel Wallace, algorithmic trading, Alvin Roth, assortative mating, Bernie Madoff, bitcoin, Black Swan, blockchain, business cycle, Capital in the Twenty-First Century by Thomas Piketty, Checklist Manifesto, computer age, corporate governance, correlation does not imply causation, cuban missile crisis, deliberate practice, discrete time, distributed ledger, en.wikipedia.org, Estimating the Reproducibility of Psychological Science, Everything should be made as simple as possible, experimental economics, first-price auction, Flash crash, Geoffrey West, Santa Fe Institute, germ theory of disease, Gini coefficient, High speed trading, impulse control, income inequality, Isaac Newton, John von Neumann, Kenneth Rogoff, knowledge economy, knowledge worker, Long Term Capital Management, loss aversion, low skilled workers, Mark Zuckerberg, market design, meta-analysis, money market fund, Nash equilibrium, natural language processing, Network effects, p-value, Pareto efficiency, pattern recognition, Paul Erdős, Paul Samuelson, phenotype, pre–internet, prisoner's dilemma, race to the bottom, random walk, randomized controlled trial, Richard Feynman, Richard Thaler, school choice, sealed-bid auction, second-price auction, selection bias, six sigma, social graph, spectrum auction, statistical model, Stephen Hawking, Supply of New York City Cabdrivers, The Bell Curve by Richard Herrnstein and Charles Murray, The Great Moderation, The Rise and Fall of American Growth, the rule of 72, the scientific method, The Spirit Level, the strength of weak ties, The Wisdom of Crowds, Thomas Malthus, Thorstein Veblen, Tragedy of the Commons, urban sprawl, value at risk, web application, winner-take-all economy, zero-sum game

The Logic of Collective Action: Public Goods and the Theory of Groups. Cambridge, MA: Harvard University Press. O’Neil, Cathy 2016. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. New York, NY: Crown. Open Science Collaboration. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349: 6251. Organization for Economic Co-operation and Development. 1996. The Knowledge Based Economy. Paris: OECD. Ormerod, Paul. 2012. Positive Linking: How Networks Can Revolutionise the World. London: Faber and Faber. Ostrom, Elinor. 2004. Understanding Institutional Diversity.