friendly AI

23 results back to index


pages: 294 words: 81,292

Our Final Invention: Artificial Intelligence and the End of the Human Era by James Barrat

AI winter, AltaVista, Amazon Web Services, artificial general intelligence, Asilomar, Automated Insights, Bayesian statistics, Bernie Madoff, Bill Joy: nanobots, brain emulation, cellular automata, Chuck Templeton: OpenTable:, cloud computing, cognitive bias, commoditize, computer vision, cuban missile crisis, Daniel Kahneman / Amos Tversky, Danny Hillis, data acquisition, don't be evil, drone strike, Extropian, finite state, Flash crash, friendly AI, friendly fire, Google Glasses, Google X / Alphabet X, Isaac Newton, Jaron Lanier, John Markoff, John von Neumann, Kevin Kelly, Law of Accelerating Returns, life extension, Loebner Prize, lone genius, mutually assured destruction, natural language processing, Nicholas Carr, optical character recognition, PageRank, pattern recognition, Peter Thiel, prisoner's dilemma, Ray Kurzweil, Rodney Brooks, Search for Extraterrestrial Intelligence, self-driving car, semantic web, Silicon Valley, Singularitarianism, Skype, smart grid, speech recognition, statistical model, stealth mode startup, stem cell, Stephen Hawking, Steve Jobs, Steve Wozniak, strong AI, Stuxnet, superintelligent machines, technological singularity, The Coming Technological Singularity, Thomas Bayes, traveling salesman, Turing machine, Turing test, Vernor Vinge, Watson beat the top human players on Jeopardy!, zero day

One path leading to global catastrophe—to someone pressing the button with a mistaken idea of what the button does—is that Artificial Intelligence comes about through a similar accretion of working algorithms, with the researchers having no deep understanding of how the combined system works. [italics mine] Not knowing how to build a Friendly AI is not deadly, of itself.… It’s the mistaken belief that an AI will be friendly which implies an obvious path to global catastrophe. Assuming that human-level AIs (AGIs) will be friendly is wrong for a lot of reasons. The assumption becomes even more dangerous after the AGI’s intelligence rockets past ours, and it becomes ASI—artificial superintelligence. So how do you create friendly AI? Or could you impose friendliness on advanced AIs after they’re already built? Yudkowsky has written a book-length online treatise about these questions entitled Creating Friendly AI: The Analysis and Design of Benevolent Goal Architectures. Friendly AI is a subject so dense yet important it exasperates its chief proponent himself, who says about it, “it only takes one error for a chain of reasoning to end up in Outer Mongolia.”

And not only what we would want, but what we would want if we “knew more, thought faster, and were more the people we thought we were.” CEV would be an oracular feature of friendly AI. It would have to derive from us our values as if we were better versions of ourselves, and be democratic about it so that humankind is not tyrannized by the norms of a few. Does this sound a little starry-eyed? Well, there are good reasons for that. First, I’m giving you a highly summarized account of Friendly AI and CEV, concepts you can read volumes about online. And second, the whole topic of Friendly AI is incomplete and optimistic. It’s unclear whether or not Friendly AI can be expressed in a formal, mathematical sense, and so there may be no way to build it or to integrate it into promising AI architectures. But if we could, what would the future look like?

In fact, we thrive. God bless you, Friendly AI! * * * Now that most (but not all) AI makers and theorists have recognized Asimov’s Three Laws of Robotics for what they were meant to be—tools for drama, not survival—Friendly AI may be the best concept humans have come up with for planning their survival. But besides not being ready yet, it’s got other big problems. First, there are too many players in the AGI sweepstakes. Too many organizations in too many countries are working on AGI and AGI-related technologies for them all to agree to mothball their projects until Friendly AI is created, or to include in their code a formal friendliness module, if one could be made. And few are even taking part in the public dialogue about the necessity for Friendly AI. Some of the AGI contestants include: IBM (with several AGI-related projects), Numenta, AGIRI, Vicarious, Carnegie Mellon’s NELL and ACT-R, SNERG, LIDA, CYC, and Google.


pages: 144 words: 43,356

Surviving AI: The Promise and Peril of Artificial Intelligence by Calum Chace

"Robert Solow", 3D printing, Ada Lovelace, AI winter, Airbnb, artificial general intelligence, augmented reality, barriers to entry, basic income, bitcoin, blockchain, brain emulation, Buckminster Fuller, cloud computing, computer age, computer vision, correlation does not imply causation, credit crunch, cryptocurrency, cuban missile crisis, dematerialisation, discovery of the americas, disintermediation, don't be evil, Elon Musk, en.wikipedia.org, epigenetics, Erik Brynjolfsson, everywhere but in the productivity statistics, Flash crash, friendly AI, Google Glasses, hedonic treadmill, industrial robot, Internet of things, invention of agriculture, job automation, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John von Neumann, Kevin Kelly, life extension, low skilled workers, Mahatma Gandhi, means of production, mutually assured destruction, Nicholas Carr, pattern recognition, peer-to-peer, peer-to-peer model, Peter Thiel, Ray Kurzweil, Rodney Brooks, Second Machine Age, self-driving car, Silicon Valley, Silicon Valley ideology, Skype, South Sea Bubble, speech recognition, Stanislav Petrov, Stephen Hawking, Steve Jobs, strong AI, technological singularity, The Future of Employment, theory of mind, Turing machine, Turing test, universal basic income, Vernor Vinge, wage slave, Wall-E, zero-sum game

It also boasts some Hollywood glamour, with Alan Alda and Morgan Freeman on the advisory board, along with technology entrepreneur Elon Musk, who has donated $10m of his personal money to the institute. 8.6 – Conclusion We do not yet have a foolproof way to ensure that the first AGI is a Friendly AI. In fact we don’t yet know how best to approach the problem. But we have only just begun, and the resources allocated to the problem are small: Nick Bostrom estimated in 2014 that only six people in the world are working full-time on the Friendly AI problem, whereas many thousands of people work full-time on projects that could well contribute to the creation of the first AGI. (50) He argued that this equation needed urgent re-balancing. A very experienced AI researcher told me in spring 2015 that Bostrom’s estimate was significantly too low, and that many more AI researchers spend much of their time thinking about Friendly AI as part of their everyday jobs. Even if that is correct, I suspect Bostrom is still right about the imbalance, and the truth will emerge if we have the kind of debate I argue for in the next, concluding chapter.

The upshot is that there are seven billion of us and we are shaping much of the planet according to our will (regardless of whether that is a good idea or not), whereas there are fewer than 300,000 chimpanzees, and whether they become extinct or not depends entirely on the actions of humans. A superintelligence could become not just twice as smart as humans, but smarter by many orders of magnitude. It is hard to escape the conclusion that our future will depend on its decisions and its actions. Would that be a good thing or a bad thing? In other words, would a superintelligence be a “Friendly AI”? (Friendly AI, or FAI, denotes an AGI that is beneficial for humans rather than one that seeks social approbation and company. It also refers to the project to make sure that AGI is beneficial.) 7.2 – Optimistic scenarios: to immortality and beyond The ultimate problem solver Imagine having a big sister endowed with superhuman wisdom, insight and ingenuity. Her cleverness enables her to solve all our personal, inter-personal, social, political and economic problems.

What is clear is that a negative outcome cannot be ruled out. So if we take seriously the idea that a superintelligence may appear on the Earth in the foreseeable future, we should certainly be thinking about how to ensure that the event is a positive one for ourselves and our descendants. We should be taking steps to ensure that the first AGI is a friendly AI. PART FOUR: FAI Friendly Artificial Intelligence CHAPTER 8 CAN WE ENSURE THAT SUPERINTELLIGENCE IS SAFE? As we saw in the last chapter, Friendly AI (FAI) is the project of ensuring that the world’s superintelligences are safe and useful for humans. The central argument of this book is that we need to address this challenge successfully. It may well turn out to be the most important challenge facing this generation and the next. Indeed it may turn out to be the most important challenge humanity ever faces. 8.1 – Stop before you start Faced with the unpalatable possibilities explored in the last chapter, perhaps we should try to avoid the problem by preventing the arrival of AGI in the first place.


pages: 271 words: 52,814

Blockchain: Blueprint for a New Economy by Melanie Swan

23andMe, Airbnb, altcoin, Amazon Web Services, asset allocation, banking crisis, basic income, bioinformatics, bitcoin, blockchain, capital controls, cellular automata, central bank independence, clean water, cloud computing, collaborative editing, Conway's Game of Life, crowdsourcing, cryptocurrency, disintermediation, Edward Snowden, en.wikipedia.org, Ethereum, ethereum blockchain, fault tolerance, fiat currency, financial innovation, Firefox, friendly AI, Hernando de Soto, intangible asset, Internet Archive, Internet of things, Khan Academy, Kickstarter, lifelogging, litecoin, Lyft, M-Pesa, microbiome, Network effects, new economy, peer-to-peer, peer-to-peer lending, peer-to-peer model, personalized medicine, post scarcity, prediction markets, QR code, ride hailing / ride sharing, Satoshi Nakamoto, Search for Extraterrestrial Intelligence, SETI@home, sharing economy, Skype, smart cities, smart contracts, smart grid, software as a service, technological singularity, Turing complete, uber lyft, unbanked and underbanked, underbanked, web application, WikiLeaks

To achieve their goals, digital intelligences will want to conduct certain transactions over the network, many of which could be managed by blockchain and other consensus mechanisms. Only Friendly AIs Are Able to Get Their Transactions Executed One of the unforeseen benefits of consensus models might be that they could possibly enforce friendly AI, which is to say cooperative, moral players within a society.196 In decentralized trust networks, an agentÕs reputation (where agents themselves remain pseudonymous) could be an important factor in whether his transactions will be executed, such that malicious players would not be able to get their transactions executed or recognized on the network. Any important transaction regarding resource access and use might require assent by consensus models. Thus, the way that friendly AI could be enforced is that even bad agents want to participate in the system to access resources and to do so, they need to look like good agents.

Advanced Concepts Terminology and Concepts Currency, Token, Tokenizing Communitycoin: Hayek’s Private Currencies Vie for Attention Campuscoin Coin Drops as a Strategy for Public Adoption Currency: New Meanings Currency Multiplicity: Monetary and Nonmonetary Currencies Demurrage Currencies: Potentially Incitory and Redistributable Extensibility of Demurrage Concept and Features 6. Limitations Technical Challenges Business Model Challenges Scandals and Public Perception Government Regulation Privacy Challenges for Personal Records Overall: Decentralization Trends Likely to Persist 7. Conclusion The Blockchain Is an Information Technology Blockchain AI: Consensus as the Mechanism to Foster “Friendly” AI Large Possibility Space for Intelligence Only Friendly AIs Are Able to Get Their Transactions Executed Smart Contract Advocates on Behalf of Digital Intelligence Blockchain Consensus Increases the Information Resolution of the Universe A. Cryptocurrency Basics Public/Private-Key Cryptography 101 B. Ledra Capital Mega Master Blockchain List Endnotes and References Index Blockchain Blueprint for a New Economy Melanie Swan Blockchain by Melanie Swan Copyright © 2015 Melanie Swan.

The blockchain is a consensus model at scale, and possibly the mechanism we have been waiting for that could help to usher in an era of friendly machine intelligence. Blockchain AI: Consensus as the Mechanism to Foster “Friendly” AI One forward-looking but important concern in the general future of technology is different ways in which artificial intelligence (AI) might arise and how to sponsor it such that it engenders a “friendly” or benevolent relationship with humans. There is the notion of a technological singularity, a moment when machine intelligence might supersede human intelligence. However, those in the field have not set forth any sort of robust plan for how to effect friendly AI, and many remain skeptical of this possibility.195 It is possible that blockchain technology could be a useful connector of humans and machines in a world of increasingly autonomous machine activity through Dapps, DAOs, and DACs that might eventually give way to AI.


Global Catastrophic Risks by Nick Bostrom, Milan M. Cirkovic

affirmative action, agricultural Revolution, Albert Einstein, American Society of Civil Engineers: Report Card, anthropic principle, artificial general intelligence, Asilomar, availability heuristic, Bill Joy: nanobots, Black Swan, carbon-based life, cognitive bias, complexity theory, computer age, coronavirus, corporate governance, cosmic microwave background, cosmological constant, cosmological principle, cuban missile crisis, dark matter, death of newspapers, demographic transition, Deng Xiaoping, distributed generation, Doomsday Clock, Drosophila, endogenous growth, Ernest Rutherford, failed state, feminist movement, framing effect, friendly AI, Georg Cantor, global pandemic, global village, Gödel, Escher, Bach, hindsight bias, Intergovernmental Panel on Climate Change (IPCC), invention of agriculture, Kevin Kelly, Kuiper Belt, Law of Accelerating Returns, life extension, means of production, meta analysis, meta-analysis, Mikhail Gorbachev, millennium bug, mutually assured destruction, nuclear winter, P = NP, peak oil, phenotype, planetary scale, Ponzi scheme, prediction markets, RAND corporation, Ray Kurzweil, reversible computing, Richard Feynman, Ronald Reagan, scientific worldview, Singularitarianism, social intelligence, South China Sea, strong AI, superintelligent machines, supervolcano, technological singularity, technoutopianism, The Coming Technological Singularity, Tunguska event, twin studies, uranium enrichment, Vernor Vinge, War on Poverty, Westphalian system, Y2K

Legislation could (for example) require researchers to publicly report their Friendliness strategies, or penalize researchers whose Ais cause damage; and while this legislation will not prevent all mistakes, it may suffice that a majority of Ais are built Friendly. Artificial Intelligence in global risk 335 We can also imagine a scenario that implies a n easy local strategy: • The first AI cannot by itself do catastrophic damage. • If even a single Friendly AI exists, that AI plus human institutions can fend off any number of unFriendly A Is. The easy scenario would hold if, for example, human institutions can reliably distinguish Friendly Ais from unFriendly ones, and give revocable power into the hands of Friendly Ais. Thus we could pick and choose our allies. The only requirement is that the Friendly AI problem must be solvable (as opposed to being completely beyond human ability) . Both of the above scenarios assume that the first AI (the first powerful, general AI) cannot by itself do global catastrophic damage. Most concrete visualizations that imply this use a g metaphor: A Is as analogous to unusually able humans.

At the time of this writing in 2007, the AI RESEARCH community still does not see Friendly AI as part of the problem. I wish I could cite a reference to this effect, but I cannot cite an absence of literature. Friendly AI is absent from the conceptual landscape, not just unpopular or unfunded. You cannot even call Friendly AI a blank spot on the map, because there is no notion that something 340 Global catastrophic risks is missing. 5 · 6 If you have read popularjsemi-technical books proposing how to build AI, such as Godel, Escher, Bach ( Hofstadter, 1 979) or The Society ofMind (Minsky, 1 986), you may think back and recall that you did not see Friendly A I discussed a s part o fthe challenge. Neither have I seen Friendly AI discussed in the technical literature as a technical problem. My attempted literature search turned up primarily brief non-technical papers, unconnected to each other, with no major reference in common except Isaac Asimov's 'Three Laws of Robotics' (Asimov, 1 942) .

A civil engineer starts by desiring a bridge; then uses a rigorous theory to select a bridge design that supports cars; then builds a real-world bridge whose structure reflects the calculated design; and thus the real-world structure supports cars, thus achieving harmony of predicted positive results and actual positive results. 15.6 Friendly Artificial Intelligence It would be a very good thing if humanity knew how to choose into existence a powerful optimization process with a particular target. In more colloquial terms, it would be nice if we knew how to build a nice AI. To describe the field of knowledge needed to address that challenge, I have proposed the term 'Friendly AI'. In addition to referring to a body oftechnique, ' Friendly AI' might also refer to the product of technique - an A I created with specified motivations. When I use the term Friendly in either sense, I capitalize it to avoid confusion with the intuitive sense of 'friendly'. One common reaction I encounter is for people to immediately declare that Friendly AI is an impossibility because any sufficiently powerful AI will be able to modify its own source code to break any constraints placed upon it. The first flaw you should notice is a Giant Cheesecake Fallacy. Any AI with free access to its own source would, in principle, possess the ability to modify its own source code in a way that changed the A I 's optimization target.


pages: 377 words: 97,144

Singularity Rising: Surviving and Thriving in a Smarter, Richer, and More Dangerous World by James D. Miller

23andMe, affirmative action, Albert Einstein, artificial general intelligence, Asperger Syndrome, barriers to entry, brain emulation, cloud computing, cognitive bias, correlation does not imply causation, crowdsourcing, Daniel Kahneman / Amos Tversky, David Brooks, David Ricardo: comparative advantage, Deng Xiaoping, en.wikipedia.org, feminist movement, Flynn Effect, friendly AI, hive mind, impulse control, indoor plumbing, invention of agriculture, Isaac Newton, John von Neumann, knowledge worker, Long Term Capital Management, low skilled workers, Netflix Prize, neurotypical, Norman Macrae, pattern recognition, Peter Thiel, phenotype, placebo effect, prisoner's dilemma, profit maximization, Ray Kurzweil, recommendation engine, reversible computing, Richard Feynman, Rodney Brooks, Silicon Valley, Singularitarianism, Skype, statistical model, Stephen Hawking, Steve Jobs, supervolcano, technological singularity, The Coming Technological Singularity, the scientific method, Thomas Malthus, transaction costs, Turing test, twin studies, Vernor Vinge, Von Neumann architecture

Even if Eliezer and the Singularity Institute have no realistic chance of creating a friendly AI, they still easily justify their institute’s existence. As Michael Anissimov, media director for the Institute, once told me in a personal conversation, at the very least the Institute has reduced the chance of humanity’s destruction by repeatedly telling artificial-intelligence programmers about the threat of unfriendly AI. The Singularity Institute works toward the only goal I consider worthy of charitable dollars: increasing the survival prospects of mankind. Anna Salamon of the Singularity Institute did a credible back-of-the-envelope calculation showing that, based on some reasonable estimates of the effectiveness of friendly AI research and the harm of an unfriendly Singularity, donating one dollar to research on friendly AI will on average save over one life because slightly decreasing the odds that the seven billion current inhabitants of Earth will die yields you a huge average expected benefit.105 This expected benefit goes way up if you factor in people who are not yet born.

Within a year, we will probably have the technical ability to activate a seed AI, but once the Chinese threat has been annihilated, our team will have no reason to hurry and could take a decade to fine-tune their seed AI. If we delay, any intelligence explosion we eventually create will have an extremely high probability of yielding a friendly AI. Some people on our team think that, given another decade, they will be able to mathematically prove that the seed AI will turn into a friendly ultra-AI. A friendly AI would allow trillions and trillions of people to eventually live their lives, and mankind and our descendants could survive to the end of the universe in utopia. In contrast, an unfriendly AI would destroy us. I have decided to make the survival of humanity my priority. Consequently, since a thermonuclear war would nontrivially increase the chance of human survival, I believe that it’s my moral duty to initiate war, even though my war will kill billions of people.

If the firm really did gain the option to achieve utopia, everyone, including the firm’s investors, would want the firm to exercise this option. The firm, therefore, wouldn’t be able to raise start-up capital from small, self-interested investors. So now pretend that at the time the firm tries to raise capital there is a well-developed theory of friendly AI, which provides programmers with a framework for creating AI that is extremely likely to be well disposed toward humanity and create a utopia if it undergoes an intelligence explosion. To raise funds from self-interested investors, an AI-building firm would need to pick a research and development path that would make it difficult for the firm ever to use the friendly AI framework. Unfortunately, this means that any intelligence explosion the firm unintentionally brings about would be less likely to be utopian than if the firm had used the friendly framework. MULTIPLE AI-BUILDERS Multiple AI-building firms would increase the odds of a bad Singularity.


pages: 1,737 words: 491,616

Rationality: From AI to Zombies by Eliezer Yudkowsky

Albert Einstein, Alfred Russel Wallace, anthropic principle, anti-pattern, anti-work, Arthur Eddington, artificial general intelligence, availability heuristic, Bayesian statistics, Berlin Wall, Build a better mousetrap, Cass Sunstein, cellular automata, cognitive bias, cognitive dissonance, correlation does not imply causation, cosmological constant, creative destruction, Daniel Kahneman / Amos Tversky, dematerialisation, different worldview, discovery of DNA, Douglas Hofstadter, Drosophila, effective altruism, experimental subject, Extropian, friendly AI, fundamental attribution error, Gödel, Escher, Bach, hindsight bias, index card, index fund, Isaac Newton, John Conway, John von Neumann, Long Term Capital Management, Louis Pasteur, mental accounting, meta analysis, meta-analysis, money market fund, Nash equilibrium, Necker cube, NP-complete, P = NP, pattern recognition, Paul Graham, Peter Thiel, Pierre-Simon Laplace, placebo effect, planetary scale, prediction markets, random walk, Ray Kurzweil, reversible computing, Richard Feynman, risk tolerance, Rubik’s Cube, Saturday Night Live, Schrödinger's Cat, scientific mainstream, scientific worldview, sensible shoes, Silicon Valley, Silicon Valley startup, Singularitarianism, Solar eclipse in 1919, speech recognition, statistical model, Steven Pinker, strong AI, technological singularity, The Bell Curve by Richard Herrnstein and Charles Murray, the map is not the territory, the scientific method, Turing complete, Turing machine, ultimatum game, X Prize, Y Combinator, zero-sum game

It’s such a compelling argument, you see. But compulsion is not a property of arguments; it is a property of minds that process arguments. So the reason I’m arguing against the ghost isn’t just to make the point that (1) Friendly AI has to be explicitly programmed and (2) the laws of physics do not forbid Friendly AI. (Though of course I take a certain interest in establishing this.) I also wish to establish the notion of a mind as a causal, lawful, physical system in which there is no irreducible central ghost that looks over the neurons/code and decides whether they are good suggestions. (There is a concept in Friendly AI of deliberately programming an FAI to review its own source code and possibly hand it back to the programmers. But the mind that reviews is not irreducible, it is just the mind that you created.

He’s just making Eliezer1997’s strategy even better by including a contingency plan for “the unlikely event that life turns out to be meaningless” . . . . . . which means that Eliezer2001 now has a line of retreat away from his mistake. I don’t just mean that Eliezer2001 can say “Friendly AI is a contingency plan,” rather than screaming “OOPS!” I mean that Eliezer2001 now actually has a contingency plan. If Eliezer2001 starts to doubt his 1997 metaethics, the intelligence explosion has a fallback strategy, namely Friendly AI. Eliezer2001 can question his metaethics without it signaling the end of the world. And his gradient has been smoothed; he can admit a 10% chance of having previously been wrong, then a 20% chance. He doesn’t have to cough out his whole mistake in one huge lump. If you think this sounds like Eliezer2001 is too slow, I quite agree. Eliezer1996–2000’s strategies had been formed in the total absence of “Friendly AI” as a consideration. The whole idea was to get a superintelligence, any superintelligence, as fast as possible—codelet soup, ad-hoc heuristics, evolutionary programming, open-source, anything that looked like it might work—preferably all approaches simultaneously in a Manhattan Project.

Good’s older term, “intelligence explosion,” to help distinguish his views from other futurist predictions, such as Ray Kurzweil’s exponential technological progress thesis.2 Technologies like smarter-than-human AI seem likely to result in large societal upheavals, for the better or for the worse. Yudkowsky coined the term “Friendly AI theory” to refer to research into techniques for aligning an AGI’s preferences with the preferences of humans. At this point, very little is known about when generally intelligent software might be invented, or what safety approaches would work well in such cases. Present-day autonomous AI can already be quite challenging to verify and validate with much confidence, and many current techniques are not likely to generalize to more intelligent and adaptive systems. “Friendly AI” is therefore closer to a menagerie of basic mathematical and philosophical questions than to a well-specified set of programming objectives. As of 2015, Yudkowsky’s views on the future of AI continue to be debated by technology forecasters and AI researchers in industry and academia, who have yet to converge on a consensus position.


pages: 48 words: 12,437

Smarter Than Us: The Rise of Machine Intelligence by Stuart Armstrong

artificial general intelligence, brain emulation, effective altruism, Flash crash, friendly AI, shareholder value, Turing test

So all is doomed and we’re heading to hell in a digitally engineered handbasket? Well, not entirely. Some effort has been made to make the AI transition safer. Kudos must be given to Eliezer Yudkowsky and Nick Bostrom, who saw and understood the risks early on. Yudkowsky uses the term “Friendly AI” to describe an AI which does what we want even as it improves its own intelligence. In 2000 he cofounded an organization now called the Machine Intelligence Research Institute (MIRI), which holds math research workshops tackling open problems in Friendly AI theory. (MIRI also commissioned and published this book.) Meanwhile, Nick Bostrom founded the Future of Humanity Institute (FHI), a research group within the University of Oxford. FHI is dedicated to analyzing and reducing all existential risks—risks that could drive humanity to extinction or dramatically curtail its potential, of which AI risk is just one example.

But why would an alien mind such as the AI react in comparable ways? Are we not simply training the AI to give the correct answer in training situations? The whole approach is a constraint problem: in the space of possible AI minds, we are going to give priority to those minds that pass successfully through this training process and reassure us that they’re safe. Is there some quantifiable way of measuring how likely this is to produce a human-friendly AI at the end of it? If there isn’t, why are we putting any trust in it? These problems remain barely addressed, so though it is possible to imagine a safe AI being developed using the current approaches (or their descendants), it feels extremely unlikely. Hence we shouldn’t put our trust in the current crop of experts to solve the problem. More work is urgently, perhaps desperately, needed. * * * 1.


pages: 523 words: 148,929

Physics of the Future: How Science Will Shape Human Destiny and Our Daily Lives by the Year 2100 by Michio Kaku

agricultural Revolution, AI winter, Albert Einstein, Asilomar, augmented reality, Bill Joy: nanobots, bioinformatics, blue-collar work, British Empire, Brownian motion, cloud computing, Colonization of Mars, DARPA: Urban Challenge, delayed gratification, double helix, Douglas Hofstadter, en.wikipedia.org, friendly AI, Gödel, Escher, Bach, hydrogen economy, I think there is a world market for maybe five computers, industrial robot, Intergovernmental Panel on Climate Change (IPCC), invention of movable type, invention of the telescope, Isaac Newton, John Markoff, John von Neumann, life extension, Louis Pasteur, Mahatma Gandhi, Mars Rover, mass immigration, megacity, Mitch Kapor, Murray Gell-Mann, new economy, oil shale / tar sands, optical character recognition, pattern recognition, planetary scale, postindustrial economy, Ray Kurzweil, refrigerator car, Richard Feynman, Rodney Brooks, Ronald Reagan, Search for Extraterrestrial Intelligence, Silicon Valley, Simon Singh, social intelligence, speech recognition, stem cell, Stephen Hawking, Steve Jobs, telepresence, The Wealth of Nations by Adam Smith, Thomas L Friedman, Thomas Malthus, trade route, Turing machine, uranium enrichment, Vernor Vinge, Wall-E, Walter Mischel, Whole Earth Review, X Prize

To prevent a robot from enslaving us in order to save us, some have advocated that we must add the zeroth law of robotics: Robots cannot harm or enslave the human race.) But many scientists are leaning toward something called “friendly AI,” where we design our robots to be benign from the very beginning. Since we are the creators of these robots, we will design them, from the very start, to perform only useful and benevolent tasks. The term “friendly AI” was coined by Eliezer Yudkowsky, a founder of the Singularity Institute for Artificial Intelligence. Friendly AI is a bit different from Asimov’s laws, which are forced upon robots, perhaps against their will. (Asimov’s laws, imposed from the outside, could actually invite the robots to devise clever ways to circumvent them.) In friendly AI, by contrast, robots are free to murder and commit mayhem. There are no rules that enforce an artificial morality.

In the future, however, more and more funding for robots will come from the civilian commercial sector, especially from Japan, where robots are designed to help rather than destroy. If this trend continues, then perhaps friendly AI could become a reality. In this scenario, it is the consumer sector and market forces that will eventually dominate robotics, so that there will be a vast commercial interest in investing in friendly AI. MERGING WITH ROBOTS In addition to friendly AI, there is also another option: merging with our creations. Instead of simply waiting for robots to surpass us in intelligence and power, we should try to enhance ourselves, becoming superhuman in the process. Most likely, I believe, the future will proceed with a combination of these two goals, i.e., building friendly AI and also enhancing ourselves. This is an option being explored by Rodney Brooks, former director of the famed MIT Artificial Intelligence Laboratory.

Douglas Hofstadter has said, “It’s as if you took a lot of good food and some dog excrement and blended it all up so that you can’t possibly figure out what’s good or bad. It’s an intimate mixture of rubbish and good ideas, and it’s very hard to disentangle the two, because these are smart people; they’re not stupid.” No one knows how this will play out. But I think the most likely scenario is the following. MOST LIKELY SCENARIO: FRIENDLY AI First, scientists will probably take simple measures to ensure that robots are not dangerous. At the very least, scientists can put a chip in robot brains to automatically shut them off if they have murderous thoughts. In this approach, all intelligent robots will be equipped with a fail-safe mechanism that can be switched on by a human at any time, especially when a robot exhibits errant behavior.


When Computers Can Think: The Artificial Intelligence Singularity by Anthony Berglas, William Black, Samantha Thalind, Max Scratchmann, Michelle Estes

3D printing, AI winter, anthropic principle, artificial general intelligence, Asilomar, augmented reality, Automated Insights, autonomous vehicles, availability heuristic, blue-collar work, brain emulation, call centre, cognitive bias, combinatorial explosion, computer vision, create, read, update, delete, cuban missile crisis, David Attenborough, Elon Musk, en.wikipedia.org, epigenetics, Ernest Rutherford, factory automation, feminist movement, finite state, Flynn Effect, friendly AI, general-purpose programming language, Google Glasses, Google X / Alphabet X, Gödel, Escher, Bach, industrial robot, Isaac Newton, job automation, John von Neumann, Law of Accelerating Returns, license plate recognition, Mahatma Gandhi, mandelbrot fractal, natural language processing, Parkinson's law, patent troll, patient HM, pattern recognition, phenotype, ransomware, Ray Kurzweil, self-driving car, semantic web, Silicon Valley, Singularitarianism, Skype, sorting algorithm, speech recognition, statistical model, stem cell, Stephen Hawking, Stuxnet, superintelligent machines, technological singularity, Thomas Malthus, Turing machine, Turing test, uranium enrichment, Von Neumann architecture, Watson beat the top human players on Jeopardy!, wikimedia commons, zero day

Or alternatively, humanity only has one opportunity to get it right. Friendly AGI research The current centre for research into friendly AI is the Machine Intelligence Research Institute (MIRI) in Berkeley, California, which was co-founded by Yudkowsky. They focus on providing mathematical solutions to the problem of producing friendly AI. In particular, they are researching how to formalize the definition of Coherent Extrapolated Volition so that it can be implemented in software. They also offer a series of very technical workshops to select mathematicians. Another research group is the Future of Humanity Institute, in Oxford UK which is lead by Nick Bostrum who wrote the book SuperIntelligence. The institute has 13 staff, but it takes a wider view than just developing friendly AI. The International Conference on Artificial General Intelligence was founded in 2008 and includes some papers that address controlling an AGI.

Thus the first computer to become hyper-intelligent would quickly dominate all other systems that have the potential to become hyper-intelligent. Thus it is only necessary to tame that first program in order to tame all hyper intelligences. There are many issues with this approach, not least of which is the fact that the military funds much of the research into artificial intelligence. They would want their money back if it turned out to be too friendly. The challenges of building a friendly AI will be discussed in detail in part III. Primary assertions and objections This book develops the following assertions:Computers will eventually become truly intelligent, and then become hyperintelligent. A computer based intelligence would have a world view very different from man’s world view. Goals are not arbitrary, but are subgoals of the need to exist. Intelligent computers’ moral values will be driven by natural selection for the same reason that human moral values have been driven by natural selection.

Experts in the latter endeavour are struggling to define basic terminology and are far from solving the problem. Yet work is being conducted and awareness is being raised. When this book was started in 2011, the only other book on the topic was by Storrs-Hall. In the previous twelve months, however, three more books have been written. It seems likely that the issue of hyper-intelligent machines will become mainstream over the next few years. Fast take off Building a friendly AI would be easier if there is a fast take off. In other words, that the first AGI capable of recursive self-improvement will quickly become exponentially more intelligent and so be able to dominate any other AGIs that are developed. If an AGI doubles its intelligence every month, then a different AGI that is produced just three months later will only have one-eighth as much intelligence as the first AGI and would not stand a chance in any competition.


pages: 428 words: 121,717

Warnings by Richard A. Clarke

active measures, Albert Einstein, algorithmic trading, anti-communist, artificial general intelligence, Asilomar, Asilomar Conference on Recombinant DNA, Bernie Madoff, cognitive bias, collateralized debt obligation, complexity theory, corporate governance, cuban missile crisis, data acquisition, discovery of penicillin, double helix, Elon Musk, failed state, financial thriller, fixed income, Flash crash, forensic accounting, friendly AI, Intergovernmental Panel on Climate Change (IPCC), Internet of things, James Watt: steam engine, Jeff Bezos, John Maynard Keynes: Economic Possibilities for our Grandchildren, knowledge worker, Maui Hawaii, megacity, Mikhail Gorbachev, money market fund, mouse model, Nate Silver, new economy, Nicholas Carr, nuclear winter, pattern recognition, personalized medicine, phenotype, Ponzi scheme, Ray Kurzweil, Richard Feynman, Richard Feynman: Challenger O-ring, risk tolerance, Ronald Reagan, Sam Altman, Search for Extraterrestrial Intelligence, self-driving car, Silicon Valley, smart grid, statistical model, Stephen Hawking, Stuxnet, technological singularity, The Future of Employment, the scientific method, The Signal and the Noise by Nate Silver, Tunguska event, uranium enrichment, Vernor Vinge, Watson beat the top human players on Jeopardy!, women in the workforce, Y2K

His work focuses on foundational mathematical research to ensure (he hopes) that artificial intelligence ultimately has only a positive impact on humanity. The ultimate problem: how to keep humanity from losing control of a machine of its own creation, to prevent artificial intelligence from becoming, in the words of James Barrat in the title of his 2013 book, Our Final Invention.6 A divisive figure, Yudkowsky is well known in academic circles and the Silicon Valley scene as the coiner of the term “friendly AI.” His thesis is simple, though his solution is not: if we are to have any hope against superintelligence, we need to code it properly from the beginning. The answer, Eliezer believes, is one of morality. AI must be programmed with a set of ethical codes that align with humanity’s. Though it is his life’s only work, Yudkowsky is pretty sure he will fail. Humanity, he tells us, is likely doomed.

Yudkowsky believes superintelligence must be designed from the start with something approximating ethics. He envisions this as a system of checks and balances so that advanced AI growth is auditable and controllable, so that even as it continues to learn, advance, and reprogram itself, it will not evolve out of its own benign coding. Such preprogrammed measures will ensure that superintelligence will “behave as we intend even in the absence of immediate human supervision.”12 Eliezer calls this “friendly AI.” According to Yudkowsky, once AI gains the ability to broadly reprogram itself, it will be far too late to implement safeguards, so society needs to prepare now for the intelligence explosion. Yet this preparation is complicated by the sporadic and unpredictable nature of scientific advancement and the numerous secret efforts to create superintelligence around the world. No supranational organization can track all of the efforts, much less predict when or which one of them will succeed.

Eliezer told us that humanity’s best hope is to perhaps create one highly funded, highly secure, multilateral effort to develop a friendly superintelligence with himself (or perhaps another futurist he approves of) at the helm. The work of this massive global Manhattan Project would be explicitly “for the benefit of humanity internationally.” It simultaneously would ban, starve, or simply outpace other, less-well-thought-out efforts to develop superintelligence. Once created, this friendly AI would be unleashed to attack and destroy any competing efforts, ensuring that the only superintelligence in existence would help, not destroy, humankind. Yudkowsky rejects the idea that a superintelligence should, or could, be tailored to parochial national security interests, believing instead that any solution must be considered at the human species level. “This stuff does not stop being lethal because it’s in American hands, or Australian hands, or even in Finland’s hands,” he told us, mildly annoyed.


pages: 761 words: 231,902

The Singularity Is Near: When Humans Transcend Biology by Ray Kurzweil

additive manufacturing, AI winter, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, Albert Einstein, anthropic principle, Any sufficiently advanced technology is indistinguishable from magic, artificial general intelligence, Asilomar, augmented reality, autonomous vehicles, Benoit Mandelbrot, Bill Joy: nanobots, bioinformatics, brain emulation, Brewster Kahle, Brownian motion, business cycle, business intelligence, c2.com, call centre, carbon-based life, cellular automata, Claude Shannon: information theory, complexity theory, conceptual framework, Conway's Game of Life, coronavirus, cosmological constant, cosmological principle, cuban missile crisis, data acquisition, Dava Sobel, David Brooks, Dean Kamen, disintermediation, double helix, Douglas Hofstadter, en.wikipedia.org, epigenetics, factory automation, friendly AI, George Gilder, Gödel, Escher, Bach, informal economy, information retrieval, invention of the telephone, invention of the telescope, invention of writing, iterative process, Jaron Lanier, Jeff Bezos, job automation, job satisfaction, John von Neumann, Kevin Kelly, Law of Accelerating Returns, life extension, lifelogging, linked data, Loebner Prize, Louis Pasteur, mandelbrot fractal, Marshall McLuhan, Mikhail Gorbachev, Mitch Kapor, mouse model, Murray Gell-Mann, mutually assured destruction, natural language processing, Network effects, new economy, Norbert Wiener, oil shale / tar sands, optical character recognition, pattern recognition, phenotype, premature optimization, randomized controlled trial, Ray Kurzweil, remote working, reversible computing, Richard Feynman, Robert Metcalfe, Rodney Brooks, scientific worldview, Search for Extraterrestrial Intelligence, selection bias, semantic web, Silicon Valley, Singularitarianism, speech recognition, statistical model, stem cell, Stephen Hawking, Stewart Brand, strong AI, superintelligent machines, technological singularity, Ted Kaczynski, telepresence, The Coming Technological Singularity, Thomas Bayes, transaction costs, Turing machine, Turing test, Vernor Vinge, Y2K, Yogi Berra

Also see note 30 above. Yudkowsky formed the Singularity Institute for Artificial Intelligence (SIAI) to develop "Friendly AI," intended to "create cognitive content, design features, and cognitive architectures that result in benevolence" before near-human or better-than-human Als become possible. SIAI has developed The SIAI Guidelines on Friendly AI: "Friendly AI," http://www.singinst.org/friendly/. Ben Goertzel and his Artificial General Intelligence Research Institute have also examined issues related to developing friendly AI; his current focus is on developing the Novamente AI Engine, a set of learning algorithms and architectures. Peter Voss, founder of Adaptive A.I., Inc., has also collaborated on friendly-AI issues: http://adaptiveai.com/. 46. Integrated Fuel Cell Technologies, http://ifctech.com. Disclosure: The author is an early investor in and adviser to IFCT. 47.

Vaupel, "Broken Limits to Life Expectancy," Science 296.5570 (May 10,2002): 1029–31. 29. Steve Bowman and Helit Barel, Weapons of Mass Destruction: The Terrorist Threat, Congressional Research Service Report for Congress, December 8, 1999, http://www.cnie.org/nle/crsreports/international/inter-75.pdf. 30. Eliezer S. Yudkowsky, "Creating Friendly AI 1.0, The Analysis and Design of Benevolent Goal Architectures" (2001), The Singularity Institute, http://www.singinst.org/CFAI/; Eliezer S. Yudkowsky, "What Is Friendly AI?" May 3, 2001, http://www.KurzweilAI.net/meme/frame.html?main=/articles/art0172.html. 31. Ted Kaczynski, "The Unabomber's Manifesto," May 14, 2001, http://www.KurzweilAI.netlmeme/frame.html?main=/articles/art0182.html. 32. Bill McKibben, Enough: Staying Human in an Engineered Age (New York: Times Books, 2003). 33.

But the dangers it presents are also profound precisely because of its amplification of intelligence. Intelligence is inherently impossible to control, so the various strategies that have been devised to control nanotechnology (for example, the "broadcast architecture" described below) won't work for strong AI. There have been discussions and proposals to guide AI development toward what Eliezer Yudkowsky calls "friendly AI"30 (see the section "Protection from 'Unfriendly' Strong AI," p. 420). These are useful for discussion, but it is infeasible today to devise strategies that will absolutely ensure that future AI embodies human ethics and values. Returning to the Past? In his essay and presentations Bill Joy eloquently describes the plagues of centuries past and how new self-replicating technologies, such as mutant bioengineered pathogens and nanobots run amok, may bring back long-forgotten pestilence.


pages: 360 words: 100,991

Heart of the Machine: Our Future in a World of Artificial Emotional Intelligence by Richard Yonck

3D printing, AI winter, artificial general intelligence, Asperger Syndrome, augmented reality, Berlin Wall, brain emulation, Buckminster Fuller, call centre, cognitive bias, cognitive dissonance, computer age, computer vision, crowdsourcing, Elon Musk, en.wikipedia.org, epigenetics, friendly AI, ghettoisation, industrial robot, Internet of things, invention of writing, Jacques de Vaucanson, job automation, John von Neumann, Kevin Kelly, Law of Accelerating Returns, Loebner Prize, Menlo Park, meta analysis, meta-analysis, Metcalfe’s law, neurotypical, Oculus Rift, old age dependency ratio, pattern recognition, RAND corporation, Ray Kurzweil, Rodney Brooks, self-driving car, Skype, social intelligence, software as a service, Stephen Hawking, Steven Pinker, superintelligent machines, technological singularity, telepresence, telepresence robot, The Future of Employment, the scientific method, theory of mind, Turing test, twin studies, undersea cable, Vernor Vinge, Watson beat the top human players on Jeopardy!, Whole Earth Review, working-age population, zero day

The resulting machine superintelligence quickly becomes more powerful than the aggregate of all of the minds on the planet. As we saw in the previous chapter, this would almost certainly be a truly alien intelligence. More importantly, there is little guarantee or even likelihood that its values, motives, and logic would align with our own. Multiple strategies have been proposed as a means of dealing with such a superintelligence, including variations on Asimov’s Three Laws, Yudkowsky’s Friendly AI theory, and Goertzel’s global AI nanny.1 Unfortunately, each is far from a foolproof strategy, and that should concern us. While there is considerable division on whether or not such an event could actually come to pass, there is probably a comparable level of disagreement about whether the outcome will be good or really, really bad. One set of scenarios mirrors those explored in the Terminator movies, such that they are often called Terminator scenarios.

The origin of consciousness in the breakdown of the bicameral mind. 1976. Houghton Mifflin. 17. Here, I’m speaking mentally, independent of their need for physical nourishment. 18. Goertzel, B. “The Mind-World Correspondence Principle (Toward a General Theory of General Intelligence).” IEEE Symposium on Computational Intelligence for Human-like Intelligence (CIHLI). 2013. Chapter 18 1. Yudkowsky, E. “Creating Friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures.” Machine Intelligence Research Institute, 2001; Goertzel, Ben. “Should Humanity Build a Global AI Nanny to Delay the Singularity Until It’s Better Understood?” Journal of consciousness studies 19.1-2: 1-2. 2012. 2. Also known as affective empathy. 3. This relation is my extrapolation and is not intended to represent the opinions of either of these scientists. 4.

., 70, 74 Empatica Srl, 70, 74 endocrine system and emotions, 16, 17, 19, 219, 244, 248 ENIAC, 210 Enigma machine, 37 epinephrine, 16, 186, 221 epistemology, 35 EPOC headset, 213 Epstein, Robert, 141 Erewhon (Butler), 228 Ex Machina (Garland), 236–238 exponential change, 40–41 “”EyeJacking: See What I See, 57 Eyeris, 72, 144–145 F FaceSense, 60, 66 Faceshift, 75 Facial Action Coding System (FACS), 55–56 facial expression recognition, 54 facial recognition software, 144 facial tracking system, 59 Fairchild Superconductor, 38 false memories, 145–146 feral children, 257–258 fiction, AIs in dystopias, 228–229 human relationships with AI, 235–239 and the Machine Age, 227–229 problems with machine intelligence, 238–239 threat of intelligent machines, 230–234 flint knapping, 8–13, 23–24 Foresight Program, University of Houston, 171 Forkhead box protein P2, 15 Forster, E. M., 229 FOXP2, 15 Frankenstein (Shelley), 228 Freud, Sigmund, 96 Frewen, Cindy, 170–171 Friendly AI theory, 262 “Friendly Faces of Industry,” 171 Frubber (flesh rubber), 87, 113 functional magnetic resonance imaging (fMRI), 126–127 “The Future of Social Robotics,” 171 G galvactivator, 57–58 gaming community and designer emotions, 217 Garver, Carolyn, 113 Gazzaniga, Michael, 247 Geminoids, 100–101 general intelligence, 255 general morphological analysis (GMA), 165–166 General Problem Solver (1957), 37 Georgia Institute of Technology, 120 geriatric physiotherapy rehabilitation robots, 152 Gibson, William, 171 Gigolo Joe (mecha), 233–234 global AI nanny, 262 GMA.


pages: 481 words: 125,946

What to Think About Machines That Think: Today's Leading Thinkers on the Age of Machine Intelligence by John Brockman

agricultural Revolution, AI winter, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, algorithmic trading, artificial general intelligence, augmented reality, autonomous vehicles, basic income, bitcoin, blockchain, clean water, cognitive dissonance, Colonization of Mars, complexity theory, computer age, computer vision, constrained optimization, corporate personhood, cosmological principle, cryptocurrency, cuban missile crisis, Danny Hillis, dark matter, discrete time, Douglas Engelbart, Elon Musk, Emanuel Derman, endowment effect, epigenetics, Ernest Rutherford, experimental economics, Flash crash, friendly AI, functional fixedness, global pandemic, Google Glasses, hive mind, income inequality, information trail, Internet of things, invention of writing, iterative process, Jaron Lanier, job automation, Johannes Kepler, John Markoff, John von Neumann, Kevin Kelly, knowledge worker, loose coupling, microbiome, Moneyball by Michael Lewis explains big data, natural language processing, Network effects, Norbert Wiener, pattern recognition, Peter Singer: altruism, phenotype, planetary scale, Ray Kurzweil, recommendation engine, Republic of Letters, RFID, Richard Thaler, Rory Sutherland, Satyajit Das, Search for Extraterrestrial Intelligence, self-driving car, sharing economy, Silicon Valley, Skype, smart contracts, social intelligence, speech recognition, statistical model, stem cell, Stephen Hawking, Steve Jobs, Steven Pinker, Stewart Brand, strong AI, Stuxnet, superintelligent machines, supervolcano, the scientific method, The Wisdom of Crowds, theory of mind, Thorstein Veblen, too big to fail, Turing machine, Turing test, Von Neumann architecture, Watson beat the top human players on Jeopardy!, Y2K

The system can be used between human parties or interspecies parties, exactly because it’s not necessary to know, trust, or understand the other entity, just the code (the language of machines). Over time, trust can grow through reputation. Blockchain technology could be used to enforce friendly AI and mutually beneficial interspecies interaction. Someday, important transactions (like identity authentication and resource transfer) will be conducted on smart networks that require confirmation by independent consensus mechanisms, such that only bona fide transactions by reputable entities are executed. While perhaps not a full answer to the problem of enforcing friendly AI, decentralized smart networks like blockchains are a system of checks and balances helping to provide a more robust solution to situations of future uncertainty. Trust-building models for interspecies digital intelligence interaction could include both game-theoretic checks-and-balances systems like blockchains and also, at the higher level, frameworks that put entities on the same plane of shared objectives.


pages: 303 words: 67,891

Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms: Proceedings of the Agi Workshop 2006 by Ben Goertzel, Pei Wang

AI winter, artificial general intelligence, bioinformatics, brain emulation, combinatorial explosion, complexity theory, computer vision, conceptual framework, correlation coefficient, epigenetics, friendly AI, G4S, information retrieval, Isaac Newton, John Conway, Loebner Prize, Menlo Park, natural language processing, Occam's razor, p-value, pattern recognition, performance metric, Ray Kurzweil, Rodney Brooks, semantic web, statistical model, strong AI, theory of mind, traveling salesman, Turing machine, Turing test, Von Neumann architecture, Y2K

Also, it is possible to directly edit the memory of a system to modify or implant certain knowledge. In theory, all of these shortcuts should be equivalent to certain possible experience of the system, so they do not conflict with the principle that all the knowledge of NARS comes, directly or indirectly, from experience. One important issue to be handled through education is ethics. Unlike argued by some other researchers, NARS is not an attempt to design a “friendly AI”. As far as its initial state is concerned, the system is ethically neutral, since it can has any beliefs and goals. To make a NARS implementation “human friendly” means to give it certain beliefs and goals, which is an education mission, not a design mission. Even if something like Asimov’s “Three Laws of Robotics” is implanted into the system’s memory (which is possible), it still cannot fully control the system’s behaviors, due to the insufficiency of knowledge and resources in the system.

[Audience]: Ben, you seem more optimistic. Could you talk about your perspective? [Ben Goertzel]: Well I have a quite different opinion than that of Steve Grand in that I don’t think an amazing conceptual breakthrough on the level of the discovery of the quantum or curved 4D space-time, or something like that, is needed to create general intelligence. It might be needed to create provably stable friendly AI, like Eliezer Yudkowsky would like. I tend to think of the brain as a complex system composed of a bunch of evolved kluges for solving particular problems, which have been hacked together and adapted by evolution. I think if you assemble subcomponents solving the appropriate set of specialized problems, as well as a fairly weak general problem solver, and they are hooked together in a knowledge representation that works for all the components, with learning mechanisms that let each component learn from each other -- then you are going to have an intelligent mind that can be taught.


pages: 222 words: 53,317

Overcomplicated: Technology at the Limits of Comprehension by Samuel Arbesman

algorithmic trading, Anton Chekhov, Apple II, Benoit Mandelbrot, citation needed, combinatorial explosion, Danny Hillis, David Brooks, digital map, discovery of the americas, en.wikipedia.org, Erik Brynjolfsson, Flash crash, friendly AI, game design, Google X / Alphabet X, Googley, HyperCard, Inbox Zero, Isaac Newton, iterative process, Kevin Kelly, Machine translation of "The spirit is willing, but the flesh is weak." to Russian and back, mandelbrot fractal, Minecraft, Netflix Prize, Nicholas Carr, Parkinson's law, Ray Kurzweil, recommendation engine, Richard Feynman, Richard Feynman: Challenger O-ring, Second Machine Age, self-driving car, software studies, statistical model, Steve Jobs, Steve Wozniak, Steven Pinker, Stewart Brand, superintelligent machines, Therac-25, Tyler Cowen: Great Stagnation, urban planning, Watson beat the top human players on Jeopardy!, Whole Earth Catalog, Y2K

naches is also a framework: The roboticist Hans Moravec has referred to our more powerful descendants as “mind children,” and a similar approach characterizes a short story by the science fiction writer Ted Chiang, in which technologically enhanced humans have long surpassed “regular” humans in their ability to make scientific discoveries. In the end, little to nothing is understood by (nonenhanced) humanity. But that’s okay, because “We need not be intimidated by the accomplishments of metahuman science. We should always remember that the technologies that made metahumans possible were originally developed by humans, and they were no smarter than we.” See Luke Muehlhauser and Nick Bostrom, “Why We Need Friendly AI,” Think 36, no. 13 (Spring 2014), 41–47; and Ted Chiang, Stories of Your Life and Others (New York: Tor Books, 2003), 203. understand the most complex parts of the world: In many cases, we might even want to have a technology too complex to understand, because it means that it is sophisticated and powerful. a grab bag of intriguing ideas: The World of Wonders: A Record of Things Wonderful in Nature, Science, and Art (London: Cassell, Petter, and Galpin, exact year of publication unknown), https://archive.org/details/worldofwondersre00londrich.


pages: 523 words: 61,179

Human + Machine: Reimagining Work in the Age of AI by Paul R. Daugherty, H. James Wilson

3D printing, AI winter, algorithmic trading, Amazon Mechanical Turk, augmented reality, autonomous vehicles, blockchain, business process, call centre, carbon footprint, cloud computing, computer vision, correlation does not imply causation, crowdsourcing, digital twin, disintermediation, Douglas Hofstadter, en.wikipedia.org, Erik Brynjolfsson, friendly AI, future of work, industrial robot, Internet of things, inventory management, iterative process, Jeff Bezos, job automation, job satisfaction, knowledge worker, Lyft, natural language processing, personalized medicine, precision agriculture, Ray Kurzweil, recommendation engine, RFID, ride hailing / ride sharing, risk tolerance, Rodney Brooks, Second Machine Age, self-driving car, sensor fusion, sentiment analysis, Shoshana Zuboff, Silicon Valley, software as a service, speech recognition, telepresence, telepresence robot, text mining, the scientific method, uber lyft

This allows human experts to exercise their judgment to resolve any ambiguities or other issues early on in the process. FUSION SKILL #4: Intelligent Interrogation Definition: Knowing how best to ask questions of AI, across levels of abstraction, to get the insights you need. How do you probe a massively complex system? How do you predict interactions between complex layers of data? People simply can’t do this on their own, so they must ask questions of their friendly AI. “Twin, how certain are you?” “Twin, what do you recommend?” At GE, maintenance professionals who have the skill of intelligent interrogation understand the capabilities and limitations of the AI system and know how to get the information they need to make an informed decision. The workers play to strengths and don’t duplicate the machine’s strengths. In the process, the machine is training the humans in how to use it, just as the humans train the machine.


pages: 574 words: 164,509

Superintelligence: Paths, Dangers, Strategies by Nick Bostrom

agricultural Revolution, AI winter, Albert Einstein, algorithmic trading, anthropic principle, anti-communist, artificial general intelligence, autonomous vehicles, barriers to entry, Bayesian statistics, bioinformatics, brain emulation, cloud computing, combinatorial explosion, computer vision, cosmological constant, dark matter, DARPA: Urban Challenge, data acquisition, delayed gratification, demographic transition, different worldview, Donald Knuth, Douglas Hofstadter, Drosophila, Elon Musk, en.wikipedia.org, endogenous growth, epigenetics, fear of failure, Flash crash, Flynn Effect, friendly AI, Gödel, Escher, Bach, income inequality, industrial robot, informal economy, information retrieval, interchangeable parts, iterative process, job automation, John Markoff, John von Neumann, knowledge worker, longitudinal study, Menlo Park, meta analysis, meta-analysis, mutually assured destruction, Nash equilibrium, Netflix Prize, new economy, Norbert Wiener, NP-complete, nuclear winter, optical character recognition, pattern recognition, performance metric, phenotype, prediction markets, price stability, principal–agent problem, race to the bottom, random walk, Ray Kurzweil, recommendation engine, reversible computing, social graph, speech recognition, Stanislav Petrov, statistical model, stem cell, Stephen Hawking, strong AI, superintelligent machines, supervolcano, technological singularity, technoutopianism, The Coming Technological Singularity, The Nature of the Firm, Thomas Kuhn: the structure of scientific revolutions, transaction costs, Turing machine, Vernor Vinge, Watson beat the top human players on Jeopardy!, World Values Survey, zero-sum game

The programmers may try to guard against this possibility by secretly monitoring the AI’s source code and the internal workings of its mind; but a smart-enough AI would realize that it might be under surveillance and adjust its thinking accordingly.2 The AI might find subtle ways of concealing its true capabilities and its incriminating intent.3 (Devising clever escape plans might, incidentally, also be a convergent strategy for many types of friendly AI, especially as they mature and gain confidence in their own judgments and capabilities. A system motivated to promote our interests might be making a mistake if it allowed us to shut it down or to construct another, potentially unfriendly AI.) We can thus perceive a general failure mode, wherein the good behavioral track record of a system in its juvenile stages fails utterly to predict its behavior at a more mature stage.

Nonzero: The Logic of Human Destiny. New York: Vintage. Yaeger, Larry. 1994. “Computational Genetics, Physiology, Metabolism, Neural Systems, Learning, Vision, and Behavior or PolyWorld: Life in a New Context.” In Proceedings of the Artificial Life III Conference, edited by C. G. Langton, 263–98. Santa Fe Institute Studies in the Sciences of Complexity. Reading, MA: Addison-Wesley. Yudkowsky, Eliezer. 2001. Creating Friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures. Machine Intelligence Research Institute, San Francisco, CA, June 15. Yudkowsky, Eliezer. 2002. “The AI-Box Experiment.” Retrieved January 15, 2012. Available at http://yudkowsky.net/singularity/aibox. Yudkowsky, Eliezer. 2004. Coherent Extrapolated Volition. Machine Intelligence Research Institute, San Francisco, CA, May. Yudkowsky, Eliezer. 2007.


pages: 252 words: 74,167

Thinking Machines: The Inside Story of Artificial Intelligence and Our Race to Build the Future by Luke Dormehl

Ada Lovelace, agricultural Revolution, AI winter, Albert Einstein, Alexey Pajitnov wrote Tetris, algorithmic trading, Amazon Mechanical Turk, Apple II, artificial general intelligence, Automated Insights, autonomous vehicles, book scanning, borderless world, call centre, cellular automata, Claude Shannon: information theory, cloud computing, computer vision, correlation does not imply causation, crowdsourcing, drone strike, Elon Musk, Flash crash, friendly AI, game design, global village, Google X / Alphabet X, hive mind, industrial robot, information retrieval, Internet of things, iterative process, Jaron Lanier, John Markoff, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John von Neumann, Kickstarter, Kodak vs Instagram, Law of Accelerating Returns, life extension, Loebner Prize, Marc Andreessen, Mark Zuckerberg, Menlo Park, natural language processing, Norbert Wiener, out of africa, PageRank, pattern recognition, Ray Kurzweil, recommendation engine, remote working, RFID, self-driving car, Silicon Valley, Skype, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, social intelligence, speech recognition, Stephen Hawking, Steve Jobs, Steve Wozniak, Steven Pinker, strong AI, superintelligent machines, technological singularity, The Coming Technological Singularity, The Future of Employment, Tim Cook: Apple, too big to fail, Turing machine, Turing test, Vernor Vinge, Watson beat the top human players on Jeopardy!

As an increasing amount of work is carried out involving autonomous AI weapons in war situations, work like Bringsjord’s is in high demand. In 2014, a multidisciplinary team at Tufts and Brown Universities, working alongside Bringsjord, received funding from the Office of Naval Research to explore the possibility of giving autonomous robots – potentially used in combat – a sense of right and wrong. While not exactly a ‘friendly AI’, this kind of computational morality would allow robots on the battlefield to make ethical decisions. Imagine, for instance, a robot medic that is transporting an injured soldier to a field hospital encounters another soldier with an injured leg. Weighing up the pros and cons of stopping its mission to administer aid, potentially administering pain relief by applying traction in the field, and other conundrums are all complex issues for a human to navigate – let alone a machine.


pages: 252 words: 79,452

To Be a Machine: Adventures Among Cyborgs, Utopians, Hackers, and the Futurists Solving the Modest Problem of Death by Mark O'Connell

3D printing, Ada Lovelace, AI winter, Airbnb, Albert Einstein, artificial general intelligence, brain emulation, clean water, cognitive dissonance, computer age, cosmological principle, dark matter, disruptive innovation, double helix, Edward Snowden, effective altruism, Elon Musk, Extropian, friendly AI, global pandemic, impulse control, income inequality, invention of the wheel, Jacques de Vaucanson, John von Neumann, knowledge economy, Law of Accelerating Returns, life extension, lifelogging, Lyft, Mars Rover, means of production, Norbert Wiener, Peter Thiel, profit motive, Ray Kurzweil, RFID, self-driving car, sharing economy, Silicon Valley, Silicon Valley ideology, Singularitarianism, Skype, Stephen Hawking, Steve Wozniak, superintelligent machines, technological singularity, technoutopianism, The Coming Technological Singularity, Travis Kalanick, trickle-down economics, Turing machine, uber lyft, Vernor Vinge

“If you consider balancing the interests of future people against those who already exist, reducing the probability of a major future catastrophe can be a very high-impact decision. If you succeed in avoiding an event that might wipe out all of future humanity, that clearly exceeds any good you might do for people currently living.” The Future of Life Institute was less focused than MIRI on the mathematical arcana of how a “friendly AI” might be engineered. The group, she said, functioned as “the outreach arm of this cluster of organizations,” raising awareness about the seriousness of this problem. It was not the attention of the media or the general public for which FLI was campaigning, Viktoriya said, but rather that of AI researchers themselves, a constituency in which the idea of existential risk was only just beginning to be taken seriously.


pages: 339 words: 94,769

Possible Minds: Twenty-Five Ways of Looking at AI by John Brockman

AI winter, airport security, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, artificial general intelligence, Asilomar, autonomous vehicles, basic income, Benoit Mandelbrot, Bill Joy: nanobots, Buckminster Fuller, cellular automata, Claude Shannon: information theory, Daniel Kahneman / Amos Tversky, Danny Hillis, David Graeber, easy for humans, difficult for computers, Elon Musk, Eratosthenes, Ernest Rutherford, finite state, friendly AI, future of work, Geoffrey West, Santa Fe Institute, gig economy, income inequality, industrial robot, information retrieval, invention of writing, James Watt: steam engine, Johannes Kepler, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John von Neumann, Kevin Kelly, Kickstarter, Laplace demon, Loebner Prize, market fundamentalism, Marshall McLuhan, Menlo Park, Norbert Wiener, optical character recognition, pattern recognition, personalized medicine, Picturephone, profit maximization, profit motive, RAND corporation, random walk, Ray Kurzweil, Richard Feynman, Rodney Brooks, self-driving car, sexual politics, Silicon Valley, Skype, social graph, speech recognition, statistical model, Stephen Hawking, Steven Pinker, Stewart Brand, strong AI, superintelligent machines, supervolcano, technological singularity, technoutopianism, telemarketer, telerobotics, the scientific method, theory of mind, Turing machine, Turing test, universal basic income, Upton Sinclair, Von Neumann architecture, Whole Earth Catalog, Y2K, zero-sum game

A superintelligent AGI will be extremely good at accomplishing its goals, and if those goals aren’t aligned with ours, we’re in trouble. People don’t think twice about flooding anthills to build hydroelectric dams, so let’s not place humanity in the position of those ants. Most researchers argue that if we end up creating superintelligence, we should make sure it’s what AI-safety pioneer Eliezer Yudkowsky has termed “friendly AI”—AI whose goals are in some deep sense beneficial. The moral question of what these goals should be is just as urgent as the technical questions about goal alignment. For example, what sort of society are we hoping to create, where we find meaning and purpose in our lives even though we, strictly speaking, aren’t needed? I’m often given the following glib response to this question: “Let’s build machines that are smarter than us and then let them figure out the answer!”


pages: 345 words: 104,404

Pandora's Brain by Calum Chace

AI winter, Any sufficiently advanced technology is indistinguishable from magic, artificial general intelligence, brain emulation, Extropian, friendly AI, hive mind, lateral thinking, mega-rich, Ray Kurzweil, self-driving car, Silicon Valley, Singularitarianism, Skype, speech recognition, stealth mode startup, Stephen Hawking, strong AI, technological singularity, theory of mind, Turing test, Wall-E

The European Commission was proposing an autonomous international agency with exceptional powers of inspection and restraint in order to stop rogue governments and individuals from doing any research that could lead to the creation of an AGI. This was a step too far for him, as it was for most Americans. ‘That game is far from over. And I wouldn’t be surprised if a coalition of the institutes which are researching human-friendly AI algorithms announced a breakthrough this spring. There’s been a lot of unusually cordial traffic between several of the bigger US-based ones during the winter. Anyway, fortunately, none of that affects what we’re doing here at the Foundation. We’ve managed to put clear blue water between brain preservation research and AI research in the public’s mind.’ ‘True. It’s funny, though,’ David mused.


pages: 1,331 words: 163,200

Hands-On Machine Learning With Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems by Aurélien Géron

Amazon Mechanical Turk, Anton Chekhov, combinatorial explosion, computer vision, constrained optimization, correlation coefficient, crowdsourcing, don't repeat yourself, Elon Musk, en.wikipedia.org, friendly AI, ImageNet competition, information retrieval, iterative process, John von Neumann, Kickstarter, natural language processing, Netflix Prize, NP-complete, optical character recognition, P = NP, p-value, pattern recognition, pull request, recommendation engine, self-driving car, sentiment analysis, SpamAssassin, speech recognition, stochastic process

(e) was reproduced from Pixabay, released under Creative Commons CC0. 6 It is often better to give the poor performers a slight chance of survival, to preserve some diversity in the “gene pool.” 7 If there is a single parent, this is called asexual reproduction. With two (or more) parents, it is called sexual reproduction. An offspring’s genome (in this case a set of policy parameters) is randomly composed of parts of its parents’ genomes. 8 OpenAI is a nonprofit artificial intelligence research company, funded in part by Elon Musk. Its stated goal is to promote and develop friendly AIs that will benefit humanity (rather than exterminate it). 9 “Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning,” R. Williams (1992). 10 We already did something similar in Chapter 11 when we discussed Gradient Clipping: we first computed the gradients, then we clipped them, and finally we applied the clipped gradients. 11 “A Markovian Decision Process,” R.


pages: 798 words: 240,182

The Transhumanist Reader by Max More, Natasha Vita-More

23andMe, Any sufficiently advanced technology is indistinguishable from magic, artificial general intelligence, augmented reality, Bill Joy: nanobots, bioinformatics, brain emulation, Buckminster Fuller, cellular automata, clean water, cloud computing, cognitive bias, cognitive dissonance, combinatorial explosion, conceptual framework, Conway's Game of Life, cosmological principle, data acquisition, discovery of DNA, Douglas Engelbart, Drosophila, en.wikipedia.org, endogenous growth, experimental subject, Extropian, fault tolerance, Flynn Effect, Francis Fukuyama: the end of history, Frank Gehry, friendly AI, game design, germ theory of disease, hypertext link, impulse control, index fund, John von Neumann, joint-stock company, Kevin Kelly, Law of Accelerating Returns, life extension, lifelogging, Louis Pasteur, Menlo Park, meta analysis, meta-analysis, moral hazard, Network effects, Norbert Wiener, pattern recognition, Pepto Bismol, phenotype, positional goods, prediction markets, presumed consent, Ray Kurzweil, reversible computing, RFID, Ronald Reagan, scientific worldview, silicon-based life, Singularitarianism, social intelligence, stem cell, stochastic process, superintelligent machines, supply-chain management, supply-chain management software, technological singularity, Ted Nelson, telepresence, telepresence robot, telerobotics, the built environment, The Coming Technological Singularity, the scientific method, The Wisdom of Crowds, transaction costs, Turing machine, Turing test, Upton Sinclair, Vernor Vinge, Von Neumann architecture, Whole Earth Review, women in the workforce, zero-sum game

One key point, though, is that the risks and rewards of AGI must be considered in the broader context of all the other technologies currently under development, and all the social, psychological, and technological change likely to come in the next decades and centuries. It’s not as though our choices are “life goes on exactly as is” versus “life as it is plus super-powerful AI.” Various technologies are advancing rapidly and society is changing accordingly, and the rate of advancement of AGI is just one aspect in the mix. I don’t think there are any magic bullets to resolve the dilemmas of AGI ethics. There will almost surely be no provably Friendly AI, in spite of the wishes of Eliezer Yudkowsky (2008) and some others. Nor, in my best guess, will there be an Artilect War in which pro-AGI and anti-AGI forces battle to the death with doomsday machines, as Hugo de Garis (2005) foresees. But I don’t pretend to be able to see exactly what the outcome will be. The important thing, as I see it, is that the human race as a whole engages as closely and intelligently as possible with AGI as it evolves – so that, as AGI comes about, it’s not a matter of “us versus them”, but rather a matter of AGIs and humans, that have become inseparable on various levels, moving forward together into new realms of science, technology, interaction, and experience.