friendly AI

23 results back to index

Our Final Invention: Artificial Intelligence and the End of the Human Era by James Barrat

AI winter, air gap, AltaVista, Amazon Web Services, artificial general intelligence, Asilomar, Automated Insights, Bayesian statistics, Bernie Madoff, Bill Joy: nanobots, Bletchley Park, brain emulation, California energy crisis, cellular automata, Chuck Templeton: OpenTable:, cloud computing, cognitive bias, commoditize, computer vision, Computing Machinery and Intelligence, cuban missile crisis, Daniel Kahneman / Amos Tversky, Danny Hillis, data acquisition, don't be evil, drone strike, dual-use technology, Extropian, finite state, Flash crash, friendly AI, friendly fire, Google Glasses, Google X / Alphabet X, Hacker News, Hans Moravec, Isaac Newton, Jaron Lanier, Jeff Hawkins, John Markoff, John von Neumann, Kevin Kelly, Law of Accelerating Returns, life extension, Loebner Prize, lone genius, machine translation, mutually assured destruction, natural language processing, Neil Armstrong, Nicholas Carr, Nick Bostrom, optical character recognition, PageRank, PalmPilot, paperclip maximiser, pattern recognition, Peter Thiel, precautionary principle, prisoner's dilemma, Ray Kurzweil, Recombinant DNA, Rodney Brooks, rolling blackouts, Search for Extraterrestrial Intelligence, self-driving car, semantic web, Silicon Valley, Singularitarianism, Skype, smart grid, speech recognition, statistical model, stealth mode startup, stem cell, Stephen Hawking, Steve Jobs, Steve Jurvetson, Steve Wozniak, strong AI, Stuxnet, subprime mortgage crisis, superintelligent machines, technological singularity, The Coming Technological Singularity, Thomas Bayes, traveling salesman, Turing machine, Turing test, Vernor Vinge, Watson beat the top human players on Jeopardy!, zero day

But if an intelligence explosion kicks in and Yudkowsky has been unsuccessful in implementing safeguards, there’s a good chance we’ll all be goo, and so will the universe. That puts Yudkowsky at the dead center of his own cosmology. I had come here to learn more about Friendly AI, a term he coined. According to Yudkowsky, Friendly AI is the kind that will preserve humanity and our values forever. It doesn’t annihilate our species or spread into the universe like a planet-eating space plague. But what is Friendly AI? How do you create it? I also wanted to hear about the AI Box Experiment. I especially wanted to know, as he role-played the part of the AGI, how he talked the Gatekeeper into setting him free.

…

[italics mine] Not knowing how to build a Friendly AI is not deadly, of itself.… It’s the mistaken belief that an AI will be friendly which implies an obvious path to global catastrophe. Assuming that human-level AIs (AGIs) will be friendly is wrong for a lot of reasons. The assumption becomes even more dangerous after the AGI’s intelligence rockets past ours, and it becomes ASI—artificial superintelligence. So how do you create friendly AI? Or could you impose friendliness on advanced AIs after they’re already built? Yudkowsky has written a book-length online treatise about these questions entitled Creating Friendly AI: The Analysis and Design of Benevolent Goal Architectures.

…

Yudkowsky has written a book-length online treatise about these questions entitled Creating Friendly AI: The Analysis and Design of Benevolent Goal Architectures. Friendly AI is a subject so dense yet important it exasperates its chief proponent himself, who says about it, “it only takes one error for a chain of reasoning to end up in Outer Mongolia.” Let’s start with a simple definition. Friendly AI is AI that has a positive rather than a negative impact on mankind. Friendly AI pursues goals, and it takes action to fulfill those goals. To describe an AI’s success at achieving its goals, theorists use a term from economics: utility. As you might recall from Econ 101, consumers behaving rationally seek to maximize utility by spending their resources in the way that gives them the most satisfaction.

Surviving AI: The Promise and Peril of Artificial Intelligence by Calum Chace

3D printing, Ada Lovelace, AI winter, Airbnb, Alvin Toffler, artificial general intelligence, augmented reality, barriers to entry, basic income, bitcoin, Bletchley Park, blockchain, brain emulation, Buckminster Fuller, Charles Babbage, cloud computing, computer age, computer vision, correlation does not imply causation, credit crunch, cryptocurrency, cuban missile crisis, deep learning, DeepMind, dematerialisation, Demis Hassabis, discovery of the americas, disintermediation, don't be evil, driverless car, Elon Musk, en.wikipedia.org, epigenetics, Erik Brynjolfsson, everywhere but in the productivity statistics, Flash crash, friendly AI, Geoffrey Hinton, Google Glasses, hedonic treadmill, hype cycle, industrial robot, Internet of things, invention of agriculture, job automation, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John von Neumann, Kevin Kelly, life extension, low skilled workers, machine translation, Mahatma Gandhi, means of production, mutually assured destruction, Neil Armstrong, Nicholas Carr, Nick Bostrom, paperclip maximiser, pattern recognition, peer-to-peer, peer-to-peer model, Peter Thiel, radical life extension, Ray Kurzweil, Robert Solow, Rodney Brooks, Second Machine Age, self-driving car, Silicon Valley, Silicon Valley ideology, Skype, South Sea Bubble, speech recognition, Stanislav Petrov, Stephen Hawking, Steve Jobs, strong AI, technological singularity, TED Talk, The future is already here, The Future of Employment, theory of mind, Turing machine, Turing test, universal basic income, Vernor Vinge, wage slave, Wall-E, zero-sum game

A superintelligence could become not just twice as smart as humans, but smarter by many orders of magnitude. It is hard to escape the conclusion that our future will depend on its decisions and its actions. Would that be a good thing or a bad thing? In other words, would a superintelligence be a “Friendly AI”? (Friendly AI, or FAI, denotes an AGI that is beneficial for humans rather than one that seeks social approbation and company. It also refers to the project to make sure that AGI is beneficial.) 7.2 – Optimistic scenarios: to immortality and beyond The ultimate problem solver Imagine having a big sister endowed with superhuman wisdom, insight and ingenuity.

…

So if we take seriously the idea that a superintelligence may appear on the Earth in the foreseeable future, we should certainly be thinking about how to ensure that the event is a positive one for ourselves and our descendants. We should be taking steps to ensure that the first AGI is a friendly AI. PART FOUR: FAI Friendly Artificial Intelligence CHAPTER 8 CAN WE ENSURE THAT SUPERINTELLIGENCE IS SAFE? As we saw in the last chapter, Friendly AI (FAI) is the project of ensuring that the world’s superintelligences are safe and useful for humans. The central argument of this book is that we need to address this challenge successfully. It may well turn out to be the most important challenge facing this generation and the next.

…

It also boasts some Hollywood glamour, with Alan Alda and Morgan Freeman on the advisory board, along with technology entrepreneur Elon Musk, who has donated $10m of his personal money to the institute. 8.6 – Conclusion We do not yet have a foolproof way to ensure that the first AGI is a Friendly AI. In fact we don’t yet know how best to approach the problem. But we have only just begun, and the resources allocated to the problem are small: Nick Bostrom estimated in 2014 that only six people in the world are working full-time on the Friendly AI problem, whereas many thousands of people work full-time on projects that could well contribute to the creation of the first AGI. (50) He argued that this equation needed urgent re-balancing.

Blockchain: Blueprint for a New Economy by Melanie Swan

23andMe, Airbnb, altcoin, Amazon Web Services, asset allocation, banking crisis, basic income, bioinformatics, bitcoin, blockchain, capital controls, cellular automata, central bank independence, clean water, cloud computing, collaborative editing, Conway's Game of Life, crowdsourcing, cryptocurrency, data science, digital divide, disintermediation, Dogecoin, Edward Snowden, en.wikipedia.org, Ethereum, ethereum blockchain, fault tolerance, fiat currency, financial innovation, Firefox, friendly AI, Hernando de Soto, information security, intangible asset, Internet Archive, Internet of things, Khan Academy, Kickstarter, Large Hadron Collider, lifelogging, litecoin, Lyft, M-Pesa, microbiome, Neal Stephenson, Network effects, new economy, operational security, peer-to-peer, peer-to-peer lending, peer-to-peer model, personalized medicine, post scarcity, power law, prediction markets, QR code, ride hailing / ride sharing, Satoshi Nakamoto, Search for Extraterrestrial Intelligence, SETI@home, sharing economy, Skype, smart cities, smart contracts, smart grid, Snow Crash, software as a service, synthetic biology, technological singularity, the long tail, Turing complete, uber lyft, unbanked and underbanked, underbanked, Vitalik Buterin, Wayback Machine, web application, WikiLeaks

Limitations Technical Challenges Business Model Challenges Scandals and Public Perception Government Regulation Privacy Challenges for Personal Records Overall: Decentralization Trends Likely to Persist 7. Conclusion The Blockchain Is an Information Technology Blockchain AI: Consensus as the Mechanism to Foster “Friendly” AI Large Possibility Space for Intelligence Only Friendly AIs Are Able to Get Their Transactions Executed Smart Contract Advocates on Behalf of Digital Intelligence Blockchain Consensus Increases the Information Resolution of the Universe A. Cryptocurrency Basics Public/Private-Key Cryptography 101 B. Ledra Capital Mega Master Blockchain List Endnotes and References Index Blockchain Blueprint for a New Economy Melanie Swan Blockchain by Melanie Swan Copyright © 2015 Melanie Swan.

…

The blockchain is a consensus model at scale, and possibly the mechanism we have been waiting for that could help to usher in an era of friendly machine intelligence. Blockchain AI: Consensus as the Mechanism to Foster “Friendly” AI One forward-looking but important concern in the general future of technology is different ways in which artificial intelligence (AI) might arise and how to sponsor it such that it engenders a “friendly” or benevolent relationship with humans. There is the notion of a technological singularity, a moment when machine intelligence might supersede human intelligence. However, those in the field have not set forth any sort of robust plan for how to effect friendly AI, and many remain skeptical of this possibility.195 It is possible that blockchain technology could be a useful connector of humans and machines in a world of increasingly autonomous machine activity through Dapps, DAOs, and DACs that might eventually give way to AI.

…

These intelligences would likely not be operating in isolation, but would be connected to communications networks. To achieve their goals, digital intelligences will want to conduct certain transactions over the network, many of which could be managed by blockchain and other consensus mechanisms. Only Friendly AIs Are Able to Get Their Transactions Executed One of the unforeseen benefits of consensus models might be that they could possibly enforce friendly AI, which is to say cooperative, moral players within a society.196 In decentralized trust networks, an agentÕs reputation (where agents themselves remain pseudonymous) could be an important factor in whether his transactions will be executed, such that malicious players would not be able to get their transactions executed or recognized on the network.

Global Catastrophic Risks by Nick Bostrom, Milan M. Cirkovic

affirmative action, agricultural Revolution, Albert Einstein, American Society of Civil Engineers: Report Card, anthropic principle, artificial general intelligence, Asilomar, availability heuristic, backpropagation, behavioural economics, Bill Joy: nanobots, Black Swan, carbon tax, carbon-based life, Charles Babbage, classic study, cognitive bias, complexity theory, computer age, coronavirus, corporate governance, cosmic microwave background, cosmological constant, cosmological principle, cuban missile crisis, dark matter, death of newspapers, demographic transition, Deng Xiaoping, distributed generation, Doomsday Clock, Drosophila, endogenous growth, Ernest Rutherford, failed state, false flag, feminist movement, framing effect, friendly AI, Georg Cantor, global pandemic, global village, Great Leap Forward, Gödel, Escher, Bach, Hans Moravec, heat death of the universe, hindsight bias, information security, Intergovernmental Panel on Climate Change (IPCC), invention of agriculture, Kevin Kelly, Kuiper Belt, Large Hadron Collider, launch on warning, Law of Accelerating Returns, life extension, means of production, meta-analysis, Mikhail Gorbachev, millennium bug, mutually assured destruction, Nick Bostrom, nuclear winter, ocean acidification, off-the-grid, Oklahoma City bombing, P = NP, peak oil, phenotype, planetary scale, Ponzi scheme, power law, precautionary principle, prediction markets, RAND corporation, Ray Kurzweil, Recombinant DNA, reversible computing, Richard Feynman, Ronald Reagan, scientific worldview, Singularitarianism, social intelligence, South China Sea, strong AI, superintelligent machines, supervolcano, synthetic biology, technological singularity, technoutopianism, The Coming Technological Singularity, the long tail, The Turner Diaries, Tunguska event, twin studies, Tyler Cowen, uranium enrichment, Vernor Vinge, War on Poverty, Westphalian system, Y2K

Artificial Intelligence in global risk 335 We can also imagine a scenario that implies a n easy local strategy: • The first AI cannot by itself do catastrophic damage. • If even a single Friendly AI exists, that AI plus human institutions can fend off any number of unFriendly A Is. The easy scenario would hold if, for example, human institutions can reliably distinguish Friendly Ais from unFriendly ones, and give revocable power into the hands of Friendly Ais. Thus we could pick and choose our allies. The only requirement is that the Friendly AI problem must be solvable (as opposed to being completely beyond human ability) . Both of the above scenarios assume that the first AI (the first powerful, general AI) cannot by itself do global catastrophic damage.

…

In more colloquial terms, it would be nice if we knew how to build a nice AI. To describe the field of knowledge needed to address that challenge, I have proposed the term 'Friendly AI'. In addition to referring to a body oftechnique, ' Friendly AI' might also refer to the product of technique - an A I created with specified motivations. When I use the term Friendly in either sense, I capitalize it to avoid confusion with the intuitive sense of 'friendly'. One common reaction I encounter is for people to immediately declare that Friendly AI is an impossibility because any sufficiently powerful AI will be able to modify its own source code to break any constraints placed upon it.

…

If huge nanocomputers make it easier to develop AI without making it easier to solve the particular challenge of Friendliness, that is a negative interaction. Thus, all else being equal, I would greatly prefer that Friendly AI precede nanotechnology in the ordering of technological developments. If we confront the challenge of AI and succeed, we can call on Friendly AI to help us with nanotechnology. If we develop nanotechnology and survive, we still have the challenge of AI to deal with after that. Generally speaking, a success on Friendly AI should help solve nearly any other problem. Thus, if a technology makes AI neither easier nor harder, but carries with it a catastrophic risk, we should prefer all else being equal to first confront the challenge of AI.

Singularity Rising: Surviving and Thriving in a Smarter, Richer, and More Dangerous World by James D. Miller

23andMe, affirmative action, Albert Einstein, artificial general intelligence, Asperger Syndrome, barriers to entry, brain emulation, cloud computing, cognitive bias, correlation does not imply causation, crowdsourcing, Daniel Kahneman / Amos Tversky, David Brooks, David Ricardo: comparative advantage, Deng Xiaoping, en.wikipedia.org, feminist movement, Flynn Effect, friendly AI, hive mind, impulse control, indoor plumbing, invention of agriculture, Isaac Newton, John Gilmore, John von Neumann, knowledge worker, Larry Ellison, Long Term Capital Management, low interest rates, low skilled workers, Netflix Prize, neurotypical, Nick Bostrom, Norman Macrae, pattern recognition, Peter Thiel, phenotype, placebo effect, prisoner's dilemma, profit maximization, Ray Kurzweil, recommendation engine, reversible computing, Richard Feynman, Rodney Brooks, Silicon Valley, Singularitarianism, Skype, statistical model, Stephen Hawking, Steve Jobs, sugar pill, supervolcano, tech billionaire, technological singularity, The Coming Technological Singularity, the scientific method, Thomas Malthus, transaction costs, Turing test, twin studies, Vernor Vinge, Von Neumann architecture

The Singularity Institute works toward the only goal I consider worthy of charitable dollars: increasing the survival prospects of mankind. Anna Salamon of the Singularity Institute did a credible back-of-the-envelope calculation showing that, based on some reasonable estimates of the effectiveness of friendly AI research and the harm of an unfriendly Singularity, donating one dollar to research on friendly AI will on average save over one life because slightly decreasing the odds that the seven billion current inhabitants of Earth will die yields you a huge average expected benefit.105 This expected benefit goes way up if you factor in people who are not yet born.

…

Within a year, we will probably have the technical ability to activate a seed AI, but once the Chinese threat has been annihilated, our team will have no reason to hurry and could take a decade to fine-tune their seed AI. If we delay, any intelligence explosion we eventually create will have an extremely high probability of yielding a friendly AI. Some people on our team think that, given another decade, they will be able to mathematically prove that the seed AI will turn into a friendly ultra-AI. A friendly AI would allow trillions and trillions of people to eventually live their lives, and mankind and our descendants could survive to the end of the universe in utopia. In contrast, an unfriendly AI would destroy us. I have decided to make the survival of humanity my priority.

…

The firm, therefore, wouldn’t be able to raise start-up capital from small, self-interested investors. So now pretend that at the time the firm tries to raise capital there is a well-developed theory of friendly AI, which provides programmers with a framework for creating AI that is extremely likely to be well disposed toward humanity and create a utopia if it undergoes an intelligence explosion. To raise funds from self-interested investors, an AI-building firm would need to pick a research and development path that would make it difficult for the firm ever to use the friendly AI framework. Unfortunately, this means that any intelligence explosion the firm unintentionally brings about would be less likely to be utopian than if the firm had used the friendly framework.

Rationality: From AI to Zombies by Eliezer Yudkowsky

Albert Einstein, Alfred Russel Wallace, anthropic principle, anti-pattern, anti-work, antiwork, Arthur Eddington, artificial general intelligence, availability heuristic, backpropagation, Bayesian statistics, behavioural economics, Berlin Wall, Boeing 747, Build a better mousetrap, Cass Sunstein, cellular automata, Charles Babbage, cognitive bias, cognitive dissonance, correlation does not imply causation, cosmological constant, creative destruction, Daniel Kahneman / Amos Tversky, dematerialisation, different worldview, discovery of DNA, disinformation, Douglas Hofstadter, Drosophila, Eddington experiment, effective altruism, experimental subject, Extropian, friendly AI, fundamental attribution error, Great Leap Forward, Gödel, Escher, Bach, Hacker News, hindsight bias, index card, index fund, Isaac Newton, John Conway, John von Neumann, Large Hadron Collider, Long Term Capital Management, Louis Pasteur, mental accounting, meta-analysis, mirror neurons, money market fund, Monty Hall problem, Nash equilibrium, Necker cube, Nick Bostrom, NP-complete, One Laptop per Child (OLPC), P = NP, paperclip maximiser, pattern recognition, Paul Graham, peak-end rule, Peter Thiel, Pierre-Simon Laplace, placebo effect, planetary scale, prediction markets, random walk, Ray Kurzweil, reversible computing, Richard Feynman, risk tolerance, Rubik’s Cube, Saturday Night Live, Schrödinger's Cat, scientific mainstream, scientific worldview, sensible shoes, Silicon Valley, Silicon Valley startup, Singularitarianism, SpaceShipOne, speech recognition, statistical model, Steve Jurvetson, Steven Pinker, strong AI, sunk-cost fallacy, technological singularity, The Bell Curve by Richard Herrnstein and Charles Murray, the map is not the territory, the scientific method, Turing complete, Turing machine, Tyler Cowen, ultimatum game, X Prize, Y Combinator, zero-sum game

But compulsion is not a property of arguments; it is a property of minds that process arguments. So the reason I’m arguing against the ghost isn’t just to make the point that (1) Friendly AI has to be explicitly programmed and (2) the laws of physics do not forbid Friendly AI. (Though of course I take a certain interest in establishing this.) I also wish to establish the notion of a mind as a causal, lawful, physical system in which there is no irreducible central ghost that looks over the neurons/code and decides whether they are good suggestions. (There is a concept in Friendly AI of deliberately programming an FAI to review its own source code and possibly hand it back to the programmers.

…

Good’s older term, “intelligence explosion,” to help distinguish his views from other futurist predictions, such as Ray Kurzweil’s exponential technological progress thesis.2 Technologies like smarter-than-human AI seem likely to result in large societal upheavals, for the better or for the worse. Yudkowsky coined the term “Friendly AI theory” to refer to research into techniques for aligning an AGI’s preferences with the preferences of humans. At this point, very little is known about when generally intelligent software might be invented, or what safety approaches would work well in such cases. Present-day autonomous AI can already be quite challenging to verify and validate with much confidence, and many current techniques are not likely to generalize to more intelligent and adaptive systems. “Friendly AI” is therefore closer to a menagerie of basic mathematical and philosophical questions than to a well-specified set of programming objectives.

…

But if you want to know why I might be reluctant to extend the graph of biological and economic growth over time, into the future and over the horizon of an AI that thinks at transistor speeds and invents self-replicating molecular nanofactories and improves its own source code, then there is my reason: you are drawing the wrong graph, and it should be optimization power in versus optimized product out, not optimized product versus time. * 146 Ghosts in the Machine People hear about Friendly AI and say—this is one of the top three initial reactions: “Oh, you can try to tell the AI to be Friendly, but if the AI can modify its own source code, it’ll just remove any constraints you try to place on it.” And where does that decision come from? Does it enter from outside causality, rather than being an effect of a lawful chain of causes that started with the source code as originally written? Is the AI the ultimate source of its own free will? A Friendly AI is not a selfish AI constrained by a special extra conscience module that overrides the AI’s natural impulses and tells it what to do.

Smarter Than Us: The Rise of Machine Intelligence by Stuart Armstrong

artificial general intelligence, brain emulation, effective altruism, Flash crash, friendly AI, machine translation, Nick Bostrom, shareholder value, Turing test

Some effort has been made to make the AI transition safer. Kudos must be given to Eliezer Yudkowsky and Nick Bostrom, who saw and understood the risks early on. Yudkowsky uses the term “Friendly AI” to describe an AI which does what we want even as it improves its own intelligence. In 2000 he cofounded an organization now called the Machine Intelligence Research Institute (MIRI), which holds math research workshops tackling open problems in Friendly AI theory. (MIRI also commissioned and published this book.) Meanwhile, Nick Bostrom founded the Future of Humanity Institute (FHI), a research group within the University of Oxford.

…

Are we not simply training the AI to give the correct answer in training situations? The whole approach is a constraint problem: in the space of possible AI minds, we are going to give priority to those minds that pass successfully through this training process and reassure us that they’re safe. Is there some quantifiable way of measuring how likely this is to produce a human-friendly AI at the end of it? If there isn’t, why are we putting any trust in it? These problems remain barely addressed, so though it is possible to imagine a safe AI being developed using the current approaches (or their descendants), it feels extremely unlikely. Hence we shouldn’t put our trust in the current crop of experts to solve the problem.

Physics of the Future: How Science Will Shape Human Destiny and Our Daily Lives by the Year 2100 by Michio Kaku

agricultural Revolution, AI winter, Albert Einstein, Alvin Toffler, Apollo 11, Asilomar, augmented reality, Bill Joy: nanobots, bioinformatics, blue-collar work, British Empire, Brownian motion, caloric restriction, caloric restriction, cloud computing, Colonization of Mars, DARPA: Urban Challenge, data science, delayed gratification, digital divide, double helix, Douglas Hofstadter, driverless car, en.wikipedia.org, Ford Model T, friendly AI, Gödel, Escher, Bach, Hans Moravec, hydrogen economy, I think there is a world market for maybe five computers, industrial robot, Intergovernmental Panel on Climate Change (IPCC), invention of movable type, invention of the telescope, Isaac Newton, John Markoff, John von Neumann, Large Hadron Collider, life extension, Louis Pasteur, Mahatma Gandhi, Mars Rover, Mars Society, mass immigration, megacity, Mitch Kapor, Murray Gell-Mann, Neil Armstrong, new economy, Nick Bostrom, oil shale / tar sands, optical character recognition, pattern recognition, planetary scale, postindustrial economy, Ray Kurzweil, refrigerator car, Richard Feynman, Rodney Brooks, Ronald Reagan, Search for Extraterrestrial Intelligence, Silicon Valley, Simon Singh, social intelligence, SpaceShipOne, speech recognition, stem cell, Stephen Hawking, Steve Jobs, synthetic biology, telepresence, The future is already here, The Wealth of Nations by Adam Smith, Thomas L Friedman, Thomas Malthus, trade route, Turing machine, uranium enrichment, Vernor Vinge, Virgin Galactic, Wall-E, Walter Mischel, Whole Earth Review, world market for maybe five computers, X Prize

But many scientists are leaning toward something called “friendly AI,” where we design our robots to be benign from the very beginning. Since we are the creators of these robots, we will design them, from the very start, to perform only useful and benevolent tasks. The term “friendly AI” was coined by Eliezer Yudkowsky, a founder of the Singularity Institute for Artificial Intelligence. Friendly AI is a bit different from Asimov’s laws, which are forced upon robots, perhaps against their will. (Asimov’s laws, imposed from the outside, could actually invite the robots to devise clever ways to circumvent them.) In friendly AI, by contrast, robots are free to murder and commit mayhem.

…

If this trend continues, then perhaps friendly AI could become a reality. In this scenario, it is the consumer sector and market forces that will eventually dominate robotics, so that there will be a vast commercial interest in investing in friendly AI. MERGING WITH ROBOTS In addition to friendly AI, there is also another option: merging with our creations. Instead of simply waiting for robots to surpass us in intelligence and power, we should try to enhance ourselves, becoming superhuman in the process. Most likely, I believe, the future will proceed with a combination of these two goals, i.e., building friendly AI and also enhancing ourselves.

…

Douglas Hofstadter has said, “It’s as if you took a lot of good food and some dog excrement and blended it all up so that you can’t possibly figure out what’s good or bad. It’s an intimate mixture of rubbish and good ideas, and it’s very hard to disentangle the two, because these are smart people; they’re not stupid.” No one knows how this will play out. But I think the most likely scenario is the following. MOST LIKELY SCENARIO: FRIENDLY AI First, scientists will probably take simple measures to ensure that robots are not dangerous. At the very least, scientists can put a chip in robot brains to automatically shut them off if they have murderous thoughts. In this approach, all intelligent robots will be equipped with a fail-safe mechanism that can be switched on by a human at any time, especially when a robot exhibits errant behavior.

When Computers Can Think: The Artificial Intelligence Singularity by Anthony Berglas, William Black, Samantha Thalind, Max Scratchmann, Michelle Estes

3D printing, Abraham Maslow, AI winter, air gap, anthropic principle, artificial general intelligence, Asilomar, augmented reality, Automated Insights, autonomous vehicles, availability heuristic, backpropagation, blue-collar work, Boston Dynamics, brain emulation, call centre, cognitive bias, combinatorial explosion, computer vision, Computing Machinery and Intelligence, create, read, update, delete, cuban missile crisis, David Attenborough, DeepMind, disinformation, driverless car, Elon Musk, en.wikipedia.org, epigenetics, Ernest Rutherford, factory automation, feminist movement, finite state, Flynn Effect, friendly AI, general-purpose programming language, Google Glasses, Google X / Alphabet X, Gödel, Escher, Bach, Hans Moravec, industrial robot, Isaac Newton, job automation, John von Neumann, Law of Accelerating Returns, license plate recognition, Mahatma Gandhi, mandelbrot fractal, natural language processing, Nick Bostrom, Parkinson's law, patent troll, patient HM, pattern recognition, phenotype, ransomware, Ray Kurzweil, Recombinant DNA, self-driving car, semantic web, Silicon Valley, Singularitarianism, Skype, sorting algorithm, speech recognition, statistical model, stem cell, Stephen Hawking, Stuxnet, superintelligent machines, technological singularity, Thomas Malthus, Turing machine, Turing test, uranium enrichment, Von Neumann architecture, Watson beat the top human players on Jeopardy!, wikimedia commons, zero day

This means that if the first real AGI can be made friendly, there will be no need to deal with any other AGI that might be developed. It only needs to be done correctly once. Or alternatively, humanity only has one opportunity to get it right. Friendly AGI research The current centre for research into friendly AI is the Machine Intelligence Research Institute (MIRI) in Berkeley, California, which was co-founded by Yudkowsky. They focus on providing mathematical solutions to the problem of producing friendly AI. In particular, they are researching how to formalize the definition of Coherent Extrapolated Volition so that it can be implemented in software. They also offer a series of very technical workshops to select mathematicians.

…

Thus it is only necessary to tame that first program in order to tame all hyper intelligences. There are many issues with this approach, not least of which is the fact that the military funds much of the research into artificial intelligence. They would want their money back if it turned out to be too friendly. The challenges of building a friendly AI will be discussed in detail in part III. Primary assertions and objections This book develops the following assertions:Computers will eventually become truly intelligent, and then become hyperintelligent. A computer based intelligence would have a world view very different from man’s world view.

…

They also offer a series of very technical workshops to select mathematicians. Another research group is the Future of Humanity Institute, in Oxford UK which is lead by Nick Bostrum who wrote the book SuperIntelligence. The institute has 13 staff, but it takes a wider view than just developing friendly AI. The International Conference on Artificial General Intelligence was founded in 2008 and includes some papers that address controlling an AGI. There is also an associated Journal. In total, these organizations are tiny. There are many more people working on the problem of trying to make software more intelligent than there are people working on the problem of making intelligent software safe for humanity.

Warnings by Richard A. Clarke

"Hurricane Katrina" Superdome, active measures, Albert Einstein, algorithmic trading, anti-communist, artificial general intelligence, Asilomar, Asilomar Conference on Recombinant DNA, Bear Stearns, behavioural economics, Bernie Madoff, Black Monday: stock market crash in 1987, carbon tax, cognitive bias, collateralized debt obligation, complexity theory, corporate governance, CRISPR, cuban missile crisis, data acquisition, deep learning, DeepMind, discovery of penicillin, double helix, Elon Musk, failed state, financial thriller, fixed income, Flash crash, forensic accounting, friendly AI, Hacker News, Intergovernmental Panel on Climate Change (IPCC), Internet of things, James Watt: steam engine, Jeff Bezos, John Maynard Keynes: Economic Possibilities for our Grandchildren, knowledge worker, Maui Hawaii, megacity, Mikhail Gorbachev, money market fund, mouse model, Nate Silver, new economy, Nicholas Carr, Nick Bostrom, nuclear winter, OpenAI, pattern recognition, personalized medicine, phenotype, Ponzi scheme, Ray Kurzweil, Recombinant DNA, Richard Feynman, Richard Feynman: Challenger O-ring, risk tolerance, Ronald Reagan, Sam Altman, Search for Extraterrestrial Intelligence, self-driving car, Silicon Valley, smart grid, statistical model, Stephen Hawking, Stuxnet, subprime mortgage crisis, tacit knowledge, technological singularity, The Future of Employment, the scientific method, The Signal and the Noise by Nate Silver, Tunguska event, uranium enrichment, Vernor Vinge, WarGames: Global Thermonuclear War, Watson beat the top human players on Jeopardy!, women in the workforce, Y2K

The ultimate problem: how to keep humanity from losing control of a machine of its own creation, to prevent artificial intelligence from becoming, in the words of James Barrat in the title of his 2013 book, Our Final Invention.6 A divisive figure, Yudkowsky is well known in academic circles and the Silicon Valley scene as the coiner of the term “friendly AI.” His thesis is simple, though his solution is not: if we are to have any hope against superintelligence, we need to code it properly from the beginning. The answer, Eliezer believes, is one of morality. AI must be programmed with a set of ethical codes that align with humanity’s. Though it is his life’s only work, Yudkowsky is pretty sure he will fail.

…

He envisions this as a system of checks and balances so that advanced AI growth is auditable and controllable, so that even as it continues to learn, advance, and reprogram itself, it will not evolve out of its own benign coding. Such preprogrammed measures will ensure that superintelligence will “behave as we intend even in the absence of immediate human supervision.”12 Eliezer calls this “friendly AI.” According to Yudkowsky, once AI gains the ability to broadly reprogram itself, it will be far too late to implement safeguards, so society needs to prepare now for the intelligence explosion. Yet this preparation is complicated by the sporadic and unpredictable nature of scientific advancement and the numerous secret efforts to create superintelligence around the world.

…

Eliezer told us that humanity’s best hope is to perhaps create one highly funded, highly secure, multilateral effort to develop a friendly superintelligence with himself (or perhaps another futurist he approves of) at the helm. The work of this massive global Manhattan Project would be explicitly “for the benefit of humanity internationally.” It simultaneously would ban, starve, or simply outpace other, less-well-thought-out efforts to develop superintelligence. Once created, this friendly AI would be unleashed to attack and destroy any competing efforts, ensuring that the only superintelligence in existence would help, not destroy, humankind. Yudkowsky rejects the idea that a superintelligence should, or could, be tailored to parochial national security interests, believing instead that any solution must be considered at the human species level.

The Singularity Is Near: When Humans Transcend Biology by Ray Kurzweil

additive manufacturing, AI winter, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, Albert Einstein, anthropic principle, Any sufficiently advanced technology is indistinguishable from magic, artificial general intelligence, Asilomar, augmented reality, autonomous vehicles, backpropagation, Benoit Mandelbrot, Bill Joy: nanobots, bioinformatics, brain emulation, Brewster Kahle, Brownian motion, business cycle, business intelligence, c2.com, call centre, carbon-based life, cellular automata, Charles Babbage, Claude Shannon: information theory, complexity theory, conceptual framework, Conway's Game of Life, coronavirus, cosmological constant, cosmological principle, cuban missile crisis, data acquisition, Dava Sobel, David Brooks, Dean Kamen, digital divide, disintermediation, double helix, Douglas Hofstadter, en.wikipedia.org, epigenetics, factory automation, friendly AI, functional programming, George Gilder, Gödel, Escher, Bach, Hans Moravec, hype cycle, informal economy, information retrieval, information security, invention of the telephone, invention of the telescope, invention of writing, iterative process, Jaron Lanier, Jeff Bezos, job automation, job satisfaction, John von Neumann, Kevin Kelly, Law of Accelerating Returns, life extension, lifelogging, linked data, Loebner Prize, Louis Pasteur, mandelbrot fractal, Marshall McLuhan, Mikhail Gorbachev, Mitch Kapor, mouse model, Murray Gell-Mann, mutually assured destruction, natural language processing, Network effects, new economy, Nick Bostrom, Norbert Wiener, oil shale / tar sands, optical character recognition, PalmPilot, pattern recognition, phenotype, power law, precautionary principle, premature optimization, punch-card reader, quantum cryptography, quantum entanglement, radical life extension, randomized controlled trial, Ray Kurzweil, remote working, reversible computing, Richard Feynman, Robert Metcalfe, Rodney Brooks, scientific worldview, Search for Extraterrestrial Intelligence, selection bias, semantic web, seminal paper, Silicon Valley, Singularitarianism, speech recognition, statistical model, stem cell, Stephen Hawking, Stewart Brand, strong AI, Stuart Kauffman, superintelligent machines, technological singularity, Ted Kaczynski, telepresence, The Coming Technological Singularity, Thomas Bayes, transaction costs, Turing machine, Turing test, two and twenty, Vernor Vinge, Y2K, Yogi Berra

Yudkowsky formed the Singularity Institute for Artificial Intelligence (SIAI) to develop "Friendly AI," intended to "create cognitive content, design features, and cognitive architectures that result in benevolence" before near-human or better-than-human Als become possible. SIAI has developed The SIAI Guidelines on Friendly AI: "Friendly AI," http://www.singinst.org/friendly/. Ben Goertzel and his Artificial General Intelligence Research Institute have also examined issues related to developing friendly AI; his current focus is on developing the Novamente AI Engine, a set of learning algorithms and architectures.

…

Steve Bowman and Helit Barel, Weapons of Mass Destruction: The Terrorist Threat, Congressional Research Service Report for Congress, December 8, 1999, http://www.cnie.org/nle/crsreports/international/inter-75.pdf. 30. Eliezer S. Yudkowsky, "Creating Friendly AI 1.0, The Analysis and Design of Benevolent Goal Architectures" (2001), The Singularity Institute, http://www.singinst.org/CFAI/; Eliezer S. Yudkowsky, "What Is Friendly AI?" May 3, 2001, http://www.KurzweilAI.net/meme/frame.html?main=/articles/art0172.html. 31. Ted Kaczynski, "The Unabomber's Manifesto," May 14, 2001, http://www.KurzweilAI.netlmeme/frame.html?main=/articles/art0182.html. 32.

…

Intelligence is inherently impossible to control, so the various strategies that have been devised to control nanotechnology (for example, the "broadcast architecture" described below) won't work for strong AI. There have been discussions and proposals to guide AI development toward what Eliezer Yudkowsky calls "friendly AI"30 (see the section "Protection from 'Unfriendly' Strong AI," p. 420). These are useful for discussion, but it is infeasible today to devise strategies that will absolutely ensure that future AI embodies human ethics and values. Returning to the Past? In his essay and presentations Bill Joy eloquently describes the plagues of centuries past and how new self-replicating technologies, such as mutant bioengineered pathogens and nanobots run amok, may bring back long-forgotten pestilence.

Heart of the Machine: Our Future in a World of Artificial Emotional Intelligence by Richard Yonck

3D printing, AI winter, AlphaGo, Apollo 11, artificial general intelligence, Asperger Syndrome, augmented reality, autism spectrum disorder, backpropagation, Berlin Wall, Bletchley Park, brain emulation, Buckminster Fuller, call centre, cognitive bias, cognitive dissonance, computer age, computer vision, Computing Machinery and Intelligence, crowdsourcing, deep learning, DeepMind, Dunning–Kruger effect, Elon Musk, en.wikipedia.org, epigenetics, Fairchild Semiconductor, friendly AI, Geoffrey Hinton, ghettoisation, industrial robot, Internet of things, invention of writing, Jacques de Vaucanson, job automation, John von Neumann, Kevin Kelly, Law of Accelerating Returns, Loebner Prize, Menlo Park, meta-analysis, Metcalfe’s law, mirror neurons, Neil Armstrong, neurotypical, Nick Bostrom, Oculus Rift, old age dependency ratio, pattern recognition, planned obsolescence, pneumatic tube, RAND corporation, Ray Kurzweil, Rodney Brooks, self-driving car, Skype, social intelligence, SoftBank, software as a service, SQL injection, Stephen Hawking, Steven Pinker, superintelligent machines, technological singularity, TED Talk, telepresence, telepresence robot, The future is already here, The Future of Employment, the scientific method, theory of mind, Turing test, twin studies, Two Sigma, undersea cable, Vernor Vinge, Watson beat the top human players on Jeopardy!, Whole Earth Review, working-age population, zero day

As we saw in the previous chapter, this would almost certainly be a truly alien intelligence. More importantly, there is little guarantee or even likelihood that its values, motives, and logic would align with our own. Multiple strategies have been proposed as a means of dealing with such a superintelligence, including variations on Asimov’s Three Laws, Yudkowsky’s Friendly AI theory, and Goertzel’s global AI nanny.1 Unfortunately, each is far from a foolproof strategy, and that should concern us. While there is considerable division on whether or not such an event could actually come to pass, there is probably a comparable level of disagreement about whether the outcome will be good or really, really bad.

…

Here, I’m speaking mentally, independent of their need for physical nourishment. 18. Goertzel, B. “The Mind-World Correspondence Principle (Toward a General Theory of General Intelligence).” IEEE Symposium on Computational Intelligence for Human-like Intelligence (CIHLI). 2013. Chapter 18 1. Yudkowsky, E. “Creating Friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures.” Machine Intelligence Research Institute, 2001; Goertzel, Ben. “Should Humanity Build a Global AI Nanny to Delay the Singularity Until It’s Better Understood?” Journal of consciousness studies 19.1-2: 1-2. 2012. 2. Also known as affective empathy. 3.

…

., 70, 74 Empatica Srl, 70, 74 endocrine system and emotions, 16, 17, 19, 219, 244, 248 ENIAC, 210 Enigma machine, 37 epinephrine, 16, 186, 221 epistemology, 35 EPOC headset, 213 Epstein, Robert, 141 Erewhon (Butler), 228 Ex Machina (Garland), 236–238 exponential change, 40–41 “”EyeJacking: See What I See, 57 Eyeris, 72, 144–145 F FaceSense, 60, 66 Faceshift, 75 Facial Action Coding System (FACS), 55–56 facial expression recognition, 54 facial recognition software, 144 facial tracking system, 59 Fairchild Superconductor, 38 false memories, 145–146 feral children, 257–258 fiction, AIs in dystopias, 228–229 human relationships with AI, 235–239 and the Machine Age, 227–229 problems with machine intelligence, 238–239 threat of intelligent machines, 230–234 flint knapping, 8–13, 23–24 Foresight Program, University of Houston, 171 Forkhead box protein P2, 15 Forster, E. M., 229 FOXP2, 15 Frankenstein (Shelley), 228 Freud, Sigmund, 96 Frewen, Cindy, 170–171 Friendly AI theory, 262 “Friendly Faces of Industry,” 171 Frubber (flesh rubber), 87, 113 functional magnetic resonance imaging (fMRI), 126–127 “The Future of Social Robotics,” 171 G galvactivator, 57–58 gaming community and designer emotions, 217 Garver, Carolyn, 113 Gazzaniga, Michael, 247 Geminoids, 100–101 general intelligence, 255 general morphological analysis (GMA), 165–166 General Problem Solver (1957), 37 Georgia Institute of Technology, 120 geriatric physiotherapy rehabilitation robots, 152 Gibson, William, 171 Gigolo Joe (mecha), 233–234 global AI nanny, 262 GMA.

What to Think About Machines That Think: Today's Leading Thinkers on the Age of Machine Intelligence by John Brockman

Adam Curtis, agricultural Revolution, AI winter, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, algorithmic trading, Anthropocene, artificial general intelligence, augmented reality, autism spectrum disorder, autonomous vehicles, backpropagation, basic income, behavioural economics, bitcoin, blockchain, bread and circuses, Charles Babbage, clean water, cognitive dissonance, Colonization of Mars, complexity theory, computer age, computer vision, constrained optimization, corporate personhood, cosmological principle, cryptocurrency, cuban missile crisis, Danny Hillis, dark matter, data science, deep learning, DeepMind, Demis Hassabis, digital capitalism, digital divide, digital rights, discrete time, Douglas Engelbart, driverless car, Elon Musk, Emanuel Derman, endowment effect, epigenetics, Ernest Rutherford, experimental economics, financial engineering, Flash crash, friendly AI, functional fixedness, global pandemic, Google Glasses, Great Leap Forward, Hans Moravec, hive mind, Ian Bogost, income inequality, information trail, Internet of things, invention of writing, iterative process, James Webb Space Telescope, Jaron Lanier, job automation, Johannes Kepler, John Markoff, John von Neumann, Kevin Kelly, knowledge worker, Large Hadron Collider, lolcat, loose coupling, machine translation, microbiome, mirror neurons, Moneyball by Michael Lewis explains big data, Mustafa Suleyman, natural language processing, Network effects, Nick Bostrom, Norbert Wiener, paperclip maximiser, pattern recognition, Peter Singer: altruism, phenotype, planetary scale, Ray Kurzweil, Recombinant DNA, recommendation engine, Republic of Letters, RFID, Richard Thaler, Rory Sutherland, Satyajit Das, Search for Extraterrestrial Intelligence, self-driving car, sharing economy, Silicon Valley, Skype, smart contracts, social intelligence, speech recognition, statistical model, stem cell, Stephen Hawking, Steve Jobs, Steven Pinker, Stewart Brand, strong AI, Stuxnet, superintelligent machines, supervolcano, synthetic biology, systems thinking, tacit knowledge, TED Talk, the scientific method, The Wisdom of Crowds, theory of mind, Thorstein Veblen, too big to fail, Turing machine, Turing test, Von Neumann architecture, Watson beat the top human players on Jeopardy!, We are as Gods, Y2K

The system can be used between human parties or interspecies parties, exactly because it’s not necessary to know, trust, or understand the other entity, just the code (the language of machines). Over time, trust can grow through reputation. Blockchain technology could be used to enforce friendly AI and mutually beneficial interspecies interaction. Someday, important transactions (like identity authentication and resource transfer) will be conducted on smart networks that require confirmation by independent consensus mechanisms, such that only bona fide transactions by reputable entities are executed. While perhaps not a full answer to the problem of enforcing friendly AI, decentralized smart networks like blockchains are a system of checks and balances helping to provide a more robust solution to situations of future uncertainty.

Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms: Proceedings of the Agi Workshop 2006 by Ben Goertzel, Pei Wang

AI winter, artificial general intelligence, backpropagation, bioinformatics, brain emulation, classic study, combinatorial explosion, complexity theory, computer vision, Computing Machinery and Intelligence, conceptual framework, correlation coefficient, epigenetics, friendly AI, functional programming, G4S, higher-order functions, information retrieval, Isaac Newton, Jeff Hawkins, John Conway, Loebner Prize, Menlo Park, natural language processing, Nick Bostrom, Occam's razor, p-value, pattern recognition, performance metric, precautionary principle, Ray Kurzweil, Rodney Brooks, semantic web, statistical model, strong AI, theory of mind, traveling salesman, Turing machine, Turing test, Von Neumann architecture, Y2K

In theory, all of these shortcuts should be equivalent to certain possible experience of the system, so they do not conflict with the principle that all the knowledge of NARS comes, directly or indirectly, from experience. One important issue to be handled through education is ethics. Unlike argued by some other researchers, NARS is not an attempt to design a “friendly AI”. As far as its initial state is concerned, the system is ethically neutral, since it can has any beliefs and goals. To make a NARS implementation “human friendly” means to give it certain beliefs and goals, which is an education mission, not a design mission. Even if something like Asimov’s “Three Laws of Robotics” is implanted into the system’s memory (which is possible), it still cannot fully control the system’s behaviors, due to the insufficiency of knowledge and resources in the system.

…

[Ben Goertzel]: Well I have a quite different opinion than that of Steve Grand in that I don’t think an amazing conceptual breakthrough on the level of the discovery of the quantum or curved 4D space-time, or something like that, is needed to create general intelligence. It might be needed to create provably stable friendly AI, like Eliezer Yudkowsky would like. I tend to think of the brain as a complex system composed of a bunch of evolved kluges for solving particular problems, which have been hacked together and adapted by evolution. I think if you assemble subcomponents solving the appropriate set of specialized problems, as well as a fairly weak general problem solver, and they are hooked together in a knowledge representation that works for all the components, with learning mechanisms that let each component learn from each other -- then you are going to have an intelligent mind that can be taught.

Overcomplicated: Technology at the Limits of Comprehension by Samuel Arbesman

algorithmic trading, Anthropocene, Anton Chekhov, Apple II, Benoit Mandelbrot, Boeing 747, Chekhov's gun, citation needed, combinatorial explosion, Computing Machinery and Intelligence, Danny Hillis, data science, David Brooks, digital map, discovery of the americas, driverless car, en.wikipedia.org, Erik Brynjolfsson, Flash crash, friendly AI, game design, Google X / Alphabet X, Googley, Hans Moravec, HyperCard, Ian Bogost, Inbox Zero, Isaac Newton, iterative process, Kevin Kelly, machine translation, Machine translation of "The spirit is willing, but the flesh is weak." to Russian and back, mandelbrot fractal, Minecraft, Neal Stephenson, Netflix Prize, Nicholas Carr, Nick Bostrom, Parkinson's law, power law, Ray Kurzweil, recommendation engine, Richard Feynman, Richard Feynman: Challenger O-ring, Second Machine Age, self-driving car, SimCity, software studies, statistical model, Steve Jobs, Steve Wozniak, Steven Pinker, Stewart Brand, superintelligent machines, synthetic biology, systems thinking, the long tail, Therac-25, Tyler Cowen, Tyler Cowen: Great Stagnation, urban planning, Watson beat the top human players on Jeopardy!, Whole Earth Catalog, Y2K

But that’s okay, because “We need not be intimidated by the accomplishments of metahuman science. We should always remember that the technologies that made metahumans possible were originally developed by humans, and they were no smarter than we.” See Luke Muehlhauser and Nick Bostrom, “Why We Need Friendly AI,” Think 36, no. 13 (Spring 2014), 41–47; and Ted Chiang, Stories of Your Life and Others (New York: Tor Books, 2003), 203. understand the most complex parts of the world: In many cases, we might even want to have a technology too complex to understand, because it means that it is sophisticated and powerful.

Human + Machine: Reimagining Work in the Age of AI by Paul R. Daugherty, H. James Wilson

3D printing, AI winter, algorithmic management, algorithmic trading, AlphaGo, Amazon Mechanical Turk, Amazon Robotics, augmented reality, autonomous vehicles, blockchain, business process, call centre, carbon footprint, circular economy, cloud computing, computer vision, correlation does not imply causation, crowdsourcing, data science, deep learning, DeepMind, digital twin, disintermediation, Douglas Hofstadter, driverless car, en.wikipedia.org, Erik Brynjolfsson, fail fast, friendly AI, fulfillment center, future of work, Geoffrey Hinton, Hans Moravec, industrial robot, Internet of things, inventory management, iterative process, Jeff Bezos, job automation, job satisfaction, knowledge worker, Lyft, machine translation, Marc Benioff, natural language processing, Neal Stephenson, personalized medicine, precision agriculture, Ray Kurzweil, recommendation engine, RFID, ride hailing / ride sharing, risk tolerance, robotic process automation, Rodney Brooks, Salesforce, Second Machine Age, self-driving car, sensor fusion, sentiment analysis, Shoshana Zuboff, Silicon Valley, Snow Crash, software as a service, speech recognition, tacit knowledge, telepresence, telepresence robot, text mining, the scientific method, uber lyft, warehouse automation, warehouse robotics

FUSION SKILL #4: Intelligent Interrogation Definition: Knowing how best to ask questions of AI, across levels of abstraction, to get the insights you need. How do you probe a massively complex system? How do you predict interactions between complex layers of data? People simply can’t do this on their own, so they must ask questions of their friendly AI. “Twin, how certain are you?” “Twin, what do you recommend?” At GE, maintenance professionals who have the skill of intelligent interrogation understand the capabilities and limitations of the AI system and know how to get the information they need to make an informed decision. The workers play to strengths and don’t duplicate the machine’s strengths.

Superintelligence: Paths, Dangers, Strategies by Nick Bostrom

agricultural Revolution, AI winter, Albert Einstein, algorithmic trading, anthropic principle, Anthropocene, anti-communist, artificial general intelligence, autism spectrum disorder, autonomous vehicles, backpropagation, barriers to entry, Bayesian statistics, bioinformatics, brain emulation, cloud computing, combinatorial explosion, computer vision, Computing Machinery and Intelligence, cosmological constant, dark matter, DARPA: Urban Challenge, data acquisition, delayed gratification, Demis Hassabis, demographic transition, different worldview, Donald Knuth, Douglas Hofstadter, driverless car, Drosophila, Elon Musk, en.wikipedia.org, endogenous growth, epigenetics, fear of failure, Flash crash, Flynn Effect, friendly AI, general purpose technology, Geoffrey Hinton, Gödel, Escher, Bach, hallucination problem, Hans Moravec, income inequality, industrial robot, informal economy, information retrieval, interchangeable parts, iterative process, job automation, John Markoff, John von Neumann, knowledge worker, Large Hadron Collider, longitudinal study, machine translation, megaproject, Menlo Park, meta-analysis, mutually assured destruction, Nash equilibrium, Netflix Prize, new economy, Nick Bostrom, Norbert Wiener, NP-complete, nuclear winter, operational security, optical character recognition, paperclip maximiser, pattern recognition, performance metric, phenotype, prediction markets, price stability, principal–agent problem, race to the bottom, random walk, Ray Kurzweil, recommendation engine, reversible computing, search costs, social graph, speech recognition, Stanislav Petrov, statistical model, stem cell, Stephen Hawking, Strategic Defense Initiative, strong AI, superintelligent machines, supervolcano, synthetic biology, technological singularity, technoutopianism, The Coming Technological Singularity, The Nature of the Firm, Thomas Kuhn: the structure of scientific revolutions, time dilation, Tragedy of the Commons, transaction costs, trolley problem, Turing machine, Vernor Vinge, WarGames: Global Thermonuclear War, Watson beat the top human players on Jeopardy!, World Values Survey, zero-sum game

The programmers may try to guard against this possibility by secretly monitoring the AI’s source code and the internal workings of its mind; but a smart-enough AI would realize that it might be under surveillance and adjust its thinking accordingly.2 The AI might find subtle ways of concealing its true capabilities and its incriminating intent.3 (Devising clever escape plans might, incidentally, also be a convergent strategy for many types of friendly AI, especially as they mature and gain confidence in their own judgments and capabilities. A system motivated to promote our interests might be making a mistake if it allowed us to shut it down or to construct another, potentially unfriendly AI.) We can thus perceive a general failure mode, wherein the good behavioral track record of a system in its juvenile stages fails utterly to predict its behavior at a more mature stage.

…

“Computational Genetics, Physiology, Metabolism, Neural Systems, Learning, Vision, and Behavior or PolyWorld: Life in a New Context.” In Proceedings of the Artificial Life III Conference, edited by C. G. Langton, 263–98. Santa Fe Institute Studies in the Sciences of Complexity. Reading, MA: Addison-Wesley. Yudkowsky, Eliezer. 2001. Creating Friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures. Machine Intelligence Research Institute, San Francisco, CA, June 15. Yudkowsky, Eliezer. 2002. “The AI-Box Experiment.” Retrieved January 15, 2012. Available at http://yudkowsky.net/singularity/aibox. Yudkowsky, Eliezer. 2004. Coherent Extrapolated Volition.

Thinking Machines: The Inside Story of Artificial Intelligence and Our Race to Build the Future by Luke Dormehl

"World Economic Forum" Davos, Ada Lovelace, agricultural Revolution, AI winter, Albert Einstein, Alexey Pajitnov wrote Tetris, algorithmic management, algorithmic trading, AlphaGo, Amazon Mechanical Turk, Apple II, artificial general intelligence, Automated Insights, autonomous vehicles, backpropagation, Bletchley Park, book scanning, borderless world, call centre, cellular automata, Charles Babbage, Claude Shannon: information theory, cloud computing, computer vision, Computing Machinery and Intelligence, correlation does not imply causation, crowdsourcing, deep learning, DeepMind, driverless car, drone strike, Elon Musk, Flash crash, Ford Model T, friendly AI, game design, Geoffrey Hinton, global village, Google X / Alphabet X, Hans Moravec, hive mind, industrial robot, information retrieval, Internet of things, iterative process, Jaron Lanier, John Markoff, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John von Neumann, Kickstarter, Kodak vs Instagram, Law of Accelerating Returns, life extension, Loebner Prize, machine translation, Marc Andreessen, Mark Zuckerberg, Menlo Park, Mustafa Suleyman, natural language processing, Nick Bostrom, Norbert Wiener, out of africa, PageRank, paperclip maximiser, pattern recognition, radical life extension, Ray Kurzweil, recommendation engine, remote working, RFID, scientific management, self-driving car, Silicon Valley, Skype, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, social intelligence, speech recognition, Stephen Hawking, Steve Jobs, Steve Wozniak, Steven Pinker, strong AI, superintelligent machines, tech billionaire, technological singularity, The Coming Technological Singularity, The Future of Employment, Tim Cook: Apple, Tony Fadell, too big to fail, traumatic brain injury, Turing machine, Turing test, Vernor Vinge, warehouse robotics, Watson beat the top human players on Jeopardy!

As an increasing amount of work is carried out involving autonomous AI weapons in war situations, work like Bringsjord’s is in high demand. In 2014, a multidisciplinary team at Tufts and Brown Universities, working alongside Bringsjord, received funding from the Office of Naval Research to explore the possibility of giving autonomous robots – potentially used in combat – a sense of right and wrong. While not exactly a ‘friendly AI’, this kind of computational morality would allow robots on the battlefield to make ethical decisions. Imagine, for instance, a robot medic that is transporting an injured soldier to a field hospital encounters another soldier with an injured leg. Weighing up the pros and cons of stopping its mission to administer aid, potentially administering pain relief by applying traction in the field, and other conundrums are all complex issues for a human to navigate – let alone a machine.

To Be a Machine: Adventures Among Cyborgs, Utopians, Hackers, and the Futurists Solving the Modest Problem of Death by Mark O'Connell

"World Economic Forum" Davos, 3D printing, Ada Lovelace, AI winter, Airbnb, Albert Einstein, AlphaGo, Amazon Picking Challenge, artificial general intelligence, Bletchley Park, Boston Dynamics, brain emulation, Charles Babbage, clean water, cognitive dissonance, computer age, cosmological principle, dark matter, DeepMind, disruptive innovation, double helix, Edward Snowden, effective altruism, Elon Musk, Extropian, friendly AI, global pandemic, Great Leap Forward, Hans Moravec, impulse control, income inequality, invention of the wheel, Jacques de Vaucanson, John von Neumann, knowledge economy, Law of Accelerating Returns, Lewis Mumford, life extension, lifelogging, Lyft, Mars Rover, means of production, military-industrial complex, Nick Bostrom, Norbert Wiener, paperclip maximiser, Peter Thiel, profit motive, radical life extension, Ray Kurzweil, RFID, San Francisco homelessness, self-driving car, sharing economy, Silicon Valley, Silicon Valley billionaire, Silicon Valley ideology, Singularitarianism, Skype, SoftBank, Stephen Hawking, Steve Wozniak, superintelligent machines, tech billionaire, technological singularity, technoutopianism, TED Talk, The Coming Technological Singularity, Travis Kalanick, trickle-down economics, Turing machine, uber lyft, Vernor Vinge

“If you consider balancing the interests of future people against those who already exist, reducing the probability of a major future catastrophe can be a very high-impact decision. If you succeed in avoiding an event that might wipe out all of future humanity, that clearly exceeds any good you might do for people currently living.” The Future of Life Institute was less focused than MIRI on the mathematical arcana of how a “friendly AI” might be engineered. The group, she said, functioned as “the outreach arm of this cluster of organizations,” raising awareness about the seriousness of this problem. It was not the attention of the media or the general public for which FLI was campaigning, Viktoriya said, but rather that of AI researchers themselves, a constituency in which the idea of existential risk was only just beginning to be taken seriously.

12 Bytes: How We Got Here. Where We Might Go Next by Jeanette Winterson

"Margaret Hamilton" Apollo, "World Economic Forum" Davos, 3D printing, Ada Lovelace, Airbnb, Albert Einstein, Alignment Problem, Amazon Mechanical Turk, Anthropocene, Apollo 11, Apple's 1984 Super Bowl advert, artificial general intelligence, Asilomar, augmented reality, autonomous vehicles, basic income, Big Tech, bitcoin, Bletchley Park, blockchain, Boston Dynamics, call centre, Cambridge Analytica, Capital in the Twenty-First Century by Thomas Piketty, cashless society, Charles Babbage, computer age, Computing Machinery and Intelligence, coronavirus, COVID-19, CRISPR, cryptocurrency, dark matter, Dava Sobel, David Graeber, deep learning, deskilling, digital rights, discovery of DNA, Dominic Cummings, Donald Trump, double helix, driverless car, Elon Musk, fake news, flying shuttle, friendly AI, gender pay gap, global village, Grace Hopper, Gregor Mendel, hive mind, housing crisis, Internet of things, Isaac Newton, Jacquard loom, James Hargreaves, Jeff Bezos, Johannes Kepler, John von Neumann, Joseph-Marie Jacquard, Kickstarter, Large Hadron Collider, life extension, lockdown, lone genius, Mark Zuckerberg, means of production, microdosing, more computing power than Apollo, move fast and break things, natural language processing, Nick Bostrom, Norbert Wiener, off grid, OpenAI, operation paperclip, packet switching, Peter Thiel, pink-collar, Plato's cave, public intellectual, QAnon, QWERTY keyboard, Ray Kurzweil, rewilding, ride hailing / ride sharing, Rutger Bregman, Sam Altman, self-driving car, sharing economy, Sheryl Sandberg, Shoshana Zuboff, Silicon Valley, Skype, Snapchat, SoftBank, SpaceX Starlink, speech recognition, spinning jenny, stem cell, Stephen Hawking, Steve Bannon, Steve Jobs, Steven Levy, Steven Pinker, superintelligent machines, surveillance capitalism, synthetic biology, systems thinking, tech billionaire, tech worker, TED Talk, telepresence, telepresence robot, TikTok, trade route, Turing test, universal basic income, Virgin Galactic, Watson beat the top human players on Jeopardy!, women in the workforce, Y Combinator

Same with that Roomba vacuum cleaner. It has a map of your house. In fact, the makers of Roomba, iRobot, are developing the idea of a house as an inside-out robot – in other words, a robot you live inside. It will come … So, I don’t want to get into the question of surveillance here. We know the price of home-friendly AI: it’s our personal data. Anyone who has voice-activated Alexa is being listened to all of the time. We’re told this isn’t like having the KGB in the kitchen. It’s ‘anonymised background sound’. How we deal with data tracking is a separate issue – a bot in the house doesn’t solve that problem, or make things worse, it’s just part of the system we’re all buying into.

Possible Minds: Twenty-Five Ways of Looking at AI by John Brockman

AI winter, airport security, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, Alignment Problem, AlphaGo, artificial general intelligence, Asilomar, autonomous vehicles, basic income, Benoit Mandelbrot, Bill Joy: nanobots, Bletchley Park, Buckminster Fuller, cellular automata, Claude Shannon: information theory, Computing Machinery and Intelligence, CRISPR, Daniel Kahneman / Amos Tversky, Danny Hillis, data science, David Graeber, deep learning, DeepMind, Demis Hassabis, easy for humans, difficult for computers, Elon Musk, Eratosthenes, Ernest Rutherford, fake news, finite state, friendly AI, future of work, Geoffrey Hinton, Geoffrey West, Santa Fe Institute, gig economy, Hans Moravec, heat death of the universe, hype cycle, income inequality, industrial robot, information retrieval, invention of writing, it is difficult to get a man to understand something, when his salary depends on his not understanding it, James Watt: steam engine, Jeff Hawkins, Johannes Kepler, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John von Neumann, Kevin Kelly, Kickstarter, Laplace demon, Large Hadron Collider, Loebner Prize, machine translation, market fundamentalism, Marshall McLuhan, Menlo Park, military-industrial complex, mirror neurons, Nick Bostrom, Norbert Wiener, OpenAI, optical character recognition, paperclip maximiser, pattern recognition, personalized medicine, Picturephone, profit maximization, profit motive, public intellectual, quantum cryptography, RAND corporation, random walk, Ray Kurzweil, Recombinant DNA, Richard Feynman, Rodney Brooks, self-driving car, sexual politics, Silicon Valley, Skype, social graph, speech recognition, statistical model, Stephen Hawking, Steven Pinker, Stewart Brand, strong AI, superintelligent machines, supervolcano, synthetic biology, systems thinking, technological determinism, technological singularity, technoutopianism, TED Talk, telemarketer, telerobotics, The future is already here, the long tail, the scientific method, theory of mind, trolley problem, Turing machine, Turing test, universal basic income, Upton Sinclair, Von Neumann architecture, Whole Earth Catalog, Y2K, you are the product, zero-sum game

A superintelligent AGI will be extremely good at accomplishing its goals, and if those goals aren’t aligned with ours, we’re in trouble. People don’t think twice about flooding anthills to build hydroelectric dams, so let’s not place humanity in the position of those ants. Most researchers argue that if we end up creating superintelligence, we should make sure it’s what AI-safety pioneer Eliezer Yudkowsky has termed “friendly AI”—AI whose goals are in some deep sense beneficial. The moral question of what these goals should be is just as urgent as the technical questions about goal alignment. For example, what sort of society are we hoping to create, where we find meaning and purpose in our lives even though we, strictly speaking, aren’t needed?

Pandora's Brain by Calum Chace

AI winter, Any sufficiently advanced technology is indistinguishable from magic, artificial general intelligence, Bletchley Park, brain emulation, Extropian, friendly AI, hive mind, lateral thinking, machine translation, mega-rich, Nick Bostrom, precautionary principle, Ray Kurzweil, self-driving car, Silicon Valley, Singularitarianism, Skype, speech recognition, stealth mode startup, Stephen Hawking, strong AI, technological singularity, theory of mind, Turing test, Wall-E

The European Commission was proposing an autonomous international agency with exceptional powers of inspection and restraint in order to stop rogue governments and individuals from doing any research that could lead to the creation of an AGI. This was a step too far for him, as it was for most Americans. ‘That game is far from over. And I wouldn’t be surprised if a coalition of the institutes which are researching human-friendly AI algorithms announced a breakthrough this spring. There’s been a lot of unusually cordial traffic between several of the bigger US-based ones during the winter. Anyway, fortunately, none of that affects what we’re doing here at the Foundation. We’ve managed to put clear blue water between brain preservation research and AI research in the public’s mind.’

What We Owe the Future: A Million-Year View by William MacAskill

Ada Lovelace, agricultural Revolution, Albert Einstein, Alignment Problem, AlphaGo, artificial general intelligence, Bartolomé de las Casas, Bletchley Park, British Empire, Brownian motion, carbon footprint, carbon tax, charter city, clean tech, coronavirus, COVID-19, cuban missile crisis, decarbonisation, deep learning, DeepMind, Deng Xiaoping, different worldview, effective altruism, endogenous growth, European colonialism, experimental subject, feminist movement, framing effect, friendly AI, global pandemic, GPT-3, hedonic treadmill, Higgs boson, income inequality, income per capita, Indoor air pollution, Intergovernmental Panel on Climate Change (IPCC), Isaac Newton, Islamic Golden Age, iterative process, Jeff Bezos, job satisfaction, lab leak, Lao Tzu, Large Hadron Collider, life extension, lockdown, long peace, low skilled workers, machine translation, Mars Rover, negative emissions, Nick Bostrom, nuclear winter, OpenAI, Peter Singer: altruism, Peter Thiel, QWERTY keyboard, Robert Gordon, Rutger Bregman, Sam Altman, seminal paper, Shenzhen special economic zone , Shenzhen was a fishing village, Silicon Valley, special economic zone, speech recognition, Stanislav Petrov, stem cell, Steven Pinker, strong AI, synthetic biology, total factor productivity, transatlantic slave trade, Tyler Cowen, William MacAskill, women in the workforce, working-age population, World Values Survey, Y Combinator

For a recent conceptual overview of the field, see Christiano (2020). Different authors have used different ways of conceptualizing the challenge of creating AI systems that are more capable than humans but lead to desirable outcomes when deployed. Yudkowsky (2001) described the issue as how to create “friendly AI”; Bostrom as the “control problem” (Bostrom 2014b, Chapter 9). (See also Christiano 2016, 2018a; Gabriel 2020; Hubinger 2020.) 83. What about worlds that are controlled by AIs but without significant lock-in? We can, for example, imagine a society of AIs that reflect, reason morally, and remain open to changing their minds.

Ways of Being: Beyond Human Intelligence by James Bridle

Ada Lovelace, Airbnb, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, Anthropocene, Any sufficiently advanced technology is indistinguishable from magic, autonomous vehicles, behavioural economics, Benoit Mandelbrot, Berlin Wall, Big Tech, Black Lives Matter, blockchain, Californian Ideology, Cambridge Analytica, carbon tax, Charles Babbage, cloud computing, coastline paradox / Richardson effect, Computing Machinery and Intelligence, corporate personhood, COVID-19, cryptocurrency, DeepMind, Donald Trump, Douglas Hofstadter, Elon Musk, experimental subject, factory automation, fake news, friendly AI, gig economy, global pandemic, Gödel, Escher, Bach, impulse control, James Bridle, James Webb Space Telescope, John von Neumann, Kickstarter, Kim Stanley Robinson, language acquisition, life extension, mandelbrot fractal, Marshall McLuhan, microbiome, music of the spheres, negative emissions, Nick Bostrom, Norbert Wiener, paperclip maximiser, pattern recognition, peer-to-peer, planetary scale, RAND corporation, random walk, recommendation engine, self-driving car, SETI@home, shareholder value, Silicon Valley, Silicon Valley ideology, speech recognition, statistical model, surveillance capitalism, techno-determinism, technological determinism, technoutopianism, the long tail, the scientific method, The Soul of a New Machine, theory of mind, traveling salesman, trolley problem, Turing complete, Turing machine, Turing test, UNCLOS, undersea cable, urban planning, Von Neumann architecture, wikimedia commons, zero-sum game

That book is often cited as a sophisticated defence of the right of human beings to use animals for their benefit, but when Fox realized that the same arguments could be used by an alien species to enslave humans, he repudiated it and moved to a position supporting animal rights. 20. Calls to develop ‘friendly’ AI originate with the Machine Intelligence Research Institute, co-founded by Eliezer Yudkowsky, who also created the LessWrong community, but have been championed by some of the founders of academic and practical AI studies. See, for example, Stuart Russell and Peter Norvig’s Artificial Intelligence: A Modern Approach, 3rd edn (Harlow: Pearson Education, 2016), the standard textbook on the subject, which cites Yudkowsky’s concerns about AI safety. 21.

Hands-On Machine Learning With Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems by Aurélien Géron

AlphaGo, Amazon Mechanical Turk, Anton Chekhov, backpropagation, combinatorial explosion, computer vision, constrained optimization, correlation coefficient, crowdsourcing, data science, deep learning, DeepMind, don't repeat yourself, duck typing, Elon Musk, en.wikipedia.org, friendly AI, Geoffrey Hinton, ImageNet competition, information retrieval, iterative process, John von Neumann, Kickstarter, machine translation, natural language processing, Netflix Prize, NP-complete, OpenAI, optical character recognition, P = NP, p-value, pattern recognition, pull request, recommendation engine, self-driving car, sentiment analysis, SpamAssassin, speech recognition, stochastic process

With two (or more) parents, it is called sexual reproduction. An offspring’s genome (in this case a set of policy parameters) is randomly composed of parts of its parents’ genomes. 8 OpenAI is a nonprofit artificial intelligence research company, funded in part by Elon Musk. Its stated goal is to promote and develop friendly AIs that will benefit humanity (rather than exterminate it). 9 “Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning,” R. Williams (1992). 10 We already did something similar in Chapter 11 when we discussed Gradient Clipping: we first computed the gradients, then we clipped them, and finally we applied the clipped gradients. 11 “A Markovian Decision Process,” R.

The Transhumanist Reader by Max More, Natasha Vita-More

"World Economic Forum" Davos, 23andMe, Any sufficiently advanced technology is indistinguishable from magic, artificial general intelligence, augmented reality, Bill Joy: nanobots, bioinformatics, brain emulation, Buckminster Fuller, cellular automata, clean water, cloud computing, cognitive bias, cognitive dissonance, combinatorial explosion, Computing Machinery and Intelligence, conceptual framework, Conway's Game of Life, cosmological principle, data acquisition, discovery of DNA, Douglas Engelbart, Drosophila, en.wikipedia.org, endogenous growth, experimental subject, Extropian, fault tolerance, Flynn Effect, Francis Fukuyama: the end of history, Frank Gehry, friendly AI, Future Shock, game design, germ theory of disease, Hans Moravec, hypertext link, impulse control, index fund, John von Neumann, joint-stock company, Kevin Kelly, Law of Accelerating Returns, life extension, lifelogging, Louis Pasteur, Menlo Park, meta-analysis, moral hazard, Network effects, Nick Bostrom, Norbert Wiener, pattern recognition, Pepto Bismol, phenotype, positional goods, power law, precautionary principle, prediction markets, presumed consent, Project Xanadu, public intellectual, radical life extension, Ray Kurzweil, reversible computing, RFID, Ronald Reagan, scientific worldview, silicon-based life, Singularitarianism, social intelligence, stem cell, stochastic process, superintelligent machines, supply-chain management, supply-chain management software, synthetic biology, systems thinking, technological determinism, technological singularity, Ted Nelson, telepresence, telepresence robot, telerobotics, the built environment, The Coming Technological Singularity, the scientific method, The Wisdom of Crowds, transaction costs, Turing machine, Turing test, Upton Sinclair, Vernor Vinge, Von Neumann architecture, VTOL, Whole Earth Review, women in the workforce, zero-sum game

It’s not as though our choices are “life goes on exactly as is” versus “life as it is plus super-powerful AI.” Various technologies are advancing rapidly and society is changing accordingly, and the rate of advancement of AGI is just one aspect in the mix. I don’t think there are any magic bullets to resolve the dilemmas of AGI ethics. There will almost surely be no provably Friendly AI, in spite of the wishes of Eliezer Yudkowsky (2008) and some others. Nor, in my best guess, will there be an Artilect War in which pro-AGI and anti-AGI forces battle to the death with doomsday machines, as Hugo de Garis (2005) foresees. But I don’t pretend to be able to see exactly what the outcome will be.