social graph

101 results back to index


pages: 255 words: 76,495

The Facebook era: tapping online social networks to build better products, reach new audiences, and sell more stuff by Clara Shih

business process, call centre, Clayton Christensen, cloud computing, commoditize, conceptual framework, corporate governance, crowdsourcing, glass ceiling, jimmy wales, Mark Zuckerberg, Metcalfe’s law, Network effects, pets.com, pre–internet, rolodex, semantic web, sentiment analysis, Silicon Valley, Silicon Valley startup, social graph, social web, software as a service, Tony Hsieh, web application

Annoying floating banners that are irrelevant to us block the screen showing the online article we are trying to read. But all hope is not lost. With the online social graph, there might be an opportunity for the first time to align what publishers and advertisers want to show with what users want to see (see Figure 2.1). From the Library of Kerri Ross 30 Pa r t I A B r i e f H i s to r y o f S o c i a l M e d i a Amount of Noise HIGH LOW Distribution Capacity Social Graph Media Internet Media HIGH PC Media LOW Traditional (Nondigital) Media Figure 2.1 Prior to the social graph, greater distribution resulted in more junk. With the social graph, we can use our friends as filters for finding the right content and data at the right time. For push content—that is, content actively pushed out to Internet users—people on Facebook and Twitter are already seeing socially filtered feeds, notifications, and SMS messages about Web pages, articles, photos, and blog posts—that is, content created or recommended by friends.

Last but not least, research shows that weak ties, rather than your most intimate circle of friends and family, tend to carry the greatest amount of social capital in business contexts. It is precisely in weak ties where Facebook, Twitter, and other social networking services excel. Welcome to the Facebook Era We are witnessing a historic movement around the online social graph—that is, the map of every person on the Internet and how they are connected. It is the World Wide Web of people, a reflection and extension of the offline social graph—the friends, family members, colleagues, mentors, classmates, neighbors, and acquaintances who are important to us, who help shape us, and for whom we live. The online social graph empowers us to be better, more effective, more efficient, and more fulfilled doing what is inherent to our nature—communicating who we are, and transacting and interacting with others across the Web. Data from social networks, such as where people are from, what they are interested in, and who their friends are, with the right privacy controls in place can then be implicitly or explicitly mined to make business interactions more tailored, personal, and precise.

In every case, regardless of prior competitive dynamics, businesses that understand and appropriately adopt the technology win, while those that fail to do so lose. In the 1970s, this was mainframe computing. In the 1980s, it was the PC. In the 1990s, it was the Internet. And today, it is the online social graph (see Figure 1.1). A Mainframe 1970s PC C 1980s Internet 1990s Social Networking Today Figure 1.1 Every decade since the advent of computing, a new wave of technology sweeps across the business landscape. But what exactly is the online social graph? Well, it is the World Wide Web of people—a map being constructed by social networking sites, such as Facebook, LinkedIn, and Hoover’s Connect, of every person on the Internet and how they are interlinked. The social graph is for people what the World Wide Web is for hyperlinked Web pages: that is, for organizing, filtering, and association. Now that all of our machines and content pages are connected, the next digital revolution will be in capturing and using information about how we as individuals are connected.


pages: 475 words: 134,707

The Hype Machine: How Social Media Disrupts Our Elections, Our Economy, and Our Health--And How We Must Adapt by Sinan Aral

Airbnb, Albert Einstein, Any sufficiently advanced technology is indistinguishable from magic, augmented reality, Bernie Sanders, bitcoin, carbon footprint, Cass Sunstein, computer vision, coronavirus, correlation does not imply causation, COVID-19, Covid-19, crowdsourcing, cryptocurrency, death of newspapers, disintermediation, Donald Trump, Drosophila, Edward Snowden, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, experimental subject, facts on the ground, Filter Bubble, global pandemic, hive mind, illegal immigration, income inequality, Kickstarter, knowledge worker, longitudinal study, low skilled workers, Lyft, Mahatma Gandhi, Mark Zuckerberg, Menlo Park, meta analysis, meta-analysis, Metcalfe’s law, mobile money, move fast and break things, move fast and break things, multi-sided market, Nate Silver, natural language processing, Network effects, performance metric, phenotype, recommendation engine, Robert Bork, Robert Shiller, Robert Shiller, Second Machine Age, sentiment analysis, shareholder value, skunkworks, Snapchat, social graph, social intelligence, social software, social web, statistical model, stem cell, Stephen Hawking, Steve Jobs, Telecommunications Act of 1996, The Chicago School, The Wisdom of Crowds, theory of mind, Tim Cook: Apple, Uber and Lyft, uber lyft, WikiLeaks, Yogi Berra

When consumers on one network can easily connect with consumers on another network, the value from network effects is diminished and competition is enhanced, giving more opportunity to new entrants and stripping away power from incumbents. The idea behind social graph portability is that consumers would own their social graphs. When they switched to a new social networking service, their connections and their friends’ identities could be transferred to facilitate exchanges on the new network as well as across networks. That said, a number of technical difficulties arise when you attempt to apply the logic of telephone number portability to social graph portability. First, social graphs and phone numbers are different. A social graph is a complex web of interconnections over which property rights are difficult to assign. Defining the property right at the level of the graph makes its management complex and, in a dynamic environment in which social graph connections change, difficult to maintain over time.

The Hype Machine structures our reality by building on what Facebook, LinkedIn, and others call the social graph. I’ve been studying the structure and function of the social graph for twenty years and am fascinated by its novel mathematical properties, such as the fact that, on average, most people’s friends have more friends than they do, a statistical regularity called the friendship paradox that was first discovered by Scott Feld in 1991, and that I will return to later in this book. Figure 3.4 The global Facebook social network in 2010. No map is shown. The contours of the continents emerge from the network connections themselves. Two regularities of the social graph directly influence what we are experiencing on the Hype Machine today. First, it’s clustered more than one would expect by chance, meaning we form dense clusters of people that are highly connected within the clusters, much more than we are connected across the clusters.

Among those introduced online, the proportion who met through common friends has declined over time, which means algorithms are displacing friends and family in guiding the formation of our romantic relationships. So thinking about how friend suggestions steer the evolution of the social graph, and in particular the degree of clustering and homophily in the Hype Machine, is fundamental to understanding political polarization, social gridlock, and the spread of misinformation and hate speech online. It may even explain the direction of human evolution, through its influence on our romantic relationships. But before I get to the process by which the Hype Machine’s intelligence directs the evolution of the network and the flow of information through it, it’s worthwhile to examine the structure of the Hype Machine’s digital network in more detail. The Hype Machine’s Social Graph Facebook studied its social graph in 2011. My friends and colleagues Johan Ugander, Brian Karrer, Lars Backstrom, and Cameron Marlow, who all worked at Facebook at the time (Brian and Lars still do), wrote a paper they called “The Anatomy of the Facebook Social Graph,” in which they studied “the entire social network of active members of Facebook in May 2011, a network then comprised of 721 million active users.”


pages: 451 words: 103,606

Machine Learning for Hackers by Drew Conway, John Myles White

call centre, centre right, correlation does not imply causation, Debian, Erdős number, Nate Silver, natural language processing, Netflix Prize, p-value, pattern recognition, Paul Erdős, recommendation engine, social graph, SpamAssassin, statistical model, text mining, the scientific method, traveling salesman

q=@EMAIL_ADDRESS&fme=1&pretty=1 If you replace the EMAIL_ADDRESS with a proper address, the API will return raw JSON to the browser window, which you can explore to see how much of your social graph Google is already storing! If you have a Twitter account registered to the email address you entered, you will likely also notice that your Twitter page’s URL shows up as one of the social graph services that resolves to the email address. Twitter is one of the many social networking sites the SGA crawls to store public social graph data. Therefore, we can use the SGA to query the social network of a specific user and then build out the network. The primary advantage of using the SGA is that unlike the pittance of hourly queries that Twitter provides, the SGA allows 50,000 queries per day. Even at Twitter’s scale, this will be more than enough queries to build the social graph for the vast majority of users. Of course, if you are Ashton Kutcher or Tim O’Reilly, the method described in this case study will not work.

Within the context of Twitter, this can give us information about the different social groups that a given user belongs to. Finally, because this book is about machine learning, we will build our own “who to follow” recommendation engine using the structure of Twitter’s social graph. We will try to avoid using jargon or niche academic terms to describe what we are doing here. There are, however, some terms that are worth using and learning for the purposes of this chapter. We have just introduced the term ego-network to describe a type of graph. As we will be referring to ego-networks many more times going forward, it will be useful to define this term. An ego-network always refers to the structure of a social graph immediately surrounding a single node in a network. Specifically, an ego-network is the subset of a network induced by a seed (or ego) and its neighbors, i.e., those nodes directly connected to the seed.

These labels can also be much more complex than a simple binary relationship. These labels could be weights indicating the strength or type of a relationship. It is important to consider the differences among these different graph types because the social graph data we will encounter online comes in many different forms. Variation in how the graphs are structured can affect how people use the service and, consequently, how we might analyze the data. Consider two popular social networking sites that vary in the class of graphs they use: Facebook and Twitter. Facebook is a massive, undirected social graph. Because “friending” requires approval, all edges imply a mutual friendship. This has caused Facebook to evolve as a relatively more closed network with dense local network structures rather than the massive central hubs of a more open service.


Mastering Structured Data on the Semantic Web: From HTML5 Microdata to Linked Open Data by Leslie Sikos

AGPL, Amazon Web Services, bioinformatics, business process, cloud computing, create, read, update, delete, Debian, en.wikipedia.org, fault tolerance, Firefox, Google Chrome, Google Earth, information retrieval, Infrastructure as a Service, Internet of things, linked data, natural language processing, openstreetmap, optical character recognition, platform as a service, search engine result page, semantic web, Silicon Valley, social graph, software as a service, SPARQL, text mining, Watson beat the top human players on Jeopardy!, web application, wikimedia commons

Social Media Applications Excellent examples for Big Data implementations on the Semantic Web are the social media graphs, such as the Facebook Social Graph, the Twitter Interest Graph, the Twitter Follow Graph, the LinkedIn Professional Graph, or the LinkedIn Economic Graph. 205 Chapter 8 ■ Big Data Applications Facebook Social Graph The Facebook Social Graph is the largest social graph in the world, containing tens of petabytes of structured data about approximately 1 billion users. Because every object is a graph node, and every relationship is a graph edge on the Facebook Social Graph (see Figure 8-4), any object can easily be accessed directly in the browser as a user and programmatically from Facebook apps. Figure 8-4. On the Facebook Social Graph, every object is a node and every connection is an edge In fact, the easy access of this vast user data is exploited well beyond Facebook, as the social connections and links of the Facebook Social Graph are also used by other social networking portals, such as Pinterest and Last.fm (social bootstrapping).

On the Facebook Social Graph, every object is a node and every connection is an edge In fact, the easy access of this vast user data is exploited well beyond Facebook, as the social connections and links of the Facebook Social Graph are also used by other social networking portals, such as Pinterest and Last.fm (social bootstrapping). Have you ever wondered how Facebook recommends friends? Using the edges of the Facebook Social Graph, it is straightforward to identify those people who have at least one friend in common (see Figure 8-5). 206 Chapter 8 ■ Big Data Applications Figure 8-5. The edges of the Facebook Social Graph make it possible to suggest people you may know The Facebook Graph API The Facebook Graph API is the core of the Facebook Platform, enabling developers to read data from and write data into Facebook user profiles. The Graph API represents the current state of the Facebook Social Graph through graph objects such as people, photos, events, and pages, as well as the connections between them, such as friend relationships, shared content, and photo tags.

The Graph API represents the current state of the Facebook Social Graph through graph objects such as people, photos, events, and pages, as well as the connections between them, such as friend relationships, shared content, and photo tags. In other words, the Graph API makes it possible to programmatically access user objects and connections from the Facebook Social Graph, which can be used for Facebook apps. The Graph API can not only query data but also post new stories, publish Open Graph stories, read information about a Facebook user, upload photos, update information in the Social Graph, and perform similar tasks used by Facebook apps. All the objects of the Facebook Social Graph (users, photo albums, photos, status messages, pages, etc.) have a unique identifier, which is a positive integer and makes it possible to refer to any node or edge. Originally, the Graph API provided data to applications exclusively in JSON.


pages: 518 words: 49,555

Designing Social Interfaces by Christian Crumlish, Erin Malone

A Pattern Language, Amazon Mechanical Turk, anti-pattern, barriers to entry, c2.com, carbon footprint, cloud computing, collaborative editing, creative destruction, crowdsourcing, en.wikipedia.org, Firefox, game design, ghettoisation, Howard Rheingold, hypertext link, if you build it, they will come, Merlin Mann, Nate Silver, Network effects, Potemkin village, recommendation engine, RFC: Request For Comment, semantic web, SETI@home, Skype, slashdot, social graph, social software, social web, source of truth, stealth mode startup, Stewart Brand, telepresence, The Wisdom of Crowds, web application

Building a network of connections is hard, and as more time goes by, becomes overly redundant as a user moves from site to site. Providing easy mechanisms for finding people and building their networks will encourage repeat use and prevent social-networking burnout. Portable Social Graph The easiest way to create a network upon joining a new site would be to bring your network with you. Although there are some contexts that may be very specialized and need only a small subset of people a user knows—for example, a fantasy sports site—for the most part, many current and future social sites are generalized enough that the network the user built on site A will be the same network of connections she wants on site B. The social graph, the network of people the user has built around herself (Figure 14-8), wants to be portable. There is a growing movement encouraging openness (http://bradfitz.com/socialgraph-problem/), just like with OpenID.

There is a growing movement encouraging openness (http://bradfitz.com/socialgraph-problem/), just like with OpenID. The idea is to create a data standard that allows users to easily bring their network from one site to another without all the work involved in finding people and adding them into the network at each site. ——continued Download at WoweBook.Com 360 Chapter 14: One of Us, One of Us Portable Social Graph Figure 14-8. The author’s social graph on Facebook as visualized by TouchGraph. The logistics are far from resolved, but as new sites are being developed, being aware of this initiative and designing new sites and using data structures that play nice will encourage interplay between sites and help the users out in the long run. Related patterns “Adding Friends” on page 361 “Sign-up or Registration” on page 45 As seen on Facebook (http://www.facebook.com) Flickr (http://www.flickr.com) LinkedIn (http://www.linkedin.com) myBlogLog (http://www.mybloglog.com) Plaxo (http://www.plaxo.com) Download at WoweBook.Com Relationships 361 Twitter (http://www.twitter.com) Upcoming (http//upcoming.yahoo.com) Adding Friends What A user has found people she knows on a social site and wants to add them to her circle of connections (Figure 14-9).

The idea is that as the Web becomes more social (that’s the word we’ve all converged on), there is an element of it that is read-write, that involves people writing and revising and responding to one another, not in a one-to-one or one-tomany fashion, but many-to-many. The problem with using social media as a generic term for the entire Internet-enabled social context is that the word “media,” already slippery (does it refer to works of creation, or to finding relevant news/media items, or to public chatter and commentary, or all of these things?), starts to add nothing to the phrase, and doesn’t really address the social graph. Most recently, we’ve seen a proliferation of social media marketing experts and gurus online, and their messages range from the sublime (that marketing can truly be turned inside out as a form of customer service, through Cluetrainful engagement* with customers, i.e., treating them as human beings through ordinary conversations and public responsiveness), to the mundane (as in the early days of the Internet, every local market has its village explainers), to the ridiculous (a glorified version of spam).


pages: 541 words: 109,698

Mining the Social Web: Finding Needles in the Social Haystack by Matthew A. Russell

Climategate, cloud computing, crowdsourcing, en.wikipedia.org, fault tolerance, Firefox, full text search, Georg Cantor, Google Earth, information retrieval, Mark Zuckerberg, natural language processing, NP-complete, Saturday Night Live, semantic web, Silicon Valley, slashdot, social graph, social web, statistical model, Steve Jobs, supply-chain management, text mining, traveling salesman, Turing test, web application

Let’s whip up a simple script for harvesting XFN data similar to the service offered by rubhub, a social search engine that crawls and indexes a large number of websites using XFN. You might also want to check out one of the many online XFN tools if you want to explore the full specification before moving on to the next section. A Breadth-First Crawl of XFN Data Let’s get social by mining some XFN data and building out a social graph from it. Given that XFN can be embedded into any conceivable web page, the bad news is that we’re about to do some web scraping. The good news, however, is that it’s probably the most trivial web scraping you’ll ever do, and the BeautifulSoup package absolutely minimizes the burden. The code in Example 2-2 uses Ajaxian, a popular blog about modern-day web development, as the basis of the graph.

Sample output follows: Dion Almaer http://www.almaer.com/blog/ [u'me'] Ben Galbraith http://weblogs.java.net/blog/javaben/ [u'co-worker'] Rey Bango http://reybango.com/ [u'friend'] Michael Mahemoff http://softwareas.com/ [u'friend'] Chris Cornutt http://blog.phpdeveloper.org/ [u'friend'] Rob Sanheim http://www.robsanheim.com/ [u'friend'] Dietrich Kappe http://blogs.pathf.com/agileajax/ [u'friend'] Chris Heilmann http://wait-till-i.com/ [u'friend'] Brad Neuberg http://codinginparadise.org/about/ [u'friend'] Assuming that the URL for each friend includes XFN or other useful information, it’s straightforward enough to follow the links and build out more social graph information in a systematic way. That approach is exactly what the next code example does: it builds out a graph in a breadth-first manner, which is to say that it does something like what is described in Example 2-3 in pseudocode. Example 2-3. Pseudocode for a breadth-first search Create an empty graph Create an empty queue to keep track of nodes that need to be processed Add the starting point to the graph as the root node Add the root node to a queue for processing Repeat until some maximum depth is reached or the queue is empty: Remove a node from the queue For each of the node's neighbors: If the neighbor hasn't already been processed: Add it to the queue Add it to the graph Create an edge in the graph that connects the node and its neighbor Note that this approach to building out a graph has the advantage of naturally creating edges between nodes in both directions, if such edges exist, without any additional bookkeeping required.

One consideration is that slight variations in URLs result in multiple nodes potentially appearing in the graph for the same person. For example, if Matthew is referenced in one hyperlink with the URL http://example.com/~matthew but as http://www.example.com/~matthew in another URL, those two nodes will remain distinct in the graph even though they most likely point to the same resource on the Web. Fortunately, XFN defines a special rel="me" value that can be used for identity consolidation. Google’s Social Graph API takes this very approach to connect a user’s various profiles, and there exist many examples of services that use rel="me" to allow users to connect profiles across multiple external sites. Another (much lesser) issue in resolving URLs is the use or omission of a trailing slash at the end. Most well-designed sites will automatically redirect one to the other, so this detail is mostly a nonissue.


pages: 274 words: 75,846

The Filter Bubble: What the Internet Is Hiding From You by Eli Pariser

A Declaration of the Independence of Cyberspace, A Pattern Language, Amazon Web Services, augmented reality, back-to-the-land, Black Swan, borderless world, Build a better mousetrap, Cass Sunstein, citizen journalism, cloud computing, cognitive dissonance, crowdsourcing, Danny Hillis, data acquisition, disintermediation, don't be evil, Filter Bubble, Flash crash, fundamental attribution error, global village, Haight Ashbury, Internet of things, Isaac Newton, Jaron Lanier, Jeff Bezos, jimmy wales, Kevin Kelly, knowledge worker, Mark Zuckerberg, Marshall McLuhan, megacity, Metcalfe’s law, Netflix Prize, new economy, PageRank, paypal mafia, Peter Thiel, recommendation engine, RFID, Robert Metcalfe, sentiment analysis, shareholder value, Silicon Valley, Silicon Valley startup, social graph, social software, social web, speech recognition, Startup school, statistical model, stem cell, Steve Jobs, Steven Levy, Stewart Brand, technoutopianism, the scientific method, urban planning, Whole Earth Catalog, WikiLeaks, Y Combinator

The term stereotyping (which in this sense comes from Walter Lippmann, incidentally) is often used to refer to malicious xenophobic patterns that aren’t true—“people of this skin color are less intelligent” is a classic example. But stereotypes and the negative consequences that flow from them aren’t fair to specific people even if they’re generally pretty accurate. Marketers are already exploring the gray area between what can be predicted and what predictions are fair. According to Charlie Stryker, an old hand in the behavioral targeting industry who spoke at the Social Graph Symposium, the U.S. Army has had terrific success using social-graph data to recruit for the military—after all, if six of your Facebook buddies have enlisted, it’s likely that you would consider doing so too. Drawing inferences based on what people like you or people linked to you do is pretty good business. And it’s not just the army. Banks are beginning to use social data to decide to whom to offer loans: If your friends don’t pay on time, it’s likely that you’ll be a deadbeat too.

.: How We Got from the Company Man, Family Dinners, and the Affluent Society to the Home Office, BlackBerry Moms, and Economic Anxiety (New York: Pantheon, 2008), 164. 130 “Model-T version of what’s possible”: Geoff Duncan, “Netflix Offers $1Mln for Good Movie Picks,” Digital Trends, Oct. 2, 2006, accessed Dec. 15, 2010, www.digitaltrends.com/computing/netflix-offers-1-mln-for-good-movie-picks. 130 “a PC and some great insight”: Katie Hafner, “And If You Liked the Movie, a Netflix Contest May Reward You Handsomely,” New York Times, Oct. 2, 2006, accessed Dec. 15, 2010, www.nytimes.com/2006/10/02/technology/02netflix.html. 131 success using social-graph data: Charlie Stryler, Marketing Panel at 2010 Social Graph Symposium, Microsoft Campus, Mountain View, CA, May 21, 2010. 132 “the creditworthiness of your friends”: Julia Angwin, “Web’s New Gold Mine,” Wall Street Journal, July 30, 2010, accessed on Feb. 7, 2011, http://online.wsj.com/article/SB10001424052748703940904575395073512989404.html. 133 reality doesn’t work that way: David Hume, An Enquiry Concerning Human Understanding, Harvard Classics, volume 37, Section VII, Part I, online edition, (P.

PayPal PeekYou persuasion profiling Phantom Public, The (Lippmann) Philby, Kim Phorm Piaget, Jean Picasa Picasso, Pablo PK List Management Plato politics electoral districts and partisans and programmers and voting Popper, Karl postmaterialism predictions present bias priming effect privacy Facebook and facial recognition and genetic Procter & Gamble product recommendations Proulx, Travis Pulitzer, Joseph push technology and pull technology Putnam, Robert Qiang, Xiao Rapleaf Rather, Dan Raz, Guy reality augmented Reality Hunger (Shields) Reddit Rendon, John Republic.com (Sunstein) retargeting RFID chips robots Rodriguez de Montalvo, Garci Rolling Stone Roombas Rotenberg, Marc Rothstein, Mark Rove, Karl Royal Caribbean Rubel, Steve Rubicon Project Rumsfeld, Donald Rushkoff, Douglas Salam, Reihan Sandberg, Sheryl schemata Schmidt, Eric Schudson, Michael Schulz, Kathryn science Scientific American Scorpion sentiment analysis Sentry serendipity Shields, David Shirky, Clay Siegel, Lee signals click Simonton, Dean Singhal, Amit Sleepwalkers, The (Koestler) smart devices Smith, J. Walker social capital social graph Social Graph Symposium Social Network, The Solove, Daniel solution horizon Startup School Steitz, Mark stereotyping Stewart, Neal Stryker, Charlie Sullivan, Danny Sunstein, Cass systematization Taleb, Nassim Nicholas Tapestry TargusInfo Taylor, Bret technodeterminism technology television advertising on mean world syndrome and Tetlock, Philip Thiel, Peter This American Life Thompson, Clive Time Tocqueville, Alexis de Torvalds, Linus town hall meetings traffic transparency Trotsky, Leon Turner, Fred Twitter Facebook compared with Últimas Noticias Unabomber uncanny valley Upshot Vaidhyanathan, Siva video games Wales, Jimmy Wall Street Journal Walmart Washington Post Web site morphing Westen, Drew Where Good Ideas Come From (Johnson) Whole Earth Catalog WikiLeaks Wikipedia Winer, Dave Winner, Langdon Winograd, Terry Wired Wiseman, Richard Woolworth, Andy Wright, David Wu, Tim Yahoo News Upshot Y Combinator Yeager, Sam Yelp You Tube LeanBack Zittrain, Jonathan Zuckerberg, Mark Table of Contents Title Page Copyright Page Dedication Introduction Chapter 1 - The Race for Relevance Chapter 2 - The User Is the Content Chapter 3 - The Adderall Society Chapter 4 - The You Loop Chapter 5 - The Public Is Irrelevant Chapter 6 - Hello, World!


pages: 302 words: 73,581

Platform Scale: How an Emerging Business Model Helps Startups Build Large Empires With Minimum Investment by Sangeet Paul Choudary

3D printing, Airbnb, Amazon Web Services, barriers to entry, bitcoin, blockchain, business process, Chuck Templeton: OpenTable:, Clayton Christensen, collaborative economy, commoditize, crowdsourcing, cryptocurrency, data acquisition, frictionless, game design, hive mind, Internet of things, invisible hand, Kickstarter, Lean Startup, Lyft, M-Pesa, Marc Andreessen, Mark Zuckerberg, means of production, multi-sided market, Network effects, new economy, Paul Graham, recommendation engine, ride hailing / ride sharing, shareholder value, sharing economy, Silicon Valley, Skype, Snapchat, social graph, social software, software as a service, software is eating the world, Spread Networks laid a new fibre optics cable between New York and Chicago, TaskRabbit, the payments system, too big to fail, transport as a service, two-sided market, Uber and Lyft, Uber for X, uber lyft, Wave and Pay

However, these multihoming costs are not as strong as they used to be. INSUFFICIENT NETWORK EFFECT Two shifts have brought about a rapid decline in multihoming costs. First, the rise of the social graph allows users to port their personal networks between different platforms. A new platform, like Instagram, can leverage the single sign-on enabled by the social graph to build an alternate network of users rapidly. Second, mobile-based access allows users to switch easily and rapidly between different apps multiple times a day. This allows multihoming at a scale that was once unimaginable. Drivers today use Uber, Lyft, and a host of other apps simultaneously and switch between them several times a day. The convenience of the social graph, coupled with the ease of switching between platforms, has eroded the lock-in that once kept users bound to a network. THINKING BEYOND THE NETWORK EFFECT The network effect isn’t quite as effective at retaining producers and consumers as it once was.

Amazon’s “People who purchased this product also purchased this product” feature is based on a collaborative filter. Many recommendation platforms allow users to filter results based on a “people like you” parameter. This, again, is a collaborative filter. The most important innovation in recent times that has led to the spread of collaborative filters is the implementation of Facebook’s social graph. Through the social graph, third-party platforms like TripAdvisor serve reviews based on a collaborative filter of people who are close to you on the graph. Finally, it is important to note that the network itself is a filter. Who you follow determines what you consume. On Twitter, who you follow is the critical filter. Relevance is almost entirely dictated by it. On Facebook, who you are connected to and how often you interact with them strengthen the newsfeed filter.

The 7c’s of trust – Confirmed Identity, Centralized Moderation, Community Feedback, Codified Behavior, Culture, Completeness, Cover – explored in this chapter, are a set of themes that platform creators may use to build trust on platforms. CONFIRMED IDENTITY Identity can be used to help build trust. The rise of Facebook’s social graph helped create real identity on the Internet, at least compared to the anonymity that was involved in much Internet participation prior to that. Today, Lyft riders link their accounts to their Facebook profiles. Tinder and a whole range of other social platforms require users to sign up through Facebook Connect. The social graph isn’t foolproof, and confirmation of identity may require more for different types of interactions. Airbnb confirms a listing by sending out photographers to a specific apartment. Sittercity babysitters must go through a stringent vetting process before being allowed onboard.


pages: 527 words: 147,690

Terms of Service: Social Media and the Price of Constant Connection by Jacob Silverman

23andMe, 4chan, A Declaration of the Independence of Cyberspace, Airbnb, airport security, Amazon Mechanical Turk, augmented reality, basic income, Brian Krebs, California gold rush, call centre, cloud computing, cognitive dissonance, commoditize, correlation does not imply causation, Credit Default Swap, crowdsourcing, don't be evil, drone strike, Edward Snowden, feminist movement, Filter Bubble, Firefox, Flash crash, game design, global village, Google Chrome, Google Glasses, hive mind, income inequality, informal economy, information retrieval, Internet of things, Jaron Lanier, jimmy wales, Kevin Kelly, Kickstarter, knowledge economy, knowledge worker, late capitalism, license plate recognition, life extension, lifelogging, Lyft, Mark Zuckerberg, Mars Rover, Marshall McLuhan, mass incarceration, meta analysis, meta-analysis, Minecraft, move fast and break things, move fast and break things, national security letter, Network effects, new economy, Nicholas Carr, Occupy movement, optical character recognition, payday loans, Peter Thiel, postindustrial economy, prediction markets, pre–internet, price discrimination, price stability, profit motive, quantitative hedge fund, race to the bottom, Ray Kurzweil, recommendation engine, rent control, RFID, ride hailing / ride sharing, self-driving car, sentiment analysis, shareholder value, sharing economy, Silicon Valley, Silicon Valley ideology, Snapchat, social graph, social intelligence, social web, sorting algorithm, Steve Ballmer, Steve Jobs, Steven Levy, TaskRabbit, technoutopianism, telemarketer, transportation-network company, Travis Kalanick, Turing test, Uber and Lyft, Uber for X, uber lyft, universal basic income, unpaid internship, women in the workforce, Y Combinator, Zipcar

When social networks like Facebook and Twitter offer to “authenticate” users (such as by the blue check mark that appears next to verified Twitter accounts), they are not speaking in terms of users being able to fulfill their own sense of authenticity, their own ideas about what it means to be true to one’s self. Rather, they are authenticating, or verifying, users for the network’s purposes. THE SOCIAL GRAPH AND FRICTIONLESS SHARING The “social graph” is a term popularized by Facebook, and it has spread throughout the industry, coming to stand in for the complex web of relationships we create and maintain, both online and off. You are probably connected to your parents online but also to friends, family, coworkers, strangers you talk to on Twitter, people you buy from on eBay, anonymous interlocutors on message boards, your college friends on Instagram, and so on. The social graph is who you talk to, how often, and what these relationships might say about you. And in the hands of Google, Facebook, and Twitter, it’s a potential gold mine of data that advertisers love.

And in the hands of Google, Facebook, and Twitter, it’s a potential gold mine of data that advertisers love. It’s human life as a clutch of data points, every feeling and expression and relationship recorded, mined, algorithmized. The social graph, however, is only as useful as the data to which it’s connected. And it’s pretty limited if you don’t share very often or if Facebook or Google or Twitter can’t learn what you do when you’re not on their sites. That’s why these companies have led the way in socializing—or surveilling—the entire Web. Each social widget, each Like or +1 button on a Web page, of which there are millions now, acts as a tracking beacon, feeding information back to the company that owns it. The practice has its roots in the advertising and tracking industry, in which third-party companies have long used online advertisements to plant cookies—small files containing information about the user—on user’s machines.

Upon learning this news, Katherine Losse followed up on Twitter, citing her own remarks about Zuckerberg wanting to turn us all into cells into a single organism, one that Facebook naturally would control. For Facebook, she wrote, “the privacy of thought is a problem for tech to overcome.” That’s why Facebook’s status bar doesn’t ask you what’s new; it asks, “What’s on your mind?” That is both a prompt and an explicit statement of intent: to know, whenever possible, what we are thinking and doing. THE SPREAD OF GOOGLE+ Facebook’s promotion of the social graph—and its attendant features, such as Like buttons, third-party apps, and universal log-ins that allow you to use your social-media identity across services—has some competition in Google and its Google+ social network. The two companies are also ideological fellow travelers, with Google+ representing the search company’s own effort to apply a social layer over the Internet and to capture and filter all user behavior through its own social network.


pages: 706 words: 202,591

Facebook: The Inside Story by Steven Levy

active measures, Airbnb, Airbus A320, Amazon Mechanical Turk, Apple's 1984 Super Bowl advert, augmented reality, Ben Horowitz, blockchain, Burning Man, business intelligence, cloud computing, computer vision, crowdsourcing, cryptocurrency, don't be evil, Donald Trump, East Village, Edward Snowden, El Camino Real, Elon Musk, Firefox, Frank Gehry, glass ceiling, indoor plumbing, Jeff Bezos, John Markoff, Jony Ive, Kevin Kelly, Kickstarter, Lyft, Mahatma Gandhi, Marc Andreessen, Mark Zuckerberg, Menlo Park, Metcalfe’s law, MITM: man-in-the-middle, move fast and break things, move fast and break things, natural language processing, Network effects, Oculus Rift, PageRank, Paul Buchheit, paypal mafia, Peter Thiel, pets.com, post-work, Ray Kurzweil, recommendation engine, Robert Mercer, Robert Metcalfe, rolodex, Sam Altman, Sand Hill Road, self-driving car, sexual politics, Shoshana Zuboff, side project, Silicon Valley, Silicon Valley startup, slashdot, Snapchat, social graph, social software, South of Market, San Francisco, Startup school, Steve Ballmer, Steve Jobs, Steven Levy, Steven Pinker, Tim Cook: Apple, web application, WikiLeaks, women in the workforce, Y Combinator, Y2K

In the course of brainstorming concepts for his speech, Zuckerberg acquired some of the language that would pepper his explanations of Facebook’s mission for years to come. The most important term was what he called the “social graph.” Though the concept had been batted around for months in late-night discussions—and was pursued as far back as Adam D’Angelo’s Buddy Zoo—that single term seemed to embody what Facebook wanted to unlock for its users. Social graph refers to the nexus of connections people have in the real world. By expediting connections to those people who were on your friend-and-acquaintance radar, Facebook was unlocking a network you already had, keeping you in close touch with people huddled next to you on this virtual constellation, and drawing lines to those who were one, two, three degrees away. “We don’t own the social graph,” Zuckerberg would explain to me later that year, going slow so even a mainstream journalist might understand this dive into network theory.

“We don’t own the social graph,” Zuckerberg would explain to me later that year, going slow so even a mainstream journalist might understand this dive into network theory. “The social graph is this thing that exists in the world, and it always has and it always will. A lot of people think that maybe Facebook’s a community site, and we think we’re not a community site at all. We’re not defining any communities. All we’re doing is taking this real-world social graph that exists with real people and their real connections, and we’re trying to get as accurate of a picture as possible of how those connections are modeled out.” Once that picture was captured, Facebook and all the other companies on the platform could exploit the social graph to, as Zuckerberg puts it, “build a set of communication utilities that help people share information with all of the people that they’re connected to.” Unsaid was Facebook’s ambition to be the only company that captured the full picture of the social graph.

It was something our brains already did when we looked at pictures. So it was simple to execute, no AI involved—just have people click on the faces in the photos and fill in a blank text box. Facebook didn’t have the artificial intelligence for facial recognition yet. But it did have fanatic users hugely motivated to share. Sittig set up a system where people could quickly note who was in the photo—if the person was on your social graph, just typing a few letters would automatically fill out the full name. The whole thing was engineered to encourage you to tag the people in the photo. Then a flywheel would take over, to extend the experience to others. When you were tagged in a photo, you’d get notified, and of course you would go to that person’s profile page to see the picture. If the person with the photo wasn’t your friend yet, you might friend the person right then.


pages: 382 words: 105,819

Zucked: Waking Up to the Facebook Catastrophe by Roger McNamee

4chan, Albert Einstein, algorithmic trading, AltaVista, Amazon Web Services, barriers to entry, Bernie Sanders, Boycotts of Israel, Cass Sunstein, cloud computing, computer age, cross-subsidies, data is the new oil, Donald Trump, Douglas Engelbart, Douglas Engelbart, Electric Kool-Aid Acid Test, Elon Musk, Filter Bubble, game design, income inequality, Internet of things, Jaron Lanier, Jeff Bezos, John Markoff, laissez-faire capitalism, Lean Startup, light touch regulation, Lyft, Marc Andreessen, Mark Zuckerberg, market bubble, Menlo Park, Metcalfe’s law, minimum viable product, Mother of all demos, move fast and break things, move fast and break things, Network effects, paypal mafia, Peter Thiel, pets.com, post-work, profit maximization, profit motive, race to the bottom, recommendation engine, Robert Mercer, Ronald Reagan, Sand Hill Road, self-driving car, Silicon Valley, Silicon Valley startup, Skype, Snapchat, social graph, software is eating the world, Stephen Hawking, Steve Jobs, Steven Levy, Stewart Brand, The Chicago School, Tim Cook: Apple, two-sided market, Uber and Lyft, Uber for X, uber lyft, Upton Sinclair, WikiLeaks, Yom Kippur War

Sean Parker described the solution this way in Adam Fisher’s Valley of Genius: “The ‘social graph’ is a math concept from graph theory, but it was a way of trying to explain to people who were kind of academic and mathematically inclined that what we were building was not a product so much as it was a network composed of nodes with a lot of information flowing between those nodes. That’s graph theory. Therefore we’re building a social graph. It was never meant to be talked about publicly.” Perhaps not, but it was brilliant. The notion that a small team in their early twenties with little or no work experience figured it out on the first try is remarkable. The founders also had the great insight that real identity would simplify the social graph, reducing each user to a single address. These two ideas would not only help Facebook overcome the performance problems that sank Friendster and MySpace, they would remain core to the company’s success as it grew past two billion users.

Without modification, the costs may be disproportionately burdensome on startups, further enhancing the competitive advantages of the largest companies, but there are ways to compensate for that without undermining a very important new regulation. My feedback to Representative Lofgren recommended modifying her article #4 to allow portability of the entire social graph—the entire friend network—as a way to promote competition from startups. If you want to compete with Facebook today, you have to solve two huge problems: finding users and then persuading them to invest in your platform to reproduce some of what they already have on Facebook. Portability of the social graph—including friends—would reduce the scope of the second problem to manageable levels, even when you factor in the need for permission from every friend. But graph portability was just the first step. I also advocated antitrust measures. In my message to Representative Lofgren, I proposed the adoption of a classic model of antitrust as the least harmful, most pro-growth form of intervention she could advocate.

The integrated data set rivaled Amazon’s, but without warehouses and inventory it generated much greater profits for Google. Best of all, combined data sets often reveal insights and business opportunities that could not have been imagined previously. The new products were free to use, but each one contributed data that transformed the value of Google’s advertising products. Facebook did something analogous with each function it added to the platform. Photo tagging expanded the social graph. News Feed enriched it further. The Like button delivered data on emotional triggers. Connect tracked users as they went around the web. The value is not really in the photos and links posted by users. The real value resides in metadata—data about data—which is what we call the data that describes where the user was when he or she posted, what they were doing, with whom they were doing it, alternatives they considered, and more.


pages: 455 words: 133,322

The Facebook Effect by David Kirkpatrick

Andy Kessler, Burning Man, delayed gratification, demand response, don't be evil, global village, happiness index / gross national happiness, Howard Rheingold, Jeff Bezos, Marc Andreessen, Mark Zuckerberg, Marshall McLuhan, Menlo Park, Network effects, Peter Thiel, rolodex, Sand Hill Road, sharing economy, Silicon Valley, Silicon Valley startup, Skype, social graph, social software, social web, Startup school, Steve Ballmer, Steve Jobs, Stewart Brand, the payments system, The Wealth of Nations by Adam Smith, Whole Earth Review, winner-take-all economy, Y Combinator

Zuckerberg was beginning to talk about what he would come to label the “social graph,” meaning the web of relationships articulated inside Facebook as the result of users connecting with their friends. With Facebook photos, your friends—your social graph—provided more information, context, and a sense of companionship. But it only worked because the photos were tagged with people’s names and Facebook alerted people when they were tagged. The tags determined how the photos were distributed through the service. “Watching the growth of tagging,” says Cohler, “was the first ‘aha’ for us about how the social graph could be used as a distribution system. The mechanism of distribution was the relationships between people.” Perhaps applying the social graph to other online activities would make them more interesting and useful, too.

By “distribution” he meant that by connecting with your friends on Facebook you had assembled a network, this so-called social graph, and it could be employed to distribute any sort of information. If you added a photo, it told your friends. Ditto if you changed your relationship status, or announced that you were heading to Mexico for the weekend. But it could also tell your friends about any action you took using any software to which your social graph was connected. So far, though, the only applications that took advantage of this distribution capability were photos, events, and a few others created by Facebook itself. Most software companies, were they to conclude that they had such an ability to create uniquely powerful applications, would create more of them. They might make shopping applications on top of their social graph, or games, or applications for businesses.

It was garnering more usage than Evite.com, which had been for years the leading website for invitations. “So why were photos and events so good?” he asked. “It was because despite all their shortcomings they had one thing no one else had. And that was integration with the social graph.” This was Facebook’s own conceptual breakthrough, and Zuckerberg was proud of the term he used to describe it. “We did some thinking and we decided that the core value of Facebook is in the set of friend connections,” he continued. “We call that the social graph, in the mathematical sense of a series of nodes and connections. The nodes are the individuals and the connections are the friendships.” Then his enthusiasm veered, it seemed at the time, toward overstatement: “We have the most powerful distribution mechanism that’s been created in a generation.”


Beautiful Visualization by Julie Steele

barriers to entry, correlation does not imply causation, data acquisition, database schema, Drosophila, en.wikipedia.org, epigenetics, global pandemic, Hans Rosling, index card, information retrieval, iterative process, linked data, Mercator projection, meta analysis, meta-analysis, natural language processing, Netflix Prize, pattern recognition, peer-to-peer, performance metric, QR code, recommendation engine, semantic web, social graph, sorting algorithm, Steve Jobs, web application, wikimedia commons

I therefore classify her as a member of neither cluster (not both clusters!) and color her purple. The final emergent social graph is shown in Figure 7-7. Figure 7-7. Emergent social graph of women based on common attendance at social events All 18 women have now been placed in the social network based on their attendance of local social events. This social network reveals a few interesting things about this small town’s social structure: Two distinct social clusters exist. The clusters are connected. This social overlap reveals some possible commonality in interests and relations between the two clusters. Various network roles emerge. Some women are connectors, bridging the two clusters, while others act as internal core members, connecting only to their own groups. Social graphs like that in Figure 7-7 can be used for marketing purposes or word-of-mouth campaigns.

Patterns formed by event attendance and object selection will give us clues into the thinking and behavior of the humans attending the events and choosing the objects. Often, our simple behaviors and choices can reveal who we are, and whom we are like. Early Social Graphs In the 1930s, a group of sociologists and ethnographers did a small “data mining” experiment. They wanted to derive the social structure of a group of women in a small town in the southern United States. They used public data that appeared in the local newspaper. Their dataset was small: 18 women attending 14 different social events. They wondered: could we figure out the social structure (today we call it a social graph) of this group of women? To this end, they posed the following questions: Who is a friend of whom? Which social circles are they all in? Who plays a key role in the social structure?

Friendship ties amongst the Wiring Room employees are illustrated in Figure 7-1. Figure 7-1. Early 20th-century social graph used in studying workflows amongst employees SNA maps a human system as nodes and links. The nodes are usually people, and the links are either relationships between people or flows between people. The links can be directional. When the nodes are of only one type—for example, people, as in the Moreno and Hawthorne studies—it is called one-mode analysis. However, the Southern Women study began as a slightly more complex form of social analysis: two-mode. There were two sets of nodes—people and events—and the links showed which people attended which events. The social graph for the two data modes are shown in Figure 7-2. The women are the blue nodes on the left, while the events that each attended are the green nodes on the right.


pages: 366 words: 76,476

Dataclysm: Who We Are (When We Think No One's Looking) by Christian Rudder

4chan, Affordable Care Act / Obamacare, bitcoin, cloud computing, correlation does not imply causation, crowdsourcing, cuban missile crisis, Donald Trump, Edward Snowden, en.wikipedia.org, Frank Gehry, Howard Zinn, Jaron Lanier, John Markoff, John Snow's cholera map, lifelogging, Mahatma Gandhi, Mikhail Gorbachev, Nate Silver, Nelson Mandela, new economy, obamacare, Occupy movement, p-value, pre–internet, race to the bottom, selection bias, Snapchat, social graph, Solar eclipse in 1919, Steve Jobs, the scientific method

One of its expressions is the amount of overlap in a pair of social graphs—Reshma’s and my embeddedness is simply how large the red portion of our graph is compared with the whole. Research using a variety of sources (e-mail, IM, telephone) has shown that the more mutual friends two people share, the stronger their relationship. More connections imply more time together, more common interests, and more stability. But unlike, say, telephone records, or even e-mail, online social networks attach rich data to a graph’s edges and nodes (not unlike how dating sites have taken the timeless ritual of courtship and added age and beauty as variables to study) and of course Facebook is the richest such network ever created. The effects of that richness are just being felt. Social-graph analysis began as, and largely remains, a matter of “who knows who.”

And then, often, and often suddenly, it’s back to the beginning with someone else. We’ve had a look so far at the ways two people come together in the first blush of attraction. I’m not sure a computer will ever capture their path to full togetherness, but we do have a picture of their lives once they get there. That pattern of a couple together, the enmeshing of what’s come to be called their “social graphs,” is now well documented. I have 384 friends on Facebook, and here they are. I’m the dot in the middle; my wife, Reshma, is in black at about three o’clock. Everyone’s connections to everyone else are shown by the gray lines: Though the groups of my friends are nicely clustered, this plot wasn’t arranged by hand—my able research assistant, James Dowdell, wrote special software to create it.

Or in a cliquey network without assimilation, “leading separate lives” can very quickly become “leading secret lives,” which might look something like this: Against assimilation, Backstrom and Kleinberg tested many other ways to evaluate a relationship, and there was one detail in their paper, presented almost as an aside, that I found particularly wry. Early on, the best predictor of a relationship doesn’t depend on the couple’s social graph at all; for the first year or so of dating, the optimal method is how often they view each other’s profile. Only over time, as the page views go down and their mutual network fills out, does assimilation come to dominate the calculus. In other words, the curiosity, discovery, and (visual) stimulation of falling for someone is eventually replaced by the graph-theory equivalent of nesting. There’s this idea in computer science that you should be your own customer—that you should at least have enough confidence in the website or software you’re foisting on the world to use it yourself.


Mining of Massive Datasets by Jure Leskovec, Anand Rajaraman, Jeffrey David Ullman

cloud computing, crowdsourcing, en.wikipedia.org, first-price auction, G4S, information retrieval, John Snow's cholera map, Netflix Prize, NP-complete, PageRank, pattern recognition, random walk, recommendation engine, second-price auction, sentiment analysis, social graph, statistical model, web application

EXERCISE 10.4.3For the Laplacian matrix constructed in Exercise 10.4.1(c), construct the third and subsequent smallest eigenvalues and their eigenvectors. 10.5Finding Overlapping Communities So far, we have concentrated on clustering a social graph to find communities. But communities are in practice rarely disjoint. In this section, we explain a method for taking a social graph and fitting a model to it that best explains how it could have been generated by a mechanism that assumes the probability that two individuals are connected by an edge (are “friends”) increases as they become members of more communities in common. An important tool in this analysis is “maximum-likelihood estimation,” which we shall explain before getting to the matter of finding overlapping communities. 10.5.1The Nature of Communities To begin, let us consider what we would expect two overlapping communities to look like. Our data is a social graph, where nodes are people and there is an edge between two nodes if the people are “friends.”

This condition is the hardest to formalize, but the intuition is that relationships tend to cluster. That is, if entity A is related to both B and C, then there is a higher probability than average that B and C are related. 10.1.2Social Networks as Graphs Social networks are naturally modeled as graphs, which we sometimes refer to as a social graph. The entities are the nodes, and an edge connects two nodes if the nodes are related by the relationship that characterizes the network. If there is a degree associated with the relationship, this degree is represented by labeling the edges. Often, social graphs are undirected, as for the Facebook friends graph. But they can be directed graphs, as for example the graphs of followers on Twitter or Google+. EXAMPLE 10.1Figure 10.1 is an example of a tiny social network. The entities are the nodes A through G. The relationship, which we might think of as “friends,” is represented by the edges.

In fact, the data involved in collaborative filtering, as was discussed in Chapter 9, often can be viewed as forming a pair of networks, one for the customers and one for the products. Customers who buy the same sorts of products, e.g., sciencefiction books, will form communities, and dually, products that are bought by the same customers will form communities, e.g., all science-fiction books. Other Examples of Social Graphs Many other phenomena give rise to graphs that look something like social graphs, especially exhibiting locality. Examples include: information networks (documents, web graphs, patents), infrastructure networks (roads, planes, water pipes, powergrids), biological networks (genes, proteins, food-webs of animals eating each other), as well as other types, like product co-purchasing networks (e.g., Groupon). 10.1.4Graphs With Several Node Types There are other social phenomena that involve entities of different types.


pages: 562 words: 153,825

Dark Mirror: Edward Snowden and the Surveillance State by Barton Gellman

4chan, A Declaration of the Independence of Cyberspace, active measures, Anton Chekhov, bitcoin, Cass Sunstein, cloud computing, corporate governance, crowdsourcing, data acquisition, Debian, desegregation, Donald Trump, Edward Snowden, financial independence, Firefox, GnuPG, Google Hangouts, informal economy, Jacob Appelbaum, job automation, Julian Assange, MITM: man-in-the-middle, national security letter, planetary scale, private military company, ransomware, Robert Gordon, Robert Hanssen: Double agent, rolodex, Ronald Reagan, Saturday Night Live, Silicon Valley, Skype, social graph, standardized shipping container, Steven Levy, telepresence, undersea cable, web of trust, WikiLeaks, zero day, Zimmermann PGP

A related tool called MapReduce condensed the trillions of data points into summary form that a human analyst could grasp. Network theory called this map a social graph. It modeled the relationships and groups that defined each person’s interaction with the world. The NSA’s analysis touched nearly all Americans because the size of the graph grew exponentially as contact chaining progressed. The whole point of chaining was to push outward from a target’s immediate contacts to the contacts of contacts, then contacts of contacts of contacts. Each step in that process was called a hop. Double a penny once a day and you reach a million dollars in less than a month. That is what exponential growth looks like with a base of two. As contact chaining steps through its hops, the social graph grows much faster. If the average person calls or is called by ten other people a year, then each hop produces a tenfold increase in the population of the NSA’s contact map.

The government may seldom care, may never abuse that knowledge in a given year. But now, for the first time in history, it had acquired the power to do so. Stewart Baker, a former general counsel of the NSA, leaped early into the fray against Snowden in television appearances, newspaper interviews, and blog posts. He sharply criticized some of my stories, too. But he minced no words about the power of the social graph. “Metadata absolutely tells you everything about somebody’s life,” he said. For purposes of signals intelligence, “if you have enough metadata, you don’t really need content.” Michael V. Hayden concurred bluntly the following spring. “We kill people based on metadata,” he said. “But that’s not what we do with this metadata.” In Washington, the day before our panel, Representative Bob Goodlatte, a Virginia Republican, asked Robert Litt, the DNI general counsel, whether intelligence officials had really believed the bulk phone collection “could be indefinitely kept secret from the American people?”

One flaw in this comparison is that it sounds like a job that will be finished eventually. MAINWAY’s job never ended. It was trying to index a book in progress, forever incomplete. The FBI brought the NSA more than a billion new records a day from the telephone companies. MAINWAY had to purge another billion a day to comply with the FISA Court’s five-year limit on retention. Every change cascaded through the social graph, redrawing the map and obliging MAINWAY to update ceaselessly. MAINWAY’s purpose, in other words, was neither storage nor preparation of a simple list. Constant, complex, and demanding operations fed another database called the Graph-in-Memory. When the Boston bombs exploded, the Graph-in-Memory was ready. Absent unlucky data gaps, it already held a summary map of the contacts revealed by the Tsarnaev brothers’ calls.


pages: 201 words: 63,192

Graph Databases by Ian Robinson, Jim Webber, Emil Eifrem

Amazon Web Services, anti-pattern, bioinformatics, commoditize, corporate governance, create, read, update, delete, data acquisition, en.wikipedia.org, fault tolerance, linked data, loose coupling, Network effects, recommendation engine, semantic web, sentiment analysis, social graph, software as a service, SPARQL, web application

For an excellent introduction to how graphs provide insight into complex events and behaviors, see David Easley and Jon Kleinberg, Networks, Crowds, and Markets: Reasoning about a Highly Connected World (Cambridge University Press, 2010) 2 | Chapter 1: Introduction the construction of a space rocket, to a system of roads, and from the supply-chain or provenance of foodstuff, to medical history for populations, and beyond. For example, Twitter’s data is easily represented as a graph. In Figure 1-1 we see a small network of followers. The relationships are key here in establishing the semantic context: namely, that Billy follows Harry, and that Harry, in turn, follows Billy. Ruth and Harry likewise follow each other, but sadly, while Ruth follows Billy, Billy hasn’t (yet) recip‐ rocated. Figure 1-1. A small social graph Of course, Twitter’s real graph is hundreds of millions of times larger than the example in Figure 1-1, but it works on precisely the same principles. In Figure 1-2 we’ve expanded the graph to include the messages published by Ruth. What is a Graph? | 3 Figure 1-2. Publishing messages Though simple, Figure 1-2 shows the expressive power of the graph model. It’s easy to see that Ruth has published a string of messages.

In their book Connec‐ ted, social scientists Nicholas Christakis and James Fowler show how, despite knowing nothing about an individual, we can better predict that person’s behavior by under‐ standing who they are connected to, than we can by accumulating facts about them.1 Social applications allows organizations to gain competitive and operational advantage by leveraging information about the connections between people, together with discrete information about individuals, to facilitate collaboration and flow of information, and predict behavior. 1. See Nicholas Christakis and James Fowler, Connected: The Amazing Power of Social Networks and How They Shape Our Lives (HarperPress, 2011) 94 | Chapter 5: Graphs in the Real World As Facebook’s use of the term social graph implies, graph data model and graph databases are a natural fit for this overtly relationship-centerd domain. Social networks help us identify the direct and indirect relationships between people, groups and the things with which they interact, allowing users to rate, review and discover each other and the things they care about. By understanding who interacts with whom, how people are connected, and what representatives within a group are likely to do or choose based on the aggregate behaviour of the group, we generate tremendous insight into the unseen forces that influence individual behaviours.

In practice this means that data like phone numbers and zip codes can be inlined in the property store file directly, rather than being pushed out to the dynamic stores. This results in reduced I/O operations and improved throughput, since only a single file access is required. In addition to inlining certain compatible property values, Neo4j also maintains space discipline on property names. For example in a social graph, there will likely be many nodes with properties like first_name and last_name. It would be wasteful if each property name was written out to disk verbatim, and so instead property names are indirectly referenced from the property store through the property index file. The prop‐ erty index allows all properties with the same name to share a single record, and thus for repetitive graphs—a very common use case-- Neo4j achieves considerable space and I/O savings.


pages: 286 words: 82,065

Curation Nation by Rosenbaum, Steven

Amazon Mechanical Turk, Andrew Keen, barriers to entry, citizen journalism, cognitive dissonance, commoditize, creative destruction, crowdsourcing, disintermediation, en.wikipedia.org, future of journalism, Jason Scott: textfiles.com, means of production, PageRank, pattern recognition, post-work, postindustrial economy, pre–internet, Sand Hill Road, Silicon Valley, Skype, social graph, social web, Steve Jobs, Tony Hsieh, Yogi Berra

“For a media brand, the concept of curating content becomes an important component of your strategy for connecting with an audience,” says Dan McCarthy, CEO of NCI, one of the United States’ largest local media companies serving the housing market. “People have a large appetite for content; the increase in the proportion of content that they are accessing through trusted connections suggests that people are looking into their social graph to ensure a good content experience. When a consumer includes a media brand in their social graph, they are inviting that brand to help guide their exploration of good content. It doesn’t all have to be original. It does all have to be useful and relevant to the brand experience.” So you can see that the PR folks and the advertising folks don’t look at the world the same way. Maheu points to campaigns like the Old Spice Man, which was done by a competitive agency, as evidence that broadcast television still drives buzz and starts consumer conversations.

THE FUTURE OF CONTENT For guys like Miller and Kurnit, who started their careers in the television business but were already looking past it to what would come next, curation isn’t a buzzword or a trend; it’s the future of content and nothing less. Says Miller, “I believe that there has to be curation, which by the way can run the gamut from traditional media like magazines and newspapers to bloggers like Michael Arrington. The social systems will do it with friends and your social graph. And the volume of content will continue to explode, which will create more content than advertising can support. So advertisers will need curation. In the end specialists win, the greater the specialization, the greater the win.” And Kurnit’s take is clearly in the same spirit: “There’s no question the crowd is more efficient than directed individuals. So it’s interesting that About.com holds core to its 750 guides one guide per topic area, and I think for About, it continues to do very well, that’s a very good idea.

No longer do you have to grab a line, bring it to your Facebook page, and share the link. You simple click “Like” on any piece of content, blog post, photo, or brand or e-commerce offering. That expression of your support is now linked to your profile and shared with your friends. From a content consumer’s perspective, it allows them to view aggregated recommendations of all their friends within their social graph. For a look at what that might look like, start here: http://likebutton.me/. Social media is both the source of much of the increased volume of data and increasingly the tools to empower curation, both accidental and purposeful. Simply put, we’re each making more data and recommending more things. The data we make comes from our posts, pictures, location data, and recommendations; the data we curate comes from what we link to, recommend, and endorse.


pages: 260 words: 76,223

Ctrl Alt Delete: Reboot Your Business. Reboot Your Life. Your Future Depends on It. by Mitch Joel

3D printing, Amazon Web Services, augmented reality, call centre, clockwatching, cloud computing, Firefox, future of work, ghettoisation, Google Chrome, Google Glasses, Google Hangouts, Khan Academy, Kickstarter, Kodak vs Instagram, Lean Startup, Marc Andreessen, Mark Zuckerberg, Network effects, new economy, Occupy movement, place-making, prediction markets, pre–internet, QR code, recommendation engine, Richard Florida, risk tolerance, self-driving car, Silicon Valley, Silicon Valley startup, Skype, social graph, social web, Steve Jobs, Steve Wozniak, Thomas L Friedman, Tim Cook: Apple, Tony Hsieh, white picket fence, WikiLeaks, zero-sum game

Lesson #4—Create a mutually beneficial world. In the case of Beats By Dre and Target, it’s not healthy to be going after each individual for the “like” on Facebook. The true opportunity is to figure out how to create a mutually beneficial world, instead of one where you are now competing with your own partners. Lesson #5—True fans. The majority of people do not want to friend or like your brand. They use their social graphs for friends, family, and those they made fun of in high school. The intrusion of brands is simply that: an intrusion. Your business will never get everyone to like it. So instead, turn to the fanatical. Find and nurture your true fans. Your heavy users. As that relationship delivers, they will become evangelists for you and you will begin to experience the network effect. IT’S NOT (PERFECTLY) CLEAR.

This means that big data is coming to marketing, and the insights that we will soon have available to us—at the business level—will make what we’re spending on computers, servers, and capital infrastructure pale in comparison. This will finally give us true knowledge of what it takes to acquire customers and keep them. Consumers are already demonstrating their desires in this area by using their smartphones to do everything from scanning QR codes to sharing their experiences with their peers on Facebook and Twitter. When you combine their usage (the linear data) with the circular data (what they’re doing in their social graph), and with all of this new big data trending information, it’s easy to see how much this will affect everything we know about connecting to our consumers. BE ACCOUNTABLE TO YOUR BRAND. Imagine a day when you could have all of the data and analytics you have ever wanted. Imagine being able to track and analyze the journey of your consumers. Imagine being able to be a fly on the wall for all of their conversations with family and friends about what they love and hate about your brand, the competitors, and the other brands that impact their lives.

For social to be social, it has to be something that people can both easily find and share. The goal to being social is to make everything that you are doing as sharable and as findable as possible. When you do this in a one-screen world, people find the brand, share it, and engage with it. When people engage with it, they are (hopefully) engaging with you as well. As they engage with something that resonates with them, they tend to share it throughout their social graphs. This makes it increasingly more findable for others. Yes, there are a few brands that are able to leverage this and have actual conversations with consumers, but those brands are few and far between. Plus, in a world of 140 characters, text messages, and +1s, is a conversation all that it’s cracked up to be? If we can simply make consumers’ lives better by providing them with what they want when they want it, is that not delivering more than the brand had initially promised?


pages: 275 words: 84,980

Before Babylon, Beyond Bitcoin: From Money That We Understand to Money That Understands Us (Perspectives) by David Birch

agricultural Revolution, Airbnb, bank run, banks create money, bitcoin, blockchain, Bretton Woods, British Empire, Broken windows theory, Burning Man, business cycle, capital controls, cashless society, Clayton Christensen, clockwork universe, creative destruction, credit crunch, cross-subsidies, crowdsourcing, cryptocurrency, David Graeber, dematerialisation, Diane Coyle, disruptive innovation, distributed ledger, double entry bookkeeping, Ethereum, ethereum blockchain, facts on the ground, fault tolerance, fiat currency, financial exclusion, financial innovation, financial intermediation, floating exchange rates, Fractional reserve banking, index card, informal economy, Internet of things, invention of the printing press, invention of the telegraph, invention of the telephone, invisible hand, Irish bank strikes, Isaac Newton, Jane Jacobs, Kenneth Rogoff, knowledge economy, Kuwabatake Sanjuro: assassination market, large denomination, M-Pesa, market clearing, market fundamentalism, Marshall McLuhan, Martin Wolf, mobile money, money: store of value / unit of account / medium of exchange, new economy, Northern Rock, Pingit, prediction markets, price stability, QR code, quantitative easing, railway mania, Ralph Waldo Emerson, Real Time Gross Settlement, reserve currency, Satoshi Nakamoto, seigniorage, Silicon Valley, smart contracts, social graph, special drawing rights, technoutopianism, the payments system, The Wealth of Nations by Adam Smith, too big to fail, transaction costs, tulip mania, wage slave, Washington Consensus, wikimedia commons

In a world based on trust it will be reputation rather than regulation that will animate trust in economic exchange (Birch 2000). The ‘social graph’ – the network of our social identities – will be the nexus of commerce, administration and interaction. In our distant past we were just as defined by our social graph as we are now (Lessin 2013). There were no identity cards or credit reference agencies or transactional histories of any kind. In the absence of such credentials, you were your reputation. Managing and maintaining these reputations among a small social group of an extended family or a clan was not a scalable solution as civilization progressed and moved on to growing trade as the source of prosperity. In the interconnected future, however, there is every reason to suspect that the social graph will resume its pre-eminent position since, as I will explore, it is the most trustworthy, reliable aspect of a persona.

Reputations are much harder to subvert since they depend not on what anyone thinks but on what everyone thinks. Reputations are a sound basis for interaction. People make judgments based on other people’s reputations, and behave better out of concern for their own (Dyson 2001). There would have been precious little chance of pretending to be someone else at the local pub in Ireland in the 1970s, and as a consequence the social graph could provide the necessary infrastructure: the landlords knew whose IOUs were good and whose were not and did not need money to substitute for their memories. There you go bringing class into it again Remember that Los Angeles Times Magazine prediction about 2013? It included a trip to the ATM to draw out cash: After parking the van, Alma stops for some cash at the bank-teller machine in the lobby of her building.

If that money is going to embody the values of the communities that create it, we are moving into new realms. Reflecting on the 17th Annual Consult Hyperion Tomorrow’s Transactions Forum, held in London in March 2014, Wendy Grossman said that ‘we are moving from money we understand to money that understands us’. I was very taken with this encapsulation of a couple of thousand years of monetary evolution, from coins made from precious metals to computations across the social graph. We no longer have money that the normal, typical member of society can understand. The public don’t understand how this crucial economic technology works, and they don’t care. The debate about what happens when we go from dumb £5 notes with no memory to Bitcoins on a blockchain that know where they have been, and then on to more sophisticated, more intelligent, more connected forms of money, is of the greatest importance and it needs to be opened up so that the public can take part.


pages: 304 words: 82,395

Big Data: A Revolution That Will Transform How We Live, Work, and Think by Viktor Mayer-Schonberger, Kenneth Cukier

23andMe, Affordable Care Act / Obamacare, airport security, barriers to entry, Berlin Wall, big data - Walmart - Pop Tarts, Black Swan, book scanning, business intelligence, business process, call centre, cloud computing, computer age, correlation does not imply causation, dark matter, double entry bookkeeping, Eratosthenes, Erik Brynjolfsson, game design, IBM and the Holocaust, index card, informal economy, intangible asset, Internet of things, invention of the printing press, Jeff Bezos, Joi Ito, lifelogging, Louis Pasteur, Mark Zuckerberg, Menlo Park, Moneyball by Michael Lewis explains big data, Nate Silver, natural language processing, Netflix Prize, Network effects, obamacare, optical character recognition, PageRank, paypal mafia, performance metric, Peter Thiel, post-materialism, random walk, recommendation engine, self-driving car, sentiment analysis, Silicon Valley, Silicon Valley startup, smart grid, smart meter, social graph, speech recognition, Steve Jobs, Steven Levy, the scientific method, The Signal and the Noise by Nate Silver, The Wealth of Nations by Adam Smith, Thomas Davenport, Turing test, Watson beat the top human players on Jeopardy!

Besides, the company is still adjusting its business model (and privacy policy) for the amount and type of data collection it wants to do. Hence much more of the criticism it has faced centers on what information it is capable of collecting than on what it has actually done with that data. Facebook had around one billion users in 2012, who were interconnected through over 100 billion friendships. The resulting social graph represents more than 10 percent of the total world population, datafied and available to a single company. The potential uses are extraordinary. A number of startups have looked into adapting the social graph to use as signals for establishing credit scores. The idea is that birds of a feather flock together: prudent people befriend like-minded types, while the profligate hang out among themselves. If it pans out, Facebook could be the next FICO, the credit-scoring agency. The rich datasets from social media firms may well form the basis of new businesses that go far beyond the superficial sharing of photos, status updates, and “likes.”

Before big data, our analysis was usually limited to testing a small number of hypotheses that we defined well before we even collected the data. When we let the data speak, we can make connections that we had never thought existed. Hence, some hedge funds parse Twitter to predict the performance of the stock market. Amazon and Netflix base their product recommendations on a myriad of user interactions on their sites. Twitter, LinkedIn, and Facebook all map users’ “social graph” of relationships to learn their preferences. Of course, humans have been analyzing data for millennia. Writing was developed in ancient Mesopotamia because bureaucrats wanted an efficient tool to record and keep track of information. Since biblical times governments have held censuses to gather huge datasets on their citizenry, and for two hundred years actuaries have similarly collected large troves of data concerning the risks they hope to understand—or at least avoid.

The idea of datafication is the backbone of many of the Web’s social media companies. Social networking platforms don’t simply offer us a way to find and stay in touch with friends and colleagues, they take intangible elements of our everyday life and transform them into data that can be used to do new things. Facebook datafied relationships; they always existed and constituted information, but they were never formally defined as data until Facebook’s “social graph.” Twitter enabled the datafication of sentiment by creating an easy way for people to record and share their stray thoughts, which had previously been lost to the winds of time. LinkedIn datafied our long-past professional experiences, just as Maury transformed old logbooks, turning that information into predictions about our present and future: whom we may know, or a job we may want. Such uses of the data are still embryonic.


pages: 588 words: 131,025

The Patient Will See You Now: The Future of Medicine Is in Your Hands by Eric Topol

23andMe, 3D printing, Affordable Care Act / Obamacare, Anne Wojcicki, Atul Gawande, augmented reality, bioinformatics, call centre, Clayton Christensen, clean water, cloud computing, commoditize, computer vision, conceptual framework, connected car, correlation does not imply causation, creative destruction, crowdsourcing, dark matter, data acquisition, disintermediation, disruptive innovation, don't be evil, Edward Snowden, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, Firefox, global village, Google Glasses, Google X / Alphabet X, Ignaz Semmelweis: hand washing, information asymmetry, interchangeable parts, Internet of things, Isaac Newton, job automation, Julian Assange, Kevin Kelly, license plate recognition, lifelogging, Lyft, Mark Zuckerberg, Marshall McLuhan, meta analysis, meta-analysis, microbiome, Nate Silver, natural language processing, Network effects, Nicholas Carr, obamacare, pattern recognition, personalized medicine, phenotype, placebo effect, RAND corporation, randomized controlled trial, Second Machine Age, self-driving car, Silicon Valley, Skype, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, Snapchat, social graph, speech recognition, stealth mode startup, Steve Jobs, the scientific method, The Signal and the Noise by Nate Silver, The Wealth of Nations by Adam Smith, Turing test, Uber for X, uber lyft, Watson beat the top human players on Jeopardy!, WikiLeaks, X Prize

Likewise, the other biologic omes are the proteome, all of your proteins; the metabolome, your metabolites; the microbiome, representing the microbes that coinhabit you; and the epigenome, comprised of the side chains of DNA and how it is packaged. Finally, there’s the exposome, referring to your environment, all that you are exposed to. Collectively, I have coined the term “panoromic” adopted from the word panoramic, meaning lots of information and covering many topics.5 A panoromic view of each individual provides a comprehensive sweep across all the omes relevant to health and medicine. Social Graph and the Phenome The term “social graph” encompasses a dense package of information, including demographics, location, family and friends, friends of friends, interests, likes, education, pets, pictures, videos, and much more. This is precisely the sort of information stored on sites like Facebook, a fact that hasn’t escaped researchers. The prominent mathematician, Stephen Wolfram, the founder of a computation knowledge engine known as Wolfram Alpha, developed a consumer software product known as “personal analytics for Facebook” that within a minute provides a remarkable set of data and graphics about oneself and one’s social network—what Wolfram calls “a dashboard for life.”6,7 If you are a Facebook registrant and haven’t seen this, I encourage you to take a look at yours for free: http://www.wolframalpha.com/facebook/.

We essentially get a phenome from this information—“the composite of an individual’s observable characteristics and traits.”9 Noteworthy is the point that for any given individual, particularly as we age, there is unlikely to be just one phenotype; instead, multiple conditions are likely to be present, which makes one’s phenome not as straightforward as it might appear. For example, blood pressure tends to rise with age, while visual acuity declines. Ideally, someday, we will have all of this data comprehensively collected as the phenome for each individual—the social graph plus the traditional medical record information—and continually updated. While the social graph is subsidiary to the phenome, there’s no question that one’s social network plays an important role in health. Sensors and the Physiome Perhaps the biggest advance in tracking an individual’s information in recent years is the outpouring of an extraordinary number of biosensors. There are now wearable wireless sensors, either commercially available or in clinical development, to capture physiologic data on a smartphone.

privacy and security concerns, 228–230 real-time test results, 121 social networking, 42–44 Smeeth, Liam, 227 Smith, Adam, 42 Snapchat, 221 Snowden, Edward, 219, 225 Snyder, Michael, 88 Social graph of the individual, 81–83 Social media clinical trials through, 212–214 data collection through, 220–221, 223 global spread increasing global autonomy, 47–48 identifying genetic commonalities in disease, 9 importance of online health communities, 10–12 machine learning, 245 managed competition, 51 open-source software, 197 predictive analysis at the individual level, 243–245 social graphs, 82–83 Social networks, 42–44 Soon-Shion, Patrick, 203–204 Spatiotemporal applications, 79–80 Spinal fusion, 214–215 Splinter, Mike, 175 Spontaneous coronary artery dissection (SCAD), 211–212 The Sports Gene (Epstein), 94 Sports injuries, 94–95 Stanford University, 112 Star Trek (television program), 286 Statins, 31–33 Stephens, Richard, 226–227 Stethoscope, 96, 119–120, 175–176, 253–256, 276, 289 Stone, Neil, 33 Sudden infant death syndrome (SIDS), 92, 103 Supreme Court, US, 74–76 Surgery Center of Oklahoma, 152–153 Surveillance, 219–224 Take Care clinics, 163 Target, 224, 239 Targeted marketing, 221–225, 243 Tay-Sachs disease, 89 TechFreedom, 69 Technion Institute, Haifa, Israel, 110 Technology adoption, 7(fig.)


pages: 270 words: 79,180

The Middleman Economy: How Brokers, Agents, Dealers, and Everyday Matchmakers Create Value and Profit by Marina Krakovsky

Affordable Care Act / Obamacare, Airbnb, Al Roth, Ben Horowitz, Black Swan, buy low sell high, Chuck Templeton: OpenTable:, Credit Default Swap, cross-subsidies, crowdsourcing, disintermediation, diversified portfolio, experimental economics, George Akerlof, Goldman Sachs: Vampire Squid, income inequality, index fund, information asymmetry, Jean Tirole, Joan Didion, Kenneth Arrow, Lean Startup, Lyft, Marc Andreessen, Mark Zuckerberg, market microstructure, Martin Wolf, McMansion, Menlo Park, Metcalfe’s law, moral hazard, multi-sided market, Network effects, patent troll, Paul Graham, Peter Thiel, pez dispenser, ride hailing / ride sharing, Robert Metcalfe, Sand Hill Road, sharing economy, Silicon Valley, social graph, supply-chain management, TaskRabbit, The Market for Lemons, too big to fail, trade route, transaction costs, two-sided market, Uber for X, uber lyft, ultimatum game, Y Combinator

Nozad might balk at this abstract, overly mathematical depiction of the ties between people, but it’s a common way to look at human connections, especially in our Web 2.0 era. When Mark Zuckerberg or Jeff Weiner talk about the “social graph,” this is what they mean, except they’re referring to users of Facebook or LinkedIn. The points, or nodes, represent individual people, while the lines or links represent the social ties between the individuals.12 Our social graphs from the online world are often a crude replica of our actual social networks. Just think of the people you may be close to who don’t use social media. (Some of your close relatives, whom you talk to by phone, may not be on Facebook at all, and your own kids might be following half their classmates on Instagram, but not you.) Conversely, think of all your LinkedIn connections whom you last saw two jobs ago, if at all. Your social graphs online certainly overlap with your real-world network, but they aren’t the same thing.

For example, graph theory in computer science uses the term “edge” for what social scientists call a “tie.” 13.For example, see Nathan Eagle, Sandy Pentland, and David Lazer, “Inferring Friendship Network Structure by Using Mobile Phone Data,” Proceedings of the National Academy of Sciences 106, no. 36 (2009): 15274–78. 14.Sociologists also use the term “sociogram,” whereas computer scientists favor “social graph.” Both refer to a network diagram. 15.See, for example, Anatol Rapoport and William J. Horvath, “A Study of a Large Sociogram,” Behavioral Science 6, no. 4 (October 1961): 279–91, and Carlo Morselli, Inside Criminal Networks (New York: Springer, 2009). 16.This is the principle of homophily: birds of a feather flock together. 17.Mark Granovetter, Getting A Job: A Study of Contacts and Careers (Cambridge, MA: Harvard University Press, 1974) and “The Strength of Weak Ties,” American Journal of Sociology 78, no. 6 (May 1973): 1360–1380. 18.Ronald Burt, Structural Holes: The Social Structure of Competition (Harvard University Press, 1992). 19.Sociologists most often use the term “network broker” or simply “broker” to describe this person.

., 15–17, 203 Reinhard Model and Talent Agency, 105, 107–8 repairman problem, 141–2 reputation brands and, 67 Certifiers and, 47, 51–2, 61–7, 72 Concierges and, 164, 166 Enforcers and, 81–6, 88–9, 92–3, 96, 101–3, 107 game theory and, 13 helpfulness and, 22 Insulators and, 179, 182–6 management of, 89 Risk Bearers and, 124, 126, 132–3, 142–3 value of, 62–5 watchdogs and, 85–6 risk Certifiers and, 51–2, 72 diversification and, 138 embracing external risk, 125–8 Enforcers and, 77, 85–6, 96, 108–9 heroes and, 133 Insulators and, 178, 183 internal vs. external, 123 investment and, 10, 13, 124–5 loyalty and, 77 online, 134–6 pooling of, 137–42 reducing, 113–20 sharing, 120–3 shifting, 111–13 supply-side vs. demand-side, 137–8 see also Risk Bearers Risk Bearers art world and, 113–17 benefiting from power laws, 128–9 contrarianism and, 132–3 explained, 7 external risk and, 125–8 fish market and, 117–20 humility and, 130–2 micro-VCs, 133–6 overview, 111–13 promise and perils of sharing risk, 120–3 role, 111 unpredictability and, 136–43 VCs and, 124–5 Robboy, Howard, 54–6, 66, 68, 70, 72 Robin Hood effect, 192 Rocky Mountain Home Staging, 193–4 Rosenhaus, Drew, 173–5, 182–3, 185–6, 190 Roth, Al, 176, 180, 194 Rysman, Marc, 38 San Francisco 49ers, 179 scale economies of, 142, 167 pooling and, 139 returns to, 43 Scheibehenne, Benjamin, 155 Schwartz, Barry, 155 Scott, Jeff, 186–90, 194 scouting, 51–7 Sears, 140 Sequoia Capital, 18–21, 125 Setai hotel, 148–9 Shamon, Carol, 67, 190 Shapiro, Carl, 63 Shapiro, Ron, 186 Shark Never Sleeps, A (Rosenhaus), 174 Shark Tank (TV series), 11 Shirky, Clay, 134 Shropshire, Kenneth, 186 Simon, Herbert, 154–6 simplicity, 168–70, 172 SitterCity, 36–46, 80, 130, 164 smuggling, 174 social graph, 23 Society of Actuaries, 122 Sports Illustrated, 173 Spulber, Daniel, 5 Steiner, Robert L., 9 stereotypes, 10 Stoxstill-Diggs, LaJuan, 30–5, 44–6, 64–5 structural holes, 25–7, 29, 44 Taibbi, Matt, 10 TaskRabbit, 5–6, 36, 38, 91, 124, 132 Templeton, Chuck, 79, 81, 83, 108 TheFunded.com, 133 Thiel, Peter, 127, 129 Thiers, Genevieve, 36–7, 39–44, 130, 164 Toys ‘R’ Us, 137 Travel + Leisure, 146 Travelocity, 145 Trident Media Group, 69 TripAdvisor, 158 Trulia, 4 Tsukiji, Japan, 117–19 Twitter, 4, 25, 124, 127–9, 134 two-sided markets balancing, 198–9 Bridges as, 38–46 economic theory and, 13 Enforcers and, 80, 84, 109 Uber, 5, 36, 38, 91, 126, 136–7, 140–1, 198, 203 Uganda, 75 Ultimatum Game, 183 unpredictability, 50, 68, 112, 116, 138–42 see also risk US Bureau of Labor Statistics, 2 venture capitalists (VCs) as Bridges, 20, 26–7, 29, 43 contrarianism and, 132–3 entrepreneurs and, 20, 29, 132–3 founders and, 26 humility and, 130–2 as middlement, 6, 13 micro-VCs, 133–6, 197 power laws and, 128–9 as Risk Bearers, 124–33, 138 VRBO.com, 90–2 W.


pages: 247 words: 81,135

The Great Fragmentation: And Why the Future of All Business Is Small by Steve Sammartino

3D printing, additive manufacturing, Airbnb, augmented reality, barriers to entry, Bill Gates: Altair 8800, bitcoin, BRICs, Buckminster Fuller, citizen journalism, collaborative consumption, cryptocurrency, David Heinemeier Hansson, disruptive innovation, Elon Musk, fiat currency, Frederick Winslow Taylor, game design, Google X / Alphabet X, haute couture, helicopter parent, illegal immigration, index fund, Jeff Bezos, jimmy wales, Kickstarter, knowledge economy, Law of Accelerating Returns, lifelogging, market design, Metcalfe's law, Minecraft, minimum viable product, Network effects, new economy, peer-to-peer, post scarcity, prediction markets, pre–internet, profit motive, race to the bottom, random walk, Ray Kurzweil, recommendation engine, remote working, RFID, Rubik’s Cube, self-driving car, sharing economy, side project, Silicon Valley, Silicon Valley startup, skunkworks, Skype, social graph, social web, software is eating the world, Steve Jobs, survivorship bias, too big to fail, US Airways Flight 1549, web application, zero-sum game

How does business go about doing business if we throw out the old methods of going to the market and marketing to people? (Mind you, marketing isn’t evil. It’s beautiful and powerful, it’s just that it needs to be used in a more human way.) If marketers embrace this philosophy, we’ll all end up better off after the interaction. The way we replace demographics is with social and interest graphs. Social graphs The social graph is the network that results from relationships that are digitally facilitated and maintained through virtual connections, which can now be spread more quickly using social-media tools. These connections are, theoretically, easier to make and easier to maintain than when our connection methods were all physical in nature. Interest graphs The interest graph is the online representation of the stuff we really care about.

INDEX 3D printing access and accessibility see also barriers; communication; digital; social media — factors of production — knowledge adoption rates advertising see also marketing; mass media; promotion; television Airbnb Alibaba Amazon antifragility Apple artisanal production creativity audience see also crowd — connecting with — vs target Away from Keyboard (AFK) banking see also crowdfunding; currencies barriers Beck (musician) big as a disadvantage bioengineering biomimicry biotechnology bitcoins blogs borrowed interest brand business strategies change see disruption and disruptive change Cluetrain Manifesto co-creation coffee culture Cold War collaboration collaborative consumption collective sentience commerce, future see also retail and retailers communication see also advertising; promotion; social media; social relationships — channels — tools community vs target competition and competitors component retail computers see also connecting and connection; internet; networks; smartphones; social media; software; technology era; 3D printing; web connecting and connection see also social media; social relationships — home/world — machines — people — things consumerism consumption silos content, delivery of coopetition corporations see also industrial era; retail and retailers; technology era costs see also finance; price co-working space creativity crowd, contribution by the crowdfunding cryptocurrencies culture — hacking — startup currencies see also banking deflation demographics device convergence digital see also computers; internet; music; smartphone; retail and retailers, online; social media; social relationships; technology; web; work — cohorts — era — footprint — revolution — skills — strategy — tools — world disruption and disruptive change DNA as an operating system drones Dunbar's number e-commerce see retail and retailers, online economic development, changing education employment, lifetime see also labour; work ephermalization Facebook see also social media finance, peer to peer see also banking; crowdfunding; currencies Ford, Henry 4Ps Foursquare fragmentation — of cities — industrial — Lego car example gadgets see also computers; smartphone; tools games and gaming behaviour gamification geo-location glass cockpit Global Financial Crisis (GFC) globalisation Google hacking hourglass strategy IFTTT (If this then that) industrialists (capital class) industry, redefining industrial era see also consumerism; marketing; retail and retailers — hacking — life in influencers information-based work infrastructure — changing — declining importance of — legacy innovation intention interest-based groups see also niches interest graphs internet see also access and accessibility; connecting and connection; social media; social relationships; web Internet.org In Real Life (IRL) isolation iTunes see also music Jumpstart Our Business Startups (JOBS) Act (USA) keyboards knowledge economy lab vs factory labour see also work — low-cost language layering legacy — industries — infrastructure — media Lego car project life — in boxes — in gaming future — hack living standards see also life location see place, work making see also artisanal production; retail and retailers; 3D printing malleable marketplace manufacturing see also artisanal production; industrial era; making; product; 3D printing; tools — desktop marketing see also advertising; consumerism; 4Ps; mass media; promotion; retail and retailers — demographics, use in — industrial era — language — mass — metrics — new — post-industrial — predictive — research — target — traditional mass media ; see also advertising; marketing; media; promotion; television — after materialism media see also communication; legacy; mass media; newspapers; niches; television — consumption — hacking — platform vs content — subscription Metcalfe's law MOOC (Massive Open Online Course) Moore's law music Napster Netflix netizens networks see also connecting and connection; media; social media; social relationships newspapers see also media niches nodes nondustrial company Oaida, Raul oDesk office, end of the omniconnection era open source parasocial interaction payment systems Pebble phones, number of mobile see also smartphones photography Pinterest piracy place — of work platforms pop culture power-generating technologies price see also costs privacy see also social media; social relationships product — unfinished production see also industrial era; product; 3D printing — mass projecteer Project October Sky promotion see also advertising; marketing; mass media; media quantified self Racovitsa, Vasilii remote controls RepRap 3D printer retail cold spot retail and retailers — changing — digital — direct — hacking — mass — online — price — small — strategies — traditional rewards robots Sans nation state economy scientific management search engines self-hacking self-publishing self-storage sensors sharing see also social media; social relationships smartphones smartwatch social graphs social media (digitally enhanced conversation) see also Facebook; social relationships; Twitter; YouTube social relationships see also social graphs; social media — digital software speed subcultures Super Awesome Micro Project see Lego car project Super Bowl mentality target tastemakers technology see also computers; digital; open source; social media; smartphones; social relationships; software; 3D printing; work — deflation — era — free — revolution — speed — stack teenagers, marketing to television Tesla Motors thingernet thinking and technology times tools see also artisanal production; communication; computers; digital; making; smartphones; social media; 3D printing — changing — old trust Twitter Uber unlearning usability gap user experience volumetric mindset wages — growth — low — minimum web see also connecting and connection; digital; internet; retail and retailers, online; social media; social relationships — three phases of — tools Wikipedia work — digital era — industrial era — location of — options words see language Yahoo YouTube Learn more with practical advice from our experts WILEY END USER LICENSE AGREEMENT Go to www.wiley.com/go/eula to access Wiley’s ebook EULA.

It’s based on the real values we have and the things we do and support, hence forming a more genuine identity. The interest graph matters because it doesn’t just track the activity undertaken by people, but also what they hope to do — where they want to go, what they want to buy, who they want to follow and meet, and what they want to change. Social + interests = intention It gets interesting where these two ideas intersect. The overlaying of the social graph and the interest graph tells us much about a person’s intentions. When people develop relationships based on a connection of interests facilitated by social-media connections we can see the true predictive persona. It’s actually how the best and most enduring relationships have always been formed; it’s just that now we can form them more quickly, develop larger cohorts and there’s less luck involved in finding similar souls.


pages: 39 words: 4,665

Data Source Handbook by Pete Warden

en.wikipedia.org, Menlo Park, openstreetmap, phenotype, social graph

ISBN: 978-1-449-30314-3 [LSI] 1295970672 Table of Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Data Source Handbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Websites WHOIS Blekko bit.ly Compete Delicious BackType PagePeeker People by Email WebFinger Flickr Gravatar Amazon AIM FriendFeed Google Social Graph MySpace Github Rapleaf Jigsaw People by Name WhitePages LinkedIn GenderFromName People by Account Klout Qwerly Search Terms BOSS 1 1 2 3 3 4 5 5 5 6 6 6 7 7 8 8 9 10 10 11 11 11 11 11 12 12 12 12 13 v Blekko Bing Google Custom Search Wikipedia Google Suggest Wolfram Alpha Locations SimpleGeo Yahoo! Google Geocoding API CityGrid Geocoder.us Geodict GeoNames US Census Zillow Neighborhoods Natural Earth US National Weather Service OpenStreetMap MaxMind Companies CrunchBase ZoomInfo Hoover’s Yahoo!

,"id":"twitter"}, {"username":"tadghin","name":"YouTube","url":"http://www.youtube.com/", "profileUrl":"http://www.youtube.com/profile?user=tadghin", "iconUrl":"...","id":"youtube"}, {"url":"http://www.facebook.com/","iconUrl":"...","id":"facebook", "profileUrl":"http://www.facebook.com/profile.php?id=544591116", "name":"Facebook"}], "nickname":"timoreilly","id":"d85e8470-25c5-11dd-9ea1-003048343a40"} Google Social Graph Though it’s an early experiment that’s largely been superseded by Webfinger, this Google API can still be useful for the rich connection information it exposes for signedup users. Unfortunately, it’s not as well-populated as you might expect. It doesn’t require any developer keys to access: 8 | Data Source Handbook curl "http://socialgraph.apis.google.com/lookup?\ q=mailto%3asearchbrowser%40gmail.com&fme=1&edi=1&edo=1&pretty=1&sgn=1&callback=" { "canonical_mapping": { "mailto:searchbrowser@gmail.com": "sgn://mailto/?


pages: 299 words: 91,839

What Would Google Do? by Jeff Jarvis

23andMe, Amazon Mechanical Turk, Amazon Web Services, Anne Wojcicki, barriers to entry, Berlin Wall, business process, call centre, cashless society, citizen journalism, clean water, commoditize, connected car, credit crunch, crowdsourcing, death of newspapers, different worldview, disintermediation, diversified portfolio, don't be evil, fear of failure, Firefox, future of journalism, G4S, Google Earth, Googley, Howard Rheingold, informal economy, inventory management, Jeff Bezos, jimmy wales, Kevin Kelly, Mark Zuckerberg, moral hazard, Network effects, new economy, Nicholas Carr, old-boy network, PageRank, peer-to-peer lending, post scarcity, prediction markets, pre–internet, Ronald Coase, search inside the book, Silicon Valley, Skype, social graph, social software, social web, spectrum auction, speech recognition, Steve Jobs, the medium is the message, The Nature of the Firm, the payments system, The Wisdom of Crowds, transaction costs, web of trust, WikiLeaks, Y Combinator, Zipcar

Scott Heiferman, founder of Meetup, also brought historical perspective to the discussion, writing a brief manifesto for change in the coming decade, chock full of hip blog references (the “social graph” to which he refers is what Mark Zuckerberg calls the architecture of personal connections on Facebook): Historically, when people are free to assemble & associate, they self-organize insurance, cooperatively. Later it became the centralized, professionalized industry we know today. I predict there’ll be some kind of massive craigslistification of insurance by April 27, 2018. It’s about de-institutionalization—not from the government borg (social security), not from the corporate borg (AIG). The New Social [graph] Security. Decentralized, self-organized. Not just DIY, but DIO (Do It Ourselves). That is the big theme for everything now. There is the great promise and power of the Google age: DIO.

Let that sip of rhetorical cabernet roll around on the palate for a minute. Elegant organization. When you think about it, that is precisely what Zuckerberg brought to Harvard—then other universities, then the rest of the world—with his social platform. Harvard’s community had been doing what it wanted to do for more than three centuries before Zuckerberg came along. He just helped them do it better. Facebook enabled people to organize their social networks—the social graph, he calls it: who they are, what they do, who they know, and, not unimportantly, what they look like. It was an instant hit because it met a need. It organized social life at Harvard. At this Davos meeting (which was off the record, but Zuckerberg gave me permission to blog it), he told the story of his Harvard art course. Zuckerberg didn’t have time to attend a single class or to study. After all, he was busy founding a $15 billion company.

See FARC The Revolution Will Not Be Televised (Trippi), 238 Rheingold, Howard, 106 Richardson, Will, 211 Rose, Kevin, 4, 132, 134 Rosenberg, Jonathan, 217 Rosen, Jay, 134–35 Roussel, Edward, 123 Rubel, Steve, 223 Rusbridger, Alan, 126 Rushkoff, Douglas, 226 Ryanair, 79 Ryan, Pat, 64 Salesforce.com, 62 Sandberg, Sheryl, 94 SANS, 180 scale, 54–57 Schmidt, Eric on Carr, 235 on Gmail, 6 on home-page sponsors, 36 on mistakes, 94 on mobile market, 79 Scion, 174 Scoble, Robert, 150 search-engine optimization (SEO), 41–42 rules, 44–45 search engines, 5, 20 Searls, Doc, 3, 82, 96–97, 149, 170 VRM and, 201–2 secrets, 97 Seed Camp, 193 Seesmic.com, 142 Segal, Rick, 15, 95 self-publishing, 73 self-searches, 20 Semel, Terry, 81 SEO. See search-engine optimization Sequoia Capital, 189 Shardanand, Upendra, 35 Shirky, Clay, 50, 60, 151, 191–92, 197, 235–36, 237 Silverman, Dwight, 13 simplicity, 114–16, 236 SimplyHired.com, 39 Sirius Satellite Radio, 131 Skype, 31, 50 Smart Mobs (Rheingold), 106 Smith, Quincy, 38 Smolan, Rick, 140 social business, 158 social graph, 49 socialization, 211–12 social-media, 172–73 social responsibility, 47 social web, 51 Sorrell, Martin, 42 Sourcetool.com, 100 specialization, 26–27, 154 speed, 103–4, 105–6 Spitzer, Eliot, 96 splogs, 43 Starbucks, 60–62 Stern, Howard, 95, 131–32 Stewart, Jon, 95–96 StudieVZ, 50 Supreme Court, 225 Surowiecki, James, 88 talent, 146, 240 Tapscott, Don, 113, 151, 225 targeting, 151, 179–80 teaching, 193, 214–15 teamwork, 217 TechCrunch, 107, 192 Technorati, 15, 20 TechTV, 132 telecommunications, 165–71 Telegraph Media Group, 123 television, 84 cable, 167 decline of, 65–66 listings, 109–10 networks, 135 Television Without Pity, 135 Tesco, 179 Tesla Motors, 175 testing, 214 Threadbanger, 180 Threadless, 57 TimesSelect, 78 Time Warner, 80–81 Tobaccowala, Rishad, 114, 121–22, 145–48, 151, 177 on Apple, 228 toilet paper, 180–81 TomEvslin.com, 31 Toto, 181 Toyota, 174–75 transparency, 83, 97–98 journalism and, 92 PR and, 223 Tribune Company, 129 Trippi, Joe, 238 trust, 74, 170 control v., 82–83 in customers, 83–84 Tumblr, 192 Turner, Ted, 134 TV Guide, 109–10 20 percent rule, 111, 114 23andMe, 205 Twitter, 20, 126 Dell and, 46 mobs and, 107 real time and, 105–6 Tyndall, Andrew, 220 Union Square Ventures, 30 University of Phoenix, 217 Updike, John, 138 The Vanishing Newspaper (Meyer), 125 Vardi, Yossi, 31–32 Vaynerchuk, Gary, 107, 157–61 VC.


pages: 366 words: 94,209

Throwing Rocks at the Google Bus: How Growth Became the Enemy of Prosperity by Douglas Rushkoff

activist fund / activist shareholder / activist investor, Airbnb, algorithmic trading, Amazon Mechanical Turk, Andrew Keen, bank run, banking crisis, barriers to entry, bitcoin, blockchain, Burning Man, business process, buy and hold, buy low sell high, California gold rush, Capital in the Twenty-First Century by Thomas Piketty, carbon footprint, centralized clearinghouse, citizen journalism, clean water, cloud computing, collaborative economy, collective bargaining, colonial exploitation, Community Supported Agriculture, corporate personhood, corporate raider, creative destruction, crowdsourcing, cryptocurrency, disintermediation, diversified portfolio, Elon Musk, Erik Brynjolfsson, Ethereum, ethereum blockchain, fiat currency, Firefox, Flash crash, full employment, future of work, gig economy, Gini coefficient, global supply chain, global village, Google bus, Howard Rheingold, IBM and the Holocaust, impulse control, income inequality, index fund, iterative process, Jaron Lanier, Jeff Bezos, jimmy wales, job automation, Joseph Schumpeter, Kickstarter, loss aversion, Lyft, Marc Andreessen, Mark Zuckerberg, market bubble, market fundamentalism, Marshall McLuhan, means of production, medical bankruptcy, minimum viable product, Mitch Kapor, Naomi Klein, Network effects, new economy, Norbert Wiener, Oculus Rift, passive investing, payday loans, peer-to-peer lending, Peter Thiel, post-industrial society, profit motive, quantitative easing, race to the bottom, recommendation engine, reserve currency, RFID, Richard Stallman, ride hailing / ride sharing, Ronald Reagan, Satoshi Nakamoto, Second Machine Age, shareholder value, sharing economy, Silicon Valley, Snapchat, social graph, software patent, Steve Jobs, TaskRabbit, The Future of Employment, trade route, transportation-network company, Turing test, Uber and Lyft, Uber for X, uber lyft, unpaid internship, Y Combinator, young professional, zero-sum game, Zipcar

Digital networks simulate the very same human social dynamics fueling the communities of artists like Palmer in order to generate goodwill and mass excitement for their corporate clients. It’s a one-sided, highly controlled relationship in which, invariably, the platforms and companies with which we engage learn more about us than we ever learn about them. Social marketing creates the illusion of a natural, nonmarketed groundswell of interest and, more importantly, provides marketers with a map of social connections and influences. These social graphs, as they’re called in the industry, are the fundamental building blocks of big data companies’ analyses. Big data is worth more than the sum of its parts. It is the technology for solving everything from terrorism to tuberculosis, as well as the purported payoff for otherwise unprofitable tech businesses, from smartphones to video games. Like pop stars, these health, entertainment, and content “plays” will make no money on their own—but the data they can glean from their users will be gold to marketers.

., 65 austerity, 136–37 auto attendants, 14 Bandcamp, 29–30 Barber, Brad, 177 Barnes & Noble, 83, 87 barter, 127 barter exchanges, 159 Basecamp, 59–60 BASF, 107 Battle-Bro, 121 Bauwens, Michael, 221 bazaars, 16–18 money and, 127 obsolescence of, caused by corporations, 70–71 Bell, Daniel, 53 Belloc, Hilaire, 229 benefit corporations, 119 Ben & Jerry’s, 80, 205 Benna, Ted, 171 BerkShare, 154–55 Best Buy, 90 Bezos, Jeff, 90, 92–93 Biewald, Lukas, 49–50 big data, 39–44 data point collection and comparisons of, 41–42 game changing product invention reduced by reliance on, 43 predicting future choices, as means of, 41, 42–43 reduction in spontaneity of customers and, 43 social graphs and, 40 suspicion of, as increasing value of data already being sold, 43–44 traditional market research, distinguished, 41 Big Shift, 76 biopiracy, 218 biotech crash of 1987, 6 Bitcoin, 143–49, 150–51, 152, 219, 222 BitTorrent, 142–43, 219 Blackboard, 95–96 Blackstone Group, 115 black swans, 183 blockchain, 144–51, 222 Bitcoin, 144, 145, 146, 147, 149, 222 decentralized autonomous corporations (DACs) and, 149–50 Blogger, 8, 31 Bloomberg, 182 Bodie, Zvie, 174 Borders, 83, 87 bot programs, 37 bounded investing, 210–15 Bovino, Beth Ann, 81–82 Brand, Russell, 36 branding, 20 social, and “likes” economy, 35–37 Branson, Richard, 121 Brin, Sergey, 92–93 Bristol Pounds, 156 British East India Company, 71–72 Brixton Pounds, 156 brokered barter system, 127 Brynjolfsson, Erik, 23, 53 Buffett, Warren, 168, 209 burn rate, 190 Bush, Jeb, 227–28 Calacanis, Jason, 201 Calvert, 209–10 Campbell Soup Company, 119 capital.

Bean, 80 local currencies, 154–65 cooperative community currencies, 160–65 free money theory, currencies based on, 156–59 local multiplier effect, 155 Long Tail theory, 26, 33 low-profit limited liability company (L3C), 120–21 Luckett, Oliver, 35–36 Lyft, 45, 47, 87 Machine Learning lab, 90–91 McAfee, Andrew, 23, 53 McCluhan, Marshall, 229 McKenna, Terence, 234 McLuhan, Marshall, 69 Magic Eraser Duo, 107 Maker’s Row, 30 malware, 37 marketing. See also advertising big data and, 42 branding and, 20, 35–37 “likes” economy and, 35–37 mass, 19–20 social graphs generated by, 40 market makers, 178–79 market money, 127–28, 130 Marx, Karl, 83, 138 mass media, 20–21 maturity, 98 Mecklenburg, George, 159 medical debt, 153 Meetup, 196–97 microfinancing platforms, 202–4 Microsoft, 83 Microventures, 202–3 Mill, John Stuart, 135 mining, of bitcoins, 145, 147 MIT Technology Review,53 Mondragon Corporation, 220, 222 money basket of commodities approach to backing of, 139 blockchains and, 144–51, 222 central currency system and (See central currency system) cooperative currencies, 160–65 debt and, 152–54 digital transaction networks and, 140–51 extractive purpose of, 128–31 free money theory, currencies based on, 156–59 gold standard and, 139 grain receipts, 128 history of, 126–31 local currencies, 154–65 manipulating human financial behavior to serve, 151–52 market, 127–28, 130 operating system nature of centrally-issued, 125–26 outlawing of local currencies and replacement with coin of the realm, 128–29 precious metals and, 128 reprogramming of, 138–51 traditional bank’s role in serving communities, 165–67 traditional purpose of, 126 as unbound, 212–13 velocity of, 140–41 monopolies chartered, 18, 56, 70, 101, 125, 131 platform, 82–93, 101 power-law dynamics and, 27–28 Monsanto, 218 Morgan Stanley, 195 Mozilla Corporation, 122–23 Mozilla Foundation, 122–23 Mr.


pages: 224 words: 64,156

You Are Not a Gadget by Jaron Lanier

1960s counterculture, accounting loophole / creative accounting, additive manufacturing, Albert Einstein, call centre, cloud computing, commoditize, crowdsourcing, death of newspapers, different worldview, digital Maoism, Douglas Hofstadter, Extropian, follow your passion, hive mind, Internet Archive, Jaron Lanier, jimmy wales, John Conway, John von Neumann, Kevin Kelly, Long Term Capital Management, Network effects, new economy, packet switching, PageRank, pattern recognition, Ponzi scheme, Ray Kurzweil, Richard Stallman, Silicon Valley, Silicon Valley startup, slashdot, social graph, stem cell, Steve Jobs, Stewart Brand, Ted Nelson, telemarketer, telepresence, The Wisdom of Crowds, trickle-down economics, Turing test, Vernor Vinge, Whole Earth Catalog

The whole artifice, the whole idea of fake friendship, is just bait laid by the lords of the clouds to lure hypothetical advertisers—we might call them messianic advertisers—who could someday show up. The hope of a thousand Silicon Valley start-ups is that firms like Face-book are capturing extremely valuable information called the “social graph.” Using this information, an advertiser might hypothetically be able to target all the members of a peer group just as they are forming their opinions about brands, habits, and so on. Peer pressure is the great power behind adolescent behavior, goes the reasoning, and adolescent choices become life choices. So if someone could crack the mystery of how to make perfect ads using the social graph, an advertiser would be able to design peer pressure biases in a population of real people who would then be primed to buy whatever the advertiser is selling for their whole lives. The situation with social networks is layered with multiple absurdities.

The advertising idea hasn’t made any money so far, because ad dollars appear to be better spent on searches and in web pages. If the revenue never appears, then a weird imposition of a database-as-reality ideology will have colored generations of teen peer group and romantic experiences for no business or other purpose. If, on the other hand, the revenue does appear, evidence suggests that its impact will be truly negative. When Facebook has attempted to turn the social graph into a profit center in the past, it has created ethical disasters. A famous example was 2007’s Beacon. This was a suddenly imposed feature that was hard to opt out of. When a Facebook user made a purchase anywhere on the internet, the event was broadcast to all the so-called friends in that person’s network. The motivation was to find a way to package peer pressure as a service that could be sold to advertisers.


pages: 903 words: 235,753

The Stack: On Software and Sovereignty by Benjamin H. Bratton

1960s counterculture, 3D printing, 4chan, Ada Lovelace, additive manufacturing, airport security, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, algorithmic trading, Amazon Mechanical Turk, Amazon Web Services, augmented reality, autonomous vehicles, basic income, Benevolent Dictator For Life (BDFL), Berlin Wall, bioinformatics, bitcoin, blockchain, Buckminster Fuller, Burning Man, call centre, carbon footprint, carbon-based life, Cass Sunstein, Celebration, Florida, charter city, clean water, cloud computing, connected car, corporate governance, crowdsourcing, cryptocurrency, dark matter, David Graeber, deglobalization, dematerialisation, disintermediation, distributed generation, don't be evil, Douglas Engelbart, Douglas Engelbart, Edward Snowden, Elon Musk, en.wikipedia.org, Eratosthenes, Ethereum, ethereum blockchain, facts on the ground, Flash crash, Frank Gehry, Frederick Winslow Taylor, future of work, Georg Cantor, gig economy, global supply chain, Google Earth, Google Glasses, Guggenheim Bilbao, High speed trading, Hyperloop, illegal immigration, industrial robot, information retrieval, Intergovernmental Panel on Climate Change (IPCC), intermodal, Internet of things, invisible hand, Jacob Appelbaum, Jaron Lanier, Joan Didion, John Markoff, Joi Ito, Jony Ive, Julian Assange, Khan Academy, liberal capitalism, lifelogging, linked data, Mark Zuckerberg, market fundamentalism, Marshall McLuhan, Masdar, McMansion, means of production, megacity, megastructure, Menlo Park, Minecraft, MITM: man-in-the-middle, Monroe Doctrine, Network effects, new economy, offshore financial centre, oil shale / tar sands, packet switching, PageRank, pattern recognition, peak oil, peer-to-peer, performance metric, personalized medicine, Peter Eisenman, Peter Thiel, phenotype, Philip Mirowski, Pierre-Simon Laplace, place-making, planetary scale, RAND corporation, recommendation engine, reserve currency, RFID, Robert Bork, Sand Hill Road, self-driving car, semantic web, sharing economy, Silicon Valley, Silicon Valley ideology, Slavoj Žižek, smart cities, smart grid, smart meter, social graph, software studies, South China Sea, sovereign wealth fund, special economic zone, spectrum auction, Startup school, statistical arbitrage, Steve Jobs, Steven Levy, Stewart Brand, Stuxnet, Superbowl ad, supply-chain management, supply-chain management software, TaskRabbit, the built environment, The Chicago School, the scientific method, Torches of Freedom, transaction costs, Turing complete, Turing machine, Turing test, undersea cable, universal basic income, urban planning, Vernor Vinge, Washington Consensus, web application, Westphalian system, WikiLeaks, working poor, Y Combinator

None of the other digital currency projects is built on the core currency over which Facebook still has privileged position of access: the microeconomies of recognized social debt from which the value of money is primordially derived (at least for humans; HST algorithms are a different story).55 But to date, Facebook's furtive and ill-conceived experiments at the monetization of that capital are based on strategies of rent more than mediation, such as reciprocal likes, selling post promotion, charging users to message each other, and so essentially taxing the graph's own growth. This may be a doubtful recipe for the conversion of public obligation into private money and back again. The magical ontology of money requires a trust so trusting that it requires no deliberation, and while social graph-based platforms may be where new currencies will be sustained in the long run, Facebook may have soiled its own punch, and so perhaps we'll see banks becoming social graph platforms before we see graph platforms becoming banks. Still Facebook is the most widely engaged social media site with well over a billion active users and so its potential for structuring human communication according to its own logics of platform sovereignty remains profound. 30.  Apple Apple has assumed a mantle from Disney for preeminence in mass-scale, closed-loop experience design.56 By comparison with the extractive micromanagement style of Facebook, Apple's closed world is ruled with luxury carrots more than with behaviorist sticks.

Today's political geographic conflicts are often defined as exceptions to that normal model, and many are driven, enabled, or enforced in significant measure by planetary computation: byzantine international and subnational bodies, a proliferation of enclaves and exclaves, noncontiguous states, diasporic nationalisms, global brand affiliations, wide-scale demographic mobilization and containment, free trade corridors and special economic zones, massive file-sharing networks both legal and illegal, material and manufacturing logistical vectors, polar and subpolar resource appropriations, panoptic satellite platforms, alternative currencies, atavistic and irredentist religious imaginaries, cloud data and social-graph identity platforms, big data biopolitics of population medicine, equities markets held in place by an algorithmic arms race of supercomputational trading, deep cold wars over data aggregation across state and party lines, and so on. In relation to the incommensurate demands of diverse protocols, these rewrite and redivide the spaces of geopolitics in ways that are inclusive of aerial volumes, atmospheric envelopes, and oceanic depths.

The Stack discussed in the following chapters is a vast software/hardware formation, a proto-megastructure built of crisscrossed oceans, layered concrete and fiber optics, urban metal and fleshy fingers, abstract identities and the fortified skins of oversubscribed national sovereignty. It is a machine literally circumscribing the planet, which not only pierces and distorts Westphalian models of state territory but also produces new spaces in its own image: clouds, networks, zones, social graphs, ecologies, megacities, formal and informal violence, weird theologies, all superimposed one on the other. This aggregate machine becomes a systematic technology according to the properties and limitations of that very spatial order. The layers of The Stack, some continental in scale and others microscopic, work in specific relation to the layer above and below it. As I have suggested, the fragile complementarity between the layers composing The Stack is discussed both as an idealized model for how platforms may be designed and as a description of some of the ways that they already work now.


pages: 285 words: 86,853

What Algorithms Want: Imagination in the Age of Computing by Ed Finn

Airbnb, Albert Einstein, algorithmic trading, Amazon Mechanical Turk, Amazon Web Services, bitcoin, blockchain, Chuck Templeton: OpenTable:, Claude Shannon: information theory, commoditize, Credit Default Swap, crowdsourcing, cryptocurrency, disruptive innovation, Donald Knuth, Douglas Engelbart, Douglas Engelbart, Elon Musk, factory automation, fiat currency, Filter Bubble, Flash crash, game design, Google Glasses, Google X / Alphabet X, High speed trading, hiring and firing, invisible hand, Isaac Newton, iterative process, Jaron Lanier, Jeff Bezos, job automation, John Conway, John Markoff, Just-in-time delivery, Kickstarter, late fees, lifelogging, Loebner Prize, Lyft, Mother of all demos, Nate Silver, natural language processing, Netflix Prize, new economy, Nicholas Carr, Norbert Wiener, PageRank, peer-to-peer, Peter Thiel, Ray Kurzweil, recommendation engine, Republic of Letters, ride hailing / ride sharing, Satoshi Nakamoto, self-driving car, sharing economy, Silicon Valley, Silicon Valley ideology, Silicon Valley startup, social graph, software studies, speech recognition, statistical model, Steve Jobs, Steven Levy, Stewart Brand, supply-chain management, TaskRabbit, technological singularity, technoutopianism, The Coming Technological Singularity, the scientific method, The Signal and the Noise by Nate Silver, The Structural Transformation of the Public Sphere, The Wealth of Nations by Adam Smith, transaction costs, traveling salesman, Turing machine, Turing test, Uber and Lyft, Uber for X, uber lyft, urban planning, Vannevar Bush, Vernor Vinge, wage slave

The place in which these machines reside is the human mind.11 This is precisely the apotheosis that Bogost calls out in his essay, suggesting that we have veiled the material realities of algorithms behind a mystical notion of computation as a universal truth. We see this faith in computation invoked repeatedly at the intersection of algorithms and culture. Facebook’s mission statement is “to give people the power to share and make the world more open and connected,” a position that embeds assumptions like the argument that its social graph algorithms will grant us power; that its closed, proprietary platform will lead to more transparency; and that transparency leads to freedom, and perhaps to empathy. Uber is “evolving the way the world moves. By seamlessly connecting riders to drivers through our apps, we make cities more accessible, opening up more possibilities for riders and more business for drivers.” The theocracy of computation will not merely change the world but evolve it, and it will open new possibilities for users, linking proprietary commerce and individual freedom.

Users construct their own narratives within the constraints of algorithmic enframing even as they click through the Skinner boxes set up for them. The algorithms structure and track these actions, gathering them like drops of rain in a catchment to be resold as a bulk data commodity. Meanwhile, the players generally perceive only a fragment of this larger market situation, often donating not just their attention (to view ads) and their social graph (to deepen their profiles with data brokers and to expand the algorithm’s reach), but also their cash, making in-game purchases to enhance their playing experience. For many of us—roughly 1.5 billion people accessed Facebook at least once a month in 2015, out of 3.2 billion Internet users worldwide—some version of these transactions constitutes a major source of “fun.”18 Work and Play Perhaps the most compelling element of Cow Clicker as a work of conceptual art is its inversion of the concept of fun.

This moves far beyond our reliance on digital address books, mail programs, or file archives: Google’s machine learning algorithms can now suggest appropriate responses to emails, and AlphaGo gives grandmasters of that venerable art form some of their most interesting games. Widening the scope further, we can begin to see how we are changing the fundamental terms of cognition and imagination. The age of the algorithm marks the moment when technical memory has evolved to store not just our data but far more sophisticated patterns of practice, from musical taste to our social graphs. In many cases we are already imagining in concert with our machines. Algorithmic systems curate the quest for knowledge, conversing and anticipating our interests and informational needs. They author with us, providing scaffolding, context, and occasionally direct material for everything from House of Cards to algorithmically vetted pop music. The horizon of imaginative possibility is increasingly determined by computational systems, which manufacture and curate the serendipity and informational flow that propels the lifecycle of ideas, of discourse, of art.


pages: 407 words: 103,501

The Digital Divide: Arguments for and Against Facebook, Google, Texting, and the Age of Social Netwo Rking by Mark Bauerlein

Amazon Mechanical Turk, Andrew Keen, business cycle, centre right, citizen journalism, collaborative editing, computer age, computer vision, corporate governance, crowdsourcing, David Brooks, disintermediation, Frederick Winslow Taylor, Howard Rheingold, invention of movable type, invention of the steam engine, invention of the telephone, Jaron Lanier, Jeff Bezos, jimmy wales, Kevin Kelly, knowledge worker, late fees, Mark Zuckerberg, Marshall McLuhan, means of production, meta analysis, meta-analysis, moral panic, Network effects, new economy, Nicholas Carr, PageRank, peer-to-peer, pets.com, Results Only Work Environment, Saturday Night Live, search engine result page, semantic web, Silicon Valley, slashdot, social graph, social web, software as a service, speech recognition, Steve Jobs, Stewart Brand, technology bubble, Ted Nelson, The Wisdom of Crowds, Thorstein Veblen, web application

But others have already taken on the characteristic of fundamental system services. Take for example the domain registries of the DNS, which are a backbone service of the Internet. Or consider CDDB, used by virtually every music application to look up the metadata for songs and albums. Mapping data from providers like Navteq and TeleAtlas is used by virtually all online mapping applications. There is a race on right now to own the social graph. But we must ask whether this service is so fundamental that it needs to be open to all. It’s easy to forget that only fifteen years ago, e-mail was as fragmented as social networking is today, with hundreds of incompatible e-mail systems joined by fragile and congested gateways. One of those systems—Internet RFC 822 e-mail—became the gold standard for interchange. We expect to see similar standardization in key Internet utilities and subsystems.

Ti Kan, Steve Scherf, and Graham Toal, the creators of CDDB, realized that the sequence of track lengths on a CD formed a unique signature that could be correlated with artist, album, and song names. Larry Page and Sergey Brin realized that a link is a vote. Marc Hedlund at Wesabe realized that every credit card swipe is also a vote, that there is hidden meaning in repeated visits to the same merchant. Mark Zuckerberg at Facebook realized that friend relationships online actually constitute a generalized social graph. They thus turn what at first appeared to be unstructured into structured data. And all of them used both machines and humans to do it. . . . >>> the rise of real time: a collective mind As it becomes more conversational, search has also gotten faster. Blogging added tens of millions of sites that needed to be crawled daily or even hourly, but microblogging requires instantaneous update—which means a significant shift in both infrastructure and approach.

Seinfeld (television series) Self-portraits Self-publishing Self-realization Self-sufficiency Semantic priming Semiotic democracy Sensory deprivation September 11, 2001 terrorist attacks Serialization SERP. See Search Engine Results Page Sesame Street Shakesville (blog) Shirky, Clay Shoutcast Simulations Six Degrees: The Science of a Connected Age (Watts) Skrenta, Rich “Skyful of Lies” and Black Swans (Gowing) Slashdot Slatalla, Michelle Slate (magazine) Sleeper Curve Slingbox SLVR phone Small world experiment Social currency Social graph Social media Social mind The Social Network (film) Social networking sites. See also specific sites activism and advertising on amount of users on development of identity setup on learning and marketing on politicians on privacy dangers and self-exposure through self-portraits on spam on weak ties on Social rules and norms Social saturation Social skills Socrates Solitude Sony Soundscape, cell phones and SourceForge.net South Korea Spamming, on social network sites Speech recognition Speed, Net Geners and Spengler, Oswald Splash screens Spotlight (blog) Squarciafico, Hieronimo Standage, Tom Starbucks Starkweather, Gary Star Trek (television series) Star Wars Stone, Linda Street Fighter II (video game) “The Strength of Weak Ties” (Granovetter) StyleDiary.net Suburbanization Sundance Resort Suriowecki, James Survival of the fittest Survivor (television series) Swados, Harvey Swarm intelligence Switch costs Tagging TakingITGlobal Task management Task switching Taylor, Frederick Winslow Teachout, Zephyr Techgnosis Technics and Civilization (Mumford) Techno-brain burnout Technology Education and Design (TED) Technomadicity Technorati TED.


pages: 184 words: 53,625

Future Perfect: The Case for Progress in a Networked Age by Steven Johnson

Airbus A320, airport security, algorithmic trading, banking crisis, barriers to entry, Bernie Sanders, call centre, Captain Sullenberger Hudson, Cass Sunstein, Charles Lindbergh, cognitive dissonance, credit crunch, crowdsourcing, dark matter, Dava Sobel, David Brooks, Donald Davies, future of journalism, hive mind, Howard Rheingold, HyperCard, Jane Jacobs, John Gruber, John Harrison: Longitude, Joi Ito, Kevin Kelly, Kickstarter, lone genius, Mark Zuckerberg, mega-rich, meta analysis, meta-analysis, Naomi Klein, Nate Silver, Occupy movement, packet switching, peer-to-peer, Peter Thiel, planetary scale, pre–internet, RAND corporation, risk tolerance, shareholder value, Silicon Valley, Silicon Valley startup, social graph, Steve Jobs, Steven Pinker, Stewart Brand, The Death and Life of Great American Cities, Tim Cook: Apple, urban planning, US Airways Flight 1549, WikiLeaks, William Langewiesche, working poor, X Prize, your tax dollars at work

The Facebook mission can be boiled down to the old E. M. Forster slogan: “Only connect.” The company wants to strengthen the social ties that allow humans around the planet to connect, organize, converse, and share. At one point, Zuckerberg writes: By helping people form these connections, we hope to rewire the way people spread and consume information. We think the world’s information infrastructure should resemble the social graph—a network built from the bottom up or peer-to-peer, rather than the monolithic, top-down structure that has existed to date. In other words, the Facebook platform is a continuation of the Web and Internet platforms that lie beneath it: it is a Baran Web, not a Legrand Star. And it considers the cultivation and proliferation of Baran Webs to be its defining mission. Zuckerberg clearly believes that the peer-network structure can and should take hold in countless industries, in both the private and public sectors.

On the one hand, the conviction that peer networks can be a transformative force for good in the world is perhaps the core belief of the peer-progressive worldview. So when you hear one of the richest and most influential young men in the world delivering that sermon—in an S-1 filing, no less—it’s hard to hold back from shouting out a few hallelujahs. But there’s a difference here, one that makes all the difference. The platforms of the Web and the Internet were pure peer networks, owned by everyone. Facebook is a private corporation; the social graph that Zuckerberg celebrates is a proprietary technology, an asset owned by the shareholders of Facebook itself. And as far as corporations go, Facebook is astonishingly top-heavy: the S-1 revealed that Zuckerberg personally controls 57 percent of Facebook’s voting stock, giving him control over the company’s destiny that far exceeds anything Bill Gates or Steve Jobs ever had. The cognitive dissonance could drown out a Sonic Youth concert: Facebook believes in peer-to-peer networks for the world, but within its own walls, the company prefers top-down control centralized in a charismatic leader.


pages: 268 words: 75,850

The Formula: How Algorithms Solve All Our Problems-And Create More by Luke Dormehl

3D printing, algorithmic trading, Any sufficiently advanced technology is indistinguishable from magic, augmented reality, big data - Walmart - Pop Tarts, call centre, Cass Sunstein, Clayton Christensen, commoditize, computer age, death of newspapers, deferred acceptance, disruptive innovation, Edward Lorenz: Chaos theory, Erik Brynjolfsson, Filter Bubble, Flash crash, Florence Nightingale: pie chart, Frank Levy and Richard Murnane: The New Division of Labor, Google Earth, Google Glasses, High speed trading, Internet Archive, Isaac Newton, Jaron Lanier, Jeff Bezos, job automation, John Markoff, Kevin Kelly, Kodak vs Instagram, lifelogging, Marshall McLuhan, means of production, Nate Silver, natural language processing, Netflix Prize, Panopticon Jeremy Bentham, pattern recognition, price discrimination, recommendation engine, Richard Thaler, Rosa Parks, self-driving car, sentiment analysis, Silicon Valley, Silicon Valley startup, Slavoj Žižek, social graph, speech recognition, Steve Jobs, Steven Levy, Steven Pinker, Stewart Brand, the scientific method, The Signal and the Noise by Nate Silver, upwardly mobile, Wall-E, Watson beat the top human players on Jeopardy!, Y Combinator

As per the promotional literature supplied by the team: In a crowded room you don’t even have to bother working out who takes your fancy. The phone does all that. If it spots another phone with a good match—male or female—the two handsets beep and exchange information using Bluetooth radio technology. The rest is up to you.25 Apps like Serendipity are part of a new trend in technology called social discovery, which has grown out of social networking. Where social networking is about connecting with people already on your social graph, social discovery is all about meeting new people. There are few better examples in this book of The Formula in action than MIT’s Serendipity system. Here is a problem (“chance”) and a task (“making it more efficient”). Executed correctly, the computer might provide an answer to the question asked by Humphrey Bogart’s character in Casablanca. “Of all the gin joints in all the towns in all the world, why did such-and-such a person walk in to yours?”

Words like “relevant” and “newsworthy” are loaded terms that encourage (but often fail to answer) the seemingly obvious follow-up question: “relevant” and “newsworthy” to whom? In the case of a company like Google, the answer is simple: to the company’s shareholders, of course. Facebook’s algorithms can similarly be viewed as a formula for maintaining and building your friendship circle—but of course the reality is that Facebook’s purpose isn’t to make you friends, but rather to monetize your social graph through advertising.41 Hopefully, this questioning process is starting to happen. A number of researchers working with recommender systems have told me how user expectations have changed in recent years. Where five or ten years ago, people would be happy with any recommendations, today an increasing number want to know why these recommendations have been made for them. With asking why we are expected to take things at “interface value” will come the ability to critique the continued algorithmization of everything.


pages: 538 words: 141,822

The Net Delusion: The Dark Side of Internet Freedom by Evgeny Morozov

"Robert Solow", A Declaration of the Independence of Cyberspace, Ayatollah Khomeini, Berlin Wall, borderless world, Buckminster Fuller, Cass Sunstein, citizen journalism, cloud computing, cognitive dissonance, Columbine, computer age, conceptual framework, crowdsourcing, Dissolution of the Soviet Union, don't be evil, failed state, Fall of the Berlin Wall, Francis Fukuyama: the end of history, global village, Google Earth, illegal immigration, invention of radio, invention of the printing press, invisible hand, John Markoff, John von Neumann, Marshall McLuhan, Mitch Kapor, Naomi Klein, Network effects, new economy, New Urbanism, Panopticon Jeremy Bentham, peer-to-peer, pirate software, pre–internet, Productivity paradox, RAND corporation, Ronald Reagan, Ronald Reagan: Tear down this wall, Silicon Valley, Silicon Valley startup, Sinatra Doctrine, Skype, Slavoj Žižek, social graph, Steve Jobs, technoutopianism, The Wisdom of Crowds, urban planning, Washington Consensus, WikiLeaks, women in the workforce

The belief that the Internet is too big to censor is dangerously naïve. As the Web becomes even more social, nothing prevents governments—or any other interested players—from building censorship engines powered by recommendation technology similar to that of Amazon and Netflix. The only difference, however, would be that instead of being prompted to check out the “recommended” pages, we’d be denied access to them. The “social graph”—a collection of all our connections across different sites (think of a graph that shows everyone you are connected to on different sites across the Web, from Facebook to Twitter to YouTube)—a concept so much beloved by the “digerati,” could encircle all of us. The main reason why censorship methods have not yet become more social is because much of our Internet browsing is still done anonymously.

Breaking the firewalls to discover that the content one seeks has been deleted by a zealous intermediary or taken down through a cyber-attack is going to be disappointing. There are plenty of things to be done to protect against this new, more aggressive kind of censorship. One is to search for ways to provide mirrors of websites that are under DDoS attacks or to train their administrators, many of whom are self-taught and may not be managing the crisis properly, to do so. Another is to find ways to disrupt, mute, or even intentionally pollute our “social graph,” rendering it useless to those who would like to restrict access to information based on user demographics. We may even want to figure out how everyone online can pretend to be an investment banker seeking to read Financial Times! One could also make it harder to hijack and delete various groups from Facebook and other social networking sites. Or one could design a way to profit from methods like “crowdsourcing” in fighting, not just facilitating, Internet censorship; surely if a group of government royalists troll the Web to find new censorship targets, another group could also be searching for websites in need of extra protection?

F., and D. G. Johnson. “Data Retention and the Panoptic Society: The Social Benefits of Forgetfulness.” Information Society 18, no. 1 (2002): 33-45. “Bloggery Soobwajut, Chto FSB Prosit Udaljat’ Posty na Temu Akcij Protesta.” Rambler-Novosti, December 24, 2008. news.rambler.ru/Russia/head/1634066/?abstroff=0 . Bonneau, J., J. Anderson, R. Anderson, and F. Stajano. “Eight Friends Are Enough: Social Graph Approximation via Public Listings.” In Proceedings of the Second ACM EuroSys Workshop on Social Network Systems, 13-18. 2009. Bunyan, T. “Just over the Horizon: The Surveillance Society and the State in the EU.” Race & Class 51, no. 3 (2010): 1. “Cambodia Shuts Off SMS Ahead of Elections.” Associated Press, April 2, 2007. Carver, G. A., Jr. “Intelligence in the Age of Glasnost.” Foreign Affairs 69 (1989): 147.


pages: 464 words: 127,283

Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia by Anthony M. Townsend

1960s counterculture, 4chan, A Pattern Language, Airbnb, Amazon Web Services, anti-communist, Apple II, Bay Area Rapid Transit, Burning Man, business process, call centre, carbon footprint, charter city, chief data officer, clean water, cleantech, cloud computing, computer age, congestion charging, connected car, crack epidemic, crowdsourcing, DARPA: Urban Challenge, data acquisition, Deng Xiaoping, digital map, Donald Davies, East Village, Edward Glaeser, game design, garden city movement, Geoffrey West, Santa Fe Institute, George Gilder, ghettoisation, global supply chain, Grace Hopper, Haight Ashbury, Hedy Lamarr / George Antheil, hive mind, Howard Rheingold, interchangeable parts, Internet Archive, Internet of things, Jacquard loom, Jane Jacobs, jitney, John Snow's cholera map, Joi Ito, Khan Academy, Kibera, Kickstarter, knowledge worker, load shedding, M-Pesa, Mark Zuckerberg, megacity, mobile money, mutually assured destruction, new economy, New Urbanism, Norbert Wiener, Occupy movement, off grid, openstreetmap, packet switching, Panopticon Jeremy Bentham, Parag Khanna, patent troll, Pearl River Delta, place-making, planetary scale, popular electronics, RFC: Request For Comment, RFID, ride hailing / ride sharing, Robert Gordon, self-driving car, sharing economy, Silicon Valley, Skype, smart cities, smart grid, smart meter, social graph, social software, social web, special economic zone, Steve Jobs, Steve Wozniak, Stuxnet, supply-chain management, technoutopianism, Ted Kaczynski, telepresence, The Death and Life of Great American Cities, too big to fail, trade route, Tyler Cowen: Great Stagnation, undersea cable, Upton Sinclair, uranium enrichment, urban decay, urban planning, urban renewal, Vannevar Bush, working poor, working-age population, X Prize, Y2K, zero day, Zipcar

In her most influential book, The Death and Life of Great American Cities, the acclaimed urbanist Jane Jacobs argued that “cities have the capability of providing something for everybody, only because, and only when, they are created by everybody.”39 Yet over fifty years later, as we set out to create the smart cities of the twenty-first century, we seem to have again forgotten this hard-learned truth. But there is hope that a new civic order will arise in smart cities, and pull every last one of us into the effort to make them better places. Cities used to be full of strangers and chance encounters. Today we can mine the social graph in an instant by simply taking a photo. Algorithms churn in the cloud, telling the little things in our pocket where we should eat and whom we should date. It’s a jarring transformation. But even as old norms fade into the past, we’re learning new ways to thrive on mass connectedness. A sharing economy has mushroomed overnight, as people swap everything from spare bedrooms to cars, in a synergistic exploitation of new technology and more earth-friendly consumption.

Dodgeball spread virally and Crowley and Rainert spun it out of the university as a for-profit venture. From the three hundred or so students and friends who used the service during their grad school days, membership expanded to a thousand at the new startup’s launch. Within a year, over thirty thousand people had logins.21 As Dodgeball became a virtual dashboard for a certain slice of Manhattan’s digerati, its social graph—the web of friendships recorded in its database, and the flow of check-ins its users created—formed a new kind of urban media that Crowley and Rainert eagerly employed to design new experiences. One tweak tried to help you make new friends. Normally you only saw the check-ins of your direct friends, but if a friend of a friend checked in nearby, you’d get an alert urging you to go say hi. Another experiment turned Dodgeball into a romantic matchmaking machine, letting you declare a “crush” on another user and alerting him or her when you checked in nearby, to give you a shot at a hookup.

If it does truly create a new global trade in smart city solutions, local officials may be under more pressure than ever to make sure their dollars go to local firms that could themselves use CityMart for a real shot at larger success. My phone buzzed with directions to my next appointment. That evening I was using Barcelona’s cafés and bars as a kind of virtual conference center, all coordinated through my Foursquare social graph. Haselmayer offered his cynical view of the smart-cities industry, which had gathered in Barcelona for one of its biggest global trade shows to date. “The debate on smart cities has become all about [technical] architecture, where IBM says a smart city is nothing else but a corporation, and you need a good kind of architecture and then everything happens. That is an unrealistic view of how a city works, and it’s a monolithic approach.


pages: 91 words: 18,831

Getting Started With OAuth 2.0 by Ryan Boyd

MITM: man-in-the-middle, social graph, web application

Connecting users with their data results in improved day-to-day efficiency by eliminating data silos and also allows developers to differentiate their applications from the competition. OAuth provides the ability for these applications to access a user’s data securely, without requiring the user to take the scary step of handing over an account password. Types of functionality provided by OAuth-enabled APIs include the following: Getting access to a user’s social graph—their Facebook friends, people they’re following on Twitter, or their Google Contacts Sharing information about a user’s activities on your site by posting to their Facebook wall or Twitter stream Accessing a user’s Google Docs or Dropbox account to store data in their online filesystem of choice Integrating business applications with one another to drive smarter decisions by mashing up multiple data sources such as a Salesforce CRM and TripIt travel plan In order to access or update private data via each of these APIs, an application needs to be delegated access by the owner of the data.


pages: 918 words: 257,605

The Age of Surveillance Capitalism by Shoshana Zuboff

Amazon Web Services, Andrew Keen, augmented reality, autonomous vehicles, barriers to entry, Bartolomé de las Casas, Berlin Wall, bitcoin, blockchain, blue-collar work, book scanning, Broken windows theory, California gold rush, call centre, Capital in the Twenty-First Century by Thomas Piketty, Cass Sunstein, choice architecture, citizen journalism, cloud computing, collective bargaining, Computer Numeric Control, computer vision, connected car, corporate governance, corporate personhood, creative destruction, cryptocurrency, dogs of the Dow, don't be evil, Donald Trump, Edward Snowden, en.wikipedia.org, Erik Brynjolfsson, facts on the ground, Ford paid five dollars a day, future of work, game design, Google Earth, Google Glasses, Google X / Alphabet X, hive mind, impulse control, income inequality, Internet of things, invention of the printing press, invisible hand, Jean Tirole, job automation, Johann Wolfgang von Goethe, John Markoff, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, Joseph Schumpeter, Kevin Kelly, knowledge economy, linked data, longitudinal study, low skilled workers, Mark Zuckerberg, market bubble, means of production, multi-sided market, Naomi Klein, natural language processing, Network effects, new economy, Occupy movement, off grid, PageRank, Panopticon Jeremy Bentham, pattern recognition, Paul Buchheit, performance metric, Philip Mirowski, precision agriculture, price mechanism, profit maximization, profit motive, recommendation engine, refrigerator car, RFID, Richard Thaler, ride hailing / ride sharing, Robert Bork, Robert Mercer, Second Machine Age, self-driving car, sentiment analysis, shareholder value, Shoshana Zuboff, Silicon Valley, Silicon Valley ideology, Silicon Valley startup, slashdot, smart cities, Snapchat, social graph, social web, software as a service, speech recognition, statistical model, Steve Jobs, Steven Levy, structural adjustment programs, The Future of Employment, The Wealth of Nations by Adam Smith, Tim Cook: Apple, two-sided market, union organizing, Watson beat the top human players on Jeopardy!, winner-take-all economy, Wolfgang Streeck

At Google she led the development of surveillance capitalism through the expansion of AdWords and other aspects of online sales operations.86 One investor who had observed the company’s growth during that period concluded, “Sheryl created AdWords.”87 In signing on with Facebook, the talented Sandberg became the “Typhoid Mary” of surveillance capitalism as she led Facebook’s transformation from a social networking site to an advertising behemoth. Sandberg understood that Facebook’s social graph represented an awe-inspiring source of behavioral surplus: the extractor’s equivalent of a nineteenth-century prospector stumbling into a valley that sheltered the largest diamond mine and the deepest gold mine ever to be discovered. “We have better information than anyone else. We know gender, age, location, and it’s real data as opposed to the stuff other people infer,” Sandberg said. Facebook would learn to track, scrape, store, and analyze UPI to fabricate its own targeting algorithms, and like Google it would not restrict extraction operations to what people voluntarily shared with the company.

The EFF also found that the company chose to hold security functions hostage to personal data flows, claiming that security updates for the operating system would not function properly if users chose to limit location reporting.117 In 2016 Microsoft acquired LinkedIn, the professional social network, for $26.2 billion. The aim here is to establish reliable supply routes to the social network dimension of surplus behavior known as the “social graph.” These powerful new flows of social surplus from 450 million users can substantially enhance Microsoft prediction products, a key fact noted by Nadella in his announcement of the acquisition to investors: “This can drive targeting and relevance to the next level.”118 Of the three key opportunities that Nadella cited to investors upon the announcement of the acquisition, one was “Accelerate monetization through individual and organization subscriptions and targeted advertising.”

According to his analysis, “Like most apps that work with the GPS in your smartphone, Pokémon Go can tell a lot of things about you based on your movement as you play: where you go, when you went there, how you got there, how long you stayed, and who else was there. And, like many developers who build those apps, Niantic keeps that information.” Whereas other location-based apps might collect similar data, Bernstein concluded that “Pokémon Go’s incredibly granular, block-by-block map data, combined with its surging popularity, may soon make it one of, if not the most, detailed location-based social graphs ever compiled.”47 The industry news site TechCrunch raised similar concerns regarding the game’s data-collection practices, questioning “the long list of permissions the app requires.” Those permissions included the camera, yes, but also permission to “read your contacts” and “find accounts on device.” Niantic’s “surveillance policy” notes that it may share “aggregated information and non-identifying information with third parties for research and analysis, demographic profiling, and other similar purposes.”


pages: 285 words: 81,743

Start-Up Nation: The Story of Israel's Economic Miracle by Dan Senor, Saul Singer

"Robert Solow", agricultural Revolution, Albert Einstein, back-to-the-land, banking crisis, Boycotts of Israel, call centre, Celtic Tiger, cleantech, Dissolution of the Soviet Union, friendly fire, immigration reform, labor-force participation, mass immigration, new economy, pez dispenser, post scarcity, profit motive, Silicon Valley, smart grid, social graph, sovereign wealth fund, Steve Ballmer, web application, women in the workforce, Yom Kippur War

Alex Vieux, CEO of Red Herring magazine, told us that he has been to “a million high-tech conferences, on multiple continents. I see Israelis like Medved give presentations all the time, alongside their peers from other countries. The others are always making a pitch for their specific company. The Israelis are always making a pitch for Israel.”9 CHAPTER 4 Harvard, Princeton, and Yale The social graph is very simple here. Everybody knows everybody. —YOSSI VARDI DAVID AMIR MET US AT HIS JERUSALEM HOME in his pilot’s uniform, but there was nothing Top Gun about him. Soft-spoken, thoughtful, and self-deprecating, he looked, even in uniform, more like an American liberal arts student than the typical pilot with crisp military bearing. Yet as he explained with pride how the Israeli Air Force trained some of the best pilots in the world—according to numerous international competitions as well as their record in battle—it became easy to see how he fit in.1 While students in other countries are preoccupied with deciding which college to attend, Israelis are weighing the merits of different military units.

It nourishes an entirely different kind of lifelong bond.”6 Indeed, relationships developed during military service form another network in what is already a very small and interconnected country. “The whole country is one degree of separation,” says Yossi Vardi, the godfather of dozens of Internet start-ups and one of the champion networkers in the wired world. Like Jon Medved, Vardi is one of Israel’s legendary business ambassadors. Vardi says he knows of Israeli companies that have stopped using help-wanted ads: “It’s now all word of mouth. . . . The social graph is very simple here. Everybody knows everybody; everybody was serving in the army with the brother of everybody; the mother of everybody was the teacher in their school; the uncle was the commander of somebody else’s unit. Nobody can hide. If you don’t behave, you cannot disappear to Wyoming or California. There is a very high degree of transparency.”7 The benefits of this kind of interconnectedness are not limited to Israel, although in Israel they are unusually intense and widespread.


pages: 369 words: 80,355

Too Big to Know: Rethinking Knowledge Now That the Facts Aren't the Facts, Experts Are Everywhere, and the Smartest Person in the Room Is the Room by David Weinberger

airport security, Alfred Russel Wallace, Amazon Mechanical Turk, Berlin Wall, Black Swan, book scanning, Cass Sunstein, commoditize, corporate social responsibility, crowdsourcing, Danny Hillis, David Brooks, Debian, double entry bookkeeping, double helix, en.wikipedia.org, Exxon Valdez, Fall of the Berlin Wall, future of journalism, Galaxy Zoo, Hacker Ethic, Haight Ashbury, hive mind, Howard Rheingold, invention of the telegraph, jimmy wales, Johannes Kepler, John Harrison: Longitude, Kevin Kelly, linked data, Netflix Prize, New Journalism, Nicholas Carr, Norbert Wiener, openstreetmap, P = NP, Pluto: dwarf planet, profit motive, Ralph Waldo Emerson, RAND corporation, Ray Kurzweil, Republic of Letters, RFID, Richard Feynman, Ronald Reagan, semantic web, slashdot, social graph, Steven Pinker, Stewart Brand, technological singularity, Ted Nelson, the scientific method, The Wisdom of Crowds, Thomas Kuhn: the structure of scientific revolutions, Thomas Malthus, Whole Earth Catalog, X Prize

Disclosure: I am a member of the Digital Public Library of America’s “technical workstream,” and the library lab that I co-direct will have entered the DPLA’s call for project ideas before this book is printed. 4 Kevin Kelly, What Technology Wants (Penguin, 2010). 5 James Aitken Wylie, The History of Protestantism with Five Hundred and Fifty Illustrations by the Best Artist, Vol. 1 (Cassell, 1899), p. 113, http://books.google.com/books?id=kFU-AAAAYAAJ. 6 See Ethan Zuckerman’s excellent post “Shortcuts in the Social Graph,” October 14, 2010, http://www.ethanzuckerman.com/blog/2010/10/14/shortcuts-in-the-social-graph/. 7 During the 2008 presidential campaign, Sarah Palin was accused of pressuring a local librarian to censor some books. See Rindi White, “Palin Pressured Wasilla Librarian,” Anchorage Daily News, September 4, 2008, http://www.adn.com/2008/09/03/515512/palin-pressured-wasilla-librarian.html. 8 Tim Berners-Lee, “Linked Data,” July 27, 2006, http://www.w3.org/DesignIssues/LinkedData.html. 9 This was the price quoted at Fisher Scientific on June 11, 2011.


pages: 286 words: 87,401

Blitzscaling: The Lightning-Fast Path to Building Massively Valuable Companies by Reid Hoffman, Chris Yeh

activist fund / activist shareholder / activist investor, Airbnb, Amazon Web Services, autonomous vehicles, bitcoin, blockchain, Bob Noyce, business intelligence, Chuck Templeton: OpenTable:, cloud computing, crowdsourcing, cryptocurrency, Daniel Kahneman / Amos Tversky, database schema, discounted cash flows, Elon Musk, Firefox, forensic accounting, George Gilder, global pandemic, Google Hangouts, Google X / Alphabet X, hydraulic fracturing, Hyperloop, inventory management, Isaac Newton, Jeff Bezos, Joi Ito, Khan Academy, late fees, Lean Startup, Lyft, M-Pesa, Marc Andreessen, margin call, Mark Zuckerberg, minimum viable product, move fast and break things, move fast and break things, Network effects, Oculus Rift, oil shale / tar sands, Paul Buchheit, Paul Graham, Peter Thiel, pre–internet, recommendation engine, ride hailing / ride sharing, Sam Altman, Sand Hill Road, Saturday Night Live, self-driving car, shareholder value, sharing economy, Silicon Valley, Silicon Valley startup, Skype, smart grid, social graph, software as a service, software is eating the world, speech recognition, stem cell, Steve Jobs, subscription business, Tesla Model S, thinkpad, transaction costs, transport as a service, Travis Kalanick, Uber for X, uber lyft, web application, winner-take-all economy, Y Combinator, yellow journalism

This dominance lets the market leader “tax” all the participants who want to use the platform, much as levies were imposed in the bygone Republic of Venice. For example, the iTunes store takes a 30 percent share of the proceeds whenever a song, a movie, a book, or an app is sold on that platform. These platform revenues tend to have very high gross margins, which generate cash that can be plowed back into making the platform even better. Amazon’s merchant platform, Facebook’s social graph, and, of course, Apple’s iOS ecosystem are great examples of the power of platforms. PROVEN PATTERN #3: FREE OR FREEMIUM “Free” has an incredible power that no other pricing does. The Duke behavioral economist Dan Ariely wrote about the power of free in his excellent book Predictably Irrational, describing an experiment in which he offered research subjects the choice of a Lindt chocolate truffle for 15 cents or a Hershey’s Kiss for a mere penny.

Network Effects We’ve already talked about how Facebook leverages classic direct network effects (the more users that join the platform, the greater the value of Facebook to every other Facebook user) and local network effects (once it becomes the dominant social network at a college, it becomes extremely difficult for any other player to pry away Facebook’s users). Facebook also experiences some helpful indirect network effects thanks to its platform services, such as the Graph API (which allows developers to leverage the Facebook social graph of users and their relationships) and Facebook Connect (which allows users to log in to a Web service using Facebook rather than create a new account for that service). Product/Market Fit Facebook achieved product/market fit for its core consumer experience almost immediately, hence its rapid growth. However, part of what makes Facebook a great company and Mark Zuckerberg a great CEO is that Facebook has been able to achieve product/market fit in additional and less obvious areas at other points in the company’s history.


pages: 339 words: 94,769

Possible Minds: Twenty-Five Ways of Looking at AI by John Brockman

AI winter, airport security, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, artificial general intelligence, Asilomar, autonomous vehicles, basic income, Benoit Mandelbrot, Bill Joy: nanobots, Buckminster Fuller, cellular automata, Claude Shannon: information theory, Daniel Kahneman / Amos Tversky, Danny Hillis, David Graeber, easy for humans, difficult for computers, Elon Musk, Eratosthenes, Ernest Rutherford, finite state, friendly AI, future of work, Geoffrey West, Santa Fe Institute, gig economy, income inequality, industrial robot, information retrieval, invention of writing, James Watt: steam engine, Johannes Kepler, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John von Neumann, Kevin Kelly, Kickstarter, Laplace demon, Loebner Prize, market fundamentalism, Marshall McLuhan, Menlo Park, Norbert Wiener, optical character recognition, pattern recognition, personalized medicine, Picturephone, profit maximization, profit motive, RAND corporation, random walk, Ray Kurzweil, Richard Feynman, Rodney Brooks, self-driving car, sexual politics, Silicon Valley, Skype, social graph, speech recognition, statistical model, Stephen Hawking, Steven Pinker, Stewart Brand, strong AI, superintelligent machines, supervolcano, technological singularity, technoutopianism, telemarketer, telerobotics, the scientific method, theory of mind, Turing machine, Turing test, universal basic income, Upton Sinclair, Von Neumann architecture, Whole Earth Catalog, Y2K, zero-sum game

For social life at a small college, you could construct a central database and keep it up to date, but its upkeep would become overwhelming if taken to any larger scale. Better to pass out free copies of a simple semi-autonomous code, hosted locally, and let the social network update itself. This code is executed by digital computers, but the analog computing performed by the system as a whole far exceeds the complexity of the underlying code. The resulting pulse-frequency coded model of the social graph becomes the social graph. It spreads wildly across the campus and then the world. What if you wanted to build a machine to capture what everything known to the human species means? With Moore’s Law behind you, it doesn’t take too long to digitize all the information in the world. You scan every book ever printed, collect every email ever written, and gather forty-nine years of video every twenty-four hours, while tracking where people are and what they do, in real time.


pages: 606 words: 157,120

To Save Everything, Click Here: The Folly of Technological Solutionism by Evgeny Morozov

3D printing, algorithmic trading, Amazon Mechanical Turk, Andrew Keen, augmented reality, Automated Insights, Berlin Wall, big data - Walmart - Pop Tarts, Buckminster Fuller, call centre, carbon footprint, Cass Sunstein, choice architecture, citizen journalism, cloud computing, cognitive bias, creative destruction, crowdsourcing, data acquisition, Dava Sobel, disintermediation, East Village, en.wikipedia.org, Fall of the Berlin Wall, Filter Bubble, Firefox, Francis Fukuyama: the end of history, frictionless, future of journalism, game design, Gary Taubes, Google Glasses, illegal immigration, income inequality, invention of the printing press, Jane Jacobs, Jean Tirole, Jeff Bezos, jimmy wales, Julian Assange, Kevin Kelly, Kickstarter, license plate recognition, lifelogging, lone genius, Louis Pasteur, Mark Zuckerberg, market fundamentalism, Marshall McLuhan, moral panic, Narrative Science, Nelson Mandela, Nicholas Carr, packet switching, PageRank, Parag Khanna, Paul Graham, peer-to-peer, Peter Singer: altruism, Peter Thiel, pets.com, placebo effect, pre–internet, Ray Kurzweil, recommendation engine, Richard Thaler, Ronald Coase, Rosa Parks, self-driving car, Silicon Valley, Silicon Valley ideology, Silicon Valley startup, Skype, Slavoj Žižek, smart meter, social graph, social web, stakhanovite, Steve Jobs, Steven Levy, Stuxnet, technoutopianism, the built environment, The Chicago School, The Death and Life of Great American Cities, the medium is the message, The Nature of the Firm, the scientific method, The Wisdom of Crowds, Thomas Kuhn: the structure of scientific revolutions, Thomas L Friedman, transaction costs, urban decay, urban planning, urban sprawl, Vannevar Bush, WikiLeaks

Amazon didn’t have an army of bored freelancers who could do virtually any job as long as they received their few pennies per hour. (And even those human freelancers might become unnecessary once automated image-recognition software gets better.) Most importantly, there was no way for all our friends to see the contents of our trash bins; fifteen years ago, even our personal websites wouldn’t get the same level of attention from our acquaintances—our entire “social graph,” as the geeks would put it—that our trash bins might receive from our Facebook friends today. Now that we are all using the same platform—Facebook—it becomes possible to steer our behavior with the help of social games and competitions; we no longer have to save the environment at our own pace using our own unique tools. There is power in standardization! These two innovations—that more and more of our life is now mediated through smart sensor-powered technologies and that our friends and acquaintances can now follow us anywhere, making it possible to create new types of incentives—will profoundly change the work of social engineers, policymakers, and many other do-gooders.

All will be tempted to exploit the power of these new techniques, either individually or in combination, to solve a particular problem, be it obesity, climate change, or congestion. Today we already have smart mirrors that, thanks to complex sensors, can track and display our pulse rates based on slight variations in the brightness of our faces; soon, we’ll have mirrors that, thanks to their ability to tap into our “social graph,” will nudge us to lose weight because we look pudgier than most of our Facebook friends. Or consider a prototype teapot built by British designer-cum-activist Chris Adams. The teapot comes with a small orb that can either glow green (making tea is okay) or red (perhaps you should wait). What determines the coloring? Well, the orb, with the help of some easily available open-source hardware and software, is connected to a site called Can I Turn It On?

Had Sigmund Freud lived long enough, he would have probably been replaced by a pedometer: in this brave new world, who needs psychoanalysis—the obsolete practice of narrative imagination—to “take stock of ourselves,” when the algorithmic option looks so tempting? If the Quantified Self movement allows us to establish our authenticity with numbers, social networking allows us to accomplish that in subtler, seemingly more creative ways: by curating the timeline of our life, by uploading our favorite photos, by using the coolest apps on the block, by maintaining a unique social graph (Facebook speak for a set of human connections that each user has). If only one looks closely enough, one can discern how the themes of fakeness and authenticity shape Facebook’s own self-presentation. So Mark Zuckerberg claims that “the social web can’t exist until you are your real self online.” Peter Thiel, the first private investor in Facebook, contrasts the authenticity offered by Facebook—where no pseudonyms are allowed—with that of its former rival, MySpace, where everything goes.


Turing's Cathedral by George Dyson

1919 Motor Transport Corps convoy, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, Albert Einstein, anti-communist, Benoit Mandelbrot, British Empire, Brownian motion, cellular automata, cloud computing, computer age, Danny Hillis, dark matter, double helix, fault tolerance, Fellow of the Royal Society, finite state, Georg Cantor, Henri Poincaré, housing crisis, IFF: identification friend or foe, indoor plumbing, Isaac Newton, Jacquard loom, John von Neumann, mandelbrot fractal, Menlo Park, Murray Gell-Mann, Norbert Wiener, Norman Macrae, packet switching, pattern recognition, Paul Erdős, Paul Samuelson, phenotype, planetary scale, RAND corporation, random walk, Richard Feynman, SETI@home, social graph, speech recognition, Thorstein Veblen, Turing complete, Turing machine, Von Neumann architecture

So as not to shoot down commercial airliners, the SAGE (Semi-Automatic Ground Environment) air defense system that developed out of MIT’s Project Whirlwind in the 1950s kept track of all passenger flights, developing a real-time model that led to the SABRE (Semi-Automatic Business-Related Environment) airline reservation system that still controls much of the passenger traffic today. Google sought to gauge what people were thinking, and became what people were thinking. Facebook sought to map the social graph, and became the social graph. Algorithms developed to model fluctuations in financial markets gained control of those markets, leaving human traders behind. “Toto,” said Dorothy in The Wizard of Oz, “I’ve a feeling we’re not in Kansas anymore.” What the Americans termed “artificial intelligence” the British termed “mechanical intelligence,” a designation that Alan Turing considered more precise. We began by observing intelligent behavior (such as language, vision, goal-seeking, and pattern-recognition) in organisms, and struggled to reproduce this behavior by encoding it into logically deterministic machines.

Pulse-frequency coding for the Internet is one way to describe the working architecture of a search engine, and PageRank for neurons is one way to describe the working architecture of the brain. These computational structures use digital components, but the analog computing being performed by the system as a whole exceeds the complexity of the digital code on which it runs. The model (of the social graph, or of human knowledge) constructs and updates itself. Complex networks—of molecules, people, or ideas—constitute their own simplest behavioral descriptions. This behavior can be more easily captured by continuous, analog networks than it can be defined by digital, algorithmic codes. These analog networks may be composed of digital processors, but it is in the analog domain that the interesting computation is being performed.


pages: 359 words: 96,019

How to Turn Down a Billion Dollars: The Snapchat Story by Billy Gallagher

Airbnb, Albert Einstein, Amazon Web Services, Apple's 1984 Super Bowl advert, augmented reality, Bernie Sanders, Black Swan, citizen journalism, Clayton Christensen, computer vision, disruptive innovation, Donald Trump, El Camino Real, Elon Musk, Frank Gehry, Google Glasses, Hyperloop, information asymmetry, Jeff Bezos, Justin.tv, Lean Startup, Long Term Capital Management, Mark Zuckerberg, Menlo Park, minimum viable product, Nelson Mandela, Oculus Rift, paypal mafia, Peter Thiel, QR code, Sand Hill Road, Saturday Night Live, side project, Silicon Valley, Silicon Valley startup, Snapchat, social graph, sorting algorithm, speech recognition, stealth mode startup, Steve Jobs, too big to fail, Y Combinator, young professional

Snapchat grew by a factor of ten, from one hundred thousand daily active users to a million in just six months. While the front-facing camera on smartphones helped Snapchat gain early traction, smartphones’ address books may have done even more to drive viral growth. Before smartphones were ubiquitous, Facebook (and others) had to work extremely hard to build a social graph on the web. But with smartphones, people had a computer in their pockets with a complete social graph—their address book. This allowed Snapchat, Instagram, WhatsApp, and others to quickly build enormously valuable services. Snapchat’s existing users were also sharing more and more photos. This put a heavy strain on Snapchat’s infrastructure, as they had to deliver millions of photographs in real time. Because users saw Snapchat as a texting replacement, they expected messages to be sent and received within seconds; if Snapchat failed to do this too often, Snapchatters would abandon it, which could cause the app to start bleeding users and spin into a death spiral.


pages: 323 words: 95,939

Present Shock: When Everything Happens Now by Douglas Rushkoff

algorithmic trading, Andrew Keen, bank run, Benoit Mandelbrot, big-box store, Black Swan, British Empire, Buckminster Fuller, business cycle, cashless society, citizen journalism, clockwork universe, cognitive dissonance, Credit Default Swap, crowdsourcing, Danny Hillis, disintermediation, Donald Trump, double helix, East Village, Elliott wave, European colonialism, Extropian, facts on the ground, Flash crash, game design, global pandemic, global supply chain, global village, Howard Rheingold, hypertext link, Inbox Zero, invention of agriculture, invention of hypertext, invisible hand, iterative process, John Nash: game theory, Kevin Kelly, laissez-faire capitalism, lateral thinking, Law of Accelerating Returns, loss aversion, mandelbrot fractal, Marshall McLuhan, Merlin Mann, Milgram experiment, mutually assured destruction, negative equity, Network effects, New Urbanism, Nicholas Carr, Norbert Wiener, Occupy movement, passive investing, pattern recognition, peak oil, price mechanism, prisoner's dilemma, Ralph Nelson Elliott, RAND corporation, Ray Kurzweil, recommendation engine, selective serotonin reuptake inhibitor (SSRI), Silicon Valley, Skype, social graph, South Sea Bubble, Steve Jobs, Steve Wozniak, Steven Pinker, Stewart Brand, supply-chain management, the medium is the message, The Wisdom of Crowds, theory of mind, Turing test, upwardly mobile, Whole Earth Catalog, WikiLeaks, Y2K, zero-sum game

It’s as if all the Facebook updates, Twitter streams, email messages, and live-streamed video could combine to create a total picture of our true personal status, or that of our business, at any given moment. And there are plenty of companies out there churning all this data in real time in order to present us with metrics and graphs claiming to represent the essence of this reality for us. And even when they work, they are mere snapshots of a moment ago. Our Facebook profile and the social graph that can be derived from it, however intricate, is still just a moment locked in time, a static picture. This quest for digital omniscience, though understandable, is self-defeating. Most of the information we get at lightning speed is so temporal as to be stale by the time it reaches us. We scramble over the buttons of the car radio in an effort to get to the right station at the right minute-after-the-hour for the traffic report.

Everything is recorded, yet almost none of it feels truly accessible. A change in file format renders decades of stored files unusable, while a silly, forgotten Facebook comment we wrote when drunk can resurface at a job interview. In the digital universe, our personal history and its sense of narrative is succeeded by our social networking profile—a snapshot of the current moment. The information itself—our social graph of friends and likes—is a product being sold to market researchers in order to better predict and guide our futures. Using past data to steer the future, however, ends up negating the present. The futile quest for omniscience we looked at earlier in this chapter encourages us, particularly businesses, to seek ever more fresh and up-to-the-minute samples, as if this will render the present coherent to us.


pages: 385 words: 111,113

Augmented: Life in the Smart Lane by Brett King

23andMe, 3D printing, additive manufacturing, Affordable Care Act / Obamacare, agricultural Revolution, Airbnb, Albert Einstein, Amazon Web Services, Any sufficiently advanced technology is indistinguishable from magic, Apple II, artificial general intelligence, asset allocation, augmented reality, autonomous vehicles, barriers to entry, bitcoin, blockchain, business intelligence, business process, call centre, chief data officer, Chris Urmson, Clayton Christensen, clean water, congestion charging, crowdsourcing, cryptocurrency, deskilling, different worldview, disruptive innovation, distributed generation, distributed ledger, double helix, drone strike, Elon Musk, Erik Brynjolfsson, Fellow of the Royal Society, fiat currency, financial exclusion, Flash crash, Flynn Effect, future of work, gig economy, Google Glasses, Google X / Alphabet X, Hans Lippershey, Hyperloop, income inequality, industrial robot, information asymmetry, Internet of things, invention of movable type, invention of the printing press, invention of the telephone, invention of the wheel, James Dyson, Jeff Bezos, job automation, job-hopping, John Markoff, John von Neumann, Kevin Kelly, Kickstarter, Kodak vs Instagram, Leonard Kleinrock, lifelogging, low earth orbit, low skilled workers, Lyft, M-Pesa, Mark Zuckerberg, Marshall McLuhan, megacity, Metcalfe’s law, Minecraft, mobile money, money market fund, more computing power than Apollo, Network effects, new economy, obamacare, Occupy movement, Oculus Rift, off grid, packet switching, pattern recognition, peer-to-peer, Ray Kurzweil, RFID, ride hailing / ride sharing, Robert Metcalfe, Satoshi Nakamoto, Second Machine Age, selective serotonin reuptake inhibitor (SSRI), self-driving car, sharing economy, Shoshana Zuboff, Silicon Valley, Silicon Valley startup, Skype, smart cities, smart grid, smart transportation, Snapchat, social graph, software as a service, speech recognition, statistical model, stem cell, Stephen Hawking, Steve Jobs, Steve Wozniak, strong AI, TaskRabbit, technological singularity, telemarketer, telepresence, telepresence robot, Tesla Model S, The Future of Employment, Tim Cook: Apple, trade route, Travis Kalanick, Turing complete, Turing test, uber lyft, undersea cable, urban sprawl, V2 rocket, Watson beat the top human players on Jeopardy!, white picket fence, WikiLeaks

In its simplest form, it’s like being told “people who did A also did B”, as a comment on what you just did or intended to do. Instructive, simple collaborative filtering. People who bought this book also bought this one. People who liked this song also liked this group. Trust Is Always Social The era of “social logins” like Facebook Connect is one of the most powerful moves made in this regard. It does this by bringing together the power of the social graph into play, augmenting the information we have access to in order to make a decision. I’m a child of the late 1950s, which means I like listening to music made between 1964 and 1977, give or take a few years. Maybe I should just say, “I like music made in the Sixties and Seventies.” Strangely enough, when I go to concerts nowadays, that statement can be interpreted differently: I now spend time listening to musicians who are in their sixties and seventies (and a few in their eighties as well, though I never was a fan of that decade).

when is this?) and relationship (who else knows this person or thing? who amongst my friends has seen this? who amongst my friends has experienced this?). Reputation and rating schemes are ways to standardise some of this feedback: my children tend to check Rotten Tomatoes before they even consider going to see a film. It’s not just about trust: there are many other ways in which the connected world, the social graph, wearables and augmentation improve our lives. Alex described how the quantified self is improved by measuring your own performance against your peers, or even working out with your peers. Peer group data, however, is being used in even simpler ways to form sort of trust “contracts”. All of these social platforms are leading us to make better decisions. One category of those decisions—whom to trust—is more important than any other, when it comes to living our lives.


pages: 159 words: 42,401

Snowden's Box: Trust in the Age of Surveillance by Jessica Bruder, Dale Maharidge

anti-communist, Bay Area Rapid Transit, Berlin Wall, blockchain, Broken windows theory, Burning Man, cashless society, Chelsea Manning, citizen journalism, computer vision, crowdsourcing, Donald Trump, Edward Snowden, Elon Musk, Ferguson, Missouri, Filter Bubble, Firefox, Internet of things, Jeff Bezos, Julian Assange, license plate recognition, Mark Zuckerberg, mass incarceration, medical malpractice, Occupy movement, off grid, pattern recognition, Peter Thiel, Robert Bork, Shoshana Zuboff, Silicon Valley, Skype, social graph, Steven Levy, Tim Cook: Apple, web of trust, WikiLeaks

The place is very remote, with the nearest utility lines some three miles away and the closest neighbor a half mile (as the spotted owl flies) across a canyon. We worked through the days and nights. I was finishing a book. Laura was editing The Program, a short documentary about William Binney, the NSA-veteran-turned-whistleblower. After a thirty-two-year career with the agency, Binney had retired in disgust following 9/11. That’s when, as he explained in the film, officials began repurposing ThinThread, a social-graphing program he’d built for use overseas, to spy on ordinary Americans instead. “This is something the KGB, the Stasi, or the Gestapo would have loved to have had about their populations,” Binney soberly told the camera. “Just because we call ourselves a democracy doesn’t mean we will stay that way. That’s the real danger.” Though no charges were ever brought against Binney, a dozen rifle-toting FBI agents raided his home in 2007.


pages: 468 words: 124,573

How to Build a Billion Dollar App: Discover the Secrets of the Most Successful Entrepreneurs of Our Time by George Berkowski

Airbnb, Amazon Web Services, barriers to entry, Black Swan, business intelligence, call centre, crowdsourcing, disruptive innovation, en.wikipedia.org, game design, Google Glasses, Google Hangouts, Google X / Alphabet X, iterative process, Jeff Bezos, Jony Ive, Kickstarter, knowledge worker, Lean Startup, loose coupling, Marc Andreessen, Mark Zuckerberg, minimum viable product, MITM: man-in-the-middle, move fast and break things, move fast and break things, Network effects, Oculus Rift, Paul Graham, QR code, Ruby on Rails, self-driving car, Silicon Valley, Silicon Valley startup, Skype, Snapchat, social graph, software as a service, software is eating the world, Steve Jobs, Steven Levy, Travis Kalanick, ubercab, Y Combinator

The startup world is very social, and most events involve pizza and beer and everyone is interested in meeting everyone else. Here are some great events that will get you talking to the right people. DEVELOPER MEETUPS. Tech companies host all kinds of events to showcase new technologies or features to developers. Facebook has one called Developer Garage. Each month the company invites developers together to talk about the latest features, such as Social Graph, Facebook Search or Facebook advertising. Companies such as Google, Yahoo!, LinkedIn and numerous other big software companies organise similar events all the time. Even though they are targeted at software developers, don’t be scared to attend if you’re not technical. Go along, pretend to be a developer (to get in), enjoy the pizza and beer, and then start talking to anyone and everyone. Not only will you get a flavour for what’s hot in terms of technology, but you’ll also start to understand how software developers think and communicate.

You can see where I am going with this. Blurring Business Models Flipboard has a very simple mission: to let people discover and share online content in beautiful, simple and meaningful ways. About 90 million people regularly use the app and it’s one of my favourite apps, on both the iPad and the iPhone. The first wave of Flipboard’s growth was fuelled by automatically creating ‘personal magazines’ directly from the social graph of its users. Flipboard pulls in stories from your friends’ tweets, Facebook posts and Google+ accounts, and then cleverly curates them into a highly readable format. Its second wave of growth was giving users (including advertisers) the power to create – and distribute – their own magazines. Within months of launching the new magazine-publishing feature, users had created some 3.5 million of them.1 The stats are impressive, with the average user spending 15 minutes browsing the app per session.


pages: 532 words: 139,706

Googled: The End of the World as We Know It by Ken Auletta

23andMe, AltaVista, Anne Wojcicki, Apple's 1984 Super Bowl advert, Ben Horowitz, bioinformatics, Burning Man, carbon footprint, citizen journalism, Clayton Christensen, cloud computing, Colonization of Mars, commoditize, corporate social responsibility, creative destruction, death of newspapers, disintermediation, don't be evil, facts on the ground, Firefox, Frank Gehry, Google Earth, hypertext link, Innovator's Dilemma, Internet Archive, invention of the telephone, Jeff Bezos, jimmy wales, John Markoff, Kevin Kelly, knowledge worker, Long Term Capital Management, Marc Andreessen, Mark Zuckerberg, Marshall McLuhan, Menlo Park, Network effects, new economy, Nicholas Carr, PageRank, Paul Buchheit, Peter Thiel, Ralph Waldo Emerson, Richard Feynman, Sand Hill Road, Saturday Night Live, semantic web, sharing economy, Silicon Valley, Skype, slashdot, social graph, spectrum auction, stealth mode startup, Stephen Hawking, Steve Ballmer, Steve Jobs, strikebreaker, telemarketer, the scientific method, The Wisdom of Crowds, Upton Sinclair, X Prize, yield management, zero-sum game

But his long pauses when asked about Google, and the way he shifted uncomfortably in his chair, suggest the tension between the two companies. He was somewhat less circumspect about MySpace, his main competitor among social networking sites: “What they’re doing is very much different from us. On a fundamental level, what they’re doing is not mapping out real connections. They’re helping people meet new people. Rather than using the social graph and the connections people have in order to facilitate decentralized communication, they’re using it as a platform to pump and push media out to people. They call themselves a next-generation media company. We don’t even think we’re a media company. We’re a technology company.” Facebook is not a content company, he said, just as a telephone company is not. In fact, in some ways Facebook is like a telephone conversation, with all your friends on the same call.

Kwan Lee is not alone in thinking that Google is mistaken to treat search as an engineering problem. John Borthwick, who created one of the first city Web sites, sold it to AOL in 1997, and later became senior vice president of technology and alliances for Time Warner, thinks Google “lacks a social gene.” (Borthwick has since founded and now runs Betaworks, which seeds money for social media.) Information, he said, “needs a social context. You need to incorporate the social graph [the connections among people] into search. Twitter becomes a platform for search. People put out Tweets—‘I’m thinking about buying a camera. What does anyone think of this camera?’” It’s the wisdom of crowds—your crowd of friends. “Google is just focused on CPU—central processing computers—and ignores the processing of the human brain.” He believes this makes its search vulnerable. Google obviously has come to share this concern for a senior Google executive confirms that they tried—and failed—to acquire Twitter.


pages: 214 words: 14,382

Monadic Design Patterns for the Web by L.G. Meredith

barriers to entry, domain-specific language, don't repeat yourself, finite state, Georg Cantor, ghettoisation, John von Neumann, Kickstarter, semantic web, social graph, type inference, web application, WebSocket

Moreover, the code is purely functional, with all of the attendant advantages of purely functional code we have been observing since Chapter 1. Obviously, in the context of the web, this particular use case is of considerable interest. Nearly every web application is of this form: navigating a tree or graph of pages. Usually, that graph of pages is somehow homomorphic, i.e. an image of, the graph of some underlying domain data structure, like the data structures of employee records in a payroll system or the social graph of a social media application Cover · Overview · Contents · Discuss · Suggest · Glossary · Index 126 Section 6.1 Chapter 6 · Zippers and Contexts and URIs, Oh My! Download from Wow! eBook <www.wowebook.com> like Twitter. Many web applications, such as so-called content management systems, also support the mutation of the graph of pages. Therefore, having a method of generating this functionality from the types of the underlying data domain, whether they are web pages or some other domain data type, is clearly pertinent to the most focused of application developers.


Martin Kleppmann-Designing Data-Intensive Applications. The Big Ideas Behind Reliable, Scalable and Maintainable Systems-O’Reilly (2017) by Unknown

active measures, Amazon Web Services, bitcoin, blockchain, business intelligence, business process, c2.com, cloud computing, collaborative editing, commoditize, conceptual framework, cryptocurrency, database schema, DevOps, distributed ledger, Donald Knuth, Edward Snowden, Ethereum, ethereum blockchain, fault tolerance, finite state, Flash crash, full text search, general-purpose programming language, informal economy, information retrieval, Internet of things, iterative process, John von Neumann, Kubernetes, loose coupling, Marc Andreessen, microservices, natural language processing, Network effects, packet switching, peer-to-peer, performance metric, place-making, premature optimization, recommendation engine, Richard Feynman, self-driving car, semantic web, Shoshana Zuboff, social graph, social web, software as a service, software is eating the world, sorting algorithm, source of truth, SPARQL, speech recognition, statistical model, undersea cable, web application, WebSocket, wikimedia commons

The rela‐ tional model can handle simple cases of many-to-many relationships, but as the con‐ nections within your data become more complex, it becomes more natural to start modeling your data as a graph. A graph consists of two kinds of objects: vertices (also known as nodes or entities) and edges (also known as relationships or arcs). Many kinds of data can be modeled as a graph. Typical examples include: Social graphs Vertices are people, and edges indicate which people know each other. The web graph Vertices are web pages, and edges indicate HTML links to other pages. Road or rail networks Vertices are junctions, and edges represent the roads or railway lines between them. Well-known algorithms can operate on these graphs: for example, car navigation sys‐ tems search for the shortest path between two points in a road network, and PageRank can be used on the web graph to determine the popularity of a web page and thus its ranking in search results.

Hellerstein: “The Declarative Imperative: Experiences and Conjec‐ tures in Distributed Logic,” Electrical Engineering and Computer Sciences, Univer‐ sity of California at Berkeley, Tech report UCB/EECS-2010-90, June 2010. [33] Jeffrey Dean and Sanjay Ghemawat: “MapReduce: Simplified Data Processing on Large Clusters,” at 6th USENIX Symposium on Operating System Design and Imple‐ mentation (OSDI), December 2004. [34] Craig Kerstiens: “JavaScript in Your Postgres,” blog.heroku.com, June 5, 2013. [35] Nathan Bronson, Zach Amsden, George Cabrera, et al.: “TAO: Facebook’s Dis‐ tributed Data Store for the Social Graph,” at USENIX Annual Technical Conference (USENIX ATC), June 2013. [36] “Apache TinkerPop3.2.3 Documentation,” tinkerpop.apache.org, October 2016. [37] “The Neo4j Manual v2.0.0,” Neo Technology, 2013. [38] Emil Eifrem: Twitter correspondence, January 3, 2014. [39] David Beckett and Tim Berners-Lee: “Turtle – Terse RDF Triple Language,” W3C Team Submission, March 28, 2011. [40] “Datomic Development Resources,” Metadata Partners, LLC, 2013. [41] W3C RDF Working Group: “Resource Description Framework (RDF),” w3.org, 10 February 2004. [42] “Apache Jena,” Apache Software Foundation. 66 | Chapter 2: Data Models and Query Languages [43] Steve Harris, Andy Seaborne, and Eric Prud’hommeaux: “SPARQL 1.1 Query Language,” W3C Recommendation, March 2013. [44] Todd J.

The opposite of bounded. 558 | Glossary Index A aborts (transactions), 222, 224 in two-phase commit, 356 performance of optimistic concurrency con‐ trol, 266 retrying aborted transactions, 231 abstraction, 21, 27, 222, 266, 321 access path (in network model), 37, 60 accidental complexity, removing, 21 accountability, 535 ACID properties (transactions), 90, 223 atomicity, 223, 228 consistency, 224, 529 durability, 226 isolation, 225, 228 acknowledgements (messaging), 445 active/active replication (see multi-leader repli‐ cation) active/passive replication (see leader-based rep‐ lication) ActiveMQ (messaging), 137, 444 distributed transaction support, 361 ActiveRecord (object-relational mapper), 30, 232 actor model, 138 (see also message-passing) comparison to Pregel model, 425 comparison to stream processing, 468 Advanced Message Queuing Protocol (see AMQP) aerospace systems, 6, 10, 305, 372 aggregation data cubes and materialized views, 101 in batch processes, 406 in stream processes, 466 aggregation pipeline query language, 48 Agile, 22 minimizing irreversibility, 414, 497 moving faster with confidence, 532 Unix philosophy, 394 agreement, 365 (see also consensus) Airflow (workflow scheduler), 402 Ajax, 131 Akka (actor framework), 139 algorithms algorithm correctness, 308 B-trees, 79-83 for distributed systems, 306 hash indexes, 72-75 mergesort, 76, 402, 405 red-black trees, 78 SSTables and LSM-trees, 76-79 all-to-all replication topologies, 175 AllegroGraph (database), 50 ALTER TABLE statement (SQL), 40, 111 Amazon Dynamo (database), 177 Amazon Web Services (AWS), 8 Kinesis Streams (messaging), 448 network reliability, 279 postmortems, 9 RedShift (database), 93 S3 (object storage), 398 checking data integrity, 530 amplification of bias, 534 of failures, 364, 495 Index | 559 of tail latency, 16, 207 write amplification, 84 AMQP (Advanced Message Queuing Protocol), 444 (see also messaging systems) comparison to log-based messaging, 448, 451 message ordering, 446 analytics, 90 comparison to transaction processing, 91 data warehousing (see data warehousing) parallel query execution in MPP databases, 415 predictive (see predictive analytics) relation to batch processing, 411 schemas for, 93-95 snapshot isolation for queries, 238 stream analytics, 466 using MapReduce, analysis of user activity events (example), 404 anti-caching (in-memory databases), 89 anti-entropy, 178 Apache ActiveMQ (see ActiveMQ) Apache Avro (see Avro) Apache Beam (see Beam) Apache BookKeeper (see BookKeeper) Apache Cassandra (see Cassandra) Apache CouchDB (see CouchDB) Apache Curator (see Curator) Apache Drill (see Drill) Apache Flink (see Flink) Apache Giraph (see Giraph) Apache Hadoop (see Hadoop) Apache HAWQ (see HAWQ) Apache HBase (see HBase) Apache Helix (see Helix) Apache Hive (see Hive) Apache Impala (see Impala) Apache Jena (see Jena) Apache Kafka (see Kafka) Apache Lucene (see Lucene) Apache MADlib (see MADlib) Apache Mahout (see Mahout) Apache Oozie (see Oozie) Apache Parquet (see Parquet) Apache Qpid (see Qpid) Apache Samza (see Samza) Apache Solr (see Solr) Apache Spark (see Spark) 560 | Index Apache Storm (see Storm) Apache Tajo (see Tajo) Apache Tez (see Tez) Apache Thrift (see Thrift) Apache ZooKeeper (see ZooKeeper) Apama (stream analytics), 466 append-only B-trees, 82, 242 append-only files (see logs) Application Programming Interfaces (APIs), 5, 27 for batch processing, 403 for change streams, 456 for distributed transactions, 361 for graph processing, 425 for services, 131-136 (see also services) evolvability, 136 RESTful, 133 SOAP, 133 application state (see state) approximate search (see similarity search) archival storage, data from databases, 131 arcs (see edges) arithmetic mean, 14 ASCII text, 119, 395 ASN.1 (schema language), 127 asynchronous networks, 278, 553 comparison to synchronous networks, 284 formal model, 307 asynchronous replication, 154, 553 conflict detection, 172 data loss on failover, 157 reads from asynchronous follower, 162 Asynchronous Transfer Mode (ATM), 285 atomic broadcast (see total order broadcast) atomic clocks (caesium clocks), 294, 295 (see also clocks) atomicity (concurrency), 553 atomic increment-and-get, 351 compare-and-set, 245, 327 (see also compare-and-set operations) replicated operations, 246 write operations, 243 atomicity (transactions), 223, 228, 553 atomic commit, 353 avoiding, 523, 528 blocking and nonblocking, 359 in stream processing, 360, 477 maintaining derived data, 453 for multi-object transactions, 229 for single-object writes, 230 auditability, 528-533 designing for, 531 self-auditing systems, 530 through immutability, 460 tools for auditable data systems, 532 availability, 8 (see also fault tolerance) in CAP theorem, 337 in service level agreements (SLAs), 15 Avro (data format), 122-127 code generation, 127 dynamically generated schemas, 126 object container files, 125, 131, 414 reader determining writer’s schema, 125 schema evolution, 123 use in Hadoop, 414 awk (Unix tool), 391 AWS (see Amazon Web Services) Azure (see Microsoft) B B-trees (indexes), 79-83 append-only/copy-on-write variants, 82, 242 branching factor, 81 comparison to LSM-trees, 83-85 crash recovery, 82 growing by splitting a page, 81 optimizations, 82 similarity to dynamic partitioning, 212 backpressure, 441, 553 in TCP, 282 backups database snapshot for replication, 156 integrity of, 530 snapshot isolation for, 238 use for ETL processes, 405 backward compatibility, 112 BASE, contrast to ACID, 223 bash shell (Unix), 70, 395, 503 batch processing, 28, 389-431, 553 combining with stream processing lambda architecture, 497 unifying technologies, 498 comparison to MPP databases, 414-418 comparison to stream processing, 464 comparison to Unix, 413-414 dataflow engines, 421-423 fault tolerance, 406, 414, 422, 442 for data integration, 494-498 graphs and iterative processing, 424-426 high-level APIs and languages, 403, 426-429 log-based messaging and, 451 maintaining derived state, 495 MapReduce and distributed filesystems, 397-413 (see also MapReduce) measuring performance, 13, 390 outputs, 411-413 key-value stores, 412 search indexes, 411 using Unix tools (example), 391-394 Bayou (database), 522 Beam (dataflow library), 498 bias, 534 big ball of mud, 20 Bigtable data model, 41, 99 binary data encodings, 115-128 Avro, 122-127 MessagePack, 116-117 Thrift and Protocol Buffers, 117-121 binary encoding based on schemas, 127 by network drivers, 128 binary strings, lack of support in JSON and XML, 114 BinaryProtocol encoding (Thrift), 118 Bitcask (storage engine), 72 crash recovery, 74 Bitcoin (cryptocurrency), 532 Byzantine fault tolerance, 305 concurrency bugs in exchanges, 233 bitmap indexes, 97 blockchains, 532 Byzantine fault tolerance, 305 blocking atomic commit, 359 Bloom (programming language), 504 Bloom filter (algorithm), 79, 466 BookKeeper (replicated log), 372 Bottled Water (change data capture), 455 bounded datasets, 430, 439, 553 (see also batch processing) bounded delays, 553 in networks, 285 process pauses, 298 broadcast hash joins, 409 Index | 561 brokerless messaging, 442 Brubeck (metrics aggregator), 442 BTM (transaction coordinator), 356 bulk synchronous parallel (BSP) model, 425 bursty network traffic patterns, 285 business data processing, 28, 90, 390 byte sequence, encoding data in, 112 Byzantine faults, 304-306, 307, 553 Byzantine fault-tolerant systems, 305, 532 Byzantine Generals Problem, 304 consensus algorithms and, 366 C caches, 89, 553 and materialized views, 101 as derived data, 386, 499-504 database as cache of transaction log, 460 in CPUs, 99, 338, 428 invalidation and maintenance, 452, 467 linearizability, 324 CAP theorem, 336-338, 554 Cascading (batch processing), 419, 427 hash joins, 409 workflows, 403 cascading failures, 9, 214, 281 Cascalog (batch processing), 60 Cassandra (database) column-family data model, 41, 99 compaction strategy, 79 compound primary key, 204 gossip protocol, 216 hash partitioning, 203-205 last-write-wins conflict resolution, 186, 292 leaderless replication, 177 linearizability, lack of, 335 log-structured storage, 78 multi-datacenter support, 184 partitioning scheme, 213 secondary indexes, 207 sloppy quorums, 184 cat (Unix tool), 391 causal context, 191 (see also causal dependencies) causal dependencies, 186-191 capturing, 191, 342, 494, 514 by total ordering, 493 causal ordering, 339 in transactions, 262 sending message to friends (example), 494 562 | Index causality, 554 causal ordering, 339-343 linearizability and, 342 total order consistent with, 344, 345 consistency with, 344-347 consistent snapshots, 340 happens-before relationship, 186 in serializable transactions, 262-265 mismatch with clocks, 292 ordering events to capture, 493 violations of, 165, 176, 292, 340 with synchronized clocks, 294 CEP (see complex event processing) certificate transparency, 532 chain replication, 155 linearizable reads, 351 change data capture, 160, 454 API support for change streams, 456 comparison to event sourcing, 457 implementing, 454 initial snapshot, 455 log compaction, 456 changelogs, 460 change data capture, 454 for operator state, 479 generating with triggers, 455 in stream joins, 474 log compaction, 456 maintaining derived state, 452 Chaos Monkey, 7, 280 checkpointing in batch processors, 422, 426 in high-performance computing, 275 in stream processors, 477, 523 chronicle data model, 458 circuit-switched networks, 284 circular buffers, 450 circular replication topologies, 175 clickstream data, analysis of, 404 clients calling services, 131 pushing state changes to, 512 request routing, 214 stateful and offline-capable, 170, 511 clocks, 287-299 atomic (caesium) clocks, 294, 295 confidence interval, 293-295 for global snapshots, 294 logical (see logical clocks) skew, 291-294, 334 slewing, 289 synchronization and accuracy, 289-291 synchronization using GPS, 287, 290, 294, 295 time-of-day versus monotonic clocks, 288 timestamping events, 471 cloud computing, 146, 275 need for service discovery, 372 network glitches, 279 shared resources, 284 single-machine reliability, 8 Cloudera Impala (see Impala) clustered indexes, 86 CODASYL model, 36 (see also network model) code generation with Avro, 127 with Thrift and Protocol Buffers, 118 with WSDL, 133 collaborative editing multi-leader replication and, 170 column families (Bigtable), 41, 99 column-oriented storage, 95-101 column compression, 97 distinction between column families and, 99 in batch processors, 428 Parquet, 96, 131, 414 sort order in, 99-100 vectorized processing, 99, 428 writing to, 101 comma-separated values (see CSV) command query responsibility segregation (CQRS), 462 commands (event sourcing), 459 commits (transactions), 222 atomic commit, 354-355 (see also atomicity; transactions) read committed isolation, 234 three-phase commit (3PC), 359 two-phase commit (2PC), 355-359 commutative operations, 246 compaction of changelogs, 456 (see also log compaction) for stream operator state, 479 of log-structured storage, 73 issues with, 84 size-tiered and leveled approaches, 79 CompactProtocol encoding (Thrift), 119 compare-and-set operations, 245, 327 implementing locks, 370 implementing uniqueness constraints, 331 implementing with total order broadcast, 350 relation to consensus, 335, 350, 352, 374 relation to transactions, 230 compatibility, 112, 128 calling services, 136 properties of encoding formats, 139 using databases, 129-131 using message-passing, 138 compensating transactions, 355, 461, 526 complex event processing (CEP), 465 complexity distilling in theoretical models, 310 hiding using abstraction, 27 of software systems, managing, 20 composing data systems (see unbundling data‐ bases) compute-intensive applications, 3, 275 concatenated indexes, 87 in Cassandra, 204 Concord (stream processor), 466 concurrency actor programming model, 138, 468 (see also message-passing) bugs from weak transaction isolation, 233 conflict resolution, 171, 174 detecting concurrent writes, 184-191 dual writes, problems with, 453 happens-before relationship, 186 in replicated systems, 161-191, 324-338 lost updates, 243 multi-version concurrency control (MVCC), 239 optimistic concurrency control, 261 ordering of operations, 326, 341 reducing, through event logs, 351, 462, 507 time and relativity, 187 transaction isolation, 225 write skew (transaction isolation), 246-251 conflict-free replicated datatypes (CRDTs), 174 conflicts conflict detection, 172 causal dependencies, 186, 342 in consensus algorithms, 368 in leaderless replication, 184 Index | 563 in log-based systems, 351, 521 in nonlinearizable systems, 343 in serializable snapshot isolation (SSI), 264 in two-phase commit, 357, 364 conflict resolution automatic conflict resolution, 174 by aborting transactions, 261 by apologizing, 527 convergence, 172-174 in leaderless systems, 190 last write wins (LWW), 186, 292 using atomic operations, 246 using custom logic, 173 determining what is a conflict, 174, 522 in multi-leader replication, 171-175 avoiding conflicts, 172 lost updates, 242-246 materializing, 251 relation to operation ordering, 339 write skew (transaction isolation), 246-251 congestion (networks) avoidance, 282 limiting accuracy of clocks, 293 queueing delays, 282 consensus, 321, 364-375, 554 algorithms, 366-368 preventing split brain, 367 safety and liveness properties, 365 using linearizable operations, 351 cost of, 369 distributed transactions, 352-375 in practice, 360-364 two-phase commit, 354-359 XA transactions, 361-364 impossibility of, 353 membership and coordination services, 370-373 relation to compare-and-set, 335, 350, 352, 374 relation to replication, 155, 349 relation to uniqueness constraints, 521 consistency, 224, 524 across different databases, 157, 452, 462, 492 causal, 339-348, 493 consistent prefix reads, 165-167 consistent snapshots, 156, 237-242, 294, 455, 500 (see also snapshots) 564 | Index crash recovery, 82 enforcing constraints (see constraints) eventual, 162, 322 (see also eventual consistency) in ACID transactions, 224, 529 in CAP theorem, 337 linearizability, 324-338 meanings of, 224 monotonic reads, 164-165 of secondary indexes, 231, 241, 354, 491, 500 ordering guarantees, 339-352 read-after-write, 162-164 sequential, 351 strong (see linearizability) timeliness and integrity, 524 using quorums, 181, 334 consistent hashing, 204 consistent prefix reads, 165 constraints (databases), 225, 248 asynchronously checked, 526 coordination avoidance, 527 ensuring idempotence, 519 in log-based systems, 521-524 across multiple partitions, 522 in two-phase commit, 355, 357 relation to consensus, 374, 521 relation to event ordering, 347 requiring linearizability, 330 Consul (service discovery), 372 consumers (message streams), 137, 440 backpressure, 441 consumer offsets in logs, 449 failures, 445, 449 fan-out, 11, 445, 448 load balancing, 444, 448 not keeping up with producers, 441, 450, 502 context switches, 14, 297 convergence (conflict resolution), 172-174, 322 coordination avoidance, 527 cross-datacenter, 168, 493 cross-partition ordering, 256, 294, 348, 523 services, 330, 370-373 coordinator (in 2PC), 356 failure, 358 in XA transactions, 361-364 recovery, 363 copy-on-write (B-trees), 82, 242 CORBA (Common Object Request Broker Architecture), 134 correctness, 6 auditability, 528-533 Byzantine fault tolerance, 305, 532 dealing with partial failures, 274 in log-based systems, 521-524 of algorithm within system model, 308 of compensating transactions, 355 of consensus, 368 of derived data, 497, 531 of immutable data, 461 of personal data, 535, 540 of time, 176, 289-295 of transactions, 225, 515, 529 timeliness and integrity, 524-528 corruption of data detecting, 519, 530-533 due to pathological memory access, 529 due to radiation, 305 due to split brain, 158, 302 due to weak transaction isolation, 233 formalization in consensus, 366 integrity as absence of, 524 network packets, 306 on disks, 227 preventing using write-ahead logs, 82 recovering from, 414, 460 Couchbase (database) durability, 89 hash partitioning, 203-204, 211 rebalancing, 213 request routing, 216 CouchDB (database) B-tree storage, 242 change feed, 456 document data model, 31 join support, 34 MapReduce support, 46, 400 replication, 170, 173 covering indexes, 86 CPUs cache coherence and memory barriers, 338 caching and pipelining, 99, 428 increasing parallelism, 43 CRDTs (see conflict-free replicated datatypes) CREATE INDEX statement (SQL), 85, 500 credit rating agencies, 535 Crunch (batch processing), 419, 427 hash joins, 409 sharded joins, 408 workflows, 403 cryptography defense against attackers, 306 end-to-end encryption and authentication, 519, 543 proving integrity of data, 532 CSS (Cascading Style Sheets), 44 CSV (comma-separated values), 70, 114, 396 Curator (ZooKeeper recipes), 330, 371 curl (Unix tool), 135, 397 cursor stability, 243 Cypher (query language), 52 comparison to SPARQL, 59 D data corruption (see corruption of data) data cubes, 102 data formats (see encoding) data integration, 490-498, 543 batch and stream processing, 494-498 lambda architecture, 497 maintaining derived state, 495 reprocessing data, 496 unifying, 498 by unbundling databases, 499-515 comparison to federated databases, 501 combining tools by deriving data, 490-494 derived data versus distributed transac‐ tions, 492 limits of total ordering, 493 ordering events to capture causality, 493 reasoning about dataflows, 491 need for, 385 data lakes, 415 data locality (see locality) data models, 27-64 graph-like models, 49-63 Datalog language, 60-63 property graphs, 50 RDF and triple-stores, 55-59 query languages, 42-48 relational model versus document model, 28-42 data protection regulations, 542 data systems, 3 about, 4 Index | 565 concerns when designing, 5 future of, 489-544 correctness, constraints, and integrity, 515-533 data integration, 490-498 unbundling databases, 499-515 heterogeneous, keeping in sync, 452 maintainability, 18-22 possible faults in, 221 reliability, 6-10 hardware faults, 7 human errors, 9 importance of, 10 software errors, 8 scalability, 10-18 unreliable clocks, 287-299 data warehousing, 91-95, 554 comparison to data lakes, 415 ETL (extract-transform-load), 92, 416, 452 keeping data systems in sync, 452 schema design, 93 slowly changing dimension (SCD), 476 data-intensive applications, 3 database triggers (see triggers) database-internal distributed transactions, 360, 364, 477 databases archival storage, 131 comparison of message brokers to, 443 dataflow through, 129 end-to-end argument for, 519-520 checking integrity, 531 inside-out, 504 (see also unbundling databases) output from batch workflows, 412 relation to event streams, 451-464 (see also changelogs) API support for change streams, 456, 506 change data capture, 454-457 event sourcing, 457-459 keeping systems in sync, 452-453 philosophy of immutable events, 459-464 unbundling, 499-515 composing data storage technologies, 499-504 designing applications around dataflow, 504-509 566 | Index observing derived state, 509-515 datacenters geographically distributed, 145, 164, 278, 493 multi-tenancy and shared resources, 284 network architecture, 276 network faults, 279 replication across multiple, 169 leaderless replication, 184 multi-leader replication, 168, 335 dataflow, 128-139, 504-509 correctness of dataflow systems, 525 differential, 504 message-passing, 136-139 reasoning about, 491 through databases, 129 through services, 131-136 dataflow engines, 421-423 comparison to stream processing, 464 directed acyclic graphs (DAG), 424 partitioning, approach to, 429 support for declarative queries, 427 Datalog (query language), 60-63 datatypes binary strings in XML and JSON, 114 conflict-free, 174 in Avro encodings, 122 in Thrift and Protocol Buffers, 121 numbers in XML and JSON, 114 Datomic (database) B-tree storage, 242 data model, 50, 57 Datalog query language, 60 excision (deleting data), 463 languages for transactions, 255 serial execution of transactions, 253 deadlocks detection, in two-phase commit (2PC), 364 in two-phase locking (2PL), 258 Debezium (change data capture), 455 declarative languages, 42, 554 Bloom, 504 CSS and XSL, 44 Cypher, 52 Datalog, 60 for batch processing, 427 recursive SQL queries, 53 relational algebra and SQL, 42 SPARQL, 59 delays bounded network delays, 285 bounded process pauses, 298 unbounded network delays, 282 unbounded process pauses, 296 deleting data, 463 denormalization (data representation), 34, 554 costs, 39 in derived data systems, 386 materialized views, 101 updating derived data, 228, 231, 490 versus normalization, 462 derived data, 386, 439, 554 from change data capture, 454 in event sourcing, 458-458 maintaining derived state through logs, 452-457, 459-463 observing, by subscribing to streams, 512 outputs of batch and stream processing, 495 through application code, 505 versus distributed transactions, 492 deterministic operations, 255, 274, 554 accidental nondeterminism, 423 and fault tolerance, 423, 426 and idempotence, 478, 492 computing derived data, 495, 526, 531 in state machine replication, 349, 452, 458 joins, 476 DevOps, 394 differential dataflow, 504 dimension tables, 94 dimensional modeling (see star schemas) directed acyclic graphs (DAGs), 424 dirty reads (transaction isolation), 234 dirty writes (transaction isolation), 235 discrimination, 534 disks (see hard disks) distributed actor frameworks, 138 distributed filesystems, 398-399 decoupling from query engines, 417 indiscriminately dumping data into, 415 use by MapReduce, 402 distributed systems, 273-312, 554 Byzantine faults, 304-306 cloud versus supercomputing, 275 detecting network faults, 280 faults and partial failures, 274-277 formalization of consensus, 365 impossibility results, 338, 353 issues with failover, 157 limitations of distributed transactions, 363 multi-datacenter, 169, 335 network problems, 277-286 quorums, relying on, 301 reasons for using, 145, 151 synchronized clocks, relying on, 291-295 system models, 306-310 use of clocks and time, 287 distributed transactions (see transactions) Django (web framework), 232 DNS (Domain Name System), 216, 372 Docker (container manager), 506 document data model, 30-42 comparison to relational model, 38-42 document references, 38, 403 document-oriented databases, 31 many-to-many relationships and joins, 36 multi-object transactions, need for, 231 versus relational model convergence of models, 41 data locality, 41 document-partitioned indexes, 206, 217, 411 domain-driven design (DDD), 457 DRBD (Distributed Replicated Block Device), 153 drift (clocks), 289 Drill (query engine), 93 Druid (database), 461 Dryad (dataflow engine), 421 dual writes, problems with, 452, 507 duplicates, suppression of, 517 (see also idempotence) using a unique ID, 518, 522 durability (transactions), 226, 554 duration (time), 287 measurement with monotonic clocks, 288 dynamic partitioning, 212 dynamically typed languages analogy to schema-on-read, 40 code generation and, 127 Dynamo-style databases (see leaderless replica‐ tion) E edges (in graphs), 49, 403 property graph model, 50 edit distance (full-text search), 88 effectively-once semantics, 476, 516 Index | 567 (see also exactly-once semantics) preservation of integrity, 525 elastic systems, 17 Elasticsearch (search server) document-partitioned indexes, 207 partition rebalancing, 211 percolator (stream search), 467 usage example, 4 use of Lucene, 79 ElephantDB (database), 413 Elm (programming language), 504, 512 encodings (data formats), 111-128 Avro, 122-127 binary variants of JSON and XML, 115 compatibility, 112 calling services, 136 using databases, 129-131 using message-passing, 138 defined, 113 JSON, XML, and CSV, 114 language-specific formats, 113 merits of schemas, 127 representations of data, 112 Thrift and Protocol Buffers, 117-121 end-to-end argument, 277, 519-520 checking integrity, 531 publish/subscribe streams, 512 enrichment (stream), 473 Enterprise JavaBeans (EJB), 134 entities (see vertices) epoch (consensus algorithms), 368 epoch (Unix timestamps), 288 equi-joins, 403 erasure coding (error correction), 398 Erlang OTP (actor framework), 139 error handling for network faults, 280 in transactions, 231 error-correcting codes, 277, 398 Esper (CEP engine), 466 etcd (coordination service), 370-373 linearizable operations, 333 locks and leader election, 330 quorum reads, 351 service discovery, 372 use of Raft algorithm, 349, 353 Ethereum (blockchain), 532 Ethernet (networks), 276, 278, 285 packet checksums, 306, 519 568 | Index Etherpad (collaborative editor), 170 ethics, 533-543 code of ethics and professional practice, 533 legislation and self-regulation, 542 predictive analytics, 533-536 amplifying bias, 534 feedback loops, 536 privacy and tracking, 536-543 consent and freedom of choice, 538 data as assets and power, 540 meaning of privacy, 539 surveillance, 537 respect, dignity, and agency, 543, 544 unintended consequences, 533, 536 ETL (extract-transform-load), 92, 405, 452, 554 use of Hadoop for, 416 event sourcing, 457-459 commands and events, 459 comparison to change data capture, 457 comparison to lambda architecture, 497 deriving current state from event log, 458 immutability and auditability, 459, 531 large, reliable data systems, 519, 526 Event Store (database), 458 event streams (see streams) events, 440 deciding on total order of, 493 deriving views from event log, 461 difference to commands, 459 event time versus processing time, 469, 477, 498 immutable, advantages of, 460, 531 ordering to capture causality, 493 reads as, 513 stragglers, 470, 498 timestamp of, in stream processing, 471 EventSource (browser API), 512 eventual consistency, 152, 162, 308, 322 (see also conflicts) and perpetual inconsistency, 525 evolvability, 21, 111 calling services, 136 graph-structured data, 52 of databases, 40, 129-131, 461, 497 of message-passing, 138 reprocessing data, 496, 498 schema evolution in Avro, 123 schema evolution in Thrift and Protocol Buffers, 120 schema-on-read, 39, 111, 128 exactly-once semantics, 360, 476, 516 parity with batch processors, 498 preservation of integrity, 525 exclusive mode (locks), 258 eXtended Architecture transactions (see XA transactions) extract-transform-load (see ETL) F Facebook Presto (query engine), 93 React, Flux, and Redux (user interface libra‐ ries), 512 social graphs, 49 Wormhole (change data capture), 455 fact tables, 93 failover, 157, 554 (see also leader-based replication) in leaderless replication, absence of, 178 leader election, 301, 348, 352 potential problems, 157 failures amplification by distributed transactions, 364, 495 failure detection, 280 automatic rebalancing causing cascading failures, 214 perfect failure detectors, 359 timeouts and unbounded delays, 282, 284 using ZooKeeper, 371 faults versus, 7 partial failures in distributed systems, 275-277, 310 fan-out (messaging systems), 11, 445 fault tolerance, 6-10, 555 abstractions for, 321 formalization in consensus, 365-369 use of replication, 367 human fault tolerance, 414 in batch processing, 406, 414, 422, 425 in log-based systems, 520, 524-526 in stream processing, 476-479 atomic commit, 477 idempotence, 478 maintaining derived state, 495 microbatching and checkpointing, 477 rebuilding state after a failure, 478 of distributed transactions, 362-364 transaction atomicity, 223, 354-361 faults, 6 Byzantine faults, 304-306 failures versus, 7 handled by transactions, 221 handling in supercomputers and cloud computing, 275 hardware, 7 in batch processing versus distributed data‐ bases, 417 in distributed systems, 274-277 introducing deliberately, 7, 280 network faults, 279-281 asymmetric faults, 300 detecting, 280 tolerance of, in multi-leader replication, 169 software errors, 8 tolerating (see fault tolerance) federated databases, 501 fence (CPU instruction), 338 fencing (preventing split brain), 158, 302-304 generating fencing tokens, 349, 370 properties of fencing tokens, 308 stream processors writing to databases, 478, 517 Fibre Channel (networks), 398 field tags (Thrift and Protocol Buffers), 119-121 file descriptors (Unix), 395 financial data, 460 Firebase (database), 456 Flink (processing framework), 421-423 dataflow APIs, 427 fault tolerance, 422, 477, 479 Gelly API (graph processing), 425 integration of batch and stream processing, 495, 498 machine learning, 428 query optimizer, 427 stream processing, 466 flow control, 282, 441, 555 FLP result (on consensus), 353 FlumeJava (dataflow library), 403, 427 followers, 152, 555 (see also leader-based replication) foreign keys, 38, 403 forward compatibility, 112 forward decay (algorithm), 16 Index | 569 Fossil (version control system), 463 shunning (deleting data), 463 FoundationDB (database) serializable transactions, 261, 265, 364 fractal trees, 83 full table scans, 403 full-text search, 555 and fuzzy indexes, 88 building search indexes, 411 Lucene storage engine, 79 functional reactive programming (FRP), 504 functional requirements, 22 futures (asynchronous operations), 135 fuzzy search (see similarity search) G garbage collection immutability and, 463 process pauses for, 14, 296-299, 301 (see also process pauses) genome analysis, 63, 429 geographically distributed datacenters, 145, 164, 278, 493 geospatial indexes, 87 Giraph (graph processing), 425 Git (version control system), 174, 342, 463 GitHub, postmortems, 157, 158, 309 global indexes (see term-partitioned indexes) GlusterFS (distributed filesystem), 398 GNU Coreutils (Linux), 394 GoldenGate (change data capture), 161, 170, 455 (see also Oracle) Google Bigtable (database) data model (see Bigtable data model) partitioning scheme, 199, 202 storage layout, 78 Chubby (lock service), 370 Cloud Dataflow (stream processor), 466, 477, 498 (see also Beam) Cloud Pub/Sub (messaging), 444, 448 Docs (collaborative editor), 170 Dremel (query engine), 93, 96 FlumeJava (dataflow library), 403, 427 GFS (distributed file system), 398 gRPC (RPC framework), 135 MapReduce (batch processing), 390 570 | Index (see also MapReduce) building search indexes, 411 task preemption, 418 Pregel (graph processing), 425 Spanner (see Spanner) TrueTime (clock API), 294 gossip protocol, 216 government use of data, 541 GPS (Global Positioning System) use for clock synchronization, 287, 290, 294, 295 GraphChi (graph processing), 426 graphs, 555 as data models, 49-63 example of graph-structured data, 49 property graphs, 50 RDF and triple-stores, 55-59 versus the network model, 60 processing and analysis, 424-426 fault tolerance, 425 Pregel processing model, 425 query languages Cypher, 52 Datalog, 60-63 recursive SQL queries, 53 SPARQL, 59-59 Gremlin (graph query language), 50 grep (Unix tool), 392 GROUP BY clause (SQL), 406 grouping records in MapReduce, 406 handling skew, 407 H Hadoop (data infrastructure) comparison to distributed databases, 390 comparison to MPP databases, 414-418 comparison to Unix, 413-414, 499 diverse processing models in ecosystem, 417 HDFS distributed filesystem (see HDFS) higher-level tools, 403 join algorithms, 403-410 (see also MapReduce) MapReduce (see MapReduce) YARN (see YARN) happens-before relationship, 340 capturing, 187 concurrency and, 186 hard disks access patterns, 84 detecting corruption, 519, 530 faults in, 7, 227 sequential write throughput, 75, 450 hardware faults, 7 hash indexes, 72-75 broadcast hash joins, 409 partitioned hash joins, 409 hash partitioning, 203-205, 217 consistent hashing, 204 problems with hash mod N, 210 range queries, 204 suitable hash functions, 203 with fixed number of partitions, 210 HAWQ (database), 428 HBase (database) bug due to lack of fencing, 302 bulk loading, 413 column-family data model, 41, 99 dynamic partitioning, 212 key-range partitioning, 202 log-structured storage, 78 request routing, 216 size-tiered compaction, 79 use of HDFS, 417 use of ZooKeeper, 370 HDFS (Hadoop Distributed File System), 398-399 (see also distributed filesystems) checking data integrity, 530 decoupling from query engines, 417 indiscriminately dumping data into, 415 metadata about datasets, 410 NameNode, 398 use by Flink, 479 use by HBase, 212 use by MapReduce, 402 HdrHistogram (numerical library), 16 head (Unix tool), 392 head vertex (property graphs), 51 head-of-line blocking, 15 heap files (databases), 86 Helix (cluster manager), 216 heterogeneous distributed transactions, 360, 364 heuristic decisions (in 2PC), 363 Hibernate (object-relational mapper), 30 hierarchical model, 36 high availability (see fault tolerance) high-frequency trading, 290, 299 high-performance computing (HPC), 275 hinted handoff, 183 histograms, 16 Hive (query engine), 419, 427 for data warehouses, 93 HCatalog and metastore, 410 map-side joins, 409 query optimizer, 427 skewed joins, 408 workflows, 403 Hollerith machines, 390 hopping windows (stream processing), 472 (see also windows) horizontal scaling (see scaling out) HornetQ (messaging), 137, 444 distributed transaction support, 361 hot spots, 201 due to celebrities, 205 for time-series data, 203 in batch processing, 407 relieving, 205 hot standbys (see leader-based replication) HTTP, use in APIs (see services) human errors, 9, 279, 414 HyperDex (database), 88 HyperLogLog (algorithm), 466 I I/O operations, waiting for, 297 IBM DB2 (database) distributed transaction support, 361 recursive query support, 54 serializable isolation, 242, 257 XML and JSON support, 30, 42 electromechanical card-sorting machines, 390 IMS (database), 36 imperative query APIs, 46 InfoSphere Streams (CEP engine), 466 MQ (messaging), 444 distributed transaction support, 361 System R (database), 222 WebSphere (messaging), 137 idempotence, 134, 478, 555 by giving operations unique IDs, 518, 522 idempotent operations, 517 immutability advantages of, 460, 531 Index | 571 deriving state from event log, 459-464 for crash recovery, 75 in B-trees, 82, 242 in event sourcing, 457 inputs to Unix commands, 397 limitations of, 463 Impala (query engine) for data warehouses, 93 hash joins, 409 native code generation, 428 use of HDFS, 417 impedance mismatch, 29 imperative languages, 42 setting element styles (example), 45 in doubt (transaction status), 358 holding locks, 362 orphaned transactions, 363 in-memory databases, 88 durability, 227 serial transaction execution, 253 incidents cascading failures, 9 crashes due to leap seconds, 290 data corruption and financial losses due to concurrency bugs, 233 data corruption on hard disks, 227 data loss due to last-write-wins, 173, 292 data on disks unreadable, 309 deleted items reappearing, 174 disclosure of sensitive data due to primary key reuse, 157 errors in transaction serializability, 529 gigabit network interface with 1 Kb/s throughput, 311 network faults, 279 network interface dropping only inbound packets, 279 network partitions and whole-datacenter failures, 275 poor handling of network faults, 280 sending message to ex-partner, 494 sharks biting undersea cables, 279 split brain due to 1-minute packet delay, 158, 279 vibrations in server rack, 14 violation of uniqueness constraint, 529 indexes, 71, 555 and snapshot isolation, 241 as derived data, 386, 499-504 572 | Index B-trees, 79-83 building in batch processes, 411 clustered, 86 comparison of B-trees and LSM-trees, 83-85 concatenated, 87 covering (with included columns), 86 creating, 500 full-text search, 88 geospatial, 87 hash, 72-75 index-range locking, 260 multi-column, 87 partitioning and secondary indexes, 206-209, 217 secondary, 85 (see also secondary indexes) problems with dual writes, 452, 491 SSTables and LSM-trees, 76-79 updating when data changes, 452, 467 Industrial Revolution, 541 InfiniBand (networks), 285 InfiniteGraph (database), 50 InnoDB (storage engine) clustered index on primary key, 86 not preventing lost updates, 245 preventing write skew, 248, 257 serializable isolation, 257 snapshot isolation support, 239 inside-out databases, 504 (see also unbundling databases) integrating different data systems (see data integration) integrity, 524 coordination-avoiding data systems, 528 correctness of dataflow systems, 525 in consensus formalization, 365 integrity checks, 530 (see also auditing) end-to-end, 519, 531 use of snapshot isolation, 238 maintaining despite software bugs, 529 Interface Definition Language (IDL), 117, 122 intermediate state, materialization of, 420-423 internet services, systems for implementing, 275 invariants, 225 (see also constraints) inversion of control, 396 IP (Internet Protocol) unreliability of, 277 ISDN (Integrated Services Digital Network), 284 isolation (in transactions), 225, 228, 555 correctness and, 515 for single-object writes, 230 serializability, 251-266 actual serial execution, 252-256 serializable snapshot isolation (SSI), 261-266 two-phase locking (2PL), 257-261 violating, 228 weak isolation levels, 233-251 preventing lost updates, 242-246 read committed, 234-237 snapshot isolation, 237-242 iterative processing, 424-426 J Java Database Connectivity (JDBC) distributed transaction support, 361 network drivers, 128 Java Enterprise Edition (EE), 134, 356, 361 Java Message Service (JMS), 444 (see also messaging systems) comparison to log-based messaging, 448, 451 distributed transaction support, 361 message ordering, 446 Java Transaction API (JTA), 355, 361 Java Virtual Machine (JVM) bytecode generation, 428 garbage collection pauses, 296 process reuse in batch processors, 422 JavaScript in MapReduce querying, 46 setting element styles (example), 45 use in advanced queries, 48 Jena (RDF framework), 57 Jepsen (fault tolerance testing), 515 jitter (network delay), 284 joins, 555 by index lookup, 403 expressing as relational operators, 427 in relational and document databases, 34 MapReduce map-side joins, 408-410 broadcast hash joins, 409 merge joins, 410 partitioned hash joins, 409 MapReduce reduce-side joins, 403-408 handling skew, 407 sort-merge joins, 405 parallel execution of, 415 secondary indexes and, 85 stream joins, 472-476 stream-stream join, 473 stream-table join, 473 table-table join, 474 time-dependence of, 475 support in document databases, 42 JOTM (transaction coordinator), 356 JSON Avro schema representation, 122 binary variants, 115 for application data, issues with, 114 in relational databases, 30, 42 representing a résumé (example), 31 Juttle (query language), 504 K k-nearest neighbors, 429 Kafka (messaging), 137, 448 Kafka Connect (database integration), 457, 461 Kafka Streams (stream processor), 466, 467 fault tolerance, 479 leader-based replication, 153 log compaction, 456, 467 message offsets, 447, 478 request routing, 216 transaction support, 477 usage example, 4 Ketama (partitioning library), 213 key-value stores, 70 as batch process output, 412 hash indexes, 72-75 in-memory, 89 partitioning, 201-205 by hash of key, 203, 217 by key range, 202, 217 dynamic partitioning, 212 skew and hot spots, 205 Kryo (Java), 113 Kubernetes (cluster manager), 418, 506 L lambda architecture, 497 Lamport timestamps, 345 Index | 573 Large Hadron Collider (LHC), 64 last write wins (LWW), 173, 334 discarding concurrent writes, 186 problems with, 292 prone to lost updates, 246 late binding, 396 latency instability under two-phase locking, 259 network latency and resource utilization, 286 response time versus, 14 tail latency, 15, 207 leader-based replication, 152-161 (see also replication) failover, 157, 301 handling node outages, 156 implementation of replication logs change data capture, 454-457 (see also changelogs) statement-based, 158 trigger-based replication, 161 write-ahead log (WAL) shipping, 159 linearizability of operations, 333 locking and leader election, 330 log sequence number, 156, 449 read-scaling architecture, 161 relation to consensus, 367 setting up new followers, 155 synchronous versus asynchronous, 153-155 leaderless replication, 177-191 (see also replication) detecting concurrent writes, 184-191 capturing happens-before relationship, 187 happens-before relationship and concur‐ rency, 186 last write wins, 186 merging concurrently written values, 190 version vectors, 191 multi-datacenter, 184 quorums, 179-182 consistency limitations, 181-183, 334 sloppy quorums and hinted handoff, 183 read repair and anti-entropy, 178 leap seconds, 8, 290 in time-of-day clocks, 288 leases, 295 implementation with ZooKeeper, 370 574 | Index need for fencing, 302 ledgers, 460 distributed ledger technologies, 532 legacy systems, maintenance of, 18 less (Unix tool), 397 LevelDB (storage engine), 78 leveled compaction, 79 Levenshtein automata, 88 limping (partial failure), 311 linearizability, 324-338, 555 cost of, 335-338 CAP theorem, 336 memory on multi-core CPUs, 338 definition, 325-329 implementing with total order broadcast, 350 in ZooKeeper, 370 of derived data systems, 492, 524 avoiding coordination, 527 of different replication methods, 332-335 using quorums, 334 relying on, 330-332 constraints and uniqueness, 330 cross-channel timing dependencies, 331 locking and leader election, 330 stronger than causal consistency, 342 using to implement total order broadcast, 351 versus serializability, 329 LinkedIn Azkaban (workflow scheduler), 402 Databus (change data capture), 161, 455 Espresso (database), 31, 126, 130, 153, 216 Helix (cluster manager) (see Helix) profile (example), 30 reference to company entity (example), 34 Rest.li (RPC framework), 135 Voldemort (database) (see Voldemort) Linux, leap second bug, 8, 290 liveness properties, 308 LMDB (storage engine), 82, 242 load approaches to coping with, 17 describing, 11 load testing, 16 load balancing (messaging), 444 local indexes (see document-partitioned indexes) locality (data access), 32, 41, 555 in batch processing, 400, 405, 421 in stateful clients, 170, 511 in stream processing, 474, 478, 508, 522 location transparency, 134 in the actor model, 138 locks, 556 deadlock, 258 distributed locking, 301-304, 330 fencing tokens, 303 implementation with ZooKeeper, 370 relation to consensus, 374 for transaction isolation in snapshot isolation, 239 in two-phase locking (2PL), 257-261 making operations atomic, 243 performance, 258 preventing dirty writes, 236 preventing phantoms with index-range locks, 260, 265 read locks (shared mode), 236, 258 shared mode and exclusive mode, 258 in two-phase commit (2PC) deadlock detection, 364 in-doubt transactions holding locks, 362 materializing conflicts with, 251 preventing lost updates by explicit locking, 244 log sequence number, 156, 449 logic programming languages, 504 logical clocks, 293, 343, 494 for read-after-write consistency, 164 logical logs, 160 logs (data structure), 71, 556 advantages of immutability, 460 compaction, 73, 79, 456, 460 for stream operator state, 479 creating using total order broadcast, 349 implementing uniqueness constraints, 522 log-based messaging, 446-451 comparison to traditional messaging, 448, 451 consumer offsets, 449 disk space usage, 450 replaying old messages, 451, 496, 498 slow consumers, 450 using logs for message storage, 447 log-structured storage, 71-79 log-structured merge tree (see LSMtrees) replication, 152, 158-161 change data capture, 454-457 (see also changelogs) coordination with snapshot, 156 logical (row-based) replication, 160 statement-based replication, 158 trigger-based replication, 161 write-ahead log (WAL) shipping, 159 scalability limits, 493 loose coupling, 396, 419, 502 lost updates (see updates) LSM-trees (indexes), 78-79 comparison to B-trees, 83-85 Lucene (storage engine), 79 building indexes in batch processes, 411 similarity search, 88 Luigi (workflow scheduler), 402 LWW (see last write wins) M machine learning ethical considerations, 534 (see also ethics) iterative processing, 424 models derived from training data, 505 statistical and numerical algorithms, 428 MADlib (machine learning toolkit), 428 magic scaling sauce, 18 Mahout (machine learning toolkit), 428 maintainability, 18-22, 489 defined, 23 design principles for software systems, 19 evolvability (see evolvability) operability, 19 simplicity and managing complexity, 20 many-to-many relationships in document model versus relational model, 39 modeling as graphs, 49 many-to-one and many-to-many relationships, 33-36 many-to-one relationships, 34 MapReduce (batch processing), 390, 399-400 accessing external services within job, 404, 412 comparison to distributed databases designing for frequent faults, 417 diversity of processing models, 416 diversity of storage, 415 Index | 575 comparison to stream processing, 464 comparison to Unix, 413-414 disadvantages and limitations of, 419 fault tolerance, 406, 414, 422 higher-level tools, 403, 426 implementation in Hadoop, 400-403 the shuffle, 402 implementation in MongoDB, 46-48 machine learning, 428 map-side processing, 408-410 broadcast hash joins, 409 merge joins, 410 partitioned hash joins, 409 mapper and reducer functions, 399 materialization of intermediate state, 419-423 output of batch workflows, 411-413 building search indexes, 411 key-value stores, 412 reduce-side processing, 403-408 analysis of user activity events (exam‐ ple), 404 grouping records by same key, 406 handling skew, 407 sort-merge joins, 405 workflows, 402 marshalling (see encoding) massively parallel processing (MPP), 216 comparison to composing storage technolo‐ gies, 502 comparison to Hadoop, 414-418, 428 master-master replication (see multi-leader replication) master-slave replication (see leader-based repli‐ cation) materialization, 556 aggregate values, 101 conflicts, 251 intermediate state (batch processing), 420-423 materialized views, 101 as derived data, 386, 499-504 maintaining, using stream processing, 467, 475 Maven (Java build tool), 428 Maxwell (change data capture), 455 mean, 14 media monitoring, 467 median, 14 576 | Index meeting room booking (example), 249, 259, 521 membership services, 372 Memcached (caching server), 4, 89 memory in-memory databases, 88 durability, 227 serial transaction execution, 253 in-memory representation of data, 112 random bit-flips in, 529 use by indexes, 72, 77 memory barrier (CPU instruction), 338 MemSQL (database) in-memory storage, 89 read committed isolation, 236 memtable (in LSM-trees), 78 Mercurial (version control system), 463 merge joins, MapReduce map-side, 410 mergeable persistent data structures, 174 merging sorted files, 76, 402, 405 Merkle trees, 532 Mesos (cluster manager), 418, 506 message brokers (see messaging systems) message-passing, 136-139 advantages over direct RPC, 137 distributed actor frameworks, 138 evolvability, 138 MessagePack (encoding format), 116 messages exactly-once semantics, 360, 476 loss of, 442 using total order broadcast, 348 messaging systems, 440-451 (see also streams) backpressure, buffering, or dropping mes‐ sages, 441 brokerless messaging, 442 event logs, 446-451 comparison to traditional messaging, 448, 451 consumer offsets, 449 replaying old messages, 451, 496, 498 slow consumers, 450 message brokers, 443-446 acknowledgements and redelivery, 445 comparison to event logs, 448, 451 multiple consumers of same topic, 444 reliability, 442 uniqueness in log-based messaging, 522 Meteor (web framework), 456 microbatching, 477, 495 microservices, 132 (see also services) causal dependencies across services, 493 loose coupling, 502 relation to batch/stream processors, 389, 508 Microsoft Azure Service Bus (messaging), 444 Azure Storage, 155, 398 Azure Stream Analytics, 466 DCOM (Distributed Component Object Model), 134 MSDTC (transaction coordinator), 356 Orleans (see Orleans) SQL Server (see SQL Server) migrating (rewriting) data, 40, 130, 461, 497 modulus operator (%), 210 MongoDB (database) aggregation pipeline, 48 atomic operations, 243 BSON, 41 document data model, 31 hash partitioning (sharding), 203-204 key-range partitioning, 202 lack of join support, 34, 42 leader-based replication, 153 MapReduce support, 46, 400 oplog parsing, 455, 456 partition splitting, 212 request routing, 216 secondary indexes, 207 Mongoriver (change data capture), 455 monitoring, 10, 19 monotonic clocks, 288 monotonic reads, 164 MPP (see massively parallel processing) MSMQ (messaging), 361 multi-column indexes, 87 multi-leader replication, 168-177 (see also replication) handling write conflicts, 171 conflict avoidance, 172 converging toward a consistent state, 172 custom conflict resolution logic, 173 determining what is a conflict, 174 linearizability, lack of, 333 replication topologies, 175-177 use cases, 168 clients with offline operation, 170 collaborative editing, 170 multi-datacenter replication, 168, 335 multi-object transactions, 228 need for, 231 Multi-Paxos (total order broadcast), 367 multi-table index cluster tables (Oracle), 41 multi-tenancy, 284 multi-version concurrency control (MVCC), 239, 266 detecting stale MVCC reads, 263 indexes and snapshot isolation, 241 mutual exclusion, 261 (see also locks) MySQL (database) binlog coordinates, 156 binlog parsing for change data capture, 455 circular replication topology, 175 consistent snapshots, 156 distributed transaction support, 361 InnoDB storage engine (see InnoDB) JSON support, 30, 42 leader-based replication, 153 performance of XA transactions, 360 row-based replication, 160 schema changes in, 40 snapshot isolation support, 242 (see also InnoDB) statement-based replication, 159 Tungsten Replicator (multi-leader replica‐ tion), 170 conflict detection, 177 N nanomsg (messaging library), 442 Narayana (transaction coordinator), 356 NATS (messaging), 137 near-real-time (nearline) processing, 390 (see also stream processing) Neo4j (database) Cypher query language, 52 graph data model, 50 Nephele (dataflow engine), 421 netcat (Unix tool), 397 Netflix Chaos Monkey, 7, 280 Network Attached Storage (NAS), 146, 398 network model, 36 Index | 577 graph databases versus, 60 imperative query APIs, 46 Network Time Protocol (see NTP) networks congestion and queueing, 282 datacenter network topologies, 276 faults (see faults) linearizability and network delays, 338 network partitions, 279, 337 timeouts and unbounded delays, 281 next-key locking, 260 nodes (in graphs) (see vertices) nodes (processes), 556 handling outages in leader-based replica‐ tion, 156 system models for failure, 307 noisy neighbors, 284 nonblocking atomic commit, 359 nondeterministic operations accidental nondeterminism, 423 partial failures in distributed systems, 275 nonfunctional requirements, 22 nonrepeatable reads, 238 (see also read skew) normalization (data representation), 33, 556 executing joins, 39, 42, 403 foreign key references, 231 in systems of record, 386 versus denormalization, 462 NoSQL, 29, 499 transactions and, 223 Notation3 (N3), 56 npm (package manager), 428 NTP (Network Time Protocol), 287 accuracy, 289, 293 adjustments to monotonic clocks, 289 multiple server addresses, 306 numbers, in XML and JSON encodings, 114 O object-relational mapping (ORM) frameworks, 30 error handling and aborted transactions, 232 unsafe read-modify-write cycle code, 244 object-relational mismatch, 29 observer pattern, 506 offline systems, 390 (see also batch processing) 578 | Index stateful, offline-capable clients, 170, 511 offline-first applications, 511 offsets consumer offsets in partitioned logs, 449 messages in partitioned logs, 447 OLAP (online analytic processing), 91, 556 data cubes, 102 OLTP (online transaction processing), 90, 556 analytics queries versus, 411 workload characteristics, 253 one-to-many relationships, 30 JSON representation, 32 online systems, 389 (see also services) Oozie (workflow scheduler), 402 OpenAPI (service definition format), 133 OpenStack Nova (cloud infrastructure) use of ZooKeeper, 370 Swift (object storage), 398 operability, 19 operating systems versus databases, 499 operation identifiers, 518, 522 operational transformation, 174 operators, 421 flow of data between, 424 in stream processing, 464 optimistic concurrency control, 261 Oracle (database) distributed transaction support, 361 GoldenGate (change data capture), 161, 170, 455 lack of serializability, 226 leader-based replication, 153 multi-table index cluster tables, 41 not preventing write skew, 248 partitioned indexes, 209 PL/SQL language, 255 preventing lost updates, 245 read committed isolation, 236 Real Application Clusters (RAC), 330 recursive query support, 54 snapshot isolation support, 239, 242 TimesTen (in-memory database), 89 WAL-based replication, 160 XML support, 30 ordering, 339-352 by sequence numbers, 343-348 causal ordering, 339-343 partial order, 341 limits of total ordering, 493 total order broadcast, 348-352 Orleans (actor framework), 139 outliers (response time), 14 Oz (programming language), 504 P package managers, 428, 505 packet switching, 285 packets corruption of, 306 sending via UDP, 442 PageRank (algorithm), 49, 424 paging (see virtual memory) ParAccel (database), 93 parallel databases (see massively parallel pro‐ cessing) parallel execution of graph analysis algorithms, 426 queries in MPP databases, 216 Parquet (data format), 96, 131 (see also column-oriented storage) use in Hadoop, 414 partial failures, 275, 310 limping, 311 partial order, 341 partitioning, 199-218, 556 and replication, 200 in batch processing, 429 multi-partition operations, 514 enforcing constraints, 522 secondary index maintenance, 495 of key-value data, 201-205 by key range, 202 skew and hot spots, 205 rebalancing partitions, 209-214 automatic or manual rebalancing, 213 problems with hash mod N, 210 using dynamic partitioning, 212 using fixed number of partitions, 210 using N partitions per node, 212 replication and, 147 request routing, 214-216 secondary indexes, 206-209 document-based partitioning, 206 term-based partitioning, 208 serial execution of transactions and, 255 Paxos (consensus algorithm), 366 ballot number, 368 Multi-Paxos (total order broadcast), 367 percentiles, 14, 556 calculating efficiently, 16 importance of high percentiles, 16 use in service level agreements (SLAs), 15 Percona XtraBackup (MySQL tool), 156 performance describing, 13 of distributed transactions, 360 of in-memory databases, 89 of linearizability, 338 of multi-leader replication, 169 perpetual inconsistency, 525 pessimistic concurrency control, 261 phantoms (transaction isolation), 250 materializing conflicts, 251 preventing, in serializability, 259 physical clocks (see clocks) pickle (Python), 113 Pig (dataflow language), 419, 427 replicated joins, 409 skewed joins, 407 workflows, 403 Pinball (workflow scheduler), 402 pipelined execution, 423 in Unix, 394 point in time, 287 polyglot persistence, 29 polystores, 501 PostgreSQL (database) BDR (multi-leader replication), 170 causal ordering of writes, 177 Bottled Water (change data capture), 455 Bucardo (trigger-based replication), 161, 173 distributed transaction support, 361 foreign data wrappers, 501 full text search support, 490 leader-based replication, 153 log sequence number, 156 MVCC implementation, 239, 241 PL/pgSQL language, 255 PostGIS geospatial indexes, 87 preventing lost updates, 245 preventing write skew, 248, 261 read committed isolation, 236 recursive query support, 54 representing graphs, 51 Index | 579 serializable snapshot isolation (SSI), 261 snapshot isolation support, 239, 242 WAL-based replication, 160 XML and JSON support, 30, 42 pre-splitting, 212 Precision Time Protocol (PTP), 290 predicate locks, 259 predictive analytics, 533-536 amplifying bias, 534 ethics of (see ethics) feedback loops, 536 preemption of datacenter resources, 418 of threads, 298 Pregel processing model, 425 primary keys, 85, 556 compound primary key (Cassandra), 204 primary-secondary replication (see leaderbased replication) privacy, 536-543 consent and freedom of choice, 538 data as assets and power, 540 deleting data, 463 ethical considerations (see ethics) legislation and self-regulation, 542 meaning of, 539 surveillance, 537 tracking behavioral data, 536 probabilistic algorithms, 16, 466 process pauses, 295-299 processing time (of events), 469 producers (message streams), 440 programming languages dataflow languages, 504 for stored procedures, 255 functional reactive programming (FRP), 504 logic programming, 504 Prolog (language), 61 (see also Datalog) promises (asynchronous operations), 135 property graphs, 50 Cypher query language, 52 Protocol Buffers (data format), 117-121 field tags and schema evolution, 120 provenance of data, 531 publish/subscribe model, 441 publishers (message streams), 440 punch card tabulating machines, 390 580 | Index pure functions, 48 putting computation near data, 400 Q Qpid (messaging), 444 quality of service (QoS), 285 Quantcast File System (distributed filesystem), 398 query languages, 42-48 aggregation pipeline, 48 CSS and XSL, 44 Cypher, 52 Datalog, 60 Juttle, 504 MapReduce querying, 46-48 recursive SQL queries, 53 relational algebra and SQL, 42 SPARQL, 59 query optimizers, 37, 427 queueing delays (networks), 282 head-of-line blocking, 15 latency and response time, 14 queues (messaging), 137 quorums, 179-182, 556 for leaderless replication, 179 in consensus algorithms, 368 limitations of consistency, 181-183, 334 making decisions in distributed systems, 301 monitoring staleness, 182 multi-datacenter replication, 184 relying on durability, 309 sloppy quorums and hinted handoff, 183 R R-trees (indexes), 87 RabbitMQ (messaging), 137, 444 leader-based replication, 153 race conditions, 225 (see also concurrency) avoiding with linearizability, 331 caused by dual writes, 452 dirty writes, 235 in counter increments, 235 lost updates, 242-246 preventing with event logs, 462, 507 preventing with serializable isolation, 252 write skew, 246-251 Raft (consensus algorithm), 366 sensitivity to network problems, 369 term number, 368 use in etcd, 353 RAID (Redundant Array of Independent Disks), 7, 398 railways, schema migration on, 496 RAMCloud (in-memory storage), 89 ranking algorithms, 424 RDF (Resource Description Framework), 57 querying with SPARQL, 59 RDMA (Remote Direct Memory Access), 276 read committed isolation level, 234-237 implementing, 236 multi-version concurrency control (MVCC), 239 no dirty reads, 234 no dirty writes, 235 read path (derived data), 509 read repair (leaderless replication), 178 for linearizability, 335 read replicas (see leader-based replication) read skew (transaction isolation), 238, 266 as violation of causality, 340 read-after-write consistency, 163, 524 cross-device, 164 read-modify-write cycle, 243 read-scaling architecture, 161 reads as events, 513 real-time collaborative editing, 170 near-real-time processing, 390 (see also stream processing) publish/subscribe dataflow, 513 response time guarantees, 298 time-of-day clocks, 288 rebalancing partitions, 209-214, 556 (see also partitioning) automatic or manual rebalancing, 213 dynamic partitioning, 212 fixed number of partitions, 210 fixed number of partitions per node, 212 problems with hash mod N, 210 recency guarantee, 324 recommendation engines batch process outputs, 412 batch workflows, 403, 420 iterative processing, 424 statistical and numerical algorithms, 428 records, 399 events in stream processing, 440 recursive common table expressions (SQL), 54 redelivery (messaging), 445 Redis (database) atomic operations, 243 durability, 89 Lua scripting, 255 single-threaded execution, 253 usage example, 4 redundancy hardware components, 7 of derived data, 386 (see also derived data) Reed–Solomon codes (error correction), 398 refactoring, 22 (see also evolvability) regions (partitioning), 199 register (data structure), 325 relational data model, 28-42 comparison to document model, 38-42 graph queries in SQL, 53 in-memory databases with, 89 many-to-one and many-to-many relation‐ ships, 33 multi-object transactions, need for, 231 NoSQL as alternative to, 29 object-relational mismatch, 29 relational algebra and SQL, 42 versus document model convergence of models, 41 data locality, 41 relational databases eventual consistency, 162 history, 28 leader-based replication, 153 logical logs, 160 philosophy compared to Unix, 499, 501 schema changes, 40, 111, 130 statement-based replication, 158 use of B-tree indexes, 80 relationships (see edges) reliability, 6-10, 489 building a reliable system from unreliable components, 276 defined, 6, 22 hardware faults, 7 human errors, 9 importance of, 10 of messaging systems, 442 Index | 581 software errors, 8 Remote Method Invocation (Java RMI), 134 remote procedure calls (RPCs), 134-136 (see also services) based on futures, 135 data encoding and evolution, 136 issues with, 134 using Avro, 126, 135 using Thrift, 135 versus message brokers, 137 repeatable reads (transaction isolation), 242 replicas, 152 replication, 151-193, 556 and durability, 227 chain replication, 155 conflict resolution and, 246 consistency properties, 161-167 consistent prefix reads, 165 monotonic reads, 164 reading your own writes, 162 in distributed filesystems, 398 leaderless, 177-191 detecting concurrent writes, 184-191 limitations of quorum consistency, 181-183, 334 sloppy quorums and hinted handoff, 183 monitoring staleness, 182 multi-leader, 168-177 across multiple datacenters, 168, 335 handling write conflicts, 171-175 replication topologies, 175-177 partitioning and, 147, 200 reasons for using, 145, 151 single-leader, 152-161 failover, 157 implementation of replication logs, 158-161 relation to consensus, 367 setting up new followers, 155 synchronous versus asynchronous, 153-155 state machine replication, 349, 452 using erasure coding, 398 with heterogeneous data systems, 453 replication logs (see logs) reprocessing data, 496, 498 (see also evolvability) from log-based messaging, 451 request routing, 214-216 582 | Index approaches to, 214 parallel query execution, 216 resilient systems, 6 (see also fault tolerance) response time as performance metric for services, 13, 389 guarantees on, 298 latency versus, 14 mean and percentiles, 14 user experience, 15 responsibility and accountability, 535 REST (Representational State Transfer), 133 (see also services) RethinkDB (database) document data model, 31 dynamic partitioning, 212 join support, 34, 42 key-range partitioning, 202 leader-based replication, 153 subscribing to changes, 456 Riak (database) Bitcask storage engine, 72 CRDTs, 174, 191 dotted version vectors, 191 gossip protocol, 216 hash partitioning, 203-204, 211 last-write-wins conflict resolution, 186 leaderless replication, 177 LevelDB storage engine, 78 linearizability, lack of, 335 multi-datacenter support, 184 preventing lost updates across replicas, 246 rebalancing, 213 search feature, 209 secondary indexes, 207 siblings (concurrently written values), 190 sloppy quorums, 184 ring buffers, 450 Ripple (cryptocurrency), 532 rockets, 10, 36, 305 RocksDB (storage engine), 78 leveled compaction, 79 rollbacks (transactions), 222 rolling upgrades, 8, 112 routing (see request routing) row-oriented storage, 96 row-based replication, 160 rowhammer (memory corruption), 529 RPCs (see remote procedure calls) Rubygems (package manager), 428 rules (Datalog), 61 S safety and liveness properties, 308 in consensus algorithms, 366 in transactions, 222 sagas (see compensating transactions) Samza (stream processor), 466, 467 fault tolerance, 479 streaming SQL support, 466 sandboxes, 9 SAP HANA (database), 93 scalability, 10-18, 489 approaches for coping with load, 17 defined, 22 describing load, 11 describing performance, 13 partitioning and, 199 replication and, 161 scaling up versus scaling out, 146 scaling out, 17, 146 (see also shared-nothing architecture) scaling up, 17, 146 scatter/gather approach, querying partitioned databases, 207 SCD (slowly changing dimension), 476 schema-on-read, 39 comparison to evolvable schema, 128 in distributed filesystems, 415 schema-on-write, 39 schemaless databases (see schema-on-read) schemas, 557 Avro, 122-127 reader determining writer’s schema, 125 schema evolution, 123 dynamically generated, 126 evolution of, 496 affecting application code, 111 compatibility checking, 126 in databases, 129-131 in message-passing, 138 in service calls, 136 flexibility in document model, 39 for analytics, 93-95 for JSON and XML, 115 merits of, 127 schema migration on railways, 496 Thrift and Protocol Buffers, 117-121 schema evolution, 120 traditional approach to design, fallacy in, 462 searches building search indexes in batch processes, 411 k-nearest neighbors, 429 on streams, 467 partitioned secondary indexes, 206 secondaries (see leader-based replication) secondary indexes, 85, 557 partitioning, 206-209, 217 document-partitioned, 206 index maintenance, 495 term-partitioned, 208 problems with dual writes, 452, 491 updating, transaction isolation and, 231 secondary sorts, 405 sed (Unix tool), 392 self-describing files, 127 self-joins, 480 self-validating systems, 530 semantic web, 57 semi-synchronous replication, 154 sequence number ordering, 343-348 generators, 294, 344 insufficiency for enforcing constraints, 347 Lamport timestamps, 345 use of timestamps, 291, 295, 345 sequential consistency, 351 serializability, 225, 233, 251-266, 557 linearizability versus, 329 pessimistic versus optimistic concurrency control, 261 serial execution, 252-256 partitioning, 255 using stored procedures, 253, 349 serializable snapshot isolation (SSI), 261-266 detecting stale MVCC reads, 263 detecting writes that affect prior reads, 264 distributed execution, 265, 364 performance of SSI, 265 preventing write skew, 262-265 two-phase locking (2PL), 257-261 index-range locks, 260 performance, 258 Serializable (Java), 113 Index | 583 serialization, 113 (see also encoding) service discovery, 135, 214, 372 using DNS, 216, 372 service level agreements (SLAs), 15 service-oriented architecture (SOA), 132 (see also services) services, 131-136 microservices, 132 causal dependencies across services, 493 loose coupling, 502 relation to batch/stream processors, 389, 508 remote procedure calls (RPCs), 134-136 issues with, 134 similarity to databases, 132 web services, 132, 135 session windows (stream processing), 472 (see also windows) sessionization, 407 sharding (see partitioning) shared mode (locks), 258 shared-disk architecture, 146, 398 shared-memory architecture, 146 shared-nothing architecture, 17, 146-147, 557 (see also replication) distributed filesystems, 398 (see also distributed filesystems) partitioning, 199 use of network, 277 sharks biting undersea cables, 279 counting (example), 46-48 finding (example), 42 website about (example), 44 shredding (in relational model), 38 siblings (concurrent values), 190, 246 (see also conflicts) similarity search edit distance, 88 genome data, 63 k-nearest neighbors, 429 single-leader replication (see leader-based rep‐ lication) single-threaded execution, 243, 252 in batch processing, 406, 421, 426 in stream processing, 448, 463, 522 size-tiered compaction, 79 skew, 557 584 | Index clock skew, 291-294, 334 in transaction isolation read skew, 238, 266 write skew, 246-251, 262-265 (see also write skew) meanings of, 238 unbalanced workload, 201 compensating for, 205 due to celebrities, 205 for time-series data, 203 in batch processing, 407 slaves (see leader-based replication) sliding windows (stream processing), 472 (see also windows) sloppy quorums, 183 (see also quorums) lack of linearizability, 334 slowly changing dimension (data warehouses), 476 smearing (leap seconds adjustments), 290 snapshots (databases) causal consistency, 340 computing derived data, 500 in change data capture, 455 serializable snapshot isolation (SSI), 261-266, 329 setting up a new replica, 156 snapshot isolation and repeatable read, 237-242 implementing with MVCC, 239 indexes and MVCC, 241 visibility rules, 240 synchronized clocks for global snapshots, 294 snowflake schemas, 95 SOAP, 133 (see also services) evolvability, 136 software bugs, 8 maintaining integrity, 529 solid state drives (SSDs) access patterns, 84 detecting corruption, 519, 530 faults in, 227 sequential write throughput, 75 Solr (search server) building indexes in batch processes, 411 document-partitioned indexes, 207 request routing, 216 usage example, 4 use of Lucene, 79 sort (Unix tool), 392, 394, 395 sort-merge joins (MapReduce), 405 Sorted String Tables (see SSTables) sorting sort order in column storage, 99 source of truth (see systems of record) Spanner (database) data locality, 41 snapshot isolation using clocks, 295 TrueTime API, 294 Spark (processing framework), 421-423 bytecode generation, 428 dataflow APIs, 427 fault tolerance, 422 for data warehouses, 93 GraphX API (graph processing), 425 machine learning, 428 query optimizer, 427 Spark Streaming, 466 microbatching, 477 stream processing on top of batch process‐ ing, 495 SPARQL (query language), 59 spatial algorithms, 429 split brain, 158, 557 in consensus algorithms, 352, 367 preventing, 322, 333 using fencing tokens to avoid, 302-304 spreadsheets, dataflow programming capabili‐ ties, 504 SQL (Structured Query Language), 21, 28, 43 advantages and limitations of, 416 distributed query execution, 48 graph queries in, 53 isolation levels standard, issues with, 242 query execution on Hadoop, 416 résumé (example), 30 SQL injection vulnerability, 305 SQL on Hadoop, 93 statement-based replication, 158 stored procedures, 255 SQL Server (database) data warehousing support, 93 distributed transaction support, 361 leader-based replication, 153 preventing lost updates, 245 preventing write skew, 248, 257 read committed isolation, 236 recursive query support, 54 serializable isolation, 257 snapshot isolation support, 239 T-SQL language, 255 XML support, 30 SQLstream (stream analytics), 466 SSDs (see solid state drives) SSTables (storage format), 76-79 advantages over hash indexes, 76 concatenated index, 204 constructing and maintaining, 78 making LSM-Tree from, 78 staleness (old data), 162 cross-channel timing dependencies, 331 in leaderless databases, 178 in multi-version concurrency control, 263 monitoring for, 182 of client state, 512 versus linearizability, 324 versus timeliness, 524 standbys (see leader-based replication) star replication topologies, 175 star schemas, 93-95 similarity to event sourcing, 458 Star Wars analogy (event time versus process‐ ing time), 469 state derived from log of immutable events, 459 deriving current state from the event log, 458 interplay between state changes and appli‐ cation code, 507 maintaining derived state, 495 maintenance by stream processor in streamstream joins, 473 observing derived state, 509-515 rebuilding after stream processor failure, 478 separation of application code and, 505 state machine replication, 349, 452 statement-based replication, 158 statically typed languages analogy to schema-on-write, 40 code generation and, 127 statistical and numerical algorithms, 428 StatsD (metrics aggregator), 442 stdin, stdout, 395, 396 Stellar (cryptocurrency), 532 Index | 585 stock market feeds, 442 STONITH (Shoot The Other Node In The Head), 158 stop-the-world (see garbage collection) storage composing data storage technologies, 499-504 diversity of, in MapReduce, 415 Storage Area Network (SAN), 146, 398 storage engines, 69-104 column-oriented, 95-101 column compression, 97-99 defined, 96 distinction between column families and, 99 Parquet, 96, 131 sort order in, 99-100 writing to, 101 comparing requirements for transaction processing and analytics, 90-96 in-memory storage, 88 durability, 227 row-oriented, 70-90 B-trees, 79-83 comparing B-trees and LSM-trees, 83-85 defined, 96 log-structured, 72-79 stored procedures, 161, 253-255, 557 and total order broadcast, 349 pros and cons of, 255 similarity to stream processors, 505 Storm (stream processor), 466 distributed RPC, 468, 514 Trident state handling, 478 straggler events, 470, 498 stream processing, 464-481, 557 accessing external services within job, 474, 477, 478, 517 combining with batch processing lambda architecture, 497 unifying technologies, 498 comparison to batch processing, 464 complex event processing (CEP), 465 fault tolerance, 476-479 atomic commit, 477 idempotence, 478 microbatching and checkpointing, 477 rebuilding state after a failure, 478 for data integration, 494-498 586 | Index maintaining derived state, 495 maintenance of materialized views, 467 messaging systems (see messaging systems) reasoning about time, 468-472 event time versus processing time, 469, 477, 498 knowing when window is ready, 470 types of windows, 472 relation to databases (see streams) relation to services, 508 search on streams, 467 single-threaded execution, 448, 463 stream analytics, 466 stream joins, 472-476 stream-stream join, 473 stream-table join, 473 table-table join, 474 time-dependence of, 475 streams, 440-451 end-to-end, pushing events to clients, 512 messaging systems (see messaging systems) processing (see stream processing) relation to databases, 451-464 (see also changelogs) API support for change streams, 456 change data capture, 454-457 derivative of state by time, 460 event sourcing, 457-459 keeping systems in sync, 452-453 philosophy of immutable events, 459-464 topics, 440 strict serializability, 329 strong consistency (see linearizability) strong one-copy serializability, 329 subjects, predicates, and objects (in triplestores), 55 subscribers (message streams), 440 (see also consumers) supercomputers, 275 surveillance, 537 (see also privacy) Swagger (service definition format), 133 swapping to disk (see virtual memory) synchronous networks, 285, 557 comparison to asynchronous networks, 284 formal model, 307 synchronous replication, 154, 557 chain replication, 155 conflict detection, 172 system models, 300, 306-310 assumptions in, 528 correctness of algorithms, 308 mapping to the real world, 309 safety and liveness, 308 systems of record, 386, 557 change data capture, 454, 491 treating event log as, 460 systems thinking, 536 T t-digest (algorithm), 16 table-table joins, 474 Tableau (data visualization software), 416 tail (Unix tool), 447 tail vertex (property graphs), 51 Tajo (query engine), 93 Tandem NonStop SQL (database), 200 TCP (Transmission Control Protocol), 277 comparison to circuit switching, 285 comparison to UDP, 283 connection failures, 280 flow control, 282, 441 packet checksums, 306, 519, 529 reliability and duplicate suppression, 517 retransmission timeouts, 284 use for transaction sessions, 229 telemetry (see monitoring) Teradata (database), 93, 200 term-partitioned indexes, 208, 217 termination (consensus), 365 Terrapin (database), 413 Tez (dataflow engine), 421-423 fault tolerance, 422 support by higher-level tools, 427 thrashing (out of memory), 297 threads (concurrency) actor model, 138, 468 (see also message-passing) atomic operations, 223 background threads, 73, 85 execution pauses, 286, 296-298 memory barriers, 338 preemption, 298 single (see single-threaded execution) three-phase commit, 359 Thrift (data format), 117-121 BinaryProtocol, 118 CompactProtocol, 119 field tags and schema evolution, 120 throughput, 13, 390 TIBCO, 137 Enterprise Message Service, 444 StreamBase (stream analytics), 466 time concurrency and, 187 cross-channel timing dependencies, 331 in distributed systems, 287-299 (see also clocks) clock synchronization and accuracy, 289 relying on synchronized clocks, 291-295 process pauses, 295-299 reasoning about, in stream processors, 468-472 event time versus processing time, 469, 477, 498 knowing when window is ready, 470 timestamp of events, 471 types of windows, 472 system models for distributed systems, 307 time-dependence in stream joins, 475 time-of-day clocks, 288 timeliness, 524 coordination-avoiding data systems, 528 correctness of dataflow systems, 525 timeouts, 279, 557 dynamic configuration of, 284 for failover, 158 length of, 281 timestamps, 343 assigning to events in stream processing, 471 for read-after-write consistency, 163 for transaction ordering, 295 insufficiency for enforcing constraints, 347 key range partitioning by, 203 Lamport, 345 logical, 494 ordering events, 291, 345 Titan (database), 50 tombstones, 74, 191, 456 topics (messaging), 137, 440 total order, 341, 557 limits of, 493 sequence numbers or timestamps, 344 total order broadcast, 348-352, 493, 522 consensus algorithms and, 366-368 Index | 587 implementation in ZooKeeper and etcd, 370 implementing with linearizable storage, 351 using, 349 using to implement linearizable storage, 350 tracking behavioral data, 536 (see also privacy) transaction coordinator (see coordinator) transaction manager (see coordinator) transaction processing, 28, 90-95 comparison to analytics, 91 comparison to data warehousing, 93 transactions, 221-267, 558 ACID properties of, 223 atomicity, 223 consistency, 224 durability, 226 isolation, 225 compensating (see compensating transac‐ tions) concept of, 222 distributed transactions, 352-364 avoiding, 492, 502, 521-528 failure amplification, 364, 495 in doubt/uncertain status, 358, 362 two-phase commit, 354-359 use of, 360-361 XA transactions, 361-364 OLTP versus analytics queries, 411 purpose of, 222 serializability, 251-266 actual serial execution, 252-256 pessimistic versus optimistic concur‐ rency control, 261 serializable snapshot isolation (SSI), 261-266 two-phase locking (2PL), 257-261 single-object and multi-object, 228-232 handling errors and aborts, 231 need for multi-object transactions, 231 single-object writes, 230 snapshot isolation (see snapshots) weak isolation levels, 233-251 preventing lost updates, 242-246 read committed, 234-238 transitive closure (graph algorithm), 424 trie (data structure), 88 triggers (databases), 161, 441 implementing change data capture, 455 implementing replication, 161 588 | Index triple-stores, 55-59 SPARQL query language, 59 tumbling windows (stream processing), 472 (see also windows) in microbatching, 477 tuple spaces (programming model), 507 Turtle (RDF data format), 56 Twitter constructing home timelines (example), 11, 462, 474, 511 DistributedLog (event log), 448 Finagle (RPC framework), 135 Snowflake (sequence number generator), 294 Summingbird (processing library), 497 two-phase commit (2PC), 353, 355-359, 558 confusion with two-phase locking, 356 coordinator failure, 358 coordinator recovery, 363 how it works, 357 issues in practice, 363 performance cost, 360 transactions holding locks, 362 two-phase locking (2PL), 257-261, 329, 558 confusion with two-phase commit, 356 index-range locks, 260 performance of, 258 type checking, dynamic versus static, 40 U UDP (User Datagram Protocol) comparison to TCP, 283 multicast, 442 unbounded datasets, 439, 558 (see also streams) unbounded delays, 558 in networks, 282 process pauses, 296 unbundling databases, 499-515 composing data storage technologies, 499-504 federation versus unbundling, 501 need for high-level language, 503 designing applications around dataflow, 504-509 observing derived state, 509-515 materialized views and caching, 510 multi-partition data processing, 514 pushing state changes to clients, 512 uncertain (transaction status) (see in doubt) uniform consensus, 365 (see also consensus) uniform interfaces, 395 union type (in Avro), 125 uniq (Unix tool), 392 uniqueness constraints asynchronously checked, 526 requiring consensus, 521 requiring linearizability, 330 uniqueness in log-based messaging, 522 Unix philosophy, 394-397 command-line batch processing, 391-394 Unix pipes versus dataflow engines, 423 comparison to Hadoop, 413-414 comparison to relational databases, 499, 501 comparison to stream processing, 464 composability and uniform interfaces, 395 loose coupling, 396 pipes, 394 relation to Hadoop, 499 UPDATE statement (SQL), 40 updates preventing lost updates, 242-246 atomic write operations, 243 automatically detecting lost updates, 245 compare-and-set operations, 245 conflict resolution and replication, 246 using explicit locking, 244 preventing write skew, 246-251 V validity (consensus), 365 vBuckets (partitioning), 199 vector clocks, 191 (see also version vectors) vectorized processing, 99, 428 verification, 528-533 avoiding blind trust, 530 culture of, 530 designing for auditability, 531 end-to-end integrity checks, 531 tools for auditable data systems, 532 version control systems, reliance on immutable data, 463 version vectors, 177, 191 capturing causal dependencies, 343 versus vector clocks, 191 Vertica (database), 93 handling writes, 101 replicas using different sort orders, 100 vertical scaling (see scaling up) vertices (in graphs), 49 property graph model, 50 Viewstamped Replication (consensus algo‐ rithm), 366 view number, 368 virtual machines, 146 (see also cloud computing) context switches, 297 network performance, 282 noisy neighbors, 284 reliability in cloud services, 8 virtualized clocks in, 290 virtual memory process pauses due to page faults, 14, 297 versus memory management by databases, 89 VisiCalc (spreadsheets), 504 vnodes (partitioning), 199 Voice over IP (VoIP), 283 Voldemort (database) building read-only stores in batch processes, 413 hash partitioning, 203-204, 211 leaderless replication, 177 multi-datacenter support, 184 rebalancing, 213 reliance on read repair, 179 sloppy quorums, 184 VoltDB (database) cross-partition serializability, 256 deterministic stored procedures, 255 in-memory storage, 89 output streams, 456 secondary indexes, 207 serial execution of transactions, 253 statement-based replication, 159, 479 transactions in stream processing, 477 W WAL (write-ahead log), 82 web services (see services) Web Services Description Language (WSDL), 133 webhooks, 443 webMethods (messaging), 137 WebSocket (protocol), 512 Index | 589 windows (stream processing), 466, 468-472 infinite windows for changelogs, 467, 474 knowing when all events have arrived, 470 stream joins within a window, 473 types of windows, 472 winners (conflict resolution), 173 WITH RECURSIVE syntax (SQL), 54 workflows (MapReduce), 402 outputs, 411-414 key-value stores, 412 search indexes, 411 with map-side joins, 410 working set, 393 write amplification, 84 write path (derived data), 509 write skew (transaction isolation), 246-251 characterizing, 246-251, 262 examples of, 247, 249 materializing conflicts, 251 occurrence in practice, 529 phantoms, 250 preventing in snapshot isolation, 262-265 in two-phase locking, 259-261 options for, 248 write-ahead log (WAL), 82, 159 writes (database) atomic write operations, 243 detecting writes affecting prior reads, 264 preventing dirty writes with read commit‐ ted, 235 WS-* framework, 133 (see also services) WS-AtomicTransaction (2PC), 355 590 | Index X XA transactions, 355, 361-364 heuristic decisions, 363 limitations of, 363 xargs (Unix tool), 392, 396 XML binary variants, 115 encoding RDF data, 57 for application data, issues with, 114 in relational databases, 30, 41 XSL/XPath, 45 Y Yahoo!


pages: 1,237 words: 227,370

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems by Martin Kleppmann

active measures, Amazon Web Services, bitcoin, blockchain, business intelligence, business process, c2.com, cloud computing, collaborative editing, commoditize, conceptual framework, cryptocurrency, database schema, DevOps, distributed ledger, Donald Knuth, Edward Snowden, Ethereum, ethereum blockchain, fault tolerance, finite state, Flash crash, full text search, general-purpose programming language, informal economy, information retrieval, Infrastructure as a Service, Internet of things, iterative process, John von Neumann, Kubernetes, loose coupling, Marc Andreessen, microservices, natural language processing, Network effects, packet switching, peer-to-peer, performance metric, place-making, premature optimization, recommendation engine, Richard Feynman, self-driving car, semantic web, Shoshana Zuboff, social graph, social web, software as a service, software is eating the world, sorting algorithm, source of truth, SPARQL, speech recognition, statistical model, undersea cable, web application, WebSocket, wikimedia commons

The relational model can handle simple cases of many-to-many relationships, but as the connections within your data become more complex, it becomes more natural to start modeling your data as a graph. A graph consists of two kinds of objects: vertices (also known as nodes or entities) and edges (also known as relationships or arcs). Many kinds of data can be modeled as a graph. Typical examples include: Social graphs Vertices are people, and edges indicate which people know each other. The web graph Vertices are web pages, and edges indicate HTML links to other pages. Road or rail networks Vertices are junctions, and edges represent the roads or railway lines between them. Well-known algorithms can operate on these graphs: for example, car navigation systems search for the shortest path between two points in a road network, and PageRank can be used on the web graph to determine the popularity of a web page and thus its ranking in search results.

Hellerstein: “The Declarative Imperative: Experiences and Conjectures in Distributed Logic,” Electrical Engineering and Computer Sciences, University of California at Berkeley, Tech report UCB/EECS-2010-90, June 2010. [33] Jeffrey Dean and Sanjay Ghemawat: “MapReduce: Simplified Data Processing on Large Clusters,” at 6th USENIX Symposium on Operating System Design and Implementation (OSDI), December 2004. [34] Craig Kerstiens: “JavaScript in Your Postgres,” blog.heroku.com, June 5, 2013. [35] Nathan Bronson, Zach Amsden, George Cabrera, et al.: “TAO: Facebook’s Distributed Data Store for the Social Graph,” at USENIX Annual Technical Conference (USENIX ATC), June 2013. [36] “Apache TinkerPop3.2.3 Documentation,” tinkerpop.apache.org, October 2016. [37] “The Neo4j Manual v2.0.0,” Neo Technology, 2013. [38] Emil Eifrem: Twitter correspondence, January 3, 2014. [39] David Beckett and Tim Berners-Lee: “Turtle – Terse RDF Triple Language,” W3C Team Submission, March 28, 2011. [40] “Datomic Development Resources,” Metadata Partners, LLC, 2013

EventSource (browser API), Pushing state changes to clients eventual consistency, Replication, Problems with Replication Lag, Safety and liveness, Consistency Guarantees(see also conflicts) and perpetual inconsistency, Timeliness and Integrity evolvability, Evolvability: Making Change Easy, Encoding and Evolutioncalling services, Data encoding and evolution for RPC graph-structured data, Property Graphs of databases, Schema flexibility in the document model, Dataflow Through Databases-Archival storage, Deriving several views from the same event log, Reprocessing data for application evolution of message-passing, Distributed actor frameworks reprocessing data, Reprocessing data for application evolution, Unifying batch and stream processing schema evolution in Avro, The writer’s schema and the reader’s schema schema evolution in Thrift and Protocol Buffers, Field tags and schema evolution schema-on-read, Schema flexibility in the document model, Encoding and Evolution, The Merits of Schemas exactly-once semantics, Exactly-once message processing, Fault Tolerance, Exactly-once execution of an operationparity with batch processors, Unifying batch and stream processing preservation of integrity, Correctness of dataflow systems exclusive mode (locks), Implementation of two-phase locking eXtended Architecture transactions (see XA transactions) extract-transform-load (see ETL) F FacebookPresto (query engine), The divergence between OLTP databases and data warehouses React, Flux, and Redux (user interface libraries), End-to-end event streams social graphs, Graph-Like Data Models Wormhole (change data capture), Implementing change data capture fact tables, Stars and Snowflakes: Schemas for Analytics failover, Leader failure: Failover, Glossary(see also leader-based replication) in leaderless replication, absence of, Writing to the Database When a Node Is Down leader election, The leader and the lock, Total Order Broadcast, Distributed Transactions and Consensus potential problems, Leader failure: Failover failuresamplification by distributed transactions, Limitations of distributed transactions, Maintaining derived state failure detection, Detecting Faultsautomatic rebalancing causing cascading failures, Operations: Automatic or Manual Rebalancing perfect failure detectors, Three-phase commit timeouts and unbounded delays, Timeouts and Unbounded Delays, Network congestion and queueing using ZooKeeper, Membership and Coordination Services faults versus, Reliability partial failures in distributed systems, Faults and Partial Failures-Cloud Computing and Supercomputing, Summary fan-out (messaging systems), Describing Load, Multiple consumers fault tolerance, Reliability-How Important Is Reliability?


pages: 598 words: 134,339

Data and Goliath: The Hidden Battles to Collect Your Data and Control Your World by Bruce Schneier

23andMe, Airbnb, airport security, AltaVista, Anne Wojcicki, augmented reality, Benjamin Mako Hill, Black Swan, Boris Johnson, Brewster Kahle, Brian Krebs, call centre, Cass Sunstein, Chelsea Manning, citizen journalism, cloud computing, congestion charging, disintermediation, drone strike, Edward Snowden, experimental subject, failed state, fault tolerance, Ferguson, Missouri, Filter Bubble, Firefox, friendly fire, Google Chrome, Google Glasses, hindsight bias, informal economy, Internet Archive, Internet of things, Jacob Appelbaum, Jaron Lanier, John Markoff, Julian Assange, Kevin Kelly, license plate recognition, lifelogging, linked data, Lyft, Mark Zuckerberg, moral panic, Nash equilibrium, Nate Silver, national security letter, Network effects, Occupy movement, Panopticon Jeremy Bentham, payday loans, pre–internet, price discrimination, profit motive, race to the bottom, RAND corporation, recommendation engine, RFID, Ross Ulbricht, self-driving car, Shoshana Zuboff, Silicon Valley, Skype, smart cities, smart grid, Snapchat, social graph, software as a service, South China Sea, stealth mode startup, Steven Levy, Stuxnet, TaskRabbit, telemarketer, Tim Cook: Apple, transaction costs, Uber and Lyft, uber lyft, undersea cable, urban planning, WikiLeaks, zero day

Amit Agarwal (2013), “Sleeping Time,” Digital Inspiration, http://sleepingtime.org. Your buddy lists and address books: Two studies of Facebook social graphs show how easy it is to predict these and other personal traits. Carter Jernigan and Behram R. T. Mistree (5 Oct 2009), “Gaydar: Facebook friendships expose sexual orientation,” First Monday 14, http://firstmonday.org/article/view/2611/2302. Michal Kosinski, David Stillwell, and Thore Graepel (11 Mar 2013), “Private traits and attributes are predictable from digital records of human behavior,” Proceedings of the National Academy of Sciences of the United States of America (Early Edition), http://www.pnas.org/content/early/2013/03/06/1218772110.abstract. Your e-mail headers reveal: The MIT Media Lab tool Immersion builds a social graph from your e-mail metadata. MIT Media Lab (2013), “Immersion: A people-centric view of your email life,” https://immersion.media.mit.edu.


pages: 554 words: 149,489

The Content Trap: A Strategist's Guide to Digital Change by Bharat Anand

Airbnb, Benjamin Mako Hill, Bernie Sanders, Clayton Christensen, cloud computing, commoditize, correlation does not imply causation, creative destruction, crowdsourcing, death of newspapers, disruptive innovation, Donald Trump, Google Glasses, Google X / Alphabet X, information asymmetry, Internet of things, inventory management, Jean Tirole, Jeff Bezos, John Markoff, Just-in-time delivery, Khan Academy, Kickstarter, late fees, Mark Zuckerberg, market design, Minecraft, multi-sided market, Network effects, post-work, price discrimination, publish or perish, QR code, recommendation engine, ride hailing / ride sharing, selection bias, self-driving car, shareholder value, Shenzhen was a fishing village, Silicon Valley, Silicon Valley startup, Skype, social graph, social web, special economic zone, Stephen Hawking, Steve Jobs, Steven Levy, Thomas L Friedman, transaction costs, two-sided market, ubercab, WikiLeaks, winner-take-all economy, zero-sum game

It cross-promoted its games on its IM platform—a user could launch the game directly from her IM screen rather than be directed to a separate site to play. It bundled its services effectively—its chat service could be used within a game, and a gamer could import her avatars. And it transferred the strength of its network effect in one product to others—with the click of a button, a user could import her social graph from QQ into a Tencent game in order to play with her friends. Tencent was doing something many companies that compete in winner-take-all markets struggle with: It successfully created connections across different products—IM, games, microblogs—where each relied on connecting users. In effect, it shifted its strength from just one network to a portfolio of connected networks. To monetize these advantages, Tencent turned again to price discrimination.

The potential for improving ad effectiveness seemed limitless. So, three sets of predictions about the impact of new features on advertising markets were clear. Targeting, measurement, and interactivity would radically improve Internet ads. Fast-forwarding through commercials would be the bane of TV advertisers and broadcast networks. And one-to-one, real-time targeting based on demographics, social graphs, and behavioral information promised unlimited advertising potential. These predictions all originated with experts; all were backed by data and charts. And all were wrong. THREE PUZZLES ABOUT ADVERTISING Twenty years after the early predictions about online advertising, its promise is still relatively obscure. The dominant ad format for most Web publishers is not much different than in 1994, when banner ads of varying size surrounded the screen text.


pages: 202 words: 59,883

Age of Context: Mobile, Sensors, Data and the Future of Privacy by Robert Scoble, Shel Israel

Albert Einstein, Apple II, augmented reality, call centre, Chelsea Manning, cloud computing, connected car, Edward Snowden, Edward Thorp, Elon Musk, factory automation, Filter Bubble, G4S, Google Earth, Google Glasses, Internet of things, job automation, John Markoff, Kickstarter, lifelogging, Marc Andreessen, Mars Rover, Menlo Park, Metcalfe’s law, New Urbanism, PageRank, pattern recognition, RFID, ride hailing / ride sharing, Robert Metcalfe, Saturday Night Live, self-driving car, sensor fusion, Silicon Valley, Skype, smart grid, social graph, speech recognition, Steve Jobs, Steve Wozniak, Steven Levy, Tesla Model S, Tim Cook: Apple, ubercab, urban planning, Zipcar

It could tell by your inquiry pattern that when you searched for “park in San Francisco” you wanted greenery and not some place to leave your car. Essentially, Google reversed the data equation. Instead of you learning to speak in a machine language, Google started to make machines recognize your natural language. This has made all the difference in the world. When Facebook rapidly evolved into the world’s biggest site, it made a series of forward leaps related to searching. First, it came up with the social graph, which examines relationships between people instead of data. It extrapolated relevant data by examining graphical representations rather than strings of text. Next, Facebook created a Graph API (Application Programming Interface) that enabled third-party developers to connect and share data with the Facebook platform using common verbs such as “read,” “listen to,” “like,” “comment on” and so forth.


pages: 232 words: 63,846

Traction: How Any Startup Can Achieve Explosive Customer Growth by Gabriel Weinberg, Justin Mares

Airbnb, Firefox, if you build it, they will come, jimmy wales, Justin.tv, Lean Startup, Marc Andreessen, Mark Zuckerberg, Network effects, Paul Graham, Peter Thiel, side project, Skype, Snapchat, social graph, software as a service, the payments system, Uber for X, web application, working poor, Y Combinator

(He targeted her by her alma mater, zip code, and interest affinities, using a picture of their son to see how long it would take for her to notice. Not very long.) The platform also allows you to reach the larger network of people connected through your fans on Facebook. As Nikhil said: When you buy a Facebook ad, you’re buying more than just a targeted fan; you’re buying the opportunity to access that fan’s social graph. With the proper incentives, fans will share and recommend your brand to their connections. StumbleUpon—With more than 25 million “stumblers,” StumbleUpon has a large potential user base looking for new and engaging content. An interesting feature about this site is that ads don’t surround the content on StumbleUpon—they are part of the content. When people hit the “Stumble” button, they will be directed to a paid piece of content that looks just like any other site on the network.


pages: 145 words: 40,897

Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps by Gabe Zichermann, Christopher Cunningham

airport security, future of work, game design, lateral thinking, minimum viable product, pattern recognition, Ruby on Rails, social graph, social web, urban planning, web application

Even when the number scale is completely meaningless or opaque, the player still feels that four million points is a lot, so it’s probably difficult to attain (unless it’s 4,000,000 Vietnamese Dong, the equivalent of about $200 U.S. dollars at press time—see the sidebar Currency Denominations earlier in this chapter). Leaderboard Types There are two kinds of leaderboards largely used today. The no-disincentive leaderboard The leaderboard of today has seen some radical redesign since the heyday of pinball machines and quarter arcades. In the era of Facebook and the social graph, leaderboards are mostly tools for creating social incentive, rather than disincentive. They accomplish this simply by taking the player and putting him right in the middle. It doesn’t matter where he falls in ranking order—whether he is #81 or #200,000—the player will see himself right in the middle of the leaderboard. Below him, he will see friends who are on his tail, and above him he will see exactly how close he is to the next best score.


The Data Journalism Handbook by Jonathan Gray, Lucy Chambers, Liliana Bounegru

Amazon Web Services, barriers to entry, bioinformatics, business intelligence, carbon footprint, citizen journalism, correlation does not imply causation, crowdsourcing, David Heinemeier Hansson, eurozone crisis, Firefox, Florence Nightingale: pie chart, game design, Google Earth, Hans Rosling, information asymmetry, Internet Archive, John Snow's cholera map, Julian Assange, linked data, moral hazard, MVC pattern, New Journalism, openstreetmap, Ronald Reagan, Ruby on Rails, Silicon Valley, social graph, SPARQL, text mining, web application, WikiLeaks

The position of the nodes is then calculated by more or less complex graph layout algorithms which allow us to immediately see the structure within the network. The trick of graph visualization in general is to find a proper way to model the network itself. Not all datasets already include relations, and even if they do, it might not be the most interesting aspect to look at. Sometimes it’s up to the journalist to define edges between nodes. A perfect example of this is the U.S. Senate Social Graph, whose edges connect senators that voted the same in more than 65% of the votes. Analyze and interpret what you see Once you have visualized your data, the next step is to learn something from the picture you created. You could ask yourself: What can I see in this image? Is it what I expected? Are there any interesting patterns? What does this mean in the context of the data? Sometimes you might end up with a visualization that, in spite of its beauty, might seem to tell you nothing of interest about your data.


pages: 272 words: 64,626

Eat People: And Other Unapologetic Rules for Game-Changing Entrepreneurs by Andy Kessler

23andMe, Andy Kessler, bank run, barriers to entry, Berlin Wall, Bob Noyce, British Empire, business cycle, business process, California gold rush, carbon footprint, Cass Sunstein, cloud computing, collateralized debt obligation, collective bargaining, commoditize, computer age, creative destruction, disintermediation, Douglas Engelbart, Eugene Fama: efficient market hypothesis, fiat currency, Firefox, Fractional reserve banking, George Gilder, Gordon Gekko, greed is good, income inequality, invisible hand, James Watt: steam engine, Jeff Bezos, job automation, Joseph Schumpeter, Kickstarter, knowledge economy, knowledge worker, libertarian paternalism, low skilled workers, Mark Zuckerberg, McMansion, Netflix Prize, packet switching, personalized medicine, pets.com, prediction markets, pre–internet, profit motive, race to the bottom, Richard Thaler, risk tolerance, risk-adjusted returns, Silicon Valley, six sigma, Skype, social graph, Steve Jobs, The Wealth of Nations by Adam Smith, transcontinental railway, transfer pricing, wealth creators, Yogi Berra

No matter.) Gruesome, the metaphor anyway, but real. It just happens slowly enough that it’s more like Chinese water torture than a flood. But how to prepare for all this change? College, when it’s not learning you how to learn, prepares students for the very jobs that Free Radicals are busy getting rid of! A Free Radical had best prepare for a world of networks and mobility and attention mining and social graphs or whatever nom du jour the social commentators come up with, and the best way is to find a set of Servers, and then eat them by rendering them obsolete. RULE #8 Markets Make Better Decisions Than Managers YOU WOULD THINK THAT, AFTER THE CREDIT CRISIS OF 2008, no one would trust markets ever again. But you should. Free Radicals embrace markets and learn to trust them more than their own instincts.


pages: 244 words: 66,977

Subscribed: Why the Subscription Model Will Be Your Company's Future - and What to Do About It by Tien Tzuo, Gabe Weisert

3D printing, Airbnb, airport security, Amazon Web Services, augmented reality, autonomous vehicles, blockchain, Build a better mousetrap, business cycle, business intelligence, business process, call centre, cloud computing, cognitive dissonance, connected car, death of newspapers, digital twin, double entry bookkeeping, Elon Musk, factory automation, fiat currency, Internet of things, inventory management, iterative process, Jeff Bezos, Kevin Kelly, Lean Startup, Lyft, manufacturing employment, minimum viable product, natural language processing, Network effects, Nicholas Carr, nuclear winter, pets.com, profit maximization, race to the bottom, ride hailing / ride sharing, Sand Hill Road, shareholder value, Silicon Valley, skunkworks, smart meter, social graph, software as a service, spice trade, Steve Ballmer, Steve Jobs, subscription business, Tim Cook: Apple, transport as a service, Uber and Lyft, uber lyft, Y2K, Zipcar

You have to speak their language, and only a segmented sales force can do so effectively. GO INTERNATIONAL Companies typically wait too long to go international. It’s a legacy of old thinking. The old way is anchored on geographical and political boundaries. But the world is different now; it’s really based on language. The reason is pretty simple—the language you use to engage with the internet and your social graph dictates the kinds of results you’re going to receive. There are no customs checks when you visit an IP address in Europe. If you’re a British newspaper like the Daily Mail that specializes in celebrity coverage, it shouldn’t be a surprise that 40 percent of your audience comes from the United States. Likewise, if you’re selling NBA gear in the United States—guess what? You’re also probably doing business in the Commonwealth countries.


Big Data at Work: Dispelling the Myths, Uncovering the Opportunities by Thomas H. Davenport

Automated Insights, autonomous vehicles, bioinformatics, business intelligence, business process, call centre, chief data officer, cloud computing, commoditize, data acquisition, disruptive innovation, Edward Snowden, Erik Brynjolfsson, intermodal, Internet of things, Jeff Bezos, knowledge worker, lifelogging, Mark Zuckerberg, move fast and break things, move fast and break things, Narrative Science, natural language processing, Netflix Prize, New Journalism, recommendation engine, RFID, self-driving car, sentiment analysis, Silicon Valley, smart grid, smart meter, social graph, sorting algorithm, statistical model, Tesla Model S, text mining, Thomas Davenport

As a group of researchers wrote about LinkedIn: LinkedIn contributes to the Voldemort distributed ­storage ­system and more than 10 more open-source projects. “We ­contribute, Chapter_07.indd 160 03/12/13 12:42 PM What You Can Learn from Start-Ups and Online Firms   161 they contribute and the code moves ­forward,” says David Henke, senior vice president of operations at LinkedIn.2 Another LinkedIn data scientist told me: We are working on some database enhancements to a social graph database. They will be open source when we’re done. There are some IP [intellectual property] considerations, but overall LinkedIn believes in building on the open-source ­framework since we benefit from it. This lesson could be taken too far, of course. All of the c­ ompanies I’ve mentioned keep some big data assets to themselves. However, given all the benefits that every firm has received from open-source software, virtually every firm should try to give something back.


pages: 666 words: 181,495

In the Plex: How Google Thinks, Works, and Shapes Our Lives by Steven Levy

23andMe, AltaVista, Anne Wojcicki, Apple's 1984 Super Bowl advert, autonomous vehicles, book scanning, Brewster Kahle, Burning Man, business process, clean water, cloud computing, crowdsourcing, Dean Kamen, discounted cash flows, don't be evil, Donald Knuth, Douglas Engelbart, Douglas Engelbart, El Camino Real, fault tolerance, Firefox, Gerard Salton, Gerard Salton, Google bus, Google Chrome, Google Earth, Googley, HyperCard, hypertext link, IBM and the Holocaust, informal economy, information retrieval, Internet Archive, Jeff Bezos, John Markoff, Kevin Kelly, Kickstarter, Mark Zuckerberg, Menlo Park, one-China policy, optical character recognition, PageRank, Paul Buchheit, Potemkin village, prediction markets, recommendation engine, risk tolerance, Rubik’s Cube, Sand Hill Road, Saturday Night Live, search inside the book, second-price auction, selection bias, Silicon Valley, skunkworks, Skype, slashdot, social graph, social software, social web, spectrum auction, speech recognition, statistical model, Steve Ballmer, Steve Jobs, Steven Levy, Ted Nelson, telemarketer, trade route, traveling salesman, turn-by-turn navigation, undersea cable, Vannevar Bush, web application, WikiLeaks, Y Combinator

Meanwhile, Google would develop its own location-based service, Latitude. By then, there were a number of location-based start-ups, all of which owed something to Dodgeball. One of the hottest was called Foursquare. Its cofounder was Dennis Crowley. Google had a built-in disadvantage in the social networking sweepstakes. It was happy to gather information about the intricate web of personal and professional connections known as the “social graph” (a term favored by Facebook’s Mark Zuckerberg) and integrate that data as signals in its search engine. But the basic premise of social networking—that a personal recommendation from a friend was more valuable than all of human wisdom, as represented by Google Search—was viewed with horror at Google. Page and Brin had started Google on the premise that the algorithm would provide the only answer.

., 140 Playboy, 153–54, 155 pornography, blocking, 54, 97, 108, 173, 174 Postini, 241 Pregibon, Daryl, 118–19 Premium Sunset, 109, 112–13, 115 privacy: and Book Settlement, 363 and browsers, 204–12, 336–37 and email, 170–78, 211–12, 378 and Google’s policies, 10, 11, 145, 173–75, 333–35, 337–40 and Google Street View, 340–43 and government fishing expeditions, 173 and interest-based ads, 263, 334–36 and security breach, 268 and social networking, 378–79, 383 and surveillance, 343 Privacy International, 176 products: beta versions of, 171 “dogfooding,” 216 Google neglect of, 372, 373–74, 376, 381 in GPS meetings, 6, 135, 171 machine-driven, 207 marketing themselves, 77, 372 speed required in, 186 Project Database (PDB), 164 property law, 6, 360 Python, 18, 37 Qiheng, Hu, 277 Queiroz, Mario, 230 Rainert, Alex, 373, 374 Rajaram, Gokul, 106 Rakowski, Brian, 161 Randall, Stephen, 153 RankDex, 27 Rasmussen, Lars, 379 Red Hat, 78 Reese, Jim, 181–84, 187, 195, 196, 198 Reeves, Scott, 153 Rekhi, Manu, 373 Reyes, George, 70, 148 Richards, Michael, 251 robotics, 246, 351, 385 Romanos, Jack, 356 Rosenberg, Jonathan, 159–60, 281 Rosenstein, Justin, 369 Rosing, Wayne, 44, 55, 82, 155, 158–59, 186, 194, 271 Rubin, Andy, 135, 213–18, 220, 221–22, 226, 227–30, 232 Rubin, Robert, 148 Rubinson, Barry, 20–21 Rubinstein, Jon, 221 Sacca, Chris, 188–94 Salah, George, 84, 128, 129, 132–33, 166 Salinger Group, The, 190–91 Salton, Gerard, 20, 24, 40 Samsung, 214, 217 Samuelson, Pamela, 362, 365 Sandberg, Sheryl, 175, 257 and advertising, 90, 97, 98, 99, 107 and customer support, 231 and Facebook, 259, 370 Sanlu Group, 297–98 Santana, Carlos, 238 Schillace, Sam, 201–3 Schmidt, Eric, 107, 193 and advertising, 93, 95–96, 99, 104, 108, 110, 112, 114, 115, 117, 118, 337 and antitrust issues, 345 and Apple, 218, 220, 236–37 and applications, 207, 240, 242 and Book Search, 350, 351, 364 and China, 267, 277, 279, 283, 288–89, 305, 310–11, 313, 386 and cloud computing, 201 and financial issues, 69–71, 252, 260, 376, 383 and Google culture, 129, 135, 136, 364 and Google motto, 145 and growth, 165, 271 and IPO, 147–48, 152, 154, 155–57 on lawsuits, 328–29 and management, 4, 80–83, 110, 158–60, 165, 166, 242, 254, 255, 273, 386, 387 and Obama, 316–17, 319, 321, 346 and privacy, 175, 178, 383 and public image, 328 and smart phones, 216, 217, 224, 236 and social networking, 372 and taxes, 90 and Yahoo, 344, 345 and YouTube, 248–49, 260, 265 Schrage, Elliot, 285–87 Schroeder, Pat, 361 search: decoding the intent of, 59 failed, 60 freshness in, 42 Google as synonymous with, 40, 41, 42, 381 mobile, 217 organic results of, 85 in people’s brains, 67–68 real-time, 376 sanctity of, 275 statelessness of, 116, 332 verticals, 58 see also web searches search engine optimization (SEO), 55–56 search engines, 19 bigram breakage in, 51 business model for, 34 file systems for, 43–44 and hypertext link, 27, 37 information retrieval via, 27 and licensing fees, 77, 84, 95, 261 name detection in, 50–52 and relevance, 48–49, 52 signals to, 22 ultimate, 35 upgrades of, 49, 61–62 Search Engine Watch, 102 SearchKing, 56 SEC regulations, 149, 150–51, 152, 154, 156 Semel, Terry, 98 Sengupta, Caesar, 210 Seti, 65–67 Shah, Sonal, 321 Shapiro, Carl, 117 Shazeer, Noam, 100–102 Sheff, David, 153 Sherman Antitrust Act, 345 Shriram, Ram, 34, 72, 74, 79 Siao, Qiang, 277 Sidekick, 213, 226 signals, 21–22, 49, 59, 376 Silicon Graphics (SGI), 131–32 Silverstein, Craig, 13, 34, 35, 36, 43, 78, 125, 129, 139 Sina, 278, 288, 302 Singh, Sanjeev, 169–70 Singhal, Amit, 24, 40–41, 48–52, 54, 55, 58 Siroker, Dan, 319–21 skunkworks, 380–81 Skype, 233, 234–36, 322, 325 Slashdot, 167 Slim, Carlos, 166 SMART (Salton’s Magical Retriever of Text), 20 smart phones, 214–16, 217–22 accelerometers on, 226–28 carrier contracts for, 230, 231, 236 customer support for, 230–31, 232 direct to consumer, 230, 232 Nexus One, 230, 231–32 Smith, Adam, 360 Smith, Bradford, 333 Smith, Christopher, 284–86 Smith, Megan, 141, 158, 184, 258, 318, 350, 355–56 social graph, 374 social networking, 369–83 Sogou, 300 Sohu, 278, 300 Sony, 251, 264 Sooner (mobile operating system), 217, 220 Southworth, Lucinda, 254 spam, 53–57, 92, 241 Spector, Alfred, 65, 66–67 speech recognition, 65, 67 spell checking, 48 Spencer, Graham, 20, 28, 201, 375 spiders, 18, 19 Stanford University: and BackRub, 29–30 and Book Search, 357 Brin in, 13–14, 16, 17, 28, 29, 34 computer science program at, 14, 23, 27, 32 Digital Library Project, 16, 17 and Google, 29, 31, 32–33, 34 and MIDAS, 16 Page in, 12–13, 14, 16–17, 28, 29, 34 and Silicon Valley, 27–28 Stanley (robot), 246, 385 Stanton, Katie, 318, 321, 322, 323–25, 327 Stanton, Louis L., 251 State Department, U.S., 324–25 Steremberg, Alan, 18, 29 Stewart, Jon, 384 Stewart, Margaret, 207 Stricker, Gabriel, 186 Sullivan, Danny, 102 Sullivan, Stacy, 134, 140, 141, 143–44, 158–59 Summers, Larry, 90 Sun Microsystems, 28, 70 Swetland, Brian, 226, 228 Taco Town, 377 Tan, Chade-Meng, 135–36 Tang, Diane, 118 Taylor, Bret, 259, 370 Teetzel, Erik, 184, 197 Tele Atlas, 341 Tesla, Nikola, 13, 32, 106 Thompson, Ken, 241 3M, 124 Thrun, Sebastian, 246, 385–86 T-Mobile, 226, 227, 230 Tseng, Erick, 217, 227 Twentieth Century Fox, 249 Twitter, 309, 322, 327, 374–77, 387 Uline, 112 Universal Music Group, 261 Universal Search, 58–60, 294, 357 University of Michigan, 352–54, 357 UNIX, 54, 80 Upson, Linus, 210, 211–12 Upstartle, 201 Urchin Software, 114 users: in A/B tests, 61 data amassed about, 45–48, 59, 84, 144, 173–74, 180, 185, 334–37 feedback from, 65 focus on, 5, 77, 92 increasing numbers of, 72 predictive clues from, 66 and security breach, 268, 269 U.S.


pages: 611 words: 188,732

Valley of Genius: The Uncensored History of Silicon Valley (As Told by the Hackers, Founders, and Freaks Who Made It Boom) by Adam Fisher

Airbnb, Albert Einstein, AltaVista, Apple II, Apple's 1984 Super Bowl advert, augmented reality, autonomous vehicles, Bob Noyce, Brownian motion, Buckminster Fuller, Burning Man, Byte Shop, cognitive dissonance, disintermediation, don't be evil, Donald Trump, Douglas Engelbart, Dynabook, Elon Musk, frictionless, glass ceiling, Hacker Ethic, Howard Rheingold, HyperCard, hypertext link, index card, informal economy, information retrieval, Jaron Lanier, Jeff Bezos, Jeff Rulifson, John Markoff, Jony Ive, Kevin Kelly, Kickstarter, knowledge worker, life extension, Marc Andreessen, Mark Zuckerberg, Marshall McLuhan, Maui Hawaii, Menlo Park, Metcalfe’s law, Mother of all demos, move fast and break things, move fast and break things, Network effects, new economy, nuclear winter, PageRank, Paul Buchheit, paypal mafia, peer-to-peer, Peter Thiel, pets.com, pez dispenser, popular electronics, random walk, risk tolerance, Robert Metcalfe, rolodex, self-driving car, side project, Silicon Valley, Silicon Valley startup, skunkworks, Skype, social graph, social web, South of Market, San Francisco, Startup school, Steve Jobs, Steve Wozniak, Steven Levy, Stewart Brand, Ted Nelson, telerobotics, The Hackers Conference, the new new thing, Tim Cook: Apple, tulip mania, V2 rocket, Whole Earth Catalog, Whole Earth Review, Y Combinator

Ruchi Sanghvi: And we would all sit together and have these intellectual conversations: “Hypothetically, if this network was a graph, how would you weight the relationship between two people? How would you weight the relationship between a person and a photo? What does that look like? What would this network eventually look like? What could we do with this network if we actually had it?” Sean Parker: The “social graph” is a math concept from graph theory, but it was a way of trying to explain to people who were kind of academic and mathematically inclined that what we were building was not a product so much as it was a network composed of nodes with a lot of information flowing between those nodes. That’s graph theory. Therefore we’re building a social graph. It was never meant to be talked about publicly. It was a way of articulating to somebody with a math background what we were building. Ruchi Sanghvi: In retrospect, I can’t believe we had those conversations back then. It seems like such a mature thing to be doing.


Designing Search: UX Strategies for Ecommerce Success by Greg Nudelman, Pabini Gabriel-Petit

access to a mobile phone, Albert Einstein, AltaVista, augmented reality, barriers to entry, business intelligence, call centre, crowdsourcing, information retrieval, Internet of things, performance metric, QR code, recommendation engine, RFID, search engine result page, semantic web, Silicon Valley, social graph, social web, speech recognition, text mining, the map is not the territory, The Wisdom of Crowds, web application, zero-sum game, Zipcar

Start by providing a large, inviting text-entry box to encourage questioners to write full sentences (like a human being) instead of query strings or Boolean operators, and label the form button with a word such as “Ask.” At the same time, expose open questions to people as a way of inviting them to answer (or route questions to likely, willing answerers based on affinities you derive from the meta-data in your social graph). Figure 14-4: Yahoo! Answers Using Images in Queries One of the most intriguing aspects of visual browsing has to do with the mismatch between images and the text used by both algorithms and people to describe the images. This text-image mismatch highlights the inherent limitation of finding and managing images through text-based queries. Typically, people see an object, determine what to call it, and then try a few keyword searches based on their interpretation.


Raw Data Is an Oxymoron by Lisa Gitelman

23andMe, collateralized debt obligation, computer age, continuous integration, crowdsourcing, disruptive innovation, Drosophila, Edmond Halley, Filter Bubble, Firefox, fixed income, Google Earth, Howard Rheingold, index card, informal economy, Isaac Newton, Johann Wolfgang von Goethe, knowledge worker, liberal capitalism, lifelogging, longitudinal study, Louis Daguerre, Menlo Park, optical character recognition, Panopticon Jeremy Bentham, peer-to-peer, RFID, Richard Thaler, Silicon Valley, social graph, software studies, statistical model, Stephen Hawking, Steven Pinker, text mining, time value of money, trade route, Turing machine, urban renewal, Vannevar Bush, WikiLeaks

Google’s incorporation of DoubleClick, one of the largest behavioral targeting companies, as well as its partnership with Verizon, would likely be the closest approximation of this single database fantasy, but there is as yet no one entity legally (and technologically) capable of aggregating the entirety of “our” data, which would include not only all governmental and financial records but also our entire search and purchase history, along with our relationship to the social graph. (The value at present is in the aggregating of just a few of these data components.) It is the more general sense that data storage is permanent Dataveillance and Countervailance that leads Viktor Mayer-Schönberger to claim that we have been produced as Borgesian figures, like Funes, who have lost the capacity to forget and thereby lost the capacity to structure a temporal narrative.35 More concretely, the consequence of total storage is that the much-heralded second act of American lives—the mythology of reinvention— cannot be possible if all of the data from the first act is easily accessible.


pages: 266 words: 80,018

The Snowden Files: The Inside Story of the World's Most Wanted Man by Luke Harding

affirmative action, airport security, Anton Chekhov, Apple's 1984 Super Bowl advert, Berlin Wall, Chelsea Manning, don't be evil, drone strike, Edward Snowden, Etonian, Firefox, Google Earth, Jacob Appelbaum, job-hopping, Julian Assange, Khan Academy, kremlinology, Mark Zuckerberg, Maui Hawaii, MITM: man-in-the-middle, national security letter, Panopticon Jeremy Bentham, pre–internet, Ralph Waldo Emerson, rolodex, Rubik’s Cube, Silicon Valley, Skype, social graph, Steve Jobs, undersea cable, web application, WikiLeaks

The idea was to perform something called ‘contact chaining’ on the records of communications, or metadata, it received. Contact chaining is a process of establishing connections between senders and recipients and their contacts. Done rigorously, it establishes a map of connections between people that doesn’t involve actually listening to their phone calls or reading the contents of their emails. Long before Facebook ever existed, the NSA was toying with what the social network would later unveil as a ‘social graph’. But there was a problem. The Justice Department’s intelligence policy branch determined in 1999 that metadata was covered under FISA’s definition of electronic surveillance. That meant that contact chaining was kosher for non-American communications, but if it ensnared Americans, the NSA would be breaking the law. Adding complexity, the transmission of electronic communications even between foreigners overseas could transit through the US, since the data splits apart into digital ‘packets’ rather than travelling from point to point over a telephone line.


pages: 284 words: 79,265

The Half-Life of Facts: Why Everything We Know Has an Expiration Date by Samuel Arbesman

Albert Einstein, Alfred Russel Wallace, Amazon Mechanical Turk, Andrew Wiles, bioinformatics, British Empire, Cesare Marchetti: Marchetti’s constant, Chelsea Manning, Clayton Christensen, cognitive bias, cognitive dissonance, conceptual framework, David Brooks, demographic transition, double entry bookkeeping, double helix, Galaxy Zoo, guest worker program, Gödel, Escher, Bach, Ignaz Semmelweis: hand washing, index fund, invention of movable type, Isaac Newton, John Harrison: Longitude, Kevin Kelly, life extension, Marc Andreessen, meta analysis, meta-analysis, Milgram experiment, Nicholas Carr, P = NP, p-value, Paul Erdős, Pluto: dwarf planet, publication bias, randomized controlled trial, Richard Feynman, Rodney Brooks, scientific worldview, social graph, social web, text mining, the scientific method, Thomas Kuhn: the structure of scientific revolutions, Thomas Malthus, Tyler Cowen: Great Stagnation

Available online: www.who.int/entity/classifications/icd/en/HistoryOfICD.pdf 205 we are up to the tenth revision: The American version even has tens of thousands more classifications than the international version. 205 Just as being exposed: Johnson, Steven. Everything Bad Is Good for You. New York: Riverhead Books, 2005. 205 This is about the number of soldiers: Christakis, Nicholas A., and James H. Fowler. Connected: The Surprising Power of Our Social Networks and How They Shape Our Lives. New York, New York, USA: Little Brown, 2009. 206 and is about 190, as of 2011: Ugander, Johan et al. “The Anatomy of the Facebook Social Graph”; http://arxiv.org/abs/1111.4503. 206 we increase the number of people we are close to: O’Malley, A. James, et al. “Egocentric Social Network Structure, Health, and Pro-Social Behaviors in a National Panel Study of Americans.” PLoS ONE. 7(5): e36250. 206 Sherlock Holmes argued this very point: Doyle, Arthur Conan. A Study in Scarlet, 1887. First published by Ward Lock & Co. in Beeton’s Christmas Annual, London.


pages: 241 words: 78,508

Lean In: Women, Work, and the Will to Lead by Sheryl Sandberg

affirmative action, business process, Cass Sunstein, constrained optimization, experimental economics, fear of failure, gender pay gap, glass ceiling, job satisfaction, labor-force participation, longitudinal study, Mark Zuckerberg, meta analysis, meta-analysis, old-boy network, Richard Thaler, risk tolerance, Silicon Valley, social graph, women in the workforce, young professional

Inside Facebook, few people noticed my TEDTalk, and those who did responded positively. But outside of Facebook, the criticism started to roll in. One of my colleagues from Treasury called to say that “others”—not him, of course—were wondering why I gave more speeches on women’s issues than on Facebook. I had been at the company for two and a half years and given countless speeches on rebuilding marketing around the social graph and exactly one speech on gender. Someone else asked me, “So is this your thing now?” At the time, I didn’t know how to respond. Now I would say yes. I made this my “thing” because we need to disrupt the status quo. Staying quiet and fitting in may have been all the first generations of women who entered corporate America could do; in some cases, it might still be the safest path. But this strategy is not paying off for women as a group.


pages: 269 words: 70,543

Tech Titans of China: How China's Tech Sector Is Challenging the World by Innovating Faster, Working Harder, and Going Global by Rebecca Fannin

Airbnb, augmented reality, autonomous vehicles, blockchain, call centre, cashless society, Chuck Templeton: OpenTable:, cloud computing, computer vision, connected car, corporate governance, cryptocurrency, data is the new oil, Deng Xiaoping, digital map, disruptive innovation, Donald Trump, El Camino Real, Elon Musk, family office, fear of failure, glass ceiling, global supply chain, income inequality, industrial robot, Internet of things, invention of movable type, Jeff Bezos, Kickstarter, knowledge worker, Lyft, Mark Zuckerberg, megacity, Menlo Park, money market fund, Network effects, new economy, peer-to-peer lending, personalized medicine, Peter Thiel, QR code, RFID, ride hailing / ride sharing, Sand Hill Road, self-driving car, sharing economy, Shenzhen was a fishing village, Silicon Valley, Silicon Valley startup, Skype, smart cities, smart transportation, Snapchat, social graph, software as a service, South China Sea, sovereign wealth fund, speech recognition, stealth mode startup, Steve Jobs, supply-chain management, Tim Cook: Apple, Travis Kalanick, Uber and Lyft, Uber for X, uber lyft, urban planning, winner-take-all economy, Y Combinator, young professional

Users open the app and access news through Toutiao’s 4,000 media partnerships without following other accounts, unlike Facebook or Twitter. Anu Hariharan, a partner with Y Combinator’s Continuity Fund in San Francisco, likens Toutiao to YouTube and technology news aggregator Techmeme in one. She finds the most interesting thing about Toutiao to be how it uses machine-and deep-learning algorithms to serve up personalized, high-quality content without any user inputs, social graphs, or product purchase history to rely on.19 From Sea to Shining Sea ByteDance has been moving up in recent years with content deals and smart acquisitions, fulfilling founder Zhang’s mission of making his startup borderless. That goal post got a lot closer when, in November 2017, ByteDance paid about $900 million to acquire Musical.ly, a social video app based in Shanghai with more than 200 million users worldwide.


pages: 1,136 words: 73,489

Working in Public: The Making and Maintenance of Open Source Software by Nadia Eghbal

Amazon Web Services, barriers to entry, Benevolent Dictator For Life (BDFL), bitcoin, Clayton Christensen, cloud computing, commoditize, continuous integration, crowdsourcing, cryptocurrency, David Heinemeier Hansson, death of newspapers, Debian, disruptive innovation, en.wikipedia.org, Ethereum, Firefox, Guido van Rossum, Hacker Ethic, Induced demand, informal economy, Jane Jacobs, Jean Tirole, Kevin Kelly, Kickstarter, Kubernetes, Mark Zuckerberg, Menlo Park, Network effects, node package manager, Norbert Wiener, pirate software, pull request, RFC: Request For Comment, Richard Stallman, Ronald Coase, Ruby on Rails, side project, Silicon Valley, Snapchat, social graph, software as a service, Steve Jobs, Steve Wozniak, Steven Levy, Stewart Brand, The Death and Life of Great American Cities, The Nature of the Firm, transaction costs, two-sided market, urban planning, web application, wikimedia commons, Zimmermann PGP

With millions of lines of code freely available today, the focus has shifted from what developers make to who they are.343 Python developer Shauna Gordon-McKeon once posed a hypothetical question to me: “Take a platform you love. Would you rather lose access to all the past content your connections have posted, or lose the connections themselves?”344 Her point was that the value created on these platforms doesn’t lie in the content itself so much as in the underlying social graph. Our relationship to content matters less than our relationships to the people who make it. As a result, we’re starting to treat content not as a private economic good but as the externalization of our social infrastructure. Platforms have helped bring about this shift more quickly. By reducing the costs of production and distribution, they’ve made it easier for creators to function as one-man operations.


pages: 301 words: 85,263

New Dark Age: Technology and the End of the Future by James Bridle

AI winter, Airbnb, Alfred Russel Wallace, Automated Insights, autonomous vehicles, back-to-the-land, Benoit Mandelbrot, Bernie Sanders, bitcoin, British Empire, Brownian motion, Buckminster Fuller, Capital in the Twenty-First Century by Thomas Piketty, carbon footprint, cognitive bias, cognitive dissonance, combinatorial explosion, computer vision, congestion charging, cryptocurrency, data is the new oil, Donald Trump, Douglas Engelbart, Douglas Engelbart, Douglas Hofstadter, drone strike, Edward Snowden, fear of failure, Flash crash, Google Earth, Haber-Bosch Process, hive mind, income inequality, informal economy, Internet of things, Isaac Newton, John von Neumann, Julian Assange, Kickstarter, late capitalism, lone genius, mandelbrot fractal, meta analysis, meta-analysis, Minecraft, mutually assured destruction, natural language processing, Network effects, oil shock, p-value, pattern recognition, peak oil, recommendation engine, road to serfdom, Robert Mercer, Ronald Reagan, self-driving car, Silicon Valley, Silicon Valley ideology, Skype, social graph, sorting algorithm, South China Sea, speech recognition, Spread Networks laid a new fibre optics cable between New York and Chicago, stem cell, Stuxnet, technoutopianism, the built environment, the scientific method, Uber for X, undersea cable, University of East Anglia, uranium enrichment, Vannevar Bush, WikiLeaks

Computation does not merely augment, frame, and shape culture; by operating beneath our everyday, casual awareness of it, it actually becomes culture. That which computation sets out to map and model it eventually takes over. Google set out to index all human knowledge and became the source and arbiter of that knowledge: it became what people actually think. Facebook set out to map the connections between people – the social graph – and became the platform for those connections, irrevocably reshaping societal relationships. Like an air control system mistaking a flock of birds for a fleet of bombers, software is unable to distinguish between its model of the world and reality – and, once conditioned, neither are we. This conditioning occurs for two reasons: because the combination of opacity and complexity renders much of the computational process illegible; and because computation itself is perceived to be politically and emotionally neutral.


pages: 283 words: 85,824

The People's Platform: Taking Back Power and Culture in the Digital Age by Astra Taylor

A Declaration of the Independence of Cyberspace, American Legislative Exchange Council, Andrew Keen, barriers to entry, Berlin Wall, big-box store, Brewster Kahle, citizen journalism, cloud computing, collateralized debt obligation, Community Supported Agriculture, conceptual framework, corporate social responsibility, creative destruction, cross-subsidies, crowdsourcing, David Brooks, digital Maoism, disintermediation, don't be evil, Donald Trump, Edward Snowden, Fall of the Berlin Wall, Filter Bubble, future of journalism, George Gilder, Google Chrome, Google Glasses, hive mind, income inequality, informal economy, Internet Archive, Internet of things, invisible hand, Jane Jacobs, Jaron Lanier, Jeff Bezos, job automation, John Markoff, Julian Assange, Kevin Kelly, Kickstarter, knowledge worker, Mark Zuckerberg, means of production, Metcalfe’s law, Naomi Klein, Narrative Science, Network effects, new economy, New Journalism, New Urbanism, Nicholas Carr, oil rush, peer-to-peer, Peter Thiel, plutocrats, Plutocrats, post-work, pre–internet, profit motive, recommendation engine, Richard Florida, Richard Stallman, self-driving car, shareholder value, sharing economy, Silicon Valley, Silicon Valley ideology, slashdot, Slavoj Žižek, Snapchat, social graph, Steve Jobs, Stewart Brand, technoutopianism, trade route, Whole Earth Catalog, WikiLeaks, winner-take-all economy, Works Progress Administration, young professional

Fred Turner, From Counterculture to Cyberculture: Stewart Brand, the Whole Earth Network, and the Rise of Digital Utopianism (Chicago: University of Chicago Press, 2006), 238 and 247. 16. For a good discussion of this history, see Evgeny Morozov’s profile of Tim O’Reilly, supporter of the open source movement and founder of O’Reilly Media. Evgeny Morozov, “The Meme Hustler,” Baffler, no. 22 (2013). 17. Openness is the “key to success,” says Jeff Jarvis in What Would Google Do? (New York: HarperBusiness, 2009), 4. 18. Rob Horning, “Social Graph vs. Social Class,” New Inquiry, March 23, 2012. 19. Lawrence Lessig, “The Architecture of Innovation,” Duke Law Journal 51, no. 1783 (2002). Related arguments about the limitations of the framework of left versus right and state versus market are made by Steven Johnson in Future Perfect: The Case for Progress in a Networked Age (New York: Riverhead Books, 2012) and his op-ed “Peer Power, from Potholes to Patents,” Wall Street Journal, September 21, 2012, as well as by Yochai Benkler in The Penguin and the Leviathan: The Triumph of Cooperation over Self-Interest (New York: Crown Business, 2011). 20.


pages: 239 words: 80,319

Lurking: How a Person Became a User by Joanne McNeil

4chan, A Declaration of the Independence of Cyberspace, Ada Lovelace, Airbnb, AltaVista, Amazon Mechanical Turk, Burning Man, Chelsea Manning, Chris Wanstrath, citation needed, cloud computing, crowdsourcing, delayed gratification, dematerialisation, don't be evil, Donald Trump, drone strike, Edward Snowden, Elon Musk, feminist movement, Firefox, Google Earth, Google Glasses, Google Hangouts, helicopter parent, Internet Archive, invention of the telephone, Jeff Bezos, jimmy wales, l'esprit de l'escalier, Marc Andreessen, Mark Zuckerberg, Marshall McLuhan, means of production, Menlo Park, moral panic, move fast and break things, move fast and break things, Network effects, packet switching, PageRank, pre–internet, profit motive, QAnon, recommendation engine, Saturday Night Live, Shoshana Zuboff, Silicon Valley, slashdot, Snapchat, social graph, Stephen Hawking, Steve Jobs, Steven Levy, Stewart Brand, technoutopianism, Ted Nelson, Tim Cook: Apple, trade route, Turing complete, We are the 99%, web application, white flight, Whole Earth Catalog

Social graces—and privacy and psychological well-being, for that matter—are just obstacles in the way of having more information. News Feed dominated a user’s full attention, which meant that Facebook dominated a user’s full attention. As the company explained it, News Feed was a time saver; but to a user, it was a time vacuum. It was a simple idea and yet groundbreaking in its rollout: a daily briefing of all possible life events and gossip of your “social graph,” like a micro-targeted digital experience of This Is Your Life—gossip compiled into a memo. And News Feed decided which of your friends were newsworthy. Facebook developers also wrote algorithms to find logic in communities and the people who are part of them. “Google edgerank,” said one of the Shiny Shiny comments in 2012. If you were to google it now, you would learn that Edgerank was the name of the algorithmic filtering system that Shiny Shiny users had an inkling of when they conducted their research.


pages: 933 words: 205,691

Hadoop: The Definitive Guide by Tom White

Amazon Web Services, bioinformatics, business intelligence, combinatorial explosion, database schema, Debian, domain-specific language, en.wikipedia.org, fault tolerance, full text search, Grace Hopper, information retrieval, Internet Archive, Kickstarter, linked data, loose coupling, openstreetmap, recommendation engine, RFID, SETI@home, social graph, web application

For example, we’ve used variants of the same algorithm[148] to do each of: Rank the most important pages in the Wikipedia linked-document collection. Google uses a vastly more refined version of this approach to identify top search hits. Identify celebrities and experts in the Twitter social graph. Users who have many more followers than their “trstrank” would imply are often spammers. Predict a school’s impact on student education, using millions of anonymized exam scores gathered over five years. Measuring Community The most interesting network in the Infochimps collection is a massive crawl of the Twitter social graph. With more than 90 million nodes, 2 billion edges, it is a marvelous instrument for understanding what people talk about and how they relate to each other. Here is an exploration, using the subgraph of “People who talk about Infochimps or Hadoop,”[149] of three ways to characterize a user’s community: Who are the people they converse with (the @reply graph)?


pages: 713 words: 93,944

Seven Databases in Seven Weeks: A Guide to Modern Databases and the NoSQL Movement by Eric Redmond, Jim Wilson, Jim R. Wilson

AGPL, Amazon Web Services, create, read, update, delete, data is the new oil, database schema, Debian, domain-specific language, en.wikipedia.org, fault tolerance, full text search, general-purpose programming language, Kickstarter, linked data, MVC pattern, natural language processing, node package manager, random walk, recommendation engine, Ruby on Rails, Skype, social graph, web application

We sit upon a vast ocean of data, yet until it’s refined into information, it’s unusable (and with a more crude comparison, there’s a lot of money in data these days). The ease of collecting and ultimately storing, mining, and refining the data out there starts with the database you choose. Deciding which database to choose is often more complex than merely considering which genre maps best to a given domain’s data. Though a social graph may seem to clearly function best with a graph database, if you’re Facebook, you simply have far too much data to choose one. You are more likely going to choose a “Big Data” implementation, such as HBase or Riak. This will force your hand into choosing a columnar or key-value store. In other cases, though you may believe a relational database is clearly the best option for bank transactions, it’s worth knowing that Neo4j also supports ACID transactions, expanding your options.


pages: 332 words: 97,325

The Launch Pad: Inside Y Combinator, Silicon Valley's Most Exclusive School for Startups by Randall Stross

affirmative action, Airbnb, AltaVista, always be closing, Amazon Mechanical Turk, Amazon Web Services, barriers to entry, Ben Horowitz, Burning Man, business cycle, California gold rush, call centre, cloud computing, crowdsourcing, don't be evil, Elon Musk, high net worth, index fund, inventory management, John Markoff, Justin.tv, Lean Startup, Marc Andreessen, Mark Zuckerberg, medical residency, Menlo Park, Minecraft, minimum viable product, Paul Buchheit, Paul Graham, Peter Thiel, QR code, Richard Feynman, Richard Florida, ride hailing / ride sharing, Sam Altman, Sand Hill Road, side project, Silicon Valley, Silicon Valley startup, Skype, social graph, software is eating the world, South of Market, San Francisco, speech recognition, Stanford marshmallow experiment, Startup school, stealth mode startup, Steve Jobs, Steve Wozniak, Steven Levy, TaskRabbit, transaction costs, Y Combinator

You just connect your Facebook account. Now, if I were logged into Facebook on this computer, I wouldn’t even have to type—I would just see an ‘Allow’ dialog. Users don’t have to verify their e-mail address or choose a new password or manage all of the issues of having a new account. Gets them on board really quickly. We pull in all the information about them from Facebook. We get their profile data and their social graph and things like that. That helps us manage our fraud a lot better. Assuming we think they’re a real human, we give them two dollars just to start off with.” “What’s the account behind it? How do they get money into their account?” asks Gaudreau. “After you spend the two dollars or three dollars or whatever it is that you get when you first sign up, we prompt you for a credit card.” Gaudreau reviews: “So the concept is, the initial pay is hassle-free and then you go, OK, now come in and set up your account.”


Data and the City by Rob Kitchin,Tracey P. Lauriault,Gavin McArdle

A Declaration of the Independence of Cyberspace, bike sharing scheme, bitcoin, blockchain, Bretton Woods, Chelsea Manning, citizen journalism, Claude Shannon: information theory, clean water, cloud computing, complexity theory, conceptual framework, corporate governance, correlation does not imply causation, create, read, update, delete, crowdsourcing, cryptocurrency, dematerialisation, digital map, distributed ledger, fault tolerance, fiat currency, Filter Bubble, floating exchange rates, global value chain, Google Earth, hive mind, Internet of things, Kickstarter, knowledge economy, lifelogging, linked data, loose coupling, new economy, New Urbanism, Nicholas Carr, open economy, openstreetmap, packet switching, pattern recognition, performance metric, place-making, RAND corporation, RFID, Richard Florida, ride hailing / ride sharing, semantic web, sentiment analysis, sharing economy, Silicon Valley, Skype, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, smart contracts, smart grid, smart meter, social graph, software studies, statistical model, TaskRabbit, text mining, The Chicago School, The Death and Life of Great American Cities, the market place, the medium is the message, the scientific method, Toyota Production System, urban planning, urban sprawl, web application

Evans himself has shown how devices, such as a smart phone, allow users to ‘dwell’ (find themselves at home) even in so called non-places (Auge 1995) through the connectivity of their devices (Evans 2015). Evans (2015: 6) makes use of Jameson’s concept of ‘social cartography’ to explain how these technologies can be used ‘as a means of understanding and regaining a capacity to act’. The apps, maps, social graphs and network updates on our mobile devices provide us with access to all kinds of urban data, enabling particular ways to act, and this changes our experience of urban space. These studies can be seen in a broader framework of urban studies that has taken an interest in ‘situatedness’ that goes all the way back to (at least) the studies of Goffman in the 1950s (Goffman 1959). As Goffman demonstrated in his work, subjects take clues from their surroundings as to what cultural codes are present, and they might attune their behaviour accordingly.


pages: 394 words: 110,352

The Art of Community: Building the New Age of Participation by Jono Bacon

barriers to entry, Benevolent Dictator For Life (BDFL), collaborative editing, crowdsourcing, Debian, DevOps, do-ocracy, en.wikipedia.org, Firefox, game design, Guido van Rossum, Johann Wolfgang von Goethe, Jono Bacon, Kickstarter, Larry Wall, Mark Shuttleworth, Mark Zuckerberg, openstreetmap, Richard Stallman, side project, Silicon Valley, Skype, slashdot, social graph, software as a service, telemarketer, union organizing, VA Linux, web application

Don’t get dragged into that; many of these folks do this to prove their abilities, knowledge, and intellect…but the real proof is not in the academia that can be cited; rather, it is in the community that you build. In this chapter we are going to focus on the substance of how social media can help you grow and build your community. orm-interview-snippet: The Community Case Book I’m not convinced that the current generation of social media sites have helped communities to prosper in the way that earlier technologies like mailing lists or even Usenet did. They create a social graph centered on the individual rather than the community. The development of tools to support communities is still an untapped opportunity. —Tim O’Reilly, on Social Media Read the full interview in Chapter 14. Being Social One of the challenges of talking about technology in books is that technology changes so rapidly that the content can quickly become outdated. This is a particular challenge for a topic such as social media, as we need to talk about specific tools to really understand how social media works.

The Arab Spring is obviously the canonical example, but #ows (Occupy Wall Street) is happening right now as an example of the interplay between social media and real-world disruption. How do you feel that social media has opened up opportunities for communities to prosper? I’m not convinced that the current generation of social media sites have helped communities to prosper in the way that earlier technologies like mailing lists or even Usenet did. They create a social graph centered on the individual rather than the community. The development of tools to support communities is still an untapped opportunity. Today social media seems to largely involve the exchange of messages. How do you think social media can evolve to further empower collaborative community beyond that of exchanging messages? If you consider Twitter, Facebook, and Google+ to be the apogee of social media, yes, it is mainly about the exchange of messages.


pages: 379 words: 109,612

Is the Internet Changing the Way You Think?: The Net's Impact on Our Minds and Future by John Brockman

A Declaration of the Independence of Cyberspace, Albert Einstein, AltaVista, Amazon Mechanical Turk, Asperger Syndrome, availability heuristic, Benoit Mandelbrot, biofilm, Black Swan, British Empire, conceptual framework, corporate governance, Danny Hillis, Douglas Engelbart, Douglas Engelbart, Emanuel Derman, epigenetics, Flynn Effect, Frank Gehry, Google Earth, hive mind, Howard Rheingold, index card, information retrieval, Internet Archive, invention of writing, Jane Jacobs, Jaron Lanier, John Markoff, Kevin Kelly, lifelogging, lone genius, loss aversion, mandelbrot fractal, Marc Andreessen, Marshall McLuhan, Menlo Park, meta analysis, meta-analysis, New Journalism, Nicholas Carr, out of africa, Paul Samuelson, peer-to-peer, Ponzi scheme, pre–internet, Richard Feynman, Rodney Brooks, Ronald Reagan, Schrödinger's Cat, Search for Extraterrestrial Intelligence, SETI@home, Silicon Valley, Skype, slashdot, smart grid, social graph, social software, social web, Stephen Hawking, Steve Wozniak, Steven Pinker, Stewart Brand, Ted Nelson, telepresence, the medium is the message, the scientific method, The Wealth of Nations by Adam Smith, theory of mind, trade route, upwardly mobile, Vernor Vinge, Whole Earth Catalog, X Prize

Now we all have blogs tethered to our mobile phones, even if they are micro in nature, with Facebook and Twitter accounts. We shouldn’t wait for facts; we should be speculating and testing assumptions as news and knowledge unfold. Facts are, of course, valuable, but speculation gets me further and builds better webs in my mind. We’ve moved from being jurors to being investigators, and the audience is onstage. Support thought bombs and the people who throw them into your social graph. It’s messy but essential. Study the reactions on either side of the aisle, because reactions can be more telling than the facts sometimes. That’s how the Internet has changed my thinking: Trust nothing, debate everything. Harmful One-Liners, an Ocean of Facts, and Rewired Minds Haim Harari Physicist, former president, Weizmann Institute of Science; author, A View from the Eye of the Storm: Terror and Reason in the Middle East It is entirely possible that the Internet is changing our way of thinking in more ways than I am willing to admit, but there are three clear changes that are palpable.


pages: 364 words: 99,897

The Industries of the Future by Alec Ross

23andMe, 3D printing, Airbnb, algorithmic trading, AltaVista, Anne Wojcicki, autonomous vehicles, banking crisis, barriers to entry, Bernie Madoff, bioinformatics, bitcoin, blockchain, Brian Krebs, British Empire, business intelligence, call centre, carbon footprint, cloud computing, collaborative consumption, connected car, corporate governance, Credit Default Swap, cryptocurrency, David Brooks, disintermediation, Dissolution of the Soviet Union, distributed ledger, Edward Glaeser, Edward Snowden, en.wikipedia.org, Erik Brynjolfsson, fiat currency, future of work, global supply chain, Google X / Alphabet X, industrial robot, Internet of things, invention of the printing press, Jaron Lanier, Jeff Bezos, job automation, John Markoff, Joi Ito, Kickstarter, knowledge economy, knowledge worker, lifelogging, litecoin, M-Pesa, Marc Andreessen, Mark Zuckerberg, Mikhail Gorbachev, mobile money, money: store of value / unit of account / medium of exchange, Nelson Mandela, new economy, offshore financial centre, open economy, Parag Khanna, paypal mafia, peer-to-peer, peer-to-peer lending, personalized medicine, Peter Thiel, precision agriculture, pre–internet, RAND corporation, Ray Kurzweil, recommendation engine, ride hailing / ride sharing, Rubik’s Cube, Satoshi Nakamoto, selective serotonin reuptake inhibitor (SSRI), self-driving car, sharing economy, Silicon Valley, Silicon Valley startup, Skype, smart cities, social graph, software as a service, special economic zone, supply-chain management, supply-chain management software, technoutopianism, The Future of Employment, Travis Kalanick, underbanked, Vernor Vinge, Watson beat the top human players on Jeopardy!, women in the workforce, Y Combinator, young professional

Because big data often relies on historical data or at least the status quo, it can easily reproduce discrimination against disadvantaged racial and ethnic minorities. The propensity models used in many algorithms can bake in a bias against someone who lived in the zip code of a low-income neighborhood at any point in his or her life. If an algorithm used by human resources companies queries your social graph and positively weighs candidates with the most existing connections to a workforce, it makes it more difficult to break in in the first place. In effect, these algorithms can hide bias behind a curtain of code. Big data is, by its nature, soulless and uncreative. It nudges us this way and that for reasons we are not meant to understand. It strips us of our privacy and puts our mistakes, secrets, and scandals on public display.


pages: 391 words: 105,382

Utopia Is Creepy: And Other Provocations by Nicholas Carr

Air France Flight 447, Airbnb, Airbus A320, AltaVista, Amazon Mechanical Turk, augmented reality, autonomous vehicles, Bernie Sanders, book scanning, Brewster Kahle, Buckminster Fuller, Burning Man, Captain Sullenberger Hudson, centralized clearinghouse, Charles Lindbergh, cloud computing, cognitive bias, collaborative consumption, computer age, corporate governance, crowdsourcing, Danny Hillis, deskilling, digital map, disruptive innovation, Donald Trump, Electric Kool-Aid Acid Test, Elon Musk, factory automation, failed state, feminist movement, Frederick Winslow Taylor, friendly fire, game design, global village, Google bus, Google Glasses, Google X / Alphabet X, Googley, hive mind, impulse control, indoor plumbing, interchangeable parts, Internet Archive, invention of movable type, invention of the steam engine, invisible hand, Isaac Newton, Jeff Bezos, jimmy wales, Joan Didion, job automation, Kevin Kelly, lifelogging, low skilled workers, Marc Andreessen, Mark Zuckerberg, Marshall McLuhan, means of production, Menlo Park, mental accounting, natural language processing, Network effects, new economy, Nicholas Carr, Norman Mailer, off grid, oil shale / tar sands, Peter Thiel, plutocrats, Plutocrats, profit motive, Ralph Waldo Emerson, Ray Kurzweil, recommendation engine, Republic of Letters, robot derives from the Czech word robota Czech, meaning slave, Ronald Reagan, self-driving car, SETI@home, side project, Silicon Valley, Silicon Valley ideology, Singularitarianism, Snapchat, social graph, social web, speech recognition, Startup school, stem cell, Stephen Hawking, Steve Jobs, Steven Levy, technoutopianism, the medium is the message, theory of mind, Turing test, Whole Earth Catalog, Y Combinator

Marketing is conversational, says Zuckerberg, and advertising is social. There is no intimacy that is not a branding opportunity, no friendship that can’t be monetized, no kiss that doesn’t carry an exchange of value. “Facebook’s ad system,” goes a company press release, “serves Social Ads that combine social actions from your friends—such as a purchase of a product or review of a restaurant—with an advertiser’s message.” What Zuckerberg calls the social graph is, it turns out, a platform for social graft. The Fortune 500 is lining up for the new Facebook service. Coke’s in, big time: The Coca-Cola Company will feature its Sprite brand on a new Facebook Page and will invite users to add an application to their account called “Sprite Sips.” People will be able to create, configure and interact with an animated Sprite Sips character. For consumers in the United States, the experience can be enhanced by entering a PIN code found under the cap of every 20 oz. bottle of Sprite to unlock special features and accessories.


pages: 540 words: 103,101

Building Microservices by Sam Newman

airport security, Amazon Web Services, anti-pattern, business process, call centre, continuous integration, create, read, update, delete, defense in depth, don't repeat yourself, Edward Snowden, fault tolerance, index card, information retrieval, Infrastructure as a Service, inventory management, job automation, Kubernetes, load shedding, loose coupling, microservices, MITM: man-in-the-middle, platform as a service, premature optimization, pull request, recommendation engine, social graph, software as a service, source of truth, the built environment, web application, WebSocket

If one part of our system needs to improve its performance, we might decide to use a different technology stack that is better able to achieve the performance levels required. We may also decide that how we store our data needs to change for different parts of our system. For example, for a social network, we might store our users’ interactions in a graph-oriented database to reflect the highly interconnected nature of a social graph, but perhaps the posts the users make could be stored in a document-oriented data store, giving rise to a heterogeneous architecture like the one shown in Figure 1-1. Figure 1-1. Microservices can allow you to more easily embrace different technologies With microservices, we are also able to adopt technology more quickly, and understand how new advancements may help us. One of the biggest barriers to trying out and adopting new technology is the risks associated with it.


pages: 371 words: 108,317

The Inevitable: Understanding the 12 Technological Forces That Will Shape Our Future by Kevin Kelly

A Declaration of the Independence of Cyberspace, AI winter, Airbnb, Albert Einstein, Amazon Web Services, augmented reality, bank run, barriers to entry, Baxter: Rethink Robotics, bitcoin, blockchain, book scanning, Brewster Kahle, Burning Man, cloud computing, commoditize, computer age, connected car, crowdsourcing, dark matter, dematerialisation, Downton Abbey, Edward Snowden, Elon Musk, Filter Bubble, Freestyle chess, game design, Google Glasses, hive mind, Howard Rheingold, index card, indoor plumbing, industrial robot, Internet Archive, Internet of things, invention of movable type, invisible hand, Jaron Lanier, Jeff Bezos, job automation, John Markoff, Kevin Kelly, Kickstarter, lifelogging, linked data, Lyft, M-Pesa, Marc Andreessen, Marshall McLuhan, means of production, megacity, Minecraft, Mitch Kapor, multi-sided market, natural language processing, Netflix Prize, Network effects, new economy, Nicholas Carr, old-boy network, peer-to-peer, peer-to-peer lending, personalized medicine, placebo effect, planetary scale, postindustrial economy, recommendation engine, RFID, ride hailing / ride sharing, Rodney Brooks, self-driving car, sharing economy, Silicon Valley, slashdot, Snapchat, social graph, social web, software is eating the world, speech recognition, Stephen Hawking, Steven Levy, Ted Nelson, the scientific method, transport as a service, two-sided market, Uber for X, uber lyft, Watson beat the top human players on Jeopardy!, Whole Earth Review, zero-sum game

Books, including fiction, will become a web of names and a community of ideas. (You can, of course, suppress links—and their connections—if you don’t want to see them, as you might while reading a novel. But novels are a tiny subset of everything that is written.) Over the next three decades, scholars and fans, aided by computational algorithms, will knit together the books of the world into a single networked literature. A reader will be able to generate a social graph of an idea, or a timeline of a concept, or a networked map of influence for any notion in the library. We’ll come to understand that no work, no idea stands alone, but that all good, true, and beautiful things are ecosystems of intertwined parts and related entities, past and present. Even when the central core of a text is authored by a lone author (as is likely for many fictional books), the auxiliary networked references, discussions, critiques, bibliography, and hyperlinks surrounding a book will probably be a collaboration.


pages: 386 words: 113,709

Why We Drive: Toward a Philosophy of the Open Road by Matthew B. Crawford

1960s counterculture, Airbus A320, airport security, augmented reality, autonomous vehicles, Bernie Sanders, Boeing 737 MAX, British Empire, Burning Man, call centre, collective bargaining, crony capitalism, deskilling, digital map, don't be evil, Donald Trump, Elon Musk, en.wikipedia.org, Fellow of the Royal Society, gig economy, Google Earth, hive mind, income inequality, informal economy, Internet of things, Jane Jacobs, labour mobility, Lyft, Network effects, New Journalism, New Urbanism, Nicholas Carr, Ponzi scheme, Ralph Nader, ride hailing / ride sharing, Ronald Reagan, Sam Peltzman, security theater, self-driving car, sharing economy, Shoshana Zuboff, Silicon Valley, smart cities, social graph, social intelligence, Stephen Hawking, technoutopianism, the built environment, The Death and Life of Great American Cities, the High Line, too big to fail, traffic fines, Travis Kalanick, Uber and Lyft, Uber for X, uber lyft, Unsafe at Any Speed, urban planning, Wall-E, Works Progress Administration

., “Your Apps Know Where You Were Last Night, and They’re Not Keeping It Secret,” New York Times, December 10, 2018, https://www.nytimes.com/interactive/2018/12/10/business/location-data-privacy-apps.html. 4.Zuboff, Age of Surveillance Capitalism, p. 8 (emphases in original). 5.Zuboff, Age of Surveillance Capitalism, pp. 217–218. 6.Zuboff, Age of Surveillance Capitalism, p. 238. 7.Zuboff, Age of Surveillance Capitalism, p. 201. 8.Zuboff, Age of Surveillance Capitalism, p. 240. 9.Monte Zweben, “Life-Pattern Marketing: Intercept People in Their Daily Routines,” SeeSaw Networks, March 2009, as cited in Zuboff, Age of Surveillance Capitalism, p. 243. 10.Dyani Sabin, “The Secret History of ‘Pokémon GO,’ as Told by Creator John Hanke,” Inverse, February 28, 2017, https://www.inverse.com/article/28485-pokemon-go-secret-history-google-maps-ingress-john-hanke-updates. 11.See Natasha Dow Schull, Addiction by Design: Machine Gambling in Las Vegas (Princeton, NJ: Princeton University Press, 2012), and the chapter “Autism as a Design Principle” in my The World Beyond Your Head. 12.Various journalists took it upon themselves to actually read the pages-long privacy policy and data-collection practices of the Pokémon Go! app and discovered that it requires you to grant it access not only to the phone’s camera, but also permission to harvest your contacts and find other accounts on the device, yielding a “detailed location-based social graph.” See Joseph Bernstein, “You Should Probably Check Your Pokémon Go Privacy Settings,” Buzzfeed, July 11, 2016, as cited in Zuboff, Age of Surveillance Capitalism, p. 317. CONCLUDING REMARKS: SOVEREIGNTY ON THE ROAD 1.Further, license plate readers are being installed on those digital road signs you may have noticed going up, creating a database that is shared by government agencies, from which a portrait of one’s movements can be drawn. https://www.fbo.gov/index?


pages: 597 words: 119,204

Website Optimization by Andrew B. King

AltaVista, bounce rate, don't be evil, en.wikipedia.org, Firefox, In Cold Blood by Truman Capote, information retrieval, iterative process, Kickstarter, medical malpractice, Network effects, performance metric, search engine result page, second-price auction, second-price sealed-bid, semantic web, Silicon Valley, slashdot, social graph, Steve Jobs, web application

Metadata generally means machine-readable "data about data," which can take many forms. Perhaps the simplest form falls under the classification of microformats, [35] which can be as simple as a single attribute value such as nofollow, described in more detail in "Step 10: Build Inbound Links with Online Promotion," earlier in this chapter. Another popular single-attribute microformat is XFN, [36] which allows individual links to be labeled as connections on a social graph, with values such as acquaintance, co-worker, spouse, or even sweetheart. A special value of me indicates that a link points to another resource from the same author, as in the following example: <a href="myothersite.example.com" rel="me">Homepage</a> Some microformats expose more structure, particularly to represent people and events and to review information, all of which can help make sites more presentable in semantic search engines.


pages: 525 words: 116,295

The New Digital Age: Transforming Nations, Businesses, and Our Lives by Eric Schmidt, Jared Cohen

access to a mobile phone, additive manufacturing, airport security, Amazon Mechanical Turk, Amazon Web Services, anti-communist, augmented reality, Ayatollah Khomeini, barriers to entry, bitcoin, borderless world, call centre, Chelsea Manning, citizen journalism, clean water, cloud computing, crowdsourcing, data acquisition, Dean Kamen, drone strike, Elon Musk, failed state, fear of failure, Filter Bubble, Google Earth, Google Glasses, hive mind, income inequality, information trail, invention of the printing press, job automation, John Markoff, Julian Assange, Khan Academy, Kickstarter, knowledge economy, Law of Accelerating Returns, market fundamentalism, means of production, MITM: man-in-the-middle, mobile money, mutually assured destruction, Naomi Klein, Nelson Mandela, offshore financial centre, Parag Khanna, peer-to-peer, peer-to-peer lending, personalized medicine, Peter Singer: altruism, Ray Kurzweil, RFID, Robert Bork, self-driving car, sentiment analysis, Silicon Valley, Skype, Snapchat, social graph, speech recognition, Steve Jobs, Steven Pinker, Stewart Brand, Stuxnet, The Wisdom of Crowds, upwardly mobile, Whole Earth Catalog, WikiLeaks, young professional, zero day

We had the opportunity to tour the command center for Plataforma México, Mexico’s impressive national crime database and perhaps the best model of an integrated data system operating today. Housed in an underground bunker in the Secretariat of Public Security compound in Mexico City, this large database integrates intelligence, crime reports and real-time data from surveillance cameras and other inputs from agencies and states across the country. Specialized algorithms can extract patterns, project social graphs and monitor restive areas for violence and crime as well as for natural disasters and other civilian emergencies. The level of surveillance and technological sophistication of Plataforma México that we saw is extraordinary—but then, so are the security challenges that Mexican authorities face. Therein lies the challenge looking ahead: Mexico is the ideal location for a pilot project like this because of its entrenched security problems, but once the model has been proven, what is to stop other states with less justifiable motivations from building something similar?


pages: 461 words: 125,845

This Machine Kills Secrets: Julian Assange, the Cypherpunks, and Their Fight to Empower Whistleblowers by Andy Greenberg

Apple II, Ayatollah Khomeini, Berlin Wall, Bill Gates: Altair 8800, Burning Man, Chelsea Manning, computerized markets, crowdsourcing, cryptocurrency, domain-specific language, drone strike, en.wikipedia.org, fault tolerance, hive mind, Jacob Appelbaum, Julian Assange, Mahatma Gandhi, Mitch Kapor, MITM: man-in-the-middle, Mohammed Bouazizi, nuclear winter, offshore financial centre, pattern recognition, profit motive, Ralph Nader, Richard Stallman, Robert Hanssen: Double agent, Silicon Valley, Silicon Valley ideology, Skype, social graph, statistical model, stem cell, Steve Jobs, Steve Wozniak, Steven Levy, undersea cable, Vernor Vinge, We are Anonymous. We are Legion, We are the 99%, WikiLeaks, X Prize, Zimmermann PGP

As Anonymous began to share the media spotlight surrounding Cablegate, Aaron Barr became increasingly preoccupied with the group. It represented a tempting case study for the kind of analysis he hoped to validate: Although Anons fiercely guarded their true names, they openly congregated and planned their actions in online chat rooms and crowd-sourced documents using pseudonyms. Despite all its proxies and masks, perhaps the entire social graph of Anonymous could be infiltrated and charted. Barr had been planning on giving a talk at the BSides security conference in San Francisco in March, in which he’d use clues built from a Web of online relationships to reveal human flaws in the security of a nuclear facility in Pennsylvania and the army intelligence group INSCOM. He had titled the talk “Who Needs the NSA When We Have Social Media?”


pages: 567 words: 122,311

Lean Analytics: Use Data to Build a Better Startup Faster by Alistair Croll, Benjamin Yoskovitz

Airbnb, Amazon Mechanical Turk, Amazon Web Services, Any sufficiently advanced technology is indistinguishable from magic, barriers to entry, Bay Area Rapid Transit, Ben Horowitz, bounce rate, business intelligence, call centre, cloud computing, cognitive bias, commoditize, constrained optimization, en.wikipedia.org, Firefox, Frederick Winslow Taylor, frictionless, frictionless market, game design, Google X / Alphabet X, Infrastructure as a Service, Internet of things, inventory management, Kickstarter, lateral thinking, Lean Startup, lifelogging, longitudinal study, Marshall McLuhan, minimum viable product, Network effects, pattern recognition, Paul Graham, performance metric, place-making, platform as a service, recommendation engine, ride hailing / ride sharing, rolodex, sentiment analysis, skunkworks, Skype, social graph, social software, software as a service, Steve Jobs, subscription business, telemarketer, transaction costs, two-sided market, Uber for X, web application, Y Combinator

Initially, numbers dropped as a result of the new focus, but by 2009, the team grew its community to 4.5 million users—and unlike the users who’d been lost in the change, these were actively engaged. The company went through some ups and downs after that, as Facebook limited applications’ abilities to spread virally. Ultimately, the company moved off Facebook, grew independently, and sold to Sugar Inc. in early 2012. Summary Circle of Friends was a social graph application in the right place at the right time—with the wrong market. By analyzing patterns of engagement and desirable behavior, then finding out what those users had in common, the company found the right market for its offering. Once the company had found its target, it focused—all the way to changing its name. Pivot hard or go home, and be prepared to burn some bridges. Analytics Lessons Learned The key to Mike’s success with Circle of Moms was his ability to dig into the data and look for meaningful patterns and opportunities.


pages: 515 words: 126,820

Blockchain Revolution: How the Technology Behind Bitcoin Is Changing Money, Business, and the World by Don Tapscott, Alex Tapscott

Airbnb, altcoin, asset-backed security, autonomous vehicles, barriers to entry, bitcoin, blockchain, Blythe Masters, Bretton Woods, business process, buy and hold, Capital in the Twenty-First Century by Thomas Piketty, carbon footprint, clean water, cloud computing, cognitive dissonance, commoditize, corporate governance, corporate social responsibility, creative destruction, Credit Default Swap, crowdsourcing, cryptocurrency, disintermediation, disruptive innovation, distributed ledger, Donald Trump, double entry bookkeeping, Edward Snowden, Elon Musk, Erik Brynjolfsson, Ethereum, ethereum blockchain, failed state, fiat currency, financial innovation, Firefox, first square of the chessboard, first square of the chessboard / second half of the chessboard, future of work, Galaxy Zoo, George Gilder, glass ceiling, Google bus, Hernando de Soto, income inequality, informal economy, information asymmetry, intangible asset, interest rate swap, Internet of things, Jeff Bezos, jimmy wales, Kickstarter, knowledge worker, Kodak vs Instagram, Lean Startup, litecoin, Lyft, M-Pesa, Marc Andreessen, Mark Zuckerberg, Marshall McLuhan, means of production, microcredit, mobile money, money market fund, Network effects, new economy, Oculus Rift, off grid, pattern recognition, peer-to-peer, peer-to-peer lending, peer-to-peer model, performance metric, Peter Thiel, planetary scale, Ponzi scheme, prediction markets, price mechanism, Productivity paradox, QR code, quantitative easing, ransomware, Ray Kurzweil, renewable energy credits, rent-seeking, ride hailing / ride sharing, Ronald Coase, Ronald Reagan, Satoshi Nakamoto, Second Machine Age, seigniorage, self-driving car, sharing economy, Silicon Valley, Skype, smart contracts, smart grid, social graph, social intelligence, social software, standardized shipping container, Stephen Hawking, Steve Jobs, Steve Wozniak, Stewart Brand, supply-chain management, TaskRabbit, The Fortune at the Bottom of the Pyramid, The Nature of the Firm, The Wisdom of Crowds, transaction costs, Turing complete, Turing test, Uber and Lyft, uber lyft, unbanked and underbanked, underbanked, unorthodox policies, wealth creators, X Prize, Y2K, Zipcar

Privacy is enhanced in other manners too. For example, spy agencies can’t conduct traffic analysis because they are unable to discern the source or destination of messages. There would also be a nifty mechanism for finding people and feeds that you might care about. In addition, distributed tools aggregate and present interesting new people or information for you to follow or friend, possibly using Facebook’s social graph to help out. Lubin calls this “bootstrapping the decentralized Web using the pillars of the centralized Web.”42 Experience shows that value ultimately wins out in the digital age. The benefits of this distributed model are huge—at least to the users and companies. The huge resources of social media companies notwithstanding, there is no end to the richness and functionality that we can develop in such an open source environment.


Virtual Competition by Ariel Ezrachi, Maurice E. Stucke

Airbnb, Albert Einstein, algorithmic trading, barriers to entry, cloud computing, collaborative economy, commoditize, corporate governance, crony capitalism, crowdsourcing, Daniel Kahneman / Amos Tversky, David Graeber, demand response, disintermediation, disruptive innovation, double helix, Downton Abbey, Erik Brynjolfsson, experimental economics, Firefox, framing effect, Google Chrome, index arbitrage, information asymmetry, interest rate derivative, Internet of things, invisible hand, Jean Tirole, John Markoff, Joseph Schumpeter, Kenneth Arrow, light touch regulation, linked data, loss aversion, Lyft, Mark Zuckerberg, market clearing, market friction, Milgram experiment, multi-sided market, natural language processing, Network effects, new economy, offshore financial centre, pattern recognition, prediction markets, price discrimination, price stability, profit maximization, profit motive, race to the bottom, rent-seeking, Richard Thaler, ride hailing / ride sharing, road to serfdom, Robert Bork, Ronald Reagan, self-driving car, sharing economy, Silicon Valley, Skype, smart cities, smart meter, Snapchat, social graph, Steve Jobs, supply-chain management, telemarketer, The Chicago School, The Myth of the Rational Market, The Wealth of Nations by Adam Smith, too big to fail, transaction costs, Travis Kalanick, turn-by-turn navigation, two-sided market, Uber and Lyft, Uber for X, uber lyft, Watson beat the top human players on Jeopardy!, women in the workforce, yield management

As the Wall Street Journal reported, “Dozens of startups that had been using Facebook data have shut down, been acquired or overhauled their businesses.”68 One venture capitalist noted the shift from joint extraction to capture: “Companies are open until they have liquidity and users. Then they start to control.”69 He too is “becoming increasingly skeptical that you can build a lasting, stand-alone business based on access to someone else’s social graph.”70 While closing one door, Facebook is opening other doors for companies to target us. For example, Facebook introduced bots for its Facebook Messenger text ing platform. The new technology, backed by powerful algorithms, will make use of user data to better target users with ads and promotions. The bots will foster communication with companies and enable Facebook’s algorithms to better learn of your preferences.


Beautiful Data: The Stories Behind Elegant Data Solutions by Toby Segaran, Jeff Hammerbacher

23andMe, airport security, Amazon Mechanical Turk, bioinformatics, Black Swan, business intelligence, card file, cloud computing, computer vision, correlation coefficient, correlation does not imply causation, crowdsourcing, Daniel Kahneman / Amos Tversky, DARPA: Urban Challenge, data acquisition, database schema, double helix, en.wikipedia.org, epigenetics, fault tolerance, Firefox, Hans Rosling, housing crisis, information retrieval, lake wobegon effect, longitudinal study, Mars Rover, natural language processing, openstreetmap, prediction markets, profit motive, semantic web, sentiment analysis, Simon Singh, social graph, SPARQL, speech recognition, statistical model, supply-chain management, text mining, Vernor Vinge, web application

Some features we were able to generate using Hadoop included various measures of friend network density and user categories based on profile features. Another internal study to understand what motivates content contribution from new users was written up in the paper “Feed Me: Motivating Newcomer Contribution in Social Network Sites,” published at the 2009 CHI conference. A more recent study from the Facebook Data team looks at how information flows through the Facebook social graph; the study is titled “Gesundheit! Modeling Contagion through Facebook News Feed,” and has been accepted for the 2009 ICWSM conference. Every day, evidence is collected, hypotheses are tested, applications are built, and new insights are generated using the shared Information Platform at Facebook. Outside of Facebook, similar systems were being constructed in parallel. MAD Skills and Cosmos In “MAD Skills: New Analysis Practices for Big Data,” a paper from the 2009 VLDB conference, the analysis environment at Fox Interactive Media (FIM) is described in detail.


pages: 543 words: 153,550

Model Thinker: What You Need to Know to Make Data Work for You by Scott E. Page

"Robert Solow", Airbnb, Albert Einstein, Alfred Russel Wallace, algorithmic trading, Alvin Roth, assortative mating, Bernie Madoff, bitcoin, Black Swan, blockchain, business cycle, Capital in the Twenty-First Century by Thomas Piketty, Checklist Manifesto, computer age, corporate governance, correlation does not imply causation, cuban missile crisis, deliberate practice, discrete time, distributed ledger, en.wikipedia.org, Estimating the Reproducibility of Psychological Science, Everything should be made as simple as possible, experimental economics, first-price auction, Flash crash, Geoffrey West, Santa Fe Institute, germ theory of disease, Gini coefficient, High speed trading, impulse control, income inequality, Isaac Newton, John von Neumann, Kenneth Rogoff, knowledge economy, knowledge worker, Long Term Capital Management, loss aversion, low skilled workers, Mark Zuckerberg, market design, meta analysis, meta-analysis, money market fund, Nash equilibrium, natural language processing, Network effects, p-value, Pareto efficiency, pattern recognition, Paul Erdős, Paul Samuelson, phenotype, pre–internet, prisoner's dilemma, race to the bottom, random walk, randomized controlled trial, Richard Feynman, Richard Thaler, school choice, sealed-bid auction, second-price auction, selection bias, six sigma, social graph, spectrum auction, statistical model, Stephen Hawking, Supply of New York City Cabdrivers, The Bell Curve by Richard Herrnstein and Charles Murray, The Great Moderation, The Rise and Fall of American Growth, the rule of 72, the scientific method, The Spirit Level, The Wisdom of Crowds, Thomas Malthus, Thorstein Veblen, urban sprawl, value at risk, web application, winner-take-all economy, zero-sum game

Tweedle, Valerie, and Robert J. Smith. 2012. “A Mathematical Model of Bieber Fever: The Most Infectious Disease of Our Time?” In Understanding the Dynamics of Emerging and Re-Emerging Infectious Diseases Using Mathematical Models, ed. Steady Mushayabasa and Claver P. Bhunu. Cham, Switzerland: Springer. Ugander, Johan, Brian Karrer, Lars Backstrom, and Cameron Marlow. 2011. “The Anatomy of the Facebook Social Graph.” arXiv:1111.4503. Updike, John. 1960. “Hub Fans Bid Adieu.” New Yorker, October 22. US Bureau of Labor Statistics. 2013. Consumer Expenditures in 2011. Report 1042, April. Washington, DC: BLS. Uzzi, Brian, Satyam Mukherjee, Michael Stringer, and Ben Jones. 2013. “Atypical Combinations and Scientific Impact.” Science 342: 468–471. Van Noorden, Richard. 2015. “Interdisciplinary Research by the Numbers.”


pages: 559 words: 155,372

Chaos Monkeys: Obscene Fortune and Random Failure in Silicon Valley by Antonio Garcia Martinez

Airbnb, airport security, always be closing, Amazon Web Services, Burning Man, Celtic Tiger, centralized clearinghouse, cognitive dissonance, collective bargaining, corporate governance, Credit Default Swap, crowdsourcing, death of newspapers, disruptive innovation, drone strike, El Camino Real, Elon Musk, Emanuel Derman, financial independence, global supply chain, Goldman Sachs: Vampire Squid, hive mind, income inequality, information asymmetry, interest rate swap, intermodal, Jeff Bezos, Kickstarter, Malcom McLean invented shipping containers, Marc Andreessen, Mark Zuckerberg, Maui Hawaii, means of production, Menlo Park, minimum viable product, MITM: man-in-the-middle, move fast and break things, move fast and break things, Network effects, orbital mechanics / astrodynamics, Paul Graham, performance metric, Peter Thiel, Ponzi scheme, pre–internet, Ralph Waldo Emerson, random walk, Ruby on Rails, Sam Altman, Sand Hill Road, Scientific racism, second-price auction, self-driving car, Silicon Valley, Silicon Valley startup, Skype, Snapchat, social graph, social web, Socratic dialogue, source of truth, Steve Jobs, telemarketer, undersea cable, urban renewal, Y Combinator, zero-sum game, éminence grise

I never heard the use of any bundling term like “buck,” but sums of millions of users were splashed around between products and test groups like chips at a one- or two-dollar poker table. What would have been an important user milestone for any consumer startup became the most minimal unit of account inside Facebook. As a geographic tangent: New Zealand was commonly used as a test bed for new user-facing products. It was perfect due to its English-language usage, its relative isolation in terms of the social graph (i.e., most friend links were internal to the country), and, frankly, its lack of newsworthiness, so any gossip or reporting of new Facebook features ran a low risk of leaking back to the real target markets of the United States and Europe. Aotearoa is the original Maori word for New Zealand, which roughly translated means “Facebook test set.” Thus does that verdant island nation, graced with stunning fjords and clear alpine lakes, sample whatever random product twiddle a twenty-three-year-old Facebook engineer in Menlo Park dreams up.


pages: 470 words: 148,730

Good Economics for Hard Times: Better Answers to Our Biggest Problems by Abhijit V. Banerjee, Esther Duflo

"Robert Solow", 3D printing, affirmative action, Affordable Care Act / Obamacare, Airbnb, basic income, Bernie Sanders, business cycle, call centre, Capital in the Twenty-First Century by Thomas Piketty, Cass Sunstein, charter city, correlation does not imply causation, creative destruction, Daniel Kahneman / Amos Tversky, David Ricardo: comparative advantage, decarbonisation, Deng Xiaoping, Donald Trump, Edward Glaeser, en.wikipedia.org, endowment effect, energy transition, Erik Brynjolfsson, experimental economics, experimental subject, facts on the ground, fear of failure, financial innovation, George Akerlof, high net worth, immigration reform, income inequality, Indoor air pollution, industrial cluster, industrial robot, information asymmetry, Intergovernmental Panel on Climate Change (IPCC), Jane Jacobs, Jean Tirole, Jeff Bezos, job automation, Joseph Schumpeter, labor-force participation, land reform, loss aversion, low skilled workers, manufacturing employment, Mark Zuckerberg, mass immigration, Network effects, new economy, New Urbanism, non-tariff barriers, obamacare, offshore financial centre, open economy, Paul Samuelson, place-making, price stability, profit maximization, purchasing power parity, race to the bottom, RAND corporation, randomized controlled trial, Richard Thaler, ride hailing / ride sharing, Robert Gordon, Ronald Reagan, school choice, Second Machine Age, secular stagnation, self-driving car, shareholder value, short selling, Silicon Valley, smart meter, social graph, spinning jenny, Steve Jobs, technology bubble, The Chicago School, The Future of Employment, The Market for Lemons, The Rise and Fall of American Growth, The Wealth of Nations by Adam Smith, total factor productivity, trade liberalization, transaction costs, trickle-down economics, universal basic income, urban sprawl, very high income, War on Poverty, women in the workforce, working-age population, Y2K

Shapiro, and Matt Taddy, “Measuring Polarization in High-Dimensional Data: Method and Application to Congressional Speech,” working paper, 2016. 63 Yuriy Gorodnickenko, Tho Pham, and Oleksandr Talavera, “Social Media, Sentiment and Public Opinions: Evidence from #Brexit and #US Election,” National Bureau of Economics Research Working Paper 24631, 2018. 64 Shanto Iyengar, Gaurav Sood, and Yphtach Lelkes, “Affect, Not Ideology: A Social Identity Perspective on Polarization,” Public Opinion Quarterly, 2012, http://doi.org/10.1093/poq/nfs038. 65 “Most Popular Social Networks Worldwide as of January 2019, Ranked by Number of Active Users (in millions),” Statista.com, 2019, accessed April 21, 2019, https://www.statista.com/statistics/272014/global-social-networks-ranked-by-number-of-users/. 66 Maeve Duggan, Nicole B. Ellison, Cliff Lampe, Amanda Lenhart, and Mary Madden,“Social Media Update 2014,” Pew Research Center, 2015, http://www.pewinternet.org/2015/01/09/social-media-update-2014/. 67 Johan Ugander, Brian Karrer, Lars Backstrom, and Cameron Marlow, “The Anatomy of the Facebook Social Graph,” Cornell University, 2011, https://arxiv.org/abs/1111.4503v1. 68 Yosh Halberstam and Brian Knight “Homophily, Group Size, and the Diffusion of Political Information in Social Networks: Evidence from Twitter,” Journal of Public Economics, 143 (November 2016), 73–88, https://doi.org/10.1016/j.jpubeco.2016.08.011. 69 David Brock, The Republican Noise Machine (New York: Crown, 2004). 70 David Yanagizawa-Drott, “Propaganda and Conflict: Evidence from the Rwandan Genocide,” Quarterly Journal of Economics 129, no. 4 (2014), https://doi.org/10.1093/qje/qju020. 71 Matthew Gentzkow and Jesse Shapiro, “Ideological Segregation Online and Offline,” Quarterly Journal of Economics 126, no. 4 (2011), http://doi.org/10.1093/qje/qjr044. 72 Levi Boxell, Matthew Gentzkow, and Jesse Shapiro, “Greater Internet Use Is Not Associated with Faster Growth in Political Polarization among US Demographic Groups,” Proceedings of the National Academy of Sciences of the United States of America, 2017, https://doi.org/10.1073/pnas.1706588114. 73 Gregory J.


pages: 552 words: 168,518

MacroWikinomics: Rebooting Business and the World by Don Tapscott, Anthony D. Williams

accounting loophole / creative accounting, airport security, Andrew Keen, augmented reality, Ayatollah Khomeini, barriers to entry, Ben Horowitz, bioinformatics, Bretton Woods, business climate, business process, buy and hold, car-free, carbon footprint, Charles Lindbergh, citizen journalism, Clayton Christensen, clean water, Climategate, Climatic Research Unit, cloud computing, collaborative editing, collapse of Lehman Brothers, collateralized debt obligation, colonial rule, commoditize, corporate governance, corporate social responsibility, creative destruction, crowdsourcing, death of newspapers, demographic transition, disruptive innovation, distributed generation, don't be evil, en.wikipedia.org, energy security, energy transition, Exxon Valdez, failed state, fault tolerance, financial innovation, Galaxy Zoo, game design, global village, Google Earth, Hans Rosling, hive mind, Home mortgage interest deduction, information asymmetry, interchangeable parts, Internet of things, invention of movable type, Isaac Newton, James Watt: steam engine, Jaron Lanier, jimmy wales, Joseph Schumpeter, Julian Assange, Kevin Kelly, Kickstarter, knowledge economy, knowledge worker, Marc Andreessen, Marshall McLuhan, mass immigration, medical bankruptcy, megacity, mortgage tax deduction, Netflix Prize, new economy, Nicholas Carr, oil shock, old-boy network, online collectivism, open borders, open economy, pattern recognition, peer-to-peer lending, personalized medicine, Ray Kurzweil, RFID, ride hailing / ride sharing, Ronald Reagan, Rubik’s Cube, scientific mainstream, shareholder value, Silicon Valley, Skype, smart grid, smart meter, social graph, social web, software patent, Steve Jobs, text mining, the scientific method, The Wisdom of Crowds, transaction costs, transfer pricing, University of East Anglia, urban sprawl, value at risk, WikiLeaks, X Prize, young professional, Zipcar

“The network works a bit like Facebook,” says Reinicke. “If you had a good ride with someone you can add them to your network and the next time you need a ride their profile is searched first.” The network extends to two degrees to increase the chances of a match. A host of other start-ups are also exploring the space of “social commuting” by developing communities around ridesharing. Some, such as GoLoco or Zimride, rely on the social graph to create groups of friends who carpool. Others such as PickupPal or Carticipate use geopositioning—either mobile or computer based—to match people departing from the same location. One such start-up called Wikit proposes a form of transportation marketplace, where drivers could advertise their daily routes using their incar GPS device and would-be passengers could publish their current location, desired destinations, and the amount they are willing to pay to get there, from the convenience of their GPS-enabled phone.


pages: 574 words: 164,509

Superintelligence: Paths, Dangers, Strategies by Nick Bostrom

agricultural Revolution, AI winter, Albert Einstein, algorithmic trading, anthropic principle, anti-communist, artificial general intelligence, autonomous vehicles, barriers to entry, Bayesian statistics, bioinformatics, brain emulation, cloud computing, combinatorial explosion, computer vision, cosmological constant, dark matter, DARPA: Urban Challenge, data acquisition, delayed gratification, demographic transition, different worldview, Donald Knuth, Douglas Hofstadter, Drosophila, Elon Musk, en.wikipedia.org, endogenous growth, epigenetics, fear of failure, Flash crash, Flynn Effect, friendly AI, Gödel, Escher, Bach, income inequality, industrial robot, informal economy, information retrieval, interchangeable parts, iterative process, job automation, John Markoff, John von Neumann, knowledge worker, longitudinal study, Menlo Park, meta analysis, meta-analysis, mutually assured destruction, Nash equilibrium, Netflix Prize, new economy, Norbert Wiener, NP-complete, nuclear winter, optical character recognition, pattern recognition, performance metric, phenotype, prediction markets, price stability, principal–agent problem, race to the bottom, random walk, Ray Kurzweil, recommendation engine, reversible computing, social graph, speech recognition, Stanislav Petrov, statistical model, stem cell, Stephen Hawking, strong AI, superintelligent machines, supervolcano, technological singularity, technoutopianism, The Coming Technological Singularity, The Nature of the Firm, Thomas Kuhn: the structure of scientific revolutions, transaction costs, Turing machine, Vernor Vinge, Watson beat the top human players on Jeopardy!, World Values Survey, zero-sum game

Even so, it would not be too difficult to identify most capable individuals with a serious long-standing interest in artificial general intelligence research. Such individuals usually leave visible trails. They may have published academic papers, presented at conferences, posted on Internet forums, or earned degrees from leading computer science departments. They may also have had communications with other AI researchers, allowing them to be identified by mapping the social graph. Projects designed from the outset to be secret could be more difficult to detect. An ordinary software development project could serve as a front.26 Only careful analysis of the code being produced would reveal the true nature of what the project was trying to accomplish. Such analysis would require a lot of (highly skilled) manpower, whence only a small number of suspect projects could be scrutinized at this level.


pages: 680 words: 157,865

Beautiful Architecture: Leading Thinkers Reveal the Hidden Beauty in Software Design by Diomidis Spinellis, Georgios Gousios

Albert Einstein, barriers to entry, business intelligence, business process, call centre, continuous integration, corporate governance, database schema, Debian, domain-specific language, don't repeat yourself, Donald Knuth, en.wikipedia.org, fault tolerance, Firefox, general-purpose programming language, iterative process, linked data, locality of reference, loose coupling, meta analysis, meta-analysis, MVC pattern, peer-to-peer, premature optimization, recommendation engine, Richard Stallman, Ruby on Rails, semantic web, smart cities, social graph, social web, SPARQL, Steve Jobs, Stewart Brand, traveling salesman, Turing complete, type inference, web application, zero-coupon bond

However, even with these new capabilities, these applications don’t yet enjoy the full power of a social utility like Facebook. The applications still need to be discovered by many users to become valuable. At the same time, not all of the internal data supporting the social platform can be made available to these external stacks. The platform creator needs to solve each of these problems, which we take in turn. product problem: For social applications to gain compelling critical mass, users on the supporting social graph must be made aware of other users’ interactions with these applications. This suggests deeper integration of the application into the social site. This problem has existed since the dawn of software: the difficulty of getting our data, product, or system out into general use. The lack of users becomes a particularly notable difficulty in the space of Web 2.0 because without users to consume and (especially) generate our content, how useful can our system ever become?


pages: 677 words: 206,548

Future Crimes: Everything Is Connected, Everyone Is Vulnerable and What We Can Do About It by Marc Goodman

23andMe, 3D printing, active measures, additive manufacturing, Affordable Care Act / Obamacare, Airbnb, airport security, Albert Einstein, algorithmic trading, artificial general intelligence, Asilomar, Asilomar Conference on Recombinant DNA, augmented reality, autonomous vehicles, Baxter: Rethink Robotics, Bill Joy: nanobots, bitcoin, Black Swan, blockchain, borderless world, Brian Krebs, business process, butterfly effect, call centre, Charles Lindbergh, Chelsea Manning, cloud computing, cognitive dissonance, computer vision, connected car, corporate governance, crowdsourcing, cryptocurrency, data acquisition, data is the new oil, Dean Kamen, disintermediation, don't be evil, double helix, Downton Abbey, drone strike, Edward Snowden, Elon Musk, Erik Brynjolfsson, Filter Bubble, Firefox, Flash crash, future of work, game design, global pandemic, Google Chrome, Google Earth, Google Glasses, Gordon Gekko, high net worth, High speed trading, hive mind, Howard Rheingold, hypertext link, illegal immigration, impulse control, industrial robot, Intergovernmental Panel on Climate Change (IPCC), Internet of things, Jaron Lanier, Jeff Bezos, job automation, John Harrison: Longitude, John Markoff, Joi Ito, Jony Ive, Julian Assange, Kevin Kelly, Khan Academy, Kickstarter, knowledge worker, Kuwabatake Sanjuro: assassination market, Law of Accelerating Returns, Lean Startup, license plate recognition, lifelogging, litecoin, low earth orbit, M-Pesa, Mark Zuckerberg, Marshall McLuhan, Menlo Park, Metcalfe’s law, MITM: man-in-the-middle, mobile money, more computing power than Apollo, move fast and break things, move fast and break things, Nate Silver, national security letter, natural language processing, obamacare, Occupy movement, Oculus Rift, off grid, offshore financial centre, optical character recognition, Parag Khanna, pattern recognition, peer-to-peer, personalized medicine, Peter H. Diamandis: Planetary Resources, Peter Thiel, pre–internet, RAND corporation, ransomware, Ray Kurzweil, refrigerator car, RFID, ride hailing / ride sharing, Rodney Brooks, Ross Ulbricht, Satoshi Nakamoto, Second Machine Age, security theater, self-driving car, shareholder value, Silicon Valley, Silicon Valley startup, Skype, smart cities, smart grid, smart meter, Snapchat, social graph, software as a service, speech recognition, stealth mode startup, Stephen Hawking, Steve Jobs, Steve Wozniak, strong AI, Stuxnet, supply-chain management, technological singularity, telepresence, telepresence robot, Tesla Model S, The Future of Employment, The Wisdom of Crowds, Tim Cook: Apple, trade route, uranium enrichment, Wall-E, Watson beat the top human players on Jeopardy!, Wave and Pay, We are Anonymous. We are Legion, web application, Westphalian system, WikiLeaks, Y Combinator, zero day

Sexual orientation, relationship status, schools attended, family tree, lists of friends, age, gender, e-mail addresses, place of birth, news interests, work history, catalogs of favorite things, religion, political affiliation, purchases, photographs, and videos—Facebook is a marketer’s dream. Advertisers know every last intimate detail about a Facebook user’s life and can thus market to him or her with extreme precision based upon the social graph Facebook has generated. Moreover, Facebook created a variety of innovations that allow it to track users across the entirety of the Web, including via its omnipresent Like button. You’ve been trained to click on the cute little blue thumbs-up button to express your support for a particular idea, status update, or photograph; after all, it’s the polite thing to do. Your friends see that you support their message, but what neither of you see is what happens with the data generated with each and every Like—data that are captured, dissected, and sold to marketers and data brokers around the world.


pages: 743 words: 201,651

Free Speech: Ten Principles for a Connected World by Timothy Garton Ash

A Declaration of the Independence of Cyberspace, activist lawyer, Affordable Care Act / Obamacare, Andrew Keen, Apple II, Ayatollah Khomeini, battle of ideas, Berlin Wall, bitcoin, British Empire, Cass Sunstein, Chelsea Manning, citizen journalism, Clapham omnibus, colonial rule, crowdsourcing, David Attenborough, don't be evil, Donald Davies, Douglas Engelbart, Edward Snowden, Etonian, European colonialism, eurozone crisis, failed state, Fall of the Berlin Wall, Ferguson, Missouri, Filter Bubble, financial independence, Firefox, Galaxy Zoo, George Santayana, global village, index card, Internet Archive, invention of movable type, invention of writing, Jaron Lanier, jimmy wales, John Markoff, Julian Assange, Mark Zuckerberg, Marshall McLuhan, mass immigration, megacity, mutually assured destruction, national security letter, Nelson Mandela, Netflix Prize, Nicholas Carr, obamacare, Peace of Westphalia, Peter Thiel, pre–internet, profit motive, RAND corporation, Ray Kurzweil, Ronald Reagan, semantic web, Silicon Valley, Simon Singh, Snapchat, social graph, Stephen Hawking, Steve Jobs, Steve Wozniak, The Death and Life of Great American Cities, The Wisdom of Crowds, Turing test, We are Anonymous. We are Legion, WikiLeaks, World Values Survey, Yom Kippur War

Another reason for the imbalance of power between state and citizen is the development of those very technologies that have given us an unprecedented increase in our ability to communicate with others. Peter Swire, a member of the panel charged by President Obama with preparing what became The NSA Report, argues that the early twenty-first century is a ‘golden age of surveillance’ for security services. He ascribes this to three technological developments in particular: the minutely detailed location data provided by mobile phones, the ‘social graph’ of contacts that we all produce, even if we are not active on social media, and the array of ‘big data’ that has created digital dossiers on us all.16 We should add to Swire’s list of technologies the phenomenon of P2, since private companies actually collect most of the information into which states tap, licitly or illicitly. Commercial surveillance, for the twin purposes of better customer service and maximising profit, feeds state surveillance, justified in the name of security.


The Code: Silicon Valley and the Remaking of America by Margaret O'Mara

"side hustle", A Declaration of the Independence of Cyberspace, accounting loophole / creative accounting, affirmative action, Airbnb, AltaVista, Amazon Web Services, Apple II, Apple's 1984 Super Bowl advert, autonomous vehicles, back-to-the-land, barriers to entry, Ben Horowitz, Berlin Wall, Bob Noyce, Buckminster Fuller, Burning Man, business climate, Byte Shop, California gold rush, carried interest, clean water, cleantech, cloud computing, cognitive dissonance, commoditize, computer age, continuous integration, cuban missile crisis, Danny Hillis, DARPA: Urban Challenge, deindustrialization, different worldview, don't be evil, Donald Trump, Doomsday Clock, Douglas Engelbart, Dynabook, Edward Snowden, El Camino Real, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, Frank Gehry, George Gilder, gig economy, Googley, Hacker Ethic, high net worth, Hush-A-Phone, immigration reform, income inequality, informal economy, information retrieval, invention of movable type, invisible hand, Isaac Newton, Jeff Bezos, Joan Didion, job automation, job-hopping, John Markoff, Julian Assange, Kitchen Debate, knowledge economy, knowledge worker, Lyft, Marc Andreessen, Mark Zuckerberg, market bubble, mass immigration, means of production, mega-rich, Menlo Park, Mikhail Gorbachev, millennium bug, Mitch Kapor, Mother of all demos, move fast and break things, move fast and break things, mutually assured destruction, new economy, Norbert Wiener, old-boy network, pattern recognition, Paul Graham, Paul Terrell, paypal mafia, Peter Thiel, pets.com, pirate software, popular electronics, pre–internet, Ralph Nader, RAND corporation, Richard Florida, ride hailing / ride sharing, risk tolerance, Robert Metcalfe, Ronald Reagan, Sand Hill Road, Second Machine Age, self-driving car, shareholder value, side project, Silicon Valley, Silicon Valley ideology, Silicon Valley startup, skunkworks, Snapchat, social graph, software is eating the world, speech recognition, Steve Ballmer, Steve Jobs, Steve Wozniak, Steven Levy, Stewart Brand, supercomputer in your pocket, technoutopianism, Ted Nelson, the market place, the new new thing, There's no reason for any individual to have a computer in his home - Ken Olsen, Thomas L Friedman, Tim Cook: Apple, transcontinental railway, Uber and Lyft, uber lyft, Unsafe at Any Speed, upwardly mobile, Vannevar Bush, War on Poverty, We wanted flying cars, instead we got 140 characters, Whole Earth Catalog, WikiLeaks, William Shockley: the traitorous eight, Y Combinator, Y2K

“There is so much accidental tourism in great things in life,” Palihapitiya later reflected, and he had hopped on the tour bus at exactly the right time.24 In 2007, Facebook opened up its network to third-party apps, bringing in games and quizzes and other content to its newsfeed, and allowing developers to tap into the treasure trove of knowledge about users’ connections and likes that Facebook called the “social graph.” In 2010, Facebook announced “Open Graph,” which connected a user’s profile and network to the other places she traveled online. It wasn’t just a social network atop the Web anymore. Facebook had remade the Web itself into something, as Zuckerberg put it, “more social, more personalized, and more semantically aware.” The company allowed academic researchers to tap into its troves of information as well, underscoring its made-in-Silicon-Valley belief that freer and more transparent flows of information served the greater good.25 Facebook and its founder were remarkably young and relentlessly future tense, but Zuckerberg had a deepening sense of his place in Valley history as the company’s wealth and influence grew.


pages: 669 words: 210,153

Tools of Titans: The Tactics, Routines, and Habits of Billionaires, Icons, and World-Class Performers by Timothy Ferriss

Airbnb, Alexander Shulgin, artificial general intelligence, asset allocation, Atul Gawande, augmented reality, back-to-the-land, Ben Horowitz, Bernie Madoff, Bertrand Russell: In Praise of Idleness, Black Swan, blue-collar work, Boris Johnson, Buckminster Fuller, business process, Cal Newport, call centre, Charles Lindbergh, Checklist Manifesto, cognitive bias, cognitive dissonance, Colonization of Mars, Columbine, commoditize, correlation does not imply causation, David Brooks, David Graeber, diversification, diversified portfolio, Donald Trump, effective altruism, Elon Musk, fault tolerance, fear of failure, Firefox, follow your passion, future of work, Google X / Alphabet X, Howard Zinn, Hugh Fearnley-Whittingstall, Jeff Bezos, job satisfaction, Johann Wolfgang von Goethe, John Markoff, Kevin Kelly, Kickstarter, Lao Tzu, lateral thinking, life extension, lifelogging, Mahatma Gandhi, Marc Andreessen, Mark Zuckerberg, Mason jar, Menlo Park, Mikhail Gorbachev, MITM: man-in-the-middle, Nelson Mandela, Nicholas Carr, optical character recognition, PageRank, passive income, pattern recognition, Paul Graham, peer-to-peer, Peter H. Diamandis: Planetary Resources, Peter Singer: altruism, Peter Thiel, phenotype, PIHKAL and TIHKAL, post scarcity, post-work, premature optimization, QWERTY keyboard, Ralph Waldo Emerson, Ray Kurzweil, recommendation engine, rent-seeking, Richard Feynman, risk tolerance, Ronald Reagan, selection bias, sharing economy, side project, Silicon Valley, skunkworks, Skype, Snapchat, social graph, software as a service, software is eating the world, stem cell, Stephen Hawking, Steve Jobs, Stewart Brand, superintelligent machines, Tesla Model S, The Wisdom of Crowds, Thomas L Friedman, Wall-E, Washington Consensus, Whole Earth Catalog, Y Combinator, zero-sum game

This gives fans a deeper connection with people they admire but do not know. [TF: Twitter also used a “Top 100” most-followed list early on to pour gasoline on competition.] Syndication of content. Emotional reaction: Users are beginning to use the nomenclature “RT” to indicate a “retweet” (this was common practice before the official retweet feature was developed). This ad hoc feature allows users to syndicate messages beyond their social graph, giving a user’s message increased visibility. The real-time nature of Twitter allows news stories to break faster than traditional media (even at the time, my startup, Digg). In allowing myself to feel these features through the eyes of the users, I can get a sense of the excitement around them. * * * This type of thinking can also be applied to larger industry trends. My colleague and friend David Prager was one of the first owners of the Tesla Model S.