data science

347 results back to index

Doing Data Science: Straight Talk From the Frontline by Cathy O'Neil, Rachel Schutt

Amazon Mechanical Turk, augmented reality, Augustin-Louis Cauchy, barriers to entry, Bayesian statistics, bike sharing, bioinformatics, computer vision, confounding variable, correlation does not imply causation, crowdsourcing, data science, distributed generation, Dunning–Kruger effect, Edward Snowden, Emanuel Derman, fault tolerance, Filter Bubble, finite state, Firefox, game design, Google Glasses, index card, information retrieval, iterative process, John Harrison: Longitude, Khan Academy, Kickstarter, machine translation, Mars Rover, Nate Silver, natural language processing, Netflix Prize, p-value, pattern recognition, performance metric, personalized medicine, pull request, recommendation engine, rent-seeking, selection bias, Silicon Valley, speech recognition, statistical model, stochastic process, tacit knowledge, text mining, the scientific method, The Wisdom of Crowds, Watson beat the top human players on Jeopardy!, X Prize

datafication, Datafication employment opportunities in, Data Science Jobs Facebook and, The Current Landscape (with a Little History) Harvard Business Review, The Current Landscape (with a Little History) history of, The Current Landscape (with a Little History)–Data Science Jobs industry vs. academia in, Getting Past the Hype LinkedIn and, The Current Landscape (with a Little History) meta-definition thought experiment, Thought Experiment: Meta-Definition privacy and, Privacy process of, The Data Science Process–A Data Scientist’s Role in This Process RealDirect case study, Case Study: RealDirect–Sample R code scientific method and, A Data Scientist’s Role in This Process scientists, A Data Science Profile–Thought Experiment: Meta-Definition sociology and, Gabriel Tarde teams, A Data Science Profile Venn diagram of, The Current Landscape (with a Little History) data science competitions, Background: Data Science Competitions Kaggle and, A Single Contestant data scientists, A Data Science Profile–Thought Experiment: Meta-Definition as problem solvers, Being Problem Solvers chief, The Life of a Chief Data Scientist defining, A Data Science Profile ethics of, Being an Ethical Data Scientist–Being an Ethical Data Scientist female, On Being a Female Data Scientist hubris and, Being an Ethical Data Scientist–Being an Ethical Data Scientist in academia, In Academia in industry, In Industry next generation of, What Are Next-Gen Data Scientists?

…

datafication, Datafication employment opportunities in, Data Science Jobs Facebook and, The Current Landscape (with a Little History) Harvard Business Review, The Current Landscape (with a Little History) history of, The Current Landscape (with a Little History)–Data Science Jobs industry vs. academia in, Getting Past the Hype LinkedIn and, The Current Landscape (with a Little History) meta-definition thought experiment, Thought Experiment: Meta-Definition privacy and, Privacy process of, The Data Science Process–A Data Scientist’s Role in This Process RealDirect case study, Case Study: RealDirect–Sample R code scientific method and, A Data Scientist’s Role in This Process scientists, A Data Science Profile–Thought Experiment: Meta-Definition sociology and, Gabriel Tarde teams, A Data Science Profile Venn diagram of, The Current Landscape (with a Little History) data science competitions, Background: Data Science Competitions Kaggle and, A Single Contestant data scientists, A Data Science Profile–Thought Experiment: Meta-Definition as problem solvers, Being Problem Solvers chief, The Life of a Chief Data Scientist defining, A Data Science Profile ethics of, Being an Ethical Data Scientist–Being an Ethical Data Scientist female, On Being a Female Data Scientist hubris and, Being an Ethical Data Scientist–Being an Ethical Data Scientist in academia, In Academia in industry, In Industry next generation of, What Are Next-Gen Data Scientists?–Being Question Askers questioning as, Being Question Askers role of, in data science process, A Data Scientist’s Role in This Process soft skills of, Cultivating Soft Skills data visualization, Data Visualization and Fraud Detection–Data Visualization Exercise at Square, Data Visualization at Square–Data Visualization at Square Before Us is the Salesman’s House (Thorp/Hansen), eBay Transactions and Books–eBay Transactions and Books Cronkite Plaza (Thorp/Rubin/Hansen), Cronkite Plaza distant reading, Franco Moretti fraud and, The Risk Challenge–Detecting suspicious activity using machine learning Hansen, Mark, Data Visualization and Fraud Detection–Goals of These Exhibits history of, Data Visualization History Lives on a Screen (Thorp/Hansen), Project Cascade: Lives on a Screen machine learning and, Data Science and Risk Moveable Type (Rubin/Hansen), New York Times Lobby: Moveable Type–New York Times Lobby: Moveable Type personal data collection thought experiment, Mark’s Thought Experiment Processing programming language, Processing risk and, Data Science and Risk–Ian’s Thought Experiment samples of, A Sample of Data Visualization Projects–Mark’s Data Visualization Projects Shakespeare Machine (Rubin/Hansen), Public Theater Shakespeare Machine sociology and, Gabriel Tarde tutorials for, Data Visualization for the Rest of Us–Data Visualization for the Rest of Us data visualization exercise, Data Visualization Exercise data-generating processes, Statistical Inference DataEDGE, Challenges in features and learning datafication, Why Now?

…

Social scientists also do tend to be good question askers and have other good investigative qualities, so a social scientist who also has the quantitative and programming chops makes a great data scientist. But it’s almost a “historical” (historical is in quotes because 2008 isn’t that long ago) artifact to limit your conception of a data scientist to someone who works only with online user behavior data. There’s another emerging field out there called computational social sciences, which could be thought of as a subset of data science. But we can go back even further. In 2001, William Cleveland wrote a position paper about data science called “Data Science: An action plan to expand the field of statistics.” So data science existed before data scientists? Is this semantics, or does it make sense?

Data Science for Business: What You Need to Know About Data Mining and Data-Analytic Thinking by Foster Provost, Tom Fawcett

Albert Einstein, Amazon Mechanical Turk, Apollo 13, big data - Walmart - Pop Tarts, bioinformatics, business process, call centre, chief data officer, Claude Shannon: information theory, computer vision, conceptual framework, correlation does not imply causation, crowdsourcing, data acquisition, data science, David Brooks, en.wikipedia.org, Erik Brynjolfsson, Gini coefficient, Helicobacter pylori, independent contractor, information retrieval, intangible asset, iterative process, Johann Wolfgang von Goethe, Louis Pasteur, Menlo Park, Nate Silver, Netflix Prize, new economy, p-value, pattern recognition, placebo effect, price discrimination, recommendation engine, Ronald Coase, selection bias, Silicon Valley, Skype, SoftBank, speech recognition, Steve Jobs, supply-chain management, systems thinking, Teledyne, text mining, the long tail, The Signal and the Noise by Nate Silver, Thomas Bayes, transaction costs, WikiLeaks

., A Firm’s Data Science Maturity structure, Machine Learning and Data Mining techniques, Data Science, Engineering, and Data-Driven Decision Making technology vs. theory of, Chemistry Is Not About Test Tubes: Data Science Versus the Work of the Data Scientist–Chemistry Is Not About Test Tubes: Data Science Versus the Work of the Data Scientist understanding, The Ubiquity of Data Opportunities, Data Processing and “Big Data” data science maturity, of firms, A Firm’s Data Science Maturity–A Firm’s Data Science Maturity data scientists academic, Attracting and Nurturing Data Scientists and Their Teams as scientific advisors, Attracting and Nurturing Data Scientists and Their Teams attracting/nurturing, Attracting and Nurturing Data Scientists and Their Teams–Attracting and Nurturing Data Scientists and Their Teams evaluating, Superior Data Scientists–Superior Data Scientists managing, Superior Data Science Management–Superior Data Science Management Data Scientists, LLC, Attracting and Nurturing Data Scientists and Their Teams data sources, Evaluation, Baseline Performance, and Implications for Investments in Data data understanding, Data Understanding–Data Understanding expected value decomposition and, From an Expected Value Decomposition to a Data Science Solution–From an Expected Value Decomposition to a Data Science Solution expected value framework and, The Expected Value Framework: Structuring a More Complicated Business Problem–The Expected Value Framework: Structuring a More Complicated Business Problem data warehousing, Data Warehousing data-analytic thinking, Data-Analytic Thinking–Data-Analytic Thinking and unbalanced classes, Problems with Unbalanced Classes for business strategies, Thinking Data-Analytically, Redux–Thinking Data-Analytically, Redux data-driven business data science vs., Data Processing and “Big Data” understanding, Data Processing and “Big Data” data-driven causal explanations, Data-Driven Causal Explanation and a Viral Marketing Example–Data-Driven Causal Explanation and a Viral Marketing Example data-driven decision-making, Data Science, Engineering, and Data-Driven Decision Making–Data Science, Engineering, and Data-Driven Decision Making benefits, Data Science, Engineering, and Data-Driven Decision Making discoveries, Data Science, Engineering, and Data-Driven Decision Making repetition, Data Science, Engineering, and Data-Driven Decision Making database queries, as analytic technique, Database Querying–Database Querying database tables, Models, Induction, and Prediction dataset entropy, Example: Attribute Selection with Information Gain datasets, Models, Induction, and Prediction analyzing, Introduction to Predictive Modeling: From Correlation to Supervised Segmentation attributes of, Overfitting in Mathematical Functions cross-validation and, From Holdout Evaluation to Cross-Validation limited, From Holdout Evaluation to Cross-Validation Davis, Miles, Example: Jazz Musicians, Example: Jazz Musicians Deanston single malt scotch, Understanding the Results of Clustering decision boundaries, Visualizing Segmentations, Classification via Mathematical Functions decision lines, Visualizing Segmentations decision nodes, Supervised Segmentation with Tree-Structured Models decision stumps, Evaluation, Baseline Performance, and Implications for Investments in Data decision surfaces, Visualizing Segmentations decision trees, Supervised Segmentation with Tree-Structured Models decision-making, automatic, Data Science, Engineering, and Data-Driven Decision Making deduction, induction vs., Models, Induction, and Prediction Dell, Data preparation, Achieving Competitive Advantage with Data Science demand, local, Example: Hurricane Frances dendrograms, Hierarchical Clustering, Hierarchical Clustering dependent variables, Models, Induction, and Prediction descriptive attributes, Data Mining and Data Science, Revisited descriptive modeling, Models, Induction, and Prediction Dictionary of Distances (Deza & Deza), * Other Distance Functions differential descriptions, * Using Supervised Learning to Generate Cluster Descriptions Digital 100 companies, Data-Analytic Thinking Dillman, Linda, Data Science, Engineering, and Data-Driven Decision Making dimensionality, of nearest-neighbor reasoning, Dimensionality and domain knowledge–Dimensionality and domain knowledge directed marketing example, Targeting the Best Prospects for a Charity Mailing–A Brief Digression on Selection Bias discoveries, Data Science, Engineering, and Data-Driven Decision Making discrete (binary) classifiers, ROC Graphs and Curves discrete classifiers, ROC Graphs and Curves discretized numeric variables, Selecting Informative Attributes discriminants, linear, Linear Discriminant Functions discriminative modeling methods, generative vs., Summary disorder, measuring, Selecting Informative Attributes display advertising, Example: Targeting Online Consumers With Advertisements distance functions, for nearest-neighbor reasoning, * Other Distance Functions–* Other Distance Functions distance, measuring, Similarity and Distance distribution Gaussian, Regression via Mathematical Functions Normal, Regression via Mathematical Functions distribution of properties, Selecting Informative Attributes Doctor Who (television show), Example: Evidence Lifts from Facebook “Likes” document (term), Representation domain knowledge data mining processes and, Dimensionality and domain knowledge nearest-neighbor reasoning and, Dimensionality and domain knowledge–Dimensionality and domain knowledge domain knowledge validation, Associations Among Facebook Likes domains, in association discovery, Associations Among Facebook Likes Dotcom Boom, Results, Formidable Historical Advantage double counting, Costs and benefits draws, statistical, * Logistic Regression: Some Technical Details E edit distance, * Other Distance Functions, * Other Distance Functions Einstein, Albert, Conclusion Elder Research, Attracting and Nurturing Data Scientists and Their Teams Ellington, Duke, Example: Jazz Musicians, Example: Jazz Musicians email, Why Text Is Important engineering, Chemistry Is Not About Test Tubes: Data Science Versus the Work of the Data Scientist, Business Understanding engineering problems, business problems vs., Other Data Science Tasks and Techniques ensemble method, Bias, Variance, and Ensemble Methods–Bias, Variance, and Ensemble Methods entropy, Selecting Informative Attributes–Selecting Informative Attributes, Selecting Informative Attributes, Example: Attribute Selection with Information Gain, Summary and Inverse Document Frequency, * The Relationship of IDF to Entropy change in, Selecting Informative Attributes equation for, Selecting Informative Attributes graphs, Example: Attribute Selection with Information Gain equations cosine distance, * Other Distance Functions entropy, Selecting Informative Attributes Euclidean distance, Similarity and Distance general linear model, Linear Discriminant Functions information gain (IG), Selecting Informative Attributes Jaccard distance, * Other Distance Functions L2 norm, * Other Distance Functions log-odds linear function, * Logistic Regression: Some Technical Details logistic function, * Logistic Regression: Some Technical Details majority scoring function, * Combining Functions: Calculating Scores from Neighbors majority vote classification, * Combining Functions: Calculating Scores from Neighbors Manhattan distance, * Other Distance Functions similarity-moderated classification, * Combining Functions: Calculating Scores from Neighbors similarity-moderated regression, * Combining Functions: Calculating Scores from Neighbors similarity-moderated scoring, * Combining Functions: Calculating Scores from Neighbors error costs, ROC Graphs and Curves error rates, Plain Accuracy and Its Problems, Error rates errors absolute, Regression via Mathematical Functions computing, Regression via Mathematical Functions false negative vs. false positive, Evaluating Classifiers squared, Regression via Mathematical Functions estimating generalization performance, From Holdout Evaluation to Cross-Validation estimation, frequency based, Probability Estimation ethics of data mining, Privacy, Ethics, and Mining Data About Individuals–Privacy, Ethics, and Mining Data About Individuals Euclid, Similarity and Distance Euclidean distance, Similarity and Distance evaluating models, Decision Analytic Thinking I: What Is a Good Model?

…

., The Fundamental Concepts of Data Science unique context of, What Data Can’t Do: Humans in the Loop, Revisited using expected values to provide framework for, The Expected Value Framework: Decomposing the Business Problem and Recomposing the Solution Pieces–The Expected Value Framework: Decomposing the Business Problem and Recomposing the Solution Pieces business strategy, Data Science and Business Strategy–A Firm’s Data Science Maturity accepting creative ideas, Be Ready to Accept Creative Ideas from Any Source case studies, examining, Examine Data Science Case Studies competitive advantages, Achieving Competitive Advantage with Data Science–Achieving Competitive Advantage with Data Science, Sustaining Competitive Advantage with Data Science–Superior Data Science Management data scientists, evaluating, Superior Data Scientists–Superior Data Scientists evaluating proposals, Be Ready to Evaluate Proposals for Data Science Projects–Flaws in the Big Red Proposal historical advantages and, Formidable Historical Advantage intangible collateral assets and, Unique Intangible Collateral Assets intellectual property and, Unique Intellectual Property managing data scientists effectively, Superior Data Science Management–Superior Data Science Management maturity of the data science, A Firm’s Data Science Maturity–A Firm’s Data Science Maturity thinking data-analytically for, Thinking Data-Analytically, Redux–Thinking Data-Analytically, Redux C Caesars Entertainment, Data and Data Science Capability as a Strategic Asset call center example, Profiling: Finding Typical Behavior–Profiling: Finding Typical Behavior Capability Maturity Model, A Firm’s Data Science Maturity Capital One, Data and Data Science Capability as a Strategic Asset, From an Expected Value Decomposition to a Data Science Solution Case-Based Reasoning, How Many Neighbors and How Much Influence?

…

This is facilitated tremendously by strong and deep professional contacts. Data scientists call on each other to help in steering them to the right solutions. The better a professional network is, the better will be the solution. And, the best data scientists have the best connections. Superior Data Science Management Possibly even more critical to success for data science in business is having good management of the data science team. Good data science managers are especially hard to find. They need to understand the fundamentals of data science well, possibly even being competent data scientists themselves. Good data science managers also must possess a set of other abilities that are rare in a single individual: They need to truly understand and appreciate the needs of the business.

Big Data at Work: Dispelling the Myths, Uncovering the Opportunities by Thomas H. Davenport

Automated Insights, autonomous vehicles, bioinformatics, business intelligence, business process, call centre, chief data officer, cloud computing, commoditize, data acquisition, data science, disruptive innovation, Edward Snowden, Erik Brynjolfsson, intermodal, Internet of things, Jeff Bezos, knowledge worker, lifelogging, Mark Zuckerberg, move fast and break things, Narrative Science, natural language processing, Netflix Prize, New Journalism, recommendation engine, RFID, self-driving car, sentiment analysis, Silicon Valley, smart grid, smart meter, social graph, sorting algorithm, statistical model, Tesla Model S, text mining, Thomas Davenport, three-martini lunch

But if you’re interviewing one from another industry, make sure that the candidate shows interest and demonstrated business problem-solving ability in the industry from which he or she comes. Horizontal versus Vertical Data Scientists There are, of course, many types of data scientists. One way to ­characterize an important set of differences between types has been coined by Vincent Granville, who operates Data Science Central, Chapter_04.indd 97 03/12/13 12:00 PM 98 big data @ work a social n ­ etwork for data scientists like himself. In a blog post (with some ­analytical jargon you can toss around at cocktail parties), he described the difference between vertical and horizontal data scientists: • Vertical data scientists have very deep knowledge in some ­narrow field.

…

Where Do You Get Data Scientists? Data Scientists from Universities I noted earlier that data scientists today often have advanced degrees in science, but that won’t always be the most efficient way to procure the ­necessary skills. How soon will there be more direct educational paths to Chapter_04.indd 101 03/12/13 12:00 PM 102 big data @ work data science? Well, as I write I believe there is no university that has yet issued a degree in data science. But there are: (a) a growing n ­ umber of courses in the field and (b) a growing number of institutions that are ­planning data science degree programs.

…

And as I hinted in chapter 3, firms such as ­Accenture, Deloitte, and IBM have begun to hire and train data scientists in larger numbers. Predominantly offshore firms such as Mu Sigma, a “math factory” with thousands of quants as employees, are also hiring data scientists in considerable numbers. One data scientist has come up with a creative approach to training new data scientists. The Insight Data Science Fellows Program, started by Jake Klamka (whose academic background is high-energy physics), takes scientists for six weeks and teaches them the skills to be a data scientist. The program includes mentoring by local companies with big data challenges (e.g., Facebook, Twitter, Google, ­LinkedIn).

Succeeding With AI: How to Make AI Work for Your Business by Veljko Krunic

AI winter, Albert Einstein, algorithmic trading, AlphaGo, Amazon Web Services, anti-fragile, anti-pattern, artificial general intelligence, autonomous vehicles, Bayesian statistics, bioinformatics, Black Swan, Boeing 737 MAX, business process, cloud computing, commoditize, computer vision, correlation coefficient, data is the new oil, data science, deep learning, DeepMind, en.wikipedia.org, fail fast, Gini coefficient, high net worth, information retrieval, Internet of things, iterative process, job automation, Lean Startup, license plate recognition, minimum viable product, natural language processing, recommendation engine, self-driving car, sentiment analysis, Silicon Valley, six sigma, smart cities, speech recognition, statistical model, strong AI, tail risk, The Design of Experiments, the scientific method, web application, zero-sum game

Closely related fields that are sometimes considered part of data science include bioinformatics and quantitative analysis. While AI and data science closely overlap, they aren’t identical, because AI includes fields such as robotics, which are traditionally not considered part of data science. Harris, Murphy, and Vaisman’s book [66] provides a good summary of the state of data science before the advancement of deep learning. Data scientist—A practitioner of the field of data science. Many sources (including this book) classify AI practitioners as data scientists. Database administrator (DBA)—A professional responsible for the maintenance of a database. Most commonly, a DBA would be responsible for maintaining a RDBMS-based database.

…

As a manager, you should look for two things when hiring data scientists for your team. You should look for a candidate who has skills in the core domain that your initial AI project is likely to use, but you also need them to have a demonstrated ability to learn new skills. Chances are good that, along the way, your data scientist will need to learn many new methods. When hiring senior data science team members, don’t just look for a strong background in one set of AI methods. Senior data scientists should have a history of solving concrete problems using a diverse set of methods. Data science is a team sport. To completely cover all of the knowledge that’s part of data science, you need a whole team, so you must assemble a team with complementary skillsets.

…

They were the best decisions you could have made based on what you knew then. Types of data scientists to hire Leading experts focused on a narrow class of algorithms are worth their weight in gold if they know how to improve the performance of an algorithm by 0.1% that’s bringing $1 billion per year to your organization. However, if your question is, “What can AI do to help my business?” you’re probably better off with a data scientist who has a command of the wide range of data science methods. A data scientist with that profile has the best chance of finding a use case in which the profit margin would be large and you don’t need to get that last 0.1% improvement for the use case to be viable.

Data Science from Scratch: First Principles with Python by Joel Grus

backpropagation, confounding variable, correlation does not imply causation, data science, deep learning, Hacker News, higher-order functions, natural language processing, Netflix Prize, p-value, Paul Graham, recommendation engine, SpamAssassin, statistical model

If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-491-90142-7 [LSI] Preface Data Science Data scientist has been called “the sexiest job of the 21st century,” presumably by someone who has never visited a fire station. Nonetheless, data science is a hot and growing field, and it doesn’t take a great deal of sleuthing to find analysts breathlessly prognosticating that over the next 10 years, we’ll need billions and billions more data scientists than we currently have. But what is data science? After all, we can’t produce data scientists if we don’t know what data science is. According to a Venn diagram that is somewhat famous in the industry, data science lies at the intersection of: Hacking skills Math and statistics knowledge Substantive expertise Although I originally intended to write a book covering all three, I quickly realized that a thorough treatment of “substantive expertise” would require tens of thousands of pages.

…

From Scratch There are lots and lots of data science libraries, frameworks, modules, and toolkits that efficiently implement the most common (as well as the least common) data science algorithms and techniques. If you become a data scientist, you will become intimately familiar with NumPy, with scikit-learn, with pandas, and with a panoply of other libraries. They are great for doing data science. But they are also a good way to start doing data science without actually understanding data science. In this book, we will be approaching data science from scratch. That means we’ll be building tools and implementing algorithms by hand in order to better understand them.

…

Now, before you start feeling too jaded: some data scientists also occasionally use their skills for good — using data to make government more effective, to help the homeless, and to improve public health. But it certainly won’t hurt your career if you like figuring out the best way to get people to click on advertisements. Motivating Hypothetical: DataSciencester Congratulations! You’ve just been hired to lead the data science efforts at DataSciencester, the social network for data scientists. Despite being for data scientists, DataSciencester has never actually invested in building its own data science practice. (In fairness, DataSciencester has never really invested in building its product either.)

The AI-First Company by Ash Fontana

23andMe, Amazon Mechanical Turk, Amazon Web Services, autonomous vehicles, barriers to entry, blockchain, business intelligence, business process, business process outsourcing, call centre, Charles Babbage, chief data officer, Clayton Christensen, cloud computing, combinatorial explosion, computer vision, crowdsourcing, data acquisition, data science, deep learning, DevOps, en.wikipedia.org, Geoffrey Hinton, independent contractor, industrial robot, inventory management, John Conway, knowledge economy, Kubernetes, Lean Startup, machine readable, minimum viable product, natural language processing, Network effects, optical character recognition, Pareto efficiency, performance metric, price discrimination, recommendation engine, Ronald Coase, Salesforce, single source of truth, software as a service, source of truth, speech recognition, the scientific method, transaction costs, vertical integration, yield management

Before creating a bunch of chatter between equations, have a single conversation with one equation to see if it answers customers’ questions. We’re not here to build a data science consulting firm, but DLEs start with data science. Most AI models are based on statistical methods. Starting with statistics allows for a smooth transition into AI when there’s enough time and money from customers to build it. Starting Small: Data Science Starting with a data scientist solving a well-defined problem saves time and money when compared to starting with a team big enough to solve an amorphous problem with machine learning. Dedicate a data scientist to serve as a consultant to customers and provide personalized, data-driven answers to a single question in order to demonstrate return on investment (ROI).

…

Linking databases, cleaning data, creating data pipelines, building features, and designing interfaces is a lot of work, but avoidable work. There’s also evidence that starting with data science works on Kaggle, where the largest community of data scientists and ML engineers compete to win prizes for solving problems. The summary is that data science methods get to the Pareto optimal solution (achieving 80 percent of the optimal solution for 20 percent of the work). Often, only the last 20 percent requires data science. Specifically, data science methods such as ensembles of decision trees—whether random forest or gradient boosted—combined with manual feature engineering win most of the competitions on structured data, and neural networks win most of the competitions on unstructured data.

…

hiring sequence Data analyst Business Low Yes First Data scientist Statistics Low Partially Second Data engineer Databases Medium Yes Third Machine learning engineer Computer science Medium No Fourth Data product manager Product management Medium No Fifth Data infrastructure engineer Distributed systems High Partially Sixth Machine learning researcher Machine learning High Maybe Seventh WHERE TO FIND THEM Starting with statistics means hiring analysts and data scientists before engineers and ML researchers. Essentially, by decoupling data science and software engineering, hiring can focus on data scientists without software engineering experience, thus broadening the pool of candidates to include every discipline in which manipulating data is part of the research process. One can find analysts and data scientists in the fields of economics, econometrics, accounting, actuarial science, biology, biostatistics, geology, geostatistics, epidemiology, demographics, engineering, and physics because these areas require high levels of mathematics and statistics.

Data-Ism: The Revolution Transforming Decision Making, Consumer Behavior, and Almost Everything Else by Steve Lohr

"World Economic Forum" Davos, 23andMe, Abraham Maslow, Affordable Care Act / Obamacare, Albert Einstein, Alvin Toffler, Bear Stearns, behavioural economics, big data - Walmart - Pop Tarts, bioinformatics, business cycle, business intelligence, call centre, Carl Icahn, classic study, cloud computing, computer age, conceptual framework, Credit Default Swap, crowdsourcing, Daniel Kahneman / Amos Tversky, Danny Hillis, data is the new oil, data science, David Brooks, driverless car, East Village, Edward Snowden, Emanuel Derman, Erik Brynjolfsson, everywhere but in the productivity statistics, financial engineering, Frederick Winslow Taylor, Future Shock, Google Glasses, Ida Tarbell, impulse control, income inequality, indoor plumbing, industrial robot, informal economy, Internet of things, invention of writing, Johannes Kepler, John Markoff, John von Neumann, lifelogging, machine translation, Mark Zuckerberg, market bubble, meta-analysis, money market fund, natural language processing, obamacare, pattern recognition, payday loans, personalized medicine, planned obsolescence, precision agriculture, pre–internet, Productivity paradox, RAND corporation, rising living standards, Robert Gordon, Robert Solow, Salesforce, scientific management, Second Machine Age, self-driving car, Silicon Valley, Silicon Valley startup, SimCity, six sigma, skunkworks, speech recognition, statistical model, Steve Jobs, Steven Levy, The Design of Experiments, the scientific method, Thomas Kuhn: the structure of scientific revolutions, Tony Fadell, unbanked and underbanked, underbanked, Von Neumann architecture, Watson beat the top human players on Jeopardy!, yottabyte

So, Hammerbacher says, “We decided to mush those two titles together and call them data scientists.” At first, a few PhDs resisted, viewing the change as a loss of title prestige. “But ultimately everyone embraced it, and it took on a life of its own,” he observes. And to him, it seemed natural. “Data science is what we did.” The origins of data science reach back half a century or more. Hammerbacher’s choice of terms wasn’t mere happenstance. As soon as he accepted the job at Facebook, Hammerbacher began poring through technical papers and books that provided clues to the evolution of data science. In the spring of 2012, he taught a course in data science at the University of California at Berkeley.

…

Alex Pentland, a computational social scientist at the Massachusetts Institute of Technology Media Lab, sees the promise of “a transition on a par with the invention of writing or the Internet.” The ranks of data scientists—people who wield their math and computing smarts to make sense of data—are modest compared to the workforce as a whole, but they loom large. Data science is hailed as the field of the future. Universities are rushing to establish data science centers, institutes, and courses, and companies are scrambling to hire data scientists. There is a trend-chasing side to the current data frenzy that invites ridicule. But it is hard to argue the direction. Jeffrey Hammerbacher was always a numbers kind of guy.

…

To explain, I think of a conversation with Claudia Perlich, the chief scientist of Dstillery, a data-science start-up in New York that specializes in ad targeting. Perlich is a former research scientist at IBM, a winner of prestigious data science contests, and a lecturer at New York University’s Stern School of Business. When I ask why she is using her skills to deliver ads, Perlich replies that digital marketing is a large, real-world testing ground where practitioners in a young field can safely learn valuable lessons. The online advertising marketplace, she says, is “a wonderful place for data scientists to experiment now. What happens if my algorithm is wrong?

Bad Data Handbook by Q. Ethan McCallum

Amazon Mechanical Turk, asset allocation, barriers to entry, Benoit Mandelbrot, business intelligence, cellular automata, chief data officer, Chuck Templeton: OpenTable:, cloud computing, cognitive dissonance, combinatorial explosion, commoditize, conceptual framework, data science, database schema, DevOps, en.wikipedia.org, Firefox, Flash crash, functional programming, Gini coefficient, hype cycle, illegal immigration, iterative process, labor-force participation, loose coupling, machine readable, natural language processing, Netflix Prize, One Laptop per Child (OLPC), power law, quantitative trading / quantitative finance, recommendation engine, selection bias, sentiment analysis, SQL injection, statistical model, supply-chain management, survivorship bias, text mining, too big to fail, web application

While they are willing and able to work on many tasks across the data science process, from munging and modeling to visualizing and presenting, it’s quite rare to find talent with extensive experience in all aspects of data science. Organizations and managers would do well to adjust their expectations accordingly. A successful data science function is made up not by one person, but at a minimum two or three individuals whose broad skills have much overlap while their unique expertise does not. Where Do Data Scientists Live Within the Organization? Finding a place for data scientists can be a bit tricky. Sometimes you’ll find them living within an engineering organization, sometimes within a product organization, sometimes within a research organization, and other times they live under some other umbrella or on their own.

…

His skills as a programmer began while assisting with the development Sahana Disaster Management System, were refined helping Sugar Labs, the software which runs the One Laptop Per Child XO. Tim has recently moved into the escience field, where he works to support the research community’s uptake of technology. Marck Vaisman is a data scientist and claims he’s been one before the term was en vogue. He is also a consultant, entrepreneur, master munger, and hacker. Marck is the principal data scientist at DataXtract, LLC where he helps clients ranging from startups to Fortune 500 firms with all kinds of data science projects. His professional experience spans the management consulting, telecommunications, Internet, and technology industries. He is the co-founder of Data Community DC, an organization focused on building the Washington DC area data community and promoting data and statistical sciences by running Meetup events (including Data Science DC and R Users DC) and other initiatives.

…

Spillover can happen for any number of reasons, including inadequate accuracy in the partitioning scheme. While you may not be able to completely eliminate spillover, you can at least be aware of it. Don’t expect that the data is partitioned perfectly. Thou Shalt Provide Your Data Scientists with a Single Tool for All Tasks There is no single tool that allows you to perform all of your data science tasks. Many different tools exist, and each tool has a specific purpose. In order to be successful, data scientists should have access to the tools they need and also the ability to configure these tools as needed—at least in a research and development (R&D) environment—without having to jump through hoops to do their work.

Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are by Seth Stephens-Davidowitz

affirmative action, AltaVista, Amazon Mechanical Turk, Asian financial crisis, Bernie Sanders, big data - Walmart - Pop Tarts, Black Lives Matter, Cass Sunstein, computer vision, content marketing, correlation does not imply causation, crowdsourcing, Daniel Kahneman / Amos Tversky, data science, desegregation, Donald Trump, Edward Glaeser, Filter Bubble, game design, happiness index / gross national happiness, income inequality, Jeff Bezos, Jeff Seder, John Snow's cholera map, longitudinal study, Mark Zuckerberg, Nate Silver, Nick Bostrom, peer-to-peer lending, Peter Thiel, price discrimination, quantitative hedge fund, Ronald Reagan, Rosa Parks, sentiment analysis, Silicon Valley, statistical model, Steve Jobs, Steven Levy, Steven Pinker, TaskRabbit, The Signal and the Noise by Nate Silver, working poor

In other words, she spotted patterns and predicted how one variable will affect another. Grandma is a data scientist. You are a data scientist, too. When you were a kid, you noticed that when you cried, your mom gave you attention. That is data science. When you reached adulthood, you noticed that if you complain too much, people want to hang out with you less. That is data science, too. When people hang out with you less, you noticed, you are less happy. When you are less happy, you are less friendly. When you are less friendly, people want to hang out with you even less. Data science. Data science. Data science. Because data science is so natural, the best Big Data studies, I have found, can be understood by just about any smart person.

…

In other words, the Columbia and Microsoft researchers wrote a groundbreaking study by utilizing the natural, obvious methodology that everybody uses to make health diagnoses. But wait. Let’s slow down here. If the methodology of the best data science is frequently natural and intuitive, as I claim, this raises a fundamental question about the value of Big Data. If humans are naturally data scientists, if data science is intuitive, why do we need computers and statistical software? Why do we need the Kolmogorov-Smirnov test? Can’t we just use our gut? Can’t we do it like Grandma does, like nurses and doctors do? This gets to an argument intensified after the release of Malcolm Gladwell’s bestselling book Blink, which extols the magic of people’s gut instincts.

…

We are often wrong, in other words, about how the world works when we rely just on what we hear or personally experience. While the methodology of good data science is often intuitive, the results are frequently counterintuitive. Data science takes a natural and intuitive human process—spotting patterns and making sense of them—and injects it with steroids, potentially showing us that the world works in a completely different way from how we thought it did. That’s what happened when I studied the predictors of basketball success. When I was a little boy, I had one dream and one dream only: I wanted to grow up to be an economist and data scientist. No. I’m just kidding. I wanted desperately to be a professional basketball player, to follow in the footsteps of my hero, Patrick Ewing, all-star center for the New York Knicks.

AIQ: How People and Machines Are Smarter Together by Nick Polson, James Scott

Abraham Wald, Air France Flight 447, Albert Einstein, algorithmic bias, Amazon Web Services, Atul Gawande, autonomous vehicles, availability heuristic, basic income, Bayesian statistics, Big Tech, Black Lives Matter, Bletchley Park, business cycle, Cepheid variable, Checklist Manifesto, cloud computing, combinatorial explosion, computer age, computer vision, Daniel Kahneman / Amos Tversky, data science, deep learning, DeepMind, Donald Trump, Douglas Hofstadter, Edward Charles Pickering, Elon Musk, epigenetics, fake news, Flash crash, Grace Hopper, Gödel, Escher, Bach, Hans Moravec, Harvard Computers: women astronomers, Higgs boson, index fund, information security, Isaac Newton, John von Neumann, late fees, low earth orbit, Lyft, machine translation, Magellanic Cloud, mass incarceration, Moneyball by Michael Lewis explains big data, Moravec's paradox, more computing power than Apollo, natural language processing, Netflix Prize, North Sea oil, Northpointe / Correctional Offender Management Profiling for Alternative Sanctions, p-value, pattern recognition, Pierre-Simon Laplace, ransomware, recommendation engine, Ronald Reagan, Salesforce, self-driving car, sentiment analysis, side project, Silicon Valley, Skype, smart cities, speech recognition, statistical model, survivorship bias, systems thinking, the scientific method, Thomas Bayes, Uber for X, uber lyft, universal basic income, Watson beat the top human players on Jeopardy!, young professional

It turns out that when she wasn’t caring for soldiers, Nightingale was also a skilled data scientist who successfully convinced hospitals that they could improve health care using statistics. In fact, no other data scientist in history can claim to have saved so many lives as Florence Nightingale. In 1859, in honor of these achievements, she became the first woman ever elected to the U.K.’s Royal Statistical Society. Nightingale’s path to unlocking the power of health-care data offers three distinct lessons for today. First, it illustrates the kind of institutional commitment necessary for a data-science revolution to take hold in a given field.

…

Her ideas formed a clear model for the international system of disease classification used today, which serves as the bedrock for all of modern epidemiology and medical data science.37 Preventable Mischiefs in the Age of AI Nightingale’s three legacies all have clear parallels today. They also raise some sharp questions. She spoke of the “foul air and preventable mischiefs” that killed the soldiers of the Crimea, and while the air in modern hospitals may be less foul, there are still mischiefs aplenty. One big question is how to staff and train a modern health-care team. After Nightingale, no hospital could function without nurses. When will the same be true of data scientists and experts in artificial intelligence, who now play almost no day-to-day role in health care?

…

Data Sharing That bring us to another big question: Will data-science teams get access to the data they’ll need to improve the existing AI systems and build new ones? If you work for a single hospital, you might have access to thousands of patient records. But wouldn’t millions of records from lots of hospitals be much better? After all, a big reason that tech firms like Google and Facebook have such good AI is the sheer scale of their data sets. There are surely millions of clinical histories of kidney disease scattered across the medical databases of the world. In principle, these could be brought together, and teams of data scientists could be hired to analyze them using cutting-edge AI tools, in a way that still ensured patient privacy.

Applied Artificial Intelligence: A Handbook for Business Leaders by Mariya Yao, Adelyn Zhou, Marlene Jia

Airbnb, algorithmic bias, AlphaGo, Amazon Web Services, artificial general intelligence, autonomous vehicles, backpropagation, business intelligence, business process, call centre, chief data officer, cognitive load, computer vision, conceptual framework, data science, deep learning, DeepMind, en.wikipedia.org, fake news, future of work, Geoffrey Hinton, industrial robot, information security, Internet of things, iterative process, Jeff Bezos, job automation, machine translation, Marc Andreessen, natural language processing, new economy, OpenAI, pattern recognition, performance metric, price discrimination, randomized controlled trial, recommendation engine, robotic process automation, Salesforce, self-driving car, sentiment analysis, Silicon Valley, single source of truth, skunkworks, software is eating the world, source of truth, sparse data, speech recognition, statistical model, strong AI, subscription business, technological singularity, The future is already here

These specialized engineers deploy models, manage infrastructure, and run operations related to machine learning projects. They are assisted by data scientists and data engineers to manage databases and build the data infrastructure necessary to support the products and services used by their customers. Data Scientists Data scientists typically work in an offline setting and do not deal directly with the production experience, which is what the end user would see. Data scientists collect data, spend most of their time cleaning it, and the rest of their time looking for patterns in the data and building predictive models. They often have degrees in statistics, data science, or a related discipline. Alternately, many have programming backgrounds and hold degrees in computer science, math, or physics.

…

Recruit from Specialized Training Programs To meet the rising demand for machine learning talent, education programs have emerged to train junior talent and help them find job placements. Abhi Jha, Director of Advanced Analytics at McKesson, initially hired data science students from Galvanize, a technical skills training provider. “We’ve had a lot of success hiring from career fairs that Galvanize organizes, where we present the unique challenges our company tackles in healthcare,” he adds.(57) Experienced Scientists and Researchers Hiring experienced data scientists and machine learning researchers requires a different approach. For these positions, employers typically look for a doctorate or extensive experience in machine learning, statistical modeling, or related fields.

…

Understand Different Job Titles Many companies struggle just to understand what “artificial intelligence” is, much less the myriad of titles, roles, skills, and technologies used to describe a prospective hire. Titles and descriptions vary from company to company, and terms are not well-standardized in the industry. However, most of the roles you encounter will resemble the following: Data Science Team Manager A data science team manager understands how best to deploy the expertise of his team in order to maximize their productivity on a project. This manager should have sufficient technical knowledge to understand what his team members are doing and how best to support them; at the same time, this manager must also have good communications skills in order to liaise with the leadership or non-technical units.

Thinking with Data by Max Shron

business intelligence, Carmen Reinhart, confounding variable, correlation does not imply causation, data science, Growth in a Time of Debt, iterative process, Kenneth Rogoff, randomized controlled trial, Richard Feynman, statistical model, The Design of Experiments, the scientific method

Thinking with Data Max Shron Praise for Thinking with Data "Thinking with Data gets to the essence of the process, and guides data scientists in answering that most important question—what’s the problem we’re really trying to solve?” — Hilary Mason Data Scientist in Residence at Accel Partners; co-founder of the DataGotham Conference “Thinking with Data does a wonderful job of reminding data scientists to look past technical issues and to focus on making an impact on the broad business objectives of their employers and clients. It’s a useful supplement to a data science curriculum that is largely focused on the technical machinery of statistics and computer science

…

Statistics as a whole is concerned with generalizing from old to new data; causal analysis is concerned with generalizing from old to new scenarios where we have deliberately altered something. Generally speaking, because data science as a field is primarily concerned with generalizing knowledge only in highly specific domains (such as for one company, one service, or one type of product), it is able to sidestep many of the issues that snarl causal analysis in more scientific domains. As of today, data scientists do little work building theories intended to capture causal relationships in entirely new scenarios. For those that do, especially if their subject matter concerns human behavior, a more thorough grounding in topics such as construct validity and quasi-experimental design is highly recommended.[10] Defining Causality Different schools of thought have defined causality differently, but a particularly simple interpretation, suitable to many of the problems that are solvable with data, is the alternate universe perspective.

…

The fields of design, argument studies, critical thinking, national intelligence, problem-solving heuristics, education theory, program evaluation, various parts of the humanities—each of them have insights that data science can learn from. Data science is already a field of bricolage. Swaths of engineering, statistics, machine learning, and graphic communication are already fundamental parts of the data science canon. They are necessary, but they are not sufficient. If we look further afield and incorporate ideas from the “softer” intellectual disciplines, we can make data science successful and help it be more than just this decade’s fad. A focus on why rather than how already pervades the work of the best data professionals.

Targeted: The Cambridge Analytica Whistleblower's Inside Story of How Big Data, Trump, and Facebook Broke Democracy and How It Can Happen Again by Brittany Kaiser

"World Economic Forum" Davos, Albert Einstein, Amazon Mechanical Turk, Asian financial crisis, Bernie Sanders, Big Tech, bitcoin, blockchain, Boris Johnson, Brexit referendum, Burning Man, call centre, Cambridge Analytica, Carl Icahn, centre right, Chelsea Manning, clean water, cognitive dissonance, crony capitalism, dark pattern, data science, disinformation, Dominic Cummings, Donald Trump, Edward Snowden, Etonian, fake news, haute couture, illegal immigration, Julian Assange, Mark Zuckerberg, Menlo Park, Nelson Mandela, off grid, open borders, public intellectual, Renaissance Technologies, Robert Mercer, rolodex, Russian election interference, sentiment analysis, Sheryl Sandberg, Silicon Valley, Silicon Valley startup, Skype, Snapchat, statistical model, Steve Bannon, subprime mortgage crisis, TED Talk, the High Line, the scientific method, WeWork, WikiLeaks, you are the product, young professional

The Mercers’ generosity to conservative causes is well known, but Bob saw in Alexander a marriage between his love for data science and his political motivations. Alexander recalled Bob’s reaction as something like “How much do you want, and where should I send it?” From there, Steve and Bekah and Bob formed the triumvirate that was the board of directors of the new company known as Cambridge Analytica, with Alexander Nix at the helm. Alexander had already hired some data scientists by then, but he went on to hire more, and he began to instruct SCL Group employees to split their time between international work and building the U.S. business. The data scientists began to purchase as much data as they could get their hands on, and within months, Cambridge Analytica had taken off.

…

That’s because San Antonio was home to Brad Parscale, of Giles-Parscale. Parscale had been a longtime website designer for Trump, and Trump had picked him to run his digital operations. The problem was that Parscale had no data science or data-driven communications experience, so Bekah knew that Trump needed Cambridge. When the early Cambridge Analytica team (which consisted of Matt Oczkowski, Molly Schweickert, and a handful of data scientists) arrived on the scene in San Antonio in June, they found Brad and the Trump campaign’s digital operations in an alarming state of disarray. Oczkowski—“Oz” for short—wrote to me on June 17, when I asked him a question about a commercial client, saying he had no time to help me, as he would need all his energy for working with Brad and getting their analytics up and running.

…

And for years, the SCL Group, Cambridge Analytica’s parent company, had been identifying and sorting people using the most sophisticated method in behavioral psychology, which gave it the capability of turning what was otherwise just a mountain of information about the American populace into a gold mine. Nix told us about his in-house army of data scientists and psychologists who had learned precisely how to know whom they wanted to message, what messaging to send them, and exactly where to reach them. He had hired the most brilliant data scientists in the world, people who could laser in on individuals wherever they were to be found (on their cell phones, computers, tablets, on television) and through any kind of medium you could imagine (from audio to social media), using “microtargeting.”

Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die by Eric Siegel

Alan Greenspan, Albert Einstein, algorithmic trading, Amazon Mechanical Turk, Apollo 11, Apple's 1984 Super Bowl advert, backtesting, Black Swan, book scanning, bounce rate, business intelligence, business process, butter production in bangladesh, call centre, Charles Lindbergh, commoditize, computer age, conceptual framework, correlation does not imply causation, crowdsourcing, dark matter, data is the new oil, data science, driverless car, en.wikipedia.org, Erik Brynjolfsson, Everything should be made as simple as possible, experimental subject, Google Glasses, happiness index / gross national happiness, information security, job satisfaction, Johann Wolfgang von Goethe, lifelogging, machine readable, Machine translation of "The spirit is willing, but the flesh is weak." to Russian and back, mass immigration, Moneyball by Michael Lewis explains big data, Nate Silver, natural language processing, Netflix Prize, Network effects, Norbert Wiener, personalized medicine, placebo effect, prediction markets, Ray Kurzweil, recommendation engine, risk-adjusted returns, Ronald Coase, Search for Extraterrestrial Intelligence, self-driving car, sentiment analysis, Shai Danziger, software as a service, SpaceShipOne, speech recognition, statistical model, Steven Levy, supply chain finance, text mining, the scientific method, The Signal and the Noise by Nate Silver, The Wisdom of Crowds, Thomas Bayes, Thomas Davenport, Turing test, Watson beat the top human players on Jeopardy!, X Prize, Yogi Berra, zero-sum game

The machine learning process is designed to accomplish this task, to mechanically develop new capabilities from data. This automation is the means by which PA builds its predictive power. The hunter returns back to the tribe, proudly displaying his kill. So, too, a data scientist posts her model on the bulletin board near the company ping-pong table. The hunter hands over the kill to the cook, and the data scientist cooks up her model, translates it to a standard computer language, and e-mails it to an engineer for integration. A well-fed tribe shows the love; a psyched executive issues a bonus. The tribe munches and the scientist crunches.

…

Special Forces: Dean Abbott, “Hiring and Selecting Key Personnel Using Predictive Analytics,” Predictive Analytics World San Francisco Conference, March 4, 2012, San Francisco, CA. www.predictiveanalyticsworld.com/sanfrancisco/2012/agenda.php#day1–1040a. LinkedIn: Manu Sharma, LinkedIn, “Data Science at LinkedIn: Iterative, Big Data Analytics and You,” Predictive Analytics World New York Conference, October 19, 2011, New York, NY. www.predictiveanalyticsworld.com/newyork/2011/agenda.php#track1-lnk. Scott Nicholson, LinkedIn, “Beyond Big Data: Better Living through Data Science,” Predictive Analytics World Boston Conference, October 1, 2012, Boston, MA. www.predictiveanalyticsworld.com/boston/2012/agenda.php#keynote-900. Scott Nicholson, LinkedIn, “Econometric Applications & Extracting Economic Insights from the LinkedIn Dataset,” Predictive Analytics World San Francisco Conference, March 5, 2012, San Francisco, CA. www.predictiveanalyticsworld.com/sanfrancisco/2012/agenda.php#day1–20a.

…

The stories range from inspiring to downright scary—read them and find out what we’ve been up to while you weren’t paying attention.” —Michael J. A. Berry, author of Data Mining Techniques, Third Edition “Eric Siegel is the Kevin Bacon of the predictive analytics world, organizing conferences where insiders trade knowledge and share recipes. Now, he has thrown the doors open for you. Step in and explore how data scientists are rewriting the rules of business.” —Kaiser Fung, VP, Vimeo; author of Numbers Rule Your World “Written in a lively language, full of great quotes, real-world examples, and case studies, it is a pleasure to read. The more technical audience will enjoy chapters on The Ensemble Effect and uplift modeling—both very hot trends.

Don't Trust Your Gut: Using Data to Get What You Really Want in LIfe by Seth Stephens-Davidowitz

affirmative action, Airbnb, cognitive bias, commoditize, correlation does not imply causation, COVID-19, Daniel Kahneman / Amos Tversky, data science, deep learning, digital map, Donald Trump, en.wikipedia.org, Erik Brynjolfsson, General Magic , global pandemic, Mark Zuckerberg, meta-analysis, Moneyball by Michael Lewis explains big data, Paul Graham, peak-end rule, randomized controlled trial, Renaissance Technologies, Sam Altman, science of happiness, selection bias, side hustle, Silicon Valley, Steve Jobs, Steve Wozniak, systematic bias, Tony Fadell, twin studies, Tyler Cowen, urban planning, Y Combinator

The artists who make it big, in contrast, present to a far wider set of places, allowing themselves to stumble upon a big break. Many people have talked about the importance in your career of showing up. But data scientists have found it’s about showing up to a wide range of places. This book isn’t meant to give advice only for single people, new parents, or aspiring artists—though there will be more lessons here for all of them. My goal is to offer some lessons in new, big datasets that are useful for you, no matter what stage of life you are in. There will be lessons recently uncovered by data scientists in how to be happier, look better, advance your career, and much more. And the idea for the book all came to me one evening while . . .

…

In the past few years, other teams of researchers have mined online dating sites, combing through large, new datasets on the traits and swipes of tens of thousands of single people to determine what predicts romantic desirability. The findings from the research on romantic desirability, unlike the research on romantic happiness, has been definitive. While data scientists have found that it is surprisingly difficult to detect the qualities in romantic partners that lead to happiness, data scientists have found it strikingly easy to detect the qualities that are catnip in the dating scene. A recent study, in fact, found that not only is it possible to predict with great accuracy whether someone will swipe left or right on a particular person on an online dating site.

…

See rich people weather and happiness, 261–262 websites athletic scholarships stats, 95, 98 equestrianism on a budget, 108 neighborhood information, 77, 87 ScenicOrNot, 259 trackyourhappiness.org, 235 twin basketball players models, 103, 104 wholesale distribution business startup, 122 West Point cadets’ success, 195–196 wholesale beverage distribution, 111–113, 122, 130 website on business startup, 122 Wikipedia entries in contrasting counties, 73 work misery factor, 221, 238–239 misery factor lessened, 239–242 quitting job per coin flip, 242 Y you are drawn to you, 15, 39–40 Youkilis, Kevin, 46–48 young entrepreneurs, 140. See also age of typical entrepreneur Z zero-profit condition, 128 Zuckerberg, Mark, 140, 142, 149 About the Author SETH STEPHENS-DAVIDOWITZ is a data scientist, author, and keynote speaker. His 2017 book, Everybody Lies, was a New York Times bestseller and an Economist Book of the Year. He has worked as a contributing op-ed writer for the New York Times, a lecturer at the Wharton School, and a Google data scientist. He received a BA in philosophy from Stanford, where he graduated Phi Beta Kappa, and a PhD in economics from Harvard. He lives in Brooklyn and is a passionate fan of the Mets, Knicks, Jets, and Leonard Cohen.

Designing Great Data Products by Jeremy Howard, Mike Loukides, Margit Zwemer

AltaVista, data science, Filter Bubble, PageRank, pattern recognition, recommendation engine, self-driving car, sentiment analysis, Silicon Valley, text mining

Great predictive modeling is an important part of the solution, but it no longer stands on its own; as products become more sophisticated, it disappears into the plumbing. Someone using Google’s self-driving car is completely unaware of the hundreds (if not thousands) of models and the petabytes of data that make it work. But as data scientists build increasingly sophisticated products, they need a systematic design approach. We don’t claim that the Drivetrain Approach is the best or only method; our goal is to start a dialog within the data science and business communities to advance our collective vision. Objective-based data products We are entering the era of data as drivetrain, where we use data not just to generate more data (in the form of predictions), but use data to produce actionable outcomes.

…

In an emergency, a data product that just produces more data is of little use. Data scientists now have the predictive tools to build products that increase the common good, but they need to be aware that building the models is not enough if they do not also produce optimized, implementable outcomes. The future for data products We introduced the Drivetrain Approach to provide a framework for designing the next generation of great data products and described how it relies at its heart on optimization. In the future, we hope to see optimization taught in business schools as well as in statistics departments. We hope to see data scientists ship products that are designed to produce desirable business outcomes.

…

We hope to see data scientists ship products that are designed to produce desirable business outcomes. This is still the dawn of data science. We don’t know what design approaches will be developed in the future, but right now, there is a need for the data science community to coalesce around a shared vocabulary and product design process that can be used to educate others on how to derive value from their predictive models. If we do not do this, we will find that our models only use data to create more data, rather than using data to create actions, disrupt industries and transform lives. About the Authors Mike Loukides is an editor for O'Reilly & Associates.

Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy by Cathy O'Neil

Affordable Care Act / Obamacare, Alan Greenspan, algorithmic bias, Bernie Madoff, big data - Walmart - Pop Tarts, call centre, Cambridge Analytica, carried interest, cloud computing, collateralized debt obligation, correlation does not imply causation, Credit Default Swap, credit default swaps / collateralized debt obligations, crowdsourcing, data science, disinformation, electronic logging device, Emanuel Derman, financial engineering, Financial Modelers Manifesto, Glass-Steagall Act, housing crisis, I will remember that I didn’t make the world, and it doesn’t satisfy my equations, Ida Tarbell, illegal immigration, Internet of things, late fees, low interest rates, machine readable, mass incarceration, medical bankruptcy, Moneyball by Michael Lewis explains big data, new economy, obamacare, Occupy movement, offshore financial centre, payday loans, peer-to-peer lending, Peter Thiel, Ponzi scheme, prediction markets, price discrimination, quantitative hedge fund, Ralph Nader, RAND corporation, real-name policy, recommendation engine, Rubik’s Cube, Salesforce, Sharpe ratio, statistical model, tech worker, Tim Cook: Apple, too big to fail, Unsafe at Any Speed, Upton Sinclair, Watson beat the top human players on Jeopardy!, working poor

,” Boston Globe, November 7, 2015, www.​bostonglobe.​com/​2015/​11/​07/​childwelfare-​bostonglobe-​com/​AZ2kZ7ziiP8c​BMOite2KKP/story.​html. ABOUT THE AUTHOR Cathy O’Neil is a data scientist and the author of the blog mathbabe.​org. She earned a PhD in mathematics from Harvard and taught at Barnard College before moving to the private sector, where she worked for the hedge fund D. E. Shaw. She then worked as a data scientist at various start-ups, building models that predict people’s purchases and clicks. O’Neil started the Lede Program in Data Journalism at Columbia and is the author of Doing Data Science. She appears weekly on the Slate Money podcast. What’s next on your reading list?

…

For diploma mills like the University of Phoenix, I think it’s safe to say, the goal is to recruit the greatest number of students who can land government loans to pay most of their tuition and fees. With that objective in mind, the data scientists have to figure out how best to manage their various communication channels so that together they generate the most bang for each buck. The data scientists start off with a Bayesian approach, which in statistics is pretty close to plain vanilla. The point of Bayesian analysis is to rank the variables with the most impact on the desired outcome. Search advertising, TV, billboards, and other promotions would each be measured as a function of their effectiveness per dollar.

…

And if it wants to find out what drives shopping recidivism, it carries out research. Its data scientists don’t just study zip codes and education levels. They also inspect people’s experience within the Amazon ecosystem. They might start by looking at the patterns of all the people who shopped once or twice at Amazon and never returned. Did they have trouble at checkout? Did their packages arrive on time? Did a higher percentage of them post a bad review? The questions go on and on, because the future of the company hinges upon a system that learns continually, one that figures out what makes customers tick. If I had a chance to be a data scientist for the justice system, I would do my best to dig deeply to learn what goes on inside those prisons and what impact those experiences might have on prisoners’ behavior.

Data Action: Using Data for Public Good by Sarah Williams

affirmative action, Amazon Mechanical Turk, Andrei Shleifer, augmented reality, autonomous vehicles, Brexit referendum, Cambridge Analytica, Charles Babbage, City Beautiful movement, commoditize, coronavirus, COVID-19, crowdsourcing, data acquisition, data is the new oil, data philanthropy, data science, digital divide, digital twin, Donald Trump, driverless car, Edward Glaeser, fake news, four colour theorem, global village, Google Earth, informal economy, Internet of things, Jane Jacobs, John Snow's cholera map, Kibera, Lewis Mumford, Marshall McLuhan, mass immigration, mass incarceration, megacity, military-industrial complex, Minecraft, neoliberal agenda, New Urbanism, Norbert Wiener, nowcasting, oil shale / tar sands, openstreetmap, place-making, precautionary principle, RAND corporation, ride hailing / ride sharing, selection bias, self-driving car, sentiment analysis, Sidewalk Labs, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, Steven Levy, the built environment, The Chicago School, The Death and Life of Great American Cities, transatlantic slave trade, Uber for X, upwardly mobile, urban planning, urban renewal, W. E. B. Du Bois, Works Progress Administration

So it is essential we investigate potential bias created by the technology used to collect data. Analysis of data acquired through creative means must include subject experts and the community represented in the data. Working with policy experts helps data scientists ask the right questions. This type of multidisciplinary team can generate more accurate and ethical results from the data. There are numerous examples of data scientists who try to predict the dynamics of human life, successfully at first; but without continued inclusion of subject experts their data models are quickly outdated. This is the hubris of big data. Why do we often think the data analyst can find the right questions to ask without asking those who have in-depth knowledge of the topics we seek to understand?

…

Method Throughout Data Action, I argue that unlocking data for policy change works best when the process engages multidisciplinary teams that include policy experts, data scientists, and data visualizers, among others. In the book's conclusion, “It's How We Work with Data That Really Matters” I stress that bringing together these experts allows the creative expression of data to truly blossom. Policy experts understand the issues, data scientists know how to develop algorithms, and graphic designers can share the results through compelling visuals. Working together these specialists extend their findings beyond the walls of academia or city hall—and reach the hands and minds of the public.

…

Kubzansky, “A Framework for Examining Social Stress and Susceptibility to Air Pollution in Respiratory Health,” Environmental Health Perspectives 117, no. 9 (September 1, 2009): 1351–1358, https://doi.org/10.1289/ehp.0900612. 54 Interview with Iyad Kheirbek of [New York City Department of Mental Health and Hygiene], January 2015. 55 “2014 West Africa Ebola Response—OpenStreetMap Wiki,” accessed January 25, 2019, https://wiki.openstreetmap.org/wiki/2014_West_Africa_Ebola_Response. 56 “Ushahidi,” accessed January 25, 2019, https://www.ushahidi.com/. 57 Ida Norheim-Hagtun and Patrick Meier, “Crowdsourcing for Crisis Mapping in Haiti,” Innovations: Technology, Governance, Globalization 5, no. 4 (2010): 81–89. 58 Jessica Ramirez, “‘Ushahidi’ Technology Saves Lives in Haiti and Chile,” Newsweek, March 3, 2010, https://www.newsweek.com/ushahidi-technology-saves-lives-haiti-and-chile-210262. 59 Norheim-Hagtun and Meier, “Crowdsourcing for Crisis Mapping in Haiti.” 60 Patrick Meier, “Crowdsourcing the Evaluation of Post-Sandy Building Damage Using Aerial Imagery,” IRevolutions (blog), November 1, 2012, https://irevolutions.org/2012/11/01/crowdsourcing-sandy-building-damage/. 61 Introduction to Data Science (IDA), “About,” IDS webpage, https://www.idsucla.org/about. 62 Ron Eglash, Juan E. Gilbert, and Ellen Foster, “Toward Culturally Responsive Computing Education,” Commun. ACM 56, no. 7 (July 2013): 33–36, https://doi.org/10.1145/2483852.2483864; Amelia McNamara and Mark Hansen, “Teaching Data Science to Teenagers,” in Proceedings of the Ninth International Conference on Teaching Statistics, 2014. 63 Nicole Lazar and Christine Franklin, “The Big Picture: Preparing Students for a Data-Centric World,” Chance 28, no. 4 (2015): 43–45.

Architecting Modern Data Platforms: A Guide to Enterprise Hadoop at Scale by Jan Kunigk, Ian Buss, Paul Wilkinson, Lars George

Amazon Web Services, barriers to entry, bitcoin, business intelligence, business logic, business process, cloud computing, commoditize, computer vision, continuous integration, create, read, update, delete, data science, database schema, Debian, deep learning, DevOps, domain-specific language, fault tolerance, Firefox, FOSDEM, functional programming, Google Chrome, Induced demand, information security, Infrastructure as a Service, Internet of things, job automation, Kickstarter, Kubernetes, level 1 cache, loose coupling, microservices, natural language processing, Network effects, platform as a service, single source of truth, source of truth, statistical model, vertical integration, web application

The architect is responsible for technical coordination and should have extensive experience working with big data and Hadoop. Data scientist Although the term “data scientist” has been around for some time, the current surge in its use in corporate IT reflects how much big data changes the complexity of data management in IT. It also reflects a shift in academia, where data science has evolved to become a fully qualified discipline at many renowned universities.1 Sometimes we see organizations that are largely indifferent to data science per se, or that simply try to rebrand all existing analyst staff as data scientists. The data scientist, however, actually does more: Statistics and classic BI The data scientist depends on classic tools to present and productize the result of his work, but before these tools can be used, a lot of exploration, cleansing, and modeling is likely to be required on the Hadoop layer.

…

End users commonly access the web interfaces of the YARN ResourceManager, MapReduce Job History Server, and Spark History Server. In addition, the Hue project offers a comprehensive user interface for many components in the stack, including HDFS, Hive, Impala, Oozie, and Solr. Additional user-oriented or specialized web UIs are also available, such as Jupyter Notebook, Apache Zeppelin, or Cloudera Data Science Workbench for data scientists, as shown in Table 11-1. Table 11-1. A summary of access mechanisms Project Programmatic Command line Web UI HDFS Java, REST (WebHDFS/HttpFS) hdfs NameNode and DataNode YARN Java, REST (RM) yarn ResourceManager and NodeManager ZooKeeper Java/C++ zookeeper-client - HBase Java, HBase REST/Thrift servera hbase shell Master and RegionServer Hive Thrift, JDBC, ODBC beeline HiveServer2 Oozie Java, REST oozie Server via extension Spark Java/Scala/Python, JDBC (via Thrift server) spark-shell, spark-submit, pyspark History Server Impala JDBC, ODBC impala-shell Statestore, catalog server, daemon Solr Java, REST solrctl Server Kudu Java/C++/Python kudu admin utility Master and tablet server Hue Python SDK - Hue Server a Apache Phoenix provides a JDBC interface to Apache HBase.

…

Coding Whereas the typical analyst or statistician understands methods and models mathematically, a good data scientist also has a solid background in parallel algorithms to build large-scale distributed applications around such models. As we already mentioned, the data scientist is well versed in coding in third-generation and functional programming languages, such as Scala, Java, and Python, in addition to the domain-specific languages of the classic analytics world. In this function, the data scientist collaborates with development departments to build fully fledged distributed applications that can be productively deployed to Hadoop.

Only Humans Need Apply: Winners and Losers in the Age of Smart Machines by Thomas H. Davenport, Julia Kirby

"World Economic Forum" Davos, AI winter, Amazon Robotics, Andy Kessler, Apollo Guidance Computer, artificial general intelligence, asset allocation, Automated Insights, autonomous vehicles, basic income, Baxter: Rethink Robotics, behavioural economics, business intelligence, business process, call centre, carbon-based life, Clayton Christensen, clockwork universe, commoditize, conceptual framework, content marketing, dark matter, data science, David Brooks, deep learning, deliberate practice, deskilling, digital map, disruptive innovation, Douglas Engelbart, driverless car, Edward Lloyd's coffeehouse, Elon Musk, Erik Brynjolfsson, estate planning, financial engineering, fixed income, flying shuttle, follow your passion, Frank Levy and Richard Murnane: The New Division of Labor, Freestyle chess, game design, general-purpose programming language, global pandemic, Google Glasses, Hans Lippershey, haute cuisine, income inequality, independent contractor, index fund, industrial robot, information retrieval, intermodal, Internet of things, inventory management, Isaac Newton, job automation, John Markoff, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, Joi Ito, Khan Academy, Kiva Systems, knowledge worker, labor-force participation, lifelogging, longitudinal study, loss aversion, machine translation, Mark Zuckerberg, Narrative Science, natural language processing, Nick Bostrom, Norbert Wiener, nuclear winter, off-the-grid, pattern recognition, performance metric, Peter Thiel, precariat, quantitative trading / quantitative finance, Ray Kurzweil, Richard Feynman, risk tolerance, Robert Shiller, robo advisor, robotic process automation, Rodney Brooks, Second Machine Age, self-driving car, Silicon Valley, six sigma, Skype, social intelligence, speech recognition, spinning jenny, statistical model, Stephen Hawking, Steve Jobs, Steve Wozniak, strong AI, superintelligent machines, supply-chain management, tacit knowledge, tech worker, TED Talk, the long tail, transaction costs, Tyler Cowen, Tyler Cowen: Great Stagnation, Watson beat the top human players on Jeopardy!, Works Progress Administration, Zipcar

If you have a background in one of these (slightly) ancillary aspects of software, you can probably find the same kind of job related to automated software. Data Scientists —A couple of years ago, Tom and D. J. Patil, now chief data scientist of the White House Office of Science and Technology Policy, wrote (with Julia’s editing help) an article suggesting that data scientists held the “sexiest job of the 21st century.”1 It’s not that the people themselves were necessarily sexy, but that the jobs were difficult and hard to fill. They still are, though the shortage may be easing a bit, with the introduction of a number of new master’s programs in data science at U.S. universities. Data scientists are likely to be highly valued when the data used by cognitive systems are highly unstructured (voice or text or human genome records, as opposed to rows and columns of numbers) or difficult to extract from its source.

…

He does that with his team for about 60 percent of his time. He also spends a lot of time with customers—roughly a couple of days a week. He hears what their needs for new capabilities are, and translates that into data science activity by his team. Whenever he can squeeze it in, he interviews and hires new data scientists, and meets with other DataXu executives. When he hires other data scientists, Catanzaro looks for three types of skills, only one of which is technical. The first is “data science smarts”—being good with big data technologies, statistics, and so forth. He’s not so much interested in knowledge of a particular set of tools, but rather the “raw horsepower” to be able to master new tools.

…

And they also are likely to have either quantitative modeling skills or natural language processing skills. What do data scientists do day to day in the development of automated decision systems? Automated systems typically use a lot of data, so the data scientist might be scouting around to figure out the next great external data source. After a promising source is identified, he or she might be determining how to get the data into the right format, or how to combine it with the data that the organization already has. The data scientist might also be working on an algorithm to extract insights from the data. Or, since data scientists tend to be good at computational skills, too, they might be architecting or helping to develop the new or modified system.

The Data Revolution: Big Data, Open Data, Data Infrastructures and Their Consequences by Rob Kitchin

Bayesian statistics, business intelligence, business process, cellular automata, Celtic Tiger, cloud computing, collateralized debt obligation, conceptual framework, congestion charging, corporate governance, correlation does not imply causation, crowdsourcing, data science, discrete time, disruptive innovation, George Gilder, Google Earth, hype cycle, Infrastructure as a Service, Internet Archive, Internet of things, invisible hand, knowledge economy, Large Hadron Collider, late capitalism, lifelogging, linked data, longitudinal study, machine readable, Masdar, means of production, Nate Silver, natural language processing, openstreetmap, pattern recognition, platform as a service, recommendation engine, RFID, semantic web, sentiment analysis, SimCity, slashdot, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, smart grid, smart meter, software as a service, statistical model, supply-chain management, technological solutionism, the scientific method, The Signal and the Noise by Nate Silver, transaction costs

The worry for many commentators is that the potential benefits of data-driven business and science will not be fully realised due to a shortage of human talent, in particular ‘data scientists, who combine the skills of software programmer, statistician and storyteller/artist to extract the nuggets of gold hidden under mountains of data’ (Cukier 2010), and managers who understand how to convert such nuggets into wise decisions. With respect to the latter, as Shah et al. (2012: 23) note, ‘[i]nvestments in analytics can be useless, even harmful, unless employees can incorporate that data into complex decision making’. Universities are now starting to create new data science programmes and research centres, and to adapt existing courses to include training in new skills sets, in an effort to ameliorate some skills gaps.

…

He argues that ‘shifting the analysis to digital data... opens up epistemic questions as to who is the most legitimate producer of knowledge – the museum collector (the clinician, or the molecular biologist) or the statistician analyzing the data’ or producing the simulation or model (2012: 87). Some data scientists are thus undoubtedly ignoring the observations of Porway (2013): Without subject matter experts available to articulate problems in advance, you get [poor] results... Subject matter experts are doubly needed to assess the results of the work, especially when you’re dealing with sensitive data about human behavior. As data scientists, we are well equipped to explain the ‘what’ of data, but rarely should we touch the question of ‘why’ on matters we are not experts in. As Porway notes, what is really needed is for data scientists and domain experts to work with each other to ensure that the data analytics used make sense and that the results from such analytics are sensibly and contextually interpreted.

…

Berry, D. (2011) ‘The computational turn: thinking about the digital humanities’, Culture Machine, 12, http://www.culturemachine.net/index.php/cm/article/view/440/470 (last accessed 3 December 2012). Bertolucci, J. (2013) ‘IBM, universities team up to build data scientists’, InformationWeek, 15 January. http://www.informationweek.com/big-data/big-data-analytics/ibm-universities-team-upto-build-data-scientists/ (last accessed 16 January 2014). Bettencourt, L.M.A., Lobo, J., Helbing, D., Kuhnert, C. and West, G.B. (2007) ‘Growth, innovation, scaling, and the pace of life in cities’, Proceedings of the National Academy of Sciences, 104(17): 7301–06.

Leadership by Algorithm: Who Leads and Who Follows in the AI Era? by David de Cremer

"Friedman doctrine" OR "shareholder theory", algorithmic bias, algorithmic management, AlphaGo, bitcoin, blockchain, business climate, business process, Computing Machinery and Intelligence, corporate governance, data is not the new oil, data science, deep learning, DeepMind, Donald Trump, Elon Musk, fake news, future of work, job automation, Kevin Kelly, Mark Zuckerberg, meta-analysis, Norbert Wiener, pattern recognition, Peter Thiel, race to the bottom, robotic process automation, Salesforce, scientific management, shareholder value, Silicon Valley, Social Responsibility of Business Is to Increase Its Profits, Stephen Hawking, The Future of Employment, Turing test, work culture , workplace surveillance , zero-sum game

Second, some departments will employ algorithms more than others, thereby creating differences in work attitudes towards automation, which will make the process of digital transformation more difficult. Integrating teams of data scientists in the daily operations of the company With the work environment gradually being automated, organizations will increasingly hire more people with an engineering and data-science background. These new hires will have an expert understanding of the new technology and the ability to work with big data. A problem, however, is that those experts usually do not share the same mindset as the people who are not trained in the fields of engineering and data science. Organizations often fail to recognize this difference in mindset and put little effort into ensuring that data scientists are integrated into the organization.

…

In a way, leaders in the 21st century show competence by bringing the right teams of experts together to optimise the use of data to bring the value that is expected. For example, leaders connecting teams of data scientists with HR and finance teams in transparent and effective ways can help to increase the success rate of digital transformation strategies. The team of data scientists will help their colleagues to see what possibilities are available to digitalize information. Equally, the other teams can help data scientists understand their needs and thus provide input in designing a more user-friendly digital environment. Finally, because successes are rarely achieved immediately, it is important that leaders provide regular updates to the different parties involved on how the challenges are being approached.

…

Organizations often fail to recognize this difference in mindset and put little effort into ensuring that data scientists are integrated into the organization. Becoming an automated organization means that all operations will be affected. Teams of data scientists thus need to understand the goals of the finance department, human resource department, sales department and so forth. Likewise, organizations need to prepare all the other departments to be open and collaborative with the team of data scientists. It is only with an open-minded attitude that successful integration and implementation of algorithms within the context of each department can be achieved.²⁰² Promoting transparency in communication and exchange of data Today there is no longer any doubt that to succeed in the future, organizations will need to have the ability to deal with data and use algorithms.

Infonomics: How to Monetize, Manage, and Measure Information as an Asset for Competitive Advantage by Douglas B. Laney

3D printing, Affordable Care Act / Obamacare, banking crisis, behavioural economics, blockchain, book value, business climate, business intelligence, business logic, business process, call centre, carbon credits, chief data officer, Claude Shannon: information theory, commoditize, conceptual framework, crowdsourcing, dark matter, data acquisition, data science, deep learning, digital rights, digital twin, discounted cash flows, disintermediation, diversification, en.wikipedia.org, endowment effect, Erik Brynjolfsson, full employment, hype cycle, informal economy, information security, intangible asset, Internet of things, it's over 9,000, linked data, Lyft, Nash equilibrium, Neil Armstrong, Network effects, new economy, obamacare, performance metric, profit motive, recommendation engine, RFID, Salesforce, semantic web, single source of truth, smart meter, Snapchat, software as a service, source of truth, supply-chain management, tacit knowledge, technological determinism, text mining, uber lyft, Y2K, yield curve

These types of algorithms can help to monetize almost any kind of information, be it granular IoT data or macro-level economic figures. Expect these kinds of algorithms to be a standard component in the vast majority of data scientist toolboxes and increasingly accepted by business leaders, despite their “black box” models. As a result, many data science tasks will become automated, increasing the productivity of data scientists and enabling a class of “citizen data scientists” to emerge. This will put signifi-cant pressure on information asset supply chains and information curation efforts, and engender a boom in information monetization ideas from all corners of the organization across all industries.

…

For example, the real estate aggregator Trulia discovered that 90 percent of its web traffic is from people clicking on photos of homes. But Trulia had no information about what was in the photos. The photos had no descriptions or tags. So Trulia’s data science team trained a one-billion-node neural network to learn what is depicted in them. Now, according to Todd Holloway, who started Trulia’s data science program, “The system can find you a home in the Hamptons with photos of wine cellars.”11 Helping a buyer find a home is one thing, but now Trulia can correlate sales data with what site users are looking at, and license this information and insight to realtors, homebuilders, appliance manufacturers, and any type of company within the periphery of the real estate market.

…

And value-focused CDOs, will deploy information assets to generate supplemental and significant revenue streams. Advanced Analytics, Data Science, and Artificial Intelligence Not just a global trend, but also a technology trend, advanced analytics solutions are becoming increasingly popular in driving business innovation and experimentation—and creating competitive advantage by monetizing available information assets inside and outside the organization. Over the foreseeable future, enterprises will be seeking to adopt advanced analytics and adapt their business models, establish specialist data science teams, and rethink their overall strategies to keep pace with the competition.

Machine Learning Design Patterns: Solutions to Common Challenges in Data Preparation, Model Building, and MLOps by Valliappa Lakshmanan, Sara Robinson, Michael Munn

A Pattern Language, Airbnb, algorithmic trading, automated trading system, business intelligence, business logic, business process, combinatorial explosion, computer vision, continuous integration, COVID-19, data science, deep learning, DevOps, discrete time, en.wikipedia.org, Hacker News, industrial research laboratory, iterative process, Kubernetes, machine translation, microservices, mobile money, natural language processing, Netflix Prize, optical character recognition, pattern recognition, performance metric, recommendation engine, ride hailing / ride sharing, selection bias, self-driving car, sentiment analysis, speech recognition, statistical model, the payments system, web application

Data engineers implement infrastructure and pipelines around data. Machine learning engineers do similar tasks to data engineers, but for ML models. They take models developed by data scientists, and manage the infrastructure and operations around training and deploying those models. ML engineers help build production systems to handle updating models, model versioning, and serving predictions to end users. The smaller the data science team at a company and the more agile the team is, the more likely it is that the same person plays multiple roles. If you are in such a situation, it is very likely that you read the above three descriptions and saw yourself partially in all three categories.

…

His team builds software solutions for business problems using Google Cloud’s data analytics and machine learning products. He founded Google’s Advanced Solutions Lab ML Immersion program. Before Google, Lak was a Director of Data Science at Climate Corporation and a Research Scientist at NOAA. Sara Robinson is a Developer Advocate on Google’s Cloud Platform team, focusing on machine learning. She inspires developers and data scientists to integrate ML into their applications through demos, online content, and events. Sara has a bachelor’s degree from Brandeis University. Before Google, she was a Developer Advocate on the Firebase team.

…

Roles Within an organization, there are many different job roles relating to data and machine learning. Below we’ll define a few common ones referenced frequently throughout the book. This book is targeted primarily at data scientists, data engineers, and ML engineers, so let’s start with those. A data scientist is someone focused on collecting, interpreting, and processing datasets. They run statistical and exploratory analysis on data. As it relates to machine learning, a data scientist may work on data collection, feature engineering, model building, and more. Data scientists often work in Python or R in a notebook environment, and are usually the first to build out an organization’s machine learning models.

Dataclysm: Who We Are (When We Think No One's Looking) by Christian Rudder

4chan, Affordable Care Act / Obamacare, bitcoin, cloud computing, correlation does not imply causation, crowdsourcing, cuban missile crisis, data science, Donald Trump, Edward Snowden, en.wikipedia.org, fake it until you make it, Frank Gehry, Howard Zinn, Jaron Lanier, John Markoff, John Snow's cholera map, lifelogging, Mahatma Gandhi, Mikhail Gorbachev, Nate Silver, Nelson Mandela, new economy, obamacare, Occupy movement, p-value, power law, pre–internet, prosperity theology / prosperity gospel / gospel of success, race to the bottom, retail therapy, Salesforce, selection bias, Snapchat, social graph, Steve Jobs, the scientific method, the strength of weak ties, Twitter Arab Spring, two and twenty

I transported the data to the previous Voronoi partition in order to maintain consistency with the previous Craigslist map. Years ago, an enterprising hacker The hacker is Pete Warden, and his post is “How to Split Up the US,” which you can find here: petewarden.com/2010/02/06/how-to-split-up-the-us/. As Warden notes in a later post, “Why You Should Never Trust a Data Scientist,” his grouping of the United States into the seven new zones is arbitrary—the data science version of “for entertainment purposes only.” I reference them here in that spirit. Matthew Zook, a geographer Professor Zook and his team maintain a fantastic geography blog called Floating Sheep, and that blog was my primary source for his work: floatingsheep.org.

…

For more information and the full study, please refer to the Facebook Data Science post on Coordinated Migration: www.facebook.com/notes/facebook-data-science/coordinated-migration/10151930946453859. As you’ll see when you visit the link, in reproducing their work, I modified their original map by removing the labels and focusing on a smaller part of the region, to make the map more readable in print. Thank you to Mike Develin, also at Facebook, for helping facilitate permission for this reproduction. All Facebook Data Science work is done on anonymized and aggregated data. Chapter 13: Our Brand Could Be Your Life But what they don’t tell you See Clare Baker, “Behind the Red Triangle: The Bass Pale Ale Brand and Logo” Logoworks.com, November 8, 2013, logoworks.com/blog/bass-pale-ale-brand-and-logo/.

…

And it’s because these few letters are such a concise description that Shazam is so fast: instead of a guitar, Paul McCartney, and just the right amount of reverb, “Yesterday” starts with •DRUUUUUUDDR. That’s a lot easier to understand. Like an app straining for a song, data science is about finding patterns. Time after time, I—and the many other people doing work like me—have had to devise methods, structures, even shortcuts to find the signal amidst the noise. We’re all looking for our own Parsons code. Something so simple and yet so powerful is a once-in-a-lifetime discovery, but luckily there are a lot of lifetimes out there. And for any problem that data science might face, this book has been my way to say: I like our odds. 1 For more on the Kafkaesque implications of the CFAA, please see “Until Today, If You Were 17, It Could Have Been Illegal to Read Seventeeen.com Under the CFAA” and “Are You a Teenager Who Reads News Online?

Big Data Analytics: Turning Big Data Into Big Money by Frank J. Ohlhorst

algorithmic trading, bioinformatics, business intelligence, business logic, business process, call centre, cloud computing, create, read, update, delete, data acquisition, data science, DevOps, extractivism, fault tolerance, information security, Large Hadron Collider, linked data, machine readable, natural language processing, Network effects, pattern recognition, performance metric, personalized medicine, RFID, sentiment analysis, six sigma, smart meter, statistical model, supply-chain management, warehouse automation, Watson beat the top human players on Jeopardy!, web application

Determining those skills is one of the first steps in putting a team together. THE DATA SCIENTIST One of the first concepts to become acquainted with is the data scientist; a relatively new title, it is not readily recognized or accepted by many organizations, but it is here to stay. A data scientist is normally associated with an employee or a business intelligence (BI) consultant who excels at analyzing data, particularly large amounts of data, to help a business gain a competitive edge. The data scientist is usually the de facto team leader during a Big Data analytics project. The title data scientist is sometimes disparaged because it lacks specificity and can be perceived as an aggrandized synonym for data analyst.

…

Much like the data themselves, the team should not be static in nature and should be able to evolve and adapt to the needs of the business. CHALLENGES REMAIN Locating the right talent to analyze data is the biggest hurdle in building a team. Such talent is in high demand, and the need for data analysts and data scientists continues to grow at an almost exponential rate. Finding this talent means that organizations will have to focus on data science and hire statistical modelers and text data–mining professionals as well as people who specialize in sentiment analysis. Success with Big Data analytics requires solid data models, statistical predictive models, and test analytic models, since these will be the core applications needed to do Big Data.

…

Nevertheless, the position is gaining acceptance with large enterprises that are interested in deriving meaning from Big Data, the voluminous amount of structured, unstructured, and semistructured data that a large enterprise produces or has access to. A data scientist must possess a combination of analytic, machine learning, data mining, and statistical skills as well as experience with algorithms and coding. However, the most critical skill a data scientist should possess is the ability to translate the significance of data in a way that can be easily understood by others. THE TEAM CHALLENGE Finding and hiring talented workers with analytics skills is the first step in creating an effective data analytics team.

Keeping Up With the Quants: Your Guide to Understanding and Using Analytics by Thomas H. Davenport, Jinho Kim

behavioural economics, Black-Scholes formula, business intelligence, business process, call centre, computer age, correlation coefficient, correlation does not imply causation, Credit Default Swap, data science, en.wikipedia.org, feminist movement, Florence Nightingale: pie chart, forensic accounting, global supply chain, Gregor Mendel, Hans Rosling, hypertext link, invention of the telescope, inventory management, Jeff Bezos, Johannes Kepler, longitudinal study, margin call, Moneyball by Michael Lewis explains big data, Myron Scholes, Netflix Prize, p-value, performance metric, publish or perish, quantitative hedge fund, random walk, Renaissance Technologies, Robert Shiller, self-driving car, sentiment analysis, six sigma, Skype, statistical model, supply-chain management, TED Talk, text mining, the scientific method, Thomas Davenport

For example, at Intuit, George Roumeliotis heads a data science group that analyzes and creates product features based on the vast amount of online data that Intuit collects. For every project in which his group engages with an internal customer, he recommends a methodology for doing and communicating about the analysis. Most of the steps have a strong business orientation: My understanding of the business problem How I will measure the business impact What data is available The initial solution hypothesis The solution The business impact of the solution Data scientists using this methodology are encouraged to create a wiki so that they can post the results of each step.

…

In the online information industry, companies have big data involving many petabytes of information. New information comes in at such volume and speed that it would be difficult for humans to comprehend it all. In this environment, the data scientists that work in such organizations (basically quantitative analysts with higher-than-normal IT skills) are often located in product development organizations. Their goal is to develop product prototypes and new product features, not reports or presentations. For example, the Data Science group at the business networking site LinkedIn is a part of the product organization, and has developed a variety of new product features and functions based on the relationships between social networks and jobs.

…

Patil eventually graduated and became a faculty member at Maryland, and worked on modeling the complexity of weather. He then worked for the US government on intelligence issues. Research funding was limited at the time, so he left to work for Skype, then owned by eBay. He next became the leader of data scientists at LinkedIn, where the people in that highly analytical position have been enormously influential in product development. Now Patil is a data scientist in residence (perhaps the first person with that title as well) at the venture capital firm Greylock Partners, helping the firm’s portfolio companies to think about data and analytics. He’s perhaps the world’s best example of latent math talent.

The Age of Surveillance Capitalism by Shoshana Zuboff

"World Economic Forum" Davos, algorithmic bias, Amazon Web Services, Andrew Keen, augmented reality, autonomous vehicles, barriers to entry, Bartolomé de las Casas, behavioural economics, Berlin Wall, Big Tech, bitcoin, blockchain, blue-collar work, book scanning, Broken windows theory, California gold rush, call centre, Cambridge Analytica, Capital in the Twenty-First Century by Thomas Piketty, Cass Sunstein, choice architecture, citizen journalism, Citizen Lab, classic study, cloud computing, collective bargaining, Computer Numeric Control, computer vision, connected car, context collapse, corporate governance, corporate personhood, creative destruction, cryptocurrency, data science, deep learning, digital capitalism, disinformation, dogs of the Dow, don't be evil, Donald Trump, Dr. Strangelove, driverless car, Easter island, Edward Snowden, en.wikipedia.org, Erik Brynjolfsson, Evgeny Morozov, facts on the ground, fake news, Ford Model T, Ford paid five dollars a day, future of work, game design, gamification, Google Earth, Google Glasses, Google X / Alphabet X, Herman Kahn, hive mind, Ian Bogost, impulse control, income inequality, information security, Internet of things, invention of the printing press, invisible hand, Jean Tirole, job automation, Johann Wolfgang von Goethe, John Markoff, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, Joseph Schumpeter, Kevin Kelly, Kevin Roose, knowledge economy, Lewis Mumford, linked data, longitudinal study, low skilled workers, Mark Zuckerberg, market bubble, means of production, multi-sided market, Naomi Klein, natural language processing, Network effects, new economy, Occupy movement, off grid, off-the-grid, PageRank, Panopticon Jeremy Bentham, pattern recognition, Paul Buchheit, performance metric, Philip Mirowski, precision agriculture, price mechanism, profit maximization, profit motive, public intellectual, recommendation engine, refrigerator car, RFID, Richard Thaler, ride hailing / ride sharing, Robert Bork, Robert Mercer, Salesforce, Second Machine Age, self-driving car, sentiment analysis, shareholder value, Sheryl Sandberg, Shoshana Zuboff, Sidewalk Labs, Silicon Valley, Silicon Valley ideology, Silicon Valley startup, slashdot, smart cities, Snapchat, social contagion, social distancing, social graph, social web, software as a service, speech recognition, statistical model, Steve Bannon, Steve Jobs, Steven Levy, structural adjustment programs, surveillance capitalism, technological determinism, TED Talk, The Future of Employment, The Wealth of Nations by Adam Smith, Tim Cook: Apple, two-sided market, union organizing, vertical integration, Watson beat the top human players on Jeopardy!, winner-take-all economy, Wolfgang Streeck, work culture , Yochai Benkler, you are the product

.… Who knows what other research they’re doing.”14 In other words, Fiske recognized that the experiment was merely an extension of Facebook’s standard practices of behavioral modification, which already flourish without sanction. Facebook data scientist and principal researcher Adam Kramer was deluged with hundreds of media queries, leading him to write on his Facebook page that the corporation really does “care” about its emotional impact. One of his coauthors, Cornell’s Jeffrey Hancock, told the New York Times that he didn’t realize that manipulating the news feeds, even modestly, would make some people feel violated.15 The Wall Street Journal reported that the Facebook data science group had run more than 1,000 experiments since its inception in 2007 and operated with “few limits” and no internal review board.

…

In studying the surveillance capitalist practices of Google, Facebook, Microsoft, and other corporations, I have paid close attention to interviews, patents, earnings calls, speeches, conferences, videos, and company programs and policies. In addition, between 2012 and 2015 I interviewed 52 data scientists from 19 different companies with a combined 586 years of experience in high-technology corporations and startups, primarily in Silicon Valley. These interviews were conducted as I developed my “ground truth” understanding of surveillance capitalism and its material infrastructure. Early on I approached a small number of highly respected data scientists, senior software developers, and specialists in the “internet of things.” My interview sample grew as scientists introduced me to their colleagues.

…

Surveillance capitalists adapted many of the highly contestable assumptions of behavioral economists as one cover story with which to legitimate their practical commitment to a unilateral commercial program of behavior modification. The twist here is that nudges are intended to encourage choices that accrue to the architect, not to the individual. The result is data scientists trained on economies of action who regard it as perfectly normal to master the art and science of the “digital nudge” for the sake of their company’s commercial interests. For example, the chief data scientist for a national drugstore chain described how his company designs automatic digital nudges that subtly push people toward the specific behaviors favored by the company: “You can make people do things with this technology.

Mindf*ck: Cambridge Analytica and the Plot to Break America by Christopher Wylie

4chan, affirmative action, Affordable Care Act / Obamacare, air gap, availability heuristic, Berlin Wall, Bernie Sanders, Big Tech, big-box store, Boris Johnson, Brexit referendum, British Empire, call centre, Cambridge Analytica, Chelsea Manning, chief data officer, cognitive bias, cognitive dissonance, colonial rule, computer vision, conceptual framework, cryptocurrency, Daniel Kahneman / Amos Tversky, dark pattern, dark triade / dark tetrad, data science, deep learning, desegregation, disinformation, Dominic Cummings, Donald Trump, Downton Abbey, Edward Snowden, Elon Musk, emotional labour, Etonian, fake news, first-past-the-post, gamification, gentleman farmer, Google Earth, growth hacking, housing crisis, income inequality, indoor plumbing, information asymmetry, Internet of things, Julian Assange, Lyft, Marc Andreessen, Mark Zuckerberg, Menlo Park, move fast and break things, Network effects, new economy, obamacare, Peter Thiel, Potemkin village, recommendation engine, Renaissance Technologies, Robert Mercer, Ronald Reagan, Rosa Parks, Sand Hill Road, Scientific racism, Shoshana Zuboff, side project, Silicon Valley, Skype, Stephen Fry, Steve Bannon, surveillance capitalism, tech bro, uber lyft, unpaid internship, Valery Gerasimov, web application, WikiLeaks, zero-sum game

But that, of course, was not why Nix gave them full access to the private data of hundreds of millions of American citizens. Nix’s dream, as he had confided in our very first meeting, was to become the “Palantir of propaganda.” One lead data scientist from Palantir began making regular trips to the Cambridge Analytica office to work with the data science team on building profiling models. He was occasionally accompanied by colleagues, but the entire arrangement was kept secret from the rest of the CA teams—and perhaps Palantir itself. I can’t speculate about why, but the Palantir staff received Cambridge Analytica database logins and emails with fairly obvious pseudonyms like “Dr.

…

Fresh out of university, I had taken a job at a London firm called SCL Group, which was supplying the U.K. Ministry of Defence and NATO armies with expertise in information operations. After Western militaries were grappling with how to tackle radicalization online, the firm wanted me to help build a team of data scientists to create new tools to identify and combat extremism online. It was fascinating, challenging, and exciting all at once. We were about to break new ground for the cyber defenses of Britain, America, and their allies and confront bubbling insurgencies of radical extremism with data, algorithms, and targeted narratives online.

…

It seemed entirely hypocritical to, on the one hand, frustrate jihadist groups in places like Pakistan and then, on the other, assist an autocratic and Islamist-backed regime in Egypt in creating its own tyranny of people. But Nix didn’t care. Business was business; he just wanted to clinch the deal. The main challenge for me and the growing team of psychologists and data scientists at SCL was in the objective substance of extremism itself. What does it mean to be an extremist? What exactly is extremism, and how can you model it? These were subjective definitions, and clearly the Egyptian government had one idea, while we had another. But if you want to be able to quantify and predict a trait, you have to be able to create a definition of it.

Artificial Unintelligence: How Computers Misunderstand the World by Meredith Broussard

"Susan Fowler" uber, 1960s counterculture, A Declaration of the Independence of Cyberspace, Ada Lovelace, AI winter, Airbnb, algorithmic bias, AlphaGo, Amazon Web Services, autonomous vehicles, availability heuristic, barriers to entry, Bernie Sanders, Big Tech, bitcoin, Buckminster Fuller, Charles Babbage, Chris Urmson, Clayton Christensen, cloud computing, cognitive bias, complexity theory, computer vision, Computing Machinery and Intelligence, crowdsourcing, Danny Hillis, DARPA: Urban Challenge, data science, deep learning, Dennis Ritchie, digital map, disruptive innovation, Donald Trump, Douglas Engelbart, driverless car, easy for humans, difficult for computers, Electric Kool-Aid Acid Test, Elon Musk, fake news, Firefox, gamification, gig economy, global supply chain, Google Glasses, Google X / Alphabet X, Greyball, Hacker Ethic, independent contractor, Jaron Lanier, Jeff Bezos, Jeremy Corbyn, John Perry Barlow, John von Neumann, Joi Ito, Joseph-Marie Jacquard, life extension, Lyft, machine translation, Mark Zuckerberg, mass incarceration, Minecraft, minimum viable product, Mother of all demos, move fast and break things, Nate Silver, natural language processing, Northpointe / Correctional Offender Management Profiling for Alternative Sanctions, One Laptop per Child (OLPC), opioid epidemic / opioid crisis, PageRank, Paradox of Choice, payday loans, paypal mafia, performance metric, Peter Thiel, price discrimination, Ray Kurzweil, ride hailing / ride sharing, Ross Ulbricht, Saturday Night Live, school choice, self-driving car, Silicon Valley, Silicon Valley billionaire, speech recognition, statistical model, Steve Jobs, Steven Levy, Stewart Brand, TechCrunch disrupt, Tesla Model S, the High Line, The Signal and the Noise by Nate Silver, theory of mind, traumatic brain injury, Travis Kalanick, trolley problem, Turing test, Uber for X, uber lyft, Watson beat the top human players on Jeopardy!, We are as Gods, Whole Earth Catalog, women in the workforce, work culture , yottabyte

I’m going to take you through a tutorial from a site called DataCamp, which was recommended as a first step for competing in data-science competitions by a different site, Kaggle.16 Kaggle, which is owned by Google’s parent company, Alphabet, is a site in which people compete to get the highest score for analyzing a dataset. Data scientists use it to compete in teams, sharpen their skills, or practice collaborating. It’s also useful for teaching students about data science or for finding datasets. We’re going to do a DataCamp Titanic tutorial using Python and a few popular Python libraries: pandas, scikit-learn, and numpy.

…

Pilhofer, “A Note to Users of DocumentCloud.” Acknowledgments Thank you to all the people who helped to bring this book to reality. I am grateful to my colleagues at New York University’s (NYU’s) Arthur L. Carter Journalism Institute, my colleagues at the Moore-Sloan Data Science Environment at NYU’s Center for Data Science, the faculty and staff at the Tow Center for Digital Journalism at Columbia Journalism School, and my former colleagues at Temple University and the University of Pennsylvania. For reading, consulting on, or otherwise midwifing this manuscript, I am eternally indebted to Elena Lahr-Vivaz, Rosalie Siegel, Jordan Ellenberg, Cathy O’Neil, Miriam Peskowitz, Samira Baird, Lori Tharps, Kira Baker-Doyle, Jane Dmochowski, Josephine Wolff, Solon Barocas, Hanna Wallach, Katy Boss, Janet Alteveer, Leslie Hunt, Elizabeth Hunt, Kay Kinsey, Karen Masse, Stevie Santangelo, Jay Kirk, Claire Wardle, Gita Manaktala, Melinda Rankin, Kathleen Caruso, Kyle Gipson, my writers’ group, and the talented team at the MIT Press.

…

There has never been, nor will there ever be, a technological innovation that moves us away from the essential problems of human nature. Why, then, do people persist in thinking there’s a sunny technological future just around the corner? I started thinking about technochauvinism one day when I was talking with a twenty-something friend who works as a data scientist. I mentioned something about Philadelphia schools that didn’t have enough books. “Why not just use laptops or iPads and get electronic textbooks?” asked my friend. “Doesn’t technology make everything faster, cheaper, and better?” He got an earful. (You’ll get one too in a later chapter.) However, his assumption stuck with me.

Exponential Organizations: Why New Organizations Are Ten Times Better, Faster, and Cheaper Than Yours (And What to Do About It) by Salim Ismail, Yuri van Geest

23andMe, 3D printing, Airbnb, Amazon Mechanical Turk, Amazon Web Services, anti-fragile, augmented reality, autonomous vehicles, Baxter: Rethink Robotics, behavioural economics, Ben Horowitz, bike sharing, bioinformatics, bitcoin, Black Swan, blockchain, Blue Ocean Strategy, book value, Burning Man, business intelligence, business process, call centre, chief data officer, Chris Wanstrath, circular economy, Clayton Christensen, clean water, cloud computing, cognitive bias, collaborative consumption, collaborative economy, commoditize, corporate social responsibility, cross-subsidies, crowdsourcing, cryptocurrency, dark matter, data science, Dean Kamen, deep learning, DeepMind, dematerialisation, discounted cash flows, disruptive innovation, distributed ledger, driverless car, Edward Snowden, Elon Musk, en.wikipedia.org, Ethereum, ethereum blockchain, fail fast, game design, gamification, Google Glasses, Google Hangouts, Google X / Alphabet X, gravity well, hiring and firing, holacracy, Hyperloop, industrial robot, Innovator's Dilemma, intangible asset, Internet of things, Iridium satellite, Isaac Newton, Jeff Bezos, Joi Ito, Kevin Kelly, Kickstarter, knowledge worker, Kodak vs Instagram, Law of Accelerating Returns, Lean Startup, life extension, lifelogging, loose coupling, loss aversion, low earth orbit, Lyft, Marc Andreessen, Mark Zuckerberg, market design, Max Levchin, means of production, Michael Milken, minimum viable product, natural language processing, Netflix Prize, NetJets, Network effects, new economy, Oculus Rift, offshore financial centre, PageRank, pattern recognition, Paul Graham, paypal mafia, peer-to-peer, peer-to-peer model, Peter H. Diamandis: Planetary Resources, Peter Thiel, Planet Labs, prediction markets, profit motive, publish or perish, radical decentralization, Ray Kurzweil, recommendation engine, RFID, ride hailing / ride sharing, risk tolerance, Ronald Coase, Rutger Bregman, Salesforce, Second Machine Age, self-driving car, sharing economy, Silicon Valley, skunkworks, Skype, smart contracts, Snapchat, social software, software is eating the world, SpaceShipOne, speech recognition, stealth mode startup, Stephen Hawking, Steve Jobs, Steve Jurvetson, subscription business, supply-chain management, synthetic biology, TaskRabbit, TED Talk, telepresence, telepresence robot, the long tail, Tony Hsieh, transaction costs, Travis Kalanick, Tyler Cowen, Tyler Cowen: Great Stagnation, uber lyft, urban planning, Virgin Galactic, WikiLeaks, winner-take-all economy, X Prize, Y Combinator, zero-sum game

For talented workers, working on and getting paid for multiple projects is a particularly welcome opportunity. But there’s another angle as well: an increase in the diversity of ideas. The data science company Kaggle, for example, offers a platform that hosts private and public algorithm contests in which more than 185,000 data scientists around the world vie for prizes and recognition. In 2011, Insurance giant Allstate, with forty of the best actuaries and data scientists money could buy, wanted to see if its claims algorithm could be improved upon, so it ran a contest on Kaggle. It turned out that the Allstate algorithm, which has been carefully optimized for over six decades, was bested within three days by 107 competing teams.

…

In fact, in every one of Kaggle’s 150 contests to date, external data scientists have beaten the internal algorithms, often by a wide margin. And in most cases outsiders (non-experts) have beaten the experts in a particular domain, which shows the power of fresh thinking and diverse perspectives. In years past, having a large workforce differentiated your enterprise and allowed it to accomplish more. Today, that same large workforce can become an anchor that reduces maneuverability and slows you down. Moreover, traditional industries have great difficulty attracting on-demand high-skill workers such as data scientists because the available positions are perceived as being low in terms of opportunity and high in terms of bureaucratic obstacles.

…

While growing exponentially as a company, Interfaces are critical if an organization is to scale seamlessly, especially on a global level. The same is true of other firms that coordinate data and oversee everything from prizes to personnel. Kaggle has its own unique mechanisms to manage its 200,000 data scientists. The X Prize Foundation has created mechanisms and dedicated teams for each of its prizes. TED has strict guidelines that help its many “franchised” TEDx events around the world deliver with consistency. And Uber has its own ways of handling its army of drivers. Most of these Interface processes are unique and proprietary to the organization that developed them, and as such comprise a unique type of intellectual property that can be of considerable market value.

The Elements of Data Analytic Style by Jeff Leek

correlation does not imply causation, data science, Netflix Prize, p-value, pattern recognition, Ronald Coase, statistical model, TED Talk

A large number of these are summarized in Karl Browman’s excellent presentation on displaying data badly. 11. Presenting data Giving data science talks can help you: Meet people Get people excited about your ideas/software/results Help people understand your ideas/software/results The importance of the first point can’t be overstated. The primary reason you are giving talks is for people to get to know you. Being well known and well regarded can make a huge range of parts of your job easier. So first and foremost make sure you don’t forget to talk to people before, after, and during your talk. Point 2 is more important than point 3. As a data scientist, it is hard to accept that the primary purpose of a talk is advertising, not data science.

…

As a data scientist, it is hard to accept that the primary purpose of a talk is advertising, not data science. See for example Hilary Mason’s great presentation Entertain, don’t teach. Here are reasons why entertainment is more important: That being said, be very careful to avoid giving a TED talk. If you are giving a data science presentation the goal is to communicate specific ideas. So while you are entertaining, don’t forget why you are entertaining. 11.1 Tailor your talk to your audience It depends on the event and the goals of the event. Here is a non-comprehensive list: Small group meeting: Goal: Update people you work with on what you are doing and get help.

…

Additional resources 15.1 Class lecture notes Johns Hopkins Data Science Specialization and Additional resources Data wrangling, exploration, and analysis with R Tools for Reproducible Research Data carpentry 15.2 Tutorials Git/github tutorial Make tutorial knitr in a knutshell Writing an R package from scratch 15.3 Leek group guides To data sharing To giving talks To developing R packages 15.4 Books An introduction to statistical learning Advanced data analysis from an elementary point of view Advanced R programming OpenIntro Statistics Statistical inference for data science

The Best Interface Is No Interface: The Simple Path to Brilliant Technology (Voices That Matter) by Golden Krishna

Airbnb, Bear Stearns, computer vision, crossover SUV, data science, en.wikipedia.org, fear of failure, impulse control, Inbox Zero, Internet Archive, Internet of things, Jeff Bezos, Jony Ive, Kickstarter, lock screen, Mark Zuckerberg, microdosing, new economy, Oculus Rift, off-the-grid, Paradox of Choice, pattern recognition, QR code, RFID, self-driving car, Silicon Valley, skeuomorphism, Skype, Snapchat, Steve Jobs, tech worker, technoutopianism, TED Talk, Tim Cook: Apple, Y Combinator, Y2K

People like Gordon Bell—through his MyLifeBits project at Microsoft Research Silicon Valley Laboratory—started collecting information about their own lives to beat memory loss.11 Over time, the digital medium’s incredible ability to remember built up to what today we call “big data.” Enter the world of the data scientist to find relevant meaning in all that digital data, to power solutions that adjust to your wonderful uniqueness. To think of you. What do you want to eat tonight? What’s the best way home? Data science is one way we can find meaning in all that cheaply stored information—whether big data or even small, relevant, searchable sets—and develop real insights and accurate answers to valuable individual questions.

…

In 2006, Jonathan Goldman, a PhD graduate in physics from Stanford working at the professional social network LinkedIn, wondered if, by studying you, the service could suggest connections you might know. Could he create an interface of smart connections built around you and your friends rather than putting you in a box as part of a generic database? Even at LinkedIn, where data scientist celebrity DJ Patil, um, coined the term “data scientist” with Jeff Hammerbacher back in 2008, the concept of creating platforms that adapted to your wonderfully distinct qualities seemed foreign just two years earlier.17 According to Patel, outspoken stakeholders in the company were said to have been “openly dismissive” of Goldman’s concept.

…

Rosenwald, “For Tablet Computer Visionary Roger Fidler, a Lot of What-Ifs,” Washington Post, March 10, 2012. http://www.washingtonpost.com/business/for-tablet-computer-visionary-roger-fidler-a-lot-of-what-ifs/2012/02/28/gIQAM0kN1R_story.html 17 “Some colleagues were openly dismissive of Goldman’s ideas. Why would users need LinkedIn to figure out their networks for them?” Thomas H. Davenport and D.J. Patil, “Data Scientist: The Sexiest Job of the 21st Century,” Harvard Business Review, October 2012. http://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century/ar/1 Chapter 15 Proactive Computing 1 “During the second World War, the robot was stored in the basement of the Weeks’s family home in Ohio, where he became 8-year-old Jack’s playmate.” Noel Sharkey, “Sign In To Read: The Return of Elektro, the First Celebrity Robot,” New Scientist, December 25, 2008. http://www.newscientist.com/article/mg20026873.000-the-return-of-elektro-the-first-celebrity-robot.html?

Outnumbered: From Facebook and Google to Fake News and Filter-Bubbles – the Algorithms That Control Our Lives by David Sumpter

affirmative action, algorithmic bias, AlphaGo, Bernie Sanders, Brexit referendum, Cambridge Analytica, classic study, cognitive load, Computing Machinery and Intelligence, correlation does not imply causation, crowdsourcing, data science, DeepMind, Demis Hassabis, disinformation, don't be evil, Donald Trump, Elon Musk, fake news, Filter Bubble, Geoffrey Hinton, Google Glasses, illegal immigration, James Webb Space Telescope, Jeff Bezos, job automation, Kenneth Arrow, Loebner Prize, Mark Zuckerberg, meta-analysis, Minecraft, Nate Silver, natural language processing, Nelson Mandela, Nick Bostrom, Northpointe / Correctional Offender Management Profiling for Alternative Sanctions, p-value, post-truth, power law, prediction markets, random walk, Ray Kurzweil, Robert Mercer, selection bias, self-driving car, Silicon Valley, Skype, Snapchat, social contagion, speech recognition, statistical model, Stephen Hawking, Steve Bannon, Steven Pinker, TED Talk, The Signal and the Noise by Nate Silver, traveling salesman, Turing test

He describes the Big Five personality traits; he outlines how surveys can be replaced by Facebook profiles; he claims that the results of one of his regression models reveals our conscientiousness and neuroticism; he talks about how political messages can be targeted to the individual and then he closes by claiming that ‘my model, given your Facebook likes, your age and your gender, can predict how agreeable you are just as well as your spouse’. One day, he says, we might fall in love with a computer that understands us better than our partner. I start to doubt whether the data scientist in the video truly believes what he is saying. I’m not sure he even expects his audience to believe it. His ‘research’ is the product of an eight-week training programme at ASI Data Science, a programme for aspiring data scientists. But even if this is just some sort of practice talk, I am deeply disturbed by what I see. This is a young man with the highest level of scientific training: a PhD in theoretical physics from Cambridge University.

…

The story of Cambridge Analytica took me deep into a web of blogs and privacy activists’ websites. Following these links, I found my way to a YouTube video of a young data scientist, who now works for Cambridge Analytica, presenting a research project he had carried out when working as an intern at the company. He starts his presentation with a reference to the film Her, in which the lead character Theodore, played by Joaquin Phoenix, falls in love with his operating system (OS). In the film, the computer forms a deep understanding of Theodore’s personality, and the human and the OS fall in love. The young data scientist uses this story to set up his own five-minute presentation: ‘Can a computer ever understand us better than a human?’

…

Glenn told me that the process of making recommendations is far from a pure science, ‘half of my job is trying to work out which computer-generated responses make sense’. When Glenn chose his job title, he asked to be called ‘data alchemist’ instead of ‘data scientist’. He sees his job not as searching for abstract truths about musical styles, but as providing classi­fications that make sense to people. This process requires humans and computers to work together. Given the vast scope of ‘Every Noise at Once’, Glenn’s modesty resonated strongly with me. Like many of the data scientists I had spoken to, he saw his job as navigating a very high-dimensional space. But he was the first person I had talked to who openly acknowledged the deeply personal and unknowable dimensions of our minds.

Tools and Weapons: The Promise and the Peril of the Digital Age by Brad Smith, Carol Ann Browne

"World Economic Forum" Davos, Affordable Care Act / Obamacare, AI winter, air gap, airport security, Alan Greenspan, Albert Einstein, algorithmic bias, augmented reality, autonomous vehicles, barriers to entry, Berlin Wall, Big Tech, Bletchley Park, Blitzscaling, Boeing 737 MAX, business process, call centre, Cambridge Analytica, Celtic Tiger, Charlie Hebdo massacre, chief data officer, cloud computing, computer vision, corporate social responsibility, data science, deep learning, digital divide, disinformation, Donald Trump, Eben Moglen, Edward Snowden, en.wikipedia.org, Hacker News, immigration reform, income inequality, Internet of things, invention of movable type, invention of the telephone, Jeff Bezos, Kevin Roose, Laura Poitras, machine readable, Mark Zuckerberg, minimum viable product, national security letter, natural language processing, Network effects, new economy, Nick Bostrom, off-the-grid, operational security, opioid epidemic / opioid crisis, pattern recognition, precision agriculture, race to the bottom, ransomware, Ronald Reagan, Rubik’s Cube, Salesforce, school vouchers, self-driving car, Sheryl Sandberg, Shoshana Zuboff, Silicon Valley, Skype, speech recognition, Steve Ballmer, Steve Jobs, surveillance capitalism, tech worker, The Rise and Fall of American Growth, Tim Cook: Apple, Wargames Reagan, WikiLeaks, women in the workforce

More than in the past this will require that those who create technology come not only from disciplines such as computer and data science but also from the social and natural sciences and humanities. If we’re to ensure that artificial intelligence makes decisions based on the best that humanity has to offer, its development must result from a multidisciplinary process. And as we think about the future of higher education, we’ll need to make certain that every computer and data scientist is exposed to the liberal arts, just as everyone who majors in the liberal arts will need a dose of computer and data science. We’ll also need to see more focus on ethics in computer and data science courses themselves.

…

Back to note reference 8. In 2018, we created a dedicated data science team to help us advance our work on key societal issues. We recruited one of Microsoft’s most experienced data scientists, John Kahan, to lead the team. He had led a large team that applied data analytics to track and analyze the company’s sales and product usage, and I had seen first-hand in weekly Senior Leadership Team meetings how this had improved our business performance. He also had a much broader set of interests, based in part on the work he and his team had pursued to use data science to better diagnose the causes of Sudden Infant Death Syndrome, or SIDS, to which John and his wife had lost their infant son, Aaron, more than a decade before.

…

He also had a much broader set of interests, based in part on the work he and his team had pursued to use data science to better diagnose the causes of Sudden Infant Death Syndrome, or SIDS, to which John and his wife had lost their infant son, Aaron, more than a decade before. Dina Bass, “Bereaved Father, Microsoft Data Scientists Crunch Numbers to Combat Infant Deaths,” Seattle Times, June 11, 2017, https://www.seattletimes.com/business/bereaved-father-microsoft-data-scientists-crunch-numbers-to-combat-infant-deaths/. One of the first projects we gave to the new team was to dig into the concerns we had developed regarding the FCC’s national data map on broadband availability. Within a few months, the team had used multiple data sets to analyze the broadband gap across the country, including data from the FCC and the Pew Research Center, as well as anonymized Microsoft data collected as part of ongoing work to improve the performance and security of our software and services.

Futureproof: 9 Rules for Humans in the Age of Automation by Kevin Roose

"World Economic Forum" Davos, adjacent possible, Airbnb, Albert Einstein, algorithmic bias, algorithmic management, Alvin Toffler, Amazon Web Services, Atul Gawande, augmented reality, automated trading system, basic income, Bayesian statistics, Big Tech, big-box store, Black Lives Matter, business process, call centre, choice architecture, coronavirus, COVID-19, data science, deep learning, deepfake, DeepMind, disinformation, Elon Musk, Erik Brynjolfsson, factory automation, fake news, fault tolerance, Frederick Winslow Taylor, Freestyle chess, future of work, Future Shock, Geoffrey Hinton, George Floyd, gig economy, Google Hangouts, GPT-3, hiring and firing, hustle culture, hype cycle, income inequality, industrial robot, Jeff Bezos, job automation, John Markoff, Kevin Roose, knowledge worker, Kodak vs Instagram, labor-force participation, lockdown, Lyft, mandatory minimum, Marc Andreessen, Mark Zuckerberg, meta-analysis, Narrative Science, new economy, Norbert Wiener, Northpointe / Correctional Offender Management Profiling for Alternative Sanctions, off-the-grid, OpenAI, pattern recognition, planetary scale, plutocrats, Productivity paradox, QAnon, recommendation engine, remote working, risk tolerance, robotic process automation, scientific management, Second Machine Age, self-driving car, Shoshana Zuboff, Silicon Valley, Silicon Valley startup, social distancing, Steve Jobs, Stuart Kauffman, surveillance capitalism, tech worker, The Future of Employment, The Wealth of Nations by Adam Smith, TikTok, Travis Kalanick, Uber and Lyft, uber lyft, universal basic income, warehouse robotics, Watson beat the top human players on Jeopardy!, work culture

Just by tweaking its algorithms, Netflix can steer users to its original shows, Amazon can steer users to its house brands, and Apple can recommend its own apps in the App Store, even when other apps might be preferable. The power to change users’ preferences at scale has made some technologists uncomfortable. Rachel Schutt, a data scientist, said as much in a 2012 interview with the Times: “Models do not just predict,” she said, “but they can make things happen.” A former product manager at Facebook went even further, telling BuzzFeed News that Facebook’s recommendation algorithms amounted to an attempt to “reprogram humans.” “It’s hard to believe that you could get humans to override all of their values that they came in with,” the former Facebook employee said.

…

They’re discouraging us from building the kind of personal autonomy that will protect us in the age of AI and automation, by allowing us to think and act for ourselves. And they’re doing it under the guise of helping us. In a 2017 paper about the history of Amazon’s recommendation algorithms, Amazon engineer Brent Smith and Microsoft data scientist Greg Linden sketched out a vision of the AI-driven future that feels, to me, both deeply dystopian and very, very plausible. “Every interaction should reflect who you are and what you like, and help you find what other people like you have already discovered,” they wrote. “It should feel hollow and pathetic when you see something that’s obviously not you; do you not know me by now?”

…

Near the end of the talk, LeCun made an unexpected prediction about the effects all of this AI and machine learning technology would have on the job market. Despite being a technologist himself, he said that the people with the best chances of coming out ahead in the economy of the future were not programmers and data scientists, but artists and artisans. To illustrate his point, he projected a slide with two photos: one of a Blu-Ray DVD player, which was being sold on Amazon for $47, and another of a handmade ceramic bowl, which was selling for $750. The difference in complexity between the two objects, he said, was stark.

The Patient Will See You Now: The Future of Medicine Is in Your Hands by Eric Topol

23andMe, 3D printing, Affordable Care Act / Obamacare, Anne Wojcicki, Atul Gawande, augmented reality, Big Tech, bioinformatics, call centre, Clayton Christensen, clean water, cloud computing, commoditize, computer vision, conceptual framework, connected car, correlation does not imply causation, creative destruction, crowdsourcing, dark matter, data acquisition, data science, deep learning, digital divide, disintermediation, disruptive innovation, don't be evil, driverless car, Edward Snowden, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, Firefox, gamification, global village, Google Glasses, Google X / Alphabet X, Ignaz Semmelweis: hand washing, information asymmetry, interchangeable parts, Internet of things, Isaac Newton, it's over 9,000, job automation, Julian Assange, Kevin Kelly, license plate recognition, lifelogging, Lyft, Mark Zuckerberg, Marshall McLuhan, meta-analysis, microbiome, Nate Silver, natural language processing, Network effects, Nicholas Carr, obamacare, pattern recognition, personalized medicine, phenotype, placebo effect, quantum cryptography, RAND corporation, randomized controlled trial, Salesforce, Second Machine Age, self-driving car, Silicon Valley, Skype, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, Snapchat, social graph, speech recognition, stealth mode startup, Steve Jobs, synthetic biology, the scientific method, The Signal and the Noise by Nate Silver, The Wealth of Nations by Adam Smith, traumatic brain injury, Turing test, Uber for X, uber lyft, Watson beat the top human players on Jeopardy!, WikiLeaks, X Prize

While there are a relatively small number of such professionals in a world inundated with data challenges in every sector, health care has not been able to attract enough gifted individuals proportional to the size and importance of this field. The irony now is that data scientists are the ones being referred to as the “high priests.”75b The iMedicine Galaxy Patients, companies, employers, doctors, government, data scientists: these major forces are like stars gravitationally bound in a galaxy, the movement of any one of them affecting all the others. What I have tried to convey here is that we have a new galaxy in the making. Just as the printing press was the great object around which modern culture has orbited, the smartphone and iMedicine are forcing a comparable transformation.

…

That seems a pretty grim prognosis. But medicine is morphing into a data science, now that big data, unsupervised algorithms, predictive analytics, machine learning, augmented reality, and neuromorphic computing are coming in. There’s still an opportunity to change medicine for the better and at least a chance for prevention. That is, if there was a surefire signal before a disease had ever manifested itself in a person—and this information was highly actionable—the individual’s illness might be preempted. This dream isn’t simply one of better data science, however. It is inextricably linked to the democratization of medicine.

…

Madrigal, “I’m Being Followed: How Google—and 104 Other Companies—Are Tracking Me on the Web,” The Atlantic, February 2014, http://www.theatlantic.com/technology/print/2012/02/im-being-foll%C951-and-104-other-companies-151-are-tracking-me-on-the-web/253758/. 26. S. Wolfram, “Data Science of the Facebook World,” Stephen Wolfram Blog, April 24, 2014, http://blog.stephenwolfram.com/2013/04/data-science-of-the-facebook-world/. 27. D. Mann, “1984 in 2014,” EP Studios Software, April 21, 2014, http://www.epstudiossoftware.com/?p=1411. 28. H. Kelly, “After Boston: The Pros and Cons of Surveillance Cameras,” CNN, April 26, 2013, http://www.cnn.com/2013/04/26/tech/innovation/security-cameras-boston-bombings/. 29.

Scikit-Learn Cookbook by Trent Hauck

bioinformatics, book value, computer vision, data science, information retrieval, p-value

ISBN 978-1-78398-948-5 www.packtpub.com www.it-ebooks.info Credits Author Project Coordinator Trent Hauck Harshal Ved Reviewers Proofreaders Anoop Thomas Mathew Simran Bhogal Xingzhong Bridget Braund Amy Johnson Commissioning Editor Kunal Parikh Indexer Tejal Soni Acquisition Editor Owen Roberts Graphics Sheetal Aute Content Development Editor Dayan Hyames Technical Editors Mrunal M. Chavan Dennis John Copy Editors Janbal Dharmaraj Ronak Dhruv Abhinash Sahu Production Coordinator Manu Joseph Cover Work Manu Joseph Sayanee Mukherjee www.it-ebooks.info About the Author Trent Hauck is a data scientist living and working in the Seattle area. He grew up in Wichita, Kansas and received his undergraduate and graduate degrees from the University of Kansas. He is the author of the book Instant Data Intensive Apps with pandas How-to, Packt Publishing—a book that can get you up to speed quickly with pandas and other associated technologies.

…

It's a pretty simple function: f ( x) = 1 1 + e−t 75 www.it-ebooks.info Working with Linear Models Visually, it looks like the following: Let's use the make_classification method, create a dataset, and get to classifying: >>> from sklearn.datasets import make_classification >>> X, y = make_classification(n_samples=1000, n_features=4) How to do it... The LogisticRegression object works in the same way as the other linear models: >>> from sklearn.linear_model import LogisticRegression >>> lr = LogisticRegression() Since we're good data scientists, we will pull out the last 200 samples to test the trained model on. Since this is a random dataset, it's fine to hold out the last 200; if you're dealing with structured data, don't do this (for example, if you deal with time series data): >>> >>> >>> >>> X_train = X[:-200] X_test = X[-200:] y_train = y[:-200] y_test = y[-200:] We'll discuss more on cross-validation later in the book.

…

Okay, so now that we've looked at how we can classify points based on distribution, let's look at how we can do this in scikit-learn: >>> >>> >>> >>> from sklearn.mixture import GMM gmm = GMM(n_components=2) X = np.row_stack((class_A, class_B)) y = np.hstack((np.ones(100), np.zeros(100))) Since we're good little data scientists, we'll create a training set: >>> train = np.random.choice([True, False], 200) >>> gmm.fit(X[train]) GMM(covariance_type='diag', init_params='wmc', min_covar=0.001, n_components=2, n_init=1, n_iter=100, params='wmc', random_state=None, thresh=0.01) 110 www.it-ebooks.info Chapter 3 Fitting and predicting is done in the same way as fitting is done for many of the other objects in scikit-learn: >>> gmm.fit(X[train]) >>> gmm.predict(X[train])[:5] array([0, 0, 0, 0, 0]) There are other methods worth looking at now that the model has been fit.

Applied Text Analysis With Python: Enabling Language-Aware Data Products With Machine Learning by Benjamin Bengfort, Rebecca Bilbro, Tony Ojeda

data science, full text search, natural language processing, quantitative easing, sentiment analysis, statistical model, the long tail

By specifically targeting text data about baseball or basketball, we reduce this ambiguity, but we also reduce the overall size of our corpus. This is a significant tradeoff, because we will need a very large corpus in order to provide sufficient training examples to our language models, thus we must find a balance between domain specificity and corpus size. Data Ingestion of Text As data scientists, we rely heavily on structure and patterns, not only in the content of our data, but in its history and provenance. In general, good data sources have a determinable structure, where different pieces of content are organized according to some schema and can be extracted systematically via the application of some logic to that schema.

…

Most modern web and social media services have APIs that developers can access, and they are typically accompanied by documentation with instructions on how to access and obtain the data. Note As a web service evolves, both the API and the documentation are usually updated as well, and as developers and data scientists, we need to stay current on changes to the APIs we use in our data products. A RESTful API is a type of web service API that adheres to representational state transfer (REST) architectural constraints. REST is a simple way to organize interactions between independent systems, allowing for lightweight interaction with clients such as mobile phones and other websites.

…

The primary reason they do this is so that they can prevent abuse of their service. Many service providers allow for registration using OAuth, which is an open authentication standard that allows a user’s information to be communicated to a third party without exposing confidential information such as their password. APIs are popular data sources among data scientists because they provide us with a source of ingestion that is authorized, structured, and well-documented. The service provider is giving us permission and access to retrieve and use the data they have in a responsible manner. This isn’t true of crawling/scraping or RSS, and for this reason, obtaining data via API is preferable whenever it is an option.

Facebook: The Inside Story by Steven Levy

active measures, Airbnb, Airbus A320, Amazon Mechanical Turk, AOL-Time Warner, Apple's 1984 Super Bowl advert, augmented reality, Ben Horowitz, Benchmark Capital, Big Tech, Black Lives Matter, Blitzscaling, blockchain, Burning Man, business intelligence, Cambridge Analytica, cloud computing, company town, computer vision, crowdsourcing, cryptocurrency, data science, deep learning, disinformation, don't be evil, Donald Trump, Dunbar number, East Village, Edward Snowden, El Camino Real, Elon Musk, end-to-end encryption, fake news, Firefox, Frank Gehry, Geoffrey Hinton, glass ceiling, GPS: selective availability, growth hacking, imposter syndrome, indoor plumbing, information security, Jeff Bezos, John Markoff, Jony Ive, Kevin Kelly, Kickstarter, lock screen, Lyft, machine translation, Mahatma Gandhi, Marc Andreessen, Marc Benioff, Mark Zuckerberg, Max Levchin, Menlo Park, Metcalfe’s law, MITM: man-in-the-middle, move fast and break things, natural language processing, Network effects, Oculus Rift, operational security, PageRank, Paul Buchheit, paypal mafia, Peter Thiel, pets.com, post-work, Ray Kurzweil, recommendation engine, Robert Mercer, Robert Metcalfe, rolodex, Russian election interference, Salesforce, Sam Altman, Sand Hill Road, self-driving car, sexual politics, Sheryl Sandberg, Shoshana Zuboff, side project, Silicon Valley, Silicon Valley startup, skeuomorphism, slashdot, Snapchat, social contagion, social graph, social software, South of Market, San Francisco, Startup school, Steve Ballmer, Steve Bannon, Steve Jobs, Steven Levy, Steven Pinker, surveillance capitalism, tech billionaire, techlash, Tim Cook: Apple, Tragedy of the Commons, web application, WeWork, WikiLeaks, women in the workforce, Y Combinator, Y2K, you are the product

For the 2010 midterm election, Facebook expanded the program, with a prominent button proclaiming “I Voted” visible to users. But not all visitors. Facebook used the midterms to conduct an elaborate experiment. Two of Facebook’s top data scientists, working with researchers at the University of California at San Diego, decided to test whether the voter button actually affected voter turnout. If you saw that your friends voted, would it influence you to do the same? Cameron Marlow, who headed Data Science for Facebook at the time, says the experiment was an innocent exercise: “We had a product that had run in every single election and we were starting to run in other countries’ elections—the goal was to get people out to vote.”

…

The work still continued—after all, research led to Growth!—but the company did not want to be misunderstood again. “I don’t think that they stopped doing experiments—they just stopped publishing them,” says Cameron Marlow, who headed Data Science but left shortly before the emotion paper was published. “So is that a good thing for society? Probably not.” As I found by informal conversations at a Data Science conference on campus in 2019, though, most of its researchers stuck around. They feel that their work is important. * * * • • • BEGINNING IN 2013, Kogan was visiting the Facebook campus regularly. He ate a lot of free lunches.

…

He became a co-founder of Cloudera, a company that stored data in the Internet cloud, and later became involved in trying to solve cancer with data analysis. Though he felt no animosity toward his former employer, at times he has expressed its motivations in phrases that speak volumes. In 2011, assessing the jobs of data scientists at Facebook and its peers, he made a remark to a BusinessWeek journalist that would reverberate for years: “The best minds of my generation are thinking about how to make people click ads,” he said. “That sucks.” * * * • • • IN THE OLD television show Mission: Impossible, each episode began when the leader, Jim, flipped through the dossiers of spies, strongmen, and honeypots, putting rejects in one pile and tossing the photos of the talented ones who were perfect for the mission into a stack on his coffee table.

Democracy for Sale: Dark Money and Dirty Politics by Peter Geoghegan

4chan, Adam Curtis, Affordable Care Act / Obamacare, American Legislative Exchange Council, anti-globalists, basic income, Berlin Wall, Big Tech, Black Lives Matter, Boris Johnson, Brexit referendum, British Empire, Cambridge Analytica, centre right, corporate raider, crony capitalism, data science, deepfake, deindustrialization, demographic winter, disinformation, Dominic Cummings, Donald Trump, East Village, Etonian, F. W. de Klerk, fake news, first-past-the-post, Francis Fukuyama: the end of history, Frank Gehry, Greta Thunberg, invisible hand, James Dyson, Jeremy Corbyn, John Bercow, Mark Zuckerberg, market fundamentalism, military-industrial complex, moral panic, Naomi Klein, Nelson Mandela, obamacare, offshore financial centre, open borders, Overton Window, Paris climate accords, plutocrats, post-truth, post-war consensus, pre–internet, private military company, Renaissance Technologies, Robert Mercer, Ronald Reagan, Silicon Valley, Snapchat, special economic zone, Steve Bannon, surveillance capitalism, tech billionaire, technoutopianism, Torches of Freedom, universal basic income, WikiLeaks, Yochai Benkler, éminence grise

“We could say, for example, we will target women between 35 and 45 who live in these particular geographical entities, who don’t have a degree,” Cummings later explained. He boasted of using physicists and experts in “quantum information” to crunch voter data. Vote Leave recorded spending over £70,000 with a firm called Advanced Skills Initiative.47 The company is better known as ASI Data Science, a tech start-up that marketed itself as a world leader in artificial intelligence, and which employed a number of data scientists that worked for Cambridge Analytica. Cummings also came up with clever ruses to find data on the “missing three million”. He ran an online competition during the 2016 European Football Championship. Win £50 million, the advertisement proclaimed, by successfully predicting the winner of all 51 games.

…

See also https://firstdraftnews.org/latest/thousands-of-misleading-conservative-ads-side-step-scrutiny-thanks-to-facebook-policy/; accessed 26 Jan. 2020. 44 Ella Hollowood and Matthew D’Ancona, ‘Big little lies’, Tortoise, December 2019. See also https://members.tortoisemedia.com/2019/12/11/lies-191211/content.html; accessed 26 Jan. 2020. 45 Dominic Cummings, ‘“Two hands are a lot” – we’re hiring data scientists, project managers, policy experts, assorted weirdos…’, Dominic Cummings’s Blog, January 2020. See also https://dominiccummings.com/2020/01/02/two-hands-are-a-lot-were-hiring-data-scientists-project-managers-policy-experts-assorted-weirdos/; accessed 26 Jan. 2020. 46 Rowland Manthorpe, ‘General election: WhatsApp messages urge British Hindus to vote against Labour’, Sky News, November 2019. 47 Jim Waterson, ‘What we learned about the media this election’, Guardian, December 2019. 48 James Cusick, ‘New evidence that LibDems sold voter data for £100,000 held back till after election’, openDemocracy, November 2019. 49 Rowland Manthorpe, ‘Data protection experts want watchdog to investigate Conservative and Labour parties’, Sky News, October 2019. 50 ‘General election 2019: Zac Goldsmith loses seat to Lib Dems again’, BBC, December 2019. 51 Kate Proctor, ‘Johnson accused of “rewarding racism” after Zac Goldsmith peerage’, Guardian, December 2019. 52 Isobel Thompson, ‘How Irish anti-abortion activists are drawing on Brexit and Trump campaigns to influence referendum’, openDemocracy, May 2018. 53 ‘Republicans Overseas UK: An Evening with GOP Strategist Matt Mackowiak’, Republicans Overseas UK.

…

Dominic Cummings Said It’s “TOP PRIORITY”’, Buzzfeed, September 2019. 3 William Norton, White Elephant: How the North East Said No (London, 2008), p. 200. 4 Dominic Cummings, ‘On the referendum #20: the campaign, physics and data science – Vote Leave’s ‘Voter Intention Collection System’ (VICS) now available for all’, Dominic Cummings’s Blog, October 2016. See also https://dominiccummings.com/2016/10/29/on-the-referendum-20-the-campaign-physics-and-data-science-vote-leaves-voter-intention-collection-system-vics-now-available-for-all/; accessed 19 Jan. 2020. 5 Alice Thomson and Rachel Sylvester, ‘Sir Nicholas Soames interview: “Johnson is nothing like Churchill and Jacob Rees-Mogg is an absolute fraud”’, The Times, September 2019. 6 Sam Knight, ‘The man who brought you Brexit’, Guardian, September 2016. 7 Tim Shipman, All Out War: The Full Story of Brexit (London, 2017), p. 27. 8 George Eaton, ‘Vote Leave head Matthew Elliott: “The Brexiteers won the battle but we could lose the war”’, New Statesman, September 2018. 9 Chloe Farand and Mat Hope, ‘Matthew and Sarah Elliott: How a UK Power Couple Links US Libertarians and Fossil Fuel Lobbyists to Brexit’, DeSmog UK, November 2018. 10 Robert Booth, ‘Who is behind the Taxpayers’ Alliance?’

Hacking Growth: How Today's Fastest-Growing Companies Drive Breakout Success by Sean Ellis, Morgan Brown

Airbnb, Amazon Web Services, barriers to entry, behavioural economics, Ben Horowitz, bounce rate, business intelligence, business process, content marketing, correlation does not imply causation, crowdsourcing, dark pattern, data science, DevOps, disruptive innovation, Elon Musk, game design, gamification, Google Glasses, growth hacking, Internet of things, inventory management, iterative process, Jeff Bezos, Khan Academy, Kickstarter, Lean Startup, Lyft, Mark Zuckerberg, market design, minimum viable product, multi-armed bandit, Network effects, Paul Graham, Peter Thiel, Ponzi scheme, recommendation engine, ride hailing / ride sharing, Salesforce, Sheryl Sandberg, side project, Silicon Valley, Silicon Valley startup, Skype, Snapchat, software as a service, Steve Jobs, Steve Jurvetson, subscription business, TED Talk, Travis Kalanick, Uber and Lyft, Uber for X, uber lyft, working poor, Y Combinator, young professional

The growth hacking practices innovated by these early practitioners and others who have followed have been honed into a finely tuned business methodology—and spawned a powerful movement with hundreds of thousands of practitioners (and growing) across the globe. This vibrant community of growth hackers includes entrepreneurs, marketers, engineers, product managers, data scientists, and more, not just from the tech start-up world but from all walks of industry, from technology, retail, business-to-business, professional services, entertainment, and even the political arena. And while the details of how it is implemented vary somewhat from company to company, the core elements of the method are: • the creation of a cross-functional team, or a set of teams that break down the traditional silos of marketing and product development and combine talents; • the use of qualitative research and quantitative data analysis to gain deep insights into user behavior and preferences; and • the rapid generation and testing of ideas, and the use of rigorous metrics to evaluate—and then act on—those results.

…

Recognizing that Walmart’s greatest asset is its data, Brian Monahan, the company’s former VP of marketing, pushed forward a unification of the company’s data platforms across all divisions, one that would allow all teams, from engineering, to merchandising, to marketing, and even external agencies and suppliers, to capitalize on the data generated and collected. Growth hacking cultivates the maximization of big data through collaboration and information sharing. Monahan highlighted the business need this approach solves: “You need marketers who can appreciate what it takes to actually write software and you need data scientists who can really appreciate consumer insights and understand business problems,” he explained.19 THE RISING COSTS AND DUBIOUS RETURNS OF TRADITIONAL MARKETING The techniques of traditional marketing—both print and television advertising, and the newer online versions that have become essential parts of the traditional marketing toolkit—are in crisis, as markets are becoming more and more fragmented and ephemeral, while advertising is becoming both more expensive and less viewed.

…

Depending on the degree of sophistication of the experiments a team is running, it might be possible for the marketing or engineering team member to play this role, as in both of those fields, a certain level of data analytics aptitude has become important. At more technically advanced companies, analysts with expertise in reporting of experiments as well as data scientists, who are mining for deep insight, should both play a role. What is essential is that data analysis not be farmed out to the intern who knows how to use Google Analytics or to a digital agency, to cite extreme but all too frequent realities. As we will discuss in detail coming up in Chapter Three, too many companies do not place enough emphasis on data analysis, and rely too heavily on prepackaged programs, such as Google Analytics, with limited capacity for combining various pools of data, such as from sales and from customer service, and limited ability to delve into that data to make discoveries.

Nervous States: Democracy and the Decline of Reason by William Davies

active measures, Affordable Care Act / Obamacare, Amazon Web Services, Anthropocene, bank run, banking crisis, basic income, Black Lives Matter, Brexit referendum, business cycle, Cambridge Analytica, Capital in the Twenty-First Century by Thomas Piketty, citizen journalism, Climategate, Climatic Research Unit, Colonization of Mars, continuation of politics by other means, creative destruction, credit crunch, data science, decarbonisation, deep learning, DeepMind, deindustrialization, digital divide, discovery of penicillin, Dominic Cummings, Donald Trump, drone strike, Elon Musk, failed state, fake news, Filter Bubble, first-past-the-post, Frank Gehry, gig economy, government statistician, housing crisis, income inequality, Isaac Newton, Jeff Bezos, Jeremy Corbyn, Johannes Kepler, Joseph Schumpeter, knowledge economy, loss aversion, low skilled workers, Mahatma Gandhi, Mark Zuckerberg, mass immigration, meta-analysis, Mont Pelerin Society, mutually assured destruction, Northern Rock, obamacare, Occupy movement, opioid epidemic / opioid crisis, Paris climate accords, pattern recognition, Peace of Westphalia, Peter Thiel, Philip Mirowski, planetary scale, post-industrial society, post-truth, quantitative easing, RAND corporation, Ray Kurzweil, Richard Florida, road to serfdom, Robert Mercer, Ronald Reagan, sentiment analysis, Silicon Valley, Silicon Valley billionaire, Silicon Valley startup, smart cities, Social Justice Warrior, statistical model, Steve Bannon, Steve Jobs, tacit knowledge, the scientific method, Turing machine, Uber for X, universal basic income, University of East Anglia, Valery Gerasimov, W. E. B. Du Bois, We are the 99%, WikiLeaks, women in the workforce, zero-sum game

Behind the scenes, this is gobbled up and mathematically processed. As the math has become more and more sophisticated, the user no longer even experiences it as mathematical. From science to data science A curiosity of big-data analytics is that its specialists are relatively uninterested in whether the data is generated by people, particles in the atmosphere, cars, financial prices, or bacteria. Data scientists are more often trained in mathematics or physics than in social science. They generate knowledge about our behavior, but they don’t profess any expertise about people, or shopping, or finance, or cities.

…

They don’t study nature or society, in the way that the archetypal expert does, but seek patterns in data that computers have already captured. As opposed to a scientist, a data scientist might better be compared to a librarian, someone who is skilled in navigating a vast collection of already-recorded information. The difference is that the data archive is growing at great speed, thanks to the mass of nonhuman sensors that gather it, and can only be sifted algorithmically. Take the example of psychology. Data science reveals a great deal that is of interest to psychologists, given the ability of algorithms to detect emotions, behaviors, and anxieties across populations.

…

The analyst’s value lies in pruning vast quantities of useless data, leaving only that which deserves our attention.16 But if they lack any intrinsic interest in the topic at hand (other than the mathematics), they also have no view of their own regarding what “something meaningful” means—they are therefore in the service of a client. Alternatively, their biases and assumptions creep in, without being consciously reflected on or criticized.17 The clients for data science are multiplying all the time. “Quants” can make big money working for Wall Street banks and hedge funds, building algorithms to analyze price movements. “Smart city” projects depend on data scientists to extract patterns of activity from the frenetic movements of urban populations, resources, and transport. Firms such as Peter Thiel’s Palantir help security services identify potential security threats, by isolating dangerous patterns of behavior.

Data and the City by Rob Kitchin,Tracey P. Lauriault,Gavin McArdle

A Declaration of the Independence of Cyberspace, algorithmic management, bike sharing, bitcoin, blockchain, Bretton Woods, Chelsea Manning, citizen journalism, Claude Shannon: information theory, clean water, cloud computing, complexity theory, conceptual framework, corporate governance, correlation does not imply causation, create, read, update, delete, crowdsourcing, cryptocurrency, data science, dematerialisation, digital divide, digital map, digital rights, distributed ledger, Evgeny Morozov, fault tolerance, fiat currency, Filter Bubble, floating exchange rates, folksonomy, functional programming, global value chain, Google Earth, Hacker News, hive mind, information security, Internet of things, Kickstarter, knowledge economy, Lewis Mumford, lifelogging, linked data, loose coupling, machine readable, new economy, New Urbanism, Nicholas Carr, nowcasting, open economy, openstreetmap, OSI model, packet switching, pattern recognition, performance metric, place-making, power law, quantum entanglement, RAND corporation, RFID, Richard Florida, ride hailing / ride sharing, semantic web, sentiment analysis, sharing economy, Silicon Valley, Skype, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, smart contracts, smart grid, smart meter, social graph, software studies, statistical model, tacit knowledge, TaskRabbit, technological determinism, technological solutionism, text mining, The Chicago School, The Death and Life of Great American Cities, the long tail, the market place, the medium is the message, the scientific method, Toyota Production System, urban planning, urban sprawl, web application

Since the architecture provides flexible techniques for data and analysis sharing, communications within organizations, between organizations and between citizens and organizations can be improved. This is an important requirement for cities and citizens. With the rise of citizen data scientist and the corporate use of urban data, sharing data about cities using various bindings is advantageous. The architecture also can efficiently communicate with big data technologies. Current city dashboards are useful tool for now-casting. With big data technologies and data science, cities need to have other systems for sharing data using polyglot bindings and providing indicators and metrics about city for future (forecasting) using predictive analytics.

…

British Academy (2012) ‘Society Counts – Quantitative Studies in the Social Sciences and Humanities’, A British Academy Position Statement available from: www.britac.ac.uk/ policy/Society_Counts.cfm [accessed 9 December 2016]. Burkert, H. (1992) ‘The legal framework of public sector information: Recent legal policy developments in the EC’, Government Publications Review 19(5): 483–496. Cabinet Office (2015) ‘Open policy making toolkit: Data science’, available from: www. gov.uk/open-policy-making-toolkit-data-science [accessed 9 December 2016]. Clarke, R. (1988) ‘Information technology and dataveillance’, Communications of the ACM 31(5): 498–512. Crampton, J. and Krygier, J. (2005) ‘An introduction to critical cartography’, ACME: An International E-Journal for Critical Geographies 4(1), available from: http://ojs.unbc. ca/index.php/acme/article/view/723/585 [accessed 9 December 2016].

…

Moreover, further research is required to understand how data influence digital labour, investigating issues such as how institutional and organizational structures change with the introduction of new databased regimes, how data ecosystems change government and corporate work practices, and how the database managers and data scientists become more important within institutions with their knowledge and expertise becoming privileged over others. Epistemology As the chapters make clear, there are a diverse set of epistemologies being deployed to make sense of urban data, data-driven systems, and the relationship between data and the city.

Beautiful Data: The Stories Behind Elegant Data Solutions by Toby Segaran, Jeff Hammerbacher

23andMe, airport security, Amazon Mechanical Turk, bioinformatics, Black Swan, business intelligence, card file, cloud computing, computer vision, correlation coefficient, correlation does not imply causation, crowdsourcing, Daniel Kahneman / Amos Tversky, DARPA: Urban Challenge, data acquisition, data science, database schema, double helix, en.wikipedia.org, epigenetics, fault tolerance, Firefox, Gregor Mendel, Hans Rosling, housing crisis, information retrieval, lake wobegon effect, Large Hadron Collider, longitudinal study, machine readable, machine translation, Mars Rover, natural language processing, openstreetmap, Paradox of Choice, power law, prediction markets, profit motive, semantic web, sentiment analysis, Simon Singh, social bookmarking, social graph, SPARQL, sparse data, speech recognition, statistical model, supply-chain management, systematic bias, TED Talk, text mining, the long tail, Vernor Vinge, web application

The most critical human component for accelerating the learning process and making use of the Information Platform is taking the shape of a new role: the Data Scientist. The Data Scientist In a recent interview, Hal Varian, Google’s chief economist, highlighted the need for employees able to extract information from the Information Platforms described earlier. As Varian puts it, “find something where you provide a scarce, complementary service to something that is getting ubiquitous and cheap. So what’s getting ubiquitous and cheap? Data. And what is complementary to data? Analysis.” INFORMATION PLATFORMS AND THE RISE OF THE DATA SCIENTIST Download at Boykma.Com 83 At Facebook, we felt that traditional titles such as Business Analyst, Statistician, Engineer, and Research Scientist didn’t quite capture what we were after for our team.

…

The workload for the role was diverse: on any given day, a team member could author a multistage processing pipeline in Python, design a hypothesis test, perform a regression analysis over data samples with R, design and implement an algorithm for some data-intensive product or service in Hadoop, or communicate the results of our analyses to other members of the organization in a clear and concise fashion. To capture the skill set required to perform this multitude of tasks, we created the role of “Data Scientist.” In the financial services domain, large data stores of past market activity are built to serve as the proving ground for complex new models developed by the Data Scientists of their domain, known as Quants. Outside of industry, I’ve found that grad students in many scientific domains are playing the role of the Data Scientist. One of our hires for the Facebook Data team came from a bioinformatics lab where he was building data pipelines and performing offline data analysis of a similar kind.

…

Recent books such as Davenport and Harris’s Competing on Analytics (Harvard Business School Press, 2007), Baker’s The Numerati (Houghton Mifflin Harcourt, 2008), and Ayres’s Super Crunchers (Bantam, 2008) have emphasized the critical role of the Data Scientist across industries in enabling an organization to improve over time based on the information it collects. In conjunction with the research community’s investigation of dataspaces, further definition for the role of the Data Scientist is needed over the coming years. By better articulating the role, we’ll be able to construct training curricula, formulate promotion hierarchies, organize conferences, write books, and fill in all of the other trappings of a recognized profession. In the process, the pool of available Data Scientists will expand to meet the growing need for expert pilots for the rapidly proliferating Information Platforms, further speeding the learning process across all organizations.

The Art of Statistics: Learning From Data by David Spiegelhalter

Abraham Wald, algorithmic bias, Anthropocene, Antoine Gombaud: Chevalier de Méré, Bayesian statistics, Brexit referendum, Carmen Reinhart, Charles Babbage, complexity theory, computer vision, confounding variable, correlation coefficient, correlation does not imply causation, dark matter, data science, deep learning, DeepMind, Edmond Halley, Estimating the Reproducibility of Psychological Science, government statistician, Gregor Mendel, Hans Rosling, Higgs boson, Kenneth Rogoff, meta-analysis, Nate Silver, Netflix Prize, Northpointe / Correctional Offender Management Profiling for Alternative Sanctions, p-value, placebo effect, probability theory / Blaise Pascal / Pierre de Fermat, publication bias, randomized controlled trial, recommendation engine, replication crisis, self-driving car, seminal paper, sparse data, speech recognition, statistical model, sugar pill, systematic bias, TED Talk, The Design of Experiments, The Signal and the Noise by Nate Silver, The Wisdom of Crowds, Thomas Bayes, Thomas Malthus, Two Sigma

This common view of statistics as a basic ‘bag of tools’ is now facing major challenges. First, we are in an age of data science, in which large and complex data sets are collected from routine sources such as traffic monitors, social media posts and internet purchases, and used as a basis for technological innovations such as optimizing travel routes, targeted advertising or purchase recommendation systems – we shall look at algorithms based on ‘big data’ in Chapter 6. Statistical training is increasingly seen as just one necessary component of being a data scientist, together with skills in data management, programming and algorithm development, as well as proper knowledge of the subject matter.

…

cox regression: See hazard ratio. data literacy: the ability to understand the principles behind learning from data, carry out basic data analyses, and critique the quality of claims made on the basis of data. data science: the study and application of techniques for deriving insights from data, including constructing algorithms for prediction. Traditional statistical science forms part of data science, which also includes a strong element of coding and data management. deep learning: a machine-learning technique that extends standard artificial neural network models to many layers representing different levels of abstraction, say going from individual pixels of an image through to recognition of objects.

…

Any conclusions generally raise more questions, and so the cycle starts over again, as when we started looking at the time of day when Shipman’s patients died. Although in practice the PPDAC cycle laid out in Figure 0.3 may not be followed precisely, it underscores that formal techniques for statistical analysis play only one part in the work of a statistician or data scientist. Statistical science is a lot more than a branch of mathematics involving esoteric formulae with which generations of students have (often reluctantly) struggled. This Book When I was a student in Britain in the 1970s, there were just three TV channels, computers were the size of a double wardrobe, and the closest thing we had to Wikipedia was on the imaginary handheld device in Douglas Adams’ (remarkably prescient) Hitchhiker’s Guide to the Galaxy.

The Art of Statistics: How to Learn From Data by David Spiegelhalter

Abraham Wald, algorithmic bias, Antoine Gombaud: Chevalier de Méré, Bayesian statistics, Brexit referendum, Carmen Reinhart, Charles Babbage, complexity theory, computer vision, confounding variable, correlation coefficient, correlation does not imply causation, dark matter, data science, deep learning, DeepMind, Edmond Halley, Estimating the Reproducibility of Psychological Science, government statistician, Gregor Mendel, Hans Rosling, Higgs boson, Kenneth Rogoff, meta-analysis, Nate Silver, Netflix Prize, Northpointe / Correctional Offender Management Profiling for Alternative Sanctions, p-value, placebo effect, probability theory / Blaise Pascal / Pierre de Fermat, publication bias, randomized controlled trial, recommendation engine, replication crisis, self-driving car, seminal paper, sparse data, speech recognition, statistical model, sugar pill, systematic bias, TED Talk, The Design of Experiments, The Signal and the Noise by Nate Silver, The Wisdom of Crowds, Thomas Bayes, Thomas Malthus, Two Sigma

This common view of statistics as a basic ‘bag of tools’ is now facing major challenges. First, we are in an age of data science, in which large and complex data sets are collected from routine sources such as traffic monitors, social media posts and internet purchases, and used as a basis for technological innovations such as optimizing travel routes, targeted advertising or purchase recommendation systems—we shall look at algorithms based on ‘big data’ in Chapter 6. Statistical training is increasingly seen as just one necessary component of being a data scientist, together with skills in data management, programming and algorithm development, as well as proper knowledge of the subject matter.

…

cox regression: See hazard ratio. data literacy: the ability to understand the principles behind learning from data, carry out basic data analyses, and critique the quality of claims made on the basis of data. data science: the study and application of techniques for deriving insights from data, including constructing algorithms for prediction. Traditional statistical science forms part of data science, which also includes a strong element of coding and data management. deep learning: a machine-learning technique that extends standard artificial neural network models to many layers representing different levels of abstraction, say going from individual pixels of an image through to recognition of objects.

…

Any conclusions generally raise more questions, and so the cycle starts over again, as when we started looking at the time of day when Shipman’s patients died. Although in practice the PPDAC cycle laid out in Figure 0.3 may not be followed precisely, it underscores that formal techniques for statistical analysis play only one part in the work of a statistician or data scientist. Statistical science is a lot more than a branch of mathematics involving esoteric formulae with which generations of students have (often reluctantly) struggled. This Book When I was a student in Britain in the 1970s, there were just three TV channels, computers were the size of a double wardrobe, and the closest thing we had to Wikipedia was on the imaginary handheld device in Douglas Adams’ (remarkably prescient) Hitchhiker’s Guide to the Galaxy.

Calling Bullshit: The Art of Scepticism in a Data-Driven World by Jevin D. West, Carl T. Bergstrom

airport security, algorithmic bias, AlphaGo, Amazon Mechanical Turk, Andrew Wiles, Anthropocene, autism spectrum disorder, bitcoin, Charles Babbage, cloud computing, computer vision, content marketing, correlation coefficient, correlation does not imply causation, crowdsourcing, cryptocurrency, data science, deep learning, deepfake, delayed gratification, disinformation, Dmitri Mendeleev, Donald Trump, Elon Musk, epigenetics, Estimating the Reproducibility of Psychological Science, experimental economics, fake news, Ford Model T, Goodhart's law, Helicobacter pylori, Higgs boson, invention of the printing press, John Markoff, Large Hadron Collider, longitudinal study, Lyft, machine translation, meta-analysis, new economy, nowcasting, opioid epidemic / opioid crisis, p-value, Pluto: dwarf planet, publication bias, RAND corporation, randomized controlled trial, replication crisis, ride hailing / ride sharing, Ronald Reagan, selection bias, self-driving car, Silicon Valley, Silicon Valley startup, social graph, Socratic dialogue, Stanford marshmallow experiment, statistical model, stem cell, superintelligent machines, systematic bias, tech bro, TED Talk, the long tail, the scientific method, theory of mind, Tim Cook: Apple, twin studies, Uber and Lyft, Uber for X, uber lyft, When a measure becomes a target

* * * — WE HAVE DEVOTED OUR careers to teaching students how to think logically and quantitatively about data. This book emerged from a course we teach at the University of Washington, also titled “Calling Bullshit.” We hope it will show you that you do not need to be a professional statistician or econometrician or data scientist to think critically about quantitative arguments, nor do you need extensive data sets and weeks of effort to see through bullshit. It is often sufficient to apply basic logical reasoning to a problem and, where needed, augment that with information readily discovered via search engine. We have civic motives for wanting to help people spot and refute bullshit.

…

If you do know some of these things, you still probably don’t remember all of the details. We, the authors, use statistics on a daily basis, but we still have to look up this sort of stuff all the time. As a result, you can’t unpack the black box; you can’t go into the details of the analysis in order to pick apart possible problems. Unless you’re a data scientist, and probably even then, you run into the same kind of problem you encounter when you read about a paper that uses the newest ResNet algorithm to reveal differences in the facial features of dog and cat owners. Whether or not this is intentional on the part of the author, this kind of black box shields the claim against scrutiny.

…

If we continue the time series across the intervening years since Vigen published this figure, the correlation completely falls apart. Vigen finds his spurious correlation examples by collecting a large number of data sets about how things change over time. Then he uses a computer program to compare each trend with every other trend. This is an extreme form of what data scientists call data dredging. With a mere one hundred data series, one can compare nearly ten thousand pairs. Some of these pairs are going to show very similar trends—and thus high correlations—just by chance. For example, check out the correlation between the numbers of deaths caused by anticoagulants and the number of sociology degrees awarded in the US: You look at these two trends and think, Wow—what are the chances they would line up that well?

The Hype Machine: How Social Media Disrupts Our Elections, Our Economy, and Our Health--And How We Must Adapt by Sinan Aral

Airbnb, Albert Einstein, algorithmic bias, AlphaGo, Any sufficiently advanced technology is indistinguishable from magic, AOL-Time Warner, augmented reality, behavioural economics, Bernie Sanders, Big Tech, bitcoin, Black Lives Matter, Cambridge Analytica, carbon footprint, Cass Sunstein, computer vision, contact tracing, coronavirus, correlation does not imply causation, COVID-19, crowdsourcing, cryptocurrency, data science, death of newspapers, deep learning, deepfake, digital divide, digital nomad, disinformation, disintermediation, Donald Trump, Drosophila, Edward Snowden, Elon Musk, en.wikipedia.org, end-to-end encryption, Erik Brynjolfsson, experimental subject, facts on the ground, fake news, Filter Bubble, George Floyd, global pandemic, hive mind, illegal immigration, income inequality, Kickstarter, knowledge worker, lockdown, longitudinal study, low skilled workers, Lyft, Mahatma Gandhi, Mark Zuckerberg, Menlo Park, meta-analysis, Metcalfe’s law, mobile money, move fast and break things, multi-sided market, Nate Silver, natural language processing, Neal Stephenson, Network effects, performance metric, phenotype, recommendation engine, Robert Bork, Robert Shiller, Russian election interference, Second Machine Age, seminal paper, sentiment analysis, shareholder value, Sheryl Sandberg, skunkworks, Snapchat, social contagion, social distancing, social graph, social intelligence, social software, social web, statistical model, stem cell, Stephen Hawking, Steve Bannon, Steve Jobs, Steve Jurvetson, surveillance capitalism, Susan Wojcicki, Telecommunications Act of 1996, The Chicago School, the strength of weak ties, The Wisdom of Crowds, theory of mind, TikTok, Tim Cook: Apple, Uber and Lyft, uber lyft, WikiLeaks, work culture , Yogi Berra

Whether you simply want to understand the how and why, or need to make business or policy decisions, this is a must-read!” —FOSTER PROVOST, NYU Stern School of Business, author of Data Science for Business “Sinan Aral is a scientist and entrepreneur, and his unique perspective makes him the perfect guide to the world we live in today. From ads to fake news, The Hype Machine is the best critical foundation for understanding the connected world and how we might navigate through it to a better future.” —HILARY MASON, founder and CEO of Fast Forward Labs, data scientist in residence at Accel, and former chief scientist at Bitly “If you want the truth about falsehoods, real information about misinformation, and rigorous analysis of hype, this is the book for you.

…

Moreover, while 92 percent of consumers read reviews, only 6 percent write reviews, which means a vocal minority is influencing the opinions of the overwhelming majority. The potential consequences of ratings herding are significant because the 6 percent have an outsize impact on how the rest of us shop. Sean Taylor, my PhD student at the time, who’s now a senior data scientist at Lyft and the former head of the statistics team in Facebook’s Core Data Science group, overheard my conversation with Lev and wandered across the hall. “Hey, what are you guys talking about?” This is how social science starts—it’s sparked by everyday puzzles that evolve into investigations of how and why things happen. Lev, Sean, and I took this nagging question about herd behavior and embarked on a research project to uncover the truth about population-scale opinion dynamics in the crowd.

…

He’s the David Austin Professor of Management, Marketing, IT, and Data Science at MIT, director of the MIT Initiative on the Digital Economy, and head of MIT’s Social Analytics Lab. He was the chief scientist of Social Amp and Humin before co-founding Manifest Capital, a VC fund that grows startups into the Hype Machine. Aral has worked closely with Facebook, Yahoo!, Twitter, LinkedIn, Snapchat, WeChat, and The New York Times, among others, and currently serves on the advisory boards of the Alan Turing Institute, the British national institute for data science, in London; the Centre for Responsible Media Technology and Innovation in Norway; and C6 Bank, one of the first all-digital banks of Brazil.

The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World by Pedro Domingos

Albert Einstein, Amazon Mechanical Turk, Arthur Eddington, backpropagation, basic income, Bayesian statistics, Benoit Mandelbrot, bioinformatics, Black Swan, Brownian motion, cellular automata, Charles Babbage, Claude Shannon: information theory, combinatorial explosion, computer vision, constrained optimization, correlation does not imply causation, creative destruction, crowdsourcing, Danny Hillis, data is not the new oil, data is the new oil, data science, deep learning, DeepMind, double helix, Douglas Hofstadter, driverless car, Erik Brynjolfsson, experimental subject, Filter Bubble, future of work, Geoffrey Hinton, global village, Google Glasses, Gödel, Escher, Bach, Hans Moravec, incognito mode, information retrieval, Jeff Hawkins, job automation, John Markoff, John Snow's cholera map, John von Neumann, Joseph Schumpeter, Kevin Kelly, large language model, lone genius, machine translation, mandelbrot fractal, Mark Zuckerberg, Moneyball by Michael Lewis explains big data, Narrative Science, Nate Silver, natural language processing, Netflix Prize, Network effects, Nick Bostrom, NP-complete, off grid, P = NP, PageRank, pattern recognition, phenotype, planetary scale, power law, pre–internet, random walk, Ray Kurzweil, recommendation engine, Richard Feynman, scientific worldview, Second Machine Age, self-driving car, Silicon Valley, social intelligence, speech recognition, Stanford marshmallow experiment, statistical model, Stephen Hawking, Steven Levy, Steven Pinker, superintelligent machines, the long tail, the scientific method, The Signal and the Noise by Nate Silver, theory of mind, Thomas Bayes, transaction costs, Turing machine, Turing test, Vernor Vinge, Watson beat the top human players on Jeopardy!, white flight, yottabyte, zero-sum game

., 91, 94–95 Connectomics, 118–119 Consciousness, 96 Consilience (Wilson), 31 Constrained optimization, 193–195, 241, 242 Constraints, support vector machines and, 193–195 Convolutional neural networks, 117–119, 303 Cope, David, 199, 307 Cornell University, Creative Machines Lab, 121–122 Cortex, 118, 138 unity of, 26–28, 299–300 Counterexamples, 67 Cover, Tom, 185 Crawlers, 8–9 Creative Machines Lab, 121–122 Credit-assignment problem, 102, 104, 107, 127 Crick, Francis, 122, 236 Crossover, 124–125, 134–136, 241, 243 Curse of dimensionality, 186–190, 196, 201, 307 Cyber Command, 19 Cyberwar, 19–21, 279–282, 299, 310 Cyc project, 35, 300 DARPA, 21, 37, 113, 121, 255 Darwin, Charles, 28, 30, 131, 235 algorithm, 122–128 analogy and, 178 Hume and, 58 on lack of mathematical ability, 127 on selective breeding, 123–124 variation and, 124 Data accuracy of held-out, 75–76 Bayes’ theorem and, 31–32 control of, 45 first principal component of the, 214 human intuition and, 39 learning from finite, 24–25 Master Algorithm and, 25–26 patterns in, 70–75 sciences and complex, 14 as strategic asset for business, 13 theory and, 46 See also Big data; Overfitting; Personal data Database engine, 49–50 Databases, 8, 9 Data mining, 8, 73, 232–233, 298, 306. See also Machine learning Data science, 8. See also Machine learning Data scientist, 9 Data sharing, 270–276 Data unions, 274–275 Dawkins, Richard, 284 Decision making, artificial intelligence and, 282–286 Decision theory, 165 Decision tree induction, 85–89 Decision tree learners, 24, 301 Decision trees, 24, 85–90, 181–182, 188, 237–238 Deduction induction as inverse of, 80–83, 301 Turing machine and, 34 Deductive reasoning, 80–81 Deep learning, 104, 115–118, 172, 195, 241, 302 DeepMind, 222 Democracy, machine learning and, 18–19 Dempster, Arthur, 209 Dendrites, 95 Descartes, René, 58, 64 Descriptive theories, normative theories vs., 141–142, 304 Determinism, Laplace and, 145 Developmental psychology, 203–204, 308 DiCaprio, Leonardo, 177 Diderot, Denis, 63 Diffusion equation, 30 Dimensionality, curse of, 186–190, 307 Dimensionality reduction, 189–190, 211–215, 255 nonlinear, 215–217 Dirty Harry (film), 65 Disney animators, S curves and, 106 “Divide and conquer” algorithm, 77–78, 80, 81, 87 DNA sequencers, 84 Downweighting attributes, 189 Driverless cars, 8, 113, 166, 172, 306 Drones, 21, 281 Drugs, 15, 41–42, 83.

…

If you’re curious what all the hubbub surrounding big data and machine learning is about and suspect that there’s something deeper going on than what you see in the papers, you’re right! This book is your guide to the revolution. If your main interest is in the business uses of machine learning, this book can help you in at least six ways: to become a savvier consumer of analytics; to make the most of your data scientists; to avoid the pitfalls that kill so many data-mining projects; to discover what you can automate without the expense of hand-coded software; to reduce the rigidity of your information systems; and to anticipate some of the new technology that’s coming your way. I’ve seen too much time and money wasted trying to solve a problem with the wrong learning algorithm, or misinterpreting what the algorithm said.

…

At the end of the day, a browser is just a standard piece of software, but a search engine requires a different mind-set. The other reason machine learners are the über-geeks is that the world has far fewer of them than it needs, even by the already dire standards of computer science. According to tech guru Tim O’Reilly, “data scientist” is the hottest job title in Silicon Valley. The McKinsey Global Institute estimates that by 2018 the United States alone will need 140,000 to 190,000 more machine-learning experts than will be available, and 1.5 million more data-savvy managers. Machine learning’s applications have exploded too suddenly for education to keep up, and it has a reputation for being a difficult subject.

The Future of the Professions: How Technology Will Transform the Work of Human Experts by Richard Susskind, Daniel Susskind

23andMe, 3D printing, Abraham Maslow, additive manufacturing, AI winter, Albert Einstein, Amazon Mechanical Turk, Amazon Robotics, Amazon Web Services, Andrew Keen, Atul Gawande, Automated Insights, autonomous vehicles, Big bang: deregulation of the City of London, big data - Walmart - Pop Tarts, Bill Joy: nanobots, Blue Ocean Strategy, business process, business process outsourcing, Cass Sunstein, Checklist Manifesto, Clapham omnibus, Clayton Christensen, clean water, cloud computing, commoditize, computer age, Computer Numeric Control, computer vision, Computing Machinery and Intelligence, conceptual framework, corporate governance, creative destruction, crowdsourcing, Daniel Kahneman / Amos Tversky, data science, death of newspapers, disintermediation, Douglas Hofstadter, driverless car, en.wikipedia.org, Erik Brynjolfsson, Evgeny Morozov, Filter Bubble, full employment, future of work, Garrett Hardin, Google Glasses, Google X / Alphabet X, Hacker Ethic, industrial robot, informal economy, information retrieval, interchangeable parts, Internet of things, Isaac Newton, James Hargreaves, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, Joseph Schumpeter, Khan Academy, knowledge economy, Large Hadron Collider, lifelogging, lump of labour, machine translation, Marshall McLuhan, Metcalfe’s law, Narrative Science, natural language processing, Network effects, Nick Bostrom, optical character recognition, Paul Samuelson, personalized medicine, planned obsolescence, pre–internet, Ray Kurzweil, Richard Feynman, Second Machine Age, self-driving car, semantic web, Shoshana Zuboff, Skype, social web, speech recognition, spinning jenny, strong AI, supply-chain management, Susan Wojcicki, tacit knowledge, TED Talk, telepresence, The Future of Employment, the market place, The Wealth of Nations by Adam Smith, The Wisdom of Crowds, Tragedy of the Commons, transaction costs, Turing test, Two Sigma, warehouse robotics, Watson beat the top human players on Jeopardy!, WikiLeaks, world market for maybe five computers, Yochai Benkler, young professional

When this term was first used, it was confined to techniques for the handling of vast bodies of data—for example, the masses of data recorded by the Large Hadron Collider. Now, Big Data is also used to refer to the use of technology to analyse much smaller bodies of information. Some speak instead of ‘data analytics’, ‘data science’, and ‘predictive analytics’, all of which seem to mean roughly the same thing.36 Specialists in the area, whatever label is preferred, are often called ‘data scientists’. There has been no shortage of hype about Big Data. There are commentators who argue, with some justification, that its claims are too extravagant and that its methodology is underdeveloped.37 What is hard to deny is the volume of data that are swilling around.

…

There is less need for the ‘sage on the stage’ and more of a job for the ‘guide on the side’—those who help students navigate through alternative sources of expertise. There are new roles and new disciplines, like education software designers who build the ‘adaptive’ learning systems, the content curators who compile and manage online content, and the data scientists who collect large data sets and develop ‘learning analytics’ to interpret them. It is not surprising, therefore, that Larry Summers, former chair of the White House Council of Economic Advisers and past President of Harvard, has said that ‘the next quarter century will see more change in higher education than the last three combined’,93 and that Sir Michael Barber, a former Downing Street adviser, anticipates transformations in education in his aptly named report, An Avalanche is Coming.94 2.3.

…

But the competition is also advancing from outside the traditional boundaries of the professions—from new people and different institutions. In Chapter 2 we see a recurring need to draw on people with very different skills, talents, and ways of working. Practising doctors, priests, teachers, and auditors did not, for example, develop the software that supports the systems that we describe. Stepping forward instead are data scientists, process analysts, knowledge engineers, systems engineers, and many more (see Chapter 6). Today, professionals still provide much of the content, but in time they may find themselves down-staged by these new specialists. We also see a diverse set of institutions entering the fray—business process outsourcers, retail brands, Internet companies, major software and service vendors, to name a few.

Super Founders: What Data Reveals About Billion-Dollar Startups by Ali Tamaseb

"World Economic Forum" Davos, 23andMe, additive manufacturing, Affordable Care Act / Obamacare, Airbnb, Anne Wojcicki, asset light, barriers to entry, Ben Horowitz, Benchmark Capital, bitcoin, business intelligence, buy and hold, Chris Wanstrath, clean water, cloud computing, coronavirus, corporate governance, correlation does not imply causation, COVID-19, cryptocurrency, data science, discounted cash flows, diversified portfolio, Elon Musk, Fairchild Semiconductor, game design, General Magic , gig economy, high net worth, hiring and firing, index fund, Internet Archive, Jeff Bezos, John Zimmer (Lyft cofounder), Kickstarter, late fees, lockdown, Lyft, Marc Andreessen, Marc Benioff, Mark Zuckerberg, Max Levchin, Mitch Kapor, natural language processing, Network effects, nuclear winter, PageRank, PalmPilot, Parker Conrad, Paul Buchheit, Paul Graham, peer-to-peer lending, Peter Thiel, Planet Labs, power law, QR code, Recombinant DNA, remote working, ride hailing / ride sharing, robotic process automation, rolodex, Ruby on Rails, Salesforce, Sam Altman, Sand Hill Road, self-driving car, shareholder value, sharing economy, side hustle, side project, Silicon Valley, Silicon Valley startup, Skype, Snapchat, SoftBank, software as a service, software is eating the world, sovereign wealth fund, Startup school, Steve Jobs, Steve Wozniak, survivorship bias, TaskRabbit, telepresence, the payments system, TikTok, Tony Fadell, Tony Hsieh, Travis Kalanick, Uber and Lyft, Uber for X, uber lyft, ubercab, web application, WeWork, work culture , Y Combinator

As a result, Lake had to learn how to be incredibly efficient with the money she had. Stitch Fix sold clothes before paying its vendors, and it turned over inventory as quickly as possible. The company changed its cash cycle so that it didn’t have to sit on unsold products. By hiring very talented and senior data scientists, she also turned Stitch Fix into a technology powerhouse and a talent magnet for the best data scientists. Stitch Fix was able to better understand a customer’s style through a series of sophisticated algorithms, which could augment the workforce of human stylists and significantly lower costs. “If I had been handed $100 million, I don’t know that I would have understood the business as well as I did,” says Lake.

…

A business like Stitch Fix—which involves buying and holding inventory, storage, shipping, and a lot of physical labor, from stylists to warehouse workers—would be considered capital intensive and inefficient at first sight. Lake managed to become more efficient by renegotiating contracts with vendors and hiring data scientists to augment stylists to scale the business. It’s true that many capital-efficient companies are capital light. (They have low capital expenditures, abbreviated as “low CapEx.”) But in practice, companies with high capital needs can also be successful in both raising funding and reaching multibillion-dollar outcomes.

…

David Vélez was a partner at Sequoia Capital looking into Latin American investment opportunities before venturing out to start Nubank, the Brazilian online bank unicorn, and Andy Rachleff co-founded Benchmark Capital before starting Wealthfront, a consumer financial advisory company. Abe Othman, head of data science at AngelList—a website enabling angel investors to invest in early-stage startups and to diversify through index investments—told me, “One of the most significant characteristics we’ve observed among successful startups is when a founder had experience as an investor or angel investor.” This could be because former investors have an easier time raising money, and because they’re better at filtering their ideas and betting on the right one to spend time on.

Deep Medicine: How Artificial Intelligence Can Make Healthcare Human Again by Eric Topol

"World Economic Forum" Davos, 23andMe, Affordable Care Act / Obamacare, AI winter, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, algorithmic bias, AlphaGo, Apollo 11, artificial general intelligence, augmented reality, autism spectrum disorder, autonomous vehicles, backpropagation, Big Tech, bioinformatics, blockchain, Cambridge Analytica, cloud computing, cognitive bias, Colonization of Mars, computer age, computer vision, Computing Machinery and Intelligence, conceptual framework, creative destruction, CRISPR, crowdsourcing, Daniel Kahneman / Amos Tversky, dark matter, data science, David Brooks, deep learning, DeepMind, Demis Hassabis, digital twin, driverless car, Elon Musk, en.wikipedia.org, epigenetics, Erik Brynjolfsson, fake news, fault tolerance, gamification, general purpose technology, Geoffrey Hinton, George Santayana, Google Glasses, ImageNet competition, Jeff Bezos, job automation, job satisfaction, Joi Ito, machine translation, Mark Zuckerberg, medical residency, meta-analysis, microbiome, move 37, natural language processing, new economy, Nicholas Carr, Nick Bostrom, nudge unit, OpenAI, opioid epidemic / opioid crisis, pattern recognition, performance metric, personalized medicine, phenotype, placebo effect, post-truth, randomized controlled trial, recommendation engine, Rubik’s Cube, Sam Altman, self-driving car, Silicon Valley, Skinner box, speech recognition, Stephen Hawking, techlash, TED Talk, text mining, the scientific method, Tim Cook: Apple, traumatic brain injury, trolley problem, War on Poverty, Watson beat the top human players on Jeopardy!, working-age population

Among today’s medical specialties, it will be the radiologist who, having a deep understanding of the nuances of such image-based diagnostic algorithms, is best positioned to communicate results to patients and provide guidance for how to respond to them. Nevertheless, although some have asserted that “radiologists of the future will be essential data scientists of medicine,” I don’t think that’s necessarily the direction we’re headed.44 Instead, they likely will be connecting far more with patients, acting as real doctors. FIGURE 6.2: Predicting longevity from a deep neural network of CT scans. Source: Adapted from L. Oakden-Rayner et al., “Precision Radiology: Predicting Longevity Using Feature Engineering and Deep Learning Methods in a Radiomics Framework,” Sci Rep (2017): 7(1), 1648.

…

With a heightened awareness of the opportunity to prevent such tragedies, in 2017, CEO Mark Zuckerberg announced new algorithms that look for patterns of posts and words for rapid review by dedicated Facebook employees: “In the future, AI will be able to understand more of the subtle nuances of language and will be able to identify different issues beyond suicide as well, including quickly spotting more kinds of bullying and hate.” Unfortunately, and critically, Facebook has refused to disclose the algorithmic details, but the company claims to have interceded with more than a hundred people intending to commit self-harm.42 Data scientists are now using machine learning on the Crisis Text Line’s 75 million texts43 to try to unravel text or emoji risk factors.44 Overall, even these very early attempts at using AI to detect depression and suicidal risk show some promising signs that we can do far better than the traditional subjective and clinical risk factors.

…

., “Assessing Suicide Risk and Emotional Distress in Chinese Social Media: A Text Mining and Machine Learning Study.” J Med Internet Res, 2017. 19(7): p. e243. 42. McConnon, “AI Helps Identify Those at Risk for Suicide.” 43. “Crisis Trends,” July 19, 2018. https://crisistrends.org/#visualizations. 44. Resnick, B., “How Data Scientists Are Using AI for Suicide Prevention,” Vox. 2018. 45. Anthes, E., “Depression: A Change of Mind.” Nature, 2014. 515(7526): pp. 185–187. 46. Firth, J., et al., “The Efficacy of Smartphone-Based Mental Health Interventions for Depressive Symptoms: A Meta-Analysis of Randomized Controlled Trials.”

Human + Machine: Reimagining Work in the Age of AI by Paul R. Daugherty, H. James Wilson

3D printing, AI winter, algorithmic management, algorithmic trading, AlphaGo, Amazon Mechanical Turk, Amazon Robotics, augmented reality, autonomous vehicles, blockchain, business process, call centre, carbon footprint, circular economy, cloud computing, computer vision, correlation does not imply causation, crowdsourcing, data science, deep learning, DeepMind, digital twin, disintermediation, Douglas Hofstadter, driverless car, en.wikipedia.org, Erik Brynjolfsson, fail fast, friendly AI, fulfillment center, future of work, Geoffrey Hinton, Hans Moravec, industrial robot, Internet of things, inventory management, iterative process, Jeff Bezos, job automation, job satisfaction, knowledge worker, Lyft, machine translation, Marc Benioff, natural language processing, Neal Stephenson, personalized medicine, precision agriculture, Ray Kurzweil, recommendation engine, RFID, ride hailing / ride sharing, risk tolerance, robotic process automation, Rodney Brooks, Salesforce, Second Machine Age, self-driving car, sensor fusion, sentiment analysis, Shoshana Zuboff, Silicon Valley, Snow Crash, software as a service, speech recognition, tacit knowledge, telepresence, telepresence robot, text mining, the scientific method, uber lyft, warehouse automation, warehouse robotics

This way, customer satisfaction doesn’t take a hit, and the company saves on energy costs.22 Enable Discovery What kind of conversations are you having with your data? Are only analysts and data scientists benefiting from analysis tools? Your goal should be to extract insights in such a way that anyone, especially less-technical business users can take advantage of the story that the data is trying to tell. Ayasdi is democratizing discovery, providing software that’s useful to data scientists and non-technical business leaders alike. One of its customers, Texas Medical Center (TMC), focuses on the analysis of high-volume, high-dimensional data sets such as data from breast-cancer patients.

…

When you start typing search terms, Google not only considers the most generally popular associations for its autocomplete feature, but also considers your geographic location, previous search terms, and other factors. It can feel as if the software is reading your thoughts. Leveraging AI to Find a Job Bot-based empowerment skills also come in handy for job searches. If there’s one guarantee for workers in these AI days, it’s that the job landscape is quickly changing. Positions such as data scientist, which barely existed five years ago, are now all the rage. And positions that focus on rote tasks like data entry are quickly fading from job listings. How can people forge new career paths, find new training opportunities, or boost their online presence or personal brand on social media? The answer is bot-based empowerment.

…

Cade Metz, “DARPA Goes Full Tron with Its Brand Battle of the Hack Bots,” Wired, July 5, 2016, https://www.wired.com/2016/07/_trashed-19/. Various security companies have their own approaches to the problem. SparkCognition, for instance, offers a product called Deep Armor, which uses a combination of AI techniques including neural networks, heuristics, data science, and natural-language processing to detect threats never seen before and remove malicious files. Another company called Darktrace offers a product called Antigena, which is modeled on the human immune system, identifying and neutralizing bugs as they’re encountered.7 Behavioral analysis of network traffic is key to another company called Vectra.

Big Data: A Revolution That Will Transform How We Live, Work, and Think by Viktor Mayer-Schonberger, Kenneth Cukier

23andMe, Affordable Care Act / Obamacare, airport security, Apollo 11, barriers to entry, Berlin Wall, big data - Walmart - Pop Tarts, Black Swan, book scanning, book value, business intelligence, business process, call centre, cloud computing, computer age, correlation does not imply causation, dark matter, data science, double entry bookkeeping, Eratosthenes, Erik Brynjolfsson, game design, hype cycle, IBM and the Holocaust, index card, informal economy, intangible asset, Internet of things, invention of the printing press, Jeff Bezos, Joi Ito, lifelogging, Louis Pasteur, machine readable, machine translation, Marc Benioff, Mark Zuckerberg, Max Levchin, Menlo Park, Moneyball by Michael Lewis explains big data, Nate Silver, natural language processing, Netflix Prize, Network effects, obamacare, optical character recognition, PageRank, paypal mafia, performance metric, Peter Thiel, Plato's cave, post-materialism, random walk, recommendation engine, Salesforce, self-driving car, sentiment analysis, Silicon Valley, Silicon Valley startup, smart grid, smart meter, social graph, sparse data, speech recognition, Steve Jobs, Steven Levy, systematic bias, the scientific method, The Signal and the Noise by Nate Silver, The Wealth of Nations by Adam Smith, Thomas Davenport, Turing test, vertical integration, Watson beat the top human players on Jeopardy!

So far, the first two of these elements get the most attention: the skills, which today are scarce, and the data, which seems abundant. A new profession has emerged in recent years, the “data scientist,” which combines the skills of the statistician, software programmer, infographics designer, and storyteller. Instead of squinting into a microscope to unlock a mystery of the universe, the data scientist peers into databases to make a discovery. The McKinsey Global Institute proffers dire predictions about the dearth of data scientists now and in the future (which today’s data scientists like to cite to feel special and to pump up their salaries). Hal Varian, Google’s chief economist, famously calls statistician the “sexiest” job around.

…

The value in data’s reuse is good news for organizations that collect or control large datasets but currently make little use of them, such as conventional businesses that mostly operate offline. They may sit on untapped informational geysers. Some companies may have collected data, used it once (if at all), and just kept it around because of low storage cost—in “data tombs,” as data scientists call the places where such old info resides. Internet and technology companies are on the front lines of harnessing the data deluge, since they collect so much information just by being online and are ahead of the rest of industry in analyzing it. But all firms stand to gain. The consultants at McKinsey & Company point to a logistics company, whose name they keep anonymous, which noticed that in the process of delivering goods, it was amassing reams of information on product shipments around the globe.

…

Data exhaust is the mechanism behind many services like voice recognition, spam filters, language translation, and much more. When users indicate to a voice-recognition program that it has misunderstood what they said, they in effect “train” the system to get better. Many businesses are starting to engineer their systems to collect and use information in this way. In Facebook’s early days, its first “data scientist,” Jeff Hammerbacher (and among the people credited with coining the term), examined its rich trove of data exhaust. He and the team found that a big predictor that people would take an action (post content, click an icon, and so on) was whether they had seen their friends do the same thing. So Facebook redesigned its system to put greater emphasis on making friends’ activities more visible, which sparked a virtuous circle of new contributions to the site.

The Happiness Industry: How the Government and Big Business Sold Us Well-Being by William Davies

"Friedman doctrine" OR "shareholder theory", "World Economic Forum" Davos, 1960s counterculture, Abraham Maslow, Airbnb, behavioural economics, business intelligence, business logic, corporate governance, data science, dematerialisation, experimental subject, Exxon Valdez, Frederick Winslow Taylor, Gini coefficient, income inequality, intangible asset, invisible hand, joint-stock company, Leo Hollis, lifelogging, market bubble, mental accounting, military-industrial complex, nudge unit, Panopticon Jeremy Bentham, Philip Mirowski, power law, profit maximization, randomized controlled trial, Richard Thaler, road to serfdom, Ronald Coase, Ronald Reagan, science of happiness, scientific management, selective serotonin reuptake inhibitor (SSRI), sentiment analysis, sharing economy, Slavoj Žižek, smart cities, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, social contagion, social intelligence, Social Responsibility of Business Is to Increase Its Profits, Steve Jobs, TED Talk, The Chicago School, The Spirit Level, theory of mind, urban planning, Vilfredo Pareto, W. E. B. Du Bois, you are the product

But possibilities for psychological and behavioural data are heavily shaped by the power structures which facilitate them. The current explosion in happiness and well-being data is really an effect of new technologies and practices of surveillance. In turn, these depend on pre-existing power inequalities. Building the new laboratory In 2012, Harvard Business Review declared that ‘data scientist’ would be the ‘sexiest job of the twenty-first century’.4 We live during a time of tremendous optimism regarding the possibilities for data collection and analysis that is refuelling the behaviourist and utilitarian ambition to manage society purely through careful scientific observation of mind, body and brain.

…

But as that science becomes ever more advanced, eventually the subjective element of it starts to drop out of the picture altogether. Bentham’s presumption, that pleasure and pain are the only real dimensions of psychology, is now leading squarely towards the philosophical riddle whereby a neuroscientist or data scientist can tell me that I am objectively wrong about my own mood. We are reaching the point where our bodies are more trusted communicators than our words. If one way of ‘seeing’ happiness as a physiological event is via the face, the other way is to get even closer to its supposed locus: the brain.

…

, theatlantic.com, 2 April 2012. 25Jeremy Gilbert, ‘Capitalism, Creativity and the Crisis in the Music Industry’, opendemocracy.net, 14 September 2012. 7 Living in the Lab 1Jennifer Scanlon, ‘Mediators in the International Marketplace: US Advertising in Latin America in the Early Twentieth Century’, The Business History Review 77: 3, 2003. 2Jeff Merron, ‘Putting Foreign Consumers on the Map: J. Walter Thompson’s Struggle with General Motors’ International Advertising Account in the 1920s’, The Business History Review 73: 3, 1999. 3Ibid. 4Thomas Davenport and D. J. Patil, ‘Data Scientist: The Sexiest Job of the 21st Century’, Harvard Business Review, October 2012. 5Viktor Mayer-Schönberger, and Kenneth Cukier, Big Data: A Revolution That Will Transform How We Live, Work and Think, London: John Murray, 2013. 6Anthony Townsend, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia, New York: W.

The Loop: How Technology Is Creating a World Without Choices and How to Fight Back by Jacob Ward

2021 United States Capitol attack, 4chan, Abraham Wald, AI winter, Albert Einstein, Albert Michelson, Amazon Mechanical Turk, assortative mating, autonomous vehicles, availability heuristic, barriers to entry, Bayesian statistics, Benoit Mandelbrot, Big Tech, bitcoin, Black Lives Matter, Black Swan, blockchain, Broken windows theory, call centre, Cass Sunstein, cloud computing, contact tracing, coronavirus, COVID-19, crowdsourcing, cuban missile crisis, Daniel Kahneman / Amos Tversky, dark matter, data science, deep learning, Donald Trump, drone strike, endowment effect, George Akerlof, George Floyd, hindsight bias, invisible hand, Isaac Newton, Jeffrey Epstein, license plate recognition, lockdown, longitudinal study, Lyft, mandelbrot fractal, Mark Zuckerberg, meta-analysis, natural language processing, non-fungible token, nudge unit, OpenAI, opioid epidemic / opioid crisis, pattern recognition, QAnon, RAND corporation, Richard Thaler, Robert Shiller, selection bias, self-driving car, seminal paper, shareholder value, smart cities, social contagion, social distancing, Steven Levy, survivorship bias, TikTok, Turing test

With enough time to refine its process, the system will, eventually, astound us with its gift for telling us that this panting poodle is a dog, and that this heifer nosing the curtains is a cow. But how did the system get there? It matters because, as we’ll see, this technology goes about answering stupid questions and world-changing questions the same way. A data scientist handed this task would want to know more about the goal of the project in order to employ the most suitable flavor of machine learning to accomplish it. If you wanted the system to identify the cows and dogs in a photograph of the stage, for instance, you’d probably feed it into a convolutional neural network, a popular means of recognizing objects in a photograph these days.

…

We knew we probably would not win the competition that way, but there was a bigger point that we needed to make.2 And they were right: they didn’t win. The organizers of the competition would not allow the judges to play with and evaluate the Duke team’s visualization tool, and a trio from IBM Research won the competition for a system based on Boolean rules that would help a human data scientist inspect the black box. In its paper describing the winning system, the IBM team rightly pointed out the need for explainability “as machine learning pushes further into domains such as medicine, criminal justice, and business, where such models complement human decision-makers and decisions can have major consequences on human lives.”

…

Wald is credited with helping to identify something that statisticians now work hard to fight off: survivorship bias, the tendency to optimize our behavior for the future based on what may in fact be the rare instances in which someone or something survives extremely long odds and only as a result comes to our attention. Forecasting probability based on limited data is something statisticians now know not to do, thanks to Wald—data scientists often post his report’s famous diagram of a shot-up plane as an online meme—but Anna Todd is the spear-tip of an industry that forecasts future success by studying survivors. When Todd and I met, roughly four million writers were posting more than twenty-four hours of reading material to WattPad every sixty seconds of every day, according to the company.

Mastering Pandas by Femi Anthony

Amazon Web Services, Bayesian statistics, correlation coefficient, correlation does not imply causation, data science, Debian, en.wikipedia.org, Internet of things, Large Hadron Collider, natural language processing, p-value, power law, random walk, side project, sparse data, statistical model, Thomas Bayes

He enjoys analyzing data and solving complex business problems using SAS, R, EViews/Gretl, Minitab, SQL, and Python. Opeyemi is also an adjunct at Northwood University where he designs and teaches undergraduate courses in microeconomics and macroeconomics. Louis Hénault is a data scientist at OgilvyOne Paris. He loves combining mathematics and computer science to solve real-world problems in an innovative way. After getting a master's degree in engineering with a major in data sciences and another degree in applied mathematics in France, he entered into the French start-up ecosystem, working on several projects. Louis has gained experience in various industries, including geophysics, application performance management, online music platforms, e-commerce, and digital advertising.

…

Note For more information, refer the Wikipedia page on Python at http://en.wikipedia.org/wiki/Python_%28programming_language%29. Among the characteristics that make Python popular for data science are its very user-friendly (human-readable) syntax, the fact that it is interpreted rather than compiled (leading to faster development time), and its very comprehensive library for parsing and analyzing data, as well as its capacity for doing numerical and statistical computations. Python has libraries that provide a complete toolkit for data science and analysis. The major ones are as follows: NumPy: The general-purpose array functionality with emphasis on numeric computation SciPy: Numerical computing Matplotlib: Graphics pandas: Series and data frames (1D and 2D array-like types) Scikit-Learn: Machine learning NLTK: Natural language processing Statstool: Statistical analysis For this book, we will be focusing on the 4th library listed in the preceding list, pandas.

…

Application of machine learning – Kaggle Titanic competition In order to illustrate how we can use pandas to assist us at the start of our machine learning journey, we will apply it to a classic problem, which is hosted on the Kaggle website (http://www.kaggle.com). Kaggle is a competition platform for machine learning problems. The idea behind Kaggle is to enable companies that are interested in solving predictive analytics problems with their data to post their data on Kaggle and invite data scientists to come up with the proposed solutions to their problems. The competition can be ongoing over a period of time, and the rankings of the competitors are posted on a leader board. At the end of the competition, the top-ranked competitors receive cash prizes. The classic problem that we will study in order to illustrate the use of pandas for machine learning with scikit-learn is the Titanic: machine learning from disaster problem hosted on Kaggle as their classic introductory machine learning problem.

Robot Rules: Regulating Artificial Intelligence by Jacob Turner

"World Economic Forum" Davos, Ada Lovelace, Affordable Care Act / Obamacare, AI winter, algorithmic bias, algorithmic trading, AlphaGo, artificial general intelligence, Asilomar, Asilomar Conference on Recombinant DNA, autonomous vehicles, backpropagation, Basel III, bitcoin, Black Monday: stock market crash in 1987, blockchain, brain emulation, Brexit referendum, Cambridge Analytica, Charles Babbage, Clapham omnibus, cognitive dissonance, Computing Machinery and Intelligence, corporate governance, corporate social responsibility, correlation does not imply causation, crowdsourcing, data science, deep learning, DeepMind, Demis Hassabis, distributed ledger, don't be evil, Donald Trump, driverless car, easy for humans, difficult for computers, effective altruism, Elon Musk, financial exclusion, financial innovation, friendly fire, future of work, hallucination problem, hive mind, Internet of things, iterative process, job automation, John Markoff, John von Neumann, Loebner Prize, machine readable, machine translation, medical malpractice, Nate Silver, natural language processing, Nick Bostrom, Northpointe / Correctional Offender Management Profiling for Alternative Sanctions, nudge unit, obamacare, off grid, OpenAI, paperclip maximiser, pattern recognition, Peace of Westphalia, Philippa Foot, race to the bottom, Ray Kurzweil, Recombinant DNA, Rodney Brooks, self-driving car, Silicon Valley, Stanislav Petrov, Stephen Hawking, Steve Wozniak, strong AI, technological singularity, Tesla Model S, The Coming Technological Singularity, The Future of Employment, The Signal and the Noise by Nate Silver, trolley problem, Turing test, Vernor Vinge

An AI system learns to play the game independently of human training, but when its actions are matched to the human descriptions, a narrative can be generated by sewing together these descriptors. Taking a different route to Riedl et al., data scientist Daniel Whitenack identifies three general capabilities required for transparency in AI: data provenance (knowing the source of all data); reproducibility (the ability to recreate a given result); and data versioning (saving snapshot copies of the AI in particular states with a view to recording which input led to which output). Whitenack suggests that in order to make these three desiderata “standards within data science, we need proper tools to integrate these characteristics into workflows”. He says that ideally, AI transparency tools will be:Language agnostic—The language wars in data science between python, R, scala, and others will continue on forever.

…

See also James Vincent, “Tencent Says There Are Only 300,000 AI Engineers Worldwide, but Millions Are Needed”, The Verge, 5 December 2017, https://​www.​theverge.​com/​2017/​12/​5/​16737224/​global-ai-talent-shortfall-tencent-report, accessed 1 June 2018. By contrast, PWC estimate that in the USA alone, there will be 2.9 m people with data science and analytics skills by 2018. Not all will be AI professionals per se, but many of their skills will overlap. “What’s Next for the 2017 Data Science and Analytics Job Market?”, PWC Website, https://​www.​pwc.​com/​us/​en/​library/​data-science-and-analytics.​html, accessed 1 June 2018. 144Katja Grace, “The Asilomar Conference: A Case Study in Risk Mitigation”, MIRI Research Institute, Technical Report, 2015–9 (Berkeley, CA: MIRI, 15 July 2015), 15. 145A constantly-updated database of tech ethics curricula is available at: https://​docs.​google.​com/​spreadsheets/​d/​1jWIrA8jHz5fYAW4​h9CkUD8gKS5V98PD​JDymRf8d9vKI/​edit#gid=​0, accessed 1 June 2018. 146 Microsoft, The Future Computed: Artificial Intelligence and Its Role in Society (Redmond, WA: Microsoft Corporation, U.S.A., 2018), 55, https://​msblob.​blob.​core.​windows.​net/​ncmedia/​2018/​01/​The-Future_​Computed_​1.​26.​18.​pdf, accessed 1 June 2018. 147See, for example, s. 1 of the UK Road Traffic Act 1988, or s. 249(1)(a) of the Canadian Criminal Code. 148“About TensorFlow”, Website of TensorfFlow, https://​www.​tensorflow.​org/​, accessed 1 June 2018. 149See, for example, the UK Government’s “Guidance: Wine Duty”, 9 November 2009, https://​www.​gov.​uk/​guidance/​wine-duty, accessed 1 June 2018. 150See, for example, Max Weber, “Politics as a Vocation”, in From Max Weber: Essays in Sociology, translated by H.H.

…

He says that ideally, AI transparency tools will be:Language agnostic—The language wars in data science between python, R, scala, and others will continue on forever. We will always need a mix of languages and frameworks to enable advancements in a field as broad as data science. However, if tools enabling data versioning/provenance are language specific, they are unlikely to be integrated as standard practice. Infrastructure Agnostic—The tools should be able to be deployed on your existing infrastructure—locally, in the cloud, or on-prem. Scalable/distributed—It would be impractical to implement changes to a workflow if they were not able to scale up to production requirements.

Mining Social Media: Finding Stories in Internet Data by Lam Thuy Vo

barriers to entry, correlation does not imply causation, data science, Donald Trump, en.wikipedia.org, Filter Bubble, Firefox, Google Chrome, Internet Archive, natural language processing, social web, web application

To that end, what follows are a few helpful resources on writing clean code with Python and pandas, as well as producing reproducible data analysis in Jupyter Notebook. While by no means a comprehensive guide, they’re a good starting point: The general Python style guide (https://docs.python-guide.org/writing/style/) and a style guide for data scientists (http://columbia-applied-data-science.github.io/pages/lowclass-python-style-guide.html) Think Python, 2nd Edition, a book by Allen B. Downey (O’Reilly, 2015), available for free under the Creative Commons license on the author’s site (https://greenteapress.com/wp/think-python-2e/) “A Beginner’s Guide to Optimizing Pandas Code for Speed,” an article by Sofia Heisler (https://engineering.upside.com/a-beginners-guide-to-optimizing-pandas-code-for-speed-c09ef2c6a4d6/) “What We’ve Learned About Sharing Our Data Analysis,” an article by Jeremy Singer-Vine (https://source.opennews.org/articles/what-weve-learned-about-sharing-our-data-analysis/) The libraries and tools we’ve used in this book have stood the test of time among Python users, but new libraries pop up all the time and may do certain things better than what is already available.

…

The Jupyter Notebook web app, which evolved out of the web app IPython Notebooks, was created to accommodate three programming languages—Julia, Python, and R (Ju-Pyt-R)—but has since evolved to support many other coding languages. Jupyter Notebook is also used by many data scientists in a diverse range of fields, including people crunching numbers to improve website performance, sociologists studying demographic information, and journalists searching for trends and anomalies in data obtained through Freedom of Information Act requests. One huge benefit of this is that many of these data scientists and researchers put their notebooks—often featuring detailed and annotated analyses—online on code-sharing platforms like GitHub, making it easier for beginning learners like you to replicate their studies.

…

But the findings in summary data come from rather messy, wildly varying replies to surveys or other databases of raw data—that is, data that has not been processed yet. Data tables provided by organizations like the US Census Bureau often have been cleaned, processed, and aggregated from thousands—if not millions—of raw data entries, many of which may contain several inconsistencies that data scientists worked to resolve. For example, in a simple table listing people’s occupations, these organizations may have resolved different but essentially equivalent responses like “attorney” and “lawyer.” Likewise, the raw data we look at in this book—data from the social web—can be quite irregular and challenging to process because it’s pro-duced by real people, each with unique quirks and posting habits.

Pandas for Everyone: Python Data Analysis by Unknown

data science

Numpy 3 http://pandas.pydata.org/pandas-docs/stable/basics.html#descriptivestatistics print(ages.mean()) 49.0 print(ages.min()) 37 print(ages.max()) 61 print(ages.std()) 16.97056274847714 The mean, min, max, and std are also methods in the numpy.ndarray Series append methods Description Concatenates 2 or more Series corr Calculate a correlation with another Series* cov Calculate a covariance with another Series* describe Calculate summary statistics* drop duplicates Returns a Series without duplicates equals Sees if a Series has the same elements get values Get values of the Series, same as the values attribute hist Draw a histogram min Return the minimum value max Returns the maximum value mean Returns the arithmetic mean median Returns the median mode Returns the mode(s) quantile Returns the value at a given quantile replace Replaces values in the Series with a specified value sample Returns a random sample of values from the Series sort values Sort values to frame Converts Series to DataFrame transpose Return the transpose unique Returns a numpy.ndarray of unique values indicates missing values will be automatically dropped 2.5.2 Boolean subsetting Series Chapter 1 showed how we can use specific indicies to subset our data. However, it is rare that we know the exact row or column index to subset the data. Typically you are looking for values that meet (or don’t meet) a particular calculation or observation. First, let’s use a larger dataset scientists pd.read_csv(’../data/scientists.csv’) We just saw how we can calculate basic descriptive metrics of vectors 4 http://does.scipy.org/doc/numpy/reference/arrays.ndarray.html ages = scientists[’Age’] print(ages) 0 37 1 61 2 90 3 66 4 56 5 45 6 41 7 77 Name: Age, dtype: int64 print(ages.mean()) 59.125 print(ages.describe()) count 8.000000 mean 59.125000 std 18.325918 min 37.000000 25% 44.000000 50% 58.500000 75% 68.750000 max 90.000000 Name: Age, dtype: float64 What if we wanted to subset our ages by those above the mean?

…

print(taxi_loop_concat_comp equals(taxi_loop_concat)) True 6.7 Summary Here I showed you how we can reshape data to a format that is conducive for data analysis, visualization, and collection. We followed Hadley Wickham's Tidy Data paper to show the various functions and methods to reshape our data. This is an important skill since various functions will need data in a certain shape, tidy or not, in order to work. Knowing how to reshape your data will be an important still as a data scientist and analyst.

…

You can see TODO USING FUNCTIONS of you need more information on using function parameters 7 http://pandas.pydata.org/pandasdocs/stable/generated/pandas.Series.to_csv.html 8 http://pandas.pydata.org/pandasdocs/stable/generated/pandas.DataFrame.to_csv.html 9 http://pandas.pydata.org/pandasdocs/stable/generated/pandas.DataFrame.to_csv.html 10 http://pandas.pydata.org/pandasdocs/stable/generated/pandas.read_csv.html 2.8.3 Excel Excel, probably the most common data type (or second most common, next to CSVs). Excel has a bad reputation within the data science community. I discuessed some of the reasons why in Chapter 1.1. The goal of this book isn’t to bash Excel, but to teach you a resonable alternative tool for data analytics. In short, the more you can do your work in a scripting language, the easier it will be to scale up to larger projects, catch and fix mistakes, and collaborate.

Personal Finance with Python by Max Humber

asset allocation, backtesting, bitcoin, cryptocurrency, data science, Dogecoin, en.wikipedia.org, Ethereum, passive income, web application

The second edition of this hands-on guide—updated for Python 3.5 and pandas 1.0 —is packed with practical cases studies that show you how to effectively solve a broad set of data analysis problems, using Python libraries such as NumPy, pandas, Matplotlib, and IPython. Python Data Science Handbook: Tools and Techniques for Developers by Jake VanderPlas For many researchers, Python is a first-class tool mainly because of its libraries for storing, manipulating, and gaining insight from data. Several resources exist for individual pieces of this data science stack, but only with this book do you get them all—IPython, NumPy, pandas, Matplotlib, scikit-learn, and other related tools. Finally, please e-mail me if you have any concerns, questions, or comments about this book.

…

Conclusion Chapter 3:​ Convert openexchangerate​s.​org Secrets Documentation Encapsulate show_​alternative .​apply Conclusion Chapter 4:​ Amortize Banks Amortization Payment Loop A Loop B Functionize Evaluate Conclusion Chapter 5:​ Budget Dates datetime Timestamp .​normalize Horizon Flows Totals Visualization Updating Vacation I English get_​dates Fun YAML Functionize Vacation II Loading YAML Conclusion Chapter 6:​ Invest Trade-Offs Instantiate Prices Orders Deposit Simulate Quotes get_​price get_​historical Portfolio Rebalance Conclusion Chapter 7:​ Spend Prophet Purchases Forecast Visualize Conclusion Appendix: Next Index About the Author and About the Technical Reviewer About the Author Max Humberis a Data Engineer interested in improving finance with technology. He works for Wealthsimple and previously served as the first data scientist for the online lending platform Borrowell. He has spoken at Pycon, ODSC, PyData, useR, and BigDataX in Colombia, London, Berlin, Brussels, and Toronto. About the Technical Reviewer Michael Thomas has worked in software development for more than 20 years as an individual contributor, team lead, program manager, and vice president of engineering.

Data for the Public Good by Alex Howard

"World Economic Forum" Davos, 23andMe, Atul Gawande, Cass Sunstein, cloud computing, crowdsourcing, data science, Hernando de Soto, Internet of things, Kickstarter, lifelogging, machine readable, Network effects, openstreetmap, Silicon Valley, slashdot, social intelligence, social software, social web, web application

If you want a deep look at what the work of digitizing data really looks like, read Carl Malamud’s interview with Slashdot on opening government data. Data for the public good, however, goes far beyond government’s own actions. In many cases, it will happen despite government action — or, often, inaction — as civic developers, data scientists and clinicians pioneer better analysis, visualization and feedback loops. For every civic startup or regulation, there’s a backstory that often involves a broad number of stakeholders. Governments have to commit to open up themselves but will, in many cases, need external expertise or even funding to do so.

…

To create public good from public goods — the public sector data that governments collect, the private sector data that is being collected and the social data that we generate ourselves — we will need to collectively forge new compacts that honor existing laws and visionary agreements that enable the new data science to put the data to work. About the Author Alexander B. Howard is the Government 2.0 Correspondent for O’Reilly Media, where he reports on technology, open government and online civics. Before joining O’Reilly, Howard was the associate editor of SearchCompliance.com at TechTarget. His work there focused on how regulations affect IT operations, including issues of data protection, privacy, security and enterprise IT strategy.

Devil's Bargain: Steve Bannon, Donald Trump, and the Storming of the Presidency by Joshua Green

4chan, Affordable Care Act / Obamacare, Ayatollah Khomeini, Bernie Sanders, Biosphere 2, Black Lives Matter, business climate, Cambridge Analytica, Carl Icahn, centre right, Charles Lindbergh, coherent worldview, collateralized debt obligation, conceptual framework, corporate raider, crony capitalism, currency manipulation / currency intervention, data science, Donald Trump, Dr. Strangelove, fake news, Fractional reserve banking, Glass-Steagall Act, Goldman Sachs: Vampire Squid, Gordon Gekko, guest worker program, hype cycle, illegal immigration, immigration reform, Jim Simons, junk bonds, liberation theology, low skilled workers, machine translation, Michael Milken, Nate Silver, Nelson Mandela, nuclear winter, obamacare, open immigration, Peace of Westphalia, Peter Thiel, quantitative hedge fund, Renaissance Technologies, Robert Mercer, Ronald Reagan, Silicon Valley, social intelligence, speech recognition, Steve Bannon, urban planning, vertical integration

Trump’s campaign rejected the attack and criticized the ADL for involving itself in partisan politics. “Darkness is good,” Bannon counseled Trump. “Don’t let up.” By this point, the campaign had curtailed most of its polling. But it wasn’t quite flying blind. A few days earlier, Trump’s team of data scientists, squirreled away in an office down in San Antonio, had delivered a report titled “Predictions: Five Days Out,” which contained stunning news that contradicted the widespread assumption that Clinton would win easily. It was suddenly clear that Comey’s FBI investigation was roiling the electorate.

…

Bannon was fixated on Michigan, constantly urging Stepien to click back over and zoom in on bellwethers like Macomb County—and with 30 percent, 40 percent, then 50 percent of precincts reporting, Trump’s lead was holding steady. Or growing. As the night wore on, he even led in Wisconsin, a scenario none of Trump’s data scientists had ever imagined. Later on, several of those crammed into the room would recall a moment when Stepien’s manic patter flagged for just a second and the room fell quiet. Then somebody—no one could remember who—muttered, “Holy shit. This is happening.” Drudge was right. The corporate media blew it.

…

The unmarked entrance is framed by palmetto trees and sits beneath a large, second-story veranda with sweeping overhead fans, where the (mostly male) staff gathers in the afternoons to smoke cigars and brainstorm. Established in 2012 to study crony capitalism and governmental malfeasance, GAI is staffed with lawyers, data scientists, and forensic investigators and has collaborated with such mainstream news outlets as Newsweek, ABC News, and CBS’s 60 Minutes on stories ranging from insider trading in Congress to credit-card fraud among presidential campaigns. It’s a mining operation for political scoops that, for two years, had trained its investigative firepower on the Clintons.

NumPy Cookbook by Ivan Idris

business intelligence, cloud computing, computer vision, data science, Debian, en.wikipedia.org, Eratosthenes, mandelbrot fractal, p-value, power law, sorting algorithm, statistical model, transaction costs, web application

By night, he cultivates his academic interests in mathematics and computer science, and plays with mathematical and scientific software. Ryan R. Rosario is a Doctoral Candidate at the University of California, Los Angeles. He works at Riot Games as a Data Scientist, and he enjoys turning large quantities of massive, messy data into gold. He is heavily involved in the open source community, particularly with R, Python, Hadoop, and Machine Learning, and has also contributed code to various Python and R projects. He maintains a blog dedicated to Data Science and related topics at http://www.bytemining.com. He has also served as a technical reviewer for NumPy 1.5 Beginner's Guide. www.PacktPub.com Support files, eBooks, discount offers and more You might want to visit www.PacktPub.com for support files and downloads related to your book.

…

., 0], 2, 0.3, 0.2) axis('off') imshow(edges) show() The code produces an image of the edges within the original picture, as shown in the following screenshot: Installing Pandas Pandas is a Python library for data analysis. It has some similarities with the R programming language, which are not coincidental. R is a specialized programming language popular with data scientists. For instance, the core DataFrame object is inspired by R. How to do it... On PyPi, the project is called pandas. So, for instance, run either of the following two command: sudo easy_install -U pandas pip install pandas If you are using a Linux package manager, you will need to install the python-pandas project.

Architects of Intelligence by Martin Ford

3D printing, agricultural Revolution, AI winter, algorithmic bias, Alignment Problem, AlphaGo, Apple II, artificial general intelligence, Asilomar, augmented reality, autonomous vehicles, backpropagation, barriers to entry, basic income, Baxter: Rethink Robotics, Bayesian statistics, Big Tech, bitcoin, Boeing 747, Boston Dynamics, business intelligence, business process, call centre, Cambridge Analytica, cloud computing, cognitive bias, Colonization of Mars, computer vision, Computing Machinery and Intelligence, correlation does not imply causation, CRISPR, crowdsourcing, DARPA: Urban Challenge, data science, deep learning, DeepMind, Demis Hassabis, deskilling, disruptive innovation, Donald Trump, Douglas Hofstadter, driverless car, Elon Musk, Erik Brynjolfsson, Ernest Rutherford, fake news, Fellow of the Royal Society, Flash crash, future of work, general purpose technology, Geoffrey Hinton, gig economy, Google X / Alphabet X, Gödel, Escher, Bach, Hans Moravec, Hans Rosling, hype cycle, ImageNet competition, income inequality, industrial research laboratory, industrial robot, information retrieval, job automation, John von Neumann, Large Hadron Collider, Law of Accelerating Returns, life extension, Loebner Prize, machine translation, Mark Zuckerberg, Mars Rover, means of production, Mitch Kapor, Mustafa Suleyman, natural language processing, new economy, Nick Bostrom, OpenAI, opioid epidemic / opioid crisis, optical character recognition, paperclip maximiser, pattern recognition, phenotype, Productivity paradox, radical life extension, Ray Kurzweil, recommendation engine, Robert Gordon, Rodney Brooks, Sam Altman, self-driving car, seminal paper, sensor fusion, sentiment analysis, Silicon Valley, smart cities, social intelligence, sparse data, speech recognition, statistical model, stealth mode startup, stem cell, Stephen Hawking, Steve Jobs, Steve Wozniak, Steven Pinker, strong AI, superintelligent machines, synthetic biology, systems thinking, Ted Kaczynski, TED Talk, The Rise and Fall of American Growth, theory of mind, Thomas Bayes, Travis Kalanick, Turing test, universal basic income, Wall-E, Watson beat the top human players on Jeopardy!, women in the workforce, working-age population, workplace surveillance , zero-sum game, Zipcar

So, it’s not surprising that when we have real robots, they’re going to be able to do those jobs. I also think that the current mindset among governments is: “Oh, well then. I guess we really need to start training people to be data scientists, because that’s the job of the future—or robot engineers.” This clearly isn’t the solution because we don’t need a billion data scientists and robot engineers: we just need a few million. This might be a strategy for a small country like Singapore; or where I am currently, in Dubai, it might also be a viable strategy. But it’s not a viable strategy for any major country because there is simply not going to be enough jobs in those areas.

…

These are not often done within a single company. Does that integration pose new challenges? DAPHNE KOLLER: Absolutely. I think the biggest challenge is actually cultural, in getting scientists and data scientists to work together as equal partners. In many companies, one group sets the direction, and the other takes a back seat. At insitro, we really need to build a culture in which scientists, engineers, and data scientists work closely together to define problems, design experiments, analyze data, and derive insights that will lead us to new therapeutics. We believe that building this team and this culture well is as important to the success of our mission as the quality of the science or the machine learning that these different groups will create.

…

At the end of the day, we’re designing these systems, and we get to say how they are deployed, we can turn the switch off. CEO & CO-FOUNDER OF AFFECTIVA Rana el Kaliouby is the co-founder and CEO of Affectiva, a startup company that specializes in AI systems that sense and understand human emotions. Affectiva is developing cutting-edge AI technologies that apply machine learning, deep learning, and data science to bring new levels of emotional intelligence to AI. Rana is an active participant in international forums that focus on ethical issues and the regulation of AI to help ensure the technology has a positive impact on society. She was selected as a Young Global Leader by the World Economic Forum in 2017.

WTF?: What's the Future and Why It's Up to Us by Tim O'Reilly

"Friedman doctrine" OR "shareholder theory", 4chan, Affordable Care Act / Obamacare, Airbnb, AlphaGo, Alvin Roth, Amazon Mechanical Turk, Amazon Robotics, Amazon Web Services, AOL-Time Warner, artificial general intelligence, augmented reality, autonomous vehicles, barriers to entry, basic income, behavioural economics, benefit corporation, Bernie Madoff, Bernie Sanders, Bill Joy: nanobots, bitcoin, Blitzscaling, blockchain, book value, Bretton Woods, Brewster Kahle, British Empire, business process, call centre, Capital in the Twenty-First Century by Thomas Piketty, Captain Sullenberger Hudson, carbon tax, Carl Icahn, Chuck Templeton: OpenTable:, Clayton Christensen, clean water, cloud computing, cognitive dissonance, collateralized debt obligation, commoditize, computer vision, congestion pricing, corporate governance, corporate raider, creative destruction, CRISPR, crowdsourcing, Danny Hillis, data acquisition, data science, deep learning, DeepMind, Demis Hassabis, Dennis Ritchie, deskilling, DevOps, Didi Chuxing, digital capitalism, disinformation, do well by doing good, Donald Davies, Donald Trump, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, fake news, Filter Bubble, Firefox, Flash crash, Free Software Foundation, fulfillment center, full employment, future of work, George Akerlof, gig economy, glass ceiling, Glass-Steagall Act, Goodhart's law, Google Glasses, Gordon Gekko, gravity well, greed is good, Greyball, Guido van Rossum, High speed trading, hiring and firing, Home mortgage interest deduction, Hyperloop, income inequality, independent contractor, index fund, informal economy, information asymmetry, Internet Archive, Internet of things, invention of movable type, invisible hand, iterative process, Jaron Lanier, Jeff Bezos, jitney, job automation, job satisfaction, John Bogle, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John Zimmer (Lyft cofounder), Kaizen: continuous improvement, Ken Thompson, Kevin Kelly, Khan Academy, Kickstarter, Kim Stanley Robinson, knowledge worker, Kodak vs Instagram, Lao Tzu, Larry Ellison, Larry Wall, Lean Startup, Leonard Kleinrock, Lyft, machine readable, machine translation, Marc Andreessen, Mark Zuckerberg, market fundamentalism, Marshall McLuhan, McMansion, microbiome, microservices, minimum viable product, mortgage tax deduction, move fast and break things, Network effects, new economy, Nicholas Carr, Nick Bostrom, obamacare, Oculus Rift, OpenAI, OSI model, Overton Window, packet switching, PageRank, pattern recognition, Paul Buchheit, peer-to-peer, peer-to-peer model, Ponzi scheme, post-truth, race to the bottom, Ralph Nader, randomized controlled trial, RFC: Request For Comment, Richard Feynman, Richard Stallman, ride hailing / ride sharing, Robert Gordon, Robert Metcalfe, Ronald Coase, Rutger Bregman, Salesforce, Sam Altman, school choice, Second Machine Age, secular stagnation, self-driving car, SETI@home, shareholder value, Silicon Valley, Silicon Valley startup, skunkworks, Skype, smart contracts, Snapchat, Social Responsibility of Business Is to Increase Its Profits, social web, software as a service, software patent, spectrum auction, speech recognition, Stephen Hawking, Steve Ballmer, Steve Jobs, Steven Levy, Stewart Brand, stock buybacks, strong AI, synthetic biology, TaskRabbit, telepresence, the built environment, the Cathedral and the Bazaar, The future is already here, The Future of Employment, the map is not the territory, The Nature of the Firm, The Rise and Fall of American Growth, The Wealth of Nations by Adam Smith, Thomas Davenport, Tony Fadell, Tragedy of the Commons, transaction costs, transcontinental railway, transportation-network company, Travis Kalanick, trickle-down economics, two-pizza team, Uber and Lyft, Uber for X, uber lyft, ubercab, universal basic income, US Airways Flight 1549, VA Linux, warehouse automation, warehouse robotics, Watson beat the top human players on Jeopardy!, We are the 99%, web application, Whole Earth Catalog, winner-take-all economy, women in the workforce, Y Combinator, yellow journalism, zero-sum game, Zipcar

CHAPTER 8: MANAGING A WORKFORCE OF DJINNS 155 breakthroughs and business processes: Steve Lohr, “The Origins of ‘Big Data’: An Etymological Detective Story,” New York Times, February 1, 2013, https://bits.blogs.nytimes.com/2013/02/01/the-origins-of-big-data-an-etymological-detective-story/. 155 speech recognition and machine translation: Alon Halevy, Peter Norvig, and Fernando Pereira, “The Unreasonable Effectiveness of Data,” IEEE Intelligent Systems, 1541–1672/09, retrieved March 31, 2017, https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf. 156 “the sexiest job of the 21st century”: Thomas Davenport and D. J. Patil, “Data Scientist: The Sexiest Job of the 21st Century,” Harvard Business Review, October 2012, https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century. Hal Varian had used this same phrase about statistics in 2009. See “Hal Varian on How the Web Challenges Managers,” McKinsey & Company, January 2009, http://www.mckinsey.com/industries/high-tech/our-insights/hal-varian-on-how-the-web-challenges-managers. 157 “the right values for these parameters is something of a black art”: Sergey Brin and Larry Page, “The Anatomy of a Large-Scale Hypertextual Web Search Engine,” Stanford University, retrieved March 31, 2017, http://infolab.stanford. edu/~backrub/google.html. 158 as many as 50,000 subsignals: Danny Sullivan, “FAQ: All About the Google RankBrain Algorithm,” Search Engine Land, June 23, 2016, http://searchengine land.com/faq-all-about-the-new-google-rankbrain-algorithm-234440. 158 “new synapses for the global brain”: Tim O’Reilly, “Freebase Will Prove Addictive,” O’Reilly Radar, March 8, 2007, http://radar.oreilly.com/2007/03/free base-will-prove-addictive.html. 158 “10 experiments for every successful launch”: Matt McGee, “BusinessWeek Dives Deep into Google’s Search Quality,” Search Engine Land, October 6, 2009, http://searchengineland.com/businessweek-dives-deep-into-googles-search-quality-27317. 159 the manual that they provide: Search Quality Evaluator Guide, Google, March 14, 2017, http://static.googleusercontent.com/media/www.google.com/en//inside search/howsearchworks/assets/search qualityevaluatorguidelines.pdf. 160 “Another big difference”: Brin and Page, “The Anatomy of a Large-Scale Hypertextual Web Search Engine,” Section 3.2.

…

Its insight, that “simple models and a lot of data trump more elaborate models based on less data,” has been fundamental to progress in field after field, and is at the heart of many Silicon Valley companies. It is even more central to the latest breakthroughs in artificial intelligence. In 2008, D. J. Patil at LinkedIn and Jeff Hammerbacher at Facebook coined the term data science to describe their jobs, naming a field that a few years later was dubbed by Harvard Business Review as “the sexiest job of the 21st century.” Understanding the data science mindset and approach and how it differs from older methods of programming is critical for anyone who is grappling with the challenges of the twenty-first century. How Google deals with search quality provides important lessons.

…

Tellingly, Jeff Hammerbacher, who worked on Wall Street before leading the data team at Facebook, once said, “The best minds of my generation are thinking about how to make people click ads. That sucks.” Jeff left Facebook and now plays a dual role as chief scientist and cofounder at big data company Cloudera and faculty member of the Icahn School of Medicine at Mount Sinai, in New York, where he runs the Hammer Lab, a team of software developers and data scientists trying to understand how the immune system battles cancer. The choice of the problems to which we apply the superpowers of our new digital workforce is ultimately up to us. We are creating a race of djinns, eager to do our bidding. What shall we ask them to do? 9 “A HOT TEMPER LEAPS O’ER A COLD DECREE” I SPOKE IN EARLY 2017 AT A GATHERING OF MINISTERS FROM the Organisation for Economic Co-operation and Development (OECD) and G20 nations to discuss the digital future.

Merchants of Truth: The Business of News and the Fight for Facts by Jill Abramson

"World Economic Forum" Davos, 23andMe, 4chan, Affordable Care Act / Obamacare, Alexander Shulgin, Apple's 1984 Super Bowl advert, barriers to entry, Bernie Madoff, Bernie Sanders, Big Tech, Black Lives Matter, Cambridge Analytica, Charles Lindbergh, Charlie Hebdo massacre, Chelsea Manning, citizen journalism, cloud computing, commoditize, content marketing, corporate governance, creative destruction, crowdsourcing, data science, death of newspapers, digital twin, diversified portfolio, Donald Trump, East Village, Edward Snowden, fake news, Ferguson, Missouri, Filter Bubble, future of journalism, glass ceiling, Google Glasses, haute couture, hive mind, income inequality, information asymmetry, invisible hand, Jeff Bezos, Joseph Schumpeter, Khyber Pass, late capitalism, Laura Poitras, Marc Andreessen, Mark Zuckerberg, move fast and break things, Nate Silver, new economy, obamacare, Occupy movement, Paris climate accords, performance metric, Peter Thiel, phenotype, pre–internet, race to the bottom, recommendation engine, Robert Mercer, Ronald Reagan, Saturday Night Live, self-driving car, sentiment analysis, Sheryl Sandberg, Silicon Valley, Silicon Valley ideology, Silicon Valley startup, skunkworks, Snapchat, social contagion, social intelligence, social web, SoftBank, Steve Bannon, Steve Jobs, Steven Levy, tech billionaire, technoutopianism, telemarketer, the scientific method, The Wisdom of Crowds, Tim Cook: Apple, too big to fail, vertical integration, WeWork, WikiLeaks, work culture , Yochai Benkler, you are the product

Her team created: Mathew Ingram, “BuzzFeed Opens Up Access to Its Viral Dashboard,” Gigaom, September 2, 2010, https://gigaom.com/2010/09/02/buzzfeed-opens-up-access-to-its-viral-dashboard/. This was called Social Lift: Dao Nguyen and Ky Harlin, “How BuzzFeed Thinks about Data Science,” BuzzFeed, September 24, 2014, https://www.buzzfeed.com/daozers/how-buzzfeed-thinks-about-data-science?utm_term=.cuPg946z1#.uqLpMYjOL. The dashboard offered more than: Felix Oberholzer-Gee, “BuzzFeed—The Promise of Native Advertising,” Harvard Business School Case 714-512, June 2014 (revised August 2014), 539; Christine Lagorio-Chafkin, “Meet BuzzFeed’s Secret Weapon,” Inc., September 2, 2014, https://www.inc.com/christine-lagorio/buzzfeed-secret-growth-weapon.html.

…

When Zuckerberg shared a photo of himself giving out full-size candy bars for Halloween, Peretti made known that he “liked” it and shared a link to BuzzFeed’s coverage of a trick-or-treat stunt, led by Matt Stopera’s fawning confession, “I personally really admire him for this.” He kept in touch with Cameron Marlow, his grad-school friend who had since become Facebook’s chief data scientist, by posting public messages back and forth on each other’s “walls.” He made “Facebook-official” friendships with the company’s heads of product, media partnerships, and global creative strategy, one of its cofounders, and a board member. Facebook was growing fast. By 2008 the number of monthly users had nearly tripled, and over the next two years it would more than quadruple to over 600 million.

…

“News might not be as big a business as entertainment, but news is the best way to have a big impact on the world.” He disliked wishy-washy barometers like “impact,” but for the time being the term would have to do. The practical value of journalism was too nebulous for his quantitatively inclined mind. Then he received a gift from Nguyen and her team of data scientists. It was a new tool built into the dashboard, which they called a Heat Map. The software noted how far down the page a reader scrolled, then collated those data in one simple visual with whether the story lost readers’ attention and where. It was just what BuzzFeed needed to apply its analytical rigor to stories that were longer and larger in scope.

If Then: How Simulmatics Corporation Invented the Future by Jill Lepore

A Declaration of the Independence of Cyberspace, Alvin Toffler, anti-communist, Apollo 11, Buckminster Fuller, Cambridge Analytica, company town, computer age, coronavirus, cuban missile crisis, data science, desegregation, don't be evil, Donald Trump, Dr. Strangelove, Elon Musk, fake news, game design, George Gilder, Grace Hopper, Hacker Ethic, Howard Zinn, index card, information retrieval, Jaron Lanier, Jeff Bezos, Jeffrey Epstein, job automation, John Perry Barlow, land reform, linear programming, Mahatma Gandhi, Marc Andreessen, Mark Zuckerberg, mass incarceration, Maui Hawaii, Menlo Park, military-industrial complex, New Journalism, New Urbanism, Norbert Wiener, Norman Mailer, packet switching, Peter Thiel, profit motive, punch-card reader, RAND corporation, Robert Bork, Ronald Reagan, Rosa Parks, self-driving car, Silicon Valley, SimCity, smart cities, social distancing, South China Sea, Stewart Brand, technoutopianism, Ted Sorensen, Telecommunications Act of 1996, urban renewal, War on Poverty, white flight, Whole Earth Catalog

Although, notably, Google abandoned “Don’t be evil” in 2015, and Alphabet Inc. adopted a code of conduct that urged employees to “do the right thing,” http://blogs.wsj.com/digits/2015/10/02/as-google-becomes-alphabet-dont-be-evil-vanishes/. Mathematicians-turned-businessmen Jeff Hammerbacher and D. J. Patil claim to have coined the term “data scientist” in 2008, and not long after, data science was also embraced both within and outside the academy as a new scientific method, a “fourth paradigm,” following the earlier paradigms of empirical, theoretical, and computational analysis. Thomas H. Davenport and D. J. Patil, “Data Scientist: The Sexiest Job of the 21st Century,” Harvard Business Review, October 2012. Tony Hey, Stewart Tansley, and Kristin Michele Tolle, The Fourth Paradigm: Data-Intensive Scientific Discovery (Redmond, WA: Microsoft Research, 2009).

…

“Don’t be evil,” the motto of Google, marked the limit of a swaggering, devil-may-care ethical ambition; doing good did not come into it.7 Incubated decades before, beneath a honeycombed, geodesic dome in Wading River, this work found a place, too, in universities. In the 2010s, a flood of money into universities attempted to make the study of data a science, with data science initiatives, data science programs, data science degrees, data science centers.8 Much academic research that fell under the label “data science” produced excellent and invaluable work, across many fields of inquiry, findings that would not have been possible with computational discovery.9 And no field should be judged by its worst practitioners. Still, the shadiest data science, like the shadiest behavioral science, grew in influence by way of self-mystification, exaggerated claims, and all-around chicanery, including fast-changing, razzle-dazzle buzzwords, from “big data” to “data analytics.”

…

Much of university life had by the 2010s followed the model of the Media Lab, collapsing the boundaries between corporate commissions, academic inquiry, and hucksterism; at its worst, behavioral data science’s self-mystification was meant to boggle the mind, daunt critics, and entice corporate sponsors and venture capitalists. “Data science is in its infancy,” wrote one MIT computer scientist in 2015. “Few individuals or organizations understand the potential of and the paradigm shift associated with Data science, let alone understand it conceptually.”13 The more mystification, the wealthier the donors. The number of data science programs stretched into the hundreds, even though little consensus had been reached on the meaning or purpose of “data science.” New disciplines and methods take time to find their way; that’s all to the good.

The Black Box Society: The Secret Algorithms That Control Money and Information by Frank Pasquale

Adam Curtis, Affordable Care Act / Obamacare, Alan Greenspan, algorithmic trading, Amazon Mechanical Turk, American Legislative Exchange Council, asset-backed security, Atul Gawande, bank run, barriers to entry, basic income, Bear Stearns, Berlin Wall, Bernie Madoff, Black Swan, bonus culture, Brian Krebs, business cycle, business logic, call centre, Capital in the Twenty-First Century by Thomas Piketty, Chelsea Manning, Chuck Templeton: OpenTable:, cloud computing, collateralized debt obligation, computerized markets, corporate governance, Credit Default Swap, credit default swaps / collateralized debt obligations, crowdsourcing, cryptocurrency, data science, Debian, digital rights, don't be evil, drone strike, Edward Snowden, en.wikipedia.org, Evgeny Morozov, Fall of the Berlin Wall, Filter Bubble, financial engineering, financial innovation, financial thriller, fixed income, Flash crash, folksonomy, full employment, Gabriella Coleman, Goldman Sachs: Vampire Squid, Google Earth, Hernando de Soto, High speed trading, hiring and firing, housing crisis, Ian Bogost, informal economy, information asymmetry, information retrieval, information security, interest rate swap, Internet of things, invisible hand, Jaron Lanier, Jeff Bezos, job automation, John Bogle, Julian Assange, Kevin Kelly, Kevin Roose, knowledge worker, Kodak vs Instagram, kremlinology, late fees, London Interbank Offered Rate, London Whale, machine readable, Marc Andreessen, Mark Zuckerberg, Michael Milken, mobile money, moral hazard, new economy, Nicholas Carr, offshore financial centre, PageRank, pattern recognition, Philip Mirowski, precariat, profit maximization, profit motive, public intellectual, quantitative easing, race to the bottom, reality distortion field, recommendation engine, regulatory arbitrage, risk-adjusted returns, Satyajit Das, Savings and loan crisis, search engine result page, shareholder value, Silicon Valley, Snapchat, social intelligence, Spread Networks laid a new fibre optics cable between New York and Chicago, statistical arbitrage, statistical model, Steven Levy, technological solutionism, the scientific method, too big to fail, transaction costs, two-sided market, universal basic income, Upton Sinclair, value at risk, vertical integration, WikiLeaks, Yochai Benkler, zero-sum game

To have been prominent at a critical point in Internet development was a similar piece of luck. Google or Facebook were once in the right place at the right time. It’s not clear whether they are still better than anyone else at online data science, or whether their prominence is such that they’ve become the permanent “default.” We also have to ask whether data science is still key here, or just the data itself. When intermediaries like Google and Facebook leverage their enormous databases of personalized information to target advertising, how much value do they add in the process? This is a matter of some dispute.

…

Yet choosing a car, or even a restaurant, is not as straightforward as optimizing an engine or routing a drive. Does the recommendation engine take into account, say, whether the restaurant or car company gives its workers health benefits or maternity leave? Could we prompt it to do so? In their race for the most profitable methods of mapping social reality, the data scientists of Silicon Valley and Wall Street tend to treat recommendations as purely technical problems. The values and prerogatives that the encoded rules enact are hidden within black boxes.23 INTRODUCTION—THE NEED TO KNOW 9 The most obvious question is: Are these algorithmic applications fair? Why, for instance, does YouTube (owned by Google) so consistently beat out other video sites in Google’s video search results?

…

“Competition is one click away,” chant the Silicon Valley antitrust lawyers when someone calls out a behemoth firm for unfair or misleading business practices.149 It’s not so. Alternatives are demonstrably worse, and likely to remain so as long as the dominant firms’ self-reinforcing data advantage grows. 84 THE BLACK BOX SOCIETY Search and Compensation At the 2013 Governing Algorithms conference at New York University, a data scientist gave a dazzling presentation of how her company maximized ad revenue for its clients. She mapped out information exchanges among networks, advertisers, publishers, and the other stars of the Internet universe, emphasizing how computers are taught by skilled programmers like herself to fi nd unexpected correlations in click-through activity.

Whistleblower: My Journey to Silicon Valley and Fight for Justice at Uber by Susan Fowler

"Susan Fowler" uber, Airbnb, Albert Einstein, Big Tech, Burning Man, cloud computing, data science, deep learning, DevOps, Donald Trump, Elon Musk, end-to-end encryption, fault tolerance, Grace Hopper, Higgs boson, Large Hadron Collider, Lyft, Maui Hawaii, messenger bag, microservices, Mitch Kapor, Richard Feynman, ride hailing / ride sharing, self-driving car, Silicon Valley, TechCrunch disrupt, Travis Kalanick, Uber for X, uber lyft, work culture

It was that infrastructure—the servers, the operating systems, the networks, and all of the code that connected the applications together—that I would be working on, that I would need to make better, more reliable, and more fault-tolerant. After Engucation came more specialized training to prepare new hires for their particular roles within the company. New data scientists spent time with their data science teams, front-end developers learned how to work with the front-end code, and I would embed with one of the site reliability engineering (SRE) teams and learn the basics before I could join my permanent team. Eamon assigned me to one of his SRE teams for this phase of training.

…

The entire time I’d been a student at Penn, I’d watched as my undergraduate classmates in both physics and philosophy interviewed with companies like Facebook and Google, did software engineering internships in San Francisco over the summers, and boasted about their high-paying job offers from companies like Palantir and Microsoft. Almost all of the graduate students in my physics lab left academia after they completed their PhDs or finished their postdocs and moved to New York or San Francisco to work as data scientists, software engineers, and product managers. Jumping from physics into the technology industry wasn’t a very big leap: physics was a technical field, and physics students were technically inclined, comfortable with mathematics and computer science. I suspected that the leap from physics into software engineering wouldn’t be that difficult for me.

…

Everyone went out for drinks that night to celebrate the two new employees: me and the new office manager, Heidi. We were the only women in the office; as I later learned, they had us start on the same day so that we wouldn’t feel “alone.” I was the only woman in the office in a technical role (the only other technical woman, a data scientist, lived and worked in Boston), but that didn’t seem strange or unusual to me; at Penn, I had been the only woman on the floor where my lab was located in the David Rittenhouse Laboratory, and there were only men’s bathrooms on the floor where I worked. As the happy hour came to an end, we all stood outside the bar while each person called an Uber ride home.

Billion Dollar Brand Club: How Dollar Shave Club, Warby Parker, and Other Disruptors Are Remaking What We Buy by Lawrence Ingrassia

air freight, Airbnb, airport security, Amazon Robotics, augmented reality, barriers to entry, call centre, commoditize, computer vision, data science, fake news, fulfillment center, global supply chain, Hacker News, industrial robot, Jeff Bezos, Kickstarter, Kiva Systems, Lyft, Mark Zuckerberg, minimum viable product, natural language processing, Netflix Prize, rolodex, San Francisco homelessness, side project, Silicon Valley, Silicon Valley startup, Snapchat, Steve Jobs, supply-chain management, Uber and Lyft, uber lyft, warehouse automation, warehouse robotics, WeWork

A “decision tree” algorithm, as its name implies, correlates a cascading series of dozens or hundreds or thousands of data points (like branches on a tree) to winnow out items you probably won’t like and add things you probably will. “Random forest” algorithms are ensembles of dozens or even hundreds of different simpler algorithms that work together and correct for possible errors. Stitch Fix, which employs a chief algorithms officer and more than a hundred data scientists, probably is the only consumer product company ever to post on its website an “Algorithms Tour.” An elaborate and lengthy online graphic, it explains how Stitch Fix uses data (some not having an obvious correlation or connection) to act as a clothing matchmaker. “Each attribute that describes a piece of merchandise can be represented as data and reconciled to each client’s unique preferences,” Eric Colson, Stitch Fix’s chief algorithms officer, explains on the company’s MultiThreaded blog.

…

As Stitch Fix has developed more sophisticated algorithms, it has incorporated the use of computer vision to help select clothing. “We have our machines look at photos of clothing that customers like (e.g., from Pinterest), and look for visually similar items,” the website explains. And while the company initially sold apparel and accessories made by others, its data scientists in 2017 started designing “Stitch Fix exclusive brand” items by combining different style characteristics from popular clothing. In-house designers create these “Hybrid Designs” by taking ideas generated by artificial intelligence about the kinds of clothing its customers might like. Predictive analysis is now employed by a wide variety of digitally native brands.

…

Then the women were asked if they liked the color better than the results they got with a do-it-yourself kit at a drugstore. “We recruited people who were willing to put this product on their hair, with no idea who we really are. The results were positive enough for us to say we have something. Now what?” he recalls. The next step: tapping their knowledge of tech and data science to figure out how to replicate color customization for sale to fifty thousand women, not just fifty. “We wanted to do something that was really innovative, not bullshit innovative,” Mourad says. “Really different and value added.” Omar Mourad, Tamim’s younger brother and another of PriceGrabber’s cofounders, read everything he could about dyeing hair.

Age of the City: Why Our Future Will Be Won or Lost Together by Ian Goldin, Tom Lee-Devlin

15-minute city, 1960s counterculture, agricultural Revolution, Alvin Toffler, Anthropocene, anti-globalists, Berlin Wall, Bonfire of the Vanities, Brixton riot, call centre, car-free, carbon footprint, Cass Sunstein, charter city, Chuck Templeton: OpenTable:, clean water, cloud computing, congestion charging, contact tracing, coronavirus, COVID-19, CRISPR, data science, David Brooks, David Ricardo: comparative advantage, decarbonisation, deindustrialization, Deng Xiaoping, desegregation, Edward Glaeser, Edward Jenner, Enrique Peñalosa, fake news, Fall of the Berlin Wall, financial engineering, financial independence, future of work, General Motors Futurama, gentrification, germ theory of disease, global pandemic, global supply chain, global village, Haight Ashbury, Hernando de Soto, high-speed rail, household responsibility system, housing crisis, Howard Rheingold, income per capita, Induced demand, industrial robot, informal economy, invention of the printing press, invention of the wheel, Jane Jacobs, Jeff Bezos, job automation, John Perry Barlow, John Snow's cholera map, Kickstarter, knowledge economy, knowledge worker, labour mobility, Lewis Mumford, lockdown, Louis Pasteur, low interest rates, low skilled workers, manufacturing employment, Marshall McLuhan, mass immigration, megacity, Neal Stephenson, Network effects, New Urbanism, offshore financial centre, open borders, open economy, Pearl River Delta, race to the bottom, Ray Oldenburg, remote working, rent control, Republic of Letters, Richard Florida, ride hailing / ride sharing, rising living standards, Salesforce, Shenzhen special economic zone , smart cities, smart meter, Snow Crash, social distancing, special economic zone, spinning jenny, Steve Jobs, Stewart Brand, superstar cities, the built environment, The Death and Life of Great American Cities, The Great Good Place, The Wealth of Nations by Adam Smith, trade liberalization, trade route, Upton Sinclair, uranium enrichment, urban decay, urban planning, urban sprawl, Victor Gruen, white flight, working poor, working-age population, zero-sum game, zoonotic diseases

The combination of globalization and rapid technological progress has led to the disappearance of many manufacturing and clerical jobs – long anchors of the middle class – as work was either eliminated entirely or sent overseas to be performed more cheaply. As these jobs have disappeared, a barbell-shaped economy has emerged, characterized by high-paid knowledge jobs, such as management consultants and data scientists, on the one hand and low-paid service jobs, like baristas and warehouse workers, on the other. In the words of economists Maarten Goos and Alan Manning, we have seen the workforce split into ‘lousy’ and ‘lovely’ jobs.23 Manufacturing jobs in particular have long been the life force behind smaller cities and also many rural towns.

…

Company value chains often cross industry boundaries. Proximity to financial and professional services, for example, is an important drawcard for many large cities. The presence of top universities is another factor. The same universities that produce investment bankers also produce lawyers, data scientists and medics. Lastly, university-educated professionals typically choose to live in places where other university-educated professionals live. Partly this is about humans’ natural tendency to gravitate towards people with similar outlooks to themselves, what sociologists call ‘homophily’. Increasingly, it is also about romance.

…

As automation makes it increasingly challenging for poor countries to follow the traditional path of industrial development, a significant expansion of investment in education will be required. While we are sceptical that the future of knowledge work is entirely remote, we do see scope for certain activities to be shifted to well-educated workers in poor countries in cases where the gains to companies from lower wages overseas outweigh the disadvantages of remote collaboration. Data science, financial analysis and doubtless other areas of knowledge work could probably be performed from abroad. The experience of Bangalore suggests that a hub for offshoring knowledge work can also mature into a global centre of innovation in its own right. It is vital that, as they become more prosperous, the cities of the developing world do not adopt the car-based sprawl of many rich world cities.

No Ordinary Disruption: The Four Global Forces Breaking All the Trends by Richard Dobbs, James Manyika

2013 Report for America's Infrastructure - American Society of Civil Engineers - 19 March 2013, access to a mobile phone, additive manufacturing, Airbnb, Amazon Mechanical Turk, American Society of Civil Engineers: Report Card, asset light, autonomous vehicles, Bakken shale, barriers to entry, business cycle, business intelligence, carbon tax, Carmen Reinhart, central bank independence, circular economy, cloud computing, corporate governance, creative destruction, crowdsourcing, data science, demographic dividend, deskilling, digital capitalism, disintermediation, disruptive innovation, distributed generation, driverless car, Erik Brynjolfsson, financial innovation, first square of the chessboard, first square of the chessboard / second half of the chessboard, Gini coefficient, global supply chain, global village, high-speed rail, hydraulic fracturing, illegal immigration, income inequality, index fund, industrial robot, intangible asset, Intergovernmental Panel on Climate Change (IPCC), Internet of things, inventory management, job automation, Just-in-time delivery, Kenneth Rogoff, Kickstarter, knowledge worker, labor-force participation, low interest rates, low skilled workers, Lyft, M-Pesa, machine readable, mass immigration, megacity, megaproject, mobile money, Mohammed Bouazizi, Network effects, new economy, New Urbanism, ocean acidification, oil shale / tar sands, oil shock, old age dependency ratio, openstreetmap, peer-to-peer lending, pension reform, pension time bomb, private sector deleveraging, purchasing power parity, quantitative easing, recommendation engine, Report Card for America’s Infrastructure, RFID, ride hailing / ride sharing, Salesforce, Second Machine Age, self-driving car, sharing economy, Silicon Valley, Silicon Valley startup, Skype, smart cities, Snapchat, sovereign wealth fund, spinning jenny, stem cell, Steve Jobs, subscription business, supply-chain management, synthetic biology, TaskRabbit, The Great Moderation, trade route, transaction costs, Travis Kalanick, uber lyft, urban sprawl, Watson beat the top human players on Jeopardy!, working-age population, Zipcar

As big data emerged as the next big opportunity in sectors ranging from finance to government, both the talent supply and employers’ understanding of the skills they need struggled to keep up. “There aren’t enough data scientists, not even close,” said Sandy Pentland, a computer scientist and management thinker at MIT. “We tend to teach people that everything that matters happens between your ears when in fact it actually mostly happens between people.” Pentland argues that the lack of data scientists makes it more difficult to fully apply the technology.5 More than two-thirds of companies are struggling against limited or no capabilities in data analytics techniques.6 The story is not restricted to data analytics positions.

…

Richard Dobbs, Anu Madgavkar, Dominic Barton, Eric Labaye, James Minyika, Charles Roxburgh, Susan Lund, and Siddarth Madhav, “The world at work: Jobs, pay, and skills for 3.5 billion people,” June 2012, McKinsey & Company, www.mckinsey.com/insights/employment_and_growth/the_world_at_work. 5. Danny Palmer, “Not enough data scientists, MIT expert tells Computing,” Computing, September 4, 2013, www.computing.co.uk/ctg/news/2292485/not-enough-data-scientists-mit-expert-tells-computing. 6. Thomas Wailgum, “Monday metric: 68% of companies struggle with big data analytics,” ASUG News, March 18, 2013, www.asugnews.com/article/monday-metric-68-of-companies-struggle-with-big-data-analytics. 7.

…

Upgrading to premium membership—monthly prices start at $59.99 per month for the Business Plus account—affords the user greater insight into who has been looking at his or her profile, the ability to send more messages to potential leads, and the use of more advanced search filters.60 A third model is monetization of big data, either through innovative business-to-business offerings (for example, crowd-sourcing business intelligence or outsourced data science services) or through developing more relevant products, services, or content for which consumers are willing to pay. LinkedIn, for example, makes 20 percent of its revenue from subscriptions, 30 percent from marketing, and 50 percent from talent solutions, a core part of which is selling targeted talent intelligence and tools to recruiters.61 You will have to keep experimenting in order to capture more consumer surplus for your business.

Prediction Machines: The Simple Economics of Artificial Intelligence by Ajay Agrawal, Joshua Gans, Avi Goldfarb

Abraham Wald, Ada Lovelace, AI winter, Air France Flight 447, Airbus A320, algorithmic bias, AlphaGo, Amazon Picking Challenge, artificial general intelligence, autonomous vehicles, backpropagation, basic income, Bayesian statistics, Black Swan, blockchain, call centre, Capital in the Twenty-First Century by Thomas Piketty, Captain Sullenberger Hudson, carbon tax, Charles Babbage, classic study, collateralized debt obligation, computer age, creative destruction, Daniel Kahneman / Amos Tversky, data acquisition, data is the new oil, data science, deep learning, DeepMind, deskilling, disruptive innovation, driverless car, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, everywhere but in the productivity statistics, financial engineering, fulfillment center, general purpose technology, Geoffrey Hinton, Google Glasses, high net worth, ImageNet competition, income inequality, information retrieval, inventory management, invisible hand, Jeff Hawkins, job automation, John Markoff, Joseph Schumpeter, Kevin Kelly, Lyft, Minecraft, Mitch Kapor, Moneyball by Michael Lewis explains big data, Nate Silver, new economy, Nick Bostrom, On the Economy of Machinery and Manufactures, OpenAI, paperclip maximiser, pattern recognition, performance metric, profit maximization, QWERTY keyboard, race to the bottom, randomized controlled trial, Ray Kurzweil, ride hailing / ride sharing, Robert Solow, Salesforce, Second Machine Age, self-driving car, shareholder value, Silicon Valley, statistical model, Stephen Hawking, Steve Jobs, Steve Jurvetson, Steven Levy, strong AI, The Future of Employment, the long tail, The Signal and the Noise by Nate Silver, Tim Cook: Apple, trolley problem, Turing test, Uber and Lyft, uber lyft, US Airways Flight 1549, Vernor Vinge, vertical integration, warehouse automation, warehouse robotics, Watson beat the top human players on Jeopardy!, William Langewiesche, Y Combinator, zero-sum game

Cardiogram, in its preliminary study, used six thousand people, including just two hundred with an irregular heart rhythm. Over time, one way to collect further data is through feedback on whether the app’s users have or develop irregular heart rhythms. Where did the six thousand come from? Data scientists have excellent tools for assessing the amount of data required given the expected reliability of the prediction and the need for accuracy. These tools are called “power calculations” and tell you how many units you need to analyze to generate a useful prediction.5 The salient management point is that you must make a trade-off: more accurate predictions require more units to study, and acquiring these additional units can be costly.

…

A sabermetric analyst develops measures for the rewards that the team would receive from signing different players. Sabermetric analysts are baseball’s reward function engineers. Now, most teams have at least one such analyst, and the role has appeared, under different names, in other sports. Better prediction created a new high-level position on the org chart. The research scientists, data scientists, and vice presidents of analytics are listed as key roles in the online front office directories. The Houston Astros even have a separate decision sciences unit headed by former NASA engineer Sig Mejdal. The strategic change also means a switch in who the team employs to pick the players. These analytics experts have mathematical skills, but the finest of them understand best what to tell the prediction machine to do.

…

Joshua holds a PhD in economics from Stanford University and, in 2008, was awarded the Economic Society of Australia’s Young Economist Award (the Australian equivalent of the John Bates Clark medal). AVI GOLDFARB is the Ellison Professor of Marketing at the Rotman School of Management, University of Toronto. Avi is also chief data scientist at the Creative Destruction Lab, senior editor at Marketing Science, and a research associate at the National Bureau of Economic Research. Avi’s research focuses on the opportunities and challenges of the digital economy, with funding from Google, Industry Canada, Bell Canada, AIMIA, SSHRC, the National Science Foundation, the Sloan Foundation, and others.

The Ransomware Hunting Team: A Band of Misfits' Improbable Crusade to Save the World From Cybercrime by Renee Dudley, Daniel Golden

2021 United States Capitol attack, Amazon Web Services, Bellingcat, Berlin Wall, bitcoin, Black Lives Matter, blockchain, Brian Krebs, call centre, centralized clearinghouse, company town, coronavirus, corporate governance, COVID-19, cryptocurrency, data science, disinformation, Donald Trump, fake it until you make it, Hacker News, heat death of the universe, information security, late fees, lockdown, Menlo Park, Minecraft, moral hazard, offshore financial centre, Oklahoma City bombing, operational security, opioid epidemic / opioid crisis, Picturephone, pirate software, publish or perish, ransomware, Richard Feynman, Ross Ulbricht, seminal paper, smart meter, social distancing, strikebreaker, subprime mortgage crisis, tech worker, Timothy McVeigh, union organizing, War on Poverty, Y2K, zero day

Concerned about whether the HTCU could provide extra attention that people with autism might need, even broad-minded Marijn had his doubts about Peter’s pitch. But an HTCU program coordinator named Yvonne Horst was a believer. At her urging, the HTCU agreed to give Mark a six-month trial internship. Mark started as a junior data science intern at the HTCU at the same time as three professional data scientists hired by the unit. His new colleagues had university degrees and career experience at major companies such as accounting giant KPMG. Mark’s inexperience starkly contrasted with their advanced skills. “My six-month introductory course at ITvitae is not comparable to four years of university studies,” he conceded.

…

However, the Dutch National Police also sought candidates to work in its ten regional cyber squads, which handled more routine cases. Over time, ITvitae sent about two dozen additional students to work on those regional squads. Some handled tedious yet essential tasks such as reviewing crime-scene footage from security cameras, and were able to home in on small details that others might have missed. Others worked as data scientists or digital investigators. “If Mark did not advance the cold case, then Tom wouldn’t have fulfilled his dream to work at the police, and twenty-five others probably wouldn’t have had the opportunity,” Peter said, his voice quaking with emotion. “They are misfits. But they are very, very important to have at your organization

…

After crippling the DCH Regional Medical Center in Tuscaloosa, Alabama, and other hospitals in 2019, it doubled down on healthcare attacks in October 2020, sowing anxiety and confusion among patients and providers across the United States. The timing suggests that Ryuk was avenging one of the biggest and most damaging actions taken against ransomware. Since 2018, Microsoft’s Digital Crimes Unit—consisting of more than forty full-time investigators, analysts, data scientists, engineers, and attorneys—had been investigating TrickBot, the Russian malware that delivered Ryuk into victims’ computers. Concerns that the Putin regime might use TrickBot to disrupt the 2020 U.S. presidential election added urgency to the task, though the fears would prove unfounded. Microsoft investigators analyzed 61,000 samples of TrickBot malware, as well as the infrastructure underpinning the network of infected computers.

The Equality Machine: Harnessing Digital Technology for a Brighter, More Inclusive Future by Orly Lobel

2021 United States Capitol attack, 23andMe, Ada Lovelace, affirmative action, Airbnb, airport security, Albert Einstein, algorithmic bias, Amazon Mechanical Turk, augmented reality, barriers to entry, basic income, Big Tech, bioinformatics, Black Lives Matter, Boston Dynamics, Charles Babbage, choice architecture, computer vision, Computing Machinery and Intelligence, contact tracing, coronavirus, corporate social responsibility, correlation does not imply causation, COVID-19, crowdsourcing, data science, David Attenborough, David Heinemeier Hansson, deep learning, deepfake, digital divide, digital map, Elon Musk, emotional labour, equal pay for equal work, feminist movement, Filter Bubble, game design, gender pay gap, George Floyd, gig economy, glass ceiling, global pandemic, Google Chrome, Grace Hopper, income inequality, index fund, information asymmetry, Internet of things, invisible hand, it's over 9,000, iterative process, job automation, Lao Tzu, large language model, lockdown, machine readable, machine translation, Mark Zuckerberg, market bubble, microaggression, Moneyball by Michael Lewis explains big data, natural language processing, Netflix Prize, Network effects, Northpointe / Correctional Offender Management Profiling for Alternative Sanctions, occupational segregation, old-boy network, OpenAI, openstreetmap, paperclip maximiser, pattern recognition, performance metric, personalized medicine, price discrimination, publish or perish, QR code, randomized controlled trial, remote working, risk tolerance, robot derives from the Czech word robota Czech, meaning slave, Ronald Coase, Salesforce, self-driving car, sharing economy, Sheryl Sandberg, Silicon Valley, social distancing, social intelligence, speech recognition, statistical model, stem cell, Stephen Hawking, Steve Jobs, Steve Wozniak, surveillance capitalism, tech worker, TechCrunch disrupt, The Future of Employment, TikTok, Turing test, universal basic income, Wall-E, warehouse automation, women in the workforce, work culture , you are the product

For example, the Bread for the World Institute recently issued a report showing that 92 percent of gender-specific economic data is missing from Africa. We are not measuring the struggles of those who are most in need. Millions remain in the shadows. But we can tackle the problem today thanks to technology, and the institute is working with volunteer coders, data scientists, statisticians, and graphic designers to begin to systematically collect what matters—to materialize the missing data and bring problems like these to light. Supreme Court Justice Louis Brandeis famously said that sunlight is the best of disinfectants, and electric light the most efficient policeman.

…

An equality machine mindset actively charts the course of the future, anticipating the many ways in which that future is unknown. This means designing governance systems and infrastructure that will continue to channel technological advancement down a progressive path. Inside Out, Outside In In late 2021, Frances Haugen, a thirty-seven-year-old data scientist, became one of the most famous whistleblowers in recent times. Testifying before both American and European legislatures, Haugen revealed that Facebook, her former employer, was time and again choosing profit over the safety and well-being of its users. Haugen asserted that Facebook consistently valued profit over safety by allowing algorithms to favor hateful content in order to bring users back to the social media platform for more traffic.

…

As we move beyond traditional litigation frameworks, government agencies also become research and development arms that incentivize, test, approve, and monitor proactive prevention programs. The immense challenge of harnessing technology for equality is one that must involve people from all disciplines and sectors. Social scientists, for example, must work with data scientists to provide context and ask the critical questions about definitions, the sources of data, and the interpretation of patterns. There are accelerated, heated debates and numerous legislative proposals to tighten the regulation of digital technology, including to amend Section 230 of the U.S. Communications Decency Act of 1996 in ways that would limit digital platform immunity and require platforms to moderate illegal content.

Advances in Financial Machine Learning by Marcos Lopez de Prado

algorithmic trading, Amazon Web Services, asset allocation, backtesting, behavioural economics, bioinformatics, Brownian motion, business process, Claude Shannon: information theory, cloud computing, complexity theory, correlation coefficient, correlation does not imply causation, data science, diversification, diversified portfolio, en.wikipedia.org, financial engineering, fixed income, Flash crash, G4S, Higgs boson, implied volatility, information asymmetry, latency arbitrage, margin call, market fragmentation, market microstructure, martingale, NP-complete, P = NP, p-value, paper trading, pattern recognition, performance metric, profit maximization, quantitative trading / quantitative finance, RAND corporation, random walk, risk free rate, risk-adjusted returns, risk/return, selection bias, Sharpe ratio, short selling, Silicon Valley, smart cities, smart meter, statistical arbitrage, statistical model, stochastic process, survivorship bias, transaction costs, traveling salesman

These features were discovered by different analysts studying a wide range of instruments and asset classes. The goal of the strategist is to make sense of all these observations and to formulate a general theory that explains them. Therefore, the strategy is merely the experiment designed to test the validity of this theory. Team members are data scientists with a deep knowledge of financial markets and the economy. Remember, the theory needs to explain a large collection of important features. In particular, a theory must identify the economic mechanism that causes an agent to lose money to us. Is it a behavioral bias? Asymmetric information? Regulatory constraints?

…

One of the scenarios of interest is how the strategy would perform if history repeated itself. However, the historical path is merely one of the possible outcomes of a stochastic process, and not necessarily the most likely going forward. Alternative scenarios must be evaluated, consistent with the knowledge of the weaknesses and strengths of a proposed strategy. Team members are data scientists with a deep understanding of empirical and experimental techniques. A good backtester incorporates in his analysis meta-information regarding how the strategy came about. In particular, his analysis must evaluate the probability of backtest overfitting by taking into account the number of trials it took to distill the strategy.

…

I have listed a few of them in the references section. The core audience of this book is investment professionals with a strong ML background. My goals are that you monetize what you learn in this book, help us modernize finance, and deliver actual value for investors. This book also targets data scientists who have successfully implemented ML algorithms in a variety of fields outside finance. If you have worked at Google and have applied deep neural networks to face recognition, but things do not seem to work so well when you run your algorithms on financial data, this book will help you. Sometimes you may not understand the financial rationale behind some structures (e.g., meta-labeling, the triple-barrier method, fracdiff), but bear with me: Once you have managed an investment portfolio long enough, the rules of the game will become clearer to you, along with the meaning of these chapters. 1.5 Requisites Investment management is one of the most multi-disciplinary areas of research, and this book reflects that fact.

Thank You for Being Late: An Optimist's Guide to Thriving in the Age of Accelerations by Thomas L. Friedman

3D printing, additive manufacturing, affirmative action, Airbnb, AltaVista, Amazon Web Services, Anthropocene, Apple Newton, autonomous vehicles, Ayatollah Khomeini, barriers to entry, Berlin Wall, Bernie Sanders, Big Tech, biodiversity loss, bitcoin, blockchain, Bob Noyce, business cycle, business process, call centre, carbon tax, centre right, Chris Wanstrath, Clayton Christensen, clean tech, clean water, cloud computing, cognitive load, corporate social responsibility, creative destruction, CRISPR, crowdsourcing, data science, David Brooks, deep learning, demand response, demographic dividend, demographic transition, Deng Xiaoping, digital divide, disinformation, Donald Trump, dual-use technology, end-to-end encryption, Erik Brynjolfsson, fail fast, failed state, Fairchild Semiconductor, Fall of the Berlin Wall, Ferguson, Missouri, first square of the chessboard / second half of the chessboard, Flash crash, fulfillment center, game design, gig economy, global pandemic, global supply chain, Great Leap Forward, illegal immigration, immigration reform, income inequality, indoor plumbing, intangible asset, Intergovernmental Panel on Climate Change (IPCC), Internet of things, invention of the steam engine, inventory management, Irwin Jacobs: Qualcomm, Jeff Bezos, job automation, John Markoff, John von Neumann, Khan Academy, Kickstarter, knowledge economy, knowledge worker, land tenure, linear programming, Live Aid, low interest rates, low skilled workers, Lyft, Marc Andreessen, Mark Zuckerberg, mass immigration, Maui Hawaii, Menlo Park, Mikhail Gorbachev, mutually assured destruction, Neil Armstrong, Nelson Mandela, ocean acidification, PalmPilot, pattern recognition, planetary scale, power law, pull request, Ralph Waldo Emerson, ransomware, Ray Kurzweil, Richard Florida, ride hailing / ride sharing, Robert Gordon, Ronald Reagan, Salesforce, Second Machine Age, self-driving car, shareholder value, sharing economy, Silicon Valley, Skype, smart cities, Solyndra, South China Sea, Steve Jobs, subscription business, supercomputer in your pocket, synthetic biology, systems thinking, TaskRabbit, tech worker, TED Talk, The Rise and Fall of American Growth, Thomas L Friedman, Tony Fadell, transaction costs, Transnistria, uber lyft, undersea cable, urban decay, urban planning, Watson beat the top human players on Jeopardy!, WikiLeaks, women in the workforce, Y2K, Yogi Berra, zero-sum game

This approach has driven a lot of innovation in education, most notably the partnership between Udacity, AT&T, and Georgia Tech to create an online master’s degree in computer science for $6,600 for the entire course—as compared with the $45,000 it would cost for two years on campus at Georgia Tech. Coursera has partnered with Johns Hopkins and Rice to create a similar certificate in data science. This is driving down the cost of education for everyone. The education “pie just got bigger,” said Blase. “We can now assist you to get the job of your dreams.” That’s intelligent assistance. “We do $250 million of training a year,” said Blase. A lot is teaching people to climb poles, install services, and run retail stores, but now a lot more is in data science, software-defined networks, Web development, introduction to programming, machine learning, and the Internet of Things.

…

It can’t just be the advocacy of abstract principles. When you put your value set together with your analysis of how the Machine works and your understanding of how it is affecting people and culture in different contexts, you have a worldview that you can then apply to all kinds of situations to produce your opinions. Just as a data scientist needs an algorithm to cut through all the unstructured data and all the noise to see the relevant patterns, an opinion writer needs a worldview to create heat and light. But to keep that worldview fresh and relevant, I suggested to Bojia, you have to be constantly reporting and learning—more so today than ever.

…

“The idea,” he explained, “was that if you know exactly how the gas turbine and combustion engine work, you can use the laws of physics and say: ‘This is how it is going to work and when it is going to break.’ There was not a belief in the traditional engineering community that the data had much to offer. They used the data to verify their physics models and then act upon them. The new breed of data scientists here say: ‘You don’t need to understand the physics to look for and find the patterns.’ There are patterns that a human mind could not find, because the signals are so weak early on that you won’t see them. But now that we have all this processing power, those weak signals just pop out at you.

The People vs Tech: How the Internet Is Killing Democracy (And How We Save It) by Jamie Bartlett

Ada Lovelace, Airbnb, AlphaGo, Amazon Mechanical Turk, Andrew Keen, autonomous vehicles, barriers to entry, basic income, Bernie Sanders, Big Tech, bitcoin, Black Lives Matter, blockchain, Boris Johnson, Californian Ideology, Cambridge Analytica, central bank independence, Chelsea Manning, cloud computing, computer vision, creative destruction, cryptocurrency, Daniel Kahneman / Amos Tversky, data science, deep learning, DeepMind, disinformation, Dominic Cummings, Donald Trump, driverless car, Edward Snowden, Elon Musk, Evgeny Morozov, fake news, Filter Bubble, future of work, general purpose technology, gig economy, global village, Google bus, Hans Moravec, hive mind, Howard Rheingold, information retrieval, initial coin offering, Internet of things, Jeff Bezos, Jeremy Corbyn, job automation, John Gilmore, John Maynard Keynes: technological unemployment, John Perry Barlow, Julian Assange, manufacturing employment, Mark Zuckerberg, Marshall McLuhan, Menlo Park, meta-analysis, mittelstand, move fast and break things, Network effects, Nicholas Carr, Nick Bostrom, off grid, Panopticon Jeremy Bentham, payday loans, Peter Thiel, post-truth, prediction markets, QR code, ransomware, Ray Kurzweil, recommendation engine, Renaissance Technologies, ride hailing / ride sharing, Robert Mercer, Ross Ulbricht, Sam Altman, Satoshi Nakamoto, Second Machine Age, sharing economy, Silicon Valley, Silicon Valley billionaire, Silicon Valley ideology, Silicon Valley startup, smart cities, smart contracts, smart meter, Snapchat, Stanford prison experiment, Steve Bannon, Steve Jobs, Steven Levy, strong AI, surveillance capitalism, TaskRabbit, tech worker, technological singularity, technoutopianism, Ted Kaczynski, TED Talk, the long tail, the medium is the message, the scientific method, The Spirit Level, The Wealth of Nations by Adam Smith, The Wisdom of Crowds, theory of mind, too big to fail, ultimatum game, universal basic income, WikiLeaks, World Values Survey, Y Combinator, you are the product

As I walked in to the ordinary looking office in central London – all offices are normal looking, except those of tech firms – I spotted a framed posted with a picture of Trump and a quote from famed US pollster Frank Luntz: ‘There are no longer any experts except Cambridge Analytica. They were Trump’s digital team who figured out how to win.’ Rows of employees were sitting staring at screens: project managers, IT specialists and data scientists.29 On a shelf in Nix’s glass office were copies of The Bad Boys of Brexit, the book written by UKIP donor Arron Banks, and Stealing Elections by John Fund. He seemed perfectly happy with these techniques, and said that micro-targeting was just getting started and represented the future of campaigning.

…

Although he rejects any single ‘why’, it’s clear that he thinks data was instrumental: One of our central ideas was that the campaign had to do things in the field of data that have never been done before. This included a) integrating data from social media, online advertising, websites, apps, canvassing, direct mail, polls, online fundraising, activist feedback and some new things we tried such as a new way to do polling . . . and b) having experts in physics and machine learning do proper data science in the way only they can – i.e. far beyond the normal skills applied in political campaigns. We were the first campaign in the UK to put almost all our money into digital communication then have it partly controlled by people whose normal work was subjects like quantum information . . . If you want to make big improvements in communication, my advice is – hire physicists, not communications people from normal companies

…

This did not seem to bother investors, since this 11-person business, with ambitions to revolutionise the entire trucking industry by building a self-driving fleet, managed to raise millions of dollars of funding from venture capitalists. ‘Everyone thought I was mad,’ 27-year-old Stefan told me when I visited Starsky’s Florida headquarters, a large rented property in a gated community, a few months ago. These days however, like many other industries, trucking is being disrupted by data science, artificial intelligence and venture capital. Stefan had agreed to let me drive in Starsky’s newest and shiniest truck with their resident driver Tony Hughes, a diminutive and friendly man with 20 years’ experience, who is perhaps better described as part-driver, part-machine supervisor. Tony is in his fifties, with a high school diploma in general studies from Shawnee Mission Northwest (Kansas) and a ‘solid track record of achieving efficient, cost-effective transportation operations’, but now finds himself training the machines that might eventually put him out of a job.

Modern Vim: Craft Your Development Environment With Vim 8 and Neovim by Drew Neil

bash_history, Bram Moolenaar, data science, Debian, DevOps, en.wikipedia.org, functional programming, microservices, pull request, remote working, text mining

Venkat Subramaniam (280 pages) ISBN: 9781934356760 $35 Data Science Essentials in Python Go from messy, unstructured artifacts stored in SQL and NoSQL databases to a neat, well-organized dataset with this quick reference for the busy data scientist. Understand text mining, machine learning, and network analysis; process numeric data with the NumPy and Pandas modules; describe and analyze data using statistical and network-theoretical methods; and see actual examples of data analysis at work. This one-stop solution covers the essential data science you need in Python. Dmitry Zinoviev (224 pages) ISBN: 9781680501841 $29 Practical Programming, Third Edition Classroom-tested by tens of thousands of students, this new edition of the best-selling intro to programming book is for anyone who wants to understand computer science.

The Costs of Connection: How Data Is Colonizing Human Life and Appropriating It for Capitalism by Nick Couldry, Ulises A. Mejias

"World Economic Forum" Davos, 23andMe, Airbnb, Amazon Mechanical Turk, Amazon Web Services, behavioural economics, Big Tech, British Empire, call centre, Cambridge Analytica, Cass Sunstein, choice architecture, cloud computing, colonial rule, computer vision, corporate governance, dark matter, data acquisition, data is the new oil, data science, deep learning, different worldview, digital capitalism, digital divide, discovery of the americas, disinformation, diversification, driverless car, Edward Snowden, emotional labour, en.wikipedia.org, European colonialism, Evgeny Morozov, extractivism, fake news, Gabriella Coleman, gamification, gig economy, global supply chain, Google Chrome, Google Earth, hiring and firing, income inequality, independent contractor, information asymmetry, Infrastructure as a Service, intangible asset, Internet of things, Jaron Lanier, job automation, Kevin Kelly, late capitalism, lifelogging, linked data, machine readable, Marc Andreessen, Mark Zuckerberg, means of production, military-industrial complex, move fast and break things, multi-sided market, Naomi Klein, Network effects, new economy, New Urbanism, PageRank, pattern recognition, payday loans, Philip Mirowski, profit maximization, Ray Kurzweil, RFID, Richard Stallman, Richard Thaler, Salesforce, scientific management, Scientific racism, Second Machine Age, sharing economy, Shoshana Zuboff, side hustle, Sidewalk Labs, Silicon Valley, Slavoj Žižek, smart cities, Snapchat, social graph, social intelligence, software studies, sovereign wealth fund, surveillance capitalism, techlash, The Future of Employment, the scientific method, Thomas Davenport, Tim Cook: Apple, trade liberalization, trade route, undersea cable, urban planning, W. E. B. Du Bois, wages for housework, work culture , workplace surveillance

Take any aspect of the social world, even ones for which we currently lack a causal model, and simply generate a proxy for it; there is no limit to what might work as such a proxy, and indeed the vagueness as to what is a “proxy variable” is a problem in legal proceedings that increasingly rely on them.64 So data scientists may ask: Could visual cues in Google Street View scenes be proxies of the likelihood of nearby crime? Could patterns in the distribution of more-expensive car models in Google Earth pictures be proxy demographic variables (income levels, relative poverty/wealth)? The temptation to pursue such proxy hunts is considerable, especially when public census data is costly and only intermittently collected.65 The scope for social experimentation that data relations provide to parts of the social quantification sector is huge but has sometimes proved controversial.66 Privacy concerns may act as a constraint, but, if so, China’s assumed lower sensitivity to privacy concerns works as a market advantage for its AI industry.67 Collect Everything If Big Data reasoning relies on the predictive power that comes from repetitive processing of unstructured data, this data can be generated, directly or indirectly, from whole populations.

…

This claim might shock those who see in “datafication . . . an essential enrichment in human comprehension” or “a great infrastructural project that rivals” the Enlightenment’s Encyclopédie.134 But let’s get beyond the hype and look at what is actually going on in the social sciences today. A good example is the work of celebrated data scientist Alex Pentland at the MIT Media Lab. Pentland contributed to the World Economic Forum’s Global Information Technology Reports in 2008, 2009, and 2014;135 his research team won the Defense Advanced Research Projects Agency’s (DARPA) prize to commemorate the internet’s fortieth anniversary.136 In his book Social Physics, Pentland reaches back to the origins of sociology.

…

There is, however, one bright note: the rise of critical information science that seeks to systematically establish the distortions woven into social caching’s presentations of the world. There is now an emerging movement for algorithmic justice, which has raised awareness of many specific issues. The critical movement within data science concerned with “Fairness, Accountability and Transparency” has regular conferences, and a number of universities have focused programs for investigating how algorithms cover the social domain.160 More generally, an important intersection between critical information science, legal theory, and social theory is opening up the question of how the social qualification sector presents the social world for action by powerful institutions.161 US civil society has generated some effective campaigning.

New Laws of Robotics: Defending Human Expertise in the Age of AI by Frank Pasquale

affirmative action, Affordable Care Act / Obamacare, Airbnb, algorithmic bias, Amazon Mechanical Turk, Anthropocene, augmented reality, Automated Insights, autonomous vehicles, basic income, battle of ideas, Bernie Sanders, Big Tech, Bill Joy: nanobots, bitcoin, blockchain, Brexit referendum, call centre, Cambridge Analytica, carbon tax, citizen journalism, Clayton Christensen, collective bargaining, commoditize, computer vision, conceptual framework, contact tracing, coronavirus, corporate social responsibility, correlation does not imply causation, COVID-19, critical race theory, cryptocurrency, data is the new oil, data science, decarbonisation, deep learning, deepfake, deskilling, digital divide, digital twin, disinformation, disruptive innovation, don't be evil, Donald Trump, Douglas Engelbart, driverless car, effective altruism, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, Evgeny Morozov, fake news, Filter Bubble, finite state, Flash crash, future of work, gamification, general purpose technology, Google Chrome, Google Glasses, Great Leap Forward, green new deal, guns versus butter model, Hans Moravec, high net worth, hiring and firing, holacracy, Ian Bogost, independent contractor, informal economy, information asymmetry, information retrieval, interchangeable parts, invisible hand, James Bridle, Jaron Lanier, job automation, John Markoff, Joi Ito, Khan Academy, knowledge economy, late capitalism, lockdown, machine readable, Marc Andreessen, Mark Zuckerberg, means of production, medical malpractice, megaproject, meta-analysis, military-industrial complex, Modern Monetary Theory, Money creation, move fast and break things, mutually assured destruction, natural language processing, new economy, Nicholas Carr, Nick Bostrom, Norbert Wiener, nuclear winter, obamacare, One Laptop per Child (OLPC), open immigration, OpenAI, opioid epidemic / opioid crisis, paperclip maximiser, paradox of thrift, pattern recognition, payday loans, personalized medicine, Peter Singer: altruism, Philip Mirowski, pink-collar, plutocrats, post-truth, pre–internet, profit motive, public intellectual, QR code, quantitative easing, race to the bottom, RAND corporation, Ray Kurzweil, recommendation engine, regulatory arbitrage, Robert Shiller, Rodney Brooks, Ronald Reagan, self-driving car, sentiment analysis, Shoshana Zuboff, Silicon Valley, Singularitarianism, smart cities, smart contracts, software is eating the world, South China Sea, Steve Bannon, Strategic Defense Initiative, surveillance capitalism, Susan Wojcicki, tacit knowledge, TaskRabbit, technological solutionism, technoutopianism, TED Talk, telepresence, telerobotics, The Future of Employment, The Turner Diaries, Therac-25, Thorstein Veblen, too big to fail, Turing test, universal basic income, unorthodox policies, wage slave, Watson beat the top human players on Jeopardy!, working poor, workplace surveillance , Works Progress Administration, zero day

At its beginning, credit scoring was confined to one area of life (determining eligibility for loans) and based on a limited set of data (repayment history). Over decades, credit scores and similar measures have come to inform other determinations, including insurance rates and employment opportunities. More recently, data scientists have proposed more data sources for credit scoring, ranging from the way people type, to their political affiliation, to the types of websites they visit online. The Chinese government has also expanded the stakes of surveillance, proposing that “social credit scores” play a role in determining what trains or planes a citizen can board, what hotels a person can stay in, and what schools a family’s children can attend.

…

Malpractice law is designed to give patients reassurance that if their physician falls below a standard of care, a penalty will be imposed and some portion of it dedicated to their recovery.22 If providers fail to use sufficiently representative datasets to develop their medical AI, lawsuits should help hold them accountable, to ensure that everyone benefits from AI in medicine (not just those lucky enough to belong to the most studied groups). Data scientists sometimes joke that AI is simply a better-marketed form of statistics. Certainly narrow AI, designed to make specific predictions, is based on quantifying probability.23 It is but one of many steps taken over the past two decades to modernize medicine with a more extensive evidence base.24 Medical researchers have seized on predictive analytics, big data, artificial intelligence, machine learning, and deep learning as master metaphors for optimizing system performance.

…

Once a single perfect way of doing some task was found, there was little rationale for a human to continue doing it. Taylorism dovetailed with the psychological school of behaviorism, which sought to develop for human beings a blend of punishments and reinforcements reminiscent of animal training. The rise of data-driven predictive analytics has given behaviorism new purchase. The chief data scientist of a Silicon Valley e-learning firm once stated, “The goal of everything we do is to change people’s actual behavior at scale. When people use our app, we can capture their behaviors, identify good and bad behaviors, and develop ways to reward the good and punish the bad. We can test how actionable our cues are for them and how profitable for us.”72 A crowded field of edtech innovators promises to drastically reduce the cost of primary, secondary, and tertiary education with roughly similar methods: broadcast courses, intricate labyrinths of computerized assessment tools, and 360-degree surveillance tools to guarantee that students are not cheating.

The Data Journalism Handbook by Jonathan Gray, Lucy Chambers, Liliana Bounegru

Amazon Web Services, barriers to entry, bioinformatics, business intelligence, carbon footprint, citizen journalism, correlation does not imply causation, crowdsourcing, data science, David Heinemeier Hansson, eurozone crisis, fail fast, Firefox, Florence Nightingale: pie chart, game design, Google Earth, Hans Rosling, high-speed rail, information asymmetry, Internet Archive, John Snow's cholera map, Julian Assange, linked data, machine readable, moral hazard, MVC pattern, New Journalism, openstreetmap, Ronald Reagan, Ruby on Rails, Silicon Valley, social graph, Solyndra, SPARQL, text mining, Wayback Machine, web application, WikiLeaks

Less guessing, less looking for quotes; instead, a journalist can build a strong position supported by data, and this can affect the role of journalism greatly. Additionally, getting into data journalism offers a future perspective. Today, when newsrooms downsize, most journalists hope to switch to public relations. Data journalists or data scientists, though, are already a sought-after group of employees, not only in the media. Companies and institutions around the world are looking for “sensemakers” and professionals who know how to dig through data and transform it into something tangible. There is a promise in data, and this is what excites newsrooms, making them look for a new type of reporter.

…

Far from it. In the information age, journalists are needed more than ever to curate, verify, analyze, and synthesize the wash of data. In that context, data journalism has profound importance for society. Today, making sense of big data, particularly unstructured data, will be a central goal for data scientists around the world, whether they work in newsrooms, Wall Street, or Silicon Valley. Notably, that goal will be substantially enabled by a growing set of common tools, whether they’re employed by government technologists opening Chicago, healthcare technologists, or newsroom developers. — Alex Howard, O’Reilly Media Our Lives Are Data Good data journalism is hard, because good journalism is hard.

…

INSEAD Working Paper, 2010 Business Models for Data Journalism Amidst all the interest and hope regarding data-driven journalism, there is one question that newsrooms are always curious about: what are the business models? While we must be careful about making predictions, a look at the recent history and current state of the media industry can give us some insight. Today there are many news organizations who have gained by adopting new approaches. Terms like “data journalism” and the newest buzzword, “data science,” may sound like they describe something new, but this is not strictly true. Instead these new labels are just ways of characterizing a shift that has been gaining strength over decades. Many journalists seem to be unaware of the size of the revenue that is already generated through data collection, data analytics, and visualization.

The Science of Storytelling: Why Stories Make Us Human, and How to Tell Them Better by Will Storr

data science, David Brooks, Demis Hassabis, Gordon Gekko, heat death of the universe, meta-analysis, Steven Pinker, TED Talk, theory of mind, Wall-E

Jockers (Allen Lane, 2016) p. 163. 4.1 Researchers downloaded 1,327: ‘The emotional arcs of stories are dominated by six basic shapes’, Andrew J. Reagan, Lewis Mitchell, Dilan Kiley, Christopher M. Danforth, Peter Sheridan Dodds, EPJ Data Science, 5:31, 4 November 2016. 4.2 For the neuroscientist Professor Beau Lotto: Deviate, Beau Lotto (W&N, 2017) Kindle location 685. When the data scientist David Robinson: Examining the arc of 100,000 stories: a tidy analysis by David Robinson, http://varianceexplained.org/r/tidytext-plots, 26 April 2017. The psychologist and story analyst Professor Jordan Peterson: Maps of Meaning video lectures.

…

It’s only by being active, and having the courage to take on the external world with all its challenges and provocations, that these core mechanisms can ever be broken down and rebuilt. For the neuroscientist Professor Beau Lotto it’s ‘not just important to be active, it is neurologically necessary’. It’s the only way we grow. When the data scientist David Robinson analysed an enormous tranche of 112,000 plots including books, movies, television episodes and video games, his algorithm found one common story shape. Robinson described this as, ‘Things get worse and worse until, at the last minute, they get better.’ The pattern he detected reveals that many stories have a point, just prior to their resolution, in which the hero endures some deeply significant test.

Work in the Future The Automation Revolution-Palgrave MacMillan (2019) by Robert Skidelsky Nan Craig

3D printing, Airbnb, algorithmic trading, AlphaGo, Alvin Toffler, Amazon Web Services, anti-work, antiwork, artificial general intelligence, asset light, autonomous vehicles, basic income, behavioural economics, business cycle, cloud computing, collective bargaining, Computing Machinery and Intelligence, correlation does not imply causation, creative destruction, data is the new oil, data science, David Graeber, David Ricardo: comparative advantage, deep learning, DeepMind, deindustrialization, Demis Hassabis, deskilling, disintermediation, do what you love, Donald Trump, driverless car, Erik Brynjolfsson, fake news, feminist movement, Ford Model T, Frederick Winslow Taylor, future of work, Future Shock, general purpose technology, gig economy, global supply chain, income inequality, independent contractor, informal economy, Internet of things, Jarndyce and Jarndyce, Jarndyce and Jarndyce, job automation, job polarisation, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, John von Neumann, Joseph Schumpeter, knowledge economy, Loebner Prize, low skilled workers, Lyft, Mark Zuckerberg, means of production, moral panic, Network effects, new economy, Nick Bostrom, off grid, pattern recognition, post-work, Ronald Coase, scientific management, Second Machine Age, self-driving car, sharing economy, SoftBank, Steve Jobs, strong AI, tacit knowledge, technological determinism, technoutopianism, TED Talk, The Chicago School, The Future of Employment, the market place, The Nature of the Firm, The Wealth of Nations by Adam Smith, Thorstein Veblen, Turing test, Uber for X, uber lyft, universal basic income, wealth creators, working poor

xii Notes on Contributors She then switched over to the private sector, working as a quant for the hedge fund D.E. Shaw in the middle of the credit crisis, and then for RiskMetrics, a risk software company that assesses risk for the holdings of hedge funds and banks. She left finance in 2011 and started working as a data scientist in the New York start-up scene, building models that predicted people’s purchases and clicks. She wrote Doing Data Science in 2013 and launched the Lede Program in Data Journalism at Columbia in 2014. She is a regular contributor to Bloomberg View and wrote the book Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. She recently founded ORCAA, an algorithmic auditing company.

…

and next you scan your memories, as well as your closet, for outfits that can optimize to that definition of success. In the case of a formal algorithm, the definition of success does not waver; it’s codified in computer code, and that precise concept of “success,” as well as the associated concept of the cost of failure, are embedded in a mathematical object called the objective function. Once the data scientist decides on the objective function, and the historical training data, the ensuing algorithm is largely determined. Sounds simple, and it sometimes is. But when the output of the algorithm (the prediction itself ) is used in a powerful way, a feedback loop is created: the algorithm doesn’t just predict the future, it causes the future.

The Seventh Sense: Power, Fortune, and Survival in the Age of Networks by Joshua Cooper Ramo

air gap, Airbnb, Alan Greenspan, Albert Einstein, algorithmic trading, barriers to entry, Berlin Wall, bitcoin, Bletchley Park, British Empire, cloud computing, Computing Machinery and Intelligence, crowdsourcing, Danny Hillis, data science, deep learning, defense in depth, Deng Xiaoping, drone strike, Edward Snowden, Fairchild Semiconductor, Fall of the Berlin Wall, financial engineering, Firefox, Google Chrome, growth hacking, Herman Kahn, income inequality, information security, Isaac Newton, Jeff Bezos, job automation, Joi Ito, Laura Poitras, machine translation, market bubble, Menlo Park, Metcalfe’s law, Mitch Kapor, Morris worm, natural language processing, Neal Stephenson, Network effects, Nick Bostrom, Norbert Wiener, Oculus Rift, off-the-grid, packet switching, paperclip maximiser, Paul Graham, power law, price stability, quantitative easing, RAND corporation, reality distortion field, Recombinant DNA, recommendation engine, Republic of Letters, Richard Feynman, road to serfdom, Robert Metcalfe, Sand Hill Road, secular stagnation, self-driving car, Silicon Valley, Skype, Snapchat, Snow Crash, social web, sovereign wealth fund, Steve Jobs, Steve Wozniak, Stewart Brand, Stuxnet, superintelligent machines, systems thinking, technological singularity, The Coming Technological Singularity, The Wealth of Nations by Adam Smith, too big to fail, Vernor Vinge, zero day

These systems run faster and better and more profitably because they are a shared system. They are gated by technology standards and by common connection. When we say that networks crave gates, this is the sort of gate we mean. If you had to look for your friends one by one on Facebook, Friendster, MySpace, and Google Plus, you’d exhaust yourself. So one winner emerges. Data scientists attribute the success of these winning nodes to preferential attachment—the idea that if Brian Arthur is using Microsoft Word, and I’m using it, you are likely to do so too. But there’s another secret: More widespread adoption makes the whole system faster. Think of five mechanics trying to fix a broken engine.

…

“seven friends in ten days”: Chamath Palihapitiya, “How We Put Facebook on the Path to 1 Billion Users” (lecture for the Udemy course “Growth Hacking: An Introduction,” published January 9, 2013, and available at https://www.youtube.com/watch?v=raIUQP71SBU). Pretty soon: Eman Yasser Daraghmi and Shyan-Ming Yuan, “We Are So Close, Less Than 4 Degrees Separating You and Me!,” Computers in Human Behavior 30 (January 2014), 273–85. Data scientists: Laurent Hébert-Dufresne et al. “Complex Networks as an Emerging Property of Hierarchical Preferential Attachment,” Physical Review E 92, 062809 (2015). But there’s another secret: Albert-László Barabási, “Network Science,” Philosophical Transactions of the Royal Society A: Mathematical, Physical, and Engineering Sciences 371, no. 1987 (March 2013).

…

Their appeal was both the potential of the new and the chance to get away from the rotting smell of old politics. This is one reason it is wrong to look at the world and consider it filled merely with random events, with so-called black swans. In fact, patterns appear everywhere. They can be searched and mapped and studied with the tools of data science, but, of course, they can also be felt. They may surprise you if you don’t know how to look for them. But they are there. There is more to human history than earthquakes alone. Even if it can’t be predicted, complexity in any system, whether it is an Indonesian coral reef or a Russian computer network, can at least be measured.

The Creativity Code: How AI Is Learning to Write, Paint and Think by Marcus Du Sautoy

3D printing, Ada Lovelace, Albert Einstein, algorithmic bias, AlphaGo, Alvin Roth, Andrew Wiles, Automated Insights, Benoit Mandelbrot, Bletchley Park, Cambridge Analytica, Charles Babbage, Claude Shannon: information theory, computer vision, Computing Machinery and Intelligence, correlation does not imply causation, crowdsourcing, data is the new oil, data science, deep learning, DeepMind, Demis Hassabis, Donald Trump, double helix, Douglas Hofstadter, driverless car, Elon Musk, Erik Brynjolfsson, Fellow of the Royal Society, Flash crash, Gödel, Escher, Bach, Henri Poincaré, Jacquard loom, John Conway, Kickstarter, Loebner Prize, machine translation, mandelbrot fractal, Minecraft, move 37, music of the spheres, Mustafa Suleyman, Narrative Science, natural language processing, Netflix Prize, PageRank, pattern recognition, Paul Erdős, Peter Thiel, random walk, Ray Kurzweil, recommendation engine, Rubik’s Cube, Second Machine Age, Silicon Valley, speech recognition, stable marriage problem, Turing test, Watson beat the top human players on Jeopardy!, wikimedia commons

Was Rembrandt’s considerable output sufficient for an algorithm to be able to learn how to create a new portrait that would be recognisably his? The internet contains millions of images of cats, but Shakespeare wrote thirty-seven plays and Beethoven nine symphonies. Will creative genius be protected from machine learning by a lack of data? Data scientists at Microsoft and Delft University of Technology were of the view that there was enough data for an algorithm to learn how to paint like Rembrandt. Ron Augustus from Microsoft, who worked on the project, believed the old master himself would approve of their project: ‘We are using technology and data like Rembrandt uses his paints and brushes to create something new.’

…

By layering the negatives on top of each other and exposing the resulting image Galton was rather shocked to see the array of distorted and ugly faces he had used transform into a handsome composite. It seems that when you smooth out the asymmetries, you end up with something quite attractive. The data scientists would have to devise a more clever plan if they were going to produce a painting that might be taken for a Rembrandt. Their algorithm would have to create new eyes, a new nose and a new mouth, as if it could see the world through Rembrandt’s eyes. Having created these features, they then investigated the proportions Rembrandt used to place these features on the faces he painted.

…

The current drive by humans to create algorithmic creativity is in the most part not one fuelled by the desire for extending artistic creation but rather enlarging a company bank balance. There is a huge amount of hype about AI. There are too many initiatives that are branded as AI but which are little more than statistics or data science. Just as any company wishing to make it at the turn of the millennium would put .com on the end of its name, today it is the addition of the tag AI or Deep which is what companies are using to jump on the bandwagon. Companies would love to be able to convince an audience that this AI is so great that it can write articles on its own, that it can compose music, paint Rembrandts.

We Are Data: Algorithms and the Making of Our Digital Selves by John Cheney-Lippold

algorithmic bias, bioinformatics, business logic, Cass Sunstein, centre right, computer vision, critical race theory, dark matter, data science, digital capitalism, drone strike, Edward Snowden, Evgeny Morozov, Filter Bubble, Google Chrome, Google Earth, Hans Moravec, Ian Bogost, informal economy, iterative process, James Bridle, Jaron Lanier, Julian Assange, Kevin Kelly, late capitalism, Laura Poitras, lifelogging, Lyft, machine readable, machine translation, Mark Zuckerberg, Marshall McLuhan, mass incarceration, Mercator projection, meta-analysis, Nick Bostrom, Norbert Wiener, offshore financial centre, pattern recognition, price discrimination, RAND corporation, Ray Kurzweil, Richard Thaler, ride hailing / ride sharing, Rosa Parks, Silicon Valley, Silicon Valley startup, Skype, Snapchat, software studies, statistical model, Steven Levy, technological singularity, technoutopianism, the scientific method, Thomas Bayes, Toyota Production System, Turing machine, uber lyft, web application, WikiLeaks, Zimmermann PGP

Big data needs to be analyzed at a distance, because as Franco Moretti claims, “distance, let me repeat it, is a condition of knowledge: it allows you to focus on units that are much smaller or much larger than the [singular] text.”66 Big-data practitioners like Viktor Mayer-Schönberger and Kenneth Cukier aren’t really concerned with analytical rigor: “moving to a large scale changes not only the expectations of precision but the practical ability to achieve exactitude.”67 Big data is instead about understanding a new version of the world, one previously unavailable to us because in an antiquated era before digital computers and ubiquitous surveillance, we didn’t have the breadth and memory to store it all. Moretti, though, is no data scientist. He’s a humanist, best known for coining the concept of “distant reading.” Unlike the disciplinary English practice of “close reading,” in which a section of a certain text is painstakingly parsed for its unique semantic and grammatological usage, distant reading finds utility in the inability to do a close reading of large quantities of texts.68 One human cannot read and remember every last word of Hemingway, and she would be far less able to do the same for other U.S.

…

Nidhi Makhija-Chimnani, “People’s Insights Volume 1, Issue 52: Vicks Mobile Ad Campaign,” MSLGroup, 2013, asia.mslgroup.com. 59. David Lazer, Ryan Kennedy, Gary King, and Alessandro Vespignani, “The Parable of Google Flu: Traps in Big Data Analysis,” Science 343 (March 14, 2014): 1203–1205. 60. Steve Lohr, “For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights,” New York Times, August 17, 2014. 61. Lazer, Kennedy, and Vespignani, “Parable of Google Flu.” 62. Foucault, “Society Must Be Defended,” 249. 63. Mauricio Santillana, D. Wendong Zhang, Benjamin Althouse, and John Ayers, “What Can Digital Disease Detection Learn from (an External Revision to) Google Flu Trends?

…

It suggests, “We can stop looking for models. We can analyze the data without hypotheses about what it might show.”119 Of course, as discussed in the preceding pages, data is never “just” data. But this antihypothetical interpretation, with all its methodological baggage, has set the trajectory for big-data science itself. Data-mining and machine-learning research is at a fever pitch. The data culled from the surveillant assemblages of our networked society has been dubbed “oil of the 21st century,” while algorithmic analytics are “the combustion engine.”120 Even with powerful critiques of this hyperpositivism coming from scholars like David Ribes, Steven J.

Python for Algorithmic Trading: From Idea to Cloud Deployment by Yves Hilpisch

algorithmic trading, Amazon Web Services, automated trading system, backtesting, barriers to entry, bitcoin, Brownian motion, cloud computing, coronavirus, cryptocurrency, data science, deep learning, Edward Thorp, fiat currency, global macro, Gordon Gekko, Guido van Rossum, implied volatility, information retrieval, margin call, market microstructure, Myron Scholes, natural language processing, paper trading, passive investing, popular electronics, prediction markets, quantitative trading / quantitative finance, random walk, risk free rate, risk/return, Rubik’s Cube, seminal paper, Sharpe ratio, short selling, sorting algorithm, systematic trading, transaction costs, value at risk

The book by Hilpisch (2020) focuses exclusively on the application of algorithms for machine and deep learning to the problem of identifying statistical inefficiencies and exploiting economic inefficiencies through algorithmic trading: Guido, Sarah, and Andreas Müller. 2016. Introduction to Machine Learning with Python: A Guide for Data Scientists. Sebastopol: O’Reilly. Hilpisch, Yves. 2020. Artificial Intelligence in Finance: A Python-Based Guide. Sebastopol: O’Reilly. VanderPlas, Jake. 2016. Python Data Science Handbook: Essential Tools for Working with Data. Sebastopol: O’Reilly. The books by Hastie et al. (2008) and James et al. (2013) provide a thorough, mathematical overview of popular machine learning techniques and algorithms: Hastie, Trevor, Robert Tibshirani, and Jerome Friedman. 2008.

…

Background information about Python as applied to finance, financial data science, and artificial intelligence can be found in the following books: Hilpisch, Yves. 2018. Python for Finance: Mastering Data-Driven Finance. 2nd ed. Sebastopol: O’Reilly. ⸻. 2020. Artificial Intelligence in Finance: A Python-Based Guide. Sebastopol: O’Reilly. McKinney, Wes. 2017. Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. 2nd ed. Sebastopol: O’Reilly. Ramalho, Luciano. 2021. Fluent Python: Clear, Concise, and Effective Programming. 2nd ed. Sebastopol: O’Reilly. VanderPlas, Jake. 2016. Python Data Science Handbook: Essential Tools for Working with Data.

…

The reader can post questions and comments in the user forum on the Quant Platform at any time (accounts are free). Online/video training (paid subscription) The Python Quants offer comprehensive online training programs that make use of the contents presented in the book and that add additional content, covering important topics such as financial data science, artificial intelligence in finance, Python for Excel and databases, and additional Python tools and skills. Contents and Structure Here’s a quick overview of the topics and contents presented in each chapter. Chapter 1, Python and Algorithmic Trading The first chapter is an introduction to the topic of algorithmic trading—that is, the automated trading of financial instruments based on computer algorithms.

The Big Nine: How the Tech Titans and Their Thinking Machines Could Warp Humanity by Amy Webb

"Friedman doctrine" OR "shareholder theory", Ada Lovelace, AI winter, air gap, Airbnb, airport security, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, algorithmic bias, AlphaGo, Andy Rubin, artificial general intelligence, Asilomar, autonomous vehicles, backpropagation, Bayesian statistics, behavioural economics, Bernie Sanders, Big Tech, bioinformatics, Black Lives Matter, blockchain, Bretton Woods, business intelligence, Cambridge Analytica, Cass Sunstein, Charles Babbage, Claude Shannon: information theory, cloud computing, cognitive bias, complexity theory, computer vision, Computing Machinery and Intelligence, CRISPR, cross-border payments, crowdsourcing, cryptocurrency, Daniel Kahneman / Amos Tversky, data science, deep learning, DeepMind, Demis Hassabis, Deng Xiaoping, disinformation, distributed ledger, don't be evil, Donald Trump, Elon Musk, fail fast, fake news, Filter Bubble, Flynn Effect, Geoffrey Hinton, gig economy, Google Glasses, Grace Hopper, Gödel, Escher, Bach, Herman Kahn, high-speed rail, Inbox Zero, Internet of things, Jacques de Vaucanson, Jeff Bezos, Joan Didion, job automation, John von Neumann, knowledge worker, Lyft, machine translation, Mark Zuckerberg, Menlo Park, move fast and break things, Mustafa Suleyman, natural language processing, New Urbanism, Nick Bostrom, one-China policy, optical character recognition, packet switching, paperclip maximiser, pattern recognition, personalized medicine, RAND corporation, Ray Kurzweil, Recombinant DNA, ride hailing / ride sharing, Rodney Brooks, Rubik’s Cube, Salesforce, Sand Hill Road, Second Machine Age, self-driving car, seminal paper, SETI@home, side project, Silicon Valley, Silicon Valley startup, skunkworks, Skype, smart cities, South China Sea, sovereign wealth fund, speech recognition, Stephen Hawking, strong AI, superintelligent machines, surveillance capitalism, technological singularity, The Coming Technological Singularity, the long tail, theory of mind, Tim Cook: Apple, trade route, Turing machine, Turing test, uber lyft, Von Neumann architecture, Watson beat the top human players on Jeopardy!, zero day