continuous integration

76 results back to index


pages: 333 words: 64,581

Clean Agile: Back to Basics by Robert C. Martin

Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, Boeing 737 MAX, c2.com, cognitive load, continuous integration, DevOps, disinformation, double entry bookkeeping, en.wikipedia.org, failed state, Frederick Winslow Taylor, index card, iterative process, Kanban, Kubernetes, loose coupling, microservices, remote working, revision control, scientific management, Turing machine

The software system did not need a feeder controller that was independent of the printer controller. The waste of human resources, not to mention the emotional angst and adversarial posturing, led to a very uncomfortable environment. I believe that this environment was instrumental, at least in part, to their eventual downfall. Continuous Integration In the early days of Agile, the practice of Continuous Integration meant that developers checked in their source code changes and merged them with the main line every “couple of hours.”6 All unit tests and acceptance tests kept passing. No feature branches remained unintegrated. Any changes that should not be active when deployed were dealt with by using toggles. 6.

Contents Cover Page About This eBook Praise for Clean Agile Half-Title Page Series-Page Title Page Copyright Page Dedication Contents Foreword Preface Acknowledgments About the Author Chapter 1 Introduction to Agile History of Agile Snowbird Agile Overview Circle of Life Conclusion Chapter 2 The Reasons for Agile Professionalism Reasonable Expectations The Bill of Rights Conclusion Chapter 3 Business Practices Planning Small Releases Acceptance Tests Whole Team Conclusion Chapter 4 Team Practices Metaphor Sustainable Pace Collective Ownership Continuous Integration Standup Meetings Conclusion Chapter 5 Technical Practices Test-Driven Development Refactoring Simple Design Pair Programming Conclusion Chapter 6 Becoming Agile Agile Values The Menagerie Transformation Coaching Certification Agile in the Large Agile Tools Coaching—An Alternative View Conclusion (Bob Again) Chapter 7 Craftsmanship The Agile Hangover Expectation Mismatch Moving Apart Software Craftsmanship Ideology versus Methodology Does Software Craftsmanship Have Practices?

Continuous Technical Readiness Stable Productivity Inexpensive Adaptability Continuous Improvement Fearless Competence QA Should Find Nothing Test Automation We Cover for Each Other Honest Estimates You Need to Say “No” Continuous Aggressive Learning Mentoring The Bill of Rights Customer Bill of Rights Developer Bill of Rights Customers Developers Conclusion Chapter 3 Business Practices Planning Trivariate Analysis Stories and Points ATM Stories Stories Story Estimation Managing the Iteration The Demo Velocity Small Releases A Brief History of Source Code Control Tapes Disks and SCCS Subversion Git and Tests Acceptance Tests Tools and Methodologies Behavior-Driven Development The Practice Whole Team Co-Location Conclusion Chapter 4 Team Practices Metaphor Domain-Driven Design Sustainable Pace Overtime Marathon Dedication Sleep Collective Ownership The X Files Continuous Integration Then Came Continuous Build The Continuous Build Discipline Standup Meetings Pigs and Chickens? Shout-out Conclusion Chapter 5 Technical Practices Test-Driven Development Double-Entry Bookkeeping The Three Rules of TDD Debugging Documentation Fun Completeness Design Courage Refactoring Red/Green/Refactor Bigger Refactorings Simple Design Design Weight Pair Programming What Is Pairing?


Agile Project Management with Kanban (Developer Best Practices) by Eric Brechner

Amazon Web Services, cloud computing, continuous integration, crowdsourcing, data science, DevOps, don't repeat yourself, en.wikipedia.org, index card, Kaizen: continuous improvement, Kanban, loose coupling, minimum viable product, pull request, software as a service

For each of these models, I cover how to assign completed items to deployments, how to track when those items are deployed, and how the Kanban approach affects your deployment cadence and customer feedback mechanisms. The topics covered are: Continuous integration Continuous push Continuous publishing Continuous deployment Checklist Continuous integration Continuous integration is a software engineering practice that dates back to the early 1990s. This approach involves automating your build and testing it to the extent that you can integrate high-quality code changes with confidence into the main branch every day, perhaps even several times a day. Kanban is a natural fit for continuous integration because the tasks you complete daily are production-ready; they certainly pass the criteria for integration.

The owners of the upstream repo accept pull requests from the downstream repos used by contributing individuals and teams, validate the changes, and accept none, some, or all of the changes, depending on the validation results. For large projects, multiple repos may be stored on the hosting service, with one or more related components in each repo. As with continuous integration, Kanban is a natural fit for continuous push because the tasks you complete daily are production-ready—they certainly pass the criteria that makes them prime for pushing to the upstream repo (or for submitting through a pull request). And as in continuous integration, large projects need to mitigate against merge and interface conflicts when new code is pushed to the upstream repo. The mitigation follows a familiar pattern: Pull from the upstream repo into your repo.

Meanwhile, they can carry on the continuous deployment of the latest version without worrying about interface conflicts. Continuous deployment of our services is a beautiful system that supports our continuous publishing of the platform and apps and our continuous integration of value for customers. Checklist Here’s a checklist of actions for deploying components, apps, and services by using Kanban: For continuous integration of changes: • Follow an integration strategy for your team branch that avoids merge and interface conflicts. • Divide the rightmost Done column of your signboard into sections for each branch you use and special sections for FastTrack changes and deployed changes. • Use the Done sections to track integrations of completed work.


pages: 224 words: 48,804

The Productive Programmer by Neal Ford

anti-pattern, business process, c2.com, continuous integration, database schema, domain-specific language, don't repeat yourself, Firefox, general-purpose programming language, knowledge worker, Larry Wall, Paradox of Choice, Ruby on Rails, side project, type inference, web application, William of Occam

Use a Canonical Build Machine The other process required in every development shop is continuous integration. Continuous integration is a process where you build the entire project, run tests, generate documentation, and do all the other activities that make software on a regular basis (the more often the better, generally you should build every time you check in code to version control). Continuous integration is supported by software of the same name. Ideally, the continuous integration server runs on a separate machine, monitoring your check-ins to version control. Every time you perform a code check-in, the continuous integration server springs to life, running a build command that you specify (in a build file like Ant, Nant, Rake, or Make) that usually includes performing a full build, setting up the database for testing, running the entire suite of unit tests, running code analysis, and deploying the application to perform a “smoke test.”

Even single developers benefit from having a continuous integration server as the lone build machine. It prevents you from inadvertently allowing tool dependencies from creeping into 72 CHAPTER 5: CANONICALITY your project. If you can build your application with a single command on a standalone machine, you obviously have the configuration correct. Numerous continuous integration servers exist, both commercial and open source. CruiseControl* is an open source project created by ThoughtWorks, and it has ports to Java, .NET, and Ruby. Other continuous integration servers include Bamboo,† Hudson,‡ TeamCity,§ and LuntBuild.‖ Indirection Platforms provide structure for heavy items.

As developers, we create too few of these little throwaway tools, frequently, because, we don’t think of tools in this way. Software development has lots of obvious automation targets: builds, continuous integration, and documentation. This chapter covers some less obvious but no less valuable ways to automate development chores, from the single keystroke all the way to little applications. Don’t Reinvent Wheels General infrastructure setup is something you have to do for every project: setting up version control, continuous integration, user IDs, etc. Buildix* is an open source project (developed by ThoughtWorks) that greatly simplifies this process for Java-based projects.


pages: 540 words: 103,101

Building Microservices by Sam Newman

airport security, Amazon Web Services, anti-pattern, business logic, business process, call centre, continuous integration, Conway's law, create, read, update, delete, defense in depth, don't repeat yourself, Edward Snowden, fail fast, fallacies of distributed computing, fault tolerance, index card, information retrieval, Infrastructure as a Service, inventory management, job automation, Kubernetes, load shedding, loose coupling, microservices, MITM: man-in-the-middle, platform as a service, premature optimization, pull request, recommendation engine, Salesforce, SimCity, social graph, software as a service, source of truth, sunk-cost fallacy, systems thinking, the built environment, the long tail, two-pizza team, web application, WebSocket

encryption of backup, Encrypt Backups retrieval via service calls, Data Retrieval via Service Calls securing at rest, Securing Data at Rest(see also security) shared, Example: Shared Data shared static, Example: Shared Static Data data encryption, Go with the Well Known data pumpsbackup, Backup Data Pump data retrieval via, Data Pumps event, Event Data Pump serial use of, Alternative Destinations database decomposition, The Database-Understanding Root Causesbreaking foreign key relationships, Example: Breaking Foreign Key Relationships incremental approach to, Cost of Change overview of, Summary refactoring databases, Staging the Break selecting separation points, Getting to Grips with the Problem selecting separation timing, Understanding Root Causes shared data, Example: Shared Data shared static data, Example: Shared Static Data shared tables, Example: Shared Tables transactional boundaries, Transactional Boundaries database integration, The Shared Database database scalingCommand-Query Responsibility Segregation (CQRS), CQRS for reads, Scaling for Reads for writes, Scaling for Writes service availability vs. data durability, Availability of Service Versus Durability of Data shared infrastructure, Shared Database Infrastructure decision-making guidelines, A Principled Approach-A Real-World Examplecustomized approach to, Combining Principles and Practices practices for, Practices principles for, Principles real-world example, A Real-World Example strategic goals, Strategic Goals decompositional techniquesdatabases (see database decomposition) identifying/packaging contexts, Breaking Apart MusicCorp modules, Modules seam concept, It’s All About Seams selecting separation points, The Reasons to Split the Monolith selecting separation timing, Understanding Root Causes shared libraries, Shared Libraries decoupling, Autonomous, Orchestration Versus Choreography degrading functionality, Degrading Functionality delivery bottlenecks, Delivery Bottlenecks deploymentartifacts, images as, Images as Artifacts artifacts, operating system, Operating System Artifacts artifacts, platform-specific, Platform-Specific Artifacts automation, Automation blue/green deployment, Separating Deployment from Release build pipeline, Build Pipelines and Continuous Delivery bundled service release, And the Inevitable Exceptions continuous integration basics, A Brief Introduction to Continuous Integration continuous integration checklist, Are You Really Doing It? continuous integration in microservices, Mapping Continuous Integration to Microservices custom images, Custom Images environment definition, Environment Definition environments to consider, Environments immutable servers, Immutable Servers interfaces, A Deployment Interface microservices vs. monolithic systems, Ease of Deployment, Deployment overview of, Summary separating from release, Separating Deployment from Release service configuration, Service Configuration virtualization approach, From Physical to Virtual virtualization, hypervisors, Traditional Virtualization virtualization, traditional, Traditional Virtualization virtualization, type 2, Traditional Virtualization deputy problem, The Deputy Problem design principles, Principles, Bringing It All Together-Highly Observable(see also architectural principles) design/delivery practicesdevelopment of, Practices real-world example, A Real-World Example directory service, Common Single Sign-On Implementations DiRT (Disaster Recovery Test), The Antifragile Organization distributed systemsfallacies of, Local Calls Are Not Like Remote Calls, Failure Is Everywhere key promises of, Composability distributed transactions, Distributed Transactions DNS service, DNS Docker, Docker documentationHAL (Hypertext Application Language), HAL and the HAL Browser importance of, Documenting Services self-describing systems, HAL and the HAL Browser Swagger, Swagger domain-driven design, Microservices Dropwizard, Tailored Service Template DRY (Don't Repeat Yourself), DRY and the Perils of Code Reuse in a Microservice World dummies, Mocking or Stubbing durability, How Much Is Too Much?

In this chapter, we’re going to look at some techniques and technology that can help us when deploying microservices into fine-grained architectures. We’re going to start off, though, by taking a look at continuous integration and continuous delivery. These related but different concepts will help shape the other decisions we’ll make when thinking about what to build, how to build it, and how to deploy it. A Brief Introduction to Continuous Integration Continuous integration (CI) has been around for a number of years at this point. It’s worth spending a bit of time going over the basics, however, as especially when we think about the mapping between microservices, builds, and version control repositories, there are some different options to consider.

synchronous vs. asynchronous, Synchronous Versus Asynchronous compensating transactions, Abort the Entire Operation composability, Composability configuration drift, Immutable Servers configuration, service, Service Configuration confused deputy problem, The Deputy Problem consistencyin CAP theorem, CAP Theorem sacrificing, Sacrificing Consistency constraints, Constraints Consul, Consul consumer-driven contracts (CDCs), Consumer-Driven Tests to the Rescue-It’s About Conversations content delivery network (CDN), Client-Side, Proxy, and Server-Side Caching content management systems (CMS), Example: CMS as a service continuous delivery (CD), Microservices, Build Pipelines and Continuous Delivery continuous integration (CI)basics, A Brief Introduction to Continuous Integration checklist for, Are You Really Doing It? mapping to microservices, Mapping Continuous Integration to Microservices Conway's lawevidence of, Evidence in reverse, Conway’s Law in Reverse statement of, Conway’s Law and System Design summary of, Summary coordination process, Distributed Transactions core team, Role of the Custodians CoreOS, Docker correlation IDs, Correlation IDs CP system, AP or CP?


pages: 779 words: 116,439

Test-Driven Development With Python by Harry J. W. Percival

business logic, continuous integration, database schema, Debian, DevOps, don't repeat yourself, duck typing, Firefox, loose coupling, MVC pattern, off-by-one error, platform as a service, pull request, web application, WebSocket

As with all things, it’s a judgement call as to whether the additional effort is worth it in particular circumstances. 364 | Chapter 19: Test Isolation, and “Listening to Your Tests” www.it-ebooks.info CHAPTER 20 Continuous Integration (CI) As our site grows, it takes longer and longer to run all of our functional tests. If this continues, the danger is that we’re going to stop bothering. Rather than let that happen, we can automate the running of functional tests by setting up a “Continuous Integration” or CI server. That way, in day-to-day development, we can just run the FT that we’re working on at that time, and rely on the CI server to run all the tests automatically, and let us know if we’ve broken anything accidentally.

Shaw, Learn Python The Hard Way: http://learnpythonthehard way.org/ • [iwp] Al Sweigart, Invent Your Own Computer Games With Python: http://invent withpython.com • [tddbe] Kent Beck, TDD By Example, Addison-Wesley • [refactoring] Martin Fowler, Refactoring, Addison-Wesley • [seceng] Ross Anderson, Security Engineering, Second Edition, Addison-Wesley: http://www.cl.cam.ac.uk/~rja14/book.html • [jsgoodparts] Douglas Crockford, JavaScript: The Good Parts, O’Reilly • [twoscoops] Daniel Greenfield and Audrey Roy, Two Scoops of Django, http:// twoscoopspress.com/products/two-scoops-of-django-1-6 • [mockfakestub] Emily Bache, Mocks, Fakes and Stubs, https://leanpub.com/mocksfakes-stubs • [GOOSGBT] Steve Freeman and Nat Pryce, Growing Object-Oriented Software Guided by Tests, Addison-Wesley 441 www.it-ebooks.info www.it-ebooks.info Index A acceptance test (see functional tests/testing (FT)) acceptance tests, 398 aesthetics (see layout and style) agile movement in software development, 81 Ajax, 249, 269 ALLOWED_HOSTS, 151 Anderson, Ross, 53 Ansible, 166, 423–426 any function, 39, 46 architectural solutions to test problems, 402 assertion messages, 16, 18, 57, 263 AssertionError, 46 assertRegex, 85 assertTemplateUsed, 90 assertTrue, 46 asynchronous JavaScript, 272–275 authentication, 241 backend, 285–293 customising, 245–248, 277 in Django, 282 login view, 281–284 minimum custom user model, 293–297 mocking (see mocks/mocking) Mozilla Persona, 242 pre-authentication, 303–306 testing logout, 299 testing view, 278 tests as documentation, 296 automation, in deployment, 132, 157–166 (see also deployment) automation, in provisioning, 166 B Bash, 141 Beck, Kent, 36, 44 Behavior-Driven Development (BDD) tools, 434 Bernhardt, Gary, 337, 399, 403 best practices in testing, 397 Big Design Up Front, 81 black box test (see functional tests/testing (FT)) Bootstrap, 118–126 jumbotron, 125 large inputs, 125 table styling, 126 boundaries, 403 browsers, 434 browsers, headless, 372 C caching, 435 CI server (see continuous integration (CI)) class-based generic views, 413–421 class-based views, 413 clean architecture, 403 We’d like to hear your suggestions for improving our indexes. Send email to index@oreilly.com. 443 www.it-ebooks.info code smell, 59, 65, 193, 301 collectstatic, 127–129 comments, 15, 86 commits, 18, 24, 30, 110 configuration management tools, 167 (see also Fabric) context managers, 177 continuous integration (CI), 365–385, cdvii adding required plugins, 368 best practices, 385 configuring Jenkins, 367 debugging with screenshots, 374–378 installing Jenkins, 365 JavaScript tests, 381–384 project setup, 369 Selenium race conditions, 378–381 for staging server test automation, 384 virtual display setup, 372–374 contracts, implicit, 356 cookies, 282, 304 Cross-Site Request Forgery (CSRF) error, 53 CSS (Cascading Style Sheets) framework, 116, 118 (see also Bootstrap) where Bootstrap won’t work, 126 cutting corners, cdvii D data migrations, 428–431 database deployment issues, 132 database location, 141 De-spiking, 251, 285–293 debugging, 21, 52, 249 Ajax, 249 Django debug screen, 146 improving error messages, 57 in continuous integration, 374–378 in JavaScript, 261 staging for, 306–310 switching DEBUG to false, 151 screenshots, for debugging, 374–378 dependencies and deployment, 132 mocking out, 278 virtualenv, 142 deployment, 411 adjusting database location, 141 automating, 152–155, 157–166 danger areas, 132 444 | dependencies and, 132 deploying to live, 163 further reading, 166 key points, 155 to live, 237 migrate, 147 Nginx, 144–146 overview, 152 production-ready, 148–152 vs. provisioning, 140 sample script, 158–161 saving progress, 156 staging, 237, 427 virtualenvs, 142–144 deployment testing, 131–156 domain name for, 135 manual provisioning for hosting, 136–140 overview, 133 design (see layout and style) Django, 4 admin site, 434 apps, 22 authentication in, 245–248, 282 class-based views, 413–421 (see also class-based views) collectstatic, 127–129 custom user model, 293–297 debugging screen, 146, 151 field types, 64 foreign key relationship, 99 forms in (see forms) FormView, 414 functional tests (FT) in (see functional tests/ testing (FT)) and Gunicorn, 148 LiveServerTestCase, 77 management commands, 311–314, 320 migrations, 62–64, 71–74, 237 model adjustment in, 97 model-layer validation, 175–187 Model-View-Controller (MVC), 24 notifications, 433 Object-Relational Mapper (ORM), 60–64 POST requests (see POST requests) as PythonAnywhere app, 410 startproject, 6 static files in, 122 static live server case, 124 template inheritance, 120–121 Index www.it-ebooks.info templates, 69–71, 90 test class in, 93 test client, 88, 93 test fixtures, 304 TestCase, 23 unit testing in, 23 URLs in, 24–30, 88, 94, 96, 102, 106, 108, 110 validation quirk, 178 view functions in, 24, 89, 94, 105–108, 326 and virtualenvs, 142–144 Django-BrowserID, 243 documentation, tests as, 296 domain names, 135 Don’t Test Constants rule, 40 double-loop TDD, 47, 323 DRY (don’t repeat yourself), 59, 396 duplicates, eliminating, 58, 211–221 E encryption, 436 end-to-end test (see functional tests/testing (FT)) error messages, 435 error pages, 434 evaluating third-party systems, 252 expected failure, 16, 19 explicit waits, 253 exploratory coding, 195, 242 (see also spiking) F Fabric, 167, 314, 426 configuration, 163 installing, 157 sample deployment script, 158–161 Fake XMLHttpRequest, 268 fixtures in functional tests, 303 in JavaScript tests, 229 on staging server, 311–317 foreign key relationship, 99 forms advanced, 211–223 autogeneration, 195 customising form field input, 194 experimenting with, 194 ModelForm, 195 save methods, 208 simple, 193–210 thin views, 210 tips for, 210 using in views, 198–207 validation testing and customising, 196 Fuctional Core, Imperative Shell architecture, 403 functional tests/testing (FT), 5, 398 automation of (see continuous integration (CI)) cleanup, 77–80, 95, 387 de-duplication, 320 defining, 14 for de-spiking, 251 for duplicate items, 211–221 isolation in, 77–80, 112 in JavaScript, 232–234 for layout and style, 115–118, 148, 173 multiple users, 387, 393–394 pros and cons, 364 in provisioning, 139 running unit tests only, 80 safeguards with, 317 splitting, 171 for staging sites, 132, 133 unittest module, 13–19 vs. unit tests, 22, 303 in views, 223 G generator expression, 39 GET requests, 198, 205 get_user, 291 Git repository setup, 8–11 reset --hard, 118 tags, 166, 238 global variables, 228 greedy regular expressions, 106 Gunicorn, 148–155, 165, 307, 425 H headless browsers, 372 helper functions/methods, 59, 172, 175, 206, 226, 350, 390–393 hexagonal architecture, 403 hosting options, 136 Index www.it-ebooks.info | 445 hosting, manual provisioning, 136–140 L Idempotency, 167 implicit waits, 18 in-memory model objects, 352 integrated tests, 351–363, 403 vs. integration test, 342 vs. isolated tests, 362, 398 pros and cons, 364 vs. unit tests, 61 integration tests, 342, 398 integrity errors, 217 isolated tests, 337, 403 (see also test isolation) vs. integrated tests, 362, 398 problems with, 400 pros and cons, 364 layout and style, 115–130 Bootstrap for (see Bootstrap) functional tests (FT) for, 173 large inputs, 125 overview, 130 rows and columns, 122 static files, 122, 127–129 table styling, 126 using a CSS framework for, 118 (see also Bootstrap) using our own CSS in, 126 what to functionally test for, 115 list comprehension, 39 LiveServerTestCase, 77 log messages, 320 logging, 307, 320 logging configuration, 318–320 J M I JavaScript, 225 de-spiking in, 251 debug console, 261 functional test (FT) building in, 232–234 jQuery and Fixtures Div, 229–231 linters, 228 MVC frameworks, 435 onload boilerplate and namespacing, 234 QUnit, 227 running tests in continuous integration, 381–384 spiking with, 242–255 (see also spiking) in TDD Cycle, 234 test runner setup, 226 testing notes, 235 Jenkins Security, 365–384 (see also continuous integration (CI)) adding required plugins, 368 configuring, 367 installing, 365 jQuery, 229–231, 234, 235 JSON fixtures, 304, 320 jumbotron, 125 446 | manage.py, 6, 24, 63, 72, 127 Meta, 196 meta-comments, 86 migrate, 147 migrations, 62–64, 71–74, 99, 237, 238 (see also data migrations) database, 427–431 deleting, 99 testing, 427–431 minimum viable application, 13–16, 81 MockMyID, 252 mocks/mocking callbacks, 272–275 checking call arguments, 267 implicit contracts, 356 in JavaScript, 241, 257–275 initialize function test, 258–264 Internet requests, 285–293 for isolation, 338–341 mock library, 301 Mock side_effects, 339 namespacing, 258 in Outside-In TDD, 331 in Python, 278–284 risks, 354 sinon.js, 265 testing Django login, 284 Index www.it-ebooks.info model-layer validation, 175–187 changes to test, 216 enforcing, 186 errors in View, 178–182 integrity errors, 217 POST requests, 183–187 preventing duplicates, 212 refactoring, 175, 184–186 unit testing, 177–178 at views level, 218 Model-View-Controller (MVC), 24, 435 ModelForm, 195 Mozilla Persona, 242 MVC frameworks, 24, 435 N namespacing, 258 Nginx, 138, 144–146, 149, 165, 424 nonroot user creation, 137 notifications, 433 O Q QuerySet, 61, 214–216 QUnit, 227, 235, 263, 268 ORM (Object-Relational Mapper), 60–64 Outside-In TDD, 323–335 advantages, 323 controller layer, 326 defined, 335 vs.

Send email to index@oreilly.com. 443 www.it-ebooks.info code smell, 59, 65, 193, 301 collectstatic, 127–129 comments, 15, 86 commits, 18, 24, 30, 110 configuration management tools, 167 (see also Fabric) context managers, 177 continuous integration (CI), 365–385, cdvii adding required plugins, 368 best practices, 385 configuring Jenkins, 367 debugging with screenshots, 374–378 installing Jenkins, 365 JavaScript tests, 381–384 project setup, 369 Selenium race conditions, 378–381 for staging server test automation, 384 virtual display setup, 372–374 contracts, implicit, 356 cookies, 282, 304 Cross-Site Request Forgery (CSRF) error, 53 CSS (Cascading Style Sheets) framework, 116, 118 (see also Bootstrap) where Bootstrap won’t work, 126 cutting corners, cdvii D data migrations, 428–431 database deployment issues, 132 database location, 141 De-spiking, 251, 285–293 debugging, 21, 52, 249 Ajax, 249 Django debug screen, 146 improving error messages, 57 in continuous integration, 374–378 in JavaScript, 261 staging for, 306–310 switching DEBUG to false, 151 screenshots, for debugging, 374–378 dependencies and deployment, 132 mocking out, 278 virtualenv, 142 deployment, 411 adjusting database location, 141 automating, 152–155, 157–166 danger areas, 132 444 | dependencies and, 132 deploying to live, 163 further reading, 166 key points, 155 to live, 237 migrate, 147 Nginx, 144–146 overview, 152 production-ready, 148–152 vs. provisioning, 140 sample script, 158–161 saving progress, 156 staging, 237, 427 virtualenvs, 142–144 deployment testing, 131–156 domain name for, 135 manual provisioning for hosting, 136–140 overview, 133 design (see layout and style) Django, 4 admin site, 434 apps, 22 authentication in, 245–248, 282 class-based views, 413–421 (see also class-based views) collectstatic, 127–129 custom user model, 293–297 debugging screen, 146, 151 field types, 64 foreign key relationship, 99 forms in (see forms) FormView, 414 functional tests (FT) in (see functional tests/ testing (FT)) and Gunicorn, 148 LiveServerTestCase, 77 management commands, 311–314, 320 migrations, 62–64, 71–74, 237 model adjustment in, 97 model-layer validation, 175–187 Model-View-Controller (MVC), 24 notifications, 433 Object-Relational Mapper (ORM), 60–64 POST requests (see POST requests) as PythonAnywhere app, 410 startproject, 6 static files in, 122 static live server case, 124 template inheritance, 120–121 Index www.it-ebooks.info templates, 69–71, 90 test class in, 93 test client, 88, 93 test fixtures, 304 TestCase, 23 unit testing in, 23 URLs in, 24–30, 88, 94, 96, 102, 106, 108, 110 validation quirk, 178 view functions in, 24, 89, 94, 105–108, 326 and virtualenvs, 142–144 Django-BrowserID, 243 documentation, tests as, 296 domain names, 135 Don’t Test Constants rule, 40 double-loop TDD, 47, 323 DRY (don’t repeat yourself), 59, 396 duplicates, eliminating, 58, 211–221 E encryption, 436 end-to-end test (see functional tests/testing (FT)) error messages, 435 error pages, 434 evaluating third-party systems, 252 expected failure, 16, 19 explicit waits, 253 exploratory coding, 195, 242 (see also spiking) F Fabric, 167, 314, 426 configuration, 163 installing, 157 sample deployment script, 158–161 Fake XMLHttpRequest, 268 fixtures in functional tests, 303 in JavaScript tests, 229 on staging server, 311–317 foreign key relationship, 99 forms advanced, 211–223 autogeneration, 195 customising form field input, 194 experimenting with, 194 ModelForm, 195 save methods, 208 simple, 193–210 thin views, 210 tips for, 210 using in views, 198–207 validation testing and customising, 196 Fuctional Core, Imperative Shell architecture, 403 functional tests/testing (FT), 5, 398 automation of (see continuous integration (CI)) cleanup, 77–80, 95, 387 de-duplication, 320 defining, 14 for de-spiking, 251 for duplicate items, 211–221 isolation in, 77–80, 112 in JavaScript, 232–234 for layout and style, 115–118, 148, 173 multiple users, 387, 393–394 pros and cons, 364 in provisioning, 139 running unit tests only, 80 safeguards with, 317 splitting, 171 for staging sites, 132, 133 unittest module, 13–19 vs. unit tests, 22, 303 in views, 223 G generator expression, 39 GET requests, 198, 205 get_user, 291 Git repository setup, 8–11 reset --hard, 118 tags, 166, 238 global variables, 228 greedy regular expressions, 106 Gunicorn, 148–155, 165, 307, 425 H headless browsers, 372 helper functions/methods, 59, 172, 175, 206, 226, 350, 390–393 hexagonal architecture, 403 hosting options, 136 Index www.it-ebooks.info | 445 hosting, manual provisioning, 136–140 L Idempotency, 167 implicit waits, 18 in-memory model objects, 352 integrated tests, 351–363, 403 vs. integration test, 342 vs. isolated tests, 362, 398 pros and cons, 364 vs. unit tests, 61 integration tests, 342, 398 integrity errors, 217 isolated tests, 337, 403 (see also test isolation) vs. integrated tests, 362, 398 problems with, 400 pros and cons, 364 layout and style, 115–130 Bootstrap for (see Bootstrap) functional tests (FT) for, 173 large inputs, 125 overview, 130 rows and columns, 122 static files, 122, 127–129 table styling, 126 using a CSS framework for, 118 (see also Bootstrap) using our own CSS in, 126 what to functionally test for, 115 list comprehension, 39 LiveServerTestCase, 77 log messages, 320 logging, 307, 320 logging configuration, 318–320 J M I JavaScript, 225 de-spiking in, 251 debug console, 261 functional test (FT) building in, 232–234 jQuery and Fixtures Div, 229–231 linters, 228 MVC frameworks, 435 onload boilerplate and namespacing, 234 QUnit, 227 running tests in continuous integration, 381–384 spiking with, 242–255 (see also spiking) in TDD Cycle, 234 test runner setup, 226 testing notes, 235 Jenkins Security, 365–384 (see also continuous integration (CI)) adding required plugins, 368 configuring, 367 installing, 365 jQuery, 229–231, 234, 235 JSON fixtures, 304, 320 jumbotron, 125 446 | manage.py, 6, 24, 63, 72, 127 Meta, 196 meta-comments, 86 migrate, 147 migrations, 62–64, 71–74, 99, 237, 238 (see also data migrations) database, 427–431 deleting, 99 testing, 427–431 minimum viable application, 13–16, 81 MockMyID, 252 mocks/mocking callbacks, 272–275 checking call arguments, 267 implicit contracts, 356 in JavaScript, 241, 257–275 initialize function test, 258–264 Internet requests, 285–293 for isolation, 338–341 mock library, 301 Mock side_effects, 339 namespacing, 258 in Outside-In TDD, 331 in Python, 278–284 risks, 354 sinon.js, 265 testing Django login, 284 Index www.it-ebooks.info model-layer validation, 175–187 changes to test, 216 enforcing, 186 errors in View, 178–182 integrity errors, 217 POST requests, 183–187 preventing duplicates, 212 refactoring, 175, 184–186 unit testing, 177–178 at views level, 218 Model-View-Controller (MVC), 24, 435 ModelForm, 195 Mozilla Persona, 242 MVC frameworks, 24, 435 N namespacing, 258 Nginx, 138, 144–146, 149, 165, 424 nonroot user creation, 137 notifications, 433 O Q QuerySet, 61, 214–216 QUnit, 227, 235, 263, 268 ORM (Object-Relational Mapper), 60–64 Outside-In TDD, 323–335 advantages, 323 controller layer, 326 defined, 335 vs.


pages: 265 words: 60,880

The Docker Book by James Turnbull

Airbnb, continuous integration, Debian, DevOps, domain-specific language, false flag, fault tolerance, job automation, Kickstarter, Kubernetes, microservices, MVC pattern, platform as a service, pull request, Ruby on Rails, software as a service, standardized shipping container, web application

This way you can build, replicate, and iterate on production applications, even complex multi-tier applications, in your local environment. Using Docker for continuous integration Up until now, all our testing examples have been very local, single developer-centric examples (i.e., how a local developer might make use of Docker to test a local website or application). Let's look at using Docker's capabilities in a multi-developer continuous integration testing scenario. Docker excels at quickly generating and disposing of one or multiple containers. There's an obvious synergy with Docker's capabilities and the concept of continuous integration testing. Often in a testing scenario you need to install software or deploy multiple hosts frequently, run your tests, and then clean up the hosts to be ready to run again.

., to provide separate containers for web, database, or application layers to better simulate an actual multi-tier production application). Other alternatives One of the more interesting parts of the Docker ecosystem is continuous integration and continuous deployment (CI/CD). Beyond integration with existing tools like Jenkins, we're also seeing people build their own tools and integrations on top of Docker. Drone One of the more promising CI/CD tools being developed on top of Docker is Drone. Drone is a SAAS continuous integration platform that connects to GitHub, Bitbucket, and Google Code repositories written in a wide variety of languages, including Python, Node.js, Ruby, Go, and numerous others.

It runs the test suites of repositories added to it inside a Docker container. Shippable Shippable is a free, hosted continuous integration and deployment service for GitHub and Bitbucket. It is blazing fast and lightweight, and it supports Docker natively. Summary In this chapter, we've seen how to use Docker as a core part of our development and testing workflow. We've looked at developer-centric testing with Docker on a local workstation or virtual machine. We've also explored scaling that testing up to a continuous integration model using Jenkins CI as our tool. We've seen how to use Docker for both point testing and how to build distributed matrix jobs.


pages: 834 words: 180,700

The Architecture of Open Source Applications by Amy Brown, Greg Wilson

8-hour work day, anti-pattern, bioinformatics, business logic, c2.com, cloud computing, cognitive load, collaborative editing, combinatorial explosion, computer vision, continuous integration, Conway's law, create, read, update, delete, David Heinemeier Hansson, Debian, domain-specific language, Donald Knuth, en.wikipedia.org, fault tolerance, finite state, Firefox, Free Software Foundation, friendly fire, functional programming, Guido van Rossum, Ken Thompson, linked data, load shedding, locality of reference, loose coupling, Mars Rover, MITM: man-in-the-middle, MVC pattern, One Laptop per Child (OLPC), peer-to-peer, Perl 6, premature optimization, recommendation engine, revision control, Ruby on Rails, side project, Skype, slashdot, social web, speech recognition, the scientific method, The Wisdom of Crowds, web application, WebSocket

And, because build and test automation is necessary for implementing continuous integration, CI is often a first step towards a continuous deployment framework wherein software updates can be deployed quickly to live systems after testing. Continuous integration is a timely subject, not least because of its prominence in the Agile software methodology. There has been an explosion of open source CI tools in recent years, in and for a variety of languages, implementing a huge range of features in the context of a diverse set of architectural models. The purpose of this chapter is to describe common sets of features implemented in continuous integration systems, discuss the architectural options available, and examine which features may or may not be easy to implement given the choice of architecture.

The process makes use of the CMake family of tools (CMake, CTest, CPack, and CDash). As the code is developed and checked into the version control system, continuous integration testing machines automatically build and test the new CMake code using CTest. The results are sent to a CDash server which notifies developers via email if there are any build errors, compiler warnings, or test failures. The process is a classic continuous integration testing system. As new code is checked into the CMake repository, it is automatically tested on the platforms supported by CMake. Given the large number of compilers and platforms that CMake supports, this type of testing system is essential to the development of a stable build system.

Defined as the ratio of used space in the cluster to total capacity of the cluster. The Architecture of Open Source Applications Amy Brown and Greg Wilson (eds.) ISBN 978-1-257-63801-7 License / Buy / Contribute Chapter 6. Continuous Integration C. Titus Brown and Rosangela Canino-Koning Continuous Integration (CI) systems are systems that build and test software automatically and regularly. Though their primary benefit lies in avoiding long periods between build and test runs, CI systems can also simplify and automate the execution of many otherwise tedious tasks.


pages: 290 words: 119,172

Beginning Backbone.js by James Sugrue

Airbnb, business logic, continuous integration, don't repeat yourself, Firefox, Google Chrome, loose coupling, MVC pattern, node package manager, single page application, web application, Y Combinator

//tests for DOM manipulation module('Fixture Test'); test('Check for paragraph', function(){ var results = fixtureEl.find('#myparagraph').length; console.log(fixtureEl); console.log(results); ok(results === 1, 'Found the correct paragraph'); }); As you can see, it’s pretty straightforward to have JavaScript tests that validate the integrity of your DOM. Recording QUnit Results If you are running your tests on a continuous integration server, as we will look at in the next chapter, you will need to have the capability of recording the results of your tests without the need to refresh the HTML page that runs the tests. While you can roll out your own solution based around the QUnit callbacks, there are plug-ins available that provide the ability to store the results in the most common format across continuous integration systems: JUnit style. As the leading unit test framework in the Java world, JUnits report style became the de facto standard.

Continuing from the previous chapter, you will also see how you can automate the execution of your unit tests with Grunt. An Introduction to Grunt The aim of continuous integration is to improve software quality and reduce the time it takes to deliver productionready applications by ensuring tests are executed as code changes. Of course, many other tasks happen in a continuous process, but running unit tests is considered the most important. Grunt (http://gruntjs.com, see Figure 9-1), created by Ben Alman in 2012, has become the default continuous integration tool for JavaScript web applications. Rather than writing your own custom build scripts to execute from the shell or adopting another languages’ build tools, Grunt scripts are written in JavaScript.

Finally, we just need to add the task to our default list. grunt.registerTask('default', ['jshint', 'uglify', 'cssmin', 'qunit_junit', 'qunit', 'jasmine']); Creating Different Task Sets The additional tasks we have created are not always suitable for all machines. A developer workspace will need to run a different set of tasks to an continuous integration server. This can be controlled quite simply with the registerTask function in Grunt. 198 Chapter 9 ■ Using Grunt for Your Build Process For example, let’s say the developers would run grunt dev on the command line, while the continuous integration server would run grunt build. We’ll create two different task sets for each of these. grunt.registerTask('dev', ['jshint', 'qunit_junit', 'qunit', 'jasmine']); grunt.registerTask('build', ['jshint', 'uglify', 'cssmin', 'qunit_junit', 'qunit', 'jasmine']); grunt.registerTask('default', ['jshint', 'uglify', 'cssmin', 'qunit_junit', 'qunit', 'jasmine']); On developer machines, the static analysis and test tasks are run, while the build machine will also run the minification tasks.


pages: 292 words: 66,588

Learning Vue.js 2: Learn How to Build Amazing and Complex Reactive Web Applications Easily With Vue.js by Olga Filipova

Amazon Web Services, business logic, continuous integration, create, read, update, delete, en.wikipedia.org, Firefox, Google Chrome, leftpad, MVC pattern, pull request, side project, single page application, single source of truth, Skype, source of truth, web application

We tackle it by showing how to write unit tests and how to develop end-to-end tests for the applications in the book. Chapter 8, Deploying – Time to Go Live!, shows how to bring your Vue application to the world, guaranteeing its quality with continuous integration tools. It explains how to connect a GitHub repository to the Travis continuous integration system and to the Heroku cloud deployment platform. Chapter 9, What Is Next, wraps up everything that has been done so far and leaves the reader with the follow up steps. Appendix, Solutions to Exercises, provides solutions to the exercises for first three chapters.

After that, we learned how to apply end-to-end testing techniques using Nightwatch. In this chapter, we will make our applications go live by deploying them to a server and making them available to the world. We will also guarantee continuous integration and continuous deployment of our applications. This means that every time we commit changes performed on the applications, they will automatically be tested and deployed. With this in mind, in this chapter, we are going to do the following: Set up a continuous integration process using Travis Set up a continuous deployment using Heroku Software deployment Before starting to deploy our applications, let's first try to define what it actually means: "Software deployment is all of the activities that make a software system available for use." – Wikipedia: https://en.wikipedia.org/wiki/Software_deployment This definition means that after we perform all the necessary activities, our software will be accessible to the public.

So, our necessary activities to deploy the application will be the following: Create GitHub repositories for the applications and move the applications into the repositories. Set up continuous integration with Travis. Connect applications to Heroku, and set up and configure them in order for Heroku to run them and to expose them to the world. In the next three subsections, I will give a small introduction to GitHub, Travis, and Heroku. What is GitHub? GitHub is a hosting provider for Git-based projects. It can be used at a small, personal scale for individual private and public projects. It can also be used for big corporate projects and all development-related activities, such as code reviews, continuous integration, and so on. Everyone who lives in the world of open source software knows GitHub.


pages: 214 words: 31,751

Software Engineering at Google: Lessons Learned From Programming Over Time by Titus Winters, Tom Manshreck, Hyrum Wright

anti-pattern, computer vision, continuous integration, defense in depth, en.wikipedia.org, functional programming, Jevons paradox, job automation, loss aversion, microservices, reproducible builds, supply-chain attack, transaction costs, Turing complete

Having automation means that tooling can be updated and new changes generated at very low cost, so losing a few cattle now and then isn’t a problem. Testing Each independent shard is tested by running it through TAP, Google’s continuous integration framework. We run every test which depends on the files in a given change transitively, which often creates high load on our continuous integration system. This may sound computationally expensive, but in practice, the vast majority of shards affect fewer than one thousand tests, out of the millions across our codebase. For those which impact more, we can group them together: first running the union of all affected tests for all shards, and then for each individual shard running just the intersection of its affected tests with those that failed the first run.

Or, better still, what sorts of policies can we put in place that provide super-linear value as the organization grows? One of our favorite internal policies is a great enabler of infrastructure teams, protecting their ability to make infrastructure changes safely. “If a product experiences outages or other problems as a result of infrastructure changes, but the issue wasn’t surfaced by tests in our Continuous Integration (CI) system, it is not the fault of the infrastructure change.” More colloquially, this is phrased as “If you liked it, you should have put a CI test on it”, which we call “The Beyoncé Rule15. From a scaling perspective, the Beyoncé Rule implies that complicated one-off bespoke tests that aren’t triggered by our common CI system do not count.

This scalability is the holy grail we’ve been working towards by defining an artifact-based build system. Remote Caching The simplest type of distributed build is one that only leverages remote caching. The system is shown in Figure 18-2: Figure 2-2. A distributed build showing remote caching Every system that performs builds, including both developer workstations and continuous integration systems, shares a reference to a common remote cache service. This service might be a fast and local short-term storage system like Redis or a cloud service like Google Cloud Storage. Whenever a user needs to build an artifact, whether directly or as a dependency, the system first checks with the remote cache to see if that artifact already exists there.


pages: 203 words: 14,242

Ship It!: A Practical Guide to Successful Software Projects by Jared R. Richardson, William A. Gwaltney

continuous integration, David Heinemeier Hansson, Donald Knuth, index card, MVC pattern, no silver bullet, place-making, Ruby on Rails, web application

Once you can build your product automatically, how often should you do so? Ideally, you will rebuild every time the code changes. That way you’ll know immediately if any change broke your build. Add a lightweight set of smoke tests to this system, and you also get a basic level of functional insurance as well. This type of system is called Continuous Integration.4 A Continuous Integration (or CI) tool sits on a clean, nondeveloper box (the build machine) and rebuilds your project every time someone commits code. Rebuilding each time code is committed keeps your code base clean by catching compile errors as soon as they occur. It also runs your test suites to catch functional errors.

It also catches the new files that you forgot to add or the existing files you modified. Automated build systems are great at catching the details that we humans are so good at missing. TIP 6 Build continuously 4 5 CruiseControl is an open-source Continuous Integration system hosted at Source- Forge.net. smoke tests Continuous Integration 30 B UILD A UTOMATICALLY Figure 2.4: CruiseControl status report web page You can also move beyond “Does it compile?” and ask “Does it run?” With a well-selected test suite, basic functionality is retested and bugs are not allowed to be reintroduced (preventing bug regression).

We’re Never Done . . . . . . . . . . . . . . . 128 129 131 132 133 135 136 138 139 141 144 146 147 150 153 155 156 158 159 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A Source Code Management 162 B Build Scripting Tools 165 C Continuous Integration Systems 169 D Issue Tracking Software 172 E Development Methodologies 175 F Testing Frameworks 178 G Suggested Reading List 181 G.1 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . 184 Index 186 vi Dogma does not mean the absence of thought, but the end of thought.


pages: 351 words: 123,876

Beautiful Testing: Leading Professionals Reveal How They Improve Software (Theory in Practice) by Adam Goucher, Tim Riley

Albert Einstein, barriers to entry, Black Swan, business logic, call centre, continuous integration, Debian, Donald Knuth, en.wikipedia.org, Firefox, Grace Hopper, index card, Isaac Newton, natural language processing, off-by-one error, p-value, performance metric, revision control, six sigma, software as a service, software patent, SQL injection, the scientific method, Therac-25, Valgrind, web application

Finally, it’s critical that test code is under source control and builds in a central location. Source control eases maintenance by enabling testers to investigate an entire history of changes to test code, overlaying infrastructure or “collateral” (noncode files used by the tests, such as media files or input data). Frequent builds or continuous integration are just as applicable to test code as they are to product code. Central builds allow the latest tests to run on every build of the production code, and ensure that tests are available from a central location. If testers build their own tests and are responsible for copying the tests to a server, mistakes are eventually bound 106 CHAPTER EIGHT to occur.

As with any large-scale change, changing and growing a little at a time is the best approach to turn an inefficient system into a beautiful one. Start by writing better tests; poor test automation is one of the biggest obstacles to automation success. Also make sure that test scripts and code are part of a source control system, and that tests go through some sort of continuous integration process to ensure some level of test quality and consistency. 116 CHAPTER EIGHT Then, set up a test lab for running automated tests. Start working out a way to distribute tests to these machines and execute the tests. Then, begin gathering and aggregating simple highlevel test results.

In this chapter we discuss some techniques that are associated with efficient testing methodologies. Testing activity encompasses all aspects of a software development life cycle, be it one of the traditional waterfall, incremental, or spiral models, or the modern agile and test-driven development (TDD) models. One of the recent project management trends is to set up continuous integration frameworks where changes are frequently integrated into the source code, the product is built, and a large part of existing tests are run to validate the correctness of code change. Although this concept was akin to the exhaustive testing methodology of the old school that was deemed as very difficult, recent advances in hardware configurations and reduced cost make this approach financially feasible.


pages: 328 words: 77,877

API Marketplace Engineering: Design, Build, and Run a Platform for External Developers by Rennay Dorasamy

Airbnb, Amazon Web Services, barriers to entry, business logic, business process, butterfly effect, continuous integration, DevOps, digital divide, disintermediation, fault tolerance, if you build it, they will come, information security, Infrastructure as a Service, Internet of things, Jeff Bezos, Kanban, Kubernetes, Lyft, market fragmentation, microservices, minimum viable product, MITM: man-in-the-middle, mobile money, optical character recognition, platform as a service, pull request, ride hailing / ride sharing, speech recognition, the payments system, transaction costs, two-pizza team, Uber and Lyft, uber lyft, underbanked, web application

Tap Debate Lifecycle Developer Experience Versioning End of Life Design Guidelines Error Handling Filtering and Pagination Software Development Kit (SDK) Summary Chapter 8:​ API Development API First, Tech Second Team Structure Delivery Lead(s) Developers Quality Assurance Delivery Approach Defining the Strategy Planning Squads Agile Methodology DevOps Continuous Integration (CI) Continuous Delivery (CD) Microservices Application Development Development Guidance API Gateway Microservices Platform Services Portal Applications Summary Chapter 9:​ Sandbox Purpose Process Sandbox Strategies beta Backend Simulation Shallow Semi-Live QA Live Third-Party Sandbox Access Building a Virtualiser Requirements Implementation Options Design Philosophy Sample Flow Summary Chapter 10:​ API Operations The Operational Universe Change and Release Management DevOps Practice Logging The ELK Stack Monitoring Environment Monitoring Application Performance Monitoring Functional Monitoring Telemetry Alerting Support Roles Transition Communication Strategy Process Issue Tracking and Reporting Service-Level Agreements Supporting Systems Platform As a Service Strategy Backend Dependencies Summary Chapter 11:​ Conclusion Team Dynamic Organizational Support Agility to Pivot Developer Centricity Where to Start Keep in Touch About the Author Rennay Dorasamy has spent the last 20 years in various technology roles, ranging from architecture to development to operations, across a number of industries.

With that being said, I retain the claim for “weaponizing” the technology and taking it from a R&D construct to an operational environment. The Power of Port-Forward As our Business Logic components could leverage several integration microservices to fulfill a specific function, it became increasingly more challenging to assemble the solution. DevOps Continuous Integration (CI) pipelines can be leveraged for code deployments to the server. CI processes are great for transitioning between environments, but for initial development tasks, I consider using a CI pipeline like having a conversation with someone on a remote planet. You say something, wait some time until the message is received and has been replied to before you continue.

Automated testing will undoubtedly save a considerable amount of time, especially for regression testing activities. In a frenetic environment, with multiple testing efforts underway, automated testing efforts are generally delayed. Automated testing must become part of the team culture as its execution is a core element of the Continuous Integration (CI) /Continuous Delivery (CD) process. This is the primary reason why organizations such as Netflix can release often to production. Delivery Approach Our Marketplace came into existence at a time when the organization had made a conscious decision to shift to a more agile delivery approach.


pages: 372 words: 67,140

Jenkins Continuous Integration Cookbook by Alan Berg

anti-pattern, continuous integration, Debian, don't repeat yourself, en.wikipedia.org, Firefox, information security, job automation, One Laptop per Child (OLPC), performance metric, revision control, web application

Jenkins Continuous Integration Cookbook * * * Jenkins Continuous Integration Cookbook Copyright © 2012 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied.

Preface Jenkins is a Java-based Continuous Integration (CI) server that supports the discovery of defects early in the software cycle. Thanks to over 400 plugins, Jenkins communicates with many types of systems, building and triggering a wide variety of tests. CI involves making small changes to software, and then building and applying quality assurance processes. Defects do not only occur in the code but also appear in the naming conventions, documentation, how the software is designed, build scripts, the process of deploying the software to servers, and so on. Continuous integration forces the defects to emerge early, rather than waiting for software to be fully produced.

There is also a highly stable long-term support release for the more conservative. Hence, there is a rapid pace of improvement. Jenkins pushes up code quality by automatically testing within a short period after code commit, and then shouting loudly if build failure occurs. Jenkins is not just a continual integration server but also a vibrant and highly active community. Enlightened self-interest dictates participation. There are a number of ways to do this: Participate on the Mailing lists and Twitter (https://wiki.jenkins-ci.org/display/JENKINS/Mailing+Lists). First, read the postings, and as you get to understand what is needed, participate in the discussions.


Seeking SRE: Conversations About Running Production Systems at Scale by David N. Blank-Edelman

Affordable Care Act / Obamacare, algorithmic trading, AlphaGo, Amazon Web Services, backpropagation, Black Lives Matter, Bletchley Park, bounce rate, business continuity plan, business logic, business process, cloud computing, cognitive bias, cognitive dissonance, cognitive load, commoditize, continuous integration, Conway's law, crowdsourcing, dark matter, data science, database schema, Debian, deep learning, DeepMind, defense in depth, DevOps, digital rights, domain-specific language, emotional labour, en.wikipedia.org, exponential backoff, fail fast, fallacies of distributed computing, fault tolerance, fear of failure, friendly fire, game design, Grace Hopper, imposter syndrome, information retrieval, Infrastructure as a Service, Internet of things, invisible hand, iterative process, Kaizen: continuous improvement, Kanban, Kubernetes, loose coupling, Lyft, machine readable, Marc Andreessen, Maslow's hierarchy, microaggression, microservices, minimum viable product, MVC pattern, performance metric, platform as a service, pull request, RAND corporation, remote working, Richard Feynman, risk tolerance, Ruby on Rails, Salesforce, scientific management, search engine result page, self-driving car, sentiment analysis, Silicon Valley, single page application, Snapchat, software as a service, software is eating the world, source of truth, systems thinking, the long tail, the scientific method, Toyota Production System, traumatic brain injury, value engineering, vertical integration, web application, WebSocket, zero day

., Context Versus Control in SRE-Context Versus Control in SRE continuous delivery/deploymentchampioning, Championing CD database reliability engineering and, Continuous Delivery: From Development to Production-Tools deployment, Deployment-Championing CD immutable infrastructure and, Continuous Integration/Continuous Deployment with Confidence third-party integrations and, Testing and staging continuous integrationimmutable infrastructure and, Continuous Integration/Continuous Deployment with Confidence third-party integrations and, Testing and staging contract termination, Decommissioning control plane, Configuration Management (Control Plane Versus Data Plane) convolutional neural networks, How and When Should We Apply Neural Networks?

., Where Percentiles Fall Down (and Histograms Step Up) SLOs and, Histograms-Where Percentiles Fall Down (and Histograms Step Up) Hopper, Grace, Approaching Operations as an Engineering Problem Horowitz, Jonah, Immutable Infrastructure and SRE, Immutable Infrastructure and SRE-Disadvantages human error, Antipattern 4: Root Cause = Human Error-Antipattern 4: Root Cause = Human Error human resources, The Primary Indicator of a Successful Team-Operations teams are bad at estimating their level of psychological safety human-factors research, Beyond Burnout Humble, Jez, SRE Patterns Loved by DevOps, SRE Patterns Loved by DevOps People Everywhere-Conclusion I immutable infrastructure, Immutable Infrastructure and SRE-Disadvantagesbase image construction, Building the Base Image continuous integration/continuous deployment with confidence, Continuous Integration/Continuous Deployment with Confidence defined, Scalability, Reliability, and Performance deploying applications, Deploying Applications disadvantages of, Disadvantages failure recovery, Failure Recovery faster startup times, Faster Startup Times known state, Known State multiregion operations, Multiregion Operations release engineering, Release Engineering scalability, reliability, and performance, Scalability, Reliability, and Performance security, Security simplicity, Simpler Operations impact analysis, Impact Analysis impact monitoring, Monitoring incentive structurefor SRE buy-in, Getting Buy-In novelty priority inversion and, Novelty Priority Inversion incident analysis, What Can You Do?

Playbook: From Staging to Production With many of the services we run, we have to adhere to some sort of playbook — a means for running our service in production. This involves a detailed explanation of what the service is intended to do, how to test it, how to deploy it, and how to deal with it should things go sideways. Testing and staging Quarterly releases are ancient history. These days, Continuous Integration (CI) and Continuous Deployment (CD) are standard. We’ve become accustomed to maintaining a deployment pipeline similar to one that is triggered by committing code to master and allowing our automation to run a suite of unit tests, followed by deployment to a staging environment, followed by Selenium testing, followed by canary deployment, and finally production deployment.


pages: 1,136 words: 73,489

Working in Public: The Making and Maintenance of Open Source Software by Nadia Eghbal

Amazon Web Services, Apollo 11, barriers to entry, Benevolent Dictator For Life (BDFL), Big Tech, bitcoin, Clayton Christensen, cloud computing, commoditize, commons-based peer production, context collapse, continuous integration, crowdsourcing, cryptocurrency, David Heinemeier Hansson, death of newspapers, Debian, disruptive innovation, Dunbar number, en.wikipedia.org, eternal september, Ethereum, Firefox, Free Software Foundation, Guido van Rossum, Hacker Ethic, Hacker News, Induced demand, informal economy, information security, Jane Jacobs, Jean Tirole, Kevin Kelly, Kickstarter, Kubernetes, leftpad, Mark Zuckerberg, Menlo Park, Neal Stephenson, Network effects, node package manager, Norbert Wiener, pirate software, pull request, RFC: Request For Comment, Richard Stallman, Ronald Coase, Ruby on Rails, side project, Silicon Valley, Snapchat, social graph, software as a service, Steve Jobs, Steve Wozniak, Steven Levy, Stewart Brand, tacit knowledge, the Cathedral and the Bazaar, The Death and Life of Great American Cities, The Nature of the Firm, TikTok, Tragedy of the Commons, transaction costs, two-sided market, urban planning, web application, wikimedia commons, Yochai Benkler, Zimmermann PGP

However, there are still a number of examples to look at. Continuous integration and automated tests Continuous integration is the practice of deploying smaller, but more frequent, changes to production, which helps developers iterate more rapidly on software. Deploying changes more frequently requires more coordination among developers, because multiple people are changing the same codebase. A number of automated services have sprung up to manage this pipeline, including Travis CI and CircleCI. Even GitHub has started offering its own continuous integration services through GitHub Actions, a system for building custom automated software workflows.

For example, the Python-based web application framework Django has a security team, an ops team to maintain tooling and infrastructure, a release team to build and manage releases, and several technical teams to handle triage, contributions, and code review.141 All of these developers could be considered maintainers, but because Django is such a big project their work is more specialized. In smaller projects, maintainers are more likely to be handling more aspects of the project themselves. They’re the ones who respond to new issues and keep them organized; answer user support questions; maintain tests, style guides, and continuous integration; and write documentation. For example, Caddy, an open source web server, is mostly maintained by its author, Matt Holt, who reviews pull requests, responds to issues, and is the primary core developer.142 Sometimes, the term “maintainer” is used interchangeably with “core developer,” which raises another question: Do maintainers write new code, or simply tend to the existing code?

Since open source developers tend toward work that they find intrinsically motivating, not only are open source projects more susceptible to technical debt but there are also fewer incentives to clean up messy code. One of the biggest pain points for open source projects is managing test infrastructure. Software is frequently developed today using continuous integration (CI), wherein smaller code revisions are regularly merged into the master, rather than merging larger revisions less frequently. To reduce the chance of introducing breaking changes, developers use automated tests to detect whether new additions are “safe” to merge (also known as “make the merge button green” on GitHub).


pages: 226 words: 17,533

Programming Scala: tackle multicore complexity on the JVM by Venkat Subramaniam

augmented reality, business logic, continuous integration, domain-specific language, don't repeat yourself, functional programming, higher-order functions, loose coupling, semantic web, type inference, web application

This can be very useful for logging results and processing them during continuous integration.3 12.5 Asserts ScalaTest provides a simple assert( ) method.4 It checks whether the expression in the parameter evaluates to true.5 If the expression evaluates to true, the assert( ) method returns silently. Otherwise, it throws an AssertionError. Here is an example of assertion failure: Download UnitTestingWithScala/AssertionFailureExample.scala class AssertionFailureExample extends org.scalatest.Suite { def testAssertFailure() { assert(2 == List().size) } } (new AssertionFailureExample).execute() 3. See “Continuous Integration” in Appendix A, on page 211, and Mike Clark’s Pragmatic Project Automation [Cla04] and Continuous Integration [DMG07] by Duvall et al. 4.

Canary Test. . . . . . http://memeagora.blogspot.com/2007/06/coalmine-canary-tests.html In this blog, Neal Ford discusses canary tests and the advantage of starting out small and simple. Command Query Separation. . . . . . http://www.martinfowler.com/bliki/CommandQuerySeparation.html In this blog, Martin Fowler discusses the term command query separation. Continuous Integration. . . . . . http://martinfowler.com/articles/continuousIntegration.html In this article, Martin Fowler discusses the practice of continuous integration. Discussion Forum for This Book . . . . . . http://forums.pragprog.com/forums/87 This is the discussion forum for this book where readers share their opinions, ask questions, respond to questions, and interact with each other.

See “Continuous Integration” in Appendix A, on page 211, and Mike Clark’s Pragmatic Project Automation [Cla04] and Continuous Integration [DMG07] by Duvall et al. 4. You can also import and use JUnit, TestNG, or Hamcrest matchers methods like assertEquals( ) and assertThat( ). Be sure to include the appropriate JAR files. 5. A variation of assert( ) checks whether the parameter evaluates to None. Download at Boykma.Com Report erratum Prepared exclusively for sam kaplan this copy is (P1.0 printing, June 2009) 172 A SSER TS When we run the previous test, we get an error message that includes the following information: Test Starting - Main$$anon$1$AssertionFailureExample.testAssertFailure TEST FAILED - Main$$anon$1$AssertionFailureExample.testAssertFailure ((virtual file):7) org.scalatest.TestFailedException: ...


pages: 266 words: 79,297

Forge Your Future with Open Source by VM (Vicky) Brasseur

AGPL, anti-pattern, Benevolent Dictator For Life (BDFL), call centre, continuous integration, Contributor License Agreement, Debian, DevOps, don't repeat yourself, en.wikipedia.org, Firefox, FOSDEM, Free Software Foundation, Guido van Rossum, information security, Internet Archive, Larry Wall, microservices, Perl 6, premature optimization, pull request, Richard Stallman, risk tolerance, Turing machine

Not so with free and open source software. Because FOSS is constantly moving, evolving, and innovating, many of the current industry best practices either originated in free and open source software development or were perfected by it. Version control, feature branches, unit and integration tests, continuous integration and deployment (CI/CD), design patterns… When you contribute to FOSS, you master many concepts and best practices that you may never get to learn in another environment. More importantly, because you’re hands on with these concepts, you have the opportunity to learn not only how they work, but also why they’re important to do at all, and learn first hand the difference they make to a successful software project.

If your contribution is documentation or some other type, test how your change will appear in the official documentation repository, website, or wherever it may appear. No matter what, don’t assume it’s right. Even if it’s just a small change, take the time to confirm not only that your change is correct, but also that you haven’t accidentally jostled something else on your way. Many projects have a continuous integration and deployment (CI/CD) service, such as Travis[81] or CircleCI.[82] This service runs all unit, integration, linter, and other tests on all submissions to confirm they meet project standards. If your selected project uses such a service, always pay attention to its results. It is, by the way, completely OK if your contribution causes CI/CD to fail (break the build).

While it may seem like these tips are only for people earlier in their programming career, nothing could be further from the truth. What follows are best practices for code reviews by people of any experience level. Whether you’re a neophyte or a master, these tips can help you spot potential problems in any code review. Does the code even pass the build? Does the project use continuous integration/continuous deployment (CI/CD) or otherwise have its test suite run automatically? If the test suite doesn’t pass after the code contribution… You’re a smart person. I don’t need to tell you that this is a big red flag that something may be wrong with that code. Politely ask the contributor to study the build/test errors and correct them before you continue to invest your time in reviewing the contribution.


pages: 270 words: 75,626

User Stories Applied: For Agile Software Development by Mike Cohn

A Pattern Language, c2.com, call centre, continuous integration, do well by doing good, Great Leap Forward, index card, iterative process, job automation, job satisfaction, phenotype, tacit knowledge, web application

For example, the practice of refactoring is made easier by the practices of pair programming, simple design, collective ownership, continuous integration, and testing. The twelve practices are not a random collection of good ideas from which an XP team is allowed to pick their favorites. After becoming experienced with XP, you may choose to drop or alter a practice but you really should delay customizing XP until becoming experienced with standard XP. In this appendix we will consider the following twelve XP practices: small releases the planning game refactoring testing pair programming sustainable pace team code ownership coding standard simple design metaphor continuous integration on-site customer Small Releases XP projects progress in a series of iterations, each of which is each typically one to three weeks long.

* * * TITLES IN THE SERIES Implementation Patterns Kent Beck, ISBN 0321413091 Test-Driven Development: By Example Kent Beck, ISBN: 0321146530 User Stories Applied: For Agile Software Development Mike Cohn, ISBN: 0321205685 Implementing Lean Software Development: From Concept to Cash Mary and Tom Poppendieck, ISBN 0321437381 * * * Refactoring Databases: Evolutionary Database Design Scott W. Ambler and Pramodkumar J. Sadalage, ISBN 0321293533 Continuous Integration: Improving Software Quality and Reducing Risk Paul M. Duvall, with Steve Matyas and Andrew Glover, 0321336380 Patterns of Enterprise Application Architecture Martin Fowler, ISBN: 0321127420 Refactoring HTML: Improving the Design of Existing Web Applications Elliotte Rusty Harold, ISBN 0321503635 Beyond Software Architecture: Creating and Sustaining Winning Solutions Luke Hohmann, ISBN: 0201775948 Enterprise Integration Patterns: Desinging, Building, and Deploying Messaging Solutions Gregor Hohpe and Bobby Woolf, ISBN: 0321200683 Refactoring to Patterns Joshua Kerievsky, ISBN 0321213351 User Stories Applied For Agile Software Development Mike Cohn Boston • San Francisco • New York • Toronto • Montreal London • Munich • Paris • Madrid Capetown • Sydney • Tokyo • Singapore • Mexico City Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks.

For example, on one project our metaphor was that the system was like a chalkboard and that various parts of the system could write on the chalkboard. When the user was done with the system she would either save the contents of the chalkboard or erase them. This greatly simplified way of thinking about the system helped us by giving us a convenient, simple way of thinking about the behavior of the system. Continuous Integration I got involved in a discussion recently with an executive at one of the largest eCommerce companies. He told that me that integrating the work of multiple developers was the largest problem for most software development teams. He liked to have his teams integrate their software once a month so they could avoid the bigger problems of integrating less frequently.


pages: 757 words: 193,541

The Practice of Cloud System Administration: DevOps and SRE Practices for Web Services, Volume 2 by Thomas A. Limoncelli, Strata R. Chalup, Christina J. Hogan

active measures, Amazon Web Services, anti-pattern, barriers to entry, business process, cloud computing, commoditize, continuous integration, correlation coefficient, database schema, Debian, defense in depth, delayed gratification, DevOps, domain-specific language, en.wikipedia.org, fault tolerance, finite state, Firefox, functional programming, Google Glasses, information asymmetry, Infrastructure as a Service, intermodal, Internet of things, job automation, job satisfaction, Ken Thompson, Kickstarter, level 1 cache, load shedding, longitudinal study, loose coupling, machine readable, Malcom McLean invented shipping containers, Marc Andreessen, place-making, platform as a service, premature optimization, recommendation engine, revision control, risk tolerance, Salesforce, scientific management, seminal paper, side project, Silicon Valley, software as a service, sorting algorithm, standardized shipping container, statistical model, Steven Levy, supply-chain management, systems thinking, The future is already here, Toyota Production System, vertical integration, web application, Yogi Berra

It has the ability to monitor an RSS feed and announce any new entries in a given room. The SRE chat room monitors an RSS feed of build completions. Every time a build completes, there is an announcement of what was built and whether it was successful, plus a link to the status page. This way the entire team has visibility to their builds. * * * 9.5 Continuous Integration Continuous integration (CI) is the practice of doing the build phase many times a day in an automated fashion. Each run of the build phase is triggered by some event, usually a code commit. All the build-phase steps then run in a fully automated fashion. All builds are done from the main trunk of the source code repository.

With each build, the testing environment is created, the automated testing runs, and the release is “delivered,” ready to be considered for use in other environments. This doesn’t mean every change is deployed to production, but rather that every change is proven to be deployable at any time. CD has similar benefits as continuous integration. In fact, it can be considered an extension to CI. CD makes it economical and low risk to work in small batches, so that problems are found sooner and, therefore, are easier to fix. (See Section 8.2.4.) CD incorporates all of continuous integration, plus system tests, performance tests, user acceptance tests, and all other automated tests. There’s really no excuse not to adopt CD once testing is automated. If some tests are not automated, CD can deliver the release to a beta environment used for manual testing. 10.6 Infrastructure as Code Recall in Figure 9.1 that the service delivery platform (SDP) pattern flow has quadrants that represent infrastructure as well as applications.

We are optimistic that someday we will be able to do so, too. 11.10 Continuous Deployment Continuous deployment means every release that passes tests is deployed to production automatically. Continuous deployment should be the goal of most companies unless constrained by external regulatory or other factors. As depicted in Figure 11.1, this requires continuous integration and continuous delivery, plus automating any other testing, approval, and code push processes. Figure 11.1: Continuous integration, delivery, and deployment build on each other. Recall that continuous delivery results in packages that are production ready, but whether to actually deploy each package into production is a business decision. Continuous deployment is a business decision to automate this approval and always deploy approved releases.


pages: 629 words: 109,663

Docker in Action by Jeff Nickoloff, Stephen Kuenzli

air gap, Amazon Web Services, cloud computing, computer vision, continuous integration, database schema, Debian, end-to-end encryption, exponential backoff, fail fast, failed state, information security, Kubernetes, microservices, MITM: man-in-the-middle, peer-to-peer, software as a service, web application

Goals of an image build pipeline In this context, pipelines automate the process for building, testing, and publishing artifacts so they can be deployed to a runtime environment. Figure 10.1 illustrates the high-level process for building software or other artifacts in a pipeline. This process should be familiar to anyone using continuous integration (CI) practices and is not specific to Docker images. Figure 10.1. Generic artifact build pipeline People often automate build pipelines with continuous integration systems such as Jenkins, Travis CI, or Drone. Regardless of the specific pipeline-modeling technology, the goal of a build pipeline is to apply a consistent set of rigorous practices in creating deployable artifacts from source definitions.

It does so by providing a complete vision for process containers and simple tooling for building and working with them. If you’re on a team that operates service software with dynamic scaling requirements, deploying software with Docker can help reduce customer impact. Containers come up more quickly and consume fewer resources than virtual machines. Teams that use continuous integration and continuous deployment techniques can build more expressive pipelines and create more robust functional testing environments if they use Docker. The containers being tested hold the same software that will go to production. The results are higher production change confidence, tighter production change control, and faster iteration.

Maintaining multiple versions of an image is as simple as maintaining multiple Dockerfiles. The Dockerfile build process itself uses extensive caching to aid rapid development and iteration. The builds are traceable and reproducible. They integrate easily with existing build systems and many continuous integration tools. With all these reasons to prefer Dockerfile builds to handmade images, it’s important to learn how to write them. The examples in this section cover the core Dockerfile instructions used in most images. The following sections show how to create downstream behavior and more maintainable Dockerfiles.


Team Topologies: Organizing Business and Technology Teams for Fast Flow by Matthew Skelton, Manuel Pais

anti-pattern, business logic, business process, call centre, cognitive load, continuous integration, Conway's law, database schema, DevOps, different worldview, Dunbar number, holacracy, information security, Infrastructure as a Service, Internet of things, Jeff Bezos, Kanban, Kickstarter, knowledge worker, Kubernetes, Lean Startup, loose coupling, meta-analysis, microservices, Norbert Wiener, operational security, platform as a service, pull request, remote working, systems thinking, two-pizza team, web application

Miguel Antunes, R&D Principle Software Engineer at OutSystems, a low-code platform vendor, relayed an example of this very challenge. Their Engineering Productivity team at OutSystems was five years old. The team’s mission was to help product teams run their builds efficiently, maintain infrastructure, and improve test execution. The team kept growing and took on extra responsibilities around continuous integration (CI), continuous delivery (CD), and infrastructure automation. Victims of their own success, sprint planning for the now eight-person-strong team was a mix and match of requests across their stack of responsibilities. Prioritization was hard, and the frequent context switching even throughout a single sprint led to a dip in people’s motivation.

While there is no formula for cognitive load, we can assess the number and relative complexity (internal to the organization) of domains for which a given team is responsible. The Engineering Productivity team at OutSystems that we mentioned in Chapter 1 realized that the different domains they were responsible for (build and continuous integration, continuous delivery, test automation, and infrastructure automation) had caused them to become overloaded. The team was constantly faced with too much work and context switching prevailed, with tasks coming in from different product areas simultaneously. There was a general sense in the team that they lacked sufficient domain knowledge, but they had no time to invest in acquiring it.

It’s common to find that one or more database-administration (DBA) teams were put in place to not only maintain the database but also coordinate changes to the database—a task they are often understaffed for—and they become a large bottleneck to delivery. Monolithic Builds (Rebuild Everything) A monolithic build uses one gigantic continuous-integration (CI) build to get a new version of a component. Application monoliths lead to monolithic builds, but even with smaller services, it’s possible that the build scripts set out to build the entire codebase instead of using standard dependency-management mechanisms between components (such as packages or containers).


Scala in Action by Nilanjan Raychaudhuri

business logic, continuous integration, create, read, update, delete, database schema, domain-specific language, don't repeat yourself, duck typing, en.wikipedia.org, failed state, fault tolerance, functional programming, general-purpose programming language, higher-order functions, index card, Kanban, MVC pattern, type inference, web application

The next section discusses setting up a continuous integration[9] environment to get continuous benefits from them. 9 Martin Fowler, “Continuous Integration,” May 1, 2006, http://martinfowler.com/articles/continuousIntegration.html. 10.3.1. Setting up your environment for TDD Once you and your team get comfortable with TDD, you need a tool that checks out the latest code from your source code repository and runs all the tests after every check-in of the source control system. This ensures that you always have a working software application. A continuous integration (CI) tool does that automatically for you.

The goal in chapter 10 is to make you comfortable writing automated tests in Scala so that you can build production-quality software. The path to writing well-crafted code is the path where you write tests for your code. Another goal is to dispel the common perception that writing tests is hard. Your first steps will be getting started with practices like test-driven development (TDD) and continuous integration for your Scala project. Chapter 6. Building web applications in functional style This chapter covers Building Scala projects with SBT (Simple Build Tool) Introduction to the Scalaz HTTP module Creating a web application in Scala called weKanban This second part of the book switches focus to more real-world applications of the Scala programming language, and what could be more practical than building a web application in Scala?

My goal for this chapter is to make you comfortable writing automated tests in Scala so that you can build production-quality software. The path to writing well-crafted code[1] is the path where you write tests for your code. The common perception about writing tests is that it’s hard, but this chapter will change that mindset. I’m going to show you how you can get started with practices like test-driven development and continuous integration for your Scala project. The idea of test-driven development (TDD) is to write the test before you write code. I know this seems backward, but I promise you that by the end of this chapter it will make sense. You’ll learn that writing tests is more like doing a design exercise than testing, and it makes sense to design your software.


pages: 313 words: 75,583

Ansible for DevOps: Server and Configuration Management for Humans by Jeff Geerling

Abraham Maslow, AGPL, Amazon Web Services, cloud computing, continuous integration, database schema, Debian, defense in depth, DevOps, fault tolerance, Firefox, full text search, Google Chrome, inventory management, loose coupling, microservices, Minecraft, MITM: man-in-the-middle, punch-card reader, Ruby on Rails, web application

Many of the developers and sysadmins I work with are at least moderately comfortable administering a Linux server via SSH, and manage between 1-100 servers. Some of these people have a little experience with configuration management tools (usually with Puppet or Chef), and maybe a little experience with deployments and continuous integration using tools like Jenkins, Capistrano, or Fabric. I am writing this book for these friends who, I think, are representative of most people who have heard of and/or are beginning to use Ansible. If you are interested in both development and operations, and have at least a passing familiarity with managing a server via the command line, this book should provide you with an intermediate- to expert-level understanding of Ansible and how you can use it to manage your infrastructure.

To update all the Server Check.in servers, all that was needed was: $ ansible centos -m yum -a "name=bash state=latest" You could even go further and create a small playbook that would patch the vulnerability, then run tests to make sure the vulnerability was no longer present, as illustrated in this playbook. This would also allow you to run the playbook in check mode or run it through a continuous integration system to verify the fix works in a non-prod environment. This infrastructure inventory is also nice in that you could create a top-level playbook that runs certain roles or tasks against all your infrastructure, others against all servers of a certain Linux flavor, and another against all servers in your entire infrastructure.

Consider Etsy, a company whose engineers are deploying code to production up to 40 times per day, with no manual intervention from the operations team. The operations team is free to work on more creative endeavors, and the developers see their code go live in near-real-time! Etsy’s production deployment schedule is enabled by a strong DevOps-oriented culture (with robust code repository management, continuous integration, well-tested code, feature flags, etc.). While it may not be immediately possible to start deploying your application to production 20 times a day, you can move a long way towards effortless deployments by automating deployments with Ansible. Deployment strategies There are dozens of ways to deploy code to servers.


pages: 1,758 words: 342,766

Code Complete (Developer Best Practices) by Steve McConnell

Ada Lovelace, Albert Einstein, Buckminster Fuller, business logic, call centre, classic study, continuous integration, data acquisition, database schema, don't repeat yourself, Donald Knuth, fault tolerance, General Magic , global macro, Grace Hopper, haute cuisine, if you see hoof prints, think horses—not zebras, index card, inventory management, iterative process, Larry Wall, loose coupling, Menlo Park, no silver bullet, off-by-one error, Perl 6, place-making, premature optimization, revision control, Sapir-Whorf hypothesis, seminal paper, slashdot, sorting algorithm, SQL injection, statistical model, Tacoma Narrows Bridge, the Cathedral and the Bazaar, the scientific method, Thomas Kuhn: the structure of scientific revolutions, Turing machine, web application

., India, Japan, and Europe found that only 20–25 percent of projects used daily builds at either the beginning or middle of their projects (Cusumano et al. 2003), so this represents a significant opportunity for improvement. Continuous Integration Some software writers have taken daily builds as a jumping-off point and recommend integrating continuously (Beck 2000). Most of the published references to continuous integration use the word "continuous" to mean "at least daily" (Beck 2000), which I think is reasonable. But I occasionally encounter people who take the word "continuous" literally. They aim to integrate each change with the latest build every couple of hours. For most projects, I think literal continuous integration is too much of a good thing. In my free time, I operate a discussion group consisting of the top technical executives from companies like Amazon.com, Boeing, Expedia, Microsoft, Nordstrom, and other Seattle-area companies.

installation tools, Code-Generation Wizards instancing objects, Look for Common Design Patterns, Look for Common Design Patterns, Handling Multiple Instances of Data with ADTs in Non-Object-Oriented Environments, Constructors ADTs, Handling Multiple Instances of Data with ADTs in Non-Object-Oriented Environments factory method, Look for Common Design Patterns singleton, Look for Common Design Patterns, Constructors integer data types, Numbers in General, Numbers in General, Integers, Guidelines for Creating Your Own Types, Relative Performance Costs of Common Operations checklist, Guidelines for Creating Your Own Types (see ) costs of operations, Relative Performance Costs of Common Operations division considerations, Numbers in General overflows, Numbers in General ranges of, Integers Integrated Development Environments (IDEs), Programming Tools integration, Developer Testing, Integration, Integration, Importance of the Integration Approach, Importance of the Integration Approach, Importance of the Integration Approach, Importance of the Integration Approach, Benefits of Incremental Integration, Benefits of Incremental Integration, Benefits of Incremental Integration, Benefits of Incremental Integration, Benefits of Incremental Integration, Benefits of Incremental Integration, Benefits of Incremental Integration, Benefits of Incremental Integration, Benefits of Incremental Integration, Top-Down Integration, Top-Down Integration, Top-Down Integration, Top-Down Integration, Bottom-Up Integration, Bottom-Up Integration, Bottom-Up Integration, Bottom-Up Integration, Bottom-Up Integration, Bottom-Up Integration, Sandwich Integration, Feature-Oriented Integration, Feature-Oriented Integration, Summary of Integration Approaches, Summary of Integration Approaches, Summary of Integration Approaches, Daily Build and Smoke Test, Daily Build and Smoke Test, Daily Build and Smoke Test, Continuous Integration, Continuous Integration, Integration benefits of, Importance of the Integration Approach, Benefits of Incremental Integration big-bang, Importance of the Integration Approach bottom-up strategy, Bottom-Up Integration broken builds, Daily Build and Smoke Test checklist, Continuous Integration classes, Importance of the Integration Approach, Benefits of Incremental Integration, Bottom-Up Integration continuous, Daily Build and Smoke Test customer relations, Benefits of Incremental Integration daily build and smoke test, Summary of Integration Approaches defined, Integration disadvantages of top-down strategy, Top-Down Integration errors, locating, Benefits of Incremental Integration feature-oriented strategy, Feature-Oriented Integration importance of approach methods, Integration interface specification, Top-Down Integration, Bottom-Up Integration key points, Integration monitoring, Benefits of Incremental Integration phased, Importance of the Integration Approach resources on, Continuous Integration risk-oriented strategy, Sandwich Integration sandwich strategy, Bottom-Up Integration scheduling, Benefits of Incremental Integration slices approach, Bottom-Up Integration smoke tests, Daily Build and Smoke Test strategies for, overview, Benefits of Incremental Integration stubs, Benefits of Incremental Integration, Top-Down Integration summary of approaches, Summary of Integration Approaches T-shaped integration, Feature-Oriented Integration testing, Developer Testing, Bottom-Up Integration top-down strategy for, Benefits of Incremental Integration unsurfaced work, Summary of Integration Approaches vertical-slice approach, Top-Down Integration integrity, Characteristics of Software Quality intellectual honesty, Intellectual Honesty intellectual toolbox approach, Applying Software Techniques: The Intellectual Toolbox intelligence, role in character, Intelligence and Humility interfaces, class, Find Real-World Objects, Find Real-World Objects, Secrets and the Right to Privacy, Build Hierarchies, Handling Multiple Instances of Data with ADTs in Non-Object-Oriented Environments, Handling Multiple Instances of Data with ADTs in Non-Object-Oriented Environments, Good Abstraction, Good Abstraction, Good Abstraction, Good Abstraction, Good Abstraction, Good Abstraction, Good Abstraction, Good Abstraction, Good Abstraction, Good Abstraction, Good Abstraction, Good Encapsulation, Good Encapsulation, Good Encapsulation, Good Encapsulation, Multiple Inheritance, Key Points, Reasons to Refactor, Class Implementation Refactorings, Class Implementation Refactorings, Class Implementation Refactorings, Class Interface Refactorings, Class Interface Refactorings, Class Interface Refactorings, Class Interface Refactorings, System-Level Refactorings, Top-Down Integration, Bottom-Up Integration, Interface Documentation Tools, Laying Out Routines, General Guidelines for Class Documentation abstraction aspect of, Find Real-World Objects, Handling Multiple Instances of Data with ADTs in Non-Object-Oriented Environments, Reasons to Refactor calls to classes, refactoring, Class Implementation Refactorings cohesion, Good Abstraction consistent level of abstraction, Good Abstraction delegation vs. inheritance, refactoring, Class Interface Refactorings documenting, Interface Documentation Tools, General Guidelines for Class Documentation erosion under modification problem, Good Abstraction evaluating abstraction of, Good Abstraction extension classes, refactoring with, Class Interface Refactorings foreign routines, refactoring with, Class Interface Refactorings formalizing as contracts, Build Hierarchies good abstraction example, Handling Multiple Instances of Data with ADTs in Non-Object-Oriented Environments guidelines for creating, Good Abstraction inconsistency with members problem, Good Abstraction inconsistent abstraction, example of, Good Abstraction information hiding role, Secrets and the Right to Privacy integration, specification during, Top-Down Integration, Bottom-Up Integration key points for, Key Points layout of, Laying Out Routines mixins, Multiple Inheritance objects, designing for, Find Real-World Objects opposites, pairs of, Good Abstraction poor abstraction example, Good Abstraction private details in, Good Encapsulation programmatic preferred to semantic, Good Abstraction public routines in interfaces concern, Good Encapsulation read-time convenience rule, Good Encapsulation refactoring, Class Implementation Refactorings, System-Level Refactorings routines, moving to refactor, Class Implementation Refactorings routines, unused, Class Interface Refactorings semantic violations of encapsulation, Good Encapsulation unrelated information, handling, Good Abstraction interfaces, graphic, Business Rules interfaces, routine, How to Use Routine Parameters, Code the Routine, Class Implementation Refactorings, Class Interface Refactorings, Class Interface Refactorings, Class Interface Refactorings, Commenting Routines commenting, Commenting Routines foreign routines, refactoring with, Class Interface Refactorings pseudocode for, Code the Routine public member variables, Class Interface Refactorings routines, hiding, Class Interface Refactorings routines, moving to refactor, Class Implementation Refactorings internationalization, Performance interoperability, Performance interpreted languages, performance of, Common Sources of Inefficiency invalid input, Assertions iteration in development, Appeal to Analogy, Determine the Kind of Software You're Working On, Determine the Kind of Software You're Working On, Choosing Between Iterative and Sequential Approaches, General Software-Development Approaches, Iterate, Pseudocode for Pros, Watch for Falling Rocks, Watch for Falling Rocks choosing, reasons for, Choosing Between Iterative and Sequential Approaches (see ) code tuning, Watch for Falling Rocks design practice, Iterate Extreme Programming, General Software-Development Approaches importance of, Watch for Falling Rocks prerequisites, Appeal to Analogy, Determine the Kind of Software You're Working On pseudocode component of, Pseudocode for Pros sequential approach compared, Determine the Kind of Software You're Working On iteration, code, Look for Common Design Patterns, Relationship Between Data Types and Control Structures, Controlling Loops, Controlling Loops, Normal Loop-With-Exit Loops, When to Use a for Loop, Iteration foreach loops, Controlling Loops, When to Use a for Loop iterative data, Relationship Between Data Types and Control Structures iterator loops, defined, Controlling Loops Iterator pattern, Look for Common Design Patterns structured programming concept of, Iteration J jamming loops, Unswitching Java, Java, Classes in General, How to Use Routine Parameters, Assertions, Exceptions, Keep Variables "Live" for as Short a Time as Possible, Comments on Minimizing Scope, C++ Conventions, Sample Naming Conventions, Common Problems with Boolean Expressions, Endline Layout assertion example in, Assertions boolean expression syntax, Common Problems with Boolean Expressions description of, Java exceptions, Exceptions layout recommended, Endline Layout live time examples, Keep Variables "Live" for as Short a Time as Possible naming conventions for, C++ Conventions, Sample Naming Conventions parameters example, How to Use Routine Parameters persistence of variables, Comments on Minimizing Scope resources for, Classes in General Javadoc, Commenting Routines, Additional Resources JavaScript, Java JUnit, Testing just in time binding, Binding Time K key construction decisions, Example of Programming into a Language killed data state, Structured Basis Testing kinds of software projects, Boss-Readiness Test L languages, programming, Choice of Programming Language Law of Demeter, Member Functions and Data layout, Layout Fundamentals, Layout Extremes, Layout Extremes, Layout Extremes, Layout Extremes, Layout Extremes, Human and Computer Interpretations of a Program, Human and Computer Interpretations of a Program, Human and Computer Interpretations of a Program, How Much Is Good Layout Worth?

Cardinal Rule of Software Evolution, Philosophy of Software Evolution CASE (computer-aided software engineering) tools, Programming Tools case statements, Inheritance ("is a" Relationships), Introduce Debugging Aids Early, case Statements, case Statements, case Statements, case Statements, case Statements, case Statements, case Statements, case Statements, Tips for Using case Statements, Tips for Using case Statements, Tips for Using case Statements, Tips for Using case Statements, Tips for Using case Statements, Key Points, Table-Driven Approach, Taming Dangerously Deep Nesting, Taming Dangerously Deep Nesting, Taming Dangerously Deep Nesting, Reasons to Refactor, Reasons to Refactor, Statement-Level Refactorings, Order Tests by Frequency, Compare Performance of Similar Logic Structures, Other Considerations alpha ordering, case Statements (see ; ; ) checklist, Tips for Using case Statements debugging, Introduce Debugging Aids Early default clauses, Tips for Using case Statements drop-throughs, Tips for Using case Statements end of case statements, Tips for Using case Statements endline layout, Other Considerations error detection in, Tips for Using case Statements frequency of execution ordering, case Statements, Order Tests by Frequency if statements, comparing performance with, Compare Performance of Similar Logic Structures key points, Key Points language support for, case Statements nested ifs, converting from, Taming Dangerously Deep Nesting, Taming Dangerously Deep Nesting normal case first rule, case Statements numeric ordering, case Statements ordering cases, case Statements parallel modifications to, Reasons to Refactor phony variables, case Statements polymorphism preferable to, Inheritance ("is a" Relationships) redesigning, Taming Dangerously Deep Nesting refactoring, Reasons to Refactor, Statement-Level Refactorings simple action guideline, case Statements table-driven methods using, Table-Driven Approach change control, Reuse Decisions character arrays, Characters and Strings (see also ) character data types, Floating-Point Numbers, Characters and Strings, Characters and Strings, Characters and Strings, Characters and Strings, Characters and Strings, Guidelines for Creating Your Own Types arrays vs. string pointers, 299 C language, Characters and Strings (see ) character sets, Characters and Strings checklist, Guidelines for Creating Your Own Types conversion strategies, Characters and Strings magic (literal) characters, Floating-Point Numbers Unicode, Characters and Strings, Characters and Strings character, personal, Personal Character, Intelligence and Humility, Intelligence and Humility, Intelligence and Humility, Curiosity, Curiosity, Curiosity, Curiosity, Curiosity, Curiosity, Curiosity, Curiosity, Intellectual Honesty, Intellectual Honesty, Intellectual Honesty, Intellectual Honesty, Intellectual Honesty, Intellectual Honesty, Intellectual Honesty, Intellectual Honesty, Creativity and Discipline, Creativity and Discipline, Creativity and Discipline, Laziness, Persistence, Persistence, Experience, Habits, Habits, Habits, Additional Resources, Watch for Falling Rocks, Iterate, Repeatedly, Again and Again, Topics Beyond Construction analysis skills, Curiosity communication skills, Intellectual Honesty compiler messages, treatment of, Intellectual Honesty computer-science graduates, Creativity and Discipline cooperation skills, Intellectual Honesty creativity, Creativity and Discipline, Topics Beyond Construction curiosity, Curiosity development process awareness, Curiosity discipline, Creativity and Discipline estimations, Intellectual Honesty experience, Persistence experimentation, Curiosity gonzo programming, Experience habits, Habits humility, Intelligence and Humility, Intellectual Honesty, Habits importance of, Personal Character intellectual honesty, Intellectual Honesty intelligence, Intelligence and Humility judgment, Watch for Falling Rocks key points, Additional Resources laziness, Laziness mistakes, admitting to, Intellectual Honesty persistence, Persistence practices compensating for weakness, Intelligence and Humility problem solving, Curiosity professional development, Curiosity reading, Curiosity religion in programming, harmful effects of, Iterate, Repeatedly, Again and Again resources on, Habits status reporting, Intellectual Honesty successful projects, learning from, Curiosity checklists, The Myth of Stable Requirements, Handling Requirements Changes During Construction, Handling Requirements Changes During Construction, General Architectural Quality, General Software-Development Approaches, Example of Programming into a Language, Example of Programming into a Language, Selection of Major Construction Practices, Selection of Major Construction Practices, Standards, Beyond Classes: Packages, Beyond Classes: Packages, Beyond Classes: Packages, Beyond Classes: Packages, Inline Routines, Inline Routines, Being Defensive About Defensive Programming, Alternatives to the PPP, Alternatives to the PPP, Using Each Variable for Exactly One Purpose, Using Each Variable for Exactly One Purpose, Kinds of Names to Avoid, Kinds of Names to Avoid, Guidelines for Creating Your Own Types, Guidelines for Creating Your Own Types, Guidelines for Creating Your Own Types, Guidelines for Creating Your Own Types, Guidelines for Creating Your Own Types, Guidelines for Creating Your Own Types, Guidelines for Creating Your Own Types, How to Reduce the Risks of Using Global Data, How to Reduce the Risks of Using Global Data, Additional Resources, Grouping Related Statements, Tips for Using case Statements, Tips for Using case Statements, Tips for Using case Statements, Correspondence Between Loops and Arrays, gotos, Stair-Step Access Tables, Other Kinds of Complexity, Other Kinds of Complexity, The General Principle of Software Quality, Keys to Success with Pair Programming, General Procedure for an Inspection, General Procedure for an Inspection, Inspections and Code Complete, Testing During Construction, Relevant Standards, Debuggers, Debuggers, Debuggers, Reasons to Refactor, System-Level Refactorings, System-Level Refactorings, System-Level Refactorings, Refactoring Strategies, Performance, Performance, Recoding in a Low-Level Language, Recoding in a Low-Level Language, Machine Configurations, Backup Plan, Continuous Integration, Continuous Integration, Additional Resources, Laying Out Files and Programs, Laying Out Files and Programs, Laying Out Files and Programs, Laying Out Files and Programs, Laying Out Files and Programs, Laying Out Files and Programs, Laying Out Files and Programs, Programming Style as Documentation, Programming Style as Documentation, Programming Style as Documentation, Programming Style as Documentation, Programming Style as Documentation, Programming Style as Documentation, Programming Style as Documentation, Additional Resources, Additional Resources abstraction, Beyond Classes: Packages architecture, General Architectural Quality arrays, Guidelines for Creating Your Own Types backups, Backup Plan boolean expressions, Other Kinds of Complexity case statements, Tips for Using case Statements character data types, Guidelines for Creating Your Own Types classes, Beyond Classes: Packages, Alternatives to the PPP, System-Level Refactorings, Laying Out Files and Programs, Programming Style as Documentation code tuning, Performance, Recoding in a Low-Level Language coding practices, Example of Programming into a Language comments, Laying Out Files and Programs, Additional Resources conditional statements, Tips for Using case Statements configuration management, Machine Configurations constants, Guidelines for Creating Your Own Types construction practices, Example of Programming into a Language control structures, Other Kinds of Complexity, Laying Out Files and Programs, Programming Style as Documentation daily build and smoke tests, Continuous Integration data organization, Programming Style as Documentation data types, Guidelines for Creating Your Own Types debugging, Debuggers defects, General Procedure for an Inspection, Debuggers defensive programming, Being Defensive About Defensive Programming design, Standards, Programming Style as Documentation documentation, Programming Style as Documentation, Additional Resources encapsulation, Beyond Classes: Packages enumerated types, Guidelines for Creating Your Own Types fixing defects, Debuggers formal inspections, General Procedure for an Inspection, Inspections and Code Complete formatting, Laying Out Files and Programs goto statements, gotos if statements, Tips for Using case Statements inheritance, Beyond Classes: Packages initialization, Using Each Variable for Exactly One Purpose integration, Continuous Integration interfaces, System-Level Refactorings layout, Laying Out Files and Programs loops, Correspondence Between Loops and Arrays names, Kinds of Names to Avoid, Programming Style as Documentation pair programming, Keys to Success with Pair Programming parameters, Inline Routines performance tuning, Performance pointers, Additional Resources prerequisites, General Software-Development Approaches programming tools, Additional Resources pseudocoding, Alternatives to the PPP quality assurance, Handling Requirements Changes During Construction, Selection of Major Construction Practices, The General Principle of Software Quality refactoring, Reasons to Refactor, System-Level Refactorings, Refactoring Strategies requirements, The Myth of Stable Requirements, Handling Requirements Changes During Construction routines, Inline Routines, Laying Out Files and Programs, Programming Style as Documentation speed, tuning for, Recoding in a Low-Level Language statements, Laying Out Files and Programs straight-line code, Grouping Related Statements strings, Guidelines for Creating Your Own Types structures, How to Reduce the Risks of Using Global Data table-driven methods, Stair-Step Access Tables testing, Testing During Construction, Relevant Standards tools, Selection of Major Construction Practices type creation, Guidelines for Creating Your Own Types variables, Using Each Variable for Exactly One Purpose, Kinds of Names to Avoid, How to Reduce the Risks of Using Global Data circular dependencies, Two Categories of Secrets class-hierarchy generators, Interface Documentation Tools classes, Program Organization, Level 3: Division into Classes, Level 3: Division into Classes, Inherit—When Inheritance Simplifies the Design, Secrets and the Right to Privacy, Keep Coupling Loose, Build Hierarchies, Working Classes, Good Encapsulation, Good Encapsulation, Good Encapsulation, Good Encapsulation, Good Encapsulation, Good Encapsulation, Good Encapsulation, Good Encapsulation, Good Encapsulation, Good Encapsulation, Good Encapsulation, Containment ("has a" Relationships), Containment ("has a" Relationships), Containment ("has a" Relationships), Inheritance ("is a" Relationships), Inheritance ("is a" Relationships), Inheritance ("is a" Relationships), Inheritance ("is a" Relationships), Inheritance ("is a" Relationships), Inheritance ("is a" Relationships), Inheritance ("is a" Relationships), Multiple Inheritance, Member Functions and Data, Member Functions and Data, Member Functions and Data, Member Functions and Data, Member Functions and Data, Constructors, Constructors, Constructors, Constructors, Constructors, Constructors, Reasons to Create a Class, Reasons to Create a Class, Reasons to Create a Class, Reasons to Create a Class, Reasons to Create a Class, Reasons to Create a Class, Reasons to Create a Class, Reasons to Create a Class, Reasons to Create a Class, Reasons to Create a Class, Reasons to Create a Class, Reasons to Create a Class, Reasons to Create a Class, Summary of Reasons to Create a Class, Summary of Reasons to Create a Class, Beyond Classes: Packages, Beyond Classes: Packages, Beyond Classes: Packages, Beyond Classes: Packages, Beyond Classes: Packages, Classes in General, Key Points, Valid Reasons to Create a Routine, Summary of Steps in Building Classes and Routines, Summary of Steps in Building Classes and Routines, Steps in Creating a Class, Steps in Creating a Class, Pseudocode for Pros, Design the Routine, Clean Up Leftovers, Clean Up Leftovers, Alternatives to the PPP, Alternatives to the PPP, Alternatives to the PPP, Alternatives to the PPP, Guidelines for Initializing Variables, Informal Naming Conventions, Guidelines for a Language-Independent Convention, Sample Naming Conventions, Sample Naming Conventions, Standardized Prefixes, Test-Support Tools, Reasons to Refactor, Reasons to Refactor, Reasons to Refactor, Routine-Level Refactorings, Routine-Level Refactorings, Class Implementation Refactorings, Class Implementation Refactorings, Class Implementation Refactorings, Class Interface Refactorings, Class Interface Refactorings, Class Interface Refactorings, Class Interface Refactorings, Class Interface Refactorings, Class Interface Refactorings, Class Interface Refactorings, System-Level Refactorings, System-Level Refactorings, System-Level Refactorings, Bad Times to Refactor, Importance of the Integration Approach, Benefits of Incremental Integration, Bottom-Up Integration, Laying Out Routines, Laying Out Routines, Laying Out Class Implementations, Laying Out Class Implementations, Laying Out Class Implementations, Laying Out Files and Programs, Programming Style as Documentation, Programming Style as Documentation, General Guidelines for Class Documentation, Watch for Falling Rocks, Watch for Falling Rocks abstract objects, modeling, Constructors (see ) abstraction checklist, Beyond Classes: Packages alternates to PPP, Clean Up Leftovers architecture prerequisites, Program Organization assumptions about users, Good Encapsulation bidirectional associations, System-Level Refactorings calls to, refactoring, Class Implementation Refactorings case statements vs. inheritance, Inheritance ("is a" Relationships) centralizing control with, Reasons to Create a Class changes, limiting effects of, Reasons to Create a Class checklists, Beyond Classes: Packages, Laying Out Files and Programs, Programming Style as Documentation coding routines from pseudocode, Design the Routine cohesion as refactoring indicator, Reasons to Refactor complexity issues, Constructors constant values returned, Routine-Level Refactorings constructors, Constructors containment, Good Encapsulation coupling considerations, Keep Coupling Loose, Good Encapsulation data-free, Reasons to Create a Class deep inheritance trees, Inheritance ("is a" Relationships) defined, Working Classes delegation vs. inheritance, refactoring, Class Interface Refactorings descendants, refactoring indicator for, Reasons to Refactor designing, Level 3: Division into Classes, Summary of Steps in Building Classes and Routines, Pseudocode for Pros, Alternatives to the PPP disallowing functions and operators, Member Functions and Data documenting, Programming Style as Documentation, General Guidelines for Class Documentation encapsulation, Good Encapsulation, Beyond Classes: Packages extension, refactoring with, Class Interface Refactorings factoring, benefit of, Reasons to Create a Class files containing, Laying Out Class Implementations foreign routines, refactoring with, Class Interface Refactorings formalizing contracts for interfaces, Build Hierarchies formatting, Laying Out Routines friend, encapsulation violation concern, Good Encapsulation global data, hiding, Reasons to Create a Class god classes, Reasons to Create a Class hacking approach to, Alternatives to the PPP hiding implementation details, Reasons to Create a Class implementation checklist, Beyond Classes: Packages indirect calls to other classes, Member Functions and Data information hiding, Inherit—When Inheritance Simplifies the Design inheritance, Containment ("has a" Relationships), Beyond Classes: Packages initializing members, Guidelines for Initializing Variables integration, Importance of the Integration Approach, Benefits of Incremental Integration, Bottom-Up Integration irrelevant classes, Reasons to Create a Class is a relationships, Containment ("has a" Relationships) key points for, Key Points, Alternatives to the PPP language-specific issues, Summary of Reasons to Create a Class layout of, Laying Out Routines limiting collaboration, Member Functions and Data Liskov Substitution Principle, Containment ("has a" Relationships) member variables, naming, Guidelines for a Language-Independent Convention, Standardized Prefixes minimizing accessibility rule, Good Encapsulation mixins, Multiple Inheritance modeling real-world objects, Constructors multiple per file, layout of, Laying Out Class Implementations naming, Sample Naming Conventions, Sample Naming Conventions number of members, Good Encapsulation number of routines, Member Functions and Data object names, differentiating from, Informal Naming Conventions objects, contrasted with, Level 3: Division into Classes overformatting, Laying Out Class Implementations overriding routines, Inheritance ("is a" Relationships), Summary of Reasons to Create a Class packages, Reasons to Create a Class parallel modifications refactoring indicator, Reasons to Refactor planning for program families, Reasons to Create a Class private vs. protected data, Inheritance ("is a" Relationships) private, declaring members as, Member Functions and Data protected data, Inheritance ("is a" Relationships) pseudocode for designing, Clean Up Leftovers public members, Good Encapsulation, Good Encapsulation, Class Interface Refactorings read-time convenience rule, Good Encapsulation reasons for creating, Constructors refactoring, Reasons to Create a Class, Routine-Level Refactorings, System-Level Refactorings, Bad Times to Refactor resources, Classes in General reusability benefit of, Reasons to Create a Class review and test step, Steps in Creating a Class routine construction step, Steps in Creating a Class routines, unused, Inheritance ("is a" Relationships), Class Interface Refactorings semantic violations of encapsulation, Good Encapsulation Set() routines, unnecessary, Class Interface Refactorings similar sub and superclasses, Class Interface Refactorings single-instance, Inheritance ("is a" Relationships) singleton property, enforcing, Constructors steps in creating, Summary of Steps in Building Classes and Routines streamlining parameter passing, Reasons to Create a Class subclasses, Valid Reasons to Create a Routine, Class Implementation Refactorings superclasses for common code, Class Implementation Refactorings test-first development, Alternatives to the PPP testing with stub objects, Test-Support Tools unidirectional associations, System-Level Refactorings visibility of, Secrets and the Right to Privacy warning signs for, Watch for Falling Rocks, Watch for Falling Rocks cleanroom development, How Many Errors Should You Expect to Find?


pages: 292 words: 62,575

97 Things Every Programmer Should Know by Kevlin Henney

A Pattern Language, active measures, Apollo 11, business intelligence, business logic, commoditize, continuous integration, crowdsourcing, database schema, deliberate practice, domain-specific language, don't repeat yourself, Donald Knuth, fixed income, functional programming, general-purpose programming language, Grace Hopper, index card, inventory management, job satisfaction, level 1 cache, loose coupling, machine readable, Silicon Valley, sorting algorithm, The Wisdom of Crowds

Let Your Project Speak for Itself Daniel Lindner YOUR PROJECT PROBABLY HAS A VERSION CONTROL SYSTEM IN PLACE. Perhaps it is connected to a continuous integration server that verifies correctness by automated tests. That's great. You can include tools for static code analysis in your continuous integration server to gather code metrics. These metrics provide feedback about specific aspects of your code, as well as their evolution over time. When you install code metrics, there will always be a red line that you do not want to cross. Let's assume you started with 20% test coverage and never want to fall below 15%. Continuous integration helps you keep track of all these numbers, but you still have to check regularly.

Chapter 33 Clint Shank Clint Shank is a software developer, consultant, and mentor at Sphere of Influence, Inc., a company that leads with design-driven innovation to make curve-jumping, mouth-watering software that's awesome inside and out. His typical consulting focus is the design and construction of enterprise applications. He is particularly interested in agile practices such as continuous integration and test-driven development; the programming languages Java, Groovy, Ruby, and Scala; frameworks like Spring and Hibernate; and general design and application architecture. He keeps a blog at http://clintshank.javadevelopersjournal.com/ and was a contributor to the book 97 Things Every Software Architect Should Know.


pages: 232 words: 71,237

Kill It With Fire: Manage Aging Computer Systems by Marianne Bellotti

anti-pattern, barriers to entry, business logic, cloud computing, cognitive bias, computer age, continuous integration, create, read, update, delete, Daniel Kahneman / Amos Tversky, data science, database schema, Dennis Ritchie, DevOps, fault tolerance, fear of failure, Google Chrome, Hans Moravec, iterative process, Ken Thompson, loose coupling, microservices, minimum viable product, Multics, no silver bullet, off-by-one error, platform as a service, pull request, QWERTY keyboard, Richard Stallman, risk tolerance, Schrödinger's Cat, side project, software as a service, Steven Levy, systems thinking, web application, Y Combinator, Y2K

How Some Useful Design Exercises for Engineering Teams Exercise: Critical Factors3 Exercise: The Saboteur4 Exercise: Shared Uncertainties5 Exercise: The 15 Percent6 Exercises Specifically for Decisions Exercise: Probabilistic Outcome-Based Decision-Making Exercise: Affinity Mapping Team Structure, Organization Structure, and Incentives Individual Incentives Minor Adjustments as Uncertainty Organization Size and Communication Manager Incentives Designing a Team: Applications of Conway’s Law Reorgs Are Traumatic Finding the Right Leadership Exercise: The Smallest Testable Unit Structuring the Team to Account for Past Failure Exercise: In-Group/Out-Group Takeaways Chapter 8: Breaking Changes Being Seen Who Draws the Line? Building Trust Through Failure Breaking Change vs. Breaking Why Break Things on Purpose? Projecting Impact The Kill Switch Communicating Failure Failure Is a Best Practice Chapter 9: How to Finish Revealing Assumptions Approach 1: Success Criteria Example: Adding Continuous Integration/Continuous Deploy Approach 2: Diagnosis-Policy-Actions Example: Upgrading a Database Comparison Marking Time Postmortems on Success Postmortem vs. Retrospective Running Postmortems The Tale of Two War Rooms Working Groups vs. Committees Success Is Not Obvious If Undefined Chapter 10: Future-Proofing Time Unescapable Migrations Failing Gracefully Less Time Between Upgrades, Not More A Word of Caution About Automation Building Something Wrong the Correct Way Feedback Loops Don’t Stop the Bus Conclusion Index Kill It with Fire Manage Aging Computer Systems (and future-Proof Modern Ones) by Marianne Bellotti San Francisco KILL IT WITH FIRE.

You can use such analysis to discover lost business requirements or track down dead code. Software renovation methodology hasn’t quite broken out of theoretical studies, but static analysis tools are available both as stand-alone products and as features of larger integrated development environments or continuous integration and deployment solutions. This is unfortunate because the methodology is what drives the bulk of the impact. The tools themselves are not as important as the phases of excavating, understanding, documenting, and ultimately rewriting and replacing legacy systems. Tools will come and go. A Guide to Not Making Things Harder Expectation management is really important.

Consultants don’t have much control over deploys, and the only way they get control is by not being consultants anymore. As software engineers, it is easy to fall into the trap of thinking that effective work will always look like writing code, but sometimes you get much closer to your desired outcome by teaching others to do the job rather than doing it yourself. Example: Adding Continuous Integration/Continuous Deploy Goal: Move service on to its own deploy pipeline. Timeline: One quarter. Success Criteria: Time to deploy drops by 20 percent. Any single person on the Service team can initiate and manage a deploy. Number of deploys in a week goes up. Approach 2: Diagnosis-Policy-Actions Developed by Richard Rumelt, this approach draws on the same information as success criteria but frames it a bit differently.1 Information is represented in three segments: diagnosis, policy, and actions.


pages: 584 words: 149,387

Essential Scrum: A Practical Guide to the Most Popular Agile Process by Kenneth S. Rubin

bioinformatics, business cycle, business intelligence, business logic, business process, continuous integration, corporate governance, fail fast, hiring and firing, index card, inventory management, iterative process, Kanban, Lean Startup, loose coupling, minimum viable product, performance metric, shareholder value, six sigma, tacit knowledge, Y2K, you are the product

Use Good Technical Practices The first approach to managing the accrual of technical debt is to stop adding naive debt to our products. Using good technical practices is an excellent starting point. Although Scrum does not formally define technical practices, every successful Scrum team that I have seen employs practices such as simple design, test-driven development, continuous integration, automated testing, refactoring, and so on (see Chapter 20 for additional discussion). Understanding and proactively using these practices will help teams stop adding many forms of naive debt to their products. In the case of accrued technical debt, code refactoring is an important tool for paying it down.

As an organization we should be asking, “Why did we have to work nights and weekends, and what should we change?” Compare that with the typical intensity profile when using Scrum, where we’ve been continuously developing, testing, and integrating working features every sprint. During each sprint the team members should be using good technical practices such as refactoring, continuous integration, and automated tests to ensure that they can deliver value at frequent, regular intervals without killing themselves. So, within a given sprint we’ll likely see intensity increase a bit near the end of the sprint as we ensure that all work associated with our strong definition of done has been met.

If you are using Scrum to develop software, team members need to be skilled in good technical practices for developing software. I’m not referring to esoteric skills but instead to skills that have been in use for decades and are essential to being successful with Scrum or arguably any software development approach—for example, continuous integration, automated testing, refactoring, test-driven development, and so on. Today the agile community refers to many of these technical practices as Extreme Programming (Beck and Andres 2004), but most are practices that predate that label (see Figure 20.5 for a subset of the Extreme Programming technical practices).


pages: 210 words: 42,271

Programming HTML5 Applications by Zachary Kessin

barriers to entry, continuous integration, fault tolerance, Firefox, functional programming, Google Chrome, higher-order functions, machine readable, mandelbrot fractal, QWERTY keyboard, SQL injection, web application, WebSocket

Finally, it is possible to write tests in a unit test framework in a programming language such as PHPUnit. Using the test runner or the programming language-based test suite allows testing with a full suite of browsers and can provide reporting and other functions. This procedure can also be integrated with continuous integration tools along with any other tests written in any of the xUnit frameworks. A Selenium test is constructed from an HTML file containing a table, with each step in the test being a row in the table. The row consists of three columns: the command to run, the element on which it will act, and an optional parameter used in some cases.

By using the API from a server-side programming language, it is possible to create a very rich environment for scripting the Web, and of course you have access to libraries on the server side to check data in a database or access web services. You can construct a test that will perform some actions in the browser, and then check the result in a database or against a logfile. Another advantage of server-side testing is that if you are using any form of continuous integration, such as CruiseControl or phpUnderControl, the Selenium tests appear to the test system as just more tests in whatever language the team is using. In a team that is using a test framework, this will leverage the existing experience of the team. Example 3-8 is a very simple Selenium test, written in PHP with the PHPUnit testing framework.

Running QUnit from Selenium Selenium can run QUnit tests as well. To do so, load the QUnit page in Selenium and run the tests. It is also possible to choose to only run a subset of tests by passing parameters to the URL string. By integrating Selenium with QUnit, you can export the results of browser tests in QUnit into a test runner for continuous integration. Selenium just opens the QUnit URL and then stands back and waits for the test to finish. To let the test runner know whether the tests passed or failed, QUnit provides a simple micro format (see Example 3-12) that shows how many tests were run and how many passed or failed. The unit test can then look for this data by an XPath selector and make sure all tests passed.


pages: 193 words: 46,550

Twisted Network Programming Essentials by Jessica McKellar, Abe Fettig

business logic, continuous integration, WebSocket

Nonblocking Database Queries | 79 More Practice and Next Steps This chapter discussed how to interact with databases in a non-blocking fashion using Twisted’s adbapi. adbapi provides an asynchronous interface to Python’s DB-API 2.0 specification, which is defined in PEP 249. The methods in the asynchronous interface map directly to methods in the blocking API, so converting a service from blocking database queries to adbapi is straightforward. For an example of how a large project uses Twisted’s relational database support, check out the Buildbot continuous integration framework. Twistar is a library that builds an object-relational mapper (ORM) on top of adbapi. 80 | Chapter 8: Databases CHAPTER 9 Authentication Twisted comes with a protocol-independent, pluggable, asynchronous authentication system called Cred that can be used to add any type of authentication support to your Twisted server.

For example, to see how Twisted Web’s Agent interface is tested, including mocking the transport, testing timeouts, and testing errors, have a look at twisted/web/test/ test_agent.py. To see how to test a protocol like twisted.words.protocols.irc, check out twisted/words/tests/test_irc.py. You can read about Twisted’s test-driven development policy in detail on the Twisted website. Twisted publishes its own coverage information as part of its continuous integration. Help improve Twisted by writing test cases! More Practice and Next Steps | 115 PART III More Protocols and More Practice CHAPTER 12 Twisted Words Twisted Words is an application-agnostic chat framework that gives you the building blocks to build clients and servers for popular chat protocols and to write new protocols.

You now have all of the tools you need to build and deploy event-driven clients and servers for any protocol, and I think you’ll find that to be a powerful tool to have in your back pocket. Twisted powers everything from networked game engines and streaming media servers to web crawling frameworks and continuous integration systems to Bit‐ Torrent clients and AMQP peers. The next time you need to programmaticaly download data from a website, test an HTTP client, process your email, or annoy your friends with an IRC bot, you know what to do. Thank you for reading! We’d love to hear your thoughts on this book.


pages: 419 words: 102,488

Chaos Engineering: System Resiliency in Practice by Casey Rosenthal, Nora Jones

Amazon Web Services, Asilomar, autonomous vehicles, barriers to entry, blockchain, business continuity plan, business intelligence, business logic, business process, cloud computing, cognitive load, complexity theory, continuous integration, cyber-physical system, database schema, DevOps, fail fast, fault tolerance, hindsight bias, human-factors engineering, information security, Kanban, Kubernetes, leftpad, linear programming, loose coupling, microservices, MITM: man-in-the-middle, no silver bullet, node package manager, operational security, OSI model, pull request, ransomware, risk tolerance, scientific management, Silicon Valley, six sigma, Skype, software as a service, statistical model, systems thinking, the scientific method, value engineering, WebSocket

The great advantage of software is that you can make explicit design decisions that improve Reversibility as well. Development of applications on the web, where a browser reload is all that is required to deploy a new codebase, drastically improved the Reversibility of applications. Version control, blue–green deployment methods, automated canaries, feature flags, continuous integration/continuous delivery (CI/CD)—these are all examples of technical solutions and architectural decisions that explicitly improve Reversibility. Any technique that improves the ability of a software engineer to get code into production, change their mind, and roll back or roll forward that decision, improves their Reversibility.

That said, clear trends have emerged that outline the future directions of this practice and its place within broader industry. This part of the book speaks to those trends. The first chapter in this part of the book, Chapter 16, “Continuous Verification,” situates Chaos Engineering within a larger category of software practices. “Like CI/CD (continuous integration/continuous delivery), the practice is born out of a need to navigate increasingly complex systems. Organizations do not have the time or other resources to validate that the internal machinations of the system work as intended, so instead they verify that the output of the system is inline with expectations.”

The chapters in this part of the book illuminate a few of those corners. Chapter 16. Continuous Verification Continuous verification is a discipline of proactive experimentation in software, implemented as tooling that verifies system behaviors. Casey Rosenthal The challenges posed by complex systems encouraged a natural evolution from continuous integration to continuous delivery to continuous verification. The latter is the subject of this chapter, describing the nascent field and the room for opportunity, and following up with a real-world example of one of the systems in production at Netflix called ChAP. Where the industry decides to take continuous verification is wide open, but the general areas of focus for future development are reviewed at the end of this chapter.


pages: 1,380 words: 190,710

Building Secure and Reliable Systems: Best Practices for Designing, Implementing, and Maintaining Systems by Heather Adkins, Betsy Beyer, Paul Blankinship, Ana Oprea, Piotr Lewandowski, Adam Stubblefield

air gap, anti-pattern, barriers to entry, bash_history, behavioural economics, business continuity plan, business logic, business process, Cass Sunstein, cloud computing, cognitive load, continuous integration, correlation does not imply causation, create, read, update, delete, cryptocurrency, cyber-physical system, database schema, Debian, defense in depth, DevOps, Edward Snowden, end-to-end encryption, exponential backoff, fault tolerance, fear of failure, general-purpose programming language, Google Chrome, if you see hoof prints, think horses—not zebras, information security, Internet of things, Kubernetes, load shedding, margin call, microservices, MITM: man-in-the-middle, NSO Group, nudge theory, operational security, performance metric, pull request, ransomware, reproducible builds, revision control, Richard Thaler, risk tolerance, self-driving car, single source of truth, Skype, slashdot, software as a service, source of truth, SQL injection, Stuxnet, the long tail, Turing test, undersea cable, uranium enrichment, Valgrind, web application, Y2K, zero day

This method of testing environmental conditions has withstood the test of time, since it explicitly captures the classes of failure from which we want to recover. Finally, look for ways to build confidence in your recovery methods via continuous validation. Recovery involves actions taken by humans, and humans are unreliable and unpredictable. Unit tests alone, or even continuous integration/delivery/deployment, cannot catch mistakes resulting from human skills or habits. For example, in addition to validating the effectiveness and interoperability of recovery workflows, you must validate that recovery instructions are readable and easy to comprehend. Emergency Access The recovery methods described in this chapter rely on a responder’s ability to interact with the system, and we’ve advocated for recovery processes that exercise the same primary services as normal operations.

Another set of conformance checks enforces isolation constraints between components, which makes it much less likely that a change in one component/module of the application results in a bug in another component. Because applications built on this framework have a fairly rigid and well-defined structure, the framework can provide out-of-the-box automation for many common development and deployment tasks—from scaffolding for new components, to automated setup of continuous integration (CI) environments, to largely automated production deployments. These benefits have made this framework quite popular among Google developers. What does all this have to do with security and reliability? The framework development team collaborated with SRE and security teams throughout the design and implementation phases, ensuring that security and reliability best practices were woven into the fabric of the framework—not just bolted on at the end.

A primary goal of Agile development workflows is to increase development and deployment velocity—in particular, to reduce the latency between feature specification and deployment. However, Agile workflows typically rely on reasonably mature unit and integration testing practices and a solid continuous integration infrastructure, which require an up-front investment to establish, in exchange for long-term benefits to velocity and stability. More generally, you can choose to prioritize initial project velocity above all else—you can develop the first iteration of your web app without tests, and with a release process that amounts to copying tarballs to production hosts.


pages: 318 words: 78,451

Kanban: Successful Evolutionary Change for Your Technology Business by David J. Anderson

airport security, anti-pattern, business intelligence, call centre, collapse of Lehman Brothers, continuous integration, corporate governance, database schema, domain-specific language, index card, Kaizen: continuous improvement, Kanban, knowledge worker, lateral thinking, loose coupling, performance metric, six sigma, systems thinking, tacit knowledge, Toyota Production System, transaction costs

This approach, in which color does not clearly designate any one attribute, seems to be acceptable to users of the kanban system and is very efficient in terms of available options for visualization. Systems Integration On some larger projects, you may have multiple teams working on different components of a system that need to be integrated later. Some of these components may involve hardware or firmware, and may be not amenable to modern continuous-integration techniques. When you have such components that need to be integrated, you need to determine an integration point based on a coarse-grained high-level planning activity. This point should then be treated as a fixed date for delivery of these dependent components. This allows each team to move forward independently with its kanban system, but also to coordinate delivery of dependent items when they are needed.

It may also be necessary to provide a brief overview of likely improvement opportunities such as bottlenecks, waste, and variability. As these opportunities for improvement are uncovered, more training in new skills and techniques may be needed. For example, if defects are a major source of waste, the development team may require training in techniques that will greatly reduce defects and improve code quality, such as continuous integration, unit testing, and pair programming. However, rather than waste too much time on education, in the first instance, it is more important that you gain a consensus around the introduction of Kanban and start using it. This chapter seeks to lay out the foundations for a successful Kanban transition and provides you with a simple 12-step guide to getting started.

If we want to interpret this for a software development–specific example, an internal, chance-cause variation would be the number of bugs created per line of code, per requirement, per task, or per unit of time. The mean number, the spread, and the distribution of the bug (or defect) rate can be affected by changing the tools and process, such as by insisting on unit tests, continuous integration, and peer code reviews. The process definition in use on your team, expressed as policies, represents the rules of the collaborative game of software development. The rules of the game determine the sources and quantity of internal variation. The irony lies in the notion that the “chance-cause” variations are actually directly under the control of the team and management through their ability to modify policies, change the process, and affect sources of internal variation.


pages: 157 words: 35,874

Building Web Applications With Flask by Italo Maia

continuous integration, create, read, update, delete, Debian, en.wikipedia.org, Firefox, full stack developer, minimum viable product, MVC pattern, premature optimization, SQL injection, web application

You could, for example, change a piece of code to fix a bug and create another bug in another point in your code. Software tests also help with that as they assure that your code does what it should do; if you change a piece of broken code and break another piece of code, you'll also be breaking a test. In this scenario, if you make use of continuous integration, the broken code will never reach your production environment. Tip Don't know what continuous integration is? Refer to http://www.martinfowler.com/articles/continuousIntegration.html and https://jenkins-ci.org/. Tests are so important that there is a software development process called Test Driven Development (TDD), which states that the test should be written before the actual code, and that the actual code is only ready when the test itself is satisfied.


pages: 282 words: 79,176

Pro Git by Scott Chacon

Chris Wanstrath, continuous integration, creative destruction, Debian, distributed revision control, GnuPG, pull request, revision control, systems thinking

If you try, you’ll see a login rejection like this: $ ssh git@gitserver fatal: What do you think I am? A shell? Connection to gitserver closed. Public Access What if you want anonymous read access to your project? Perhaps instead of hosting an internal private project, you want to host an open source project. Or maybe you have a bunch of automated build servers or continuous integration servers that change a lot, and you don’t want to have to generate SSH keys all the time — you just want to add simple anonymous read access. Probably the simplest way for smaller setups is to run a static web server with its document root where your Git repositories are, and then enable that post-update hook we mentioned in the first section of this chapter.

Again, this is for unauthenticated read-only access. If you’re running this on a server outside your firewall, it should only be used for projects that are publicly visible to the world. If the server you’re running it on is inside your firewall, you might use it for projects that a large number of people or computers (continuous integration or build servers) have read-only access to, when you don’t want to have to add an SSH key for each. In any case, the Git protocol is relatively easy to set up. Basically, you need to run this command in a daemonized manner: git daemon --reuseaddr --base-path=/opt/git/ /opt/git/ --reuseaddr allows the server to restart without waiting for old connections to time out, the --base-path option allows people to clone projects without specifying the entire path, and the path at the end tells the Git daemon where to look for repositories to export.

You can use this hook to do things like make sure none of the updated references are non-fast-forwards; or to check that the user doing the pushing has create, delete, or push access or access to push updates to all the files they’re modifying with the push. The post-receive hook runs after the entire process is completed and can be used to update other services or notify users. It takes the same stdin data as the pre-receive hook. Examples include e-mailing a list, notifying a continuous integration server, or updating a ticket-tracking system — you can even parse the commit messages to see if any tickets need to be opened, modified, or closed. This script can’t stop the push process, but the client doesn’t disconnect until it has completed; so, be careful when you try to do anything that may take a long time.


pages: 448 words: 84,462

Testing Extreme Programming by Lisa Crispin, Tip House

business logic, c2.com, continuous integration, data acquisition, database schema, Donner party, Drosophila, fail fast, hypertext link, index card, job automation, systems thinking, web application

The programmers write the unit tests before they write the code, then add unit tests whenever one is found to be missing. No modification or refactoring of code is complete until 100% of the unit tests have run successfully. Acceptance tests validate larger blocks of system functionality, such as user stories. When all the acceptance tests pass for a given user story, that story is considered complete. Continuous integration. Additions and modifications to the code are integrated into the system on at least a daily basis, and the unit tests must run 100% successfully, both before and after each integration. Small releases. The smallest useful feature set is identified for the first release, and releases are performed as early and often as possible, with a few new features added each time.

This is especially frustrating to testers, because they really can't do anything about it. No matter how thorough, efficient, and/or automated are the tests they develop, they can't put either functionality or quality into the system; they can only detect its presence or absence. XP avoids this scenario completely through the practices of 100% unit-test automation and continuous integration. Programmer pairs detect and correct unit and integration bugs during the coding sessions. By the time the code gets to acceptance testing, it's doing what the programmers intend, and the acceptance tests can focus on determining how well that intent matches the customer's expectations. Missing and Out-of-Date Requirements Moving upstream in the process, another common problem in developing tests occurs when requirements are missing or out of date.

Summary XP is a lightweight but disciplined approach to software development that has testing and quality at its core. XP is based on four values: communication, simplicity, feedback, and courage. Twelve practices comprise the rules of XP: Onsite customer Pair programming Coding standards Metaphor Simple design Refactoring Testing Continuous integration Small releases Planning game Collective code ownership Sustainable pace XP solves three major testing and quality assurance problems: Unit and integration bugs during system and acceptance testing Lack of requirements from which to develop tests Large gaps between customer expectations and delivered product Chapter 2.


pages: 355 words: 81,788

Monolith to Microservices: Evolutionary Patterns to Transform Your Monolith by Sam Newman

Airbnb, business logic, business process, continuous integration, Conway's law, database schema, DevOps, fail fast, fault tolerance, ghettoisation, inventory management, Jeff Bezos, Kubernetes, loose coupling, microservices, MVC pattern, price anchoring, pull request, single page application, single source of truth, software as a service, source of truth, sunk-cost fallacy, systems thinking, telepresence, two-pizza team, work culture

In Building Microservices, I shared the experiences of the SnapCI product team at ThoughtWorks. Despite knowing the domain of continuous integration really well, their initial stab at coming up with service boundaries for their hosted-CI solution wasn’t quite right. This led to a high cost of change and high cost of ownership. After several months fighting this problem, the team decided to merge the services back into one big application. Later, when the feature-set of the application had stabilized somewhat and the team had a firmer understanding of the domain, it was easier to find those stable boundaries. SnapCI was a hosted continuous integration and continuous delivery tool. The team had previously worked on another similar tool, Go-CD, a now open source continuous delivery tool that can be deployed locally rather than being hosted in the cloud.

The challenge is that once the change in the branch has been completed, these changes have to be merged back, which can often cause significant challenges. The longer the branch exists, the bigger these problems are. I won’t go into detail now as to the problems associated with long-lived source code branches, other than to say they run contrary to the principles of continuous integration. I could also throw in that data gathered from “The 2017 State of DevOps Report” shows that embracing trunk-based development (where changes are made directly on the main line and branches are avoided) and using short-lived branches contributes to higher performance of IT teams. Let’s just say that I am not a fan of long-lived branches, and I’m not alone.


Producing Open Source Software: How to Run a Successful Free Software Project by Karl Fogel

active measures, AGPL, barriers to entry, Benjamin Mako Hill, collaborative editing, continuous integration, Contributor License Agreement, corporate governance, Debian, Donald Knuth, en.wikipedia.org, experimental subject, Firefox, Free Software Foundation, GnuPG, Hacker Ethic, Hacker News, intentional community, Internet Archive, iterative process, Kickstarter, natural language processing, off-by-one error, patent troll, peer-to-peer, pull request, revision control, Richard Stallman, selection bias, slashdot, software as a service, software patent, SpamAssassin, the Cathedral and the Bazaar, Wayback Machine, web application, zero-sum game

Canned Hosting A canned hosting site is an online service that offers some or all of the online collaboration tools needed to run a free software project. At a minimal, a canned hosting site offers public version control repositories and bug tracking; most also offer wiki space, many offer mailing list hosting too, and some offer continuous integration testing and other services.[27] There are two main advantages to using a canned site. The first is server capacity and bandwidth: their servers are hefty boxes sitting on really fat pipes. No matter how successful your project gets, you're not going to run out of disk space or swamp the network connection.

So if you want to provide hosting, do so—but if you are actually trying to create a public association between your company and the project, make sure that the level of support you provide is matched to the amount of credit you claim, whether you claim it via URL, banner ads, or some other means. Providing Build Farms and Development Servers Many projects have infrastructure needs beyond just hosting of code, bug tracker, etc. For example, projects often use continuous integration (CI) testing (a.k.a. build farms) to automatically ensure that the changes developers are committing both integrate into the mainline trunk and pass all regression tests[53]. However, depending on the size and complexity of the codebase, the number of developers checking in changes, and other factors, running a responsive build farm can cost more money than any individual developer has at their disposal.

However, depending on the size and complexity of the codebase, the number of developers checking in changes, and other factors, running a responsive build farm can cost more money than any individual developer has at their disposal. A good way to help, and gain some goodwill in the process, is to donate the server space and bandwidth and the technical expertise to set up the continuous integration and automated testing. If you don't have the technical expertise available on staff, you could hire someone from the project to do it, or at the very least give some of the project's developers administrative access to the CI servers so they can set things up themselves. Sponsoring Conferences, Hackathons, and other Developer Meetings A very effective use of funds is to sponsor in-person contact between developers who might not otherwise meet.


pages: 549 words: 134,988

Pro Git by Scott Chacon, Ben Straub

Chris Wanstrath, continuous integration, creative destruction, Debian, distributed revision control, GnuPG, pull request, remote working, revision control, systems thinking, web application

Remember that since it’s not an authenticated service, anything you serve over this protocol is public within its network. If you’re running this on a server outside your firewall, it should only be used for projects that are publicly visible to the world. If the server you’re running it on is inside your firewall, you might use it for projects that a large number of people or computers (continuous integration or build servers) have read-only access to, when you don’t want to have to add an SSH key for each. In any case, the Git protocol is relatively easy to set up. Basically, you need to run this command in a daemonized manner: git daemon --reuseaddr --base-path=/opt/git/ /opt/git/ --reuseaddr allows the server to restart without waiting for old connections to time out, the --base-path option allows people to clone projects without specifying the entire path, and the path at the end tells the Git daemon where to look for repositories to export.

Both the Hooks and Services integrations can be found in the Settings section of your repository, where we previously looked at adding Collaborators and changing the default branch of your project. Under the “Webhooks and Services” tab you will see something like Figure 6-49. Figure 6-49. Services and Hooks configuration section. There are dozens of services you can choose from, most of them integrations into other commercial and open source systems. Most of them are for Continuous Integration services, bug and issue trackers, chat room systems and documentation systems. We’ll walk through setting up a very simple one, the Email hook. If you choose “email” from the “Add Service” dropdown, you’ll get a configuration screen like Figure 6-50. Figure 6-50. Email service configuration.

You can use the API to do just about anything you can do on the website — creating and setting milestones, assigning people to Issues and Pull Requests, creating and changing labels, accessing commit data, creating new commits and branches, opening, closing or merging Pull Requests, creating and editing teams, commenting on lines of code in a Pull Request, searching the site and on and on. Changing the Status of a Pull Request One final example we’ll look at since it’s really useful if you’re working with Pull Requests. Each commit can have one or more statuses associated with it and there is an API to add and query that status. Most of the Continuous Integration and testing services make use of this API to react to pushes by testing the code that was pushed, and then report back if that commit has passed all the tests. You could also use this to check if the commit message is properly formatted, if the submitter followed all your contribution guidelines, if the commit was validly signed — any number of things.


pages: 196 words: 58,122

AngularJS by Brad Green, Shyam Seshadri

business logic, combinatorial explosion, continuous integration, Firefox, Google Chrome, Kickstarter, MVC pattern, node package manager, single page application, systems thinking, web application, WebSocket

test main require module last 'test/spec/main.js' ]; // list of files to exclude exclude = []; // test results reporter to use // possible values: dots || progress reporter = 'progress'; // web server port port = 8989; // cli runner port runnerPort = 9898; // enable/disable colors in the output (reporters and logs) colors = true; // level of logging logLevel = LOG_INFO; // enable/disable watching file and executing tests whenever any file changes autoWatch = true; // Start these browsers, currently available: // - Chrome // - ChromeCanary // - Firefox // - Opera // - Safari // - PhantomJS // - IE if you have a windows box browsers = ['Chrome']; // Continuous Integration mode // if true, it captures browsers, runs tests, and exits singleRun = false; We use a slightly different format to define our dependencies (the included: false is quite important). We also add the dependency on REQUIRE_JS and its adapter. The final thing to get all this working is main.js, which triggers our tests. // This file is test/spec/main.js require.config({ // !!

./'; // list of files / patterns to load in the browser files = [ ANGULAR_SCENARIO, ANGULAR_SCENARIO_ADAPTER, 'test/e2e/*.js' ]; // list of files to exclude exclude = []; // test results reporter to use // possible values: dots || progress reporter = 'progress'; // web server port port = 8989; // cli runner port runnerPort = 9898; // enable / disable colors in the output (reporters and logs) colors = true; // level of logging logLevel = LOG_INFO; // enable / disable watching file and executing tests whenever any file changes autoWatch = true; urlRoot = '/_karma_/'; proxies = { '/': 'http://localhost:8000/' }; // Start these browsers, currently available: browsers = ['Chrome']; // Continuous Integration mode // if true, it capture browsers, run tests and exit singleRun = false; Chapter 4. Analyzing an AngularJS App We talked about some of the commonly used features of AngularJS in Chapter 2, and then dived into how your development should be structured in Chapter 3. Rather than continuing with similarly deep dives into individual features, Chapter 4 will look at a small, real-life application.


Designing Web APIs: Building APIs That Developers Love by Brenda Jin, Saurabh Sahni, Amir Shevat

active measures, Amazon Web Services, augmented reality, Big Tech, blockchain, business logic, business process, cognitive load, continuous integration, create, read, update, delete, exponential backoff, Google Hangouts, if you build it, they will come, Lyft, machine readable, MITM: man-in-the-middle, premature optimization, pull request, Salesforce, Silicon Valley, Snapchat, software as a service, the market place, uber lyft, web application, WebSocket

Automated Testing It’s important that everyone who impacts the API understands and supports consistency, but it’s difficult to enforce as an organizational value. And you cannot expect people to always make the right deci‐ sions for the system, especially when they need to reconsider the 120 | Chapter 7: Managing Change minutiae of a complex system for every decision. That’s where auto‐ mated testing comes into play. Continuous integration (CI) is the practice of merging all developers’ working copies to a single shared repository, often multiple times a day. The workflow in which a code change goes from being written to approved and merged is called a CI pipeline (Figure 7-1). Adding an automated testing step (see Step 3 in Figure 7-1) before develop‐ ers are allowed to merge their code is a great choice for preventing unwanted changes from sneaking into the API—especially the back‐ ward incompatible ones.

API Implementation Checklist: ❏ Define specific developer problem to solve ❏ Write internal API specification ❏ Get internal feedback on API specification ❏ Build API ❏ Authentication ❏ Authorization ❏ Error handling ❏ Rate-limiting ❏ Pagination ❏ Monitoring and logging ❏ Write documentation ❏ Run beta test with partners on new API ❏ Gather feedback from beta partners and make changes ❏ Create communication plan to notify developers of changes ❏ Release API changes API Design Worksheets | 205 Index A additions to APIs, 131 additive-change strategy, 133 Amazon Web Services (AWS), 83 ambassador programs, 195 analytics dashboards, 56 Apache Thrift, 14 API testers, 179 APIs about, 1 attributes and traits of good APIs, 199 business case for, 3-7 APIs as a product, 6 APIs for external developers first, internal developers second, 5 APIs for internal developers first, external developers second, 4 characteristics of great APIs, 7 description language, 122-126 design paradigms, 9 designing (see designing APIs, best practices; designing APIs, practical exercise in) event-driven, 19-25 comparison of different types, 24 HTTP Streaming, 23-24 WebHooks, 19-22 WebSockets, 22-23 request–response comparison of types, 18 GraphQL, 14-18 REST, 10-13 RPC, 13-14 security (see security) uses of, 2 application names, misleading, pro‐ hibiting in OAuth, 40 application-level caching for APIs, 87 archiving a GitHub repository, HTTP request for, 12 asynchronous operations, 89 authentication, 27 choosing mechanism for MyFiles API (example), 66 authorization, 28 OAuth 2.0 as standard for, 29 Authorization header (HTTP) for Basic Authentication, 28 with OAuth tokens and scope, 34 automated testing (see testing) awareness tactics examples, 156 AWS (see Amazon Web Services) B backward compatibility, maintaining, 58, 127-128 Basic Authentication, 28 beta programs, 188-190 beta testers, 79 Botkit framework, 177 bottlenecks, finding, 82 207 breadth and depth analysis for devel‐ oper programs, 185 breadth developer programs, 192-197 bulk operation endpoints, support‐ ing, 95 business case for APIs, 3-7 business objectives, defining, 62-64 business-focused tech savvy audience, 145 C cache invalidation, 87 caching in developer SDKs, 115 using to scale thoughput, 87 change, managing, 117-142 aiming for consistency, 117-126 using automated testing, 120-126 backward compatibility, 127-128 planning for and communicating change, 128-141 additions to the API, 131 communication plan, 129 removing fields or endpoints, 132 versioning, 133-141 changelogs, 170 CI pipeline, 121 clickjacking, 40 client secret, ability to reset in OAuth, 39 cloud hosting providers, solutions for measuring bottlenecks, 83 code samples, 173 code snippets, 174 communication means, preferred, of developers, 149 communication plan for changes, 129 community contributions, 182-183 community, building, 192 computing resources, adding to scale applications, 85 consistency in an API, 50, 117-126 hallmarks of consistency, 118 using automated testing, 120-126 continuous integration (CI), 121 Conversations APIs (Slack), 13, 93 208 | Index CPU bottlenecks, 83 Create, Read, Update, and Delete operations (see CRUD operations) credit programs, 197 cross-site request forgery (CSRF), 27, 38 cross-site scripting (XSS), 27 CRUD operations, 10 HTTP verbs and REST conven‐ tions, 11 in MyFiles API (example), 67 in request–response API para‐ digms, 18 pros and cons of API paradigms for MyFiles API (example), 65 current actual numbers for developer funnel indicators, 155 cursor-based pagination, 99-101 advantages and disadvantages, 100 ID as cursor, 101 opaque strings as cursor, 101 timestamp as cursor, 101 custom HTTP headers rate-limit response headers, 110 with OAuth token and scope, 34 D dark launching rate-limiting, 109 data access patterns (new), introduc‐ ing in your API, 90 database indexes, 86 database profiling, 84 database replication, 85 database sharding, 85 date filters, 96 debugging tools, 179 DELETE method (HTTP), 10 (see also CRUD operations) depth developer programs, 187-192 design sprints, 160, 191 designing APIs, best practices, 47-59 design for great developer experi‐ ence, 48-59 making it fast and easy to get started, 48 making troubleshooting easy, 52-56 making your API extensible, 56 working toward consistency, 50 design for real-life use cases, 47-48 focusing on users, 2 designing APIs, practical exercise in, 61-79 scenario 1, 62-72 defining business objectives, 62-64 outlining key user stories, 64 selecting technology architec‐ ture, 65 writing an API specification, 68-72 scenario 2, 72-79 defining problem and impact statement, 73 getting feedback on the API spec, 77-79 selecting technology architec‐ ture, 74 writing an API specification, 74-77 developer ecosystem strategy, build‐ ing, 143-161 building a developer strategy, 147 defining your developer funnel, 152-154 funnel indicators, 154 deriving measurements, 160 developer segmentation, 147-150 common use cases and tasks, 148 examples of segmentation analysis, 149 identity, 147 market size and geographical distribution, 149 platform of choice, 148 preferred development lan‐ guage, framework, and tools, 148 preferred means of communi‐ cation, 149 proficiency, 148 developers, 144-147 business-focused tech savvy audience, 145 hackers, 145 hobbyist, 144 professional developers, 146 variations on categories men‐ tioned, 146 distilling the value proposition, 151 mapping current and future state, 155-156 outlining your tactics, 156-160 awareness tactics examples, 156 developer relations high-level quarterly plan, 159 proficiency tactics examples, 157 success tactics examples, 158 usage tactics examples, 157 developer programs, 185-198 breadth and depth analysis for, 185 breadth programs, 192-197 credit programs, 197 hackathons, 194 meetups and community, 192 online videos and streaming, 196 speaking at events and event sponsorships, 194 support, forums, and StackO‐ verflow, 196 train-the-trainer and ambassa‐ dor programs, 195 depth programs, 187-192 design sprints, 191 early access/beta program, 188 top partner program, 187 measuring, 197 developer relations, 143 Developer Relations core activities, 185 developer resources, 163-184 API documentation, 163-172 code samples and snippets, 172-175 Index | 209 community contribution, 182-183 development tools, 179 frameworks, 177 office hours, 181 rich media, 180 videos, 180 software development kits (SDKs), 175-177 webinars and online training, 182 developer SDKs (see software devel‐ opment kits) developers APIs for external developers first, internal developers second, 5 APIs for internal developers first, external developers second, 4 communicating with about API changes, 129 removal of fields or endpoints, 132 rate limits and, 110-112 trying APIs without signing up, 49 disk I/O, 83 documentation for APIs, 49, 163-172 changelog, 170 frequently asked questions, 168 Getting Started guides, 163 landing page, 169 reference documentation, 165 terms of service (ToS), 171 tutorials, 167 E early access/beta program, 188 edge caching, 87 error handling and exponential back‐ off in SDKs, 115 errors HTTP status codes in MyFiles API specification (example), 71 meaningful, 52-55 actionable errors and recom‐ mended error codes, 52 grouping errors into high-level categories, 53 210 | Index organizing into status codes, headers, and machinereadable and humanreadable codes, 54 event objects for MyFiles API techni‐ cal spec (example), 75 event-driven APIs, 19-25 comparison of different types, 24 HTTP Streaming, 23-24 pros and cons for MyFiles API (example), 74 WebHooks, 19-22 WebSockets, 22-23 events (developer), 194 Events API (Slack), 91 evolving API design, 90-97 adding new API methods, 92 best practices, 97 introducing new data access pat‐ terns, 90 new options to filter results, 95 supporting bulk endpoints, 95 explicit-version strategy, 134-138 exponential back-off, 115 extensibility of APIs, 56 F Facebook, ToS violations, 41 failures and retries (WebHooks), 20 feedback, getting on API specifica‐ tion, 77-79 fields in API responses, filtering, 96 filtering results, providing options for, 95 firewalls, WebHooks and, 21 fixed-window counter (rate-limiting), 107 Flannel (Slack), 88 forums, 196 frameworks, 177 frequently asked questions (FAQ), 168 function names, versioned, 136 G geographical distribution (develop‐ ers), 149 GET method (HTTP), 10 (see also CRUD operations) Getting Started guides, 163 GitHub, 5 addressing scalability challenges, 91 archiving a repository, HTTP request for, 12 OAuth scope headers in API response, 35 rate-limit response header, 111 rate-limiting at, 112 Gmail phishing attack on, 40 thin WebHook message payload, 45 Google Cloud Platform (GCP), 83 Google Developer Groups (GDG), 192 Google Hangouts, versioning case study, 140 GraphQL, 6, 14-18, 91 advantages over REST and RPC, 16 comparison to REST and RPC APIs, 18 Object Field deprecation, 133 pros and cons for MyFiles API (example), 66 gRPC, 14 gzip compression, using in SDKs, 114 custom rate-limit response head‐ ers X-RateLimit-Limit, 111 X-RateLimit-Remaining, 111 X-RateLimit-Reset, 111 organizing errors into, 54 specifying API versions in, 135 X-Frame-Options, 40 HTTP methods CRUD operations and REST con‐ ventions, 11 in REST APIs, 10 in REST, RPC, and GraphQL APIs, 18 in RPC-style APIs, 13 HTTP status codes description for errors in MyFiles API specification (example), 71 in REST APIs, 10 indicating redirection for moved/ moving resources, 135 organizing errors into, 54 returning for rate limits, 110 HTTP Streaming, 23-24 comparison with WebHooks and WebSockets, 24 pros and cons for MyFiles eventdriven API (example), 74 HTTPs endpoints (OAuth), 39 human-readable errors, 52 H I hackathons, 194 hackers, 145 hash-based message authentication code (HMAC), 43 Hello World exercise, 163 hobbyist developers, 144 horizontal scaling, 85 HTTP, 9 (see also request–response APIs) in RPC-style APIs, 13 HTTP headers custom OAuth headers X-Accepted-OAuth-Scopes, 35 X-OAuth-Scopes, 35 ID as cursor, 101 identity (developers), 147 iframes, rendering of authorization screen, disallowing, 40 impact statement for MyFiles API (example), 63 scenario 2, 73 indexes (database), 86 integrated development environ‐ ments (IDEs), 148 interface definition language, 122-126 interface description language (IDL), 122 interfaces, 1 Index | 211 J JavaScript object notation (JSON), 122 JSON responses in REST APIs, 11 JSON web APIs, 122-126 describing and validating requests, 125 describing and validating respon‐ ses, 123 K key indicators status report (devel‐ oper funnel), 155 key performance indicators (KPIs), connecting to developer activities, 160 L landing page for API documentation, 169 load testing, 84 logging changelog, 170 use in troubleshooting developer issues, 55 M machine-readable error codes, 52 Macys.com responsive checkout, 78 MAJOR, MINOR, and PATCH ver‐ sions, 137 managing change (see change, man‐ aging) market potential (developer funnel indicators), 155 market size, 149 measurements of developer activities, 160 measuring developer programs, 197 meetups and community, 192 memory bottlenecks, 83 methods, adding to APIs, 92 MINOR versions, 137 mocking data for interactive user testing, 78 Mutual TLS (Transport Layer Secu‐ rity), 44 212 | Index N network I/O, 83 noise (in WebHooks), 21 non-CRUD operations in REST APIs, 12 O OAuth, 28-42, 50 benefits of, 29 best practices, 38 listing and revoking authoriza‐ tions, 37 scopes, 32 Slack's move to granular OAuth scopes, 34 selection for use in MyFiles API (example), 66 token and scope validation, 34 token expiry and refresh tokens, 35 token generation, 30 objective key results (OKRs), 159 office hours, 181 offset-based pagination, 97 advantages and disadvantages, 98 opaque strings as cursor, 101 OpenAPI, 125 order filters, 96 P paginating APIs, 97-102 best practices, 102 cursor-based pagination, 99-101 advantages and disadvantages, 100 choosing cursor contents, 101 offset-based pagination, 97 advantages and disadvantages, 98 pagination support in developer SDKs, 114 partner engineers, 187 PATCH method (HTTP), 10 (see also CRUD operations) PATCH versions, 137 personally identifiable information (PII), 55 phishing attacks using misleading application names, 40 platform of choice (developers), 148 polling, 19 solving as API scaling problem in REST APIs, 90 WebHooks vs., 19 POST method (HTTP), 10 (see also CRUD operations) problem and impact statement for MyFiles API (example), 63 scenario 2, 73 professional developers, 146 proficiency (developer), 148 proficiency tactics examples, 157 profiling code, 83 programming languages, 148 implementing code snippets in, 174 PUT method (HTTP), 10 (see also CRUD operations) Q quarterly plan for developer relations, 159 R rate-limit response headers, 110 rate-limiting APIs, 102-114 best practices, 112 implementation strategies, 105-110 fixed-window counter, 107 sliding-window counter, 108 token bucket algorithm, 105 rate limits and developers, 110-112 documenting rate limits, 111 rate-limit status API, 111 rate-limiting policy, 103 Slack's rate-limiting, lessons learned from, 113 Stripe's rate-limiting strategies, 104 Read method, 10 (see also CRUD operations) read/write scopes, 32 Real-Time Messaging API, 91 reference apps, 173 reference documentation, 165 refresh tokens, 36 one-time-use, 39 remote procedure calls (RPCs), 13 (see also RPC APIs) removing endpoints or fields from APIs, 132 replay attacks, 43 request logs, providing for develop‐ ers, 55 requests adding request parameters to con‐ trol output, 131 describing and validating, 125 request parameters in version schemes, 136 request–response APIs, 9-19 comparison of different types, 18 GraphQL, 14-17 REST, 10-13 RPC, 13-14 resources (in REST APIs), 10 showing relationships between, 11 responses adding response fields, 131 describing and validating, 123-125 REST APIs, 10-13 comparison to RPC and GraphQL, 18 CRUD operations, HTTP verbs, and REST conventions, 11 general rules for, 10 non-CRUD operations, 12 payload creep, 17 polling as scaling problem, solv‐ ing, 90 pros and cons for MyFiles API (example), 66 showing relationships among resources, 11 retries (WebHooks), 20 rich media, 180-181 rich site summary (RSS) feed, adding to changelog, 170 RPC APIs, 13-14 Index | 213 comparison to REST and GraphQL, 18 general rules for, 13 HTTP request to Slack API, 13 pros and cons for MyFiles API (example), 66 Slack, Conversations APIs, 13 using protocols other than HTTP, 14 RSpec test using JSON Schema speci‐ fication, 125 S sandboxes and API testers, 179 scaling APIs, 81-116 evolving your API design, 90-97 adding new API methods, 92 best practices, 97 introducing new data access patterns, 90 providing new options to filter results, 95 supporting bulk endpoints, 95 providing developer SDKs, 114-116 caching frequently used data, 115 error handling and exponen‐ tial back-off, 115 pagination support, 114 rate-limiting support, 114 SDK best practices, 115 using gzip compression in SDKs, 114 scaling throughput, 82-90 adding computing resources, 85 best practices, 89 caching, 87 database indexes, 86 doing expensive operations asynchronously, 89 finding bottlenecks, 82 using pagination, 97-102 best practices, 102 cursor-based pagination, 99-101 offset-based pagination, 97 214 | Index using rate-limiting, 102-114 best practices, 112 implementation strategies, 105-110 rate limits and developers, 110-112 scopes (OAuth), 32 for sensitive information, 39 for use in MyFiles API (example), 66 in MyFiles API (example) scopes, operations, and resour‐ ces, 67 Slack's move to granular OAuth scopes, lessons learned from, 34 SDKs (see software development kits) search filters, 96 search operations in REST APIs, 12 security, 27-46 authentication and authorization, 27 for WebHooks, 20 OAuth, 28-42 best practices, 38 listing and revoking authoriza‐ tions, 37 scopes, 32 token and scope validation, 34 token expiry and refresh tokens, 35 token generation, 30 WebHooks, 42-45 semantic versioning specification (SemVer), 137 server-sent events (SSE), streaming data via, 24 short-lived authorization codes (OAuth), 39 short-term targets and market poten‐ tial (developer funnel indicators), 155 signatures (WebHook), 43 Slack APIs, 4 adding new API methods, 92 addressing scalability challenges with Events API, 91 API Metadata, 126 app credentials of Slack app with verification token, 42 changelog, 170 Conversations API, 132 supporting bulk operations, 95 developer segmentation for Slack, 149 early access/beta program, 189 Flannel, application-level edge cache, 87 inconsistency in, 118 long-lived tokens, 37 missing field on message pay‐ loads, 127 move to granular OAuth scopes, lessons learned from, 34 rate-limiting, lessons learned from, 113 RPC-style web API, 13 translation layer to maintain backward compatibility, 57 WebSocket-based real-time mes‐ saging API, 22 sliding-window counter (ratelimiting), 108 snippets (code), 174 software as a service (SaaS) compa‐ nies, 4 software development kits (SDKs), 50, 148, 175-177 developer SDKs, 114-116 best practices, 115 caching frequently used data, 115 error handling and exponen‐ tial back-off, 115 pagination support, 114 rate-limiting support, 114 using gzip compression, 114 maintaining, 178 SoundCloud API, value proposition, 151 speaking at developer events, 194 specification (spec), writing for an API, 68-72 MyFiles API WebHooks (exam‐ ple), 74-77 SQL databases, queries based on cur‐ sor values, 99 Stack Overflow, 196 Stackdrivers, 83 stakeholders, reviewing API specifi‐ cation with, 77 state parameter support (OAuth), 38 streaming, 196 Stripe online testing of API by develop‐ ers without signing up, 49 rate-limiting, 104 value proposition, 151 versioning case study, 139 subresources in APIs, 11 success tactics examples, 158 support for developers, 196 T task queues, 89 TCP (Transport Control Protocol), 22 technology architecture, selecting scenario 1 for MyFiles API (exam‐ ple), 65 scenario 2 for MyFiles API (exam‐ ple), 74 terms of service (ToS) violations of Facebook ToS, 41 writing, 171 testing automated, 120-126 describing and validating requests, 125 describing and validating responses, 123-125 sandboxes and API testers, 179 Thrift, 14 throughput, scaling, 82-90 adding computing resources, 85 best practices, 89 caching, 87 database indexes, 86 doing expensive operations asyn‐ chronously, 89 finding bottlenecks, 82 the Time to Hello World (TTHW), 164 Index | 215 timestamps, using as cursors, 101 TLS (Transport Layer Security), 44 token bucket algorithm (ratelimiting), 105 tokens (OAuth) and scope, validation of, 34 expiry and refresh tokens, 35 generation of, 30 Slack's long-lived tokens, 37 top partner program, 187 train-the-trainer and ambassador programs, 195 transformations between versions, 137 Transport Control Protocol (see TCP) transport patterns for MyFiles API (example), 65 troubleshooting, making easy for developers, 52-56 building tooling, 55 meaningful errors, 52-55 providing troubleshooting tools, 179 tutorials for APIs, 49, 167 Twitch, deprecation of an API, 59 Twitter, 90 cursor-based pagination, 99 U Uber developers and rate-limiting, 109 Unix timestamp as cursor, 99 Update method, 10 (see also CRUD operations) URI components, specifying versions in, 135 URIs, specification for MyFiles API (example), 70 usage tactics examples, 157 user interfaces (UIs), 88 user stories (key), outlining scenario 1 for MyFiles API (exam‐ ple), 64 scenario 2 for MyFiles API (exam‐ ple), 73 users, focusing on in API design, 2 216 | Index users.conversations API method (Slack), 94 V value proposition, distilling for your API, 151 verification tokens, 42 versioning APIs, 57, 133-141 additive-change strategy, 133 case study, Google Hangouts, 140 case study, Stripe, 139 explicit-version strategy, 134-138 policies for MAJOR and MINOR changes, 138 process management, 141 vertical scaling, 85 videos creating, 180 online videos and streaming, 196 W WebHooks, 19-22, 90 comparison with WebSockets and HTTP Streaming, 24 considerations for use in MyFiles API (example), 66 MyFiles API Webhooks Spec (example), 74-77 polling vs., 19 pros and cons for MyFiles eventdriven API (example), 74 security, 42-45 best practices, 45 Mutual Transport Layer Secu‐ rity, 44 request signing and WebHook signatures, 43 thin payloads and API retrieval, 44 verification tokens, 42 supporting, additional complexi‐ ties added by, 20 webinars and online training, 182 WebSockets, 22-23, 90 comparison with WebHooks and HTTP Streaming, 24 pros and cons for MyFiles eventdriven API (example), 74 Y YouTube API, value proposition, 151 X XML responses, REST APIs, 11 Index | 217 About the Authors Brenda Jin is an entrepreneur and software engineer.


pages: 40 words: 11,939

9 Lessons in Brexit by Ivan Rogers

Boris Johnson, continuous integration, imperial preference, non-tariff barriers

Forgive me for pointing out that, as some of us forecast well over two years ago, it did not turn out like that. And that the Brussels theologians actually exhibited rather more flexibility than the key Member States when it came to the crunch. And that not a peep was heard from the titans of corporate Europe, except to back very robustly the position in capitals that the continued integrity of the Single Market project was vastly more important to them than the terms of a framework agreement with the UK. A position which won’t change during the trade negotiations ahead either. The “No deal +” fantasy is that if we just had the guts to walk away, refuse to sign the Withdrawal Agreement with the backstop in it, and withhold a good half of the money the Prime Minister promised this time last year, capitals, suddenly realising we were serious, would come running for a series of mini deals which assured full trading continuity in all key sectors on basically unchanged Single Market and Customs Union terms.


pages: 444 words: 118,393

The Nature of Software Development: Keep It Simple, Make It Valuable, Build It Piece by Piece by Ron Jeffries

Amazon Web Services, anti-pattern, bitcoin, business cycle, business intelligence, business logic, business process, c2.com, call centre, cloud computing, continuous integration, Conway's law, creative destruction, dark matter, data science, database schema, deep learning, DevOps, disinformation, duck typing, en.wikipedia.org, fail fast, fault tolerance, Firefox, Hacker News, industrial robot, information security, Infrastructure as a Service, Internet of things, Jeff Bezos, Kanban, Kubernetes, load shedding, loose coupling, machine readable, Mars Rover, microservices, Minecraft, minimum viable product, MITM: man-in-the-middle, Morris worm, move fast and break things, OSI model, peer-to-peer lending, platform as a service, power law, ransomware, revision control, Ruby on Rails, Schrödinger's Cat, Silicon Valley, six sigma, software is eating the world, source of truth, SQL injection, systems thinking, text mining, time value of money, transaction costs, Turing machine, two-pizza team, web application, zero day

Build Server as Attack Vector Any widely used piece of server software will be used for an attack. That includes build servers such as Jenkins, Bamboo, or GoCD. At least one major software vendor was attacked by means of the build environment. The attacker compromised a plugin to the vendor’s continuous integration server. The plugin injected code that targeted a well-known customer of this vendor (relayed in personal communication to the author). This vendor kept its libraries in a controlled artifact repository but had overlooked the plugins to the build system itself. Those were downloaded directly from the Net.

As a result, a containerized application may still have operating system vulnerabilities that IT patched days or weeks ago. The solution is to treat container images as perishable goods. You need an automated build process that creates new images from an upstream base and your local application code. Ideally this comes from your continuous integration pipeline. Be sure to configure timed builds for any application that isn’t still under active development, though. Configured Passwords Passwords are the Brazil nut of application security; every mix has them, but nobody wants to deal with them. There’s obviously no way that somebody can interactively key in passwords every time an application server starts up.

This overview won’t be enough for you to pick up Chef and start writing deployment recipes, but it will put Chef and tools like it into context so we know what to do with our ingredients. The first tool of interest is the build pipeline. It picks up after someone commits a change to version control. (Some teams like to build every commit to master; others require a particular tag to trigger a build.) In some ways, the build pipeline is an overgrown continuous integration (CI) server. (In fact, build pipelines are often implemented with CI servers.) The pipeline spans both development and operations activities. It starts exactly like CI with steps that cover development concerns like unit tests, static code analysis, and compilation. See the figure that follows.


pages: 283 words: 78,705

Principles of Web API Design: Delivering Value with APIs and Microservices by James Higginbotham

Amazon Web Services, anti-pattern, business intelligence, business logic, business process, Clayton Christensen, cognitive dissonance, cognitive load, collaborative editing, continuous integration, create, read, update, delete, database schema, DevOps, fallacies of distributed computing, fault tolerance, index card, Internet of things, inventory management, Kubernetes, linked data, loose coupling, machine readable, Metcalfe’s law, microservices, recommendation engine, semantic web, side project, single page application, Snapchat, software as a service, SQL injection, web application, WebSocket

The reach of web APIs goes beyond a few internal systems and partners. Today’s web-based APIs connect organizations to their customers, partners, and workforce using the standards of the web. Hundreds of libraries and frameworks exist to make it cheap and fast to deliver APIs to a marketplace or for internal use. Continuous integration and delivery (CI/CD) tools make it easier than ever to build automation pipelines to ensure APIs are delivered with speed and efficiency. Yet, the biggest challenge for today’s API programs continues to be successfully designing APIs that can be understood and integrated by developers in a consistent and scalable fashion.

While Git supports writing scripts to react to these kinds of events within a source code repository, GitHub was one of the first vendors to turn these script-based hooks into Webhooks. Any individual or organization hosting their code with GitHub could be notified, via an HTTP-based POST, when new code was available and trigger a new build process. Over time, continuous integration and delivery (CI/CD) tools that were previously restricted to on-premises installation could now be offered via a software-as-a-service (SaaS) model. These solutions would be granted permission to receive the Webhook-based notification and start a new build process. This one async API notification ultimately created an entire SaaS market of hosted CI/CD tools.


Machine Learning Design Patterns: Solutions to Common Challenges in Data Preparation, Model Building, and MLOps by Valliappa Lakshmanan, Sara Robinson, Michael Munn

A Pattern Language, Airbnb, algorithmic trading, automated trading system, business intelligence, business logic, business process, combinatorial explosion, computer vision, continuous integration, COVID-19, data science, deep learning, DevOps, discrete time, en.wikipedia.org, Hacker News, industrial research laboratory, iterative process, Kubernetes, machine translation, microservices, mobile money, natural language processing, Netflix Prize, optical character recognition, pattern recognition, performance metric, recommendation engine, ride hailing / ride sharing, selection bias, self-driving car, sentiment analysis, speech recognition, statistical model, the payments system, web application

Of course, there is some overhead involved in converting the different pieces of our ML workflow into an organized pipeline. In this section, we’ll look at some variations and extensions of the Workflow Pipeline design pattern: creating containers manually, automating a pipeline with tools for continuous integration and continuous delivery (CI/CD), processes for moving from a development to production workflow pipeline, and alternative tools for building and orchestrating pipelines. We’ll also explore how to use pipelines for metadata tracking. Creating custom components Instead of using pre-built or customizable TFX components to construct our pipeline, we can define our own containers to use as components, or convert a Python function to a component.

Building pipelines to automate these steps enables more efficient workflows and repeatable processes that improve future model development, and allows for increased agility in solving problems that arise. Today, open source tools like Kubeflow provide this functionality and many large software companies have developed their own end-to-end ML platforms, like Uber’s Michelangelo or Google’s TFX, which are also open source. Successful operationalization incorporates components of continuous integration and continuous delivery (CI/CD) that are the familiar best practices of software development. These CI/CD practices are focused on reliability, reproducibility, speed, security, and version control within code development. ML/AI workflows benefit from the same considerations, though there are some notable differences.

, Data and Model Tooling, Concept, Saving predictions Cloud AI Platform Pipelines, Solution, Running the pipeline on Cloud AI Platform Cloud AI Platform Predictions, Lambda architecture Cloud AI Platform Training, Solution, Running the pipeline on Cloud AI Platform Cloud Build, Integrating CI/CD with pipelines Cloud Composer/Apache Airflow, Scheduled retraining Cloud Dataflow, Lambda architecture Cloud Functions, Triggers for retraining, Integrating CI/CD with pipelines Cloud Run, Create web endpoint, Other serverless versioning tools Cloud Spanner, Cached results of batch serving clustering, Models and Frameworks clustering models, Models and Frameworks CNN, Images as tiled structures, Why It Works-Why It Works cold start, Problem, Cold start combinatorial explosion, Grid search and combinatorial explosion completeness, Data Quality components, definition of, Solution computer vision, Computer Vision concept drift, Problem, Estimating retraining interval confidence, Inputs with overlapping labels, When human experts disagree, Saving predictions confusion matrix, Problem, Evaluating model performance consistency, Data Quality-Data Quality containers, Design Pattern 25: Workflow Pipeline, Solution, Why It Works context language models, Context language models-Context language models(see also BERT, Word2Vec) Continued Model Evaluation design pattern, Design Patterns for Resilient Serving, Design Pattern 18: Continued Model Evaluation-Estimating retraining interval, Model versioning with a managed service, Responsible AI, Automating data evaluation, Pattern Interactions Continuous Bag of Words (see CBOW) continuous evaluation, Continuous evaluation-Continuous evaluation continuous integration and continuous delivery (see CI/CD) convolutional neural network (see CNN) Coral Edge TPU, Phase 1: Building the offline model counterfactual analysis, Counterfactual analysis and example-based explanations-Counterfactual analysis and example-based explanations counterfactual reasoning, Capturing ground truth cryptographic algorithms, Cryptographic hash custom serving function, Custom serving function D DAG, Why It Works, Apache Airflow and Kubeflow Pipelines Darwin, Charles, Genetic algorithms data accuracy, Data Quality data analysts, Roles data augmentation, Data augmentation data collection bias, Before training, Before training data distribution bias, Problem data drift, Data Drift-Data Drift, Problem, Estimating retraining interval, Continuous evaluation for offline models, Problem data engineers, Roles, Scale, Solution data parallelism, Solution-Solution, Synchronous training, Why It Works, Model parallelism data preprocessing, Data and Feature Engineering(see also data transformation, feature engineering) data representation, Data Representation Design Patterns-Data Representation Design Patterns data representation bias, Before training data scientistsrole of, Roles, Multiple Objectives-Multiple Objectives, Why It Works, Problem tasks of, Problem, Problem, Solution data transformation, Data and Feature Engineering data validation, Data and Feature Engineering, Data validation with TFX data warehouses, Embeddings in a data warehouse-Embeddings in a data warehouse dataset-level transformations, Efficient transformations with tf.transform datasets, definition of, Data and Feature Engineering Datastore, Cached results of batch serving decision trees, Models and Frameworks, Data Representation Design Patterns-Data Representation Design Patterns, Decreased model interpretability, Choosing a model architecture, Typical Training Loop, Solution Deep Galerkin Method, Data-driven discretizations-Unbounded domains deep learning, Models and Frameworks-Models and Frameworks, Multimodal feature representations and model interpretability deep neural network (see DNN model) default, definition of, Model versioning with a managed service Dense layers, Solution, Using images with metadata design patterns, definition of, What Are Design Patterns?


pages: 761 words: 80,914

Ansible: Up and Running: Automating Configuration Management and Deployment the Easy Way by Lorin Hochstein

Amazon Web Services, cloud computing, continuous integration, Debian, DevOps, domain-specific language, don't repeat yourself, general-purpose programming language, Infrastructure as a Service, job automation, machine readable, MITM: man-in-the-middle, pull request, side project, smart transportation, web application

Push Docker images up from your local machine to the registry. Pull Docker images down to your remote hosts from the registry. Start up Docker containers on the remote hosts, passing in any configuration information to the containers on startup. You typically create your Docker image on your local machine, or on a continuous integration system that supports creating Docker images, such as Jenkins or CircleCI. Once you’ve created your image, you need to store it somewhere it will be convenient for downloading onto your remote hosts. Docker images typically reside in a repository called a registry. The Docker project runs a registry called Docker Hub, which can host both public and private Docker images, and where the Docker command-line tools have built-in support for pushing images up to a registry and for pulling images down from a registry.

Ansible does contain a module for building Docker images, called docker_image. However, that module has been deprecated because building images isn’t a good fit for a tool like Ansible. Image building is part of the build process of an application’s lifecycle; building Docker images and pushing them up an image registry is the sort of thing that your continuous integration system should be doing, not your configuration management system. Deploying the Dockerized Application Note We use the docker module for deploying the application. As of this writing, there are several known issues with the docker module that ships with Ansible. The volumes_from parameter does not work with recent versions of Docker.


pages: 422 words: 86,414

Hands-On RESTful API Design Patterns and Best Practices by Harihara Subramanian

blockchain, business logic, business process, cloud computing, continuous integration, create, read, update, delete, cyber-physical system, data science, database schema, DevOps, disruptive innovation, domain-specific language, fault tolerance, information security, Infrastructure as a Service, Internet of things, inventory management, job automation, Kickstarter, knowledge worker, Kubernetes, loose coupling, Lyft, machine readable, microservices, MITM: man-in-the-middle, MVC pattern, Salesforce, self-driving car, semantic web, single page application, smart cities, smart contracts, software as a service, SQL injection, supply-chain management, web application, WebSocket

Precisely speaking, in order to decisively reap all of the originally envisioned benefits (business, technical, and user), applications are being designed through microservices, developed using one of the integrated development platforms and frameworks, and deployed in cloud environments. The microservices-based software design is faster than the traditional, agile programming models, and methods are made available in plenty to speed up the process of software construction. The continued adoption of enterprise DevOps tools and processes accelerates the continuous integration, delivery, and deployment of software applications, and the emergence of site-reliability engineering (SRE) concepts bring forth additional automation and acceleration for software engineering. Hence, the future belongs to cloud-native applications. That is, all kinds of enterprise, mobile, social, embedded, transactional, operational, and analytical applications are being built as cloud-native applications.

This way, little wastage of time is guaranteed for legacy modernization, which happens to be a time-consuming job. Once a microservice is readied, it has to be taken to one or more software repositories (public and/or private) to enable access and usage by software developers and testers. This kind of centralized storage also contributes for source-code version control. There are continuous integration (CI) solutions to integrate, build, and test software services. With the overwhelming adoption of the enterprise DevOps concept, business applications get continuous and consistent integration, delivery, and deployment in order to take applications and services to their end users quickly, without any risk.


Service Design Patterns: Fundamental Design Solutions for SOAP/WSDL and RESTful Web Services by Robert Daigneau

Amazon Web Services, business intelligence, business logic, business process, continuous integration, create, read, update, delete, en.wikipedia.org, fault tolerance, loose coupling, machine readable, MITM: man-in-the-middle, MVC pattern, OSI model, pull request, RFC: Request For Comment, Ruby on Rails, software as a service, web application

Git has the powerful concept of submodules. • Enforce contracts with every change to a service: Contract tests should be run with every change to a service, regardless of whether the change occurs to the API or in the internal service logic. Automated, self-checking tests can be incorporated into a continuous integration pipeline, where they will be executed with every check-in of service code. • Modifying contracts: Tests written in the early stages of a service’s design and development are different from those written once a service API has been published. The former helps to shape the API, but may be modified as service and client owners negotiate the scope and composition of the API.

ConsumerDriven Contracts 256 C HAPTER 7 W EB S ERVICE E VOLUTION Example: A Consumer Contract for Service Behavior Implemented in C# and NUnit This example shows a simple consumer contract for a news service Resource API (38). The contract has been written by developers of one of the service’s clients, and then given to the service owner who has incorporated it, together with other consumer contracts, in the service’s continuous integration pipeline to form a consumer-driven contract. [TestFixture] public class NewsServiceConsumerContract { private IIntegrationContext context; private HttpResponseMessage response; [SetUp] public void Init() { context = CreateContext(); string xml = @"<entry xmlns=""http://www.w3.org/2005/Atom""> <title>Lilliput Siezes Blefuscudian Fleet</title> <id>urn:uuid:897B5900-7805-4A61-BC63-03691EEE752D</id> <updated>2011-06-01T06:30:00Z</updated> <author><name>Jonathan Swift</name></author> <content>Lilliput's Man-Mountain this morning...


pages: 161 words: 30,412

Creating Development Environments With Vagrant - Second Edition by Michael Peacock

Amazon Web Services, cloud computing, continuous integration, Debian, domain-specific language, web application

He is the author of Creating Development Environments with Vagrant, PHP 5 Social Networking, PHP 5 E-Commerce Development, Drupal 7 Social Networking, and Selling online with Drupal e-Commerce and Building Websites with TYPO3, all by Packt Publishing. The other publications Michael has been involved in include Advanced API Security, Mobile Web Development, Jenkins Continuous Integration Cookbook, and Drupal for Education and E-Learning; for these he acted as a technical reviewer. Michael has also presented at a number of user groups and technical conferences, including PHP UK Conference, Dutch PHP Conference, ConFoo, PHPNE, PHPNW, and Could Connect Santa Clara. You can follow Michael on Twitter (@michaelpeacock), or find out more about him through his website (www.michaelpeacock.co.uk).


Real-World Kanban by Mattias Skarin

call centre, continuous integration, Great Leap Forward, Kanban, loose coupling, pull request

The first step was to have the two development leads check in with each other on a weekly call. We also kicked off a developer exchange program where developers from other teams came and learned how we worked. We covered topics such as branching, TDD, design patterns, integrated development environment (IDE) setup, continuous integration, and testing environments. After all the developers went through the exchange program, we were confident we would be able to work The team, working together together as one team to maintain a higher stan- to maintain higher standards. dard of quality. report erratum • discuss How We Continuously Improved • 79 How We Continuously Improved Everything so far was on the defense as we fought problems as they cropped up.


Large Scale Apps with Vue 3 and TypeScript by Damiano Fusco

continuous integration, en.wikipedia.org, node package manager, place-making, single source of truth, source of truth

MOCHA Testing... configsMap ✓ instance should have "mock" key ✓ instance should have "local" key ✓ instance should have "beta" key ✓ instance should have "live" key config ✓ instance should have "global" section ✓ instance should have "items" section items ✓ section should have "apiUrls" property apiUrls ✓ section should have "fetchItems" property HttpClient.get ✓ should succeed and return data (102ms) HttpClient.get ✓ should reject and return 400 (103ms) Item.component.vue ✓ renders an Item correctly ✓ has expected css class when selected is false ✓ has selected css class when selected is true 13 passing (231ms) MOCHA Tests completed successfully NOTE: In a continuous integration (CI) setup, you might have a task that will run only the text-xyz command that is related to the CI environment. If you CI builds for both production (live) and QA (beta) you could add an additional entry that calls both of them at once: i.e. npm run test-beta && npm run test-live Now that we have verified our configuration, we need to also update the other commands like serve/build etc.


pages: 648 words: 108,814

Solr 1.4 Enterprise Search Server by David Smiley, Eric Pugh

Amazon Web Services, bioinformatics, cloud computing, continuous integration, database schema, domain-specific language, en.wikipedia.org, fault tolerance, Firefox, information retrieval, Ruby on Rails, SQL injection, Wayback Machine, web application, Y Combinator

This approach makes your function queries more comparable by using a common baseline, as the values will fit within the same range. If your data changes in ways causing you to alter the constants in your function queries, then consider implementing a periodic automated test of your Solr data to ensure that the data fits within expected bounds. A Continuous Integration (CI) server might be configured to do this task. An approach is to run a search simply sorting by the data field in question to get the highest or lowest value. As you tweak the boost, you'll want to look at the proportion of the function query's score contribution (which includes the raw function query, the boost, and the queryNorm) relative to the total score.

See coord collapse.facet, field collapsing 192 collapse.field, field collapsing 192 collapse.info.doc, field collapsing 193 collapse.maxdocs, field collapsing 193 collapse.threshold, field collapsing 193 collapse.type, field collapsing 192 combined index 32 CommonsHttpSolrServer 235 complex systems, tuning about 271 CPU usage 272 memory usage 272 scale deep 273 scale high 273 scale wide 273 system changes 272 components about 111, 159 solrconfig.xml 159 compressed, field option 41 configuration files, Solr <requestHandler> tag 25 solrconfig.xml file 25 standard request handler 26 Configuration Management. See CM ConsoleHandler 204 Content Construction Kit 252 Content Management System. See CMS Continuous Integration. See CI coord 112 copyField directive about 46 uses 46 CoreDescriptor classes 231 core, managing 209, 210 count, Stats component 189 CPU usage 272 cron 289 CSV, sending to Solr about 72 configuration options 73, 74 curl using, to interact with Solr 66, 68 [ 302 ] Download at Boykma.Com This material is copyright and is licensed for the sole use by William Anderson on 26th August 2009 4310 E Conway Dr.


pages: 461 words: 106,027

Zero to Sold: How to Start, Run, and Sell a Bootstrapped Business by Arvid Kahl

business logic, business process, centre right, Chuck Templeton: OpenTable:, cognitive load, content marketing, continuous integration, coronavirus, COVID-19, crowdsourcing, domain-specific language, financial independence, functional programming, Google Chrome, hockey-stick growth, if you build it, they will come, information asymmetry, information retrieval, inventory management, Jeff Bezos, job automation, Kanban, Kubernetes, machine readable, minimum viable product, Network effects, performance metric, post-work, premature optimization, risk tolerance, Ruby on Rails, sentiment analysis, side hustle, Silicon Valley, single source of truth, software as a service, solopreneur, source of truth, statistical model, subscription business, sunk-cost fallacy, supply-chain management, the long tail, trickle-down economics, value engineering, web application

Many web development frameworks like Ruby on Rails and Elixir/Phoenix have this feature built-in, but you need to know how to use it. Making bidirectional migrations a part of your release flow from the start will save you from a lot of potential trouble further down the road. Automate the process. You will benefit immensely from removing manual steps from the release process. Continuous Integration, the concept of automating build and tests, makes releasing extremely easy and manageable. This level of automation makes your business more sellable as well: if a developer you hire (or the one that replaces you after you sold the company) can release a new version of the software at the push of a button, this will net you a premium when your business gets acquired.

Competition, a changing regulatory landscape, human error, bad press, changing preferences can all derail a working business. That’s why we always kept an ear on the pulse of our community. We screened public forums and groups for complaints and new developments. In the SaaS world, we have Continuous Integration. I think we should also have the concept of Continuous Validation. Regularly and frequently, assess where you are in terms of still being in touch with the market. Are you still solving their most painful problem? Did that change? Are there new issues that you didn’t encounter before? That will keep you on your toes—and keep your customers happy.


Kanban in Action by Marcus Hammarberg, Joakim Sunden

Buckminster Fuller, business logic, call centre, continuous integration, en.wikipedia.org, fail fast, index card, Kaizen: continuous improvement, Kanban, Lean Startup, performance metric, place-making, systems thinking, the scientific method, Toyota Production System, transaction costs, Two Sigma

Avoiding rework can have a big effect on flow and cycle time. It’s no coincidence that one of the key principles of Lean software development[4] is to build quality in from the start. A good way to accomplish this is to invest in technical practices such as pair programming, test-driven development, continuous integration, and test automation.[5] 4 Coined by Mary and Tom Poppendieck. 5 More on this in the excellent pair of books Clean Code: A Handbook of Agile Software Craftsmanship (Prentice Hall, 2008, http://amzn.com/0132350882) and The Clean Coder: A Code of Conduct for Professional Programmers (Prentice Hall, 2011, http://amzn.com/0137081073) by Robert C.

Andon boards animals as avatars annotating cards Aptitud B balancing flow speed and work items batches, working with smaller BDD (behavior-driven development) Bellware, Scott big daily boredom bugs giving precedence to types of work items business impact, 2nd C call centers cartoons on cards celebrations frequent flyer miles WIP limit CFD (cumulative flow diagram) data needed for drawing diagram overview reading diagram charts. See diagrams. Coaching Kata Cockburn, Alistair coin flipping game. See also Pass the Pennies. component teams constraints. See also Theory of Constraints. context switching effects of excess WIP wastes in software development continuous improvement continuous integration converting story points CONWIP coordination cost core practices cost, balancing with scope and time counting down as progress cumulative flow diagram. See CFD. current situation customers defining favoring some over others importance of in planning promoting preferences on work items value and D Daily Kata daily standup best practices focused on time regular short length, 2nd blocked items focusing on smells focusing on work instead of people multiteams prioritization simplifying visualizations spontaneous kaizen walking board work not displayed on board dates, as workflow metric days, avoiding use of in estimates debt, technical decisions, facilitating using cards defects Defects class delays effects of excess WIP wastes in software development design principles for cards Development column discarded ideas, as metric analyzing capturing overview discussion, as goal of estimation Disneyland wait times double bookkeeping downstream drawings on cards Drive: The Surprising Truth About What Motivates Us due date performance analyzing capturing overview visualizing E electronic cards and tools, 2nd emotional data empty wall space entry/exit criteria for queues estimated data, vs. real data estimating defined need for diminishes No Estimates movement story points advantages of disadvantages of overview techniques Goldilocks line of cards Planning Poker using t-shirt sizes.


pages: 179 words: 43,441

The Fourth Industrial Revolution by Klaus Schwab

"World Economic Forum" Davos, 3D printing, additive manufacturing, Airbnb, Amazon Mechanical Turk, Amazon Web Services, Anthropocene, augmented reality, autonomous vehicles, barriers to entry, Baxter: Rethink Robotics, bitcoin, blockchain, Buckminster Fuller, call centre, circular economy, clean water, collaborative consumption, commoditize, conceptual framework, continuous integration, CRISPR, cross-border payments, crowdsourcing, digital divide, digital twin, disintermediation, disruptive innovation, distributed ledger, driverless car, Edward Snowden, Elon Musk, epigenetics, Erik Brynjolfsson, future of work, global value chain, Google Glasses, hype cycle, income inequality, Internet Archive, Internet of things, invention of the steam engine, job automation, job satisfaction, John Maynard Keynes: Economic Possibilities for our Grandchildren, John Maynard Keynes: technological unemployment, life extension, Lyft, Marc Benioff, mass immigration, megacity, meta-analysis, more computing power than Apollo, mutually assured destruction, Narrative Science, Network effects, Nicholas Carr, nuclear taboo, OpenAI, personalized medicine, precariat, precision agriculture, Productivity paradox, race to the bottom, randomized controlled trial, reshoring, RFID, rising living standards, Sam Altman, Second Machine Age, secular stagnation, self-driving car, sharing economy, Silicon Valley, smart cities, smart contracts, social contagion, software as a service, Stephen Hawking, Steve Jobs, Steven Levy, Stuxnet, supercomputer in your pocket, synthetic biology, TaskRabbit, The Future of Employment, The Spirit Level, total factor productivity, transaction costs, Uber and Lyft, uber lyft, Watson beat the top human players on Jeopardy!, Wayback Machine, WikiLeaks, winner-take-all economy, women in the workforce, working-age population, Y Combinator, Zipcar

Operating in an increasingly complex and disruptive environment requires the intellectual and social agility of the fox rather than fixed and narrow focus of the hedgehog. In practical terms, this means that leaders cannot afford to think in silos. Their approach to problems, issues and challenges must be holistic, flexible and adaptive, continuously integrating many diverse interests and opinions. Emotional intelligence – the heart As a complement to, not a substitute for, contextual intelligence, emotional intelligence is an increasingly essential attribute in the fourth industrial revolution. As management psychologist David Caruso of the Yale Center for Emotional Intelligence has stated, it should not be seen as the opposite of rational intelligence or “the triumph of heart over head – it is the unique intersection of both.”71 In academic literature, emotional intelligence is credited with allowing leaders to be more innovative and enabling them to be agents of change.


pages: 131 words: 41,052

Why Europe Will Run the 21st Century by Mark Leonard

Berlin Wall, Celtic Tiger, continuous integration, cuban missile crisis, different worldview, European colonialism, facts on the ground, failed state, global reserve currency, Global Witness, invisible hand, knowledge economy, mass immigration, non-tariff barriers, North Sea oil, one-China policy, Panopticon Jeremy Bentham, pension reform, reserve currency, Robert Gordon, shareholder value, South China Sea, The Wealth of Nations by Adam Smith, Thomas Malthus, trade liberalization, Washington Consensus

With its current low levels of consumption, and increases predicted at a modest 0.7 per cent per year, Europe may soon reap the rewards of its ‘energy independence’.22 The Power of Integration and Enlargement Possibly the biggest boon to European economies is the European project itself. Europe’s economic strength rests on two pillars: the overall size of its economy which assures it power in the world, and the quality of living standards it can deliver for its citizens. Both have been improved by the continuing integration and enlargement of the European Union. The impact of the creation of the Single Market in 1992 has already been dramatic. The European Commission estimates that European Union’s GDP in 2002 was almost 2 per cent higher than it would have been without the creation of the internal market, while employment was almost 1.5 per cent higher.


pages: 394 words: 118,929

Dreaming in Code: Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software by Scott Rosenberg

A Pattern Language, AOL-Time Warner, Benevolent Dictator For Life (BDFL), Berlin Wall, Bill Atkinson, c2.com, call centre, collaborative editing, Computer Lib, conceptual framework, continuous integration, Do you want to sell sugared water for the rest of your life?, Donald Knuth, Douglas Engelbart, Douglas Engelbart, Douglas Hofstadter, Dynabook, en.wikipedia.org, Firefox, Ford Model T, Ford paid five dollars a day, Francis Fukuyama: the end of history, Free Software Foundation, functional programming, General Magic , George Santayana, Grace Hopper, Guido van Rossum, Gödel, Escher, Bach, Howard Rheingold, HyperCard, index card, intentional community, Internet Archive, inventory management, Ivan Sutherland, Jaron Lanier, John Markoff, John Perry Barlow, John von Neumann, knowledge worker, L Peter Deutsch, Larry Wall, life extension, Loma Prieta earthquake, machine readable, Menlo Park, Merlin Mann, Mitch Kapor, Neal Stephenson, new economy, Nicholas Carr, no silver bullet, Norbert Wiener, pattern recognition, Paul Graham, Potemkin village, RAND corporation, Ray Kurzweil, Richard Stallman, Ronald Reagan, Ruby on Rails, scientific management, semantic web, side project, Silicon Valley, Singularitarianism, slashdot, software studies, source of truth, South of Market, San Francisco, speech recognition, stealth mode startup, stem cell, Stephen Hawking, Steve Jobs, Stewart Brand, Strategic Defense Initiative, Ted Nelson, the Cathedral and the Bazaar, Therac-25, thinkpad, Turing test, VA Linux, Vannevar Bush, Vernor Vinge, Wayback Machine, web application, Whole Earth Catalog, Y2K

So as the date for 0.2 drew closer, the developers began to try to take the work they had been pursuing on their own—like Andi Vajda’s new repository and Andy Hertzfeld’s work on agents—and add it to the main trunk of Chandler code. Most projects today embrace the idea of continuous integration: The programmers always keep their latest code checked in to the main trunk of the code tree, and everyone is responsible for making sure that their new additions haven’t thrown a spanner in the works. Later on, OSAF would end up achieving a higher level of continuous integration, but for 0.2 the process was more like what software-development analysts call “big-bang integration”: all the programmers try to integrate their code at the end, and everything breaks.


Text Analytics With Python: A Practical Real-World Approach to Gaining Actionable Insights From Your Data by Dipanjan Sarkar

bioinformatics, business intelligence, business logic, computer vision, continuous integration, data science, deep learning, Dr. Strangelove, en.wikipedia.org, functional programming, general-purpose programming language, Guido van Rossum, information retrieval, Internet of things, invention of the printing press, iterative process, language acquisition, machine readable, machine translation, natural language processing, out of africa, performance metric, premature optimization, recommendation engine, self-driving car, semantic web, sentiment analysis, speech recognition, statistical model, text mining, Turing test, web application

Python can be used to invoke code for other languages, and there are Python bindings for invoking Python code from other languages. This helps in easy integration of Python code wherever necessary. The most important advantage, though, is that it is very easy to develop Python code and deploy it no matter how complex your codebase might be. If you follow the right continuous integration (CI) processes and manage your Python codebase properly, deployment usually involves updating your latest code and starting the necessary processes in your production environment. It is extremely easy to get proper working code in minimal time, which is often difficult to do with other languages.

Index A Adjective phrase (ADJP) Advanced word vectorization models Adverb phrase (ADVP) Affinity propagation (AP) description exemplars feature matrix K-means clustering, movie data message-passing steps number of movies, clusters AFINN lexicon The American National Corpus (ANC) Anonymous functions Antonyms Application programming interfaces (APIs) Artificial intelligence (AI) Automated document summarization abstraction-based techniques definition elephants extraction-based techniques gensim, normalization LSA mathematical and statistical models product description Python summary_ratio TextRank Automated text classification binary classification description learning methods multi-class classification prediction process reinforcement learning semi-supervised learning supervised learning training process unsupervised learning Averaged word vectors B Backus-Naur Form (BNF) Bag of Words model BigramTagger Bing Liu’s lexicon Blueprint, text classification The British National Corpus (BNC) C Case conversion operations Centroid-based clustering models Centrum Wiskunde and Informatica (CWI) The Child Language Data Exchange System (CHILDES) ChunkRule Classification algorithms evaluation multinomial naïve Bayes supervised ML algorithms SVM training tuning types ClassifierBasedPOSTagger class Cleaning text Collocations Common Language Runtime (CLR) Comprehensions Conda package management conll_tag_chunks() Constituency-based parsing Continuous integration (CI) processes Contractions The Corpus of Contemporary American English (COCA) Cosine distance and similarity CPython D Database management systems (DBMS) Deep learning Density-based clustering models Dependency-based parsing code Language Syntax and Structure nltk rule-based dependency sample sentence scaling spacy’s output textual tokens tree/graph DependencyGrammar class Dictionaries Distribution-based clustering models Document clustering AP SeeAffinity propagation (AP) BIRCH and CLARANS centroid-based clustering models definition density-based clustering models distribution-based clustering models hierarchical clustering models IMDb K-meansclustering SeeK-means clustering movie data normalization and feature extraction scikit-learn Ward’s hierarchicalclustering SeeWard’s agglomerative hierarchical clustering Document similarity build_feature_matrix() corpus of cosine distance and similarity HB-distance mathematical computations Okapi BM25 TF-IDF features toy_corpus index E Entailments Euclidean distance F Feature-extraction techniques advanced word vectorization models averaged word vectors Bag of Words model definition implementations, modules TF-IDFmodel SeeTerm Frequency-Inverse Document Frequency (TF-IDF) model TF-IDF weighted averaged word vectors Vector Space Model First order logic (FOL) Flow of code FOL SeeFirst order logic (FOL) Functions functools module G Gaussian mixture models (GMM) Generators gensim library Global Interpreter Lock (GIL) Grammar classification constituency conjunctions coordinating conjunction lexical category model noun phrases phrase structure rules prepositional phrases recursive properties rules and conventions syntax trees verb phrases course of time dependencies models rules syntax and structure, language Graphical user interfaces (GUIs) H Hamming distance Handling exceptions Hellinger-Bhattacharya distance (HB-distance) Hierarchical clustering models Higher order logic (HOL) High-level language (HLL) Holonyms Homographs Homonyms Human-computer interaction (HCI) Hypernyms Hyperparameter tuning Hyponyms I IMDb SeeInternet Movie Database (IMDb) Indexing Information retrieval (IR) Integrated development environments (IDEs) Internet Movie Database (IMDb) movie reviews datasets feature-extraction getting and formatting, data lexicons SeeLexicons model performance metrics and visualization positive and negative setting up dependencies supervisedML SeeSupervised machine learning technique text normalization Iterators J JAVA_HOME environment variable Java Runtime Environment (JRE) Java Virtual Machine (JVM) K Keyphrase extraction collocations definition text analytics weighted tag–based phrase extraction K-means clustering analysis data data structure definition functions IMDb movie data iterative procedure movie data multidimensional scaling (MDS) Kullback-Leibler divergence L Lancaster stemmer Language semantics antonyms capitonyms definition FOL collection of well-defined formal systems components HOL natural language statements quantifiers and variables universal generalization heterographs heteronyms homographs homonyms homophones hypernyms hyponyms lemma lexical linguistic networks and models PL SeePropositional logic (PL) polysemes representation synonyms syntax and rules wordforms Language syntax and structure clauses categories declarative exclamations imperative independent sentences interrogative relationship relative collection of words constituent units English grammar SeeGrammar hierarchical tree phrases rules, conventions and principles sentence word order typology Latent dirichlet allocation (LDA) algorithm black box end-to-end framework gensim get_topics_terms_weights() function LdaModel class parameters plate notation print_topics_udf() function Latent semantic analysis (LSA) Latent semantic indexing (LSI) description dictionary framework function, thresholds gensim and toy corpus low_rank_svd() function matrix computations parameters terms and weights TF-IDF feature matrix TF-IDF–weighted model thresholds Lemmatization nltk package normalizing root word speech wordnet corpus Levenshtein edit distance Lexical Functional Grammar (LFG) Lexical semantics Lexical similarity Lexicons AFINN Bing Liu description MPQA subjectivity pattern SeePattern lexicon SentiWordNet VADER Linguistics definition discourse analysis lexicon morphology phonetics Phonology pragmatics semantics semiotics stylistics syntax term Lists Looping constructs LSA SeeLatent semantic analysis (LSA) M Machine learning (ML) algorithms Manhattan distance max(candidates, key=WORD_COUNTS.get) function MaxentClassifier Meronyms Multi-class text classification system confusion matrix and SVM feature-extraction techniques metrics, prediction performance misclassified documents multinomial naïve Bayes and SVM normalization scikit-learn training and testing datasets Multinomial naïve Bayes Multi-Perspective Question Answering (MPQA) subjectivity lexicon N Named entity recognition Natural language acquisition and cognitive learning and usage analysis, data communication database DBMS direction, fit representation human languages linguistics SeeLinguistics NLP NLP SeeNatural language processing (NLP) origins of language philosophy processing semantics SeeLanguage semantics sensors SQL Server syntax and structure phrases techniques and algorithms textcorpora SeeText corpora/text corpus triangle of reference model usage Natural language processing (NLP) contextual recognition and resolution definition HCI machine translation QAS speech recognition text analytics text categorization text summarization The Natural Language Toolkit (NLTK) NGramTagChunker Non-negative matrix factorization (NNMF) Normalization contractions corpus, text documents lemmatization stopwords symbols and characters techniques Noun phrase (NP) Numeric types O Object-oriented programming (OOP) Okapi BM25 ranking P Parts of speech (POS) tagging Pattern lexicon description mood and modality, sampled test movie reviews mood and modality, text documents sentiment prediction performance sentiment statistics Phrases adjective phrase (ADJP) adverb phrase (ADVP) annotated categories noun phrase (NP) prepositional phrase (PP) principle verb phrase (VP) Pip package management PL SeePropositional logic (PL) Polarity analysis Polysemous Popular corpora ANC BNC Brown Corpus CHILDES COCA Collins Corpus Google N-gram Corpus KWIC LOB Corpus Penn Treebank reuters Web, chat, email, tweets WordNet Porter stemmer POS taggers bigram models building classification-based approach ContextTagger class input tokens MaxentClassifier NaiveBayesClassifier nltk pattern module trigram models Prepositional phrase (PP) ProjectiveDependencyParser Propositional logic (PL) atomic units complex units connectors constructive dilemma declarative Disjunctive Syllogism Hypothetical Syllogism Modus Ponens Modus Tollens operators with symbols and precedence sentential logic/statement logic truth values PunktSentenceTokenizer PyEnchant Python ABC language advantages and benefits built-in methods classes code conditional code flow database programming data types dictionaries disadvantages environment formatting hands-on approach identity implementations and versions lists machine learning manipulations and operations numeric types OS principles programming language programming paradigms Scientific computing scripting strings structure syntax systems programming text analytics text data type value versions virtual environment web development Python 2.7.x Python 2.x Python 3.0 Python 3.x Python Package Index (PyPI) Python Reserved Words Python standard library (PSL) Q Question Answering Systems (QAS) R range() Recursive functions RegexpStemmer RegexpTokenizer class Regular expressions (Regexes) Repeating characters Rich internet applications (RIA) Robust ecosystem S SciPy libraries Semantic analysis FOL frameworks messages named entity recognition natural language parts of speech (POS), chunking, and grammars PL WordNet SeeWordNet word sense disambiguation Sentence tokenization delimiters German text Gutenberg corpus nltk interfaces nltk.sent_tokenize function pre-trained German language pre-trained tokenizer PunktSentenceTokenizer class snippet text corpora text samples Sentiment analysis description IMDb moviereviews SeeInternet Movie Database (IMDb) movie reviews polarity analysis techniques textual data SentiWordNet Sets Shallow parsing chunking process code snippet conll2000 corpus conlltags2tree() function Evaluating Classification Models expression-based patterns generic functions IOB format noun phrases parse() function parser performance POS tags sentence tree snippet tagger_classes parameter tokens/sequences treebank corpus treebank training data visual representation Singular Value Decomposition (SVD) description extraction-based techniques low rank matrix approximation LSA LSI NNMF Slicing SnowballStemmer Special characters removal Speech recognition system Spelling correction candidate words case_of function code director of research English language preceding function replacements vocabulary dictionary StemmerI interface Stemming affixes code inflections snippet Snowball Project user-defined rules Stopwords Strings indexing syntax literals operations and methods Supervised machine learning technique confusion matrix normalization and feature-extraction performance metrics positive and negative emotions predictions support vector machine (SVM) test dataset reviews text classification Support vector machines (SVM) SVD SeeSingular Value Decomposition (SVD) SVM SeeSupport vector machines (SVM) Synonyms T Term frequency-inverse document frequency (TF-IDF) model Bag of Words feature vectors CORPUS CountVectorizer definition diagonal matrix Euclidean norm mathematical equations matrix multiplication tfidf feature vectors tfidf weights TfidfTransformer class TfidfTransformer TfidfVectorizer Text analytics textblob Text classification applications and uses automated SeeAutomated text classification blueprint conceptual representation definition documents feature-extraction SeeFeature-extraction techniques inherent properties learning machine learning (ML) normalization prediction performance, metrics accuracy confusion matrix emails, spam and ham F1 score precision recall products types Text corpora/text corpus access Brown Corpus NLTK Reuters Corpus WordNet annotation and utilities collection of texts/data monolingual multilingual origins popular Text normalization Text pre-processing techniques TextRank Text semantics Text similarity Bag of Characters vectorization character vectorization cosine distance and similarity description distance metrics Euclidean distance feature-extraction Hamming distance information retrieval (IR) Levenshtein edit distance Manhattan distance normalization similarity measures terms and computing text data unsupervised machine learning algorithms vector representations Text summarization description documents feature extraction feature matrix information extraction automated documentsummarization SeeAutomated document summarization information overload Internet Keyphraseextraction SeeKeyphrase extraction production of books techniques topicmodeling SeeTopic modeling information overload normalization social media SVD Text syntax Graphviz installation libraries machine learning concepts nltk processing and normalization TF-IDF weighted averaged word vectors tokenize_text function Tokenizing text Topic modeling definition frameworks and algorithms gensim and scikit-learn LDA SeeLatent Dirichlet allocation (LDA) LSI SeeLatent semantic indexing (LSI) NNMF product reviews treebank corpus treebank data TreebankWordTokenizer TrigramTagger Tuples U Unicode characters UnigramTagger V VADERlexicon SeeValence Aware Dictionary and sEntiment Reasoner (VADER) lexicon Valence Aware Dictionary and sEntiment Reasoner (VADER) lexicon Vector Space Model Verb phrase (VP) W, X, Y Ward’s agglomerative hierarchical clustering cosine similarity defintion dendrogram distance metric IMDb movie data linkage criterion Ward’s minimum variance method Weighted tag–based phrase extraction WordNet definition entailments holonyms and meronyms homonyms and homographs hyponyms and hypernyms lexical semantic relations semantic relationships and similarity synonyms and antonyms synsets web application interface WordNetLemmatizer class Word order typology Words Adverbs annotated, POS tags correction meaning morphemes N(oun) parts of speech plural nouns pronouns PRON tag sense disambiguation singular nouns singular proper nouns verbs Word tokenization interfaces lemmatization nltk.word_tokenize function patterns regular expressions snippet stemming Z Zen of Python


From Airline Reservations to Sonic the Hedgehog: A History of the Software Industry by Martin Campbell-Kelly

Apple II, Apple's 1984 Super Bowl advert, barriers to entry, Bill Gates: Altair 8800, business process, card file, Charles Babbage, computer age, computer vision, continuous integration, Dennis Ritchie, deskilling, Donald Knuth, Gary Kildall, Grace Hopper, history of Unix, hockey-stick growth, independent contractor, industrial research laboratory, information asymmetry, inventory management, John Markoff, John von Neumann, Larry Ellison, linear programming, longitudinal study, machine readable, Menlo Park, Mitch Kapor, Multics, Network effects, popular electronics, proprietary trading, RAND corporation, Robert X Cringely, Ronald Reagan, seminal paper, Silicon Valley, SimCity, software patent, Steve Jobs, Steve Wozniak, Steven Levy, Thomas Kuhn: the structure of scientific revolutions, vertical integration

Because these software developments were so capital intensive, the two market leaders, 154 Chapter 5 Anacomp and Hogan Systems, were both funded through non-market sources. Anacomp, established as a programming services operation in 1968, received funding from 30 major banks to develop its CIS (Continuous Integrated System) real-time banking software product. By 1983, Anacomp had pulled ahead of all its competitors and was among the top 10 software products firms, with annual sales of $172 million. However, the company’s sales plummeted in 1984 as a result of product-development delays with CIS. After losing $116 million on sales of $132 million, “for all practical purposes, CIS was dead.”49 Anacomp’s software was acquired by EDS in 1985, and Anacomp faded from sight.

., 80, 84 Computer Information Management, 148 Computer Leasing Company, 80 Computer Machinery Group, 77 Computer Sciences Corporation, 5, 12, 50–54, 57, 70–73, 76, 83, 84, 115, 117, 195, 205, 304, 305 Computer Sciences International, 76 Computer Services and Software Industry, 57, 59 Computer Services Association, 77 Computer Space, 272 Computer Usage Company, 5, 24, 31, 50–52, 57, 59, 71, 73 Computer Vision, 128, 129, 160–162, 244, 245 ComputerLand, 209, 255 Computers, numbers of, 89, 90 Computers and Software Inc., 62 Computicket, 73 Compuware, 168, 172 Comshare, 63 Configuration Utilization Evaluator, 102 Connecticut Mutual Insurance Company, 219 Consolidated Edison, 149 Continental Software, 227, 294 Continuous Integrated System, 154 Control Data Corporation, 62, 82, 102, 109, 250 CDC 1604 computer, 80 Cook, Scott, 261, 294, 295 Copyright, See Intellectual property issues Corel, 261, 263 Cornfeld, Bernie, 83 COSMIC, 130 CP/M, 206, 215–218, 239–241, 249 Creative Strategies International, 26 Cricket Software, 183 Cross-platform software, 262 Cullinane Corporation, 86, 122, 126, 148, 165 Cullinane, John, 148, 169, 189, 190 Cullinet, 168, 184, 187–190 Cyrix, 239 Cytation Inc., 291 Data Decisions, 132 Data General, 159, 160, 254 Data Pro, 132 Data Products Corporation, 80 Data Systems Analysts, 58, 59 364 Index Data Transmission Company, 84 Databases, 6, 101, 113, 116, 123, 133, 134, 145–149, 156, 176, 184–191, 203, 210, 213–216, 219–221, 234, 244, 251, 256–259, 263, 301, 307 relational, 31, 149, 168, 169, 185–191, 198, 307, 310 DATACOM/DB, 116, 146–148, 151, 184 Dataskil, 78 Datasolv, 78 Datran, 84, 85 dBase II–IV, 7, 203, 210–212, 215, 219, 220, 254–257, 259, 263 Defense Advanced Research Projects Agency, 41 Desktop publishing software, 236, 261, 263 DESQ, 249 DeVries, John, 82 Diana project, 48, 49 Digital Computer Association, 32, 33 Digital Equipment Corporation, 143–145, 159, 160, 173, 188, 195, 213, 306 Digital Research, 203, 206, 207, 217, 234, 239, 241, 248, 249, 264, 289 Disruptive technologies, 186, 187 DistribuPro, 183 Dorling Kindersley, 292 Dow Chemical, 195 Drake, Dan, 243 Draw!


Mathematical Finance: Theory, Modeling, Implementation by Christian Fries

Black-Scholes formula, Brownian motion, continuous integration, discrete time, financial engineering, fixed income, implied volatility, interest rate derivative, martingale, quantitative trading / quantitative finance, random walk, short selling, Steve Jobs, stochastic process, stochastic volatility, volatility smile, Wiener process, zero-coupon bond

FOUNDATIONS Definition 31 (Itô integral for elementary processes): A stochastic process φ is called elementary, if X φ(t, ω) = e j (ω) · 1(t j ,t j+1 ] (t), q j∈N∪{0} where {t j | j ∈ N ∪ {0}} is a strictly monotone sequence in [0, ∞) with t0 := 0 and {e j | j ∈ N ∪ {0}} a sequence of Ft j -measurable random variables and 1(t j ,t j+1 ] denotes the indicator function15 .16 For an elementary process we define the Itô integral as Z ∞ X φ(t, ω)dW(t) := e j (ω) · (W(t j+1 , ω) − W(t j , ω)), 0 T Z j∈N∪{0} ∞ φ(t, ω)dW(t) := S Z φ(t, ω) · 1(S ,T ] (t)dW(t). 0 y Remark 32 (on the one sided continuity of the integrand): In some textbooks (e.g. [25]) the elementary integral is defined using the indicator function 1[t j ,t j+1 ) in place of 1(t j ,t j+1 ] . For continuous integrators, as we consider here (W(t)), it makes no difference which variant we use. However if jump processes are considered (see e.g. [27]) and also with respect to the interpretation of the integral as trading strategy (see page 73) our definition is the better suited. Lemma 33 (Itô Isometry): Let φ denote an elementary process such that φ(·, ω) is bounded.

Let X(t) denote a previsible process. Then we define Z t2 Z t2 Z t2 X(t) dY(t) := X(t) dA(t) + X(t) dM(t). t1 t1 t1 y Remark 57 (Stochastic Integral): The class of processes (integrands) for which we may define a stochastic integral depends on the properties of the integrators (and vice versa). For continuous integrators (as the Brownian motion) the integrands merely have to be adapted processes. For more general integrators one has to restrict to a smaller class of integrands, e.g. previsible processes. Compare Example 4.1 and Remark 4.4 in [12]. For more detailed discussion of the stochastic integral see [6], §5.5 and (especially for more general integrators) [12], §4 and [18], §3.


pages: 255 words: 55,018

Architecting For Scale by Lee Atchison

Amazon Web Services, business logic, business process, cloud computing, continuous integration, DevOps, Internet of things, microservices, platform as a service, risk tolerance, software as a service, web application

The ideas in this chapter will help you manage your application and your team to avoid this cycle and keep your availability high. 1 According to Werner Vogels, CTO of Amazon, in 2014 Amazon did 50 million deploys to individual hosts. That’s about one every second. 2 This could be, but does not need to be a modern continuous integration and continuous deploy (CI/CD) process. Part II. Risk Management You cannot possibly manage the risk in your system if you cannot identify the risk in your system. …but there are also unknown unknowns—the ones we don’t know we don’t know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones.


pages: 184 words: 12,922

Pragmatic Version Control Using Git by Travis Swicegood

AGPL, continuous integration, David Heinemeier Hansson, en.wikipedia.org, Firefox, George Santayana, revision control

Coupled with an automated test suite that can be run from the command line, you can use it to trace bugs quickly. The enterprising development team could even tie it into their automated build system to help isolate bugs. This can be a tremendous time-saver when a project moves too rapidly to be tested with a traditional continuous integration system where each change kicks off a new build. This brings us to the end of the basics chapter and the end of Part II. Whether you’ve just started your journey as a developer and are learning your first VCS or you are an old hand looking to pick up Git, you now know what you need to start being productive with Git.


pages: 443 words: 51,804

Handbook of Modeling High-Frequency Data in Finance by Frederi G. Viens, Maria C. Mariani, Ionut Florescu

algorithmic trading, asset allocation, automated trading system, backtesting, Bear Stearns, Black-Scholes formula, book value, Brownian motion, business process, buy and hold, continuous integration, corporate governance, discrete time, distributed generation, fear index, financial engineering, fixed income, Flash crash, housing crisis, implied volatility, incomplete markets, linear programming, machine readable, mandelbrot fractal, market friction, market microstructure, martingale, Menlo Park, p-value, pattern recognition, performance metric, power law, principal–agent problem, random walk, risk free rate, risk tolerance, risk/return, short selling, statistical model, stochastic process, stochastic volatility, transaction costs, value at risk, volatility smile, Wiener process

(13.34) Here, L = L(x, t) is a second-order elliptic operator in nondivergence form, namely, L(x, t) := d i,j=1 ∂2 ∂ + bi (x, t) + c(x, t). ∂xi ∂xj ∂xi i=1 d aij (x, t) The integro-differential operator is defined by F(x, t, u, ∇u) = f (x, t, y, u(x, t), ∇u(x, t)) dy. (13.35) This integro-differential operator will be a continuous integral operator as the ones defined in Equations 13.28 and 13.32 modeling the jump. The case in which f is decreasing respect to u and all jumps are positive corresponds to the evolution of a call option near a crash. Throughout this section, we impose the following assumptions: A(1) The coefficients aij (x, t), bi (x, t), c(x, t) belong to the Hölder space C δ,δ/2 (Q T ).

See also Value at risk (VaR) 423 Conditional variances, 203, 206, 208 of the GARCH(1,1) process, 180 Confidence intervals, for forecasts, 187–188 Consecutive trades, 129 Consensus indicators, 62 Constant coefficient case, 311 Constant default correlation, 79–81 Constant default correlation model, 76 Constant rebalanced portfolio technical analysis (CRP-TA) trading algorithm, 65–66 Constant variance, 181 Constant volatility, 353 Constructed indices, comparison of, 106–107 Constructed volatility index (VIX). See also Volatility index (VIX) comparing, 105–106 convergence of, 105 Contaminated returns, variance and covariance of, 257 Continuous integral operator, 367 Continuous semimartingales, 246, 253 Continuous-time long-memory stochastic volatility (LMSV) model, 220 Continuous-time stochastic modeling, 3 Continuous-time vintage, 78 Convergence-of-interests hypothesis, 54 Convex duality method, 296 Copula models, 77 Copulas, 75–76 CorpInterlock, 62, 63 Corporate governance, 53–54 of S&P500 companies, 54–60 Corporate governance best practices, 59 Corporate governance scorecards, 51–52 Corporate governance variables, 69 interpreting S&P500 representative ADTs with, 58–59 Corporate performance, predicting, 69 Correlation coefficient, 400 Correlation fluctuations impact on securitized structures, 75–95 products and models related to, 77–79 Cost structures, 392 424 Covariance(s) estimating, 244 forecasting, 280–285 Covariance function, 252 Covariance matrix, 170 Covariance stationarity, 177, 179, 181 Covariation-realized covariance estimator, 266 Covolatility function, 249 Covolatility measurement/forecasting, as a key issue in finance, 243 Cox, Ingersoll, Ross (CIR) square-root model, 257 cpVIX, 103 Crash imminence, precautions against, 121 Creamer, Germán, xiii, 47 Crisis detection, 131 Crisis-related equity behavior, 150 Cubic-type kernels, 261, 263 Cumulative abnormal return, 62, 63 Cumulative consumption process, 297, 305–306 Cumulative distribution curve, 346 Cumulative distribution function, 176 Current market volatility distribution, estimating, 115 Current weighting, 49 Customer perspective, 51 Cutting frequency, 258, 259 cVIX-1, 101, 102.


Learning Ansible 2 - Second Edition by Fabio Alessandro Locati

Amazon Web Services, anti-pattern, business logic, cloud computing, continuous integration, Debian, DevOps, don't repeat yourself, Infrastructure as a Service, inventory management, Kickstarter, revision control, source of truth, web application

In this chapter, we'll look at syntactic checks, testing without applying the code on the machines (the no-op mode), and functional testing for playbooks, which are at the core of Ansible and trigger the various tasks you want to perform on the remote hosts. It is recommended that you integrate some of these into your Continuous Integration (CI) system that you have for Ansible to better test your playbooks. We'll be looking at the following points: • Syntax checking • Checking the mode with and without diff • Functional testing As part of functional testing, we will be looking at: • • • • Assertions on the end state of the system Testing with tags Serverspec (a different tool, but can work wonderfully with Ansible) Using the --syntax-check option Whenever you run a playbook, Ansible first checks the syntax of the playbook file.


pages: 274 words: 58,675

Puppet 3 Cookbook by John Arundel

Amazon Web Services, cloud computing, continuous integration, Debian, defense in depth, DevOps, don't repeat yourself, Free Software Foundation, GnuPG, Larry Wall, place-making, Ruby on Rails, web application

Vagrant drives VirtualBox or another virtualization layer to automate the process of creating a VM, provisioning it with Chef or Puppet, setting up networking, port forwarding, and packaging running VMs into images for others to use. You can use Vagrant to manage your development VMs on your own desktop, or on a shared machine such as a continuous integration server. For example, you might use a CI tool such as Jenkins to boot a VM with Vagrant, deploy your app, and then run your tests against it as though it were in production. Getting ready… In order to use Vagrant, you'll need to install VirtualBox (which actually runs the virtual machine) and Vagrant (which controls VirtualBox).


Demystifying Smart Cities by Anders Lisdorf

3D printing, artificial general intelligence, autonomous vehicles, backpropagation, behavioural economics, Big Tech, bike sharing, bitcoin, business intelligence, business logic, business process, chief data officer, circular economy, clean tech, clean water, cloud computing, computer vision, Computing Machinery and Intelligence, congestion pricing, continuous integration, crowdsourcing, data is the new oil, data science, deep learning, digital rights, digital twin, distributed ledger, don't be evil, Elon Musk, en.wikipedia.org, facts on the ground, Google Glasses, hydroponic farming, income inequality, information security, Infrastructure as a Service, Internet of things, Large Hadron Collider, Masdar, microservices, Minecraft, OSI model, platform as a service, pneumatic tube, ransomware, RFID, ride hailing / ride sharing, risk tolerance, Salesforce, self-driving car, smart cities, smart meter, software as a service, speech recognition, Stephen Hawking, Steve Jobs, Steve Wozniak, Stuxnet, Thomas Bayes, Turing test, urban sprawl, zero-sum game

There are also different frameworks for iterative development like Scrum and SAFe, but a traditional team working in a line of business on a solution area also typically works iteratively. It has been used for decades for business as usual and later with agile development as an emerging standard for Net New development too. With continuous integration/continuous delivery, it has entered the realm of maintenance and infrastructure development. It is also well suited for experimental work since it is good at adapting to new situations. Iterative development is often challenged in its ability to make long-term plans and achieve strategic results since everything is done in small incremental loops.


pages: 239 words: 68,598

The Vanishing Face of Gaia: A Final Warning by James E. Lovelock

Ada Lovelace, Alan Greenspan, An Inconvenient Truth, butterfly effect, carbon footprint, Clapham omnibus, cognitive dissonance, continuous integration, David Attenborough, decarbonisation, discovery of DNA, disinformation, Edward Lorenz: Chaos theory, Garrett Hardin, Henri Poincaré, Herman Kahn, Intergovernmental Panel on Climate Change (IPCC), mandelbrot fractal, mass immigration, megacity, Northern Rock, ocean acidification, oil shale / tar sands, phenotype, Pierre-Simon Laplace, planetary scale, quantum entanglement, short selling, Stewart Brand, Tragedy of the Commons, University of East Anglia, Virgin Galactic

As if this were not enough to damn wind energy, the construction of a 1 GW wind farm would use a quantity of concrete, 2 million tons, sufficient to build a town for 100,000 people living in 30,000 homes; making and using that quantity of concrete would release about 1 million tons of carbon dioxide into the air. For us to survive as a civilized nation our cities need that safe, secure and constant supply of electricity that only coal, gas or nuclear can provide, and only with nuclear can we be assured of a constant supply of fuel. We have already seen how vulnerable gas supplies are to the continued integrity of pipelines perhaps a thousand miles long, and to the aggressive politics of autocrats. Coal is expensive in the UK and imports are insecure. Wind farms are hopelessly inadequate to the UK as a source of energy and as I have indicated can do little to halt global heating even when used on a global scale; moreover, experience in Western Europe shows them to be costly and inefficient sources of electricity.


pages: 536 words: 73,482

Programming Clojure by Stuart Halloway, Aaron Bedra

continuous integration, duck typing, en.wikipedia.org, functional programming, general-purpose programming language, Gödel, Escher, Bach, higher-order functions, Neal Stephenson, Paul Graham, Ruby on Rails, type inference, web application

(With no validation, the regression error might just be telling you that the old code was broken and the new code fixed it.) How hard is it write a program that should produce exactly the same output? Call only pure functions from the program, which is exactly what our score-inputs function does. Wiring this kind of regression test into a continuous integration build is not difficult. If you do it, think about contributing it to whatever testing framework you use. Now we have partially answered the question, “How do I make sure that my code is correct?” In summary: Build with small, composable pieces (most should be pure functions). Test forms from the inside out at the REPL.


Raw Data Is an Oxymoron by Lisa Gitelman

23andMe, collateralized debt obligation, computer age, continuous integration, crowdsourcing, disruptive innovation, Drosophila, Edmond Halley, Filter Bubble, Firefox, fixed income, folksonomy, Google Earth, Howard Rheingold, index card, informal economy, information security, Isaac Newton, Johann Wolfgang von Goethe, knowledge worker, Large Hadron Collider, liberal capitalism, lifelogging, longitudinal study, Louis Daguerre, Menlo Park, off-the-grid, optical character recognition, Panopticon Jeremy Bentham, peer-to-peer, RFID, Richard Thaler, Silicon Valley, social graph, software studies, statistical model, Stephen Hawking, Steven Pinker, text mining, time value of money, trade route, Turing machine, urban renewal, Vannevar Bush, WikiLeaks

True interpellation—in his terms “a complicated configuration of unconsciousness, indirection, automation, and absentmindedness”—requires a coercive system, a “superpanopticon,” capable of rendering us as both subjects of and subjects to that particular assemblage that David Mitchell, in a fictional context, calls a corpocracy.22 For Kevin Robins and Frank Webster, this is the essence of “cybernetic capitalism,” by which they mean the whole of the socioeconomic control system that is in part dependent on the capacity of state and corporate entities to collect and aggregate personal data to the extent that each individual can be easily monitored, managed, and hence controlled.23 As my epigraphs indicate, Robins and Webster are far from alone in their concern with our dynamic incorporation within a totalizing technological system of data management. 24 Greg Elmer also explicates the techniques by which consumer profiles are developed and individuals are “continuously integrated into a larger information economy and technological apparatus.”25 But for Elmer and Lyon and others, a crucial aspect of this incorporation is our voluntary participation: the composition of consumer profiles in part results from solicitation—whether in the form of a request for feedback or personal data so as to be granted access to a particular service or program—which means we are interpellated as “self-communicating” actors.26 To be sure, to participate in the project of modernity has arguably always meant that one becomes a calculable subject by voluntarily surrendering data.


pages: 278 words: 83,468

The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses by Eric Ries

3D printing, barriers to entry, Benchmark Capital, call centre, Clayton Christensen, clean tech, clean water, cloud computing, commoditize, Computer Numeric Control, continuous integration, corporate governance, disruptive innovation, experimental subject, Ford Model T, Frederick Winslow Taylor, hockey-stick growth, Kanban, Lean Startup, Marc Andreessen, Mark Zuckerberg, Metcalfe’s law, minimum viable product, Mitch Kapor, Network effects, payday loans, Peter Thiel, pets.com, Ponzi scheme, pull request, reality distortion field, risk tolerance, scientific management, selection bias, Silicon Valley, Silicon Valley startup, six sigma, skunkworks, social bookmarking, stealth mode startup, Steve Jobs, the scientific method, Toyota Production System, transaction costs

Why do we add additional gems that we don’t intend to use right away? Answer: In preparation for a code push we wanted to get all new gems ready in the production environment. Even though our code deployments are fully automated, gems are not. Proportional investment: Bennett—Automate gem management and installation into Continuous Integration and Continuous Deployment process. Bonus—Why are we doing things in production on Friday nights? Answer: Because no one says we can’t and it was a convenient time for the developer to prepare for a deployment we’d be doing on Monday. Proportional investment: Tony—Make an announcement to the team.


pages: 314 words: 94,600

Business Metadata: Capturing Enterprise Knowledge by William H. Inmon, Bonnie K. O'Neil, Lowell Fryman

affirmative action, bioinformatics, business cycle, business intelligence, business process, call centre, carbon-based life, continuous integration, corporate governance, create, read, update, delete, database schema, en.wikipedia.org, folksonomy, informal economy, knowledge economy, knowledge worker, semantic web, tacit knowledge, The Wisdom of Crowds, web application

Appendix Metadata System of Record Example Table 1 Metadata object system of record Meta Object Entity Name Entity Type Entity Definition Entity Scope Entity Active Ind Entity Logical Business Rule Logical Application Name Entity Synonym Name Logical Attribute Logical Attribute Definition Attribute Logical Business Rule Attribute Logical FK Ind Attribute Business Area Logical Business Function Data Subject Area Physical Column Name Physical Column Data Type Physical Column Length Physical Column Precision Strategic Modeling Tool Tactical Modeling Tool C C C C C U U C U DBMS Tool Data Integration Tool Reporting Tool U C C C C U C U C C U U C C U U C U C U C U C U (Cont.) 283 284 Appendix Table 1 Metadata object system of record (continued) Meta Object Strategic Modeling Tool Physical Column Decimal places Physical Column Default Value Physical Column Nullable Ind Physical Column Comment Physical Column Primary Key Ind Physical Column Foreign Key Ind Table Name Table Owner Table Type Table Comments Physical Table Name Physical Column Name Physical View Name Physical Database Name Physical Schema Name ETL Object Name Source Table Source Column Target Table Target Column ETL Job Name ETL Transformation Rule ETL Job Run Date ETL Job Execution Time ETL Job Row Count ETL Job Status Report Name Report Element Name Report Table Name Report Database Name Report DB Sequence Report Element Business Rule Tactical Modeling Tool DBMS Tool C U C U C U C U C C C C C U U U U U C C C C C Data Integration Tool Reporting Tool C R R R R C C C C C C C R R R R C Appendix 285 Metadata Usage Matrix Example The following table summarizes the metadata objects and the anticipated usage of each object in the following functions: Table 1 Summary of metadata objects usage Source – Metadata Object Entity Name Entity Type Entity Definition Entity Scope Entity Container Entity Active Ind Entity Logical Business Rule Logical Application Name Entity Synonym Name Logical Attribute Logical Attribute Definition Attribute Logical Business Rule Attribute Logical FK Ind Attribute Business Area Logical Business Function Data Subject Area Physical Column Name Physical Column Data Type Physical Column Length Physical Column Precision Physical Column Decimal Places Physical Column Default Value Physical Column Nullable Ind Data Lineage Impact Analysis Definition and or Glossary Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y (Cont.) 286 Appendix Table 1 Summary of metadata objects usage (continued) Source – Metadata Object Physical Column Comment Physical Column Primary Key Ind Physical Column Foreign Key Ind Table Name Table Owner Table Type Table Comments Physical Table Name Physical Column Name Physical View Name Physical Database Name Physical Schema Name ETL Object Name Source Table Source Column Target Table Target Column ETL Job Name ETL Transformation Rule ETL Job Run Date ETL Job Execution Time ETL Job Row Count ETL Job Status Report Name Report Element Name Report Table Name Report Database Name Report DB Sequence Report Element Business Rule Data Lineage Impact Analysis Definition and or Glossary Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Index Abstraction, linking structured and unstructured data, 230–231 Accuracy, metadata information, 33–34 Administration, infrastructure issues functionality requirements, 169 history keeping, 169–170 BI, see Business intelligence Broader term, definition component, 61, 63–64 Business Glossary features, 111–112 integrated technical and business metadata delivery, 152–153 Business intelligence (BI), business metadata delivery infrastructure, 163 Business metadata capture, see Capture, business metadata components, 13 definition, 38 delivery, see Delivery, business metadata funding, see Funding, business metadata historical perspective, 3–11 importance, 274–275 locations corporate forms, 15–16 reports, 14, 29–31 screens, 13–14 origins, 19–20 repository construction, see Metadata project resources, 281–282 technical metadata comparison, 12–13 conversion, 135–136, 182–186 infrastructure for integration, 165–166 separation, 140 tracking over time, 20–21 types, 158–160 Business rules business management, 238–239 business metadata, 237, 245 capturing rationale, 238–239 definition, 235–236 maintenance, 242 management, 243 metadata repository, 244–245 ruleflow, 242–243 sources, 237–238 systems, 239, 242–243 Call center volume, search problem quantification, 70 Capability Maturity Model Integration (CMMI), note-taking as asset producing, 265 Capture, business metadata barriers, 275 corporate knowledge base, 93–94 culture, 95–96 editing automation, 128–129 expansion of definition and descriptions, 129–131 granularization, 129 homonym resolution, 132–134 manual editing, 134–135 staging area, 134 synonym resolution, 131–132 Governance Lite™, 107–109, 111 individual documentation problem, 114–115 knowledge socialization, see Knowledge socialization metadata sources comparison of sources, 127 data warehouse, 126 database management system system catalogs, 124 documents, 123 enterprise resource planning applications, 122 extract-transform-load, 124–125 legacy systems, 125–126 on-line analytical processing tools, 124 on-line transaction processing, 125–126 reports, 122–123 spreadsheets, 123 principles, 95–96 publicity, 112–113 rationale, 90–93 technical metadata conversion to business metadata, 135–136 287 288 Index Capture, business metadata (Continued) technology search, 109–111 Web 2.0 folksonomy, 118–119 mashups, 115–116 overview, 115 Card catalog, see Library card catalog CDC, see Centers for Disease Control Centers for Disease Control (CDC), linking structured and unstructured data, 231 CIF, see Corporate Information Factory C-map, see Concept Map CMMI, see Capability Maturity Model Integration Collective intelligence, knowledge socialization, 97 Communications audits, 251 clarity problems bad business decisions, 57 English language limitations, 59 everyday communications, 56–57 faulty rollups, 57–58 units of measure differences, 59 classification, see Taxonomy definitions components, 61–62 guidelines, 60–61 importance, 59–60 miscellaneous guidelines, 64 usage notes, 62–64 historic library creation, 253 human/computer communication problem, 276–278 screening, 251–253 search problem information and knowledge workers, 65–66 information provider guidelines, 71–72 quantification, 67–70 search techniques, 71 tracking down information, 66–67 Compliance communications audits, 251 historic library creation, 253 screening, 251–253 data profiling, 254–256 financial audit metadata utility, 250 transaction background activities, 250–251 prospects, 280 Sarbanes-Oxley Act provisions, 248–240 types, 249–251 Concept Map (C-map), semantic framework, 205, 209 Conceptual model, semantic framework, 204 Controlled vocabulary (CV), semantic framework, 200–201 Corporate forms, business metadata content, 15–16 Corporate Information Factory (CIF), implementation, 81–82 Corporate knowledge base, components, 93–94 Create Read Update Delete (CRUD), conflict resolution, 167 CRM, see Customer relationship management Cross selling, business metadata capture rationale, 92 CRUD, see Create Read Update Delete Customer relationship management (CRM), business metadata capture rationale, 92 Customer, definition, 56 CV, see Controlled vocabulary DASD, see Direct access storage device Data, definition, 176 Database management system (DBMS) historical perspective, 5 metadata storage, 19 system catalog as metadata resource, 124 Data Flux, data quality presentation, 190–191 Data Governance Council, metadata stewardship, 42–43 Data quality continuum, 190 Data Warehousing report, 49 definition, 177 presentation, 189–190 Data Stewardship Council, metadata stewardship, 43–44 Data warehouse historical perspective, 7–9 infrastructure, 160–161 metadata resource, 126 metadata warehouse features, 161–162 DBMS, see Database management system Decision table, business rule representation, 239–242 Decision tree, business rule representation, 239, 241 Index Definition components, 61–62 dictionary role in information quality, 186 guidelines, 60–61 importance, 59–60 miscellaneous guidelines, 64 usage notes, 62–64 Delivery, business metadata examples corporate dictionary, 147–148 integrated technical and business metadata delivery, 152–153 mashups, 149–150 technical use, 151–152 training, 148–149 visual analytic techniques, 150–151 indirect usage accessibility from multiple places, 142–143 application access, 147 interactive reports, 145–146 overview, 141–142 Web delivery, 143–144 information quality business metadata, 188–190 infrastructure considerations business intelligence environments, 163 graphical affinity, 163–164 legacy environment, 162–163 mashups, 164 principles, 140–141 prospects, 280 Description Logics (DL), semantic framework, 206 Dictionary, see Glossary Direct access storage device (DASD), historical perspective, 4–6 Disk storage, historical perspective, 4–6 DL, see Description Logics Documents, metadata resource, 123 Editing, metadata automation, 128–129 expansion of definition and descriptions, 129–131 granularization, 129 homonym resolution, 132–134 manual editing, 134–135 staging area, 134 synonym resolution, 131–132 Employee turnover, business metadata capture rationale, 91–92 289 Enterprise resource planning (ERP), metadata resource, 85, 122 Entity/relationship (ER) model, semantic framework, 203–204, 208, 210–211 ER model, see Entity/relationship model ERP, see Enterprise resource planning ETL, see Extract-transform-load Extract-transform-load (ETL), metadata resource, 85, 124–125 Federated metadata, integrated metadata management, 168 Financial audit metadata utility, 250 transaction background activities, 250–251 First-order logic (FOL), semantic framework, 206 FOL, see First-order logic Folksonomy knowledge capture, 118–119 self-organizing tags, 77 Forms, see Corporate forms Fourth generation language historical perspective, 6–7 metadata handling, 11 Funding, business metadata advantages and disadvantages of approaches, 52–53 centralized implementation, 51 localized implementation, 51–52 overview, 50–51 Glossary business functions, 60 information quality role, 186 semantic framework, 201 Governance Lite™, knowledge capture, 107–109, 111 Granularization, metadata, 129 Graphical affinity, business metadata delivery infrastructure, 163–164 Grid, metadata representation, 18 Groupware, knowledge socialization, 100–103, 279 Homonyms, resolution, 132–134 Industrial recognition, text, 227 Information quality business and technical metadata interaction, 177–186 business metadata delivery, 188–190 290 Index Information quality (Continued) definition, 177 dictionary role, 186 methodology, 187–188 Information Technology (IT) department challenges, 278–279 metadata responsibility, 38–39 Information, definition, 177 IT department, see Information Technology department KB, see Knowledge base KM, see Knowledge management Knowledge base (KB) definition, 267 building, 267 Knowledge management (KM) business metadata intersection artifact generation, 262–263 corporate dictionary example, 263 definition, 260 goals, 260–261 importance, 261 social issues graying work force, 269–270 socialization effect on knowledge, 270 tacit knowledge, see Tacit knowledge techniques, 267–268 Knowledge socialization collective intelligence, 97 experts, 97–98 groupware, 100–103, 279 knowledge management, 268, 270 technology fostering portal and collaboration servers, 100–103 social networking, 99 wikis, 103–106 Knowledge worker metadata capture, 94 search problem, 65–66 Legacy systems business metadata delivery infrastructure, 162–163 metadata resource, 125–126 Library card catalog, metadata analogy, 27–29 Life cycle, metadata, 45–48 Magnetic tape data storage, 4 languages for data reading, 10 Mashup business metadata delivery, 149–150, 164 knowledge capture, 115–116 Master data management (MDM) conflict resolution, 167 overview, 22 MDM, see Master data management Metadata definition, 9, 26–27 examples, 9 grid representation, 18 management importance, 32 metamodel, 158–160 system of record example, 283–284 usage matrix example, 285–286 Metadata project business metadata versus technical metadata, 83–84 buying versus building, 170–172 classification, 82–83 funding, see Funding, business metadata iterations of development, 84 local metadata tools, 85 metadata sources, 86–87 preexisting repositories, 172–173 rationale, 80–82 scope defining, 85–87 Metadata Stewardship Council, responsibilities, 44–45 Narrower term, definition component, 61, 63–64 National Cancer Institute (NCI), semantic vocabulary implementation, 214–216 NCI, see National Cancer Institute Null, data profiling, 183, 185–186 ODS, see Operational data store OLAP, see Online analytical processing On-line analytical processing, metadata resource, 85124 On-line transaction processing, metadata resource, 125–126 Ontology, semantic framework, 207 Operational data store (ODS), data warehousing, 8 Opportunity cost, search problem quantification, 69 OWL, see Web Ontology Language Ownership, definition, 40 Patterns, identification, 183–184, 186 PC, see Personal computer Index Personal computer (PC) historical perspective, 7 metadata handling, 11 Preferred term, definition component, 62 Punch cards historical perspective, 4 metadata, 9–10 Quality, see Data quality; Information quality Range of values, data profiling, 183, 186 RDF, see Resource Definition Framework Reference file overview, 21–22 updating, 22 Regulations, see Compliance Related term, definition component, 61 Reports interactive reports, 145–146 metadata resource, 14, 29–31, 122–123 Repository, see Metadata project Resource Definition Framework (RDF), semantic framework, 204–205 Reuse, metadata, 32–33 Sales, search problem quantification, 70 Sarbanes-Oxley Act, provisions, 248–240 Screen, business metadata content, 13–14 Search problem enterprise search, 279 information and knowledge workers, 65–66 information provider guidelines, 71–72 quantification, 67–70 search techniques, 71 tracking down information, 66–67 Self-organizing map, linking structured and unstructured data, 233 Self-organizing tags, taxonomy, 77 Semantics business metadata delivering definitions and relationships, 208–209 exposing semantics to business, 210–211 expression, 209–210 overview, 207–208 context sensitivity, 197–199 framework Concept Map, 205, 209 conceptual model, 204 controlled vocabulary, 200–201 Description Logics, 206 291 entity/relationship model, 203–204, 208, 210–211 first-order logic, 206 glossary, 201 ontology, 207 Resource Definition Framework, 204–205 taxonomy, 202 thesauri, 203 topic map, 205 UML, 205–206 Web Ontology Language, 204–205, 210 human/computer concept, 199–200, 278 importance, 196–197 practical issues integration, 211–212 National Cancer Institute semantic vocabulary implementation, 214–216 service-oriented architecture, 214 Web Services, 212–213 prospects for integration and discovery, 280 semantic Web, 195–196 spectrum, 200–201, 208 Semistructured data, examples and technologies, 222–223 Serial transfer, knowledge management, 267–268 Service-oriented architecture (SOA) integrated metadata management, 168 semantics integration, 214 SOA, see Service-oriented architecture SOAP, semantic interface, 212 Socialization, see Knowledge socialization Social networking, knowledge socialization, 99 Spreadsheets, metadata resource, 123 Stemmed words, text distillation, 225 Stewardship business metadata approaches, 44–45 artifacts, 44 Data Governance Council, 42–43 Data Stewardship Council, 43–44 historical perspective, 41–42 Metadata Stewardship Council, 44–45 definition, 40–41 Structured metadata characteristics, 16–18 examples and technologies, 219–220 linking structured and unstructured data abstraction, 230–231 examples, 231, 233 292 Index Structured metadata (Continued) integration, 230 unstructured data comparison, 221 Synonyms, resolution, 131–132 Tacit knowledge definition, 94, 264 note-taking as asset producing, 265–266 transfer nurturing, 266 Taxonomy basic rules, 73, 75 document categorization, 76 governance and taxonomy, 77 language and vocabulary, 75–76 lowest common denominator, 75 overview, 72 self-organizing tags, 77 semantic framework, 202 simplicity, 76 Team Room, knowledge socialization, 100–103 Technical metadata business metadata comparison, 12–13 conversion, 135–136 infrastructure for integration, 165–166 separation, 140 categories, 2 Technical metadata conversion to business metadata, 135–136, 182–186 profiling, 179–182 Text business metadata terms, 227–228 communications audits, 252–253 distillation, 224–229 extraneous words, 225 industrial recognition, 227 pulling, 223 relationship recognition, 228–229 stemmed words, 225 word counting, 226 Thesauri, semantic framework, 203 Topic map, semantic framework, 205 UML, semantic framework, 205–206 Unstructured metadata characteristics, 16–18 examples and technologies, 220–221 mining prospects, 281 structured data comparison, 221 text business metadata terms, 227–228 distillation, 224–229 extraneous words, 225 industrial recognition, 227 pulling, 223 relationship recognition, 228–229 stemmed words, 225 word counting, 226 Value/frequency report, data profiling, 181, 184–186 Web 2.0 knowledge capture folksonomy, 118–119 mashups, 115–116 overview, 115 semantic Web, 195–196 Web Ontology Language (OWL), semantic framework, 204–205, 210 Web Services, semantics interface, 212–213 Wiki governance, 106 knowledge capture, 104 limitations, 105–106 portal collaboration comparison, 105 wikinomics, 104 Wikipedia, 103–104, 212 Words, see Text


pages: 509 words: 92,141

The Pragmatic Programmer by Andrew Hunt, Dave Thomas

A Pattern Language, Broken windows theory, business logic, business process, buy low sell high, c2.com, combinatorial explosion, continuous integration, database schema, domain-specific language, don't repeat yourself, Donald Knuth, Ford Model T, Free Software Foundation, general-purpose programming language, George Santayana, Grace Hopper, higher-order functions, if you see hoof prints, think horses—not zebras, index card, Kaizen: continuous improvement, lateral thinking, loose coupling, Menlo Park, MVC pattern, off-by-one error, premature optimization, Ralph Waldo Emerson, revision control, Schrödinger's Cat, slashdot, sorting algorithm, speech recognition, systems thinking, the Cathedral and the Bazaar, traveling salesman, urban decay, Y2K

The earlier a bug is found, the cheaper it is to remedy. "Code a little, test a little" is a popular saying in the Smalltalk world,[6] and we can adopt that mantra as our own by writing test code at the same time (or even before) we write the production code. [6] eXtreme Programming [URL 45] calls this concept "continuous integration, relentless testing." In fact, a good project may well have more test code than production code. The time it takes to produce this test code is worth the effort. It ends up being much cheaper in the long run, and you actually stand a chance of producing a product with close to zero defects.


pages: 322 words: 87,181

Straight Talk on Trade: Ideas for a Sane World Economy by Dani Rodrik

3D printing, airline deregulation, Asian financial crisis, bank run, barriers to entry, behavioural economics, Berlin Wall, Bernie Sanders, blue-collar work, Bretton Woods, BRICs, business cycle, call centre, capital controls, Capital in the Twenty-First Century by Thomas Piketty, carbon tax, Carmen Reinhart, carried interest, central bank independence, centre right, collective bargaining, conceptual framework, continuous integration, corporate governance, corporate social responsibility, currency manipulation / currency intervention, David Ricardo: comparative advantage, deindustrialization, Donald Trump, endogenous growth, Eugene Fama: efficient market hypothesis, eurozone crisis, export processing zone, failed state, financial deregulation, financial innovation, financial intermediation, financial repression, floating exchange rates, full employment, future of work, general purpose technology, George Akerlof, global value chain, income inequality, inflation targeting, information asymmetry, investor state dispute settlement, invisible hand, Jean Tirole, Kenneth Rogoff, low interest rates, low skilled workers, manufacturing employment, market clearing, market fundamentalism, meta-analysis, moral hazard, Nelson Mandela, new economy, offshore financial centre, open borders, open economy, open immigration, Pareto efficiency, postindustrial economy, precautionary principle, price stability, public intellectual, pushing on a string, race to the bottom, randomized controlled trial, regulatory arbitrage, rent control, rent-seeking, Richard Thaler, Robert Gordon, Robert Shiller, Ronald Reagan, Sam Peltzman, Silicon Valley, Solyndra, special economic zone, spectrum auction, Steven Pinker, tacit knowledge, The Rise and Fall of American Growth, the scientific method, The Wealth of Nations by Adam Smith, Thomas L Friedman, too big to fail, total factor productivity, trade liberalization, transaction costs, Tyler Cowen, unorthodox policies, Washington Consensus, World Values Survey, zero-sum game, éminence grise

Advanced democracies have built—and retain (despite recent setbacks)—extensive social safety nets in the form of unemployment insurance, retirement pensions, and family benefits. The world economy now has functional international institutions—such as the International Monetary Fund and the World Trade Organization—that it lacked prior to the Second World War. Global value chains have entrenched strong business lobbies for continued integration that even an instinctive protectionist like Donald Trump will find hard to overcome. Last but not least, truly extremist political movements such as fascism and communism have been largely discredited. Still, the conflicts between a hyperglobalized economy and social cohesion are real, and mainstream political elites ignore them at their peril.


pages: 313 words: 34,042

Tools for Computational Finance by Rüdiger Seydel

bioinformatics, Black-Scholes formula, Brownian motion, commoditize, continuous integration, discrete time, financial engineering, implied volatility, incomplete markets, interest rate swap, linear programming, London Interbank Offered Rate, mandelbrot fractal, martingale, random walk, risk free rate, stochastic process, stochastic volatility, transaction costs, value at risk, volatility smile, Wiener process, zero-coupon bond

.: Introduction to Étale Cohomology Tondeur, P.: Foliations on Riemannian Manifolds Toth, G.: Finite Möbius Groups, Minimal Immersions of Spheres, and Moduli Verhulst, F.: Nonlinear Differential Equations and Dynamical Systems Wong, M. W.: Weyl Transforms Xambó-Descamps, S.: Block Error-Correcting Codes Zaanen, A.C.: Continuity, Integration and Fourier Theory Zhang, F.: Matrix Theory Zong, C.: Sphere Packings Zong, C.: Strange Phenomena in Convex and Discrete Geometry Zorich, V. A.: Mathematical Analysis I Zorich, V. A.: Mathematical Analysis II


pages: 304 words: 22,886

Nudge: Improving Decisions About Health, Wealth, and Happiness by Richard H. Thaler, Cass R. Sunstein

Al Roth, Albert Einstein, asset allocation, availability heuristic, behavioural economics, call centre, carbon tax, Cass Sunstein, choice architecture, continuous integration, currency risk, Daniel Kahneman / Amos Tversky, desegregation, diversification, diversified portfolio, do well by doing good, endowment effect, equity premium, feminist movement, financial engineering, fixed income, framing effect, full employment, George Akerlof, index fund, invisible hand, late fees, libertarian paternalism, loss aversion, low interest rates, machine readable, Mahatma Gandhi, Mason jar, medical malpractice, medical residency, mental accounting, meta-analysis, Milgram experiment, money market fund, pension reform, presumed consent, price discrimination, profit maximization, rent-seeking, Richard Thaler, Right to Buy, risk tolerance, Robert Shiller, Saturday Night Live, school choice, school vouchers, systems thinking, Tragedy of the Commons, transaction costs, Vanguard fund, Zipcar

Despite the attention they receive in the media, market-based programs like vouchers are available to relatively few students nationwide. One popular alternative is a policy known as controlled choice, which emerged in the wake of 1970s court rulings prohibiting busing for the purpose of achieving desegregation. The idea was to continue integration by guaranteeing students a priority space at a nearby school or a school that a sibling attended, while giving them the option to apply for enrollment somewhere else. School administrators in Boston adopted a computer algorithm designed to assign as many students as possible to their first-choice schools, while still giving priority to the neighborhood students.


pages: 462 words: 172,671

Clean Code: A Handbook of Agile Software Craftsmanship by Robert C. Martin

business logic, continuous integration, database schema, disinformation, domain-specific language, don't repeat yourself, Donald Knuth, en.wikipedia.org, Eratosthenes, finite state, G4S, Ignaz Semmelweis: hand washing, iterative process, place-making, Rubik’s Cube, web application

How will we be able to discover if our code has failures when we do not know where to look? Here are a few ideas: • Monte Carlo Testing. Make tests flexible, so they can be tuned. Then run the test over and over—say on a test server—randomly changing the tuning values. If the tests ever fail, the code is broken. Make sure to start writing those tests early so a continuous integration server starts running them soon. By the way, make sure you carefully log the conditions under which the test failed. • Run the test on every one of the target deployment platforms. Repeatedly. Continuously. The longer the tests run without failure, the more likely that – The production code is correct or – The tests aren’t adequate to expose problems


pages: 405 words: 117,219

In Our Own Image: Savior or Destroyer? The History and Future of Artificial Intelligence by George Zarkadakis

3D printing, Ada Lovelace, agricultural Revolution, Airbnb, Alan Turing: On Computable Numbers, with an Application to the Entscheidungsproblem, animal electricity, anthropic principle, Asperger Syndrome, autonomous vehicles, barriers to entry, battle of ideas, Berlin Wall, bioinformatics, Bletchley Park, British Empire, business process, carbon-based life, cellular automata, Charles Babbage, Claude Shannon: information theory, combinatorial explosion, complexity theory, Computing Machinery and Intelligence, continuous integration, Conway's Game of Life, cosmological principle, dark matter, data science, deep learning, DeepMind, dematerialisation, double helix, Douglas Hofstadter, driverless car, Edward Snowden, epigenetics, Flash crash, Google Glasses, Gödel, Escher, Bach, Hans Moravec, income inequality, index card, industrial robot, intentional community, Internet of things, invention of agriculture, invention of the steam engine, invisible hand, Isaac Newton, Jacquard loom, Jacques de Vaucanson, James Watt: steam engine, job automation, John von Neumann, Joseph-Marie Jacquard, Kickstarter, liberal capitalism, lifelogging, machine translation, millennium bug, mirror neurons, Moravec's paradox, natural language processing, Nick Bostrom, Norbert Wiener, off grid, On the Economy of Machinery and Manufactures, packet switching, pattern recognition, Paul Erdős, Plato's cave, post-industrial society, power law, precautionary principle, prediction markets, Ray Kurzweil, Recombinant DNA, Rodney Brooks, Second Machine Age, self-driving car, seminal paper, Silicon Valley, social intelligence, speech recognition, stem cell, Stephen Hawking, Steven Pinker, Strategic Defense Initiative, strong AI, Stuart Kauffman, synthetic biology, systems thinking, technological singularity, The Coming Technological Singularity, The Future of Employment, the scientific method, theory of mind, Turing complete, Turing machine, Turing test, Tyler Cowen, Tyler Cowen: Great Stagnation, Vernor Vinge, Von Neumann architecture, Watson beat the top human players on Jeopardy!, Y2K

If the brain is made out of base logic units that process information, then intelligence ought to emerge from the interconnectedness between these units. The brain is therefore a cybernetic system. As Dehaene’s research into consciousness has shown, the brain uses feedback loops that pass information from neuron to neuron and from groups of neurons to groups of neurons. Sensory inputs from the nervous system are continuously integrated at a neural level. These integrations affect internal states in the brain, such as memories and thoughts. Intelligence is an emergent behaviour as the brain instructs the body how to react and respond to external stimuli. If this hypothesis is true then the brain can be replicated in any medium that can process information in a similar, granular, logic unit base fashion.


pages: 470 words: 109,589

Apache Solr 3 Enterprise Search Server by Unknown

bioinformatics, business logic, continuous integration, database schema, en.wikipedia.org, fault tolerance, Firefox, full text search, functional programming, information retrieval, natural language processing, performance metric, platform as a service, Ruby on Rails, SQL injection, Wayback Machine, web application

With luck, you may find some websites that will suffice, perhaps http://www.wolframapha.com. If your data changes in ways causing you to alter the constants in your function queries, then consider implementing a periodic automated test of your Solr data to ensure that the data fits within expected bounds. A Continuous Integration (CI) server might be configured to do this task. An approach is to run a search simply sorting by the data field in question to get the highest or lowest value. Formula: Logarithm The logarithm is a popular formula for inputs that grow without bounds, but the output is also unbounded.


pages: 514 words: 111,012

The Art of Monitoring by James Turnbull

Amazon Web Services, anti-pattern, business logic, cloud computing, continuous integration, correlation does not imply causation, Debian, DevOps, domain-specific language, failed state, functional programming, Kickstarter, Kubernetes, microservices, performance metric, pull request, Ruby on Rails, single source of truth, software as a service, source of truth, web application, WebSocket

Caitie has a degree in Computer Science from Cornell University, and has worked on several video games, including Gears of War 2, Gears of War 3, Halo 4, and Halo 5. She maintains a blog at CaitieM.com and frequently discusses technology on Twitter @Caitie. Paul Stack Paul Stack is an infrastructure coder and is passionate about continuous integration, continuous delivery, and good operational procedures—and how they should be part of what developers and system administrators do on a day-to-day basis. He believes that reliably delivering software is as important as its development. Jamie Wilkinson Jamie is a Site Reliability Engineer in Google's Storage Infrastructure team.


pages: 426 words: 117,027

Mind in Motion: How Action Shapes Thought by Barbara Tversky

Apple's 1984 Super Bowl advert, Asperger Syndrome, augmented reality, clean water, cognitive load, continuous integration, double helix, en.wikipedia.org, fundamental attribution error, Hans Rosling, Intergovernmental Panel on Climate Change (IPCC), John Snow's cholera map, Lao Tzu, meta-analysis, mirror neurons, natural language processing, neurotypical, patient HM, Richard Feynman, Steven Pinker, TED Talk, the new new thing, theory of mind, urban planning

Or why they seem so active at importune times—one of them kept popping my dress up and down during my PhD orals. Mercifully, bodies do far more than kick. They eventually perform an astounding assortment of activities. The harmonious coordination underlying those diverse behaviors depends on the continuous integration of a variable stream of information from many senses with the articulated actions of dozens of muscles (apologies for beginning with such a mouthful!). Although our skin encloses and separates our bodies from the surrounding world, accomplishing those activities entails countless interactions with the world.


pages: 395 words: 110,994

The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win by Gene Kim, Kevin Behr, George Spafford

air freight, anti-work, antiwork, Apollo 13, business intelligence, business process, centre right, cloud computing, continuous integration, dark matter, database schema, DevOps, fail fast, friendly fire, Gene Kranz, index card, information security, inventory management, Kanban, Lean Startup, shareholder value, systems thinking, Toyota Production System

Otherwise, some Chinese company will steal our idea, have them on our competitors store shelves, and take the majority of the market. “In these competitive times, the name of the game is quick time to market and to fail fast. We just can’t have multiyear product development timelines, waiting until the end to figure out whether we have a winner or loser on our hands. We need short and quick cycle times to continually integrate feedback from the marketplace. “But that’s just half the picture,” she continues. “The longer the product development cycle, the longer the company capital is locked up and not giving us a return. Dick expects that on average, our R&D investments return more than ten percent. That’s the internal hurdle rate.


pages: 350 words: 114,454

Docker: Up & Running: Shipping Reliable Containers in Production by Sean P. Kane, Karl Matthias

Airbnb, Amazon Web Services, business logic, business process, cloud computing, Colossal Cave Adventure, continuous integration, Debian, DevOps, don't repeat yourself, false flag, interchangeable parts, Kubernetes, loose coupling, Lyft, microservices, revision control, software as a service, source of truth, web application

That container might then also be tested against any outside dependencies like databases or caches without having to mock them. None of this guarantees success, but it gets them a lot closer to that guarantee than the dependency roulette often experienced by production deployment systems. If you use Jenkins for continuous integration or are looking for a good way to test scaling Docker, there are many plug-ins for Docker, Mesos, and Kubernetes that are worth investigating. Outside Dependencies But what about those external dependencies we glossed over? Things like the data‐ base, or Memcache or Redis instances that we need to run our tests against our con‐ tainer?


pages: 312 words: 93,504

Common Knowledge?: An Ethnography of Wikipedia by Dariusz Jemielniak

Andrew Keen, barriers to entry, Benevolent Dictator For Life (BDFL), citation needed, collaborative consumption, collaborative editing, commons-based peer production, conceptual framework, continuous integration, crowdsourcing, Debian, deskilling, digital Maoism, disinformation, en.wikipedia.org, Filter Bubble, Free Software Foundation, Gabriella Coleman, Google Glasses, Guido van Rossum, Hacker Ethic, hive mind, Internet Archive, invisible hand, Jaron Lanier, jimmy wales, job satisfaction, Julian Assange, knowledge economy, knowledge worker, Menlo Park, moral hazard, online collectivism, pirate software, RFC: Request For Comment, Richard Stallman, selection bias, Silicon Valley, Skype, slashdot, social software, Stewart Brand, the Cathedral and the Bazaar, The Hackers Conference, The Nature of the Firm, the strength of weak ties, The Wisdom of Crowds, transaction costs, Wayback Machine, WikiLeaks, wikimedia commons, Wikivoyage, Yochai Benkler, zero-sum game

., & Speckbacher, G. (2010). The effect of interorganizational trust on make-or-cooperate decisions: Disentangling opportunism-dependent 2 5 6    R e f e r e n c e s and opportunism-independent effects of trust. European Management Review, 7(2), 101–115. Holck, J., & Jørgensen, N. (2007). Continuous integration and quality assurance: A case study of two open source projects. Australasian Journal of Information Systems, 11(1), 40–53. Hollander, E. P. (1958). Conformity, status, and idiosyncrasy credit. Psychological Review, 65(2), 117–127. Hollander, E. P. (1992). The essential interdependence of leadership and followership.


pages: 960 words: 125,049

Mastering Ethereum: Building Smart Contracts and DApps by Andreas M. Antonopoulos, Gavin Wood Ph. D.

air gap, Amazon Web Services, bitcoin, blockchain, business logic, continuous integration, cryptocurrency, Debian, digital divide, Dogecoin, domain-specific language, don't repeat yourself, Edward Snowden, en.wikipedia.org, Ethereum, ethereum blockchain, fault tolerance, fiat currency, Firefox, functional programming, Google Chrome, information security, initial coin offering, intangible asset, Internet of things, litecoin, machine readable, move fast and break things, node package manager, non-fungible token, peer-to-peer, Ponzi scheme, prediction markets, pull request, QR code, Ruby on Rails, Satoshi Nakamoto, sealed-bid auction, sharing economy, side project, smart contracts, transaction costs, Turing complete, Turing machine, Vickrey auction, Vitalik Buterin, web application, WebSocket

(yes) Now, we can install the dependencies that we will use to make working with truffle easier: $ npm install dotenv truffle-wallet-provider ethereumjs-wallet We now have a node_modules directory with several thousand files inside our Faucet directory. Prior to deploying a DApp to a cloud production or continuous integration environment, it is important to specify the engines field so that your DApp is built with the correct Node.js version and its associated dependencies are installed. For details on configuring this field, see the documentation. Configuring truffle truffle creates some empty configuration files, truffle.js and truffle-config.js.


pages: 580 words: 125,129

Androids: The Team That Built the Android Operating System by Chet Haase

Andy Rubin, Apple II, Apple's 1984 Super Bowl advert, augmented reality, barriers to entry, Beos Apple "Steve Jobs" next macos , Big Tech, Bill Atkinson, commoditize, continuous integration, crowdsourcing, en.wikipedia.org, fault tolerance, Firefox, General Magic , Google Chrome, Ken Thompson, lock screen, machine readable, Menlo Park, PalmPilot, Parkinson's law, pull request, QWERTY keyboard, side project, Silicon Valley, skunkworks, speech recognition, stealth mode startup, Steve Ballmer, Steve Jobs, Steve Wozniak, Tony Fadell, turn-by-turn navigation, web application

Recursion is a powerful technique, but can be tricky to think through, and to ensure that it will actually terminate. 130 Taligent was a company formed by Apple and IBM with the goal of providing a new operating system, at a time when Apple was trying to come up with a successor to the aging MacOS. Taligent eventually failed and Apple continued its attempts internally before eventually acquiring Steve Jobs’s NeXT Computer and adopting NeXTSTEP OS instead. 131 Continuous Integration, or CI, is the practice in software development of integrating all of a team’s changes as often as possible for building and testing. It helps maintain a constant measure of the quality and stability of the product so that it doesn’t spin out of control before someone notices. 132 Ideally, you have tests that run automatically, all the time, to make sure that changes don’t break things.


Version Control With Git: Powerful Tools and Techniques for Collaborative Software Development by Jon Loeliger, Matthew McCullough

continuous integration, Debian, distributed revision control, GnuPG, Larry Wall, peer-to-peer, peer-to-peer model, pull request, revision control, Snow Crash, web application, web of trust

That code was stored as a Git submodule and when the superproject was cloned, the permissions denied the majority of developers from being able to clone that submodule. The build system for this project was carefully constructed to adapt to the missing source of the cryptographic component, outputting a developer-only build. The SSH key of the continuous integration server, on the other hand, does have permission to retrieve the cryptography submodule, thus producing the feature-complete builds that customers will ultimately receive. Multilevel Nesting of Repos The use of submodules discussed thus far can be extended to another level of recursion. submodules can in turn be superprojects, and thus contain submodules.


pages: 298 words: 151,238

Excession by Iain M. Banks

continuous integration, gravity well, heat death of the universe, hive mind, place-making

However, I have no evidence to the contrary. I have given my word and I will not go public with all this, but I will consider that agreement dependent on the continued well-being and freedom from persecution of both theSerious Callers Only and theShoot Them Later , as well, of course, as being contingent upon my own continued integrity. I don't doubt you will think me either paranoid or ridiculous for systematising this arrangement with various other friends and colleagues, particularly given the hostilities which commenced yesterday. I am thinking of taking some sabbatical time myself soon, and going off course-schedule.


The Volatility Smile by Emanuel Derman,Michael B.Miller

Albert Einstein, Asian financial crisis, Benoit Mandelbrot, Black Monday: stock market crash in 1987, book value, Brownian motion, capital asset pricing model, collateralized debt obligation, continuous integration, Credit Default Swap, credit default swaps / collateralized debt obligations, discrete time, diversified portfolio, dividend-yielding stocks, Emanuel Derman, Eugene Fama: efficient market hypothesis, financial engineering, fixed income, implied volatility, incomplete markets, law of one price, London Whale, mandelbrot fractal, market bubble, market friction, Myron Scholes, prediction markets, quantitative trading / quantitative finance, risk tolerance, riskless arbitrage, Sharpe ratio, statistical arbitrage, stochastic process, stochastic volatility, transaction costs, volatility arbitrage, volatility smile, Wiener process, yield curve, zero-coupon bond

Denote a standard Brownian motion evaluated at time t by B(t), where B(t0 ) = 0 E[dB(t)] = 0 E[dB(t)2 ] = dt (B.4a) (B.4b) (B.4c) 425 Appendix B: Backward Itô Integrals Forward Approach As an example, we would like to approximate the continuous stochastic Itô integral t ∫0 B(s)dB(s) We can approximate the continuous integral as a summation using our forward approach as follows: AF (n) = n−1 ∑ [B(ti+1 ) − B(ti )]B(ti ) (B.5) i=0 Here, t0 = 0 and tn = t. It might seem counterproductive at first, but we can rewrite each term in the summation as follows: ] 1 1[ B(ti+1 )2 − B(ti )2 − [B(ti+1 ) − B(ti )]2 2 2 [B(ti+1 ) − B(ti )]B(ti ) = (B.6) Substituting into Equation B.5 and rearranging, we obtain AF (n) = n−1 ∑ ] 1 n−1 1 ∑[ B(ti+1 )2 − B(ti )2 − [B(ti+1 ) − B(ti )]2 2 i=0 2 i=0 (B.7) Most of the terms in the first summation cancel each other out, leaving ∑ ] 1 n−1 1[ 2 2 B(tn ) − B(t0 ) − AF (n) = [B(ti+1 ) − B(ti )]2 2 2 i=0 n−1 1 1∑ = B(t)2 − [B(ti+1 ) − B(ti )]2 2 2 i=0 (B.8) In the limit, as n increases and the width of each rectangle decreases, ]] [[ lim E {B(ti+1 ) − B(ti )}2 = E[dB(t)2 ] = dt = ti+1 − ti n→∞ (B.9) In the limit, the remaining summation in Equation B.8 is then lim n→∞ n−1 ∑ n−1 ∑ i=0 i=0 [B(ti+1 ) − B(ti )]2 = ti+1 − ti (B.10) 426 APPENDIX B: BACKWARD ITÔ INTEGRALS As before, most of the terms in the summation cancel, leaving us with lim n→∞ n−1 ∑ [B(ti+1 ) − B(ti )]2 = tn − t0 = t (B.11) i=0 Substituting back into Equation B.8, we now have: 1 1 B(t)2 − t 2 2 lim AF (n) = n→∞ (B.12) The solution to our Itô integral is then: t ∫0 B(s)dB(s) = 1 1 B(t)2 − t 2 2 (B.13) Throughout the rest of the book, unless noted otherwise, when we refer to an Itô integral, or a standard Itô integral, we mean an Itô integral based on the forward approach as shown here.


pages: 578 words: 168,350

Scale: The Universal Laws of Growth, Innovation, Sustainability, and the Pace of Life in Organisms, Cities, Economies, and Companies by Geoffrey West

"World Economic Forum" Davos, Alfred Russel Wallace, Anthropocene, Anton Chekhov, Benoit Mandelbrot, Black Swan, British Empire, butterfly effect, caloric restriction, caloric restriction, carbon footprint, Cesare Marchetti: Marchetti’s constant, clean water, coastline paradox / Richardson effect, complexity theory, computer age, conceptual framework, continuous integration, corporate social responsibility, correlation does not imply causation, cotton gin, creative destruction, dark matter, Deng Xiaoping, double helix, driverless car, Dunbar number, Edward Glaeser, endogenous growth, Ernest Rutherford, first square of the chessboard, first square of the chessboard / second half of the chessboard, Frank Gehry, Geoffrey West, Santa Fe Institute, Great Leap Forward, Guggenheim Bilbao, housing crisis, Index librorum prohibitorum, invention of agriculture, invention of the telephone, Isaac Newton, Jane Jacobs, Jeff Bezos, Johann Wolfgang von Goethe, John von Neumann, Kenneth Arrow, laissez-faire capitalism, Large Hadron Collider, Larry Ellison, Lewis Mumford, life extension, Mahatma Gandhi, mandelbrot fractal, Marc Benioff, Marchetti’s constant, Masdar, megacity, Murano, Venice glass, Murray Gell-Mann, New Urbanism, Oklahoma City bombing, Peter Thiel, power law, profit motive, publish or perish, Ray Kurzweil, Richard Feynman, Richard Florida, Salesforce, seminal paper, Silicon Valley, smart cities, Stephen Hawking, Steve Jobs, Stewart Brand, Suez canal 1869, systematic bias, systems thinking, technological singularity, The Coming Technological Singularity, The Death and Life of Great American Cities, the scientific method, the strength of weak ties, time dilation, too big to fail, transaction costs, urban planning, urban renewal, Vernor Vinge, Vilfredo Pareto, Von Neumann architecture, Whole Earth Catalog, Whole Earth Review, wikimedia commons, working poor

Consequently, each of these urban characteristics, each metric—whether wages, the length of all the roads, the number of AIDS cases, or the amount of crime—is interrelated and interconnected with every other one and together they form an overarching multiscale quintessentially complex adaptive system that is continuously integrating and processing energy, resources, and information. The result is the extraordinary collective phenomenon we call a city, whose origins emerge from the underlying dynamics and organization of how people interact with one another through social networks. To repeat: cities are an emergent self-organizing phenomenon that has resulted from the interaction and communication between human beings exchanging energy, resources, and information.


pages: 680 words: 157,865

Beautiful Architecture: Leading Thinkers Reveal the Hidden Beauty in Software Design by Diomidis Spinellis, Georgios Gousios

Albert Einstein, barriers to entry, business intelligence, business logic, business process, call centre, continuous integration, corporate governance, database schema, Debian, domain-specific language, don't repeat yourself, Donald Knuth, duck typing, en.wikipedia.org, fail fast, fault tolerance, financial engineering, Firefox, Free Software Foundation, functional programming, general-purpose programming language, higher-order functions, iterative process, linked data, locality of reference, loose coupling, meta-analysis, MVC pattern, Neal Stephenson, no silver bullet, peer-to-peer, premature optimization, recommendation engine, Richard Stallman, Ruby on Rails, semantic web, smart cities, social graph, social web, SPARQL, Steve Jobs, Stewart Brand, Strategic Defense Initiative, systems thinking, the Cathedral and the Bazaar, traveling salesman, Turing complete, type inference, web application, zero-coupon bond

Decisions about some of the basic concerns were made at this point to ensure that the code would grow easily and cohesively, including: The top-level file structure How we would name things A “house” presentation style Common coding idioms The choice of unit test framework The supporting infrastructure (e.g., source control, a suitable build system, and continuous integration) These “fine detail” factors were very important: they allied closely with the software architecture and, in turn, influenced many later design decisions. The Story Unfolds Once the initial design had been established by the team, the Design Town project proceeded following the XP process.


pages: 725 words: 168,262

API Design Patterns by Jj Geewax

Amazon Web Services, anti-pattern, bitcoin, blockchain, business logic, cognitive load, continuous integration, COVID-19, database schema, en.wikipedia.org, exponential backoff, imposter syndrome, Internet of things, Kubernetes, lateral thinking, loose coupling, machine readable, microservices, natural language processing, Paradox of Choice, ride hailing / ride sharing, social graph, sorting algorithm

This leads to a problem: how do we make these improvements without seriously inconveniencing users who have come to depend on things looking and acting as they did before? In this chapter, we’ll explore the most common mechanism for addressing this problem: versioning. 24.1 Motivation Not only is software development rarely static, it is also rarely a continuous process. Even with continuous integration and deployment, we often have checkpoints or launches where some new functionality “goes live” and is visible to users, which holds especially true for web APIs. This need is exacerbated by the fact that APIs are both rigid and public, meaning changes are difficult to make safely. This leads to an obvious question: how can we make changes to a web API without causing damage to those using the API already?


pages: 719 words: 181,090

Site Reliability Engineering: How Google Runs Production Systems by Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy

"Margaret Hamilton" Apollo, Abraham Maslow, Air France Flight 447, anti-pattern, barriers to entry, business intelligence, business logic, business process, Checklist Manifesto, cloud computing, cognitive load, combinatorial explosion, continuous integration, correlation does not imply causation, crowdsourcing, database schema, defense in depth, DevOps, en.wikipedia.org, exponential backoff, fail fast, fault tolerance, Flash crash, George Santayana, Google Chrome, Google Earth, if you see hoof prints, think horses—not zebras, information asymmetry, job automation, job satisfaction, Kubernetes, linear programming, load shedding, loose coupling, machine readable, meta-analysis, microservices, minimum viable product, MVC pattern, no silver bullet, OSI model, performance metric, platform as a service, proprietary trading, reproducible builds, revision control, risk tolerance, side project, six sigma, the long tail, the scientific method, Toyota Production System, trickle-down economics, warehouse automation, web application, zero day

Of course, any high-level programming environment creates the opportunity for complexity, so Borgmon provides a way to build extensive unit and regression tests by synthesizing time-series data, in order to ensure that the rules behave as the author thinks they do. The Production Monitoring team runs a continuous integration service that executes a suite of these tests, packages the configuration, and ships the configuration to all the Borgmon in production, which then validate the configuration before accepting it. In the vast library of common templates that have been created, two classes of monitoring configuration have emerged.


pages: 823 words: 206,070

The Making of Global Capitalism by Leo Panitch, Sam Gindin

accounting loophole / creative accounting, active measures, airline deregulation, Alan Greenspan, anti-communist, Asian financial crisis, asset-backed security, bank run, banking crisis, barriers to entry, Basel III, Bear Stearns, Big bang: deregulation of the City of London, bilateral investment treaty, book value, Branko Milanovic, Bretton Woods, BRICs, British Empire, business cycle, call centre, capital controls, Capital in the Twenty-First Century by Thomas Piketty, carbon credits, Carmen Reinhart, central bank independence, classic study, collective bargaining, continuous integration, corporate governance, creative destruction, Credit Default Swap, crony capitalism, currency manipulation / currency intervention, currency peg, dark matter, democratizing finance, Deng Xiaoping, disintermediation, ending welfare as we know it, eurozone crisis, facts on the ground, financial deregulation, financial innovation, Financial Instability Hypothesis, financial intermediation, floating exchange rates, foreign exchange controls, full employment, Gini coefficient, Glass-Steagall Act, global value chain, guest worker program, Hyman Minsky, imperial preference, income inequality, inflation targeting, interchangeable parts, interest rate swap, Kenneth Rogoff, Kickstarter, land reform, late capitalism, liberal capitalism, liquidity trap, London Interbank Offered Rate, Long Term Capital Management, low interest rates, manufacturing employment, market bubble, market fundamentalism, Martin Wolf, means of production, military-industrial complex, money market fund, money: store of value / unit of account / medium of exchange, Monroe Doctrine, moral hazard, mortgage debt, mortgage tax deduction, Myron Scholes, new economy, Nixon triggered the end of the Bretton Woods system, non-tariff barriers, Northern Rock, oil shock, precariat, price stability, proprietary trading, quantitative easing, Ralph Nader, RAND corporation, regulatory arbitrage, reserve currency, risk tolerance, Ronald Reagan, Savings and loan crisis, scientific management, seigniorage, shareholder value, short selling, Silicon Valley, sovereign wealth fund, special drawing rights, special economic zone, stock buybacks, structural adjustment programs, subprime mortgage crisis, Tax Reform Act of 1986, The Chicago School, The Great Moderation, the payments system, The Wealth of Nations by Adam Smith, too big to fail, trade liberalization, transcontinental railway, trickle-down economics, union organizing, vertical integration, very high income, Washington Consensus, We are all Keynesians now, Works Progress Administration, zero-coupon bond, zero-sum game

This pattern did not change all that much until the 1980s, when the political conditions were established—in the North as well as increasingly in the South—that laid the grounds for a truly global capitalism. Integrating Europe The accelerated push towards economic and monetary union in the 1980s, emerging at a time when Europe was mired in internal stagnation, needs to be understood in the context of the continuing integration of European and American capitalism. The abandonment of the Bretton Woods framework, wherein all European currencies had been fixed in a hub-and-spokes relationship to the dollar, was initially compensated for by the European states adopting a “currency snake” designed to prevent competitive devaluations; but “the instability of the snake reflected the fact that the 1970s was a low point for European cooperation.”1 The formation of the European Monetary System in 1979—“the most significant development in the EC arising out of the long crisis from 1969 to 1983”—was the first major step towards the common currency, a development that the US did not oppose.2 What it was much more concerned about, even in the wake of the election that brought Mrs.


pages: 999 words: 194,942

Clojure Programming by Chas Emerick, Brian Carper, Christophe Grand

Amazon Web Services, Benoit Mandelbrot, cloud computing, cognitive load, continuous integration, database schema, domain-specific language, don't repeat yourself, drop ship, duck typing, en.wikipedia.org, failed state, finite state, Firefox, functional programming, game design, general-purpose programming language, Guido van Rossum, higher-order functions, Larry Wall, mandelbrot fractal, no silver bullet, Paul Graham, platform as a service, premature optimization, random walk, Ruby on Rails, Schrödinger's Cat, semantic web, software as a service, sorting algorithm, SQL injection, Turing complete, type inference, web application

Clojure web applications will run side by side with your other Java web applications without a hitch, you can call existing Java libraries from code you write in Clojure, and you can call Clojure functions and create instances of Clojure types from code you write in Java (or any other language that runs on the JVM, including JRuby, Jython, JavaScript [via Rhino], Scala, Groovy, and so on). Everything you’ve learned about builds, packaging, continuous integration, and JVM operations and tuning applies to the work you’ll do with Clojure. Clojure is an incremental addition to your existing JVM investment, not a radical departure. “Clojure is just another .jar file”. A corollary of the fact that Clojure lets you reuse your investments in Java is that Clojure really is “just another .jar file.”


The Code: Silicon Valley and the Remaking of America by Margaret O'Mara

A Declaration of the Independence of Cyberspace, accounting loophole / creative accounting, affirmative action, Airbnb, Alan Greenspan, AltaVista, Alvin Toffler, Amazon Web Services, An Inconvenient Truth, AOL-Time Warner, Apple II, Apple's 1984 Super Bowl advert, autonomous vehicles, back-to-the-land, barriers to entry, Ben Horowitz, Berlin Wall, Big Tech, Black Lives Matter, Bob Noyce, Buckminster Fuller, Burning Man, business climate, Byte Shop, California gold rush, Californian Ideology, carried interest, clean tech, clean water, cloud computing, cognitive dissonance, commoditize, company town, Compatible Time-Sharing System, computer age, Computer Lib, continuous integration, cuban missile crisis, Danny Hillis, DARPA: Urban Challenge, deindustrialization, different worldview, digital divide, Do you want to sell sugared water for the rest of your life?, don't be evil, Donald Trump, Doomsday Clock, Douglas Engelbart, driverless car, Dynabook, Edward Snowden, El Camino Real, Elon Musk, en.wikipedia.org, Erik Brynjolfsson, Fairchild Semiconductor, Frank Gehry, Future Shock, Gary Kildall, General Magic , George Gilder, gig economy, Googley, Hacker Ethic, Hacker News, high net worth, hockey-stick growth, Hush-A-Phone, immigration reform, income inequality, industrial research laboratory, informal economy, information retrieval, invention of movable type, invisible hand, Isaac Newton, It's morning again in America, Jeff Bezos, Joan Didion, job automation, job-hopping, John Gilmore, John Markoff, John Perry Barlow, Julian Assange, Kitchen Debate, knowledge economy, knowledge worker, Larry Ellison, Laura Poitras, Lyft, Marc Andreessen, Mark Zuckerberg, market bubble, Mary Meeker, mass immigration, means of production, mega-rich, Menlo Park, Mikhail Gorbachev, military-industrial complex, millennium bug, Mitch Kapor, Mother of all demos, move fast and break things, mutually assured destruction, Neil Armstrong, new economy, Norbert Wiener, old-boy network, Palm Treo, pattern recognition, Paul Graham, Paul Terrell, paypal mafia, Peter Thiel, pets.com, pirate software, popular electronics, pre–internet, prudent man rule, Ralph Nader, RAND corporation, Richard Florida, ride hailing / ride sharing, risk tolerance, Robert Metcalfe, ROLM, Ronald Reagan, Salesforce, Sand Hill Road, Second Machine Age, self-driving car, shareholder value, Sheryl Sandberg, side hustle, side project, Silicon Valley, Silicon Valley ideology, Silicon Valley startup, skunkworks, Snapchat, social graph, software is eating the world, Solyndra, speech recognition, Steve Ballmer, Steve Jobs, Steve Wozniak, Steven Levy, Stewart Brand, Strategic Defense Initiative, supercomputer in your pocket, Susan Wojcicki, tacit knowledge, tech billionaire, tech worker, technoutopianism, Ted Nelson, TED Talk, the Cathedral and the Bazaar, the market place, the new new thing, The Soul of a New Machine, There's no reason for any individual to have a computer in his home - Ken Olsen, Thomas L Friedman, Tim Cook: Apple, Timothy McVeigh, transcontinental railway, Twitter Arab Spring, Uber and Lyft, uber lyft, Unsafe at Any Speed, upwardly mobile, Vannevar Bush, War on Poverty, Wargames Reagan, WarGames: Global Thermonuclear War, We wanted flying cars, instead we got 140 characters, Whole Earth Catalog, WikiLeaks, William Shockley: the traitorous eight, work culture , Y Combinator, Y2K

Educational experts questioned the relevance of teaching with paper and textbooks in an age when nearly every American family had a television in their living room, and enthusiastically endorsed the idea of introducing programmable “teaching machines” into the classroom. On top of it all, American schools remained civil rights battlegrounds, as continuing integration struggles had given way to fiercely contested court-ordered busing. These measures prompted some white parents to violently resist, and many others to opt out of public school systems altogether. Alternative schooling movements like Montessori and Waldorf spiked in popularity.8 A growing number of educational advocates had begun talking about computers as critical features of the new and improved schools of the future.


pages: 1,409 words: 205,237

Architecting Modern Data Platforms: A Guide to Enterprise Hadoop at Scale by Jan Kunigk, Ian Buss, Paul Wilkinson, Lars George

Amazon Web Services, barriers to entry, bitcoin, business intelligence, business logic, business process, cloud computing, commoditize, computer vision, continuous integration, create, read, update, delete, data science, database schema, Debian, deep learning, DevOps, domain-specific language, fault tolerance, Firefox, FOSDEM, functional programming, Google Chrome, Induced demand, information security, Infrastructure as a Service, Internet of things, job automation, Kickstarter, Kubernetes, level 1 cache, loose coupling, microservices, natural language processing, Network effects, platform as a service, single source of truth, source of truth, statistical model, vertical integration, web application

For scalable productive BI reports, the analyst uses a range of algorithms on multidimensional views on the data in tabular form (also known as cubes) based on star or snowflake database schemas in the data warehouse. This achieves a high degree of abstraction for the analyst, who focuses on the data algorithms that often have been optimized for cubes in the BI solution. Software developer Software developers not only write, build, and maintain code, but also are involved with the continuous integration and operation of the underlying infrastructure. In our example, a software developer would likely be involved in providing the presentation layer of the overall solution, producing dashboards according to the requirements of the analyst team. Note So as to avoid confusion with the term systems engineer (introduced in the text that follows), we do not use the term software engineer.


Understanding Power by Noam Chomsky

anti-communist, Ayatollah Khomeini, Berlin Wall, Bretton Woods, British Empire, Burning Man, business climate, business cycle, cognitive dissonance, continuous integration, Corn Laws, cuban missile crisis, dark matter, David Ricardo: comparative advantage, deindustrialization, Deng Xiaoping, deskilling, disinformation, European colonialism, Fall of the Berlin Wall, feminist movement, gentrification, global reserve currency, guns versus butter model, Howard Zinn, junk bonds, Korean Air Lines Flight 007, liberation theology, Mahatma Gandhi, Mikhail Gorbachev, military-industrial complex, Monroe Doctrine, mortgage tax deduction, Nixon triggered the end of the Bretton Woods system, Paul Samuelson, Ralph Nader, reserve currency, Ronald Reagan, Rosa Parks, school choice, Strategic Defense Initiative, strikebreaker, structural adjustment programs, systems thinking, the scientific method, The Wealth of Nations by Adam Smith, union organizing, wage slave, women in the workforce

It would be like asking the New York City police force whether they would like to turn Harlem over to local mercenaries to patrol, while they hold on to Wall Street, the Upper East Side, Madison Avenue, and so on—if you asked the New York City police force that, I’m sure they’d be delighted. Who wants to patrol Harlem? Well, that’s in effect what’s happening in the Occupied Territories right now: the idea is, see if you can get local mercenaries, who are still always under your whip, to run the place for you, while you continue integrating the area into Israel. Actually, some Israeli commentators have used the term “neocolonialism” to describe what’s being done with the Territories, and that’s essentially correct, I think. 114 In fact, I think what’s been taking place in the Middle East is really just a part of something much broader that’s happened throughout the West in recent years, particularly since the Gulf War: there’s been a real revival of traditional European racism and imperialism, in a very dramatic way.


pages: 496 words: 174,084

Masterminds of Programming: Conversations With the Creators of Major Programming Languages by Federico Biancuzzi, Shane Warden

Benevolent Dictator For Life (BDFL), business intelligence, business logic, business process, cellular automata, cloud computing, cognitive load, commoditize, complexity theory, conceptual framework, continuous integration, data acquisition, Dennis Ritchie, domain-specific language, Douglas Hofstadter, Fellow of the Royal Society, finite state, Firefox, follow your passion, Frank Gehry, functional programming, general-purpose programming language, Guido van Rossum, higher-order functions, history of Unix, HyperCard, industrial research laboratory, information retrieval, information security, iterative process, Ivan Sutherland, John von Neumann, Ken Thompson, Larry Ellison, Larry Wall, linear programming, loose coupling, machine readable, machine translation, Mars Rover, millennium bug, Multics, NP-complete, Paul Graham, performance metric, Perl 6, QWERTY keyboard, RAND corporation, randomized controlled trial, Renaissance Technologies, Ruby on Rails, Sapir-Whorf hypothesis, seminal paper, Silicon Valley, slashdot, software as a service, software patent, sorting algorithm, SQL injection, Steve Jobs, traveling salesman, Turing complete, type inference, Valgrind, Von Neumann architecture, web application

This view should penetrate everything you do and the practices that you deploy when developing software. There are basically two approaches to managing legacy systems and improve them. The first is to just deploy practices that don’t really change the product but improve the way you work, such as iterative development, continuous integration, test-driven development, use-case-driven development, user stories, pair programming, and cross-cutting teams. The cost and risks of introducing such practices are marginal, but for big companies still substantial. The second approach is more fundamental: change the actual product via practices such as architecture (at a simple level), enterprise architecture, product-line architecture, components, etc.


pages: 846 words: 250,145

The Cold War: A World History by Odd Arne Westad

Able Archer 83, Albert Einstein, American ideology, anti-communist, Ayatollah Khomeini, Berlin Wall, Bolshevik threat, Bretton Woods, British Empire, capital controls, collective bargaining, colonial rule, continuous integration, cuban missile crisis, Deng Xiaoping, disinformation, Dissolution of the Soviet Union, energy security, European colonialism, facts on the ground, failed state, Fall of the Berlin Wall, financial deregulation, full employment, Great Leap Forward, household responsibility system, imperial preference, Internet Archive, land reform, Les Trente Glorieuses, liberal capitalism, long peace, means of production, Mikhail Gorbachev, military-industrial complex, mutually assured destruction, Nelson Mandela, new economy, Nixon shock, Nixon triggered the end of the Bretton Woods system, oil shock, out of africa, post-industrial society, Ronald Reagan, Ronald Reagan: Tear down this wall, South China Sea, special economic zone, Strategic Defense Initiative, Suez crisis 1956, union organizing, urban planning, War on Poverty, women in the workforce, Yom Kippur War, young professional, zero-sum game

.… I think those sons of bitches want to eat us any day they can.”15 Johnson did believe, however, that Germany was less of an immediate Cold War issue because of West Germany’s safe anchoring in NATO. While de Gaulle huffed and hawed and students—not least in West Germany—protested against US imperialism, both main parties in the Federal Republic, the Christian Democrats and the Social Democrats, saw their country’s continued integration with the West as crucial for Germany’s future. Indeed, de Gaulle’s insistence on building his “new Europe” around a French-German axis seemed to confirm West Germany’s place. Western European economic integration became an instrument both for further growth and for Cold War cohesion. Increasingly, the European integration project had West Germany’s spectacular industrial and commercial success as its center.


pages: 870 words: 259,362

Austerity Britain: 1945-51 by David Kynaston

Alistair Cooke, anti-communist, Arthur Marwick, British Empire, Chelsea Manning, collective bargaining, continuous integration, deindustrialization, deskilling, Etonian, full employment, garden city movement, hiring and firing, industrial cluster, invisible hand, job satisfaction, labour mobility, Lewis Mumford, light touch regulation, mass immigration, moral panic, Neil Kinnock, occupational segregation, price mechanism, public intellectual, rent control, reserve currency, road to serfdom, Ronald Reagan, shared worldview, stakhanovite, strikebreaker, the market place, upwardly mobile, urban planning, urban renewal, very high income, wage slave, washing machines reduced drudgery, wealth creators, women in the workforce, young professional

Each had significantly different characteristics, but in all of them the conveyor-belt assembly line – ‘the track’ – was the relentless, remorseless, unforgiving nerve centre of operations. Dagenham was the British home of Ford – a Detroit in miniature since the early 1930s. The works put a premium on continuous, integrated production and included a blast furnace, coke ovens, a powerhouse, iron and steel foundries, and fully mechanised jetties for loading and unloading that reached out into the Thames. ‘Ford has always applied the principle that higher wages and higher standards of living for all depend on lower costs and lower selling prices through increasingly large-scale production’ was how the British chairman, Lord Perry, summed up the Ford philosophy soon after the war.