Download to read offline
![978-1-491-92435-8
[LSI]
Infrastructure as Code
by Kief Morris
Copyright © 2016 Kief Morris. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
also available for most titles (http://safaribooksonline.com). For more information, contact our corporate/
institutional sales department: 800-998-9938 or corporate@oreilly.com.
Editor: Brian Anderson Indexer: Judy McConville
Production Editor: Kristen Brown Interior Designer: David Futato
Copyeditor: Amanda Kersey Cover Designer: Karen Montgomery
Proofreader: Jasmine Kwityn Illustrator: Rong Tang and Rebecca Demarest
June 2016: First Edition
Revision History for the First Edition
2016-06-07: First Release
See http://oreilly.com/catalog/errata.csp?isbn=9781491924358 for release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Infrastructure as Code, the cover
image, and related trade dress are trademarks of O’Reilly Media, Inc.
While the publisher and the author have used good faith efforts to ensure that the information and
instructions contained in this work are accurate, the publisher and the author disclaim all responsibility
for errors or omissions, including without limitation responsibility for damages resulting from the use of
or reliance on this work. Use of the information and instructions contained in this work is at your own
risk. If any code samples or other technology this work contains or describes is subject to open source
licenses or the intellectual property rights of others, it is your responsibility to ensure that your use
thereof complies with such licenses and/or rights.](https://image.slidesharecdn.com/74980-250616015204-cbdae5a6/75/Infrastructure-as-code-managing-servers-in-the-cloud-Morris-7-2048.jpg)
![Photo by] Relics of the Ice Age. [C. W. Mason
A walrus tusk from Kelsey Hill and the tooth of a mammoth from
the cliffs at Atwick.[2]
In this way was formed the ‘great mass of gravel, clay, and sand
... east of the Yorkshire Wolds’ which we know as the Plain of
Holderness. Here is what one of our foremost local geologists has to
say of its beginnings:—
‘Let us imagine the probable appearance of East Yorkshire on the
final melting of the ice. Huge fans or sheets of gravel occur at
Bridlington and other places as a result of the floods. Rounded
hillocks of gravel and clay stand out in all directions; the hollows in
between are filled with water, forming miniature lakes or meres. Of
animal or plant life there is little or none. The climate gradually
becomes milder; at first Arctic plants and animals exist in small
numbers. Later, the margins of the meres become clothed in
vegetation; peat is eventually formed, and huge trees of Oak and Fir
thrive. The Red Deer, Beaver, Short-horned Ox, Otter, and Wild
Horse, haunt the woods, and finally primitive man makes his
appearance.’](https://image.slidesharecdn.com/74980-250616015204-cbdae5a6/75/Infrastructure-as-code-managing-servers-in-the-cloud-Morris-37-2048.jpg)
![it must have been; for there was evidence that it had been built in
the first place by a race of people whose tools were of flint and
bone, and that this race had been ousted many years later by
another more advanced race who had weapons and tools of bronze.
That the dwellers here were mighty hunters and mighty eaters was
proved by enormous accumulations of animal bones under and
around the platform. That they were also cannibals is likely from the
presence of human bones among this refuse.
So much for the ‘lake-dwellers’ of Ulrome. Up on the Wolds there
were men living a somewhat different life. These hunted and ate the
same kinds of creatures, and they used the same kinds of weapons,
but their dwellings were dug out of the soil—shallow circular or
elliptical pits each covered over with a conical roof of branches and
turf, supported on a central post; or deeper troughs covered over
with sods and scrub laid on slabs of chalk, so that the roof was level
with the surrounding earth and indistinguishable from it.
Of the former kind of pit-dwelling an example has been discovered
in the hollow known as Garton Slack, the pit measuring rather less
than 9 feet by 6 feet in length and breadth, and 5 feet in depth;
while one of the latter kind has come to light under Kemp Howe, a
few miles north of Driffield.[3]
The underground chamber here
measured 25 feet by 4½ feet, had a depth of 6 feet at its deeper
end, and was approached by a sloping passage 11 feet in length, the
entrance to which would doubtless be hidden with scrub. The roof
had been supported on six upright posts, and for twelve feet along
one side of the chamber ran a stone ledge—this last being evidently
a luxury.
It is probable that these two kinds of dwellings may have been
respectively the summer and winter houses of the same people. For
the Roman historian Tacitus says of the ancient tribes on the other
side of the North Sea:—
Besides their ordinary habitations, they have a number of
subterranean caves, dug by their own labour and carefully](https://image.slidesharecdn.com/74980-250616015204-cbdae5a6/75/Infrastructure-as-code-managing-servers-in-the-cloud-Morris-41-2048.jpg)
![The earliest kind of Axe used in East Yorkshire.
It is impossible to give any definite answers to these questions. A
century ago, however, the village people answered them all very
easily. The Devil, they said, objected to the building of the church,
and flung this stone to destroy it before its completion. But his aim
was not so accurate as it was intended to be, and the missile missed
its mark. Asked for a proof of their wonderful story, they would point
to the stone itself. There it was for everyone to see. What further
proof could be needed?[4]
Whether we believe this legend or not,
two things are certain. First, that the stone is as old as the barrows
in the surrounding wolds; secondly, that there is no rock of the same
nature nearer to it than Filey Brig and the Brimham Rocks. Was it
brought down by the great ice sheet and then erected by the men of
the Stone Age to serve some purpose in their heathen rites, or did
they bring it up from Filey or down from the hills of the North Riding
on wooden rollers? Perhaps it is not more difficult to conceive of
their doing this than of their raising such a huge barrow as that
which stands unopened at the foot of Garrowby Hill—a mound 250
feet in diameter at its base and 50 feet in height.](https://image.slidesharecdn.com/74980-250616015204-cbdae5a6/75/Infrastructure-as-code-managing-servers-in-the-cloud-Morris-50-2048.jpg)
![Photo by] [C.W. Mason
British Gold Coin.
Found at Atwick by Mr. W. Morfitt
(1/1).
How a British Chieftain’s Wife was buried in Garton Slack.
In neither of these two barrows was
there any sign of a bronze implement.
Weapons and implements of bronze
are rare among those found in the
barrows of East Yorkshire, and the
few discovered are dagger or knife
heads and prickers. The Brigantes
were far behind the Britons of the
south in their knowledge of the use of
metal; and at the time when the latter
were making use of bronze, the wild
and savage tribes of the north were content still to make use of
greenstone and flint.
Personal ornaments, too, are rare, and were found accompanying
only fifty-seven out of eight hundred and ninety-three burials that
Mr. Mortimer excavated. They include dress-fastenings, such as rings
and links of jet, and buttons of amber, jet and bone. With only one
British interment was gold found, and of silver ornaments none were
discovered at all.](https://image.slidesharecdn.com/74980-250616015204-cbdae5a6/75/Infrastructure-as-code-managing-servers-in-the-cloud-Morris-57-2048.jpg)
Infrastructure as code managing servers in the cloud Morris
- 1. Infrastructure as codemanaging servers in the cloud Morris download https://textbookfull.com/product/infrastructure-as-code-managing- servers-in-the-cloud-morris/ Download full version ebook from https://textbookfull.com
- 2. We believe theseproducts will be a great fit for you. Click the link to download now, or visit textbookfull.com to discover even more! Infrastructure as Code: Dynamic Systems for the Cloud Age 2nd Edition Kief Morris https://textbookfull.com/product/infrastructure-as-code-dynamic- systems-for-the-cloud-age-2nd-edition-kief-morris/ Infrastructure as Code 2nd Edition Early Access Kief Morris https://textbookfull.com/product/infrastructure-as-code-2nd- edition-early-access-kief-morris/ Managing Distributed Cloud Applications and Infrastructure: A Self-Optimising Approach Theo Lynn https://textbookfull.com/product/managing-distributed-cloud- applications-and-infrastructure-a-self-optimising-approach-theo- lynn/ Terraform Up and Running Writing Infrastructure as Code 1st Edition Yevgeniy Brikman https://textbookfull.com/product/terraform-up-and-running- writing-infrastructure-as-code-1st-edition-yevgeniy-brikman/
- 3. Terraform Up RunningWriting Infrastructure as Code 2 / converted Edition Yevgeniy Brikman https://textbookfull.com/product/terraform-up-running-writing- infrastructure-as-code-2-converted-edition-yevgeniy-brikman/ Cybersecurity in the Electricity Sector Managing Critical Infrastructure Rafa■ Leszczyna https://textbookfull.com/product/cybersecurity-in-the- electricity-sector-managing-critical-infrastructure-rafal- leszczyna/ The Cloud DBA-Oracle : Managing Oracle Database in the Cloud 1st Edition Abhinivesh Jain https://textbookfull.com/product/the-cloud-dba-oracle-managing- oracle-database-in-the-cloud-1st-edition-abhinivesh-jain/ DevOps in Python: Infrastructure as Python Zadka https://textbookfull.com/product/devops-in-python-infrastructure- as-python-zadka/ Leading and Managing Professional Services Firms in the Infrastructure Sector First Edition Ellis https://textbookfull.com/product/leading-and-managing- professional-services-firms-in-the-infrastructure-sector-first- edition-ellis/
- 4. Kief Morris Infrastructure as Code MANAGINGSERVERS IN THE CLOUD
- 6. Kief Morris Infrastructure asCode Managing Servers in the Cloud Boston Farnham Sebastopol Tokyo Beijing Boston Farnham Sebastopol Tokyo Beijing
- 7. 978-1-491-92435-8 [LSI] Infrastructure as Code byKief Morris Copyright © 2016 Kief Morris. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://safaribooksonline.com). For more information, contact our corporate/ institutional sales department: 800-998-9938 or corporate@oreilly.com. Editor: Brian Anderson Indexer: Judy McConville Production Editor: Kristen Brown Interior Designer: David Futato Copyeditor: Amanda Kersey Cover Designer: Karen Montgomery Proofreader: Jasmine Kwityn Illustrator: Rong Tang and Rebecca Demarest June 2016: First Edition Revision History for the First Edition 2016-06-07: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781491924358 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Infrastructure as Code, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.
- 8. Table of Contents Preface.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Part I. Foundations 1. Challenges and Principles. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Why Infrastructure as Code? 3 What Is Infrastructure as Code? 5 Goals of Infrastructure as Code 5 Challenges with Dynamic Infrastructure 6 Server Sprawl 6 Configuration Drift 7 Snowflake Servers 7 Fragile Infrastructure 8 Automation Fear 9 Erosion 10 Principles of Infrastructure as Code 10 Systems Can Be Easily Reproduced 10 Systems Are Disposable 11 Systems Are Consistent 12 Processes Are Repeatable 12 Design Is Always Changing 13 Practices 13 Use Definition Files 13 Self-Documented Systems and Processes 14 Version All the Things 15 Continuously Test Systems and Processes 16 Small Changes Rather Than Batches 16 iii
- 9. Keep Services AvailableContinuously 17 Antifragility: Beyond “Robust” 17 The Secret Ingredient of Antifragile IT Systems 18 Conclusion 18 What’s Next? 19 2. Dynamic Infrastructure Platforms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 What Is a Dynamic Infrastructure Platform? 21 Requirements for a Dynamic Infrastructure Platform 22 Programmable 23 On-Demand 24 Self-Service 25 Infrastructure Resources Provided by the Platform 25 Compute Resources 26 Storage Resources 26 Network Resources 28 Types of Dynamic Infrastructure Platforms 30 Public IaaS Cloud 30 Community IaaS Cloud 30 Private IaaS Cloud 30 Antipattern: Hand-Cranked Cloud 31 Hybrid and Mixed Cloud Options 32 Bare-Metal Clouds 32 Deciding on a Dynamic Infrastructure Platform 34 Public or Private? 34 Cloud Portability 37 Mechanical Sympathy with the Cloud and Virtualization 38 Conclusion 40 3. Infrastructure Definition Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Choosing Tools for Infrastructure as Code 42 Requirement: Scriptable Interface 42 Requirement: Unattended Mode for Command-Line Tools 42 Requirement: Support for Unattended Execution 43 Requirement: Externalized Configuration 45 Configuration Definition Files 48 Reusability with Configuration Definitions 49 Working with Infrastructure Definition Tools 50 Provisioning Infrastructure with Procedural Scripts 51 Defining Infrastructure Declaratively 52 Using Infrastructure Definition Tools 54 Configuring Servers 54 iv | Table of Contents
- 10. Configuration Registries 55 LightweightConfiguration Registries 56 Is a Configuration Registry a CMDB? 57 The CMDB Audit and Fix Antipattern 58 The Infrastructure-as-Code Approach to CMDB 58 Conclusion 59 4. Server Configuration Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 Goals for Automated Server Management 62 Tools for Different Server Management Functions 62 Tools for Creating Servers 63 Tools for Configuring Servers 64 Tools for Packaging Server Templates 65 Tools for Running Commands on Servers 66 Using Configuration from a Central Registry 68 Server Change Management Models 69 Ad Hoc Change Management 69 Configuration Synchronization 69 Immutable Infrastructure 70 Containerized Services 70 Containers 70 Managing Ruby Applications with and without Containers 72 Are Containers Virtual Machines? 73 Using Containers Rather than Virtual Machines 74 Running Containers 75 Security and Containers 76 Conclusion 78 5. General Infrastructure Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Considerations for Infrastructure Services and Tools 81 Prefer Tools with Externalized Configuration 83 Prefer Tools That Assume Infrastructure Is Dynamic 84 Prefer Products with Cloud-Compatible Licensing 84 Prefer Products That Support Loose Coupling 85 Sharing a Service Between Teams 85 Service Instance Templates 86 Monitoring: Alerting, Metrics, and Logging 87 Alerting: Tell Me When Something Is Wrong 87 Metrics: Collect and Analyze Data 89 Log Aggregation and Analysis 89 Service Discovery 90 Server-Side Service Discovery Pattern 91 Table of Contents | v
- 11. Client-Side Service DiscoveryPattern 91 Distributed Process Management 91 Orchestrating Processes with Server Roles 92 Orchestrating Processes with Containers 92 Scheduling Short Jobs 92 Container Orchestration Tools 92 Software Deployment 93 Deployment Pipeline Software 93 Packaging Software 94 Conclusion 96 Part II. Patterns 6. Patterns for Provisioning Servers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Server Provisioning 100 A Server’s Life 100 What Goes onto a Server 105 Types of Things on a Server 105 Server Roles 107 Patterns for Creating Servers 108 Antipattern: Handcrafted Server 109 Practice: Wrap Server Creation Options in a Script 110 Antipattern: Hot Cloned Server 111 Pattern: Server Template 111 Antipattern: Snowflake Factory 112 Patterns for Bootstrapping New Servers 112 Pushing to Bootstrap 113 Pulling to Bootstrap 113 Practice: Smoke Test Every New Server Instance 114 Conclusion 115 7. Patterns for Managing Server Templates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 Stock Templates: Can’t Someone Else Do It? 117 Provisioning Servers Using Templates 118 Provisioning at Creation Time 118 Provisioning in the Template 119 Balancing Provisioning Across Template and Creation 120 The Process for Building a Server Template 121 Creating Templates for Multiple Platforms 122 Origin Images 123 Antipattern: Hot Cloned Server Template 123 vi | Table of Contents
- 12. Baking a Templatefrom an OS Installation Image 123 Baking a Template from a Stock Image 124 Building a Template from a Unikernel 125 Customizing a Server Template without Booting It 125 Updating Server Templates 126 Reheating a Template 126 Baking a Fresh Template 126 Versioning Server Templates 126 Building Templates for Roles 129 Pattern: Layered Template 129 Sharing Base Scripts for Templates 130 Automating Server Template Management 131 Customizing Servers Before Baking 131 Practice: Automatically Test Server Templates 132 Conclusion 132 8. Patterns for Updating and Changing Servers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Models for Server Change Management 134 Ad Hoc Change Management 134 Continuous Configuration Synchronization 134 Immutable Servers 135 Containerized Servers 136 General Patterns and Practices 136 Practice: Minimize Server Templates 136 Practice: Replace Servers When the Server Template Changes 137 Pattern: Phoenix Servers 137 Patterns and Practices for Continuous Deployment 138 Pattern: Masterless Configuration Management 139 Practice: Apply Cron 139 Continuous Synchronization Flow 140 The Unconfigured Country 141 Patterns and Practices for Immutable Servers 143 Server Image as Artifact 144 Simplifying Confirmation Management Tooling with Immutable Servers 144 Immutable Server Flow 144 Bootstrap Configuration with Immutable Servers 145 Transactional Server Updates 147 Practices for Managing Configuration Definitions 148 Practice: Keep Configuration Definitions Minimal 148 Organizing Definitions 149 Practice: Use Test-Driven Development (TDD) to Drive Good Design 149 Conclusion 150 Table of Contents | vii
- 13. 9. Patterns forDefining Infrastructure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 Environments 152 Antipattern: Handcrafted Infrastructure 153 Defining Infrastructure Stacks as Code 153 Antipattern: Per-Environment Definition Files 155 Pattern: Reusable Definition Files 155 Practice: Test and Promote Stack Definitions 157 Self-Service Environments 158 Organizing Infrastructure 158 Antipattern: Monolithic Stack 158 Avoid “Lift and Shift” When Migrating Infrastructure 160 Dividing an Application Environment into Multiple Stacks 160 Managing Configuration Parameters Between Stacks 162 Sharing Infrastructure Elements 164 Practice: Manage Application Code and Infrastructure Code Together 166 Approaches to Sharing Definitions 167 Practice: Align Infrastructure Design with the Scope of Change 168 Example: An Infrastructure Design for Microservices 169 Running Definition Tools 174 Conclusion 175 Part III. Practices 10. Software Engineering Practices for Infrastructure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 System Quality 180 Poor-Quality Systems Are Difficult to Change 180 High-Quality Systems Are Easier and Safer to Change 181 Infrastructure Quality Through Code 181 Fast Feedback 181 VCS for Infrastructure Management 182 What to Manage in a VCS 182 Continuous Integration (CI) 183 Continuously Testing Branches Is Not Continuous Integration 183 Who Broke the Build? 185 Ignoring Tests That Fail 186 CI for Infrastructure 187 Continuous Delivery (CD) 187 The Problem with the Integration Phase 187 Deployment Pipelines and Change Pipelines 188 Continuous Delivery Is Not Continuous Deployment 189 Code Quality 190 viii | Table of Contents
- 14. Clean Code 190 Practice:Manage Technical Debt 191 Managing Major Infrastructure Changes 192 Feature Toggles 193 Conclusion 194 11. Testing Infrastructure Changes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 The Agile Approach to Testing 196 Automating Tests for Fast Feedback 197 Organically Building a Test Suite 197 Structuring the Test Suite: The Test Pyramid 198 Avoiding an Unbalanced Test Suite 199 Practice: Test at the Lowest Level Possible 200 Practice: Only Implement the Layers You Need 201 Practice: Prune the Test Suite Often 202 Practice: Continuously Review Testing Effectiveness 202 Implementing a Balanced Test Suite 202 Low-Level Testing 204 Mid-Level Testing 206 Higher-Level Tests 209 Testing Operational Quality 211 Managing Test Code 212 Practice: Keep Test Code with the Code It Tests 212 Anti-Pattern: Reflective Tests 213 Techniques to Isolate Components for Testing 213 Refactoring Components so They Can Be Isolated 215 Managing External Dependencies 215 Test Setup 216 Roles and Workflow for Testing 218 Principle: People Should Write Tests for What They Build 219 The Test-Writing Habit 219 Principle: Everyone Should Have Access to the Testing Tools 219 The Value of a Quality Analyst 220 Test-Driven Development (TDD) 221 Conclusion 222 12. Change Management Pipelines for Infrastructure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 Benefits of a Change Management Pipeline 225 Guidelines for Designing Pipelines 226 Ensure Consistency Across Stages 226 Get Immediate Feedback for Every Change 227 Run Automated Stages Before Manual Stages 227 Table of Contents | ix
- 15. Get Production-Like SoonerRather Than Later 228 Basic Pipeline Designs 229 The Local Development Stage 229 The Build Stage 230 Publishing a Configuration Artifact 231 Automated Testing Stages 233 Manual Validation Stages 234 Apply to Live 235 The Rhythm of the Pipeline 235 Practices for Using a Pipeline 236 Practice: Prove Production Readiness for Every Change 237 Practice: Start Every Change from the Beginning of the Pipeline 237 Practice: Stop the Line on Any Failure 238 Scaling Pipelines to More Complex Systems 238 Pattern: Fan-In Pipelines 239 Practice: Keep Pipelines Short 243 Practice: Decouple Pipelines 243 Integration Models 243 Techniques for Handling Dependencies Between Components 246 Pattern: Library Dependency 246 Pattern: Self-Provisioned Service Instance 248 Providing Pre-Release Library Builds 248 Providing Test Instances of a Service to Consumers 249 Using Test Instances of a Service as a Consumer 250 Practices for Managing Interfaces Between Components 252 Practice: Ensure Backward Compatibility of Interfaces 252 Practice: Decouple Deploying from Releasing 253 Practice: Use Version Tolerance 253 Practice: Provide Test Doubles 254 Practice: Test the Provider with Contract Tests 254 Practice: Test with a Reference Consumer 255 Practice: Smoke Test the Provider Interface 255 Practice: Run Consumer-Driven Contract (CDC) Tests 255 Conclusion 256 13. Workflow for the Infrastructure Team. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 Automate Anything That Moves 258 Make the Change Manually 259 Ad Hoc Automation 259 Autonomic Automation 260 Autonomic Automation Workflow 261 Using a Local Sandbox 262 x | Table of Contents
- 16. Using Local Virtualizationfor a Sandbox 263 Example Workflow with Local Testing 265 Using the Virtualization Platform for Sandboxes 266 Codebase Organization Patterns 267 Antipattern: Branch-Based Codebases 268 Pattern: One Trunk per Component 268 Pattern: Single Trunk 269 Workflow Effectiveness 269 Expediting Changes 269 Code Reviews 270 Fitting Governance into the Workflow 271 Conclusion 273 14. Continuity with Dynamic Infrastructure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 Service Continuity 276 True Availability 276 Using Dynamic Server Pools for Recovery 277 Software Design for Dynamic Infrastructure 279 Compartmentalizing for Continuity 281 Zero-Downtime Changes 282 Pattern: Blue-Green Replacement 282 Pattern: Phoenix Replacement 283 Practice: Reduce the Scope of Replacement 284 Pattern: Canary Replacement 285 Routing Traffic for Zero-Downtime Replacements 286 Zero-Downtime Changes with Data 288 Data Continuity 289 Replicating Data Redundantly 289 Regenerating Data 290 Delegating Data Persistence 290 Backing Up to Persistent Storage 291 Disaster Recovery 292 Continuous Disaster Recovery 293 The DR Plan: Planning for Disaster 294 Practice: Prefer Rebuilding Things over Cold Standby 295 Continuous Monitoring through the Pipeline 296 Security 298 Automatically Papering over Compromises 298 Reliable Updates as a Defense 299 Provenance of Packages 299 Automated Hardening 301 Automating Security Validation in the Pipeline 302 Table of Contents | xi
- 17. The Change Pipelineas a Vulnerability 302 Managing Security Risks with Cloud Accounts 303 Conclusion 305 15. Organizing for Infrastructure as Code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 Evolutionary Architecture 308 Learning under Fire 309 Start with a Trailblazer Pipeline 310 Measuring Effectiveness 311 Agree on Desired Outcomes First 311 Choose Metrics that Help the Team 312 Track and Improve Cycle Time 313 Use Kanban to Make Work Visible 315 Retrospectives and Post-Mortems 316 Organize to Empower Users 317 Pitfalls of the Divided Function Model 317 Adopt a Self-Service Model 319 Take Full Responsibility: You Build It, You Run It 320 Organizing into Cross-Functional Teams 321 Governance through Continuous Change Management 322 Provide Solid Building Blocks 322 Prove Operational Readiness in the Pipeline 323 Sharing Ownership of Operational Quality 324 Review and Audit Automated Processes 324 Optimize for Time to Detect and Fix Problems 324 Conclusion: It’s Never Finished 325 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327 xii | Table of Contents
- 18. 1 Andrew ClayShafer and Patrick Debois triggered the DevOps movement with a talk at the Agile 2008 confer‐ ence. The movement grew, mainly driven by the series of DevOpsDays conferences organized by Debois. 2 A BBS is a bulletin board system. Preface Infrastructure and software development teams are increasingly building and manag‐ ing infrastructure using automated tools that have been described as “infrastructure as code.” These tools expect users to define their servers, networking, and other ele‐ ments of an infrastructure in files modeled after software source code. The tools then compile and interpret these files to decide what action to take. This class of tool has grown naturally with the DevOps movement.1 The DevOps movement is mainly about culture and collaboration between software developers and software operations people. Tooling that manages infrastructure based on a soft‐ ware development paradigm has helped bring these communities together. Managing infrastructure as code is very different from classic infrastructure manage‐ ment. I’ve met many teams who have struggled to work out how to make this shift. But ideas, patterns, and practices for using these tools effectively have been scattered across conference talks, blog posts, and articles. I’ve been waiting for someone to write a book to pull these ideas together into a single place. I haven’t seen any sign of this, so finally took matters into my own hands. You’re now reading the results of this effort! How I Learned to Stop Worrying and to Love the Cloud I set up my first server, a dialup BBS,2 in 1992. This led to Unix system administra‐ tion and then to building and running hosted software systems (before we called it SaaS, aka “Software as a Service”) for various companies, from startups to enterprises. I’ve been on a journey to infrastructure as code the entire time, before I’d ever heard the term. Preface | xiii
- 19. Things came toa head with virtualization. The story of my stumbling adoption of virtualization and the cloud may be familiar, and it illustrates the role that infrastruc‐ ture as code has to play in modern IT operations. My First Virtual Server Farm I was thrilled when my team got the budget to buy a pair of beefy HP rack servers and licenses for VMware ESX Server back in 2007. We had in our office’s server racks around 20 1U and 2U servers named after fruits (Linux servers) and berries (Windows database servers) running test environments for our development teams. Stretching these servers to test various releases, branches, and high-priority, proof-of-concept applications was a way of life. Network services like DNS, file servers, and email were crammed onto servers running multiple appli‐ cation instances, web servers, and database servers. So we were sure these new virtual servers would change our lives. We could cleanly split each of these services onto its own virtual machine (VM), and the ESX hypervi‐ sor software would help us to squeeze the most out of the multicore server machines and gobs of RAM we’d allocated. We could easily duplicate servers to create new environments and archive those servers that weren’t needed onto disk, confident they could be restored in the future if needed. Those servers did change our lives. But although many of our old problems went away, we discovered new ones, and we had to learn completely different ways of thinking about our infrastructure. Virtualization made creating and managing servers much easier. The flip side of this was that we ended up creating far more servers than we could have imagined. The product and marketing people were delighted that we could give them a new envi‐ ronment to demo things in well under a day, rather than need them to find money in the budget and then wait a few weeks for us to order and set up hardware servers. The Sorcerer’s Apprentice A year later, we were running well over 100 VMs and counting. We were well under‐ way with virtualizing our production servers and experimenting with Amazon’s new cloud hosting service. The benefits virtualization had brought to the business people meant we had money for more ESX servers and for shiny SAN devices to feed the surprising appetite our infrastructure had for storage. But we found ourselves a bit like Mickey Mouse in “The Sorcerer’s Apprentice” from Fantasia. We spawned virtual servers, then more, then even more. They over‐ whelmed us. When something broke, we tracked down the VM and fixed whatever was wrong with it, but we couldn’t keep track of what changes we’d made where. xiv | Preface
- 20. Well, a perfecthit! See how he is split! Now there’s hope for me, and I can breathe free! Woe is me! Both pieces come to life anew, now, to do my bidding I have servants two! Help me, O great powers! Please, I’m begging you! —Excerpted from Brigitte Dubiel’s translation of “Der Zauberlehrling” (“The Sor‐ cerer’s Apprentice”) by Johann Wolfgang von Goethe As new updates to operating systems, web servers, app servers, database servers, JVMs, and various other software packages came out, we would struggle to install them across all of our systems. We would apply them successfully to some servers, but on others the upgrades broke things, and we didn’t have time to stomp out every incompatibility. Over time, we ended up with many combinations of versions of things strewn across hundreds of servers. We had been using configuration automation software even before we virtualized, which should have helped with these issues. I had used CFEngine in previous compa‐ nies, and when I started this team, I tried a new tool called Puppet. Later, when spik‐ ing out ideas for an AWS infrastructure, my colleague Andrew introduced Chef. All of these tools were useful, but particularly in the early days, they didn’t get us out of the quagmire of wildly different servers. The problem was that, although Puppet (and Chef and the others) should have been set up and left running unattended across all of our servers, we couldn’t trust it. Our servers were just too different. We would write manifests to configure and manage a particular application server. But when we ran it against another, theoretically similar app server, we found that different versions of Java, application software, and OS components would cause the Puppet run to fail, or worse, break the application server. So we ended up using Puppet ad hoc. We could safely run it against new VMs, although we might need to make some tweaks after it ran. We would write manifests for a specific task and then run them against servers one at a time, carefully checking the result and making fixes as needed. So configuration automation was a useful aid, somewhat better than shell scripts, but the way we used it didn’t save us from our sprawl of inconsistent servers. Preface | xv
- 21. Cloud from Scratch Thingschanged when we began moving things onto the cloud. The technology itself wasn’t what improved things; we could have done the same thing with our own VMware servers. But because we were starting fresh, we adopted new ways of manag‐ ing servers based on what we had learned with our virtualized farm and on what we were reading and hearing from IT Ops teams at companies like Flickr, Etsy, and Netflix. We baked these new ideas into the way we managed services as we migrated them onto the cloud. The key idea of our new approach was that every server could be automatically rebuilt from scratch, and our configuration tooling would run continuously, not ad hoc. Every server added into our new infrastructure would fall under this approach. If automation broke on some edge case, we would either change the automation to handle it, or else fix the design of the service so it was no longer an edge case. The new regime wasn’t painless. We had to learn new habits, and we had to find ways of coping with the challenges of a highly automated infrastructure. As the members of the team moved on to other organizations and got involved with communities such as DevOpsDays, we learned and grew. Over time, we reached the point where we were habitually working with automated infrastructures with hundreds of servers, with much less effort and headache than we had been in our “Sorcerer’s Apprentice” days. Joining ThoughtWorks was an eye-opener for me. The development teams I worked with were passionate about using XP engineering practices like test-driven develop‐ ment (TDD), continuous integration (CI) and continuous delivery (CD). Because I had already learned to manage infrastructure scripts and configuration files in source control systems, it was natural to apply these rigorous development and testing approaches to them. Working with ThoughtWorks has also brought me into contact with many IT opera‐ tions teams, most of whom are using virtualization, cloud, and automation tools to handle a variety of challenges. Working with them to share and learn new ideas and techniques has been a fantastic experience. Why I’m Writing This Book I’ve run across many teams who are in the same place I was a few years ago: people who are using cloud, virtualization, and automation tools but haven’t got it all run‐ ning as smoothly as they know they could. Much of the challenge is time. Day-to-day life for system administrators is coping with a never-ending flow of critical work. Fighting fires, fixing problems, and setting xvi | Preface
- 22. up new business-criticalprojects doesn’t leave much time to work on the fundamen‐ tal improvements that will make the routine work easier. My hope is that this book provides a practical vision for how to manage IT infra‐ structure, with techniques and patterns that teams can try and use. I will avoid the details of configuring and using specific tools so that the content will be useful for working with different tools, including ones that may not exist yet. Meanwhile, I will use examples from existing tools to illustrate points I make. The infrastructure-as-code approach is essential for managing cloud infrastructure of any real scale or complexity, but it’s not exclusive to organizations using public cloud providers. The techniques and practices in this book have proven effective in virtual‐ ized environments and even for bare-metal servers that aren’t virtualized. Infrastructure as Code is one of the cornerstones of DevOps. It is the “A” in “CAMS”: culture, automation, measurement, and sharing. Who This Book Is For This book is for people who work with IT infrastructure, particularly at the level of managing servers and collections of servers. You may be a system administrator, infrastructure engineer, team lead, architect, or a manager with technical interest. You might also be a software developer who wants to build and use infrastructure. I’m assuming you have some exposure to virtualization or IaaS (Infrastructure as a Service) cloud, so you know how to create a server, and the concepts of configuring operating systems. You’ve probably at least played with configuration automation software like Ansible, Chef, or Puppet. While this book may introduce some readers to infrastructure as code, I hope it will also be interesting to people who work this way already and a vehicle through which to share ideas and start conversations about how to do it even better. What Tools Are Covered This book doesn’t offer instructions in using specific scripting languages or tools. There are code examples from specific tools, but these are intended to illustrate con‐ cepts and approaches, rather than to provide instruction. This book should be helpful to you regardless of whether you use Chef on OpenStack, Puppet on AWS, Ansible on bare metal, or a completely different stack. The specific tools that I do mention are ones which I’m aware of, and which seem to have a certain amount of traction in the field. But this is a constantly changing land‐ scape, and there are plenty of other relevant tools. Preface | xvii
- 23. The tools Iuse in examples tend to be ones with which I am familiar enough to write examples that demonstrate the point I’m trying to make. For example, I use Terra‐ form for examples of infrastructure definitions because it has a nice, clean syntax, and I’ve used it on multiple projects. Many of my examples use Amazon’s AWS cloud platform because it is likely to be the most familiar to readers. How to Read This Book Read Chapter 1, or at least skim it, to understand the terms this book uses and the principles this book advocates. You can then use this to decide which parts of the book to focus on. If you’re new to this kind of automation, cloud, and infrastructure orchestration tool‐ ing, then you’ll want to focus on Part I, and then move on to Part II. Get comfortable with those topics before proceeding to Part III. If you’ve been using the types of automation tools described here, but don’t feel like you’re using them the way they’re intended after reading Chapter 1, then you may want to skip or skim the rest of Part I. Focus on Part II, which describes ways of using dynamic and automated infrastructure that align with the principles outlined in Chapter 1. If you’re comfortable with the dynamic infrastructure and automation approaches described in Chapter 1, then you may want to skim Parts I and II and focus on Part III, which gets more deeply into the infrastructure management regime: archi‐ tectural approaches as well as team workflow. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. xviii | Preface
- 24. This element signifiesa tip or suggestion. This element signifies a general note. This element indicates a warning or caution. Safari® Books Online Safari Books Online is an on-demand digital library that deliv‐ ers expert content in both book and video form from the world’s leading authors in technology and business. Technology professionals, software developers, web designers, and business and crea‐ tive professionals use Safari Books Online as their primary resource for research, problem solving, learning, and certification training. Safari Books Online offers a range of plans and pricing for enterprise, government, education, and individuals. Members have access to thousands of books, training videos, and prepublication manuscripts in one fully searchable database from publishers like O’Reilly Media, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technology, and hundreds more. For more information about Safari Books Online, please visit us online. Preface | xix
- 25. How to ContactUs Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://bit.ly/infrastructureAsCode_1e. To comment or ask technical questions about this book, send email to bookques‐ tions@oreilly.com. For more information about our books, courses, conferences, and news, see our web‐ site at http://www.oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Acknowledgments When I started working on this book, I assumed the result would be a product that was entirely my own. But in the end, it’s just the opposite: this book is the product of the ideas, thoughts, opinions, and experiences of far more people than I could have imagined. I have probably mangled, oversimplified, and misrepresented this input. But this book would not exist as it is without these contributions. The people I worked with on the University of Tennessee’s computer science lab staff taught me my Unix chops and inducted me into the culture. Chad Mynhier was par‐ ticularly responsible for hooking me into the Unix world. He explained why I could no longer cd into my own home area after I had experimented with the chmod com‐ mand. Working with a series of companies, including Syzygy, Vizyon, Cellectivity, and the Map of Medicine, gave me the arena to develop my understanding and to learn how to apply infrastructure automation to real-world business and user problems. I owe much to the many good people of those organizations. I’ll specifically call out Jona‐ than Waywell and Ketan Patel for their unending support and encouragement, xx | Preface
- 26. 3 Sadly, asof early 2016, Infrastructures.org hasn’t been updated since 2007. Andrew Fulcher for quickly learning what I had to teach and then teaching me even more, and Nat Billington for his inspiration. This book would truly never have happened without my current home, Thought‐ Works. I’ve learned much about the ideas in this book and how to think about and explain them to other people, thanks to more than five years of exposure to a dizzy‐ ing number of organizations of various sizes, sectors, and technologies. The endless curiosity of my colleagues, past and present, and their heartfelt drive to improve our industry and the experiences of people in it, continually challenges me. Generous support and encouragement from ThoughtWorks as an organization has been vital for this book, especially as my energy flagged with the finish line coming into view. Chris Murphy, Dave Elliman, Maneesh Subherwal, and Suzi Edwards- Alexander are a few among many who have made this more than a personal project for me. An incomplete list of past and present ThoughtWorks colleagues who have gone out of their way to contribute suggestions, feedback, and other support include: Abigail Bangser, Ashok Subramanian, Barry O’Reilly, Ben Butler-Cole, Chris Bird (DevOops!), Chris Ford, David Farley, Gurpreet Luthra, Inny So, Jason Yip, Jim Gumbley, Kesha Stickland, Marco Abis, Nassos Antoniou, Paul Hammant, Peter Gillard-Moss, Peter Staples, Philip Potter, Rafael Gomes, Sam Newman, Simon Brun‐ ning, Tom Duckering, Venu Murthy, and Vijay Raghavan Aravamudhan. Martin Fowler has given me tremendous encouragement and practical support in writing this book. He gave me far more time than I could have asked, thoroughly reviewing the manuscript several times. Martin gave me detailed, useful feedback and advice based on his considerable experience of organizing and conveying technical concepts. He has been a true champion of this book. My colleague Rong Tang created the images for this book. She was extremely patient with my muddled explanations of what I wanted. Any failings of clarity or consis‐ tency is down to me, but the great look is a credit to her. The folks behind the long-dormant Infrastructures.org exposed me to the ideas of Infrastructure as Code before the term existed.3 I owe a great debt to the people of the DevOpsDays community, who are collectively responsible for bringing the ideas of DevOps and Infrastructure as Code to promi‐ nence. Regardless of who actually coined the term “Infrastructure as Code,” people like Adam Jacob, Andrew Clay-Shafer, John Allspaw, John Willis, Luke Kaines, Mark Burgess, and of course, Patrick Debois (“The Godfather of DevOps”) have given me inspiration and many great ideas. Preface | xxi
- 27. A number ofother people have given me feedback and advice on earlier drafts of this book, including Axel Fontaine, Jon Cowie, Jose Maria San Jose Juarez, Marcos Her‐ mida, and Matt Jones. I also want to thank Kent Spillner, although I don’t recall why. Ovine and dairy. Putney Bridge. Last, but the furthest from least, everlasting love to Ozlem and Erel, who endured my obsession with this book. xxii | Preface
- 28.
- 30. CHAPTER 1 Challenges andPrinciples The new generation of infrastructure management technologies promises to trans‐ form the way we manage IT infrastructure. But many organizations today aren’t see‐ ing any dramatic differences, and some are finding that these tools only make life messier. As we’ll see, infrastructure as code is an approach that provides principles, practices, and patterns for using these technologies effectively. Why Infrastructure as Code? Virtualization, cloud, containers, server automation, and software-defined network‐ ing should simplify IT operations work. It should take less time and effort to provi‐ sion, configure, update, and maintain services. Problems should be quickly detected and resolved, and systems should all be consistently configured and up to date. IT staff should spend less time on routine drudgery, having time to rapidly make changes and improvements to help their organizations meet the ever-changing needs of the modern world. But even with the latest and best new tools and platforms, IT operations teams still find that they can’t keep up with their daily workload. They don’t have the time to fix longstanding problems with their systems, much less revamp them to make the best use of new tools. In fact, cloud and automation often makes things worse. The ease of provisioning new infrastructure leads to an ever-growing portfolio of systems, and it takes an ever-increasing amount of time just to keep everything from collapsing. Adopting cloud and automation tools immediately lowers barriers for making changes to infrastructure. But managing changes in a way that improves consistency and reliability doesn’t come out of the box with the software. It takes people to think through how they will use the tools and put in place the systems, processes, and hab‐ its to use them effectively. 3
- 31. 1 “Shadow IT”is when people bypass formal IT governance to bring in their own devices, buy and install unap‐ proved software, or adopt cloud-hosted services. This is typically a sign that internal IT is not able to keep up with the needs of the organization it serves. 2 The phrase “infrastructure as code” doesn’t have a clear origin or author. While writing this book, I followed a chain of people who have influenced thinking around the concept, each of whom said it wasn’t them, but offered suggestions. This chain had a number of loops. The earliest reference I could find was from the Veloc‐ ity conference in 2009, in a talk by Andrew Clay-Shafer and Adam Jacob. John Willis may be the first to document the phrase, in an article about the conference. Luke Kaines has admitted that he may have been involved, the closest anyone has come to accepting credit. Some IT organizations respond to this challenge by applying the same types of pro‐ cesses, structures, and governance that they used to manage infrastructure and soft‐ ware before cloud and automation became commonplace. But the principles that applied in a time when it took days or weeks to provision a new server struggle to cope now that it takes minutes or seconds. Legacy change management processes are commonly ignored, bypassed, or overruled by people who need to get things done.1 Organizations that are more successful in enforcing these processes are increasingly seeing themselves outrun by more techni‐ cally nimble competitors. Legacy change management approaches struggle to cope with the pace of change offered by cloud and automation. But there is still a need to cope with the ever- growing, continuously changing landscape of systems created by cloud and automa‐ tion tools. This is where infrastructure as code2 comes in. The Iron Age and the Cloud Age In the “iron age” of IT, systems were directly bound to physical hardware. Provision‐ ing and maintaining infrastructure was manual work, forcing humans to spend their time pointing, clicking, and typing to keep the gears turning. Because changes involved so much work, change management processes emphasized careful up-front consideration, design, and review work. This made sense because getting it wrong was expensive. In the “cloud age” of IT, systems have been decoupled from the physical hardware. Routine provisioning and maintenance can be delegated to software systems, freeing the humans from drudgery. Changes can be made in minutes, if not seconds. Change management can exploit this speed, providing better reliability along with faster time to market. 4 | Chapter 1: Challenges and Principles
- 32. What Is Infrastructureas Code? Infrastructure as code is an approach to infrastructure automation based on practices from software development. It emphasizes consistent, repeatable routines for provi‐ sioning and changing systems and their configuration. Changes are made to defini‐ tions and then rolled out to systems through unattended processes that include thorough validation. The premise is that modern tooling can treat infrastructure as if it were software and data. This allows people to apply software development tools such as version control systems (VCS), automated testing libraries, and deployment orchestration to manage infrastructure. It also opens the door to exploit development practices such as test- driven development (TDD), continuous integration (CI), and continuous delivery (CD). Infrastructure as code has been proven in the most demanding environments. For companies like Amazon, Netflix, Google, Facebook, and Etsy, IT systems are not just business critical; they are the business. There is no tolerance for downtime. Amazon’s systems handle hundreds of millions of dollars in transactions every day. So it’s no surprise that organizations like these are pioneering new practices for large scale, highly reliable IT infrastructure. This book aims to explain how to take advantage of the cloud-era, infrastructure-as- code approaches to IT infrastructure management. This chapter explores the pitfalls that organizations often fall into when adopting the new generation of infrastructure technology. It describes the core principles and key practices of infrastructure as code that are used to avoid these pitfalls. Goals of Infrastructure as Code The types of outcomes that many teams and organizations look to achieve through infrastructure as code include: • IT infrastructure supports and enables change, rather than being an obstacle or a constraint. • Changes to the system are routine, without drama or stress for users or IT staff. • IT staff spends their time on valuable things that engage their abilities, not on routine, repetitive tasks. • Users are able to define, provision, and manage the resources they need, without needing IT staff to do it for them. • Teams are able to easily and quickly recover from failures, rather than assuming failure can be completely prevented. What Is Infrastructure as Code? | 5
- 33. Exploring the Varietyof Random Documents with Different Content
- 34. II. HOW THE EASTRIDING WAS MADE. Stand on the very highest point of the white limestone cliffs that stretch northwards from Flamborough Head, and realise that you are standing on what was once the bed of the sea. Strange though this be, it is nevertheless true. Countless ages ago what now towers up 450 feet above sea-level had over it the ceaseless rolling of the waters of the ocean, and during countless ages it was slowly formed out of the shells and teeth and bones of the creatures that lived in these waters. Men who know tell us that the layer of chalk at the bottom of the ocean to-day is composed principally of the remains of creatures so minute as to be visible only by the aid of a microscope, and that this layer grows in thickness at the rate of not more than one-tenth of an inch per year. They tell us also that the layer of chalk which extends under our county is not less than 1200 feet in thickness, and thus a simple calculation will help us to form some idea of the extent of time necessary for its formation. But however long this time actually was, it came to an end with a tremendous upheaval of a portion of the ocean bed, and the formation of a new area of ‘dry land.’ All the coast line of the East Riding, however, does not consist of chalk cliffs. North of Bempton and Speeton lie cliffs of sandstone and clay, which have yielded the fossil remains of living beings that once inhabited the water and the shore. Such are the belemnites and ammonites—the ‘thunderbolts’ and ‘St. Hilda’s snakes’ we may have heard them called—and the Ichthyosaurus, whose skeleton was recently discovered embedded in the clay cliffs at Speeton and may
- 35. now be seenin the Hull Museum. Not a very handsome gentleman in the flesh he must have been, unless appearances are deceptive. One of the First Inhabitants of the East Riding. Actual length about twelve feet. Again, walk southwards from Flamborough Head, and the chalk cliffs are found to get less and less in height until they disappear altogether, and their place is taken by cliffs of clay. Then these disappear, and are succeeded by the long, flat bank of sand and shingle which is known as Spurn Point; and if we round this point and follow the river bank, we find it nothing but mud and clay until we get past the mouth of the river Hull. At Hessle the chalk cliffs break out once more, and we know, from investigations, that the bed of chalk comes to the surface completely westwards of a line drawn from Flamborough to this point. Draw on a map of the East Riding a line from Sewerby, through Driffield and Beverley, to Hessle, and you are drawing the line of the old sea-beach when the upheaval previously mentioned had taken place. This was the shore of a land inhabited by races of animals now found living only in tropical regions. The elephant, rhinoceros, hippopotamus and hyena ranged the land for food, and bones of these creatures have been found in considerable numbers in the caves that exist at Kirkdale in the North Riding. Then came a great change. The climate of Northern Europe became colder and colder till there prevailed what scientists call the
- 36. ‘Great Ice Age.’This was the time of formation of huge glaciers which spread from the mountains of Scandinavia, Scotland, and north-west England southwards and eastwards into the sea, until they met and made its whole area a slowly moving mass of ice. With the ice were carried sand, gravel, clay, boulders torn from projecting rocks, and bones of Arctic animals, such as the walrus, the reindeer, and the Irish elk; and as the ice gradually melted, all these were deposited at the base of the line of chalk cliffs, or even on the summit of the cliffs where these were low. From the gravel pits at Burstwick excavations of ballast for the embankments of the North Eastern Railway brought to light animal bones in such quantities that many tons were sold to chemical manure manufacturers, and it is probable that many tons still remain undiscovered.
- 37. Photo by] Relicsof the Ice Age. [C. W. Mason A walrus tusk from Kelsey Hill and the tooth of a mammoth from the cliffs at Atwick.[2] In this way was formed the ‘great mass of gravel, clay, and sand ... east of the Yorkshire Wolds’ which we know as the Plain of Holderness. Here is what one of our foremost local geologists has to say of its beginnings:— ‘Let us imagine the probable appearance of East Yorkshire on the final melting of the ice. Huge fans or sheets of gravel occur at Bridlington and other places as a result of the floods. Rounded hillocks of gravel and clay stand out in all directions; the hollows in between are filled with water, forming miniature lakes or meres. Of animal or plant life there is little or none. The climate gradually becomes milder; at first Arctic plants and animals exist in small numbers. Later, the margins of the meres become clothed in vegetation; peat is eventually formed, and huge trees of Oak and Fir thrive. The Red Deer, Beaver, Short-horned Ox, Otter, and Wild Horse, haunt the woods, and finally primitive man makes his appearance.’
- 38. Skull and Antlersof a Red Deer found in the Hornsea Peat-bed.
- 39. III. MEN OF THESTONE AGE. What sort of man was it who first inhabited Holderness and how did he live? Artists in his day were few and far between, and the few who did exist in Europe gave pleasure to themselves and to their companions by drawing portraits of reindeer and horses on pieces of bone. To draw portraits of their fellows was probably the last thing they would think of doing. Reindeer and horses are graceful creatures, but the artists’ fellows were anything but graceful. As far as we know, the first inhabitants of Holderness were a race of short, dark-haired men, who depended for their food and clothing on the animals of the forest and the mere, who pursued their prey and fought one another with weapons of stone, and who lived in dwellings built on piles driven into the bed of a lake in exactly the same way as the New Guinea islanders live to-day. Something definite about their dwelling-places we know; for what is appropriately called a lake-dwelling was discovered thirty years ago at Ulrome. This was a structure made of tree trunks laid side by side and held together by piles driven into the bed of what was then a large mere.
- 40. Bone Implements andWeapons from Barrows on the Wolds. A, B. Hammer head and pick made from the shed antlers of a red deer (1/1, 1/4). C. Bodkin or needle (1/1). D. Dagger made from a man’s thigh-bone (1/3). On this rough sort of platform, which measured 90 feet by 60 feet, dwelling-places had been constructed, and a ‘popular watering-place’
- 41. it must havebeen; for there was evidence that it had been built in the first place by a race of people whose tools were of flint and bone, and that this race had been ousted many years later by another more advanced race who had weapons and tools of bronze. That the dwellers here were mighty hunters and mighty eaters was proved by enormous accumulations of animal bones under and around the platform. That they were also cannibals is likely from the presence of human bones among this refuse. So much for the ‘lake-dwellers’ of Ulrome. Up on the Wolds there were men living a somewhat different life. These hunted and ate the same kinds of creatures, and they used the same kinds of weapons, but their dwellings were dug out of the soil—shallow circular or elliptical pits each covered over with a conical roof of branches and turf, supported on a central post; or deeper troughs covered over with sods and scrub laid on slabs of chalk, so that the roof was level with the surrounding earth and indistinguishable from it. Of the former kind of pit-dwelling an example has been discovered in the hollow known as Garton Slack, the pit measuring rather less than 9 feet by 6 feet in length and breadth, and 5 feet in depth; while one of the latter kind has come to light under Kemp Howe, a few miles north of Driffield.[3] The underground chamber here measured 25 feet by 4½ feet, had a depth of 6 feet at its deeper end, and was approached by a sloping passage 11 feet in length, the entrance to which would doubtless be hidden with scrub. The roof had been supported on six upright posts, and for twelve feet along one side of the chamber ran a stone ledge—this last being evidently a luxury. It is probable that these two kinds of dwellings may have been respectively the summer and winter houses of the same people. For the Roman historian Tacitus says of the ancient tribes on the other side of the North Sea:— Besides their ordinary habitations, they have a number of subterranean caves, dug by their own labour and carefully
- 42. covered over withsoil, in winter their retreat from cold and the repository for their corn. In these recesses they not only find a shelter from the rigour of the seasons, but in times of foreign invasions their effects are safely concealed. Of the men who lived on the Yorkshire Wolds we know a great deal; for it was their custom to raise over the burial places of their chiefs circular mounds of earth, some still very large, others now only a foot or two high. The relative size of a burial mound, which we speak of either by the Latin name tumulus or by the English names barrow and howe, marks the importance of the chieftain whose body or ashes once lay under it. These tumuli, or barrows, are very plentifully strewn over the Yorkshire Wolds, and for more than fifty years the late Mr. J. R. Mortimer, of Driffield, devoted all his leisure time to their excavation. The results of his labours are to be seen in his private museum—the Mortimer Museum—and details of his ‘finds’ are recorded in his large book on the Burial Mounds of East Yorkshire, some of the illustrations in which are here reproduced. A general idea of how a barrow has been constructed, and of what it may contain, can be gained from the illustration on the next page. Howe Hill, Duggleby, is one of the larger barrows, built on a sloping hillside, and having at its base a diameter of 125 feet and at its flattened top one of 47 feet.
- 43. Section of HoweHill, Duggleby. A-K. Skeletons in position as buried. O. Cremated remains. Y. Band of blue clay impervious to water. W. Inner mound of clay. Z. Outer mound of chalk. X. Bed of chalk grit. * Probable summit of the barrow when built. From the diagram we see that the bodies first interred have been placed at the bottom of a cavity dug out of the solid chalk. This hole not proving large enough for the numbers to be buried, an extension has been begun, but not finished. Time was evidently pressing, for some bodies have been buried above the surface of the ground. They have been placed in different positions, but the legs of all have been bent at the knees and all are enclosed in a low mound of clay. Above this lie the remains of numerous other bodies, which have been burnt before burial; and over them comes a twelve-inch layer of a blue clay which is impervious to water. Then a large mound of
- 44. soil and piecesof chalk has been raised over all, the mound being originally much higher than it is to-day. Such has been the building of Howe Hill. But it must not be thought that all barrows contain the remains of a large number of bodies. Most contain one only, and the body has either been buried as it was when life left it or been burnt and the calcined bones gathered up in an earthenware vessel, or pinned in a skin garment. The eight full-grown skeletons discovered under Howe Hill are those of men, and we may suppose that they represent a chieftain and his relatives killed in the onslaught by a hostile clan. The cremated bodies, forty of which were discovered in the digging of a trench through the barrow, would be those of his dependants, who died fighting in defence of their lord and master. But the barrow contains evidence of the lives of the people of the time as well as of their deaths. Scattered through the soil under the band of blue clay were found many broken bones of the ox, roebuck, red deer, fox, goat, and pig, the remains of the burial feast; and among these were human bones which had quite evidently been broken and cooked. It is horrible to think of the people of our East Riding as having once been cannibals, but the evidence to that effect is indisputable. Here and there were also found portions of the weapons with which the defenders of the settlement had fought—the hammer head shown on page 9, made from the shed antler of a red deer, and the broken javelin head of flint shown on page 15. In this barrow was also found the wonderfully made flint knife represented below—an implement fashioned out of a piece of flint with no other tools than such as are mentioned below, and yet fashioned so delicately that its greatest thickness is only one-sixteenth of an inch.
- 45. Polished Flint Knifefound in Duggleby Howe (1/1). A clever workman he must have been who made this wonderful knife. But such beautifully wrought implements are very rare. Only one similar knife—found in a barrow at Aldro—was known to its discoverer, and he had himself superintended the excavation of no fewer than two hundred and eighty-eight barrows. The weapons and tools which have been buried with their owners are more commonly of the rougher types figured on the opposite page. They include knives, chisels, spear heads, saws, and arrow heads, all made from flints by the processes of chipping and flaking, with hammer heads, picks, needles and daggers of bone. Compare the figures A and B given on page 9 with the illustration of the antlers of a red deer on page 7, and see how cleverly the hammer head and the pick have been fashioned. Equally clever has been the adaptation of a bone in the making of the very primitive dagger figured at D on the same page. But in this case it has been not the antler of a red deer that has been brought into use, but the thigh-bone of a man.
- 46. Flint Implement andWeapons. A. Chisel from Aldro (1/1). B. Barbed arrow head from Grimston (1/1) C. Javelin head from Duggleby Howe (1/1). So far we have spoken of weapons and implements of bone and of flint. Others were then in use made of whinstone and greenstone, such as the axe heads figured overleaf. Notice the different arrangement of the cutting edge in these two implements, and notice also that in the first one the hole intended for the insertion of a wooden handle has, for some reason or other, not been finished. Perhaps the maker was killed before he had time to finish it, or perhaps he grew tired of his work and threw it away. At any rate this unfinished adze head was found loose on the surface of the ground, and not buried under a howe as was the other.
- 47. Unfinished Stone AdzeHead picked up on Acklam Wold (1/1). Whinstone Axe Head from a Barrow on Calais Wold (2/3). Weapons and implements of stone! May we not justly call their makers Men of the Stone Age? They lived before man knew how to dig metals from the earth, and how, having obtained them, to melt and mould them to his wish. But besides these weapons which have lain buried with their owners for some thousands of years, there are yielded up by the barrows earthenware vessels of different sizes and shapes. Some, like that shown below, are wide-mouthed and have a thick rim; others are narrower, and their rim is not thickened. Then others have an overhanging rim; and others, again, are small, only an inch or two in height, and have from two to six holes perforated in their
- 48. sides. All aremarked with simple patterns, made by pressing the pointed end of a stick or the thumb-nail into the moist clay, or by pressing round it a twisted thong of hide. There has been no glazing and no attempt to make use of artificial colour. Food Vessel from a Barrow on Acklam Wold (1/2). Each of these vessels has had its particular use. The first-named vessels, which are by far the most common, are always found to be stained with some decomposed matter on the inside of the bottom, and their use has undoubtedly been as food vessels. So also we may consider the second group to be drinking vessels. The food and drink which these two contained when they were buried have been intended for their owners in the new life to come, when food and drink would be again required. The vessels of the third kind are always found to contain remains of a body which has been cremated before burial—hence their name cinerary urns—and the last-named and smallest, which are found with them, have probably been used to hold the precious spark of fire which lit the funeral pyre.
- 49. The Rudston Monolith Letus leave these howes and barrows and examine another example of the work of the Men of the Stone Age. Close to the wall of the village church at Rudston stands a huge upright stone, or monolith. Twenty-five feet is its height above the ground, and sixteen feet its girth, while it is said to be embedded in the ground as deep as it is high above the surface. Its weight is estimated as not far short of forty tons. What is it doing in a village churchyard, and who put it there? When and how was it placed where it now stands?
- 50. The earliest kindof Axe used in East Yorkshire. It is impossible to give any definite answers to these questions. A century ago, however, the village people answered them all very easily. The Devil, they said, objected to the building of the church, and flung this stone to destroy it before its completion. But his aim was not so accurate as it was intended to be, and the missile missed its mark. Asked for a proof of their wonderful story, they would point to the stone itself. There it was for everyone to see. What further proof could be needed?[4] Whether we believe this legend or not, two things are certain. First, that the stone is as old as the barrows in the surrounding wolds; secondly, that there is no rock of the same nature nearer to it than Filey Brig and the Brimham Rocks. Was it brought down by the great ice sheet and then erected by the men of the Stone Age to serve some purpose in their heathen rites, or did they bring it up from Filey or down from the hills of the North Riding on wooden rollers? Perhaps it is not more difficult to conceive of their doing this than of their raising such a huge barrow as that which stands unopened at the foot of Garrowby Hill—a mound 250 feet in diameter at its base and 50 feet in height.
- 51. IV. MEN OF THEBRONZE AGE.
- 52. Bronze Celt or AxeHead found at Swine. The Ancient Britons. With the coming of Julius Caesar to Britain in the middle of the first century before the birth of Christ, we reach the time in the history of our country when definite facts about its people begin to be recorded. Thus we know from Caesar’s own writings that the Britons lived in houses like those of the Gauls, that they had great numbers of cattle, that they used copper coins, that many of the inland tribes did not grow corn but lived on milk and flesh and went clothed in skins, that in war time they dyed their bodies with a blue stain to give them a more terrible aspect, and that they wore long hair on their heads and their upper lips. So also, with regard to their religion, Caesar tells us that their priests were called Druids; that if any crime had been committed, or if there were any dispute about an inheritance or a boundary, it was the Druids who gave judgment; that they had vast stores of learning, all of which was committed to memory and none committed to writing; and that their chief doctrine was that the soul of man did not perish, but passed after death into another body, so that no man should fear death. From these accounts we see that there had been great progress made since the times described in the last chapter. This was due to the migration westwards of a new race of people—the Kelts—who had gained a knowledge of the use of metal, and who, consequently, had weapons and implements made of bronze instead of stone. Their greater knowledge gave them greater power, and the extinction of the men of the Stone Age was only a question of time. For not often was the bronze-weaponed warrior slain by a weapon of stone.
- 53. But the accountwritten by Julius Caesar refers to the inhabitants of the southern parts of our island. ‘Many of the inland tribes do not grow corn, but live on milk and flesh and go clothed in skins.’ This passage may be taken as true of the tribes living north of the Humber, known—so later Roman writers tell us—as the Brigantes, the wildest and most savage of the tribes inhabiting Britain. Let us see what Mr. Mortimer’s discoveries have to tell us of these Brigantes. The most interesting discovery, perhaps, was that made in a barrow on Calais Wold, the highest point of the Yorkshire Wolds, 807 feet above sea-level. Here, on the mound being removed, a double row of stake-holes was exposed in the surface of the ground. These were from 3 to 15 inches in diameter, and were arranged in circles having diameters of 21½ and 28 feet. Outside these were four other stake-holes, and beyond these again a circular trench 100 feet in diameter, 3 feet 9 inches deep, 9 feet across at the top, and 1 foot across at the bottom. Within the double circle of stake-holes was a cavity cut in the chalk and containing a skeleton lying on its side, with its knees bent. The plan on the opposite page shows the arrangement exactly, and the drawing which accompanies it gives Mr. Mortimer’s clever conjecture of the meaning of the stake-holes. The space enclosed between the inner and outer walls would be used, Mr. Mortimer thought, as a storage place for food, skins, and weapons. It would also serve to keep the inside living-room warm in winter.
- 54. Ideal Restoration ofthe Site of Burial.
- 55. Plan of aBarrow on Calais Wold, showing the Encircling Trench and Stake-holes. ‘We will bury our chieftain in his home, which no one after him shall have power to defile.’ So, probably, thought those who buried him. But, if so, time has played them false; for men of a race undreamt of and speaking a tongue of which he would understand hardly one word, have ruthlessly laid bare his burial place, and have carted away his bones to be measured with tape and pencil, and his skull to have its brain cavity estimated with grains of millet seed. What an insult added to injury! A mighty chieftain he had doubtless been, and it must be his favourite weapon that lies buried with him, so placed that he should be buried as he slept—grasping its handle firmly in his right hand. One wonders how many of his enemies’ skulls that weapon of his had beaten in before its master ceased to use it. Perhaps it had
- 56. been wielded againstthe Roman legions brought north of the Humber by Ostorius Scapula in A.D. 50. Who knows? If you would see the head of the weapon you must go to the museum at Driffield; its likeness you will find on page 16. The Brigantes buried their dead chiefs just as the earlier tribes had done, and the photograph on page 25 shows very clearly the curious way in which the legs were doubled and the head bent back. This skeleton was obtained from a barrow in Garton Slack, and here is what its discoverer says of the pains taken to obtain it:— ‘Being desirous of possessing this skeleton in its entirety, we obtained a quantity of stiff, mortar-like material, scraped from the adjoining high road, with which we covered the remains, in order to keep all the bones in position. We then passed three broad pieces of sheet iron under it without displacing any of the bones. The remains were then lifted on a prepared board, and conveyed to Fimber. After being carefully cleaned, the skeleton was mounted in a glass case, and now, with its relics, and part of the ground on which it was found, forms a highly interesting relic in the museum at Driffield.’ The skeleton is that of a woman, and with it, you will notice, are two objects. There is no need to say what has been the use of the bone ornament lying behind the head, but the use of the flint implement placed before the jaw is not so obvious. This is one of a class of implements known to us as scrapers—roughly chipped pieces of flint used by the women of a household in scraping the insides of animal skins when preparing them for human wear, and in scraping the roots that went into the ‘stock-pot’ with the flesh of the animals that provided also garments and beds for the household.
- 57. Photo by] [C.W.Mason British Gold Coin. Found at Atwick by Mr. W. Morfitt (1/1). How a British Chieftain’s Wife was buried in Garton Slack. In neither of these two barrows was there any sign of a bronze implement. Weapons and implements of bronze are rare among those found in the barrows of East Yorkshire, and the few discovered are dagger or knife heads and prickers. The Brigantes were far behind the Britons of the south in their knowledge of the use of metal; and at the time when the latter were making use of bronze, the wild and savage tribes of the north were content still to make use of greenstone and flint. Personal ornaments, too, are rare, and were found accompanying only fifty-seven out of eight hundred and ninety-three burials that Mr. Mortimer excavated. They include dress-fastenings, such as rings and links of jet, and buttons of amber, jet and bone. With only one British interment was gold found, and of silver ornaments none were discovered at all.
- 58. Welcome to ourwebsite – the ideal destination for book lovers and knowledge seekers. With a mission to inspire endlessly, we offer a vast collection of books, ranging from classic literary works to specialized publications, self-development books, and children's literature. Each book is a new journey of discovery, expanding knowledge and enriching the soul of the reade Our website is not just a platform for buying books, but a bridge connecting readers to the timeless values of culture and wisdom. With an elegant, user-friendly interface and an intelligent search system, we are committed to providing a quick and convenient shopping experience. Additionally, our special promotions and home delivery services ensure that you save time and fully enjoy the joy of reading. Let us accompany you on the journey of exploring knowledge and personal growth! textbookfull.com