Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Adrian Hornsby, Technical Evangelist @ AWS
Twitt...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A bit about me
Technical Evangelist, Developer A...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why 10 lessons?
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Transient state
do not belong in
the database.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Immutable Infrastructure
• No updates on live sy...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Infrastructure as Code
• Template of the infrast...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Message passing for async. patterns
A
Queue
B
A
...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Async. Architecture (part 1)
Web
Instances
Worke...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Async. Architecture (part 2)
Worker
Instance
Wor...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
RDS DB Instance
Active (Multi-AZ)
Availability Z...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Event driven pattern
A B CEvent on B by A trigge...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How Lambda works
S3 event
notifications
DynamoDB...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Event-driven using Lambda
AWS Lambda:
Resize Ima...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Read / Write Sharding
RDS DB Instance
Read Repli...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database Federation
Users
DB
Products
DB
App
Ins...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database Sharding
User ShardID
002345 A
002346 B...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Measure __everything__
• Nurture a culture of me...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DynamoDB at Amazon
“A deep dive on how we were u...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Measuring without Operational Targets is often
U...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The best way to know if you
are resilient to fai...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
https://medium.com/@NetflixTechBlog
https://engi...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliat...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1. EMBRACE FAILURE
2. LOCAL STATE IS A CLOUD ANT...
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
Questions?
Twitter: @adhorn
Upcoming SlideShare
Loading in …5
×

10 Lessons from 10 Years of AWS

6,420 views

Published on

Slides from my talk at the first AWS Community Day in Bangalore
https://www.meetup.com/awsugblr/events/243819403/

Speaker notes: https://medium.com/@adhorn/10-lessons-from-10-years-of-aws-part-1-258b56703fcf
and https://medium.com/@adhorn/10-lessons-from-10-years-of-aws-part-2-5dd92b533870

The list is not in any particular order :)

Published in: Technology

10 Lessons from 10 Years of AWS

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Adrian Hornsby, Technical Evangelist @ AWS Twitter: @adhorn 10 Lessons from 10 years of AWS
  2. 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. A bit about me Technical Evangelist, Developer Advocate, … Software Engineer My home is in Finland Previously: • Solutions Architect @AWS • Lead Cloud Architect, Backend developer, … @Dreambroker • Director of Engineering, Software Engineer, DevOps, ... @Hdm • Researcher @Nokia Research Center • and a bunch of other stuff. Love climbing and ginger shots.
  3. 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Why 10 lessons?
  4. 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EMBRACE FAILURE 1
  5. 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. “It is not failure itself that holds you back; it is the fear of failure that paralyzes you.” Brian Tracy
  6. 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. "organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations." M. Conway
  7. 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. LOCAL STATE IS A CLOUD ANTI-PATTERN 2
  8. 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Transient state do not belong in the database.
  9. 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. IMMUTABLE INFRASTRUCTURE 3
  10. 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Immutable components are replaced for every deployment, rather than being updated in-place.
  11. 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Immutable Infrastructure • No updates on live systems • Always start from a new instance being provisioned • Deploy the new software • Test in different environments (dev, staging) • Deploy to prod (inactive) • Change references (DNS or Load Balancer) • Keep old version around (inactive) • Fast rollback if things go wrong
  12. 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. INFRASTRUCTURE AS CODE 4
  13. 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Infrastructure as Code • Template of the infrastructure in code. • Version controlled infrastructure. • Repeatable template. • Testable infrastructure. • Automate it!
  14. 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  15. 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ASYNCHRONOUS AND EVENT-DRIVEN PATTERNS WILL HELP YOU SCALE 5
  16. 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Message passing for async. patterns A Queue B A Queue BListener Pub-Sub SNS, SQS, Redis, RabbitMQ
  17. 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Async. Architecture (part 1) Web Instances Worker Instance Worker Instance Queue API Instance API Instance API Instance API: {DO foo} PUT JOB: {JobID: 0001, Task: DO foo} API: {JobID: 0001} GET JOB: {JobID: 0001, Task: DO foo} ElastiCache Result: { JobID: 0001, Result: bar }
  18. 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Async. Architecture (part 2) Worker Instance Worker Instance Queue API Instance API Instance API Instance ElastiCache Amazon SNS Push Notification User
  19. 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. RDS DB Instance Active (Multi-AZ) Availability Zone Elastic Load Balancer Web Instance Web Instance Amazon Route 53User Amazon S3 Amazon Cloudfront ElastiCache Worker Instance Worker Instance Decoupling Queue Amazon SNS
  20. 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Event driven pattern A B CEvent on B by A triggers C Invocation Lambda functions Action
  21. 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How Lambda works S3 event notifications DynamoDB Streams Kinesis events Cognito events SNS events Custom events CloudTrail events LambdaDynamoDB Kinesis S3 Any custom Invoked in response to events - Changes in data - Changes in state Redshift SNS Access any service, including your own Such as… Lambda functions CloudWatch events
  22. 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Event-driven using Lambda AWS Lambda: Resize Images Users upload photos S3: Source Bucket S3: Destination Bucket Triggered on PUTs
  23. 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. DON’T FORGET TO SCALE THE DATABASE (SHARDING TO THE RESCUE) 6
  24. 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Read / Write Sharding RDS DB Instance Read Replica App Instance App Instance App Instance RDS DB Instance Master (Multi-AZ) RDS DB Instance Read Replica RDS DB Instance Read Replica
  25. 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Database Federation Users DB Products DB App Instance App Instance App Instance
  26. 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Database Sharding User ShardID 002345 A 002346 B 002347 C 002348 B 002349 A CBA App Instance App Instance App Instance
  27. 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. MEASURE, MEASURE, AND MEASURE 7
  28. 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Measure __everything__ • Nurture a culture of measuring everything. • Make it easy to measure. • Setup alarms and escalation paths. • Learn and work to automate responses.
  29. 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. DynamoDB at Amazon “A deep dive on how we were using our existing databases revealed that they were frequently not used for their relational capabilities. About 70 percent of operations were of the key-value kind, where only a primary key was used and a single row would be returned. About 20 percent would return a set of rows, but still operate on only a single table.” Werner Vogels
  30. 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Measuring without Operational Targets is often Useless • How can you progress if you have no targets? • Load test • Set target and make data driven decisions, not hunch based.
  31. 31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. PLAN FOR THE WORST, PREPARE FOR THE UNEXPECTED 8
  32. 32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  33. 33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The best way to know if you are resilient to failure is to TEST failures.
  34. 34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. YOU BUILD IT, YOU RUN IT! 9
  35. 35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  36. 36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. BE HUMBLE, LEARN FROM OTHERS, MAKE HISTORY 10
  37. 37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. https://medium.com/@NetflixTechBlog https://engineering.instagram.com/ http://www.allthingsdistributed.com/ http://highscalability.com/ https://segment.com/blog/categories/engineering/ https://reinvent.awsevents.com/ https://www.meetup.com/topics/amazon-web-services/
  38. 38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. RECAP
  39. 39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 1. EMBRACE FAILURE 2. LOCAL STATE IS A CLOUD ANTI-PATTERN 3. IMMUTABLE INFRASTRUCTURE 4. INFRASTRUCTURE AS CODE 5. ASYNCHRONOUS AND EVENT-DRIVEN PATTERNS WILL HELP YOU SCALE 6. DON’T FORGET TO SCALE THE DATABASE (SHARDING TO THE RESCUE) 7. MEASURE, MEASURE, AND MEASURE 8. PLAN FOR THE WORST, PREPARE FOR THE UNEXPECTED 9. YOU BUILD IT, YOU RUN IT! 10.LEARN FROM OTHERS. 10 Lessons from 10 Years of AWS
  40. 40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you! Questions? Twitter: @adhorn

×