The document discusses preparing to operate systems at scale on AWS. It covers reviewing the Well Architected operational excellence pillar, setting operational priorities, designing with operations in mind, ensuring operational readiness, defining success, and enabling self-service. It also provides examples from Skyscanner's journey to operating on AWS, including planning considerations, cost management, security, and automating tasks like project creation and user management.

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Isabel Huerga Ayza
Technical Account Manager, Amazon Web Services
Paul Moran - @pdmoran_aws
Technical Account Manager, Amazon Web Services
Operating at Scale: Preparing for the
Journey

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our Goal for Today…
A clearer understanding of what
Operational Excellence
should look like in your
Organisation

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Well Architected…
A quick review of the Well Architected Framework -
Operational Excellence
Security
Reliability
Performance Efficiency
Cost Optimisation

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Where do we Start?…
The Operational Excellence
Design Principles

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Where do we Start?…
Perform Operations as Code

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Where do we Start?…
Well Annotated Documentation

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Where do we Start?…
Frequent, Small, Reversible
Changes

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Where do we Start?…
Anticipate failure

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Where do we Start?…
Learn from all operational failures

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
This Means…
You need to:
Prepare
Operate
Evolve

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
This Means…
You need to:
Prepare
Operate
Evolve

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
This Means…
You need to:
Prepare
Operate
Evolve

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
This Means…
You need to:
Prepare
Operate
Evolve

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
We’re Focusing on…
You need to:
Prepare
Operate
Evolve

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you prepare…
Set Your Operational Priorities

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you prepare…
Supporting Resources
AWS Cloud Compliance
AWS Trusted Advisor
AWS Business Support
AWS Enterprise Support

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Think about Support...
Basic Support level includes:
 Customer Service 24x7
 Support Forums
 Support for Health check
 Documentation, White Papers, Best Practice Guides
Developer Support level includes:
 All of Basic support
+ Email Technical Support
+ 1 Named contact
+ Response Time < 12 hours & by Cloud Support Associate
Business Support level includes:
 All of Developer support
+ Email / Phone / Chat / Screen Sharing Technical Support
+ Response Time < 1 hours & by Cloud Support Engineer
+ 3rd Party Software Support (OS, DB, Web-servers, Networking, etc)
+ AWS Trusted Advisor & Support API
Enterprise Support level includes:
 All of Business support
+ Technical Account Manager (TAM)
+ Response Time < 15 min & by Sr. Support Engineer
+ Dedicated Concierge Services
+ Infrastructure Event Management
+ Management Business Reviews
https://aws.amazon.com/premiumsupport/enterprise-support/

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you prepare?…
Design with Operations in Mind

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you prepare?…
Design with Operations in Mind

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you prepare?…
Supporting Resources
AWS CloudFormation
AWS Developer Tools
AWS X-Ray

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you prepare?…
Operational Readiness

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you prepare?…
Operational Readiness

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you prepare?…
Supporting Resources
AWS Config
Amazon EC2 Systems Manager

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you prepare?…
What does Success look like?

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you prepare?…
What does Success look like?

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you prepare?…
Food for thought

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What Helps?…
Culture

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What Helps?…
=
Culture
Values + Behaviours

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What Helps?…
Hire and Develop the Best
Customer Obsession
Ownership
Invent and Simplify
Are Right, A Lot
Think Big
Learn and Be CuriousInsist on the Highest Standards
Bias for Action
Dive Deep
Have Backbone; Disagree and Commit
Deliver Results
Earn Trust
Frugality

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What Helps?…
Think Product

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What Helps?…
Security

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What Helps?…
Openness

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What Helps?…
Change

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What Helps?…
Never stop iterating
Well-
Architected
Build
Trusted Advisor
Check Run
Cloud Operations Review
Cloud
Adoption
Framework
Plan

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What Helps?…
Don't let the failures of today be
the reason for the failures of
tomorrow
- Bobby Kennedy

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
William Fairbrother
Engineering Manager, Skyscanner
Alan Cooper
Principle Engineer, Skyscanner
How do you prepare?

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Intros
Alan Cooper
Principal Engineer within Developer Enablement Tribe
and former Infrastructure Manager.
William Fairbrother
Engineering Manager within Developer Enablement and
former Squad Lead in Cloud Operations.

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Quick Facts
Over 70m active users per month
Record 5m+ sessions per day
30 languages, 150 currencies
Gross bookings Value 11bn per annum +
Available on iOS, Android, Desktop and B2B channels

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our Journey
Kicked off with 4 regions in Hybrid Mode
Learnings throughout the journey
Running multiple production workloads
Using many different AWS service

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Systems operating at scale

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Solution planning considerations
How will your users smoothly interact with AWS?
CloudFormation access – our story

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Solution planning considerations
Planning your deployment strategy
Training

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Solution planning considerations
Living with limits

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Solution planning considerations
Ensuring you have capacity when you
need it

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Solution planning considerations
Account structure/design

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cost Management Planning
Plan to focus regularly on usage to help spend wisely

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cost Management Planning
Choose your cost management tools and methods
wisely

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cost Management Planning
Learn lessons early regarding cost

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Planning for security
Bake security into every layer
Secure your perimeters

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Planning for security
IAM security
Other security initiatives

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you prepare…
You can’t do it all your self…
Very quickly we realised a few important points
• We didn’t have a 3 year view on how things would look
• We didn’t, couldn’t and shouldn’t do all admin tasks centrally
• Automate and Delegate – Tooling and SelfServe Portal

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you prepare…
You can’t do it all your self…
Very quickly we realised a few important points
• We didn’t have a 3 year view on how things would look
• We didn’t, couldn’t and shouldn’t do all admin tasks centrally
• Automate and Delegate – Tooling and SelfServe Portal

54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you prepare…

55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do you prepare…
SelfServe
(Organisationally Scale)
Project creation
Tooling Integration
New User creation and Password Reset

56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A specific challenge
NAT Gateway
EC2 NAT Gateways very cost effective compared to Managed NAT
EC2 NAT gateways give better visibility of traffic – Linux stats &
TCPDump
Scale is an issue – Lambda & Boto is our saviour

57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A specific challenge
NAT Gateway
EC2 NAT Gateways very cost effective compared to Managed NAT
EC2 NAT gateways give better visibility of traffic – Linux stats &
TCPDump
Scale is an issue – Lambda & Boto is our saviour

58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Final thoughts
william.fairbrother@skyscanner.net
alan.cooper@skyscanner.net

59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Handy Links…
https://aws.amazon.com/architecture/well-architected/
https://aws.amazon.com/compliance/
https://aws.amazon.com/premiumsupport/trustedadvisor/
https://aws.amazon.com/cloudformation/
https://aws.amazon.com/developers/getting-started/
https://aws.amazon.com/xray/
https://aws.amazon.com/config/
https://aws.amazon.com/systems-manager/

60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Isabel Huerga Ayza
Technical Account Manager, Amazon Web Services
Paul Moran - @pdmoran_aws
Technical Account Manager, Amazon Web Services
Operating at Scale: Preparing for the
Journey