My talk from the Auckland DevOps meetup in April 2019 https://www.meetup.com/DevOps-Auckland/events/260240205/

1. Serverless
DevOps Lessons Learned From Production

2. About Me - Steve Hogg
● Past - Founder of Web Drive Hosting Company
○ I ran the Operations team for shared hosting platforms with 20,000+ websites circa 2003
○ I ran large scale migrations of customer platforms to cloud based servers
● Now - AWS Contractor
○ Built many Serverless platforms over the last 2 years with IoT Labs and Blockchain Labs
○ Honed my Dev skills with TDD, Integration, Functional testing etc

3. Serverless - DevOps Lesson Learned
This presentation:
● Concepts - Abstract Ideas
● Demos - Those ideas in-action

4. What is Serverless?

5. What is Serverless
● Services where you can pretend that servers don’t exist
○ They do, but you’re not responsible for them
● Not just Functions-as-a-Service (FaaS)
○ Storage
○ Databases
■ DynamoDB
■ Athena
○ Queues
○ API services like API Gateway and AppSync
○ 3rd party services

6. Some Typical Uses
● Functions called directly
● Worker processes that read from queues
● The backend for APIs
● Single Page Apps
● Functions that customise cloud services
○ Pre and post event triggers
● Often used in combination with third-party services e.g. Auth0

7. Resources
Things I got started with

8. The “Evolution” of Application Platforms
https://www.youtube.com/watch?v=oE5lrNn7bAg - Yochay Kiriaty

9. Serverless Security
https://www.youtube.com/watch?v=CiyUD_rI8D8
Dependency vulnerability analysis: https://snyk.io/

10. Serverless - NoOps Is A Myth
“The cost and pain of developing software is approximately zero compared to the
operational cost of maintaining it over time.” - Charity Majors
https://www.youtube.com/watch?v=hG39tB5qqMc
● NoOps is a myth, Operations has just changed focus.
○ Serverless === MoreOps; you give up observability e.g. you can’t strace
○ The system still needs Operations
■ You need logging, monitoring, backups, triage (e.g. slow queries), pipelines
○ 10 to 30% of your budget should be on observability tools
○ Plan for failure e.g. fail-safe queues between services

11. Lambda Demos

12. Demos
Simple Lambda Examples

13. Serverless State Machines

14. State Machine Services - e.g. Step Functions

15. Demos 2
State Machines Examples

16. Lessons Learned

17. Lessons Learned - How Do You Eat An Elephant?
● Minimise the size of the learning pit: get started with low-hanging fruit
○ Worker jobs that get triggered with events are a good choice
■ Run on a timer - e.g. cron
■ Run when an image is uploaded
■ Process messages in a queue
The Learning Pit - Dr John Edwards

18. Lessons Learned - 1 - Getting Started
● IaC is great; deploy examples and learn from them
○ https://github.com/awslabs/serverless-application-model/tree/master/examples/apps
● Embrace the concept of time-to-value
○ Have a bias towards efficiency - “What is the smallest amount of work I can do to get this
working?”
○ Not due to laziness, but in the interest of getting a great result quickly
○ This makes you look for existing services you can compose into what you need
○ This makes you “sharpen your tools”, to learn more efficient ways of working
○ Be result focussed and take charge of the fundamental attribution error
■ We judge ourselves on our intentions, everyone else judges us on the result
■ The only powerful position we can take to to judge ourselves on the result to
● Have a Serverless-first approach
○ Put effort into coming up with a Serverless solution, as the up-front effort pays off

19. Lessons Learned - 2 - Getting Started
● Avoid premature optimisation
● Build things in multiple passes, adding layers of sophistication as you go
● It is OK to start building things with the console first before moving to IaC *
● It is OK to deploy IaC manually first before automating it with a pipeline *
● It is OK to prototype things without TDD when you only have a fuzzy idea *
● It is OK to have low-coverage integration tests at first *
● * Communication is critical. The whole team needs to buy into the per-service
level of engineering being applied, and that refactoring is natural part of the
process.

20. Lesson Learned - 2 continued - Agreed Levels
Define and agree on the per-service level of engineering being targeted i.e. quality
Beginner Novice Competent Proficient Expert
Level 1 - Prototype Level 2 - Beta Level 3 - Small Scale Level 4 Level 5
- TDD + TDD (50%) + TDD (80%)
- Integration Tests
- Smoke Tests
- Functional Tests
+ Continuous
Integration
- Continuous
Delivery
- Continuous
Deployment
+ Infrastructure-as-
Code
- Feature flags
- Software publish
as modules
+ TDD (90%)
+ Integration Tests
+ Smoke Tests
+ Functional Tests
+ Continuous
Integration
+ Continuous
Delivery
- Continuous
Deployment
+ Infrastructure-as-
Code
- Feature flags
- Software publish
as modules
+ TDD (90%)
+ Integration Tests
+ Smoke Tests
+ Functional Tests
+ Continuous
Integration
+ Continuous
Delivery
+ Continuous
Deployment
+ Infrastructure-as-
Code
+ Feature flags
+ Software publish
as modules

21. Lessons Learned - 3
● Deploy. Lots.
○ The whole system is defined as a IaC template, which is easy to deploy.
○ You only pay for resources when you use them.
○ Allows you to create multiple environments like Dev, Staging, UAT, Prod.
○ You can spin up an environment, run integration tests and a report, then tear it all down.
○ A developer can deploy their own copy to get familiar with the system.
○ You can start up a new environment for a customer demo quickly

22. Lessons Learned - 4
● You can use multiple accounts as an alternative to multi-tenancy
○ Customers can pay for their own resources
○ Customers are responsible for their own Operations (backups, account security etc.)
● Use multiple accounts as a permission boundary
○ Lock down sensitive data, Lambdas and state machines into their own account, and use
cross-account permissions to allow another named account to use them.

23. Lessons Learned - 5
● Version your templates
○ It is useful to have a parameter in the template that has the version number.
○ This helps when you have multiple deployments
○ It can be given to the functions too via environment variables e.g. read the version number
from package.json and include it as a param

24. Lessons Learned - 6
● Have do and undo actions e.g. Saga Pattern
○ https://theburningmonk.com/2017/07/applying-the-saga-pattern-with-aws-lambda-and-step-functions/

25. Lessons Learned - 7 - Use Fail-Safe Queues

26. Lessons Learned - 8
● Pub/Sub is a nice way to decouple services
○ https://www.jeremydaly.com/serverless-microservice-patterns-for-aws/

27. Lessons Learned - 9
● Allow resume from any state
○ https://aws.amazon.com/blogs/compute/resume-aws-step-functions-from-any-state/

28. Lessons Learned - 10
● Make your state machines idempotent
○ When a state machine fails due to a transient error, e.g. a third-party service is down
○ If the state machine stages are idempotent, you can “copy and paste” the output for the failed
state machine into a new one that will pick-up where the last one left off.

29. Lessons Learned - 11 - Compose Services
● https://www.jeremydaly.com/serverless-microservice-patterns-for-aws/

30. Serverless Application
Services

31. AWS AppSync and AWS Amplify
● AWS AppSync is a GraphQL API service with integrations with Lambda,
DynamoDB, ElasticSearch etc.
● AWS Amplify is an application framework and SDK for the rapid creation of
applications. It works with API GateWay and/or AWS AppSync

32. Demos 3
Amplify and AppSync

33. Serverless and IaC
My thoughts on DevOps

34. Serverless and Infrastructure-as-Code are a great fit
The whole system: code and infrastructure can be defined in set of templates,
allowing simple CI/CD
https://github.com/aws-samples/codepipeline-nested-cfn/blob/master/codepipeline-cfn-codebuild.yml

35. DevOps - Responsibility
Who is responsible for this mess? Who’s fault is this?
● The word responsible and fault are often used interchangeably
● “Fault” looks to attribute blame
● Taking “Responsible” back to its origin, it means able to respond
● Developers used to just be responsible for code, Operations used to be just
responsible for running code
● DevOps culture aims to create shared responsibility: reliable software running
in production
● DevSecOps aims for secure, reliable software running in production
● Mark Schwartz talks about DevSecFinBizOps

36. DevOps - Responsibility Example
A practical example of DevSecFinBizOps using Serverless and IaC

37. DevOps - Responsibility - Future Focus (R)
● The New Zealand Olympic Team do not do a review of their performance at
the previous games.
● They do a preview of the next Olympic Games.
○ Data from the previous games will naturally be introduced during the preview, but not for its
own sake.
○ This avoids getting bogged down in the current reality, instead building excitement for the
future.
● Similar to SRE Blameless “Postmortem”

38. The End
Questions?

39. My Details
● blog.h4.nz
● steve@h4.nz
● https://www.linkedin.com/in/steve-hogg/