Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Deploying large-scale, serverless and asynchronous systems - without integrated tests

384 views

Published on

Modern distributed architectures are increasingly composed of large numbers of decoupled, asynchronous components. In AWS, these components are plumbed together via services like SQS, SNS, Kinesis and S3 often integrated via small and frequent numbers of microservices or lambdas. But how do you test these architectures if they are cloud native?

It’s 2018, and we can do better than deploying the entire stack and running a battery of E2E tests against them.

In his talk, Matt will demonstrate how you can scale development of large-scale systems across teams, technology and process, and unlock the agility of your cloud-native architecture.

WARNING: there will be code (https://github.com/mefellows/serverless-testing-example)

Published in: Technology
  • Be the first to comment

Deploying large-scale, serverless and asynchronous systems - without integrated tests

  1. 1. Deploying large-scale, serverless and asynchronous systems Matt Fellows @matthewfellows ...without integrated tests!
  2. 2. Architecture
  3. 3. 3 Evolution Architecture
  4. 4. 4 Release Architecture > Evolution MoreEvolved Time Monolith SOA Serverless Microservices Testing Culture Unit tests, TDD, BDD Contract tests Waterfall Outsourcing API Testing Scenario Tests Fn Not all primitives are invisible! Co-evolved practices: XP AgileLean Multiple E2E Test Env. Chaos Engineering DevOps CD CI
  5. 5. 5 If you can’t build a good monolith you shouldn’t be doing microservices Architecture > Microservices
  6. 6. Entry criteria to make µServices ● Ability to write a modular monolith ● Strong grasp on your domain model ● Ability to design loosely coupled services ○ = state + transactions ● Ability to scale and decouple teams ● Ability to deploy continuously ● A strong “DevOps” culture ● Be comfortable with decentralised, complex systems + uncertainty Architecture > Microservices
  7. 7. ...the µServices ride ● Ability to write a modular monolith ● Strong grasp on your domain model ● Ability to design loosely coupled services ○ = state + transactions ● Ability to scale and decouple teams ● Ability to deploy continuously ● A strong “DevOps” culture ● Be comfortable with decentralised, complex systems + uncertainty Architecture > Microservices
  8. 8. ...the µServices ride ● Ability to write a modular monolith ● Strong grasp on your domain model ● Ability to design loosely coupled services ○ = state + transactions ● Ability to scale and decouple teams ● Ability to deploy continuously ● A strong “DevOps” culture ● Be comfortable with decentralised, complex systems + uncertainty Architecture > Microservices
  9. 9. ...the µServices ride ● Ability to write a modular monolith ● Strong grasp on your domain model ● Ability to design loosely coupled services ○ = state + transactions ● Ability to scale and decouple teams ● Ability to deploy continuously ● A strong “DevOps” culture ● Be comfortable with decentralised, complex systems + uncertainty Architecture > Microservices
  10. 10. WHAT WE ARE KNOWN FOR 10 Microservices Evolution Drivers for levelling up ● Increase time-to-value ● Increase utilisation / cost reduction ● Raise abstraction; focus on composition ● Scale product teams ● Massively distributed, real-time apps ● API contracts (e.g. OAS) ● Ubiquity of cloud computing ● Rise of containers ● Unreliable networking / components ● Rise of the Service Mesh ● Improved observability Architecture > Microservices
  11. 11. Functions + Serverless
  12. 12. 12 Architecture > Functions + Serverless (Cloud Native) https://aws.amazon.com/retail/data-analytics/
  13. 13. 13 https://www.slideshare.net/AmazonWebServices/new-launch-data-driven-apps-with-graphql-aws-appsync-deep-dive-mbl402-reinvent-2017 Architecture > Functions + Serverless (Cloud Native)
  14. 14. 14 https://aws.amazon.com/blogs/big-data/analyze-a-time-series-in-real-time-with-aws-lambda-amazon-kinesis-and-amazon-dynamodb-streams/ Architecture > Functions + Serverless (Cloud Native)
  15. 15. 15 https://aws.amazon.com/blogs/big-data/automating-analytic-workflows-on-aws/ Architecture > Functions + Serverless (Cloud Native)
  16. 16. 16 https://www.slideshare.net/nateware/aws-gdc2015-gameanalyticsfinal Architecture > Functions + Serverless (Cloud Native)
  17. 17. 17 Architecture > Functions + Serverless (Cloud Native) https://aws.amazon.com/retail/data-analytics/
  18. 18. 18 Much Compose! So speed! Architecture > Functions + Serverless (Cloud Native)Architecture > Functions + Serverless (Cloud Native)
  19. 19. 19 BUT. Architecture > Functions + Serverless (Cloud Native)
  20. 20. 20 If you can’t build good microservices you shouldn’t be doing serverless Architecture > Functions + Serverless (Cloud Native)
  21. 21. 21 If you can’t build a good Fn you shouldn’t be doing FaaS Architecture > Functions + Serverless (Cloud Native)
  22. 22. 22 So, what practices do we need? Architecture > Functions + Serverless (Cloud Native)
  23. 23. AWS Meetup Lounge
  24. 24. 24 Demo: AWS Meetup Lounge CloudFront API Gateway Authentication You! awslounge.onegeek.com.au Twitter Fetch Twitter feed Twitter Push IoT Message Broker Sentiment Analysis Checkpoint Amazon ML #awsmelb
  25. 25. Challenges w Serverless Blasphemy!
  26. 26. 26 Vendor lock-in Challenges > Vendor lock-in
  27. 27. 27 https://twitter.com/alambert/status/958050456101642241 Challenges > Vendor lock-in
  28. 28. WHAT WE ARE KNOWN FOR 28 Options ● Multi-cloud = lowest common denominator ● There is always a vendor (even if it’s you!) ● You’ll never be completely vendor agnostic ● Multi-cloud abstraction ● Avoid altogether ● Structure your code better Challenges Challenges > Vendor lock-in
  29. 29. WHAT WE ARE KNOWN FOR 29 Options ● Multi-cloud = lowest common denominator ● There is always a vendor (even if it’s you!) ● You’ll never be completely vendor agnostic ● Multi-cloud abstraction ● Avoid altogether ● Structure your code better Challenges Challenges > Vendor lock-in
  30. 30. Lambda Handler Challenges > Vendor lock-in > Lambda Anti-pattern { "Records": [ { "EventVersion": "1.0", "EventSubscriptionArn": "somearn", "EventSource": "aws:sns", "Sns": { ... } } ] } { "Success": true }
  31. 31. MQ Handler WebSocket handler MQ Adapter WS Adapter Azure Function Adapter Google Cloud Fn Adapter Azure Function Handler Google Cloud Fn Handler Lambda Handler Lambda Adapter Service Repository Adapter (Anti-corruption layer) Ports Business Logic Collaborators ... Challenges > Vendor lock-in > Ports and Adaptors
  32. 32. 32 Challenges > Vendor lock-in > Ports and Adaptors > Sentiment Analysis Example Twitter feed Sentiment Analysis IoT Message Broker
  33. 33. Challenges > Vendor lock-in > Ports and Adaptors > Lambda Port
  34. 34. Challenges > Vendor lock-in > Ports and Adaptors > SNS Adapter
  35. 35. Challenges > Vendor lock-in > Ports and Adaptors > Lambda Sentiment Service
  36. 36. Summary Architecture > Microservices ● Separate protocol handling from business logic ● Business logic shouldn’t change with introduction of a new protocol ● Port, Adapter and Business Logic are independently testable
  37. 37. 37 How do you test locally? Challenges > Testing Locally
  38. 38. WHAT WE ARE KNOWN FOR 38 Options Challenges > Testing Locally ● Stub services (e.g. Localstack) ● Stub API (e.g. Moto) ● Multiple handlers ● Unit tests
  39. 39. 39 Challenges > Testing Locally
  40. 40. WHAT WE ARE KNOWN FOR 40 Options ● Feature arms race ● Trustworthy? ● Integration style = slower ● All cloud providers? ● Stub services (e.g. Localstack) ● Stub API (e.g. Moto) ● Multiple handlers ● Unit tests Challenges > Testing Locally Challenges
  41. 41. WHAT WE ARE KNOWN FOR 41 Writing good unit tests ● Follow test pyramid ● Stub out AWS APIs ● Test your business logic separately from any serverless features (e.g. lambda handler) ● Callback pyramid of death ● Tests are hard to write ● Your business logic knows about AWS ● You have to stub multiple services per test ● Large methods ● Confusing control flow Challenges > Testing Locally > Unit Test Code Smells
  42. 42. 42 Challenges > Testing Locally > Unit Test > Twitter Fetch Checkpoint Twitter Fetch
  43. 43. Challenges > Testing Locally > Unit Test > Lambda Service Look ma, no AWS Code
  44. 44. Challenges > Testing Locally > Unit Test > Lambda Service Look ma, no AWS Code
  45. 45. 45 Challenges > Testing Locally > Unit Test > Sentiment Analysis Example Twitter feed Sentiment Analysis
  46. 46. Integration-style test of handler Challenges > Testing Locally > Unit Test > Lambda Handler
  47. 47. Summary Architecture > Microservices ● Business logic should be tested separately from port + adapter ● Majority of tests, should be fast unit tests ● Stub AWS Services where required ● (Lambda) Handlers can be locally integration tested with pre-canned JSON events
  48. 48. 48 Async - who is my consumer? Challenges > Async
  49. 49. 49 Async - delay in feedback Challenges > Async
  50. 50. WHAT WE ARE KNOWN FOR 50 Options ● Local integration testing ● Cloud env. integration testing ● Contract testing Challenges > Async
  51. 51. WHAT WE ARE KNOWN FOR 51 Options ● Can’t run complete local environment ● Many moving parts ● Problems with integration testing... ● Local integration testing ● Cloud env. integration testing ● Contract testing Challenges Challenges > Async
  52. 52. WHAT WE ARE KNOWN FOR 52 Options ● Can’t run complete local environment ● Many moving parts ● Problems with integration testing... ● Local integration testing ● Cloud env. integration testing ● Contract testing Challenges Challenges > Async
  53. 53. 53 Contract tests Challenges > Async
  54. 54. 54 IoT Message Broker Sentiment Analysis Contract { 'Positive': 0.234, 'Negative': 10.123, 'Neutral': 7.334, 'Mixed': 0 } Consumer Provider Challenges > Async > Contract Tests
  55. 55. 55 CloudFront API Gateway Authentication You! awslounge.onegeek.com.au Twitter Fetch Twitter feed Twitter Push IoT Message Broker Sentiment Analysis Checkpoint Amazon ML #awsmelb Challenges > Async > Contract Tests
  56. 56. 56 Authentication Twitter Fetch Twitter feed Twitter Push IoT Message Broker Sentiment Analysis Checkpoint #awsmelb Challenges > Async > Contract Tests Contract Contract Contract Contract Contract Contract Contract Contract
  57. 57. WHAT WE ARE KNOWN FOR 57 Problems with automated e2e tests ● Slow ● Easy to break ● Hard to fix ● Scales badly across teams ● Lots of set up maintenance ● $$ potentially costly Challenges > Async > Contract Tests
  58. 58. 58 C P mock Challenges > Async > Contract Tests Understanding Test Symmetry
  59. 59. WHAT WE ARE KNOWN FOR 59 Test symmetry Solved problems New problems ● Hard to keep both sides in sync ● Fast feedback ● Few dependencies ● No dedicated environment ● Reliable ● Easy to debug Challenges > Async > Contract Tests
  60. 60. 60 C P mock Challenges > Async > Contract Tests
  61. 61. 61 C P mock Challenges > Async > Contract Tests
  62. 62. PPCC mock contract Challenges > Async > Contract Tests
  63. 63. Bug turnaround - minutes Contracts tests Challenges > Async > Contract Tests
  64. 64. Know before you commit Contracts tests Challenges > Async > Contract Tests
  65. 65. Make changes with speed and confidence Contracts tests Challenges > Async > Contract Tests
  66. 66. Deploy independently Contracts tests Challenges > Async > Contract Tests
  67. 67. Better API design (Consumer) Contracts tests Challenges > Async > Contract Tests
  68. 68. 68 ● Open source ● Multiple languages ○ Javascript ○ Go ○ Python ○ JVM ○ .NET ○ + more pact.io ● HTTP contracts ● Message contracts
  69. 69. How to be taller ...and ride that Fn
  70. 70. Summary 70 Summary Problems Solutions ● Microservices+ ● Create cloud and protocol anti-corruption layer ● Mock out AWS SDK in unit tests ● Use contract tests ● Reduce reliance on end-to-end environments ● Ad-hoc exploratory testing ● Serverless complexity > uServices ● Testing cloud native services ● Avoiding vendor lock-in ● Handling asynchronous Fn’s
  71. 71. Deploying large-scale, serverless and asynchronous systems Matt Fellows @matthewfellows ...without integrated tests!

×