Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
David Dooling & Ryan Richt
October 2015
Cloud Fi...
What to Expect from the Session
Theory of Cloud
Scientists Turned Developers Turned Architects
Ryan
David
Scientists Turned Developers Turned Architects
Monsanto
Theory of Cloud
Theory of Cloud
Automated
Elastic
Highly Available
Security
Software defined everything
Unlimited scale + pay-as-you-go
Ho...
Theory of Cloud Cloud Architecture
Automated Higher-Order Automation
Elastic Ephemeral Environments
Highly Available Fault...
Higher-Order Automation
Automated Tests
Continuous Integration
Continuous Delivery
Automated Infrastructure
Automated Faul...
Fallacies of Internal Apps
1. The hardware is reliable
2. The network is reliable
3. The database is reliable
4. Other ser...
Fault Tolerant
Fallacies of 1st Generation Cloud
1. Other people’s fault tolerant
code is actually fault tolerant
2. Every...
Elastic, Ephemeral, Cost-Effective
time
cost
Cloud
On Prem
Dynamic Env Replication
time
cost
Cloud
On Prem
Experiments
A Do-Over for Secure by Construction
Secure by Assumption
Secure by Design
Security Automation
Horizontally Scalable
1. The overhead of scaling
grows at most linearly with
additional nodes
2. Reads and writes both
sca...
Infrastructure Automation
Federation – 1000 VPCs
Amazon VPC
Amazon VPC
Amazon VPC
Amazon VPC
Amazon VPC
Amazon VPC
Amazon VPC
Amazon VPC
Amazon VPC
...
Cloud Architecture
Cloud Architecture
Cloud Architecture
Cloud Architecture
Cloud Architecture
AWS
CloudFormation
"IPAddress" : {
"Type" : "AWS::EC2::EIP",
"DependsOn" : "AttachGateway",
"Properties" : {
"Domain" : "v...
Cloud Architecture
CloudFormation Template Generator
https://github.com/MonsantoCo/cloudformation-template-generator
CloudFormation
Template
Generator
Referential Integrity
Auto Scaling
Group
CFTG: Security Groups
Stax
$ ./stax --help
Usage: stax [OPTIONS] COMMAND [COMMAND_ARGS]
add Add functionality to an existing VPC
auto-services L...
$ ./stax create
[ ---- ] creating stax
[ NAME ] vpc-stax-36918-outfitting
[ ---- ] creating parameter file
[ ---- ] checki...
$ ./stax connect
[ ---- ] checking if stax build is complete
[ ---- ] describe stax
[ NAME ] vpc-stax-36918-outfitting
[ -...
Stax as a Service - Create
Stax as a Service – List
Stax as a Service – Describe
Stax as a Service – Services
Monsanto
Microservices Lifecycle
Microservices: Cupcakes, Not Wedding Cakes
A modern language for software engineering
Abstract Data Types (ADTs)
Enforced Immutability
Pattern Matching & Destructuri...
Advanced Abstractions
Algebraic Data Types (ADTs)
Enforced Immutability
Pattern Matching & Destructuring
Assignment
Type-L...
Project-as-a-Service 1 – Create Code Repo/Wiki/Issues
Project-as-a-Service 2 – Simple Service Template
Runs giter8 to create a fully functional service written in
Scala based o...
Project-as-a-Service 3 – CI & Dockerization
New check-in Test and Build Dockerize
Project-as-a-Service 4 – Continuous Deployment
fleet
Router
Route Updater
Registrator
A commit is made to GitHub1
1
https://github.com/MonsantoCo/etcd-aws-cluster
https:...
fleet
Router
Route Updater
Registrator
GitHub notifies Jenkins that new code is available.
Jenkins runs automated tests to...
fleet
Router
Route Updater
Registrator
Jenkins builds a Docker container and pushes it to our private Docker registry.3
3
...
fleet
Router
Route Updater
Registrator
Jenkins registers the service with etcd, our key/value store, since it doesn’t exis...
fleet
Router
Route Updater
Registrator
Jenkins calls fleet to deploy the container running our service.5
5
service-1:1
ser...
fleet
Router
Route Updater
Registrator
Registrator notices the service is deployed and registers the location in etcd.6
6
...
fleet
Router
Route Updater
Registrator
When a request is received, the router determines the current revision for the serv...
fleet
Router
Route Updater
Registrator
Next commit (rev 2) is received, Jenkins will test/build/push and look up the revis...
fleet
Router
Route Updater
Registrator
Jenkins deploys the new container to fleet. It runs side-by-side with the previous
...
fleet
Router
Route Updater
Registrator
Registrator notices the new service is deployed and registers the location in etcd ...
fleet
Router
Route Updater
Registrator
Traffic continues to flow to the old service as the current revision has not change...
fleet
Router
Route Updater
Registrator
Traffic can be directed to a particular version by using a header for testing purpo...
fleet
Router
Route Updater
Registrator
Periodically, Route Updater queries etcd to look for cases where there is a revisio...
fleet
Router
Route Updater
Registrator
If there is a newer revision, route updater will attempt to call the smoketest endp...
fleet
Router
Route Updater
Registrator
Now traffic will start flowing to the new revision of the service automatically.15
...
fleet
Router
Route Updater
Registrator
Route Updater will notice that there is a stale revision running. It will instruct ...
fleet
Router
Route Updater
Registrator
Registrator will notice the container is no longer running and remove its location ...
fleet
Router
Route Updater
Registrator
The system continues as-is until a new revision is deployed.18
service-1:1
service-...
Comprehensive
Service – log4j
Container – logspout
CoreOS – journal forwarder
Bastion/NAT – rsyslog
ELB – S3 (ELK coming s...
Instrumentation & Shipping
• Kamon to Prometheus
Exporter, preserves more
metrics than Prometheus JVM
• Improved tracing
•...
What’s Next
Improvements & Evolution
AWS Service Catalog – API?
EC2 Container Service
AWS IAM
• EC2 CS Roles
• RDS Roles – per VPC/DB ...
Higher-Order Automation
Automated Tests
Continuous Integration
Continuous Delivery
Automated Infrastructure
Automated Faul...
Monsanto IT
Acknowledgements
Larry Anderson
Chris Coffman
TJ Corrigan
Phil Cryer
Dave D’Alessandro
Daniel Solano Gómez
Justin Honold
K...
Thank you!
engineering.monsanto.com
@MonsantoPlatformEng
@ddgenome @ryan_richt
Remember to complete
your evaluations!
Related Sessions
ARC309 - From Monolithic to Microservices: Evolving
Architecture Patterns in the Cloud
Thursday, Oct 8, 4...
http://engineering.monsanto.com/code
@MonsantoPlatformEng
https://github.com/MonsantoCo/cloudformation-template-generator
...
(ARC401) Cloud First: New Architecture for New Infrastructure
(ARC401) Cloud First: New Architecture for New Infrastructure
(ARC401) Cloud First: New Architecture for New Infrastructure
(ARC401) Cloud First: New Architecture for New Infrastructure
Upcoming SlideShare
Loading in …5
×

(ARC401) Cloud First: New Architecture for New Infrastructure

11,778 views

Published on

What do companies with internal platforms have to change to succeed in the cloud? The five pillars at the heart of IT solutions in the cloud are automation, fault tolerance, horizontal scalability, security, and cost-effectiveness. This talk discusses tools that facilitate the development and automate the deployment of secure, highly available microservices. The tools were developed using AWS CloudFormation, AWS SDKs, AWS CLI, Amazon RDS, and various open-source software such as Docker. The talk provides concrete examples of how these tools can help developers and architects move from beginning/intermediate AWS practitioners to cloud deployment experts.

Published in: Technology

(ARC401) Cloud First: New Architecture for New Infrastructure

  1. 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. David Dooling & Ryan Richt October 2015 Cloud First New Architecture for New Infrastructure @ddgenome & @ryan_richt ARC401
  2. 2. What to Expect from the Session Theory of Cloud
  3. 3. Scientists Turned Developers Turned Architects
  4. 4. Ryan David Scientists Turned Developers Turned Architects
  5. 5. Monsanto
  6. 6. Theory of Cloud
  7. 7. Theory of Cloud Automated Elastic Highly Available Security Software defined everything Unlimited scale + pay-as-you-go Horizontally Scalable Multi-AZ/region + shards/replicas Provision more like things any time “Do over” + Correct by construction
  8. 8. Theory of Cloud Cloud Architecture Automated Higher-Order Automation Elastic Ephemeral Environments Highly Available Fault Tolerant Security Secure by Construction Horizontally Scalable Parallel, Commodity ⇒
  9. 9. Higher-Order Automation Automated Tests Continuous Integration Continuous Delivery Automated Infrastructure Automated Fault Detection Automated Recovery …and automated tools to build more automation!
  10. 10. Fallacies of Internal Apps 1. The hardware is reliable 2. The network is reliable 3. The database is reliable 4. Other services are available 5. Inside the network is secure 6. … Fault Tolerant
  11. 11. Fault Tolerant Fallacies of 1st Generation Cloud 1. Other people’s fault tolerant code is actually fault tolerant 2. Everything is stateless 3. Everything can be retried 4. Applications should handle all faults 5. Data is magically handled by someone else
  12. 12. Elastic, Ephemeral, Cost-Effective time cost Cloud On Prem Dynamic Env Replication time cost Cloud On Prem Experiments
  13. 13. A Do-Over for Secure by Construction Secure by Assumption Secure by Design Security Automation
  14. 14. Horizontally Scalable 1. The overhead of scaling grows at most linearly with additional nodes 2. Reads and writes both scale out 3. The system can continue to provide this scalability under loss of any node * This (CAP) requires apps to understand conflicts
  15. 15. Infrastructure Automation
  16. 16. Federation – 1000 VPCs Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VAmazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VPC Amazon VP Amazon Amazon VPC Amazon VPC
  17. 17. Cloud Architecture
  18. 18. Cloud Architecture
  19. 19. Cloud Architecture
  20. 20. Cloud Architecture
  21. 21. Cloud Architecture
  22. 22. AWS CloudFormation "IPAddress" : { "Type" : "AWS::EC2::EIP", "DependsOn" : "AttachGateway", "Properties" : { "Domain" : "vpc", "InstanceId" : { "Ref" : "WebServerInstance" } } }, "InstanceSecurityGroup" : { "Type" : "AWS::EC2::SecurityGroup", "Properties" : { "VpcId" : { "Ref" : "VPC" }, "GroupDescription" : "Enable SSH access via port 22", "SecurityGroupIngress" : [ {"IpProtocol":"tcp","FromPort":"22","ToPort":"22", "CidrIp" : { "Ref" : "SSHLocation"}}, {"IpProtocol":"tcp","FromPort":"80","ToPort":"80", "CidrIp" : "0.0.0.0/0"} ] } }, "WebServerInstance" : { "Type" : "AWS::EC2::Instance", "DependsOn" : "AttachGateway", "Metadata" : { "Comment" : "Install a simple application", …
  23. 23. Cloud Architecture
  24. 24. CloudFormation Template Generator https://github.com/MonsantoCo/cloudformation-template-generator
  25. 25. CloudFormation Template Generator Referential Integrity
  26. 26. Auto Scaling Group
  27. 27. CFTG: Security Groups
  28. 28. Stax $ ./stax --help Usage: stax [OPTIONS] COMMAND [COMMAND_ARGS] add Add functionality to an existing VPC auto-services Lanch multiple services on fleet using template/NAME.services file check Run various tests against an existing stax clean Remove keys and buckets of non-existant stacks connect [TARGET] Connect to bastion|gateway|service in the VPC stax over SSH create Create a new VPC stax in AWS describe Describe the stax created from this host delete Delete the existing VPC stax dockerip-update Fetch docker IP addresses and update related files fleet Run various fleetctl commands against the fleet cluster help Output this message history View history of recently created/deleted stax list List all completely built and running stax rds PASSWORD Create an RDS instance in the DB subnet rds-delete RDSIN Delete RDS instance RDSIN remove ADD Remove the previously added ADD services List servers that are available to run across a stax slack Post usage report to Slack, define hook in stax.config sleep Turn on/off bastion host which allows ssh access into the VPC start SERVICE Start service SERVICE in the fleet cluster test Automated test to exercise functionality of stax update Update an existing VPC with changes from Cloudformation validate Validate CloudFormation template For more help, check the docs: https://github.com/MonsantoCo/stax Create and manage CloudFormation stacks in AWS
  29. 29. $ ./stax create [ ---- ] creating stax [ NAME ] vpc-stax-36918-outfitting [ ---- ] creating parameter file [ ---- ] checking for valid json file format [ ---- ] creating ssh key pair in aws [ ---- ] creating key pair [ ---- ] create bucket [ ---- ] creating bucket vpc-stax-36918-outfitting [ ---- ] uploading template [ ---- ] validate template [ ---- ] validating template https://s3.amazonaws.com/… [ ---- ] uploading vpc assets [ ---- ] creating stax in aws [ ---- ] stax creation complete [ ---- ] querying aws [ ---- ] query complete [ ---- ] see run/vpc-stax-36918-outfitting.json for details
  30. 30. $ ./stax connect [ ---- ] checking if stax build is complete [ ---- ] describe stax [ NAME ] vpc-stax-36918-outfitting [ ---- ] querying aws [ ---- ] query complete [ ---- ] see run/vpc-stax-36918-outfitting.json for details [ ---- ] stack vpc-stax-36918-outfitting build complete [ ---- ] connecting to stax: bastion __| __|_ ) _| ( / Amazon Linux AMI ___|___|___| https://aws.amazon.com/amazon-linux-ami/2014.09-release-notes/ [ec2-user@ip-10-183-1-195 ~]$
  31. 31. Stax as a Service - Create
  32. 32. Stax as a Service – List
  33. 33. Stax as a Service – Describe
  34. 34. Stax as a Service – Services
  35. 35. Monsanto
  36. 36. Microservices Lifecycle
  37. 37. Microservices: Cupcakes, Not Wedding Cakes
  38. 38. A modern language for software engineering Abstract Data Types (ADTs) Enforced Immutability Pattern Matching & Destructuring Assignment Type-Level Programming Futures, Actors, Async Type classes Scala, Haskell, Swift, OCaML, SML Scala, Haskell, Clojure, Erlang, OCaML, SML CoffeeScript, Scala, Haskell, Swift, OCaML, Erlang, SML Haskell, Scala, C++ Erlang, Scala, Java Haskell, Scala, ~OCaML Hybrid OO/FP Provides transition from and backward compatibility with Java
  39. 39. Advanced Abstractions Algebraic Data Types (ADTs) Enforced Immutability Pattern Matching & Destructuring Assignment Type-Level Programming Futures, Actors, Async Type classes Scala: A Modern Language for Software Engineering Advanced Type Constraints Advanced Generics & Variance Higher Kinds F-bounded Polymorphism Self-Types Type Projections Type Members Path Dependent Types Type Refinements Turing-complete!
  40. 40. Project-as-a-Service 1 – Create Code Repo/Wiki/Issues
  41. 41. Project-as-a-Service 2 – Simple Service Template Runs giter8 to create a fully functional service written in Scala based off our current best practices: • Standard libraries (Slick, Spray, Akka, etc.) for microservices • Automated tests with ScalaTest • Administrative REST endpoints • Built in (remote) logging and metric capabilities • Auto-Docker-ization • Local Vagrant environment
  42. 42. Project-as-a-Service 3 – CI & Dockerization New check-in Test and Build Dockerize
  43. 43. Project-as-a-Service 4 – Continuous Deployment
  44. 44. fleet Router Route Updater Registrator A commit is made to GitHub1 1 https://github.com/MonsantoCo/etcd-aws-cluster https://github.com/MonsantoCo/docker-aws https://github.com/MonsantoCo/fleet-client
  45. 45. fleet Router Route Updater Registrator GitHub notifies Jenkins that new code is available. Jenkins runs automated tests to validate that code is functional. 2 2
  46. 46. fleet Router Route Updater Registrator Jenkins builds a Docker container and pushes it to our private Docker registry.3 3 service-1:1
  47. 47. fleet Router Route Updater Registrator Jenkins registers the service with etcd, our key/value store, since it doesn’t exist.4 4 service-1:1 name version revision service-1 => 1
  48. 48. fleet Router Route Updater Registrator Jenkins calls fleet to deploy the container running our service.5 5 service-1:1 service-1 => 1 service v1 rev1 10.183.0.100:8080
  49. 49. fleet Router Route Updater Registrator Registrator notices the service is deployed and registers the location in etcd.6 6 service-1:1 service-1 => 1 service-1-1 => [10.183.0.100:8080] service v1 rev1 10.183.0.100:8080
  50. 50. fleet Router Route Updater Registrator When a request is received, the router determines the current revision for the service as well as the location of the service. 7 7 service-1:1 service-1 => 1 service-1-1 => [10.183.0.100:8080] service v1 rev1 10.183.0.100:8080
  51. 51. fleet Router Route Updater Registrator Next commit (rev 2) is received, Jenkins will test/build/push and look up the revision from etcd. The revision is newer so it continues but does not update the current revision. 8 8 service-1:1 service-1 => 1 service-1-1 => [10.183.0.100:8080] service v1 rev1 service-1:2 10.183.0.100:8080
  52. 52. fleet Router Route Updater Registrator Jenkins deploys the new container to fleet. It runs side-by-side with the previous revision at a different location. 9 9 service-1:1 service-1 => 1 service-1-1 => [10.183.0.100:8080] service v1 rev1 service-1:2 service v1 rev2 10.183.0.100:8081 10.183.0.100:8080
  53. 53. fleet Router Route Updater Registrator Registrator notices the new service is deployed and registers the location in etcd under a different key. 10 10 service-1:1 service-1 => 1 service-1-1 => [10.183.0.100:8080] service-1-2 => [10.183.0.100:8081]service v1 rev1 service-1:2 service v1 rev2 10.183.0.100:8081 10.183.0.100:8080
  54. 54. fleet Router Route Updater Registrator Traffic continues to flow to the old service as the current revision has not changed.11 11 service-1:1 service-1 => 1 service-1-1 => [10.183.0.100:8080] service-1-2 => [10.183.0.100:8081]service v1 rev1 service-1:2 service v1 rev2 10.183.0.100:8081 10.183.0.100:8080
  55. 55. fleet Router Route Updater Registrator Traffic can be directed to a particular version by using a header for testing purposes.12 12 service-1:1 service-1 => 1 service-1-1 => [10.183.0.100:8080] service-1-2 => [10.183.0.100:8081]service v1 rev1 service-1:2 service v1 rev2 X-Service-Revision: 2 10.183.0.100:8081 10.183.0.100:8080
  56. 56. fleet Router Route Updater Registrator Periodically, Route Updater queries etcd to look for cases where there is a revision deployed that is newer than the current route. 13 service-1:1 service-1 => 1 service-1-1 => [10.183.0.100:8080] service-1-2 => [10.183.0.100:8081]service v1 rev1 service-1:2 service v1 rev2 13 10.183.0.100:8081 10.183.0.100:8080
  57. 57. fleet Router Route Updater Registrator If there is a newer revision, route updater will attempt to call the smoketest endpoint. If this returns true, it updates the current route. 14 service-1:1 service-1 => 2 service-1-1 => [10.183.0.100:8080] service-1-2 => [10.183.0.100:8081]service v1 rev1 service-1:2 service v1 rev2 14 /admin/smoketest 10.183.0.100:8081 10.183.0.100:8080
  58. 58. fleet Router Route Updater Registrator Now traffic will start flowing to the new revision of the service automatically.15 service-1:1 service-1 => 2 service-1-1 => [10.183.0.100:8080] service-1-2 => [10.183.0.100:8081]service v1 rev1 service-1:2 service v1 rev2 15 10.183.0.100:8081 10.183.0.100:8080
  59. 59. fleet Router Route Updater Registrator Route Updater will notice that there is a stale revision running. It will instruct the service to cleanly exit by making a call to the /admin/shutdown endpoint. 16 service-1:1 service-1 => 2 service-1-1 => [10.183.0.100:8080] service-1-2 => [10.183.0.100:8081]service v1 rev1 service-1:2 service v1 rev2 16 /admin/shutdown 10.183.0.100:8081 10.183.0.100:8080
  60. 60. fleet Router Route Updater Registrator Registrator will notice the container is no longer running and remove its location from etcd. 17 service-1:1 service-1 => 2 service-1-1 => [10.183.0.100:8080] service-1-2 => [10.183.0.100:8081] service-1:2 service v1 rev2 17 10.183.0.100:8081
  61. 61. fleet Router Route Updater Registrator The system continues as-is until a new revision is deployed.18 service-1:1 service-1 => 2 service-1-2 => [10.183.0.100:8081] service-1:2 service v1 rev2 10.183.0.100:8081
  62. 62. Comprehensive Service – log4j Container – logspout CoreOS – journal forwarder Bastion/NAT – rsyslog ELB – S3 (ELK coming soon) S3 – S3 (ELK coming soon) CloudTrail – S3 → TrailDash RDS – (coming soon) Logging with ScalaLogging and ELK Easy to use • Standard ScalaLogging interface • Auto custom formats (stack traces) • JSON-format log messages • Direct-to-ELK writing • Standard Fields (container ID, code version, service name, etc)
  63. 63. Instrumentation & Shipping • Kamon to Prometheus Exporter, preserves more metrics than Prometheus JVM • Improved tracing • Improved complex data mapping • Periodically collect and push Spray metrics to Kamon Automating Kamon and Prometheus Auto-discovery, Dashboards, Alerts • Custom Docker containers with more automation – etcd discovery • Custom default dashboards • Auto EC2/EBS/RDS standup • OAuth integration • SNS notification integration • Default Alerts https://github.com/MonsantoCo/spray-kamon-metrics
  64. 64. What’s Next
  65. 65. Improvements & Evolution AWS Service Catalog – API? EC2 Container Service AWS IAM • EC2 CS Roles • RDS Roles – per VPC/DB Subnet Groups Amazon API Gateway VPC Flow Logs – CloudFormation support? Inverting control for deployment CloudFormation update predictability IAM role Amazon RDS Amazon EC2 Container Service
  66. 66. Higher-Order Automation Automated Tests Continuous Integration Continuous Delivery Automated Infrastructure Automated Fault Detection Automated Recovery …and automated tools to build more automation!
  67. 67. Monsanto IT
  68. 68. Acknowledgements Larry Anderson Chris Coffman TJ Corrigan Phil Cryer Dave D’Alessandro Daniel Solano Gómez Justin Honold Kyle Jones Jessica Kerr Kevin Meredith Jorge Montero Brian Rodgers Chris Shafer Niranjan Vengavasi Dick Wall Russ Wilson Stuart Wong
  69. 69. Thank you! engineering.monsanto.com @MonsantoPlatformEng @ddgenome @ryan_richt
  70. 70. Remember to complete your evaluations!
  71. 71. Related Sessions ARC309 - From Monolithic to Microservices: Evolving Architecture Patterns in the Cloud Thursday, Oct 8, 4:15 PM - 5:15 PM – Palazzo N MBL203 - From Drones to Cars: Connecting the Devices in Motion to the Cloud Friday, Oct 9, 10:15 AM - 11:15 AM – Delfino 4005
  72. 72. http://engineering.monsanto.com/code @MonsantoPlatformEng https://github.com/MonsantoCo/cloudformation-template-generator https://github.com/MonsantoCo/docker-aws https://github.com/MonsantoCo/etcd-aws-cluster https://github.com/MonsantoCo/fleet-client https://github.com/MonsantoCo/spray-kamon-metrics https://github.com/MonsantoCo/stax More to come… @ddgenome @ryan_richt

×