Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building a Cloud Native Service - Docker Meetup Santa Clara (July 20, 2017)

In this talk, we share our experience of building up a cloud native service with Docker, Kubernetes, and CoreDNS. It is a customer-facing, multi-tenant, and globally available service that helps customers defending against various Internet attacks.

The global availability of the service is achieved through Anycast so that all customers only need to access one IP address across different regions. Deploying Anycast turns out to be a challenge because of the limitations on certain clouds. We overcome those limitations through containerization of different components with Docker.

We also share our experiences in container orchestration, container networking, load balancing, and service registration & discovery. We use a simplified architecture for container networking, and the service registration & discovery is done through CoreDNS. The overall design have helped our deployed service with improved elasticity, ease of use, and lowered maintenance cost.

  • Login to see the comments

Building a Cloud Native Service - Docker Meetup Santa Clara (July 20, 2017)

  1. 1. Cloud Native Security Service with Docker Yong Tang Infoblox Inc. Maintainer, Docker/SwarmKit/CoreDNS GitHub: yongtang
  2. 2. Overview • ActiveTrust Cloud in Infoblox • Deployed on a Common Platform (Next Gen Platform) • Simple L3 Container Networking and Auto Scaling Group • Service Registration & Discovery • Demo (https://github.com/yongtang/kubernetes-cfn) • Anycast Support, CoreDNS 07/20/2017 1
  3. 3. ActiveTrust Cloud in Infoblox 07/20/2017 2 Block Malicious Communications Extends Protection to Roaming Clients and Branch Offices Helps with Prioritization of Threats Get a Deep Understanding around Infected Devices Detect and Contain Malware using DNS Prevent DNS Based Data Exfiltration That Other Systems Can’t Detect Improved Visibility and Context Detect and Contain Malware using DNS Prevent DNS Based Data Exfiltration That Other Systems Can’t Detect Improved Visibility and Context
  4. 4. Common Platform (Next Gen Platform) 07/20/2017 3 Service Delivery Platform (SDP) • Customer Identity • Service Catalog • Licensing • Subscriptions • Provisioning • Service Assurance • Cost Tracking • Problem & Incident • Metering & Billing IaaS PaaS (Next Gen Platform) SaaS ActiveTrust Threat Feed Threat Insight Analytics ActiveTrust Cloud AWS Infrastructure AWS Direct Connect AWS Services (S3, EMR, etc.) CONTROL PLANE APPLICATION INFRASTRUCTURE (Persistence, Messaging, etc.) Containerized Microservices Architecture in Infoblox
  5. 5. Current Next Gen Platform 07/20/2017 4 Zookeeper/Mesos/Marathon/Consul Container Networking Container Runtime Container Orchestration Service R & D Policy Management API CONTROL PLANE APPLICATION INFRASTRUCTURE SUPPORT Anycast, IPVS & ELB Integration Hook MESSAGING Data Processing Pipeline Logging Tracing Metrics API Auth PERSISTENCE ANALITICS
  6. 6. Future Next Gen Platform 07/20/2017 5 Kubernetes/CoreDNS Container Networking Container Runtime Container Orchestration Service R & D Policy Management API CONTROL PLANE APPLICATION INFRASTRUCTURE SUPPORT Anycast, IPVS & ELB Integration Hook MESSAGING PERSISTENCE ANALITICS Data Processing Pipeline Logging Tracing Metrics API Auth
  7. 7. Networking vs Service Registration/Discovery • Container Networking • Container lifetime, flexibility vs. stability (network topology) • Necessity of fixed service IP, or dynamic service IP • Service Registration & Discovery • Fixed network topology and dynamic service IP could be combined with service registration for a better solution in a containerized microservices architecture • Next Gen Platform, achieved the goal through: • Simple L3 networking with route management on AWS • Simple address spacing naturally fitted with docker0 (/24 network on host) • Implemented a better service registration through background process 07/20/2017 6
  8. 8. Container Networking with Auto Scaling 07/20/2017 7 Auto Scaling Group 10.2.1.2 10.2.1.3 10.2.1.4 docker0 10.2.1.1/24 eth0 172.28.1.100/24 10.2.2.2 10.2.2.3 10.2.2.4 docker0 10.2.2.1/24 eth0 172.28.1.200/24 10.2.3.2 10.2.3.3 10.2.3.4 docker0 10.2.3.1/24 eth0 172.28.1.300/24 i-02fcc38b9ced4ec91 i-05d3bc68d5d647b33 i-0c72256b7f55086e3 host mode host mode host mode host mode host mode host mode Destination Target 10.2.1.0/24 i-02fcc38b9ced4ec91 10.2.2.0/24 i-05d3bc68d5647b33 10.2.3.0/24 i-0c72256b7f55086e3
  9. 9. Container Networking with Auto Scaling 07/20/2017 8 Auto Scaling Group 10.2.1.2 10.2.1.3 10.2.1.4 docker0 10.2.1.1/24 eth0 172.28.1.100/24 10.2.2.2 10.2.2.3 10.2.2.4 docker0 10.2.2.1/24 eth0 172.28.1.200/24 10.2.3.2 10.2.3.3 10.2.3.4 docker0 10.2.3.1/24 eth0 172.28.1.300/24 i-02fcc38b9ced4ec91 i-05d3bc68d5d647b33 i-0c72256b7f55086e3 host mode host mode host mode host mode host mode host mode Destination Target Elastic Network Interface (eni) Lock 10.2.1.0/24 i-02fcc38b9ced4ec91
  10. 10. Container Networking with Auto Scaling 07/20/2017 9 Auto Scaling Group 10.2.1.2 10.2.1.3 10.2.1.4 docker0 10.2.1.1/24 eth0 172.28.1.100/24 10.2.2.2 10.2.2.3 10.2.2.4 docker0 10.2.2.1/24 eth0 172.28.1.200/24 10.2.3.2 10.2.3.3 10.2.3.4 docker0 10.2.3.1/24 eth0 172.28.1.300/24 i-02fcc38b9ced4ec91 i-05d3bc68d5d647b33 i-0c72256b7f55086e3 host mode host mode host mode host mode host mode host mode Destination Target 10.2.1.0/24 i-02fcc38b9ced4ec91 Elastic Network Interface (eni) Lock
  11. 11. Container Networking with Auto Scaling 07/20/2017 10 Auto Scaling Group 10.2.1.2 10.2.1.3 10.2.1.4 docker0 10.2.1.1/24 eth0 172.28.1.100/24 10.2.2.2 10.2.2.3 10.2.2.4 docker0 10.2.2.1/24 eth0 172.28.1.200/24 10.2.3.2 10.2.3.3 10.2.3.4 docker0 10.2.3.1/24 eth0 172.28.1.300/24 i-02fcc38b9ced4ec91 i-05d3bc68d5d647b33 i-0c72256b7f55086e3 host mode host mode host mode host mode host mode host mode Destination Target Elastic Network Interface (eni) Lock 10.2.1.0/24 i-02fcc38b9ced4ec91 10.2.2.0/24 i-05d3bc68d5647b33
  12. 12. Container Networking with Auto Scaling 07/20/2017 11 Auto Scaling Group 10.2.1.2 10.2.1.3 10.2.1.4 docker0 10.2.1.1/24 eth0 172.28.1.100/24 10.2.2.2 10.2.2.3 10.2.2.4 docker0 10.2.2.1/24 eth0 172.28.1.200/24 10.2.3.2 10.2.3.3 10.2.3.4 docker0 10.2.3.1/24 eth0 172.28.1.300/24 i-02fcc38b9ced4ec91 i-05d3bc68d5d647b33 i-0c72256b7f55086e3 host mode host mode host mode host mode host mode host mode Destination Target 10.2.1.0/24 i-02fcc38b9ced4ec91 Elastic Network Interface (eni) Lock 10.2.2.0/24 i-05d3bc68d5647b33
  13. 13. Container Networking with Auto Scaling 07/20/2017 12 Auto Scaling Group 10.2.1.2 10.2.1.3 10.2.1.4 docker0 10.2.1.1/24 eth0 172.28.1.100/24 10.2.2.2 10.2.2.3 10.2.2.4 docker0 10.2.2.1/24 eth0 172.28.1.200/24 10.2.3.2 10.2.3.3 10.2.3.4 docker0 10.2.3.1/24 eth0 172.28.1.300/24 i-02fcc38b9ced4ec91 i-05d3bc68d5d647b33 i-0c72256b7f55086e3 host mode host mode host mode host mode host mode host mode Destination Target Elastic Network Interface (eni) Lock 10.2.1.0/24 i-02fcc38b9ced4ec91 10.2.2.0/24 i-05d3bc68d5647b33 10.2.3.0/24 i-0c72256b7f55086e3
  14. 14. Container Networking with Auto Scaling 07/20/2017 13 Auto Scaling Group 10.2.1.2 10.2.1.3 10.2.1.4 docker0 10.2.1.1/24 eth0 172.28.1.100/24 10.2.2.2 10.2.2.3 10.2.2.4 docker0 10.2.2.1/24 eth0 172.28.1.200/24 10.2.3.2 10.2.3.3 10.2.3.4 docker0 10.2.3.1/24 eth0 172.28.1.300/24 i-02fcc38b9ced4ec91 i-05d3bc68d5d647b33 i-0c72256b7f55086e3 host mode host mode host mode host mode host mode host mode Destination Target 10.2.1.0/24 i-02fcc38b9ced4ec91 10.2.2.0/24 i-05d3bc68d5647b33 10.2.3.0/24 i-0c72256b7f55086e3 Elastic Network Interface (eni) Lock
  15. 15. Service Registration & Discovery • Implemented a background process, listening to Docker events • Sync metadata from Docker Engine • Sync metadata from orchestration (mesos/k8s) • Updates triggered by listened Docker events 07/20/2017 14 background progress
  16. 16. Misc. & Management • Disable Source Destination Check of EC2 instances • Change FORWARD chain policy to ACCEPT (since docker 1.13) • Source NAT (SNAT) rule to allow connection to outside • Managed by CloudFormation template with CloudInit (#cloud-config) • Amazon Linux (AMI) for better performance/driver support (on AWS) • Instance Profile (IAM) for better security • Integrate Elastic Load Balancing with Auto Scaling/Kubernetes • Limit on the number of routes per route table (50), addressable by: • Pre-populate the routes to individual hosts with `ip route` in Linux • Use ENIs for host-level IP pool, available for attachments by instances. 07/20/2017 15
  17. 17. Demo • Kubernetes with Auto Scaling Group • No external storage or server setup • Managed by CloudFormation • Available: https://github.com/yongtang/kubernetes-cfn 07/20/2017 16
  18. 18. ActiveTrust Cloud Anycast (VA Region) 07/20/2017 17 Equinix Data Center (VA) US-East-1 (VA) application container Containerized White Box Router (WBR) application container application container application container application container application container application container application container application container IPVS container (host mode) IPVS container (host mode) ECMP Equal-cost multi-path routing
  19. 19. ActiveTrust Cloud Anycast Network 07/20/2017 18 Internet ActiveTrust Cloud Anycast Deployment in Different AWS Regions Replace vendor routers with White Box Routers running on NOA VGW Router ATC in AP Northeast Tokyo Equinix Data Center VGW Router ATC in EU West London Equinix Data Center VGW Router ATC in US West CA Equinix Data Center VGW Router ATC in US East CA Equinix Data Center AWS VPC AWS Virtual Gateway Equinix Data Center Router- Publish 52.119.40.0/24 Anycast route to Internet via BGP ATC running in AWS EC2 Instances Advertise 52.119.40.100/32 to Router via BGP AWS Direct Connect Internet uplink via HE Internet uplink via GTT
  20. 20. CoreDNS and CNCF • https://github.com/coredns/coredns • Cloud native, authoritative DNS server written in Go • Not a recursive DNS server (yet...?) • Successor to SkyDNS2 for dynamic DNS-based service discovery • Flexible, middleware-based, extensible request pipeline • Started and led by Miek Gieben • Sponsored by Infoblox and soon to be used in its SaaS offerings • Hosted as an inception level project at CNCF 07/20/2017 19
  21. 21. CoreDNS and CNCF 07/20/2017 20
  22. 22. THANK YOU 07/20/2017 21

×