Downloaded 64 times
![@SpringBootApplication
@EnableAutoConfiguration
@EnableZuulProxy
public class EdgeGateway {
public static void main(String[] args) throws Exception {
SpringApplication.run(EdgeGateway.class, args);
}
}
Getting started with Netflix Zuul is easy
Edge gateway](https://image.slidesharecdn.com/elasticon-final-170329075333/85/Microservices-Continuous-Delivery-and-Elasticsearch-at-Capital-One-18-320.jpg)
![@SpringBootApplication
@EnableDiscoveryClient
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}
Making Spring Boot app discoverable with Eureka
Discover service](https://image.slidesharecdn.com/elasticon-final-170329075333/85/Microservices-Continuous-Delivery-and-Elasticsearch-at-Capital-One-26-320.jpg)
![Hardware playbook example
roles:
- role: servers
instances:
- name: Elasticsearch_Master
instance_type: m4.2xlarge
number_of_instances: 3
- name: Elasticsearch_Data
instance_type: m4.4xlarge
number_of_instances: 100
additional_volume_sizes: [1000, 1000, 1000]](https://image.slidesharecdn.com/elasticon-final-170329075333/85/Microservices-Continuous-Delivery-and-Elasticsearch-at-Capital-One-46-320.jpg)
![- hosts: tag_{{ ansible_ec2_tag }}_Elasticsearch_Data
become: true
roles:
- role: elasticsearch
es_heap_size: '{{ [(ansible_memtotal_mb / 1024) / 2, 16] | min | int }}g'
es_plugins:
- '{{ es_plugin_license }}'
- '{{ es_plugin_marvel_agent }}'
- '{{ es_plugin_cloud_aws }}'
es_config:
cluster.name: '{{ elasticsearch_cluster_name }}'
node.name: '{{ ansible_default_ipv4.address }}'
node.master: false
node.data: true
indices.fielddata.cache.size: 10%
indices.breaker.fielddata.limit: 15%
indices.breaker.request.limit: 15%
indices.breaker.total.limit: 30%
network.breaker.inflight_requests.limit: 75%
Software playbook example](https://image.slidesharecdn.com/elasticon-final-170329075333/85/Microservices-Continuous-Delivery-and-Elasticsearch-at-Capital-One-47-320.jpg)
Uploaded byNoriaki Tatsumi
Microservices, Continuous Delivery, and Elasticsearch at Capital One
Capital One's document outlines their transition to microservices and continuous delivery, utilizing Elasticsearch for data management and processing. They discuss the challenges faced with security, compliance, and operational efficiencies, and how microservices have improved deployment speed and system manageability. Key takeaways emphasize the importance of a robust monitoring system and the integration of Elastic Stack for performance optimization.
More Related Content
Similar to Microservices, Continuous Delivery, and Elasticsearch at Capital One
Microservices, Continuous Delivery, and Elasticsearch at Capital One
- 1. Capital One 3/8/2017 Microservices, ContinuousDelivery, and Elasticsearch at Capital One Noriaki (Nori) Tatsumi, Bingchen (Ben) Hu, Anne Cather
- 3. Security breaches dominatethe news
- 4.
- 5. Build vs. buy •Industry tools only meet ~80% of our requirements • Vendors’ priorities don’t align with ours • Elasticsearch is an open source solution • Open source technology is extensible
- 6. 30+ data sources,3B events and 6TB data/day
- 8. How we gothere
- 9. Scale New featuresNFRs • More data • More processing • Longer data retention • More consumers • Alerts console • Cyber threat intelligence repository • And more! Our initial requirements • Uptime and DR • Security • Compliance • Data management
- 10. The prototype wehad Elasticsearch Data Nodes Elasticsearch Master Nodes Elasticsearch Client NodeKibana Fork w/ SSO Integration AD SSO
- 11. MORE REQUIREMENTS, DELIVERY DATES, BIGGERTEAMS = HIGHER COMPLEXITY
- 12.
- 13. • Work inparallel • Do one scope of things well • Easy to understand and maintain • Technology stack choice for features and teams • Quicker, smaller, & independent deploys • Fault isolation What we wanted
- 14.
- 15. No SSO Integration! Embracingmicroservices Elasticsearch Data Nodes Elasticsearch Master Nodes Elasticsearch Client NodeKibana Fork w/ SSO Integration AD SSO Alerts-API Alerts-UI CTI Repo
- 16. • A wellknown entry point to the system • Security • Dynamic routing • Resiliency • Latency and fault tolerance • Monitoring and stats collection Edge gateway Align same qualities to downstream services
- 17. • Spring Bootfor developer productivity • JVM-based for production supportability • Netflix OSS that’s proven microservices technology Spring Cloud Foundation for our web microservices
- 18. @SpringBootApplication @EnableAutoConfiguration @EnableZuulProxy public class EdgeGateway{ public static void main(String[] args) throws Exception { SpringApplication.run(EdgeGateway.class, args); } } Getting started with Netflix Zuul is easy Edge gateway
- 19.
- 20. Elasticsearch Client NodeKibana Elasticsearch Client NodeKibana Zuul:the edge gateway Elasticsearch Data Nodes Elasticsearch Master Nodes Edge Gateway Elasticsearch Client NodeKibana AD SSO Alerts API Alerts UI Reports UI CyberTech Reports Repo Auth
- 21. Asking engineers tomaintain IP addresses
- 22. • Use cases •Service connection information lookup • Automated configuration of load balancing and failover • Alternatives to Eureka with Spring Cloud • HashiCorp Consul • Apache Zookeeper Discover service Automate orchestration with Netflix Eureka
- 23. <application> <name>...</name> <instance> <instanceId>... </instanceId> <hostName>... </hostName> <app>...</app> <ipAddr>...</ipAddr> <status>UP</status> <overriddenstatus>UNKNOWN</overriddenstatus> <portenabled="false">...</port> <securePort enabled="true">...</securePort> <countryId>1</countryId> <dataCenterInfo class="com.netflix.appinfo.AmazonInfo"> <name>Amazon</name> <metadata> <accountId>...</accountId> <local-hostname>... </local-hostname> <instance-id>...</instance-id> <local-ipv4>...</local-ipv4> <instance-type>...</instance-type> <vpc-id>...</vpc-id> <ami-id>...</ami-id> <mac>...</mac> <availability-zone>...</availability-zone> </metadata> </dataCenterInfo> <leaseInfo> <renewalIntervalInSecs>...</renewalIntervalInSecs> <durationInSecs>...</durationInSecs> …..
- 24.
- 25. zuul.routes.kibana.path=/kibana/** zuul.routes.kibana.serviceId=kibana Routing with Zuulwith Eureka Discover service
- 26. @SpringBootApplication @EnableDiscoveryClient public class Application{ public static void main(String[] args) { SpringApplication.run(Application.class, args); } } Making Spring Boot app discoverable with Eureka Discover service
- 27. • Eureka Client(Java) • Eureka-js-client (JavaScript) • Eureka REST API (Polyglot) • *Sidecar/App gateway (Polyglot) Discover service Making any app discoverable with Eureka
- 28. Solving the configurationnightmare Elasticsearch Data Nodes Elasticsearch Master Nodes Edge Gateway AD SSO Kibana Gateway Elasticsearch Client Node Kibana Kibana Gateway Elasticsearch Client Node Kibana Kibana Gateway Elasticsearch Client Node Kibana Eureka Discovery Service /kibana Alerts-UI CyberTech Reports UI Alerts-API CyberTech Reports API
- 29. Multi-config Kibanas Elasticsearch Data Nodes Elasticsearch MasterNodes Edge Gateway AD SSO Kibana Gateway Elasticsearch Client Node Kibana Kibana Gateway Elasticsearch Client Node Kibana Kibana Gateway Elasticsearch Client NodeKibana (Console Off) Kibana Gateway Elasticsearch Client NodeKibana (Console On) Authorization Service /kibana /kibana-admin
- 30. Protected Elasticsearch gate Elasticsearch DataNodes Elasticsearch Master Nodes Edge Gateway AD SSO Kibana Elasticsearch Client Node Elasticsearch Gateway Kibana Gateway Kibana Elasticsearch Client Node Elasticsearch Gateway Kibana Gateway Kibana (Console OFF) Elasticsearch Client Node Elasticsearch Gateway Kibana Gateway Kibana (Console ON) Elasticsearch Client Node Elasticsearch Gateway Kibana-Admin Gateway Authorization Service /kibana-admin /kibana /esclient
- 31. Spring Boot Adminfor Spring Cloud microservices https://github.com/codecentric/spring-boot-admin
- 32. Distributed tracing withSpring Cloud Sleuth https://cloud.spring.io/spring-cloud-sleuth/
- 33. Distributed tracing withSpring Cloud Sleuth
- 34. Distributed tracing withSpring Cloud Sleuth
- 35. • Successes • Shortcircuited • Thread timeouts • Thread-pool rejections • Failures/exceptions • Error percentage (Rolling 10 second counters) Circuit breaker monitoring
- 36. Crushed it! Elasticsearch Kibana Product deliveredand released on time
- 37.
- 38.
- 39. Cluster on fire! •Stability issues from end user queries • Data ingestion latency problems • Insufficient monitoring
- 40. Finding the causes •Inconsistent OS, JVM, and Elasticsearch configurations across cluster • No circuit breakers • Elasticsearch index templates were missing • Shards improperly sized • Incorrect field mappings • Improper cluster sizing
- 41.
- 42.
- 43.
- 44.
- 45. Hardware Playbook • Spin upAWS infrastructure • Tag for purpose • Configure subnet, security group, VPC, etc. Software Playbook • Install common dependencies • AWS tags determine software • Deploy latest artifacts per environment Ansible deployment breakdown
- 46. Hardware playbook example roles: -role: servers instances: - name: Elasticsearch_Master instance_type: m4.2xlarge number_of_instances: 3 - name: Elasticsearch_Data instance_type: m4.4xlarge number_of_instances: 100 additional_volume_sizes: [1000, 1000, 1000]
- 47. - hosts: tag_{{ansible_ec2_tag }}_Elasticsearch_Data become: true roles: - role: elasticsearch es_heap_size: '{{ [(ansible_memtotal_mb / 1024) / 2, 16] | min | int }}g' es_plugins: - '{{ es_plugin_license }}' - '{{ es_plugin_marvel_agent }}' - '{{ es_plugin_cloud_aws }}' es_config: cluster.name: '{{ elasticsearch_cluster_name }}' node.name: '{{ ansible_default_ipv4.address }}' node.master: false node.data: true indices.fielddata.cache.size: 10% indices.breaker.fielddata.limit: 15% indices.breaker.request.limit: 15% indices.breaker.total.limit: 30% network.breaker.inflight_requests.limit: 75% Software playbook example
- 48. ./hardware-playbook.yml --extra-vars @dev-vars.yml ./software-playbook.yml--extra-vars @dev-vars.yml How to use
- 49. Monitor everything! Don’t runa black box
- 50. • Cloud metrics •Server metrics • JVM metrics (even built our own JVM agent) • Application metrics • … What we should monitor
- 52. Time-series dashboards withGrafana
- 53.
- 54. Metrics cluster integration Elasticsearch CyberLakeNodes Edge Gateway AD SSO Kibana Elasticsearch Client Node Elasticsearch Gateway Kibana Gateway Kibana Elasticsearch Client Node Elasticsearch Gateway Kibana Gateway Kibana Elasticsearch Client Node Elasticsearch Gateway Kibana Gateway Kibana Elasticsearch Gateway Kibana-Metrics Gateway Elasticsearch Client Node /metrics /kibana /esclient Elasticsearch Metrics Cluster Eureka Discovery Service ES query data ES query data Service Availability Data Service Availability Data
- 55.
- 56.
- 57. • Microservices architectureworks for us • Increase velocity and reduce maintenance effort • Elastic stack can integrate easily • Continuous Delivery must be a requirement • Monitor everything! Takeaways
- 58.
- 59. More Questions? Visit usat the AMA