Haufe #msaday - Building a Microservice Ecosystem by Daniel Bryant

267 views

Published on

Daniel Bryant's slides from his talk at Haufe Microservices Architecture Day 2016.
Microservice platforms are finally becoming a reality: Mesos, Kubernetes, and a whole bunch of PaaS-style offerings are available, but the reality is that these platforms still don’t provide everything you need in order to build a fully functional microservice ecosystem. Come to this session to learn about the essential deployment, orchestration, and glue components that often have to be self-assembled. The presentation begins by looking at deployment techniques and tools and examines where to test (QA, staging, or production), how to test (integration and contracts), and how to separate deployment and release. It then discusses orchestration, configuration, and service discovery. Finally it looks at essential glue such as logging, monitoring, and alerting.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
267
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
15
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Haufe #msaday - Building a Microservice Ecosystem by Daniel Bryant

  1. 1. Building a microservice ecosystem Daniel Bryant @danielbryantuk OpencRedo
  2. 2. Today • What is a ‘microservice ecosystem’? • Local dev of services becomes challenging • Think about your build pipeline(s) • Testing requires a paradigm shift • Push to production as early as possible 24/11/2016 @danielbryantuk
  3. 3. @danielbryantuk • Chief Scientist at OpenCredo, CTO at SpectoLabs ü Transforming organisations through technology and teams ü Agile, Lean, Architecture, CI/CD, DevOps ü Microservices, cloud, Containers, Java, Go, Docker, Kubernetes • London Java Community Associate • Adopt OpenJDK and JSR • InfoQ Editor, DZone MVB, VOXXED, O'Reilly 24/11/2016 @danielbryantuk
  4. 4. My Biggest Claim to Fame in Microservices? 24/11/2016 @danielbryantuk
  5. 5. Seriously Though...… 24/11/2016 @danielbryantuk skillsmatter.com/skillscasts/7004-our-journey-to-world-gifting- domination-how-notonthehighstreet-com-embraced-docker
  6. 6. So, What is a microservice? “Loosely coupled service oriented architecture with bounded contexts” Adrian Cockcroft “Applications that fit in your head” James Lewis 24/11/2016 @danielbryantuk
  7. 7. Why? The move towards Cloud-native • Hypothesis-driven business (and development) • Microservice architecture (and 'Micro' teams) • Devops mindset/culture (shared, learning/feedback & Mechanical sympathy) • Continuous delivery • Automated, self-service platform Hat tip @caseywest 24/11/2016 @danielbryantuk
  8. 8. Exploring the Ecosystem 24/11/2016 @danielbryantuk
  9. 9. Adrian Cockcroft’s Thoughts 24/11/2016 @danielbryantuk www.slideshare.net/adriancockcroft/microxchg-microservices
  10. 10. Alexis Richardson’s Thoughts 24/11/2016 @danielbryantuk gotocon.com/goto-london-2015/#!#schedulePopupExtras-7011
  11. 11. wikibon.com/wp-content/uploads/container_implementations.png Technology Choices 24/11/2016 @danielbryantuk
  12. 12. 24/11/2016
  13. 13. What do I mean by ‘ecosystem’? • Build – Local development, pipelines and integration • Test – From local integration to End-to-end • Deploy • Operate • Observe – Monitoring/logging/alerting 24/11/2016 @danielbryantuk
  14. 14. Don’t forget about (RE)ArchitecturE Complexity 24/11/2016 @danielbryantuk
  15. 15. 24/11/2016 @danielbryantuk www.infoq.com/news/2015/04/raffi-krikorian-rearchitecting
  16. 16. What do I mean by ‘ecosystem’? • Build – Local development, pipelines and integration • Test – From local integration to End-to-end • Deploy • Operate • Observe – Monitoring/logging/alerting 24/11/2016 @danielbryantuk
  17. 17. Build 24/11/2016 @danielbryantuk
  18. 18. Developing Locally: The Basics • GitHub’s Boxen (Puppet) • Pivotal’s Sprout (Chef) • Mac-dev-playbook (Ansible) • Hashicorp Vagrant • Docker Compose (Docker machine/native) 24/11/2016 @danielbryantuk
  19. 19. Developing Locally • The naive approach – Replication of env per service – …and dependencies and data stores and... – Soon gets crazy • Local profiles + mocking/stubbing – Spring profiles + Mockito etc 24/11/2016 @danielbryantuk
  20. 20. Developing Locally • Service virtualisation – Hoverfly, Mountebank, Wiremock (Saboteur) • ‘Production-in-a-box’ (IFTTT) – Docker Compose, Vagrant, cf_nise_installer • Environment leasing – Create your own env (e.g. Hailo) 24/11/2016 @danielbryantuk
  21. 21. Developing Locally www.opencredo.com/2015/09/20/working-locally-with-microservices/ 24/11/2016 @danielbryantuk
  22. 22. Create a Pipeline 24/11/2016 @danielbryantuk
  23. 23. Key lesson learned (the Hard Way) • Push through to production as early as possible! • Detect and fix issues – Technology – Process – People 24/11/2016 @danielbryantuk
  24. 24. Test 24/11/2016 @danielbryantuk
  25. 25. Always Remember... 24/11/2016 @danielbryantuk blog.bbv.ch/2012/06/13/acceptance-test-driven-development/
  26. 26. Microservice Testing Basics • Toby Clemson’s article martinfowler.com/articles/microservice-testing 24/11/2016 @danielbryantuk
  27. 27. Microservice Testing Basics • My article https://www.specto.io/blog/recipe-for-designing-building- testing-microservices.html 24/11/2016 @danielbryantuk
  28. 28. (multiple) Pipelines • Big-bang release – Beware of rubber stamping (distributed monolith) – ‘semver’ if you must (semver.org) e.g. 1.2.1 • Gated release – Critical path testing in Stage • Single service Continuous delivery – Consumer-based contracts – Backwards compatibility 24/11/2016 @danielbryantuk
  29. 29. Service-level Integration Testing • Contracts – Pact-JVM github.com/DiUS/pact-jvm – PACT broker github.com/bethesque/pact_broker – Examples: github.com/mstine/microservices-pact 24/11/2016 @danielbryantuk
  30. 30. My Opinions • BDD critical paths throughout application – Including API journey • Contract tests (failure is a conversation) • BDD services API (e.g. Serenity BDD) • Component test and unit test (as normal) – Maven surefire/failsafe 24/11/2016 @danielbryantuk
  31. 31. Final Words on Testing • Don’t forget the ‘ilities” • Security / reliability – ZAP (from the OWASP team) – github.com/continuumsecurity/bdd-security • Performance / scalability – Jmeter (Jenkins Performance plugin) – Gatling – flood.io 24/11/2016 @danielbryantuk
  32. 32. Deploy 24/11/2016 @danielbryantuk
  33. 33. Separate Deploy and Release • Feature flags – Difficult at scale (and distribution) – Enable at ingress • Incremental (phased) rollout • Canary vs blue/green • Avoid datastore migrations (if possible) 24/11/2016 @danielbryantuk
  34. 34. Centralise Configuration • Consul & consul-template • Etcd/ZK & confd • Netflix Archaius • Spring Cloud config • Watch for application rollback! 24/11/2016 @danielbryantuk
  35. 35. Operate 24/11/2016 @danielbryantuk
  36. 36. Building Blocks • HashiCorp Terraform – VMWare? cloud-init and vcloud-tools – “Boot my secure government cloud” • “CAPS” – Chef, Ansible, Puppet, SaltStack – Automated sysadmin • Docker / Ami / RPM / JAR – Machine vs Os/lang artifact • Standardise on an OS – Amazon Linux vs mainstream distros 24/11/2016 @danielbryantuk
  37. 37. DevOps and Programmable Infrastructure • Devops involves programming… • Introduce SOLID principles • Good CI/CD principles – Gitflow etc – Testing 24/11/2016 @danielbryantuk
  38. 38. Service Discovery • External – HAProxy / nginx / ELB etc • Client-side – Netflix Ribbon (with Prana) – ‘Baker Street’ (extending SmartStack) – srv-router • Kubernetes and CF are good to go 24/11/2016 @danielbryantuk
  39. 39. External 24/11/2016 @danielbryantuk
  40. 40. Client-side 24/11/2016 @danielbryantuk
  41. 41. Observe 24/11/2016 @danielbryantuk
  42. 42. When bad things happen, people are always involved 24/11/2016 @danielbryantuk | @oakinger
  43. 43. Monitoring and People 24/11/2016 @danielbryantuk www.infoq.com/news/2015/06/too-big-to-fail
  44. 44. Start with the Basics • Health checks – Coda Hale (DropWizard) Metrics – Spring Boot actuator • KPIs for apps (and business) – Assertions / invariants – Throughput – Queue length 24/11/2016 @danielbryantuk
  45. 45. Logging • What every engineer should know • 10 Tips for Proper Application Logging • ElasticSearch-Logstash-Kibana (ELK) – Buffer/proxy log sending or… – Mount directory into container 24/11/2016 @danielbryantuk
  46. 46. Monitoring • Push – Spring Boot actuator e.g. InfluxDbExporter • Pull – E.g. Telegraf (shout to Tareq Abedrabbo) • InfluxDB vs prometheus vs graphite vs opentsdb • Information radiators – Aggregate vs individual 24/11/2016 @danielbryantuk
  47. 47. Aggregation: Sick Cattle, Not Sick Pets
  48. 48. Opentracing & OpenZipkin 24/11/2016 @danielbryantuk
  49. 49. Problems? • Rob Ewaschuk’s “Philosophy on Alerting” • Brendan Gregg’s USE method – “check utilization, saturation, and errors.” • “DevOps Troubleshooting” – Kyle Rankin 24/11/2016 @danielbryantuk
  50. 50. Let’s wrap this up... 24/11/2016 @danielbryantuk
  51. 51. Summary • development of 3+ services is challenging • Think about your build pipeline(s) • Testing requires a paradigm shift • Operations requires much more automation • Push through to prod as early as possible 24/11/2016 @danielbryantuk
  52. 52. Bedtime reading 24/11/2016 @danielbryantuk
  53. 53. THANKS... @danielbryantuk daniel.bryant@opencredo.com http://muservicesweekly.com/ (Credit to Tareq Abedrabbo for inspiration/guidance) 24/11/2016 @danielbryantuk
  54. 54. Logging vs monitoring • Logging – Primarily used post-incident – Machine readable – Hard to do right – Noise/signal • Monitoring – Useful in Real time during incident – Trend analysis – Easier to do 24/11/2016 @danielbryantuk

×