Building a microservice ecosystem
Daniel Bryant
@danielbryantuk
OpencRedo

Today
• What is a ‘microservice ecosystem’?
• Local dev of services becomes challenging
• Think about your build pipeline(s)
• Testing requires a paradigm shift
• Push to production as early as possible
24/11/2016 @danielbryantuk

@danielbryantuk
• Chief Scientist at OpenCredo, CTO at SpectoLabs
ü Transforming organisations through technology and teams
ü Agile, Lean, Architecture, CI/CD, DevOps
ü Microservices, cloud, Containers, Java, Go, Docker, Kubernetes
• London Java Community Associate
• Adopt OpenJDK and JSR
• InfoQ Editor, DZone MVB, VOXXED, O'Reilly
24/11/2016 @danielbryantuk

My Biggest Claim to Fame in Microservices?
24/11/2016 @danielbryantuk

Seriously Though...…
24/11/2016 @danielbryantuk
skillsmatter.com/skillscasts/7004-our-journey-to-world-gifting-
domination-how-notonthehighstreet-com-embraced-docker

So, What is a microservice?
“Loosely coupled service oriented architecture
with bounded contexts”
Adrian Cockcroft
“Applications that fit in your head”
James Lewis
24/11/2016 @danielbryantuk

Why? The move towards Cloud-native
• Hypothesis-driven business (and development)
• Microservice architecture (and 'Micro' teams)
• Devops mindset/culture (shared, learning/feedback & Mechanical sympathy)
• Continuous delivery
• Automated, self-service platform
Hat tip @caseywest
24/11/2016 @danielbryantuk

Exploring the Ecosystem
24/11/2016 @danielbryantuk

Adrian Cockcroft’s Thoughts
24/11/2016 @danielbryantuk
www.slideshare.net/adriancockcroft/microxchg-microservices

Alexis Richardson’s Thoughts
24/11/2016 @danielbryantuk
gotocon.com/goto-london-2015/#!#schedulePopupExtras-7011

wikibon.com/wp-content/uploads/container_implementations.png
Technology Choices
24/11/2016 @danielbryantuk

24/11/2016

What do I mean by ‘ecosystem’?
• Build
– Local development, pipelines and integration
• Test
– From local integration to End-to-end
• Deploy
• Operate
• Observe
– Monitoring/logging/alerting
24/11/2016 @danielbryantuk

Don’t forget about (RE)ArchitecturE Complexity
24/11/2016 @danielbryantuk

24/11/2016 @danielbryantuk
www.infoq.com/news/2015/04/raffi-krikorian-rearchitecting

What do I mean by ‘ecosystem’?
• Build
– Local development, pipelines and integration
• Test
– From local integration to End-to-end
• Deploy
• Operate
• Observe
– Monitoring/logging/alerting
24/11/2016 @danielbryantuk

Build
24/11/2016 @danielbryantuk

Developing Locally: The Basics
• GitHub’s Boxen (Puppet)
• Pivotal’s Sprout (Chef)
• Mac-dev-playbook (Ansible)
• Hashicorp Vagrant
• Docker Compose (Docker machine/native)
24/11/2016 @danielbryantuk

Developing Locally
• The naive approach
– Replication of env per service
– …and dependencies and data stores and...
– Soon gets crazy
• Local profiles + mocking/stubbing
– Spring profiles + Mockito etc
24/11/2016 @danielbryantuk

Developing Locally
• Service virtualisation
– Hoverfly, Mountebank, Wiremock (Saboteur)
• ‘Production-in-a-box’ (IFTTT)
– Docker Compose, Vagrant, cf_nise_installer
• Environment leasing
– Create your own env (e.g. Hailo)
24/11/2016 @danielbryantuk

Developing Locally
www.opencredo.com/2015/09/20/working-locally-with-microservices/
24/11/2016 @danielbryantuk

Create a Pipeline
24/11/2016 @danielbryantuk

Key lesson learned (the Hard Way)
• Push through to production as early as possible!
• Detect and fix issues
– Technology
– Process
– People
24/11/2016 @danielbryantuk

Test
24/11/2016 @danielbryantuk

Always Remember...
24/11/2016 @danielbryantuk
blog.bbv.ch/2012/06/13/acceptance-test-driven-development/

Microservice Testing Basics
• Toby Clemson’s article
martinfowler.com/articles/microservice-testing
24/11/2016 @danielbryantuk

Microservice Testing Basics
• My article
https://www.specto.io/blog/recipe-for-designing-building-
testing-microservices.html
24/11/2016 @danielbryantuk

(multiple) Pipelines
• Big-bang release
– Beware of rubber stamping (distributed monolith)
– ‘semver’ if you must (semver.org) e.g. 1.2.1
• Gated release
– Critical path testing in Stage
• Single service Continuous delivery
– Consumer-based contracts
– Backwards compatibility
24/11/2016 @danielbryantuk

Service-level Integration Testing
• Contracts
– Pact-JVM github.com/DiUS/pact-jvm
– PACT broker
github.com/bethesque/pact_broker
– Examples:
github.com/mstine/microservices-pact
24/11/2016 @danielbryantuk

My Opinions
• BDD critical paths throughout application
– Including API journey
• Contract tests (failure is a conversation)
• BDD services API (e.g. Serenity BDD)
• Component test and unit test (as normal)
– Maven surefire/failsafe
24/11/2016 @danielbryantuk

Final Words on Testing
• Don’t forget the ‘ilities”
• Security / reliability
– ZAP (from the OWASP team)
– github.com/continuumsecurity/bdd-security
• Performance / scalability
– Jmeter (Jenkins Performance plugin)
– Gatling
– flood.io
24/11/2016 @danielbryantuk

Deploy
24/11/2016 @danielbryantuk

Separate Deploy and Release
• Feature flags
– Difficult at scale (and distribution)
– Enable at ingress
• Incremental (phased) rollout
• Canary vs blue/green
• Avoid datastore migrations (if possible)
24/11/2016 @danielbryantuk

Centralise Configuration
• Consul & consul-template
• Etcd/ZK & confd
• Netflix Archaius
• Spring Cloud config
• Watch for application rollback!
24/11/2016 @danielbryantuk

Operate
24/11/2016 @danielbryantuk

Building Blocks
• HashiCorp Terraform
– VMWare? cloud-init and vcloud-tools
– “Boot my secure government cloud”
• “CAPS”
– Chef, Ansible, Puppet, SaltStack
– Automated sysadmin
• Docker / Ami / RPM / JAR
– Machine vs Os/lang artifact
• Standardise on an OS
– Amazon Linux vs mainstream distros
24/11/2016 @danielbryantuk

DevOps and Programmable Infrastructure
• Devops involves programming…
• Introduce SOLID principles
• Good CI/CD principles
– Gitflow etc
– Testing
24/11/2016 @danielbryantuk

Service Discovery
• External
– HAProxy / nginx / ELB etc
• Client-side
– Netflix Ribbon (with Prana)
– ‘Baker Street’ (extending SmartStack)
– srv-router
• Kubernetes and CF are good to go
24/11/2016 @danielbryantuk

External
24/11/2016 @danielbryantuk

Client-side
24/11/2016 @danielbryantuk

Observe
24/11/2016 @danielbryantuk

When bad things happen, people are always involved
24/11/2016 @danielbryantuk | @oakinger

Monitoring and People
24/11/2016 @danielbryantuk
www.infoq.com/news/2015/06/too-big-to-fail

Start with the Basics
• Health checks
– Coda Hale (DropWizard) Metrics
– Spring Boot actuator
• KPIs for apps (and business)
– Assertions / invariants
– Throughput
– Queue length
24/11/2016 @danielbryantuk

Logging
• What every engineer should know
• 10 Tips for Proper Application Logging
• ElasticSearch-Logstash-Kibana (ELK)
– Buffer/proxy log sending or…
– Mount directory into container
24/11/2016 @danielbryantuk

Monitoring
• Push
– Spring Boot actuator e.g. InfluxDbExporter
• Pull
– E.g. Telegraf (shout to Tareq Abedrabbo)
• InfluxDB vs prometheus vs graphite vs
opentsdb
• Information radiators
– Aggregate vs individual
24/11/2016 @danielbryantuk

Aggregation: Sick Cattle, Not Sick Pets

Opentracing & OpenZipkin
24/11/2016 @danielbryantuk

Problems?
• Rob Ewaschuk’s “Philosophy on Alerting”
• Brendan Gregg’s USE method
– “check utilization, saturation, and errors.”
• “DevOps Troubleshooting”
– Kyle Rankin
24/11/2016 @danielbryantuk

Let’s wrap this up...
24/11/2016 @danielbryantuk

Summary
• development of 3+ services is challenging
• Think about your build pipeline(s)
• Testing requires a paradigm shift
• Operations requires much more automation
• Push through to prod as early as possible
24/11/2016 @danielbryantuk

Bedtime reading
24/11/2016 @danielbryantuk

THANKS...
@danielbryantuk
daniel.bryant@opencredo.com
http://muservicesweekly.com/
(Credit to Tareq Abedrabbo for inspiration/guidance)
24/11/2016 @danielbryantuk

Logging vs monitoring
• Logging
– Primarily used post-incident
– Machine readable
– Hard to do right
– Noise/signal
• Monitoring
– Useful in Real time during incident
– Trend analysis
– Easier to do
24/11/2016 @danielbryantuk