3 DevOps teams are building high tech big data web applications in a private cloud environment to support police work. These teams use microservices architectures, with separate front-end and back-end components developed independently but integrated via service discovery. Testing includes unit, integration, and load testing across feature branches in a continuous integration environment before production deployment. Future work includes improving zero downtime deployments, modularizing the front-end, and automating security testing.

1. bertjan@openvalue.nl
Microservices in action
at the Dutch National Police
Bert Jan Schrijver
@bjschrijver

2. Bert Jan Schrijver
L e t ’ s m e e t
@bjschrijver

3. Architecture and
platform
Frontend
Methodology
and culture
Introduction
Development and
testing
Build tools, deployments
and running in production
Challenges and
looking ahead
Outline
W h a t ‘ s n e x t ?
Backend

4. The police protects the
democracy, maintains the law
and is the authority on the
streets. Around 65.000 people
work at the Dutch police, of
which over 1500 IT
professionals.
Dutch National Police

5. CLOUD
PLATFORM
ANALYSE
PATRONEN
BIG DATA
SECURITY3 DevOps teams are building
high tech big data web
applications in a private cloud
environment. These
applications support police
related themes.
Product line
Cloud | Big Data | Internet

6. Methodology and culture

7. • 3 teams, separate backlogs
• Overall planning at start of sprint
• Minimal planning ritual
• Usability tests as part of sprint
• Phabricator as tool of choice
• (Almost) no meetings
Methodology

8. • Continuous Delivery & DevOps
• Short feedback loops
• Embrace change
• Minimal dependencies outside team
• Invest in people, not in products
• Open, transparent, verifiable
Culture
Source: http://kids.nationalgeographic.com/explore/countries/netherlands/#netherlands-tulip-fields.jpg

9. • How to handle support, monitoring and
(pro-active) maintenance during sprint?
Make this a role that cycles through the
team.
“Operator of the day” and “Developer of
the day”
Culture
Source: http://kids.nationalgeographic.com/explore/countries/netherlands/#netherlands-tulip-fields.jpg

10. Architecture and platform

11. • End-to-end security and encryption
• Version control for everything
• Horizontally scalable, no single points of failure
• No runtime dependencies on external systems
• Standardised naming
• Application config lives with code
• Services defined by business functionality
• Right tool for the job (polyglot)
Architecture

12. Architecture
• This slide is intentionally left blank…

13. Current architecture

14. Current architecture

15. Current architecture

16. Current architecture

17. Current architecture

18. Current architecture

19. Current architecture

20. Current architecture

21. Source: https://www.google.com/about/datacenters
• OpenStack private cloud
• Ceph distributed storage
• General cloud services for police
organisation
• 3000 managed desktops
• Automation starts when hardware boots
Platform

22. Source: https://www.google.com/about/datacenters
• How to manage 100’s of physical servers,
1000’s of virtual servers and 1000’s of
desktops?
Infrastructure as code.
Terraform for cloud, Puppet for servers,
Ansible for desktops
Platform

23. Frontend

24. • Angular 4.x, TypeScript, RxJS
• Bootstrap, responsive design
• Graceful degradation when backend fails
Frontend

25. • How to decouple building a feature from
release and deployment?
Create feature toggles.
Frontend

26. • How to prevent creating a monolithic
frontend?
Apply microservice principles to the
frontend too.
Create re-usable frontend components as
standalone component libraries.
Frontend

27. Backend

28. • Small in size, single responsibility
• Runs in its own process
• Independently develop, deploy, upgrade, scale
• Has its own data store
• Distributed by default
• Potentially heterogeneous/polyglot
• Light-weight communication
Anatomy of a microservice

29. • Did we build microservices right from the start?
Nope.
• Why did we move to a microservices architecture?
• Scalability: both for performance/load and
development teams
• Modularity: independent development and
deployment of each part of the system
• The ‘cool factor’ helped bit ;-)
The path to microservices

30. • How did we move to microservices?
• Split the system in modules (bounded
contexts)
• For each module, create a microservice
• For every microservice, check if it now only
serves 1 part of the domain
• If so: cool. If not: repeat - split it up more.
The path to microservices

31. • Spring Boot, Java 8, Maven
• Stateless
• 1 service in 1 jar on 1 JVM on 1 host
• Minimal amount of shared code:
• Security
• Logging and metrics
• Past: high available via load balancers
• Present: moving from LB’s to service discovery
Backend

32. Service discovery is an advanced pattern.
Do you need it?
Probably not.
Do we need it?
We think we do.
Service discovery

33. Service discovery
• Why do we use service discovery?
• HA & load balancing without single points of failure
• Direct secure end-to-end client-service communication
• Kerberos: picky on DNS verification
• Dynamic horizontal scaling
• Resilience: automated response to failure
• Location transparency
• Zero downtime deployments

34. Consul cluster
service-example0
service-example1
Gebruiker
ui-example0
(webserver)
consul-template
sseserver
Service discovery setup
• Client-side discovery
• Self-registration

35. Development and testing

36. • Feature branch based development
• Master branch must always be releasable
• Test environment on OpenStack runs the
master branch
• Feature branches only live locally and on
the CI server
Development

37. • How to locally run a system that consists
of ~50 services?
Run only the component(s) you work on
locally.
For other components, local env connects
to test env on OpenStack
Development

38. • Unit tests
• Mutation tests
• Service/integration tests: Spring boot
integration, embedded in-memory data
stores, REST assured
• End-to-end test: Protractor
• Load tests: Gatling
Testing

39. • How to test feature branches in a
microservices environment?
Spin up branch version of component in a
container on the CI environment.
For dependencies, connect to test env.
Avoid changing multiple components at once.
Testing

40. Build tools, deployments
and running in production

41. • Gitlab
• Jenkins with Docker swarm slave nodes
• Jenkins 2 pipelines
• Nexus
• Sonar
Build tools

42. • How to manage >100 builds with a single
team?
Create modular, reusable build
definitions.
(See https://virtualjug.com/pipeline-as-code-building-
continuous-delivery-pipelines-with-jenkins-2/)
Build tools

43. • Every push to master is a release
• Config embedded in executable jar
• Deployments via Rundeck and Puppet
Deployments

44. • How to know when, what and which version(s)
to deploy?
• Minimize administration and think time.
• Development: deploy component on commit
• Everything from dev -> acc during sprint
• Everything from acc -> prod after sprint
• Single component dev -> acc -> prod when needed
Deployments

45. • Logging and dashboards via Graylog
• Metrics:
• Spring Boot actuator
• Distributed tracing with Zipkin
• Grafana
• Kafka stats via Burrow
• Monitoring via Sensu and Flapjack
Running in production

46. Photo: Dave Lehl
Challenges and looking ahead

47. Challenges
01
Share as little as possible; prefer
duplication over coupling.
Sharing code between services
04Authentication and authorisation
happen at every request. Find the
balance between performance and
security.
Running stateless has a cost
When moving fast,
don’t forget to finish up before
starting something new.
Switching focus has a cost
06
Throwing something away and
starting over can work out better
than refactoring.
Don’t be afraid to rebuild03
Microservices are not just for the
backend. Modularity is just as
important on the frontend.
Monolithic frontend
02
Minimalize dependencies on
other teams, or it will slow you
down.
Cross functional team
composition is vital
05
and lessons learned

48. Looking ahead
Upgrades and fixes without users even
noticing.
0-downtime deployments
Our plans for the (near) future.
@bjschrijver
Cross-functional teams with vertical
(full stack) responsibilities.
Product teams
Split the frontend in products and re-
usable components.
Modular frontend
There is no silver bullet here, but useful
tools and practices do exist.
Automated security testing
Get the teams the information they
need, but only when they need it.
Better dashboards and alerting

49. Questions?
@bjschrijver

50. Thanks for your time.
Got feedback? Tweet it!
All pictures belong
to their respective
authors
@bjschrijver