3 DevOps teams at the Dutch National Police are building big data applications in a private cloud using microservices architecture. They develop independently but coordinate planning. Testing includes unit, integration, end-to-end, load, and mutation tests. Features are developed on branches and deployed continuously to test and production after sprints. Challenges include balancing security and performance for stateless services and preventing monolithic frontends. They aim to improve with zero downtime deployments, product teams, and automated security testing.

1. bertjan@openvalue.nl
Microservices in action
at the Dutch National Police
Bert Jan Schrijver
@bjschrijver

2. Bert Jan Schrijver
L e t ’ s m e e t
@bjschrijver

3. Architecture and
platform
Frontend
Methodology
and culture
Introduction
Development and
testing
Build tools, deployments
and running in production
Challenges and
looking ahead
Outline
W h a t ‘ s n e x t ?
Backend
@bjschrijver

4. The police protects the
democracy, maintains the law
and is the authority on the
streets. Around 65.000 people
work at the Dutch police, of
which over 1500 IT
professionals.
Dutch National Police

5. CLOUD
PLATFORM
ANALYSE
PATRONEN
BIG DATA
SECURITY3 DevOps teams are building
high tech big data web
applications in a private cloud
environment. These
applications support police
related themes.
Product line
Cloud | Big Data | Internet

6. Methodology and culture

7. • 3 teams, separate backlogs
• Overall planning at start of sprint
• Minimal planning ritual
• Usability tests as part of sprint
• (Almost) no meetings
• Phabricator as tool of choice
Methodology

8. • Continuous Delivery & DevOps
• Short feedback loops
• Embrace change
• Minimal dependencies outside team
• Invest in people, not in products
• Open, transparent, verifiable
Culture
Source: http://kids.nationalgeographic.com/explore/countries/netherlands/#netherlands-tulip-fields.jpg

9. • How to handle support, monitoring and
(pro-active) maintenance during sprint?
Make this a role that cycles through the
team.
“Operator of the day” and “Developer of
the day”
Culture
Source: http://kids.nationalgeographic.com/explore/countries/netherlands/#netherlands-tulip-fields.jpg

10. Architecture and platform

11. • End-to-end security and encryption
• Version control for everything
• Horizontally scalable, no single points of failure
• No runtime dependencies on external systems
• Standardised naming
• Right tool for the job (polyglot)
Architecture

12. Architecture
• This slide is intentionally left blank…

13. Current architecture

14. Current architecture

15. Current architecture

16. Current architecture

17. Current architecture

18. Current architecture

19. Current architecture

20. Current architecture

21. Source: https://www.google.com/about/datacenters
• OpenStack private cloud
• Ceph distributed storage
• General cloud services for police
organisation
• 3000 managed desktops
• Automation starts when hardware boots
Platform

22. Source: https://www.google.com/about/datacenters
• 100’s of physical servers
• 14 physical disks per server
• It’s not a matter IF stuff breaks but WHEN
Platform

23. Source: https://www.google.com/about/datacenters
• How to manage 100’s of physical servers,
1000’s of virtual servers and 1000’s of
desktops?
Infrastructure as code.
Terraform for cloud, Puppet for servers,
Ansible for desktops
Platform

24. Frontend

25. • Angular 4.x, TypeScript, RxJS
• Bootstrap, responsive design
• Graceful degradation when backend fails
Frontend

26. • How to decouple building a feature from
release and deployment?
Use feature toggles.
Frontend

27. • How to prevent creating a monolithic
frontend?
Apply microservice principles to the
frontend too.
Create re-usable frontend components as
standalone component libraries.
Frontend

28. Backend

29. • Small in size, single responsibility
• Runs in its own process
• Independently develop, deploy, upgrade, scale
• Has its own data store
• Distributed by default
• Potentially heterogeneous/polyglot
• Light-weight communication
Anatomy of a microservice

30. • Did we build microservices right from the start?
Nope.
• Why did we move to a microservices architecture?
• Scalability: both for performance/load and
development teams
• Modularity: independent development and
deployment of each part of the system
• The ‘cool factor’ helped bit ;-)
The path to microservices

31. • How did we move to microservices?
• Split the existing system in modules (bounded
contexts) - defined by business functionality
• For each module, create a microservice
• For every microservice, check if it now only
serves 1 part of the domain
• If so: cool. If not: repeat - split it up more.
The path to microservices

32. • Spring Boot, Java 8, Maven
• Stateless
• 1 service in 1 jar on 1 JVM on 1 host (and 1 git project)
• Minimal amount of shared code:
• Security
• Logging and metrics
• Past: high available via load balancers
• Present: moving from LB’s to service discovery
Backend

33. Service discovery is an advanced pattern.
Do you need it?
Probably not.
Do we need it?
We think we do.
Service discovery

34. Service discovery
Why do we use service discovery?
• HA & load balancing without single points of failure
• Direct secure end-to-end client-service comm.
• Kerberos: picky on DNS verification
• Dynamic horizontal scaling
• Resilience: automated response to failure
• Location transparency
• Zero downtime deployments

35. Consul cluster
service-example0
service-example1
Gebruiker
ui-example0
(webserver)
consul-template
sseserver
Service discovery setup
• Client-side discovery
• Self-registration

36. Development and testing

37. • Feature branch based development
• Master branch must always be releasable
• Test environment on OpenStack runs the
master branch
• Feature branches only live locally and on
the CI server
Development

38. • How to locally run a system that consists
of ~50 services?
Run only the component(s) you work on
locally.
For other components, local env connects
to test env on OpenStack
Development

39. • Unit tests
• Mutation tests
• Service/integration tests: Spring boot
integration, embedded in-memory data
stores, REST assured
• End-to-end test: Protractor
• Load tests: Gatling
Testing

40. • How to test feature branches in a
microservices environment?
Spin up branch version of component in a
container on the CI environment.
For dependencies, connect to test env.
Avoid changing multiple components at once.
Testing

41. Build tools, deployments
and running in production

42. • Gitlab
• Jenkins with Docker swarm slave nodes
• Jenkins 2 pipelines
• Nexus
• Sonar
Build tools

43. • How to manage >100 builds with a single
team?
Create modular, reusable build
definitions.
(See https://virtualjug.com/pipeline-as-code-building-
continuous-delivery-pipelines-with-jenkins-2/)
Build tools

44. • Every push to master is a release
• Config embedded in executable jar
• Deployments via Rundeck and Puppet
Deployments

45. • How to know when, what and which version(s)
to deploy?
• Minimise administration and think time.
• Test environment: deploy component on commit
• Everything from test -> acc during sprint
• Everything from acc -> prod after sprint
• Single component test -> acc -> prod when needed
Deployments

46. • Logging and dashboards via Graylog
• Metrics:
• Spring Boot actuator
• Distributed tracing with Zipkin
• Grafana for graphs
• Kafka stats via Burrow
• Monitoring via Sensu and Flapjack
Running in production

47. Photo: Dave Lehl
Challenges and looking ahead

48. Challenges
01
Share as little as possible; prefer
duplication over coupling.
Sharing code between services
04Authentication and authorisation
happen at every request. Find the
balance between performance and
security.
Running stateless has a cost
When moving fast,
don’t forget to finish up before
starting something new.
Switching focus has a cost
06
Throwing something away and
starting over can work out better
than refactoring.
Don’t be afraid to rebuild03
Microservices are not just for the
backend. Modularity is just as
important on the frontend.
Monolithic frontend
02
Minimalize dependencies on
other teams, or it will slow you
down.
Cross functional team
composition is vital
05
and lessons learned
@bjschrijver

49. Looking ahead
Upgrades and fixes without users even
noticing.
0-downtime deployments
Our plans for the (near) future.
@bjschrijver
Cross-functional teams with vertical
(full stack) responsibilities.
Product teams
Split the frontend in products and re-
usable components.
Modular frontend
There is no silver bullet here, but useful
tools and practices do exist.
Automated security testing
Get the teams the information they
need, but only when they need it.
Better dashboards and alerting

50. Questions?
@bjschrijver

51. Thanks for your time.
Got feedback? Tweet it!
All pictures belong
to their respective
authors
@bjschrijver