SpringOne 2021 Session Title: Winning the Lottery with Spring: A Microservices Case Study for the Dutch Lotteries Speaker: Joris Kuipers, CTO at Trifork

1. Winning the Lottery
with Spring
A Micro Microservices Case Study for Nederlandse Loterij

2. About Me
 Joris Kuipers
 @jkuipers
 CTO, hands-on architect and
fly-by-night Spring trainer
@ Trifork Amsterdam

3. Setting The Stage
“When we are born we cry that we are come to this great stage of fools.”
― William Shakespeare

4. Introduction
 Integration platform for Nederlandse Loterij
 Expose many (mostly new) backends to clients by
providing APIs based on custom domain model
 Encapsulate backend details
 Facilitate future replacement
 Serves ca. 2 million users
 peaks during state lottery draws and big soccer games

5. Architecture
 Separate domains
(verticals)
and service types
(horizontals)
 Verticals:
specific games,
subscriptions, players, …

6. Project Setup
 Every service is Spring Boot / Cloud app
 Running on AWS EKS (Kubernetes)
 Blocking HTTP for inter-service communication
 Too early for reactive
 Mono-repo using Gradle build
 More services (~30) than teams (1)
 Easy to have shared libs

7. Chapter 1: Autoconfiguration
for the people
“I like the elitism of the art world.
I think art for the people is a terrible idea.”
― John Waters

8. Spring Boot Autoconfiguration
 Spring Boot’s main and most visible feature
 Used extensively within the framework
 But not restricted to Spring Boot itself
 Conditional configuration classes automatically
applied
 Listed in META-INF/spring.factories

9. Custom Autoconfiguration
Benefits of using autoconfiguration yourself:
 Encapsulation
 Dynamic configuration
 Conditions
 Overridable defaults
 Easy for clients
 Esp. when providing @ConfigurationProperties!

10. Autoconfiguration Examples:
HTTP Client Config
 Apache Component HTTP client
 sane defaults for timeouts, connections (# and TTL), etc.
plus @ConfigurationProperties to override those
 RestTemplateCustomizers
 Ensure HTTP client is used
 Configure request/response logging, incl.
BufferingClientHttpRequestFactory wrapper

11. Discover Configuration Options
@ConfigurationProperties allows IDE auto-completion
 Discover available options
 Inline documentation incl. defaults
 Enum / boolean values also auto-complete!

12. Autoconfiguration Examples:
Client library for 3rd-party service
 Sets up client for calls to external service, adds:
 Custom JSON marshalling config
 Error handling
 Often involves parsing JSON response
 Authentication (OAuth bearer token, basic auth, …)
 Might overlap with error handling:
Spring Retry to redo request after refreshing expired token

13. Chapter 2: Observability for Messaging
“To acquire knowledge, one must study;
but to acquire wisdom, one must observe.”
― Marilyn vos Savant

14. Distributed Tracing
 Spring Sleuth allows distribute tracing
 Propagating correlation ID per logical request
 Called Trace ID
 Instruments many Spring components out of the box

15. Sleuth and SQS
 Spring Cloud AWS for SQS integration
 Template for sending, listener container for receiving
 Added Sleuth integration ourselves
 Tracing info added as message headers
 Allows for two things:
 Correlate async logging as part of regular flow
 Admin tool to link error logs for dead-lettered messages

16. Async Logging Correlation
Example for single trace ID
 Blocking: player-experience -> ~-process -> ~- system
 Async: player-process -> marketing-system

17. Dealing with Dead Lettering
 Messages failing repeatedly go to dead letter queue
 Problem: what do you do with them?
 Answer: it depends…

18. Dealing with Dead Lettering
 Have to understand cause behind failure
 Bug: fix it, retry unless message itself is broken
 Unavailable back-end: wait until available & retry
 Data consistency problem: fix it first, then it depends…
 …
 Also, ensure idempotent message handlers as much
as possible!

19. DLQ Admin Tool
Built a tool to allow customer* to manage this
 Overview of DLQs with # of messages
 Trace logging per message
 Move (retry) or delete individual messages
 Move all messages back from a DLQ
* We reserve pruning to our operations personnel ☺

22. Single click to related logging

23. Chapter 3: Spring Cloud Netflix
“You only grow by coming to the end of something and
by beginning something else.”
― John Irving, The World According to Garp

24. Spring Cloud Netflix
 Part of the start of Spring Cloud
 Allowed Netflix OSS stack to be used with Spring Boot
 Eureka, Ribbon, Hystrix, Zuul, …
 Netflix stopped maintaining most of those libs
 Spring support announced to stop as well,
alternatives provided

25. Our Spring Cloud Netflix Usage
 Ribbon
 For service discovery & client-side load balancing
 Moved to server-side load balancing
 Zuul
 For both inter-service proxies and some proxying to 3rd parties
 Hystrix
 Circuit breaking
 Bulk heading, esp. to prevent all request-handling threads
from blocking on downstream I/O

26. Replacing Zuul: inter-service proxies
 Looked at Spring Cloud Gateway, but not designed
to be embedded
 Simple solution: don’t use proxying anymore

27. Replacing Zuul: 3rd party proxies
 Still needed solution for other proxies
 We use Spring Cloud Gateway MVC
 This is NOT Spring Cloud Gateway!
 Rather a simple RestTemplate wrapper with
Spring-MVC integration for building proxies
 Reactive version exists as well

28. Spring Cloud Gateway MVC
 ProxyExchange passed to controller method
 Allows for adding headers & query params, changing path and
making actual request
 Configure default and “sensitive” headers (not copied)
 Sensitive applies to both request and response
 Should always include host!
spring.cloud.gateway.proxy.sensitive=authorization,cookie,host
spring.cloud.gateway.proxy.headers.x-api-key=${subscription-api.key}

29. @RequestMapping(SubsMgmtProxyController.PATH_PREFIX)
public class SubsMgmtProxyController extends AbstractProxyController {
static final String PATH_PREFIX = "/subscription-mgmt-experience";
…
@RequestMapping(path = "/**", method = {GET, POST})
ResponseEntity<?> proxyOperation(HttpServletRequest request,
@RequestParam Map<String, String> params,
ProxyExchange<byte[]> proxy) {
Player player = determinePlayerId(request, params);
ProxyExchange<byte[]> proxyWithUri = proxy
.headers(new ServletServerHttpRequest(request).getHeaders())
.uri(subsUrl + proxy.path(PATH_PREFIX) + "/" + player.id + queryParams());
return "GET".equals(request.getMethod()) ? proxyWithUri.get()
: proxyWithUri.post();
}
// can be combined with more specific, regular request mapping methods
…
}
Our URI prefix,
which should be stripped
Proxy all GETs & POSTs
without specific mapping
Copy non-sensitive headers
Base URL + stripped path +
custom path segment + query
params
Perform actual proxy call

30. Replacing Hystrix
 Circuit breaking: use Resilience4J instead
 Didn’t like bulk heading as solution to prevent
running out of request handling threads
 Solution: use additional Tomcat connector on
separate port with custom thread pool
 Point Kubernetes liveness/readiness endpoints to that

31. Custom Tomcat Connector
@Configuration
@EnableConfigurationProperties(HealthConnectorProperties.class)
public class TomcatAutoConfiguration {
@Bean
WebServerFactoryCustomizer<TomcatServletWebServerFactory>
tcHealthConnectorCustomizer(HealthConnectorProperties connector) {
return server -> {
Connector healthConnector = new Connector();
healthConnector.setPort(connector.getPort());
AbstractProtocol protocol = (AbstractProtocol) healthConnector.getProtocolHandler();
protocol.setMinSpareThreads(connector.getMinThreads());
protocol.setMaxThreads(connector.getMaxThreads());
server.addAdditionalTomcatConnectors(healthConnector);
};
}
}
probes:
liveness:
enabled: false
path: /actuator/health/liveness
port: 8888
readiness:
enabled: false
path: /actuator/health/readiness
port: 8888

32. Conclusion
 Use what the frameworks provide you with
 Don’t be afraid to extend them
 Build your own tools to support your system
 Know what’s out there, know what’s supported
 Follow me to Q&A for questions & discussions!