Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Practical
non-blocking microservices
in Java 8
Michał	Baliński
System	Architect
Oleksandr	Goldobin
System	Architect
Cloud of microservices for secure IoT
gateway backing
service
core
service
gateway
gateway
core
service
backing
service
The demand and characteristics of our domain
Crowded IoT environment	
Slow,	long lived connections
Big	amount of	concurren...
OTA Gateway – Use Case
TSM
Trusted
Service
Manager
OTA Gateway – Use Case
TSM
Trusted
Service
Manager
Security Module
OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1.	submit
scripts
OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1.	submit
scripts	
H...
OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1.	submit
scripts	
H...
OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1.	submit
scripts	
H...
OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1.	submit
scripts	
H...
OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1.	submit
scripts	
H...
OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1.	submit
scripts	
H...
OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP HTTP
HTTP
TCP
OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
Log
Storage
HTTP HTTP
HTT...
Let’s test blocking approach!
OTA Gateway Blocking - technologies
17
TSM
DB Log
Storage
HTTP
TCP STDOUT
OTA
Gateway
JAX-RS
Logback
appender
Security Mod...
OTA Gateway – Blocking - test
OTA
(Over-the-Air)
Gateway
send
2 scripts per
request
Security Module
DB
Log
Storage
HTTP HT...
Blocking – 1k threads
OTA
(Over-the-Air)
Gateway
send
2 scripts per
request
Security Module
DB
Log
Storage
HTTP HTTP
HTTP
...
Blocking – 1k threads
500 req/s 1000 req/s 1500 req/s 2000 req/s
The drawbacks of classic synchronous I/O
One	thread per	connection
Threads waiting instead of	running
Context switches
Res...
Let’s switch from blocking to non-blocking
OTA Gateway Non-blocking - technologies
23
TSM
DB Log
Storage
HTTP
TCP STDOUT
OTA
Gateway
JAX-RS 2.0
Logback
async
appende...
Non-blocking – 16 threads
OTA
(Over-the-Air)
Gateway
send
2 scripts per
request
Security Module
DB
Log
Storage
HTTP HTTP
H...
Non-blocking – 16 threads
1k req/s 2k req/s 2.5k req/s 3k req/s
Blocking Non-blocking
1000	threads 56	threads
1.2	GB 0.5	GB
~1.2k	r/s 2.5k	r/s
Let’s talk about challenges
OTA Gateway Blocking – sequence diagram
OTA
Gateway
TSM
Security
Module DB Logs
loop
1.	submit
scripts	
2.	encrypt
script
...
OTA Gateway Non-blocking – sequence diagram
OTA
Gateway
TSM
Security
Module DB Logs
loop
1.	submit
scripts	
2.	encrypt
scr...
OTA Non-blocking – realityOTA
Gateway
TSM
Security
Module DB Logs
„loopedprocessingchain”
1.	submit
scripts	
4.	submition
...
Code. Bird view
3110	October	2016
package org.demo.ota.blocking.rest;
@Path("se")
public class ScriptSubmissionResource ex...
Code. Blocking. Submission 1
10	October	2016 32
@POST
@Path("/{seId}/scripts")
@Consumes(MediaType.APPLICATION_JSON)
@Prod...
Code. Blocking. Submission 2
10	October	2016 33
log.debug("Processing {} scripts submission", scripts.size(), seId);
for (...
Code. Non-blocking. Submission 1
10	October	2016 34
@POST
@Path("/{seId}/scripts")
@Consumes(MediaType.APPLICATION_JSON)
@...
Code. Non-blocking. Submission 1
10	October	2016 35
@POST
@Path("/{seId}/scripts")
@Consumes(MediaType.APPLICATION_JSON)
@...
Code. Non-blocking. Submission 2
10	October	2016 36
METRICS.instrumentStage(() -> {
log.debug(
"{} Processing {} scripts s...
Code. Non-blocking. Submission 3
10	October	2016 37
private CompletionStage<Void> encryptAndStoreAllScripts(
final Diagnos...
Code. Non-blocking. Submission 4
10	October	2016 38
private CompletionStage<Void> encryptAndStoreSingleScript(
final Diagn...
Code. Blocking. Integration
10	October	2016 39
private final JedisPool pool;
public void storeScript(String seId, Script s...
Code. Non-Blocking. Integration
10	October	2016 40
private final RedisAsyncCommands<String, String> commands;
public Compl...
Diagnostics in non-blocking systems
No	clear	stack	traces, need for	good logs
Name	your	threads properly
MDC becomes	usele...
NIO technology landscape
JDK	1.4
NIO
JDK	1.7
NIO.2
JAX-RS	2.x
Servlet API	3.x
Lessons learned, part 1
Vanila Java	8 for	NIO	µ-services
Netty best	for	custom	protocols	in	NIO
Unit	tests should	be	synch...
Lessons learned, part 2
Functional programming patterns	for	readability
Immutability as 1-st	class	citizen
Scala may be	go...
Conclusion
Non-blocking	processing	can	really	
save	your resources	(== money)
But!
Weight	all	pros	and	cons	and	use	non-bl...
Thank you.
Michał	Baliński
m.balinski@oberthur.com
Oleksandr	Goldobin
o.goldobin@oberthur.com
goldobin
@goldobin
balonus
@...
Upcoming SlideShare
Loading in …5
×

JDD 2016 - Michał Balinski, Oleksandr Goldobin - Practical Non Blocking Microservices in Java 8

48 views

Published on

We will show how to write application in Java 8 that do not waste resources and which can maximize effective utilization of CPU/RAM. There will be presented comparison of blocking and non-blocking approach for I/O and application services. Based on microservices implementing simple business logic in security/cryptography/payments domain, we will demonstrate following aspects: * NIO at all edges of application * popular libraries that support NIO * single instance scalability * performance metrics (incl. throughput and latency) * resources utilization * code readability with CompletableFuture * application maintenance and debugging All above based on our experiences gathered during development of software platforms at Oberthur Technologies R&D Poland.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

JDD 2016 - Michał Balinski, Oleksandr Goldobin - Practical Non Blocking Microservices in Java 8

  1. 1. Practical non-blocking microservices in Java 8 Michał Baliński System Architect Oleksandr Goldobin System Architect
  2. 2. Cloud of microservices for secure IoT gateway backing service core service gateway gateway core service backing service
  3. 3. The demand and characteristics of our domain Crowded IoT environment Slow, long lived connections Big amount of concurrent connections Scalability and resilience Rather I/O intensive than CPU intensive
  4. 4. OTA Gateway – Use Case TSM Trusted Service Manager
  5. 5. OTA Gateway – Use Case TSM Trusted Service Manager Security Module
  6. 6. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB
  7. 7. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts
  8. 8. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts
  9. 9. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts TCP 3. store scripts
  10. 10. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts TCP 4. submition response 3. store scripts
  11. 11. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts TCP 4. submition response HTTP 5. poll scripts 3. store scripts
  12. 12. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts TCP 3. store scripts 4. submition response HTTP 5. poll scripts 6. search scripts
  13. 13. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts TCP 3. store scripts 4. submition response HTTP 5. poll scripts 6. search scripts 7. response with scripts
  14. 14. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP HTTP HTTP TCP
  15. 15. OTA Gateway – Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB Log Storage HTTP HTTP HTTP TCP File I/OMonitoring System HTTP
  16. 16. Let’s test blocking approach!
  17. 17. OTA Gateway Blocking - technologies 17 TSM DB Log Storage HTTP TCP STDOUT OTA Gateway JAX-RS Logback appender Security Module JAX-RS Jedis JAX-RS Client
  18. 18. OTA Gateway – Blocking - test OTA (Over-the-Air) Gateway send 2 scripts per request Security Module DB Log Storage HTTP HTTP HTTP TCP File I/O emulated latency 200 ms expected latency < 450 ms verified with throughput over 7k req/s
  19. 19. Blocking – 1k threads OTA (Over-the-Air) Gateway send 2 scripts per request Security Module DB Log Storage HTTP HTTP HTTP TCP File I/O emulated latency 200 ms max 1000 connections max 1000 threads expected latency < 450 ms
  20. 20. Blocking – 1k threads 500 req/s 1000 req/s 1500 req/s 2000 req/s
  21. 21. The drawbacks of classic synchronous I/O One thread per connection Threads waiting instead of running Context switches Resource wasting (~1 MB per thread - 64bit)
  22. 22. Let’s switch from blocking to non-blocking
  23. 23. OTA Gateway Non-blocking - technologies 23 TSM DB Log Storage HTTP TCP STDOUT OTA Gateway JAX-RS 2.0 Logback async appender Async Http Client 2 (Netty) Security Module JAX-RS 2.0 Lettuce (Netty)
  24. 24. Non-blocking – 16 threads OTA (Over-the-Air) Gateway send 2 scripts per request Security Module DB Log Storage HTTP HTTP HTTP TCP File I/O emulated latency 200 ms no limit for connections max 16 threads expected latency < 450 ms
  25. 25. Non-blocking – 16 threads 1k req/s 2k req/s 2.5k req/s 3k req/s
  26. 26. Blocking Non-blocking 1000 threads 56 threads 1.2 GB 0.5 GB ~1.2k r/s 2.5k r/s
  27. 27. Let’s talk about challenges
  28. 28. OTA Gateway Blocking – sequence diagram OTA Gateway TSM Security Module DB Logs loop 1. submit scripts 2. encrypt script 3a. store script 4. submition response 3b. count scripts
  29. 29. OTA Gateway Non-blocking – sequence diagram OTA Gateway TSM Security Module DB Logs loop 1. submit scripts 2. encrypt script 3a. store script 4. submition response 3b. count scripts
  30. 30. OTA Non-blocking – realityOTA Gateway TSM Security Module DB Logs „loopedprocessingchain” 1. submit scripts 4. submition response 3b. count scripts HTTP Server 2. encrypt script 3a. store script Logging DB Client Security Client
  31. 31. Code. Bird view 3110 October 2016 package org.demo.ota.blocking.rest; @Path("se") public class ScriptSubmissionResource extends Application { private static final Logger log = LoggerFactory.getLogger(ScriptSubmissionResource.class); private static final ResourceMetrics METRICS = new ResourceMetrics("ota_submission"); private SecureModuleClient secureModuleClient = SecureModuleClient.instance(); private ScriptStorageClient scriptStorageClient = ScriptStorageClient.instance(); @POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public long submitScripts(@PathParam("seId") String seId, List<Script> scripts) { MDC.put("flow", "submission"); MDC.put("se", seId); return METRICS.instrument(() -> { log.debug("Processing {} scripts submission", scripts.size(), seId); for (int i = 0; i < scripts.size(); i++) { final Script script = scripts.get(i); log.debug("Encrypting {} script", i); final String encryptedPayload = secureModuleClient.encrypt(seId, script.getPayload()); script.setPayload(encryptedPayload); log.debug("Storing encrypted script {}", i); scriptStorageClient.storeScript(seId, script); } long numberOfScripts = scriptStorageClient.numberOfScriptsForSe(seId); log.debug("Request processed", seId); return numberOfScripts; }); } @Override public Set<Object> getSingletons() { return Collections.singleton(this); } } package org.demo.ota.nonblocking.rest; @Path("se") public class ScriptSubmissionResource extends Application { private static final Logger log = LoggerFactory.getLogger(ScriptSubmissionResource.class); private static final ResourceMetrics METRICS = new ResourceMetrics("ota_submission"); private SecureModuleClient secureModuleClient = SecureModuleClient.instance(); private ScriptStorageClient scriptStorageClient = ScriptStorageClient.instance(); @POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public void submitScripts( @PathParam("seId") String seId, List<Script> scripts, @Suspended final AsyncResponse asyncResponse ) { final DiagnosticContext diagnosticContext = new DiagnosticContext("submission", seId); METRICS.instrumentStage(() -> { log.debug("{} Processing {} scripts submission", diagnosticContext, scripts.size()); return encryptAndStoreAllScripts(diagnosticContext, seId, scripts) .thenCompose( ignore -> scriptStorageClient.numberOfScriptsForSe(seId) ); }) .whenComplete((numberOfScripts, e) -> { if (e != null) { asyncResponse.resume(e); } else { log.debug("{} Request processed", diagnosticContext); asyncResponse.resume(numberOfScripts); } }); } private CompletionStage<Void> encryptAndStoreAllScripts( final DiagnosticContext diagnosticContext, final String seId, final List<Script> scripts ) { CompletionStage<Void> stage = null; // <- non final field, potential concurrent access bug! for (int i = 0; i < scripts.size(); i++) { final int scriptIndex = i; final Script script = scripts.get(scriptIndex); if (stage == null) { stage = encryptAndStoreSingleScript(diagnosticContext, seId, scriptIndex, script); } else { stage = stage.thenCompose(ignore -> encryptAndStoreSingleScript(diagnosticContext, seId, scriptIndex, script)); } } return stage; } private CompletionStage<Void> encryptAndStoreSingleScript( final DiagnosticContext diagnosticContext, final String seId, final int scriptIndex, final Script script ) { log.debug("{} Encrypting script {}", diagnosticContext, scriptIndex); return secureModuleClient .encrypt(seId, script.getPayload()) .thenCompose( encryptedPayload -> { log.debug("{} Storing encrypted script {}", diagnosticContext, scriptIndex); return scriptStorageClient.storeScript(seId, new Script(encryptedPayload)); } ); } @Override public Set<Object> getSingletons() { return new HashSet<>(Collections.singletonList(this)); } }
  32. 32. Code. Blocking. Submission 1 10 October 2016 32 @POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public long submitScripts(@PathParam("seId") String seId, List<Script> scripts) { MDC.put("flow", "submission"); // ß Setting diagnostic context MDC.put("se", seId); return METRICS.instrument(() -> { // ß Instrumenting with metrics //...
  33. 33. Code. Blocking. Submission 2 10 October 2016 33 log.debug("Processing {} scripts submission", scripts.size(), seId); for (int i = 0; i < scripts.size(); i++) { ß Cycle through the scripts final Script script = scripts.get(i); log.debug("Encrypting {} script", i); final String encryptedPayload = secureModuleClient .encrypt(seId, script.getPayload()); ß Encrypting the script script.setPayload(encryptedPayload); log.debug("Storing encrypted script {}", i); scriptStorageClient.storeScript(seId, script); ß Saving the script into DB } long numberOfScripts = scriptStorageClient.numberOfScriptsForSe(seId); ß Getting current number of scripts in DB log.debug("Request processed", seId); return numberOfScripts;
  34. 34. Code. Non-blocking. Submission 1 10 October 2016 34 @POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public void submitScripts( @PathParam("seId") String seId, List<Script> scripts, @Suspended final AsyncResponse asyncResponse ) { final DiagnosticContext diagnosticContext = new DiagnosticContext("submission", seId); ß Creating diagnostic context METRICS.instrumentStage(() -> { ß Instrumenting with metrics
  35. 35. Code. Non-blocking. Submission 1 10 October 2016 35 @POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public void submitScripts( @PathParam("seId") String seId, List<Script> scripts, @Suspended final AsyncResponse asyncResponse ) { final DiagnosticContext diagnosticContext = new DiagnosticContext("submission", seId); ß Creating diagnostic context METRICS.instrumentStage(() -> { ß Instrumenting with metrics
  36. 36. Code. Non-blocking. Submission 2 10 October 2016 36 METRICS.instrumentStage(() -> { log.debug( "{} Processing {} scripts submission", diagnosticContext, scripts.size()); return encryptAndStoreAllScripts(diagnosticContext, seId, scripts) .thenCompose( ignore -> scriptStorageClient.numberOfScriptsForSe(seId) ); }) .whenComplete((numberOfScripts, e) -> { if (e != null) { asyncResponse.resume(e); } else { log.debug("{} Request processed", diagnosticContext); asyncResponse.resume(numberOfScripts); } });
  37. 37. Code. Non-blocking. Submission 3 10 October 2016 37 private CompletionStage<Void> encryptAndStoreAllScripts( final DiagnosticContext diagnosticContext, final String seId, final List<Script> scripts ) { CompletionStage<Void> stage = null; // <- non final field, potential // concurrent access bug! for (int i = 0; i < scripts.size(); i++) { ß Cycle through the scripts final int scriptIndex = i; final Script script = scripts.get(scriptIndex); if (stage == null) { stage = encryptAndStoreSingleScript( diagnosticContext, seId, scriptIndex, script); } else { stage = stage.thenCompose(ignore -> encryptAndStoreSingleScript( diagnosticContext, seId, scriptIndex, script)); } } return stage; }
  38. 38. Code. Non-blocking. Submission 4 10 October 2016 38 private CompletionStage<Void> encryptAndStoreSingleScript( final DiagnosticContext diagnosticContext, final String seId, final int scriptIndex, final Script script ) { log.debug("{} Encrypting script {}", diagnosticContext, scriptIndex); return secureModuleClient .encrypt(seId, script.getPayload()) ß Encrypting the script .thenCompose( encryptedPayload -> { log.debug( "{} Storing encrypted script {}", diagnosticContext, scriptIndex); return scriptStorageClient.storeScript( ß Saving the script into seId, the DB new Script(encryptedPayload)); } ); }
  39. 39. Code. Blocking. Integration 10 October 2016 39 private final JedisPool pool; public void storeScript(String seId, Script script) { try (Jedis jedis = pool.getResource()) { jedis.rpush(seId, script.getPayload()); } } public long numberOfScriptsForSe(String seId) { try (Jedis jedis = pool.getResource()) { return jedis.llen(seId); } } public Optional<Script> nextScript(String seId) { try (Jedis jedis = pool.getResource()) { return Optional.ofNullable(jedis.lpop(seId)).map(Script::new); } } public String encrypt(String keyDiversifier, String payload) { // ... }
  40. 40. Code. Non-Blocking. Integration 10 October 2016 40 private final RedisAsyncCommands<String, String> commands; public CompletionStage<Void> storeScript(String seId, Script script) { return commands .rpush(seId, script.getPayload()) .thenApply(ignore -> null); } public CompletionStage<Long> numberOfScriptsForSe(String seId) { return commands.llen(seId); } public CompletionStage<Optional<String>> nextScript(String seId) { return commands.lpop(seId).thenApply(Optional::ofNullable); } public CompletionStage<String> encrypt(String keyDiversifier, String payload) { // ... }
  41. 41. Diagnostics in non-blocking systems No clear stack traces, need for good logs Name your threads properly MDC becomes useless (thread locals) Explicitly pass debug context to trace flows Be prepared for debuging non-obvious errors
  42. 42. NIO technology landscape JDK 1.4 NIO JDK 1.7 NIO.2 JAX-RS 2.x Servlet API 3.x
  43. 43. Lessons learned, part 1 Vanila Java 8 for NIO µ-services Netty best for custom protocols in NIO Unit tests should be synchronous Load/stress testing is a must Make bulkheading and plan your resources
  44. 44. Lessons learned, part 2 Functional programming patterns for readability Immutability as 1-st class citizen Scala may be good choice ;-)
  45. 45. Conclusion Non-blocking processing can really save your resources (== money) But! Weight all pros and cons and use non-blocking processing only if you really need it.
  46. 46. Thank you. Michał Baliński m.balinski@oberthur.com Oleksandr Goldobin o.goldobin@oberthur.com goldobin @goldobin balonus @MichalBalinski Readings: • https://github.com/balonus/blocking-vs-nonblocking-demo • C10k Problem, C10M Problem, Asynchronous I/O • Boost application performance using asynchronous I/O (M. Tim Jones, 2006) • Zuul 2 : The Netflix Journey to Asynchronous, Non-Blocking Systems (Netflix, 2016) • Thousands of Threads and Blocking I/O: The Old Way to Write Java Servers Is New Again (and Way Better) (Paul Tyma, 2008) • Why Non-Blocking? (Bozhidar Bozhanov, 2011)

×