Download to read offline
Uploaded byMichal Balinski
Practical non blocking microservices in java 8
The document discusses the implementation and benefits of non-blocking microservices in Java 8, particularly in the context of an Over-The-Air (OTA) gateway for IoT environments. It contrasts blocking and non-blocking I/O approaches, demonstrating the scalability and resource efficiency of non-blocking systems through various use cases and coding examples. Additionally, it highlights potential challenges, best practices, and lessons learned in developing non-blocking architectures for microservices.
![[BlackHat USA 2016] Nonce-Disrespecting Adversaries: Practical Forgery Attack...](https://cdn.slidesharecdn.com/ss_thumbnails/us-16-devlin-nonce-disrespecting-adversaries-practical-forgery-attacks-on-gcm-in-tls-170329234029-thumbnail.jpg?width=640&height=640&fit=bounds)
![[CB20] Vulnerabilities of Machine Learning Infrastructure by Sergey Gordeychik](https://cdn.slidesharecdn.com/ss_thumbnails/sergeygordeychikcb20-210112133639-thumbnail.jpg?width=640&height=640&fit=bounds)
Practical non blocking microservices in java 8
- 1. Practical non-blocking microservices in Java8 Michał Baliński System Architect Oleksandr Goldobin System Architect
- 2. Cloud of microservicesfor secure IoT gateway backing service core service gateway gateway core service backing service
- 3. The demand andcharacteristics of our domain Crowded IoT environment Slow, long lived connections Big amount of concurrent connections Scalability and resilience Rather I/O intensive than CPU intensive
- 4. OTA Gateway –Use Case TSM Trusted Service Manager
- 5. OTA Gateway –Use Case TSM Trusted Service Manager Security Module
- 6. OTA Gateway –Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB
- 7. OTA Gateway –Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts
- 8. OTA Gateway –Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts
- 9. OTA Gateway –Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts TCP 3. store scripts
- 10. OTA Gateway –Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts TCP 4. submition response 3. store scripts
- 11. OTA Gateway –Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts TCP 4. submition response HTTP 5. poll scripts 3. store scripts
- 12. OTA Gateway –Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts TCP 3. store scripts 4. submition response HTTP 5. poll scripts 6. search scripts
- 13. OTA Gateway –Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP 1. submit scripts HTTP 2. encrypt scripts TCP 3. store scripts 4. submition response HTTP 5. poll scripts 6. search scripts 7. response with scripts
- 14. OTA Gateway –Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB HTTP HTTP HTTP TCP
- 15. OTA Gateway –Use Case OTA (Over-the-Air) Gateway TSM Trusted Service Manager Security Module DB Log Storage HTTP HTTP HTTP TCP File I/OMonitoring System HTTP
- 16. Let’s test blockingapproach!
- 17. OTA Gateway Blocking- technologies 25 February 2017 17 TSM DB Log Storage HTTP TCP STDOUT OTA Gateway JAX-RS Logback appender Security Module JAX-RS Jedis JAX-RS Client
- 18. OTA Gateway –Blocking - test OTA (Over-the-Air) Gateway send 2 scripts per request Security Module DB Log Storage HTTP HTTP HTTP TCP File I/O emulated latency 200 ms expected latency < 450 ms verified with throughput over 7k req/s
- 19. Blocking – 1kthreads OTA (Over-the-Air) Gateway send 2 scripts per request Security Module DB Log Storage HTTP HTTP HTTP TCP File I/O emulated latency 200 ms max 1000 connections max 1000 threads expected latency < 450 ms
- 20. Blocking – 1kthreads 500 req/s 1000 req/s 1500 req/s 2000 req/s
- 21. The drawbacks ofclassic synchronous I/O One thread per connection Threads waiting instead of running Context switches Resource wasting (~1 MB per thread - 64bit)
- 22. Let’s switch fromblocking to non-blocking
- 23. OTA Gateway Non-blocking- technologies 25 February 2017 23 TSM DB Log Storage HTTP TCP STDOUT OTA Gateway JAX-RS 2.0 Logback async appender Async Http Client 2 (Netty) Security Module JAX-RS 2.0 Lettuce (Netty)
- 24. Non-blocking – 16threads OTA (Over-the-Air) Gateway send 2 scripts per request Security Module DB Log Storage HTTP HTTP HTTP TCP File I/O emulated latency 200 ms no limit for connections max 16 threads expected latency < 450 ms
- 25. Non-blocking – 16threads 1k req/s 2k req/s 2.5k req/s 3k req/s
- 26. Blocking Non-blocking 1000 threads56 threads 1.2 GB 0.5 GB ~1.2k r/s 2.5k r/s
- 27. Let’s talk aboutchallenges
- 28. OTA Gateway Blocking– sequence diagram OTA Gateway TSM Security Module DB Logs loop 1. submit scripts 2. encrypt script 3a. store script 4. submition response 3b. count scripts
- 29. OTA Gateway Non-blocking– sequence diagram OTA Gateway TSM Security Module DB Logs loop 1. submit scripts 2. encrypt script 3a. store script 4. submition response 3b. count scripts
- 30. OTA Non-blocking –realityOTA Gateway TSM Security Module DB Logs „loopedprocessingchain” 1. submit scripts 4. submition response 3b. count scripts HTTP Server 2. encrypt script 3a. store script Logging DB Client Security Client
- 31. Code. Bird view 3125February 2017 package org.demo.ota.blocking.rest; @Path("se") public class ScriptSubmissionResource extends Application { private static final Logger log = LoggerFactory.getLogger(ScriptSubmissionResource.class); private static final ResourceMetrics METRICS = new ResourceMetrics("ota_submission"); private SecureModuleClient secureModuleClient = SecureModuleClient.instance(); private ScriptStorageClient scriptStorageClient = ScriptStorageClient.instance(); @POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public long submitScripts(@PathParam("seId") String seId, List<Script> scripts) { MDC.put("flow", "submission"); MDC.put("se", seId); return METRICS.instrument(() -> { log.debug("Processing {} scripts submission", scripts.size(), seId); for (int i = 0; i < scripts.size(); i++) { final Script script = scripts.get(i); log.debug("Encrypting {} script", i); final String encryptedPayload = secureModuleClient.encrypt(seId, script.getPayload()); script.setPayload(encryptedPayload); log.debug("Storing encrypted script {}", i); scriptStorageClient.storeScript(seId, script); } long numberOfScripts = scriptStorageClient.numberOfScriptsForSe(seId); log.debug("Request processed", seId); return numberOfScripts; }); } @Override public Set<Object> getSingletons() { return Collections.singleton(this); } } package org.demo.ota.nonblocking.rest; @Path("se") public class ScriptSubmissionResource extends Application { private static final Logger log = LoggerFactory.getLogger(ScriptSubmissionResource.class); private static final ResourceMetrics METRICS = new ResourceMetrics("ota_submission"); private SecureModuleClient secureModuleClient = SecureModuleClient.instance(); private ScriptStorageClient scriptStorageClient = ScriptStorageClient.instance(); @POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public void submitScripts( @PathParam("seId") String seId, List<Script> scripts, @Suspended final AsyncResponse asyncResponse ) { final DiagnosticContext diagnosticContext = new DiagnosticContext("submission", seId); METRICS.instrumentStage(() -> { log.debug("{} Processing {} scripts submission", diagnosticContext, scripts.size()); return encryptAndStoreAllScripts(diagnosticContext, seId, scripts) .thenCompose( ignore -> scriptStorageClient.numberOfScriptsForSe(seId) ); }) .whenComplete((numberOfScripts, e) -> { if (e != null) { asyncResponse.resume(e); } else { log.debug("{} Request processed", diagnosticContext); asyncResponse.resume(numberOfScripts); } }); } private CompletionStage<Void> encryptAndStoreAllScripts( final DiagnosticContext diagnosticContext, final String seId, final List<Script> scripts ) { CompletionStage<Void> stage = null; // <- non final field, potential concurrent access bug! for (int i = 0; i < scripts.size(); i++) { final int scriptIndex = i; final Script script = scripts.get(scriptIndex); if (stage == null) { stage = encryptAndStoreSingleScript(diagnosticContext, seId, scriptIndex, script); } else { stage = stage.thenCompose(ignore -> encryptAndStoreSingleScript(diagnosticContext, seId, scriptIndex, script)); } } return stage; } private CompletionStage<Void> encryptAndStoreSingleScript( final DiagnosticContext diagnosticContext, final String seId, final int scriptIndex, final Script script ) { log.debug("{} Encrypting script {}", diagnosticContext, scriptIndex); return secureModuleClient .encrypt(seId, script.getPayload()) .thenCompose( encryptedPayload -> { log.debug("{} Storing encrypted script {}", diagnosticContext, scriptIndex); return scriptStorageClient.storeScript(seId, new Script(encryptedPayload)); } ); } @Override public Set<Object> getSingletons() { return new HashSet<>(Collections.singletonList(this)); } }
- 32. Code. Blocking. Submission1 25 February 2017 32 @POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public long submitScripts(@PathParam("seId") String seId, List<Script> scripts) { MDC.put("flow", "submission"); // Setting diagnostic context MDC.put("se", seId); return METRICS.instrument(() -> { // Instrumenting with metrics //...
- 33. Code. Blocking. Submission2 25 February 2017 33 log.debug("Processing {} scripts submission", scripts.size(), seId); for (int i = 0; i < scripts.size(); i++) { Cycle through the scripts final Script script = scripts.get(i); log.debug("Encrypting {} script", i); final String encryptedPayload = secureModuleClient .encrypt(seId, script.getPayload()); Encrypting the script script.setPayload(encryptedPayload); log.debug("Storing encrypted script {}", i); scriptStorageClient.storeScript(seId, script); Saving the script into DB } long numberOfScripts = scriptStorageClient.numberOfScriptsForSe(seId); Getting current number of scripts in DB log.debug("Request processed", seId); return numberOfScripts;
- 34. Code. Non-blocking. Submission1 25 February 2017 34 @POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public void submitScripts( @PathParam("seId") String seId, List<Script> scripts, @Suspended final AsyncResponse asyncResponse ) { final DiagnosticContext diagnosticContext = new DiagnosticContext("submission", seId); Creating diagnostic context METRICS.instrumentStage(() -> { Instrumenting with metrics
- 35. Code. Non-blocking. Submission1 25 February 2017 35 @POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public void submitScripts( @PathParam("seId") String seId, List<Script> scripts, @Suspended final AsyncResponse asyncResponse ) { final DiagnosticContext diagnosticContext = new DiagnosticContext("submission", seId); Creating diagnostic context METRICS.instrumentStage(() -> { Instrumenting with metrics
- 36. Code. Non-blocking. Submission2 25 February 2017 36 METRICS.instrumentStage(() -> { log.debug( "{} Processing {} scripts submission", diagnosticContext, scripts.size()); return encryptAndStoreAllScripts(diagnosticContext, seId, scripts) .thenCompose( ignore -> scriptStorageClient.numberOfScriptsForSe(seId) ); }) .whenComplete((numberOfScripts, e) -> { if (e != null) { asyncResponse.resume(e); } else { log.debug("{} Request processed", diagnosticContext); asyncResponse.resume(numberOfScripts); } });
- 37. Code. Non-blocking. Submission3 25 February 2017 37 private CompletionStage<Void> encryptAndStoreAllScripts( final DiagnosticContext diagnosticContext, final String seId, final List<Script> scripts ) { CompletionStage<Void> stage = null; // <- non final field, potential // concurrent access bug! for (int i = 0; i < scripts.size(); i++) { Cycle through the scripts final int scriptIndex = i; final Script script = scripts.get(scriptIndex); if (stage == null) { stage = encryptAndStoreSingleScript( diagnosticContext, seId, scriptIndex, script); } else { stage = stage.thenCompose(ignore -> encryptAndStoreSingleScript( diagnosticContext, seId, scriptIndex, script)); } } return stage; }
- 38. Code. Non-blocking. Submission4 25 February 2017 38 private CompletionStage<Void> encryptAndStoreSingleScript( final DiagnosticContext diagnosticContext, final String seId, final int scriptIndex, final Script script ) { log.debug("{} Encrypting script {}", diagnosticContext, scriptIndex); return secureModuleClient .encrypt(seId, script.getPayload()) Encrypting the script .thenCompose( encryptedPayload -> { log.debug( "{} Storing encrypted script {}", diagnosticContext, scriptIndex); return scriptStorageClient.storeScript( Saving the script into seId, the DB new Script(encryptedPayload)); } ); }
- 39. Code. Blocking. Integration 25February 2017 39 private final JedisPool pool; public void storeScript(String seId, Script script) { try (Jedis jedis = pool.getResource()) { jedis.rpush(seId, script.getPayload()); } } public long numberOfScriptsForSe(String seId) { try (Jedis jedis = pool.getResource()) { return jedis.llen(seId); } } public Optional<Script> nextScript(String seId) { try (Jedis jedis = pool.getResource()) { return Optional.ofNullable(jedis.lpop(seId)).map(Script::new); } } public String encrypt(String keyDiversifier, String payload) { // ... }
- 40. Code. Non-Blocking. Integration 25February 2017 40 private final RedisAsyncCommands<String, String> commands; public CompletionStage<Void> storeScript(String seId, Script script) { return commands .rpush(seId, script.getPayload()) .thenApply(ignore -> null); } public CompletionStage<Long> numberOfScriptsForSe(String seId) { return commands.llen(seId); } public CompletionStage<Optional<String>> nextScript(String seId) { return commands.lpop(seId).thenApply(Optional::ofNullable); } public CompletionStage<String> encrypt(String keyDiversifier, String payload) { // ... }
- 41. Diagnostics in non-blockingsystems No clear stack traces, need for good logs Name your threads properly MDC becomes useless (thread locals) Explicitly pass debug context to trace flows Be prepared for debuging non-obvious errors
- 42. NIO technology landscape JDK1.4 NIO JDK 1.7 NIO.2 JAX-RS 2.x Servlet API 3.x
- 43. Lessons learned, part1 Vanila Java 8 for NIO µ-services Netty best for custom protocols in NIO Unit tests should be synchronous Load/stress testing is a must Make bulkheading and plan your resources
- 44. Lessons learned, part2 Functional programming patterns for readability Immutability as 1-st class citizen Scala may be good choice ;-)
- 45. Conclusion Non-blocking processing canreally save your resources (== money) But! Weight all pros and cons and use non-blocking processing only if you really need it.
- 46. Thank you. Michał Baliński m.balinski@oberthur.com OleksandrGoldobin o.goldobin@oberthur.com goldobin @goldobin balonus @MichalBalinski Readings: • https://github.com/balonus/blocking-vs-nonblocking-demo • C10k Problem, C10M Problem, Asynchronous I/O • Boost application performance using asynchronous I/O (M. Tim Jones, 2006) • Zuul 2 : The Netflix Journey to Asynchronous, Non-Blocking Systems (Netflix, 2016) • Thousands of Threads and Blocking I/O: The Old Way to Write Java Servers Is New Again (and Way Better) (Paul Tyma, 2008) • Why Non-Blocking? (Bozhidar Bozhanov, 2011)
Editor's Notes
- #3 Michał
- #4 Alex Our products are operating in pretty "crowded" environment The clients of our services are: from the one site: service providers like MNO or Banks from the other site: crowds of devices like secure elements in mobile and IoT devices those devices maintaining SLOOOW, long living connections The demand of the market is to process pretty big amount of concurrent connections in a secure way We also should be scalable and resilient -> micro-services architecture The microservices in IT are generally performing some kind of gateway role: receive request request other web services/database for some data synchronize and process responses respond (or not) to the caller So primarily they are IO intensive rather then CPU intensive The most CPU intensive part is actually marshalling/unmarshaling of requests/responses
- #5 Michał
- #6 Michał
- #7 Michał
- #8 Michał
- #9 Michał
- #10 Michał
- #11 Michał
- #12 Michał
- #13 Michał
- #14 Michał
- #15 Michał
- #16 Alex
- #18 Alex
- #19 Alex
- #20 Alex
- #21 Michał
- #22 Alex TODO reconsider order of slides Your are forced to have one thread per connection Those threads actually waiting for the data instead of perform real work This leads to big amount of context switches And also this leads to resource wasting because threads are expensive For example you can't create more than 4000 threads (on the latest Mac Book Pro -> OOM ) Christopher Batey - $8 per year on AWS EC2
- #23 Alex
- #24 Alex
- #25 Alex
- #26 Alex
- #27 Michał
- #28 TODO kick out?
- #29 Michał
- #30 Michał
- #31 Michał
- #32 Alex
- #42 Alex It is harder to debug: no clear stack traces; you're more relying on logs; It is harder to trace business flows: MDC becomes useless (thread locals); To trace business flows you need to pass (or close on) some immutable (!!!) debugging context;
- #43 Michał Servers: JaxRS 2.x implementation like RestEasy 3.x with @Suspended annotation support Netty Akka (IO, Http) – different story, still better used with Scala (sorry ) Clients: Async Http Client 2.0 Latest database clients: Lettuce for Redis Cassandra Mongo DB JDBC drivers by nature blocking :( Akka (IO, Http)
- #44 Alex + Michał (last one) JDK 8 + some libraries like Netty (vanilla Java) are sufficient to implement non-blocking processing and build micro-services (no additional huge frameworks really required) Async code can be readable but it requires to apply functional programming patterns which are not common for classic approach As soon as you're using several threads to process your flows immutability becomes 1-st class citizen Unit tests should be synchronous -- no concurrency at all or fully synchronized on test's execution thread (which can be challenging sometimes) Sometimes you need to develop your own tools to make tests locking like scenarios and not as complex asynchronous code; Load/stress testing is a must - you really need to stress your system to check that it behaves properly
- #45 Michał + Alex (last one) Do not use MDC use some context passing technique (like we showed in example); Name your threads properly: it will help you to investigate/profile issues; Make bulkheading and plan your resources utilization in advance, adjust configuration while performing load/stress tests; Be prepared to solve some non-obvious errors (like direct memory OOM or virtual memory OOM); Implementing graceful shutdown is tricky, design it from the beginning; Netty is a best library to implement custom protocols in non-blocking way in Java; Technically Scala still provides better toolset for non-blocking asynchronous data processing (standard library and/or Akka) -- but it is different topic ;)
- #46 Alex