The document discusses various architectural patterns for distributed in-memory caching in Kubernetes microservices including embedded, embedded distributed, client-server, cloud, sidecar, reverse proxy, and reverse proxy sidecar patterns. It provides examples of implementing each pattern using the Hazelcast in-memory data grid and summaries of the pros and cons of each approach.

1. Architectural Patterns for
High-Performance Microservices in
Kubernetes
Rafał Leszko
@RafalLeszko
rafalleszko.com
Hazelcast

2. About me
● Cloud-Native Team Lead at Hazelcast
● Worked at Google and CERN
● Author of the book "Continuous Delivery
with Docker and Jenkins"
● Trainer and conference speaker
● Live in Kraków, Poland

3. About Hazelcast
● Distributed Company
● Open Source Software
● 140+ Employees
● Products:
○ Hazelcast IMDG
○ Hazelcast Jet
○ Hazelcast Cloud
@Hazelcast
www.hazelcast.com

4. ● Introduction
● Distributed In-Memory Store Patterns
○ Embedded
○ Embedded Distributed
○ Client-Server
○ Cloud
○ Sidecar
○ Reverse Proxy
○ Reverse Proxy Sidecar
● Summary
Agenda

5. Introduction

6. High-Performance Microservices
● Systems have tons of microservices
○ Netflix has over 1000!
● High Performance allows for
complex use-cases, e.g.
○ AI processing
○ Mainframe optimization
○ ...
● Cloud cost optimization

7. High Performance is no longer
nice-to-have

8. High Performance is a must!

9. Service 1
Service 2
v1
Service 2
v2
Service 3
Service 4
v1
Service 4
v2
Service 4
v3
Ruby
Microservice World

10. Microservice World
Service 1
Service 2
v1
Service 2
v2
Service 1
Service 4
v1
Service 4
v2
Service 4
v3
Ruby
library
library
library
library

11. Microservice World
Service 1
Service 2
v1
Service 2
v2
Service 3
Service 4
v1
Service 4
v2
Service 4
v3
Ruby
in-mem DB
in-mem DB
in-mem DB

12. Microservice World
Service 1
Service 2
v1
Service 2
v2
Service 3
Service 4
v1
Service 4
v2
Service 4
v3
Ruby
reverse
proxy
reverse
proxy
reverse
proxy

13. ● Introduction ✔
● Distributed In-Memory Store Patterns
○ Embedded
○ Embedded Distributed
○ Client-Server
○ Cloud
○ Sidecar
○ Reverse Proxy
○ Reverse Proxy Sidecar
● Summary
Agenda

14. 1. Embedded

15. Application
Kubernetes
Service
library
Application
library
Request
Embedded
Kubernetes Pod
Kubernetes Pod

16. private ConcurrentHashMap<String, String> cache =
new ConcurrentHashMap<>();
private String processRequest(String request) {
if (cache.contains(request)) {
return cache.get(request);
}
String response = process(request);
cache.put(request, response);
return response;
}
Embedded Cache (Pure Java implementation)

17. private ConcurrentHashMap<String, String> cache =
new ConcurrentHashMap<>();
private String processRequest(String request) {
if (cache.contains(request)) {
return cache.get(request);
}
String response = process(request);
cache.put(request, response);
return response;
}
Embedded Cache (Pure Java implementation)

18. ● No Eviction Policies
● No Max Size Limit
(OutOfMemoryError)
● No Statistics
● No built-in Cache Loaders
● No Expiration Time
● No Notification Mechanism
Java Collection is not a Cache!

19. CacheBuilder.newBuilder()
.initialCapacity(300)
.expireAfterAccess(Duration.
ofMinutes(10))
.maximumSize(1000)
.build();
Embedded Cache (Java libraries)

20. Embedded Cache (Java libraries)
CacheBuilder.newBuilder()
.initialCapacity(300)
.expireAfterAccess(Duration.
ofMinutes(10))
.maximumSize(1000)
.build();

21. Caching Application Layer
@Service
public class BookService {
@Cacheable("books")
public String getBookNameByIsbn(String isbn) {
return findBookInSlowSource(isbn);
}
}

22. Caching Application Layer
@Service
public class BookService {
@Cacheable("books")
public String getBookNameByIsbn(String isbn) {
return findBookInSlowSource(isbn);
}
}
Be Careful, Spring uses ConcurrentHashMap by default!

23. Application
Kubernetes
Service
library
Application
library
Request
Embedded
Kubernetes Pod
Kubernetes Pod

24. 1*. Embedded Distributed

25. Application
Application
Kubernetes
Service
library
library
Request
Hazelcast
Cluster
Embedded Distributed
Kubernetes Pod
Kubernetes Pod

26. @Configuration
public class HazelcastConfiguration {
@Bean
CacheManager cacheManager() {
return new HazelcastCacheManager(
Hazelcast.
newHazelcastInstance());
}
}
Embedded Distributed Cache (Spring with Hazelcast)

27. Hazelcast Discovery Plugins

28. Hazelcast Discovery Plugins

29. Application
Application
Kubernetes
Service
library
library
Request
Hazelcast
Cluster
Embedded Distributed
Kubernetes Pod
Kubernetes Pod

30. Embedded (Distributed)
Pros Cons
● Simple configuration /
deployment
● Low-latency data access
● No separate Ops Team
needed
● Not flexible management
(scaling, backup)
● Limited to JVM-based
applications
● Data collocated with
applications

31. ● Introduction ✔
● Distributed In-Memory Store Patterns
○ Embedded ✔
○ Embedded Distributed ✔
○ Client-Server
○ Cloud
○ Sidecar
○ Reverse Proxy
○ Reverse Proxy Sidecar
● Summary
Agenda

32. 2. Client-Server

33. Application
Kubernetes
Service
Application
Request
In-Memory Store /
Computing Platform
Client-Server
Kubernetes Pod
Kubernetes Pod

34. Client-Server
Application
Kubernetes
Service
Application
Request
In-Memory Store /
Computing Platform
Kubernetes Pod
Kubernetes Pod

35. Client-Server
Separate Management:
● backups
● (auto) scaling
● security
Ops Team
Application
Kubernetes
Service
Application
Request
In-Memory Store /
Computing Platform
Kubernetes Pod
Kubernetes Pod

36. Client-Server
Application
Kubernetes
Service
Application
Request
In-Memory Store /
Computing Platform
Kubernetes Pod
Kubernetes Pod

37. Client-Server
Application
Kubernetes
Service
Application
Request
In-Memory Store /
Computing Platform
Kubernetes Pod
Kubernetes Pod

38. Client-Server

39. Client-Server
Starting Hazelcast Server
$ helm install hazelcast/hazelcast

40. Client-Server
Hazelcast Client
@Configuration
public class HazelcastClientConfiguration {
@Bean
CacheManager cacheManager() {
return new HazelcastCacheManager(HazelcastClient
.
newHazelcastClient());
}
}
Starting Hazelcast Server
$ helm install hazelcast/hazelcast

41. Client-Server
Hazelcast Client
HazelcastInstance instance = HazelcastCacheManager(
HazelcastClient.
newHazelcastClient());
IMap<String, String> map = instance.getMap("my-map");
// Use in-memory storage
map.put("key", "value");
// Use in-memory computing
map.executeOnKeys(...);
// Use stream processing, etc.
Starting Hazelcast Server
$ helm install hazelcast/hazelcast

42. Client-Server
Separate Management:
● backups
● (auto) scaling
● security
Ops Team
Application
Kubernetes
Service
Application
Request
In-Memory Store /
Computing Platform
Kubernetes Pod
Kubernetes Pod

43. 2*. Cloud

44. Application
Kubernetes
Service
Application
Request
Cloud (Hazelcast as a Service)
Kubernetes Pod
Kubernetes Pod

45. Cloud (Hazelcast as a Service)
Management:
● backups
● (auto) scaling
● security
Ops Team
Application
Kubernetes
Service
Application
Request
Kubernetes Pod
Kubernetes Pod

46. Cloud (Hazelcast as a Service)
Management:
● backups
● (auto) scaling
● security
Ops Team
Application
Kubernetes
Service
Application
Request
Kubernetes Pod
Kubernetes Pod

47. Cloud (Hazelcast as a Service)
Application
Kubernetes
Service
Application
Request
Kubernetes Pod
Kubernetes Pod

48. ClientConfig clientConfig = new ClientConfig();
clientConfig.getNetworkConfig().getCloudConfig()
.setEnabled(
true)
.setDiscoveryToken(
"KSXFDTi5HXPJGR0wRAjLgKe45tvEEhd");
clientConfig.setClusterName(
"test-cluster");
HazelcastInstance hazelcast =
HazelcastClient.
newHazelcastClient(clientConfig));
Cloud (Hazelcast as a Service)

49. Client-Server (Cloud)
Pros
● Data separate from
applications
● Separate management
(scaling, backup)
● Programming-language
agnostic
Cons
● Separate Ops effort
● Higher latency
● Server network requires
adjustment (same region,
same VPC)

50. ● Introduction ✔
● Distributed In-Memory Store Patterns
○ Embedded ✔
○ Embedded Distributed ✔
○ Client-Server ✔
○ Cloud ✔
○ Sidecar
○ Reverse Proxy
○ Reverse Proxy Sidecar
● Summary
Agenda

51. 3. Sidecar

52. Kubernetes Service
Request
Hazelcast
Cluster
Kubernetes POD
Application Container
In-Mem DB Container
Application Container
In-Mem DB Container
Kubernetes POD
Sidecar

53. Sidecar
Similar to Embedded:
● the same physical machine
● the same resource pool
● scales up and down together
● no discovery needed (always localhost)
Similar to Client-Server:
● different programming language
● uses client to connect
● some isolation between app and server

54. ClientConfig clientConfig = new ClientConfig();
clientConfig.getNetworkConfig()
.addAddress(
"localhost:5701");
HazelcastInstance hazelcast = HazelcastClient
.newHazelcastClient(clientConfig));
Sidecar

55. Sidecar Cache
apiVersion: apps/v1
kind: Deployment
...
spec:
template:
spec:
containers:
- name: application
image: leszko/application
- name: hazelcast
image: hazelcast/hazelcast

56. Sidecar
Pros Cons
● Simple configuration
● Programming-language
agnostic
● Low latency
● Some isolation of data and
applications
● Not flexible management
(scaling, backup)
● Data collocated with
application PODs

57. ● Introduction ✔
● Distributed In-Memory Store Patterns
○ Embedded ✔
○ Embedded Distributed ✔
○ Client-Server ✔
○ Cloud ✔
○ Sidecar ✔
○ Reverse Proxy
○ Reverse Proxy Sidecar
● Summary
Agenda

58. 4. Reverse Proxy

59. Application
Kubernetes
Service
Cache
Application
Request
Reverse Proxy
Kubernetes Pod
Kubernetes Pod

60. Reverse Proxy
http {
...
proxy_cache_path /data/nginx/cache
keys_zone=one:10m;
...
}

61. ● Only for HTTP
● Not distributed
● No High Availability
● Data stored on the disk
NGINX Reverse Proxy Issues

62. 4*. Reverse Proxy Sidecar

63. Kubernetes Service
Request
Hazelcast
Cluster
Kubernetes POD
Application Container
Reverse Proxy
Container
Application Container
Reverse Proxy
Container
Kubernetes POD
Reverse Proxy Sidecar

65. Good

66. Reverse Proxy Sidecar
Service 1
Service 2
v1
Service 2
v2
Service 3
Service 4
v1
Service 4
v2
Service 4
v3
Ruby

67. Reverse Proxy Sidecar
Service 1
Service 2
v1
Service 2
v2
Service 3
Service 4
v1
Service 4
v2
Service 4
v3
Ruby

68. Reverse Proxy Sidecar
apiVersion: apps/v1
kind: Deployment
...
spec:
template:
spec:
initContainers:
- name: init-networking
image: leszko/init-networking
containers:
- name: caching-proxy
image: leszko/caching-proxy
- name: application
image: leszko/application

69. Reverse Proxy Sidecar
Service 1
Service 2
v1
Service 2
v2
Service 3
Service 4
v1
Service 4
v2
Service 4
v3
Ruby

71. Bad

73. Reverse Proxy
@CacheEvict(value = "someValue", allEntries = true)
public void evictAllCacheValues() {}
Application Cache:

74. Reverse Proxy
@CacheEvict(value = "someValue", allEntries = true)
public void evictAllCacheValues() {}
Application Cache:
Proxy Cache:
http {
...
location / {
add_header Cache-Control public;
expires 86400;
etag on;
}
}

75. Reverse Proxy (Sidecar)
Pros Cons
● Configuration-based (no
need to change
applications)
● Programming-language
agnostic
● Consistent with containers
and microservice world
● Difficult cache invalidation
● No mature solutions yet
● Protocol-based (e.g. works
only with HTTP)

76. ● Introduction ✔
● Distributed In-Memory Store Patterns
○ Embedded ✔
○ Embedded Distributed ✔
○ Client-Server ✔
○ Cloud ✔
○ Sidecar ✔
○ Reverse Proxy ✔
○ Reverse Proxy Sidecar ✔
● Summary
Agenda

77. Summary

78. application-aware?

79. application-aware?
early adopter?
no

80. application-aware?
Reverse Proxy
no
no
early adopter?

81. application-aware?
Reverse Proxy
Reverse Proxy
Sidecar
no
yes no
early adopter?

82. application-aware?
Reverse Proxy
Reverse Proxy
Sidecar
lot of data?
security restrictions?
yes no
yes no
early adopter?

83. application-aware?
Reverse Proxy
Reverse Proxy
Sidecar
lot of data?
security restrictions?
language-agnostic?
yes no
yes no
no
early adopter?

84. application-aware?
Reverse Proxy
Reverse Proxy
Sidecar
lot of data?
security restrictions?
language-agnostic?
Embedded
(Distributed)
yes no
yes no
no
no
early adopter?

85. application-aware?
Reverse Proxy
Reverse Proxy
Sidecar
lot of data?
security restrictions?
language-agnostic?
Embedded
(Distributed)
Sidecar
yes no
yes
yes no
no
no
early adopter?

86. application-aware?
Reverse Proxy
Reverse Proxy
Sidecar
lot of data?
security restrictions?
language-agnostic?
Embedded
(Distributed)
Sidecar
cloud?
yes no
yes
yes
yes no
no
no
early adopter?

87. application-aware?
Reverse Proxy
Reverse Proxy
Sidecar
lot of data?
security restrictions?
language-agnostic?
Embedded
(Distributed)
Sidecar
cloud?
Client-Server
yes no
yes
yes
yes no
no
no
no
early adopter?

88. application-aware?
Reverse Proxy
Reverse Proxy
Sidecar
lot of data?
security restrictions?
language-agnostic?
Embedded
(Distributed)
Sidecar
cloud?
Client-Server
Cloud
yes no
yes
yes yes
yes no
no
no
no
early adopter?

89. Resources
● Code for this talk:
https://github.com/leszko/caching-patterns
● Hazelcast Sidecar Container Pattern:
https://hazelcast.com/blog/hazelcast-sidecar-container-pattern/
● Hazelcast Reverse Proxy Sidecar Prototype:
https://github.com/leszko/caching-injector
● NGINX HTTP Reverse Proxy Caching:
https://www.nginx.com/resources/videos/best-practices-for-caching/

90. Thank You!
Rafał Leszko
@RafalLeszko