Talk at In-Memory Computing Summit North America 2019

1. Where is my cache?
Architectural patterns for caching
microservices by example
Rafał Leszko
Cloud Software Engineer
Hazelcast

2. About me
● Cloud Software Engineer at Hazelcast
● Worked at Google and CERN
● Author of the book "Continuous Delivery with
Docker and Jenkins"
● Trainer and conference speaker
● Live in Kraków, Poland

3. About Hazelcast
● Distributed Company
● Open Source Software
● 140+ Employees
● Products:
○ Hazelcast IMDG
○ Hazelcast Jet
○ Hazelcast Cloud
@Hazelcast
www.hazelcast.com

4. ● Introduction
● Caching Architectural Patterns
○ Embedded
○ Embedded Distributed
○ Client-Server
○ Cloud
○ Sidecar
○ Reverse Proxy
○ Reverse Proxy Sidecar
● Summary
Agenda

5. Why Caching?
● Performance
○ Decrease latency
○ Reduce load
● Resilience
○ High availability
○ Lower downtime

6. Service 1
Service 2 v1
Service 2 v2
Service 1
Service 4 v1
Service 4 v2
Service 4 v3
Ruby
Microservice World

7. Service 1
Service 2 v1
Service 2 v2
Service 1
Service 4 v1
Service 4 v2
Service 4 v3
Ruby
Microservice World
cache
cache
cache
cache

8. Service 1
Service 2 v1
Service 2 v2
Service 1
Service 4 v1
Service 4 v2
Service 4 v3
Ruby
Microservice World
cache
cache
cache

9. Service 1
Service 2 v1
Service 2 v2
Service 1
Service 4 v1
Service 4 v2
Service 4 v3
Ruby
Microservice World
cache
cache
cache

10. ● Introduction ✔
● Caching Architectural Patterns
○ Embedded
○ Embedded Distributed
○ Client-Server
○ Cloud
○ Sidecar
○ Reverse Proxy
○ Reverse Proxy Sidecar
● Summary
Agenda

11. 1. Embedded

12. Application
Load Balancer
Cache
Application
Cache
Request
Embedded Cache

13. private ConcurrentHashMap<String, String> cache =
new ConcurrentHashMap<>();
private String processRequest(String request) {
if (cache.contains(request)) {
return cache.get(request);
}
String response = process(request);
cache.put(request, response);
return response;
}
Embedded Cache (Pure Java implementation)

14. private ConcurrentHashMap<String, String> cache =
new ConcurrentHashMap<>();
private String processRequest(String request) {
if (cache.contains(request)) {
return cache.get(request);
}
String response = process(request);
cache.put(request, response);
return response;
}
Embedded Cache (Pure Java implementation)

15. ● No Eviction Policies
● No Max Size Limit
(OutOfMemoryError)
● No Statistics
● No built-in Cache Loaders
● No Expiration Time
● No Notification Mechanism
Java Collection is not a Cache!

16. CacheBuilder.newBuilder()
.initialCapacity(300)
.expireAfterAccess(Duration.ofMinutes(10))
.maximumSize(1000)
.build();
Embedded Cache (Java libraries)

17. Embedded Cache (Java libraries)
CacheBuilder.newBuilder()
.initialCapacity(300)
.expireAfterAccess(Duration.ofMinutes(10))
.maximumSize(1000)
.build();

18. Caching Application Layer
@Service
public class BookService {
@Cacheable("books")
public String getBookNameByIsbn(String isbn) {
return findBookInSlowSource(isbn);
}
}

19. Caching Application Layer
@Service
public class BookService {
@Cacheable("books")
public String getBookNameByIsbn(String isbn) {
return findBookInSlowSource(isbn);
}
}
Be Careful, Spring uses ConcurrentHashMap by default!

20. Application
Load Balancer
Cache
Application
Cache
Request
Embedded Cache

21. 1*. Embedded Distributed

22. Application
Application
Load Balancer
Cache
Cache
Request
Hazelcast
Cluster
Embedded Distributed Cache

23. @Configuration
public class HazelcastConfiguration {
@Bean
CacheManager cacheManager() {
return new HazelcastCacheManager(
Hazelcast.newHazelcastInstance());
}
}
Embedded Distributed Cache (Spring with Hazelcast)

24. DEMO

25. Hazelcast Discovery Plugins

26. Hazelcast Discovery Plugins

27. Hazelcast Discovery Plugins

28. Application
Application
Load Balancer
Cache
Cache
Request
Embedded Distributed Cache
Hazelcast
Cluster

29. Embedded Cache
Pros Cons
● Simple configuration /
deployment
● Low-latency data access
● No separate Ops Team needed
● Not flexible management (scaling,
backup)
● Limited to JVM-based
applications
● Data collocated with applications

30. ● Introduction ✔
● Caching Architectural Patterns
○ Embedded ✔
○ Embedded Distributed ✔
○ Client-Server
○ Cloud
○ Sidecar
○ Reverse Proxy
○ Reverse Proxy Sidecar
● Summary
Agenda

31. 2. Client-Server

32. Application
Load Balancer
Application
Request
Cache Server
Client-Server Cache

33. Application
Load Balancer
Application
Request
Cache Server
Client-Server Cache

34. Application
Load Balancer
Application
Request
Cache Server
Client-Server Cache
Separate Management:
● backups
● (auto) scaling
● security
Ops Team

35. Application
Load Balancer
Application
Request
Cache Server
Client-Server Cache

36. Application
Load Balancer
Application
Request
Cache Server
Client-Server Cache

37. Client-Server Cache

38. Client-Server Cache

39. Client-Server Cache
$ ./start.sh
Starting Hazelcast Cache Server (standalone)

40. Client-Server Cache
Starting Hazelcast Cache Server (Kubernetes)
$ helm install hazelcast/hazelcast

41. Client-Server Cache
Hazelcast Client (Kubernetes):
@Configuration
public class HazelcastClientConfiguration {
@Bean
CacheManager cacheManager() {
ClientConfig clientConfig = new ClientConfig();
clientConfig.getNetworkConfig().getKubernetesConfig()
.setEnabled(true);
return new HazelcastCacheManager(HazelcastClient
.newHazelcastClient(clientConfig));
}
}
Starting Hazelcast Cache Server (Kubernetes)
$ helm install hazelcast/hazelcast

42. Application
Load Balancer
Application
Request
Cache Server
Client-Server Cache
Ops Team
Separate Management:
● backups
● (auto) scaling
● security

43. 2*. Cloud

44. Application
Load Balancer
Application
Request
Cloud (Cache as a Service)

45. Application
Load Balancer
Application
Request
Cloud (Cache as a Service)
Management:
● backups
● (auto) scaling
● security
Ops Team

46. Application
Load Balancer
Application
Request
Cloud (Cache as a Service)
Management:
● backups
● (auto) scaling
● security
Ops Team

47. Application
Load Balancer
Application
Request
Cloud (Cache as a Service)

48. @Configuration
public class HazelcastCloudConfiguration {
@Bean
CacheManager cacheManager() {
ClientConfig clientConfig = new ClientConfig();
clientConfig.getNetworkConfig().getCloudConfig()
.setEnabled(true)
.setDiscoveryToken("KSXFDTi5HXPJGR0wRAjLgKe45tvEEhd");
clientConfig.setGroupConfig(
new GroupConfig("test-cluster", "b2f984b5dd3314"));
return new HazelcastCacheManager(
HazelcastClient.newHazelcastClient(clientConfig));
}
}
Cloud (Cache as a Service)

49. DEMO
cloud.hazelcast.com

50. Client-Server (Cloud) Cache
Pros
● Data separate from applications
● Separate management (scaling,
backup)
● Programming-language agnostic
Cons
● Separate Ops effort
● Higher latency
● Server network requires
adjustment (same region, same
VPC)

51. ● Introduction ✔
● Caching Architectural Patterns
○ Embedded ✔
○ Embedded Distributed ✔
○ Client-Server ✔
○ Cloud ✔
○ Sidecar
○ Reverse Proxy
○ Reverse Proxy Sidecar
● Summary
Agenda

52. 3. Sidecar

53. Kubernetes Service
(Load Balancer)
Request
Kubernetes POD
Application Container
Cache Container
Application Container
Cache Container
Kubernetes POD
Sidecar Cache
Hazelcast
Cluster

54. Sidecar Cache
Similar to Embedded:
● the same physical machine
● the same resource pool
● scales up and down together
● no discovery needed (always localhost)
Similar to Client-Server:
● different programming language
● uses cache client to connect
● clear isolation between app and cache

55. @Configuration
public class HazelcastSidecarConfiguration {
@Bean
CacheManager cacheManager() {
ClientConfig clientConfig = new ClientConfig();
clientConfig.getNetworkConfig()
.addAddress("localhost:5701");
return new HazelcastCacheManager(HazelcastClient
.newHazelcastClient(clientConfig));
}
}
Sidecar Cache

56. Sidecar Cache
apiVersion: apps/v1
kind: Deployment
...
spec:
template:
spec:
containers:
- name: application
image: leszko/application
- name: hazelcast
image: hazelcast/hazelcast

57. Sidecar Cache
Pros Cons
● Simple configuration
● Programming-language agnostic
● Low latency
● Some isolation of data and
applications
● Limited to container-based
environments
● Not flexible management (scaling,
backup)
● Data collocated with application
PODs

58. ● Introduction ✔
● Caching Architectural Patterns
○ Embedded ✔
○ Embedded Distributed ✔
○ Client-Server ✔
○ Cloud ✔
○ Sidecar ✔
○ Reverse Proxy
○ Reverse Proxy Sidecar
● Summary
Agenda

59. 4. Reverse Proxy

60. Application
Load Balancer Cache
Application
Request
Reverse Proxy Cache

61. Reverse Proxy Cache
http {
...
proxy_cache_path /data/nginx/cache
keys_zone=one:10m;
...
}

62. ● Only for HTTP
● Not distributed
● No High Availability
● Data stored on the disk
NGINX Reverse Proxy Cache Issues

63. NGINX Reverse Proxy Cache Issues
● Only for HTTP
● Not distributed
● No High Availability
● Data stored on the disk

64. 4*. Reverse Proxy Sidecar

65. Kubernetes Service
(Load Balancer)
Request
Kubernetes POD
Application Container
Reverse Proxy Cache
Container
Application Container
Reverse Proxy Cache
Container
Kubernetes POD
Reverse Proxy Sidecar Cache
Hazelcast
Cluster

67. Good

68. Reverse Proxy Sidecar Cache
Service 1
Service 2 v1
Service 2 v2
Service 1
Service 4 v1
Service 4 v2
Service 4 v3
Ruby

69. Reverse Proxy Sidecar Cache
Service 1
Service 2 v1
Service 2 v2
Service 1
Service 4 v1
Service 4 v2
Service 4 v3
Ruby

70. Reverse Proxy Sidecar Cache
apiVersion: apps/v1
kind: Deployment
...
spec:
template:
spec:
initContainers:
- name: init-networking
image: leszko/init-networking
containers:
- name: caching-proxy
image: leszko/caching-proxy
- name: application
image: leszko/application

71. Reverse Proxy Cache
Container
Application
Container
eth0
:80
Reverse Proxy Sidecar Cache
:80
Pod
:8000
lo
:80

72. Reverse Proxy Sidecar Cache
#!/bin/bash
# Forward TCP traffic on port 80 to port 8000 on the eth0 interface.
iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 80 -j REDIRECT --to-port 8000

73. Reverse Proxy Sidecar Cache
Service 1
Service 2 v1
Service 2 v2
Service 1
Service 4 v1
Service 4 v2
Service 4 v3
Ruby

75. Reverse Proxy Sidecar Cache (Istio)

76. Bad

78. Reverse Proxy Cache
@CacheEvict(value = "someValue", allEntries = true)
public void evictAllCacheValues() {}
Application Cache:

79. Reverse Proxy Cache
@CacheEvict(value = "someValue", allEntries = true)
public void evictAllCacheValues() {}
Application Cache:
Proxy Cache:
http {
...
location / {
add_header Cache-Control public;
expires 86400;
etag on;
}
}

80. Reverse Proxy (Sidecar) Cache
Pros Cons
● Configuration-based (no need to
change applications)
● Programming-language agnostic
● Consistent with containers and
microservice world
● Difficult cache invalidation
● No mature solutions yet
● Protocol-based (e.g. works only
with HTTP)

81. ● Introduction ✔
● Caching Architectural Patterns
○ Embedded ✔
○ Embedded Distributed ✔
○ Client-Server ✔
○ Cloud ✔
○ Sidecar ✔
○ Reverse Proxy ✔
○ Reverse Proxy Sidecar ✔
● Summary
Agenda

82. Summary

83. application-aware?

84. application-aware?
containers?
no

85. application-aware?
containers?
Reverse Proxy
no
no

86. application-aware?
containers?
Reverse ProxyReverse Proxy Sidecar
no
yes no

87. application-aware?
containers?
Reverse ProxyReverse Proxy Sidecar
lot of data?
security restrictions?
yes no
yes no

88. application-aware?
containers?
Reverse ProxyReverse Proxy Sidecar
lot of data?
security restrictions?
language-agnostic?
containers?
yes no
yes nono

89. application-aware?
containers?
Reverse ProxyReverse Proxy Sidecar
lot of data?
security restrictions?
language-agnostic?
containers?
Embedded
(Distributed)
yes no
yes no
no
no

90. application-aware?
containers?
Reverse ProxyReverse Proxy Sidecar
lot of data?
security restrictions?
language-agnostic?
containers?
Embedded
(Distributed)
Sidecar
yes no
yes
yes no
no
no

91. application-aware?
containers?
Reverse ProxyReverse Proxy Sidecar
lot of data?
security restrictions?
language-agnostic?
containers?
Embedded
(Distributed)
Sidecar
cloud?
yes no
yes
yes
yes no
no
no

92. application-aware?
containers?
Reverse ProxyReverse Proxy Sidecar
lot of data?
security restrictions?
language-agnostic?
containers?
Embedded
(Distributed)
Sidecar
cloud?
Client-Server
yes no
yes
yes
yes no
nono
no

93. application-aware?
containers?
Reverse ProxyReverse Proxy Sidecar
lot of data?
security restrictions?
language-agnostic?
containers?
Embedded
(Distributed)
Sidecar
cloud?
Client-ServerCloud
yes no
yes
yes yes
yes no
nono
no

94. Resources
● Hazelcast Sidecar Container Pattern:
https://hazelcast.com/blog/hazelcast-sidecar-container-pattern/
● Hazelcast Reverse Proxy Sidecar Caching Prototype:
https://github.com/leszko/caching-injector
● Caching Best Practices: https://vladmihalcea.com/caching-best-practices/
● NGINX HTTP Reverse Proxy Caching:
https://www.nginx.com/resources/videos/best-practices-for-caching/

95. Thank You!
Rafał Leszko
@RafalLeszko