Why, how, and production considerations

1. March 2018
Spring Session Redis
Oded Shopen
Why, How and Production Considerations

2. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs2
About Me
▪Oded Shopen
▪Software Architect @ Amdocs Israel
▪Working on Event-Driven, Cloud Native Microservices
▪🤓 Redis Geek
▪http://odedia.org
▪@odedia on twitter

3. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs3
Why?

4. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs4
Why Spring Session?
▪Once upon a time, there was the servlet container. And it was good…
Tomcat 🍪 123-456 = David

5. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs5
Why Spring Session?
Tomcat
🍪 123-456 = David
Tomcat Tomcat
Load Balancer
🍪 789-111 = Larry

6. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs6
Why Spring Session?
🍪 123-456 = ?
Tomcat Tomcat
Load Balancer
🍪 789-111 = Larry

7. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs7
Why Spring Session?
😞

8. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs8
Why Spring Session?
Tomcat Tomcat
Load Balancer
Tomcat Tomcat
Tomcat Tomcat
Tomcat Tomcat
Tomcat Tomcat
Tomcat Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat

9. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs9
Why Spring Session?
Jetty Tomcat
Load Balancer
Jetty Tomcat
Tomcat Tomcat
Tomcat Tomcat
Tomcat Tomcat
Jetty Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat

10. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs10
Why Spring Session?
Tomcat
Session Data

11. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs11
Why Spring Session?
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Jetty
Session Data

12. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs12
Why Spring Session?
▪Replaces the built-in HttpSession with another implementation
▪Transparent drop-in replacement when using Spring Boot
▪Makes your servers truly stateless
▪Sessions survive application restarts
▪No need for Load Balancer sticky sessions
▪Conforms to the cloud-native apps 12-factor principals

13. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs13
Twelve-factor processes are stateless
and share-nothing.
Any data that needs to persist must be
stored in a stateful backing service,
typically a database.

14. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs14
Why Spring Session Redis?
https://docs.spring.io/spring-session/docs/1.3.1.RELEASE/reference/html5/

15. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs15
Why Spring Session Redis?
▪The application requires frequent, fast access to the session
▪A fast database is critical
▪Redis is really fast
▪No user experience degradation by externalizing the session to Redis
▪Sessions needs to expire after some time
▪Redis expiring keys are a great solution
▪With sharding and clustering, Redis scales when your user base scales

16. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs16
How?

17. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs17
Demo
https://github.com/odedia/spring-session-redis-sample

18. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs18
Production
Considerations

19. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs19
#1 Be Prepared to Scale
▪At the very minimum, use a master/slave setup

20. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs20
#1 Be Prepared to Scale
▪Consider sharding your sessions in a Redis Cluster
▪RedisLabs + DNS Proxy will hide the cluster topology and let you focus on
a simple configuration on the client side
▪No need to configure sentinels
Node 1
Node 2
Node 3
Proxyspring.redis.url=proxy:6379
spring.redis.sentinel.master=MasterNode
spring.redis.sentinel.nodes=Node1:6379,Node2:6379,Node3:6379

21. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs21
#2 Pool Settings
▪ Spring Data Redis uses a JedisConnectionFactory with the following defaults:
spring.redis.pool.max-active=8
spring.redis.pool.max-idle=8
spring.redis.pool.max-wait=-1
spring.redis.pool.min-idle=0
spring.redis.timeout=0
▪ Behind the scenes - an Apache Commons GenericObjectPool.
▪ Use max-active based on your tomcat/jetty threads.
▪ Use max-wait based on your setup
▪ Too high - all tomcat threads can get stuck, waiting
▪ Too low - your consumers may get errors
▪ Use min-idle of at least 8 to offset “sudden load” issues (such as mass login on a business day).
▪ Set spring.redis.timeout to a reasonably low number (such as 1000ms)
▪ Too high - your connection pool can become full quickly.
▪ Too low - your consumers may get errors

22. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs22
#3 Server-Side Metrics
▪Spring Boot Actuator provides a production-ready monitoring metrics.
▪Simply add spring-boot-starter-actuator to your gradle/maven dependencies
▪/metrics endpoint provides valuable monitoring data
▪However, Spring Data Redis is not available…
▪Luckily, actuator is extensible!
▪https://github.com/nysd/spring-boot-redis-metrics

23. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs23
#4 Monitor Redis
▪Redislabs monitoring dashboards provide
excellent visibility to identify issues, as we’ll soon
see.
▪If you use open source and monitor yourself
on-prem, keep in mind that Redis is a single-
threaded database.
▪A 2-core virtual machine would report 50%
CPU utilization
▪However, redis itself might be at 100% at that
time
▪Monitor the process, not the VM

24. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs24
#5 Please Monitor Responsibly
▪Be careful how you monitor…
▪KEYS * runs over all the keys in the DB. Your DB is unresponsive until operation is over!
▪SMEMBERS returns all keys in a given set. Some sets may contain all keys in the system.
▪In our use case, we kept track of all logins from a certain vendor, which is pretty
much O(n) cardinality, so same as KEYS command
▪SCARD simply returns the size of a given set with cardinality of O(1), so it can be used
safely in production
▪ SCARD “spring:session:index:org.springframework.session.FindByIndexNameSessionRepository.PRINCIPAL_NAME_INDEX_NAME:UI_USERS"
▪INFO provides valuable information (such as instantaneous_ops_per_sec) and can be
safely used in production.

25. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs25

26. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs26

27. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs27
#6 Inside Spring Session Redis
▪What happens when your system is completely idle (0 users)?
▪You would expect redis to be idle as well
▪However…

28. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs28
#6 Inside Spring Session Redis
▪Multiple logins using:
▪for ((i=1;i<=10000000;i++));
do curl -s -u gateway:password localhost:8080/user >/dev/null;
done
▪One would expect consistent load on
redis, but in reality, we got:

29. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs29
#6 Inside Spring Session Redis

30. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs30
#6 Inside Spring Session Redis
▪Time to dig into the code… (thanks, open source!)
▪Inside RedisOperationsSessionRepository:
@Scheduled(cron = "${spring.session.cleanup.cron.expression:0 * * * * *}")
public void cleanupExpiredSessions() {
this.expirationPolicy.cleanExpiredSessions();
}
▪cleanupExpiredSessions() would loop over expired keys, delete them, and “touch”
them to make sure they are immediately deleted.
▪This means that by default, every minute on the minute, Spring will call delete on every
expired key in Redis.

31. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs31
#6 Inside Spring Session Redis
▪Why? Redis can self-expire keys just fine, thank you very much
▪The main reason:
▪Spring wants to conduct cleanup activities when a session is expired.
▪The API exposes SessionDeletedEvent and SessionExpiredEvent to let the
program cleanup a user’s session (close web sockets, notify SSO server etc.)
▪Spring needs the session details available when the cleanup activity is
performed

32. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs32
#6 Inside Spring Session Redis
▪To solve this issue, two separate keys are managed for each session
▪An “expiration notification” key expires after 30 minutes
▪The actual session details key expires 5 minutes later
HMSET spring:session:sessions:33fdd1b6-b496-4b33-9f7d-df96679d32fe creationTime
1404360000000
maxInactiveInterval 1800
lastAccessedTime 1404360000000
sessionAttr:attrName someAttrValue
sessionAttr2:attrName someAttrValue2
EXPIRE spring:session:sessions:33fdd1b6-b496-4b33-9f7d-df96679d32fe 2100
APPEND spring:session:sessions:expires:33fdd1b6-b496-4b33-9f7d-df96679d32fe ""
EXPIRE spring:session:sessions:expires:33fdd1b6-b496-4b33-9f7d-df96679d32fe 1800
SADD spring:session:expirations:1439245080000 expires:33fdd1b6-b496-4b33-9f7d-
df96679d32fe
EXPIRE spring:session:expirations1439245080000 2100

33. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs33
#6 Inside Spring Session Redis
▪Redis guarantees expired keys would be passively cleaned, but makes no
guarantees on when
▪Spring Session Redis wants to make sure the “expire” key is expired within the 5
minutes window.
▪For this reason, the cronjob updates the expired keys every minute
▪But, there is a risk here…

34. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs
#6 Inside Spring Session Redis
34
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Jetty
Session Data

35. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs35
#6 Inside Spring Session Redis
Session Data
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Jetty

36. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs36
#6 Inside Spring Session Redis
▪To recap:
▪Every minute, each server tries to delete all “expires” keys.
▪A lot of redundant calls, since only the first instance would actually achieve this
purpose.
▪Redis would then notify every instance that the “expires” key was deleted.
▪Every instance would then connect to Redis to get the session details about the
expiring session to handle the expiration.
▪A little bit of math:
▪100 servers * 2500 expiring sessions = 500,000 access at the same second
▪The more you scale, the bigger the problem

37. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs37
#6 Inside Spring Session Redis
Solutions:
▪ If you don’t care about session expiration events, set the cron job to an impossible value such as
February 31st:
spring.session.cleanup.cron.expression=0 0 5 31 2 ?
▪ You can also disable registering to those expiration and deletion events. Note that all servers need to
disable the registering for this to work:
@EnableRedisHttpSession
public class RedisOperationsSessionRepositoryConfigNoOp {
@Bean
public static ConfigureRedisAction configureRedisAction() {
return ConfigureRedisAction.NO_OP;
}
}

38. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs38
#6 Inside Spring Session Redis
▪If you do care about session expiration events, set only a specific instance(s)
to handle those events.
▪On all other servers, disable receiving the events:
@EnableRedisHttpSession
public class RedisOperationsSessionRepositoryConfigNoListener {
@Bean
public RedisMessageListenerContainer redisMessageListenerContainer(
RedisConnectionFactory connectionFactory,
RedisOperationsSessionRepository messageListener) {
return new RedisMessageListenerContainer();
}
}

39. Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs39
To Recap
▪Be prepared to scale
▪Monitor well, monitor with caution
▪Open source rocks 🤟. Get involved! Tweak the
framework for your own use case.

40. Thank you
http://odedia.org