RedisConf19

1. PRESENTED BY
Resilient Testing at Scale
using Redis Queues
to Manage Concurrency
Aaron Evans
Sauce Labs, Solutions Architect

2. PRESENTED BY
1 Test Automation with Selenium & Sauce Labs
Agenda
2 How to implement a queue to manage concurrency
3 Redis data types (STRING, LIST, SET, HASH, ZSET)
4 Notifications with PUBLISH / SUBSCRIBE
5 Using RedisJSON to store complex data (test results)
6 Analyzing historical data with RediSearch
7 Predicting test failures with machine learning

3. PRESENTED BY
• My name is Aaron Evans
• Accidental software tester
• Burned out and went to Fiji to live on the beach
• Came back to buy a sailboat but got married instead
• Moved to Ecuador and started freelance consulting
• Traveled around the USA with my family in an RV
• Live in Montana on a farm in the woods & built a yurt
About me

4. PRESENTED BY
• Sauce Labs is a testing infrastructure company
• Founded by creator of Selenium, Jason Huggins
• Allows you to run test automation in the cloud on different OS, browsers & mobile
devices
• Using Selenium & Appium
• Essentially a managed remote Selenium grid
• Also provides a nice UI, dashboard, reports, manual testing via remote control, and
test archives & analytics
About Sauce Labs

5. PRESENTED BY
• Many tests run in parallel
• Need to manage test concurrency
• Across multiple teams & users
• Triggered by CI server or manually by developers
• Flaky / Slow Tests
• Analysis & Prediction
The Problem

6. PRESENTED BY
• Use Redis as a single source of truth
• Keep track of how many tests are running
• Create a "pool" of sessions
• Move from pending, to active, to complete
– Like a ledger
• Check out a session at the start of each test
• Check in when complete
• Store test data in Redis for future analysis
A Solution

7. PRESENTED BY
• KEY/VALUE store
– Memcached, Berkeley DB
• Relational database
– MySQL, PostgreSQL, Oracle
• NoSQL database / document store
– MongoDB, Cassandra
• Shared File System
– NFS, SMB
• Web Service
Alternatives to Redis

8. PRESENTED BY
• It’s fast!
• Simple API with available client for all major platforms
• Nice middle ground between Key/Value and NoSQL
• I didn’t want the hassle of setting up hosting or managing infrastructure
• Redis Labs has a free service tier that is sufficient
• I wanted to learn something new
Why Redis

9. PRESENTED BY
Implementation

10. PRESENTED BY
• Use a STRING to keep a count of concurrent sessions
• Numeric values can be incremented/decremented
• Subtract from counter when each session starts
• Add to counter after each session finishes
Keep track of available sessions with a counter

11. PRESENTED BY
String operations for simple counter
redis.test:6379> set available_sessions 100
OK
redis.test:6379> GET available_sessions
"100"
redis.test:6379> DECR available_sessions
(integer) 99
redis.test:6379> INCR available_sessions
(integer) 100
redis.test:6379> DECRBY available_sessions 50
(integer) 50
redis.test:6379> INCRBY available_sessions 25
(integer) 75
redis.test:6379> GETSET available_sessions 150
"75"

12. PRESENTED BY
• Add session_id to LIST when started
• Remove session_id from LIST when complete
• Get length of active_sessions to make sure you don’t exceed concurrency
• Trim sessions to appropriate size for cleanup
Create a LIST of active sessions

13. PRESENTED BY
List operations for active sessions
redis.test:6379> LPUSH active_sessions session:123
(integer) 1
redis.test:6379> LPUSH active_sessions session:124
(integer) 2
redis.test:6379> LPUSH active_sessions session:125
(integer) 3
redis.test:6379> LREM active_sessions 1 session:124
(integer) 1
redis.test:6379> LLEN active_sessions
(integer) 2
redis.test:6379> LRANGE active_sessions 0 -1
1) "session:125"
2) "session:123"
redis.test:6379> LTRIM active_sessions 0 99
OK

14. PRESENTED BY
• Use a separate list for requested test sessions
• First In / First Out (FIFO) queue
• LPUSH when a test session is requested
• RPOP when an active session becomes available
• RPUSHLPOP to atomically push from pending to active
Create a queue of pending tests

15. PRESENTED BY
List operations for requested sessions queue
redis.test:6379> LPUSH requested_sessions requested:126
(integer) 1
redis.test:6379> LPUSH requested_sessions requested:127
(integer) 2
redis.test:6379> RPOP requested_sessions
"requested:126"
redis.test:6379> LPUSH active_sessions session:126
(integer) 4
redis.test:6379> RPOPLPUSH requested_sessions active_sessions
"requested:127"
redis.test:6379> LSET active_sessions 0 session:127
OK
redis.test:6379> LRANGE active_sessions 0 1
1) "session:127"
2) "session:126"

16. PRESENTED BY
Transaction for moving from requested to active session
redis.test:6379> MULTI
OK
redis.test:6379> LREM active_sessions 1 session:126
QUEUED
redis.test:6379> BRPOPLPUSH requested_sessions active_sessions 60
QUEUED
redis.test:6379> LSET active_sessions 1 session:128
QUEUED
redis.test:6379> EXEC
1) (integer) 1
2) "requested:128"
3) OK
redis.test:6379> LINDEX active_sessions 0
"session:128"

17. PRESENTED BY
• Notify client when sessions are available
• PUBLISH sends to all clients listening to channel
• SUBSCRIBE listens (may be blocking)
• Single worker publishes
• Avoids need for constant polling for available sessions
Notifications with PUBLISH / SUBSCRIBE

18. PRESENTED BY
PUBLISH / SUBSCRIBE operations
# CLIENT LISTENS
redis.test:6379> SUBSCRIBE sessions_available
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "sessions_available"
3) (integer) 1
1) "message"
2) "sessions_available"
3) "requested:130"
# WORKER PUBLISHES
redis.test:6379> PUBLISH sessions_available requested:131
(integer) 1

19. PRESENTED BY
• Use a SET instead of a LIST
• Keys must be unique
• Sets aren’t ordered (no push / pop) * SPOP is random
• Ok because active_sessions are not necessarily in sequence
• Can move elements from one set to another
– (e.g. active to completed, passed, failed, errored)
• Allows grouping with SUNION, SINTER, SDIFF
• Can SORT if needed (and STORE as a list)
Avoid duplicate sessions with a SET

20. PRESENTED BY
Set operations for managing sessions
redis.test:6379> SADD active_sessions session:123
(integer) 1
redis.test:6379> SADD active_sessions session:123
(integer) 0
redis.test:6379> SCARD active_sessions
(integer) 1
redis.test:6379> SMOVE active_sessions completed_sessions session:123
(integer) 1
redis.test:6379> SADD errored_sessions session:124
(integer) 1
redis.test:6379> SUNION completed_sessions errored_sessions
1) "session:124"
2) "session:123"

21. PRESENTED BY
Move errored tests to retry queue
redis.test:6379> SMEMBERS errored_sessions
1) "session:126"
2) "session:124"
3) "session:125”
redis.test:6379> SORT errored_sessions ALPHA STORE retry_queue
(integer) 3
redis.test:6379> LRANGE retry_queue 0 -1
1) "session:124"
2) "session:125"
3) "session:126”

22. PRESENTED BY
• Use a ZSET (Sorted Set)
• Combines the best features of both LIST and SET
– *but it comes at a cost
• Unique keys
• SCORE allows you to rank tests to specify priority
• POP elements by highest or lowest score
– ZPOPMAX / ZPOPMIN
Implement a priority queue with a sorted test

23. PRESENTED BY
ZSET commands for priority queue
redis.test:6379> ZADD priority_queue 1 session:1
(integer) 1
redis.test:6379> ZADD priority_queue 2 session:2
(integer) 1
redis.test:6379> ZINCRBY priority_queue 2 session:1
"3"
redis.test:6379> ZRANGE priority_queue 0 -1 WITHSCORES
1) "session:2"
2) "2"
3) "session:1"
4) "3"
redis.test:6379> ZPOPMAX priority_queue
1) "session:1"
2) "3"

24. PRESENTED BY
• Store multiple Name/Value pairs
• Use session ID as HASH key
• Store test metadata
– Execution environment capabilities (e.g. OS, browser, version)
– Test name, build id, tags
– Test status (Passed/Failed, Completed/Errored)
Capture test data with a HASH

25. PRESENTED BY
Hash operations for storing test data
redis.test:6379> HSET session:123 name "login test" platform "Windows
10" browser "IE" version "11" tag "regression"
(integer) 5
redis.test:6379> HEXISTS session:123 status
(integer) 0
redis.test:6379> HSET session:123 status passed
(integer) 0
redis.test:6379> HGET session:123 status
"passed"
redis.test:6379> HGETALL session:123
1) "name"
2) "login test"
…
11) "status"
12) "passed"

26. PRESENTED BY
• Hashes only store 1 dimension
• Can fake it with multipart keys
– foo.bar.baz=quux
• Use a common key between records for relationships
– testdata:{123}
– testresult:{123}
• Store as a JSON blob in a STRING
• Use Redis JSON
• Use Redis Search
Storing complex (hierarchical) test data

27. PRESENTED BY
• Additional Module
• Can be compiled or used with RedisLabs
• Allows for storing and querying JSON data
Using RedisJSON

28. PRESENTED BY
RedisJSON operations
redis.test:6379> JSON.SET session:123 . '{ "name":"login test",
"status": "failed" }'
OK
redis.test:6379> JSON.GET session:123 status
""failed""

29. PRESENTED BY
• Additional Module
• Can be compiled or use with RedisLabs
• Allows for full text search
• Numeric or Tags
• Add unstructured data
– Test steps
– Log files
– Error messages & stack traces
Using RediSearch

30. PRESENTED BY
RediSearch for test results
redis.test:6379> FT.CREATE testresults SCHEMA name TEXT platform TEXT
browser TEXT version TEXT exec_time NUMERIC status TAG
OK
redis.test:6379> FT.ADD testresults test:123 1 FIELDS name "Login
Test" browser "Chrome" exec_time 123.456 status PASSED
OK
redis.test:6379> FT.SEARCH testresults login
1) (integer) 1
2) "test:123"
3) 1) name
2) "Login Test"
3) browser
4) "Chrome"
5) exec_time
6) "123.456"

31. PRESENTED BY
RediSearch for test results by status tag
redis.test:6379> FT.SEARCH testresults "@status:{ PASSED | COMPLETE }"
1) (integer) 1
2) "test:123"
3) 1) name
2) "Login Test"
3) browser
4) "Chrome"
5) exec_time
6) "123.456"
7) status
8) "PASSED"

32. PRESENTED BY
• Finding patterns and anticipating failures
• Group By
– test failures
– platform / browser / device
– feature or tags
– release / over time series
– other data points that you can't predict
• Using Machine Learning
Looking forwards

33. Thank you!

34. PRESENTED BY