Talk given at RedisConf 2019 about management concurrent parallel test automation within an organization. A case study about building a prioritized queue to enable more efficient testing and introduction to common Redis data types and the Redis JSON on Redisearch modules.

1. PRESENTED BY
Resilient Testing at Scale
using Redis Queues
to Manage Concurrency
Aaron Evans
Sauce Labs, Solutions Architect

2. PRESENTED BY
1 Test Automation with Selenium & Sauce Labs
Agenda
2 How to implement a queue to manage concurrency
3 Redis data types (STRING, LIST, SET, HASH, ZSET)
4 Notifications with PUBLISH / SUBSCRIBE
5 Using RedisJSON to store complex data (test results)
6 Analyzing historical data with RediSearch
7 Predicting test failures with machine learning

3. PRESENTED BY
• My name is Aaron Evans
• Accidental software tester
• Burned out and went to Fiji to live on the beach
• Came back to buy a sailboat but got married instead
• Moved to Ecuador and started freelance consulting
• Traveled around the USA with my family in an RV
• Live in Montana on a farm in the woods & built a yurt
About me

4. PRESENTED BY
• Sauce Labs is a testing infrastructure company
• Founded by creator of Selenium, Jason Huggins
• Allows you to run test automation in the cloud on different OS, browsers & mobile
devices
• Using Selenium & Appium
• Essentially a managed remote Selenium grid
• Also provides a nice UI, dashboard, reports, manual testing via remote control, and
test archives & analytics
About Sauce Labs

5. PRESENTED BY
• Many tests run in parallel
• Need to manage test concurrency
• Across multiple teams & users
• Triggered by CI server or manually by developers
• Flaky / Slow Tests
• Analysis & Prediction
The Problem

6. PRESENTED BY
• Use Redis as a single source of truth
• Keep track of how many tests are running
• Create a "pool" of sessions
• Move from pending, to active, to complete
– Like a ledger
• Check out a session at the start of each test
• Check in when complete
• Store test data in Redis for future analysis
A Solution

7. PRESENTED BY
• KEY/VALUE store
– Memcached, Berkeley DB
• Relational database
– MySQL, PostgreSQL, Oracle
• NoSQL database / document store
– MongoDB, Cassandra
• Shared File System
– NFS, SMB
• Web Service
Alternatives to Redis

8. PRESENTED BY
• It’s fast!
• Simple API with available client for all major platforms
• Nice middle ground between Key/Value and NoSQL
• I didn’t want the hassle of setting up hosting or managing infrastructure
• Redis Labs has a free service tier that is sufficient
• I wanted to learn something new
Why Redis

9. PRESENTED BY
Implementation

10. PRESENTED BY
• Use a STRING to keep a count of concurrent sessions
• Numeric values can be incremented/decremented
• Subtract from counter when each session starts
• Add to counter after each session finishes
Keep track of available sessions with a counter

11. PRESENTED BY
String operations for simple counter
redis.test:6379>setavailable_sessions 100
OK
redis.test:6379>GETavailable_sessions
"100"
redis.test:6379>DECRavailable_sessions
(integer) 99
redis.test:6379>INCRavailable_sessions
(integer) 100
redis.test:6379>DECRBYavailable_sessions50
(integer) 50
redis.test:6379>INCRBYavailable_sessions25
(integer) 75
redis.test:6379>GETSETavailable_sessions 150
"75"

12. PRESENTED BY
• Add session_id to LIST when started
• Remove session_id from LIST when complete
• Get length of active_sessions to make sure you don’t exceed concurrency
• Trim sessions to appropriate size for cleanup
Create a LIST of active sessions

13. PRESENTED BY
List operations for active sessions
redis.test:6379>LPUSHactive_sessions session:123
(integer) 1
redis.test:6379>LPUSHactive_sessions session:124
(integer) 2
redis.test:6379>LPUSHactive_sessions session:125
(integer) 3
redis.test:6379>LREMactive_sessions 1 session:124
(integer) 1
redis.test:6379>LLENactive_sessions
(integer) 2
redis.test:6379>LRANGEactive_sessions 0 -1
1) "session:125"
2) "session:123"
redis.test:6379>LTRIMactive_sessions 0 99
OK

14. PRESENTED BY
• Use a separate list for requested test sessions
• First In / First Out (FIFO) queue
• LPUSH when a test session is requested
• RPOP when an active session becomes available
• RPUSHLPOP to atomically push from pending to active
Create a queue of pending tests

15. PRESENTED BY
List operations for requested sessions queue
redis.test:6379>LPUSHrequested_sessionsrequested:126
(integer) 1
redis.test:6379>LPUSHrequested_sessionsrequested:127
(integer) 2
redis.test:6379>RPOPrequested_sessions
"requested:126"
redis.test:6379>LPUSHactive_sessions session:126
(integer) 4
redis.test:6379>RPOPLPUSHrequested_sessions active_sessions
"requested:127"
redis.test:6379>LSETactive_sessions 0 session:127
OK
redis.test:6379>LRANGEactive_sessions 0 1
1) "session:127"
2) "session:126"

16. PRESENTED BY
Transaction for moving from requested to active session
redis.test:6379>MULTI
OK
redis.test:6379>LREMactive_sessions 1 session:126
QUEUED
redis.test:6379>BRPOPLPUSHrequested_sessionsactive_sessions 60
QUEUED
redis.test:6379>LSETactive_sessions 1 session:128
QUEUED
redis.test:6379>EXEC
1) (integer) 1
2) "requested:128"
3) OK
redis.test:6379>LINDEXactive_sessions 0
"session:128"

17. PRESENTED BY
• Notify client when sessions are available
• PUBLISH sends to all clients listening to channel
• SUBSCRIBE listens (may be blocking)
• Single worker publishes
• Avoids need for constant polling for available sessions
Notifications with PUBLISH / SUBSCRIBE

18. PRESENTED BY
PUBLISH / SUBSCRIBE operations
# CLIENTLISTENS
redis.test:6379>SUBSCRIBEsessions_available
Readingmessages...(pressCtrl-Ctoquit)
1) "subscribe"
2) "sessions_available"
3) (integer) 1
1) "message"
2) "sessions_available"
3) "requested:130"
# WORKERPUBLISHES
redis.test:6379>PUBLISHsessions_availablerequested:131
(integer) 1

19. PRESENTED BY
• Use a SET instead of a LIST
• Keys must be unique
• Sets aren’t ordered (no push / pop) * SPOP is random
• Ok because active_sessions are not necessarily in sequence
• Can move elements from one set to another
– (e.g. active to completed, passed, failed, errored)
• Allows grouping with SUNION, SINTER, SDIFF
• Can SORT if needed (and STORE as a list)
Avoid duplicate sessions with a SET

20. PRESENTED BY
Set operations for managing sessions
redis.test:6379>SADDactive_sessions session:123
(integer) 1
redis.test:6379>SADDactive_sessions session:123
(integer) 0
redis.test:6379>SCARDactive_sessions
(integer) 1
redis.test:6379>SMOVEactive_sessions completed_sessionssession:123
(integer) 1
redis.test:6379>SADDerrored_sessionssession:124
(integer) 1
redis.test:6379>SUNIONcompleted_sessions errored_sessions
1) "session:124"
2) "session:123"

21. PRESENTED BY
Move errored tests to retry queue
redis.test:6379>SMEMBERSerrored_sessions
1) "session:126"
2) "session:124"
3) "session:125”
redis.test:6379>SORTerrored_sessions ALPHASTOREretry_queue
(integer) 3
redis.test:6379>LRANGEretry_queue0 -1
1) "session:124"
2) "session:125"
3) "session:126”

22. PRESENTED BY
• Use a ZSET (Sorted Set)
• Combines the best features of both LIST and SET
– *but it comes at a cost
• Unique keys
• SCORE allows you to rank tests to specify priority
• POP elements by highest or lowest score
– ZPOPMAX / ZPOPMIN
Implement a priority queue with a sorted test

23. PRESENTED BY
ZSET commands for priority queue
redis.test:6379>ZADDpriority_queue1 session:1
(integer) 1
redis.test:6379>ZADDpriority_queue2 session:2
(integer) 1
redis.test:6379>ZINCRBYpriority_queue2 session:1
"3"
redis.test:6379>ZRANGEpriority_queue0 -1WITHSCORES
1) "session:2"
2) "2"
3) "session:1"
4) "3"
redis.test:6379>ZPOPMAX priority_queue
1) "session:1"
2) "3"

24. PRESENTED BY
• Store multiple Name/Value pairs
• Use session ID as HASH key
• Store test metadata
– Execution environment capabilities (e.g. OS, browser, version)
– Test name, build id, tags
– Test status (Passed/Failed, Completed/Errored)
Capture test data with a HASH

25. PRESENTED BY
Hash operations for storing test data
redis.test:6379>HSETsession:123name"login test"platform"Windows10"browser"IE"version "11"tag "regression"
(integer) 5
redis.test:6379>HEXISTSsession:123status
(integer) 0
redis.test:6379>HSETsession:123statuspassed
(integer) 0
redis.test:6379>HGETsession:123 status
"passed"
redis.test:6379>HGETALLsession:123
1) "name"
2) "login test"
…
11) "status"
12) "passed"

26. PRESENTED BY
• Hashes only store 1 dimension
• Can fake it with multipart keys
– foo.bar.baz=quux
• Use a common key between records for relationships
– testdata:{123}
– testresult:{123}
• Store as a JSON blob in a STRING
• Use Redis JSON
• Use Redis Search
Storing complex (hierarchical) test data

27. PRESENTED BY
• Additional Module
• Can be compiled or used with RedisLabs
• Allows for storing and querying JSON data
Using RedisJSON

28. PRESENTED BY
RedisJSON operations
redis.test:6379>JSON.SETsession:123. '{"name":"logintest","status":"failed"}'
OK
redis.test:6379>JSON.GETsession:123status
""failed""

29. PRESENTED BY
• Additional Module
• Can be compiled or use with RedisLabs
• Allows for full text search
• Numeric or Tags
• Add unstructured data
– Test steps
– Log files
– Error messages & stack traces
Using RediSearch

30. PRESENTED BY
RediSearch for test results
redis.test:6379>FT.CREATEtestresultsSCHEMA nameTEXTplatformTEXTbrowserTEXTversion TEXT exec_timeNUMERICstatusTAG
OK
redis.test:6379>FT.ADDtestresultstest:1231 FIELDSname"LoginTest"browser"Chrome"exec_time 123.456statusPASSED
OK
redis.test:6379>FT.SEARCHtestresultslogin
1) (integer) 1
2) "test:123"
3) 1) name
2) "LoginTest"
3) browser
4) "Chrome"
5) exec_time
6) "123.456"

31. PRESENTED BY
RediSearch for test results by status tag
redis.test:6379>FT.SEARCHtestresults"@status:{PASSED|COMPLETE}"
1) (integer) 1
2) "test:123"
3) 1) name
2) "LoginTest"
3) browser
4) "Chrome"
5) exec_time
6) "123.456"
7) status
8) "PASSED"

32. PRESENTED BY
• Finding patterns and anticipating failures
• Group By
– test failures
– platform / browser / device
– feature or tags
– release / over time series
– other data points that you can't predict
• Using Machine Learning
Looking forwards

33. Thank you!

34. PRESENTED BY

35. PRESENTED BY