High performance (as measured by sub-millisecond response time for queries) is a key characteristic of the Redis database, and it is one of the main reasons why Redis is the most popular key-value database in the world. In order to continue improving performance across all of the different Redis components, we’ve developed a framework for automatically triggering performance tests, telemetry gathering, profiling, and data visualization upon code commit. In this talk, we describe how this type of automation and “zero-touch” profiling scaled our ability to pursue performance regressions and find opportunities to improve the efficiency of our code, helping us (as a company) to start shifting from a reactive to a more proactive performance mindset.

1. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
E2E Performance Testing, Profiling, and
Analysis at Redis
Filipe Oliveira
CMG Atlanta 2024

2. Filipe Oliveira
Principal Performance Engineer at Redis
> whoami

3. ■ Working on continuous performance analysis
■ Open Source Contributor (C, Go):
● github.com/redis-performance
● github.com/redis/redis
● https://github.com/HdrHistogram/hdrhistogram-go
● https://github.com/RedisBloom/t-digest-c
> whoami

4. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
Agenda
Performance @Redis
The “old behaviour”
The do’s and dont’s
Our approach
KEY TAKEAWAYS

5. Performance…

6. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
Ordinarily, on our companies core products
We have...
● automatic extensive tests to catch functional failures
...but when
● we accidentally commit a performance regression, nothing intercepts it*!

7. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
> a real case from the past

8. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
> a real case from the past
1. RediSearch minor version bump
2. Required multiple patch
a. Feedback cycle took us at-least 1 day
b. prioritized over other projects
c. Siloed
d. Jul. 30, Nov. 27, 2019
You can relate to...
● your team run performance tests before releasing

9. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
> a real case from the past
1. RediSearch minor version bump
2. Required multiple patch
a. Feedback cycle took us at-least 1 day
b. prioritized over other projects
c. Siloed
d. Jul. 30, Nov. 27, 2019
You can relate to...
● your team run performance tests before releasing

10. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
Ordinarily, on our companies core products
You can state...
● your team run performance tests before releasing
...but solving slowdowns just before releasing is...
● dangerous
● time-consuming
● one of the most difficult tasks to estimate time to

11. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
Ordinarily, on our companies core products
You can state...
● your team run performance tests before releasing
...doing so is just buffering potential issues!

12. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
> goal: reduce feedback cycle. avoid silos
Requirements for valid tests
- Stable testing environment
- Deterministic testing tools
- Deterministic outcomes
- Reduced testing/probing overhead
- Reduce tested changes to the minimal
Requirements for acceptance in
products
- Acceptable duration
- No manual work
- Actionable items
- Well defined key performance
indicators
CODE REVIEW
PREVIEW /
UNSTABLE
RELEASE
MANUAL
PERF
CHECK
from:
CODE REVIEW
PREVIEW /
UNSTABLE
RELEASE
ZERO TOUCH
PERF CHECK
ZERO TOUCH
PERF CHECK
ZERO TOUCH
PERF CHECK
to:

13. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
> this is not new / disruptive
Elastic
https://elasticsearch-benchmarks.elastic.co/#
Lucene
https://home.apache.org/~mikemccand/lucenebench/

14. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
> this is not new / disruptive
mongoDB

15. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
> how we’re actually doing it
CODE REVIEW
PREVIEW /
UNSTABLE
RELEASE
ZERO TOUCH
PERF CHECK
ZERO TOUCH
PERF CHECK
ZERO TOUCH
PERF CHECK

16. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
> how we’re actually doing it

17. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
> How we’re
actually
doing it…

18. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
- summary dashboard details for Redis Ltd Performance CI tracking -
> 200 Active steady stable VMs on peak > 100K benchmark runs in ~= 2 years
> how we’re actually doing it

19. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
> our approach
Vanilla Redis (purely OSS project)
1. Created an OSS SPEC
a. [follow link]
2. Extend the spec and use it
a. for historical data
b. for regression analysis
c. for docs
Redis Developers Group +
(Redis Ltd, AWS, Ericson, Alibaba, …. )

20. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
> our approach
Redis Ltd
1. Started by the small scale
projects
a. Redis Module’s
2. Initial OSS deployments
3. local and remote triggers
4. Used for testing, profiling
a. Regression analysis
i. and fix
b. Approval of features
c. Proactive optimization

21. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
by branch
scalability analysis
by version
> our approach
Redis Ltd
1. Started by the small scale
projects
a. Redis Module’s
2. Initial OSS deployments
3. local and remote triggers
4. Used for testing, profiling
a. Regression analysis
i. and fix
b. Approval of features
c. Proactive optimization

22. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
scalability analysis by branch/version including client and server metrics
> our approach
Redis Ltd
1. Started by the small scale
projects
a. Redis Module’s
2. Initial OSS deployments
3. local and remote triggers
4. Used for testing, profiling
a. Regression analysis
i. and fix
b. Approval of features
c. Proactive optimization

23. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
> our approach
Redis Ltd
1. Started by the small scale
projects
a. Redis Module’s
2. Initial OSS deployments
3. local and remote triggers
4. Used for testing, profiling
a. Regression analysis
i. and fix
b. Approval of features
c. Proactive optimization

24. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
nightly:
feature* / perf* / v*:
> our approach
Redis Ltd
1. Started by the small scale
projects
a. Redis Module’s
2. Initial OSS deployments
3. local and remote triggers
4. Used for testing, profiling
a. Regression analysis
i. and fix
b. Approval of features
c. Proactive optimization

25. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
> our approach
Redis Ltd
1. Started by the small scale
projects
a. Redis Module’s
2. Initial OSS deployments
3. local and remote triggers
4. Used for testing, profiling
a. Regression analysis
i. and fix
b. Approval of features
c. Proactive optimization

26. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
1. Full process Flame Graph + main thread Flame Graph
2. perf report per dso
3. perf report per dso,sym (w/wout callgraph)
4. perf report per dso,sym,srcline (w/wout callgraph)
5. identical stacks collapsed
6. hotpath callgraph
1
3
2
4
6
> our approach
Redis Ltd
1. Started by the small scale
projects
a. Redis Module’s
2. Initial OSS deployments
3. local and remote triggers
4. Used for testing, profiling
a. Regression analysis
i. and fix
b. Approval of features
c. Proactive optimization

27. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
> our approach
Redis Ltd
1. Started by the small scale
projects
a. Redis Module’s
2. Initial OSS deployments
3. local and remote triggers
4. Used for testing, profiling
a. Regression analysis
i. and fix
b. Approval of features
c. Proactive optimization
> 300 individual profiles all merged…

28. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
> our approach
Redis Ltd
1. Started by the small scale
projects
a. Redis Module’s
2. Initial OSS deployments
3. local and remote triggers
4. Used for testing, profiling
a. Regression analysis
i. and fix
b. Approval of features
c. Proactive optimization
> 300 individual profiles all merged…

29. © 2024 Redis Ltd. All rights reserved. Confidential (Internal use only)
Analysis:
https://github.com/RedisTimeSeries/RedisTimeSeries/issues/793
PR:
https://github.com/RedisTimeSeries/RedisTimeSeries/pull/794
WIP:
https://github.com/RedisTimeSeries/RedisTimeSeries/issues/907
> our approach
Redis Ltd
1. Started by the small scale
projects
a. Redis Module’s
2. Initial OSS deployments
3. local and remote triggers
4. Used for testing, profiling
a. Regression analysis
i. and fix
b. Approval of features
c. Proactive optimization

30. concrete outcomes…

31. improved Redis
performance by up to 4x![1]
[1] - https:/
/redis.com/blog/redis-intel-performance-testing/
https:/
/redis.com/blog/redis-7-geographic-commands/

32. > what we’ve gained (1/4)
● Deeply reduced the feedback cycle ( days -> 1 hour )
● Dev’s can easily add tests (> 300 full suites)
● Scaled + more challenging!
● performance is now everyone’s power/responsibility

33. > what we’ve gained (2/4)
● A/B test new tech/state-of-the-art HW/SW components
● Continuous up-to-date numbers for use-cases that matter
● Foster openness community/cross-company efforts

34. > what we’ve gained (3/4)
● Able to commit to reduce overhead per operation on our cloud/SW
● Competitive advantage/Leading
● Shift proactive/reactive + predictiveness
● Reduce costs
○ manual work/detection is at least 17.5X more expensive vs automation detection

35. > what we’ve gained (4/4)
● ability to reproduce the performance of > 10 years of
development of Redis
■ “go back in time”

36. what’s next…

37. ● Extend profiler daemon to bpf tooling, vtune
○ off-cpu analysis
○ threading/locking
○ vectorization reports
VISIBILITY for Points of Improvement
> what’s next feature wise (1/2)

38. > what’s next feature wise (2/2)
● extend tools to characterize further the workload
○ mem bound/cpu bound
○ HW counters
■ stalls on memory
○ % time off-cpu
○ extend partner tooling and HW
■ beta versions of their next gen HW
■ multi-arch comparisons (we do it manually now)
○ extended io statistics
● include low(*er) overhead profilers / tracers
○ call count analysis
○ off cpu flame charts analysis
○ syscall
○ …

39. > what’s next product wise
● Improve anomaly/regression detection
● Increase OSS / Company adoption
○ expose data on docs

40. Follow up links
● Redis Performance Group
● Redis Benchmarks Specification
● Making the fast faster blog

41. Thank you.
ping us at: performance <at> redis <dot> com