NDBench is a pluggable, cloud-enabled benchmarking tool developed by Netflix to test the performance and honesty of data stores. It can benchmark systems like Cassandra, Elasticsearch, and DynamoDB. NDBench has a REST API and UI, and allows dynamic configuration of load patterns, client drivers, and other parameters. Netflix uses NDBench for integration testing, deployment validation, and to benchmark upgrades like Cassandra version changes and hardware migrations. The presenter discusses NDBench's achievements in benchmarking Netflix's cloud infrastructure and databases. Future roadmap items include performance profiling, automated metrics analysis, and dynamic load generation.

1. Honest Performance Testing with
NDBench
Netflix Data Benchmark

5. • CDE, Cloud Database
Engineering
• Providing data stores as a
service
○Cassandra
○Dynomite
○Elasticsearch and RDS
Who are we?

6. Vinay Chella
Cloud Data Architect
Cassandra MVP
Cloud Database Engineering @ Netflix

7. • 98% of streaming data is stored in
Cassandra
• Data ranges from customer
details to Viewing history /
streaming bookmarks to billing
and payment
Cassandra @ Netflix

8. Agenda
• Background
• Why NDBench?
• Architecture
• Usage
• Achievements
• Roadmap
• Take away

9. Perf testing persistence layer?

10. Capacity in my existing fleet?

11. Why?
Why not already existing Perf testing tools?

12. What is NDBench?
Netflix Data Benchmark (NDBench) is a Pluggable
cloud-enabled benchmarking tool that can be used
across any data store system.

13. Side by Side comparison

14. Different driver/ software versions

15. Different instance types

16. Dynamically tunable configurations

17. Varying data models

18. Pluggable Patterns & Loads

19. Different Client APIs

20. Netflix homegrown
• Well integrated with
netflix OSS infrastructure

21. Architecture

22. What is Pluggable?
• Load Patterns
• Load tests

23. Load Patterns
• Random
• Sliding Window

24. Load Tests
• Cassandra-JavaDriver
• Elastic Search
• Dynomite
• Cassandra-Astyanax
• In-Memory Test

25. What can be configured?
• ndbench.driver.numKeys - 1000000
• ndbench.driver.dataSize - 200 bytes
• ndbench.driver.numWriters - 1
• ndbench.driver.numReaders - 1
• ndbench.driver.writeRateLimit - 100
• ndbench.driver.readRateLimit - 200
• ndbench.driver.useVariableDataSize - false
• Many more….

26. Dynamic Script

27. How to use it
• REST API
• UI

28. REST API
• Initialization
- Initialize: /pappy/driver/init/{DriverName}
- Init Script: /pappy/driver/initfromscript
• Perf API
- Start writes: /pappy/driver/startWrites
- Start reads: /pappy/driver/startReads
- Stop everything: /pappy/driver/stop
• Sanity check
- Verify Read: /pappy/driver/readSingle/key
- Verify Write: /pappy/driver/writeSingle/key
- Verify Read: /pappy/driver/readSingle/key
• Backfill
- Data Backfill: /pappy/driver/startDataFill
- DataBackfill Async: /pappy/driver/startDataFillAsync
• Status API
- /pappy/driver/{getRead/Write}Status
- /pappy/driver/getserverstatus

29. NDBench Demo...

31. NDBench @ Netflix
• As a
- Benchmarking Tool
- Integration Tests
- Deployment Validation

32. NDBench’s Achievements @ Netflix

33. N+1

34. C* 1.2 → C* 2.0, C*2.0 → C* 2.1

35. C* 2.0 vs C* 2.1 (Reads - Thrift)

36. C* 2.0 vs C* 2.1 (Writes - Thrift)

37. CentOS ---> Trusty

38. CentOS -> Trusty Migration

39. LCS on CentOS vs Trusty (writes)

40. LCS on CentOS vs Trusty (Reads)

41. Java 7 → Java 8

42. C* on Java 7 vs Java 8 (Writes)

43. C* on Java 7 vs Java 8 (Reads)

44. C* AMI Certification Pipeline

45. Dynomite benchmarking
• Generating Millions of
Ops/ sec

46. Dynomite benchmarking

47. Roadmap
• Performance profile management
• Automated metrics analysis
• Dynamic load generation based on destination
schema

48. https://github.com/Netflix/ndbench

49. Take away
“Test the honesty of your data models, persistence
layers in Cloud ecosystem using NDBench”

50. Q & A