Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
LDBC SNB Benchmark Auditing
Moritz Kaufmann
Technische Universität München
Process
Rules
+
Metrics
+
2014-03-05 3
Where to start?
Download driver
https://github.com/ldbc/ldbc_driver
Implement driver for SUT
Examples: https:...
Base setting
System with 3 years of data will be
benchmarked for 2h of simulation time
Main metric: Throughput @ Scale
Mai...
2014-03-05 5
Query Implementation Dataset @ SF
Clean Execution
Environment
System under test
Serializability Impl. (option...
2014-03-05 6
Install base system
Setup benchmark configuration
Validation run
Audit workflow
1
2014-03-05 7
Reset system
Setup loading configuration
Load data (timed)
Audit workflow
2
2014-03-05 8
Optional system warmup
Setup benchmark configuration
Benchmark >= 2h of
simulation time
Audit workflow
3
2014-03-05 9
Crash System +Recovery testing
Audit workflow
Verify benchmark run4
Continue run
2014-03-05 10
Audit workflow
Additional tooling
Validation of loaded data (potentially requires
implementation by the spon...
2014-03-05 11
Audit result
Report
Throughput @ Scale
Load time
Recovery time
System configuration, cost, …
2014-03-05 12
Rules
Inspiration from TPC, but we want to support
systems without a declarative query language.
Prohibit us...
2014-03-05 13
Rules
Rough guidelines
Datatypes have to conform to specification
Indexes have to be updatable
System has to...
2014-03-05 14
Rules
Intricate edges
How to define the weight of each query?
How to determine cost of system?
- Cloud? Open...
Discussion
Upcoming SlideShare
Loading in …5
×

LDBC SNB Benchmark Auditing

620 views

Published on

LDBC SNB Benchmark Auditing by Boritz Kaufmann (TUM)

Published in: Technology
  • Login to see the comments

  • Be the first to like this

LDBC SNB Benchmark Auditing

  1. 1. LDBC SNB Benchmark Auditing Moritz Kaufmann Technische Universität München
  2. 2. Process Rules + Metrics +
  3. 3. 2014-03-05 3 Where to start? Download driver https://github.com/ldbc/ldbc_driver Implement driver for SUT Examples: https://github.com/ldbc/ldbc_snb_implementations Verify it using validation dataset + validation file https://github.com/ldbc-dev/ldbc_snb_interactive_validation Download data generator + generate dataset https://github.com/ldbc-dev/ldbc_snb_datagen_0.2
  4. 4. Base setting System with 3 years of data will be benchmarked for 2h of simulation time Main metric: Throughput @ Scale Main obstacles: Query latency constraints + Crash recovery
  5. 5. 2014-03-05 5 Query Implementation Dataset @ SF Clean Execution Environment System under test Serializability Impl. (optional) Preparation
  6. 6. 2014-03-05 6 Install base system Setup benchmark configuration Validation run Audit workflow 1
  7. 7. 2014-03-05 7 Reset system Setup loading configuration Load data (timed) Audit workflow 2
  8. 8. 2014-03-05 8 Optional system warmup Setup benchmark configuration Benchmark >= 2h of simulation time Audit workflow 3
  9. 9. 2014-03-05 9 Crash System +Recovery testing Audit workflow Verify benchmark run4 Continue run
  10. 10. 2014-03-05 10 Audit workflow Additional tooling Validation of loaded data (potentially requires implementation by the sponsor) Query latency checks from driver log Serializability verificatoin (potentially requires implementation by the sponsor) Recovery verification
  11. 11. 2014-03-05 11 Audit result Report Throughput @ Scale Load time Recovery time System configuration, cost, …
  12. 12. 2014-03-05 12 Rules Inspiration from TPC, but we want to support systems without a declarative query language. Prohibit using hinted query plans, e.g. for join ordering Disallow precomputed indexes Force serializable consistency
  13. 13. 2014-03-05 13 Rules Rough guidelines Datatypes have to conform to specification Indexes have to be updatable System has to be generally obtainable within 6 months
  14. 14. 2014-03-05 14 Rules Intricate edges How to define the weight of each query? How to determine cost of system? - Cloud? Open-source?
  15. 15. Discussion

×