Practical SPARQL Benchmarking Revisited

1. 1 Rob Vesse rvesse@yarcdata.com @RobVesse

2. 2 1. Rewind to 2012 2. Limitations 3. Evolving the Framework 4. Examples 5. Future Work

4. 4  Presentation I gave at this conference in 2012  Slides at http://www.slideshare.net/RobVesse/practical-sparql-benchmarking  Highlighted some issues with SPARQL Benchmarking:  Standard Benchmarks all have know deficiencies  Lack of standardized methodology  Best benchmark is the one you run with your data and workload  Introduced the 1.x version of our SPARQL Query Benchmarker tool  Java tool and API for benchmarking  Used a methodology based upon combination of the BSBM runner and Revelytix SP2B white paper  Reports various appropriate statistics  Various configuration options to change what exactly is benchmarked e.g. whether results are fully parsed and counted

5. 5  The 1.x tool was open sourced shortly after the 2012 conference under a 3 clause BSD License  Available on SourceForge  http://sourceforge.net/projects/sparql-query-bm/files/1.0.0/  Also as Maven artifacts (in Maven Central):  Group ID: net.sf.sparql-query-bm  Artifact IDs:  cmd  core  Latest 1.x Version: 1.1.0

7.  The 1.x tool can only benchmark SPARQL queries  SPARQL 1.1 has been standardized since the 1.x version of the tool was written and adds various additional SPARQL features that you may want to test: 7  SPARQL Updates  SPARQL Graph Store Protocol  Queries are fixed  No parameterization support  Can't pass custom endpoint parameters in  For example enable/disable reasoning  Also no way to test endpoint specific extensions  e.g. transactions

8. 8  Requires using HTTP endpoints to access the SPARQL system to be tested  Adds communication overheads to the results  Sometimes this may be desirable  No ability to test SPARQL operations in-memory  i.e. can't test lower level APIs

9.  Only supports a single benchmarking methodology  Methodology is hard coded  Can't do things like run a subset of the provided operations on each run 9  Or repeat an operation within a run  Or retry an operation under specific failure conditions  Configuration of the methodology is tightly coupled to the methodology  Many aspects are actually independent of the methodology

10. 1 0  Used a simplistic text based format  One query file per line  No way to specify additional parameters  No way to assign a friendly name to queries  Assigns each query the filename

11.  There is a progress monitoring API but it is limited  E.g. Gets called after a query completes but not before it starts  Makes it awkward/impossible to implement some kinds of monitoring 1 1  e.g. crash detection, memory usage

12. 1 2  In the interests of speed over usability we rolled our own command line arguments parser  Means argument parsing is awkward to extend

13. 1 3

14. 1 4  Earlier this year we found a compelling reason to rewrite the tool and address the various limitations  First 2.x release was made 9th June 2014  Minor bug fix and maintenance releases since  Releases available at:  http://sourceforge.net/projects/sparql-query-bm/files/  Code is now using Git  http://git.code.sf.net/p/sparql-query-bm/git sparql-query-bm-git  Mirrors available on GitHub for those who think that it is the one true source  https://github.com/rvesse/sparql-query-bm  Maven artifacts available through Maven Central as before:  Group ID: net.sf.sparql-query-bm  Artifact IDs: core, cmd and dist  Latest 2.x version: 2.0.1

15.  Concept of Queries replaced with the general concept of Operations  Also divorces the definition of an operation with how to run said operation 1 5  Makes it easier to change runtime behaviour of operations  20 built-in operations provided  API allows defining and plugging in new operations as desired  http://sparql-query-bm.sourceforge.net/javadoc/latest/core/

16. 1 6  Several kinds of query/update  Fixed  Parameterized  Dataset Size  Variants for both remote endpoints and in-memory datasets  Remote variants have additional NVP variants  Allows adding custom parameters to the remote request  Accounts for 13 of the built in operations

17. 1 7  One for each graph store protocol operation:  DELETE  GET  HEAD  POST  PUT  Accounts for a further 5 of the built-in operations

18. 1 8  Sleep  Do nothing for some period  Useful for simulating quiet periods as part of testing  Mix  Allow grouping a set of operations into a single operation  Lets you compose mixes from other mixes

19. 1 9  As already noted in-memory variants of some operations are now available  These run tests against a Dataset implementation  Part of Apache Jena ARQ API  Removes SPARQL Protocol and HTTP overhead from testing  Of course depending on Dataset implementation may still be some communication overhead  But this is likely using lower level back end native communications protocols instead

20. 2 0  Addresses the limitation of hard coded methodology  Separates test running into three components:  Overall runner  Mix runner  Operation runner  Each has own API and can be customized as desired  Various useful base/abstract implementations provided  Four different test runners are provided:  Benchmark  Smoke  Soak  Stress

21. 2 1  Smoke  Runs the mix once and indicates whether it passes/fails  Pass is defined as all operations pass  Soak  Run the mix continuously for some period of time  Test how a system reacts under continuous load  Stress  Run the mix with increasingly high load  Test how a system reacts under increasing load  AbstractRunner provides a basic framework and helper method to make it easy to add custom runners or customize existing runs

22. 2 2  Allows customizing how mixes and individual operations are run  Some alternative implementations built in:  E.g. SamplingOperationMixRunner  Runs a sample of the operations in the mix  May include repeats  E.g. RetryingOperationRunner  Retries an operation if it doesn't succeed  Easy to implement your own

23. 2 3  Separates test configuration from the test runner  Interface with all common configuration defined  Endpoints  Timeouts  Progress Listeners  etc  NB - Runners are typically defined such that they restrict their input options to sub-interfaces that add runner specific configuration e.g.  Warm-ups for benchmarks  Total runtime for soak testing  Ramp up factor for stress testing

24. 2 4  Now using TSV as the file format  Still wanted to be simple enough that someone with zero RDF/SPARQL knowledge can configure  Each line is a series of parameters separated by a tab character  First parameter is an identifier for the type of the operation  Used to decide how to interpret the remaining parameters  Can define your own mix file format and register a loader for it  Possible to override the loader for a specific operation identifier since this has an API  Means you can do neat tricks like use a mix designed for remote endpoints against an in-memory dataset

25. query 806670-warmup1.rq 806670 Warmup Query 1 query 806670-warmup2.rq 806670 Warmup Query 2 query 806670-nofilter.rq 806670 Query with No Filter query 806670-filter3.rq 806670 Query with Filter (Variant 3) param-query 806670-filter3-params.rq instances.tsv Parameterized Query with Filter (Variant 3) query 806670-filter4.rq 806670 Query with Filter (Variant 4) query 806670-filter4a.rq 806670 Query with Filter (Variant 4a - Zero Results) param-query 806670-filter4-params.rq instances.tsv Parameterized Query with Filter (Variant 4) query 806238-warmup1.rq 806238 Warmup Query 1 query 806238-warmup2.rq 806238 Warmup Query 2 query 806238-comment43.rq 806238 Query (Comment 43) query 806238-comment43a.rq 806238 Query (Comment 43 - SELECT * sub-query) query 806238-comment45.rq 806238 Query (Comment 45 - Multiple sub-queries) query 806238-comment54.rq 806238 Query (Comment 54) param-update load-full1m.ru graph-names.tsv Load 1M Dataset into named graph param-query count-loaded.rq graph-names.tsv Count named graph param-update drop-loaded.ru graph-names.tsv Drop named graph query count.rq Count quads checkpoint10 Checkpoint every 10 runs sleep 180 3 minute sleep 2 5

26.  Now provides notifications before and after operation and mix runs  Improvements to how some of the built-in implementations handle multi-threaded output 2 6  Makes it easier to distinguish where errors occurred when running multi-threaded benchmarks

27. 2 7  Now based upon the powerful open source Airline library  https://github.com/airlift/airline  Provides a command line interface to each built-in runner  Also provides AbstractCommandwith all standard options exposed  Standardized exit codes across all commands  Comprehensive built-in help  Can help you define operation mixes  ./operations  ./operation --op param-query

28. 2 8

29.  These are things we've done (or are currently doing) with the framework that aren't in the open source releases  However the 2.x framework makes these (hopefully) easy to replicate yourself 2 9

30. 3 0  Many stores often have rich REST APIs in addition to their SPARQL APIs  Can be useful to include testing of these in your mixes  Requires implementing two interfaces:  Operation  OperationCallable  Abstract implementations of both available to give you the boiler plate bits  Internally we have 9 different custom operations defined which test a subset of our REST API:  Database Management  Asynchronous Queries  Import Management

31.  One thing we're particularly interested in is how operations affect memory usage 3 1  We added custom progress listeners that track and monitor memory usage  Reports on min, max and average memory usage  We also have another progress listener that tracks processes to identify when a test run may have been impacted by other activity on the system

32. 3 2 public class RetryOnAuthFailureOperationRunner extends RetryingOperationRunner { public RetryOnAuthFailureOperationRunner() { this(1); } public RetryOnAuthFailureOperationRunner(int maxRetries) { super(maxRetries); } @Override protected <T extends Options> boolean shouldRetry(Runner<T> runner, T options, Operation op, OperationRun run) { return run.getErrorCategory() == ErrorCategories.AUTHENTICATION; } }  Extends the built-in RetryingOperationRunner  Simply adds a constraint on retries by overriding the shouldRetry() method

33. 3 3

34. 3 4  Embrace Java 7 features fully  Use ServiceLoader to automatically discover new operations and mix formats  Make it even easier to customize runners  i.e. provide more abstraction of the current implementations

35. 3 5 Questions? rvesse@yarcdata.com @RobVesse