Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ycsb benchmarking


Published on

YCSB benchmarking report with some comparative info for Accumulo and HBase

Published in: Technology
  • Be the first to comment

Ycsb benchmarking

  1. 1. YCSB++ BenchmarkingToolPerformanceDebuggingAdvancedFeaturesofScalableTableStoresSwapnil PatilMilo Polte,WittawatTantisiriroj, Kai Ren, Lin Xiao, Julio Lopez, Garth Gibson, Adam Fuchs *, Billie Rinaldi *CarnegieMellonUniversity *NationalSecurityAgencyOpen Cirrus Summit, October 2011, Atlanta GA
  2. 2. Scalable table stores are critical systemsSwapnil Patil, CMU 2•  For data processing & analysis (e.g. Pregel, Hive)•  For systems services (e.g., Google Colossus metadata)
  3. 3. Evolution of scalable table storesSimple, lightweight  complex, feature-rich storesSupports a broader range of applications and servicesHard to debug and understand performance problemsComplex behavior and interaction of various componentsGrowingsetofHBasefeatures2008 2009 2010 2011+RangeRowFiltersBatch updatesBulk load toolsRegEx filteringScan optimizationsHBASEreleaseCo-processorsAccess Control⏏
3Swapnil Patil, CMU
  4. 4. YCSB++ FUNCTIONALITYZooKeeper-based distributed and coordinated testingAPI and extensions the new Apache ACCUMULO DBFine-grained, correlated monitoring usingOTUSFEATURES TESTED USING YCSB++Batch writing Table pre-splitting  Bulk loadingWeak consistency  Server-side filtering  Fine-grained securityTool released at Patil, CMU 4Need richer tools for understandingadvanced features in table stores …
  5. 5. Outline•  Problem•  YCSB++ design•  Illustrative examples•  Ongoing work and summarySwapnil Patil, CMU 5
  6. 6. Yahoo Cloud Serving Benchmark [Cooper2010]Swapnil Patil, CMU 6•  For CRUD (create-read-update-delete) benchmarking•  Single-node system with an extensible APIStorage ServersHBASEOTHERDBSWorkloadExecutorThreadsStatsDBClientsCommand-lineParametersWorkloadParameterFile
  7. 7. YCSB++: New extensionsSwapnil Patil, CMU 7Added support for the new Apache ACCUMULO DB− New parameters and workload executorsStorage ServersHBASEOTHERDBSWorkloadExecutorThreadsStatsDBClientsWorkloadParameterFileCommand-lineParametersEXTENSIONSEXTENSIONS ACCUMULO
  8. 8. YCSB++: Distributed & parallel testsSwapnil Patil, CMU 8Multi-client, multi-phase coordination using ZooKeeper− Enables testing at large scales and testing asymmetric featuresStorage ServersHBASEOTHERDBSWorkloadExecutorThreadsStatsDBClientsWorkloadParameterFileCommand-lineParametersEXTENSIONSEXTENSIONSMULTI-PHASEACCUMULOYCSB clientsCOORDINATION
  9. 9. YCSB++: Collective monitoringSwapnil Patil, CMU 9OTUS monitor built on Ganglia [Ren2011]− Collects information fromYCSB, table stores, HDFS and OSStorage ServersHBASEOTHERDBSWorkloadExecutorThreadsStatsDBClientsWorkloadParameterFileCommand-lineParametersEXTENSIONSEXTENSIONSMULTI-PHASEACCUMULOYCSB clientsCOORDINATION OTUS MONITORING
  10. 10. Example ofYCSB++ debuggingSwapnil Patil, CMU 10OTUS collects fine-grained information− Both HDFS process andTabletServer process on same node02040608010000:00 04:00 08:00 12:00 16:00 20:00 00:00 04:000816243240CPUUsage(%)AvgNumberofStoreFilesPerTabletTime (Minutes)Monitoring Resource Usage and TableStore MetricsAccumulo Avg. StoreFiles per TabletHDFS DataNode CPU UsageAccumulo TabletServer CPU Usage
  11. 11. Outline•  Problem•  YCSB++ design•  Illustrative examples− YCSB++ on HBASE and ACCUMULO (Bigtable-like stores)•  Ongoing work and summarySwapnil Patil, CMU 11
  12. 12. Tablet ServersRecap of Bigtable-like table storesSwapnil Patil, CMU 12HDFS nodesTabletTNMemtable(Fewer)SortedIndexedFilesSortedIndexedFilesMINORCOMPACTIONMAJORCOMPACTIONWriteAheadLogDataInsertion1 23Write-path: in-memory buffering & async FS writes1) Mutations logged in memory tables (unsorted order)2) Minor compaction: Memtables -> sorted, indexed files in HDFS3) Major compaction: LSM-tree based file merging in backgroundRead-path: lookup both memtables and on-disk files
  13. 13. Apache ACCUMULOStarted at NSA; now an Apache Incubator project− Designed for for high-speed ingest and scan workloads− features in ACCUMULO− Iterator framework for user-specified programs placed inbetween different stages of the DB pipeline  E.g., Support joins and stream processing using iterators− Also supports fine-grained cell-level access controlSwapnil Patil, CMU 13
  14. 14. ILLUSTRATIVE EXAMPLE #1Analyzing thefast inserts vs. weak consistencytradeoff usingYCSB++Swapnil Patil, CMU 14
  15. 15. Client-side batch writingFeature: clients batch inserts, delay writes to server•  Improves insert throughput and latency•  Newly inserted data may not be immediately visible toother clientsSwapnil Patil, CMU 15⏏
Table store serversZooKeeperClusterManagerYCSB++Store clientBatchYCSB++Store clientCLIENT #1 CLIENT #2Read{K}
  16. 16. Batch writing improves throughput6 clients creating 9 million 1-Kbyte records on 6 servers− Small batches - high client CPU utilization, limits throughput− Large batches - saturate servers, limited benefit from batchingSwapnil Patil, CMU 16010203040506010 KB 100 KB 1 MB 10 MBInsertspersecond(1000s)Batch sizeHbase Accumulo
  17. 17. Table store serversZooKeeperBatch writing causes weak consistencySwapnil Patil, CMU 17Test setup: ZooKeeper-based client coordination•  Share producer-consumer queue between readers/writers•  R-W lag = delay before C2 can read C1’s most recent writeYCSB++Store clientBatchYCSB++Store client12 34CLIENT #1 CLIENT #2Insert{K:V}(106 records)EnqueueK(sample 1% records)PollanddequeueKRead{K}
  18. 18. Batch writing causes weak consistencyDeferred write wins, but lag can be ~100 seconds− (N%) = fraction of requests that needed multiple read()s− Implementation of batching affects the median latencySwapnil Patil, CMU 1800. 10 100 1000 10000 100000Fractionofrequestsread-after-write time lag (ms)(a) HBase: Time lag for different buffer sizes10 KB ( <1%)100 KB (7.4%)1 MB ( 17%)10 MB ( 23%) 10 100 1000 10000 100000Fractionofrequestsread-after-write time lag (ms)(b) Accumulo: Time lag for different buffer sizes10 KB ( <1%)100 KB (1.2%)1 MB ( 14%)10 MB ( 33%)
  19. 19. ILLUSTRATIVE EXAMPLE #2Benchmarkinghigh-speed ingest featuresusingYCSB++Swapnil Patil, CMU 19
  20. 20. Features for high-speed insertionsMost table stores have high-speed ingest features− Periodically insert large amounts of data or migrate olddata in bulk− Classic relational DB techniques applied to new storesTwo features: bulk loading and table pre-splitting•  Less data migration during inserts•  Engages more tablet servers immediately•  Need careful tuning and configuration [Sasha2002]Swapnil Patil, CMU 20⏏

  21. 21. 8-phase test setup: table bulk loadingBulk loading involves two steps− Hadoop-based data formatting− Importing store files into table storePre-load phase (1 and 2)− Bulk load 6M rows in an empty table− Goal: parallelism by engaging all serversLoad phase (4 and 5)− Load 48M new rows− Goal: study rebalancing during ingestR/U measurements (3, 6 and 7)− Correlate latency with rebalancing workSwapnil Patil, CMU 21Load (importing)Read/Update workloadLoad (re-formatting)Read/Update workloadSleepRead/Update workloadPre-Load (importing)Pre-Load (re-formatting)Phases12345678
  22. 22. Read latency affected by rebalancing workSwapnil Patil, CMU 22Load (importing)Read/Update workloadLoad (re-formatting)Read/Update workloadSleepRead/Update workloadPre-Load (importing)Pre-Load (re-formatting)Phases1234567811010010000 60 120 180 240 300AccumuloReadLatency(ms)Measurement Phase RunningTime (Seconds)R/U 1 (Phase 3) R/U 2 (Phase 6) R/U 3 (Phase 8)•  High latency after high insertion periods thatcause servers to rebalance (compactions)•  Latency drops after store is in a steady state
  23. 23. Rebalancing on ACCUMULO serversSwapnil Patil, CMU 23Load (importing)Read/Update workloadLoad (re-formatting)Read/Update workloadSleepRead/Update workloadPre-Load (importing)Pre-Load (re-formatting)Phases12345678•  OTUS monitor shows the server-sidecompactions during post-ingestmeasurement phases11010010000 300 600 900 1200 1500 1800Experiment RunningTime (sec)StoreFilesTabletsCompactions
  24. 24. HBASE is slower: Different compaction policiesSwapnil Patil, CMU 241101001000100000 60 120 180 240 300AccumuloReadLatency(ms)Measurement Phase RunningTime (Seconds)R/U 1 (Phase 3) R/U 2 (Phase 6) R/U 3 (Phase 8)11010010000 300 600 900 1200 1500 1800Accumulo Experiment RunningTime (sec)StoreFilesTabletsCompactions1101001000100000 60 120 180 240 300HBaseReadLatency(ms)Measurement Phase RunningTime (Seconds)11010010000 300 600 900 1200 1500 1800HBase Experiment RunningTime (sec)
  25. 25. Extending to table pre-splittingSwapnil Patil, CMU 25Tablepre-splittingtestLoadPre-loadPre-split into N rangesRead/Update workloadSleepRead/Update workloadLoad (importing)Read/Update workloadLoad (re-formatting)Read/Update workloadSleepRead/Update workloadPre-Load (importing)Pre-Load (re-formatting)BulkloadingtestPre-split a key range into N partitions to avoid splitting during insertion
  26. 26. Outline•  Problem•  YCSB++ design•  Illustrative examples•  Ongoing work and summarySwapnil Patil, CMU 26
  27. 27. Things not covered in this talkMore features: function shipping to servers− Data filtering at the servers− Fine-grained, cell-level access controlMoredetailsintheACMSOCC2011paperOngoing work− Analyze more table stores: Cassandra,CouchDB, MongoDB− Continue research through the new Intel Science andTechnology Center for Cloud Computing at CMU (withGaTech)Swapnil Patil, CMU 27
  28. 28. Summary:YCSB++ tool•  Tool for performance debugging and benchmarkingadvanced features using new extensions toYCSB•  Two case-studies: Apache HBASE and ACCUMULO•  Tool available at consistency semantics Distributed clients using ZooKeeperFast insertions (pre-splits & bulk loads) Multi-phase testing (with Hadoop)Server-side filtering New workload generators anddatabase client API extensionsFine-grained access control28Swapnil Patil, CMU