On HBase Integration Testing  Andrew Purtell, Hadoop Group at Trend Micro             apurtell@apache.org
Introducing Apache Bigtop                            (and other custom                            distros)
Introducing Apache BigtopA new project in incubation with committers from Cloudera, Ubuntu, and othersThe goal is to do fo...
Introducing Apache BigtopA new project in incubation with committers from Cloudera, Ubuntu, and othersThe goal is to do fo...
iTestA framework for integration testingProvides a high-level abstraction for querying and  driving OS level packaging sys...
HBASE-6201 (and friends)From Enis SöztutarAdd ability to run all integration/system tests locally or on a given cluster  W...
HBASE-6201 (and friends)Convert candidate unit tests that run on a full MiniCluster into integration tests that can run on...
HBASE-6201 (and friends)Add more tests, especially, long running ingestion tests and chaos monkey  LoadTestTool  YCSB  Gor...
Going beyond HBASE-6201Add full system application test cases  I have a Heritrix / hbase-writer based webtable thingy ?  J...
End. Discuss.
Upcoming SlideShare
Loading in …5
×

On HBase Integration Testing

2,493 views

Published on

Published in: Technology

On HBase Integration Testing

  1. 1. On HBase Integration Testing Andrew Purtell, Hadoop Group at Trend Micro apurtell@apache.org
  2. 2. Introducing Apache Bigtop (and other custom distros)
  3. 3. Introducing Apache BigtopA new project in incubation with committers from Cloudera, Ubuntu, and othersThe goal is to do for Hadoop what Debian did for LinuxBuild and packaging infrastructure Makefile based (yes!) RPM and DEB packaging targetsDeployment infrastructure PuppetIntegration test infrastructure iTest
  4. 4. Introducing Apache BigtopA new project in incubation with committers from Cloudera, Ubuntu, and othersThe goal is to do for Hadoop what Debian did for LinuxBuild and packaging infrastructure Makefile based (yes!) RPM and DEB packaging targetsDeployment infrastructure PuppetIntegration test infrastructure iTest This seems like it could be useful
  5. 5. iTestA framework for integration testingProvides a high-level abstraction for querying and driving OS level packaging systems during full system integration tests Install package Run tests Check results Remove package Start/stop SysV servicesTransparent shell-outs written in Java and Groovy Convenient to write tests that are scripting (Groovy) layered over command line invocations ("hadoop jar ...")Helper functions for unpacking data from JAR files, filesystem paths, classpath, etc.
  6. 6. HBASE-6201 (and friends)From Enis SöztutarAdd ability to run all integration/system tests locally or on a given cluster With mvn verify though Stack suggests this hasnt been possible to date using Maven machinery alone Or, by importing and using iTest (and exposing this also to BigTop) – I personally think this is promising MiniCluster mvn verify iTest Actual cluster
  7. 7. HBASE-6201 (and friends)Convert candidate unit tests that run on a full MiniCluster into integration tests that can run on a real cluster TestAcidGuarantees TestAtomicOperation TestRegionBalancing TestFullLogReconstruction TestMasterFailover TestImportExport TestRollingRestart TestMultiVersions TestKeepDeletes TestFromClientSide TestSplitTransactionOnCluster ... and more
  8. 8. HBASE-6201 (and friends)Add more tests, especially, long running ingestion tests and chaos monkey LoadTestTool YCSB GoraCI Resurrect Gremlins? https://github.com/toddlipcon/gremlins
  9. 9. Going beyond HBASE-6201Add full system application test cases I have a Heritrix / hbase-writer based webtable thingy ? JGit backed by HBase https://github.com/apurtell/jgit-hbase ? Canned queries for Titan (http://thinkaurelius.github.com/titan/ ) or something that uses Apache Giraphs HBase storage adapter ? Canned data and scripts for Apache Pig and Hive ?Add Kerberos setup and usage helper functions to iTest so then we can finesse integration testing with security enabled at the Hadoop, HBase, and/or ZooKeeper layers
  10. 10. End. Discuss.

×