Your SlideShare is downloading. ×
0
On HBase Integration Testing
On HBase Integration Testing
On HBase Integration Testing
On HBase Integration Testing
On HBase Integration Testing
On HBase Integration Testing
On HBase Integration Testing
On HBase Integration Testing
On HBase Integration Testing
On HBase Integration Testing
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

On HBase Integration Testing

1,848

Published on

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,848
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
15
Comments
0
Likes
5
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. On HBase Integration Testing Andrew Purtell, Hadoop Group at Trend Micro apurtell@apache.org
  • 2. Introducing Apache Bigtop (and other custom distros)
  • 3. Introducing Apache BigtopA new project in incubation with committers from Cloudera, Ubuntu, and othersThe goal is to do for Hadoop what Debian did for LinuxBuild and packaging infrastructure Makefile based (yes!) RPM and DEB packaging targetsDeployment infrastructure PuppetIntegration test infrastructure iTest
  • 4. Introducing Apache BigtopA new project in incubation with committers from Cloudera, Ubuntu, and othersThe goal is to do for Hadoop what Debian did for LinuxBuild and packaging infrastructure Makefile based (yes!) RPM and DEB packaging targetsDeployment infrastructure PuppetIntegration test infrastructure iTest This seems like it could be useful
  • 5. iTestA framework for integration testingProvides a high-level abstraction for querying and driving OS level packaging systems during full system integration tests Install package Run tests Check results Remove package Start/stop SysV servicesTransparent shell-outs written in Java and Groovy Convenient to write tests that are scripting (Groovy) layered over command line invocations ("hadoop jar ...")Helper functions for unpacking data from JAR files, filesystem paths, classpath, etc.
  • 6. HBASE-6201 (and friends)From Enis SöztutarAdd ability to run all integration/system tests locally or on a given cluster With mvn verify though Stack suggests this hasnt been possible to date using Maven machinery alone Or, by importing and using iTest (and exposing this also to BigTop) – I personally think this is promising MiniCluster mvn verify iTest Actual cluster
  • 7. HBASE-6201 (and friends)Convert candidate unit tests that run on a full MiniCluster into integration tests that can run on a real cluster TestAcidGuarantees TestAtomicOperation TestRegionBalancing TestFullLogReconstruction TestMasterFailover TestImportExport TestRollingRestart TestMultiVersions TestKeepDeletes TestFromClientSide TestSplitTransactionOnCluster ... and more
  • 8. HBASE-6201 (and friends)Add more tests, especially, long running ingestion tests and chaos monkey LoadTestTool YCSB GoraCI Resurrect Gremlins? https://github.com/toddlipcon/gremlins
  • 9. Going beyond HBASE-6201Add full system application test cases I have a Heritrix / hbase-writer based webtable thingy ? JGit backed by HBase https://github.com/apurtell/jgit-hbase ? Canned queries for Titan (http://thinkaurelius.github.com/titan/ ) or something that uses Apache Giraphs HBase storage adapter ? Canned data and scripts for Apache Pig and Hive ?Add Kerberos setup and usage helper functions to iTest so then we can finesse integration testing with security enabled at the Hadoop, HBase, and/or ZooKeeper layers
  • 10. End. Discuss.

×