HCatalog is a table management and metadata layer for Hadoop that allows users to access data in different formats like RCFile and Hive in a uniform manner. It provides a metadata abstraction layer that allows data producers to declare schema for output data, and consumers to understand the schema to make sense of that data. This allows data sharing across different tools like Pig, MapReduce, Hive and more without needing to know the internal data representation. The project was recently accepted into the Apache incubator and the 0.1 release supports reading and writing data from Pig, MapReduce and Hive. Future plans include adding support for storing data in HBase and improving integration with other Hadoop components.

1. HCatalogTable Management for HadoopAlan F. Gates

2. Committer and mentor for Apache HCatalogCommitter and PMC member for Apache PigCo-founder HortonworksTwitter @alanfgatesWho Am I?Photo credit: Charles Dawley

3. MotivationPigHiveMap ReduceHCatSerDeCustomLoaderHiveColumnarLoaderRCFileInputFormatCustomInputFormatColumnarSerDeCustomSerDeHCatLoaderHCatInputFormatHCatalogRCFileStorageDriverCustomStorageDriverCustom FormatRCFile

4. More MotivationData WarehouseHiveBI ToolsAnalysisData FactoryPig/MapReducePipelinesIterative ProcessingResearch

5. End User Exampleraw = load ‘/rawevents/20100819/data’using MyLoader()as (ts:long, user:chararray, url:chararray);botless= filterraw byNotABot(user);…storeoutput into ‘/processedevents/20100819/data’;Processedevents consumersmust be manually informed by producer thatdata isavailable, or poll on HDFS (= bad for the NameNode)raw = load ‘rawevents’ using HCatLoader();botless= filterraw by date = ‘20100819’and NotABot(user);…storeoutput into ‘processedevents’using HCatStorage(“date=20100819”);Processedevents consumers will be notified by HCatalogdata is available and canthen start their jobs

6. Metadata ArchitectureHCatLoaderHCatStorageHCatInputFormatHCatOutputFormatCLINotificationHive metadata interfaceThrift serverRDBMS= Current HCatalog= Hive= Future HCatalog

7. Storage ArchitectureHCatLoaderHCatStorageHCatInputFormatHCatOutputFormatInputStorageDriverOutputStorageDriverHDFSHBase

8. Project StatusHCatalog was accepted to the Apache Incubator last March0.1 released this month, includesRead/write from PigRead/write from MapReduceRead/write from HiveWorks only with secure HadoopStorageDrivers for RCFile and Text

9. Future Plans0.2, plan to release in JulyNotification via JMS when data is availableStore to multiple partitions simultaneouslyImport/Export toolsLater this yearStoring in HBaseIntegration with Hadoop streamingBytearray/blob typeRCFile compression improvementsHigh Availability for Thrift serverEventuallyData management interfaces for archivers, cleaners, etc.Statistics storage

10. Get Involvedincubator.apache.org/hcatalogJoin the mailing lists User list: hcatalog-user@incubator.apache.orgDev list: hcatalog-dev@incubator.apache.org

11. Questions?