Evolving Hadoop Into An Operational Platform With Data Applications
•
2 likes•1,057 views
Hadoop Summit 2015
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Evolving Hadoop Into An Operational Platform With Data Applications
Similar to Evolving Hadoop Into An Operational Platform With Data Applications (20)
More from DataWorks Summit
More from DataWorks Summit (20)
Recently uploaded
Recently uploaded (20)
Evolving Hadoop Into An Operational Platform With Data Applications
- 1. @nmotgi Nitin Motgi Evolving Hadoop Into An Opera5onal Pla7orm With Data Applica5ons
- 2. PROPRIETARY & CONFIDENTIAL2 • Introduction to operational data applications • Challenges with building operational data applications on Hadoop • Goals and Motivation for CDAP • Introduction to CDAP and Architecture Overview • Building Blocks • Datasets • Programs • Application and Application Template • Use-‐cases Agenda
- 3. PROPRIETARY & CONFIDENTIAL3 Applications that use data insights to enhance the customers/user experience, achieve a business objective or improve a business process. What are Operational Data Applications?
- 4. PROPRIETARY & CONFIDENTIAL4 • 360-‐Degree Customer View • Recommendation Engine • Predictive Modeling • Fraud Analysis • Network Threat Detection • Telemetry • Time Series Analysis • And many more Examples
- 5. Challenges
- 6. Technology Explosion Core Hadoop HDFS, MR 2006 Hbase ZooKeeper Core Hadoop 2008 Hive Pig Mahout Hbase ZooKeeper Core Hadoop 2009 Sqoop Whirr Avro Hive Pig Mahout Hbase Zookeeper Core Hadoop 2010 Flume Bigtop Oozie MRUnit HCatalog Sqoop Whirr Avro Hive Pig Mahout Hbase Zookeeper Core Hadoop 2011 Spark Impala Solr Kafka Flume Bigtop Oozie MRUnit HCatalog Sqoop Whirr Avro Hive Pig Mahout Hbase Zookeeper Core Hadoop 2012 Sentry Tez Parquet YARN Spark YARN Impala Solr Kafka Flume Bigtop Oozie MRUnit HCatalog Sqoop Whirr Avro Hive Pig Mahout Hbase Zookeeper Core Hadoop Knox Present
- 7. APPLICATION COMPLEXITY MANY DOMAINS TO BRIDGE LOTS OF BOILERPLATE INCONSISTENT APIS NO REUSABILITY LACK OF DEVELOPER PRODUCTIVITY Challenges
- 9. Mo5va5on
- 10. Motivation • Simple yet powerful platform for developers to build applications on Hadoop • Expose capabilities rather than features • Make Hadoop accessible to developers with no Hadoop knowledge
- 11. Goals • Unified platform for building solutions on Hadoop • Simpler application development lifecycle • Reusable Data and Processing Patterns • Framework level correctness and consistency
- 12. Introduc5on to Cask Data Applica5on Pla7orm
- 13. An open source, integrated, distributed and extensible platform for building data applications on Hadoop. Cask Data Application Platform
- 14. Provides
- 15. Supports developers, operations, and organizations through the entire enterprise data application lifecycle. CASK DATA APP PLATFORM Data Lifecycle Ingest Explore Transform Serve Application Lifecycle Develop Test Deploy Scale Enterprise Lifecycle Secure Manage Monitor Operate Supports
- 16. Datasets Programs Tools & Experience • Standardized containers providing consistency for diverse processing paradigms • Services for developers to enable richer apps with less hassle; and production to enable application and data management • Libraries to build reusable data access patterns spanning multiple storage technologies Runtime Services 16 Programs Batch Programs Realtime Programs CASK DATA APPLICATION PLATFORM (CDAP) Event /Data Ingestion Tools and User Experience Datasets Runtime Services BATCH PROCESSING (MapReduce, Hive, pig) ANALYTIC SQL (Impala) SEARCH ENGINE (Cloudera Search) MACHINE LEARNING (Spark, MapReduce, Mahout) STREAM PROCESSING (Spark) 3RD PARTY APPS (Partners) DATA MANAGEMEN Egress Cloudera’s Enterprise Data Hub Adapters Data Application Examples Anomaly Detection 360o Consumer profile Network Analytics Multi-log Correlation Analytics Architecture HADOOP • Maven Archetype, Testing Framework, Debugging Tools, Monitoring Tool, Web based Application Management
- 17. 17 ServeTransformExploreIngest Unification ACID Dataset Streams Realtime - Tigon JDBC Query RPC SparkMR Dataset Dataset MR Spark Ad-hoc query Dataset API, SPI & Management Services Application Structure
- 18. 18 Deployment • Services • Master • Router • Auth Server CDAP Server • Highly Available (HA) • Installed on edge node(s) • Supports Kerberos - Impersonation & Permitter Security • Manager system services in YARN CDAP Server System Services (Twill Containers) • Transactions (Tephra) • Metrics Aggregation • Log Aggregation • Dataset Services • Metadata Management Service • Explore Service • Stream Management Service & more
- 19. Building Blocks
- 20. Building Blocks Dataset Program Encapsulated data access paBerns and data model in a reusable, domain-‐specific API Standardized containers for processing paradigms ProgramaTc abstracTon for composing mulTple Datasets and Programs that integrates ingesTon, exploraTon, transformaTon and serving Application Dataset ProgramProgramDataset
- 21. Dataset
- 22. PROPRIETARY & CONFIDENTIAL22 RDBMS Hadoop Dataset Raw Storage Interfaces, Data Modeling, Data Layout, OpTmizaTons and Schema Raw Storage Raw Distributed Storage, Model, Layout, Op5miza5ons and op5onal Schema • OpTmizaTons are pushed closer to storage • ApplicaTons use SQL to access data (store or retrieve) • Simpler ApplicaTons! • Modeling, layout and opTmizaTons are embedded within applicaTons • Hard to scale -‐ lack of reusability • Access through domain specific APIs with opTonal SQL Interface • OpTmizaTons are encapsulated within datasets • Simpler ApplicaTons! Dataset Motivation
- 23. PROPRIETARY & CONFIDENTIAL23 • Encapsulate a data access paBern and data model in a reusable, domain-‐specific API • Establishes best prac5ces in schema definiTon • Abstract away underlying storage plaorm • Reusable as data storage templates • Easy sharing of stored data: • Between applicaTons • Batch and real-‐Tme processing • Integrated with TransacTons for consistency • Integrated tes5ng • Extensible to create your own soluTons • Transparent Integra5on with • Hive metastore • MR Input/Output Formats • Spark RDDs Building Blocks - Dataset
- 24. PROPRIETARY & CONFIDENTIAL24 • Secondary Indexes • Example use case: Entity storage - store customer records indexed by location • Object Mapping • Example use case: Entity storage - easily store User instances for user profiles • Timeseries Data • Example use case: any data organized around a time dimension • Data Cube • Example use case: Retail product sales reports, web analytics • ParTToned Fileset • Example use case: Time partitioned processing of feeds • And many more Dataset - Types
- 25. PROPRIETARY & CONFIDENTIAL25 Dataset - Example • A Java Library • Table Dataset • First Name, Last Name and Link to Picture in a Table • Fileset Dataset • Pictures in a Fileset • Instance of Dataset as • HBase Table and • HDFS Directory • Access using SQL (HIVE) • Tigon, MR & Spark can access
- 26. public class ContactsDataset extends AbstractDataset { private ObjectMappedTable<Contact> contacts; private FileSet pictures; public ContactsDataset(DatasetSpecification spec, @EmbeddedDataset("contacts") ObjectMappedTable<Contact> contacts, @EmbeddedDataset("pictures") FileSet pictures) { super(spec.getName(), contacts, pictures); this.contacts = contacts; this.pictures = pictures; } public void addContact(String nick, Contact contact) { contacts.write(nick, contact); } public Contact getContact(String nick) { return contacts.read(nick); } // continued... PROPRIETARY & CONFIDENTIAL26 Dataset - Composite Embedded Datasets
- 27. PROPRIETARY & CONFIDENTIAL27 public class ContactsDataset extends AbstractDataset { // ...continued public void addPhoto(String nick, byte[] photoBytes) throws IOException { Contact contact = getContact(nick); if (contact.getPicturePath() != null) { // delete picture path } String picturePath = "pic." + nick; Location location = pictures.getLocation(picturePath); try { ByteStreams.copy(new ByteArrayInputStream(photoBytes), location.getOutputStream()); contact.setPicturePath(picturePath); contacts.write(nick, contact); } catch (IOException e) { LOG.error("Got exception: ", e); // delete path throw e; } } } Dataset - Transactional Update
- 28. PROPRIETARY & CONFIDENTIAL28 public class ContactsDataset extends AbstractDataset implements RecordScannable<StructuredRecord> { //.. @Override public Type getRecordType() { return StructuredRecord.class; } @Override public List<Split> getSplits() { return contacts.getSplits(); } @Override public RecordScanner<StructuredRecord> createSplitRecordScanner(Split split) { return contacts.createSplitRecordScanner(split); } } Dataset - Explorable
- 29. PROPRIETARY & CONFIDENTIAL29 Dataset Example - Usage public class Contacts extends AbstractApplication { @Override public void configure() { try { setName("Contacts"); setDescription("An application to manage contacts and their pictures"); createDataset("contacts", ContactsDataset.class); // Define programs, other datasets... } catch (UnsupportedTypeException e) { // cannot happen with Contact } } }
- 30. Programs
- 31. PROPRIETARY & CONFIDENTIAL31 • Standardized containers for processing paradigms • Establishes unified way of extracTng logs & metrics • Compose complex applicaTons -‐ real-‐5me or batch • Seamless Integra5on with Datasets -‐ simple or composite. • Provides conceptual integrity across different processing paradigms • Integrated end-‐to-‐end tes5ng • Extensible to add new processing paradigms. • Leverage common services to ease • version management • deployment • management Building Blocks - Programs
- 32. Applica5on
- 33. PROPRIETARY & CONFIDENTIAL33 ProgramaTc abstracTon for composing a use case by combining Datasets and Programs to perform ingesTon, transformaTon and serving. Building Blocks - Application public class PurchaseApp extends AbstractApplication { @Override public void configure() { . . . addStream(new Stream("purchaseStream")); createDataset("frequentCustomers", KeyValueTable.class); createDataset("userProfiles", KeyValueTable.class); addFlow(new PurchaseFlow()); addWorkflow(new PurchaseHistoryWorkflow()); addService(new PurchaseHistoryService()); addService(UserProfileServiceHandler.SERVICE_NAME, new UserProfileServiceHandler()); addService(new CatalogLookupService()); try { createDataset("history", PurchaseHistoryStore.class, PurchaseHistoryStore.properties()); ObjectStores.createObjectStore(getConfigurer(), "purchases", Purchase.class); } catch (UnsupportedTypeException e) { throw new RuntimeException(e); } } }
- 34. PROPRIETARY & CONFIDENTIAL34 • Is a use-‐case Blueprint • Composed using one or more Programs and Datasets • Supports real-‐5me or batch or combina5on • Highly reusable through configuraTon & extensible through plugins • Is an applicaTon that is reusable through configuraTon and extensible through plugins. • Plugins extend the ApplicaTon Template by implemenTng an interface expected by the template. • Support with an end to end tes5ng framework Building Blocks - Application Template Application Template Pluggable Interface Adapter1 Plugin Config1 Config2 Config3 Adapter2 Plugin Adapter3 Plugin
- 35. PROPRIETARY & CONFIDENTIAL35 • Scalable and reliable real-‐time business critical analytics • Closed Loop Recommendation and Analytics • Data Ingestion As A Service -‐ Realtime and Batch • Extendable and Reusable use-‐case blueprints • Data As A Service • Reduce application development and operational complexity • ETL Automation -‐ Real-‐time and Batch Use-cases
- 36. Want to Learn More? Open-source (Apache License v2) Website: http://cdap.io Mailing List: cdap-user@googlegroups.com cdap-dev@googlegroups.com IRC: #cdap on freenode.net
- 37. QUESTIONS? Want to work on these and other challenges? http://cask.co/careers/