Your SlideShare is downloading. ×
Introduction to the Hortonworks YARN Ready Program
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Introduction to the Hortonworks YARN Ready Program

2,521
views

Published on

The recently launched YARN Ready Program will accelerate multi-workload Hadoop in the Enterprise. The program enables developers to integrate new and existing applications with YARN-based Hadoop. We …

The recently launched YARN Ready Program will accelerate multi-workload Hadoop in the Enterprise. The program enables developers to integrate new and existing applications with YARN-based Hadoop. We will cover:

--the program and it's benefits
--why it is important to customers
--tools and guides to help you get started
--technical resources to support you
--marketing recognition you can leverage

Published in: Technology

0 Comments
12 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,521
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
106
Comments
0
Likes
12
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Page1 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Introduction to YARN Ready John Kreisa – VP Strategic Marketing Ajay Singh - Director, Partner Solutions
  • 2. Page2 © Hortonworks Inc. 2011 – 2014. All Rights Reserved YARN Ready Program Overview John Kreisa
  • 3. Page3 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Why YARN Ready? -> Apache Hadoop YARN • YARN is the architecture center of Apache Hadoop 2 enabling more efficient cluster utilization • More and more customers are asking about tools and applications that are integrated into YARN to maximize the value of their Hadoop cluster • YARN Ready is a program offered by Hortonworks the leading developers of YARN –YARN was conceived of by Arun Murthy Hortonworks founder –He submitted JIRA MapReduce 279 in January 2008 and has been working on it since –Hortonworks has written 80% of the YARN code Page 3
  • 4. Page4 © Hortonworks Inc. 2011 – 2014. All Rights Reserved The Two YARN Ready Program Goals  Expand the Hadoop ecosystem by enabling ISVs to build & validate YARN-based applications, using Apache technologies  Allow enterprises to know which technologies maximize Hadoop resources in an ecosystem-friendly way
  • 5. Page5 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Two Achievements Available  HDP Certified  YARN Ready Badge Description HDP (Certification) Integrates with HDP 2.x YARN Ready (Validation) Uses the YARN resource scheduler
  • 6. Page6 © Hortonworks Inc. 2011 – 2014. All Rights Reserved YARN Technical Overview and Value Ajay Singh
  • 7. Page7 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Hadoop 1.0: Batch HADOOP 1.0 Built for Web-Scale Batch Apps Single App BATCH HDFS Single App INTERACTIVE Single App BATCH HDFS • All other usage patterns must leverage that same infrastructure • Forces the creation of silos for managing mixed workloads Single App BATCH HDFS Single App ONLINE
  • 8. Page8 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Hadoop 2.0: Multi Workload HADOOP 1.0 HDFS (redundant, reliable storage) MapReduce (cluster resource management & data processing) HDFS2 (redundant, reliable storage) YARN (cluster resource management) MapReduce (data processing) Others (data processing) HADOOP 2.0 Single Use System Batch Apps Multi Purpose Platform Batch, Interactive, Online, Streaming, …
  • 9. Page9 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Key Driver Of Hadoop Adoption: Enterprise Data Lake Flexible Enables other purpose-built data processing models beyond MapReduce (batch), such as interactive and streaming Efficient Double processing IN Hadoop on the same hardware while providing predictable performance & quality of service Shared Provides a stable, reliable, secure foundation and shared operational services across multiple workloads Data Processing Engines Run Natively IN Hadoop BATCH MapReduce INTERACTIVE Tez STREAMING Storm IN-MEMORY Spark GRAPH Giraph ONLINE HBase, Accumulo OTHERS HDFS: Redundant, Reliable Storage YARN: Cluster Resource Management
  • 10. Page10 © Hortonworks Inc. 2011 – 2014. All Rights Reserved YARN Development Framework YARN : Data Operating System °1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° °° ° ° ° ° ° ° ° ° ° ° ° ° ° N HDFS (Hadoop Distributed File System) System Batch MapReduce Interactive Tez Engine Long Running Slider Direct ISV Apps Scripting Pig SQL Hive Cascading Java Scala NoSQL HBase Accumulo Stream Storm API ISV Apps ISV Aps Applications Others Spark ISV Apps ISV Apps
  • 11. Page11 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Tez – Next Generation Data Processing Framework YARN ApplicationMaster to run DAG of Tez Tasks Task with pluggable Input, Processor and Output Tez Task - <Input, Processor, Output> Task ProcessorInput Output
  • 12. Page12 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Slider – Onboarding Long Running Services To YARN  A YARN application to deploy existing distributed applications, monitor them and make them larger or smaller as desired - even while the cluster is running  Has a plug-in provider architecture to support different applications  Automatic Deploy, Manage, Monitor via Ambari – Standard management APIs for Ambari to use – Standard ways to collect metrics  Clusters can be stopped, "frozen" and restarted, "thawed" later – Distribution of the deployed application across the YARN cluster is persisted.
  • 13. Page13 © Hortonworks Inc. 2011 – 2014. All Rights Reserved # of Apps / Ease of Integration Control Integration Option Tradeoffs YARN Native Apps Tez Apps Services on YARN (Slider) MapReduce Apps Hive
  • 14. Page14 © Hortonworks Inc. 2011 – 2014. All Rights Reserved HADOOP YARN HDFS SQL Datanode HDFS Visual Data & Analytics Workflow Actian Analytics Platform – YARN Native Integration Example Transform Hadoop into a High Performance Analytics Platform Datanode HDFS Datanode HDFS Datanode HDFS X100X100X100 Read Load Actian Vector Blend & Enrich Data Science & Analytics Datanode HDFS X100 HDFS Vector • Original file format • Standard block replication • Column-based blocks • Binary • Compressed • Partitioned • Faster Loading • Faster SQL • Standard SQL • Better Scaling Namenode High Performance, Industrialized SQL Database High Performance, Data Science & Analytics
  • 15. Page15 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Partners with YARN Ready Apps
  • 16. Page16 © Hortonworks Inc. 2011 – 2014. All Rights Reserved YARN Ready Program Details John Kreisa
  • 17. Page17 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Partner Benefits - Marketing  Recognition  Recognition as Hortonworks Certified Technology Partner on our website along with the YARN Ready badge  Hortonworks quote for reference in partner marketing assets  Promotion to customers in Hortonworks led events and activities  Joint Marketing & Field Alignment  Invitation to Modern Data Architecture Webinar Series  Hortonworks Sales Briefing and alignment with Field BD  Automatic participation in digital marketing initiatives  LinkedIn and FB campaigns  Twitter and blog activity
  • 18. Page18 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Partner Benefits - Technical  Tech Expertise  YARN implementation guide and sample code  Integration best practice guidance  Access to the architects and builders of YARN  Roadmap Input  YARN, Tez and Stinger roadmap visibility  Input into the direction of YARN and the YARN engines  Access to pre-release builds to speed integration with new features
  • 19. Page19 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Process for Partners  Become a Hortonworks partner  Sign up at http://hortonworks.com/partners/become-a-partner/  Become Certified  Download the certification kit (a .zip archive) from the partner portal  Validate with HDP (Sandbox OK)  Complete the provided templates (samples are included) for  Solution Overview  Reference Architecture  Certification Report  Submit the three documents to technology-certification@hortonworks.com.  SLA: 10 working days for Hortonworks response.  Collaborate with Hortonworks on joint marketing – via ChannelMarketing@Hortonworks.com
  • 20. Page20 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Certification Kit  A .zip file available to partners  Contents:  Program Guide  Solution Overview template (DOC)  Reference Architecture template (PPT)  Certification Report template & sample (DOC)  Sample YARN App code*  Sample Slider provider code*  YARN app test recommendations*  Logos (available after certification)  PR Template (available after certification)  README file (text) *YARN Ready, Coming – target July
  • 21. Page21 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Calls to Action  Engage us on your YARN initiative  Become HDP Certified and YARN Ready  Develop a joint Sandbox Tutorial  Collaborate on “Art of Possible” Demonstration  Co-Market & Co-Sell
  • 22. Page22 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Questions? Join us for the next series of webinars on integrating to YARN: YARN native July 24 @ 9am PT Slider integration August 7 @ 9am PT Tez integration August 21 @ 9am PT Page 22
  • 23. Page23 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Thank You