Welcome to Apache Hadoop's Teenage Years, Arun Murthy Keynote
•Download as PPTX, PDF•
0 likes•505 views
The document discusses the evolution of Apache Hadoop from its origins in 2006 to the present day. It describes how Hadoop has grown from early implementations focused on HDFS, MapReduce, and batch applications to a full ecosystem that enables enterprise interoperability, deployment in data centers and clouds, and assembly of modern data applications. The document also presents several talks that will take place at the conference, focusing on topics like YARN, security, data governance, and running Hadoop in the cloud.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Welcome to Apache Hadoop's Teenage Years, Arun Murthy Keynote
Similar to Welcome to Apache Hadoop's Teenage Years, Arun Murthy Keynote (20)
More from DataWorks Summit/Hadoop Summit
More from DataWorks Summit/Hadoop Summit (20)
Recently uploaded
Recently uploaded (20)
Welcome to Apache Hadoop's Teenage Years, Arun Murthy Keynote
- 1. Welcome to Apache Hadoop’s Teenage Years Arun Murthy Co-Founder Hortonworks
- 2. © Hortonworks Inc. 2011 – 2016. All Rights Reserved Our Hadoop Journey Begins… 1 ° ° ° ° ° ° N HDFS MapReduce Batch apps 2006
- 3. © Hortonworks Inc. 2011 – 2016. All Rights Reserved Today Our Hadoop Journey: Ecosystem Innovation Accelerates 2006 2011
- 4. © Hortonworks Inc. 2011 – 2016. All Rights Reserved Our Hadoop Journey: Enterprise Interoperability 2006 Today
- 5. © Hortonworks Inc. 2011 – 2016. All Rights Reserved Our Hadoop Journey: Data Center, Cloud, Everywhere! 2006 Today
- 6. © Hortonworks Inc. 2011 – 2016. All Rights Reserved Asad Khan Group Program Manager Microsoft Cloud Optimized Enterprise Ready Productivity Day 1
- 7. © Hortonworks Inc. 2011 – 2016. All Rights Reserved MUST BE 100X EASIER TO ASSEMBLE, DEPLOY, SECURE AND GOVERN MODERN APPS ARE DATA APPS WHAT’S NEXT?
- 8. © Hortonworks Inc. 2011 – 2016. All Rights Reserved Modern Credit Fraud Detection Application Data at Rest Data in Motion Adaptive Application Fueled by Real-time Actionable Intelligence Customer Service AnalystCustomer DATA SOURCES
- 9. © Hortonworks Inc. 2011 – 2016. All Rights Reserved Service Assemble HADOOP Select Engines and Services Wire Containers Secure and Operate Make Assembly of Modern Data Applications 100X Easier Container Container Container Security Admin UI Container Container Container Data Container Container Container Engine Container Container Container Service Container Container Container Service Engine Data UI
- 10. © Hortonworks Inc. 2011 – 2016. All Rights Reserved Vinod Vavilapalli Hadoop YARN Development Lead @Hortonworks Tuesday Breakout Session A Multi-Colored YARN: Apps and First-Class Support for Services Assemblies Demo
- 11. © Hortonworks Inc. 2011 – 2016. All Rights Reserved Repeatable and Portable Secure and Well Governed Modern Data Applications Must Be… Easy to Use and Operate
- 12. © Hortonworks Inc. 2011 – 2016. All Rights Reserved How to Secure and Govern Access to Your Data? Classification Prohibition Time Location Streams Pipelines Feeds Hive Tables HDFS Files HBase Tables Entities in Data Lake Policies ?
- 13. © Hortonworks Inc. 2011 – 2016. All Rights Reserved Secure and Govern Your Data with Tag-Based Access Policies Classification Prohibition Time Location Policies PDP Resource Cache Ranger Manage Access Policies and Audit Logs Track Metadata and Lineage Atlas Client Subscribers to Topic Gets Metadata Updates Atlas Metastore Tags Assets Entitles Streams Pipelines Feeds Hive Tables HDFS Files HBase Tables Entities in Data Lake
- 14. © Hortonworks Inc. 2011 – 2016. All Rights Reserved Learn More About What’s Coming Next… A Multi-Colored YARN: Apps and First-Class Support for Services What the #$* is a business catalog and why you need it! Fine-grained security for Spark and Hive (with LLAP and more) Keynote: The Elephant in the Clouds Hadoop in the Cloud – The What, Why and How from the Experts Tuesday: Wednesday: Thursday:
- 15. Thank You
Editor's Notes
- Speaker: Arun Murthy, Hortonworks Co-Founder Title: Welcome to Hadoop’s Teenage Years Duration: 25 minutes Abstract: In 2006, Apache Hadoop had its first line of code committed to what has become a breakthrough technology. A decade later, we are witness to open source innovation that has literally changed the face of business. Hadoop and related technologies have become the enterprise data platform, fueled by a rich ecosystem capable of supporting any application, any data, anywhere. Join Hortonworks Co-Founder Arun Murthy as he as he drills down into the current and future state of Hadoop and reviews community initiatives aimed at enabling the next wave of modern data applications that are well governed and easy to deploy on-premises and in the cloud.
- Our Hadoop journey began in 2006 focused on executing batch MapReduce jobs on petabytes of data. Yahoo’s decision to contribute Hadoop to the Apache Software Foundation was critical because a vibrant set of related technologies began to appear around Hadoop. [NEXT]
- Fast forward to 2011 and the concept of YARN began to emerge. Its goal? Enable Hadoop to move from its batch-only roots and become a data platform capable of running batch, interactive, and real-time applications. The emergence of YARN further accelerated the innovation around Hadoop with the emergence of Spark, Kafka, Storm, and many other projects that started life as Apache Incubator proposals. [NEXT]
- Enterprises began to deploy Hadoop at scale and for any enterprise-viable data platform, enterprise interoperability became a critical area of investment. The formation of the commercial ecosystem around accelerated. The ODPi was formed to focus on enterprise Hadoop adoption and enabling more apps on top of Hadoop. ODPi is a Linux Foundation initiative that has 30 members now consisting of multiple vendors, apps-developers and end-users. ODPi released its first common reference specs this year and we’re proud to be one of the first to be certified. [NEXT]
- So after 10 years, the Hadoop ecosystem is available everywhere. In the Data Center, within appliances, across public and private clouds. This maximizes choice for people interested in getting started with Hadoop and deploying it at scale for transformational use cases. [NEXT]
- Speaking of cloud, please join me in welcoming Asad Khan from Microsoft on stage to show how the cloud is making it easier for analysts and data scientists to get value from their data. [ASAD COMES ON STAGE AMID MILD APPLAUSE FROM THE AUDIENCE] Asad will use this slide to set the stage for the demo. Asad will do the demo. We will come back to this slide after Asad’s demo when Arun thanks him and his team for making Hadoop in the cloud a reality. [NEXT]
- 7
- 8
- We’ve been working within the community on enabling this notion of Assemblies that make it a lot easier to assemble and deploy these complex modern data applications. Docker containers play a key role in this process. [NEXT]
- So with that as context, let me hand things over to Vinod. Vinod thanks Arun, then dive into the demo starting with the main Ambari assemblies view where he highlights the different kinds of assemblies such as Credit Fraud application, Cybersecurity application powered by Metron, Log Search application, etc. He’ll then duck under the hood and look at the Log Search JSON files to show how the app, packaged as Docker containers, is wired together. He’ll then surf through the other highlights including showing the visual gauges of the running assemblies, etc., etc. 1. Start in Ambari Assemblies view and talk through list by mentioning Credit Fraud app, Metron Cybersecurity app, and Log Search app 2. Click on Log Search app 3. Change to tab that shows the JSON descriptor for the Log Search app, and briefly talk through 4. Go back to Ambari Assemblies view 5. Click on Credit Fraud app and talk through the components that power it (Spark, HBase, etc., etc.); click on the Deploy link to show how the user can deploy (cancel out of this since you already have the app running) 6. Change to tab that shows the Credit Fraud analyst app within the context of a given event (you should have double-clicked on a specific event so we can see the charts, map, etc.) 7. Let’s take a look under the hood by changing to tab that shows the Slider App Master view, and briefly talk through the running services 8. For those that prefer a command-line, we also have the HSHELL; goto shell and show some highlights 9. One final thing worth noting is the new YARN UI; change to tab that shows YARN UI and show some highlights 10. Go back to Ambari Assemblies view…and close out by saying…so those are the highlights of where things stand with Assemblies… If you’re interested in learning more, join me and my team members in our session later today: A Multi-Colored YARN: Apps and First-Class Support for Services We will come back to this slide after Vinod’s demo when Arun thanks him and the community for making helping to make it 100X easier to assemble and deploy modern data applications. [NEXT SLIDE]
- So Vinod showed you where we are with Assemblies, and the focus of that efforts is really aimed at making modern data applications easy to use and operate. Since assemblies are based on Docker containers, we can achieve a repeatable and portable experience. But our work is not done. Modern data applications muct be secure and well governed if enterprises are to embrace them. I’ll close out things out by touching on the community efforts focused in this area. [NEXT]
- So the enterprise has invested in integrating Hadoop into its data lake architecture. Landing petabyte of data from streams, pipelines, data feeds into HDFS files, Hive and HBase tables, etc. The question arises of how we can setup policies for these data sets that enable us to secure and govern access to it. [NEXT ]
- The community has been hard at work on integrating Apache Atlas as a metadata catalog and Apache Ranger as the centralized security system to address this need. The result is tag-based authorization model driven by the metadata catalog (i.e. Atlas) with access and audit policies applied to those tags (via Ranger). This enables a more flexible way to govern access to data and data sets than traditional role/group based access policies. Ex. as data pipelines land data, they can tag that data as the data lands and the access policies setup for those tags immediately apply. Moreover, Ranger has added the notions of time-based and location-based access policies, so users can do things like limit access to data that’s older than 90 days (for example) or limit access to data from certain geographies. This provides important enterprise-focused capabilities that will help businesses deploy more modern data applications in a way where they have the confidence their data is secure and well-governed. [NEXT]
- So in closing the past 10 years have been exciting, but I’m really looking forward to the next 10 years of innovation in the Hadoop ecosystem focused on the world of modern data applications. To learn more about what’s coming next, I encourage you to attend these sessions today, tomorrow, and Thursday where folks will go into more details. [NEXT]
- THANK YOU AND HAVE A GREAT CONFERENCE!