Hortonworks Yarn Code Walk Through January 2014

•Download as PPTX, PDF•

13 likes•4,407 views

This slide deck accompanies the Webinar recording YARN Code Walk through on Jan. 22, 2014, on Hortonworks.com/webinars under Past Webinars, or https://hortonworks.webex.com/hortonworks/lsr.php?AT=pb&SP=EC&rID=129468197&rKey=b645044305775657

YARN Code Overview
Ocular bleeding is no reason to stop programing!

© Hortonworks Inc. 2013

Page 1

Quick Bio – Joseph Niemiec
• Hadoop user for 2+ years
• 1 of 5 Author’s for Apache Hadoop YARN
• Originally used Hadoop for location based services
(March 2014)

– Destination Prediction
– Traffic Analysis
– Effects of weather at client locations on call center call types

• Pending Patent in Automotive/Telematics domain
• Defensive Paper on M2M Validation
• Started on analytics to be better at an MMORPG

© Hortonworks Inc. 2013

Agenda
• What Is YARN
• YARN Concepts & Architecture
• Code and more Code
• Q&A

© Hortonworks Inc. 2013

Page 3

From Batch To Anything
Single Use System

Multi Purpose Platform

Batch Apps

Batch, Interactive, Online, Streaming, …

HADOOP 1.0

HADOOP 2.0
MapReduce
(data processing)

MapReduce

Others
(data processing)

YARN

(cluster resource management
& data processing)

(cluster resource management)

HDFS

HDFS2

(redundant, reliable storage)

(redundant, reliable storage)

© Hortonworks Inc. 2013

Page 4

Concepts
• Application
–Application is a job submitted to the framework
–Examples
– Map Reduce Job
– MoYa Cluster

• Container
–Basic unit of allocation
–Fine-grained resource allocation across multiple resource
types (memory, cpu, disk, network, gpu etc.)
– container_0 = 2GB, 1CPU
– container_1 = 1GB, 6 CPU

–Replaces the fixed map/reduce slots

© Hortonworks Inc. 2013

5

Architecture
• Resource Manager
–Global resource scheduler
–Hierarchical queues

• Node Manager
–Per-machine agent
–Manages the life-cycle of container
–Container resource monitoring

• Application Master
–Per-application
–Manages application scheduling and task execution
–E.g. MapReduce Application Master
© Hortonworks Inc. 2013

6

To the code!

© Hortonworks Inc. 2013

Page 7

YARN - ApplicationMaster
• ApplicationMaster
– ApplicationSubmissionContext is the complete specification of the
ApplicationMaster, provided by Client
– ResourceManager responsible for allocating and launching
ApplicationMaster container

ApplicationSubmissionContext
resourceRequest
containerLaunchContext
appName
queue

© Hortonworks Inc. 2013

Page 9

YARN – Resource Allocation & Usage
• ContainerLaunchContext
– The context provided by ApplicationMaster to NodeManager to
launch the Container
– Complete specification for a process
– LocalResource used to specify container binary and
dependencies
– NodeManager responsible for downloading from shared namespace
(typically HDFS)

ContainerLaunchContext
container
commands
environment
localResources

LocalResource
uri
type

© Hortonworks Inc. 2013

Page 10

YARN – Resource Allocation & Usage
• ResourceRequest

priority

1

© Hortonworks Inc. 2013

<4gb, 1 core>

numContainers
1

rack0

1

*

<2gb, 1 core>

resourceName
host01

0

capability

1

*

1

Page 11

YARN – Resource Allocation & Usage
• Container
– The basic unit of allocation in YARN
– The result of the ResourceRequest provided by ResourceManager
to the ApplicationMaster
– A specific amount of resources (cpu, memory etc.) on a specific
machine
Container
containerId
resourceName
capability

tokens

© Hortonworks Inc. 2013

Page 12

Apache Solr is the open source platform for searching data stored in Hadoop. Solr powers search on many of the world's largest Internet sites, enabling powerful full-text search and near real-time indexing. Whether users search for tabular, text, geo-location or sensor data in Hadoop, they find it quickly with Apache Solr. Hortonworks Data Platform 2.1 includes Apache Solr. In this deck from their 30-minute webinar, Rohit Bakhshi, Hortonworks product manager, and Paul Codding, Hortonworks solution engineer describe how Solr works within HDP's YARN-based architecture.

Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS

Hortonworks

This is the presentation from the "Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS" webinar on May 28, 2014. Rohit Bahkshi, a senior product manager at Hortonworks, and Vinod Vavilapalli, PMC for Apache Hadoop, discuss an overview of YARN in HDFS and new features in HDP 2.1. Those new features include: HDFS extended ACLs, HTTPs wire encryption, HDFS DataNode caching, resource manager high availability, application timeline server, and capacity scheduler pre-emption.

Discover HDP 2.2: Even Faster SQL Queries with Apache Hive and Stinger.next

Hortonworks

Earlier this year, the Apache open source community delivered the Stinger Initiative to improve speed, scale and SQL semantics in Apache Hive. Now Stinger.next is underway, to build on those initial successes. In this presentation, from a webinar hosted by Hortonworks co-founder Alan Gates and Hortonworks Hive product manager Raj Baines, you can learn more about Stinger.next and innovation in Apache Hive. Alan and Raj cover new Hive functionality for more speed, scale and SQL in HDP 2.2. Specific topics include transactions with ACID semantics, the cost based optimizer and dynamic query optimizations. The presentation also shows future plans for the Stinger.next initiative.

Rescue your Big Data from Downtime with HP Operations Bridge and Apache Hadoop

Hortonworks

YARN Ready: Integrating to YARN with Tez

Hortonworks

Discover HDP 2.1: Apache Falcon for Data Governance in Hadoop

Hortonworks

Beginning with HDP 2.1, Hortonworks Data Platform ships with Apache Falcon for Hadoop data governance. Himanshu Bari, Hortonworks senior product manager, and Venkatesh Seetharam, Hortonworks co-founder and committer to Apache Falcon, lead this 30-minute webinar, including: + Why you need Apache Falcon + Key new Falcon features + Demo: Defining data pipelines with replication; policies for retention and late data arrival; managing Falcon server with Ambari

Enrich a 360-degree Customer View with Splunk and Apache Hadoop

Hortonworks

What if your organization could obtain a 360 degree view of the customer across offline, online and social and mobile channels? Attend this webinar with Splunk and Hortonworks and see examples of how marketing, business and operations analysts can reach across disparate data sets in Hadoop to spot new opportunities for up-sell and cross-sell. We'll also cover examples of how to measure buyer sentiment and changes in buyer behavior. Along with best practices on how to use data in Hadoop with Splunk to assign customer influence scores that online, call-center, and retail branches can use to customize more compelling products and promotions.

An organization’s information is spread across multiple repositories, on-premise and in the cloud, with limited ability to correlate information and derive insights. The Smart Content Hub solution from HP and Hortonworks enables a shared content infrastructure that transparently synchronizes information with existing systems and offers an open standards-based platform for deep analysis and data monetization. - Leverage 100% of your data: Text, images, audio, video, and many more data types can be automatically consumed and enriched using HP Haven (powered by HP IDOL and HP Vertica), making it possible to integrate this valuable content and insights into various line of business applications. - Democratize and enable multi-dimensional content analysis: - Empower your analysts, business users, and data scientists to search and analyze Hadoop data with ease, using the 100% open source Hortonworks Data Platform. - Extend the enterprise data warehouse: Synchronize and manage content from content management systems, and crack open the files in whatever format they happen to be in. - Dramatically reduce complexity with enterprise-ready SQL engine: Tap into the richest analytics that support JOINs, complex data types, and other capabilities only available with HP Vertica SQL on the Hortonworks Data Platform. Speakers: - Ajay Singh, Director, Technical Channels, Hortonworks - Will Gardella, Product Management, HP Big Data

Hortonworks and HP Vertica Webinar

Hortonworks

Discover HDP 2.2: Apache Falcon for Hadoop Data Governance

Hortonworks

Hortonworks Data Platform 2.2 includes Apache Falcon for Hadoop data governance. In this 30-minute webinar, we discussed why the enterprise needs Falcon for governance, and demonstrated data pipeline construction, policies for data retention and management with Ambari. We also discussed new innovations including: integration of user authentication, data lineage, an improved interface for pipeline management, and the new Falcon capability to establish an automated policy for cloud backup to Microsoft Azure or Amazon S3.

Hortonworks Technical Workshop: Real Time Monitoring with Apache Hadoop

Hortonworks

Real Time Monitoring requires a high scalable infrastructure of message bus, database, distributed event processing and scalable analytics engine. By bringing together leading open source projects of Apache Kafka, Apache HBase, Apache Storm and Apache Hive, the Hortonworks Data Platform offers a comprehensive Real Time Analysis platform. In this session, we will provide an in-depth overview all the key technology components and demonstrate a working solution for monitoring a fleet of trucks. Audience: Developers, Architects and System Engineers from the Hortonworks Technology Partner community. Recording: https://hortonworks.webex.com/hortonworks/lsr.php?RCID=0278dc8aa49a9991e1ce436c71f53d30

Webinar - Accelerating Hadoop Success with Rapid Data Integration for the Mod...

Hortonworks

Many enterprises are turning to Apache Hadoop to enable Big Data Analytics and reduce the costs of traditional data warehousing. Yet, it is hard to succeed when 80% of the time is spent on moving data and only 20% on using it. It’s time to swap the 80/20! The Big Data experts at Attunity and Hortonworks have a solution for accelerating data movement into and out of Hadoop that enables faster time-to-value for Big Data projects and a more complete and trusted view of your business. Join us to learn how this solution can work for you.

Combine SAS High-Performance Capabilities with Hadoop YARN

Hortonworks

What if you could assemble all your data in one system and run your critical analytic applications in parallel, regardless of the format, age or location of the data? Today, thanks to the economics of Apache Hadoop-based data platforms, in particular YARN, this is possible. SAS applications bring highly advanced, in-memory analytic processing to the data in Hadoop and enable a rich set of additional use cases with high performance analytic needs. Join this webinar and learn how, by combining their solutions, Hortonworks and SAS offer more flexibility to choose best of breed SAS HPA and LASR analytic applications in conjunction with trusted Hadoop workloads. Hear how it is possible to leverage Hadoop clusters to extend the power of SAS analytics. You will hear directly from our experts how SAS HPA and LASR have been integrated with Hadoop YARN to: Enable a modern data architecture without the need for fragmented processing clusters for each workload. Ensure low-latency local data access directly from the data nodes. Create Unified Resource Management window-panes for managing SAS HPA, LASR and HDP resources. Speakers: Arun Murthy, Founder and Architect at Hortonworks Arun is a Apache Hadoop PMC member and has been a full time contributor to the project since the inception in 2006. He is also the lead of the MapReduce project and has focused on building NextGen MapReduce (YARN). Prior to co-founding Hortonworks, Arun was responsible for all MapReduce code and configuration deployed across the 42,000+ servers at Yahoo! He jointly holds the current world sorting record using Apache Hadoop. Paul Kent, Vice President Big Data at SAS Paul Kent is Vice President of Big Data initiatives at SAS. He spends his time between Customers, Partners and the Research & Development teams discussing, evangelizing and developing software at the confluence of big data and high performance computing. A datacenter rack full of current-generation 64bit x86 processors represents a very large aggregate memory space, thousands of threads and plentiful IO that can be harnessed to solve problems at a much larger scale than we have traditionally been accustomed to.

Discover.hdp2.2.h base.final[2]

Hortonworks

State of the Union with Shaun Connolly

Hortonworks

In 2012, we released Hortonworks Data Platform powered by Apache Hadoop and established partnerships with major enterprise software vendors including Microsoft and Teradata that are making enterprise ready Hadoop easier and faster to consume. As we start 2013, we invite you to join us for this live webinar where Shaun Connolly, VP of Strategy at Hortonworks, will cover the highlights of 2012 and the road ahead in 2013 for Hortonworks and Apache Hadoop.

YARN webinar series: Using Scalding to write applications to Hadoop and YARN

Hortonworks

Enterprise Hadoop with Hortonworks and Nimble Storage

Hortonworks

Join us to learn how Hortonworks Data Platform and Nimble Storage provide an enterprise-ready data platform for multi-workload data processing. HDP supports an array of processing methods — from batch through interactive to real-time, with key capabilities required of an enterprise data platform — spanning Governance, Security and Operations. Nimble Storage provides the performance, capacity, and availability for HDP and allows you to take advantage of Hadoop with minimal changes to existing data architectures and skillsets.

Optimizing your Modern Data Architecture - with Attunity, RCG Global Services...

Hortonworks

Today’s enterprises are challenged with capturing large amounts of data from a number of sources in a variety of formats, and then storing it in a cost-effective, timely manner. With your current data warehouse, this may seem overwhelming. It doesn’t have to be. With a Hadoop-based modern data warehouse, you can overcome these challenges and get meaningful insights from real-time data. Want to learn how? Join experts from Attunity, Hortonworks, and RCG Global Services for a live webinar - where we will be discussing enterprise data warehouse optimization. You will learn how to: •Rebalance your data warehouse by identifying unused data and resource-intensive workloads that can be moved to Hadoop. •Seamlessly integrate your current enterprise data warehouse with a Modern Data Architecture. •Better utilize data assets to reduce costs while realizing more value from your data. •Develop a roadmap for implementing the Hadoop-based Modern Data Architecture and Data Lake.

Discover hdp 2.2: Data storage innovations in Hadoop Distributed Filesystem (...

Hortonworks

Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) ...

Hortonworks

No matter if you are new to Hadoop or have a mature cluster in production, scale will be a critical factor of your success with Hadoop. Are you ready to take the next big step as you scale out your data architecture? Talend and Hortonworks discuss where we will help you learn how to implement an effective big data and Hadoop strategy across your IT infrastructure. You will learn: How to grow a pilot into production How to scale-out architecture & systems affordably How to leverage the flexibility of Hadoop to optimize your data integration processes Recording: http://www.talend.com/resources/webinars/starting-small-and-scaling-big-with-hadoop

Discover HDP 2.1: Interactive SQL Query in Hadoop with Apache Hive

Hortonworks

In February 2013, the open source community launched the Stinger Initiative to improve speed, scale and SQL semantics in Apache Hive. After thirteen months of constant, concerted collaboration (and more than 390,000 new lines of Java code) Stinger is complete with Hive 0.13. In this presentation, Carter Shanklin, Hortonworks director of product management, and Owen O'Malley, Hortonworks co-founder and committer to Apache Hive, discuss how Hive enables interactive query using familiar SQL semantics.

Apache Hadoop on the Open Cloud

Hortonworks

Hp Converged Systems and Hortonworks - Webinar Slides

Hortonworks

Our experts will walk you through some key design considerations when deploying a Hadoop cluster in production. We'll also share practical best practices around HP and Hortonworks Data Platform to get you started on building your modern data architecture. Learn how to: - Leverage best practices for deployment - Choose a deployment model - Design your Hadoop cluster - Build a Modern Data Architecture and vision for the Data Lake

Supporting Financial Services with a More Flexible Approach to Big Data

Hortonworks

Financial services companies can reap tremendous benefits from 'Big Data' and they have moved quickly to deploy it. But these companies also place heavy demands on 'Big Data' infrastructure for flexibility, reliability and performance. In this webinar, Hortonworks joins WANDisco to look at three examples of using 'Big Data' to get a more comprehensive view of customer behavior and activity in the banking and insurance industries. Then we'll pull out the common threads from these examples, and see how a flexible next-generation Hadoop architecture lets you get a step up on improving your business performance. Join us to learn: How to leverage data from across an entire global enterprise How to analyze a wide variety of structured and unstructured data to get quick, meaningful answers to critical questions What industry leaders have put in place

C-BAG Big Data Meetup Chennai Oct.29-2014 Hortonworks and Concurrent on Casca...

Hortonworks

Big Data is moving to the next level of maturity and it’s all about the applications. Dhruv Kumar, one of the minds behind Cascading, the most widely used and deployed development framework for building Big Data applications, will discuss how Cascading can enable developers to accelerate the time to market for their data applications, from development to production. In this session, Dhruv will introduce how to easily and reliably develop, test, and scale your data applications and then deploy them on Hadoop and Hortonworks Data Platform. He will show a demo using the Hortonworks Sandbox and Cascading. Recording is at https://hortonworks.webex.com/hortonworks/lsr.php?RCID=e5582bcbc0516d35fc2dcf0bce86146e

Hortonworks and Platfora in Financial Services - Webinar

Hortonworks

Big Data Analytics is transforming how banks and financial institutions unlock insights, make more meaningful decisions, and manage risk. Join this webinar to see how you can gain a clear understanding of the customer journey by leveraging Platfora to interactively analyze the mass of raw data that is stored in your Hortonworks Data Platform. Our experts will highlight use cases, including customer analytics and security analytics. Speakers: Mark Lochbihler, Partner Solutions Engineer at Hortonworks, and Bob Welshmer, Technical Director at Platfora

YARN Ready - Integrating to YARN using Slider Webinar

Hortonworks

Hortonworks Technical Workshop - build a yarn ready application with apache ...

Hortonworks

YARN has fundamentally transformed the Hadoop landscape. It has opened hadoop from a single workload system to one that can now support a multitude of fit for purpose processing. In this workshop we will provide an overview of Apache Slider that enables custom applications to run natively in the cluster as a YARN Ready Application. The workshop will include working examples and provide an overview of work being pursued in the community around YARN Docker integration.

What's hot

Splunk-hortonworks-risk-management-oct-2014

Hortonworks

Create a Smarter Data Lake with HP Haven and Apache Hadoop

Hortonworks

Hortonworks and HP Vertica Webinar

Hortonworks

Discover HDP 2.2: Apache Falcon for Hadoop Data Governance

Hortonworks

Hortonworks Technical Workshop: Real Time Monitoring with Apache Hadoop

Hortonworks

Webinar - Accelerating Hadoop Success with Rapid Data Integration for the Mod...

Hortonworks

Combine SAS High-Performance Capabilities with Hadoop YARN

Hortonworks

Discover.hdp2.2.h base.final[2]

Hortonworks

State of the Union with Shaun Connolly

Hortonworks

YARN webinar series: Using Scalding to write applications to Hadoop and YARN

Hortonworks

Enterprise Hadoop with Hortonworks and Nimble Storage

Hortonworks

Optimizing your Modern Data Architecture - with Attunity, RCG Global Services...

Hortonworks

Discover hdp 2.2: Data storage innovations in Hadoop Distributed Filesystem (...

Hortonworks

Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) ...

Hortonworks

Discover HDP 2.1: Interactive SQL Query in Hadoop with Apache Hive

Hortonworks

Apache Hadoop on the Open Cloud

Hortonworks

Hp Converged Systems and Hortonworks - Webinar Slides

Hortonworks

Supporting Financial Services with a More Flexible Approach to Big Data

Hortonworks

C-BAG Big Data Meetup Chennai Oct.29-2014 Hortonworks and Concurrent on Casca...

Hortonworks

Hortonworks and Platfora in Financial Services - Webinar

Hortonworks

What's hot (20)

Splunk-hortonworks-risk-management-oct-2014

Create a Smarter Data Lake with HP Haven and Apache Hadoop

Hortonworks and HP Vertica Webinar

Discover HDP 2.2: Apache Falcon for Hadoop Data Governance

Hortonworks Technical Workshop: Real Time Monitoring with Apache Hadoop

Webinar - Accelerating Hadoop Success with Rapid Data Integration for the Mod...

Combine SAS High-Performance Capabilities with Hadoop YARN

Discover.hdp2.2.h base.final[2]

State of the Union with Shaun Connolly

YARN webinar series: Using Scalding to write applications to Hadoop and YARN

Enterprise Hadoop with Hortonworks and Nimble Storage

Optimizing your Modern Data Architecture - with Attunity, RCG Global Services...

Discover hdp 2.2: Data storage innovations in Hadoop Distributed Filesystem (...

Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) ...

Discover HDP 2.1: Interactive SQL Query in Hadoop with Apache Hive

Apache Hadoop on the Open Cloud

Hp Converged Systems and Hortonworks - Webinar Slides

Supporting Financial Services with a More Flexible Approach to Big Data

C-BAG Big Data Meetup Chennai Oct.29-2014 Hortonworks and Concurrent on Casca...

Hortonworks and Platfora in Financial Services - Webinar

Viewers also liked

YARN Ready - Integrating to YARN using Slider Webinar

Hortonworks

Hortonworks Technical Workshop - build a yarn ready application with apache ...

Hortonworks

Developing YARN Applications - Integrating natively to YARN July 24 2014Hortonworks

Discover HDP2.1: Apache Storm for Stream Data Processing in Hadoop

Hortonworks

For the first time, Hortonworks Data Platform ships with Apache Storm for processing stream data in Hadoop. In this presentation, Himanshu Bari, Hortonworks senior product manager, and Taylor Goetz, Hortonworks engineer and committer to Apache Storm, cover Storm and stream processing in HDP 2.1: + Key requirements of a streaming solution and common use cases + An overview of Apache Storm + Q & A

Get Started Building YARN Applications

Hortonworks

Dataguise hortonworks insurance_feb25

Hortonworks

Insurance companies of all sizes are challenged to keep up with emerging technologies that deliver a competitive advantage. Recording: https://www.brighttalk.com/webcast/9573/192877 Big data holds the key to greater customer insight and stronger customer relationships. But risk of sensitive data exposure — and compliance violations — keeps many insurers from pursuing big data initiatives and reaping the rewards of business-driven analytics. Join Dataguise and Hortonworks for this live webinar to learn how you can free your organization from traditional information security constraints and unlock the power of your most valuable business assets. • What do you need to know about PII/PHI privacy before embarking on big data initiatives? • Why do so many big data initiatives fail before they’ve even begun—and what can you do about it? • How can IT security organizations help data scientists extract more business value from their data? • How are leading insurance companies leveraging big data to gain competitive advantage?

Hortonworks sqrrl webinar v5.pptx

Hortonworks

Almost every week, news of a proprietary or customer data breach hits the news wave. While attackers have increased the level of sophistication in their tactics, so too have organizations advanced in their ability to build a robust, data-driven defense. Join Hortonworks and Sqrrl to learn how a Modern Data Architecture with Hortonworks Data Platform (HDP) and Sqrrl Enterprise enables intuitive exploration, discovery, and pattern recognition over your big cybersecurity data. In this webinar you will learn: --How Apache Hadoop makes it the perfect fit to accumulate cybersecurity data and diagnose the latest attacks --The effective ways for pinpointing and reasoning about correlated events within your data, and assessing your network security posture. --How a Modern Data Architecture that includes the power of Hadoop with Hortonworks Data Platform with the massive, secure, entity-centric data models in Sqrrl Enterprise can discover hidden patterns and detect anomalies within your data using linked data analysis.

Apache Ambari: Managing Hadoop and YARN

Hortonworks

Accelerate Big Data Application Development with Cascading and HDP, Hortonwor...

Hortonworks

Discover HDP 2.1: Using Apache Ambari to Manage Hadoop Clusters

Hortonworks

YARN Ready: Apache Spark

Hortonworks

http://hortonworks.com/hadoop/spark/ Recording: https://hortonworks.webex.com/hortonworks/lsr.php?RCID=03debab5ba04b34a033dc5c2f03c7967 As the ratio of memory to processing power rapidly evolves, many within the Hadoop community are gravitating towards Apache Spark for fast, in-memory data processing. And with YARN, they use Spark for machine learning and data science use cases along side other workloads simultaneously. This is a continuation of our YARN Ready Series, aimed at helping developers learn the different ways to integrate to YARN and Hadoop. Tools and applications that are YARN Ready have been verified to work within YARN.

Discover.hdp2.2.storm and kafka.final

Hortonworks

Discover HDP 2.2: Comprehensive Hadoop Security with Apache Ranger and Apache...

Hortonworks

This presentation was included in a 30-minute webinar Balaji Ganesan, Hortonworks senior director for enterprise security strategy and Vinay Shukla, director of product management. They discussed Hortonworks Data Platform 2.2’s features for delivering comprehensive security in HDP. Balaji and Vinay discussed Apache Ranger and Apache Knox and how they are integrated in HDP 2.2 to provide fine grain authorization, auditing and API security that can be centrally administered.

Leverage Big Data to Enhance Customer Experience in Telecommunications – with...

Hortonworks

Take a look at this presentation from Hortonworks and Skytree and learn how Communications Service Providers can enhance their customers experience by: – Creating a Data Lake for a 360 degree customer view. – Building dynamic customer profiles. – Leveraging a next-best-action streaming engine. You will learn more about how Hortonworks Hadoop Distribution Platform and Skytree Machine Learning Solution can help you do so. Speakers: Dr. Alexander Gray, CTO at Skytree, and Sanjay Kumar, General Manager, Hortonworks

Hortonworks tech workshop in-memory processing with spark

Hortonworks

Apache Spark offers unique in-memory capabilities and is well suited to a wide variety of data processing workloads including machine learning and micro-batch processing. With HDP 2.2, Apache Spark is a fully supported component of the Hortonworks Data Platform. In this session we will cover the key fundamentals of Apache Spark and operational best practices for executing Spark jobs along with the rest of Big Data workloads. We will also provide a working example to showcase micro-batch and machine learning processing using Apache Spark.

HPE and Hortonworks join forces to Deliver Healthcare Transformation

Hortonworks

Hortonworks Data In Motion Webinar Series Pt. 2

Hortonworks

How Hortonworks DataFlow (HDF), powered by Apache NIFi, MiNiFi, Kafka and Storm, and it’s associated HDF Certification Program make it easier and faster to integrate different systems together. Highlights on the latest partner integrations from HPE, SAS, Attunity, Impetus Technologies, Kepware and Midfin Systems. “ Watch the webinar on-demand: http://hortonworks.com/webinar/make-big-data-ecosystem-work-better/ HDF Partner certification program: http://hortonworks.com/partners/product-integration-certification/#hdf-integration

Hortonworks Data in Motion Webinar Series - Part 1

Hortonworks

Hortonworks Technical Workshop: Apache Ambari

Hortonworks

Apache Ambari is the only 100% open source management and provisioning tool for Apache Hadoop and Hortonworks Data Platform (HDP). Recent innovations of Apache Ambari have focused on opening Apache Ambari into a pluggable management platform that can automate cluster provisioning, deploy 3rd party software and provide custom operational and developers views to the end user. In this session Hortonworks will cover 3 key integration points of Apache Ambari including Stacks, Views and Blueprints and deliver working examples of each.

Viewers also liked (19)

YARN Ready - Integrating to YARN using Slider Webinar

Hortonworks Technical Workshop - build a yarn ready application with apache ...

Developing YARN Applications - Integrating natively to YARN July 24 2014

Discover HDP2.1: Apache Storm for Stream Data Processing in Hadoop

Get Started Building YARN Applications

Dataguise hortonworks insurance_feb25

Hortonworks sqrrl webinar v5.pptx

Apache Ambari: Managing Hadoop and YARN

Accelerate Big Data Application Development with Cascading and HDP, Hortonwor...

Discover HDP 2.1: Using Apache Ambari to Manage Hadoop Clusters

YARN Ready: Apache Spark

Discover.hdp2.2.storm and kafka.final

Discover HDP 2.2: Comprehensive Hadoop Security with Apache Ranger and Apache...

Leverage Big Data to Enhance Customer Experience in Telecommunications – with...

Hortonworks tech workshop in-memory processing with spark

HPE and Hortonworks join forces to Deliver Healthcare Transformation

Hortonworks Data In Motion Webinar Series Pt. 2

Hortonworks Data in Motion Webinar Series - Part 1

Hortonworks Technical Workshop: Apache Ambari

Similar to Hortonworks Yarn Code Walk Through January 2014

Developing Applications with Hadoop 2.0 and YARN by Abhijit Lele

Hakka Labs

Apache Hadoop YARN: best practicesDataWorks Summit

Yarnthug2014

Joseph Niemiec

YARN - Presented At Dallas Hadoop User Group

Rommel Garcia

Apache Hadoop YARN - Enabling Next Generation Data Applications

Hortonworks

Running Non-MapReduce Big Data Applications on Apache Hadoop

hitesh1892

Apache Hadoop has become popular from its specialization in the execution of MapReduce programs. However, it has been hard to leverage existing Hadoop infrastructure for various other processing paradigms such as real-time streaming, graph processing and message-passing. That was true until the introduction of Apache Hadoop YARN in Apache Hadoop 2.0. YARN supports running arbitrary processing paradigms on the same Hadoop cluster. This allows for development of newer frameworks as well as more efficient implementations of existing frameworks that can all run on and share the resources of a single multi-tenant YARN cluster. This talk gives a brief introduction to YARN. We will illustrate how to create applications and how to best make use of YARN. We will show examples of different applications such as Apache Tez and Apache Samza that can leverage YARN and present best practices/guidelines on building applications on top of Apache Hadoop YARN.

Yarn

Ayub Mohammad

MHUG - YARN

Joseph Niemiec

Apache Hadoop YARN - The Future of Data Processing with Hadoop

Hortonworks

YARN: Future of Data Processing with Apache HadoopHortonworks

Bikas saha：the next generation of hadoop– hadoop 2 and yarn

hdhappy001

Apache Hadoop YARN: Understanding the Data Operating System of Hadoop

Hortonworks

YARN - Hadoop Next Generation Compute Platform

Bikas Saha

Apache Tez: Accelerating Hadoop Query Processing

Hortonworks

Apache Tez - A New Chapter in Hadoop Data ProcessingDataWorks Summit

Apache Tez - Accelerating Hadoop Data Processing

hitesh1892

Apache Tez : Accelerating Hadoop Query Processing

Bikas Saha

Apache Tez is the new data processing framework in the Hadoop ecosystem. It runs on top of YARN - the new compute platform for Hadoop 2. Learn how Tez is built from the ground up to tackle a broad spectrum of data processing scenarios in Hadoop/BigData - ranging from interactive query processing to complex batch processing. With a high degree of automation built-in, and support for extensive customization, Tez aims to work out of the box for good performance and efficiency. Apache Hive and Pig are already adopting Tez as their platform of choice for query execution.

YARN - Next Generation Compute Platform fo Hadoop

Hortonworks

Hadoop: Beyond MapReduce

Steve Loughran

Tez big datacamp-la-bikas_saha

Data Con LA

Similar to Hortonworks Yarn Code Walk Through January 2014 (20)

Developing Applications with Hadoop 2.0 and YARN by Abhijit Lele

Apache Hadoop YARN: best practices

Yarnthug2014

YARN - Presented At Dallas Hadoop User Group

Apache Hadoop YARN - Enabling Next Generation Data Applications

Running Non-MapReduce Big Data Applications on Apache Hadoop

Yarn

MHUG - YARN

Apache Hadoop YARN - The Future of Data Processing with Hadoop

YARN: Future of Data Processing with Apache Hadoop

Bikas saha：the next generation of hadoop– hadoop 2 and yarn

Apache Hadoop YARN: Understanding the Data Operating System of Hadoop

YARN - Hadoop Next Generation Compute Platform

Apache Tez: Accelerating Hadoop Query Processing

Apache Tez - A New Chapter in Hadoop Data Processing

Apache Tez - Accelerating Hadoop Data Processing

Apache Tez : Accelerating Hadoop Query Processing

YARN - Next Generation Compute Platform fo Hadoop

Hadoop: Beyond MapReduce

Tez big datacamp-la-bikas_saha

More from Hortonworks

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level

Hortonworks

IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy

Hortonworks

Forrester forecasts* that direct spending on the Internet of Things (IoT) will exceed $400 Billion by 2023. From manufacturing and utilities, to oil & gas and transportation, IoT improves visibility, reduces downtime, and creates opportunities for entirely new business models. But successful IoT implementations require far more than simply connecting sensors to a network. The data generated by these devices must be collected, aggregated, cleaned, processed, interpreted, understood, and used. Data-driven decisions and actions must be taken, without which an IoT implementation is bound to fail. https://hortonworks.com/webinar/iot-predictions-2019-beyond-data-heart-iot-strategy/

Getting the Most Out of Your Data in the Cloud with Cloudbreak

Hortonworks

Johns Hopkins - Using Hadoop to Secure Access Log Events

Hortonworks

Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys

Hortonworks

Cybersecurity today is a big data problem. There’s a ton of data landing on you faster than you can load, let alone search it. In order to make sense of it, we need to act on data-in-motion, use both machine learning, and the most advanced pattern recognition system on the planet: your SOC analysts. Advanced visualization makes your analysts more efficient, helps them find the hidden gems, or bombs in masses of logs and packets. https://hortonworks.com/webinar/catch-hacker-real-time-live-visuals-bots-bad-guys/

HDF 3.2 - What's New

Hortonworks

Curing Kafka Blindness with Hortonworks Streams Messaging Manager

Hortonworks

With the growth of Apache Kafka adoption in all major streaming initiatives across large organizations, the operational and visibility challenges associated with Kafka are on the rise as well. Kafka users want better visibility in understanding what is going on in the clusters as well as within the stream flows across producers, topics, brokers, and consumers. With no tools in the market that readily address the challenges of the Kafka Ops teams, the development teams, and the security/governance teams, Hortonworks Streams Messaging Manager is a game-changer. https://hortonworks.com/webinar/curing-kafka-blindness-hortonworks-streams-messaging-manager/

Interpretation Tool for Genomic Sequencing Data in Clinical Environments

Hortonworks

The healthcare industry—with its huge volumes of big data—is ripe for the application of analytics and machine learning. In this webinar, Hortonworks and Quanam present a tool that uses machine learning and natural language processing in the clinical classification of genomic variants to help identify mutations and determine clinical significance. Watch the webinar: https://hortonworks.com/webinar/interpretation-tool-genomic-sequencing-data-clinical-environments/

IBM+Hortonworks = Transformation of the Big Data Landscape

Hortonworks

Premier Inside-Out: Apache Druid

Hortonworks

Accelerating Data Science and Real Time Analytics at Scale

Hortonworks

TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA

Hortonworks

Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...

Hortonworks

Trimble Transportation Enterprise is a leading provider of enterprise software to over 2,000 transportation and logistics companies. They have designed an architecture that leverages Hortonworks Big Data solutions and Machine Learning models to power up multiple Blockchains, which improves operational efficiency, cuts down costs and enables building strategic partnerships. https://hortonworks.com/webinar/blockchain-with-machine-learning-powered-by-big-data-trimble-transportation-enterprise/

Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense

Hortonworks

For years, the healthcare industry has had problems of data scarcity and latency. Clearsense solved the problem by building an open-source Hortonworks Data Platform (HDP) solution while providing decades worth of clinical expertise. Clearsense is delivering smart, real-time streaming data, to its healthcare customers enabling mission-critical data to feed clinical decisions. https://hortonworks.com/webinar/delivering-smart-real-time-streaming-data-healthcare-customers-clearsense/

Making Enterprise Big Data Small with Ease

Hortonworks

Webinewbie to Webinerd in 30 Days - Webinar World Presentation

Hortonworks

Driving Digital Transformation Through Global Data Management

Hortonworks

Using your data smarter and faster than your peers could be the difference between dominating your market and merely surviving. Organizations are investing in IoT, big data, and data science to drive better customer experience and create new products, yet these projects often stall in ideation phase to a lack of global data management processes and technologies. Your new data architecture may be taking shape around you, but your goal of globally managing, governing, and securing your data across a hybrid, multi-cloud landscape can remain elusive. Learn how industry leaders are developing their global data management strategy to drive innovation and ROI. Presented at Gartner Data and Analytics Summit Speaker: Dinesh Chandrasekhar Director of Product Marketing, Hortonworks

HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features

Hortonworks

Hortonworks DataFlow (HDF) is the complete solution that addresses the most complex streaming architectures of today’s enterprises. More than 20 billion IoT devices are active on the planet today and thousands of use cases across IIOT, Healthcare and Manufacturing warrant capturing data-in-motion and delivering actionable intelligence right NOW. “Data decay” happens in a matter of seconds in today’s digital enterprises. To meet all the needs of such fast-moving businesses, we have made significant enhancements and new streaming features in HDF 3.1. https://hortonworks.com/webinar/series-hdf-3-1-technical-deep-dive-new-streaming-features/

Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...

Hortonworks

Unlock Value from Big Data with Apache NiFi and Streaming CDC

Hortonworks

Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. It provides an end-to-end platform that can collect, curate, analyze, and act on data in real-time, on-premises, or in the cloud with a drag-and-drop visual interface. It’s being used across industries on large amounts of data that had stored in isolation which made collaboration and analysis difficult. Join industry experts from Hortonworks and Attunity as they explain how Apache NiFi and streaming CDC technology provides a distributed, resilient platform for unlocking the value of data in new ways.

More from Hortonworks (20)

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level

IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy

Getting the Most Out of Your Data in the Cloud with Cloudbreak

Johns Hopkins - Using Hadoop to Secure Access Log Events

Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys

HDF 3.2 - What's New

Curing Kafka Blindness with Hortonworks Streams Messaging Manager

Interpretation Tool for Genomic Sequencing Data in Clinical Environments

IBM+Hortonworks = Transformation of the Big Data Landscape

Premier Inside-Out: Apache Druid

Accelerating Data Science and Real Time Analytics at Scale

TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA

Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...

Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense

Making Enterprise Big Data Small with Ease

Webinewbie to Webinerd in 30 Days - Webinar World Presentation

Driving Digital Transformation Through Global Data Management

HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features

Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...

Unlock Value from Big Data with Apache NiFi and Streaming CDC

Recently uploaded

Elevating Tactical DDD Patterns Through Object Calisthenics

Dorra BARTAGUIZ

After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!

Connector Corner: Automate dynamic content and events by pushing a button

DianaGray10

Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to: Create a campaign using Mailchimp with merge tags/fields Send an interactive Slack channel message (using buttons) Have the message received by managers and peers along with a test email for review But there’s more: In a second workflow supporting the same use case, you’ll see: Your campaign sent to target colleagues for approval If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team But—if the “Reject” button is pushed, colleagues will be alerted via Slack message Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors. And... Speakers: Akshay Agnihotri, Product Manager Charlie Greenberg, Host

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

UiPathCommunity

💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™: See how to accelerate model training and optimize model performance with active learning Learn about the latest enhancements to out-of-the-box document processing – with little to no training required Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath. Speakers: 👨‍🏫 Andras Palfi, Senior Product Manager, UiPath 👩‍🏫 Lenka Dulovicova, Product Program Manager, UiPath

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

Tobias Schneck

As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other? Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.

Key Trends Shaping the Future of Infrastructure.pdf

Cheryl Hung

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

91mobiles

Knowledge engineering: from people to machines and back

Elena Simperl

PCI PIN Basics Webinar from the Controlcase Team

ControlCase

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

FIDO Alliance

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

Product School

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Paul Groth

Essentials of Automations: Optimizing FME Workflows with Parameters

Safe Software

Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place. Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects. Here’s what you’ll gain: - Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows. - Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy. - Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency. - Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity. We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic. Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

Securing your Kubernetes cluster_ a step-by-step guide to success !

KatiaHIMEUR1

Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster. However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks. In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.

Leading Change strategies and insights for effective change management pdf 1.pdf

OnBoard

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

Designing Great Products: The Power of Design and Leadership by Chief Designe...

Product School

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

Ramesh Iyer

In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Product School

UiPath Test Automation using UiPath Test Suite series, part 3

DianaGray10

Recently uploaded (20)