Cloudera Manager Webinar | Cloudera Enterprise 3.7Cloudera, Inc.
Managing Hadoop just got easier!
In this webinar, Cloudera VP, Product Charles Zedlewski will introduce and explain the new features and functionality of Cloudera Manager, a core component of Cloudera Enterprise 3.7.
What is Cloudera Manager?
Cloudera Manager is the industry’s first end-to-end management application for Apache Hadoop. With Cloudera Manager, you can easily deploy and centrally operate a complete Hadoop stack. The application automates the installation process, reducing deployment time from weeks to minutes; gives you a cluster wide, real time view of nodes and services running; provides a single, central place to enact configuration changes across your cluster; and incorporates a full range of reporting and diagnostic tools to help you optimize cluster performance and utilization.
What is Cloudera Enterprise?
Cloudera Enterprise enables data driven enterprises to run Apache Hadoop environments in production cost effectively with repeatable success. Comprised of Cloudera Support and Cloudera Manager, a software layer that delivers deep visibility into and across Hadoop clusters, Cloudera Enterprise gives Hadoop operators an efficient way to precisely provision and manage cluster resources. It also allows IT shops to apply familiar business metrics – such as measurable SLAs and chargebacks – to Hadoop environments so they can run at optimal utilization. Built-in predictive capabilities anticipate shifts in the Hadoop infrastructure, ensuring reliable operation.
Why is everyone interested in Big Data and Hadoop?
Why you should use Hadoop?
Read this to and you as well you quickly and easily be the proud owner of a Hadoop kit of your own, using Cloudera Free Edition.
************************NOTE**********************
This presentation is still being edited and new slides added every day. Stay tuned...
****************************************************
Accumulo includes a remarkable breadth of testing frameworks, which helps to ensure its correctness, performance, robustness, and protection of your vital data. This presentation takes you on a tour from Accumulo's basic unit testing up through performance and scalability testing exercised on running clusters. Learn the extent to which Accumulo is put through its paces before it is released, and get ideas for how you can similarly enhance testing of your own code.
Find this talk and others at http://www.slideshare.net/AccumuloSummit.
Cloudera Manager Webinar | Cloudera Enterprise 3.7Cloudera, Inc.
Managing Hadoop just got easier!
In this webinar, Cloudera VP, Product Charles Zedlewski will introduce and explain the new features and functionality of Cloudera Manager, a core component of Cloudera Enterprise 3.7.
What is Cloudera Manager?
Cloudera Manager is the industry’s first end-to-end management application for Apache Hadoop. With Cloudera Manager, you can easily deploy and centrally operate a complete Hadoop stack. The application automates the installation process, reducing deployment time from weeks to minutes; gives you a cluster wide, real time view of nodes and services running; provides a single, central place to enact configuration changes across your cluster; and incorporates a full range of reporting and diagnostic tools to help you optimize cluster performance and utilization.
What is Cloudera Enterprise?
Cloudera Enterprise enables data driven enterprises to run Apache Hadoop environments in production cost effectively with repeatable success. Comprised of Cloudera Support and Cloudera Manager, a software layer that delivers deep visibility into and across Hadoop clusters, Cloudera Enterprise gives Hadoop operators an efficient way to precisely provision and manage cluster resources. It also allows IT shops to apply familiar business metrics – such as measurable SLAs and chargebacks – to Hadoop environments so they can run at optimal utilization. Built-in predictive capabilities anticipate shifts in the Hadoop infrastructure, ensuring reliable operation.
Why is everyone interested in Big Data and Hadoop?
Why you should use Hadoop?
Read this to and you as well you quickly and easily be the proud owner of a Hadoop kit of your own, using Cloudera Free Edition.
************************NOTE**********************
This presentation is still being edited and new slides added every day. Stay tuned...
****************************************************
Accumulo includes a remarkable breadth of testing frameworks, which helps to ensure its correctness, performance, robustness, and protection of your vital data. This presentation takes you on a tour from Accumulo's basic unit testing up through performance and scalability testing exercised on running clusters. Learn the extent to which Accumulo is put through its paces before it is released, and get ideas for how you can similarly enhance testing of your own code.
Find this talk and others at http://www.slideshare.net/AccumuloSummit.
A brief introduction to YARN: how and why it came into existence and how it fits together with this thing called Hadoop.
Focus given to architecture, availability, resource management and scheduling, migration from MR1 to MR2, job history and logging, interfaces, and applications.
Hadoop is often viewed as needing racks of dedicated boxes -despite the fact that in sheer number terms, the majority of Hadoop clusters ever created have been brought up on public cloud infrastructures -particularly Amazon`s. Yet the rest of datacenter computing is moving towards virtualization -be it in-cloud startups or in-enterprise IT departments. Some organizations are standing up private clouds: a rack or two of servers with an API for VM creation. Hadoop can live there -it just needs to integrate better. At the same time, OpenStack is emerging as the de-facto standard open source cloud platform for private use, and is available publicly from a number of cloud infrastructure service providers. This talk looks at what we`ve done -and are doing- to integrate Hadoop with OpenStack. This is taking it beyond Hadoop`s current support for Amazon`s infrastructure, making a combined Hadoop + OpenStack cluster something to consider in-house -and in-cloud. This work is being done in collaboration with members of the OpenStack community, showing how cloud and big data projects can not only co-exist, we can co-develop our platforms.
Hadoop Operations for Production Systems (Strata NYC)Kathleen Ting
Hadoop is emerging as the standard for big data processing and analytics. However, as usage of the Hadoop clusters grow, so do the demands of managing and monitoring these systems.
In this full-day Strata Hadoop World tutorial, attendees will get an overview of all phases for successfully managing Hadoop clusters, with an emphasis on production systems — from installation, to configuration management, service monitoring, troubleshooting and support integration.
We will review tooling capabilities and highlight the ones that have been most helpful to users, and share some of the lessons learned and best practices from users who depend on Hadoop as a business-critical system.
January 2015 HUG: Using HBase Co-Processors to Build a Distributed, Transacti...Yahoo Developer Network
Monte Zweben Co-Founder and CEO of Splice Machine, will discuss how to use HBase co-processors to build an ANSI-99 SQL database with 1) parallelization of SQL execution plans, 2) ACID transactions with snapshot isolation and 3) consistent secondary indexing.
Transactions are critical in traditional RDBMSs because they ensure reliable updates across multiple rows and tables. Most operational applications require transactions, but even analytics systems use transactions to reliably update secondary indexes after a record insert or update.
In the Hadoop ecosystem, HBase is a key-value store with real-time updates, but it does not have multi-row, multi-table transactions, secondary indexes or a robust query language like SQL. Combining SQL with a full transactional model over HBase opens a whole new set of OLTP and OLAP use cases for Hadoop that was traditionally reserved for RDBMSs like MySQL or Oracle. However, a transactional HBase system has the advantage of scaling out with commodity servers, leading to a 5x-10x cost savings over traditional databases like MySQL or Oracle.
HBase co-processors, introduced in release 0.92, provide a flexible and high-performance framework to extend HBase. In this talk, we show how we used HBase co-processors to support a full ANSI SQL RDBMS without modifying the core HBase source. We will discuss how endpoint transactions are used to serialize SQL execution plans over to regions so that computation is local to where the data is stored. Additionally, we will show how observer co-processors simultaneously support both transactions and secondary indexing.
The talk will also discuss how Splice Machine extended the work of Google Percolator, Yahoo Labs’ OMID, and the University of Waterloo on distributed snapshot isolation for transactions. Lastly, performance benchmarks will be provided, including full TPC-C and TPC-H results that show how Hadoop/HBase can be a replacement of traditional RDBMS solutions.
Resource Management in Impala - StampedeCon 2016StampedeCon
Want to run queries in Impala as fast as possible without choking other workloads and services? If you are a Hadoop cluster administrator or a big data application developer, this course will help you understand how Impala Admission Control can help you make good use of available resources, avoid bad performance issues, and provide better user experiences in a multi-tenancy environment.
Cloudera Impala provides a fast, ad hoc query capability to Apache Hadoop, complementing traditional MapReduce batch processing. Learn the design choices and architecture behind Impala, and how to use near-ubiquitous SQL to explore your own data at scale.
As presented to Portland Big Data User Group on July 23rd 2014.
http://www.meetup.com/Hadoop-Portland/events/194930422/
A brief introduction to YARN: how and why it came into existence and how it fits together with this thing called Hadoop.
Focus given to architecture, availability, resource management and scheduling, migration from MR1 to MR2, job history and logging, interfaces, and applications.
Hadoop is often viewed as needing racks of dedicated boxes -despite the fact that in sheer number terms, the majority of Hadoop clusters ever created have been brought up on public cloud infrastructures -particularly Amazon`s. Yet the rest of datacenter computing is moving towards virtualization -be it in-cloud startups or in-enterprise IT departments. Some organizations are standing up private clouds: a rack or two of servers with an API for VM creation. Hadoop can live there -it just needs to integrate better. At the same time, OpenStack is emerging as the de-facto standard open source cloud platform for private use, and is available publicly from a number of cloud infrastructure service providers. This talk looks at what we`ve done -and are doing- to integrate Hadoop with OpenStack. This is taking it beyond Hadoop`s current support for Amazon`s infrastructure, making a combined Hadoop + OpenStack cluster something to consider in-house -and in-cloud. This work is being done in collaboration with members of the OpenStack community, showing how cloud and big data projects can not only co-exist, we can co-develop our platforms.
Hadoop Operations for Production Systems (Strata NYC)Kathleen Ting
Hadoop is emerging as the standard for big data processing and analytics. However, as usage of the Hadoop clusters grow, so do the demands of managing and monitoring these systems.
In this full-day Strata Hadoop World tutorial, attendees will get an overview of all phases for successfully managing Hadoop clusters, with an emphasis on production systems — from installation, to configuration management, service monitoring, troubleshooting and support integration.
We will review tooling capabilities and highlight the ones that have been most helpful to users, and share some of the lessons learned and best practices from users who depend on Hadoop as a business-critical system.
January 2015 HUG: Using HBase Co-Processors to Build a Distributed, Transacti...Yahoo Developer Network
Monte Zweben Co-Founder and CEO of Splice Machine, will discuss how to use HBase co-processors to build an ANSI-99 SQL database with 1) parallelization of SQL execution plans, 2) ACID transactions with snapshot isolation and 3) consistent secondary indexing.
Transactions are critical in traditional RDBMSs because they ensure reliable updates across multiple rows and tables. Most operational applications require transactions, but even analytics systems use transactions to reliably update secondary indexes after a record insert or update.
In the Hadoop ecosystem, HBase is a key-value store with real-time updates, but it does not have multi-row, multi-table transactions, secondary indexes or a robust query language like SQL. Combining SQL with a full transactional model over HBase opens a whole new set of OLTP and OLAP use cases for Hadoop that was traditionally reserved for RDBMSs like MySQL or Oracle. However, a transactional HBase system has the advantage of scaling out with commodity servers, leading to a 5x-10x cost savings over traditional databases like MySQL or Oracle.
HBase co-processors, introduced in release 0.92, provide a flexible and high-performance framework to extend HBase. In this talk, we show how we used HBase co-processors to support a full ANSI SQL RDBMS without modifying the core HBase source. We will discuss how endpoint transactions are used to serialize SQL execution plans over to regions so that computation is local to where the data is stored. Additionally, we will show how observer co-processors simultaneously support both transactions and secondary indexing.
The talk will also discuss how Splice Machine extended the work of Google Percolator, Yahoo Labs’ OMID, and the University of Waterloo on distributed snapshot isolation for transactions. Lastly, performance benchmarks will be provided, including full TPC-C and TPC-H results that show how Hadoop/HBase can be a replacement of traditional RDBMS solutions.
Resource Management in Impala - StampedeCon 2016StampedeCon
Want to run queries in Impala as fast as possible without choking other workloads and services? If you are a Hadoop cluster administrator or a big data application developer, this course will help you understand how Impala Admission Control can help you make good use of available resources, avoid bad performance issues, and provide better user experiences in a multi-tenancy environment.
Cloudera Impala provides a fast, ad hoc query capability to Apache Hadoop, complementing traditional MapReduce batch processing. Learn the design choices and architecture behind Impala, and how to use near-ubiquitous SQL to explore your own data at scale.
As presented to Portland Big Data User Group on July 23rd 2014.
http://www.meetup.com/Hadoop-Portland/events/194930422/
If you also got the Big Data itch, here is something to ease the pain :-)
Answers to this questions will be available soon (more info in the attached link)
Which Big Data Appliance should YOU use?
(click on the attached link for Poll results)
Appliances are Small and Quick, Right?
Revealing the 6 Types of Big Data Appliances
Uncovering the Main Players
Challenges, Pitfalls, and Winning the Big Data Game
Where is all this leading YOU to?
Technical details with lot of numbers. Git, Redmine, Hipchat, 10 654 working hours and more. Now, and only now, you could see background details of very sophisticated e-commerce solution.
Technical details with lot of numbers. Git, Redmine, Hipchat, 10 654 working hours and more. Now and only now you could see background details of very sophisticated e-commerce solution. You could look under the hood.
Thinking DevOps in the era of the Cloud - Demi Ben-AriDemi Ben-Ari
The lines between Development and Operations people have gotten blurry and lots of skills needs to be held by both sides.
In the talk we'll talk about all of the considerations that are needed to be taken when creating a development and production environment, mentioning Continuous Integration, Continuous Deployment and the Buzzword "DevOps", also talking about some real implementations in the industry.
Of course how can we leave out the real enabler of the whole deal,
"The Cloud", Giving us a tool set that makes life much easier when implementing all of these practices.
The Fifth Elephant 2016: Self-Serve Performance Tuning for Hadoop and SparkAkshay Rai
Dr. Elephant is a self serve performance monitoring and tuning tool for the users who run Hadoop and Spark jobs.
Conference Link: https://fifthelephant.talkfunnel.com/2016/19-dr-elephant-self-serve-performance-tuning-for-hado
Github Repo & Documentation: https://github.com/linkedin/dr-elephant
Mailing List: https://groups.google.com/forum/#!forum/dr-elephant-users
During Kylin OLAP development, we setup many engineering principles in the team. These principles are very important to delivery Kylin with high quality and on schedule.
PHP At 5000 Requests Per Second: Hootsuite’s Scaling Storyvanphp
Bill Monkman, Lead Engineer at Hootsuite, presenting on how Hootsuite went from zero to hundreds of millions of requests per day with its PHP codebase, and how dealing with that growth has shaped its future direction. Tips, optimizations, and horror stories from a rapidly-scaling PHP startup.
Video: https://www.youtube.com/watch?v=TZGeBAIMPII
In the session, we discussed the End-to-end working of Apache Airflow that mainly focused on "Why What and How" factors. It includes the DAG creation/implementation, Architecture, pros & cons. It also includes how the DAG is created for scheduling the Job and what all steps are required to create the DAG using python script & finally with the working demo.
In the session, we discussed the End-to-end working of Apache Airflow that mainly focused on "Why What and How" factors. It includes the DAG creation/implementation, Architecture, pros & cons. It also includes how the DAG is created for scheduling the Job and what all steps are required to create the DAG using python script & finally with the working demo.
Running Airflow Workflows as ETL Processes on Hadoopclairvoyantllc
While working with Hadoop, you'll eventually encounter the need to schedule and run workflows to perform various operations like ingesting data or performing ETL. There are a number of tools available to assist you with this type of requirement and one such tool that we at Clairvoyant have been looking to use is Apache Airflow. Apache Airflow is an Apache Incubator project that allows you to programmatically create workflows through a python script. This provides a flexible and effective way to design your workflows with little code and setup. In this talk, we will discuss Apache Airflow and how we at Clairvoyant have utilized it for ETL pipelines on Hadoop.
Mtc learnings from isv & enterprise interactionGovind Kanshi
This is one of the dated presentation for which I keep getting requests for, please do reach out to me for status on various things as Azure keeps fixing/innovating whole of things every day.
There are bunch of other things I can help you on to ensure you can take advantage of Azure platform for oss, .net frameworks and databases.
Similar to Cluster management and automation with cloudera manager (20)
Mark Burgess' presentation at #lspe on "Configuration Management" http://www.meetup.com/SF-Bay-Area-Large-Scale-Production-Engineering/events/73725682/
Kuyper Hoffmann's presentation from the #lspe "Private Clouds" event: http://www.meetup.com/SF-Bay-Area-Large-Scale-Production-Engineering/events/48901162/
Vladimir Vuksan's presentation on Ganglia at the "Not Nagios" episode of The Bay Area Large-Scale Production Engineering meetup: http://www.meetup.com/SF-Bay-Area-Large-Scale-Production-Engineering/events/15481164/
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
3. Hadoop is...
● Fast-changing
– New features all the time
● Different from other IT projects
– One application on many hosts; not vice-versa
● Complex
– Things you might run: HDFS, MapReduce, Yarn, ZooKeeper,
Oozie, Hive, Pig, HBase, Sqoop, Solr, Cloudera Impala...
● Useful
4. Many Common Setup Issues
●
Operating system issues
– Transparent Huge Pages
– Ulimits
– Clock Skew
●
Networking issues
– Reverse-lookup does must report FQDN
– NICs can negotiate less than full speed
These are just examples. There are many more!
5. Let others do the work for you
●
Cloudera's Distribution including Apache
Hadoop (CDH)
– Enterprise-Ready: Tested and deployed in production on 10s of
1000s of nodes
– Enterprise-grade features and innovation
●
Fine-grained Authorization (Sentry)
●
Impala, Search
– 100% open source and Apache licensed
6. Cloudera Manager
●
Available for free
– Any number of nodes
– Manage all services available in CDH
– Set up, configure, monitor, diagnose, and upgrade
– Complex workflows
– Kerberos
– API
●
5 Years of expertise baked into product
15. Installation Complete
● Everything is up and running – Great!
● Add users and start running jobs, and get
a whole new set of challenges – Great...
16. Next Challenges
● Find, Diagnose and fix problems
– Why are my HBase queries slow?
● View cluster activity
– Who ran the MapReduce job that made my HBase
queries slow?
● Get alerts for any problems that come up
– Outage at 2AM, you want that wake-up call...right?
17. Health Tests
● Common problems that are easy to check
– Are any processes down?
– Are HDFS reads and writes working?
– Are HDFS checkpoints too slow?
– Has a host been swapping?
– Is there too much Clock Skew?
19. Log Search
● Grep works great on 1 machine, not 100's
● Useful to answer
– What errors/warnings occurred when my service was slow?
– Has this error occurred before?
– When did a problem start happening?
24. Metrics and Charts
● Like Log search, a must-have for any distributed
system
● Hadoop services expose many metrics
● Collect and visualize these with
– Cloudera Manager
– Ganglia
28. Next Challenges
● We know how to set up a cluster manually
● We know how to identify, diagnose and fix
issues
● Also need to handle regular tasks
– Grow cluster
– Replace hardware
29. Cloudera Manager API
●
Setup
– Create / configure cluster and services
– Configure new host to run on cluster
●
Workflows
– Enable HDFS High Availability
– Enable MapReduce JobTracker High Availability
– Decommission / Recommission host
●
Monitoring
– Metrics used for charting available via API
– Health checks, including export to Nagios
– Events
30. Cloudera Manager API
● http://cloudera.github.com/cm_api/
● Java and Python client bindings
● Shell
● Export health information into Nagios
31. Common Integration Questions
● Nagios – yes
● Even have tools to help integrate
● Chef – not yet
● Puppet – yes
● Customers use CM and puppet together to press button
and stamp out new cluster
● Snmp – yes
● events published and can be integrated
32. Links
● Hadoop Operations - A Guide for Developers and Administrators
– Book by Eric Sammer
● CM Architecture blog
– http://blog.cloudera.com/blog/2013/07/how-does-cloudera-manager-work/
● API Examples and Tutorials
– http://cloudera.github.io/cm_api/
– http://blog.cloudera.com/blog/2013/05/how-to-automate-your-hadoop-cluster-from-java/
– http://blog.cloudera.com/blog/2012/09/automating-your-cluster-with-cloudera-manager-api/
● Cloudera Manager installer link and docs
– http://www.cloudera.com/content/support/en/downloads.html
– http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/Cloudera-Manager-
Installation-Guide/Cloudera-Manager-Installation-Guide.html
33. Enterprise Features
● Easily upload support bundle
– Enables proactive support
– Fix problems more quickly
● Rolling Upgrades and Restarts
● Backup and Disaster Recovery
●
Auditing
●
Operational Reports
●
Configuration History and Rollback
● LDAP