• Share
  • Email
  • Embed
  • Like
  • Private Content
How CBS Interactive uses Cloudera Manager to effectively manage their Hadoop cluster
 

How CBS Interactive uses Cloudera Manager to effectively manage their Hadoop cluster

on

  • 1,495 views

Manoj Murumkar, Senior Manager, Data Engineering at CBS Interactive and Bala Venkatrao, Director of Products at Cloudera provide insight into how CBSi uses Cloudera Manager to effectively manage the ...

Manoj Murumkar, Senior Manager, Data Engineering at CBS Interactive and Bala Venkatrao, Director of Products at Cloudera provide insight into how CBSi uses Cloudera Manager to effectively manage the complete lifecycle of CBSi’s Hadoop operations. They will highlight the Cloudera Manager features that have been most helpful to CBSi and share best practices on how to use the tool. This webinar will conclude with a preview of the future roadmap for Cloudera Manager.

Statistics

Views

Total Views
1,495
Views on SlideShare
1,442
Embed Views
53

Actions

Likes
3
Downloads
26
Comments
0

2 Embeds 53

http://www.cloudera.com 47
http://blog.cloudera.com 6

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • CBS Interactive
  • Cost – just because you can doesn’t mean you should. You could cut your grass with hedge clippers. But why would you?

How CBS Interactive uses Cloudera Manager to effectively manage their Hadoop cluster How CBS Interactive uses Cloudera Manager to effectively manage their Hadoop cluster Presentation Transcript

  • WEBINARHow CBS Interactive Uses Cloudera Manager to EffectivelyManage their Hadoop ClusterWednesday, September 19th, 2012Manoj Murumkar - Senior Manager, Data Engineering, CBS InteractiveBala Venkatrao – Director of Products, Cloudera
  • AgendaIntroductionsCBSi • Hadoop Use Case • Operational Challenges • How Cloudera Manager helps CBSi & Demo • Benefits of using Cloudera ManagerCloudera Manager • Overview & Benefits • Key Features • RoadmapQ&A 2 ©2012 Cloudera, Inc. All Rights Reserved.
  • IntroductionsManoj MurumkarSenior Manager, Data Engineering at CBS Interactive Manoj has been working with data technologies since 1998. His team currently responsible for providing data infrastructure solutions and operating them for internet division of CBS corporation. He has been involved with Hadoop for more around 3 years, around 2 years of which working with Cloudera. His team has built big data infrastructure from ground up that helps in clickstream analysis using Hadoop streaming.Bala VenkatraoDirector, Products at Cloudera Bala Venkatrao is part of the product management team at Cloudera and leads the efforts around Cloudera Manager. In addition, he is involved in several other initiatives, including customer advocacy, partnership development, marketing etc. 3 ©2012 Cloudera, Inc. All Rights Reserved.
  • Building web analytics for Top 10 global web property on Hadoop  235M worldwide monthly unique users Challenge Solution Results  Requires advanced analytics  Web analytics platform on  Optimizing what content is placed on click stream data in near Hadoop processes >1B global beside that which user is currently real time events/day reading  Reduced processing time by 6+ hrs.  Weblog processing time on  >1PB on Hadoop; 42 nodes to reach SLA proprietary platform hit limit while data volumes  Tracking clicks, page views,  Accommodates 50% data volume continuously increased downloads, streaming video increase per year events, ad events, etc.  Ability/Cost to store historical  Reduce cost of storing/processing data for analyses  Hadoop Components: HDFS, data Hive, MapReduce, Pig, Hadoop Streaming  Greater ad revenues achievedSource: Hadoop World 2012 presentation. Michael Sun, Lead Software Engineer & Manager of DW Operations, CBS Interactive.http://www.cloudera.com/resource/hadoop-world-2012-presentation-slides-building-web-analytics-processing-on-hadoop-at-cbs-interactive/ 4 ©2012 Cloudera, Inc. All Rights Reserved.
  • CBSi Hadoop Operational ChallengesPrior to Cloudera Manager Lack of…  Holistic view  Configuration control  No audit trail/history of changes Existing solutions were…  Ganglia , Hadoop web UI pages and custom scripts  Difficult to maintain No visibility into activity failures • Reactive to user complaints on failed/long running jobs 5 ©2012 Cloudera, Inc. All Rights Reserved.
  • How Cloudera Manager helps CBSi withHadoop OperationsIntuitive visual interface  Can manage and monitor the whole cluster  Overall health status/dashboard  Ability to drill down from services > roles > hostsService Monitoring and Alerting  Makes Hadoop operations pro-active  Heatmaps provides an easy way to identify outliersActivity Monitoring • Helps identify failed or slow running jobs  Notify end-users on failed jobs and manage SLA’sWorkflows  Simple to add new ‘data nodes’, hosts, clients etc. 6 ©2012 Cloudera, Inc. All Rights Reserved.
  • CLOUDERA MANAGERCBSi Demo
  • Key Benefits ofUsing Cloudera Manager at CBSiLowers the barrier for Hadoop administration  Do not need to rely on experts solelyMakes life easier – saves money & time  Avoid licensing costs associated with managing multiple tools  Cuts technical and human resource costs  Reduces time to manage and maintain the clusterProvides a “one-stop” holistic view  Easy to understand how the overall cluster is performingHelps create repeatable processes & workflows forHadoop operations  Improves efficiency of the Operations team 8 ©2012 Cloudera, Inc. All Rights Reserved.
  • The 6 Characteristics ofEnterprise Grade Hadoop 9 ©2012 Cloudera, Inc. All Rights Reserved.
  • Why You Need Cloudera Manager1 COMPLEXITY HADOOP IS MORE THAN A DOZEN SERVICES RUNNING ACROSS MANY MACHINES  HUNDREDS OF HARDWARE COMPONENTS  THOUSANDS OF SETTINGS  LIMITLESS PERMUTATIONS2 CONTEXT HADOOP IS A SYSTEM, NOT JUST A COLLECTION OF PARTS  EVERYTHING IS INTERRELATED  RAW DATA ABOUT INDIVIDUAL PIECES IS NOT ENOUGH  MUST EXTRACT WHAT’S IMPORTANT3 EFFICIENCY MANAGING HADOOP WITH MULTIPLE TOOLS & MANUAL PROCESSES TAKES LONGER  COMPLICATED, ERROR PRONE WORKFLOWS  LONGER ISSUE RESOLUTION  LACK OF CONSISTENT AND REPEATABLE PROCESSES 10 ©2012 Cloudera, Inc. All Rights Reserved.
  • Cloudera Manager ProvidesEnd-to-End CDH Administration1 DEPLOY INSTALL, CONFIGURE AND START YOUR CLUSTER IN 3 SIMPLE STEPS2 CONFIGURE & OPTIMIZE ENSURE OPTIMAL SETTINGS FOR ALL HOSTS AND SERVICES3 MONITOR, DIAGNOSE & REPORT FIND AND FIX PROBLEMS QUICKLY, VIEW CURRENT AND HISTORICAL ACTIVITY AND RESOURCE USAGE CDH 11 ©2012 Cloudera, Inc. All Rights Reserved.
  • Managing Complexity One Tool For EverythingDEPLOYMENT & ACTIVITY MONITORING WORKFLOWS EVENTS & ALERTS LOG SEARCH DIAGNOSTICS REPORTINGCONFIGURATION MONITORINGDO-IT-YOURSELF +CLOUDERA ENTERPRISE“In a recent Cloudera survey, >95% of respondents emphasized the need for a single end-to-end tool to manage their Hadoop Operations” 12 ©2012 Cloudera, Inc. All Rights Reserved.
  • Providing ContextRaw Data vs. Hadoop Intelligence 1 SMART CONFIGURATION AUTO-SETS CONFIGURATIONS & GUARDS AGAINST USER ERROR ? VS. 2 WORKFLOWS ENSURES THAT MULTI-STEP TASKS ARE ACCOMPLISHED COMPLETELY & IN THE CORRECT SEQUENCE 3 DEPENDENCIES AWARE OF HOW A PARTICULAR ACTION AFFECTS THE REST OF THE CLUSTER & MANAGES THE IMPACT 4 EVENTS & ALERTS MAKES YOU AWARE OF WHAT’S IMPORTANT AT A HADOOP SYSTEM LEVEL 5 HISTORY COMPARES CURRENT & PAST ACTIVITIES FOR CONTEXT 13 ©2012 Cloudera, Inc. All Rights Reserved.
  • Cloudera Manager Key Features Installs the complete Hadoop stack in minutes via a wizard-based interface Gives you complete, end-to-end visibility and control over your Hadoop cluster from a single interface Allows you to manage multiple clusters from a single instance of Cloudera Manager Integrate Cloudera Manager with Active Directory Establishes the time context globally for almost all views Correlates jobs, activities, logs, system changes, configuration changes and service metrics along a single timeline to simplify diagnosis Set server roles, configure services and manage security across the cluster Gracefully start, stop and restart of services as needed Supports Administrator and Read-Only users Maintains a complete record of configuration changes with the ability to roll back to previous states Monitors dozens of service performance metrics and alerts you when you approach critical thresholds14 ©2012 Cloudera, Inc. All Rights Reserved.
  • Cloudera Manager Key Features Gather, view and search Hadoop logs collected from across the cluster Scans Hadoop logs for irregularities and warns you before they impact the cluster Creates and aggregates relevant Hadoop events pertaining to system health, log messages, user services and activities and make them available for alerting and searching Generates email alerts when certain events occur Consolidates all cluster activity into a single, real-time view View information pertaining to hosts in your cluster including status, resident memory, virtual memory and roles Visualize health status and metrics across the cluster to quickly identify problem nodes and take action Visualize current and historical disk usage by user, group and directory Track MapReduce activity on the cluster by job or user Takes a snapshot of the cluster state and automatically sends it to Cloudera support to assist with resolution Easily integrate Cloudera Manager with your existing enterprise-wide management and monitoring tools15 ©2012 Cloudera, Inc. All Rights Reserved.
  • Cloudera Manager RoadmapMaintenance modePlatform Support  Manage additional services like Flume, Hive etc.Monitoring  ZooKeeper monitoring  Advanced HBase monitoringRolling UpgradesUsability enhancements  Improved error handling  Log search enhancements  Enhanced charting 16 ©2012 Cloudera, Inc. All Rights Reserved.
  • Why Enterprises are Standardizing onCloudera Manager 1 SIMPLE END-TO-END HADOOP ADMINISTRATION IN A SINGLE TOOL 2 INTELLIGENT MANAGES HADOOP AT THE SYSTEM LEVEL - CLOUDERA’S EXPERIENCE REALIZED IN SOFTWARE 3 EFFICIENT SIMPLIFIES COMPLEX WORKFLOWS & MAKES ADMINISTRATORS MORE EFFICIENT 4 BEST-IN-CLASS THE ONLY ENTERPRISE-GRADE HADOOP MANAGEMENT APPLICATION AVAILABLE 17 ©2012 Cloudera, Inc. All Rights Reserved.
  • Next Steps• Try out FREE edition of Cloudera Manager• Download from:http://www.cloudera.com/products-services/tools/• Support available via scm-users@cloudera.org• For Cloudera Enterprise subscriptions, please contact: sales@cloudera.com 18 ©2012 Cloudera, Inc. All Rights Reserved.
  • Q&AFor more information go to www.cloudera.com
  • THANK YOU!We appreciate your time and interest in Cloudera!For more information: www.cloudera.comSales: (888)789-1488