Big Data Integration Webinar: Getting Started With Hadoop Big Data
•
3 likes•1,452 views
This document discusses getting started with big data analytics using Hadoop and Pentaho. It provides an overview of installing and configuring Hadoop and Pentaho on a single machine or cluster. Dell's Crowbar tool is presented as a way to quickly deploy Hadoop clusters on Dell hardware in about two hours. The document also covers best practices like leveraging different technologies, starting with small datasets, and not overloading networks. A demo is given and contact information provided.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (15)
Similar to Big Data Integration Webinar: Getting Started With Hadoop Big Data
Similar to Big Data Integration Webinar: Getting Started With Hadoop Big Data (20)
More from Pentaho
More from Pentaho (7)
Recently uploaded
Recently uploaded (20)
Big Data Integration Webinar: Getting Started With Hadoop Big Data
- 1. Getting Started & Successful with Big Data © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 @Pentaho #BigDataWebSeries
- 2. Your Hosts Today Paul Brook Cloud EMEA Program Manager Dell 2© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 Davy Nys VP EMEA & APAC Pentaho Chuck Yarbrough Technical Solutions Marketing Pentaho
- 3. Pentaho Webinar Series 3© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 Sign-up at: pentaho.com
- 4. Goals for Today 4© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 To Understand: • How to get a Hadoop cluster up and running • Where Hadoop and other pieces fit into the architecture • How you can easily get data in & out Hadoop • How to leverage Hadoop with Pentaho • Initial Best Practices
- 5. Complete Analytics and Visual Data Management HadoopNoSQL Databases Data Discovery & Visualization Enterprise & Ad Hoc Reporting Predictive Analytics & Machine Learning Data Ingestion, Manipulation & Integration Analytic Databases © 2012, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 5
- 6. Data Warehouse Optimization Data Sources Big Data Architecture Data Warehouse (Master & Transactional Data) ERP CRM CDR Analytic Data Mart(s) Analytic Data Mart(s) Analytic Data Mart(s) Logs Logs Other Data Raw Data Parsed Data Analytic Datasets Master Data Tape Archive
- 7. Steps To Start with Hadoop 7© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 Hadoop Installation • Install locally – as Pseudo-Distributed mode • Leverage tools like Dell Crowbar • Cloud sandbox • Easy download & installation • Start on desktop Pentaho Installation 1 2 • Extract or access data from source systems • Load it (in its raw form) into Hadoop • Tokenize & parse as required • Transform & enrich • Load into destination Start Loading3
- 8. Data Architecture and Integration Challenges Data Sources Big Data Architecture Data Warehouse (Master & Transactional Data) ERP CRM CDR Analytic Data Mart(s) Analytic Data Mart(s) Analytic Data Mart(s) Logs Logs Other Data Raw Data Parsed Data Analytic Datasets Master Data NOSQL
- 9. Data Architecture and Integration Challenges Data Sources Big Data Architecture Data Warehouse (Master & Transactional Data) ERP CRM CDR Analytic Data Mart(s) Analytic Data Mart(s) Analytic Data Mart(s) Logs Logs Other Data Raw Data Parsed Data Analytic Datasets Master Data NOSQL Extract Transform Load Orchestration & Integration MR
- 10. Data Architecture and Integration Challenges Data Sources Big Data Architecture Data Warehouse (Master & Transactional Data) ERP CRM CDR Analytic Data Mart(s) Analytic Data Mart(s) Analytic Data Mart(s) Logs Logs Other Data Raw Data Parsed Data Analytic Datasets Master Data NOSQL Extract Transform Load Orchestration & Integration MR
- 11. Solution Architecture & Demo 11© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 Fast and easy way to deploy Hadoop clusters with Dell1
- 12. Global Marketing Fast and easy way to deploy Hadoop clusters with Dell
- 13. Global Marketing Well we are ready, but how will the Hardware Team know how to size and design the Hadoop cluster……..? I don’t know….and it may take a long time to build the Hadoop cluster Time is a critical factor, we need to get this project moving
- 14. Global Marketing Reduce time to Cluster Sizing, Design & Deployment Faster time to productive operations Optimize and adapt for your needs Deliver the best return on investment Reduce risk & increase flexibility with Dell Dell.com/Crowbar
- 15. Global Marketing Dell | Hadoop Solution “Dell … was one of the first of the hardware vendors to grasp the fact that cloud is about provisioning services, not about the hardware.” Maxwell Cooter, Cloud Pro Excels at supporting complex big data analyses across large collections of structured and unstructured data • Hadoop handles a variety of workloads, including search, log processing, data warehousing, recommendation systems and video/image analysis • Work on the most modern scale-out architectures using a clean-sheet design data framework • Without vendor lock-in Apache Hadoop software Crowbar software framework with a Hadoop barclamp PowerEdge C8000 Series, C6220, R720, R720XD Force10 or PowerConnect switches Reference Architecture Deployment Guide Joint Service and Support Proven solutions Proven components Partner Ecosystem
- 16. Global Marketing Crowbar • Accelerates multi-node deployments • Simplifies maintenance • Streamlines ongoing updates Built with DevOps • Provides an operational model for managing big data clusters and cloud Field-proven technologies • Build on locally deployed Chef Server • Raw servers to full cluster in <2 hours • Hardened with more than a year of deployments Apache 2 open source • Multi-apps (Hadoop & OpenStack) • Multi-OS (Ubuntu, RHEL, CentOS, SUSE) NOT limited to Dell hardware Crowbar Software Framework A modular, open source framework
- 17. Global Marketing Deploy a Hadoop cluster in ~2 hours Reduce software licensing fees 100% Use Crowbar to: • Automate the deployment and configuration of a Hadoop cluster • Quickly provision bare-metal servers from box to cluster with minimal intervention • Maintain, upgrade and evolve your Hadoop cluster over time • Leverage an open source framework backed by a growing global developer ecosystem Reduce development time 4-6 mo. Crowbar software framewo rk Evolve to meet your needs over time with built in DevOps
- 18. Global Marketing Crowbar dashboard provides visibility
- 19. Global Marketing Leverage developer expertise worldwide Download the open source software: https://github.com/dellcloudedge/crowb ar Participate in an active community http://lists.us.dell.com/mailman/listinfo /crowbar Get resources on the Wiki: https://github.com/dellcloudedge/crowb ar/wiki Visit Dell.com/Crowbar, Crowbar@Dell.com
- 21. pentaho.com/download • Install on a local desktop – no need for a cluster • “Managed Code” no additional installations • Pentaho will write to the Hadoop Distributed Cache for execution Pentaho Installation 21© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 Scheduling Integration Manipulation Orchestration 2
- 22. Start Loading Loading into HDFS & HIVE – Hadoop Copy Files – Specify source files / destination 22© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 3 Loading into HBASE – Zookeeper host & port – Specify HBASE Mapping
- 23. Solution Architecture & Demo 23© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 Demo
- 24. Maximize Performance 24© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 As much as 15x faster than hand-written code. Parallel execution as MapReduce in the Hadoop cluster.
- 25. Additional Best Practices 25© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 Leverage Hadoop • Don’t do database lookups inside a Mapper/Reducer – bring the data set into HDFS • Don’t transfer data between two clustering technologies – network overload • Start with a small data set and validate logic & performance outside the cluster • Gradually increase volumes and fine tune the application, cluster, data stores & network Don’t Boil the Ocean • Leverage the various technologies available • A combination of easy to use tools, powerful scripting and custom coding provides the best mix It’s AND…AND
- 26. Solution Architecture & Demo 26© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 Q & A
- 27. 27© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 Contact Us or Sign-up at: pentaho.com
Editor's Notes
- Two major
- TAKE-AWAYSPentaho provides complete integrated DI+BI for every leading big data platform.
- The company decided to invest in Hadoop to ingest the raw CDR data into Hadoop along with other data. This frees DW capacity for high value transactional data; while also, lowering cost and meeting compliance requirements. And since the data is rescued from tape, it becomes available for reporting and analysis.
- The company decided to invest in Hadoop to ingest the raw CDR data into Hadoop along with other data. This frees DW capacity for high value transactional data; while also, lowering cost and meeting compliance requirements. And since the data is rescued from tape, it becomes available for reporting and analysis.
- The company decided to invest in Hadoop to ingest the raw CDR data into Hadoop along with other data. This frees DW capacity for high value transactional data; while also, lowering cost and meeting compliance requirements. And since the data is rescued from tape, it becomes available for reporting and analysis.
- The company decided to invest in Hadoop to ingest the raw CDR data into Hadoop along with other data. This frees DW capacity for high value transactional data; while also, lowering cost and meeting compliance requirements. And since the data is rescued from tape, it becomes available for reporting and analysis.
- Delivered as a hardware, software, and services Reference Architecture (RA) which can scale from 6-nodes up to 720-nodesCurrently utilizes PowerEdge C 2100/C6100/C6105 R720, R720XD servers and PowerConnect 6248 or Force 10 switchesDell CrowbarAutomated solution deployment and configuration (Bare metal, OS, Solution Stack, and Monitoring)CDH3 EnterpriseCloudera Hadoop DistributionCloudera Management ToolsCloudera SupportPartner EcosystemSoftware and services capabilities to address broader customer needs around HadoopEnabling non-technical business users to leverage HadoopSimplify getting data into HadoopIntuitive analytics reporting and dashboardsSolution Provided viaReference ArchitectureDeployment GuideDell Digital LockerDell Deployment Services
- First OpenStack cloud solution provider in marketPioneer OpenStack partner (Only Day 1 hardware provider)Most history with the OpenStack technology = expertize + RA’s that have been tested longer and fuller than newcomersDell offers a deep partnership ecosystemSingle point of support and purchase to reduce the problem of dealing with multiple vendorsONLY company providing automated software to do multi-node OpenStack provisioning: CrowbarDell developed software that we opensourced in the community.OpenStack expertsize