Cloudera – One Platform to Rule Them All

•

1 like•819 views

Presentation about the Cloudera platform – a bit of history and the use cases. By Nuno Barreto – Partner & Big Data Lead @XpandIT

Technology

Cloudera – One
Platform to Rule
Them All

• A Bit of History
• The Platform
• The Use Cases
• Summary
AGENDA

RDBMS VS CLOUDERA
…according to Google Trends (since January 2010)

HADOOP – THE EARLY DAYS
• Hadoop Distributed Filesystem (HDFS)
• Hadoop MapReduce

X86 X86 X86 X86
NODE1 NODE2 NODE3 NODEN
...

HADOOP OVERTIME
• Hive – SQL-like query
• Pig - Programming model
• HBase – NoSQL database (operational)
• YARN – Resource Manager
• Impala – Online SQL (analytics)
• Spark – Streaming, Batch, ML
• Kafka – Messaging
… new additions

CLOUDERA ENTERPRISE DATA HUB
• Processing and Storage Core is 100% open source
• Only tested – at large scale – Apache components
get in
• Value added features (operations and governance)
• Reactive and Pro-active & Predictive Support
• Easy, Fast & Secure
• The best partners – us 

CLOUDERA MANAGER
• Operations
• Monitoring
• Configuration Management
• Multi-tenant Management
• Backup & Disaster Recovery
• Extensible Integration
… for cluster operations

CLOUDERA DIRECTOR
… makes Cloudera cloud ready

CLOUDERA NAVIGATOR
• Audit & Trace
• Alert
• Lineage
• Encryption
• Optimizer
… for cluster governance & security

DATA LAKE / ENTERPRISE DATA HUB
Sensor
Data
Blogs
Emails
Web
Logs
Docs
(e.g.PDF)
Images
Videos
CRM ERP Legacy
3rd
Patry
Extract (includesFileTansfer),TransformandLoad
Scale-out DistributedDatabase
Visualization(Reporting,ExplorationandSandboxing)
RawDataSources Operational Systems
DW&DATAMARTs

IOT
Devices with sensors &
actuators
Devices with sensors &
actuators
Gateway
Gateway
EDH

CLOUDERA DATA
SCIENCE WORKBENCH
Use R, Python or Scala
No need to sample
Collaborative research
Bring Analysis to the data
Secure by default
Flexible Deployment

SUMMARY
• Hadoop is an eco-system, not two projects
• Spark will not replace Hadoop, Spark “is”
Hadoop
• Cloudera has a complete offering
• Cloudera is for Batch & NRT
• Cloudera is for Analytics & Operational
… key takeaways

Credits
• Includes icons (pages 8 and 15) made by Freepik from www.flaticon.com
• Cloudera images from www.cloudera.com

What's hot

Ten Commandants For Picking NoSQL Database

Bansilal Haudakari

The Holy Grail of Data Analytics

Dan Lynn

This Presentation Covers: Improving application deployment by introducing database automation to your CI/CD pipeline. Current DB deployment challenges. Modern day deployment options. Automated Deployment with Ansible Benefits of Ansible If you liked the demo and would like to play with the code yourself, you can find it on Github Repository: https://github.com/sameerkasi200x/self-provisioned-edb-multiplatform. You can use it to get a head-start on Ansible-ized EDB deployment or to dig into the code used for the demo.

Automating EDB Postgres using Ansible by Sameer Kumar - Senior Solution Archi...

Ashnikbiz

Database Camp 2016 @ United Nations, NYC - Michael Glukhovsky, Co-Founder, Re...

✔ Eric David Benari, PMP

A couple of thousands of servers for a big data system is also a big investment. Microsoft Bing has figured out a way to fulfill our needs without signing a huge check. We have the technology to harvest spare cycles on underutilized servers. And we tweak the configurations in Hadoop and Spark to fit the flexible capacity base. We have saved hundreds of millions of dollars per year. Bing is adopting open source big data technologies for our offline data processing system. It requires a massive amount of capacity, which implies a significant bill. With collaboration with Windows and Azure, Research teams, we can harvest most of the needed capacity from our existing server fleet. We make use of the capacity on reserve servers while keeping them instantly available for emergency use; we allocate compute and storage to servers when they are not fully occupied. We updated Hadoop node decommission, HDFS block placement, YARN node label mapping, and a few other policies so that they can adapt to the capacity that is even less reliable than commodity servers. We brought open source capacity to Bing product with less than 1 percent of the cost we had done it through normal approach. We also extend the YARN and Spark framework to better fit the need of deep learning training and inferencing workloads in our system. This extension is equipping Bing with direct questioning and answering type of interactive query features. Big does not mean expensive. The audience can learn about the approach from Bing that they can make better use of their existing servers to do additional big data systems.

Free Servers to Build Big Data System on: Bing’s Approach

DataWorks Summit

Most DBAs are aware something interesting is going on with big data and the Hadoop product ecosystem that underpins it, but aren't so clear about what each component in the stack does, what problem each part solves and why those problems couldn't be solved using the old approach. We'll look at where it's all going with the advent of Spark and machine learning, what's happening with ETL, metadata and analytics on this platform ... why IaaS and datawarehousing-as-a-service will have such a big impact, sooner than you think

New World Hadoop Architectures (& What Problems They Really Solve) for Oracle...

Rittman Analytics

Journey to the Cloud: Database Modernization Best Practices

Datavail

This presentation covers What is Polyglot Persistence? And how should you choose the right Database Technology for a scalable architecture and introduction to Emerging world of Polyglot Persistence using open source database ecosystem. Polyglot Persistence is not something which can be used as an out of the box product, but instead needs to be designed for each individual enterprise for its unique Data Architecture.

Polyglot Persistence and Database Deployment by Sandeep Khuperkar CTO and Dir...

Ashnikbiz

Smartsheet’s Transition to Snowflake and Databricks: The Why and Immediate Im...

Databricks

EDB Postgres in DBaaS & Container Platforms

Ashnikbiz

Architecting Modern Data Platforms

Ankit Rathi

Database Camp 2016 @ United Nations, NYC - Brad Bebee, CEO, Blazegraph

✔ Eric David Benari, PMP

In this deck from the DDN booth at SC18, Laura Carriere from NCCS presents: Centralizing Storage without going off the Rails. "The National Climate Change Secretariat (NCCS) was established on 1 July 2010 under the Prime Minister’s Office (PMO) to develop and implement Singapore's domestic and international policies and strategies to tackle climate change. NCCS is part of the Strategy Group which supports the Prime Minister and his Cabinet to establish priorities and strengthen strategic alignment across Government. The inclusion of NCCS enhances strategy-making and planning on vital issues that span multiple Government ministries and agencies." Learn more: https://www.nccs.gov.sg/about-us/about-nccs and http://ddn.com

Centralizing Storage without going off the Rails

inside-BigData.com

Cosmosdb graph

Mohit Chhabra

Hubble - Accelerated Reporting, Analytics and Planning

strongandagile.co.uk

AWS Database Services

Mackenzie LeJeune

KNIME Meetup 2016-04-16

W. Daniel Cox, III CMA, CFM

Snaplogic Live: Big Data in Motion

SnapLogic

Your business wants to solve problems for your customers, not spend time managing silos of disconnected data that comes from on-premises solutions and new cloud applications. More and more organizations are looking to solve this problem by investing in cloud-based storage and analytics platforms such as Snowflake. However, data from systems such as mainframes can be a challenge to bring into cloud data warehouses. Together, Snowflake and Syncsort offer you the ability to get the full picture of your data – whether its mainframe or from a cloud application. View this webinar on how Snowflake and Syncsort are working together to get you back to what is essential for your business. View this webcast on-demand to learn: • Best practices for extracting your mainframe data • Advantages of using Snowflake for your cloud data warehouse needs • Common challenges faced by businesses trying to access mainframe data for use in cloud data warehouses • How Syncsort is helping organizations gain strategic value from their mainframe data

Snowflake + Syncsort: Get Value from Your Mainframe Data

Precisely

Building a Star Schema v1.1

Patrick Cuba

What's hot (20)

Ten Commandants For Picking NoSQL Database

The Holy Grail of Data Analytics

Automating EDB Postgres using Ansible by Sameer Kumar - Senior Solution Archi...

Database Camp 2016 @ United Nations, NYC - Michael Glukhovsky, Co-Founder, Re...

Free Servers to Build Big Data System on: Bing’s Approach

New World Hadoop Architectures (& What Problems They Really Solve) for Oracle...

Journey to the Cloud: Database Modernization Best Practices

Polyglot Persistence and Database Deployment by Sandeep Khuperkar CTO and Dir...

Smartsheet’s Transition to Snowflake and Databricks: The Why and Immediate Im...

EDB Postgres in DBaaS & Container Platforms

Architecting Modern Data Platforms

Database Camp 2016 @ United Nations, NYC - Brad Bebee, CEO, Blazegraph

Centralizing Storage without going off the Rails

Cosmosdb graph

Hubble - Accelerated Reporting, Analytics and Planning

AWS Database Services

KNIME Meetup 2016-04-16

Snaplogic Live: Big Data in Motion

Snowflake + Syncsort: Get Value from Your Mainframe Data

Building a Star Schema v1.1

Similar to Cloudera – One Platform to Rule Them All

Search On Hadoop

bigdatagurus_meetup

Cloudera Impala - Las Vegas Big Data Meetup Nov 5th 2014

cdmaxime

We present a software model built on the Apache software stack (ABDS) that is well used in modern cloud computing, which we enhance with HPC concepts to derive HPC-ABDS. We discuss layers in this stack We give examples of integrating ABDS with HPC We discuss how to implement this in a world of multiple infrastructures and evolving software environments for users, developers and administrators We present Cloudmesh as supporting Software-Defined Distributed System as a Service or SDDSaaS with multiple services on multiple clouds/HPC systems. We explain the functionality of Cloudmesh as well as the 3 administrator and 3 user modes supported

Cloud Services for Big Data Analytics

Geoffrey Fox

Cloud Services for Big Data Analytics

Geoffrey Fox

Introduction To Hadoop Ecosystem

InSemble

http://bit.ly/1BTaXZP – As organizations look for even faster ways to derive value from big data, they are turning to Apache Spark is an in-memory processing framework that offers lightning-fast big data analytics, providing speed, developer productivity, and real-time processing advantages. The Spark software stack includes a core data-processing engine, an interface for interactive querying, Spark Streaming for streaming data analysis, and growing libraries for machine-learning and graph analysis. Spark is quickly establishing itself as a leading environment for doing fast, iterative in-memory and streaming analysis. This talk will give an introduction the Spark stack, explain how Spark has lighting fast results, and how it complements Apache Hadoop. By the end of the session, you’ll come away with a deeper understanding of how you can unlock deeper insights from your data, faster, with Spark.

Intro to Apache Spark by CTO of Twingo

MapR Technologies

Hadoop distributions - ecosystem

Jakub Stransky

Advanced Analytics and Big Data (August 2014)

Thomas W. Dinsmore

Search onhadoopsfhug081413

gregchanan

Hadoop and OpenStack

DataWorks Summit

Hadoop and OpenStack - Hadoop Summit San Jose 2014

spinningmatt

Over the past several years, the Hadoop ecosystem has made great strides in its real-time access capabilities, narrowing the gap compared to traditional database technologies. With systems such as Impala and Spark, analysts can now run complex queries or jobs over large datasets within a matter of seconds. With systems such as Apache HBase and Apache Phoenix, applications can achieve millisecond-scale random access to arbitrarily-sized datasets. Despite these advances, some important gaps remain that prevent many applications from transitioning to Hadoop-based architectures. Users are often caught between a rock and a hard place: columnar formats such as Apache Parquet offer extremely fast scan rates for analytics, but little to no ability for real-time modification or row-by-row indexed access. Online systems such as HBase offer very fast random access, but scan rates that are too slow for large scale data warehousing workloads. This talk will investigate the trade-offs between real-time transactional access and fast analytic performance from the perspective of storage engine internals. It will also describe Kudu, the new addition to the open source Hadoop ecosystem that fills the gap described above, complementing HDFS and HBase to provide a new option to achieve fast scans and fast random access from a single API.

Introduction to Kudu - StampedeCon 2016

StampedeCon

Data Pipelines in Hadoop - SAP Meetup in Tel Aviv

larsgeorge

Enterprises are starting to deploy large scale Hadoop clusters to extract value out of the data that they are generating. These clusters often span hundreds of nodes. To speed up the time to value, a lot of the newer deployments are happening in AWS, moving from the traditional on-premises, bare-metal world. Cloudera supports just such deployments. In this session, Cloudera shares the lessons learned and best practices for deploying multi-tenant Hadoop clusters in AWS. They will cover what reference deployments look like, what services are relevant for Hadoop deployments, network configurations, instance types, backup and disaster recovery considerations, and security considerations. They will also talk about what works well, what doesn't, and what has to be done going forward to improve the operability of Hadoop on AWS.

(BDT305) Lessons Learned and Best Practices for Running Hadoop on AWS | AWS r...

Amazon Web Services

Introduction to Kudu: Hadoop Storage for Fast Analytics on Fast Data - Rüdige...

Dataconomy Media

Real time fraud detection at 1+M scale on hadoop stack

DataWorks Summit/Hadoop Summit

Analytics using big data technologies

Balakrishnan Vinchu

Enterprise Holding’s first started with Hadoop as a POC in 2013. Today, we have clusters on premises and in the cloud. This talk will explore our experience with Big Data and outline three common big data architectures (batch, lambda, and kappa). Then, we’ll dive into the decision points to necessary for your own cluster, for example: cloud vs on premises, physical vs virtual, workload, and security. These decisions will help you understand what direction to take. Finally, we’ll share some lessons learned with the pieces of our architecture worked well and rant about those which didn’t. No deep Hadoop knowledge is necessary, architect or executive level.

Innovation in the Data Warehouse - StampedeCon 2016

StampedeCon

SQL and Machine Learning on Hadoop

Mukund Babbar

Introduction to Impala

markgrover

Similar to Cloudera – One Platform to Rule Them All (20)

Search On Hadoop

Cloudera Impala - Las Vegas Big Data Meetup Nov 5th 2014

Cloud Services for Big Data Analytics

Introduction To Hadoop Ecosystem

Intro to Apache Spark by CTO of Twingo

Hadoop distributions - ecosystem

Advanced Analytics and Big Data (August 2014)

Search onhadoopsfhug081413

Hadoop and OpenStack

Hadoop and OpenStack - Hadoop Summit San Jose 2014

Introduction to Kudu - StampedeCon 2016

Data Pipelines in Hadoop - SAP Meetup in Tel Aviv

(BDT305) Lessons Learned and Best Practices for Running Hadoop on AWS | AWS r...

Introduction to Kudu: Hadoop Storage for Fast Analytics on Fast Data - Rüdige...

Real time fraud detection at 1+M scale on hadoop stack

Analytics using big data technologies

Innovation in the Data Warehouse - StampedeCon 2016

SQL and Machine Learning on Hadoop

Introduction to Impala

More from Xpand IT

The Xray and Xporter Winter Tour kicked off last Wednesday with the Jira & Confluence Solutions. During Sérgio Freire’s (Xray Product Manager) presentation, he showed Jira as a Test Management tool and how to empower test teams to manage and deliver rock-solid software solutions with Xray. If you missed it or you want to know more about Testing in Jira, you can check it here. For more visit https://www.xpand-addons.com/

Xray & Xporter were in Austria: Jira & Confluence Solutions Day 2018

Xpand IT

Using Xamarin for your Mobile+ Apps – Xamarin Experience London 2017

Xpand IT

Xporter for Jira - Overview

Xpand IT

Xray for Jira - How to automate your QA process

Xpand IT

Xpand Addons - Addon Discovery Day 2017

Xpand IT

Xray for Jira 3.0 - What's New?

Xpand IT

Xray for Jira - Overview

Xpand IT

Xporter for Jira - Advanced topics

Xpand IT

Keynote - Xamarin Experience London 2017

Xpand IT

Welcome & Introduction – Xamarin Experience London 2017

Xpand IT

Gathering Customer Insights with Sitecore - Xamarin Experience London 2017

Xpand IT

Why Speed Matters in Mobile Apps – Xamarin Experience London 2017

Xpand IT

Mobile & Cognitive Services | Harnessing the Power of IoT – Xamarin Experienc...

Xpand IT

Atlassian Tools in Practice: A Customer Success Story – Xpand IT & Atlassian ...

Xpand IT

The Secret Sauce of Successful Teams - Xpand IT & Atlassian JAM Sessions 2017

Xpand IT

Quality Assurance Made Easy in JIRA - Xpand IT & Atlassian JAM Sessions 2017

Xpand IT

Improved Reporting with JIRA Add-ons - Xpand IT & Atlassian JAM Sessions 2017

Xpand IT

How our Team Collaborates with Atlassian Tools - Xpand IT & Atlassian JAM Ses...

Xpand IT

Welcome & Introduction - Xpand IT & Atlassian JAM Sessions 2017

Xpand IT

The Real World with OpenShift - Red Hat DevOps & Microservices Conference 2017

Xpand IT

More from Xpand IT (20)