Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfaces to Large-scale Data

•

1 like•989 views

You like to use R, and you need to use big data. dplyr, one of the most popular packages for R, makes it easy to query large data sets in scalable processing engines like Apache Spark and Apache Impala. But there can be pitfalls: dplyr works differently with different data sources—and those differences can bite you if you don’t know what you’re doing. Ian Cook is a data scientist, an R contributor, and a curriculum developer at Cloudera University. In this webinar, Ian will show you exactly what you need to know about sparklyr (from RStudio) and the package implyr (from Cloudera). He will show you how to write dplyr code that works across these different interfaces. And, he will solve mysteries: Do I need to know SQL to use dplyr? When is a “tbl” not a “tibble”? Why is 1 not always equal to 1? When should you collect(), collapse(), and compute()? How can you use dplyr to combine data stored in different systems? 3 things to learn: Do I need to know SQL to use dplyr? When should you collect(), collapse(), and compute()? How can you use dplyr to combine data stored in different systems?

Technology

1© Cloudera, Inc. All rights reserved.
dplyr Interfaces to Large-Scale Data
Ian Cook
@ianmcook
ian@cloudera.com

2© Cloudera, Inc. All rights reserved.
Mission for Cloudera: Provide a platform for data analysts, data scientists to
efficiently query, analyze, model large-scale data in clusters, cloud storage
• By distributing Apache Spark, Apache Impala, other tools
• By enabling productive use of these tools
Python and R users often have difficulty moving from smaller data to large-scale
distributed data
• Familiar packages, methods don’t work the same way on distributed data
Context

3© Cloudera, Inc. All rights reserved.
Poll question

4© Cloudera, Inc. All rights reserved.
]
SQLPySpark
SparkR
SQL
SQL or
DataFrame API
SQL or
DataFrame API
SQL or
DataFrame API
SQL or
DataFrame API
SQL or
DataFrame API
SQL or
DataFrame API

5© Cloudera, Inc. All rights reserved.
Poll question

6© Cloudera, Inc. All rights reserved.
]
SQLPySpark
SparkR
SQL
dplyr

7© Cloudera, Inc. All rights reserved.
dplyr provides a set of verbs that perform common data manipulation steps
• select() to select columns
• filter() to filter rows
• arrange() to order rows
• mutate() to create new columns
• summarise() to aggregate
• group_by() to perform operations by group
dplyr works on local data and with remote data sources
• For remote sources, dplyr commands are translated into SQL
dplyr

8© Cloudera, Inc. All rights reserved.
Poll question

9© Cloudera, Inc. All rights reserved.
Demonstration
Example code at
github.com/ianmcook/dplyr-examples

10© Cloudera, Inc. All rights reserved.
dplyr SQL backends
dplyr
↕
dbplyr
↕
dplyr SQL backend package*
↕
DBI
↕
DBI-compatible interface package
↕
database driver or connector
↕
database/engine
* optional

11© Cloudera, Inc. All rights reserved.
• Provides a SQL backend to dplyr for Spark
• Also exposes the MLlib API and a subset of the Spark DataFrames API
• Developed by RStudio
spark.rstudio.com
sparklyr

12© Cloudera, Inc. All rights reserved.
• Provides a SQL backend to dplyr for Impala
• Uses ODBC or JDBC to connect to Impala
• Developed at Cloudera
tiny.cloudera.com/implyr
implyr
implyr

13© Cloudera, Inc. All rights reserved.
Five tips for using dplyr
with SQL data sources

14© Cloudera, Inc. All rights reserved.
Use show_query()
1

15© Cloudera, Inc. All rights reserved.
filter() early
arrange() late
2

16© Cloudera, Inc. All rights reserved.
Check your data types
3

17© Cloudera, Inc. All rights reserved.
Know your SQL engine
4

18© Cloudera, Inc. All rights reserved.
Know when to collect()
5

19© Cloudera, Inc. All rights reserved.
Questions?
Ian Cook
@ianmcook
ian@cloudera.com

20© Cloudera, Inc. All rights reserved.
Cloudera Data Science Workbench
More information
tiny.cloudera.com/cdsw
OnDemand training
tiny.cloudera.com/cdsw-training

The document discusses Cloudera's certification program and training options. It provides information on: 1) The different certification levels like CCA and CCP and their focus areas like administration, development, and data analytics. 2) The benefits of Cloudera training like its alignment with best practices, experienced instructors, and global coverage. 3) The various training modalities like public/private classes, virtual training, and on-demand options. 4) Suggested curricula for different roles and tips for studying for and taking the certification exams.

Unlock Hadoop Success with Cloudera Navigator Optimizer

Cloudera, Inc.

Supercharge Splunk with Cloudera 

Cloudera, Inc.

Cloudera can help optimize Splunk deployments by providing more cost-effective scalability, increased data flexibility, and enhanced analytics capabilities. Cloudera can ingest data from Splunk indexes and apply enrichment using open-source machine learning before storing the data in its data hub. This provides a single platform for advanced analytics like SQL and Python/R scripts across both historical and new data. Initial use cases include offloading event data from Splunk to reduce costs and loading additional context sources to gain better insights.

Solr consistency and recovery internals

Cloudera, Inc.

Big Data Fundamentals

Cloudera, Inc.

Discover the origins of big data, discuss existing and new projects, share common use cases for those projects, and explain how you can modernize your architecture using data analytics, data operations, data engineering and data science. Big Data Fundamentals is your prerequisite to building a modern platform for machine learning and analytics optimized for the cloud. We’ll close out with a live Q&A with some of our technical experts as well. Stretch your brain with a packed agenda: Open source software Data storage Data ingestion Data analytics Data engineering IoT and life after Lambda architectures Data science Cybersecurity Cluster management Big data in the cloud Success stories

A deep dive into running data analytic workloads in the cloud

Cloudera, Inc.

This document discusses running data analytic workloads in the cloud using Cloudera Altus. It introduces Altus, which provides a platform-as-a-service for analyzing and processing data at scale in public clouds. The document outlines Altus features like low cost per-hour pricing, end-user focus, and cloud-native deployment. It then describes hands-on examples using Altus Data Engineering for ETL and the Altus Analytic Database for exploration and analytics. Workload analytics capabilities are also introduced for troubleshooting and optimizing jobs.

Data Science and CDSW

Jason Hubbard

SFHUG presentation from February 2, 2016. One of the key values of the Hadoop ecosystem is its flexibility. There is a myriad of components that make up this ecosystem, allowing Hadoop to tackle otherwise intractable problems. However, having so many components provides a significant integration, implementation, and usability burden. Features that ought to work in all the components often require sizable per-component effort to ensure correctness across the stack. Lenni Kuff explores RecordService, a new solution to this problem that provides an API to read data from Hadoop storage managers and return them as canonical records. This eliminates the need for components to support individual file formats, handle security, perform auditing, and implement sophisticated IO scheduling and other common processing that is at the bottom of any computation. Lenni discusses the architecture of the service and the integration work done for MapReduce and Spark. Many existing applications on those frameworks can take advantage of the service with little to no modification. Lenni demonstrates how this provides fine grain (column level and row level) security, through Sentry integration, and improves performance for existing MapReduce and Spark applications by up to 5×. Lenni concludes by discussing how this architecture can enable significant future improvements to the Hadoop ecosystem. About the speaker: Lenni Kuff is an engineering manager at Cloudera. Before joining Cloudera, he worked at Microsoft on a number of projects including SQL Server storage engine, SQL Azure, and Hadoop on Azure. Lenni graduated from the University of Wisconsin-Madison with degrees in computer science and computer engineering.

Faster Batch Processing with Cloudera 5.7: Hive-on-Spark is ready for production

Cloudera, Inc.

It’s no secret that Apache Spark is becoming the successor to MapReduce for data processing in Hadoop. With it’s easy development, flexible API, and performance benefits, Spark is a powerful data processing engine that has quickly gained popularity within the community. On the other hand Hive continues to be the most widely used data warehouse/ETL engine with large scale adoption across enterprises. Therefore, it’s imperative to enable Spark as the underlying execution engine for Hive to seamlessly allow existing and future Hive workloads to leverage the advantages of Spark. With the recent release of Cloudera 5.7, we have delivered on this goal by adding support for Hive-on-Spark. Data engineers and ETL developers can now transition from MR to Spark for their Hive workloads seamlessly thereby benefitting from the advantages of Spark without any disruption on their end. Join Santosh Kumar, Senior Product Manager at Cloudera, and Rui Li, Apache Hive committer and engineer at Intel, as we discuss: An Introduction to Spark and its advantages over MR An introduction of Hive-on-Spark: Goals and Design Principles Migrating to HoS and a live demo Configuring and tuning for batch workloads What’s next for both tools

A Community Approach to Fighting Cyber Threats

Cloudera, Inc.

Part 3: Models in Production: A Look From Beginning to End

Cloudera, Inc.

The document discusses the different roles involved in developing machine learning models from beginning to end. It describes the typical workflow as including data engineering to prepare data, exploratory data science to develop models, and operational model deployment to production applications. It provides examples of tasks for each role such as data engineers ingesting and transforming sensor data, data scientists building and evaluating predictive models, and model deployment engineers validating models and creating APIs.

Extreme Sports & Beyond: Exploring a new frontier in data with GoPro

Cloudera, Inc.

GoPro is a powerful global brand, thanks in large part to its innovative cameras and accessories that capture moments other cameras just miss: surfing in Maui, skiing in Tahoe, recording your child’s first steps. And today, the company is nearly as well known for its user-generated social and content networks. Join us for this special webinar hosted by Tableau, Trifacta, and Cloudera—featuring GoPro. We’ll dive into GoPro’s data strategy and architecture, from ingest and processing to data prep and reporting, all on AWS.

Part 1: Lambda Architectures: Simplified by Apache Kudu

Cloudera, Inc.

Spark One Platform Webinar

Cloudera, Inc.

This document discusses Cloudera's initiative to make Spark the standard execution engine for Hadoop. It outlines how Spark improves on MapReduce by leveraging distributed memory and having a simpler developer experience. It also describes Cloudera's investments in areas like management, security, scale, and streaming to further Spark's capabilities and make it production-ready. The goal is for Spark to replace MapReduce as the execution engine and for specialized engines like Impala to handle specific workloads, with all sharing the same data, metadata, resource management, and other platform services.

Cloudera Altus: Big Data in the Cloud Made Easy

Cloudera, Inc.

Cloudera Altus makes it easier for data engineers, ETL developers, and anyone who regularly works with raw data to process that data in the cloud efficiently and cost effectively. In this webinar we introduce our new platform-as-a-service offering and explore challenges associated with data processing in the cloud today, how Altus abstracts cluster overhead to deliver easy, efficient data processing, and unique features and benefits of Cloudera Altus.

How Data Drives Business at Choice Hotels

Cloudera, Inc.

3 Things to Learn: -How data is driving digital transformation to help businesses innovate rapidly -How Choice Hotels (one of largest hoteliers) is using Cloudera Enterprise to gain meaningful insights that drive their business -How Choice Hotels has transformed business through innovative use of Apache Hadoop, Cloudera Enterprise, and deployment in the cloud — from developing customer experiences to meeting IT compliance requirements

Gartner Data and Analytics Summit: Bringing Self-Service BI & SQL Analytics ...

Cloudera, Inc.

For self-service BI and exploratory analytic workloads, the cloud can provide a number of key benefits, but the move to the cloud isn’t all-or-nothing. Gartner predicts nearly 80 percent of businesses will adopt a hybrid strategy. Learn how a modern analytic database can power your business-critical workloads across multi-cloud and hybrid environments, while maintaining data portability. We'll also discuss how to best leverage the increased agility cloud provides, while maintaining peak performance.

Hadoop on Cloud: Why and How?

Cloudera, Inc.

The document discusses running Hadoop on the cloud using Cloudera Director. It begins with an introduction of the speaker and Cloudera Director. Several common architectural patterns for running Hadoop in the cloud are presented, including using object storage and running short-term ETL/modeling clusters versus long-term analytics clusters. The presentation envisions a future with a more portable, self-service, self-healing, and granularly secure experience for managing Hadoop in the cloud.

Data Science at Scale Using Apache Spark and Apache Hadoop

Cloudera, Inc.

This document provides information about a data science course taught using Apache Spark and Apache Hadoop. It introduces the instructors Sean Owen and Tom White and describes what data science is and the roles of data scientists. Data scientists have skills in engineering, statistics, and business domains. The document discusses why companies need data scientists due to the growth of data and its value. It presents the tools used in data science, including Apache Spark, and how Spark can be used for both investigative and operational analytics. The course teaches a complete data science problem process through hands-on examples using tools like Hadoop, Python, R, Hive, and Spark MLlib.

Analyzing Hadoop Data Using Sparklyr 

Cloudera, Inc.

The document discusses how Sparklyr allows data scientists to access and work with data stored in Cloudera Enterprise using the popular RStudio IDE. It describes the challenges data scientists face in accessing secured Hadoop clusters and limitations of notebook environments. Sparklyr integration with RStudio provides a familiar environment for data scientists to access Hadoop data and compute using Spark, enabling distributed data science workflows directly in R. The presentation demonstrates how to analyze over a billion records using Spark and R through Sparklyr.

Multi-Tenant Operations with Cloudera 5.7 & BT

Cloudera, Inc.

One Hadoop, Multiple Clouds - NYC Big Data Meetup

Andrei Savu

Part 2: Apache Kudu: Extending the Capabilities of Operational and Analytic D...

Cloudera, Inc.

Hadoop Distributed File System (HDFS) Encryption with Cloudera Navigator Key ...

Cloudera, Inc.

This document provides an overview of Cloudera's Navigator Key Trustee, which is a key management server that acts as a proxy between CDH components and an external key store. It discusses how Key Trustee uses encryption zone keys stored in an external hardware security module to encrypt data encryption keys, which are then used to encrypt data at rest in HDFS. The document also covers Key Trustee's architecture, deployment considerations, access control lists, and troubleshooting steps.

Hadoop Hadoop & Spark meetup - Altiscale

Mark Kerzner

Altiscale provides a big data-as-a-service platform based on Apache Hadoop and related technologies like Spark, Hive, and Tez. Interest in big data is growing rapidly but many independent implementations fail. Altiscale aims to help with its experienced team and fully managed platform that offers fast time to value, scalability, security, and lower total cost of ownership. The platform core is Apache open source components like Hadoop, Spark, Hive and Tez. Altiscale handles administration of the Hadoop cluster including hardware, upgrades, tuning, and addressing failures so customers can focus on their data and jobs.

How Big Data Can Enable Analytics from the Cloud (Technical Workshop)

Cloudera, Inc.

In this workshop, we will look outside the box and help expand the problem space to include issues you may not have thought were possible before Big Data. From Near Real Time (NRT) recommendation engines, loan applications to churn detection, Big Data is answering new questions and providing organisations with a competitive edge through revenue increase, cost savings and risk mitigation. We will take a special look at the role the Cloud can play in elevating your analytics environment. We will discuss real world examples of how Big Data answers these questions and does it at a lower cost outlay.

dplyr Interfaces to Large-Scale Data

Cloudera, Inc.

Part 2: A Visual Dive into Machine Learning and Deep Learning  

Cloudera, Inc.

What's hot

Part 2: Cloudera’s Operational Database: Unlocking New Benefits in the Cloud

Cloudera, Inc.

Simplifying Hadoop with RecordService, A Secure and Unified Data Access Path ...

Cloudera, Inc.

Faster Batch Processing with Cloudera 5.7: Hive-on-Spark is ready for production

Cloudera, Inc.

A Community Approach to Fighting Cyber Threats

Cloudera, Inc.

Part 3: Models in Production: A Look From Beginning to End

Cloudera, Inc.

Extreme Sports & Beyond: Exploring a new frontier in data with GoPro

Cloudera, Inc.

Part 1: Lambda Architectures: Simplified by Apache Kudu

Cloudera, Inc.

Spark One Platform Webinar

Cloudera, Inc.

Cloudera Altus: Big Data in the Cloud Made Easy

Cloudera, Inc.

How Data Drives Business at Choice Hotels

Cloudera, Inc.

Gartner Data and Analytics Summit: Bringing Self-Service BI & SQL Analytics ...

Cloudera, Inc.

Hadoop on Cloud: Why and How?

Cloudera, Inc.

Data Science at Scale Using Apache Spark and Apache Hadoop

Cloudera, Inc.

Analyzing Hadoop Data Using Sparklyr 

Cloudera, Inc.

Multi-Tenant Operations with Cloudera 5.7 & BT

Cloudera, Inc.

One Hadoop, Multiple Clouds - NYC Big Data Meetup

Andrei Savu

Part 2: Apache Kudu: Extending the Capabilities of Operational and Analytic D...

Cloudera, Inc.

Hadoop Distributed File System (HDFS) Encryption with Cloudera Navigator Key ...

Cloudera, Inc.

Hadoop Hadoop & Spark meetup - Altiscale

Mark Kerzner

How Big Data Can Enable Analytics from the Cloud (Technical Workshop)

Cloudera, Inc.

What's hot (20)

Part 2: Cloudera’s Operational Database: Unlocking New Benefits in the Cloud

Simplifying Hadoop with RecordService, A Secure and Unified Data Access Path ...

Faster Batch Processing with Cloudera 5.7: Hive-on-Spark is ready for production

A Community Approach to Fighting Cyber Threats

Part 3: Models in Production: A Look From Beginning to End

Extreme Sports & Beyond: Exploring a new frontier in data with GoPro

Part 1: Lambda Architectures: Simplified by Apache Kudu

Spark One Platform Webinar

Cloudera Altus: Big Data in the Cloud Made Easy

How Data Drives Business at Choice Hotels

Gartner Data and Analytics Summit: Bringing Self-Service BI & SQL Analytics ...

Hadoop on Cloud: Why and How?

Data Science at Scale Using Apache Spark and Apache Hadoop

Analyzing Hadoop Data Using Sparklyr 

Multi-Tenant Operations with Cloudera 5.7 & BT

One Hadoop, Multiple Clouds - NYC Big Data Meetup

Part 2: Apache Kudu: Extending the Capabilities of Operational and Analytic D...

Hadoop Distributed File System (HDFS) Encryption with Cloudera Navigator Key ...

Hadoop Hadoop & Spark meetup - Altiscale

How Big Data Can Enable Analytics from the Cloud (Technical Workshop)

Similar to Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfaces to Large-scale Data

dplyr Interfaces to Large-Scale Data

Cloudera, Inc.

Part 2: A Visual Dive into Machine Learning and Deep Learning  

Cloudera, Inc.

Data Science Languages and Industry Analytics

Wes McKinney

Cloudera Analytics and Machine Learning Platform - Optimized for Cloud

Stefan Lipp

Large-Scale Data Science on Hadoop (Intel Big Data Day)

Uri Laserson

The document discusses data science workflows on Hadoop. It describes data science as involving three phases - data plumbing to ingest and transform data, exploratory analytics to investigate and analyze data, and operational analytics to build and deploy models. It provides examples of tools used for each phase including Spark, Hadoop streaming, SAS, and Python for exploratory analytics, and MLlib and Spark for operational analytics. The document also discusses lambda architectures for handling both batch and real-time analytics.

Applications on Hadoop

markgrover

This document discusses building applications on Hadoop and introduces the Kite SDK. It provides an overview of Hadoop and its components like HDFS and MapReduce. It then discusses that while Hadoop is powerful and flexible, it can be complex and low-level, making application development challenging. The Kite SDK aims to address this by providing higher-level APIs and abstractions to simplify common use cases and allow developers to focus on business logic rather than infrastructure details. It includes modules for data, ETL processing with Morphlines, and tools for working with datasets and jobs. The SDK is open source and supports modular adoption.

Twitter with hadoop for oow

Gwen (Chen) Shapira

New data dictionary an internal server api that matters

Alexander Nozdrin

A new Data Dictionary based on transactional tables is being developed for the MySQL server. That project is a huge step forward improving many aspects of the server. The new Data Dictionary provides API which is intended to be used by all the participants of the MySQL Server Ecosystem. The slides make a brief introduction about what general Data Dictionary is, provide overview of the MySQL traditional Data Dictionary and its limitations. Then, the presentation shows the design goals of the new Data Dictionary and sketch the main architectural decisions. It also provides the description of a few visible advantages for the MySQL users. These are the slides for my session on OOW 2014.

Cloudera Big Data Integration Speedpitch at TDWI Munich June 2017

Stefan Lipp

Power of the AWR Warehouse- HotSos Symposium 2015

Kellyn Pot'Vin-Gorman

This document discusses the Power of the AWR Warehouse and beyond in Oracle Enterprise Manager 12c Release 4. It provides an overview of the architecture and features of the AWR Warehouse, including how it allows centralized long-term storage and analysis of AWR data across multiple databases without runtime overhead. The key benefits are space savings, offloading resource demands for deep analysis to the warehouse, and centralizing data identified by database to analyze multiple databases.

Big Data Management System: Smart SQL Processing Across Hadoop and your Data ...

DataWorks Summit

The document discusses smart SQL processing for databases, Hadoop and beyond. It describes how Oracle teaches its database about Hadoop by publishing Hadoop metadata like SerDe, RecordReader and InputFormat information to Oracle's catalog. This allows SQL queries to be executed on Hadoop data. However, directly sending SQL queries to Hadoop data nodes presents bottlenecks, so the document discusses how Oracle makes SQL processing smarter by applying techniques like smart scan, storage indexing and caching utilized in Oracle Exadata to minimize data movement and improve performance.

Building Efficient Pipelines in Apache Spark

Jeremy Beard

This document provides an overview of techniques for optimizing Apache Spark pipelines. It discusses fundamentals of Spark execution including jobs, stages and tasks. It then provides recommendations for tuning aspects like sizing executors, using DataFrames/Datasets over RDDs, caching frequently used data, joining techniques to avoid shuffling large datasets, and addressing skew. The document aims to help debug and optimize Spark applications.

OOW-TBE-12c-CON7307-Sharable

Obaidur (OB) Rashid

This document discusses the successful migration of Oracle's Taleo Business Edition cloud service to Oracle Database 12c. It provides context on TBE's rapid growth necessitating a more robust database platform. It describes how Oracle 12c was well-suited for the migration due to its optimizations for multi-tenancy and cloud deployments. The document also outlines the transition lifecycle and lessons learned from the project.

Oracle Database Cloud Service

Jean-Philippe PINTE

Oracle NoSQL Database release 3.0 overview

Paulo Fagundes

The document discusses NoSQL databases and Oracle's NoSQL Database product. It outlines key features of Oracle NoSQL Database including its scalability, high availability, elastic configuration, ACID transactions, and commercial support. Benchmark results show Oracle NoSQL Database can achieve over 1 million operations per second and scale linearly with additional servers. The document also provides information on licensing and support options for Oracle NoSQL Database Community Edition and Enterprise Edition.

PySpark Best Practices

Cloudera, Inc.

This document discusses best practices for using PySpark. It covers: - Core concepts of PySpark including RDDs and the execution model. Functions are serialized and sent to worker nodes using pickle. - Recommended project structure with modules for data I/O, feature engineering, and modeling. - Writing testable, serializable code with static methods and avoiding non-serializable objects like database connections. - Tips for testing like unit testing functions and integration testing the full workflow. - Best practices for running jobs like configuring the Python environment, managing dependencies, and logging to debug issues.

Impala 2.0 - The Best Analytic Database for Hadoop

Cloudera, Inc.

Turning Relational Database Tables into Hadoop Datasources by Kuassi Mensah

Data Con LA

Apache Spark in Scientific Applications

Dr. Mirko Kämpf

This document provides an overview of Apache Spark, including: - Apache Spark is a next generation data processing engine for Hadoop that allows for fast in-memory processing of huge distributed and heterogeneous datasets. - Spark offers tools for data science and components for data products and can be used for tasks like machine learning, graph processing, and streaming data analysis. - Spark improves on MapReduce by being faster, allowing parallel processing, and supporting interactive queries. It works on both standalone clusters and Hadoop clusters.

Apache Spark in Scientific Applciations

Dr. Mirko Kämpf

Similar to Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfaces to Large-scale Data (20)

dplyr Interfaces to Large-Scale Data

Part 2: A Visual Dive into Machine Learning and Deep Learning  

Data Science Languages and Industry Analytics

Cloudera Analytics and Machine Learning Platform - Optimized for Cloud

Large-Scale Data Science on Hadoop (Intel Big Data Day)

Applications on Hadoop

Twitter with hadoop for oow

New data dictionary an internal server api that matters

Cloudera Big Data Integration Speedpitch at TDWI Munich June 2017

Power of the AWR Warehouse- HotSos Symposium 2015

Big Data Management System: Smart SQL Processing Across Hadoop and your Data ...

Building Efficient Pipelines in Apache Spark

OOW-TBE-12c-CON7307-Sharable

Oracle Database Cloud Service

Oracle NoSQL Database release 3.0 overview

PySpark Best Practices

Impala 2.0 - The Best Analytic Database for Hadoop

Turning Relational Database Tables into Hadoop Datasources by Kuassi Mensah

Apache Spark in Scientific Applications

Apache Spark in Scientific Applciations

More from Cloudera, Inc.

Partner Briefing_January 25 (FINAL).pptx

Cloudera, Inc.

The document discusses using Cloudera DataFlow to address challenges with collecting, processing, and analyzing log data across many systems and devices. It provides an example use case of logging modernization to reduce costs and enable security solutions by filtering noise from logs. The presentation shows how DataFlow can extract relevant events from large volumes of raw log data and normalize the data to make security threats and anomalies easier to detect across many machines.

Cloudera Data Impact Awards 2021 - Finalists

Cloudera, Inc.

The document outlines the 2021 finalists for the annual Data Impact Awards program, which recognizes organizations using Cloudera's platform and the impactful applications they have developed. It provides details on the challenges, solutions, and outcomes for each finalist project in the categories of Data Lifecycle Connection, Cloud Innovation, Data for Enterprise AI, Security & Governance Leadership, Industry Transformation, People First, and Data for Good. There are multiple finalists highlighted in each category demonstrating innovative uses of data and analytics.

2020 Cloudera Data Impact Awards Finalists

Cloudera, Inc.

Cloudera is proud to present the 2020 Data Impact Awards Finalists. This annual program recognizes organizations running the Cloudera platform for the applications they've built and the impact their data projects have on their organizations, their industries, and the world. Nominations were evaluated by a panel of independent thought-leaders and expert industry analysts, who then selected the finalists and winners. Winners exemplify the most-cutting edge data projects and represent innovation and leadership in their respective industries.

Edc event vienna presentation 1 oct 2019

Cloudera, Inc.

The document outlines the agenda for Cloudera's Enterprise Data Cloud event in Vienna. It includes welcome remarks, keynotes on Cloudera's vision and customer success stories. There will be presentations on the new Cloudera Data Platform and customer case studies, followed by closing remarks. The schedule includes sessions on Cloudera's approach to data warehousing, machine learning, streaming and multi-cloud capabilities.

Machine Learning with Limited Labeled Data 4/3/19

Cloudera, Inc.

Data Driven With the Cloudera Modern Data Warehouse 3.19.19

Cloudera, Inc.

Introducing Cloudera DataFlow (CDF) 2.13.19

Cloudera, Inc.

Introducing Cloudera Data Science Workbench for HDP 2.12.19

Cloudera, Inc.

Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19

Cloudera, Inc.

Leveraging the cloud for analytics and machine learning 1.29.19

Cloudera, Inc.

Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19

Cloudera, Inc.

Leveraging the Cloud for Big Data Analytics 12.11.18

Cloudera, Inc.

Modern Data Warehouse Fundamentals Part 3

Cloudera, Inc.

Modern Data Warehouse Fundamentals Part 2

Cloudera, Inc.

Modern Data Warehouse Fundamentals Part 1

Cloudera, Inc.

The document discusses the benefits and trends of modernizing a data warehouse. It outlines how a modern data warehouse can provide deeper business insights at extreme speed and scale while controlling resources and costs. Examples are provided of companies that have improved fraud detection, customer retention, and machine performance by implementing a modern data warehouse that can handle large volumes and varieties of data from many sources.

Extending Cloudera SDX beyond the Platform

Cloudera, Inc.

Federated Learning: ML with Privacy on the Edge 11.15.18

Cloudera, Inc.

Analyst Webinar: Doing a 180 on Customer 360

Cloudera, Inc.

Build a modern platform for anti-money laundering 9.19.18

Cloudera, Inc.

Introducing the data science sandbox as a service 8.30.18

Cloudera, Inc.

More from Cloudera, Inc. (20)

Partner Briefing_January 25 (FINAL).pptx

Cloudera Data Impact Awards 2021 - Finalists

2020 Cloudera Data Impact Awards Finalists

Edc event vienna presentation 1 oct 2019

Machine Learning with Limited Labeled Data 4/3/19

Data Driven With the Cloudera Modern Data Warehouse 3.19.19

Introducing Cloudera DataFlow (CDF) 2.13.19

Introducing Cloudera Data Science Workbench for HDP 2.12.19

Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19

Leveraging the cloud for analytics and machine learning 1.29.19

Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19

Leveraging the Cloud for Big Data Analytics 12.11.18

Modern Data Warehouse Fundamentals Part 3

Modern Data Warehouse Fundamentals Part 2

Modern Data Warehouse Fundamentals Part 1

Extending Cloudera SDX beyond the Platform

Federated Learning: ML with Privacy on the Edge 11.15.18

Analyst Webinar: Doing a 180 on Customer 360

Build a modern platform for anti-money laundering 9.19.18

Introducing the data science sandbox as a service 8.30.18

Recently uploaded

Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...

Zilliz

Data structures and Algorithms in Python.pdf

TIPNGVN2

GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...

Neo4j

Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.

Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf

Malak Abu Hammad

Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers: * What is Vector Search? * Importance and benefits of vector search * Practical use cases across various industries * Step-by-step implementation guide * Live demos with code snippets * Enhancing LLM capabilities with vector search * Best practices and optimization strategies Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications. #MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology

Cosa hanno in comune un mattoncino Lego e la backdoor XZ?

Speck&Tech

ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune. Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile. BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack

shyamraj55

GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...

Neo4j

Dr. Sean Tan, Head of Data Science, Changi Airport Group Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.

Large Language Model (LLM) and it’s Geospatial Applications

Rohit Gautam

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

Paige Cruz

Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack. While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack. I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:

Monitoring Java Application Security with JDK Tools and JFR Events

Ana-Maria Mihalceanu

Communications Mining Series - Zero to Hero - Session 1

DianaGray10

This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered: • Communication Mining Overview • Why is it important? • How can it help today’s business and the benefits • Phases in Communication Mining • Demo on Platform overview • Q/A

Artificial Intelligence for XMLDevelopment

Octavian Nadolu

In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject. We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup. Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved. The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring. The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise. By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.

Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...

名前です男

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Aggregage

Securing your Kubernetes cluster_ a step-by-step guide to success !

KatiaHIMEUR1

Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster. However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks. In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.

RESUME BUILDER APPLICATION Project for students

KAMESHS29

Building RAG with self-deployed Milvus vector database and Snowpark Container...

Zilliz

GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...

Neo4j

Leonard Jayamohan, Partner & Generative AI Lead, Deloitte This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.

Presentation of the OECD Artificial Intelligence Review of Germany

innovationoecd

Recently uploaded (20)

Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...

Data structures and Algorithms in Python.pdf

GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...

Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf

Cosa hanno in comune un mattoncino Lego e la backdoor XZ?

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack

GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...

Large Language Model (LLM) and it’s Geospatial Applications

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

Monitoring Java Application Security with JDK Tools and JFR Events

Communications Mining Series - Zero to Hero - Session 1

Artificial Intelligence for XMLDevelopment

Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Securing your Kubernetes cluster_ a step-by-step guide to success !

RESUME BUILDER APPLICATION Project for students

Building RAG with self-deployed Milvus vector database and Snowpark Container...

GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...

Presentation of the OECD Artificial Intelligence Review of Germany

Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfaces to Large-scale Data

2. 2© Cloudera, Inc. All rights reserved. Mission for Cloudera: Provide a platform for data analysts, data scientists to efficiently query, analyze, model large-scale data in clusters, cloud storage • By distributing Apache Spark, Apache Impala, other tools • By enabling productive use of these tools Python and R users often have difficulty moving from smaller data to large-scale distributed data • Familiar packages, methods don’t work the same way on distributed data Context

7. 7© Cloudera, Inc. All rights reserved. dplyr provides a set of verbs that perform common data manipulation steps • select() to select columns • filter() to filter rows • arrange() to order rows • mutate() to create new columns • summarise() to aggregate • group_by() to perform operations by group dplyr works on local data and with remote data sources • For remote sources, dplyr commands are translated into SQL dplyr

10. 10© Cloudera, Inc. All rights reserved. dplyr SQL backends dplyr ↕ dbplyr ↕ dplyr SQL backend package* ↕ DBI ↕ DBI-compatible interface package ↕ database driver or connector ↕ database/engine * optional

11. 11© Cloudera, Inc. All rights reserved. • Provides a SQL backend to dplyr for Spark • Also exposes the MLlib API and a subset of the Spark DataFrames API • Developed by RStudio spark.rstudio.com sparklyr

Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfaces to Large-scale Data

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfaces to Large-scale Data

Similar to Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfaces to Large-scale Data (20)

More from Cloudera, Inc.

More from Cloudera, Inc. (20)

Recently uploaded

Recently uploaded (20)

Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfaces to Large-scale Data