Apache Spark Essentials
•
0 likes•213 views
Srikanth Kodeboyina completed the DEV 360 - Apache Spark Essentials course offered by MapR Academy on December 12, 2015. A certificate of completion was issued with the certificate number 56vho9jk85o3, which can be verified online at http://verify.skilljar.com/c/56vho9jk85o3.
Report
Share
Report
Share
Download to read offline
Recommended
Apachee hadoop essentials
Srikanth Kodeboyina has completed the HDE 100 - Apache Hadoop Essentials course offered by MapR Academy. He was issued a Certificate of Completion on December 9, 2015 with certificate number xrcjm9k2m36r that can be verified online at http://verify.skilljar.com/c/xrcjm9k2m36r.
DEV 361 - Build and Monitor Apache Spark Applications
This certificate of completion recognizes that RAVITEJA ADARI has finished the DEV 361 course on building and monitoring Apache Spark applications offered by MapR Academy. The certificate was issued on November 20, 2016 with the identification number hcbe4yihfqmi and a link is provided to verify the authenticity of the certificate online.
Spark Solution for Rank Product
This document describes a Spark solution for calculating the rank product of gene expression data across multiple studies. It contains sections on the author's biography, definitions of key concepts like rank and rank product, a description of the input data format and storage, and an overview of the Spark-based algorithm to calculate rank product in two steps by finding the mean per gene per study and then computing the overall rank product. The solution uses either groupByKey() or combineByKey() to aggregate the rank values by gene.
Docker 基本概念與指令操作
This document discusses Docker, a tool that allows users to run applications securely isolated in a container that runs on the host operating system. It explores key Docker concepts like images, containers, repositories and how they work. It also provides examples of common Docker commands to pull, run, stop and manage images and containers.
Performance in Spark 2.0, PDX Spark Meetup 8/18/16
Brad Carlile of Oracle gave an overview of performance in Spark 2.0 and potential ways to enhance it.
Joker'16 Spark 2 (API changes; Structured Streaming; Encoders)
This is first part of Spark 2 new features overview
This topic covers API changes; Structured Streaming; Encoders; Memory Management in Spark; Tungsten issues; Catalyst features)
Meetup Spark 2.0
Spark 2.0 estará entre nosotros en los próximos meses, nuestro amado Framework de procesamiento en paralelo va a sufrir un gran cambio y debemos estar preparados para afrontarlo.
Durante meses la comunidad y Databricks han estado trabajando en Spark para atender a todas las peticiones de usuarios, se han esforzado en desarrollar potentes utilidades y en continuar haciendo de Spark la herramienta numero uno de Big Data.
使用 CLI 管理 OpenStack 平台
The document discusses using the OpenStack command line interface (CLI) to manage an OpenStack platform. It provides examples of commands for various OpenStack services like Keystone, Glance, Neutron, Nova, Cinder, Swift, Heat, and Ceilometer. The commands can be used to create, delete, modify, query and display resources and include examples like keystone user create, neutron network create, nova boot instance, and ceilometer meter list.
Recommended
Apachee hadoop essentials
Srikanth Kodeboyina has completed the HDE 100 - Apache Hadoop Essentials course offered by MapR Academy. He was issued a Certificate of Completion on December 9, 2015 with certificate number xrcjm9k2m36r that can be verified online at http://verify.skilljar.com/c/xrcjm9k2m36r.
DEV 361 - Build and Monitor Apache Spark Applications
This certificate of completion recognizes that RAVITEJA ADARI has finished the DEV 361 course on building and monitoring Apache Spark applications offered by MapR Academy. The certificate was issued on November 20, 2016 with the identification number hcbe4yihfqmi and a link is provided to verify the authenticity of the certificate online.
Spark Solution for Rank Product
This document describes a Spark solution for calculating the rank product of gene expression data across multiple studies. It contains sections on the author's biography, definitions of key concepts like rank and rank product, a description of the input data format and storage, and an overview of the Spark-based algorithm to calculate rank product in two steps by finding the mean per gene per study and then computing the overall rank product. The solution uses either groupByKey() or combineByKey() to aggregate the rank values by gene.
Docker 基本概念與指令操作
This document discusses Docker, a tool that allows users to run applications securely isolated in a container that runs on the host operating system. It explores key Docker concepts like images, containers, repositories and how they work. It also provides examples of common Docker commands to pull, run, stop and manage images and containers.
Performance in Spark 2.0, PDX Spark Meetup 8/18/16
Brad Carlile of Oracle gave an overview of performance in Spark 2.0 and potential ways to enhance it.
Joker'16 Spark 2 (API changes; Structured Streaming; Encoders)
This is first part of Spark 2 new features overview
This topic covers API changes; Structured Streaming; Encoders; Memory Management in Spark; Tungsten issues; Catalyst features)
Meetup Spark 2.0
Spark 2.0 estará entre nosotros en los próximos meses, nuestro amado Framework de procesamiento en paralelo va a sufrir un gran cambio y debemos estar preparados para afrontarlo.
Durante meses la comunidad y Databricks han estado trabajando en Spark para atender a todas las peticiones de usuarios, se han esforzado en desarrollar potentes utilidades y en continuar haciendo de Spark la herramienta numero uno de Big Data.
使用 CLI 管理 OpenStack 平台
The document discusses using the OpenStack command line interface (CLI) to manage an OpenStack platform. It provides examples of commands for various OpenStack services like Keystone, Glance, Neutron, Nova, Cinder, Swift, Heat, and Ceilometer. The commands can be used to create, delete, modify, query and display resources and include examples like keystone user create, neutron network create, nova boot instance, and ceilometer meter list.
Use of spark for proteomic scoring seattle presentation
This document discusses using Apache Spark to parallelize proteomic scoring, which involves matching tandem mass spectra against a large database of peptides. The author developed a version of the Comet scoring algorithm and implemented it on a Spark cluster. This outperformed single machines by over 10x, allowing searches that took 8 hours to be done in under 30 minutes. Key considerations for running large jobs in parallel on Spark are discussed, such as input formatting, accumulator functions for debugging, and smart partitioning of data. The performance improvements allow searching larger databases and considering more modifications.
Java BigData Full Stack Development (version 2.0)
This document is a presentation by Alexey Zinovyev about Java Big Data full stack development. It discusses Alexey's background and contacts, required skills for Java Big Data development like SQL, Linux, Java and backend skills. It then covers topics like NoSQL databases, Hadoop, Spark, machine learning with MLlib and deep learning. It provides different ways to learn these topics including books, online courses, conferences and mentoring. It encourages learning through hands-on projects and recommends starting with tools like Weka, MongoDB, Hadoop and AWS.
London Spark Meetup Project Tungsten Oct 12 2015
Building on a previous talk about how Spark beat Hadoop @ 100TB Daytona GraySort, we present low-level details of Project Tungsten which includes many CPU and Memory optimizations.
Scala and spark
This document provides an introduction to Apache Spark, including its architecture and programming model. Spark is a cluster computing framework that provides fast, in-memory processing of large datasets across multiple cores and nodes. It improves upon Hadoop MapReduce by allowing iterative algorithms and interactive querying of datasets through its use of resilient distributed datasets (RDDs) that can be cached in memory. RDDs act as immutable distributed collections that can be manipulated using transformations and actions to implement parallel operations.
Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Tech-talk at Bay Area Apache Spark Meetup.
Apache Spark 2.0 will ship with the second generation Tungsten engine. Building upon ideas from modern compilers and MPP databases, and applying them to data processing queries, we have started an ongoing effort to dramatically improve Spark’s performance and bringing execution closer to bare metal. In this talk, we’ll take a deep dive into Apache Spark 2.0’s execution engine and discuss a number of architectural changes around whole-stage code generation/vectorization that have been instrumental in improving CPU efficiency and gaining performance.
Spark Summit EU talk by Ted Malaska
This document summarizes a presentation about unit testing Spark applications. The presentation discusses why it is important to run Spark locally and as unit tests instead of just on a cluster for faster feedback and easier debugging. It provides examples of how to run Spark locally in an IDE and as ScalaTest unit tests, including how to create test RDDs and DataFrames and supply test data. It also discusses testing concepts for streaming applications, MLlib, GraphX, and integration testing with technologies like HBase and Kafka.
Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap...
* Title *
Spark After Dark 1.5: Deep Dive Into Latest Perf and Scale Improvements in Spark Ecosystem
* Abstract *
Combining the most popular and technically-deep material from his wildly popular Advanced Apache Spark Meetup, Chris Fregly will provide code-level deep dives into the latest performance and scalability advancements within the Apache Spark Ecosystem by exploring the following:
1) Building a Scalable and Performant Spark SQL/DataFrames Data Source Connector such as Spark-CSV, Spark-Cassandra, Spark-ElasticSearch, and Spark-Redshift
2) Speeding Up Spark SQL Queries using Partition Pruning and Predicate Pushdowns with CSV, JSON, Parquet, Avro, and ORC
3) Tuning Spark Streaming Performance and Fault Tolerance with KafkaRDD and KinesisRDD
4) Maintaining Stability during High Scale Streaming Ingestion using Approximations and Probabilistic Data Structures from Spark, Redis, and Twitter's Algebird
5) Building Effective Machine Learning Models using Feature Engineering, Dimension Reduction, and Natural Language Processing with MLlib/GraphX, ML Pipelines, DIMSUM, Locality Sensitive Hashing, and Stanford's CoreNLP
6) Tuning Core Spark Performance by Acknowledging Mechanical Sympathy for the Physical Limitations of OS and Hardware Resources such as CPU, Memory, Network, and Disk with Project Tungsten, Asynchronous Netty, and Linux epoll
* Demos *
This talk features many interesting and audience-interactive demos - as well as code-level deep dives into many of the projects listed above.
All demo code is available on Github at the following link: https://github.com/fluxcapacitor/pipeline/wiki
In addition, the entire demo environment has been Dockerized and made available for download on Docker Hub at the following link: https://hub.docker.com/r/fluxcapacitor/pipeline/
* Speaker Bio *
Chris Fregly is a Principal Data Solutions Engineer for the newly-formed IBM Spark Technology Center, an Apache Spark Contributor, a Netflix Open Source Committer, as well as the Organizer of the global Advanced Apache Spark Meetup and Author of the Upcoming Book, Advanced Spark.
Previously, Chris was a Data Solutions Engineer at Databricks and a Streaming Data Engineer at Netflix.
When Chris isn’t contributing to Spark and other open source projects, he’s creating book chapters, slides, and demos to share knowledge with his peers at meetups and conferences throughout the world.
Data Source API in Spark
Video of the presentation can be seen here: https://www.youtube.com/watch?v=uxuLRiNoDio
The Data Source API in Spark is a convenient feature that enables developers to write libraries to connect to data stored in various sources with Spark. Equipped with the Data Source API, users can load/save data from/to different data formats and systems with minimal setup and configuration. In this talk, we introduce the Data Source API and the unified load/save functions built on top of it. Then, we show examples to demonstrate how to build a data source library.
Deep Dive Into Catalyst: Apache Spark 2.0’s Optimizer
Catalyst is becoming one of the most important components in Apache Spark, as it underpins all the major new APIs in Spark 2.0, from DataFrames, Datasets, to streaming. At its core, Catalyst is a general library for manipulating trees. Based on this library, we have built a modular compiler frontend for Spark, including a query analyzer, optimizer, and an execution planner. In this talk, I will first introduce the concepts of Catalyst trees, followed by major features that were added in order to support Spark’s powerful API abstractions. Audience will walk away with a deeper understanding of how Spark 2.0 works under the hood.
Introduction to Apache Spark 2.0
Spark 2.0 is a major release of Apache Spark. This release has brought many changes to API(s) and libraries of Spark. So in this KnolX, we will be looking at some improvements that are made in Spark 2.0. Also, in these slides we will be getting an introduction to some new features in Spark 2,0 like SparkSession API and Structured Streaming.
Spark Kernel Talk - Apache Spark Meetup San Francisco (July 2015)
NOTE: This was converted to Powerpoint from Keynote. Slideshare does not play the embedded videos. You can download the powerpoint from slideshare and import it into keynote. The videos should work in the keynote.
Abstract:
In this presentation, we will describe the "Spark Kernel" which enables applications, such as end-user facing and interactive applications, to interface with Spark clusters. It provides a gateway to define and run Spark tasks and to collect results from a cluster without the friction associated with shipping jars and reading results from peripheral systems. Using the Spark Kernel as a proxy, applications can be hosted remotely from Spark.
Robust and Scalable ETL over Cloud Storage with Apache Spark
The majority of reported Spark deployments are now in the cloud. In such an environment, it is preferable for Spark to access data directly from services such as Amazon S3, thereby decoupling storage and compute. However, there are limitations to object stores such as S3. Chained or concurrent ETL jobs often run into issues on S3 due to inconsistent file listings and the lack of atomic rename support. Metadata performance also becomes an issue when running jobs over many thousands to millions of files.
Speaker: Eric Liang
This talk was originally presented at Spark Summit East 2017.
Keeping Spark on Track: Productionizing Spark for ETL
ETL is the first phase when building a big data processing platform. Data is available from various sources and formats, and transforming the data into a compact binary format (Parquet, ORC, etc.) allows Apache Spark to process it in the most efficient manner. This talk will discuss common issues and best practices for speeding up your ETL workflows, handling dirty data, and debugging tips for identifying errors.
Speakers: Kyle Pistor & Miklos Christine
This talk was originally presented at Spark Summit East 2017.
Collaborative Filtering with Spark
This document summarizes an approach for scaling implicit matrix factorization to large datasets using Apache Spark. It discusses three attempts at implementing alternating least squares for collaborative filtering in Spark. The first two attempts shuffle data across nodes on each iteration. The third attempt partitions and caches the user/item vectors, then builds mappings to join local blocks of data and update vectors within each partition, avoiding shuffles between iterations for more efficient distributed computation.
Exceptions are the Norm: Dealing with Bad Actors in ETL
Stable and robust data pipelines are a critical component of the data infrastructure of enterprises. Most commonly, data pipelines ingest messy data sources with incorrect, incomplete or inconsistent records and produce curated and/or summarized data for consumption by subsequent applications.
In this talk, we go over new and upcoming features in Spark that enabled it to better serve such workloads. Such features include isolation of corrupt input records and files, useful diagnostic feedback to users and improved support for nested type handling which is common in ETL jobs.
Speaker: Sameer Agarwal
This talk was originally presented at Spark Summit East 2017.
Trends for Big Data and Apache Spark in 2017 by Matei Zaharia
Big data remains a rapidly evolving field with new applications and infrastructure appearing every year. In this talk, I’ll cover new trends in 2016 / 2017 and how Apache Spark is moving to meet them. In particular, I’ll talk about work Databricks is doing to make Apache Spark interact better with native code (e.g. deep learning libraries), support heterogeneous hardware, and simplify production data pipelines in both streaming and batch settings through Structured Streaming.
Hive on spark is blazing fast or is it final
This presentation was given at the Strata + Hadoop World, 2015 in San Jose.
Apache Hive is the most popular and most widely used SQL solution for Hadoop. To keep pace with Hadoop’s increasingly vital role in the Enterprise, Hive has transformed from a batch-only, high-latency system into a modern SQL engine capable of both batch and interactive queries over large datasets. Hive’s momentum is accelerating: With Spark integration and a shift to in-memory processing on the horizon, Hive continues to expand the boundaries of Big Data.
In this talk the speakers examined Hive performance, past, present and future. In particular they looked at Hive’s origins as a petabyte scale SQL engine.
Through some numbers and graphs, they showed how Hive became 100x faster by moving beyond MapReduce, by vectorizing execution and by introducing a cost-based optimizer.
They detailed and discussed the challenges of scalable SQL on Hadoop.
The looked into Hive’s sub-second future, powered by LLAP and Hive on Spark.
And showed just how fast Hive on Spark really is.
More Related Content
Viewers also liked
Use of spark for proteomic scoring seattle presentation
This document discusses using Apache Spark to parallelize proteomic scoring, which involves matching tandem mass spectra against a large database of peptides. The author developed a version of the Comet scoring algorithm and implemented it on a Spark cluster. This outperformed single machines by over 10x, allowing searches that took 8 hours to be done in under 30 minutes. Key considerations for running large jobs in parallel on Spark are discussed, such as input formatting, accumulator functions for debugging, and smart partitioning of data. The performance improvements allow searching larger databases and considering more modifications.
Java BigData Full Stack Development (version 2.0)
This document is a presentation by Alexey Zinovyev about Java Big Data full stack development. It discusses Alexey's background and contacts, required skills for Java Big Data development like SQL, Linux, Java and backend skills. It then covers topics like NoSQL databases, Hadoop, Spark, machine learning with MLlib and deep learning. It provides different ways to learn these topics including books, online courses, conferences and mentoring. It encourages learning through hands-on projects and recommends starting with tools like Weka, MongoDB, Hadoop and AWS.
London Spark Meetup Project Tungsten Oct 12 2015
Building on a previous talk about how Spark beat Hadoop @ 100TB Daytona GraySort, we present low-level details of Project Tungsten which includes many CPU and Memory optimizations.
Scala and spark
This document provides an introduction to Apache Spark, including its architecture and programming model. Spark is a cluster computing framework that provides fast, in-memory processing of large datasets across multiple cores and nodes. It improves upon Hadoop MapReduce by allowing iterative algorithms and interactive querying of datasets through its use of resilient distributed datasets (RDDs) that can be cached in memory. RDDs act as immutable distributed collections that can be manipulated using transformations and actions to implement parallel operations.
Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Tech-talk at Bay Area Apache Spark Meetup.
Apache Spark 2.0 will ship with the second generation Tungsten engine. Building upon ideas from modern compilers and MPP databases, and applying them to data processing queries, we have started an ongoing effort to dramatically improve Spark’s performance and bringing execution closer to bare metal. In this talk, we’ll take a deep dive into Apache Spark 2.0’s execution engine and discuss a number of architectural changes around whole-stage code generation/vectorization that have been instrumental in improving CPU efficiency and gaining performance.
Spark Summit EU talk by Ted Malaska
This document summarizes a presentation about unit testing Spark applications. The presentation discusses why it is important to run Spark locally and as unit tests instead of just on a cluster for faster feedback and easier debugging. It provides examples of how to run Spark locally in an IDE and as ScalaTest unit tests, including how to create test RDDs and DataFrames and supply test data. It also discusses testing concepts for streaming applications, MLlib, GraphX, and integration testing with technologies like HBase and Kafka.
Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap...
* Title *
Spark After Dark 1.5: Deep Dive Into Latest Perf and Scale Improvements in Spark Ecosystem
* Abstract *
Combining the most popular and technically-deep material from his wildly popular Advanced Apache Spark Meetup, Chris Fregly will provide code-level deep dives into the latest performance and scalability advancements within the Apache Spark Ecosystem by exploring the following:
1) Building a Scalable and Performant Spark SQL/DataFrames Data Source Connector such as Spark-CSV, Spark-Cassandra, Spark-ElasticSearch, and Spark-Redshift
2) Speeding Up Spark SQL Queries using Partition Pruning and Predicate Pushdowns with CSV, JSON, Parquet, Avro, and ORC
3) Tuning Spark Streaming Performance and Fault Tolerance with KafkaRDD and KinesisRDD
4) Maintaining Stability during High Scale Streaming Ingestion using Approximations and Probabilistic Data Structures from Spark, Redis, and Twitter's Algebird
5) Building Effective Machine Learning Models using Feature Engineering, Dimension Reduction, and Natural Language Processing with MLlib/GraphX, ML Pipelines, DIMSUM, Locality Sensitive Hashing, and Stanford's CoreNLP
6) Tuning Core Spark Performance by Acknowledging Mechanical Sympathy for the Physical Limitations of OS and Hardware Resources such as CPU, Memory, Network, and Disk with Project Tungsten, Asynchronous Netty, and Linux epoll
* Demos *
This talk features many interesting and audience-interactive demos - as well as code-level deep dives into many of the projects listed above.
All demo code is available on Github at the following link: https://github.com/fluxcapacitor/pipeline/wiki
In addition, the entire demo environment has been Dockerized and made available for download on Docker Hub at the following link: https://hub.docker.com/r/fluxcapacitor/pipeline/
* Speaker Bio *
Chris Fregly is a Principal Data Solutions Engineer for the newly-formed IBM Spark Technology Center, an Apache Spark Contributor, a Netflix Open Source Committer, as well as the Organizer of the global Advanced Apache Spark Meetup and Author of the Upcoming Book, Advanced Spark.
Previously, Chris was a Data Solutions Engineer at Databricks and a Streaming Data Engineer at Netflix.
When Chris isn’t contributing to Spark and other open source projects, he’s creating book chapters, slides, and demos to share knowledge with his peers at meetups and conferences throughout the world.
Data Source API in Spark
Video of the presentation can be seen here: https://www.youtube.com/watch?v=uxuLRiNoDio
The Data Source API in Spark is a convenient feature that enables developers to write libraries to connect to data stored in various sources with Spark. Equipped with the Data Source API, users can load/save data from/to different data formats and systems with minimal setup and configuration. In this talk, we introduce the Data Source API and the unified load/save functions built on top of it. Then, we show examples to demonstrate how to build a data source library.
Deep Dive Into Catalyst: Apache Spark 2.0’s Optimizer
Catalyst is becoming one of the most important components in Apache Spark, as it underpins all the major new APIs in Spark 2.0, from DataFrames, Datasets, to streaming. At its core, Catalyst is a general library for manipulating trees. Based on this library, we have built a modular compiler frontend for Spark, including a query analyzer, optimizer, and an execution planner. In this talk, I will first introduce the concepts of Catalyst trees, followed by major features that were added in order to support Spark’s powerful API abstractions. Audience will walk away with a deeper understanding of how Spark 2.0 works under the hood.
Introduction to Apache Spark 2.0
Spark 2.0 is a major release of Apache Spark. This release has brought many changes to API(s) and libraries of Spark. So in this KnolX, we will be looking at some improvements that are made in Spark 2.0. Also, in these slides we will be getting an introduction to some new features in Spark 2,0 like SparkSession API and Structured Streaming.
Spark Kernel Talk - Apache Spark Meetup San Francisco (July 2015)
NOTE: This was converted to Powerpoint from Keynote. Slideshare does not play the embedded videos. You can download the powerpoint from slideshare and import it into keynote. The videos should work in the keynote.
Abstract:
In this presentation, we will describe the "Spark Kernel" which enables applications, such as end-user facing and interactive applications, to interface with Spark clusters. It provides a gateway to define and run Spark tasks and to collect results from a cluster without the friction associated with shipping jars and reading results from peripheral systems. Using the Spark Kernel as a proxy, applications can be hosted remotely from Spark.
Robust and Scalable ETL over Cloud Storage with Apache Spark
The majority of reported Spark deployments are now in the cloud. In such an environment, it is preferable for Spark to access data directly from services such as Amazon S3, thereby decoupling storage and compute. However, there are limitations to object stores such as S3. Chained or concurrent ETL jobs often run into issues on S3 due to inconsistent file listings and the lack of atomic rename support. Metadata performance also becomes an issue when running jobs over many thousands to millions of files.
Speaker: Eric Liang
This talk was originally presented at Spark Summit East 2017.
Keeping Spark on Track: Productionizing Spark for ETL
ETL is the first phase when building a big data processing platform. Data is available from various sources and formats, and transforming the data into a compact binary format (Parquet, ORC, etc.) allows Apache Spark to process it in the most efficient manner. This talk will discuss common issues and best practices for speeding up your ETL workflows, handling dirty data, and debugging tips for identifying errors.
Speakers: Kyle Pistor & Miklos Christine
This talk was originally presented at Spark Summit East 2017.
Collaborative Filtering with Spark
This document summarizes an approach for scaling implicit matrix factorization to large datasets using Apache Spark. It discusses three attempts at implementing alternating least squares for collaborative filtering in Spark. The first two attempts shuffle data across nodes on each iteration. The third attempt partitions and caches the user/item vectors, then builds mappings to join local blocks of data and update vectors within each partition, avoiding shuffles between iterations for more efficient distributed computation.
Exceptions are the Norm: Dealing with Bad Actors in ETL
Stable and robust data pipelines are a critical component of the data infrastructure of enterprises. Most commonly, data pipelines ingest messy data sources with incorrect, incomplete or inconsistent records and produce curated and/or summarized data for consumption by subsequent applications.
In this talk, we go over new and upcoming features in Spark that enabled it to better serve such workloads. Such features include isolation of corrupt input records and files, useful diagnostic feedback to users and improved support for nested type handling which is common in ETL jobs.
Speaker: Sameer Agarwal
This talk was originally presented at Spark Summit East 2017.
Trends for Big Data and Apache Spark in 2017 by Matei Zaharia
Big data remains a rapidly evolving field with new applications and infrastructure appearing every year. In this talk, I’ll cover new trends in 2016 / 2017 and how Apache Spark is moving to meet them. In particular, I’ll talk about work Databricks is doing to make Apache Spark interact better with native code (e.g. deep learning libraries), support heterogeneous hardware, and simplify production data pipelines in both streaming and batch settings through Structured Streaming.
Hive on spark is blazing fast or is it final
This presentation was given at the Strata + Hadoop World, 2015 in San Jose.
Apache Hive is the most popular and most widely used SQL solution for Hadoop. To keep pace with Hadoop’s increasingly vital role in the Enterprise, Hive has transformed from a batch-only, high-latency system into a modern SQL engine capable of both batch and interactive queries over large datasets. Hive’s momentum is accelerating: With Spark integration and a shift to in-memory processing on the horizon, Hive continues to expand the boundaries of Big Data.
In this talk the speakers examined Hive performance, past, present and future. In particular they looked at Hive’s origins as a petabyte scale SQL engine.
Through some numbers and graphs, they showed how Hive became 100x faster by moving beyond MapReduce, by vectorizing execution and by introducing a cost-based optimizer.
They detailed and discussed the challenges of scalable SQL on Hadoop.
The looked into Hive’s sub-second future, powered by LLAP and Hive on Spark.
And showed just how fast Hive on Spark really is.
Viewers also liked (18)
Use of spark for proteomic scoring seattle presentation
Use of spark for proteomic scoring seattle presentation
Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap...
Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap...
Deep Dive Into Catalyst: Apache Spark 2.0’s Optimizer
Deep Dive Into Catalyst: Apache Spark 2.0’s Optimizer
Spark Kernel Talk - Apache Spark Meetup San Francisco (July 2015)
Spark Kernel Talk - Apache Spark Meetup San Francisco (July 2015)
Robust and Scalable ETL over Cloud Storage with Apache Spark
Robust and Scalable ETL over Cloud Storage with Apache Spark
Keeping Spark on Track: Productionizing Spark for ETL
Keeping Spark on Track: Productionizing Spark for ETL
Exceptions are the Norm: Dealing with Bad Actors in ETL
Exceptions are the Norm: Dealing with Bad Actors in ETL
Trends for Big Data and Apache Spark in 2017 by Matei Zaharia
Trends for Big Data and Apache Spark in 2017 by Matei Zaharia
Apache Spark Essentials
- 1. Certificate of Completion Srikanth Kodeboyina has completed DEV 360 - Apache Spark Essentials offered by MapR Academy Issued: December 12, 2015 Certificate No: 56vho9jk85o3 View: http://verify.skilljar.com/c/56vho9jk85o3