Scala: the unpredicted lingua franca for data science

Distributed Data Science… * A genomics use case * Spark Notebook * Interactive Distributed Data Science Distributed Data Science… Pipeline * Pipeline: productizing Data Science * Demo of Distributed Pipeline (ADAM, Akka, Cassandra, Parquet, Spark) * Why Micro Services? * Painful points: * Data science is Discontiguous * Context Lost in Translation * Solution: Data Fellas’ Agile Data Science Toolkit

Agile data science: Distributed, Interactive, Integrated, Semantic, Micro Ser...

The central premise of DataXu is to apply data science to better marketing. At its core, is the Real Time Bidding Platform that processes 2 Petabytes of data per day and responds to ad auctions at a rate of 2.1 million requests per second across 5 different continents. Serving on top of this platform is Dataxu’s analytics engine that gives their clients insightful analytics reports addressed towards client marketing business questions. Some common requirements for both these platforms are the ability to do real-time processing, scalable machine learning, and ad-hoc analytics. This talk will showcase DataXu’s successful use-cases of using the Apache Spark framework and Databricks to address all of the above challenges while maintaining its agility and rapid prototyping strengths to take a product from initial R&D phase to full production. The team will share their best practices and highlight the steps of large scale Spark ETL processing, model testing, all the way through to interactive analytics.

Fishing Graphs in a Hadoop Data Lake by Jörg Schad and Max Neunhoeffer at Big...

Big Data Spain

R&D to Product Pipeline Using Apache Spark in AdTech: Spark Summit East talk ...

Spark Summit

Ensemble machine learning methods are often used when the true prediction function is not easily approximated by a single algorithm. Practitioners may prefer ensemble algorithms when model performance is valued above other factors such as model complexity and training time. The Super Learner algorithm, also called "stacking", learns the optimal combination of the base learner fits. The latest version of H2O now contains a "Stacked Ensemble" method, which allows the user to stack H2O models into a Super Learner. The Stacked Ensemble method is the the native H2O version of stacking, previously only available in the h2oEnsemble R package, and now enables stacking from all the H2O APIs: Python, R, Scala, etc. Erin is a Statistician and Machine Learning Scientist at H2O.ai. Before joining H2O, she was the Principal Data Scientist at Wise.io (acquired by GE Digital) and Marvin Mobile Security (acquired by Veracode) and the founder of DataScientific, Inc. Erin received her Ph.D. from University of California, Berkeley. Her research focuses on ensemble machine learning, learning from imbalanced binary-outcome data, influence curve based variance estimation and statistical computing.

Stacked Ensembles in H2O

Sri Ambati

Serverless data pipelines gcp

Catherine Kimani

Distributed machine learning 101 using apache spark from a browser devoxx.b...

Way too many enterprise decision makers have clouded and uninformed views of how Hadoop works and what it does. Donald Miner offers high-level observations about Hadoop technologies and explains how Hadoop can shift the paradigms inside of an organization, based on his report Hadoop: What You Need To Know—Hadoop Basics for the Enterprise Decision Maker, forthcoming from O’Reilly Media. After a basic introduction to Hadoop and the Hadoop ecosystem, Donald outlines 10 basic concepts you need to understand to master Hadoop: Hadoop masks being a distributed system: what it means for Hadoop to abstract away the details of distributed systems and why that’s a good thing Hadoop scales out linearly: why Hadoop’s linear scalability is a paradigm shift (but one with a few downsides) Hadoop runs on commodity hardware: an honest definition of commodity hardware and why this is a good thing for enterprises Hadoop handles unstructured data: why Hadoop is better for unstructured data than other data systems from a storage and computation perspective In Hadoop, you load data first and ask questions later: the differences between schema-on-read and schema-on-write and the drawbacks this represents Hadoop is open source: what it really means for Hadoop to be open source from a practical perspective, not just a “feel good” perspective HDFS stores the data but has some major limitations: an overview of HDFS (replication, not being able to edit files, and the NameNode) YARN controls everything going on and is mostly behind the scenes: an overview of YARN and the pitfalls of sharing resources in a distributed environment and the capacity scheduler MapReduce may be getting a bad rap, but it’s still really important: an overview of MapReduce (what it’s good at and bad at and why, while it isn’t used as much these days, it still plays an important role) The Hadoop ecosystem is constantly growing and evolving: an overview of current tools such as Spark and Kafka and a glimpse of some things on the horizon

PyData: The Next Generation

Wes McKinney

10 concepts the enterprise decision maker needs to understand about Hadoop

Donald Miner

Machine Learning with Spark

elephantscale

The combination of Deep Learning with Apache Spark has the potential for tremendous impact in many sectors of the industry. This webinar, based on the experience gained in assisting customers with the Databricks Virtual Analytics Platform, will present some best practices for building deep learning pipelines with Spark. Rather than comparing deep learning systems or specific optimizations, this webinar will focus on issues that are common to deep learning frameworks when running on a Spark cluster, including: * optimizing cluster setup; * configuring the cluster; * ingesting data; and * monitoring long-running jobs. We will demonstrate the techniques we cover using Google’s popular TensorFlow library. More specifically, we will cover typical issues users encounter when integrating deep learning libraries with Spark clusters. Clusters can be configured to avoid task conflicts on GPUs and to allow using multiple GPUs per worker. Setting up pipelines for efficient data ingest improves job throughput, and monitoring facilitates both the work of configuration and the stability of deep learning jobs.

Deep Learning on Apache® Spark™ : Workflows and Best Practices

Jen Aman

Spark Summit East Keynote by Ion Stoica A long-standing grand challenge in computing is to enable machines to act autonomously and intelligently: to rapidly and repeatedly take appropriate actions based on information in the world around them. To address this challenge, at UC Berkeley we are starting a new five year effort that focuses on the development of data-intensive systems that provide Real-Time Intelligence with Secure Execution (RISE). Following in the footsteps of AMPLab, RISELab is an interdisciplinary effort bringing together researchers across AI, robotics, security, and data systems. In this talk I’ll present our research vision and then discuss some of the applications that will be enabled by RISE technologies.

RISELab:Enabling Intelligent Real-Time Decisions

Jen Aman

What's hot (20)

Enabling Python to be a Better Big Data Citizen

Introduction to Analytics with Azure Notebooks and Python

Analyzing Data With Python

Apache Spark for Everyone - Women Who Code Workshop

CuRious about R in Power BI? End to end R in Power BI for beginners

DataFrames: The Good, Bad, and Ugly

Paris Data Geek - Spark Streaming

Talend spark meetup 03042017 - Paris Spark Meetup

Briefing on the Modern ML Stack with R

Agile data science: Distributed, Interactive, Integrated, Semantic, Micro Ser...

Fishing Graphs in a Hadoop Data Lake by Jörg Schad and Max Neunhoeffer at Big...

R&D to Product Pipeline Using Apache Spark in AdTech: Spark Summit East talk ...

Stacked Ensembles in H2O

Serverless data pipelines gcp

Distributed machine learning 101 using apache spark from a browser devoxx.b...

PyData: The Next Generation

10 concepts the enterprise decision maker needs to understand about Hadoop

Machine Learning with Spark

Deep Learning on Apache® Spark™ : Workflows and Best Practices

RISELab:Enabling Intelligent Real-Time Decisions

Viewers also liked

We are at a time where biotech allow us to get personal genomes for $1000. Tremendous progress since the 70s in DNA sequencing have been done, e.g. more samples in an experiment, more genomic coverages at higher speeds. Genomic analysis standards that have been developed over the years weren't designed with scalability and adaptability in mind. In this talk, we’ll present a game changing technology in this area, ADAM, initiated by the AMPLab at Berkeley. ADAM is framework based on Apache Spark and the Parquet storage. We’ll see how it can speed up a sequence reconstruction to a factor 150.

Lightning fast genomics with Spark, Adam and Scala

Share and analyze geonomic data at scale by Andy Petrella and Xavier Tordoir

Spark Summit

What went right or wrong on election day - Nov 8th, 2016 USA

Marshall Sponder

Ga4 gh meeting at the the sanger institute

Matt Massie

ADAM—Spark Summit, 2014

fnothaft

Virdata: lessons learned from the Internet of Things and M2M Cloud Services @...

Nathan Bijnens

Spark Summit EU 2015: Reynold Xin Keynote

Our software needs to become reactive, this realization is widely understood: we need to consider responsiveness, maintainability, elasticity and scalability from the outset. Not all systems need to implement all these to the same degree, specific project requirements will determine where effort is most wisely spent, but in the vast majority of cases the need to go reactive will demand that we design our applications differently. In this presentation we explore several architecture elements that are commonly found in reactive systems (like the circuit breaker, various replication techniques, or flow control protocols). These patterns are language agnostic and also independent of the abundant choice of reactive programming frameworks and libraries, they are well-specified starting points for exploring the design space of a concrete problem: thinking is strictly required!

Reactive Design Patterns — J on the Beach

Roland Kuhn

Strata Big Data Science Talk on ADAM

Matt Massie

Spark is a powerful new tool for processing large volumes of data quickly across a cluster of networked computers. Typical bioinformatics workflow requirements are well-matched to Spark’s capabilities. However, Spark is not commonly used because many legacy bioinformatics applications make assumptions about their computing environment. These assumptions present a barrier to integrating the tools into more modern computing environments. These barriers are quickly coming down. ADAM is a software library and set of tools built on top of Spark that make it easy work with file formats commonly used for genome analysis like FastQ, BAM, and VCF. In this presentation, we’ll explore how a step that is common to many bioinformatics workflows, sequence alignment, can done with Bowtie and ADAM inside a Spark environment to quickly align short reads to a reference genome. A complete code example is demonstrated and provided at https://github.com/allenday/spark-genome-alignment-demo

Genome Analysis Pipelines with Spark and ADAM

Allen Day, PhD

Spark Summit EU 2015: Revolutionizing Big Data in the Enterprise with Spark

R is a favorite language of many data scientists. In addition to a language and runtime, R is a rich ecosystem of libraries for a wide range of use cases from statistical inference to data visualization. However, handling large datasets with R is challenging, especially when data scientists use R with frameworks or tools written in other languages. In this mode most of the friction is at the interface of R and the other systems. For example, when data is sampled by a big data platform, results need to be transferred to and imported in R as native data structures. In this talk we show how SparkR solves these problems to enable a much smoother experience. In this talk we will present an overview of the SparkR architecture, including how data and control is transferred between R and JVM. This knowledge will help data scientists make better decisions when using SparkR. We will demo and explain some of the existing and supported use cases with real large datasets inside a notebook environment. The demonstration will emphasize how Spark clusters, R and interactive notebook environments, such as Jupyter or Databricks, facilitate exploratory analysis of large data.

Enabling exploratory data science with Spark and R

2015 was a year of continued growth for Spark, with numerous additions to the core project and very fast growth of use cases across the industry. In this talk, I’ll look back at how the Spark community is has grown and changed in 2015, based on a large Apache Spark user survey conducted by Databricks. We see some interesting trends in the diversity of runtime environments (which are increasingly not just Hadoop); the types of applications run on Spark; and the types of users, now that features like R support and DataFrames are available in Spark. I’ll also cover the ongoing work in the upcoming releases of Spark to support new use cases.

Spark Summit EU 2015: Matei Zaharia keynote

This talk discusses integrating common data science tools like Python pandas, scikit-learn, and R with MLlib, Spark’s distributed Machine Learning (ML) library. Integration is simple; migration to distributed ML can be done lazily; and scaling to big data can significantly improve accuracy. We demonstrate integration with a simple data science workflow. Data scientists often encounter scaling bottlenecks with single-machine ML tools. Yet the overhead in migrating to a distributed workflow can seem daunting. In this talk, we demonstrate such a migration, taking advantage of Spark and MLlib’s integration with common ML libraries. We begin with a small dataset which runs on a single machine. Increasing the size, we hit bottlenecks in various parts of the workflow: hyperparameter tuning, then ETL, and eventually the core learning algorithm. As we hit each bottleneck, we parallelize that part of the workflow using Spark and MLlib. As we increase the dataset and model size, we can see significant gains in accuracy. We end with results demonstrating the impressive scalability of MLlib algorithms. With accuracy comparable to traditional ML libraries, combined with state-of-the-art distributed scalability, MLlib is a valuable new tool for the modern data scientist.

Introduction to Apache Kafka- Part 2

Knoldus Inc.

Spark Summit EU 2015: Combining the Strengths of MLlib, scikit-learn, and R

At Databricks, we have a unique view into over a hundred different companies trying out Spark for development and production use-cases, from their support tickets and forum posts. Having seen so many different workflows and applications, some discernible patterns emerge when looking at common performance and scalability issues that our users run into. This talk will discuss some of these common common issues from an engineering and operations perspective, describing solutions and clarifying misconceptions.

Introduction to Apache Kafka- Part 1

Knoldus Inc.

Spark Summit EU 2015: Lessons from 300+ production users

Reactive streaming is becoming the best approach to handle data flows across asynchronous boundaries. Here, we present the implementation of a real-world application based on Akka Streams. After reviewing the basics, we will discuss the development of a data processing pipeline that collects real-time sensor data and sends it to a Kinesis stream. There are various possible point of failures in this architecture. What should happen when Kinesis is unavailable? If the data flow is not handled in the correct way, some information may get lost. Akka Streams are the tools that enabled us to build a reliable processing logic for the pipeline that avoids data losses and maximizes the robustness of the entire system.

A dive into akka streams: from the basics to a real-world scenario

Gioia Ballin

Scenario Building Workshop - How to Build and Use Scenarios

Adam Vigdor Gordon

Viewers also liked (19)

Lightning fast genomics with Spark, Adam and Scala

Share and analyze geonomic data at scale by Andy Petrella and Xavier Tordoir

What went right or wrong on election day - Nov 8th, 2016 USA

Ga4 gh meeting at the the sanger institute

ADAM—Spark Summit, 2014

Virdata: lessons learned from the Internet of Things and M2M Cloud Services @...

Spark Summit EU 2015: Reynold Xin Keynote

Reactive Design Patterns — J on the Beach

Strata Big Data Science Talk on ADAM

Genome Analysis Pipelines with Spark and ADAM

Spark Summit EU 2015: Revolutionizing Big Data in the Enterprise with Spark

Enabling exploratory data science with Spark and R

Spark Summit EU 2015: Matei Zaharia keynote

Introduction to Apache Kafka- Part 2

Spark Summit EU 2015: Combining the Strengths of MLlib, scikit-learn, and R

Introduction to Apache Kafka- Part 1

Spark Summit EU 2015: Lessons from 300+ production users

A dive into akka streams: from the basics to a real-world scenario

Scenario Building Workshop - How to Build and Use Scenarios

Similar to Scala: the unpredicted lingua franca for data science

Spark 101

Shahaf Azriely {TopLinked} ☁

Big Data Processing with Spark and Scala

Spark For Faster Batch Processing

Big Data Processing with Apache Spark 2014

mahchiev

spark_v1_2

Frank Schroeter

From Pandas to Koalas: Reducing Time-To-Insight for Virgin Hyperloop's Data

Happiest Minds Technologies

A short introduction to Spark and its benefits

Johan Picard

Started with-apache-spark

Apache Spark & Scala

Given at Data Day Seattle 2015. Bitly generates over 9 billion clicks on shortened links a month, as well as over 100 million unique link shortens. Analyzing data of this scale is not without its challenges. At Bitly, we have started adopting Apache Spark as a way to process our data. In this talk, I’ll elaborate on how I use Spark as part of my data science workflow. I’ll cover how Spark fits into our existing architecture, the kind of problems I’m solving with Spark, and the benefits and challenges of using Spark for large-scale data science.

Introduction to spark

Home

Data Science at Scale: Using Apache Spark for Data Science at Bitly

Sarah Guido

Scalable Machine Learning with PySpark

Ladle Patel

20160512 apache-spark-for-everyone

Amanda Casari

http://bit.ly/1BTaXZP – As organizations look for even faster ways to derive value from big data, they are turning to Apache Spark is an in-memory processing framework that offers lightning-fast big data analytics, providing speed, developer productivity, and real-time processing advantages. The Spark software stack includes a core data-processing engine, an interface for interactive querying, Spark Streaming for streaming data analysis, and growing libraries for machine-learning and graph analysis. Spark is quickly establishing itself as a leading environment for doing fast, iterative in-memory and streaming analysis. This talk will give an introduction the Spark stack, explain how Spark has lighting fast results, and how it complements Apache Hadoop. By the end of the session, you’ll come away with a deeper understanding of how you can unlock deeper insights from your data, faster, with Spark.

Intro to Apache Spark by CTO of Twingo

MapR Technologies

Spark SQL | Apache Spark

Big Data Processing With Spark

Spark - The Ultimate Scala Collections by Martin Odersky

Spark Summit

Big data Processing with Apache Spark & Scala

Apache spark-melbourne-april-2015-meetup

Ned Shawa

Scaling up with hadoop and banyan at ITRIX-2015, College of Engineering, Guindy

Rohit Kulkarni

Similar to Scala: the unpredicted lingua franca for data science (20)

Spark 101

Big Data Processing with Spark and Scala

Spark For Faster Batch Processing

Big Data Processing with Apache Spark 2014

spark_v1_2

From Pandas to Koalas: Reducing Time-To-Insight for Virgin Hyperloop's Data

A short introduction to Spark and its benefits

Started with-apache-spark

Apache Spark & Scala

Introduction to spark

Data Science at Scale: Using Apache Spark for Data Science at Bitly

Scalable Machine Learning with PySpark

20160512 apache-spark-for-everyone

Intro to Apache Spark by CTO of Twingo

Spark SQL | Apache Spark

Big Data Processing With Spark

Spark - The Ultimate Scala Collections by Martin Odersky

Big data Processing with Apache Spark & Scala

Apache spark-melbourne-april-2015-meetup

Scaling up with hadoop and banyan at ITRIX-2015, College of Engineering, Guindy

More from Andy Petrella

Data Observability Best Pracices

How to Build a Global Data Mapping

Interactive notebooks

Governance compliance

Data science governance and GDPR

Data science governance : what and how

What is a distributed data science pipeline. how with apache spark and friends.

Nowadays, Data Science is buzzing all over the place. But what is a, so-called, Data Scientist? Some will argue that a Data Scientist is a person able to report and present insights in a data set. Others will say that a Data Scientist can handle a high throughput of values and expose them in services. Yet another definition includes the capacity to create meaningful visualizations on the data. However, we enter an age where velocity is a key. Not only the velocity of your data is high, but the time to market is shortened. Hence, the time separating the moment you receive a set of data and the time you’ll be able to deliver added value is crucial. In this talk, we’ll review the legacy Data Science methodologies, what it meant in terms of delivered work and results. Afterwards, we’ll slightly move towards different concepts, techniques and tools that Data Scientists will have to learn and appropriate in order to accomplish their tasks in the age of Big Data. The dissertation is closed by exposing the Data Fellas view on a solution to the challenges, specially thanks to the Spark Notebook and the Shar3 product we develop.

Towards a rebirth of data science (by Data Fellas)

Spark Summit Europe: Share and analyse genomic data at scale

Leveraging mesos as the ultimate distributed data science platform

Data science requires so many skills, people and time before the results can be accessed. Moreover, these results cannot be static anymore. And finally, the Big Data comes to the plate and the whole tool chain needs to change. In this talk Data Fellas introduces Shar3, a tool kit aiming to bridged the gaps to build a interactive distributed data processing pipeline, or loop! Then the talk covers genomics nowadays problems including data types, processing, discovery by introducing the GA4GH initiative and its implementation using Shar3.

Data Enthusiasts London: Scalable and Interoperable data services. Applied to...

Genomics and Health data is nowadays one of the hot topics requiring lots of computations and specially machine learning. This helps science with a very relevant societal impact to get even better outcome. That is why Apache Spark and its ADAM library is a must have. This talk will be twofold. First, we'll show how Apache Spark, MLlib and ADAM can be plugged all together to extract information from even huge and wide genomics dataset. Everything will be packed into examples from the Spark Notebook, showing how bio-scientists can work interactively with such a system. Second, we'll explain how these methodologies and even the datasets themselves can be shared at very large scale between remote entities like hospitals or laboratories using micro services leveraging Apache Spark, ADAM, Play Framework 2, Avro and Tachyon.

Spark meetup london share and analyse genomic data at scale with spark, adam...

Distributed machine learning 101 using apache spark from the browser

Liège créative: Open Science

BioBankCloud: Machine Learning on Genomics + GA4GH @ Med at Scale

What is Distributed Computing, Why we use Apache Spark

Spark devoxx2014

Machine Learning and GraphX

Quanti-litative Revolution in GIS

Scala and-fp-in-big-data