The Little Warehouse That Couldn't Or: How We Learned to Stop Worrying and Move to Spark-(Yandu Oppacher, Shopify)

•

4 likes•2,503 views

The document discusses the challenges faced by Shopify in using its existing data warehouse and ETL processes due to increasing data volume and complexity. It describes Shopify's attempts to use Pig and Luigi as well as Platfora to address these issues, but notes they did not meet Shopify's needs. Shopify then moved to using Spark due to its fast performance, nice development model using Python, and ability to better handle their data and query complexity. The summary provides an overview of why Shopify changed its data warehousing approach and the key technology it adopted.

Data & Analytics

The Little Warehouse That
Couldn’t Or: How We Learned to
Stop Worrying and Move to Spark
1
Yandu Oppacher (@yandu)
Data Infrastructure

2
Shopify Stores

ETL Warehouse Reporting
August 2013
TilllerRuby Vertica
3

Why we had to move
• Data volume
• Data/Query complexity
• Performance issues
4

Couple of false starts
5
Pig + Luigi
Pig + Oozie
Platfora

–platfora.com
“Without coding or ETL, data
warehousing, BI tools, or breaking a
sweat.”
6

Enter Spark
• Fast
• Nice development model
• Python
7

88
The Good Book

GMV
A Case Study
9
165,000+
ACTIVE SHOPIFY MERCHANTS
$8 BILLION+
CUMULATIVE GMV

Growing pains
• Joins
• Groupings
• General data skew
• Getting to know python’s performance quirks
10

Starscream
11
• specialized joins
• resolvers
• range
• cassandra
• overby
• contracts
• incrementalized fact
builds

Our current stack
12
Kafka
OLTP
HDFS
Cassandra
Spark
FrontroomBackroom
Redshift
Tableau

Thank you
13
Yandu Oppacher (@yandu)
Data Infrastructure

More Related Content

What's hot

Zeppelin at Twitter

Zeppelin at Twitter

Zeppelin at Twitter

ASPgems - kappa architecture

ASPgems - kappa architecture

ASPgems - kappa architecture

Juantomás García Molina

PixieDust is a new open source library that helps data scientists and developers working in Jupyter Notebooks and Apache Spark be more efficient. PixieDust speeds up data manipulation and display with features like: auto-visualization of Spark DataFrames, real-time Spark job progress monitoring, automated local install of Python and Scala kernels running with Spark, and much more. Come along and learn how you can use this tool in your own projects to visualize and explore data effortlessly with no coding. Oh, and if you prefer working with a Scala Notebook, this session is also for you, as PixieDust can also run on a Scala Kernel. Imagine being able to visualize your favorite Python chart engines from a Scala Notebook! We’ll finish the session with a demo combining Twitter, Watson Tone Analyzer, Spark Streaming, and some fun real-time visualizations–all running within a Notebook.

Taking Jupyter Notebooks and Apache Spark to the Next Level PixieDust with Da...

Taking Jupyter Notebooks and Apache Spark to the Next Level PixieDust with Da...

Taking Jupyter Notebooks and Apache Spark to the Next Level PixieDust with Da...

Disrupting Big Data with Apache Spark in the Cloud

Disrupting Big Data with Apache Spark in the Cloud

Disrupting Big Data with Apache Spark in the Cloud

We’re always told to ‘Go for the Gold!,’ but how do we get there? This talk will walk you through the process of moving your data to the finish fine to get that gold metal! A common data engineering pipeline architecture uses tables that correspond to different quality levels, progressively adding structure to the data: data ingestion (‘Bronze’ tables), transformation/feature engineering (‘Silver’ tables), and machine learning training or prediction (‘Gold’ tables). Combined, we refer to these tables as a ‘multi-hop’ architecture. It allows data engineers to build a pipeline that begins with raw data as a ‘single source of truth’ from which everything flows. In this session, we will show how to build a scalable data engineering data pipeline using Delta Lake, so you can be the champion in your organization.

Simplify and Scale Data Engineering Pipelines with Delta Lake

Simplify and Scale Data Engineering Pipelines with Delta Lake

Simplify and Scale Data Engineering Pipelines with Delta Lake

Apache Spark was designed as a batch analytics system. By caching RDDs, Spark speeds up jobs that iteratively process the same data. This pattern is also applicable to online analytics. We use Bloomberg’s Spark Server as a server runtime for online analytics. Our framework implements certain useful patterns applicable to online query processing and is centered on the idea of “Managed” DataFrames that can be refreshed and updated as per user requirements, without violating the immutability of RDDs/DataFrames. However, Spark presents significant challenges with respect to availability and resilience in an online setting where Spark is required to respond to queries with high SLAs. In this talk, we try to identify specific areas where slow-down or failures can result in the largest hits on online-query performance and potential solutions to address these.

Spark and Online Analytics: Spark Summit East talky by Shubham Chopra

Spark and Online Analytics: Spark Summit East talky by Shubham Chopra

Spark and Online Analytics: Spark Summit East talky by Shubham Chopra

How Apache Spark Is Helping Tame the Wild West of Wi-Fi

How Apache Spark Is Helping Tame the Wild West of Wi-Fi

How Apache Spark Is Helping Tame the Wild West of Wi-Fi

Spark Summit EU talk by Pat Patterson

Spark Summit EU talk by Pat Patterson

Spark Summit EU talk by Pat Patterson

Shifting Data Science into High Gear

Shifting Data Science into High Gear

Shifting Data Science into High Gear

Workday Prism Analytics enables data discovery and interactive Business Intelligence analysis for Workday customers. Workday is a “pure SaaS” company, providing a suite of Financial and HCM (Human Capital Management) apps to about 2000 companies around the world, including more than 30% from Fortune-500 list. There are significant business and technical challenges to support millions of concurrent users and hundreds of millions daily transactions. Using memory-centric graph-based architecture allowed to overcome most of these problems. As Workday grew, data transactions from existing and new customers generated vast amounts of valuable and highly sensitive data. The next big challenge was to provide in-app analytics platform, which for the multiple types of accumulated data, and also would allow using blend in external datasets. Workday users wanted it to be super-fast, but also intuitive and easy-to-use both for the financial and HR analysts and for regular, less technical users. Existing backend technologies were not a good fit, so we turned to Apache Spark. In this presentation, we will share the lessons we learned when building highly scalable multi-tenant analytics service for transactional data. We will start with the big picture and business requirements. Then describe the architecture with batch and interactive modules for data preparation, publishing, and query engine, noting the relevant Spark technologies. Then we will dive into the internals of Prism’s Query Engine, focusing on Spark SQL, DataFrames and Catalyst compiler features used. We will describe the issues we encountered while compiling and executing complex pipelines and queries, and how we use caching, sampling, and query compilation techniques to support interactive user experience. Finally, we will share the future challenges for 2018 and beyond.

Lightning-Fast Analytics for Workday Transactional Data with Pavel Hardak and...

Lightning-Fast Analytics for Workday Transactional Data with Pavel Hardak and...

Lightning-Fast Analytics for Workday Transactional Data with Pavel Hardak and...

Quby, an Amsterdam-based technology company, offers solutions to empower homeowners to stay in control of their electricity, gas and water usage. Using Europe’s largest energy dataset, consisting of petabytes of IoT data, the company has developed AI powered products that are used by hundreds of thousands of users on a daily basis. Delta Lake ensures the quality of incoming records though schema enforcement and evolution. But it is the Data Engineers role to check whether the expected data is ingested in to the Delta Lake at the right time with expected metrics so that downstream processes will function their duties. Re-training models and serving on the fly might go wrong unless we put the right monitoring infrastructure too.

Monitoring Half a Million ML Models, IoT Streaming Data, and Automated Qualit...

Monitoring Half a Million ML Models, IoT Streaming Data, and Automated Qualit...

Monitoring Half a Million ML Models, IoT Streaming Data, and Automated Qualit...

Apache Spark 2.0 introduced Structured Streaming which allows users to continually and incrementally update your view of the world as new data arrives while still using the same familiar Spark SQL abstractions. Michael Armbrust from Databricks talks about the progress made since the release of Spark 2.0 on robustness, latency, expressiveness and observability, using examples of production end-to-end continuous applications. Speaker: Michael Armbrust Video: http://go.databricks.com/videos/spark-summit-east-2017/using-structured-streaming-apache-spark This talk was originally presented at Spark Summit East 2017.

Insights Without Tradeoffs: Using Structured Streaming

Insights Without Tradeoffs: Using Structured Streaming

Insights Without Tradeoffs: Using Structured Streaming

Apple leverages Apache Spark for processing large datasets to power key components of Apple's production services. The majority of users rely on Spark SQL to benefit from state-of-the-art optimizations in Catalyst and Tungsten. As there are multiple APIs to interact with Spark SQL, users have to make a wise decision which one to pick. While DataFrames and SQL are widely used, they lack type safety so that the analysis errors will not be detected during the compile time such as invalid column names or types. Also, the ability to apply the same functional constructions as on RDDs is missing in DataFrames. Datasets expose a type-safe API and support for user-defined closures at the cost of performance. This talk will explain cases when Spark SQL cannot optimize typed Datasets as much as it can optimize DataFrames. We will also present an effort to use bytecode analysis to convert user-defined closures into native Catalyst expressions. This helps Spark to avoid the expensive conversion between the internal format and JVM objects as well as to leverage more Catalyst optimizations. A consequence, we can bridge the gap in performance between Datasets and DataFrames, so that users do not have to sacrifice the benefits of Datasets for performance reasons.

Bridging the Gap Between Datasets and DataFrames

Bridging the Gap Between Datasets and DataFrames

Bridging the Gap Between Datasets and DataFrames

Using Hadoop to build a Data Quality Service for both real-time and batch data

Using Hadoop to build a Data Quality Service for both real-time and batch data

Using Hadoop to build a Data Quality Service for both real-time and batch data

DataWorks Summit/Hadoop Summit

Spark SQL is a very effective distributed SQL engine for OLAP and widely adopted in Baidu production for many internal BI projects. However, Baidu has also been facing many challenges for large scale including tuning the shuffle parallelism for thousands of jobs, inefficient execution plan, and handling data skew. In this talk, we will explore Intel and Baidu’s joint efforts to address challenges in large scale and offer an overview of an adaptive execution mode we implemented for Baidu’s Big SQL platform which is based on Spark SQL. At runtime, adaptive execution can change the execution plan to use a better join strategy and handle skewed join automatically. It can also change the number of reducer to better fit the data scale. In general, adaptive execution decreases the effort involved in tuning SQL query parameters and improves the execution performance by choosing a better execution plan and parallelism at runtime. We’ll also share our experience of using adaptive execution in Baidu’s production cluster with thousands of server, where adaptive execution helps to improve the performance of some complex queries by 200%. After further analysis we found that several special scenarios in Baidu data analysis can benefit from the optimization of choosing better join type. We got 2x performance improvement in the scenario where the user wanted to analysis 1000+ advertisers’ cost from both web and mobile side and each side has a full information table with 10 TB parquet file per-day. Now we are writing probe jobs to detect more scenarios from current daily jobs of our users. We are also considering to expose the strategy interface based on the detailed metrics collected form adaptive execution mode for the upper users.

Spark SQL Adaptive Execution Unleashes The Power of Cluster in Large Scale wi...

Spark SQL Adaptive Execution Unleashes The Power of Cluster in Large Scale wi...

Spark SQL Adaptive Execution Unleashes The Power of Cluster in Large Scale wi...

Apache Spark in Scientific Applciations

Apache Spark in Scientific Applciations

Apache Spark in Scientific Applciations

Dr. Mirko Kämpf

Spark at Airbnb

Spark at Airbnb

Spark at Airbnb

At Netflix, we've spent a lot of time thinking about how we can make our analytics group move quickly. Netflix's Data Engineering & Analytics organization embraces the company's culture of "Freedom & Responsibility". How does a company with a $40 billion market cap and $6 billion in annual revenue keep their data teams moving with the agility of a tiny company? How do hundreds of data engineers and scientists make the best decisions for their projects independently, without the analytics environment devolving into chaos? We'll talk about how Netflix equips its business intelligence and data engineers with: the freedom to leverage cloud-based data tools - Spark, Presto, Redshift, Tableau and others - in ways that solve our most difficult data problems the freedom to find and introduce right software for the job - even if it isn't used anywhere else in-house the freedom to create and drop new tables in production without approval the freedom to choose when a question is a one-off, and when a question is asked often enough to require a self-service tool the freedom to retire analytics and data processes whose value doesn't justify their support costs Speaker Bios Monisha Kanoth is a Senior Data Architect at Netflix, and was one of the founding members of the current streaming Content Analytics team. She previously worked as a big data lead at Convertro (acquired by AOL) and as a data warehouse lead at MySpace. Jason Flittner is a Senior Business Intelligence Engineer at Netflix, focusing on data transformation, analysis, and visualization as part of the Content Data Engineering & Analytics team. He previously led the EC2 Business Intelligence team at Amazon Web Services and was a business intelligence engineer with Cisco. Chris Stephens is a Senior Data Engineer at Netflix. He previously served as the CTO at Deep 6 Analytics, a machine learning & content analytics company in Los Angeles, and on the data warehouse teams at the FOX Audience Network and Anheuser-Busch.

Rapid Data Analytics @ Netflix

Rapid Data Analytics @ Netflix

Rapid Data Analytics @ Netflix

"Most data practitioners grapple with data quality issues and data pipeline complexities—it's the bane of their existence. Data engineers, in particular, strive to design and deploy robust data pipelines that serve reliable data in a performant manner so that their organizations can make the most of their valuable corporate data assets. Databricks Delta, part of Databricks Runtime, is a next-generation unified analytics engine built on top of Apache Spark. Built on open standards, Delta employs co-designed compute and storage and is compatible with Spark API’s. It powers high data reliability and query performance to support big data use cases, from batch and streaming ingests, fast interactive queries to machine learning. In this tutorial we will discuss the requirements of modern data pipelines, the challenges data engineers face when it comes to data reliability and performance and how Delta can help. Through presentation, code examples and notebooks, we will explain pipeline challenges and the use of Delta to address them. You will walk away with an understanding of how you can apply this innovation to your data architecture and the benefits you can gain. This tutorial will be both instructor-led and hands-on interactive session. Instructions in how to get tutorial materials will be covered in class. WHAT YOU’LL LEARN: – Understand the key data reliability and performance data pipelines challenges – How Databricks Delta helps build robust pipelines at scale – Understand how Delta fits within an Apache Spark™ environment – How to use Delta to realize data reliability improvements – How to deliver performance gains using Delta PREREQUISITES: – A fully-charged laptop (8-16GB memory) with Chrome or Firefox – Pre-register for Databricks Community Edition" Speakers: Steven Yu, Burak Yavuz

Building Robust Production Data Pipelines with Databricks Delta

Building Robust Production Data Pipelines with Databricks Delta

Building Robust Production Data Pipelines with Databricks Delta

Realtime streaming architecture in INFINARIO

Realtime streaming architecture in INFINARIO

Realtime streaming architecture in INFINARIO

What's hot (20)

Zeppelin at Twitter

Zeppelin at Twitter

Zeppelin at Twitter

ASPgems - kappa architecture

ASPgems - kappa architecture

ASPgems - kappa architecture

Taking Jupyter Notebooks and Apache Spark to the Next Level PixieDust with Da...

Taking Jupyter Notebooks and Apache Spark to the Next Level PixieDust with Da...

Taking Jupyter Notebooks and Apache Spark to the Next Level PixieDust with Da...

Disrupting Big Data with Apache Spark in the Cloud

Disrupting Big Data with Apache Spark in the Cloud

Disrupting Big Data with Apache Spark in the Cloud

Simplify and Scale Data Engineering Pipelines with Delta Lake

Simplify and Scale Data Engineering Pipelines with Delta Lake

Simplify and Scale Data Engineering Pipelines with Delta Lake

Spark and Online Analytics: Spark Summit East talky by Shubham Chopra

Spark and Online Analytics: Spark Summit East talky by Shubham Chopra

Spark and Online Analytics: Spark Summit East talky by Shubham Chopra

How Apache Spark Is Helping Tame the Wild West of Wi-Fi

How Apache Spark Is Helping Tame the Wild West of Wi-Fi

How Apache Spark Is Helping Tame the Wild West of Wi-Fi

Spark Summit EU talk by Pat Patterson

Spark Summit EU talk by Pat Patterson

Spark Summit EU talk by Pat Patterson

Shifting Data Science into High Gear

Shifting Data Science into High Gear

Shifting Data Science into High Gear

Lightning-Fast Analytics for Workday Transactional Data with Pavel Hardak and...

Lightning-Fast Analytics for Workday Transactional Data with Pavel Hardak and...

Lightning-Fast Analytics for Workday Transactional Data with Pavel Hardak and...

Monitoring Half a Million ML Models, IoT Streaming Data, and Automated Qualit...

Monitoring Half a Million ML Models, IoT Streaming Data, and Automated Qualit...

Monitoring Half a Million ML Models, IoT Streaming Data, and Automated Qualit...

Insights Without Tradeoffs: Using Structured Streaming

Insights Without Tradeoffs: Using Structured Streaming

Insights Without Tradeoffs: Using Structured Streaming

Bridging the Gap Between Datasets and DataFrames

Bridging the Gap Between Datasets and DataFrames

Bridging the Gap Between Datasets and DataFrames

Using Hadoop to build a Data Quality Service for both real-time and batch data

Using Hadoop to build a Data Quality Service for both real-time and batch data

Using Hadoop to build a Data Quality Service for both real-time and batch data

Spark SQL Adaptive Execution Unleashes The Power of Cluster in Large Scale wi...

Spark SQL Adaptive Execution Unleashes The Power of Cluster in Large Scale wi...

Spark SQL Adaptive Execution Unleashes The Power of Cluster in Large Scale wi...

Apache Spark in Scientific Applciations

Apache Spark in Scientific Applciations

Apache Spark in Scientific Applciations

Spark at Airbnb

Spark at Airbnb

Spark at Airbnb

Rapid Data Analytics @ Netflix

Rapid Data Analytics @ Netflix

Rapid Data Analytics @ Netflix

Building Robust Production Data Pipelines with Databricks Delta

Building Robust Production Data Pipelines with Databricks Delta

Building Robust Production Data Pipelines with Databricks Delta

Realtime streaming architecture in INFINARIO

Realtime streaming architecture in INFINARIO

Realtime streaming architecture in INFINARIO

Viewers also liked

Open Stack Cheat Sheet V1

Open Stack Cheat Sheet V1

Open Stack Cheat Sheet V1

Anuchit Chalothorn

Tachyon-2014-11-21-amp-camp5

Tachyon-2014-11-21-amp-camp5

Tachyon-2014-11-21-amp-camp5

Linux Filesystems, RAID, and more

Linux Filesystems, RAID, and more

Linux Filesystems, RAID, and more

Lessons Learned with Spark at the US Patent & Trademark Office-(Christopher B...

Lessons Learned with Spark at the US Patent & Trademark Office-(Christopher B...

Lessons Learned with Spark at the US Patent & Trademark Office-(Christopher B...

In an environment where cloud-scaling applications is becoming more and more important, client-server architectures paradigms, as shown by memcached, are back with vengeance. In this talk, Galder will talk about Hot Rod, Infinispan's new client/server binary protocol, explaining the key differences compared to memcached's binary protocol, such as the possibility of receiving cluster topology changes. Audience of this talk will learn of the importance of Hot Rod in 'cloud-scale' application server clustering, where stateless application server instances could use Infinispan Hot Rod clients to retrieve state from an elastic farm of Infinispan Hot Rod servers, improving capabilities to run application server instances as a PaaS. The talk will finish with a brief demo of a cluster of Infinispan Hot Rod servers running on EC2 being accessed from a non-Java client. The audience is expected to have an intermediate understanding of client-server software architectures and cloud deployments.

The Hot Rod Protocol in Infinispan

The Hot Rod Protocol in Infinispan

The Hot Rod Protocol in Infinispan

Galder Zamarreño

Accelerating Cassandra Workloads on Ceph with All-Flash PCIE SSDS

Accelerating Cassandra Workloads on Ceph with All-Flash PCIE SSDS

Accelerating Cassandra Workloads on Ceph with All-Flash PCIE SSDS

Lightning talk from the OpenStack NYC meetup on October 8, 2014. http://bit.ly/ibm-os-meetup By Gil Vernik The integration between Apache Spark and Swift, and the use of Storlets for smart retrieval via filtering and privacy-support. The content of this talk is a statement from the IBM Research division, not IBM product divisions, and is not a statement from IBM regarding its plans, directions or product intents. Any activities described by this talk are subject to change.

Advanced Data Retrieval and Analytics with Apache Spark and Openstack Swift

Advanced Data Retrieval and Analytics with Apache Spark and Openstack Swift

Advanced Data Retrieval and Analytics with Apache Spark and Openstack Swift

Scaling up genomic analysis with ADAM

Scaling up genomic analysis with ADAM

Scaling up genomic analysis with ADAM

ELC-E 2010: The Right Approach to Minimal Boot Times

ELC-E 2010: The Right Approach to Minimal Boot Times

ELC-E 2010: The Right Approach to Minimal Boot Times

andrewmurraympc

Velox: Models in Action

Velox: Models in Action

Velox: Models in Action

Naïveté vs. Experience

Naïveté vs. Experience

Naïveté vs. Experience

SparkR: Enabling Interactive Data Science at Scale

SparkR: Enabling Interactive Data Science at Scale

SparkR: Enabling Interactive Data Science at Scale

SampleClean: Bringing Data Cleaning into the BDAS Stack

SampleClean: Bringing Data Cleaning into the BDAS Stack

SampleClean: Bringing Data Cleaning into the BDAS Stack

OpenStack Cheat Sheet V2

OpenStack Cheat Sheet V2

OpenStack Cheat Sheet V2

Anuchit Chalothorn

A Curious Course on Coroutines and Concurrency

A Curious Course on Coroutines and Concurrency

A Curious Course on Coroutines and Concurrency

David Beazley (Dabeaz LLC)

Lab 5: Interconnecting a Datacenter using Mininet

Lab 5: Interconnecting a Datacenter using Mininet

Lab 5: Interconnecting a Datacenter using Mininet

Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)

Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)

Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)

Join this webinar to discuss best practices for designing and building a solid, robust and flexible Hadoop platform on an enterprise virtual infrastructure. Attendees will learn the flexibility and operational advantages of Virtual Machines such as fast provisioning, cloning, high levels of standardization, hybrid storage, vMotioning, increased stabilization of the entire software stack, High Availability and Fault Tolerance. This is a can`t miss presentation for anyone wanting to understand design, configuration and deployment of Hadoop in virtual infrastructures.

Best Practices for Virtualizing Apache Hadoop

Best Practices for Virtualizing Apache Hadoop

Best Practices for Virtualizing Apache Hadoop

In Search of the Perfect Global Interpreter Lock

In Search of the Perfect Global Interpreter Lock

In Search of the Perfect Global Interpreter Lock

David Beazley (Dabeaz LLC)

Python in Action (Part 2)

Python in Action (Part 2)

Python in Action (Part 2)

David Beazley (Dabeaz LLC)

Viewers also liked (20)

Open Stack Cheat Sheet V1

Open Stack Cheat Sheet V1

Open Stack Cheat Sheet V1

Tachyon-2014-11-21-amp-camp5

Tachyon-2014-11-21-amp-camp5

Tachyon-2014-11-21-amp-camp5

Linux Filesystems, RAID, and more

Linux Filesystems, RAID, and more

Linux Filesystems, RAID, and more

Lessons Learned with Spark at the US Patent & Trademark Office-(Christopher B...

Lessons Learned with Spark at the US Patent & Trademark Office-(Christopher B...

Lessons Learned with Spark at the US Patent & Trademark Office-(Christopher B...

The Hot Rod Protocol in Infinispan

The Hot Rod Protocol in Infinispan

The Hot Rod Protocol in Infinispan

Accelerating Cassandra Workloads on Ceph with All-Flash PCIE SSDS

Accelerating Cassandra Workloads on Ceph with All-Flash PCIE SSDS

Accelerating Cassandra Workloads on Ceph with All-Flash PCIE SSDS

Advanced Data Retrieval and Analytics with Apache Spark and Openstack Swift

Advanced Data Retrieval and Analytics with Apache Spark and Openstack Swift

Advanced Data Retrieval and Analytics with Apache Spark and Openstack Swift

Scaling up genomic analysis with ADAM

Scaling up genomic analysis with ADAM

Scaling up genomic analysis with ADAM

ELC-E 2010: The Right Approach to Minimal Boot Times

ELC-E 2010: The Right Approach to Minimal Boot Times

ELC-E 2010: The Right Approach to Minimal Boot Times

Velox: Models in Action

Velox: Models in Action

Velox: Models in Action

Naïveté vs. Experience

Naïveté vs. Experience

Naïveté vs. Experience

SparkR: Enabling Interactive Data Science at Scale

SparkR: Enabling Interactive Data Science at Scale

SparkR: Enabling Interactive Data Science at Scale

SampleClean: Bringing Data Cleaning into the BDAS Stack

SampleClean: Bringing Data Cleaning into the BDAS Stack

SampleClean: Bringing Data Cleaning into the BDAS Stack

OpenStack Cheat Sheet V2

OpenStack Cheat Sheet V2

OpenStack Cheat Sheet V2

A Curious Course on Coroutines and Concurrency

A Curious Course on Coroutines and Concurrency

A Curious Course on Coroutines and Concurrency

Lab 5: Interconnecting a Datacenter using Mininet

Lab 5: Interconnecting a Datacenter using Mininet

Lab 5: Interconnecting a Datacenter using Mininet

Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)

Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)

Spark on Mesos-A Deep Dive-(Dean Wampler and Tim Chen, Typesafe and Mesosphere)

Best Practices for Virtualizing Apache Hadoop

Best Practices for Virtualizing Apache Hadoop

Best Practices for Virtualizing Apache Hadoop

In Search of the Perfect Global Interpreter Lock

In Search of the Perfect Global Interpreter Lock

In Search of the Perfect Global Interpreter Lock

Python in Action (Part 2)

Python in Action (Part 2)

Python in Action (Part 2)

Similar to The Little Warehouse That Couldn't Or: How We Learned to Stop Worrying and Move to Spark-(Yandu Oppacher, Shopify)

Pig on Spark

5 Things that Make Hadoop a Game Changer Webinar by Elliott Cordo, Caserta Concepts There is much hype and mystery surrounding Hadoop's role in analytic architecture. In this webinar, Elliott presented, in detail, the services and concepts that makes Hadoop a truly unique solution - a game changer for the enterprise. He talked about the real benefits of a distributed file system, the multi workload processing capabilities enabled by YARN, and the 3 other important things you need to know about Hadoop. To access the recorded webinar, visit the event site: https://www.brighttalk.com/webcast/9061/131029 For more information the services and solutions that Caserta Concepts offers, please visit http://casertaconcepts.com/

5 Things that Make Hadoop a Game Changer

5 Things that Make Hadoop a Game Changer

5 Things that Make Hadoop a Game Changer

Spark Summit EU 2015: Revolutionizing Big Data in the Enterprise with Spark

Spark Summit EU 2015: Revolutionizing Big Data in the Enterprise with Spark

Spark Summit EU 2015: Revolutionizing Big Data in the Enterprise with Spark

100 Exadata Implementations Later-Tim Fox

100 Exadata Implementations Later-Tim Fox

100 Exadata Implementations Later-Tim Fox

The Briefing Room with Dr. Robin Bloor and Splice Machine Live Webcast August 11, 2015 Watch the archive: https://bloorgroup.webex.com/bloorgroup/onstage/g.php?MTID=e1b33c9d45b178e13784b4a971a4c1349 The ETL process was born out of necessity, and for decades it has been the glue between data sources and target applications. But as data growth soars and increased competition demands real-time data, standard ETL has become brittle and often unmanageable. Scaling up resources can do the trick, but it’s very costly and only a matter of time before the processes hit another bottleneck. When outmoded ETL stands in the way of real-time analytics, it might be time to consider a completely new approach. Register for this episode of The Briefing Room to learn from veteran Analyst Dr. Robin Bloor as he explains how modern, data-driven architectures must adopt an equally capable data integration strategy. He’ll be briefed by Rich Reimer of Splice Machine, who will discuss how his company solves ETL performance issues and enables real-time analytics and reports on big data. He will show that by leveraging the scale-out power of Hadoop and the in-memory speed of Spark, users can bring both analytical and operational systems together, eventually performing transformations only when needed. Visit InsideAnalysis.com for more information.

Goodbye, Bottlenecks: How Scale-Out and In-Memory Solve ETL

Goodbye, Bottlenecks: How Scale-Out and In-Memory Solve ETL

Goodbye, Bottlenecks: How Scale-Out and In-Memory Solve ETL

Inside Analysis

PyData: The Next Generation | Data Day Texas 2015

PyData: The Next Generation | Data Day Texas 2015

PyData: The Next Generation | Data Day Texas 2015

http://www.bigdataspain.org/2014/conference/state-of-play-data-science-on-hadoop-in-2015-keynote Machine Learning is not new. Big Machine Learning is qualitatively different: More data beats algorithm improvement, scale trumps noise and sample size effects, can brute-force manual tasks. Session presented at Big Data Spain 2014 Conference 18th Nov 2014 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Slides: https://speakerdeck.com/bigdataspain/state-of-play-data-science-on-hadoop-in-2015-by-sean-owen-at-big-data-spain-2014

State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...

State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...

State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...

Teradata Partners Conference Oct 2014 Big Data Anti-Patterns

Teradata Partners Conference Oct 2014 Big Data Anti-Patterns

Teradata Partners Conference Oct 2014 Big Data Anti-Patterns

Hadoop and SAP BI

Hadoop and SAP BI

Hadoop and SAP BI

Praveen Kumar (Tyagi)

Michael Stonebraker: Big Data, Disruption, and the 800 Pound Gorilla in the ...

Michael Stonebraker: Big Data, Disruption, and the 800 Pound Gorilla in the ...

Michael Stonebraker: Big Data, Disruption, and the 800 Pound Gorilla in the ...

Splice machine-bloor-webinar-data-lakes

Splice machine-bloor-webinar-data-lakes

Splice machine-bloor-webinar-data-lakes

Edgar Alejandro Villegas

Cloud-native Semantic Layer on Data Lake

Cloud-native Semantic Layer on Data Lake

Cloud-native Semantic Layer on Data Lake

Stream based Data Integration

Stream based Data Integration

Stream based Data Integration

Jeffrey T. Pollock

SnappyData Toronto Meetup Nov 2017

SnappyData Toronto Meetup Nov 2017

SnappyData Toronto Meetup Nov 2017

Unlocked: the Hybrid Cloud - 12th May 2014 / All Slides (morning)

Unlocked: the Hybrid Cloud - 12th May 2014 / All Slides (morning)

Unlocked: the Hybrid Cloud - 12th May 2014 / All Slides (morning)

Rackspace Academy

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

Big Data, Big Dream

Big Data, Big Dream

Big Data, Big Dream

Apache Tajo - BWC 2014

Apache Tajo - BWC 2014

Apache Tajo - BWC 2014

Spark Summit EU talk by Bas Geerdink

Spark Summit EU talk by Bas Geerdink

Spark Summit EU talk by Bas Geerdink

Distributed Data Quality - Technical Solutions for Organizational Scaling

Distributed Data Quality - Technical Solutions for Organizational Scaling

Distributed Data Quality - Technical Solutions for Organizational Scaling

Justin Cunningham

Similar to The Little Warehouse That Couldn't Or: How We Learned to Stop Worrying and Move to Spark-(Yandu Oppacher, Shopify) (20)

Pig on Spark

5 Things that Make Hadoop a Game Changer

5 Things that Make Hadoop a Game Changer

5 Things that Make Hadoop a Game Changer

Spark Summit EU 2015: Revolutionizing Big Data in the Enterprise with Spark

Spark Summit EU 2015: Revolutionizing Big Data in the Enterprise with Spark

Spark Summit EU 2015: Revolutionizing Big Data in the Enterprise with Spark

100 Exadata Implementations Later-Tim Fox

100 Exadata Implementations Later-Tim Fox

100 Exadata Implementations Later-Tim Fox

Goodbye, Bottlenecks: How Scale-Out and In-Memory Solve ETL

Goodbye, Bottlenecks: How Scale-Out and In-Memory Solve ETL

Goodbye, Bottlenecks: How Scale-Out and In-Memory Solve ETL

PyData: The Next Generation | Data Day Texas 2015

PyData: The Next Generation | Data Day Texas 2015

PyData: The Next Generation | Data Day Texas 2015

State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...

State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...

State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...

Teradata Partners Conference Oct 2014 Big Data Anti-Patterns

Teradata Partners Conference Oct 2014 Big Data Anti-Patterns

Teradata Partners Conference Oct 2014 Big Data Anti-Patterns

Hadoop and SAP BI

Hadoop and SAP BI

Hadoop and SAP BI

Michael Stonebraker: Big Data, Disruption, and the 800 Pound Gorilla in the ...

Michael Stonebraker: Big Data, Disruption, and the 800 Pound Gorilla in the ...

Michael Stonebraker: Big Data, Disruption, and the 800 Pound Gorilla in the ...

Splice machine-bloor-webinar-data-lakes

Splice machine-bloor-webinar-data-lakes

Splice machine-bloor-webinar-data-lakes

Cloud-native Semantic Layer on Data Lake

Cloud-native Semantic Layer on Data Lake

Cloud-native Semantic Layer on Data Lake

Stream based Data Integration

Stream based Data Integration

Stream based Data Integration

SnappyData Toronto Meetup Nov 2017

SnappyData Toronto Meetup Nov 2017

SnappyData Toronto Meetup Nov 2017

Unlocked: the Hybrid Cloud - 12th May 2014 / All Slides (morning)

Unlocked: the Hybrid Cloud - 12th May 2014 / All Slides (morning)

Unlocked: the Hybrid Cloud - 12th May 2014 / All Slides (morning)

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

Big Data, Big Dream

Big Data, Big Dream

Big Data, Big Dream

Apache Tajo - BWC 2014

Apache Tajo - BWC 2014

Apache Tajo - BWC 2014

Spark Summit EU talk by Bas Geerdink

Spark Summit EU talk by Bas Geerdink

Spark Summit EU talk by Bas Geerdink

Distributed Data Quality - Technical Solutions for Organizational Scaling

Distributed Data Quality - Technical Solutions for Organizational Scaling

Distributed Data Quality - Technical Solutions for Organizational Scaling

More from Spark Summit

In this session we will present a Configurable FPGA-Based Spark SQL Acceleration Architecture. It is target to leverage FPGA highly parallel computing capability to accelerate Spark SQL Query and for FPGA’s higher power efficiency than CPU we can lower the power consumption at the same time. The Architecture consists of SQL query decomposition algorithms, fine-grained FPGA based Engine Units which perform basic computation of sub string, arithmetic and logic operations. Using SQL query decomposition algorithm, we are able to decompose a complex SQL query into basic operations and according to their patterns each is fed into an Engine Unit. SQL Engine Units are highly configurable and can be chained together to perform complex Spark SQL queries, finally one SQL query is transformed into a Hardware Pipeline. We will present the performance benchmark results comparing the queries with FGPA-Based Spark SQL Acceleration Architecture on XEON E5 and FPGA to the ones with Spark SQL Query on XEON E5 with 10X ~ 100X improvement and we will demonstrate one SQL query workload from a real customer.

FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang

FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang

FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang

In this talk, we’ll present techniques for visualizing large scale machine learning systems in Spark. These are techniques that are employed by Netflix to understand and refine the machine learning models behind Netflix’s famous recommender systems that are used to personalize the Netflix experience for their 99 millions members around the world. Essential to these techniques is Vegas, a new OSS Scala library that aims to be the “missing MatPlotLib” for Spark/Scala. We’ll talk about the design of Vegas and its usage in Scala notebooks to visualize Machine Learning Models.

VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...

VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...

VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...

This presentation introduces how we design and implement a real-time processing platform using latest Spark Structured Streaming framework to intelligently transform the production lines in the manufacturing industry. In the traditional production line there are a variety of isolated structured, semi-structured and unstructured data, such as sensor data, machine screen output, log output, database records etc. There are two main data scenarios: 1) Picture and video data with low frequency but a large amount; 2) Continuous data with high frequency. They are not a large amount of data per unit. However the total amount of them is very large, such as vibration data used to detect the quality of the equipment. These data have the characteristics of streaming data: real-time, volatile, burst, disorder and infinity. Making effective real-time decisions to retrieve values from these data is critical to smart manufacturing. The latest Spark Structured Streaming framework greatly lowers the bar for building highly scalable and fault-tolerant streaming applications. Thanks to the Spark we are able to build a low-latency, high-throughput and reliable operation system involving data acquisition, transmission, analysis and storage. The actual user case proved that the system meets the needs of real-time decision-making. The system greatly enhance the production process of predictive fault repair and production line material tracking efficiency, and can reduce about half of the labor force for the production lines.

Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu

Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu

Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu

As common sense would suggest, weather has a definite impact on traffic. But how much? And under what circumstances? Can we improve traffic (congestion) prediction given weather data? Predictive traffic is envisioned to significantly impact how driver’s plan their day by alerting users before they travel, find the best times to travel, and over time, learn from new IoT data such as road conditions, incidents, etc. This talk will cover the traffic prediction work conducted jointly by IBM and the traffic data provider. As a part of this work, we conducted a case study over five large metropolitans in the US, 2.58 billion traffic records and 262 million weather records, to quantify the boost in accuracy of traffic prediction using weather data. We will provide an overview of our lambda architecture with Apache Spark being used to build prediction models with weather and traffic data, and Spark Streaming used to score the model and provide real-time traffic predictions. This talk will also cover a suite of extensions to Spark to analyze geospatial and temporal patterns in traffic and weather data, as well as the suite of machine learning algorithms that were used with Spark framework. Initial results of this work were presented at the National Association of Broadcasters meeting in Las Vegas in April 2017, and there is work to scale the system to provide predictions in over a 100 cities. Audience will learn about our experience scaling using Spark in offline and streaming mode, building statistical and deep-learning pipelines with Spark, and techniques to work with geospatial and time-series data.

Improving Traffic Prediction Using Weather Data with Ramya Raghavendra

Improving Traffic Prediction Using Weather Data with Ramya Raghavendra

Improving Traffic Prediction Using Weather Data with Ramya Raghavendra

Graph is on the rise and it’s time to start learning about scalable graph analytics! In this session we will go over two Spark-based Graph Analytics frameworks: Tinkerpop and GraphFrames. While both frameworks can express very similar traversals, they have different performance characteristics and APIs. In this Deep-Dive by example presentation, we will demonstrate some common traversals and explain how, at a Spark level, each traversal is actually computed under the hood! Learn both the fluent Gremlin API as well as the powerful GraphFrame Motif api as we show examples of both simultaneously. No need to be familiar with Graphs or Spark for this presentation as we’ll be explaining everything from the ground up!

A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...

A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...

A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...

Building accurate machine learning models has been an art of data scientists, i.e., algorithm selection, hyper parameter tuning, feature selection and so on. Recently, challenges to breakthrough this “black-arts” have got started. In cooperation with our partner, NEC Laboratories America, we have developed a Spark-based automatic predictive modeling system. The system automatically searches the best algorithm, parameters and features without any manual work. In this talk, we will share how the automation system is designed to exploit attractive advantages of Spark. The evaluation with real open data demonstrates that our system can explore hundreds of predictive models and discovers the most accurate ones in minutes on a Ultra High Density Server, which employs 272 CPU cores, 2TB memory and 17TB SSD in 3U chassis. We will also share open challenges to learn such a massive amount of models on Spark, particularly from reliability and stability standpoints. This talk will cover the presentation already shown on Spark Summit SF’17 (#SFds5) but from more technical perspective.

No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...

No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...

No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...

In Sweden, from the Rise ICE Data Center at www.hops.site, we are providing to reseachers both Spark-as-a-Service and, more recently, Tensorflow-as-a-Service as part of the Hops platform. In this talk, we examine the different ways in which Tensorflow can be included in Spark workflows, from batch to streaming to structured streaming applications. We will analyse the different frameworks for integrating Spark with Tensorflow, from Tensorframes to TensorflowOnSpark to Databrick’s Deep Learning Pipelines. We introduce the different programming models supported and highlight the importance of cluster support for managing different versions of python libraries on behalf of users. We will also present cluster management support for sharing GPUs, including Mesos and YARN (in Hops Hadoop). Finally, we will perform a live demonstration of training and inference for a TensorflowOnSpark application written on Jupyter that can read data from either HDFS or Kafka, transform the data in Spark, and train a deep neural network on Tensorflow. We will show how to debug the application using both Spark UI and Tensorboard, and how to examine logs and monitor training.

Apache Spark and Tensorflow as a Service with Jim Dowling

Apache Spark and Tensorflow as a Service with Jim Dowling

Apache Spark and Tensorflow as a Service with Jim Dowling

In Sweden, from the Rise ICE Data Center at www.hops.site, we are providing to reseachers both Spark-as-a-Service and, more recently, Tensorflow-as-a-Service as part of the Hops platform. In this talk, we examine the different ways in which Tensorflow can be included in Spark workflows, from batch to streaming to structured streaming applications. We will analyse the different frameworks for integrating Spark with Tensorflow, from Tensorframes to TensorflowOnSpark to Databrick’s Deep Learning Pipelines. We introduce the different programming models supported and highlight the importance of cluster support for managing different versions of python libraries on behalf of users. We will also present cluster management support for sharing GPUs, including Mesos and YARN (in Hops Hadoop). Finally, we will perform a live demonstration of training and inference for a TensorflowOnSpark application written on Jupyter that can read data from either HDFS or Kafka, transform the data in Spark, and train a deep neural network on Tensorflow. We will show how to debug the application using both Spark UI and Tensorboard, and how to examine logs and monitor training.

Apache Spark and Tensorflow as a Service with Jim Dowling

Apache Spark and Tensorflow as a Service with Jim Dowling

Apache Spark and Tensorflow as a Service with Jim Dowling

With the rapid growth of available datasets, it is imperative to have good tools for extracting insight from big data. The Spark ML library has excellent support for performing at-scale data processing and machine learning experiments, but more often than not, Data Scientists find themselves struggling with issues such as: low level data manipulation, lack of support for image processing, text analytics and deep learning, as well as the inability to use Spark alongside other popular machine learning libraries. To address these pain points, Microsoft recently released The Microsoft Machine Learning Library for Apache Spark (MMLSpark), an open-source machine learning library built on top of SparkML that seeks to simplify the data science process and integrate SparkML Pipelines with deep learning and computer vision libraries such as the Microsoft Cognitive Toolkit (CNTK) and OpenCV. With MMLSpark, Data Scientists can build models with 1/10th of the code through Pipeline objects that compose seamlessly with other parts of the SparkML ecosystem. In this session, we explore some of the main lessons learned from building MMLSpark. Join us if you would like to know how to extend Pipelines to ensure seamless integration with SparkML, how to auto-generate Python and R wrappers from Scala Transformers and Estimators, how to integrate and use previously non-distributed libraries in a distributed manner and how to efficiently deploy a Spark library across multiple platforms.

MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...

MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...

MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...

The Next Accelerator Logging Service (NXCALS) is a new Big Data project at CERN aiming to replace the existing Oracle-based service. The main purpose of the system is to store and present Controls/Infrastructure related data gathered from thousands of devices in the whole accelerator complex. The data is used to operate the machines, improve their performance and conduct studies for new beam types or future experiments. During this talk, Jakub will speak about NXCALS requirements and design choices that lead to the selected architecture based on Hadoop and Spark. He will present the Ingestion API, the abstractions behind the Meta-data Service and the Spark-based Extraction API where simple changes to the schema handling greatly improved the overall usability of the system. The system itself is not CERN specific and can be of interest to other companies or institutes confronted with similar Big Data problems.

Next CERN Accelerator Logging Service with Jakub Wozniak

Next CERN Accelerator Logging Service with Jakub Wozniak

Next CERN Accelerator Logging Service with Jakub Wozniak

In Between (A mobile App for couples, downloaded 20M in Global), from daily batch for extracting metrics, analysis and dashboard. Spark is widely used by engineers and data analysts in Between, thanks to the performance and expendability of Spark, data operating has become extremely efficient. Entire team including Biz Dev, Global Operation, Designers are enjoying data results so Spark is empowering entire company for data driven operation and thinking. Kevin, Co-founder and Data Team leader of Between will be presenting how things are going in Between. Listeners will know how small and agile team is living with data (how we build organization, culture and technical base) after this presentation.

Powering a Startup with Apache Spark with Kevin Kim

Powering a Startup with Apache Spark with Kevin Kim

Powering a Startup with Apache Spark with Kevin Kim

As common sense would suggest, weather has a definite impact on traffic. But how much? And under what circumstances? Can we improve traffic (congestion) prediction given weather data? Predictive traffic is envisioned to significantly impact how driver’s plan their day by alerting users before they travel, find the best times to travel, and over time, learn from new IoT data such as road conditions, incidents, etc. This talk will cover the traffic prediction work conducted jointly by IBM and the traffic data provider. As a part of this work, we conducted a case study over five large metropolitans in the US, 2.58 billion traffic records and 262 million weather records, to quantify the boost in accuracy of traffic prediction using weather data. We will provide an overview of our lambda architecture with Apache Spark being used to build prediction models with weather and traffic data, and Spark Streaming used to score the model and provide real-time traffic predictions. This talk will also cover a suite of extensions to Spark to analyze geospatial and temporal patterns in traffic and weather data, as well as the suite of machine learning algorithms that were used with Spark framework. Initial results of this work were presented at the National Association of Broadcasters meeting in Las Vegas in April 2017, and there is work to scale the system to provide predictions in over a 100 cities. Audience will learn about our experience scaling using Spark in offline and streaming mode, building statistical and deep-learning pipelines with Spark, and techniques to work with geospatial and time-series data.

Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra

Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra

Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra

In many cases, Big Data becomes just another buzzword because of the lack of tools that can support both the technological requirements for developing and deploying of the projects and/or the fluency of communication between the different profiles of people involved in the projects. In this talk, we will present Moriarty, a set of tools for fast prototyping of Big Data applications that can be deployed in an Apache Spark environment. These tools support the creation of Big Data workflows using the already existing functional blocks or supporting the creation of new functional blocks. The created workflow can then be deployed in a Spark infrastructure and used through a REST API. For better understanding of Moriarty, the prototyping process and the way it hides the Spark environment to the Big Data users and developers, we will present it together with a couple of examples based on a Industry 4.0 success cases and other on a logistic success case.

Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...

Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...

Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...

Large-scale testing of new data products or enhancements to existing products in a research and development environment can be a technical challenge for data scientists. In some cases, tools available to data scientists lack production-level capacity, whereas other tools do not provide the algorithms needed to run the methodology. At Nielsen, the Databricks platform provided a solution to both of these challenges. This breakout session will cover a specific Nielsen business case where two methodology enhancements were developed and tested at large-scale using the Databricks platform. Development and large-scale testing of these enhancements would not have been possible using standard database tools.

How Nielsen Utilized Databricks for Large-Scale Research and Development with...

How Nielsen Utilized Databricks for Large-Scale Research and Development with...

How Nielsen Utilized Databricks for Large-Scale Research and Development with...

Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...

Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...

Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...

Since the invention of SQL and relational databases, data production has been about specifying how data is transformed through queries. While Apache Spark can certainly be used as a general distributed query engine, the power and granularity of Spark’s APIs enables a revolutionary increase in data engineering productivity: goal-based data production. Goal-based data production concerns itself with specifying WHAT the desired result is, leaving the details of HOW the result is achieved to a smart data warehouse running on top of Spark. That not only substantially increases productivity, but also significantly expands the audience that can work directly with Spark: from developers and data scientists to technical business users. With specific data and architecture patterns spanning the range from ETL to machine learning data prep and with live demos, this session will demonstrate how Spark users can gain the benefits of goal-based data production.

Goal Based Data Production with Sim Simeonov

Goal Based Data Production with Sim Simeonov

Goal Based Data Production with Sim Simeonov

Have you imagined a simple machine learning solution able to prevent revenue leakage and monitor your distributed application? To answer this question, we offer a practical and a simple machine learning solution to create an intelligent monitoring application based on simple data analysis using Apache Spark MLlib. Our application uses linear regression models to make predictions and check if the platform is experiencing any operational problems that can impact in revenue losses. The application monitor distributed systems and provides notifications stating the problem detected, that way users can operate quickly to avoid serious problems which directly impact the company’s revenue and reduce the time for action. We will present an architecture for not only a monitoring system, but also an active actor for our outages recoveries. At the end of the presentation you will have access to our training program source code and you will be able to adapt and implement in your company. This solution already helped to prevent about US$3mi in losses last year.

Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...

Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...

Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...

Getting Ready to use Redis with Apache Spark is a technical tutorial designed to address integrating Redis with an Apache Spark deployment to increase the performance of serving complex decision models. To set the context for the session, we start with a quick introduction to Redis and the capabilities Redis provides. We cover the basic data types provided by Redis and cover the module system. Using an ad serving use-case, we look at how Redis can improve the performance and reduce the cost of using complex ML-models in production. Attendees will be guided through the key steps of setting up and integrating Redis with Spark, including how to train a model using Spark then load and serve it using Redis, as well as how to work with the Spark Redis module. The capabilities of the Redis Machine Learning Module (redis-ml) will be discussed focusing primarily on decision trees and regression (linear and logistic) with code examples to demonstrate how to use these feature. At the end of the session, developers should feel confident building a prototype/proof-of-concept application using Redis and Spark. Attendees will understand how Redis complements Spark and how to use Redis to serve complex, ML-models with high performance.

Getting Ready to Use Redis with Apache Spark with Dvir Volk

Getting Ready to Use Redis with Apache Spark with Dvir Volk

Getting Ready to Use Redis with Apache Spark with Dvir Volk

Here we present a general supervised framework for record deduplication and author-disambiguation via Spark. This work differentiates itself by – Application of Databricks and AWS makes this a scalable implementation. Compute resources are comparably lower than traditional legacy technology using big boxes 24/7. Scalability is crucial as Elsevier’s Scopus data, the biggest scientific abstract repository, covers roughly 250 million authorships from 70 million abstracts covering a few hundred years. – We create a fingerprint for each content by deep learning and/or word2vec algorithms to expedite pairwise similarity calculation. These encoders substantially reduce compute time while maintaining semantic similarity (unlike traditional TFIDF or predefined taxonomies). We will briefly discuss how to optimize word2vec training with high parallelization. Moreover, we show how these encoders can be used to derive a standard representation for all our entities namely such as documents, authors, users, journals, etc. This standard representation can simplify the recommendation problem into a pairwise similarity search and hence it can offer a basic recommender for cross-product applications where we may not have a dedicate recommender engine designed. – Traditional author-disambiguation or record deduplication algorithms are batch-processing with small to no training data. However, we have roughly 25 million authorships that are manually curated or corrected upon user feedback. Hence, it is crucial to maintain historical profiles and hence we have developed a machine learning implementation to deal with data streams and process them in mini batches or one document at a time. We will discuss how to measure the accuracy of such a system, how to tune it and how to process the raw data of pairwise similarity function into final clusters. Lessons learned from this talk can help all sort of companies where they want to integrate their data or deduplicate their user/customer/product databases.

Deduplication and Author-Disambiguation of Streaming Records via Supervised M...

Deduplication and Author-Disambiguation of Streaming Records via Supervised M...

Deduplication and Author-Disambiguation of Streaming Records via Supervised M...

The use of large-scale machine learning and data mining methods is becoming ubiquitous in many application domains ranging from business intelligence and bioinformatics to self-driving cars. These methods heavily rely on matrix computations, and it is hence critical to make these computations scalable and efficient. These matrix computations are often complex and involve multiple steps that need to be optimized and sequenced properly for efficient execution. This work presents new efficient and scalable matrix processing and optimization techniques based on Spark. The proposed techniques estimate the sparsity of intermediate matrix-computation results and optimize communication costs. An evaluation plan generator for complex matrix computations is introduced as well as a distributed plan optimizer that exploits dynamic cost-based analysis and rule-based heuristics The result of a matrix operation will often serve as an input to another matrix operation, thus defining the matrix data dependencies within a matrix program. The matrix query plan generator produces query execution plans that minimize memory usage and communication overhead by partitioning the matrix based on the data dependencies in the execution plan. We implemented the proposed matrix techniques inside the Spark SQL, and optimize the matrix execution plan based on Spark SQL Catalyst. We conduct case studies on a series of ML models and matrix computations with special features on different datasets. These are PageRank, GNMF, BFGS, sparse matrix chain multiplications, and a biological data analysis. The open-source library ScaLAPACK and the array-based database SciDB are used for performance evaluation. Our experiments are performed on six real-world datasets are: social network data ( e.g., soc-pokec, cit-Patents, LiveJournal), Twitter2010, Netflix recommendation data, and 1000 Genomes Project sample. Experiments demonstrate that our proposed techniques achieve up to an order-of-magnitude performance.

MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...

MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...

MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...

More from Spark Summit (20)

FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang

FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang

FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang

VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...

VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...

VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...

Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu

Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu

Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu

Improving Traffic Prediction Using Weather Data with Ramya Raghavendra

Improving Traffic Prediction Using Weather Data with Ramya Raghavendra

Improving Traffic Prediction Using Weather Data with Ramya Raghavendra

A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...

A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...

A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...

No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...

No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...

No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...

Apache Spark and Tensorflow as a Service with Jim Dowling

Apache Spark and Tensorflow as a Service with Jim Dowling

Apache Spark and Tensorflow as a Service with Jim Dowling

Apache Spark and Tensorflow as a Service with Jim Dowling

Apache Spark and Tensorflow as a Service with Jim Dowling

Apache Spark and Tensorflow as a Service with Jim Dowling

MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...

MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...

MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...

Next CERN Accelerator Logging Service with Jakub Wozniak

Next CERN Accelerator Logging Service with Jakub Wozniak

Next CERN Accelerator Logging Service with Jakub Wozniak

Powering a Startup with Apache Spark with Kevin Kim

Powering a Startup with Apache Spark with Kevin Kim

Powering a Startup with Apache Spark with Kevin Kim

Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra

Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra

Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra

Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...

Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...

Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...

How Nielsen Utilized Databricks for Large-Scale Research and Development with...

How Nielsen Utilized Databricks for Large-Scale Research and Development with...

How Nielsen Utilized Databricks for Large-Scale Research and Development with...

Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...

Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...

Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...

Goal Based Data Production with Sim Simeonov

Goal Based Data Production with Sim Simeonov

Goal Based Data Production with Sim Simeonov

Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...

Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...

Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...

Getting Ready to Use Redis with Apache Spark with Dvir Volk

Getting Ready to Use Redis with Apache Spark with Dvir Volk

Getting Ready to Use Redis with Apache Spark with Dvir Volk

Deduplication and Author-Disambiguation of Streaming Records via Supervised M...

Deduplication and Author-Disambiguation of Streaming Records via Supervised M...

Deduplication and Author-Disambiguation of Streaming Records via Supervised M...

MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...

MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...

MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...

Recently uploaded

ALSO dropshipping via API with DroFx.pptx

ALSO dropshipping via API with DroFx.pptx

ALSO dropshipping via API with DroFx.pptx

olyaivanovalion

Ashok Vihar Call Girls in Delhi (–9953330565) Escort Service In Delhi NCR PROVIDE 100% REAL GIRLS ALL ARE GIRLS LOOKING MODELS AND RAM MODELS ALL GIRLS” INDIAN , RUSSIAN ,KASMARI ,PUNJABI HOT GIRLS AND MATURED HOUSE WIFE BOOKING ONLY DECENT GUYS AND GENTLEMAN NO FAKE PERSON FREE HOME SERVICE IN CALL FULL AC ROOM SERVICE IN SOUTH DELHI Ultimate Destination for finding a High Profile Independent Escorts in Delhi.Gurgaon.Noida..!.Like You Feel 100% Real Girl Friend Experience. We are High Class Delhi Escort Agency offering quality services with discretion. We only offer services to gentlemen people. We have lots of girls working with us like students, Russian, models, house wife, and much More We Provide Short Time and Full Night Service Call ☎☎+91–9953330565 ❤꧂ • In Call and Out Call Service in Delhi NCR • 3* 5* 7* Hotels Service in Delhi NCR • 24 Hours Available in Delhi NCR • Indian, Russian, Punjabi, Kashmiri Escorts • Real Models, College Girls, House Wife, Also Available • Short Time and Full Time Service Available • Hygienic Full AC Neat and Clean Rooms Avail. In Hotel 24 hours • Daily New Escorts Staff Available • Minimum to Maximum Range Available. Location;- Delhi, Gurgaon, NCR, Noida, and All Over in Delhi Hotel and Home Services HOTEL SERVICE AVAILABLE :-REDDISSON BLU,ITC WELCOM DWARKA,HOTEL-JW MERRIOTT,HOLIDAY INN MAHIPALPUR AIROCTY,CROWNE PLAZA OKHALA,EROSH NEHRU PLACE,SURYAA KALKAJI,CROWEN PLAZA ROHINI,SHERATON PAHARGANJ,THE AMBIENC,VIVANTA,SURAJKUND,ASHOKA CONTINENTAL , LEELA CHANKYAPURI,_ALL 3* 5* 7* STARTS HOTEL SERVICE BOOKING CALL Call WHATSAPP Call ☎+91–9953330565❤꧂ NIGHT SHORT TIME BOTH ARE AVAILABLE

Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service

Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service

Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service

9953056974 Low Rate Call Girls In Saket, Delhi NCR

BigBuy dropshipping via API with DroFx.pptx

BigBuy dropshipping via API with DroFx.pptx

BigBuy dropshipping via API with DroFx.pptx

olyaivanovalion

Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage with sex Booking Contact Details :- WhatsApp Chat :- +91-9920725232 4-May-2024(SMW) Call Girls In Model Towh +91-9920725232 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in NCR 24 Hours Available Service Call Girls, Contact Us +91-9920725232 (Any Time. Any Where) Call Girls in , Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service NCRWelcome To Escorts Service – An All Over New Very Sexy Hot Call Girls Agency Service Escorts In South NCR’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —9920725232 We are available 24*7 all days of the year. Call us — 9920725232 Thank you for Visiting.

Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...

Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...

Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...

FESE Capital Markets Fact Sheet 2024 Q1.pdf

FESE Capital Markets Fact Sheet 2024 Q1.pdf

FESE Capital Markets Fact Sheet 2024 Q1.pdf

MarinCaroMartnezBerg

Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore Escorts Service Booking Contact Details :- WhatsApp Chat :- +91-7737669865 2-May-2024(SMW) Call Girls In Model Towh Bangalore +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in Bangalore NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in Bangalore, Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service Bangalore NCRWelcome To Bangalore Escorts Service – An All Over New Bangalore Very Sexy Hot Call Girls Agency Service Escorts In South BangaloreNCRBangalore’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Bangalore Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...

Call Girl In Dwarka ☎92055#41914 ¶¶ Indian,Russian Best Quality full Educated And Full Cooperative Independent Call Girls Escort Services In New Delhi- I Have Extremely Beautiful Broad Minded Cute Sexy & Hot Call Girls and Escorts, We Are Located in 3* 4* 5* Hotels in Delhi. Safe & Secure High Class Services Affordable Rate 100% Satisfaction, Unlimited Enjoyment. Any Time for Model/Teens Escort in Delhi High class luxury and premium escorts agency Indian Russian Call Girls In Delhi Booking Good High Profile Escorts (Call Girls) In Delhi 5 Star Hotel ,Incall Service,OutCall Service, We provide services by Call Girls,College Girls,Modals Get High Profile queens,Well Educated,Good Looking,Full Cooperative Model, Russian Models,Punjabi Girls Kashmeri Girls Services etc… We Provide Hottest Female With Safe And Consensual With Most Limits Respected Complete Satisfaction Guaranteed…Service. Call Me Spacial For Including Incall//outcall Service In New Delhi Indian Russian Escorts Service,

Cheap Rate Call girls Sarita Vihar Delhi 9205541914 shot 1500 night

Cheap Rate Call girls Sarita Vihar Delhi 9205541914 shot 1500 night

Cheap Rate Call girls Sarita Vihar Delhi 9205541914 shot 1500 night

Delhi Call girls

Ravak dropshipping via API with DroFx.pptx

Ravak dropshipping via API with DroFx.pptx

Ravak dropshipping via API with DroFx.pptx

olyaivanovalion

Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (Bangalore) Booking Contact Details :- WhatsApp Chat :- +91-7737669865 2-May-2024(SMW) Call Girls In Model Towh Bangalore +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in Bangalore NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in Bangalore, Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service Bangalore NCRWelcome To Bangalore Escorts Service – An All Over New Bangalore Very Sexy Hot Call Girls Agency Service Escorts In South BangaloreNCRBangalore’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Bangalore Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...

Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...

Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...

Halmar dropshipping via API with DroFx

Halmar dropshipping via API with DroFx

Halmar dropshipping via API with DroFx

olyaivanovalion

Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore Booking Contact Details :- WhatsApp Chat :- +91-7737669865 2-May-2024(SMW) Call Girls In Model Towh Bangalore +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in Bangalore NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in Bangalore, Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service Bangalore NCRWelcome To Bangalore Escorts Service – An All Over New Bangalore Very Sexy Hot Call Girls Agency Service Escorts In South BangaloreNCRBangalore’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Bangalore Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...

Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...

Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...

ELKO dropshipping via API with DroFx.pptx

ELKO dropshipping via API with DroFx.pptx

ELKO dropshipping via API with DroFx.pptx

olyaivanovalion

Digital advertising, or paid media, encompasses the strategic deployment of online advertisements to reach target audiences efficiently and effectively. This includes any digital platform that supports advertising to deliver unique messages for any objective. Understanding the mechanics of digital advertising platforms, along with insights into audience behaviors and preferences, allows marketers to optimize their ad spend and achieve significant engagement and conversion rates. This lecture is for Advanced Digital & Social Media Strategy (MGMTX 466.05) at UCLA Extension.

Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...

Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...

Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...

Valters Lauzums

Capstone Project on IBM Data Analytics Program

Capstone Project on IBM Data Analytics Program

Capstone Project on IBM Data Analytics Program

MoniSankarHazra

Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha Call girls in dubai Call girls at dubai Dubai Call girl mistaken. Just Call girl dubai Call girl in dubai Indian Call girls dubai Indian Call girl dubai Pakistan Call girls in dubai the Pakistani Call girl dubai https://services.tochat.be/whatsapp-business-directory/b22b3d16-5b26-4e59-b6c1-db53f14dfea0?utm_medium=social&utm_source=heylink.me Dubai Call girls service Dubai Call girl services Call girl service in dubai Dubai Call girl agency Dubai Call girls agency Verified Call girls dubai correct motivation is required. Her smile enlarges as if she were Young Call girls in dubai Marina Call girls Dubai marina Call girls Jumeirah Call girls Dubai Jumeirah Call girls Bur dubai Call girls Indian Call girls in bur dubai Call girls bur dubai hiding a tremendous secret. Al qusais Call girls Al nahda dubai Call girls Independent Call girls dubai Independent Call girl dubai Russian Call girls in dubai Dubai russian Call girls Young Call girls in dubai Dubai young Call girls Call girls numbers in dubai How about leaving your father's home, being wealthy, and being able to help your sister? Even though I know what she is going to say won't be good, my ears are ringing. To have this chat, I waited until Dubai Call girls number Call girls near me dubai Call girls near my hotel Cute Call girls in dubai Model Call girl in dubai Rent a girlfriend dubai you were eighteen years old. Do you understand what I do, Eden? Since I have no idea, I shake my head and my mind races. She must be some kind of successful businesswoman, I suppose. "I own a business. Do you recognize that? Knowing my best. She left. She said that Dad told her that Dubai Call girls Call girls dubai Call girls in dubai Call girls at dubai we didn’t need her anymore when he came home. I was sad.Dubai Call girl Call girl dubai Call girl in dubai Indian Call girls dubai Indian Call girl dubai Can you tell her to come back? I like her.” Her little face is Pakistan Call girls in dubai Pakistani Call girl dubai Dubai Call girls service Dubai Call girl services all pinched. So sweet. Call girl service in dubai Dubai Call girl agency Dubai Call girls agency Verified Call girls dubai But I'm pissed off. How can he Young Call girls in dubai Marina Call girls Dubai marina Call girls Jumeirah Call girls Dubai Jumeirah Call girls Bur dubai Call girls Indian Call girls in bur dubai Call girls bur dubai turn down someone I'm paying for? “So, who's here with you?” I ask her,Al qusais Call girls Al nahda dubai Call girls Independent Call girls dubai Independent Call girl dubai Russian Call girls in dubai Dubai russian Call girls fervently hoping she wasn’t here alone. “Dad's downstairs, I think Young Call girls in dubai Dubai young Call girls Call girls numbers in dubai Dubai Call girls number Call girls near me dubai Call girls near my hotel Cute Call girls in dubai Model Call girl in dubai Rent a girlfriend

Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha

Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha

Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha

Saudi Arabia [ Abortion pills) Jeddah/riaydh/dammam/+966572737505☎️] cytotec tablets uses abortion pills 💊💊 How effective is the abortion pill? 💊💊 +966572737505) "Abortion pills in Jeddah" how to get cytotec tablets in Riyadh " Abortion pills in dammam*💊💊 The abortion pill is very effective. If you’re taking mifepristone and misoprostol, it depends on how far along the pregnancy is, and how many doses of medicine you take:💊💊 +966572737505) how to buy cytotec pills At 8 weeks pregnant or less, it works about 94-98% of the time. +966572737505[ 💊💊💊 At 8-9 weeks pregnant, it works about 94-96% of the time. +966572737505) At 9-10 weeks pregnant, it works about 91-93% of the time. +966572737505)💊💊 If you take an extra dose of misoprostol, it works about 99% of the time. At 10-11 weeks pregnant, it works about 87% of the time. +966572737505) If you take an extra dose of misoprostol, it works about 98% of the time. In general, taking both mifepristone and+966572737505 misoprostol works a bit better than taking misoprostol only. +966572737505 Taking misoprostol alone works to end the+966572737505 pregnancy about 85-95% of the time — depending on how far along the+966572737505 pregnancy is and how you take the medicine. +966572737505 The abortion pill usually works, but if it doesn’t, you can take more medicine or have an in-clinic abortion. +966572737505 When can I take the abortion pill?+966572737505 In general, you can have a medication abortion up to 77 days (11 weeks)+966572737505 after the first day of your last period. If it’s been 78 days or more since the first day of your last+966572737505 period, you can have an in-clinic abortion to end your pregnancy.+966572737505 Why do people choose the abortion pill? Which kind of abortion you choose all depends on your personal+966572737505 preference and situation. With+966572737505 medication+966572737505 abortion, some people like that you don’t need to have a procedure in a doctor’s office. You can have your medication abortion on your own+966572737505 schedule, at home or in another comfortable place that you choose.+966572737505 You get to decide who you want to be with during your abortion, or you can go it alone. Because+966572737505 medication abortion is similar to a miscarriage, many people feel like it’s more “natural” and less invasive. And some+966572737505 people may not have an in-clinic abortion provider close by, so abortion pills are more available to+966572737505 them. +966572737505 Your doctor, nurse, or health center staff can help you decide which kind of abortion is best for you. +966572737505 More questions from patients: Saudi Arabia+966572737505 CYTOTEC Misoprostol Tablets. Misoprostol is a medication that can prevent stomach ulcers if you also take NSAID medications. It reduces the amount of acid in your stomach, which protects your stomach lining. The brand name of this medication is Cytotec®.+966573737505) Unwanted Kit is a combination of two medici

Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec

Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec

Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec

Abortion pills in Riyadh +966572737505 get cytotec

Discover Why Less is More in B2B Research

Discover Why Less is More in B2B Research

Discover Why Less is More in B2B Research

Mg Road Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore Escorts Service Booking Contact Details :- WhatsApp Chat :- +91-7737669865 4-May-2024(SMW) Call Girls In Model Towh +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in , Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service NCRWelcome To Escorts Service – An All Over New Very Sexy Hot Call Girls Agency Service Escorts In South NCR’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

Mg Road Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Banga...

Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore Booking Contact Details :- WhatsApp Chat :- +91-7737669865 2-May-2024(SMW) Call Girls In Model Towh Bangalore +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in Bangalore NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in Bangalore, Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service Bangalore NCRWelcome To Bangalore Escorts Service – An All Over New Bangalore Very Sexy Hot Call Girls Agency Service Escorts In South BangaloreNCRBangalore’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Bangalore Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...

Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...

Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore Booking Contact Details :- WhatsApp Chat :- +91-7737669865 2-May-2024(SMW) Call Girls In Model Towh Bangalore +91-7737669865 !! Best Woman Seeking Man Call Girls Service, Escorts Service in Home Hotel in Bangalore NCR 24 Hours Available Service Call Girls, Contact Us +91-7737669865 (Any Time. Any Where) Call Girls in Bangalore, Noida, Gurgaon, Ghaziabad,Sexy Indian Female Escorts Service Bangalore NCRWelcome To Bangalore Escorts Service – An All Over New Bangalore Very Sexy Hot Call Girls Agency Service Escorts In South BangaloreNCRBangalore’s No. 1 High Profile Independent Female Escorts Service. We Provide Good Quality Educated Profile At Very Regnebal Price 100% Safe And Original.We Are Provide Escorts Service All OYO Hotels ,3*,4*,5* Star Hotel And Home Flat, Apartment. Guest-House. Services In -Call And Out – Call Both Are Services Available. 24Hrs. Any Time Any Where. In All Over Bangalore Noida Gurgaon Ghaziabad Faridabad.More Information And Contact Profile Real Pic Visit Our Website City Wise Escorts Service Agency.Good Looking Cheap And Best Models Girls U Can Get Best Click On Link……Night Call Girls Now In Hotel Le Meridien Gurgaon Near Female Escort One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7737669865 We are available 24*7 all days of the year. Call us — 7737669865 Thank you for Visiting.

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...

Recently uploaded (20)

ALSO dropshipping via API with DroFx.pptx

ALSO dropshipping via API with DroFx.pptx

ALSO dropshipping via API with DroFx.pptx

Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service

Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service

Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service

BigBuy dropshipping via API with DroFx.pptx

BigBuy dropshipping via API with DroFx.pptx

BigBuy dropshipping via API with DroFx.pptx

Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...

Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...

Vip Mumbai Call Girls Thane West Call On 9920725232 With Body to body massage...

FESE Capital Markets Fact Sheet 2024 Q1.pdf

FESE Capital Markets Fact Sheet 2024 Q1.pdf

FESE Capital Markets Fact Sheet 2024 Q1.pdf

Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...

Cheap Rate Call girls Sarita Vihar Delhi 9205541914 shot 1500 night

Cheap Rate Call girls Sarita Vihar Delhi 9205541914 shot 1500 night

Cheap Rate Call girls Sarita Vihar Delhi 9205541914 shot 1500 night

Ravak dropshipping via API with DroFx.pptx

Ravak dropshipping via API with DroFx.pptx

Ravak dropshipping via API with DroFx.pptx

Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...

Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...

Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...

Halmar dropshipping via API with DroFx

Halmar dropshipping via API with DroFx

Halmar dropshipping via API with DroFx

Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...

Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...

Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...

ELKO dropshipping via API with DroFx.pptx

ELKO dropshipping via API with DroFx.pptx

ELKO dropshipping via API with DroFx.pptx

Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...

Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...

Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...

Capstone Project on IBM Data Analytics Program

Capstone Project on IBM Data Analytics Program

Capstone Project on IBM Data Analytics Program

Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha

Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha

Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha

Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec

Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec

Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec

Discover Why Less is More in B2B Research

Discover Why Less is More in B2B Research

Discover Why Less is More in B2B Research

Mg Road Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Banga...

Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...

Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...

Call Girls Jalahalli Just Call 👗 7737669865 👗 Top Class Call Girl Service Ban...

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...

Call Girls Hsr Layout Just Call 👗 7737669865 👗 Top Class Call Girl Service Ba...

The Little Warehouse That Couldn't Or: How We Learned to Stop Worrying and Move to Spark-(Yandu Oppacher, Shopify)

1. The Little Warehouse That Couldn’t Or: How We Learned to Stop Worrying and Move to Spark 1 Yandu Oppacher (@yandu) Data Infrastructure

2. 2 Shopify Stores

3. ETL Warehouse Reporting August 2013 TilllerRuby Vertica 3

4. Why we had to move • Data volume • Data/Query complexity • Performance issues 4

5. Couple of false starts 5 Pig + Luigi Pig + Oozie Platfora

6. –platfora.com “Without coding or ETL, data warehousing, BI tools, or breaking a sweat.” 6

7. Enter Spark • Fast • Nice development model • Python 7

8. 88 The Good Book

9. GMV A Case Study 9 165,000+ ACTIVE SHOPIFY MERCHANTS $8 BILLION+ CUMULATIVE GMV

10. Growing pains • Joins • Groupings • General data skew • Getting to know python’s performance quirks 10

11. Starscream 11 • specialized joins • resolvers • range • cassandra • overby • contracts • incrementalized fact builds

12. Our current stack 12 Kafka OLTP HDFS Cassandra Spark FrontroomBackroom Redshift Tableau

13. Thank you 13 Yandu Oppacher (@yandu) Data Infrastructure