Datatypes for the real world

•Download as PPT, PDF•

5 likes•2,221 views

“Make illegal state unrepresentable” - Yaron Minsky Starting from Yaron’s quote we will see where it traditionally applies: values sanitization (escaping), data structures (NonEmptyList), CS constructs (parsers). Does it still apply to the rest of the day-to-day programming where things look… complicated? In particular we will develop the example of executing applications which can run on single Amazon EC2 instances (for testing) or on full EMR clusters (for production) and which can take data from S3 (meaning that they need to download it and put it somewhere first) or use local data. Can we make sure we get the right paths? Can we know when it is possible to download data efficiently (using distcopy to go directly from S3 to the cluster for example)? The answer is: create specific data types to model the different cases and tame the complexity. The take-away: datatypes are not only for “data” but they can also encapsulate “logic”

Data types for the
Real World ™
Eric Torreborre
YOW LambdaJam 2015

Yaron Minsky Make illegal state
unrepresentable!
Make illegal state
unrepresentable!

Make illegal state
unrepresentable!
Make illegal state
unrepresentable!
Real W
orld?
Constraints
CS
Data
structures

sync
sync
sync
storage
execution execution

sync
storage
“Hadoop – half sync”
execution

“sync” files
download?
where?
how?
upload?

location execution
local local
hdfs
S3 local
hdfs
hdfs local
hdfs

createSynchronizedLocation
9 cases
unrepresentable!

Locations
Location
ExecutionLocation
SynchronizedLocation
Datatypes are
cheap!

Make illegal state
unrepresentable!
Make illegal state
unrepresentable!
For the Real World ™

M3D stands for Metadata Driven Development and is a cloud and platform-agnostic framework for the automated creation, management, and governance of metadata and data flows from multiple sources to multiple target systems. It is a tool comprised of two components: m3d-API and m3d-engine. At Adidas the framework is used for the creation of data lake environments, management and governance of metadata, data flows from multiple sources and algorithms as data frame transformations. It is feeding 500+ tables and over 900 views. Author: Jorge Cespedes (Adidas)

Airflow - An Open Source Platform to Author and Monitor Data PipelinesDataWorks Summit

Running Fast, Interactive Queries on Petabyte Datasets using Presto - AWS Jul...

Amazon Web Services

Learn how to deploy a managed Presto environment to interactively query log data on AWS Organizations often need to quickly analyze large amounts of data, such as logs, generated from a wide variety of sources and formats. However, traditional approaches require a lot of time and effort designing complex data transformation and loading processes; and configuring data warehouses. Using AWS, you can start querying your datasets within minutes In this webinar you will learn how you can deploy a managed Presto environment in minutes to interactively query log data using plain ANSI SQL. Presto is a popular open source SQL engine for running interactive analytic queries against data sources of all sizes. We will talk about common use cases and best practices for running Presto on Amazon EMR. Learning Objectives: • Learn how to deploy a managed Presto environment running on Amazon EMR • Understand best practices for running Presto on Amazon EMR, including use of Amazon EC2 Spot instances • Learn how other customers are using Presto to analyze large data sets

Amazon EMR Facebook Presto Meetup

stevemcpherson

Analyzing Mixpanel Data into Amazon Redshift

George Psistakis

This presentation is an overview guide to help us define a process or data pipeline, to load data from Mixpanel into Amazon Redshift for further analysis. We will see how to: - access and extract data from Mixpanel through its API - how to load it into Redshift This is not a full solution as it will require to writing the code to get the data and make sure that this process will run every time new data are generated.

Serverless to author, schedule, execute and monitor data workflows.

Anselmo Rodrigues da Silva

Expedia Affiliates Network (EAN), is the B2B partnership brand of Expedia, Inc. Our technology powers the hotel offering of thousands of partners around the world. Everyday EAN produces, processes and consumes more than 10TB of data to drive applications, services and business through granular insights. EAN new data cloud platform is an opportunity to be nimble, simplify and consolidate the authoring, schedule, execution and monitoring of complex data workflows. The talk explores EAN’s strategy on data processing in the cloud, moving from traditional workflow tools to a serverless approach, leveraging step-function, lambda, API gateway, EMR and Cloud Watch.

Building Robust Pipelines with Airflow

Erin Shellman

The data science team at Zymergen is applying machine learning techniques to identify genetic targets, work that is supported by extensive analytical automation that systematically identifies outliers, removes process-related bias, and quantifies performance improvements. We’re using Apache Airflow to construct robust data pipelines that allow us to produce clean, reliable inputs to our predictive models. In this talk, I’ll discuss the unique data processing challenges we face in working with high-throughput, biological data and provide an overview of how we’re using Apache Airflow to meet those challenges.

Movie Recommendations using Map Reduce and Python

Avinash Pandu

In this session, we discuss how Spark and Presto complement the Netflix big data platform stack that started with Hadoop, and the use cases that Spark and Presto address. Also, we discuss how we run Spark and Presto on top of the Amazon EMR infrastructure; specifically, how we use Amazon S3 as our data warehouse and how we leverage Amazon EMR as a generic framework for data-processing cluster management.

Dataflow in 104corp - DataConTW2018

Gavin Lin

Atlantisで実現するTerraformのGitOps

理弘山崎

Presentation kyushu-2018

masahitojp

Honu - A Large Scale Streaming Data Collection and Processing Pipeline__Hadoo...

Yahoo Developer Network

Hw09 Building Data Intensive Apps A Closer Look At Trending Topics.OrgCloudera, Inc.

Tech Thursday - Beer & DevOps 24.11.

Nebula Oy

Dataflow in 104corp - AWS UserGroup TW 2018

Gavin Lin

«Почему Spark отнюдь не так хорош»

Olga Lavrentieva

Алексей Дёмин (Java Developer в InData Labs) Доклад: «Почему Spark отнюдь не так хорош» О чём: Сейчас по всем каналам идёт обсуждение новой революционной технологии обработки данных Spark. Алексей предлагает взглянуть чуть глубже и узнать, действительно ли Spark так хорош, как нам рассказывает об этом маркетинг.

PyconUK-2015Heeren Sharma

Simplify Big Data with AWS

Julien SIMON

Best Practices for Hosting Web Applications on AWSAmazon Web Services

MongoDB World 2018: Spark and Machine Learning

MongoDB

Building High-availability Websites on AWSAmazon Web Services

High-Availability Websites and Web Applications with AWSAmazon Web Services

BDA305 Building Data Lakes and Analytics on AWS

Amazon Web Services

Over 90% of today’s data was generated in the last 2 years, and the rate of data growth isn’t slowing down. In this session, we’ll step through the challenges and best practices on how to capture all the data that is being generated, understand what data you have, and start driving insights and even predict the future using purpose built AWS Services. We’ll frame the session and demonstrations around common pitfalls of building Data Lakes and how to successful drive analytics and insights from the data. This session will focus on the architecture patterns bringing together key AWS Services and rather than a deep dive on any single service. We’ll show how services such as Amazon S3, Amazon Glue, AWS Glue, Amazon Redshift, Amazon Athena, Amazon EMR, and Amazon Kinesis, and Amazon Machine Learning services are put together to build a successful data lake for various role including both data scientists and business users.

Websites on AWS

Amazon Web Services

Connecting Your Data Analytics Pipeline

Amazon Web Services

CTO Night & Days 2015 Winter - AWS Mobile Development

崇之清水

Build Data Lakes & Analytics on AWS: Patterns & Best Practices - BDA305 - Ana...

Amazon Web Services

In this session, we show you how to understand what data you have, how to drive insights, and how to make predictions using purpose-built AWS services. Learn about the common pitfalls of building data lakes, and discover how to successfully drive analytics and insights from your data. Also learn how services such as Amazon S3, AWS Glue, Amazon Redshift, Amazon Athena, Amazon EMR, Amazon Kinesis, and Amazon ML services work together to build a successful data lake for various roles, including data scientists and business users.

What's hot

Riot Games - Player Focused Pipeline - Stampedecon 2015

sean_seannery

Presto @ Netflix: Interactive Queries at Petabyte Scale

DataWorks Summit

(BDT303) Running Spark and Presto on the Netflix Big Data Platform

Amazon Web Services

Dataflow in 104corp - DataConTW2018

Gavin Lin

Atlantisで実現するTerraformのGitOps

理弘山崎

Presentation kyushu-2018

masahitojp

Honu - A Large Scale Streaming Data Collection and Processing Pipeline__Hadoo...

Yahoo Developer Network

Hw09 Building Data Intensive Apps A Closer Look At Trending Topics.OrgCloudera, Inc.

Tech Thursday - Beer & DevOps 24.11.

Nebula Oy

Dataflow in 104corp - AWS UserGroup TW 2018

Gavin Lin

«Почему Spark отнюдь не так хорош»

Olga Lavrentieva

PyconUK-2015Heeren Sharma

What's hot (12)

Riot Games - Player Focused Pipeline - Stampedecon 2015

Presto @ Netflix: Interactive Queries at Petabyte Scale

(BDT303) Running Spark and Presto on the Netflix Big Data Platform

Dataflow in 104corp - DataConTW2018

Atlantisで実現するTerraformのGitOps

Presentation kyushu-2018

Honu - A Large Scale Streaming Data Collection and Processing Pipeline__Hadoo...

Hw09 Building Data Intensive Apps A Closer Look At Trending Topics.Org

Tech Thursday - Beer & DevOps 24.11.

Dataflow in 104corp - AWS UserGroup TW 2018

«Почему Spark отнюдь не так хорош»

PyconUK-2015

Similar to Datatypes for the real world

Simplify Big Data with AWS

Julien SIMON

Best Practices for Hosting Web Applications on AWSAmazon Web Services

MongoDB World 2018: Spark and Machine Learning

MongoDB

Building High-availability Websites on AWSAmazon Web Services

High-Availability Websites and Web Applications with AWSAmazon Web Services

BDA305 Building Data Lakes and Analytics on AWS

Amazon Web Services

Websites on AWS

Amazon Web Services

Connecting Your Data Analytics Pipeline

Amazon Web Services

CTO Night & Days 2015 Winter - AWS Mobile Development

崇之清水

Build Data Lakes & Analytics on AWS: Patterns & Best Practices - BDA305 - Ana...

Amazon Web Services

Scalable Event Analytics with MongoDB & Ruby on Rails

Jared Rosoff

B3 - Business intelligence apps on aws

Amazon Web Services

Business intelligence is often described as a set of methodologies and technologies that transform raw data into meaningful and useful information for business purposes. But this simple description hides many technical challenges IT teams struggle with. This session will show how to build business intelligence applications leveraging AWS, from the raw data import, consumption and storage down to the information production. We will also cover best practices for services such as Amazon Redshift or Amazon RDS, and how to use applications such as SAP Hana, Jaspersoft and others.

(BDT308) Using Amazon Elastic MapReduce as Your Scalable Data Warehouse | AWS...

Amazon Web Services

In this presentation, we will demonstrate how to use Amazon Elastic MapReduce as your scalable data warehouse. Amazon EMR supports clusters with thousands of nodes and is used to access petabyte scale data warehouses. Amazon EMR is not only fast, but it is also easy to use for rapid development and adhoc analysis. We will show you how access the large scale data warehouses with emerging tools such as Hue, Hive, low latency SQL applications like Presto, and alternative execution engines like Apache Spark. We will also show you how these tools integrate directly with other AWS big data services such as Amazon S3, Amazon DynamoDB, and Amazon Kinesis.

The Cloud as a Platform - Cloud Connections 2011 Keynote - Jinesh Varia

Amazon Web Services

Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...

Amazon Web Services

(BDT402) Performance Profiling in Production: Analyzing Web Requests at Scale...

Amazon Web Services

Code profiling gives a rich, detailed view of runtime performance. However, it's difficult to achieve in production: for even a small fraction of web requests, huge challenges in scalability, access, and ease of use appear. Despite this, Yelp profiles a nontrivial fraction of its traffic by combining Amazon EC2, Amazon EMR, and Amazon S3. Developers can search, sort, filter, and combine interesting profiles; during a site slowdown or page failure, this allows a fast diagnosis and speedy recovery. Some of our analyses run nightly, while others run in real-time via Storm topologies. This session includes our use cases for code profiling, its benefits, and the implementation of its handlers and analysis flows. We include both performance results and implementation challenges of our MapReduce and Storm jobs, including code overviews. We also touch on issues such as concurrent logging, cross-data center replication, job scheduling, and API definitions.

Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...

Amazon Web Services

In this session, we show you how to understand what data you have, how to drive insights, and how to make predictions using purpose-built AWS services. Learn about the common pitfalls of building data lakes and discover how to successfully drive analytics and insights from your data. Also learn how services such as Amazon S3, AWS Glue, Amazon Redshift, Amazon Athena, Amazon EMR, Amazon Kinesis, and Amazon ML services work together to build a successful data lake for various roles, including data scientists and business users.

CliqueSquare processing

INRIA-OAK

Docker bdxio

Nicolas De Loof

Lambda Jam 2015: Event Processing in Clojure

Andy Marks

Riemann (riemann.io) is an event stream processor, usually deployed as a router for events generated by software systems. As Kyle Kingsbury, the author states: “Riemann’s job is to process, analyze and react to the event stream”. In effect, event stream processing is functional transformation writ large; events from various sources (e.g., hardware stats, application metrics, network data etc) are received then grouped, filtered, mapped, averaged, coalesced, forwarded, etc. Riemann is written in Clojure. Riemann is also configured via Clojure. It’s “parentheses all the way down”. But does a tool for functional transformation of data require a functional language? To what extent does the implementation language impact the capabilities Riemann exposes to the end user? This presentation will take a deep dive into both the capabilities of the configuration system and the core Clojure implementation to see why a functional language is such a good choice to implement this type of application.

Similar to Datatypes for the real world (20)

Simplify Big Data with AWS

Best Practices for Hosting Web Applications on AWS

MongoDB World 2018: Spark and Machine Learning

Building High-availability Websites on AWS

High-Availability Websites and Web Applications with AWS

BDA305 Building Data Lakes and Analytics on AWS

Websites on AWS

Connecting Your Data Analytics Pipeline

CTO Night & Days 2015 Winter - AWS Mobile Development

Build Data Lakes & Analytics on AWS: Patterns & Best Practices - BDA305 - Ana...

Scalable Event Analytics with MongoDB & Ruby on Rails

B3 - Business intelligence apps on aws

(BDT308) Using Amazon Elastic MapReduce as Your Scalable Data Warehouse | AWS...

The Cloud as a Platform - Cloud Connections 2011 Keynote - Jinesh Varia

Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...

(BDT402) Performance Profiling in Production: Analyzing Web Requests at Scale...

Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...

CliqueSquare processing

Docker bdxio

Lambda Jam 2015: Event Processing in Clojure

More from Eric Torreborre

wire-all-the-things-lambda-days-2023.pdf

Datatypes for the real world

Recommended

Recommended

More Related Content

What's hot

What's hot (12)

Similar to Datatypes for the real world

Similar to Datatypes for the real world (20)

More from Eric Torreborre

More from Eric Torreborre (20)

Recently uploaded

Recently uploaded (20)

Datatypes for the real world