Aws education meetup - Large scale data preprocessing with sagemaker - Weiyih

•

0 likes•57 views

Yap Wei Yih is a senior data scientist at Firemark Labs Singapore who has a master's degree in electrical engineering. He discusses solving three key problems: 1) processing terabytes of geospatial data using AWS EMR and Athena, 2) a scalable model development lifecycle platform using SageMaker, and 3) exposing a scalable API endpoint using Lambda and API Gateway without devops resources. EMR is used to spin up Spark clusters for data cleansing, aggregation, and geospatial functions on large datasets. SageMaker provides full model lifecycle management with notebooks, training, versioning, testing, and deployment. Lambda and API Gateway expose the SageMaker endpoint to external parties through a controlled API.

Technology

About Me – Yap Wei Yih
• Senior Data Scientist @ Firemark Labs Singapore (IAG)
• NYP Alumni – Specialist Diploma in Business and Big Data Analytics
• Master of Science (Electrical Engineering)
• linkedin.com/in/yapweiyih
• Interest - Applied Data Science, Machine Learning

Topic
Large scale data pre-processing, model training and deployment using
AWS EMR, Athena and SageMaker

Key Problems
1. Processing terabytes (billions observation) of geospatial data,
Spark cluster setup
2. A model development lifecycle platform
3. Scalable API endpoint, lack of DevOps resources

#1 - EMR & Athena
Elastic Map Reduce (EMR)
✓ Spin up Spark cluster with just a few clicks
✓ Multi user JupyterHub
✓ Data cleansing and aggregation with Scala/PySpark
✓ Easily configure number of nodes or Autoscaling

Athena
✓ Support geo - Point, Polygon
✓ Geospatial function – ST_Intersect, ST_Contains,
ST_DISTANCE
#1 - EMR & Athena

#2 - SageMaker
• Provides full Model Development Lifecycle Management capability
• Key requirements that is important for data science work:
✓ Jupyter Notebook, Exploratory Data Analysis (EDA)
✓ Support custom algorithm container
✓ Model Training/Versioning
✓ A/B Testing
✓ Model Endpoint Deployment
• Install both R and Python to support two main user groups
• Minimize the work of DevOps for model deployment

#3 - Lambda & API Gateway
Frontend
• SageMaker endpoint is only available within AWS services
• Lambda and API Gateway is used to expose model to external party
• Access by partners is controlled via API key

High Level Solution Architecture
Frontend
Geospatial
Data
Data
Pre-processing
Data
Ingestion
EDA,
Modeling
Deployment Serving

What's hot

16th Athens Big Data Meetup - 1st Talk - An Introduction to Machine Learning ...

Athens Big Data

Saurav Sengupta Resume

SauravSengupta12

Scalable Machine Learning using R and Azure HDInsight - Parashar

Parashar Shah

An emerging trend in automobile insurance is to charge drivers based on how much they drive, as well as how, where and when they drive. Data scientists need to analyze trip-level indicators that affect auto accidents, namely characteristics specific to a driver going from point A to point B. This detailed level of analysis presents several new challenges. First, the collection of detailed trip-level data results in huge volumes of data that must be processed efficiently. Second, automobile accidents are low-frequency events, and a large portion of trips have zero insurance loss. Third, the driver-level variation is influential on statistical inference, but the modeling of a large number of driver-level effects leads to a high-dimensional estimation problem. Learn how Uber uses a novel implementation of the Tweedie mixed-effect model on the Spark distributed computing engine to address these challenges. The Tweedie compound Poisson distribution is employed to model zero-inflated insurance losses, and random effects are used to account for the inherent heterogeneity of drivers. This implementation on Spark enables efficient processing of high volumes of trip-level data and fast estimation of high-dimensional models. Hear about the challenges of estimating mixed-effects models in the distributed environment, and the details of Uber’s implementation.

Large-Scaled Insurance Analytics Using Tweedie Models in Apache Spark with Ya...

Databricks

This talk describes migrating a large random forest classifier from scikit-learn to Spark's MLlib. We cut training time from 2 days to 2 hours, reduced failed runs, and track experiments better with MLflow. Kount provides certainty in digital interactions like online credit card transactions. One of our scores uses a random forest classifier with 250 trees and 100,000 nodes per tree. We used scikit-learn to train using 60 million samples that each contained over 150 features. The in-memory requirements exceeded 750 GB, took 2 days, and were not robust to disruption in our database or training execution. To migrate workflow to Spark, we built a 6-node cluster with HDFS. This provides 1.35 TB of RAM and 484 cores. Using MLlib and parallelization, the training time for our random forests are now less than 2 hours. Training data stays in our production environment, which used to require a deploy cycle to move locally-developed code onto our training server. The new implementation uses Jupyter notebooks for remote development with server-side execution. MLflow tracks all input parameters, code, and git revision number, while the performance and model itself are retained as experiment artifacts. The new workflow is robust to service disruption. Our training pipeline begins by pulling from a Vertica database. Originally, this single connection took over 8 hours to complete with any problem causing a restart. Using sqoop and multiple connections, we pull the data in 45 minutes. The old technique used volatile storage and required the data for each experiment. Now, we pull the data from Vertica one time and then reload much faster from HDFS. While a significant undertaking, moving to the Spark ecosystem converted an ad hoc and hands-on training process into a fully repeatable pipeline that meets regulatory and business goals for traceability and speed. Speaker: Josh Johnston

Moving a Fraud-Fighting Random Forest from scikit-learn to Spark with MLlib, ...

Databricks

"Something worth ten Microsofts" - Azure Machine Learning 101

Haritha Thilakarathne

Vulnerabilities such as Spectre and Meltdown continue to plague many production servers, based on Intel CPUs. Our solution involves software-based monitoring of hardware counters and sending that data to Apache Spark clusters for threat detection. We leverage Spark's support for support vector machine (SVM) inference. Our machine learning models are trained off-line by a data scientist within a Jupyter notebook environment. As new models are validated, they can be easily deployed to the Spark cluster from the notebook. We have standardized model export and import using the ONNX machine learning open file format. In our presentation, we will demo the full pipeline, from model training to deployment. We will discuss the various challenges when deploying ML-based cyber-threat detection at scale using Apache Spark. For example, we found that gaps in detection can occur when Spark models are updated. We will describe a novel data ingestion architecture, based on Apache Kafka, that we developed to deal with this issue.

Scaling ML-Based Threat Detection For Production Cyber Attacks

Databricks

Resume chao han_tsai

CHAO-HAN TSAI

Denver AWS Users' Group meeting - September 2017

David McDaniel

Computation of spatial data on Hadoop Cluster

Abhishek Sagar

Have you imagined a simple machine learning solution able to prevent revenue leakage and monitor your distributed application? To answer this question, we offer a practical and a simple machine learning solution to create an intelligent monitoring application based on simple data analysis using Apache Spark MLlib. Our application uses linear regression models to make predictions and check if the platform is experiencing any operational problems that can impact in revenue losses. The application monitor distributed systems and provides notifications stating the problem detected, that way users can operate quickly to avoid serious problems which directly impact the company’s revenue and reduce the time for action. We will present an architecture for not only a monitoring system, but also an active actor for our outages recoveries. At the end of the presentation you will have access to our training program source code and you will be able to adapt and implement in your company. This solution already helped to prevent about US$3mi in losses last year.

Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...

Spark Summit

Data cleansing and data prep with synapse data flows

Mark Kromer

Running Spark and Python data science workloads can be challenging given the complexity of the various data science tools in the ecosystem like sci-kit Learn, TensorFlow, Spark, Pandas, and MLlib. All these various tools and architectures, provide important trade-offs to consider when it comes to moving to proofs of concept and going to production. While proof of concepts may be relatively straightforward, moving to production can be challenging because it’s difficult to understand not just the short term effort to develop a solution, but the long term cost of supporting projects over the long term. This talk will discuss important tactical patterns for evaluating projects, running proofs of concept to inform going to production, and finally the key tactics we use internally at Databricks to take data and machine learning projects into production. This session will cover some architectural choices involving Spark, PySpark, Pandas, notebooks, various machine learning toolkits, as well as frameworks and technologies necessary to support them.

Tactical Data Science Tips: Python and Spark Together

Databricks

Spark Summit EU: IBM Keynote

sparktc

What's hot (14)

16th Athens Big Data Meetup - 1st Talk - An Introduction to Machine Learning ...

Saurav Sengupta Resume

Scalable Machine Learning using R and Azure HDInsight - Parashar

Large-Scaled Insurance Analytics Using Tweedie Models in Apache Spark with Ya...

Moving a Fraud-Fighting Random Forest from scikit-learn to Spark with MLlib, ...

"Something worth ten Microsofts" - Azure Machine Learning 101

Scaling ML-Based Threat Detection For Production Cyber Attacks

Resume chao han_tsai

Denver AWS Users' Group meeting - September 2017

Computation of spatial data on Hadoop Cluster

Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...

Data cleansing and data prep with synapse data flows

Tactical Data Science Tips: Python and Spark Together

Spark Summit EU: IBM Keynote

Similar to Aws education meetup - Large scale data preprocessing with sagemaker - Weiyih

How do you go from a strictly typed object-based streaming pipeline with simple operations to a structured streaming pipeline with higher order complex relational operations? This is what the Data Engineering team did at GoPro to scale up the development of streaming pipelines for the rapidly growing number of devices and applications. When big data frameworks such as Hadoop first came to exist, developers were happy because we could finally process large amounts of data without writing complex multi-threaded code or worse yet writing complicated distributed code. Unfortunately, only very simple operations were available such as map and reduce. Almost immediately, higher level operations were desired similar to relational operations. And so Hive and dozens (hundreds?) of SQL-based big data tools became available for more developer-efficient batch processing of massive amounts of data. In recent years, big data has moved from batch processing to stream-based processing since no one wants to wait hours or days to gain insights. Dozens of stream processing frameworks exist today and the same trend that occurred in the batch-based big data processing realm has taken place in the streaming world, so that nearly every streaming framework now supports higher level relational operations. In this talk, we will discuss in a very hands-on manner how the streaming data pipelines for GoPro devices and apps have moved from the original Spark streaming with its simple RDD-based operations in Spark 1.x to Spark's structured streaming with its higher level relational operations in Spark 2.x. We will talk about the differences, advantages, and necessary pain points that must be addressed in order to scale relational-based streaming pipelines for massive IoT streams. We will also talk about moving from “hand built” Hadoop/Spark clusters running in the cloud to using a Spark-based cloud service. DAVID WINTERS, Big Data Architect, GoPro and HAO ZOU, Senior Software Engineer, GoPro

Adding structure to your streaming pipelines: moving from Spark streaming to ...

DataWorks Summit

The presentation topic for this meet-up was covered in two sections without any breaks in-between Section 1: Business Aspects (20 mins) Speaker: Rasmi Mohapatra, Product Owner, Experian https://www.linkedin.com/in/rasmi-m-428b3a46/ Once your data science application is in the production, there are many typical data science operational challenges experienced today - across business domains - we will cover a few challenges with example scenarios Section 2: Tech Aspects (40 mins, slides & demo, Q&A ) Speaker: Santanu Dey, Solution Architect, Iguazio https://www.linkedin.com/in/santanu/ In this part of the talk, we will cover how these operational challenges can be overcome e.g. automating data collection & preparation, making ML models portable & deploying in production, monitoring and scaling, etc. with relevant demos.

Challenges of Operationalising Data Science in Production

iguazio

PradeepDWH

Pradeep Pandey

Scaling up with Cisco Big Data: Data + Science = Data Science

eRic Choo

Spark One Platform Webinar

Cloudera, Inc.

Machine learning model to production

Georg Heiler

Big Data Introduction - Solix empower

Durga Gadiraju

Teradata - Architecture of Teradata

Vibrant Technologies & Computers

Spark Summit EU talk by Kent Buenaventura and Willaim Lau

Spark Summit

Introducing new AIOps innovations in Oracle 19c - San Jose AICUG

Sandesh Rao

DataOps with Project Amaterasu

DataWorks Summit/Hadoop Summit

AnilKumarT_Resume_latest

anil_thyagarajan

EnterpriseDB Postgres Plus Advanced Server provides Oracle compatibility with enterprise performance features built upon the legendary open source PostgreSQL platform, all certified on IBM’s latest Linux on Power servers. The highlights of this presentation include: * An overview of the database landscape – past, present and future * Postgres NoSQL capabilities for document and key-value store work loads * How you can lower your Total-Cost-of-Ownership (TCO) with Postgres in conjunction with your current database * What resources are available to assess the right decision * How the IBM Power Systems™ platform is fueling performance, reliability, security, TCO and virtualization for new applications, markets and geographies. * Suggested audience: This presentation is intended for strategic IT and Business Decision-Makers involved in IT infrastructure and application development.

Save money with Postgres on IBM PowerLinux

EDB

Resume

Vamshi Krishna Reddy

Storage Challenges for Production Machine Learning

Nisha Talagala

Multi dimension aggregations using spark and dataframes

Romi Kuntsman

6 GigaSpaces Principles to Survive Black Friday

Ali Hodroj

SAP HANA_class1.pptx

SudhaVukkalkar1

http://www.bigdataspain.org/2014/conference/state-of-play-data-science-on-hadoop-in-2015-keynote Machine Learning is not new. Big Machine Learning is qualitatively different: More data beats algorithm improvement, scale trumps noise and sample size effects, can brute-force manual tasks. Session presented at Big Data Spain 2014 Conference 18th Nov 2014 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Slides: https://speakerdeck.com/bigdataspain/state-of-play-data-science-on-hadoop-in-2015-by-sean-owen-at-big-data-spain-2014

State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...

Big Data Spain

At Linkedin, we have thousands of Hadoop and Spark users ranging from amateurs to experts who run a variety of jobs on our huge 2000-plus node clusters. In just a few years, the number of Hadoop and Spark jobs have grown from hundreds to thousands. With this ever increasing number of users and jobs, it becomes very crucial to have an efficient way to find answers to frequently asked questions like: 1) Why is my job running slow? 2) Why did my job get killed? 3) Can you send me an alert when my job is about to fail or miss SLA? 4) Do we have enough resources on the Hadoop cluster? Having this information available with us will help in quicker debugging, alert based on anomalies, perform root cause analysis(RCA), identify workload patterns and perform capacity planning. To address this problem, we at Linkedin have built a Unified Grid Metrics Platform that captures and stores, current and historical job metrics. In our experience debugging and tuning jobs and interacting with our users, we have learnt a lot of lessons and have been integrating ideas and solutions into this system. For example, we have learned that capturing and storing the complete set of metrics and its history though fascinating is actually rarely useful just like the verbose logs in Spark. We have come up with some derived metrics and curated list of metrics which we track very closely at LinkedIn. In this talk, we will discuss the architecture of how we built this platform for both Hadoop and Spark along with the huge challenges in collecting all the standard, derived and custom user metrics in real-time. We will see how this system allows users to build reporting dashboards, perform trend analysis, dimension analysis and view correlated metrics together.

Conquering Hadoop and Apache Spark with Operational Intelligence with Akshay Rai

Databricks

Similar to Aws education meetup - Large scale data preprocessing with sagemaker - Weiyih (20)

Adding structure to your streaming pipelines: moving from Spark streaming to ...

Challenges of Operationalising Data Science in Production

PradeepDWH

Scaling up with Cisco Big Data: Data + Science = Data Science

Spark One Platform Webinar

Machine learning model to production

Big Data Introduction - Solix empower

Teradata - Architecture of Teradata

Spark Summit EU talk by Kent Buenaventura and Willaim Lau

Introducing new AIOps innovations in Oracle 19c - San Jose AICUG

DataOps with Project Amaterasu

AnilKumarT_Resume_latest

Save money with Postgres on IBM PowerLinux

Resume

Storage Challenges for Production Machine Learning

Multi dimension aggregations using spark and dataframes

6 GigaSpaces Principles to Survive Black Friday

SAP HANA_class1.pptx

State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...

Conquering Hadoop and Apache Spark with Operational Intelligence with Akshay Rai

Recently uploaded

الأمن السيبراني - ما لا يسع للمستخدم جهله

Mohamed Sweelam

Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...

ScyllaDB

At Skynet Technologies, our team of accessibility experts performs automated, semi-automated, and manual audits of websites and web applications as per WCAG 2.2 level AA, ADA, and section 508. Based on evaluations of the accessibility compliance level of the website’s UI, design, source code, navigation, interactive elements, and overall usability, we will provide a digital accessibility evaluation report with in-depth details of potential accessibility barriers and remediation recommendations. Get a manual website WCAG audit (2.0, 2.1, 2.2 level AA) for a small website: 10 pages: $2,500 within 7 business days 30 pages: $7,500 within 14 business days 50 pages: $12,500 within 28 business days For medium websites: 100 pages: $25,000 within 70 business days For larger websites or audits of all pages, please reach out hello@skynettechnologies.com.

Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...

Skynet Technologies

Overview of Hyperledger Foundation

Hyperleger Tokyo Meetup

The presentation was made in “Web3 Fusion: Embracing AI and Beyond” is more than a conference; it's a journey into the heart of digital transformation. The conference a provided a platform where the future of technology meets practical application. This three-day hybrid event, set in the heart of innovation, served as a gateway to the latest trends and transformative discussions in AI, Blockchain, IoT, AR/VR, and their collective impact on the information space.

AI in Action: Real World Use Cases by Anitaraj

AnitaRaj43

Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx

MasterG

Welcome to this session on UiPath Manufacturing and AI. In this session, we will discuss the importance UiPath plays with technology in the manufacturing and we will do an overview of AI and Document Understanding. Please join us to hear from UiPath and Community experts on why you might consider leveraging UiPath in manufacturing. Topics covered Community team overview The importance of UiPath technology for manufacturing UiPath AI and Document Understanding overview What is Document Understanding? DU Process Studio Template Components of DU Framework Q&A Speakers: Sebastian Seutter, Senior Industry Practice Director, UiPath, Inc. Priya Darshini, UiPath MVP RPA Solutions Architect Dzmitry Belanovski, Account Executive, UiPath, Inc.

UiPath manufacturing technology benefits and AI overview

DianaGray10

WebRTC and SIP not just audio and video @ OpenSIPS 2024

Lorenzo Miniero

TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...

marcuskenyatta275

JohnPollard-hybrid-app-RailsConf2024.pptx

JohnPollard37

In the dynamic field of DevOps, the quest for efficiency and productivity is endless. This talk introduces a revolutionary toolkit: Large Language Models (LLMs), including ChatGPT, Gemini, and Claude, extending far beyond traditional coding assistance. We'll explore how LLMs can automate not just code generation, but also transform day-to-day operations such as crafting compelling cover letters for TPS reports, streamlining client communications, and architecting innovative DevOps solutions. Attendees will learn effective prompting strategies and examine real-life use cases, demonstrating LLMs' potential to redefine productivity in the DevOps landscape. Join us to discover how to harness the power of LLMs for a comprehensive productivity boost across your DevOps activities.

ChatGPT and Beyond - Elevating DevOps Productivity

VictorSzoltysek

Event-Driven Architecture Masterclass: Challenges in Stream Processing

ScyllaDB

CORS (Kitworks Team Study 양다윗 발표자료 240510)

Wonjun Hwang

Discover the top CodeIgniter development companies that can elevate your project to new heights. Our blog explores the best firms known for their expertise in CodeIgniter framework development. From robust web applications to scalable solutions, these companies deliver excellence. Whether you're a startup or an enterprise, find the perfect match for your development needs on Top CSS Gallery's blog.

Top 10 CodeIgniter Development Companies

TopCSSGallery

Introduction to use of FHIR Documents in ABDM

Kumar Satyam

Because observability is such a broad topic – and often something we learn on the job – it can feel like there’s too much to learn at once. But you don’t have to tackle everything and can start with the basics and build from there! Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack. No matter what tooling is in place, there are still observability fundamentals that developers should know. That’s why I’ve put together a primer on the different telemetry types, when to use them, how to understand the data journey, and what to look for in time series graphs.

Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)

Paige Cruz

Intro to Passkeys and the State of Passwordless.pptx

FIDO Alliance

Microsoft CSP Briefing Pre-Engagement - Questionnaire

Exakis Nelite

Webinar Recording: https://www.panagenda.com/webinars/alles-neu-macht-der-mai-wir-durchleuchten-den-verbesserten-notes-eigenschaftendialog/ Haben Sie sich schon einmal über den zu kleinen Eigenschaftendialog in Notes geärgert? Mussten Sie einen Agenten oder eine Aktion erstellen, um schnell mal ein Feld zu ändern? Haben Sie jedes mal endlos nach dem zu vergleichenden Feld gesucht, nachdem Sie ein neues Dokument ausgewählt haben? Wollten Sie das verdammte Ding einfach nur größer machen? Zum Glück gibt es dafür eine Lösung – und sie ist wahrscheinlich bereits installiert! Mit dem kostenlosen panagenda Document Properties (Pro) erhalten Sie den Eigenschaftendialog, den Sie schon immer haben wollten. Größer, anpassbar, und im Volltext durchsuchbar. Sehen Sie mehrere Dokumente gleichzeitig oder vergleichen Sie mit einem Diff-Viewer. Ändern Sie beliebige Felder und haben Sie endlich eine einfache Möglichkeit, Profildokumente für alle Benutzer zu verwalten. Entdecken Sie mit HCL Ambassador Marc Thomas, wie Document Properties Ihre Arbeit vereinfachen und Sie bei der täglichen Verwendung von Domino-Anwendungen unterstützen kann – im Client oder im Designer. Sie werden es nicht bereuen! Für Sie in diesem Webinar - Was Document Properties ist, welche Editionen es gibt und wo es in Notes und Domino Designer zu finden ist - Wie Sie nach einem beliebigen Feld suchen und es bearbeiten, Dokumente vergleichen oder alle Daten per CSV exportieren können - Suchen, Bearbeiten und auch Löschen von Profildokumenten - Welche Konfigurationseinstellungen verfügbar sind, um Funktionen anzupassen - Wie Ihre Endbenutzer davon profitieren - Sehen Sie alles in einer Live-Demo

Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...

panagenda

WebAssembly is Key to Better LLM Performance

Samy Fodil

Recently uploaded (20)

الأمن السيبراني - ما لا يسع للمستخدم جهله

Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...

Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...

Overview of Hyperledger Foundation

AI in Action: Real World Use Cases by Anitaraj

Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx

UiPath manufacturing technology benefits and AI overview

WebRTC and SIP not just audio and video @ OpenSIPS 2024

TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...

JohnPollard-hybrid-app-RailsConf2024.pptx

ChatGPT and Beyond - Elevating DevOps Productivity

Event-Driven Architecture Masterclass: Challenges in Stream Processing

CORS (Kitworks Team Study 양다윗 발표자료 240510)

Top 10 CodeIgniter Development Companies

Introduction to use of FHIR Documents in ABDM

Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)

Intro to Passkeys and the State of Passwordless.pptx

Microsoft CSP Briefing Pre-Engagement - Questionnaire

Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...

WebAssembly is Key to Better LLM Performance

Aws education meetup - Large scale data preprocessing with sagemaker - Weiyih

1. About Me – Yap Wei Yih • Senior Data Scientist @ Firemark Labs Singapore (IAG) • NYP Alumni – Specialist Diploma in Business and Big Data Analytics • Master of Science (Electrical Engineering) • linkedin.com/in/yapweiyih • Interest - Applied Data Science, Machine Learning

2. Topic Large scale data pre-processing, model training and deployment using AWS EMR, Athena and SageMaker

3. Key Problems 1. Processing terabytes (billions observation) of geospatial data, Spark cluster setup 2. A model development lifecycle platform 3. Scalable API endpoint, lack of DevOps resources

4. #1 - EMR & Athena Elastic Map Reduce (EMR) ✓ Spin up Spark cluster with just a few clicks ✓ Multi user JupyterHub ✓ Data cleansing and aggregation with Scala/PySpark ✓ Easily configure number of nodes or Autoscaling

5. #1 - EMR & Athena

6. Athena ✓ Support geo - Point, Polygon ✓ Geospatial function – ST_Intersect, ST_Contains, ST_DISTANCE #1 - EMR & Athena

7. Why geospatial function is important?

8. Why geospatial function is important?

9. #2 - SageMaker • Provides full Model Development Lifecycle Management capability • Key requirements that is important for data science work: ✓ Jupyter Notebook, Exploratory Data Analysis (EDA) ✓ Support custom algorithm container ✓ Model Training/Versioning ✓ A/B Testing ✓ Model Endpoint Deployment • Install both R and Python to support two main user groups • Minimize the work of DevOps for model deployment

10. #3 - Lambda & API Gateway Frontend • SageMaker endpoint is only available within AWS services • Lambda and API Gateway is used to expose model to external party • Access by partners is controlled via API key

11. High Level Solution Architecture Frontend Geospatial Data Data Pre-processing Data Ingestion EDA, Modeling Deployment Serving

12. Questions

Aws education meetup - Large scale data preprocessing with sagemaker - Weiyih

Recommended

Recommended

More Related Content

What's hot

What's hot (14)

Similar to Aws education meetup - Large scale data preprocessing with sagemaker - Weiyih

Similar to Aws education meetup - Large scale data preprocessing with sagemaker - Weiyih (20)

Recently uploaded

Recently uploaded (20)

Aws education meetup - Large scale data preprocessing with sagemaker - Weiyih