Aws education meetup - Large scale data preprocessing with sagemaker - Weiyih
•
0 likes•57 views
Yap Wei Yih is a senior data scientist at Firemark Labs Singapore who has a master's degree in electrical engineering. He discusses solving three key problems: 1) processing terabytes of geospatial data using AWS EMR and Athena, 2) a scalable model development lifecycle platform using SageMaker, and 3) exposing a scalable API endpoint using Lambda and API Gateway without devops resources. EMR is used to spin up Spark clusters for data cleansing, aggregation, and geospatial functions on large datasets. SageMaker provides full model lifecycle management with notebooks, training, versioning, testing, and deployment. Lambda and API Gateway expose the SageMaker endpoint to external parties through a controlled API.
Report
Share
Report
Share
Download to read offline
Recommended
Best Practices for Engineering Production-Ready Software with Apache Spark
This document summarizes best practices for engineering production-ready software with Apache Spark. It discusses how data scientists focus on iterative experimentation and model performance while engineers focus on stability, maintainability, and performance. Simply taking prototypes to production can cause issues. The document demonstrates building well-engineered software with Spark that enables both research and experimentation, getting the best of data science and engineering. It emphasizes using practices like modular and testable code, and designing interfaces that allow experimentation on engineered artifacts.
Automate your Machine Learning
Automated machine learning (AutoML) can automate time-consuming tasks in the machine learning lifecycle like data preprocessing, model training, and tuning. This allows data scientists to focus on higher-level work. The presentation demonstrated AutoML on the Titanic dataset in Microsoft Azure Machine Learning service. It showed how AutoML can iterate through various algorithms and hyperparameters, measure model performance, enable model interpretability, facilitate model hosting and drift detection, and support code-based MLOps workflows. AutoML aims to make machine learning more accessible and productive.
Cortana Analytics Workshop: Using the Cortana Analytics Process
Sachin Chouksey, Wee Hyong Tok. In this talk, we will describe a cloud-centric data science process to go from raw data on premise or in the cloud to operationalized machine learning models using only a browser and elastic resources in Azure using various tools like SQL Server on Azure, Azure VMs, Azure Machine Learning, and Azure HDInsight (Hadoop). Go to https://channel9.msdn.com/ to find the recording of this session.
Microsoft DevOps for AI with GoDataDriven
Artificial Intelligence (AI) and machine learning (ML) technologies extend the capabilities of software applications that are now found throughout our daily life: digital assistants, facial recognition, photo captioning, banking services, and product recommendations. The difficult part about integrating AI or ML into an application is not the technology, or the math, or the science or the algorithms. The challenge is getting the model deployed into a production environment and keeping it operational and supportable. Software development teams know how to deliver business applications and cloud services. AI/ML teams know how to develop models that can transform a business. But when it comes to putting the two together to implement an application pipeline specific to AI/ML — to automate it and wrap it around good deployment practices — the process needs some effort to be successful.
Different usages of Machine Learning Open Studio
Machine Learning Open Studio from Activeeon has been designed for different types of users to build machine learning pipelines and meet their profession-specific needs : non-AI engineers, data analysts, data scientists, AI architects.
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
Nielsen used Databricks to test new digital advertising rating methodologies on a large scale. Databricks allowed Nielsen to run analyses on thousands of advertising campaigns using both small panel data and large production data. This identified edge cases and performance gains faster than traditional methods. Using Databricks reduced the time required to test and deploy improved rating methodologies to benefit Nielsen's clients.
Apache Hadoop India Summit 2011 talk "Making Hadoop Enterprise Ready with Am...
Apache Hadoop India Summit 2011 talk "Making Hadoop Enterprise Ready with Am...Yahoo Developer Network
1) Amazon Elastic MapReduce enables customers to easily process vast amounts of data by launching Hadoop clusters across AWS infrastructure.
2) It provides features for managing, monitoring, and debugging Hadoop jobs and clusters without the operational complexities of Hadoop.
3) New features were announced that provide more flexibility for enterprises including expanding and shrinking running clusters, using spot instances to reduce costs, and additional support options.resume
Akhil Katta has experience working as an intern at three tech companies - Actifio, Alpha-Sense, and Semtech Corporation. At each internship, he gained skills in areas like REST APIs, PostgreSQL, Apache Solr, Hadoop, and embedded systems. In school, he achieved a 3.47 GPA as an Electrical Engineering and Computer Science student at UC Berkeley. His relevant projects include creating an interpreter for Scheme in Python, using Hadoop and MapReduce to solve Connect 4, and designing a functional CPU in Logisim. He has taken classes relevant to algorithms, AI, security, machine learning, and more.
Recommended
Best Practices for Engineering Production-Ready Software with Apache Spark
This document summarizes best practices for engineering production-ready software with Apache Spark. It discusses how data scientists focus on iterative experimentation and model performance while engineers focus on stability, maintainability, and performance. Simply taking prototypes to production can cause issues. The document demonstrates building well-engineered software with Spark that enables both research and experimentation, getting the best of data science and engineering. It emphasizes using practices like modular and testable code, and designing interfaces that allow experimentation on engineered artifacts.
Automate your Machine Learning
Automated machine learning (AutoML) can automate time-consuming tasks in the machine learning lifecycle like data preprocessing, model training, and tuning. This allows data scientists to focus on higher-level work. The presentation demonstrated AutoML on the Titanic dataset in Microsoft Azure Machine Learning service. It showed how AutoML can iterate through various algorithms and hyperparameters, measure model performance, enable model interpretability, facilitate model hosting and drift detection, and support code-based MLOps workflows. AutoML aims to make machine learning more accessible and productive.
Cortana Analytics Workshop: Using the Cortana Analytics Process
Sachin Chouksey, Wee Hyong Tok. In this talk, we will describe a cloud-centric data science process to go from raw data on premise or in the cloud to operationalized machine learning models using only a browser and elastic resources in Azure using various tools like SQL Server on Azure, Azure VMs, Azure Machine Learning, and Azure HDInsight (Hadoop). Go to https://channel9.msdn.com/ to find the recording of this session.
Microsoft DevOps for AI with GoDataDriven
Artificial Intelligence (AI) and machine learning (ML) technologies extend the capabilities of software applications that are now found throughout our daily life: digital assistants, facial recognition, photo captioning, banking services, and product recommendations. The difficult part about integrating AI or ML into an application is not the technology, or the math, or the science or the algorithms. The challenge is getting the model deployed into a production environment and keeping it operational and supportable. Software development teams know how to deliver business applications and cloud services. AI/ML teams know how to develop models that can transform a business. But when it comes to putting the two together to implement an application pipeline specific to AI/ML — to automate it and wrap it around good deployment practices — the process needs some effort to be successful.
Different usages of Machine Learning Open Studio
Machine Learning Open Studio from Activeeon has been designed for different types of users to build machine learning pipelines and meet their profession-specific needs : non-AI engineers, data analysts, data scientists, AI architects.
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
Nielsen used Databricks to test new digital advertising rating methodologies on a large scale. Databricks allowed Nielsen to run analyses on thousands of advertising campaigns using both small panel data and large production data. This identified edge cases and performance gains faster than traditional methods. Using Databricks reduced the time required to test and deploy improved rating methodologies to benefit Nielsen's clients.
Apache Hadoop India Summit 2011 talk "Making Hadoop Enterprise Ready with Am...
Apache Hadoop India Summit 2011 talk "Making Hadoop Enterprise Ready with Am...Yahoo Developer Network
1) Amazon Elastic MapReduce enables customers to easily process vast amounts of data by launching Hadoop clusters across AWS infrastructure.
2) It provides features for managing, monitoring, and debugging Hadoop jobs and clusters without the operational complexities of Hadoop.
3) New features were announced that provide more flexibility for enterprises including expanding and shrinking running clusters, using spot instances to reduce costs, and additional support options.resume
Akhil Katta has experience working as an intern at three tech companies - Actifio, Alpha-Sense, and Semtech Corporation. At each internship, he gained skills in areas like REST APIs, PostgreSQL, Apache Solr, Hadoop, and embedded systems. In school, he achieved a 3.47 GPA as an Electrical Engineering and Computer Science student at UC Berkeley. His relevant projects include creating an interpreter for Scheme in Python, using Hadoop and MapReduce to solve Connect 4, and designing a functional CPU in Logisim. He has taken classes relevant to algorithms, AI, security, machine learning, and more.
16th Athens Big Data Meetup - 1st Talk - An Introduction to Machine Learning ...
Title: An Introduction to Machine Learning with Python and Scikit-Learn
Speaker: Julien Simon (https://linkedin.com/in/juliensimon/)
Date: Thursday, March 14, 2019
Event: https://www.meetup.com/Athens-Big-Data/events/259091496/
Saurav Sengupta Resume
Saurav Sengupta is a data scientist with a Master's degree from the University of Virginia. He has experience as an associate consultant at Capgemini developing REST APIs and deploying code to cloud platforms. He also interned at STMicroelectronics designing embedded neural networks. His technical skills include Python, R, Java, C, databases, and machine learning packages. His projects include classifying biopsy images using CNNs achieving 98% accuracy and classifying music genres using ensemble models.
Scalable Machine Learning using R and Azure HDInsight - Parashar
How to build a scalable machine learning-based data processing pipeline using Microsoft R Server with Apache Spark on Azure HDInsight.
Large-Scaled Insurance Analytics Using Tweedie Models in Apache Spark with Ya...
This document summarizes research on using Bayesian hierarchical models and distributed computing on Apache Spark to perform large-scale insurance analytics on trip-level driving data. It addresses challenges like big data volumes, extreme sparsity of claims events, and dependency in correlated driving data. The approach enables efficient Bayesian computation that is 65 times faster than standard methods. An empirical analysis fits a Bayesian hierarchical Tweedie model to a dataset of 13.2 million auto insurance records to demonstrate scalability. The distributed Bayesian simulation approach enables practical Bayesian analysis of extremely large insurance datasets.
Moving a Fraud-Fighting Random Forest from scikit-learn to Spark with MLlib, ...
This talk describes migrating a large random forest classifier from scikit-learn to Spark's MLlib. We cut training time from 2 days to 2 hours, reduced failed runs, and track experiments better with MLflow. Kount provides certainty in digital interactions like online credit card transactions. One of our scores uses a random forest classifier with 250 trees and 100,000 nodes per tree. We used scikit-learn to train using 60 million samples that each contained over 150 features. The in-memory requirements exceeded 750 GB, took 2 days, and were not robust to disruption in our database or training execution. To migrate workflow to Spark, we built a 6-node cluster with HDFS. This provides 1.35 TB of RAM and 484 cores. Using MLlib and parallelization, the training time for our random forests are now less than 2 hours. Training data stays in our production environment, which used to require a deploy cycle to move locally-developed code onto our training server. The new implementation uses Jupyter notebooks for remote development with server-side execution. MLflow tracks all input parameters, code, and git revision number, while the performance and model itself are retained as experiment artifacts. The new workflow is robust to service disruption. Our training pipeline begins by pulling from a Vertica database. Originally, this single connection took over 8 hours to complete with any problem causing a restart. Using sqoop and multiple connections, we pull the data in 45 minutes. The old technique used volatile storage and required the data for each experiment. Now, we pull the data from Vertica one time and then reload much faster from HDFS. While a significant undertaking, moving to the Spark ecosystem converted an ad hoc and hands-on training process into a fully repeatable pipeline that meets regulatory and business goals for traceability and speed.
Speaker: Josh Johnston
"Something worth ten Microsofts" - Azure Machine Learning 101
Azure Machine learning is a fully managed cloud service that enables you to easily build, deploy, and share predictive analytics solutions. Here's a jump start for Azure ML.
Scaling ML-Based Threat Detection For Production Cyber Attacks
The document discusses best practices for integrating machine learning models into production pipelines. It describes the full data science product lifecycle from identifying business needs to deploying models through APIs. Key aspects covered include maintainable code through functions/classes, unit testing and code reviews, using Jenkins and a tool called Apparate to schedule Spark jobs and automatically update libraries, and deploying APIs on Kubernetes through Spinnaker for continuous delivery. Lessons learned emphasize leveraging existing tools and infrastructure while addressing pain points to streamline the end-to-end process.
Resume chao han_tsai
Chao-Han Tsai is seeking a full-time software engineer position and has relevant work experience. He interned at IBM where he developed a web application in Ruby on Rails and implemented file transfer logic. Tsai has a Master's degree from the University of Michigan and a Bachelor's from National Chiao-Tung University, where he graduated with academic achievement. His skills include languages like Python, Ruby, and Java as well as frameworks like Rails and Spark.
Denver AWS Users' Group meeting - September 2017
Great presentation on Athena, QuickSight, S3 and related technologies by AWS Solution Architect Ben Snively.
Computation of spatial data on Hadoop Cluster
This document provides an overview of distributed computation on spatial data using Hadoop. It discusses GIS background including common spatial data types and operations. It then introduces Hadoop and MapReduce, describing HDFS architecture, MapReduce programming model, and how Hadoop compares to distributed databases. The document outlines experiments on Hadoop including speedup from spatial joins and optimization, building R-tree indexes, nearest neighbor queries, and graph computations. It concludes with discussing results and outlines future work.
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Have you imagined a simple machine learning solution able to prevent revenue leakage and monitor your distributed application? To answer this question, we offer a practical and a simple machine learning solution to create an intelligent monitoring application based on simple data analysis using Apache Spark MLlib. Our application uses linear regression models to make predictions and check if the platform is experiencing any operational problems that can impact in revenue losses. The application monitor distributed systems and provides notifications stating the problem detected, that way users can operate quickly to avoid serious problems which directly impact the company’s revenue and reduce the time for action. We will present an architecture for not only a monitoring system, but also an active actor for our outages recoveries. At the end of the presentation you will have access to our training program source code and you will be able to adapt and implement in your company. This solution already helped to prevent about US$3mi in losses last year.
Data cleansing and data prep with synapse data flows
This document contains links to resources about using Azure Synapse Analytics for data cleansing and preparation with Data Flows. It includes links to videos and documentation about removing null values, saving data profiler summary statistics, and using metadata functions in Azure Data Factory data flows.
Tactical Data Science Tips: Python and Spark Together
This document summarizes a talk given by Bill Chambers on processing data with Spark and Python. It discusses 5 ways to process data: RDDs, DataFrames, Koalas, UDFs, and pandasUDFs. It then covers two data science use cases - growth forecasting and churn prediction - and how they were implemented using these different processing methods based on characteristics like the number of input rows, features, and required models. The talk emphasizes using DataFrames and pandasUDFs for optimal performance and flexibility. It also highlights tracking models with MLFlow for consistency in production.
Spark Summit EU: IBM Keynote
Weren't at Spark Summit Europe? Have no fear, you can now view IBM's Keynote by Anjul Bhambhri, VP Product Development, IBM. This presentation covers IBM's advancements in bringing Spark mainstream. Take a look and enjoy.
Adding structure to your streaming pipelines: moving from Spark streaming to ...
How do you go from a strictly typed object-based streaming pipeline with simple operations to a structured streaming pipeline with higher order complex relational operations? This is what the Data Engineering team did at GoPro to scale up the development of streaming pipelines for the rapidly growing number of devices and applications.
When big data frameworks such as Hadoop first came to exist, developers were happy because we could finally process large amounts of data without writing complex multi-threaded code or worse yet writing complicated distributed code. Unfortunately, only very simple operations were available such as map and reduce. Almost immediately, higher level operations were desired similar to relational operations. And so Hive and dozens (hundreds?) of SQL-based big data tools became available for more developer-efficient batch processing of massive amounts of data.
In recent years, big data has moved from batch processing to stream-based processing since no one wants to wait hours or days to gain insights. Dozens of stream processing frameworks exist today and the same trend that occurred in the batch-based big data processing realm has taken place in the streaming world, so that nearly every streaming framework now supports higher level relational operations.
In this talk, we will discuss in a very hands-on manner how the streaming data pipelines for GoPro devices and apps have moved from the original Spark streaming with its simple RDD-based operations in Spark 1.x to Spark's structured streaming with its higher level relational operations in Spark 2.x. We will talk about the differences, advantages, and necessary pain points that must be addressed in order to scale relational-based streaming pipelines for massive IoT streams. We will also talk about moving from “hand built” Hadoop/Spark clusters running in the cloud to using a Spark-based cloud service. DAVID WINTERS, Big Data Architect, GoPro and HAO ZOU, Senior Software Engineer, GoPro
Challenges of Operationalising Data Science in Production
The presentation topic for this meet-up was covered in two sections without any breaks in-between
Section 1: Business Aspects (20 mins)
Speaker: Rasmi Mohapatra, Product Owner, Experian
https://www.linkedin.com/in/rasmi-m-428b3a46/
Once your data science application is in the production, there are many typical data science operational challenges experienced today - across business domains - we will cover a few challenges with example scenarios
Section 2: Tech Aspects (40 mins, slides & demo, Q&A )
Speaker: Santanu Dey, Solution Architect, Iguazio
https://www.linkedin.com/in/santanu/
In this part of the talk, we will cover how these operational challenges can be overcome e.g. automating data collection & preparation, making ML models portable & deploying in production, monitoring and scaling, etc.
with relevant demos.
PradeepDWH
Pradeep Kumar Pandey has over 10 years of experience as a data/systems integration specialist and ETL expert. He has extensive experience designing and implementing data warehouses using tools like IBM DataStage, Informatica, Oracle OBIEE, and Oracle OBIA. He has led teams and taken on roles such as developer, technical lead, and team lead. Pradeep has worked on projects across various industries including telecom, financial services, HR, and retail.
Scaling up with Cisco Big Data: Data + Science = Data Science
Discover how IoT data is ingested with real-time technologies such as Apache Spark and analysed using Data Science techniques to glean insights.
Spark One Platform Webinar
This document discusses Cloudera's initiative to make Spark the standard execution engine for Hadoop. It outlines how Spark improves on MapReduce by leveraging distributed memory and having a simpler developer experience. It also describes Cloudera's investments in areas like management, security, scale, and streaming to further Spark's capabilities and make it production-ready. The goal is for Spark to replace MapReduce as the execution engine and for specialized engines like Impala to handle specific workloads, with all sharing the same data, metadata, resource management, and other platform services.
Machine learning model to production
This document discusses moving machine learning models from prototype to production. It outlines some common problems with the current workflow where moving to production often requires redevelopment from scratch. Some proposed solutions include using notebooks as APIs and developing analytics that are accessed via an API. It also discusses different data science platforms and architectures for building end-to-end machine learning systems, focusing on flexibility, security, testing and scalability for production environments. The document recommends a custom backend integrated with Spark via APIs as the best approach for the current project.
Big Data Introduction - Solix empower
This document provides an overview of big data and the Spark framework. It discusses the big data ecosystem, including file systems, data ingestion tools, batch and real-time data processing frameworks, visualization tools, and support technologies. It outlines common big data job roles and their associated skills. The document then focuses on Spark, describing its core functionality, modules like DataFrames and MLlib, and execution modes. It provides guidance on learning Spark, emphasizing programming skills and Spark APIs. A demo of Spark fundamentals on a big data lab is also proposed.
Teradata - Architecture of Teradata
Parsing Engine(PE),
Request and Response Parcel,
Access Module Processors (AMPs),
Bynet,
Data access Handling,
TD Config Utilities,
Config and Reconfig,
More Related Content
What's hot
16th Athens Big Data Meetup - 1st Talk - An Introduction to Machine Learning ...
Title: An Introduction to Machine Learning with Python and Scikit-Learn
Speaker: Julien Simon (https://linkedin.com/in/juliensimon/)
Date: Thursday, March 14, 2019
Event: https://www.meetup.com/Athens-Big-Data/events/259091496/
Saurav Sengupta Resume
Saurav Sengupta is a data scientist with a Master's degree from the University of Virginia. He has experience as an associate consultant at Capgemini developing REST APIs and deploying code to cloud platforms. He also interned at STMicroelectronics designing embedded neural networks. His technical skills include Python, R, Java, C, databases, and machine learning packages. His projects include classifying biopsy images using CNNs achieving 98% accuracy and classifying music genres using ensemble models.
Scalable Machine Learning using R and Azure HDInsight - Parashar
How to build a scalable machine learning-based data processing pipeline using Microsoft R Server with Apache Spark on Azure HDInsight.
Large-Scaled Insurance Analytics Using Tweedie Models in Apache Spark with Ya...
This document summarizes research on using Bayesian hierarchical models and distributed computing on Apache Spark to perform large-scale insurance analytics on trip-level driving data. It addresses challenges like big data volumes, extreme sparsity of claims events, and dependency in correlated driving data. The approach enables efficient Bayesian computation that is 65 times faster than standard methods. An empirical analysis fits a Bayesian hierarchical Tweedie model to a dataset of 13.2 million auto insurance records to demonstrate scalability. The distributed Bayesian simulation approach enables practical Bayesian analysis of extremely large insurance datasets.
Moving a Fraud-Fighting Random Forest from scikit-learn to Spark with MLlib, ...
This talk describes migrating a large random forest classifier from scikit-learn to Spark's MLlib. We cut training time from 2 days to 2 hours, reduced failed runs, and track experiments better with MLflow. Kount provides certainty in digital interactions like online credit card transactions. One of our scores uses a random forest classifier with 250 trees and 100,000 nodes per tree. We used scikit-learn to train using 60 million samples that each contained over 150 features. The in-memory requirements exceeded 750 GB, took 2 days, and were not robust to disruption in our database or training execution. To migrate workflow to Spark, we built a 6-node cluster with HDFS. This provides 1.35 TB of RAM and 484 cores. Using MLlib and parallelization, the training time for our random forests are now less than 2 hours. Training data stays in our production environment, which used to require a deploy cycle to move locally-developed code onto our training server. The new implementation uses Jupyter notebooks for remote development with server-side execution. MLflow tracks all input parameters, code, and git revision number, while the performance and model itself are retained as experiment artifacts. The new workflow is robust to service disruption. Our training pipeline begins by pulling from a Vertica database. Originally, this single connection took over 8 hours to complete with any problem causing a restart. Using sqoop and multiple connections, we pull the data in 45 minutes. The old technique used volatile storage and required the data for each experiment. Now, we pull the data from Vertica one time and then reload much faster from HDFS. While a significant undertaking, moving to the Spark ecosystem converted an ad hoc and hands-on training process into a fully repeatable pipeline that meets regulatory and business goals for traceability and speed.
Speaker: Josh Johnston
"Something worth ten Microsofts" - Azure Machine Learning 101
Azure Machine learning is a fully managed cloud service that enables you to easily build, deploy, and share predictive analytics solutions. Here's a jump start for Azure ML.
Scaling ML-Based Threat Detection For Production Cyber Attacks
The document discusses best practices for integrating machine learning models into production pipelines. It describes the full data science product lifecycle from identifying business needs to deploying models through APIs. Key aspects covered include maintainable code through functions/classes, unit testing and code reviews, using Jenkins and a tool called Apparate to schedule Spark jobs and automatically update libraries, and deploying APIs on Kubernetes through Spinnaker for continuous delivery. Lessons learned emphasize leveraging existing tools and infrastructure while addressing pain points to streamline the end-to-end process.
Resume chao han_tsai
Chao-Han Tsai is seeking a full-time software engineer position and has relevant work experience. He interned at IBM where he developed a web application in Ruby on Rails and implemented file transfer logic. Tsai has a Master's degree from the University of Michigan and a Bachelor's from National Chiao-Tung University, where he graduated with academic achievement. His skills include languages like Python, Ruby, and Java as well as frameworks like Rails and Spark.
Denver AWS Users' Group meeting - September 2017
Great presentation on Athena, QuickSight, S3 and related technologies by AWS Solution Architect Ben Snively.
Computation of spatial data on Hadoop Cluster
This document provides an overview of distributed computation on spatial data using Hadoop. It discusses GIS background including common spatial data types and operations. It then introduces Hadoop and MapReduce, describing HDFS architecture, MapReduce programming model, and how Hadoop compares to distributed databases. The document outlines experiments on Hadoop including speedup from spatial joins and optimization, building R-tree indexes, nearest neighbor queries, and graph computations. It concludes with discussing results and outlines future work.
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Have you imagined a simple machine learning solution able to prevent revenue leakage and monitor your distributed application? To answer this question, we offer a practical and a simple machine learning solution to create an intelligent monitoring application based on simple data analysis using Apache Spark MLlib. Our application uses linear regression models to make predictions and check if the platform is experiencing any operational problems that can impact in revenue losses. The application monitor distributed systems and provides notifications stating the problem detected, that way users can operate quickly to avoid serious problems which directly impact the company’s revenue and reduce the time for action. We will present an architecture for not only a monitoring system, but also an active actor for our outages recoveries. At the end of the presentation you will have access to our training program source code and you will be able to adapt and implement in your company. This solution already helped to prevent about US$3mi in losses last year.
Data cleansing and data prep with synapse data flows
This document contains links to resources about using Azure Synapse Analytics for data cleansing and preparation with Data Flows. It includes links to videos and documentation about removing null values, saving data profiler summary statistics, and using metadata functions in Azure Data Factory data flows.
Tactical Data Science Tips: Python and Spark Together
This document summarizes a talk given by Bill Chambers on processing data with Spark and Python. It discusses 5 ways to process data: RDDs, DataFrames, Koalas, UDFs, and pandasUDFs. It then covers two data science use cases - growth forecasting and churn prediction - and how they were implemented using these different processing methods based on characteristics like the number of input rows, features, and required models. The talk emphasizes using DataFrames and pandasUDFs for optimal performance and flexibility. It also highlights tracking models with MLFlow for consistency in production.
Spark Summit EU: IBM Keynote
Weren't at Spark Summit Europe? Have no fear, you can now view IBM's Keynote by Anjul Bhambhri, VP Product Development, IBM. This presentation covers IBM's advancements in bringing Spark mainstream. Take a look and enjoy.
What's hot (14)
16th Athens Big Data Meetup - 1st Talk - An Introduction to Machine Learning ...
16th Athens Big Data Meetup - 1st Talk - An Introduction to Machine Learning ...
Scalable Machine Learning using R and Azure HDInsight - Parashar
Scalable Machine Learning using R and Azure HDInsight - Parashar
Large-Scaled Insurance Analytics Using Tweedie Models in Apache Spark with Ya...
Large-Scaled Insurance Analytics Using Tweedie Models in Apache Spark with Ya...
Moving a Fraud-Fighting Random Forest from scikit-learn to Spark with MLlib, ...
Moving a Fraud-Fighting Random Forest from scikit-learn to Spark with MLlib, ...
"Something worth ten Microsofts" - Azure Machine Learning 101
"Something worth ten Microsofts" - Azure Machine Learning 101
Scaling ML-Based Threat Detection For Production Cyber Attacks
Scaling ML-Based Threat Detection For Production Cyber Attacks
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Data cleansing and data prep with synapse data flows
Data cleansing and data prep with synapse data flows
Tactical Data Science Tips: Python and Spark Together
Tactical Data Science Tips: Python and Spark Together
Similar to Aws education meetup - Large scale data preprocessing with sagemaker - Weiyih
Adding structure to your streaming pipelines: moving from Spark streaming to ...
How do you go from a strictly typed object-based streaming pipeline with simple operations to a structured streaming pipeline with higher order complex relational operations? This is what the Data Engineering team did at GoPro to scale up the development of streaming pipelines for the rapidly growing number of devices and applications.
When big data frameworks such as Hadoop first came to exist, developers were happy because we could finally process large amounts of data without writing complex multi-threaded code or worse yet writing complicated distributed code. Unfortunately, only very simple operations were available such as map and reduce. Almost immediately, higher level operations were desired similar to relational operations. And so Hive and dozens (hundreds?) of SQL-based big data tools became available for more developer-efficient batch processing of massive amounts of data.
In recent years, big data has moved from batch processing to stream-based processing since no one wants to wait hours or days to gain insights. Dozens of stream processing frameworks exist today and the same trend that occurred in the batch-based big data processing realm has taken place in the streaming world, so that nearly every streaming framework now supports higher level relational operations.
In this talk, we will discuss in a very hands-on manner how the streaming data pipelines for GoPro devices and apps have moved from the original Spark streaming with its simple RDD-based operations in Spark 1.x to Spark's structured streaming with its higher level relational operations in Spark 2.x. We will talk about the differences, advantages, and necessary pain points that must be addressed in order to scale relational-based streaming pipelines for massive IoT streams. We will also talk about moving from “hand built” Hadoop/Spark clusters running in the cloud to using a Spark-based cloud service. DAVID WINTERS, Big Data Architect, GoPro and HAO ZOU, Senior Software Engineer, GoPro
Challenges of Operationalising Data Science in Production
The presentation topic for this meet-up was covered in two sections without any breaks in-between
Section 1: Business Aspects (20 mins)
Speaker: Rasmi Mohapatra, Product Owner, Experian
https://www.linkedin.com/in/rasmi-m-428b3a46/
Once your data science application is in the production, there are many typical data science operational challenges experienced today - across business domains - we will cover a few challenges with example scenarios
Section 2: Tech Aspects (40 mins, slides & demo, Q&A )
Speaker: Santanu Dey, Solution Architect, Iguazio
https://www.linkedin.com/in/santanu/
In this part of the talk, we will cover how these operational challenges can be overcome e.g. automating data collection & preparation, making ML models portable & deploying in production, monitoring and scaling, etc.
with relevant demos.
PradeepDWH
Pradeep Kumar Pandey has over 10 years of experience as a data/systems integration specialist and ETL expert. He has extensive experience designing and implementing data warehouses using tools like IBM DataStage, Informatica, Oracle OBIEE, and Oracle OBIA. He has led teams and taken on roles such as developer, technical lead, and team lead. Pradeep has worked on projects across various industries including telecom, financial services, HR, and retail.
Scaling up with Cisco Big Data: Data + Science = Data Science
Discover how IoT data is ingested with real-time technologies such as Apache Spark and analysed using Data Science techniques to glean insights.
Spark One Platform Webinar
This document discusses Cloudera's initiative to make Spark the standard execution engine for Hadoop. It outlines how Spark improves on MapReduce by leveraging distributed memory and having a simpler developer experience. It also describes Cloudera's investments in areas like management, security, scale, and streaming to further Spark's capabilities and make it production-ready. The goal is for Spark to replace MapReduce as the execution engine and for specialized engines like Impala to handle specific workloads, with all sharing the same data, metadata, resource management, and other platform services.
Machine learning model to production
This document discusses moving machine learning models from prototype to production. It outlines some common problems with the current workflow where moving to production often requires redevelopment from scratch. Some proposed solutions include using notebooks as APIs and developing analytics that are accessed via an API. It also discusses different data science platforms and architectures for building end-to-end machine learning systems, focusing on flexibility, security, testing and scalability for production environments. The document recommends a custom backend integrated with Spark via APIs as the best approach for the current project.
Big Data Introduction - Solix empower
This document provides an overview of big data and the Spark framework. It discusses the big data ecosystem, including file systems, data ingestion tools, batch and real-time data processing frameworks, visualization tools, and support technologies. It outlines common big data job roles and their associated skills. The document then focuses on Spark, describing its core functionality, modules like DataFrames and MLlib, and execution modes. It provides guidance on learning Spark, emphasizing programming skills and Spark APIs. A demo of Spark fundamentals on a big data lab is also proposed.
Teradata - Architecture of Teradata
Parsing Engine(PE),
Request and Response Parcel,
Access Module Processors (AMPs),
Bynet,
Data access Handling,
TD Config Utilities,
Config and Reconfig,
Spark Summit EU talk by Kent Buenaventura and Willaim Lau
This document summarizes Unity Technologies' journey migrating their data pipeline from a legacy Hive-based system to using Spark. Some key points:
- They moved to Spark for its scaling, performance, and ability to handle both batch and streaming workloads from a single stack.
- The new Spark-based pipeline uses Airflow for workflow management and saves processed data to Parquet files stored in S3 for backup.
- Taking a test-driven development approach with unit and integration tests helped ensure a smooth migration. Staging the pipeline in an environment similar to production also helped address issues early.
- The new Spark pipeline completed analysis stages up to 2x faster than the previous Hive-based system and
Introducing new AIOps innovations in Oracle 19c - San Jose AICUG
The document discusses Oracle's use of machine learning for autonomous database services. It provides an overview of Oracle's machine learning tools and algorithms, including how they are used for applications like anomaly detection and predictive queries. It also describes Oracle's machine learning notebook interface within the Autonomous Data Warehouse Cloud for building and deploying machine learning models.
DataOps with Project Amaterasu
This document discusses Project Amaterasu, a tool for simplifying the deployment of big data applications. Amaterasu uses Mesos to deploy Spark jobs and other frameworks across clusters. It defines workflows, actions, and environments in YAML and JSON files. Workflows contain a series of actions like Spark jobs. Actions are written in Scala and interface with Amaterasu's context. Environments configure settings for different clusters. Amaterasu aims to improve collaboration and testing for big data teams through continuous integration and deployment of data pipelines.
AnilKumarT_Resume_latest
Anil Kumar Thyagarajan is a senior software engineer with over 15 years of experience in areas like big data analytics, cloud computing, payment gateways, and supply chain products. He is currently a senior data engineer at Microsoft working on their Azure HDInsight platform. Previously he held roles at Nokia, Yahoo, and AOL where he led teams and worked on projects involving Hadoop, Amazon Web Services, data migration, monitoring tools, and distributed systems. He has expertise in technologies like Perl, Java, Python, Linux, Hadoop, Spark, and Amazon Web Services.
Save money with Postgres on IBM PowerLinux
EnterpriseDB Postgres Plus Advanced Server provides Oracle compatibility with enterprise performance features built upon the legendary open source PostgreSQL platform, all certified on IBM’s latest Linux on Power servers.
The highlights of this presentation include:
* An overview of the database landscape – past, present and future
* Postgres NoSQL capabilities for document and key-value store work loads
* How you can lower your Total-Cost-of-Ownership (TCO) with Postgres in conjunction with your current database
* What resources are available to assess the right decision
* How the IBM Power Systems™ platform is fueling performance, reliability, security, TCO and virtualization for new applications, markets and geographies.
* Suggested audience: This presentation is intended for strategic IT and Business Decision-Makers involved in IT infrastructure and application development.
Resume
Vamshi Krishna Reddy has over 7 months of experience using Informatica Power Center and Teradata for data warehousing projects. He has strong skills in ETL development, data modeling, and debugging Teradata utilities. He has extensive experience developing complex mappings from varied data sources and transformations for data loading and analytics.
Storage Challenges for Production Machine Learning
Intersection of Storage and Machine Learning. The storage problems that AI/ML faces and how to use AI/ML to improve storage
Multi dimension aggregations using spark and dataframes
How Totango uses Apache Spark DataFrames to perform hundreds of aggregations in scale for Customer Success analytics
6 GigaSpaces Principles to Survive Black Friday
GigaSpaces Professional Services July 2014 E-Commerce Customer Summit presentation on architecting for black friday traffic.
SAP HANA_class1.pptx
This document provides an overview of an SAP HANA training course. The course will cover topics such as SAP HANA modeling, development, ETL processes, and big data environments. Students will learn HANA concepts and tools like the HANA studio. They will also practice tasks like modeling, querying, security, and provisioning data. The course aims to prepare students to work with HANA systems and answer related interview questions. It consists of 14 modules taught over several hours and includes hands-on assignments.
State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...
http://www.bigdataspain.org/2014/conference/state-of-play-data-science-on-hadoop-in-2015-keynote
Machine Learning is not new. Big Machine Learning is qualitatively different: More data beats algorithm improvement, scale trumps noise and sample size effects, can brute-force manual tasks.
Session presented at Big Data Spain 2014 Conference
18th Nov 2014
Kinépolis Madrid
http://www.bigdataspain.org
Event promoted by: http://www.paradigmatecnologico.com
Slides: https://speakerdeck.com/bigdataspain/state-of-play-data-science-on-hadoop-in-2015-by-sean-owen-at-big-data-spain-2014
Conquering Hadoop and Apache Spark with Operational Intelligence with Akshay Rai
At Linkedin, we have thousands of Hadoop and Spark users ranging from amateurs to experts who run a variety of jobs on our huge 2000-plus node clusters. In just a few years, the number of Hadoop and Spark jobs have grown from hundreds to thousands. With this ever increasing number of users and jobs, it becomes very crucial to have an efficient way to find answers to frequently asked questions like:
1) Why is my job running slow?
2) Why did my job get killed?
3) Can you send me an alert when my job is about to fail or miss SLA?
4) Do we have enough resources on the Hadoop cluster?
Having this information available with us will help in quicker debugging, alert based on anomalies, perform root cause analysis(RCA), identify workload patterns and perform capacity planning. To address this problem, we at Linkedin have built a Unified Grid Metrics Platform that captures and stores, current and historical job metrics. In our experience debugging and tuning jobs and interacting with our users, we have learnt a lot of lessons and have been integrating ideas and solutions into this system. For example, we have learned that capturing and storing the complete set of metrics and its history though fascinating is actually rarely useful just like the verbose logs in Spark. We have come up with some derived metrics and curated list of metrics which we track very closely at LinkedIn.
In this talk, we will discuss the architecture of how we built this platform for both Hadoop and Spark along with the huge challenges in collecting all the standard, derived and custom user metrics in real-time. We will see how this system allows users to build reporting dashboards, perform trend analysis, dimension analysis and view correlated metrics together.
Similar to Aws education meetup - Large scale data preprocessing with sagemaker - Weiyih (20)
Adding structure to your streaming pipelines: moving from Spark streaming to ...
Adding structure to your streaming pipelines: moving from Spark streaming to ...
Challenges of Operationalising Data Science in Production
Challenges of Operationalising Data Science in Production
Scaling up with Cisco Big Data: Data + Science = Data Science
Scaling up with Cisco Big Data: Data + Science = Data Science
Spark Summit EU talk by Kent Buenaventura and Willaim Lau
Spark Summit EU talk by Kent Buenaventura and Willaim Lau
Introducing new AIOps innovations in Oracle 19c - San Jose AICUG
Introducing new AIOps innovations in Oracle 19c - San Jose AICUG
Storage Challenges for Production Machine Learning
Storage Challenges for Production Machine Learning
Multi dimension aggregations using spark and dataframes
Multi dimension aggregations using spark and dataframes
State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...
State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...
Conquering Hadoop and Apache Spark with Operational Intelligence with Akshay Rai
Conquering Hadoop and Apache Spark with Operational Intelligence with Akshay Rai
Recently uploaded
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Removing Uninteresting Bytes in Software Fuzzing
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Communications Mining Series - Zero to Hero - Session 1
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
HCL Notes and Domino License Cost Reduction in the World of DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
National Security Agency - NSA mobile device best practices
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
20240609 QFM020 Irresponsible AI Reading List May 2024
Everything I found interesting about the irresponsible use of machine intelligence in May 2024
Building Production Ready Search Pipelines with Spark and Milvus
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
20240605 QFM017 Machine Intelligence Reading List May 2024
Everything I found interesting about machines behaving intelligently during May 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Imagine a world where machines not only perform tasks but also learn, adapt, and make decisions. This is the promise of Artificial Intelligence (AI), a technology that's not just enhancing our lives but revolutionizing entire industries.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
Infrastructure Challenges in Scaling RAG with Custom AI models
Building Retrieval-Augmented Generation (RAG) systems with open-source and custom AI models is a complex task. This talk explores the challenges in productionizing RAG systems, including retrieval performance, response synthesis, and evaluation. We’ll discuss how to leverage open-source models like text embeddings, language models, and custom fine-tuned models to enhance RAG performance. Additionally, we’ll cover how BentoML can help orchestrate and scale these AI components efficiently, ensuring seamless deployment and management of RAG systems in the cloud.
Recently uploaded (20)
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Infrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI models
Aws education meetup - Large scale data preprocessing with sagemaker - Weiyih
- 1. About Me – Yap Wei Yih • Senior Data Scientist @ Firemark Labs Singapore (IAG) • NYP Alumni – Specialist Diploma in Business and Big Data Analytics • Master of Science (Electrical Engineering) • linkedin.com/in/yapweiyih • Interest - Applied Data Science, Machine Learning
- 2. Topic Large scale data pre-processing, model training and deployment using AWS EMR, Athena and SageMaker
- 3. Key Problems 1. Processing terabytes (billions observation) of geospatial data, Spark cluster setup 2. A model development lifecycle platform 3. Scalable API endpoint, lack of DevOps resources
- 4. #1 - EMR & Athena Elastic Map Reduce (EMR) ✓ Spin up Spark cluster with just a few clicks ✓ Multi user JupyterHub ✓ Data cleansing and aggregation with Scala/PySpark ✓ Easily configure number of nodes or Autoscaling
- 5. #1 - EMR & Athena
- 6. Athena ✓ Support geo - Point, Polygon ✓ Geospatial function – ST_Intersect, ST_Contains, ST_DISTANCE #1 - EMR & Athena
- 7. Why geospatial function is important?
- 8. Why geospatial function is important?
- 9. #2 - SageMaker • Provides full Model Development Lifecycle Management capability • Key requirements that is important for data science work: ✓ Jupyter Notebook, Exploratory Data Analysis (EDA) ✓ Support custom algorithm container ✓ Model Training/Versioning ✓ A/B Testing ✓ Model Endpoint Deployment • Install both R and Python to support two main user groups • Minimize the work of DevOps for model deployment
- 10. #3 - Lambda & API Gateway Frontend • SageMaker endpoint is only available within AWS services • Lambda and API Gateway is used to expose model to external party • Access by partners is controlled via API key
- 11. High Level Solution Architecture Frontend Geospatial Data Data Pre-processing Data Ingestion EDA, Modeling Deployment Serving
- 12. Questions