presentation
•
0 likes•264 views
The document discusses a project analyzing Twitter spam and fraudulent websites. It outlines two proposed architectures - one using Hadoop/HDFS/Flume/Hive/Postgres/Spark and the other using Spark/Scala/Streaming for data ingestion from Twitter APIs, analysis using text analysis and machine learning, and storing results in HDFS for retrieval and display in dashboards and browser plugins. It also discusses future work including clustering, error recovery, API development, improved algorithms, and additional applications of the data pipeline.
Report
Share
Report
Share
Download to read offline
Recommended
Scalable AutoML for Time Series Forecasting using Ray
Time Series Forecasting is widely used in real world applications, such as network quality analysis in Telcos, log analysis for data center operations, predictive maintenance for high-value equipment, etc
Building an open source high performance data analytics platform
This document discusses Twister2, an open source dataflow system for batch and streaming data processing. It provides high-level and lower-level APIs and can execute tasks on HPC clusters, clouds, and containers. The document outlines Twister2's components like its task graph system, data access APIs, and programming models. It also provides examples of how batch and streaming jobs are configured and run on Twister2.
Superset druid realtime
This document discusses using real-time data with Superset by ingesting data streams using Apache Flink and storing the data in Apache Druid. It outlines use cases like operational insights and service debugging. It proposes using Flink for its scalable streaming capabilities and SQL interface and integrating with Druid for its columnar storage and geospatial support. It also describes transforming the data through UDFs before querying and visualizing it in Superset.
Python and H2O with Cliff Click at PyData Dallas 2015
This document discusses H2O.ai, an open source in-memory machine learning platform. It can perform distributed machine learning on large datasets using algorithms like generalized linear modeling, gradient boosted machines, random forests, and deep learning. The platform provides APIs and interfaces for R, Python, Scala, Spark, and other languages. It can handle big data from sources like HDFS, S3, and NFS without sampling. The document includes an overview of H2O's architecture and demonstrates its use on a bike sharing dataset with over 10 million rows.
Sydney Apache Spark Meetup - Spark Natural Language Processing
In this talk, we shared our experience in using Spark to perform natural language processing tasks to drive business value for our clients. We demonstrated the capabilities of word embedding using Spark Word2Vec followed by showing how third party natural language models can be incorporated into Spark applications.
H2O Rains with Databricks Cloud - Parisoma SF
Michal Malohlava's meetup on H2O Rains with Databricks Cloud at Parisoma SF, 02.02.16
- Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai
- To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata
Realtime streaming architecture in INFINARIO
About our experience with realtime analyses on never-ending stream of user events. Discuss Lambda architecture, Kappa, Apache Kafka and our own approach.
Sydney Spark Meetup - September 2015
Presentation at Sydney Spark Meet-up on using Jdbc extraction with Apache Spark for sandbox data discovery. Code examples and walk-through on creating dataframe using Jdbc and materializing as parquet files onto hadoop file system. Demonstrates how to perform partitioning on non-integer data fields from source database.
Recommended
Scalable AutoML for Time Series Forecasting using Ray
Time Series Forecasting is widely used in real world applications, such as network quality analysis in Telcos, log analysis for data center operations, predictive maintenance for high-value equipment, etc
Building an open source high performance data analytics platform
This document discusses Twister2, an open source dataflow system for batch and streaming data processing. It provides high-level and lower-level APIs and can execute tasks on HPC clusters, clouds, and containers. The document outlines Twister2's components like its task graph system, data access APIs, and programming models. It also provides examples of how batch and streaming jobs are configured and run on Twister2.
Superset druid realtime
This document discusses using real-time data with Superset by ingesting data streams using Apache Flink and storing the data in Apache Druid. It outlines use cases like operational insights and service debugging. It proposes using Flink for its scalable streaming capabilities and SQL interface and integrating with Druid for its columnar storage and geospatial support. It also describes transforming the data through UDFs before querying and visualizing it in Superset.
Python and H2O with Cliff Click at PyData Dallas 2015
This document discusses H2O.ai, an open source in-memory machine learning platform. It can perform distributed machine learning on large datasets using algorithms like generalized linear modeling, gradient boosted machines, random forests, and deep learning. The platform provides APIs and interfaces for R, Python, Scala, Spark, and other languages. It can handle big data from sources like HDFS, S3, and NFS without sampling. The document includes an overview of H2O's architecture and demonstrates its use on a bike sharing dataset with over 10 million rows.
Sydney Apache Spark Meetup - Spark Natural Language Processing
In this talk, we shared our experience in using Spark to perform natural language processing tasks to drive business value for our clients. We demonstrated the capabilities of word embedding using Spark Word2Vec followed by showing how third party natural language models can be incorporated into Spark applications.
H2O Rains with Databricks Cloud - Parisoma SF
Michal Malohlava's meetup on H2O Rains with Databricks Cloud at Parisoma SF, 02.02.16
- Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai
- To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata
Realtime streaming architecture in INFINARIO
About our experience with realtime analyses on never-ending stream of user events. Discuss Lambda architecture, Kappa, Apache Kafka and our own approach.
Sydney Spark Meetup - September 2015
Presentation at Sydney Spark Meet-up on using Jdbc extraction with Apache Spark for sandbox data discovery. Code examples and walk-through on creating dataframe using Jdbc and materializing as parquet files onto hadoop file system. Demonstrates how to perform partitioning on non-integer data fields from source database.
An efficient data mining solution by integrating Spark and Cassandra
Integrating C* and Spark gives us a system that combines the best of both worlds. The goal of this integration is to obtain a better result than using Spark over HDFS because Cassandra´s philosophy is much closer to RDD's philosophy than what HDFS is. The goal with Cassandra is to have a system that mines all the information stored in C* in a much more efficient way than having the information stored in HDFS. Cassandra data storage and Spark data mining power: an unrivalled mix.
"Managing the Complete Machine Learning Lifecycle with MLflow"
Machine Learning development brings many new complexities beyond the traditional software development lifecycle. Unlike in traditional software development, ML developers want to try multiple algorithms, tools, and parameters to get the best results, and they need to track this information to reproduce work. In addition, developers need to use many distinct systems to productionize models. To address these problems, many companies are building custom “ML platforms” that automate this lifecycle, but even these platforms are limited to a few supported algorithms and to each company’s internal infrastructure.
In this session, we introduce MLflow, a new open-source project from Databricks that aims to design an open ML platform where organizations can use any ML library and development tool of their choice to reliably build and share ML applications. MLflow introduces simple abstractions to package reproducible projects, track results, and encapsulate models that can be used with many existing tools, accelerating the ML lifecycle for organizations of any size.
Metail at Cambridge AWS User Group Main Meetup #3
An overview of Metail's data pipeline architecture with particular emphasis on our event collection and batch processing layer, based on Hadoop. A brief overview of our use of business intelligence and data science sections is also given.
Metail and Elastic MapReduce
Metail allows users to discover clothes on their body shape online with minimum measurements from the user. With your avatar you can create outfits and coupled with our size advice this gives you a confidence in the size and fit.
I'm part of the team within Metail that has built a pipeline to collection, enriched and serve data to the company and our clients, and which has been used to validate Metail's product. This talk was given at the AWS Loft in London 21st April 2016 where I gave an overview of the end-to-end pipeline and then went into detail how we're using AWS' EMR to perform a batch processing of the collected data which is then served internally with Redshift.
Big Data in the Cloud
AWS Summit 2014 Brisbane - Breakout 5
Most organisations are facing ever growing volumes of data that need to be stored and processed but most importantly analysed to bring value to the business. Big Data appears to have solutions to address these challenges but the landscape is littered with acronyms and obscure naming conventions such as MPP, NoSQL, Hadoop, Hive and HBase. Attend this Session to find out
- What is the value proposition for each of these technologies
- How do they fit with more traditional Big Data solutions such as data warehouses?
- How AWS can help organisations get maximum value from their data
Presenter: Russell Nash, Solutions Architect, APAC, Amazon Web Services
Building Scalable Big Data Pipelines
Christian Gügi presented on building scalable big data pipelines. He discussed opportunities and challenges with big data, integrating Hadoop into existing systems, and the Lambda architecture for batch and real-time processing. He provided an example pipeline for online marketing analytics and recommendations for adopting the Lambda architecture approach.
Spark Summit EU 2015: Combining the Strengths of MLlib, scikit-learn, and R
This talk discusses integrating common data science tools like Python pandas, scikit-learn, and R with MLlib, Spark’s distributed Machine Learning (ML) library. Integration is simple; migration to distributed ML can be done lazily; and scaling to big data can significantly improve accuracy. We demonstrate integration with a simple data science workflow. Data scientists often encounter scaling bottlenecks with single-machine ML tools. Yet the overhead in migrating to a distributed workflow can seem daunting. In this talk, we demonstrate such a migration, taking advantage of Spark and MLlib’s integration with common ML libraries. We begin with a small dataset which runs on a single machine. Increasing the size, we hit bottlenecks in various parts of the workflow: hyperparameter tuning, then ETL, and eventually the core learning algorithm. As we hit each bottleneck, we parallelize that part of the workflow using Spark and MLlib. As we increase the dataset and model size, we can see significant gains in accuracy. We end with results demonstrating the impressive scalability of MLlib algorithms. With accuracy comparable to traditional ML libraries, combined with state-of-the-art distributed scalability, MLlib is a valuable new tool for the modern data scientist.
Data Warehousing with Spark Streaming at Zalando
Zalandos AI-driven products and distributed landscape of analytical data marts cannot wait for long-running, hard-to-recover, monolithic batch jobs taking all night to calculate already outdated data. Modern data integration pipelines need to deliver fast and easy to consume data sets in high quality. Based on Spark Streaming and Delta, the central data warehousing team was able to deliver widely-used master data as S3 or Kafka streams and snapshots at the same time.
The talk will cover challenges in our fashion data platform and a detailed architectural deep dive about separation of integration from enrichment, providing streams as well as snapshots and feeding the data to distributed data marts. Finally, lessons learned and best practices about Delta’s MERGE command, Scala API vs Spark SQL and schema evolution give more insights and guidance for similar use cases.
H2O Rains with Databricks Cloud - NY 02.16.16
Michal Malohlava's presentation on H2O Rains with Databricks Cloud, New York, NY 02.16.16
- Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai
- To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata
SparkR: The Past, the Present and the Future-(Shivaram Venkataraman and Rui S...
SparkR allows users to perform large-scale data analysis from R. It provides a unified approach for working with big data using DataFrames. The current release of SparkR includes DataFrame support released in Spark 1.4 for improved performance. Future developments will expand SparkR's capabilities for large-scale machine learning, simple parallel programming APIs, and tighter integration with existing R packages. The open source project has an growing community of over 20 contributors working to advance SparkR.
On Improving Broadcast Joins in Apache Spark SQL
- The document discusses improving broadcast joins in Apache Spark SQL, which are more efficient than shuffle joins when the broadcasted data fits in memory.
- Experimenting with increasing the broadcast threshold showed that executor-side broadcasting performs better than driver-side broadcasting by avoiding data shuffling to the driver.
- Comparing the cost models of shuffle joins and broadcast joins showed that shuffle joins perform better with more cores while broadcast joins perform better when the size difference between tables is larger.
- Applying these techniques to joins in Workday HR customer data pipelines showed that increasing the broadcast threshold did not always improve performance due to the presence of self-joins and outer joins.
JEEConf 2015 Big Data Analysis in Java World
The document discusses different technologies for handling big data: MapReduce, MPP analytical databases, and in-memory data fabrics. It then describes a real-life project involving sensor data that is growing rapidly. The requirements are for structured analytics on the data in near real-time. Vertica, an MPP database, is proposed as the solution since it can handle the volumes of structured time-series data, perform analytics using SQL, and scale to meet future growth needs. R analytics would also be integrated as user-defined functions running on the Vertica cluster.
Petabytes, Exabytes, and Beyond: Managing Delta Lakes for Interactive Queries...
Data production continues to scale up and the techniques for managing it need to scale too. Building pipelines that can process petabytes per day in turn create data lakes with exabytes of historical data. At Databricks, we help our customers turn these data lakes into gold mines of valuable information using Apache Spark. This talk will cover techniques to optimize access to these data lakes using Delta Lakes, including range partitioning, file-based data skipping, multi-dimensional clustering, and read-optimized files. We'll cover sample implementations and see examples of querying petabytes of data in seconds, not hours. We'll also discuss tradeoffs that data engineers deal with everyday like read speed vs. write throughput, managing storage costs, and duplicating data to support multiple query profiles. We'll also discuss combining batch with streaming to achieve desired query performance. After this session, you will have new ideas for managing truly massive Delta Lakes.
Multi Model Machine Learning by Maximo Gurmendez and Beth Logan
1. DataXu uses Spark for large-scale machine learning to power real-time bidding for display ads, processing 2 petabytes of data daily and making 1.6 million bid decisions per second across 5 continents.
2. Some challenges in using Spark included smart dataset partitioning, handling categorical and functional features, and enabling real-time model instantiations for bidding.
3. Spark SQL and UDFs helped address these challenges by enabling declarative feature encoding and selection, reusable transformations for both training and bidding, and efficient categorical encoding and top K feature value extraction.
Graph computation
~ Graph of the Internet
~ Table data
~ Compute configuration
~ Graph data structures
~ Tweaking for memory
Media_Entertainment_Veriticals
This document discusses applying Apache Spark to data science challenges in media and entertainment. It introduces Spark as a unifying framework for content personalization using recommendation systems and streaming data, as well as social media analytics using GraphFrames. Specific use cases discussed include content personalization with recommendations, churn analysis, analyzing social networks with GraphFrames, sentiment analysis, and viewership prediction using topic modeling. The document also discusses continuous applications with Spark Streaming, and how Spark ML can be used for machine learning workflows and optimization.
Lessons Learned from Modernizing USCIS Data Analytics Platform
U.S. Citizenship and Immigration Services (USCIS) is the government agency that oversees lawful immigration to the United States.
Minería de Datos en Sql Server 2008
Inteligencia de Negocios en SQL Server 2008 y Minería de Datos.
Ing. Eduardo Castro Martinez, PhD
Microsoft SQL Server MVP
http://ecastrom.blogspot.com
http://comunidadwindows.org
Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy - Paris Spark...
Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy - Paris Spark...Modern Data Stack France
Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy
Retour d'expérience sur la mise en place d'un Datalab avec Hadoop, Spark et ElasticSearch dans un environnement contraint. Nous allons exposer les méthodes qui nous ont permis d'améliorer la conception, le développement, les performances et la recette d'une application complexe en Spark.
Jonathan Winandy est MOE, développeur Java/Scala spécialisé dans les pipelines de données.Graph Processing with Apache TinkerPop
This document discusses Apache TinkerPop, a graph computing framework. It provides an overview of TinkerPop and the graph landscape, describes common graph domains and the Gremlin property graph model. It also demonstrates hands-on examples with Titan and Spark/Giraph and discusses using graphs to analyze dependency management and the NPM registry. The document emphasizes that TinkerPop allows seamless use of OLTP and OLAP graphs via Gremlin and supports graph-based thinking for multi-model data.
Paris Data Geek - Spark Streaming
This document discusses Spark Streaming and its use for near real-time ETL. It provides an overview of Spark Streaming, how it works internally using receivers and workers to process streaming data, and an example use case of building a recommender system to find matches using both batch and streaming data. Key points covered include the streaming execution model, handling data receipt and job scheduling, and potential issues around data loss and (de)serialization.
Austin Data Meetup 092014 - Spark
The Spark Summit was attended by over 1,100 people from 450+ companies and featured keynotes and community presentations. Spark is an active Apache project with over 250 contributors from 50+ companies. It includes subprojects for SQL, streaming, machine learning and graph processing. Popular use cases include real-time recommendations, cancer genomics, and media analytics.
More Related Content
What's hot
An efficient data mining solution by integrating Spark and Cassandra
Integrating C* and Spark gives us a system that combines the best of both worlds. The goal of this integration is to obtain a better result than using Spark over HDFS because Cassandra´s philosophy is much closer to RDD's philosophy than what HDFS is. The goal with Cassandra is to have a system that mines all the information stored in C* in a much more efficient way than having the information stored in HDFS. Cassandra data storage and Spark data mining power: an unrivalled mix.
"Managing the Complete Machine Learning Lifecycle with MLflow"
Machine Learning development brings many new complexities beyond the traditional software development lifecycle. Unlike in traditional software development, ML developers want to try multiple algorithms, tools, and parameters to get the best results, and they need to track this information to reproduce work. In addition, developers need to use many distinct systems to productionize models. To address these problems, many companies are building custom “ML platforms” that automate this lifecycle, but even these platforms are limited to a few supported algorithms and to each company’s internal infrastructure.
In this session, we introduce MLflow, a new open-source project from Databricks that aims to design an open ML platform where organizations can use any ML library and development tool of their choice to reliably build and share ML applications. MLflow introduces simple abstractions to package reproducible projects, track results, and encapsulate models that can be used with many existing tools, accelerating the ML lifecycle for organizations of any size.
Metail at Cambridge AWS User Group Main Meetup #3
An overview of Metail's data pipeline architecture with particular emphasis on our event collection and batch processing layer, based on Hadoop. A brief overview of our use of business intelligence and data science sections is also given.
Metail and Elastic MapReduce
Metail allows users to discover clothes on their body shape online with minimum measurements from the user. With your avatar you can create outfits and coupled with our size advice this gives you a confidence in the size and fit.
I'm part of the team within Metail that has built a pipeline to collection, enriched and serve data to the company and our clients, and which has been used to validate Metail's product. This talk was given at the AWS Loft in London 21st April 2016 where I gave an overview of the end-to-end pipeline and then went into detail how we're using AWS' EMR to perform a batch processing of the collected data which is then served internally with Redshift.
Big Data in the Cloud
AWS Summit 2014 Brisbane - Breakout 5
Most organisations are facing ever growing volumes of data that need to be stored and processed but most importantly analysed to bring value to the business. Big Data appears to have solutions to address these challenges but the landscape is littered with acronyms and obscure naming conventions such as MPP, NoSQL, Hadoop, Hive and HBase. Attend this Session to find out
- What is the value proposition for each of these technologies
- How do they fit with more traditional Big Data solutions such as data warehouses?
- How AWS can help organisations get maximum value from their data
Presenter: Russell Nash, Solutions Architect, APAC, Amazon Web Services
Building Scalable Big Data Pipelines
Christian Gügi presented on building scalable big data pipelines. He discussed opportunities and challenges with big data, integrating Hadoop into existing systems, and the Lambda architecture for batch and real-time processing. He provided an example pipeline for online marketing analytics and recommendations for adopting the Lambda architecture approach.
Spark Summit EU 2015: Combining the Strengths of MLlib, scikit-learn, and R
This talk discusses integrating common data science tools like Python pandas, scikit-learn, and R with MLlib, Spark’s distributed Machine Learning (ML) library. Integration is simple; migration to distributed ML can be done lazily; and scaling to big data can significantly improve accuracy. We demonstrate integration with a simple data science workflow. Data scientists often encounter scaling bottlenecks with single-machine ML tools. Yet the overhead in migrating to a distributed workflow can seem daunting. In this talk, we demonstrate such a migration, taking advantage of Spark and MLlib’s integration with common ML libraries. We begin with a small dataset which runs on a single machine. Increasing the size, we hit bottlenecks in various parts of the workflow: hyperparameter tuning, then ETL, and eventually the core learning algorithm. As we hit each bottleneck, we parallelize that part of the workflow using Spark and MLlib. As we increase the dataset and model size, we can see significant gains in accuracy. We end with results demonstrating the impressive scalability of MLlib algorithms. With accuracy comparable to traditional ML libraries, combined with state-of-the-art distributed scalability, MLlib is a valuable new tool for the modern data scientist.
Data Warehousing with Spark Streaming at Zalando
Zalandos AI-driven products and distributed landscape of analytical data marts cannot wait for long-running, hard-to-recover, monolithic batch jobs taking all night to calculate already outdated data. Modern data integration pipelines need to deliver fast and easy to consume data sets in high quality. Based on Spark Streaming and Delta, the central data warehousing team was able to deliver widely-used master data as S3 or Kafka streams and snapshots at the same time.
The talk will cover challenges in our fashion data platform and a detailed architectural deep dive about separation of integration from enrichment, providing streams as well as snapshots and feeding the data to distributed data marts. Finally, lessons learned and best practices about Delta’s MERGE command, Scala API vs Spark SQL and schema evolution give more insights and guidance for similar use cases.
H2O Rains with Databricks Cloud - NY 02.16.16
Michal Malohlava's presentation on H2O Rains with Databricks Cloud, New York, NY 02.16.16
- Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai
- To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata
SparkR: The Past, the Present and the Future-(Shivaram Venkataraman and Rui S...
SparkR allows users to perform large-scale data analysis from R. It provides a unified approach for working with big data using DataFrames. The current release of SparkR includes DataFrame support released in Spark 1.4 for improved performance. Future developments will expand SparkR's capabilities for large-scale machine learning, simple parallel programming APIs, and tighter integration with existing R packages. The open source project has an growing community of over 20 contributors working to advance SparkR.
On Improving Broadcast Joins in Apache Spark SQL
- The document discusses improving broadcast joins in Apache Spark SQL, which are more efficient than shuffle joins when the broadcasted data fits in memory.
- Experimenting with increasing the broadcast threshold showed that executor-side broadcasting performs better than driver-side broadcasting by avoiding data shuffling to the driver.
- Comparing the cost models of shuffle joins and broadcast joins showed that shuffle joins perform better with more cores while broadcast joins perform better when the size difference between tables is larger.
- Applying these techniques to joins in Workday HR customer data pipelines showed that increasing the broadcast threshold did not always improve performance due to the presence of self-joins and outer joins.
JEEConf 2015 Big Data Analysis in Java World
The document discusses different technologies for handling big data: MapReduce, MPP analytical databases, and in-memory data fabrics. It then describes a real-life project involving sensor data that is growing rapidly. The requirements are for structured analytics on the data in near real-time. Vertica, an MPP database, is proposed as the solution since it can handle the volumes of structured time-series data, perform analytics using SQL, and scale to meet future growth needs. R analytics would also be integrated as user-defined functions running on the Vertica cluster.
Petabytes, Exabytes, and Beyond: Managing Delta Lakes for Interactive Queries...
Data production continues to scale up and the techniques for managing it need to scale too. Building pipelines that can process petabytes per day in turn create data lakes with exabytes of historical data. At Databricks, we help our customers turn these data lakes into gold mines of valuable information using Apache Spark. This talk will cover techniques to optimize access to these data lakes using Delta Lakes, including range partitioning, file-based data skipping, multi-dimensional clustering, and read-optimized files. We'll cover sample implementations and see examples of querying petabytes of data in seconds, not hours. We'll also discuss tradeoffs that data engineers deal with everyday like read speed vs. write throughput, managing storage costs, and duplicating data to support multiple query profiles. We'll also discuss combining batch with streaming to achieve desired query performance. After this session, you will have new ideas for managing truly massive Delta Lakes.
Multi Model Machine Learning by Maximo Gurmendez and Beth Logan
1. DataXu uses Spark for large-scale machine learning to power real-time bidding for display ads, processing 2 petabytes of data daily and making 1.6 million bid decisions per second across 5 continents.
2. Some challenges in using Spark included smart dataset partitioning, handling categorical and functional features, and enabling real-time model instantiations for bidding.
3. Spark SQL and UDFs helped address these challenges by enabling declarative feature encoding and selection, reusable transformations for both training and bidding, and efficient categorical encoding and top K feature value extraction.
Graph computation
~ Graph of the Internet
~ Table data
~ Compute configuration
~ Graph data structures
~ Tweaking for memory
Media_Entertainment_Veriticals
This document discusses applying Apache Spark to data science challenges in media and entertainment. It introduces Spark as a unifying framework for content personalization using recommendation systems and streaming data, as well as social media analytics using GraphFrames. Specific use cases discussed include content personalization with recommendations, churn analysis, analyzing social networks with GraphFrames, sentiment analysis, and viewership prediction using topic modeling. The document also discusses continuous applications with Spark Streaming, and how Spark ML can be used for machine learning workflows and optimization.
Lessons Learned from Modernizing USCIS Data Analytics Platform
U.S. Citizenship and Immigration Services (USCIS) is the government agency that oversees lawful immigration to the United States.
Minería de Datos en Sql Server 2008
Inteligencia de Negocios en SQL Server 2008 y Minería de Datos.
Ing. Eduardo Castro Martinez, PhD
Microsoft SQL Server MVP
http://ecastrom.blogspot.com
http://comunidadwindows.org
Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy - Paris Spark...
Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy - Paris Spark...Modern Data Stack France
Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy
Retour d'expérience sur la mise en place d'un Datalab avec Hadoop, Spark et ElasticSearch dans un environnement contraint. Nous allons exposer les méthodes qui nous ont permis d'améliorer la conception, le développement, les performances et la recette d'une application complexe en Spark.
Jonathan Winandy est MOE, développeur Java/Scala spécialisé dans les pipelines de données.Graph Processing with Apache TinkerPop
This document discusses Apache TinkerPop, a graph computing framework. It provides an overview of TinkerPop and the graph landscape, describes common graph domains and the Gremlin property graph model. It also demonstrates hands-on examples with Titan and Spark/Giraph and discusses using graphs to analyze dependency management and the NPM registry. The document emphasizes that TinkerPop allows seamless use of OLTP and OLAP graphs via Gremlin and supports graph-based thinking for multi-model data.
What's hot (20)
An efficient data mining solution by integrating Spark and Cassandra
An efficient data mining solution by integrating Spark and Cassandra
"Managing the Complete Machine Learning Lifecycle with MLflow"
"Managing the Complete Machine Learning Lifecycle with MLflow"
Spark Summit EU 2015: Combining the Strengths of MLlib, scikit-learn, and R
Spark Summit EU 2015: Combining the Strengths of MLlib, scikit-learn, and R
SparkR: The Past, the Present and the Future-(Shivaram Venkataraman and Rui S...
SparkR: The Past, the Present and the Future-(Shivaram Venkataraman and Rui S...
Petabytes, Exabytes, and Beyond: Managing Delta Lakes for Interactive Queries...
Petabytes, Exabytes, and Beyond: Managing Delta Lakes for Interactive Queries...
Multi Model Machine Learning by Maximo Gurmendez and Beth Logan
Multi Model Machine Learning by Maximo Gurmendez and Beth Logan
Lessons Learned from Modernizing USCIS Data Analytics Platform
Lessons Learned from Modernizing USCIS Data Analytics Platform
Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy - Paris Spark...
Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy - Paris Spark...
Similar to presentation
Paris Data Geek - Spark Streaming
This document discusses Spark Streaming and its use for near real-time ETL. It provides an overview of Spark Streaming, how it works internally using receivers and workers to process streaming data, and an example use case of building a recommender system to find matches using both batch and streaming data. Key points covered include the streaming execution model, handling data receipt and job scheduling, and potential issues around data loss and (de)serialization.
Austin Data Meetup 092014 - Spark
The Spark Summit was attended by over 1,100 people from 450+ companies and featured keynotes and community presentations. Spark is an active Apache project with over 250 contributors from 50+ companies. It includes subprojects for SQL, streaming, machine learning and graph processing. Popular use cases include real-time recommendations, cancer genomics, and media analytics.
Spark ML Pipeline serving
My talk at Data Science Labs conference in Odessa.
Training a model in Apache Spark while having it automatically available for real-time serving is an essential feature for end-to-end solutions.
There is an option to export the model into PMML and then import it into a separated scoring engine. The idea of interoperability is great but it has multiple challenges, such as code duplication, limited extensibility, inconsistency, extra moving parts. In this talk we discussed an alternative solution that does not introduce custom model formats and new standards, not based on export/import workflow and shares Apache Spark API.
It takes two to tango! : Is SQL-on-Hadoop the next big step?
This document discusses the evolution of technologies for processing large datasets from before Hadoop to modern SQL-on-Hadoop approaches. It describes the early limitations of technologies like partitioned databases and data warehouses that led to the development of Hadoop. It then examines different approaches for adding SQL capabilities to Hadoop like Cloudera Impala's distributed query processing, Microsoft Polybase's split query processing, and faster implementations of Hive. The document provides architectural diagrams and explanations of how various SQL-on-Hadoop technologies work.
Big Data Journey
Generic presentation about Big Data Architecture/Components. This presentation was delivered by David Pilato and Tugdual Grall during JUG Summer Camp 2015 in La Rochelle, France
Machine Learning with H2O, Spark, and Python at Strata 2015
Machine Learning with H2O, Spark, and Python at Strata SJ 2015-by Cliff Click and Michal Malohlava
- Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai
- To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata
2014 sept 26_thug_lambda_part1
The document discusses the Lambda architecture, which combines batch and stream processing. It provides an example implementation using Hadoop, Kafka, Storm and other tools. The Lambda architecture handles batch loading and querying of large datasets as well as real-time processing of data streams. It also discusses using YARN and Spark for distributed processing and refreshing enrichments.
Spark and machine learning in microservices architecture
Modern architecture for machine learning and state-safe data processing.
Data Pipelines and machine learning models deployment in production.
HKOSCon18 - Chetan Khatri - Scaling TB's of Data with Apache Spark and Scala ...
This document summarizes a presentation about scaling terabytes of data with Apache Spark and Scala. The key points are:
1) The presenter discusses how to use Apache Spark and Scala to process large scale data in a distributed manner across clusters. Spark operations like RDDs, DataFrames and Datasets are covered.
2) A case study is presented about reengineering a data processing platform for a retail business to improve performance. Changes included parallelizing jobs, tuning Spark hyperparameters, and building a fast data architecture using Spark, Kafka and data lakes.
3) Performance was improved through techniques like dynamic resource allocation in YARN, reducing memory and cores per executor to better utilize cluster resources, and processing data
TWDI Accelerate Seattle, Oct 16, 2017: Distributed and In-Database Analytics ...
Event: TDWI Accelerate Seattle, October 16, 2017
Topic: Distributed and In-Database Analytics with R
Presenter: Debraj GuhaThakurta
Description: How to develop scalable and in-DB analytics using R in Spark and SQL-Server
TDWI Accelerate, Seattle, Oct 16, 2017: Distributed and In-Database Analytics...
R is a popular statistical programming language used for data analysis and machine learning. It has over 3 million users and is taught widely in universities. While powerful, R has some scaling limitations for big data. Several Apache Spark integrations with R like SparkR and sparklyr enable distributed, parallel processing of large datasets using R on Spark clusters. Other options for scaling R include H2O for in-memory analytics, Microsoft ML Server for on-premises scaling, and ScaleR for portable parallel processing across platforms. These solutions allow R programs and models to be trained on large datasets and deployed for operational use on big data in various cloud and on-premises environments.
Big data vahidamiri-tabriz-13960226-datastack.ir
کارگاه کلان داده در سومین کنفرانس ملی محاسبات توزیعی و پردازش داده های بزرگ
دانشگاه شهید مدنی آذربایجان
تبریز
اردیبهشت 96
AWSug.nl Data recap Jan 2023
This document summarizes announcements from re:Invent 2022 related to AWS data services. Over 40% of announcements were data-related. Key announcements included the introduction of Data Zone as a central platform for data organization and access control, new features for Redshift like dynamic data masking and auto copy from S3, updates to Glue including support for Pandas and data quality checks, and new connectors for Appflow totaling 22 additions. RDS and S3 also received updates around deployment improvements and multi-region access point failover controls.
Document Conversion & Retrieve and Rank 一問一答
This document provides information about using Watson services like Document Conversion and Retrieve and Rank through their APIs. It includes examples of calling the APIs to index documents, search a Solr collection, create a ranker, and perform a ranked search.
Sanath pabba hadoop resume 1.0
Sanath Pabba has over 5 years of experience working with big data technologies like Hadoop, Spark, Hive, Pig, Kafka and NoSQL databases. He has expertise in data extraction, transformation and loading processes. Some of his responsibilities include writing Sqoop and Spark jobs to load and prepare data, developing automation scripts to monitor cluster utilization, and implementing validation rules for data quality. He has worked on various projects involving data warehousing, reporting, stream processing and analytics using technologies like SQL Server, Hive and Spark.
Lambda Architecture Using SQL
Keynote of HadoopCon 2014 Taiwan:
* Data analytics platform architecture & designs
* Lambda architecture overview
* Using SQL as DSL for stream processing
* Lambda architecture using SQL
Dev Ops Training
The document provides an agenda for a DevOps advanced class on Spark being held in June 2015. The class will cover topics such as RDD fundamentals, Spark runtime architecture, memory and persistence, Spark SQL, PySpark, and Spark Streaming. It will include labs on DevOps 101 and 102. The instructor has over 5 years of experience providing Big Data consulting and training, including over 100 classes taught.
Frustration-Reduced Spark: DataFrames and the Spark Time-Series Library
In this talk I talk about my recent experience working with Spark Data Frames and the Spark TimeSeries library. For data frames, the focus will be on usability. Specifically, a lot of the documentation does not cover common use cases like intricacies of creating data frames, adding or manipulating individual columns, and doing quick and dirty analytics. For the time series library, I dive into the kind of use cases it supports and why it’s actually super useful.
5 Reasons why Spark is in demand!
Spark is in high demand for several reasons: it offers low-latency processing by keeping data in memory, supports streaming analytics, machine learning algorithms, and graph processing. It also introduces DataFrames for easier data analysis and integrates well with Hadoop for processing large datasets. Spark can sort 100TB of data 3 times faster than MapReduce using fewer resources, making it a popular big data processing engine.
sam_resume - updated
Sam has over 8 years of experience in big data technologies including Hadoop, Spark, Hive, Pig, Kafka and Cassandra. He has expertise in data ingestion, processing, ETL and analytics. Some of his responsibilities include developing MapReduce programs, Spark applications, data pipelines, streaming jobs and visualizations. He is proficient in Java, Python, Scala and various big data tools.
Similar to presentation (20)
It takes two to tango! : Is SQL-on-Hadoop the next big step?
It takes two to tango! : Is SQL-on-Hadoop the next big step?
Machine Learning with H2O, Spark, and Python at Strata 2015
Machine Learning with H2O, Spark, and Python at Strata 2015
Spark and machine learning in microservices architecture
Spark and machine learning in microservices architecture
HKOSCon18 - Chetan Khatri - Scaling TB's of Data with Apache Spark and Scala ...
HKOSCon18 - Chetan Khatri - Scaling TB's of Data with Apache Spark and Scala ...
TWDI Accelerate Seattle, Oct 16, 2017: Distributed and In-Database Analytics ...
TWDI Accelerate Seattle, Oct 16, 2017: Distributed and In-Database Analytics ...
TDWI Accelerate, Seattle, Oct 16, 2017: Distributed and In-Database Analytics...
TDWI Accelerate, Seattle, Oct 16, 2017: Distributed and In-Database Analytics...
Frustration-Reduced Spark: DataFrames and the Spark Time-Series Library
Frustration-Reduced Spark: DataFrames and the Spark Time-Series Library
presentation
- 1. Twitter Spam & Fraudulent Websites Kyle Hamilton, Carlos Rodriguez, and Sharmila Velamur
- 4. Concept stage HDFS - Batch Layer Data Ingestion: Custom Implementation - Scraping - Scheduled Job Data Analysis & Precomputation: Text Analysis - Scheduled Job Browser Plug-in Dashboard Data Storing & Retrieval: Hive, NoSql (or) RDBMS Distributed Workflow & Scheduling Coordinator
- 8. Twitter API Object-Relationship Model (Selective & Partial)
- 10. OPTION A Architecture Hadoop-HDFS Flume Hive Postgres Spark-SQL Spark-Pyspark S3 Cron Shiny App (R, HTML5) Javascript
- 11. OPTION B Architecture Hadoop-HDFS Spark-Scala Streaming Lib Twitter Utils MLLib/ML Spark-JDBC Shiny App (R, HTML5) Zeppelin Optimization: Spark RDDs Scala (JVM) Streaming
- 12. GO-DAWG-GO! w205Project_V2.0 AMI wget https://s3-us-west-2.amazonaws. com/w205twitterproject/provision.sh . provision.sh . bootstrap.sh crontab simple_sched_cron.txt
- 13. FUTURE WORK ● Clustering and parallelization; using EMR. ● Recovering from errors automagically. ● Tweaking config such as heap space and other JVM parameters. ● RESTful API for dashboard as well as browser plug-in. ● Better classification algorithms. ● Magnify “spam” URL extension ● Utilizing the same data pipeline for multiple areas: ○ Online marketing - semantic analysis. ○ Political campaigns - popular topics, alternative polling. ○ Social experiments on user behavior. ○ Geolocation based analysis.