Spark Summit EU talk by Javier Aguedes
•
3 likes•1,037 views
This document discusses generating origin-destination matrices from mobile network data using Spark. It describes using telco event data to dynamically compute OD matrices over time and by mode of transport between geographic regions. The approach involves processing raw telco data to estimate locations, detect trips, determine mode of transport, and aggregate the results into OD matrices for given time periods and regions. Spark is used to efficiently process large datasets in parallel. The matrices are validated against ground truth data and checked for logical consistency.
Report
Share
Report
Share
Download to read offline
Recommended
Spark Summit EU talk by Chris Pool and Jeroen Vlek
This document discusses a predictive maintenance project for switch failures on Dutch railways. The goal is to predict switch failures 3 weeks in advance using sensor data collected from switches. Features are derived from the sensor data and a decision tree model is used to classify switches into failure risk categories. The current system uses SQL but will transition to Spark for increased data volumes and streaming capabilities. Initial results show high precision and recall. Future work involves deep learning models and predicting specific failure types.
Spark Summit EU talk by Miha Pelko and Til Piffl
1) NorCom is an IT company that provides consulting services for big data and information management, with customers in automotive, public, media, and finance sectors.
2) Time series analysis of sensor data from vehicles is important for automotive R&D, but requires parallel processing due to the large volumes of data.
3) NorCom developed a Spark API called DaSense to simplify working with multi-sensor time series data at scale, and explored parallelizing state machine analysis on vehicle data using Spark.
Spark Summit EU talk by Reza Karimi
The document discusses mentorship modeling based on authorship graphs from Scopus data. It describes building features from co-authorship and correspondence graphs using Spark, validating predictions via crowdsourcing, and visualizing mentorship subgraphs. Key points include normalizing authorship data, aggregating node and edge features, applying pairwise mentorship models, obtaining training data via email campaigns, and using D3.js to interactively display mentorship subgraphs in Spark applications.
Spark Summit EU talk by Zoltan Zvara
This document discusses a project called Data-Aware Spark that aims to make Spark a data-aware distributed data processing framework. It addresses the problem of data skew where some partitions contain disproportionately more data than others, causing slow tasks. The project develops a dynamic repartitioning mechanism that collects data characteristics from executors, determines when and how to repartition, and constructs a new hash function to more evenly distribute the data across partitions. It provides tracing and visualization capabilities to help debug and monitor applications. Evaluation results show the approach reduces data skew and task runtime compared to naive hashing.
Spark Summit EU talk by Brij Bhushan Ravat
The document discusses productizing a Spark and Cassandra based solution for a telecom voucher server. It describes challenges faced during evolution including moving from Oracle to Cassandra, improving report generation performance using Spark, and key operations and maintenance challenges of running Cassandra and Spark together such as repairing inconsistencies, handling lagging compaction, and balancing real-time and batch processing loads. The presentation provides examples of how these challenges were addressed during the productization process.
Spark Summit EU talk by Nick Pentreath
This document summarizes a presentation about scaling factorization machines on Apache Spark using parameter servers. The presentation introduces factorization machines as a method for modeling feature interactions in large datasets. It then discusses how distributed factorization machine models can be implemented on Spark using a parameter server approach, as demonstrated by the GlintFM library. Performance results show that GlintFM achieves significant speedups over the existing spark-libFM implementation on a large Criteo dataset, scaling to billions of parameters. Some remaining challenges are also outlined, along with ideas for future work.
Download It
The document discusses using Map-Reduce for machine learning algorithms on multi-core processors. It describes rewriting machine learning algorithms in "summation form" to express the independent computations as Map tasks and aggregating results as Reduce tasks. This formulation allows the algorithms to be parallelized efficiently across multiple cores. Specific machine learning algorithms that have been implemented or analyzed in this Map-Reduce framework are listed.
Spark Summit EU talk by Sameer Agarwal
This document discusses Project Tungsten, which aims to substantially improve the memory and CPU efficiency of Spark. It describes how Spark has optimized IO but the CPU has become the bottleneck. Project Tungsten focuses on improving execution performance through techniques like explicit memory management, code generation, cache-aware algorithms, whole-stage code generation, and columnar in-memory data formats. It shows how these techniques provide significant performance improvements, such as 5-30x speedups on operators and 10-100x speedups on radix sort. Future work includes cost-based optimization and improving performance on many-core machines.
Recommended
Spark Summit EU talk by Chris Pool and Jeroen Vlek
This document discusses a predictive maintenance project for switch failures on Dutch railways. The goal is to predict switch failures 3 weeks in advance using sensor data collected from switches. Features are derived from the sensor data and a decision tree model is used to classify switches into failure risk categories. The current system uses SQL but will transition to Spark for increased data volumes and streaming capabilities. Initial results show high precision and recall. Future work involves deep learning models and predicting specific failure types.
Spark Summit EU talk by Miha Pelko and Til Piffl
1) NorCom is an IT company that provides consulting services for big data and information management, with customers in automotive, public, media, and finance sectors.
2) Time series analysis of sensor data from vehicles is important for automotive R&D, but requires parallel processing due to the large volumes of data.
3) NorCom developed a Spark API called DaSense to simplify working with multi-sensor time series data at scale, and explored parallelizing state machine analysis on vehicle data using Spark.
Spark Summit EU talk by Reza Karimi
The document discusses mentorship modeling based on authorship graphs from Scopus data. It describes building features from co-authorship and correspondence graphs using Spark, validating predictions via crowdsourcing, and visualizing mentorship subgraphs. Key points include normalizing authorship data, aggregating node and edge features, applying pairwise mentorship models, obtaining training data via email campaigns, and using D3.js to interactively display mentorship subgraphs in Spark applications.
Spark Summit EU talk by Zoltan Zvara
This document discusses a project called Data-Aware Spark that aims to make Spark a data-aware distributed data processing framework. It addresses the problem of data skew where some partitions contain disproportionately more data than others, causing slow tasks. The project develops a dynamic repartitioning mechanism that collects data characteristics from executors, determines when and how to repartition, and constructs a new hash function to more evenly distribute the data across partitions. It provides tracing and visualization capabilities to help debug and monitor applications. Evaluation results show the approach reduces data skew and task runtime compared to naive hashing.
Spark Summit EU talk by Brij Bhushan Ravat
The document discusses productizing a Spark and Cassandra based solution for a telecom voucher server. It describes challenges faced during evolution including moving from Oracle to Cassandra, improving report generation performance using Spark, and key operations and maintenance challenges of running Cassandra and Spark together such as repairing inconsistencies, handling lagging compaction, and balancing real-time and batch processing loads. The presentation provides examples of how these challenges were addressed during the productization process.
Spark Summit EU talk by Nick Pentreath
This document summarizes a presentation about scaling factorization machines on Apache Spark using parameter servers. The presentation introduces factorization machines as a method for modeling feature interactions in large datasets. It then discusses how distributed factorization machine models can be implemented on Spark using a parameter server approach, as demonstrated by the GlintFM library. Performance results show that GlintFM achieves significant speedups over the existing spark-libFM implementation on a large Criteo dataset, scaling to billions of parameters. Some remaining challenges are also outlined, along with ideas for future work.
Download It
The document discusses using Map-Reduce for machine learning algorithms on multi-core processors. It describes rewriting machine learning algorithms in "summation form" to express the independent computations as Map tasks and aggregating results as Reduce tasks. This formulation allows the algorithms to be parallelized efficiently across multiple cores. Specific machine learning algorithms that have been implemented or analyzed in this Map-Reduce framework are listed.
Spark Summit EU talk by Sameer Agarwal
This document discusses Project Tungsten, which aims to substantially improve the memory and CPU efficiency of Spark. It describes how Spark has optimized IO but the CPU has become the bottleneck. Project Tungsten focuses on improving execution performance through techniques like explicit memory management, code generation, cache-aware algorithms, whole-stage code generation, and columnar in-memory data formats. It shows how these techniques provide significant performance improvements, such as 5-30x speedups on operators and 10-100x speedups on radix sort. Future work includes cost-based optimization and improving performance on many-core machines.
Airfare prediction using Machine Learning with Apache Spark on 1 billion obse...
Prediction using Machine Learning (ML) techniques on Big Data is a computationally and system-wide challenging problem. Especially in the case when the system is processing approximately 10^9 observations per day scalability is the prime concern. In order to be able to rapidly train models covering whole multivariate space the time series vectors, which exhibit significant similarities, are clustered into the groups. Consequently the resulting vector clusters could be modelled using ML tools capable of coefficient estimation at the massive scale (Apache Spark with Scikit Learn). Presentation describes application of the Linear Regression and Support Vector Regression with Radial Basis Function kernel. This approach enables training models fast enough to complete the task within a couple of hours, allowing daily or even real time updates of the coefficients. The above machine learning framework is used to predict the airfares used as support tool for the Revenue Management systems.
Breaking Down Analytical and Computational Barriers Across the Energy Industr...
DNV GL is presenting on breaking down barriers in energy data analytics through their work in statistical computing and machine learning. They discuss using big data from smart grids and advanced metering infrastructure to better understand demand behavior and integrate disruptive technologies like electric vehicles and solar power. DNV GL also demonstrates a pilot project using Databricks and Spark to perform real-time energy forecasting and grid reliability analysis at scale.
Willump: Optimizing Feature Computation in ML Inference
Systems for performing ML inference are increasingly important, but are far slower than they could be because they use techniques designed for conventional data serving workloads, neglecting the statistical nature of ML inference. As an alternative, this talk presents Willump, an optimizer for ML inference.
Apache Spark for Machine Learning with High Dimensional Labels: Spark Summit ...
This talk will cover the tools we used, the hurdles we faced and the work arounds we developed with the help from Databricks support in our attempt to build a custom machine learning model and use it to predict the TV ratings for different networks and demographics.
The Apache Spark machine learning and dataframe APIs make it incredibly easy to produce a machine learning pipeline to solve an archetypal supervised learning problem. In our applications at Cadent, we face a challenge with high dimensional labels and relatively low dimensional features; at first pass such a problem is all but intractable but thanks to a large number of historical records and the tools available in Apache Spark, we were able to construct a multi-stage model capable of forecasting with sufficient accuracy to drive the business application.
Over the course of our work we have come across many tools that made our lives easier, and others that forced work around. In this talk we will review our custom multi-stage methodology, review the challenges we faced and walk through the key steps that made our project successful.
Continuous Evaluation of Deployed Models in Production Many high-tech industr...
Many high-tech industries rely on machine-learning systems in production environments to automatically classify and respond to vast amounts of incoming data. Despite their critical roles, these systems are often not actively monitored. When a problem first arises, it may go unnoticed for some time. Once it is noticed, investigating its underlying cause is a time-consuming, manual process. Wouldn’t it be great if the model’s output were automatically monitored? If they could be visualized, sliced by different dimensions? If the system could automatically detect performance degradation and trigger alerts? In this presentation, we describe our experience from building such a core machine-learning services: Model Evaluation.
Our service provides automated, continuous evaluation of the performance of a deployed model over commonly-used metrics like the area-under-the-curve (AUC), root-mean-square-error (RMSE) etc. In addition, summary statistics about the model’s output, their distributions are also computed. The service also provides a dashboard to visualize the performance metrics, summary statistics and distributions of a model over time along with REST APIs to retrieve these metrics programmatically.
These metrics can be sliced by input features (e.g. Geography, Product type) to provide insights into model performance over different segments. The talk will describe various components that are required in building such a service and metrics of interest. Our system has a backend component built with spark on Azure Databricks. The backend can scale to analyze TBs of data to generate model evaluation metrics.
We will talk about how we modified Spark MLLib for computing AUC sliced by different dimensions and other optimizations in Spark to improve compute and performance. Our front-end and middle-tier, built with Docker and Azure Webapp provides visuals and REST APIs to retrieve the above metrics. This talk will cover various aspects of building, deploying and using the above system.
Spark Summit EU talk by Josef Habdank
Prediction as a Service with Ensemble Model Trained in SparkML on 1 billion Observed Flight Prices Daily
Huawei Advanced Data Science With Spark Streaming
This document discusses streamDM, an open source machine learning library for stream mining in Spark Streaming. It summarizes streamDM's capabilities for incremental learning on data streams using algorithms like SGD, Naive Bayes, clustering and decision trees. Examples of using streamDM in Huawei's network alarm analysis and fault localization systems are provided, demonstrating improvements in efficiency, accuracy and ability to handle large volumes of streaming data. The document encourages researchers to apply for Huawei's Innovation Research Program grants to further collaborative work on stream mining algorithms and applications.
Real time, streaming advanced analytics, approximations, and recommendations ...
Real time, streaming advanced analytics, approximations, and recommendations ...DataWorks Summit/Hadoop Summit
This document discusses various techniques for building recommendation systems using Apache Spark. It begins with an overview of scaling techniques using parallelism and composability. Various similarity measures are then covered, including Euclidean, cosine, Jaccard, and word embeddings. Recommendation approaches like item-to-item graphs and personalized PageRank are demonstrated. The document also discusses feature engineering, modeling techniques, and evaluating recommendations. Live demos are provided of word similarity, movie recommendations, sentiment analysis and more.FlinkML - Big data application meetup
My slides from the Big Data Applications meetup on 27th of July, talking about FlinkML. Also some things about open-source ML development and an illustation of interactive Flink machine learning with Apache Zeppelin.
Scalable Deep Learning Platform On Spark In Baidu
This document summarizes Baidu's work on scaling deep learning using Spark. It discusses:
1) Baidu's goals of implementing Spark ML abstractions to train deep learning models with minimal code changes and leveraging Paddle's distributed training capabilities.
2) The system architecture which uses Spark and YARN for resource management and Paddle for distributed training.
3) Experiments showing Paddle can efficiently train large models like image recognition on ImageNet using many machines and GPUs, as well as sparse models.
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Pipelines have become ubiquitous, as the need for stringing multiple functions to compose applications has gained adoption and popularity. Common pipeline abstractions such as “fit” and “transform” are even shared across divergent platforms such as Python Scikit-Learn and Apache Spark.
Scaling pipelines at the level of simple functions is desirable for many AI applications, however is not directly supported by Ray’s parallelism primitives. In this talk, Raghu will describe a pipeline abstraction that takes advantage of Ray’s compute model to efficiently scale arbitrarily complex pipeline workflows. He will demonstrate how this abstraction cleanly unifies pipeline workflows across multiple platforms such as Scikit-Learn and Spark, and achieves nearly optimal scale-out parallelism on pipelined computations.
Attendees will learn how pipelined workflows can be mapped to Ray’s compute model and how they can both unify and accelerate their pipelines with Ray.
Spark Autotuning: Spark Summit East talk by Lawrence Spracklen
While the performance delivered by Spark has enabled data scientists to undertake sophisticated analyses on big and complex data in actionable timeframes, too often, the process of manually configuring the underlying Spark jobs (including the number and size of the executors) can be a significant and time consuming undertaking. Not only it does this configuration process typically rely heavily on repeated trial-and-error, it necessitates that data scientists have a low-level understanding of Spark and detailed cluster sizing information. At Alpine Data we have been working to eliminate this requirement, and develop algorithms that can be used to automatically tune Spark jobs with minimal user involvement,
In this presentation, we discuss the algorithms we have developed and illustrate how they leverage information about the size of the data being analyzed, the analytical operations being used in the flow, the cluster size, configuration and real-time utilization, to automatically determine the optimal Spark job configuration for peak performance.
26 Trillion App Recomendations using 100 Lines of Spark Code - Ayman Farahat
This document summarizes an approach to build an app recommendation engine using collaborative filtering on data from 53,793 apps and 496 million users. It implements collaborative filtering using 100 lines of Spark code to generate low-dimensional user and app features and compute 26 trillion user-app ratings. Key aspects included using data frames, caching joined data, broadcasting app features for efficient BLAS-3 matrix multiplication, and evaluating recommendations.
Managing the Complete Machine Learning Lifecycle with MLflow
ML development brings many new complexities beyond the traditional software development lifecycle. Unlike in traditional software development, ML developers want to try multiple algorithms, tools and parameters to get the best results, and they need to track this information to reproduce work. In addition, developers need to use many distinct systems to productionize models.
To solve for these challenges, Databricks unveiled last year MLflow, an open source project that aims at simplifying the entire ML lifecycle. MLflow introduces simple abstractions to package reproducible projects, track results, and encapsulate models that can be used with many existing tools, accelerating the ML lifecycle for organizations of any size.
In the past year, the MLflow community has grown quickly: over 120 contributors from over 40 companies have contributed code to the project, and over 200 companies are using MLflow.
In this tutorial, we will show you how using MLflow can help you:
Keep track of experiments runs and results across frameworks.
Execute projects remotely on to a Databricks cluster, and quickly reproduce your runs.
Quickly productionize models using Databricks production jobs, Docker containers, Azure ML, or Amazon SageMaker.
We will demo the building blocks of MLflow as well as the most recent additions since the 1.0 release.
What you will learn:
Understand the three main components of open source MLflow (MLflow Tracking, MLflow Projects, MLflow Models) and how each help address challenges of the ML lifecycle.
How to use MLflow Tracking to record and query experiments: code, data, config, and results.
How to use MLflow Projects packaging format to reproduce runs on any platform.
How to use MLflow Models general format to send models to diverse deployment tools.
Prerequisites:
A fully-charged laptop (8-16GB memory) with Chrome or Firefox
Python 3 and pip pre-installed
Pre-Register for a Databricks Standard Trial
Basic knowledge of Python programming language
Basic understanding of Machine Learning Concepts
Auto-Pilot for Apache Spark Using Machine Learning
At Qubole, users run Spark at scale on cloud (900+ concurrent nodes). At such scale, for efficiently running SLA critical jobs, tuning Spark configurations is essential. But it continues to be a difficult undertaking, largely driven by trial and error. In this talk, we will address the problem of auto-tuning SQL workloads on Spark. The same technique can also be adapted for non-SQL Spark workloads. In our earlier work[1], we proposed a model based on simple rules and insights. It was simple yet effective at optimizing queries and finding the right instance types to run queries. However, with respect to auto tuning Spark configurations we saw scope of improvement. On exploration, we found previous works addressing auto-tuning using Machine learning techniques. One major drawback of the simple model[1] is that it cannot use multiple runs of query for improving recommendation, whereas the major drawback with Machine Learning techniques is that it lacks domain specific knowledge. Hence, we decided to combine both techniques. Our auto-tuner interacts with both models to arrive at good configurations. Once user selects a query to auto tune, the next configuration is computed from models and the query is run with it. Metrics from event log of the run is fed back to models to obtain next configuration. Auto-tuner will continue exploring good configurations until it meets the fixed budget specified by the user. We found that in practice, this method gives much better configurations compared to configurations chosen even by experts on real workload and converges soon to optimal configuration. In this talk, we will present a novel ML model technique and the way it was combined with our earlier approach. Results on real workload will be presented along with limitations and challenges in productionizing them. [1] Margoor et al,'Automatic Tuning of SQL-on-Hadoop Engines' 2018,IEEE CLOUD
How to use Apache TVM to optimize your ML models
Apache TVM is an open source machine learning compiler that distills the largest, most powerful deep learning models into lightweight software that can run on the edge. This allows the outputed model to run inference much faster on a variety of target hardware (CPUs, GPUs, FPGAs & accelerators) and save significant costs.
In this deep dive, we’ll discuss how Apache TVM works, share the latest and upcoming features and run a live demo of how to optimize a custom machine learning model.
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
As common sense would suggest, weather has a definite impact on traffic. But how much? And under what circumstances? Can we improve traffic (congestion) prediction given weather data? Predictive traffic is envisioned to significantly impact how driver’s plan their day by alerting users before they travel, find the best times to travel, and over time, learn from new IoT data such as road conditions, incidents, etc. This talk will cover the traffic prediction work conducted jointly by IBM and the traffic data provider. As a part of this work, we conducted a case study over five large metropolitans in the US, 2.58 billion traffic records and 262 million weather records, to quantify the boost in accuracy of traffic prediction using weather data. We will provide an overview of our lambda architecture with Apache Spark being used to build prediction models with weather and traffic data, and Spark Streaming used to score the model and provide real-time traffic predictions. This talk will also cover a suite of extensions to Spark to analyze geospatial and temporal patterns in traffic and weather data, as well as the suite of machine learning algorithms that were used with Spark framework. Initial results of this work were presented at the National Association of Broadcasters meeting in Las Vegas in April 2017, and there is work to scale the system to provide predictions in over a 100 cities. Audience will learn about our experience scaling using Spark in offline and streaming mode, building statistical and deep-learning pipelines with Spark, and techniques to work with geospatial and time-series data.
Enhancements on Spark SQL optimizer by Min Qiu
This document summarizes enhancements made to Spark SQL's optimizer including rule-based optimizations and cost-based optimizations. Rule-based optimizations included join condition push down through predicate rewrite and join order adjustment, as well as data volume reduction through column pruning enhancements. Cost-based optimizations leveraged statistics and histograms to select optimal join types, partitions, and join orders. Future work focuses on enumerating the full space of query plans and improving estimation accuracy.
Harnessing Spark Catalyst for Custom Data Payloads
- The document discusses harnessing Spark Catalyst to efficiently build machine learning models with earth observation data like satellite imagery.
- It describes registering a Tile user-defined type (UDT) to encode raster tile data payloads in Spark DataFrames and SQL, allowing the use of powerful Spark APIs.
- The approach prototyped leverages the GeoTrellis library to ingest raster data into RDDs of tiles, then encodes tiles as a UDT to represent them as columns in DataFrames for further analysis and machine learning.
Analyzing 2TB of Raw Trace Data from a Manufacturing Process: A First Use Cas...
As the development of semiconductor devices, manufacturing system leads to improve productivity and efficiency for wafer fabrication. Owing to such improvement, the number of wafers yielded from the fabrication process has been rapidly increasing. However, current software systems for semiconductor wafers are not aimed at processing large number of wafers. To resolve this issue, the BISTel (a world-class provider of manufacturing intelligence solutions and services for manufacturers) tries to build several products for big data such as Trace Analyzer (TA) and Map Analyzer (MA) using Apache Spark. TA is to analyze raw trace data from a manufacturing process. It captures details on all variable changes, big and small and give the traces' statistical summary (i.e.: min, max, slope, average, etc.). Several BISTel's customers, which are the top-tier semiconductor company in the world use the TA to analyze the massive raw trace data from their manufacturing process. Especially, TA is able to manage terabytes of data by applying Apache Spark's APIs. MA is an advanced pattern recognition tool that sorts wafer yield maps and automatically identify common yield loss patterns. Also, some semiconductor companies use MA to identify clustering patterns for more than 100,000 wafers, which can be considered as big data in the semiconductor area. This talk will introduce these two products which are developed based on the Apache Spark and present how to handle the large-scale semiconductor data in the aspects of software techniques.
Speakers: Seungchul Lee, Daeyoung Kim
flat_presentation_time_evolving_OD_matrix_estimation
This document summarizes a research paper on estimating time-evolving origin-destination (O-D) matrices using high-speed GPS data streams. It discusses using online machine learning techniques to build and maintain O-D matrices and histograms over time in order to model variables like travel time. A real-world case study using taxi GPS data from Porto, Portugal is also presented. Experimental results show the proposed time-evolving O-D matrix and multidimensional discretization techniques outperform static grid-based approaches and offline regression models in estimating travel times.
Spark Summit EU talk by Patrick Baier and Stanimir Dragiev
This document discusses Zalando's transition from using Scikit-learn and Python for machine learning tasks to using Spark and Scala on AWS. Zalando is a large European fashion platform that processes over 55 million orders per year. Their Payment Analytics team uses machine learning to estimate fraud risk for orders. Previously, their setup had issues with scalability, maintenance, and coupling of data and models. By moving to Spark on AWS using Scala, they were able to gain horizontal scaling, increase learning and prediction speeds by 2x, and unify their data sources. There were initial steep learning curves for Scala and debugging distributed systems, but overall Spark and AWS eliminated all of their previous pain points around scalability.
More Related Content
What's hot
Airfare prediction using Machine Learning with Apache Spark on 1 billion obse...
Prediction using Machine Learning (ML) techniques on Big Data is a computationally and system-wide challenging problem. Especially in the case when the system is processing approximately 10^9 observations per day scalability is the prime concern. In order to be able to rapidly train models covering whole multivariate space the time series vectors, which exhibit significant similarities, are clustered into the groups. Consequently the resulting vector clusters could be modelled using ML tools capable of coefficient estimation at the massive scale (Apache Spark with Scikit Learn). Presentation describes application of the Linear Regression and Support Vector Regression with Radial Basis Function kernel. This approach enables training models fast enough to complete the task within a couple of hours, allowing daily or even real time updates of the coefficients. The above machine learning framework is used to predict the airfares used as support tool for the Revenue Management systems.
Breaking Down Analytical and Computational Barriers Across the Energy Industr...
DNV GL is presenting on breaking down barriers in energy data analytics through their work in statistical computing and machine learning. They discuss using big data from smart grids and advanced metering infrastructure to better understand demand behavior and integrate disruptive technologies like electric vehicles and solar power. DNV GL also demonstrates a pilot project using Databricks and Spark to perform real-time energy forecasting and grid reliability analysis at scale.
Willump: Optimizing Feature Computation in ML Inference
Systems for performing ML inference are increasingly important, but are far slower than they could be because they use techniques designed for conventional data serving workloads, neglecting the statistical nature of ML inference. As an alternative, this talk presents Willump, an optimizer for ML inference.
Apache Spark for Machine Learning with High Dimensional Labels: Spark Summit ...
This talk will cover the tools we used, the hurdles we faced and the work arounds we developed with the help from Databricks support in our attempt to build a custom machine learning model and use it to predict the TV ratings for different networks and demographics.
The Apache Spark machine learning and dataframe APIs make it incredibly easy to produce a machine learning pipeline to solve an archetypal supervised learning problem. In our applications at Cadent, we face a challenge with high dimensional labels and relatively low dimensional features; at first pass such a problem is all but intractable but thanks to a large number of historical records and the tools available in Apache Spark, we were able to construct a multi-stage model capable of forecasting with sufficient accuracy to drive the business application.
Over the course of our work we have come across many tools that made our lives easier, and others that forced work around. In this talk we will review our custom multi-stage methodology, review the challenges we faced and walk through the key steps that made our project successful.
Continuous Evaluation of Deployed Models in Production Many high-tech industr...
Many high-tech industries rely on machine-learning systems in production environments to automatically classify and respond to vast amounts of incoming data. Despite their critical roles, these systems are often not actively monitored. When a problem first arises, it may go unnoticed for some time. Once it is noticed, investigating its underlying cause is a time-consuming, manual process. Wouldn’t it be great if the model’s output were automatically monitored? If they could be visualized, sliced by different dimensions? If the system could automatically detect performance degradation and trigger alerts? In this presentation, we describe our experience from building such a core machine-learning services: Model Evaluation.
Our service provides automated, continuous evaluation of the performance of a deployed model over commonly-used metrics like the area-under-the-curve (AUC), root-mean-square-error (RMSE) etc. In addition, summary statistics about the model’s output, their distributions are also computed. The service also provides a dashboard to visualize the performance metrics, summary statistics and distributions of a model over time along with REST APIs to retrieve these metrics programmatically.
These metrics can be sliced by input features (e.g. Geography, Product type) to provide insights into model performance over different segments. The talk will describe various components that are required in building such a service and metrics of interest. Our system has a backend component built with spark on Azure Databricks. The backend can scale to analyze TBs of data to generate model evaluation metrics.
We will talk about how we modified Spark MLLib for computing AUC sliced by different dimensions and other optimizations in Spark to improve compute and performance. Our front-end and middle-tier, built with Docker and Azure Webapp provides visuals and REST APIs to retrieve the above metrics. This talk will cover various aspects of building, deploying and using the above system.
Spark Summit EU talk by Josef Habdank
Prediction as a Service with Ensemble Model Trained in SparkML on 1 billion Observed Flight Prices Daily
Huawei Advanced Data Science With Spark Streaming
This document discusses streamDM, an open source machine learning library for stream mining in Spark Streaming. It summarizes streamDM's capabilities for incremental learning on data streams using algorithms like SGD, Naive Bayes, clustering and decision trees. Examples of using streamDM in Huawei's network alarm analysis and fault localization systems are provided, demonstrating improvements in efficiency, accuracy and ability to handle large volumes of streaming data. The document encourages researchers to apply for Huawei's Innovation Research Program grants to further collaborative work on stream mining algorithms and applications.
Real time, streaming advanced analytics, approximations, and recommendations ...
Real time, streaming advanced analytics, approximations, and recommendations ...DataWorks Summit/Hadoop Summit
This document discusses various techniques for building recommendation systems using Apache Spark. It begins with an overview of scaling techniques using parallelism and composability. Various similarity measures are then covered, including Euclidean, cosine, Jaccard, and word embeddings. Recommendation approaches like item-to-item graphs and personalized PageRank are demonstrated. The document also discusses feature engineering, modeling techniques, and evaluating recommendations. Live demos are provided of word similarity, movie recommendations, sentiment analysis and more.FlinkML - Big data application meetup
My slides from the Big Data Applications meetup on 27th of July, talking about FlinkML. Also some things about open-source ML development and an illustation of interactive Flink machine learning with Apache Zeppelin.
Scalable Deep Learning Platform On Spark In Baidu
This document summarizes Baidu's work on scaling deep learning using Spark. It discusses:
1) Baidu's goals of implementing Spark ML abstractions to train deep learning models with minimal code changes and leveraging Paddle's distributed training capabilities.
2) The system architecture which uses Spark and YARN for resource management and Paddle for distributed training.
3) Experiments showing Paddle can efficiently train large models like image recognition on ImageNet using many machines and GPUs, as well as sparse models.
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Pipelines have become ubiquitous, as the need for stringing multiple functions to compose applications has gained adoption and popularity. Common pipeline abstractions such as “fit” and “transform” are even shared across divergent platforms such as Python Scikit-Learn and Apache Spark.
Scaling pipelines at the level of simple functions is desirable for many AI applications, however is not directly supported by Ray’s parallelism primitives. In this talk, Raghu will describe a pipeline abstraction that takes advantage of Ray’s compute model to efficiently scale arbitrarily complex pipeline workflows. He will demonstrate how this abstraction cleanly unifies pipeline workflows across multiple platforms such as Scikit-Learn and Spark, and achieves nearly optimal scale-out parallelism on pipelined computations.
Attendees will learn how pipelined workflows can be mapped to Ray’s compute model and how they can both unify and accelerate their pipelines with Ray.
Spark Autotuning: Spark Summit East talk by Lawrence Spracklen
While the performance delivered by Spark has enabled data scientists to undertake sophisticated analyses on big and complex data in actionable timeframes, too often, the process of manually configuring the underlying Spark jobs (including the number and size of the executors) can be a significant and time consuming undertaking. Not only it does this configuration process typically rely heavily on repeated trial-and-error, it necessitates that data scientists have a low-level understanding of Spark and detailed cluster sizing information. At Alpine Data we have been working to eliminate this requirement, and develop algorithms that can be used to automatically tune Spark jobs with minimal user involvement,
In this presentation, we discuss the algorithms we have developed and illustrate how they leverage information about the size of the data being analyzed, the analytical operations being used in the flow, the cluster size, configuration and real-time utilization, to automatically determine the optimal Spark job configuration for peak performance.
26 Trillion App Recomendations using 100 Lines of Spark Code - Ayman Farahat
This document summarizes an approach to build an app recommendation engine using collaborative filtering on data from 53,793 apps and 496 million users. It implements collaborative filtering using 100 lines of Spark code to generate low-dimensional user and app features and compute 26 trillion user-app ratings. Key aspects included using data frames, caching joined data, broadcasting app features for efficient BLAS-3 matrix multiplication, and evaluating recommendations.
Managing the Complete Machine Learning Lifecycle with MLflow
ML development brings many new complexities beyond the traditional software development lifecycle. Unlike in traditional software development, ML developers want to try multiple algorithms, tools and parameters to get the best results, and they need to track this information to reproduce work. In addition, developers need to use many distinct systems to productionize models.
To solve for these challenges, Databricks unveiled last year MLflow, an open source project that aims at simplifying the entire ML lifecycle. MLflow introduces simple abstractions to package reproducible projects, track results, and encapsulate models that can be used with many existing tools, accelerating the ML lifecycle for organizations of any size.
In the past year, the MLflow community has grown quickly: over 120 contributors from over 40 companies have contributed code to the project, and over 200 companies are using MLflow.
In this tutorial, we will show you how using MLflow can help you:
Keep track of experiments runs and results across frameworks.
Execute projects remotely on to a Databricks cluster, and quickly reproduce your runs.
Quickly productionize models using Databricks production jobs, Docker containers, Azure ML, or Amazon SageMaker.
We will demo the building blocks of MLflow as well as the most recent additions since the 1.0 release.
What you will learn:
Understand the three main components of open source MLflow (MLflow Tracking, MLflow Projects, MLflow Models) and how each help address challenges of the ML lifecycle.
How to use MLflow Tracking to record and query experiments: code, data, config, and results.
How to use MLflow Projects packaging format to reproduce runs on any platform.
How to use MLflow Models general format to send models to diverse deployment tools.
Prerequisites:
A fully-charged laptop (8-16GB memory) with Chrome or Firefox
Python 3 and pip pre-installed
Pre-Register for a Databricks Standard Trial
Basic knowledge of Python programming language
Basic understanding of Machine Learning Concepts
Auto-Pilot for Apache Spark Using Machine Learning
At Qubole, users run Spark at scale on cloud (900+ concurrent nodes). At such scale, for efficiently running SLA critical jobs, tuning Spark configurations is essential. But it continues to be a difficult undertaking, largely driven by trial and error. In this talk, we will address the problem of auto-tuning SQL workloads on Spark. The same technique can also be adapted for non-SQL Spark workloads. In our earlier work[1], we proposed a model based on simple rules and insights. It was simple yet effective at optimizing queries and finding the right instance types to run queries. However, with respect to auto tuning Spark configurations we saw scope of improvement. On exploration, we found previous works addressing auto-tuning using Machine learning techniques. One major drawback of the simple model[1] is that it cannot use multiple runs of query for improving recommendation, whereas the major drawback with Machine Learning techniques is that it lacks domain specific knowledge. Hence, we decided to combine both techniques. Our auto-tuner interacts with both models to arrive at good configurations. Once user selects a query to auto tune, the next configuration is computed from models and the query is run with it. Metrics from event log of the run is fed back to models to obtain next configuration. Auto-tuner will continue exploring good configurations until it meets the fixed budget specified by the user. We found that in practice, this method gives much better configurations compared to configurations chosen even by experts on real workload and converges soon to optimal configuration. In this talk, we will present a novel ML model technique and the way it was combined with our earlier approach. Results on real workload will be presented along with limitations and challenges in productionizing them. [1] Margoor et al,'Automatic Tuning of SQL-on-Hadoop Engines' 2018,IEEE CLOUD
How to use Apache TVM to optimize your ML models
Apache TVM is an open source machine learning compiler that distills the largest, most powerful deep learning models into lightweight software that can run on the edge. This allows the outputed model to run inference much faster on a variety of target hardware (CPUs, GPUs, FPGAs & accelerators) and save significant costs.
In this deep dive, we’ll discuss how Apache TVM works, share the latest and upcoming features and run a live demo of how to optimize a custom machine learning model.
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
As common sense would suggest, weather has a definite impact on traffic. But how much? And under what circumstances? Can we improve traffic (congestion) prediction given weather data? Predictive traffic is envisioned to significantly impact how driver’s plan their day by alerting users before they travel, find the best times to travel, and over time, learn from new IoT data such as road conditions, incidents, etc. This talk will cover the traffic prediction work conducted jointly by IBM and the traffic data provider. As a part of this work, we conducted a case study over five large metropolitans in the US, 2.58 billion traffic records and 262 million weather records, to quantify the boost in accuracy of traffic prediction using weather data. We will provide an overview of our lambda architecture with Apache Spark being used to build prediction models with weather and traffic data, and Spark Streaming used to score the model and provide real-time traffic predictions. This talk will also cover a suite of extensions to Spark to analyze geospatial and temporal patterns in traffic and weather data, as well as the suite of machine learning algorithms that were used with Spark framework. Initial results of this work were presented at the National Association of Broadcasters meeting in Las Vegas in April 2017, and there is work to scale the system to provide predictions in over a 100 cities. Audience will learn about our experience scaling using Spark in offline and streaming mode, building statistical and deep-learning pipelines with Spark, and techniques to work with geospatial and time-series data.
Enhancements on Spark SQL optimizer by Min Qiu
This document summarizes enhancements made to Spark SQL's optimizer including rule-based optimizations and cost-based optimizations. Rule-based optimizations included join condition push down through predicate rewrite and join order adjustment, as well as data volume reduction through column pruning enhancements. Cost-based optimizations leveraged statistics and histograms to select optimal join types, partitions, and join orders. Future work focuses on enumerating the full space of query plans and improving estimation accuracy.
Harnessing Spark Catalyst for Custom Data Payloads
- The document discusses harnessing Spark Catalyst to efficiently build machine learning models with earth observation data like satellite imagery.
- It describes registering a Tile user-defined type (UDT) to encode raster tile data payloads in Spark DataFrames and SQL, allowing the use of powerful Spark APIs.
- The approach prototyped leverages the GeoTrellis library to ingest raster data into RDDs of tiles, then encodes tiles as a UDT to represent them as columns in DataFrames for further analysis and machine learning.
Analyzing 2TB of Raw Trace Data from a Manufacturing Process: A First Use Cas...
As the development of semiconductor devices, manufacturing system leads to improve productivity and efficiency for wafer fabrication. Owing to such improvement, the number of wafers yielded from the fabrication process has been rapidly increasing. However, current software systems for semiconductor wafers are not aimed at processing large number of wafers. To resolve this issue, the BISTel (a world-class provider of manufacturing intelligence solutions and services for manufacturers) tries to build several products for big data such as Trace Analyzer (TA) and Map Analyzer (MA) using Apache Spark. TA is to analyze raw trace data from a manufacturing process. It captures details on all variable changes, big and small and give the traces' statistical summary (i.e.: min, max, slope, average, etc.). Several BISTel's customers, which are the top-tier semiconductor company in the world use the TA to analyze the massive raw trace data from their manufacturing process. Especially, TA is able to manage terabytes of data by applying Apache Spark's APIs. MA is an advanced pattern recognition tool that sorts wafer yield maps and automatically identify common yield loss patterns. Also, some semiconductor companies use MA to identify clustering patterns for more than 100,000 wafers, which can be considered as big data in the semiconductor area. This talk will introduce these two products which are developed based on the Apache Spark and present how to handle the large-scale semiconductor data in the aspects of software techniques.
Speakers: Seungchul Lee, Daeyoung Kim
What's hot (20)
Airfare prediction using Machine Learning with Apache Spark on 1 billion obse...
Airfare prediction using Machine Learning with Apache Spark on 1 billion obse...
Breaking Down Analytical and Computational Barriers Across the Energy Industr...
Breaking Down Analytical and Computational Barriers Across the Energy Industr...
Willump: Optimizing Feature Computation in ML Inference
Willump: Optimizing Feature Computation in ML Inference
Apache Spark for Machine Learning with High Dimensional Labels: Spark Summit ...
Apache Spark for Machine Learning with High Dimensional Labels: Spark Summit ...
Continuous Evaluation of Deployed Models in Production Many high-tech industr...
Continuous Evaluation of Deployed Models in Production Many high-tech industr...
Real time, streaming advanced analytics, approximations, and recommendations ...
Real time, streaming advanced analytics, approximations, and recommendations ...
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Spark Autotuning: Spark Summit East talk by Lawrence Spracklen
Spark Autotuning: Spark Summit East talk by Lawrence Spracklen
26 Trillion App Recomendations using 100 Lines of Spark Code - Ayman Farahat
26 Trillion App Recomendations using 100 Lines of Spark Code - Ayman Farahat
Managing the Complete Machine Learning Lifecycle with MLflow
Managing the Complete Machine Learning Lifecycle with MLflow
Auto-Pilot for Apache Spark Using Machine Learning
Auto-Pilot for Apache Spark Using Machine Learning
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
Harnessing Spark Catalyst for Custom Data Payloads
Harnessing Spark Catalyst for Custom Data Payloads
Analyzing 2TB of Raw Trace Data from a Manufacturing Process: A First Use Cas...
Analyzing 2TB of Raw Trace Data from a Manufacturing Process: A First Use Cas...
Viewers also liked
flat_presentation_time_evolving_OD_matrix_estimation
This document summarizes a research paper on estimating time-evolving origin-destination (O-D) matrices using high-speed GPS data streams. It discusses using online machine learning techniques to build and maintain O-D matrices and histograms over time in order to model variables like travel time. A real-world case study using taxi GPS data from Porto, Portugal is also presented. Experimental results show the proposed time-evolving O-D matrix and multidimensional discretization techniques outperform static grid-based approaches and offline regression models in estimating travel times.
Spark Summit EU talk by Patrick Baier and Stanimir Dragiev
This document discusses Zalando's transition from using Scikit-learn and Python for machine learning tasks to using Spark and Scala on AWS. Zalando is a large European fashion platform that processes over 55 million orders per year. Their Payment Analytics team uses machine learning to estimate fraud risk for orders. Previously, their setup had issues with scalability, maintenance, and coupling of data and models. By moving to Spark on AWS using Scala, they were able to gain horizontal scaling, increase learning and prediction speeds by 2x, and unify their data sources. There were initial steep learning curves for Scala and debugging distributed systems, but overall Spark and AWS eliminated all of their previous pain points around scalability.
Spark Summit EU talk by Oscar Castaneda
The document discusses transforming Excel spreadsheets into Spark DataFrames by automatically translating Excel formulas into Spark code. It presents a program transformation pipeline that takes Excel formulas, parses them using a grammar and parser to generate a parse tree, and then generates Spark code from the parse tree. Key aspects covered include using an existing grammar and parser called XLParser to parse Excel formulas, treating Excel as a domain-specific language, and generating code by writing a pretty printer for the target Spark language. The talk concludes with a demonstration of the code generation approach.
The Spark (R)evolution in The Netherlands
The document discusses Spark and big data technologies. It announces the Spark Summit Europe 2016 conference in the Netherlands to discuss Spark, machine learning, cloud platforms, APIs, and customer data. The document encourages attending the conference and notes the company is hiring.
Spark Summit EU talk by Heiko Korndorf
Heiko Korndorf presented on scaling SparkR in production and lessons from real-world projects. The presentation covered classifying SparkR as both a data science and data engineering tool. It discussed SparkR's architecture in versions 1.x and 2.x, approaches to parallelizing R code with Spark 1.5/1.6 and YARN, and dynamic R deployment including dependencies. Advanced techniques like 2-level parallelization using GPGPU were also presented. The talk concluded with an outlook on further integrating data engineering and data science and new technical approaches like simplifying data pipelines and moving calculations to GPUs.
Spark Summit EU talk by Jorg Schad
This document discusses containers and virtual machines. It explains that containers provide a lightweight virtualization method that isolates applications but shares the host operating system kernel. Containers use resource isolation features like cgroups and namespaces to limit CPU, memory, storage, and networking usage. In contrast, virtual machines run their own full operating system and provide stronger isolation but are more resource intensive.
Spark Summit EU talk by Erwin Datema and Roeland van Ham
The document discusses KeyGene's use of Apache Spark for high-throughput genomics data analysis. KeyGene is a crop innovation company that analyzes genomic data from thousands of plants to improve crop traits like yield and quality. They previously used conventional HPC clusters for genomics pipelines but found Spark enabled more interactive analysis. KeyGene developed a "Sparkified" genomics pipeline using tools like BWA, GATK and their own Guacamole variant caller. This allowed interactive variant selection and GWAS using Spark SQL queries, demonstrating Spark is well-suited for interactive plant genomics analysis.
Democratizing AI with Apache Spark
This document discusses democratizing AI using Apache Spark. It summarizes that while AI is advancing rapidly, it has not been fully democratized due to challenges with data management, developing productive teams, and establishing production-ready applications. Databricks aims to close these gaps with its just-in-time data platform that provides integrated workspaces, automated Spark management, and supports deep learning use cases across industries.
Spark Summit EU talk by Sudeep Das and Aish Faenton
The researcher wants to plot popular Netflix titles by country using Spark DataFrames. Doing this manually would be painful, requiring filtering, aggregation, faceting and other operations on the DataFrame before plotting. Vegas provides a declarative grammar for statistical visualization in Scala, allowing the user to specify what to do with the data without worrying about how. It works by compiling a Scala DSL to Vega-Lite JSON, then to Vega JSON and D3JS code to generate the visualization.
Spark Summit EU talk by Luc Bourlier
This document discusses dynamic resource allocation in Spark clusters. It explains how Spark can add or remove executors from a cluster based on workload to optimize resource usage for jobs with variable loads. It also describes how the external shuffle service moves shuffle data management out of executors to improve performance and fault tolerance. The document provides details on configuring dynamic allocation and the external shuffle service and demonstrates dynamic allocation in action. It also discusses applying these techniques to Spark Streaming workloads.
Spark Summit EU talk by Dean Wampler
Dean Wampler gave a talk at the Spark Summit Europe 2016 titled "Just Enough Scala for Spark". He demonstrated coding a demo in Scala for Spark, based on his free online tutorial. The tutorial and live coding showed attendees the essential Scala concepts for working with Spark. Wampler also provided resources for additional Scala and Spark help.
Spark Summit EU talk by Sital Kedia
This document summarizes a case study of using Apache Spark at scale to process over 60 TB of data at Facebook. It describes the previous implementation using Hive, which involved many small jobs that were unmanageable and slow. The new Spark implementation processes the entire dataset in a single job with two stages, shuffling over 90 TB of intermediate data. It provides performance comparisons showing significant reductions in CPU time, latency, and resource usage. It also details reliability and performance improvements made to Spark, such as fixing memory leaks, enabling seamless cluster restarts, and reducing shuffle write latency. Configuration tuning tips are provided to optimize memory usage and shuffle processing.
Spark Summit EU talk by Ruben Pulido and Behar Veliqi
The document describes IBM's transition from a single-tenant Hadoop architecture for their Watson Analytics for Social Media product to a multitenant Apache Spark architecture supporting over 3000 tenants. Key aspects of the new architecture included splitting analytics into tenant-specific and language-specific components, aggregating social media feeds from all tenants into a single stream for processing, and removing tenant state from processing components to enable low-latency switching between tenants. This resulted in a scalable, robust pipeline for real-time social media analytics based on Spark, Kafka and Zookeeper.
Emerging Business Models for the Open Data Industry and Open Data Value Capab...
This presentation presents a comprehensive model and the relations between Innovation, Business Model, Business/Industry, Open Data and Entrepreneurship. Later slides presents Open Data Business Model (the 6-Value Model) and Open Data Capability Matrix.
MmmooOgle: From Big Data to Decisions for Dairy Cows
The document discusses the benefits of exercise for mental health. Regular physical activity can help reduce anxiety and depression and improve mood and cognitive functioning. Exercise causes chemical changes in the brain that may help protect against mental illness and improve symptoms.
Spark Summit EU talk by Ted Malaska
This document summarizes a presentation about unit testing Spark applications. The presentation discusses why it is important to run Spark locally and as unit tests instead of just on a cluster for faster feedback and easier debugging. It provides examples of how to run Spark locally in an IDE and as ScalaTest unit tests, including how to create test RDDs and DataFrames and supply test data. It also discusses testing concepts for streaming applications, MLlib, GraphX, and integration testing with technologies like HBase and Kafka.
The Next AMPLab: Real-Time, Intelligent, and Secure Computing
The document discusses the goals of establishing a new research lab called RISELab to develop a secure real-time decision stack that can enable real-time decisions on live data with strong security guarantees. It outlines some of the challenges in building such a system and presents early work on Drizzle, a low-latency streaming engine, and Opaque, which leverages hardware enclaves to provide encryption and hide data access patterns. The goal is to build an open source platform and tools over the next 5 years to enable applications requiring sophisticated, accurate, and robust real-time decisions on private data.
Simplifying Big Data Applications with Apache Spark 2.0
Apache Spark 2.0 is a major new release that simplifies the Spark API and improves performance. Some key points:
1) It remains highly compatible with Spark 1.x while building on lessons learned to simplify the API with over 2000 patches from 280 contributors.
2) It introduces structured APIs like DataFrames that allow Spark to optimize queries via whole-stage code generation, providing up to 10x performance gains.
3) It launches a new higher-level streaming API called Structured Streaming that allows developers to write streaming jobs that behave like batch jobs and integrate easily with static data and batch jobs.
Spark Summit EU talk by Qifan Pu
This document discusses improving Spark performance on many-core machines by implementing an in-memory shuffle. It finds that Spark's disk-based shuffle is inefficient on such hardware due to serialization, I/O contention, and garbage collection overhead. An in-memory shuffle avoids these issues by copying data directly between memory pages. This results in a 31% median performance improvement on TPC-DS queries compared to the default Spark shuffle. However, more work is needed to address other performance bottlenecks and extend the in-memory shuffle to multi-node clusters.
Viewers also liked (19)
flat_presentation_time_evolving_OD_matrix_estimation
flat_presentation_time_evolving_OD_matrix_estimation
Spark Summit EU talk by Patrick Baier and Stanimir Dragiev
Spark Summit EU talk by Patrick Baier and Stanimir Dragiev
Spark Summit EU talk by Erwin Datema and Roeland van Ham
Spark Summit EU talk by Erwin Datema and Roeland van Ham
Spark Summit EU talk by Sudeep Das and Aish Faenton
Spark Summit EU talk by Sudeep Das and Aish Faenton
Spark Summit EU talk by Ruben Pulido and Behar Veliqi
Spark Summit EU talk by Ruben Pulido and Behar Veliqi
Emerging Business Models for the Open Data Industry and Open Data Value Capab...
Emerging Business Models for the Open Data Industry and Open Data Value Capab...
MmmooOgle: From Big Data to Decisions for Dairy Cows
MmmooOgle: From Big Data to Decisions for Dairy Cows
The Next AMPLab: Real-Time, Intelligent, and Secure Computing
The Next AMPLab: Real-Time, Intelligent, and Secure Computing
Simplifying Big Data Applications with Apache Spark 2.0
Simplifying Big Data Applications with Apache Spark 2.0
Similar to Spark Summit EU talk by Javier Aguedes
Blu Fax Recent Advances 2 24 2011
This document provides an overview of Traffax Inc.'s Bluetooth-based traffic monitoring equipment and software called BluFax. It describes how BluFax uses Bluetooth detectors to capture unique device IDs from passing vehicles to calculate travel times and origin-destination patterns. The document outlines BluFax's applications for traffic studies, work zone monitoring, and real-time traffic monitoring. It also summarizes the components, software, sample sizes needed, and provides examples of BluFax installations.
Secure Benchmarking
This document discusses secure benchmarking and its applications. Secure benchmarking allows companies to anonymously share and compare sensitive performance data through encryption techniques. This allows benchmarking without revealing raw confidential data. The document provides examples of secure benchmarking for transportation procurement bids and shipment costs. Regression analysis is performed on encrypted data to identify cost structures and compare performance between companies and markets over time. Insights from benchmarking can guide strategy and value creation for shippers and carriers.
Transport for London - London's Operations Digital Twin
1) London Transport is developing an Operations Digital Twin to provide a real-time simulation of traffic conditions on London's roads.
2) The Digital Twin integrates multiple real-time and historical data sources into a common framework and graph database aligned by road links and time.
3) This allows the Digital Twin to identify traffic incidents and disruptions, help manage traffic, and support planning and analysis across Transport for London.
CH7-INFORMATION MANAGEMENT TECHNOLOGY.pdf
An Intelligent Transportation System (ITS) uses advanced technologies to provide innovative services that improve transportation network usage and safety. Some key applications of IT in supply chains include electronic data interchange, barcoding, and track and trace technologies. Emerging technologies like RFID tags and internet-based applications may further improve areas like visibility, collaboration, and optimization. Overall, IT applications in transportation and supply chain management can generate benefits like reduced costs, improved efficiency, increased information sharing, and enhanced responsiveness.
KTH-Texxi Project 2010
This document summarizes a study on implementing demand-responsive transit (DRT) services in Stockholm, Sweden. The researchers:
1. Developed a methodology to analyze existing trip data, demographics, and other factors to model demand and identify optimal areas for a DRT pilot project. This included developing a gravity model to estimate potential trips.
2. Applied the methodology to identify the top 3 candidate clusters for the pilot: Sollentuna, Hammarbyhöjden/Björkhagen, and Södertälje.
3. Discussed opportunities to improve the analysis, such as incorporating additional data sources and parameters or using more advanced clustering algorithms.
Analyzing Streets, Traffic, and Drivetimes in MapInfo Pro
Street networks connect our world. From understanding where a customer lives in relation to your store, to identifying the fastest route to take in an emergency, street data informs the spatial analytics that power your business. This street network analysis is essential for analyzing how people, places, and things are connected across space. Precisely Streets data is designed to work seamlessly with MapInfo Pro, enabling you to derive network and accessibility insights accurately and efficiently.
Join Precisely Streets product manager, Dan Edwards as he describes the creation and curation of street data for MapInfo Pro, and how data analysts worldwide are leveraging spatial insights in their industry-specific workflows.
Learn how to:
- Understand the accessibility of retail locations, define store catchments, and manage logistics and deliveries
- Find the most efficient, effective, and economic routes to deliver and transport goods
- Analyze the time it takes to access particular locations, and the impact of planning applications and road works
- Understand the local environment and the impact decisions may have on specific places and individuals
Driving Efficiency with Splunk Cloud at Gatwick Airport
Gatwick Airport, the busiest single runway airport in the world, needed to ensure a high degree of efficiency for a record-breaking 925 daily flights and 38 million annual passengers. This presentation covers how they:
- Combine historical fact with "in the moment" data and events to predict success or failure, enabling the operation to prevent issues before they occur
- Support other organisations (e.g., airlines and ground handlers) with dashboards to improve their performance
- Moved from "how did we do?" to "how are we doing?" and are on the edge of answering "How will we do?”
- Plan to expand the use of Splunk Cloud in the future: tracking travel disruption, predicting passenger flow and getting real-time feedback via social media monitoring
Also, learn why a cloud solution gives Gatwick Airport the agility and scalability to achieve what they need.
Road hotspot warning system based cooperative concept
This presentation describe a Intelligent Transport System designed for developing traffic safety at road hotspot.
Transport for London - Using Data to Keep London Moving
Transport for London (TfL) and WSO2 have been working together on integration projects that focus on making the most use of London road networks and public transport. TfL collects and brings together a wide range of data from multiple disparate systems. Then they use this data for operational purposes and also make it open and available to everyone in real-time.
This webinar will explore how TfL, with the help of the WSO2 analytics platform,
Uses IoT and real-time streaming techniques to understand the current and predicted transport network status.
Innovates heterogeneous data sources by combining the TfL Unified API with traffic, air quality, and passenger flow data
Provides better travel time and transit suggestions for Londoners.
Real time traffic management - challenges and solutions
Presentation from NORTHMOST - a new biannual series of meetings on the topic of mathematical modelling in transport.
Hosted at its.leeds.ac.uk, NORTHMOST 01 focussed on academic research, to encourage networking and collaboration between academics interested in the methodological development of mathematical modelling applied to transport.
The focus of the meetings will alternate; NORTHMOST 02 - planned for Spring 2017 - will be led by practitioners who are modelling experts. Practitioners will give presentations, with academic researchers in the audience. In addition to giving a forum for expert practitioners to meet and share best practice, a key aim of the series is to close the gap between research and practice, establishing a feedback loop to communicate the needs of practitioners to those working in university research.
Ite Vancourver Nelson Corridors
This document discusses the use of Bluetooth-based data collection using BluFax technology for urban corridor studies. BluFax devices record the unique Bluetooth ID of detectable devices passing within their detection zone, along with a timestamp and location data. The data is processed by BluStat software to output graphics and results like travel time, speed, and origin-destination pairs between stations. Early applications focused on travel time studies, but recent applications now combine travel time with origin-destination data for uses like signal timing studies, interchange evaluations, and pedestrian studies. Key benefits are the ability to track Bluetooth signals throughout a study area to obtain travel time and origin-destination data.
MMauch HOC presentation-oct-04-2013
PSU Friday Transportation Seminar 10/4/2013, featuring Michael Mauch of DKS Associates: Real-world traffic trends observed in PORTAL and INRIX traffic data are used to expand the performance measures that can be obtained from Portland Metro's travel demand model to include the number of hours of congestion that can be expected during a typical weekday and travel time reliability measures for congested freeway corridors.
Analyze performance and operations of truck fleets in real time
Altair’s event processing and data visualization tools enable fleet operators to analyze critical data streaming in from sensors and other sources. This real-time visibility into vehicle and driver performance helps reduce operating costs, improve driver safety, and increase fleet productivity. Analysts can display maps showing the current position of all assets, examine route deviations, program alerts on any set of parameters, and compare drivers’ behavior. Analysts can design and modify analytical dashboards as needed without writing a single line of code.
Capstone Poster Final
The document summarizes an autonomous vehicle learning system (AVLS) that uses computer vision techniques like object detection and line detection for navigation. It describes the robotic platform, control system, and testing methods. The goal of the AVLS is to design and control an autonomous vehicle that can use computer vision, networking, and route optimization to navigate simulated city environments and detect traffic signs and lanes.
CAST Terminal
CAST Terminal is a leading passenger terminal simulation system that offers flexible and realistic modeling of passenger flows. It can be used to determine facility requirements, validate new terminal designs, and optimize existing terminals and processes. The simulation allows analyzing an entire airport or individual parts in detail to plan infrastructure, calculate facility needs, identify bottlenecks, and improve capacity without reducing operational quality. The user-friendly interface allows fast modeling of complex real-world terminals.
20180308 coptra wac_pub
Presentation performed by COPTRA during the World ATM Congress 2018 during the DATA DRIVEN ATM: GOING DIGITAL! Workshop
A Big Data Telco Solution by Dr. Laura Wynter
Presented during the WKWSCI Symposium 2014
21 March 2014
Marina Bay Sands Expo and Convention Centre
Organized by the Wee Kim Wee School of Communication and Information at Nanyang Technological University
Traffic data fusion methodology
1) The document discusses the development of a traffic data fusion methodology that intelligently combines multiple data sources to obtain more accurate and complete traffic information than any single source can provide alone.
2) Different data sources have strengths and weaknesses depending on traffic conditions, and understanding these strengths and weaknesses helps to resolve differences between sources.
3) Intelligent data fusion using quality measures from multiple sources can provide near-complete traffic coverage and high quality information, improving transport network management and planning.
Big Data Pipelines and Machine Learning at Uber
How does the Marketplace org at Uber ingest, store, query and analyze big data? What does our ML infrastructure look like?
Machine Learning Approach to Report Prioritization with an ...
The document describes a machine learning approach to prioritize reports in a vehicle communication network. Reports contain information like travel times and are stored in local databases with limited size. A model is developed to learn the relevance of reports using attributes like age and road type. The model is applied to prioritize travel time reports to disseminate updated speeds to vehicles for route planning. Evaluation shows logistic regression accurately predicts relevant reports to change routes.
Similar to Spark Summit EU talk by Javier Aguedes (20)
Transport for London - London's Operations Digital Twin
Transport for London - London's Operations Digital Twin
Analyzing Streets, Traffic, and Drivetimes in MapInfo Pro
Analyzing Streets, Traffic, and Drivetimes in MapInfo Pro
Driving Efficiency with Splunk Cloud at Gatwick Airport
Driving Efficiency with Splunk Cloud at Gatwick Airport
Road hotspot warning system based cooperative concept
Road hotspot warning system based cooperative concept
Transport for London - Using Data to Keep London Moving
Transport for London - Using Data to Keep London Moving
Real time traffic management - challenges and solutions
Real time traffic management - challenges and solutions
Analyze performance and operations of truck fleets in real time
Analyze performance and operations of truck fleets in real time
Machine Learning Approach to Report Prioritization with an ...
Machine Learning Approach to Report Prioritization with an ...
More from Spark Summit
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
In this session we will present a Configurable FPGA-Based Spark SQL Acceleration Architecture. It is target to leverage FPGA highly parallel computing capability to accelerate Spark SQL Query and for FPGA’s higher power efficiency than CPU we can lower the power consumption at the same time. The Architecture consists of SQL query decomposition algorithms, fine-grained FPGA based Engine Units which perform basic computation of sub string, arithmetic and logic operations. Using SQL query decomposition algorithm, we are able to decompose a complex SQL query into basic operations and according to their patterns each is fed into an Engine Unit. SQL Engine Units are highly configurable and can be chained together to perform complex Spark SQL queries, finally one SQL query is transformed into a Hardware Pipeline. We will present the performance benchmark results comparing the queries with FGPA-Based Spark SQL Acceleration Architecture on XEON E5 and FPGA to the ones with Spark SQL Query on XEON E5 with 10X ~ 100X improvement and we will demonstrate one SQL query workload from a real customer.
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
In this talk, we’ll present techniques for visualizing large scale machine learning systems in Spark. These are techniques that are employed by Netflix to understand and refine the machine learning models behind Netflix’s famous recommender systems that are used to personalize the Netflix experience for their 99 millions members around the world. Essential to these techniques is Vegas, a new OSS Scala library that aims to be the “missing MatPlotLib” for Spark/Scala. We’ll talk about the design of Vegas and its usage in Scala notebooks to visualize Machine Learning Models.
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
This presentation introduces how we design and implement a real-time processing platform using latest Spark Structured Streaming framework to intelligently transform the production lines in the manufacturing industry. In the traditional production line there are a variety of isolated structured, semi-structured and unstructured data, such as sensor data, machine screen output, log output, database records etc. There are two main data scenarios: 1) Picture and video data with low frequency but a large amount; 2) Continuous data with high frequency. They are not a large amount of data per unit. However the total amount of them is very large, such as vibration data used to detect the quality of the equipment. These data have the characteristics of streaming data: real-time, volatile, burst, disorder and infinity. Making effective real-time decisions to retrieve values from these data is critical to smart manufacturing. The latest Spark Structured Streaming framework greatly lowers the bar for building highly scalable and fault-tolerant streaming applications. Thanks to the Spark we are able to build a low-latency, high-throughput and reliable operation system involving data acquisition, transmission, analysis and storage. The actual user case proved that the system meets the needs of real-time decision-making. The system greatly enhance the production process of predictive fault repair and production line material tracking efficiency, and can reduce about half of the labor force for the production lines.
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
Graph is on the rise and it’s time to start learning about scalable graph analytics! In this session we will go over two Spark-based Graph Analytics frameworks: Tinkerpop and GraphFrames. While both frameworks can express very similar traversals, they have different performance characteristics and APIs. In this Deep-Dive by example presentation, we will demonstrate some common traversals and explain how, at a Spark level, each traversal is actually computed under the hood! Learn both the fluent Gremlin API as well as the powerful GraphFrame Motif api as we show examples of both simultaneously. No need to be familiar with Graphs or Spark for this presentation as we’ll be explaining everything from the ground up!
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
Building accurate machine learning models has been an art of data scientists, i.e., algorithm selection, hyper parameter tuning, feature selection and so on. Recently, challenges to breakthrough this “black-arts” have got started. In cooperation with our partner, NEC Laboratories America, we have developed a Spark-based automatic predictive modeling system. The system automatically searches the best algorithm, parameters and features without any manual work. In this talk, we will share how the automation system is designed to exploit attractive advantages of Spark. The evaluation with real open data demonstrates that our system can explore hundreds of predictive models and discovers the most accurate ones in minutes on a Ultra High Density Server, which employs 272 CPU cores, 2TB memory and 17TB SSD in 3U chassis. We will also share open challenges to learn such a massive amount of models on Spark, particularly from reliability and stability standpoints. This talk will cover the presentation already shown on Spark Summit SF’17 (#SFds5) but from more technical perspective.
Apache Spark and Tensorflow as a Service with Jim Dowling
In Sweden, from the Rise ICE Data Center at www.hops.site, we are providing to reseachers both Spark-as-a-Service and, more recently, Tensorflow-as-a-Service as part of the Hops platform. In this talk, we examine the different ways in which Tensorflow can be included in Spark workflows, from batch to streaming to structured streaming applications. We will analyse the different frameworks for integrating Spark with Tensorflow, from Tensorframes to TensorflowOnSpark to Databrick’s Deep Learning Pipelines. We introduce the different programming models supported and highlight the importance of cluster support for managing different versions of python libraries on behalf of users. We will also present cluster management support for sharing GPUs, including Mesos and YARN (in Hops Hadoop). Finally, we will perform a live demonstration of training and inference for a TensorflowOnSpark application written on Jupyter that can read data from either HDFS or Kafka, transform the data in Spark, and train a deep neural network on Tensorflow. We will show how to debug the application using both Spark UI and Tensorboard, and how to examine logs and monitor training.
Apache Spark and Tensorflow as a Service with Jim Dowling
In Sweden, from the Rise ICE Data Center at www.hops.site, we are providing to reseachers both Spark-as-a-Service and, more recently, Tensorflow-as-a-Service as part of the Hops platform. In this talk, we examine the different ways in which Tensorflow can be included in Spark workflows, from batch to streaming to structured streaming applications. We will analyse the different frameworks for integrating Spark with Tensorflow, from Tensorframes to TensorflowOnSpark to Databrick’s Deep Learning Pipelines. We introduce the different programming models supported and highlight the importance of cluster support for managing different versions of python libraries on behalf of users. We will also present cluster management support for sharing GPUs, including Mesos and YARN (in Hops Hadoop). Finally, we will perform a live demonstration of training and inference for a TensorflowOnSpark application written on Jupyter that can read data from either HDFS or Kafka, transform the data in Spark, and train a deep neural network on Tensorflow. We will show how to debug the application using both Spark UI and Tensorboard, and how to examine logs and monitor training.
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
With the rapid growth of available datasets, it is imperative to have good tools for extracting insight from big data. The Spark ML library has excellent support for performing at-scale data processing and machine learning experiments, but more often than not, Data Scientists find themselves struggling with issues such as: low level data manipulation, lack of support for image processing, text analytics and deep learning, as well as the inability to use Spark alongside other popular machine learning libraries. To address these pain points, Microsoft recently released The Microsoft Machine Learning Library for Apache Spark (MMLSpark), an open-source machine learning library built on top of SparkML that seeks to simplify the data science process and integrate SparkML Pipelines with deep learning and computer vision libraries such as the Microsoft Cognitive Toolkit (CNTK) and OpenCV. With MMLSpark, Data Scientists can build models with 1/10th of the code through Pipeline objects that compose seamlessly with other parts of the SparkML ecosystem. In this session, we explore some of the main lessons learned from building MMLSpark. Join us if you would like to know how to extend Pipelines to ensure seamless integration with SparkML, how to auto-generate Python and R wrappers from Scala Transformers and Estimators, how to integrate and use previously non-distributed libraries in a distributed manner and how to efficiently deploy a Spark library across multiple platforms.
Next CERN Accelerator Logging Service with Jakub Wozniak
The Next Accelerator Logging Service (NXCALS) is a new Big Data project at CERN aiming to replace the existing Oracle-based service.
The main purpose of the system is to store and present Controls/Infrastructure related data gathered from thousands of devices in the whole accelerator complex.
The data is used to operate the machines, improve their performance and conduct studies for new beam types or future experiments.
During this talk, Jakub will speak about NXCALS requirements and design choices that lead to the selected architecture based on Hadoop and Spark. He will present the Ingestion API, the abstractions behind the Meta-data Service and the Spark-based Extraction API where simple changes to the schema handling greatly improved the overall usability of the system. The system itself is not CERN specific and can be of interest to other companies or institutes confronted with similar Big Data problems.
Powering a Startup with Apache Spark with Kevin Kim
In Between (A mobile App for couples, downloaded 20M in Global), from daily batch for extracting metrics, analysis and dashboard. Spark is widely used by engineers and data analysts in Between, thanks to the performance and expendability of Spark, data operating has become extremely efficient. Entire team including Biz Dev, Global Operation, Designers are enjoying data results so Spark is empowering entire company for data driven operation and thinking. Kevin, Co-founder and Data Team leader of Between will be presenting how things are going in Between. Listeners will know how small and agile team is living with data (how we build organization, culture and technical base) after this presentation.
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
As common sense would suggest, weather has a definite impact on traffic. But how much? And under what circumstances? Can we improve traffic (congestion) prediction given weather data? Predictive traffic is envisioned to significantly impact how driver’s plan their day by alerting users before they travel, find the best times to travel, and over time, learn from new IoT data such as road conditions, incidents, etc. This talk will cover the traffic prediction work conducted jointly by IBM and the traffic data provider. As a part of this work, we conducted a case study over five large metropolitans in the US, 2.58 billion traffic records and 262 million weather records, to quantify the boost in accuracy of traffic prediction using weather data. We will provide an overview of our lambda architecture with Apache Spark being used to build prediction models with weather and traffic data, and Spark Streaming used to score the model and provide real-time traffic predictions. This talk will also cover a suite of extensions to Spark to analyze geospatial and temporal patterns in traffic and weather data, as well as the suite of machine learning algorithms that were used with Spark framework. Initial results of this work were presented at the National Association of Broadcasters meeting in Las Vegas in April 2017, and there is work to scale the system to provide predictions in over a 100 cities. Audience will learn about our experience scaling using Spark in offline and streaming mode, building statistical and deep-learning pipelines with Spark, and techniques to work with geospatial and time-series data.
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
In many cases, Big Data becomes just another buzzword because of the lack of tools that can support both the technological requirements for developing and deploying of the projects and/or the fluency of communication between the different profiles of people involved in the projects.
In this talk, we will present Moriarty, a set of tools for fast prototyping of Big Data applications that can be deployed in an Apache Spark environment. These tools support the creation of Big Data workflows using the already existing functional blocks or supporting the creation of new functional blocks. The created workflow can then be deployed in a Spark infrastructure and used through a REST API.
For better understanding of Moriarty, the prototyping process and the way it hides the Spark environment to the Big Data users and developers, we will present it together with a couple of examples based on a Industry 4.0 success cases and other on a logistic success case.
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
Nielsen used Databricks to test new digital advertising rating methodologies on a large scale. Databricks allowed Nielsen to run analyses on thousands of advertising campaigns using both small panel data and large production data. This identified edge cases and performance gains faster than traditional methods. Using Databricks reduced the time required to test and deploy improved rating methodologies to benefit Nielsen's clients.
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Data lineage tracking is one of the significant problems that financial institutions face when using modern big data tools. This presentation describes Spline – a data lineage tracking and visualization tool for Apache Spark. Spline captures and stores lineage information from internal Spark execution plans and visualizes it in a user-friendly manner.
Goal Based Data Production with Sim Simeonov
Since the invention of SQL and relational databases, data production has been about specifying how data is transformed through queries. While Apache Spark can certainly be used as a general distributed query engine, the power and granularity of Spark’s APIs enables a revolutionary increase in data engineering productivity: goal-based data production. Goal-based data production concerns itself with specifying WHAT the desired result is, leaving the details of HOW the result is achieved to a smart data warehouse running on top of Spark. That not only substantially increases productivity, but also significantly expands the audience that can work directly with Spark: from developers and data scientists to technical business users. With specific data and architecture patterns spanning the range from ETL to machine learning data prep and with live demos, this session will demonstrate how Spark users can gain the benefits of goal-based data production.
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Have you imagined a simple machine learning solution able to prevent revenue leakage and monitor your distributed application? To answer this question, we offer a practical and a simple machine learning solution to create an intelligent monitoring application based on simple data analysis using Apache Spark MLlib. Our application uses linear regression models to make predictions and check if the platform is experiencing any operational problems that can impact in revenue losses. The application monitor distributed systems and provides notifications stating the problem detected, that way users can operate quickly to avoid serious problems which directly impact the company’s revenue and reduce the time for action. We will present an architecture for not only a monitoring system, but also an active actor for our outages recoveries. At the end of the presentation you will have access to our training program source code and you will be able to adapt and implement in your company. This solution already helped to prevent about US$3mi in losses last year.
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Getting Ready to use Redis with Apache Spark is a technical tutorial designed to address integrating Redis with an Apache Spark deployment to increase the performance of serving complex decision models. To set the context for the session, we start with a quick introduction to Redis and the capabilities Redis provides. We cover the basic data types provided by Redis and cover the module system. Using an ad serving use-case, we look at how Redis can improve the performance and reduce the cost of using complex ML-models in production. Attendees will be guided through the key steps of setting up and integrating Redis with Spark, including how to train a model using Spark then load and serve it using Redis, as well as how to work with the Spark Redis module. The capabilities of the Redis Machine Learning Module (redis-ml) will be discussed focusing primarily on decision trees and regression (linear and logistic) with code examples to demonstrate how to use these feature. At the end of the session, developers should feel confident building a prototype/proof-of-concept application using Redis and Spark. Attendees will understand how Redis complements Spark and how to use Redis to serve complex, ML-models with high performance.
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Here we present a general supervised framework for record deduplication and author-disambiguation via Spark. This work differentiates itself by – Application of Databricks and AWS makes this a scalable implementation. Compute resources are comparably lower than traditional legacy technology using big boxes 24/7. Scalability is crucial as Elsevier’s Scopus data, the biggest scientific abstract repository, covers roughly 250 million authorships from 70 million abstracts covering a few hundred years. – We create a fingerprint for each content by deep learning and/or word2vec algorithms to expedite pairwise similarity calculation. These encoders substantially reduce compute time while maintaining semantic similarity (unlike traditional TFIDF or predefined taxonomies). We will briefly discuss how to optimize word2vec training with high parallelization. Moreover, we show how these encoders can be used to derive a standard representation for all our entities namely such as documents, authors, users, journals, etc. This standard representation can simplify the recommendation problem into a pairwise similarity search and hence it can offer a basic recommender for cross-product applications where we may not have a dedicate recommender engine designed. – Traditional author-disambiguation or record deduplication algorithms are batch-processing with small to no training data. However, we have roughly 25 million authorships that are manually curated or corrected upon user feedback. Hence, it is crucial to maintain historical profiles and hence we have developed a machine learning implementation to deal with data streams and process them in mini batches or one document at a time. We will discuss how to measure the accuracy of such a system, how to tune it and how to process the raw data of pairwise similarity function into final clusters. Lessons learned from this talk can help all sort of companies where they want to integrate their data or deduplicate their user/customer/product databases.
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
The use of large-scale machine learning and data mining methods is becoming ubiquitous in many application domains ranging from business intelligence and bioinformatics to self-driving cars. These methods heavily rely on matrix computations, and it is hence critical to make these computations scalable and efficient. These matrix computations are often complex and involve multiple steps that need to be optimized and sequenced properly for efficient execution. This work presents new efficient and scalable matrix processing and optimization techniques based on Spark. The proposed techniques estimate the sparsity of intermediate matrix-computation results and optimize communication costs. An evaluation plan generator for complex matrix computations is introduced as well as a distributed plan optimizer that exploits dynamic cost-based analysis and rule-based heuristics The result of a matrix operation will often serve as an input to another matrix operation, thus defining the matrix data dependencies within a matrix program. The matrix query plan generator produces query execution plans that minimize memory usage and communication overhead by partitioning the matrix based on the data dependencies in the execution plan. We implemented the proposed matrix techniques inside the Spark SQL, and optimize the matrix execution plan based on Spark SQL Catalyst. We conduct case studies on a series of ML models and matrix computations with special features on different datasets. These are PageRank, GNMF, BFGS, sparse matrix chain multiplications, and a biological data analysis. The open-source library ScaLAPACK and the array-based database SciDB are used for performance evaluation. Our experiments are performed on six real-world datasets are: social network data ( e.g., soc-pokec, cit-Patents, LiveJournal), Twitter2010, Netflix recommendation data, and 1000 Genomes Project sample. Experiments demonstrate that our proposed techniques achieve up to an order-of-magnitude performance.
Indicium: Interactive Querying at Scale Using Apache Spark, Zeppelin, and Spa...
This document discusses Indicium, a system for enabling interactive querying at scale on Spark. It describes a unified data platform using Spark for storing and processing data, as well as enabling scheduled jobs, BI reports, and real-time lookups. It then discusses two parts of Indicium: (1) a managed context pool using Apache Zeppelin and Spark Job Server for multi-user SQL queries, and (2) a smart query scheduler for dynamic scheduling, load balancing, and high availability of queries. The smart scheduler aims to improve on the managed context pool by addressing issues like FIFO scheduling and lack of failure handling.
More from Spark Summit (20)
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
Next CERN Accelerator Logging Service with Jakub Wozniak
Next CERN Accelerator Logging Service with Jakub Wozniak
Powering a Startup with Apache Spark with Kevin Kim
Powering a Startup with Apache Spark with Kevin Kim
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
Indicium: Interactive Querying at Scale Using Apache Spark, Zeppelin, and Spa...
Indicium: Interactive Querying at Scale Using Apache Spark, Zeppelin, and Spa...
Recently uploaded
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Discussion on Vector Databases, Unstructured Data and AI
https://www.meetup.com/unstructured-data-meetup-new-york/
This meetup is for people working in unstructured data. Speakers will come present about related topics such as vector databases, LLMs, and managing data at scale. The intended audience of this group includes roles like machine learning engineers, data scientists, data engineers, software engineers, and PMs.This meetup was formerly Milvus Meetup, and is sponsored by Zilliz maintainers of Milvus.
原版制作(swinburne毕业证书)斯威本科技大学毕业证毕业完成信一模一样
学校原件一模一样【微信：741003700 】《(swinburne毕业证书)斯威本科技大学毕业证》【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
【主营项目】
一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
University of New South Wales degree offer diploma Transcript
澳洲UNSW毕业证书制作新南威尔士大学假文凭定制Q微168899991做UNSW留信网教留服认证海牙认证改UNSW成绩单GPA做UNSW假学位证假文凭高仿毕业证申请新南威尔士大学University of New South Wales degree offer diploma Transcript
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Round table discussion of vector databases, unstructured data, ai, big data, real-time, robots and Milvus.
A lively discussion with NJ Gen AI Meetup Lead, Prasad and Procure.FYI's Co-Found
Natural Language Processing (NLP), RAG and its applications .pptx
1. In the realm of Natural Language Processing (NLP), knowledge-intensive tasks such as question answering, fact verification, and open-domain dialogue generation require the integration of vast and up-to-date information. Traditional neural models, though powerful, struggle with encoding all necessary knowledge within their parameters, leading to limitations in generalization and scalability. The paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" introduces RAG (Retrieval-Augmented Generation), a novel framework that synergizes retrieval mechanisms with generative models, enhancing performance by dynamically incorporating external knowledge during inference.
Learn SQL from basic queries to Advance queries
Dive into the world of data analysis with our comprehensive guide on mastering SQL! This presentation offers a practical approach to learning SQL, focusing on real-world applications and hands-on practice. Whether you're a beginner or looking to sharpen your skills, this guide provides the tools you need to extract, analyze, and interpret data effectively.
Key Highlights:
Foundations of SQL: Understand the basics of SQL, including data retrieval, filtering, and aggregation.
Advanced Queries: Learn to craft complex queries to uncover deep insights from your data.
Data Trends and Patterns: Discover how to identify and interpret trends and patterns in your datasets.
Practical Examples: Follow step-by-step examples to apply SQL techniques in real-world scenarios.
Actionable Insights: Gain the skills to derive actionable insights that drive informed decision-making.
Join us on this journey to enhance your data analysis capabilities and unlock the full potential of SQL. Perfect for data enthusiasts, analysts, and anyone eager to harness the power of data!
#DataAnalysis #SQL #LearningSQL #DataInsights #DataScience #Analytics
The Building Blocks of QuestDB, a Time Series Database
Talk Delivered at Valencia Codes Meetup 2024-06.
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
毕业原版【微信:41543339】【(爱大毕业证书)爱丁堡大学毕业证】【微信:41543339】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信41543339】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信41543339】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
办(uts毕业证书)悉尼科技大学毕业证学历证书原版一模一样
原版一模一样【微信：741003700 】【(uts毕业证书)悉尼科技大学毕业证学历证书】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
一比一原版(BCU毕业证书)伯明翰城市大学毕业证如何办理
原版定制【微信:41543339】【(BCU毕业证书)伯明翰城市大学毕业证】【微信:41543339】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信41543339】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信41543339】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
原版制作(Deakin毕业证书)迪肯大学毕业证学位证一模一样
学校原件一模一样【微信：741003700 】《(Deakin毕业证书)迪肯大学毕业证学位证》【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
【主营项目】
一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
End-to-end pipeline agility - Berlin Buzzwords 2024
We describe how we achieve high change agility in data engineering by eliminating the fear of breaking downstream data pipelines through end-to-end pipeline testing, and by using schema metaprogramming to safely eliminate boilerplate involved in changes that affect whole pipelines.
A quick poll on agility in changing pipelines from end to end indicated a huge span in capabilities. For the question "How long time does it take for all downstream pipelines to be adapted to an upstream change," the median response was 6 months, but some respondents could do it in less than a day. When quantitative data engineering differences between the best and worst are measured, the span is often 100x-1000x, sometimes even more.
A long time ago, we suffered at Spotify from fear of changing pipelines due to not knowing what the impact might be downstream. We made plans for a technical solution to test pipelines end-to-end to mitigate that fear, but the effort failed for cultural reasons. We eventually solved this challenge, but in a different context. In this presentation we will describe how we test full pipelines effectively by manipulating workflow orchestration, which enables us to make changes in pipelines without fear of breaking downstream.
Making schema changes that affect many jobs also involves a lot of toil and boilerplate. Using schema-on-read mitigates some of it, but has drawbacks since it makes it more difficult to detect errors early. We will describe how we have rejected this tradeoff by applying schema metaprogramming, eliminating boilerplate but keeping the protection of static typing, thereby further improving agility to quickly modify data pipelines without fear.
一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理
原版定制【微信:41543339】【(Bradford毕业证书)布拉德福德大学毕业证】【微信:41543339】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信41543339】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信41543339】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
毕业原版【微信:176555708】【(UCSF毕业证书)旧金山分校毕业证】【微信:176555708】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信176555708】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信176555708】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
毕业原版【微信:41543339】【(Harvard毕业证书)哈佛大学毕业证】【微信:41543339】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信41543339】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信41543339】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
The Modern Marketing Reckoner (MMR) is a comprehensive resource packed with POVs from 60+ industry leaders on how AI is transforming the 4 key pillars of marketing – product, place, price and promotions.
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
This webinar will explore cutting-edge, less familiar but powerful experimentation methodologies which address well-known limitations of standard A/B Testing. Designed for data and product leaders, this session aims to inspire the embrace of innovative approaches and provide insights into the frontiers of experimentation!
My burning issue is homelessness K.C.M.O.
My burning issue is homelessness in Kansas City, MO
To: Tom Tresser
From: Roger Warren
State of Artificial intelligence Report 2023
Artificial intelligence (AI) is a multidisciplinary field of science and engineering whose goal is to create intelligent machines.
We believe that AI will be a force multiplier on technological progress in our increasingly digital, data-driven world. This is because everything around us today, ranging from culture to consumer products, is a product of intelligence.
The State of AI Report is now in its sixth year. Consider this report as a compilation of the most interesting things we’ve seen with a goal of triggering an informed conversation about the state of AI and its implication for the future.
We consider the following key dimensions in our report:
Research: Technology breakthroughs and their capabilities.
Industry: Areas of commercial application for AI and its business impact.
Politics: Regulation of AI, its economic implications and the evolving geopolitics of AI.
Safety: Identifying and mitigating catastrophic risks that highly-capable future AI systems could pose to us.
Predictions: What we believe will happen in the next 12 months and a 2022 performance review to keep us honest.
Recently uploaded (20)
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
University of New South Wales degree offer diploma Transcript
University of New South Wales degree offer diploma Transcript
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
Natural Language Processing (NLP), RAG and its applications .pptx
Natural Language Processing (NLP), RAG and its applications .pptx
The Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series Database
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Spark Summit EU talk by Javier Aguedes
- 1. Origin-Destination Matrix Using Mobile Network Data with Spark Dr. Javiera Guedes Teralytics
- 2. Teralytics We transform raw, human mobile activity data into valuable insights (near)
- 3. We Provide Insights into Human Behavior
- 6. OD Matrix In transportation processes, it is important to understand how many people travel between zones. This interchange is organized in an origin-destination table or matrix. One approach to statically computing this matrix is: These factors can be estimated from mobility and census data, surveys, ground truth collection data extrapolation, etc. With telco data we can do this dynamically, over time, and including mode of transport to answer major transportation questions.
- 7. OD Matrix Germany Zip code level 5 geometries, Germany Region 1 2 3 1 120 12 58 2 12 80 95 3 56 90 34 1 Matrix per • Time bucket (15 min, hour, day) • Mode of transport (flight, car, train) 8199 zip code level 5 geometries 40 days, 100 Billion events
- 8. Approach
- 9. ApproachTelco Data HDFS Network Info PostgreSQL USA: 25 Billion events / day Germany: 3 Billion events / day Raw Data Processing Filtering Location Estimation MoT Detection Routing (optional) Trip Aggregation Group By Subscriber Scala / Spark K- anonymization
- 10. Cell phone tower Event Generation O D
- 11. 8 AM 9 AM 9:05 8:05 8:20 9:15 9:10 10 AM 10:25 10:05 3 pm 4 pm 4:05 10 pm 10:05 11 pm O D Event at cell location Coverage area Coverage Areas
- 12. O D 8 AM 9 AM 9:05 8:20 9:15 9:10 10 AM 10:25 10:05 3 pm 4 pm 4:05 pm 10 pm 10:05 pm 11 pm 8:05 Event at cell location Event at est. location Confidence area Location Estimation
- 13. O Event at cell location Location estimation Most probable route Alternative route D Trip Estimation
- 14. O D Event at cell location Location estimation Most probable route Subway Mode of Transport Estimation
- 15. Coverage Estimation: • Expectation Maximization around cell locations using estimated locations of events associated with the cells. Coverage Area Estimation
- 16. Aggregation: • Can be done in any desired geometry (e.g zip code) • Reduces the amount of data for visualizations and ensures k-anonymization • Hexagons are great geometries because they are space-filling and can be approximately nested to archive varying levels of resolution • Sampling an underlying distribution with hexagons is 13% more accurate than squares • Allows for fast data analysis and visualization as data can be pre-aggregated into different hexagon levels
- 18. Detecting the mode of transportation of a trip is a classification problem. Mostly, we want to assign trips to major mode of transportations such as flight, train, or car. Flights: • Stationary at origin and destination • Nearby Airports • Speed constrains • Feasible flight routes stationary events on-flight events Traces that contain no flights are further classified as land-based Mode of Transport Estimation
- 19. Detecting the mode of transportation of a trip is a classification problem. Mostly, we want to assign trips to major mode of transportations such as flight, train, or car. Train / Car: • Features derived from e.g. distance to railways or highways pcar / ptrain > t MoT = Car ptrain / pcar > t MoT = Train • Routing Mode of Transport Estimation
- 20. PRODUCT
- 21. OD Matrix Product: • Departure time from origin • Arrival time at destination • Mode of Transport • Aggregated by desired region (state, city, zip code, etc.) • Bucketed in any desired time (15 min, 1 hr, 1 week, etc.)
- 22. OD Matrix Germany Goal: • Compute a matrix between all zip code level 5 areas in Germany (8199 PLZ5)2 • Bucketed by 15 min, hour, day, or week • Split by mode of transport: flight, car, train Data: • 40 days • 100 Billion events Zip code level 5 geometries, Germany
- 23. Extrapolation The extrapolation factors for each entry of the OD matrix are computed using the market share of the origin and destination regions. Estimated local market share based on telco and census data in Germany 0 < MS < 1
- 24. OD Matrix Germany • Telco data is intrinsically dynamic, patterns can be seen varying over time, or even over the course of day. • Data accompanied by visualizations that allow customers to dice and slice the data for specific regions Matrix is k-anonymized, i.e. all flow with less than five commuters is removed.
- 25. Validation
- 26. We validate out results using several criteria and data sets: • MoT validation using limited calibration data labeled to measure performance. • Limited ground truth for train and flight trips from our providers and customers • Common sense checks, e.g. whether counts are balanced: OD[i,j] ~ OD[j,i] Criteria for Validation precision recall flights 0.96 0.81 car 0.96 0.88 train 0.95 0.91
- 27. Building up a strong network of women in tech — Javiera Guedes — javiera.guedes@teralytics.ch — July 14, 2016 27
- 28. Borussia Dortmund vs M’gladbach September 11, 2015 54,000 fans Borussia Park Bayern Munich vs FC Augsburg September 12, 2015 75,000 fans Allianz Arena Bundesliga Games
- 29. Berlin, Germany
- 30. Guangzhou, China
- 32. Why Spark Our production code is written entirely in Scala / Spark: • Terabyte data sets processed on a single job • Fast and compatible with Hadoop HDFS and Accumulo • The domain is easy to parallelize since every trip is independent Dark Matter Streams | Kavli Institute Stanford University
- 33. Why Spark Our production code is written entirely in Scala / Spark: • Terabyte data sets processed on a single job • Fast and compatible with Hadoop HDFS and Accumulo • The domain is easy to parallelize since every trip is independent • Natural choice of the map / reduce nature of our algorithmic approach Spark is used for slicing and dicing data in dashboards Spark was not our top choice for data visualization dashboard, since only small batches of data are queried