Spark meets Telemetry
•
4 likes•1,819 views
This document discusses using Spark to analyze telemetry data from Mozilla. It describes how telemetry data is currently collected and processed using a map-reduce framework in Python. It then introduces Spark as a faster alternative that supports interactive analysis, machine learning, and other capabilities. The document demonstrates launching a Spark cluster on EC2 to analyze Mozilla telemetry data in an interactive tutorial.
Report
Share
Report
Share
Download to read offline
Recommended
Elixir 5 minute intro
This document discusses building systems using Erlang and functional programming concepts. It provides code examples of defining functions and modules in Erlang, using concurrency with tasks, and structuring code into pipelines. It also mentions using macros to generate code at compile time and provides an example of defining routes in a Phoenix router module. The talk was given by Doug Goldie at HackerDojo in November 2014 and emphasizes that programming can be fun using these approaches.
ns2-training material
This document discusses an NS-2 workshop. It begins by asking what NS-2 is and what is needed to use it. It explains that NS-2 is a network simulator and the requirements include installing NS-2, having a supported operating system like Ubuntu, development tools, and documentation. It also discusses how to program in NS-2 using TCL for simulations and other languages like C++, AWK, Python and Perl for processing output. The document provides examples of writing TCL code for creating nodes, links, agents and applications to simulate network traffic in NS-2.
Google map and navigation
this PPt cover all relevant point of Google map
you can easily explain each and every topic with the help of this ppt.
Graph 500 DISLIB powered optimized version
This document discusses an optimized Graph500 implementation using the DISLIB library. Key points:
- DISLIB is a library for message passing and synchronization that aims to improve productivity for Graph500 and other applications.
- The Graph500 algorithm uses a simple 1D decomposition with a CRS graph storage format and BFS kernel. DISLIB handles complexity like message aggregation and synchronization.
- For the June 2011 submissions, DISLIB was built on DCMF for BlueGene/P and GASNET for InfiniBand clusters. The November 2011 submission used MPI with improved tuning.
Tower Of Hanoi
This presentation gives details about Tower of Hanoi and basic understanding of Recursion in algorithms.
Communication Patterns with Apache Spark-(Reza Zadeh, Stanford)
1) Spark programs involve creating RDDs from input data, transforming them lazily through operations like map and filter, and then launching actions like count to trigger computation across clusters.
2) Spark uses communication patterns like shuffle to perform reductions across partitions during operations like reduceByKey. It employs techniques like sorting and broadcasting to efficiently distribute data.
3) The Spark scheduler splits jobs into stages of tasks, coordinates execution on worker nodes, and retries failed tasks to provide fault tolerance across the cluster.
Tower of hanoi
The document discusses the Tower of Hanoi puzzle, including its history, rules, and various algorithms and solutions. It describes the basic recursive algorithm for solving the puzzle by breaking it down into smaller subproblems. It also presents non-recursive algorithms and discusses representing the puzzle as a graph.
Incremental and parallel computation of structural graph summaries for evolvi...
This document presents an incremental and parallel algorithm for computing structural graph summaries of evolving graphs. The algorithm incrementally updates graph summaries when the input graph changes, which is often faster than recomputing from scratch. The algorithm partitions the graph and computes summaries in parallel. Experimental results on real-world and benchmark graphs show the incremental algorithm outperforms batch computation even when 50% of the graph changes. The algorithm runs in linear time with respect to graph changes and degree.
Recommended
Elixir 5 minute intro
This document discusses building systems using Erlang and functional programming concepts. It provides code examples of defining functions and modules in Erlang, using concurrency with tasks, and structuring code into pipelines. It also mentions using macros to generate code at compile time and provides an example of defining routes in a Phoenix router module. The talk was given by Doug Goldie at HackerDojo in November 2014 and emphasizes that programming can be fun using these approaches.
ns2-training material
This document discusses an NS-2 workshop. It begins by asking what NS-2 is and what is needed to use it. It explains that NS-2 is a network simulator and the requirements include installing NS-2, having a supported operating system like Ubuntu, development tools, and documentation. It also discusses how to program in NS-2 using TCL for simulations and other languages like C++, AWK, Python and Perl for processing output. The document provides examples of writing TCL code for creating nodes, links, agents and applications to simulate network traffic in NS-2.
Google map and navigation
this PPt cover all relevant point of Google map
you can easily explain each and every topic with the help of this ppt.
Graph 500 DISLIB powered optimized version
This document discusses an optimized Graph500 implementation using the DISLIB library. Key points:
- DISLIB is a library for message passing and synchronization that aims to improve productivity for Graph500 and other applications.
- The Graph500 algorithm uses a simple 1D decomposition with a CRS graph storage format and BFS kernel. DISLIB handles complexity like message aggregation and synchronization.
- For the June 2011 submissions, DISLIB was built on DCMF for BlueGene/P and GASNET for InfiniBand clusters. The November 2011 submission used MPI with improved tuning.
Tower Of Hanoi
This presentation gives details about Tower of Hanoi and basic understanding of Recursion in algorithms.
Communication Patterns with Apache Spark-(Reza Zadeh, Stanford)
1) Spark programs involve creating RDDs from input data, transforming them lazily through operations like map and filter, and then launching actions like count to trigger computation across clusters.
2) Spark uses communication patterns like shuffle to perform reductions across partitions during operations like reduceByKey. It employs techniques like sorting and broadcasting to efficiently distribute data.
3) The Spark scheduler splits jobs into stages of tasks, coordinates execution on worker nodes, and retries failed tasks to provide fault tolerance across the cluster.
Tower of hanoi
The document discusses the Tower of Hanoi puzzle, including its history, rules, and various algorithms and solutions. It describes the basic recursive algorithm for solving the puzzle by breaking it down into smaller subproblems. It also presents non-recursive algorithms and discusses representing the puzzle as a graph.
Incremental and parallel computation of structural graph summaries for evolvi...
This document presents an incremental and parallel algorithm for computing structural graph summaries of evolving graphs. The algorithm incrementally updates graph summaries when the input graph changes, which is often faster than recomputing from scratch. The algorithm partitions the graph and computes summaries in parallel. Experimental results on real-world and benchmark graphs show the incremental algorithm outperforms batch computation even when 50% of the graph changes. The algorithm runs in linear time with respect to graph changes and degree.
Where in the world is Franz Kafka? | Will LaForest, Confluent
Apache Kafka is the de-facto standard for event streaming and creating data pipelines that can feed a variety of different tools. It is very common for the data to have geospatial characteristics but to date there has been relatively little work done around how to leverage this natively in Kafka. The common approach is to just dump all the data into some geospatial store or toolset and do retrospective analysis and queries. This of course loses all the advantages of handling it in realtime before it ever goes to an external tool. In this talk I will discuss the creation and demonstrate the usage of geospatial UDFs in ksqlDB. I will also talk through the advantages of doing geospatial processing directly in Apache Kafka.
S2
S2 Library deep dive.
How does it work, features, usage, visualisations and more.
Based on real life work experience @Google
CArcMOOC 03.04 - Gate-level design
Computer Architecture
Degree Program in Applied Computer Science
University of Urbino
http://informatica.uniurb.it/
Pgrouting_foss4guk_ross_mcdonald
1) pgRouting is an extension to PostGIS that adds routing functionality to PostgreSQL databases.
2) It allows users to calculate shortest and fastest path routes on road networks.
3) The document discusses various Ordnance Survey and OpenStreetMap datasets that can be used with pgRouting, tools for loading spatial data, and performance enhancements for routing queries.
MapReduce with Hadoop
Hadoop is an open-source software framework for distributed storage and processing of large datasets across clusters of computers using MapReduce. It allows for the parallel processing of large datasets across clusters of computers using a simple programming model. MapReduce programs are divided into Map and Reduce phases, where the Map phase processes input key-value pairs to generate intermediate output and the Reduce phase merges all intermediate values associated with the same key. Experiments showed that Hadoop and MapReduce were able to process 1 million documents faster than a single computer by distributing the work across multiple machines in a cluster.
A parallel gpu version of the traveling salesman problem slides
This document summarizes a parallel GPU implementation of the traveling salesman problem (TSP). The TSP algorithm used is an iterative hill climbing search that generates random initial tours and refines them through opt-2 moves until a local minimum is reached. The algorithm was optimized for GPUs by distributing independent climbers across threads and minimizing memory accesses through techniques like caching distance matrices in shared memory. Evaluation on a Tesla GPU found it was 7.8x faster than an 8-core CPU implementation and produced optimal tours in 4 out of 5 test cases using 100,000-200,000 climbers.
Doom in SpaceX
This document discusses SpaceX's approach to rocket hardware and software compared to traditional aerospace approaches. It notes that SpaceX hires many software engineers from the gaming industry and that rocket software is viewed as easier than game software. It outlines how SpaceX uses smaller teams and more agile development practices compared to NASA, focusing on skills like C++, LabVIEW, and JavaScript. It also discusses SpaceX's approach of learning through experience rather than anticipating all interactions.
Flink meetup
The document discusses timely dataflow, a data-parallel programming model that enables parallelism, state, and iteration in streaming computations. It proposes adding structured programming languages and zero-overhead abstractions to timely dataflow to provide modular constructs while maintaining performance. A worked example of graph connectivity demonstrates how timely dataflow could implement common graph algorithms today with time-generic operators for sending, receiving, and notifying on data.
scikit-cuda
scikit-cuda is a Python package that provides interfaces to popular GPU-powered libraries like CUBLAS, CUFFT and CUSOLVER. It exposes both low-level wrappers of C functions using ctypes as well as high-level functions with an interface similar to SciPy to make GPU capabilities more accessible to Python programmers. The package originated from using CULA's GPU SVD routines to accelerate time series decoding algorithms in Python. Future goals include improving wrappers, adding support for MAGMA, and enabling library backend selection.
3.5 equivalence of pushdown automata and cfl
The document discusses the equivalence between context-free grammars (CFGs) and pushdown automata (PDAs). It states that for any CFG, an equivalent PDA can be constructed to accept the language generated by the grammar, and vice versa. This allows a programming language to be specified by a CFG and implemented with a PDA in a compiler. The document also provides procedures for converting between CFGs and PDAs, including an example of constructing a PDA from a given CFG.
Google Cluster Innards
The document summarizes key aspects of Google's early development and architecture for web search. It discusses how Google was founded by Sergey Brin and Larry Page in 1998 and developed the PageRank algorithm to rank pages based on backlinks. It then describes the anatomy of Google's large-scale clusters, including how they use commodity hardware, a modified Linux OS, the Google File System, and MapReduce programming model to process massive amounts of data across thousands of servers.
Riding the Elephant - Hadoop 2.0
Hadoop 2.0, and in particular YARN has opened up a lot of potential applications beyond MapReduce. This presentation explains some of the ways this happened, and what you can now do that you couldn't before. It also introduces some new tools (Spark) and infrastructure pieces (Mesos) to achieve even more efficient cluster use.
Three Functional Programming Technologies for Big Data
Learn about the future of Functional Programming and Big Data with this summary on a recent evaluation of three related open source technologies.
Quill - 一個 Scala 的資料庫存取利器
在 Java ecosystem, 資料庫應用系統通常會使用 JDBC 直接存取資料庫, 或採用其他 ORM framework 資料庫存取工具.
Quill 是 Scala 的一個簡單又優美的 ORM 工具, 它提供了一組 DSL(Domain Specific Language). 只要使用 case class 加上 function, 我們即可使用 Scala 語法寫出優美的資料庫查詢指令. 在 compile-time 階段產生出對應資料庫的 SQL statement, 也因此在 run-time 階段, 系統不再須要動態組出 SQL statement, 提升了系統整體的執行效能.
Source code: https://github.com/vitojeng/jcconf2020-quill
Aws Quick Dirty Hadoop Mapreduce Ec2 S3
This document discusses using Amazon Web Services (AWS) and Hadoop MapReduce for quick and dirty parallel processing on the cloud. It provides an overview of AWS tools like EC2, S3, and Elastic MapReduce that can be used to launch and manage Hadoop clusters in the cloud. It also summarizes how MapReduce works and gives an example of analyzing Wikipedia page view logs with Hadoop Streaming and S3 for storage. The document concludes that Hadoop MapReduce provides an easy way to do distributed computing in the cloud.
Open Source Routing Machine - FOSS4G 2016 Bonn
My presentation on OSRM
VIDEO:
http://ftp5.gwdg.de/pub/misc/openstreetmap/FOSS4G-2016/foss4g-2016-1332-dealing_with_change_-_osrm_version_5-hd.webm
http://ftp5.gwdg.de/pub/misc/openstreetmap/FOSS4G-2016/foss4g-2016-1332-dealing_with_change_-_osrm_version_5-hd.mp4
Graph database
Graph databases are well-suited for storing and querying multi-relational data. They provide better performance, flexibility, and agility than relational databases for such data. Tests showed graph databases like Neo4j outperforming relational databases by returning results faster and for more records as depth and complexity of queries increased. Cypher is the query language for Neo4j that allows starting queries, matching patterns, returning and filtering results through clauses like START, MATCH, RETURN, and WHERE. Graph databases are used successfully by many large companies needing to handle complex relationships in data.
State of OSRM - SOTM 2016
The document summarizes recent updates to the Open Source Routing Machine (OSRM) project. OSRM version 5 now supports dynamic speed data to provide more accurate arrival time estimates, and improved guidance instructions for turns, lanes, ramps, and destinations. The speaker encourages using OSRM for in-car navigation and bringing your own dynamic speed data, as it has guidance capabilities ready for vehicle integration.
Nips2016 mlgkernel
The document proposes a Multiscale Laplacian Graph Kernel (MLG) that captures graph structure at multiple scales. The MLG computes the graph Laplacian at increasing levels of resolution, then takes the sum of the normalized dot products between the Laplacian matrices at matching scales. This allows the kernel to compare both global and local structure between graphs. The MLG outperforms other graph kernels on bioinformatics datasets, demonstrating it can effectively capture multiresolution graph information.
Automating AWS Infrastructure Provisioning Using Concourse and Terraform
Building a CI/CD pipeline with Concourse and terraform.
Resources:
- concourse-pipelines - Collection of concourse pipelines including terraform-pr and terraform-commit pipeline
https://github.com/cesar-rodriguez/concourse-pipelines
- terrascan - Static code analysis of terraform templates. https://github.com/cesar-rodriguez/terrascan
- hello-hug - Example terraform project that uses concourse-pipelines
https://github.com/cesar-rodriguez/hello-hug
Tuning and Debugging in Apache Spark
Video: https://www.youtube.com/watch?v=kkOG_aJ9KjQ
This talk gives details about Spark internals and an explanation of the runtime behavior of a Spark application. It explains how high level user programs are compiled into physical execution plans in Spark. It then reviews common performance bottlenecks encountered by Spark users, along with tips for diagnosing performance problems in a production application.
RBootcam Day 2
This document summarizes content from Day 2 of an R bootcamp, including:
1) Creating variables as vectors and accessing their values through indexing. Summing and subsetting vectors.
2) Binding vectors together into a dataframe and accessing values through row/column indexing or column names.
3) Importing and exporting data from dataframes to CSV files. Manipulating dataframes through functions like mutate().
3) Basic plotting of dataframes and vectors using plot(). Customizing plots with parameters like main, xlab, ylab, and colors. Using low-level functions like line() for more control.
More Related Content
What's hot
Where in the world is Franz Kafka? | Will LaForest, Confluent
Apache Kafka is the de-facto standard for event streaming and creating data pipelines that can feed a variety of different tools. It is very common for the data to have geospatial characteristics but to date there has been relatively little work done around how to leverage this natively in Kafka. The common approach is to just dump all the data into some geospatial store or toolset and do retrospective analysis and queries. This of course loses all the advantages of handling it in realtime before it ever goes to an external tool. In this talk I will discuss the creation and demonstrate the usage of geospatial UDFs in ksqlDB. I will also talk through the advantages of doing geospatial processing directly in Apache Kafka.
S2
S2 Library deep dive.
How does it work, features, usage, visualisations and more.
Based on real life work experience @Google
CArcMOOC 03.04 - Gate-level design
Computer Architecture
Degree Program in Applied Computer Science
University of Urbino
http://informatica.uniurb.it/
Pgrouting_foss4guk_ross_mcdonald
1) pgRouting is an extension to PostGIS that adds routing functionality to PostgreSQL databases.
2) It allows users to calculate shortest and fastest path routes on road networks.
3) The document discusses various Ordnance Survey and OpenStreetMap datasets that can be used with pgRouting, tools for loading spatial data, and performance enhancements for routing queries.
MapReduce with Hadoop
Hadoop is an open-source software framework for distributed storage and processing of large datasets across clusters of computers using MapReduce. It allows for the parallel processing of large datasets across clusters of computers using a simple programming model. MapReduce programs are divided into Map and Reduce phases, where the Map phase processes input key-value pairs to generate intermediate output and the Reduce phase merges all intermediate values associated with the same key. Experiments showed that Hadoop and MapReduce were able to process 1 million documents faster than a single computer by distributing the work across multiple machines in a cluster.
A parallel gpu version of the traveling salesman problem slides
This document summarizes a parallel GPU implementation of the traveling salesman problem (TSP). The TSP algorithm used is an iterative hill climbing search that generates random initial tours and refines them through opt-2 moves until a local minimum is reached. The algorithm was optimized for GPUs by distributing independent climbers across threads and minimizing memory accesses through techniques like caching distance matrices in shared memory. Evaluation on a Tesla GPU found it was 7.8x faster than an 8-core CPU implementation and produced optimal tours in 4 out of 5 test cases using 100,000-200,000 climbers.
Doom in SpaceX
This document discusses SpaceX's approach to rocket hardware and software compared to traditional aerospace approaches. It notes that SpaceX hires many software engineers from the gaming industry and that rocket software is viewed as easier than game software. It outlines how SpaceX uses smaller teams and more agile development practices compared to NASA, focusing on skills like C++, LabVIEW, and JavaScript. It also discusses SpaceX's approach of learning through experience rather than anticipating all interactions.
Flink meetup
The document discusses timely dataflow, a data-parallel programming model that enables parallelism, state, and iteration in streaming computations. It proposes adding structured programming languages and zero-overhead abstractions to timely dataflow to provide modular constructs while maintaining performance. A worked example of graph connectivity demonstrates how timely dataflow could implement common graph algorithms today with time-generic operators for sending, receiving, and notifying on data.
scikit-cuda
scikit-cuda is a Python package that provides interfaces to popular GPU-powered libraries like CUBLAS, CUFFT and CUSOLVER. It exposes both low-level wrappers of C functions using ctypes as well as high-level functions with an interface similar to SciPy to make GPU capabilities more accessible to Python programmers. The package originated from using CULA's GPU SVD routines to accelerate time series decoding algorithms in Python. Future goals include improving wrappers, adding support for MAGMA, and enabling library backend selection.
3.5 equivalence of pushdown automata and cfl
The document discusses the equivalence between context-free grammars (CFGs) and pushdown automata (PDAs). It states that for any CFG, an equivalent PDA can be constructed to accept the language generated by the grammar, and vice versa. This allows a programming language to be specified by a CFG and implemented with a PDA in a compiler. The document also provides procedures for converting between CFGs and PDAs, including an example of constructing a PDA from a given CFG.
Google Cluster Innards
The document summarizes key aspects of Google's early development and architecture for web search. It discusses how Google was founded by Sergey Brin and Larry Page in 1998 and developed the PageRank algorithm to rank pages based on backlinks. It then describes the anatomy of Google's large-scale clusters, including how they use commodity hardware, a modified Linux OS, the Google File System, and MapReduce programming model to process massive amounts of data across thousands of servers.
Riding the Elephant - Hadoop 2.0
Hadoop 2.0, and in particular YARN has opened up a lot of potential applications beyond MapReduce. This presentation explains some of the ways this happened, and what you can now do that you couldn't before. It also introduces some new tools (Spark) and infrastructure pieces (Mesos) to achieve even more efficient cluster use.
Three Functional Programming Technologies for Big Data
Learn about the future of Functional Programming and Big Data with this summary on a recent evaluation of three related open source technologies.
Quill - 一個 Scala 的資料庫存取利器
在 Java ecosystem, 資料庫應用系統通常會使用 JDBC 直接存取資料庫, 或採用其他 ORM framework 資料庫存取工具.
Quill 是 Scala 的一個簡單又優美的 ORM 工具, 它提供了一組 DSL(Domain Specific Language). 只要使用 case class 加上 function, 我們即可使用 Scala 語法寫出優美的資料庫查詢指令. 在 compile-time 階段產生出對應資料庫的 SQL statement, 也因此在 run-time 階段, 系統不再須要動態組出 SQL statement, 提升了系統整體的執行效能.
Source code: https://github.com/vitojeng/jcconf2020-quill
Aws Quick Dirty Hadoop Mapreduce Ec2 S3
This document discusses using Amazon Web Services (AWS) and Hadoop MapReduce for quick and dirty parallel processing on the cloud. It provides an overview of AWS tools like EC2, S3, and Elastic MapReduce that can be used to launch and manage Hadoop clusters in the cloud. It also summarizes how MapReduce works and gives an example of analyzing Wikipedia page view logs with Hadoop Streaming and S3 for storage. The document concludes that Hadoop MapReduce provides an easy way to do distributed computing in the cloud.
Open Source Routing Machine - FOSS4G 2016 Bonn
My presentation on OSRM
VIDEO:
http://ftp5.gwdg.de/pub/misc/openstreetmap/FOSS4G-2016/foss4g-2016-1332-dealing_with_change_-_osrm_version_5-hd.webm
http://ftp5.gwdg.de/pub/misc/openstreetmap/FOSS4G-2016/foss4g-2016-1332-dealing_with_change_-_osrm_version_5-hd.mp4
Graph database
Graph databases are well-suited for storing and querying multi-relational data. They provide better performance, flexibility, and agility than relational databases for such data. Tests showed graph databases like Neo4j outperforming relational databases by returning results faster and for more records as depth and complexity of queries increased. Cypher is the query language for Neo4j that allows starting queries, matching patterns, returning and filtering results through clauses like START, MATCH, RETURN, and WHERE. Graph databases are used successfully by many large companies needing to handle complex relationships in data.
State of OSRM - SOTM 2016
The document summarizes recent updates to the Open Source Routing Machine (OSRM) project. OSRM version 5 now supports dynamic speed data to provide more accurate arrival time estimates, and improved guidance instructions for turns, lanes, ramps, and destinations. The speaker encourages using OSRM for in-car navigation and bringing your own dynamic speed data, as it has guidance capabilities ready for vehicle integration.
Nips2016 mlgkernel
The document proposes a Multiscale Laplacian Graph Kernel (MLG) that captures graph structure at multiple scales. The MLG computes the graph Laplacian at increasing levels of resolution, then takes the sum of the normalized dot products between the Laplacian matrices at matching scales. This allows the kernel to compare both global and local structure between graphs. The MLG outperforms other graph kernels on bioinformatics datasets, demonstrating it can effectively capture multiresolution graph information.
Automating AWS Infrastructure Provisioning Using Concourse and Terraform
Building a CI/CD pipeline with Concourse and terraform.
Resources:
- concourse-pipelines - Collection of concourse pipelines including terraform-pr and terraform-commit pipeline
https://github.com/cesar-rodriguez/concourse-pipelines
- terrascan - Static code analysis of terraform templates. https://github.com/cesar-rodriguez/terrascan
- hello-hug - Example terraform project that uses concourse-pipelines
https://github.com/cesar-rodriguez/hello-hug
What's hot (20)
Where in the world is Franz Kafka? | Will LaForest, Confluent
Where in the world is Franz Kafka? | Will LaForest, Confluent
A parallel gpu version of the traveling salesman problem slides
A parallel gpu version of the traveling salesman problem slides
Three Functional Programming Technologies for Big Data
Three Functional Programming Technologies for Big Data
Automating AWS Infrastructure Provisioning Using Concourse and Terraform
Automating AWS Infrastructure Provisioning Using Concourse and Terraform
Similar to Spark meets Telemetry
Tuning and Debugging in Apache Spark
Video: https://www.youtube.com/watch?v=kkOG_aJ9KjQ
This talk gives details about Spark internals and an explanation of the runtime behavior of a Spark application. It explains how high level user programs are compiled into physical execution plans in Spark. It then reviews common performance bottlenecks encountered by Spark users, along with tips for diagnosing performance problems in a production application.
RBootcam Day 2
This document summarizes content from Day 2 of an R bootcamp, including:
1) Creating variables as vectors and accessing their values through indexing. Summing and subsetting vectors.
2) Binding vectors together into a dataframe and accessing values through row/column indexing or column names.
3) Importing and exporting data from dataframes to CSV files. Manipulating dataframes through functions like mutate().
3) Basic plotting of dataframes and vectors using plot(). Customizing plots with parameters like main, xlab, ylab, and colors. Using low-level functions like line() for more control.
Lecture 2: Data-Intensive Computing for Text Analysis (Fall 2011)
Data-Intensive Computing for Text Analysis CS395T / INF385T / LIN386M
University of Texas at Austin, Fall 2011
Lecture 2 September 1, 2011
Jason Baldridge and Matt Lease
https://sites.google.com/a/utcompling.com/dicta-f11/
Structuring Spark: DataFrames, Datasets, and Streaming by Michael Armbrust
This document summarizes Spark's structured APIs including SQL, DataFrames, and Datasets. It discusses how structuring computation in Spark enables optimizations by limiting what can be expressed. The structured APIs provide type safety, avoid errors, and share an optimization and execution pipeline. Functions allow expressing complex logic on columns. Encoders map between objects and Spark's internal data format. Structured streaming provides a high-level API to continuously query streaming data similar to batch queries.
Tuning and Debugging in Apache Spark
This talk gives details about Spark internals and an explanation of the runtime behavior of a Spark application. It explains how high level user programs are compiled into physical execution plans in Spark. It then reviews common performance bottlenecks encountered by Spark users, along with tips for diagnosing performance problems in a production application.
Deathstar
deathstar is a tool for seamlessly running R code across local and cloud computing nodes. It uses ZeroMQ to distribute data and code across nodes. The deathstar daemon runs on each node and listens for work. The R package exposes a zmq.cluster.lapply function similar to regular lapply to distribute jobs. Examples show using deathstar to estimate pi across nodes and calculate stock betas in a distributed manner. deathstar allows scaling R code to multiple cores locally or launching EC2 instances on AWS for more capacity.
Hadoop classes in mumbai
Hadoop classes in mumbai
best android classes in mumbai with job assistance.
our features are:
expert guidance by it industry professionals
lowest fees of 5000
practical exposure to handle projects
well equiped lab
after course resume writing guidance
Barcelona MUG MongoDB + Hadoop Presentation
- The document discusses MongoDB and Hadoop, two popular big data platforms, and the MongoDB + Hadoop Connector which allows interoperation between the two.
- It provides an overview of MongoDB and Hadoop's key features for scalability, availability and processing large datasets.
- The connector allows processing data across MongoDB and Hadoop through MapReduce jobs without needing custom exports/imports.
- Examples show building a graph of email sender/recipient relationships from an Enron dataset stored in MongoDB using Hadoop Streaming, Pig and Hive.
Shrug2017 arcpy data_and_you
This document provides an agenda and overview for a presentation on bridging the ESRI platform with Python and other databases. It discusses connection files in ESRI, using ArcPy cursors to read, insert and update data, connecting to databases like Oracle and SQL Server with Python libraries, and using SQLite as an alternative spatial database. It provides code examples for inserting CSV data into a SQLite database table and then inserting that data into a feature class. The final example shows updating data in a SQL Server database table.
Scala meetup - Intro to spark
This document provides an introduction to Apache Spark, including its core components, architecture, and programming model. Some key points:
- Spark uses Resilient Distributed Datasets (RDDs) as its fundamental data structure, which are immutable distributed collections that allow in-memory computing across a cluster.
- RDDs support transformations like map, filter, reduce, and actions like collect that return results. Transformations are lazy while actions trigger computation.
- Spark's execution model involves a driver program that coordinates tasks on worker nodes using an optimized scheduler.
- Spark SQL, MLlib, GraphX, and Spark Streaming extend the core Spark API for structured data, machine learning, graph processing, and stream processing
So you think you can stream.pptx
- Spark Streaming allows processing of live data streams using Spark's batch processing engine by dividing streams into micro-batches.
- A Spark Streaming application consists of input streams, transformations on those streams such as maps and filters, and output operations. The application runs continuously processing each micro-batch.
- Key aspects of operationalizing Spark Streaming jobs include checkpointing to ensure fault tolerance, optimizing throughput by increasing parallelism, and debugging using Spark UI.
Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming - by Mi...
This document summarizes key aspects of structuring computation and data in Apache Spark using SQL, DataFrames, and Datasets. It discusses how structuring computation and data through these APIs enables optimizations like predicate pushdown and efficient joins. It also describes how data is encoded efficiently in Spark's internal format and how encoders translate between domain objects and Spark's internal representations. Finally, it introduces structured streaming as a high-level streaming API built on top of Spark SQL that allows running the same queries continuously on streaming data.
2015-10-23_wim_davis_r_slides.pptx on consumer
R is an open-source statistical programming language that can be used for data analysis and visualization. The document provided an introduction to R including how to install R, create variables, import and assemble data, perform basic statistical analyses like t-tests and linear regression, and create plots and graphs. Key functions and concepts introduced included using c() to combine values into vectors, reading in data from CSV files, using lm() for linear regression, and the basic plot() function.
State of the Art Web Mapping with Open Source
This document discusses the importance of open source tools and data for web mapping. It begins by providing background on TileMill and Mapbox, which provide open source tools for making maps. It then discusses key concepts in web mapping like geospatial data formats, tile rendering, and minimal code examples. Modern approaches to web mapping involve preprocessing data, using tile renderers and caches, and gradually rendering more client-side. Upcoming improvements may optimize tiled formats and storage. TileMill is demonstrated as an open source tool for making maps. The talk concludes by emphasizing other open mapping tools like CartoDB, Stamen, and CartoDB that build on these concepts.
MapReduce basics
Covers:
Distributed processing issues, MR programming model
Sample MR job
How MR can be implemented
Pros and cons of MR, tips for better performance
NVIDIA HPC ソフトウエア斜め読み
2022 年 4 月 26 日 (火) に開催した「GTC 2022 テクニカルフォローアップセミナー 〜新アーキテクチャ & HPC〜」にて、NVIDIA 丹が講演した「NVIDIA HPC ソフトウエア斜め読み」のスライドです。
MAP REDUCE IN DATA SCIENCE.pptx
Hadoop is an open-source software framework for distributed storage and processing of large datasets across clusters of computers. It provides reliable storage through HDFS and distributed processing via MapReduce. HDFS handles storage and MapReduce provides a programming model for parallel processing of large datasets across a cluster. The MapReduce framework consists of a mapper that processes input key-value pairs in parallel, and a reducer that aggregates the output of the mappers by key.
Map Reduce
A MapReduce job usually splits the input data-set into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system.
Geoff Rothman Presentation on Parallel Processing
Presentation to University of Kentucky Computer Science graduate studentrs on high level Cloud Computing, how MapReduce works, and the current competition for Parallel Processing on a Massive Scale
A Brief Introduction of TiDB (Percona Live)
TiDB is an open-source distributed SQL database that supports high availability, horizontal scalability, and consistent distributed transactions. It provides a MySQL compatible API and seamless online expansion. TiDB uses Raft for consensus and implements the MVCC model to support high concurrency. It also provides distributed transactions through a two-phase commit protocol. The architecture consists of a stateless SQL layer (TiDB) and a distributed transactional key-value storage (TiKV).
Similar to Spark meets Telemetry (20)
Lecture 2: Data-Intensive Computing for Text Analysis (Fall 2011)
Lecture 2: Data-Intensive Computing for Text Analysis (Fall 2011)
Structuring Spark: DataFrames, Datasets, and Streaming by Michael Armbrust
Structuring Spark: DataFrames, Datasets, and Streaming by Michael Armbrust
Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming - by Mi...
Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming - by Mi...
More from Roberto Agostino Vitillo
Telemetry Onboarding
This document provides an overview of Mozilla's telemetry onboarding process. It outlines the typical steps of asking a question, getting the relevant telemetry data, analyzing the data, and reporting results. It includes links to documentation on Mozilla's telemetry systems, available telemetry datasets, tools for analyzing telemetry data, and resources for getting help throughout the process.
Growing a Data Pipeline for Analytics
This document discusses growing a data pipeline for analytics. It recommends using existing tools like Apache Spark, Hive, and Pig rather than building a custom pipeline. It suggests using schemas and columnar storage formats like Parquet. Finally, it advises exploiting modern SQL's support for complex data structures and using SQL-based analytics tools like Presto and Redash.
Telemetry Datasets
Telemetry datasets from Mozilla are accessible from Spark and Presto for analysis. The datasets include raw JSON data, derived views with summarized columns, and longitudinal views tracking client usage over time. An elective session on Thursday will teach how to access the telemetry data using different tools like Heka, Spark, and Presto.
Growing a SQL Query
The document discusses how SQL queries have evolved to handle complex data structures and analytics tasks. SQL now supports variables, loops, and window functions to query nested data, iterate over rows, and perform window calculations respectively. This enables SQL to answer questions by analyzing raw and columnar data stored in databases and data platforms. The document demonstrates these SQL capabilities through a demo query.
Telemetry Onboarding
This document provides an overview of Mozilla's telemetry system. It describes what telemetry is and how it measures Firefox usage to identify issues and conduct experiments. It explains the different types of pings and dashboards used in telemetry as well as how to add new probes, run experiments, and ensure privacy and policy compliance when collecting and analyzing data.
All you need to know about Statistics
Setting a 95% confidence interval means that if repeated random samples were taken from a population and the statistics and confidence intervals were calculated for each sample, 95% of the confidence intervals would contain the true value. The central limit theorem states that the distribution of sample means will be normally distributed even if the population is not normally distributed. Confidence intervals can be derived once the theoretical sampling distribution is known. Bootstrap resampling involves generating bootstrap samples by sampling with replacement from the original sample and calculating a bootstrap statistic for each sample to approximate the sampling distribution and derive a 95% confidence interval.
Vectorization on x86: all you need to know
The document discusses vectorization techniques on x86 processors. It describes how vectorization can significantly improve performance by processing multiple data elements in parallel using SIMD instructions. Vectorization is most effective when combined with multithreading. The document outlines several techniques for writing vectorized code, including using vector instruction sets like SSE and AVX, compiler auto-vectorization and vectorization directives. It also discusses challenges like memory alignment and dependency analysis that compilers have to handle for effective vectorization.
Sharing C++ objects in Linux
Sharing C++ objects across processes in Linux requires consideration of static data, pointers/references, virtual tables, and virtual inheritance. The key requirements are: 1) objects must be allocated in shared memory segments mapped to the same virtual address in all processes; 2) virtual tables and code must be mapped the same across processes; and 3) shared libraries containing virtual tables must be loaded at fixed addresses. With these techniques, C++ objects can be shared without deep changes to tooling.
Performance tools developments
The document discusses performance tool developments for the Linux kernel. It describes the perf events subsystem, which allows measuring workloads using performance counters. It introduces tools like perf-record and perf-report for collecting and displaying performance data. It outlines efforts to build visualizers and leverage last branch records to gain deeper insights into performance bottlenecks from hardware event data. The goal is to make it easier for non-experts to understand profiling information from the Linux performance subsystem.
Exploiting vectorization with ISPC
1) ISPC (Intel SPMD Program Compiler) is a language that extends C with "single program, multiple data" constructs to enable easy vectorization of code. It describes the behavior of a single program instance while executing a "gang" of instances in parallel.
2) ISPC was shown to provide significant speedups (up to 5x) over non-vectorized C/C++ code for problems like matrix multiplication and Kalman filtering.
3) While OpenCL can also enable vectorization, ISPC offers advantages like explicit control over vectorized vs serial code, performance warnings, and easier composition with task schedulers than OpenCL.
GOoDA tutorial
This document provides an overview of GOODA, a low overhead open source CPU profiler. GOODA consists of four main components: a kernel subsystem that interfaces with the PMU, a data collection tool called perf, an analyzer that creates call graphs and spreadsheets, and a web-based GUI. It was developed by Google and LBNL to provide low overhead profiling, call counts, microarchitectural insights, and a lightweight user interface. The document discusses how GOODA works, applying it to different application classes, and how it decomposes cycles using hardware event data to identify optimization opportunities like load latency, branch misses, and function call overhead.
Callgraph analysis
1) Callgraph analysis of ATLAS software identified clusters of heavily called functions that could benefit from inlining to reduce instruction counts. Inlining requires changes to code and use of link-time optimization with profile guidance.
2) Avoiding position independent code may improve performance but reduce code sharing. Static libraries could allow link-time optimization.
3) Tools like IgProf, SystemTap and perf events can profile memory and performance, but a visualizer is needed to analyze object-oriented software. Sampling branch records may improve basic block counts.
Vectorization in ATLAS
The document discusses vectorization in the ATLAS experiment. It begins by providing background on ATLAS and describing how exploiting SIMD (single instruction, multiple data) parallelism can improve performance. It then examines four approaches to vectorization: 1) hand-tuning numerical hotspots, 2) using vectorized linear algebra libraries, 3) relying on compiler auto-vectorization, and 4) using language extensions. For each approach, it provides examples and discusses advantages and limitations. The overall summary is that vectorization can provide significant performance gains but requires careful consideration of algorithms and data structures.
Inter-process communication on steroids
This document discusses inter-process communication techniques for high performance computing applications. It presents a message passing model using channels and sockets as a way to provide concurrency. Several data transfer techniques like zero-copy, double copy, and single copy are evaluated. Benchmarks show that for small messages, a pipe using vmsplice performs best in terms of latency, while for large messages, a spliced pipe provides the highest bandwidth with a single copy. The library provides a uniform message passing abstraction and supports different communication patterns and topologies to optimize performance.
More from Roberto Agostino Vitillo (14)
Recently uploaded
Essentials of Automations: Exploring Attributes & Automation Parameters
Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they?
Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality.
You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
This is a session that details how PostgreSQL's features and Azure AI Services can be effectively used to significantly enhance the search functionality in any application.
In this session, we'll share insights on how we used PostgreSQL to facilitate precise searches across multiple fields in our mobile application. The techniques include using LIKE and ILIKE operators and integrating a trigram-based search to handle potential misspellings, thereby increasing the search accuracy.
We'll also discuss how the azure_ai extension on PostgreSQL databases in Azure and Azure AI Services were utilized to create vectors from user input, a feature beneficial when users wish to find specific items based on text prompts. While our application's case study involves a drug search, the techniques and principles shared in this session can be adapted to improve search functionality in a wide range of applications. Join us to learn how PostgreSQL and Azure AI can be harnessed to enhance your application's search capability.
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
How we managed to grow and scale a RAG application from zero to thousands of users in 7 months. Lessons from technical challenges around managing high load for LLMs, RAGs and Vector databases.
AppSec PNW: Android and iOS Application Security with MobSF
Mobile Security Framework - MobSF is a free and open source automated mobile application security testing environment designed to help security engineers, researchers, developers, and penetration testers to identify security vulnerabilities, malicious behaviours and privacy concerns in mobile applications using static and dynamic analysis. It supports all the popular mobile application binaries and source code formats built for Android and iOS devices. In addition to automated security assessment, it also offers an interactive testing environment to build and execute scenario based test/fuzz cases against the application.
This talk covers:
Using MobSF for static analysis of mobile applications.
Interactive dynamic security assessment of Android and iOS applications.
Solving Mobile app CTF challenges.
Reverse engineering and runtime analysis of Mobile malware.
How to shift left and integrate MobSF/mobsfscan SAST and DAST in your build pipeline.
The Microsoft 365 Migration Tutorial For Beginner.pptx
This presentation will help you understand the power of Microsoft 365. However, we have mentioned every productivity app included in Office 365. Additionally, we have suggested the migration situation related to Office 365 and how we can help you.
You can also read: https://www.systoolsgroup.com/updates/office-365-tenant-to-tenant-migration-step-by-step-complete-guide/
Demystifying Knowledge Management through Storytelling
The Department of Veteran Affairs (VA) invited Taylor Paschal, Knowledge & Information Management Consultant at Enterprise Knowledge, to speak at a Knowledge Management Lunch and Learn hosted on June 12, 2024. All Office of Administration staff were invited to attend and received professional development credit for participating in the voluntary event.
The objectives of the Lunch and Learn presentation were to:
- Review what KM ‘is’ and ‘isn’t’
- Understand the value of KM and the benefits of engaging
- Define and reflect on your “what’s in it for me?”
- Share actionable ways you can participate in Knowledge - - Capture & Transfer
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
HERE IS YOUR WEBINAR CONTENT! 'Mastering Customer Journey Management with Dr. Graham Hill'. We hope you find the webinar recording both insightful and enjoyable.
In this webinar, we explored essential aspects of Customer Journey Management and personalization. Here’s a summary of the key insights and topics discussed:
Key Takeaways:
Understanding the Customer Journey: Dr. Hill emphasized the importance of mapping and understanding the complete customer journey to identify touchpoints and opportunities for improvement.
Personalization Strategies: We discussed how to leverage data and insights to create personalized experiences that resonate with customers.
Technology Integration: Insights were shared on how inQuba’s advanced technology can streamline customer interactions and drive operational efficiency.
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Dmitrii Kamaev, PhD
Senior Product Owner - QIAGEN
Apps Break Data
How information systems are built or acquired puts information, which is what they should be about, in a secondary place. Our language adapted accordingly, and we no longer talk about information systems but applications. Applications evolved in a way to break data into diverse fragments, tightly coupled with applications and expensive to integrate. The result is technical debt, which is re-paid by taking even bigger "loans", resulting in an ever-increasing technical debt. Software engineering and procurement practices work in sync with market forces to maintain this trend. This talk demonstrates how natural this situation is. The question is: can something be done to reverse the trend?
From Natural Language to Structured Solr Queries using LLMs
This talk draws on experimentation to enable AI applications with Solr. One important use case is to use AI for better accessibility and discoverability of the data: while User eXperience techniques, lexical search improvements, and data harmonization can take organizations to a good level of accessibility, a structural (or “cognitive” gap) remains between the data user needs and the data producer constraints.
That is where AI – and most importantly, Natural Language Processing and Large Language Model techniques – could make a difference. This natural language, conversational engine could facilitate access and usage of the data leveraging the semantics of any data source.
The objective of the presentation is to propose a technical approach and a way forward to achieve this goal.
The key concept is to enable users to express their search queries in natural language, which the LLM then enriches, interprets, and translates into structured queries based on the Solr index’s metadata.
This approach leverages the LLM’s ability to understand the nuances of natural language and the structure of documents within Apache Solr.
The LLM acts as an intermediary agent, offering a transparent experience to users automatically and potentially uncovering relevant documents that conventional search methods might overlook. The presentation will include the results of this experimental work, lessons learned, best practices, and the scope of future work that should improve the approach and make it production-ready.
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
These are the slides for the presentation, "Component Testing: Bridging the gap between frontend applications" that was presented at QA or the Highway 2024 in Columbus, OH by Zachary Hamm.
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
The typical problem in product engineering is not bad strategy, so much as “no strategy”. This leads to confusion, lack of motivation, and incoherent action. The next time you look for a strategy and find an empty space, instead of waiting for it to be filled, I will show you how to fill it in yourself. If you’re wrong, it forces a correction. If you’re right, it helps create focus. I’ll share how I’ve approached this in the past, both what works and lessons for what didn’t work so well.
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Pitangent Analytics & Technology Solutions Pvt. Ltd
Discover top-tier mobile app development services, offering innovative solutions for iOS and Android. Enhance your business with custom, user-friendly mobile applications.Must Know Postgres Extension for DBA and Developer during Migration
Mydbops Opensource Database Meetup 16
Topic: Must-Know PostgreSQL Extensions for Developers and DBAs During Migration
Speaker: Deepak Mahto, Founder of DataCloudGaze Consulting
Date & Time: 8th June | 10 AM - 1 PM IST
Venue: Bangalore International Centre, Bangalore
Abstract: Discover how PostgreSQL extensions can be your secret weapon! This talk explores how key extensions enhance database capabilities and streamline the migration process for users moving from other relational databases like Oracle.
Key Takeaways:
* Learn about crucial extensions like oracle_fdw, pgtt, and pg_audit that ease migration complexities.
* Gain valuable strategies for implementing these extensions in PostgreSQL to achieve license freedom.
* Discover how these key extensions can empower both developers and DBAs during the migration process.
* Don't miss this chance to gain practical knowledge from an industry expert and stay updated on the latest open-source database trends.
Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top three open-source databases: MySQL, MongoDB, and PostgreSQL.
Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability.
Contact us: info@mydbops.com
Visit: https://www.mydbops.com/
Follow us on LinkedIn: https://in.linkedin.com/company/mydbops
For more details and updates, please follow up the below links.
Meetup Page : https://www.meetup.com/mydbops-databa...
Twitter: https://twitter.com/mydbopsofficial
Blogs: https://www.mydbops.com/blog/
Facebook(Meta): https://www.facebook.com/mydbops/
Leveraging the Graph for Clinical Trials and Standards
Katja Glaß
OpenStudyBuilder Community Manager - Katja Glaß Consulting
Marius Conjeaud
Principal Consultant - Neo4j
High performance Serverless Java on AWS- GoTo Amsterdam 2024
Java is for many years one of the most popular programming languages, but it used to have hard times in the Serverless community. Java is known for its high cold start times and high memory footprint, comparing to other programming languages like Node.js and Python. In this talk I'll look at the general best practices and techniques we can use to decrease memory consumption, cold start times for Java Serverless development on AWS including GraalVM (Native Image) and AWS own offering SnapStart based on Firecracker microVM snapshot and restore and CRaC (Coordinated Restore at Checkpoint) runtime hooks. I'll also provide a lot of benchmarking on Lambda functions trying out various deployment package sizes, Lambda memory settings, Java compilation options and HTTP (a)synchronous clients and measure their impact on cold and warm start times.
Mutation Testing for Task-Oriented Chatbots
Conversational agents, or chatbots, are increasingly used to access all sorts of services using natural language. While open-domain chatbots - like ChatGPT - can converse on any topic, task-oriented chatbots - the focus of this paper - are designed for specific tasks, like booking a flight, obtaining customer support, or setting an appointment. Like any other software, task-oriented chatbots need to be properly tested, usually by defining and executing test scenarios (i.e., sequences of user-chatbot interactions). However, there is currently a lack of methods to quantify the completeness and strength of such test scenarios, which can lead to low-quality tests, and hence to buggy chatbots.
To fill this gap, we propose adapting mutation testing (MuT) for task-oriented chatbots. To this end, we introduce a set of mutation operators that emulate faults in chatbot designs, an architecture that enables MuT on chatbots built using heterogeneous technologies, and a practical realisation as an Eclipse plugin. Moreover, we evaluate the applicability, effectiveness and efficiency of our approach on open-source chatbots, with promising results.
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Join us to learn how UiPath Apps can directly and easily interact with prebuilt connectors via Integration Service--including Salesforce, ServiceNow, Open GenAI, and more.
The best part is you can achieve this without building a custom workflow! Say goodbye to the hassle of using separate automations to call APIs. By seamlessly integrating within App Studio, you can now easily streamline your workflow, while gaining direct access to our Connector Catalog of popular applications.
We’ll discuss and demo the benefits of UiPath Apps and connectors including:
Creating a compelling user experience for any software, without the limitations of APIs.
Accelerating the app creation process, saving time and effort
Enjoying high-performance CRUD (create, read, update, delete) operations, for
seamless data management.
Speakers:
Russell Alfeche, Technology Leader, RPA at qBotic and UiPath MVP
Charlie Greenberg, host
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Tomaz Bratanic
Graph ML and GenAI Expert - Neo4j
Astute Business Solutions | Oracle Cloud Partner |
Your goto partner for Oracle Cloud, PeopleSoft, E-Business Suite, and Ellucian Banner. We are a firm specialized in managed services and consulting.
Recently uploaded (20)
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
From Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMs
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
Spark meets Telemetry
- 1. SPARK MEETS TELEMETRY Mozlandia 2014 Roberto Agostino Vitillo
- 3. TELEMETRY PINGS • If Telemetry is enabled, a ping is generated for each session • Pings are sent to our backend infrastructure as json blobs • Backend validates and stores pings on S3
- 5. TELEMETRY MAP-REDUCE import json def map(k, d, v, cx): j = json.loads(v) os = j['info']['OS'] cx.write(os, 1) def reduce(k, v, cx): cx.write(k, sum(v)) • Processes pings from S3 using a map reduce framework written in Python • https://github.com/mozilla/telemetry-server
- 6. SHORTCOMINGS • Not distributed, limited to a single machine • Doesn’t support chains of map/reduce ops • Doesn’t support SQL-like queries • Batch oriented
- 9. WHAT IS SPARK? • In-memory data analytics cluster computing framework (up to 100x faster than Hadoop) • Comes with over 80 distributed operations for grouping, filtering etc. • Runs standalone or on Hadoop, Mesos and TaskCluster in the future (right Jonas?)
- 10. WHY DO WE CARE? • In memory caching • Interactive command line interface for EDA (think R command line) • Comes with higher level libraries for machine learning and graph processing • Works beautifully on a single machine without tedious setup; doesn’t depend on Hadoop/HDFS • Scala, Python, Clojure and R APIs are available
- 11. WHY DO WE REALLY CARE? The easier we make it to get answers, the more questions we will ask
- 12. MASHUP DEMO
- 13. HOW DOES IT WORK? • User creates Resilient Distributed Datasets (RDDs), transforms and executes them • RDD operations are compiled to a DAG of operators • DAG is compiled into stages • A stage is executed in parallel as a series of tasks
- 14. RDD A parallel dataset with partitions Var A Var B Var C observation observation observation observation Partition Partition
- 15. DAG Logical graph of RDD operations sc.textFile("input") .map(line => line.split(",")) .map(line => (line(0), line(1).toInt)) .reduceByKey(_ + _, 3) RDD[String] RDD[Array[String]] RDD[(String, Int)] RDD[(String, Int)] map map reduceByKey read P1 P2 P3 P4
- 16. RDD[String] RDD[Array[String]] RDD[(String, Int)] RDD[(String, Int)] map map reduceByKey read STAGE Stage 1 Stage 2 P1 P2 P3 P4
- 17. Stage 1 map map STAGE shuffle RDD[String] RDD[Array[String]] RDD[(String, Int)] read input output read map map shuffle P1 P2 P3 P4 T1 T2 T3 T4 Set of tasks that can run in parallel Stage 1
- 18. STAGE Set of tasks that can run in parallel Stage 1 Stage 2
- 19. STAGE Set of tasks that can run in parallel • Tasks are the fundamental unit of work • Tasks are serialised and shipped to workers • Task execution 1. Fetch input 2. Execute 3. Output result task 1 task 2 task 3 task 4
- 20. HANDS-ON
- 21. HANDS-ON 1. Visit telemetry-dash.mozilla.org and sign in using Persona. 2. Click “Launch an ad-hoc analysis worker”. 3. Upload your SSH public key (this allows you to log in to the server once it’s started up). 4. Click “Submit” 5. A Ubuntu machine will be started up on Amazon’s EC2 infrastructure.
- 22. HANDS-ON • Connect to the machine through ssh • Clone the starter template: 1. git clone https://github.com/vitillo/mozilla-telemetry-spark.git 2. cd mozilla-telemetry-spark && source aws/setup.sh 3. sbt console • Open http://bit.ly/1wBHHDH
- 23. TUTORIAL