Big and fast a quest for relevant and real-time analytics
•
2 likes•3,029 views
Our retail banking market demands now more than ever to stay close to our customers, and to carefully understand what services, products, and wishes are relevant for each customer at any given time. This sort of marketing research is often beyond the capacity of traditional BI reporting frameworks. In this talk, we illustrate how we team up data scientists and big data engineers in order to create and scale distributed analyses on a big data platform.
Report
Share
Report
Share
Download to read offline
Recommended
Towards Real-Time banking API's: Introducing Coral, a web api for realtime st...
Banks are innovating. The purpose of this innovation is to transform bank services into meaningful and frictionless customer experiences. A key element in order to achieve that ambitious goal is by providing well tailored and reactive APIs and provide them as the building blocks for greater and smoother customer journeys and experiences. For these API’s to work, internal processes have to evolve as well from batch processing to real time event processing.
In this talk, after providing a brief introduction of the streaming computing landscape, we describe a RESTful API called “Coral” meant to design and deploy customized and flexible data flows. The user can compose data flow for a number of data streaming goals such as on-the-fly data clustering and classifiers, streaming analytics, per-event predictive analysis , real time recommenders. Once the events are processed, Coral passes the resulting analysis as auctionable events for alerting, messaging or further processing to other systems. Coral is a flexible and generic event processing platform to transform streaming data into actionable events via a RESTful API. Those data flows are defined via the Web API by connecting together basic streaming processing elements named “coral actors”. The Coral framework manages those coral actors on a distributed and scalable architecture.
Streaming and real time data processing and analytics are the key elements to an improved customer experience. In this way, you can get the most targeted processing for your domain (marketing customization, personalized recommenders, fraud detection, real time security alerting, etc.). This streaming “data flow” model implies processing customers’ events as soon as they enter via web APIs. This approach borrows a lot from distributed “data flow” concepts developed for processor architectures back in the 80’s. The “Coral” streaming processing engine is generic and built on top of world class libraries such as Akka and Spark, and fully exposed via a RESTful web API.
Share and analyze geonomic data at scale by Andy Petrella and Xavier Tordoir
This document discusses analyzing genomic data at scale using distributed machine learning tools like Spark, ADAM, and the Spark Notebook. It outlines challenges with genomic data like its large size and need for distributed teams in research projects. The document proposes sharing data, processes, and results more efficiently through tools like Shar3 that can streamline the data analysis lifecycle and allow distributed collaboration on genomic research projects and datasets.
Data Engineering Challenges - DSE Day at Bandung Institute of Technology
Data engineers build and maintain robust data pipelines to ingest, process, store and retrieve large amounts of data. They face challenges around throughput and scalability during ingestion, dependencies and performance during processing, efficiency and queries during storage, and ease of use during retrieval. Data engineering involves taking on roles in database administration, architecture, programming and analytics simultaneously.
Big data solutions for advanced marketing analytics
Our retail banking market demands now more than ever to stay close to our customers, and to carefully understand what services, products, and wishes are relevant for each customer at any given time. This sort of marketing research is often beyond the capacity of traditional BI reporting frameworks. In this talk, we illustrate how we team up data scientists and big data engineers in order to create and scale distributed analyses on a big data platform. By using Hadoop and open source statistical language and tools such R and Python, we can execute a variety of machine learning algorithms, and scale them out on a distributed computing framework.
Pinterest’s Story of Streaming Hundreds of Terabytes of Pins from MySQL to S3...
Pinterest moved 100TB of data from MySQL databases to S3 and Hadoop continuously using a new data pipeline. The pipeline uses Kafka to stream database change events in real-time. It incorporates periodic compaction to merge snapshots and deltas into a compact format with 15 minute latency. The new system provides reliability, scalability, and enables features like real-time search and recommendations.
Are we reaching a Data Science Singularity? How Cognitive Computing is emergi...
https://www.bigdataspain.org/2016/program/fri-are-we-reaching-data-science-singularity.html
https://www.youtube.com/watch?v=w1WRk-RAkp0&t=262s&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik&index=43
Spark and Cassandra: An Amazing Apache Love Story by Patrick McFadin
This document discusses the use of Apache Spark and Apache Cassandra together for storing and analyzing large amounts of data. It provides examples of companies like Microsoft and British Gas that use Spark and Cassandra to handle high volumes of event data and perform real-time analytics on time series data. The document also briefly mentions regulatory issues around data protection and shipping data between the US and EU.
Critical Breakthroughs and Challenges in Big Data and Analytics
Critical Breakthroughs and technical Challenges in Big Data Driven Innovation discusses 4 key breakthroughs in Google Cloud Platform's approach to big data:
1. Batch and streaming data processing can be combined using Cloud Dataflow.
2. Real-time data ingestion at massive scales is enabled through technologies like Cloud Bigtable which can process billions of events per hour.
3. Analytics can be done at the speed of thought through BigQuery which allows complex queries on petabytes of data to return results in seconds.
4. Machine learning is made available to everyone through services that offer pre-trained models via APIs and allow custom modeling using TensorFlow on Google Cloud.
Recommended
Towards Real-Time banking API's: Introducing Coral, a web api for realtime st...
Banks are innovating. The purpose of this innovation is to transform bank services into meaningful and frictionless customer experiences. A key element in order to achieve that ambitious goal is by providing well tailored and reactive APIs and provide them as the building blocks for greater and smoother customer journeys and experiences. For these API’s to work, internal processes have to evolve as well from batch processing to real time event processing.
In this talk, after providing a brief introduction of the streaming computing landscape, we describe a RESTful API called “Coral” meant to design and deploy customized and flexible data flows. The user can compose data flow for a number of data streaming goals such as on-the-fly data clustering and classifiers, streaming analytics, per-event predictive analysis , real time recommenders. Once the events are processed, Coral passes the resulting analysis as auctionable events for alerting, messaging or further processing to other systems. Coral is a flexible and generic event processing platform to transform streaming data into actionable events via a RESTful API. Those data flows are defined via the Web API by connecting together basic streaming processing elements named “coral actors”. The Coral framework manages those coral actors on a distributed and scalable architecture.
Streaming and real time data processing and analytics are the key elements to an improved customer experience. In this way, you can get the most targeted processing for your domain (marketing customization, personalized recommenders, fraud detection, real time security alerting, etc.). This streaming “data flow” model implies processing customers’ events as soon as they enter via web APIs. This approach borrows a lot from distributed “data flow” concepts developed for processor architectures back in the 80’s. The “Coral” streaming processing engine is generic and built on top of world class libraries such as Akka and Spark, and fully exposed via a RESTful web API.
Share and analyze geonomic data at scale by Andy Petrella and Xavier Tordoir
This document discusses analyzing genomic data at scale using distributed machine learning tools like Spark, ADAM, and the Spark Notebook. It outlines challenges with genomic data like its large size and need for distributed teams in research projects. The document proposes sharing data, processes, and results more efficiently through tools like Shar3 that can streamline the data analysis lifecycle and allow distributed collaboration on genomic research projects and datasets.
Data Engineering Challenges - DSE Day at Bandung Institute of Technology
Data engineers build and maintain robust data pipelines to ingest, process, store and retrieve large amounts of data. They face challenges around throughput and scalability during ingestion, dependencies and performance during processing, efficiency and queries during storage, and ease of use during retrieval. Data engineering involves taking on roles in database administration, architecture, programming and analytics simultaneously.
Big data solutions for advanced marketing analytics
Our retail banking market demands now more than ever to stay close to our customers, and to carefully understand what services, products, and wishes are relevant for each customer at any given time. This sort of marketing research is often beyond the capacity of traditional BI reporting frameworks. In this talk, we illustrate how we team up data scientists and big data engineers in order to create and scale distributed analyses on a big data platform. By using Hadoop and open source statistical language and tools such R and Python, we can execute a variety of machine learning algorithms, and scale them out on a distributed computing framework.
Pinterest’s Story of Streaming Hundreds of Terabytes of Pins from MySQL to S3...
Pinterest moved 100TB of data from MySQL databases to S3 and Hadoop continuously using a new data pipeline. The pipeline uses Kafka to stream database change events in real-time. It incorporates periodic compaction to merge snapshots and deltas into a compact format with 15 minute latency. The new system provides reliability, scalability, and enables features like real-time search and recommendations.
Are we reaching a Data Science Singularity? How Cognitive Computing is emergi...
https://www.bigdataspain.org/2016/program/fri-are-we-reaching-data-science-singularity.html
https://www.youtube.com/watch?v=w1WRk-RAkp0&t=262s&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik&index=43
Spark and Cassandra: An Amazing Apache Love Story by Patrick McFadin
This document discusses the use of Apache Spark and Apache Cassandra together for storing and analyzing large amounts of data. It provides examples of companies like Microsoft and British Gas that use Spark and Cassandra to handle high volumes of event data and perform real-time analytics on time series data. The document also briefly mentions regulatory issues around data protection and shipping data between the US and EU.
Critical Breakthroughs and Challenges in Big Data and Analytics
Critical Breakthroughs and technical Challenges in Big Data Driven Innovation discusses 4 key breakthroughs in Google Cloud Platform's approach to big data:
1. Batch and streaming data processing can be combined using Cloud Dataflow.
2. Real-time data ingestion at massive scales is enabled through technologies like Cloud Bigtable which can process billions of events per hour.
3. Analytics can be done at the speed of thought through BigQuery which allows complex queries on petabytes of data to return results in seconds.
4. Machine learning is made available to everyone through services that offer pre-trained models via APIs and allow custom modeling using TensorFlow on Google Cloud.
Final deck
The document provides an introduction to big data concepts including what big data is, how issues with processing large amounts of data were addressed, and what Apache Hadoop is. It then discusses two use case scenarios involving using Hadoop and Apache Cassandra to solve challenges around product recommendations and managing large volumes of email data at scale.
Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...
So, you finally have a data ecosystem with Kafka and Hadoop both deployed and operating correctly at scale. Congratulations. Are you done? Far from it.
As the birthplace of Kafka and an early adopter of Hadoop, LinkedIn has 13 years of combined experience using Kafka and Hadoop at scale to run a data-driven company. Both Kafka and Hadoop are flexible, scalable infrastructure pieces, but using these technologies without a clear idea of what the higher-level data ecosystem should be is perilous. Shirshanka Das and Yael Garten share best practices around data models and formats, choosing the right level of granularity of Kafka topics and Hadoop tables, and moving data efficiently and correctly between Kafka and Hadoop and explore a data abstraction layer, Dali, that can help you to process data seamlessly across Kafka and Hadoop.
Beyond pure technology, Shirshanka and Yael outline the three components of a great data culture and ecosystem and explain how to create maintainable data contracts between data producers and data consumers (like data scientists and data analysts) and how to standardize data effectively in a growing organization to enable (and not slow down) innovation and agility. They then look to the future, envisioning a world where you can successfully deploy a data abstraction of views on Hadoop data, like a data API as a protective and enabling shield. Along the way, Shirshanka and Yael discuss observations on how to enable teams to be good data citizens in producing, consuming, and owning datasets and offer an overview of LinkedIn’s governance model: the tools, process and teams that ensure that its data ecosystem can handle change and sustain #datasciencehappiness.
Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...
Twitter generates billions and billions of events per day. Analyzing these events in real time presents a massive challenge. Twitter designed and deployed a new streaming system called Heron. Heron has been in production nearly 2 years and is widely used by several teams for diverse use cases. This talk looks at Twitter's operating experiences and challenges of running Heron at scale and the approaches taken to solve those challenges.
Big Data LDN 2016: Out of the Data Warehouses, and into the Data Lakes and St...
This document discusses using data streams and lakes for big data analytics in utilities. It begins by explaining how smart meter rollouts have increased meter reads from 80 million to 350 billion per year. Traditional data warehousing is challenged by the time criticality, resource demands, and scale of this unbounded data. The document then introduces data streams for real-time analytics and data lakes for flexible storage of huge amounts of structured, semi-structured, and unstructured data. It describes Valo, an open lambda architecture that uses multiple repositories and stream processing to enable both real-time and historical analysis across disparate data sources.
Dataiku Flow and dctc - Berlin Buzzwords
This document discusses Dataiku Flow and DCTC. Dataiku Flow is a data-driven orchestration framework for complex data pipelines that manages data dependencies and parallelization. It allows defining datasets and tasks to transform data. DCTC is a tool that can manipulate files across different storage systems like S3, GCS, HDFS to perform operations like copying, synchronizing, dispatching files. It aims to simplify common data transfer pains. The presentation concludes with contacting information for Dataiku executives.
The of Operational Analytics Data Store
Apache Druid basics and history. Where it fits. What it's best for. How is it different from other similar tech.
Big Data - Part I
This document discusses big data concepts including data science, the evolution of data storage technologies from relational databases to modern data lakes, and Hadoop. It describes data science as extracting insights from large diverse data sets. It outlines the progression from relational databases and data warehouses to NoSQL and NewSQL systems. It also explains key aspects of Hadoop including its origins, components like HDFS and MapReduce, and its evolution towards enterprise capabilities in versions like Hadoop 2.x.
What's next for Big Data? -- Apache Spark
What's next for Big Data? -- Apache SparkTUMRA | Big Data Science - Gain a competitive advantage through Big Data & Data Science
Michael Cutler (CTO cofounder of TUMRA) provides a high-level introduction to Apache Spark in a presentation given at ‘Big Data Week 2014’ #BDW14 held at University College London.
TUMRA were early adopters of Spark after a brief PoC in Dec ‘12 and took it to production just a few months later. The main motivation to do so was the inflexibility and high-latency of Hadoop Map/Reduce jobs and the knock-on effect for technology that utilises it (Mahout machine learning, Hive data warehousing, Cascading).
With two primary uses case ‘Ecommerce Personalisation’ and ‘Marketing Automation’ TUMRA are currently flowing around 29 million ‘user engagement events’ (JSON) each day through Apache Kafka and Spark Streaming at peak rates of up to 800 events per second.
TUMRA use Apache Spark on Amazon Web Services (EC2) in production for a mix of machine learning model building, graph analytics and near-real-time reporting.
To learn more about how we use Spark and the services we can deliver through our Platform please contact: hello@tumra.com
Data Analytics and Processing at Snap - Druid Meetup LA - September 2018
Charles Allen covers data processing, analytics, and insights systems at Snap. Strength points for Druid use cases are called out as are differences in some of the processing systems used.
This is the slide collection from the second talk from:
https://www.meetup.com/druidio-la/events/254080924/
Self Service Analytics at Twitch
As Twitch grew, both the amount of data we received and the number of employees interested in the data grew rapidly. In order to continue empowering decision making as we scaled, we turned to using Druid and Imply to provide self service analytics to both our technical and non technical staff allowing them to drill into high level metrics in lieu of reading generated reports.
In this talk, learn how Twitch implemented a common analytics platform for the needs of many different teams supporting hundreds of users, thousands of queries, and ~5 billion events each day. This session will explain our Druid architecture in detail, including:
-The end-to-end architecture deployed on Amazon that includes Kinesis, RDS, S3, Druid, Pivot and Tableau
-How the data is brought together to deliver a unified view of live customer engagement and historical trends
-Operational best practices we learnt scaling Druid
-An example walk through using the platform
Clickstream & Social Media Analysis using Apache Spark
Clickstream & Social Media Analysis using Apache SparkTUMRA | Big Data Science - Gain a competitive advantage through Big Data & Data Science
Use cases and examples using Apache Spark, presented at the Hadoop User Group (UK) November 2014 Hadoop Meetup
http://www.meetup.com/hadoop-users-group-uk/events/217791892/Apache Druid: The Foundation of Fortune 500 “Analytical Decision-Making"
This document discusses Apache Druid, an open-source distributed real-time analytics database. It summarizes Druid's evolution, architecture, use cases, and how companies use it. The document outlines Druid's ability to handle large, high-dimensional datasets with sub-second queries and discusses its core components like segments for efficient storage and parallelism. It concludes by inviting the reader to join the Druid community.
Microservices Live
Glynn Bird - Building the "microservices way" involves breaking monolithic IT systems into small, decoupled services that each to one job well. This talk builds a practical microservices architecture during the talk using small Node.js apps that perform storage, analytics and visualisation tasks. Learn how to orchestrate your own microservice architecture using simple, easily-tested building blocks.
OSCON 2015
Slides from my talk at Oscon 2015 talking about the things we are measuring in an attempt to understand our data.
Big Graph Analytics on Neo4j with Apache Spark
In this talk I will introduce you to a Docker container that provides you an easy way to do distributed graph processing using Apache Spark GraphX and a Neo4j graph database. You'll learn how to analyze big data graphs that are exported from Neo4j and consequently updated from the results of a Spark GraphX analysis. The types of analysis I will be talking about are PageRank, connected components, triangle counting, and community detection.
Database technologies have evolved to be able to store big data, but are largely inflexible. For complex graph data models stored in a relational database there may be tedious transformations and shuffling around of data to perform large scale analysis.
Fast and scalable analysis of big data has become a critical competitive advantage for companies. There are open source tools like Apache Hadoop and Apache Spark that are providing opportunities for companies to solve these big data problems in a scalable way. Platforms like these have become the foundation of the big data analysis movement.
Speakers
Introduction_OF_Hadoop_and_BigData
Big data refers to large, complex datasets that are difficult to process using traditional database management tools. It has become a business strategy for leveraging information resources generated by social media, scientific instruments, mobile devices, sensors, and networks. While more data can be collected than ever before, the challenges lie in managing, analyzing, summarizing, visualizing, and discovering knowledge from the data in a timely and scalable way. Hadoop is an open-source software framework that addresses these challenges through distributed storage and processing of large datasets across clusters of computers using simple programming models. It provides reliable storage of data via its Hadoop Distributed File System and scalable processing of that data using the MapReduce programming model.
Non-Relational Databases: This hurts. I like it.
The document discusses non-relational databases, providing an overview of their characteristics and comparing them to relational databases. It outlines some popular non-relational database platforms, and uses the example of an open government project to demonstrate how CouchDB could be used to store and query schema-less data in a scalable way.
Strata London 16: sightseeing, venues, and friends
Which venues have similar visiting patterns? How can we detect when a user is on vacation? Can we predict which venues will be favorited by users by examining their friends' preferences? Natalino Busa explains how these predictive analytics tasks can be accomplished by using Spark SQL, Spark ML, and just a few lines of Scala code.
Blue Pill/Red Pill: The Matrix of Thousands of Data Streams
Designing a streaming application which has to process data from 1 or 2 streams is easy. Any streaming framework which provides scalability, high-throughput, and fault-tolerance would work. But when the number of streams start growing in order 100s or 1000s, managing them can be daunting. How would you share resources among 1000s of streams with all of them running 24×7? Manage their state, Apply advanced streaming operations, Add/Delete streams without restarting? This talk explains common scenarios & shows techniques that can handle thousands of streams using Spark Structured Streaming.
Strata Online_road_to_enterprise_data_2011
This document discusses the transition from traditional business intelligence (BI) to big data. It notes that BI focuses on structured transactional data and answering questions about the past, while big data leverages both structured and unstructured behavioral data from diverse sources to answer questions about the future. The document outlines technologies like Hadoop, NoSQL databases, and cloud computing that enable organizations to capture and analyze large, dynamic datasets. It also discusses the roles of data scientists and new types of visualizations and devices that support deriving insights from big data.
Lunar lander - The Game
O documento descreve a história do jogo Lunar Lander desde 1969 até 1979, quando foi lançado para Atari. Começou como um jogo de texto baseado na missão Apollo 11 e evoluiu para versões com gráficos que simulavam o pouso lunar, exigindo controle do empuxo e combustível para evitar uma queda ou colisão.
More Related Content
What's hot
Final deck
The document provides an introduction to big data concepts including what big data is, how issues with processing large amounts of data were addressed, and what Apache Hadoop is. It then discusses two use case scenarios involving using Hadoop and Apache Cassandra to solve challenges around product recommendations and managing large volumes of email data at scale.
Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...
So, you finally have a data ecosystem with Kafka and Hadoop both deployed and operating correctly at scale. Congratulations. Are you done? Far from it.
As the birthplace of Kafka and an early adopter of Hadoop, LinkedIn has 13 years of combined experience using Kafka and Hadoop at scale to run a data-driven company. Both Kafka and Hadoop are flexible, scalable infrastructure pieces, but using these technologies without a clear idea of what the higher-level data ecosystem should be is perilous. Shirshanka Das and Yael Garten share best practices around data models and formats, choosing the right level of granularity of Kafka topics and Hadoop tables, and moving data efficiently and correctly between Kafka and Hadoop and explore a data abstraction layer, Dali, that can help you to process data seamlessly across Kafka and Hadoop.
Beyond pure technology, Shirshanka and Yael outline the three components of a great data culture and ecosystem and explain how to create maintainable data contracts between data producers and data consumers (like data scientists and data analysts) and how to standardize data effectively in a growing organization to enable (and not slow down) innovation and agility. They then look to the future, envisioning a world where you can successfully deploy a data abstraction of views on Hadoop data, like a data API as a protective and enabling shield. Along the way, Shirshanka and Yael discuss observations on how to enable teams to be good data citizens in producing, consuming, and owning datasets and offer an overview of LinkedIn’s governance model: the tools, process and teams that ensure that its data ecosystem can handle change and sustain #datasciencehappiness.
Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...
Twitter generates billions and billions of events per day. Analyzing these events in real time presents a massive challenge. Twitter designed and deployed a new streaming system called Heron. Heron has been in production nearly 2 years and is widely used by several teams for diverse use cases. This talk looks at Twitter's operating experiences and challenges of running Heron at scale and the approaches taken to solve those challenges.
Big Data LDN 2016: Out of the Data Warehouses, and into the Data Lakes and St...
This document discusses using data streams and lakes for big data analytics in utilities. It begins by explaining how smart meter rollouts have increased meter reads from 80 million to 350 billion per year. Traditional data warehousing is challenged by the time criticality, resource demands, and scale of this unbounded data. The document then introduces data streams for real-time analytics and data lakes for flexible storage of huge amounts of structured, semi-structured, and unstructured data. It describes Valo, an open lambda architecture that uses multiple repositories and stream processing to enable both real-time and historical analysis across disparate data sources.
Dataiku Flow and dctc - Berlin Buzzwords
This document discusses Dataiku Flow and DCTC. Dataiku Flow is a data-driven orchestration framework for complex data pipelines that manages data dependencies and parallelization. It allows defining datasets and tasks to transform data. DCTC is a tool that can manipulate files across different storage systems like S3, GCS, HDFS to perform operations like copying, synchronizing, dispatching files. It aims to simplify common data transfer pains. The presentation concludes with contacting information for Dataiku executives.
The of Operational Analytics Data Store
Apache Druid basics and history. Where it fits. What it's best for. How is it different from other similar tech.
Big Data - Part I
This document discusses big data concepts including data science, the evolution of data storage technologies from relational databases to modern data lakes, and Hadoop. It describes data science as extracting insights from large diverse data sets. It outlines the progression from relational databases and data warehouses to NoSQL and NewSQL systems. It also explains key aspects of Hadoop including its origins, components like HDFS and MapReduce, and its evolution towards enterprise capabilities in versions like Hadoop 2.x.
What's next for Big Data? -- Apache Spark
What's next for Big Data? -- Apache SparkTUMRA | Big Data Science - Gain a competitive advantage through Big Data & Data Science
Michael Cutler (CTO cofounder of TUMRA) provides a high-level introduction to Apache Spark in a presentation given at ‘Big Data Week 2014’ #BDW14 held at University College London.
TUMRA were early adopters of Spark after a brief PoC in Dec ‘12 and took it to production just a few months later. The main motivation to do so was the inflexibility and high-latency of Hadoop Map/Reduce jobs and the knock-on effect for technology that utilises it (Mahout machine learning, Hive data warehousing, Cascading).
With two primary uses case ‘Ecommerce Personalisation’ and ‘Marketing Automation’ TUMRA are currently flowing around 29 million ‘user engagement events’ (JSON) each day through Apache Kafka and Spark Streaming at peak rates of up to 800 events per second.
TUMRA use Apache Spark on Amazon Web Services (EC2) in production for a mix of machine learning model building, graph analytics and near-real-time reporting.
To learn more about how we use Spark and the services we can deliver through our Platform please contact: hello@tumra.com
Data Analytics and Processing at Snap - Druid Meetup LA - September 2018
Charles Allen covers data processing, analytics, and insights systems at Snap. Strength points for Druid use cases are called out as are differences in some of the processing systems used.
This is the slide collection from the second talk from:
https://www.meetup.com/druidio-la/events/254080924/
Self Service Analytics at Twitch
As Twitch grew, both the amount of data we received and the number of employees interested in the data grew rapidly. In order to continue empowering decision making as we scaled, we turned to using Druid and Imply to provide self service analytics to both our technical and non technical staff allowing them to drill into high level metrics in lieu of reading generated reports.
In this talk, learn how Twitch implemented a common analytics platform for the needs of many different teams supporting hundreds of users, thousands of queries, and ~5 billion events each day. This session will explain our Druid architecture in detail, including:
-The end-to-end architecture deployed on Amazon that includes Kinesis, RDS, S3, Druid, Pivot and Tableau
-How the data is brought together to deliver a unified view of live customer engagement and historical trends
-Operational best practices we learnt scaling Druid
-An example walk through using the platform
Clickstream & Social Media Analysis using Apache Spark
Clickstream & Social Media Analysis using Apache SparkTUMRA | Big Data Science - Gain a competitive advantage through Big Data & Data Science
Use cases and examples using Apache Spark, presented at the Hadoop User Group (UK) November 2014 Hadoop Meetup
http://www.meetup.com/hadoop-users-group-uk/events/217791892/Apache Druid: The Foundation of Fortune 500 “Analytical Decision-Making"
This document discusses Apache Druid, an open-source distributed real-time analytics database. It summarizes Druid's evolution, architecture, use cases, and how companies use it. The document outlines Druid's ability to handle large, high-dimensional datasets with sub-second queries and discusses its core components like segments for efficient storage and parallelism. It concludes by inviting the reader to join the Druid community.
Microservices Live
Glynn Bird - Building the "microservices way" involves breaking monolithic IT systems into small, decoupled services that each to one job well. This talk builds a practical microservices architecture during the talk using small Node.js apps that perform storage, analytics and visualisation tasks. Learn how to orchestrate your own microservice architecture using simple, easily-tested building blocks.
OSCON 2015
Slides from my talk at Oscon 2015 talking about the things we are measuring in an attempt to understand our data.
Big Graph Analytics on Neo4j with Apache Spark
In this talk I will introduce you to a Docker container that provides you an easy way to do distributed graph processing using Apache Spark GraphX and a Neo4j graph database. You'll learn how to analyze big data graphs that are exported from Neo4j and consequently updated from the results of a Spark GraphX analysis. The types of analysis I will be talking about are PageRank, connected components, triangle counting, and community detection.
Database technologies have evolved to be able to store big data, but are largely inflexible. For complex graph data models stored in a relational database there may be tedious transformations and shuffling around of data to perform large scale analysis.
Fast and scalable analysis of big data has become a critical competitive advantage for companies. There are open source tools like Apache Hadoop and Apache Spark that are providing opportunities for companies to solve these big data problems in a scalable way. Platforms like these have become the foundation of the big data analysis movement.
Speakers
Introduction_OF_Hadoop_and_BigData
Big data refers to large, complex datasets that are difficult to process using traditional database management tools. It has become a business strategy for leveraging information resources generated by social media, scientific instruments, mobile devices, sensors, and networks. While more data can be collected than ever before, the challenges lie in managing, analyzing, summarizing, visualizing, and discovering knowledge from the data in a timely and scalable way. Hadoop is an open-source software framework that addresses these challenges through distributed storage and processing of large datasets across clusters of computers using simple programming models. It provides reliable storage of data via its Hadoop Distributed File System and scalable processing of that data using the MapReduce programming model.
Non-Relational Databases: This hurts. I like it.
The document discusses non-relational databases, providing an overview of their characteristics and comparing them to relational databases. It outlines some popular non-relational database platforms, and uses the example of an open government project to demonstrate how CouchDB could be used to store and query schema-less data in a scalable way.
Strata London 16: sightseeing, venues, and friends
Which venues have similar visiting patterns? How can we detect when a user is on vacation? Can we predict which venues will be favorited by users by examining their friends' preferences? Natalino Busa explains how these predictive analytics tasks can be accomplished by using Spark SQL, Spark ML, and just a few lines of Scala code.
Blue Pill/Red Pill: The Matrix of Thousands of Data Streams
Designing a streaming application which has to process data from 1 or 2 streams is easy. Any streaming framework which provides scalability, high-throughput, and fault-tolerance would work. But when the number of streams start growing in order 100s or 1000s, managing them can be daunting. How would you share resources among 1000s of streams with all of them running 24×7? Manage their state, Apply advanced streaming operations, Add/Delete streams without restarting? This talk explains common scenarios & shows techniques that can handle thousands of streams using Spark Structured Streaming.
Strata Online_road_to_enterprise_data_2011
This document discusses the transition from traditional business intelligence (BI) to big data. It notes that BI focuses on structured transactional data and answering questions about the past, while big data leverages both structured and unstructured behavioral data from diverse sources to answer questions about the future. The document outlines technologies like Hadoop, NoSQL databases, and cloud computing that enable organizations to capture and analyze large, dynamic datasets. It also discusses the roles of data scientists and new types of visualizations and devices that support deriving insights from big data.
What's hot (20)
Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...
Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...
Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...
Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...
Big Data LDN 2016: Out of the Data Warehouses, and into the Data Lakes and St...
Big Data LDN 2016: Out of the Data Warehouses, and into the Data Lakes and St...
Data Analytics and Processing at Snap - Druid Meetup LA - September 2018
Data Analytics and Processing at Snap - Druid Meetup LA - September 2018
Clickstream & Social Media Analysis using Apache Spark
Clickstream & Social Media Analysis using Apache Spark
Apache Druid: The Foundation of Fortune 500 “Analytical Decision-Making"
Apache Druid: The Foundation of Fortune 500 “Analytical Decision-Making"
Strata London 16: sightseeing, venues, and friends
Strata London 16: sightseeing, venues, and friends
Blue Pill/Red Pill: The Matrix of Thousands of Data Streams
Blue Pill/Red Pill: The Matrix of Thousands of Data Streams
Viewers also liked
Lunar lander - The Game
O documento descreve a história do jogo Lunar Lander desde 1969 até 1979, quando foi lançado para Atari. Começou como um jogo de texto baseado na missão Apollo 11 e evoluiu para versões com gráficos que simulavam o pouso lunar, exigindo controle do empuxo e combustível para evitar uma queda ou colisão.
Historyofcomputers
This document provides a brief history of computers from early counting methods like fingers and pebbles to modern electronic digital computers. It describes the evolution of mechanical computing devices like the abacus and Babbage's analytical engine. Important early electronic computers are highlighted such as the Atanasoff-Berry Computer, ENIAC, and UNIVAC. The development of key technologies like the integrated circuit, microprocessor, and internet are noted. The history shows how computers advanced rapidly from vacuum tubes to transistors to semiconductors.
Unix.part1.history
UNIX was created in 1969 by Kenneth Thompson and Dennis Ritchie at Bell Labs as a simpler alternative to the Multics operating system. It originated from an earlier game called "Space Travel" created by Thompson for the PDP-7 computer. Early versions included the System V file system and in 1973, the first paper on UNIX was published titled "The UNIX Time-Sharing System". Over time, UNIX evolved through various versions including BSD releases from Berkeley and commercial versions from Microsoft and SCO.
software History
The document discusses the history and development of computer software. It covers the different generations of software from the first generation where programs were entered manually using switches and dials, to modern programming languages. It also defines different types of software including system software, programming tools, and application software. Key topics covered include software architecture, quality, licensing, and the reasons why new programming languages and software are continually designed.
Software Testing: History, Trends, Perspectives - a Brief Overview
In this presentation you can learn about different types of software testing, new technologies and methodologies. It contains an overview of software testing perspectives.
Advanced API Design: how an awesome API can help you make friends, get rich, ...
The document discusses how to design advanced APIs that can attract users, generate revenue, and impact the world. It provides examples of successful APIs and recommends designing APIs that follow REST conventions, are well documented, supported, validated, versioned, and performant. APIs that are easy to use and solve major problems can attract many users, lead to business opportunities, and drive innovation with widespread adoption.
Streaming Api Design with Akka, Scala and Spray
The document proposes designing a streaming API that aggregates data from wristbands worn by runners. It defines endpoints to create a stream, add filters to the stream to calculate metrics like maximum heart rate per second within groups, ingest data, and retrieve the filtered output. The API is built on Akka actors for asynchronous and reactive processing and on Spray for building HTTP services on top of Akka actors.
Computer Software introduction
in Short words we explain Computer software introduction , it will give u a short introduction about software
Animated Visualization of Software History Using Software Evolution Storyboards
This document proposes and demonstrates evolution storyboards, a new technique for visualizing software evolution over time. Evolution storyboards display the evolution of a software system through a series of panels that depict important events and periods in the system's lifetime. Nodes represent code elements and edges show code changes. Properties of the layout algorithm and approach are described. The technique is illustrated using open source systems and shown to help understand software structure, explain decay, and identify refactoring opportunities.
Operating Systems: Network Management
The document discusses different types of computer network topologies and configurations. It describes star, ring, bus, tree, and hybrid topologies. It also discusses network operating systems (NOS) and distributed operating systems (DOS). Finally, it defines local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), and wireless local area networks (WLANs).
Operating Systems: A History of Linux
This document provides a history and overview of Linux. It discusses the prehistory of Linux including early operating systems like Unix, BSD, and MINIX. It then describes how Linus Torvalds created the first version of the Linux kernel in 1991 and made it open source. The document outlines some major versions of the Linux kernel and provides timelines showing the development and releases of popular Linux distributions like Debian, Red Hat, Ubuntu, and Android. It also summarizes several popular Linux desktop environments such as KDE, GNOME, Xfce, and Cinnamon.
Viewers also liked (12)
Software Testing: History, Trends, Perspectives - a Brief Overview
Software Testing: History, Trends, Perspectives - a Brief Overview
Advanced API Design: how an awesome API can help you make friends, get rich, ...
Advanced API Design: how an awesome API can help you make friends, get rich, ...
Animated Visualization of Software History Using Software Evolution Storyboards
Animated Visualization of Software History Using Software Evolution Storyboards
Similar to Big and fast a quest for relevant and real-time analytics
Traitement d'événements
Après-midi - Track 2 - S3 - Traitement d'evenements
Cette session vous a plû ? Si c'est le cas, n'hésitez pas à vous inscrire à notre Summit !
Big Data Use Cases and Solutions in the AWS Cloud
The AWS cloud computing platform has disrupted big data. Managing big data applications used to be for only well-funded research organizations and large corporations, but not any longer. Hear from Ben Butler, Big Data Solutions Marketing Manager for AWS, to learn how our customers are using big data services in the AWS cloud to innovate faster than ever before. Not only is AWS technology available to everyone, but it is self-service, on-demand, and featuring innovative technology and flexible pricing models at low cost with no commitments. Learn from customer success stories, as Ben shares real-world case studies describing the specific big data challenges being solved on AWS. We will conclude with a discussion around the tutorials, public datasets, test drives, and our grants program - all of the resources needed to get you started quickly.
Data Apps with the Lambda Architecture - with Real Work Examples on Merging B...
The document discusses the Lambda architecture, which provides a common pattern for integrating real-time and batch processing systems. It describes the key components of Lambda - the batch layer, speed layer, and serving layer. The challenges of implementing Lambda are that it requires multiple systems and technologies to be coordinated. Real-world examples are needed to help practical application. The document also provides examples of medical and customer analytics use cases that could benefit from a Lambda approach.
Prassnitha Sampath - Real Time Big Data Analytics with Kafka, Storm & HBase -...
Relevance and Personalization is crucial to building personalized local commerce experience at Groupon. Talk covers overview of the real time analytics infrastructure that handles over 3 million events/ second and stores and scales to billions of data points. Solution covers how our Kafka -> Storm -> Redis/ HBase pipeline is used to generate real time analytics for hundreds of millions of users of Groupon. Solution includes various architecture design choices and tradeoffs including some interesting algorithmic choices such as Bloom Filters & Hyper Log Log. Attendees can take away learnings from our real-life experience that can help them understand various tuning methods, their tradeoffs and apply them in their solutions
Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, Oath
Offline and stream processing of big data sets can be done with tools such as Hadoop, Spark, and Storm, but what if you need to process big data at the time a user is making a request? Vespa (http://www.vespa.ai) allows you to search, organize and evaluate machine-learned models from e.g TensorFlow over large, evolving data sets with latencies in the tens of milliseconds. Vespa is behind the recommendation, ad targeting, and search at Yahoo where it handles billions of daily queries over billions of documents.
Big Data Architectural Patterns
This document provides an overview of big data architectural patterns and best practices on AWS. It discusses challenges of big data and how to simplify big data processing. It covers ingestion, storage, analysis and visualization technologies to use as well as design patterns. Key technologies discussed include Amazon Kinesis, DynamoDB, S3, Redshift, EMR, Lambda and design approaches like decoupled data bus and using the right tool for each job.
Big data business case
This document discusses big data concepts including what constitutes big data, dimensions of big data, the big data lifecycle, and where big data is used. It provides examples of big data use cases in industries like automotive, healthcare, and retail. Key aspects of big data technologies like Hadoop, HDFS, MapReduce, and NoSQL databases are explained at a high level. The challenges of big data including the CAP theorem and approaches to distributed and parallel processing are also summarized.
AWS APAC Webinar Week - Big Data on AWS. RedShift, EMR, & IOT
The world is producing an ever-increasing volume, velocity, and variety of data including data from devices. As we step into the era of Internet of things (IOT), for many consumers, batch analytics is no longer enough; they need sub-second analysis on fast-moving data. AWS delivers many technologies for solving big data and IOT problems. But what services should you use, why, when, and how? In this webinar where we simplify big data processing as a pipeline comprising various stages: ingest, store, process, analyze & visualize. Next, we discuss how to choose the right technology in each stage based on criteria such as data structure, query latency, cost, request rate, item size, data volume, and durability. Finally, we provide a reference architecture, design patterns, and best practices for assembling these technologies to solve your big data problems.
UnConference for Georgia Southern Computer Science March 31, 2015
I presented to the Georgia Southern Computer Science ACM group. Rather than one topic for 90 minutes, I decided to do an UnConference. I presented them a list of 8-9 topics, let them vote on what to talk about, then repeated.
Each presentation was ~8 minutes, (Except Career) and was by no means an attempt to explain the full concept or technology. Only to wake up their interest.
GraphLab Conference 2014 Keynote - Carlos Guestrin
This document introduces GraphLab Create, a machine learning toolkit that aims to help data scientists unleash the power of data science from inspiration to production. It highlights key features of GraphLab Create including scalable data structures that allow analyzing big data on a single machine without running out of memory, robust machine learning algorithms, and tools for deploying predictive applications and services to production environments from the same code used for prototyping. The document provides examples of using GraphLab Create for tasks like recommender systems, fraud detection, and deep learning. It emphasizes that GraphLab Create allows users to be productive on a single machine, at scale, and in production.
Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...
Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...Yahoo Developer Network
Offline and stream processing of big data sets can be done with tools such as Hadoop, Spark, and Storm, but what if you need to process big data at the time a user is making a request?
This presentation introduces Vespa (http://vespa.ai) – the open source big data serving engine.
Vespa allows you to search, organize, and evaluate machine-learned models from e.g TensorFlow over large, evolving data sets with latencies in the tens of milliseconds. Vespa is behind the recommendation, ad targeting, and search at Yahoo where it handles billions of daily queries over billions of documents and was recently open sourced at http://vespa.ai.(ARC311) Decoding The Genetic Blueprint Of Life On A Cloud Ecosystem
Thermo Fisher Scientific, a world leader in biotechnology, has built a new polymerase chain reaction (PCR) system for DNA sequencing. Designed for low- to midlevel throughput laboratories that conduct real time PCR experiments, the system runs on individual QuantStudio devices. These devices are connected to Thermo Fisher’s cloud computing platform, which is built on AWS using Amazon EC2, Amazon DynamoDB, and Amazon S3. With this single platform, applied and clinical researchers can learn, analyze, share, collaborate, and obtain support. Researchers worldwide can now collaborate online in real time and access their data wherever and whenever necessary. Laboratories can also share experimental conditions and results with their partners while providing a uniform experience for every user and helping to minimize training and errors. The net result is increased collaboration, faster time to market, fewer errors, and lower cost. We have architected a solution that uses Amazon EMR, DynamoDB, Amazon Elasticache, and S3. In this presentation, we share our architecture, lessons learned, best design patterns for NoSQL, strategies for leveraging EMR with DynamoDB, and a flexible solution that our scientist use. We also share our next step in architecture evolution.
Redis and Bloom Filters - Atlanta Java Users Group 9/2014
This document discusses using Redis to enable failing fast with Bloom filters. It provides an overview of Redis as a data structure server and key-value cache, as well as Bloom filters as a space-efficient probabilistic data structure for membership testing. By storing Bloom filters in Redis, they can be made updatable and persistent while avoiding the high costs of updating and querying large Bloom filters stored in databases. Examples are given where Redis-backed Bloom filters can be used to optimize queries and decide whether expensive operations need to be performed.
Big Data & Analytics - Innovating at the Speed of Light
Amazon Web Services proporciona una amplia gama de servicios que le ayudarán a crear e implementar aplicaciones de análisis de big data de forma rápida y sencilla. AWS ofrece un acceso rápido a recursos de TI económicos y flexibles, algo que permitirá escalar prácticamente cualquier aplicación de big data con rapidez, incluidos almacenamiento de datos, análisis de clics, detección de elementos fraudulentos, motores de recomendación, proceso ETL impulsado por eventos, informática sin servidor y procesamiento del Internet de las cosas. Con AWS no necesita hacer grandes inversiones iniciales de tiempo o dinero para crear y mantener la infraestructura. En su lugar, puede aprovisionar exactamente el tipo y el tamaño adecuado de los recursos que necesita para impulsar sus aplicaciones de análisis de big data. Puede obtener acceso a tantos recursos como necesite, prácticamente al instante, y pagar únicamente por los utilice.
Scaling APIs: Predict, Prepare for, Overcome the Challenges
This document discusses strategies for scaling APIs to handle increasing volumes of traffic. It outlines considerations and approaches for handling throughput from 1 transaction per second up to 100,000 transactions per second. Lower volumes can typically be handled through database and application tuning. Higher volumes require caching, database sharding or NoSQL databases, and scaling out application servers and other infrastructure. API and client design can also impact scaling by reducing unnecessary calls and payloads. Latency and network bandwidth also become important concerns at high volumes.
ConFoo 2017: Introduction to performance optimization of .NET web apps
This document discusses performance optimization of .NET web apps. It defines performance as response time and resource utilization. It emphasizes measuring performance before and during optimization to identify the right things to optimize. A variety of tools are presented for different levels of performance monitoring and profiling, including Application Performance Monitoring, lightweight code profilers, and code profilers for highest detail. Best practices like leveraging load balancers and avoiding implicit transactions are also recommended.
Kellogg XML Holland Speech
The document discusses the evolution of databases and the rise of content applications. It argues that traditional databases are no longer suitable for next-generation applications that require flexible querying of content. New specialized database systems optimized for XML and content are needed to power applications that help users complete tasks and gain insights from unstructured data. Content applications will blend software and content to seamlessly support users in interactive, read-write experiences.
Need for Time series Database
This document discusses the need for a time series database and introduces OpenTSDB as an option. Some key points:
- Time series data is useful for analyzing metrics and patterns over time but is currently scattered across different databases.
- OpenTSDB is an open source time series database that can store trillions of data points, scale using HBase, and never loses precision.
- It is optimized for write throughput and can handle thousands of data points per second. Reads depend on the cardinality of metrics but it supports time-based queries.
- OpenTSDB uses HBase under the hood and stores tags with metrics to allow for flexible filtering of time series data without affecting performance.
Database and Analytics on the AWS Cloud
Antoine Genereux takes us on a detailed overview of the Database solutions available on the AWS Cloud, addressing the needs and requirements of customers at all levels. He also discusses Business Intelligence and Analytics solutions.
Big Data LDN 2017: Delivering Instant Experience with Redid Enterprise
Date: 15th November 2017
Location: Fast Data Theatre
Time: 15:10 - 15:40
Speaker: Manish Gupta
Organisation: Redis Labs
Similar to Big and fast a quest for relevant and real-time analytics (20)
Data Apps with the Lambda Architecture - with Real Work Examples on Merging B...
Data Apps with the Lambda Architecture - with Real Work Examples on Merging B...
Prassnitha Sampath - Real Time Big Data Analytics with Kafka, Storm & HBase -...
Prassnitha Sampath - Real Time Big Data Analytics with Kafka, Storm & HBase -...
Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, Oath
Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, Oath
AWS APAC Webinar Week - Big Data on AWS. RedShift, EMR, & IOT
AWS APAC Webinar Week - Big Data on AWS. RedShift, EMR, & IOT
UnConference for Georgia Southern Computer Science March 31, 2015
UnConference for Georgia Southern Computer Science March 31, 2015
GraphLab Conference 2014 Keynote - Carlos Guestrin
GraphLab Conference 2014 Keynote - Carlos Guestrin
Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...
Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...
(ARC311) Decoding The Genetic Blueprint Of Life On A Cloud Ecosystem
(ARC311) Decoding The Genetic Blueprint Of Life On A Cloud Ecosystem
Redis and Bloom Filters - Atlanta Java Users Group 9/2014
Redis and Bloom Filters - Atlanta Java Users Group 9/2014
Big Data & Analytics - Innovating at the Speed of Light
Big Data & Analytics - Innovating at the Speed of Light
Scaling APIs: Predict, Prepare for, Overcome the Challenges
Scaling APIs: Predict, Prepare for, Overcome the Challenges
ConFoo 2017: Introduction to performance optimization of .NET web apps
ConFoo 2017: Introduction to performance optimization of .NET web apps
Big Data LDN 2017: Delivering Instant Experience with Redid Enterprise
Big Data LDN 2017: Delivering Instant Experience with Redid Enterprise
More from Natalino Busa
Data Production Pipelines: Legacy, practices, and innovation
Modern engineering requires machine learning engineers, who are needed to monitor and implement ETL and machine learning models in production. Natalino Busa shares technologies, techniques, and blueprints on how to robustly and reliably manage data science and ETL flows from inception to production.
In particular, Natalino explains how to solve one of the most annoying problems in modern data pipelines—migrating and managing legacy ETL—by generating Spark jobs from a textual representation (NLP and SQL). Natalino also demonstrates an open source web UI implemented in React that transforms high-level representations to Spark code and shows how users are able to capture and discover data in the organization by accessing a metadata service. Natalino also introduces the datalabframework, a Jupyter-powered lightweight framework that allows machine learning scientists and engineers to build a robust production ML system only using notebooks.
https://conferences.oreilly.com/strata/strata-sg/public/schedule/detail/64481
Data science apps powered by Jupyter Notebooks
Jupyter notebooks are transforming the way we look at computing, coding, and science. But is this the only "data scientist experience" that this technology can provide? In this presentation we will look at how to create interactive web applications for data exploration and machine learning. In the background this code is still powered by the well-understood and well-documented Jupyter Notebooks.
Code on github: https://github.com/natbusa/kernelgateway_demos
7 steps for highly effective deep neural networks
Natalino will provide a fast-paced lecture on recognizing handwritten numbers with ANNs. In the past years much progress has been booked on the recognition of images: multi-layer perceptrons, inception networks, skip and residual nets, lstm, attention networks.
In this talk, Natalino will go through 7 different architectures on how to classify handwritten numbers using keras and tensorflow. For each neural network architecture, he will spend some time on the theory, the coding, and the quality of the results. Code and Fun included!.
original file:
https://drive.google.com/file/d/0BwNrPuGaMi8PbVhUYUVKWUhGRjQ
Data science apps: beyond notebooks
Jupyter notebooks are transforming the way we look at computing, coding and problem solving. But is this the only “data scientist experience” that this technology can provide?
In this webinar, Natalino will sketch how you could use Jupyter to create interactive and compelling data science web applications and provide new ways of data exploration and analysis. In the background, these apps are still powered by well understood and documented Jupyter notebooks.
They will present an architecture which is composed of four parts: a jupyter server-only gateway, a Scala/Spark Jupyter kernel, a Spark cluster and a angular/bootstrap web application.
[Ai in finance] AI in regulatory compliance, risk management, and auditing
AI to Improve Regulatory Compliance, Governance & Auditing. How AI identifies and prevents risks, above and beyond traditional methods. Techniques and analytics that protect customers and firms from cyber-attacks and fraud. Using AI to quickly and efficiently provide evidence for auditing requests.
Data in Action
A brief intro about me, my past experience, and a small history of data analytics and data science and the latest trends in data science such as deep learning, graph analysis, distributed computing with spark and streaming computing.
Real-Time Anomaly Detection with Spark MLlib, Akka and Cassandra
We present a solution for streaming anomaly detection, named “Coral”, based on Spark, Akka and Cassandra. In the system presented, we run Spark to run the data analytics pipeline for anomaly detection. By running Spark on the latest events and data, we make sure that the model is always up-to-date and that the amount of false positives is kept low, even under changing trends and conditions. Our machine learning pipeline uses Spark decision tree ensembles and k-means clustering. Once the model is trained by Spark, the model’s parameters are pushed to the Streaming Event Processing Layer, implemented in Akka. The Akka layer will then score 1000s of event per seconds according to the last model provided by Spark. Spark and Akka communicate which each other using Cassandra as a low-latency data store. By doing so, we make sure that every element of this solution is resilient and distributed. Spark performs micro-batches to keep the model up-to-date while Akka detects the new anomalies by using the latest Spark-generated data model. The project is currently hosted on Github. Have a look at : http://coral-streaming.github.io
The evolution of data analytics
In the past decade a number of technologies have revolutionized the way we do analytics in banking. In this talk we would like to summarize this journey from classical statistical offline modeling to the latest real-time streaming predictive analytical techniques.
In particular, we will look at hadoop and how this distributing computing paradigm has evolved with the advent of in-memory computing. We will introduce Spark, an engine for large-scale data processing optimized for in-memory computing.
Finally, we will describe how to make data science actionable and how to overcome some of the limitations of current batch processing with streaming analytics.
Hadoop + Cassandra: Fast queries on data lakes, and wikipedia search tutorial.
Today’s services rely on massive amount of data to be processed, but require at the same time to be fast and responsive. Building fast services on big data batch- oriented frameworks is definitely a challenge. At ING, we have worked on a stack that can alleviate this problem. Namely, we materialize data model by map-reducing Hadoop queries from Hive to Cassandra. Instead of sinking the results back to hdfs, we propagate the results into Cassandra key-values tables. Those Cassandra tables are finally exposed via a http API front-end service.
Awesome Banking API's
How do you combine comprehensive analysis running on large amount of data with the demand for responsiveness of today's api services?
This talk illustrates one of recipes that we currently use at ING to tackle this problem. Our analytical stack combines machine learning algorithms running on hadoop cluster and api services executed by an akka cluster.
Cassandra is used as a 'latency adapter' between the fast and the slow path. Our api services are executed by the akka/spray layer. Those services consume both live data sources as well as intermediate results as promoted by the hadoop layer via cassandra. This approach allows us to provide internal api services which are both complete and responsive.
Yo. big data. understanding data science in the era of big data.
We talk a lot these days about data science, and how it will pave our paths with beautiful insights and unexpected new relations and connections in our given datasets, and even across datasets.
But how to maintain the "Science" part in "Data Science"? After some time working in this field I appreciate more and more the critical thinking which has characterized the progress in science.
Hypothesis, facts, prove and/or disprove the thesis. This is how science has progressed in the past centuries. This method has been formalized by Popper and categorize as non-science all disciplines where the statements cannot be falsified. In other words, if a statement cannot be disproved, we cannot talk of science, since there is no mechanism to left to verify the solution or to prove it wrong.
When that happens the argument can still be accepted, but not scientifically accepted. Ways of accepting or refuting a non falsifiable statement are for instance based on aesthetic, authority or pragmatic or philosophical considerations. All valid but not scientific. This applies for instance to statements in the disciplines of politics, teology, ethics, etc.
Science has definitely progressed since then. For instance, Bayesian networks and statistical inductions are currently part of the arsenal of the (data) scientist weapons. But, no matter how the baseline is set, critical thinking and a rigorous method are definitely helpful in understanding the results produced by science in particular when this is based on large amount of data and computational in nature, rather than formula/model driven.
Data Science has currently many different connotations. On one side it praises the "artistry", the genius of laying out connections between disciplines and concepts. This is a truly great aspect of scientists and creativity is definitely very welcome in all data science profiles.
With the fun of creating new insights and new data golden eggs, a data scientist has to put up with those annoying criteria of reproducibility, falsifiability and peer reviewing. Sometimes these elements are postponed or left behind in name of the artistry. Granted, it's just hard to find metrics and baselines in order to compare models and data science solutions. But the scientific method has proven to be solid over the centuries and has proven to allow factual scientific discussion between scientists and a to allow selection between models based on objective agreed criteria.
Big Data and APIs - a recon tour on how to successfully do Big Data analytics
This document discusses big data and APIs. It provides examples of the large amount of data generated daily by Facebook and WhatsApp. It then discusses using MapReduce to parallelize processing of big data across distributed systems. It describes challenges like dealing with large sparse matrices and context in big data. It proposes using machine learning and time series analysis to help process big data. Finally, it discusses distributed computing paradigms like Hadoop and architectures used to manage complex, distributed big data systems at large scale.
Strata 2014: Data science and big data trending topics
The hot and the new at strata 2014. An overview about big data and data science are revolutionising the world around us. The latest tools, techniques, insight, and hands on experience from the industry leaders.
Streaming computing: architectures, and tchnologies
Some loosen thoughts about the latest buzzwords, streaming computing, realtime processing, and in memory computing.
Big data landscape
An overview about several technologies which contribute to the landscape of Big Data.
An intro about the technology challenges of Big Data, follow by key open-source components which help out in dealing with various big data aspects such as OLAP, Real-Time Online
Analytics, Machine Learning on Map-Reduce. I conclude with an enumeration of the key areas where those technologies are most likely unleashing new opportunity for various businesses.
More from Natalino Busa (15)
Data Production Pipelines: Legacy, practices, and innovation
Data Production Pipelines: Legacy, practices, and innovation
[Ai in finance] AI in regulatory compliance, risk management, and auditing
[Ai in finance] AI in regulatory compliance, risk management, and auditing
Real-Time Anomaly Detection with Spark MLlib, Akka and Cassandra
Real-Time Anomaly Detection with Spark MLlib, Akka and Cassandra
Hadoop + Cassandra: Fast queries on data lakes, and wikipedia search tutorial.
Hadoop + Cassandra: Fast queries on data lakes, and wikipedia search tutorial.
Yo. big data. understanding data science in the era of big data.
Yo. big data. understanding data science in the era of big data.
Big Data and APIs - a recon tour on how to successfully do Big Data analytics
Big Data and APIs - a recon tour on how to successfully do Big Data analytics
Strata 2014: Data science and big data trending topics
Strata 2014: Data science and big data trending topics
Streaming computing: architectures, and tchnologies
Streaming computing: architectures, and tchnologies
Recently uploaded
2024 case study-SHEIN Analysis-Final.pdf
A brief analysis of SHEIN's digital transformation.
SHEIN’s business model:
1. D2C cross-border ecommerce: SHEIN integrate the manufactures from Guanzhou to make clothes and deliver direct to customers.
2. Digital marketing: Data driven online marketing for user acquisition.
3. Digital transforming vendor chain: the most core of the revolution to shorten the innovation and lead time.
4. Outstanding user experience: International delivery in high efficiency
Leverage four parts of the user satisfaction process and integrate related resource and information flow, which making SHEIN an international leading D2C ecommerce company.
• Keeping utilizing data in all process is another core capability. From the page click, sales metrics, fabric sourcing to manufacturing time, all data is integrated for decision making, leading an upward customer preference and much efficient business decision making process.
Social Samosa's #30Under30 Guidebook.pdf
With nominations for Social Samosa's #30Under30 now open, here's a guidebook to help you discover every detail of our flagship initiative.
Snapshot of Consumer Behaviors of May 2024-EOLiSurvey (EN).pdf
PARTⅠ: Intension of cooking or dining out under the influence of food safety issues
PARTⅡ: New consuming behavior following up
How to Maximize Sales Using Social Commerce
Discover the best practices and strategies on using social commerce to grow your business and drive sales.
NIMA2024 | AI en de 50% afname van je traffic. Wat nu! | Folkje Berends en Sj...
We nemen je mee in de grote veranderingen in search met de komst van AI, hoe je je website hier op inricht en je data AI ready maakt.
01 Field+Guide+to+Human-Centered+Design_IDEOorg_English GUIA COMPLETA DETALLA...
descripcion detallada de la metodologia HCD
Boost Your Instagram Views Instantly Proven Free Strategies.
Supercars use advanced materials and tech for top-speed performance. Join Performance Car Exclusive to experience driving excellence.
https://instblast.com/instagram/free-instagram-views
Evaluating the Effectiveness of Women-Focused Marketing
Women centric marketing is a vital part in reaching one of the most influential groups of consumers. Here is a guide to know and measure the impact of women-centric marketing efforts-
一比一原版(TMU毕业证)多伦多都会大学毕业证如何办理
TMU毕业证假文凭【微信95270640】（多伦多都会大学毕业证成绩单本科学历）Q微信95270640(补办TMU学位文凭证书)多伦多都会大学留信网学历认证怎么办理多伦多都会大学毕业证成绩单精仿本科学位证书硕士文凭证书认证Seneca College diplomaoffer,Transcript办理硕士学位证书造假多伦多都会大学假文凭学位证书制作TMU本科毕业证书硕士学位证书精仿多伦多都会大学学历认证成绩单修改制作，办理真实认证、留信认证、使馆公证、购买成绩单，购买假文凭，购买假学位证，制造假国外大学文凭、毕业公证、毕业证明书、录取通知书、Offer、在读证明、雅思托福成绩单、假文凭、假毕业证、请假条、国际驾照、网上存档可查！
1:1完美还原海外各大学毕业材料上的工艺：水印阴影底纹钢印LOGO烫金烫银LOGO烫金烫银复合重叠。文字图案浮雕激光镭射紫外荧光温感复印防伪。
可办理以下真实多伦多都会大学存档留学生信息存档认证：
1多伦多都会大学真实留信网认证（网上可查永久存档无风险百分百成功入库）；
2真实教育部认证（留服）等一切高仿或者真实可查认证服务（暂时不可办理）；
3购买英美真实学籍（不用正常就读直接出学历）；
4英美一年硕士保毕业证项目（保录取学校挂名不用正常就读保毕业）
留学本科/硕士毕业证书成绩单制作流程：
1客户提供办理信息：姓名生日专业学位毕业时间等（如信息不确定可以咨询顾问：我们有专业老师帮你查询多伦多都会大学多伦多都会大学毕业证学位证）；
2开始安排制作多伦多都会大学毕业证成绩单电子图；
3多伦多都会大学毕业证成绩单电子版做好以后发送给您确认；
4多伦多都会大学毕业证成绩单电子版您确认信息无误之后安排制作成品；
5多伦多都会大学成品做好拍照或者视频给您确认；
6快递给客户（国内顺丰国外DHLUPS等快读邮寄）
— — — — — — — — — — — 《文凭顾问Q/微：95270640》大街小巷人潮涌动山娃一路张望一路惊叹他发现城里的桥居然层层叠叠扭来扭去桥下没水却有着水一般的车水马龙山娃惊诧于城里的公交车那么大那么美不用买票乖乖地掷下二枚硬币空调享受还能坐着看电视呢屡经辗转山娃终于跟着父亲到家了山娃没想到父亲城里的家会如此寒碜更没料到父亲的城里竟有如此简陋的鬼地方父亲的家在高楼最底屋最下面很矮很黑是很不显眼的地下室父亲的家安在别人脚底下须绕过高楼旁边的垃圾堆下八个台阶才到父亲将
在线办理(worc毕业证书)伍斯特大学毕业证成绩单一模一样
学校原件一模一样【微信：741003700 】《(worc毕业证书)伍斯特大学毕业证成绩单》【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
【主营项目】
一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
Lyrics Video Document for artist Kevin Aguirre
A document explaining the elements that will be used for Kevin's lyrics video
Content Marketing Blueprint For Content Strategy, content creation, content d...
If you’re at all interested in digital
marketing and in making a name for
your brand online, then it is crucial that
you understand how to properly make
use of content marketing. Content
marketing is currently one of the
biggest trends in digital marketing as a
whole and is an area that many website owners and brands are investing in
heavily right now thanks to the impressive returns that they are seeing.
How to Start Affiliate Marketing with ChatGPT- A Step-by-Step Guide (1).pdf
Discover the power of affiliate marketing with ChatGPT! This comprehensive guide takes you through the process of starting and scaling your affiliate marketing business using the latest AI technology. Learn how to leverage ChatGPT to generate content ideas, create engaging articles, and connect with your audience through personalized interactions. From building your strategy and optimizing conversions to analyzing performance and staying updated with industry trends, this eBook provides everything you need to know to succeed in affiliate marketing. Whether you're a beginner looking to start your online business or an experienced marketer wanting to take your efforts to the next level, this guide is your roadmap to success in the world of affiliate marketing.
DTA Promotion - Marketing Specialist Assignment
This document was submitted as part of interview process for Marketing Specialist position at DTA Promotion, an Indonesian company which offers 360 degree marketing services, including ATL and BTL advertising platform.
What is Digital Marketing: A Comprehensive Guide
Digital technologies have transformed marketing. Traditional methods like print and TV ads are giving way to digital strategies, reshaping how brands connect with consumers online. Welcome to the era of digital marketing, where engagement in the digital realm is key. Let's delve into what digital marketing entails in our interconnected world.
Embark on style journeys Indian clothing store denver guide.pptx
Finding the perfect "Indian Clothing Store Denver" is essential for those seeking vibrant, authentic, and culturally rich attire in the heart of Colorado. Denver, a city known for its diverse culture and eclectic fashion scene, offers a variety of options for those in search of traditional and contemporary Indian clothing. Whether you're preparing for a wedding, festival, or cultural event, or simply wish to incorporate the elegance and beauty of Indian fashion into your wardrobe, discovering the right store can make all the difference.
Recently uploaded (20)
Practical Progress from a Theory by Steven Kingpdf
Practical Progress from a Theory by Steven Kingpdf
Snapshot of Consumer Behaviors of May 2024-EOLiSurvey (EN).pdf
Snapshot of Consumer Behaviors of May 2024-EOLiSurvey (EN).pdf
NIMA2024 | AI en de 50% afname van je traffic. Wat nu! | Folkje Berends en Sj...
NIMA2024 | AI en de 50% afname van je traffic. Wat nu! | Folkje Berends en Sj...
01 Field+Guide+to+Human-Centered+Design_IDEOorg_English GUIA COMPLETA DETALLA...
01 Field+Guide+to+Human-Centered+Design_IDEOorg_English GUIA COMPLETA DETALLA...
Boost Your Instagram Views Instantly Proven Free Strategies.
Boost Your Instagram Views Instantly Proven Free Strategies.
Evaluating the Effectiveness of Women-Focused Marketing
Evaluating the Effectiveness of Women-Focused Marketing
Content Marketing Blueprint For Content Strategy, content creation, content d...
Content Marketing Blueprint For Content Strategy, content creation, content d...
How to Start Affiliate Marketing with ChatGPT- A Step-by-Step Guide (1).pdf
How to Start Affiliate Marketing with ChatGPT- A Step-by-Step Guide (1).pdf
Learn more about affiliate marketing as a beginner
Learn more about affiliate marketing as a beginner
Embark on style journeys Indian clothing store denver guide.pptx
Embark on style journeys Indian clothing store denver guide.pptx
Big and fast a quest for relevant and real-time analytics
- 1. Big & Fast: A quest for relevant and real-time analytics Natalino Busa @natalinobusa
- 2. Parallelism Mathematics Programming Languages Machine Learning Statistics Big Data Algorithms Cloud Computing Natalino Busa @natalinobusa www.natalinobusa.com
- 3. Big and Fast. Methodology Architecture Roles and organization
- 4. Conversion is the ultimate form of permission marketing Permission marketing is about the honour of being heard. How to earn it ? Provide the right suggestions, at the right time. This is what makes data analysis valuable
- 5. When do you really know your customer ? know about last unique: 5 songs? 100 songs? 10’000 songs?
- 6. Old & New stuff. We evolve slowly, our personality, our habits. But events and trends can affect us on a short notice How do you combine old with new?
- 7. The customer’s context Complex on many dimensions: Personal history: amount of transactions ever done Long term Interaction: how the users’ action correlate with others Real time events: Trends and recent events
- 8. The customer’s context context is related to time: slow changing: the defining characteristic of a person fast changing: events which influence our lives, trends Require very different technology solutions !!!
- 9. Challenges millions of billions of Not much time to react window of opportunity sometimes is just a few seconds Load of information to process you want to understand well the user history
- 10. Slow and fast ranking and preference analysis segmentation and clustering short term trending topics rule-based recommendations 10’s Terabytes of Data. This can take hours …. 100’s of events per second. This must be fast ….
- 11. Hadoop: Distributed Data OS Reliable Distributed, Replicated File System Low cost ↓ Cost vs ↑ Performance/Storage Computing Powerhouse All clusters CPU’s working in parallel for running queries
- 12. Scala / Akka / Spray: a WEB API reactive framework Actor A Actor B Actor C msg 1 msg 2 msg 3 msg 4 ● it scales horizontally (can run in cluster mode) ● maximum use of the available cores/memory 1. processing is non-blocking, threads are re-used 2. can parallelize computing power across many actors Very fast: 1000’s messages/sec Very reliable: auto recovery
- 13. Distributed computing: lambda architecture Batch Computing HTTP RESTful API In-Memory Distributed Database In-memory Distributed DB’s Lambda Architecture Batch + Streaming low-latency Web API services Streaming Computing Data Warehouses Messaging Busses
- 14. Distributed computing: some techs Hadoop Cassandra millions of billions of λ= conversions ( lamda )
- 15. All Things Distributed Distributing computing and storage more machines = more storage/computing Open Source software solutions mature enough for pragmatic adopters Near realtime + big data technologies Hadoop, Scala, Akka, Spray, Cassandra
- 16. Science & Engineering Statistics, Data Science Python R Visualization IT Infra Big Data Java Scala SQL Hadoop: Big Data Infrastructure, Data Science on large datasets Big Data and Fast Data requires different profiles to be able to achieve the best results
- 17. Parallelism Mathematics Programming Languages Machine Learning Statistics Big Data Algorithms Cloud Computing Natalino Busa @natalinobusa www.natalinobusa.com Thanks ! Any questions?