The Evolution of Big Data Pipelines at Intuit
•Download as PPT, PDF•
1 like•1,559 views
The document summarizes the evolution of Intuit's big data pipelines over time from disparate and chaotic early stages to their current integrated cloud-based architecture. It describes how Intuit transitioned from siloed data storage to a single cohesive data pipeline using Apache Kafka and real-time processing. It outlines the key components of their current big data pipeline including real-time data collection, processing, profile storage, and monitoring systems and how this pipeline supports use cases like personalization, fraud detection and more.
Report
Share
Report
Share
Recommended
What's new in SQL on Hadoop and Beyond
Presto is an open source distributed SQL query engine that allows interactive analysis of data across multiple data stores. At Facebook, Presto is used for ad-hoc queries of their Hadoop data warehouse, which processes trillions of rows and scans petabytes of data daily. Presto's low latency also makes it suitable for powering analytics in user-facing products. New features of Presto include improved SQL support, performance optimizations, and connectors to additional data sources like Redis and MongoDB.
Introduction to Apache Apex by Thomas Weise
https://www.bigdataspain.org/2016/program/thu-introduction-apache-apex.html
https://www.youtube.com/watch?v=93mWU2k8AaU&index=41&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik&t=199s
Active Learning for Fraud Prevention
This document discusses using active learning for fraud prevention at PayPal. It introduces fraud prevention techniques at PayPal, including machine learning models at the transaction, account, and network levels. It then describes an active learning framework that uses deep learning and gradient boosted trees models along with a query by committee strategy. The experiments show that active learning is able to improve the area under the ROC curve performance of these models while significantly reducing labeling costs compared to random sampling for training data.
Lambda-less Stream Processing @Scale in LinkedIn
Lambda-less Stream Processing @Scale in LinkedIn
The document discusses challenges with stream processing including data accuracy and reprocessing. It proposes a "lambda-less" approach using windowed computations and handling late and out-of-order events to produce eventually correct results. Samza is used in LinkedIn's implementation to store streaming data locally using RocksDB for processing within configurable windows. The approach avoids code duplication compared to traditional lambda architectures while still supporting reprocessing through resetting offsets. Challenges remain in merging online and reprocessed results at large scale.
Lego-like building blocks of Storm and Spark Streaming Pipelines
This document discusses an approach to building streaming analytics applications using reusable building blocks like Lego blocks. It presents a centralized approach where common streaming patterns like ingestion, filtering, classification, enrichment etc. are defined as reusable pipelines. These pre-built pipelines can then be connected together to rapidly build streaming analytics apps for various use cases across industries. The document demonstrates how different streaming engines can be used within pipelines and how routing between pipelines can be configured dynamically for A/B testing and model optimization. This approach aims to provide a unified visual platform for collaborative development of efficient streaming analytics solutions at scale.
Next Gen Big Data Analytics with Apache Apex
Next Gen Big Data Analytics with Apache Apex discusses Apache Apex, an open source stream processing framework. It provides an overview of Apache Apex's capabilities for processing continuous, real-time data streams at scale. Specifically, it describes how Apache Apex allows for in-memory, distributed stream processing using a programming model of operators in a directed acyclic graph. It also covers Apache Apex's features for fault tolerance, dynamic scaling, and integration with Hadoop and YARN.
Apache Flink for IoT: How Event-Time Processing Enables Easy and Accurate Ana...
This document discusses Apache Flink for IoT event-time stream processing. It begins by introducing streaming architectures and Flink. It then discusses how IoT data has important properties like continuous data production and event timestamps that require event-time based processing. Examples are provided of companies like King and Bouygues Telecom using Flink for billions of events per day with challenges like out-of-order data and flexible windowing. Event-time processing in Flink is able to handle these challenges through features like watermarks.
The Stream is the Database - Revolutionizing Healthcare Data Architecture
The Stream is the Database - Revolutionizing Healthcare Data ArchitectureDataWorks Summit/Hadoop Summit
This document discusses using event streams as the system of record for data, rather than traditional databases. It argues that streams can serve as the single source of truth for data, providing benefits like data lineage, auditing, and integrity. It also describes how healthcare company Liaison uses a streaming platform from MapR to power their data integration platform, gaining the advantages of streams while meeting various compliance requirements.Recommended
What's new in SQL on Hadoop and Beyond
Presto is an open source distributed SQL query engine that allows interactive analysis of data across multiple data stores. At Facebook, Presto is used for ad-hoc queries of their Hadoop data warehouse, which processes trillions of rows and scans petabytes of data daily. Presto's low latency also makes it suitable for powering analytics in user-facing products. New features of Presto include improved SQL support, performance optimizations, and connectors to additional data sources like Redis and MongoDB.
Introduction to Apache Apex by Thomas Weise
https://www.bigdataspain.org/2016/program/thu-introduction-apache-apex.html
https://www.youtube.com/watch?v=93mWU2k8AaU&index=41&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik&t=199s
Active Learning for Fraud Prevention
This document discusses using active learning for fraud prevention at PayPal. It introduces fraud prevention techniques at PayPal, including machine learning models at the transaction, account, and network levels. It then describes an active learning framework that uses deep learning and gradient boosted trees models along with a query by committee strategy. The experiments show that active learning is able to improve the area under the ROC curve performance of these models while significantly reducing labeling costs compared to random sampling for training data.
Lambda-less Stream Processing @Scale in LinkedIn
Lambda-less Stream Processing @Scale in LinkedIn
The document discusses challenges with stream processing including data accuracy and reprocessing. It proposes a "lambda-less" approach using windowed computations and handling late and out-of-order events to produce eventually correct results. Samza is used in LinkedIn's implementation to store streaming data locally using RocksDB for processing within configurable windows. The approach avoids code duplication compared to traditional lambda architectures while still supporting reprocessing through resetting offsets. Challenges remain in merging online and reprocessed results at large scale.
Lego-like building blocks of Storm and Spark Streaming Pipelines
This document discusses an approach to building streaming analytics applications using reusable building blocks like Lego blocks. It presents a centralized approach where common streaming patterns like ingestion, filtering, classification, enrichment etc. are defined as reusable pipelines. These pre-built pipelines can then be connected together to rapidly build streaming analytics apps for various use cases across industries. The document demonstrates how different streaming engines can be used within pipelines and how routing between pipelines can be configured dynamically for A/B testing and model optimization. This approach aims to provide a unified visual platform for collaborative development of efficient streaming analytics solutions at scale.
Next Gen Big Data Analytics with Apache Apex
Next Gen Big Data Analytics with Apache Apex discusses Apache Apex, an open source stream processing framework. It provides an overview of Apache Apex's capabilities for processing continuous, real-time data streams at scale. Specifically, it describes how Apache Apex allows for in-memory, distributed stream processing using a programming model of operators in a directed acyclic graph. It also covers Apache Apex's features for fault tolerance, dynamic scaling, and integration with Hadoop and YARN.
Apache Flink for IoT: How Event-Time Processing Enables Easy and Accurate Ana...
This document discusses Apache Flink for IoT event-time stream processing. It begins by introducing streaming architectures and Flink. It then discusses how IoT data has important properties like continuous data production and event timestamps that require event-time based processing. Examples are provided of companies like King and Bouygues Telecom using Flink for billions of events per day with challenges like out-of-order data and flexible windowing. Event-time processing in Flink is able to handle these challenges through features like watermarks.
The Stream is the Database - Revolutionizing Healthcare Data Architecture
The Stream is the Database - Revolutionizing Healthcare Data ArchitectureDataWorks Summit/Hadoop Summit
This document discusses using event streams as the system of record for data, rather than traditional databases. It argues that streams can serve as the single source of truth for data, providing benefits like data lineage, auditing, and integrity. It also describes how healthcare company Liaison uses a streaming platform from MapR to power their data integration platform, gaining the advantages of streams while meeting various compliance requirements.Make streaming processing towards ANSI SQL
SQL is the most widely used language for data processing. It allows users to concisely and easily declare their business logic. Data analysts usually do not have complex software programing backgrounds, but they can program SQL and use it on a regular basis to analyze data and power the business decisions. Apache Flink is one of streaming engines that supports SQL. Besides Flink, some other stream processing frameworks, like Kafka and Spark structured streaming, have SQL-like DSL, but they do not have the same semantics as Flink. Flink’s SQL implementation follows ANSI SQL standard while others do not.
In this talk, we will present why following ANSI SQL standard is essential characteristic of Flink SQL and how we achieved this. The core business of Alibaba is now fully driven by the data processing engine: Blink, a project based on Flink with Alibaba’s improvements. About 90% of blink jobs are written by Flink SQL. We will show the use cases and the experience of running large scale Flink SQL jobs at Alibaba in the talk.
Speakers
Shaoxuan Wang, Senior Engineering Manager, Alibaba
Xiaowei Jiang, Senior Director, Alibaba
Accelerating Data Warehouse Modernization
Modern data warehouses need to be modernized to handle big data, integrate multiple data silos, reduce costs, and reduce time to market. A modern data warehouse blueprint includes a data lake to land and ingest structured, unstructured, external, social, machine, and streaming data alongside a traditional data warehouse. Key challenges for modernization include making data discoverable and usable for business users, rethinking ETL to allow for data blending, and enabling self-service BI over Hadoop. Common tactics for modernization include using a data lake as a landing zone, offloading infrequently accessed data to Hadoop, and exploring data in Hadoop to discover new insights.
Building a Graph Database in Neo4j with Spark & Spark SQL to gain new insight...
Building a Graph Database in Neo4j with Spark & Spark SQL to gain new insight...DataWorks Summit/Hadoop Summit
This document discusses how TiVo used a graph database and Spark to gain insights from their log data. TiVo collects log data from their set top boxes but faced challenges analyzing it to optimize the user experience. Their solution was to build a graph of user sessions in Neo4j connected by edges of user actions. This allowed them to analyze paths users take and understand how users discover and access content. They demonstrated analyzing the most popular paths and apps. The graph provided advantages over SQL for understanding interconnected user behavior.Reliable and Scalable Data Ingestion at Airbnb
The document discusses reliable and scalable data ingestion at Airbnb. It describes the challenges they previously faced with unreliable and low quality data. It then outlines the five phases taken to rebuild their data ingestion system to be reliable: 1) auditing each component, 2) auditing the end-to-end system, 3) enforcing schemas, 4) implementing anomaly detection, and 5) building a real-time ingestion pipeline. The new system is able to ingest over 5 billion events per day with less than 100 events lost.
RUNNING A PETASCALE DATA SYSTEM: GOOD, BAD, AND UGLY CHOICES by Alexey Kharlamov
https://www.bigdataspain.org/2016/program/thu-running-petascale-data-system-good-bad-ugly-choices.html
https://www.youtube.com/watch?v=gMrFSwT_O-g&t=18s&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik&index=14
Pivotal Real Time Data Stream Analytics
This document discusses using Pivotal's Big Data Suite to build a real-time analytics solution for processing taxi trip data streams. It presents an architecture that uses Spring XD for data ingestion, Spark Streaming for in-memory analytics on 10-second windows, Gemfire for fast data retrieval, and Pivotal HD for long-term storage. The solution demonstrates filtering inconsistent data, finding top traffic areas, and available taxis in real-time. The document highlights how the Big Data Suite provides a complete toolset for data-driven enterprises through its optimized Hadoop distribution, in-memory processing, stream processing, and low-latency data stores.
Make Streaming Analytics work for you: The Devil is in the Details
This document discusses building a streaming analytics platform and its requirements and capabilities. It covers 5 key attributes of a streaming platform including ingest, process, analyze, visualize and respond. It discusses 6 focus areas for building a streaming platform including common abstraction layer, latency, lambda architecture, scale-out, rapid application development and data visualization. It provides examples of capabilities including continuous processing, event sources, storage, filtering, transformations, correlations and pattern detection. It uses a cybersecurity use case to demonstrate how streaming analytics can be applied for threat protection through exposing, ingesting, prioritizing, analyzing and remediating steps. It also discusses core data platform requirements and interesting problems around offline and local processing for Neustar.
Apache Eagle: Secure Hadoop in Real Time
Apache Eagle is a distributed real-time monitoring and alerting engine for Hadoop that was created by eBay and later open sourced as an Apache Incubator project. It provides security for Hadoop systems by instantly identifying access to sensitive data, recognizing attacks/malicious activity, and blocking access in real time through complex policy definitions and stream processing. Eagle was designed to handle the huge volume of metrics and logs generated by large-scale Hadoop deployments through its distributed architecture and use of technologies like Apache Storm and Kafka.
Solving Performance Problems on Hadoop
My presentation slides from Hadoop Summit, San Jose, June 28, 2016. See live video at http://www.makedatauseful.com/vid-solving-performance-problems-hadoop/ and follow along for context.
Moving analytic workloads into production - specific technical challenges and best practices for engineering SQL in Hadoop solutions. Highlighting the next generation engineering approaches to the secret sauce we have implemented in the Actian VectorH database.
Optimizing industrial operations using the big data ecosystem
GE Digital is undertaking a journey to optimize the reliability, availability, and efficiency of assets in the industrial sector and converge IT and OT. To do so, GE Digital is building cloud-based products that enable customers to analyze the asset data, detect anomalies, and provide recommendations for operating plants efficiently while increasing productivity. In a energy sector such as oil and gas, power, or renewables, a single plant comprises multiple complex assets, such as steam turbines, gas turbines, and compressors, to generate power. Each system contains various sensors to detect the operating conditions of the assets, generating large volumes of variety of data. A highly scalable distributed environment is required to analyze such a large volume of data and provide operating insights in near real time.
In this session I will share the challenges encountered when analyzing the large volumes of data, in-stream data analysis and how we standardized the industrial data based on data frames, and performance tuning.
Building a Scalable Data Science Platform with R
This document discusses building a scalable data science platform with R. It describes R as a popular statistical programming language with over 2.5 million users. It notes that while R is widely used, its open source nature means it lacks enterprise capabilities for large-scale use. The document then introduces Microsoft R Server as a way to bring enterprise capabilities like scalability, efficiency, and support to R in order to make it suitable for production use on big data problems. It provides examples of using R Server with Hadoop and HDInsight on the Azure cloud to operationalize advanced analytics workflows from data cleaning and modeling to deployment as web services at scale.
Real-time Distributed Stream Processing @ Scale
This document summarizes a presentation given by Jerome Boulon, CEO of CaliStream.com, about their data streaming platform. CaliStream provides a schema-less data processing pipeline to stream large volumes of event data from applications to Hadoop/Hive easily and without prior Hadoop knowledge. The presentation covered challenges with traditional data pipelines, how CaliStream addresses these through its SaaS offering, examples of real-time stream analysis using Apache Samza, and new features for Samza.
The Future of Hadoop by Arun Murthy, PMC Apache Hadoop & Cofounder Hortonworks
Arun Murthy will be discussing the future of Hadoop and the next steps in what the big data world would start to look like in the future. With the advent of tools like Spark and Flink and containerization of apps using Docker, there is a lot of momentum currently in this space. Arun will share his thoughts and ideas on what the future holds for us.
Bio:-
Arun C. Murthy
Arun is a Apache Hadoop PMC member and has been a full time contributor to the project since the inception in 2006. He is also the lead of the MapReduce project and has focused on building NextGen MapReduce (YARN). Prior to co-founding Hortonworks, Arun was responsible for all MapReduce code and configuration deployed across the 42,000+ servers at Yahoo!. In essence, he was responsible for running Apache Hadoop’s MapReduce as a service for Yahoo!. Also, he jointly holds the current world sorting record using Apache Hadoop. Follow Arun on Twitter: @acmurthy.
Spark Streaming and IoT by Mike Freedman
This document discusses using Spark Streaming for IoT applications and the challenges involved. It notes that while Spark simplifies programming across different processing intervals from batch to stream, programming models alone are not sufficient as IoT data streams can have varying rates and delays. It proposes a unified data infrastructure with abstractions like data series that support joining real-time and historical data while handling delays transparently. It also suggests approaches for Spark Streaming to better support processing many independent low-volume IoT streams concurrently and improving resource utilization for such applications. Finally, it introduces the Device-Model-Infra framework for addressing these IoT analytics challenges through combined programming models and data abstractions.
"Who Moved my Data? - Why tracking changes and sources of data is critical to...
Speaker: Russ Savage, from Cask
Big Data Applications Meetup, 09/14/2016
Palo Alto, CA
More info here: http://www.meetup.com/BigDataApps/
Link to talk: https://youtu.be/4j78g3WvC4Y
About the talk:
As data lake sizes grow, and more users begin exploring and including that data in their everyday analysis, keeping track of the sources for data becomes critical. Understanding how a dataset was generated and who is using it allows users and companies to ensure their analysis is leveraging the most accurate and up to date information. In this talk, we will explore the different techniques available to keep track of your data in your data lake and demonstrate how we at Cask approached and attempted to mitigate this issue.
From Batch to Streaming ET(L) with Apache Apex
Stream data processing is increasingly required to support business needs for faster actionable insight with growing volume of information from more sources. Apache Apex is a true stream processing framework for low-latency, high-throughput and reliable processing of complex analytics pipelines on clusters. Apex is designed for quick time-to-production, and is used in production by large companies for real-time and batch processing at scale.
This session will use an Apex production use case to walk through the incremental transition from a batch pipeline with hours of latency to an end-to-end streaming architecture with billions of events per day which are processed to deliver real-time analytical reports. The example is representative for many similar extract-transform-load (ETL) use cases with other data sets that can use a common library of building blocks. The transform (or analytics) piece of such pipelines varies in complexity and often involves business logic specific, custom components.
Topics include:
* Pipeline functionality from event source through queryable state for real-time insights.
* API for application development and development process.
* Library of building blocks including connectors for sources and sinks such as Kafka, JMS, Cassandra, HBase, JDBC and how they enable end-to-end exactly-once results.
* Stateful processing with event time windowing.
* Fault tolerance with exactly-once result semantics, checkpointing, incremental recovery
* Scalability and low-latency, high-throughput processing with advanced engine features for auto-scaling, dynamic changes, compute locality.
* Who is using Apex in production, and roadmap.
Following the session attendees will have a high level understanding of Apex and how it can be applied to use cases at their own organizations.
Debunking Common Myths in Stream Processing
Stream Processing is emerging as a popular paradigm for data processing architectures, because it handles the continuous nature of most data and computation and gets rid of artificial boundaries and delays. In this talk, we are going to look at some of the most common misconceptions about stream processing and debunk them.
- Myth 1: Streaming is approximate and exactly-once is not possible.
- Myth 2: Streaming is for real-time only.
- Myth 4: Streaming is harder to learn than Batch Processing.
- Myth 3: You need to choose between latency and throughput.
We will look at these and other myths and debunk them at the example of Apache Flink. We will discuss Apache Flink's approach to high performance stream processing with state, strong consistency, low latency, and sophisticated handling of time. With such building blocks, Apache Flink can handle classes of problems previously considered out of reach for stream processing. We also take a sneak preview at the next steps for Flink.
Apache Flink: Real-World Use Cases for Streaming Analytics
This face to face talk about Apache Flink in Sao Paulo, Brazil is the first event of its kind in Latin America! It explains how Apache Flink 1.0 announced on March 8th, 2016 by the Apache Software Foundation (link), marks a new era of Big Data analytics and in particular Real-Time streaming analytics. The talk maps Flink's capabilities to real-world use cases that span multiples verticals such as: Financial Services, Healthcare, Advertisement, Oil and Gas, Retail and Telecommunications.
In this talk, you learn more about:
1. What is Apache Flink Stack?
2. Batch vs. Streaming Analytics
3. Key Differentiators of Apache Flink for Streaming Analytics
4. Real-World Use Cases with Flink for Streaming Analytics
5. Who is using Flink?
6. Where do you go from here?
Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka
At NMC (Nielsen Marketing Cloud) we provide our customers (marketers and publishers) real-time analytics tools to profile their target audiences.
To achieve that, we need to ingest billions of events per day into our big data stores, and we need to do it in a scalable yet cost-efficient manner.
In this session, we will discuss how we continuously transform our data infrastructure to support these goals.
Specifically, we will review how we went from CSV files and standalone Java applications all the way to multiple Kafka and Spark clusters, performing a mixture of Streaming and Batch ETLs, and supporting 10x data growth.
We will share our experience as early-adopters of Spark Streaming and Spark Structured Streaming, and how we overcame technical barriers (and there were plenty...).
We will present a rather unique solution of using Kafka to imitate streaming over our Data Lake, while significantly reducing our cloud services' costs.
Topics include :
* Kafka and Spark Streaming for stateless and stateful use-cases
* Spark Structured Streaming as a possible alternative
* Combining Spark Streaming with batch ETLs
* "Streaming" over Data Lake using Kafka
Data Ingest Self Service and Management using Nifi and Kafka
We’re feeling the growing pains of maintaining a large data platform. Last year we went from 50 to 150 unique data feeds by adding them all by hand. In this talk we will share the best practices developed to handle our 300% increase in feeds through self service. Having self-service capabilities will increase your teams velocity and decrease your time to value and insight.
* Self service data feed design and ingest
* configuration management
* automatic debugging
* light weight data governance
Timeline service V2 at the Hadoop Summit SJ 2016
This document summarizes the new YARN Timeline Service version 2, which was developed to address scalability, reliability, and usability challenges in version 1. Key highlights of version 2 include a distributed collector architecture for scalable and fault-tolerant writing of timeline data, an enhanced data model with first-class configuration and metrics, and metrics aggregation. It stores data in HBase for scalability and provides a richer REST API for querying. Milestone goals include integration with more frameworks and production readiness.
The Columnar Era: Leveraging Parquet, Arrow and Kudu for High-Performance Ana...
The Columnar Era: Leveraging Parquet, Arrow and Kudu for High-Performance Ana...DataWorks Summit/Hadoop Summit
1) Columnar formats like Parquet, Kudu and Arrow provide more efficient data storage and querying by organizing data by column rather than row.
2) Parquet provides an immutable columnar format well-suited for storage, while Kudu allows for mutable updates but is optimized for scans. Arrow provides an in-memory columnar format focused on CPU efficiency.
3) By establishing common in-memory and on-disk columnar standards, Arrow and Parquet enable more efficient data sharing and querying across systems without serialization overhead.More Related Content
What's hot
Make streaming processing towards ANSI SQL
SQL is the most widely used language for data processing. It allows users to concisely and easily declare their business logic. Data analysts usually do not have complex software programing backgrounds, but they can program SQL and use it on a regular basis to analyze data and power the business decisions. Apache Flink is one of streaming engines that supports SQL. Besides Flink, some other stream processing frameworks, like Kafka and Spark structured streaming, have SQL-like DSL, but they do not have the same semantics as Flink. Flink’s SQL implementation follows ANSI SQL standard while others do not.
In this talk, we will present why following ANSI SQL standard is essential characteristic of Flink SQL and how we achieved this. The core business of Alibaba is now fully driven by the data processing engine: Blink, a project based on Flink with Alibaba’s improvements. About 90% of blink jobs are written by Flink SQL. We will show the use cases and the experience of running large scale Flink SQL jobs at Alibaba in the talk.
Speakers
Shaoxuan Wang, Senior Engineering Manager, Alibaba
Xiaowei Jiang, Senior Director, Alibaba
Accelerating Data Warehouse Modernization
Modern data warehouses need to be modernized to handle big data, integrate multiple data silos, reduce costs, and reduce time to market. A modern data warehouse blueprint includes a data lake to land and ingest structured, unstructured, external, social, machine, and streaming data alongside a traditional data warehouse. Key challenges for modernization include making data discoverable and usable for business users, rethinking ETL to allow for data blending, and enabling self-service BI over Hadoop. Common tactics for modernization include using a data lake as a landing zone, offloading infrequently accessed data to Hadoop, and exploring data in Hadoop to discover new insights.
Building a Graph Database in Neo4j with Spark & Spark SQL to gain new insight...
Building a Graph Database in Neo4j with Spark & Spark SQL to gain new insight...DataWorks Summit/Hadoop Summit
This document discusses how TiVo used a graph database and Spark to gain insights from their log data. TiVo collects log data from their set top boxes but faced challenges analyzing it to optimize the user experience. Their solution was to build a graph of user sessions in Neo4j connected by edges of user actions. This allowed them to analyze paths users take and understand how users discover and access content. They demonstrated analyzing the most popular paths and apps. The graph provided advantages over SQL for understanding interconnected user behavior.Reliable and Scalable Data Ingestion at Airbnb
The document discusses reliable and scalable data ingestion at Airbnb. It describes the challenges they previously faced with unreliable and low quality data. It then outlines the five phases taken to rebuild their data ingestion system to be reliable: 1) auditing each component, 2) auditing the end-to-end system, 3) enforcing schemas, 4) implementing anomaly detection, and 5) building a real-time ingestion pipeline. The new system is able to ingest over 5 billion events per day with less than 100 events lost.
RUNNING A PETASCALE DATA SYSTEM: GOOD, BAD, AND UGLY CHOICES by Alexey Kharlamov
https://www.bigdataspain.org/2016/program/thu-running-petascale-data-system-good-bad-ugly-choices.html
https://www.youtube.com/watch?v=gMrFSwT_O-g&t=18s&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik&index=14
Pivotal Real Time Data Stream Analytics
This document discusses using Pivotal's Big Data Suite to build a real-time analytics solution for processing taxi trip data streams. It presents an architecture that uses Spring XD for data ingestion, Spark Streaming for in-memory analytics on 10-second windows, Gemfire for fast data retrieval, and Pivotal HD for long-term storage. The solution demonstrates filtering inconsistent data, finding top traffic areas, and available taxis in real-time. The document highlights how the Big Data Suite provides a complete toolset for data-driven enterprises through its optimized Hadoop distribution, in-memory processing, stream processing, and low-latency data stores.
Make Streaming Analytics work for you: The Devil is in the Details
This document discusses building a streaming analytics platform and its requirements and capabilities. It covers 5 key attributes of a streaming platform including ingest, process, analyze, visualize and respond. It discusses 6 focus areas for building a streaming platform including common abstraction layer, latency, lambda architecture, scale-out, rapid application development and data visualization. It provides examples of capabilities including continuous processing, event sources, storage, filtering, transformations, correlations and pattern detection. It uses a cybersecurity use case to demonstrate how streaming analytics can be applied for threat protection through exposing, ingesting, prioritizing, analyzing and remediating steps. It also discusses core data platform requirements and interesting problems around offline and local processing for Neustar.
Apache Eagle: Secure Hadoop in Real Time
Apache Eagle is a distributed real-time monitoring and alerting engine for Hadoop that was created by eBay and later open sourced as an Apache Incubator project. It provides security for Hadoop systems by instantly identifying access to sensitive data, recognizing attacks/malicious activity, and blocking access in real time through complex policy definitions and stream processing. Eagle was designed to handle the huge volume of metrics and logs generated by large-scale Hadoop deployments through its distributed architecture and use of technologies like Apache Storm and Kafka.
Solving Performance Problems on Hadoop
My presentation slides from Hadoop Summit, San Jose, June 28, 2016. See live video at http://www.makedatauseful.com/vid-solving-performance-problems-hadoop/ and follow along for context.
Moving analytic workloads into production - specific technical challenges and best practices for engineering SQL in Hadoop solutions. Highlighting the next generation engineering approaches to the secret sauce we have implemented in the Actian VectorH database.
Optimizing industrial operations using the big data ecosystem
GE Digital is undertaking a journey to optimize the reliability, availability, and efficiency of assets in the industrial sector and converge IT and OT. To do so, GE Digital is building cloud-based products that enable customers to analyze the asset data, detect anomalies, and provide recommendations for operating plants efficiently while increasing productivity. In a energy sector such as oil and gas, power, or renewables, a single plant comprises multiple complex assets, such as steam turbines, gas turbines, and compressors, to generate power. Each system contains various sensors to detect the operating conditions of the assets, generating large volumes of variety of data. A highly scalable distributed environment is required to analyze such a large volume of data and provide operating insights in near real time.
In this session I will share the challenges encountered when analyzing the large volumes of data, in-stream data analysis and how we standardized the industrial data based on data frames, and performance tuning.
Building a Scalable Data Science Platform with R
This document discusses building a scalable data science platform with R. It describes R as a popular statistical programming language with over 2.5 million users. It notes that while R is widely used, its open source nature means it lacks enterprise capabilities for large-scale use. The document then introduces Microsoft R Server as a way to bring enterprise capabilities like scalability, efficiency, and support to R in order to make it suitable for production use on big data problems. It provides examples of using R Server with Hadoop and HDInsight on the Azure cloud to operationalize advanced analytics workflows from data cleaning and modeling to deployment as web services at scale.
Real-time Distributed Stream Processing @ Scale
This document summarizes a presentation given by Jerome Boulon, CEO of CaliStream.com, about their data streaming platform. CaliStream provides a schema-less data processing pipeline to stream large volumes of event data from applications to Hadoop/Hive easily and without prior Hadoop knowledge. The presentation covered challenges with traditional data pipelines, how CaliStream addresses these through its SaaS offering, examples of real-time stream analysis using Apache Samza, and new features for Samza.
The Future of Hadoop by Arun Murthy, PMC Apache Hadoop & Cofounder Hortonworks
Arun Murthy will be discussing the future of Hadoop and the next steps in what the big data world would start to look like in the future. With the advent of tools like Spark and Flink and containerization of apps using Docker, there is a lot of momentum currently in this space. Arun will share his thoughts and ideas on what the future holds for us.
Bio:-
Arun C. Murthy
Arun is a Apache Hadoop PMC member and has been a full time contributor to the project since the inception in 2006. He is also the lead of the MapReduce project and has focused on building NextGen MapReduce (YARN). Prior to co-founding Hortonworks, Arun was responsible for all MapReduce code and configuration deployed across the 42,000+ servers at Yahoo!. In essence, he was responsible for running Apache Hadoop’s MapReduce as a service for Yahoo!. Also, he jointly holds the current world sorting record using Apache Hadoop. Follow Arun on Twitter: @acmurthy.
Spark Streaming and IoT by Mike Freedman
This document discusses using Spark Streaming for IoT applications and the challenges involved. It notes that while Spark simplifies programming across different processing intervals from batch to stream, programming models alone are not sufficient as IoT data streams can have varying rates and delays. It proposes a unified data infrastructure with abstractions like data series that support joining real-time and historical data while handling delays transparently. It also suggests approaches for Spark Streaming to better support processing many independent low-volume IoT streams concurrently and improving resource utilization for such applications. Finally, it introduces the Device-Model-Infra framework for addressing these IoT analytics challenges through combined programming models and data abstractions.
"Who Moved my Data? - Why tracking changes and sources of data is critical to...
Speaker: Russ Savage, from Cask
Big Data Applications Meetup, 09/14/2016
Palo Alto, CA
More info here: http://www.meetup.com/BigDataApps/
Link to talk: https://youtu.be/4j78g3WvC4Y
About the talk:
As data lake sizes grow, and more users begin exploring and including that data in their everyday analysis, keeping track of the sources for data becomes critical. Understanding how a dataset was generated and who is using it allows users and companies to ensure their analysis is leveraging the most accurate and up to date information. In this talk, we will explore the different techniques available to keep track of your data in your data lake and demonstrate how we at Cask approached and attempted to mitigate this issue.
From Batch to Streaming ET(L) with Apache Apex
Stream data processing is increasingly required to support business needs for faster actionable insight with growing volume of information from more sources. Apache Apex is a true stream processing framework for low-latency, high-throughput and reliable processing of complex analytics pipelines on clusters. Apex is designed for quick time-to-production, and is used in production by large companies for real-time and batch processing at scale.
This session will use an Apex production use case to walk through the incremental transition from a batch pipeline with hours of latency to an end-to-end streaming architecture with billions of events per day which are processed to deliver real-time analytical reports. The example is representative for many similar extract-transform-load (ETL) use cases with other data sets that can use a common library of building blocks. The transform (or analytics) piece of such pipelines varies in complexity and often involves business logic specific, custom components.
Topics include:
* Pipeline functionality from event source through queryable state for real-time insights.
* API for application development and development process.
* Library of building blocks including connectors for sources and sinks such as Kafka, JMS, Cassandra, HBase, JDBC and how they enable end-to-end exactly-once results.
* Stateful processing with event time windowing.
* Fault tolerance with exactly-once result semantics, checkpointing, incremental recovery
* Scalability and low-latency, high-throughput processing with advanced engine features for auto-scaling, dynamic changes, compute locality.
* Who is using Apex in production, and roadmap.
Following the session attendees will have a high level understanding of Apex and how it can be applied to use cases at their own organizations.
Debunking Common Myths in Stream Processing
Stream Processing is emerging as a popular paradigm for data processing architectures, because it handles the continuous nature of most data and computation and gets rid of artificial boundaries and delays. In this talk, we are going to look at some of the most common misconceptions about stream processing and debunk them.
- Myth 1: Streaming is approximate and exactly-once is not possible.
- Myth 2: Streaming is for real-time only.
- Myth 4: Streaming is harder to learn than Batch Processing.
- Myth 3: You need to choose between latency and throughput.
We will look at these and other myths and debunk them at the example of Apache Flink. We will discuss Apache Flink's approach to high performance stream processing with state, strong consistency, low latency, and sophisticated handling of time. With such building blocks, Apache Flink can handle classes of problems previously considered out of reach for stream processing. We also take a sneak preview at the next steps for Flink.
Apache Flink: Real-World Use Cases for Streaming Analytics
This face to face talk about Apache Flink in Sao Paulo, Brazil is the first event of its kind in Latin America! It explains how Apache Flink 1.0 announced on March 8th, 2016 by the Apache Software Foundation (link), marks a new era of Big Data analytics and in particular Real-Time streaming analytics. The talk maps Flink's capabilities to real-world use cases that span multiples verticals such as: Financial Services, Healthcare, Advertisement, Oil and Gas, Retail and Telecommunications.
In this talk, you learn more about:
1. What is Apache Flink Stack?
2. Batch vs. Streaming Analytics
3. Key Differentiators of Apache Flink for Streaming Analytics
4. Real-World Use Cases with Flink for Streaming Analytics
5. Who is using Flink?
6. Where do you go from here?
Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka
At NMC (Nielsen Marketing Cloud) we provide our customers (marketers and publishers) real-time analytics tools to profile their target audiences.
To achieve that, we need to ingest billions of events per day into our big data stores, and we need to do it in a scalable yet cost-efficient manner.
In this session, we will discuss how we continuously transform our data infrastructure to support these goals.
Specifically, we will review how we went from CSV files and standalone Java applications all the way to multiple Kafka and Spark clusters, performing a mixture of Streaming and Batch ETLs, and supporting 10x data growth.
We will share our experience as early-adopters of Spark Streaming and Spark Structured Streaming, and how we overcame technical barriers (and there were plenty...).
We will present a rather unique solution of using Kafka to imitate streaming over our Data Lake, while significantly reducing our cloud services' costs.
Topics include :
* Kafka and Spark Streaming for stateless and stateful use-cases
* Spark Structured Streaming as a possible alternative
* Combining Spark Streaming with batch ETLs
* "Streaming" over Data Lake using Kafka
Data Ingest Self Service and Management using Nifi and Kafka
We’re feeling the growing pains of maintaining a large data platform. Last year we went from 50 to 150 unique data feeds by adding them all by hand. In this talk we will share the best practices developed to handle our 300% increase in feeds through self service. Having self-service capabilities will increase your teams velocity and decrease your time to value and insight.
* Self service data feed design and ingest
* configuration management
* automatic debugging
* light weight data governance
What's hot (20)
Building a Graph Database in Neo4j with Spark & Spark SQL to gain new insight...
Building a Graph Database in Neo4j with Spark & Spark SQL to gain new insight...
RUNNING A PETASCALE DATA SYSTEM: GOOD, BAD, AND UGLY CHOICES by Alexey Kharlamov
RUNNING A PETASCALE DATA SYSTEM: GOOD, BAD, AND UGLY CHOICES by Alexey Kharlamov
Make Streaming Analytics work for you: The Devil is in the Details
Make Streaming Analytics work for you: The Devil is in the Details
Optimizing industrial operations using the big data ecosystem
Optimizing industrial operations using the big data ecosystem
The Future of Hadoop by Arun Murthy, PMC Apache Hadoop & Cofounder Hortonworks
The Future of Hadoop by Arun Murthy, PMC Apache Hadoop & Cofounder Hortonworks
"Who Moved my Data? - Why tracking changes and sources of data is critical to...
"Who Moved my Data? - Why tracking changes and sources of data is critical to...
Apache Flink: Real-World Use Cases for Streaming Analytics
Apache Flink: Real-World Use Cases for Streaming Analytics
Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka
Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka
Data Ingest Self Service and Management using Nifi and Kafka
Data Ingest Self Service and Management using Nifi and Kafka
Viewers also liked
Timeline service V2 at the Hadoop Summit SJ 2016
This document summarizes the new YARN Timeline Service version 2, which was developed to address scalability, reliability, and usability challenges in version 1. Key highlights of version 2 include a distributed collector architecture for scalable and fault-tolerant writing of timeline data, an enhanced data model with first-class configuration and metrics, and metrics aggregation. It stores data in HBase for scalability and provides a richer REST API for querying. Milestone goals include integration with more frameworks and production readiness.
The Columnar Era: Leveraging Parquet, Arrow and Kudu for High-Performance Ana...
The Columnar Era: Leveraging Parquet, Arrow and Kudu for High-Performance Ana...DataWorks Summit/Hadoop Summit
1) Columnar formats like Parquet, Kudu and Arrow provide more efficient data storage and querying by organizing data by column rather than row.
2) Parquet provides an immutable columnar format well-suited for storage, while Kudu allows for mutable updates but is optimized for scans. Arrow provides an in-memory columnar format focused on CPU efficiency.
3) By establishing common in-memory and on-disk columnar standards, Arrow and Parquet enable more efficient data sharing and querying across systems without serialization overhead.Big Data Ready Enterprise
The document describes Big Data Ready Enterprise (BDRE), an open source product that addresses common challenges in implementing and operating big data solutions at large scale. It provides out-of-the-box features to accelerate implementations using pluggable architecture, community support, and distribution compatibility. The document outlines BDRE's key benefits and capabilities for data ingestion, workflow automation, operational metadata management, and more. It also provides examples of BDRE implementations and screenshots of the product's interface.
The Stream Processor as a Database Apache Flink
Apache Flink enables stream processing on continuously produced data through its DataStream and DataSet APIs. It allows for streaming and batch processing as first class citizens. Flink programs are composed of sources that ingest data, transformations on those data streams, and sinks that output the results. Queryable state in Flink allows for querying the system state without writing to an external database, improving performance over traditional architectures that rely on writing intermediate results to external key-value stores. Flink's use of lightweight snapshots for fault tolerance and its log-based approach to persistence allows queryable state to have high throughput and low latency.
Managing a Multi-Tenant Data Lake
The document discusses managing a multi-tenant data lake at Comcast over time. It began as an experiment in 2013 with 10 nodes and has grown significantly to over 1500 nodes currently. Governance was instituted to manage the diverse user community and workloads. Tools like the Command Center were developed to provide monitoring, alerting and visualization of the large Hadoop environment. SLA management, support processes, and ongoing training are needed to effectively operate the multi-tenant data lake at scale.
Data-Driven Marketing in Plain English
My presentation that I used in my speech during the Middle East Big Data Conference 2015 in Kuwait City on December 3rd.
Bill Loconzolo talks #bigdata at #StrataHadoop 2016
VP of Data Engineering & Analytics, Bill Loconzolo talks about building a scalable, secure data platform at the 2016 Strata Hadoop Conference. If you missed his presentation, check it out. http://intuit.me/22T5Beh
Creating a Data-Positive Culture at Your Organization
This document discusses how to create a data-positive culture within a nonprofit organization. It recommends demonstrating the value of data on an individual level, tying data collection and analysis to the organization's mission, and dispersing data responsibilities throughout the organization. The example of YNPN Boston is provided, which aligns its data on professional development, events, and membership with its mission of engaging young nonprofit professionals to build a stronger nonprofit sector.
Big data cloud architecture
The document discusses using cloud infrastructure to provide fast access to large datasets for customers globally. It describes:
1. Creating an Excel plug-in that provides thousands of data points on demand from global servers to customers anywhere in the world.
2. Moving frequently used large datasets to the cloud to enable single millisecond data retrieval times, compared to keeping everything in an on-premise US data center.
3. Employing techniques like smart caching and pre-sending relevant data to optimize performance, based on analyzing usage logs to predict customer needs.
Real time, streaming advanced analytics, approximations, and recommendations ...
Real time, streaming advanced analytics, approximations, and recommendations ...DataWorks Summit/Hadoop Summit
This document discusses various techniques for building recommendation systems using Apache Spark. It begins with an overview of scaling techniques using parallelism and composability. Various similarity measures are then covered, including Euclidean, cosine, Jaccard, and word embeddings. Recommendation approaches like item-to-item graphs and personalized PageRank are demonstrated. The document also discusses feature engineering, modeling techniques, and evaluating recommendations. Live demos are provided of word similarity, movie recommendations, sentiment analysis and more.HIPAA Compliance in the Cloud
This document discusses HIPAA compliance in the cloud and using Amazon Web Services (AWS) for collaborative medical research. It provides an overview of how AWS services like S3, EC2, DynamoDB, and EMR can be used in a HIPAA-compliant manner. It also discusses challenges of ensuring HIPAA compliance for Hadoop and how tools like encryption, access controls, monitoring, and auditing help address those challenges. The document concludes that security is critical, AWS has tools to make HIPAA compliance easier, and users can move fast while staying safe on AWS.
Launching your advanced analytics program for success in a mature industry
Launching your advanced analytics program for success in a mature industryDataWorks Summit/Hadoop Summit
This document provides recommendations for launching a successful advanced analytics program in a mature industry. It recommends:
1. Starting at the top by gaining executive support and framing analytics as a strategic capability.
2. Finding the right people by hiring data scientists and multi-disciplinary talent, as well as growing internal skills, but prioritizing industry knowledge over tool expertise.
3. Getting organized by establishing processes, connecting different parts of the organization, and promoting internal evangelism for analytics.Hive acid and_2.x new_features
The document discusses new features in Hive 2.0 including Hive LLAP (Live Long And Process) and Hive on ACID (Atomic, Consistent, Isolated, Durable). Hive LLAP introduces an in-memory caching mechanism that provides sub-second query performance for Hive. Hive on ACID allows for transactions on Hive tables including updates, deletes, and streaming ingestion while maintaining consistency and concurrency. The document provides overviews of how both features work and improvements they provide for analytics workloads on Hive.
Big Data Heterogeneous Mixture Learning on Spark
This document provides an overview of NEC's Heterogeneous Mixture Learning (HML) technology and its implementation on Apache Spark. It introduces the speakers and their backgrounds working on distributed computing and machine learning. The agenda discusses HML, applications of HML, the HML algorithm, and benchmark performance evaluations showing HML achieves competitive prediction accuracy compared to other Spark ML algorithms while maintaining good scalability. Distributed HML on Spark aims to enable fast, large-scale machine learning by balancing work across executors and leveraging high-performance matrix libraries.
High-Scale Entity Resolution in Hadoop
eBay maintains hundreds of millions of accounts across its properties that are unstructured and in different formats. Identifying which accounts belong to the same person enables eBay to personalize customer experiences, provide customer service, and fight fraud. MapReduce provides a robust design pattern to simplify high-scale entity resolution through parallelized modular operations, including linking accounts pairwise, identifying connected components through iterative MapReduce jobs, and validating the results.
Adding ACID Transactions, Inserts, Updates, and Deletes in Apache Hive
This document discusses adding ACID transaction support to Hive to allow for updating, deleting, and inserting rows. It describes how HDFS storage is organized using base and delta files. Transactions are managed through a new transaction manager that uses the metastore database. Locking is implemented to control concurrent access. Streaming ingest is supported through a new interface that allows small batches to be written and committed. The goal is to support SQL commands like UPDATE and DELETE while providing scalable reads and writes through compactions.
Quark Virtualization Engine for Analytics
Rajat Venkatesh from Qubole presented on Quark, a virtualization engine for analytics. Quark uses a multi-store architecture to optimize queries using materialized views, predicate injection, and denormalized/sorted tables. It supports multiple SQL and storage engines. The roadmap includes improvements to the cost-based optimizer, support for OLAP cubes, and developing Quark as a service. Coordinates for the Quark GitHub and mailing list were provided.
Hive acid-updates-strata-sjc-feb-2015
Hive 0.14 adds ACID transactional support which allows for inserting, updating, and deleting rows in Hive tables. It uses a new transaction manager and lock manager to provide snapshot isolation across DML statements. Data is stored in HDFS in a layout of base files and transactional delta files which are compacted periodically. This allows Hive to support use cases beyond batch loads such as streaming data ingest and updating dimension tables.
Sparksee overview
This document describes Sparksee, a graph database created by Sparsity Technologies. Sparksee is designed for large, labeled graphs and uses vertical partitioning and bitmaps to store graph data compactly. It provides high performance for queries and analytics on social networks and bibliographic databases. The document outlines Sparksee's architecture, capabilities including efficiency, capacity, and scalability. It also describes the desktop and mobile APIs available for accessing Sparksee and provides examples of common API methods.
GNW01: In-Memory Processing for Databases
This document discusses in-memory execution for databases. It begins with introductions and background on the author. It then discusses how databases can offload data to memory to improve query performance 2-24x by analyzing storage use and access patterns. It covers concepts like how RAM access is now the performance bottleneck and how CPU cache-friendly data structures are needed. It shows examples measuring performance differences when scanning data in memory versus disk. Finally, it discusses future directions like more integrated storage and memory and new data formats optimized for CPU caches.
Viewers also liked (20)
The Columnar Era: Leveraging Parquet, Arrow and Kudu for High-Performance Ana...
The Columnar Era: Leveraging Parquet, Arrow and Kudu for High-Performance Ana...
Bill Loconzolo talks #bigdata at #StrataHadoop 2016
Bill Loconzolo talks #bigdata at #StrataHadoop 2016
Creating a Data-Positive Culture at Your Organization
Creating a Data-Positive Culture at Your Organization
Real time, streaming advanced analytics, approximations, and recommendations ...
Real time, streaming advanced analytics, approximations, and recommendations ...
Launching your advanced analytics program for success in a mature industry
Launching your advanced analytics program for success in a mature industry
Adding ACID Transactions, Inserts, Updates, and Deletes in Apache Hive
Adding ACID Transactions, Inserts, Updates, and Deletes in Apache Hive
Similar to The Evolution of Big Data Pipelines at Intuit
Hadoop Summit 2016 - Evolution of Big Data Pipelines At Intuit
This document summarizes the evolution of Intuit's big data pipelines over time. It discusses how the pipelines started as disparate and moved to a single cohesive pipeline. The stages of evolution included moving to real-time processing, high availability with mirroring, and integrating streaming and batch processing. Key components of the modern pipeline include Apache Kafka for event collection, real-time processing engines, and data storage in Hive. Testing ensures the pipeline meets throughput, latency, and data integrity requirements as the volumes and complexity increase.
Mesoscon 2015
This document discusses Mesos implementation at Bloomberg. It notes that Bloomberg runs one of the largest private networks and was an early adopter of cloud computing and software as a service. It describes how Mesos is used to provide elastic data processing and analytics across Bloomberg's 3000+ developers. Key parts of the Mesos implementation include using Marathon for application deployment, Kafka for processing topologies, and ELK/InfluxDB/Grafana for centralized monitoring. The document also discusses lessons learned around access control, Zookeeper protection, and cleaning up sandbox data.
Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022
There is an increasing need to unleash analytical capabilities directly to the end-users to democratize decision-making. User-Facing Analytics is a new frontier that will shape the products of tomorrow and push the limits of existing technology. It demands a solution that will scale to millions of users to provide fast, real-time insights. In this session, Xiang will talk about his journey to build Apache Pinot to tackle the analytics problem space with the architectural changes and technology inventions made over the past decade. He will also talk about how other big data companies such as LinkedIn, Uber, and Stripe power their user-facing analytical applications.
Digital Transformation Mindset - More Than Just Technology
The document discusses how digital transformation requires an event-driven mindset and streaming platforms. It describes how a streaming platform can act as the central nervous system of a modern digital enterprise by enabling event-centric thinking, serving as the foundation for data integration and storage, and allowing data to drive applications in real-time. The document outlines Confluent's role in developing the open-source Apache Kafka streaming platform and discusses how various companies are using streaming platforms for applications like real-time customer experiences, fraud detection, and rewriting client experiences.
[WSO2Con EU 2018] The Rise of Streaming SQL
This session takes an in-depth look at:
- Trends in stream processing
- How streaming SQL has become a standard
- The advantages of Streaming SQL
- Ease of development with streaming SQL: Graphical and Streaming SQL query editors
- Business value of streaming SQL and its related tools: Domain-specific UIs
- Scalable deployment of streaming SQL: Distributed processing
Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...
https://www.bigdataspain.org/2016/program/thu-big-data-analytics-machine-learning-to-real-time-processing.html
https://www.youtube.com/watch?v=QiQ6LKKvd9k&t=3s&index=19&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik
Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Ser...
This document provides an overview of streaming analytics and compares different streaming analytics frameworks. It begins with real-world use cases in various industries and then defines what a data stream is. The core components of a streaming analytics processing pipeline are described, including ingestion, preprocessing, and real-time and batch processing. Popular open-source frameworks like Apache Storm and AWS Kinesis are highlighted. The document concludes by noting that both streaming analytics frameworks and products are growing significantly to enable real-time analytics on streaming data.
Moving Targets: Harnessing Real-time Value from Data in Motion
The Briefing Room with David Loshin and Datawatch
Live Webcast Feb. 17, 2015
Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=4a053043c45cf0c2f6453dfb8577c72a
Patience may be a virtue, but when it comes to streaming analytics, waiting is no option. Between Big Data and the Internet of Things, businesses are faced with more data and greater complexity than ever before. Traditional information architectures simply cannot support the kind of processing necessary to make use of this fast-moving resource. The modern context requires a shorter path to analytics, one that narrows the gap between governance and discovery
Register for this episode of The Briefing Room to hear veteran Analyst David Loshin as he explains how the prevalence of streaming data is changing business pace and processes. He’ll be briefed by Dan Potter of Datawatch, who will tout his company’s real-time data discovery platform for data in motion. He will show how self-service data preparation can lead to faster insights, ultimately fostering the ability to make precise decisions at the right time.
Visit InsideAnalysis.com for more information.
Analytical Innovation: How to Build the Next Generation Data Platform
There was a time when the Enterprise Data Warehouse (EDW) was the only way to provide a 360-degree analytical view of the business. In recent years many organizations have deployed disparate analytics alternatives to the EDW, including: cloud data warehouses, machine learning frameworks, graph databases, geospatial tools, and other technologies. Often these new deployments have resulted in the creation of analytical silos that are too complex to integrate, seriously limiting global insights and innovation.
Join guest speaker, 451 Research’s Jim Curtis and Pivotal’s Jacque Istok for an interactive discussion about some of the overarching trends affecting the data warehousing market, as well as how to build a next generation data platform to accelerate business innovation. During this webinar you will learn:
- The significance of a multi-cloud, infrastructure-agnostic analytics
- What is working and what isn’t, when it comes to analytics integration
- The importance of seamlessly integrating all your analytics in one platform
- How to innovate faster, taking advantage of open source and agile software
Speakers: James Curtis, Senior Analyst, Data Platforms & Analytics, 451 Research & Jacque Istok, Head of Data, Pivotal
Analytics in Your Enterprise
To view recording of this webinar please use below URL:
http://wso2.com/library/webinars/2016/06/analytics-in-your-enterprise/
Big data spans many fields and brings together technologies like distributed systems, machine learning, statistics and Internet of Things (IoT). It has now become a multi-billion dollar industry with use cases ranging from targeted advertising and fraud detection to product recommendations and market surveys.
Some use cases such as urban planning can be slower (done in batch mode), while others such as the stock market needs results in milliseconds (done is a streaming fashion). Different technologies are used for each case; MapReduce for batch analytics, complex event processing for real-time analytics and machine learning for predictive analytics. Furthermore, the type of analysis ranges from basic statistics to complicated prediction models.
This webinar will discuss the big data landscape including
Concepts, use cases and technologies
Capabilities and applications of the WSO2 analytics platform
WSO2 Data Analytics Server
WSO2 Complex Event Processor
WSO2 Machine Learner
Scaling Your Architecture with Services and Events
This session is a deep dive into the modern best practices around asynchronous decoupling, resilience, and scalability that allow us to implement a large-scale software system from the building blocks of events and services, based on the speaker's experiences implementing such systems at Google, eBay, and other high-performing technology organizations. We will outline the various options for handling event delivery and event ordering in a distributed system. We will cover data and persistence in an event-driven architecture. Finally, we will describe how to combine events, services, and so-called 'serverless' functions into a powerful overall architecture. You will leave with practical suggestions to help you accelerate your development velocity and drive business results.
"Fintech inside of a SaaS powered by 2000+ Microservices", Volodymyr Malyk
The way Architecture is done in BigTech, top-down “rules” and technology choices.
We will talk about freedom in making architecture decisions for Product and Core teams.
We'll go deeper into Fintech solution, built as a crucial component of global SaaS: Microservices, APIs and Queues, Event Sourcing, Feature Toggles, SDLC, CI/CD, DevOps, Monitoring, Analytics and more.
MicroStrategy at Badoo
This document discusses Badoo's use of MicroStrategy for business intelligence and analytics. It describes how MicroStrategy helped Badoo overcome challenges with their previous BI tool by providing dimensional modeling, self-service reports, and weekly releases. It highlights how MicroStrategy enabled data discovery, analysis delivery, and reporting for over 90 users across various teams. The document also provides examples of query optimizations in MicroStrategy that improved performance. Finally, it discusses how MicroStrategy has enabled Badoo to empower users through visual insights, transaction services, command manager automation, and streamlined web deployments.
Extracting Insights from Data at Twitter
Prasad Wagle's talk discussed how Twitter extracts insights from its large volumes of data. Twitter collects hundreds of millions of tweets and interactions per day from over 300 million monthly active users, creating big data challenges around velocity, volume, and variety. Twitter stores this data in hundreds of petabytes across large Hadoop clusters and processes it using batch tools like Hadoop and Spark as well as real-time tools like Heron. Insights are generated through basic analytics like user counts, A/B testing of new features, and custom data science work including machine learning models for recommendations, content filtering, and ad targeting. Systems, programming, and statistical skills are needed to effectively extract value from Twitter's big data.
Pivotal Digital Transformation Forum: Journey to Become a Data-Driven Enterprise
The document discusses Pivotal's Big Data Suite for helping enterprises become data-driven. It outlines challenges in analyzing large amounts of data and the value that can be gained. The suite includes tools for ingesting, processing, storing and analyzing streaming and batch data at scale. It also provides examples of how the suite can be used for applications like financial compliance monitoring and connected cars.
CQRS and Event Sourcing for IoT applications
In the Internet of things, data and commands between things and servers are sent as streams of events, which are often aggregated and processed to provide up to date information to end users. Because of this, CQRS and Event Sourcing patterns are a natural fit for IoT applications. In this presentation we provide an overview of these patterns, how they apply to IoT applications and their benefits. A prototype application of Event Sourcing is then demonstrated using the Sense Tecnic FRED platform based on Node-RED - a data flow programming tool for wiring up the internet of things
SQL Server 2008 R2 StreamInsight
Massive amounts of data are being generated from various sources like cell phones, sensors, web logs etc. This ambient data needs to be processed in real-time to enable scenarios like fraud detection, manufacturing process control, network monitoring etc. SQL Server StreamInsight provides a platform to process data streams with low latency queries, enabling near real-time analytics and action. Key capabilities include filtering, correlating, aggregating events over windows using a LINQ-like declarative query language.
Effective Microservices In a Data-centric World
From a talk at GOTOChicago 2017, these slides discuss the speaker's experiences at Stitch Fix with
* Organizational, Process, and Cultural prerequisites for being successful with Microservices: small teams, TDD / CD, DevOps
* How to handle shared data when your data is split among microservices
* How to handle "joins" across microservices
* How to simulate "transactions" across microservices
Slides link: https://gotochgo.com/3/sessions/79/slides
Video link: https://gotochgo.com/3/sessions/79/video
Presto @ Treasure Data - Presto Meetup Boston 2015
Treasure Data simplifies event analytics for the complex digital
world. Our customers send us 1,000,000 events per second and issue 30,000+ Presto queries everyday to understand their customers better. One of the challenges is designing a cloud database with zero downtime to support a global customer base. We have achieved this goal by developing several open-source technologies; Fluentd and Embulk enable seamless log collection from stream/batch sources, and with MessagePack we can provide an extensible columnar store that accommodates future schema changes. Finally, Presto allows us to serve a wide variety of data processing our customers perform on our service. In this talk, I will present an overview of our system, and how our customers keep using Presto while collecting and extending their data set.
Ronan Corkery, kdb+ developer at Kx Systems: “Kdb+: How Wall Street Tech can ...
This document discusses how Wall Street technology from Kx Systems can speed up data processing for various industries. Kx's time-series database, kdb+, can process and analyze large volumes of real-time and historical data extremely fast with low latency. It is scalable, integrates with other technologies, and provides powerful tools and dashboards. Kx is currently used widely in financial services and is now being adopted in industries like manufacturing, utilities, healthcare, and more.
Similar to The Evolution of Big Data Pipelines at Intuit (20)
Hadoop Summit 2016 - Evolution of Big Data Pipelines At Intuit
Hadoop Summit 2016 - Evolution of Big Data Pipelines At Intuit
Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022
Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022
Digital Transformation Mindset - More Than Just Technology
Digital Transformation Mindset - More Than Just Technology
Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...
Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...
Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Ser...
Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Ser...
Moving Targets: Harnessing Real-time Value from Data in Motion
Moving Targets: Harnessing Real-time Value from Data in Motion
Analytical Innovation: How to Build the Next Generation Data Platform
Analytical Innovation: How to Build the Next Generation Data Platform
Scaling Your Architecture with Services and Events
Scaling Your Architecture with Services and Events
"Fintech inside of a SaaS powered by 2000+ Microservices", Volodymyr Malyk
"Fintech inside of a SaaS powered by 2000+ Microservices", Volodymyr Malyk
Pivotal Digital Transformation Forum: Journey to Become a Data-Driven Enterprise
Pivotal Digital Transformation Forum: Journey to Become a Data-Driven Enterprise
Presto @ Treasure Data - Presto Meetup Boston 2015
Presto @ Treasure Data - Presto Meetup Boston 2015
Ronan Corkery, kdb+ developer at Kx Systems: “Kdb+: How Wall Street Tech can ...
Ronan Corkery, kdb+ developer at Kx Systems: “Kdb+: How Wall Street Tech can ...
More from DataWorks Summit/Hadoop Summit
Running Apache Spark & Apache Zeppelin in Production
This document discusses running Apache Spark and Apache Zeppelin in production. It begins by introducing the author and their background. It then covers security best practices for Spark deployments, including authentication using Kerberos, authorization using Ranger/Sentry, encryption, and audit logging. Different Spark deployment modes like Spark on YARN are explained. The document also discusses optimizing Spark performance by tuning executor size and multi-tenancy. Finally, it covers security features for Apache Zeppelin like authentication, authorization, and credential management.
State of Security: Apache Spark & Apache Zeppelin
This document discusses Spark security and provides an overview of authentication, authorization, encryption, and auditing in Spark. It describes how Spark leverages Kerberos for authentication and uses services like Ranger and Sentry for authorization. It also outlines how communication channels in Spark are encrypted and some common issues to watch out for related to Spark security.
Unleashing the Power of Apache Atlas with Apache Ranger
The document discusses the Virtual Data Connector project which aims to leverage Apache Atlas and Apache Ranger to provide unified metadata and access governance across data sources. Key points include:
- The project aims to address challenges of understanding, governing, and controlling access to distributed data through a centralized metadata catalog and policies.
- Apache Atlas provides a scalable metadata repository while Apache Ranger enables centralized access governance. The project will integrate these using a virtualization layer.
- Enhancements to Atlas and Ranger are proposed to better support the project's goals around a unified open metadata platform and metadata-driven governance.
- An initial minimum viable product will be built this year with the goal of an open, collaborative ecosystem around shared
Enabling Digital Diagnostics with a Data Science Platform
This document discusses using a data science platform to enable digital diagnostics in healthcare. It provides an overview of healthcare data sources and Yale/YNHH's data science platform. It then describes the data science journey process using a clinical laboratory use case as an example. The goal is to use big data and machine learning to improve diagnostic reproducibility, throughput, turnaround time, and accuracy for laboratory testing by developing a machine learning algorithm and real-time data processing pipeline.
Revolutionize Text Mining with Spark and Zeppelin
This document discusses using Apache Spark and MLlib for text mining on big data. It outlines common text mining applications, describes how Spark and MLlib enable scalable machine learning on large datasets, and provides examples of text mining workflows and pipelines that can be built with Spark MLlib algorithms and components like tokenization, feature extraction, and modeling. It also discusses customizing ML pipelines and the Zeppelin notebook platform for collaborative data science work.
Double Your Hadoop Performance with Hortonworks SmartSense
This document compares the performance of Hive and Spark when running the BigBench benchmark. It outlines the structure and use cases of the BigBench benchmark, which aims to cover common Big Data analytical properties. It then describes sequential performance tests of Hive+Tez and Spark on queries from the benchmark using a HDInsight PaaS cluster, finding variations in performance between the systems. Concurrency tests are also run by executing multiple query streams in parallel to analyze throughput.
Hadoop Crash Course
The document discusses modern data applications and architectures. It introduces Apache Hadoop, an open-source software framework for distributed storage and processing of large datasets across clusters of commodity hardware. Hadoop provides massive scalability and easy data access for applications. The document outlines the key components of Hadoop, including its distributed storage, processing framework, and ecosystem of tools for data access, management, analytics and more. It argues that Hadoop enables organizations to innovate with all types and sources of data at lower costs.
Data Science Crash Course
This document provides an overview of data science and machine learning. It discusses what data science and machine learning are, including extracting insights from data and computers learning without being explicitly programmed. It also covers Apache Spark, which is an open source framework for large-scale data processing. Finally, it discusses common machine learning algorithms like regression, classification, clustering, and dimensionality reduction.
Apache Spark Crash Course
This document provides an overview of Apache Spark, including its capabilities and components. Spark is an open-source cluster computing framework that allows distributed processing of large datasets across clusters of machines. It supports various data processing workloads including streaming, SQL, machine learning and graph analytics. The document discusses Spark's APIs like DataFrames and its libraries like Spark SQL, Spark Streaming, MLlib and GraphX. It also provides examples of using Spark for tasks like linear regression modeling.
Dataflow with Apache NiFi
This document provides an overview of Apache NiFi and dataflow. It begins with an introduction to the challenges of moving data effectively within and between systems. It then discusses Apache NiFi's key features for addressing these challenges, including guaranteed delivery, data buffering, prioritized queuing, and data provenance. The document outlines NiFi's architecture and components like repositories and extension points. It also previews a live demo and invites attendees to further discuss Apache NiFi at a Birds of a Feather session.
Schema Registry - Set you Data Free
Many Organizations are currently processing various types of data and in different formats. Most often this data will be in free form, As the consumers of this data growing it’s imperative that this free-flowing data needs to adhere to a schema. It will help data consumers to have an expectation of about the type of data they are getting and also they will be able to avoid immediate impact if the upstream source changes its format. Having a uniform schema representation also gives the Data Pipeline a really easy way to integrate and support various systems that use different data formats.
SchemaRegistry is a central repository for storing, evolving schemas. It provides an API & tooling to help developers and users to register a schema and consume that schema without having any impact if the schema changed. Users can tag different schemas and versions, register for notifications of schema changes with versions etc.
In this talk, we will go through the need for a schema registry and schema evolution and showcase the integration with Apache NiFi, Apache Kafka, Apache Storm.
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...DataWorks Summit/Hadoop Summit
There is increasing need for large-scale recommendation systems. Typical solutions rely on periodically retrained batch algorithms, but for massive amounts of data, training a new model could take hours. This is a problem when the model needs to be more up-to-date. For example, when recommending TV programs while they are being transmitted the model should take into consideration users who watch a program at that time.
The promise of online recommendation systems is fast adaptation to changes, but methods of online machine learning from streams is commonly believed to be more restricted and hence less accurate than batch trained models. Combining batch and online learning could lead to a quickly adapting recommendation system with increased accuracy. However, designing a scalable data system for uniting batch and online recommendation algorithms is a challenging task. In this talk we present our experiences in creating such a recommendation engine with Apache Flink and Apache Spark.Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...DataWorks Summit/Hadoop Summit
DeepLearning is not just a hype - it outperforms state-of-the-art ML algorithms. One by one. In this talk we will show how DeepLearning can be used for detecting anomalies on IoT sensor data streams at high speed using DeepLearning4J on top of different BigData engines like ApacheSpark and ApacheFlink. Key in this talk is the absence of any large training corpus since we are using unsupervised machine learning - a domain current DL research threats step-motherly. As we can see in this demo LSTM networks can learn very complex system behavior - in this case data coming from a physical model simulating bearing vibration data. Once draw back of DeepLearning is that normally a very large labaled training data set is required. This is particularly interesting since we can show how unsupervised machine learning can be used in conjunction with DeepLearning - no labeled data set is necessary. We are able to detect anomalies and predict braking bearings with 10 fold confidence. All examples and all code will be made publicly available and open sources. Only open source components are used.Mool - Automated Log Analysis using Data Science and ML
QE automation for large systems is a great step forward in increasing system reliability. In the big-data world, multiple components have to come together to provide end-users with business outcomes. This means, that QE Automations scenarios need to be detailed around actual use cases, cross-cutting components. The system tests potentially generate large amounts of data on a recurring basis, verifying which is a tedious job. Given the multiple levels of indirection, the false positives of actual defects are higher, and are generally wasteful.
At Hortonworks, we’ve designed and implemented Automated Log Analysis System - Mool, using Statistical Data Science and ML. Currently the work in progress has a batch data pipeline with a following ensemble ML pipeline which feeds into the recommendation engine. The system identifies the root cause of test failures, by correlating the failing test cases, with current and historical error records, to identify root cause of errors across multiple components. The system works in unsupervised mode with no perfect model/stable builds/source-code version to refer to. In addition the system provides limited recommendations to file/open past tickets and compares run-profiles with past runs.
How Hadoop Makes the Natixis Pack More Efficient
Improving business performance is never easy! The Natixis Pack is like Rugby. Working together is key to scrum success. Our data journey would undoubtedly have been so much more difficult if we had not made the move together.
This session is the story of how ‘The Natixis Pack’ has driven change in its current IT architecture so that legacy systems can leverage some of the many components in Hortonworks Data Platform in order to improve the performance of business applications. During this session, you will hear:
• How and why the business and IT requirements originated
• How we leverage the platform to fulfill security and production requirements
• How we organize a community to:
o Guard all the players, no one gets left on the ground!
o Us the platform appropriately (Not every problem is eligible for Big Data and standard databases are not dead)
• What are the most usable, the most interesting and the most promising technologies in the Apache Hadoop community
We will finish the story of a successful rugby team with insight into the special skills needed from each player to win the match!
DETAILS
This session is part business, part technical. We will talk about infrastructure, security and project management as well as the industrial usage of Hive, HBase, Kafka, and Spark within an industrial Corporate and Investment Bank environment, framed by regulatory constraints.
HBase in Practice
HBase is a distributed, column-oriented database that stores data in tables divided into rows and columns. It is optimized for random, real-time read/write access to big data. The document discusses HBase's key concepts like tables, regions, and column families. It also covers performance tuning aspects like cluster configuration, compaction strategies, and intelligent key design to spread load evenly. Different use cases are suitable for HBase depending on access patterns, such as time series data, messages, or serving random lookups and short scans from large datasets. Proper data modeling and tuning are necessary to maximize HBase's performance.
The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)DataWorks Summit/Hadoop Summit
There has been an explosion of data digitising our physical world – from cameras, environmental sensors and embedded devices, right down to the phones in our pockets. Which means that, now, companies have new ways to transform their businesses – both operationally, and through their products and services – by leveraging this data and applying fresh analytical techniques to make sense of it. But are they ready? The answer is “no” in most cases.
In this session, we’ll be discussing the challenges facing companies trying to embrace the Analytics of Things, and how Teradata has helped customers work through and turn those challenges to their advantage.Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
In this talk, we will present a new distribution of Hadoop, Hops, that can scale the Hadoop Filesystem (HDFS) by 16X, from 70K ops/s to 1.2 million ops/s on Spotiy's industrial Hadoop workload. Hops is an open-source distribution of Apache Hadoop that supports distributed metadata for HSFS (HopsFS) and the ResourceManager in Apache YARN. HopsFS is the first production-grade distributed hierarchical filesystem to store its metadata normalized in an in-memory, shared nothing database. For YARN, we will discuss optimizations that enable 2X throughput increases for the Capacity scheduler, enabling scalability to clusters with >20K nodes. We will discuss the journey of how we reached this milestone, discussing some of the challenges involved in efficiently and safely mapping hierarchical filesystem metadata state and operations onto a shared-nothing, in-memory database. We will also discuss the key database features needed for extreme scaling, such as multi-partition transactions, partition-pruned index scans, distribution-aware transactions, and the streaming changelog API. Hops (www.hops.io) is Apache-licensed open-source and supports a pluggable database backend for distributed metadata, although it currently only support MySQL Cluster as a backend. Hops opens up the potential for new directions for Hadoop when metadata is available for tinkering in a mature relational database.
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...DataWorks Summit/Hadoop Summit
In high-risk manufacturing industries, regulatory bodies stipulate continuous monitoring and documentation of critical product attributes and process parameters. On the other hand, sensor data coming from production processes can be used to gain deeper insights into optimization potentials. By establishing a central production data lake based on Hadoop and using Talend Data Fabric as a basis for a unified architecture, the German pharmaceutical company HERMES Arzneimittel was able to cater to compliance requirements as well as unlock new business opportunities, enabling use cases like predictive maintenance, predictive quality assurance or open world analytics. Learn how the Talend Data Fabric enabled HERMES Arzneimittel to become data-driven and transform Big Data projects from challenging, hard to maintain hand-coding jobs to repeatable, future-proof integration designs.
Talend Data Fabric combines Talend products into a common set of powerful, easy-to-use tools for any integration style: real-time or batch, big data or master data management, on-premises or in the cloud.Backup and Disaster Recovery in Hadoop
While you could be tempted assuming data is already safe in a single Hadoop cluster, in practice you have to plan for more. Questions like: "What happens if the entire datacenter fails?, or "How do I recover into a consistent state of data, so that applications can continue to run?" are not a all trivial to answer for Hadoop. Did you know that HDFS snapshots are handling open files not as immutable? Or that HBase snapshots are executed asynchronously across servers and therefore cannot guarantee atomicity for cross region updates (which includes tables)? There is no unified and coherent data backup strategy, nor is there tooling available for many of the included components to build such a strategy. The Hadoop distributions largely avoid this topic as most customers are still in the "single use-case" or PoC phase, where data governance as far as backup and disaster recovery (BDR) is concerned are not (yet) important. This talk first is introducing you to the overarching issue and difficulties of backup and data safety, looking at each of the many components in Hadoop, including HDFS, HBase, YARN, Oozie, the management components and so on, to finally show you a viable approach using built-in tools. You will also learn not to take this topic lightheartedly and what is needed to implement and guarantee a continuous operation of Hadoop cluster based solutions.
More from DataWorks Summit/Hadoop Summit (20)
Running Apache Spark & Apache Zeppelin in Production
Running Apache Spark & Apache Zeppelin in Production
Unleashing the Power of Apache Atlas with Apache Ranger
Unleashing the Power of Apache Atlas with Apache Ranger
Enabling Digital Diagnostics with a Data Science Platform
Enabling Digital Diagnostics with a Data Science Platform
Double Your Hadoop Performance with Hortonworks SmartSense
Double Your Hadoop Performance with Hortonworks SmartSense
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Mool - Automated Log Analysis using Data Science and ML
Mool - Automated Log Analysis using Data Science and ML
The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)
Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
Recently uploaded
Taking AI to the Next Level in Manufacturing.pdf
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
Generating privacy-protected synthetic data using Secludy and Milvus
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
June Patch Tuesday
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
TrustArc Webinar - 2024 Global Privacy Survey
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
National Security Agency - NSA mobile device best practices
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
HCL Notes and Domino License Cost Reduction in the World of DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
20240609 QFM020 Irresponsible AI Reading List May 2024
Everything I found interesting about the irresponsible use of machine intelligence in May 2024
Digital Marketing Trends in 2024 | Guide for Staying Ahead
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
Building Production Ready Search Pipelines with Spark and Milvus
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Recently uploaded (20)
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
The Evolution of Big Data Pipelines at Intuit
- 1. The Evolution of Big Data Pipelines At Intuit June 30, 2016 #hadoopsummit #HS16SJ
- 2. Your Speakers Lokesh Rajaram Senior Software Engineer, Intuit likes Photography Rekha Joshi Principal Software Engineer, Intuit Currently likes Chopped
- 3. The Plan
- 6. Evolution of Big Data
- 7. Our Mission To improve our customers’ financial lives so profoundly … they can’t imagine going back to the old way!
- 8. Consumers Small Businesses Accounting Professionals Who we serve
- 9. 42M 2.3M 7MFile their own taxes with TurboTax Run their small businesses with QuickBooks Manage their personal finances with Mint The Numbers Are Growing 65+ Applications, 25% of US GDP
- 10. Era of Windows Era of Web E ra of th e Cl o u d Era of DOS Intuit - An Evolution Case Study Compliant data M o bi le Fi rs t 1980s 1990s 2000s • Employees: 150 • Customers: 1.3M customers • Revenue: $33M • Employees: 4,500 • Customers: 5.6M • Revenue: $1.04B • Employees: 7,700 • Customers: 37M • Revenue: $4.2B 20162010 Regulatory data Transactional data Batch data Real time data Complex, secure data
- 11. Data Is The Decision Maker
- 12. Evolution of Big Data Pipelines – The Need Secure Cloud Environment Single Cohesive Data Pipeline AB Testing Personalization Streaming Profile Store Fraud Detection Support Varied Use Cases and more..
- 13. Evolution of Big Data Pipelines Thin Slices - Minimal Viable Product
- 14. Evolution of Big Data Pipelines – The Recipe Taking the Data In Transforming Data Handling The Indigestion With Scale
- 15. Evolution of Big Data Pipelines – The Recipe No Snowflakes Solutions Getting Vested Stakeholders Agreements Establishing The Standards
- 16. Evolution of Big Data Pipelines – The Recipe Breaking The Silos Moving Organization In One Direction
- 17. Evolution of Big Data Pipelines – The Recipe ● Making The Configuration Knobs Work ● At Scale o Latency o Throughput ● Schema, PII, Metadata, Changes, Audit, Governance ● Controlled Access←→ Innovation ● Error Monitoring ● Cluster Deployment
- 18. Organization Evolution Data Evolution
- 19. SDK User-entered data Apache Kafka Collector: User-entered and clickstream data Real-time processing Personalization Engine Profile Store Big Data Pipeline Slice View
- 20. Big Data Pipeline Components
- 21. Monitoring The Pipeline AWS resource alarms Custom App MetricsJVM and App Metrics Custom process alerts Logging and alert
- 23. Evolution - Stage 0: Disparate And Chaotic Disparate Databases
- 24. Data Pipeline (an example) • Collect event stream data into one location • Handle ~ 200k events / sec • Payload ~ 3-5KB • Enrich message and load it into Hive in defined SLA
- 25. Evolution - Stage 1 Event Stream Oozie Sqoop Netezza Loader Hive QL operations Storm Samza Flume
- 26. Evolution - Stage 2 Event Stream { ReST }
- 27. Evolution - Stage 3 (HA & DR) SDK { ReST } SDK { ReST } Mirrorin g
- 31. Set of Changes • Network upgrades • Increase pipe • Broker • Mirrormaker • Host TCP
- 33. Evolution - Stage 4 (Streaming + Batch) SDK { ReST } SDK { ReST } Mirrorin g
- 34. Evolution - Stage 5 (Cloud only) SDK { ReST } Kafka Connectors
- 35. Evolution - Stage 5 (Cloud only - Future state)
- 36. Pipeline Essentials SDK { ReST } SDK { ReST }
- 38. Trust by Verification • Test all Observable End-points • Functional • Data Loss • Data Parity • Measure for SLA • Baseline Tests
- 41. Thank You!