WSO2Con Asia 2014 - Simultaneous Analysis of Massive Data Streams in real-time and Batch

•

1 like•837 views

This document discusses analyzing massive data streams in real-time and batch processing. It describes how data streams are created from sources like sensors and social media. The data can be stored in databases or data stores and analyzed either through batch processing using Hadoop/Spark or real-time processing using Storm/CEP. The document then outlines WSO2's big data architecture for simultaneously handling real-time and batch analysis of the same data streams using their BAM and CEP products. It concludes with a demo of this architecture.

Technology

Simultaneous Analysis of Massive Data
Streams in Real-Time and Batch
Anjana Fernando
Technical Lead
WSO2

Agenda
• How massive data streams created
• How to receive
• How to store
• How to analyze, batch vs real-time
• WSO2 Big Data solution
• Demo

Massive Data Streams -> Data Streams with Big Data

What is Big Data?
❏ The 3 Vs
❏ Velocity
❏ Volume
❏ Variety
Image Source: http://akrayasolutions.com/big-data/

Where does it originate from?
• Machine logs
• Social media
• Archives
• Traffic information
• Weather data
• Sensor data (IoT)

What do I do with it?
Create intelligence..
• Should I take an umbrella to work today?
• What is the best route to go back home?
• What are the current market trends?
• Are my servers running healthily?

Protocols used to publish data..
• HTTP
• MQTT
• Zigbee
• Thrift
• Avro
• ProtoBuf

How to store the data?
• Relational databases
• Block data stores
-> HDFS
• Column oriented
-> HBase
-> Cassandra
• Document based
-> MongoDB
-> CouchDB
• In-Memory
-> VoltDB
A
C P

How to analyse data?
• Two options:
-> Batch processing: Schedule data processing jobs
and receive the processed data later
-> Real-time processing: The queries are executed
and the results are retrieved instantly

Analysing data..
• Batch processing
-> Apache Hadoop: Map/Reduce processing system
and a distributed file system

Analysing data..
• Batch processing - Data Warehouse
-> Apache Hive - Hadoop based framework for working
on large scale data stores with SQL-like queries
INSERT OVERWRITE TABLE UserTable SELECT userName, COUNT(DISTINCT
orderID),SUM(quantity) FROM PhoneSalesTable WHERE version= "1.0.0"
GROUP BY userName;

Analysing data..
• Batch processing - In-Memory Computing
-> Apache Spark - Functional programming model, in-
memory computing, claims 10x - 100x faster than
Hadoop

Analysing data..
• Real-time processing - Stream Processing
-> Apache Storm - Distributed and fault-tolerant
Spouts Bolts

Analysing data..
• Real-time processing - Complex Event Processing
-> WSO2 Siddhi:

$Big Data Architecture with WSO2.. • Data Streams { 'name':'phone.retail.shop', 'version':'1.0.0', 'nickName': 'Phone_Retail_Shop', 'description': 'Phone Sales', 'metaData':[ {'name':'clientType','type':'STRING'} ], 'payloadData':[ {'name':'brand','type':'STRING'}, {'name':'quantity','type':'INT'}, {'name':'total','type':'INT'}, {'name':'user','type':'STRING'} ] } The common stream format used in both CEP and BAM; The stream definition contains the stream name, version and other attributes that makes up the stream.$

Big Data Architecture with WSO2..
• WSO2 BAM
-> Data Receiver - High performance binary format data
publishing with Apache Thrift, shared with WSO2 CEP
-> Data Storage - Cassandra for highly scalable data store
-> Data Analyzer - Hive based batch processing

Big Data Architecture with WSO2..
• WSO2 BAM..
-> Activity Monitoring: Implemented using a custom indexing
mechanism to instantly search for events of a specific activity in
the system

Big Data Architecture with WSO2..
• WSO2 BAM..
-> Incremental Data Processing - Customized Hive to support
incremental data processing:
@Incremental (name="salesAnalysis" , tables="PhoneSalesTable")
SELECT brandname,
Count(DISTINCT orderid),
Sum(quantity)
FROM phonesalestable
WHERE version = "1.0.0"
GROUP BY brandname;

Big Data Architecture with WSO2..
• WSO2 CEP
-> Same data receiver as BAM, where this is the point where the
same event is sent to both servers, where BAM for batch
processing and CEP for real-time processing of the same data
streams
-> Real-time in-memory processing, based on WSO2 Siddhi
engine, with data adapters for receiving and sending event with
different data types and transports, e.g. XML, JSON, Text, HTTP,
JMS, SMTP

This document discusses analyzing massive data streams in real-time and batch processing. It describes how data streams are created from sources like machine logs and sensors. The data can be stored in databases or data stores like HDFS. The data can then be analyzed through either batch processing using Hadoop/Hive or real-time processing using Storm/CEP engines like WSO2 Siddhi. The document presents WSO2's big data architecture for receiving and analyzing the same data streams in both batch using BAM and real-time using CEP.

Real-time analytics with HBase

Data Science Society

Schema Design Best Practices with Buzz Moschetti

MongoDB

제3회 사내기술세미나-hadoop(배포용)-dh kim-2014-10-1

Donghan Kim

The document provides a summary of the history and current state of big data and Hadoop. It discusses how the concept of big data emerged in the late 1990s and was further popularized by McKinsey in 2011. It then outlines the evolution of definitions and technologies around big data, including the development of Hadoop from the early 2000s onward. The document also analyzes current adoption trends, use cases, and market forecasts for big data and Hadoop.

Overview of Big data zoo

Gurvinder Singh

Data analysis as a service allows organizations to leverage big data technologies like Hadoop to store and process large amounts of data in a distributed, scalable manner. Key components of big data solutions include commodity hardware, network capacity, and software that uses a divide and conquer approach. Data can be stored unstructured in Hadoop Distributed File System or structured in NoSQL databases like HBase. MapReduce and Spark are common frameworks for processing data across clusters, and tools like Hive and Pig support analyzing structured data. Performance benchmarks are provided to help understand storage and processing speeds at different levels.

Apache Drill: An Active, Ad-hoc Query System for large-scale Data Sets

MapR Technologies

Make it fast for everyone - performance and middleware design

Sriskandarajah Suhothayan

This document discusses various techniques for optimizing application performance, including reducing latency and increasing throughput. It covers strategies like using data structures like linked lists, bloom filters, and Merkle trees efficiently. Other topics include removing contention through approaches like the disruptor pattern, optimizing for network performance, and leveraging the reactor pattern. Performance of transports like XML/JSON and SOAP/REST is also evaluated. Monitoring tools like Java Flight Recorder are also mentioned.

Webinar: Best Practices for Getting Started with MongoDB

MongoDB

MongoDB adoption continues to grow at a record pace due to the significant enhancements in developer productivity and scalability that the database provides. Occasionally, however, organizations new to the technology make mistakes that limit their ability to leverage the significant advantages MongoDB provides. This webinar will discuss some of the common mistakes made by users when they first start working with MongoDB, how to identify when you've made those mistakes, and how to resolve them.

Data analysis is an exploratory process that requires a variety of tools and a flexible data store. Data analysis projects are easy to start but quickly become difficult to manage and error prone when depending on file-based data storage. Relational databases are poorly equipped to accommodate the dynamic demands complex analysis. This talk describes best practices for using MongoDB for analytics projects. Examples will be drawn from a large scale text mining project (approximately 25 million documents) that applies machine learning (neural networks and support vector machines) and statistical analysis. Tools discussed include R, Spark, Python scientific stack, and custom pre-processing scripts but the focus is on using these with the document database.

Webinar: Managing Real Time Risk Analytics with MongoDB

MongoDB

Leveraging Big Data and Real-Time Analytics at Cxense

Simon Lia-Jonassen

Abstract: Cxense Insight helps companies to understand their audience and build great online experiences. Our interactive UI and APIs help customers to annotate, filter, segment and target their users based on the visited content and actions in realtime. Today we already track more than half a billion of unique user identities across more than 5000 web-sites, contributing to more than 10 billions of analytics events on a monthly basis. To leverage these amounts of data in realtime, we built a large distributed system relying on the concepts familiar from databases, information retrieval and data mining. The first part of this talk will therefore give an insight into the challenges, the architecture and the techniques we have used. While the second part of the talk will briefly demonstrate our UI and APIs in action. We hope that both parts will be interesting for undergraduate students taking IR/DB courses as well as PhD students, experienced researchers and staff.

Data integration

Ballerina

With systems composed of multiple, collaborating services, there is a vast quantity of enterprise data scattered across heterogeneous data sources. Data integration allows businesses to combine data residing in different sources to provide users a consolidated view. How you choose to use, integrate with, and analyze enterprise data may be different than what you're used to. This session will discuss how Ballerina can be used to solve this problem. The following key aspects will also be discussed during the talk. Connecting to different data sources using endpoints First class support for SQL result-sets, JSON data, XML data, etc. Dealing with the problem of consistent state using transactions Streaming a large quantity of data

Introduction to MongoDB (Webinar Jan 2011)

MongoDB

The document introduces MongoDB, an open source document database. It discusses MongoDB's document data model, flexible queries and updates, and deployment architectures. Key features highlighted include the document model, atomic operations, indexing, and replication support for scalability and high availability. Use cases mentioned include replacing relational databases for web applications, content management, and analytics.

Look Ma! No more blobs

Aparna Chaudhary

This document discusses using different data storage solutions like file systems, relational databases, MongoDB, and JCR for storing and querying documents. It notes the advantages of each system and highlights that MongoDB supports dynamic schemas, namespaces up to 123 characters, and GridFS for storing file metadata and content in chunks, making it a good fit for storing variable length documents with metadata. Performance tests showed MongoDB met the requirements of writing 1MB in under a millisecond and retrieving 1MB in under a specified time.

Large-Scale Real-Time Data Management for Engagement and Monetization

Simon Lia-Jonassen

Invited talk at the Workshop on Large-Scale and Distributed Systems for Information Retrieval 2015. Cxense helps companies understand their audience and build great online experiences. Cxense Insight and DMP let customers annotate, filter, segment and target their users based on the consumed content and performed actions in real-time. With more than 5000 active websites, Insight alone tracks more than a billion unique users with more than 15 billions page views per month. To leverage the huge amounts of data in real-time, we have built a large distributed system relying on techniques familiar from databases, information retrieval and data mining. In this talk, we outline our solutions and give some insight into the technology we use and the challenges we face. This introduction should be interesting to undergraduate and PhD students as well as experienced researchers and engineers.

MongoDB .local Munich 2019: Managing a Heterogeneous Stack with MongoDB & SQL

MongoDB

MongoDB for Time Series Data Part 1: Setting the Stage for Sensor Management

MongoDB

The document discusses schema design considerations for time series traffic monitoring data stored in MongoDB. It evaluates alternatives like storing a document per event, per minute, or per hour. Storing aggregated data improves write performance by changing inserts to updates, improves analytic queries by reducing document reads, and reduces memory requirements by decreasing index size. The best schema depends on the read and write workload of the specific application.

How to leverage MongoDB for Big Data Analysis and Operations with MongoDB's A...

Gianfranco Palumbo

Intro to bigdata on gcp (1)

SahilRaina21

Big data is characterized by large and complex datasets that are difficult to process using traditional software. These massive volumes of data, known by characteristics like volume, velocity, variety, veracity, value, and volatility, can provide insights to address business problems. Google Cloud Platform offers tools like Cloud Storage, Big Query, PubSub, Dataflow, and Cloud Storage storage classes that can handle big data according to these characteristics and help extract value from large and diverse datasets.

Mongo db and hadoop driving business insights - final

MongoDB

MongoDB and Hadoop can work together to solve big data problems facing today's enterprises. We will take an in-depth look at how the two technologies complement and enrich each other with complex analyses and greater intelligence. We will take a deep dive into the MongoDB Connector for Hadoop and how it can be applied to enable new business insights with MapReduce, Pig, and Hive, and demo a Spark application to drive product recommendations.

MongoDB + Spring

Norberto Leite

Mongodb

Thiago Veiga

MongoDB is an open-source document database that provides high performance, high availability, and automatic scaling. It stores data in flexible, JSON-like documents, enabling storage of data with complex relationships easily and supporting polyglot persistence. MongoDB can be used for applications such as content management systems, user profiles, logs, and more. It provides indexing, replication, load balancing and aggregation capabilities.

Socialite, the Open Source Status Feed Part 2: Managing the Social Graph

MongoDB

Webinar: Choosing the Right Shard Key for High Performance and Scale

MongoDB

Agility and Scalability with MongoDB

MongoDB

MongoDB has taken a clear lead in adoption among the new generation of databases, including the enormous variety of NoSQL offerings. A key reason for this lead has been a unique combination of agility and scalability. Agility provides business units with a quick start and flexibility to maintain development velocity, despite changing data and requirements. Scalability maintains that flexibility while providing fast, interactive performance as data volume and usage increase. We'll address the key organizational, operational, and engineering considerations to ensure that agility and scalability stay aligned at increasing scale, from small development instances to web-scale applications. We will also survey some key examples of highly-scaled customer applications of MongoDB.

MongoDB Schema Design Tips & Tricks

Juan Antonio Roy Couto

Intelligent integration with WSO2 ESB & WSO2 CEP

Sriskandarajah Suhothayan

This document discusses how WSO2 Enterprise Service Bus (ESB) and WSO2 Complex Event Processing (CEP) can be used together intelligently for integration. It notes there are many different software systems that need to be integrated in connected businesses, but doing so is challenging due to issues like disparate systems and dynamic requirements. The document proposes using an integration platform containing an ESB like WSO2 ESB to connect systems, along with other WSO2 products. It then discusses how the ESB can integrate with CEP for real-time monitoring, predictive actions, auto-recovery, and ensuring security and quality of service. Several patterns are provided that show how the ESB and CEP can work together, such

Data Science Stack with MongoDB and RStudio

Winston Chen

This document discusses using RStudio server with MongoDB to enable data science work on MongoDB data. Key points: - Fliptop uses MongoDB to store sales lead, opportunity, and contact data from their CRM. They also aggregate additional data sources. - They use RStudio server to pull MongoDB data into R data frames to enable exploring, analyzing, and prototyping machine learning algorithms on the data in an interactive way. - The rmongodb package allows transforming MongoDB data into R data frames. This allows working with the data in R for tasks like text processing, algorithm prototyping, and insights extraction.

MongoDB Best Practices

Lewis Lin 🦊

This document provides MongoDB best practices for schema design and data modeling. It discusses when to embed documents versus reference them, and how that impacts performance. It also uses an example of medical device data to illustrate how aggregating data at regular intervals (e.g. hourly instead of per minute) can significantly reduce storage requirements and improve query performance. Proper schema design is important for determining the required hardware resources.

Day 5 - Real-time Data Processing/Internet of Things (IoT) with Amazon Kinesis

Amazon Web Services

This document discusses Amazon Kinesis, a fully managed service for real-time processing of streaming data. It provides an overview of Kinesis and how it can be used to ingest, store, and process streaming data. Examples are given of how companies are using Kinesis for applications like game analytics, digital advertising metrics, and IoT data processing. The key benefits of Kinesis are also summarized such as its ease of use, real-time performance, elastic scalability, integration with other AWS services, and low cost.

What's hot

Dan Sullivan - Data Analytics and Text Mining with MongoDB - NoSQL matters Du...

NoSQLmatters

Webinar: Managing Real Time Risk Analytics with MongoDB

MongoDB

Leveraging Big Data and Real-Time Analytics at Cxense

Simon Lia-Jonassen

Data integration

Ballerina

Introduction to MongoDB (Webinar Jan 2011)

MongoDB

Look Ma! No more blobs

Aparna Chaudhary

Large-Scale Real-Time Data Management for Engagement and Monetization

Simon Lia-Jonassen

MongoDB .local Munich 2019: Managing a Heterogeneous Stack with MongoDB & SQL

MongoDB

MongoDB for Time Series Data Part 1: Setting the Stage for Sensor Management

MongoDB

How to leverage MongoDB for Big Data Analysis and Operations with MongoDB's A...

Gianfranco Palumbo

Intro to bigdata on gcp (1)

SahilRaina21

Mongo db and hadoop driving business insights - final

MongoDB

MongoDB + Spring

Norberto Leite

Mongodb

Thiago Veiga

Socialite, the Open Source Status Feed Part 2: Managing the Social Graph

MongoDB

Webinar: Choosing the Right Shard Key for High Performance and Scale

MongoDB

Agility and Scalability with MongoDB

MongoDB

MongoDB Schema Design Tips & Tricks

Juan Antonio Roy Couto

Intelligent integration with WSO2 ESB & WSO2 CEP

Sriskandarajah Suhothayan

Data Science Stack with MongoDB and RStudio

Winston Chen

What's hot (20)

Dan Sullivan - Data Analytics and Text Mining with MongoDB - NoSQL matters Du...

Webinar: Managing Real Time Risk Analytics with MongoDB

Leveraging Big Data and Real-Time Analytics at Cxense

Data integration

Introduction to MongoDB (Webinar Jan 2011)

Look Ma! No more blobs

Large-Scale Real-Time Data Management for Engagement and Monetization

MongoDB .local Munich 2019: Managing a Heterogeneous Stack with MongoDB & SQL

MongoDB for Time Series Data Part 1: Setting the Stage for Sensor Management

How to leverage MongoDB for Big Data Analysis and Operations with MongoDB's A...

Intro to bigdata on gcp (1)

Mongo db and hadoop driving business insights - final

MongoDB + Spring

Mongodb

Socialite, the Open Source Status Feed Part 2: Managing the Social Graph

Webinar: Choosing the Right Shard Key for High Performance and Scale

Agility and Scalability with MongoDB

MongoDB Schema Design Tips & Tricks

Intelligent integration with WSO2 ESB & WSO2 CEP

Data Science Stack with MongoDB and RStudio

Similar to WSO2Con Asia 2014 - Simultaneous Analysis of Massive Data Streams in real-time and Batch

MongoDB Best Practices

Lewis Lin 🦊

Day 5 - Real-time Data Processing/Internet of Things (IoT) with Amazon Kinesis

Amazon Web Services

Big data from the trenches

Azrul MADISA

This document provides an overview of big data concepts from the perspective of an enterprise data architect. It discusses the data journey from acquisition to analytics and highlights best practices for data quality, sandboxing data on the cloud, balancing real-time and batch analytics, and putting these concepts together under a tiered data architecture. The architecture proposes investing more in fast data sources and less in cold data, using data virtualization to provide a unified view, and keeping sensitive customer data localized to comply with regulations.

Filtering From the Firehose: Real Time Social Media Streaming

Cloud Elements

Introduction to WSO2 Data Analytics Platform

Srinath Perera

This document provides an introduction to the WSO2 Analytics Platform. It discusses how the platform allows users to collect data from various sources using a sensor API, then perform analysis on the data through both batch and real-time means. Batch analysis uses technologies like Apache Spark and Hadoop to perform tasks like finding averages, max/min, and building KPIs. Real-time analysis uses complex event processing to run queries over streaming data and detect patterns. The platform also enables predictive analytics using machine learning algorithms and anomaly detection. Results are then communicated through dashboards and alerts.

Real-time big data analytics based on product recommendations case study

deep.bi

We started as an ad network. The challenge was to recommend the best product (out of millions) to the right person in a given moment (thousands of users within a second). We have delivered 5 billion ad views since 24 months. To put it in the scale context: If we would serve 1 ad per second it will take 160 years to serve 5 billion ads. So we needed a solution. SQL databases did not work. Popular NoSQL databases did not work. Standard data warehouse approaches (pre-aggregations, creating schemas) - did not work too. Re-thinking all the problems with huge data streams flowing to us every second we have built a complete solution based on open-source technologies and fresh, smart ideas from our engineering team. It is called deep.bi and now we make it available to other companies. deep.bi lets high-growth companies solve fast data problems by providing scalable, flexible and real-time data collection, enrichment and analytics. It was built using: - Node.js - API - Kafka - collecting and distributing data - Spark Streaming - ETL, data enrichments - Druid - real-time analytics - Cassandra - user events store - Hadoop + Parquet + Spark - raw data store + ad-hoc queries

Introduction to Azure DocumentDB

Denny Lee

Denny Lee introduced Azure DocumentDB, a fully managed NoSQL database service. DocumentDB provides elastic scaling of throughput and storage, global distribution with low latency reads and writes, and supports querying JSON documents with SQL and JavaScript. Common scenarios that benefit from DocumentDB include storing product catalogs, user profiles, sensor telemetry, and social graphs due to its ability to handle hierarchical and de-normalized data at massive scale.

Getting Started with Real-time Analytics

Amazon Web Services

This session is recommended for anyone interested in understanding how to use AWS big data services to develop real-time analytics applications. In this session, you will get an overview of a number of Amazon's big data and analytics services that enable you to build highly scaleable cloud applications that immediately and continuously analyze large sets of distributed data. We'll explain how services like Amazon Kinesis, EMR and Redshift can be used for data ingestion, processing and storage to enable real-time insights and analysis into customer, operational and machine generated data and log files. We'll explore system requirements, design considerations, and walk through a specific customer use case to illustrate the power of real-time insights on their business.

Analytics in Your Enterprise

WSO2Con Asia 2014 - Simultaneous Analysis of Massive Data Streams in real-time and Batch

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to WSO2Con Asia 2014 - Simultaneous Analysis of Massive Data Streams in real-time and Batch

Similar to WSO2Con Asia 2014 - Simultaneous Analysis of Massive Data Streams in real-time and Batch (20)

More from WSO2

More from WSO2 (20)

Recently uploaded

Recently uploaded (20)

WSO2Con Asia 2014 - Simultaneous Analysis of Massive Data Streams in real-time and Batch