Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Big Data- GITPRO 2013
By - Sameer Wadkar
Co-Founder & Big Data Architect / Data Scientist at Axiomine
Agenda
• What is Big Data
• Big Data Characteristics
• Big Data and Business Intelligence Applications
• Big Data and Tran...
What is Big Data?
Volume
Velocity
Big
Data
Variety
Big Data monitors 12 Terabytes
of Tweets each day to improve
product se...
Visualizing Big Data
• 1 Petabyte is 54000 movies in digital format
• Reading 1 Terabyte of data sequentially from a singl...
Big Data vs. Traditional
Big Data Architecture
…
In Big Data architectures the application is moves to the data. Why?
User...
Why is Big Data hard?
Divide out and conquer in place is a Big Data Strategy
• Goal is to divide the data on multiple node...
Big Data Characteristics
Scale-out, Fault Tolerance & Graceful Recovery are essential features
• Big Data Systems must sca...
Big Data Applications
Big Data design is dictated by the nature of the applications
• Business Intelligence applications
•...
BI - Sample Use-Case
A very simple query but size makes all the difference
• “Select SUM(SALES_AMT) from SALES where state...
BI Big Data Flavors
We discuss three flavors in increasing order of scale-out capability
Big Data Flavor Products
In-Memor...
In Memory Databases
If State=‘VA’ is next query & cache is only big enough to hold one state
Simplified version - Data is ...
In Memory Databases (cont.)
In-Memory DBs provide real-time querying on moderate sized data
• Specialized hardware
• Speci...
MPP (Typical Architecture)
Data is partitioned horizontally across all slave nodes. Assume “Sale Year” is the distribution...
MPP (cont.)
MPP supports familiar RDBMS paradigm for medium scalability
• Balances throughput with responsiveness.
• Some ...
MapReduce
Data is partitioned randomly/redundantly across all data nodes. Every data node contain sales
data for every sta...
MapReduce (cont.)
Map Reduce – How it works
Year Sales
1990 $1M
1982 $2M
… ..
1999 $20M
Map Process 1
Year Sales
1998 $6M
...
MapReduce (cont.)
MapReduce is general purpose but requires complex skills.
• Batch oriented - Maximizes throughput not re...
Summary of BI Apps
Each option has tradeoffs. Choose based on requirements
Big Data Flavor How much data can it typically ...
Transactional System - Use-Case
How many items in stock do users A and B on their second access?
Web Based Online
Store Da...
Context – CAP Theorem
You can get any two but not all three features in any system
Characteristic
Consistency All nodes (a...
CA – Single RDBMS
A single RDBMS instance is both consistent and available
Web Based Online
Store RDBMS
User A Looks up
it...
CP – Distributed RDBMS
A Distributed RDBMS is consistent and resilient to failure of nodes
Web Based Online
Store
East Reg...
AP – Distributed RDBMS
Eventual Consistency is the key to Big Data Transactional Systems
Web Based Online Store
User A Loo...
Hybrid Solution
Big Data Techniques – Not an either or choice!
Large
Structured DB
Large
Unstructured
DB
Map Reduce
based ...
Exploring over Millions US Patent Pages at the Speed of Thought
www.axiomine.com/patents/
Demo- US Patent Explorer
Patent Explorer Goals
Seamlessly navigate Structured and Unstructured data in real-time
• Navigate 3 million US Patents Da...
Patent Metadata
Cannot answer – What is the title of Patent No 8086905?
Source – National Bureau of Economic Research
http...
Patent Text
Need to merge both metadata & text
Source – Google
http://www.google.com/googlebooks/uspto.html
Sample File
High Level Architecture
Need to merge both metadata & text
Hadoop
Patent
Metadata
Patent Text
Navigation, Search
& Text An...
Big Data Flavors – Summary
Choose a Big Data tool and product based on requirements
Flavor Characteristics
Map-Reduce • Ma...
Upcoming SlideShare
Loading in …5
×

Big Data presentation at GITPRO 2013

919 views

Published on

Published in: Technology
  • Be the first to like this

Big Data presentation at GITPRO 2013

  1. 1. Big Data- GITPRO 2013 By - Sameer Wadkar Co-Founder & Big Data Architect / Data Scientist at Axiomine
  2. 2. Agenda • What is Big Data • Big Data Characteristics • Big Data and Business Intelligence Applications • Big Data and Transactional Applications • Demo
  3. 3. What is Big Data? Volume Velocity Big Data Variety Big Data monitors 12 Terabytes of Tweets each day to improve product sentiment analysis (source :IBM) Amazon and PayPal use Big-Data for real time fraud detection (source: McKinsey) In 15 of the US economy’s 17 sectors, companies with upward of 1,000 employees store, on average, more information than the Library of Congress (source: McKinsey) Big Data monitors 12 TB of Tweets each day to improve product sentiment analysis (source :IBM) Most Big Data applications are based around the Volume dimension
  4. 4. Visualizing Big Data • 1 Petabyte is 54000 movies in digital format • Reading 1 Terabyte of data sequentially from a single disk drive takes 3 hours • Typical speed to read from the hard-disk – 80 MB/sec • Traversing 1 Terabyte of data randomly over 1 disk (a typical database access scenario) requires orders of magnitude longer • Disk transfer rates are significantly higher than disk seek rate Single node processing capacity will drown in the face of Big Data
  5. 5. Big Data vs. Traditional Big Data Architecture … In Big Data architectures the application is moves to the data. Why? User launches a batch job 1 Three Tier Architecture App Request Data from Data Tier 2 Data Tier sends data to the App Tier 3 App Tier processes data4 App Tier sends the report 5 User requests a report 1 Master Distributes Application 2 Master launches App on nodes 35 User downloads results 4 All Nodes process the data on their nodes Master Node Application & Data Tier Data Tier Application Tier
  6. 6. Why is Big Data hard? Divide out and conquer in place is a Big Data Strategy • Goal is to divide the data on multiple nodes and conquer by processing the data in-place of the node. • Real world processing cannot be always divided into smaller sub- problems (Divide and Conquer is not always feasible) • Data has dependencies • Normalization v/s Denormalization • There are processing dependencies. Later phase of the process may require results of an earlier phase • Single Pass v/s Multi-pass
  7. 7. Big Data Characteristics Scale-out, Fault Tolerance & Graceful Recovery are essential features • Big Data Systems must scale out • Adding more nodes should lead to greater parallelization • Big Data Systems must be resilient to partial failure • If one part of the system fails other parts should continue to function • Big Data Systems must be able to self-recover from partial failure • If any part of the system fails another part of the system will attempt to recover from the failure. • Data must be replicated on separate nodes • Loss of any node does not lose data or processing. • Recovery should be transparent to the end-user.
  8. 8. Big Data Applications Big Data design is dictated by the nature of the applications • Business Intelligence applications • Read-only systems • ETL Systems • Query massive data for purpose of generating reports or for large scale transformations and import into destination data-source • Transactional Applications • One part of the system updates data while another part reads the data • Example Systems – Imagine running a online store of Amazon.com scale.
  9. 9. BI - Sample Use-Case A very simple query but size makes all the difference • “Select SUM(SALES_AMT) from SALES where state=‘MD’ group by YEAR order by YEAR” • Find me total sales revenue by year for “Maryland” and order them by year • What if SALES table has billions of rows over 20 years? Sales Transactions Table Big Data Reporting Year Sales Revenue 1980 11 Million 1981 13 Million … … 2010 10 Billion Input Output
  10. 10. BI Big Data Flavors We discuss three flavors in increasing order of scale-out capability Big Data Flavor Products In-Memory Databases Oracle Exalytics, SAP HANA Massively Parallel Computing (MPP) Greenplum, Netezza Map Reduce Hadoop
  11. 11. In Memory Databases If State=‘VA’ is next query & cache is only big enough to hold one state Simplified version - Data is partitioned randomly across all nodes. Selection Phase 1. Each data node contains fast Memory (SSD) and mechanism to apply “Where” clause 2. Only the necessary data (“MD” records) are passed over the expensive Network I/O to the processing node Processing Phase 1. The processing nodes will compute the SUM(sales_amt) by year 2. Order the results 3. Place it In-Memory cache • First execution of the query is slow. • Subsequent executions are very fast (almost real-time) as the cache is hot. • Cache has SQL-Interface. User experiences “Real-Time”!! Data Node Data Node …… Data Node Processing Node In-Memory TB Cache with SQL Interface User SQL Interface Fetch Phase The user is served the results from the cache through the familiar SQL Interface
  12. 12. In Memory Databases (cont.) In-Memory DBs provide real-time querying on moderate sized data • Specialized hardware • Specialized I/O and Flash Memory for faster I/O • Massive in-memory cache (Multi-Terabyte TB) with SQL Interface Characteristics • Familiar model (SQL Interface) • Can integrate with standard toolkits and BI Solutions • Unified software/hardware solution Pros • Vendor lock-in • Expensive – Hardware as well as licensing cost • Typically cannot scale beyond 1-2 TB of data • Works best when same data is read often (Cache remains hot). Cons
  13. 13. MPP (Typical Architecture) Data is partitioned horizontally across all slave nodes. Assume “Sale Year” is the distribution key. Secondary indexes by other keys can be added to each slave node. Distributed Query Phase 1. Each salve node will compute the query for the data contained in its own node. 2. Each year data is completely held in its own node 3. This phase produces partial query results which are complete for each year Slave Node (1980 & 1990 data) Slave Node (1981 & 1991 data) .. Slave Node (2000 & 2010 data) Master Node Accumulation Phase 1. All slave results are aggregated and sorted. • Scale Out – More nodes means less years of data per node. • Redundancy & Failover – Each node will have a backup node. • Data distribution strategy & access patterns compatibility determine performance. • Enormous network overhead if access-patterns do not respect distribution strategy
  14. 14. MPP (cont.) MPP supports familiar RDBMS paradigm for medium scalability • Balances throughput with responsiveness. • Some implementations use specialized hardware (Ex. Netezza uses FPGA) • Familiar RDBMS (SQL) paradigm • Can scale to 10’s of Terabytes in most cases Characteristics • Familiar model (SQL Interface) • Can integrate with standard toolkits and BI Solutions Pros • Vendor lock-in • Cannot scale for ad-hoc queries • Queries must respect data distribution strategy for acceptable performance. Cons
  15. 15. MapReduce Data is partitioned randomly/redundantly across all data nodes. Every data node contain sales data for every state and every year. Map Phase 1. Each data node reads all of its records sequentially. 2. It filters out all non- “MD” state records 3. It computes a SUM(sales_amt) by year for each year Data Node Data Node … Data Node Reduce Node Reduce Phase 1. Reduce node receives SUM(sales_amt) for state “MD” by each year from each node 2. Add all map results by year and compute the final SUM(sales_amt) by year for “MD” sales 3. Orders results by year • Data blocks (order of 128 MB) are stored and accessed contiguously • Scales out efficiently and degrades gracefully. • If a task fails the framework restarts automatically (on another node if necessary) – Redundancy and Graceful Recovery Master Node Map Nodes
  16. 16. MapReduce (cont.) Map Reduce – How it works Year Sales 1990 $1M 1982 $2M … .. 1999 $20M Map Process 1 Year Sales 1998 $6M 1982 $5M … .. 2010 $30M Map Process 20 …… Reduce Node adds up all the map results, sorts by year to give final result Year Sales 1980 $100M 1981 $102M … .. 2010 $250M
  17. 17. MapReduce (cont.) MapReduce is general purpose but requires complex skills. • Batch oriented - Maximizes throughput not responsiveness Characteristics • Simple programming model • Scales out efficiently • Failure and redundancy built in • Adapts well to a wide variety of problems Pros • Requires custom programming • Higher level languages (SQL-like) exist but programming skills are often critical • Requires a complex array of skills to manage & maintain a MapReduce System Cons
  18. 18. Summary of BI Apps Each option has tradeoffs. Choose based on requirements Big Data Flavor How much data can it typically handle? In Memory Databases Order of 1TB Massively Parallel Databases Order of 10 TB MapReduce Order of 100’s of TB into the Petabyte range
  19. 19. Transactional System - Use-Case How many items in stock do users A and B on their second access? Web Based Online Store Database User A Looks up item X User B Looks up item X User C buys item X Updates inventory User A Looks up item X again User B Looks up item X again
  20. 20. Context – CAP Theorem You can get any two but not all three features in any system Characteristic Consistency All nodes (and users) see the same data at the same time. Availability A guarantee that every request receives a valid response. Site does not go down or appear down under heavy load. Partition Tolerance The system continues to function regardless of loss of one of its components
  21. 21. CA – Single RDBMS A single RDBMS instance is both consistent and available Web Based Online Store RDBMS User A Looks up item X User B Looks up item X User C buys item X Updates inventory User A Looks up item X again User B Looks up item X again • When setup in “Read Committed” every user sees the same inventory count • System responds with last committed inventory count even during updates • Consistent • Available
  22. 22. CP – Distributed RDBMS A Distributed RDBMS is consistent and resilient to failure of nodes Web Based Online Store East Region RDBMS User A Looks up item X User B Looks up item X User C buys item X Updates inventory User A Looks up item X again User B Looks up item X again • Under “Read Committed” mode all user see consistent counts. • If one DB fails the other one will serve all users(Partition Tolerance) • During two phase commit system is unavailable. • Consistent • Partition Tolerant West Region RDBMS 2- Phase Commit
  23. 23. AP – Distributed RDBMS Eventual Consistency is the key to Big Data Transactional Systems Web Based Online Store User A Looks up item X User B Looks up item X User C buys item X Updates inventory User A Looks up item X again User B Looks up item X again • Amazon Dynamo and Apache Cassandra work on this principle • If one DB fails the other one will serve all users(Partition Tolerance) • Users will always be able to browse all products but occasionally some users will see a stale count of inventory (Eventual Consistency) • Available • Partition Tolerant • Eventually Consistent
  24. 24. Hybrid Solution Big Data Techniques – Not an either or choice! Large Structured DB Large Unstructured DB Map Reduce based ETL MPP DB In-Memory DB Business Users can use familiar SQL based tools in real-time. In- Memory DB allows that No-SQL DB Programmers, System Admins with no real-time requirements can use all three techniques. NoSQL DB’s allow technical users to gain real-time benefits in ways which suite their complex needs. Familiar BI Solution Programs & Scripts 100 TB to 1 PB 5-10 TB 1 TB Few 100 GB
  25. 25. Exploring over Millions US Patent Pages at the Speed of Thought www.axiomine.com/patents/ Demo- US Patent Explorer
  26. 26. Patent Explorer Goals Seamlessly navigate Structured and Unstructured data in real-time • Navigate 3 million US Patents Data (Text and Metadata) from 1963 to 1999 at the speed of thought. • Data Sources • Patent Metadata - National Bureau of Economic Research • Patent Text – Bulk Download from Google Site • Each week granted patents are published to the Google Site as an archive. • Size of uncompressed data • Structured Metadata – Approximately 2 GB • Patent Text Data – Approximately 300 GB
  27. 27. Patent Metadata Cannot answer – What is the title of Patent No 8086905? Source – National Bureau of Economic Research http://data.nber.org/patents/ Patent Master Pairwise Citations * Inventors * Patent Master Other Master Data Company Master Country Master Classification Master Contains only meta-data. No text data such as Patent Title available. Ex. Pairwise citations contains millions of patent id pairs
  28. 28. Patent Text Need to merge both metadata & text Source – Google http://www.google.com/googlebooks/uspto.html Sample File
  29. 29. High Level Architecture Need to merge both metadata & text Hadoop Patent Metadata Patent Text Navigation, Search & Text Analytics Apache Solr Patent Details MongoDB Text Enhanced Citation Data Raw Data Tier ETL & Text Analytics Tier Search & Visualization Navigate, Search & Visualize Drill down to Patent Details
  30. 30. Big Data Flavors – Summary Choose a Big Data tool and product based on requirements Flavor Characteristics Map-Reduce • Massive 100 TB to 1 PB Scale ELT • Complex Analytics on Massive Data • Large Scale Unstructured Data Analysis Massively Parallel Processing (MPP) • Batch oriented aggregations • Analytics on Moderately Large Structured Data with predictable access patterns In-Memory DB • Similar to MPP but with real-time access patterns required. • Rich and Interactive Business Intelligence Apps NoSQL databases • Similar to In-Memory DB but simpler (Non SQL) access patterns • Provide fast access to detail data where other techniques are used to serve summary data GPGPU • Real time Value At Risk (Financial Risk Management) • Compute intensive analytics Ex. Simulation of a Hospital Waiting Room over 1 years

×