SlideShare a Scribd company logo
1 of 41
Download to read offline
Marko Grobelnik 
marko.grobelnik@ijs.si 
Jozef Stefan Institute 
Ljubljana, Slovenia 
Stavanger, May 8th 2012
 
Introduction 
◦ 
What is Big data? 
◦ 
Why Big-Data? 
◦ 
When Big-Data is really a problem? 
 
Techniques 
 
Tools 
 
Applications 
 
Literature
 
‘Big-data’ is similar to ‘Small-data’, but bigger 
 
…but having data bigger consequently requires different approaches: 
◦ 
techniques, tools & architectures 
 
…to solve: 
◦ 
New problems… 
◦ 
…and old problems in a better way.
From “Understanding Big Data” by IBM
Big-Data
 
Key enablers for the growth of “Big Data” are: 
◦ 
Increase of storage capacities 
◦ 
Increase of processing power 
◦ 
Availability of data
 
NoSQL 
◦ 
DatabasesMongoDB, CouchDB, Cassandra, Redis, BigTable, Hbase, Hypertable, Voldemort, Riak, ZooKeeper 
 
MapReduce 
◦ 
Hadoop, Hive, Pig, Cascading, Cascalog, mrjob, Caffeine, S4, MapR, Acunu, Flume, Kafka, Azkaban, Oozie, Greenplum 
 
Storage 
◦ 
S3, Hadoop Distributed File System 
 
Servers 
◦ 
EC2, Google App Engine, Elastic, Beanstalk, Heroku 
 
Processing 
◦ 
R, Yahoo! Pipes, Mechanical Turk, Solr/Lucene, ElasticSearch, Datameer, BigSheets, Tinkerpop
 
…when the operations on data are complex: 
◦ 
…e.g. simple counting is not a complex problem 
◦ 
Modeling and reasoning with data of different kinds can get extremely complex 
 
Good news about big-data: 
◦ 
Often, because of vast amount of data, modeling techniques can get simpler (e.g. smart counting can replace complex model based analytics)… 
◦ 
…as long as we deal with the scale
 
Research areas (such as IR, KDD, ML, NLP, SemWeb, …) are sub- cubes within the data cube 
Scalability 
Dynamicity 
Context 
Quality 
Usage
 
Good recommendations can make a big difference when keeping a user on a web site 
◦ 
…the key is how rich context model a system is using to select information for a user 
◦ 
Bad recommendations <1% users, good ones >5% users click 
Contextual personalized recommendations generated in ~20ms
 
Domain 
 
Sub-domain 
 
Page URL 
 
URL sub-directories 
 
Page Meta Tags 
 
Page Title 
 
Page Content 
 
Named Entities 
 
Has Query 
 
Referrer Query 
 
Referring Domain 
 
Referring URL 
 
Outgoing URL 
 
GeoIP Country 
 
GeoIP State 
 
GeoIP City 
 
Absolute Date 
 
Day of the Week 
 
Day period 
 
Hour of the day 
 
User Agent 
 
Zip Code 
 
State 
 
Income 
 
Age 
 
Gender 
 
Country 
 
Job Title 
 
Job Industry
Log Files 
(~100M page clicks per day) 
User profiles 
NYT articles 
Stream of profiles 
Advertisers 
Segment 
Keywords 
Stock Market 
Stock Market, mortgage, banking, investors, Wall Street, turmoil, New York Stock Exchange 
Health 
diabetes, heart disease, disease, heart, illness 
Green Energy 
Hybrid cars, energy, power, model, carbonated, fuel, bulbs, 
Hybrid cars 
Hybrid cars, vehicles, model, engines, diesel 
Travel 
travel, wine, opening, tickets, hotel, sites, cars, search, restaurant 
… 
… 
Segments 
Trend Detection System 
Stream of clicks 
Trends and 
updated segments 
Campaign 
to sell segments 
$ 
Sales
 
50Gb of uncompressed log files 
 
10Gb of compressed log files 
 
0.5Gb of processed log files 
 
50-100M clicks 
 
4-6M unique users 
 
7000 unique pages with more then 100 hits 
 
Index size 2Gb 
 
Pre-processing & indexing time 
◦ 
~10min on workstation (4 cores & 32Gb) 
◦ 
~1hour on EC2 (2 cores & 16Gb)
 
Alarms Explorer Server implements three real-time scenarios on the alarms stream: 
1. 
Root-Cause-Analysis – finding which device is responsible for occasional “flood” of alarms 
2. 
Short-Term Fault Prediction – predict which device will fail in next 15mins 
3. 
Long-Term Anomaly Detection – detect unusual trends in the network 
 
…system is used in British Telecom 
Alarms Server 
Alarms Explorer Server 
Live feed of data 
Operator 
Big board display 
Telecom Network (~25 000 devices) 
Alarms ~10-100/sec
 
Presented in “Planetary-Scale Views on a Large Instant-Messaging Network” by Jure Leskovec and Eric Horvitz WWW2008
 
Observe social and communication phenomena at a planetary scale 
 
Largest social network analyzed to date 
Research questions: 
How does communication change with user demographics (age, sex, language, country)? 
How does geography affect communication? 
What is the structure of the communication network? 33
 
We collected the data for June 2006 
 
Log size: 
150Gb/day (compressed) 
 
Total: 1 month of communication data: 
4.5Tb of compressed data 
 
Activity over June 2006 (30 days) 
◦ 
245 million users logged in 
◦ 
180 million users engaged in conversations 
◦ 
17,5 million new accounts activated 
◦ 
More than 30 billion conversations 
◦ 
More than 255 billion exchanged messages 34
35
36
 
Count the number of users logging in from particular location on the earth 37
 
Logins from Europe 38
 
6 degrees of separation [Milgram ’60s] 
 
Average distance between two random users is 6.6 
 
90% of nodes can be reached in < 8 hops 
Hops 
Nodes 
1 
10 
2 
78 
3 
396 
4 
8648 
5 
3299252 
6 
28395849 
7 
79059497 
8 
52995778 
9 
10321008 
10 
1955007 
11 
518410 
12 
149945 
13 
44616 
14 
13740 
15 
4476 
16 
1542 
17 
536 
18 
167 
19 
71 
20 
29 
21 
16 
22 
10 
23 
3 
24 
2 
25 
3
Big data tutorial_part4
Big data tutorial_part4

More Related Content

What's hot

Spark Social Media
Spark Social Media Spark Social Media
Spark Social Media suresh sood
 
Systemof insight
Systemof insightSystemof insight
Systemof insightsuresh sood
 
Geospatial Intelligence Middle East 2013_Big Data_Steven Ramage
Geospatial Intelligence Middle East 2013_Big Data_Steven RamageGeospatial Intelligence Middle East 2013_Big Data_Steven Ramage
Geospatial Intelligence Middle East 2013_Big Data_Steven RamageSteven Ramage
 
A brief history of "big data"
A brief history of "big data"A brief history of "big data"
A brief history of "big data"Nicola Ferraro
 
The Web3 Data Economy: Ocean Protocol
The Web3 Data Economy: Ocean ProtocolThe Web3 Data Economy: Ocean Protocol
The Web3 Data Economy: Ocean ProtocolTrent McConaghy
 
Enterprise Data Governance: Leveraging Knowledge Graph & AI in support of a d...
Enterprise Data Governance: Leveraging Knowledge Graph & AI in support of a d...Enterprise Data Governance: Leveraging Knowledge Graph & AI in support of a d...
Enterprise Data Governance: Leveraging Knowledge Graph & AI in support of a d...Connected Data World
 
Big Data and Classification
Big Data and ClassificationBig Data and Classification
Big Data and Classification303Computing
 
Graph Database in Graph Intelligence
Graph Database in Graph IntelligenceGraph Database in Graph Intelligence
Graph Database in Graph IntelligenceChen Zhang
 
Web search-metrics-tutorial-www2010-section-1of7-introduction
Web search-metrics-tutorial-www2010-section-1of7-introductionWeb search-metrics-tutorial-www2010-section-1of7-introduction
Web search-metrics-tutorial-www2010-section-1of7-introductionAli Dasdan
 
The future of mobile and big data
The future of mobile and big dataThe future of mobile and big data
The future of mobile and big dataspirit conference
 
Challenging Problems for Scalable Mining of Heterogeneous Social and Informat...
Challenging Problems for Scalable Mining of Heterogeneous Social and Informat...Challenging Problems for Scalable Mining of Heterogeneous Social and Informat...
Challenging Problems for Scalable Mining of Heterogeneous Social and Informat...BigMine
 
Software is eating the world data is digesting it DF2013
Software is eating the world data is digesting it DF2013Software is eating the world data is digesting it DF2013
Software is eating the world data is digesting it DF2013James Governor
 
Neo4j GraphTalks Munich - Einführung
Neo4j GraphTalks Munich - EinführungNeo4j GraphTalks Munich - Einführung
Neo4j GraphTalks Munich - EinführungNeo4j
 

What's hot (20)

Spark Social Media
Spark Social Media Spark Social Media
Spark Social Media
 
Datapreneurs
DatapreneursDatapreneurs
Datapreneurs
 
Big Data
Big DataBig Data
Big Data
 
Systemof insight
Systemof insightSystemof insight
Systemof insight
 
Geospatial Intelligence Middle East 2013_Big Data_Steven Ramage
Geospatial Intelligence Middle East 2013_Big Data_Steven RamageGeospatial Intelligence Middle East 2013_Big Data_Steven Ramage
Geospatial Intelligence Middle East 2013_Big Data_Steven Ramage
 
A brief history of "big data"
A brief history of "big data"A brief history of "big data"
A brief history of "big data"
 
The Web3 Data Economy: Ocean Protocol
The Web3 Data Economy: Ocean ProtocolThe Web3 Data Economy: Ocean Protocol
The Web3 Data Economy: Ocean Protocol
 
Big data
Big dataBig data
Big data
 
Spark
SparkSpark
Spark
 
Big-Data Computing on the Cloud
Big-Data Computing on the CloudBig-Data Computing on the Cloud
Big-Data Computing on the Cloud
 
Enterprise Data Governance: Leveraging Knowledge Graph & AI in support of a d...
Enterprise Data Governance: Leveraging Knowledge Graph & AI in support of a d...Enterprise Data Governance: Leveraging Knowledge Graph & AI in support of a d...
Enterprise Data Governance: Leveraging Knowledge Graph & AI in support of a d...
 
Big Data and Classification
Big Data and ClassificationBig Data and Classification
Big Data and Classification
 
Graph Database in Graph Intelligence
Graph Database in Graph IntelligenceGraph Database in Graph Intelligence
Graph Database in Graph Intelligence
 
Web search-metrics-tutorial-www2010-section-1of7-introduction
Web search-metrics-tutorial-www2010-section-1of7-introductionWeb search-metrics-tutorial-www2010-section-1of7-introduction
Web search-metrics-tutorial-www2010-section-1of7-introduction
 
The future of mobile and big data
The future of mobile and big dataThe future of mobile and big data
The future of mobile and big data
 
Javier Godar
Javier Godar Javier Godar
Javier Godar
 
Challenging Problems for Scalable Mining of Heterogeneous Social and Informat...
Challenging Problems for Scalable Mining of Heterogeneous Social and Informat...Challenging Problems for Scalable Mining of Heterogeneous Social and Informat...
Challenging Problems for Scalable Mining of Heterogeneous Social and Informat...
 
Graph Realities
Graph RealitiesGraph Realities
Graph Realities
 
Software is eating the world data is digesting it DF2013
Software is eating the world data is digesting it DF2013Software is eating the world data is digesting it DF2013
Software is eating the world data is digesting it DF2013
 
Neo4j GraphTalks Munich - Einführung
Neo4j GraphTalks Munich - EinführungNeo4j GraphTalks Munich - Einführung
Neo4j GraphTalks Munich - Einführung
 

Viewers also liked

Big data mc_05_2014_long
Big data mc_05_2014_longBig data mc_05_2014_long
Big data mc_05_2014_longAxel Poestges
 
Big Data Tutorial - Marko Grobelnik - 25 May 2012
Big Data Tutorial - Marko Grobelnik - 25 May 2012Big Data Tutorial - Marko Grobelnik - 25 May 2012
Big Data Tutorial - Marko Grobelnik - 25 May 2012Marko Grobelnik
 
Big Data in e-Commerce
Big Data in e-CommerceBig Data in e-Commerce
Big Data in e-CommerceDivante
 
How to Spark Creativity, Drive Innovation, and Ensure Sustainability
How to Spark Creativity, Drive Innovation, and Ensure Sustainability How to Spark Creativity, Drive Innovation, and Ensure Sustainability
How to Spark Creativity, Drive Innovation, and Ensure Sustainability Faisal Hoque
 

Viewers also liked (8)

Big data mc_05_2014_long
Big data mc_05_2014_longBig data mc_05_2014_long
Big data mc_05_2014_long
 
Big Data: Kunden auf der Spur
Big Data: Kunden auf der SpurBig Data: Kunden auf der Spur
Big Data: Kunden auf der Spur
 
Big Data Tutorial V4
Big Data Tutorial V4Big Data Tutorial V4
Big Data Tutorial V4
 
Big Data - einfach erklärt!
Big Data - einfach erklärt!Big Data - einfach erklärt!
Big Data - einfach erklärt!
 
Big Data and E-Commerce
Big Data and E-CommerceBig Data and E-Commerce
Big Data and E-Commerce
 
Big Data Tutorial - Marko Grobelnik - 25 May 2012
Big Data Tutorial - Marko Grobelnik - 25 May 2012Big Data Tutorial - Marko Grobelnik - 25 May 2012
Big Data Tutorial - Marko Grobelnik - 25 May 2012
 
Big Data in e-Commerce
Big Data in e-CommerceBig Data in e-Commerce
Big Data in e-Commerce
 
How to Spark Creativity, Drive Innovation, and Ensure Sustainability
How to Spark Creativity, Drive Innovation, and Ensure Sustainability How to Spark Creativity, Drive Innovation, and Ensure Sustainability
How to Spark Creativity, Drive Innovation, and Ensure Sustainability
 

Similar to Big data tutorial_part4

Big data tutorial_part4
Big data tutorial_part4Big data tutorial_part4
Big data tutorial_part4GV prasad
 
Big data tutorial_part4
Big data tutorial_part4Big data tutorial_part4
Big data tutorial_part4Pragati Singh
 
ESWC SS 2012 - Friday Keynote Marko Grobelnik: Big Data Tutorial
ESWC SS 2012 - Friday Keynote Marko Grobelnik: Big Data TutorialESWC SS 2012 - Friday Keynote Marko Grobelnik: Big Data Tutorial
ESWC SS 2012 - Friday Keynote Marko Grobelnik: Big Data Tutorialeswcsummerschool
 
EDF2013: Big Data Tutorial: Marko Grobelnik
EDF2013: Big Data Tutorial: Marko GrobelnikEDF2013: Big Data Tutorial: Marko Grobelnik
EDF2013: Big Data Tutorial: Marko GrobelnikEuropean Data Forum
 
Big Data - Umesh Bellur
Big Data - Umesh BellurBig Data - Umesh Bellur
Big Data - Umesh BellurSTS FORUM 2016
 
MongoDB Days Silicon Valley: Jumpstart: The Right and Wrong Use Cases for Mon...
MongoDB Days Silicon Valley: Jumpstart: The Right and Wrong Use Cases for Mon...MongoDB Days Silicon Valley: Jumpstart: The Right and Wrong Use Cases for Mon...
MongoDB Days Silicon Valley: Jumpstart: The Right and Wrong Use Cases for Mon...MongoDB
 
Big Data By Vijay Bhaskar Semwal
Big Data By Vijay Bhaskar SemwalBig Data By Vijay Bhaskar Semwal
Big Data By Vijay Bhaskar SemwalIIIT Allahabad
 
Big data and Internet
Big data and InternetBig data and Internet
Big data and InternetSanoj Kumar
 
Big Data and Data Science: The Technologies Shaping Our Lives
Big Data and Data Science: The Technologies Shaping Our LivesBig Data and Data Science: The Technologies Shaping Our Lives
Big Data and Data Science: The Technologies Shaping Our LivesRukshan Batuwita
 
Big Data made easy in the era of the Cloud - Demi Ben-Ari
Big Data made easy in the era of the Cloud - Demi Ben-AriBig Data made easy in the era of the Cloud - Demi Ben-Ari
Big Data made easy in the era of the Cloud - Demi Ben-AriDemi Ben-Ari
 
Data analytics introduction
Data analytics introductionData analytics introduction
Data analytics introductionamiyadash
 
AI, ML and Graph Algorithms: Real Life Use Cases with Neo4j
AI, ML and Graph Algorithms: Real Life Use Cases with Neo4jAI, ML and Graph Algorithms: Real Life Use Cases with Neo4j
AI, ML and Graph Algorithms: Real Life Use Cases with Neo4jIvan Zoratti
 

Similar to Big data tutorial_part4 (20)

Big data tutorial_part4
Big data tutorial_part4Big data tutorial_part4
Big data tutorial_part4
 
Big data tutorial_part4
Big data tutorial_part4Big data tutorial_part4
Big data tutorial_part4
 
Big data tutorial_part4
Big data tutorial_part4Big data tutorial_part4
Big data tutorial_part4
 
Big data tutorial
Big data tutorialBig data tutorial
Big data tutorial
 
ESWC SS 2012 - Friday Keynote Marko Grobelnik: Big Data Tutorial
ESWC SS 2012 - Friday Keynote Marko Grobelnik: Big Data TutorialESWC SS 2012 - Friday Keynote Marko Grobelnik: Big Data Tutorial
ESWC SS 2012 - Friday Keynote Marko Grobelnik: Big Data Tutorial
 
EDF2013: Big Data Tutorial: Marko Grobelnik
EDF2013: Big Data Tutorial: Marko GrobelnikEDF2013: Big Data Tutorial: Marko Grobelnik
EDF2013: Big Data Tutorial: Marko Grobelnik
 
Big Data - Umesh Bellur
Big Data - Umesh BellurBig Data - Umesh Bellur
Big Data - Umesh Bellur
 
Big data business case
Big data   business caseBig data   business case
Big data business case
 
Big data
Big dataBig data
Big data
 
MongoDB Days Silicon Valley: Jumpstart: The Right and Wrong Use Cases for Mon...
MongoDB Days Silicon Valley: Jumpstart: The Right and Wrong Use Cases for Mon...MongoDB Days Silicon Valley: Jumpstart: The Right and Wrong Use Cases for Mon...
MongoDB Days Silicon Valley: Jumpstart: The Right and Wrong Use Cases for Mon...
 
SKILLWISE-BIGDATA ANALYSIS
SKILLWISE-BIGDATA ANALYSISSKILLWISE-BIGDATA ANALYSIS
SKILLWISE-BIGDATA ANALYSIS
 
Big Data By Vijay Bhaskar Semwal
Big Data By Vijay Bhaskar SemwalBig Data By Vijay Bhaskar Semwal
Big Data By Vijay Bhaskar Semwal
 
Big data and Internet
Big data and InternetBig data and Internet
Big data and Internet
 
Big Data et eGovernment
Big Data et eGovernmentBig Data et eGovernment
Big Data et eGovernment
 
Big Data and Data Science: The Technologies Shaping Our Lives
Big Data and Data Science: The Technologies Shaping Our LivesBig Data and Data Science: The Technologies Shaping Our Lives
Big Data and Data Science: The Technologies Shaping Our Lives
 
Big Data made easy in the era of the Cloud - Demi Ben-Ari
Big Data made easy in the era of the Cloud - Demi Ben-AriBig Data made easy in the era of the Cloud - Demi Ben-Ari
Big Data made easy in the era of the Cloud - Demi Ben-Ari
 
Big Data World
Big Data WorldBig Data World
Big Data World
 
Internet of Things
Internet of ThingsInternet of Things
Internet of Things
 
Data analytics introduction
Data analytics introductionData analytics introduction
Data analytics introduction
 
AI, ML and Graph Algorithms: Real Life Use Cases with Neo4j
AI, ML and Graph Algorithms: Real Life Use Cases with Neo4jAI, ML and Graph Algorithms: Real Life Use Cases with Neo4j
AI, ML and Graph Algorithms: Real Life Use Cases with Neo4j
 

Recently uploaded

Non Text Magic Studio Magic Design for Presentations L&P.pdf
Non Text Magic Studio Magic Design for Presentations L&P.pdfNon Text Magic Studio Magic Design for Presentations L&P.pdf
Non Text Magic Studio Magic Design for Presentations L&P.pdfPratikPatil591646
 
Role of Consumer Insights in business transformation
Role of Consumer Insights in business transformationRole of Consumer Insights in business transformation
Role of Consumer Insights in business transformationAnnie Melnic
 
Statistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfStatistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfnikeshsingh56
 
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Boston Institute of Analytics
 
DATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etcDATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etclalithasri22
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBoston Institute of Analytics
 
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...Jack Cole
 
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfEnglish-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfblazblazml
 
Presentation of project of business person who are success
Presentation of project of business person who are successPresentation of project of business person who are success
Presentation of project of business person who are successPratikSingh115843
 
Digital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfDigital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfNicoChristianSunaryo
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaManalVerma4
 
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelDecoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelBoston Institute of Analytics
 
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...Dr Arash Najmaei ( Phd., MBA, BSc)
 
Digital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksDigital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksdeepakthakur548787
 

Recently uploaded (17)

Non Text Magic Studio Magic Design for Presentations L&P.pdf
Non Text Magic Studio Magic Design for Presentations L&P.pdfNon Text Magic Studio Magic Design for Presentations L&P.pdf
Non Text Magic Studio Magic Design for Presentations L&P.pdf
 
Role of Consumer Insights in business transformation
Role of Consumer Insights in business transformationRole of Consumer Insights in business transformation
Role of Consumer Insights in business transformation
 
Statistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdfStatistics For Management by Richard I. Levin 8ed.pdf
Statistics For Management by Richard I. Levin 8ed.pdf
 
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
 
DATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etcDATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etc
 
2023 Survey Shows Dip in High School E-Cigarette Use
2023 Survey Shows Dip in High School E-Cigarette Use2023 Survey Shows Dip in High School E-Cigarette Use
2023 Survey Shows Dip in High School E-Cigarette Use
 
Data Analysis Project: Stroke Prediction
Data Analysis Project: Stroke PredictionData Analysis Project: Stroke Prediction
Data Analysis Project: Stroke Prediction
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
 
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
 
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfEnglish-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
 
Presentation of project of business person who are success
Presentation of project of business person who are successPresentation of project of business person who are success
Presentation of project of business person who are success
 
Digital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfDigital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdf
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in India
 
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelDecoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
 
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...
 
Insurance Churn Prediction Data Analysis Project
Insurance Churn Prediction Data Analysis ProjectInsurance Churn Prediction Data Analysis Project
Insurance Churn Prediction Data Analysis Project
 
Digital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksDigital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing works
 

Big data tutorial_part4

  • 1. Marko Grobelnik marko.grobelnik@ijs.si Jozef Stefan Institute Ljubljana, Slovenia Stavanger, May 8th 2012
  • 2.  Introduction ◦ What is Big data? ◦ Why Big-Data? ◦ When Big-Data is really a problem?  Techniques  Tools  Applications  Literature
  • 3.
  • 4.
  • 5.  ‘Big-data’ is similar to ‘Small-data’, but bigger  …but having data bigger consequently requires different approaches: ◦ techniques, tools & architectures  …to solve: ◦ New problems… ◦ …and old problems in a better way.
  • 6. From “Understanding Big Data” by IBM
  • 7.
  • 9.  Key enablers for the growth of “Big Data” are: ◦ Increase of storage capacities ◦ Increase of processing power ◦ Availability of data
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.  NoSQL ◦ DatabasesMongoDB, CouchDB, Cassandra, Redis, BigTable, Hbase, Hypertable, Voldemort, Riak, ZooKeeper  MapReduce ◦ Hadoop, Hive, Pig, Cascading, Cascalog, mrjob, Caffeine, S4, MapR, Acunu, Flume, Kafka, Azkaban, Oozie, Greenplum  Storage ◦ S3, Hadoop Distributed File System  Servers ◦ EC2, Google App Engine, Elastic, Beanstalk, Heroku  Processing ◦ R, Yahoo! Pipes, Mechanical Turk, Solr/Lucene, ElasticSearch, Datameer, BigSheets, Tinkerpop
  • 21.
  • 22.  …when the operations on data are complex: ◦ …e.g. simple counting is not a complex problem ◦ Modeling and reasoning with data of different kinds can get extremely complex  Good news about big-data: ◦ Often, because of vast amount of data, modeling techniques can get simpler (e.g. smart counting can replace complex model based analytics)… ◦ …as long as we deal with the scale
  • 23.  Research areas (such as IR, KDD, ML, NLP, SemWeb, …) are sub- cubes within the data cube Scalability Dynamicity Context Quality Usage
  • 24.
  • 25.
  • 26.  Good recommendations can make a big difference when keeping a user on a web site ◦ …the key is how rich context model a system is using to select information for a user ◦ Bad recommendations <1% users, good ones >5% users click Contextual personalized recommendations generated in ~20ms
  • 27.  Domain  Sub-domain  Page URL  URL sub-directories  Page Meta Tags  Page Title  Page Content  Named Entities  Has Query  Referrer Query  Referring Domain  Referring URL  Outgoing URL  GeoIP Country  GeoIP State  GeoIP City  Absolute Date  Day of the Week  Day period  Hour of the day  User Agent  Zip Code  State  Income  Age  Gender  Country  Job Title  Job Industry
  • 28. Log Files (~100M page clicks per day) User profiles NYT articles Stream of profiles Advertisers Segment Keywords Stock Market Stock Market, mortgage, banking, investors, Wall Street, turmoil, New York Stock Exchange Health diabetes, heart disease, disease, heart, illness Green Energy Hybrid cars, energy, power, model, carbonated, fuel, bulbs, Hybrid cars Hybrid cars, vehicles, model, engines, diesel Travel travel, wine, opening, tickets, hotel, sites, cars, search, restaurant … … Segments Trend Detection System Stream of clicks Trends and updated segments Campaign to sell segments $ Sales
  • 29.  50Gb of uncompressed log files  10Gb of compressed log files  0.5Gb of processed log files  50-100M clicks  4-6M unique users  7000 unique pages with more then 100 hits  Index size 2Gb  Pre-processing & indexing time ◦ ~10min on workstation (4 cores & 32Gb) ◦ ~1hour on EC2 (2 cores & 16Gb)
  • 30.
  • 31.  Alarms Explorer Server implements three real-time scenarios on the alarms stream: 1. Root-Cause-Analysis – finding which device is responsible for occasional “flood” of alarms 2. Short-Term Fault Prediction – predict which device will fail in next 15mins 3. Long-Term Anomaly Detection – detect unusual trends in the network  …system is used in British Telecom Alarms Server Alarms Explorer Server Live feed of data Operator Big board display Telecom Network (~25 000 devices) Alarms ~10-100/sec
  • 32.  Presented in “Planetary-Scale Views on a Large Instant-Messaging Network” by Jure Leskovec and Eric Horvitz WWW2008
  • 33.  Observe social and communication phenomena at a planetary scale  Largest social network analyzed to date Research questions: How does communication change with user demographics (age, sex, language, country)? How does geography affect communication? What is the structure of the communication network? 33
  • 34.  We collected the data for June 2006  Log size: 150Gb/day (compressed)  Total: 1 month of communication data: 4.5Tb of compressed data  Activity over June 2006 (30 days) ◦ 245 million users logged in ◦ 180 million users engaged in conversations ◦ 17,5 million new accounts activated ◦ More than 30 billion conversations ◦ More than 255 billion exchanged messages 34
  • 35. 35
  • 36. 36
  • 37.  Count the number of users logging in from particular location on the earth 37
  • 38.  Logins from Europe 38
  • 39.  6 degrees of separation [Milgram ’60s]  Average distance between two random users is 6.6  90% of nodes can be reached in < 8 hops Hops Nodes 1 10 2 78 3 396 4 8648 5 3299252 6 28395849 7 79059497 8 52995778 9 10321008 10 1955007 11 518410 12 149945 13 44616 14 13740 15 4476 16 1542 17 536 18 167 19 71 20 29 21 16 22 10 23 3 24 2 25 3