SlideShare a Scribd company logo
1 of 59
Marko Grobelnik
marko.grobelnik@ijs.si
 Jozef Stefan Institute




                     Dublin, April 9th 2013
   Introduction
    ◦ What is Big data?
    ◦ Why Big-Data?
    ◦ When Big-Data is really a problem?
   Market Overview
   Tools
   Techniques
   Applications
   Literature
   ‘Big-data’ is similar to ‘Small-data’, but bigger

   …but having data bigger it requires different
    approaches:
    ◦ techniques, tools, architectures


   …with an aim to solve new problems
    ◦ …or old problems in a better way.
From “Understanding Big Data” by IBM
Big-Data
   Key enablers for the appearance and growth
    of “Big Data” are:

    ◦ Increase of storage capacities

    ◦ Increase of processing power

    ◦ Availability of data
Source: WikiBon report on “Big Data Vendor Revenue and Market Forecast 2012-2017”, 2013
   Where processing is hosted?
    ◦ Distributed Servers / Cloud (e.g. Amazon EC2)
   Where data is stored?
    ◦ Distributed Storage (e.g. Amazon S3)
   What is the programming model?
    ◦ Distributed Processing (e.g. MapReduce)
   How data is stored & indexed?
    ◦ High-performance schema-free databases (e.g.
      MongoDB)
   What operations are performed on data?
    ◦ Analytic / Semantic Processing (e.g. R, OWLIM)
   Computing and storage are typically hosted
    transparently on cloud infrastructures
    ◦ …providing scale, flexibility and high fail-safety


   Distributed Servers
    ◦ Amazon-EC2, Google App Engine, Elastic,
      Beanstalk, Heroku
   Distributed Storage
    ◦ Amazon-S3, Hadoop Distributed File System
   Distributed processing of Big-Data requires non-
    standard programming models
    ◦ …beyond single machines or traditional parallel
      programming models (like MPI)
    ◦ …the aim is to simplify complex programming tasks

   The most popular programming model is
    MapReduce approach

   Implementations of MapReduce
    ◦ Hadoop (http://hadoop.apache.org/), Hive, Pig,
      Cascading, Cascalog, mrjob, Caffeine, S4, MapR, Acunu,
      Flume, Kafka, Azkaban, Oozie, Greenplum
   The key idea of the MapReduce approach:
    ◦ A target problem needs to be parallelizable

    ◦ First, the problem gets split into a set of smaller problems (Map step)
    ◦ Next, smaller problems are solved in a parallel way
    ◦ Finally, a set of solutions to the smaller problems get synthesized
      into a solution of the original problem (Reduce step)
   NoSQL class of databases have in common:
    ◦   To support large amounts of data
    ◦   Have mostly non-SQL interface
    ◦   Operate on distributed infrastructures (e.g. Hadoop)
    ◦   Are based on key-value pairs (no predefined schema)
    ◦   …are flexible and fast
   Implementations
    ◦ MongoDB, CouchDB, Cassandra, Redis, BigTable, Hbase,
      Hypertable, Voldemort, Riak, ZooKeeper…
   …when the operations on data are complex:
    ◦ e.g. simple counting is not a complex problem
    ◦ Modeling and reasoning with data of different kinds
      can get extremely complex

   Good news about big-data:
    ◦ Often, because of vast amount of data, modeling
      techniques can get simpler (e.g. smart counting can
      replace complex model-based analytics)…
    ◦ …as long as we deal with the scale
   Research areas (such
    as IR, KDD, ML, NLP,
                            Usage
    SemWeb, …) are sub-
    cubes within the data   Quality
    cube
                            Context

                            Streaming

                            Scalability
   A risk with “Big-Data mining” is that an
    analyst can “discover” patterns that are
    meaningless
   Statisticians call it Bonferroni’s principle:
    ◦ Roughly, if you look in more places for interesting
      patterns, than your amount of data will support
      almost anything, …and you are bound to find lots
      of nonsense




                    Example taken from: Rajaraman, Ullman: Mining of Massive Datasets
Example:
 We want to find (unrelated) people who at least twice
  have stayed at the same hotel on the same day
    ◦   109 people being tracked.
    ◦   1000 days.
    ◦   Each person stays in a hotel 1% of the time (1 day out of 100)
    ◦   Hotels hold 100 people (so 105 hotels).
    ◦   If everyone behaves randomly (i.e., no terrorists) will the data
        mining detect anything suspicious?
   Expected number of “suspicious” pairs of people:
    ◦ 250,000
    ◦ … too many combinations to check – we need to have some
      additional evidence to find “suspicious” pairs of people in
      some more efficient way


                           Example taken from: Rajaraman, Ullman: Mining of Massive Datasets
   Smart sampling of data
    ◦ …reducing the original data while not losing the
      statistical properties of data
   Finding similar items
    ◦ …efficient multidimensional indexing
   Incremental updating of the models
    ◦ (vs. building models from scratch)
    ◦ …crucial for streaming data
   Distributed linear algebra
    ◦ …dealing with large sparse matrices
   On the top of the previous ops we perform
    usual data mining/machine learning/statistics
    operators:
    ◦ Supervised learning (classification, regression, …)
    ◦ Non-supervised learning (clustering, different types
      of decompositions, …)
    ◦ …


   …we are just more careful which algorithms
    we choose (typically linear or sub-linear
    versions)
   An excellent overview of the algorithms
    covering the above issues is the book
    “Rajaraman, Leskovec, Ullman: Mining of
    Massive Datasets”
   Good recommendations
    can make a big
    difference when keeping
    a user on a web site
    ◦ …the key is how rich the
      context model a system is
      using to select information
      for a user
    ◦ Bad recommendations <1%
      users, good ones >5% users
      click
    ◦ 200clicks/sec

                      Contextual
                     personalized
                  recommendations
                 generated in ~20ms
   Domain                   Referring Domain      Zip Code
   Sub-domain               Referring URL         State
   Page URL                 Outgoing URL          Income
   URL sub-directories                             Age
                             GeoIP Country         Gender
   Page Meta Tags           GeoIP State           Country
   Page Title               GeoIP City            Job Title
   Page Content                                    Job Industry
   Named Entities           Absolute Date
                             Day of the Week
   Has Query                Day period
   Referrer Query           Hour of the day
                             User Agent
Trend Detection System

                            User                   Stream of
 Log Files     Stream
              of clicks    profiles                 profiles
  (~100M
page clicks
 per day)


                                                                                                       Sales
                          Trends and
                          updated segments                                          Segments
                          Segment       Keywords

        NYT               Stock         Stock Market, mortgage, banking,
                          Market        investors, Wall Street, turmoil, New
       articles                         York Stock Exchange
                                                                                                Campaign
                          Health        diabetes, heart disease, disease, heart,
                                        illness                                                   to sell
                                                                                                segments
                                                                                      $
                          Green         Hybrid cars, energy, power, model,
                          Energy        carbonated, fuel, bulbs,

                          Hybrid cars   Hybrid cars, vehicles, model, engines,
                                        diesel

                          Travel        travel, wine, opening, tickets, hotel,
                                        sites, cars, search, restaurant


                                                                                               Advertisers
                          …             …
   50Gb of uncompressed log files
   50-100M clicks
   4-6M unique users
   7000 unique pages with more then 100 hits
Alarms Server

      Telecom
      Network                                                          Alarms
                   Alarms                       Live feed of data      Explorer
     (~25 000     ~10-100/sec
      devices)                                                         Server


   Alarms Explorer Server implements three
    real-time scenarios on the alarms stream:
    1. Root-Cause-Analysis – finding which device is
       responsible for occasional “flood” of alarms
    2. Short-Term Fault Prediction – predict which
       device will fail in next 15mins
    3. Long-Term Anomaly Detection – detect
       unusual trends in the network
   …system is used in British Telecom


                                                       Operator     Big board display
   The aim is to use analytic techniques to
    visualize documents in different ways:
    ◦ Topic view
    ◦ Social view
    ◦ Temporal view
Query

Search
Results

 Topic Map


Selected
group of news




 Selected
 story
Query




Named
entities
in relation
US Elections
                               US Budget
  Query


 Result set

                    NATO-Russia
Topic Trends
Visualization
                               Mid-East
                               conflict



 Topics
 description
Dec 7th 1941
Apr 6th 1941
June 1944
Query
Conceptual map

Search Point


 Dynamic
 contextual
 ranking based
 on the search
 point
   Observe social and communication
     phenomena at a planetary scale
    Largest social network analyzed till 2010

 Research questions:
  How does communication change with user
   demographics (age, sex, language, country)?
  How does geography affect communication?
  What is the structure of the communication
   network?

“Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008
                                                                                          51
   We collected the data for June 2006
     Log size:
         150Gb/day (compressed)
     Total: 1 month of communication data:
         4.5Tb of compressed data
     Activity over June 2006 (30 days)
      ◦   245 million users logged in
      ◦   180 million users engaged in conversations
      ◦   17,5 million new accounts activated
      ◦   More than 30 billion conversations
      ◦   More than 255 billion exchanged messages
“Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008
                                                                                          52
“Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008   53
“Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008   54
   Count the number of users logging in from
     particular location on the earth
“Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008
                                                                                          55
   Logins from Europe




“Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008   56
Hops     Nodes
                                                                                              1         10
                                                                                              2         78
                                                                                              3        396
                                                                                              4       8648
                                                                                              5     3299252
                                                                                              6    28395849
                                                                                              7    79059497
                                                                                              8    52995778
                                                                                              9    10321008
                                                                                              10    1955007
                                                                                              11    518410
                                                                                              12    149945
                                                                                              13     44616
                                                                                              14     13740
                                                                                              15      4476
                                                                                              16      1542
                                                                                              17       536
                                                                                              18       167
                                                                                              19        71

   6 degrees of separation [Milgram ’60s]          20                                                  29

    Average distance between two random users is 6.622
                                                    21                                                  16
                                                                                                       10
   90% of nodes can be reached in < 8 hops         23                                                   3
                                                                                              24         2
    “Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008   25         3
   Big-Data is everywhere, we are just not used to
    deal with it

   The “Big-Data” hype is very recent
    ◦ …growth seems to be going up
    ◦ …evident lack of experts to build Big-Data apps

   Can we do “Big-Data” without big investment?
    ◦ …yes – many open source tools, computing machinery is
      cheap (to buy or to rent)
    ◦ …the key is knowledge on how to deal with data
    ◦ …data is either free (e.g. Wikipedia) or to buy (e.g.
      twitter)

More Related Content

What's hot

Introduction to machine learning
Introduction to machine learningIntroduction to machine learning
Introduction to machine learningPruet Boonma
 
[Webinar] How Big Data and Machine Learning Are Transforming ITSM
[Webinar] How Big Data and Machine Learning Are Transforming ITSM[Webinar] How Big Data and Machine Learning Are Transforming ITSM
[Webinar] How Big Data and Machine Learning Are Transforming ITSMSunView Software, Inc.
 
Big Data and Data Science: The Technologies Shaping Our Lives
Big Data and Data Science: The Technologies Shaping Our LivesBig Data and Data Science: The Technologies Shaping Our Lives
Big Data and Data Science: The Technologies Shaping Our LivesRukshan Batuwita
 
Machine Learning in Big Data
Machine Learning in Big DataMachine Learning in Big Data
Machine Learning in Big DataDataWorks Summit
 
Machine Learning Introduction for Digital Business Leaders
Machine Learning Introduction for Digital Business LeadersMachine Learning Introduction for Digital Business Leaders
Machine Learning Introduction for Digital Business LeadersSudha Jamthe
 
Introduction to Machine Learning
Introduction to Machine LearningIntroduction to Machine Learning
Introduction to Machine LearningRaveen Perera
 
Machine learning with Big Data power point presentation
Machine learning with Big Data power point presentationMachine learning with Big Data power point presentation
Machine learning with Big Data power point presentationDavid Raj Kanthi
 
Applications of Machine Learning at USC
Applications of Machine Learning at USCApplications of Machine Learning at USC
Applications of Machine Learning at USCSri Ambati
 
Big Data & Machine Learning - TDC2013 Sao Paulo
Big Data & Machine Learning - TDC2013 Sao PauloBig Data & Machine Learning - TDC2013 Sao Paulo
Big Data & Machine Learning - TDC2013 Sao PauloOCTO Technology
 
Introduction to Machine learning
Introduction to Machine learningIntroduction to Machine learning
Introduction to Machine learningKnoldus Inc.
 
Training in Analytics and Data Science
Training in Analytics and Data ScienceTraining in Analytics and Data Science
Training in Analytics and Data ScienceAjay Ohri
 
Introduction to machine learning and deep learning
Introduction to machine learning and deep learningIntroduction to machine learning and deep learning
Introduction to machine learning and deep learningShishir Choudhary
 
Data science presentation 2nd CI day
Data science presentation 2nd CI dayData science presentation 2nd CI day
Data science presentation 2nd CI dayMohammed Barakat
 
10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning Systems10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning SystemsXavier Amatriain
 
Introduction To Data Science
Introduction To Data ScienceIntroduction To Data Science
Introduction To Data ScienceSpotle.ai
 
GTU GeekDay Data Science and Applications
GTU GeekDay Data Science and ApplicationsGTU GeekDay Data Science and Applications
GTU GeekDay Data Science and ApplicationsKürşat İNCE
 
Data science | What is Data science
Data science | What is Data scienceData science | What is Data science
Data science | What is Data scienceShilpaKrishna6
 
Data Science, Machine Learning and Neural Networks
Data Science, Machine Learning and Neural NetworksData Science, Machine Learning and Neural Networks
Data Science, Machine Learning and Neural NetworksBICA Labs
 

What's hot (20)

Introduction to machine learning
Introduction to machine learningIntroduction to machine learning
Introduction to machine learning
 
[Webinar] How Big Data and Machine Learning Are Transforming ITSM
[Webinar] How Big Data and Machine Learning Are Transforming ITSM[Webinar] How Big Data and Machine Learning Are Transforming ITSM
[Webinar] How Big Data and Machine Learning Are Transforming ITSM
 
Big Data and Data Science: The Technologies Shaping Our Lives
Big Data and Data Science: The Technologies Shaping Our LivesBig Data and Data Science: The Technologies Shaping Our Lives
Big Data and Data Science: The Technologies Shaping Our Lives
 
Machine Learning in Big Data
Machine Learning in Big DataMachine Learning in Big Data
Machine Learning in Big Data
 
Machine Learning Introduction for Digital Business Leaders
Machine Learning Introduction for Digital Business LeadersMachine Learning Introduction for Digital Business Leaders
Machine Learning Introduction for Digital Business Leaders
 
Introduction to Machine Learning
Introduction to Machine LearningIntroduction to Machine Learning
Introduction to Machine Learning
 
Machine learning with Big Data power point presentation
Machine learning with Big Data power point presentationMachine learning with Big Data power point presentation
Machine learning with Big Data power point presentation
 
Applications of Machine Learning at USC
Applications of Machine Learning at USCApplications of Machine Learning at USC
Applications of Machine Learning at USC
 
Big Data & Machine Learning - TDC2013 Sao Paulo
Big Data & Machine Learning - TDC2013 Sao PauloBig Data & Machine Learning - TDC2013 Sao Paulo
Big Data & Machine Learning - TDC2013 Sao Paulo
 
Introduction to Machine learning
Introduction to Machine learningIntroduction to Machine learning
Introduction to Machine learning
 
Training in Analytics and Data Science
Training in Analytics and Data ScienceTraining in Analytics and Data Science
Training in Analytics and Data Science
 
Introduction to machine learning and deep learning
Introduction to machine learning and deep learningIntroduction to machine learning and deep learning
Introduction to machine learning and deep learning
 
Data science presentation 2nd CI day
Data science presentation 2nd CI dayData science presentation 2nd CI day
Data science presentation 2nd CI day
 
10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning Systems10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning Systems
 
Managing machine learning
Managing machine learningManaging machine learning
Managing machine learning
 
Introduction To Data Science
Introduction To Data ScienceIntroduction To Data Science
Introduction To Data Science
 
Unit 3 part 2
Unit  3 part 2Unit  3 part 2
Unit 3 part 2
 
GTU GeekDay Data Science and Applications
GTU GeekDay Data Science and ApplicationsGTU GeekDay Data Science and Applications
GTU GeekDay Data Science and Applications
 
Data science | What is Data science
Data science | What is Data scienceData science | What is Data science
Data science | What is Data science
 
Data Science, Machine Learning and Neural Networks
Data Science, Machine Learning and Neural NetworksData Science, Machine Learning and Neural Networks
Data Science, Machine Learning and Neural Networks
 

Viewers also liked

Dynamic Models with Django
Dynamic Models with DjangoDynamic Models with Django
Dynamic Models with Djangoschacki
 
Semantic Wiki For The Enterprise
Semantic Wiki For The EnterpriseSemantic Wiki For The Enterprise
Semantic Wiki For The EnterpriseJosef Holy
 
Social Media: You Can't Measure If You Don't Plan
Social Media: You Can't Measure If You Don't PlanSocial Media: You Can't Measure If You Don't Plan
Social Media: You Can't Measure If You Don't PlanDerek Beere
 
Secret to Successful Financial Management TASN
Secret to Successful Financial Management TASNSecret to Successful Financial Management TASN
Secret to Successful Financial Management TASNPrimeroEdge
 
An Introduction of Big data; Big data for beginners; Overview of Big Data; Bi...
An Introduction of Big data; Big data for beginners; Overview of Big Data; Bi...An Introduction of Big data; Big data for beginners; Overview of Big Data; Bi...
An Introduction of Big data; Big data for beginners; Overview of Big Data; Bi...SpringPeople
 
How to Create a Web Analytics Measurement Plan - Strategy
How to Create a Web Analytics Measurement Plan - StrategyHow to Create a Web Analytics Measurement Plan - Strategy
How to Create a Web Analytics Measurement Plan - StrategyRyan Stewart
 
Automobile Industry - Benchmarking
Automobile Industry - BenchmarkingAutomobile Industry - Benchmarking
Automobile Industry - Benchmarkingdigiqom
 
Trends in Successful Financial Advisor Business Practices
Trends in Successful Financial Advisor Business Practices Trends in Successful Financial Advisor Business Practices
Trends in Successful Financial Advisor Business Practices Maximizer Software
 
Building a Digital Measurement Plan - PSE Web 2013
Building a Digital Measurement Plan - PSE Web 2013Building a Digital Measurement Plan - PSE Web 2013
Building a Digital Measurement Plan - PSE Web 2013ISL Digital Marketing
 
Admin social media management training mos training
Admin social media management training mos trainingAdmin social media management training mos training
Admin social media management training mos trainingDonna Gilliland
 
Big Data Benchmarking Tutorial
Big Data Benchmarking TutorialBig Data Benchmarking Tutorial
Big Data Benchmarking TutorialTilmann Rabl
 
AWS re:Invent 2016: Transforming Industrial Processes with Deep Learning (MAC...
AWS re:Invent 2016: Transforming Industrial Processes with Deep Learning (MAC...AWS re:Invent 2016: Transforming Industrial Processes with Deep Learning (MAC...
AWS re:Invent 2016: Transforming Industrial Processes with Deep Learning (MAC...Amazon Web Services
 
12 analisa kelayakan proyek
12 analisa kelayakan proyek12 analisa kelayakan proyek
12 analisa kelayakan proyekSimon Patabang
 
03 bisnis plan dried fruit
03 bisnis plan dried fruit03 bisnis plan dried fruit
03 bisnis plan dried fruitstiemberau2
 
Introduction to Machine Learning
Introduction to Machine LearningIntroduction to Machine Learning
Introduction to Machine LearningLior Rokach
 
Financial Planning presentation
Financial Planning presentationFinancial Planning presentation
Financial Planning presentationjhumur_sinha
 

Viewers also liked (20)

Dynamic Models with Django
Dynamic Models with DjangoDynamic Models with Django
Dynamic Models with Django
 
Semantic Wiki For The Enterprise
Semantic Wiki For The EnterpriseSemantic Wiki For The Enterprise
Semantic Wiki For The Enterprise
 
Social Media: You Can't Measure If You Don't Plan
Social Media: You Can't Measure If You Don't PlanSocial Media: You Can't Measure If You Don't Plan
Social Media: You Can't Measure If You Don't Plan
 
Secret to Successful Financial Management TASN
Secret to Successful Financial Management TASNSecret to Successful Financial Management TASN
Secret to Successful Financial Management TASN
 
An Introduction of Big data; Big data for beginners; Overview of Big Data; Bi...
An Introduction of Big data; Big data for beginners; Overview of Big Data; Bi...An Introduction of Big data; Big data for beginners; Overview of Big Data; Bi...
An Introduction of Big data; Big data for beginners; Overview of Big Data; Bi...
 
How to Create a Web Analytics Measurement Plan - Strategy
How to Create a Web Analytics Measurement Plan - StrategyHow to Create a Web Analytics Measurement Plan - Strategy
How to Create a Web Analytics Measurement Plan - Strategy
 
Automobile Industry - Benchmarking
Automobile Industry - BenchmarkingAutomobile Industry - Benchmarking
Automobile Industry - Benchmarking
 
Trends in Successful Financial Advisor Business Practices
Trends in Successful Financial Advisor Business Practices Trends in Successful Financial Advisor Business Practices
Trends in Successful Financial Advisor Business Practices
 
Web Analytics 2
Web Analytics 2Web Analytics 2
Web Analytics 2
 
Web Analytics 3
Web Analytics 3Web Analytics 3
Web Analytics 3
 
Building a Digital Measurement Plan - PSE Web 2013
Building a Digital Measurement Plan - PSE Web 2013Building a Digital Measurement Plan - PSE Web 2013
Building a Digital Measurement Plan - PSE Web 2013
 
Admin social media management training mos training
Admin social media management training mos trainingAdmin social media management training mos training
Admin social media management training mos training
 
Big Data Benchmarking Tutorial
Big Data Benchmarking TutorialBig Data Benchmarking Tutorial
Big Data Benchmarking Tutorial
 
AWS re:Invent 2016: Transforming Industrial Processes with Deep Learning (MAC...
AWS re:Invent 2016: Transforming Industrial Processes with Deep Learning (MAC...AWS re:Invent 2016: Transforming Industrial Processes with Deep Learning (MAC...
AWS re:Invent 2016: Transforming Industrial Processes with Deep Learning (MAC...
 
How to Create a Business Plan
How to Create a Business PlanHow to Create a Business Plan
How to Create a Business Plan
 
12 analisa kelayakan proyek
12 analisa kelayakan proyek12 analisa kelayakan proyek
12 analisa kelayakan proyek
 
03 bisnis plan dried fruit
03 bisnis plan dried fruit03 bisnis plan dried fruit
03 bisnis plan dried fruit
 
Big Data Tutorial V4
Big Data Tutorial V4Big Data Tutorial V4
Big Data Tutorial V4
 
Introduction to Machine Learning
Introduction to Machine LearningIntroduction to Machine Learning
Introduction to Machine Learning
 
Financial Planning presentation
Financial Planning presentationFinancial Planning presentation
Financial Planning presentation
 

Similar to EDF2013: Big Data Tutorial: Marko Grobelnik

Big Data Tutorial - Marko Grobelnik - 25 May 2012
Big Data Tutorial - Marko Grobelnik - 25 May 2012Big Data Tutorial - Marko Grobelnik - 25 May 2012
Big Data Tutorial - Marko Grobelnik - 25 May 2012Marko Grobelnik
 
ESWC SS 2012 - Friday Keynote Marko Grobelnik: Big Data Tutorial
ESWC SS 2012 - Friday Keynote Marko Grobelnik: Big Data TutorialESWC SS 2012 - Friday Keynote Marko Grobelnik: Big Data Tutorial
ESWC SS 2012 - Friday Keynote Marko Grobelnik: Big Data Tutorialeswcsummerschool
 
Big data tutorial_part4
Big data tutorial_part4Big data tutorial_part4
Big data tutorial_part4heyramzz
 
Big data tutorial_part4
Big data tutorial_part4Big data tutorial_part4
Big data tutorial_part4GV prasad
 
Big data tutorial_part4
Big data tutorial_part4Big data tutorial_part4
Big data tutorial_part4Pragati Singh
 
The information supernova
The information supernovaThe information supernova
The information supernovaAlaa Al-Agamawi
 
Big data-analytics-changing-way-organizations-conducting-business
Big data-analytics-changing-way-organizations-conducting-businessBig data-analytics-changing-way-organizations-conducting-business
Big data-analytics-changing-way-organizations-conducting-businessAmit Bhargava
 
Big data in marketing at harvard business club nick1 june 15 2013
Big data in marketing at harvard business club nick1 june 15 2013Big data in marketing at harvard business club nick1 june 15 2013
Big data in marketing at harvard business club nick1 june 15 2013nkabra
 
Data Mining: Future Trends and Applications
Data Mining: Future Trends and ApplicationsData Mining: Future Trends and Applications
Data Mining: Future Trends and ApplicationsIJMER
 
Introduction To Data Mining
Introduction To Data MiningIntroduction To Data Mining
Introduction To Data Miningdataminers.ir
 
Introduction To Data Mining
Introduction To Data Mining   Introduction To Data Mining
Introduction To Data Mining Phi Jack
 
Data Mining and Knowledge Discovery in Large Databases
Data Mining and Knowledge Discovery in Large DatabasesData Mining and Knowledge Discovery in Large Databases
Data Mining and Knowledge Discovery in Large DatabasesSSA KPI
 
Big data unit 2
Big data unit 2Big data unit 2
Big data unit 2RojaT4
 
The AI Platform Business Revolution: Matchmaking, Empathetic Technology, and ...
The AI Platform Business Revolution: Matchmaking, Empathetic Technology, and ...The AI Platform Business Revolution: Matchmaking, Empathetic Technology, and ...
The AI Platform Business Revolution: Matchmaking, Empathetic Technology, and ...Steve Omohundro
 

Similar to EDF2013: Big Data Tutorial: Marko Grobelnik (20)

Big Data Tutorial - Marko Grobelnik - 25 May 2012
Big Data Tutorial - Marko Grobelnik - 25 May 2012Big Data Tutorial - Marko Grobelnik - 25 May 2012
Big Data Tutorial - Marko Grobelnik - 25 May 2012
 
ESWC SS 2012 - Friday Keynote Marko Grobelnik: Big Data Tutorial
ESWC SS 2012 - Friday Keynote Marko Grobelnik: Big Data TutorialESWC SS 2012 - Friday Keynote Marko Grobelnik: Big Data Tutorial
ESWC SS 2012 - Friday Keynote Marko Grobelnik: Big Data Tutorial
 
Big data tutorial_part4
Big data tutorial_part4Big data tutorial_part4
Big data tutorial_part4
 
Big data tutorial_part4
Big data tutorial_part4Big data tutorial_part4
Big data tutorial_part4
 
Big data tutorial_part4
Big data tutorial_part4Big data tutorial_part4
Big data tutorial_part4
 
Big data tutorial_part4
Big data tutorial_part4Big data tutorial_part4
Big data tutorial_part4
 
Big data tutorial
Big data tutorialBig data tutorial
Big data tutorial
 
The information supernova
The information supernovaThe information supernova
The information supernova
 
Big data-analytics-changing-way-organizations-conducting-business
Big data-analytics-changing-way-organizations-conducting-businessBig data-analytics-changing-way-organizations-conducting-business
Big data-analytics-changing-way-organizations-conducting-business
 
Big data in marketing at harvard business club nick1 june 15 2013
Big data in marketing at harvard business club nick1 june 15 2013Big data in marketing at harvard business club nick1 june 15 2013
Big data in marketing at harvard business club nick1 june 15 2013
 
Big Data Analytics V2
Big Data Analytics V2Big Data Analytics V2
Big Data Analytics V2
 
Big data business case
Big data   business caseBig data   business case
Big data business case
 
Data mining
Data miningData mining
Data mining
 
Data Mining: Future Trends and Applications
Data Mining: Future Trends and ApplicationsData Mining: Future Trends and Applications
Data Mining: Future Trends and Applications
 
Dma unit 1
Dma unit   1Dma unit   1
Dma unit 1
 
Introduction To Data Mining
Introduction To Data MiningIntroduction To Data Mining
Introduction To Data Mining
 
Introduction To Data Mining
Introduction To Data Mining   Introduction To Data Mining
Introduction To Data Mining
 
Data Mining and Knowledge Discovery in Large Databases
Data Mining and Knowledge Discovery in Large DatabasesData Mining and Knowledge Discovery in Large Databases
Data Mining and Knowledge Discovery in Large Databases
 
Big data unit 2
Big data unit 2Big data unit 2
Big data unit 2
 
The AI Platform Business Revolution: Matchmaking, Empathetic Technology, and ...
The AI Platform Business Revolution: Matchmaking, Empathetic Technology, and ...The AI Platform Business Revolution: Matchmaking, Empathetic Technology, and ...
The AI Platform Business Revolution: Matchmaking, Empathetic Technology, and ...
 

More from European Data Forum

EDF2014: Ralf-Peter Schaefer, Head of Traffic Product Unit, TomTom, Germany: ...
EDF2014: Ralf-Peter Schaefer, Head of Traffic Product Unit, TomTom, Germany: ...EDF2014: Ralf-Peter Schaefer, Head of Traffic Product Unit, TomTom, Germany: ...
EDF2014: Ralf-Peter Schaefer, Head of Traffic Product Unit, TomTom, Germany: ...European Data Forum
 
EDF2014: BIG - NESSI Networking Session: Edward Curry, National University of...
EDF2014: BIG - NESSI Networking Session: Edward Curry, National University of...EDF2014: BIG - NESSI Networking Session: Edward Curry, National University of...
EDF2014: BIG - NESSI Networking Session: Edward Curry, National University of...European Data Forum
 
EDF2014: BIG - NESSI Networking Session: Nuria de Lama, Representative to the...
EDF2014: BIG - NESSI Networking Session: Nuria de Lama, Representative to the...EDF2014: BIG - NESSI Networking Session: Nuria de Lama, Representative to the...
EDF2014: BIG - NESSI Networking Session: Nuria de Lama, Representative to the...European Data Forum
 
EDF2014: BIG - NESSI Networking Session: Intro Presentation
EDF2014: BIG - NESSI Networking Session: Intro PresentationEDF2014: BIG - NESSI Networking Session: Intro Presentation
EDF2014: BIG - NESSI Networking Session: Intro PresentationEuropean Data Forum
 
EDF2014: Kush Wadhwa, Senior Partner, Trilateral Research & Consulting: Addre...
EDF2014: Kush Wadhwa, Senior Partner, Trilateral Research & Consulting: Addre...EDF2014: Kush Wadhwa, Senior Partner, Trilateral Research & Consulting: Addre...
EDF2014: Kush Wadhwa, Senior Partner, Trilateral Research & Consulting: Addre...European Data Forum
 
EDF2014: Adrian Cristal, Barcelona Supercomputing Center, RETHINK big Project...
EDF2014: Adrian Cristal, Barcelona Supercomputing Center, RETHINK big Project...EDF2014: Adrian Cristal, Barcelona Supercomputing Center, RETHINK big Project...
EDF2014: Adrian Cristal, Barcelona Supercomputing Center, RETHINK big Project...European Data Forum
 
EDF2014: Dimitris Vassiliadis, Head of Unit, EXUS Innovation Attractor: From ...
EDF2014: Dimitris Vassiliadis, Head of Unit, EXUS Innovation Attractor: From ...EDF2014: Dimitris Vassiliadis, Head of Unit, EXUS Innovation Attractor: From ...
EDF2014: Dimitris Vassiliadis, Head of Unit, EXUS Innovation Attractor: From ...European Data Forum
 
EDF2014: Rüdiger Eichin, Research Manager at SAP AG, Germany: Deriving Value ...
EDF2014: Rüdiger Eichin, Research Manager at SAP AG, Germany: Deriving Value ...EDF2014: Rüdiger Eichin, Research Manager at SAP AG, Germany: Deriving Value ...
EDF2014: Rüdiger Eichin, Research Manager at SAP AG, Germany: Deriving Value ...European Data Forum
 
EDF2014: Paul Groth, Department of Computer Science & The Network Institute, ...
EDF2014: Paul Groth, Department of Computer Science & The Network Institute, ...EDF2014: Paul Groth, Department of Computer Science & The Network Institute, ...
EDF2014: Paul Groth, Department of Computer Science & The Network Institute, ...European Data Forum
 
EDF2014: Christian Lindemann, Wolters Kluwer Germany & Christian Dirschl, Wol...
EDF2014: Christian Lindemann, Wolters Kluwer Germany & Christian Dirschl, Wol...EDF2014: Christian Lindemann, Wolters Kluwer Germany & Christian Dirschl, Wol...
EDF2014: Christian Lindemann, Wolters Kluwer Germany & Christian Dirschl, Wol...European Data Forum
 
EDF2014: Marta Nagy-Rothengass, Head of Unit Data Value Chain, Directorate Ge...
EDF2014: Marta Nagy-Rothengass, Head of Unit Data Value Chain, Directorate Ge...EDF2014: Marta Nagy-Rothengass, Head of Unit Data Value Chain, Directorate Ge...
EDF2014: Marta Nagy-Rothengass, Head of Unit Data Value Chain, Directorate Ge...European Data Forum
 
EDF2014: Stefan Wrobel, Institute Director, Fraunhofer IAIS / Member of the b...
EDF2014: Stefan Wrobel, Institute Director, Fraunhofer IAIS / Member of the b...EDF2014: Stefan Wrobel, Institute Director, Fraunhofer IAIS / Member of the b...
EDF2014: Stefan Wrobel, Institute Director, Fraunhofer IAIS / Member of the b...European Data Forum
 
EDF2014: Michele Vescovi, Researcher, Semantic & Knowledge Innovation Lab, It...
EDF2014: Michele Vescovi, Researcher, Semantic & Knowledge Innovation Lab, It...EDF2014: Michele Vescovi, Researcher, Semantic & Knowledge Innovation Lab, It...
EDF2014: Michele Vescovi, Researcher, Semantic & Knowledge Innovation Lab, It...European Data Forum
 
EDF2014: Allan Hanbury, Senior Researcher, Vienna University of Technology, A...
EDF2014: Allan Hanbury, Senior Researcher, Vienna University of Technology, A...EDF2014: Allan Hanbury, Senior Researcher, Vienna University of Technology, A...
EDF2014: Allan Hanbury, Senior Researcher, Vienna University of Technology, A...European Data Forum
 
EDF2014: Nikolaos Loutas, Manager at PwC Belgium, Business Models for Linked ...
EDF2014: Nikolaos Loutas, Manager at PwC Belgium, Business Models for Linked ...EDF2014: Nikolaos Loutas, Manager at PwC Belgium, Business Models for Linked ...
EDF2014: Nikolaos Loutas, Manager at PwC Belgium, Business Models for Linked ...European Data Forum
 
EDF2014: Vedran Sabol, Head of the Knowledge Visualisation Area, Know-Center,...
EDF2014: Vedran Sabol, Head of the Knowledge Visualisation Area, Know-Center,...EDF2014: Vedran Sabol, Head of the Knowledge Visualisation Area, Know-Center,...
EDF2014: Vedran Sabol, Head of the Knowledge Visualisation Area, Know-Center,...European Data Forum
 
EDF2014: Daniel Vila-Suero, Researcher, Ontology Engineering Group, Universid...
EDF2014: Daniel Vila-Suero, Researcher, Ontology Engineering Group, Universid...EDF2014: Daniel Vila-Suero, Researcher, Ontology Engineering Group, Universid...
EDF2014: Daniel Vila-Suero, Researcher, Ontology Engineering Group, Universid...European Data Forum
 
EDF2014: Piek Vossen, Professor Computational Lexicology, VU University Amste...
EDF2014: Piek Vossen, Professor Computational Lexicology, VU University Amste...EDF2014: Piek Vossen, Professor Computational Lexicology, VU University Amste...
EDF2014: Piek Vossen, Professor Computational Lexicology, VU University Amste...European Data Forum
 
EDF2014: Taru Rastas, Senior Advisor, Ministry of Communications of Finland: ...
EDF2014: Taru Rastas, Senior Advisor, Ministry of Communications of Finland: ...EDF2014: Taru Rastas, Senior Advisor, Ministry of Communications of Finland: ...
EDF2014: Taru Rastas, Senior Advisor, Ministry of Communications of Finland: ...European Data Forum
 

More from European Data Forum (20)

EDF2014: Ralf-Peter Schaefer, Head of Traffic Product Unit, TomTom, Germany: ...
EDF2014: Ralf-Peter Schaefer, Head of Traffic Product Unit, TomTom, Germany: ...EDF2014: Ralf-Peter Schaefer, Head of Traffic Product Unit, TomTom, Germany: ...
EDF2014: Ralf-Peter Schaefer, Head of Traffic Product Unit, TomTom, Germany: ...
 
Barbato leit ict 15-16-17
Barbato leit ict 15-16-17Barbato leit ict 15-16-17
Barbato leit ict 15-16-17
 
EDF2014: BIG - NESSI Networking Session: Edward Curry, National University of...
EDF2014: BIG - NESSI Networking Session: Edward Curry, National University of...EDF2014: BIG - NESSI Networking Session: Edward Curry, National University of...
EDF2014: BIG - NESSI Networking Session: Edward Curry, National University of...
 
EDF2014: BIG - NESSI Networking Session: Nuria de Lama, Representative to the...
EDF2014: BIG - NESSI Networking Session: Nuria de Lama, Representative to the...EDF2014: BIG - NESSI Networking Session: Nuria de Lama, Representative to the...
EDF2014: BIG - NESSI Networking Session: Nuria de Lama, Representative to the...
 
EDF2014: BIG - NESSI Networking Session: Intro Presentation
EDF2014: BIG - NESSI Networking Session: Intro PresentationEDF2014: BIG - NESSI Networking Session: Intro Presentation
EDF2014: BIG - NESSI Networking Session: Intro Presentation
 
EDF2014: Kush Wadhwa, Senior Partner, Trilateral Research & Consulting: Addre...
EDF2014: Kush Wadhwa, Senior Partner, Trilateral Research & Consulting: Addre...EDF2014: Kush Wadhwa, Senior Partner, Trilateral Research & Consulting: Addre...
EDF2014: Kush Wadhwa, Senior Partner, Trilateral Research & Consulting: Addre...
 
EDF2014: Adrian Cristal, Barcelona Supercomputing Center, RETHINK big Project...
EDF2014: Adrian Cristal, Barcelona Supercomputing Center, RETHINK big Project...EDF2014: Adrian Cristal, Barcelona Supercomputing Center, RETHINK big Project...
EDF2014: Adrian Cristal, Barcelona Supercomputing Center, RETHINK big Project...
 
EDF2014: Dimitris Vassiliadis, Head of Unit, EXUS Innovation Attractor: From ...
EDF2014: Dimitris Vassiliadis, Head of Unit, EXUS Innovation Attractor: From ...EDF2014: Dimitris Vassiliadis, Head of Unit, EXUS Innovation Attractor: From ...
EDF2014: Dimitris Vassiliadis, Head of Unit, EXUS Innovation Attractor: From ...
 
EDF2014: Rüdiger Eichin, Research Manager at SAP AG, Germany: Deriving Value ...
EDF2014: Rüdiger Eichin, Research Manager at SAP AG, Germany: Deriving Value ...EDF2014: Rüdiger Eichin, Research Manager at SAP AG, Germany: Deriving Value ...
EDF2014: Rüdiger Eichin, Research Manager at SAP AG, Germany: Deriving Value ...
 
EDF2014: Paul Groth, Department of Computer Science & The Network Institute, ...
EDF2014: Paul Groth, Department of Computer Science & The Network Institute, ...EDF2014: Paul Groth, Department of Computer Science & The Network Institute, ...
EDF2014: Paul Groth, Department of Computer Science & The Network Institute, ...
 
EDF2014: Christian Lindemann, Wolters Kluwer Germany & Christian Dirschl, Wol...
EDF2014: Christian Lindemann, Wolters Kluwer Germany & Christian Dirschl, Wol...EDF2014: Christian Lindemann, Wolters Kluwer Germany & Christian Dirschl, Wol...
EDF2014: Christian Lindemann, Wolters Kluwer Germany & Christian Dirschl, Wol...
 
EDF2014: Marta Nagy-Rothengass, Head of Unit Data Value Chain, Directorate Ge...
EDF2014: Marta Nagy-Rothengass, Head of Unit Data Value Chain, Directorate Ge...EDF2014: Marta Nagy-Rothengass, Head of Unit Data Value Chain, Directorate Ge...
EDF2014: Marta Nagy-Rothengass, Head of Unit Data Value Chain, Directorate Ge...
 
EDF2014: Stefan Wrobel, Institute Director, Fraunhofer IAIS / Member of the b...
EDF2014: Stefan Wrobel, Institute Director, Fraunhofer IAIS / Member of the b...EDF2014: Stefan Wrobel, Institute Director, Fraunhofer IAIS / Member of the b...
EDF2014: Stefan Wrobel, Institute Director, Fraunhofer IAIS / Member of the b...
 
EDF2014: Michele Vescovi, Researcher, Semantic & Knowledge Innovation Lab, It...
EDF2014: Michele Vescovi, Researcher, Semantic & Knowledge Innovation Lab, It...EDF2014: Michele Vescovi, Researcher, Semantic & Knowledge Innovation Lab, It...
EDF2014: Michele Vescovi, Researcher, Semantic & Knowledge Innovation Lab, It...
 
EDF2014: Allan Hanbury, Senior Researcher, Vienna University of Technology, A...
EDF2014: Allan Hanbury, Senior Researcher, Vienna University of Technology, A...EDF2014: Allan Hanbury, Senior Researcher, Vienna University of Technology, A...
EDF2014: Allan Hanbury, Senior Researcher, Vienna University of Technology, A...
 
EDF2014: Nikolaos Loutas, Manager at PwC Belgium, Business Models for Linked ...
EDF2014: Nikolaos Loutas, Manager at PwC Belgium, Business Models for Linked ...EDF2014: Nikolaos Loutas, Manager at PwC Belgium, Business Models for Linked ...
EDF2014: Nikolaos Loutas, Manager at PwC Belgium, Business Models for Linked ...
 
EDF2014: Vedran Sabol, Head of the Knowledge Visualisation Area, Know-Center,...
EDF2014: Vedran Sabol, Head of the Knowledge Visualisation Area, Know-Center,...EDF2014: Vedran Sabol, Head of the Knowledge Visualisation Area, Know-Center,...
EDF2014: Vedran Sabol, Head of the Knowledge Visualisation Area, Know-Center,...
 
EDF2014: Daniel Vila-Suero, Researcher, Ontology Engineering Group, Universid...
EDF2014: Daniel Vila-Suero, Researcher, Ontology Engineering Group, Universid...EDF2014: Daniel Vila-Suero, Researcher, Ontology Engineering Group, Universid...
EDF2014: Daniel Vila-Suero, Researcher, Ontology Engineering Group, Universid...
 
EDF2014: Piek Vossen, Professor Computational Lexicology, VU University Amste...
EDF2014: Piek Vossen, Professor Computational Lexicology, VU University Amste...EDF2014: Piek Vossen, Professor Computational Lexicology, VU University Amste...
EDF2014: Piek Vossen, Professor Computational Lexicology, VU University Amste...
 
EDF2014: Taru Rastas, Senior Advisor, Ministry of Communications of Finland: ...
EDF2014: Taru Rastas, Senior Advisor, Ministry of Communications of Finland: ...EDF2014: Taru Rastas, Senior Advisor, Ministry of Communications of Finland: ...
EDF2014: Taru Rastas, Senior Advisor, Ministry of Communications of Finland: ...
 

Recently uploaded

So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Mark Goldstein
 
Kuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialKuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialJoão Esperancinha
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesThousandEyes
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...itnewsafrica
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfAarwolf Industries LLC
 
Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...itnewsafrica
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxfnnc6jmgwh
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Mark Simos
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentMahmoud Rabie
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkPixlogix Infotech
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFMichael Gough
 
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical InfrastructureVarsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructureitnewsafrica
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
Irene Moetsana-Moeng: Stakeholders in Cybersecurity: Collaborative Defence fo...
Irene Moetsana-Moeng: Stakeholders in Cybersecurity: Collaborative Defence fo...Irene Moetsana-Moeng: Stakeholders in Cybersecurity: Collaborative Defence fo...
Irene Moetsana-Moeng: Stakeholders in Cybersecurity: Collaborative Defence fo...itnewsafrica
 

Recently uploaded (20)

So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
 
Kuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialKuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorial
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdf
 
Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career Development
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App Framework
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDF
 
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical InfrastructureVarsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
Irene Moetsana-Moeng: Stakeholders in Cybersecurity: Collaborative Defence fo...
Irene Moetsana-Moeng: Stakeholders in Cybersecurity: Collaborative Defence fo...Irene Moetsana-Moeng: Stakeholders in Cybersecurity: Collaborative Defence fo...
Irene Moetsana-Moeng: Stakeholders in Cybersecurity: Collaborative Defence fo...
 

EDF2013: Big Data Tutorial: Marko Grobelnik

  • 1. Marko Grobelnik marko.grobelnik@ijs.si Jozef Stefan Institute Dublin, April 9th 2013
  • 2. Introduction ◦ What is Big data? ◦ Why Big-Data? ◦ When Big-Data is really a problem?  Market Overview  Tools  Techniques  Applications  Literature
  • 3.
  • 4. ‘Big-data’ is similar to ‘Small-data’, but bigger  …but having data bigger it requires different approaches: ◦ techniques, tools, architectures  …with an aim to solve new problems ◦ …or old problems in a better way.
  • 5. From “Understanding Big Data” by IBM
  • 6.
  • 8.
  • 9. Key enablers for the appearance and growth of “Big Data” are: ◦ Increase of storage capacities ◦ Increase of processing power ◦ Availability of data
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20. Source: WikiBon report on “Big Data Vendor Revenue and Market Forecast 2012-2017”, 2013
  • 21.
  • 22.
  • 23.
  • 24. Where processing is hosted? ◦ Distributed Servers / Cloud (e.g. Amazon EC2)  Where data is stored? ◦ Distributed Storage (e.g. Amazon S3)  What is the programming model? ◦ Distributed Processing (e.g. MapReduce)  How data is stored & indexed? ◦ High-performance schema-free databases (e.g. MongoDB)  What operations are performed on data? ◦ Analytic / Semantic Processing (e.g. R, OWLIM)
  • 25. Computing and storage are typically hosted transparently on cloud infrastructures ◦ …providing scale, flexibility and high fail-safety  Distributed Servers ◦ Amazon-EC2, Google App Engine, Elastic, Beanstalk, Heroku  Distributed Storage ◦ Amazon-S3, Hadoop Distributed File System
  • 26. Distributed processing of Big-Data requires non- standard programming models ◦ …beyond single machines or traditional parallel programming models (like MPI) ◦ …the aim is to simplify complex programming tasks  The most popular programming model is MapReduce approach  Implementations of MapReduce ◦ Hadoop (http://hadoop.apache.org/), Hive, Pig, Cascading, Cascalog, mrjob, Caffeine, S4, MapR, Acunu, Flume, Kafka, Azkaban, Oozie, Greenplum
  • 27. The key idea of the MapReduce approach: ◦ A target problem needs to be parallelizable ◦ First, the problem gets split into a set of smaller problems (Map step) ◦ Next, smaller problems are solved in a parallel way ◦ Finally, a set of solutions to the smaller problems get synthesized into a solution of the original problem (Reduce step)
  • 28. NoSQL class of databases have in common: ◦ To support large amounts of data ◦ Have mostly non-SQL interface ◦ Operate on distributed infrastructures (e.g. Hadoop) ◦ Are based on key-value pairs (no predefined schema) ◦ …are flexible and fast  Implementations ◦ MongoDB, CouchDB, Cassandra, Redis, BigTable, Hbase, Hypertable, Voldemort, Riak, ZooKeeper…
  • 29.
  • 30. …when the operations on data are complex: ◦ e.g. simple counting is not a complex problem ◦ Modeling and reasoning with data of different kinds can get extremely complex  Good news about big-data: ◦ Often, because of vast amount of data, modeling techniques can get simpler (e.g. smart counting can replace complex model-based analytics)… ◦ …as long as we deal with the scale
  • 31. Research areas (such as IR, KDD, ML, NLP, Usage SemWeb, …) are sub- cubes within the data Quality cube Context Streaming Scalability
  • 32. A risk with “Big-Data mining” is that an analyst can “discover” patterns that are meaningless  Statisticians call it Bonferroni’s principle: ◦ Roughly, if you look in more places for interesting patterns, than your amount of data will support almost anything, …and you are bound to find lots of nonsense Example taken from: Rajaraman, Ullman: Mining of Massive Datasets
  • 33. Example:  We want to find (unrelated) people who at least twice have stayed at the same hotel on the same day ◦ 109 people being tracked. ◦ 1000 days. ◦ Each person stays in a hotel 1% of the time (1 day out of 100) ◦ Hotels hold 100 people (so 105 hotels). ◦ If everyone behaves randomly (i.e., no terrorists) will the data mining detect anything suspicious?  Expected number of “suspicious” pairs of people: ◦ 250,000 ◦ … too many combinations to check – we need to have some additional evidence to find “suspicious” pairs of people in some more efficient way Example taken from: Rajaraman, Ullman: Mining of Massive Datasets
  • 34. Smart sampling of data ◦ …reducing the original data while not losing the statistical properties of data  Finding similar items ◦ …efficient multidimensional indexing  Incremental updating of the models ◦ (vs. building models from scratch) ◦ …crucial for streaming data  Distributed linear algebra ◦ …dealing with large sparse matrices
  • 35. On the top of the previous ops we perform usual data mining/machine learning/statistics operators: ◦ Supervised learning (classification, regression, …) ◦ Non-supervised learning (clustering, different types of decompositions, …) ◦ …  …we are just more careful which algorithms we choose (typically linear or sub-linear versions)
  • 36. An excellent overview of the algorithms covering the above issues is the book “Rajaraman, Leskovec, Ullman: Mining of Massive Datasets”
  • 37.
  • 38. Good recommendations can make a big difference when keeping a user on a web site ◦ …the key is how rich the context model a system is using to select information for a user ◦ Bad recommendations <1% users, good ones >5% users click ◦ 200clicks/sec Contextual personalized recommendations generated in ~20ms
  • 39. Domain  Referring Domain  Zip Code  Sub-domain  Referring URL  State  Page URL  Outgoing URL  Income  URL sub-directories  Age  GeoIP Country  Gender  Page Meta Tags  GeoIP State  Country  Page Title  GeoIP City  Job Title  Page Content  Job Industry  Named Entities  Absolute Date  Day of the Week  Has Query  Day period  Referrer Query  Hour of the day  User Agent
  • 40. Trend Detection System User Stream of Log Files Stream of clicks profiles profiles (~100M page clicks per day) Sales Trends and updated segments Segments Segment Keywords NYT Stock Stock Market, mortgage, banking, Market investors, Wall Street, turmoil, New articles York Stock Exchange Campaign Health diabetes, heart disease, disease, heart, illness to sell segments $ Green Hybrid cars, energy, power, model, Energy carbonated, fuel, bulbs, Hybrid cars Hybrid cars, vehicles, model, engines, diesel Travel travel, wine, opening, tickets, hotel, sites, cars, search, restaurant Advertisers … …
  • 41. 50Gb of uncompressed log files  50-100M clicks  4-6M unique users  7000 unique pages with more then 100 hits
  • 42. Alarms Server Telecom Network Alarms Alarms Live feed of data Explorer (~25 000 ~10-100/sec devices) Server  Alarms Explorer Server implements three real-time scenarios on the alarms stream: 1. Root-Cause-Analysis – finding which device is responsible for occasional “flood” of alarms 2. Short-Term Fault Prediction – predict which device will fail in next 15mins 3. Long-Term Anomaly Detection – detect unusual trends in the network  …system is used in British Telecom Operator Big board display
  • 43. The aim is to use analytic techniques to visualize documents in different ways: ◦ Topic view ◦ Social view ◦ Temporal view
  • 46. US Elections US Budget Query Result set NATO-Russia Topic Trends Visualization Mid-East conflict Topics description
  • 50. Query Conceptual map Search Point Dynamic contextual ranking based on the search point
  • 51. Observe social and communication phenomena at a planetary scale  Largest social network analyzed till 2010 Research questions:  How does communication change with user demographics (age, sex, language, country)?  How does geography affect communication?  What is the structure of the communication network? “Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008 51
  • 52. We collected the data for June 2006  Log size: 150Gb/day (compressed)  Total: 1 month of communication data: 4.5Tb of compressed data  Activity over June 2006 (30 days) ◦ 245 million users logged in ◦ 180 million users engaged in conversations ◦ 17,5 million new accounts activated ◦ More than 30 billion conversations ◦ More than 255 billion exchanged messages “Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008 52
  • 53. “Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008 53
  • 54. “Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008 54
  • 55. Count the number of users logging in from particular location on the earth “Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008 55
  • 56. Logins from Europe “Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008 56
  • 57. Hops Nodes 1 10 2 78 3 396 4 8648 5 3299252 6 28395849 7 79059497 8 52995778 9 10321008 10 1955007 11 518410 12 149945 13 44616 14 13740 15 4476 16 1542 17 536 18 167 19 71  6 degrees of separation [Milgram ’60s] 20 29 Average distance between two random users is 6.622 21 16  10  90% of nodes can be reached in < 8 hops 23 3 24 2 “Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008 25 3
  • 58.
  • 59. Big-Data is everywhere, we are just not used to deal with it  The “Big-Data” hype is very recent ◦ …growth seems to be going up ◦ …evident lack of experts to build Big-Data apps  Can we do “Big-Data” without big investment? ◦ …yes – many open source tools, computing machinery is cheap (to buy or to rent) ◦ …the key is knowledge on how to deal with data ◦ …data is either free (e.g. Wikipedia) or to buy (e.g. twitter)