Introduction to Big Data processing (FGRE2016)

•

0 likes•854 views

Thomas Vanhove

Introduction to Big Data processing: Lecture on the FGRE 2016 Summer School

Data & Analytics

Big data processing
Thomas Vanhove, Bruno Volckaert

Data flow
Ingest Process Store Represent

Data processing - Batch
Collecting Processing Assembling

Data processing - Batch - Frameworks
MapReduce

Data processing - Batch - Hadoop
HDFS
YARN

Data processing - Batch - MapReduce
HDFS
YARN
MapReduce

Data processing - Batch - MapReduce - Example
(“lorem”, 1)
(“ipsum”, 1)
(“dolor”, 1)
(“lorem”, 1)
(“lorem”, 1)
(“ipsum”, 1)
(“ipsum”, 1)
(“ipsum”, 1)
(“dolor”, 1)
(“dolor”, 1)
(“dolor”, 1)
Lorem, 14
Ipsum, 5
Dolor, 22
%% BOOK

Data processing - Batch - Software Stacks
HDFS
YARN
MapReduce
Hive

Data processing - Batch - Hive
Hive QL MapReduce

Data processing - Batch - Hive
Hive QL MapReduce
Hive Shell
Web UI

Data processing - Batch - Hive
LATENCYCOMPLEXITY

Data processing - Batch - Frameworks
HDFS
YARN
MapReduce Spark
Hive

Data processing - Batch - Spark
RESILIENT DISTRIBUTED DATASETS

Data processing - Batch - Spark
Execution time
Hadoop Spark

Data processing - Batch - Spark
Execution time
Hadoop Spark
30x - 100x faster

Data processing - Batch - Spark
HDFS
YARN
MapReduce Spark
Hive MLlib
Spark
SQL
Streaming

Data processing - Batch - Spark - Examples

Data processing - Stream - Storm
SPOUT BOLT

Data processing - Stream - Storm - Examples

Data processing - Beyond
Batch
Stream
Data Query
Query

Data processing - Lambda
Batch
Layer
Stream
Layer
Data Query
Query

Big data is like teenage sex:
everyone talks about it,
nobody really knows how to do it,
everyone thinks everyone else is doing it,
so everyone claims they are doing it...
Dan Ariely
Duke University

What's hot

Bigdata Nedir? Hadoop Nedir? MapReduce Nedir? Big Data.Zekeriya Besiroglu

Introduction to Bigdata and HADOOP vinoth kumar

Big data Analytics HadoopMishika Bharadwaj

Hadoop Tutorial For Beginners | Apache Hadoop Tutorial For Beginners | Hadoop...Simplilearn

Basics of big data analytics hadoopAmbuj Kumar

Introduction to Big Data and HadoopEdureka!

Setting up Hadoop YARN ClusteringDanairat Thanabodithammachari

hadoop&zingzingopen

Big dataSampath Bhargav Pinnam

Hadoop/Spark Non-Technical BasicsZitao Liu

Introduction of Big data and Hadoop Arohi Khandelwal

R, Hadoop and Amazon Web ServicesPortland R User Group

Open source analyticsAjay Ohri

Big Data and Hadoop IntroductionDzung Nguyen

Hadoop Case Studies in the Real WorldMobin Ranjbar

Hadoop summit 2010 frameworks panel elephant birdKevin Weil

Bigdata & HadoopPinto Das

Scaling AncestryDNA with the Hadoop Ecosystem. Presented at the San Jose Hado...William Yetman

simple introduction to hadoopvishnu rao

Hadoop Tutorial For BeginnersDataflair Web Services Pvt Ltd

What's hot (20)

Bigdata Nedir? Hadoop Nedir? MapReduce Nedir? Big Data.

Introduction to Bigdata and HADOOP

Big data Analytics Hadoop

Hadoop Tutorial For Beginners | Apache Hadoop Tutorial For Beginners | Hadoop...

Basics of big data analytics hadoop

Introduction to Big Data and Hadoop

Setting up Hadoop YARN Clustering

hadoop&zing

Big data

Hadoop/Spark Non-Technical Basics

Introduction of Big data and Hadoop

R, Hadoop and Amazon Web Services

Open source analytics

Big Data and Hadoop Introduction

Hadoop Case Studies in the Real World

Hadoop summit 2010 frameworks panel elephant bird

Bigdata & Hadoop

Scaling AncestryDNA with the Hadoop Ecosystem. Presented at the San Jose Hado...

simple introduction to hadoop

Hadoop Tutorial For Beginners

Viewers also liked

Big data Processing with Apache Spark & ScalaEdureka!

A Novel methodology for handling Document Level Security in Search Based Appl...lucenerevolution

Integrate ManifoldCF with Solrfrancelabs

Processing Complex Workflows in Advertising using HadoopDataWorks Summit

Hive - SerDe and LazySerdeZheng Shao

Introduction to Apache Flink - Fast and reliable big data processingTill Rohrmann

An intriduction to hiveReza Ameri

Unified Log London (May 2015) - Why your company needs a unified logAlexander Dean

Ten tools for ten big data areas 04_Apache HiveWill Du

Hive ICDE 2010ragho

Data Engineering with Spring, Hadoop and Hive Alex Silva

20081030linkedinJeff Hammerbacher

Hive analytic workloads hadoop summit san jose 2014alanfgates

Hadoop Summit 2009 HiveNamit Jain

Hive Object ModelZheng Shao

Unified Log Processing ArchitectureGuido Schmutz

Cost-based query optimization in Apache HiveJulian Hyde

Learning Apache HIVE - Data Warehouse and Query Language for HadoopSomeshwar Kale

Internal HiveRecruit Technologies

Hive Anatomynzhang

Viewers also liked (20)

Big data Processing with Apache Spark & Scala

A Novel methodology for handling Document Level Security in Search Based Appl...

Integrate ManifoldCF with Solr

Processing Complex Workflows in Advertising using Hadoop

Hive - SerDe and LazySerde

Introduction to Apache Flink - Fast and reliable big data processing

An intriduction to hive

Unified Log London (May 2015) - Why your company needs a unified log

Ten tools for ten big data areas 04_Apache Hive

Hive ICDE 2010

Data Engineering with Spring, Hadoop and Hive

20081030linkedin

Hive analytic workloads hadoop summit san jose 2014

Hadoop Summit 2009 Hive

Hive Object Model

Unified Log Processing Architecture

Cost-based query optimization in Apache Hive

Learning Apache HIVE - Data Warehouse and Query Language for Hadoop

Internal Hive

Hive Anatomy

Similar to Introduction to Big Data processing (FGRE2016)

Big data with javaStefan Angelov

Big Data ConceptsAhmed Salman

Big Data on the CloudSercan Karaoglu

20th Athens Big Data Meetup - 1st Talk - Druid: the open source, performant, ...Athens Big Data

Big data and hadoopAshishRathore72

Hadoop at the Center: The Next Generation of HadoopAdam Muise

Big Data - HDInsight and Power BIPrasad Prabhu (PP)

Introduction to Big data & Hadoop -IEdureka!

Hadoop DeveloperEdureka!

August meetup - All about Apache Druid Imply

Big Data Analytics with Hadoop, MongoDB and SQL ServerMark Kromer

Inroduction to Big DataOmnia Safaan

Introduction to Big Data & HadoopEdureka!

Data Science Toolchain 101Francis Michael Bautista

Handling not so big dataSATOSHI TAGOMORI

Big Data: Architecture and Performance Considerations in Logical Data LakesDenodo

Evolution of spark framework for simplifying data analysis.Anirudh Gangwar

Overview of stinger interactive query for hiveDavid Kaiser

Druid: Under the Covers (Virtual Meetup)Imply

Hadoop overviewSiva Pandeti

Similar to Introduction to Big Data processing (FGRE2016) (20)

Big data with java

Big Data Concepts

Big Data on the Cloud

20th Athens Big Data Meetup - 1st Talk - Druid: the open source, performant, ...

Big data and hadoop

Hadoop at the Center: The Next Generation of Hadoop

Big Data - HDInsight and Power BI

Introduction to Big data & Hadoop -I

Hadoop Developer

August meetup - All about Apache Druid

Big Data Analytics with Hadoop, MongoDB and SQL Server

Inroduction to Big Data

Introduction to Big Data & Hadoop

Data Science Toolchain 101

Handling not so big data

Big Data: Architecture and Performance Considerations in Logical Data Lakes

Evolution of spark framework for simplifying data analysis.

Overview of stinger interactive query for hive

Druid: Under the Covers (Virtual Meetup)

Hadoop overview

Recently uploaded

Brighton SEO | April 2024 | Data StorytellingNeil Barnes

办理学位证纽约大学毕业证(NYU毕业证书）原版一比一fhwihughh

RadioAdProWritingCinderellabyButleri.pdfgstagge

Industrialised data - the key to AI success.pdfLars Albertsson

Decoding Loan Approval: Predictive Modeling in ActionBoston Institute of Analytics

Customer Service Analytics - Make Sense of All Your Data.pptxEmmanuel Dauda

Amazon TQM (2) Amazon TQM (2)Amazon TQM (2).pptxAbdelrhman abooda

PKS-TGC-1084-630 - Stage 1 Proposal.pptxPramod Kumar Srivastava

EMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM TRACKING WITH GOOGLE ANALYTICS.pptxthyngster

9654467111 Call Girls In Munirka Hotel And Home ServiceSapana Sha

VIP High Class Call Girls Jamshedpur Anushka 8250192130 Independent Escort Se...Suhani Kapoor

VIP High Profile Call Girls Amravati Aarushi 8250192130 Independent Escort Se...Suhani Kapoor

Building on a FAIRly Strong Foundation to Connect Academic Research to Transl...Jack DiGiovanna

RA-11058_IRR-COMPRESS Do 198 series of 1998YohFuh

办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一F La

20240419 - Measurecamp Amsterdam - SAM.pdfHuman37

{Pooja: 9892124323 } Call Girl in Mumbai | Jas Kaur Rate 4500 Free Hotel Del...Pooja Nehwal

(PARI) Call Girls Wanowrie ( 7001035870 ) HI-Fi Pune Escorts Serviceranjana rawat

Call Girls in Saket 99530🔝 56974 Escort Service9953056974 Low Rate Call Girls In Saket, Delhi NCR

Call Girls In Dwarka 9654467111 Escorts ServiceSapana Sha

Recently uploaded (20)

Brighton SEO | April 2024 | Data Storytelling

办理学位证纽约大学毕业证(NYU毕业证书）原版一比一

RadioAdProWritingCinderellabyButleri.pdf

Industrialised data - the key to AI success.pdf

Decoding Loan Approval: Predictive Modeling in Action

Customer Service Analytics - Make Sense of All Your Data.pptx

Amazon TQM (2) Amazon TQM (2)Amazon TQM (2).pptx

PKS-TGC-1084-630 - Stage 1 Proposal.pptx

EMERCE - 2024 - AMSTERDAM - CROSS-PLATFORM TRACKING WITH GOOGLE ANALYTICS.pptx

9654467111 Call Girls In Munirka Hotel And Home Service

VIP High Class Call Girls Jamshedpur Anushka 8250192130 Independent Escort Se...

VIP High Profile Call Girls Amravati Aarushi 8250192130 Independent Escort Se...

Building on a FAIRly Strong Foundation to Connect Academic Research to Transl...

RA-11058_IRR-COMPRESS Do 198 series of 1998

办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一

20240419 - Measurecamp Amsterdam - SAM.pdf

{Pooja: 9892124323 } Call Girl in Mumbai | Jas Kaur Rate 4500 Free Hotel Del...

(PARI) Call Girls Wanowrie ( 7001035870 ) HI-Fi Pune Escorts Service

Call Girls in Saket 99530🔝 56974 Escort Service

Call Girls In Dwarka 9654467111 Escorts Service

Introduction to Big Data processing (FGRE2016)

1. Big data processing Thomas Vanhove, Bruno Volckaert

4. What is big data?

5. What is big data? - Volume

6. What is big data? - Velocity

7. What is big data? - Variety

8. What is big data? - 3 Vs

9. What is big data? - 2 Vs

10. What is big data? - 3 Vs

11. Data flow Ingest Process Store Represent

12. Data flow Ingest Process Store Represent

13. Data flow Ingest Process Store Represent

20. Data processing - Batch Collecting Processing Assembling

21. Data processing - Batch Collecting Processing Assembling

22. Data processing - Batch Collecting Processing Assembling

23. Data processing - Batch - Frameworks

24. Data processing - Batch - Frameworks MapReduce

25. Data processing - Batch - Frameworks MapReduce

26. Data processing - Batch - Hadoop HDFS YARN

27. Data processing - Batch - Hadoop

28. Data processing - Batch - MapReduce HDFS YARN MapReduce

29. Data processing - Batch - MapReduce

30. Data processing - Batch - MapReduce - Example (“lorem”, 1) (“ipsum”, 1) (“dolor”, 1) (“lorem”, 1) (“lorem”, 1) (“ipsum”, 1) (“ipsum”, 1) (“ipsum”, 1) (“dolor”, 1) (“dolor”, 1) (“dolor”, 1) Lorem, 14 Ipsum, 5 Dolor, 22 %% BOOK

31. Data processing - Batch - MapReduce

32. Data processing - Batch - Software Stacks HDFS YARN MapReduce Hive

33. Data processing - Batch - Hive Hive QL

34. Data processing - Batch - Hive Hive QL MapReduce

35. Data processing - Batch - Hive Hive QL MapReduce Hive Shell Web UI

36. Data processing - Batch - Hive Hive QL MapReduce Hive Shell Web UI

37. Data processing - Batch - Hive LATENCYCOMPLEXITY

38. Data processing - Batch - Frameworks HDFS YARN MapReduce Spark Hive

39. Data processing - Batch - Spark RESILIENT DISTRIBUTED DATASETS

40. Data processing - Batch - Spark RESILIENT DISTRIBUTED DATASETS

41. Data processing - Batch - Spark RESILIENT DISTRIBUTED DATASETS

42. Data processing - Batch - Spark RESILIENT DISTRIBUTED DATASETS

43. Data processing - Batch - Spark Execution time Hadoop Spark

44. Data processing - Batch - Spark Execution time Hadoop Spark 30x - 100x faster

45. Data processing - Batch - Spark HDFS YARN MapReduce Spark Hive MLlib Spark SQL Streaming

46. Data processing - Batch - Spark - Examples

47. Data processing - Batch LATENCY

48. Data processing - Batch LATENCY

49. Data processing - Stream

50. Data processing - Stream

51. Data processing - Stream - Frameworks

52. Data processing - Stream - Storm SPOUT BOLT

53. Data processing - Stream - Storm

54. Data processing - Stream - Storm - Examples

55. Data processing - Stream HISTORIC DATA?

56. Data processing - Stream HISTORIC DATA?

57. Data processing - Beyond Batch Stream Data Query Query

58. Data processing - Lambda Batch Layer Stream Layer Data Query Query

59. Data processing - Lambda Batch Layer Stream Layer Data Query Query

60. Data processing - Lambda Batch Layer Stream Layer Data Query Query

61. Data processing - Lambda - Examples

62. What is big data?

63. What is big data? - 3 Vs

64. What is big data? - 3 Vs

65. What is big data? - 3 Vs

66. What is big data? - Veracity

67. What is big data? - 4 Vs

68. Big Data Landscape 2016

69.

70. It’s a jungle out there

71. Tengu

72.

73. Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it... Dan Ariely Duke University

74. Big data processing Thomas Vanhove, Bruno Volckaert

Editor's Notes

3 STEP
HIGH THROUGHPUTLOW LATENCY
HIGH THROUGHPUTLOW LATENCY
IMMUTABLE DATA SETS
NASA JET PROPULSION LAB - DEEP SPACE NETWORK - TELECOM NETWORK FOR SPACE EXPLORATION EBAY - LOG DATA AGGREGATION AND ANALYSIS
NO REALTIME FEEDBACK
CONTINUOUS STREAM OF DATA
LOW LATENCY PROCESS MESSAGE BY MESSAGE
STORMFLINK
TWO ELEMENTS SPOUT INJECTS DATA INTO A TOPOLOGYBOLT PROCESS
TOPOLOGY
SPOTIFY MONITORING AND REAL-TIME RECOMMENDATIONS GROUPON ANALYZE, NORMALIZE, CLEAN
METAMARKETS - ADVERTISING ANALYTICS YAHOO - ADVERTISING WAREHOUSE ANALYTICS
CHANGES EVERY YEAR GROWING COMPUTING POWER => CHANGING BIG DATA DEFINITION
UNCERTAINTY OF DATA

Introduction to Big Data processing (FGRE2016)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Introduction to Big Data processing (FGRE2016)

Similar to Introduction to Big Data processing (FGRE2016) (20)

Recently uploaded

Recently uploaded (20)

Introduction to Big Data processing (FGRE2016)

Editor's Notes