Have you ever heard the buzzword "big data"? Big data is briefly described to collect massive amounts of data and extract all the small details and larger trends that are available. Summarize the output and generate important insight about customers and competitors. Enterprises seem to have sensed that something is in the air and have started to shop technology. So what has the world to offer for enterprises that have an unknown amount of petabytes flowing through their systems on a daily basis? There are a few options, but really few that can catch up with the popularity of Hadoop. Hadoop can store and process large amounts of data. It has a large and diverse toolset for integrations, operations and processing and it is open source!

1. Introduction to Hadoop

2. • Tarjei Romtveit
• Co-founder of Monokkel AS
• Former CTO – Integrasco AS
• My story with Hadoop
www.monokkel.io

3. • Daglig leder i Monokkel AS
• Tidligere COO i Integrasco AS
• Persistering, Prosessering og Presentasjon av data
Persistering – Prosessering – Presentasjon

4. Bombshell
If you work with data today and not start to
learn the Hadoop ecosystem: You may be
unemployed soon

5. Agenda
• Context – Big Data and how to handle it
• What is Hadoop?
• Demo
• Distributions and/or demo
• “Deepdive” into Hadoop - Architecure
– HDFS
– YARN
– MapReduce
• Languages and ecosystem

6. What we not will cover
• Security
• Integrations with database X or system Y
• Running Hadoop in production

7. Big Data

8. Big Data – hype and hipsters

9. Big Data

10. Big Data – Let’s add some letters
• Volume
• Variety
• Velocity
• Variability
• Veracity / Data quality
and the step-brother
• Complexity

11. Big Data – Example

12. The Nordic Hotel Tycoon
1600 Hotels in 5 countries

13. I am a digital champion:
The website

14. I am a digital champion:
The desk

15. I am a digital champion:
The external provider

16. I am a digital champion
The IoT case

17. I am a digital champion
Social

18. Houston we have a problem
• Sales is declining and my stock price is
tumbling

19. The CEO

20. How can the CEO manage his
problem?
• Get control over the data
• Implement analytical
processes to aid sales

22. The data he need to handle
• Volume – Gigabytes/Terabyte
• Variety – Click stream, Voice, emails, sensor data,
social data, different languages, timestamp data,
transactional data, third party data
• Variability – Various quality
• Velocity – MB per second

23. The data he need to handle
• Veracity / Data quality – Inconsistent data quality
• Complexity – Many legacy domain models

24. How to handle ?
Web
Emails
Sensors
Social
Processing
RDBMS
Search

25. How to understand ?
Web
Emails
Sensors
Social
Processing
RDBMS
Search

26. So what do Hadoop solve?
Processing

27. What is Hadoop?

28. What is Hadoop?
An operating system for data

29. An OS need software on top

30. Distributions
'

31. Distributions
• ”Stable” compilation of the Hadoop Ecosystem
• Operational tools
• Integration tools and frameworks
• Data governance and data management tools
• Security

32. Distributions

33. HADOOP
An operating system for data
Layman’s terms
• Store huge files (unstructured) on many
machines
• Query and modify data
• Can run sophisticated analytics on top

34. How to start:
Alt 1
• https://hadoop.apache.org/
• Getting Started
• Download
• Unzip
• bin/hadoop <commandline arguments>
Alt 2
• http://hortonworks.com/products/hortonworks-sandbox/#install
• Install VMWare Player or VirtualBox
• Download image (6 GB)
• Install and run (give it lots of memory)

35. DEMO
– Transform and modify data
– Machine learning with Spark
– Integrate with ElasticSearch
NEXT: ARCHITECHTURE AND HOW IT WORKS

36. DEMO
• Hortonworks Sandbox
• Hortonworks Ambari
• Hortonworks Hue

37. Hadoop - Architecture
HDFS
YARN
MapReduce

38. 2.X.X
• Hadoop Distributed File System (HDFS)
• YARN (Yet Another Resource Negotiator)
• MapReduce

39. HDFS
D1
D2
DX
Name
Node
Failover
Name
Node
Client

40. HDFS
Block index
D1
D2
D3
Data
Nodes
B: 1, D1
B: 2, D2
B: 3, D3
B: 4, D1
B: 5, D2
B: 6, D3
Name node

41. HDFS
Block index
D1
D2
D3
Data
Nodes
B: 1, D1
B: 2, D2
B: 3, D3
B: 4, D1
B: 5, D2
B: 6, D3
Name node

42. HDFS Write
Client
/path/to/document1, R:2, B:{1,2}
Name Node
I need to write a
document!

43. Client
/path/to/document1, R:2, B:{1,2}
Name Node
I need to write
/path/to/document1, R:2, B:{3,4}
/path/to/document1, R:2, B:{5,6}
HDFS Write

44. Client
Name
Node
You can write to
: D1,D2,D3 D1
D2
D3
Data Nodes
HDFS Write

45. Client
Name
Node
D1
D2
D3
B:{D2:5,D3:6}
B:{D3:3,D1:4}
B:{D1:1,D2:2}
Split and write
HDFS Write

46. HDFS Write
Client
Name
Node
D1
D2
D3
Replicate
B:1 to
D2:2
Success

47. HDFS Read
Client
Name
Node
D1
D2
D3B:{D3:3,D3:6}
B:{D2:2,D2:5}

48. • HDFS blocks are immutable you can not change them!
• Deletes and updates are written as new blocks
• The node name takes care of overwriting deleted
blocks
• Small files are consuming a lot of name node memory
HDFS Delete/Update

49. HDFS Scalability
D1
D2
DX
Name
Node
Failover
Name
Node

50. YARN
HOW DOES HADOOP PROCESS
THE DATA STORED IN HDFS?

51. YARN
Client
Resource Manager
Scheduler
Applications manager
I want to process file
“docuemt1” with
my-app.jar?

52. YARN
Resource Manager
Scheduler
Applications manager
You can process on D1!

53. YARN
D1 D2
Node Manager Node Manager
Resource Manager
Scheduler
Applications manager
Start my-app.jar
Application Master

54. YARN
D1 D2
Node Manager Node Manager
Resource Manager
Scheduler
Applications manager
Application Master
AM to RM: “document1” is
located on d1 and d2 and I
need X Gb RAM

55. YARN
D1 D2
Node Manager Node Manager
Application Master Container
Resource Manager
Scheduler
Applications manager
my-app.jar is running here!
Start my-app.jar

56. YARN + HDFS
D1
D2
D3
Name
Node
Client
Client
Client
• YARN will try to make
sure data is processed
where it is stored
• ….. data locality

57. YARN + HDFS
• Blocks are immutable. This enables high write speeds
• Data is schema free! You can store any data you want
• Data locality is what differentiates HDFS from other data
storage
• You can read massive amounts of data only limited by
disk read speeds

58. MapReduce and others
OK… BUT HOW DO I
PROCESS ?

59. YARN
Tez MapReduce <Name here>
Libraries: Mahout, MLib, GraphX, Oryx
Languages: Hive, Pig, R, Spark SQL, Stinger

60. YARN
Tez <Name here>
Languages: Hive, Pig, R, Spark SQL, Stinger
Libraries: Mahout, Crunch, Mlib, GraphX, Oryx
MapReduce

61. MapReduce

62. Document
Deer Bear River
Car Car River
Deer Car BearDocument
stored in HDFS

63. Splitting
Deer Bear River
Deer Car Bear
Deer Bear River
Car Car River
Car Car River
Deer Car Bear

64. Mapping
Deer Bear River
Car Car River
Deer Car Bear
Deer 1
Bear 1
River 1
Car 1
Car 1
River 1
Deer 1
Car 1
Bear 1

65. Shuffling
Deer 1
Bear 1
River 1
Deer 1
Car 1
Bear 1
Car 1
Car 1
River 1
Deer 1
Deer 1
Deer 1
Bear 1
Bear 1
Car 1
Car 1
River 1
River 1

66. Reduce
Deer 1
Deer 1
Deer 1
Bear 1
Bear 1
Car 1
Car 1
River 1
River 1
Deer 3
Bear 2
Car 2
River 2
Deer 3
Bear 2
Car 2
River 2
HDFS

67. API: Mapper
interface

68. API: Reduce
interface

69. API: Main

70. How to run
$ bin/hadoop jar wc.jar WordCount /hdfs/dir/in /hdfs/dir/out

71. MapReduce
• Mappers and reducers are distributed in YARN
containers
• Chaining of MapReduce jobs make them slow
• Easy to scale but difficult to code
• … use the data DSL languages instead

72. Languages

73. YARN
Tez MapReduce <Name here>
Languages: Hive, Pig, R, Spark SQL, Stinger
Libraries: Mahout, Crunch, MLib, GraphX, Oryx

74. ”Languages”

75. PIG
• Procedural language
• Execute on YARN
• Great for
• Structuring
• Moving
• Transforming

76. Hive/Drill/Spark
SQL
• Declarative / SQL-like languages
• Great for
• Column data / Database dumps
• Aggregations
• Connect BI tools and Dashboards
• Data Warehouse for Hadoop++

77. Spark
• Core language (runs in YARN or standalone)
• Great for
• Anything that MapReduce can do
• Analytics, Machine Learning
• In memory and languages in Java, Scala and
Python

78. Summary
• Hadoop is designed to handle/process massive amounts of data
through HDFS and/or YARN
• The data do not need to be structured before it is stored in HDFS
• Hadoop is an ecosystem and have languages/frameworks for data
extraction, data management, data analysis and data integration
• It is most convenient to begin with Hadoop by testing distributions.
E.g. Hortonworks, Cloudera, MapR etc.
• Learn MapReduce and learn to understand languages and a few
integration tools

79. Is it a fad?