This document outlines an introductory workshop on big data held by the BigData Community. The workshop agenda includes an introduction to big data and the Hadoop ecosystem, demonstrations of Hadoop installation in standalone and pseudo-distributed modes, and a hands-on Java application example. Attendees are guided through setting up a test environment, downloading and configuring Hadoop, and testing the installation. The goal is to provide 120 students and 5 universities with an awareness of big data science and engineering through hands-on training.

1. Intro to Big Data
Workshop
14 March
Big Data Hub
https://www.facebook.com/BigDataHub

2. Introduction
2
Make it Easy
BidData Community : Intro to BigData Workshop

3. Introduction
3BidData Community : Intro to BigData Workshop
Who, we are…..

4. Introduction
4BidData Community : Intro to BigData Workshop
Community Vision
Building a Big Data Knowledge hub in Egypt.

5. Introduction
5BidData Community : Intro to BigData Workshop
Mission
Spreading BD science and engineering
awareness

6. Introduction
6BidData Community : Intro to BigData Workshop
Objectives
● Training 120 4th year student from
Engineering and computer science faculties.
● Running 5 awareness sessions targeting five
universities in the first phase.

7. Introduction
7BidData Community : Intro to BigData Workshop
Monofia University

8. Introduction
8BidData Community : Intro to BigData Workshop
Monofia University

9. Introduction
9BidData Community : Intro to BigData Workshop
Call for volunteers
Business
Development
Administration
Development
Administrative
& Logistics

10. Introduction
10BidData Community : Intro to BigData Workshop
Today RoadMap
1. Big Data Introduction
2. Hadoop installation
3. Small Java application

11. Introduction
11BidData Community : Intro to BigData Workshop
Ready ?

12. Eng. Youssef Eassa
BigData and Ecosystem

13. BigData & Ecosystem
13BidData Community : Intro to BigData Workshop
Data Vs. Information Vs. Knowledge

14. BigData & Ecosystem
14BidData Community : Intro to BigData Workshop
Data Scientist VS Data Engineer

15. BigData & Ecosystem
15BidData Community : Intro to BigData Workshop
What is the maximum file size
you have?
Movies/Files/Streaming video that you have used?
What is the maximum download
speed you get?
How much time to just transfer?

16. BigData & Ecosystem
16BidData Community : Intro to BigData Workshop
How do you process
Distributed System
Massive data?
Distributed Computing System

17. BigData & Ecosystem
17BidData Community : Intro to BigData Workshop
Now, What is difference between
Big DataMassive Data and

18. BigData & Ecosystem
18BidData Community : Intro to BigData Workshop
From Where Data are Generated?
Social media and networks
(all of us are generating data)
Scientific instruments
(collecting all sorts of data)
Mobile devices
(tracking all objects all the time)
Sensor technology and networks
(measuring all kinds of data)

19. BigData & Ecosystem
19BidData Community : Intro to BigData Workshop
Big Data in one Minute

20. BigData & Ecosystem
20BidData Community : Intro to BigData Workshop
What makes Big Data that big?

21. BigData & Ecosystem
21BidData Community : Intro to BigData Workshop
Data Evolution

22. BigData & Ecosystem
22BidData Community : Intro to BigData Workshop
Data Evolution

23. BigData & Ecosystem
23BidData Community : Intro to BigData Workshop
The Problem

24. BigData & Ecosystem
24BidData Community : Intro to BigData Workshop
Hadoop !=
Database
Hadoop
Hadoop is a
Distributed Storage and
Computation Framework

25. BigData & Ecosystem
25BidData Community : Intro to BigData Workshop
BIGDATA is not just HADOOP

26. BigData & Ecosystem
26BidData Community : Intro to BigData Workshop
BigData Platforms

27. BigData & Ecosystem
27BidData Community : Intro to BigData Workshop
Core Components of Hadoop
Storage (HDFS) Processing (MapReduce)

28. BigData & Ecosystem
28BidData Community : Intro to BigData Workshop

29. BigData & Ecosystem
29BidData Community : Intro to BigData Workshop
HDFS to Store Big Data

30. BigData & Ecosystem
30BidData Community : Intro to BigData Workshop

31. BigData & Ecosystem
31BidData Community : Intro to BigData Workshop
MapReduce :

32. BigData & Ecosystem
32BidData Community : Intro to BigData Workshop
Hadoop 2.0: Next-gen platform

33. BigData & Ecosystem
33BidData Community : Intro to BigData Workshop
Store all data in one place
Interact with data in multiple ways

34. BigData & Ecosystem
34BidData Community : Intro to BigData Workshop
Hadoop 2.0 Projects
• YARN
• HDFS Federation Aka HDFS 2.0

35. BigData & Ecosystem
35BidData Community : Intro to BigData Workshop
YARN: Architecture

36. BigData & Ecosystem
36BidData Community : Intro to BigData Workshop
HDFS Federation: Architecture

37. BigData & Ecosystem
37BidData Community : Intro to BigData Workshop
Big Data Ecosystem

38. BigData & Ecosystem
38BidData Community : Intro to BigData Workshop

39. BigData & Ecosystem
39BidData Community : Intro to BigData Workshop

40. BigData & Ecosystem
40BidData Community : Intro to BigData Workshop
It’s the time for
Implementaion

41. Eng. Ahmed Shouman
BigData Practical Activity

42. Hadoop Installation
BigData practice
42BidData Community : Intro to BigData Workshop

43. What’s Hadoop?
Is a Java-based programming framework that supports the
processing of large data sets in a distributed computing
environment.
43BidData Community : Intro to BigData Workshop
BigData practice

44. Hadoop installation modes
○ Stand alone mode.
○ Pseudo distributed mode.
○ Fully distributed mode.
44BidData Community : Intro to BigData Workshop
BigData practice

45. What’s Hadoop?
45BidData Community : Intro to BigData Workshop
BigData practice

46. Setting up our Lab:
46BidData Community : Intro to BigData Workshop
BigData practice

47. Prerequisites:
47BidData Community : Intro to BigData Workshop
BigData practice

48. Install JAVA & Configure hosts file
○ sudo apt-get update
○ sudo apt-get install sun-java6-jdk
○ Java –version
○ Hosts File
○ #gedit /etc/hosts
48BidData Community : Intro to BigData Workshop
BigData practice

49. Install & Configure SSH
○ sudo apt-get install openssh-server
○ ssh-keygen -t rsa -P "“
49BidData Community : Intro to BigData Workshop
BigData practice

50. Download & Install
50BidData Community : Intro to BigData Workshop
BigData practice

51. Download & Install
51BidData Community : Intro to BigData Workshop
BigData practice

52. Download & Install
52BidData Community : Intro to BigData Workshop
BigData practice

53. Download & Install
53BidData Community : Intro to BigData Workshop
BigData practice

54. Download & Install
54BidData Community : Intro to BigData Workshop
BigData practice

55. Download & Install
55BidData Community : Intro to BigData Workshop
BigData practice

56. Download & Install
○ In the Linux Terminal, Write:
“ wget http://supergsego.com/apache/hadoop/common/hadoop-1.2.1/hadoop-
1.2.1-bin.tar.gz ” & hit ENTER
56BidData Community : Intro to BigData Workshop
BigData practice

57. Editing .bashrc file
○ #gedite ~/.bashrc
○ Add the following lines at the end of the file
57BidData Community : Intro to BigData Workshop
BigData practice

58. Main Installation
○ #tar –zxvf hadoop-1.2.1-bin.tar.gz
58BidData Community : Intro to BigData Workshop
BigData practice

59. Editing hadoop-env.sh
○ #gedite /opt/hadoop/conf/hadoop-env.sh
59BidData Community : Intro to BigData Workshop
BigData practice

60. Editing conf/*-site.xml files
○ 1- “Core-site.xml” File:
○ #gedit /opt/hadoop/conf/core-site.xml
60BidData Community : Intro to BigData Workshop
BigData practice

61. Editing conf/*-site.xml files
○ 2-”Mapred-site.xml” File
○ #gedit /opt/hadoop/conf/mapred-site.xml
61BidData Community : Intro to BigData Workshop
BigData practice

62. Editing conf/*-site.xml files
○ 3-”hdfs-site.xml” File.
○ #gedit /opt/hadoop/conf/hdfs-site.xml
62BidData Community : Intro to BigData Workshop
BigData practice

63. Formatting Namenode F.S.
○ #hadoop namenode –format
63BidData Community : Intro to BigData Workshop
BigData practice

64. Firing Hadoop Deamons
○ #start-all.sh
64BidData Community : Intro to BigData Workshop
BigData practice

65. Testing Installation
○ Localhost:50070
65BidData Community : Intro to BigData Workshop
BigData practice

66. Testing Installation
○ Localhost:50070
66BidData Community : Intro to BigData Workshop
BigData practice

67. 67BidData Community : Intro to BigData Workshop
BigData practice

68. Thank You
Special thanks to
68BidData Community : Intro to BigData Workshop
BigData practice

69. Questions?
Like us on FB:
https://www.facebook.com/BigDataHub
69BidData Community : Intro to BigData Workshop
BigData Workshop