This document discusses cloud and big data technologies. It provides an overview of Hadoop and its ecosystem, which includes components like HDFS, MapReduce, HBase, Zookeeper, Pig and Hive. It also describes how data is stored in HDFS and HBase, and how MapReduce can be used for parallel processing across large datasets. Finally, it gives examples of using MapReduce to implement algorithms for word counting, building inverted indexes and performing joins.

1. Cloud and Big Data
Farzad Nozarian (fnozarian@aut.ac.ir)
Amirkabir University of Technology
With the help of Dr. Amir H. Payberah (amir@sics.se)

3. Big Data Analytics Stack

4. Hadoop Big Data Analytics
Stack

5. Spark Big Data Analytics
Stack

6. Big Data - File systems
• Traditional file-systems are not well-designed for large-scale data
processing systems.
• Efficiency has a higher priority than other features, e.g., directory
service.
• Massive size of data tends to store it across multiple machines in a
distributed way.
• HDFS/GFS, Amazon S3, ...

7. Big Data - Database
• Relational Databases Management Systems (RDMS) were not designed to be
distributed.
• NoSQL databases relax one or more of the ACID properties: BASE
• Different data models: key/value, column-family, graph, document.
• Hbase/BigTable, Dynamo, Scalaris, Cassandra, MongoDB, Voldemort, Riak,
Neo4J, ...

8. Big Data - Resource Management
• Different frameworks require different computing resources.
• Large organizations need the ability to share data and resources between
multiple frameworks.
• Resource management share resources in a cluster between multiple
frameworks while providing resource isolation.
• Mesos, YARN, Quincy, ...

9. Big Data - Execution Engine
• Scalable and fault tolerance parallel data processing on clusters of unreliable
machines.
• Data-parallel programming model for clusters of commodity machines.
• MapReduce, Spark, Stratosphere, Dryad, Hyracks, ...

10. Big Data - Query/Scripting Language
• Low-level programming of execution engines, e.g., MapReduce, is not easy
for end users.
• Need high-level language to improve the query capabilities of execution
engines.
• It translates user-defined functions to low-level API of the execution
engines.
• Pig, Hive, Shark, Meteor, DryadLINQ, SCOPE, ...

11. Big Data - Stream Processing
• Providing users with fresh and low latency results.
• Database Management Systems (DBMS) vs. Data Stream Management Systems (DSMS)
• Storm, S4, SEEP, D-Stream, Naiad, ...

12. Big Data - Graph Processing
• Many problems are expressed using graphs: sparse computational
dependencies, and multiple iterations to converge.
• Data-parallel frameworks, such as MapReduce, are not ideal for these
problems: slow
• Graph processing frameworks are optimized for graph-based problems.
• Pregel, Giraph, GraphX, GraphLab, PowerGraph, GraphChi, ...

13. Big Data - Machine Learning
• Implementing and consuming machine learning techniques at scale are
difficult tasks for developers and end users.
• There exist platforms that address it by providing scalable machine learning
and data mining libraries.
• Mahout, MLBase, SystemML, Ricardo, Presto, ...

14. Big Data - Configuration and Synchronization
Service
• A means to synchronize distributed applications accesses to shared resources.
• Allows distributed processes to coordinate with each other.
• Zookeeper, Chubby, ...

15. Hadoop Ecosystem

16. Hadoop Ecosystem-HDFS
• A foundational component of the Hadoop ecosystem is the Hadoop
Distributed File System (HDFS).
• HDFS is the mechanism by which a large amount of data can be distributed
over a cluster of computers, and data is written once, but read many times
for analytics.
• It provides the foundation for other tools, such as HBase.

17. Hadoop Ecosystem-MapReduce
• Hadoop’s main execution framework is MapReduce, a programming model
for distributed, parallel data processing, breaking jobs into mapping phases
and reduce phases.
• Developers write MapReduce jobs for Hadoop, using data stored in HDFS for
fast data access.
• Because of the nature of how MapReduce works, Hadoop brings the
processing to the data in a parallel fashion, resulting in fast implementation.

18. Hadoop Ecosystem-HBase
• A column-oriented NoSQL database built on top of HDFS, HBase is used
for fast read/write access to large amounts of data.
• HBase uses Zookeeper for its management to ensure that all of its
components are up and running.

19. Hadoop Ecosystem-Zookeeper
• Zookeeper is Hadoop’s distributed coordination service.
• Designed to run over a cluster of machines, it is a highly available service
used for the management of Hadoop operations, and many components of
Hadoop depend on it.

20. Hadoop Ecosystem-Oozie
• A scalable workflow system
• Oozie is integrated into the Hadoop stack, and is used to coordinate
execution of multiple MapReduce jobs.
• It is capable of managing a significant amount of complexity, basing
execution on external events that include timing and presence of required
data.

21. Hadoop Ecosystem-Pig
• An abstraction over the complexity of MapReduce programming
• the Pig platform includes an execution environment and a scripting language
(Pig Latin) used to analyze Hadoop data sets.
• Its compiler translates Pig Latin into sequences of MapReduce programs.

22. Hadoop Ecosystem-Hive
• An SQL-like, high-level language used to run queries on data stored in
Hadoop
• Hive enables developers not familiar with MapReduce to write data queries
that are translated into MapReduce jobs in Hadoop.

23. Hadoop Ecosystem-Sqoop
• a connectivity tool for moving data between relational databases and data
warehouses and Hadoop.
• Sqoop leverages database to describe the schema for the imported/exported
data and MapReduce for parallelization operation and fault tolerance.

24. Hadoop Ecosystem-Flume
• a distributed, reliable, and highly available service for efficiently collecting,
aggregating, and moving large amounts of data from individual machines to
HDFS.
• provides a streaming of data flows
• allowing to move data from multiple machines within an enterprise into
Hadoop.

25. Beyond the core components
• Whirr — This is a set of libraries that allows users to easily spin-up Hadoop
clusters on top of Amazon EC2, Rackspace, or any virtual infrastructure.
• Mahout — This is a machine-learning and data-mining library that provides
MapReduce implementations for popular algorithms used for clustering, regression
testing, and statistical modeling.
• BigTop — This is a formal process and framework for packaging and
interoperability testing of Hadoop’s sub-projects and related components.
• Ambari — This is a project aimed at simplifying Hadoop management by
providing support for provisioning, managing, and monitoring Hadoop clusters.

26. Storing Data in Hadoop
HDFS - HBase

27. HDFS-Architecture
• The HDFS design is based on the design of the Google File System (GFS).
• To be able to store a very large amount of data (terabytes or petabytes)
• HDFS is designed to spread the data across a large number of machines, and to
support much larger file sizes compared to distributed filesystems such as NFS.
• HDFS uses data replication
• To better integrate with Hadoop’s MapReduce, HDFS allows data to be read and
processed locally.

28. HDFS-Architecture
HDFS is implemented as a block-structured file system

29. HDFS-Using HDFS Files
• User applications access the HDFS file system using an HDFS client
• ACCESSING HDFS
• FileSystem (FS) shell
• HDFS Java APIs

30. HDFS-Using HDFS Files

31. HBase-Architecture
• HBase is a distributed, versioned, column-oriented, multidimensional storage system,
designed for high performance and high availability.
• HBase is an open source implementation of Google’s BigTable architecture.
• Similar to traditional relational database management systems (RDBMSs), data in HBase is
organized in tables.
• Unlike RDBMSs, however, HBase supports a very loose schema definition, and does not
provide any joins, query language, or SQL.
• The main focus of HBase is on Create, Read, Update, and Delete (CRUD) operations on
wide sparse tables.
• HBase leverages HDFS for its persistent data storage.

32. Processing Data with MapReduce

33. MapReduce-Roadmap
• Understanding MapReduce fundamentals
• Getting to know MapReduce application execution
• Understanding MapReduce application design

34. MAPREDUCE-GETTING TO KNOW
• MapReduce is a framework for executing highly parallelizable and
distributable algorithms across huge data sets using a large number of
commodity computers.
• inspired by these concepts and introduced by Google in 2004
• MapReduce was introduced to solve large-data computational problems, and
is specifically designed to run on commodity hardware.
• It is based on divide-and-conquer principles — the input data sets are split into
independent chunks, which are processed by the mappers in parallel.

35. MAPREDUCE-GETTING TO KNOW

37. MAPREDUCE-Execution
Pipeline

38. MAPREDUCE-Runtime Coordination and
Task Management

39. word count implementation-Map Phase

40. word count implementation-Reduce Phase

41. word count implementation-
Driver

42. DESIGNING MAPREDUCE
IMPLEMENTATIONS

43. Necessary questions to reformulate the initial
problem in terms of MapReduce
• How do you break up a large problem into smaller tasks? More specifically,
how do you decompose the problem so that the smaller tasks can be
executed in parallel?
• Which key/value pairs can you use as inputs/outputs of every task?
• How do you bring together all the data required for calculation? More
specifically, how do you organize processing the way that all the data
necessary for calculation is in memory at the same time?

44. Simple Data Processing with
MapReduce

45. Inverted Indexes Example

47. Building Joins with MapReduce
• Two “standard” implementations exist for joining data in MapReduce:
• Reduce-side join
• Map-side join
• A most common implementation of a join is a reduce-side join.
• Map-side join is very well in the case of one-to-one joins, where at most one
record from every data set has the same key.

48. Road Enrichment Example

49. A simplified road enrichment algorithm
1. Find all links connected to a given node. For example, as shown in Figure,
node N1 has links L1, L2, L3, and L4, while node N2 has links L4, L5, and
L6.
2. Based on the number of lanes for every link at the node, calculate the road
width at the intersection.
3. Based on the road width, calculate the intersection geometry.
4. Based on the intersection geometry, move the road’s end point to tie it to
the intersection geometry.

50. Algorithm assumptions
• A node is described with an object N with the key NN1… NNm. For example, node
N1 can be described as NN1and N2 as NN2. All nodes are stored in the nodes input
file.
• A link is described with an object L with the key LL1… LLm. For example, link L1
can be described as LL1 , L2 as LL2, and so on. All the links are stored in the links
source file.
• Also introduce an object of the type link or node (LN), which can have any key.
• Finally, it is necessary to define two more types — intersection (S) and road (R).

51. Phase 1
Calculation of Intersection Geometry and
Moving the Road’s End Points Job

52. Phase 2
Merge Roads Job

53. Links Elevation Example
• This problem can be defined as follows. Given a links graph and terrain
model, convert two dimensional (x,y) links into three-dimensional (x, y, z)
links. This process is called link elevation.

54. Simplified link elevation
algorithm
1. Split every link into fixed-length
fragments (for example, 10 meters).
2. For every piece, calculate heights (from
the terrain model) for both start and
end points of each link.
3. Combine pieces together into original
links.

55. Phase 1
Split Links into Pieces and Elevate Each
Piece Job

56. Phase 2
Combine Link’s Pieces into Original Links Job