This document provides an overview of cloud computing and distributed systems. It discusses large scale distributed systems, cloud computing paradigms and models, MapReduce and Hadoop. MapReduce is introduced as a programming model for distributed computing problems that handles parallelization, load balancing and fault tolerance. Hadoop is presented as an open source implementation of MapReduce and its core components are HDFS for storage and the MapReduce framework. Example use cases and running a word count job on Hadoop are also outlined.

1. the cloud
COMPUTING

2. Outline
 Large-scale Distributed Systems
 Introduction to Cloud Computing
 Cloud Computing paradigms and models
 Introduction to MapReduce
 Alternative architectures
 Writing Application using Hadoop

3. Distributed Systems
 Set of discrete machines which cooperate to perform
computation
 Give the notion of a single “machine”
 Keep the distribution transparent
 Examples:
 Compute clusters
 Distributed storage systems, such as Dropbox, Google Drive, etc.
 The Web

4. Characteristics
 Ordering
 Time is used to ensure ordering
 In most cases, only need to know that a happened before
b, known as the happens-before relation
 Distributed Mutual Exclusion
 Concurrent access to shared resources needs to be
synchronized
 Central lock server: All lock requests are handled by a
central server
 Token passing: Arrange nodes into a ring and a token is
passed around
 Totally-ordered multicast: Clients multicast requests to
each other

5. Characteristics (2)
 Distributed transactions
 Distributed transactions span multiple transaction processing servers
 Actions need to be coordinated across multiple parties
 Replication
A number of distributed systems involve replication
 Data replication: Multiple copies of some object stored at different servers
 Computation replication: Multiple servers capable of providing an
operation
 Advantages
1. Load balancing: Work spread out across clients
2. Lower latency: Better performance if replica close to the client
3. Fault tolerance: Failure of some replicas can be tolerated

6. CAP
 CAP
 Consistency: All nodes see the same state
 Availability: All requests get a response
 Partitioning: System continues to operate even in the
face of node failure
 Brewer‟s conjecture states that in a distributed
system only 2 out of 3 possible
 In the current setup, partitioning is a given:
Hardware/software fails all the time
 Therefore, systems need to choose between
consistency and availability

7. Advantages
 Scalability:
 The scale of the Internet (think how many queries Google servers handle daily)
 Only a matter of adding more machines
 Cheaper than super computers
 More machines means more parallelism, hence better performance
 Sharing:
 The same resource is shared between multiple users
 Just like the Internet is shared between millions of users
 Communication:
 Communication between (potentially geographically isolated) machines and
users (via email, Facebook, etc.)
 Reliability:
 The service can remain active even if multiple machines go down

8. Challenges
 Concurrency:
 Concurrent execution requires some form of coordination
 Fault-tolerance:
 Any component can fail at any instant due to a software or a
hardware bug
 Security:
 One machine can compromise the entire system
 Coordination:
 No global time so non-trivial to coordinate
 Trouble shooting:
 Hard to trouble shoot because hard to reason about the
system

9. Introduction to Cloud
Computing
 An emerging IT development, deployment, and delivery model that enables
real-time delivery of a broad range of IT products, services and solutions over
the internet
 A realization of utility computing in which computation, storage, and services
are offered as a metered service
 Grid Computing: form of distributed computing, acting
in concert to perform very large tasks
 Utility Computing: metered service similar to a
traditional public utility
 Autonomic Computing: capable of self-management
 Cloud Computing: deployments as of 2009 depend on
grids, have autonomic characteristics and bill like
utilities

10. Characteristics
 On-demand self-service: allows users to obtain,
configure and deploy cloud services themselves using
cloud service catalogues, without requiring the
assistance of IT.
 Broad network access: capabilities are available over
the network and accessed through standard
mechanisms that promote use by heterogeneous thin
or thick client platforms
 Resource pooling: The provider‟s computing resources
are pooled to serve multiple consumers using a multi-
tenant model, with different physical and virtual
resources dynamically assigned and reassigned
according to consumer demand.

11. Characteristics (2)
 Rapid elasticity: Capabilities can be rapidly and
elastically provisioned, in some cases automatically,
to quickly scale out and rapidly released to quickly
scale in. To the consumer, the capabilities available
for provisioning often appear to be unlimited and
can be purchased in any quantity at any time.
 Measured service: Cloud systems automatically
control and optimize resource use by leveraging a
metering capability at some level of abstraction
appropriate to the type of service (e.g., storage,
processing, bandwidth, and active user accounts).

12. Cloud Service Models
 SaaS – Software as a Service: Network-hosted
application
 PaaS– Platform as a Service: Network-hosted software
development platform
 IaaS – Infrastructure as a Service: Provider hosts
customer VMs or provides network storage
 DaaS – Data as a Service: Customer queries against
provider’s database
 IPMaaS – Identity and Policy Management as a
Service: Provider manages identity and/or access
control policy for customer
 NaaS – Network as a Service: Provider offers virtualized
networks (e.g. VPNs)

13. Deployment Models
 Private Cloud: infrastructure is operated solely for an
organization.
 Public Cloud: infrastructure is made available to the general
public as a pay-as-you-go model, e.g. Amazon Web Services,
Google AppEngine, and Microsoft Azure
 Community Cloud: infrastructure between several
organizations from a specific community with common
concerns (security, compliance, jurisdiction, etc.), whether
managed internally or by a third-party and hosted internally or
externally.
 Hybrid Cloud: infrastructure is a combination of two or more
clouds(private, community, or public) that remain unique
entities but are bound together by standardized or proprietary
technology that enables data and application portability
between environments.

14. Private Cloud

15. Private Outsourced Cloud

16. Public Cloud

17. Hybrid Cloud

18. Advantages
Advantages to both service providers and end users
 Service providers:
 Simplified software installation and maintenance
 Centralized control over versioning
 No need to build, provision, and maintain a datacenter
 On the fly scaling
 End users:
 “Anytime, anywhere” access
 Share data and collaborate easily
 Safeguard data stored in the infrastructure

19. Obstacles
 Bugs in large-scale distributed systems: Hard to
debug large-scale applications in full deployment
 Scaling quickly: Automatically scaling while
conserving resources and money is an open ended
problem
 Reputation fate sharing: Bad behavior by one
tenant can reflect badly on the rest
 Software licensing: Gap between pay-as-you-go
model and software licensing

20. Obstacles (2)
 Service availability: Possibility of cloud outage
 Data lock-in: Dependence on cloud specific APIs
 Security: Requires strong encrypted storage, VLANs,
and network middle-boxes (firewalls, etc.)
 Data transfer bottlenecks: Moving large amounts of
data in and out is expensive
 Performance unpredictability: Resource sharing
between applications
 Scalable storage: No standard model to arbitrarily
scale storage up and down on-demand while
ensuring data durability and high availability

21. Introduction to MapReduce
 A simple programming model that applies to many
large-scale computing problems
 Hide messy details in MR runtime library:
 Automatic parallelization
 Load balancing
 Network and disk transfer optimization
 Handling of machine failures
 Robustness
 Improvements to core library benefit all users of library

22. Google MapReduce – Idea
 The core idea behind MapReduce is mapping your
data set into a collection of <key, value> pairs, and
then reducing over all pairs with the same key.
 Map
 Apply function to all elements of a list
 square x = x * x;
 Map square [1, 2, 3, 4, 5];
 [1, 4, 9, 16, 25]
 Reduce
 Combine all elements of a list
 Reduce (+)[1, 2, 3, 4, 5];
 15

23. Google MapReduce –
Overview

24. MapReduce architecture
 Master: In charge of all meta data, work scheduling
and distribution, and job orchestration
 Workers: Contain slots to execute map or reduce
functions
 Mappers:
 A map worker reads the contents of the input split that it has
been assigned
 It parses the file and converts it to key/value pairs and invokes
the user-defined map function for each pair
 The intermediate key/value pairs after the application of the
map logic are collected (buffered) in memory
 Once the buffered key/value pairs exceed a threshold they are
written to local disk and partitioned (using a partitioning
function) into R partitions. The location of each partition is passed
to the master

25. MapReduce architecture (2)
 Workers: Contain slots to execute map or reduce
functions
 Reducers:
 A reduce worker gets locations of its input partitions from the
master and uses HTTP requests to retrieve them
 Once it has read all its input, it sorts it by key to group
together all occurrences of the same key
 It then invokes the user-defined reduce for each key and
passes it the key and its associated values
 The key/value pairs generated after the application of the
reduce logic are then written to a final output file, which is
subsequently written to the distributed filesystem

26. Google File System - GFS
 In-house distributed file system at Google
 Stores all input an output files
 Stores files…
 divided into 64 MB blocks
 on at least 3 different machines
 Machines running GFS also run MapReduce

27. MapReduce job phases
A MapReduce job can be divided into 4 phases:
 Input split: The input dataset is sliced into M splits, one
per map task
 Map logic: The user-supplied map function is invoked
 In tandem a sort phase is also applied that ensures that
map output is locally sorted by key
 In addition, the key space is also partitioned amongst the
reducers
 Shuffle: Map output is relayed to all reduce tasks
 Reduce logic: The user-provided reduce function is
invoked
 Before the application of the reduce function, the input
keys are merged to get globally sorted key/value pairs

28. Google MapReduce –
Example

29. Wordcount map in Java
1. public void map(Object key, Text value , Context
context) {
2. StringTokenizer itr = new
StringTokenizer(value.toString());
3. while (itr.hasMoreTokens()) {
4. word.set(itr.nextToken());
5. context.write(word , one);
6. }
7. }

30. Wordcount reduce in Java
1. public void reduce(Text key, Iterable <IntWritable >
values ,
Context context) {
2. int sum = 0;
3. for (IntWritable val : values) {
4. sum += val.get();
5. }
6. result.set(sum);
7. context.write(key, result);
8. }

31. Hadoop
 Open-source implementation of MapReduce,
developed by Doug Cutting originally at Yahoo! in
2004
 Now a top-level Apache open-source project
 Implemented in Java (Google‟s in-house
implementation is in C++)
 Jobs can be written in C++, Java, Python, etc.
 Comes with an associated distributed filesystem,
HDFS (clone of GFS)

32. Hadoop Components
 Hadoop consists of two core components
– The Hadoop Distributed File System (HDFS)
– MapReduce Software Framework
 There are many other projects based around
core Hadoop
– Often referred to as the „Hadoop
Ecosystem‟
– Pig, Hive, HBase,
Flume, Oozie, Sqoop, etc

33. Hadoop Users
 Adobe: Several areas from social services to
unstructured data storage and processing
 eBay: 532 nodes cluster storing 5.3PB of data
 Facebook: Used for reporting/analytics; one cluster
with 1100 nodes (12PB) and another with 300 nodes
(3PB)
 LinkedIn: 3 clusters with collectively 4000 nodes
 Twitter: To store and process Tweets and log files
 Yahoo!: Multiple clusters with collectively 40000
nodes; largest cluster has 4500 nodes!

34. Running a Hadoop Application
 The first order of the day is to format the Hadoop
DFS
 Jump to the Hadoop directory and execute:
bin/hadoop namenode -format
 Running Hadoop
 To run Hadoop and HDFS:
bin/start-all.sh
 To terminate them:
bin/stop-all.sh

35. Running a Hadoop Application
 Generating a dataset
 Create a temporary directory to hold the data:
 mkdir /tmp/gutenberg
 Jump to it:
 cd /tmp/gutenberg
 Download text files:
 wget www.gutenberg.org/etext/20417
 wget www.gutenberg.org/etext/5000
 wget www.gutenberg.org/etext/4300

36. Running a Hadoop Application
 Copying the dataset to the HDFS
 Jump to the Hadoop directory and execute:
 bin/hadoop dfs -copyFromLocal /tmp/gutenberg
/ccw/Gutenberg
 Running Wordcount
 bin/hadoop jar hadoop-examples-1.0.4.jar wordcount
/ccw/gutenberg /ccw/gutenberg-output
 Retrieving results from the HDFS
 Copy to the local FS:
 bin/hadoop dfs –getmerge /ccw/gutenberg-output
/tmp/gutenberg-output

37. Running a Hadoop Application
 Accessing the web interface
 JobTracker: http://localhost:50030
 TaskTracker: http://localhost:50060
 Reference: Running Hadoop on Ubuntu Linux
(Single-Node Cluster):
 http://www.michael-noll.com/tutorials/running-
hadoop-on-ubuntu-linux-single-node-cluster/

38. Thanks