The document discusses GreenHDFS, a self-adaptive variant of HDFS that aims to reduce energy consumption. It does this through techniques like data classification to place "hot" and "cold" data in different zones, power management policies to transition servers to low-power states, and machine learning to predict file access patterns and inform placement. An evaluation of GreenHDFS found it reduced energy consumption by 24% and saved $2.1 million annually in a 38,000 server cluster.

1. EEDC
34330
Self-Adapting, Energy-
Execution Conserving Distributed
Environments for File Systems
Distributed
Computing
European Master in Distributed
Computing - EMDC
EEDC Presentation
Mário Almeida– 4knahs[@]gmail.com
www.marioalmeida.eu

2. Outline
● Introduction ● Conclusions
○ Green Computing ● References
○ Distributed File Systems
○ DFS issues
● Hadoop Distributed File System
○ Overview
○ Evaluation
● Green HDFS
○ Overview
○ Design
○ Goal
○ Energy-management
policies
○ Machine learning
○ Evaluation
*

3. Introduction - Green Computing
● Environmentally sustainable computing with minimal
impact on the environment.
● Reduction of the energy consumption, the GreenHouse
Gas emissions and the operational costs.
*

4. Introduction - Distributed FS
● A Distributed File System (DFS) is any file system that
allows access to files from multiple hosts sharing via a
computer network.
● May include facilities for transparent replication and
fault tolerance.
*

5. Introduction - DFS Issues
● Distributed File Systems are often built to run on a large
number of commodity servers.
● Which means that:
○ it generates heat and consumes large amounts of
energy.
○ costs are dependent on the initial acquisition costs
and power, cooling, etc.
*

6. Introduction - DFS Issues
● Common approach:
○ Scale-Down -Transitioning servers into low power
consumption states.
○ Other approaches not exclusive to DFS might
include renewable energy, free cooling, etc.
*

7. HDFS Overview
● Hadoop Distributed File System (HDFS) is the primary
storage system used by Hadoop applications.
● HDFS creates multiple replicas of data blocks and
distributes them on compute nodes throughout a cluster
of enable reliable, extremely rapid computations.
*

8. HDFS Evaluation
In 2010, a detailed analysis of files was done in a
production Yahoo! Hadoop cluster with the following
characteristics:
● 2600 servers
● 34 million files
● Over 5 PB of data
● 3 months of observation
*

9. HDFS Evaluation
Key observations:
● Files are heterogeneous in access and lifespan patterns.
● 60% of data is "cold" or dormant.
● 95-98% of files have a very short "hotness" lifespan of
less than 3 days.
● 90% of files were dormant or "cold" for more than 18
days.
● Majority of the data had a news-server-like access
pattern.
*

10. GHDFS Overview
● Self-Adaptive - depends only on HDFS and file access
patterns
● Applies Data-Classification techniques
● Energy-Aware placement of data
● Trades cost, performance and power by separating
cluster into logical zones.
*

11. GHDFS Design
Hot Zone Cold Zone
Files currently Files with low to
accessed and rare access
newly created
Low energy use
High energy usage and Sleeping
and performance mode
*

12. GHDFS - Management Policies
GreenHDFS uses three different management
policies:
● FMP - File Migration Policy Hot Zone
● SCP - Server Power Conserver Policy
Cold Zone
● FRP - File Reversal Policy
*

13. GHDFS - File Migration Policy
● FMP monitors the dormancy of files
● Runs in the Hot Zone
● Gives higher storage effiency for the Hot Zone as less
accessed files are moved to the Cold Zone
Coldness > Threshold
Hot Zone Cold Zone
Hotness > Threshold
*

14. GHDFS - Power Conserver Policy
● SCP runs in the ColdZone
● Determines which servers can go to stanby/sleep mode.
● Uses hardware techniques to transfer CPU, Disks and
FRAM into low power state.
● Wakes the server up only if:
○ Data on that server is accessed
○ New data needs to be placed on that server
Cold
Zone
*

15. GHDFS - File Reversal Policy
● FRP runs in the ColdZone.
● Ensures QoS, bandwidth and response time is well
managed in case a file becomes popular.
#accesses > Threshold
Hot Zone Cold Zone
*

16. GHDFS - Machine Learning
● Designing and developing algorithms that allow
computers to evolve behaviors based on empirical
data.
● Recognize patterns and make decisions based on data.
*

17. GHDFS - Machine Learning
GHDFS uses:
● Supervised machine learning.
● A variant of Multiple Linear Regression to find the
statistical correlation between directory and file
attributions.
● Training data preparation - audit logs and metadata.
● Predicts the files Lifespan, Size and Heat upon creation
of file.
It works because there is a high correlation between the
directory hierarchy and file attributes in a well-laid out and
partitioned name space!!
*

18. GHDFS - Machine Learning
*

19. GHDFS - Evaluation
*

20. GHDFS - Evaluation
*

21. GHDFS - Evaluation
*

22. GHDFS - Evaluation
*

23. GHDFS - Evaluation
● Energy consumption reduced by 24% and saved $2.1
millions saved in energy costs per annum (38000
servers).
● Maximizes the usage of the power budget by allowing
the infrastructure to expand. More Hot Zone servers
offer more availability and performance.
*

24. Conclusions
● Machine learning can be applied for a predictive self-
managed energy control system that achieves better
results than reactive approaches.
● Good Energy Management Policies can result in high
savings in energy consumption.
● Data-Classification techniques can help achieving a
better energy-aware placement of data in Distributed
File Systems.
● The presented techniques applied in conjunction to
other more common green computing technologies can
impact significantly the maintenance costs of the cluster.
*

25. References
● GreenHDFS : Torwards an Energy-Conserving Storage-
Efficient, Hybrid Hadoop Compute Cluster
● Evaluation and Analysis of GreenHDFS: A Self-
Adaptive, Energy-Conserving Variant of the Hadoop
Distributed File System
● Predictive Data and Energy Management in
GreenHDFS
● The Hadoop Distributed File System
● Introduction to Machine Learning (Adaptive Computation
and Machine Learning)
*

26. EEDC
34330
Self-Adapting, Energy-
Execution Conserving Distributed
Environments for File Systems
Distributed
Computing
European Master in Distributed
Computing - EMDC
EEDC Presentation
Mário Almeida– 4knahs[@]gmail.com
www.marioalmeida.eu