An Experiment on Bare-Metal BigData Provisioning: Many BigData customers use on-demand platforms in the cloud, where they can get a dedicated virtual cluster in a couple of minutes and pay only for the time they use. Increasingly, there is a demand for bare-metal bigdata solutions for applications that cannot tolerate the unpredictability and performance degradation of virtualized systems. Existing bare-metal solutions can introduce delays of 10s of minutes to provision a cluster by installing operating systems and applications on the local disks of servers. This has motivated recent research developing sophisticated mechanisms to optimize this installation. These approaches assume that using network mounted boot disks incur unacceptable run-time overhead. Our analysis suggest that while this assumption is true for application data, it is incorrect for operating systems and applications, and network mounting the boot disk and applications result in negligible run-time impact while leading to faster provisioning time.

1. An Experiment on Bare-Metal
BigData Provisioning
Ata Turk, Ravi S. Gudimetla, Emine Ugur Kaynar, Jason Hennessey,
Sahil Tikale, Peter Desnoyers, Orran Krieger
1

2. BigData Analytics on the Cloud
• BigData deployments are
moving to the cloud
• On-demand usage (Cost), Elasticity,
Agility, Simplicity, …
• Virtualized IaaS solutions: Amazon
EMR, Azure HDInsight, …
• Virtualization drawbacks
• Overhead, unpredictability, security
concerns, device functionality, …
• Bare-metal cloud solutions: IBM,
Rackspace, and Internap, …
2

3. Bare-Metal BigData
Cloud Solutions
• Bare-Metal cloud provisioning
• Automated provisioning: Ironic,
MaaS, …
• Image copy to local disk => long
waits => loss of agility & elasticity
• OS streaming*, Lazy copy &
de-virtualization**
• What about network booting?
• incur an ongoing unacceptable
overhead during runtime
** Y. Omote, T. Shinagawa, and K. Kato, “Improving Agility and Elasticity in Bare-metal Clouds,” in ASPLOS’15, pp. 145–159, 2015.
3
* David Clerc, “OS Streaming Deployment”, in IPCCC’10, pp. 169–179, 2010.

4. • Large parts of the HPC community has been
doing it for the last 20 years.
• Virtualized IaaS is doing it all the time.
• Why not bare-metal cloud?
4

5. Network-Mounted BigData
System
• Clients access kernel and init
ramdisk via PXE
• Mount OS & BigData apps
from a remote iSCSI volume
• Use local disk for ephemeral
storage (HDFS, /swap, /tmp,
…)
5

6. Bare-Metal BigData
Provisioning Prototype
6

7. Bare-Metal BigData
Provisioning Prototype
Hardware Isolation Layer:
A service to allocate bare-
metal nodes out of a shared
pool and isolate network
7

8. Bare-Metal BigData
Provisioning Prototype
CEPH: Central image
store hosting user images
with BigData applications
8

9. Bare-Metal BigData
Provisioning Prototype
Provisioning VM:
Gateway between
isolated servers
and image store
9

10. Bare-Metal BigData
Provisioning Prototype
10

11. Bare-Metal BigData
Provisioning Prototype
11

12. Bare-Metal BigData
Provisioning Prototype
12

13. Bare-Metal BigData
Provisioning Prototype
13

14. Provisioning Time
Local Disk iSCSI
0
200
400
600
800
1000
1200
1400
ElapsedTime(Secs)
Bigdata Configuration
Bigdata Installation
OS Reboot
Firmware Initialization
Post Setup Software Installation
Package Installation
OS Boot(inc. Kernel+Initrd Download)
DHCP request
Firmware Initialization
Haas Power Cycle
Ceph Cloning
Haas Initilization
14
Firmware initialization
OS Boot (inc. kernel+initrd download)
OS Installation
OS Reboot
BigData App Installation
Post Config
DHCP Request
Local disk
installation
iSCSI boot

15. Provisioning Time
Local Disk iSCSI
0
200
400
600
800
1000
1200
1400
ElapsedTime(Secs)
Bigdata Configuration
Bigdata Installation
OS Reboot
Firmware Initialization
Post Setup Software Installation
Package Installation
OS Boot(inc. Kernel+Initrd Download)
DHCP request
Firmware Initialization
Haas Power Cycle
Ceph Cloning
Haas Initilization
15
Firmware initialization
OS Boot (inc. kernel+initrd download)
OS Installation
OS Reboot
BigData App Installation
Post Config
DHCP Request
Local disk
installation
iSCSI boot

16. Provisioning Time Scaling
2 Node 4 Node 8 Node
0
50
100
150
200
250
300
ElapsedTime(Secs)
Bigdata Post Script
Booting
Ceph Cloning
Haas Initilization
16

17. Read Traffic over Boot Drive
Initial
Provisioning
Data
Generation1
Sort1
Data
Generation2
Sort2
Data
Generation3
Sort3
Data
Generation4
Sort4
Data
Generation5
Sort5
0
100
200
300
CumulativeiSCSIreadspernode(MB)
iSCSI Reads: Runs with 256GB Data
iSCSI Reads: Runs with 128GB Data
17

18. Read Traffic over Boot Drive
Initial
Provisioning
Data
Generation1
Sort1
Data
Generation2
Sort2
Data
Generation3
Sort3
Data
Generation4
Sort4
Data
Generation5
Sort5
0
100
200
300
CumulativeiSCSIreadspernode(MB)
iSCSI Reads: Runs with 256GB Data
iSCSI Reads: Runs with 128GB Data
~170MB / 8GB Boot Image => 2%
18

19. Read Traffic over Boot Drive
Initial
Provisioning
Data
Generation1
Sort1
Data
Generation2
Sort2
Data
Generation3
Sort3
Data
Generation4
Sort4
Data
Generation5
Sort5
0
100
200
300
CumulativeiSCSIreadspernode(MB)
iSCSI Reads: Runs with 256GB Data
iSCSI Reads: Runs with 128GB Data
3KB/s
read
after
initial
boot
19

20. Write Traffic over Boot Drive
Initial
Provisioning
Data
Generation1
Sort1
Data
Generation2
Sort2
Data
Generation3
Sort3
Data
Generation4
Sort4
Data
Generation5
Sort5
0
100
200
300
400
500
600
700
CumulativeiSCSIwritespernode(MB)
iSCSI Writes - Runs with 256GB Data
iSCSI Writes - Runs with 128GB Data
20

21. Write Traffic over Boot Drive
Initial
Provisioning
Data
Generation1
Sort1
Data
Generation2
Sort2
Data
Generation3
Sort3
Data
Generation4
Sort4
Data
Generation5
Sort5
0
100
200
300
400
500
600
700
CumulativeiSCSIwritespernode(MB)
iSCSI Writes - Runs with 256GB Data
iSCSI Writes - Runs with 128GB Data 14KB/s
write
21

22. Runtime Performance of
Network-Mounted Boot Drive171
319
616
1187
2314
171
318
617
1176
2281
64
115
300
542
1073
69
120
238
555
1361
60
75
76
118
199
52
63
86
125
201
Data Size (GB)
0
400
800
1200
1600
2000
2400
2800
ElapsedTime(secs)
WordCount - Local Disk
WordCount - iSCSI Mounted
Sort - Local Disk
Sort - iSCSI Mounted
Grep - Local Disk
Grep - iSCSI Mounted
8GB 16GB 32GB 64GB 128GB
22

23. Runtime Performance of
Network-Mounted Boot Drive171
319
616
1187
2314
171
318
617
1176
2281
64
115
300
542
1073
69
120
238
555
1361
60
75
76
118
199
52
63
86
125
201
Data Size (GB)
0
400
800
1200
1600
2000
2400
2800
ElapsedTime(secs)
WordCount - Local Disk
WordCount - iSCSI Mounted
Sort - Local Disk
Sort - iSCSI Mounted
Grep - Local Disk
Grep - iSCSI Mounted
8GB 16GB 32GB 64GB 128GB
23

24. Runtime Performance of
Network-Mounted Boot Drive171
319
616
1187
2314
171
318
617
1176
2281
64
115
300
542
1073
69
120
238
555
1361
60
75
76
118
199
52
63
86
125
201
Data Size (GB)
0
400
800
1200
1600
2000
2400
2800
ElapsedTime(secs)
WordCount - Local Disk
WordCount - iSCSI Mounted
Sort - Local Disk
Sort - iSCSI Mounted
Grep - Local Disk
Grep - iSCSI Mounted
8GB 16GB 32GB 64GB 128GB
24

25. Take-aways
• Network booting the OS for bare-metal BigData
• uses only a fraction of boot disk during start-up
• improves provisioning time with no runtime degradation
• provisioning time < 5 mins, boot disk reads: ~3KB/s, writes: ~14KB/s
• Enormous effort on bare-metal provisioning on local disks
may be unnecessary, especially for BigData deployments
• We are building a new Bare Metal Imaging Service using
remote network boot mechanisms
• enable capabilities available on virtualized platforms (e.g.
snapshotting, cloning, …) to bare metal cloud solutions
25

26. Questions
26

27. Provisioning Time
Local Disk iSCSI
0
200
400
600
800
1000
1200
1400
ElapsedTime(Secs)
Bigdata Configuration
Bigdata Installation
OS Reboot
Firmware Initialization
Post Setup Software Installation
Package Installation
OS Boot(inc. Kernel+Initrd Download)
DHCP request
Firmware Initialization
Haas Power Cycle
Ceph Cloning
Haas Initilization
27
Local disk
installation
iSCSI boot Emulab* Ironic*
* A. Chandrasekar and G. Gibson, “A comparative study of baremetal provisioning frameworks,” Parallel Data Laboratory,
Carnegie Mellon University, Tech. Rep. CMU-PDL-14-109, 2014.

28. Provisioning Time
Local Disk iSCSI
0
200
400
600
800
1000
1200
1400
ElapsedTime(Secs)
Bigdata Configuration
Bigdata Installation
OS Reboot
Firmware Initialization
Post Setup Software Installation
Package Installation
OS Boot(inc. Kernel+Initrd Download)
DHCP request
Firmware Initialization
Haas Power Cycle
Ceph Cloning
Haas Initilization
28
Local disk
installation
iSCSI boot Emulab* Ironic*
* A. Chandrasekar and G. Gibson, “A comparative study of baremetal provisioning frameworks,” Parallel Data Laboratory,
Carnegie Mellon University, Tech. Rep. CMU-PDL-14-109, 2014.
copy (rather than install)
an image to local disk