We all agree that recurring operational tasks are time-consuming nuisances, which should be eradicated using automation. However, sometimes they require careful coordination, hardware manipulation and worst of all - human interaction. Recently, we found that our code doesn’t really need to pass a Turing test in order to successfully interact with humans, and convince them to partake in an automated process. In this talk I’ll describe how we automated disk replacement for our HDFS clusters - despite having to communicate with the hosting provider by emails, while preventing the process from failing at scale.

1. ReverseEngineering
the"HumanAPI"
forAutomationandProfit
Nati Cohen / SimilarWeb
@nocoot

2. Hello,mynameisNati
and I hate hard disks
ZERO Hard Disks
(and look how
happy I was…)

3. Asingleharddiskisgreat
● Will store ALL your porn data
○ 2015: 10TB helium-filled HDD
● Cheap
○ 2015: 0.032$ per-gigabyte
● Will rarely fail
○ Unless it’s really inconvenient

4. harddiskfailureinnumbers
Pinheiro, Eduardo, Wolf-Dietrich Weber, and Luiz André Barroso.
"Failure Trends in a Large Disk Drive Population." FAST. Vol. 7.
2007.

5. Thedarksecretofself-hosted“Big-Data”
● 150 nodes * 8 disks
● First year: ~24 failures
● Second year: ~96 failures
● Bad disks can cause
○ Slow tasks
○ Under replicated blocks
○ Missing blocks

6. Diskreplacementprocess
1. Detect
2. Stop usage
3. Request replacement
4. <Physical replacement>
5. Initialize disk
6. Resume usage

7. YUNOAUTOMATE?!
● Recurring task? CHECK
● Time consuming? CHECK
● Error prone? CHECK
Let’s do it!

8. Problem
1. Detect
2. Stop usage
3. Request replacement
4. <Physical replacement>
5. Initialize disk
6. Resume usage
No API for actual replacement
:,(
We want:
replace(server_id, disk_id)
is_replaced(server_id, disk_id)

9. is_replaced(server_id,disk_id)
● We have RAID adapters on all the servers
● They can emit the Disk Serial Number
is_replaced(server_id, disk_id):
return STORED_SERIAL_NUMBERS(server_id, disk_id) !=
get_serial_number(server_id, disk_id)

10. replace(server_id,disk_id)
Luckily our hosting provider accepts emails!
Goal: Craft the Perfect*
Email
*
The correct disk will be replaced,
without any extra correspondence

11. replace(server_id,disk_id):#1Attempt
Hi,
Please replace the following Disk on the Server:
Server Ip : <SERVER_ID>
Disk : <DISK_ID>
Thanks
Which one is it?

12. replace(server_id,disk_id):#2Attempt
Hi,
Please replace the following Disk (it is blinking)
on the Server:
Server Ip : <SERVER_ID>
Disk : <DISK_ID>
Thanks
Why?

13. replace(server_id,disk_id):#3Attempt
Hi,
Please replace the following Disk (it is blinking)
on the Server:
Server Ip : <SERVER_ID>
Disk : <DISK_ID>
Description : <ERROR_MSG>
Thanks
Now?
Are you sure?
Do you have
backups?

14. replace(server_id,disk_id):#4Attempt
Hi,
Please replace the following Disk (it is blinking)
on the Server:
Server Ip : <SERVER_ID>
Disk : <DISK_ID>
Description : <ERROR_MSG>
The server is backed up and ready for maintenance.
Thanks

15. replace(server_id,disk_id)

16. Diskreplacementprocess
1. Detect
a. SMART / Vendor utility
b. Monitoring service API
2. Stop usage
a. Application API
b. Operating system (kill processes, umount, delete SCSI device)
3. Request replacement
a. replace(server_id, disk_id)
4. <Physical replacement>
a. while not is_replace(server_id, disk_id): sleep(t)
5. Initialize disk
a. [RAID], Partition Table, Filesystem, Directories, Permissions
6. Resume usage
a. Application API

17. Safetyis#1priority!
● Check Pre-conditions
○ Under-replicated blocks
○ Missing blocks
● Single replacement per-cluster
● Handle special cases
○ Root device
○ User facing services

18. Notifications&Audit
● All emails originate from PE group address
● CC to PE group and Slack
● Audit log sent to ElasticSearch
○ SOON: Kibana dashboard

19. Summary
● One disk GOOD
● Many disks BAD
● Avoid Fail at Scale
● No need to pass a Turing Test to interact with humans
replace(1.2.3.4, 0:1:2)

20. Questions?
Nati Cohen / SimilarWeb
@nocoot

21. Architecture
See also Oz Katz’s talk about Faberge