How Facebook runs ad-hoc commands on and controls access to the entire fleet

2. Hypershell
Sameen Jalal
Production Engineer, Cache -> Security

3. Hypershell is a push-based massively parallel
command execution tool
What is Hypershell?

4. Hypershell is a push-based massively parallel
command execution tool
$ hsh exec <targets> <command>
What is Hypershell?

5. Hypershell is a push-based massively parallel
command execution tool
$ hsh exec <targets> <command>
$ hsh deploy <targets> <file> [location]
What is Hypershell?

6. • Ad hoc queries
• Service deployment
• System and service configuration
• Monitoring and Reporting
• Emergency situations
• … many other things
What could it be used for?

7. • Emergency situations
• Nothing else!
What SHOULD it be used for?

8. • Massively scalable
• Flexible
• Easy to specify targets
• Persistent results
• Easy to use
• Command execution tool works
• There is a maintainer
Why was it built?

9. agent target
ssh
agent vs target

10. agent target
ssh
agent vs target vs database
database

11. clusters, racks and arbitrary groups of entities vs database
agents targets
database

12. server vs agents vs database
agents targets
cluster / rack /
host group
server
database
thrift

13. server vs clusters of agents vs database = job
server
databases
c1
c2
c3

14. server
CLI / script / origin
job invocation
thrift

15. jobs
c1
c2
c3
c4
c5
c6
c7
c8
c9
job 1
job 2
server
origin

16. • Staged jobs with confirmation between each step
• Explicit read-only vs write invocations
• Make usage more visible with invocation
monitoring
• Security logging
• Job invocation approval from service owners
• Tiered authentication
Features?

17. “hypershell...”
— Uncle Ben
“With great power comes great responsibility.”
— Engineers

18. Thrift: http://thrift.apache.org/static/files/thrift-20070401.pdf
Services using thrift:
https://code.facebook.com/posts/1468950976659943/
Resources