On-demand Computing for Scientific Workflows using Amazon Web Services

1. On-demand Computing for Scientific
Workflows using Amazon Web Services
Claudio Pontili Supervisors:
Gabriele Garzoglio
Steven Timm
Grid and Cloud Services Department, Fermilab

2. Presentation Outline
• Introduction: why do we want to use Commercial Cloud?
• GlideinWMS: a simple way to access resources
• Running up to 1000 simultaneously jobs on Amazon Web
Services and FermiCloud
• Task A: Moving software and data to Commercial Cloud
• Task B: Adding and removing HTCondor to GlideinWMS
using AWS
• Task C: Hybrid Cloud solution AWS+Fermicloud
• Conclusions
11/12/2014Claudio Pontili | AWS On-Demand2

3. Introduction: Fermilab Experiment Schedule
• Measurements at all
frontiers – Electroweak
physics, neutrino
oscillations, muon g-2,
dark energy, dark matter
• 8 major experiments in 3
frontiers running
simultaneously in 2016
• Sharing both beam and
computing resources
• Impressive breadth of
experiments at FNAL
3 11/12/2014Claudio Pontili | AWS On-Demand
FY16

4. Introduction: Computing requirements for
experiments
• higher intensity and higher precision
measurements are driving request
for more computing resources than
previous “small” experiments
• beam simulations to optimize
experiments - make every particle
count
• detector design studies - cost
effectiveness and sensitivity
projections
• higher bandwidth DAQ and greater
detector granularity
• event generation and detector
response simulation
• reconstruction and analysis
algorithms
4 11/12/2014Claudio Pontili | AWS On-Demand

5. Introduction: Slots Fermilab, OSG, & Clouds
• Current full capacity of FermiGrid  ~30k slots
• Full capacity of OSG (Open Science Grid)  ~85k slots
• Additional OSG opportunistic slots  15k – 30k
• Additional per-pay slots at commercial Clouds
11/12/2014Claudio Pontili | AWS On-Demand
5

6. Federation via GlideinWMS – Grid and Cloud Bursting
11/12/2014Claudio Pontili | AWS On-Demand6
Fermi
Cloud
Job Queue
FrontEnd
GlideinWMS
Pilot Factory
Gcloud
KISTI
Amazon
AWS
Jobsub_submit
FermiGrid
OSG
Unified submit tool for grid and cloud
using HTCondor
Users
Pilots
Jobs

7. Running AWS NovA Jobs as function of time, Oct 23. 2014
11/12/2014Claudio Pontili | AWS On-Demand7
0
200
400
600
800
1000
1200 2.21
2.28
2.35
2.42
2.49
2.56
3.03
3.10
3.17
3.24
3.31
3.38
3.45
3.52
3.59
4.06
4.13
4.20
4.27
4.34
4.41
4.48
4.55
5.02
5.09
5.16
5.23
5.30
5.37
5.44
5.51
5.58
6.05
6.12
6.19
6.26
6.33
6.40
6.47
6.54
7.01
7.08
7.15
7.22
7.29
7.36
7.43
7.50
Jobs
Time
• 3300 jobs
• 525 m3.large
• Total cost
$449
• Prices are
getting down
1 hour

8. Task A: Moving software and data to commercial cloud
using Scalable Squid Servers
• Need to transport software and data to the Cloud
• Auto-scalable squid servers, deploying and destroying in 30
seconds using CloudFormation script
11/12/2014Claudio Pontili | AWS On-Demand8

9. Task B: Auto-Scaling GlideinWMS adding/removing
HTCondor using Amazon Web Services
• New resources are made available through the WMS
(HTCondor)
• The system is designed to scale by adding servers
11/12/2014Claudio Pontili | AWS On-Demand9
• Problem: the
submission system is
a stateful service
– Easy to scale up
– Hard to scale down
• Solution? Manage
lifecycle of each
server using AWS
Hooks and Standby
(released July 30th
2014)
Pending
Pending:Wait
(Lifecycle hook)
InService
Standby
Terminating:Wait
(Lifecycle Hook)
Terminated
State diagram

10. • At least 7 different Amazon services
• 2 different programming languages (Java and bash scripting)
• Role based authentication: auto-generating and auto-rotating
logins and passwords
Task B: Auto-Scaling HTCondor using AWS - 2
11/12/2014Claudio Pontili | AWS On-Demand10
Custom
Metrics
(Idle and
running
jobs)
AWS
CLI
S3 Logs and Init
Scripts
Auto Scaling Group and
ELB controlled by Custom
Metrics
Scale Up
Event
Instance Launched
Lifecycle Hook
Hook
queueSNS
Custom Action:
looking for standby
instance instead of
creating a new one
Java
Instance attached to Auto
Scaling Group and ELB
Scale
Down
Event
Instance Removed from
the Auto Scaling Group
and ELB
Standby Instance (it ll be
terminated after 5 days)
Role Based
Authentication
Permissions
Queue
SNS
Custom Action:
finding the right VM
to terminate and
changing ASG state to
standby
Java

11. • No change for the final user!!! He just wants to deploy his job
in the same way and to computed it as soon as possible
• Now we can handle spikes of traffic using commercial cloud
• We pay AWS only during spikes
Task C: Hybrid cloud – Fermicloud and Amazon Web
Service
11/12/2014Claudio Pontili | AWS On-Demand11

12. Conclusions
• Experiments have an increased need for computing
resources with an increased diversity of requirements
• Managing these needs (especially peak demand) is a major
focus
• Experiments are being enabled to use a diverse set of
resources: Local, Grid, and Cloud
• Need to demonstrate sustainability and cost effectiveness of
the commercial Cloud solution for physics use cases (e.g.
NOvA MonteCarlo)
• This work demonstrated the scaling of on-demand services in
support of scientific workflows using native Amazon Services
11/12/2014Claudio Pontili | AWS On-Demand12

13. Thank you
• Questions?
11/12/2014Claudio Pontili | AWS On-Demand13