Astrophysics has just entered a new era where computational power (parallel and distributed) and state-of-the-art technology are mandatory as off the-shelf tools for daily activity. In this talk we focus on two scenarios in which AWS services (Amazon EC2, Amazon SQS, AWS Lambda, and Amazon S3) have greatly improved project performance. In the first, we explore the use of EC2 coupled with AWS Lambda in heavily parallelized CUDA based simulations to design innovative instrumentation in search for life on extrasolar planets. Next, we dive into serverless HTC architecture used to create simulations of astrophysical sources such as astronomical high energy gamma rays. You'll leave understanding how these architectures propelled innovation and decreased execution times all while optimizing overall cost.

1. P U B L I C S E C T O R
S U M M I T
WASHINGTON DC 2019

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Speeding Up Scientific
Computation in Astrophysics with
Amazon Web Services
Marco Landoni
Ph.D.
INAF – Italian National Institute of Astrophysics
Brera Astronomical Observatory
3 2 4 2 8 7
Antonio Corina
AWS Sales specialist
Amazon Inc.

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Agenda
Setting the stage – Introduction to INAF, Brera Observatory
I’ll tell you two stories today …
• The first one is related to the hunt of extrasolar planets with AWS
Accelerated Instances (EC2 + NVIDIA CUDA)
• In the second, we’ll dive into the world of gamma-ray astrophysics
with Amazon EC2, SQS and Lambda

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Who are we?

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
INAF – The National Institute of Astrophysics

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
INAF – The National Institute of Astrophysics

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Our roots

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
The INAF Brera Observatory

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Our Infrastructures … today

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
…. and tomorrow

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
The first story. The hunt for habitable
exoplanets with the Extremely Large
European Telescope with HIRES.
PI: Prof. Alessandro Marconi, University of Florence

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
De l’infinito universo et mondi Giordano Bruno (α1548-
ω1600)
<<There are countless Suns and countless lands all
revolving around their suns, We see only mine
because they are the largest and most luminous
bodies and their planets are invisible because they
are smaller and not luminous. The innumerable
worlds in the universe are neither worse nor less
inhabited than our earth>>

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
From what we thought is was a rule…… to the great diversity of exoplanetary systems

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
And this is routine…

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
How do we hunt them ?

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
A rule of thumb…
Earth induced
RV on Sun = 9 cm/s
Jupiter induced
RV on Sun = 12.5 m/s

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Our I/O ‘data stream’ is a telescope with a spectrograph

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
How does it work ?

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
And at the end, you will get this!

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
A rule of thumb…
Earth induced
RV on Sun = 9 cm/s

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
This is already a day-by-day reality with an ESPRESSO

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
What are we really looking for ?

23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
… and our final goal is to study their atmosphere

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Toward the Extremely Large Telescope

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
We will use a very complex spectrograph

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
HIRES simulations

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Our journey to the Cloud with AWS for HIRES simulations

28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
To simulate a full HIRES detector frame we need…

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
10 pixels box for each image
10pixelsboxforeachimage
10 sub pixels
10subpixels

30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
The problem is computationally heavy…

31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Using NVIDIA CUDA (8.0) on Amazon EC2
Block Block
Thread Thread Thread Thread Thread ThreadIn parallel
Block Block Block Block Block Block
…
… Block
10 pixels box for each image
10pixelsboxforeachimage
10 sub pixels
10subpixels
Amazon EC2G3 instance

32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Scaling up to full architecture with EC2 cluster
AWS Lambda
AWS Cloud
Spot Fleet
1) Upload
2)Triggers
3)
Instantiate
G3 instanceG3 instance G3 instance G3 instanceG3 instance
G3 instanceG3 instance G3 instance G3 instanceG3 instance
G3 instanceG3 instance G3 instance G3 instanceG3 instance
4) Illumination profiles are sent back
AMI
Amazon S3 Glacier
4) Toward on premises farm for further analyses
Scheduled move of old data to
Glacier

33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Sample result
Calibration light
Object + sky
contribution
G2V star with mv = 10, Exp. Time = 50 sec
Genoni M. , Landoni M et al 2018
SPIE Astronomical Telescopes + Instrumentation, 2018, Austin, Texas, United States

34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
How did the AWS Cloud help the HIRES team ?

35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
The second story: A journey toward
gamma-ray astrophysics in the cloud.
With the Cherenkov Telescope Array

36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Why 𝛾 rays astronomy is important ?
• Studing the violent Universe!

37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
How are very high energy gamma rays detected ?

38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
The Concept of the Cherenkov Telescope Array

39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Scientific simulations for the CTA observatory
• As in the case of HIRES, simulations of astrophysical sources MUST be
done in order to asses the expected performance of the instrument
(with quickly changing setup, efficiency, etc.) and test what kind of
science could be carried out.
• We used the official software to perform our simulation in the context
of extragalactic Astrophysics.
• We did it for many sources
• … with great results and speed thanks to the AWS architecture (I’ll
show you)

40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
What is a simulation in this case ?

41. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
But you need to repeat this process many times…
You end up with thousands and
thousands simulations…

42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
The proposed architecture on AWS
Amazon Simple Storage
Service (S3)
AWS Lambda
Input Queue
Failed Queue
Spot Fleet
AMI
Instances
Amazon S3 Glacier
1) Upload
2)Triggers
3) Pushes commands
3) Instantiates
Pull
Push if marked to terminate6)TowardINAF
4) Compute
5) Output is sent to S2
bucket
AWS Cloud

43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
A reloaded version…
Amazon Elastic
Container Registry
Containers
Amazon DynamoDBSimulationResults
AWS Batch
Auto Scaling group
Spot fleet
Instances
Amazon Simple Storage
Service (S3)
1) Model upload
2)Startjobarray
AWS Glue
ETL
AWS Cloud
Results to
Dynamo

44. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Benefits
• We were able to run thousands and thousands of simulations by deploying the
architectures in minutes.
• We optimized the overall performance by a factor of x100 by consuming about half
million CPU/hrs and simulating hundreds of TBs of data events.
• We were able to smartly use resources without impact on the on premises INAF
clusters and grid mainly dedicated to long, frequent and network bounded
simulations (for example in the context of Cosmology).
• Remarkably: we published 4 papers and various proceedings in 1 year !!!!!

45. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
AWS has been a part of our research with our teams!

46. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Conclusions and take home messages
• It’s just the beginning….
• Each computational problem (especially in Astrophysics) has its own
optimized model to be solved
• AWS platform demonstrated to be powerful solution for a number of
computational astrophysics tasks
• …and the number of situation where the Cloud Computing could be
successfully applied in this sector is constantly increasing!

47. Thank you!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Marco Landoni
INAF – Italian National Institute of Astrophysics
Brera Observatory

48. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T