Jupiter is a cloud-native company that delivers hyperlocal environmental information in a changing climate, primarily using AWS Batch. Through AWS Batch’s capability to execute thousands of scientific modeling jobs while managing scale and cost, Jupiter scientists can focus on data analysis and developing sophisticated machine learning (ML)-based applications to support private sector and local municipality customers; AWS Batch takes care of the rest. In this chalk talk, we demonstrate how AWS Batch, through managing resource provisioning and scheduling, enables flexibility across changing requirements to allow various modeling applications to run quickly and at scale.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How Jupiter Intel Uses AWS to
Forecast Weather Impact &
Climate Risk
John Exby
Systems Architect
Jupiter / Research and Science
C M P 3 4 0

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“We live in a world
designed for an
environment that
no longer exists.”

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
$330B
lost from natural catastrophes in 2017

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Path Towards Resiliency
Understand
physical
vulnerabilities
1
Quantify
costs and
damageability
2
Identify
adaptation options,
risk transfer,
mitigation
3

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
There Are Five Key Characteristics
Scientifically Rigorous
Peer-reviewed models
incorporating shifting climate
baseline
Verification & Validation
Model outputs verified
against observed historical
data
Periodically Updated
Continuously updated with
latest climate and built-
environment data as well as
modeling methodologies
High-Resolution
Property-level resolution to
support decision making
Scenario-Based
Flexibility to accommodate
different assumptions and
provide transparency for
comparability

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Science to Action

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
工欲善其事, 必先利其器。
- 孔子
“For a craftsman to perfect his trade,
he must first fine-tune his tools.”
- Confucius

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Tools and Services for Dynamic Science
Amazon S3
Amazon EFS Amazon ECR
Amazon EC2
AWS Batch
Lambda
function
AWS Step Functions
Images

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Batch
Fully Managed
No software to install or
servers to manage. AWS
Batch provisions, manages,
and scales your infrastructure
Integrated with AWS
Natively integrated with AWS
products and services
Cost-Optimized
Resource Provisioning
AWS Batch automatically
provisions compute resources
tailored to the needs of your jobs
using Amazon Elastic Compute
Cloud (Amazon EC2) and EC2
Spot Instances

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Improved Managed Compute Environments
• Use EC2 Launch Templates in your compute environment
• Example use cases include:
• Increase Size/Encrypt Container Volume
• Support custom User-Data: Mount Amazon Elastic File System (Amazon EFS) on instance
launch, without needing to create a custom AMI
• Override ECS Image Cleanup
• New instance types
• Z1d
• R5
• R5d
• M5d
• C5d
• X1e

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Batch Regional Expansion
AWS Batch is available in 15 regions:
us-east-1 (N. Virginia) ap-south-1 (Mumbai)
us-east-2 (Ohio) ap-northeast-1 (Tokyo)
us-west-1 (N. California) ap-northeast-2 (Seoul)
us-west-2 (Oregon) ap-southeast-1 (Singapore)
eu-west-1 (Ireland) ap-southeast-2 (Sydney)
eu-west-2 (London) ca-central-1 (Canada Central)
eu-west-3 (Paris) sa-east-1 (São Paulo)
eu-central-1 (Frankfurt)

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Elements of a Scientific Pipeline
Optimized
AMI
Spot Fleet
ECS
container
Amazon
ECR
NetCDF / HDF5
To
Zarr
transformations
S3
Objects
Dask
Analytics
Cluster
S3 with
Geographic Data
Generated
Storm Scenarios
Scientists
AWS
Batch
Job Definition
Docker Image
Model XYZ

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scaling the Science Pipeline
Ibtracs (.csv)
Desired
Perturbations
JupiterTrack generator
Processing (model-dep)
Determine Landfall
Calculate Tides
Infill missing data
Unprepped JupiterTrack object
Perturb
1 in, X out
Tide info (.csv)
Specify
downstream
model
Track Generator
Parallel Model container
Process (usually .py)
Containers / BatchHuman input
Data storage (S3 or local)
Legend
Prepped baseline JupiterTrack
object
Prepped perturbed JupiterTrack
objects
(optional)
Post-processing
NetCDF, stm, trk files
Plots, csv of data
Validation against
observation
Tweak params, run again
Amazon
EFS
Amazon
S3
Amazon
S3
Amazon ECR
Amazon ECR
AWS Batch
AWS
Lambda
AWS Batch
Amazon
EFS
List of
Storms
AWS Batch
Amazon
S3
Scientist: “Can you
build this?”
Statistician:
“Can you
execute this
5,000 times?”

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Batch Optimization Considerations
• Using Spot for $ savings, with a sane retry strategy
• Unstable multi-hour modeling runs may repeatedly fail
• AWS Batch can be used for MPI containers
• Don’t try to over specify compute environments
• Use Job Definitions :ver accordingly for resource allocation
• AWS CloudFormation, JSONs, Boto 3 lead to maximum flexibility
• Tag and re-tag for billing filtering
• AMIs and Templates can implement specialized use cases

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Support for Multi-Node Parallel Jobs
Container 1 Container 2
Container 3 Container 4

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Batch JSON CloudFormation
{"computeEnvironmentName": ”wrf-precipcity-
spot",
"type": "MANAGED",
"computeResources": {
"type": "SPOT",
"minvCpus": 0, "maxvCpus": 2000,
"desiredvCpus": 0,
"instanceTypes": ["c5"]
"bidPercentage": 80,
"imageId": "ami-a83327c1",
"instanceRole":
"arn:aws:iam::555017055555:instance-
profile/ecsInstanceRole",
"tags": {
"Product": "floodscore",
"Env": ”dev",
"Tier": "model",
"Domain": ”city001",
"Application": ”precip",
"Project": ”proto-city",
"Name": ”wrf-precip-spotnode"}

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Code for Disposable, Automated Environments
batch_client = boto3.client('batch')
compute_environment = 'stormsurge-dev’
new_ce =
batch_client.create_compute_environment(
computeEnvironmentName= compute_environment,
type='MANAGED' state='ENABLED',
computeResources={
'type': 'SPOT','minvCpus': 0,
'maxvCpus': 2000, 'desiredvCpus': 0,
'instanceTypes’: [ 'c5’ ],
‘ imageId': 'ami-a83327c1',
newqueue = batch_client.create_job_queue(
computeEnvironmentOrder=[
{ 'computeEnvironment':
compute_environment,
'order’: 1, },
],
jobQueueName='stormsurgeruns',
priority=10,
state='ENABLED',
)

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Additional Resources for Research on AWS
https://aws.amazon.com/rtc/
https://aws.amazon.com/rcp/
https://aws.amazon.com/research-credits/
https://registry.opendata.aws/

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!
john.exby@jupiterintel.com

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.