The document provides an overview of AWS Batch, a service designed to simplify batch job processing on AWS. It includes details on how to create job queues, compute environments, and submit jobs using JSON configuration files, while highlighting the benefits of resource optimization and cost management. Additionally, it contains links to case studies and resources for deploying batch workflows, particularly in genomics and other high-throughput environments.

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2018

2. n and ff Predictable peaksWASTE
O L S E L
( , ) , , T G

3. 1. https://aws.amazon.com/ko/solutions/case-studies/novartis/
2. https://aws.amazon.com/blogs/aws/experiment-that-discovered-the-higgs-boson-uses-aws-to-probe-nature/
3. https://www.slideshare.net/AmazonWebServices/bdt311-megarun-behind-the-156000-core-hpc-run-on-aws-and-experience-of-
ondemand-clusters-for-manufacturing-production-workloads-aws-reinvent-2014
1 ,, N8D
6 7E ED R C2
1 6 H E
D R 7E R C2
1 0 5 3 9 D D
9 3 7 R C2

4. • S3/EC2/Spot Fleet
• Auto-Scaling
• SNS/SQS/CloudWatch
• S3/ECS
• SQS/CloudWatch

6. t 12
a eW
v Dm A
So E m
1 6357 ,
r C
B
A -
53 76 SnS
187: SnSI p
eW C
B 21 06 6:
B
B

7. g P m
,
(
o h
) u
d MW C
A m
s
, /
c il
s
il
( C
l l a
m S l
t E
e
e
C U v
n
L I E
2/ z )
L g P
E

8. 21
) )Batch Queue (2)
Batch Queue (1)
Batch Queue (0)
priority
(
Container Property
Compute
Resources
DependsOn
Container Property
Container Property
( ) )
( )
( )
( )
…(
Define the
Batch Job
Create
Compute
Environment(s)
Create Job
Queue Submit Job Job Scheduling
Monitor Job and
Outputs

9. IAM Role for
Batch Job
Input Files
Queue of
Runnable Jobs
S3 Events Trigger
Lambda Function
Submits Batch Job
Compute
Environments (ECS)
Job Definition
Batch Execution
Application
Image (ECR)
Batch Scheduler
Batch Job
Input
Batch Job
Output
Output Files

11. (
• B 2 CEA
A 2 B
aws batch submit-job --cli-input-json file://submit_job.json
( )
SUBMITTED
PENDING
RUNNABLE
STARTING
RUNNING
SUCCEEDED
FAILED

12. ()( ( )
• - C A ,
A
aws batch register-job-definition --cli-input-json file://jdef.json

13. ) ( (
• ,
aws batch create-job-queue --cli-input-json file://job_queue.json

14. ) ( )( (
• . "
" , O
aws batch create-compute-environment --cli-input-json file://job_env.json

15. ,
!

17. https://aws.amazon.com/blogs/compute/using-aws-cloudformation-to-create-and-manage-aws-batch-resources/

18. 2

19. 2

21. $ aws batch create-compute-environment --compute-environment-name c4Spot50 --type MANAGED
--state ENABLED --compute-resources
type=SPOT,instanceTypes=c4,minvCpus=0,maxvCpus=10000,desiredvCpus=2500,bidpercentage=50,s
ubnets=subnet-220c0e0a,subnet-1a95556d,subnet-978f6dce,instanceRole=ecsInstanceRole,
securityGroupIds=secGrp01,secGrp02,secGrp03,spotIamFleetRole=arn:aws:iam::012345678910:ro
le/aws-ec2-spot-fleet-role,serviceRole=arn:aws:iam::012345678910:role/AWSBatchServiceRole
$ aws batch create-compute-environment --compute-environment-name C4-RI --type MANAGED --
state ENABLED --compute-resources
type=EC2,instanceTypes=c4.2xlarge,minvCpus=0,maxvCpus=6000,desiredvCpus=1000,subnets=subn
et-220c0e0a,instanceRole=ecsInstanceRole,
securityGroupIds=secGrp01,spotIamFleetRole=arn:aws:iam::012345678910:role/aws-ec2-spot-
fleet-role, serviceRole=arn:aws:iam::012345678910:role/AWSBatchServiceRole
.2 12
1 12

22. 2 .

23. F A
{
...
"SubmitJob": {
"Type": "Task",
"Resource": ”arn…",
"Next": "GetJobStatus"
},
...
}

25. Annotation
Variant
Calling
QC
Alignment
• https://aws.amazon.com/ko/blogs/compute/building-high-throughput-
genomics-batch-workflows-on-aws-introduction-part-1-of-4/
• https://github.com/aws-samples/aws-batch-genomics

26. 3 .
• M
•
• 1
• M
• C
•
Job A Job C
Job B:0
Job B:1
Job B:n
…
$ aws batch submit-job --job-name BigBatch --job-
queue ProdQueue --job-definition monte-carlo:8 --
array-properties “size=10000” ...
{
"jobName": ”BigBatch", "jobId": "350f4655-
5d61-40f0-aa0b-03ad787db329”
}
Job Name: BigBatch
Job ID: 350f4655-5d61-40f0-aa0b-03ad787db329
Job Name: BigBatch
Job ID: 350f4655-5d61-40f0-aa0b-03ad787db329:0
Job Name: BigBatch
Job ID: 350f4655-5d61-40f0-aa0b-03ad787db329:1
Job Name: BigBatch
Job ID: 350f4655-5d61-40f0-aa0b-03ad787db329:9999
…

27. Job Depends on Array Job
“Job-B”
B:0
…
B:1
B:99
“Job-A”
$ aws batch submit-job –cli-input-json file://./Job-A.json
<Job-A.json>
{
"jobName": ”Job-A",
"jobQueue": "ProdQueue",
"jobDefinition": ”job-s3-list-validate:1",
}
$ aws batch submit-job –cli-input-json file://./Job-B.json
<Job-B.json>
{
"jobName": ”Job-B",
"jobQueue": "ProdQueue",
"jobDefinition": ”job-s3-CPU-intensive:1",
”containerOverride": { ”vcpus”: 32, “memory”: 4096 }
”arrayProperties": { “size”: 100 }
"dependsOn": [
{"jobId": "<job ID for Job A>" }
]
}
)3 ( ,
J
A
S B

28. Two Equally-Sized Array Jobs,
a.k.a. “N-to-N”
“Job-A”
A:0
…
A:1
A:2
A:3
A:9999
B:0
B:1
B:2
B:3
B:n
“Job-B”
“Job-A”
“Job-C”
C:0
…
C:1
C:2
C:3
C:9999
D:0
D:1
D:2
D:3
D:9999
“Job-D”
“Job-B”
C is dependent on A and B
D has an N_TO_N dependency on C

29. “Job-A”
“Job-C”
C:0
…
C:1
C:2
C:3
C:9999
D:0
D:1
D:2
D:3
D:9999
“Job-D”“Job-B”
B:0
…
B:1
B:9
B:2
“Job-E”
Heavy
Network I/O
CPU
Intensive
Large
Memory
Setup Cleanup

30. • E n
• 1 - E u n
• / E , 2 ,-2 n 13 -1 St
• Mm a n
• G E
• P , 2 u
• P h l UI W oi W ,
U C
• zu d
• r p 2 : A - 5 n
• c r p 5B : A 5 t F

32. N A D
https://aws.amazon.com/ko/batch/use-cases/

33. 9
2 E9CG9 89
, E B 9E
0- 9 8
4-
Deletion
E 6
96
.15E
.15 G E
99
8 99
68
.15E
.15 G E
99
8 99
9
9
9
8 9
8 9
8 9
9
9
9
.15E
.15 G E
.15E
0- -3
.15 G E
9
AWS re:Invent 2017: AWS Batch: Easy and Efficient Batch Computing on AWS (CMP323) https://www.youtube.com/watch?v=8dApnlJLY54

34. API Server
(ECS)
SWF
Workflow
Job Manager
(ECS)
Generate Variants
Solve Variants
start
Workflow
poll
for
Decision
submit Batch Job
poll for Activity
submit Batch Job
poll for Activity
task Completed
task Completed
§ :
§ 2 ) A: :
§ ( 2 ) A: : B )
AWS re:Invent 2017: AWS Batch: Easy and Efficient Batch Computing on AWS (CMP323) https://www.youtube.com/watch?v=8dApnlJLY54

35. Deploy an 8K HEVC pipeline using Amazon EC2 P3 instances with AWS Batch
https://aws.amazon.com/ko/blogs/compute/deploy-an-8k-hevc-pipeline-using-amazon-ec2-p3-
instances-with-aws-batch/

36. 1.2 MILLION JOBS
SUBMITTED
300K INSTANCES
CHURNED
2 TEAMS USE IN
PRODUCTION
• 5 D
• , 0 2 c
• 6 0 c
600 CPU YEARS
SPENT
) ( 7 2 1
AWS re:Invent 2017: AWS Batch: Easy and Efficient Batch Computing on AWS (CMP323) https://www.youtube.com/watch?v=8dApnlJLY54

37. 비용 최적화 (Cost-optimized)
• 인프라 설정 및 관리 운영 부담 최소화
• 언제나 원하는 시간에 따라 다양한 작업 실행 가능
• 별도 SW 구매 없이도 다양한 배치 작업에 맞는 작업 구성 관리 가능
비지니스 현장에서 발생하는 다양한 요구 사항에 대해 민첩하게 대응 가능한 배치
작업 아키텍처 제공 가능
자원 최적화 (Resource-optimized)
• 예산이 한정되어 있는 경우
• 다중 업무에 따른 사내 작업 순위와 컴퓨팅 환경을 공유해야 하는 경우
• 기존 구매 자원 (Reserved Instance) 등의 활용을 못하고 있는 경우
RI
시간 최적화(Time-optimized)
• 작업에 데드라인이 있는 경우, 작업 전체 시간을 줄이고 싶을 때
• 다양한 컴퓨팅 인스턴스와 지불 방식을 조합하여 빠르게 끝낼 수 있음

38. • / P
( T o S - a no r
• A: . SL T w
/ A dr P s r fc
• A: / W f
He T h o S
• / / : / / fc W
b lt P u Ws g k
• m
.) / P p i
CC B :C D EB B DBC A
F G B R !
/ D
FMS P RC EF

39. • FFB H IA A F F
• FFB H IA A A F C
• : FFB H IA A F : FF : F DF
•
FFB A H IA A A D F F F G D:G H F F F
•
• B A AD F F B G F A - F :D F A
FFB : F G A H H F : A
• A BGF A: /A F D : AH FA F /+
FFB H IA A A: A BGF D F : BD A F
• B . D : A F
FFB H IA A A: A BGF B D : A H F

40. :
:
- / .! .