This document discusses using NOVASTO's batch processing solution on AWS ECS. It describes building a Docker image containing the batch processing application and dependencies, pushing it to Docker Hub, and then using ecs-cli to deploy the batch processing task on ECS configured with the necessary IAM roles, security groups, and other parameters defined in an ecs-params file. The document also mentions monitoring batch jobs and results with CloudWatch.

1. AWS ECS
batch
NOVASTO

2. •
• NOVASTO join
•

3. API
aws
aws

4. •
•
•

5. 1.
2. batch

6. 1.
2. batch

7. 1. Batch
2. Docker
3. Ecs

8. Batch

9. Batch
batch - 1 1 1

10. -Docker-

11. -Docker-
-
Docker p
OS

12. -Docker-
-
image
image DockerHub

13. -Docker-
-
(http://www.itmedia.co.jp/enterprise/articles/1701/23/news036.html)

14. Ecs(Elastic Container Service)
Docker AWS

15. Ecs(Elastic Container Service)

16. Ecs(Elastic Container Service)
(https://thinkit.co.jp/article/9701)

17. Ecs(Elastic Container Service)
——
1.
2.
3.
4. …

18. aws aws

19. 1.
2. batch

20. Docker Engine Docker Engine Docker Engine Docker Engine
batch
Batch

21. (https://stackoverflow.com/questions/23735149/what-is-the-difference-between-a-docker-image-and-a-container)

22. Docker-Compose.yml
Batchfile(~~.rb)
DockerFile
image
Task
image

23. Ecs

24. Docker
Docker-file
FROM ruby:2.5.1
RUN mkdir -p $APP_DIR
WORKDIR $APP_DIR
COPY src/Gemfile $APP_DIR
RUN gem install bundler & bundle install
RUN git clone git@github.com:novasto/~~~
WORKDIR /usr/share/batch/
RUN git checkout $branch
Dockerfile

25. Batchfile
Docker-file
FROM ruby:2.5.1
RUN mkdir -p $APP_DIR
WORKDIR $APP_DIR
COPY src/Gemfile $APP_DIR
RUN gem install bundler & bundle install
RUN git clone git@github.com:novasto/~~~
WORKDIR /usr/share/batch/
RUN git checkout $branch Git clone
batch

26. DockerHub
VERSION=1.0
MINOR_VERSION=1
#
docker build -t novasto/leaves.batch:$VERSION.$MINOR_VERSION
# VERSION latest
docker tag novasto/leaves.batch:$VERSION.$MINOR_VERSION novasto/
leaves.batch:latest
# DockerHub push VERSION
docker push novasto/leaves.batch:$VERSION.$MINOR_VERSION

27. Docker-compose.yml
Docker-compose.yml
version: '3'
services:
ruby:
image: novasto/batch
container_name: scraping_batch
tty: true
command: ruby /usr/share/batch/app/batch/run.rb
env_file:
- ./ruby/app.env
DockerHub
ECS
batch

28. # ecs-cli compose --file docker-compose.yml --ecs-params ecs-params.yml
--project-name ruby create --launch-type EC2
# ecs-cli compose --file docker-compose.yml --ecs-params ecs-params.yml --project-name
ruby create --launch-type version: 1
task_definition:
ecs_network_mode: awsvpc
task_execution_role: secretsTaskExecutionRole3
task_size:
cpu_limit: 256
mem_limit: 512
services:
fluentd:
essential: true
- sg-0d7a8b34dd3cfee84
ecs-params

29. ecs-cli compose --file ./$VERSION/docker-compose.yml --task-role-arn
secretsTaskExecutionRole3 --ecs-params ./$VERSION/ecs-params.yml --project-name ruby
start --launch-type EC2 -c leaves-batch
ecs-params

30. OK!!!
ecs-params

31. Cloud Watch
ecs-params

32. Cloud Watch
ecs-params
1 11 16 24
2 12 1 24

33. ecs-params
ECS
•
•
•

34. ecs-params
ECS AWS
• Subnet
• VPC
• SecurityGroup
• I AM Role

35. ecs-params
•
•
• Web
•