第10回elasticsearch勉強会で使用した資料です。

1. Amazon Web Searvice
elasticsearch
×

2. • DMP AudienceSearch
•

3. • IM elasticsearch
• elasticsearch
•
• AutoScaling
•

4. elasticsearch
ID
•
•
• User Agent
• IP
ID

5. AudienceSearch

6. • AWS
• Cluster: 1, Type: 1, Node: 86
• master: 3 (master:true, data: false)
• data: 80 (master:false, data:true)
• searcher: 2 (master:false, data: false)
• indexer: 1 (master:false, data: false)
• version: 1.3.8 → 1.5.2 (5/26 )

7. masterdatasearcher
x2
indexer
indexer AutoScaling
x3x80
ELB

9. Cluster Size
• Shard = Node
• 10 200
• 100
• 200 Aggregation
→
• 80

10. Shard
• Node = 80 c3.large
(1) shard: 40, replica: 1 shard/node = 0.5
(2) shard: 80, reprica: 1 shard/node = 1
(3) shard: 160, replica:1 shard/node = 2
• shard/node
※ shard/node shard/cpu

11. Instance Size
• SSD
→ Ephemeral Disk c3
• xlarge CP
→
large→xlarge 30GB→80GB
• JVM doc_value

12. AutoScalling

13. Why AutoScaling ?
AutoScaling
•
• terminate
•
•

14. recovery
• LifeCycle
• recovery
• 0
indices:
recovery:
max_bytes_per_sec: 500mb
concurrent_streams: 4

15. recovery
• LifeCycle
• : 40GB 10 c3.xlarge

16. • elasticsearch-cloud-aws
• discovery master node
• SG
discovery:
type: ec2
ec2:
tag:
es_cluster: production

17. • master node
• aws:autoscaling:groupName

18. ELB
• ELB
master, data
• 9200 9300

20. • Multi-AZ 3
• shard replica AZ
cluster:
routing:
allocation:
awareness:
attributes: aws_availability_zone

21. • ”aws_availability_zone” AZ
elasticsearch-cloud-aws
•
{
"attributes" : {
"aws_availability_zone" : "ap-northeast-1c",
"master" : "false"
}
}

22. hostname
• hostname(ip-10-0-0-1) ES
• AMI
node:
name: ${HOSTNAME}
export HOSTNAME=$(hostname -s)
/etc/sysconfig/elasticsearch
/etc/elasticsearch/elasticsearch.yml

23. • ELB
• Mackerel
• Papertrail
• / Slack webhook

24. Papertrail
master
data
searcher
x2
x3
x80
SSH tail
syslog

25. • AWS ES
• AutoScalling
• Cluster Size
•
• 1 index elasticsearch