Who doesn't love building high-available, scalable systems holding multiple Terabytes of data? Recently we had the pleasure to crack some tough nuts to solve the problems and we'd love to share our findings designing, building up and operating a 120 Node, 6TB Elasticsearch (and Hadoop) cluster with the community.

1. You know, for search
querying 24 000 000 000 Records in 900ms
@jodok

2. @jodok

8. source: www.searchmetrics.com

12. First Iteration

13. The anatomy of a tweet
http://www.readwriteweb.com/archives/what_a_tweet_can_tell_you.php

14. c1.xlarge 5 x m2.2xlarge
EBS
ES as document store
- bash - 5 instances
- find - weekly indexes
- zcat - 2 replicas
- curl - EBS volume

15. • Map/Reduce to push
to Elasticsearch
• via NFS to HDFS
storage HDFS
ES
• no dedicated nodes
MAPRED

16. - Disk IO
- concated gzip files
- compression

19. Hadoop Storage - Index “Driver”
Namenode Jobtracker Hive
Datanode Secondary NN Datanode
2 Tasktracker Datanode 2 Tasktracker
6x 500 TB HDFS 2 Tasktracker 6x 500 TB HDFS
6x 500 TB HDFS
Datanode Datanode Datanode
4 Tasktracker 4 Tasktracker 4 Tasktracker
6x 500 TB HDFS 6x 500 TB HDFS 6x 500 TB HDFS
Tasktracker
Spot Instances

20. Adding S3 / External Tables to Hive
create external table $tmp_table_name
(size bigint, path string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY 't'
stored as
INPUTFORMAT "org.apache.hadoop.mapred.lib.NLineInputFormat"
OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"
location s3n://...;
SET ...
from (
select transform (size, path)
using './current.tar.gz/bin/importer transform${max_lines}' as
(crawl_ts int, screen_name string, ... num_tweets int)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '001'
COLLECTION ITEMS TERMINATED BY '002'
MAP KEYS TERMINATED BY '003'
LINES TERMINATED BY 'n'
from $tmp_table_name
) f
INSERT overwrite TABLE crawls PARTITION (crawl_day='${day}')
select
crawl_ts, ... user_json, tweets,

21. https://launchpad.net/ubuntu/+source/cloud-init
http://www.netfort.gr.jp/~dancer/software/dsh.html.en

22. packages:
- puppet
# Send pre-generated ssh private keys to the server
ssh_keys:
rsa_private: | ${SSH_RSA_PRIVATE_KEY}
rsa_public: ${SSH_RSA_PUBLIC_KEY}
dsa_private: | ${SSH_DSA_PRIVATE_KEY}
dsa_public: ${SSH_DSA_PUBLIC_KEY}
# set up mount points
# remove default mount points
mounts:
- [ swap, null ]
- [ ephemeral0, null ]
# Additional YUM Repositories
repo_additions:
- source: "lovely-public"
name: "Lovely Systems, Public Repository for RHEL 6 compatible Distributions"
filename: lovely-public.repo
enabled: 1
gpgcheck: 1
key: "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-lovely"
baseurl: "https://yum.lovelysystems.com/public/release"
runcmd:
- [ hostname, "${HOST}" ]
- [ sed, -i, -e, "s/^HOSTNAME=.*/HOSTNAME=${HOST}/", /etc/sysconfig/network ]
- [ wget, "http://169.254.169.254/latest/meta-data/local-ipv4", -O, /tmp/local-ipv4 ]
- [ sh, -c, echo "$(/bin/cat /tmp/local-ipv4) ${HOST} ${HOST_NAME}" >> /etc/hosts ]
- [ rpm, --import, "https://yum.lovelysystems.com/public/RPM-GPG-KEY-lovely"]
- [ mkdir, -p, /var/lib/puppet/ssl/private_keys ]
- [ mkdir, -p, /var/lib/puppet/ssl/public_keys ]
- [ mkdir, -p, /var/lib/puppet/ssl/certs ]
${PUPPET_PRIVATE_KEY}
- [ mv, /tmp/puppet_private_key.pem, /var/lib/puppet/ssl/private_keys/${HOST}.pem ]
${PUPPET_PUBLIC_KEY}
- [ mv, /tmp/puppet_public_key.pem, /var/lib/puppet/ssl/public_keys/${HOST}.pem ]
${PUPPET_CERT}
- [ mv, /tmp/puppet_cert.pem, /var/lib/puppet/ssl/certs/${HOST}.pem ]
- [ sh, -c, echo " server = ${PUPPET_MASTER}" >> /etc/puppet/puppet.conf ]
- [ sh, -c, echo " certname = ${HOST}" >> /etc/puppet/puppet.conf ]
- [ /etc/init.d/puppet, start ]

23. - IO
- ES Memory
- ES Backup
- ES Replicas
- Load while indexing
- AWS Limits

24. EBS performance
http://blog.dt.org

25. • Shard allocation
• Avoid rebalancing (Discovery Timeout)
• Uncached Facets
https://github.com/lovelysystems/elasticsearch-ls-plugins
• LUCENE-2205
Rework of the TermInfosReader class to remove the
Terms[], TermInfos[], and the index pointer long[] and create
a more memory efficient data structure.

26. 3 AP server / MC
c1.xlarge
6 ES Master Nodes 6 Node Hadoop Cluster
c1.xlarge + Spot Instances
40 ES nodes per zone
m1.large
8 EBS Volumes

31. Everything fine?

33. Cutting the cost
• Reduce the amount of Data
use Hadoop/MapRed transform to
eliminate SPAM, irrelevant Languages,...
• no more time-based indizes
• Dedicated Hardware
• SSD Disks
• Share Hardware for ES and Hadoop

34. Jenkins for Workflows

35. distcp
S3 HDFS
transform
https://github.com/lovelysystems/ls-hive
https://github.com/lovelysystems/ls-thrift-py-hadoop

38. That's thirty
minutes away.
I'll be there in ten.
@jodok