This document discusses EyeEm's use of Elasticsearch in production. It provides an overview of EyeEm, the technologies in its API stack including Elasticsearch, its current Elasticsearch cluster configuration, production use cases for Elasticsearch including album, people and location searches, lessons learned, and the company's outlook.

1. EyeEm
Lars Fronius
@LarsFronius
ElasticSearch in production at EyeEm

2. • H A P P Y G R U M P Y C AT O F
E Y E E M
• S TA R T E D A S A N O P S I N A
S C I E N T I F I C D ATA C E N T E R
• N O W D E V
• D E V E L O P E R S H AT E M E
S O M E T I M E S
M E
A B O U T M E

3. EyeEm is the world’s premier community and
marketplace for the photographer inside all of us

6. A P I S TA C K
• PHP
• MySQL (~10k commands per second)
• Memcached (~50k commands per second)
• Redis (~3k commands per second)
• S3 (~1k commands per second, 40m photos stored)
• Elasticsearch (~250 commands per second - elasticsearch-php)
• All writes are async
• Metrics everywhere

7. C U R R E N T C L U S T E R S P E C S
• 3 x m3.xlarge (4 cores, 15GiB Mem, 2 x 40GB SSD)
• cloud-aws plugin to interconnect.
• OpenJDK 1.6
• 60% heap size (9 GiB)
• 4 Indexes, 5 Shards each. From 1GB to 15GB

8. C U R R E N T P R O D U C T I O N U S E - C A S E S

9. C U R R E N T P R O D U C T I O N U S E - C A S E S
A L B U M S E A R C H

10. C U R R E N T P R O D U C T I O N U S E - C A S E S
P E O P L E S E A R C H

11. C U R R E N T P R O D U C T I O N U S E - C A S E S
• C I T Y- S E A R C H
• L I V E N E A R B Y
D I S C O V E R

12. C U R R E N T P R O D U C T I O N U S E - C A S E S
L I V E N E A R B Y

13. C U R R E N T B E TA U S E - C A S E S

14. C U R R E N T B E TA U S E - C A S E S

15. L O N G S T O RY
• MyISAM full-text search
• Album Search on one ElasticSearch node
• People Search added
• Scale-Out to 3 instances for Photo Search (+ Live
Nearby)

16. E L A S T I C S E A R C H - I N T E R N A L S
• Index
• What your application sees.
• View for a logical namespace inside ElasticSearch.
• Consists of a fixed number of shards
• “To Index” means to “put” your data into
ElasticSearch to make it available for search and for
persistence.

17. E L A S T I C S E A R C H - I N T E R N A L S
• Inverted-Index/Mapping
• The Mapping tells Lucene how to create the
inverted-index in order to make data searchable.
• e.g. “EyeEm” as an nGram{2,3} gets “indexed” as
[“Ey”,”ye”,”eE”,”Em”,”Eye”,”yeE”,”eEm”],
“yeah” would be [“ye”,”ah”,”yea”, “eah”]

18. E L A S T I C S E A R C H - I N T E R N A L S
• Inverted Index/Mapping by example
Ey 1
ye 1,2
eE 1
Em 1
Eye 1
yeE 1
eEm 1
ah 2
yea 2
eah 2

19. S C H E M A - L E S S O R W H AT ?
• Yes and No.

20. S C H E M A - L E S S O R W H AT ?
• Yes - You can put anything that can be formatted as a
JSON in your index, and you get a readable
document.

21. S C H E M A - L E S S O R W H AT ?
• No - you have to think first, because changing your
Mapping is expensive, since you have to reindex.

22. E L A S T I C S E A R C H - I N T E R N A L S
• Shard
• Instance of Lucene
• Consists of multiple Lucene segments
• Manages segments (Merging, fsync, deletion etc.)

23. E L A S T I C S E A R C H - I N T E R N A L S
segments API
http://example.es:9200/yourindex/_segments
indices: { eyephoto6: { shards: { 0: [!
{!
routing: {!
state: "STARTED",!
primary: true,!
node: "PiVDZW-VRYmeaVOy7afoWQ"!
},!
num_committed_segments: 2,!
num_search_segments: 3,!
segments: {!
_l: {!
generation: 21,!
num_docs: 13,!
deleted_docs: 0,!
size_in_bytes: 30810,!
memory_in_bytes: 589,!
committed: true,!
search: true,!
version: "4.7",!
compound: true!
},!
!
!
!
!
!
_m: {!
generation: 22,!
num_docs: 371,!
deleted_docs: 16,!
size_in_bytes: 408548,!
memory_in_bytes: 7365,!
committed: false,!
search: true,!
version: "4.7",!
compound: false!
},!
_n: {!
generation: 23,!
num_docs: 16,!
deleted_docs: 0,!
size_in_bytes: 38514,!
memory_in_bytes: 615,!
committed: false,!
search: true,!
version: "4.7",!
compound: true!
}!
}!
}!
],!
1: [!

24. E L A S T I C S E A R C H - I N T E R N A L S
• Segments
• Managed by ElasticSearch
• Is the storage for the inverted index

25. E L A S T I C S E A R C H - I N T E R N A L S
• Basically ElasticSearch is a Lucene cluster manager
and API

26. L E S S O N S L E A R N E D - S H A R D S /
S E G M E N T S
• Deletion does only mark documents as deleted and
does not delete them immediately.
• Updating a document does only create a new one and
marks old one as deleted.
• The actual cleanup process happens in background
and can result in nice performance surprises.

27. L E S S O N S L E A R N E D - S H A R D S /
S E G M E N T S
• Nested documents live in the same Lucene Segment.
• Can bloat up memory usage a lot.
• They are treated as every other document.
• If you don’t necessarily always have to search in them,
go for parent-child.

28. L E S S O N S L E A R N E D - E L A S T I C S E A R C H
• Start with more than one instance - just too simple
• Major upgrades are a pain (0.90 -> 1.1)
• PHP Client Libraries mostly do not handle connection
pools properly, use elasticsearch-php
• ‘connectionPoolClass' => ‘Elasticsearch
ConnectionPoolStaticConnectionPool'
• let an intermediate webserver handle it

29. L E S S O N S L E A R N E D - E L A S T I C S E A R C H
• You will index more than one time. Promise.
Be prepared.
• Rebalancing is smooth, don’t worry.
• Have your metrics ready.
• “You can have a good time with ElasticSearch, if you
don't ignore the complexity and internals of this
distributed database.”

30. L E S S O N S L E A R N E D - E L A S T I C S E A R C H

31. L E S S O N S L E A R N E D - E L A S T I C S E A R C H

32. L E S S O N S L E A R N E D - I N D E X / M A P P I N G
• Different analysers should go into separate fields
• Score individually - iterative optimisations possible
• Keep a raw field
• Use dynamic_templates if you found the holy grail of
field analysis.
• Filter first! Querying and scoring is expensive.

33. L E S S O N S L E A R N E D - I N D E X / M A P P I N G

34. L E S S O N S L E A R N E D - I N D E X / M A P P I N G
GET /eyephoto/_mapping!
{!
"eyephoto6": {!
"mappings": {!
"photo": {!
"dynamic_templates": [!
{!
"string": {!
"mapping": {!
"type": "string",!
"index_analyzer": "photo_names",!
"search_analyzer": "photo_standard",!
"fields": {!
"raw": {!
"type": "string",!
"index": "not_analyzed"!
},!
"split": {!
"type": "string",!
"analyzer": "standard"!
}!
}!
},!
"match": "*",!
"match_mapping_type": "string"!
}!
}!
]
• Different analysers should go into separate fields

35. L E S S O N S L E A R N E D - I N D E X / M A P P I N G
{!
"took": 18,!
"timed_out": false,!
"_shards": {!
##########!
},!
"hits": {!
"total": 125,!
"max_score": 6.44889,!
"hits": [!
{!
#####!
"_id": "167480",!
#####!
}!
}!
]!
},!
"facets": {!
"topic": {!
"_type": "terms",!
"missing": 0,!
"total": 138,!
"other": 57,!
"terms": [!
{!
"term": "Coffee",!
"count": 81!
}!
]!
}!
}!
}
• Different analysers should go
into separate fields
POST /eyephoto/photo/_search!
{!
"size": 1,!
"fields": [!
"id"!
],!
"query": {!
"multi_match": {!
"query": "coff",!
"fields": [!
"topics"!
]!
}!
},!
"facets": {!
"topic": {!
"terms": {!
"field": "topics.raw",!
"size": 1!
}!
}!
}!
}

36. L E S S O N S L E A R N E D - I N D E X / M A P P I N G
POST /eyephoto/photo/_search!
{!
"query": {!
"bool": {!
"should": [!
{!
"multi_match": {!
"query": "lars",!
"operator": "and",!
"fields": [!
“name.raw^3",!
“name.split^2”,!
“name"!
]!
}!
},!
{!
"multi_match": {!
"query": "lars",!
"fields": [!
“name.raw^3”,!
“name.split^2”,!
“name”!
]!
}!
}!
]!
}!
• Different analysers should go
into separate fields

37. L E S S O N S L E A R N E D - I N D E X / M A P P I N G
• Read and write only to index aliases.
Index Name Index Aliases
eyephoto5 “eyephotoread”
eyephoto6 “eyephotowrite”

38. L E S S O N S L E A R N E D - I N D E X / M A P P I N G
• If you have a string or integer field, you can put an
array into it as well.
Ey 1
ye 1,2
eE 1
Em 1
Eye 1
yeE 1
eEm 1
ah 2
yea 2
eah 2

39. L E S S O N S L E A R N E D - I N D E X / M A P P I N G
• Use geohash wherever you query on lat/lng.
POST /eyephoto/photo/_search!
{!
"query": {!
"function_score": {!
"query": {!
"filtered": {!
"query": {!
"match_all": []!
},!
"filter": {!
"geohash_cell": {!
"location": {!
"lat": 52.5311,!
"lon": 13.404!
},!
"precision": 4,!
"neighbors": true!
} } } },!
"functions": [!
{!
"gauss": {!
"location": {!
"origin": "52.5311,13.404",!
"scale": "10km"!
}!
}!
},!
{!
"exp": {!
"uploaded": {!
"origin": "now",!
"scale": "2d"!
}!
}!
}!

40. L E S S O N S L E A R N E D - A G G R E G AT I O N S
• Aggregations give you recursive facets, handle with
care. "aggregations": {!
“user_fullname": {!
"filter": {!
"query": {!
"match": {!
"topics": {!
"query": "lars beer",!
"operator": "or"!
} } } },!
"aggs": {!
“user_fullname": {!
"terms": {!
"field": “user_fullname.raw”,!
"size": 3!
},!
"aggs": {!
“topics": {!
"filter": {!
"query": {!
"match": {!
“topics": {!
"query": "lars beer",!
"operator": "or"!
} } } },!
"aggs": {!
“topics": {!
"terms": {!
"field": “topics.raw”,!
"size": 3!
}!
}!
}!
},!

41. L E S S O N S L E A R N E D - A G G R E G AT I O N S
• Aggregations give you recursive facets, handle with
care. "user_fullname": {!
"doc_count": 678,!
"user_fullname": {!
"buckets": [!
{!
"key": "Lars 🍻 ",!
"doc_count": 678,!
"topics": {!
"doc_count": 5,!
"topics": {!
"buckets": [!
{!
"key": "Beer",!
"doc_count": 1!
},!
{!
"key": "BeerOps",!
"doc_count": 1!
},!
{!
"key": "Birthday beer in the snow",!
"doc_count": 1!
}!
]!
}!
}!
}!
]!
}!

42. O U T L O O K

43. O U T L O O K
• 1-liner search
• public release
• Localisation (snowball / stopwords)
• Keep indexed documents (e.g. albums) updated

44. N E X T I T E R AT I O N ( E VA L U AT I N G )
• Elasticsearch 1.1
• Oracle Java 1.8 (GC)
• more indexes and even more shards.
• restore API

45. 46
WE
ARE
HIRING