@molly_struve
Elasticsearch 5
or and
Bust
1

@molly_struve
2

@molly_struve
TL;DR
3

@molly_struve
4

@molly_struve
5

@molly_struve
6

@molly_struve
7

@molly_struve
8

@molly_struve
9

@molly_struve
Elasticsearch Lingo
10

@molly_struve
11
Elasticsearch
ES

@molly_struve
12
Node NodeNode
Servers

@molly_struve
13
Node NodeNode
Cluster

@molly_struve
14
2.x 5.x

@molly_struve
15
The Story

@molly_struve
16
2017
March

@molly_struve
17
Cluster

@molly_struve
18

@molly_struve
19

@molly_struve
Upgrade Steps
20
1
2
3
4
Shutdown the cluster

@molly_struve
Upgrade Steps
21
1
2
3
4
Shutdown the cluster
Upgrade Elasticsearch on all nodes

@molly_struve
Upgrade Steps
22
1
2
3
4
Shutdown the cluster
Upgrade Elasticsearch on all nodes
Deploy Elasticsearch 5 code changes

@molly_struve
Upgrade Steps
23
1
2
3
4
Shutdown the cluster
Upgrade Elasticsearch on all nodes
Deploy Elasticsearch 5 code changes
Start Elasticsearch on all nodes
👍

@molly_struve
24
Elasticsearch 5.x Cluster
cpu
load
cpu
load
cpu
load
cpu
load
cpu
load
cpu
load

@molly_struve
25
Elasticsearch 5.x Cluster
cpu
load
cpu
load
cpu
load
cpu
load
cpu
load
cpu
load
😬 😬 😬

@molly_struve
26

@molly_struve
27
x

@molly_struve
28
Elasticsearch 5.x Cluster
☠
☠ ☠ ☠
☠

@molly_struve
29
Elasticsearch 5.x Cluster

@molly_struve
30
Elasticsearch 5.x Cluster
cpu
load
cpu
load
cpu
load
cpu
load
cpu
load
cpu
load

@molly_struve
31
Elasticsearch 5.x Cluster
☠
☠ ☠ ☠
☠

@molly_struve
32
Elasticsearch 5.x Cluster
☠

@molly_struve
33
Debugging Mode

@molly_struve
34

@molly_struve
35
Elasticsearch 5 upgrade followed by cluster crash

@molly_struve
36
Elasticsearch 5 cluster instability

@molly_struve
37
Why does Elasticsearch 5 suck so much?

@molly_struve
38

@molly_struve
The Story
39

@molly_struve
40

@molly_struve
41
x x

@molly_struve
42

@molly_struve
43
😩 15+ hours 😩

@molly_struve
44

@molly_struve
45

@molly_struve
46
Rollback

@molly_struve
47

@molly_struve
48
Rollback
2.x5.x

@molly_struve
49
Rollback
5 days
2.x5.x

@molly_struve
50
x x x x
x x

@molly_struve
51

@molly_struve
52

@molly_struve
53
🙌

@molly_struve
54
Workaround Deployed

@molly_struve
55
Workaround Deployed

@molly_struve
56

@molly_struve
Lessons Learned
57

@molly_struve
58
1
2
3
4
Have a Rollback Plan
5
6
Lessons Learned

@molly_struve
Rollback Plan
Can you rollback the software inline?
59

@molly_struve
Rollback Plan
Can you rollback the software inline?
60
How long and hard will a rollback be?

@molly_struve
Rollback Plan
Can you rollback the software inline?
61
How long and hard will a rollback be?
Worst case scenario the shit out of the upgrade

@molly_struve
62
1
2
3
4
Have a Rollback Plan
5
Do Performance Testing
6
Lessons Learned

@molly_struve
63
The last upgrade was great, this one
will be too!

@molly_struve
64
The last upgrade was great, this one
will be too!

@molly_struve
65
Performance Test

@molly_struve
66
Performance Test

@molly_struve
67
1
2
3
4
Have a Rollback Plan
5
Do Performance Testing
Don't Ignore Small Warning Signs
6
Lessons Learned

@molly_struve
68
Node
Local
Elasticsearch
☠

@molly_struve
69
Don’t ignore small warning
signs

@molly_struve
70

@molly_struve
71
1
2
3
4
Have a Rollback Plan
5
Do Performance Testing
Don't Ignore Small Warning Signs
6
Lessons Learned

@molly_struve
72
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
6
Lessons Learned

@molly_struve
73
Community

@molly_struve
74

@molly_struve
75
Don’t Wait

@molly_struve
76
Ask!

@molly_struve
77
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
Leader and Management Support Are Crucial
6
Lessons Learned

@molly_struve
78
Engineers

@molly_struve
79
Vice President of
Engineering

@molly_struve
80
Vice President of
Engineering

@molly_struve
81
Vice President of
Engineering
🛡

@molly_struve
82
Trust

@molly_struve
83
Fail Forward

@molly_struve
84
VP’s

@molly_struve
85
VP’s
Managers

@molly_struve
86
VP’s
Managers
C Suite Execs

@molly_struve
87

@molly_struve
88
🛡

@molly_struve
89
Trust

@molly_struve
90
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
Leader and Management Support Are Crucial
6
Lessons Learned
Your Team Matters

@molly_struve
91
Developer/Engineer
🖥

@molly_struve
92
Developer/Engineer
🖥 😊

@molly_struve
93
😊😊😊

@molly_struve
94
😱😳😩

@molly_struve
95
😱😳😩
😐😟😬
😭😡🤔

@molly_struve
96
Character is
everything

@molly_struve
97
Character is
everything

@molly_struve
98
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
Leader and Management Support Are Crucial
6
Lessons Learned
Your Team Matters

@molly_struve
99
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
Leader and Management Support Are Crucial
6
Lessons Learned
Your Team Matters

@molly_struve
100
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
Leader and Management Support Are Crucial
6
Lessons Learned
Your Team Matters

@molly_struve
101
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
Leader and Management Support Are Crucial
6
Lessons Learned
Your Team Matters

@molly_struve
102
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
Leader and Management Support Are Crucial
6
Lessons Learned
Your Team Matters

@molly_struve
103
Elasticsearch Outage
2017

@molly_struve
104
Embrace Your
Mistakes

@molly_struve
105
Embrace Your
Mistakes

@molly_struve
106
1
2
3
4
Have a Rollback Plan
Use the Community
5
Do Performance Testing
Don't Ignore Small Warning Signs
Leader and Management Support Are Crucial
6
Lessons Learned
Your Team Matters

@molly_struve
107
Embrace it

@molly_struve
108
Learn from it
Embrace it

@molly_struve
109
Learn from it
Share it with others
Embrace it

@molly_struve
110
Questions?