XPDays 2018

1. Immutable Database
Safe Way To Migrate Large Data Stores
@YEVHEN

2. The true story of one
internal Hackathon
that went South

3. Go PROD!
BRAVO!

9. Евгений Бобров
17 лет в IT
Архитектор
Платформа
OSS
YouScan.io
yevhen

11. Social Media Monitoring

12. Now using YouScan

16. 2.5Bln docs
7K indices
14K shards
10TB (1 replica)

17. Data Architecture
T101
T42
T345
20GB
1 shard
40GB
2 shards
120GB
4 shards

18. Data Schema
"mention": {
"properties": {
"cityId": {
"index": "not_analyzed",
"type": "string"
},
"reactionsWow": {
"type": "integer"
},
"imageScenes": {
"properties": {
"confidence": {
"type": "float“
…
120 fields
10KB avg

19. Schema Evolution
2015 2016 2017 2018
You’re
here

21. REINDEXING

22. V2V1
copy

23. REINDEXING

24. Aliases
POST /_aliases/
{
"actions" : [{
"add" : {
"index" : "101_1",
"alias" : "101"
}
}]
}

26. Custom Alias Table
Topic Index Version
42 42_1 2
101 101_1 3
345 345_1 3

27. Client Singleton
var es = Elastic.Init(table);
Topic Index Version
42 42_1 2
101 101_1 3
345 345_1 3

28. Client Code
var result = client.Search(query).Path(topic);

29. Proxy Function
var result = es.Execute(topic, (index, client) =>
client.Search(query).Path(index));

30. Transfer

31. Snapshot Copy
Progress
1
2
1
2
3 3while (lastId <= maxId)

32. Snapshot Copy
1
2
1
2
3 3
W W W
missing
stale
Progress
DONE
lastId = maxId

33. DOWNTIME

34. Service Downtime
Availability % per year per month per week per day
99% 3.65 days 7.31 hours 1.68 hours 14.40 minutes
99.9% 8.77 hours 43.83 minutes 10.08 minutes 1.44 minutes
99.99% 52.60 minutes 4.38 minutes 1.01 minutes 8.64 seconds
50M
2K/sec
= ~7 hours

35. Event
Source
All the
Things!

36. Event Sourcing
… Tag
Added
Mention
Created
N 2 1
T42
Projector
Log (FIFO)
1.apply(e) 2.rec(e)
3. apply(e) -> POST/PUT/DELETE
4.query()

37. Temporal Event Sourcing
… .. …
N 2 1
T42
Projector
1.apply(e)
3.query()
X
2. apply(e) -> POST/PUT/DELETE

38. Events?

39. Direct Index Access
T42

40. CQRS/ES
T42
AddTag API Projector
GET/SEARCH
TagAdded

41. Event Store
… Tag
Added
Mention
Created
N 2 1
Stream42
… Tag
Removed
Mention
Favorited
N 2 1
Stream102
T42
T102

42. Projector

43. Migration
Automata
powered by

44. Preparing
old
3. GetMaxId()
new
2. Create()
Tag
Added
2 1
1. StartLog()

45. Copying
Preparing
2 2
while (lastId <= maxId)

46. …
Tag
Removed
Tag
Added
Mention
Created
4 3 2 1
new
while (remaining > SwitchThreshold)
apply(e)
Replaying
Copying

47. Switching
Replaying
catchup(tail)
old
Projector
new
tail
Tag
Added
Mention
Created
2 1

48. Switching
Replaying
old
Projector
new
update(alias)
100%
…
Tag
Added
Mention
Created
2 1
Topic Index Version
42 42_1 2
101 101_1 3
345 345_2 3

49. Switching
Replaying
new old
Topic Index Version
42 42_1 2
101 101_1 3
345 345_2 3
1.apply(e)
3.query()
Projector
Point Of
No Return!
2.apply(e)
X … Tag
Added
Mention
Created
2 1

55. New Cluster Migration
YsTokenizer 1.3
ys-es-01
YsTokenizer 2.1
ys-es-02

56. Minor Changes
Topic Index Version Cluster
42 42_1 2 ys-es-01
101 101_1 3 ys-es-02
345 345_1 3 ys-es-02
var result = es.Execute(topic, (index, client) =>
client.Search(query).Path(index));

58. transform

59. Non-Destructive Transformation
text : "Lorem ipsum
http://bit.ly/17f42"
V1
links : ["http://bit.ly/17f42"]
text : "Lorem ipsum
http://bit.ly/17f42"
V2

60. Convert On Write/Copy
if (version == IndexVersion.Next)
doc.Links = ExtractLinks(doc);
copy.Links = ExtractLinks(src);
Projector:
Copying:

61. Destructive Transformation
region : "Киев",
city : "Киев"
V1
V2
region : "Киев",
city : "Киев"
regionId : 38044
cityId : 546232

62. Progressive Destruction
region : "Киев",
city : "Киев"
V1
V3
region : "Киев",
city : "Киев"
V2
region : "Киев",
city : "Киев"
regionId : 38044
cityId : 546232

63. Backward Compatibility
var result = es.Execute(topic, (index, client) =>
{
var city = index.Version == IndexVersion.Current
? criteria.City
: LookupId(criteria.City);
query = query.Term(“city”).Equals(city);
return client.Search(query).Path(index));
}

65. V2V1

66. Switching
Replaying
old
Topic Index Version
42 42_1 2
101 101_1 3
345 345_2 3
1.apply(e)
3.query()
Projector X
Point Of
No Return!
… Tag
Added
Mention
Created
2 1
new
2.apply(e)

68. Syncing
Switching
… Tag
Added
Mention
Created
2 1
new
Topic Index Version
42 42_1 2
101 101_1 3
345 345_2 3
1.apply(e)
4.query()
Projector
3.apply(e)
2.rec(e)
old

69. Rollback
Syncing
tail
Tag
Added
Mention
Created
2 1
new
Projector
1.catchup(tail)
Topic Index Version
42 42_1 2
101 101_1 3
345 345_2 3
old

70. Rollback
Syncing
tail
Tag
Added
Mention
Created
2 1
new
Projector
100%
Topic Index Version
42 42_1 2
101 101_1 3
345 345_1 2
update(alias)
old

71. Rollback
Syncing
…
Tag
Added
Mention
Created
2 1
old new
3.query()
Projector
1.apply(e)
Topic Index Version
42 42_1 2
101 101_1 3
345 345_1 2
X
2.apply(e)

72. Parallel Versions
V2V1
sync

74. Safe Migration

75. Challenges

79. Projector
Topic Index Version
42 42_1 2
101 101_1 3
345 345_2 3
… Tag
Added
Mention
Created
2 1
Highly Available
Infrastructure

80. Conclusions

81. Immutable infrastructure is
comprised of immutable components
that are replaced for every deployment,
rather than being updated in-place

82. Immutable Database is
Scalable
Atomic
Fault-tolerant
Efficient

83. Immutable Database for apps
Stability
Availability
Flexibility
Evolvability

84. Immutable Database as CD practice
Stability
Availability
Flexibility
Evolvability
SRE
Agile