This document provides an overview of the distributed search capabilities of Yokozuna. It discusses how Yokozuna indexes commit logs to enable full-text search of code commits. An inverted index is used to search commit summaries and bodies. The document also covers replication, high availability, and self-healing capabilities such as Active Anti-Entropy and hinted handoff that allow the system to repair indexes and continue serving search requests even when nodes fail.

1. Yokozuna
Distributed SearchYou Don’t Think About
Ryan Zezeski May 14th 2013
1Tuesday, May 21, 13

2. Live Demo
2Tuesday, May 21, 13

3. Live Demos
3Tuesday, May 21, 13

4. PROBLEM?
4Tuesday, May 21, 13

5. PROBLEM?
SOLUTION!
5Tuesday, May 21, 13

6. Solution Pre-made
6Tuesday, May 21, 13

7. Piece At A Time
7Tuesday, May 21, 13

8. Goals
• Don’t screw up
• Show howYokozuna
doesn’t make you think
(too hard)
• Teach you about Search
• Neat things you can do
withYokozuna
8Tuesday, May 21, 13

9. PROBLEM: SEARCH
FOR COMMITS
ABOUT SPECIFIC
FEATURE/BUG
MAKE IT GOOGLE-LIKE
9Tuesday, May 21, 13

10. SOLUTION: INDEX
COMMITS IN
YOKOZUNA -
“COMMIT LOG
SEARCHER” (CLS)
10Tuesday, May 21, 13

11. Anatomy of a Commit
Msg
11Tuesday, May 21, 13

12. Primary Key
12Tuesday, May 21, 13

13. Any Node Will Do
13Tuesday, May 21, 13

14. Term Query
14Tuesday, May 21, 13

15. Query Any Node
15Tuesday, May 21, 13

16. Boolean (1)
repo:riak_kv repo:riak_core
16Tuesday, May 21, 13

17. Boolean (2)
repo:riak_kv AND author:”Ryan Zezeski”
17Tuesday, May 21, 13

18. Boolean (3)
commit_author:"Ryan Zezeski" OR
commit_author:"Joseph Blomstedt" NOT
commit_repo:riak_kv
18Tuesday, May 21, 13

19. Range (1)
commit_repo:riak_core
AND commit_dt:[NOW-1YEAR TO NOW]
19Tuesday, May 21, 13

20. Range (2)
commit_repo:riak_core
AND commit_dt:[NOW-1YEAR TO NOW]
I RAN THIS ON
2013-05-10
sort=dt asc
20Tuesday, May 21, 13

21. Wildcard (1)
*:*
GET TOTAL
COUNT FIRST
21Tuesday, May 21, 13

22. Wildcard (2)
commit_repo:riak_*
NOTICE COUNT
IS LOWER
22Tuesday, May 21, 13

23. WHAT ABOUT
SEARCHING
SUMMARY AND
BODY?
23Tuesday, May 21, 13

24. THE INVERTED
INDEX
24Tuesday, May 21, 13

25. AN INDEX - BUT
INVERTED
25Tuesday, May 21, 13

26. EVERYONE KNOWS
WHAT IT IS
26Tuesday, May 21, 13

27. EVEN NON-TECH
PEOPLE
27Tuesday, May 21, 13

28. YES...EVENYOUR
PARENTS
28Tuesday, May 21, 13

29. What’s In A Book?
29Tuesday, May 21, 13

30. • WORDS
• PARAGRAPHS
• SECTIONS
• CHAPTERS
• ETC.
30Tuesday, May 21, 13

31. AND PAGE
NUMBERS
31Tuesday, May 21, 13

32. 32Tuesday, May 21, 13

33. PAGE NUMBERS ARE
AN IMPLICIT INDEX
33Tuesday, May 21, 13

34. PAGE NUMBER TO
WORDS
WORD TO PAGE
NUMBERS
INVERTED
34Tuesday, May 21, 13

35. STOLEN FROM BLOG OF RICKY HO: http://
horicky.blogspot.com/2013/02/text-processing-
part-2-inverted-index.html
35Tuesday, May 21, 13

36. HOW DOYOU GET
THE WORDS IN THE
FIRST PLACE?
36Tuesday, May 21, 13

37. Analysis - The Iceberg
That Sunk The Titanic
37Tuesday, May 21, 13

38. Phrase (1)
subject:hinted OR subject:handoff
OR body:hinted OR body:handoff
38Tuesday, May 21, 13

39. Phrase (2)
subject:”hinted handoff”
OR body:”hinted handoff”
39Tuesday, May 21, 13

40. Phrase (3)
subject:”partition vnode”
OR body:”partition vnode”
40Tuesday, May 21, 13

41. Phrase (4)
subject:”partition vnode”~4
OR body:”partition vnode”~4
41Tuesday, May 21, 13

42. Exact Term
subject:behavior OR body:behavior
42Tuesday, May 21, 13

43. Fuzzy Term
subject:behavior~1 OR body:behavior~1
43Tuesday, May 21, 13

44. Ranking
ADD SCORE TO
FL
SCORE ADDED
TO EVERY
RESULT
44Tuesday, May 21, 13

45. RECALL, PRECISION,
AND RELEVANCY, OH
MY!
45Tuesday, May 21, 13

46. RELEVANCY - FOR A
GIVEN QUERY &
DOC SET THERE IS
AN IDEAL ANSWER
OF ONLY RELEVANT
DOCS
46Tuesday, May 21, 13

47. RECALL = WHAT %
OF IDEAL ANSWER
SET WAS RETRIEVED
47Tuesday, May 21, 13

48. PRECISION = WHAT
% OF ANSWER IS
RELEVANT
48Tuesday, May 21, 13

49. RECALLvs.PRECISION
ASYOU INCREASE
RECALLYOU
DEGRADE PRECISION
49Tuesday, May 21, 13

50. SOLR DETERMINES
RELEVANCYVIA THE
NOTION OF
SIMILARITY
50Tuesday, May 21, 13

51. SOLR USES TF-IDF:
TERM FREQUENCY,
INVERSE DOCUMENT
FREQUENCY
51Tuesday, May 21, 13

52. Dismax + Facets +
Highlighting
FACETS
HIGHLIGHTING
DISMAX
52Tuesday, May 21, 13

53. FACET - A
TAXONOMY OF
YOUR QUERY BASED
ON FIELD’SVALUES
53Tuesday, May 21, 13

54. FACETS ALLOW
“DRILL DOWN” -
THEY GUIDE THE
USER
54Tuesday, May 21, 13

55. HIGHLIGHTING
GIVESYOUR RESULTS
CONTEXT - ALLOWS
QUICKER
DETERMINATION OF
RELEVANCY
55Tuesday, May 21, 13

56. DISMAX - DISjunction
MAX - A QUERY
HANDLER MEANT
FOR DIRECT USER
INPUT
56Tuesday, May 21, 13

57. All Nodes Up
57Tuesday, May 21, 13

58. All Nodes Up - Query
58Tuesday, May 21, 13

59. Node 4 Down
59Tuesday, May 21, 13

60. Node 4 Down - Query
60Tuesday, May 21, 13

61. Node 3 & 4 Down
61Tuesday, May 21, 13

62. Node 3 & 4 Down -
Query
62Tuesday, May 21, 13

63. REPLICATION
PROVIDES HIGH
AVAILABILITY
2 3 41
START WITH 4 NODES
63Tuesday, May 21, 13

64. Write 3 Replicas
2 3 41
64Tuesday, May 21, 13

65. Take 2 Nodes Down
2 3 41
1 REPLICA STILL
AVAILABLE
65Tuesday, May 21, 13

66. WHAT IF DATA IS
WRITTEN WHILE
NODES ARE DOWN?
66Tuesday, May 21, 13

67. YZ Not StoredYet
67Tuesday, May 21, 13

68. StoreYZ Log
68Tuesday, May 21, 13

69. QueryYZ - Node 1& 2
69Tuesday, May 21, 13

70. Set XFer Limit To 0
70Tuesday, May 21, 13

71. Start Nodes 3 & 4
71Tuesday, May 21, 13

72. Query Solr Direct
WHEN MAKING THIS
DEMO I WAS
EXPECTING THIS TO BE
0 BUT I FORGOT ABOUT
AAE WHICH STARTED
KICKING IN BEFORE
HANDOFF - SELF
HEALING FTW!
72Tuesday, May 21, 13

73. Set Xfer Limit To 64
73Tuesday, May 21, 13

74. Handoff Occurs
74Tuesday, May 21, 13

75. 0 Pending Xfers
75Tuesday, May 21, 13

76. Solr Direct (Again)
NOTICE IT’S NOW 301,
UP FROM 54, MORE
PROOF THAT HANDOFF
OCCURRED - NOTE
THIS QUERY IS GOING
DIRECT TO ONLY 1
SHARD
76Tuesday, May 21, 13

77. Query Node 4YZ
NOW HITYOKOZUNA
ON NODE4 (NOTICE
CHANGE IN PORT #) -
THIS WILL RUN A DIST
SEARCH AND THUS
RETURN CORRECT
COUNT
77Tuesday, May 21, 13

78. Data Ownership
AVNODE
THE RING
78Tuesday, May 21, 13

79. Node Down
X
X
X
XX
X
X
X
X
X
79Tuesday, May 21, 13

80. Write Fallback
X
X
X
XX
X
X
X
X
X
80Tuesday, May 21, 13

81. Node Up
HINTED
HANDOFF WILL
MOVE REPLICA
TO PRIMARY
81Tuesday, May 21, 13

82. WHAT IFYOU RM -RF
THE INDEX DIR?
82Tuesday, May 21, 13

83. Kill The Data
RM -RF THE INDEX DIRECTORY
KILL THE SOLR PROC
83Tuesday, May 21, 13

84. Auto Restart
YOKOZUNA NOTICES SOLR DIED
AND AUTOMATICALLY RESTARTS
IT
84Tuesday, May 21, 13

85. Node 4 - 0 Results
85Tuesday, May 21, 13

86. AAE Notices Missing
Data
86Tuesday, May 21, 13

87. Node 4 - 13 Results
DATA IS RE-INDEXED OVER TIME
87Tuesday, May 21, 13

88. More AAE Repair
88Tuesday, May 21, 13

89. Node 4 - 128 Results
MORE INDEXES ARE REPAIRED,
THIS CONTINUES UNTIL AAE
REPAIRS ALL INDEXES
89Tuesday, May 21, 13

90. WHAT EVEN IS
ACTIVE ANTI-
ENTROPY?
90Tuesday, May 21, 13

91. Mo Systems Mo Failure
• index update could get
lost
• files can become
truncated/corrupted
• accidental `rm -rf`
• segfault at right time
• etc...
91Tuesday, May 21, 13

92. MYRAID OF FAILURE
SCENARIOS - FROM
OBVIOUS TO NEARLY
INVISIBLE
92Tuesday, May 21, 13

93. ENTROPY IS DAMAGE
AAE IS SELF HEALING
STRIKER!!!! EHEM, I
MEAN, ENTROPY!!!!
93Tuesday, May 21, 13

94. REPAIR EFFICIENTLY -
NOT STUPIDLY
94Tuesday, May 21, 13

95. LearnYou Some Merkle
For A Great Good
BIG UPS TO @jtuple FOR THE AAE DIAGRAMS
95Tuesday, May 21, 13

96. Segments
EACH SEGMENT IS LIST OF
KEY-HASH PAIRS
96Tuesday, May 21, 13

97. Segment Hashes
HASH OF HASHES IN
SEGMENT
97Tuesday, May 21, 13

98. Hash O’ Hashes
98Tuesday, May 21, 13

99. WHAT HAPPENS
DURING
EXCHANGE?
99Tuesday, May 21, 13

100. Start With 2 Trees
100Tuesday, May 21, 13

101. Compare Top Hashes
TOP HASHES DON’T
MATCH -
SOMETHING IS
DIFFERENT
101Tuesday, May 21, 13

102. Compare Child Hashes
NARROW DOWN
THE DIVERGENT
SEGMENT
102Tuesday, May 21, 13

103. Recur
NARROW DOWN
THE DIVERGENT
SEGMENT CONT...
103Tuesday, May 21, 13

104. Iter Key-Hash Pairs
ITER FINAL LIST OF
HASHES TO FIND
DIVERGENT KEYS
104Tuesday, May 21, 13

105. Repair Divergent Keys
REPAIR (RE-INDEX)
KEYS THAT ARE
DIVERGENT (RED)
105Tuesday, May 21, 13

106. CODE FOR
DETECTION AND
REPAIR - NOT
PREVENTION
106Tuesday, May 21, 13

107. WHAT HAPPENS IF 3
NODES GO DOWN?
107Tuesday, May 21, 13

108. Stop 3 Nodes
108Tuesday, May 21, 13

109. Query
109Tuesday, May 21, 13

110. CONSISTENCY
vs.
AVAILABILITY
110Tuesday, May 21, 13

111. Uptime - Story of 9s
UPTIME = (MTBF - MTTR) / MTBF
111Tuesday, May 21, 13

112. Uptime is Flawed
IF THE SYSTEM IS
DOWN, BUT NO
ONE MAKES A
REQUEST, IS IT
REALLY DOWN?
112Tuesday, May 21, 13

113. Yield - Uptime of the
People
YIELD = QUERIES COMPLETED / QUERIES
OFFERED
113Tuesday, May 21, 13

114. Harvest vs.Yield
HARVEST = DATA AVAIL / COMPLETE DATA
IF FACE OF FAILUREYOU CAN’T HAVE BOTH
FOR A SINGLE REQUEST
114Tuesday, May 21, 13

115. IN TIMES OF
TROUBLE -
YOKOZUNA
CHOOSES HARVEST
FOR QUERIES
115Tuesday, May 21, 13

116. TECHNICALLY -
YOKOZUNA IS
ALWAYS < 100%
HARVEST IN A NON-
QUIESCENT CLUSTER
116Tuesday, May 21, 13

117. YOKOZUNA FAVORS
YIELD FOR WRITES
117Tuesday, May 21, 13

118. ONCE RIAK 1.4 SHIPS
-YOKOZUNA LANDS
IN MASTER
118Tuesday, May 21, 13

119. THANKYOU
HTTP://GITHUB.COM/BASHO/YOKOZUNA
119Tuesday, May 21, 13