Speedy, Sleek Searching - Drayton Benner, BibleTech 2019

“Speedy, Sleek Searching”
Drayton C. Benner
Founder/President, Miklal Software Solutions
DraytonBenner@MiklalSoftware.com

Bible Software: bread and butter
• Library of Bibles and Bible reference resources
• Links among resources
• Search
Drayton Benner | Miklal Software Solutions | DraytonBenner@MiklalSoftware.com

Boolean Text Search: goals
• Functionality
• Powerful
• Flexible
• Enabling biblical researchers as well as laypersons
• Not resource intensive
• Fast
• Low memory usage
• Handle old devices
• Handle server requests without a server farm
• Scale to large libraries

Boolean Text Search: functionality
• Multilingual
• Hebrew, Aramaic, and Greek
• Bible translations in any language
• Surface forms and lemmas
• AND, OR searches
• (Jesus or Christ) and lord
• Ordered or not ordered
• Proximity for AND searches
• [within 3 verses and (1 chapter or 1 paragraph)]
• Imprecise words
• Wildcards: Abra*m, wom?n
• Regular expressions

Information Retrieval Basics: lexicon
…
conquer
conquering
conqueror
conscience
consciousness
consecrate
consent
consequently
consider
considerable
consign
consist
consolation
console
conspicuous
conspiracy
conspire
constant
constantly
constrain
construct
consult
consultation
consume
consuming
contain
container
contempt
contend
content
contentious
contentment
continual
continually
continue
contradict
contradiction
contrary
contribute
contribution
contrived
control
controversy
conversation
converse
conversion
convert
convict
conviction
convince
convinced
convincingly
convulse
cool
…

Information Retrieval Basics: inverted index
conquer 2 Kings 16:5 (32); Isaiah 7:6 (15); Revelation 6:2 (36); Revelation 11:7 (23);
Revelation 13:7 (14); Revelation 17:14 (13)
conquering Isaiah 18:2 (41); Isaiah 18:7 (27); Revelation 6:2 (21)
conqueror Micah 1:15 (6)
conscience 1 Samuel 25:31 (17); Acts 23:1 (24); Acts 24:16 (9); Romans 2:15 (16); Romans
9:1 (16); Romans 13:5 (14); 1 Corinthians 8:7 (23); 1 Corinthians 8:10 (8); 1
Corinthians 8:12 (4); 1 Corinthians 10:25 (23); 1 Corinthians 10:27 (31); 1
Corinthians 10:28 (7); 1 Corinthians 10:29 (12); 1 Corinthians 10:29 (22); 2
Corinthians 1:12 (17); 2 Corinthians 4:2 (6); 2 Corinthians 5:11 (27); 1 Timothy
1:5 (23); 1 Timothy 1:19 (15); 1 Timothy 3:9 (20); 1 Timothy 4:2 (3); 2 Timothy
1:3 (18); Titus 1:15 (6); Hebrews 9:9 (24); Hebrews 9:14 (30); Hebrews 10:22
(21); Hebrews 13:18 (14); 1 Peter 3:16 (8); 1 Peter 3:21 (19)

Inverted Index
conquer 698; 19,203; 54,897; 246,488; 565,574; 645,106; 802,193
conquering 89,654; 480,737; 749,240
conqueror 513,238
conscience 112,592; 118,761; 192,708; 223,519; 229,955; 326,459; 331,294; 372,501;
418,133; 436,530; 456,026; 458,388; 508,750; 510,470; 538,332; 547,994;
561,011; 564,817; 595,180; 595,757; 643,096; 668,186; 677,311; 781,107;
797,371; 808,857; 810,062

Inverted Index Compression
conquer 698; 19,203; 54,897; 246,488; 565,574;
645,106; 802,193
conquering 89,654; 480,737; 749,240
conqueror 513,238
conscience 112,592; 118,761; 192,708; 223,519;
229,955; 326,459; 331,294; 372,501;
418,133; 436,530; 456,026; 458,388;
508,750; 510,470; 538,332; 547,994;
561,011; 564,817; 595,180; 595,757;
643,096; 668,186; 677,311; 781,107;
797,371; 808,857; 810,062
• Store the first number, then
difference between each number
and the previous number
• Use compression that works well
for small numbers

Compression for Small Integers
• Fixed bit width
• Delta code
• Use gamma code for
length
• Offset as with delta
code
• Golomb code
• Parameterized on
token frequency
• Achieves close to
optimal code length
Table from Introduction to Information Retrieval by Christopher D. Manning et al.

Lexicon Retrieval from Disk
Word Same as previous Remaining
conquer 0 conquer
conquering 7 ing
conqueror 7 or
conscience 3 science
consciousness 6 ousness
consecrate 4 ecrate
consent 5 nt

Expanded Lexicon
• conquer ->
• conquer$
• onquer$c
• nquer$co
• quer$con
• uer$conq
• er$conqu
• r$conque
• $conquer
…
tinued$con
tinues$con
tinuing$con
tiny$des
tioch$an
tion$abomina
tion$ac
tion$accusa
tion$addi
tion$adjura
tion$adop
tion$affec
tion$afflic
tion$ambi
tion$appor
tion$associa
tion$atten
tion$celebra
tion$collec
tion$commemora
tion$commenda
tion$commo
tion$compensa
tion$comple
tion$composi
tion$concep
tion$condemna
tion$condi
tion$confirma
tion$confisca
tion$congrega
tion$connec
tion$consecra
tion$consola
tion$construc
tion$consulta
tion$conten
tion$contribu
tion$conversa
tion$convic
tion$convoca
tion$correc
tion$corrup
tion$crea
tion$decep
tion$declara
tion$dedica
tion$deduc
tion$delega
tion$demonstra
tion$deporta
tion$descrip
tion$desola
tion$destruc
tion$devasta
tion$devo
tion$dicta
tion$direc
tion$discre
tion$dissipa
tion$distinc
…

Expanded Lexicon
Expanded word Word ID Num rotations
tinued$con 2871 3
tinues$con 2872 3
tinuing$con 2873 3
tiny$des 3486 3
tioch$an 894 2
tion$abomina 382 7
tion$ac 481 2

book
chapter chapter
verseverse
token token token token token token
verse
token token token
verse
token token
Proximity

book
chapter chapter
verseverse
token token token token token token
verse
token token token
verse
token token
Proximity: multiple hierarchies
word word word word word wordwordwordword
• Hierarchies need not be ordered identically
• Allows for multiple orders of books (e.g. Protestant canon, Hebrew Bible, etc.)

Demos
• 2nd generation e-ink Kindle
• KJV surface forms search
• Entire app given 2MB memory max
• Java
• Old Android tablet
• Greek lemmas search
• Java
• Laptop
• ESV surface forms search
• C++

Questions?

Speedy, Sleek Searching - Drayton Benner, BibleTech 2019

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

Speedy, Sleek Searching - Drayton Benner, BibleTech 2019