Paper: http://drops.dagstuhl.de/opus/volltexte/2019/10481/ Given a string $T$ on an alphabet of size $\sigma$, we describe a bidirectional Burrows-Wheeler index that takes $O(|T|\log{\sigma})$ bits of space, and that supports the addition \emph{and removal} of one character, on the left or right side of any substring of $T$, in constant time. Previously known data structures that used the same space allowed constant-time addition to any substring of $T$, but they could support removal only from specific substrings of $T$. We also describe an index that supports bidirectional addition and removal in $O(\log{\log{|T|}})$ time, and that takes a number of words proportional to the number of left and right extensions of the maximal repeats of $T$. We use such fully-functional indexes to implement bidirectional, frequency-aware, variable-order de Bruijn graphs with no upper bound on their order, and supporting natural criteria for increasing and decreasing the order during traversal.

1. Fully-functional bidirectional
Burrows-Wheeler indexes
Djamal Belazzougui1
Fabio Cunial2,3
1 DTISI, CERIST, Algiers.
2 Max Planck Institute for Molecular Cell Biology and Genetics, Dresden.
3 Center for Systems Biology Dresden

2. Constant-space descriptor:
Bidirectional index

3. Bidirectional index
: all starting/ending positions of W in T
Suffix tree of T and of the reverse of T
Affix tree, affix array, bidirectional BWT...
Synchronous and asynchronous...
Applied to read mapping with mismatches,
searching for RNA secondary structures...
Cánovas, Rivals. Full compressed affix tree representations. DCC 2017.

4. Bidirectional contraction
Martin
Djamal and Fabio
ok, but i'd need bidirectional
*contraction* for the sequencing
software i'm working on...
sorry, that's only possible from right-
maximal or left-maximal substrings...
or for substrings that occur only once
Belazzougui et al. Versatile succinct representations of the bidirectional BWT. ESA 2013.
Munro et al. Space-efficient construction of compressed indexes in deterministic linear time. SODA 2017.

5. Bidirectional contraction
Martin
Djamal and Fabio
ok, but i'd need bidirectional
*contraction* for the sequencing
software i'm working on...
...
sorry, that's only possible from right-
maximal or left-maximal substrings...
or for substrings that occur only once
are you sure you need contraction?
Belazzougui et al. Versatile succinct representations of the bidirectional BWT. ESA 2013.
Munro et al. Space-efficient construction of compressed indexes in deterministic linear time. SODA 2017.

6. Bidirectional contraction
New
New

7. Variable-order de Bruijn graphs
longest right-extension with same freq.
longest prefix with different freq.

8. Variable-order de Bruijn graphs
Freq. Change k by δ Query time
(constant σ)
Space
(constant σ)
No upper
bound on k
Bidirectional
Belazzougui
et al. 2016
Boucher
et al. 2015
Díaz-Domínguez
et al. 2018
New
New constant
constant
bits
bits
bits
words
bits
Boucher et al. Variable-order de Bruijn graphs. DCC 2015.
Belazzougui et al. Bidirectional variable-order de Bruijn graphs. LATIN 2016.
Díaz-Domínguez et al. Assembling omnitigs using hidden-order de Bruijn graphs. arXiv 2018.

9. Preliminaries

10. C
A
A•# #
#G
#
#
#
C
C•
C•A
G
A
GG
A A C•
G
G
G
A C
ST
A
A
G• G•
G•
G
C•
G•
Suffix tree

11. Suffix tree
C
A
A•# #
#G
#
#
#
C
C•
C•A
G
A
GG
A A C•
G
G
G
A C
ST
A
A
G• G•
G•
G
C•
G•

12. Maximal repeats subgraph
C
A
A•# #
#G
#
#
#
C
C•
C•A
G
A
GG
A A C•
G
G
G
A C
ST
A
A
G• G•
G•
G
C•
G•

13. C
A
A
#
#
C
C
G
A
GG
A A C
G
G
G
A C
SLT *
A
C
G
Suffix-link tree of the reverse

14. C
A
A
#
#
C
C
G
A
GG
A A C
G
G
G
A C
SLT *
A
C
G
Suffix-link tree of the reverse

15. SLTT
1
2 6
7 9
8 10
11
12
13
3
4
5
1
2 9 16
10 15 17
11 14 18 23 24
12 13 19 22 28
20 21 25
26 27
3 6
4 5 7 8
STT
Length of a maximal repeat
isMaximalRepeat2 = 1010111101001
isMaximalRepeat1 = 1100010000000001110000011000
Belazzougui, Cunial. A framework for space-efficient string kernels. Algorithmica 2017.

16. Bidirectional BWT index
All operations take linear time
in the size of their output
Can be built in randomized time, bits.
bits
Belazzougui et al. Versatile succinct representations of the bidirectional BWT. ESA 2013.

17. Minimal compact automaton that recognizes the suffixes of a string.
Nodes (except the sink) are maximal repeats.
Compact Directed Acyclic Word Graph
AG# C G
AGC
A AC C
CG # G AG C
A A #
A A
#
# G
C CG G
C C
G
ST
A
G
A
C
A
A
C
C
A G
C
#
G
A
#
C
C
#
CDAWG

18. Minimal compact automaton that recognizes the suffixes of a string.
Nodes (except the sink) are maximal repeats.
Compact Directed Acyclic Word Graph
A
G
A
C
A
A
C
C
A G
C
#
A
#
C
C
#
CDAWG
maxrep
chain of explicit
Weiner links
rightmax
but not
leftmax
W
...
V
G

19. Minimal compact automaton that recognizes the suffixes of a string.
Nodes (except the sink) are maximal repeats.
Compact Directed Acyclic Word Graph
A
G
A
C
A
A
C
C
A G
C
#
A
#
C
C
#
CDAWG
WX
X
C
G

20. Minimal compact automaton that recognizes the suffixes of a string.
Nodes (except the sink) are maximal repeats.
Compact Directed Acyclic Word Graph
A
G
A
C
A
A
C
C
A G
C
#
A
#
C
C
#
CDAWG
W
Y
X
CC
G

21. Minimal compact automaton that recognizes the suffixes of a string.
Nodes (except the sink) are maximal repeats.
Compact Directed Acyclic Word Graph
A
G
A
C
A
A
C
C
A G
C
#
A
#
C
C
#
CDAWG
W
Y
X
CC
parent in ST
G

22. Representing the ST with the CDAWG
BWT
c
a
c
c
c
c
Belazzougui, Cunial. Representing the suffix tree with the CDAWG. CPM 2017.

23. Representing the ST with the CDAWG
A
G
A
C
A
A
C
C
A G
C
#
A
#
C
C
#
CDAWG
CA
A
G
Belazzougui, Cunial. Representing the suffix tree with the CDAWG. CPM 2017.

24. Representing the ST with the CDAWG
Belazzougui, et al. Composite repetition-aware data structures. CPM 2015.

25. Contracting in constant time

26. Left-contraction for non-right-maximal
C
A
A•# #
#G
#
#
#
C
C•
C•A
G
A
GG
A A C•
G
G
G
A C
ST
A
A
G• G•
G•
G
C•
G•

27. Left-contraction for non-right-maximal
C
A
A•# #
#G
#
#
#
C
C•
C•A
G
A
GG
A A C•
G
G
G
A C
ST
A
A
G• G•
G•
G
C•
G•

28. Left-contraction for non-right-maximal
C
A
A•# #
#G
#
#
#
C
C•
C•A
G
A
GG
A A C•
G
G
G
A C
ST
A
A
G• G•
G•
G
C•
G•

29. Left-contraction for non-right-maximal
C
A
A•# #
#G
#
#
#
C
C•
C•A
G
A
GG
A A C•
G
G
G
A C
ST
A
A
G• G•
G•
G
C•
G•

30. Left-contraction for non-right-maximal
C
A
A•# #
#G
#
#
#
C
C•
C•A
G
A
GG
A A C•
G
G
G
A C
ST
A
A
G• G•
G•
G
C•
G•

31. Left-contraction for non-right-maximal
A
#G
#
#
#C•
G
A
G
A A C•
G
G
A C
ST
G•
C•
G• #
#
G
A
G
A A C
G
G
A C
SLT *
C
G

32. Left-contraction for non-right-maximal
A
#G
#
#
#C•
G
A
G
A A C•
G
G
A C
ST
G•
C•
G• #
#
G
A
G
A A C
G
G
A C
SLT *
C
G

33. Left-contraction for non-right-maximal
A
#G
#
#
#C•
G
A
G
A A C•
G
G
A C
ST
G•
C•
G• #
#
G
A
G
A A C
G
G
A C
SLT *
C
G

34. Storing maxrep lengths in practice
bytes per character
0 0.2 0.4 0.6
Human genome
Bacterial genomes
E. coli
Proteins
Human reads
Metagenome
Repetitive proteins
S. cerevisiae
S. paradoxus
H. influenzae
5.9 Gbp
Joint work with Jarno Alanko
34 Gbp
29 Gbp
33 Gbp
28 Gbp

35. "Infinite-order" de Bruijn graph
No upper limit

36. "Infinite-order" de Bruijn graph
linear time in difference between orders
unsupported
No upper limit

37. DBG ⟷CDAWG

38. CDAWG
RLBWT
ST
WLA
Weighted level-
ancestor on
maxreps
Affix links for
maxreps
words
time

39. CDAWG
RLBWT
ST
WLA
Weighted level-
ancestor on
maxreps
Affix links for
maxreps
WLA
Weighted level-
ancestor on
reverse maxreps
Affix links for
reverse
maxreps
CDAWG
RLBWT
Reverse ST
words
time

40. CDAWG
RLBWT
ST
WLA
Weighted level-
ancestor on
maxreps
Affix links for
maxreps
WLA
Weighted level-
ancestor on
reverse maxreps
Affix links for
reverse
maxreps
CDAWG
RLBWT
Reverse ST
words
time

41. WLA
Weighted level-
ancestor on
maxreps
WA
Weighted
ancestor on
maxreps
words
time
words
time
C
A
A•# #
#G
#
#
#
C
C•
C•A
G
A
GG
A A C•
G
G
G
A C
ST
A
A
G• G•
G•
G
C•
G•
C
A
A
#
#
C
C
G
A
GG
A A C
G
G
G
A C
SLT *
A
C
G

42. CDAWG
RLBWT
ST
WLA
Weighted level-
ancestor on
maxreps
Affix links for
maxreps
WLA
Weighted level-
ancestor on
reverse maxreps
Affix links for
reverse
maxreps
CDAWG
RLBWT
Reverse ST
words
time

43. longest right-extension with same freq. longest suffix with different freq.
W VW
W
Hidden-order operations
Díaz-Domínguez et al. Assembling omnitigs using hidden-order de Bruijn graphs. arXiv 2018.

44. Illumina
Genomes
CCS
0.2% avg error rate
13 kb avg length
Wenger et al. Highly-accurate long-read sequencing improves variant detection and assembly of a human genome. bioRxiv 2019.

45. Fully-functional bidirectional
Burrows-Wheeler indexes
Djamal Belazzougui1
Fabio Cunial2,3
1 DTISI, CERIST, Algiers.
2 Max Planck Institute for Molecular Cell Biology and Genetics, Dresden.
3 Center for Systems Biology Dresden