The document discusses implementing the Smith-Waterman algorithm for genomic sequence alignment in Spark. It motivates the problem of scaling gene alignment to large genomic datasets. It describes testing the algorithm on reference datasets ranging from 49MB to 234MB, with reference sequences from 1 to over 30,000. Key challenges included understanding the dynamic programming algorithm, adapting it to Spark's functional programming model, and distributing either the algorithm or large reference datasets across a cluster.

1. Implementing
Smith-Waterman in Spark
Elizabeth Fong
Insight Data Engineering NY
September - October 2015

2. Motivation
- What is your genomic sequence?
- Gene Therapy: personalised treatment based on your genomic sequence
- Many gene alignment algorithms available
- Not many distributed

3. Motivation
- What is your genomic sequence?
- Gene Therapy: personalised treatment based on your genomic sequence
- Many gene alignment algorithms available
- Not many distributed
The Problem
Where does this align?
ACTCA
ATGCAGAC
ACTCA
ATGCAGAC
ACTCA
ATGCAGAC
ACTC_ A
A _TGCAGAC

4. About the Data
Dataset
- NCBI ftp site for dataset download
- Total size = 26.5GB out of approx. 200TB
- Dataset contains 518 files of 12,321,160 sequences of mRNA
- mean (average) number of base pairs per sequence = 2,160
- median number of base pairs per sequence = 1,609
Input Files
- Randomly selected sequences from the dataset

5. Metadata
The sequence

6. Many
Reference
Sequences

7. The sequence

8. Input
Best-Aligned
Reference Sequence
where the
sequences
match

9. Reference
Input
Legend
a alignment
i insertion
d deletionOptimal
Alignment

10. Scaling up: Tests and Data
Reference File
Size
/ MB
# Reference
Sequences
Input File Size
/ KB
# Inputs # Comparisons
Time (distribute
reference)
/ h
49.0 17,864 2 19 339416 0.75
69.0 23,398 3 31 725338 2.05
83.8 35,751 3 32 1144032 2.64
97.9 30,096 2 19 571824 1.50
118.6 ? 2 ? N/A
234.9 ? 3 ? N/A

11. Scaling up: Tests and Data

12. Scaling up: Additional Tests
read length 80
# reference sequences 10
# reference files 1

13. Scaling up: Additional Tests
read length 80
# reference sequences 10
# reference files 1

14. Scaling up: Additional Tests
# reads 5
read length 80
reference sequence length 400
# reference files 1

15. Scaling up: Additional Tests
# reads 5
read length 80
# reference sequences 1
# reference files 1

16. Scaling up: Parallelization
Parallelize the algorithm, reads or the reference set
- Algorithm
- Dynamic programming algorithm
- Fills up a matrix where each cell depends on 3 of its neighbouring cells
- Reads
- reads are much shorter
- but need to store all the scores for each read on reduce
- Reference Set
- much longer than reads
- # references per file range from 1 to 73,030

17. Challenges Encountered
- Understanding the algorithm
- Object-Oriented Java → Functional Spark
- OutOfMemory: Java heap space
- Distributing the algorithm
- Distributing either dataset
- ftp → S3

18. BA Computer Science, minor in Molecular Biology
Mount Holyoke College

19. http://vlab.amrita.edu/?sub=3&brch=274&sim=1433&cnt=1

20. http://vlab.amrita.edu/?sub=3&brch=274&sim=1433&cnt=1

21. Let the following scores be:
Match = 5
Mismatch = -3
Gap = -4
- C G T G A A T T C A T
-
G
A
C
T
T 13 9
A 14
C
0
alignment
insertion
deletion
score = max
{

22. - C G T G A A T T C A T
-
G
A
C
T
T 13 9
A 14
Alignment (diagonal)
Score = 13 + (match or mismatch)
= 13 + 5
= 18

23. Insertion (vertical)
Score = 9 + (gap)
= 9 - 4
= 5
- C G T G A A T T C A T
-
G
A
C
T
T 13 9
A 14
C

24. Deletion (horizontal)
Score = 14 + (gap)
= 9 - 4
= 10
- C G T G A A T T C A T
-
G
A
C
T
T 13 9
A 14
C

25. score = max
{
0
alignment
insertion
deletion
score = max{ 0 , 18 , 5 , 10 }
= 18
- C G T G A A T T C A T
-
G
A
C
T
T 13 9
A 14 18
C

26. http://vlab.amrita.edu/?sub=3&brch=274&sim=1433&cnt=1

27. http://vlab.amrita.edu/?sub=3&brch=274&sim=1433&cnt=1

28. align align align align align del align
G A A T T C C
G A C T T _ C

29. align align align align align ins align
G A A T T _ C
G A C T T A C

30. Both alignment scores = 18
Therefore both are optimal alignments
http://vlab.amrita.edu/?sub=3&brch=274&sim=1433&cnt=1