The document describes running a sample analysis using the CloudBurst tool on Hadoop to map 100,000 Illumina reads against a Streptococcus suis reference genome, allowing up to 3 mismatches. It provides details of the sample input data formats and locations, commands to format the data and run CloudBurst, and snippets of the output showing the mapping job progress and completing successfully in under 20 minutes.

1. 참고 http://sourceforge.net/apps/mediawiki/cloudburst-bio/index.php?title=Sample_Results
[hadoop@skcc-nebdap02 CloudBurst-1.1.0]$ ll
total 24332
-rw-r--r-- 1 hadoop hadoop 1995984 Jun 15 17:19 100k.3.txt
-rw-r--r-- 1 hadoop hadoop 1995984 Dec 5 2008 100k.3.txt.gold
-rw-r--r-- 1 hadoop hadoop 4493593 Jun 15 17:06 100k.br
-rw-r--r-- 1 hadoop hadoop 4388895 Dec 5 2008 100k.fa
-rw-r--r-- 1 hadoop hadoop 1177790 Jun 15 17:06 100k.fa.map
-rw-r--r-- 1 hadoop hadoop 8337 Dec 5 2008 cloudburst.err.gold
-rw-r--r-- 1 hadoop hadoop 57014 Jul 9 2010 CloudBurst.jar
-rw-r--r-- 1 hadoop hadoop 4067962 Jul 9 2010 ConvertFastaForCloud.jar
-rw-r--r-- 1 hadoop hadoop 4067959 Jul 9 2010 PrintAlignments.jar
-rw-r--r-- 1 hadoop hadoop 1452 Jul 9 2010 README.txt
drwxr-xr-x 2 hadoop hadoop 4096 Jun 15 17:19 results
-rw-r--r-- 1 hadoop hadoop 579773 Jun 15 17:06 s_suis.br
-rw-r--r-- 1 hadoop hadoop 2040970 Dec 5 2008 s_suis.fa
-rw-r--r-- 1 hadoop hadoop 21 Jun 15 17:06 s_suis.fa.map
[hadoop@skcc-nebdap02 CloudBurst-1.1.0]$ cat README.txt
Sample data for CloudBurst
==========================
CloudBurst has several parameters to control the sensitivity of the
alignment algorithm. Here it finds the unambiguous best alignment for
100,000 reads allowing up to 3 mismatches when mapping to the corresponding
S. suis genome.
== Sample input data
s_suis.fa: Streptococcus suis reference genome sequence
100k.fa: 100,000 36bp Illumina reads available from
http://www.sanger.ac.uk/Projects/S_suis/
== Format the input data
$ java -jar ConvertFastaForCloud.jar s_suis.fa s_suis.br
$ java -jar ConvertFastaForCloud.jar 100k.fa 100k.br

2. s_suis.br: reference genome in CloudBurst binary format
100k.br: Reads in CloudBurst binary format
... 생략 ...
[hadoop@skcc-nebdap02 CloudBurst-1.1.0]$ head s_suis.fa
>Streptococcus_suis
atgaaccaagaacaacttttttggcaacgatttattgaattggcaaaggtaaattttaag
ccatctatttatgatttttatgtcgctgatgcaaaattactcggaatcaaccagcaagtt
gccaatattttcttaaatcgtccatttaaaaaagatttctgggaaaaaaacttcgaagag
ttaatgattgccgctagttttgaaagctacggagagcctcttaccatccaatatcaattt
... 생략 ...
acagaggatgaacaggagattaggaatactacaaacacaagaagttcaatagttcaccag
gtacagacacttgagccggctactcctcaagaaacttttaaaccggttcattctgatata
aaatcccagtacacctttgctaattttgtacaaggagacaataatcactgggcaaaggct
gcagctttagctgtatctgataacctaggtgagctctacaatccattattcatttttggt
ggtcctggtcttggaaaaactcatattttaaatgcgattggaaataaggttctagccgat
[hadoop@skcc-nebdap02 CloudBurst-1.1.0]$ wc -l s_suis.fa
33460 s_suis.fa
[hadoop@skcc-nebdap02 CloudBurst-1.1.0]$ head 100k.fa
>1
GCCTGTTCTTTACATGATTTTTGGTCTAGTGTATGG
>2
AACCGCTGTAAAGGCTTCTGCCACACCGATTTCTTG
>3
GAGGTGATTGTGGTATTGT.GGTAAATCGGTGATTG
>4
GCTTTAGCCGACCTGAACT.GACTACAAGTTGACCA
>5
AAAGGCTACCCGCGGTTGAACCTTACGTGACACATT
[hadoop@skcc-nebdap02 CloudBurst-1.1.0]$ tail 100k.fa
>99996
AATGCCCGTAACAACGGGCTTTTATCTTGTTCTAAA
>99997
GTCAGATAGCGCAGGAATTTCAAAGGAATTTGGACC
>99998
AGTTAACTCTTCAGCTGTAAAGTTGTAGTTTTCTAA
>99999

3. GCGGCATAAATTGGATAAAGAAAGAACTGAAGGACA
>100000
GTTACCATGTATTGTGACAGATAACCACGGTGGAGT
[hadoop@skcc-nebdap02 hadoop]$ bin/hadoop dfs -mkdir /data/cloudburst
[hadoop@skcc-nebdap02 hadoop]$ bin/hadoop dfs -put ../CloudBurst-1.1.0/s_suis.br /data/cloudburst
[hadoop@skcc-nebdap02 hadoop]$ bin/hadoop dfs -put ../CloudBurst-1.1.0/100k.br /data/cloudburst
[hadoop@skcc-nebdap02 hadoop]$ bin/hadoop jar ../CloudBurst-1.1.0/CloudBurst.jar
Usage: CloudBurst refpath qrypath outpath minreadlen maxreadlen k allowdifferences filteralignments #mappers #reduces
#fmappers #freducers blocksize redundancy
1. refpath: path in hdfs to the reference file
2. qrypath: path in hdfs to the query file
3. outpath: path to a directory to store the results (old results are automatically deleted)
4. minreadlen: minimum length of the reads
5. maxreadlen: maximum read length
6. k: number of mismatches / differences to allow (higher number requires more time)
7. allowdifferences: 0: mismatches only, 1: indels as well
8. filteralignments: 0: all alignments, 1: only report unambiguous best alignment (results identical to RMAP)
9. #mappers: number of mappers to use. suggested: #processor-cores * 10
10. #reduces: number of reducers to use. suggested: #processor-cores * 2
11. #fmappers: number of mappers for filtration alg. suggested: #processor-cores
12. #freducers: number of reducers for filtration alg. suggested: #processor-cores
13. blocksize: number of qry and ref tuples to consider at a time in the reduce phase. suggested: 128
14. redundancy: number of copies of low complexity seeds to use. suggested: # processor cores
[hadoop@skcc-nebdap02 hadoop]$ bin/hadoop jar ../CloudBurst-1.1.0/CloudBurst.jar /data/cloudburst/s_suis.br
/data/cloudburst/100k.br /data/results 36 36 3 0 1 240 48 24 24 128 16 >& cloudburst.err
[hadoop@skcc-nebdap02 hadoop]$ cat cloudburst.err
refath: /data/cloudburst/s_suis.br
qrypath: /data/cloudburst/100k.br
outpath: /data/results-alignments
MIN_READ_LEN: 36
MAX_READ_LEN: 36
K: 3
SEED_LEN: 9
FLANK_LEN: 30
ALLOW_DIFFERENCES: 0
FILTER_ALIGNMENTS: true
NUM_MAP_TASKS: 240

4. NUM_REDUCE_TASKS: 48
BLOCK_SIZE: 128
REDUNDANCY: 16
Removing old results
12/06/15 17:11:27 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
12/06/15 17:11:28 INFO mapred.FileInputFormat: Total input paths to process : 2
12/06/15 17:11:28 INFO mapred.JobClient: Running job: job_201206112243_0018
12/06/15 17:11:29 INFO mapred.JobClient: map 0% reduce 0%
12/06/15 17:11:47 INFO mapred.JobClient: map 12% reduce 0%
12/06/15 17:11:48 INFO mapred.JobClient: map 14% reduce 0%
12/06/15 17:11:49 INFO mapred.JobClient: map 15% reduce 0%
12/06/15 17:11:50 INFO mapred.JobClient: map 17% reduce 0%
12/06/15 17:11:51 INFO mapred.JobClient: map 19% reduce 0%
12/06/15 17:11:52 INFO mapred.JobClient: map 21% reduce 0%
12/06/15 17:11:53 INFO mapred.JobClient: map 36% reduce 0%
12/06/15 17:11:54 INFO mapred.JobClient: map 40% reduce 0%
12/06/15 17:11:55 INFO mapred.JobClient: map 45% reduce 0%
12/06/15 17:11:56 INFO mapred.JobClient: map 49% reduce 0%
12/06/15 17:11:57 INFO mapred.JobClient: map 56% reduce 0%
12/06/15 17:11:58 INFO mapred.JobClient: map 57% reduce 0%
12/06/15 17:11:59 INFO mapred.JobClient: map 74% reduce 0%
12/06/15 17:12:00 INFO mapred.JobClient: map 80% reduce 1%
12/06/15 17:12:01 INFO mapred.JobClient: map 80% reduce 2%
12/06/15 17:12:02 INFO mapred.JobClient: map 83% reduce 3%
12/06/15 17:12:03 INFO mapred.JobClient: map 91% reduce 4%
12/06/15 17:12:05 INFO mapred.JobClient: map 95% reduce 6%
12/06/15 17:12:06 INFO mapred.JobClient: map 95% reduce 9%
12/06/15 17:12:07 INFO mapred.JobClient: map 95% reduce 10%
12/06/15 17:12:08 INFO mapred.JobClient: map 100% reduce 14%
12/06/15 17:12:09 INFO mapred.JobClient: map 100% reduce 17%
12/06/15 17:12:10 INFO mapred.JobClient: map 100% reduce 18%
12/06/15 17:12:11 INFO mapred.JobClient: map 100% reduce 22%
12/06/15 17:12:13 INFO mapred.JobClient: map 100% reduce 23%
12/06/15 17:12:14 INFO mapred.JobClient: map 100% reduce 28%
12/06/15 17:12:15 INFO mapred.JobClient: map 100% reduce 31%
12/06/15 17:12:17 INFO mapred.JobClient: map 100% reduce 51%
12/06/15 17:12:18 INFO mapred.JobClient: map 100% reduce 65%

5. 12/06/15 17:12:19 INFO mapred.JobClient: map 100% reduce 70%
12/06/15 17:12:20 INFO mapred.JobClient: map 100% reduce 87%
12/06/15 17:12:21 INFO mapred.JobClient: map 100% reduce 92%
12/06/15 17:12:22 INFO mapred.JobClient: map 100% reduce 94%
12/06/15 17:12:23 INFO mapred.JobClient: map 100% reduce 98%
12/06/15 17:12:26 INFO mapred.JobClient: map 100% reduce 100%
12/06/15 17:12:31 INFO mapred.JobClient: Job complete: job_201206112243_0018
12/06/15 17:12:32 INFO mapred.JobClient: Counters: 31
12/06/15 17:12:32 INFO mapred.JobClient: Job Counters
12/06/15 17:12:32 INFO mapred.JobClient: Launched reduce tasks=48
12/06/15 17:12:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=2980992
12/06/15 17:12:32 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
12/06/15 17:12:32 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
12/06/15 17:12:32 INFO mapred.JobClient: Rack-local map tasks=158
12/06/15 17:12:32 INFO mapred.JobClient: Launched map tasks=241
12/06/15 17:12:32 INFO mapred.JobClient: Data-local map tasks=83
12/06/15 17:12:32 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=1106915
12/06/15 17:12:32 INFO mapred.JobClient: File Input Format Counters
12/06/15 17:12:32 INFO mapred.JobClient: Bytes Read=5587101
12/06/15 17:12:32 INFO mapred.JobClient: File Output Format Counters
12/06/15 17:12:32 INFO mapred.JobClient: Bytes Written=2707836
12/06/15 17:12:32 INFO mapred.JobClient: FileSystemCounters
12/06/15 17:12:32 INFO mapred.JobClient: FILE_BYTES_READ=140515797
12/06/15 17:12:32 INFO mapred.JobClient: HDFS_BYTES_READ=6112267
12/06/15 17:12:32 INFO mapred.JobClient: FILE_BYTES_WRITTEN=288167030
12/06/15 17:12:32 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=2707836
12/06/15 17:12:32 INFO mapred.JobClient: Map-Reduce Framework
12/06/15 17:12:32 INFO mapred.JobClient: Map output materialized bytes=140584917
12/06/15 17:12:32 INFO mapred.JobClient: Map input records=100032
12/06/15 17:12:32 INFO mapred.JobClient: Reduce shuffle bytes=140436273
12/06/15 17:12:32 INFO mapred.JobClient: Spilled Records=5558658
12/06/15 17:12:32 INFO mapred.JobClient: Map output bytes=134956851
12/06/15 17:12:32 INFO mapred.JobClient: Total committed heap usage (bytes)=57936314368
12/06/15 17:12:32 INFO mapred.JobClient: CPU time spent (ms)=1693370
12/06/15 17:12:32 INFO mapred.JobClient: Map input bytes=5073092
12/06/15 17:12:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=24638
12/06/15 17:12:32 INFO mapred.JobClient: Combine input records=0
12/06/15 17:12:32 INFO mapred.JobClient: Reduce input records=2774585

6. 12/06/15 17:12:32 INFO mapred.JobClient: Reduce input groups=254196
12/06/15 17:12:32 INFO mapred.JobClient: Combine output records=0
12/06/15 17:12:32 INFO mapred.JobClient: Physical memory (bytes) snapshot=57459982336
12/06/15 17:12:32 INFO mapred.JobClient: Reduce output records=81128
12/06/15 17:12:32 INFO mapred.JobClient: Virtual memory (bytes) snapshot=754874736640
12/06/15 17:12:32 INFO mapred.JobClient: Map output records=2779329
CloudBurst Finished
Alignment time: 65.36
NUM_FMAP_TASKS: 24
NUM_FREDUCE_TASKS: 24
Removing old results
12/06/15 17:12:32 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
12/06/15 17:12:32 INFO mapred.FileInputFormat: Total input paths to process : 48
12/06/15 17:12:39 INFO mapred.JobClient: Running job: job_201206112243_0019
12/06/15 17:12:40 INFO mapred.JobClient: map 0% reduce 0%
12/06/15 17:12:54 INFO mapred.JobClient: map 62% reduce 0%
12/06/15 17:12:55 INFO mapred.JobClient: map 100% reduce 0%
12/06/15 17:13:06 INFO mapred.JobClient: map 100% reduce 16%
12/06/15 17:13:07 INFO mapred.JobClient: map 100% reduce 33%
12/06/15 17:13:09 INFO mapred.JobClient: map 100% reduce 58%
12/06/15 17:13:10 INFO mapred.JobClient: map 100% reduce 75%
12/06/15 17:13:12 INFO mapred.JobClient: map 100% reduce 87%
12/06/15 17:13:13 INFO mapred.JobClient: map 100% reduce 91%
12/06/15 17:13:15 INFO mapred.JobClient: map 100% reduce 100%
12/06/15 17:13:20 INFO mapred.JobClient: Job complete: job_201206112243_0019
12/06/15 17:13:20 INFO mapred.JobClient: Counters: 31
12/06/15 17:13:20 INFO mapred.JobClient: Job Counters
12/06/15 17:13:20 INFO mapred.JobClient: Launched reduce tasks=24
12/06/15 17:13:20 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=207232
12/06/15 17:13:20 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
12/06/15 17:13:20 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
12/06/15 17:13:20 INFO mapred.JobClient: Rack-local map tasks=5
12/06/15 17:13:20 INFO mapred.JobClient: Launched map tasks=48
12/06/15 17:13:20 INFO mapred.JobClient: Data-local map tasks=43
12/06/15 17:13:20 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=245651
12/06/15 17:13:20 INFO mapred.JobClient: File Input Format Counters
12/06/15 17:13:20 INFO mapred.JobClient: Bytes Read=2707836

7. 12/06/15 17:13:20 INFO mapred.JobClient: File Output Format Counters
12/06/15 17:13:20 INFO mapred.JobClient: Bytes Written=2485042
12/06/15 17:13:20 INFO mapred.JobClient: FileSystemCounters
12/06/15 17:13:20 INFO mapred.JobClient: FILE_BYTES_READ=2188332
12/06/15 17:13:20 INFO mapred.JobClient: HDFS_BYTES_READ=2713260
12/06/15 17:13:20 INFO mapred.JobClient: FILE_BYTES_WRITTEN=6039532
12/06/15 17:13:20 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=2485042
12/06/15 17:13:20 INFO mapred.JobClient: Map-Reduce Framework
12/06/15 17:13:20 INFO mapred.JobClient: Map output materialized bytes=2195100
12/06/15 17:13:20 INFO mapred.JobClient: Map input records=81128
12/06/15 17:13:20 INFO mapred.JobClient: Reduce shuffle bytes=2153793
12/06/15 17:13:20 INFO mapred.JobClient: Spilled Records=162088
12/06/15 17:13:20 INFO mapred.JobClient: Map output bytes=2028200
12/06/15 17:13:20 INFO mapred.JobClient: Total committed heap usage (bytes)=14471921664
12/06/15 17:13:20 INFO mapred.JobClient: CPU time spent (ms)=95390
12/06/15 17:13:20 INFO mapred.JobClient: Map input bytes=2703324
12/06/15 17:13:20 INFO mapred.JobClient: SPLIT_RAW_BYTES=5424
12/06/15 17:13:20 INFO mapred.JobClient: Combine input records=81128
12/06/15 17:13:20 INFO mapred.JobClient: Reduce input records=81044
12/06/15 17:13:20 INFO mapred.JobClient: Reduce input groups=76511
12/06/15 17:13:20 INFO mapred.JobClient: Combine output records=81044
12/06/15 17:13:20 INFO mapred.JobClient: Physical memory (bytes) snapshot=13169172480
12/06/15 17:13:20 INFO mapred.JobClient: Reduce output records=74502
12/06/15 17:13:20 INFO mapred.JobClient: Virtual memory (bytes) snapshot=193761902592
12/06/15 17:13:20 INFO mapred.JobClient: Map output records=81128
FilterAlignments Finished
Filtering time: 48.481
Total Running time: 113.841
[hadoop@skcc-nebdap02 hadoop]$
[hadoop@skcc-nebdap02 hadoop]$ bin/hadoop dfs -get /data/results/ ../CloudBurst-1.1.0/results
[hadoop@skcc-nebdap02 hadoop]$ cd ../CloudBurst-1.1.0
[hadoop@skcc-nebdap02 CloudBurst-1.1.0]$ java -jar PrintAlignments.jar results | sort -nk4 > 100k.3.txt
[hadoop@skcc-nebdap02 CloudBurst-1.1.0]$ head -n 20 100k.3.txt
1 766133 766169 1 1 +
1 297899 297935 2 0 -
1 1325118 1325154 4 1 +
1 145970 146006 7 1 -
1 553513 553549 8 0 -

8. 1 1779842 1779878 9 0 -
1 86299 86335 10 0 -
1 1503808 1503844 11 2 +
1 397758 397794 12 0 +
1 241778 241814 13 0 -
1 626711 626747 14 0 +
1 142141 142177 15 1 +
1 1401129 1401165 16 1 -
1 306289 306325 17 1 +
1 628571 628607 18 1 -
1 815172 815208 19 0 -
1 1624600 1624636 20 0 +
1 13779 13815 21 0 +
1 129064 129100 22 1 +
1 1382938 1382974 24 2 +
[hadoop@skcc-nebdap02 CloudBurst-1.1.0]$ tail -n 20 100k.3.txt
1 1796768 1796804 99976 2 -
1 1021128 1021164 99978 0 -
1 1350005 1350041 99980 1 +
1 799280 799316 99981 2 -
1 139518 139554 99983 0 +
1 57158 57194 99985 0 +
1 1663030 1663066 99986 2 +
1 549235 549271 99987 0 -
1 1400509 1400545 99988 0 +
1 880593 880629 99989 0 +
1 918064 918100 99990 0 +
1 937994 938030 99992 1 -
1 94456 94492 99993 0 +
1 1144320 1144356 99994 0 +
1 1441627 1441663 99995 0 +
1 1281557 1281593 99996 0 +
1 1323611 1323647 99997 2 -
1 800095 800131 99998 0 -
1 1956458 1956494 99999 1 +
1 134848 134884 100000 2 -
[hadoop@skcc-nebdap02 CloudBurst-1.1.0]$ wc -l 100k.3.txt
74502 100k.3.txt