Presentation to cover the data and file formats commonly used in next generation sequencing (high throughput sequencing) analyses. From nucleotide ambiguity codes, FASTA and FASTQ, quality scores to SAM and BAM, CIGAR strings and variant calling format. This was given as part of the EPIZONE Workshop on Next Generation Sequencing applications and Bioinformatics in Brussels, Belgium in April 2016.

1. NGS data formats and analyses
Richard Orton
Viral Genomics and Bioinformatics
Centre for Virus Research
University of Glasgow
1

2. NGS – HTS – 1st – 2nd – 3rd Gen
• Next Generation Sequencing is now the
Current Generation Sequencing
• NGS = High-Throughput Sequencing (HTS)
• 1st Generation: The automated Sanger
sequencing method
• 2nd Generation: NGS (Illumina, Roche 454,
Ion Torrent etc)
• 3rd Generation: PacBio & Oxford Nanopore:
The Next Next Generation Sequencing.
Single molecule sequencing.
2

3. FASTA Format
• Text based format for representing nucleotide or protein sequences.
• Typical file extensions
– .fasta, .fas, .fa, .fna, .fsa
• Two components of a FASTA sequence
– Header line – begins with a `>` character – followed by a sequence name and an optional description
– Sequence line(s) – the sequence data either all on a single line or spread across multiple lines
• Typically, sequence data is spread across multiple lines of length 80, 70, or 60 bases
>gi|661348725|gb|KM034562.1| Zaire ebolavirus isolate Ebola virus/H.sapiens-wt/SLE/2014/Makona-G3686.1, complete genome
CGGACACACAAAAAGAAAGAAGAATTTTTAGGATCTTTTGTGTGCGAATAACTATGAGGAAGATTAATAA
TTTTCCTCTCATTGAAATTTATATCGGAATTTAAATTGAAATTGTTACTGTAATCATACCTGGTTTGTTT
CAGAGCCATATCACCAAGATAGAGAACAACCTAGGTCTCCGGAGGGGGCAAGGGCATCAGTGTGCTCAGT
TGAAAATCCCTTGTCAACATCTAGGCCTTATCACATCACAAGTTCCGCCTTAAACTCTGCAGGGTGATCC
AACAACCTTAATAGCAACATTATTGTTAAAGGACAGCATTAGTTCACAGTCAAACAAGCAAGATTGAGAA
TTAACTTTGATTTTGAACCTGAACACCCAGAGGACTGGAGACTCAACAACCCTAAAGCCTGGGGTAAAAC
ATTAGAAATAGTTTAAAGACAAATTGCTCGGAATCACAAAATTCCGAGTATGGATTCTCGTCCTCAGAAA
GTCTGGATGACGCCGAGTCTCACTGAATCTGACATGGATTACCACAAGATCTTGACAGCAGGTCTGTCCG
Name Description
Sequence lines
Header line `>`
3

4. FASTA Format
• Can have multiple sequences in a single FASTA file
• The ‘>’ character signifies the end of one sequence and the beginning of the next
one
• Sequences do not need to be the same length
>gi|123456780|gb|KM012345.1| Epizone test sequence 1
CGGACACACAAAAAGAAAGAAGAATTTTTAGGATCTTTTGTGTGCGAATAACTATGAGGAAGATTAATAA
TTTTCCTCTCATTGAAATTTATATCGGAATTTAAAT
>gi|123456781|gb|KM012346.1| Epizone test sequence 2
CAGAGCCATATCACCAAGATAGAGAACAACCTAGGTCTCCGGAGGGGGCAAGGGCATCAGTGTGCTCAGT
TGAAAATCCCTTGTCAACATCTAGGCCTTATCACATCACAAGTTCCGCCTTAAAC
>gi|123456782|gb|KM012347.1| Epizone test sequence 3
AACAACCTTAATAGCAACATTATTGTTAAAGGACAGCATTAGTTCACAGTCAAACAAGCAAGATTGAGAA
TTAACTTTGATTTTGAACCTGAACACCCAGAGGACTGGAGACTCAAC
>gi|123456783|gb|KM012348.1| Epizone test sequence 4
ATTAGAAATAGTTTAAAGACAAATTGCTCGGAATCACAAAATTCCGAGTATGGATTCTCGTCCTCAGAAA
GTCTGGATGACGCCGAGTCTCACTGAATCTGACATGGATTACCACAAGATCTTGACAGCAGGG
>gi|123456784|gb|KM012349.1| Epizone test sequence 5
ATTAGAAATAGTTTAAAGACAAATTGCTCGGAATCACAAAATTCCGAGTATGGATTCTCGTCCTCAGAAA
GTCTGGATGACGCCGAGTCTCACTGAATCTGACATGGAAAAAAAAAAAAAAAAAAAAGCAGGTCTGTCCG
4

5. Sequence Characters – IUPAC Codes
• The nucleic acid notation currently in use was first formalized by the
International Union of Pure and Applied Chemistry (IUPAC) in 1970.
Symbol Description Bases Represented Num
A Adenine A 1
C Cytosine C 1
G Guanine G 1
T Thymine T 1
U Uracil U 1
W Weak A T 2
S Strong C G 2
M aMino A C 2
K Keto G T 2
R puRine A G 2
Y pYrimidine C T 2
B not A (B comes after A) C G T 3
D not C (D comes after C) A G T 4
H not G (H comes after G) A C T 4
V not T (V comes after T & U) A C G 4
N Any Nucleotide A C G T 4
- Gap 0
5

6. FASTA Names
• GenBank, at the National Center for Biotechnology Information (NCBI), is the NIH genetic sequence
database, an annotated collection of all publicly available DNA sequences.
– Part of the International Nucleotide Sequence Database Collaboration, which also comprises the DNA
DataBank of Japan (DDBJ), the European Molecular Biology Laboratory (EMBL).
• GI (genInfo identifier) number
– A unique integer number which identifies a particular sequence (DNA or prot)
– As of September 2016, the integer sequence identifiers known as "GIs" will no longer be included in the
GenBank
• Accession Number
– A unique alphanumeric unique identifier given to a sequence (DNA or prot)
– Accession.Version
>gi|661348725|gb|KM034562.1| Zaire ebolavirus isolate Ebola virus/H.sapiens-wt/SLE/2014/Makona-G3686.1, complete genome
CGGACACACAAAAAGAAAGAAGAATTTTTAGGATCTTTTGTGTGCGAATAACTATGAGGAAGATTAATAA
TTTTCCTCTCATTGAAATTTATATCGGAATTTAAATTGAAATTGTTACTGTAATCATACCTGGTTTGTTT
CAGAGCCATATCACCAAGATAGAGAACAACCTAGGTCTCCGGAGGGGGCAAGGGCATCAGTGTGCTCAGT
TGAAAATCCCTTGTCAACATCTAGGCCTTATCACATCACAAGTTCCGCCTTAAACTCTGCAGGGTGATCC
AACAACCTTAATAGCAACATTATTGTTAAAGGACAGCATTAGTTCACAGTCAAACAAGCAAGATTGAGAA
TTAACTTTGATTTTGAACCTGAACACCCAGAGGACTGGAGACTCAACAACCCTAAAGCCTGGGGTAAAAC
ATTAGAAATAGTTTAAAGACAAATTGCTCGGAATCACAAAATTCCGAGTATGGATTCTCGTCCTCAGAAA
GTCTGGATGACGCCGAGTCTCACTGAATCTGACATGGATTACCACAAGATCTTGACAGCAGGTCTGTCCG
GI number
Description
Accession Number
Version Number
6

7. FASTQ Format
• Typically you will get your NGS reads in FASTQ format
– Oxford Nanopore – FAST5 format
– PacBio – HDF5 format
• FASTQ is a text-based format for storing both a nucleotide sequence and its corresponding quality
score.
• Four lines per sequence (read)
– @ - The UNIQUE sequence name
– The nucleotide sequence {A, C, G, T, N} – on 1 line
– + - The quality line break – sometimes see +SequenceName
– The quality scores {ASCII characters} – on 1 line
• The quality line is always the same length as the sequence line
– The 1st quality score corresponds to the quality of the 1st sequence
– The 14th quality score corresponds to the quality of the 14th sequence
• Multiple FASTQ sequence per file – millions
• Sequences can be different length
@EAS139:136:FC706VJ:2:5:1000:12850 1:Y:18:ATCACG
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+
BBBBCCCC?<A?BC?7@@???????DBBA@@@@A@@ Quality scores
Sequence bases
Sequence name
Quality line break
7

8. FASTQ Names
• Specific to Illumina Sequences
• @EAS139:136:FC706VJ:2:5:1000:12850 1:Y:18:ATCACG - FASTQ Sequence Start
• @EAS139:136:FC706VJ:2:5:1000:12850 1:Y:18:ATCACG - Instrument Name
• @EAS139:136:FC706VJ:2:5:1000:12850 1:Y:18:ATCACG - Run ID
• @EAS139:136:FC706VJ:2:5:1000:12850 1:Y:18:ATCACG - Flowcell ID
• @EAS139:136:FC706VJ:2:5:1000:12850 1:Y:18:ATCACG - Lane Number
• @EAS139:136:FC706VJ:2:5:1000:12850 1:Y:18:ATCACG - Tile Number
• @EAS139:136:FC706VJ:2:5:1000:12850 1:Y:18:ATCACG - X-coordinate of cluster
• @EAS139:136:FC706VJ:2:5:1000:12850 1:Y:18:ATCACG - Y-coordinate of cluster
• @EAS139:136:FC706VJ:2:5:1000:12850 1:Y:18:ATCACG - read Number (Paired 1/2)
• @EAS139:136:FC706VJ:2:5:1000:12850 1:Y:18:ATCACG - Filtered
• @EAS139:136:FC706VJ:2:5:1000:12850 1:Y:18:ATCACG – Control Number
• @EAS139:136:FC706VJ:2:5:1000:12850 1:Y:18:ATCACG – Index Sequence
• Filtered by Illumina Real Time Analysis (RTA) software during the run
– Y = is filtered (did not pass)
– N = is not filtered (did pass)
• Control number
– 0 if sequence NOT identified as control (PhiX)
– > 0 is a control sequence
• However, can be just: @SRR1553467.1 1 8

9. Illumina Flowcell
Surface of flow cell coated with a dense lawn of oligos
8 Lanes/Channels
2 Columns Per Lane
50 tiles per column, 100 tiles per lane (GAII)
Each column contains 50 tiles
Each tile is imaged 4 times per cycle – 1 image
per base {ACGT}
For 70bp run: 28,000 images per lane, 224,000
images per flowcell

10. FASTQ Quality Scores
• Each nucleotide in a sequence has an associated quality score
@EAS139:136:FC706VJ:2:5:1000:12850 1:Y:18:ATCACG
ACGTGTACACAGTCAGATGATAGCAGATAGGAAAT
+
BBBBCCCC?<A?BC?7@@???????DBBA@@@@A@
• FASTQ quality scores are ASCII characters
– ASCII - American Standard Code for Information Interchange
– letters {A-Z, a-z}, numbers {0-9}, special characters {@ £ # $ % ^ & * ( ) ,
. ? “ / | ~ ` { } }
– Each character has a numeric whole number associated with it
• When people speak of quality scores, it is typically in the region
of 0 to 40
– For example, trimming the ends of reads below Q25
• To translate from ASCII characters to Phred scale quality scores
– YOU SUBTRACT 33 (-33)
• But … what does a quality score actually mean???
10
Q-Symbol Q-ASCII Q-Score
I 73 40
H 72 39
G 71 38
F 70 37
E 69 36
D 68 35
C 67 34
B 66 33
A 65 32
@ 64 31
? 63 30
> 62 29
= 61 28
< 60 27
; 59 26
: 58 25
9 57 24
8 56 23
7 55 22
6 54 21
5 53 20
4 52 19
3 51 18
2 50 17
1 49 16
0 48 15
/ 47 14
. 46 13
- 45 12
, 44 11
+ 43 10
* 42 9
) 41 8
( 40 7
' 39 6
& 38 5
% 37 4
$ 36 3
# 35 2
" 34 1
! 33 0

11. Q = Probability of Error
Q-Symbol Q-ASCII Q-Score P-Error
I 73 40 0.00010
H 72 39 0.00013
G 71 38 0.00016
F 70 37 0.00020
E 69 36 0.00025
D 68 35 0.00032
C 67 34 0.00040
B 66 33 0.00050
A 65 32 0.00063
@ 64 31 0.00079
? 63 30 0.00100
> 62 29 0.00126
= 61 28 0.00158
< 60 27 0.00200
; 59 26 0.00251
: 58 25 0.00316
9 57 24 0.00398
8 56 23 0.00501
7 55 22 0.00631
6 54 21 0.00794
5 53 20 0.01000
4 52 19 0.01259
3 51 18 0.01585
2 50 17 0.01995
1 49 16 0.02512
0 48 15 0.03162
/ 47 14 0.03981
. 46 13 0.05012
- 45 12 0.06310
, 44 11 0.07943
+ 43 10 0.10000
* 42 9 0.12589
) 41 8 0.15849
( 40 7 0.19953
' 39 6 0.25119
& 38 5 0.31623
% 37 4 0.39811
$ 36 3 0.50119
# 35 2 0.63096
" 34 1 0.79433
! 33 0 1.00000
Q40 is not the maximum quality score
• A Phred quality score is a measure of the quality of the
identification of the nucleotide generated by automated DNA
sequencing.
• The quality score is associated to a probability of error – the
probability of an incorrect base call
• The calculation takes into account the ambiguity of the signal
for the respective base as well as the quality of neighboring
bases and the quality of the entire read.
• P = 10(-Q/10)
• Q40 = 1 in 10,000 error rate
• Q = - 10 x log10(P)
• If you have high coverage (~30,000) you expect errors even if
everything was the best quality i.e you would expect 3 errors on
average.

12. Read Quality Scores
• Quality scores tend to decrease along the read
• The read represents the sequence of a Cluster
– Cluster is comprised of ~1,000 DNA molecules
• As the sequencing progresses more and more of the DNA
molecules in the cluster get out of sync
– Phasing: falling behind: missing and incorporation cycle,
incomplete removal of the 3’ terminators/fluorophores
– Pre-phasing: jumping ahead: incorporation of multiple bases in a
cycle due to NTs without effective 3’ blocking
• Proportion of sequences in each cluster which are affected
by Phasing & Pre-Phasing increases with cycle number –
hampering correct base identification.
Q-Score P-error
40 0.0001
39 0.000125893
38 0.000158489
37 0.000199526
36 0.000251189
35 0.000316228
34 0.000398107
33 0.000501187
32 0.000630957
31 0.000794328
30 0.001
Trim low quality
ends off reads
Delete poor quality
reads overall
FASTQC
FASTX
ConDeTri

13. Illumina Quality Scores
• Quality scores tend to decrease along the read
• The read represents the sequence of a Cluster
– Cluster is comprised of ~1,000 DNA molecules
• As the sequencing progresses more and more of the DNA
molecules in the cluster get out of sync
– Phasing: falling behind: missing and incorporation cycle,
incomplete removal of the 3’ terminators/fluorophores
– Pre-phasing: jumping ahead: incorporation of multiple bases in a
cycle due to NTs without effective 3’ blocking
• Proportion of sequences in each cluster which are affected
by Phasing & Pre-Phasing increases with cycle number –
hampering correct base identification.
– A/C and G/ T fluorophore spectra overlap
– T accumulation – in certain chemistries – due to a lower removal
rate of T fluorophores

14. Viral Reference Genome
Sequencing
HTS Reads
Genome
Mapping
Aligning Reads to a Reference
Viral
DNA

15. Short Read Alignment
• Find which bit of the genome the short read has come from
– Given a reference genome and a short read, find the part of the
reference genome that is identical (or most similar) to the read
• It is basically a String matching problem
– BLAST – use hash tables
– MUMmer – use suffix trees
– Bowtie – use Burrows-Wheeler Transform
• There are now many alignment tools for short reads:
– BWA, Bowtie, MAQ, Novoalign, Stampy … … … … …
– Lots of options such as maximum no of mismatches between read/ref,
gap penalties etc
• Aligning millions of reads can be time consuming process.
• So you want to save the alignments for each read to a file:
– So the process doesn’t need to be repeated
– So you can view the results
– In a generic format that other tools can utilise (variant calling etc)

16. SAM Header
• The first section of the SAM file is called the header.
• All header lines start with an @
• HD – The Header Line – 1st line
– SO – Sorting order of alignments (unknown (default), unsorted, queryname and coordinate)
• SQ - Reference sequence dictionary.
– SN - Reference sequence name.
– LN - Reference sequence length.
• PG – Program
– ID – The program ID
– PN – The program name
– VN – Program version number
– CL – The Command actually used to create the SAM file
• RG – Read Group
– "a set of reads that were all the product of a single sequencing run on one lane"
16
@HDVN:1.0 SO:coordinate
@SQ SN:KM034562.G3686.1 LN:18957
@PG ID:bowtie2PN:bowtie2 VN:2.2.4 CL:"/usr/bin/../lib/bowtie2/bin/bowtie2-align-s --wrapper basic-
0 --local -x ../ref/ebov -S ebov1.sam -1 ebov1_1.fastq -2 ebov1_2.fastq"

17. SAM File Format
QNAME FLAG RNAME POS MAPQ CIGAR RNEXT PNEXT TLEN SEQ QUALITY
Read1 10 FMDVgenome 3537 57 70M CCAGTACGTA >AAA=>?AA>
• SAM (Sequence Alignment/Map format) data files are outputted from aligners that read FASTQ files and
assign sequences/reads to a position with respect to a known reference genome.
– Readable Text format – tab delimited
– Each line contains alignment information for a read to the reference
• Each line contains:
– QNAME: Read Name
– FLAG: Info on if the read is mapped, part of a pair, strand etc
– RNAME: Reference Sequence Name that the read aligns to
– POS: Leftmost position of where this alignment maps to the reference
– MAPQ: Mapping quality of read to reference (phred scale P that mapping is wrong)
– CIGAR: Compact Idiosyncratic Gapped Alignment Report: 50M, 30M1I29M
– RNEXT: Paired Mate Read Name
– PNEXT: Paired Mate Position
– TLEN: Template length/Insert Size (difference in outer co-ordinates of paired reads)
– SEQ: The actual read DNA sequence
– QUAL: ASCII Phred quality scores (+33)
– TAGS: Optional data – Lots of options e.g. MD=String for mismatches

18. SAM FLAGS
• Example Flag = 99
• 99 < 128, 256, 512, 1024, 2048 … so all these flags don’t apply
• 99 > 64 … {99 – 64 = 35} … This read is the first in a pair
• 35 > 32 … {35 – 32 = 3} … The paired mate read is on the reverse strand
• 3 < 16 … {3} … This read is not on the reverse strand
• 3 < 8 … {3} … The paired mate read is not unmapped
• 3 > 2 … {3 – 2 = 1} … The read is mapped in proper pair
• 1 = 1 … {1 – 1 = 0} … The read is paired 18
# Flag Description
1 1 Read paired
2 2 Read mapped in proper pair
3 4 Read unmapped
4 8 Mate unmapped
5 16 Read reverse strand
6 32 Mate reverse strand
7 64 First in pair
8 128 Second in pair
9 256 Not primary alignment
10 512 Read fails platform/vendor quality checks
11 1024 Read is PCR or optical duplicate
12 2048 Supplementary alignment
8 > (4+2+1) = 7
16 > (8+4+2+1) = 15
32 > (16+8+4+2+1) = 31
etc

19. 19
• H : Hard Clipping: the clipped nucleotides have been removed from the read
• S : Soft Clipping: the clipped nucleotides are still present in the read
• M : Match: the nucleotides align to the reference – BUT the nucleotide can be an alignment match or mismatch.
• X : Mismatch: the nucleotide aligns to the reference but is a mismatch
• = : Match: the nucleotide aligns to the reference and is a match to it
• I : Insertion: the nucleotide(s) is present in the read but not in the reference
• D : Deletion; the nucleotide(s) is present in the reference but not in the read
• N : Skipped region in reference (for mRNA-to-genome alignment, an N operation represents an intron)
• P: Padding (silent deletion from the padded reference sequence) – not a true deletion
CIGAR String
• 8M2I4M1D3M
• 8Matches
• 2 insertions
• 4 Matches
• 1 Deletion
• 3 Matches

20. BAM File
• SAM (Sequence Alignment/Map) files are typically converted to
Binary Alignment/Map (BAM) format
• They are binary
– Can NOT be opened like a text file
– Compressed = Smaller (great for server storage)
– Sorted and indexed = Daster
• Once you have a file in a BAM format you can delete your aligned
read files
– You can recover the FASTQ reads from the BAM format
– Delete the aligned reads – trimmed/filtered reads
– Do not delete the RAW reads – zip them and save them
20

21. Variant Call Format
• Once reads are aligned to a reference, you may want to call
mutations and indels to the reference
• Many tools output this data in the VCF (Variant Call Format)
• Text file – tab delimited
21

22. Variant Call Format
22
CHROM: Chromosome Name (i.e. viral genome name)
POS: Position the viral genome
ID: Name of the known variant from DB
REF: The reference allele (i.e. the reference base)
ALT: The alternate allele (i.e. the variant/mutation observed)
QUAL: The quality of the variant on a Phred scale
FILTER: Did the variant pass the tools filters
INFO: Information
DP = Depth
AF = Alternate Allele Frequency
SB = Strand Bias P-value
DP4 = Extra Depth Information
(forward ref; reverse ref; forward non-ref; reverse non-ref)

23. Acknowledgements
• Viral Genomics and
Bioinformatics group
– Centre for Virus Research
– University of Glasgow
23