Published on

Introduction to NGS data, processing and mapping to detect polymorphisms.

Published in: Science, Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1.
  2. 2. Obtaining, QC, mapping and analysis of NGS data. Richard Emes Associate Professor & Reader in Bioinformatics. School of Veterinary Medicine and Science Director Advanced Data Analysis Centre @rdemes @ADAC_UoN 2
  3. 3. What is ADAC? The University of Nottingham Advanced Data Analysis Centre (ADAC). •  Bioinformatics and data analysis support. Why is this important? •  Complex data underpins much current research. •  Innovative analysis can prompt new discoveries. •  Excellent research can often be stalled due to a lack of expertise in conducting data analysis, availability or cost of inclusion of diverse specialists. Why ADAC? •  ADAC supports high-class research by providing analysts with expertise in a range of bioinformatics and computer science disciplines. •  Flexible support •  Consultancy, collaboration, bespoke analysis. •  Leadership from recognized experts in the fields of bioinformatics and computer science. •  Track record of funding •  Pivotal role in collaborations funded by amongst others Zoetis, BBSRC, NERC and Technology Strategy Board. •  ADAC is conducting transcriptome analysis for a multinational FP7 funded project (EU Prohealth).
  4. 4. Enquiries: or @ADAC_UoN or @rdemes Current Areas of expertise relevant here: •  Transcriptomics (Microarray, NGS) •  Comparative genomics (eukaryotic, prokaryotic) •  Identification of biomarkers from genetic and epigenetic datasets •  Artificial intelligence for decision support •  Machine Learning •  Integration of complex datasets •  Data Management
  5. 5. @HWI-_FC_20BTNAAXX:2:1:215:593 ACAGTGCATGACATGCATAGCAGCATAGACTAC +HWI-_FC_20BTNAAXX:2:1:215:593 GhhhhhhhhhhhUhhEGhhhGhhhhhhhhhhhhh
  6. 6. Some common terms •  Library: collection of molecules. This is the “complexity” of what you sequence. •  Flowcell: slide where sequencing is attached to a solid platform. •  Lane: unique sequencing unit of the flowcell. •  Reads: Raw sequence of bases and imputed quality scores. •  Fragment: Original molecule being sequenced (fragment of genome/gene). Ie PE are reads form the same fragment. •  Cluster: DNA bound to slide, local amplification of product to amplify signal to measure fluorescence. •  Mapping: Finds where your sequence matches to a reference. Importantly gives a probability that this is the correct location.
  7. 7. Illumina sequencing
  8. 8. Illumina sequencing
  9. 9. What coverage do I need?
  10. 10. Obtaining NGS Data •  Short Read Archive (SRA) •  European Nucleotide Archive (ENA)
  11. 11. Obtaining NGS Data •  Short Read Archive (SRA) •  Will need to convert to FastQ using sra toolkit
  12. 12. Deciphering a fastq file @HWI-_FC_20BTNAAXX:2:1:215:593#0/1 ACAGTGCATGACATGCATAGCAGCATAGACTAC +HWI-_FC_20BTNAAXX:2:1:215:593#0/1 GhhhhhhhhhhhUhhEGhhhGhhhhhhhhhhhhh Header: @HWI-_FC_20BTNAAXX:2:1:215:593#0/1 HWI-_FC_20BTNAAXX instrument identifier 2 flowcell lane 1 tile number in flowcell lane 215 x - coordinate of cluster in the tile 593 y - coordinate of cluster in the tile #0 index of multiplexed samples /1 member of pair /1 or /2 if Paired end
  13. 13. Deciphering a fastq file @HWI-_FC_20BTNAAXX:2:1:215:593#0/1 ACAGTGCATGACATGCATAGCAGCATAGACTAC +HWI-_FC_20BTNAAXX:2:1:215:593#0/1 GhhhhhhhhhhhUhhEGhhhGhhhhhhhhhhhhh Sequence: ACAGTGCATGACATGCATAGCAGCATAGACTAC Quality Header: +HWI-_FC_20BTNAAXX:2:1:215:593 Quality: GhhhhhhhhhhhUhhEGhhhGhhhhhhhhhhhhh
  14. 14. Deciphering a fastq file Quality: GhhhhhhhhhhhUhhEGhhhGhhhhhhhhhhhhh Sanger encoding = ASCII table lookup – 33 Solexa encoding = ASCII table lookup – 64 G = 71 – 64 = 7 h = 104- 64 = 40
  15. 15. Deciphering a fastq file Quality: GhhhhhhhhhhhUhhEGhhhGhhhhhhhhhhhhh
  16. 16. SNP Calling •  Genotyping: identifying variants in a single genome (i.e. from each parent) •  SNP Calling: identifying variants between individual genomes ACGTGCAGCATAGCA?CGACATCGACATACGC TGCACGTCGTATCGT?GCTGTAGCTGTATGCG ****A******* ***A***** **T****** *****T********** ******T******** ACGTGCAGCATAGCATCGACATCGACATACGC TGCACGTCGTATCGTAGCTGTAGCTGTATGCG Sample Genome(s) Reads Reference Genome
  17. 17. INDEL Calling •  INDEL: Insertion/deletion ACGTGCAGCATAGCA???CGACATCGACATACGC TGCACGTCGTATCGT???GCTGTAGCTGTATGCG ****ACG******* ***ACG***** **---******** *****---********** ******---******** ACGTGCAGCATAGCACGTCGACATCGACATACGC TGCACGTCGTATCGTGCAGCTGTAGCTGTATGCG Sample Genome(s) Reads Reference Genome
  18. 18. A pipeline for SNP identification •  Quality Control –  FastQC, Fastx toolkit •  Trimming: –  Sickle, Trimgalore, Trimomatic, Cutadapt •  Mapping –  BWA, Bowtie, Stampy •  Remove Duplicates –  Picard tools, Samtools rmdup •  Call SNPs / INDELS –  Samtoools mpileup, VarScan, GATK, Many others!
  19. 19. Galaxy: Toolbox Workflows
  20. 20. Visualize the data •  FASTQC •  Stand Alone or non-interactive –  Basic Statistics module, includes: •  Filename: The original filename of the file which was analyzed •  Encoding: Says which ASCII encoding of quality values was found in this file. •  Total Sequences: A count of the total number of sequences processed. •  Sequence Length: Provides the length of the shortest and longest sequence in the set. If all sequences are the same length only one value is reported.
  21. 21. Visualize the data •  FASTQC: Per Base Sequence Quality: Red line = Median quality Yellow box = IQR Whiskers = 10%-90% Blue line = Mean quality If the lower quartile for any base is less than 10, or if the median for any base is less than 25. If the lower quartile for any base is less than 5 or if the median for any base is less than 20.
  22. 22. Visualize the data •  FASTQC: Per Sequence Quality Scores: If the most frequently observed mean quality is below 27 - this equates to a 0.2% error rate. If the most frequently observed mean quality is below 20 - this equates to a 1% error rate.
  23. 23. Visualize the data •  FASTQC: Per Base Sequence Content Proportion of each base position in a file for which ATCG DNA bases has been called. If the difference between A and T, or G and C is greater than 10% in any position. If the difference between A and T, or G and C is greater than 20% in any position. Possibly adapters or affect of trimming.
  24. 24. Visualize the data •  FASTQC: Sequence Length Distribution Distribution of fragment sizes in the file. If all sequences are not the same length. If any of the sequences have zero length.
  25. 25. Visualize the data •  FASTQC: Duplicate Sequences Degree of duplication within first 200,000 reads of file. Distribution of duplication levels in dataset If non-unique sequences make up more than 20% of the total. If non-unique sequences make up more than 50% of the total.
  26. 26. Visualize the data •  FASTQC: Overrepresented Sequences •  FASTQC: Adapter Content Lists all of the sequences which make up more than 0.1% of the total. If any sequence is found to represent more than 0.1% of the total. If any sequence is found to represent more than 1% of the total. To know if your library contains a significant amount of adapter in order to be able to assess whether you need to adapter trim or not. If any sequence is present in more than 5% of all reads. If any sequence is present in more than 10% of all reads.
  27. 27. Cut adapters -f = the type of file (in this case fastq) -q CUTOFF, Trim low-quality ends from reads before adapter removal. -a ADAPTER, Sequence of an adapter that was ligated to the 3' end. The adapter itself and anything that follows is trimmed. -m 100 minimum length of reads following adapter removal. Reads less than 100 will be discarded --discard-untrimmed any reads without an adapter will be discarded. -o output file also in fastq format. cutadapt -f fastq -q 20 -a AGATCGGAAGAG -m 100 --discard-untrimmed -o SNP.test.trimmed.fastq SNP.test.fastq
  28. 28. Cut adapters Galaxy: Fastx_clipper from fastx_toolkit
  29. 29. Quality filters
  30. 30. Quality filters Fastx_toolkit -q = Minimum quality score to keep. -p = Minimum percent of bases in a read that must have [-q] quality. - i input file (output of adapter trimming step) -v verbose -Q quality encoding -o output file also in fastq format. fastq_quality_filter -q 20 -p 70 -i SNP.test.trimmed.fastq -v –Q64 -o SNP.test.trimmed.QC.fastq
  31. 31. Quality filters •  Galaxy: filter by quality
  32. 32. Align to genome The problem •  Generally a large genome (Human > 3Gb) •  Large number of short reads The solution •  Index genome into hash of kmers or short sequences •  Use efficient aligners –  Large number of aligners available. •  Common aligners: Bowtie1/2, BWA, Stampy
  33. 33. Align to genome Example Bowtie 2 alignment Build index -f fasta formatted genome file ./bowtie.index.files/chr17 output location for index files Galaxy: select pre-built index when using bowtie or BWA bowtie2-build -f ./genome/chr17.fa ./bowtie.index.files/chr17
  34. 34. Align to genome Align to reference -p number of processors --end-to-end alignment is not local -k 1 number of positions read is allowed to align k = 1 means all non- uniquely mapping reads are discarded -x path to indexed genome file to align reads to. -U reads are unpaired (in this case -S output in SAM format bowtie2 -p 4 --end-to-end -k 1 -x ./bowtie.index.files/chr17 -U SNP.test.trimmed.QC.fastq -S SNP.test.trimmed.QC.fastq.sam
  35. 35. Alignment file formats SAM - Sequence Alignment/Map format. (BAM is a binary compressed equivalent) –  TAB-delimited text format –  header section (optional) @HD = Header, VN[format version], SO[sorting order] @SQ = Reference sequence line @PG = Program, ID[program ID], CL[command line] @HD VN:1.0 SO:unsorted @SQ SN:chr17 LN:81195210 @PG ID:bowtie2 PN:bowtie2 VN:2.2.1 CL:"/ home/rde/tools/bowtie2-2.2.1/bowtie2-align-s --wrapper basic-0 -p 4 --end-to-end -k 1 -x ./bowtie.index.files/ chr17 -S SNP.test.trimmed.QC.fastq.sam -U SNP.test.trimmed.QC.fastq”
  36. 36. Alignment file formats •  SAM - Sequence Alignment/Map format. –  TAB-delimited text format –  alignment section mandatory 11 columns –  Optional fields –  For details –  Samflags instrument_name:100:flowcellID:1:1:32575:625 0 chr17 72387574 255 141M * 0 0 GGAAGAGCTGGGACCAGGCCCAGCAATTACCTCACCATGGTTGGGTGCAACCAAG TGGGGCAACTCTTTGGCCAGAAAGCAAAAGTCTTTTTAGCTTCAATGTAGGCCAT TCTGGGTCCCAGACCCACAGCTTTGGACATT Dgga^h`hedac`hcc`c^_f`ChggeDfC`e`_h_ddgd^_`be^bcg^acchC `^^bCdagfbgf_`^dc^^cD`dbha^Dbc`fb^d`CdaD^``dDb_edffdDDh _chceabh`heecd_h`gb_b^Ch`Dd_^c` AS:i:0 XN:i: 0 XM:i:0 XO:i:0 XG:i:0 NM:i:0 MD:Z: 141 YT:Z:UU 1  Query template 2  Bitwise flag 3  Reference Name 4  1-based leftmost position 5  Mapping Quality 6  CIGAR string 7  Reference Name of mate 8  Position of mate 9  Observed template length 10  Sequence 11  Phred-scaled Quality
  37. 37. Remove duplicates •  Duplicate reads generated as artifact in the library generation step results in false confidence in variants. •  samtools [rmdup], Picard tools [MarkDuplicates] samtools view -bS SNP.test.trimmed.QC.fastq.sam -o SNP.test.trimmed.QC.fastq.bam samtools rmdup -s SNP.test.trimmed.QC.fastq.bam SNP.test.trimmed.QC.fastq.rmdup.bam
  38. 38. Remove duplicates
  39. 39. Call Variants •  Identify regions in alignment where sequence differs ****A******* ***A***** **T****** *****T********** ******T******** ACGTGCAGCATAGCATCGACATCGACATACGC Reads Reference Genome
  40. 40. Call Variants samtools sort SNP.test.trimmed.QC.fastq.rmdup.bam SNP.test.trimmed.QC.fastq.rmdup.sorted samtools index SNP.test.trimmed.QC.fastq.rmdup.sorted.bam samtools faidx ./genome/chr17.fa samtools mpileup -f ./genome/chr17.fa SNP.test.trimmed.QC.fastq.rmdup.sorted.bam | java -jar ./tool/VarScan.v2.3.6.jar mpileup2snp --output-vcf – strand-filter 0 samtools mpileup -f faidx indexed reference sequence file VarScan mpileup2snp or mpileup2indel --min-coverage Minimum read depth at a position to make a call [8] --min-reads2 Minimum supporting reads at a position to call variants [2] --min-avg-qual Minimum base quality at a position to count a read [15] --min-var-freq Minimum variant allele frequency threshold [0.01] --strand-filter Ignore variants with >90% support on one strand [1]
  41. 41. Call Variants
  42. 42. Call Variants
  43. 43. VCF Variant Call Format file •  Header text marked with ## •  Column headings marked with # •  Mandatory columns –  CHROM Chromosome –  POS Position of variant start –  ID Unique variant ID –  REF Reference Allele –  ALT Alternate non-reference alleles (comma separated) –  QUAL Phred quality score –  FILTER Filtering information –  INFO User annotation
  44. 44. Visualize: IGV
  45. 45. So Many SNPS – So What? •  Get gene •  Functional Analysis to identify key candidates Identify Homologues Locate variants Identify Ontologies Pathway and interaction analysis Locate SNPs on structure Compare to current data
  46. 46. •  Functional Analysis to identify key candidates A step by step example (don’t do this with lots of variants!) •  Get gene •  Modify bases as shown in VCF file. •  BLASTx to identify reading frame. •  Produce mRNA, and encoded peptide sequence fasta files (provided) •  Determine variant positions in mRNA & peptide sequences
  47. 47. Compare to current data •  dbSNP –  SNP already known? •  Repositories such as Ensembl, UCSC –  In splice variant? –  In known regions, domain etc? •  Variant effect predictor (more later)
  48. 48. •  Visualize the position in genome – Ensembl/UCSC –  Add custom track using GFF/bed file •  Coding/non-coding –  Synonymous / non-synonymous –  Codon usage •  Locate in relation to known domains –  Pfam –  SMART •  Repeat regions Locate variants
  49. 49. •  For single genes –  Search for available information PubMed, interpro etc •  For multiple genes –  BLAST2GO –  DAVID Identify Ontologies
  50. 50. •  Pathway analysis –  Understand the process of your gene. Does it make biological sense? –  IPA –  DAVID –  Webgestalt •  Interaction analysis –  BioGRID –  STRING –  PSICQUIC Pathway and interaction analysis
  51. 51. •  Structure prediction –  BLAST of PDB –  Predict structure •  PSIPRED – 2° structure •  ITASSER – 3° structure •  Phyre – 3° structure –  Locate in 3D •  Swiss PDB viewer Locate SNPs on structure
  52. 52. •  Predict effect of SNPs •  Suspect •  VEP Locate SNPs on structure Arg Ser
  53. 53. Identify Homologues Emes R.D. Inferring function from homology. in Methods in Molecular Biology 453: Humana Press 2008.
  54. 54. Links and websites 1 •  Many at •  SRA: •  SRA toolkit •  ENA: •  Galaxy: •  FASTQC: •  Fastx_toolkit: •  Bowtie 1: •  Bowtie 2: •  BWA: •  Stampy: •  SAMtools: •  PicardTools: •  VarScan: •  IGV:
  55. 55. Links and websites 2 •  dbSNP: •  Ensembl: •  UCSC Genome Browser: •  Pfam: •  SMART: •  GO: •  BLAST2GO: •  IPA: •  DAVID: •  Webgestalt: •  BioGRID: •  PSICQUIC: •  ITASSER: •  Phyre: •  SuSPect: •  PSIPRED: •  SwissPDB:
  56. 56. Richard Emes Associate Professor & Reader in Bioinformatics. School of Veterinary Medicine and Science Director Advanced Data Analysis Centre @rdemes @ADAC_UoN 61