Neural Networks on Bioinformatics

Applications of Artificial Neural Networks in Bioinformatics Armando Vieira ISEP and Centro de Física Computacional, Coimbra, Portugal www.defi.isep.ipp.pt/~asv [email_address]

XXI: the century of BIO

Reverse Engineering

BIOINFORMATICS: bringing together two worlds apart

Outline
Review of Neural Networks and its specificity in Bioinformatics:
Data pre-processing; Training (care and tricks); Interpretation of results.
Applications :
Protein Secondary Structure
Loop Classification
Selenoproteins discovery
Gene Expression Microarrays

Input builds up on receptors (dendrites)
Cell has an input threshold
Upon breech of cell’s threshold, activation is fired down the axon.
Modeling the Human Brain?

Artificial Neural Networks are essentially maps
Supervised: {input}  {output}
Unsupersived: cluster the inputs
Universal computation machines but much more that simple look up tables

ANN can generalize. … But not that much!

Artificial Neural Networks
A problem-solving paradigm modeled after the physiological functioning of the human brain.
Synapses in the brain are modeled by computational nodes.
The firing of a synapse is modeled by input, output, and threshold functions.
The network “learns” based on problems to which answers are known ( supervised learning ).
The network can then produce answers to entirely new problems of the same type.

Artificial Neuron W i,j a j Input Links Input Function Trigger Function Output a i = g(in i )

MultiLayer Perceptron Output Layer Input Layer Hidden Layers

ANN in Bioinformatics
Protein Secondary Structure prediction
Gene identification / prediction
Protein-Protein / Gene-Gene Interaction.
Pathway discovery
Microarray analysis
DNA binding site recognition

Applications
Time series prediction (weather forecasting, adaptative filters, stock market)
Processes / machine control
Bankrutpcy / fraud prediction
Checkers players – reach the level of master
Topological maps: human brain, meterological profiles

ANN: Advanced Learning Machines
ANN can learn higher order correlations.
What does this mean?
0 0 => 0 0 1 => 1 1 0 => 1 1 1 => 0 No linear function can learn this pattern XOR Problem

Neural Networks are Advanced Learning Machines
They can learn higher order correlations.
What does this mean?
Money Age FRUSTATION

Learning higher order correlation 0 0 => 0; 1 0 => 1 1 1 => 0; 0 1 => 1 Has no solution! X 1 W 1 W 2 X 2 0 W 11 X 1 W 22 X 2 0 W 21 W 12 V 2 V 1 h 1 h s Solution

Pre-processing and data cleaning
Normalization of inputs and outputs [0.1,.9] for sigmoid or [-.9, +.9] for tanh.
Reduce dimensionality of inputs through feature selection (correlations, PCA, sensitivity). DO IT!
Remove outliers
Use adequate transformations (logarithm, inverse,…)
In classification always use balanced datasets

How to build a balance dataset
Suppose we have a training set composed of:
100 examples class x
1000 examples class y
Oversampling: replicate 10-fold class x, by repetition of interpolation.
Undersampling: compress 10-fold class y using a cluster algorithm like k-means.

How to train ANN
1)Choose a network topology: number layers and hidden nodes (as small as possible)
2) Separate training (90%) and test set (10%)
3) Minimize a cost function:
Sum of square errors
Cross entropy

Balancing Act in Neural Network Training
The network should NOT just memorize the training set.
The network should be able to generalize from the training set so that it can solve similar but not identical problems.
It’s a matter of balancing the # of training patterns vs. # network weights vs. # hidden nodes vs. # of training iterations

Overfitting in action Simpler hypothesis has lower error rate

Overfitting, an important risk! Use small networks and k-fold cross validation (k = 5 or 10)

ANN for classification
Two classes:
negatives ( y = 0)
positives ( y = 1)
True positives ratio = true positives / number positives in the sample (as high as possible)
False positives ratio = false positives / number negatives in the sample (as low as possible)
Average accuracy = number correctly predicted / total number

Evaluation of prediction accuracy A roc =0.5 A roc =0.8 Roc curves Pearson correlation ROC: Received Operational Characteristic TP FP TP TN

Programs Evaluation
Sensitivity =
Specificity =
number of correct exons number of actual exons number of correct exons number of predicted exons TN FP FN TN TN TP FN TP FN REALITY PREDICTION

Hidden Layer Vector Quantization- HLVQ Traditional NN x x x x x x x x o o o o o o o HLVQ Main advantage: detect and correct prediction for outliers z x x x x x x x x o o o o o o o

Output Inputs unit vectors

ANN are not a “silver bullet”!
Neural networks are good when: many training data available; continuous variables; relevant features are known; unicity of the mapping .
Neural networks are less useful when: problem is linear; few data compared to the size of search space; data high dimensional; long range correlations.
They are black boxes and cannot explain why a given pattern has been classified as x rather than y. They don’t tell us anything about underlying principles.

Disadvantages to Neural Networks
They are black boxes . They cannot explain why a given pattern has been classified as x rather than y. Unless we associate other methods with them, they don’t tell us anything about underlying principles.

Theory: how about applications?!

Applications I Protein Secondary Structure

One million putative protein but only 5% have its structure determined.
Secondary Structure is the first, and simplest step, towards full 3D. However, accuracy < 76%. Why?
Training with proteins of known secondary structure.
Then give it new proteins with unknown structure and determine their structure with the neural network.
Check if the prediction makes sense from a biological point of view – e.g., you need at least 4 amino acids in a row for an α -helix.

The Gist of it
Amino acid sequence Physical structure Function

What is a protein?
All proteins are chains of 20 amino acids
Not all chains of amino acids are proteins
Fold rapidly and repeatedly
Proteins are the machinery of live
Essential to all (known) organisms

From sequence to secondary structure to backbone angles (3º structure)
Residue Sequence
MEKKEFHIVAETGIHARPATLLVQTASLFNSDINLETLGKSVNLKSIMGVMSLGVGQGSDVTITVDGADEADGMAAIVETLQLQGLAQ...
3 ° Structure
( φ 1 , ψ 1 ), ( φ 2 , ψ 2 ), ( φ 3 , ψ 3 ), ( φ 4 , ψ 4 ), ( φ 5 , ψ 5 ), ( φ 6 , ψ 6 ), ( φ 7 , ψ 7 ), ( φ 8 , ψ 8 ), ( φ 9 , ψ 9 ), ( φ 10 , ψ 10 ), ( φ 11 , ψ 11 ), ( φ 12 , ψ 12 ), ( φ 13 , ψ 13 ), ( φ 14 , ψ 14 ), ( φ 15 , ψ 15 ),...

Ramachandran Alphabet φ ψ -180 ° 180° -180° 180° 90° -90° 0° 0° -90° 90° A B E G

5-letter alphabet
Residue Sequence
MEKKEFHIVAETGIHARPATLLVQTASLFNSDINLETLGKSVNLKSIMGVMSLGVGQGSDVTITVDGADEADGMAAIVETLQLQGLAQ...
3 ° Structure
ACCDECBAABDECBDABCDBEABDBCBDBAEBDBDBAEBABDCBBDBADDCBDBCBDBEBDBCBBDCAABDEDCDCEAABACAAAADC…

PROBLEM VERY HARD! Start by predicting only the secondary structure Amino acid sequence (Helix, Sheet, Coil)

Approaches
Ab initio :
Quantum Mechanics + fast computers + large # configurations
= huge problems…
Machine Learning :
Use known cases to learn a suitable map (black box approach)

How a Neural Network is trained Training pattern One of n inputs, each with 21 bits

Inputs to the Network
Both the residues and target classes are encoded in unary format , for example
Input:
Alanine: {1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0}
Cysteine: {0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0} ...
Output:
Helix = {1, 0, 0}, Sheet {0, 1, 0}, Coil { 0, 0, 1}
Each pattern presented to the network requires n 21-bit inputs for a window of size n. (One bit is required per residue to indicate when the window overlaps the end of the chain).
The advantage of this sparse encoding scheme is that it does not pay attention to ordering of the amino acids
Disadvantage is that it requires a lot of input.

How to train
Input values at each layer are multiplied by weights.
Weights are adjusted after the output is computed based on how close the output is to the “right” answer.
When the full training session is completed, the weights have settled on certain values.
These weights are then used to compute output for new problems that weren’t part of the training set.

Training Set
A typical training set contains over 100 non-homologous protein chains comprising more than 15,000 training patterns.
The number of training patterns is equal to the total number of residues in the 100 proteins.
For example, if there are 100 proteins and 150 residues per protein there would be 15,000 training patterns.

Neural Network Architecture
A typical architecture has a window-size of n and 5 hidden layer nodes.
Then a fully-connected would be 17(21)-5-3 network, i.e. a net with an input window of 17, five hidden nodes in a single hidden layer and three outputs.
Such a network has 357 input nodes and 1,808 weights: ((17 * 21) * 5) + (5 * 3) + 5 + 3 = 1808?
*This information is adapted from “Protein Secondary Structure Prediction with Neural Networks: A Tutorial” by Adrian Shepherd (UCL),
http://www.biochem.ucl.ac.uk/~shepherd/sspred_tutorial/ss-index.html .)

Window
The n-residue window is moved across the protein, one residue at a time.
Each time the window is moved, the center residue becomes the focus.
The neural network “learns” what secondary structure that residue is a part of. It keeps adjusting weights until it gets the right answer within a certain tolerance. Then the window is moved to the right.

Weight matrices from multiple alignments (BLAST)
Estimate amino acid frequencies from alignment. A weight matrix is given by
W ij = log(p ij /q j )
Here i is a position in the motif, and j an amino acid. q j is the background frequency for amino acid j.
In nature not all amino acids are found equally often
P A = 0.07, P W = 0.013
Finding 6% A is hence not significant, but 6% W highly significant
W is a L x 20 matrix, L is motif length

Scoring sequences to a weight matrix A R N D C Q E G H I L K M F P S T W Y V 1 0.6 0.4 -3.5 -2.4 -0.4 -1.9 -2.7 0.3 -1.1 1.0 0.3 0.0 1.4 1.2 -2.7 1.4 -1.2 -2.0 1.1 0.7 2 -1.6 -6.6 -6.5 -5.4 -2.5 -4.0 -4.7 -3.7 -6.3 1.0 5.1 -3.7 3.1 -4.2 -4.3 -4.2 -0.2 -5.9 -3.8 0.4 3 0.2 -1.3 0.1 1.5 0.0 -1.8 -3.3 0.4 0.5 -1.0 0.3 -2.5 1.2 1.0 -0.1 -0.3 -0.5 3.4 1.6 0.0 4 -0.1 -0.1 -2.0 2.0 -1.6 0.5 0.8 2.0 -3.3 0.1 -1.7 -1.0 -2.2 -1.6 1.7 -0.6 -0.2 1.3 -6.8 -0.7 5 -1.6 -0.1 0.1 -2.2 -1.2 0.4 -0.5 1.9 1.2 -2.2 -0.5 -1.3 -2.2 1.7 1.2 -2.5 -0.1 1.7 1.5 1.0 6 -0.7 -1.4 -1.0 -2.3 1.1 -1.3 -1.4 -0.2 -1.0 1.8 0.8 -1.9 0.2 1.0 -0.4 -0.6 0.4 -0.5 -0.0 2.1 7 1.1 -3.8 -0.2 -1.3 1.3 -0.3 -1.3 -1.4 2.1 0.6 0.7 -5.0 1.1 0.9 1.3 -0.5 -0.9 2.9 -0.4 0.5 8 -2.2 1.0 -0.8 -2.9 -1.4 0.4 0.1 -0.4 0.2 -0.0 1.1 -0.5 -0.5 0.7 -0.3 0.8 0.8 -0.7 1.3 -1.1 9 -0.2 -3.5 -6.1 -4.5 0.7 -0.8 -2.5 -4.0 -2.6 0.9 2.8 -3.0 -1.8 -1.4 -6.2 -1.9 -1.6 -4.9 -1.6 4.5 PSSM

Predictions Based on Output
Predictions are made on a winner-takes-all basis.
That is, the prediction is determined by the strongest of the three outputs. For example, the output (0.3, 0.1, 0.1) is interpreted as a helix prediction.

Performance Measurements
How do you know if your neural network performs well?
Test it on proteins that are not included in the training set but whose structure is known.
Determine how often it gets the right answer.
What differentiates one neural network from another?
Its architecture – whether or not it has hidden layers, how many nodes are used.
Its mathematical functions – the trigger function, the back-propagation algorithm.

PSI-PRED uses PSSM matrix.
Average accuracy reach up to 76%, but largely dependent on similarities found by BLAST.
In some proteins PSI-PRED have very poor accuracies (less than 35% ~ almost random). Why?

Application II Protein Loops Classification

Loops everywhere!

How to look for a loop…

Geometry of the Motif

Loop Types  :  strand -  -helix  :  -helix -  -helix  :  -helix –  strand  -hairpin :  strand -  strand  - link :  strand -  strand

 Similar conformation aa{ b }aa / aa{p}aa Identical geometry (4,6)(0,45)(45,90)(180,225) © Baldomero Oliva 1.3.1 aa{p}aa 1.1.2 aa{b}aa Pro 75% Ser 75%

Class 

ArchDB database
~ 20 000 loops classified into ~ 3000 classes . EE-3.4.1
Loop type - loop size . consensus . motif
TASK: classify a loop from sequence alone
If not possible, get as much information as possible

Problems
Coding of amino acids
Huge searching space, sparsely populated
How to assign the loop classes?
High dimensionality -> Lar ge Networks -> poor generalization

Results on loop class prediction
I – loop type
1) Sequence alone
2) Sequence + geometry motif
II – loop class
1) Sequence alone
2) Sequence + geometry

 -  link and  -  hairpins from sequence Total error 12.4% 1260 (1208) 180 (232) Real  -  harpin 104 (185) 795 (725) Real  -  link Predicted  -  harpin Predicted  -  link HLVQ (MLP)

Prediction of all loop types from sequence alone 374 150 51 143 16 α - α 165 1000 138 236 36  - α 43 127 1787 111 22  -  hp 96 183 103 507 40 α -  32 132 105 105 195  -  lk α - α  - α  -  hp α -   -  lk

Prediction of loop type 536 186 12 144 37 α - α 149 1348 41 125 76  - α 12 46 2152 21 9  -  hp 47 183 12 684 89 α -  15 139 26 200 322  -  lk α - α  - α  -  hp α -   -  lk

Ramachandran alphabet from sequence + geometry  -  harpins Work in progress Goal: accuracy > 70% E G B A E G B A

What’s it all mean?
Given a loop residue sequence, we can (usually) identify its native structure.
Not ab initio : We cannot tell the structure of a novel sequence.
HLVQ is superior to MLP
Non-local effects does not allow a 100% accuracy.

Application III Selenoprotein Discovery Miguel Lupi Alves & Carlos Lourenço Departamento de Informática Faculdade de Ciências Universidade de Lisboa

Selenoproteins
Selenium is a micronutrient found in proteins on eubacterial, archeal and eukaryotic domains of life. It is present in selenoproteins in the form of Selenocysteine (Sec), the 21 st amino acid.
Cancer prevention;
Immune function;
Male reproduction;
Degenerative diseases of the cardiac and skeletal muscles.
Selenoproteins play an important role in:

Prediction Problem
Difficult to predict selenoproteins, since Selenocysteine is the only amino acid genetically coded;
The UGA serves both as stop and as Selenocystine codon;
How can the cells’ translational machinery distinguish these two functions?

SECIS
This is the function of the stem-loop structure in the 3’-untranslated region of eukaryotic mRNAs that encode selenoproteins, known as Selenocysteine Insertion Sequence (SECIS) elements.

Prediction Problem Overview
Gene prediction programs rely on the standard stop codon;
SECIS patterns are too common;
The distance between the SECIS elements and a UGA codon influences whether the UGA is read as stop or a Selenocysteine codon;
There are examples were the UGA codon specify both selenocysteine and termination (rodent and sheep selenoprotein W);
SECIS elements have different structures in eukaryotes, archea and bacteria.

Selenoproteins prediction protocol overview Genome SECIS prediction
Use a program like PatScan to search for the SECIS patterns in a genome;
Use a program like RNAFold to assess the thermodynamic stability of the previous results.
Gene prediction
With the location of the potential SECIS found, use a program like Geneid to find genes with an in-frame TGA;
Remove, from the results, already known genes, check for incompatibility with expressed sequence tags, strange secondary structures, etc.
Lab Testing Selenoproteins
If, after all computational analysis, the researcher believes there is nothing more he/she can do to prove that the predicted genes do not code for selenoproteins, then send the results to the laboratory for confirmation and labeling.

Why Neural Networks?
Because the complexity of the problem demands a system that is robust, fault tolerant, easily adaptable to new environments without specialized programming, and which can deal with fuzzy, probabilistic, noisy or inconsistent data.
The intention is not to replace RNAFold but to learn from it.

Materials
35876 potential SECIS extracted from the first release of Drosophila M . genome;
1415 thermodynamically stable SECIS evaluated as such by RNAFold;
267 several species real SECIS;
14182 potential SECIS extracted from the 4 th release of Drosophila M. genome;
46625 potential SECIS extracted from the human mRNA.
All sequences extracted from genomes using PatScan.

Pre-processing
Use competition networks to encode each sequence;
They solve the sequence variable length problem;
They allow for a high data compression ratio.

Competition networks

Competition Networks
The neuron closer to the answer wins;
The encoded values represent the distance between the final neuron weight vectors and each of the input vectors;
Use 10 competition networks for SECIS encoding.

Results
Learning Set: 120 stable + 300 unstable SECIS from Drosophila M. ;
Test Set: 1415 stable + 7915 unstable + 3 real SECIS sequences from Drosophila M. + 267 real SECIS from other species (9600 sequences);
Prediction: 1155 including 194 real SECIS from several species

Application IV Gene Expression Microarrays
Clustering
Causality
A.Krishna, A.Narayanan, E.C.Keedwell
Department of computer Science, University of Exeter

Quick review of gene expression
A gene is expressed when it is directing protein production
Transcription of DNA to mRNA is the first step in expression
By measuring the products of transcription, we can assay gene expression

A more nuanced view
Genes are expressed at varying levels (not just on/off)
mRNA isn't just copied, but processed
Mature mRNA has
Introns removed
PolyA tail, 5' cap
Alternative splicings
...

Expression is central because...
Differentiation : All cells in a body have the same genome. Expression is what differentiates, e.g. brain cells from liver.
Physiology : Cells do their business (dividing, sending signals, digesting, etc.) largely via changes in expression
Response to stimuli : Environmental changes (like drugs or disease) often cause changes in expression
Disease markers and drug targets: changes in expression associated with disease can be diagnostic markers and/or suggest novel pharmaceutical approaches.

Arrays Technology
Always compares two samples
Uses two different fluors (red/green)
Chips have cDNAs arrayed on them

Clustering genes on expression profiles Green = Expression level low with respect to reference sample. Red = Expression level high with respect to reference sample. Black = Expression level comparable to reference sample. The columns are ordered such that similar expression profiles neighbor each other. Proceedings of the National Academy of Science, U S A. 1998 Dec 8. 95. 25. 14863-8. “Cluster analysis and display of genome-wide expression patterns”. MB Eisen, PT Spellman, PO Brown, and D Botstein. Clustered display of data from time course of serum stimulation of primary human fibroblasts. Serum was added back and samples taken at time 0, 15 min, 30 min, 1 hr, 2 hr, 3 hr, 4 hr, 8 hr, 12 hr, 16 hr, 20 hr, 24 hr. The final datapoint was from a separate unsynchronized sample. Data were measured by using a cDNA microarray with elements representing approximately 8,600 distinct human genes.

Idea is to find causality instead of just Clustering

Data preparation Training Target 0.4 0.5 4.2 2.4 G9 0.4 3.4 0.3 1.9 G8 0.1 0.2 2.5 0 G7 0.2 4.6 3.1 0 G6 4.6 0.4 0.3 0 G5 4.2 0.2 0 0.5 G4 0.6 0.2 2.5 0.9 G3 0.2 0.5 0.5 1.3 G2 0.5 3.8 0 0.4 G1 T4 T3 T2 T1

T0 sample T1sample The idea is to predict T1 from T0, i.e, discover causality: a) Which genes are controling subsequent ones b) Which genes are important

Once a neural network has been successfully trained, sensitivity analysis is introduced by perturbing the expression values of genes to check for the effect of ‘gene silencing’ or ‘gene activation’ on genes at the output layer.
Sensitivity analysis consists of experimenting with the expression values and letting the weights connecting every input gene to every output gene determine how varying the expression of an input gene affects the expression value of an output gene.
If a gene passes the sensitivity analysis as either an input (affecting gene) or as an output (affected gene), it is kept for subsequent participation in a gene network.

What is gene sensitivity analysis? A methodology for reverse engineering the interconnective relationships from temporal gene expression data. Sensitivity analysis measures the effect of small changes (perturbations) in the input channel (gene expression at time t) on the output (gene expression at time t+1). Sensitivity analysis is performed to determine the effect that each of the network inputs (gene expression values at one time step) has on the network output (gene expression at subsequent time step).

This provides feedback as to which input and output channels are the most significant. If perturbing a gene’s expression value has little effect on genes at the output layer, this is considered an ‘insignificant’ channel that can be removed (pruned) from the input space. The gene’s expression value in these experiments is varied between its mean and ± 1 standard deviation while all other inputs are fixed at their respective means. This range of sensitivity can be varied if desired. The network output is computed for a number of steps above and below the mean. This process is repeated for each input gene to determine which genes have most effect on which output genes.

where y ip is the i th output obtained with the fixed weights for the p th pattern, is the mean value for the i th output on pattern p , P is the total number of patterns, n is the number of network outputs, and is the variance of input perturbation. The sensitivity for a particular gene can be expressed as:

Métodos de validacao: train e test set, k-fold cross validation
Como lidar com outliers
Levar materiais para eles fotocopiarem

Many false positives (unbalanced)
Evolucao do erro no test ( ~10% train) e no train
One or several hidden layers?
Number weights / size training data
Eliminar redundancias

ArchDB database
Protein Data Bank (PDB) http:// www.rcsb.org contains ~ 25 000 proteins with known structure of ~ 10 6 entries in SWISS-PROT
ArchDB ~ 20 000 classified loops

Falar trabalho Heitor (pedir-lhe)
ANN usando combinacao de metodos
Dificuldades redes: codificacao amino,
Problemas bio: baixa performance. Porque? Porque funciona em redes, muitos minimos, muito contexto

Gene Prediction Problem (1/2)
Given a DNA sequence, find out:
Which region codes for a protein?
Which DNA strand is used to encode the gene?
Where does the gene starts and ends?
Where are the exon-intron boundaries?

Gene Prediction Problem (2/2)
A gene finding problem can be decomposed into two problems:
identification of coding potential of a region in a particular frame
identification of boundaries between coding and non-coding regions

Artificial Neural Networks (3/3)
An output layer comprising one neuron indicates whether or not the region is likely to be an exon.
Each neuron receives information from a set in the layer above, some with a positive value and others with a negative value; sums these values; and then converts them to an output of approximately 0 or 1.
The system is trained using a set of known coding sequences, and as each sequence is utilized, the strengths and types of connections (positive or negative) between the neurons are adjusted, decreasing or increasing the signal to the next neuron in a manner that produces the correct output.

Neural Networks on Bioinformatics

by Armando Vieira on Jun 14, 2010

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 2

Statistics

Neural Networks on Bioinformatics — Presentation Transcript