Welcome to Jozi ruby
(Building for the web)
(AI Fanatic)
(Community support)
This talk is about CI,
namely, Computational
Intelligence – the study
of adaptive
mechanisms to enable
or facilitate intel...
Computational
Intelligence
AI
Deterministic
Stochastic
Weak AI
“specific”
Strong AI
“general”
New era AI
“CI”
Old era AI
“brute force”
EC
FS
SI
AIS
NN
Evolutionary
Computation Swarm
Intelligence
Neural
Networks
Artificial
Immune
Systems
Fuzzy
Systems
4
Deterministic StochasticKnowledgeNoKnowledge
1 2
3
1. Search Space ‘fitness landscape’
2. Candidate Solution
3. Fitness Function (heuristic)
0. Global Search
concepts
HELLO WORLD
FOR AI
The Problem
Write a program that will print “hello world” to
standard out in these four cases:
!
1. Deterministic algorith...
Deterministic & Knowledge
puts "hello world"!
1
Stochastic & Knowledge
if rand() < 0.95 # 5% failure rate!
puts "hello world"!
else!
fail "a statistically unlikely death"...
No Knowledge
Write a program that will
print something to
standard out, without
knowing exactly what it is
that should be ...
No Knowledge
(uninformed)
No Knowledge
(uninformed)
Define: (3) heuristic
Computing proceeding to a solution by trial and
error or by rules that are...
No Knowledge
(uninformed)
Define: (3) fitness function
A single real value that reflects to some accuracy
how close a solu...
No Knowledge
(uninformed)
Lexical Distance
ALPHABET = ('a'..'z').to_a + [" "]!
!
def distance a, b!
return nil unless a.le...
(x,y,z)
Deterministic & No Knowledge3
xy
z
(‘h’,’e’,’l’)
’e’
’h’
’l’
(1) Search space
Deterministic & No Knowledge3
“Arrow solution”
require_relative 'distance'!
# D = domain (all possible values)!
D = ALPHA!...
Deterministic & No Knowledge3
Problem: takes far too long for a simple hello world, imagine
something that required actual...
Stochastic & No Knowledge4
Clearly, we need an intelligent solution…
(0) Global Search!!!!!!!!!!!!!!!!!
RECAP
1. Search Space ‘fitness landscape’
2. Candidate Solution
3. Fitness Function (heuristic)
0. Global Search
concepts
Evolving a solution to
hello world
with natural selection
“The sequencing is just enormously complex”
~ Kevin
“But, didn’t you write it?”
~ Sam
“Ha, some of it. The rest is just, b...
EC
FS
SI
AIS
NN
Evolutionary
Computation Swarm
Intelligence
Neural
Networks
Artificial
Immune
Systems
Fuzzy
Systems
GA: “Genetic Algorithm”
EC
Algorithmic model
developed to simulate
biological evolution.
Jean-Baptiste Lamarck’s theory of evolution was
that of heredity, i.e. the inheritance of acquired
traits. The main idea i...
Charles Darwin ~ Individuals with the “best”
characteristics (traits/genes) are more likely to
survive and to reproduce, a...
Genetic Algorithm Overview
“hllew ordjg”
“lpainbsy is”
“heldnn hjao”
“gn oojafh o”
“woolnhloea ”
Population
“hllew ordjg”
...
Genetic Algorithm Overview
@individuals = array_of_individuals!
generations.times do |generation|!
# Calculate fitness!
@i...
Crossover / Recombination
h e n n o x o r d l
0 1 1 0 1 0 0 0 1 1 0
g n l b y b w r o l d
h n l n y x o o l l
+
=
+
=
“Her...
Mutation
h e n n o x o r d l
0 1 -1 0 1 0 0 0 -1 -1 0
+
=
h f m n p x o q c l
“Diversity”
Selection
def select entities, population!
# Rank by fitness!
fitnesses = entities.map{ |e| e.fitness }!
sum = fitnesses.r...
Elitism
badassoftheweek.com
Make sure the best
individual(s) survive to the
next generation
DEMO
Questions
Break
Diagnosing Cancer
with NN & PSO
Artificial
Neural
Networks
~10b neurons
each connected
to thousands
others
via synapse
Synapse are
elastic
Learn by
thickening
Excited /
Inhibited
States
EC
FS
SI
AIS
NN
Evolutionary
Computation Swarm
Intelligence
Neural
Networks
Artificial
Immune
Systems
Fuzzy
Systems
NN
Neural
Networks
FFNN: “Feed Forward Neural Network”
Artificial model developed to
approximate the generalization
of kno...
Classification Problem
Attr1 Attr2 … AttrN Class
17.99 10.38 1.78 B
0.5 595.9 0.03 M
122.8 103.2 9.2 M
9.34 90 2.5 B
…
Inp...
Classification Problem Training
17.99 10.38 1.78 B B
Output Target
0.5 595.9 0.03 B M
122.8 103.2 9.2 B M
9.34 90 2.5 M B
...
Classification Problem Evaluation
Attr1 Attr2 … AttrN
17.99 10.38 1.78
(Malignant or
Benign)FNN: Rn -> {M,B}
Class
B
The Perceptron
f(net)
w1
w2
w3
w4
x1
x2
x3
x4
Activation strength
f(net) = f(x1w1 + x2w2 + x3w3 + x4w4)
f is an activation...
The Perceptron: OR
f(net)
w1
w2
x1
x2
x x f(net)
0 0 0
0 1 1
1 0 1
1 1 1
Guess values for
x1 and x2…
(Step activation)
f(n...
The Perceptron: OR & XOR
x x f(net)
0 0 0
0 1 1
1 0 1
1 1 1
x x f(net)
0 0 0
0 1 1
1 0 1
1 1 0
1
0
0 1
1
0
0 1
OR: XOR:
The 3 Layer FFNN
We can compose Perceptrons (which have
activation functions) to create a higher order
function from them ...
The 3 Layer FFNN Output
FNN: Rn -> {M,B}
!
meaning…
!
output = fout(net)
= fout(sum(wfmiddle(net)))
= fout(sum(wfmiddle(su...
How do we train it?
1. Gradient descent
2. Simulated annealing
3. RPROP
4. Global search? PSO!
Training our NN
with A PSO
Particle
Swarm
Optimization
Simple individuals,
working socially to
exhibit complex,
emergent,
behaviour
EC
FS
SI
AIS
NN
Evolutionary
Computation Swarm
Intelligence
Neural
Networks
Artificial
Immune
Systems
Fuzzy
Systems
SI
Swarm
Intelligence
!
PSO: “Particle Swarm Optimizer”
Algorithmic model
developed to simulate
complex emergent
behaviour...
PSO Objects
Particle:
velocity
position
personal best position
inertia
social_weight
cognitive_weight
(dimensions)
Swarm:
...
PSO Algorithm
initialize_swarm!
@iterations.times do |i|!
@swarm.each do |particle|!
particle.update_velocity!
particle.up...
PSO Algorithm: Velocity & Position Updates
Position update
position = position + v
Velocity update
v = wv + c1r1(pbest - p...
Ones Problem
Use a PSO to generate a
vector of float values where
each value equals exactly 1
The solution: [1.0, 1.0, 1.0...
DEMO
Questions
Bringing it together
Swarm
Update
personal best
How well does
it classify?
set solution vector
as weights in NN
report fit...
DEMO
Questions
fin.
Diagnosing cancer with Computational Intelligence
Upcoming SlideShare
Loading in …5
×

Diagnosing cancer with Computational Intelligence

5,270 views

Published on

An introduction to key Computational Intelligence (CI) concepts, using Hello World as an introductory example, and moving onto Diagnosing Cancer with Neural Networks.

The problem of diagnosing cancer is actually a very simple problem for CI to solve, yet it's impact can be large. It really is just up to what kind of data we have access to, that will determine our creativity in the problems we can solve with CI.

Enjoy

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,270
On SlideShare
0
From Embeds
0
Number of Embeds
4,738
Actions
Shares
0
Downloads
7
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Diagnosing cancer with Computational Intelligence

  1. 1. Welcome to Jozi ruby
  2. 2. (Building for the web) (AI Fanatic) (Community support)
  3. 3. This talk is about CI, namely, Computational Intelligence – the study of adaptive mechanisms to enable or facilitate intelligent behaviour in complex and changing environments. These mechanisms include those CI paradigms that exhibit an ability to learn or adapt to new situations, to generalise, abstract, discover and associate.
  4. 4. Computational Intelligence
  5. 5. AI Deterministic Stochastic Weak AI “specific” Strong AI “general” New era AI “CI” Old era AI “brute force”
  6. 6. EC FS SI AIS NN Evolutionary Computation Swarm Intelligence Neural Networks Artificial Immune Systems Fuzzy Systems
  7. 7. 4 Deterministic StochasticKnowledgeNoKnowledge 1 2 3
  8. 8. 1. Search Space ‘fitness landscape’ 2. Candidate Solution 3. Fitness Function (heuristic) 0. Global Search concepts
  9. 9. HELLO WORLD FOR AI
  10. 10. The Problem Write a program that will print “hello world” to standard out in these four cases: ! 1. Deterministic algorithm with knowledge (informed) 2. Stochastic algorithm with knowledge (informed) 3. Deterministic algorithm with no knowledge (uninformed) 4. Stochastic algorithm with no knowledge (uninformed)
  11. 11. Deterministic & Knowledge puts "hello world"! 1
  12. 12. Stochastic & Knowledge if rand() < 0.95 # 5% failure rate! puts "hello world"! else! fail "a statistically unlikely death"! end! 2
  13. 13. No Knowledge Write a program that will print something to standard out, without knowing exactly what it is that should be outputted… (uninformed)
  14. 14. No Knowledge (uninformed)
  15. 15. No Knowledge (uninformed) Define: (3) heuristic Computing proceeding to a solution by trial and error or by rules that are only loosely defined.
  16. 16. No Knowledge (uninformed) Define: (3) fitness function A single real value that reflects to some accuracy how close a solution is from being correct Sample solutions "hello steve" #=> half way there! "hello Mr. T" #=> half way there! " ello world" #=> almost there!!
  17. 17. No Knowledge (uninformed) Lexical Distance ALPHABET = ('a'..'z').to_a + [" "]! ! def distance a, b! return nil unless a.length == b.length! a.chars.each_index.inject(0) do |sum, i|! sum + char_distance(a[i], b[i])! end! end! ! def char_distance ai, bi! ia = ALPHABET.index(ai)! ib = ALPHABET.index(bi)! ia = (ia - ALPHABET.length).abs if ia > (ALPHABET.length/2)! ib = (ib - ALPHABET.length).abs if ib > (ALPHABET.length/2)! (ia - ib).abs! end "hello steve" #=> 21! "hello mr t" #=> 34! " ello world" #=> 6! "iello world" #=> 1! "hello world" #=> 0 (2) Candidate solution
  18. 18. (x,y,z) Deterministic & No Knowledge3 xy z (‘h’,’e’,’l’) ’e’ ’h’ ’l’ (1) Search space
  19. 19. Deterministic & No Knowledge3 “Arrow solution” require_relative 'distance'! # D = domain (all possible values)! D = ALPHA! # "hello world".length = 11 chars! best = "aaaaaaaaaaa"! D.each do |a|! D.each do |b|! D.each do |c|! D.each do |d|! D.each do |e|! D.each do |f|! D.each do |g|! D.each do |h|! D.each do |i|! D.each do |j| ##! D.each do |k| ###! candidate = "#{a}#{b}#{c}#{d}#{e}#{f}#{g}#{h}#{i}#{j}#{k}"! best = distance(candidate) < distance(best) ? candidate : best! puts "solution = #{best}" and exit 0 if distance(best) == 0! end ###! end ##! end! end! end! end! end! end! end! end! end!
  20. 20. Deterministic & No Knowledge3 Problem: takes far too long for a simple hello world, imagine something that required actual computing power… Combinations = 27^11 = 5.5590606e+15 for brute force ! Combinations = 27*11 = 297 for optimising each dimension independently (since the problem is ‘separable’)
  21. 21. Stochastic & No Knowledge4 Clearly, we need an intelligent solution… (0) Global Search!!!!!!!!!!!!!!!!!
  22. 22. RECAP 1. Search Space ‘fitness landscape’ 2. Candidate Solution 3. Fitness Function (heuristic) 0. Global Search concepts
  23. 23. Evolving a solution to hello world with natural selection
  24. 24. “The sequencing is just enormously complex” ~ Kevin “But, didn’t you write it?” ~ Sam “Ha, some of it. The rest is just, beyond me” ~ Kevin
  25. 25. EC FS SI AIS NN Evolutionary Computation Swarm Intelligence Neural Networks Artificial Immune Systems Fuzzy Systems
  26. 26. GA: “Genetic Algorithm” EC Algorithmic model developed to simulate biological evolution.
  27. 27. Jean-Baptiste Lamarck’s theory of evolution was that of heredity, i.e. the inheritance of acquired traits. The main idea is that individuals adapt during their lifetimes, and transmit their traits to their offspring
  28. 28. Charles Darwin ~ Individuals with the “best” characteristics (traits/genes) are more likely to survive and to reproduce, and those characteristics will be passed on to their offspring. These desirable characteristics are inherited by the following generations, and (over time) become dominant among the population.
  29. 29. Genetic Algorithm Overview “hllew ordjg” “lpainbsy is” “heldnn hjao” “gn oojafh o” “woolnhloea ” Population “hllew ordjg” “lpainbsy is” “heldnn hjao” “gn oojafh o” “woolnhloea ” recom bine “hllewgwooln” “lpan hjy is” “hnbsoojaeldn” “gn aoifh o” “ ordjhloea ” “hllew ordjg” “lpainbsy is” “heldnn hjao” “gn oojafh o” “woolnhloea ” “holfwgxoono” “lpam ijy is” “hnarnojaeoen” “gn aoifh o” “ oqejhloez ” m utate “heldnn hjao” “woolnhloea ” “holfwgxoono” “hnarnojaeoen” “gn aoifh o” selection
  30. 30. Genetic Algorithm Overview @individuals = array_of_individuals! generations.times do |generation|! # Calculate fitness! @individuals.each{ |individual| individual.calc_fitness }! ! # Recombine (copulate)! offspring = @crossover_strategy.crossover(@individuals)! ! # Mutate offspring! mutated_offspring = @mutation_strategy.mutate(offspring, problem)! ! # Select next generation! generation_pool = (@individuals + mutated_offspring)! new_population = @selection_strategy.select(generation_pool, @population)! ! # Ensure elitism! @individuals = elitism(new_population, generation_pool)! end! ! return best_solution_found! !
  31. 31. Crossover / Recombination h e n n o x o r d l 0 1 1 0 1 0 0 0 1 1 0 g n l b y b w r o l d h n l n y x o o l l + = + = “Heredity”
  32. 32. Mutation h e n n o x o r d l 0 1 -1 0 1 0 0 0 -1 -1 0 + = h f m n p x o q c l “Diversity”
  33. 33. Selection def select entities, population! # Rank by fitness! fitnesses = entities.map{ |e| e.fitness }! sum = fitnesses.reduce(:+).to_f! normalized_ranks = ranks.map{ |r| r.to_f/sum }! ! # Calculate, form array of tuples to keep a reference to rank & entity! tuples = []! entities.each_with_index{ |e, i| tuples << [e, normalized_ranks[i]] }! tuples.sort!{ |a,b| a[1] <=> b[1] }! ! # Select probabilistically based on rank! size = population! selected = []! while selected.length < size! tuples.each_with_index do |tuple, index|! if rand() < tuple[1]! selected << tuples[index][0]! end! end! end! selected! end!
  34. 34. Elitism badassoftheweek.com Make sure the best individual(s) survive to the next generation
  35. 35. DEMO Questions Break
  36. 36. Diagnosing Cancer with NN & PSO
  37. 37. Artificial Neural Networks
  38. 38. ~10b neurons each connected to thousands others via synapse
  39. 39. Synapse are elastic Learn by thickening
  40. 40. Excited / Inhibited States
  41. 41. EC FS SI AIS NN Evolutionary Computation Swarm Intelligence Neural Networks Artificial Immune Systems Fuzzy Systems
  42. 42. NN Neural Networks FFNN: “Feed Forward Neural Network” Artificial model developed to approximate the generalization of knowledge & discovery.
  43. 43. Classification Problem Attr1 Attr2 … AttrN Class 17.99 10.38 1.78 B 0.5 595.9 0.03 M 122.8 103.2 9.2 M 9.34 90 2.5 B … Input vector “Relevant data” Classfication “Target vector” Each row is 1 persons tissue measurements
  44. 44. Classification Problem Training 17.99 10.38 1.78 B B Output Target 0.5 595.9 0.03 B M 122.8 103.2 9.2 B M 9.34 90 2.5 M B 7.69 1.38 2.33 B M
  45. 45. Classification Problem Evaluation Attr1 Attr2 … AttrN 17.99 10.38 1.78 (Malignant or Benign)FNN: Rn -> {M,B} Class B
  46. 46. The Perceptron f(net) w1 w2 w3 w4 x1 x2 x3 x4 Activation strength f(net) = f(x1w1 + x2w2 + x3w3 + x4w4) f is an activation function: [step, sigmoid, h tan, linear] sigmoid:
  47. 47. The Perceptron: OR f(net) w1 w2 x1 x2 x x f(net) 0 0 0 0 1 1 1 0 1 1 1 1 Guess values for x1 and x2… (Step activation) f(net) = f(x1w1 + x2w2 + x3w3 + x4w4)
  48. 48. The Perceptron: OR & XOR x x f(net) 0 0 0 0 1 1 1 0 1 1 1 1 x x f(net) 0 0 0 0 1 1 1 0 1 1 1 0 1 0 0 1 1 0 0 1 OR: XOR:
  49. 49. The 3 Layer FFNN We can compose Perceptrons (which have activation functions) to create a higher order function from them that has more “information capacity”.
  50. 50. The 3 Layer FFNN Output FNN: Rn -> {M,B} ! meaning… ! output = fout(net) = fout(sum(wfmiddle(net))) = fout(sum(wfmiddle(sum(vz)))) w vz error = patterns.each do |pattern| sum(difference(target,output)) end / patterns.length Output Fitness (accuracy)
  51. 51. How do we train it? 1. Gradient descent 2. Simulated annealing 3. RPROP 4. Global search? PSO!
  52. 52. Training our NN with A PSO
  53. 53. Particle Swarm Optimization
  54. 54. Simple individuals, working socially to exhibit complex, emergent, behaviour
  55. 55. EC FS SI AIS NN Evolutionary Computation Swarm Intelligence Neural Networks Artificial Immune Systems Fuzzy Systems
  56. 56. SI Swarm Intelligence ! PSO: “Particle Swarm Optimizer” Algorithmic model developed to simulate complex emergent behaviour of swarms.
  57. 57. PSO Objects Particle: velocity position personal best position inertia social_weight cognitive_weight (dimensions) Swarm: [particle] global best position floats
  58. 58. PSO Algorithm initialize_swarm! @iterations.times do |i|! @swarm.each do |particle|! particle.update_velocity! particle.update_position! particle.update_pbest! update_gbest particle! end! end
  59. 59. PSO Algorithm: Velocity & Position Updates Position update position = position + v Velocity update v = wv + c1r1(pbest - position) + c2r2(gbest - position)
  60. 60. Ones Problem Use a PSO to generate a vector of float values where each value equals exactly 1 The solution: [1.0, 1.0, 1.0, 1.0, 1.0]
  61. 61. DEMO Questions
  62. 62. Bringing it together Swarm Update personal best How well does it classify? set solution vector as weights in NN report fitness to update gbest in the swarm
  63. 63. DEMO Questions
  64. 64. fin.

×