This document provides an overview of neural networks. It discusses how neural networks were inspired by biological neural systems and attempt to model their massive parallelism and distributed representations. It covers the perceptron algorithm for learning basic neural networks and the development of backpropagation for learning in multi-layer networks. The document discusses concepts like hidden units, representational power of neural networks, and successful applications of neural networks.
An artificial neuron network (ANN) is a computational model based on the structure and functions of biological neural networks. It works on real-valued, discrete-valued and vector valued.
A Threshold Logic Unit (TLU) is a mathematical function conceived as a crude model, or abstraction of biological neurons. Threshold logic units are the constitutive units in an artificial neural network. In this paper a positive clock-edge triggered T flip-flop is designed using Perceptron Learning Algorithm, which is a basic design algorithm of threshold logic units. Then this T flip-flop is used to design a two-bit up-counter that goes through the states 0, 1, 2, 3, 0, 1… Ultimately, the goal is to show how to design simple logic units based on threshold logic based perceptron concepts.
Inroduction to Perceptron and how it is used in Machine Learning and Artificial Neural Network.
This presentation is prepared by Zaid Al-husseini, as a lectur for third stage of undergraduate students in Softwrae department - faculity of IT - University of Babylon, Iraq.
It is publicly availabe for the beginners to learn in theory and mathmatically how the Perceptron is working.
Notice: the slides are not detailed. And need a teacher to explain them deeply.
Contains description of CPN.
CP algorithm consists of a input, hidden and output layer.
In this case the hidden layer is called the Kohonen layer & the output layer is called the Grossberg layer.
Abstract: This PDSG workshop introduces basic concepts of the grandfather of neural networks - the Perceptron. Concepts covered are history, algorithm and limitations.
Level: Fundamental
Requirements: No prior programming or statistics knowledge required.
I think this could be useful for those who works in the field of Coputational Intelligence. Give your valuable reviews so that I can progree in my research
Understanding the Basics of Personal Data: Vendors, Users, and You (Web 2.0 NYC)daniela barbosa
Presentation used for Web 2.0 NYC session lead by Chis Saad and Daniela Barbosa titled: Understanding the Basics of Personal Data: Vendors, Users, and You
My notes and thoughts available here:
http://danielabarbosa.blogspot.com/2008/09/web-20-nyc-presentation-understanding.html
An artificial neuron network (ANN) is a computational model based on the structure and functions of biological neural networks. It works on real-valued, discrete-valued and vector valued.
A Threshold Logic Unit (TLU) is a mathematical function conceived as a crude model, or abstraction of biological neurons. Threshold logic units are the constitutive units in an artificial neural network. In this paper a positive clock-edge triggered T flip-flop is designed using Perceptron Learning Algorithm, which is a basic design algorithm of threshold logic units. Then this T flip-flop is used to design a two-bit up-counter that goes through the states 0, 1, 2, 3, 0, 1… Ultimately, the goal is to show how to design simple logic units based on threshold logic based perceptron concepts.
Inroduction to Perceptron and how it is used in Machine Learning and Artificial Neural Network.
This presentation is prepared by Zaid Al-husseini, as a lectur for third stage of undergraduate students in Softwrae department - faculity of IT - University of Babylon, Iraq.
It is publicly availabe for the beginners to learn in theory and mathmatically how the Perceptron is working.
Notice: the slides are not detailed. And need a teacher to explain them deeply.
Contains description of CPN.
CP algorithm consists of a input, hidden and output layer.
In this case the hidden layer is called the Kohonen layer & the output layer is called the Grossberg layer.
Abstract: This PDSG workshop introduces basic concepts of the grandfather of neural networks - the Perceptron. Concepts covered are history, algorithm and limitations.
Level: Fundamental
Requirements: No prior programming or statistics knowledge required.
I think this could be useful for those who works in the field of Coputational Intelligence. Give your valuable reviews so that I can progree in my research
Understanding the Basics of Personal Data: Vendors, Users, and You (Web 2.0 NYC)daniela barbosa
Presentation used for Web 2.0 NYC session lead by Chis Saad and Daniela Barbosa titled: Understanding the Basics of Personal Data: Vendors, Users, and You
My notes and thoughts available here:
http://danielabarbosa.blogspot.com/2008/09/web-20-nyc-presentation-understanding.html
S2DS final project presentation: Building a recommendation engine for RefMEMartina Pugliese
Final project for the Science to Data Science school in London (Aug. 2014) where we worked with company RefME to build the prototype of their recommendation engine.
Check this out - See what LinkedIn had to say about activity of Pradyp Kanabar's Outsourcing Business Expert Profile.
Please Tweet, Like, etc Now ! ... and save the math till later
Repair, maintenance services and other services often require the skills of professional servicemen. However, often theses issues are simple, and professional servicemen tend to overcharge their service fees. Furthermore, it will take time for these servicemen to reach the clients houses. What more if services are required on odd hours, more expensive prices tend to be quoted. This is where the JND come in handy, with the multi-cross platform application services, you can find the services you need around your neighbourhood in flash and hustle-less. Just Next Door!
¿Ya sabes que harás estas vacaciones? Las vacaciones son para disfrutarse con tranquilidad y sin gastar demasiado dinero.
Disfruta este verano de tus mejores vacaciones, sin remordimientos, poniendo en práctica los siguientes consejos que te servirán para planear las mejores vacaciones de tu vida.
Top 10 reasons people love listly and list postsNick Kellet
Listly is three things
Lists : Collect & Organize things you love
Curation : Assemble great content
Crowdsourcing : Get others to help create and share your story.
This is short simple slidedeck
The top 5 Things Consumers love about Lists (and List Posts)
The top 5 Things Publishers Love about Lists (and List Posts)
5-day process for answering critical business questionsFixir
"A practical guide to answering critical business questions, Sprint is a book for teams of any size, from small startups to Fortune 100s, from teachers to nonprofits. It’s for anyone with a big opportunity, problem, or idea who needs to get answers now."
An artificial neural network (ANN) is the piece of a computing system designed to simulate the way the human brain analyzes and processes information. It is the foundation of artificial intelligence (AI) and solves problems that would prove impossible or difficult by human or statistical standards. ANNs have self-learning capabilities that enable them to produce better results as more data becomes available.
final Year Projects, Final Year Projects in Chennai, Software Projects, Embedded Projects, Microcontrollers Projects, DSP Projects, VLSI Projects, Matlab Projects, Java Projects, .NET Projects, IEEE Projects, IEEE 2009 Projects, IEEE 2009 Projects, Software, IEEE 2009 Projects, Embedded, Software IEEE 2009 Projects, Embedded IEEE 2009 Projects, Final Year Project Titles, Final Year Project Reports, Final Year Project Review, Robotics Projects, Mechanical Projects, Electrical Projects, Power Electronics Projects, Power System Projects, Model Projects, Java Projects, J2EE Projects, Engineering Projects, Student Projects, Engineering College Projects, MCA Projects, BE Projects, BTech Projects, ME Projects, MTech Projects, Wireless Networks Projects, Network Security Projects, Networking Projects, final year projects, ieee projects, student projects, college projects, ieee projects in chennai, java projects, software ieee projects, embedded ieee projects, "ieee2009projects", "final year projects", "ieee projects", "Engineering Projects", "Final Year Projects in Chennai", "Final year Projects at Chennai", Java Projects, ASP.NET Projects, VB.NET Projects, C# Projects, Visual C++ Projects, Matlab Projects, NS2 Projects, C Projects, Microcontroller Projects, ATMEL Projects, PIC Projects, ARM Projects, DSP Projects, VLSI Projects, FPGA Projects, CPLD Projects, Power Electronics Projects, Electrical Projects, Robotics Projects, Solor Projects, MEMS Projects, J2EE Projects, J2ME Projects, AJAX Projects, Structs Projects, EJB Projects, Real Time Projects, Live Projects, Student Projects, Engineering Projects, MCA Projects, MBA Projects, College Projects, BE Projects, BTech Projects, ME Projects, MTech Projects, M.Sc Projects, Final Year Java Projects, Final Year ASP.NET Projects, Final Year VB.NET Projects, Final Year C# Projects, Final Year Visual C++ Projects, Final Year Matlab Projects, Final Year NS2 Projects, Final Year C Projects, Final Year Microcontroller Projects, Final Year ATMEL Projects, Final Year PIC Projects, Final Year ARM Projects, Final Year DSP Projects, Final Year VLSI Projects, Final Year FPGA Projects, Final Year CPLD Projects, Final Year Power Electronics Projects, Final Year Electrical Projects, Final Year Robotics Projects, Final Year Solor Projects, Final Year MEMS Projects, Final Year J2EE Projects, Final Year J2ME Projects, Final Year AJAX Projects, Final Year Structs Projects, Final Year EJB Projects, Final Year Real Time Projects, Final Year Live Projects, Final Year Student Projects, Final Year Engineering Projects, Final Year MCA Projects, Final Year MBA Projects, Final Year College Projects, Final Year BE Projects, Final Year BTech Projects, Final Year ME Projects, Final Year MTech Projects, Final Year M.Sc Projects, IEEE Java Projects, ASP.NET Projects, VB.NET Projects, C# Projects, Visual C++ Projects, Matlab Projects, NS2 Projects, C Projects, Microcontroller Projects, ATMEL Projects, PIC Projects, ARM Projects, DSP Projects, VLSI Projects, FPGA Projects, CPLD Projects, Power Electronics Projects, Electrical Projects, Robotics Projects, Solor Projects, MEMS Projects, J2EE Projects, J2ME Projects, AJAX Projects, Structs Projects, EJB Projects, Real Time Projects, Live Projects, Student Projects, Engineering Projects, MCA Projects, MBA Projects, College Projects, BE Projects, BTech Projects, ME Projects, MTech Projects, M.Sc Projects, IEEE 2009 Java Projects, IEEE 2009 ASP.NET Projects, IEEE 2009 VB.NET Projects, IEEE 2009 C# Projects, IEEE 2009 Visual C++ Projects, IEEE 2009 Matlab Projects, IEEE 2009 NS2 Projects, IEEE 2009 C Projects, IEEE 2009 Microcontroller Projects, IEEE 2009 ATMEL Projects, IEEE 2009 PIC Projects, IEEE 2009 ARM Projects, IEEE 2009 DSP Projects, IEEE 2009 VLSI Projects, IEEE 2009 FPGA Projects, IEEE 2009 CPLD Projects, IEEE 2009 Power Electronics Projects, IEEE 2009 Electrical Projects, IEEE 2009 Robotics Projects, IEEE 2009 Solor Projects, IEEE 2009 MEMS Projects, IEEE 2009 J2EE P
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
When stars align: studies in data quality, knowledge graphs, and machine lear...
Neural
1. 1
CS 343: Artificial Intelligence
Neural Networks
Raymond J. Mooney
University of Texas at Austin
2. 2
Neural Networks
• Analogy to biological neural systems, the most
robust learning systems we know.
• Attempt to understand natural biological systems
through computational modeling.
• Massive parallelism allows for computational
efficiency.
• Help understand “distributed” nature of neural
representations (rather than “localist”
representation) that allow robustness and graceful
degradation.
• Intelligent behavior as an “emergent” property of
large number of simple units rather than from
explicitly encoded symbolic rules and algorithms.
3. 3
Neural Speed Constraints
• Neurons have a “switching time” on the order of a
few milliseconds, compared to nanoseconds for
current computing hardware.
• However, neural systems can perform complex
cognitive tasks (vision, speech understanding) in
tenths of a second.
• Only time for performing 100 serial steps in this
time frame, compared to orders of magnitude
more for current computers.
• Must be exploiting “massive parallelism.”
• Human brain has about 1011
neurons with an
average of 104
connections each.
4. 4
Neural Network Learning
• Learning approach based on modeling
adaptation in biological neural systems.
• Perceptron: Initial algorithm for learning
simple neural networks (single layer)
developed in the 1950’s.
• Backpropagation: More complex algorithm
for learning multi-layer neural networks
developed in the 1980’s.
6. 6
Neural Communication
• Electrical potential across cell membrane exhibits spikes
called action potentials.
• Spike originates in cell body, travels down
axon, and causes synaptic terminals to
release neurotransmitters.
• Chemical diffuses across synapse to
dendrites of other neurons.
• Neurotransmitters can be excititory or
inhibitory.
• If net input of neurotransmitters to a neuron from other
neurons is excititory and exceeds some threshold, it fires an
action potential.
7. 7
Real Neural Learning
• Synapses change size and strength with
experience.
• Hebbian learning: When two connected
neurons are firing at the same time, the
strength of the synapse between them
increases.
• “Neurons that fire together, wire together.”
8. 8
Artificial Neuron Model
• Model network as a graph with cells as nodes and synaptic
connections as weighted edges from node i to node j, wji
• Model net input to cell as
• Cell output is:
1
32 54 6
w12
w13
w14
w15
w16
i
i
jij ownet ∑=
(Tj is threshold for unit j)
ji
jj
j
Tnet
Tnet
o
≥
<
=
if1
if0
netj
oj
Tj
0
1
9. 9
Neural Computation
• McCollough and Pitts (1943) showed how such model
neurons could compute logical functions and be used to
construct finite-state machines.
• Can be used to simulate logic gates:
– AND: Let all wji be Tj/n, where n is the number of inputs.
– OR: Let all wji be Tj
– NOT: Let threshold be 0, single input with a negative weight.
• Can build arbitrary logic circuits, sequential machines, and
computers with such gates.
• Given negated inputs, two layer network can compute any
boolean function using a two level AND-OR network.
10. 10
Perceptron Training
• Assume supervised training examples
giving the desired output for a unit given a
set of known input activations.
• Learn synaptic weights so that unit
produces the correct output for each
example.
• Perceptron uses iterative update algorithm
to learn a correct set of weights.
11. 11
Perceptron Learning Rule
• Update weights by:
where η is the “learning rate”
tj is the teacher specified output for unit j.
• Equivalent to rules:
– If output is correct do nothing.
– If output is high, lower weights on active inputs
– If output is low, increase weights on active inputs
• Also adjust threshold to compensate:
ijjjiji ootww )( −+= η
)( jjjj otTT −−= η
12. 12
Perceptron Learning Algorithm
• Iteratively update weights until convergence.
• Each execution of the outer loop is typically
called an epoch.
Initialize weights to random values
Until outputs of all training examples are correct
For each training pair, E, do:
Compute current output oj for E given its inputs
Compare current output to target value, tj , for E
Update synaptic weights and threshold using learning rule
13. 13
Perceptron as a Linear Separator
• Since perceptron uses linear threshold function, it is
searching for a linear separator that discriminates the
classes.
1313212 Towow >+o3
o2
13
1
2
13
12
3
w
T
o
w
w
o +−>
??
Or hyperplane in
n-dimensional space
14. 14
Concept Perceptron Cannot Learn
• Cannot learn exclusive-or, or parity
function in general.
o3
o2
??
+1
0
1
–
+–
15. 15
Perceptron Limits
• System obviously cannot learn concepts it
cannot represent.
• Minksy and Papert (1969) wrote a book
analyzing the perceptron and demonstrating
many functions it could not learn.
• These results discouraged further research
on neural nets; and symbolic AI became the
dominate paradigm.
16. 16
Perceptron Convergence
and Cycling Theorems
• Perceptron convergence theorem: If the data is
linearly separable and therefore a set of weights
exist that are consistent with the data, then the
Perceptron algorithm will eventually converge to a
consistent set of weights.
• Perceptron cycling theorem: If the data is not
linearly separable, the Perceptron algorithm will
eventually repeat a set of weights and threshold at
the end of some epoch and therefore enter an
infinite loop.
– By checking for repeated weights+threshold, one can
guarantee termination with either a positive or negative
result.
17. 17
Perceptron as Hill Climbing
• The hypothesis space being search is a set of weights and a
threshold.
• Objective is to minimize classification error on the training set.
• Perceptron effectively does hill-climbing (gradient descent) in
this space, changing the weights a small amount at each point
to decrease training set error.
• For a single model neuron, the space is well behaved with a
single minima.
weights0
training
error
18. 18
Perceptron Performance
• Linear threshold functions are restrictive (high bias) but
still reasonably expressive; more general than:
– Pure conjunctive
– Pure disjunctive
– M-of-N (at least M of a specified set of N features must be
present)
• In practice, converges fairly quickly for linearly separable
data.
• Can effectively use even incompletely converged results
when only a few outliers are misclassified.
• Experimentally, Perceptron does quite well on many
benchmark data sets.
19. 19
Multi-Layer Networks
• Multi-layer networks can represent arbitrary functions, but
an effective learning algorithm for such networks was
thought to be difficult.
• A typical multi-layer network consists of an input, hidden
and output layer, each fully connected to the next, with
activation feeding forward.
• The weights determine the function computed. Given an
arbitrary number of hidden units, any boolean function can
be computed with a single hidden layer.
output
hidden
input
activation
20. 20
Hill-Climbing in Multi-Layer Nets
• Since “greed is good” perhaps hill-climbing can be used to
learn multi-layer networks in practice although its
theoretical limits are clear.
• However, to do gradient descent, we need the output of a
unit to be a differentiable function of its input and weights.
• Standard linear threshold function is not differentiable at
the threshold.
netj
oi
Tj
0
1
21. 21
Differentiable Output Function
• Need non-linear output function to move beyond linear
functions.
– A multi-layer linear network is still linear.
• Standard solution is to use the non-linear, differentiable
sigmoidal “logistic” function:
netj
Tj
0
1
)(
1
1
jj Tnetj
e
o −−
+
=
Can also use tanh or Gaussian output function
22. 22
Gradient Descent
• Define objective to minimize error:
where D is the set of training examples, K is the set of
output units, tkd and okd are, respectively, the teacher and
current output for unit k for example d.
• The derivative of a sigmoid unit with respect to net input is:
• Learning rule to change weights to minimize error is:
2
)()( kd
Dd Kk
kd otWE −= ∑∑∈ ∈
)1( jj
j
j
oo
net
o
−=
∂
∂
ji
ji
w
E
w
∂
∂
−=∆ η
23. 23
Backpropagation Learning Rule
• Each weight changed by:
where η is a constant called the learning rate
tj is the correct teacher output for unit j
δj is the error measure for unit j
ijji ow ηδ=∆
unitoutputanisif))(1( jotoo jjjjj −−=δ
unithiddenaisif)1( jwoo
k
kjkjjj ∑−= δδ
24. 24
Error Backpropagation
• First calculate error of output units and use this to
change the top layer of weights.
output
hidden
input
Current output: oj=0.2
Correct output: tj=1.0
Error δj = oj(1–oj)(tj–oj)
0.2(1–0.2)(1–0.2)=0.128
Update weights into j
ijji ow ηδ=∆
25. 25
Error Backpropagation
• Next calculate error for hidden units based on
errors on the output units it feeds into.
output
hidden
input
∑−=
k
kjkjjj woo δδ )1(
26. 26
Error Backpropagation
• Finally update bottom layer of weights based on
errors calculated for hidden units.
output
hidden
input
∑−=
k
kjkjjj woo δδ )1(
Update weights into j
ijji ow ηδ=∆
27. 27
Backpropagation Training Algorithm
Create the 3-layer network with H hidden units with full connectivity
between layers. Set weights to small random real values.
Until all training examples produce the correct value (within ε), or
mean squared error ceases to decrease, or other termination criteria:
Begin epoch
For each training example, d, do:
Calculate network output for d’s input values
Compute error between current output and correct output for d
Update weights by backpropagating error and using learning rule
End epoch
28. 28
Comments on Training Algorithm
• Not guaranteed to converge to zero training error,
may converge to local optima or oscillate
indefinitely.
• However, in practice, does converge to low error
for many large networks on real data.
• Many epochs (thousands) may be required, hours
or days of training for large networks.
• To avoid local-minima problems, run several trials
starting with different random weights (random
restarts).
– Take results of trial with lowest training set error.
– Build a committee of results from multiple trials
(possibly weighting votes by training set accuracy).
29. 29
Representational Power
• Boolean functions: Any boolean function can be
represented by a two-layer network with sufficient
hidden units.
• Continuous functions: Any bounded continuous
function can be approximated with arbitrarily
small error by a two-layer network.
– Sigmoid functions can act as a set of basis functions for
composing more complex functions, like sine waves in
Fourier analysis.
• Arbitrary function: Any function can be
approximated to arbitrary accuracy by a three-
layer network.
30. 30
Sample Learned XOR Network
3.11
−7.386.96
−5.24
−3.6
−3.58
−5.57
−5.74
−2.03A
X Y
B
Hidden Unit A represents: ¬(X ∧ Y)
Hidden Unit B represents: ¬(X ∨ Y)
Output O represents: A ∧ ¬B = ¬(X ∧ Y) ∧ (X ∨
Y)
= X ⊕ Y
O
31. 31
Hidden Unit Representations
• Trained hidden units can be seen as newly
constructed features that make the target concept
linearly separable in the transformed space.
• On many real domains, hidden units can be
interpreted as representing meaningful features
such as vowel detectors or edge detectors, etc..
• However, the hidden layer can also become a
distributed representation of the input in which
each individual unit is not easily interpretable as a
meaningful feature.
32. 32
Successful Applications
• Text to Speech (NetTalk)
• Fraud detection
• Financial Applications
– HNC (eventually bought by Fair Isaac)
• Chemical Plant Control
– Pavillion Technologies
• Automated Vehicles
• Game Playing
– Neurogammon
• Handwriting recognition