Neural Networks Introduction

  • Computational paradigm based on biological nervous system

Brain: $10^{10}$ neurons, $10^{4}$ fan in, $10^{14}$ connection strengths

NN learning

NN does not know anything they learnt.

  • Initial random paramters (weights)
  • Small modifications to weights on each presentation of data
  • Simple networks can be set up directly

NN processing

  • Input new patterns
  • Propagate activations along (weighted) links
  • Repeat training

Computational model of a single neuron

McCulloch & Pitts

N binary inputs x1,x2,…,xN
1 binary output y
threshold $\theta$
N weights w1,w2,…,wN
w is 1 or -1
y(x) = 1 iff xi * wi >= threshold for all i

Gating network with memory

Rosenblatt

weights not fixed
random interconnections
learn from experience

Gradient descent

Partial derivative of a function tells us how to change $w$ to minimise $f(w)$.
Gradient descent updates the parameter vector $w$, using the partial derivatives of the error function to minimise the error.

Computational model of Perceptron

weights learned from data