An artificial neural network is a web of very simple units, each just multiplying, adding and passing on a number. Arrange enough of them in layers, adjust their weights by back propagation of errors, and the network can learn to recognise faces, translate speech or drive a car.
An artificial neural network (ANN) is made of nodes (artificial neurons) arranged in an input layer, one or more hidden layers and an output layer. Each connection has a weight. A node calculates the weighted sum of its inputs plus a bias and passes it through an activation function. Deep learning uses ANNs with many hidden layers that learn features from raw data by themselves. The network is trained by back propagation of errors: compare the output with the target, pass the error backwards through the layers, and adjust every weight a little to reduce the error.
Fig. 1: a small neural network with weights and biases; the values in the nodes are a forward pass for inputs 1 and 0.5
Section 01
From neuron to network.
An artificial neuron is loosely modelled on a brain cell: it receives signals, combines them, and fires a signal of its own. Its behaviour is set entirely by its weights and bias, which are the numbers the network learns.
Weighted sum: multiply each input by the weight of its connection, add them up and add the node's bias.
Activation function: turns the sum into the node's output. A step function outputs 1 if the sum is at least 0, otherwise 0; ReLU outputs the sum if it is positive, otherwise 0; sigmoid squashes the sum into the range 0 to 1. Non-linear activations let the network learn relationships that are not straight lines.
Layers: the input layer receives the data (e.g. pixel values); hidden layers combine it; the output layer gives the result (e.g. the probability of each class). In a fully connected network every node in one layer connects to every node in the next.
Forward pass: data flows from the input layer through the hidden layers to the output layer to make a prediction.
Worked example A forward pass
Use the weights and biases in Fig. 1 to calculate the output of the network for inputs x1 = 1 and x2 = 0.5. The hidden nodes use ReLU; the output node uses no activation.
A single neuron has weights 0.6 and 0.6, bias −1 and a step activation (output 1 if the sum is at least 0). For which pair of binary inputs does it output 1?
Only 1 and 1: 0.6 + 0.6 − 1 = 0.2, which is at least 0. Every other pair gives a negative sum (−1 or −0.4), so the neuron behaves like an AND gate.
Examiner trap
Do not forget the bias. It is added to the weighted sum before the activation function, and leaving it out changes the answer.
Section 02
Deep learning.
Older machine learning methods needed people to decide which features to measure (the length of an email, the number of links). Neural networks with many layers can learn useful features straight from raw data, which is why they transformed machine learning.
How ANNs helped machine learning: they can model very complex, non-linear relationships; they learn from examples instead of hand-written rules; and they keep improving as they are given more data.
Deep learning: an ANN with many hidden layers. Each layer learns more abstract features than the last: in image recognition, early layers detect edges, middle layers shapes, and later layers whole objects such as faces.
Reasons for using it: very high accuracy on unstructured data (images, sound, natural language) where features are hard to define by hand; used for speech recognition, translation, medical image analysis and self-driving cars.
Costs and limits: needs very large labelled data sets and a lot of processing power (often GPUs) to train; behaves as a black box, so it is hard to explain why it made a decision; can overfit (memorise the training data) if the data set is too small.
Machine learning (classical)
Deep learning
Features
chosen by people
learned automatically by the layers
Data needed
can work with smaller data sets
very large data sets
Hardware and time
modest
heavy: GPUs, long training
Explaining decisions
often easier (e.g. a regression line)
hard: a black box
Examiner trap
Deep learning is not a separate fourth category alongside supervised, unsupervised and reinforcement learning. It is a technique (deep neural networks) that can be used within any of them.
Stage 1 · Learn
Check what the sim just showed you
Four quick checks. Each correct answer earns XP and records this skill on your profile.
Quick check+10 XP
What does a node in a hidden layer calculate before applying its activation function?
Quick check+10 XP
What makes a neural network a deep learning network?
Quick check+10 XP
During back propagation, in which direction is the error passed through the network?
Quick check+10 XP
Using Fig. 1, what is the value of the top hidden node?
Section 03
Back propagation of errors.
A new network starts with random weights, so its first predictions are poor. Back propagation is how it learns: each mistake is used to work out how much each weight was to blame, and every weight is nudged in the direction that reduces the error.
1. Forward pass: feed a training example through the network to get an output.
2. Calculate the error: compare the output with the target (label) using an error (loss) function, such as E = ½(output − target)2.
3. Propagate backwards: pass the error back from the output layer through each hidden layer, working out how much each weight contributed to it (its gradient).
4. Update the weights: new weight = old weight − learning rate × gradient. This is gradient descent: a step downhill on the error.
5. Repeat for every training example, over many passes through the data (epochs), until the error is acceptably small.
Learning rate: too large and the weights overshoot and may never settle; too small and training is very slow.
Worked example One weight update
A single neuron with no bias outputs y = w × x. For x = 2 the target is 3, and w = 0.5. The error is E = ½(y − target)2, whose gradient with respect to w is (y − target) × x. Carry out one update with learning rate 0.1.
Forward pass: y = 0.5 × 2 = 1.
Error: y − target = 1 − 3 = −2, so E = ½ × 4 = 2.
Gradient: (−2) × 2 = −4.
Update: w = 0.5 − 0.1 × (−4) = 0.5 + 0.4 = 0.9.
Check: new y = 0.9 × 2 = 1.8, error −1.2, E = 0.72. The error has fallen from 2 to 0.72.
Answer: w becomes 0.9
Your turn
Carry out the next update from w = 0.9 (same x, target and learning rate).
y = 1.8, y − target = −1.2, gradient = −1.2 × 2 = −2.4, so w = 0.9 + 0.24 = 1.14 (new output 2.28, closer to 3).
Examiner trap
"The error is sent back and the weights are changed" is too vague for full marks. Say that the error is calculated at the output, propagated backwards layer by layer, and each weight is adjusted in proportion to its contribution to the error.
Stage 2 · Exam
Exam-style questions
Unlocks once the four checks above are done. Worth more XP, written to A Level Paper 3 and Paper 4 standard.
Finish the four checks above to unlock the exam questions
Exam style+20 XP
A neuron has inputs 1 and 0, weights 0.7 and 0.4, a bias of −0.5 and a step activation (output 1 if the sum is at least 0). What is the weighted sum and the output?
Exam style+20 XP
A weight is 0.8, its gradient is 2.5 and the learning rate is 0.2. What is the weight after one gradient descent update?
Exam style+20 XP
Which is the best reason to use deep learning rather than a simple regression model?
Exam style+20 XP
What is the likely effect of setting the learning rate too high during training?
Skill unlocked
Neural networks and back propagation, mastered.
This skill is now recorded on your profile. Keep the chain going.