The shape of the lesson
By the end of the lesson, learners can
- From neuron to network: An artificial neuron is loosely modelled on a brain cell: it receives signals, combines them, and fires a signal of its own.
- Deep learning: Older machine learning methods needed people to decide which features to measure (the length of an email, the number of links).
- Back propagation of errors: A new network starts with random weights, so its first predictions are poor.
- Node: weighted sum of inputs + bias, then the activation function.
- Deep learning: many hidden layers that learn features; needs lots of data and processing power.
- Back propagation: output error, passed backwards, each weight adjusted by its contribution.
- Update: new weight = old weight − learning rate × gradient.
One picture every idea returns to
An artificial neural network (ANN) is made of nodes (artificial neurons) arranged in an input layer, one or more hidden layers and an output layer. Each connection has a weight. A node calculates the weighted sum of its inputs plus a bias and passes it through an activation function. Deep learning uses ANNs with many hidden layers that learn features from raw data by themselves. The network is trained by back propagation of errors: compare the output with the target, pass the error backwards through the layers, and adjust every weight a little to reduce the error.
Forty-five minutes, phase by phase
| Time | Phase | What happens in the room | Grouping |
|---|---|---|---|
| 0 to 5 min | Hook: predict | Open with the question in the title, Networks that learn.. An artificial neural network is a web of very simple units, each just multiplying, adding and passing on a number. Put the figure on the board: it is the one picture every idea in the lesson returns to. | Whole class, then pairs |
| 5 to 13 min | Teach and check: From neuron to network | An artificial neuron is loosely modelled on a brain cell: it receives signals, combines them, and fires a signal of its own. Model the worked example A forward pass on the board, one line at a time. Your turn: A single neuron has weights 0.6 and 0.6, bias −1 and a step activation (output 1 if the sum is at least 0). For which pair of binary inputs does it output 1? | Whole class, then pairs |
| 13 to 21 min | Teach and check: Deep learning | Older machine learning methods needed people to decide which features to measure (the length of an email, the number of links). | Pairs |
| 21 to 30 min | Teach and check: Back propagation of errors | A new network starts with random weights, so its first predictions are poor. Model the worked example One weight update on the board, one line at a time. Your turn: Carry out the next update from w = 0.9 (same x, target and learning rate). | Frayer Model |
| 30 to 36 min | Check | The four Stage 1 questions from the student lesson, answered on mini whiteboards or by show of fingers. Re-teach the one most get wrong. | Individual, then whole class |
| 36 to 43 min | Plenary | The exam-style questions from Stage 2, one at a time: thirty seconds alone, then a random call. The feedback lines in the key give the examiner's wording. | Individual, random call |
| 43 to 45 min | Exit and homework | Set the worksheet below (6 questions, 20 marks). Learners self-assess against the objectives before they leave. | Individual |
Protect the parts that carry the learning
The sequence carries about fifty minutes of material, so the lab is driven by one student while the class predicts rather than explored freely. Times are a guide.
Protected: the check at 30 minutes. It is the only individual read on whether the key idea landed, and it is not cut.
Cut first if the build overruns: the last check in Back propagation of errors, never the check or the plenary.
In a 60 minute block: run worksheet questions 1 to 3 in class as a Rally Coach set and model both the hardest worked example and its check live.
Frayer Model
Pairs fill a four-box grid for the key term: definition, characteristics, examples and non-examples. The non-examples box is where the misconceptions surface, so give it the most time.
The full structure, with its variations and the other twelve, is on the cooperative-learning page.
What to head off, and how
| Trap learners fall into | Teaching move that pre-empts it |
|---|---|
| Do not forget the bias. | It is added to the weighted sum before the activation function, and leaving it out changes the answer. |
| Deep learning is not a separate fourth category alongside supervised, unsupervised and reinforcement learning. | It is a technique (deep neural networks) that can be used within any of them. |
| "The error is sent back and the weights are changed" is too vague for full marks. | Say that the error is calculated at the output, propagated backwards layer by layer, and each weight is adjusted in proportion to its contribution to the error. |
Neural networks, deep learning and back propagation: worksheet
6 questions, 20 marks. Written in the style of the papers; no past paper question is reproduced.
Describe the structure of an artificial neural network.
A network has inputs x1 = 2 and x2 = 1. Hidden node H1 has weights 0.5 (from x1) and −1.5 (from x2) and bias 0.2. Hidden node H2 has weights 0.25 and 0.5 and bias 0. Both hidden nodes use ReLU. The output node has weights 2 (from H1) and 0.6 (from H2), bias 0.1 and no activation function. Calculate the output.
Explain how back propagation of errors is used to train an artificial neural network.
A neuron with no bias outputs y = w × x. The error is E = ½(y − target)2 and its gradient with respect to w is (y − target) × x. For x = 3 the target is 1.5. The weight is 0.2 and the learning rate is 0.05. Carry out one weight update and show that the error has decreased.
Explain why deep learning is used for tasks such as speech recognition and image classification.
An estate agent has the sale prices and floor areas of 60 houses and wants to predict prices of new houses, and to explain each prediction to customers. A developer suggests a deep neural network. Evaluate this suggestion.
Answer key and marking notes
- Nodes / artificial neurons arranged in layers: an input layer, one or more hidden layers and an output layer.
- Nodes in one layer are connected to nodes in the next layer, and each connection has a weight.
- Each node adds the weighted inputs and a bias and applies an activation function to produce its output.
- H1: 0.5 × 2 − 1.5 × 1 + 0.2 = −0.3; ReLU gives 0.
- H2: 0.25 × 2 + 0.5 × 1 + 0 = 1.0; ReLU gives 1.0.
- Output: 2 × 0 + 0.6 × 1.0 + 0.1 = 0.7.
- Training data is passed forward through the network to produce an output.
- The output is compared with the expected / target output to calculate the error.
- The error is propagated backwards from the output layer through the hidden layers, finding each weight's contribution to the error.
- Each weight is adjusted (by the learning rate times its gradient) to reduce the error; this is repeated over many examples / epochs until the error is acceptable.
- y = 0.2 × 3 = 0.6; y − target = −0.9; E = ½ × 0.81 = 0.405.
- Gradient = −0.9 × 3 = −2.7.
- New w = 0.2 − 0.05 × (−2.7) = 0.2 + 0.135 = 0.335.
- New y = 0.335 × 3 = 1.005; error −0.495; E = 0.1225 (to 4 d.p.), which is less than 0.405.
- The data (sound, pixels) is unstructured and its useful features are very hard to define by hand.
- The many hidden layers learn the features automatically, building from simple features to complex ones.
- Given large data sets and enough processing power, deep networks reach much higher accuracy than classical methods on these tasks.
- Deep learning needs very large data sets; with only 60 examples the network is likely to overfit and predict poorly on new houses.
- A deep network is a black box, so the agent could not easily explain why a price was predicted.
- Linear regression (price against floor area) needs little data and processing and is easy to explain, so it is more suitable here.
6 multiple-choice questions on this lesson
From the topic question bank. Assign them in the app for a marked score in your gradebook, or run the topic as a class game.
A neuron has two inputs, both equal to 1, with weights 0.5 and 0.3 and a bias of −0.9. It uses a step activation: output 1 if the sum is at least 0, otherwise 0. What are the weighted sum and the output?
- A−0.1, so the output is 0
- B0.8, so the output is 1
- C−0.1, so the output is 1
- D1.7, so the output is 1
A hidden node receives inputs 2 and 3 along connections with weights 0.3 and −0.5, and has a bias of 0.4. It uses the ReLU activation function. What does it output?
- A0
- B−0.5
- C0.5
- D0.3
During back propagation a weight of 0.6 has a gradient of −1.8. The learning rate is 0.05. What is the weight after one gradient descent update?
- A0.69
- B0.51
- C2.4
- D0.09
Which is a disadvantage of using deep learning compared with a simpler machine learning model?
- AIt is a black box, so it is hard to explain why it made a decision
- BIt cannot learn features from raw data such as image pixels
- CIt only works on small data sets with a few numerical inputs
- DIt cannot be used for supervised learning tasks
A network has inputs x1 = 2 and x2 = 1. Hidden node H1 has weights 0.5 (from x1) and 0.25 (from x2) and bias −0.5. Hidden node H2 has weights −1 (from x1) and 0.5 (from x2) and bias 0.5. Both hidden nodes use ReLU. The output node has weights 2 (from H1) and −1 (from H2), bias 0.3 and no activation function. What is the output?
- A1.8
- B2.8
- C1.5
- D0.75
A neuron with no bias outputs y = w × x. The error is E = ½(y − target)2 and its gradient with respect to w is (y − target) × x. For x = 4 the target is 2. The weight is 0.25 and the learning rate is 0.05. What is the weight after one update?
- A0.45
- B0.05
- C0.30
- D4.25
Quiz key
0.5 × 1 + 0.3 × 1 − 0.9 = −0.1, which is less than 0, so the step function outputs 0. 0.8 leaves out the bias, −0.1 with output 1 applies the step the wrong way round, and 1.7 adds 0.9 instead of the bias of −0.9.
0.3 × 2 + (−0.5) × 3 + 0.4 = 0.6 − 1.5 + 0.4 = −0.5. ReLU outputs the sum if it is positive and 0 otherwise, so the output is 0. −0.5 forgets the activation function, 0.5 drops the minus sign (ReLU does not take the size of a negative sum), and 0.3 pairs each weight with the wrong input (0.3 × 3 − 0.5 × 2 + 0.4).
0.6 − 0.05 × (−1.8) = 0.6 + 0.09 = 0.69. 0.51 subtracts 0.09 instead of adding it (the gradient is negative, so the weight rises), 2.4 ignores the learning rate and subtracts the whole gradient, and 0.09 is only the size of the step, not the new weight.
With many hidden layers and millions of weights, a deep network's reasons for a decision are very hard to explain: it behaves as a black box. Learning features straight from raw data is its main advantage, it needs very large data sets rather than small ones, and deep learning is often used for supervised tasks such as image classification.
H1 = 0.5 × 2 + 0.25 × 1 − 0.5 = 0.75 and ReLU leaves it as 0.75. H2 = −1 × 2 + 0.5 × 1 + 0.5 = −1.0, and ReLU makes it 0. Output = 2 × 0.75 − 1 × 0 + 0.3 = 1.8. 2.8 forgets ReLU on H2 and passes −1.0 on, 1.5 leaves out the output bias, and 0.75 is only the value of H1.
y = 0.25 × 4 = 1, so y − target = −1 and the gradient is −1 × 4 = −4. New w = 0.25 − 0.05 × (−4) = 0.45; the output becomes 1.8 and E falls from 0.5 to 0.02. 0.05 adds the step in the wrong direction (0.25 − 0.2), 0.30 forgets to multiply by x in the gradient, and 4.25 leaves out the learning rate.
Slides for this lesson
A projectable deck for every pack is on the way. Until then, present the student lesson full screen: its sections, figure and lab tabs follow the same order as the plan above.