Introduction to Neural Networks

Kevin Gurney
University of Sheffield

Contents
Preface vii
1 Neural networks—an overview 1
1.1 What are neural networks? 1
1.2 Why study neural networks? 3
1.3 Summary 4
1.4 Notes 4
2 Real and artificial neurons 5
2.1 Real neurons: a review 5
2.2 Artificial neurons: the TLU 8
2.3 Resilience to noise and hardware failure 10
2.4 Non-binary signal communication 11
2.5 Introducing time 12
2.6 Summary 14
2.7 Notes 15
3 TLUs, linear separability and vectors 16
3.1 Geometric interpretation of TLU action 16
3.2 Vectors 18
3.3 TLUs and linear separability revisited 22
3.4 Summary 23
3.5 Notes 24
4 Training TLUs: the perceptron rule 25
4.1 Training networks 25
4.2 Training the threshold as a weight 25
4.3 Adjusting the weight vector 26
4.4 The perceptron 28
4.5 Multiple nodes and layers 29
4.6 Some practical matters 31
4.7 Summary 33
4.8 Notes 33
5 The delta rule 34
5.1 Finding the minimum of a function: gradient descent 34
5.2 Gradient descent on an error 36
5.3 The delta rule 37
5.4 Watching the delta rule at work 39
5.5 Summary 40
6 Multilayer nets and backpropagation 41
6.1 Training rules for multilayer nets 41
6.2 The backpropagation algorithm 42
6.3 Local versus global minima 43
6.4 The stopping criterion 44
6.5 Speeding up learning: the momentum term 44
6.6 More complex nets 45
6.7 The action of well-trained nets 46
6.8 Taking stock 50
6.9 Generalization and overtraining 50
6.10 Fostering generalization 52
6.11 Applications 54
6.12 Final remarks 56
6.13 Summary 56
6.14 Notes 56
7 Associative memories: the Hopfield net 57
7.1 The nature of associative memory 57
7.2 Neural networks and associative memory 58
7.3 A physical analogy with memory 58
7.4 The Hopfield net 59
7.5 Finding the weights 64
7.6 Storage capacity 66
7.7 The analogue Hopfield model 66
7.8 Combinatorial optimization 67
7.9 Feedforward and recurrent associative nets 68
7.10 Summary 69
7.11 Notes 69
8 Self-organization 70
8.1 Competitive dynamics 70
8.2 Competitive learning 72
8.3 Kohonen’s self-organizing feature maps 75
8.4 Principal component analysis 85
8.5 Further remarks 87
8.6 Summary 88
8.7 Notes 88
9 Adaptive resonance theory: ART 89
9.1 ART’s objectives 89
9.2 A hierarchical description of networks 90
9.3 ART1 91
9.4 The ART family 98
9.5 Applications 98
9.6 Further remarks 99
9.7 Summary 100
9.8 Notes 100
10 Nodes, nets and algorithms: further alternatives 101
10.1 Synapses revisited 101
10.2 Sigma-pi units 102
10.3 Digital neural networks 103
10.4 Radial basis functions 110
10.5 Learning by exploring the environment 112
10.6 Summary 115
10.7 Notes 116
11 Taxonomies, contexts and hierarchies 117
11.1 Classifying neural net structures 117
11.2 Networks and the computational hierarchy 120
11.3 Networks and statistical analysis 122
11.4 Neural networks and intelligent systems: symbols versus neurons 122
11.5 A brief history of neural nets 126
11.6 Summary 127
11.7 Notes 127
A The cosine function 128
References 130
Index 135


Another Neural Network Books
Download