Neural Network Systems Techniques and Applications: Implementation Techniques

Neural Network Systems Techniques and Applications: Implementation Techniques

by Cornelius T. Leondes (editor)
Published by Academic Press, San Diego, CA, 1998
401 pages, $70

reviewed by
Dan Simon
Innovatia Software
dansimon@innovatia.com

This book, the third volume in its series, contains eight academically-oriented research monographs. Each monograph discusses a different facet of neural network implementation.

The first contribution is about recurrent neural networks (RNNs). RNNs are very general networks in that their neurons contain feedforward, feedback, and lateral connections. This chapter is an impressive treatise, highly mathematical and theoretical, and contains some original and previously unpublished contributions. The chapter views RNNs as a nonlinear control system, and as such may be particularly interesting to readers of Control Engineering Practice. The authors use rigorous mathematics to discuss the observability, identifiability, minimality, and controllability of both continuous-time and discrete-time RNNs. The chapter closes with a list of open problems related to the topics under discussion.

The second chapter is about Botzmann machines, which are recurrent stochastic neural networks with binary outputs. A Boltzmann machine without any hidden layers can be viewed as a stochastic version of a Hopfield network. However, a Boltzmann machine can also contain hidden layers. This type of network is particularly interesting from a statistical point of view in that it can be used to estimate the parameters of an exponential probability distribution. The authors discuss various ways of training Boltzmann machines for this purpose. This chapter is more practical than the first chapter in that it provides some simple numerical examples of the training methods and draws conclusions based on the results.

The third chapter is an excellent and comprehensive treatment of constructive training techniques for classification and regression networks. Classification networks assign an input to one of a number of distinct output classes, and regression networks provide a continuous mapping between inputs and outputs. Constructive algorithms not only optimize the typical network parameters (weights, thresholds, etc.) but also determine the architecture of the network. These techniques typically have the advantages of fast training times and good incremental learning of new data. They also have some basis in biological neural learning. However, they exhibit a tendency to overtrain. This chapter discusses several constructive training algorithms, including some that add entire subnetworks, some that prune an already-trained network, and some that use variable activation functions to add further flexibility. The comprehensiveness of this chapter can be seen by the 143 references listed at the end.

The fourth contribution discusses the relatively new topic of modular neural networks. Modularity is a fundamental engineering principle. A modular neural network is a group of loosely connected neural networks (just as a neural network is a group of loosely connected neurons). The authors survey various ways of connecting neural networks and provide some experimental results. The use of modular neural nets simplifies training, because if a task can be broken into several simpler subtasks, then separate networks can be trained more easily to solve the subtasks. The authors also touch on constructive training for modular networks, in which new modules are added to a network during training.

The fifth chapter is an extensive and excellent discussion of neural networks as associative memories. The authors study the subject from a dynamic systems point of view, noting that dynamic systems can contain equilibrium points. Stable equilibria are known as attractors. An associative memory can be viewed as a dynamic system whose attractors represent stored memory patterns. Hence the retrieval of a stored pattern is not a direct mapping, but rather a process by which the network's state reaches equilibrium. The authors present many associative memory architectures, discuss their stability and convergence properties, and delve into strategies for encoding associative memories from given desired attractors. They also discuss the tradeoff between storage capacity and error-correcting ability. Several interesting examples illustrate the principles of this chapter.

The sixth study is perhaps the most unusual in the book. The authors propose a logical and philosophical basis for the design and construction of neural networks. They discuss the reasons why neural networks are (or should be) designed as they are. They quote ancient Chinese philosophers, dissect the meaning of induction, and finally end up with a neural architecture named an Inductive Logic Unit (ILU). The chapter concludes with a simulation of an ILU. Overall the chapter is far-sighted and thought-provoking, although it may be difficult for most engineers to firmly grasp.

The seventh contribution is an impressively complete categorization of classification methods using neural networks. The chapter is notable for its many examples. The authors review methods of data analysis for determining the complexity of a classification problem. They discuss methods by which data's separability can be determined. They consider ways of selecting a classifier so as to provide the most confidence in performance. Finally they survey various ways of monitoring the stability and generalization ability of a classifier. Each of their discussions is full of interesting and illustrative examples.

The final chapter challenges the long-held assumption of neurons as simple processing elements. Current neural networks use simple neurons. Most researchers agree that the computational power of neural networks arises from the connection of many simple neurons. This chapter presents biological evidence for much more complex neurons, even neurons that change their operation under various circumstances. Most of the chapter is spent in giving circuit models for such neurons.

Overall, this book would make a valuable addition to most libraries (personal or institutional). Its academic bent makes it more useful for research than practical application, but its depth and breadth and leading-edge flavor will be of interest to many neural network engineers.

Home Credentials Publications White Papers

Last Revised: March 13, 2001