LITMIR.BIZ

3 Feedforward Neural Networks

      3.1 Introduction

      A feedforward neural network, also known as a multilayer perceptron (MLP), is a popular supervised learning method that provides a parameterized form for the nonlinear map from an input to a predicted label [6]. The form of here can be depicted graphically as a directed layered network, where the directed edges go upward from nodes in one layer to nodes in the next layer. The neural network has been seen to be a very powerful model, as they are able to approximate any Borel measurable function to an arbitrary degree, provided that parameters are chosen correctly.

      3.2 Model Description

We start by describing a simple MLP with three layers, as depicted in Figure 1.

      The bottom layer of a three‐layer MLP is called the input layer, with each node representing the respective elements of an input vector. The top layer is known as the output layer and represents the final output of the model, a predicted vector. Again, each node in the output layer represents the respective predicted score of different classes. The middle layer is called the hidden layer and captures the unobserved latent features of the input. This is the only layer where the number of nodes is determined by the user of the model, rather than the problem itself.

      The directed edges in the network represent weights from a node in one layer to another node in the next layer. We denote the weight from a node in the input layer to a node in the hidden layer as . The weight from a node in the hidden layer to a node in the output layer will be denoted . In each of the input and hidden layers, we introduce intercept nodes, denoted and , respectively. Weights from them to any other node are called biases. Each node in a given layer is connected by a weight to every node in the layer above except the intercept node.

      The value of each node in the hidden and output layers is determined as a nonlinear transformation of the linear combination of the values of the nodes in the previous layers and the weights from each of those nodes to the node of interest. That is, the value of , , is given by , where , , and is a nonlinear transformation with range in the interval . Similarly, the value of , , is given by , where , , and is also a nonlinear transformation with a range in the interval .

      More formally, the map provided by an MLP from a sample to can be written as follows: