LITMIR.BIZ

Figure 9 Architecture of long short‐term memory network (LSTM).

In LSTM, if the loss is evaluated at , the gradient w.r.t. calculated via backpropagation can be written as

      (15)

      where represents other terms in the partial derivative calculation. Since the sigmoid function is used when calculating the values of , this implies that they will be close to either 0 or 1. When is close to 1, the gradient does not vanish, and when it is close to 0, it means that the previous information is not useful for the current state and should be forgotten.

7 Conclusion

      We discussed the architectures of four types of neural networks and their extensions in this chapter. There have been many other neural networks proposed in the past years, but the ones discussed in this chapter are the classical ones that served as foundations for many other works. Though DNNs have achieved breakthroughs in many fields, the performances in many fields are far from perfect. Developing new architectures that can improve the performances on various tasks or solve new problems is an important research direction. Analyzing the properties and problems of existing architectures is also of great interest to the community.

References