Author ORCID Identifier
Date Available
4-28-2020
Year of Publication
2020
Document Type
Doctoral Dissertation
Degree Name
Doctor of Philosophy (PhD)
College
Arts and Sciences
Department/School/Program
Mathematics
Advisor
Dr. Qiang Ye
Abstract
Despite the recent success of various machine learning techniques, there are still numerous obstacles that must be overcome. One obstacle is known as the vanishing/exploding gradient problem. This problem refers to gradients that either become zero or unbounded. This is a well known problem that commonly occurs in Recurrent Neural Networks (RNNs). In this work we describe how this problem can be mitigated, establish three different architectures that are designed to avoid this issue, and derive update schemes for each architecture. Another portion of this work focuses on the often used technique of batch normalization. Although found to be successful in decreasing training times and in preventing overfitting, it is still unknown why this technique works. In this paper we describe batch normalization and provide a potential alternative with the end goal of improving our understanding of how batch normalization works.
Digital Object Identifier (DOI)
https://doi.org/10.13023/etd.2020.223
Recommended Citation
Helfrich, Kyle Eric, "Orthogonal Recurrent Neural Networks and Batch Normalization in Deep Neural Networks" (2020). Theses and Dissertations--Mathematics. 70.
https://uknowledge.uky.edu/math_etds/70
Included in
Applied Statistics Commons, Artificial Intelligence and Robotics Commons, Numerical Analysis and Computation Commons, Numerical Analysis and Scientific Computing Commons
