Laboratory for Control, Learning, and Systems Biology

subanalytic sets

1996
  1. ▪E.D. Sontag, "Critical points for least-squares problems involving certain analytic functions, with applications to sigmoidal nets", Adv. Comput. Math., vol. 5, no. 2-3, pp. 245–268, 1996. pdf
    machine learning · artificial intelligence · subanalytic sets · semianalytic sets · critical points · approximation theory · neural networks · real-analytic functions · gradient dynamics · gradient descent · gradient systems · gradient descent · numerical methods · dynamics of algorithms
    Abstract

    This paper deals with nonlinear least-squares problems involving the fitting to data of parameterized analytic functions. For generic regression data, a general result establishes the countability, and under stronger assumptions finiteness, of the set of functions giving rise to critical points of the quadratic loss function. In the special case of what are usually called "single-hidden layer neural networks", which are built upon the standard sigmoidal activation tanh(x) or equivalently 1/(1+exp(-x)), a rough upper bound for this cardinality is provided as well.