Florida International University
Bin Shi is a Ph.D. candidate at Florida International University’s School of Computing and Information Sciences under the supervision of Dr. Sitharama S. Iyengar. Bin’s preliminary research focus is on the theory of machine learning, specifically on optimization. Bin received his B.Sc. of Applied Math from Ocean University of China in 2006, M.Sc. of Pure Mathematics from Fudan University, China in 2011 and M.Sc. of Theoretical Physics from University of Massachusetts, Dartmouth. Bin’s research interests focus on statistical machine learning and optimization, some theoretical computer science. Bin’s research contributions have been published in NIPS OPT-2017 Workshop and Informs Journal on Optimization — Special Issue for Machine Learning.
Researchers are exploring efficient algorithms to find local minima as well as global minima from Newton Second Law without friction. More specifically, some algorithms are proposed to find local minima in non-convex optimization domain and to obtain global minima in some degree from the Newton Second Law without friction. With these key observations, we present high performance algorithms that will simulate the Newton Second Law without friction based on symplectic Euler scheme. From the intuitive analysis of analytical solution, we then present a theoretical analysis for the high-speed convergence in the algorithm, and also we propose a set of experiments for testing the convex function/non-convex function which have multi attribute properties in higher dimension.
The optimality and adapability of choosing step sizes of gradient descent for escaping strict saddle points are studied. We first describe for a fixed step size, the maximum allowable step size for gradient descent to find a local minimizer is twice the inverse of gradient Lipschitz constant when all the strict saddle points are well defined. Since the step size is smaller than twice the inverse of gradient Lipschitz constant it is necessary for the convergence properties. Therefore, the optimal step size of gradient descent for strict saddle non-convex optimization problem is derived. Furthermore, the main theme is to prove this result is as long as the induced mapping by gradient descent is a local diffeomorphism (isomorphism of manifolds), thus the points that converge to any strict saddle point has Lebesgue measure 0. All the previous work require this mapping in a global diffeomorphism structure. Next, we study whether the step size can be chosen in an adaptive way for convergence properties. We also present and show that as long as the step size at each iteration is proportional to the inverse of the local gradient Lipschitz constant, gradient descent will not converge to any strict saddle point. To our knowledge, this is the first result showing gradient descent with varying step size can also escape saddle points. This property is proved by applying a generalization of Hartman-Grobman Linearization Theorem from dynamical systems theory.