<  Back to the Polytechnique Montréal portal

Stochastic damped L-BFGS with controlled norm of the Hessian approximation

Sanae Lotfi, Tiphaine Bonniot de Ruisselet, Dominique Orban and Andrea Lodi

Technical Report (2020)

An external link is available for this item
Show abstract
Hide abstract

Abstract

We propose a new stochastic variance-reduced damped L-BFGS algorithm, where we leverage estimates of bounds on the largest and smallest eigenvalues of the Hessian approximation to balance its quality and conditioning. Our algorithm, VARCHEN, draws from previous work that proposed a novel stochastic damped L-BFGS algorithm called SdLBFGS. We establish almost sure convergence to a stationary point and a complexity bound. We empirically demonstrate that VARCHEN is more robust than SdLBFGS-VR and SVRG on a modified DavidNet problem - a highly nonconvex and ill-conditioned problem that arises in the context of deep learning, and their performance is comparable on a logistic regression problem and a nonconvex support-vector machine problem.

Department: Department of Mathematics and Industrial Engineering
Research Center: GERAD - Research Group in Decision Analysis
PolyPublie URL: https://publications.polymtl.ca/46915/
Report number: 2020-52
Official URL: https://www.gerad.ca/fr/papers/G-2020-52
Date Deposited: 18 Apr 2023 15:01
Last Modified: 05 Apr 2024 11:47
Cite in APA 7: Lotfi, S., Bonniot de Ruisselet, T., Orban, D., & Lodi, A. (2020). Stochastic damped L-BFGS with controlled norm of the Hessian approximation. (Technical Report n° 2020-52). https://www.gerad.ca/fr/papers/G-2020-52

Statistics

Stats are not available on this system.

Repository Staff Only

View Item View Item