<  Retour au portail Polytechnique Montréal

Stochastic damped L-BFGS with controlled norm of the Hessian approximation

Sanae Lotfi, Tiphaine Bonniot de Ruisselet, Dominique Orban et Andrea Lodi

Rapport technique (2020)

Un lien externe est disponible pour ce document
Afficher le résumé
Cacher le résumé

Abstract

We propose a new stochastic variance-reduced damped L-BFGS algorithm, where we leverage estimates of bounds on the largest and smallest eigenvalues of the Hessian approximation to balance its quality and conditioning. Our algorithm, VARCHEN, draws from previous work that proposed a novel stochastic damped L-BFGS algorithm called SdLBFGS. We establish almost sure convergence to a stationary point and a complexity bound. We empirically demonstrate that VARCHEN is more robust than SdLBFGS-VR and SVRG on a modified DavidNet problem - a highly nonconvex and ill-conditioned problem that arises in the context of deep learning, and their performance is comparable on a logistic regression problem and a nonconvex support-vector machine problem.

Département: Département de mathématiques et de génie industriel
Centre de recherche: GERAD - Groupe d'études et de recherche en analyse des décisions
URL de PolyPublie: https://publications.polymtl.ca/46915/
Numéro du rapport: 2020-52
URL officielle: https://www.gerad.ca/fr/papers/G-2020-52
Date du dépôt: 18 avr. 2023 15:01
Dernière modification: 03 nov. 2023 09:23
Citer en APA 7: Lotfi, S., Bonniot de Ruisselet, T., Orban, D., & Lodi, A. (2020). Stochastic damped L-BFGS with controlled norm of the Hessian approximation. (Rapport technique n° 2020-52). https://www.gerad.ca/fr/papers/G-2020-52

Statistiques

Aucune statistique n'est disponible.

Actions réservées au personnel

Afficher document Afficher document