Boltzmann machine

A Boltzmann machine (also called Sherrington–Kirkpatrick model with external field or stochastic Ising–Lenz–Little model) is a stochastic spin-glass model with an external field, i.e., a Sherrington–Kirkpatrick model, that is a stochastic Ising model. It is a statistical physics technique applied in the context of cognitive science. It is also classified as a Markov random field. Boltzmann machines are theoretically intriguing because of the locality and Hebbian nature of their training algorithm (being trained by Hebb's rule), and because of their parallelism and the resemblance of their dynamics to simple physical processes. Boltzmann machines with unconstrained connectivity have not been proven useful for practical problems in machine learning or inference, but if the connectivity is properly constrained, the learning can be made efficient enough to be useful for practical problems. They are named after the Boltzmann distribution in statistical mechanics, which is used in their sampling function. They were heavily popularized and promoted by Geoffrey Hinton, Terry Sejnowski and Yann LeCun in cognitive sciences communities and in machine learning. As a more general class within machine learning these models are called "energy based models" (EBM), because Hamiltonians of spin glasses are used as a starting point to define the learning task. A Boltzmann machine, like a Sherrington–Kirkpatrick model , is a network of units with a total "energy" (Hamiltonian) defined for the overall network. Its units produce binary results. Boltzmann machine weights are stochastic. The global energy in a Boltzmann machine is identical in form to that of Hopfield networks and Ising models: Where: is the connection strength between unit and unit . is the state, , of unit . is the bias of unit in the global energy function. ( is the activation threshold for the unit.) Often the weights are represented as a symmetric matrix with zeros along the diagonal.

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Hristo Georgiev Papazov, Scott William Pesme

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size

\gamma

and momentum parameter

\beta

that allows u ...

2024

Sparse autoregressive neural networks for classical spin systems

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

KID-PPG: Knowledge Informed Deep Learning for Extracting Heart Rate from a Smartwatch

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Sparse autoregressive neural networks for classical spin systems

KID-PPG: Knowledge Informed Deep Learning for Extracting Heart Rate from a Smartwatch