FLOPS | EPFL Graph Search

In computing, floating point operations per second (FLOPS, flops or flop/s) is a measure of computer performance, useful in fields of scientific computations that require floating-point calculations. For such cases, it is a more accurate measure than measuring instructions per second. Floating-point arithmetic Floating-point arithmetic is needed for very large or very small real numbers, or computations that require a large dynamic range. Floating-point representation is similar to scientific notation, except everything is carried out in base two, rather than base ten. The encoding scheme stores the sign, the exponent (in base two for Cray and VAX, base two or ten for IEEE floating point formats, and base 16 for IBM Floating Point Architecture) and the significand (number after the radix point). While several similar formats are in use, the most common is ANSI/IEEE Std. 754-1985. This standard defines the format for 32-bit numbers called single precision, as well as 64-bit

Stop Wasting my FLOPS: Improving the Efficiency of Deep Learning Models

Angelos Katharopoulos

Deep neural networks have completely revolutionized the field of machinelearning by achieving state-of-the-art results on various tasks ranging fromcomputer vision to protein folding. However, their application is hindered bytheir large computational and memory requirements. In this thesis, we proposemethods for improving the efficiency of deep neural networks.Firstly, we tackle the sample inefficiency of neural network training with animportance sampling algorithm suitable for deep neural networks. This algorithmallows us to focus computation on datapoints that are going to provide usefulgradients for training our models and ignore the ones that will have negligiblegradients. We show that our algorithm can improve the performance of variousneural networks when compared to uniform sampling under a fixed computationalbudget.Secondly, we design a model that is suitable for processing large input imageswith a fraction of the computational and memory requirements of traditionalapproaches. We achieve this by sampling from a data-dependent attentiondistribution in order to only process a portion of the input in highresolution. We demonstrate that our model can learn both the attention and thefeatures in an end-to-end fashion using only single image-wise labels forsupervision.Subsequently, we shift our attention to transformer architectures and introducea kernelized formulation for self-attention that reduces its quadraticcomplexity to linear with respect to the input sequence's length. Furthermore,we uncover the relationship between autoregressive transformers and recurrentneural networks and show that our formulation enables up to 3 orders ofmagnitude faster autoregressive inference.Finally, we develop clustered, attention a method that can approximate softmaxtransformers with reduced computation. This is achieved by grouping elements ofthe input using clustering. We showcase that our formulation provides a bettertrade-off between performance and computation in comparison to the originaltransformer architecture. In addition, we demonstrate that clustered attentioncan approximate pretrained transformer models without any fine-tuning and withminimal loss in performance.

EPFL2022

Torque magnetometry study of magnetically ordered state and spin reorientation in the quasi-one-dimensional S=1/2 Heisenberg antiferromagnet CuSb2O6

Helmuth Berger

The antiferromagnetically ordered state of the monoclinic quasi-one-dimensional S = 1/2 Heisenberg antiferromagnet CuSb2O6 was studied combining torque magnetometry with a phenomenological approach to magnetic anisotropy. This system is known to have a number of different twins in the monoclinic beta phase, which differ in the orientation of the two CuO6 octahedra in the unit cell resulting in different orientation of magnetic axes with respect to crystal axes for each twin. We performed torque measurements in magnetic fields H H-SF is applied along the easy axis direction. Our analysis of magnetocrystalline anisotropy energy predicts two possibilities for the easy axis direction in this system, b or a, connected to different crystallographic twins that can be realized in CuSb2O6. These results offer a possibility to reconcile the different reports of easy axis direction found in literature for this system and also nicely demonstrate how a combination of torque magnetometry and a phenomenological approach to magnetic anisotropy can be used to determine the value of the spin-flop field and the direction of spin axis in antiferromagnets in bothH < H-SF andH > H-SF by performing measurements in fields significantly smaller than H-SF.

Amer Physical Soc2015

Stop Wasting my FLOPS: Improving the Efficiency of Deep Learning Models

Angelos Katharopoulos

EPFL2022