Computational memory-based inference and training of deep neural networks

In-memory computing is an emerging computing paradigm where certain computational tasks are performed in place in a computational memory unit by exploiting the physical attributes of the memory devices, Here, we present an overview of the application of in-memory computing in deep learning, a branch of machine learning that has significantly contributed to the recent explosive growth in artificial intelligence. The methodology for both inference and training of deep neural networks is presented along with experimental results using phase-change memory (PCM) devices,

Learning spatio-temporal patterns in the presence of input noise using phase-change memristors

Evangelos Eleftheriou, Angeliki Pantazi, Stanislaw Andrzej Wozniak

Neuromorphic systems increasingly attract research interest owing to their ability to provide biologically inspired methods of computing, alternative to the classic von Neumann architecture. In these systems, computing relies on spike-based communication between neurons, and memory is represented by evolving states of the synaptic interconnections. In this work, we first demonstrate how spike-timing-dependent plasticity (STDP) based synapses can be realized using the crystal-growth dynamics of phase-change memristors. Then, we present a novel learning architecture comprising an integrate-and-fire neuron and an array of phase-change synapses that is capable of detecting temporal correlations in parallel input streams. We demonstrate a continuous re-learning operation on a sequence of binary 20×20 pixel images in the presence of significant background noise. Experimental results using an array of phase-change cells as synaptic elements confirm the functionality and performance of the proposed learning architecture.

2016

Multi-memristive synaptic architectures for training neural networks

Irem Boybat Kara

Highly data-centric AI workloads require that new computing paradigms be adopted because the performance of traditional CPU- and GPU-based systems are limited by data access and transfer. Training deep neural networks with millions of tunable parameters takes days or even weeks using relatively powerful heterogeneous systems, and it consumes more than hundreds of kilowatts of power. In-memory computing with memristive devices is a very promising avenue to accelerate deep learning because computations take place within the memory itself, eliminating the need to move the data around. The synaptic weights can be represented by the analog conductance states of memristive devices organized in crossbar arrays. The computationally expensive operations associated with deep learning training can be performed in place by exploiting the physical attributes and state dynamics of memristive devices and circuit laws. Memristive cores are also of particular interest due to the non-volatility, scalability, CMOS-compatibility and fast access time of the constituent devices. In addition, the multi-level storage capability of certain memristive technologies is especially attractive for increasing the information storage capacity of such cores. Large-scale demonstrations that combine memristive synapses with digital or analog CMOS circuitry indicate the potential of in-memory computing to accelerate deep learning. However, these implementations are highly vulnerable to non-ideal memristive device behavior. In particular, the limited weight representation capability, intra-device and array-level variability and temporal variations of conductance states pose significant challenges to achieving training accuracies that are comparable to conventional von Neumann implementations. Design solutions that can address these non-idealities without introducing significant implementation complexities will be critical for future memristive systems. This thesis proposes a novel synaptic architecture that can overcome a multitude of the aforementioned device non-idealities. In particular, it investigates the use of multiple memristive devices as a single computational primitive to represent a neural network weight and examines experimentally how such a compute primitive can improve non-desired memristive behavior. We propose a novel technique to arbitrate between the constituent devices of the synapse that can easily be implemented in hardware and adds only minimal energy overhead. We explore the proposed concept over various networks such as conventional non-spiking and spiking neural networks. The efficacy of this synaptic architecture is demonstrated for different training approaches including fully memristive and mixed-precision in-memory training by means of experiments using more than 1 million phase-change memory devices. Furthermore, we show that the proposed concept can be a key enabler to exploit binary memristive devices for deep-learning training.

EPFL2020

Phase-Change Memory Models for Deep Learning Training and Inference

Irem Boybat Kara, Evangelos Eleftheriou, Abu Sebastian

Non-volatile analog memory devices such as phase-change memory (PCM) enable designing dedicated connectivity matrices for the hardware implementation of deep neural networks (DNN). In this in-memory computing approach, the analog conductance states of the memory device can be gradually updated to train DNNs on-chip or software trained connection strengths may be programmed one-time to the devices to create efficient inference engines. Reliable and computationally simple models that capture the non-ideal programming and temporal evolution of the devices are needed for evaluating the training and inference performance of the deep learning hardware based on in-memory computing. In this paper, we present statistically accurate models for PCM, based on the characterization of more than 10,000 devices, that capture the state-dependent nature and variability of the conductance update, conductance drift, and read noise. Integrating the computationally simple device models with deep learning frameworks such as TensorFlow enables us to realistically evaluate training and inference performance of the PCM array based hardware implementations of DNNs.

IEEE2019

Multi-memristive synaptic architectures for training neural networks

Irem Boybat Kara

EPFL2020