Towards neural network approaches for point cloud compression

Point cloud imaging has emerged as an efficient and popular solution to represent immersive visual information. However, the large volume of data generated in the acquisition process reveals the need of efficient compression solutions in order to store and transmit such contents. Several standardization committees are in the process of finalizing efficient compression schemes to cope with the large volume of information that point clouds require. At the same time, recent efforts on learning-based compression approaches have been shown to exhibit good performance in the coding of conventional image and video contents. It is currently an open question how learning-based coding performs when applied to point cloud data. In this study, we extend recent efforts on the matter by exploring neural network implementations for separate, or joint compression of geometric and textural information from point cloud contents. Two alternative architectures are presented and compared; that is, a unified model that learns to encode point clouds in a holistic way, allowing fine-tuning for quality preservation per attribute, and a second paradigm consisting of two cascading networks that are trained separately to encode geometry and color, individually. A baseline configuration from the best-performing option is compared to the MPEG anchor, showing better performance for geometry and competitive performance for color encoding at low bit-rates. Moreover, the impact of a series of parameters is examined on the network performance, such as the selection of input block resolution for training and testing, the color space, and the loss functions. Results provide guidelines for future efforts in learning-based point cloud compression.

(Geometry Aware) Deep Learning-based Omnidirectional Image Compression

Yamin Sepehri

Omnidirectional images are the spherical visual signals that provide a wide, 360◦, view of a scene from a specific position. Such images are becoming increasingly popular in fields like virtual reality and robotics. Compared to conventional 2D images, the storage and badwidth requirements of omnidirectional signals are much higher, due to the specific nature of them. Thus, there is a need for image compression schemes to reduce the dedicated storage space of omnidirectional images. Image compression algorithms can be broadly classified into two groups: lossless and lossy. Lossless schemes are able to reconstruct the exact original data but they cannot reduce the size beyond a specific criteria. Lossy methods are generally better solutions if they do not add a high visual distortion to the reconstructed image, as long as they provide a decent compression rate. If a planar, lossy image compression scheme is applied on omnidirectional images, some problems show up. It is possible to apply a planar compression scheme on a projected version of a 360◦image; however, in these projection schemes (such as equirectangular projection) the sampling rate is different in the poles and the center. Consequently, the filters of the planar compression schemes that do not consider this difference ends in suboptimal result and distortions in the reconstructed images. Recently, with the success of deep neural networks in many image processing tasks, researchers began to use them for the image compression as well. In this study, we propose a deep learning-based method for the compression of omnidirectional images by combining some state of the art approaches from the deep learning-based image compression schemes and some special convolutional layers that take into account the geometry of the omnidirectional image. In comparison to the available methods, it is the first method that can be applied directly on the equirectangularly projected version of omnidirectional images and considers the geometry in the scheme and the layers themselves. To propose this method, different geometry-aware convolutional layers have been tried. We exploited various methods of downsampling and upsampling, such as spherical pooling layers, strided or transposed convolutions, bilinear interpolation, and pixel shuffle. In the end, a method is proposed that benefits from specific spherical convolutional layers which contain sampling methods considering the geometry of omnidirectional images. The sampling positions differ in the different heights of the image based on the nature of the projected omnidirectional image. Additionally, as it benefits from an iterative training method that calculates the residual between the output and input and feeds it again to the network as input of the next iteration, it can provide different compression rates with just one pass of training. Finally, it benefits from a novel method of patching that is well-aligned with the spherical convolution layers and helps the method to run efficiently without a need for a high computational power. The model was compared with a similar architecture without spherical convolutions and spherical patching and showed some improvements. The architecture has been optimized and improved and it has the potential to compete with popular image compression schemes such as JPEG especially in terms of reconstructing the colors.

2020

Efficient second-order methods for model compression

Sidak Pal Singh

Second-order information, in the form of Hessian- or Inverse-Hessian-vector products, is a fundamental tool for solving optimization problems. Recently, there has been a tremendous amount of work on utilizing this information for the current compute and memory-intensive deep neural networks, usually via coarse-grained approximations (such as diagonal, blockwise, or Kronecker-factorization). However, not much is known about the quality of these approximations. Our work addresses this question, and in particular, we propose a method called ‘WoodFisher’ that leverages the structure of the empirical Fisher information matrix, along with the Woodbury matrix identity, to compute a faithful and efficient estimate of the inverse Hessian. Our main application is to the task of compressing neural networks, where we build on the classical Optimal Brain Damage/Surgeon framework (LeCun et al., 1990; Hassibi and Stork, 1993). We demonstrate that WoodFisher significantly outperforms magnitude pruning (isotropic Hessian), as well as methods that maintain other diagonal estimates. Further, even when gradual pruning is considered, our method results in a gain in test accuracy over the state-of-the-art approaches, for standard image classification datasets such as CIFAR-10, ImageNet. We also propose a variant called ‘WoodTaylor’, which takes into account the first-order gradient term, and can lead to additional improvements. An important advantage of our methods is that they allow us to automatically set the layer-wise pruning thresholds, avoiding the need for any manual tuning or sensitivity analysis.

2020

Perceptual Quality of Point Clouds with application to Compression

Evangelos Alexiou

Modern information technologies and human-centric communication systems employ advanced content representations for richer portrayals of the real world. The newly adopted imaging modalities offer additional information cues and permit the depiction of realistic sceneries, enabling immersive experiences and promoting the engagement of the user with the content. In this context, point clouds have emerged as an attractive option to represent immersive media. This type of visual data has seen a revived interest in the recent years, following the release of low-cost depth sensors and the wide integration of modern graphics processing units in mobile phones and personal computers. Point clouds can be naturally employed in extended reality applications that involve 6 degree-of-freedom interactions, allowing adjustments of the 3-D visual information in a per-point basis. At the same time, complexity reductions are promoted when compared to mesh modelling counterpart, due to the absence of connectivity information and the elimination of corresponding constraints from acquisition to rendering. Yet, the vast amount of information that is required for faithful content representations implies the necessity for efficient data structures and compression algorithms. In particular, new coding schemes must be designed in order to reduce the amount of data and by extension the costs in processing, storage, and transmission of point clouds, while lossy compression solutions should restrain degradations for more appealing results. Furthermore, adequate subjective quality evaluation methodologies tailored to the nature of this 3D imaging modality are essential in order to obtain ground-truth data, and to better understand the impact of compression and rendering artifacts in visual quality. The development of high-performing objective metrics is also fundamental to accurately predict the perceptual quality of degraded models. In this thesis, we address the aforementioned challenges by proposing new subjective quality assessment methodologies that better simulate realistic use-cases of 3D model consumption. We examine several aspects related to point cloud visualization and display means, by exploring different rendering approaches and by introducing experimental set-ups that offer different degrees of interactivity to the end-user. The behavior of human observers in 6 degrees-of-freedom virtual reality scenes is analysed, and visual attention maps are constructed using head and gaze trajectories recorded from eye-tracking experiments. Moreover, navigation data obtained from interactive subjective evaluations in desktop arrangements are exploited to improve image-based quality metrics, whose performance is examined in predicting visual impairments on point cloud contents. In the same line of research, we design novel point-based quality predictors for point cloud topology and texture degradations, and we rigorously analyse their performance using several subjectively annotated data sets. Furthermore, adopting well-established subjective evaluation methodologies, state-of-the-art compression solutions are benchmarked and best practices for rate allocation between geometry and texture encoding are derived. Lastly, a learning-based point cloud compression solution for encoding of both geometric and color information is proposed, and the impact of a series of parameters is examined on the obtained performance to pave the path for future efforts on the field.

EPFL2021

(Geometry Aware) Deep Learning-based Omnidirectional Image Compression

Yamin Sepehri

2020

Perceptual Quality of Point Clouds with application to Compression

Evangelos Alexiou

EPFL2021