Visual Attention and Perceptual Quality in Omnidirectional Imaging

Omnidirectional imaging has reached a level of widespread availability driven by recent advances in integrated circuit technology, image sensors, and computer graphics which allow now capturing, rendering and displaying of such type of immersive content in spatial resolutions sufficient to convey visual information directly to humans, as opposed to its previous use almost solely in computer vision for robotics and surveillance. In addition to its main property of covering full spherical field of view, omnidirectional imaging nowadays is an interactive multimedia; and, when experienced by means of virtual reality head-mounted displays, it achieves a remarkably high level of immersiveness. The paradigm, thus, has shifted toward human consumption of omnidirectional images and video.

Automatic prediction of salient regions in images is a well-developed topic in the field of computer vision. Yet, omnidirectional imaging brings new challenges to this topic, due to a different representation of visual information and additional degrees of freedom available to viewers. Having a model for visual attention in omnidirectional imaging is important to continue research in this subject. We develop such a model for interpreting experimental head-direction trajectories with a goal to construct a visual attention heat-map representing salient regions of an omnidirectional image. The developed model is further used in objective assessment of perceptual visual quality of omnidirectional visual content.

The problem of objectively measuring perceptual quality of omnidirectional visual content arises in many immersive imaging applications; and it is particularly important in compression and delivery. The interactive nature of this type of content limits the performance of earlier methods designed for static images or for video with a predefined dynamic. We aim to address the non-deterministic impact by using a statistical approach. In particular, we attempt to describe and analyze viewer interactions in omnidirectional imaging through estimation of visual attention. We propose an objective metric to measure perceptual quality of omnidirectional visual content considering visual attention information.

Additionally, we explore certain related extensions and applications in omnidirectional imaging. Firstly, we investigate a possible extension to 3+ degrees of freedom by considering an individual case of rendering narrow baseline light~filed images with limited translational interactions. We provide also results of extensive analysis of those iterations, including: circular histograms of directions of user movements, average vectors for a next perspective view, and charts of time spent on a view. Secondly, we look into privacy protection which is yet another field drawing more attention with the advances in image processing, visual and social media. We present a method for protecting user privacy in omnidirectional media, by removing parts of the content selected by the user, in a reversible manner. Results on distinct contents indicate our object removal methodology on the viewport enhances perceived quality, thereby improves privacy protection as the user is able to hide objects with less distortion in the overall image.

A simple method to obtain visual attention data in head mounted virtual reality

Touradj Ebrahimi, Evgeniy Upenik

Automatic prediction of salient regions in images is a well developed topic in the field of computer vision. Yet, virtual reality omnidirectional visual content brings new challenges to this topic, due to a different representation of visual information and additional degrees of freedom available to viewers. Having a model for visual attention is important to continue research in this direction. In this paper we develop such a model for head direction trajectories. The method consists of three basic steps: First, a computed head angular speed is used to exclude the parts of a trajectory where motion is too fast to fixate viewer's attention. Second, fixation locations of different subjects are fused together, optionally preceded by a re-sampling step to conform to the equal distribution of points on a sphere. Finally, a Gaussian based filtering is performed to produce continuous fixation maps. The developed model can be used to obtain ground truth experimental data when eye tracking is not available.

IEEE2017

Saliency Driven Perceptual Quality Metric for Omnidirectional Visual Content

Touradj Ebrahimi, Evgeniy Upenik

The problem of objectively measuring perceptual quality of omnidirectional visual content arises in many immersive imaging applications and particularly in compression. The interactive nature of this type of content limits the performance of earlier methods designed for static images or for video with a predefined dynamic. The non-deterministic impact must be addressed using statistical approach. One of the ways to describe, analyze and predict viewer interactions in omnidirectional imaging is through estimation of visual attention. We propose an objective metric to measure perceptual quality of omnidirectional visual content considering visual attention information.

IEEE2019

Towards Modelling of Visual Saliency in Point Clouds for Immersive Applications

Evangelos Alexiou, Touradj Ebrahimi, Peisen Xu

Modelling human visual attention is of great importance in the field of computer vision and has been widely explored for 3D imaging. Yet, in the absence of ground truth data, it is unclear whether such predictions are in alignment with the actual human viewing behavior in virtual reality environments. In this study, we work towards solving this problem by conducting an eye-tracking experiment in an immersive 3D scene that offers 6 degrees of freedom. A wide range of static point cloud models is inspected by human subjects, while their gaze is captured in real-time. The visual attention information is used to extract fixation density maps, that can be further exploited for saliency modelling. To obtain high quality fixation points, we devise a scheme that utilizes every recorded gaze measurement from the two eye-cameras of our set-up. The obtained fixation density maps together with the recorded gaze and head trajectories are made publicly available, to enrich visual saliency datasets for 3D models.

2019