Review of subjective quality assessment methodologies and standards for compressed images evaluation

Acquiring and exchanging a large number of pictures has become nowadays a common practice. Therefore, new image compression solutions to optimize the storage resources are in constant demand. In this context, it is essential to have a solid methodology to evaluate the performance of compression techniques. Such performance is usually measured through objective quality metrics, which are fast and inexpensive but not always reliable. However, performance is best assessed through subjective image quality assessment experiments, which are expensive and time-consuming, but reliable as based on the subjective opinion of a large number of subjects. These experiments are usually conducted in a controlled laboratory environment, with high-quality monitors and controlled lighting conditions. Recently, encouraged by the COVID-19 pandemic, crowdsourcing-based subjective image quality experiments are gaining popularity, and have demonstrated to be a faster and cheaper alternative to traditional approaches in subjective quality assessment. In this paper, different methodologies for subjective image quality assessment experiments are examined, including a review of the released standards as well as a list of publicly available tools. Moreover, the analysis is extended to novel plenoptic imaging techniques, i.e. point clouds and light fields, and to visually lossless quality assessment approaches. Authors hope that this work will help researchers interested in conducting subjective experiments for assessing the quality of compressed media, to better select the appropriate methodology for their use cases.