Motion Segmentation and Its Applications to Depth Ordering and Frame Rate Up-Conversion

In this thesis, layered motion segmentation problem and its applications to 2.5D scene representation and frame interpolation are investigated. As an initial step towards a layered design, building blocks of a generic motion segmentation algorithm; motion estimation and region segmentation, are studied. In addition, in an attempt to ensure correct propagation of information in temporal domain, a fast shot boundary detection algorithm based on color and histogram dissimilarity measures is also exploited. Several motion estimation algorithms are compared in terms of objective and subjective criteria, such as peak signal-to-noise-ratio (PSNR) of displaced frame difference and visual quality of motion-compensated frames. Feature point detection, tracking and outlier rejection problems for correspondence-based parametric motion estimation are discussed within the same context. Various state-of-the-art image segmentation algorithms are investigated and their over-segmentation performances are evaluated based on a region consistency error measure between estimated and ground truth boundaries. Based on these basic tools, three motion segmentation algorithms with increasing algorithmic and computational complexity are examined. Utilization of over- segmentation and occlusion handling towards obtaining a layered representation of motion video are discussed in the context of motion segmentation. In order to obtain temporally smooth motion layers, a temporal coherence cost term based on overlapping layer visibility voting is proposed and incorporated in a multi-component cost function. Finally, two application oriented systems utilizing estimated motion layers are proposed for layer depth ordering and region-based motion-compensated frame rate up-conversion. The results indicate that the proposed layer depth ordering algorithm provides a satisfactorily accurate sense of depth even in the presence of independently moving objects (IMO). Similarly, interpolated frames are quite promising both visually and in terms of PSNR values.