Optimal Layered Representation for Adaptive Interactive Multiview Video Streaming

We consider an interactive multiview video streaming (IMVS) system where clients select their preferred viewpoint in a given navigation window. To provide high quality IMVS, many high quality views should be transmitted to the clients. However, this is not always possible due to the limited and heterogeneous capabilities of the clients. In this paper, we propose a novel adaptive IMVS solution based on a layered multiview representation where camera views are organized into layered subsets to match the different clients constraints. We formulate an optimization problem for the joint selection of the views subsets and their encoding rates. Then, we propose an optimal and a reduced computational complexity greedy algorithms, both based on dynamic-programming. Simulation results show the good performance of our novel algorithms compared to a baseline algorithm, proving that an effective IMVS adaptive solution should consider the scene content and the client capabilities and their preferences in navigation.

Adaptive Streaming in Interactive Multiview Video Systems

Pascal Frossard, Laura Toni, Xue Zhang, Yao Zhao

Multiview applications endow final users with the possibility to freely navigate within 3D scenes with minimum-delay. A real feeling of scene navigation is enabled by transmitting multiple high-quality camera views, which can be used to synthesize additional virtual views to offer a smooth navigation. However, when network resources are limited, not all camera views can be sent at high quality. It is therefore important, yet challenging, to find the right tradeoff between coding artifacts ( reducing the quality of camera views) and virtual synthesis artifacts ( reducing the number of camera views sent to users). To this aim, we propose an optimal transmission strategy for interactive multiview HTTP adaptive streaming. We propose a problem formulation to select the optimal set of camera views that the client requests for downloading, such that the navigation quality experienced by the user is optimized while the bandwidth constraints are satisfied. We show that our optimization problem is NP-hard, and we therefore develop an optimal solution based on the dynamic programming algorithm with polynomial time complexity. To further simplify the deployment, we present a suboptimal greedy algorithm with effective performance and lower complexity. The proposed controller is evaluated in theoretical and realistic settings characterized by realistic network statistics estimation, buffer management, and server-side representation optimization. Simulation results show significant improvement in terms of navigation quality compared with alternative baseline multiview adaptation logic solutions.

2019

Optimized Coding Strategies for Interactive Multiview Video

Ana Karina De Abreu Goes

The natural next step in improving the realistic experience in multimedia services is interactive multiview video (IMV). IMV promises to enable the users to freely navigate through a scene by selecting their preferred viewpoints from any view position for which the corresponding view is generated. A smooth navigation could be achieved with camera views and views synthesized at the decoder. However, the large amount of data required for such navigation experience still represents a challenge for the current systems, which implies the need for new efficient coding strategies that permit to save on storage and transmission resources, while preserving interactivity in the navigation. In this thesis, we focus on the optimization of coding strategies for IMV systems. First, we address the issues related to the coding techniques for IMV in a multiview video plus depth (MVD) scenario, where texture and depth maps are available for view synthesis at the decoder. We propose a low complexity algorithm for the selection of the interview prediction structures (PSs) and associated texture and depth quantization parameters (QPs) for IMV under transmission and storage constraints. Simulation results show that our novel low complexity algorithm has near-optimal compression efficiency while preserving interactivity properties at the decoder. Then, considering the limited and heterogeneous capabilities of current networks and decoding devices, we propose a novel adaptive solution for IMV based on a layered multiview representation where camera views are organized into layered subsets to offer different levels of navigation quality depending on the different client constraints. We propose an optimal and a reduced computational complexity greedy algorithms that jointly select the different view subsets and their encoding rates. Simulation results show the good performance of our novel algorithms compared to a baseline algorithm, proving that an effective IMV adaptive solution should consider the scene content, the client capabilities and their preferences, in building adaptive systems for multiview navigation. Finally, we build on the solution proposed in our second problem and present a general solution to rate allocation problems in multiview video. In particular, we propose a new algorithm to find the optimal Lagrange multiplier in a Lagrangian-based rate allocation problem. We show the performance of our proposed algorithm in both multiview and monoview video scenarios and show that the proposed method is able to compete with complex state-of-the-art rate control techniques. In summary, this thesis addresses important issues for coding multiview video in the design of efficient IMV systems under resource constraints. Our algorithm to select the optimal PS and QPs in a MVD scenario can improve the quality of the rendered views and it can indeed provide new insights for a deeper understanding of specific IMV coding requirements. We show that our algorithm for a layered representation of multiview video provides an effective adaptive streaming solution for IMV systems with users with limited and heterogeneous capabilities. Finally, our proposed Lagrangian-based rate allocation algorithm with an optimized selection of the Lagrange multiplier represents a general contribution that can be used in multiple video scenarios.

EPFL2015

Distortion Optimized Multipath Video Streaming

Pascal Frossard, Dan Jurca

This paper addresses the problem of choosing the best streaming policy for distortion optimal multipath video delivery, under delay constraints. The streaming policy consists in a joint selection of the video packets to be transmitted, as well as their sending time, and the transmission path. A simple streaming model is introduced, which takes into account the video packet importance, and the dependencies among packets, and allows to compute the quality perceived by the receiver, as a function of the streaming policy. We derive an optimization problem based on the video abstraction model, under the assumption that the server knows the state of the network. A detailed analysis of the timing constraints in multipath video streaming provides helpful insights that lead to an efficient algorithm to solve the NP-hard policy optimization problem. We eventually propose a fast heuristic-based algorithm, that still provides close to optimal performance. Thanks to its limited complexity, this novel algorithm is finally implemented in live streaming scenarios, where it only induces a negligible distortion penalty compared to the optimal strategy. Simulation results finally show that the proposed scheduling solutions perform better than common scheduling algorithms, and represent very efficient strategies for both stored and live video streaming scenarios.

2004

Optimized Coding Strategies for Interactive Multiview Video

Ana Karina De Abreu Goes

EPFL2015