This paper deals with a general matrix extension problem with structural constraints and provides a recursive solution in terms of an inverse scattering experiment. Both the stationary and nonstationary cases are considered, in addition to connections to tensor algebra.