Projection pursuit regression

In statistics, projection pursuit regression (PPR) is a statistical model developed by Jerome H. Friedman and Werner Stuetzle which is an extension of additive models. This model adapts the additive models in that it first projects the data matrix of explanatory variables in the optimal direction before applying smoothing functions to these explanatory variables. The model consists of linear combinations of ridge functions: non-linear transformations of linear combinations of the explanatory variables. The basic model takes the form where xi is a 1 × p row of the design matrix containing the explanatory variables for example i, yi is a 1 × 1 prediction, {βj} is a collection of r vectors (each a unit vector of length p) which contain the unknown parameters, {fj} is a collection of r initially unknown smooth functions that map from R → R, and r is a hyperparameter. Good values for r can be determined through cross-validation or a forward stage-wise strategy which stops when the model fit cannot be significantly improved. As r approaches infinity and with an appropriate set of functions {fj}, the PPR model is a universal estimator, as it can approximate any continuous function in Rp. For a given set of data , the goal is to minimize the error function over the functions and vectors . No method exists for solving over all variables at once, but it can be solved via alternating optimization. First, consider each pair individually: Let all other parameters be fixed, and find a "residual", the variance of the output not accounted for by those other parameters, given by The task of minimizing the error function now reduces to solving for each j in turn. Typically new pairs are added to the model in a forward stage-wise fashion. Aside: Previously-fitted pairs can be readjusted after new fit-pairs are determined by an algorithm known as backfitting, which entails reconsidering a previous pair, recalculating the residual given how other pairs have changed, refitting to account for that new information, and then cycling through all fit-pairs this way until parameters converge.