LOESS: locally weighted polynomial regression

LOESS blends the simplicity of linear least squares regression with the adaptability of nonlinear regression. It achieves this by fitting simple model to localized subset of the data, gradually constructing a function that captures the deterministic pattern of the variation in the data – effectively filtering out the random component that follows some probability distribution.

Degree of local polynomials: The local polynomials fitted to each subset of the data are typically of either first or second degree. Employing a zero-degree polynomial reduces LOESS to a weighted moving average. While higher-degree polynomials could theoretically be used, they are not aligned with the spirit of LOESS. Such polynomials are prone to overfitting within each subset and thus often lead to numerical instability.

Weight function: The weight function, gives the moset weight to the data points nearest the point of estimation and the least weight to the data points that are furthest away. The use of the weights is based on the idea that points near each other are more likely to be related to each other in a simple way than points that are further apart. The traditional weight function used for LOESS is the tricube weight function: w(x) = (1 - |x|^3)^3 for |x| < 1 and 0 otherwise. The main criteria for the weight function are the following (Cleveland, 1979):

• w(x) > 0 for |x| < 1 since negative weights do not make sense
• w(-x) = w(x): there is no reason to treat points to the left of x differently from those to the right
• w(x) is a nonincreasing function for x >= 0: it seems unreasonable to allow a point that is closer to x to have less weight than the one that is further away
• w(x) = 0 for |x| >= 1 In addition it seems desirable that w(x) decrease smoothly to 0 as x goes from 0 to 1. Such a weight function is more likely to produce a smoothed result. The tricube has been chosen since it enhances a chi-squared distributional approximation of an estimate of the error variance. So it should provide an adequate smooth in many situations. The weight for a specific point in any localized subset of data is obtained by evaluating the weight function at the distance between that point and the point of estimation, after scaling the distance so that the maximum absolute distance over all of the points in the subset of data is exactly one.

Apparently, LOESS is equivalent to Savitzky-Golay filtering.

References:

• Blog post with python implementation
• The LOESS algorithm is proposed by William S. Cleveland in 1979.
• NIST example of LOESS Computation: