The term independent variable suggests that its value can be chosen at will, and the dependent variable is an effect, i.e., causally dependent on the independent variable, as in a stimulus-response model. Although many linear regression models are formulated as models of cause and effect, the direction of causation may just as well go the other way, or indeed there need not be any causal relation at all.

Regression, in general, is the problem of estimating a conditional expected value. Linear regression is called "linear" because the relation of the dependent to the independent variables is a linear function of some parameters. Regression models which are not a linear function of the parameters are called nonlinear regression models. A neural network is an example of a nonlinear regression model.

Still more generally, regression may be viewed as a special case of density estimation. The joint distribution of the dependent and independent variables can be constructed from the conditional distribution of the dependent variable and the marginal distribution of the independent variables. In some problems, it is convenient to work in the other direction: from the joint distribution, the conditional distribution of the dependent variable can be derived.

Table of contents

1 Historical remarks 2 Statement of the linear regression model 3 Parameter estimation 3.1 Summarizing the data 3.2 Estimating beta 3.3 Estimating alpha 3.4 Displaying the residuals 3.5 Ancillary statistics 4 References 4.6 Historical 4.7 Modern theory 4.8 Modern practice 5 External links

Historical remarks

The earliest form of linear regression was the method of least squares, which was published by Legendre in 1805, and by Gauss in 1809. The term "least squares" is from Legendre's term, moindres quarrés. However, Gauss claimed that he had known the method since 1795.

Legendre and Gauss both applied the method to the problem of determining, from astronomical observations, the orbits of bodies about the sun. Euler had worked on the same problem (1748) without success. Gauss published a further development of the theory of least squares in 1821, including a version of the Gauss-Markov theorem.

The term "reversion" was used in the nineteenth century to describe a biological phenomenon, namely that the progeny of exceptional individuals tend to be less exceptional, and more like their more distant ancestors, than their parents. Francis Galton studied this phenomenon, and applied the term "regression" to it. For Galton, regression had only this biological meaning, but his work (1877, 1885) was extended by Karl Pearson and G.U. Yule to a more general context (1897, 1903). In the work of Pearson and Yule, the joint distribution of the dependent and independent variables is assumed to be Gaussian. This assumption was weakened by R.A. Fisher in his works of 1922 and 1925. Fisher assumed that the conditional distribution of the dependent variable is Gaussian, but the joint distribution need not be. In this respect, Fisher's assumption is closer to Gauss's formulation of 1821.

Statement of the linear regression model

A linear regression model is typically stated in the form

An equivalent formulation which explicitly shows the linear regression as a model of conditional expectation is

A linear regression model need not be affine, let alone linear, in the independent variables x. For example,

Parameter estimation

Often in linear regression problems statisticians rely on the Gauss-Markov assumptions:

• The random errors ε_i have expected value 0.
• The random errors ε_i are uncorrelated (this is weaker than an assumption of probabilistic independence).
• The random errors ε_i are "homoscedastic", i.e., they all have the same variance.

Sometimes stronger assumptions are relied on:

• The random errors ε_i have expected value 0.
• They are independent.
• They are normally distributed.
• They all have the same variance.

A statistician will usually estimate the unobservable values of the parameters α and β by the method of least squares, which consists of finding the values of and that minimize the sum of squares of the residuals

Notice that, whereas the errors are independent, the residuals cannot be independent because the use of least-squares estimates implies that the sum of the residuals must be 0, and the dot-product of the vector of residuals with the vector of -values must be 0, i.e., we must have

These facts make it possible to use Student's t-distribution with n − 2 degrees of freedom (so named in honor of the pseudonymous "Student") to find confidence intervals for α and β.

Denote by capital Y the column vector whose ith entry is y_i, and by capital X the n x 2 matrix whose second column contains the x_i as its ith entry, and whose first column contains n 1s. Let ε be the column vector containing the errors ε_i. Let δ and d be respectively the 2x1 column vector containing α and β and the 2x1 column vector containing the estimates a and b. Then the model can be written as

Then it can be shown that

The matrix I_n - X (X' X)^-1 X' that appears above is a symmetric idempotent matrix of rank n - 2. Here is an example of the use of that fact in the theory of linear regression. The finite-dimensional spectral theorem of linear algebra says that any real symmetric matrix M can be diagonalized by an orthogonal matrix G, i.e., the matrix G'MG is a diagonal matrix. If the matrix M is also idempotent, then the diagonal entries in G'MG must be idempotent numbers. Only two real numbers are idempotent: 0 and 1. So I_n-X(X'X)^-1X', after diagonalization, has n − 2 0s and two 1s on the diagonal. That is most of the work in showing that the sum of squares of residuals has a chi-square distribution with n-2 degrees of freedom.

Note: A useful alternative to linear regression is robust regression in which mean absolute error is minimized instead of mean squared error as in linear regression. Robust regression is computationally much more intensive than linear regression and is somewhat more difficult to implement as well.

The first method of displaying the residuals use the histogram or cumulative distribution to depict the similarity (or lack thereof) to a normal distribution. Non-normality suggests that the model may not be a good summary description of the data.

We plot the residuals, against the independent variable, X. There should be no discernible trend or pattern if the model is satisfactory for this data. Some of the possible problems are:

• Residuals increase (or decrease) as the independent variable increases -- indicates mistakes in the calculations -- find the mistakes and correct them.
• Residuals first rise and then fall (or first fall and then rise) -- indicates that the appropriate model is (at least) quadratic. See polynomial regression.
• One residual is much larger than the others and opposite in sign -- suggests that there is one unusual observation which is distorting the fit --
  • Verify its value before publishing or
  • Eliminate it, document your decision to do so, and recalculate the statistics.

The sum of squared deviations can be partitioned as in ANOVA to indicate what part of the dispersion of the dependent variable is explained by the independent variable.

The correlation coefficient, r, can be calculated by

References

Historical

Modern practice

External links