Prof. Bryan Caplan

bcaplan@gmu.edu

http://www3.gmu.edu/departments/economics/bcaplan

Econ 637

Spring, 1999

Weeks 3-4: The k-Variable Linear Equation

I. Quick Review of Linear Algebra

A. A matrix is a rectangular array of numbers; a vector is a matrix with only 1 column (or only 1 row); a scalar is a matrix with 1 row and 1 column.

B. Two matrices are equal iff they have the same size and the corresponding entries in the two matrices are equal.

C. If A and B are matrices, then their sum is obtained by adding the corresponding entries in each matrix. You can only add matrices of the same size.

D. If A and B are matrices, then their product can be obtained as follows: entry (i,j) in matrix AB equals (the first element of row i in column A times the first element of column j in column B), plus (the second element of row i in column A times the second element of column j in column B)+...+(the last element of row i in column A times the last element of column j in column B). Two matrices can only be multiplied if they are conformable: to get product AB, the # of columns in A must equal the # of rows in B. .

E. I is the "identity matrix" - a matrix with 1's on its diagonal and 0's everywhere else. A*I=A. A’, read "A-transpose" or "A-prime" is simply a matrix in which the 1st column of A is the first row of A’, the 2nd column of A is the second row of A’, etc.

F. A-1 is the matrix such that AA-1=I. Finding the inverse of a big matrix is extremely time consuming for a person, but computers are great at it!

G. The rank of a matrix is its number of linearly independent columns. (A set of columns is linearly independent iff has only 1 solution, with all of the k's=0).

H. A matrix can only be inverted if its rank is equal to its # of columns (aka if it has "full rank"). This will be very important for multiple regression, because a regression of Y on a set of variables that do not have full rank will not have a solution.

II. Multiple Regression

A. Intuitively: More than one factor often matters in the real world. And people frequently claim that some matter and others don't. Is there any way to extend the bivariate regression to shed light on this? E.g. finding the impact of IQ controlling for education, or finding the impact of spending controlling for deficits.

B. Mathematically: If you can find the "best" statistical fit between one dependent variable and one independent variable, can you find the "best" fit of one dependent variable on any number of independent variables?

C. It is very convenient at this point to switch to matrix notation. Suppose you have N observations of all variables you are interested in. Then just write the dependent variable, Y, as a 1-column matrix (i.e., a vector) with N rows: . And write each independent variable Xi as a vector with N rows: . Note further that we can think of the constant as itself one of the independent variables: .

D. Now we could write a regression equation with k variables (including the constant) as: , where b i is the coefficient for variable i, and u is a vector of disturbance terms.

E. This can be written even more compactly by combining all k columns of independent variables into one big matrix X with N rows and k columns. Similarly, combine all k coefficient scalars into a single (kx1) matrix of coefficients b . Then we can rewrite the above equation as: !

F. An example: Y is a student’s grade on the final exam, and it is regressed on a constant, the # of hours the student studied, and their GPA. What do Y, X, and b look like? Are the dimensions right?
  1. The Mathematics of Multiple Regression
    1. Now we will redo the results established for the bivariate case for the k-variate case, enabling us to regress any dependent variable on any set of independent variables we like.
    2. The vector of error terms, . Let’s choose b to minimize the SSE - which can now be conveniently written as e’e. (Double-check that e’e is a scalar).
    3. Note that Y’Xb is a scalar, so Y’Xb=b’X’Y.
    4. Now simply differentiate above expression wrt b. The derivative of Y’Y wrt b is obviously a vector of 0’s, since it does not contain b. The derivative of -2b’X’Y wrt b is just -2X’Y. The derivative of b’X’Xb is 2X’Xb.
    5. Setting these terms equal to a vector of zeros yields: .
    6. Rearranging and dividing by 2 gives: .
    7. Finally, pre-multiply by (X’X)-1 to solve for b: . This is an enormously powerful result.
    8. The simple vs. the expectations-augmented Phillips curve: an example of multivariate regression.
  2. Bivariate Regression as a Special Case
    1. Double-check that this formula implies the bivariate results: if , then and . Then .
    2. Writing these equations out explicitly gives:
    3. Notice that these are identical to our results for last week: just divide the first equation by N to get , and substitute this expression for b1 into the second equation.
    simplifies all of the work for the bivariate regression - and helps even more as the number of variables increases.
  3. The Full Rank Condition
    1. Mathematically, will only have a solution if (X’X) is invertible. And it will only be invertible if (X’X) has full rank. But what does this mean, and is there any intuitive interpretation?
    2. Suppose you put the same variable into your regression twice. Does this make sense?
    3. Suppose you include 3 variables: a constant, a "male" dummy (=1 if male, 0 otherwise), and a "female" dummy (=1 if female, 0 otherwise). Does this make sense?
    4. Suppose you include a variable that is always equal to zero. Does this make sense?
    5. Suppose you include both a constant (a vector of 1’s), and a vector of 3’s. Does this make sense?
    6. Suppose you have 10 variables and only 3 observations. Does this make sense?
    7. All are examples of violation of the full rank condition. In general, these are cases when your regression makes no sense - i.e., there cannot be a unique "best" way to fit the data to an equation.
    8. Rule of thumb: violation of the full rank condition never arises by chance. It can only occur if there is conceptual confusion in your initial setup - if one of the columns of X is just a linear combination of some other columns. (Although figuring out your confusion isn’t always easy!)
  4. R2 and Multiple Regression
    1. R2, as you recall, measures the "goodness of fit" of an equation. We now extend it to the multivariate case.
    2. Designate demeaned variables by their lower-case counterparts. Then: y=xb+e, so y’y=b’x’xb+e’e (recall that the correlation between X and e is always 0).
    3. This is the multivariate version of TSS=ESS+SSE. R2=ESS/TSS=1-(SSE/TSS). The square root of R2 is known as the coefficient of multiple correlation.
    4. Fun fact #1: Adding more variables to an equation cannot decrease the R2. If every non-zero coefficient on a new variable makes the fit worse, you can always just set its coefficient equal to zero to keep the fit the same as before.
    5. Fun fact #2: If you have N observations, you can always get a perfect fit with R2=1 by simply having N explanatory variables. Implications?
    6. Without proof: You must have a constant vector in your regression to be assured that R2 lies between 0 and 1.
  5. Mean of b
    1. We maintain the earlier assumption that the disturbance terms are iid N(0,s 2).
    2. . Sub in for Y to get: . Then simply apply the expectations operator: . b is an unbiased estimator for b (given our assumptions).
    3. In order to derive var(b), use the fact that E(b)=b : var(b)=E[(b-b )(b-b )'].
    4. (Verify that the dimensionality is right: the diagonals give variances, the off-diagonals give covariances).
    5. E[(b-b )(b-b )']= . Remembering that E(u'u)=s 2, and canceling inverses: var(b)=s 2(X'X)-1.
    6. This formula allows us to derive the formulas for the variance of the constant and slope coefficient in the bivariate case. That will be left as an exercise.
    7. How do you get from a calculation of var(b) to the SE's? The variance of the first coefficient is the first diagonal term; the variance of the second coefficient is the second diagonal term; etc. To get SEs, just take the square root. (How can you be sure that all of the diagonal terms will be positive? Must the off-diagonals be positive?)
    8. Application: more Phillips curve regressions.
  6. Estimating s 2
    1. If you know the value of s 2, you're already done. However, in real life you have to estimate s 2. How can this be done?
    2. Note that e=Y-Xb=Y-X(X'X)-1X'Y=(I-X(X'X)-1X')Y. Define M=I-X(X'X)-1X'; then e=MY. Furthermore, note that MY=M(Xb +u)=Mu (since MX=0).
    3. E(e'e)=E(u'M'Mu)=E(u'Mu).
    4. Aside: The trace of a matrix, tr(X), is the sum of its diagonal entries.
      1. tr(AB)=tr(BA)
      2. tr(A+B)=tr(A)+tr(B)
    5. Since the trace of a scalar is just itself (it's only got 1 diagonal!): E(u'Mu)=E[tr(u'Mu)]=E[tr(u'uM)]= s 2tr(M) (since E(u'u)=s 2 and E(M)=M).
    6. s 2tr(M)=s 2[tr(I)-tr(X(X'X)-1X']= s 2[tr(I)-tr((X'X)-1X'X]=s 2(N-k).
    7. Therefore, since E(e'e)=s 2(N-k), an unbiased estimator for s 2=e'e/(N-k)
    8. The Gauss-Markov theorem (stated without proof): the OLS estimator is BLUE (best linear unbiased estimator).
  7. Hypothesis Testing with Multivariate Regressions
    1. The simplest hypotheses to test concern only a single variable. In this case, everything learned from the bivariate case carries over: just set up an appropriate CI for the single variable you're interested in, and use the t-distribution (with N-k degrees of freedom) to get critical values.
    2. Alternately, one can use the square root of the F-test with (1, N-k) degrees of freedom. It's equivalent.
  8. Restricted vs. Unrestricted Regression
    1. For hypotheses involving more than 1 variable, it is much easier to use the F-test. There are numerous equivalent versions, so I'll just give you the easiest.
    2. First, do a vanilla regression of the dependent variable on all of the independent variables. Calculate the R2.
    3. Second, do the restricted regression that imposes the conditions you want to test. Calculate the new R2.
      1. If you are testing the hypothesis that some coefficients are zero, just re-run the regression without those variables.
      2. If you are testing the hypothesis that two variables have the same coefficient, just add them together (forcing them to have 1 coefficient).
      3. If you are testing the hypothesis that a variable equals an exact #, multiply the variable by that #, then subtract it from both sides of the equation (or combine it with the constant).
    4. Calculate the test statistic: it is equal to , where N is the # of observations, k is the # of variables in the UN-restricted regression, k2 is the # of restrictions imposed on the 2nd regression, R2 is the R2 of the UN-restricted regression, and D R2 is the change in the R2's.
    5. Compare the test statistic to the appropriate critical value: the F(k2,N-k) distribution. If your test statistic exceeds the critical value, you can reject the hypothesis.
    6. Example: I have 34 years of data on inflation and unemployment, and regress Unemployment on a constant, current inflation, lagged inflation, and lagged unemployment. Suppose I want to test two hypotheses: first, the coefficients on inflation and lagged inflation are equal and opposite; second, the coefficient on lagged unemployment =.8.
    7. First, I estimate the unrestricted equation, finding that its R2=.85. I then regress unemployment-.8*lagged unemployment on a constant and (inflation-lagged inflation), with R2=.20. So the test statistic is: (34-4)/2*.65/.15=65. The 5% critical value for F(2,34)=3.32, so this hypothesis is strongly rejected.
    8. In practice, single coefficient tests are by far the most common - mainly because they can be done without the use of a table.