Higher Variance due to Overfitting.
up vote
0
down vote
favorite
This is a homework question.
Consider the following two regression models
$$mathbf y=mathbf X_1mathbf {beta_1} + mathbf {varepsilon}$$
$$mathbf y=mathbf X_1mathbf {beta_1} + mathbf X_2mathbf {beta_2} + mathbf {varepsilon}$$
Where $operatorname{Disp} (mathbf {varepsilon}) =sigma^2mathbf I$
Now, if the first model is correct, show that for any vector $mathbf u$ one has
$$operatorname{Var} left( mathbf {u'}hat {mathbf {beta_1}}^{(2)}right) ge operatorname{Var} left( mathbf {u'}hat {mathbf {beta_1}}^{(1)}right)$$
Where $hat{mathbf beta_1}^{(1)}$ and $hat {mathbf beta_1}^{(2)}$ are the Least-Square estimators of $mathbf beta_1$ from the first and second models respectively.
My attempt: The Least-Square (LS) estimators of $mathbf beta_1$ obtained from the first and second models are
$$hat {mathbf {beta_1}}^{(1)}=mathbf {(X_1'X_1)^{-1}X_1'y}$$
$$hat {mathbf {beta_1}}^{(2)}=mathbf {(X_1'X_1)^{-1}X_1'}( mathbf y - mathbf {X_2hat {beta_2}^{(2)}})$$
Now since $mathbf {hat beta_2^{(2)}}$ is the LS estimator of $mathbf beta_2$, it is a linear function of $mathbf y$. So we may write $mathbf {X_2hat beta_2^{(2)}}= mathbf {Sy}$. Hence
$operatorname{Var} left( mathbf {u'}hat {mathbf {beta_1}}^{(2)}right) - operatorname{Var} left( mathbf {u'}hat {mathbf {beta_1}}^{(1)}right)$
$= sigma^2Bigl( mathbf {u'left(X_1'X_1right)^{-1}X_1'left(I - Sright)'left(I - Sright)X_1left(X_1'X_1right)^{-1}u - u'left(X_1'X_1right)^{-1}X_1'X_1left(X_1'X_1right)^{-1}u}Bigr)$
$= sigma^2Bigl(mathbf {z'(I - S)'(I - S)z - z'z}Bigr)$
$Bigl($Where $mathbf z = mathbf {X_1left(X_1'X_1right)^{-1}u}$ $Bigr)$
$= sigma^2Bigl(mathbf {z'SS'z - z'S'z -z'Sz}Bigr)$
Now this is where I am stuck. I know that the first term in the brackets is non-negative as the matrix $mathbf {SS'}$ is non-negative definite, but the two negative terms are something that I am completely unable to comment on. Any help would be appreciated. Thanks.
statistics self-learning regression regression-analysis
add a comment |
up vote
0
down vote
favorite
This is a homework question.
Consider the following two regression models
$$mathbf y=mathbf X_1mathbf {beta_1} + mathbf {varepsilon}$$
$$mathbf y=mathbf X_1mathbf {beta_1} + mathbf X_2mathbf {beta_2} + mathbf {varepsilon}$$
Where $operatorname{Disp} (mathbf {varepsilon}) =sigma^2mathbf I$
Now, if the first model is correct, show that for any vector $mathbf u$ one has
$$operatorname{Var} left( mathbf {u'}hat {mathbf {beta_1}}^{(2)}right) ge operatorname{Var} left( mathbf {u'}hat {mathbf {beta_1}}^{(1)}right)$$
Where $hat{mathbf beta_1}^{(1)}$ and $hat {mathbf beta_1}^{(2)}$ are the Least-Square estimators of $mathbf beta_1$ from the first and second models respectively.
My attempt: The Least-Square (LS) estimators of $mathbf beta_1$ obtained from the first and second models are
$$hat {mathbf {beta_1}}^{(1)}=mathbf {(X_1'X_1)^{-1}X_1'y}$$
$$hat {mathbf {beta_1}}^{(2)}=mathbf {(X_1'X_1)^{-1}X_1'}( mathbf y - mathbf {X_2hat {beta_2}^{(2)}})$$
Now since $mathbf {hat beta_2^{(2)}}$ is the LS estimator of $mathbf beta_2$, it is a linear function of $mathbf y$. So we may write $mathbf {X_2hat beta_2^{(2)}}= mathbf {Sy}$. Hence
$operatorname{Var} left( mathbf {u'}hat {mathbf {beta_1}}^{(2)}right) - operatorname{Var} left( mathbf {u'}hat {mathbf {beta_1}}^{(1)}right)$
$= sigma^2Bigl( mathbf {u'left(X_1'X_1right)^{-1}X_1'left(I - Sright)'left(I - Sright)X_1left(X_1'X_1right)^{-1}u - u'left(X_1'X_1right)^{-1}X_1'X_1left(X_1'X_1right)^{-1}u}Bigr)$
$= sigma^2Bigl(mathbf {z'(I - S)'(I - S)z - z'z}Bigr)$
$Bigl($Where $mathbf z = mathbf {X_1left(X_1'X_1right)^{-1}u}$ $Bigr)$
$= sigma^2Bigl(mathbf {z'SS'z - z'S'z -z'Sz}Bigr)$
Now this is where I am stuck. I know that the first term in the brackets is non-negative as the matrix $mathbf {SS'}$ is non-negative definite, but the two negative terms are something that I am completely unable to comment on. Any help would be appreciated. Thanks.
statistics self-learning regression regression-analysis
If you're going to simultaneously cross-post the same question on two (or more?) different forums, please list the other forums and keep each forum informed as to the status (especially if you've accepted an answer). Otherwise you're wasting folks' time. stats.stackexchange.com/questions/377500/…
– JimB
Nov 17 at 15:21
Sure, I'll gladly keep the forum(s) informed if someone graces my question with an answer. Until then, what else can I do?
– Neel
Nov 17 at 16:42
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
This is a homework question.
Consider the following two regression models
$$mathbf y=mathbf X_1mathbf {beta_1} + mathbf {varepsilon}$$
$$mathbf y=mathbf X_1mathbf {beta_1} + mathbf X_2mathbf {beta_2} + mathbf {varepsilon}$$
Where $operatorname{Disp} (mathbf {varepsilon}) =sigma^2mathbf I$
Now, if the first model is correct, show that for any vector $mathbf u$ one has
$$operatorname{Var} left( mathbf {u'}hat {mathbf {beta_1}}^{(2)}right) ge operatorname{Var} left( mathbf {u'}hat {mathbf {beta_1}}^{(1)}right)$$
Where $hat{mathbf beta_1}^{(1)}$ and $hat {mathbf beta_1}^{(2)}$ are the Least-Square estimators of $mathbf beta_1$ from the first and second models respectively.
My attempt: The Least-Square (LS) estimators of $mathbf beta_1$ obtained from the first and second models are
$$hat {mathbf {beta_1}}^{(1)}=mathbf {(X_1'X_1)^{-1}X_1'y}$$
$$hat {mathbf {beta_1}}^{(2)}=mathbf {(X_1'X_1)^{-1}X_1'}( mathbf y - mathbf {X_2hat {beta_2}^{(2)}})$$
Now since $mathbf {hat beta_2^{(2)}}$ is the LS estimator of $mathbf beta_2$, it is a linear function of $mathbf y$. So we may write $mathbf {X_2hat beta_2^{(2)}}= mathbf {Sy}$. Hence
$operatorname{Var} left( mathbf {u'}hat {mathbf {beta_1}}^{(2)}right) - operatorname{Var} left( mathbf {u'}hat {mathbf {beta_1}}^{(1)}right)$
$= sigma^2Bigl( mathbf {u'left(X_1'X_1right)^{-1}X_1'left(I - Sright)'left(I - Sright)X_1left(X_1'X_1right)^{-1}u - u'left(X_1'X_1right)^{-1}X_1'X_1left(X_1'X_1right)^{-1}u}Bigr)$
$= sigma^2Bigl(mathbf {z'(I - S)'(I - S)z - z'z}Bigr)$
$Bigl($Where $mathbf z = mathbf {X_1left(X_1'X_1right)^{-1}u}$ $Bigr)$
$= sigma^2Bigl(mathbf {z'SS'z - z'S'z -z'Sz}Bigr)$
Now this is where I am stuck. I know that the first term in the brackets is non-negative as the matrix $mathbf {SS'}$ is non-negative definite, but the two negative terms are something that I am completely unable to comment on. Any help would be appreciated. Thanks.
statistics self-learning regression regression-analysis
This is a homework question.
Consider the following two regression models
$$mathbf y=mathbf X_1mathbf {beta_1} + mathbf {varepsilon}$$
$$mathbf y=mathbf X_1mathbf {beta_1} + mathbf X_2mathbf {beta_2} + mathbf {varepsilon}$$
Where $operatorname{Disp} (mathbf {varepsilon}) =sigma^2mathbf I$
Now, if the first model is correct, show that for any vector $mathbf u$ one has
$$operatorname{Var} left( mathbf {u'}hat {mathbf {beta_1}}^{(2)}right) ge operatorname{Var} left( mathbf {u'}hat {mathbf {beta_1}}^{(1)}right)$$
Where $hat{mathbf beta_1}^{(1)}$ and $hat {mathbf beta_1}^{(2)}$ are the Least-Square estimators of $mathbf beta_1$ from the first and second models respectively.
My attempt: The Least-Square (LS) estimators of $mathbf beta_1$ obtained from the first and second models are
$$hat {mathbf {beta_1}}^{(1)}=mathbf {(X_1'X_1)^{-1}X_1'y}$$
$$hat {mathbf {beta_1}}^{(2)}=mathbf {(X_1'X_1)^{-1}X_1'}( mathbf y - mathbf {X_2hat {beta_2}^{(2)}})$$
Now since $mathbf {hat beta_2^{(2)}}$ is the LS estimator of $mathbf beta_2$, it is a linear function of $mathbf y$. So we may write $mathbf {X_2hat beta_2^{(2)}}= mathbf {Sy}$. Hence
$operatorname{Var} left( mathbf {u'}hat {mathbf {beta_1}}^{(2)}right) - operatorname{Var} left( mathbf {u'}hat {mathbf {beta_1}}^{(1)}right)$
$= sigma^2Bigl( mathbf {u'left(X_1'X_1right)^{-1}X_1'left(I - Sright)'left(I - Sright)X_1left(X_1'X_1right)^{-1}u - u'left(X_1'X_1right)^{-1}X_1'X_1left(X_1'X_1right)^{-1}u}Bigr)$
$= sigma^2Bigl(mathbf {z'(I - S)'(I - S)z - z'z}Bigr)$
$Bigl($Where $mathbf z = mathbf {X_1left(X_1'X_1right)^{-1}u}$ $Bigr)$
$= sigma^2Bigl(mathbf {z'SS'z - z'S'z -z'Sz}Bigr)$
Now this is where I am stuck. I know that the first term in the brackets is non-negative as the matrix $mathbf {SS'}$ is non-negative definite, but the two negative terms are something that I am completely unable to comment on. Any help would be appreciated. Thanks.
statistics self-learning regression regression-analysis
statistics self-learning regression regression-analysis
asked Nov 17 at 12:25
Neel
516
516
If you're going to simultaneously cross-post the same question on two (or more?) different forums, please list the other forums and keep each forum informed as to the status (especially if you've accepted an answer). Otherwise you're wasting folks' time. stats.stackexchange.com/questions/377500/…
– JimB
Nov 17 at 15:21
Sure, I'll gladly keep the forum(s) informed if someone graces my question with an answer. Until then, what else can I do?
– Neel
Nov 17 at 16:42
add a comment |
If you're going to simultaneously cross-post the same question on two (or more?) different forums, please list the other forums and keep each forum informed as to the status (especially if you've accepted an answer). Otherwise you're wasting folks' time. stats.stackexchange.com/questions/377500/…
– JimB
Nov 17 at 15:21
Sure, I'll gladly keep the forum(s) informed if someone graces my question with an answer. Until then, what else can I do?
– Neel
Nov 17 at 16:42
If you're going to simultaneously cross-post the same question on two (or more?) different forums, please list the other forums and keep each forum informed as to the status (especially if you've accepted an answer). Otherwise you're wasting folks' time. stats.stackexchange.com/questions/377500/…
– JimB
Nov 17 at 15:21
If you're going to simultaneously cross-post the same question on two (or more?) different forums, please list the other forums and keep each forum informed as to the status (especially if you've accepted an answer). Otherwise you're wasting folks' time. stats.stackexchange.com/questions/377500/…
– JimB
Nov 17 at 15:21
Sure, I'll gladly keep the forum(s) informed if someone graces my question with an answer. Until then, what else can I do?
– Neel
Nov 17 at 16:42
Sure, I'll gladly keep the forum(s) informed if someone graces my question with an answer. Until then, what else can I do?
– Neel
Nov 17 at 16:42
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3002305%2fhigher-variance-due-to-overfitting%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
If you're going to simultaneously cross-post the same question on two (or more?) different forums, please list the other forums and keep each forum informed as to the status (especially if you've accepted an answer). Otherwise you're wasting folks' time. stats.stackexchange.com/questions/377500/…
– JimB
Nov 17 at 15:21
Sure, I'll gladly keep the forum(s) informed if someone graces my question with an answer. Until then, what else can I do?
– Neel
Nov 17 at 16:42