Information theory - Intuition of channel capacity
up vote
0
down vote
favorite
Question
As stated in Elements of Information theory, given $p(y|x)$, the Information channel capacity formula is
$C = max_{p(x)} I(X; Y)$
where $X, Y$ are input and output symbols, $p(x)$ is the p.m.f of the distribution of $X$
Can anyone tell me the intuition of this formula ?
My understanding
$C = max_{p(x)} I(X; Y) = max_{p(x)} [H(X) - H(X|Y)]$
$H(X)$ can be understood as the expected amount of unknown information (measured in bits) of $X$. $H(X|Y)$ is the expected the expected amount of remaining unknown information (measured in bits) of $X$ given $Y$ is known. Thus, $I(X; Y)$ is the expected amount of known information (measured in bits) about $X$ given by $Y$.
Thus, $C =max_{p(x)} I(X; Y)$ gives the maximum number of bits in $X$, which can be exactly given by $Y$.
There are two problems with my understanding:
According to my understanding, for any input symbol $X$, there are only $I(X; Y)$ bits can be transmitted without error (i.e. no matter whether $H(X)$ exceeds the channel capacity $C$ or not)
In Elements of Information theory, $C$ is usually written as $C = max_{p(x)}[H(Y) - H(Y|X)]$, instead of $max_{p(X)}[H(X) - H(X|Y)]$ (i.e. my formula). I think $H(Y) - H(Y|X)$ says something about the right intuition of the formula $C = max_{p(x)} I(X; Y)$.
probability statistics information-theory entropy
asked Nov 18 at 3:57
HOANG GIANG
53
add a comment |
up vote
0
down vote
favorite
Question
As stated in Elements of Information theory, given $p(y|x)$, the Information channel capacity formula is
$C = max_{p(x)} I(X; Y)$
where $X, Y$ are input and output symbols, $p(x)$ is the p.m.f of the distribution of $X$
Can anyone tell me the intuition of this formula ?
My understanding
$C = max_{p(x)} I(X; Y) = max_{p(x)} [H(X) - H(X|Y)]$
$H(X)$ can be understood as the expected amount of unknown information (measured in bits) of $X$. $H(X|Y)$ is the expected the expected amount of remaining unknown information (measured in bits) of $X$ given $Y$ is known. Thus, $I(X; Y)$ is the expected amount of known information (measured in bits) about $X$ given by $Y$.
Thus, $C =max_{p(x)} I(X; Y)$ gives the maximum number of bits in $X$, which can be exactly given by $Y$.
There are two problems with my understanding:
According to my understanding, for any input symbol $X$, there are only $I(X; Y)$ bits can be transmitted without error (i.e. no matter whether $H(X)$ exceeds the channel capacity $C$ or not)
In Elements of Information theory, $C$ is usually written as $C = max_{p(x)}[H(Y) - H(Y|X)]$, instead of $max_{p(X)}[H(X) - H(X|Y)]$ (i.e. my formula). I think $H(Y) - H(Y|X)$ says something about the right intuition of the formula $C = max_{p(x)} I(X; Y)$.
probability statistics information-theory entropy
asked Nov 18 at 3:57
HOANG GIANG
53
1
Your question is vague and too broad. You really need to read carefully that chapter of the book, it's all explained there. "$I(X; Y)$ is the expected amount of known information (measured in bits) about $X$ given by $Y$" is more or less right... but " the maximum number of bits in X, which can be exactly given by Y" makes little sense . Capacity is not related with knowing exactly some amount of bits. In general (think of the BSC channel) you cannot do that.
– leonbloy
Nov 19 at 4:01
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
Question
As stated in Elements of Information theory, given $p(y|x)$, the Information channel capacity formula is
$C = max_{p(x)} I(X; Y)$
where $X, Y$ are input and output symbols, $p(x)$ is the p.m.f of the distribution of $X$
Can anyone tell me the intuition of this formula ?
My understanding
$C = max_{p(x)} I(X; Y) = max_{p(x)} [H(X) - H(X|Y)]$
$H(X)$ can be understood as the expected amount of unknown information (measured in bits) of $X$. $H(X|Y)$ is the expected the expected amount of remaining unknown information (measured in bits) of $X$ given $Y$ is known. Thus, $I(X; Y)$ is the expected amount of known information (measured in bits) about $X$ given by $Y$.
Thus, $C =max_{p(x)} I(X; Y)$ gives the maximum number of bits in $X$, which can be exactly given by $Y$.
There are two problems with my understanding:
According to my understanding, for any input symbol $X$, there are only $I(X; Y)$ bits can be transmitted without error (i.e. no matter whether $H(X)$ exceeds the channel capacity $C$ or not)
In Elements of Information theory, $C$ is usually written as $C = max_{p(x)}[H(Y) - H(Y|X)]$, instead of $max_{p(X)}[H(X) - H(X|Y)]$ (i.e. my formula). I think $H(Y) - H(Y|X)$ says something about the right intuition of the formula $C = max_{p(x)} I(X; Y)$.
probability statistics information-theory entropy
asked Nov 18 at 3:57
HOANG GIANG
53
Question
As stated in Elements of Information theory, given $p(y|x)$, the Information channel capacity formula is
$C = max_{p(x)} I(X; Y)$
where $X, Y$ are input and output symbols, $p(x)$ is the p.m.f of the distribution of $X$
Can anyone tell me the intuition of this formula ?
My understanding
$C = max_{p(x)} I(X; Y) = max_{p(x)} [H(X) - H(X|Y)]$
$H(X)$ can be understood as the expected amount of unknown information (measured in bits) of $X$. $H(X|Y)$ is the expected the expected amount of remaining unknown information (measured in bits) of $X$ given $Y$ is known. Thus, $I(X; Y)$ is the expected amount of known information (measured in bits) about $X$ given by $Y$.
Thus, $C =max_{p(x)} I(X; Y)$ gives the maximum number of bits in $X$, which can be exactly given by $Y$.
There are two problems with my understanding:
According to my understanding, for any input symbol $X$, there are only $I(X; Y)$ bits can be transmitted without error (i.e. no matter whether $H(X)$ exceeds the channel capacity $C$ or not)
In Elements of Information theory, $C$ is usually written as $C = max_{p(x)}[H(Y) - H(Y|X)]$, instead of $max_{p(X)}[H(X) - H(X|Y)]$ (i.e. my formula). I think $H(Y) - H(Y|X)$ says something about the right intuition of the formula $C = max_{p(x)} I(X; Y)$.
probability statistics information-theory entropy
probability statistics information-theory entropy
asked Nov 18 at 3:57
HOANG GIANG
53
asked Nov 18 at 3:57
HOANG GIANG
53
asked Nov 18 at 3:57
HOANG GIANG
53
asked Nov 18 at 3:57
HOANG GIANG
53
asked Nov 18 at 3:57
HOANG GIANG
53
53
1
Your question is vague and too broad. You really need to read carefully that chapter of the book, it's all explained there. "$I(X; Y)$ is the expected amount of known information (measured in bits) about $X$ given by $Y$" is more or less right... but " the maximum number of bits in X, which can be exactly given by Y" makes little sense . Capacity is not related with knowing exactly some amount of bits. In general (think of the BSC channel) you cannot do that.
– leonbloy
Nov 19 at 4:01
add a comment |
1
Your question is vague and too broad. You really need to read carefully that chapter of the book, it's all explained there. "$I(X; Y)$ is the expected amount of known information (measured in bits) about $X$ given by $Y$" is more or less right... but " the maximum number of bits in X, which can be exactly given by Y" makes little sense . Capacity is not related with knowing exactly some amount of bits. In general (think of the BSC channel) you cannot do that.
– leonbloy
Nov 19 at 4:01
1
1
Your question is vague and too broad. You really need to read carefully that chapter of the book, it's all explained there. "$I(X; Y)$ is the expected amount of known information (measured in bits) about $X$ given by $Y$" is more or less right... but " the maximum number of bits in X, which can be exactly given by Y" makes little sense . Capacity is not related with knowing exactly some amount of bits. In general (think of the BSC channel) you cannot do that.
– leonbloy
Nov 19 at 4:01
Your question is vague and too broad. You really need to read carefully that chapter of the book, it's all explained there. "$I(X; Y)$ is the expected amount of known information (measured in bits) about $X$ given by $Y$" is more or less right... but " the maximum number of bits in X, which can be exactly given by Y" makes little sense . Capacity is not related with knowing exactly some amount of bits. In general (think of the BSC channel) you cannot do that.
– leonbloy
Nov 19 at 4:01
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3003117%2finformation-theory-intuition-of-channel-capacity%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Your question is vague and too broad. You really need to read carefully that chapter of the book, it's all explained there. "$I(X; Y)$ is the expected amount of known information (measured in bits) about $X$ given by $Y$" is more or less right... but " the maximum number of bits in X, which can be exactly given by Y" makes little sense . Capacity is not related with knowing exactly some amount of bits. In general (think of the BSC channel) you cannot do that.
– leonbloy
Nov 19 at 4:01