吉布斯不等式維基百科,自由的 encyclopedia 若 ∑ i = 1 n p i = ∑ i = 1 n q i = 1 {\displaystyle \sum _{i=1}^{n}p_{i}=\sum _{i=1}^{n}q_{i}=1} ,且 p i , q i ∈ ( 0 , 1 ] {\displaystyle p_{i},q_{i}\in (0,1]} ,則有: − ∑ i = 1 n p i log p i ≤ − ∑ i = 1 n p i log q i {\displaystyle -\sum _{i=1}^{n}p_{i}\log p_{i}\leq -\sum _{i=1}^{n}p_{i}\log q_{i}} ,等號成立當且僅當 p i = q i ∀ i {\displaystyle p_{i}=q_{i}\forall i} 約西亞·吉布斯 吉布斯不等式說明: 在資訊論和概率論,它能應用在法諾不等式和訊號源編碼定理的證明。 約西亞·吉布斯在19世紀提出它。 證明 吉布斯不等式等價於: 0 ≥ ∑ i = 1 n p i log q i − ∑ i = 1 n p i log p i = ∑ i = 1 n p i log ( q i / p i ) = − D K L ( P ‖ Q ) {\displaystyle 0\geq \sum _{i=1}^{n}p_{i}\log q_{i}-\sum _{i=1}^{n}p_{i}\log p_{i}=\sum _{i=1}^{n}p_{i}\log(q_{i}/p_{i})=-D_{\mathrm {KL} }(P\|Q)} (見相對熵) 證明最右的項小於或等於0的方法有幾種: 已知 ln ( x ) ≤ x − 1 {\displaystyle \ln(x)\leq x-1} ,等號成立當且僅當 x = 1 {\displaystyle x=1} : ∑ i = 1 n p i log ( q i / p i ) ≤ ∑ i = 1 n p i ( q i / p i − 1 ) = ∑ i = 1 n ( q i − p i ) = ∑ i = 1 n q i − ∑ i = 1 n p i = 0 {\displaystyle \sum _{i=1}^{n}p_{i}\log(q_{i}/p_{i})\leq \sum _{i=1}^{n}p_{i}(q_{i}/p_{i}-1)=\sum _{i=1}^{n}(q_{i}-p_{i})=\sum _{i=1}^{n}q_{i}-\sum _{i=1}^{n}p_{i}=0} 根據對數求和不等式或延森不等式: ∑ i p i log q i p i ≤ log ∑ i p i q i p i = log ∑ i q i ≤ 0 {\displaystyle \sum _{i}p_{i}\log {\frac {q_{i}}{p_{i}}}\leq \log \sum _{i}p_{i}{\frac {q_{i}}{p_{i}}}=\log \sum _{i}q_{i}\leq 0} 引理 對於n個變量的概率分佈P,其熵的最大值是: H ( p 1 , … , p n ) ≤ log n {\displaystyle H(p_{1},\ldots ,p_{n})\leq \log n}
若 ∑ i = 1 n p i = ∑ i = 1 n q i = 1 {\displaystyle \sum _{i=1}^{n}p_{i}=\sum _{i=1}^{n}q_{i}=1} ,且 p i , q i ∈ ( 0 , 1 ] {\displaystyle p_{i},q_{i}\in (0,1]} ,則有: − ∑ i = 1 n p i log p i ≤ − ∑ i = 1 n p i log q i {\displaystyle -\sum _{i=1}^{n}p_{i}\log p_{i}\leq -\sum _{i=1}^{n}p_{i}\log q_{i}} ,等號成立當且僅當 p i = q i ∀ i {\displaystyle p_{i}=q_{i}\forall i} 約西亞·吉布斯 吉布斯不等式說明: 在資訊論和概率論,它能應用在法諾不等式和訊號源編碼定理的證明。 約西亞·吉布斯在19世紀提出它。 證明 吉布斯不等式等價於: 0 ≥ ∑ i = 1 n p i log q i − ∑ i = 1 n p i log p i = ∑ i = 1 n p i log ( q i / p i ) = − D K L ( P ‖ Q ) {\displaystyle 0\geq \sum _{i=1}^{n}p_{i}\log q_{i}-\sum _{i=1}^{n}p_{i}\log p_{i}=\sum _{i=1}^{n}p_{i}\log(q_{i}/p_{i})=-D_{\mathrm {KL} }(P\|Q)} (見相對熵) 證明最右的項小於或等於0的方法有幾種: 已知 ln ( x ) ≤ x − 1 {\displaystyle \ln(x)\leq x-1} ,等號成立當且僅當 x = 1 {\displaystyle x=1} : ∑ i = 1 n p i log ( q i / p i ) ≤ ∑ i = 1 n p i ( q i / p i − 1 ) = ∑ i = 1 n ( q i − p i ) = ∑ i = 1 n q i − ∑ i = 1 n p i = 0 {\displaystyle \sum _{i=1}^{n}p_{i}\log(q_{i}/p_{i})\leq \sum _{i=1}^{n}p_{i}(q_{i}/p_{i}-1)=\sum _{i=1}^{n}(q_{i}-p_{i})=\sum _{i=1}^{n}q_{i}-\sum _{i=1}^{n}p_{i}=0} 根據對數求和不等式或延森不等式: ∑ i p i log q i p i ≤ log ∑ i p i q i p i = log ∑ i q i ≤ 0 {\displaystyle \sum _{i}p_{i}\log {\frac {q_{i}}{p_{i}}}\leq \log \sum _{i}p_{i}{\frac {q_{i}}{p_{i}}}=\log \sum _{i}q_{i}\leq 0} 引理 對於n個變量的概率分佈P,其熵的最大值是: H ( p 1 , … , p n ) ≤ log n {\displaystyle H(p_{1},\ldots ,p_{n})\leq \log n}