.

1. Ma Yihsin, 04 June
The Shannon Channel
Coding Theorem

2. Content
Shamonlhe Gding Theorem coptiy
Channel Coding Theorem
Mavk Wilde Chap 2.22 , Chap 14
Cover,
Thomas Chap 7.7 .
CQ Channel Coding Theorem
Hayashi Chap 8

3. Entropy ,
Mutual Information ( Classical )
Givenarandomuariabe Xanditsdistribution Rl 。
) ,
1 1 11
howtoquantifythe surprisenesslīnfomation )
wegetatthemomentweknowXEX ? Ansi Entropy ! !
•
Entropyof X : H (X) : = [
Blxixjlog
1
-71EX Prlx= X
)
•
londitionalentropyofx.Y.INotethati HMX ) 三
Hly
HMYKEYIHMY:D) =
前 無Blxixnmglyl

4. .
Mutual Informationof
XY
xz.chamelO-iPIMYHHIN-HMYIHMNoteth.atHMY) ⼆
Hly knowing
Themutualinformation HMYI Y
圓 ⼯ 必 炒
鼠器。
幾恐器品蕊紫 穆_
.in
iiiǘi焱 ⼼ 州 州 州州
ÒOIlxj Y ) = I M i X ) 品 , Y ) 、

5. The Channel Gding Problem Discrete Memisschannel
l DMS ) N : = plylx)
decodernn.it#O'-T-nCoderateR-f.Error
probabilitgiR-P.im#injShannon'squestioniWhatisthemaxcoderate
Ésuchthatlim pě
) *
= o ?
Isthereasequenceofhtoo
maximal.li?n)coaekiii'Dnexists?

6. Arate Ris
"
achievable
"
if 彐
{( ⼼, D
(
以及
⼦ eīm Pě
"
= 0 .
pǒnmiPrlhmlhsao
Channel Capacity CW) : ⼆
SUPIRIR isachivable}
S
hannonschannelcapacitytheorem.letN-Pylylxjisac.lassicalchannel
(W) =
max Ilx ; Y ) = : I W)
Wewantto Pxlx) Interpretation : IcxjY )
lstleamowwtofinfocho.se
prob-s-nofxwhichcanbeinferfromY.infotsiiir

7. Wecansplitthetheoremintotwopart
II) Thedīrectcodingtheorem
Fora DMCN ,
allrates R below IW)
isachievable.ern-o.c.n.se/-ofachievablerates-nInsCWRIW )I W)
(⼆) Theconversetheorem
Ifareliablesequenceofli只 )
codesexisttetherateRofthesequenceofc.desislessthan IW)
[ -
ˋ ˋ _ _ _ _
. n e t CCN) EIW)
Ìm

8. Typicakty
WeaklawoflargenumbeiletfS-IRwithfilflsnjc.no
È 三点 fcs :)
⼀⽇
yfls》
⇐> H > 0
,
比 E (0,1 ),
⼆ hothz.no
Prli 三点 fcsi) EGHSD-8.IT》 +8 ] } = 1 -
E
Nowweconsiderflsi ) =
logfy
V 870,
limprlliilog点,
⼀
Esllog 前) 1 _
< S } = 1
MN cnn.no
lnersampleentropyĀlsn) Hls)

9. Apartīcularsequencesn
isatypialsequenceifītssampleentropyÀlsnjisdosetotheentropy HIS )
letS~R.li )
5 e 5
isaf.typicalsequenceiffliilogf.tl( S ) 1 : 8
sn
The S-typicalsetfg-ISEShlsnisgtypi.at
withrespectto S } ,

10. Propertiesoftgpicalset
① Unitprobabilīty
比 ⼼ 1),
以 0
,
largeenoughn.ph ET⾏ = 1 -
E
②
Exponentiallysmallcardinalitglijl-iznHDVEEIQD.largen.ltDznl
HD-8 )
: 1 Tjkzhl
H +8 )
PMEsyiz.tn
Hcs )
、
③ Equīpartitionivsn ETja_znl.gl
xli"
器{哵
""" 打 50。 站 𠴕

11. Conditional
Typicalset.considertwovariablex.Ywherex~RM.Y~PrlYlxjthef.conditional typical set isdefinedas
Tì
"
住 Ijiihg,
花⼀
⼀
HMX ) 1 三 8
}
ftp.yongwithcardinalityznloglylwithcardinalityiznHMX)

12. Propertiesofonditionaltgpicalset
② Unitprobabilitg
UEE ⼼ 1 ) , 870 ,
Iargen Exn {Pryxn METJ
"
}} : 1 -
E
② Exponentidlysmallercardinaliy Kjlxhl 三 zn
HMX)
HjMkzhl
HMX ) +8 )
,
EHHMYJ zlrcjznlHMXM
③ Equipartition Ruxnlynlxh) 三
z-nHMDz-nlHMXHSIEPYM.lynlxyszhlHMXH )

13. Proofofthe Directpart .
Formdy : Fora DMCM Vrate RRIW)
⼆ aseqoflzhR.ru ) codeswīthmaxerrprob
*
oiputtypical
Pe以 ⼆
miPrliimj.es
0
seauenceh→ N
Overviewoftheproof
->
斷節:恐是
iiiiiif2
"
是 zn
HN )
Üx)
⼆ zh
Ilxiy

14. Thefdlowingpnofisreferedfrom Markwildésbok .
」
- _ -
randomlgselectuniformy
acode-tr-PMxjdecoderii.itÜto-_-
Trahdomnesscstept) randomyselectacodebook ?20 Whyrandomcode ?
e =
[
⼤ ⼝) ,
如 ⼼ ,
- _ -
,
Xnczj
_
codewordtrimsg I
x.li ) , _ _ _ _ .
Xnl2"3 -0
formsg 2
"
?
let Pxcx) istheprobdistthatmax IMY)
each
Xilj) ofeisiid.se/ectedfromyx)

15. lstepz )
Gding
Alie Sends
Xncmjīutothechannel 。
( Step 3 ) Decoding Bobrecēwesywfromthechanneloutput
Then Bobhastodothefollowingtests
② WhetheryhETYorrespondingtoR.ly) =
f[Pylylxj
If yna TY reporterror
② Checkwhether ⼆
someintjETYNnIfisuc.hnreport 不
If ⺺ Such 不 reporterror (I )
If 彐 肛, inandh 千 元 reporterrorg

16. Define 3 kindoferrorevents ,
Eolm ) :
yna TY
am )
iyhETY.yhqfjxhlmlczlmiiynETY.am4 miyne 壪
⼼以
Theexpectatiowofaverageerrorprob.is
成 ⼀ Ec 偷 丟 Prkolm)
UE.cm?UEzlmBWlOG-nr.2
= Ec Pri Eoll ) UEHDU 92 ( 1 ) }
EIEPrkillBThisgivetheanswerofl.tl0

17. Let IA (X ) ⼆ ⼆ ( XEA )
Focuson 9。 ( I ) :
Ecpr化 𠮨} =
Exnnyn { 1 -
Iij M ) }
= 1 -
Eyn { Iij M ) } =
BIYGTY} f
Focuson E.CI ) :
因為 typicalseq 7占 pnob 很 ⼤
ˊ
E I
E {Prlc )}} =
EnniilfjhllIyxn, ⼼ ) }
E E
… ( 喊ùii
" "
} ] E E

18. Focusonhll ) :
EIR 192 𠮨 了 = Ec,
及 Iji Iuīynmilyn) }
m⽜ 1
EEc.nl Iiynly以
密Iiiicmilyn ) }
⼆
距回去啊 怔 的 ⽔ 啊 𠮩
⼆產, 㮺,yn Pxnynlilmhjkyyh) Iyjicmilyn )
11 independent
Rnlxinpnlj )

19. Rnlxinpnlj ) Iyyh) Iijxnmilyn )⼆
蝱 㮺,yn
enrnnlnzhCHCYj-SJ.equipartition.EE以
8点iìnlxinj Iiitnyn )
Nlnnenen Fnenn
蝱⼯ ⼆ IMI -1 Hjhynki
比比州
Ei
呧以 8了
ㄝ比比州
mki
叿比川 -28了
Ml

20. 因为要證 uxil ) 的 aohievabiliy .
= i
吐⼼ 以
2811叫 l
Ifwechoose kukzh
(⼯ ⼼ ㄚ ) -
3 8 ]
EIRIGCIB } : 2
8
TheaverageerrorprobabilityTYEZE.tt :
E
'

21. 刪去
Doingexpurgatiow original 12㕧 n ) code
,
在 ⼼
throwouthalfofcodewords-tlzhlRF.nl ǗWKZE
original
expurgatiowrate.IN)-38
,
感 i rate = ⼯⼼ ㄚ ) -38 ,
ÜUK
Letg != 3 8 + i .
E
'
- 2[ +2
的
whereccanbechosenarbitraysmanaslongasnislargeenough.fincewechoose Pxlx) =
argmax IMY )
Wehavealn.IN )-
S
'
,
2 E
'
) channelode
⼆
7 IMY )
hereisfgi.EE
lo.it/andlargeenoughn.equaltoHenceIW)isanachievablerate IWI ⼀