Demystifying Information Theory

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Looking at the way our methods and systems of
communication have evolved by nature, nurture and
design so we may operate e...

I N F O R M A T I O N T H E O R Y
Origin: Claude E. Shannon in 1948 "A Mathematical Theory of Communication”
Genetic: Loss...

Communication
I N F O R M A T I O N
Actions
Gestures
Facial Expression
Body Langage
Pheromones
Grunts - Noises
Spoken Lang...

Computing Power
G R 0 W T H
216 × 162 - sphinxbazaar.com
480 × 432 - sunreed.com
E x p o n e n t i a l l y m o r e p r o c...

D a t a S t o r a g e
480 × 432 - sunreed.com
G R 0 W T H
E x p o n e n t i a l l y m o r e d a t a c r e a t e d a n d s ...

Communication
G R 0 W T H
E x p o n e n t i a l l y m o r e d e v i c e s a n d p r o d u c t s o n - l i n e
f o r e x p ...

The ~10m limit of humans
I N F O R M A T I O N
Conversation - Eye Contact
Facial Expressions - Pheromones
Body Language - ...

The ~10m limit of humans
I N F O R M A T I O N
As we physically move apart all forms
of human communication progressively ...

V e r y e a r l y i n s i g h t i n t o i n f o r m a t i o n
“ T h e m o s t i m p o r t a n t m e s s a g e i s
t h e l ...

Victory/Nelson/Trafalgar
1805 England expects every
man to do his duty
At sea the ships are rolling,
the wind is blowing, ...

S E M A P H O R E
Chappe - Napoleonic War - 1791
England expects every
man to do his duty
Lille to Paris communication
wit...

S E M A P H O R E
Surrey - Napoleonic War - 1822
England expects every
man to do his duty
Wide ranger of message sets
Slow...

S E M A P H O R E
Modern times USA and UK engineers combined the heliotrope surveying instrument (1821 Carl Friedrich Gaus...

R A I L W A Y S
Changed everything
T h e h i g h s p e e d
transportation of
people and goods
a t a n i n h u m a n
s p e ...

T E L E G R A P H
Saw a dash dot bubble
1837 & on

T E L E G R A P H
I N V E N T E D
Railways = new industry
Standardised, and very simple
signalling format used to
ensure h...

A collaboration resulting in an iconic result
Vail refined the prototype produced by Morse, who retained all the patents. ...

A collaboration resulting in an iconic result
The efficiency of the symbol and code relationship can be gaged from the abo...

Extended to other languages/character sets
All other language adaptations lead to significant inefficiencies !
M O R S G E...

A path resulting in a transformative result
C H A R A C T E R S E T S
A n e v o l u t i o n t o w a r d
f l e x i b i l i ...

The transmission, processing, extraction, and utilisation of information
I love thee to the depth and breadth and height M...

R E D U D A N C Y
Readable with missing bits
O ce m re unto th bre h, dear frie s, once m re;
Or close the wa l up ith our...

R E D U D A N C Y
Purposely shortened TXT
How r u 2day
Want 2 go 4 a drink @ 8 2nigt
Get a curry 2 ?

R E D U D A N C Y
Image Paradox and More

R E D U D A N C Y
Image Paradox and More

R E D U D A N C Y
Image Paradox and More

R E D U D A N C Y
Image Paradox and More

R E D U D A N C Y
Image Paradox and More

R E D U D A N C Y
Image Paradox and More

R E D U D A N C Y
Image Paradox and More

R E D U D A N C Y
Image Paradox and More

R E D U D A N C Y
Image Paradox and More

R E D U D A N C Y
Image Paradox and More

R E D U D A N C Y
Image Paradox and More

R E D U D A N C Y
Image Paradox and More

R E D U D A N C Y
Image Paradox and More

R E D U D A N C Y
Image Paradox and More

R E D U D A N C Y
Image Paradox and More

T E L E G R A P H
Connecting every empire

T E L E G R A P H
T i t a n i c 1 9 1 2 w i re l e s s

T E L E G R A P H
W W I I M i l i t a r y - N a va l
F re q u e n c y s e l e c t i ve f a d i n g
T h e r m a l + s p e c...

T E L E G R A P H
WWII Telex in use ~1986
D e v e l o p e d t o m e e t t h e n e e d
f o r s p e e d + m o re a c c u ra ...

M o d e r n e r a B L I N K I N G
Admiral Jeremiah Denton Blinks T-O-R-T-U-R-E using Morse Code as P.O.W.
S E M A P H O R E

M o d e r n e ra A l d i s L a m p
WW1 & WWII
Ship - Ship
Ship - Shore
Aircraft - Aircraft
Ship - Aircraft
+++
Still in us...

C o m m e r c i a l A i r l i n e r - ATC
S P E E C H R A D I O
F o r m a l i s e d o p e ra t i o n s / p ro c e d u re s...

B i r d S t r i k e E m e r g e n c y
S P E E C H R A D I O
C a p t a i n C h e s l e y S u l l e n b u r g e r
l a n d s ...

M i l i t a r y A i r c r a f t C o m b a t
S P E E C H R A D I O

M i l i t a r y H F S S B C h a n n e l
S P E E C H R A D I O

In the hierarchy to wisdom
Information theory studies the transmission,
processing, extraction & utilisation of informatio...

Describe, define & quantify
I N F O R M A T I O N
Bits
Bytes
Bauds
Signal
Noise
Coding
Storage
Decoding
Bandwidth
Processi...

Describe, define & quantify
I N F O R M A T I O N
Two States = 1; 0Bits = Binary Digits =
Bytes = Collection of Bits = Typ...

I N F O R M A T I O N
Claude Shannon Original
Claude Elwood Shannon
1916 – 2001) mathematician, electrical engineer,
and c...

E N T R 0 P Y
A measure of order
Order implies less information
Disorder implies a lot of information
Expected messages im...

The basic of what we mean
I N F O R M A T I O N
y
x

The basics of what we mean
I N F O R M A T I O N
y
x
Information = x;y;white

The basic of what we mean
I N F O R M A T I O N
y
x
Information = x;y;white; (dot colour, size, location)

The basic of what we mean
I N F O R M A T I O N
y
x
Information = x;y;white; (dots, colours, sizes, locations)

The basic of what we mean
I N F O R M A T I O N
y
x
Info= x;y;colourgrad; (shapes, colours, sizes, locations)

Fundamental relationship
I N F O R M A T I O N
Complexity
Information
Units ??
Units ??
What are the Law(s) and/or
relatio...

Channel
Copper Wire
Optical Fibre
Wireless
Receiver
Demodulation
Decoding
Error Correction
Transmitter
Coding
Modulation
D...

Dictionary definition - basic
I N F O R M A T I O N
noun [mass noun]

1 facts provided or learned about something or someo...

Describe, define & quantify
I N F O R M A T I O N
Storage
Magnetic
Optical
Thermal
Decoding
Error Correction
Coding
Modula...

Describe, define & quantify
I N F O R M A T I O N
Channel
Copper Wire
Optical Fibre
Wireless
Receiver
Demodulation
Decodin...

Of an Info Source
E N T R O P Y
All information theory analysis assumes binary (Base 2) as the foundation level and is the...

Information Source
E N T R O P Y
The entropy of a discrete random bit stream (or discrete variable X) is a measure of the
...

Information Source
E N T R O P Y
The entropy of a discrete random bit stream (or discrete variable X) is a measure of the
...

Information Source
E N T R O P Y
Plotting the entropy of a binary case with a range of bit probabilities using this formul...

Mutual Information
E N T R O P Y
A measure of the information obtained about one signal by observing another
Used to maxim...

T e l e c o m m u n i c a t i o n
I N F O R M A T I O N
1138 × 1364 - atlantic-cable.com
A p r i m a r y d r i v e r > 1 0...

How many bit/s ??
C A P I C I T Y
Channel
Copper Wire
Optical Fibre
Wireless
Receiver
Demodulation
Decoding
Error Correcti...

If it is wrong
D E M O I

L o s s y c o d i n g
D E M O 2

L o w / N o S i g n a l
D E M O 3

S/ N Analogue
D E M O 4
L o g O u t G OTO L I V E A P P S 1

Transmission/Storage upper bound
S H A N N O N L I M I T
The Shannon - Hartley Law was not originally derived as detailed ...

Transmission/Storage upper bound
S H A N N O N L I M I T
It is also assumed that the: Signal = vs(t) and Noise = vn(t) are...

Transmission/Storage upper bound
S H A N N O N L I M I T
It is also assumed that the: Signal = vs(t) and Noise = vn(t) are...

Transmission/Storage upper bound
S H A N N O N L I M I T
Rearranging further, this becomes:
If we now make M - b Bit measu...

Transmission/Storage upper bound
S H A N N O N L I M I T
And just a little more rearranging gets us to the ‘bottom line’:
...

S/N dB
BW Hz
Duration
T seconds
I = B.T log2(1 + k.S/N)
I ~ B.T.K.S/NdB
vv
Transmission/Storage upper bound
S H A N N O N ...

The same information
t r a n s m i t t e d o v e r a
channel in three different
modes using S/N, BW and
T as variable fact...

The same information
t r a n s m i t t e d o v e r a
channel in three different
modes using S/N, BW and
T as variable fact...

Transmission/Storage noise impact
E R R O R R A T E - S / N
Suppose we transmit a
binary signal represented
by two voltage...

Transmission/Storage noise impact
E R R O R R A T E - S / N
P1 = 0.5
P0 = 0.50
1
0
1 P1 = ?
P0 = ?

Transmission/Storage noise impact
E R R O R R A T E - S / N

≠
E R R O R S
Decisions

E R R O R S
Decisions

E R R O R S
Decisions

E R R O R S
Decisions

Upper bound/Limit
S H R N N N O N

A London cenral city location
R E A L I T Y W I F I
T
seconds
S + N
Power
B = 22MHz

S/ N Analogue
D E M O 4
L o g O u t G OTO L I V E A P P S 2

Any further questions
cochrane.org.uk
or thoughts ??

Share SlideShare

Facebook
Twitter
LinkedIn

Embed

Size (px)

Show related SlideShares at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 469091 views
Storytelling For The Web: Integrate... by Chiara Aliotta 375892 views
Artificial Intelligence, Data and C... by OECD Directorate ... 266102 views
How to Leverage AI to Boost Employe... by SocialHRCamp 115151 views
2024 State of Marketing Report – by... by Marius Sescu 107739 views
Everything You Need To Know About C... by Expeed Software 143146 views

Top clipped slide

View on SlideShare

1 of 93 Ad

1 of 93 Ad