• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Semantic information theory in 20 minutes
 

Semantic information theory in 20 minutes

on

  • 5,451 views

 

Statistics

Views

Total Views
5,451
Views on SlideShare
5,440
Embed Views
11

Actions

Likes
1
Downloads
37
Comments
2

2 Embeds 11

http://www.linkedin.com 10
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

12 of 2 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • because there are less tyrannosauruses than dogs in the current world.
    Are you sure you want to
    Your message goes here
    Processing…
  • On page 10, why is it more surprising to have the piece of knowledge that 'Rex is not a dog' ? Thank you.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • PB: Expressed message => Technical Message encoding is deterministic Replace agent
  • What’s new? How to validate? Who cares?

Semantic information theory in 20 minutes Semantic information theory in 20 minutes Presentation Transcript

  • Towards a Theory of Semantic Communication Jie Bao , Prithwish Basu, Mike Dean, Craig Partridge, Ananthram Swami, Will Leland and Jim Hendler RPI, Raytheon BBN, and ARL IEEE Network Science Workshop 2011, West Point, June 23rd, 20011
  • Outline
    • Background
    • A general semantic communication model
    • Semantic data compression (source coding)
    • Semantic reliable communication (channel coding)
    • Path ahead
  • Shannon, 1948
    • “ The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning ;... These semantic aspects of communication are irrelevant to the engineering problem .”
    Claude E. Shannon. A mathematical theory of communication. Bell System Technical Journal, 27:379-423, 625-56, 1948. message message Signal Signal
  • However, are these just bits?
    • Movie streams
    • Software codes
    • DNA sequences
    • Emails
    • Tweets
    • ……
    “ The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning;..” “ These semantic aspects of communication are irrelevant to the engineering problem ”?
  • Our Contributions
    • We develop a generic model of semantic communication, extending the classic model-theoretical work of (Carnap and Bar-Hillel 1952) ;
    • We discuss the role of semantics in reducing source redundancy , and potential approaches for lossless and lossy semantic data compression;
    • We define the notions of semantic noise, semantic channel , and obtain the semantic capacity of a channel.
  • Shannon, 1948 message message Shannon Model Signal Signal Expressed Message (e.g., commands and reports) Expressed Message From IT to SIT (Classical) Information Theory Semantic Information Theory Semantic Channel
  • A 3-level Model (adapted from Weaver) Transmitter Receiver Destination Source Physical Channel Technical message Technical Noise Intended message Expressed message Semantic Transmitter Semantic Receiver Semantic Noise Shared knowledge Local knowledge Local knowledge C: Effectiveness B: Semantic A: Technical Context, Utility, Trust etc.
  • A Semantic Communication Model Message generator World model Background Knowledge Inference Procedure Messages Sender Message interpreter World model Background Knowledge Inference Procedure Receiver W s W r K s K r I s I r {m} World M: Message Syntax Feedback (?) observations Ms Mr
  • Semantic Sources
    • Key : A semantic source tells something that is “ true ”
      • Engineering bits are neither true or false!
    • Goals : 1) more correctness (sent as “true”->received as “true”); 2) less ambiguity
  • Which message is more “surprising”? Rex is not a tyrannosaurus Rex is not a dog This slide contains animation
  • Semantics of Messages
    • If a message (an expression) is more commonly true , it contains less semantic information
      • inf (Sunny & Cold) > inf (Cold)
      • inf (Cold) > inf (Cold or Warm)
    Shannon Information How often a message appears Semantic Information How often a message is true
  • Semantics of Messages
    • Carnap & Bar-Hillel (1952) - “An outline of a theory of semantic information”
      • m(exp) = |mod(exp)| / |all models|
      • inf(exp) = - log m(exp)
    • Example
      • m(A v B) = 3/4, m(A ^ B)=1/4
      • Inf(A v B)=0.415, inf(A^B )= 2
  • Knowledge Entropy
    • Extending Carnap & Bar-Hillel (1952)
      • Models have a distribution
      • Background knowledge may present
    Weekend=2/7, Saturday=1/7
  • Semantic Information Calculator
    • http://www.cs.rpi.edu/~baojie/sit/index.php
    • Semantic Information and Coding
      • Data compressing (source coding)
      • Reliable communication (channel coding)
  • Compression with Shared Knowledge
    • Background knowledge (A->B), when shared, helps compress the source
      • Side information in the form of entailment
      • Not addressed by classical information theory
  • Lossless Message Compression
    • Intuition: by removing synonyms
      • “ pig” = “swine”
      • a->(a^b)v(b^c) = a->b
    • Theorem: There is a semantically lossless code for source X, with message entropy H >= H(X eq )
      • X eq are equivalence classes of X
    • Other lossless compressing strategies may exist
      • e.g., by using semantic ambiguity
  • Other Source Coding Strategies
    • Lossless model compression
      • E.g., using minimal models
    • Lossy message compression
      • Sometime, a semantic loss is intentional compression
        • How about having lunch at 1pm? See you soon! => “lunch@1? C U”
        • Textual description of an image
    • Ongoing Work
  • Semantic Errors and Noises
    • Examples
    • From engineering noise: “copy machine” or “coffee machine”?
    • Semantic mismatch: The source / receiver use different background knowledge or inference
    • Lost in translation : The word “Uncle” in English has no exact correspondence in Chinese.
    • Note :
    • Not all syntactical errors are semantic!
    • Neither that all semantic errors are syntactical.
    W X Y W’ Model Sent message Received message Model Goal: W’ |=x Noise
  • Semantic Channel Coding Theorem
    • Theorem: If transmission rate is smaller than C s (semantic channel capacity), semantic-error-free coding exists
    Engineering channel capacity Encoder’s semantic ambiguity Decode’s “smartness” W X Y W’ Model Sent message Received message Model Goal: W’ |=x
  • Path Ahead
    • Extensions
      • First-order languages [probabilistic logics]
      • Inconsistent KBs (misinformation) [paraconsistent logics]
      • Lossy source coding [clustering and similarity measurement]
      • Semantic mismatches
    • Applications
      • Semantic compression for RDF/OWL
      • Semantic retrieval, e.g., extending TF*IDF
  • Questions?
    • Slides: http://www.slideshare.net/baojie_iowa
    • Tech Report: http://www.cs.rpi.edu/~baojie/pub/2011-03-28_nsw_tr.pdf
    • Contact: baojie@gmail.com
    Image courtesy: http://www.addletters.com/pictures/bart-simpson-generator/900788.htm
  • backup
  • Measuring Semantic Information
    • Statistical approach : Inference may change the distribution of symbols, hence the entropy of the source.
    • Model-theoretical approach : The less “likely” a message is to be true , the more information it contains.
    • Algorithmic approach : What’s the minimal program needed to describe messages and their deductions?
    • Situation-theoretical approach : measuring the divergence of messages to “truth”.
    Our Approach
  • Shannon: Information = “surpriseness” H( tyrannosaurus ) > H(dog) Captured from: http://www.wordcount.org/main.php
  • Model Semantics
    • tyrannosaurus
    • dog
    ?? ??
  • Conditional Knowledge Entropy
    • When there is background knowledge, the set of possible worlds decreases.
  • Semantic Noise and Channel Coding “ coffee machine” “ copy machine” “ Xerox ” “ Xerox” “ copy machine” p->ff ? ? 0.9 0.1 1.0 W X Y W’ Scenario developed based on reports in http://english.visitkorea.or.kr/enu/AK/AK_EN_1_6_8_5.jsp and   http://blog.cleveland.com/metro/2011/03/identifying_photocopy_machine.html
  • Compressing by semantic Ambiguity Sunny Rain Light Rain Heavy Rain Sunny Light Rain Heavy Rain Status 0.5 0.25 0.25 0.5 0.2 0.2 0.1 (a) (b) H(X)=1.5 H(X’)=1.76