Towards a theory of semantic communication

4,200 views
4,045 views

Published on

Published in: Technology, Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,200
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
28
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • International Security Assistance Force  ( ISAF )
  • PB: Expressed message => Technical Message encoding is deterministic Replace agent
  • What’s new? How to validate? Who cares?
  • For another example of information source in a broader sense, information carried by DNA are encoded using a four-letter alphabet (bases A, G, C, U). DNA's syntactical entropy can be obtained using statistical studies of bases or sequences of bases, with estimation ranging from 1.6 to 1.9 bits per base~\\cite{LY96,citeulike:769969,schmidt1997}. However, the ``semantics'' of DNA is only expressed after a complex process, producing functional gene products such as RNAs or proteins. The process is not yet fully understood, but it has been observed that variations of DNA do not necessarily result in different gene products [Citation of degenracy of DNA code], nor DNA will be expressed in the exactly same way under different conditions [citation of genetic expression variation]. If we measure the amount of information carried in a DNA based on its functional gene products, our conjecture is that it might be different from the DNA's syntactical entropy.
  • Towards a theory of semantic communication

    1. 1. Towards a Theory of Semantic Communication Jie Bao, RPI Joint work with Prithwish Basu, Mike Dean, Craig Partridge, Ananthram Swami, Will Leland and Jim Hendler
    2. 2. Outline <ul><li>Background </li></ul><ul><li>A general semantic communication model </li></ul><ul><li>Measuring semantics </li></ul><ul><li>Semantic data compression (source coding) </li></ul><ul><li>Semantic reliable communication (channel coding) </li></ul><ul><li>Path ahead </li></ul>
    3. 3. Shannon, 1948 <ul><li>“ The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning;... These semantic aspects of communication are irrelevant to the engineering problem .” </li></ul>Claude E. Shannon. A mathematical theory of communication. Bell System Technical Journal, 27:379-423, 625-56, 1948. message message Signal Signal
    4. 4. But, are these just sequences of bits? <ul><li>Movie streams </li></ul><ul><li>Software codes </li></ul><ul><li>DNA sequences </li></ul><ul><li>Emails </li></ul><ul><li>Tweets </li></ul><ul><li>…… </li></ul>“ The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning;..” “ These semantic aspects of communication are irrelevant to the engineering problem ”?
    5. 5. Between a Talent Manager & Me <ul><li>“ Are you open to discuss greener pastures”? </li></ul>“ Thanks for contacting me. However, I'm not sure if my research is related to &quot; greener pastures &quot;. I'm a computer scientist.”
    6. 6. Misunderstanding can be costly Mars Climate Orbiter (1998-1999), $125 million Expressed Pound (lb F ) Interpreted Newton (N) Image Source: Wikipedia, http://en.wikipedia.org/wiki/Mars_Climate_Orbiter#Communications_loss
    7. 7. Misunderstanding can be deadly <ul><li>Afghan National Army (ANA) to ISAF </li></ul><ul><li>“ Launch flares over the left side of the village ” </li></ul><ul><li>Received and Understood as </li></ul><ul><li>“ fire on the left side of the village ” </li></ul><ul><li>Alternative semantic coding (e.g., illuminating shell) may save lives! </li></ul>Scenario based on report from http://www.closeprotectionworld.co.uk/security-news-asia/37466-afghanistan-war-what-happens-when-war-interpreter-doesnt-know-language.html (Noisy) Battlefield Communication
    8. 8. Our Contributions <ul><li>We develop a generic model of semantic communication, extending the classic model-theoretical work of (Carnap and Bar-Hillel 1952) ; </li></ul><ul><li>We discuss the role of semantics in reducing source redundancy , and potential approaches for lossless and lossy semantic data compression; </li></ul><ul><li>We define the notions of semantic noise, semantic channel , and obtain the semantic capacity of a channel. </li></ul>
    9. 9. Outline <ul><li>Background </li></ul><ul><li>A general semantic communication model </li></ul><ul><li>Measuring Semantics </li></ul><ul><li>Semantic data compression (source coding) </li></ul><ul><li>Semantic reliable communication (channel coding) </li></ul><ul><li>Path ahead </li></ul>
    10. 10. Shannon, 1948 message message Shannon Model Signal Signal Expressed Message (e.g., commands and reports) Expressed Message From IT to SIT (Classical) Information Theory Semantic Information Theory Semantic Channel
    11. 11. A Three-level Model (Weaver) Transmitter Receiver Destination Source Physical Channel Technical message Technical Noise Intended message Expressed message Semantic Transmitter Semantic Receiver Semantic Noise Shared knowledge Local knowledge Local knowledge (effectiveness factors) C: Effectiveness B: Semantic A: Technical
    12. 12. A Semantic Communication Model Message generator World model Background Knowledge Inference Procedure Messages Sender Message interpreter World model Background Knowledge Inference Procedure Receiver W s W r K s K r I s I r {m} World M: Message Syntax Feedback (?) observations Ms Mr
    13. 13. Semantic Sources <ul><li>Key : A semantic source tells something that is “ true ” </li></ul><ul><ul><li>Engineering bits are neither true or false! </li></ul></ul><ul><li>Goal : 1) more soundness (sent as “true”->received as “true”); 2) less ambiguity </li></ul>
    14. 14. Outline <ul><li>Background </li></ul><ul><li>A general semantic communication model </li></ul><ul><li>Measuring semantics </li></ul><ul><li>Semantic data compression (source coding) </li></ul><ul><li>Semantic reliable communication (channel coding) </li></ul><ul><li>Path ahead </li></ul>
    15. 15. Measuring Semantic Information <ul><li>Basic Problem : What is the amount of “semantics” carried by a source and its messages? </li></ul>
    16. 16. Measuring Semantic Information <ul><li>Statistical approach : Inference may change the distribution of symbols, hence the entropy of the source. </li></ul><ul><li>Model-theoretical approach : The less “likely” a message is to be true , the more information it contains. </li></ul><ul><li>Algorithmic approach : What’s the minimal program needed to describe messages and their deductions? </li></ul><ul><li>Situation-theoretical approach : measuring the divergence of messages to “truth”. </li></ul>Our Approach
    17. 17. Shannon: Information = “surpriseness” H( tyrannosaurus ) > H(dog) Captured from: http://www.wordcount.org/main.php
    18. 18. Which sentence is more “surprising”? ``Rex is not a tyrannosaurus'' ``Rex is not a dog''
    19. 19. Model Semantics <ul><li>tyrannosaurus </li></ul><ul><li>dog </li></ul>?? ??
    20. 20. “ Semantics” of DNA Image courtesy: http://www.yourdictionary.com/dna http://www.pnl.gov/biology/images/protein_molecule.jpg “ Syntax” Model (“Semantics”) Gene expression
    21. 21. Stone-age Semantic Communication <ul><li>Semantic communication predates symbolic communications </li></ul>Altamira Cave Painting http://mandyking.files.wordpress.com/2011/02/altamira-cave.jpg
    22. 22. Semantics of Messages <ul><li>Messages are expressions , not just sequences of symbols </li></ul><ul><ul><li>E.g., Saturday->Weekend, Sunny & Cold </li></ul></ul><ul><li>If an expression is more commonly true , it contains less semantic information </li></ul><ul><ul><li>inf (Sunny & Cold) > inf (Cold) </li></ul></ul><ul><ul><li>inf (Cold) > inf (Cold or Warm) </li></ul></ul>
    23. 23. Semantics of Messages <ul><li>Carnap & Bar-Hillel (1952) - “An outline of a theory of semantic information” </li></ul><ul><ul><li>m(exp) = |mod(exp)| / |all models| </li></ul></ul><ul><ul><li>inf(exp) = - log m(exp) </li></ul></ul><ul><li>Example </li></ul><ul><ul><li>m(A v B) = ¾, m(A ^ B)=1/4 </li></ul></ul><ul><ul><li>Inf(A v B)=0.415, inf(A^B )= 2 </li></ul></ul>
    24. 24. Knowledge Entropy <ul><li>Extending Carnap & Bar-Hillel (1952) </li></ul><ul><ul><li>Models have a distribution </li></ul></ul><ul><ul><li>Background knowledge may present </li></ul></ul>Weekend=2/7, Saturday=1/7
    25. 25. Knowledge Entropy <ul><li>Logical prob. and knowledge entropy of Messages </li></ul><ul><li>Model entropy of an information source </li></ul>model distribution logical probability
    26. 26. Semantic Information Calculator (Demo) <ul><li>http://www.cs.rpi.edu/~baojie/sit/index.php </li></ul>
    27. 27. Outline <ul><li>Background </li></ul><ul><li>A general semantic communication model </li></ul><ul><li>Measuring Semantics </li></ul><ul><li>Semantic data compression (source coding) </li></ul><ul><li>Semantic reliable communication (channel coding) </li></ul><ul><li>Path ahead </li></ul>
    28. 28. Conditional Knowledge Entropy <ul><li>When there is background knowledge, the set of possible worlds decreases. </li></ul>
    29. 29. Model Compression with Shared Knlg <ul><li>Background knowledge (A->B), when shared, help compress the source </li></ul><ul><ul><li>Side information in the form of entailment </li></ul></ul>
    30. 30. Lossless Message Compression <ul><li>Theorem : There is a semantically lossless code for source X, with message entropy H >= H(X eq ); no such code exists for H < H(X eq ) </li></ul><ul><ul><li>X eq are equivalent classes of X </li></ul></ul><ul><li>Example: no need for coding both “pig” and “swine”, using one of them is sufficient. </li></ul><ul><li>Example 2: a->(a^b)v(b^c) = a->b </li></ul><ul><li>Sometime, the loss is intentional compression </li></ul><ul><ul><li>Textual description of an image </li></ul></ul><ul><ul><li>Abstract of a paper </li></ul></ul>
    31. 31. Other Source Coding Strategies <ul><li>Lossless model compression </li></ul><ul><ul><li>E.g., using minimal models </li></ul></ul><ul><li>Lossy message compression </li></ul><ul><ul><li>E.g., compressing based on semantic similarity </li></ul></ul><ul><li>Leave as future work </li></ul>
    32. 32. Outline <ul><li>Background </li></ul><ul><li>A general semantic communication model </li></ul><ul><li>Measuring Semantics </li></ul><ul><li>Semantic data compression (source coding) </li></ul><ul><li>Semantic reliable communication (channel coding) </li></ul><ul><li>Path ahead </li></ul>
    33. 33. Semantic Noise <ul><li>Examples </li></ul><ul><li>The meaning of a message is changed due to technical noises , e.g., from ``flare'' to ``fire'‘; </li></ul><ul><li>Semantic mismatch: The source / receiver use different background knowledge or inference (e.g., during the loss of the Mars Climate Orbiter); </li></ul><ul><li>Lost in translation : “Uncle” in English has no exact correspondence in Chinese. </li></ul>
    34. 34. Semantic Noise and Channel Coding “ coffee machine” “ copy machine” “ Xerox ” “ Xerox” “ copy machine” p->ff ? ? 0.9 0.1 1.0 W X Y W’ Scenario developed based on reports in http://english.visitkorea.or.kr/enu/AK/AK_EN_1_6_8_5.jsp and   http://blog.cleveland.com/metro/2011/03/identifying_photocopy_machine.html
    35. 35. Semantic Channel Coding Theorem <ul><li>In the simplified model, assume no semantic mismatch (K s =K r , I s =I r ) </li></ul><ul><li>Theorem 3: If transmission rate is smaller than C s (semantic channel capacity), error-free coding exists </li></ul><ul><li>Semantic channel capacity may be higher or lower than the engineering channel capacity (sup I(X;Y)) ! </li></ul><ul><ul><li>H(W|X) stands for encoder’s semantic ambiguity </li></ul></ul><ul><ul><li>avg(inf(Y)) is receiver’s “smartness” </li></ul></ul>
    36. 36. Outline <ul><li>Background </li></ul><ul><li>A general semantic communication model </li></ul><ul><li>Measuring Semantics </li></ul><ul><li>Semantic data compression (source coding) </li></ul><ul><li>Semantic reliable communication (channel coding) </li></ul><ul><li>Path ahead </li></ul>
    37. 37. Application in Coding & Validation <ul><li>Hypothesis 1: using semantics we can achieve better data compression </li></ul><ul><li>Hypothesis 2: using semantics we can achieve more reliable communication </li></ul><ul><li>Validation with comparison to non-semantic algorithms </li></ul>
    38. 38. Extensions <ul><li>Extensions & connections to other fields </li></ul><ul><ul><li>First-order languages [probabilistic logics] </li></ul></ul><ul><ul><li>Inconsistent KBs (misinformation) [paraconsistent logics] </li></ul></ul><ul><ul><li>Lossy source coding [clustering and similarity measurement] </li></ul></ul><ul><ul><li>Semantic mismatches [extending Juba & Sudan 2011] </li></ul></ul><ul><ul><li>… … </li></ul></ul>
    39. 39. Path ahead – Broad Impact <ul><ul><li>Communications (e.g., coding) </li></ul></ul><ul><ul><li>Linguistics (e.g., entropy of English) </li></ul></ul><ul><ul><li>Biology (e.g., semantics of genes) </li></ul></ul><ul><ul><li>Economics </li></ul></ul><ul><ul><li>… . </li></ul></ul><ul><ul><li>Areas wherever Shannon’s theory applies </li></ul></ul><ul><ul><li>And beyond (e.g., Semantic Web, ontology engineering) </li></ul></ul>
    40. 40. Questions? Image courtesy: http://www.addletters.com/pictures/bart-simpson-generator/900788.htm

    ×