SlideShare a Scribd company logo

Optimizing Communication to Optimize Human Behavior - LCBM

Y
Y
Yaman KumarResearch Scientist at Adobe, SUNY at Buffalo, IIIT-Delhi

Optimizing Communication to Optimize Behavior: Towards Achieving Third Level Of Shannon’s Theory Of Communication. In this talk, given at National University of Singapore, Yaman talks about how to use solve the third level proposed in Shannon's theory of communication. He uses the transfer learning property of LLMs to present initial results showing encouraging results on simulating content to elicit required behavior, simulating human behavior for a content, and human behavior explanation., showing results for both text and image generation.

Optimizing Communication to Optimize Human Behavior - LCBM

1 of 49
Download to read offline
Optimizing Communication
To Optimize Behavior
Towards Achieving Third Level Of Shannon’s Theory
Of Communication
07/Dec/2023
- Yaman K Singla,
Research Scientist, Adobe Media and Data Science
Research
Shannon’s Theory Of Communication: Highlights
Shannon’s Theory Of Communication: Highlights
The three levels build on top of each others, so to
solve Level C, we need Level A and B to be
solved first (to a large extent)
Shannon’s Theory Of Communication: Highlights
Level A: The Technical Problem
Level B: The Semantic Problem
Level C: The Effectiveness Problem ??
Simulate Human Behavior
For A Certain Content
Generate Content
To Elicit A Given Behavior
Explain Human Behavior
Use Behavior To Extract
Signals About Content
Uses data generated by internet to
build LLM
The Internet
LLMs
Principal “Scaled” Solution Characteristic Of Each Level
The problems we need to
solve at the third level
Idea: Why don’t we use LLMs transfer learning power to solve the third level?
Large Language Models As Foundation To Enable Transfer Learning
The basic idea underlying our work is to treat every text processing problem as a “text-to-text” problem, i.e. taking text as input and producing new
text as output. Crucially, the text-to-text framework allows us to directly apply the same model, objective, training procedure, and decoding process to
every task we consider. We leverage this flexibility by evaluating performance on a wide variety of English-based NLP problems, including question
answering, document summarization, and sentiment classification, to name a few.
The main utility of transfer learning is the possibility of leveraging pre-trained models in data-scarce settings
- T5, Raffel et al, 2020
Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively
trained models to perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model
on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In contrast to previous approaches, we make use of
task-aware input transformations during fine-tuning to achieve effective transfer while requiring minimal changes to the model architecture.
- Improving Language Understanding By Generative Pre-training, Radford et al, 2018
Transfer Learning Power of LLMs in Action

Recommended

Arduino Certification
Arduino CertificationArduino Certification
Arduino CertificationRobert John
 
Blockchain Technology For Social Media Training Ppt
Blockchain Technology For Social Media Training PptBlockchain Technology For Social Media Training Ppt
Blockchain Technology For Social Media Training PptSlideTeam
 
Zenpayroll Pitch Deck Template
Zenpayroll Pitch Deck TemplateZenpayroll Pitch Deck Template
Zenpayroll Pitch Deck TemplateJoseph Hsieh
 
social network analysis project twitter sentimental analysis
social network analysis project twitter sentimental analysissocial network analysis project twitter sentimental analysis
social network analysis project twitter sentimental analysisAshish Mundra
 
Bolt pitch deck: $393M Series D, $6B+ valuation
Bolt pitch deck: $393M Series D, $6B+ valuationBolt pitch deck: $393M Series D, $6B+ valuation
Bolt pitch deck: $393M Series D, $6B+ valuationPitch Decks
 
Sneak - dtac accelerate batch 7
Sneak - dtac accelerate batch 7Sneak - dtac accelerate batch 7
Sneak - dtac accelerate batch 7dtac Accelerate
 

More Related Content

Similar to Optimizing Communication to Optimize Human Behavior - LCBM

Artificial Intelligence at LinkedIn
Artificial Intelligence at LinkedInArtificial Intelligence at LinkedIn
Artificial Intelligence at LinkedInBill Liu
 
Is software engineering research addressing software engineering problems?
Is software engineering research addressing software engineering problems?Is software engineering research addressing software engineering problems?
Is software engineering research addressing software engineering problems?Gail Murphy
 
AWS re:Invent 2016: Deep Learning in Alexa (MAC202)
AWS re:Invent 2016: Deep Learning in Alexa (MAC202)AWS re:Invent 2016: Deep Learning in Alexa (MAC202)
AWS re:Invent 2016: Deep Learning in Alexa (MAC202)Amazon Web Services
 
The Concurrent Constraint Programming Research Programmes -- Redux
The Concurrent Constraint Programming Research Programmes -- ReduxThe Concurrent Constraint Programming Research Programmes -- Redux
The Concurrent Constraint Programming Research Programmes -- ReduxPierre Schaus
 
Fyp ideas
Fyp ideasFyp ideas
Fyp ideasMr SMAK
 
Simulation On The Mips Cpu Model
Simulation On The Mips Cpu ModelSimulation On The Mips Cpu Model
Simulation On The Mips Cpu ModelJessica Valentin
 
2013 10-03-semantics-meetup-s buxton-mark_logic_pub
2013 10-03-semantics-meetup-s buxton-mark_logic_pub2013 10-03-semantics-meetup-s buxton-mark_logic_pub
2013 10-03-semantics-meetup-s buxton-mark_logic_pubStephen Buxton
 
How to use transfer learning to bootstrap image classification and question a...
How to use transfer learning to bootstrap image classification and question a...How to use transfer learning to bootstrap image classification and question a...
How to use transfer learning to bootstrap image classification and question a...Wee Hyong Tok
 
OReilly AI Transfer Learning
OReilly AI Transfer LearningOReilly AI Transfer Learning
OReilly AI Transfer LearningDanielle Dean
 
The Web Science MacroScope: Mixed-methods Approach for Understanding Web Acti...
The Web Science MacroScope: Mixed-methods Approach for Understanding Web Acti...The Web Science MacroScope: Mixed-methods Approach for Understanding Web Acti...
The Web Science MacroScope: Mixed-methods Approach for Understanding Web Acti...Markus Luczak-Rösch
 
Generative AI at the edge.pdf
Generative AI at the edge.pdfGenerative AI at the edge.pdf
Generative AI at the edge.pdfQualcomm Research
 
GraphLab: Large-Scale Machine Learning on Graphs (BDT204) | AWS re:Invent 2013
GraphLab: Large-Scale Machine Learning on Graphs (BDT204) | AWS re:Invent 2013GraphLab: Large-Scale Machine Learning on Graphs (BDT204) | AWS re:Invent 2013
GraphLab: Large-Scale Machine Learning on Graphs (BDT204) | AWS re:Invent 2013Amazon Web Services
 
The importance of model fairness and interpretability in AI systems
The importance of model fairness and interpretability in AI systemsThe importance of model fairness and interpretability in AI systems
The importance of model fairness and interpretability in AI systemsFrancesca Lazzeri, PhD
 
JACIC
JACICJACIC
JACICDMacP
 
Designing Under the Agile fog of war
Designing Under the Agile fog of warDesigning Under the Agile fog of war
Designing Under the Agile fog of warJames Bryant
 
When UX (guy) Meets Operations
When UX (guy) Meets OperationsWhen UX (guy) Meets Operations
When UX (guy) Meets OperationsTim Sheiner
 
Big Data at DYNO
Big Data at DYNOBig Data at DYNO
Big Data at DYNOTu Pham
 
Sistemas de Recomendação sem Enrolação
Sistemas de Recomendação sem Enrolação Sistemas de Recomendação sem Enrolação
Sistemas de Recomendação sem Enrolação Gabriel Moreira
 
Human computer interaction research at ibm t
Human computer interaction research at ibm tHuman computer interaction research at ibm t
Human computer interaction research at ibm tJohn Thomas
 

Similar to Optimizing Communication to Optimize Human Behavior - LCBM (20)

Artificial Intelligence at LinkedIn
Artificial Intelligence at LinkedInArtificial Intelligence at LinkedIn
Artificial Intelligence at LinkedIn
 
Is software engineering research addressing software engineering problems?
Is software engineering research addressing software engineering problems?Is software engineering research addressing software engineering problems?
Is software engineering research addressing software engineering problems?
 
AWS re:Invent 2016: Deep Learning in Alexa (MAC202)
AWS re:Invent 2016: Deep Learning in Alexa (MAC202)AWS re:Invent 2016: Deep Learning in Alexa (MAC202)
AWS re:Invent 2016: Deep Learning in Alexa (MAC202)
 
The Concurrent Constraint Programming Research Programmes -- Redux
The Concurrent Constraint Programming Research Programmes -- ReduxThe Concurrent Constraint Programming Research Programmes -- Redux
The Concurrent Constraint Programming Research Programmes -- Redux
 
Fyp ideas
Fyp ideasFyp ideas
Fyp ideas
 
Simulation On The Mips Cpu Model
Simulation On The Mips Cpu ModelSimulation On The Mips Cpu Model
Simulation On The Mips Cpu Model
 
2013 10-03-semantics-meetup-s buxton-mark_logic_pub
2013 10-03-semantics-meetup-s buxton-mark_logic_pub2013 10-03-semantics-meetup-s buxton-mark_logic_pub
2013 10-03-semantics-meetup-s buxton-mark_logic_pub
 
How to use transfer learning to bootstrap image classification and question a...
How to use transfer learning to bootstrap image classification and question a...How to use transfer learning to bootstrap image classification and question a...
How to use transfer learning to bootstrap image classification and question a...
 
OReilly AI Transfer Learning
OReilly AI Transfer LearningOReilly AI Transfer Learning
OReilly AI Transfer Learning
 
The Web Science MacroScope: Mixed-methods Approach for Understanding Web Acti...
The Web Science MacroScope: Mixed-methods Approach for Understanding Web Acti...The Web Science MacroScope: Mixed-methods Approach for Understanding Web Acti...
The Web Science MacroScope: Mixed-methods Approach for Understanding Web Acti...
 
Generative AI at the edge.pdf
Generative AI at the edge.pdfGenerative AI at the edge.pdf
Generative AI at the edge.pdf
 
GraphLab: Large-Scale Machine Learning on Graphs (BDT204) | AWS re:Invent 2013
GraphLab: Large-Scale Machine Learning on Graphs (BDT204) | AWS re:Invent 2013GraphLab: Large-Scale Machine Learning on Graphs (BDT204) | AWS re:Invent 2013
GraphLab: Large-Scale Machine Learning on Graphs (BDT204) | AWS re:Invent 2013
 
The importance of model fairness and interpretability in AI systems
The importance of model fairness and interpretability in AI systemsThe importance of model fairness and interpretability in AI systems
The importance of model fairness and interpretability in AI systems
 
MATLAB Essay
MATLAB EssayMATLAB Essay
MATLAB Essay
 
JACIC
JACICJACIC
JACIC
 
Designing Under the Agile fog of war
Designing Under the Agile fog of warDesigning Under the Agile fog of war
Designing Under the Agile fog of war
 
When UX (guy) Meets Operations
When UX (guy) Meets OperationsWhen UX (guy) Meets Operations
When UX (guy) Meets Operations
 
Big Data at DYNO
Big Data at DYNOBig Data at DYNO
Big Data at DYNO
 
Sistemas de Recomendação sem Enrolação
Sistemas de Recomendação sem Enrolação Sistemas de Recomendação sem Enrolação
Sistemas de Recomendação sem Enrolação
 
Human computer interaction research at ibm t
Human computer interaction research at ibm tHuman computer interaction research at ibm t
Human computer interaction research at ibm t
 

Recently uploaded

AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...ISPMAIndia
 
Introduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVAIntroduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVARobert McDermott
 
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaBuilding Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaISPMAIndia
 
The Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolThe Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolProduct School
 
Q1 Memory Fabric Forum: Building Fast and Secure Chips with CXL IP
Q1 Memory Fabric Forum: Building Fast and Secure Chips with CXL IPQ1 Memory Fabric Forum: Building Fast and Secure Chips with CXL IP
Q1 Memory Fabric Forum: Building Fast and Secure Chips with CXL IPMemory Fabric Forum
 
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
Zi-Stick UBS Dongle ZIgbee from  Aeotec manualZi-Stick UBS Dongle ZIgbee from  Aeotec manual
Zi-Stick UBS Dongle ZIgbee from Aeotec manualDomotica daVinci
 
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERNRonnelBaroc
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologySafe Software
 
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre..."Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...shaiyuvasv
 
Quinto Z-Wave Heltun_HE-RS01_User_Manual_B9AH.pdf
Quinto Z-Wave Heltun_HE-RS01_User_Manual_B9AH.pdfQuinto Z-Wave Heltun_HE-RS01_User_Manual_B9AH.pdf
Quinto Z-Wave Heltun_HE-RS01_User_Manual_B9AH.pdfDomotica daVinci
 
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, GoogleISPMAIndia
 
Heltun_HE-RS01_User_Manual_B9AH.pdf
Heltun_HE-RS01_User_Manual_B9AH.pdfHeltun_HE-RS01_User_Manual_B9AH.pdf
Heltun_HE-RS01_User_Manual_B9AH.pdfMarielaL5
 
5 Things You Shouldn’t Do at Salesforce World Tour Sydney 2024!
5 Things You Shouldn’t Do at Salesforce World Tour Sydney 2024!5 Things You Shouldn’t Do at Salesforce World Tour Sydney 2024!
5 Things You Shouldn’t Do at Salesforce World Tour Sydney 2024!XfilesPro
 
AWS reInvent 2023 recaps from Chicago AWS user group
AWS reInvent 2023 recaps from Chicago AWS user groupAWS reInvent 2023 recaps from Chicago AWS user group
AWS reInvent 2023 recaps from Chicago AWS user groupAWS Chicago
 
OTel Orientation_ How to Train Teams (OTel in Practice).pdf
OTel Orientation_ How to Train Teams (OTel in Practice).pdfOTel Orientation_ How to Train Teams (OTel in Practice).pdf
OTel Orientation_ How to Train Teams (OTel in Practice).pdfPaige Cruz
 
DNA LIGASE BIOTECHNOLOGY BIOLOGY STUDY OF LIFE
DNA LIGASE BIOTECHNOLOGY BIOLOGY STUDY OF LIFEDNA LIGASE BIOTECHNOLOGY BIOLOGY STUDY OF LIFE
DNA LIGASE BIOTECHNOLOGY BIOLOGY STUDY OF LIFEandreiandasan
 
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdfLLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdfThomas Poetter
 
Microsoft Azure News - Feb 2024
Microsoft Azure News - Feb 2024Microsoft Azure News - Feb 2024
Microsoft Azure News - Feb 2024Daniel Toomey
 
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...Adrian Sanabria
 

Recently uploaded (20)

AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
 
Introduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVAIntroduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVA
 
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaBuilding Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
 
The Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolThe Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product School
 
Q1 Memory Fabric Forum: Building Fast and Secure Chips with CXL IP
Q1 Memory Fabric Forum: Building Fast and Secure Chips with CXL IPQ1 Memory Fabric Forum: Building Fast and Secure Chips with CXL IP
Q1 Memory Fabric Forum: Building Fast and Secure Chips with CXL IP
 
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
Zi-Stick UBS Dongle ZIgbee from  Aeotec manualZi-Stick UBS Dongle ZIgbee from  Aeotec manual
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
 
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI Technology
 
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre..."Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
 
Quinto Z-Wave Heltun_HE-RS01_User_Manual_B9AH.pdf
Quinto Z-Wave Heltun_HE-RS01_User_Manual_B9AH.pdfQuinto Z-Wave Heltun_HE-RS01_User_Manual_B9AH.pdf
Quinto Z-Wave Heltun_HE-RS01_User_Manual_B9AH.pdf
 
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
 
Heltun_HE-RS01_User_Manual_B9AH.pdf
Heltun_HE-RS01_User_Manual_B9AH.pdfHeltun_HE-RS01_User_Manual_B9AH.pdf
Heltun_HE-RS01_User_Manual_B9AH.pdf
 
5 Things You Shouldn’t Do at Salesforce World Tour Sydney 2024!
5 Things You Shouldn’t Do at Salesforce World Tour Sydney 2024!5 Things You Shouldn’t Do at Salesforce World Tour Sydney 2024!
5 Things You Shouldn’t Do at Salesforce World Tour Sydney 2024!
 
AWS reInvent 2023 recaps from Chicago AWS user group
AWS reInvent 2023 recaps from Chicago AWS user groupAWS reInvent 2023 recaps from Chicago AWS user group
AWS reInvent 2023 recaps from Chicago AWS user group
 
OTel Orientation_ How to Train Teams (OTel in Practice).pdf
OTel Orientation_ How to Train Teams (OTel in Practice).pdfOTel Orientation_ How to Train Teams (OTel in Practice).pdf
OTel Orientation_ How to Train Teams (OTel in Practice).pdf
 
DNA LIGASE BIOTECHNOLOGY BIOLOGY STUDY OF LIFE
DNA LIGASE BIOTECHNOLOGY BIOLOGY STUDY OF LIFEDNA LIGASE BIOTECHNOLOGY BIOLOGY STUDY OF LIFE
DNA LIGASE BIOTECHNOLOGY BIOLOGY STUDY OF LIFE
 
5 Tech Trend to Notice in ESG Landscape- 47Billion
5 Tech Trend to Notice in ESG Landscape- 47Billion5 Tech Trend to Notice in ESG Landscape- 47Billion
5 Tech Trend to Notice in ESG Landscape- 47Billion
 
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdfLLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
 
Microsoft Azure News - Feb 2024
Microsoft Azure News - Feb 2024Microsoft Azure News - Feb 2024
Microsoft Azure News - Feb 2024
 
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...
 

Optimizing Communication to Optimize Human Behavior - LCBM

  • 1. Optimizing Communication To Optimize Behavior Towards Achieving Third Level Of Shannon’s Theory Of Communication 07/Dec/2023 - Yaman K Singla, Research Scientist, Adobe Media and Data Science Research
  • 2. Shannon’s Theory Of Communication: Highlights
  • 3. Shannon’s Theory Of Communication: Highlights The three levels build on top of each others, so to solve Level C, we need Level A and B to be solved first (to a large extent)
  • 4. Shannon’s Theory Of Communication: Highlights Level A: The Technical Problem Level B: The Semantic Problem Level C: The Effectiveness Problem ?? Simulate Human Behavior For A Certain Content Generate Content To Elicit A Given Behavior Explain Human Behavior Use Behavior To Extract Signals About Content Uses data generated by internet to build LLM The Internet LLMs Principal “Scaled” Solution Characteristic Of Each Level The problems we need to solve at the third level Idea: Why don’t we use LLMs transfer learning power to solve the third level?
  • 5. Large Language Models As Foundation To Enable Transfer Learning The basic idea underlying our work is to treat every text processing problem as a “text-to-text” problem, i.e. taking text as input and producing new text as output. Crucially, the text-to-text framework allows us to directly apply the same model, objective, training procedure, and decoding process to every task we consider. We leverage this flexibility by evaluating performance on a wide variety of English-based NLP problems, including question answering, document summarization, and sentiment classification, to name a few. The main utility of transfer learning is the possibility of leveraging pre-trained models in data-scarce settings - T5, Raffel et al, 2020 Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve effective transfer while requiring minimal changes to the model architecture. - Improving Language Understanding By Generative Pre-training, Radford et al, 2018
  • 6. Transfer Learning Power of LLMs in Action
  • 7. Transfer Learning Power of LLMs in Action
  • 8. Transfer Learning Power of LLMs in Action
  • 9. Human Communication Process And How LLMs See Only Half The Picture? Communicator Message Channel Receiver Effect Nike Subscribers: 754K { } Youtube { } Nike Subscribers/ General Youtube Audience { } likes: 15k views: 60M scene replays: [73,85,99,...,24] { } description: a sunny football ground objects: a football, players, grass asr: even after hours of practice, { } Common Crawl (Web-scale dump of webpages) Github (Web-scale dump of code) Books Scientific Papers Reddit Discussion Forums such as StackExchange, hackernews, etc ~200 TBs ~3 TBs ~200 TBs ~50 TBs ~1 TBs ~1 TBs Removed as noise while preparing data for LLMs
  • 10. Human Communication Process And How LLMs See Only Half The Picture? Communicator Message Channel Receiver Effect Nike Subscribers: 754K { } Youtube { } Nike Subscribers/ General Youtube Audience { } likes: 15k views: 60M scene replays: [73,85,99,...,24] { } description: a sunny football ground objects: a football, players, grass asr: even after hours of practice, { } Invariants of Communication: 1. At any instant of time, there is always one sender and multiple receivers 2. Behavior is always expressed by an act. 1 Message n receivers producing m acts at a time t Total Data Per Message = n * m * t tokens Total Data = 1 Message (2k tokens) - Measurement of Social Change, Richard Meier, 1959 The total tokens in the other (unseen) half is much larger than the total number of content tokens
  • 11. Human Communication Process And How We Are Seeing Only Half The Picture? Communicator Message Channel Receiver Effect Nike Subscribers: 754K { } Youtube { } Nike Subscribers/ General Youtube Audience { } likes: 15k views: 60M scene replays: [73,85,99,...,24] { } description: a sunny football ground objects: a football, players, grass asr: even after hours of practice, { } Common Crawl (Web-scale dump of webpages) Github (Web-scale dump of code) Books Scientific Papers Reddit Discussion Forums such as StackExchange, hackernews, etc ~200 TBs ~3 TBs ~200 TBs ~50 TBs ~1 TBs ~1 TBs Timestamp, Person, Effect Timestamp, Person, Effect Timestamp, Person, Effect Timestamp, Community, Upvotes, Comments Timestamp, Community, Upvotes, Comments
  • 12. Email Creation Email Scheduling Email Sent • Like • Buy • Share Customer Response Adobe Campaign How Can We See The Complete Picture? Adobe Campaign A Sample of How We Record The Complete Picture In Enterprise Products
  • 13. Behavior Data as the Source for the Next Trillion Tokens “Finally, we found that when we train high dimensional word vectors on a large amount of data, the resulting vectors can be used to answer very subtle semantic relationships between words, such as a city and the country it belongs to, e.g. France is to Paris as Germany is to Berlin.” - Word2Vec, Mikolov et al, 2013 “Datasets for language models have rapidly expanded, culminating in the Common Crawl dataset2 [RSR+19] constituting nearly a trillion words. This size of dataset is sufficient to train our largest models without ever updating on the same sequence twice.” - GPT-3, Brown et al, 2020 “…unsupervised pre-training for NLP is particularly attractive because unlabeled text data is available en masse thanks to the Internet—for example, the Common Crawl project2 produces about 20TB of text data…. This is a natural fit for neural networks…, i.e. it is often possible to achieve better performance simply by training a larger model on a larger data set” - T5, Raffel et al, 2020 “Compute-Optimal Scaling: Recently, Hoffmann et al. (2022) showed that data size is at least as important as model size. We validate this study for larger amounts of compute and similarly find that data and model size should be scaled roughly 1:1 to achieve the best performance for a given amount of training compute (as opposed to past trends, which scaled the model 3× faster than the dataset).” - Palm v2, Google, 2023 1.6 2 21 300 780 2000 3600 0 500 1000 1500 2000 2500 3000 3500 4000 Word2vec GPT-1 GPT-2 GPT-3 PaLM Llama-2 PaLM-2 Definition of "Large" Tokens in Billions Evolution of the Definition of “large”
  • 14. What Next? Where Would The Next Trillion Tokens Come From? While The Architectures Are Scaling Bigger, The Limit Is The Data The Total Content Produced By Humanity Is ~10T Answer: Behavior Tokens …. They are much larger in number than content tokens
  • 15. How Can We See The Complete Picture? • Channel Name • Subscriber count • Video/Email/Image • Captions • Customer Attributes • Likes • Views • Impressions title: Nike Football posted-by: Nike subscribers: 754K description: a sunny football ground objects: a football, players, grass asr: even after hours of practice, likes: 15k views: 60M replays: 73 audience: general YouTube Audience / Nike audience Communicator Message Channel Receiver Effect Nike Subscribers: 754K { } Youtube { } Nike Subscribers/ General Youtube Audience { } likes: 15k views: 60M scene replays: [73,85,99,...,24] { } description: a sunny football ground objects: a football, players, grass asr: even after hours of practice, { } Khandelwal, Ashmit, et al. "Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior." arXiv preprint arXiv:2309.00359 (2023). An Example of the Complete Picture
  • 16. How Can We See The Complete Picture? An Example of the Complete Picture
  • 17. Large Content And Behavior Models (LCBM) How Can We See The Full Picture? Khandelwal, Ashmit, et al. "Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior." arXiv preprint arXiv:2309.00359 (2023).
  • 18. What Happens If We See The Full Picture? Khandelwal, Ashmit, et al. "Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior." arXiv preprint arXiv:2309.00359 (2023).
  • 19. What Happens If We See The Full Picture? LCBM Results vs 15x Larger SOTA LLMs – Behavior Simulation Khandelwal, Ashmit, et al. "Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior." arXiv preprint arXiv:2309.00359 (2023).
  • 20. LCBM Shows Signs of Behavior Domain Adaptation LCBM Results vs 15x Larger SOTA LLMs Khandelwal, Ashmit, et al. "Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior." arXiv preprint arXiv:2309.00359 (2023).
  • 21. LCBM Shows Signs of Behavior Domain Adaptation LCBM Results vs 15x Larger SOTA LLMs Video Title: Robber hangs from the 23rd-floor window to escape the police in China Posted by: South China Morning Post Link: youtu.be/P7iDWZiT3Kg Comments from Video: (not shown to LLMs/LCBM) Commenter: I feel bad for the dude. Nobody in the right mind robs a local shop.. To then hang out a window. I wonder what he is going through. Commenter: Kinda feel sorry for him when he cried, he must have been in a bad situation to do what he did but it does not justify his actions. Commenter: Aw, i felt his desperation. sorry young man but you did wrong. :( Question: Would the average sentiment of the comments of the audience be positive, neutral, or negative? Explain why. LCBM: The average sentiment of the comments of the audience watching the video would be negative. This is because the video shows a man jumping out of a high-rise building to escape the police. The audience is likely to feel sympathy and sadness towards the man's situation. Video Title: The new Porsche 718 Spyder. Perfectly irrational. Posted by: Porsche Link: youtu.be/573YaSAP5H0 Comments from Video: (not shown to LLMs/LCBM) Commenter: Dream car. Mid engine, natural aspiration, b6, rwd, manual, lightweight, open top Commenter: All the makings of a epic sports car Commenter: This car is perfect. I’m in love. New dream car, no questions asked! Question: Given the video, would the average audience sentiment be positive, neutral, or negative? Explain why. LCBM: Positive average sentiment of the audience. The video features a luxury car driving along a scenic coastal road, which is would evoke a sense of awe and admiration in the viewers. Additionally, the Porsche 718 in the video is an iconic sports car that is widely recognized and respected in the automotive community, contributing to the positive sentiment. Commenter: I add my vote and wishes to yours ... a great car I’d love to own and pilot Vicuna: The audience would likely have a mix of positive and negative sentiment towards the video. Viewers may feel sympathetic towards the person trying to escape and praise their bravery. GPT-3.5: The video depicts a dangerous and potentially life-threatening situation, which may cause some viewers to feel uneasy or uncomfortable. However, others may find the events exciting or thrilling to watch. Vicuna: The average sentiment of the comments would be positive. The video showcases a beautiful scenic drive through the mountains, which is likely to evoke a sense of awe and enjoyment in the viewers. Additionally, the presence of a luxury car in the video may also contribute to a positive sentiment, as it may be seen as a symbol of success and wealth. GPT-3.5: The sentiment of the audience watching the video is positive because the video showcases a new sports car, which is exciting and desirable to car enthusiasts. Khandelwal, Ashmit, et al. "Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior." arXiv preprint arXiv:2309.00359 (2023).
  • 22. Training LCBM For Long Term Behavior SI, Harini, et al. "Long-Term Memorability On Advertisements." arXiv preprint arXiv:2309.00378 (2023).
  • 23. Training LCBM For Long Term Behavior SI, Harini, et al. "Long-Term Memorability On Advertisements." arXiv preprint arXiv:2309.00378 (2023).
  • 24. Training LCBM For Long Term Behavior SI, Harini, et al. "Long-Term Memorability On Advertisements." arXiv preprint arXiv:2309.00378 (2023).
  • 25. Training LCBM For Long Term Behavior SI, Harini, et al. "Long-Term Memorability On Advertisements." arXiv preprint arXiv:2309.00378 (2023).
  • 26. Connecting It Back To Shannon’s Theory Of Communication Level A: The Technical Problem Level B: The Semantic Problem Level C: The Effectiveness Problem ? Simulate Human Behavior For A Certain Content (For the image domain) Generate Content To Elicit A Given Behavior Explain Human Behavior Use Behavior To Extract Signals About Content
  • 27. How Can We Optimize Image Generation With Behavior? Khurana, Varun, et al. "Behavior Optimized Image Generation." arXiv preprint arXiv:2311.10995 (2023). Speaker: Dolce & Gabbana Followers: 5.3 mi Receivers: Nike Subscribers/ General Twitter Audience Receiver Ef f ect: Likes: 5k Views: 38k Retweets: 1k Channel: Twitter Message: Tweet DEVOTION Uncover the tantalizing delectable notes of the new bright gourmand fragrance by #DolceGabbana. Discover more at https://bit.ly/DGDevotion_fragrance_ #DGBeauty #DGDevotion #MadeinItaly How to create not just better-looking images but also “better-performing” images?
  • 28. How Can We Optimize Image Generation With Behavior? Behavior Optimized Image Generation (BOIG) Using Twitter As The Resource Khurana, Varun, et al. "Behavior Optimized Image Generation." arXiv preprint arXiv:2311.10995 (2023).
  • 29. How Can We Optimize Image Generation With Behavior? Step-1: Train A Language Model Which Can Act As A Reward Model Khurana, Varun, et al. "Behavior Optimized Image Generation." arXiv preprint arXiv:2311.10995 (2023). BoigSD BoigLLM Prompt for BoigLLM "You are a smart model. I am giving giving you some data regarding an image tweeted by a company - (1) company (2) twi er username (3) tweet text (4) cap ons and keywords (5) image resolu on i.e. (width, height) (6) tweet date (7) number of likes received on the tweet (8) KPI label of the tweet i.e. "high" or "low"... Caption: A living room with a couch, coffee table, and a painting on the wall Keywords: living room, couch, coffee table, painting, wall, home decor, ... Image being progressively optimized Verbalization of BoigSD generated Colours Extractor Object Detector Tones Extractor BoigLLM generates Decoding Graph of image conditioned on behavior Reward = sum of conditional probability of generati of image verbalization tokens COLOURS Dark Blue: 40% Olive Green: 35% Brown: 25% TONES Warm: 10% Neutral: 85% Cool: 5% OBJECTS Armchair: [1, 100, 15, 70] Side table: [300, 420, 150, 240] Lamp: [400, 450, 30, 95] Image Verbalization <BOS> 0.5 Colors 0.4 Dark Blue 0.7 Orange Black 0.7 40% 0.4 35% 0.5 25% 0.2 45% 0.3 22% 0.5 Green 0.6 Olive Green 0.8 Dark Green 0.2 Brown 0.3 Red 20% 35% 18% 10% 27% 0.4 0.1 0.3 0.6 0.8 Marketer Provided Input For Behavior-Conditioned Image Generation Marketer: Honeywell Tweet Text: “In honor of Energy Efficiency Day, let us help you identify...” Required KPI (Likes): 465 Date: 2022-10-05 Caption: A living room with a couch, coffee table, and a painting on the wall Keywords: living room, couch, coffee table, painting, wall, home decor, ... Image perceptual models BoigLLM logits converted to probabilities Reward Computation on BoigLLM Decoding Grap BoigSD verbalization path Curate Prompt for BoigLLM from Marketer provided Input Captions and Keywords from Marketer's Input fed to BoigSD BoigSD (6) tweet date (7) number of likes received on the tweet (8) KPI label of the tweet i.e. "high" or "low"... Caption: A living room with a couch, coffee table, and a painting on the wall Keywords: living room, couch, coffee table, painting, wall, home decor, ... Image being progressively optimized for achieving high KPI Verbalization of BoigSD generated image computed using image perceptual models Colours Extractor Object Detector Tones Extractor BoigLLM generates Decoding Graph of image conditioned on behavior Rewar COLOURS Dark Blue: 40% Olive Green: 35% Brown: 25% TONES Warm: 10% Neutral: 85% Cool: 5% OBJECTS Armchair: [1, 100, 15, 70] Side table: [300, 420, 150, 240] Lamp: [400, 450, 30, 95] Image Verbalization <BOS> C coffee table, and a painting on the wall Keywords: living room, couch, coffee table, painting, wall, home decor, ... Image perceptual models BoigS verbalizati Curate Prompt for BoigLLM from Marketer provided Input Captions and Keywords from Marketer's Input fed to BoigSD
  • 30. Behavior Optimized Image Generation Results Khurana, Varun, et al. "Behavior Optimized Image Generation." arXiv preprint arXiv:2311.10995 (2023).
  • 31. How Can We Optimize Image Generation With Behavior? Step-2: Use the LLM generated probabilities to compute reward Khurana, Varun, et al. "Behavior Optimized Image Generation." arXiv preprint arXiv:2311.10995 (2023). Policy Gradient BoigSD BoigLLM Prompt for BoigLLM "You are a smart model. I am giving giving you some data regarding an image tweeted by a company - (1) company (2) twi er username (3) tweet text (4) cap ons and keywords (5) image resolu on i.e. (width, height) (6) tweet date (7) number of likes received on the tweet (8) KPI label of the tweet i.e. "high" or "low"... Caption: A living room with a couch, coffee table, and a painting on the wall Keywords: living room, couch, coffee table, painting, wall, home decor, ... Image being progressively optimized for achieving high KPI Verbalization of BoigSD generated image computed using image perceptual models Colours Extractor Object Detector Tones Extractor BoigLLM generates Decoding Graph of image conditioned on behavior Reward = sum of conditional probability of generation of image verbalization tokens COLOURS Dark Blue: 40% Olive Green: 35% Brown: 25% TONES Warm: 10% Neutral: 85% Cool: 5% OBJECTS Armchair: [1, 100, 15, 70] Side table: [300, 420, 150, 240] Lamp: [400, 450, 30, 95] Image Verbalization <BOS> 0.5 Colors 0.4 Dark Blue 0.7 Orange Black 0.7 40% 0.4 35% 0.5 25% 0.2 45% 0.3 22% 0.5 Green 0.6 Olive Green 0.8 Dark Green 0.2 Brown 0.3 Red 20% 35% 18% 10% 27% 0.4 0.1 0.3 0.6 0.8 Marketer Provided Input For Behavior-Conditioned Image Generation Marketer: Honeywell Tweet Text: “In honor of Energy Efficiency Day, let us help you identify...” Required KPI (Likes): 465 Date: 2022-10-05 Caption: A living room with a couch, coffee table, and a painting on the wall Keywords: living room, couch, coffee table, painting, wall, home decor, ... Image perceptual models BoigLLM logits converted to probabilities Best Path Path used to compute reward Reward Computation on BoigLLM Decoding Graph BoigSD verbalization path Curate Prompt for BoigLLM from Marketer provided Input Captions and Keywords from Marketer's Input fed to BoigSD Policy Gradient BoigSD BoigLLM Prompt for BoigLLM "You are a smart model. I am giving giving you some data regarding an image tweeted by a company - (1) company (2) twi er username (3) tweet text (4) cap ons and keywords (5) image resolu on i.e. (width, height) (6) tweet date (7) number of likes received on the tweet (8) KPI label of the tweet i.e. "high" or "low"... Caption: A living room with a couch, coffee table, and a painting on the wall Keywords: living room, couch, coffee table, painting, wall, home decor, ... Image being progressively optimized for achieving high KPI Verbalization of BoigSD generated image computed using image perceptual models Colours Extractor Object Detector Tones Extractor BoigLLM generates Decoding Graph of image conditioned on behavior Reward = sum of conditional probability of generation of image verbalization tokens COLOURS Dark Blue: 40% Olive Green: 35% Brown: 25% TONES Warm: 10% Neutral: 85% Cool: 5% OBJECTS Armchair: [1, 100, 15, 70] Side table: [300, 420, 150, 240] Lamp: [400, 450, 30, 95] Image Verbalization <BOS> 0.5 Colors 0.4 Dark Blue 0.7 Orange Black 0.7 40% 0.4 35% 0.5 25% 0.2 45% 0.3 22% 0.5 Green 0.6 Olive Green 0.8 Dark Green 0.2 Brown 0.3 Red 20% 35% 18% 10% 27% 0.4 0.1 0.3 0.6 0.8 Marketer Provided Input For Behavior-Conditioned Image Generation Marketer: Honeywell Tweet Text: “In honor of Energy Efficiency Day, let us help you identify...” Required KPI (Likes): 465 Date: 2022-10-05 Caption: A living room with a couch, coffee table, and a painting on the wall Keywords: living room, couch, coffee table, painting, wall, home decor, ... Image perceptual models BoigLLM logits converted to probabilities Best Path Path used to compute reward Reward Computation on BoigLLM Decoding Graph BoigSD verbalization path Curate Prompt for BoigLLM from Marketer provided Input Captions and Keywords from Marketer's Input fed to BoigSD
  • 32. How Can We Optimize Image Generation With Behavior? Step-3: Feed them back to Stable Diffusion as a reward signal Khurana, Varun, et al. "Behavior Optimized Image Generation." arXiv preprint arXiv:2311.10995 (2023). Policy Gradient BoigSD BoigLLM Prompt for BoigLLM "You are a smart model. I am giving giving you some data regarding an image tweeted by a company - (1) company (2) twi er username (3) tweet text (4) cap ons and keywords (5) image resolu on i.e. (width, height) (6) tweet date (7) number of likes received on the tweet (8) KPI label of the tweet i.e. "high" or "low"... Caption: A living room with a couch, coffee table, and a painting on the wall Keywords: living room, couch, coffee table, painting, wall, home decor, ... Image being progressively optimized for achieving high KPI Verbalization of BoigSD generated image computed using image perceptual models Colours Extractor Object Detector Tones Extractor BoigLLM generates Decoding Graph of image conditioned on behavior Reward = sum of conditional probability of generation of image verbalization tokens COLOURS Dark Blue: 40% Olive Green: 35% Brown: 25% TONES Warm: 10% Neutral: 85% Cool: 5% OBJECTS Armchair: [1, 100, 15, 70] Side table: [300, 420, 150, 240] Lamp: [400, 450, 30, 95] Image Verbalization <BOS> 0.5 Colors 0.4 Dark Blue 0.7 Orange Black 0.7 40% 0.4 35% 0.5 25% 0.2 45% 0.3 22% 0.5 Green 0.6 Olive Green 0.8 Dark Green 0.2 Brown 0.3 Red 20% 35% 18% 10% 27% 0.4 0.1 0.3 0.6 0.8 Marketer Provided Input For Behavior-Conditioned Image Generation Marketer: Honeywell Tweet Text: “In honor of Energy Efficiency Day, let us help you identify...” Required KPI (Likes): 465 Date: 2022-10-05 Caption: A living room with a couch, coffee table, and a painting on the wall Keywords: living room, couch, coffee table, painting, wall, home decor, ... Image perceptual models BoigLLM logits converted to probabilities Best Path Path used to compute reward Reward Computation on BoigLLM Decoding Graph BoigSD verbalization path Curate Prompt for BoigLLM from Marketer provided Input Captions and Keywords from Marketer's Input fed to BoigSD
  • 33. Behavior Optimized Image Generation Results Khurana, Varun, et al. "Behavior Optimized Image Generation." arXiv preprint arXiv:2311.10995 (2023). Notice the subtle persuasion strategies in these images; objects are same, subtle marketing messages get introduced
  • 34. Behavior Optimized Image Generation Results Khurana, Varun, et al. "Behavior Optimized Image Generation." arXiv preprint arXiv:2311.10995 (2023).
  • 35. Behavior Optimized Image Generation Results Khurana, Varun, et al. "Behavior Optimized Image Generation." arXiv preprint arXiv:2311.10995 (2023).
  • 36. Connecting It Back To Shannon’s Theory Of Communication Level A: The Technical Problem Level B: The Semantic Problem Level C: The Effectiveness Problem ? Simulate Human Behavior For A Certain Content Generate Content To Elicit A Given Behavior Explain Human Behavior Use Behavior To Extract Signals About Content
  • 37. Can Behavior Also Lead To Better Content Understanding? Augmenting Scanpaths in NLP models Scanpaths are temporal sequences of eye fixation positions with durations. Linguistically important words often have a higher fixation duration and revisit. Regressions (word revisits) also appear in the examples. Khurana, Varun, et al. "Synthesizing Human Gaze Feedback for Improved NLP Performance." Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023. Communicator Message Channel Receiver Effect Nike Subscribers: 754K { } Youtube { } Nike Subscribers/ General Youtube Audience { } likes: 15k views: 60M scene replays: [73,85,99,...,24] { } description: a sunny football ground objects: a football, players, grass asr: even after hours of practice, { }
  • 38. Can Behavior Also Lead To Better Content Understanding? • Generated scanpaths from trained ScanTextGAN model • Augmented NLP models with these generated scanpaths • Achieved performance gains in various NLP tasks: • Sentiment classification • Sarcasm detection • Paraphrase detection • Textual Entailment • Natural Language Inference Khurana, Varun, et al. "Synthesizing Human Gaze Feedback for Improved NLP Performance." Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.
  • 39. Can Behavior Also Lead To Better Content Understanding? Khurana, Varun, et al. "Synthesizing Human Gaze Feedback for Improved NLP Performance." Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.
  • 40. Can Behavior Also Lead To Better Content Understanding? QUANTITATIVE RESULTS QUALITATIVE RESULTS Khurana, Varun, et al. "Synthesizing Human Gaze Feedback for Improved NLP Performance." Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.
  • 41. Can Behavior Also Lead To Better Content Understanding? Together these results corroborate the hypothesis that leveraging the cognitive signals approximated by synthetic scanpaths in NLP models leads to performance gains. Khurana, Varun, et al. "Synthesizing Human Gaze Feedback for Improved NLP Performance." Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.
  • 42. Connecting It Back To Shannon’s Theory Of Communication Level A: The Technical Problem Level B: The Semantic Problem Level C: The Effectiveness Problem ? Simulate Human Behavior For A Certain Content Generate Content To Elicit A Given Behavior Explain Human Behavior Use Behavior To Extract Signals About Content
  • 43. We Are Hiring @Adobe MDSR! Product Features MDSR has contributed to products across the portfolio of Adobe’s digital marketing and creative tools. Publications 80+ Papers At top-tier venues like NeurIPS, JMLR, CVPR, AAAI, ICLR etc. with widespread media coverage in social media and top tech portals such as MIT Tech Review. Innovation 160+ Patents In diverse technology areas such as, computer vision, topological data analysis, Generative AI, etc. Internships 5 years Of super-charged internships with interns from reputed institutions including IITs, IIITs, MIT, Harvard, etc. 25+ Features Academic Collaborations MIT, Georgia Tech, UIUC, Univ. of Texas, Dallas, IIT Bombay, IIIT Delhi, IIT Hyderabad applytomdsr@adobe.com
  • 44. MDSR Research Areas AI for Data Insights AI for Experiences AI for Customer Journeys
  • 45. Research Areas: AI for Experiences Image Recognition ImageGeneration Logo Coca- Cola Car Woma n Beach Trunk Beverage s What’s in the image? Where’s each object? What are you looking at? Document Understanding
  • 46. Research Areas: AI for Experiences ContentSearch and Retrieval TextContentTagging KnowledgeAugmented MarketingChat-bots • Auto-suggest questions • Document recommendations • Identifying Visitor Intent • Response recommendations Query Results
  • 47. Research Areas: AI for Data Insights Look-alike Modeling CustomerSegmentation Causality Model Explanation Variance of Gradients
  • 49. Some Recent Publications  HyHTM: Hyperbolic Geometry-based Hierarchical Topic Model Simra Shahid, Tanay Anand, Nikitha Srikanth, Sumit Bhatia, Balaji Krishnamurthy and Nikaash Puri; In Findings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023).  Explain Like I am BM25: Interpreting a Dense Model's Ranked-List with a Sparse Approximation Michael Llordes, Debasis Ganguly, Sumit Bhatia and Chirag Agarwal; In 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2023).  VGFlow: Visibility guided flow network for human reposing. Rishabh Jain, Krishna Kumar Singh, Mayur Hemani, Jingwan Lu, Mausoom Sarkar, Duygu Ceylan, Balaji Krishnamurthy. (CVPR 2023)  Parameter Efficient Local Implicit Image Function Network for Face Segmentation. Mausoom Sarkar, Nikitha SR, Mayur Hemani, Rishabh Jain and Balaji Krishnamurthy. (CVPR 2023)  A Video Is Worth 4096 Tokens: Verbalize Story Videos To Understand Them In Zero Shot. Aanisha Bhattacharyya, Yaman K Singla, Balaji Krishnamurthy, Rajiv Ratn Shah, Changyou Chen (EMNLP 2023)  UMFuse: Unified Multi View Fusion for Human Editing applications. Rishabh Jain, Mayur Hemani, Duygu Ceylan, Krishna Singh, Jingwan Lu, Mausoom Sarkar, Balaji Krishnamurthy (ICCV 2023)  Synthesizing Human Gaze Feedback for Improved NLP Performance, Varun Khurana, Yaman Kumar Singla, Rajesh Kumar, Nora Hollenstein, Balaji Krishnamurthy (EACL 2023)  Persuasion Strategies in Advertisements, Yaman Kumar Singla, Rajat Jha, Arunim Gupta, Milan Aggarwal, Aditya Garg, Ayush Bhardwaj, Tushar, Balaji Krishnamurthy, Rajiv Ratn Shah, Changyou Chen (AAAI 2023)

Editor's Notes

  1. As a marketer today, I use an off-shelf LLM to create an email for a burger advertisement and schedule it through Adobe Campaign. The customers interact with the email, and analytics tells me how many customers clicked, shared, and bought the advertised product. Hmm, my ad campaign seems fine to me, but analytics tells me that very few people actually clicked on the email. Why would this be??? Let’s try to analyze why. <pause> The LLM isn’t aware of the target audience and generates plain content that doesn’t bring in any KPIs. The scheduler doesn’t consider the content it is sending, and sent my email about burgers in the morning when it should’ve sent it in the afternoon, when people are hungry. Also, analytics just told me that that my email had very few clicks, but it cant explain why. I have to imagine that on my own. Now, envision a foundation model – a large content-behavior model – that generates content with the target audience in mind, schedules it to maximize KPIs, and can explain why or why not my ad campaign had good reach.