SlideShare a Scribd company logo

Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models.pdf

PPT Created for Short story assignment for CMPEE255 Data Mining

1 of 15
Download to read offline
Enhancing Financial Sentiment Analysis via
Retrieval Augmented Large Language Models
A short Story Presentation
By Aditi Patil
Introduction
Key Focus:
● Financial sentiment analysis: Essential for interpreting emotions in financial texts for market forecasting and investor
insights.
● Challenges with traditional NLP and LLMs: Limited understanding of complex financial content and generalization
issues.
Large Language Models (LLMs) in Finance:
● Superior performance yet facing challenges in precise sentiment prediction due to training-objective discrepancy and
lack of contextual depth.
Our Innovative Approach:
● Introducing a Retrieval-Augmented Large Language Model Framework.
● Combines instruction-finetuned LLMs with contextual enrichment from external sources.
● Achieves 15% to 48% better accuracy and F1 score in financial sentiment analysis compared to existing models.
Background
A. Financial Sentiment Analysis
● Early Research: Focused on fine-tuning pre-trained models. Limited in complex scenarios, especially with
numerical data.
● LLMs Emergence: Increasing model size and data enhanced in-context learning and zero-shot predictions.
B. Instruction Tuning in LLMs
● Training Techniques: Causal Language Modeling leading to randomness in outputs.
● Solution: Instruction tuning for precise, user-directed output generation.
C. Retrieval Augmented Generation (RAG)
● Methodology: Combines context retrieval and LLMs for language generation.
● Dual Knowledge Source: Uses both parametric memory (LLMs) and nonparametric memory (external
documents).
Approach used
Module 1 :Instruction-tuned LLM Module
● Objective: Align LLM behavior with financial sentiment prediction.
● Process: Fine-tune an open-source LLM (e.g., LLaMA, ChatGLM) using a custom instruction-following dataset specific to financial sentiment.
Module 2: Retrieval Augmented Generation (RAG) Module
● Function: Retrieves relevant background information from various external sources.
● Sources: Includes Bloomberg, Reuters, Goldman Sachs, Citi Velocity, Twitter, and Reddit.
● Method: Uses a multi-source query and similarity-based retrieval for context enrichment.
Integration and Outcome
● Combining Modules: The retrieved context is merged with the original query to form the final query.
● Sentiment Prediction: The instruction-tuned LLM generates sentiment predictions based on this enriched query, enhancing prediction
accuracy.
Instruction Tuned LLMs
Overview of Instruction Tuning
● Purpose: Align LLMs with financial sentiment prediction tasks.
● Effectiveness: Proven to adhere well to user instructions in predicting financial sentiments.
Process of Instruction Tuning
1. Constructing the Dataset:Create an instruction-following dataset with paired instructions and expected
sentiment responses (positive, negative, neutral).
2. Fine-Tuning LLMs: Utilize the dataset for fine-tuning, teaching LLMs to generate accurate responses based
on financial sentiment instructions.
3. Mapping Outputs to Sentiment Classes: Convert the model's generative outputs back into predefined
sentiment classes for proper evaluation.
Instruction Tuned LLMs
Detailed Steps
● Dataset Creation:
○ Convert existing financial sentiment datasets into
instruction-following format. Include human-written
instructions and corresponding outputs.
● Fine-Tuning Methodology:
○ Tokenize texts using Byte-Pair Encoding (BPE).
○ Apply Causal Language Modeling (CLM) to optimize
prediction of next tokens.
● Output Mapping:
○ Sequentially check generated text for sentiment indicators
(“negative”, “neutral”, “positive”) and categorize accordingly.
RAG Module
Two-Step Knowledge Retrieval Process
Multi-Source Knowledge Query:
● Preprocess text to remove irrelevant content.
● Utilize retrieval APIs to extract information relevant to the financial query.
Similarity-Based Retrieval:
● Use the Szymkiewicz-Simpson coefficient for higher relevance.
● Focus on exact matches and minimize irrelevant retrievals.
Integration with Instruction-tuned LLM
● Combine retrieved context with original input to form enriched data for LLM
response generation.
The Retrieval Augmented Generation (RAG) Module
Overview of RAG
● Purpose: Enhances LLM accuracy by injecting external
knowledge into response generation.
● Process: Involves setting up external knowledge sources,
two-step knowledge retrieval, and integration with input data.
Setting Up External Knowledge Sources
● News Sources: Bloomberg, Yahoo Finance, Reuters, CNBC,
Market Screener for consistent, reliable information.
● Research Publications: Centralized (Goldman Sachs, Citi) and
crowd-based (Seeking Alpha) platforms for diverse financial
insights.
● Social Media: Twitter, Reddit for real-time updates; requires
careful analysis due to volatility.
Why Szymkiewicz-Simpson Coefficient?
Why Szymkiewicz-Simpson Over Semantic Similarity?
Focus on Exact Matches:
● Essential for financial terms like tickers.
● Ensures precision in matching specific financial terms, crucial for accurate sentiment analysis.
Effectively Manages Short-to-Long Text Matching:
● Addresses the challenge of aligning short financial texts (tweets, headlines) with longer contextual documents.
● Prevents the overshadowing of relevant content by document length.
Szymkiewicz-Simpson Coefficient in Detail:
Overlap Measure:
● Calculates similarity by the proportion of overlapping words between input and context.
Advantages in Financial Analysis:
● Prioritizes hard matching for crucial financial terms.
● More effective in specific term detection compared to semantic similarity.
Datasets used
Training Datasets
● Twitter Financial News Dataset:
○ Corpus: News tweets related to the financial sector.
○ Purpose: Classify financial sentiment within Twitter discussions.
○ Size: 9,540 samples.
○ Labels: Bearish, Bullish, Neutral.
● FiQA Dataset:
○ Inclusion: Diverse financial texts.
○ Size: 961 samples.
○ Labels: Positive, Neutral, Negative.
Testing Datasets
● Twitter Financial News Sentiment Validation (Twitter Val):
○ Split: Validation part of the Twitter dataset.
○ Size: 2,388 samples.
○ Challenge: Often lacks clear sources and context.
● Financial PhraseBank (FPB) Dataset:
○ Source: Financial news articles from Lexis-Nexis.
○ Size: 4,840 samples.
○ Annotation: By 16 finance and business professionals for high-quality sentiment labels.
Evaluations and Analysis
Overview of Evaluation
● Objective: Assess effectiveness of instruction tuning and RAG.
● Method: Compare against state-of-the-art sentiment analysis models and general-purpose LLMs.
Training and Testing Datasets
● Training Data: Combination of Twitter Financial News and FiQA datasets (10,501 samples).
● Testing Data: Twitter Val and Financial PhraseBank (FPB) datasets.
Model Training
● Base Model: Llama-7B.
● Training Process: 10 epochs, AdamW optimizer, batch size of 32, max input text length of 512 tokens.
● Hardware: DeepSpeed on 8×A100 GPUs, 58 minutes total training time.
Baseline Models for Comparison
● Models: BloombergGPT, ChatGPT, Llama-7B, ChatGLM2-6B, FinBERT.
● Performance Metrics: Accuracy and F1-score.
Performance Results
● Our Model's Performance: Significantly outperforms baselines in accuracy and F1 score.
● Instruction Tuning Impact: Demonstrates improved ability to discern sentiment.
Case Study of RAG Module
Assessing RAG's Effectiveness
● Tested On: Instruction-tuned model and ChatGPT 4.0 using the Twitter Val dataset.
● Results: Introduction of RAG enhances LLMs' performance, validating the impact of added context.
Comparative Performance
● Without RAG:
○ ChatGPT 4.0: 78.8% Accuracy, 65.2% F1 score.
○ Our Model: 86.3% Accuracy, 81.1% F1 score.
● With RAG:
○ ChatGPT 4.0: 81.3% Accuracy, 70.8% F1 score.
○ Our Model: 88.1% Accuracy, 84.2% F1 score.
Case Study Highlight
● Example: $ENR - Energizer shakes off JPMorgan’s bear call.
● Without RAG: Classified as "Neutral".
● With RAG: Context from Seeking Alpha clarifies the phrase, reclassified as "Positive".
● Significance: Showcases how RAG enhances comprehension and nuanced understanding of financial sentiment.
Conclusion
1. Conclusion
● Innovative Framework: Developed a retrieval-augmented LLM framework for enhanced
financial sentiment analysis.
● Instruction Tuning Method: Successfully realigned LLMs for improved accuracy in predicting
financial sentiments.
● Contextual Enrichment: Integrated external knowledge retrieval for more nuanced predictions.
2. Limitation and Future Work
● Current Limitation: Reliance on textual similarity for data retrieval, potentially missing key
economic information.
● Future Direction: Plan to incorporate macroeconomic and microeconomic data to refine
analysis accuracy and reliability, aiming for a comprehensive approach in financial sentiment
analysis with LLMs.
References
Paper that is Summarized:
Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models (Boyu Zhang, Hongyang Yang, Tianyu Zhou, Ali Babar, Xiao-Yang Liu)
Link: https://arxiv.org/pdf/2310.04027.pdf
Additional Papers read:
● Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.,
“Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems, vol. 35, pp. 27730–27744, 2022.
Link: https://arxiv.org/pdf/2203.02155.pdf
● Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal
Azhar, et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023. Link: https://arxiv.org/pdf/2302.13971.pdf
● Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich uttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, et al.,
“Retrieval-augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474, 2020. Link:
https://arxiv.org/pdf/2005.11401.pdf
Thank You !

Recommended

CMPE255: Short Story Assignment FinGPT
CMPE255: Short Story Assignment FinGPTCMPE255: Short Story Assignment FinGPT
CMPE255: Short Story Assignment FinGPTshawnchumbar
 
Towards Purposeful Reuse of Semantic Datasets Through Goal-Driven Summarization
Towards Purposeful Reuse of Semantic Datasets Through Goal-Driven SummarizationTowards Purposeful Reuse of Semantic Datasets Through Goal-Driven Summarization
Towards Purposeful Reuse of Semantic Datasets Through Goal-Driven SummarizationPanos Alexopoulos
 
CV-Grace-DataAnalytics-UCL
CV-Grace-DataAnalytics-UCLCV-Grace-DataAnalytics-UCL
CV-Grace-DataAnalytics-UCLHan Yang
 
Analytics @ Marketing Service Center - discussion document
Analytics @ Marketing Service Center - discussion documentAnalytics @ Marketing Service Center - discussion document
Analytics @ Marketing Service Center - discussion documentAditya Madiraju
 
Adam Suchley - Predictive Delivery Assurance - APM Assurance SIG Conference 2018
Adam Suchley - Predictive Delivery Assurance - APM Assurance SIG Conference 2018Adam Suchley - Predictive Delivery Assurance - APM Assurance SIG Conference 2018
Adam Suchley - Predictive Delivery Assurance - APM Assurance SIG Conference 2018Association for Project Management
 
Hr competency mapping
Hr competency mappingHr competency mapping
Hr competency mappingyashika2202
 
Stock price prediction using stock eod of day price
Stock price prediction using stock eod of day priceStock price prediction using stock eod of day price
Stock price prediction using stock eod of day priceEkaantik Software Solutions
 

More Related Content

Similar to Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models.pdf

IRJET- Aspect based Sentiment Analysis on Financial Data using Transferred Le...
IRJET- Aspect based Sentiment Analysis on Financial Data using Transferred Le...IRJET- Aspect based Sentiment Analysis on Financial Data using Transferred Le...
IRJET- Aspect based Sentiment Analysis on Financial Data using Transferred Le...IRJET Journal
 
my model genuines.
my model genuines.my model genuines.
my model genuines.Teng Xiaolu
 
Andre Jankowitz CV - Full Version
Andre Jankowitz CV - Full VersionAndre Jankowitz CV - Full Version
Andre Jankowitz CV - Full VersionAndre Jankowitz
 
MUTUAL FUND RECOMMENDATION SYSTEM WITH PERSONALIZED EXPLANATIONS
MUTUAL FUND RECOMMENDATION SYSTEM WITH PERSONALIZED EXPLANATIONSMUTUAL FUND RECOMMENDATION SYSTEM WITH PERSONALIZED EXPLANATIONS
MUTUAL FUND RECOMMENDATION SYSTEM WITH PERSONALIZED EXPLANATIONSIRJET Journal
 
IRJET- Strength and Workability of High Volume Fly Ash Self-Compacting Concre...
IRJET- Strength and Workability of High Volume Fly Ash Self-Compacting Concre...IRJET- Strength and Workability of High Volume Fly Ash Self-Compacting Concre...
IRJET- Strength and Workability of High Volume Fly Ash Self-Compacting Concre...IRJET Journal
 
IRJET- Implementing Social CRM System for an Online Grocery Shopping Platform...
IRJET- Implementing Social CRM System for an Online Grocery Shopping Platform...IRJET- Implementing Social CRM System for an Online Grocery Shopping Platform...
IRJET- Implementing Social CRM System for an Online Grocery Shopping Platform...IRJET Journal
 
LeadTeam start-up: step 9 Key activities and Key partners
LeadTeam start-up: step 9 Key activities and Key partnersLeadTeam start-up: step 9 Key activities and Key partners
LeadTeam start-up: step 9 Key activities and Key partnersClémence Lagaüzère
 
Graph embedding approach to analyze sentiments on cryptocurrency
Graph embedding approach to analyze sentiments on cryptocurrencyGraph embedding approach to analyze sentiments on cryptocurrency
Graph embedding approach to analyze sentiments on cryptocurrencyIJECEIAES
 
Tanmoy patra resume
Tanmoy patra resumeTanmoy patra resume
Tanmoy patra resumeTanmoy Patra
 
Sentiment Analysis of Twitter Data
Sentiment Analysis of Twitter DataSentiment Analysis of Twitter Data
Sentiment Analysis of Twitter DataSumit Raj
 
Business statistics and analytics
Business statistics and analyticsBusiness statistics and analytics
Business statistics and analyticsVijay K S
 
CHRIS_SMITH_CV_2015
CHRIS_SMITH_CV_2015CHRIS_SMITH_CV_2015
CHRIS_SMITH_CV_2015Chris Smith
 
Performance Management to Program Evaluation: Creating a Complementary Connec...
Performance Management to Program Evaluation: Creating a Complementary Connec...Performance Management to Program Evaluation: Creating a Complementary Connec...
Performance Management to Program Evaluation: Creating a Complementary Connec...nicholes21
 

Similar to Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models.pdf (20)

IRJET- Aspect based Sentiment Analysis on Financial Data using Transferred Le...
IRJET- Aspect based Sentiment Analysis on Financial Data using Transferred Le...IRJET- Aspect based Sentiment Analysis on Financial Data using Transferred Le...
IRJET- Aspect based Sentiment Analysis on Financial Data using Transferred Le...
 
my model genuines.
my model genuines.my model genuines.
my model genuines.
 
CWAYITA NYELI (CV) 2
CWAYITA NYELI (CV) 2CWAYITA NYELI (CV) 2
CWAYITA NYELI (CV) 2
 
Andre Jankowitz CV - Full Version
Andre Jankowitz CV - Full VersionAndre Jankowitz CV - Full Version
Andre Jankowitz CV - Full Version
 
MUTUAL FUND RECOMMENDATION SYSTEM WITH PERSONALIZED EXPLANATIONS
MUTUAL FUND RECOMMENDATION SYSTEM WITH PERSONALIZED EXPLANATIONSMUTUAL FUND RECOMMENDATION SYSTEM WITH PERSONALIZED EXPLANATIONS
MUTUAL FUND RECOMMENDATION SYSTEM WITH PERSONALIZED EXPLANATIONS
 
ST_Short_Story.pptx
ST_Short_Story.pptxST_Short_Story.pptx
ST_Short_Story.pptx
 
Matthew Week 3 Feedback & Project
Matthew Week 3 Feedback & Project Matthew Week 3 Feedback & Project
Matthew Week 3 Feedback & Project
 
IRJET- Strength and Workability of High Volume Fly Ash Self-Compacting Concre...
IRJET- Strength and Workability of High Volume Fly Ash Self-Compacting Concre...IRJET- Strength and Workability of High Volume Fly Ash Self-Compacting Concre...
IRJET- Strength and Workability of High Volume Fly Ash Self-Compacting Concre...
 
IRJET- Implementing Social CRM System for an Online Grocery Shopping Platform...
IRJET- Implementing Social CRM System for an Online Grocery Shopping Platform...IRJET- Implementing Social CRM System for an Online Grocery Shopping Platform...
IRJET- Implementing Social CRM System for an Online Grocery Shopping Platform...
 
Upload.pptx
Upload.pptxUpload.pptx
Upload.pptx
 
LeadTeam start-up: step 9 Key activities and Key partners
LeadTeam start-up: step 9 Key activities and Key partnersLeadTeam start-up: step 9 Key activities and Key partners
LeadTeam start-up: step 9 Key activities and Key partners
 
Graph embedding approach to analyze sentiments on cryptocurrency
Graph embedding approach to analyze sentiments on cryptocurrencyGraph embedding approach to analyze sentiments on cryptocurrency
Graph embedding approach to analyze sentiments on cryptocurrency
 
Tanmoy patra resume
Tanmoy patra resumeTanmoy patra resume
Tanmoy patra resume
 
Just Text PPM March 20 2015
Just Text PPM March 20 2015Just Text PPM March 20 2015
Just Text PPM March 20 2015
 
Sentiment Analysis of Twitter Data
Sentiment Analysis of Twitter DataSentiment Analysis of Twitter Data
Sentiment Analysis of Twitter Data
 
Business statistics and analytics
Business statistics and analyticsBusiness statistics and analytics
Business statistics and analytics
 
CHRIS_SMITH_CV_2015
CHRIS_SMITH_CV_2015CHRIS_SMITH_CV_2015
CHRIS_SMITH_CV_2015
 
Bijendra singh
Bijendra singhBijendra singh
Bijendra singh
 
A. rathi (2)
A. rathi (2)A. rathi (2)
A. rathi (2)
 
Performance Management to Program Evaluation: Creating a Complementary Connec...
Performance Management to Program Evaluation: Creating a Complementary Connec...Performance Management to Program Evaluation: Creating a Complementary Connec...
Performance Management to Program Evaluation: Creating a Complementary Connec...
 

Recently uploaded

Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...DrSumathyV
 
HayleyDerby_Market_Research_Spotify.docx
HayleyDerby_Market_Research_Spotify.docxHayleyDerby_Market_Research_Spotify.docx
HayleyDerby_Market_Research_Spotify.docxHayleyDerby
 
presentation big data analytics on Apache spark
presentation big data analytics on Apache sparkpresentation big data analytics on Apache spark
presentation big data analytics on Apache sparkVarun Garg
 
Growing through Artificial Intelligence (AI)
Growing  through Artificial Intelligence (AI)Growing  through Artificial Intelligence (AI)
Growing through Artificial Intelligence (AI)OmChou6
 
AWS_projects related AWS services such as feature store store and clarify
AWS_projects related AWS services such as feature store store and clarifyAWS_projects related AWS services such as feature store store and clarify
AWS_projects related AWS services such as feature store store and clarifyVarun Garg
 
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDFEXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDFProject Cubicle
 
Choose your perfect jacket.pdf
Choose your perfect jacket.pdfChoose your perfect jacket.pdf
Choose your perfect jacket.pdfAlexia Trejo
 
Pereira_EBM_Assign3.docx1234123412341234
Pereira_EBM_Assign3.docx1234123412341234Pereira_EBM_Assign3.docx1234123412341234
Pereira_EBM_Assign3.docx1234123412341234LR1709MUSIC
 
Basics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelBasics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelTope Osanyintuyi
 
TainoMoreiraPereiraLuis_Assignment_4.docx
TainoMoreiraPereiraLuis_Assignment_4.docxTainoMoreiraPereiraLuis_Assignment_4.docx
TainoMoreiraPereiraLuis_Assignment_4.docxLR1709MUSIC
 
introduction-to-crimean-congo-haemorrhagic-fever.pdf
introduction-to-crimean-congo-haemorrhagic-fever.pdfintroduction-to-crimean-congo-haemorrhagic-fever.pdf
introduction-to-crimean-congo-haemorrhagic-fever.pdfSalamaAdel
 
Prometheus Grafana Dashboard for Cassandra 5
Prometheus Grafana Dashboard for Cassandra 5Prometheus Grafana Dashboard for Cassandra 5
Prometheus Grafana Dashboard for Cassandra 5Sarma Pydipally
 
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptxWOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptxyosra Saidani
 
Ratio analysis, Formulas, Advantage PPt.pptx
Ratio analysis, Formulas, Advantage PPt.pptxRatio analysis, Formulas, Advantage PPt.pptx
Ratio analysis, Formulas, Advantage PPt.pptxSugumarVenkai
 
Discover the Best Free Web Hosting Services with SSL in 2023
Discover the Best Free Web Hosting Services with SSL in 2023Discover the Best Free Web Hosting Services with SSL in 2023
Discover the Best Free Web Hosting Services with SSL in 2023maker Money
 
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
Artificial Intelligence for Vision:  A walkthrough of recent breakthroughsArtificial Intelligence for Vision:  A walkthrough of recent breakthroughs
Artificial Intelligence for Vision: A walkthrough of recent breakthroughsNikolas Markou
 
Why Choose Canada for your next film project
Why Choose Canada for your next film projectWhy Choose Canada for your next film project
Why Choose Canada for your next film projectnubreed2019
 
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...Samuel Chukwuma
 
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdfEIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdfEarley Information Science
 
Customer Satisfaction Data - Multiple Linear Regression Model.pdf
Customer Satisfaction Data -  Multiple Linear Regression Model.pdfCustomer Satisfaction Data -  Multiple Linear Regression Model.pdf
Customer Satisfaction Data - Multiple Linear Regression Model.pdfruwanp2000
 

Recently uploaded (20)

Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...
 
HayleyDerby_Market_Research_Spotify.docx
HayleyDerby_Market_Research_Spotify.docxHayleyDerby_Market_Research_Spotify.docx
HayleyDerby_Market_Research_Spotify.docx
 
presentation big data analytics on Apache spark
presentation big data analytics on Apache sparkpresentation big data analytics on Apache spark
presentation big data analytics on Apache spark
 
Growing through Artificial Intelligence (AI)
Growing  through Artificial Intelligence (AI)Growing  through Artificial Intelligence (AI)
Growing through Artificial Intelligence (AI)
 
AWS_projects related AWS services such as feature store store and clarify
AWS_projects related AWS services such as feature store store and clarifyAWS_projects related AWS services such as feature store store and clarify
AWS_projects related AWS services such as feature store store and clarify
 
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDFEXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
 
Choose your perfect jacket.pdf
Choose your perfect jacket.pdfChoose your perfect jacket.pdf
Choose your perfect jacket.pdf
 
Pereira_EBM_Assign3.docx1234123412341234
Pereira_EBM_Assign3.docx1234123412341234Pereira_EBM_Assign3.docx1234123412341234
Pereira_EBM_Assign3.docx1234123412341234
 
Basics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelBasics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft Excel
 
TainoMoreiraPereiraLuis_Assignment_4.docx
TainoMoreiraPereiraLuis_Assignment_4.docxTainoMoreiraPereiraLuis_Assignment_4.docx
TainoMoreiraPereiraLuis_Assignment_4.docx
 
introduction-to-crimean-congo-haemorrhagic-fever.pdf
introduction-to-crimean-congo-haemorrhagic-fever.pdfintroduction-to-crimean-congo-haemorrhagic-fever.pdf
introduction-to-crimean-congo-haemorrhagic-fever.pdf
 
Prometheus Grafana Dashboard for Cassandra 5
Prometheus Grafana Dashboard for Cassandra 5Prometheus Grafana Dashboard for Cassandra 5
Prometheus Grafana Dashboard for Cassandra 5
 
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptxWOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
 
Ratio analysis, Formulas, Advantage PPt.pptx
Ratio analysis, Formulas, Advantage PPt.pptxRatio analysis, Formulas, Advantage PPt.pptx
Ratio analysis, Formulas, Advantage PPt.pptx
 
Discover the Best Free Web Hosting Services with SSL in 2023
Discover the Best Free Web Hosting Services with SSL in 2023Discover the Best Free Web Hosting Services with SSL in 2023
Discover the Best Free Web Hosting Services with SSL in 2023
 
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
Artificial Intelligence for Vision:  A walkthrough of recent breakthroughsArtificial Intelligence for Vision:  A walkthrough of recent breakthroughs
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
 
Why Choose Canada for your next film project
Why Choose Canada for your next film projectWhy Choose Canada for your next film project
Why Choose Canada for your next film project
 
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
 
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdfEIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
 
Customer Satisfaction Data - Multiple Linear Regression Model.pdf
Customer Satisfaction Data -  Multiple Linear Regression Model.pdfCustomer Satisfaction Data -  Multiple Linear Regression Model.pdf
Customer Satisfaction Data - Multiple Linear Regression Model.pdf
 

Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models.pdf

  • 1. Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models A short Story Presentation By Aditi Patil
  • 2. Introduction Key Focus: ● Financial sentiment analysis: Essential for interpreting emotions in financial texts for market forecasting and investor insights. ● Challenges with traditional NLP and LLMs: Limited understanding of complex financial content and generalization issues. Large Language Models (LLMs) in Finance: ● Superior performance yet facing challenges in precise sentiment prediction due to training-objective discrepancy and lack of contextual depth. Our Innovative Approach: ● Introducing a Retrieval-Augmented Large Language Model Framework. ● Combines instruction-finetuned LLMs with contextual enrichment from external sources. ● Achieves 15% to 48% better accuracy and F1 score in financial sentiment analysis compared to existing models.
  • 3. Background A. Financial Sentiment Analysis ● Early Research: Focused on fine-tuning pre-trained models. Limited in complex scenarios, especially with numerical data. ● LLMs Emergence: Increasing model size and data enhanced in-context learning and zero-shot predictions. B. Instruction Tuning in LLMs ● Training Techniques: Causal Language Modeling leading to randomness in outputs. ● Solution: Instruction tuning for precise, user-directed output generation. C. Retrieval Augmented Generation (RAG) ● Methodology: Combines context retrieval and LLMs for language generation. ● Dual Knowledge Source: Uses both parametric memory (LLMs) and nonparametric memory (external documents).
  • 4. Approach used Module 1 :Instruction-tuned LLM Module ● Objective: Align LLM behavior with financial sentiment prediction. ● Process: Fine-tune an open-source LLM (e.g., LLaMA, ChatGLM) using a custom instruction-following dataset specific to financial sentiment. Module 2: Retrieval Augmented Generation (RAG) Module ● Function: Retrieves relevant background information from various external sources. ● Sources: Includes Bloomberg, Reuters, Goldman Sachs, Citi Velocity, Twitter, and Reddit. ● Method: Uses a multi-source query and similarity-based retrieval for context enrichment. Integration and Outcome ● Combining Modules: The retrieved context is merged with the original query to form the final query. ● Sentiment Prediction: The instruction-tuned LLM generates sentiment predictions based on this enriched query, enhancing prediction accuracy.
  • 5. Instruction Tuned LLMs Overview of Instruction Tuning ● Purpose: Align LLMs with financial sentiment prediction tasks. ● Effectiveness: Proven to adhere well to user instructions in predicting financial sentiments. Process of Instruction Tuning 1. Constructing the Dataset:Create an instruction-following dataset with paired instructions and expected sentiment responses (positive, negative, neutral). 2. Fine-Tuning LLMs: Utilize the dataset for fine-tuning, teaching LLMs to generate accurate responses based on financial sentiment instructions. 3. Mapping Outputs to Sentiment Classes: Convert the model's generative outputs back into predefined sentiment classes for proper evaluation.
  • 6. Instruction Tuned LLMs Detailed Steps ● Dataset Creation: ○ Convert existing financial sentiment datasets into instruction-following format. Include human-written instructions and corresponding outputs. ● Fine-Tuning Methodology: ○ Tokenize texts using Byte-Pair Encoding (BPE). ○ Apply Causal Language Modeling (CLM) to optimize prediction of next tokens. ● Output Mapping: ○ Sequentially check generated text for sentiment indicators (“negative”, “neutral”, “positive”) and categorize accordingly.
  • 7. RAG Module Two-Step Knowledge Retrieval Process Multi-Source Knowledge Query: ● Preprocess text to remove irrelevant content. ● Utilize retrieval APIs to extract information relevant to the financial query. Similarity-Based Retrieval: ● Use the Szymkiewicz-Simpson coefficient for higher relevance. ● Focus on exact matches and minimize irrelevant retrievals. Integration with Instruction-tuned LLM ● Combine retrieved context with original input to form enriched data for LLM response generation.
  • 8. The Retrieval Augmented Generation (RAG) Module Overview of RAG ● Purpose: Enhances LLM accuracy by injecting external knowledge into response generation. ● Process: Involves setting up external knowledge sources, two-step knowledge retrieval, and integration with input data. Setting Up External Knowledge Sources ● News Sources: Bloomberg, Yahoo Finance, Reuters, CNBC, Market Screener for consistent, reliable information. ● Research Publications: Centralized (Goldman Sachs, Citi) and crowd-based (Seeking Alpha) platforms for diverse financial insights. ● Social Media: Twitter, Reddit for real-time updates; requires careful analysis due to volatility.
  • 9. Why Szymkiewicz-Simpson Coefficient? Why Szymkiewicz-Simpson Over Semantic Similarity? Focus on Exact Matches: ● Essential for financial terms like tickers. ● Ensures precision in matching specific financial terms, crucial for accurate sentiment analysis. Effectively Manages Short-to-Long Text Matching: ● Addresses the challenge of aligning short financial texts (tweets, headlines) with longer contextual documents. ● Prevents the overshadowing of relevant content by document length. Szymkiewicz-Simpson Coefficient in Detail: Overlap Measure: ● Calculates similarity by the proportion of overlapping words between input and context. Advantages in Financial Analysis: ● Prioritizes hard matching for crucial financial terms. ● More effective in specific term detection compared to semantic similarity.
  • 10. Datasets used Training Datasets ● Twitter Financial News Dataset: ○ Corpus: News tweets related to the financial sector. ○ Purpose: Classify financial sentiment within Twitter discussions. ○ Size: 9,540 samples. ○ Labels: Bearish, Bullish, Neutral. ● FiQA Dataset: ○ Inclusion: Diverse financial texts. ○ Size: 961 samples. ○ Labels: Positive, Neutral, Negative. Testing Datasets ● Twitter Financial News Sentiment Validation (Twitter Val): ○ Split: Validation part of the Twitter dataset. ○ Size: 2,388 samples. ○ Challenge: Often lacks clear sources and context. ● Financial PhraseBank (FPB) Dataset: ○ Source: Financial news articles from Lexis-Nexis. ○ Size: 4,840 samples. ○ Annotation: By 16 finance and business professionals for high-quality sentiment labels.
  • 11. Evaluations and Analysis Overview of Evaluation ● Objective: Assess effectiveness of instruction tuning and RAG. ● Method: Compare against state-of-the-art sentiment analysis models and general-purpose LLMs. Training and Testing Datasets ● Training Data: Combination of Twitter Financial News and FiQA datasets (10,501 samples). ● Testing Data: Twitter Val and Financial PhraseBank (FPB) datasets. Model Training ● Base Model: Llama-7B. ● Training Process: 10 epochs, AdamW optimizer, batch size of 32, max input text length of 512 tokens. ● Hardware: DeepSpeed on 8×A100 GPUs, 58 minutes total training time. Baseline Models for Comparison ● Models: BloombergGPT, ChatGPT, Llama-7B, ChatGLM2-6B, FinBERT. ● Performance Metrics: Accuracy and F1-score. Performance Results ● Our Model's Performance: Significantly outperforms baselines in accuracy and F1 score. ● Instruction Tuning Impact: Demonstrates improved ability to discern sentiment.
  • 12. Case Study of RAG Module Assessing RAG's Effectiveness ● Tested On: Instruction-tuned model and ChatGPT 4.0 using the Twitter Val dataset. ● Results: Introduction of RAG enhances LLMs' performance, validating the impact of added context. Comparative Performance ● Without RAG: ○ ChatGPT 4.0: 78.8% Accuracy, 65.2% F1 score. ○ Our Model: 86.3% Accuracy, 81.1% F1 score. ● With RAG: ○ ChatGPT 4.0: 81.3% Accuracy, 70.8% F1 score. ○ Our Model: 88.1% Accuracy, 84.2% F1 score. Case Study Highlight ● Example: $ENR - Energizer shakes off JPMorgan’s bear call. ● Without RAG: Classified as "Neutral". ● With RAG: Context from Seeking Alpha clarifies the phrase, reclassified as "Positive". ● Significance: Showcases how RAG enhances comprehension and nuanced understanding of financial sentiment.
  • 13. Conclusion 1. Conclusion ● Innovative Framework: Developed a retrieval-augmented LLM framework for enhanced financial sentiment analysis. ● Instruction Tuning Method: Successfully realigned LLMs for improved accuracy in predicting financial sentiments. ● Contextual Enrichment: Integrated external knowledge retrieval for more nuanced predictions. 2. Limitation and Future Work ● Current Limitation: Reliance on textual similarity for data retrieval, potentially missing key economic information. ● Future Direction: Plan to incorporate macroeconomic and microeconomic data to refine analysis accuracy and reliability, aiming for a comprehensive approach in financial sentiment analysis with LLMs.
  • 14. References Paper that is Summarized: Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models (Boyu Zhang, Hongyang Yang, Tianyu Zhou, Ali Babar, Xiao-Yang Liu) Link: https://arxiv.org/pdf/2310.04027.pdf Additional Papers read: ● Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al., “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems, vol. 35, pp. 27730–27744, 2022. Link: https://arxiv.org/pdf/2203.02155.pdf ● Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023. Link: https://arxiv.org/pdf/2302.13971.pdf ● Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich uttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, et al., “Retrieval-augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474, 2020. Link: https://arxiv.org/pdf/2005.11401.pdf