describe

1. Productionizing ML Final
Project
A fully integrated app combining image captioning and question
answering services.
Shuchismit Jha | Sahil Shaheen | RK Yashasvi

2. Services
Image Captioning
• Model:
• InceptionV3 feature extractor (ImageNET
weights)
• Encoder & Decoder with Attention mechanism
• Trained on a subset of COCO dataset
(computational limitations)
1
Question Answering
• Model:
• Distilbert-base-cased-distilled-squad
• Fine-tuned using (a second step of) knowledge
distillation on SQuAD v1.1
• F1 score of 87.1
2

3. System Design

4. Demo
Link - https://animated-pudding-88c6c9.netlify.app/

5. Web
Application

6. API (Lambda backend)
Takes POST payload and identifies task using "task" parameter
Generates unique UUID for request
Image Classification subtasks
• Decodes Base64-encoded image
• Stores it in S3 bucket
• Sends message with ID and path attributes to SQS queue
Question Answering subtasks
• Sends message with ID and concatenated context-question text data
Polls S3 bucket for result file corresponding to ID

7. Container Apps
Image Captioning
• Polls the queue for requests
• Gets image object
corresponding to request ID
from S3 bucket
(/results/{object})
• Inferences through the
captioning model
• Stores the results in S3 bucket
(/results/{id}.json)
Question Answering
• Polls the queue for request
• Inferences through the pipeline
• Stores the result in S3 bucket
(/{id}.json)