This was my Final Year Project thesis for my Software Engineering Degree at Staffordshire University, UK. An Augmented Reality based Real-time Products and Advertisements identification System developed upon, Real-Time Image Processing, Machine Learning, Computer Vision Technologies, SURF (Speeded Up Robust Feature Detection) Feature detection algorithm, Android, Cloud technology. The .Net platform for the Server side implementation and Java Android development for Client side development was being used.

1. AUGMENTED REALITY BASED
PRODUCT IDENTIFICATION
AND ADVERTISING SYSTEM
- UDARA ABHILASH ALWIS (CB003982)

2. INTRODUCTION

3. PROBLEM BACKGROUND
Problem Statement
There are huge product varieties in the market and advertising has
a great impact in today’s life. Customers are always looking for
extended information when they buy products or even regarding
an advertisement, whereas it is very crucial to provide users with
updated and accurate information regarding them. QR Codes are
being deprecated in modern end consumerism due to its
disadvantages and real time visual recognitions technologies are
coming up with powerful mobile devices. Augmented Reality is
becoming the next mass media advertising in today’s world.

4. PROBLEM BACKGROUND
• Product information could change anytime
• Advertisements can not display all the information at once
• Customers are looking for more extended information
• No proper method to look for information, time waste
• Augmented Reality as the next mass advertising media
• Extinction of QR Codes from end user consumerism

5. PROBLEM OBJECTIVES
• Identify Product or Advertisement
• View the extended information, and related advertisements or
products
• Implement Augmented Reality based concepts in the
application
• Insert Products and Advertisements information as
Administrators
• Simulate Product or Advertisement Identification process in
admin system.

6. SOLUTION OUTLINE
• Capture image when the user points at a product or an
advertisement.
• Identify the product or the advertisement from the server side.
• Provide extended information about the product or the
advertisement to the user.
• Present the information in according to Augmented Reality
principles.
• Allow admins to add new information to the System.

7. MAIN CHALLENGES
• Implementing Augmented Reality into the Mobile Application.
• Getting the identification results from the server from the
fastest way possible.
• Implementing an Image Matching Algorithms.
• Identifying a product or an advertisement by its Brand Logo.
• Adding and maintaining Product and Advertisement
information

8. RESEARCH

9. RESEARCH
• User Requirements Investigation
• Customers
• Difficulty of Finding information
• Having to wait for a Salesperson to show up
• QR Codes, useless
• A New Experience
• Advertisers
• The search for new creative advertising strategies
• Modern Adverts focus simplicity, therefore less information
• Visual Recognition over QR Code Recognition

10. RESEARCH
• Similar Systems Study
• Aurasma
• Vuforia Qualcomm
• Layar
• The IBM Augmented Reality Personal
Shopping Assist
• Conclusions
• Used for Advertising and real time
Image target identification
• Only available for US and Europe
based countries
• Cloud technologies
• No Local implementation

11. RESEARCH
• Augmented Reality
• Mobile Augmented Reality
• Mobile Application Development Platform
• Apple
• Android
• Windows Phone
• Mobile Video Stream Frame Monitoring
• Image Processing

12. RESEARCH
• Image Enhancements
• Spatial Domain Enchantement
• Image Negatives
• Contrast Stretching
• Histogram Processing
• Frequency Domain Enchantement
• Smoothing Frequency-Domain Filters
• Low pass Filtering
• High pass Filtering
• Computer Vision

13. RESEARCH
• Feature Extraction
• SIFT - Scale Invariant Feature Transforms
• SURF – Speeded Up Robust Features
• Matching
• 3D Object Detection in SURF
• FAST - Corner Detection
• FREAK - Fast Retina Key point Detection
• BRISK - Binary Robust Invariant Scalable Key-points
• Character Recognition and Reading

14. RESEARCH
• Selection of APIs and Tools
• Image Processing and Image Matching APIs
• OpenCV
• MatLab
• Simple CV
• Character Recognition APIs
• Tesseract
• ABBY OCR

15. RESEARCH
• Augmented Realty APIs and SDKs
• Vuforia
• FastCV
• Aurasma
• Layar

16. DEVELOPMENT METHODOLOGY
Feature Driven Development
• Optimized for Scalable Systems
• Executes on iterative development
cycles
• Suited for time intensive application
• Features list that is being built at the
beginning of the development method
cycle
• Feature by Feature
• Each feature gets much better focus

17. ANALYSIS
• Augmented Realty APIs and SDKs
• Mobile Augmented Reality
• Mobile Application Development Platform
• Apple
• Android
• Windows Phone
• Mobile Video Stream Frame Monitoring
• Image Processing

18. RESEARCH

19. RESEARCH
• Similar Systems Study Analysis
• Augmented Reality
• Image Processing and Recognition
• Cloud Computing
• User Experience
• Availability for 3rd Parties
• Not available for 3rd World Countries
• No actual implementation in Sri Lanka (Experimental)

20. RESEARCH
• Mobile Application Development Analysis – Android
• Mobile Platform Usage
• Developer Support and Features
• Multitasking and Heavy Weight Processing Support
• Image Processing
• User Friendliness and UI Features

21. RESEARCH
• Computer Vision Analysis
• Robust
• Faster
• High Accuracy
• Feature Extraction Analysis - SIFT, SURF, FREAK and BRISK
• SURF
• Speed and Accuracy
• Implementation Support

22. REQUIREMENT SPECIFICATION

23. REQUIREMENT SPECIFICATION
• Functional Requirements
• Ability to identify Products or Advertisements
• Augmented Reality should be implemented
• Provide Related Products and Advertisements
• Automatically Sort and categorize related Products and
Advertisements

24. REQUIREMENT SPECIFICATION
• Non-functional requirements
• User friendly and easy to use Interface.
• An Interface suitable for Augmented Reality based UI designs.
• Mobile application should be smooth and quickly responsive.
• Recognize Products and advertisements from the fastest way
possible.

25. DESIGN

26. DESIGN
• Use case – Mobile Application

27. DESIGN
• Use case – Admin Server System

28. DESIGN • Class Diagram – Admin Server System

29. DESIGN • Activity Diagram – SURF Image Matching

30. DESIGN • Sequence Diagram – Brand Logo Identification

31. DESIGN • Sequence Diagram – Product Identification

32. DESIGN • Sequence Diagram – Mobile Application

33. IMPLEMENTATION
• Image matching using SURF
• Character Recognition and key words identification
• Mobile Application UI development
• Mobile Application Backend
• Hosting a Web Service

34. IMPLEMENTATION
• SURF Feature Extraction and Matching
• Extract Feature Points and Descriptors
• Brute Force Match and Vote for Feature Point Matches

35. IMPLEMENTATION
• Further Voting and Filtering
• Draw Matching Feature Points and Return Results

36. IMPLEMENTATION
• SURF Feature Extraction and Matching Sample

37. IMPLEMENTATION
• Experimentations
• In need of Experimentation and Modifications
• Could use “mask” Matrix
• Started to give even greater matched points count of
incorrect images

38. IMPLEMENTATION
• Experimentation Conclusions
• Get the number of matched feature points count
• Calculate the Area of the formed Polygon
• Identify whether the formed Polygon is a proper Polygon
• Calculate the percentage Ratio of the Detected Key points of
both images
• Adjust the Uniqueness threshold

39. IMPLEMENTATION
• Determining Best Match

40. IMPLEMENTATION
• Threshold of an Image Matching Process
• Image Pre-Processing Usage
• Character Recognition and key words identification

41. IMPLEMENTATION
• Mobile Image Processing
• Image Resizing and Scaling

42. IMPLEMENTATION
• Image Encoding and Decoding – Base64

43. IMPLEMENTATION
• Web Service Connectivity

44. IMPLEMENTATION
• Hosting a Web Service
• Runtime Exception – Heavy EmguCV Libraries
• Cv.Invoke Exceptions
• Server Out Of Memory Exceptions

45. IMPLEMENTATION
• Dummy Web Service

46. IMPLEMENTATION
• Overall Admin System Implementation
• Object Oriented Methodologies
• Surf Execution Class

47. IMPLEMENTATION
• Product Identification and Advertisement Identification Process
• Find the Brand Logo Information
• Load the Product Images under that Brand Logo
• Take the given image through that Product Images list to find
the best match and retrieve information.

48. TESTING
• Unit Testing
• Functional Testing
• Performance Testing

49. TESTING
• Unit Testing - Server System
Test Case 1 Test Name : Recognize Given Product Image’s Brand Logo at Server Simulation
Test Criteria Check if the Given Product’s Brand type is identified.
Input / Action Performed Drag and Drop a Product Image and Click on “Search and Detect”
Expected Results System should identify the brand logo and Display the Brand Information
Actual Results The System successfully identified the Brand Type and showed the information of the
Brand.
Test Case 2 Test Name : Recognize Given Product Image at Server Simulation
Test Criteria Check if the given Product’s front view Image is Recognized and Show the information
of the Product
Input / Action Performed Drag and drop Product Image and click on “Search and Detect” button.
Expected Results System should recognize the Product and show information about it as well as Related
Products.
Actual Results Product was Successfully Recognized and information were showed.

50. TESTING
• Unit Testing - Server System
Test Case 3 Test Name : Determine when a given Image of a Product or an Advertisement cannot be
Identified
Test Criteria Check if the given Threshold condition is able to determine whether a given image can
be Recognized or not.
Input / Action Performed Insert a random unrelated Product/advertisement Image to the Matching Algorithm and
execute matching algorithm
Expected Results The algorithm should determine that the given Image cannot be Recognized based on
the Threshold condition.
Actual Results The Matching process successfully determines the given Image cannot be identified.

51. TESTING
• Unit Testing – Server System
Test Case 4 Test Name : Automatically Recognize the Brand Type of the Product/Advert image when
adding a new Product/Advert info
Test Criteria Check if the System is able to Automatically Recognize the Brand Type of a given
Product or Advertisement Image when Inserting their information.
Input / Action Performed Drag and drop the Product/Advert Image and click on “Auto Detect”
Expected Results The matching algorithm will detect the Brand logo of the Product/Advert Image and
identify the Brand.
Actual Results The system successfully identifies the Brand Type by detecting the brand logo.

52. TESTING
• Unit Testing – Mobile Application
Test Case 5 Test Name : Retrieve Recognition results from the Web Service
Test Criteria Check if the Application is able to retrieve the identification result of a capture image
Input / Action Performed Click on “Scan and Detect” button
Expected Results After sending the image to server application should wait for the response and
retrieve the results.
Actual Results The Application send the image to the server and waits till the response is being
given, while the other UI Controls are being disabled. When the results are retrieved,
they are being displayed successfully.

53. TESTING
• Performance Testing
Case Expected Accuracy Actual Result
Capturing Image and Sending to Web
100% 100%
service
Retrieving Product/Advert Recognition
Results
100% 100%
Capturing product/advert image from
a different angle and recognizing it
100% 50%
Overall Recognition Results 90% 70%
푃표푠푖푡푖푣푒 푅푒푠푢푙푡푠
푁푢푚푏푒푟 표푓 푇푒푠푡푠
× 100%

54. CRITICAL EVALUATION
• Efficiency
• Execute all the heavy processing in Server
• Holding the Pre-Processed – SURF Features Extracted Images
in the Memory
• Mobile Application to Server Communication – Data
compression and Base64 Encoding
• Utilizing the SURF based Image Matching process to achieve
more accuracy and Speed
• Utilizing the Recognition Process
• Automated Brand Logo detection Process

55. CRITICAL EVALUATION
• Ease of Use
• Accuracy
• Increased User Satisfaction
• Maintainability and Extendibility
• The Centralized Database
• Web Service Implementation

56. CRITICAL EVALUATION
• System Utility and Usability
• Consumers, Advertisers and Marketers
• Providing Access for Extended information
• Information are updated and centralized
• implement creative strategies into Augmented Reality
• intuitive way to recognize products for end users rather than
QR Codes
• easily upgraded and extended due to the architecture
• The interface has been designed in a very minimalistic, simple,
and user friendly

57. CRITICAL EVALUATION
• Further Enhancements
• Further Developing the Image Matching Process from ground
up to use even more efficient Algorithms such as FREAK or
BRISK Feature Extraction algorithms
• Adopt an Augmented Reality API such as Vuforia
• High end Cloud Service which will radically increase the
processing power.
• Adopt a new lightweight API Library for Image Processing

58. DEMONSTRATION

59. Q & A

60. THANK YOU !