The advances of mobile computing and sensor technology have turned the mobile devices into powerful instruments. The integration of thermal and visual cameras extends the capability of computer vision, due to the fact that both images reveal different characteristics in images; however, image alignment is a challenge. This paper proposes an effective approach to align image pairs for event detection on mobile through image recognition. We leverage thermal and visual cameras as multi-modality sources for image recognition. By analyzing the heat pattern, the proposed APP can identify the heating sources and help users inspect their house heating system; on the other hand, with applying image recognition, the proposed APP furthermore can help field workers identify the asset condition and provide the guidance to solve their issues.

1. Jui-Hsin(Larry) Lai, Chung-Ching Lin,
Chun-Fu(Richard) Chen, and Ching-Yung Li
n
IBM T.J. Watson Research Center
Multi-Modality Mobile Image
Recognition Based on
Thermal and Visual Cameras
Demo Videos
www.larry-lai.com/mobilethermal.html

2. IBM :: IBM Confidential :: ©2015 IBM Corporation 2

3. IBM :: IBM Confidential :: ©2015 IBM Corporation
Expert Tech APP with SmallBlue
3
• SmallBlue
• find your colleagues with the
knowledge that your are looking for
• Expert Tech APP
• give field technicians the right tools
for the job of customer service

4. IBM :: IBM Confidential :: ©2015 IBM Corporation
Demo: Expert Tech APP with SmallBlue
4

5. IBM :: IBM Confidential :: ©2015 IBM Corporation
More Functions in Expert Tech APP
5
• Expert Tech APP
• give field technicians the right tools
for the job of customer service
A trouble to workers— when facing a device, technicians have no
idea/experience to fix it. How can Expert Tech do the help?

6. IBM :: IBM Confidential :: ©2015 IBM Corporation
The Help from Image Recognition
6
• Where the computation take place
• running image recognition on the cloud server while it
connects to Internet
• running image recognition on the mobile device while there
is no Internet connection
• Steps
• (1) take a picture of an asset,
• (2) run image recognition algorithm,
• (3) and show the instructions to fix it step-by-step

7. IBM :: IBM Confidential :: ©2015 IBM Corporation
Image Recognition Running on Mobile Devices
7
• Challenges
• low computation capability on mobile devices
• require immediate response for real-time applications
• Computation Acceleration
• Feature detection: Harris Corner Detector is implemented
because it can be easily modified into pixel-wise operation
=> by leveraging CIKernal, the iOS image processing
framework, to use GPU’s for calculation
• Feature description: modify SIFT algorithm to reduce feature
vectors
=> by leveraging multi-thread for acceleration
• Feature matching: image features are stored with CoreDate,
the iOS SQL framework, for efficient data access

8. IBM :: IBM Confidential :: ©2015 IBM Corporation
Not Only Recognition, But Appearance Inspection
8
Cancel
Check the power signal
The LED should be green light as
a normal condition.
View instructions
Fixing Point 2
• Geometry Mapping & Local Feature Extraction
• Register image features for assets under normal conditions,
like the light color, component shape
• Compare features of captured images to the database

9. IBM :: IBM Confidential :: ©2015 IBM Corporation
Expert Tech APP with Image Recognition
9

10. IBM :: IBM Confidential :: ©2015 IBM Corporation
Do More Beyond Human Eye
• Asset Temperature Readings
• Take thermal and visible images of an asset, run image recognition to
identify the asset’s model, and automatically pull temperature readings
to check asset’s condition
• Show instructions to fix it and recommend the experts to help
10
• Identify a Heat Source
• Point the camera at an asset, analyze the real-time video stream, and
overlay the image with indicators showing the hear source
• Analyze the heat pattern change on temporal domain => computation
in real-time

11. IBM :: IBM Confidential :: ©2015 IBM Corporation
Do More Beyond Human Eye
11
• Image recognition with Visual and Thermal Cameras
• leverage mobile computation for real-time applications
Visible Image
from smartphone
Thermal and visible
images from external
camera
Smartphone with
external camera

12. IBM :: IBM Confidential :: ©2015 IBM Corporation
Image Overlap — Thermal and Visual Images
12
(b)Thermal image (c)Visible image
(d)Overlapping thermal
and visible images
without displacement
(a)External camera
• Problem: the mismatch in overlapping images

13. IBM :: IBM Confidential :: ©2015 IBM Corporation
Image Overlap — Thermal and Visual Images
13
(e)Overlapping with
displacement d1 to focus
on the closer mug
(f)Overlapping with
displacement d2 to focus
on the farther mug
• Different object in different depth has its own displacement d

14. IBM :: IBM Confidential :: ©2015 IBM Corporation
Thermal-Visible Image Alignment
14
• The image texture is different
• Conventional image registration using key features is not working
• An observation: temperature distribution is correlated to object shape
• => Minimize the difference between edge maps
Et(x,y) is the edge map of thermal image
E ̃d(x,y) is the edge map of visible image
with a displacement d

15. IBM :: IBM Confidential :: ©2015 IBM Corporation
Demo — Thermal-Visual Image Alignment
15

16. IBM :: IBM Confidential :: ©2015 IBM Corporation
Asset Inspection with Thermal and Visual Images
16

17. IBM :: IBM Confidential :: ©2015 IBM Corporation
Heat Source and Pattern Detection
17
frame m
frame m+1
frame n frame t
frame n+1 frame t+1
(a)Lighting source (b)Spot heating source (c)Strip heating source
• An application scenario
• field workers would like to inspect heating system and understand the
heat distribution in every corner of a house
• predefine some common patterns with distinguishable traits

18. IBM :: IBM Confidential :: ©2015 IBM Corporation
Detection Criteria for Heat Patterns
18
• Light Source
• A region with higher temperature than its surroundings is detected,
whose luminance distribution on visible video frames is high as well
• Spot Heating Source
• A region with higher temperature than its surroundings is detected, whose
distribution like a circle and stable over a period of time, and the luminance
distribution on visible video frames is not higher than its surroundings.
• Strip Heating Source
• A region with higher temperature than its surroundings is detected, whose
distribution like a strip and stable over a period of time, and the luminance
distribution on visible video frames is not higher than its surroundings

19. IBM :: IBM Confidential :: ©2015 IBM Corporation
Screenshot of Heat Source Detection and Pattern Analysis
19
Cancel Heat Camera List
1
3
2
Cancel
Spot Heat
Detect a spot heat with 135 F in the
center. Heat flow slightly goes up.
Checking Point 2

20. IBM :: IBM Confidential :: ©2015 IBM Corporation
Conclusion
20
• Applications with visible and thermal image processing
• image recognition for helping field workers identify asset model,
inspect asset condition, and show instructions step-by-step
• thermal and visual image processing for heat pattern detection and
operation condition check
• Mobile analytics is a huge demand and can help users
to solve problems
• More and more mobile analytics on the way
• viewpoints from research to product
• viewpoints from demand to research

21. IBM :: IBM Confidential :: ©2015 IBM Corporation
Thanks for Your Attention
21