Can we use information from social media and crowdsourced images to detect smoke and assist rescue forces? While there are computer vision methods for detecting smoke, they require movement information extracted from video data. In this paper we propose SmokeBlock: a method that is able to segment and detect smoke in still images. SmokeBlock uses superpixel segmentation and extracts local color and texture features from images to spot smoke. We used real data from Flickr and compared SmokeBlock against state-of-the-art methods for feature extraction. Our method achieved performance superior than the competitors, for the task of smoke detection. Our findings shall support further investigations in the field of image analysis, in particular, concerning images captured with mobile devices.

1. Unveiling smoke in social
images with the SmokeBlock
approach
Jessica Andressa de Souza (speaker)
jessicasouza@usp.br
Mirela T. Cazzolato, Marcos V. N. Bedo, Alceu F. Costa, Caetano
Traina Jr., Jose F. Rodrigues Jr. and Agma J. M. Traina
SAC 2016 – 31 𝑠𝑡
ACM Symposium on Applied Computing – Pisa, Italy (April 4-8, 2016)

2. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Outline
Introduction
Proposed Approach: SmokeBlock
Experimental Results
Conclusions
2

3. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
RESCUER Project
• The RESCUER project is a Brazil-Europe
consortium
• Goal: develop solutions to improve the decision-
making process in disaster situations
 Industrial plants
 Densely populated area
 Crowded events
3

4. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
RESCUER Project
4
The user may also upload
multimedia content such as
photo and video
Multimedia
data are
automatically
analyzed
http://www.rescuer-project.org
Smartphone user sends data
(including photos) of the
situation
Uploaded
multimedia data is
automatically
analyzed

5. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Research Question
Given a collection of RESCUER user reports
containing images of the situation, can we
automatically detect the presence of smoke?
5
Smoke
Detection

6. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Problem Definition
6
Given a set of images from social
media/crowdsourcing, find the subset
of images that depict smoke while
minimizing the rate of false-positives.

7. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Smoke Detection - Challenges
• Absence of movement
• Harder than smoke detection in videos
• Smoke may have different colors:
• Depends on: temperature, material, illumination
• Smoke may be transparent
• May be visually similar to clouds, mist, rain
7

8. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Outline
Introduction
Proposed Approach: SmokeBlock
Experimental Results
Conclusions
10

9. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Proposed Method: SmokeBlock
• Input:
• Unlabeled image
• Set of labeled super-pixels
• Output:
• Segmented image
• Global classification
(Smoke vs. Not-smoke)
11
…
Smoke

10. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016 12
SmokeBlock: Pipeline
Superpixel
Extraction
Feature
Extraction
Superpixel
Classification
Global Smoke
Detection
• Problem: not all regions of the input
image may contain smoke
• Solution: break the image into visually
homogeneous regions
• Algorithm: SLIC

11. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016 13
SmokeBlock: Pipeline
Superpixel
Extraction
Feature
Extraction
Superpixel
Classification
Global Smoke
Detection
• From each super-pixel we extract a
feature vector
• Numerical representation of low-level
visual content
V1 = (0.31, 0.41, 0.59, …, 0.26)
V2 = (1.41, 4.21, 3.56, ..., 2.37)
VN = (0.31, 0.41, 0.59, …, 0.26)

12. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016 14
SmokeBlock: Pipeline
Superpixel
Extraction
Feature
Extraction
Superpixel
Classification
Global Smoke
Detection
Why extract features from superpixels?
1. Parts of the input image may not
contain smoke
• Global features are not adequate
2. By analyzing group of pixels we can
extract texture features
• Not possible when analyzing individual
pixels

13. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016 15
SmokeBlock: Pipeline
Superpixel
Extraction
Feature
Extraction
Superpixel
Classification
Global Smoke
Detection
• SmokeBlock uses a binary classifier that
predict the class of a superpixel given
the feature vectors
• Classes: smoke vs. not-smoke
Training set: manually annotated superpixels
from different images

14. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016 16
SmokeBlock: Pipeline
Superpixel
Extraction
Feature
Extraction
Superpixel
Classification
Global Smoke
Detection
• Goal: decide if image contains smoke
• Application: find all images uploaded to
RESCUER that contain smoke
• Naive approach:
• Classify an image as positive (has smoke) if
at least one superpixel is classified as
containing smoke
• Problem: if a single superpixel is wrongly
classified, then the all the image will be
wrongly classified

15. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016 17
SmokeBlock: Pipeline
Superpixel
Extraction
Feature
Extraction
Superpixel
Classification
Global Smoke
Detection
VGlobal = (0.31, 0.41, …, 0.59)
Classifier
Class: Smoke

16. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Outline
Introduction
Proposed Approach: SmokeBlock
Experimental Results
Conclusions
18

17. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Flickr-Smoke Dataset
• Downloaded a set of images from Flickr API:
• Image retrieval with textual descriptors as `smoke
fire´ and `smoke forest´
• 7 subjects annotated the images as containing or
not traces of smoke
• 832 images were labeled as smoke, and 834 as
non-smoke
19

18. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Flickr-Smoke Dataset
• Dataset available at:
github.com/alceufc/SmokeBlockDataset
20
{smoke}
{not smoke}

19. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Outline
Introduction
Proposed Approach: SmokeBlock
Experimental Results
Visual features
Smoke segmentation
Global classification
Conclusions
21

20. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Experimental Analysis
• Identification of the two best feature
extraction methods for smoke
detection
22
1202 superpixels
Higher is
better
Lower is better

21. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Experimental Analysis
• Identification of the two best feature
extraction methods for smoke
detection
23
1202 superpixels
Higher is
better
Lower is better

22. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Experimental Analysis
• Identification of the two best feature
extraction methods for smoke
detection
24
1202 superpixels
Higher is
better
Lower is better

23. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Outline
Introduction
Proposed Approach: SmokeBlock
Experimental Results
Visual features
Smoke segmentation
Global classification
Conclusions
25

24. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Evaluation: smoke segmentation
• We compared SmokeBlock to:
• Celik, Chen
• Both methods classify individual pixels based
on color information
26

25. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
• 1st output: smoke segmentation
27
Not smoke
(a) Unlabeled image (b) SmokeBlock (c) Celik (d) Chen
positivenegative
Evaluation: smoke segmentation

26. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Outline
Introduction
Proposed Approach: SmokeBlock
Experimental Results
Visual features
Smoke segmentation
Global classification
Conclusions
28

27. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Evaluation: Global classification
Problem: Given a set of unlabeled images,
assign a class (smoke vs. not-smoke) to each
image.
• Celik and Chen are not able to classify
images
• Baselines:
• Global feature vectors
29

28. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016 30
Higher is
better
SmokeBlock was the most accurate approach
Evaluation: Global classification

29. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Outline
Introduction
Proposed Approach: SmokeBlock
Experimental Results
Conclusions
35

30. ACM SAC 2016 (Pisa, Italy) SmokeBlock April 04-08, 2016
Conclusions
• We designed a new approach for smoke
detection: SmokeBlock
• More accurate than state-of-the-art approaches
when analyzing still images
• Visual feature evaluation:
• We analyzed different feature extraction methods
w.r.t. smoke detection accuracy
• Flickr-Smoke dataset:
• Annotated and publicly available dataset
36

31. Unveiling smoke in social
images with the SmokeBlock
approach
Mirela T. Cazzolato, Marcos V. N. Bedo, Alceu F. Costa,
Jessica A. de Souza, Caetano Traina Jr., Jose F. Rodrigues Jr.
and Agma J. M. Traina
SAC 2016 – 31 𝑠𝑡
ACM Symposium on Applied Computing – Pisa, Italy (April 4-8, 2016)
jessicasouza@usp.br
Thank You!