This document describes a proposed method for estimating sky view factor (SVF) using semantic segmentation with deep learning networks. Specifically: - It develops a system using SegNet and U-Net deep learning models to perform pixel-wise semantic segmentation of sky and non-sky areas from images to calculate SVF ratios. - The system was trained on 300 manually segmented images and tested on 100 fisheye photographs, achieving 98% accuracy in estimating SVF under different sky conditions. - Future work is needed to apply the system to live video streams rather than static images. The method provides an efficient, high-precision way to estimate important urban environmental metrics like SVF.

1. 大阪大学 工学研究科 環境・エネルギー工学専攻
環境設計情報学領域 M1
28H16084 朱閲晗

2. 大阪大学 工学研究科 環境・エネルギー工学専攻
環境設計情報学領域 M1
28H16084 朱閲晗

3. Contents
1. Introduction
2. Literature Review
3. Proposed Method
4. System Development and Experiments
5. Conclusion
4

4. 5
1. Introduction

5. 6
Introduction
Urban visual environment evaluation index
Important criteria to measure the achievements of urban
construction
Urban environmental
management
Planar
Height regulationGreen coverage rate
Green view index (GVI)
Sky view factor (SVF)

6. 7
Introduction
Application of SVF in building design
Sky view factor (SVF)
Ratio of radiation received by a planar surface from the sky
to that received by the entire hemispheric radiating
environment
Efficient, low cost, high precision, easy to operate, rapid
SVF estimation method is necessary.
Definition of sky view factor
Sb(Building projected area)

7. 8
2. Literature Review

8.  Steyn (1980) verified the theory of using fisheye photos to correctly
estimate SVF. However, at that time, this theory was unable to perform
actual application because limited by technology. Afterward, with the
development of photographic equipment and computer technology, the
practical application of this theory had been achieved by manual
operation.
9
Literature Review
 Honma et al. (2009) proposed a new algorithm to estimate the SVF by
calculating the solid angles of urban landscapes based on computational
geometry.
 Grimmond et al. (2001) proposed a method based on plant analysis
software to measure automatically diffuse non‐interceptance (DIFN) light
by using a fisheye optical sensor. From another perspective, he proved that
the way to estimate SVF by using fisheye camera image is efficient.
Honma, Y., Kurita, O., Suzuki, A.:2009, An Analysis of Urban Landscapes with Respect to Solid Angles based on Computational Geometry Algorithms,
Journal of the City Planning Institute of Japan 2009 Volume 74 Issue 643 Pages 2035-2042.
Steyn, G. :1980, The calculation of view factors from fisheye‐lens photographs: Research note. pp. 254-258.
Grimmond, B., Potter, K., Zutter, N., Souch, C. :2001, Rapid methods to estimate sky‐view factors applied to urban areas. International Journal of
Climatology, 21(7), 903-913.

9. 10
Objective
Development of estimating SVF as visual environment
 Efficient, low cost, high precision, easy to operate, rapid
 Use semantic segmentation using deep learning network
 Train the model in advance to avoid each pre-processing

10. 11
3. Proposed Method

11. 12
Semantic Segmentation and Deep Learning
Caffe
Tensorflow
Chainer
Theano
Keras
Torch
Mxnet
……
Deep Learning network
Frameｗork
Programming Language
C++, Python,,,,
 Semantic segmentation defines the process of associating each
pixel of an image with a class label.
 Recently, image semantic segmentation systems based on deep
learning have attracted considerable research attention.

12. 13
Proposed Method
 A new SVF estimation system by training and constructing a deep
learning network for semantic segmentation of natural pixels in
landscape images
 Segnet (Badrinarayanan et al. 2017) and U-Net (Olaf et al. 2015) are selected
as the basic network.
 The focus of the entire system is how to build a deep learning
network to perform accurate semantic segmentation operations at
a high processing speed in real-time.
Badrinarayanan, V., Kendall, A. and Cipolla, R.: 2017, SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation,
IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2481–2495.
Olaf R., Philipp F., Thomas B.: 2015, U-Net: Convolutional Networks for Biomedical Image Segmentation, arXiv:1505.04597 [cs.CV]

13. 14
System Flow
Mobile
terminal
Step4
Transfer segmented
image or video to mobile
terminal
Step2
Transfer visual data to server
 Laptop
 Tablet
Statistic
Sky area
Urban
Environment
Step1
Capturing visual data
(Still images/Video)
Step3
Segmentation
The transferred
data is processed
at the same time,
and each region is
generated by the
element
Server

14. 15
System Flow

15. 16
4. System Development and
Experiments

16. 17
SVF Estimation by Segnet
(1)𝑆𝑉𝐹 =
𝑃𝑖𝑥𝑒𝑙 𝑆𝑘𝑦
𝑃𝑖𝑥𝑒𝑙 𝑇𝑜𝑡𝑎𝑙
×
4
𝜋
Sky view segmentation
by Segnet network

17. 18
Learning Images for U-Net
 We used a 360°camera to collect and prepare 300 original images
including sky and buildings
 We segmented the images manually into two parts: sky area and the others
to get the label mask images. In addition, the original images were in three
channels and the label mask images were 8-bit gray scale in a single
channel.
 Both 300 original images and their 300 label mask images were prepared.

18. 19
Inspection of Semantic Segmentation Training
 During the training process, after 30 epochs, the accuracy and current loss
of our model appeared to be smooth.
 At the 19th epoch, abnormal fluctuations occurred in the accuracy curve;
We filtered the training pictures that caused the fluctuations and continued
the training.
 The accuracy curve started to converge at a high level. In contrast, the
average loss curve started to converge at a low level.
Average Validation Accuracy vs Epochs Average Loss vs Epochs

19. 20
Verification method
Scene in HoloLens
Scene in real world
Semantic segmentation accuracy of our system to estimate SVF was verified
by the following method.
1. Create correct images
2. Calculate the red region in the
images that were outputted by
the proposed method
3. Combined for comprehensive
verification
（a）Original image （b）Correct image (by photoshop)
（c）Output by proposed method （d）Composite image
（a）Original image （b）Correct image (by photoshop)
（c）Output by proposed method （d）Composite image
(a) Original (b) Correct image (Photoshop)
(c) Output by
Proposed Method
(d) Composite Image
Definition
Yellow (Y) Accurate extracted pixel
Black (B) Accurate unextracted pixel
Red (R) Over-extracted pixel
Green (G) Unextracted area pixel
Calculation
formula
Extract accuracy
rate [%]
Y
X
× 100
Unextracted
accuracy rate [%]
B
X
× 100
Accuracy rate [%] (
Y
X
+
B
X
) × 100
Over extracted rate
[%]
R
X
× 100
Unextracted
inaccuracy rate [%]
G
X
× 100
Inaccuracy rate [%] (
R
X
+
G
X
) × 100
Formulas of accuracy rate and inaccuracy rate
Definition of each color in composite image

20. 21
Verification Experiment Method
Scene in HoloLens
Scene in real world
1. 100 fisheye photographs including sky and buildings are manually
shot as the original images.
2. Considering the difference in sky color distribution under different
weather conditions, these 100 fisheye photographs contain 3 types
of sky: Sunny weather, cloudy and rain sky.
 Date: 29th November, 1st December, 3rd December, 2018
 Weather: Cloudy
 Location: Osaka University, Suita campus
 Camera: RICOH THETA V
 Photograph parameter: 1920p
 Amount of photographs: 100
 Vertical distance of lens: 1.6 m from the ground
3. Upload them to the server, and Segment the transferred images
into the sky area part and the others with the estimation results of
SVF.

21. 22
Results
Scene in HoloLensComparison between correct image and the output image generated using the proposed method
(A) Sunny weather (B) Rainy (C) Cloudy
(a) Original (b) Correct (Photoshop)
(c) Proposed Method (d) Composite Image
(a) Original (b) Correct (Photoshop) (a) Original (b) Correct (Photoshop)
(c) Proposed Method (d) Composite Image (c) Proposed Method (d) Composite Image
Extract accuracy rate [%] 45.7
Unextracted accuracy rate [%] 53.2
Accuracy rate [%] 98.9
Over extracted rate [%] 0.9
Unextracted inaccuracy rate [%] 0.2
Inaccuracy rate [%] 1.1
Accuracy and inaccuracy rate of SVF estimation

22. 23
5. Conclusion

23. 24
Conclusion
 We developed an easy-to-operate visual estimation system of sky view
factor at high precision without detailed pre-process using semantic
segmentation technique of deep learning.
 It demonstrated the same precision as previous studies (98%).
Conclusion
Future Work
 The system can only estimate the SVF from a still image or a video; if the
data source is from a live web camera, it will not work well.
Acknowledgments
 This research has been partly supported by JSPS KAKENHI Grant Number
JP16K00707.