Personalized Real-Time Virtual Tours in Places with Cultural Interest

Personalized Real-Time Virtual Tours in
Places with Cultural Interest
Emmanouil Skondras1
, Konstantina Siountri1,2
, Angelos Michalas3
, Dimitrios D. Vergados1
1
University of Piraeus, Greece, Email: {skondras, ksiountri, vergados}@unipi.gr
2
University of Aegean, Greece, Email: ksiountri@aegean.gr
3
Technological Educational Institute of Western Macedonia, Greece, Email:
amichalas@kastoria.teiwm.gr
ABSTRACT
Virtual tours using drones enhance the experience the users perceive from a place with cultural interest.
Drones equipped with 360o
cameras perform real-time video streaming of the cultural sites. The user
preferences about each monument type should be considered in order the appropriate flying route for the
drone to be selected. This paper describes a scheme for supporting personalized real-time virtual tours in
sites with cultural interest using drones. The user preferences are modeled using the MPEG-21 and the
MPEG-7 standards, while Web Ontology Language (OWL) ontologies are used for the description of the
metadata structure and semantics. The Metadata-aware Analytic Network Process (MANP) algorithm is
proposed in order the weights about the user preferences for each monument type to be estimated.
Subsequently, the Trapezoidal Fuzzy Topsis for Heritage Route Selection (TFT-HRS) algorithm
accomplishes ranks the candidate heritage routes. Finally, after each virtual tour, the user preferences
metadata are updated in order the scheme to continuously learn about the user preferences.
Keywords: Digital Culture, Virtual Tours, Personalization, Drones, Web Ontology Language (OWL),
MPEG-21, MPEG-7, 5G Networks
INTRODUCTION
Virtual tourism (Beck J. & Egger R., 2018) reduces time or spatial limitations of real tourism. Services such
as 360o
video streaming (Qian,F., Ji,L., Han,B. & Gopalakrishnan,V., 2016), 3D animation (Bustillo,A.,
Alaguero,M., Miguel,I., Saiz,J. M. & Iglesias, L.S., 2015), Augmented Reality (AR) (Marques,L.F.,
Tenedório,J.A., Burns,M., Romão,T., Birra,F., Marques,J. & Pires,A., 2017) and Mixed Reality (MR)
(Debandi,F., Iacoviello,R., Messina,A., Montagnuolo,M., Manuri,F., Sanna,A. & Zappia,D., 2018) are used
to construct a virtual world for the user. Drones (Mirk,D. & Hlavacs,H., 2015) could be used for the 360o
video capture of places with cultural interest. The video can be streamed to users in real time, enriched with
additional audio descriptions, 3D, AR or MR content. To support such services, Fifth Generation (5G)
(Akpakwu,G.A., Silva,B.J., Hancke,G.P. & Abu-Mahfouz,A.M., 2018) mobile infrastructures could be
used, providing plenty of networking, computational and storage resources. Indicatively, the enriched 360o
video is streamed to the user through a 5G Mobile Edge Computing (MEC) or Fog (Roman,R., Lopez,J. &
Mambo,M., 2018) infrastructure, which assures the satisfaction of its strict constraints in terms of
throughput, delay, jitter and packet loss. The 5G infrastructure could support heterogeneous network access
technologies, such as the 3GPP Long Term Evolution Advanced (LTE-A) (TS36.300-V13.2.0., 2016), the
IEEE 802.11p Wireless Access for Vehicular Environment Road Side Units (WAVE RSUs) (1609.3-2016,
2016) and the IEEE 802.16 WiMAX (802.16q-2015, 2015).

Virtual tours with drones can be used in numerous cases dealing with protection, preservation and
enhancement of tangible heritage, as well as servicing special groups of people, i.e. the elderly, children,
persons with disabilities that cannot reach inaccessible monuments. Each drone is remotely controlled by
the user or it is autonomous navigated (Kan,M., Okamoto,S. & Lee,J.H., 2018). A critical task of the
autonomous navigation service is the selection of the most appropriate flying route for the drone, as in
heritage sites where multiple monument types exist, the user preference for each type should be considered,
in order personalized user experience to be provided.
User preferences could be modeled using the MPEG-21 (Metta,S., Montagnuolo,M. & Messina,A., 2015)
standard, which defines a framework for both multimedia resources and user preferences manipulation.
MPEG-21 uses the architectural concept of the Digital Item (DI), which is a combination of resources (such
as audiovisual content), metadata (such as descriptors) and structures describing the relationships between
resources. DIs are declared using the Digital Item Declaration Language (DIDL). The MPEG-21 Digital
Item Adaptation (DIA) architecture and the MPEG-7 (Park,K.W., Hong,H.K., Lee,M., & Lee,D.H., 2014)
Multimedia Description Schemes (MDS) for content and service personalization provide a Usage
Environment that models user preferences. Additionally, description of the multimedia that enriches the
360o
video is required in order the appropriate content to be retrieved and placed in certain places into the
video. This description could be performed using the MPEG-7 standard.
Both MPEG-21 and MPEG-7 (MPEG-21/7) metadata are stored in XML format allowing efficient
indexing, searching and filtering. The metadata structure could be described using Web Ontology Language
(OWL) (Sengupta,K. & Hitzler,P., 2014). OWL is a knowledge representation language used for composing
ontologies. The ontologies are described in OWL documents by defining classes, attributes and individuals.
Classes are collection of concepts, attributes are properties of classes and individuals represent the objects
of a class. Thus, the system will be intelligent enough to autonomous retrieve information of the metadata
files without requiring additional information about the metadata structure from the user. OWL ontologies
could be queried using the SPARQL (Peng,P., Zou,L., Özsu,M.T., Chen,L. & Zhao,D., 2016) language.
SPARQL features include conjunctive or disjunctive patterns as well as value filters. Subsequently, when
the system has learned about the metadata structure, it can query them using Mpeg Query Format (MPQF)
(Tous,R. & Delgado,J., 2010) queries. MPQF specifies the message format for querying the metadata
repositories.
In this paper, a scheme for supporting real-time virtual tours in places with cultural interest using drones is
proposed. The user preferences for several monument types are modeled using the MPEG-21/7 standards,
in order a personalized virtual tour experience to be provided to the user. Furthermore, after each virtual
tour, the user preferences are updated considering feedback information received from the user according
to the experience he perceived from the tour. Thus, the scheme’s knowledge about user preferences is
continuously updated. A mayor advantage of the proposed scheme is that the use of standardized metadata
to manage user preferences ensures interoperability with third-party systems that use the same standards.
The rest of the paper is organized as follows: The “Background” section discusses the related work, while
the “Modeling the User Preferences” section describes the proposed methodology for modeling the user
preferences. Subsequently, the “Simulation Setup and Results” section presents a case study of the proposed
scheme in a simulation environment. Finally, the “Conclusion” section concludes the discussed work.
BACKGROUND
The rapid increase of multimedia users has challenged the academic and industrial communities into the
design of multimedia personalization solutions. Several systems have been proposed to address the complex
task of making recommendations considering multiple parameters, including the model of user preferences
in some cases (Marlin,B.M., 2004). Indicatively, in (Santos,F., Almeida,A., Martins,C., Oliveira,P., &
Gonçalves,R., 2016) a recommendation system that considers the existence of several Points of Interest
(PoIs) in an area is described. User profiles are defined considering parameters including user gender, age

or nationality, as well as possible user’s disabilities (e.g. vision or motion disabilities). Subsequently, PoIs
are recommended according to each user profile. Accordingly, in (Zhao,Y., Wang,S., Zou,Y., Ng,J. &
Ng,T., 2017) a machine learning approach is proposed. Specifically, the discussed approach is based in
historical data to learn the user preferences. Subsequently, considering the estimated user preferences,
personalized data retrieval services are provided to the users. Also, in (Kovacikova,T., Petersen,F.,
Pluke,M., Alvarez,V.A., Bartolomeo,G., Frisiello,A. & Cadzow,S., 2009) a wide view of personalization
and user profiles, that makes the preferences available to a range of services and devices, is discussed. A
user profile that stores the user preferences, context of use and other information capable to deliver user
experiences that describe individual user needs, is described. It is based upon the fact that user needs depend
on the context and current situation, (e.g. “the user is at home” or “the user is in the Car”).
Multi Attribute Decision Making (MADM) weighting methods could also be used to model the preferences
of multimedia users, by performing a set of pairwise comparisons between multiple criteria, sometimes
even contradictory. Widely used methods include the Analytic Hierarchy Process (AHP) (Ho,W., & Ma,X.,
2017) and the Analytic Network Process (ANP) (Lee,J., Jun,S., Chang,T.W. & Park,J., 2017), which
accomplish satisfactory results in terms of user preferences (or weights) estimation. However, these
methods do not apply any well-defined standard for the storage, manipulation or future use of their results.
Thus, limited interoperability with other current or future systems could be observed, since the
compatibility of the obtained user preferences values depends on the specific design of each service which
usually does not follow well-defined data manipulation standards.
As described in (Pavlidis,G., 2018), the use of recommendation systems in cultural heritage applications
has obtain increased interest. Indicatively, in (Frikha,M., Mhiri,M. & Gargouri,F., 2017) a scheme for
personalized touristic recommendations is proposed. The scheme defines a set of trusted friends using a
calculation method which considers the interactions between the users and his friends in social media.
Subsequently, the user preferences are modeled using an ontology, while the scheme proposes cultural
places to the user considering both his preferences and the preferences of his friends. Furthermore, some
recommendation systems deal with the specific case of museums (Keller,I. & Viennet,E., 2015). A work
that belong to this category is described in (Osche,P. E., Castagnos,S., Napoli,A. & Naudet,Y., 2016),
where a user model for museum items recommendation is proposed. The model considers parameters such
as congestion points inside the museum, average time the user spends in each item or the user’s distance
from each exhibit, in order a list of items to be recommended to the user with the assumption that he prefers
to see them.
Although the aforementioned works propose useful solutions, they lack in the interoperability of their data
with third-party systems, since each work defines individually its data sets. To address the data
interoperability issues, well-defined standards could be applied. Indicatively, in (Kohncke,B. &
Balke,W.T., 2010) the concept of Universal Multimedia Access (UMA) is discussed. Specifically, both
user preferences and terminal device characteristics could be considered in order the personalized
multimedia content to be adapted according to terminal’s specifications. For example, video content could
be transmitted to the terminal device in the most appropriate resolution considering the device’s screen
capabilities to enhance the Quality of Experience (QoE) the user perceives from the video service. To
accomplish these tasks, the authors consider the MPEG-21/7 standards, as well as the corresponding OWL
ontology for the modeling of the user preferences. Furthermore, in (Cobos,Y., Sarasua,C., Linaza,M.T.,
Jimenez,I. & Garcia,A., 2008), an MPEG-7 based Multimedia Retrieval System for Film Heritage is
presented. The multimedia content has been indexed considering the MPEG-7 standard. An MPEG-7
Compliant OWL ontology has been developed to fulfill the requirements of the system. This ontology has
been instantiated so that the retrieval process could be handled.
As it can be observed, the described works either propose specialized solutions for accomplishing
personalization without using multimedia standards, or apply metadata standards without implementing an
optimal underlying algorithm for their efficient manipulation. The main contribution of the model proposed

in this work, is the combination of the strong aspects of MADM algorithms, MPEG-21/7 standards and
OWL ontologies in order to perform the recommendation task.
MODELING THE USER PREFERENCES
The proposed methodology for the modeling of the user preferences consists of two subprocesses, namely
the calculation of the user preferences weights, which is performed before a virtual tour, and the update of
user preferences which is performed after each virtual tour.
Calculate user preferences weights
MPEG-21 and MPEG-7 standards are used for the description of user preferences about monument types.
Specifically, in this paper an extension of the Analytic Network Process (ANP) is proposed. The method is
called Metadata-based ANP (MANP) and it is composed of six major steps:
1. Model construction and problem structuring: The problem is analyzed and decomposed into a
rational system, consisted of nodes (or cluster elements). Arcs are also included, denoting the
dependencies between the cluster elements.
2. Construction of Evaluation Matrix: The user preference values described into the MPEG-21/7
metadata are organized into the matrix E which can be expressed as:
𝑷 = | 𝒑 𝟏 ⋯ 𝒑𝒊 ⋯ 𝒑 𝒛| (1)
where 𝒑𝒊 is obtained from the user metadata and represents the preference value of the user for the
monument type i. Also, z represents the count of the monument types. Each 𝒑𝒊 obtains a value
between 1 and 9, by applying the Saaty’s nine-point importance scale (Table 1).
Table 1. The interest scale considered in the MPEG-21/7 user metadata
Possible 𝒑 𝒖,𝒊 values Definition for 𝒑𝒊 values Saaty’s definition about the Importance
1 The Less Interest Equal Importance
3 Low Interest Moderate Importance
5 Medium Interest Strong Importance
7 High Interest Very Strong Importance
9 The Most Interest Extreme Importance
2, 4, 6, 8 Intermediate Values Intermediate Values
3. Pairwise comparison matrices and priority vectors: The pairwise comparison matrix A is derived
from the aforementioned importance scale. The standard form of the matrix is expressed as follows:

𝑨 =
|
|
|
|
𝟏 … 𝒂 𝟏𝒋 … 𝒂 𝟏𝒏
⋮ ⋮ ⋮
𝒂𝒊𝟏 … 𝟏 … 𝒂𝒊 𝒏
⋮ ⋮ ⋮
𝒂 𝒏𝟏 … 𝒂 𝒏𝒋 … 𝟏 |
|
|
|
(2)
where n denotes the number of the cluster elements. The value of each aij cluster element is
calculated using the following formula:
𝒂𝒊𝒋 = 𝒑𝒊/𝒑𝒋 (3)
4. Supermatrix formation: During this step, the MANP supermatrix is constructed to represent the
dependencies between the clusters. It is a partitioned matrix, where each submatrix represents a
relationship between two clusters. To construct the supermatrix the local priority vectors obtained
in Step 2 are grouped and placed in the appropriate positions in a supermatrix based on the flow of
influence from one cluster to another, or from a cluster to itself, as in the loop. Then, the supermatrix
is transformed to the weighted supermatrix. Finally, the weighted supermatrix is raised to limiting
powers until all the entries converge to calculate the overall priorities, and thus the cumulative
influence of each element on every other element with which it interacts is obtained. At this point,
all the columns of the new matrix, the limit supermatrix, are the same and their values show the
global priority of each element of the network. Indicatively, if we assume a network with n clusters,
where each cluster Qk; k=1; 2;…;n; and has mn elements, denoted as qk1;qk2;…;qkmk, then the
standard form for a supermatrix can be expressed as:
𝑸 𝟏 … 𝑸 𝒌 … 𝑸 𝒏
𝒒 𝟏𝟏…𝒒 𝒎 𝟏
𝒒 𝒌𝟏…𝒒 𝒎 𝒌
𝒒 𝒏𝟏…𝒒 𝒎 𝒏
𝑾 =
𝑸 𝟏
𝒒 𝟏𝟏
⋮
𝒒 𝟏𝒎 𝟏
⋮
𝑸 𝒌
𝒒 𝒌𝟏
⋮
𝒒 𝒌𝒎 𝒌
⋮
𝑸 𝒏
𝒒 𝒏𝟏
⋮
𝒒 𝒏𝒎 𝒏
|
|
|
𝒘 𝟏𝟏 … 𝒘 𝟏𝒌 … 𝒘 𝟏𝒏
⋮ ⋮ ⋮ ⋮ ⋮
𝒘 𝒌𝟏 … 𝒘 𝒌𝒌 … 𝒘 𝒌 𝒏
⋮ ⋮ ⋮ ⋮ ⋮
𝒘 𝒏𝟏 … 𝒘 𝒏𝒌 … 𝒘 𝒏 𝒏
|
|
|
(4)
5. Obtain the priority weights: During this step, the priority weights of the alternatives can be found
in the corresponding columns of the normalized limited supermatrix.
The entire process is introduced in the following algorithm:
Input: User preference values described into MPEG-21/7 metadata
Output: Weights about monument types

function Problem_structuring(monument_types):
Model the relations between monument_types;
function Evaluation_matrix_construction(relations between monument_types):
Construct the Evaluation_matrix;
function Pairwise_comparison_matrix_construction(Evaluation_matrix):
Construct the Pairwise_comparison_matrix;
function Supermatrix_construction(Pairwise_comparison_matrix):
Derive the Supermatrix;
function Weighted_Supematrix_construction(Supermatrix):
Derive the Weighted_Supematrix;
function Limited_Supematrix_construction(Weighted_Supematrix):
Derive the Limited_Supematrix;
Indicate the weights about monument types;
Update user preferences metadata according to user feedback
After each virtual tour, the user preference metadata are updated considering the user feedback about each
monument type. Specifically, the user is asked to evaluate each monument type presented in the tour,
considering the nine-point importance scale. Subsequently, the new user preference 𝒑𝒊,𝒏𝒆𝒘 about each
monument type is calculated using the following formula:
𝒑𝒊,𝒏𝒆𝒘 = ⌈(𝒑𝒊,𝒐𝒍𝒅 + 𝒑𝒊,𝒇)/𝟐⌉ (5)
where 𝒑𝒊,𝒐𝒍𝒅 is the previous 𝒑𝒊 value and 𝒑𝒊,𝒇 is the feedback value about the corresponding monument type
the user sent after his virtual tour. Thus, the user preference for each monument type is updated according
to the user feedback and could be considered for his future virtual tours.
The entire process is introduced in the following algorithm:
Input: User feedback about each monument type
Output: Updated MPEG-21/7 metadata about user preference
function Update_User_Preferences(user_feedback):
Update the user preferences stored into the MPEG-21/7 metadata;
SIMULATION SETUP AND RESULTS
In this section, the proposed scheme is evaluated. Firstly, in the simulated environment is described.
Subsequently, a case study is presented where the scheme’s functionalities and the corresponding results
are analyzed.
Simulation setup
The proposed scheme is applied to a fully virtualized 5G architecture which provides Smart Cultural
Heritage as a Service (SCHaaS) services (Figure 1) (Siountri,K., Skondras,E. & Vergados,D.D., 2018). The
architecture is simulated using the Network Simulator 3 (NS3) and includes a Cloud and a Fog
infrastructure. The Cloud includes a set of Virtual Machines (VMs), while each VM hosts audiovisual
material about monuments (audio tour files and visual models), MPEG-21/7 metadata that describe the
audiovisual material, and OWL ontologies describing the structure and the semantics of the aforementioned
metadata. Accordingly, the Fog infrastructure includes a heterogeneous network access environment

consisting of LTE and WiMAX Macrocells and Femtocells, as well as of WAVE RSUs. Inside the Fog
coverage area, a number of Ancient, Byzantine, Modern and Natural Beauty monuments exist. A Software
Defined Network (SDN) controller provides centralized control of the entire architecture (Kreutz,D.,
Ramos,F.M., Verissimo,P.E., Rothenberg,C.E., Azodolmolky, S. & Uhlig, S., 2015).
Figure 1. The simulated architecture
Case Study
The case where 5 users need to perform a personalized virtual tour using a drone is considered. Initially the
preferences of each user for each type of monument are modeled in the user equipment (UE) using the
MPEG-21/7 standards. In this case, the produced metadata are called User Preferences Metadata (UPM).
Indicatively, the UPM metadata that describe the preferences of user 1 about the considered monument
types are as follows:
<mpeg21:DIDL xmlns:mpeg21="urn:mpeg:mpeg21:2002:02-mpeg21-NS">
<mpeg21:Container><mpeg21:Item><mpeg21:Descriptor>
<mpeg21:Statement mimeType="text/plain">Metadata about user preferences.
</mpeg21:Statement></mpeg21:Descriptor><mpeg21:Component>
<mpeg21:Resource mimeType="application/xml">
<Mpeg7 xmlns="http://www.w3.org/2000/XMLSchema-instance" type="complete">
<UserPreferences>
<UserIdentifier protected="true"><UserName>User 1</UserName>

</UserIdentifier>
<UsagePreferences allowAutomaticUpdate="true">
<FilteringAndSearchPreferences protected="true">
<ClassificationPreference>
<Genre href="urn:mpeg:GenreCS" preferenceValue="9">
<Name>Ancient</Name></Genre>
<Name>Byzantine</Name></Genre>
<Name>Modern</Name></Genre>
<Name>Natural Beauty</Name></Genre>
</ClassificationPreference>
</FilteringAndSearchPreferences>
</UsagePreferences>
</UserPreferences>
</Mpeg7>
</mpeg21:Resource></mpeg21:Component></mpeg21:Item>
</mpeg21:Container></mpeg21:DIDL>
Results
Each UE interacts with the Fog infrastructure, sends UPM metadata along with the corresponding OWL
ontology (Figure 2) and requests to perform a personalized real-time virtual tour using a drone. Thereafter,
the Fog forwards the UE’s UPM and their ontology to the SDN controller, in order the most appropriate
flying route to be selected for the drone. Specifically, the SDN controller manipulates the UPM considering
the OWL ontology. This ontology describes the structure and the semantics of the UPM, providing the
required intelligence to the SDN controller to understand the UPM content.
Figure 2. The OWL ontology about User Preferences Metadata
For each user, a MANP pairwise comparison matrix is created considering the UPM metadata values about
the user preferences, as well as the MANP network model and the relations among the four monument
types depicted in Figure 3. Then, these pairwise comparison decision matrices are used to evaluate the
priority vectors of monument types and form the supermatrix. Subsequently, the weighted supermatrices
and finally the limit supermatrices are obtained. Indicatively, Table 2 presents the produced MANP

pairwise comparison matrix for user 1, while the corresponding weighted and limited supermatrices are
presented in
Table 3 and Table 4, respectively. Thus, the weights that concern the users’ preferences for each monument
type are estimated. As it is presented in Figure 4, user 1 mostly prefers the Ancient monuments, user 2
prefers both Byzantine and Modern monuments, user 3 prefers Natural Beauty monuments, user 4 prefers
Modern monuments and user 5 prefers Byzantine monuments.
Figure 3. The MANP network model and the relations among the monument types
Table 2. The MANP pairwise comparison matrix produced according to user 1 preferences metadata
Ancient Byzantine Modern Natural Beauty
Ancient 1 3 3 9
Byzantine 0,333 1 1 3
Modern 0,333 1 1 3
Natural Beauty 0,111 0,333 0,333 1
Table 3. The MANP weighted supermatrix for user 1
Ancient 0,562579 0,562588 0,562588 0,562588
Byzantine 0,187479 0,187482 0,187482 0,187482
Modern 0,187479 0,187482 0,187482 0,187482
Natural Beauty 0,0624619 0,0624473 0,0624473 0,0624473
Table 4. The MANP limited supermatrix for user 1
Ancient 0,562583 0,562583 0,562583 0,562583
Byzantine 0,187481 0,187481 0,187481 0,187481
Modern 0,187481 0,187481 0,187481 0,187481
Natural Beauty 0,0624555 0,0624555 0,0624555 0,0624555

Figure 4. The user preferences for each Monument Type.
Subsequently, the SDN controller applies the Trapezoidal Fuzzy Topsis for Heritage Route Selection (TFT-
HRS) algorithm (Skondras,E., Siountri,K., Michalas,A. & Vergados,D.D., 2018) which is considered as a
black-box in this work. Specifically, TFT-HRS constructs a decision matrix expressing the evaluation value
of each alternative heritage route for each monument (namely the considered dataset). The evaluation values
are expressed using Interval-Valued Trapezoidal Fuzzy Numbers (IVTFNs) (Wei,S.H. & Chen,S.M., 2009)
and are obtained considering the percentage of the monument covered by the route. The linguistic terms
used are presented in Table 5, while the produced decision matrix is presented in Table 6. Subsequently,
TFT-HRS considers the MANP weights to determine the importance of each monument type for each user.
Afterwards the positive and negative ideal solutions are estimated, while the flying route with the shortest
distance from the positive ideal solution and the longer distance from the negative ideal solution is selected
for the drone.
Table 5. Linguistic terms and the corresponding IVTFNs used for the definition of the evaluation values
of each monument in each heritage route
Linguistic term Interval-Valued Trapezoidal Fuzzy Number (IVTFN)
Absolutely-Poor (AP) [(0.0, 0.0, 0.0, 0.0, 0.9),(0.0, 0.0, 0.0, 0.0, 1.0)]
Very-Poor (VP) [(0.01, 0.02, 0.03, 0.07, 0.9),(0.0, 0.01, 0.05, 0.08, 1.0)]
Poor (P) [(0.04, 0.1, 0.18, 0.23, 0.9),(0.02, 0.08, 0.2, 0.25, 1.0)]
Medium-Poor (MP) [(0.17, 0.22, 0.36, 0.42, 0.9),(0.14, 0.18, 0.38, 0.45, 1.0)]
Medium (M) [(0.32, 0.41, 0.58, 0.65, 0.9),(0.28, 0.38, 0.6, 0.7, 1.0)]
Medium-Good (MG) [(0.58, 0.63, 0.8, 0.86, 0.9),(0.5, 0.6, 0.9, 0.92, 1.0)]
Good (G) [(0.72, 0.78, 0.92, 0.97, 0.9),(0.7, 0.75, 0.95, 0.98, 1.0)]
Very-Good (VG) [(0.93, 0.98, 1.0, 1.0, 0.9),(0.9, 0.95, 1.0, 1.0, 1.0)]
Absolutely-Good (AG) [(1.0, 1.0, 1.0, 1.0, 0.9),(1.0, 1.0, 1.0, 1.0, 1.0)]
Table 6. The decision matrix of the TFT-HRS algorithm
Monument Route-1 Route-2 Route-3 Route-4 Route-5 Route-6 Route-7 Route-8 Route-9 Route-10
Ancient
Monument 1
P AG G MG AP MG MG MG AP MG
Ancient
Monument 2
AP G VG G MP AG AG AG VP AG

Ancient
Monument 3
MP MG AG G AG AG MG P VP G
Byzantine
Monument 1
MG AP MG VG G VP MP AP MP MG
Byzantine
Monument 2
AG MP AG AG VG MG AG AG VG AG
Modern
Monument 1
AG P VG G VG AP G G G P
Modern
Monument 2
MG G VG VP MG AG G VG G P
Modern
Monument 3
VG AP AG M G AG VG MP VG AP
Natural
Beauty 1
G VG G AG MG G AG AP MG MG
Natural
Beauty 2
G MP MG P AG MG P MG G AG
When the execution of the TFT-HRS algorithm is completed, the rank of each heritage route is available
(Figure 5), while the route with the higher rank is selected. Specifically, for user 1 the route-3 is selected
which provides the best fuzzy values (G, VG and AG) for the Ancient monument type. Likewise, the route-
3 is also selected for the user 2, route-5 is selected for the user 3, route-3 is selected for the user 4 and,
finally, route-7 is selected for user 5.
Figure 5. The TFT-HRS ranking for each heritage route according to each user preferences
Also, the SDN controller retrieves the corresponding audiovisual material (audio tour files and visual
models) from the Cloud using the MPEG-21/7 metadata about audio tour files and visual models.
Indicatively, the following code presents an example of MPEG-21/7 metadata content that describe an audio
tour file, while similar metadata are also used for the description of the visual models:
<mpeg21:DIDL xmlns:mpeg21="urn:mpeg:mpeg21:2002:02-mpeg21-NS"
xmlns:mpeg7="http://www.mpeg.org/MPEG7/2000">
<mpeg21:Container><mpeg21:Item> …
<mpeg7:Mpeg7><mpeg7:CreationPreferences>

<mpeg7:Title mpeg7:preferenceValue="12" xml:lang=“en”>
monument2_audio.mp3</mpeg7:Title></mpeg7:CreationPreferences>
<mpeg7:CreationInformation><mpeg7:Creation><mpeg7:Creator>
<mpeg7:Role mpeg7:href=“urn:mpeg:mpeg7:cs:RoleCS:2001:AUTHOR” />
<mpeg7:Agent xsi:type=“PersonType”>
<mpeg7:Name>
<mpeg7:GivenName>AuthorName</mpeg7:GivenName>
<mpeg7:FamilyName>AuthorSurname</mpeg7:FamilyName>
</mpeg7:Name></mpeg7:Agent></mpeg7:Creator> …
<mpeg7:Abstract>mpeg7:FreeTextAnnotation>
This audio file describes monument2
</mpeg7:FreeTextAnnotation></mpeg7:Abstract>
<mpeg7:CreationCoordinates> …
<mpeg7:Location>
<mpeg7:GeographicPosition>
<mpeg7:Point longitude="37.941780" />
<mpeg7:Point latitude="23.652443" />
</mpeg7:GeographicPosition></mpeg7:Location>
</mpeg7:CreationCoordinates></mpeg7:Creation></mpeg7:CreationInformation>
…<mpeg7:MediaLocator><mpeg7:MediaUri>audio_tour_files/monument2.mp3
</mpeg7:MediaUri></mpeg7:MediaLocator>
<mpeg7:MediaTime>
<mpeg7:MediaTimePoint>T00:00:00F100</mpeg7:MediaTimePoint>
<mpeg7:MediaDuration>T00:03:07F100</mpeg7:MediaDuration>
</mpeg7:MediaTime>
<mpeg7:MediaFormat>
<mpeg7:Content mpeg7:href="urn:mpeg:mpeg7:cs:ContentCS:2001:2">
<mpeg7:Name xml:lang="en">audio</mpeg7:Name></mpeg7:Content>
<mpeg7:Medium mpeg7:href="urn:mpeg:mpeg7:cs:MediumCS:2001:2.1.1 ">
<mpeg7:Name xml:lang="en">HD</mpeg7:Name></mpeg7:Medium>
<mpeg7:FileFormat mpeg7:href="urn:mpeg:mpeg7:cs:FileFormatCS:2001:3">
<mpeg7:Name xml:lang="en">MP3</mpeg7:Name></mpeg7:FileFormat>
<mpeg7:FileSize>191488</mpeg7:FileSize>
<mpeg7:BitRate mpeg7:minimum="N/A" mpeg7:average="8000"
mpeg7:maximum="N/A"></mpeg7:BitRate>
<mpeg7:AudioCoding><mpeg7:Format
mpeg7:href="urn:mpeg:mpeg7:cs:AudioCodingFormatCS:2001:1">
<mpeg7:Name xml:lang="en">MP3</mpeg7:Name></mpeg7:Format>
<mpeg7:AudioChannels mpeg7:track="2"></mpeg7:AudioChannels>
<mpeg7:Sample mpeg7:rate="44100" mpeg7:bitPer="0"></mpeg7:Sample>
</mpeg7:AudioCoding></mpeg7:MediaFormat></mpeg7:Mpeg7>…
</mpeg21:Item></mpeg21:Container></mpeg21:DIDL>
Τhe metadata manipulation is performed considering the OWL ontology for audiovisual metadata presented
in Figure 6. Afterwards the audiovisual material is forwarded to the Fog, along with the information
required for its embedment is the 360o
video that will be produced from the drone. Subsequently, the drone
flights along the selected route, while the captured 360o
video is enriched with the audiovisual material and
streamed to the user in real-time. Finally, after the virtual tour’s completion, each user sends his feedback
𝒑𝒊,𝒇 about each monument type presented in his tour.

Figure 6. The OWL ontology about audiovisual MPEG21/7 metadata
Table 7 presents the 𝑝𝑖,𝑛𝑒𝑤 values for all users. These values are estimated using formula 5, where the
previous 𝑝𝑖,𝑛𝑒𝑤 and the feedback 𝑝𝑖,𝑓 values are considered. These values will be considered for the next
real-time virtual tour of each user. Specifically, if the 5 users request to perform a new virtual tour each,
the corresponding MANP weights, which will be estimated considering the new preference values, will
have the distribution presented in Figure 7. As a result, the new weights for each user differ from the ones
presented in Figure 4. Specifically, according to the new weights, user 1 prefers both Ancient and Byzantine
monuments, user 2 mostly prefers Byzantine monuments, user 3 mostly prefers Natural Beauty monuments
but he also prefers Ancient and Byzantine monuments, user 4 mostly prefers Modern monuments but he
also seems to prefer Byzantine monuments and, finally, user 5 prefers both Byzantine and Modern
monuments.
Table 7. The updated MPEG-21 and MPEG-7 metadata preference values for each user
User
Updated MPEG-21 and MPEG-7 metadata preference values
⌈(𝒑𝒊+𝒑𝒊,𝒇)/𝟐⌉
→ 𝒑𝒊,𝒏𝒆𝒘
⌈(𝒑𝒊+𝒑𝒊,𝒇)/𝟐⌉
⌈(𝒑𝒊+𝒑𝒊,𝒇)/𝟐⌉ → 𝒑𝒊,𝒏𝒆𝒘 ⌈(𝒑𝒊+𝒑𝒊,𝒇)/𝟐⌉
1 ⌈(𝟗 + 𝟔)/𝟐⌉ → 𝟖 ⌈(𝟑 + 𝟗)/𝟐⌉ → 𝟔 ⌈(𝟑+𝟑)/𝟐⌉ → 𝟑 ⌈(𝟏 + 𝟏)/𝟐⌉ → 𝟏

2 ⌈(𝟑+𝟏)/𝟐⌉ → 𝟐 ⌈(𝟗 + 𝟗)/𝟐⌉ → 𝟗 ⌈(𝟔 + 𝟏)/𝟐⌉ → 𝟒 ⌈(𝟏+𝟔)/𝟐⌉ → 𝟒
3 ⌈(𝟏 + 𝟔)/𝟐⌉ → 𝟒 ⌈(𝟏+𝟗)/𝟐⌉ → 𝟓 ⌈(𝟏+𝟏)/𝟐⌉ → 𝟏 ⌈(𝟗 + 𝟔)/𝟐⌉ → 𝟖
4 ⌈(𝟑 + 𝟑)/𝟐⌉ → 𝟑 ⌈(𝟑 + 𝟔)/𝟐⌉ → 𝟓 ⌈(𝟗+𝟗)/𝟐⌉ → 𝟗 ⌈(𝟑 + 𝟏)/𝟐⌉ → 𝟐
5 ⌈(𝟏 + 𝟏)/𝟐⌉ → 𝟏 ⌈(𝟗 + 𝟔)/𝟐⌉ → 𝟖 ⌈(𝟑 + 𝟗)/𝟐⌉ → 𝟔 ⌈(𝟏+𝟏)/𝟐⌉ → 𝟏
Figure 7. The user preferences for each monument type after the users feedeback
Thus, since the user preferences weights changed, the TFT-HRS ranks for the available route also changed.
Specifically, as observed in Figure 8, considering the new MANP weights the route-3 is selected for all
users, while at the same time the entire routes obtain different ranks from the ones presented in Figure 5.
The SDN controller interacts with the OWL ontologies using SPARQL queries, while the interaction with
the corresponding MPEG-21 and MPEG-7 metadata is performed in a standardized way using MPQF
queries. Figure 9 illustrates the entire process.
Figure 8. The ranking of each heritage route after users feedback

Figure 9. The sequence diagram about the proposed procedure
CONCLUSION
In this paper, a scheme for supporting personalized real-time virtual tours in sites with cultural interest
using drones is proposed. The user preferences about the Ancient, Byzantine, Modern and Natural Beauty
monument types are modeled using the MPEG-21/7 standards. Also, the structure and semantics of the used
MPEG-21/7 metadata are described using the corresponding OWL ontologies. Subsequently, the MANP
algorithm is used for the estimation of the weights about the user preferences for each monument type.
Considering these weights, the TFT-HRS algorithm ranks the candidate heritage routes, while the route
with the higher ranking is selected for the drone. Furthermore, after each virtual tour, the MPEG-21/7
metadata about user preferences are updated, to maintain the scheme’s knowledge. In this way, the proposed
scheme combines the advantages of both MADM algorithms, MPEG-21/7 standards and OWL ontologies
for accomplishing the personalization, which could be noted as a lack of existing alternative solutions.
Future work includes the optimization of the heritage route selection algorithm considering additional
parameters such as weather conditions, time of day, monument availability, qualitative chrematistics of the
used equipment and information about user demographics. Also, the situation of having several versions of
multimedia content to support persons with disabilities (e.g. persons with protanopia) will be studied.
ACKNOWLEDGMENT
The publication of this paper has been partly supported by the University of Piraeus Research Center
(UPRC).
REFERENCES

1609.3-2016.(2016). Standard for wireless access in vehicular environments (wave)-networking
services. Technical Specifications Book.IEEE.
802.16q-2015.(2015). Standard for air interface for broadband wireless access systems–
amendment 3 multi-tier networks. Technical Specifications Book.IEEE.
Akpakwu,G.A., Silva,B.J., Hancke,G.P. & Abu-Mahfouz,A.M.(2018). A survey on 5G networks
for the Internet of Things: Communication technologies and challenges. IEEE Access,
(vol.6,pp.3619-3647).
Beck,J. & Egger,R.(2018). Emotionalise Me: Self-reporting and Arousal Measurements in Virtual
Tourism Environments. Information and Communication Technologies in Tourism,(pp.3-
15).Springer.
Bustillo,A., Alaguero,M., Miguel,I., Saiz,J. M. & Iglesias, L.S.(2015). A flexible platform for the
creation of 3D semi-immersive environments to teach cultural heritage. Digital Applications
in Archaeology and Cultural Heritage,(pp.248-259).
Cobos,Y., Sarasua,C., Linaza,M.T., Jimenez,I. & Garcia,A.(2008). Retrieving Film Heritage
content using an MPEG-7 Compliant Ontology. International Workshop on Semantic Media
Adaptation and Personalization,(pp.63-68).IEEE.
Debandi,F., Iacoviello,R., Messina,A., Montagnuolo,M., Manuri,F., Sanna,A. & Zappia,D.
(2018). Enhancing cultural tourism by a mixed reality application for outdoor navigation and
information browsing using immersive devices. Materials Science and Engineering,(pp.1-
8).IOP Publishing.
Frikha,M., Mhiri,M., & Gargouri,F.(2017). A User Interest Ontology Based on Trusted Friends
Preferences for Personalized Recommendation. European, Mediterranean, and Middle
Eastern Conference on Information Systems,(pp.54-67).Springer.
Ho,W., & Ma,X.(2017). The state-of-the-art integrations and applications of the analytic hierarchy
process. European Journal of Operational Research,(pp.399-414).Elsevier.
Kan,M., Okamoto,S., & Lee,J.H.(2018). Development of Drone Capable of Autonomous Flight
Using GPS. International Multi-Conference of Engineers and Computer Scientists,(pp.665-
669).
Keller,I., & Viennet,E.(2015). Recommender Systems for Museums: Evaluation on a Real Dataset.
International Conference on Advances in Information Mining and Management,(pp.65-71).
Kohncke,B., & Balke,W.T.(2010). MPEG-7/21: Structured Metadata for Handling and
Personalizing Multimedia Content. The Handbook of MPEG Applications,(pp.363-387).
Kovacikova,T., Petersen,F., Pluke,M., Alvarez,V.A., Bartolomeo,G., Frisiello,A. &
Cadzow,S.(2009). Personalization and user profile standardization. In Seventh International
Conference on Communication Systems and Networks,(pp.629-043).
Kreutz,D., Ramos,F.M., Verissimo,P.E., Rothenberg,C.E., Azodolmolky, S. & Uhlig, S.(2015).
Software-defined networking: A comprehensive survey. Proceedings of the IEEE,(pp.14-
76).
Lee,J., Jun,S., Chang,T.W. & Park,J.(2017). A Smartness Assessment Framework for Smart
Factories Using Analytic Network Process. Sustainability,(pp.794-808).

Marlin,B.M.(2004). Modeling user rating profiles for collaborative filtering. In Advances in neural
information processing systems,(pp.627-634).
Marques,L.F., Tenedório,J.A., Burns,M., Romão,T., Birra,F., Marques,J. & Pires,A.(2017).
Cultural Heritage 3D Modelling and visualisation within an Augmented Reality
Environment, based on Geographic Information Technologies and mobile platforms, ACE:
Architecture, City and Environment (pp.117-136).
Metta,S., Montagnuolo,M. & Messina,A.(2015). MPEG-21 UD: a solution for horizontal
integration of media recommendation systems. International Broadcasting Convention
Conference,(pp.1-8).
Mirk,D. & Hlavacs,H.(2015). Virtual tourism with drones: experiments and lag compensation.
First Workshop on Micro Aerial Vehicle Networks, Systems, and Applications for Civilian
Use,(pp.45-50).ACM.
Osche,P. E., Castagnos,S., Napoli,A. & Naudet,Y.(2016). Walk the line: Toward an efficient user
model for recommendations in museums. International Workshop on Semantic and Social
Media Adaptation and Personalization,(pp.83-88).
Park,K.W., Hong,H.K., Lee,M. & Lee,D.H.(2014). Unified multimedia annotation system using
MPEG-7 visual descriptors and semantic event-templates. Electronics, Information and
Communications, International Conference on.(pp.1-2).IEEE.
Pavlidis,G.(2018). Recommender systems, cultural heritage applications, and the way forward.
Journal of Cultural Heritage.Elsevier,(pp.1-14).
Peng,P., Zou,L., Özsu,M.T., Chen,L. & Zhao,D.(2016). Processing SPARQL queries over
distributed RDF graphs. The VLDB Journal—The International Journal on Very Large Data
Bases,(Vol.2, pp.243-268).
Qian,F., Ji,L., Han,B. & Gopalakrishnan,V.(2016). Optimizing 360 video delivery over cellular
networks. 5th Workshop on All Things Cellular: Operations, Applications and
Challenges,(pp.1-6).ACM.
Roman,R., Lopez,J., & Mambo,M.(2018). Mobile edge computing, fog et al.: A survey and
analysis of security threats and challenges. Future Generation Computer Systems,(Vol.78,
pp.680-698).
Santos,F., Almeida,A., Martins,C., Oliveira,P., & Gonçalves,R.(2016). Tourism recommendation
system based in user functionality and points-of-interest accessibility levels. International
Conference on Software Process Improvement,(pp.275-284).Springer.
Sengupta,K., & Hitzler,P.(2014). Web ontology language (OWL). Encyclopedia of Social
Network Analysis and Mining,(pp.2374-2378).
Siountri,K., Skondras,E., & Vergados,D.D.(2018). A Delivery Model for Cultural Heritage
Services in Smart Cities Environments. Euro-Mediterranean Conference,(pp.279-
288).Springer.
Skondras,E., Siountri,K., Michalas,A., & Vergados,D.D.(2018). A Route Selection Scheme for
supporting Virtual Tours in Sites with Cultural Interest using Drones. International
Conference on Information, Intelligence, Systems & Applications (IISA),(pp.1-6).IEEE.

Tous,R., & Delgado,J.(2010). Semantic-driven multimedia retrieval with the MPEG Query
Format. Multimedia Tools and Applications,(Vol.1, pp.213-233).
TS36.300-V13.2.0.(2016). Evolved Universal Terrestrial Radio Access Network (E-UTRAN-
Rel.13). Technical Specifications Book.3GPP.
Wei,S.H., & Chen,S.M.(2009). Fuzzy risk analysis based on interval-valued fuzzy numbers.
Expert Systems with Applications,(Issue 36, Vol.2, pp.2285-2299).
Zhao,Y., Wang,S., Zou,Y., Ng,J., & Ng,T.(2017). Automatically Learning User Preferences for
Personalized Service Composition. International Conference on Web Services (ICWS)
(pp.776-783).IEEE.

Personalized Real-Time Virtual Tours in Places with Cultural Interest

Recommended

Recommended

More Related Content

Similar to Personalized Real-Time Virtual Tours in Places with Cultural Interest

Similar to Personalized Real-Time Virtual Tours in Places with Cultural Interest (20)

More from University of Piraeus

More from University of Piraeus (20)

Recently uploaded

Recently uploaded (20)

Personalized Real-Time Virtual Tours in Places with Cultural Interest