Quality of Experience:
What End-users Say About
Web Services?
Bipin
Upadhyaya
Introduction
• Web services provided by different vendors
have different qualities
• Quality of Service(QoS) refers to non...
Service Selection Approaches
QoS aware architecture filters the low quality web
services from a set of functionally equiva...
Limitation of QoS
• Static Release
• Not frequently updated
• Environment & platform specific
• Runtime Monitoring
• Evalu...
User Experience
On average, users tell 15 people
about good experiences and 24
people about bad experiences
A word-of-mout...
Reviews
• Web 2.0 has enabled people to
broadcast their experience to
the mass
• Online user reviews are
examples of such ...
Heterogeneity in Reviews
7
Extract Relevant Information
from Reviews
• Extract service features such as
• Parts and Properties
• Opinions about the f...
Overview of Our Approach
9
POS Tagged
Sentence
Extract
Sentence
& Tag POS
QoE Attributes
& Opinions
Extract
QoE &
Opinion
...
POS Tag for Review Sentences
Dropbox has great
synchronization and folder
sharing capability.
Dropbox/NNP has/VBZ
great/JJ...
Extract QoE Attributes and
Opinion
(great, synchronization)
(great, Folder sharing capability)
Dropbox/NNP has/VBZ great/J...
Polarity of Opinion
• SentiWordNet is used to assign a score of
opinion
• Each word in WordNet is assigned a score
• Alloc...
Cluster QoE Attributes
Performance (6)
Download (3)
Install (2)Ease to Use(4)
Operation (4)
Sync (3)
Synchronization(4)
Ba...
Select Candidate Element
• Rank each word in a cluster based on
frequency and semantic similarity with other
words
• Candi...
Research Questions
RQ1. Can our approach extract QoE from online
reviews?
RQ2. Does user sentiment about QoE attributes
ch...
Case Study Setup
448 440 422 419
6980
5866
6611
4110
0
1000
2000
3000
4000
5000
6000
7000
8000
Trip Shopping Storage Mappi...
RQ1. Can our approach extract
QoE from online reviews?
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Travel Shopping Storage Map...
RQ2. Does user sentiment about
QoE attributes change over time?
18
User’s sentiment about QoE attributes
changes over time
RQ3. How do QoE attributes
relate with QoS attributes?
High correlation between QoS and QoE
Attributes indicates the possi...
20
Upcoming SlideShare
Loading in …5
×

2014 icws research bipinv2

685 views

Published on

Published in: Technology
1 Comment
0 Likes
Statistics
Notes
  • Be the first to like this

No Downloads
Views
Total views
685
On SlideShare
0
From Embeds
0
Number of Embeds
184
Actions
Shares
0
Downloads
5
Comments
1
Likes
0
Embeds 0
No embeds

No notes for slide
  • Different sources
    Different formats of reviews
    Written in natural language, difficult for machine to understand
  • 2014 icws research bipinv2

    1. 1. Quality of Experience: What End-users Say About Web Services? Bipin Upadhyaya
    2. 2. Introduction • Web services provided by different vendors have different qualities • Quality of Service(QoS) refers to non- functional requirements, such as performance or availability requirements • QoS deals with how a service behave at run- time 2
    3. 3. Service Selection Approaches QoS aware architecture filters the low quality web services from a set of functionally equivalent services 3
    4. 4. Limitation of QoS • Static Release • Not frequently updated • Environment & platform specific • Runtime Monitoring • Evaluation of QoS at client-side • resource intensive, time consuming & expensive End users barely understand QoS attributes 4
    5. 5. User Experience On average, users tell 15 people about good experiences and 24 people about bad experiences A word-of-mouth recommendation is the primary factor behind 20% to 50% of all purchasing decisions Ref: http://www.mckinsey.com/insights/marketing_sales/a_new_way_to_measure_word-of-mouth_marketing
    6. 6. Reviews • Web 2.0 has enabled people to broadcast their experience to the mass • Online user reviews are examples of such phenomenon • Credible source of information 6
    7. 7. Heterogeneity in Reviews 7
    8. 8. Extract Relevant Information from Reviews • Extract service features such as • Parts and Properties • Opinions about the features are expressed as • Adjectives and Adverbs #Hotel Pool ElevatorFood Breakfast Staff Location Room 8
    9. 9. Overview of Our Approach 9 POS Tagged Sentence Extract Sentence & Tag POS QoE Attributes & Opinions Extract QoE & Opinion Aggregated QoE Attributes Group Similar QoE Attributes
    10. 10. POS Tag for Review Sentences Dropbox has great synchronization and folder sharing capability. Dropbox/NNP has/VBZ great/JJ synchronization/NN &/CC folder/NN sharing/NN capability/NN ./. POS Tagger 10
    11. 11. Extract QoE Attributes and Opinion (great, synchronization) (great, Folder sharing capability) Dropbox/NNP has/VBZ great/JJ synchronization/NN &/CC folder/NN sharing/NN capability/NN ./. Detect negation and reverse adjectives Detect POS patterns 11
    12. 12. Polarity of Opinion • SentiWordNet is used to assign a score of opinion • Each word in WordNet is assigned a score • Allocates a sentiment score of positive, negative or objective to words 12 (great, Synchronization) (great, Folder sharing capability) Great –> 0.75
    13. 13. Cluster QoE Attributes Performance (6) Download (3) Install (2)Ease to Use(4) Operation (4) Sync (3) Synchronization(4) Back-up (5) Store (3) Fast (4) Upload (3) Download (3) Install (2)Ease to Use (4) Setup(2) Sync (3) Synchronization(4) Back-up (5) Store (3) Performance (6) Operation (4) Fast (4)Upload (3) Setup(2) Clustering 13
    14. 14. Select Candidate Element • Rank each word in a cluster based on frequency and semantic similarity with other words • Candidate elements can be used by users to specify preferences Performance (6) Operation (4) Fast (4) Upload (3) QoE Attribute Rank Performance 7.04 Operation 5.168 Upload 3.66 Fast 5.982 14
    15. 15. Research Questions RQ1. Can our approach extract QoE from online reviews? RQ2. Does user sentiment about QoE attributes changes over time? RQ3. How do QoE attributes relate with QoS attributes? 15
    16. 16. Case Study Setup 448 440 422 419 6980 5866 6611 4110 0 1000 2000 3000 4000 5000 6000 7000 8000 Trip Shopping Storage Mapping Service Sentences containing QoE attributes and opinions Sentences without QoE attributes and opinions • Collect reviews for a periods of 13 months • Collect QoS attributes such as execution time and availability for all services 16
    17. 17. RQ1. Can our approach extract QoE from online reviews? 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Travel Shopping Storage Mapping Service Recall Precision Extract QoE attributes with high precision and recall 17 90% 72%
    18. 18. RQ2. Does user sentiment about QoE attributes change over time? 18 User’s sentiment about QoE attributes changes over time
    19. 19. RQ3. How do QoE attributes relate with QoS attributes? High correlation between QoS and QoE Attributes indicates the possibility of using QoE attributes for service selection. 19
    20. 20. 20

    ×