ACM Multimedia Systems 2011Quantifying QoS Requirements ofNetwork Services:A Cheat-Proof FrameworkKuan-Ta Chen Academia Si...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 2The Long-Lasting QuestionFor a...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 3More Concretely…Minimum networ...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 4MotivationUnderstanding QoS re...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 5Our ContributionsA general, ch...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 6Properties of Our FrameworkSim...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 7Our AmbitionA simple, cost-eff...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 8Intuitive Solution: MOS Rating...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 9Why Not MOS: ReasonsSlow in sc...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 10Talk ProgressOverviewMethodol...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 11Method Overview“Method of Lim...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 12Intolerance Threshold Measure...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 13Probing StageExplore users’ i...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 14Cheating DetectionDetect part...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 15Randomness in MeasurementProc...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 16Cheat Proof MechanismA measur...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 17Talk ProgressOverviewMethodol...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 18Pilot StudyTo verify the effi...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 19Experiment Setup
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 20Participants InstructionThe o...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 21Summary of Experiment Results...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 22Consistency ChecksConsistency...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 23ITS Consistency Check (1)
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 24ITS Consistency Check (2)
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 25ITS Consistency Check (3)
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 26Consistency Check acrossParti...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 27Intra-Service ComparativeAsse...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 28Intra-Service ComparativeAsse...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 29Intra-Service ComparativeAsse...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 30Inter-Service ComparisonCoinc...
Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 31Talk ProgressOverviewMethodol...
ConclusionA general, cheat-proof framework for quantifying QoSrequirements of network servicesSimpleEven untrained partici...
ACM Multimedia Systems 2011Thank You!Kuan-Ta Chen Academia SinicaChen-Chi Wu National Taiwan UniversityYu-Chun Chang Natio...
Upcoming SlideShare
Loading in …5
×

Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework

468 views

Published on

Despite all the efforts devoted to improving the QoS of networked multimedia services, the baseline for such improvements has yet to be defined. In other words, although it is well recognized that better network conditions generally yield better service quality, the exact minimum level of network QoS required to ensure satisfactory user experience remains an open question.

In this paper, we propose a general, cheat-proof framework that enables researchers to systematically quantify the minimum QoS needs for real-time networked multimedia services. Our framework has two major features: 1) it measures the quality of a service that users find intolerable by intuitive responses and therefore reduces the burden on experiment participants; and 2) it is cheat-proof because it supports systematic verification of the participants' inputs. Via a pilot study involving 38 participants, we verify the efficacy of our framework by proving that even inexperienced participants can easily produce consistent judgments. In addition, by cross-application and cross-service comparative analysis, we demonstrate the usefulness of the derived QoS thresholds. Such knowledge will serve important reference in the evaluation of competitive applications, application recommendation, network planning, and resource arbitration.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
468
On SlideShare
0
From Embeds
0
Number of Embeds
60
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • For example, if the longest acceptable delay for FPS games is 200 ms, an FPS game provider can ensure that game play is at least tolerable by hosting its game servers within 200 ms network delay from the majority of the players’ locations. For example, suppose that the minimum bandwidth required for conference calls is 80 Kbps. To guarantee the quality of such calls, we can allocate at least 80 Kbps of bandwidth to each call, as long as other simultaneous needs do not have stricter real-time requirements
  • It is generalizable across a variety of networked multimedia services. Thus, it can be applied to compare the resource demands of various services and a service's different implementations. The participants do not have to describe the intensity of their sensations on a categorical or numerical scale. They only need to decide whether or not the current service quality is acceptable; thus, the burden on participants is much less than in the MOS rating experiments. The framework is cheat-proof in that the experiment results can be verified. The verification relies on the consistency of each participant's inputs; that is, the service quality that a participant finds intolerable should be at similar levels in repeated tests. By employing this property, we can detect inconsistent judgments and remove problematic data before performing further analysis and modeling.
  • Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework

    1. 1. ACM Multimedia Systems 2011Quantifying QoS Requirements ofNetwork Services:A Cheat-Proof FrameworkKuan-Ta Chen Academia SinicaChen-Chi Wu National Taiwan UniversityYu-Chun Chang National Taiwan UniversityChin-Laung Lei National Taiwan University
    2. 2. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 2The Long-Lasting QuestionFor a real-time interactive networkservice, what is the minimum level ofnetwork QoS required to providesatisfactory user experience?
    3. 3. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 3More Concretely…Minimum network bandwidthMaximum packet loss rateMaximum network delayfor a smoothSkypeMSN MessengerAIMGoogle TalkLineageWorld of WarcraftUnreal Tournamentuser experienceWhat is the?
    4. 4. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 4MotivationUnderstanding QoS requirements can enable …Network planningE.g., how to place game servers if we know the maximumacceptable RTT of certain online game.Resource arbitrationE.g., guarantee network bandwidth for conferencing calls athome gateway
    5. 5. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 5Our ContributionsA general, cheat-proof framework for quantifyingthe minimum QoS needscross-application comparative analysis ofapplications minimum network QoS needE.g., Skype vs. Google Talkcross-service comparative analysis of networkservices resource demandsE.g., VoIP vs. online games
    6. 6. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 6Properties of Our FrameworkSimple experiment procedureeven inexperienced participants can make consistentjudgments easilyCheating detection  enabling crowdsourcingNo artificial thresholds are used/defined
    7. 7. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 7Our AmbitionA simple, cost-effective andcheat-proofing wayto measure QoS requirements of anetwork service
    8. 8. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 8Intuitive Solution: MOS RatingExcellent?Good?Fair?Poor?Bad?Vote?
    9. 9. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 9Why Not MOS: ReasonsSlow in scoring (think/interpretation time)Not cheat-proofNo justifiable threshold representing“barely acceptable” level
    10. 10. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 10Talk ProgressOverviewMethodologyExperiment DesignCheat Proof MechanismPilot StudySetupConsistency CheckIntra-Service ComparisonInter-Service ComparisonConclusion
    11. 11. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 11Method Overview“Method of Limits” approach from PsychophysicsRepeat measurements:Gradually decrease application quality until thequality becomes not acceptableWe record the network QoS that correspond tominimum acceptable application quality as“intolerance threshold”
    12. 12. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 12Intolerance Threshold MeasurementPlateau stageRemind users the “normal” service qualityProbingExplore users’ intolerance thresholdsQuality boostingEnter the next round of measurement
    13. 13. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 13Probing StageExplore users’ intolerance thresholds in a graceful wayBasically following the exponential decay functionConceptually like “slow start” + “congestion avoidance”
    14. 14. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 14Cheating DetectionDetect participants who do not pay attention toexperiments even in laboratoryAlso, enable crowdsourcingCrowdsourcing = Crowd + OutsourcingTo reduce experiment costUsers may give erroneous feedback perfunctorily,carelessly, or dishonestlyDishonest users have more incentives to perform tasksNot every Internet user is trustworthy!
    15. 15. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 15Randomness in MeasurementProceduresTo prevent users from guessing the currentservice quality based on time, run-timeparameters are randomly decidedPlateau stageDuration: 2 – 6 secondsProbing stageDuration: 15 – 25 secondsQuality boostingIncreased to a random service quality
    16. 16. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 16Cheat Proof MechanismA measured intolerance threshold an intolerance threshold sample (ITS)Basic idea: If a user’s ITS samples arestatistically self-consistent over time• Assume a user made n ITS samples v = (v1, v2, …, vn)• Repeat m times: randomly divide v into va and vb test if va ~ vbusing Wilcoxon rank-sum hypothesis test• The p-value of hypothesis tests is adjusted using theBonferroni method (significance level = α/m)• See if all the p-values are higher than α/m
    17. 17. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 17Talk ProgressOverviewMethodologyExperiment DesignCheat Proof MechanismPilot StudySetupConsistency ChecksIntra-Service ComparisonInter-Service ComparisonConclusion
    18. 18. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 18Pilot StudyTo verify the efficiency and effectiveness of theour frameworkQoS factorsNetwork bandwidth, packet loss rate, network delayApplications chosen in the studyVoIP: AIM, MSN Messenger, Skype, Google TalkConferencing: AIM, MSN Messenger, SkypeGames FPS: Unreal Tournament RPG: Lineage, World of Warcraft
    19. 19. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 19Experiment Setup
    20. 20. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 20Participants InstructionThe only guideline given was“Click the SPACE key whenever you find theservice quality intolerable.”
    21. 21. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 21Summary of Experiment Results38 part-time employees20 service-application-QoS factor combinations1,037 experiment (47.6 hours)13,184 click actions
    22. 22. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 22Consistency ChecksConsistency of individual participants97% passed the cheat-proofing test with signif. level 0.05Consistency of overall InputsGenerally consistentITS for some application-factor pairs are more variable thanothers “service quality may not be dentical even if the networkconditions are exactly the same” (due to different workload)
    23. 23. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 23ITS Consistency Check (1)
    24. 24. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 24ITS Consistency Check (2)
    25. 25. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 25ITS Consistency Check (3)
    26. 26. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 26Consistency Check acrossParticipantsDifferent participants agree more on the threshold of anapplication than the same participant agrees on thethresholds of different applications
    27. 27. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 27Intra-Service ComparativeAssessment (VoIP)Skype is most demanding inbandwidth (in contrast toAIM and Google talk)Google talk is least robust topacket loss (in contrast toSkype)Can see easily the strengthand weakness of eachapplication
    28. 28. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 28Intra-Service ComparativeAssessment (Conferencing)Skype and MSN Messengerare more demanding inbandwidthSkype is most resilient topacket loss, but MSNMessenger is notNone of the applicationsexcel in all aspects
    29. 29. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 29Intra-Service ComparativeAssessment (Games)UT, the only FPS game, ismost demanding inbandwidthWorld of Warcraft is morebandwidth demanding, andmore resilient to packet lossthan Lineage
    30. 30. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 30Inter-Service ComparisonCoincidentally, the relative bandwidth needs ofconferencing, VoIP, FPS, and RPG is approx.8:4:2:1 (70, 35, 17, 8 Kbps)
    31. 31. Quantifying QoS Requirements of Network Services: A Cheat-Proof Framework / ACM MMSys 2011 31Talk ProgressOverviewMethodologyExperiment DesignCheat Proof MechanismPilot StudySetupConsistency CheckIntra-Service ComparisonInter-Service ComparisonConclusion
    32. 32. ConclusionA general, cheat-proof framework for quantifying QoSrequirements of network servicesSimpleEven untrained participants can produce consistent inputsCrowd-sourcing possible!We hope the framework will be helpful forEvaluation of competing applicationsApplication recommendationNetwork planningResource arbitration…
    33. 33. ACM Multimedia Systems 2011Thank You!Kuan-Ta Chen Academia SinicaChen-Chi Wu National Taiwan UniversityYu-Chun Chang National Taiwan UniversityChin-Laung Lei National Taiwan UniversityPresented by Cheng-Chu Tu (Stony Brook University)

    ×