You Tube Video Quality Characterization Study

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

0 comments

Post a comment

    Post a comment
    Embed Video
    Edit your comment Cancel

    Notes on slide 1

    Favorites, Groups & Events

    You Tube Video Quality Characterization Study - Presentation Transcript

    1. January 25, 2009 A structured analysis utilizing Six Sigma and Taguchi Robust Engineering tools YouTube User Name: mjpindy
    2.  About the author:  Intermediate skill level for video editing projects  Video editing is considered a hobby  Frustration experienced in past on videos posted to YouTube (some post well while others do not)  Motives / Comments:  No financial motives for this project  Desire to learn more about the rendering/posting process in order to maximize YouTube video quality  Approaching this evaluation as a process characterization study utilizing Six Sigma & Taguchi Robust Engineering tools  Analysis results used to generate data driven conclusions  Results may vary based on other user considerations  Intend to share results with others for critical feedback  Where can the study be improved?
    3.  Problem Statements:  Video quality loss is too high between desktop rendering and YouTube posting  Optimal rendering setup on video editing software not understood or well defined  Objectives:  Develop a test vehicle and methodology to evaluate video quality for videos posted on YouTube  Make conclusions and document findings for the best setup configuration for optimal video quality on YouTube  Target Audience for Report:  Beginning to Intermediate video editors who are attempting to post video clips to the YouTube site  Interested readers wanting to read about using Six Sigma and Robust engineering tools for a characterization study
    4.  Current Status: What video format to render?  Video YouTube High Quality What frame size to render?  Quality Posting after What is the best frame rate?  Process Desktop Why are the YouTube videos  Rendering blurry? Video Why isn‟t the posted quality the  Quality same as desktop quality? after posting on Why does shooting environment  YouTube seem to affect YouTube viewing quality? Low Quality Should I use YouTube differently?  Time / Effort  Desired Status: Common video format defined  High Quality Common display size, frame rates,  Video Video bit rates utilized for optimal YouTube Quality Quality Posting performance after after Process posting on Desktop Posted videos on YouTube show  YouTube Rendering consistent high quality performance regardless of shooting conditions Broad cross-section of viewers  can enjoy high video quality Recommendations based on  Low Quality consideration of all criteria Time / Effort This analysis will make an attempt to improve knowledge base to move from „current‟ to „desired‟ status
    5.  Digital video goes through a series of compression process steps between video camera and posting to Internet Compression reduces video file size and processing load   Smaller file sizes reduce demand on hard drive space  Compressed videos are easier to process by PC system  Compressed videos are easier to upload to Internet  Each compression step results in some video quality loss as compared to original source video in camera  Compression process utilizes CODEC‟s (Compression/Decompression) algorithms to carry out the task  Objective is to evaluate which factors have the least amount of impact on video quality through the entire video editing process What are the key variables that affect posted video quality on YouTube?
    6.  Traditional problem solving approach involves trial and error using one factor at a time (OFAT approach) The video production and posting process is a complex  process with many variables Structured problem solving techniques involve the “tools” to  efficiently evaluate these variables This report is a characterization study to determine what  factors are most important to video quality Additional parameter design studies can be conducted to  study critical variables in more detail The following slide documents the „Methodology‟ or process  steps planned for this characterization study
    7.  Step 1: Define the input parameters to evaluate  Step 2: Develop analysis tool  Step 3: Design the test matrix (or inference space)  Step 4: Render and post videos to YouTube  Step 5: Develop the measurement system  Step 6: Score video clips  Step 7: Analyze Data  Step 8: Complete weighted decision matrix  Step 9: Confirm results by posting a new video to YouTube  Step 10: Document recommendations Presentation will outline each of the 10 steps above in detail
    8. Define Input Parameters to Evaluate
    9.  Project „Inputs‟ and „Outputs‟ documented using the Six Sigma SIPOC tool (Suppliers-Inputs-Process-Output-Customers)  SIPOC tool allows reader to understand boundary conditions of the study  Completed SIPOC in attached file details complete project  Information from SIPOC establishes foundation for rest of project Sampling of lines from SIPOC
    10. Develop Analysis Tool
    11.  Objective of study is to characterize the video quality of videos posted on YouTube  What can we measure to evaluate quality?  A standardized video clip was developed to provide a common tool for all factors of the study  Considerations for standardized clip:  1) Include multiple segments that cover various shooting conditions and challenges  2) Segments must contain enough detail to allow inspector to discriminate video quality differences between clips  3) Proposed material for standardized clip must be capability to be rendered into different formats and settings
    12.  Standardized clip was developed using several ~20 sec clips from personal video collection  Strategy was to incorporate a wide variety of shooting conditions that have generated issues in the past  Final version of standardized video contain the following 8 conditions: Outdoors – Sunny 1. Outdoors – Cloudy 2. Indoors – Bright 3. Indoors – Dark Each original clip was in MPEG2 4. Action – Inside format prior to rendering on desktop 5. Action – Outside 6. Colorful Scenery 7. Text Overlay 8. Resulting standardized clip contains the 8 segments listed (20 sec each) for a total length of 2 min 40 sec
    13. Design the Test Matrix
    14.  The test matrix defines what input factors need to be evaluated in this study  SIPOC defines the various formats originally considered at the start of the study  Formats include: AVI, MPEG4, MPEG2, WMV  SIPOC also documents the „display size‟ of the rendered video as another factor  Display size include: 320x240, 640x480, 1280x720  Other potential factors include: Bit rate, frames per second, audio settings  Objective was to design the test matrix large enough to cover the most popular format and setups that most users encounter  Rendering quality of each of these considerations was not fully understood before study (resulting in unpredictable results)
    15.  Test Matrix designed to provide wide variety of formats and conditions  Intent is to capture cross-section of formats that „normal‟ users may try to upload to YouTube  Challenge is to learn which alternative is best AVI MPEG4 MPEG1/2 WMV 320 640 320 640 320 640 320 640 1280 1280 1280 1280 x x x x x x x x x x x x 720 720 720 720 240 480 240 480 240 480 240 480 Test plan resulted in 12 unique videos targeted for posting on YouTube
    16. Render and Post Videos on YouTube
    17.  Standardized video clip was rendered into each of the 12 configurations listed on previous slide  Pinnacle Studio 11 software setup to render to each configuration  Individual video files were generated and stored on hard drive  YouTube standard upload feature was used to upload the 12 videos from desktop PC hard drive to YouTube‟s server Process Map for Posting Evaluation Videos on YouTube: Outputs Process Desired Standardized Video AVI, Mpeg1/2, Mpeg4, 320x240, 680x240, Video Files stored on Videos viewable on Clip defined - 8 WMV formats 1280x720 display sizes desktop PC hard drive YouTube site conditions Step Generate Select Render Upload to Select format video timeline Configuration Video YouTube Process Step Clips of various Select Commonly used Select commonly used Editing software Format compatibility Inputs to shooting conditions formats used on display size capability with YouTube YouTube Select common video Hard drive storage Internet upload speed Short Segments (20 bit rates space Editing software sec) Active YouTube compatibility for Select common frame PC processing account rendering rate capability
    18.  Attached screen shot image shows the Pinnacle software rendering screen (labeled „make movie‟)  Desired formats and configurations selected on this Pinnacle screen for the for each of the 12 video files Format Selected Display Size Selected
    19.  Attached screen shot image shows the YouTube page and the listing of the 12 videos for this study  User Account Name: mjpindy  AVI file formats did NOT upload properly to YouTube and could not be used in this study Naming Convention: •Each video file name lists the file format •Files labeled „Medium Quality‟ refer to the 320x240 size •Files labeled „High Quality‟ refer to the 640x480 size •Files labeled „HD Quality‟ refer to the 1280x720 size
    20.  The following are the hyperlinks to each of the 12 evaluation videos posted on YouTube:  Mpeg4 – 320x240: http://www.youtube.com/watch?v=3v8uJHgrTDM  Mpeg4 – 640x480: http://www.youtube.com/watch?v=tkWLdN-mwSY  Mpeg4 – 1280x720: http://www.youtube.com/watch?v=hn8J9jO-tIo  Mpeg1/2 – 320x240: http://www.youtube.com/watch?v=T4SG42yxvHI  Mpeg1/2 – 640x480: http://www.youtube.com/watch?v=0iRln2HigUw  Mpeg1/2 – 1280x720: http://www.youtube.com/watch?v=L22qSrEv8ew  WMV – 320x240: http://www.youtube.com/watch?v=iFUgtu1nhE0  WMV – 640x480: http://www.youtube.com/watch?v=sEK8KZqJQIs  WMV – 1280x720: http://www.youtube.com/watch?v=TDIJAB8hmvE  AVI – 320x240: http://www.youtube.com/watch?v=_CSPKLhfUbo All 3 AVI files did NOT post properly to YouTube.  AVI – 640x480: http://www.youtube.com/watch?v=_CSPKLhfUbo As a result, the AVI files  AVI – 1280x720: http://www.youtube.com/watch?v=atO4quN-fRE were not included in scoring activity
    21. Develop the Measurement System
    22.  Measurement system is critical for accurately and reliably measuring the video quality for each video  Measurement Tool Options:  Continuous Measure: This is ideal but not possible since we can‟t take a „physical measure‟ of video quality using a gage  Example: Using a tape measure to measure the length of a board (32.233 inches long) – not possible to apply to video quality measures  Scoring Measure: This is the next best alternative for evaluating video quality  Example: Define a scoring system so inspector can assign scores to each clip to represent level of perceived quality  Visual Scoring System Development:  Recommend a scoring scale from 1 to 10  Following slide outlines the targeted quality characteristics for scoring values across spectrum
    23.  Visual Scoring – Quality Criteria Defined Perceived „HD equivalent‟ quality video resolution Score = 10 Perceived „DVD equivalent‟ quality video resolution Score = 9 Perceived „Very High‟ quality video resolution with a few Score = 8 noticeable areas of diminished resolution Perceived „High‟ quality video resolution with less Score = 7 sharpness with fine details in clip Perceived „Good‟ quality video resolution with less sharpness Score = 6 throughout video and many fine details become blurry Perceived „Average‟ quality video resolution where the video is Score = 5 viewable but lacks detail and sharpness throughout Perceived „Below Average‟ quality video resolution where Score = 4 details are lost but viewer can still detect images throughout Perceived „Poor‟ quality video resolution where viewer starts having Score = 3 difficulty in detecting objects in video – very blurry throughout Perceived „Very Poor‟ quality video resolution where viewer can only Score = 2 detect major objects in video - very blurry and choppy Perceived „Not Viewable‟ quality video resolution where viewer cannot Score = 1 adequately distinguish objects or follow the sequence of events
    24.  Confirmation of the proposed Measurement System needs to be conducted  Is the proposed system repeatable?  Can we rely on the measurement system results?  Ideally, 2 or more visual inspectors are utilized to confirm reproducibility  This evaluation will only utilize 1 visual inspector (i.e. author) for simplicity  Intra-class Correlation Evaluation utilized to verify repeatability of proposed visual scoring system  Select representative clips and measure 2 times each and record scores  Determine the correlation factor between the 2 data sets (scores #1 to scores #2)  Correlation value between data sets should be 0.9 or higher to confirm visual scoring system
    25.  Intra-class Correlation Study Methodology:  First Step: Identify 10 specific segments from the videos posted to YouTube site  9 posted videos with 8 segments/video = 72 total segments  Clips targeted should represent the expected quality range of the videos in the evaluation  Important to capture the widest range in quality possible in order to define how significant measurement error is for this study  Second Step: Inspector reviews and scores each of the 10 clips Third Step: Inspector randomizes the same 10 clips and  reviews and scores again Fourth Step: Data is recorded for each inspection score in  a spreadsheet Fifth Step: Calculate the correlation value between the 2  data sets Sixth Step: Determine if correlation value is greater than  0.9
    26.  Target specific clips from the videos posted on YouTube to represent the range in quality  Utilize experiences in past and/or observations from rendering process to identify clips  Expect to see lower quality on clips with darker background and smaller video display sizes  Target 4-5 clips that meet this criteria  Expect to see higher quality on clips with brighter background and larger video display sizes  Target 4-5 clips that meet this criteria  Table on the next page defines the 10 video clips selected for the measurement system evaluation Objective is to identify clips that represent „range of quality‟ from all clips in study
    27.  Video Segments Identified and scored  Each clip reviewed and scored based on criteria defined  Clips reviewed in order listed in table  Scores documented in table for scores #1
    28.  Same clips need to be reviewed and scored a 2 nd time  Inspector randomizes the order of inspection for clips  Inspector documents scores for Scores #2
    29.  Correlation value between Score #1 and Score #2 calculated to 0.96  Exceeds the 0.90 specification  Visual scores between data sets are not identical  Statistical correlation is sufficient given the range in quality we are trying to detect  Values range from 3 to 8 in this study Recommendation is to use the proposed visual scoring system outlined for the characterization study Excel command used to determine correlation factor (=correl (array1, array2) where array1 is Scores #1 and array 2 is Scores #2
    30. Score Video Clips
    31.  Next step is to visual score the video clips posted on YouTube  Instructions: Each of the 9 video clips posted must individually be scored 1. Each of the 8 shooting conditions must also be individually scored 2. for each clip Inspector generates a total of 72 scores for all the videos posted for 3. each video display size and quality level listed on scoring sheet Process Map for Inspecting and Scoring Videos: Outputs Desired Inspector views in Std Inspector locates Inspector evaluates Inspector can assign a Repeat process for all 8 window, full-screen – also proper video name quality of segment score for the segment shooting conditions std or high quality mode posted on YouTube Determine View the first Stop video Record Inputs to Process Identify first clip Process Step Step viewing shooting after first scores on to view format to use condition condition sheet Score sheet to identify Score sheet to identify YouTube access to Media program on PC Blank scoring sheet correct video names the proper viewing size video has easy access to available to inspector (std window, full stop/start button Access to YouTube Inspector PC screen) User identifies correct channel with posts or functioning properly Access to view boxes to assign scores hyperlinks in this Score sheet to identify segment again if Internet service presentation proper quality level needed download capability (default or high quality sufficient mode)
    32.  Video Display Size Description  Image on left is a screen shot representing the YouTube Default size window (smaller)  Image on right is a screen shot representing the full screen viewing size (larger) YouTube Standard Viewing Window: Full Screen Viewing Window: Button to allow viewing in Full Screen Mode
    33.  Quality Level Instruction  The test plan calls out 2 quality levels YouTube Standard Quality 1. YouTube High Quality 2.  YouTube Default Quality is the standard quality level when you open up a link  This is the quality level most YouTube viewers utilize  YouTube High Quality is an option that the user selects after you open a link  User must select the button in order to view in higher quality Button to allow viewing in high quality mode
    34.  Scoring Sheet – Evaluation #1  The following sheet is used to capture all the scoring data from inspector  Special Conditions for Evaluation #1:  Inspector needs to view all videos in „full screen‟ mode  Inspector needs to view all videos in YouTube „default quality‟ mode Evaluation Condition #1: Viewing Mode: Full Screen Quality Level: YouTube Default quality mode Medium High HD Medium High HD Medium High HD Mpeg 4 Mpeg 4 Mpeg 4 Mpeg 1/2 Mpeg 1/2 Mpeg 1/2 WMV WMV WMV 320x240 640x480 1280x720 320x240 640x480 1280x720 320x240 640x480 1280x720 Condition 1 Outside - Sunny 4 4 4 4 4 5 4 4 4 Condition 2 Outside - Cloudy 4 4 4 4 4 5 4 4 3 Condition 3 Inside - Bright 4 4 4 4 4 4 4 4 3 Condition 4 Inside - Dark 3 4 4 4 5 5 4 4 4 Condition 5 Indoor - Action 3 4 3 3 3 4 3 3 4 Condition 6 Outdoor - Action 4 4 4 4 4 4 4 4 4 Condition 7 Colors 4 4 4 4 4 4 4 4 4 Condition 8 Text 2 3 3 2 3 3 3 3 3
    35.  Scoring Sheet – Evaluation #2  The following sheet is used to capture all the scoring data from inspector  Special Conditions for Evaluation #2:  Inspector needs to view all videos in „full screen‟ mode  Inspector needs to view all videos in YouTube „High quality‟ mode Evaluation Condition #2: Viewing Mode: Full Screen Quality Level: YouTube High Quality mode Medium High HD Medium High HD Medium High HD Mpeg 4 Mpeg 4 Mpeg 4 Mpeg 1/2 Mpeg 1/2 Mpeg 1/2 WMV WMV WMV 320x240 640x480 1280x720 320x240 640x480 1280x720 320x240 640x480 1280x720 Condition 1 Outside - Sunny 4 6 5 4 6 5 4 6 5 Condition 2 Outside - Cloudy 4 7 6 4 7 6 4 7 6 Condition 3 Inside - Bright 4 7 6 4 7 6 4 7 6 Condition 4 Inside - Dark 3 6 6 4 7 6 4 7 6 Condition 5 Indoor - Action 3 7 6 3 7 6 3 7 6 Condition 6 Outdoor - Action 4 7 6 4 7 6 4 7 6 Condition 7 Colors 4 7 6 4 7 6 4 7 6 Condition 8 Text 2 5 5 2 6 5 3 5 6
    36.  Scoring Sheet – Evaluation #3  The following sheet is used to capture all the scoring data from inspector  Special Conditions for Evaluation #3:  Inspector needs to view all videos in „YouTube Standard size‟ mode  Inspector needs to view all videos in YouTube „High quality‟ mode Evaluation Condition #3: Viewing Mode: Standard YouTube window size Quality Level: YouTube High Quality mode Medium High HD Medium High HD Medium High HD Mpeg 4 Mpeg 4 Mpeg 4 Mpeg 1/2 Mpeg 1/2 Mpeg 1/2 WMV WMV WMV 320x240 640x480 1280x720 320x240 640x480 1280x720 320x240 640x480 1280x720 Condition 1 Outside - Sunny 6 8 7 5 8 8 5 6 7 Condition 2 Outside - Cloudy 6 7 7 5 8 7 4 7 7 Condition 3 Inside - Bright 5 8 7 5 8 7 5 7 7 Condition 4 Inside - Dark 5 8 7 5 8 7 5 7 7 Condition 5 Indoor - Action 4 8 7 4 8 7 4 7 7 Condition 6 Outdoor - Action 4 8 7 5 8 7 4 7 7 Condition 7 Colors 5 9 7 5 8 7 5 8 7 Condition 8 Text 4 8 7 5 8 7 4 7 8
    37. Analyze Data
    38.  Six Sigma the „Practical-Graphical-Analytical‟ approach to analyze experimental data  Objective is to analyze the data to make conclusions on performance differences between test conditions  Practical Approach:  Document observations while inspecting videos  Sort raw data to look for obvious trends  Graphical Approach:  Spider charts used to show trends in data  Analytical Approach:  Review descriptive statistics on scoring data  Taguchi Robust Engineering methods used to review scoring data
    39.  Practical Analysis:  Inspector Observations: Appeared during inspection that the default YouTube quality level 1. yielded approximately the same video quality regardless of input formatting Video quality differences did appear once videos were viewed in 2. „High Quality‟ mode Video quality generally less when viewed in full screen mode as 3. compared to default YouTube window Large video sizes (1280x720) viewed in High Quality (or HD mode) 4. seemed to have higher resolution but were “choppy” when compared to videos with 640x480 formatting Quality differences were observed between the 8 shooting 5. conditions within each segment Generally, darker shooting conditions and the „text‟ condition 6. scored lower than other conditions Generally, the „high definition‟ option for viewing videos significantly 7. improved overall video quality
    40.  Practical Analysis – ANOG: (Analysis of Goodness)  Each unique variable color coded  Scores sorted from highest quality to lowest  Performance differences can be observed by color coded patterns in each column (end point counting) Average Scores: Format: Size: Viewing Mode: Mpeg 4 640x480 Std Window - high quality 8.0 Mpeg 1/2 640x480 Std Window - high quality 8.0 Mpeg 1/2 1280x720 Std Window - high quality 7.1 WMV 1280x720 Std Window - high quality 7.1 Mpeg 4 1280x720 Std Window - high quality 7.0 WMV 640x480 Std Window - high quality 7.0 Mpeg 1/2 640x480 Full Screen - high quality 6.8 WMV 640x480 Full Screen - high quality 6.6 Mpeg 4 640x480 Full Screen - high quality 6.5 WMV 1280x720 Full Screen - high quality 5.9 Mpeg 4 1280x720 Full Screen - high quality 5.8 Mpeg 1/2 1280x720 Full Screen - high quality 5.8 Mpeg 4 320x240 Std Window - high quality 4.9 Mpeg 1/2 320x240 Std Window - high quality 4.9 WMV 320x240 Std Window - high quality 4.5 Mpeg 1/2 1280x720 Full Screen - def quality 4.3 Mpeg 4 640x480 Full Screen - def quality 3.9 Mpeg 1/2 640x480 Full Screen - def quality 3.9 Mpeg 4 1280x720 Full Screen - def quality 3.8 WMV 320x240 Full Screen - def quality 3.8 WMV 640x480 Full Screen - def quality 3.8 WMV 320x240 Full Screen - high quality 3.8 Mpeg 1/2 320x240 Full Screen - def quality 3.6 WMV 1280x720 Full Screen - def quality 3.6 Mpeg 1/2 320x240 Full Screen - high quality 3.6 Mpeg 4 320x240 Full Screen - def quality 3.5 Mpeg 4 320x240 Full Screen - high quality 3.5 Note: Average scores calculated over all 8 shooting conditions
    41.  ANOG Comments:  Color coded cells in rank order for quality indicate some patterns in the data  Observable patterns provide input for potential significant factors  Comments:  For viewing mode, the Std Window in High Quality appears to be a significant factor  Full Screen in High Quality appears to also be significant  For display size, the 640x480 values generated the highest average scores at 8  1280x720 screen sizes were significant but second in rankings  For formats, no significant patterns are evident  ANOG comments appear to align with the practical analysis comments
    42.  Graphical Analysis:  Scoring data can also be reviewed graphically to obtain a different perspective on significance  The following slides show a series of „spider charts‟ showing video quality performance  Spider charts contain difference axis representing each criteria  Each axis is labeled from 1 to 10 to represent score values  Same quality level definitions apply to the axis values (10=best)  Graph format makes visual comparisons between input conditions easier
    43.  Spider Chart – Full Screen Viewing at Default Resolution  Observations:  Most scores are between 2 and 4 for each video format and screen size  The „Colorful Scenery‟ condition scored highest (~ 4 in quality)  „Text‟ condition scored lowest (between 2-3 in quality)  Generally, every format scored low when full screen viewing and default YouTube quality level (regardless of input format)
    44.  Spider Chart – Full Screen Viewing at High Quality  Observations:  Significant higher scores with all of the 640x480 and the 1280x720 display sizes  YouTube does not allow the „High Quality‟ view feature for videos rendered in the 320x240 size (therefore, results are same as previous slide)  The „Colorful Scenery‟ condition scored highest (6-7 in quality)  „Text‟ condition scored lowest (between 5-6 in quality)
    45.  Spider Chart – Default Screen Size at High Quality  Observations:  640x480 and the 1280x720 display scores are the highest (7-8 range)  „Text‟ condition also scored very high (7-8 range)  Range of scores across all 8 shooting conditions appears smaller
    46.  Spider Chart – Comparison of Best and Worst Shooting Conditions  Observations:  Both conditions generated consistently low scores in the full screen at default quality mode (3-4 range)  Colorful scenery condition generated a tighter range of values in the „High Quality‟ mode at both full screen and default screen sizes  Text Overlay condition generated a higher range of values in the „High Quality‟ mode at both full screen and default screen sizes Best Case Condition: Worst Case Condition:
    47.  Analytical Analysis:  Descriptive statistics is one analytical analysis option for evaluating the scoring data  Mean, Standard Deviation, Range, etc. are all normal measures  Taguchi Signal-to-Noise analysis is another option for evaluating the scoring data generated  Signal-to-Noise is a calculated value that combines both the mean and variance of the data set  Signal-to-noise is NOT the same measure as traditional electrical loss or frequency loss  Taguchi Signal-to-noise is a measure of robustness for the defined output response  Robust process solutions are desirable since the response (i.e. video quality) is less vulnerable to the impact of process noise  This evaluation utilizes the 8 shooting conditions as „process noise‟  Signal-to-Noise Formula: S/N = -10LOG(1/n*(Sum of Squares)2)
    48.  Analytical Analysis (results):  Taguchi Signal-to-Noise values calculated for each of the 3 response tables:  Condition #1: YouTube „Default Quality‟ setting at Full Screen  Condition #2: YouTube „High Quality‟ setting at Full Screen  Condition #3: YouTube „High Quality‟ setting at Default Screen Size  Each table shows the Rank Order for all 9 response tables  More positive S/N values represent more robust value Evaluation Condition #2: Evaluation Condition #3: Evaluation Condition #1: Viewing Mode: Standard YouTube window size Viewing Mode: Full Screen Viewing Mode: Full Screen Quality Level: YouTube High Quality mode Quality Level: YouTube High Quality mode Quality Level: YouTube Default quality mode Format: Size: S/N Ratio: Rank: Format: Size: S/N Ratio: Rank: Format: Size: S/N Ratio: Rank: Mpeg 1/2 640x480 -6.0 1 Mpeg 1/2 640x480 -10.3 1 Mpeg 1/2 1280x720 -15.3 1 Mpeg 4 640x480 -6.3 2 WMV 640x480 -10.7 2 Mpeg 4 640x480 -15.8 2 Mpeg 1/2 1280x720 -9.2 3 Mpeg 4 640x480 -11.1 3 Mpeg 1/2 640x480 -15.8 2 WMV 1280x720 -9.2 3 WMV 1280x720 -12.3 4 Mpeg 4 1280x720 -15.9 4 Mpeg 4 1280x720 -9.5 5 Mpeg 4 1280x720 -12.6 5 WMV 320x240 -15.9 4 WMV 640x480 -9.7 6 Mpeg 1/2 1280x720 -12.6 5 WMV 640x480 -15.9 4 Mpeg 1/2 320x240 -14.2 7 WMV 320x240 -15.9 7 Mpeg 1/2 320x240 -16.1 7 Mpeg 4 320x240 -14.3 8 Mpeg 1/2 320x240 -16.1 8 WMV 1280x720 -16.1 7 WMV 320x240 -14.8 9 Mpeg 4 320x240 -16.3 9 Mpeg 4 320x240 -16.3 9 Rank Order for S/N represents „Most Robust‟ Condition to „Least Robust‟ Condition
    49.  Conclusions for Analyzing Data:  Analysis included techniques for a practical, graphical, and analytical approach Each method offers a different perspective on the  same testing data Important not too solely rely on only 1 method for data  analysis – multiple perspectives are better Data analysis results do NOT represent final  conclusions Analysis results will be factored into the weighted  decision matrix to determine the best condition tested
    50. Complete Weighted Decision Matrix
    51.  Weighted decision matrix is the preferred tool for documenting final decision process  Factors in study represent user‟s key interests and considerations  Users assign „weights‟ to each factor which represent the relative importance  Users then score each alternative based on the evaluation results  Weighted Factors: Video file size on PC hard drive 1. Rendering speed 2. Video format compatibility to YouTube 3. Video format compatibility to Windows Media Player (preferred media player) 4. Video aspect ratio compatibility 5. Ease of use for the YouTube viewing option 6. Video quality on Desktop 7. Video quality on YouTube 8. Final decisions are NOT based on quality considerations alone – other factors must be considered and factored into decision process
    52.  Weights are first assigned to each factor in matrix: Author assigned weights based on personal needs and preferences   Total sum total of scores MUST equal 40 (average of 5 pts/factor)  Controlling sum total forces user to allocate points carefully Assigned Decision Factors: Weights: Video File Size 3 Rendering Speed 2 Video format compatibility to Internet site 9 Video format compatibility to PC desktop 5 Video Aspect Ratio Compatibility 3 Ease of upload to Internet site 2 Video Quality - Desktop 8 Video Quality - Internet Site 8 40  Factors that received higher weighted values are:  Video format compatibility to Internet site – Videos must be compatible to YouTube  Video Quality – Desktop and Internet – Recommendations need to generate quality results whether clips are viewed on desktop or on YouTube
    53.  Next step is to provide individual scores for each alternative  Additional information shown in table for each consideration Medium High HD Medium High HD Medium High HD Medium High HD Format: Mpeg 4 Mpeg 4 Mpeg 4 Mpeg 1/2 Mpeg 1/2 Mpeg 1/2 WMV WMV WMV AVI AVI AVI Display Size: 320x240 640x480 1290x720 320x240 640x480 1288x720 320x240 640x480 1280x720 320x240 640x480 1290x720 Video Bit Rate: 300 kbps 3800 kbps 3800 kbps 400 kbps 4000 kbps 6000 kbps 400 kbps 1943 kbps 5000 kbps 90% 90% 90% Frames per Second: 29.97 fps 29.97 fps 29.97 fps 29.97 fps 29.97 fps 29.97 fps 29.97 fps 29.97 fps 29.97 fps 29.97 fps 29.97 fps 59.94 fps Audio Format: Mpeg Layer 4 Mpeg Layer 4 Mpeg Layer 4 Mpeg Layer 2 Mpeg Layer 2 Mpeg Layer 2 WMA WMA WMA ?? ?? ?? Audio Bit Rate: 128 kbps 128 kbps 128 kbps 224 kbps 224 kbps 224 kbps 16 bit 16 bit 16 bit ?? ?? ?? Frequency: 48 kHz 48 kHz 48 kHz 44.1 kHz 44.1 kHz 48 kHz 48 kHz 48 kHz 48 kHz 48 kHz 48 kHz 48 kHz Not Not Not Video Compatibility to Internet Site: Compatible Compatible Compatible Compatible Compatible Compatible Compatible Compatible Compatible compatible compatible compatible Video Compatibility to Desktop Player: Not compatible Not compatible Not compatible Compatible Compatible Compatible Compatible Compatible Compatible Compatible Compatible Compatible Video Aspect Ratio: 4:3 4:3 Wide 4:3 4:3 Wide 4:3 4:3 Wide 4:3 4:3 Wide Length of Video Clip: 160 sec 160 sec 160 sec 160 sec 160 sec 160 sec 160 sec 160 sec 160 sec 160 sec 160 sec 160 sec Rendered Video File Size: 7.4 MB 67.3 MB 76.8 MB 65.6 MB 76.2 MB 124 MB 7.85 MB 43 MB 101.2 MB 84.2 MB 246.9 MB 680.7 MB Est Time to Render: 5 min 10 min 15 min 5 min 7 min 10 min 5 min 10 min 15 min 2 min 5 min 10 min Input Video Format (used in Pinnacle 11): Mpeg 2 Mpeg 2 Mpeg 2 Mpeg 2 Mpeg 2 Mpeg 2 Mpeg 2 Mpeg 2 Mpeg 2 Mpeg 2 Mpeg 2 Mpeg 2  Each alternative is scored based on data collected Scores from 1-10 for each cell  Weighted Scores: Data Input Sheet - Individual Scores for each Video Format 320x240 640x480 1280x720 320x240 640x480 1280x720 320x240 640x480 1280x720 320x240 640x480 1280x720 Decision Factors: Mpeg 4 Mpeg 4 Mpeg 4 Mpeg 1/2 Mpeg 1/2 Mpeg 1/2 WMV WMV WMV AVI AVI AVI 10 7 6 7 6 4 10 8 5 5 3 1 Video File Size 8 5 2 8 6 4 8 5 2 10 8 4 Rendering Speed 10 10 10 10 10 10 10 10 10 0 0 0 Video format compatibility to Internet site 3 3 3 10 10 10 10 10 10 10 10 10 Video format compatibility to PC desktop 10 10 7 10 10 7 10 10 7 10 10 7 Video Aspect Ratio Compatibility 8 8 8 8 8 8 8 8 8 8 8 8 Ease of upload to Internet site 7 9 10 7 9 10 7 9 10 7 9 10 Video Quality - Desktop 5 8 7 5 8 8 5 7 8 0 0 0 Video Quality - Internet Site
    54.  Weighted matrix generated by multiplying each score by the weighted factor score  Sum total represents the „most desirable‟ score generated for all the factors considered Weighted Score Calculations for each Video Format Medium High HD Medium High HD Medium High HD Medium High HD Decision Factors: Mpeg 4 Mpeg 4 Mpeg 4 Mpeg 1/2 Mpeg 1/2 Mpeg 1/2 WMV WMV WMV AVI AVI AVI 30 21 18 21 18 12 30 24 15 15 9 3 Video File Size 3 16 10 4 16 12 8 16 10 4 20 16 8 Rendering Speed 2 90 90 90 90 90 90 90 90 90 0 0 0 Video format compatibility to Internet site 9 15 15 15 50 50 50 50 50 50 50 50 50 Video format compatibility to PC desktop 5 30 30 21 30 30 21 30 30 21 30 30 21 Video Aspect Ratio Compatibility 3 16 16 16 16 16 16 16 16 16 16 16 16 Ease of upload to Internet site 2 56 72 80 56 72 80 56 72 80 56 72 80 Video Quality - Desktop 8 40 64 56 40 64 64 40 56 64 0 0 0 Video Quality - Internet Site 8 293 318 300 319 352 341 328 348 340 187 193 178 Scoring Total: 9 7 8 6 1 3 5 2 4 10 11 12 Rank Order:  Top 3 total scores are: Mpeg 1 – 640x480 display size (score of 352) 1. 2. WMV – 640x480 display size (score of 348) 3. Mpeg2 – 1280x720 display size (score of 341) 4. WMV – 1280x720 display size (score of 340)
    55. Confirm Results by Posting New Video to YouTube
    56.  New video rendered using the Mpeg 1 format and the 640x480 display size – video posted to YouTube for confirmation of quality level  YouTube Link to Confirmation Video:  http://www.youtube.com/watch?v=yHpDyapRQmc Evaluation Condition #1 Evaluation Condition #2 Evaluation Condition #3 Viewed in YouTube „Full Screen‟ Viewed in YouTube „Full Screen‟ Viewed in YouTube „Default size in „Default Quality‟ mode size in „High Quality‟ mode Window‟ size in „High Quality‟ mode Quality Score: 5 Quality Score: 7 Quality Score: 8 Score confirms expected quality Score confirms expected quality Score confirms expected quality level for this condition level for this condition level for this condition Scoring results from Confirmation Profile match expected quality level for each viewing mode – Ready to document findings and conclusions
    57. Document Recommendations
    58.  Recommendations from study based on results generated using structured problem solving tools Objective was to find best alternatives which resulted in  most „robust‟ solution for video quality on YouTube Recommendations intended to provide additional  knowledge to video editing process Additional studies may be planned to further evaluate  other aspects of video editing and Internet site posting The following page is a summary of recommendations  form this evaluation
    59.  Study Recommendations:  Render videos using Mpeg 1 video format  Render videos using 640x480 display size  YouTube videos must be viewed in „High Definition‟ mode to realize best video quality possible  YouTube has allowed automatic „high quality‟ viewing to be enabled when the following prefix (&fmt=6)is added to a URL address  YouTube „default window‟ viewing is best-case quality using these conditions  YouTube „full screen‟ viewing can also be used with acceptable quality results  WMV format at 640x480 display size is also acceptable (close 2nd to Mpeg 1)  Mpeg2 and WMV 1280x720 options are also acceptable  File sizes significantly larger  Resolution is very good but observed choppy video display Recommendations target configuration for videos resulting in most robust quality level
    60.  Six Sigma Tools:  SIPOC  Process MAP (PMAP)  Visual Scoring Scale  Measurement System Evaluation: Intra-class Correlation  Practical-Graphical-Analytical Analysis  Analysis of Goodness (ANOG)  Weighted Decision Matrix  One Factor at a Time Approach (OFAT)  Taguchi Robust Engineering Tools:  Process Noise Evaluation Method  Signal-to-Noise Calculations  Rank Order – Robustness
    61.  Many benefits exist when utilizing a structured methodology for making decisions Process to identify the scope of the project is thorough 1. Data collection process is also more thorough and 2. efficient Testing methodology is clearly documented and 3. statistically sound (practical-graphical-analytical) Decision matrix takes numerous considerations into 4. account (not just the obvious factors such as quality) Recommendations are clear and concise 5. Documentation of project is thorough and understood by 6. those interested in the topic (even years after completion)
    SlideShare Zeitgeist 2009

    + Mike Mike Nominate

    custom

    652 views, 0 favs, 1 embeds more stats

    The attached is a characterization study on YouTube more

    More info about this document

    © All Rights Reserved

    Go to text version

    • Total Views 652
      • 650 on SlideShare
      • 2 from embeds
    • Comments 0
    • Favorites 0
    • Downloads 36
    Most viewed embeds
    • 2 views on http://mjpindy.wordpress.com

    more

    All embeds
    • 2 views on http://mjpindy.wordpress.com

    less

    Flagged as inappropriate Flag as inappropriate
    Flag as inappropriate

    Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

    Cancel
    File a copyright complaint
    Having problems? Go to our helpdesk?

    Categories

    Tags

    -->
    What's New

    © 2009 SlideShare Inc. All rights reserved.