Streaming Video Quality & User Engagement Whitepaper: IDC & Akamai
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Streaming Video Quality & User Engagement Whitepaper: IDC & Akamai

on

  • 694 views

Streaming Video Quality & User Engagement Whitepaper from July, 2011. In an industry first, IDC conducted a statistical analysis of the server log files of six major 2010 sports events that were ...

Streaming Video Quality & User Engagement Whitepaper from July, 2011. In an industry first, IDC conducted a statistical analysis of the server log files of six major 2010 sports events that were streamed live to consumers in both North America and Western Europe with a total of more than2 million users. The analysis found that both user engagement (measured as session length) and, consequently, unique user numbers were influenced by video quality. Several factors were shown to have an impact:

--Higher bit rates do increase user engagement. For each event, after a certain bit rate threshold, a further increase of bit rates had no additional positive effect on user engagement anymore.
--An important factor negatively impacting user engagement was the number of rebuffering events per hour.
--Other, less influential negative factors were the share of time the video player spent rebuffering during users' sessions and the number of dropped frames per hour.
--Our research suggests that measuring and monitoring key performance indicators (KPIs) for video quality is of critical importance for publishers because they affect user engagement and audience reach and therefore publishers' revenue and competitiveness.

Visit http://www.akamai.com/html/solutions/sola_analytics.html for more information about Akamai's solutions.

Statistics

Views

Total Views
694
Views on SlideShare
694
Embed Views
0

Actions

Likes
0
Downloads
7
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Streaming Video Quality & User Engagement Whitepaper: IDC & Akamai Document Transcript

  • 1. WHITE P APER Streaming Video Quality and User Engagement Sponsored by: Akamai Karsten Weide July 2011 IDC OPINIONwww.idc.com In an industry first, IDC conducted a statistical analysis of the server log files of six major 2010 sports events that were streamed live to consumers in both North America and Western Europe with a total of more than 2 million users. The analysis found that both user engagement (measured as session length) and,F.508.935.4015 consequently, unique user numbers were influenced by video quality. Several factors were shown to have an impact:  Higher bit rates do increase user engagement. For each event, after a certain bit rate threshold, a further increase of bit rates had no additional positive effect onP.508.872.8200 user engagement anymore.  An important factor negatively impacting user engagement was the number of rebuffering events per hour.  Other, less influential negative factors were the share of time the video playerGlobal Headquarters: 5 Speen Street Framingham, MA 01701 USA spent rebuffering during users sessions and the number of dropped frames per hour. Our research suggests that measuring and monitoring key performance indicators (KPIs) for video quality is of critical importance for publishers because they affect user engagement and audience reach and therefore publishers revenue and competitiveness. METHODOLOGY Akamai tasked IDC with a research project to explore the impact of different aspects of online streaming video quality on user engagement. To that end, Akamai provided IDC with the server log files of six major 2010 sports events that were streamed live to consumers via Akamais HD Network employing HTTP streaming, using adaptive bit rate technology as outlined in Table 1. For all of the following analyses, keep in mind that the bit rates offered by publishers were vastly different between events. The FIFA World Cup soccer events in particular were offered at comparatively low bit rates because of the massive crowds expected to watch.
  • 2. TABLE 1 Event Overview Event Region Number of Users Bit Rates Served Soccer A: FIFA World Cup 2010 North America 235,052 400, 750, 1000, 1300, 1800 kbps Soccer B: FIFA World Cup 2010 Western Europe 76,843 700, 1300, 2200, 3000 kbps Soccer C: FIFA World Cup 2010 Western Europe 125,244 400, 800, 1200, 1600 kbps Sports Event A: Major 2010 sports event North America 1,376,727 564, 1064, 1564, 2200 kbps Sports Event B: Major 2010 sports event North America 25,644 564, 1064, 1564, 2200 kbps Sports Event C: Major 2010 sports event North America 469,876 Four bit rates under 2000 kbps Source: IDC, 2011The log files were cleaned up before statistical analysis as follows: Sessions were consolidated by user ID. Each log file entry originally represented one viewing session. Where there were two or more separate sessions for the same user ID, these sessions were consolidated so that log entries represented the complete viewing experiences for each user ID for each event. Logs were cleaned up. We removed any entry where it was clear from the data that it was either impossible for the respective user to have seen the video (average playback bit rate was zero, number of average frames per second [FPS] was zero) or where more than 20% of total session time was spent rebuffering (with the picture frozen), making it unlikely for the user to have endured that bad of a viewing experience. We also ignored cases where the total aggregated session time was less than one minute, assuming that shorter sessions could not be counted as "viewing" a live video.We also assumed that each user ID related to one person, even though severalpersons or different persons at different times may have watched the video.Furthermore, we assumed that users had spent the entire total aggregated sessiontime watching the video. In practice, users might have walked away from their PC orcould have had the video run in the background. For both assumptions, there was noway for us to determine from the log files whether they held true.IDC then conducted a statistical analysis of the remaining cases using the statisticalsoftware package SPSS. The approach was to correlate user engagement (measuredas the total aggregated session time per user ID per event [short: session length orsession duration]) with certain measurements of streaming video quality for thatusers session during that event (see the Correlation section for details on thestatistical method): Average playback bit rate: The average bit rate at which the video was rendered on the users screen as reported to the server by the users video player2 #229083 ©2011 IDC
  • 3.  Rebuffering events per hour: The number of times the buffer ran out of data and had to be replenished, possibly with the picture frozen if the rebuffering event was long enough to be noticeable by the user Percent of time spent rebuffering: The share of the total aggregated session time that was spent rebuffering Dropped frames per hour: The number of frames that the users video player did not show(The project did not analyze the impact of video start-up times on user engagementbecause the log files did not include that information.)The hypotheses were that: Where positive KPIs such as playback bit rate or average FPS were higher (i.e., video quality was better), user engagement would also be higher (i.e., session times would be longer). The expected correlation coefficient would be > 0. Where positive KPIs were lower (i.e., video quality was worse), we expected user engagement to also be lower (i.e., session times would be shorter). The expected correlation coefficient would be < 0. Conversely, where negative KPIs were higher (i.e., video quality was worse), we expected user engagement to also be lower (i.e., session times would be shorter). The expected correlation coefficient would be < 0. Where negative KPIs were lower (i.e., video quality was better), user engagement would also be higher (i.e., session times would be shorter). The expected correlation coefficient would be > 0.All correlation coefficients reported in this document were significant at the 0.01 level.This means that mathematically, there is only a 1% likelihood that the reportedcorrelation occurred by chance.CorrelationCorrelation is a statistical method that analyzes the relationship between two sets ofdata and expresses the closeness of their relation in a "correlation coefficient," asingle number between 1 and -1.For instance, we compared the average bit rates at which thousands of userswatched a video and the total time they spent watching the video. If higher bit rates in each case translate into longer viewing times in a certain proportion, the correlation coefficient would be 1. If there was no relation at all between bit rates and viewing times, the coefficient would be 0. If higher bit rates in each case translate into shorter viewing times in a certain proportion, the correlation coefficient would be -1. Values between 0 and 1 and 0 and -1 would express varying degrees of relationship.©2011 IDC #229083 3
  • 4. Correlation does not necessarily indicate causation (i.e., two sets of data might beshown to relate to each other statistically even though there is no relation betweenthe two data sets in the real world).IN THIS WHITE P APERThis IDC white paper explores the impact of different aspects of online streamingvideo quality on user engagement based on the statistical analysis of server log filesof six major 2010 sports events that were streamed live to consumers.SITUATION OVERVIEWStatistical Analysis of the Impact ofStreaming Video Quality on User EngagementIDCs statistical analysis of the server log files of six major sports events that werestreamed live to users found that user engagement was influenced by video quality.We found the following two factors had a positive impact on session durations (i.e.,they tended to improve user engagement): Higher playback bit rates (which of course are based on higher transferred bit rates) had the greatest impact in terms of extending session lengths, but only up to a certain optimal bit rate threshold. If bit rates were further increased beyond that threshold, session durations were not further increased, or they were not increased as much. Higher frame rates (frames per second) also had a positive impact on session lengths, but to a lesser extent than higher bit rates.The following factors had a negative impact on session lengths — that is, they tendedto worsen user engagement (in sequence of their level of impact): The number of rebuffering events The share of the session time spent rebuffering The number of dropped framesOf the preceding factors negatively impacting user engagement, one of the mostimportant was the number of rebuffering events. The share of viewing time spentrebuffering and the number of dropped frames/s had less of an impact.Our research suggests that measuring and monitoring KPIs for video quality is ofcritical importance for publishers because they affect user engagement and audiencereach and therefore publisher revenue and competitiveness.4 #229083 ©2011 IDC
  • 5. Playback Bit RateOnline video publishers all adopt high-quality or high-definition video for competitiveand branding purposes, also based on the experience in cable TV, where higherresolutions translated into greater user engagement.There has been a lot of discussion in the industry about whether increasing the bitrate available to the user and thereby improving video resolution has a positiveimpact on user engagement. Therefore, we began our analysis by correlating usersaverage playback bit rates and session lengths. We expected a positive correlation(i.e., that higher bit rates come with longer sessions).Our statistical analysis showed that users did watch the video streams for a longertime if they watched the event at higher playback bit rates (i.e., at higher videoresolutions) — but only up to a certain bit rate. That is, for each event, if we analyzedonly the cases up to that events optimal bit rate threshold, correlation betweenplayback bit rates and session length was positive, which means that higher bit ratestended to go with longer sessions (see Figure 1). The impact was slight, butstatistically significant.FIGURE 1Correlation Between Session Time and Playback Bit Rates forCases up to the Optimal Bit Rate Threshold for Each Event Soccer A Soccer B Soccer C Sports Event A Sports Event B Sports Event C -0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4 (Correlation coef f icient)Note: For the optimal bit rate threshold (i.e., the playback bit rates up to which cases wereanalyzed for each of the above events), see Table 2.Source: IDC, 2011©2011 IDC #229083 5
  • 6. After that threshold, there was no additional positive effect on session times, orthe effect decreased. For the six events analyzed, the threshold was at differentlevels (see Table 2). For publishers, this means that it is necessary to carefullymeasure and monitor the impact of bit rate on session lengths to establish the optimalbit rate range. TABLE 2 Maximum Playback Bit Rate Level Showing Positive Impact on Session Length Maximum Bit Rate (kbps) up to Which Higher Bit Rates Further Improved Impact on Session Lengths Correlation Coefficient Soccer A 1000 0.120 Soccer B 2500 0.148 Soccer C 1000 0.031 Sports Event A 2000 0.021 Sports Event B 3500 0.048 Sports Event C 1500 0.023 Source: IDC, 2011It is difficult to arrive at a formula that would express how much user engagement(i.e., session lengths) increases as bit rate increases given the many factors that havean impact on video performance (see next paragraph). But based on the kind ofperformance increases we have seen in the data, we would expect to see, as a ruleof thumb, an increase of 10% in session lengths per 500 kbps increase in averageplayback bit rate.Keep in mind that the bit rates offered by publishers were vastly different betweenevents. This may be one reason why the cutoff is at different levels for differentevents. We also theorize that there may be other effects at work as well. For instance,those users who watch video at the highest bit rates also must have the infrastructure(e.g., a high broadband access speed) in place to be able to watch at these rates.Those users are also more likely to have incomes and busier lives, which couldexplain why they are more likely to watch for shorter periods of time. More research isneeded, taking into account cultural, social, and situational factors.RebufferingRebuffering events are among the most frustrating experiences when watching a livevideo stream. We analyzed the impact of rebuffering events per hour. These areincidents where the buffer of the users video player runs out of data and must be6 #229083 ©2011 IDC
  • 7. replenished by the server while the picture freezes. Rebuffering is caused either by aconnection slowdown or by bad heuristics (i.e., when the player waits too long toswitch to a lower bit rate).We expected a negative correlation between the number of rebuffering events andsession durations (i.e., for more rebuffering events to go with shorter sessions)because with more interruptions of the video stream, users would become morefrustrated and more likely to stop watching it.That is precisely what we found. The number of rebuffering events turned out to beone of the worst factors impacting user engagement (see Figure 2).FIGURE 2Correlation Between Sessi on Time and Rebuffering Events perHour Soccer A Soccer B Soccer C Sports Event A Sports Event B Sports Event C -0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4 (Correlation coef f icient)Source: IDC, 2011Then we looked at the impact of the total share of the session time that users had tospend waiting for the rebuffering to complete and the video to resume (aggregatingthe waiting time incurred by all rebuffering events) on session lengths.Again, we expected a negative correlation (i.e., for higher rebuffering time shares togo with shorter session durations). And again, we found that to be the case (seeFigure 3). Rebuffering duration was the second most important factor negativelyimpacting user engagement. This also means the negative impact of rebuffering timewas smaller than that of the number of rebuffering events. Apparently, viewers findthat disruption as such is worse than waiting for it to end.©2011 IDC #229083 7
  • 8. FIGURE 3Correlation Between Session Time and Percent of Time SpentRebuffering Soccer A Soccer B Soccer C Sports Event A Sports Event B Sports Event C -0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4 (Correlation coef f icient)Source: IDC, 2011Video FramesThe number of frames per second (FPS) or frame rate expresses the number ofconsecutive images shown in a video transmission per second that create the illusionof motion. Higher FPS numbers translate into better video quality because the videoplays more smoothly; lower FPS numbers conversely result in worse video streams.Dropped frames are images that are not displayed by the users video player, eitherbecause local resources (CPU, graphics adapter, memory, etc.) are not sufficient orbecause there is a disruption in the video transmission. From the users perspective,dropped frames translate into choppy video.For the number of dropped frames per hour, we expected a negative correlation withsession durations (i.e., for more dropped frames to coincide with lower userengagement) because dropped frames disrupt the viewing experience. This is whatwe found in the numbers, too. Dropped frames per hour were the third most importantfactor negatively impacting session lengths. However, the impact was fairly minimal(see Figure 4).8 #229083 ©2011 IDC
  • 9. FIGURE 4Correlation Between Session Time and Average Number ofDropped Frames per Hour Soccer A Soccer B Soccer C Sports Event A Sports Event B Sports Event C -0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4 (Correlation coef f icient)Source: IDC, 2011FUTURE OUTLOOKThis research has established that there is an impact of streaming video quality onuser engagement (i.e., session durations) and, therefore, on audience reach. Asconsumers embrace online video distribution as a viable alternative to cable andbroadcast TV, their expectations of video quality will continue to increase. This will beeven more so as online video makes its way into consumers living rooms, where itends up on high-definition television sets and will have to compete with the qualitythat cable routinely provides. Here, a reliable online transmission at a resolution of720p is only the beginning.Publishers will need to embrace measuring and monitoring video quality such asbuffering, drop-off, and bit rate consumption on an ongoing, routine basis to tune theexperience and avoid dips in video quality and the resulting drop in user engagementin order to protect their financial performance and competitiveness.Given the wide range of bit rates offered in the six events analyzed, and the differentinfrastructures given for them, it is difficult to arrive at a universal bit rate benchmark.Two of the three soccer events in particular offered comparatively low bit ratesbecause of the expected huge numbers of viewers. If one looked only at the UnitedStates, recommended bit rates would have to be set quite a bit higher.Based on the given events, for the bit rate provided by publishers, IDC suggestsmaintaining a level of at least 1200 kbps to attract the kind of audience numbers wesaw in the events analyzed. If one wanted to increase audience reach and userengagement beyond that level, it would be prudent to increase the average bit rate to1500 kbps.©2011 IDC #229083 9
  • 10. The single most negative impact on engagement was the number of rebufferingevents. We believe the best practice is not about keeping rebuffering events to acertain exact number; rather, it is about ensuring that the largest share of youraudience experiences no buffering at all.Of course, only part of the occurrence of rebuffering events can be controlled bypublishers. Again, an optimized distribution technology and measuring and monitoringrebuffering events are key to an optimized user engagement.The second most important negative impact on viewer engagement is the amount oftime spent rebuffering. IDC recommends, as a rule of thumb, maintaining a level ofvideo quality at which users experience rebuffering for a maximum of 1% of the time.This lines up with the experiences that publishers have had in practice. In aninterview with IDC, Glenn Goldstein, MTVs VP, Video Technology Strategy, said,"Once rebuffering time hits 1% of the playback time, we know were in trouble."More research is needed regarding the impact of start-up times on user engagement(which was not explored in this research) and the influence of demographic,psychographic, and situational factors on video consumption.Copyright NoticeExternal Publication of IDC Information and Data — Any IDC information that is to beused in advertising, press releases, or promotional materials requires prior writtenapproval from the appropriate IDC Vice President or Country Manager. A draft of theproposed document should accompany any such request. IDC reserves the right todeny approval of external usage for any reason.Copyright 2011 IDC. Reproduction without written permission is completely forbidden.10 #229083 ©2011 IDC