• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Skyfire log files100411
 

Skyfire log files100411

on

  • 631 views

 

Statistics

Views

Total Views
631
Views on SlideShare
631
Embed Views
0

Actions

Likes
0
Downloads
3
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Skyfire log files100411 Skyfire log files100411 Document Transcript

    • Skyfire Rocket Logs Guidance and Outline Robert Oberhofer October 4, 2011 Skyfire Labs, Inc. 779 E. Evelyn Ave. Sunnyvale, CA 94041
    • PAGE 2 © 2011 Skyfire Labs, Inc Copyright and Trademarks The information in this document is subject to change without notice and describes only the product defined in the introduction of this documentation. This document is not an official customer document and does not take responsibility for any errors or omissions in this document. This document is intended for the use of Skyfire Labs Inc.(Skyfire) personnel and their customers. No part of this documentation may be used, reproduced, modified or transmitted in any form or means without the prior written permission of Skyfire. The information or statements given in this documentation concerning the suitability, capacity or performance of the mentioned hardware or software products are given “as is” and all liability arising in connection with such hardware or software products shall be defined conclusively and finally in a separate agreement between Skyfire and the customer. IN NO EVENT WILL SKYFIRE BE LIABLE FOR ERRORS IN THIS DOCUMENTATION OR FOR ANY DAMAGES, INCLUDING BUT NOT LIMITED TO SPECIAL, DIRECT, INDIRECT, INCIDENTAL OR CONSEQUENTIAL OR ANY LOSSES SUCH AS BUT NOT LIMITED TO LOSS OF PROFIT, REVENUE, BUSINESS INTERRUPTION, BUSINESS OPPORTUNITY OR DATA, that might arise from the use of this document or the information in it. THE CONTENTS OF THIS DOCUMENT ARE PROVIDED "AS IS". EXCEPT AS REQUIRED BY APPLICABLE MANDATORY LAW, NO WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT, ARE MADE IN RELATION TO THE ACCURACY, RELIABILITY OR CONTENTS OF THIS DOCUMENT. SKYFIRE RESERVES THE RIGHT TO REVISE THIS DOCUMENT OR WITHDRAW IT AT ANY TIME WITHOUT PRIOR NOTICE. This document and the product it describes are considered protected by copyrights and other intellectual property rights according to the applicable laws. Copyright © Skyfire Labs, Inc. 2011. All rights reserved.
    • © 2011 Skyfire Labs, Inc PAGE 3 Contents Contents Overview 5 Generic Log Elements 6 Rocket Controller Log 7 LogMetaDataStats ......................................................................................................................7 MP4 MetaData .......................................................................................................................................... 8 FLV MetaData 9.......................................................................................................................................... VideoDetectionStats................................................................................................................... 11 Field Definitions 11........................................................................................................................................ Rocket Optimizer Log 14 VideoTranscoderStats ................................................................................................................14 Field Definitions 14........................................................................................................................................ VideoServerStats .......................................................................................................................17 Field Descriptions .................................................................................................................................... 18
    • PAGE 4 © 2011 Skyfire Labs, Inc
    • Overview © 2011 Skyfire Labs, Inc PAGE 5 Overview The Skyfire Rocket solution has two main components:  Rocket Controller  Rocket Optimizer The main functions of the Rocket Controller is to:  Provide deep video packet inspection capability  Pre-determine whether a video can and will be transcoded  Handle Adaptive Bitrate Video requests  Determine the video optimiztion profile that applies for a specific video and flow and inform the optimzier of the respective parameters.  Maintain state of ongoing video transactions  Provide video related logging information – even if the video is not optimized/transcoded. The main function of the Rocket Optimizer is to:  Instant, Real-time Transcode the video Thus with regards to logging it will be necessary to review and correlate information from both the Optimizer and the Controller to get a comprehensive picture of video activity (whether optimized or not). The log files for the Controller and Optimizer can be found on each server under: /var/log/SkyController.log /var/log/SkyOtpimizer.log
    • PAGE 6 © 2011 Skyfire Labs, Inc Generic Log Elements Generic Log Elements The following is an example of a log line: [09:08:2011:13:56:14.394233] 0.1 SkyController qa-ctlr004 !NOTIFY [3225- 1169942848] ../src/Common/MP4Utils.cpp:236 LogMetaDataStats():236 lc=6, MetaData: MetaDataFormat=MP4, MetaDataFound=true, creation_time=3303238425, duration=38998, modification_time=3303238425, timescale=600, volume=256 It contains the following generic elements common to all log lines: Date and Time: [09:08:2011:13:56:14.394233] Log Version: 0.1 Process creating the log file: SkyController Server name: qa-ctlr004 Log Level of the entry: !NOTIFY Unique ID (TBD): [3225-1169942848] Source File & line generic entry: ../src/Common/MP4Utils.cpp:236 Type of Entry / Function: LogMetaDataStats()
    • Rocket Controller Log © 2011 Skyfire Labs, Inc PAGE 7 Rocket Controller Log By default, the Rocket Controller will be run in production with log level set to !Notify. When set to the !Notify level, all messages of level !Notify, !WARN and !Error will be visible. For analytics purposes, the following are the two main log lines of interest:  LogMetaDataStats  VideoDetectionStats The purpose of ‘LogMetaDataStats’ is to provide Meta Data contained in the headers of a specific video file. This data is useful for videos that might not be transcoded. The purpose of ‘VideoDetectionStats’ is to inform about the decision process within the Controller. LogMetaDataStats Metadata may or may not be present in the log file. The system merely attempts to capture the metadata found. The following two examples are log lines for a LogMetaDataStats event: [09:08:2011:13:56:14.394233] 0.1 SkyController qa-ctlr004 !NOTIFY [3225- 1169942848] ../src/Common/MP4Utils.cpp:236 LogMetaDataStats():236 lc=6, MetaData: MetaDataFormat=MP4, MetaDataFound=true, creation_time=3303238425, duration=38998, modification_time=3303238425, timescale=600, volume=256 [09:08:2011:14:03:40.896518] 0.1 SkyController qa-ctlr004 !NOTIFY [3225- 1232881984] ../src/Common/FLVUtils.cpp:584 LogMetaDataStats():584 lc=21, MetaData: MetaDataFormat=FLV, MetaDataFound=true, duration=180.768000, width=512.000000, height=288.000000, videodatarate=495.000000, framerate=30.000000, videocodecid=7.000000, audiodatarate=64.000000, audiosamplerate=2097152000.000000, audiosamplesize=16.000000, stereo=true, audiocodecid=10.000000, filesize=12622924.000000 As can be seen these entries, depending on the type of video file processed, the metadata stats will contain very different levels of detail. MetaDataFormat=MP4 This field indicates that the source file format is MP4, which is a complex format. Because MP4 is complex, the Controller does not always have enough data available to complete a full inspection of all the headers contained within. Thus the meta-data available might be limited. NOTE For MP4 ,the Optimizer has more detailed information available in its log files.
    • PAGE 8 © 2011 Skyfire Labs, Inc Rocket Controller Log MetaDataFormat=FLV This field indicates that the source file is FLV. FLV stands for “FLash Video”, and is used only for video that has an Adobe Flash video player as its destination. FLV has a simpler metadata structure than MP4, which allows the Controller to provide full more metadata detail. NOTE Flash Video players can also support MP4 video format. MP4 MetaData The MP4 MetaData log entry contains the following information elements: [09:08:2011:13:56:14.394233] 0.1 SkyController qa-ctlr004 !NOTIFY [3225- 1169942848] ../src/Common/MP4Utils.cpp:236 LogMetaDataStats():236 lc=6, MetaData: MetaDataFormat=MP4, MetaDataFound=true, creation_time=3303238425, duration=38998, modification_time=3303238425, timescale=600, volume=256 Field Definitions Field MetaDataFound Description Should always be true, otherwise there will be no log entry. Value(s) True | False Field: creation_time Description The timestamp from when the video file was created by the publisher. Value(s): Time in ms from 1 Jan 1970 (TBD) Field: Duration Description Length of the video in ms Value(s): Integer Field: modification_time Description Timestamp of the last time the video has been modified. Normally this is the same time as creation_time. Value(s): Time in ms from 1 Jan 1970 (TBD) Field: Timescale Description TBD Value(s): Integer
    • Rocket Controller Log © 2011 Skyfire Labs, Inc PAGE 9 Field: Volume Description Volume of the video. Max = 256. Value(s): Integer from 1 to 256 FLV MetaData The FLV MetaData log entry contains the following information elements: [09:16:2011:00:37:37.912094] 0.1 SkyController vzw-flv003 !NOTIFY [6875- 1099671872] ../src/Common/FLVUtils.cpp:544 LogMetaDataStats():544 lc=863620, MetaData: MetaDataFormat=FLV, MetaDataFound=true, duration=10.449000, width=240.000000, height=160.000000, videodatarate=250.000000, framerate=12.000000, videocodecid=2.000000, audiodatarate=31.250000, audiosamplerate=11025.000000, audiosamplesize=16.000000, stereo=true, audiocodecid=2.000000, filesize=392352.000000 Field: MetaDataFound Description Should always be true Value(s): True Field: Duration Description Duration in seconds and fraction of seconds Value(s): Decimal string Field: Width Description Width of video in pixels Value(s): Integer Field: Height Description Height if video in pixels Value(s): Integer Field: Videodatarate Description Bitrate of the video substream in kbps Value(s): Decimal string
    • PAGE 10 © 2011 Skyfire Labs, Inc Rocket Controller Log Field: Framerate Description Frames per second Value(s): Decimal string Field: videocodecid Description ID of the used codec in the format TBD XXX Value(s): Integer Field: Audiodatarate Description Bitrate of the audio substream in kbps Value(s): Decimal String Field: audiosamplerate Description Same rate of the audio substream in Hz. The previous sample code suggests that the metadata was incorrect. Value(s): Decimal String Field: audiosamplesize Description Number of bits per sample. Value(s): Decimal String Field: stereo Description Volume of the video. Max = 256. Value(s): Integer from 1 to 256 Field: audiocodecid Description Volume of the video. Max = 256. Value(s): Integer from 1 to 256 Field: filesize Description Volume of the video. Max = 256. Value(s): Integer from 1 to 256
    • Rocket Controller Log © 2011 Skyfire Labs, Inc PAGE 11 VideoDetectionStats These message lines give key information regarding video detection. The following is an example of a VideoDetectionStats log line: [09:20:2011:12:27:34.200701] 0.1 SkyController sfo-ctlr002 !NOTIFY [17178- 1198274880] ../src/SkyController/VideoAnalysis.cpp:1201 OnDataReceived(): 1201 lc=6838, VideoDetectionStats: sessionID=5cbb91d0f69e4e22aefae1e7791fc3d5, contentType=video%2fx-flv, srcFileFmt=FLV, Player=Flash, srcVideoCodec=On2VP6, srcAudioCodec=Unknown, transcoderAvail=true, srcBitrate=909547, bitrateThreshold=773000, bitrateTarget=350000, minOptimizationTest=passed, srcDuration=1.79s, srcSize=203625, bypass=false, redirectURL=http%3a%2f%2f209.118.181.135%3a4243%2fskyRequest_5c1d62ea-f592- 4322-abdd- 5c10e1f76486%2fskyWidth_480%2fskyHeight_320%2fskyBitrate_350%2fskyMinComp_15 %2fskyABRMinComp_30%2fskyMinCompTest_1%2fskyTranscode_1%2fskyOuttype_flv%2fs kyCookie_%2fskyStatsId_sfo- ctlr002%2fskySessionId_5cbb91d0f69e4e22aefae1e7791fc3d5%2fskyVideo_l.yimg.co m%252fcv%252feng%252ffox%252f110920%252ff%252fxfactor_to_1000.flv, srcURL=http%3a%2f%2fl.yimg.com%2fcv%2feng%2ffox%2f110920%2ff%2fxfactor_to_10 00.flv, UA=Mozilla%2f5.0%20(compatible%3b%20MSIE%209.0%3b%20Windows%20NT%206.0%3b%20 Trident%2f5.0%3b%20FunWebProducts) Field Definitions Field: sessionID Definition The unique identifier as programmed in the Controller API's flow ID field Value(s): Customer defined variable. Field: contentType Definition The HTTP content type of the payload Value(s): See the HTTP specification Field: Player Definition The player used by the client. Value(s): Flash | Non-Flash. Note: it is possible for Flash players to render MP4 content, but a Non-Flash player cannot render FLV content.
    • PAGE 12 © 2011 Skyfire Labs, Inc Rocket Controller Log Field: transcoderAvail Definition The transcoder health check at the time the video was detected Value(s): True | False. If true, the transcoder was online If false, the transcoder was offline Field: minOptimizationTest Definition The optimization test result. Value(s): Passed | Failed | Deferred Passed indicates that video is detected and its format is deemed worthy of optimization, given the current congestion level. Failed indicates that video is detected and its format is deemed not worthy of being optimized, given the current congestion level. Deferred indicates that not enough information was available to make the optimization decision, so the Optimizer makes the decision at a future date. Field: bypass Definition Whether or not the video optimization was bypassed. Value(s): False | variable False indicates the video will be optimized Variable: Any other value provides a text description of why we do not intend to optimize the video. Typical variable values might be: transcoder_unavailable url-blacklisted min-optimization-failed Field: redirectURL Definition When video was found, the client redirected to this URL. Value(s): Variable URL format Field: UA Definition The User Agent of the client application Value(s): Variable string
    • Rocket Controller Log © 2011 Skyfire Labs, Inc PAGE 13 When the field M3U8Playlist=true, the following fields are included: Field: M3U8Type Definition The type of variant playlist that was found Value(s): HLS:variants: HLS:mixed | HLS:unknown An HLS:variant indicates a pure variant playlist is detected HLS:mixed indicates a combined variant playlist and content playlist in a single file is detected HLS:unknown indicates an HLS format is detected, but that no other determination could be made.
    • PAGE 14 © 2011 Skyfire Labs, Inc Rocket Optimizer Log Rocket Optimizer Log When conducting analytics for the Optimizer, the two fields of most interest are:  VideoTranscoderStats  VideoServerStats VideoTranscoderStats This field provides optimization information. A typical example of this log entry is: [10:03:2011:16:25:39.758386] 0.1 SkyOptimizerServer qa-opt102.dc.skyfire.com !NOTIFY [28889- 1119922496] ../src/SkyOptimizerServer/SFTranscoderWrapper.cpp:1270 DumpSt atistics():1270 lc=21283, inst={82447.82444.82441.8} VideoTranscoderStats: sessionId=92c2acfc5f234a08bda4c69b711da6d1, statsId=qa-flv002, srcDuration=80.05s, playTime=79.74s, fetchedInBytes=37882522, processedInBytes=37882522, outBytes=6241702, optimizationRatio=83.53, srcDimensions=1920x1080, srcBitrate=3786.07kbps, srcVideoCodec=H.264, srcAudioCodec=AAC, srcContainer=MP4, outDimensions=852x480, outBitrate=626.00kbps, outVideoCodec=H.264, outAudioCodec=AAC, outContainer=FLV, inputFileWaitTime=0.00s, outputFileWaitTime=0.01s, sourceFileFirstDataTime=0.01s, firstDataInToOut=2.58s, srcURL=qa- web001.dc.skyfire.com%2fhosted_vids%2fmp4s%2fMain_HiRes_1080P.mp4 Field Definitions Field: sessionID Definition The unique identifier as programmed in the Controller API's flow ID field Value(s): Customer defined variable. Field: statsId Definition The hostname of the controller that intercepted the video. Value(s): Customer defined variable. Field: srcDuration Definition The duration in seconds of the original non-transcoded source video. Value(s): Variable, floating point
    • Rocket Optimizer Log © 2011 Skyfire Labs, Inc PAGE 15 Field: playTime Definition The duration in seconds that the customer actually watched the transcoded video. Value(s): Variable, floating point. Field: fetchedInBytes Definition The total amount of video, in bytes, fetched by the Optimizer. Value(s): Variable, Integer Field: processedInBytes Definition The optimized portion of the video sent to the client (in outBytes size). The optimizer fetches content ahead of the current position from the internet; in order to calculate the true optimization ratio of the currently delivered content, the following calculation is used: optimizationRatio = 1- (outBytes/processedInBytes) in % Value(s): Variable, floating point Field: srcBytes Definition The size in bytes of the original non-transcoded data in bytes the user received. Value(s): Variable, integer Field: outBytes Definition The amount of output transcoded data in bytes the user received. Value(s): Variable, integer. Field: optimizationRatio Definition The average optimization percentage of video transcoded so far. Value(s): Variable, integer. Field: srcDimensions Definition The dimensions of the original non-transcoded source video. Value(s): Integer x integer Field: srcBitrate
    • PAGE 16 © 2011 Skyfire Labs, Inc Rocket Optimizer Log Definition The average bitrate in kbps of the original non-transcoded source video. Value(s): Variable, floating point. Field: srcVideoCodec Definition The video codec of the original non-transcoded source video. Value(s): Variable, free-form string, may be unknown Field: srcAudioCodec Definition The audio codec of the original non-transcoded source video. Value(s): Variable, free-form string, may be unknown Field: srcContainer Definition The container of the original non-transcoded source video. Value(s): Either FLV or MP4 Field: outDimensions Definition The dimensions of output transcoded data Value(s): Integer x integer Field: outBitrate Definition The average bitrate in kbps of the transcoded video Value(s): Variable, floating point Field: outVideoCodec Definition The video codec of the transcoded video. Value(s): Variable, free-form string, should always be H.264 Field: outAudioCodec Definition The audio codec of the transcoded video. Value(s): Variable, free-form string, should always be AAC Field: outContainer Definition The container of the transcoded video.
    • Rocket Optimizer Log © 2011 Skyfire Labs, Inc PAGE 17 Value(s): FLV | MP4. The field should never be MP if the scrContainer is FLV. Field: inputFileWaitTime Definition The time the SkyTranscoderHost spends waiting for input data to transcode. If this number is high, it indicates the Optimizer can transcode faster than the network can feed it data from the origin server.. Value(s): Variable, floating point Field: outputFileWaitTime Definition The time the SkyTranscoderHost spends waiting to push output data to SkyTranscoderServer. Value(s): Variable, floating point Field: sourceFileFirstDataTime Definition The approximate time from when the server received the request to transcode to the time it received the first packet of source data from the origin server. Unlike the field, inputFileWaitTime, this field does not include mid-download stalls. Value(s): Variable, floating point Field: firstDataInToOut Definition The time from when the server receives the first input data from the origin server to when transcoded sends the first output data to the client. The fields, sourceFileFirstDataTime + firstDatainToOut = the time from the initial request to the first packet of output data. Value(s): Variable, floating point Field: srcURL Definition The URL to the original non-transcoded video. Value(s): Variable, URL VideoServerStats Typical log line looks like:
    • PAGE 18 © 2011 Skyfire Labs, Inc Rocket Optimizer Log [09:27:2011:11:37:54.51804] 0.1 SkyOptimizerServer qa-opt101 !NOTIFY [20970- 1170946368] ../src/Server/HTTPVideoServer.cpp:110 DumpStatistics():110 lc=25, inst={1086.1083.1107.8} VideoServerStats: sessionId=ff6ff7b4bcac31d5, statsId=qa-flv001, servedVideo=true, rejectedVideo=false, sourceRequestTimeout=false, dataSentTimeout=false, transcoderFail=false, srcURL=server7.rooftopcomedy.com%2fm%2f2010%2f10%2frivera_thai_mexican_or_no t_HD_DRT_PMSTR_deint.mp4, UA=Mozilla%2f5.0%20(Windows%20NT%206.1%3b%20WOW64%3b%20rv%3a6.0.2)%20Gecko%2 f20100101%20Firefox%2f6.0.2 Field Descriptions Field: sessionID Definition The unique identifier as programmed in the Controller API's flow ID field Value(s): Customer defined variable. Field: statsId Definition The hostname of the controller that intercepted the video. Value(s): Customer defined variable. Field: servedVideo Definition Indicates whether the server served the video or not. Value(s): True | False. True indicates normal operation; False indicates no transcoded data was sent to the client. Field: rejectedVideo Definition Indicates whether or not the server rejected the video Value(s): True | False. True indicates the Optimizer rejected the video, triggering a redirect back to the source. This condition usually means the content failed our minimum optimization criteria. It does not indicate a problem with the Skyfire transcoder. Fale indicates normal operation. Field: sourceRequestTimeout Definition Indicates whether the server received content from the origin server or not. Value(s): True | False. True indicates the origin server did not respond to the request for data, indicating a problem at the source. False indicates normal operation. Field: dataSentTimeout
    • Rocket Optimizer Log © 2011 Skyfire Labs, Inc PAGE 19 Definition Indicates whether the transcoder did not send data to the client. Value(s): True | False. True indicates the transcoder did not provide data to the client, indicating a problem with the Skyfire transcoder. False indicates normal operation. Field: TranscoderFail Definition Indicates whether the transcoder detected a failure in the process of converting the video. Value(s): True | False. True indicates the transcoder was not able to process the original video, indicating a problem with the Skyfire transcoder. False indicates normal operation. Field: srcURL Definition The URL to the original non-transcoded video. Value(s): Variable, URL