Skyfire log files100411

Skyfire Rocket Logs
Guidance and Outline
Robert Oberhofer
October 4, 2011
Skyfire Labs, Inc.
779 E. Evelyn Ave.
Sunnyvale, CA 94041

© 2011 Skyfire Labs, Inc
Copyright and Trademarks
The information in this document is subject to change without notice and describes only the product defined in the
introduction of this documentation. This document is not an official customer document and does not take
responsibility for any errors or omissions in this document. This document is intended for the use of Skyfire Labs
Inc.(Skyfire) personnel and their customers. No part of this documentation may be used, reproduced, modified or
transmitted in any form or means without the prior written permission of Skyfire.
The information or statements given in this documentation concerning the suitability, capacity or performance of
the mentioned hardware or software products are given “as is” and all liability arising in connection with such
hardware or software products shall be defined conclusively and finally in a separate agreement between Skyfire
and the customer.
IN NO EVENT WILL SKYFIRE BE LIABLE FOR ERRORS IN THIS DOCUMENTATION OR FOR ANY DAMAGES,
INCLUDING BUT NOT LIMITED TO SPECIAL, DIRECT, INDIRECT, INCIDENTAL OR CONSEQUENTIAL OR ANY
LOSSES SUCH AS BUT NOT LIMITED TO LOSS OF PROFIT, REVENUE, BUSINESS INTERRUPTION, BUSINESS
OPPORTUNITY OR DATA, that might arise from the use of this document or the information in it.
THE CONTENTS OF THIS DOCUMENT ARE PROVIDED "AS IS". EXCEPT AS REQUIRED BY APPLICABLE
MANDATORY LAW, NO WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NON-INFRINGEMENT, ARE MADE IN RELATION TO THE ACCURACY, RELIABILITY OR CONTENTS OF THIS
DOCUMENT. SKYFIRE RESERVES THE RIGHT TO REVISE THIS DOCUMENT OR WITHDRAW IT AT ANY TIME
WITHOUT PRIOR NOTICE.
This document and the product it describes are considered protected by copyrights and other intellectual property
rights according to the applicable laws.
Copyright © Skyfire Labs, Inc. 2011. All rights reserved.

© 2011 Skyfire Labs, Inc PAGE 3
Contents
Contents
Overview 5
Generic Log Elements 6
Rocket Controller Log 7
LogMetaDataStats ......................................................................................................................7
MP4 MetaData .......................................................................................................................................... 8
FLV MetaData 9..........................................................................................................................................
VideoDetectionStats................................................................................................................... 11
Field Definitions 11........................................................................................................................................
Rocket Optimizer Log 14
VideoTranscoderStats ................................................................................................................14
Field Definitions 14........................................................................................................................................
VideoServerStats .......................................................................................................................17
Field Descriptions .................................................................................................................................... 18

Overview
Overview
The Skyfire Rocket solution has two main components:
 Rocket Controller
 Rocket Optimizer
The main functions of the Rocket Controller is to:
 Provide deep video packet inspection capability
 Pre-determine whether a video can and will be transcoded
 Handle Adaptive Bitrate Video requests
 Determine the video optimiztion profile that applies for a specific video and flow and inform
the optimzier of the respective parameters.
 Maintain state of ongoing video transactions
 Provide video related logging information – even if the video is not optimized/transcoded.
The main function of the Rocket Optimizer is to:
 Instant, Real-time Transcode the video
Thus with regards to logging it will be necessary to review and correlate information from
both the Optimizer and the Controller to get a comprehensive picture of video activity
(whether optimized or not).
The log files for the Controller and Optimizer can be found on each server under:
/var/log/SkyController.log
/var/log/SkyOtpimizer.log

Generic Log Elements
Generic Log Elements
The following is an example of a log line:
[09:08:2011:13:56:14.394233] 0.1 SkyController qa-ctlr004 !NOTIFY [3225-
1169942848] ../src/Common/MP4Utils.cpp:236 LogMetaDataStats():236 lc=6,
MetaData: MetaDataFormat=MP4, MetaDataFound=true, creation_time=3303238425,
duration=38998, modification_time=3303238425, timescale=600, volume=256
It contains the following generic elements common to all log lines:
Date and Time: [09:08:2011:13:56:14.394233]
Log Version: 0.1
Process creating the log file: SkyController
Server name: qa-ctlr004
Log Level of the entry: !NOTIFY
Unique ID (TBD): [3225-1169942848]
Source File & line generic entry: ../src/Common/MP4Utils.cpp:236
Type of Entry / Function: LogMetaDataStats()

Rocket Controller Log
By default, the Rocket Controller will be run in production with log level set to !Notify.
When set to the !Notify level, all messages of level !Notify, !WARN and !Error will be
visible.
For analytics purposes, the following are the two main log lines of interest:
 LogMetaDataStats
 VideoDetectionStats
The purpose of ‘LogMetaDataStats’ is to provide Meta Data contained in the headers of a
specific video file. This data is useful for videos that might not be transcoded.
The purpose of ‘VideoDetectionStats’ is to inform about the decision process within the
Controller.
LogMetaDataStats
Metadata may or may not be present in the log file. The system merely attempts to
capture the metadata found.
The following two examples are log lines for a LogMetaDataStats event:
1232881984] ../src/Common/FLVUtils.cpp:584 LogMetaDataStats():584 lc=21,
MetaData: MetaDataFormat=FLV, MetaDataFound=true, duration=180.768000,
width=512.000000, height=288.000000, videodatarate=495.000000,
framerate=30.000000, videocodecid=7.000000, audiodatarate=64.000000,
audiosamplerate=2097152000.000000, audiosamplesize=16.000000, stereo=true,
audiocodecid=10.000000, filesize=12622924.000000
As can be seen these entries, depending on the type of video file processed, the metadata
stats will contain very different levels of detail.
MetaDataFormat=MP4
This field indicates that the source file format is MP4, which is a complex format. Because
MP4 is complex, the Controller does not always have enough data available to complete
a full inspection of all the headers contained within. Thus the meta-data available might
be limited.
NOTE For MP4 ,the Optimizer has more detailed information available in its log files.

MetaDataFormat=FLV
This field indicates that the source file is FLV. FLV stands for “FLash Video”, and is used
only for video that has an Adobe Flash video player as its destination. FLV has a simpler
metadata structure than MP4, which allows the Controller to provide full more metadata
detail.
NOTE Flash Video players can also support MP4 video format.
MP4 MetaData
The MP4 MetaData log entry contains the following information elements:
Field Definitions
Field MetaDataFound
Description Should always be true, otherwise there will be no log entry.
Value(s) True | False
Field: creation_time
Description The timestamp from when the video file was created by the publisher.
Value(s): Time in ms from 1 Jan 1970 (TBD)
Field: Duration
Description Length of the video in ms
Value(s): Integer
Field: modification_time
Description
Timestamp of the last time the video has been modified. Normally this is
the same time as creation_time.
Value(s): Time in ms from 1 Jan 1970 (TBD)
Field: Timescale
Description TBD
Value(s): Integer

Field: Volume
Description Volume of the video. Max = 256.
Value(s): Integer from 1 to 256
FLV MetaData
The FLV MetaData log entry contains the following information elements:
[09:16:2011:00:37:37.912094] 0.1 SkyController vzw-flv003 !NOTIFY [6875-
1099671872] ../src/Common/FLVUtils.cpp:544 LogMetaDataStats():544
lc=863620, MetaData: MetaDataFormat=FLV, MetaDataFound=true,
duration=10.449000, width=240.000000, height=160.000000,
videodatarate=250.000000, framerate=12.000000, videocodecid=2.000000,
audiodatarate=31.250000, audiosamplerate=11025.000000,
audiosamplesize=16.000000, stereo=true, audiocodecid=2.000000,
filesize=392352.000000
Field: MetaDataFound
Description Should always be true
Value(s): True
Field: Duration
Description Duration in seconds and fraction of seconds
Value(s): Decimal string
Field: Width
Description Width of video in pixels
Value(s): Integer
Field: Height
Description Height if video in pixels
Value(s): Integer
Field: Videodatarate
Description Bitrate of the video substream in kbps

Field: Framerate
Description Frames per second
Field: videocodecid
Description ID of the used codec in the format TBD XXX
Value(s): Integer
Field: Audiodatarate
Description Bitrate of the audio substream in kbps
Value(s): Decimal String
Field: audiosamplerate
Description
Same rate of the audio substream in Hz.
The previous sample code suggests that the metadata was incorrect.
Field: audiosamplesize
Description Number of bits per sample.
Field: stereo
Field: audiocodecid
Field: filesize

VideoDetectionStats
These message lines give key information regarding video detection.
The following is an example of a VideoDetectionStats log line:
[09:20:2011:12:27:34.200701] 0.1 SkyController sfo-ctlr002 !NOTIFY [17178-
1198274880] ../src/SkyController/VideoAnalysis.cpp:1201 OnDataReceived():
1201 lc=6838, VideoDetectionStats:
sessionID=5cbb91d0f69e4e22aefae1e7791fc3d5, contentType=video%2fx-flv,
srcFileFmt=FLV, Player=Flash, srcVideoCodec=On2VP6, srcAudioCodec=Unknown,
transcoderAvail=true, srcBitrate=909547, bitrateThreshold=773000,
bitrateTarget=350000, minOptimizationTest=passed, srcDuration=1.79s,
srcSize=203625, bypass=false,
redirectURL=http%3a%2f%2f209.118.181.135%3a4243%2fskyRequest_5c1d62ea-f592-
4322-abdd-
5c10e1f76486%2fskyWidth_480%2fskyHeight_320%2fskyBitrate_350%2fskyMinComp_15
%2fskyABRMinComp_30%2fskyMinCompTest_1%2fskyTranscode_1%2fskyOuttype_flv%2fs
kyCookie_%2fskyStatsId_sfo-
ctlr002%2fskySessionId_5cbb91d0f69e4e22aefae1e7791fc3d5%2fskyVideo_l.yimg.co
m%252fcv%252feng%252ffox%252f110920%252ff%252fxfactor_to_1000.flv,
srcURL=http%3a%2f%2fl.yimg.com%2fcv%2feng%2ffox%2f110920%2ff%2fxfactor_to_10
00.flv,
UA=Mozilla%2f5.0%20(compatible%3b%20MSIE%209.0%3b%20Windows%20NT%206.0%3b%20
Trident%2f5.0%3b%20FunWebProducts)
Field Definitions
Field: sessionID
Definition The unique identifier as programmed in the Controller API's flow ID field
Value(s): Customer defined variable.
Field: contentType
Definition The HTTP content type of the payload
Value(s): See the HTTP specification
Field: Player
Definition The player used by the client.
Value(s):
Flash | Non-Flash.
Note: it is possible for Flash players to render MP4 content, but a Non-Flash
player cannot render FLV content.

Field: transcoderAvail
Definition The transcoder health check at the time the video was detected
Value(s):
True | False.
If true, the transcoder was online
If false, the transcoder was offline
Field: minOptimizationTest
Definition The optimization test result.
Value(s):
Passed | Failed | Deferred
Passed indicates that video is detected and its format is deemed worthy of
optimization, given the current congestion level.
Failed indicates that video is detected and its format is deemed not worthy of
being optimized, given the current congestion level.
Deferred indicates that not enough information was available to make the
optimization decision, so the Optimizer makes the decision at a future date.
Field: bypass
Definition Whether or not the video optimization was bypassed.
Value(s):
False | variable
False indicates the video will be optimized
Variable: Any other value provides a text description of why we do not intend
to optimize the video. Typical variable values might be:
transcoder_unavailable
url-blacklisted
min-optimization-failed
Field: redirectURL
Definition When video was found, the client redirected to this URL.
Value(s): Variable URL format
Field: UA
Definition The User Agent of the client application
Value(s): Variable string

When the field M3U8Playlist=true, the following fields are included:
Field: M3U8Type
Definition The type of variant playlist that was found
Value(s):
HLS:variants: HLS:mixed | HLS:unknown
An HLS:variant indicates a pure variant playlist is detected
HLS:mixed indicates a combined variant playlist and content playlist in a
single file is detected
HLS:unknown indicates an HLS format is detected, but that no other
determination could be made.

Rocket Optimizer Log
When conducting analytics for the Optimizer, the two fields of most interest are:
 VideoTranscoderStats
 VideoServerStats
VideoTranscoderStats
This field provides optimization information.
A typical example of this log entry is:
[10:03:2011:16:25:39.758386] 0.1 SkyOptimizerServer qa-opt102.dc.skyfire.com
!NOTIFY [28889-
1119922496] ../src/SkyOptimizerServer/SFTranscoderWrapper.cpp:1270 DumpSt
atistics():1270 lc=21283, inst={82447.82444.82441.8} VideoTranscoderStats:
sessionId=92c2acfc5f234a08bda4c69b711da6d1, statsId=qa-flv002,
srcDuration=80.05s, playTime=79.74s, fetchedInBytes=37882522,
processedInBytes=37882522, outBytes=6241702, optimizationRatio=83.53,
srcDimensions=1920x1080, srcBitrate=3786.07kbps, srcVideoCodec=H.264,
srcAudioCodec=AAC, srcContainer=MP4, outDimensions=852x480,
outBitrate=626.00kbps, outVideoCodec=H.264, outAudioCodec=AAC,
outContainer=FLV, inputFileWaitTime=0.00s, outputFileWaitTime=0.01s,
sourceFileFirstDataTime=0.01s, firstDataInToOut=2.58s, srcURL=qa-
web001.dc.skyfire.com%2fhosted_vids%2fmp4s%2fMain_HiRes_1080P.mp4
Field Definitions
Field: sessionID
Field: statsId
Definition The hostname of the controller that intercepted the video.
Field: srcDuration
Definition The duration in seconds of the original non-transcoded source video.
Value(s): Variable, floating point

Field: playTime
Definition
The duration in seconds that the customer actually watched the transcoded
video.
Value(s): Variable, floating point.
Field: fetchedInBytes
Definition The total amount of video, in bytes, fetched by the Optimizer.
Value(s): Variable, Integer
Field: processedInBytes
Definition
The optimized portion of the video sent to the client (in outBytes size). The
optimizer fetches content ahead of the current position from the internet; in
order to calculate the true optimization ratio of the currently delivered
content, the following calculation is used:
optimizationRatio = 1- (outBytes/processedInBytes) in %
Field: srcBytes
Definition
The size in bytes of the original non-transcoded data in bytes the user
received.
Value(s): Variable, integer
Field: outBytes
Definition The amount of output transcoded data in bytes the user received.
Value(s): Variable, integer.
Field: optimizationRatio
Definition The average optimization percentage of video transcoded so far.
Value(s): Variable, integer.
Field: srcDimensions
Definition The dimensions of the original non-transcoded source video.
Value(s): Integer x integer
Field: srcBitrate

Definition The average bitrate in kbps of the original non-transcoded source video.
Value(s): Variable, floating point.
Field: srcVideoCodec
Definition The video codec of the original non-transcoded source video.
Value(s): Variable, free-form string, may be unknown
Field: srcAudioCodec
Definition The audio codec of the original non-transcoded source video.
Value(s): Variable, free-form string, may be unknown
Field: srcContainer
Definition The container of the original non-transcoded source video.
Value(s): Either FLV or MP4
Field: outDimensions
Definition The dimensions of output transcoded data
Value(s): Integer x integer
Field: outBitrate
Definition The average bitrate in kbps of the transcoded video
Field: outVideoCodec
Definition The video codec of the transcoded video.
Value(s): Variable, free-form string, should always be H.264
Field: outAudioCodec
Definition The audio codec of the transcoded video.
Value(s): Variable, free-form string, should always be AAC
Field: outContainer
Definition The container of the transcoded video.

Value(s): FLV | MP4. The field should never be MP if the scrContainer is FLV.
Field: inputFileWaitTime
Definition
The time the SkyTranscoderHost spends waiting for input data to transcode.
If this number is high, it indicates the Optimizer can transcode faster than the
network can feed it data from the origin server..
Field: outputFileWaitTime
Definition
The time the SkyTranscoderHost spends waiting to push output data to
SkyTranscoderServer.
Field: sourceFileFirstDataTime
Definition
The approximate time from when the server received the request to transcode
to the time it received the first packet of source data from the origin server.
Unlike the field, inputFileWaitTime, this field does not include mid-download
stalls.
Field: firstDataInToOut
Definition
The time from when the server receives the first input data from the origin
server to when transcoded sends the first output data to the client. The fields,
sourceFileFirstDataTime + firstDatainToOut = the time from the initial request
to the first packet of output data.
Field: srcURL
Definition The URL to the original non-transcoded video.
Value(s): Variable, URL
VideoServerStats
Typical log line looks like:

[09:27:2011:11:37:54.51804] 0.1 SkyOptimizerServer qa-opt101
!NOTIFY [20970-
1170946368] ../src/Server/HTTPVideoServer.cpp:110 DumpStatistics():110
lc=25, inst={1086.1083.1107.8} VideoServerStats:
sessionId=ff6ff7b4bcac31d5, statsId=qa-flv001, servedVideo=true,
rejectedVideo=false, sourceRequestTimeout=false, dataSentTimeout=false,
transcoderFail=false,
srcURL=server7.rooftopcomedy.com%2fm%2f2010%2f10%2frivera_thai_mexican_or_no
t_HD_DRT_PMSTR_deint.mp4,
UA=Mozilla%2f5.0%20(Windows%20NT%206.1%3b%20WOW64%3b%20rv%3a6.0.2)%20Gecko%2
f20100101%20Firefox%2f6.0.2
Field Descriptions
Field: sessionID
Field: statsId
Definition The hostname of the controller that intercepted the video.
Field: servedVideo
Definition Indicates whether the server served the video or not.
Value(s):
True | False. True indicates normal operation; False indicates no transcoded
data was sent to the client.
Field: rejectedVideo
Definition Indicates whether or not the server rejected the video
Value(s):
True | False. True indicates the Optimizer rejected the video, triggering a
redirect back to the source. This condition usually means the content failed
our minimum optimization criteria. It does not indicate a problem with the
Skyfire transcoder. Fale indicates normal operation.
Field: sourceRequestTimeout
Definition Indicates whether the server received content from the origin server or not.
Value(s):
True | False. True indicates the origin server did not respond to the request for
data, indicating a problem at the source. False indicates normal operation.
Field: dataSentTimeout

Definition Indicates whether the transcoder did not send data to the client.
Value(s):
True | False. True indicates the transcoder did not provide data to the client,
indicating a problem with the Skyfire transcoder. False indicates normal
operation.
Field: TranscoderFail
Definition
Indicates whether the transcoder detected a failure in the process of
converting the video.
Value(s):
True | False. True indicates the transcoder was not able to process the
original video, indicating a problem with the Skyfire transcoder. False
indicates normal operation.
Field: srcURL
Definition The URL to the original non-transcoded video.
Value(s): Variable, URL

Skyfire log files100411

More Related Content

What's hot

Viewers also liked

More from navaidkhan

Recently uploaded

Skyfire log files100411