How data rule the worldKristoffer Benjaminsson CTO
Easy?Easy is a studio within EA Games that do free to play gamesOriginates from DICE3 live titlesBattlefield HeroesBattleforgeLord of Ultima
Trailer
Telemetry - Wikipedia” Telemetry (synonymous with telematics) is a technology that allows remote measurement and reporting of information. The word is derived from Greek roots tele = remote, and metron = measure.”
Telemetry - EasyData that helps us make decisions to provide a better service for our playersProvides answers to specific questionsMultiple sourcesFinancial dataGame statisticsWeb behaviour dataGame telemetryVertical slice of the player population
Why bother?We prefer to know what is going on instead of guessingWe can measure results from actionsData do not (usually) lieForums not representative Less than 20% read forum postsLess than 5% post in forums
Game telemetry examplesLatency dataPunkbuster kicksMatchmakingTime to levelHardware profilesPlay time
ArchitectureGame ClientMagmaDBReportingGame Server
ImplementationTelemetryManager sends data to our backend for database storageHooks in the game code collects dataHooks are records packed as name value pairsEach record will create its own table in the databaseAll telemtry includes a common identifer so we can cross reference data between telemetry hooks
Example recordclass PingRecord : public ITelemetryRecord{public:PingRecord(const std::string& clientIp, const std::string& serverIp, int ping, const 	std::string& dataCenter, const std::string& community);public:	virtual boolhasType( inttypeId ) const { return typeId == RTPing; }	virtual bool assimilate( const ITelemetryRecord* other ) { return false; }	virtual IDataCollector::DataRef buildData() const;private:	std::string m_clientIp;	std::string m_serverIp;	int m_ping;	std::string m_dataCenter;	std::string m_community;};
Example usageplayer->addTelemetry(new backend::telemetry::PingRecord(clientIp,  serverAddressAndPort, 				  conn->getAveragePing(), 				  serverSettings->getProdDataCenter(), 				  serverSettings->getServerCommunity()));
Real life examples
High latency - before
Why were people in Europe being matchmade to Australia and North America?
High latency - after
How did we do it?Generated an HTML pageGeoIP CityA bit of C#Google Maps APITelemetry dataA bit of SQL
Example page<html>    <head>        <meta name="viewport" content="initial-scale=1.0, user-scalable=no" />        <script type="text/javascript" src="http://maps.google.com/maps/api/js?sensor=false"></script>        <script type="text/javascript">        function initialize()        {            var latlng = new google.maps.LatLng(45.5735931396484, 12.1060943603516);            var myOptions =                {                zoom: 2,                center: latlng,                mapTypeId: google.maps.MapTypeId.ROADMAP                };            var map = new google.maps.Map(document.getElementById("map_canvas"), myOptions);            var marker = null;            var coords = null;            var path = null;            marker = new google.maps.Marker({ position: new google.maps.LatLng(59.433, 24.728), map: map, title:"gva, 62.65.243.167" , icon: "client.png" });            marker = new google.maps.Marker({ position: new google.maps.LatLng(52.350, 4.916), map: map, title:"gva" , icon: "server.png" });            coords = [new google.maps.LatLng(59.433, 24.7283), new google.maps.LatLng(52.350, 4.916)];            path = new google.maps.Polyline({ path: coords, strokeColor: "#ff0000", strokeOpcaity: 0.5, strokeWeight: 1});            path.setMap(map);        }        </script>    </head>    <body onload="initialize()">        <div id="map_canvas" style="width: 100%; height: 100%"></div>    </body></html>
Google Maps
Punkbuster - before
Punkbuster - after
How did we do it?Microsoft SQL Server 2008: Reporting ServicesStandard BI toolEasy to create charts and tablesUpdates on a daily basis
During development
AssertsWe collect all asserts fromboth client and serverWe differentiate between content asserts and code assertsWe have leaderboards and summariesTop 10 assertsTotal content/code asserts per branchProvides guidance to what we need to fix
FragalyzerLevel design helper toolCollects information about how a map is playedKill positionKilled from positionPlayer/Vehicle movementEtcCurrently collected locally on a game server
SummaryTelemetry is awesome!Collect data to answer specific queriesBuild a flexible system to make it super easy to add new telemetry hooksChose the right reporting tools to mine and report on the data
Questions?

How data rules the world: Telemetry in Battlefield Heroes

  • 1.
    How data rulethe worldKristoffer Benjaminsson CTO
  • 2.
    Easy?Easy is astudio within EA Games that do free to play gamesOriginates from DICE3 live titlesBattlefield HeroesBattleforgeLord of Ultima
  • 3.
  • 4.
    Telemetry - Wikipedia”Telemetry (synonymous with telematics) is a technology that allows remote measurement and reporting of information. The word is derived from Greek roots tele = remote, and metron = measure.”
  • 5.
    Telemetry - EasyDatathat helps us make decisions to provide a better service for our playersProvides answers to specific questionsMultiple sourcesFinancial dataGame statisticsWeb behaviour dataGame telemetryVertical slice of the player population
  • 6.
    Why bother?We preferto know what is going on instead of guessingWe can measure results from actionsData do not (usually) lieForums not representative Less than 20% read forum postsLess than 5% post in forums
  • 7.
    Game telemetry examplesLatencydataPunkbuster kicksMatchmakingTime to levelHardware profilesPlay time
  • 8.
  • 9.
    ImplementationTelemetryManager sends datato our backend for database storageHooks in the game code collects dataHooks are records packed as name value pairsEach record will create its own table in the databaseAll telemtry includes a common identifer so we can cross reference data between telemetry hooks
  • 10.
    Example recordclass PingRecord: public ITelemetryRecord{public:PingRecord(const std::string& clientIp, const std::string& serverIp, int ping, const std::string& dataCenter, const std::string& community);public: virtual boolhasType( inttypeId ) const { return typeId == RTPing; } virtual bool assimilate( const ITelemetryRecord* other ) { return false; } virtual IDataCollector::DataRef buildData() const;private: std::string m_clientIp; std::string m_serverIp; int m_ping; std::string m_dataCenter; std::string m_community;};
  • 11.
    Example usageplayer->addTelemetry(new backend::telemetry::PingRecord(clientIp, serverAddressAndPort, conn->getAveragePing(), serverSettings->getProdDataCenter(), serverSettings->getServerCommunity()));
  • 12.
  • 13.
  • 14.
    Why were peoplein Europe being matchmade to Australia and North America?
  • 15.
  • 16.
    How did wedo it?Generated an HTML pageGeoIP CityA bit of C#Google Maps APITelemetry dataA bit of SQL
  • 17.
    Example page<html> <head> <meta name="viewport" content="initial-scale=1.0, user-scalable=no" /> <script type="text/javascript" src="http://maps.google.com/maps/api/js?sensor=false"></script> <script type="text/javascript"> function initialize() { var latlng = new google.maps.LatLng(45.5735931396484, 12.1060943603516); var myOptions = { zoom: 2, center: latlng, mapTypeId: google.maps.MapTypeId.ROADMAP }; var map = new google.maps.Map(document.getElementById("map_canvas"), myOptions); var marker = null; var coords = null; var path = null; marker = new google.maps.Marker({ position: new google.maps.LatLng(59.433, 24.728), map: map, title:"gva, 62.65.243.167" , icon: "client.png" }); marker = new google.maps.Marker({ position: new google.maps.LatLng(52.350, 4.916), map: map, title:"gva" , icon: "server.png" }); coords = [new google.maps.LatLng(59.433, 24.7283), new google.maps.LatLng(52.350, 4.916)]; path = new google.maps.Polyline({ path: coords, strokeColor: "#ff0000", strokeOpcaity: 0.5, strokeWeight: 1}); path.setMap(map); } </script> </head> <body onload="initialize()"> <div id="map_canvas" style="width: 100%; height: 100%"></div> </body></html>
  • 18.
  • 19.
  • 20.
  • 21.
    How did wedo it?Microsoft SQL Server 2008: Reporting ServicesStandard BI toolEasy to create charts and tablesUpdates on a daily basis
  • 22.
  • 23.
    AssertsWe collect allasserts fromboth client and serverWe differentiate between content asserts and code assertsWe have leaderboards and summariesTop 10 assertsTotal content/code asserts per branchProvides guidance to what we need to fix
  • 24.
    FragalyzerLevel design helpertoolCollects information about how a map is playedKill positionKilled from positionPlayer/Vehicle movementEtcCurrently collected locally on a game server
  • 25.
    SummaryTelemetry is awesome!Collectdata to answer specific queriesBuild a flexible system to make it super easy to add new telemetry hooksChose the right reporting tools to mine and report on the data
  • 26.