Christian Corsano (Io-Interactive) - Hitman – the first episodic AAA game and Azure
•Download as PPTX, PDF•
1 like•256 views
Hitman, by Io-Interactive, published by Square Enix, is a game about Agent 47, the world’s greatest assassin. In March 2016 the latest version of Hitman was released in the form of the world’s first episodic AAA game. Meaning a game that will evolve over time, with features that might only be accessible if you are connected to the cloud. This session will give insights on building such a game on Azure.
Recommended
Recommended
More Related Content
What's hot
What's hot (11)
Similar to Christian Corsano (Io-Interactive) - Hitman – the first episodic AAA game and Azure
Similar to Christian Corsano (Io-Interactive) - Hitman – the first episodic AAA game and Azure (20)
More from DevGAMM Conference
More from DevGAMM Conference (20)
Recently uploaded
Recently uploaded (20)
Christian Corsano (Io-Interactive) - Hitman – the first episodic AAA game and Azure
- 1. Hitman From the box to the cloud Christian Corsano Lead Online Programmer @ccorsano
- 2. Video
- 3. Hitman • Sixth game of a nearly 20 years old franchise • Third person, stealth-action puzzle game • Huge sandbox levels to explore • Figure out the clockwork, take out the target • AAA title, shipping on PC, PS4 and XBoxOne
- 4. From Absolution to Hitman • Hitman Absolution (2012) • Modern, accessible • New game engine (Glacier 2) • Story driven • Online, Contracts mode • Less appealing to historical Hitman player • Hitman • Keep it accessible • Bring back more freedom of approach • Deliver it in a brand new way
- 5. Hitman as a Service • Ever expanding, ever evolving • Digital first • Episodic • Listen and react to the community • Live content
- 6. Backend • Extensibility • Server delivered missions • Server delivered menus • Server driven game modes • Authoritative server • Scoring • Progression • Contracts creation • Supporting services • Analytics, authorization, configuration, administration
- 7. Why Azure • Platform as a Service leader • Extensive toolbox of services • Choosing the right tool for the right job ! • Familiar toolset for game developers • Visual Studio • C#
- 8. Services overview WebApps ˃ Shared services & web tools CloudServices ˃ Game logic and stateful user sessions Search ˃ UGC discoverability Storage ˃ Blob, Key value storage & Queues SQL ˃ Relational storage StreamAnalytics ˃ Analytic processing DocumentDB ˃ Object storage AppInsights ˃ Service monitoring & diagnostics DataFactory ˃ Analytic pipeline orchestration HDInsight ˃ On-demand big data processing EventHub ˃ Metric events pipeline Redis Cache ˃ Managed Redis
- 9. Event-based evaluation • Progression, Scoring and more • Examples: NPC Kill, Disguise, Trespassing, Shots fired • C# ”Scripts” evaluation as needed • Bigger evaluation at the end of a session
- 10. Development process • DevOps-like, Iterative development • Built first services and started iterating • Live internally in the first year • Regular releases • Been running live ever since • Service Oriented Architecture • Actor Model through Microsoft Orleans
- 11. Going deeper: Game services Game Service Storage Account Storage tables Config blobs (public) Config blobs (public) Resource blobs (private) Resource blobs (private) Service blobs (private) Service blobs (private) Game Cloud Service Service Web RolesService Web Roles Service Worker Roles Service Worker Roles Orleans Game Service SQL DB SQL Database Hitman Game Client Azure Redis Cache Azure Search
- 12. Orleans • Actor model for .net • Open source framework • Isolated, persistent, light weight, single threaded objects • Dynamically instanciated • Blocking concurrency at the actor level • Each player is an actor • Great way to scale: run more actors • Isolate user sessions • Automated lifecycle • Handle concurrency with minimal contention
- 13. Service Fabric • Same approach, but fully managed • Actors running on a scalable cluster • Better local emulation for development
- 14. Analytics • Vital system • Business indicators • Gameplay feedback • Issues investigation • Custom assembly of Azure services • Web collector • EventHub • Stream Analytic jobs • Data Factory • HDInsights (Hadoop)
- 15. Data pipeline Collector Event Hub Stream processing Data Orchestration Aggregation Prepared data store Fine-grained aggregations
- 17. Tutorial level
- 21. Looking back • One year being live
- 22. Looking forward • Long term plans ahead !
Editor's Notes
- Good afternoon, and thanks for coming to this session. I am Christian Corsano, and serve as Lead Online Programmer at Io-Interactive, and I am here to talk about how we took Hitman from a traditional game to a running online experience, and how Microsoft Azure helped us to do that. But first, let’s introduce the game with a small video.
- Hitman is nearly 20 years old now, and the our new release is the sixth game in the franchise. It can be best described as a third person stealth action puzzle game, where you explore massive sandbox levels, and through extensive replay, figure out their intricate clockwork, and use it at your advantage to take out your targets. It is a major title, targetting PC PS4 and XBoxOne. So, taking one step back.
- Before Hitman was Hitman Absolution, released in 2012, just before I joined Absolution was a great game, and a big evolution from previous Hitman games It was made more accessible, shipped on the two major consoles of the time, and was powered by the all new iteration of our internal game engine. The game itself was more story driven, and as such a lot more linear than previous installments. For the first time the game had online features, with leaderboards and more importantly Contracts mode, which let the players create their own assassination contracts to challenge the community. But it ended up being less appealing to core Hitman fans, mostly because the reduced freedom of approach. After Absolution came out, we started working on what would become simply Hitman We had the opportunity to build with the learnings from Absolution with its modern, accessible gameplay, and from previous titles such as Blood Money, with its unmatched freedom of approach. More importantly, we have great ambition for this game, and we wanted to deliver it in a suitable and innovative way.
- We called our approach ”Hitman as a Service” Hitman is the first digital-first, episodic AAA game. Our huge sandbox levels are made to be explored over and over again, discovering the many layers of content hidden deep inside them, delivering them in episodes to let the players discover them. It is also an opportunity to constantly listen to the community and deliver custom tailored content based on their feedback. We also constantly released weekly content for the past year, including Elusive Targets where you get a limited time to take out a hidden target in a modified location. You have one shot, if you miss it you will not be able to restart the mission, if you finish it you will not be able to improve your score.
- This means that Hitman had to move from a very traditional boxed release to something extremely flexible and reactive. Having this constant stream of releases and competitive modes called for a brand new online backend. Of course Hitman is not a MMO, the data is installed on the player’s hard drive, but within these levels the objectives definitions are delivered from the server, allowing us to build and deliver more very easily. The game menus which presents the content are also assembled from server data, allowing us to tweak them live if needed. Game modes such as Elusive Targets need very tight time constraints and fair evaluation, which is provided by the backend as well. These are all requirements very specific to our game and the model we chose, but more generally any modern game requires analytics, and might use some more common online services.
- All these requirements made us consider how we could best support the game, with a reasonably sized Online team. We chose to target Azure very early on. Azure started strongly as a Platform as a Service provider, meaning that while the other cloud providers were mostly focusing on selling VM hours, they were offering fully managed services. We are game developers, we want to keep the operational overhead to a minimum and focus on building our solutions: PaaS is the best way to go for us. Azure is providing a very complete set of services, and is always expanding. This allow us to explore their offering and choose the right tool for the right job, like picking the right data storage based on our projected access pattern. Another big advantage for game developers like us are the development tools. We are building our game in Visual Studio, primarilly in C++, but our engine tooling is mainly C#. Azure is very nicely integrated within Visual Studio, and C# SDK is one of the primary languages. This makes it very easy for anybody from the team to jump in and start working on our solutions.
- Here is a summary of the services we use within our current solution. Some of them are core to the solution, such as CloudServices, Azure Storage, SQL, Redis, some other actually drove the development of complete features, such as Azure Search which is powering the discoverability of huge number of user created contracts in a fun and meaningful way. As we grew smarter in using them, we were able to tweak our usage, refine our implementation to gain performance and reduce cost. We also sometime discover a new service or pricing tier in preview and jump on it, such as recently with a new Azure SQL offering more suited to run part of our Analytics pipeline.
- So, in more practical terms, what exactly makes Hitman an online game ? I mentioned some of our core meta-game systems were online, such as Progression (which is mostly based on Challenges, an achievement-like system), or Scoring. These are powered by a stream of rich gameplay events triggered by the players and sent to the backend. These events range from the obvious - and very detailed - ”Kill” event when a NPC is eliminated in the game, to taking one of our many disguises or trespassing in a forbidden area. Each event is evaluated against a set of rules expressed in C#, and stored in memory for the duration of the session. A chain of evaluation is triggered against the full session when the player exits, or fails, to compute and award XP, challenges (achievements) and of course score and rank on global leaderboards.
- The way we approached this product in the online team was fairly new, we got a first iteration of the events system and the ”contract” services running quickly, and kept iterating ever since. Within the first year, we released a first internal build of the game, hosted on internal servers, then we migrated to Azure and made that available to the whole Square Enix group through Steam. The goal there was to practice regular releases and learning how to run a live product over extended period of time. Our online services are direct descendants of these initial baby steps, there were no throwaway prototypes, just continuous development and refactoring. Our approach was service oriented from day one, and we early on tried to have per-player service semantics to cope with scale and concurrency. To do this, we developed our own little .Net framework to declare our services and their operations, and their execution context. We had an early try on integrating a then private Microsoft research endeavour, code named Orleans. We ended up using it as our core distribution model, and I’ll go into more details about that part in a bit.
- This is a bird-view description of our game services, we have one of these for each of the 3 platforms we support : PC, PS4 and XBoxOne. Our game is communicating with the server through standard HTTPS, and the connection is established with what is called in Azure Cloud Services terms a Web Role, which is basically a managed node running a specific web application on an IIS server. This Web Role scales as needed depending on average CPU load, based on rules we defined, and is in charge of deserializing payloads and forwarding calls to our service framework. That service framework is powered by Microsoft Orleans, of which I will speak more in a bit, and actual service logic sits within a Worker Role, which can be described as a managed, distributed Windows Service. One nice thing about this is that each role is dealing with very different jobs, and are configured to run and scale independently. The webrole is stateless, doing a lot of serialization and deserialization work on top of dealing with HTTPS termination through IIS, while the service worker role is stateful, much more memory and CPU intensive. As a consequence, the web role is running on less core and memory, and scale very frequently, while the worker role is running on a bigger setup and scale less often. For storage we are following the general principle of using the right tool for the right job, and depending on access pattern we are saving our state to different services: Azure Redis for real time leaderboards Azure SQL for player inventory and contracts Azure Table Storage for high volume player data Azure Blob Storage for static configuration
- Our approach was service oriented from day one, and we early on tried to have per-player service semantics to cope with scale and concurrency. To do this, we developed our own little .Net framework to declare and execute service operations, and after testing different implementations, landed with Microsoft Orleans, which . Orleans is an Actor model framework for .net. Those of you knowing Erlang (or Elixir) probably know what this is about. An actor is an isolated, light weight single threaded ”program” running on a distributed system. On Orleans, actors are dynamically instanciated and the framework ensure that no concurrent calls can reach a given actor at any time. It means for Hitman that each of our players has his own actor as soon as he connects to the system, and will keep that running context for the duration as long as he stays active. This allows us to keep an in-memory state for each player, to reduce storage access to our various solutions. In case the cluster needs to scale up or down, actors will be migrated by the framework and we will only have to ensure the state is writted to persistent storage and restored on the new node. One of the nicest guarantee Orleans gives us is no concurrent calls on a stateful actor: this means we have (almost) no shared state to worry about on our system.
- One thing to note is that around the time we implemented our system with Orleans on top of Cloud Services (which are quite old-school now), Microsoft came up with a managed version of it, which is now the preferred way of running massively distributed code on Azure. It is called Service Fabric, and is doing the same thing more natively, with better tooling, and more efficiently. As we have our service running we have not started migrating yet, but if you are going to do anything with Azure you should definitely start there.
- Another aspect of our backend I would like to talk about is Analytics. Metrics are obviously a key part of game development, and serves multiple purpose. Of course there are some base KPI that you would like to have for any game, in order to follow things like acquisition and retention, but there is a world of other opportunities when gathering metrics more specific to your own product. We are sending telemetry to a separate data pipeline solely dedicated to analytics, and we use this collect data that helps us evaluate how our players play our game, and also keep an eye on error indicators, such as disconnection to the servers. This pipeline is built using some of the extensive Azure analytics offering. You can see the list here, but let’s take a closer look.
- The game sends telemetry to a collector webapp, which allow us to route the events to different endpoints when needed. This web app then writes the metrics to an Azure EventHub, which is a managed service similar to Apache Kafka in the non Azure world. This event hub has several Azure stream analytics jobs doing near-realtime aggregation. Some of these aggregations are directly written to Table storage and are used for monitoring and also reinjected into the game service, for things such as contracts trending. Another stream outputs the raw data into hourly raw files, to be consumed by pig scripts running on a on-demand Hadoop cluster every night. Then the aggregated files are pumped into SQL stored procedure to be exposed and used in reports. All of it is orchestrated by Data Factory, which keeps track of what tranform has been run for what time slice.
- This is what that setup looks like The box on the left is the raw output from the event stream, big flat files of JSON records stored in Azure Blob Storage. Then we have a fairly big PIG script running on an on-demand HDInsight (Hadoop) cluster, and spitting out a large number of data aggregated to tables. This is still put in blob storage, and is then passed on to dozen of activities reading these files and sending them to SQL stored procedures to insert into tables.
- Now having data is nice, but what did we do with it ? Beside all traditional reporting and the occasional investigation, we also are looking at ways to improve our game and level design. For instance, all Hitman players starts in the tutorial level you can see above, and we took a look at how they played on
- This is a map of where players get spotted when infiltrating the boat on the first tutorial mission. I only got static images in that presentation, but this is taken from an interactive report where we can filter points for the first minute of playtime for instance.
- Usually what happen when players get spotted is fairly straightforward: this is the map of where players get killed in the first 5 minutes of the playthrough. You can see they get killed a lot, and some in places that we considered very safe.
- And here we can see where they equip different disguises. All this data helped us to understand what could be improved when guiding first timers in our game, which is notoriously difficult, and very different from traditionnal stealth action games.
- Continuous release More than one release per month Automation of live tools (release, community, configuration) Game improvements Client resilience to network issues Contract mode improvements Contract search Player profiles Pro1 difficulty mode Live mindset The game is running out there