Data Management andStreaming Strategiesin Drakensang OnlineAndre “Floh” WeissflogHead of Development BerlinBigpoint GmbHBr...
2• Hack & Slay F2P MMO.• Embedded in browser.• No client download or installation.• Impressive 3D graphics.• 8 months from...
3
4
Team & Technology Background5Drakensang Online is developed by the same team that also worked on the previoussingle-player...
Data Management Core Demands6• Browser Game Experience Click & play, no installation.• Fast Startup Time Start in seconds,...
Drakensang Online: The Assets7• 50 interconnected maps• 240 monsters• 120 NPCs• 1400 items• 300 animated effects317 MB unc...
Two Main Achievements8Aggressive Data Size ReductionWe’re working with 3%of the data size we’ve been used to.This is mainl...
Maps as “static web pages”9This was really the initial idea behind our datamanagement.Our maps reference many small asset ...
Asset Size Reduction: Extreme Granularity10Our maps are built from really small and simple 3D objects:Extremely granular s...
Asset Size Reduction: Textures11Typical texturedimensions:256 x 256128 x 12864 x 64All textures are DXT compressed (and th...
Asset Size: Lights, Shadows& Decals12All lighting and shadowing is completely dynamic. No light-maps needed.We use deferre...
Two Streaming Strategies:131) ON DEMAND:Client requests data right when it’s needed.IO requests are handled asynchronously...
The web as a (really unreliable) hard-disc14Nebula3 always had a powerful I/O system:• async I/O is standard, not the exce...
“Massively Multi-Threaded IO”15IO DispatcherThreadMain ThreadRender ThreadAudio Thread… ThreadIncoming IO RequestsIO Threa...
Hierarchical File Cache16ROOT MD5 KEYe4680c8372bc6c527043a97631b404d7Directory “Table Of Content” FileLeaf TOCs Leaf TOCs ...
The Cache Chain17We basically have a 3-level cache for assets:RAM DISCCDNPROXYCDNORIGINLevel 1 Level 2 Level 3RAM Cache Hi...
404’s Considered Harmful18Remember this from the previous page:RAM DISCCDNPROXYCDNORIGIN1st 2nd 3rd 4th“Sh*t happens”.Tryi...
Conclusion19• Data management is the job of the whole team.• Careful, early asset size planning.• Team must commit on size...
Thank You Questions?20
Find us onBigpoint GmbHAlexanderstraße 510178 BerlinGermanyBigpoint Inc.500 Howard StreetSuite 300San Francisco, CA 94105B...
Upcoming SlideShare
Loading in …5
×

Data Management and Streaming Strategies in Drakensang Online

2,099 views
1,985 views

Published on

One of my older presentations from the Browser Games Forum 2011 about how we're managing data size optimizations and streaming in Drakensang Online. Absolute asset size number are about 3x..4x bigger today, since the game has grown a lot since 2011.

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,099
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
18
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Data Management and Streaming Strategies in Drakensang Online

  1. 1. Data Management andStreaming Strategiesin Drakensang OnlineAndre “Floh” WeissflogHead of Development BerlinBigpoint GmbHBrowser Games Forum 2011
  2. 2. 2• Hack & Slay F2P MMO.• Embedded in browser.• No client download or installation.• Impressive 3D graphics.• 8 months from concept to Closed Beta.
  3. 3. 3
  4. 4. 4
  5. 5. Team & Technology Background5Drakensang Online is developed by the same team that also worked on the previoussingle-player Drakensang “boxed titles”.Very skilled and experienced team, very solid and proven production pipelineand processes (and very little online experience).The technology behind our client and server is the Nebula3 engine (~250k lines of C/C++, 12years of continuous development).Epic stories will be written about how we went from building offline boxed-titles tocreating an MMO running in a browser in 8 months.…but not Today…Today we’ll talk about data managementand streaming in Drakensang Online…
  6. 6. Data Management Core Demands6• Browser Game Experience Click & play, no installation.• Fast Startup Time Start in seconds, not minutes• On-Demand Streaming: Only download what’s needed• Client-Side Caching: CDN traffic cost, user experience• Robustness: The web is slow and unreliable.• Minimize Data Size: But keep graphics quality high!
  7. 7. Drakensang Online: The Assets7• 50 interconnected maps• 240 monsters• 120 NPCs• 1400 items• 300 animated effects317 MB uncompressed139 MB compressed27035 filesAsset TypesTexturesAudioMeshesNavigationLocaleMapsAnimsOther
  8. 8. Two Main Achievements8Aggressive Data Size ReductionWe’re working with 3%of the data size we’ve been used to.This is mainly an organizational achievement.All departments have to agree on data size budgets……even if it’s hard (e.g. for the graphics guys)Seamless Data Streaming During GameplayThat’s mainly a programming achievement.Client engine must support asynchronous I/O from theground up.
  9. 9. Maps as “static web pages”9This was really the initial idea behind our datamanagement.Our maps reference many small asset files “somewhereon the web”, just like a static web page.All asset files are loaded asynchronously. Render loopcan deal with assets which are not yet loaded.Really clever local file caching prevents unnecessaryHTTP requests (down to no HTTP traffic at all if thecache is completely up-to-date).
  10. 10. Asset Size Reduction: Extreme Granularity10Our maps are built from really small and simple 3D objects:Extremely granular set ofreusable objects. Each withonly a few dozen to a fewhundred triangles.One map may have 100k of thosemicro 3D objects, but only a fewdozen DIFFERENT objects.That’s very good for datasize. But…OMG TEHDRAW CALLS !!Nebula3 uses hardwareinstancing and runtime-baking to reduce drawcalls from about 5000down to 250 per frame.
  11. 11. Asset Size Reduction: Textures11Typical texturedimensions:256 x 256128 x 12864 x 64All textures are DXT compressed (and then – like all other files – ZIP compressed)Bump textures use DXT5NM compression, aka the Carmack-Trick(that’s why they’re grey, not purple).Texture types:colorbumpemissivespecularTypical download sizes for textures are somewhere between 5 kByteand 40 kByte.Textures are aggressively shared and reused.
  12. 12. Asset Size: Lights, Shadows& Decals12All lighting and shadowing is completely dynamic. No light-maps needed.We use deferred Pre-Pass-Lighting to enable “infinite point-lights” withmoderate fill-rate requirements.Volumetric Decals are used to hide the tile-nature of the ground.Post-Effects add bloom, fog, color saturation and color balancing.
  13. 13. Two Streaming Strategies:131) ON DEMAND:Client requests data right when it’s needed.IO requests are handled asynchronously.May take several seconds until data is ready.Until then:Render a placeholder or…Render nothing.2) BACKGROUND STREAMING:Low-priority background thread just for cache warm-up.One streaming list per map created during build process.Download and update items in local file cache.Very speculative, can’t really predict what’s needed next.
  14. 14. The web as a (really unreliable) hard-disc14Nebula3 always had a powerful I/O system:• async I/O is standard, not the exception• “massively multi-threaded”, currently 10 I/O-threads• pluggable filesystems• all file paths are URLsWe built an “HTTP filesystem” on top of this:• only uses HTTP GET requests• transparently replaces local-disc file I/O• MD5 checksums for all files• ZIP compression for all files• hierarchical caching system• directory walking and file-exists checks• CDN support• reasonably fail-safe (high latencies, droppedconnections, corruptdownloads, tampering…)
  15. 15. “Massively Multi-Threaded IO”15IO DispatcherThreadMain ThreadRender ThreadAudio Thread… ThreadIncoming IO RequestsIO ThreadsCache(Local HD)CacheHit?YesWebServerNoUpdate CacheServed IO Requests• up to 10 HTTP-GET requests concurrently in flight (tweakable)• low-prio IO requests don’t block high-prio requests• no difference in high-level code between file I/O and http I/O
  16. 16. Hierarchical File Cache16ROOT MD5 KEYe4680c8372bc6c527043a97631b404d7Directory “Table Of Content” FileLeaf TOCs Leaf TOCs Leaf TOCs1. Daily-Build-Process: compress files and create per-file MD5 hashes.2. Each directory has a “Table Of Content” file with all MD5 hashes.3. TOCs are also compressed and MD5’d.4. ...with one single “Directory TOC” at the top.5. …and a single Root MD5 Key for the Directory TOC.• Client receives Root MD5 Key at startup.• Downloads TOC files if not in cache…• TOC files provide MD5 keys for files.• Zero HTTP GETs if cache is up-to-date!
  17. 17. The Cache Chain17We basically have a 3-level cache for assets:RAM DISCCDNPROXYCDNORIGINLevel 1 Level 2 Level 3RAM Cache Hit?Resource objects are shared in RAM using their ResourceIDs.DISC Cache Hit?MD5-named file must exist in local file cache.MD5 of actual file-content must match build-time MD5 hash.CDN Proxy Cache Hit?MD5-hash appended to HTTP GET URL, forces a cache miss if proxy only hasout-of-date file of the same name.CDN Origin Server has the definite data.
  18. 18. 404’s Considered Harmful18Remember this from the previous page:RAM DISCCDNPROXYCDNORIGIN1st 2nd 3rd 4th“Sh*t happens”.Trying to load non-existing assets will happen with such a complex data set.HTTP requests for non-existing files will cut through all the way to the origin server.CDN’s usually cache 404’s for a few seconds, but it’s better to not bother the web servers with404’s at all.Our Table-Of-Content-Files let us detect non-existing files on the client without a serverroundtrip.
  19. 19. Conclusion19• Data management is the job of the whole team.• Careful, early asset size planning.• Team must commit on size budgets.• “Programming Magic” alone doesn’t solve problems.• Programming, Level Design and Art Department must work hand in hand.• Oh, and: Check your CDN cost plan (if you pay per HTTP request, our solution is sub-optimal)
  20. 20. Thank You Questions?20
  21. 21. Find us onBigpoint GmbHAlexanderstraße 510178 BerlinGermanyBigpoint Inc.500 Howard StreetSuite 300San Francisco, CA 94105Bigpoint Distribuição deEntretenimento Online Ltda.Av. Brig. Faria Lima3729 cj. 52804538-905 São PauloBrazilBigpoint GmbHAndre WeissflogHead of Development BerlinDrehbahn 47-4820354 HamburgGermanyTel +49 40.88 14 13 - 0Fax +49 40.88 14 13 - 11info@bigpoint.netwww.bigpoint.netContact usBigpoint International ServicesLimited1 Villa ZimmermannTa’Xbiex TerraceXBX 1035 Ta’XbiexMalta21

×