It’s  Still  Loading? Designing an Efficient File System Scott Bilas Gas Powered Games
Introduction <ul><li>Me </li></ul><ul><ul><li>Scott Bilas, scott@gaspowered.com </li></ul></ul><ul><li>You </li></ul><ul><...
Introduction (cont.) <ul><li>What is a “file system”? </li></ul><ul><ul><li>Not particularly sexy, impressive, or difficul...
Four Basic Requirements <ul><li>My requirements for a “good file system” </li></ul><ul><ul><li>Easy to Use </li></ul></ul>...
Requirement 1: Easy to Use <ul><li>This is the top priority! </li></ul><ul><ul><li>Easy for  entire  team. </li></ul></ul>...
Requirement 1 (cont.): Easy to Use <ul><li>Easy for content developers </li></ul><ul><ul><li>Easy to drop in and modify re...
Requirement 2: Fast and Efficient <ul><li>Stay close to the hardware </li></ul><ul><ul><li>Exploit advantages inherent in ...
Requirement 3: Use Package Files <ul><li>Definition </li></ul><ul><ul><li>Packaged file format is a huge file containing m...
Requirement 3 (cont.): Use Package Files <ul><li>Advantages (cont.) </li></ul><ul><ul><li>Can be statically indexed for su...
Requirement 4: Solid Build Process <ul><li>Something needs to build the packages </li></ul><ul><ul><li>This is a standalon...
Six-Part Solution <ul><li>This should apply to nearly any type of game </li></ul><ul><ul><li>Especially effective with gam...
Solution Part 1: Organized Network Resources <ul><li>What does that mean? </li></ul><ul><ul><li>Put resources out on the n...
Solution Part 1 (cont.): Organized Network Resources <ul><li>Set it in stone </li></ul><ul><ul><li>Before production begin...
Solution Part 2: Simple File Access API <ul><li>“ FileHandle” API </li></ul><ul><ul><li>Provides Open, Read, Close functio...
Solution Part 2 (cont.):  Simple File Access API <ul><li>Need abstracted file system </li></ul><ul><ul><li>Requirement 1 s...
Solution Part 2 (cont.):  Simple File Access API <ul><li>Need abstracted file system (cont.) </li></ul><ul><ul><li>Abstrac...
Solution Part 2 (cont.): Simple File Access API <ul><li>Unique naming (Gabriel Knight 3) </li></ul><ul><ul><li>Data set is...
Solution Part 2 (cont.): Simple File Access API <ul><li>Relative unique naming (Dungeon Siege) </li></ul><ul><ul><li>Data ...
Solution Part 3: Path-Based File System <ul><li>Definition </li></ul><ul><ul><li>An interim file system needed for develop...
Solution Part 4: Flexible Package File Format <ul><li>Features </li></ul><ul><ul><li>Versioning (plan for content patches ...
Solution Part 5: Package File Builder <ul><li>It’s like WinZip or link.exe </li></ul><ul><ul><li>Everybody is familiar wit...
Solution Part 5 (cont.): Package File Builder <ul><li>Support filters </li></ul><ul><ul><li>Use a plug-in format to make i...
Solution Part 5 (cont.): Package File Builder <ul><li>Support data ordering </li></ul><ul><ul><li>Will speed up loads by m...
Solution Part 5 (cont.): Package File Builder <ul><li>Support compression </li></ul><ul><ul><li>Games keep getting more fi...
Solution Part 5 (cont.): Package File Builder <ul><li>Data drive the builder </li></ul><ul><ul><li>Using a scripting langu...
Solution Part 6: Package-Based File System <ul><li>Definition </li></ul><ul><ul><li>File manager that owns one or more pac...
Memory-Mapped Files: Overview <ul><li>Definitions </li></ul><ul><ul><li>File map: an object attached to a file from which ...
Memory-Mapped Files: How a Mapped View Works <ul><li>Built on a “section” object in OS memory manager </li></ul><ul><ul><l...
Memory-Mapped Files: Advantages <ul><li>Important:  memory-mapped files are your friends . </li></ul><ul><ul><li>If you’re...
Memory-Mapped Files: Usage <ul><li>To open a memory mapped package file </li></ul><ul><ul><li>CreateFile() read-only on yo...
Memory-Mapped Files: Error Handling <ul><li>Win32 Structured Exception Handling </li></ul><ul><ul><li>Off the end of the v...
Some Potential Issues <ul><li>Multi-CD games </li></ul><ul><ul><li>Separate out your data set, enforce with code. </li></u...
Some (More) Potential Issues <ul><li>Win9x </li></ul><ul><ul><li>There are many – read the knowledge base articles referen...
Future <ul><li>Extensions </li></ul><ul><ul><li>“ File mapping” over the Internet. </li></ul></ul><ul><ul><li>Local hard d...
Contact Info <ul><li>Scott Bilas [email_address] http://www.drizzle.com/~scottb/gdc </li></ul>
References <ul><li>Microsoft Developer Network (MSDN) http://msdn.microsoft.com </li></ul><ul><ul><li>KB articles Q125713,...
Upcoming SlideShare
Loading in …5
×

Its still loading

798 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
798
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Hi, my name is Scott Bilas. I was the technical lead on Sierra’s Gabriel Knight 3 (GK3) project. I’m currently working at Gas Powered Games doing yet another file system among other things for our Dungeon Siege project.
  • Now a file system isn’t sexy, impressive, or even difficult to engineer. It’s one of those back-end services where you see it as a success if nobody notices it. Like networking or memory management, people will only say something if it sucks – everybody hates slider bars that say “now loading level”. But there’s more to it than just performance, and that’s what this talk is about.
  • Combine: the mini16 format as an example of knowing how your data is used effectively. Other filters like shared polys lists and LOD generation.
  • Avoiding AutoPlay: see the Platform SDK docs on how to disable, though the Win95 recommendation sucks. Have your AutoPlay app check to see if your game is running at the top of its WinMain() (before creating any windows) and then bail immediately if it is.
  • Future thoughts: shouldn’t be difficult to abstract the concept and add files “mapped” across the internet. Implement memory mapping via virtual memory, where page faults will cause a page to be pulled over the internet and stored in a local map. Updates would be simple, too, with a properly designed resource database – updates of the header could invalidate and flush local pages, which when re-faulted, would transfer over the latest data. Make a big deal of abstracting those pointers. Talk about how barns work for reconnecting on a CD swap and/or dynamic resource update (possibly via Internet). Be very careful not to succumb to the temptation to just turn everything into pointers and use them. That’s not flexible, plus dangling pointers are Satan.
  • Its still loading

    1. 1. It’s Still Loading? Designing an Efficient File System Scott Bilas Gas Powered Games
    2. 2. Introduction <ul><li>Me </li></ul><ul><ul><li>Scott Bilas, scott@gaspowered.com </li></ul></ul><ul><li>You </li></ul><ul><ul><li>Win32 developers </li></ul></ul><ul><ul><li>Technical producers </li></ul></ul><ul><ul><li>Developers for other OS’s don’t leave! Win32 concepts should port over nicely. </li></ul></ul>
    3. 3. Introduction (cont.) <ul><li>What is a “file system”? </li></ul><ul><ul><li>Not particularly sexy, impressive, or difficult to engineer. (Sigh) </li></ul></ul><ul><ul><li>Executable code: find resources such as BMP’s or WAV’s, load or stream them into physical memory. </li></ul></ul><ul><ul><li>Content: the directory structure, naming conventions, and other process involved in content creation. </li></ul></ul><ul><li>Gabriel Knight 3 – the guinea pig </li></ul><ul><ul><li>Over 36,000 unique files in 2 gig of raw data. No pathing, all files uniquely named. </li></ul></ul><ul><ul><li>800 meg compressed ordered data over 3 CD’s. </li></ul></ul><ul><ul><li>File system 100% based on memory mapped files. </li></ul></ul>
    4. 4. Four Basic Requirements <ul><li>My requirements for a “good file system” </li></ul><ul><ul><li>Easy to Use </li></ul></ul><ul><ul><li>Fast and Efficient </li></ul></ul><ul><ul><li>Use Package Files </li></ul></ul><ul><ul><li>Solid Build Process </li></ul></ul>
    5. 5. Requirement 1: Easy to Use <ul><li>This is the top priority! </li></ul><ul><ul><li>Easy for entire team. </li></ul></ul><ul><li>Easy for engineers </li></ul><ul><ul><li>Simple and safe to code for engineers (especially junior level) using familiar file access conventions. </li></ul></ul><ul><ul><li>They are probably used to fopen(), fread(), fclose() style functions – provide an API like that first. </li></ul></ul><ul><ul><li>Provide a second API that goes directly to the metal using file mapping (more on this later). </li></ul></ul>
    6. 6. Requirement 1 (cont.): Easy to Use <ul><li>Easy for content developers </li></ul><ul><ul><li>Easy to drop in and modify resources. </li></ul></ul><ul><ul><li>Easy to verify that resources work properly - just run the game and see if it works. </li></ul></ul><ul><ul><li>Best if this can be done without restarting the game. </li></ul></ul><ul><ul><li>Fastest development process is: </li></ul></ul><ul><ul><ul><li>1. Alt-tab away from game. 4. Tell game to “rescan”. 2. Update content. 5. See results immediately! 3. Alt-tab back to game. </li></ul></ul></ul>
    7. 7. Requirement 2: Fast and Efficient <ul><li>Stay close to the hardware </li></ul><ul><ul><li>Exploit advantages inherent in a virtual memory OS by using memory mapped files. </li></ul></ul><ul><li>Support basic file access scenarios </li></ul><ul><ul><li>Block: probably the most common – one-shot block read used to initialize something (bitmaps). </li></ul></ul><ul><ul><li>Random: pure random access (huge file paging). </li></ul></ul><ul><ul><li>Streaming: like Random except serial (sounds). </li></ul></ul>
    8. 8. Requirement 3: Use Package Files <ul><li>Definition </li></ul><ul><ul><li>Packaged file format is a huge file containing multiple resources, possibly compressed. </li></ul></ul><ul><ul><li>Examples are WAD, HOG, ZIP, BRN. </li></ul></ul><ul><li>Advantages </li></ul><ul><ul><li>Use only a single file handle left open all the time. Opening and closing handles is expensive, especially on NT. </li></ul></ul><ul><ul><li>Maximize disc space usage efficiency. </li></ul></ul>
    9. 9. Requirement 3 (cont.): Use Package Files <ul><li>Advantages (cont.) </li></ul><ul><ul><li>Can be statically indexed for super-fast access. </li></ul></ul><ul><ul><li>Support easy versioning for future expansion and resource patching. </li></ul></ul><ul><ul><li>Requires a build process – a very good thing. </li></ul></ul><ul><ul><li>More professional, easier to manage install (one file vs. thousands). </li></ul></ul><ul><ul><li>Source data is more difficult to rip from the game. </li></ul></ul>
    10. 10. Requirement 4: Solid Build Process <ul><li>Something needs to build the packages </li></ul><ul><ul><li>This is a standalone tool that “compiles and links” all resources together into the big package file(s). </li></ul></ul><ul><li>Support “filtering” of the data as it goes in </li></ul><ul><ul><li>Ideal opportunity to verify data integrity. </li></ul></ul><ul><ul><li>Easy to extend build tool to modify the data as it packages it. </li></ul></ul><ul><li>Streamline build process </li></ul><ul><ul><li>One big batch file can do it all! </li></ul></ul>
    11. 11. Six-Part Solution <ul><li>This should apply to nearly any type of game </li></ul><ul><ul><li>Especially effective with games where frame-to-frame content requirements greatly exceed available system memory. </li></ul></ul><ul><li>This solution shipped with Gabriel Knight 3. </li></ul><ul><li>A variation of this will ship with Dungeon Siege. </li></ul>
    12. 12. Solution Part 1: Organized Network Resources <ul><li>What does that mean? </li></ul><ul><ul><li>Put resources out on the net and have the development team run exclusively from net data. </li></ul></ul><ul><ul><li>Source control systems make this easy – Visual SourceSafe offers “shadow directories”. </li></ul></ul><ul><ul><li>Choose sensible naming conventions, enforce from code if possible. </li></ul></ul>
    13. 13. Solution Part 1 (cont.): Organized Network Resources <ul><li>Set it in stone </li></ul><ul><ul><li>Before production begins and inertia takes over. </li></ul></ul><ul><li>Everybody is in sync </li></ul><ul><ul><li>All developers guaranteed to run from the same data. Nobody should have outdated or incorrect content. </li></ul></ul><ul><ul><li>Minimizes the always mysterious “it works fine on my machine” phenomenon. </li></ul></ul>
    14. 14. Solution Part 2: Simple File Access API <ul><li>“ FileHandle” API </li></ul><ul><ul><li>Provides Open, Read, Close functions. </li></ul></ul><ul><ul><li>Familiar to all programmers, easy to port code. </li></ul></ul><ul><ul><li>Base it on memory-mapped files underneath. </li></ul></ul><ul><li>“ MemHandle” API </li></ul><ul><ul><li>Provides Map, GetPointer, UnMap functions. </li></ul></ul><ul><ul><li>Directly uses memory mapped files. </li></ul></ul><ul><ul><li>Easy to understand and use. </li></ul></ul>
    15. 15. Solution Part 2 (cont.): Simple File Access API <ul><li>Need abstracted file system </li></ul><ul><ul><li>Requirement 1 says to run from network resources. This requires a system that works with raw files found on local or remote paths. </li></ul></ul><ul><ul><li>Requirement 3 says to use package files. This requires a system that knows how to pull resources from one or more package files. </li></ul></ul>
    16. 16. Solution Part 2 (cont.): Simple File Access API <ul><li>Need abstracted file system (cont.) </li></ul><ul><ul><li>Abstract both the FileHandle and MemHandle API’s in a base class and derive both file manager types from it. </li></ul></ul><ul><ul><li>Allow simultaneous use of multiple file systems. </li></ul></ul><ul><li>Solution affected by naming conventions </li></ul><ul><ul><li>Must choose “unique” or “relative unique” convention for names (this affects indexing). </li></ul></ul>
    17. 17. Solution Part 2 (cont.): Simple File Access API <ul><li>Unique naming (Gabriel Knight 3) </li></ul><ul><ul><li>Data set is treated as a flat array of resources with no path information. </li></ul></ul><ul><ul><li>Example: OpenFile( “wood.bmp” ) where file may exist anywhere on the available path trees. </li></ul></ul><ul><ul><li>Advantages: simple to use, packaged files are easy to implement, packaged file index lookups are fast. </li></ul></ul><ul><ul><li>Disadvantages: requires indexing of paths in development builds (slow), can have problems with duplicate filenames, harder to enforce naming conventions, takes longer to implement. </li></ul></ul>
    18. 18. Solution Part 2 (cont.): Simple File Access API <ul><li>Relative unique naming (Dungeon Siege) </li></ul><ul><ul><li>Data set is treated like a directory tree. Files are accessed through relative paths. </li></ul></ul><ul><ul><li>Example: OpenFile( “artmapswood.bmp” ) where relative path to file must always be specified. </li></ul></ul><ul><ul><li>Advantages: no indexing required in development builds, duplicate names are not possible, easy to enforce file location standards, quick to implement. </li></ul></ul><ul><ul><li>Disadvantages: implementation of package file system more difficult, lookups in package files may be slower. </li></ul></ul><ul><li>Which is better? </li></ul>
    19. 19. Solution Part 3: Path-Based File System <ul><li>Definition </li></ul><ul><ul><li>An interim file system needed for development. </li></ul></ul><ul><ul><li>Give it a set of root paths and it pulls files from those trees. </li></ul></ul><ul><ul><li>Optionally support absolute and override pathing. </li></ul></ul><ul><li>Features </li></ul><ul><ul><li>Minimize tree iteration across the network by using indexes. </li></ul></ul><ul><ul><li>Should be able to use new or changed resources without restarting the game. </li></ul></ul><ul><ul><li>Don’t just limit it to development builds? </li></ul></ul>
    20. 20. Solution Part 4: Flexible Package File Format <ul><li>Features </li></ul><ul><ul><li>Versioning (plan for content patches and expansion packs). </li></ul></ul><ul><ul><li>Special data formats (such as compression or encryption) transparent to client code. </li></ul></ul><ul><ul><li>Use the Win32 Portable Executable file format and put a VERSIONINFO resource in there. </li></ul></ul><ul><li>Check your project requirements </li></ul><ul><ul><li>If multi-CD game, will require cross-package indexing. </li></ul></ul><ul><ul><li>May want to support adding new packages on the fly (via Web download?) </li></ul></ul>
    21. 21. Solution Part 5: Package File Builder <ul><li>It’s like WinZip or link.exe </li></ul><ul><ul><li>Everybody is familiar with the “package file builder concept” </li></ul></ul><ul><ul><li>Process is: 1. choose files, 2. do something to them, 3. pack them end to end in the output file. </li></ul></ul><ul><ul><li>Keep it as simple as possible. </li></ul></ul><ul><li>Command line or GUI? </li></ul><ul><ul><li>Command line! Easier to implement and can be batched for auto-generating packaged files. </li></ul></ul>
    22. 22. Solution Part 5 (cont.): Package File Builder <ul><li>Support filters </li></ul><ul><ul><li>Use a plug-in format to make it easy to add new filters later. </li></ul></ul><ul><ul><li>Examples of mutation filters: PSD-to-raw bitmap conversion, script precompiler, MP3 encoder. </li></ul></ul><ul><ul><li>Examples of verification filters: naming convention checker, zero file size checker, content syntax verifier. </li></ul></ul><ul><ul><li>Probably don’t want to do anything really advanced. </li></ul></ul>
    23. 23. Solution Part 5 (cont.): Package File Builder <ul><li>Support data ordering </li></ul><ul><ul><li>Will speed up loads by minimizing slow CD seeks. </li></ul></ul><ul><ul><li>Will speed up transfers by maximizing locality of reference for caches. The system and drive hardware will read ahead anyway – don’t throw that data away! </li></ul></ul><ul><ul><li>Implement in game’s EXE code via journaling. </li></ul></ul>
    24. 24. Solution Part 5 (cont.): Package File Builder <ul><li>Support compression </li></ul><ul><ul><li>Games keep getting more files, more content. </li></ul></ul><ul><ul><li>Reading data from CD and decompressing it may be faster than reading uncompressed data. </li></ul></ul><ul><ul><li>zlib is high compression ratio, supports streaming, good decompression rate. </li></ul></ul><ul><ul><li>LZO is good compression ratio, does not support streaming, but ultra-fast decompression rate. </li></ul></ul><ul><ul><li>GK3 used zlib for streaming files, LZO for all others. </li></ul></ul>
    25. 25. Solution Part 5 (cont.): Package File Builder <ul><li>Data drive the builder </li></ul><ul><ul><li>Using a scripting language is probably bad judging from GK3 experience. </li></ul></ul><ul><ul><li>Use a configuration (INI style) language instead. </li></ul></ul>
    26. 26. Solution Part 6: Package-Based File System <ul><li>Definition </li></ul><ul><ul><li>File manager that owns one or more packaged files. </li></ul></ul><ul><ul><li>Can quickly search through index(es) to find a resource in one of those packages. </li></ul></ul><ul><ul><li>A resource is identified through an entry giving its offset and size within the package. </li></ul></ul><ul><ul><li>Exclusively uses memory-mapped files. </li></ul></ul><ul><li>Easy to implement </li></ul><ul><li>But first, what are memory-mapped files? </li></ul>
    27. 27. Memory-Mapped Files: Overview <ul><li>Definitions </li></ul><ul><ul><li>File map: an object attached to a file from which views can be created. A file mapping object is just a memory structure and does not actually allocate any memory or address space. Create one per file. </li></ul></ul><ul><ul><li>View: a region of virtual address space “backed” by the file. Create many per file. </li></ul></ul>
    28. 28. Memory-Mapped Files: How a Mapped View Works <ul><li>Built on a “section” object in OS memory manager </li></ul><ul><ul><li>Pages in virtual memory are directly mapped onto a file in 4K chunks. </li></ul></ul><ul><li>Lazy evaluation is key </li></ul><ul><ul><li>Mapping a view only reserves contiguous address space, does not take up any physical memory at first. </li></ul></ul><ul><ul><li>Accessing a page that has not been assigned physical memory is called a “page fault”. </li></ul></ul><ul><ul><li>File chunks are only brought into memory (“paged in”) on demand. </li></ul></ul>
    29. 29. Memory-Mapped Files: Advantages <ul><li>Important: memory-mapped files are your friends . </li></ul><ul><ul><li>If you’re not using them, you should. They’re fast and easy! </li></ul></ul><ul><ul><li>Can be used for many things – here we only use them for accessing read-only data. See docs for other uses. </li></ul></ul><ul><li>Advantages </li></ul><ul><ul><li>Ideal for read-only package files. Files mapped read-only are “backed” by the file itself. Old pages will just be tossed, not swapped out to the page file. </li></ul></ul><ul><ul><li>Eliminates unnecessary layers of code. </li></ul></ul><ul><ul><li>Eliminates typical read/process/discard file procedures. Just get a pointer and dereference it. </li></ul></ul><ul><ul><li>Usage is more intuitive: it’s just read-only memory. </li></ul></ul>
    30. 30. Memory-Mapped Files: Usage <ul><li>To open a memory mapped package file </li></ul><ul><ul><li>CreateFile() read-only on your packaged file. </li></ul></ul><ul><ul><li>CreateFileMapping() read-only on the entire file. </li></ul></ul><ul><li>To access a packed resource </li></ul><ul><ul><li>Look up the resource name in the packed file index. Get its size and offset. </li></ul></ul><ul><ul><li>MapViewOfFile() to the offset and size where it’s located (aligned to system allocation granularity). </li></ul></ul><ul><ul><li>Use the returned const void* pointer. </li></ul></ul>
    31. 31. Memory-Mapped Files: Error Handling <ul><li>Win32 Structured Exception Handling </li></ul><ul><ul><li>Off the end of the view? Dangling or uninitialized pointer? No surprise: access violation! </li></ul></ul><ul><ul><li>Special case: damaged files, dirty CD? Not a read error, but an “in-page error”. </li></ul></ul><ul><li>Write a top-level SEH handler for in-page error </li></ul><ul><ul><li>Not much you can do about it. </li></ul></ul><ul><ul><li>Win95 will probably blue-screen (but not BSOD). </li></ul></ul><ul><ul><li>Just reformat the error so end users don’t freak out. </li></ul></ul>
    32. 32. Some Potential Issues <ul><li>Multi-CD games </li></ul><ul><ul><li>Separate out your data set, enforce with code. </li></ul></ul><ul><ul><li>Possibility of emergency CD swaps. </li></ul></ul><ul><ul><li>Detecting CD changes, avoiding AutoPlay problems. </li></ul></ul><ul><ul><li>Complications with deciding what to install. </li></ul></ul><ul><li>Running from network resources </li></ul><ul><ul><li>Files will be locked while accessed by game. </li></ul></ul><ul><ul><li>Should special-case file system code to work around this. Detect files accessed from network and copy them to local memory buffers. </li></ul></ul>
    33. 33. Some (More) Potential Issues <ul><li>Win9x </li></ul><ul><ul><li>There are many – read the knowledge base articles referenced at the end of this talk. </li></ul></ul><ul><ul><li>It’s “emulated” and so certain features are not supported (this wasn’t a problem for GK3) </li></ul></ul><ul><ul><li>1 gig shared memory region </li></ul></ul>
    34. 34. Future <ul><li>Extensions </li></ul><ul><ul><li>“ File mapping” over the Internet. </li></ul></ul><ul><ul><li>Local hard drive caching of CD data. </li></ul></ul><ul><ul><li>Possibility of shipping the package builder as part of the installer. </li></ul></ul><ul><ul><li>Incremental builder rather than straight compile. </li></ul></ul><ul><li>Q&A. </li></ul>
    35. 35. Contact Info <ul><li>Scott Bilas [email_address] http://www.drizzle.com/~scottb/gdc </li></ul>
    36. 36. References <ul><li>Microsoft Developer Network (MSDN) http://msdn.microsoft.com </li></ul><ul><ul><li>KB articles Q125713, Q108231 </li></ul></ul><ul><ul><li>Working sample FASTFILE.C in Platform SDK </li></ul></ul><ul><ul><li>Article: “Improving the Performance of Windows 95 Games” </li></ul></ul><ul><li>Inside Windows NT (2 nd Ed.) by David Solomon Microsoft Press </li></ul><ul><li>zlib, a free compression library http://www.cdrom.com/pub/infozip/zlib/ </li></ul><ul><li>LZO, a free compression library http://wildsau.idv.uni-linz.ac.at/mfx/lzo.html </li></ul>

    ×