SlideShare a Scribd company logo
1 of 44
Download to read offline
One Year of Porting
Post-mortem of two Linux/SteamOS launches
Leszek Godlewski
Who is this guy?
Leszek Godlewski
Programmer, Nordic Games (early 2014 – now)
– Unannounced project
Freelance Programmer (Sep 2013 – early 2014)
– Linux port of Painkiller Hell & Damnation
– Linux port of Deadfall Adventures
Generalist Programmer, The Farm 51 (Mar 2010 – Aug
2013)
– Painkiller Hell & Damnation, Deadfall Adventures
Focus
● Not sales figures
● Not business viability
● Not game-specific bugs
● Not the Steam Controller – oops! �
● Platform-specific problems
● Mistakes made & mitigation attempts
Agenda
● The ports
● Laying down the foundations
– Build system
– Compilers
– Linking
– Boilerplate
● Release and feedback
– User issues
– Crash handling
– GLSL shader linking
The ports
Painkiller Hell & Damnation (The Farm 51)
Deadfall Adventures (The Farm 51)
Facts
● Unreal Engine 3
● All major Linux distros
– SteamOS, Debian, Ubuntu, Fedora, Arch, Gentoo
● All official drivers
– NVIDIA, AMD, Intel (i965)
● Some open-source drivers
– Gallium r600, Gallium radeonsi
Facts
● Most of UE3 middlewares have Linux versions
– In our case: PhysX, FaceFX, Scaleform Gfx, lzopro, Bink...
● Introduced open-source middlewares
– SDL 2.x, GLEW, Steam Runtime
● UE3's build system – Unreal Build Tool
– Handles everything make does
– Written in C#, fixed up to run in Mono on Linux
● UE3's content packaging (cooking) system
– Linux target based on Mac OSX
Facts
● QA department unfamiliar with Linux
– Basic training was required
●
Installing & running software (including from the command line), file
permissions, driver installation, gathering system information...
– Mostly reported false positives in the beginning
● Spare time project over ~13 months
– After leaving The Farm 51 employment – contracted for further
outsourcing directly by TF51
– Occasional support from individual members of TF51 staf
●
Kudos to Piotr Bąk and Wojciech Knopf!
Overlap
● Noticed how a lot of work was based on OSX code?
● Happens all the time
– POSIX
– OpenGL/OpenAL
MacOS X
Linux Mobile
Agenda
● The ports
● Laying down the foundations
– Build system
– Compilers
– Linking
– Boilerplate
● Release and feedback
– User issues
– Crash handling
– GLSL shader linking
Laying down the foundations
Starting point
● Epic's OpenGL 2.1 and OpenAL back-ends
– OpenGL mode somewhat functional in Windows developer builds
● Epic's Mac OSX port
– Limited test builds for Mac OSX had been made before
– Mac OSX binary builds supported via remote compiltion
– Existing Mac OSX target for game content packaging (cooking)
● Both of the above – somewhat... unfinished �
● On Windows, the games shipped 32-bit binaries only
Building the build tool – C# & Mono
● Patched the Unreal Build Tool to build & run on Mono in
Linux
– Mono can handle most .NET commandline apps all right
● Added support for Linux toolchains (duh)
● Fixed hardcoding of backslashes in paths
– Path.Join() instead
● Fixed regexes on large strings (C++ sources) blowing up
the stack
– Break up the string into smaller parts
Cross-compiling for 32/64-bit
● Yes, I agree, 32-bit should die, but one may not be
allowed to kill it
● gcc -m32/-m64 is not enough!
– Sets target code generation
– But not headers & libraries (CRT, OpenMP, libgcc etc.)
● Fixed (on Debian & friends) by installing gcc-multilib
– Dependency package for non-default architectures (i.e. i386 on an amd64
system and vice versa)
Clang
● Clang is faster
– gcc: 3m47s
– Clang: 3m05s
● Clang has diferent diagnostics than gcc
● Clang has C++ preprocessor macro compatibility with gcc
– Declares __GNUC__ etc.
● Clang has commandline compatibility with gcc
– Can easily switch back & forth between gcc and Clang
Clang - caveats
● Object files may be incompatible with gcc & fail to link
(need full rebuilds)
● gcc is more mature than Clang
– Clang has generated faulty code for me (YMMV)
● Slight inconsistencies in C++ standard strictness
– Templates
– Anonymous structs/unions
– May need to add this-> in some places
– May need to name some anonymous types
So – Clang or gcc?
Both:
● Clang – quick iterations during development
● gcc – final shipping binaries
Linking – GNU ld
● Default linker on Linux
● Ancient
● Single-threaded
● Requires specification of libraries in the order of reverse
dependency...
● We are not doomed to use it!
Linking – GNU gold
● Multi-threaded linker for ELF binaries
– ld: 18s
– gold: 5s
● Developed at Google, now officially part of GNU binutils
● Drop-in replacement for ld
– May need an additional parameter or toolchain setup
●
clang++ -B/usr/lib/gold-ld ...
●
g++ -fuse-ld=gold ...
● Still needs libs in the order of reverse dependency...
Linking – library groups
● Major headache/game-breaker with circular dependencies
– ”Proper” fix: re-specify the same libraries over and over again
● Declare library groups instead
– Wrap library list with --start-group –end-group
●
Shorthand: -(, -)
●
g++ foo.obj -Wl,-( -lA -lB -Wl,-)
● Caveat: results in exhaustive symbol search within the group
– Manual warns of possible performance hit
– Not observed here, but keep that in mind!
Caching the gdb-index
● Large codebase generates heavy debug symbols (hundreds of megabytes)
● gdb generates the index for quick symbol lookup...
● ...at every single gdb startup �
– Takes several minutes for said codebases
– Massive waste of time!
● Solution: cache the index, fold it into the build process!
– Full description in the gdb manual
– gdb -batch -ex "save gdb-index $(OUTPUT_PATH)/gdb-index" $
(BINARY)
– objcopy --add-section .gdb_index=$(OUTPUT_PATH)/gdb-index/$
(BINARY).gdb-index --set-section-flags .gdb_index=readonly $
(BINARY) $(BINARY)
Raw X11 or SDL?
● Initially tried rolling my own boilerplate
– Basic X11 mouse, window and key press events are easy
– Unicode text input is not
– Useful windowing is not
– Correct GLX is not
– Linux joystick API is not
– Above all, X11 seems to be on its way out
●
Wayland & Mir will have emulation layers, but that's bound to have
overhead
● You really want to use SDL 2 instead, trust me
– Shameless plug: see my talk from WGK 2013 for benefits of using SDL 2 ☺
Agenda
● The ports
● Laying down the foundations
– Build system
– Compilers
– Linking
– Boilerplate
● Release and feedback
– User issues
– Crash handling
– GLSL shader linking
Release and feedback
What we shipped initially with the beta
● 32-bit binaries (64-bit added later on)
● Launch script (~20 lines)
– Architecture detection
●
Initially a stub for 64-bit with fallback to 32-bit
– Steam Runtime injection (if not already present)
● That's about it ☺
● Explicit dependency on the Steam Runtime
– Allows shifting some responsibility to Valve
– And, admittedly, to users who insist on using their own dependencies
User issues
● Missing/incompatbile libraries
– Resulting from disabling the Steam Runtime
●
Gentoo users, mostly... Maintainer of steam package had chosen to disable
it by default
– Usually fixed by force-starting Steam with STEAM_RUNTIME=1
●
$ STEAM_RUNTIME=1 steam
● ”Missing” 32-bit NVIDIA OpenGL libraries on 64-bit systems
– Apparently, they might end up unreachable by the dynamic linker
– Fixed by adding /usr/lib32 to LD_LIBRARY_PATH in the launch script
– Also, prompt user to make sure they did install them
●
It's an option - ”install compatibility 32-bit libraries”
User issues
● No support for DXT texture compression despite capable
hardware (GL_EXT_texture_compression_s3tc)
– Concerns the open-source drivers
– For legal reasons (S3/VIA patents), some distros don't ship it or install it
automatically
●
E.g. Fedora
– If extension not advertised by driver, suggest the user to install
libtxc_dxtn
●
Often a distro package, so no hassle
More user issues...
● Graphical glitches...
● Broken V-sync...
● Broken NVIDIA Optimus with open-source multiplexer...
● Looong & unresponsive loading times...
● A whole lot of crashes...
● Most of the above was my fault – not going to bore you
with all of this!
Crash handling
● Unix signals
– Asynchronous IPC notification mechanism in POSIX-compliant systems
●
Sources can be the process itself, other processes or the kernel
– Default handler terminates process & dumps core for most signals
– Can (must?!) specify custom handlers
● Get/set handlers via the sigaction(2) system call
– Handler prototype: void sa_handler(int signal,
siginfo_t *siginfo,
void *context);
● More information
– G. Ben-Yossef, Crash N' Burn: Writing Linux application fault handlers
Interesting siginfo_t fields
● si_errno – errno value
– Possibly more detailed error code
● si_code – reason for sending the signal
– Both general and per signal type
– Examples: issued by user, issued by kernel, illegal addressing mode, FP
over/underflow, invalid memory permissions, unmapped address etc.
● si_addr – memory location at which fault happened
– If applicable: SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGTRAP
Signal handler caveats
● Not safe to allocate or free heap memory!
– Fault may have corrupted the allocator's data structures
● Prone to race conditions
– Can't share locks with the main program!
●
If signalled after locking, you'll deadlock
– Can't call async-unsafe functions!
●
See manual for signal(7) for a list of safe ones
● Custom handlers do not dump core (a.k.a. minidump)
– Mitigated by restoring default handler after custom logging and re-signalling self
●
signal(signum, SIG_DFL); raise(signum);
Safe stack walking
● glibc provides backtrace() and friends
● Symbols are read from the dynamic symbol table
– Must pass -rdynamic to gcc/Clang to populate
● Calling backtrace_symbols() allocates heap
memory
– Not safe... ☹
– Still, can get away with it most of the time
– Proper solution involves a separate watchdog process & pipes (heap-less
backtrace_symbols_fd() call instead)
Long load times? Unresponsiveness?
● Profiling quickly places blame on shader linking
– OpenGL shader model operates on program objects, created by linking shader pipeline
combinations
●
Introduces lots of redundancy (see glGetProgramiv() & glGetShaderiv())
●
Drivers often defer actual compilation until ”link time”
●
Increased memory consumption
– UE3 OpenGL renderer blocks the render thread for linking
●
Render thread blocked → Frozen loading screen!
– Both games have thousands of shaders
●
An awful lot of vertex/fragment shader combinations (programs) �
● Moreover – makes async level streaming blocking!
– Bad stuttering during gameplay
● Situation better on subsequent loads on NVIDIA due to in-driver cache
Shader linking
● Short-term fix: background shader linking
– Worker thread with a separate OpenGL context, sharing data with the main one
– Queue all shader link jobs, execute on the worker only
– If on a loading screen, keep spinning it while waiting for the shaders
– Defer ”async streaming done” notifications till shader link queue is empty
● Pros:
– Quick & easy to implement
– Fixes gameplay stuttering
● Cons:
– Only fixes unresponsiveness, not the long load times ☹
Shader linking
● Disaster on the official AMD Catalyst driver!
– Total system hang (PC needs hard reset)
– Apparently, exposed a race condition in AMD driver
– AMD has yet to ship the fix...
● Fallback to old, blocking code path if Catalyst detected
Shader linking
● Possible improvement (suggested by Epic):
ARB_separate_shader_objects
– Replaces programs (and linking) with much lighter pipeline objects
●
Removes a lot of redundancy
– Makes use of separate vertex/fragment shaders (D3D-like)
– Would play well with UE3's RHI, modelled mostly after D3D
● Not implemented ☹ – requires shader syntax upgrade
and a refactor of UE3's OpenGL renderer
– Explicit locations for attributes and varyings required for SSO
– Need to bump GLSL from 1.20 (OpenGL 2.1) to at least 1.40 (OpenGL 3.1)
Shader linking
● Proper fix: deferred shader access
– Modern drivers queue shader compiles and links internally and process
them in a multithreaded manner
●
Official NVIDIA & AMD Catalyst
●
Open-source Mesa drivers in SteamOS (patches pushed upstream
recently by Valve)
– Kick all the jobs (i.e. create shader objects) at level load
– Do not access the objects (query, draw) until they are needed
– Not even the compile/link status! This creates a sync point!
● Not implemented ☹ – requires a considerable refactor of
UE3's OpenGL renderer
Summary
Takeaway 1/2
● Porting .NET-based tools to Linux is viable
● Many 32/64-bit cross-compile issues are solved with
gcc-multilib
● Switching back and forth between Clang and gcc is easy
and useful
● Link times can be greatly improved by using gold
● Caching the gdb-index improves debugging experience
● Using SDL 2 is way better than rolling your own boilerplate
Takeaway 2/2
● Using the Steam Runtime is good for you
● Crash handling in Linux is easy to do, tricky to get right
● OpenGL shader model is significantly diferent from D3D's
● GLSL linking is slow, so defer access if possible
● Multiple concurrent OpenGL contexts can still bite you
● Test on different GPU drivers to avoid unpleasant
surprises!
@ l go d l ews k i @ n o rd i c ga m e s . at
t @ T h e I n e Q u ati o n
K w w w. i n e q u ati o n . o rg
Questions?
F u rt h e r N o rd i c G a m e s i nfo rm ati o n :
K w ww. n o rd i c ga m e s . at
Deve l o p me nt i nfo rmati o n :
K ww w. gr i m l o re ga m e s . co m
Thank you!

More Related Content

What's hot

Digging for Android Kernel Bugs
Digging for Android Kernel BugsDigging for Android Kernel Bugs
Digging for Android Kernel BugsJiahong Fang
 
Android Variants, Hacks, Tricks and Resources
Android Variants, Hacks, Tricks and ResourcesAndroid Variants, Hacks, Tricks and Resources
Android Variants, Hacks, Tricks and ResourcesOpersys inc.
 
Android Variants, Hacks, Tricks and Resources presented at AnDevConII
Android Variants, Hacks, Tricks and Resources presented at AnDevConIIAndroid Variants, Hacks, Tricks and Resources presented at AnDevConII
Android Variants, Hacks, Tricks and Resources presented at AnDevConIIOpersys inc.
 
01 linux-quick-start
01 linux-quick-start01 linux-quick-start
01 linux-quick-startNguyen Vinh
 
Why kernelspace sucks?
Why kernelspace sucks?Why kernelspace sucks?
Why kernelspace sucks?OpenFest team
 
A War Story: Porting Android 4.0 to a Custom Board (ELCE 2012)
A War Story: Porting Android 4.0 to a Custom Board (ELCE 2012)A War Story: Porting Android 4.0 to a Custom Board (ELCE 2012)
A War Story: Porting Android 4.0 to a Custom Board (ELCE 2012)Matthias Brugger
 
Android Hacks, Variants, Tricks and Resources ESC SV 2012
Android Hacks, Variants, Tricks and Resources ESC SV 2012Android Hacks, Variants, Tricks and Resources ESC SV 2012
Android Hacks, Variants, Tricks and Resources ESC SV 2012Opersys inc.
 
Understanding the Android System Server
Understanding the Android System ServerUnderstanding the Android System Server
Understanding the Android System ServerOpersys inc.
 
Linux: the first second
Linux: the first secondLinux: the first second
Linux: the first secondAlison Chaiken
 
Using Erlang in an Embedded and Cross-Compiled World
Using Erlang in an Embedded and Cross-Compiled WorldUsing Erlang in an Embedded and Cross-Compiled World
Using Erlang in an Embedded and Cross-Compiled WorldFrank Hunleth
 
Don't Give Credit: Hacking Arcade Machines
Don't Give Credit: Hacking Arcade MachinesDon't Give Credit: Hacking Arcade Machines
Don't Give Credit: Hacking Arcade MachinesMichael Scovetta
 
Jagan Teki - U-boot from scratch
Jagan Teki - U-boot from scratchJagan Teki - U-boot from scratch
Jagan Teki - U-boot from scratchlinuxlab_conf
 
Valerio Di Giampietro - Introduction To IoT Reverse Engineering with an examp...
Valerio Di Giampietro - Introduction To IoT Reverse Engineering with an examp...Valerio Di Giampietro - Introduction To IoT Reverse Engineering with an examp...
Valerio Di Giampietro - Introduction To IoT Reverse Engineering with an examp...linuxlab_conf
 
BUD17-104: Scripting Languages in IoT: Challenges and Approaches
BUD17-104: Scripting Languages in IoT: Challenges and ApproachesBUD17-104: Scripting Languages in IoT: Challenges and Approaches
BUD17-104: Scripting Languages in IoT: Challenges and ApproachesLinaro
 
Build your own embedded linux distributions by yocto project
Build your own embedded linux distributions by yocto projectBuild your own embedded linux distributions by yocto project
Build your own embedded linux distributions by yocto projectYen-Chin Lee
 
Inside Android's UI
Inside Android's UIInside Android's UI
Inside Android's UIOpersys inc.
 
Michele Dionisio & Pietro Lorefice - Developing and testing a device driver w...
Michele Dionisio & Pietro Lorefice - Developing and testing a device driver w...Michele Dionisio & Pietro Lorefice - Developing and testing a device driver w...
Michele Dionisio & Pietro Lorefice - Developing and testing a device driver w...linuxlab_conf
 

What's hot (20)

Digging for Android Kernel Bugs
Digging for Android Kernel BugsDigging for Android Kernel Bugs
Digging for Android Kernel Bugs
 
Android Variants, Hacks, Tricks and Resources
Android Variants, Hacks, Tricks and ResourcesAndroid Variants, Hacks, Tricks and Resources
Android Variants, Hacks, Tricks and Resources
 
Android Variants, Hacks, Tricks and Resources presented at AnDevConII
Android Variants, Hacks, Tricks and Resources presented at AnDevConIIAndroid Variants, Hacks, Tricks and Resources presented at AnDevConII
Android Variants, Hacks, Tricks and Resources presented at AnDevConII
 
Porting Android
Porting AndroidPorting Android
Porting Android
 
01 linux-quick-start
01 linux-quick-start01 linux-quick-start
01 linux-quick-start
 
Why kernelspace sucks?
Why kernelspace sucks?Why kernelspace sucks?
Why kernelspace sucks?
 
A War Story: Porting Android 4.0 to a Custom Board (ELCE 2012)
A War Story: Porting Android 4.0 to a Custom Board (ELCE 2012)A War Story: Porting Android 4.0 to a Custom Board (ELCE 2012)
A War Story: Porting Android 4.0 to a Custom Board (ELCE 2012)
 
Android Hacks, Variants, Tricks and Resources ESC SV 2012
Android Hacks, Variants, Tricks and Resources ESC SV 2012Android Hacks, Variants, Tricks and Resources ESC SV 2012
Android Hacks, Variants, Tricks and Resources ESC SV 2012
 
Understanding the Android System Server
Understanding the Android System ServerUnderstanding the Android System Server
Understanding the Android System Server
 
Android Attacks
Android AttacksAndroid Attacks
Android Attacks
 
Linux: the first second
Linux: the first secondLinux: the first second
Linux: the first second
 
Using Erlang in an Embedded and Cross-Compiled World
Using Erlang in an Embedded and Cross-Compiled WorldUsing Erlang in an Embedded and Cross-Compiled World
Using Erlang in an Embedded and Cross-Compiled World
 
Don't Give Credit: Hacking Arcade Machines
Don't Give Credit: Hacking Arcade MachinesDon't Give Credit: Hacking Arcade Machines
Don't Give Credit: Hacking Arcade Machines
 
An introduction to Arduino
An introduction to ArduinoAn introduction to Arduino
An introduction to Arduino
 
Jagan Teki - U-boot from scratch
Jagan Teki - U-boot from scratchJagan Teki - U-boot from scratch
Jagan Teki - U-boot from scratch
 
Valerio Di Giampietro - Introduction To IoT Reverse Engineering with an examp...
Valerio Di Giampietro - Introduction To IoT Reverse Engineering with an examp...Valerio Di Giampietro - Introduction To IoT Reverse Engineering with an examp...
Valerio Di Giampietro - Introduction To IoT Reverse Engineering with an examp...
 
BUD17-104: Scripting Languages in IoT: Challenges and Approaches
BUD17-104: Scripting Languages in IoT: Challenges and ApproachesBUD17-104: Scripting Languages in IoT: Challenges and Approaches
BUD17-104: Scripting Languages in IoT: Challenges and Approaches
 
Build your own embedded linux distributions by yocto project
Build your own embedded linux distributions by yocto projectBuild your own embedded linux distributions by yocto project
Build your own embedded linux distributions by yocto project
 
Inside Android's UI
Inside Android's UIInside Android's UI
Inside Android's UI
 
Michele Dionisio & Pietro Lorefice - Developing and testing a device driver w...
Michele Dionisio & Pietro Lorefice - Developing and testing a device driver w...Michele Dionisio & Pietro Lorefice - Developing and testing a device driver w...
Michele Dionisio & Pietro Lorefice - Developing and testing a device driver w...
 

Viewers also liked

Social Media For Busy Entrepreneurs and Small Businesses
Social Media For Busy Entrepreneurs and Small Businesses Social Media For Busy Entrepreneurs and Small Businesses
Social Media For Busy Entrepreneurs and Small Businesses Fikriyyah George
 
El presidencialismo mexicano antes y después
El presidencialismo mexicano antes y después El presidencialismo mexicano antes y después
El presidencialismo mexicano antes y después espejodeoesed
 
Crisis Subprime en España
Crisis Subprime en EspañaCrisis Subprime en España
Crisis Subprime en Españaespejodeoesed
 
CriminalEFS-PowerPoint
CriminalEFS-PowerPointCriminalEFS-PowerPoint
CriminalEFS-PowerPointJenn Amabile
 
Хипстеры в энтерпрайзе
Хипстеры в энтерпрайзеХипстеры в энтерпрайзе
Хипстеры в энтерпрайзеAleksandr Tarasov
 
Cross-platform game engine development with SDL 2.0
Cross-platform game engine development with SDL 2.0Cross-platform game engine development with SDL 2.0
Cross-platform game engine development with SDL 2.0Leszek Godlewski
 
каталог керасис
каталог керасискаталог керасис
каталог керасисNastasik
 
Service Discovery. Spring Cloud Internals
Service Discovery. Spring Cloud InternalsService Discovery. Spring Cloud Internals
Service Discovery. Spring Cloud InternalsAleksandr Tarasov
 

Viewers also liked (15)

Social Media For Busy Entrepreneurs and Small Businesses
Social Media For Busy Entrepreneurs and Small Businesses Social Media For Busy Entrepreneurs and Small Businesses
Social Media For Busy Entrepreneurs and Small Businesses
 
Imágenes inmersivas
Imágenes inmersivasImágenes inmersivas
Imágenes inmersivas
 
Gamedev-grade debugging
Gamedev-grade debuggingGamedev-grade debugging
Gamedev-grade debugging
 
El presidencialismo mexicano antes y después
El presidencialismo mexicano antes y después El presidencialismo mexicano antes y después
El presidencialismo mexicano antes y después
 
Crisis Subprime en España
Crisis Subprime en EspañaCrisis Subprime en España
Crisis Subprime en España
 
Suir img
Suir imgSuir img
Suir img
 
El barrroco
El barrrocoEl barrroco
El barrroco
 
Ecosistemas
EcosistemasEcosistemas
Ecosistemas
 
CriminalEFS-PowerPoint
CriminalEFS-PowerPointCriminalEFS-PowerPoint
CriminalEFS-PowerPoint
 
Green Peace y WWF
Green Peace y WWFGreen Peace y WWF
Green Peace y WWF
 
Хипстеры в энтерпрайзе
Хипстеры в энтерпрайзеХипстеры в энтерпрайзе
Хипстеры в энтерпрайзе
 
Cross-platform game engine development with SDL 2.0
Cross-platform game engine development with SDL 2.0Cross-platform game engine development with SDL 2.0
Cross-platform game engine development with SDL 2.0
 
каталог керасис
каталог керасискаталог керасис
каталог керасис
 
Docker In Bank Unrated
Docker In Bank UnratedDocker In Bank Unrated
Docker In Bank Unrated
 
Service Discovery. Spring Cloud Internals
Service Discovery. Spring Cloud InternalsService Discovery. Spring Cloud Internals
Service Discovery. Spring Cloud Internals
 

Similar to One Year of Porting - Post-mortem of two Linux/SteamOS launches

UWE Linux Boot Camp 2007: Hacking embedded Linux on the cheap
UWE Linux Boot Camp 2007: Hacking embedded Linux on the cheapUWE Linux Boot Camp 2007: Hacking embedded Linux on the cheap
UWE Linux Boot Camp 2007: Hacking embedded Linux on the cheapedlangley
 
Lightweight Virtualization with Linux Containers and Docker | YaC 2013
Lightweight Virtualization with Linux Containers and Docker | YaC 2013Lightweight Virtualization with Linux Containers and Docker | YaC 2013
Lightweight Virtualization with Linux Containers and Docker | YaC 2013dotCloud
 
Lightweight Virtualization with Linux Containers and Docker I YaC 2013
Lightweight Virtualization with Linux Containers and Docker I YaC 2013Lightweight Virtualization with Linux Containers and Docker I YaC 2013
Lightweight Virtualization with Linux Containers and Docker I YaC 2013Docker, Inc.
 
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo..."Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...Yandex
 
Embedded Erlang, Nerves, and SumoBots
Embedded Erlang, Nerves, and SumoBotsEmbedded Erlang, Nerves, and SumoBots
Embedded Erlang, Nerves, and SumoBotsFrank Hunleth
 
Docker and-containers-for-development-and-deployment-scale12x
Docker and-containers-for-development-and-deployment-scale12xDocker and-containers-for-development-and-deployment-scale12x
Docker and-containers-for-development-and-deployment-scale12xrkr10
 
LMG Lightning Talks - SFO17-205
LMG Lightning Talks - SFO17-205LMG Lightning Talks - SFO17-205
LMG Lightning Talks - SFO17-205Linaro
 
Linux Kernel Platform Development: Challenges and Insights
 Linux Kernel Platform Development: Challenges and Insights Linux Kernel Platform Development: Challenges and Insights
Linux Kernel Platform Development: Challenges and InsightsGlobalLogic Ukraine
 
Tech Days 2015: ARM Programming with GNAT and Ada 2012
Tech Days 2015: ARM Programming with GNAT and Ada 2012Tech Days 2015: ARM Programming with GNAT and Ada 2012
Tech Days 2015: ARM Programming with GNAT and Ada 2012AdaCore
 
Introduction to Docker, December 2014 "Tour de France" Bordeaux Special Edition
Introduction to Docker, December 2014 "Tour de France" Bordeaux Special EditionIntroduction to Docker, December 2014 "Tour de France" Bordeaux Special Edition
Introduction to Docker, December 2014 "Tour de France" Bordeaux Special EditionJérôme Petazzoni
 
Docker Introduction + what is new in 0.9
Docker Introduction + what is new in 0.9 Docker Introduction + what is new in 0.9
Docker Introduction + what is new in 0.9 Jérôme Petazzoni
 
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQDocker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQJérôme Petazzoni
 
The Deck by Phil Polstra GrrCON2012
The Deck by Phil Polstra GrrCON2012The Deck by Phil Polstra GrrCON2012
The Deck by Phil Polstra GrrCON2012Philip Polstra
 
Systemd: the modern Linux init system you will learn to love
Systemd: the modern Linux init system you will learn to loveSystemd: the modern Linux init system you will learn to love
Systemd: the modern Linux init system you will learn to loveAlison Chaiken
 
My "Perfect" Toolchain Setup for Grails Projects
My "Perfect" Toolchain Setup for Grails ProjectsMy "Perfect" Toolchain Setup for Grails Projects
My "Perfect" Toolchain Setup for Grails ProjectsGR8Conf
 
Let's Containerize New York with Docker!
Let's Containerize New York with Docker!Let's Containerize New York with Docker!
Let's Containerize New York with Docker!Jérôme Petazzoni
 
Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)Jérôme Petazzoni
 
Let's trace Linux Lernel with KGDB @ COSCUP 2021
Let's trace Linux Lernel with KGDB @ COSCUP 2021Let's trace Linux Lernel with KGDB @ COSCUP 2021
Let's trace Linux Lernel with KGDB @ COSCUP 2021Jian-Hong Pan
 
Introduction to Docker at SF Peninsula Software Development Meetup @Guidewire
Introduction to Docker at SF Peninsula Software Development Meetup @GuidewireIntroduction to Docker at SF Peninsula Software Development Meetup @Guidewire
Introduction to Docker at SF Peninsula Software Development Meetup @GuidewiredotCloud
 
High-Performance Computing with C++
High-Performance Computing with C++High-Performance Computing with C++
High-Performance Computing with C++JetBrains
 

Similar to One Year of Porting - Post-mortem of two Linux/SteamOS launches (20)

UWE Linux Boot Camp 2007: Hacking embedded Linux on the cheap
UWE Linux Boot Camp 2007: Hacking embedded Linux on the cheapUWE Linux Boot Camp 2007: Hacking embedded Linux on the cheap
UWE Linux Boot Camp 2007: Hacking embedded Linux on the cheap
 
Lightweight Virtualization with Linux Containers and Docker | YaC 2013
Lightweight Virtualization with Linux Containers and Docker | YaC 2013Lightweight Virtualization with Linux Containers and Docker | YaC 2013
Lightweight Virtualization with Linux Containers and Docker | YaC 2013
 
Lightweight Virtualization with Linux Containers and Docker I YaC 2013
Lightweight Virtualization with Linux Containers and Docker I YaC 2013Lightweight Virtualization with Linux Containers and Docker I YaC 2013
Lightweight Virtualization with Linux Containers and Docker I YaC 2013
 
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo..."Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...
 
Embedded Erlang, Nerves, and SumoBots
Embedded Erlang, Nerves, and SumoBotsEmbedded Erlang, Nerves, and SumoBots
Embedded Erlang, Nerves, and SumoBots
 
Docker and-containers-for-development-and-deployment-scale12x
Docker and-containers-for-development-and-deployment-scale12xDocker and-containers-for-development-and-deployment-scale12x
Docker and-containers-for-development-and-deployment-scale12x
 
LMG Lightning Talks - SFO17-205
LMG Lightning Talks - SFO17-205LMG Lightning Talks - SFO17-205
LMG Lightning Talks - SFO17-205
 
Linux Kernel Platform Development: Challenges and Insights
 Linux Kernel Platform Development: Challenges and Insights Linux Kernel Platform Development: Challenges and Insights
Linux Kernel Platform Development: Challenges and Insights
 
Tech Days 2015: ARM Programming with GNAT and Ada 2012
Tech Days 2015: ARM Programming with GNAT and Ada 2012Tech Days 2015: ARM Programming with GNAT and Ada 2012
Tech Days 2015: ARM Programming with GNAT and Ada 2012
 
Introduction to Docker, December 2014 "Tour de France" Bordeaux Special Edition
Introduction to Docker, December 2014 "Tour de France" Bordeaux Special EditionIntroduction to Docker, December 2014 "Tour de France" Bordeaux Special Edition
Introduction to Docker, December 2014 "Tour de France" Bordeaux Special Edition
 
Docker Introduction + what is new in 0.9
Docker Introduction + what is new in 0.9 Docker Introduction + what is new in 0.9
Docker Introduction + what is new in 0.9
 
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQDocker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
 
The Deck by Phil Polstra GrrCON2012
The Deck by Phil Polstra GrrCON2012The Deck by Phil Polstra GrrCON2012
The Deck by Phil Polstra GrrCON2012
 
Systemd: the modern Linux init system you will learn to love
Systemd: the modern Linux init system you will learn to loveSystemd: the modern Linux init system you will learn to love
Systemd: the modern Linux init system you will learn to love
 
My "Perfect" Toolchain Setup for Grails Projects
My "Perfect" Toolchain Setup for Grails ProjectsMy "Perfect" Toolchain Setup for Grails Projects
My "Perfect" Toolchain Setup for Grails Projects
 
Let's Containerize New York with Docker!
Let's Containerize New York with Docker!Let's Containerize New York with Docker!
Let's Containerize New York with Docker!
 
Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)
 
Let's trace Linux Lernel with KGDB @ COSCUP 2021
Let's trace Linux Lernel with KGDB @ COSCUP 2021Let's trace Linux Lernel with KGDB @ COSCUP 2021
Let's trace Linux Lernel with KGDB @ COSCUP 2021
 
Introduction to Docker at SF Peninsula Software Development Meetup @Guidewire
Introduction to Docker at SF Peninsula Software Development Meetup @GuidewireIntroduction to Docker at SF Peninsula Software Development Meetup @Guidewire
Introduction to Docker at SF Peninsula Software Development Meetup @Guidewire
 
High-Performance Computing with C++
High-Performance Computing with C++High-Performance Computing with C++
High-Performance Computing with C++
 

Recently uploaded

The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...kalichargn70th171
 
Introduction to Firebase Workshop Slides
Introduction to Firebase Workshop SlidesIntroduction to Firebase Workshop Slides
Introduction to Firebase Workshop Slidesvaideheekore1
 
Osi security architecture in network.pptx
Osi security architecture in network.pptxOsi security architecture in network.pptx
Osi security architecture in network.pptxVinzoCenzo
 
Strategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsStrategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsJean Silva
 
2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shardsChristopher Curtin
 
What’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesWhat’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesVictoriaMetrics
 
eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolsosttopstonverter
 
Advantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxAdvantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxRTS corp
 
Effort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsEffort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsDEEPRAJ PATHAK
 
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxUnderstanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxSasikiranMarri
 
Ronisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited CatalogueRonisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited Catalogueitservices996
 
Effectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryErrorEffectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryErrorTier1 app
 
Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldRoberto Pérez Alcolea
 
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingOpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingShane Coughlan
 
Mastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxMastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxAS Design & AST.
 
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdfEnhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdfRTS corp
 
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...OnePlan Solutions
 
Best Angular 17 Classroom & Online training - Naresh IT
Best Angular 17 Classroom & Online training - Naresh ITBest Angular 17 Classroom & Online training - Naresh IT
Best Angular 17 Classroom & Online training - Naresh ITmanoharjgpsolutions
 
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics
 
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdfAndrey Devyatkin
 

Recently uploaded (20)

The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
 
Introduction to Firebase Workshop Slides
Introduction to Firebase Workshop SlidesIntroduction to Firebase Workshop Slides
Introduction to Firebase Workshop Slides
 
Osi security architecture in network.pptx
Osi security architecture in network.pptxOsi security architecture in network.pptx
Osi security architecture in network.pptx
 
Strategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsStrategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero results
 
2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards
 
What’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesWhat’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 Updates
 
eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration tools
 
Advantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxAdvantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptx
 
Effort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsEffort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software Projects
 
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxUnderstanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
 
Ronisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited CatalogueRonisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited Catalogue
 
Effectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryErrorEffectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryError
 
Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository world
 
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingOpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
 
Mastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxMastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptx
 
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdfEnhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
 
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
 
Best Angular 17 Classroom & Online training - Naresh IT
Best Angular 17 Classroom & Online training - Naresh ITBest Angular 17 Classroom & Online training - Naresh IT
Best Angular 17 Classroom & Online training - Naresh IT
 
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
 
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
 

One Year of Porting - Post-mortem of two Linux/SteamOS launches

  • 1. One Year of Porting Post-mortem of two Linux/SteamOS launches Leszek Godlewski
  • 2. Who is this guy? Leszek Godlewski Programmer, Nordic Games (early 2014 – now) – Unannounced project Freelance Programmer (Sep 2013 – early 2014) – Linux port of Painkiller Hell & Damnation – Linux port of Deadfall Adventures Generalist Programmer, The Farm 51 (Mar 2010 – Aug 2013) – Painkiller Hell & Damnation, Deadfall Adventures
  • 3. Focus ● Not sales figures ● Not business viability ● Not game-specific bugs ● Not the Steam Controller – oops! � ● Platform-specific problems ● Mistakes made & mitigation attempts
  • 4. Agenda ● The ports ● Laying down the foundations – Build system – Compilers – Linking – Boilerplate ● Release and feedback – User issues – Crash handling – GLSL shader linking
  • 6. Painkiller Hell & Damnation (The Farm 51)
  • 8. Facts ● Unreal Engine 3 ● All major Linux distros – SteamOS, Debian, Ubuntu, Fedora, Arch, Gentoo ● All official drivers – NVIDIA, AMD, Intel (i965) ● Some open-source drivers – Gallium r600, Gallium radeonsi
  • 9. Facts ● Most of UE3 middlewares have Linux versions – In our case: PhysX, FaceFX, Scaleform Gfx, lzopro, Bink... ● Introduced open-source middlewares – SDL 2.x, GLEW, Steam Runtime ● UE3's build system – Unreal Build Tool – Handles everything make does – Written in C#, fixed up to run in Mono on Linux ● UE3's content packaging (cooking) system – Linux target based on Mac OSX
  • 10. Facts ● QA department unfamiliar with Linux – Basic training was required ● Installing & running software (including from the command line), file permissions, driver installation, gathering system information... – Mostly reported false positives in the beginning ● Spare time project over ~13 months – After leaving The Farm 51 employment – contracted for further outsourcing directly by TF51 – Occasional support from individual members of TF51 staf ● Kudos to Piotr Bąk and Wojciech Knopf!
  • 11. Overlap ● Noticed how a lot of work was based on OSX code? ● Happens all the time – POSIX – OpenGL/OpenAL MacOS X Linux Mobile
  • 12. Agenda ● The ports ● Laying down the foundations – Build system – Compilers – Linking – Boilerplate ● Release and feedback – User issues – Crash handling – GLSL shader linking
  • 13. Laying down the foundations
  • 14. Starting point ● Epic's OpenGL 2.1 and OpenAL back-ends – OpenGL mode somewhat functional in Windows developer builds ● Epic's Mac OSX port – Limited test builds for Mac OSX had been made before – Mac OSX binary builds supported via remote compiltion – Existing Mac OSX target for game content packaging (cooking) ● Both of the above – somewhat... unfinished � ● On Windows, the games shipped 32-bit binaries only
  • 15. Building the build tool – C# & Mono ● Patched the Unreal Build Tool to build & run on Mono in Linux – Mono can handle most .NET commandline apps all right ● Added support for Linux toolchains (duh) ● Fixed hardcoding of backslashes in paths – Path.Join() instead ● Fixed regexes on large strings (C++ sources) blowing up the stack – Break up the string into smaller parts
  • 16. Cross-compiling for 32/64-bit ● Yes, I agree, 32-bit should die, but one may not be allowed to kill it ● gcc -m32/-m64 is not enough! – Sets target code generation – But not headers & libraries (CRT, OpenMP, libgcc etc.) ● Fixed (on Debian & friends) by installing gcc-multilib – Dependency package for non-default architectures (i.e. i386 on an amd64 system and vice versa)
  • 17. Clang ● Clang is faster – gcc: 3m47s – Clang: 3m05s ● Clang has diferent diagnostics than gcc ● Clang has C++ preprocessor macro compatibility with gcc – Declares __GNUC__ etc. ● Clang has commandline compatibility with gcc – Can easily switch back & forth between gcc and Clang
  • 18. Clang - caveats ● Object files may be incompatible with gcc & fail to link (need full rebuilds) ● gcc is more mature than Clang – Clang has generated faulty code for me (YMMV) ● Slight inconsistencies in C++ standard strictness – Templates – Anonymous structs/unions – May need to add this-> in some places – May need to name some anonymous types
  • 19. So – Clang or gcc? Both: ● Clang – quick iterations during development ● gcc – final shipping binaries
  • 20. Linking – GNU ld ● Default linker on Linux ● Ancient ● Single-threaded ● Requires specification of libraries in the order of reverse dependency... ● We are not doomed to use it!
  • 21. Linking – GNU gold ● Multi-threaded linker for ELF binaries – ld: 18s – gold: 5s ● Developed at Google, now officially part of GNU binutils ● Drop-in replacement for ld – May need an additional parameter or toolchain setup ● clang++ -B/usr/lib/gold-ld ... ● g++ -fuse-ld=gold ... ● Still needs libs in the order of reverse dependency...
  • 22. Linking – library groups ● Major headache/game-breaker with circular dependencies – ”Proper” fix: re-specify the same libraries over and over again ● Declare library groups instead – Wrap library list with --start-group –end-group ● Shorthand: -(, -) ● g++ foo.obj -Wl,-( -lA -lB -Wl,-) ● Caveat: results in exhaustive symbol search within the group – Manual warns of possible performance hit – Not observed here, but keep that in mind!
  • 23. Caching the gdb-index ● Large codebase generates heavy debug symbols (hundreds of megabytes) ● gdb generates the index for quick symbol lookup... ● ...at every single gdb startup � – Takes several minutes for said codebases – Massive waste of time! ● Solution: cache the index, fold it into the build process! – Full description in the gdb manual – gdb -batch -ex "save gdb-index $(OUTPUT_PATH)/gdb-index" $ (BINARY) – objcopy --add-section .gdb_index=$(OUTPUT_PATH)/gdb-index/$ (BINARY).gdb-index --set-section-flags .gdb_index=readonly $ (BINARY) $(BINARY)
  • 24. Raw X11 or SDL? ● Initially tried rolling my own boilerplate – Basic X11 mouse, window and key press events are easy – Unicode text input is not – Useful windowing is not – Correct GLX is not – Linux joystick API is not – Above all, X11 seems to be on its way out ● Wayland & Mir will have emulation layers, but that's bound to have overhead ● You really want to use SDL 2 instead, trust me – Shameless plug: see my talk from WGK 2013 for benefits of using SDL 2 ☺
  • 25. Agenda ● The ports ● Laying down the foundations – Build system – Compilers – Linking – Boilerplate ● Release and feedback – User issues – Crash handling – GLSL shader linking
  • 27. What we shipped initially with the beta ● 32-bit binaries (64-bit added later on) ● Launch script (~20 lines) – Architecture detection ● Initially a stub for 64-bit with fallback to 32-bit – Steam Runtime injection (if not already present) ● That's about it ☺ ● Explicit dependency on the Steam Runtime – Allows shifting some responsibility to Valve – And, admittedly, to users who insist on using their own dependencies
  • 28. User issues ● Missing/incompatbile libraries – Resulting from disabling the Steam Runtime ● Gentoo users, mostly... Maintainer of steam package had chosen to disable it by default – Usually fixed by force-starting Steam with STEAM_RUNTIME=1 ● $ STEAM_RUNTIME=1 steam ● ”Missing” 32-bit NVIDIA OpenGL libraries on 64-bit systems – Apparently, they might end up unreachable by the dynamic linker – Fixed by adding /usr/lib32 to LD_LIBRARY_PATH in the launch script – Also, prompt user to make sure they did install them ● It's an option - ”install compatibility 32-bit libraries”
  • 29. User issues ● No support for DXT texture compression despite capable hardware (GL_EXT_texture_compression_s3tc) – Concerns the open-source drivers – For legal reasons (S3/VIA patents), some distros don't ship it or install it automatically ● E.g. Fedora – If extension not advertised by driver, suggest the user to install libtxc_dxtn ● Often a distro package, so no hassle
  • 30. More user issues... ● Graphical glitches... ● Broken V-sync... ● Broken NVIDIA Optimus with open-source multiplexer... ● Looong & unresponsive loading times... ● A whole lot of crashes... ● Most of the above was my fault – not going to bore you with all of this!
  • 31. Crash handling ● Unix signals – Asynchronous IPC notification mechanism in POSIX-compliant systems ● Sources can be the process itself, other processes or the kernel – Default handler terminates process & dumps core for most signals – Can (must?!) specify custom handlers ● Get/set handlers via the sigaction(2) system call – Handler prototype: void sa_handler(int signal, siginfo_t *siginfo, void *context); ● More information – G. Ben-Yossef, Crash N' Burn: Writing Linux application fault handlers
  • 32. Interesting siginfo_t fields ● si_errno – errno value – Possibly more detailed error code ● si_code – reason for sending the signal – Both general and per signal type – Examples: issued by user, issued by kernel, illegal addressing mode, FP over/underflow, invalid memory permissions, unmapped address etc. ● si_addr – memory location at which fault happened – If applicable: SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGTRAP
  • 33. Signal handler caveats ● Not safe to allocate or free heap memory! – Fault may have corrupted the allocator's data structures ● Prone to race conditions – Can't share locks with the main program! ● If signalled after locking, you'll deadlock – Can't call async-unsafe functions! ● See manual for signal(7) for a list of safe ones ● Custom handlers do not dump core (a.k.a. minidump) – Mitigated by restoring default handler after custom logging and re-signalling self ● signal(signum, SIG_DFL); raise(signum);
  • 34. Safe stack walking ● glibc provides backtrace() and friends ● Symbols are read from the dynamic symbol table – Must pass -rdynamic to gcc/Clang to populate ● Calling backtrace_symbols() allocates heap memory – Not safe... ☹ – Still, can get away with it most of the time – Proper solution involves a separate watchdog process & pipes (heap-less backtrace_symbols_fd() call instead)
  • 35. Long load times? Unresponsiveness? ● Profiling quickly places blame on shader linking – OpenGL shader model operates on program objects, created by linking shader pipeline combinations ● Introduces lots of redundancy (see glGetProgramiv() & glGetShaderiv()) ● Drivers often defer actual compilation until ”link time” ● Increased memory consumption – UE3 OpenGL renderer blocks the render thread for linking ● Render thread blocked → Frozen loading screen! – Both games have thousands of shaders ● An awful lot of vertex/fragment shader combinations (programs) � ● Moreover – makes async level streaming blocking! – Bad stuttering during gameplay ● Situation better on subsequent loads on NVIDIA due to in-driver cache
  • 36. Shader linking ● Short-term fix: background shader linking – Worker thread with a separate OpenGL context, sharing data with the main one – Queue all shader link jobs, execute on the worker only – If on a loading screen, keep spinning it while waiting for the shaders – Defer ”async streaming done” notifications till shader link queue is empty ● Pros: – Quick & easy to implement – Fixes gameplay stuttering ● Cons: – Only fixes unresponsiveness, not the long load times ☹
  • 37. Shader linking ● Disaster on the official AMD Catalyst driver! – Total system hang (PC needs hard reset) – Apparently, exposed a race condition in AMD driver – AMD has yet to ship the fix... ● Fallback to old, blocking code path if Catalyst detected
  • 38. Shader linking ● Possible improvement (suggested by Epic): ARB_separate_shader_objects – Replaces programs (and linking) with much lighter pipeline objects ● Removes a lot of redundancy – Makes use of separate vertex/fragment shaders (D3D-like) – Would play well with UE3's RHI, modelled mostly after D3D ● Not implemented ☹ – requires shader syntax upgrade and a refactor of UE3's OpenGL renderer – Explicit locations for attributes and varyings required for SSO – Need to bump GLSL from 1.20 (OpenGL 2.1) to at least 1.40 (OpenGL 3.1)
  • 39. Shader linking ● Proper fix: deferred shader access – Modern drivers queue shader compiles and links internally and process them in a multithreaded manner ● Official NVIDIA & AMD Catalyst ● Open-source Mesa drivers in SteamOS (patches pushed upstream recently by Valve) – Kick all the jobs (i.e. create shader objects) at level load – Do not access the objects (query, draw) until they are needed – Not even the compile/link status! This creates a sync point! ● Not implemented ☹ – requires a considerable refactor of UE3's OpenGL renderer
  • 41. Takeaway 1/2 ● Porting .NET-based tools to Linux is viable ● Many 32/64-bit cross-compile issues are solved with gcc-multilib ● Switching back and forth between Clang and gcc is easy and useful ● Link times can be greatly improved by using gold ● Caching the gdb-index improves debugging experience ● Using SDL 2 is way better than rolling your own boilerplate
  • 42. Takeaway 2/2 ● Using the Steam Runtime is good for you ● Crash handling in Linux is easy to do, tricky to get right ● OpenGL shader model is significantly diferent from D3D's ● GLSL linking is slow, so defer access if possible ● Multiple concurrent OpenGL contexts can still bite you ● Test on different GPU drivers to avoid unpleasant surprises!
  • 43. @ l go d l ews k i @ n o rd i c ga m e s . at t @ T h e I n e Q u ati o n K w w w. i n e q u ati o n . o rg Questions?
  • 44. F u rt h e r N o rd i c G a m e s i nfo rm ati o n : K w ww. n o rd i c ga m e s . at Deve l o p me nt i nfo rmati o n : K ww w. gr i m l o re ga m e s . co m Thank you!