White Paper: High-Performance Coding, Building and Testing for Multiple Platforms and Devices
High-Perform ance Coding, Building & Testing for Multiple Platform sand DevicesJethro Villegasjet@adobe.comOverviewToday’s software development landscape requires that all the associated costs yield the maximum benefitfor large user bases across increasingly divergent hardware and software platforms. In real terms, can thevalue of a single line of HTML markup be weighed against a single line of ARM5 assembly language?This talk aims to show how careful management of a large code base that targets multiple platforms can beused to lower the cost of generic cross-platform core code, and maximize the return on investment onexpensive but competitively advantageous platform-specific source code. The real-world data set for thistalk will be the Adobe Flash Runtime products. This talk will describe the problems we had to solve, thefactors that influenced our decisions, the solutions we came up with, what worked, what didnt, and ourplans for the future.Brief HistoryThe Flash Runtime is actually several different products and technologies that evolved over 12 years. TheFlash Player, Adobe Integrated Runtime (AIR), and Flash Lite (Mobile) products were built on severalcodelines and were becoming very hard to manage. An enormous amount of effort was being spent portingfeatures and fixes from one product to the other. Our ship schedules were in disarray, and it was verydifficult to predict when a product would be ready to ship. In 2008, a goal was set to consolidate thecodebases into one common trunk now known as FlashRuntime/Main."Flash Player 10 on all devices" was the mission. The promise was to provide the complete Internet toevery supported device. At the time, our mobile binaries were built on an older code base that did notinclude the latest Flash Player features. By the time we finally shipped Flash Player 10.1 on the desktopand mobile platforms, we actually made more code changes than when we went from Flash Player 9 to 10.In 2008, the mobile computing landscape was in a confusing state. It was unclear which devices would becapable of supporting the full Flash Player 10 experience. The availability of the 1GHz QualcommSnapdragon chip later that year made it viable to support plans to bring our mobile codebase up to the samelevel as the desktop.The Mobile & Desktop Flash Player and AIR products are now built from a single highly portable unifiedcode base, separated into platform and core components. The same code base produces hundreds ofdifferent binaries on dozens of software and hardware platforms, from low-power phones to high-end 64-bit workstations. Every time a developer checks in to the shared mainline trunk, her code must build, run,and pass quality tests on hundreds of different devices. She must write code that passes a rigorous codereview process, and adhere to an architectural model that allows the core code to evolve while minimizingdivergent behavior across the different runtime environments. • All features are supported on the main desktop platforms, at parity. • All features are supported in software. Hardware is optionally used for acceleration, but not differing functionality. • All code compiles as straight C++, without a requirement for assembly. (Although we certainly optimize using machine code.) • The SDK and mobile player supports full web browsing, although at a loss of fidelity where appropriate, but not in a way that breaks functional compatibility.
Careful consideration was made for the planned proliferation of mobile platforms. The entire source tree isdivided into 3 directories: • code The code folder contains all the source code to build all the products. This directory is further divided into several module areas, under 3 major categories: 1. core – this directory contains code that is common to any Flash product on any platform. For example, the parsing of the SWF file format is in this directory. Most of the Flash Player is core code. 2. platform – this directory contains the platform-specific code for each Flash product. For example, the code to rasterize specifically on DirectX on Windows could be found here. 3. third_party – this directory contains code we license and use from other organizations. Video and audio codecs are typically found here. • tools The tools directory contains utilities and other non-code components of the Flash Runtime products. This folder is under version control to ensure that the non-code components are in lock=step with the matching code folder. • buildconfig This directory holds the configuration data for the build system. This folder is also under source control so that we can build and ship from any code line with the same build system.Given the diverse build and deployment configurations that Flash needs to run on, the developer doesn’ttypically have every type of device or development environment available. Writing code for dozens ofproducts requires a system that can validate code changes across all the supported configurations. Thesystem shouldnt require the developer to have every supported build environment and toolchain availablelocally. The system also has to build and test the new code in a limited amount of time.
A sandbox build and test system is used to validate her code before the check-in into the mainline trunk.She checks her code into her branch and points the sandbox build system at her code, tools, and buildconfigdirectories. This gives her maximum flexibility and reduces the need to branch all 3 main directories, as shecan use her personal code branch, while using production tools and buildconfig directories.She authenticates with her LDAP username and password, then supplies the Perforce info needed by thebuild system to pull her code into each instance of a builder that will compile, link, and test.She can then select all the targets that her code will be built and tested against.
The developer specifies the Perforce path and changelist from which to build her code. She can also specifythe tools used for the build, and the parent branch used to run tests against her code. Because Flash isalways shipping, we typically have 3 to 6 production branches in addition to the mainline. Each productionbranch may have different features or fixes, and the tests to run need to account for that. After making herselections and choosing ‘Start Build’, dozens of computers spin up to either build or test her code. After 1to 2 hours, she can check the results of her build on the status page.
Other developers keep their own sandbox branches and often share branches with each other to conservecomputing resources and minimize integration time.Nightly, Continuous, Sandbox Build ConfigurationsThe Flash Runtime team supports 3 levels of configuration:Sandbox. The sandbox configuration was described above, and is typically run on new, untested, orexperimental code.Continuous. Our production branches are continuously built and tested 24 hours a day to validate codechecked in. These builds are synced incrementally and built about once an hour.
Nightly. A scheduled script increments the version number in Perforce prior to building the officialproduction builds. This occurs once a day on a typical release. Once a build becomes a Release Candidate(RC) the nightly builds stop and official builds are built on demand.Tracking quality and performance metricsAll new code has to be run against a set of tests that are appropriate for the type of builds that are running.A sandbox or continuous build is tested against a quick set of automated tests that exercise corefunctionality in under 10 minutes. An official nightly build is tested against an automated battery of teststhat run in about 30 minutes. A release candidate is typically tested with automated and manual tests thatrun between 48 and 72 hours.As new devices are added to the test matrix, the system has to be made aware of their availability andcapabilities. Even as some devices may lack certain features, a core set of capabilities need to be validated
in an automated manner. Graphics, audio, and video tests are difficult to automate and some tests have tobe done manually.References:Lee Thomason & Jim Corbett – Flash Player Architecture InfoLeah Zagreus & Alvin Luison – Flash Player Testing Info