Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.



Published on


By Renato Golin, Linaro

Published in: Technology
  • Be the first to comment


  1. 1. Oopstreaming Tales from upstreaming experiences Renato Golin
  2. 2. Agenda ● Upstreaming 101 ○ For those new to the trade ● Successful stories ○ Happy moments ● Unsuccessful stories ○ Most of them, honestly ● Tragic stories ○ I wish I could forget them...
  3. 3. Upstreaming 101
  4. 4. Upstreaming 101 ● Every community is unique ○ Origin: company offshoot, grew from another community, standards implementation ○ Focus: solve a particular problem, fit a specific role in a larger community ○ Steering: ruler/amorphous, meritocratic, foundation, companies (and which companies!) ○ People: ultimately, the engineers are the ones making things happen ● Their place in the larger OSS arena matters ○ Linux is used by a very large of different communities, open, closed and anything in between ○ FreeBSD has similarly wide adoption, but with completely different patterns ○ Not many companies have their own GCC releases, not many use upstream LLVM releases ● License matters ○ Strong/permissive licenses create very different communities ○ Different kinds of freedom, different kinds of zealots ○ Patent issues affect both sides and is the most controversial issue for companies
  5. 5. Upstreaming 101 ● Open Source is a long process ○ Winning individual battles are important, but not enough to make people care ○ Maintainers will not accept your short term benefit for their long term loss ○ Most of closed source development rely on that premise one way or another ● Modus Operandi ○ Maintainers have to adopt the project’s stance, or risk forks, irrelevance, removal ○ Most of them are paid by some company, but have their own public personas ○ It’s hard to tell how much is one or the other (most people rely on appearances only) ● Steering is a multi-layered process ○ Committees and foundations are the easy ways, but only worth on bigger projects ○ Direct influences: code, infrastructure, reviews, support ○ Indirect influences: closed door meetings, regional agreements, propaganda ○ Perverse influences: fast commit rate, delaying tactics, late night changes ○ Evil influences: paying people to be obnoxious (or hiring obnoxious people)
  6. 6. Upstreaming 101 ● Open Source is not Woodstock ○ Peace and love helps making better people, not solve engineering problems (we need both) ● Open Source is not Robin Hood ○ Taking from the rich (companies) and giving to the poor (who?) ○ Most engineers work at companies, if one fails, so does the other ● Finding the best long term solution to all engineering problems ○ Freedoms are required for re-use, both in space and time! ○ Forced collaboration makes companies understand the long term goals ○ Licenses make legal teams afraid of the big bad wolf ○ And in turn, push companies to behave and stop thinking short term It almost looks like a world wide conspiracy, driven by engineers, to get their job done in the way they know it’s right, while avoiding the unwinnable battle of the short term profit.
  7. 7. Successful stories
  8. 8. Successful stories ● EDG on LLVM ○ While successful internally to Arm, and to LLVM, it was a complete disaster for EDG ○ Two years later, Sony described how that changed their whole product to use Clang instead ○ Now, both Arm and Sony are using Clang as their front-ends, collaborating in the open ● LLVM Vectoriser ○ Mostly Nadav/Arnold from scratch, I just made sure it also worked on ARM ○ Arnold completed the whole cost-model for x86/Arm not much later ○ The original model was tailored to Apple’s design, which prompted adaptation to other cores ● Stride vectorisation ○ Initial implementation interrupted due to lack of support in vectoriser ○ Attempts to bootstrap infrastructure created basic infrastructure for OpenMP simd pragma ○ Years later, Arm wrote internal implementation that had serious issues ○ Draft was used later by Arm to do the actual implementation, from scratch
  9. 9. Successful stories ● LLVMLinux ○ Basic effort from another community: LLVMLinux ○ Linaro was the point-of-contact for LLVM changes, especially around Arm cores ○ Extended such contract to Android, Chromium, helped move Android to LLVM ○ Helped fix dozens of bugs in user code, kernel code, compiler code ● RISC-V / BPF Backends ○ Initial proposers enquired how to do it and what they had before proposing ○ They quickly acted on every request by the community, constantly changing large patch-sets ○ Guaranteed maintainership of their code for the foreseeable future ● Warewulf on Arm ○ PXELinux didn’t have AArch64 support, Arm clusters had to use a Grub PXE trick ○ We wanted identical experience on x86_64 and AArch64, to ease adoption ○ Arm and Linaro worked with OpenHPC and Warewulf communities to find a better solution ○ Turns out iPXE works identically on both architectures, we validated, now it’s upstream
  10. 10. Unsuccessful stories
  11. 11. Unsuccessful stories ● Multi-arch Clang driver ○ Clang supports multiple back-ends at the same time, unlike GCC ○ But it still has to emulate GCC, and pretend to behave like it on different systems ○ The driver has to understand all paths, multilib/multiarch, embedded/desktop/server, etc. ○ Also cross-compile run-time libraries (RT, libunwind, libc++abi) for all targets ○ Large problem space, fixes were always too narrow in scope ○ Most fixes generated more bugs than they fixed, many of them mine ● Target Parser ○ CPU/FPU names were duplicated in the back-end (C++ code), Target description (TableGen) ○ As well as every possible front-end (Clang, Flang) and development tools (lli, llc, etc) ○ Linaro worked to common up all strings into a single TargetParser ○ End goal was to generate the strings from TableGen, so that we had a single source ○ TableGen turned out too complex to refactor (without breaking the rest of the back end) ○ Class design was also too slow and no one other than Arm used it, so ultimately, unnecessary
  12. 12. Unsuccessful stories ● AAP Backend ○ Not an actual back-end, but a meta backend to build deeply embedded backends ○ They have taken all steps to make it the perfect submission, exactly like RISC-V ○ They have responded to every question, rebased the patches on every change ○ Provided documentation, presentations in conferences, offered code ownership ○ But there was not enough support by the other members of the community to be accepted ● GCC & LLVM collaboration ○ Linaro proposed a stronger collaboration between GCC and LLVM at the design level ○ Some people supported the idea, which is why we proposed it in the first place ○ But there were two crucial areas of disagreement, which made that all moot ■ Hubris on both sides, believing their design choices were better informed ■ Different speed for different things: quick-fix for one was a slow fix for another ○ We also had bigotry from both sides, albeit only from a minority group ■ On its own, probably wouldn’t be enough to moot it, but together with the other two problems...
  13. 13. Unsuccessful stories ● Initial SVE upstreaming ○ First impressions are important, the most common rookie mistake is to dump code in GitHub ○ Arm hasn’t exactly done that, but they referred to the GitHub repo and have not had a patch-set ■ It looks exactly the same to seasoned open sourcers ○ Opted for sending core changes first, to avoid time waste while rebasing backend patches ○ Community won’t accept core changes without factual reasons, implemented in code ■ Ended up wasting more time than if had been rebasing changes from start ■ And ultimately will have to follow through the plan anyway, wasting the same time + initial wait ○ Companies need to understand that upstreaming is a long and painful process ○ But that it will also save a huge cost (downstream) and have benefits (upstream changes) ■ Hard to argue this with companies that need to sell tools ○ Luckily, Arm has changed course and they have made considerable progress this last quarter
  14. 14. Tragic stories
  15. 15. ● EHABI ○ Back in 2009, LLVM didn’t support Arm’s exception handling ABI ○ We wrote a quick hack and tried to upstream, but were blocked on already existing work ○ When the work shown up 6 months later, it was the same we already had (that didn’t work) ○ It took 2 years to get something working, with a lot of help from Code Aurora ○ Turns out it was company pressure to keep downstream costs low (community manipulation) ● New Pass Manager ○ LLVM’s pass manager has a number of deficiencies and a big effort was created to fix them ○ Years in preparation culminated in the complete rewrite and move, but still using the old one ○ Many years later, we’re still using the old pass manager ■ Too many entangled problems, too hard to benchmark, harder to fix problems without creating new ones ■ People left the project, others were intermittent: too big a task for any small group of people ○ Current efforts being made again (lots of tests, benchmarking) look very promising... Tragic stories
  16. 16. Tragic stories ● LLD Arm backends ○ By 2013, MediaTek had made the MCLinker work on ARM hardware ○ But that wasn’t an approved project by the upstream LLVM community ○ So Linaro vowed to work on LLD for both Arm and AArch64, the latter coming first (easiest) ○ But after 3 months work and bootstrap working, some developers wiped the whole backend ○ One year later, we picked up the new backend and implemented it again for AArch64 ○ Thought, the Arm backend is in review for almost one year already... ● GitHub move ○ LLVM still uses SVN, with an official Git repo + multiple unofficial GitHub repos ○ We wanted to move to Git, and to GitHub while at it ○ But no one could agree on how we join the projects (llvm, clang, rt, libc++) ○ We tried a public process, but downstream companies couldn’t agree ○ So the process was hijacked and is being done downstream ○ We’ll hear of their decision and will have no choice but to accept whatever comes...
  17. 17. Tragic stories ● Flang ○ There was a Fortran front end before, as a GSOC project to compile two benchmarks ○ It followed all LLVM’s developers’ policies, code standards, etc. but couldn’t compile much ○ PGI had a front end and decided to move it to LLVM, and upstream while they were at it ○ For multiple years they have promised to upstream and failed to deliver ○ One day we received a GitHub dump… Needless to say no one was impressed ○ Too many problems to fix ■ Code is controlled by a single proprietary tool from a single company (PGI) ■ Needs changes to Clang/LLVM which are not upstream ■ Don’t follow either developer policies nor coding standards ■ Doesn’t use the LLVM API ■ Forks three times, uses temporary text files as communication between the stages ■ No one has tried to upstream a single patch, so far ○ Both Arm and PGI have promised to fix that, but without public efforts, people are sceptical ■ Given current track record, it’s hardly surprising...
  18. 18. Conclusions
  19. 19. Conclusions ● Communities are different and often follow chaotic patterns ○ Understand the community before you interact with it (or ask someone who does) ○ Don’t try to change or challenge the community’s status quo (you will get burned) ● Open source is a mix of public, personal and corporate interests ○ For both individuals and companies, not all of them clear or honest ○ Unique to each and every project, regardless of who participates in it ● There is no silver bullet ○ Same behaviour, same person, different topic can have drastically different outcomes ○ Failure can come in many ways, often unpredictable, learn to cope with it is the best strategy ● Count on failure, embrace the downstream cost ○ When using / contributing to OSS projects, make sure your project will cope with delays/rewrites ○ It is your job to adapt your project to upstream, not the other way round ○ Success in upstreaming should be largely orthogonal to internal project roadmaps
  20. 20. Thank You