LAS16-101: Efficient kernel backporting Speakers: Alex Shi Date: September 26, 2016 ★ Session Description ★ In computer/mobile product world, due to the stability, project timeline, etc considerations, latest upstream kernel isn’t their preference. The long term stable kernel is. But if you want to some of the latest features which only is in upstream kernel,you will have to backport them to old stable kernel. This presentation will share the kernel feature backport experience with audience, help them understand how to do backports quickly and effectively without detailed knowledge of the target feature, thus giving more flexibility and Improving productivity when making products. We will use some examples, to discuss how to get info from backport request, how to find necessary commits, how to get dependency, how to resolve conflicts, and finally how to test it. ★ Resources ★ Etherpad: pad.linaro.org/p/las16-101 Presentations & Videos: http://connect.linaro.org/resource/las16/las16-101/ ★ Event Details ★ Linaro Connect Las Vegas 2016 – #LAS16 September 26-30, 2016 http://www.linaro.org http://connect.linaro.org

1. LAS16-101 - Efficient Kernel Backporting
Alex Shi

2. ENGINEERS AND DEVICES
WORKING TOGETHER
Agenda
● Why need do backporting on old
kernel
● Preparations of backporting
● How to do Backporting
● Conflicts & solution
● Post backporting
● Expectations on upstreamENGINEERS
AND DEVICES
WORKING
TOGETHER
ENGINEERS
AND DEVICES
WORKING
TOGETHER

3. ENGINEERS AND DEVICES
WORKING TOGETHER
Why is old stable kernel?
● Product considerations
● Product Time line, life cycle
● Kernel/user API, in-kernel API
− /sys, /proc,
− Driver API
● Stability
ENGINEERS
AND DEVICES
WORKING
TOGETHER

4. ENGINEERS AND DEVICES
WORKING TOGETHER
Extra Features Needed
● Desire for extra features?
● Latest drivers, functions or security
updates, like
− Arm64 PCI-E support on 3.18/4.1
− Cgroup writeback, hibernation on arm64,
KASAN on 4.1
− Or out of tree features:
HMP/EAS on ARM for big.LITTLE arch.
ENGINEERS
AND DEVICES
WORKING
TOGETHER

5. ENGINEERS AND DEVICES
WORKING TOGETHER
Solutions and Disadvantage
● Solution: LTS + feature backporting
− An example: Linaro Stable Kernel
● https://wiki.linaro.org/LSK
● Perfect solution?
− Repeated work for enabled feature
− Less review/testing from community
− Trouble with newer API or coupled kernel
componentsENGINEERS
AND DEVICES
WORKING
TOGETHER

6. ENGINEERS AND DEVICES
WORKING TOGETHER
Ideal Solutions?
● A ideal upstream kernel?
● A bug free kernel? no
● Consistent kernel API, like: driver API,
/proc, /sys, ioctl ? no
ENGINEERS
AND DEVICES
WORKING
TOGETHER

7. ENGINEERS AND DEVICES
WORKING TOGETHER
Before Backporting

8. ENGINEERS AND DEVICES
WORKING TOGETHER
Before Backporting
● Get requested feature resources:
● Get feature info from requester,
what/why
● Get feature profile from lwn.net or wiki
● Know the feature versions
● First target is always the upstream
patchset
● An old version feature maybe
acceptable, if it's using old kernel API
ENGINEERS
AND DEVICES
WORKING
TOGETHER

9. ENGINEERS AND DEVICES
WORKING TOGETHER
Get all related commits
● Know feature patches from lkml
● Get commits in git tree
− Get commits list from feature related source files and
headers
− All commits mentioned feature name
− get all writeback cgroup patches by author
git log -i -G'KASAN' v3.18..v4.4 (54 commits)
git log -i --grep=kasan v3.18..v4.4 (126 commits)
git log --author=tj@kernel.org --reverse v4.1.18.
ENGINEERS
AND DEVICES
WORKING
TOGETHER

10. ENGINEERS AND DEVICES
WORKING TOGETHER
Backporting

11. ENGINEERS AND DEVICES
WORKING TOGETHER
Backporting
● Pick up wanted changes/commits
● $git cherry-pick -sx
commit1..commit2
● git cherry-pick a patchset range
− Get all commit at once, easily disturbed by
dependent missing.
● git cherry-pick patches one by one.
− Easy to control for every commits
ENGINEERS
AND DEVICES
WORKING
TOGETHER

12. ENGINEERS AND DEVICES
WORKING TOGETHER
Backporting Path
● An example of backporting organization:
-----LTS 4.1.17 –---- LTS 4.1.18
--------------- LSK 16.02 ---------- LSK 16.03
/ /
---LTS 4.1.17+PAN /
/
-------------------LTS 4.1.18 + Cgroup WB
ENGINEERS
AND DEVICES
WORKING
TOGETHER

13. ENGINEERS AND DEVICES
WORKING TOGETHER
Conflicts & Solutions

14. ENGINEERS AND DEVICES
WORKING TOGETHER
Conflicts & Solutions
● Conflicts: middle changes missed
● Direct code base changed
− Know reasons of the changes, pick or skip
● Miss some dependencies
− Find and pick them
● Feature coupled with other kernel
components
− Cut off the connection maybe better
− Or bring trouble on the other kernel part
ENGINEERS
AND DEVICES
WORKING
TOGETHER

15. ENGINEERS AND DEVICES
WORKING TOGETHER
Feature/LTS Conflicts
● Possible upstream situation:
other blk/mm change
/
------ 4.1 -- bug fix f1 ----- conflicts -- v4.2 ---- bug fix f2/f3 ----
/
cgroup feature -- a -- b
● Usual solution for cgroup writeback LTS
branch:
backported cgwb -- a' – b'-- c'
/
--- 4.1 -- 4.1.x ---- 4.1.18
/
bug fix f1/f2/f3
ENGINEERS
AND DEVICES
WORKING
TOGETHER

16. ENGINEERS AND DEVICES
WORKING TOGETHER
Feature/LTS Conflicts
● A better way: Do the backporting on v4.1 kernel, and
left the conflicts on merging to 4.1.18
backported cgwb -- a -- b
/
--- 4.1 -- 4.1.x --------------- 4.1.18
/
bug fix f1/f2/f3
ENGINEERS
AND DEVICES
WORKING
TOGETHER

17. ENGINEERS AND DEVICES
WORKING TOGETHER
Tips for Conflicts
● Reduce conflicts
● Find out all related changes on feature
code area
− Pick up from base to upstream direction
● Pick up the related set commits instead
of only single related patch,
− Use gitk not git log --graph to find out the
commit set
ENGINEERS
AND DEVICES
WORKING
TOGETHER

18. ENGINEERS AND DEVICES
WORKING TOGETHER
Tips for Conflicts
● Conflicts solution reuse
− Use git rerere to reuse conflicts
solutions
● Find commits that trigger conflicts
− Tools git log, git blame
− git log -G'extern void inet_twsk_put' –
include/net/inet_timewait_sock.h
− git log v3.10..v3.18 -S'newsk->sk_v6_rcv_saddr =
ireq6->loc_addr' --net/dccp/ipv6.c
ENGINEERS
AND DEVICES
WORKING
TOGETHER

19. ENGINEERS AND DEVICES
WORKING TOGETHER
Don't Do ...
● Don't change kernel/Uese API /sys,
/proc etc
− System applications
● Don't pick up big coupled kernel
parts
− Cut off early
● Don't change driver API
− Downstream drivers
ENGINEERS
AND DEVICES
WORKING
TOGETHER

20. ENGINEERS AND DEVICES
WORKING TOGETHER
Post Backporting
● Check if all necessary patches picked
● Scan and compare all changed and related commits
● git log --cherry v4.4...v4.1/topic/KASAN --
./mm/kasan ./include/config/have/arch/kasan.h
./include/linux/kasan.h
./arch/arm64/mm/kasan_init.c
./arch/arm64/include/asm/kasan.h
● Testing
● Seek testing method from community
ENGINEERS
AND DEVICES
WORKING
TOGETHER

21. ENGINEERS AND DEVICES
WORKING TOGETHER
Post Backporting
● Scan bug fix for picked commits
● https://git.linaro.org/people/alex.shi/scripts.git
/blob/HEAD:/chkfix.sh
● The latest features are often buggy, so scan the
upstream kernel, to see if any commits which
you picked was mentioned by others, that's
probably a bug fix.
ENGINEERS
AND DEVICES
WORKING
TOGETHER

22. ENGINEERS AND DEVICES
WORKING TOGETHER
Post Backporting
● Notice your users of Any API
changes if you have to do so.
− Explain reasons
− And give compatible solution for changes
ENGINEERS
AND DEVICES
WORKING
TOGETHER

23. ENGINEERS AND DEVICES
WORKING TOGETHER
Expectation for Upstream
● Stable, stable, stable
● The less API change, the better.
● Standardlization of API?
− POSIX, Libcgroup, DRM, memfd, etc.
− Driver API
● keep old function when new ones
introduced
− Good examples, cgroup v2 coexist with
cgroup v1
● More collaboration on backporting
ENGINEERS
AND DEVICES
WORKING
TOGETHER

24. Thank You
#LAS16
For further information: www.linaro.org
LAS16 keynotes and videos on: connect.linaro.org