The document discusses the importance of syncing local patches with upstream changes in open-source projects, particularly in the context of HP's cloud development efforts and the tool 'git-upstream'. It highlights the challenges faced in managing local backports and the benefits of automating this process, asserting that it can significantly reduce time spent on these tasks. The document also addresses the workflow and technical aspects of using git-upstream to maintain alignment with upstream projects.

1. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Syncing with Upstream
Darragh Bailey/Tokyo/Oct, 2015

2. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2
Bio – who am I?
• Work for HP since 2005
• Mostly focused on source control/CI/release over the 10 years
• Involved in HP Public cloud and recently Helion 2.0 (just released)
• OpenStack projects mostly involved with:
– jenkins-job-builder, python-jenkins, git-upstream
• IRC
– electrofelix (was pelix) in #openstack-infra on freenode
• GitHub
– electrofelix (recent work around vagrant plugins)

3. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.3
What does ‘Syncing with upstream’ mean?
• Direct consumer & contributor to OpenStack  This talk
– Managing local backports/fixes where/when required
– Encouraging upstreaming of fixes and automatic discarding of
backports
– Workflow used by development of HP Helion to sync with upstream
• Consuming distribution packages?
– Still mention a few interesting tools that might be useful

4. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Why?
Why do we need to align local patches with upstream?
And why do we need git-upstream to handle this?

5. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5
Why would anyone want a tool to align
patches?
• Syncing daily
• Manually it could take 5-30 minutes
(ave 15)
• For 25+ projects and growing?
• Worth spending ~90 engineer days
per year to have this take <1 minute
on average
• Currently just works >90%
https://xkcd.com/1205/

6. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6
Ideally “Contribute upstream and consume”, sounds great but is that possible?
Why are we carrying local patches?
Assumptions:
• All bugs can be debugged and fixed
within a virtual environment
• Acceptance upstream is always quick
• There is no risk of new bugs being added
• Issues never impact a production
environment or stop teams from working
– The only customers are end users
Get latest
upstream
Find
issues
(stress
test)
Develop
Fix
Submit
upstream
Release

7. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7
Probably more likely…
Get latest
upstream
Develop Fix
Submit
upstream
Release*
Apply/Drop
local fixes
Apply locally
Find issues
(stress test)
None found (ideal)
Bugs

8. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8
Types of changes
• Packaging files (including pinning against older python packages)
• Compatibility hacks for older libraries/applications for
migration/upgrades
• Backports where can’t roll forward, but need a recent change
• Dumb fixes  breaks for other environments or configurations but
allows progress for local configurations until proper solution
• Scale testing  proving a fix really works; can’t always test upstream at
sufficient scale
Anything that needs to be used in an environment besides your personal
devstack area.

9. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9
Where git-upstream fits
Get latest
upstream
Develop Fix
Submit
upstream
Release
Apply/Drop
local fixes
Apply locally
Try to automate
apply/drop of fixes
in an submit
upstream
compatible manner
Find issues
(stress test)

10. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10
Is there a benefit to upstream if we carry?
• Verify change actually solves the problem
• Run debug patches on a live system
– Provide data for discussion on fix upstream
• Allow time for review and improvements
– Avoid unnecessary pressure for changes to be accepted

11. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11
So we wrote a tool to support a work-flow
• To clone: https://github.com/openstack/git-upstream
• PyPI: https://pypi.python.org/pypi/git-upstream
• IRC: #git-upstream
• Basic description:
– git-upstream is an open source Python application that can be used to
keep in sync with upstream open source projects, mainly OpenStack.

12. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
How?
How do we go about staying in sync?

13. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13
Two phases to the work-flow
Syncing
• Fetch upstream changes
• Push to namespace in local
repo
• master  upstream/master
JJB example:
contrib/jjb/mirror.yaml
Importing/Re-aligning
• Clone local project repo
• Run ‘git upstream import’
….
• Profit
Tends to be a little more…

14. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14
The basics
Git-upstream expects to find upstream refs under ‘upstream/*’, i.e.
upstream/stable/kilo, upstream/master
To sync all patches on your master branch against upstream/master:
• git upstream import
More complete version
• git upstream import --into master upstream/master
Followed by upload to internal gerrit

15. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15
Quiet
By default git-upstream is quiet:

16. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16
Verbose
But you can increase the verbosity (spam debug messages to console)

17. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.17
Separate logging
Or request a separate log file for debug:

18. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.18
git-upstream commands available

19. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
How does this work?
Under the hood.

20. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.20
Two main approaches
Merge requests
• Proven for Linux (kernel)
• Well supported by GitHub/GitLab
• Requires main maintainer in house
for each project repo
• Conflicts likely unavoidable unless
patch accepted upstream identical
• Doesn’t map well to OpenStack
review workflow???
Patches
• Proven for OpenStack
• Well supported by Gerrit
• Possibly needs patch owner to
help resolve conflicts
• Duplicate changes will cause
conflicts
• Adding a set of patches to a repo
as separate files will not be fun to
review

21. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.21
Tools in this area already
• Patch queue
• guilt (quilt in git)
• Branches
• stgit
• topgit
• Packaging
• git-dpm
None appeared to play well with Gerrit
workflow
https://xkcd.com/927/

22. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.22
So to the requirements we worked out…
• Wanted all developers to use same workflow as upstream (local Gerrit)
• Avoid reviewing commits containing patches (diff of a diff)
• Automated discard of patches landed upstream
• A workflow that encourages us to upstream any local changes

23. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.23
Git is already great at managing patches as commits
What did we want?
previous import
next desired
import
So can we get from old to
new state without causing
issues?
current
desired
landed
upstream so
drop next time

24. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.24
But how do we avoid this?

25. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.25
The secret sauce git merge -s ours --no-commit <desired>
git read-tree -u --reset <desired>
git commit --no-edit
• We use ‘ours’ to merge from desired
to current and then replace the
contents to make things easier for
devs to follow, not because it’s
necessary. Could use ‘ours’ and
merge from current into desired.
landed
upstream so
drop next time
previous import
next desired
import
current
desired
Outstanding fixes
can be merged by
Gerrit or cleanly
rebased if conflicting
with upstream

26. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Usage in HP Helion and
status

27. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.27
Example drift
Currently worked out by a custom script, like to have git-upstream report the
raw information.

28. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.28
Current state of affairs
Git-upstream is still in a alpha/beta phase
Lack of reporting
• Difficulty in comparing between upstream and local
• Difficulty in understanding what changes are being carried
Manual interaction is poor
• Conflicts require manual finishing after resolving
• No resume + checking of locally carried changes
Lack of syncing
• Notes used for supersede must be explicitly synced using git commands
• No ‘git-upstream status’ to let user know

29. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.29
Git-upstream roadmap
List of main changes to be targeted
• https://etherpad.openstack.org/p/git-upstream
Project available on stackforge
• https://github.com/openstack/git-upstream
• https://review.openstack.org/#/q/status:open+project:openstack/git-upstream,n,z
Any help greatly appreciated!

30. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Fin
To clone: https://github.com/openstack/git-upstream
PyPI: https://pypi.python.org/pypi/git-upstream
IRC: #git-upstream (look for electrofelix/pelix)