Developing Enterprise andDeveloping Enterprise and
Community distributions at theCommunity distributions at the
same time, impossible ?same time, impossible ?
openSUSE / SUSE example: Tumbleweed / Leap and SLEopenSUSE / SUSE example: Tumbleweed / Leap and SLE
Frédéric Crozat <fcrozat@suse.com>
SUSE Linux Enterprise Release Manager

openSUSE/SUSE terminology
●
SLE = SUSE Linux Enterprise (Server / Desktop)
– Enterprise distribution, developed by SUSE
●
openSUSE Factory:
– Development repository
●
Tumbleweed:
– openSUSE Rolling release, by openSUSE, using only Factory as
basis, tested by openQA
●
Leap:
– openSUSE Stable release, based on SLE common code +
Packages from Factory (or specific repository)
2

openSUSE & SUSE Linux Enterprise
Developed together - Reminder
openSUSE Tumbleweed
Leap
42.2
SLE
12 SP2
Shared Core
12.2
Leap
42.3
SLE
12 SP3
Shared Core
12.3
Leap
15
SLE
15
Shared Core
15
Packages list with their origin:
https://build.opensuse.org/package/view_file/openSUSE:Leap:15.0/00Meta/lookup.yml

Lessons from the pastLessons from the past
4

Mistakes were made (a few examples)
●
In SLE12 (SP0), we forked GNOME 3.10.3..
●
Even worse, we didn’t backport our features to
openSUSE:Factory !
●
We were saying “we’ll do that later...”
●
For SLE12 SP1, people were too busy bug fixing
●
“We’ll do that later...”
5

We started to fix those mistake
●
Goal was to sync back SLE 12 GNOME with openSUSE one
●
Could we share the same SRPM between SLE 12 SP2 and
Leap 42.2 ?
6

We did it !
●
More than 300 packages to sync
●
A lot of discussion and interaction between SUSE desktop
teams and openSUSE GNOME team
●
Tooling was essential, to get overview of divergeance
between SLE and openSUSE packages
●
Very few patches were enabled only on SLE 12 SP2
●
Sometime, in later bug reports, we discovered Leap 42.2 was
suffering from bugs not present in SP2, because of the above.
7

Main points
●
Work was done first internally and then pushed to openSUSE
●
Changelog integration
– Packages between SP should never loose FATE / CVE /
BSC
– openSUSE was very helpful in accepting some older
changelog entries to preserve this
●
Update handling for bug reported on Leap for packages
inherited from SLE
8

Scenes from current episode inScenes from current episode in
production (aka SLE15 / Leap 15)production (aka SLE15 / Leap 15)
9

Submitting to SLE/Factory
Submit
Request
Target:
Distro
Developer creates a
Submit Request
Distro:Staging:Foobar
SR is moved to a specific staging for testing
Mini-ISO is generated
Check-in team and various
bot
review SR
Everything is fine
(openQA is green and
review was ok)
SR committed to
Distro
ISO are
generated for all arch
OpenQA runs
full testsuite
openQA runs a
small testsuite

Factory first policy (for SLE)
●
New guidelines in effect for development since SLE12 SP3 and now for SLE15
●
Whenever possible, development should be done on OBS
(openSUSE:Factory) and pushed back to SLE15
●
When submission is reviewed and accepted on Factory, it is automatically
submitted to SLE15 (unless packages was branched in SLE15). This was in
place for most of the development period.
●
When a submission is sent to SLE15, a automated check will ensure similar
submission was done to openSUSE:Factory
●
Based on this knowledge, SLE Release Managers decide what to do with
those submit requests
●
You could see SLE15 development “live” since we were in Beta phase
11

Factory first policy

Factory first policy enforcement / bot reviews
●
Legal bot (license OK)
●
Maintenance bot (does the package build in the developer project)
●
Changelog checker bot (each SLE submission must have a bug or feature
number in its changelog)
●
Leaper bot (check if the package is supposed to be following Factory or is
allowed to fork):
– If the package is inherited from Factory, ensure the same changes are
already in Factory or waiting to be approved by Factory maintainer. If it is
not the case, submission is automatically REJECTED
– If package is allowed to branch, still check if identical changes have been
pushed to Factory and give the results as a comment to the submission
(will not auto-reject)
13

Lessons learned
●
Using automation as much as possible:
– People are less emotional if they get an rejection by a
automated tool which is enforcing a policy than a human
– If the rejection can be informative, it is better
– Have a way to override the tool, if needed
– Empower your contributors, ensure they don’t have to do
work several times
14

Give me numbers !Give me numbers !
16

SLE packages origin
17
SLE12 SP1 SLE12 SP2 SLE12 SP3 SLE15
0
500
1000
1500
2000
2500
3000
3500
4000
4500
Factory
FORK

SLE12 packages origin
●
SLE 12 SP3
– 459 “source” packages
– 328 FORK (but usually equivalent submission in OBS was
done)
– 127 are Factory packages (28%)
●
SLE 12 SP2: 1010 “source” packages
●
SLE 12 SP1: 550 “source” packages
●
SLE 12: 2971 “source” packages
18

SLE15 (post RC4) package origin
●
3850 “source” packages
●
325 FORK (but usually equivalent submission in OBS was
done) : 8.4%
●
Everything else identical to openSUSE:Factory package
(some patches might only apply on openSUSE or SLE, but
SRPM are identical) => 91.6%
19

Leap Packages origin
20
Leap 42.1 Leap 42.2 Leap 42.3 Leap 15.0
0
2000
4000
6000
8000
10000
12000
14000
Leap
Devel
FORK
Factory
SLE

Leap 42.x packages origin
●
Leap 42.1
– 7630 “source” packages
– 209 FORK
– 5698 from Factory
– 221 from Devel projects (GNOME 3.16 mostly)
– 1501 inherit from SLE12 codebase (256 from SP1, the rest from GA)
●
Leap 42.2:
– 8968 “source” packages
– 82 FORK
– 2478 are Factory packages !
– 1865 from SLE12 codebase (580 from SP2)
– Only 43 packages from Devel project (KDE 5 LTS)
●
Leap 42.3
– 10189 “source” packages
– 1990 inherit from SLE12 (270 from SP3) => 20%
– 2227 from Factory => 22 %
– 193 from Devel projects (KDE 5 LTS)
– 5780 from Leap 42.2
21

Leap 15.0 packages origin
●
Leap 15.0 (GM)
– 11600 “source” packages
– 2945 inherit from SLE15 (25%)
– 8641 from Factory
– 20 from Devel projects
22

Questions / Reactions ?
●
Nothing is set in stone
●
We are improving and smoothing our processes
23