The HPCC Systems Platform Team has modified their release approach to prioritize more frequent and smaller updates, moving from a longer release cycle to every 1-2 weeks for point releases and every three months for minor releases. Automation of the release process has improved efficiency, allowing for near 'push-button' releases and rigorous daily testing across multiple environments. Challenges still exist in testing and managing multiple active branches, but the changes have significantly reduced testing times and improved the testing infrastructure.

1. 2019 HPCC
Systems®
Community Day
Challenge Yourself –
Challenge the Status Quo
Attila Vámos
Consulting Software Engineer
HPCC Systems Platform Team

2. Release on the
schedule approach
in the HPCC
Systems Platform
Team
Demand
Concept
Realisation
Challenges
Release Cycle
Changes

3. Demand
• A few words about past release cycles
Release on schedule approach in HPCC Systems Platform Team

4. • Every release:
• Has lots of changes, bug fixes, features, etc.
• Impacts on a wide range of HPCC Systems Platform components
• If something goes wrong outside the Platform team:
• It is hard to identify and extract the culprit from source tree and build a new
install package
• Therefore the whole release should roll back to an earlier release.
• It can be weeks or months old, sometimes from the previous main release
• It can hold back to develop/release new products
• It can cause problems in operations and existing products
• It can make everyone life much harder than necessary
Release on schedule approach in HPCC Systems Platform Team
Demand

5. Concept
• After the last Conference, we (The HPCC Systems
Platform Team) discussed and agreed to change
our release concept
Release on schedule approach in HPCC Systems Platform Team
 However this wasn’t an entirely new idea, more of a decision instead.
From large amount
of changes
released in couple
of month or longer
To more frequent
and smaller, Agile
Sprint like
releases

6. Concept
• Releases:
• More frequent (1-2 weeks) and smaller point release
– Contains only a couple changes
– Easy to roll back and the fix can be there in the next release
• Minor release every 3 months
• Major release every year (if it is necessary).
Release on schedule approach in HPCC Systems Platform Team

7. • Point:
• Defect fixes
• Small changes which
• cannot be seen outside the platform (doesn’t impact compiled ECL code)
• doesn’t change client tools behaviour
• Small refactoring work
• Minor:
• Small changes
• New features
• Larger refactoring work (encapsulates the Platform Code)
• Defect fixes
• Major:
• A change that is too large to be considered a minor release
• It can break the backward compatibility (ECL queries should recompile)
• Defect fixes
Release on schedule approach in HPCC Systems Platform Team
Release Types

8. • The .x branches introduced, for example 6.4.x, 7.0.x, 7.2.x, 7.4.x and 7.6.x, for
collect changes for point and minor releases
Realisation
Master 8.x features
7.6.x next development
7.4.x to fix bugs,
new features etc.
7.2.x to fix regressions
7.2.0G
2019-04-04
7.2.2G
2019-04-11
7.2.42RC-1
2019-09-05
7.4.0G
2019-07-03
7.4.2G …
2019-07-23
7.4.16G
2019-09-05
2019-09-05
…
Release on schedule approach in HPCC Systems Platform Team

9. • At the push of a button release
• Previously the release process was a manual process
• Merge all reviewed and tested Pull Request into the target branch
• Use git command to create tags
• Now it is automatized
• Scripts created to do a lots of previously manual operations
• Now it is nearly a push button release.
Realisation
Release on schedule approach in HPCC Systems Platform Team

10. Challenges
• The point release branches are kept in-house for
a short period of time only and we should test
them as rigorously as we can
• We should test all pre release candidates on a daily basis in
different environments:
– On large and small HW
– Different settings (e.g.: Thor slaves and channels/slave)
Release on schedule approach in HPCC Systems Platform Team

11. • The new .x branches don’t challenge Smoketest because:
• It already handles different base branches of Pull Requests
• Smoketest utilises the new Draft Pull Request feature of GitHub, where the PR
• can be used as proof of concept without any chance to infer its base branch because it
can’t be merged
• The initial commit is fully tested by Smoketest (the PR owner can manage if they want
all further commits will be tested or not)
• can observe, review by other members
• can move to standard (able to merge) PR
Release on schedule approach in HPCC Systems Platform Team
Impacts in Smoketest system

12. • Previously ( ~1.5 years ago) we tested only one selected branch on a daily bases.
• But we have demand to do more because we have 2 or more active branches
• Internal and Unit test case execution is reviewed and highly time consuming
cases are removed from the daily test and executed less frequently.
• Average test session time decreased from 3.5 ~ 4 hours to 1.5 ~ 2 hours
• It heavily uses versioning
• RUN_0=("BRANCH_ID=candidate-7.2.x")
• RUN_1=("BRANCH_ID=candidate-7.4.x")
• RUN_2=("BRANCH_ID=candidate-7.6.x")
• RUN_3=("BRANCH_ID=master")
• Now we have the chance to run whole test sets twice a day if necessary
Changes in our OBT system
Release on schedule approach in HPCC Systems Platform Team

13. Changes in our OBT system
• Automated versioning is improved and now we can test
– On a large HW (32 Cores, 128 GB RAM)
• 2 pre point release branches (7.4.x, 7.6.x ) in standard 4 Thor slaves
• 3 (7.2.x and 7.4.x, 7.6.x) in 4 Thor slaves and 4 channels/slave settings
• the master in every day
– On a small HW (4 cores, 16 GB RAM)
• 4 pre point release branches (7.2.x , 7.4.x and 7.6.x) in 4 Thor slaves
setting
• the master in every day
• Further information about our automated test system can
be found in my presentation from last year’s conference.
Release on schedule approach in HPCC Systems Platform Team

14. Major and minor releases in last 3.5 years.
Release on schedule approach in HPCC Systems Platform Team
Number of Gold
releases in
each branches:
5.6
6.0
6.2
6.4
7.0
7.2
7.4
5
9
15
21
21
23
10

15. Thank you for your attention.
If you have any questions, please send it to: attila.vamos@lexisnexisrisk.com
Release on schedule approach in HPCC Systems Platform Team

16. Release on schedule approach in HPCC Systems Platform Team
View this presentation on YouTube:
https://www.youtube.com/watch?v=Z1A3nOuhv3A&list=PL-
8MJMUpp8IKH5-d56az56t52YccleX5h&index=11&t=43s (1:08:01)