SlideShare a Scribd company logo
2019 HPCC
Systems®
Community Day
Challenge Yourself –
Challenge the Status Quo
Attila Vámos
Consulting Software Engineer
HPCC Systems Platform Team
Release on the
schedule approach
in the HPCC
Systems Platform
Team
Demand
Concept
Realisation
Challenges
Release Cycle
Changes
Demand
• A few words about past release cycles
Release on schedule approach in HPCC Systems Platform Team
• Every release:
• Has lots of changes, bug fixes, features, etc.
• Impacts on a wide range of HPCC Systems Platform components
• If something goes wrong outside the Platform team:
• It is hard to identify and extract the culprit from source tree and build a new
install package
• Therefore the whole release should roll back to an earlier release.
• It can be weeks or months old, sometimes from the previous main release
• It can hold back to develop/release new products
• It can cause problems in operations and existing products
• It can make everyone life much harder than necessary
Release on schedule approach in HPCC Systems Platform Team
Demand
Concept
• After the last Conference, we (The HPCC Systems
Platform Team) discussed and agreed to change
our release concept
Release on schedule approach in HPCC Systems Platform Team
 However this wasn’t an entirely new idea, more of a decision instead.
From large amount
of changes
released in couple
of month or longer
To more frequent
and smaller, Agile
Sprint like
releases
Concept
• Releases:
• More frequent (1-2 weeks) and smaller point release
– Contains only a couple changes
– Easy to roll back and the fix can be there in the next release
• Minor release every 3 months
• Major release every year (if it is necessary).
Release on schedule approach in HPCC Systems Platform Team
• Point:
• Defect fixes
• Small changes which
• cannot be seen outside the platform (doesn’t impact compiled ECL code)
• doesn’t change client tools behaviour
• Small refactoring work
• Minor:
• Small changes
• New features
• Larger refactoring work (encapsulates the Platform Code)
• Defect fixes
• Major:
• A change that is too large to be considered a minor release
• It can break the backward compatibility (ECL queries should recompile)
• Defect fixes
Release on schedule approach in HPCC Systems Platform Team
Release Types
• The .x branches introduced, for example 6.4.x, 7.0.x, 7.2.x, 7.4.x and 7.6.x, for
collect changes for point and minor releases
Realisation
Master 8.x features
7.6.x next development
7.4.x to fix bugs,
new features etc.
7.2.x to fix regressions
7.2.0G
2019-04-04
7.2.2G
2019-04-11
7.2.42RC-1
2019-09-05
7.4.0G
2019-07-03
7.4.2G …
2019-07-23
7.4.16G
2019-09-05
2019-09-05
…
Release on schedule approach in HPCC Systems Platform Team
• At the push of a button release
• Previously the release process was a manual process
• Merge all reviewed and tested Pull Request into the target branch
• Use git command to create tags
• Now it is automatized
• Scripts created to do a lots of previously manual operations
• Now it is nearly a push button release.
Realisation
Release on schedule approach in HPCC Systems Platform Team
Challenges
• The point release branches are kept in-house for
a short period of time only and we should test
them as rigorously as we can
• We should test all pre release candidates on a daily basis in
different environments:
– On large and small HW
– Different settings (e.g.: Thor slaves and channels/slave)
Release on schedule approach in HPCC Systems Platform Team
• The new .x branches don’t challenge Smoketest because:
• It already handles different base branches of Pull Requests
• Smoketest utilises the new Draft Pull Request feature of GitHub, where the PR
• can be used as proof of concept without any chance to infer its base branch because it
can’t be merged
• The initial commit is fully tested by Smoketest (the PR owner can manage if they want
all further commits will be tested or not)
• can observe, review by other members
• can move to standard (able to merge) PR
Release on schedule approach in HPCC Systems Platform Team
Impacts in Smoketest system
• Previously ( ~1.5 years ago) we tested only one selected branch on a daily bases.
• But we have demand to do more because we have 2 or more active branches
• Internal and Unit test case execution is reviewed and highly time consuming
cases are removed from the daily test and executed less frequently.
• Average test session time decreased from 3.5 ~ 4 hours to 1.5 ~ 2 hours
• It heavily uses versioning
• RUN_0=("BRANCH_ID=candidate-7.2.x")
• RUN_1=("BRANCH_ID=candidate-7.4.x")
• RUN_2=("BRANCH_ID=candidate-7.6.x")
• RUN_3=("BRANCH_ID=master")
• Now we have the chance to run whole test sets twice a day if necessary
Changes in our OBT system
Release on schedule approach in HPCC Systems Platform Team
Changes in our OBT system
• Automated versioning is improved and now we can test
– On a large HW (32 Cores, 128 GB RAM)
• 2 pre point release branches (7.4.x, 7.6.x ) in standard 4 Thor slaves
• 3 (7.2.x and 7.4.x, 7.6.x) in 4 Thor slaves and 4 channels/slave settings
• the master in every day
– On a small HW (4 cores, 16 GB RAM)
• 4 pre point release branches (7.2.x , 7.4.x and 7.6.x) in 4 Thor slaves
setting
• the master in every day
• Further information about our automated test system can
be found in my presentation from last year’s conference.
Release on schedule approach in HPCC Systems Platform Team
Major and minor releases in last 3.5 years.
Release on schedule approach in HPCC Systems Platform Team
Number of Gold
releases in
each branches:
5.6
6.0
6.2
6.4
7.0
7.2
7.4
5
9
15
21
21
23
10
Thank you for your attention.
If you have any questions, please send it to: attila.vamos@lexisnexisrisk.com
Release on schedule approach in HPCC Systems Platform Team
Release on schedule approach in HPCC Systems Platform Team
View this presentation on YouTube:
https://www.youtube.com/watch?v=Z1A3nOuhv3A&list=PL-
8MJMUpp8IKH5-d56az56t52YccleX5h&index=11&t=43s (1:08:01)

More Related Content

What's hot

Introduction to Foreman Maintain
Introduction to Foreman MaintainIntroduction to Foreman Maintain
Introduction to Foreman Maintain
Kavita Gaikwad
 
Getting Started with Performance Co-Pilot
Getting Started with Performance Co-PilotGetting Started with Performance Co-Pilot
Getting Started with Performance Co-Pilot
Paul V. Novarese
 
Nagios Conference 2011 - Mike Guthrie - Distributed Monitoring With Nagios
Nagios Conference 2011 - Mike Guthrie - Distributed Monitoring With NagiosNagios Conference 2011 - Mike Guthrie - Distributed Monitoring With Nagios
Nagios Conference 2011 - Mike Guthrie - Distributed Monitoring With Nagios
Nagios
 
Daniel dauwe ece 561 Benchmarking Results
Daniel dauwe   ece 561 Benchmarking ResultsDaniel dauwe   ece 561 Benchmarking Results
Daniel dauwe ece 561 Benchmarking Results
cinedan
 
Git Basic
Git BasicGit Basic
Git Basic
Lin Liu
 
Upgrading Your System: A Telco User Perspectiv
Upgrading Your System: A Telco User PerspectivUpgrading Your System: A Telco User Perspectiv
Upgrading Your System: A Telco User Perspectiv
OPNFV
 
101 3.6 modify process execution priorities v2
101 3.6 modify process execution priorities v2101 3.6 modify process execution priorities v2
101 3.6 modify process execution priorities v2
Acácio Oliveira
 
Sql disaster recovery
Sql disaster recoverySql disaster recovery
Sql disaster recovery
Sqlperfomance
 
Sep 6 cdm
Sep 6 cdmSep 6 cdm
Sep 6 cdm
rainman1985
 
Deploying puppet code at light speed
Deploying puppet code at light speedDeploying puppet code at light speed
Deploying puppet code at light speed
Tomas Doran
 
Deploying Puppet Code At Light Speed - Puppet Camp Silicon Valley
Deploying Puppet Code At Light Speed - Puppet Camp Silicon ValleyDeploying Puppet Code At Light Speed - Puppet Camp Silicon Valley
Deploying Puppet Code At Light Speed - Puppet Camp Silicon Valley
Puppet
 
Nagios Conference 2012 - Mike Guthrie - Nagios Fusion 2012
Nagios Conference 2012 - Mike Guthrie - Nagios Fusion 2012Nagios Conference 2012 - Mike Guthrie - Nagios Fusion 2012
Nagios Conference 2012 - Mike Guthrie - Nagios Fusion 2012
Nagios
 
BKK16-208 EAS
BKK16-208 EASBKK16-208 EAS
BKK16-208 EAS
Linaro
 
3.6 modify process execution priorities v2
3.6 modify process execution priorities v23.6 modify process execution priorities v2
3.6 modify process execution priorities v2
Acácio Oliveira
 
Relax and Recover rear-server Proposal 1.0
Relax and Recover rear-server Proposal 1.0Relax and Recover rear-server Proposal 1.0
Relax and Recover rear-server Proposal 1.0
Schlomo Schapiro
 
HKG15-305: Real Time processing comparing the RT patch vs Core isolation
HKG15-305: Real Time processing comparing the RT patch vs Core isolationHKG15-305: Real Time processing comparing the RT patch vs Core isolation
HKG15-305: Real Time processing comparing the RT patch vs Core isolation
Linaro
 
He Pi Xii2003
He Pi Xii2003He Pi Xii2003
He Pi Xii2003
FNian
 
SIPfoundry CoLab 2013 - sipXecs Cloud Architecture (UCCS)
SIPfoundry CoLab 2013 - sipXecs Cloud Architecture (UCCS)SIPfoundry CoLab 2013 - sipXecs Cloud Architecture (UCCS)
SIPfoundry CoLab 2013 - sipXecs Cloud Architecture (UCCS)
SIPfoundry
 
Pcp
PcpPcp

What's hot (19)

Introduction to Foreman Maintain
Introduction to Foreman MaintainIntroduction to Foreman Maintain
Introduction to Foreman Maintain
 
Getting Started with Performance Co-Pilot
Getting Started with Performance Co-PilotGetting Started with Performance Co-Pilot
Getting Started with Performance Co-Pilot
 
Nagios Conference 2011 - Mike Guthrie - Distributed Monitoring With Nagios
Nagios Conference 2011 - Mike Guthrie - Distributed Monitoring With NagiosNagios Conference 2011 - Mike Guthrie - Distributed Monitoring With Nagios
Nagios Conference 2011 - Mike Guthrie - Distributed Monitoring With Nagios
 
Daniel dauwe ece 561 Benchmarking Results
Daniel dauwe   ece 561 Benchmarking ResultsDaniel dauwe   ece 561 Benchmarking Results
Daniel dauwe ece 561 Benchmarking Results
 
Git Basic
Git BasicGit Basic
Git Basic
 
Upgrading Your System: A Telco User Perspectiv
Upgrading Your System: A Telco User PerspectivUpgrading Your System: A Telco User Perspectiv
Upgrading Your System: A Telco User Perspectiv
 
101 3.6 modify process execution priorities v2
101 3.6 modify process execution priorities v2101 3.6 modify process execution priorities v2
101 3.6 modify process execution priorities v2
 
Sql disaster recovery
Sql disaster recoverySql disaster recovery
Sql disaster recovery
 
Sep 6 cdm
Sep 6 cdmSep 6 cdm
Sep 6 cdm
 
Deploying puppet code at light speed
Deploying puppet code at light speedDeploying puppet code at light speed
Deploying puppet code at light speed
 
Deploying Puppet Code At Light Speed - Puppet Camp Silicon Valley
Deploying Puppet Code At Light Speed - Puppet Camp Silicon ValleyDeploying Puppet Code At Light Speed - Puppet Camp Silicon Valley
Deploying Puppet Code At Light Speed - Puppet Camp Silicon Valley
 
Nagios Conference 2012 - Mike Guthrie - Nagios Fusion 2012
Nagios Conference 2012 - Mike Guthrie - Nagios Fusion 2012Nagios Conference 2012 - Mike Guthrie - Nagios Fusion 2012
Nagios Conference 2012 - Mike Guthrie - Nagios Fusion 2012
 
BKK16-208 EAS
BKK16-208 EASBKK16-208 EAS
BKK16-208 EAS
 
3.6 modify process execution priorities v2
3.6 modify process execution priorities v23.6 modify process execution priorities v2
3.6 modify process execution priorities v2
 
Relax and Recover rear-server Proposal 1.0
Relax and Recover rear-server Proposal 1.0Relax and Recover rear-server Proposal 1.0
Relax and Recover rear-server Proposal 1.0
 
HKG15-305: Real Time processing comparing the RT patch vs Core isolation
HKG15-305: Real Time processing comparing the RT patch vs Core isolationHKG15-305: Real Time processing comparing the RT patch vs Core isolation
HKG15-305: Real Time processing comparing the RT patch vs Core isolation
 
He Pi Xii2003
He Pi Xii2003He Pi Xii2003
He Pi Xii2003
 
SIPfoundry CoLab 2013 - sipXecs Cloud Architecture (UCCS)
SIPfoundry CoLab 2013 - sipXecs Cloud Architecture (UCCS)SIPfoundry CoLab 2013 - sipXecs Cloud Architecture (UCCS)
SIPfoundry CoLab 2013 - sipXecs Cloud Architecture (UCCS)
 
Pcp
PcpPcp
Pcp
 

Similar to Release Cycle Changes

OpenHPC: Project Overview and Updates
OpenHPC: Project Overview and UpdatesOpenHPC: Project Overview and Updates
OpenHPC: Project Overview and Updates
inside-BigData.com
 
HPCC Systems 6.0.0 Highlights
HPCC Systems 6.0.0 HighlightsHPCC Systems 6.0.0 Highlights
HPCC Systems 6.0.0 Highlights
HPCC Systems
 
PostgreSQL 10: What to Look For
PostgreSQL 10: What to Look ForPostgreSQL 10: What to Look For
PostgreSQL 10: What to Look For
Amit Langote
 
Mixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting exampleMixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting example
corehard_by
 
ELC-E 2016 Neil Armstrong - No, it's never too late to upstream your legacy l...
ELC-E 2016 Neil Armstrong - No, it's never too late to upstream your legacy l...ELC-E 2016 Neil Armstrong - No, it's never too late to upstream your legacy l...
ELC-E 2016 Neil Armstrong - No, it's never too late to upstream your legacy l...
Neil Armstrong
 
Run-time Patching and updating Impact Estimation
Run-time Patching and updating Impact EstimationRun-time Patching and updating Impact Estimation
Run-time Patching and updating Impact Estimation
CREST
 
Parallel Computing - Lec 6
Parallel Computing - Lec 6Parallel Computing - Lec 6
Parallel Computing - Lec 6
Shah Zaib
 
Sanger, upcoming Openstack for Bio-informaticians
Sanger, upcoming Openstack for Bio-informaticiansSanger, upcoming Openstack for Bio-informaticians
Sanger, upcoming Openstack for Bio-informaticians
Peter Clapham
 
Flexible compute
Flexible computeFlexible compute
Flexible compute
Peter Clapham
 
L-2 (Computer Performance).ppt
L-2 (Computer Performance).pptL-2 (Computer Performance).ppt
L-2 (Computer Performance).ppt
ImranKhan997082
 
What is the merge window?
What is the merge window?What is the merge window?
What is the merge window?
Macpaul Lin
 
Architecting for the cloud storage build test
Architecting for the cloud storage build testArchitecting for the cloud storage build test
Architecting for the cloud storage build test
Len Bass
 
SQL Server DevOps Jumpstart
SQL Server DevOps JumpstartSQL Server DevOps Jumpstart
SQL Server DevOps Jumpstart
Ori Donner
 
Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.com
Aviran Mordo
 
GIT In Detail
GIT In DetailGIT In Detail
GIT In Detail
Haitham Raik
 
Consul administration at scale
Consul administration at scaleConsul administration at scale
Consul administration at scale
Pierre Souchay
 
Node.js what's next (Index 2018)
Node.js what's next (Index 2018)Node.js what's next (Index 2018)
Node.js what's next (Index 2018)
Gibson Fahnestock
 
DevOps: Automate all the things
DevOps: Automate all the thingsDevOps: Automate all the things
DevOps: Automate all the things
Mat Mannion
 
2016 09-dev opsjourney-devopsdaysoslo
2016 09-dev opsjourney-devopsdaysoslo2016 09-dev opsjourney-devopsdaysoslo
2016 09-dev opsjourney-devopsdaysoslo
Jon Arild Tørresdal
 
OpenHPC: A Comprehensive System Software Stack
OpenHPC: A Comprehensive System Software StackOpenHPC: A Comprehensive System Software Stack
OpenHPC: A Comprehensive System Software Stack
inside-BigData.com
 

Similar to Release Cycle Changes (20)

OpenHPC: Project Overview and Updates
OpenHPC: Project Overview and UpdatesOpenHPC: Project Overview and Updates
OpenHPC: Project Overview and Updates
 
HPCC Systems 6.0.0 Highlights
HPCC Systems 6.0.0 HighlightsHPCC Systems 6.0.0 Highlights
HPCC Systems 6.0.0 Highlights
 
PostgreSQL 10: What to Look For
PostgreSQL 10: What to Look ForPostgreSQL 10: What to Look For
PostgreSQL 10: What to Look For
 
Mixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting exampleMixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting example
 
ELC-E 2016 Neil Armstrong - No, it's never too late to upstream your legacy l...
ELC-E 2016 Neil Armstrong - No, it's never too late to upstream your legacy l...ELC-E 2016 Neil Armstrong - No, it's never too late to upstream your legacy l...
ELC-E 2016 Neil Armstrong - No, it's never too late to upstream your legacy l...
 
Run-time Patching and updating Impact Estimation
Run-time Patching and updating Impact EstimationRun-time Patching and updating Impact Estimation
Run-time Patching and updating Impact Estimation
 
Parallel Computing - Lec 6
Parallel Computing - Lec 6Parallel Computing - Lec 6
Parallel Computing - Lec 6
 
Sanger, upcoming Openstack for Bio-informaticians
Sanger, upcoming Openstack for Bio-informaticiansSanger, upcoming Openstack for Bio-informaticians
Sanger, upcoming Openstack for Bio-informaticians
 
Flexible compute
Flexible computeFlexible compute
Flexible compute
 
L-2 (Computer Performance).ppt
L-2 (Computer Performance).pptL-2 (Computer Performance).ppt
L-2 (Computer Performance).ppt
 
What is the merge window?
What is the merge window?What is the merge window?
What is the merge window?
 
Architecting for the cloud storage build test
Architecting for the cloud storage build testArchitecting for the cloud storage build test
Architecting for the cloud storage build test
 
SQL Server DevOps Jumpstart
SQL Server DevOps JumpstartSQL Server DevOps Jumpstart
SQL Server DevOps Jumpstart
 
Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.com
 
GIT In Detail
GIT In DetailGIT In Detail
GIT In Detail
 
Consul administration at scale
Consul administration at scaleConsul administration at scale
Consul administration at scale
 
Node.js what's next (Index 2018)
Node.js what's next (Index 2018)Node.js what's next (Index 2018)
Node.js what's next (Index 2018)
 
DevOps: Automate all the things
DevOps: Automate all the thingsDevOps: Automate all the things
DevOps: Automate all the things
 
2016 09-dev opsjourney-devopsdaysoslo
2016 09-dev opsjourney-devopsdaysoslo2016 09-dev opsjourney-devopsdaysoslo
2016 09-dev opsjourney-devopsdaysoslo
 
OpenHPC: A Comprehensive System Software Stack
OpenHPC: A Comprehensive System Software StackOpenHPC: A Comprehensive System Software Stack
OpenHPC: A Comprehensive System Software Stack
 

More from HPCC Systems

Natural Language to SQL Query conversion using Machine Learning Techniques on...
Natural Language to SQL Query conversion using Machine Learning Techniques on...Natural Language to SQL Query conversion using Machine Learning Techniques on...
Natural Language to SQL Query conversion using Machine Learning Techniques on...
HPCC Systems
 
Improving Efficiency of Machine Learning Algorithms using HPCC Systems
Improving Efficiency of Machine Learning Algorithms using HPCC SystemsImproving Efficiency of Machine Learning Algorithms using HPCC Systems
Improving Efficiency of Machine Learning Algorithms using HPCC Systems
HPCC Systems
 
Towards Trustable AI for Complex Systems
Towards Trustable AI for Complex SystemsTowards Trustable AI for Complex Systems
Towards Trustable AI for Complex Systems
HPCC Systems
 
Welcome
WelcomeWelcome
Welcome
HPCC Systems
 
Closing / Adjourn
Closing / Adjourn Closing / Adjourn
Closing / Adjourn
HPCC Systems
 
Community Website: Virtual Ribbon Cutting
Community Website: Virtual Ribbon CuttingCommunity Website: Virtual Ribbon Cutting
Community Website: Virtual Ribbon Cutting
HPCC Systems
 
Path to 8.0
Path to 8.0 Path to 8.0
Path to 8.0
HPCC Systems
 
Geohashing with Uber’s H3 Geospatial Index
Geohashing with Uber’s H3 Geospatial Index Geohashing with Uber’s H3 Geospatial Index
Geohashing with Uber’s H3 Geospatial Index
HPCC Systems
 
Advancements in HPCC Systems Machine Learning
Advancements in HPCC Systems Machine LearningAdvancements in HPCC Systems Machine Learning
Advancements in HPCC Systems Machine Learning
HPCC Systems
 
Docker Support
Docker Support Docker Support
Docker Support
HPCC Systems
 
Expanding HPCC Systems Deep Neural Network Capabilities
Expanding HPCC Systems Deep Neural Network CapabilitiesExpanding HPCC Systems Deep Neural Network Capabilities
Expanding HPCC Systems Deep Neural Network Capabilities
HPCC Systems
 
Leveraging Intra-Node Parallelization in HPCC Systems
Leveraging Intra-Node Parallelization in HPCC SystemsLeveraging Intra-Node Parallelization in HPCC Systems
Leveraging Intra-Node Parallelization in HPCC Systems
HPCC Systems
 
DataPatterns - Profiling in ECL Watch
DataPatterns - Profiling in ECL Watch DataPatterns - Profiling in ECL Watch
DataPatterns - Profiling in ECL Watch
HPCC Systems
 
Leveraging the Spark-HPCC Ecosystem
Leveraging the Spark-HPCC Ecosystem Leveraging the Spark-HPCC Ecosystem
Leveraging the Spark-HPCC Ecosystem
HPCC Systems
 
Work Unit Analysis Tool
Work Unit Analysis ToolWork Unit Analysis Tool
Work Unit Analysis Tool
HPCC Systems
 
Community Award Ceremony
Community Award Ceremony Community Award Ceremony
Community Award Ceremony
HPCC Systems
 
Dapper Tool - A Bundle to Make your ECL Neater
Dapper Tool - A Bundle to Make your ECL NeaterDapper Tool - A Bundle to Make your ECL Neater
Dapper Tool - A Bundle to Make your ECL Neater
HPCC Systems
 
A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...
A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...
A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...
HPCC Systems
 
Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...
Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...
Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...
HPCC Systems
 
Using High Dimensional Representation of Words (CBOW) to Find Domain Based Co...
Using High Dimensional Representation of Words (CBOW) to Find Domain Based Co...Using High Dimensional Representation of Words (CBOW) to Find Domain Based Co...
Using High Dimensional Representation of Words (CBOW) to Find Domain Based Co...
HPCC Systems
 

More from HPCC Systems (20)

Natural Language to SQL Query conversion using Machine Learning Techniques on...
Natural Language to SQL Query conversion using Machine Learning Techniques on...Natural Language to SQL Query conversion using Machine Learning Techniques on...
Natural Language to SQL Query conversion using Machine Learning Techniques on...
 
Improving Efficiency of Machine Learning Algorithms using HPCC Systems
Improving Efficiency of Machine Learning Algorithms using HPCC SystemsImproving Efficiency of Machine Learning Algorithms using HPCC Systems
Improving Efficiency of Machine Learning Algorithms using HPCC Systems
 
Towards Trustable AI for Complex Systems
Towards Trustable AI for Complex SystemsTowards Trustable AI for Complex Systems
Towards Trustable AI for Complex Systems
 
Welcome
WelcomeWelcome
Welcome
 
Closing / Adjourn
Closing / Adjourn Closing / Adjourn
Closing / Adjourn
 
Community Website: Virtual Ribbon Cutting
Community Website: Virtual Ribbon CuttingCommunity Website: Virtual Ribbon Cutting
Community Website: Virtual Ribbon Cutting
 
Path to 8.0
Path to 8.0 Path to 8.0
Path to 8.0
 
Geohashing with Uber’s H3 Geospatial Index
Geohashing with Uber’s H3 Geospatial Index Geohashing with Uber’s H3 Geospatial Index
Geohashing with Uber’s H3 Geospatial Index
 
Advancements in HPCC Systems Machine Learning
Advancements in HPCC Systems Machine LearningAdvancements in HPCC Systems Machine Learning
Advancements in HPCC Systems Machine Learning
 
Docker Support
Docker Support Docker Support
Docker Support
 
Expanding HPCC Systems Deep Neural Network Capabilities
Expanding HPCC Systems Deep Neural Network CapabilitiesExpanding HPCC Systems Deep Neural Network Capabilities
Expanding HPCC Systems Deep Neural Network Capabilities
 
Leveraging Intra-Node Parallelization in HPCC Systems
Leveraging Intra-Node Parallelization in HPCC SystemsLeveraging Intra-Node Parallelization in HPCC Systems
Leveraging Intra-Node Parallelization in HPCC Systems
 
DataPatterns - Profiling in ECL Watch
DataPatterns - Profiling in ECL Watch DataPatterns - Profiling in ECL Watch
DataPatterns - Profiling in ECL Watch
 
Leveraging the Spark-HPCC Ecosystem
Leveraging the Spark-HPCC Ecosystem Leveraging the Spark-HPCC Ecosystem
Leveraging the Spark-HPCC Ecosystem
 
Work Unit Analysis Tool
Work Unit Analysis ToolWork Unit Analysis Tool
Work Unit Analysis Tool
 
Community Award Ceremony
Community Award Ceremony Community Award Ceremony
Community Award Ceremony
 
Dapper Tool - A Bundle to Make your ECL Neater
Dapper Tool - A Bundle to Make your ECL NeaterDapper Tool - A Bundle to Make your ECL Neater
Dapper Tool - A Bundle to Make your ECL Neater
 
A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...
A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...
A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...
 
Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...
Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...
Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...
 
Using High Dimensional Representation of Words (CBOW) to Find Domain Based Co...
Using High Dimensional Representation of Words (CBOW) to Find Domain Based Co...Using High Dimensional Representation of Words (CBOW) to Find Domain Based Co...
Using High Dimensional Representation of Words (CBOW) to Find Domain Based Co...
 

Recently uploaded

DataScienceConcept_Kanchana_Weerasinghe.pptx
DataScienceConcept_Kanchana_Weerasinghe.pptxDataScienceConcept_Kanchana_Weerasinghe.pptx
DataScienceConcept_Kanchana_Weerasinghe.pptx
Kanchana Weerasinghe
 
Fine-Tuning of Small/Medium LLMs for Business QA on Structured Data
Fine-Tuning of Small/Medium LLMs for Business QA on Structured DataFine-Tuning of Small/Medium LLMs for Business QA on Structured Data
Fine-Tuning of Small/Medium LLMs for Business QA on Structured Data
kevig
 
the unexpected potential of Dijkstra's Algorithm
the unexpected potential of Dijkstra's Algorithmthe unexpected potential of Dijkstra's Algorithm
the unexpected potential of Dijkstra's Algorithm
huseindihon
 
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
janvikumar4133
 
Busty Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And...
Busty Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And...Busty Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And...
Busty Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And...
tanupasswan6
 
potential development of the A* search algorithm specifically
potential development of the A* search algorithm specificallypotential development of the A* search algorithm specifically
potential development of the A* search algorithm specifically
huseindihon
 
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
vrvipin164
 
Supervised Learning (Data Science).pptx
Supervised Learning  (Data Science).pptxSupervised Learning  (Data Science).pptx
Supervised Learning (Data Science).pptx
TARIKU ENDALE
 
Willis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdfWillis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdf
LINAT
 
Nipissing University degree offer Nipissing diploma Transcript
Nipissing University degree offer Nipissing diploma TranscriptNipissing University degree offer Nipissing diploma Transcript
Nipissing University degree offer Nipissing diploma Transcript
zyqedad
 
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
saadkhan1485265
 
New Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And N...
New Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And N...New Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And N...
New Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And N...
tanupasswan6
 
Universidad de Valladolid degree offer diploma Transcript
Universidad de Valladolid  degree offer diploma TranscriptUniversidad de Valladolid  degree offer diploma Transcript
Universidad de Valladolid degree offer diploma Transcript
taqyea
 
DU degree offer diploma Transcript
DU degree offer diploma TranscriptDU degree offer diploma Transcript
DU degree offer diploma Transcript
uapta
 
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).docbai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
PhngThLmHnh
 
Potential Uses of the Floyd-Warshall Algorithm as appropriate
Potential Uses of the Floyd-Warshall Algorithm as appropriatePotential Uses of the Floyd-Warshall Algorithm as appropriate
Potential Uses of the Floyd-Warshall Algorithm as appropriate
huseindihon
 
Artificial Intelligence (AI) Technology Project Proposal _ by Slidesgo.pptx
Artificial Intelligence (AI) Technology Project Proposal _ by Slidesgo.pptxArtificial Intelligence (AI) Technology Project Proposal _ by Slidesgo.pptx
Artificial Intelligence (AI) Technology Project Proposal _ by Slidesgo.pptx
vaishnavisharma877623
 
Data analytics and Access Program Recommendations
Data analytics and Access Program RecommendationsData analytics and Access Program Recommendations
Data analytics and Access Program Recommendations
hemantsharmaus
 
CHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptxCHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptx
girewiy968
 
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
birajmohan012
 

Recently uploaded (20)

DataScienceConcept_Kanchana_Weerasinghe.pptx
DataScienceConcept_Kanchana_Weerasinghe.pptxDataScienceConcept_Kanchana_Weerasinghe.pptx
DataScienceConcept_Kanchana_Weerasinghe.pptx
 
Fine-Tuning of Small/Medium LLMs for Business QA on Structured Data
Fine-Tuning of Small/Medium LLMs for Business QA on Structured DataFine-Tuning of Small/Medium LLMs for Business QA on Structured Data
Fine-Tuning of Small/Medium LLMs for Business QA on Structured Data
 
the unexpected potential of Dijkstra's Algorithm
the unexpected potential of Dijkstra's Algorithmthe unexpected potential of Dijkstra's Algorithm
the unexpected potential of Dijkstra's Algorithm
 
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
 
Busty Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And...
Busty Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And...Busty Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And...
Busty Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And...
 
potential development of the A* search algorithm specifically
potential development of the A* search algorithm specificallypotential development of the A* search algorithm specifically
potential development of the A* search algorithm specifically
 
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
 
Supervised Learning (Data Science).pptx
Supervised Learning  (Data Science).pptxSupervised Learning  (Data Science).pptx
Supervised Learning (Data Science).pptx
 
Willis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdfWillis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdf
 
Nipissing University degree offer Nipissing diploma Transcript
Nipissing University degree offer Nipissing diploma TranscriptNipissing University degree offer Nipissing diploma Transcript
Nipissing University degree offer Nipissing diploma Transcript
 
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
 
New Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And N...
New Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And N...New Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And N...
New Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And N...
 
Universidad de Valladolid degree offer diploma Transcript
Universidad de Valladolid  degree offer diploma TranscriptUniversidad de Valladolid  degree offer diploma Transcript
Universidad de Valladolid degree offer diploma Transcript
 
DU degree offer diploma Transcript
DU degree offer diploma TranscriptDU degree offer diploma Transcript
DU degree offer diploma Transcript
 
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).docbai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
 
Potential Uses of the Floyd-Warshall Algorithm as appropriate
Potential Uses of the Floyd-Warshall Algorithm as appropriatePotential Uses of the Floyd-Warshall Algorithm as appropriate
Potential Uses of the Floyd-Warshall Algorithm as appropriate
 
Artificial Intelligence (AI) Technology Project Proposal _ by Slidesgo.pptx
Artificial Intelligence (AI) Technology Project Proposal _ by Slidesgo.pptxArtificial Intelligence (AI) Technology Project Proposal _ by Slidesgo.pptx
Artificial Intelligence (AI) Technology Project Proposal _ by Slidesgo.pptx
 
Data analytics and Access Program Recommendations
Data analytics and Access Program RecommendationsData analytics and Access Program Recommendations
Data analytics and Access Program Recommendations
 
CHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptxCHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptx
 
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
 

Release Cycle Changes

  • 1. 2019 HPCC Systems® Community Day Challenge Yourself – Challenge the Status Quo Attila Vámos Consulting Software Engineer HPCC Systems Platform Team
  • 2. Release on the schedule approach in the HPCC Systems Platform Team Demand Concept Realisation Challenges Release Cycle Changes
  • 3. Demand • A few words about past release cycles Release on schedule approach in HPCC Systems Platform Team
  • 4. • Every release: • Has lots of changes, bug fixes, features, etc. • Impacts on a wide range of HPCC Systems Platform components • If something goes wrong outside the Platform team: • It is hard to identify and extract the culprit from source tree and build a new install package • Therefore the whole release should roll back to an earlier release. • It can be weeks or months old, sometimes from the previous main release • It can hold back to develop/release new products • It can cause problems in operations and existing products • It can make everyone life much harder than necessary Release on schedule approach in HPCC Systems Platform Team Demand
  • 5. Concept • After the last Conference, we (The HPCC Systems Platform Team) discussed and agreed to change our release concept Release on schedule approach in HPCC Systems Platform Team  However this wasn’t an entirely new idea, more of a decision instead. From large amount of changes released in couple of month or longer To more frequent and smaller, Agile Sprint like releases
  • 6. Concept • Releases: • More frequent (1-2 weeks) and smaller point release – Contains only a couple changes – Easy to roll back and the fix can be there in the next release • Minor release every 3 months • Major release every year (if it is necessary). Release on schedule approach in HPCC Systems Platform Team
  • 7. • Point: • Defect fixes • Small changes which • cannot be seen outside the platform (doesn’t impact compiled ECL code) • doesn’t change client tools behaviour • Small refactoring work • Minor: • Small changes • New features • Larger refactoring work (encapsulates the Platform Code) • Defect fixes • Major: • A change that is too large to be considered a minor release • It can break the backward compatibility (ECL queries should recompile) • Defect fixes Release on schedule approach in HPCC Systems Platform Team Release Types
  • 8. • The .x branches introduced, for example 6.4.x, 7.0.x, 7.2.x, 7.4.x and 7.6.x, for collect changes for point and minor releases Realisation Master 8.x features 7.6.x next development 7.4.x to fix bugs, new features etc. 7.2.x to fix regressions 7.2.0G 2019-04-04 7.2.2G 2019-04-11 7.2.42RC-1 2019-09-05 7.4.0G 2019-07-03 7.4.2G … 2019-07-23 7.4.16G 2019-09-05 2019-09-05 … Release on schedule approach in HPCC Systems Platform Team
  • 9. • At the push of a button release • Previously the release process was a manual process • Merge all reviewed and tested Pull Request into the target branch • Use git command to create tags • Now it is automatized • Scripts created to do a lots of previously manual operations • Now it is nearly a push button release. Realisation Release on schedule approach in HPCC Systems Platform Team
  • 10. Challenges • The point release branches are kept in-house for a short period of time only and we should test them as rigorously as we can • We should test all pre release candidates on a daily basis in different environments: – On large and small HW – Different settings (e.g.: Thor slaves and channels/slave) Release on schedule approach in HPCC Systems Platform Team
  • 11. • The new .x branches don’t challenge Smoketest because: • It already handles different base branches of Pull Requests • Smoketest utilises the new Draft Pull Request feature of GitHub, where the PR • can be used as proof of concept without any chance to infer its base branch because it can’t be merged • The initial commit is fully tested by Smoketest (the PR owner can manage if they want all further commits will be tested or not) • can observe, review by other members • can move to standard (able to merge) PR Release on schedule approach in HPCC Systems Platform Team Impacts in Smoketest system
  • 12. • Previously ( ~1.5 years ago) we tested only one selected branch on a daily bases. • But we have demand to do more because we have 2 or more active branches • Internal and Unit test case execution is reviewed and highly time consuming cases are removed from the daily test and executed less frequently. • Average test session time decreased from 3.5 ~ 4 hours to 1.5 ~ 2 hours • It heavily uses versioning • RUN_0=("BRANCH_ID=candidate-7.2.x") • RUN_1=("BRANCH_ID=candidate-7.4.x") • RUN_2=("BRANCH_ID=candidate-7.6.x") • RUN_3=("BRANCH_ID=master") • Now we have the chance to run whole test sets twice a day if necessary Changes in our OBT system Release on schedule approach in HPCC Systems Platform Team
  • 13. Changes in our OBT system • Automated versioning is improved and now we can test – On a large HW (32 Cores, 128 GB RAM) • 2 pre point release branches (7.4.x, 7.6.x ) in standard 4 Thor slaves • 3 (7.2.x and 7.4.x, 7.6.x) in 4 Thor slaves and 4 channels/slave settings • the master in every day – On a small HW (4 cores, 16 GB RAM) • 4 pre point release branches (7.2.x , 7.4.x and 7.6.x) in 4 Thor slaves setting • the master in every day • Further information about our automated test system can be found in my presentation from last year’s conference. Release on schedule approach in HPCC Systems Platform Team
  • 14. Major and minor releases in last 3.5 years. Release on schedule approach in HPCC Systems Platform Team Number of Gold releases in each branches: 5.6 6.0 6.2 6.4 7.0 7.2 7.4 5 9 15 21 21 23 10
  • 15. Thank you for your attention. If you have any questions, please send it to: attila.vamos@lexisnexisrisk.com Release on schedule approach in HPCC Systems Platform Team
  • 16. Release on schedule approach in HPCC Systems Platform Team View this presentation on YouTube: https://www.youtube.com/watch?v=Z1A3nOuhv3A&list=PL- 8MJMUpp8IKH5-d56az56t52YccleX5h&index=11&t=43s (1:08:01)