Low code declarative templates and workflows for generating dynamic pipelines with inputs from web hooks or pipeline job triggers.

1. Accordion Pipelines – A Cloud-native declarative Pipelines and
Dynamic workflows
Accordion Ci/cd Cloud-native Workflows
Pavan Kumar Hemadri
Thursday, September 14th, 2023
6 min read
Cisco Tech Blog Home Topics Research Contact Us
Pipelines as code have evolved in the past decade from the early days of Groovy scripts to various
forms of declarative constructs with many open source and commercial offerings. In general,
declarative pipelines use structured syntax with predefined keywords to offer better readability and
abstractions of complexity. Still, they are far from true declarative in spirit by including domain-
specific semantics.
With the increasing adoption of declarative pipelines over conventional CI jobs, standardization, and
pipeline code quality are of utmost importance in large-scale cloud native projects which involve
multiple pipelines from one or more version control repositories. Inheritance of pipelines from one
another helps improve productivity by sharing and reusing pipelines across the projects.
Similarly, the workflows in various OSS or commercial offerings require conditional constructs or
domain-specific language semantics to decide the flow of complex pipelines. This approach
contradicts declarative conventions and is unsuitable for dynamically generating pipelines based on
declarative workflows. An alternative implementation is proposed using reusable and extensible
templates to address these limitations.
In large organizations dealing with multiple projects or in open source development environments,
there is a need to share these templates within the organization or globally across private or public
Git-based version control repositories offered by GitHub, Bit Bucket, Gerrit, etc. This empowers a
true distributed pipelines development ecosystem like go modules in Golang programming language.
This blog discusses how effectively we can automate the workflows with templates on Accordion’s
cloud native CI/CD platform. Accordion pipelines with dynamic workflows are truly declarative, which
can be compared to the likes of Kubernetes YAML, which manifests free from domain-specific
semantics. This brings in powerful dynamic pipelines with a set of manifest files effectively dictated by
the process embedded in the YAML manifest.
Resources
Accordion CI/CD platform
Jenkins Pipelines
Low Code Extensible Pipeline Templates
In the context of CI/CD (Continuous Integration/Continuous Deployment) pipelines, templates refer to
predefined, reusable sets of instructions or configurations that help streamline the process of setting
up and managing pipelines for different projects or components within a software development
environment. Templates are designed to promote consistency, reduce duplication of effort, and
improve the overall efficiency of building, testing, and deploying software.
The Accordion pipeline template’s scope is granular, pervasive, and not specific to a stage or the full
pipeline. In fact, inheritance works at a block level or even for a single step within a stage. One can
create a template for an agent definition or environment or pipeline options, stages, or steps or with
Share

2. all of them in the most abstract form. Also, these templates can be nested to abstract out the least
relevant data. Here is the sample template:
From the above illustrations, template.yaml is a primary template that contains the template name,
i.e., “master_template,” and different pipeline blocks like environment, options, and stages. Whereas
pipeline.yaml is a purely abstracted pipeline that imports the template from a private GitHub
repository with the URL “https://aci-
github.cisco.com/accordion/pipeline_templates/templates/master_template.yaml” and its git revision
“8f1463c” (this is for pipeline reproducibility). Credentials are optional depending on the repository
type, i.e., public or private. Credentials can be pulled from the CI system’s native credential store or
Kubernetes secrets. The imported template is tagged as “template” in the pipeline – as seen in item
4 in pipeline.yaml. Once a template is imported, one can either inherit the entire template or a
particular block from the template.yaml. In the above example, pipeline.yaml only inherits the block
“stages,” as shown in item 1.
If we were to inherit the entire pipeline from the template, the pipeline.yaml would just be:
The above illustrations demonstrate the behavior of the generation of pipelines dynamically from the
templates. When this template inheritance feature is combined with the feature of policy-based
workflows, this software unveils easy self-serviced pipelines with dynamic workflows. Most of the
work is one-time, i.e., declaration of templates and all workflows, thereby reusing them for various
pipeline requirements.
Share

3. Dynamic Workflows
Workflows define the flow of a pipeline based on several constructs like parameters, environment, or
even webhook trigger information. Usually, these workflows are developed with the underlying
pipeline’s CI system DSL and custom syntaxes. There is a learning curve involved for pipeline
developers to understand these semantics. The solution shared provides a way of defining semantic-
free workflows with YAML specification.
Accordion’s dynamic workflows have a pure declarative approach in which one can declare the
stages’ (or tasks) names applicable to a workflow and assign a name to the workflow. When the
workflow name is attached to the Gerrit code change or pull request of GitHub / Bit Bucket in the
form of a tag or a comment, the Accordion CICD software applies the workflow name to a given
workflow and template files to render all stages with their definitions and generate the resultant
pipeline dynamically. While the actual pipeline code remains constant as a boilerplate code for any
workflow, just the workflow name gets attached to the code commit or job parameters (external to
the scope of pipeline code) to define the pipeline’s flow.
Dynamic workflows offer:
An abstraction from the workflow policy name to dynamically generate flow with required
stages using templates.
An abstraction defined in the “context” field gives flexibility to support various trigger points.
Here are some examples to illustrate the behavior of policy-based dynamic workflows in Accordion
CICD.
Share

4. In the above example, pipeline.yaml will remain the same in any workflow. For example, when
reviewing the workflow with the policy name “a7n:lint”
, the policy label “a7n:lint” is attached to a
GitHub or BitBucket PR. The Accordion CI software dynamically generates a pipeline with the blocks
defined in the workflow file and the lint stage by rendering them from templates.
Even if a developer wants to execute a pipeline with a different workflow, only the relevant workflow
name attached to the GitHub pull request changes. For example, upon attaching “a7n:build”
, it
dynamically generates the resultant pipeline for stages build, lint, and test – all this happens with no
changes to the underlying pipeline code. This feature does not need any conditional constructs or
custom DSL syntax in pipelines or workflow files. This also does not require importing multiple
workflow files per workflow for a complex pipeline.
Highlighting important aspects of dynamic workflows:
Ease of use/simplicity: no need to be a pipeline developer to define a workflow. Any user can
define the flow of execution based on the requirement. Integration is straightforward.
Abstraction: confers flexibility and scalability. For example, one can define the number of
workflows with policy labels and stages participating in the pipeline without the knowledge of
underlying pipeline code semantics.
Readability: flows can be easily understood since no conditional and additional logic is used.
Scalability and Flexibility: multiple workflows can be easily imported and referenced in a
pipeline definition.
Real-world example pipeline:
Share

5. Cisco.com Innovation Labs Research Who We Are Careers Contact 
Help Terms & Conditions Statement Trademarks © 2023 Cisco Systems, Inc.
Conclusion
The Accordion dynamic workflows and templates bring in the power of distributed pipeline
development for large-scale projects, avoiding repeatability of the pipeline code development, low
maintenance, abstraction of secrets, and secure logic. This enables the generation of workflows on
the fly with a selected GitHub label, a comment on a pull request, or any input from any VCS
software. Pure declarative manifests depict the flow of the process from which the pipeline flow is
built dynamically with VCS triggers and selections.
Acknowledgments
We sincerely thank Kalyan Ghosh, Debasish Chakraborty and the Data Center and Cloud Networking
DevOps team for their valuable contributions to this project. Share