Apache NiFi SDLC Improvements

© Cloudera, Inc. All rights reserved.
Apache NiFi SDLC Improvements
Bryan Bende / @bbende
November 2019

© Cloudera, Inc. All rights reserved. 2© Cloudera, Inc. All rights reserved.
OUTLINE
• NiFi 1.10.0
• Parameterized Flows
• Force commit
• Auto-select external controller services
• Track enabled/disabled state
• Change version with nested versioning
• NiFi Registry 0.5.0
• Granular proxy permissions
• Public buckets
• Versioned Extension Bundles

PARAMETERIZED FLOWS

PROBLEMS
• Variables are referenced through expression language (EL)…
• Some properties don’t support EL and can’t be parameterized
• Can’t apply access control because references are ambiguous
• Ex: ${foo} could be a flow file attribute, variable, system property, or environment variable
• Without access control, can’t have sensitive variables
• Without sensitive variables, can’t parameterize sensitive properties!

SOLUTION – INTRODUCE PARAMETER CONTEXTS
• Parameter contexts created outside of the flow
• Context has a name, description, and one or more parameters
• Parameter has a name, description, and sensitivity flag
• Process group can be bound to one parameter context
• Components in the process group can reference parameters in the bound context
• New syntax for referencing parameters in properties: #{param-name}
• All properties support parameters regardless of expression language
• Sensitive properties can only reference sensitive parameters (vice versa)
• Integration with NiFi registry when migrating flow between environments

MANAGE PARAMETER CONTEXTS
• Control who can create
parameter contexts
• Control “view” & “modify”
permissions for each context
• Sensitive parameter values
encrypted and never
returned

BIND PROCESS GROUP TO CONTEXT
• Configure process group to select a
parameter context
• Select from contexts the current user
has “view” permissions for
• Requires “modify” on process group

REFERENCE PARAMETERS IN FLOW
• Reference parameters in any property,
regardless of EL support
• Sensitive properties can only reference
sensitive parameters
• Easily promote property values to
parameters from up-arrow icon

VERSION CONTROL FLOW WITH PARAMETERS
"parameterContexts" : {
"SFTP Params" : {
"name" : "SFTP Params",
"parameters" : [
{
"name" : "sftp.password",
"sensitive" : true
}, {
"name" : "sftp.host",
"sensitive" : false,
"value" : "localhost"
}, {
"name" : "sftp.user",
"sensitive" : false,
"value" : "myuser"
}
]
}
• Saved to registry with snapshots of referenced
parameter contexts
• Values of sensitive parameters scrubbed, set
once after importing to target environment
• Sensitive properties in versioned flow retain
parameter references like #{password}

IMPORT/UPGRADE VERSION CONTROLLED FLOW
• For each parameter context in incoming versioned flow…
• If no existing context with same name, create new context using initial values from versioned flow
• Requires permissions to create a new context
• If existing context with same name, add new parameters not already in existing context
• Requires “view” & "modify” permissions to the existing context
• After import/upgrade, set sensitive parameter values in given contexts

MANAGE PARAMETERS WITH NIFI CLI
• CLI commands for…
• create-param-context
• list-param-contexts
• get-param-context
• set-param
• delete-param
• pg-set-param-context
• export-param-context
• import-param-context
• merge-param-context

GENERAL NIFI SDLC IMPROVEMENTS

PROBLEM – CAN’T PROCEED AFTER REVERTING
• If latest version of a flow is bad, change version back to previous (i.e. revert),
BUT now local changes put flow into conflict state
• No way to move forward based on previous version

SOLUTION – FORCE COMMIT
• Allow committing local changes as next version regardless of available
upgrades (i.e. force commit next version)

PROBLEM – UNLINKED CONTROLLER SERVICES
• If a component references a controller service from outside the versioned
process group, service must be re-selected on import (first time only)

SOLUTION – AUTO-SLECET CONTROLLER SERVICES BY NAME
• Track names of external controller services referenced by versioned flow
• During import, find all services from parent groups…
• If only one service matching the desired type with same name, auto-select
• If multiple services matching desired type with same name, require user to select
• Example:
• Dev – service named ‘DBCPConnectionPool’ in root group
• Prod - service name ‘DBCPConnectionPool’ in root group
• Import flow from dev environment to prod environment
• Processors referencing ‘DBCPConnectionPool’ get correctly linked to prod service by name

OTHER IMPROVEMENTS…
• Store enabled/disabled state of components in registry
• Retain appropriate state on import of versioned flow
• https://issues.apache.org/jira/browse/NIFI-6025
• Recursively change version on nested versioned process groups when
changing version on a parent
• Ignore changes in local flow caused by new properties with default values

NIFI REGISTRY IMPROVEMENTS

PROBLEM – PROD SHOULDN’T BE ABLE MODIFY REGISTRY
• Many teams want to enforce a development workflow
• Dev -> Staging -> Prod
• If a problem is found in staging or prod, start back in dev
• Previously no way to enforce that a NiFi instance can’t write to a registry

SOLUTION – GRANULAR PROXY PERMISSIONS
• Proxy permissions allow NiFi to make
requests to registry on behalf of an end user
• Previously a single permission for Proxy
(yes or no)
• Proxy permissions now split into ‘Read’,
‘Write’, ‘Delete’
• A proxy with only ‘Read’ can import flows,
but can’t save new versions

PROBLEM – ANONYMOUS ACCESS TO SOME BUCKETS
• Secured registry requires all access to come from authenticated users
• No way to make some items public so that anyone can retrieve them
• Requires all users to have accounts

SOLUTION – DECLARE BUCKETS PUBLICLY VISIBLE
• Allow a bucket to be marked as public
• All items in a public bucket are read-only
for unauthenticated users
• Configure anonymous access
• nifi.registry.security.needClientAuth=false
• When no client cert is presented, user sent to
home page seeing publicly visible items

PROBLEM – VERSION CONTROL OF EXTENSIONS
• Versioned flows reference specific versions of extensions bundles
{
"type" : "org.apache.nifi.processors.standard.LookupRecord",
"bundle" : {
"artifact" : "nifi-standard-nar",
"group" : "org.apache.nifi",
"version" : "1.10.0"
}
...
}
• In order to deploy a flow, we also need the correct extensions bundles
• Previously no way to version control bundles along side the flows

SOLUTION – VERSIONED EXTENSION BUNDLES
• New type of versioned item in registry – ‘bundle’
• Currently one type of bundle – ‘NAR’
• Bundle must provide extension manifest (more info later)
• Registry REST API for interacting with bundles
• Bundles show in registry UI similar to flows

VERSIONED BUNDLES - DEEPER DIVE

EXTENSION MANIFESTS
• Extension manifest describes all extensions contained in the bundle
• XSD
• https://gist.github.com/bbende/8df60c186bd94ed1dbfd42d61cfc63ef
• Example
• https://github.com/apache/nifi-registry/blob/master/nifi-registry-core/nifi-registry-bundle-
utils/src/test/resources/descriptors/extension-manifest-hadoop-nar.xml
• Plan to support different types of bundles for NiFi, MiNiFi CPP, etc.
• Same extension manifest regardless of bundle type
• Extractors to read extension manifest from given bundle types

NAR BUNDLES
• NAR Maven Plugin version 1.3.1 generates extension manifests
• Requires NAR built against nifi-api 1.10.0
• Example from nifi-hadoop-nar
META-INF/
├── docs
| ├── additional-details│
| | ├── org.apache.nifi.processors.hadoop.CreateHadoopSequenceFile│
| | | └── additionalDetails.html│
| | ├── org.apache.nifi.processors.hadoop.ListHDFS│
| | └── org.apache.nifi.processors.hadoop.PutHDFS│
| └── extension-manifest.xml

REGISTRY REST API
• Consult Swagger documentation at:
• http://<registry-host>:18080/nifi-registry-api/swagger/ui.html
• Consult Admin Guide at:
• https://nifi.apache.org/docs/nifi-registry-docs/html/user-guide.html#manage-bundles

NIFI CLI
• Commands to make working with registry REST API easier…
• upload-bundle
• upload-bundles
• download-bundle
• list-bundle-groups
• list-bundle-artifacts
• list-bundle-versions
• list-extensions
• list-extension-tags

EXAMPLE – GENERATE AND BUILD NAR
mvn archetype:generate
-DarchetypeGroupId=org.apache.nifi
-DarchetypeArtifactId=nifi-processor-bundle-archetype
-DarchetypeVersion=1.10.0
-DnifiVersion=1.10.0
Define value for property 'groupId': org.apache.nifi
Define value for property 'artifactId': nifi-test-bundle
Define value for property 'version' 1.0-SNAPSHOT: : 1.0.0
Define value for property 'artifactBaseName': test
Define value for property 'package' org.apache.nifi.processors.test: :
cd nifi-test-bundle
mvn clean package
[1] https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions

EXAMPLE – UPLOAD BUNDLE
• Download nifi-toolkit-1.10.0-bin.tar.gz from
https://nifi.apache.org/download.html
• Launch CLI from nifi-toolkit-1.10.0/bin/cli.sh
• Execute upload-bundle command:
• registry upload-bundle -u http://localhost:18080 -b 1005e90f-5751-4f10-8ae5-
69e0961fc02f -ebf /path/to/nifi-test-nar-1.0.0.nar -ebt nifi-nar

EXAMPLE – VIEW IN REGISTRY UI
• Navigate to the registry UI and view bundle as a versioned item

EXAMPLE - BROWSE EXTENSION REPOSITORY API
• Registry REST API exposes a hierarchical linked API for browsing bundles
• Level 1 – Buckets the user is authorized for
• http://localhost:18080/nifi-registry-api/extension-repository
• Level 2 – Bundle group ids within a selected bucket
• http://localhost:18080/nifi-registry-api/extension-repository/Bundles
• Level 3 – Bundle artifact ids within a selected group
• http://localhost:18080/nifi-registry-api/extension-repository/Bundles/org.apache.nifi
• Level 4 – Bundle versions within a selected artifact
• http://localhost:18080/nifi-registry-api/extension-repository/Bundles/org.apache.nifi/nifi-test-nar
• Level 5 – Version specific info (download, checksum, docs)
• http://localhost:18080/nifi-registry-api/extension-repository/Bundles/org.apache.nifi/nifi-test-nar/1.0.0

EXAMPLE – DOWNLOAD BUNDLE
• Use CLI to download bundle to NiFi’s auto-load directory…
• registry download-bundle -u http://localhost:18080 -bn "Bundles" -gr
org.apache.nifi -ar nifi-test-bundle -ver 1.0.0 -od /path/to/nifi-home/extensions
• Alternatively, curl can be used:
• curl http://localhost:18080/nifi-registry-api/extension-
repository/Bundles/org.apache.nifi/nifi-test-nar/1.0.0/content > /path/to/nifi-home/nifi-
test-nar-1.0.0.nar
• NAR will automatically load after a few seconds
• Currently requires hard refresh of NiFi UI to show in the ‘Add Processor’ list

Apache NiFi SDLC Improvements

More Related Content

What's hot

Similar to Apache NiFi SDLC Improvements

More from Bryan Bende

Recently uploaded

Apache NiFi SDLC Improvements