More Related Content Similar to Hortonworks Data In Motion Series Part 3 - HDF Ambari (20) More from Hortonworks (20) Hortonworks Data In Motion Series Part 3 - HDF Ambari 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks DataFlow Manages Data-in-Motion
Core
InfrastructureSources
Constrained
High-latency
Localized context
Hybrid – cloud / on-premises
Low-latency
Global context
Regional
Infrastructure
Apache NiFi, Apache MiNiFi, Apache Kafka, Apache Storm are trademarks of the Apache Software Foundation
4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What is Apache Ambari?
Health Checks, Alerts
Stacks, Views
Lifecycle controls, Rolling
Restarts, Decommission/Re-
commission
Host Groups, Versioning,
Compare, Revert,
Recommendations, Security
Setup
Install Wizard (UI),
Blueprints (API)
Config
Management
ExtensibilityMonitoring
Service
Management
Cluster
Provisioning
5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Apache Ambari Integration in HDF 2.0
• Deployment
• Configuration
• Debugging
• Monitoring
7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Deployment
Deploy Apache NiFi through:
– Apache Ambari install wizard or
– Operationalize via blueprints automation
Choose which nodes Apache NiFi can
be installed on
So either:
– Choose Apache NiFi hosts at time of
cluster install
8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Deployment
– Or add Apache NiFi to existing host (post-cluster install) and then start it
9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Deployment
–Can configure which user/group Apache NiFi runs as (when cluster installed or when
Apache NiFi is added to existing cluster)
10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Deployment
Can remove Apache NiFi service from Apache Ambari
– Note: this does not remove bits
Can stop/start/configure Apache NiFi across cluster via:
– Apache Ambari UI or
– Apache Ambari REST API
11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Deployment
Can stop/start Apache NiFi on individual hosts via Apache Ambari UI and also via
REST API
12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Deployment
Quicklink for easy access to Apache
NiFi UI
– URL dynamically determined based on:
• whether SSL enabled and
• which ports user configured
14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Demo #1: Deploy HDF 2.0
•Deploy HDF cluster using Apache Ambari Install Wizard:
http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.0/bk_ambari-installation/content/ch03s01.html
–Install Apache Ambari
–Install HDF mpack
–Open Apache Ambari WebUI
–Run through Install Wizard
16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration
Configure once via Apache Ambari (instead of repeating config on each node)
Support for most Apache NiFi config files (e.g. nifi.properties, bootstrap.conf etc)
Assistance for admin during Apache NiFi configuration:
– Help text displayed on hover with property descriptions
– Checkboxes instead of true/false values
– User friendly labels and default values
– ‘Computed’ values can be automatically handled (e.g. node address)
Get benefit of Apache Ambari config features:
– Config history:
• diff between versions, revert to older version etc
– Update configs via Apache Ambari REST API
17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration
18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration
•Manage host specific configs using Config groups feature
–e.g. to bump memory for Apache NiFi on one of the nodes, click Override
button:
19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration
...create new config group and add desired host to it
20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration
‘Common’ configs grouped together and exposed in first config section
(‘Advanced nifi-ambari-config’) to allow configuration of:
– Ports (nonSSL, SSL, protocol)
– Initial and max memory (Xms, Xmx)
– Repo default dir locations (provenance, content, db, flow file)
– ‘Internal’ dir location - contains files NiFi will write to
• ‘conf’ subdir for flow/tar.gz, authorizations.xml
• ‘state’ subdir for internal state
• Can change subdir names by prefixing the desired subdir name with
‘{nifi_internal_dir}/’
– Sensitive props key (used to encrypt sensitive property values)
– Zookeeper znode for nifi
21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration
22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration
nifi.properties exposed under ‘Advanced nifi-properties’ as key/value pairs with
helptext
– values replaced by Apache Ambari shown surrounded by double braces e.g.{{ }} but
can be overridden by end user
– Properties can be updated or added via ‘Custom nifi-properties’ and will get
written to all nodes
– ‘Computed’ values handled:
• ‘Node address’ fields replaced with each host's own FQDN
24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration
Other property based config files exposed as jinja templates
– Values replaced by Apache Ambari shown surrounded by double braces e.g. {{ }} but can be overridden by
end user
– Properties can be added/updated and will get written to all nodes
26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration
Other xml based config files also exposed as jinja templates
– Values replaced by Apache Ambari shown surrounded by double braces e.g. {{ }} but can be overridden
– Elements can be updated/added and will get written to all nodes
– Config files written out with
either 0400 or 0600 perm
• Why?
• Some property files contain
plaintext passwords
28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Debugging
Logsearch integration for ease of visualizing/debugging Apache NiFi logs w/o connecting via SSH to
hosts (e.g nifi_app.log, nifi_user.log, nifi_bootstrap.log)
By default, monitors FATAL,ERROR,WARN messages (for all HDF services)
29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Debugging
Can view/drill into errors at component level or host level
Can filter errors based on severity (fatal, error, warn, info, debug, trace)
Can exclude ‘noisy’ messages to find the needle in the haystack
31. 31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring
Apache NiFi Service check: Used to ensure that Apache NiFi UI is up. Can also be
invoked via REST API for automation
32. 32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring
Apache NiFi Alert: Host level Apache Ambari alert when Apache NiFi process
goes down
– Can temporarily be disabled by turning on maintenance mode
33. 33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring
Alerts tab allows:
– Disabling and Configuring alerts (e.g. change interval)
34. 34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring
Can setup email or
SNMP notifications to
admins
35. 35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring
Apache Ambari Metrics (AMS) integration
– Service autocreates Apache Ambari reporting task, pointing to AMS collector host/port (autodetected)
36. 36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring
This is achieved by setting up an initial flow.xml (configurable via Amari)...
Tip: can auto-deploy any Apache NiFi flow by pasting its flow.xml here
37. 37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring
...and passing arguments via bootstrap.conf
Ensures Apache NiFi autodetects updated location of AMS collector (in case it’s moved)
38. 38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring
Results in AMS dashboard
for Apache NiFi metrics,
such as:
– Flowfiles sent/received
– MBs read/written
– JVM usage/thread counts
39. 39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring
Dashboard widgets can:
– be drilled into to see
results from last 1,2,4
hours, day, week etc
– export metrics data as csv
or JSON
40. 40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring
Same metrics can be viewed in Grafana dashboard
– Access via quicklink under ‘Ambari metrics’ service
– Allows users to easily create custom dashboards for each component
41. 41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Summary - Apache NiFi in Apache Ambari
• HDF available via management pack (mpack) that defines required services, stack
definitions etc
• Wizard Driven Setup
• Provides access to Apache NiFi configurations, start/stop/restart
• Has a Reporting Task for Apache Ambari Metrics
• In Cluster Mode by Default
• Leverages Zookeeper on HDF for State Management (not Embedded Zookeeper)
• Some Stack Advisor configurations (Ranger integration, Apache NiFi Certificate
Authority)
• Audit logging via Apache Ambari Infra (Internal Solr)
* You cannot install Apache Ambari to manage an HDF cluster on a system where
HDP is already installed (requires a dedicated instance)
43. 43 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Demo #2: Explore HDF 2.0
Explore HDF 2.0
– Apache NiFi configurations
– Configuration groups
– Metrics
– Logsearch integration
– Service checks
– Alerts
45. 45 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Try it out!
Deploy HDF cluster using either:
– a) Install Wizard: http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.0/bk_ambari-installation/content/ch03s01.html
– b) Blueprints: https://community.hortonworks.com/articles/56849/automate-deployment-of-hdf-20-clusters-using-ambar.html
* Reminder: You cannot install Apache Ambari to manage an HDF cluster on a system where HDP is
already installed (requires a dedicated instance)
46. 46 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Questions? https://community.hortonworks.com/
Hortonworks Community Connection:
Data Ingestion and Streaming