Downloaded 67 times
Uploaded byGuido Schmutz
Self-Service Data Ingestion Using NiFi, StreamSets & Kafka
The document discusses the technologies and methodologies for self-service data ingestion using Apache NiFi, StreamSets, and Kafka. It provides an overview of each tool's features, challenges in data ingestion, and potential use cases with relevant demo cases. The author, Guido Schmutz, is an experienced consultant in IT, specializing in big data and data flow processing.
In this document
Powered by AI
Introduction of the presentation on self-service data ingestion solutions using NiFi, StreamSets & Kafka.
Guido Schmutz’s extensive experience, roles at Trivadis, and expertise in IT, Oracle, and Big Data.
Trivadis as a leading IT consulting firm with 600+ employees, expertise in multiple fields, and a strong operational presence.
Outline of the agenda covering Data Flow Processing, Apache NiFi, StreamSets Data Collector, Kafka Connect.
Introduction to Data Flow Processing as a framework for managing data ingestion and movement.
Description of traditional big data architecture involving batch processing, data warehouses, and BI tools.
Role of Event Hub in processing event stream data, including various sources like CRM and sensor data.
Architecture considerations for using Event Hubs with a focus on microservices and data integration.
Details on IoT data ingestion pipelines and handling data from multiple sensors to Cloud platforms.
Key characteristics of data flow pipelines focusing on high-throughput, data lineage, and processing methodologies.
Overview of key challenges faced during data ingestion including infrastructure, data quality, and transformation issues.
Different types of transformations during data ingestion, including zero transformation and enriching data.
Demonstration case of processing truck data with raw data illustrations.
Overview of Apache NiFi including its history, features, and unique flow-based programming approach.
List of processors used in NiFi for source and sink operations, enabling data movement and transformation.
Practical demonstrations of data flow from MQTT to Kafka using various NiFi processors.
Introduction to StreamSets as an innovative solution for big data ingestion incorporating visual approaches.
Detailing origin and processor stages in StreamSets, highlighting types of processing and data flow management.
Demonstrations showcasing data ingestion concepts within StreamSets including MQTT and Kafka integration.
Overview of Kafka Connect, its architecture, source and sink connectors, and transformation functionality.
Demonstration of configuring and using Kafka Connect for streaming data workflows.
Comparative summary of Apache NiFi, StreamSets, and Kafka Connect addressing their strengths and functions.
Final reflection on the importance of correctly utilizing technology for effective data management.
More Related Content
![[DSC Europe 22] Overview of the Databricks Platform - Petar Zecevic](https://cdn.slidesharecdn.com/ss_thumbnails/petarzecevic-databricksoverview-221130080703-c60d93de-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Self-Service Data Ingestion Using NiFi, StreamSets & Kafka
![[BDD 2025 - Artificial Intelligence] AI for the Underdogs: Innovation for Sma...](https://cdn.slidesharecdn.com/ss_thumbnails/ai-aifortheunderdogsinnovationforsmallbusinesses-251124030839-72a599a4-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] PHP in AI Age: The Laravel Way. (Rizqy Hi...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-phpinaiagethelaravelway-251125012602-ef9d330e-thumbnail.jpg?width=640&height=640&fit=bounds)
Self-Service Data Ingestion Using NiFi, StreamSets & Kafka
- 1. BASEL BERN BRUGGDÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH Self-Service Data Ingestion Using NiFi, StreamSets & Kafka Guido Schmutz – 25.4.2018 @gschmutz guidoschmutz.wordpress.com
- 2. Guido Schmutz Working atTrivadis for more than 21 years Oracle ACE Director for Fusion Middleware and SOA Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data Head of Trivadis Architecture Board Technology Manager @ Trivadis More than 30 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz
- 3. Our company. Trivadis isa market leader in IT consulting, system integration, solution engineering and the provision of IT services focusing on and technologies in Switzerland, Germany, Austria and Denmark. We offer our services in the following strategic business fields: Trivadis Services takes over the interacting operation of your IT systems. O P E R A T I O N
- 4. COPENHAGEN MUNICH LAUSANNE BERN ZURICH BRUGG GENEVA HAMBURG DÜSSELDORF FRANKFURT STUTTGART FREIBURG BASEL VIENNA With over 600specialists and IT experts in your region. 14 Trivadis branches and more than 600 employees 200 Service Level Agreements Over 4,000 training participants Research and development budget: CHF 5.0 million Financially self-supporting and sustainably profitable Experience from more than 1,900 projects per year at over 800 customers
- 5. Agenda 1. Data FlowProcessing 2. Apache NiFi 3. StreamSets Data Collector 4. Kafka Connect 5. Summary
- 6.
- 7. Hadoop Clusterd Hadoop Cluster BigData Cluster Traditional Big Data Architecture BI Tools Enterprise Data Warehouse Billing & Ordering CRM / Profile Marketing Campaigns File Import / SQL Import SQL Search / Explore Online & Mobile Apps Search SQL Export Service NoSQL Parallel Batch Processing Distributed Filesystem • Machine Learning • Graph Algorithms • Natural Language Processing
- 8. Event Hub Event Hub Hadoop Clusterd Hadoop Cluster BigData Cluster Event Hub – handle event stream data BI Tools Enterprise Data Warehouse Location Social Click stream Sensor Data Billing & Ordering CRM / Profile Marketing Campaigns Event Hub Call Center Weather Data Mobile Apps File Import / SQL Import SQL Search / Explore Online & Mobile Apps Search SQL Export ServiceData Flow Data Flow NoSQL Parallel Batch Processing Distributed Filesystem • Machine Learning • Graph Algorithms • Natural Language Processing C hange D ata C apture
- 9. Hadoop Clusterd Hadoop Cluster BigData Cluster Event Hub – taking Velocity into account Location Social Click stream Sensor Data Billing & Ordering CRM / Profile Marketing Campaigns Call Center Mobile Apps Batch Analytics Streaming Analytics Results Parallel Batch Processing Distributed Filesystem Stream Analytics NoSQL Reference / Models SQL Search SQL Export Service Dashboard BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps File Import / SQL Import Weather Data Event Hub Event Hub Event Hub Data Flow Data Flow Data Flow Change Data Capture
- 10. Container Hadoop Clusterd Hadoop Cluster BigData Cluster Event Hub – Asynchronous Microservice Architecture Location Social Click stream Sensor Data Billing & Ordering CRM / Profile Marketing Campaigns Call Center Mobile Apps Parallel Batch ProcessingDistributed Filesystem Microservice NoSQLRDBMS SQL Search SQL Export Service BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps File Import / SQL Import Weather Data { } API Event Hub Event Hub Event Hub Data Flow Data Flow Data Flow Change Data Capture
- 11. Container Hadoop Clusterd Hadoop Cluster BigData Cluster Location Social Click stream Sensor Data Billing & Ordering CRM / Profile Marketing Campaigns Call Center Mobile Apps Parallel Batch ProcessingDistributed Filesystem Microservice NoSQLRDBMS SQL Search SQL Export Service BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps File Import / SQL Import Weather Data { } API Event Hub Event Hub Event Hub Data Flow Data Flow Data Flow Change Data Capture Integrate Sanitize / Normalize Deliver
- 12. IoT GW MQTT Broker ContinuousIngestion - DataFlow Pipelines DB Source Big Data Log Stream Processing IoT Sensor Event Hub Topic Topic REST Topic IoT GW CDC GW Connect CDC DB Source Log CDC Native IoT Sensor IoT Sensor 12 Dataflow GW Topic Topic Queue Messaging GWTopic Dataflow GW Dataflow Topic REST 12 File Source Log Log Log Social Native
- 13. DataFlow Pipeline • Flow-based”programming” • Ingest Data from various sources • Extract – Transform – Load • High-Throughput, straight-through data flows • Data Lineage • Batch- or Stream-Processing • Visual coding with flow editor • Event Stream Processing (ESP) but not Complex Event Processing (CEP) Source: Confluent
- 14. Why is DataIngestion Difficult? Physical and Logical Infrastructure changes rapidly Key Challenges: Infrastructure Automation Edge Deployment Infrastructure Drift Data Structures and formats evolve and change unexpectedly Key Challenges: Consumption Readiness Corruption and Loss Structure Drift Data semantics change with evolving applications Key Challenges Timely Intervention System Consistency Semantic Drift Source: Streamsets
- 15. Challenges for IngestingSensor Data • Multitude of sensors • Real-Time Streaming • Multiple Firmware versions • Bad Data from damaged sensors • Regulatory Constraints • Data Quality Source: Cloudera
- 16. Ingestion with/without Transformation? ZeroTransformation • No transformation, plain ingest, no schema validation • Keep the original format – Text, CSV, … • Allows to store data that may have errors in the schema Format Transformation • Prefer name of Format Translation • Simply change the format • Change format from Text to Avro • Does schema validation Enrichment Transformation • Add new data to the message • Do not change existing values • Convert a value from one system to another and add it to the message Value Transformation • Replaces values in the message • Convert a value from one system to another and change the value in-place • Destroys the raw data!
- 17. Demo Case Truck-2 truck/nn/ position Truck-1 Truck-3 truck position raw? truck/nn/ positionTruck-4 Truck-5 RawData Store ? {"truckid":"57","driverid":"15","routeid":"1927624662 ","eventtype":"Normal","latitude":"38.65","longitude": "-90.21","correlationId":"4412891759760421296"}
- 18.
- 19. Apache NiFi • Originatedat NSA as Niagarafiles – developed behind closed doors for 8 years • Open sourced December 2014, Apache Top Level Project July 2015 • Look-and-Feel modernized in 2016 • Opaque, “file-oriented” payload • Distributed system of processors with centralized control • Based on flow-based programming concepts • Data Provenance and Data Lineage • Web-based user interface
- 20. Processors for Sourceand Sink • ConsumeXXXX (AMQP, EWS, IMAP, JMS, Kafka, MQTT, POP3, …) • DeleteXXXX (DynamoDB, Elasticsearch, HDFS, RethinkDB, S3, SQS, ...) • FetchXXXX (AzureBlobStorage, ElasticSearch, File, FTP, HBase, HDFS, S3 ...) • ExecuteXXXX (FlumeSink, FlumeSource, Script, SQL, ...) • GetXXXX (AzureEventHub, Couchbase, DynamoDB, File, FTP, HBase, HDFS, HTTP, Ignite, JMSQueue, JMSTopic, Kafka, Mongo, Solr, Splunk, SQS, TCP, ...) • ListenXXXX (HTTP, RELP, SMTP, Syslog, TCP, UDP, WebSocket, ...) • PublishXXXX (Kafka, MQTT) • PutXXXX (AzureBlobStorage, AzureEventHub, CassandraQL, CloudWatchMetric, Couchbase, DynamoDB, Elasticsearch, Email, FTP, File, Hbase, HDFS, HiveQL, Kudu, Lambda, Mongo, Parquet, Slack, SQL, TCP, ....) • QueryXXXX (Cassandra, DatabaseTable, DNS, Elasticserach)
- 21. Processors for Processing •ConvertXxxxToYyyy • ConvertRecord • EnforceOrder • EncryptContent • ExtractXXXX (AvroMetdata, EmailAttachments, Grok, HL7Attributes, ImageMetadata, ...) • GeoEnrichIP • JoltTransformJSON • MergeContent • ReplaceText • ResizeImage • SplitXXXX (Avro, Content, JSON, Record, Xml, ...) • TailFile • TransformXML • UpdateAttribute
- 22. Demo Case Truck-2 truck/nn/ position Truck-1 Truck-3 truck position raw truck/nn/ positionTruck-4 Truck-5 RawData Store MQTT to Kafka Kafka to Raw {"truckid":"57","driverid":"15","routeid":"1927624662 ","eventtype":"Normal","latitude":"38.65","longitude": "-90.21","correlationId":"4412891759760421296"} Port: 1883 Port: 1884
- 23. Demo: Dataflow forMQTT to Kafka
- 24.
- 25.
- 26. Demo: Masking Fieldwith ReplaceText Processor
- 27.
- 28. StreamSets Data Collector •Founded by ex-Cloudera, Informatica employees • Continuous open source, intent-driven, big data ingest • Visible, record-oriented approach fixes combinatorial explosion • Batch or stream processing • Standalone, Spark cluster, MapReduce cluster • IDE for pipeline development by ‘civilians’ • Relatively new - first public release September 2015 • So far, vast majority of commits are from StreamSets staff
- 29. StreamSets Origins Source: https://streamsets.com/connectors Anorigin stage represents the source for the pipeline. You can use a single origin stage in a pipeline Origins on the right are available out of the box API for writing custom origins
- 30. StreamSets Processors A processorstage represents a type of data processing that you want to perform use as many processors in a pipeline as you need Programming languages supported • Java • JavaScript • Jython • Groovy • Java Expression Language (EL) Spark Some of processors available out-of-the- box: • Expression Evaluator • Field Flattener • Field Hasher • Field Masker • Field Merger • Field Order • Field Splitter • Field Zip • Groovy Evaluator • JDBC Lookup • JSON Parser • Spark Evaluator • …
- 31. StreamSets Destinations A destinationstage represents the target for a pipeline. You can use one or more destinations in a pipeline Destinations on the right are available out of the box API for writing custom origins Source: https://streamsets.com/connectors
- 32. Demo Case Truck-2 truck/nn/ position Truck-1 Truck-3 truck position raw truck/nn/ positionTruck-4 Truck-5 RawData Store MQTT-1 to Kafka Kafka to Raw {"truckid":"57","driverid":"15","routeid":"1927624662 ","eventtype":"Normal","latitude":"38.65","longitude": "-90.21","correlationId":"4412891759760421296"} MQTT-2 to Kafka Edge Port: 1883 Port: 1884
- 33. Demo: Dataflow forMQTT to Kafka
- 34.
- 35.
- 36. Demo: Dataflow forMQTT to Kafka
- 37.
- 38. Demo: Sending Messageto Kafka in Avro
- 39.
- 40. Kafka Connect -Overview Source Connector Sink Connector
- 41. Kafka Connect –Single Message Transforms (SMT) Simple Transformations for a single message Defined as part of Kafka Connect • some useful transforms provided out-of-the-box • Easily implement your own Optionally deploy 1+ transforms with each connector • Modify messages produced by source connector • Modify messages sent to sink connectors Makes it much easier to mix and match connectors Some of currently available transforms: • InsertField • ReplaceField • MaskField • ValueToKey • ExtractField • TimestampRouter • RegexRouter • SetSchemaMetaData • Flatten • TimestampConverter
- 42. Kafka Connect –Many Connectors 60+ since first release (0.9+) 20+ from Confluent and Partners Source: http://www.confluent.io/product/connectors Confluent supported Connectors Certified Connectors Community Connectors
- 43. Demo Case Truck-2 truck/nn/ position Truck-1 Truck-3 truck position raw truck/nn/ positionTruck-4 Truck-5 RawData Store MQTT-1 to Kafka Kafka to Raw {"truckid":"57","driverid":"15","routeid":"1927624662 ","eventtype":"Normal","latitude":"38.65","longitude": "-90.21","correlationId":"4412891759760421296"} MQTT-2 to Kafka Port: 1883 Port: 1884
- 44. Demo: Dataflow forMQTT to Kafka #!/bin/bash curl -X "POST" "http://192.168.69.138:8083/connectors" -H "Content-Type: application/json" -d $'{ "name": "mqtt-source", "config": { "connector.class": "com.datamountaineer.streamreactor.connect.mqtt.source.MqttSourceConnector", "connect.mqtt.connection.timeout": "1000", "tasks.max": "1", "connect.mqtt.kcql": "INSERT INTO truck_position SELECT * FROM truck/+/position", "name": "MqttSourceConnector", "connect.mqtt.service.quality": "0", "connect.mqtt.client.id": "tm-mqtt-connect-01", "connect.mqtt.converter.throw.on.error": "true", "connect.mqtt.hosts": "tcp://mosquitto-1:1883" } }'
- 45.
- 46. Summary Apache NiFi • visualdataflow modelling • very powerful – “with power comes responsibility” • special package for Edge computing • data lineage and data provenance • supports for backpressure • no transport mechanism (DEV/TST/PROD) • custom processors • supported by Hortonworks StreamSets • visual dataflow modelling • very powerful – “with power comes responsibility” • special package for Edge computing • data lineage and data provenance • no transport mechanism • custom sources, sinks, processors • supported by StreamSets Kafka Connect • declarative style data flows • simplicity - “simple things done simple” • very well integrated with Kafka – comes with Kafka • Single Message Transforms (SMT) • use Kafka Streams for complex data flows • custom connectors • supported by Confluent
- 47. Technology on itsown won't help you. You need to know how to use it properly.