Ingesting Data from Kafka to JDBC with Transformation and Enrichment

•Download as PPTX, PDF•

1 like•897 views

Presenter - Dr Sandeep Deshmukh, Committer Apache Apex, DataTorrent engineer Abstract: Ingesting and extracting data from Hadoop can be a frustrating, time consuming activity for many enterprises. Apache Apex Data Ingestion is a standalone big data application that simplifies the collection, aggregation and movement of large amounts of data to and from Hadoop for a more efficient data processing pipeline. Apache Apex Data Ingestion makes configuring and running Hadoop data ingestion and data extraction a point and click process enabling a smooth, easy path to your Hadoop-based big data project. In this series of talks, we would cover how Hadoop Ingestion is made easy using Apache Apex. The third talk in this series would focus on ingesting unbounded data from Kafka to JDBC with couple of processing operators -Transform and enrichment.

Technology

Meetup Series on
Hadoop Ingestion Made Easy
Part 3: Ingestion of Unbounded Data
(Kafka -> Enrich -> Transform -> JDBC)
Dr. Sandeep Deshmukh
Committer@Apache Apex
Engineer@DataTorrent
Yogi Devendra
Committer@Apache Apex
Engineer@DataTorrent

2
What is Ingestion
Data ingestion
• A process of obtaining, importing, and analyzing data for later use or storage
in a database
Big Data Ingestion
• Discovering the data sources
• Importing the data
• Processing data to produce intermediate data
• Sending data out to durable data stores
ETL + Big Data => Data ingestion

3
Requirements
• Read from stream
• Kafka
• JMS
• JDBC
• ….
• Enrich
• Transform
• Store
• JDBC

4
Use Case
Kafka
Reader
Enrich Transform
JDBC
Output
● Kafka 0.9
● Multi-topic
● Multi-partition
● Enrich State
using City
● Mark a
transaction as
HIGH or LOW
value
● Store in
DB
Parallel Partition can scale to large volume

5 5
Let us walk-through the code and appreciate how simple it is to
use Apex
● Code walk-through for an data ingestion pipeline using
○ Operators/Modules in Malhar
○ Drag-n-drop user interface
● Validate
○ Partitioning
○ Scalability
Let us create and run the app

6
Resources
• http://apex.apache.org/
• Learn more: http://apex.apache.org/docs.html
• Subscribe - http://apex.apache.org/community.html
• Download - http://apex.apache.org/downloads.html
• Follow @ApacheApex - https://twitter.com/apacheapex
• Meetups – http://www.meetup.com/pro/apacheapex/
• More examples: https://github.com/DataTorrent/examples
• Slideshare: http://www.slideshare.net/ApacheApex/presentations
• https://www.youtube.com/results?search_query=apache+apex
• Free Enterprise License for Startups -
https://www.datatorrent.com/product/startup-accelerator/

What's hot

Introduction to Apache Apex

Apache Apex

Presenter - Siyuan Hua, Apache Apex PMC Member & DataTorrent Engineer Apache Apex provides a DAG construction API that gives the developers full control over the logical plan. Some use cases don't require all of that flexibility, at least so it may appear initially. Also a large part of the audience may be more familiar with an API that exhibits more functional programming flavor, such as the new Java 8 Stream interfaces and the Apache Flink and Spark-Streaming API. Thus, to make Apex beginners to get simple first app running with familiar API, we are now providing the Stream API on top of the existing DAG API. The Stream API is designed to be easy to use yet flexible to extend and compatible with the native Apex API. This means, developers can construct their application in a way similar to Flink, Spark but also have the power to fine tune the DAG at will. Per our roadmap, the Stream API will closely follow Apache Beam (aka Google Data Flow) model. In the future, you should be able to either easily run Beam applications with the Apex Engine or express an existing application in a more declarative style.

Java High Level Stream API

Apache Apex

Presenter: Devendra Tagare - DataTorrent Engineer, Contributor to Apex, Data Architect experienced in building high scalability big data platforms. Apache Apex is a next generation native Hadoop big data platform. This talk will cover details about how it can be used as a powerful and versatile platform for big data. Apache Apex is a native Hadoop data-in-motion platform. We will discuss architectural differences between Apache Apex features with Spark Streaming. We will discuss how these differences effect use cases like ingestion, fast real-time analytics, data movement, ETL, fast batch, very low latency SLA, high throughput and large scale ingestion. We will cover fault tolerance, low latency, connectors to sources/destinations, smart partitioning, processing guarantees, computation and scheduling model, state management and dynamic changes. We will also discuss how these features affect time to market and total cost of ownership.

Intro to Apache Apex (next gen Hadoop) & comparison to Spark Streaming

Apache Apex

Intro to Apache Apex - Next Gen Native Hadoop Platform - Hackac

Apache Apex

Deep Dive into Apache Apex App Development

Apache Apex

Introduction to Apache Apex - CoDS 2016

Bhupesh Chawda

David Yan offers an overview of Apache Apex, a stream processing engine used in production by several large companies for real-time data analytics. Apache Apex uses a programming paradigm based on a directed acyclic graph (DAG). Each node in the DAG represents an operator, which can be data input, data output, or data transformation. Each directed edge in the DAG represents a stream, which is the flow of data from one operator to another. As part of Apex, the Malhar library provides a suite of connector operators so that Apex applications can read from or write to various data sources. It also includes utility operators that are commonly used in streaming applications, such as parsers, deduplicators and join, and generic building blocks that facilitate scalable state management and checkpointing. In addition to processing based on ingression time and processing time, Apex supports event-time windows and session windows. It also supports windowing, watermarks, allowed lateness, accumulation mode, triggering, and retraction detailed by Apache Beam as well as feedback loops in the DAG for iterative processing and at-least-once and “end-to-end” exactly-once processing guarantees. Apex provides various ways to fine-tune applications, such as operator partitioning, locality, and affinity. Apex is integrated with several open source projects, including Apache Beam, Apache Samoa (distributed machine learning), and Apache Calcite (SQL-based application specification). Users can choose Apex as the backend engine when running their application model based on these projects. David explains how to develop fault-tolerant streaming applications with low latency and high throughput using Apex, presenting the programming model with examples and demonstrating how custom business logic can be integrated using both the declarative high-level API and the compositional DAG-level API.

Developing streaming applications with apache apex (strata + hadoop world)

Apache Apex

The presentation covers how Apache Apex is used to deliver actionable insights in real-time for Ad-tech. It includes a reference architecture to provide dimensional aggregates on TB scale for billions of events per day. The reference architecture covers concepts around Apache Apex, with Kafka as source and dimensional compute. Slides from Devendra Tagare at Apache Big Data North America in Miami 2017.

Actionable Insights with Apache Apex at Apache Big Data 2017 by Devendra Tagare

Apache Apex

Slides from http://www.meetup.com/Hadoop-User-Group-Munich/events/230313355/ This is an overview of architecture with use cases for Apache Apex, a big data analytics platform. It comes with a powerful stream processing engine, rich set of functional building blocks and an easy to use API for the developer to build real-time and batch applications. Apex runs natively on YARN and HDFS and is used in production in various industries. You will learn more about two use cases: A leading Ad Tech company serves billions of advertising impressions and collects terabytes of data from several data centers across the world every day. Apex was used to implement rapid actionable insights, for real-time reporting and allocation, utilizing Kafka and files as source, dimensional computation and low latency visualization. A customer in the IoT space uses Apex for Time Series service, including efficient storage of time series data, data indexing for quick retrieval and queries at high scale and precision. The platform leverages the high availability, horizontal scalability and operability of Apex.

Apache Apex: Stream Processing Architecture and Applications

Thomas Weise

This is an overview of architecture with use cases for Apache Apex, a big data analytics platform. It comes with a powerful stream processing engine, rich set of functional building blocks and an easy to use API for the developer to build real-time and batch applications. Apex runs natively on YARN and HDFS and is used in production in various industries. You will learn more about two use cases: A leading Ad Tech company serves billions of advertising impressions and collects terabytes of data from several data centers across the world every day. Apex was used to implement rapid actionable insights, for real-time reporting and allocation, utilizing Kafka and files as source, dimensional computation and low latency visualization. A customer in the IoT space uses Apex for Time Series service, including efficient storage of time series data, data indexing for quick retrieval and queries at high scale and precision. The platform leverages the high availability, horizontal scalability and operability of Apex.

Hadoop Summit SJ 2016: Next Gen Big Data Analytics with Apache Apex

Apache Apex

Apache Apex is a next gen big data analytics platform. Originally developed at DataTorrent it comes with a powerful stream processing engine, rich set of functional building blocks and an easy to use API for the developer to build real-time and batch applications. Apex runs natively on YARN and HDFS and is used in production in various industries. You will learn about the Apex architecture, including its unique features for scalability, fault tolerance and processing guarantees, programming model and use cases. http://apachebigdata2016.sched.org/event/6M0L/next-gen-big-data-analytics-with-apache-apex-thomas-weise-datatorrent

Apache Big Data 2016: Next Gen Big Data Analytics with Apache Apex

Apache Apex

Building your first aplication using Apache Apex

Yogi Devendra Vyavahare

DataTorrent Presentation @ Big Data Application Meetup

Thomas Weise

This presentation discusses architectural differences between Apache Apex features with Spark Streaming. It discusses how these differences effect use cases like ingestion, fast real-time analytics, data movement, ETL, fast batch, very low latency SLA, high throughput and large scale ingestion. Also, it will cover fault tolerance, low latency, connectors to sources/destinations, smart partitioning, processing guarantees, computation and scheduling model, state management and dynamic changes. Further, it will discuss how these features affect time to market and total cost of ownership.

Architectual Comparison of Apache Apex and Spark Streaming

Apache Apex

Introduction to Apache Apex

Apache Apex

Processing big data often requires running the same computations parallelly in multiple processes or threads, called partitions, with each partition handling a subset of the data. This becomes all the more necessary when processing live data streams where maintaining SLA is paramount. Furthermore, multiple different computations make up an application and each of them may have different partitioning needs. Partitioning also needs to adapt to changing data rates, input sources and other application requirements like SLA. In this talk, we will introduce how Apache Apex, a distributed stream processing platform on Hadoop, handles partitioning. We will look at different partitioning schemes provided by Apex some of which are unique in this space. We will also look at how Apex does dynamic partitioning, a feature unique to and pioneered by Apex to handle varying data needs with examples. We will also talk about the different utilities and libraries that Apex provides for users to be able to affect their own custom partitioning.

Smart Partitioning with Apache Apex (Webinar)

Apache Apex

Ingestion and Dimensions Compute and Enrich using Apache Apex

Apache Apex

Introduction to Apache Apex - The next generation native Hadoop platform, and writing a native Hadoop big data Apache Apex streaming application. This talk will cover details about how Apex can be used as a powerful and versatile platform for big data. Apache apex is being used in production by customers for both streaming and batch use cases. Common usage of Apache Apex includes big data ingestion, streaming analytics, ETL, fast batch. alerts, real-time actions, threat detection, etc. Presenter : <b>Pramod Immaneni</b> Apache Apex PPMC member and senior architect at DataTorrent Inc, where he works on Apex and specializes in big data applications. Prior to DataTorrent he was a co-founder and CTO of Leaf Networks LLC, eventually acquired by Netgear Inc, where he built products in core networking space and was granted patents in peer-to-peer VPNs. Before that he was a technical co-founder of a mobile startup where he was an architect of a dynamic content rendering engine for mobile devices. This is a video of the webcast of an Apache Apex meetup event organized by Guru Virtues at 267 Boston Rd no. 9, North Billerica, MA, on <b>May 7th 2016</b> and broadcasted from San Jose, CA. If you are interested in helping organize i.e., hosting, presenting, community leadership Apache Apex community, please email apex-meetup@datatorrent.com

Introduction to Apache Apex and writing a big data streaming application

Apache Apex

Kafka is a vital part of data infrastructure in many organizations. When the Kafka cluster grows and more data is stored in Kafka for a longer duration, several issues related to scalability, efficiency, and operations become important to address. Kafka cluster storage is typically scaled by adding more broker nodes to the cluster. But this also adds needless memory and CPUs to the cluster making overall storage cost less efficient compared to storing the older data in external storage. Tiered storage is introduced to extend Kafka's storage beyond the local storage available on the Kafka cluster by retaining the older data in cheaper stores, such as HDFS, S3, Azure or GCS with minimal impact on the internals of Kafka. We will talk about - How tiered storage addresses the above problems and also brings several other advantages. - High level architecture of tiered storage - Future work planned as part of tiered storage.

Kafka Tiered Storage | Satish Duggana and Sriharsha Chintalapani, Uber

HostedbyConfluent

Apache Apex Fault Tolerance and Processing Semantics

Apache Apex

What's hot (20)

Introduction to Apache Apex

Java High Level Stream API

Intro to Apache Apex (next gen Hadoop) & comparison to Spark Streaming

Intro to Apache Apex - Next Gen Native Hadoop Platform - Hackac

Deep Dive into Apache Apex App Development

Introduction to Apache Apex - CoDS 2016

Developing streaming applications with apache apex (strata + hadoop world)

Actionable Insights with Apache Apex at Apache Big Data 2017 by Devendra Tagare

Apache Apex: Stream Processing Architecture and Applications

Hadoop Summit SJ 2016: Next Gen Big Data Analytics with Apache Apex

Apache Big Data 2016: Next Gen Big Data Analytics with Apache Apex

Building your first aplication using Apache Apex

DataTorrent Presentation @ Big Data Application Meetup

Architectual Comparison of Apache Apex and Spark Streaming

Introduction to Apache Apex

Smart Partitioning with Apache Apex (Webinar)

Ingestion and Dimensions Compute and Enrich using Apache Apex

Introduction to Apache Apex and writing a big data streaming application

Kafka Tiered Storage | Satish Duggana and Sriharsha Chintalapani, Uber

Apache Apex Fault Tolerance and Processing Semantics

Viewers also liked

Apache Apex Kafka Input Operator

Apache Apex

Intro to Big Data Hadoop

Apache Apex

Introduction to Yarn

Apache Apex

Fault-Tolerant File Input & Output

Apache Apex

Apache Apex connector with Kafka 0.9 consumer API

Apache Apex

Gobblin: Unifying Data Ingestion for Hadoop

Yinan Li

This talk will explore the area of real-time data ingest into Hadoop and present the architectural trade-offs as well as demonstrate alternative implementations that strike the appropriate balance across the following common challenges: * Decentralized writes (multiple data centers and collectors) * Continuous Availability, High Reliability * No loss of data * Elasticity of introducing more writers * Bursts in Speed per syslog emitter * Continuous, real-time collection * Flexible Write Targets (local FS, HDFS etc.)

High Speed Continuous & Reliable Data Ingest into Hadoop

DataWorks Summit

Meetup presentation: How Apache Apex consumes from Kafka topics for real-time time processing and analytics. Learn about features of the Apex Kafka Connector, which is one of the most popular operators in the Apex Malhar operator library, and powers several production use cases. We explain the advanced features this operator provides for high throughput, low latency ingest and how it enables fault tolerant topologies with exactly once processing semantics.

Stream data from Apache Kafka for processing with Apache Apex

Apache Apex

Spark Streaming into context

David Martínez Rego

* Title * Spark After Dark 1.5: Deep Dive Into Latest Perf and Scale Improvements in Spark Ecosystem * Abstract * Combining the most popular and technically-deep material from his wildly popular Advanced Apache Spark Meetup, Chris Fregly will provide code-level deep dives into the latest performance and scalability advancements within the Apache Spark Ecosystem by exploring the following: 1) Building a Scalable and Performant Spark SQL/DataFrames Data Source Connector such as Spark-CSV, Spark-Cassandra, Spark-ElasticSearch, and Spark-Redshift 2) Speeding Up Spark SQL Queries using Partition Pruning and Predicate Pushdowns with CSV, JSON, Parquet, Avro, and ORC 3) Tuning Spark Streaming Performance and Fault Tolerance with KafkaRDD and KinesisRDD 4) Maintaining Stability during High Scale Streaming Ingestion using Approximations and Probabilistic Data Structures from Spark, Redis, and Twitter's Algebird 5) Building Effective Machine Learning Models using Feature Engineering, Dimension Reduction, and Natural Language Processing with MLlib/GraphX, ML Pipelines, DIMSUM, Locality Sensitive Hashing, and Stanford's CoreNLP 6) Tuning Core Spark Performance by Acknowledging Mechanical Sympathy for the Physical Limitations of OS and Hardware Resources such as CPU, Memory, Network, and Disk with Project Tungsten, Asynchronous Netty, and Linux epoll * Demos * This talk features many interesting and audience-interactive demos - as well as code-level deep dives into many of the projects listed above. All demo code is available on Github at the following link: https://github.com/fluxcapacitor/pipeline/wiki In addition, the entire demo environment has been Dockerized and made available for download on Docker Hub at the following link: https://hub.docker.com/r/fluxcapacitor/pipeline/ * Speaker Bio * Chris Fregly is a Principal Data Solutions Engineer for the newly-formed IBM Spark Technology Center, an Apache Spark Contributor, a Netflix Open Source Committer, as well as the Organizer of the global Advanced Apache Spark Meetup and Author of the Upcoming Book, Advanced Spark. Previously, Chris was a Data Solutions Engineer at Databricks and a Streaming Data Engineer at Netflix. When Chris isn’t contributing to Spark and other open source projects, he’s creating book chapters, slides, and demos to share knowledge with his peers at meetups and conferences throughout the world.

Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap...

Chris Fregly

Spark streaming: Best Practices

Prakash Chockalingam

Data Ingestion, Extraction & Parsing on Hadoop

skaluska

Workshop - How to Build Recommendation Engine using Spark 1.6 and HDP Hands-on - Build a Data analytics application using SPARK, Hortonworks, and Zeppelin. The session explains RDD concepts, DataFrames, sqlContext, use SparkSQL for working with DataFrames and explore graphical abilities of Zeppelin. b) Follow along - Build a Recommendation Engine - This will show how to build a predictive analytics (MLlib) recommendation engine with scoring This will give a better understanding of architecture and coding in Spark for ML.

Spark Advanced Analytics NJ Data Science Meetup - Princeton University

Alex Zeltov

This talk was prepared for the November 2013 DataPhilly Meetup: Data in Practice ( http://www.meetup.com/DataPhilly/events/149515412/ ) Map Reduce: Beyond Word Count by Jeff Patti Have you ever wondered what map reduce can be used for beyond the word count example you see in all the introductory articles about map reduce? Using Python and mrjob, this talk will cover a few simple map reduce algorithms that in part power Monetate's information pipeline Bio: Jeff Patti is a backend engineer at Monetate with a passion for algorithms, big data, and long walks on the beach. Prior to working at Monetate he performed software R&D for Lockheed Martin, where he worked on projects ranging from social network analysis to robotics.

Map reduce: beyond word count

Jeff Patti

Recipes for Running Spark Streaming Applications in Production-(Tathagata Das...

Spark Summit

Spark+flume seattle

Hari Shreedharan

Scala - The Simple Parts, SFScala presentation

Martin Odersky

Production debugging is hard, and it’s getting harder. With architectures becoming more distributed and code more asynchronous and reactive, pinpointing and resolving errors that happen in production is no child’s game. This session covers some essential tools and more advanced techniques Scala developers can use to debug live applications and resolve errors quickly. It explores crucial techniques for distributed debugging - and some of the pitfalls that make resolution much harder, and can lead to downtime. The talk also touches on some little-known JVM tools and capabilities that give you super-deep visibility at high scale without making you restart it or attach debuggers.

Advanced Production Debugging

Takipi

HDFS Internals

Apache Apex

Introduction to Map Reduce

Apache Apex

Viewers also liked (20)

Apache Apex Kafka Input Operator

Intro to Big Data Hadoop

Introduction to Yarn

Fault-Tolerant File Input & Output

Apache Apex connector with Kafka 0.9 consumer API

Gobblin: Unifying Data Ingestion for Hadoop

High Speed Continuous & Reliable Data Ingest into Hadoop

Stream data from Apache Kafka for processing with Apache Apex

Spark Streaming into context

Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap...

Spark streaming: Best Practices

Data Ingestion, Extraction & Parsing on Hadoop

Spark Advanced Analytics NJ Data Science Meetup - Princeton University

Map reduce: beyond word count

Recipes for Running Spark Streaming Applications in Production-(Tathagata Das...

Spark+flume seattle

Scala - The Simple Parts, SFScala presentation

Advanced Production Debugging

HDFS Internals

Introduction to Map Reduce

Similar to Ingesting Data from Kafka to JDBC with Transformation and Enrichment

Apache Hive is a rapidly evolving project, many people are loved by the big data ecosystem. Hive continues to expand support for analytics, reporting, and bilateral queries, and the community is striving to improve support along with many other aspects and use cases. In this lecture, we introduce the latest and greatest features and optimization that appeared in this project last year. This includes benchmarks covering LLAP, Apache Druid's materialized views and integration, workload management, ACID improvements, using Hive in the cloud, and performance improvements. I will also tell you a little about what you can expect in the future.

What's New in Apache Hive 3.0 - Tokyo

DataWorks Summit

What's New in Apache Hive 3.0?

DataWorks Summit

Azure Data Engineering.pptx

priyadharshini626440

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Alluxio, Inc.

Otimizações de Projetos de Big Data, Dw e AI no Microsoft Azure

Luan Moreno Medeiros Maciel

Whats new in Oracle Database 12c release 12.1.0.2

Connor McDonald

Artur Borycki - Beyond Lambda - how to get from logical to physical - code.ta...

AboutYouGmbH

Slide 2 collecting, storing and analyzing big data

Trieu Nguyen

Kafka & Hadoop in Rakuten

Rakuten Group, Inc.

Gruter TECHDAY 2014 Realtime Processing in Telco

Gruter

Building Apps with Distributed In-Memory Computing Using Apache Geode

PivotalOpenSourceHub

Analyzing transactional data residing in Oracle databases is becoming increasingly common, especially as the data sizes and complexity increase and transactional stores are no longer to keep pace with the ever-increasing storage. Although there are many techniques available for loading Oracle data, getting up-to-date data into your data warehouse store is a more difficult problem. VMware Continuent provides provides data replication from Oracle to data warehouses and analytics engines, to derive insight from big data for better business decisions. Learn practical tips on how to get your data warehouse loading projects off the ground quickly and efficiently when replicating from Oracle into Hadoop, Amazon Redshift, and HP Vertica.

Replicate from Oracle to data warehouses and analytics

Continuent

Hadoop and IDW - When_to_use_which

Dan TheMan

I would like you to join me on our journey from a complex, multi instance Oracle topology to a single logical database in PostgreSQL. Each technology and architectural decision point will be discussed describing how we arrived at our destination. There are five keys areas that will be covered: - Target architecture - Migration of database objects (tables, indexes, views, synonyms, etc) - Migration of database code (packages, functions, procedures, triggers) - Application tier - Migration of Data - with minimal downtime during cutover The target architecture is a BDR cluster, where the physical data model and data stored is different between the logical standbys and the lead master/shadow master. Will discuss how this allowed for the simplification of the topology, and the benefits this delivered. Before you go there, yes I know PostgreSQL does no have synonyms, but an alternative approach was needed. There is a significant amount of business logic in the database tier all of which needed to be translated into database code. Will look at the tools and extensions available to reproduce the functionality in PostgreSQL. Look at common non-ISO standard SQL embedded in the application tier, along with jdbc challenges. Finally a look at some of the data movement tools available. Full disclosure, we are still on the journey but have learnt a lot on the way.

A Journey from Oracle to PostgreSQL

EDB

Oracle big data appliance and solutions

solarisyougood

JDBC Connectivity Model

kunj desai

Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010

Bhupesh Bansal

Hadoop and Voldemort @ LinkedIn

Hadoop User Group

Modernize Your Existing EDW with IBM Big SQL & Hortonworks Data Platform

Hortonworks

Presentation big dataappliance-overview_oow_v3

xKinAnx

Similar to Ingesting Data from Kafka to JDBC with Transformation and Enrichment (20)

What's New in Apache Hive 3.0 - Tokyo

What's New in Apache Hive 3.0?

Azure Data Engineering.pptx

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Otimizações de Projetos de Big Data, Dw e AI no Microsoft Azure

Whats new in Oracle Database 12c release 12.1.0.2

Artur Borycki - Beyond Lambda - how to get from logical to physical - code.ta...

Slide 2 collecting, storing and analyzing big data

Kafka & Hadoop in Rakuten

Gruter TECHDAY 2014 Realtime Processing in Telco

Building Apps with Distributed In-Memory Computing Using Apache Geode

Replicate from Oracle to data warehouses and analytics

Hadoop and IDW - When_to_use_which

A Journey from Oracle to PostgreSQL

Oracle big data appliance and solutions

JDBC Connectivity Model

Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010

Hadoop and Voldemort @ LinkedIn

Modernize Your Existing EDW with IBM Big SQL & Hortonworks Data Platform

Presentation big dataappliance-overview_oow_v3

More from Apache Apex

Data science is fast becoming a complementary approach and process to solve business challenges today. The explosion of frameworks to help data scientists build models bears a testimony to this. However when a model needs to be turned into a production version in very low latency and enterprise grade environments, there are a very few choices with each one having their own strengths and weaknesses. Adding to this is the current disconnect between a data scientists world which is all about modelling and an engineers world which is about SLAs and service guarantees. A framework like Apache Apex can complement each of these roles and provide constructs for both these worlds. This would help enterprises to drastically cut down the cost of model deployment to production environments.

Low Latency Polyglot Model Scoring using Apache Apex

Apache Apex

Hadoop Interacting with HDFS

Apache Apex

Introduction to Real-Time Data Processing

Apache Apex

Big Data Berlin v8.0 Stream Processing with Apache Apex

Apache Apex

Presenter: Kenn Knowles, Software Engineer, Google & Apache Beam (incubating) PPMC member Apache Beam (incubating) is a programming model and library for unified batch & streaming big data processing. This talk will cover the Beam programming model broadly, including its origin story and vision for the future. We will dig into how Beam separates concerns for authors of streaming data processing pipelines, isolating what you want to compute from where your data is distributed in time and when you want to produce output. Time permitting, we might dive deeper into what goes into building a Beam runner, for example atop Apache Apex.

Apache Beam (incubating)

Apache Apex

Making sense of Apache Bigtop's role in ODPi and how it matters to Apache Apex

Apache Apex

Chinmay Kolhatkar: Engineer, DataTorrent & Committer, Apache Apex For ease of use and deployment, Apache Apex leverages Apache Bigtop. Apex, being part of bigtop stack, can be easily deployed in both debian and rpm based cluster system and run validation tests for installation. This talk will cover a demo on how to install apex-bigtop and use it. It also covers a test sandbox docker environment, having pre-installed bigtop-hadoop and bigtop-apex, for quickly getting started with apex.

Apache Apex & Bigtop

Apache Apex

Building Your First Apache Apex Application

Apache Apex

More from Apache Apex (8)

Low Latency Polyglot Model Scoring using Apache Apex

Hadoop Interacting with HDFS

Introduction to Real-Time Data Processing

Big Data Berlin v8.0 Stream Processing with Apache Apex

Apache Beam (incubating)

Making sense of Apache Bigtop's role in ODPi and how it matters to Apache Apex

Apache Apex & Bigtop

Building Your First Apache Apex Application

Recently uploaded

ScyllaDB has the potential to deliver impressive performance and scalability. The better you understand how it works, the more you can squeeze out of it. But before you squeeze, make sure you know what to monitor! Watch our experienced Postgres developer work through monitoring and performance strategies that help him understand what mistakes he’s made moving to NoSQL. And learn with him as our database performance expert offers friendly guidance on how to use monitoring and performance tuning to get his sample Rust application on the right track. This webinar focuses on using monitoring and performance tuning to discover and correct mistakes that commonly occur when developers move from SQL to NoSQL. For example: - Common issues getting up and running with the monitoring stack - Using the CQL optimizations dashboard - Common issues causing high latency in a node - Common issues causing replica imbalance - What a healthy system looks like in terms of memory - Key metrics to keep an eye on This isn’t “Death-by-Powerpoint.” We’ll walk through problems encountered while migrating a real application from Postgres to ScyllaDB – and try to fix them live as well.

Optimizing NoSQL Performance Through Observability

ScyllaDB

In today’s fast-paced digital world, harnessing the power of artificial intelligence (AI) can significantly enhance productivity and creativity across various domains. With the advent of advanced language models like ChatGPT, developers, marketers, data analysts, and professionals in numerous other fields can now leverage AI-generated prompts to spark innovative ideas and streamline their workflows.

1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT

iSEO AI

FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...

FIDO Alliance

Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf

FIDO Alliance

This talk focuses on the practical aspects of integrating various telephony systems with Salesforce, drawing on examples from implementations in the Czech scene. It aims to inform attendees about the spectrum of telephony solutions available, from small to large scale, and their compatibility with Salesforce. The presentation will highlight key considerations for selecting a telephony provider that integrates smoothly with Salesforce, including important questions to support the decision-making process. It will also discuss methods for integrating existing telephony systems with Salesforce, aimed at companies contemplating or in the process of adopting this CRM platform. The discussion is designed to provide a straightforward overview of the steps and considerations involved in telephony and Salesforce integration, with an emphasis on functionality, compatibility, and the practical experiences of Czech companies.

Integrating Telephony Systems with Salesforce: Insights and Considerations, B...

CzechDreamin

Speed Wins: From Kafka to APIs in Minutes

confluent

Webinar Recording: https://www.panagenda.com/webinars/easier-faster-and-more-powerful-notes-document-properties-reimagined/ Have you ever felt frustrated by the small properties dialog in Notes? Had to create an agent or button to quickly change a field? Searched endlessly for the field you wanted to compare each time you selected a new document? Wished you could just make the damned thing bigger? Luckily, there is a solution – and you probably already have it installed! With the free panagenda Document Properties (Pro) you get the properties dialog you always needed. Big, resizable, full-text searchable. View multiple documents at once or compare them with a diff viewer. Modify any field, and finally have an easy way to handle profile documents for all users. Join HCL Lifetime Ambassador Julian Robichaux to discover how Document Properties can simplify your work and assist you daily when using Domino applications – in the client or the designer. You will never look back! Key takeaways from this session - What Document Properties is, which editions there are, and how you can find it in Notes and Domino Designer - How you can search for and edit any field, compare documents, or CSV export all data - How to find, edit, and even delete profile documents - Which configuration settings are available to customize feature

Easier, Faster, and More Powerful – Notes Document Properties Reimagined

panagenda

The presentation underscores the strategic advantage of treating design systems not just as technical assets but as vital business components that require thoughtful management, robust planning, and strategic alignment with organizational goals. Key Points Covered: - Understanding Design Systems as Business Entities: Conceptualizing design systems as internal business entities can streamline their integration and evolution within a company. - Adoption and Expansion: Elaborating on the importance of tactical adoption across organizational structures, enhancing product suites to cater to user needs and broadening scope to mobile and content authoring solutions. - Data-Driven Development: Utilizing data insights for component development ensures that resources are allocated to create valuable, widely used features. - Financial Modeling for Design Systems: Developing sustainable funding models is crucial for long-term support and success of design systems. - Promoting Internal Buy-In: Stressing on strategies for promoting design systems within the organization to increase engagement and investment from internal stakeholders.

A Business-Centric Approach to Design System Strategy

UXDXConf

IESVE for Early Stage Design and Planning

IES VE

I'm excited to share my latest predictions on how AI, robotics, and other technological advancements will reshape industries in the coming years. The slides explore the exponential growth of computational power, the future of AI and robotics, and their profound impact on various sectors. Why this matters: The success of new products and investments hinges on precise timing and foresight into emerging categories. This deck equips founders, VCs, and industry leaders with insights to align future products with upcoming tech developments. These insights enhance the ability to forecast industry trends, improve market timing, and predict competitor actions. Highlights: ▪ Exponential Growth in Compute: How $1000 will soon buy the computational power of a human brain ▪ Scaling of AI Models: The journey towards beyond human-scale models and intelligent edge computing ▪ Transformative Technologies: From advanced robotics and brain interfaces to automated healthcare and beyond ▪ Future of Work: How automation will redefine jobs and economic structures by 2040 With so many predictions presented here, some will inevitably be wrong or mistimed, especially with potential external disruptions. For instance, a conflict in Taiwan could severely impact global semiconductor production, affecting compute costs and related advancements. Nonetheless, these slides are intended to guide intuition on future technological trends.

Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl

Peter Udo Diehl

Using IESVE for Room Loads Analysis - UK & Ireland

IES VE

Explore the core of Salesforce success in 'Salesforce Adoption – Metrics, Methods, and Motivation.' We will discuss essential metrics, effective methods to drive adoption, and the driving force behind user engagement and explore strategies for onboarding, training, and continuous support that empower users to navigate the platform seamlessly. By leveraging these tools, you can effectively measure adoption against your company’s goals and create an environment where users not only adopt Salesforce but actively contribute to its ongoing success.

Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom

CzechDreamin

Heather Hedden, Senior Consultant at Enterprise Knowledge, presented “Enterprise Knowledge Graphs: The Importance of Semantics” on May 9, 2024, at the annual Data Summit in Boston. In her presentation, Hedden describes the components of an enterprise knowledge graph and provides further insight into the semantic layer – or knowledge model – component, which includes an ontology and controlled vocabularies, such as taxonomies, for controlled metadata. While data experts tend to focus on the graph database components (RDF triple store or a label property graph), Hedden emphasizes they should not overlook the importance of the semantic layer.

Enterprise Knowledge Graphs - Data Summit 2024

Enterprise Knowledge

Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf

FIDO Alliance

We're living the AI revolution and Salesforce is adapting and bring new value to their customers. Einstein products are evolving rapidly and navigating their limitations, language support, and use cases can be challenging. Let's make review of what Einstein product are available currently, what are the capabilities and what can be used for in CEE region and how Rossie.ai can help to learn Salesforce speak Czech. We will explore the Einstein roadmap and I will make a short live demo (based on your vote) of some Einstein feature.

AI revolution and Salesforce, Jiří Karpíšek

CzechDreamin

This our Twelfth semiannual report on the global Cryptocurrency mining industry. Bitcoin is the world’s largest special purpose supercomputer. And it is globally decentralized. Millions of nodes all run the same open-source code to secure the Bitcoin network, create value, and put new transactions onto the distributed ledger. The latest Top500 list has just been announced at the ISC 2024 conference in Hamburg, and once again the Frontier supercomputer with 1.2 Exaflops peak performance is number one on the list. If assigned to SHA-256 hashing, Frontier would provide only the equivalent hash rate of about three cabinets of the latest high-end Bitcoin mining systems, costing less than 0.1% of Frontier’s cost. Michael Saylor, Chairman of MicroStrategy, has pointed out that GPUs are two orders of magnitude slower than the 5-nanometer technology of custom ASICs used for Bitcoin mining today. He makes the point that the Bitcoin network is unassailable by all of the hyperscale computing resources combined in AWS, Google, and Microsoft Azure cloud data centers today.

TopCryptoSupers 12thReport OrionX May2024

Stephen Perrenod

Intro in Product Management - Коротко про професію продакт менеджера

Mark Opanasiuk

Join me in this session where I'll share our journey of building a fully serverless application that flawlessly managed check-ins for an event with a staggering 80 thousand registrations. We'll dive into three key strategies that made this possible. Firstly, by harnessing DynamoDB global tables, we ensured global service availability and data replication across regions, boosting performance and disaster recovery. Next, we'll explore how we seamlessly integrated real-time updates into the app using Appsync subscriptions, making the experience dynamic and engaging for users. Finally, I'll discuss how provisioned concurrency not only improved performance but also kept costs in check, highlighting the cost-effectiveness of serverless architectures. Through these strategies and the inherent scalability of serverless technology, our application effortlessly handled massive user loads without manual intervention. This session is a real world example to the power and efficiency of modern cloud-based solutions in enabling seamless scalability and robust performance with Serverless

How we scaled to 80K users by doing nothing!.pdf

Srushith Repakula

Designing inclusive products is not only a social responsibility but also a business imperative. This talk delves into the journey of creating accessible hardware products that cater to diverse user needs. Key Topics Covered: 1. Introduction to Inclusive Design - Importance of accessibility in product design - Overview of Comcast's commitment to making products accessible to a wide audience 2. Case Study: Xfinity Large Button Voice Remote - Initial challenges and the evolution of the product - User research and feedback that shaped the design - Key features of the final product and their benefits 3. Designing for Diverse Needs - Understanding human-centered design and its historical context - The impact of designing for people with disabilities on overall product quality - Examples from other industries, such as architecture and industrial design 4. Integrating Accessibility from the Beginning - The cost and efficiency benefits of designing for accessibility from the start - The process of embedding accessibility as a core trait rather than an optional feature 5. Real-World Impact and Continuous Improvement - Insights from in-home studies with users having assistive needs - How continuous feedback and iterative design lead to better products - The role of inclusive research and development practices

Designing for Hardware Accessibility at Comcast

UXDXConf

Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...

FIDO Alliance

Recently uploaded (20)

Optimizing NoSQL Performance Through Observability

1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT

FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...

Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf

Integrating Telephony Systems with Salesforce: Insights and Considerations, B...

Speed Wins: From Kafka to APIs in Minutes

Easier, Faster, and More Powerful – Notes Document Properties Reimagined

A Business-Centric Approach to Design System Strategy

IESVE for Early Stage Design and Planning

Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl

Using IESVE for Room Loads Analysis - UK & Ireland

Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom

Enterprise Knowledge Graphs - Data Summit 2024

Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf

AI revolution and Salesforce, Jiří Karpíšek

TopCryptoSupers 12thReport OrionX May2024

Intro in Product Management - Коротко про професію продакт менеджера

How we scaled to 80K users by doing nothing!.pdf

Designing for Hardware Accessibility at Comcast

Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...

Ingesting Data from Kafka to JDBC with Transformation and Enrichment

1. Meetup Series on Hadoop Ingestion Made Easy Part 3: Ingestion of Unbounded Data (Kafka -> Enrich -> Transform -> JDBC) Dr. Sandeep Deshmukh Committer@Apache Apex Engineer@DataTorrent Yogi Devendra Committer@Apache Apex Engineer@DataTorrent

2. 2 What is Ingestion Data ingestion • A process of obtaining, importing, and analyzing data for later use or storage in a database Big Data Ingestion • Discovering the data sources • Importing the data • Processing data to produce intermediate data • Sending data out to durable data stores ETL + Big Data => Data ingestion

3. 3 Requirements • Read from stream • Kafka • JMS • JDBC • …. • Enrich • Transform • Store • JDBC

4. 4 Use Case Kafka Reader Enrich Transform JDBC Output ● Kafka 0.9 ● Multi-topic ● Multi-partition ● Enrich State using City ● Mark a transaction as HIGH or LOW value ● Store in DB Parallel Partition can scale to large volume

5. 5 5 Let us walk-through the code and appreciate how simple it is to use Apex ● Code walk-through for an data ingestion pipeline using ○ Operators/Modules in Malhar ○ Drag-n-drop user interface ● Validate ○ Partitioning ○ Scalability Let us create and run the app

6. 6 Resources • http://apex.apache.org/ • Learn more: http://apex.apache.org/docs.html • Subscribe - http://apex.apache.org/community.html • Download - http://apex.apache.org/downloads.html • Follow @ApacheApex - https://twitter.com/apacheapex • Meetups – http://www.meetup.com/pro/apacheapex/ • More examples: https://github.com/DataTorrent/examples • Slideshare: http://www.slideshare.net/ApacheApex/presentations • https://www.youtube.com/results?search_query=apache+apex • Free Enterprise License for Startups - https://www.datatorrent.com/product/startup-accelerator/

7. Q&A 7

Ingesting Data from Kafka to JDBC with Transformation and Enrichment

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Ingesting Data from Kafka to JDBC with Transformation and Enrichment

Similar to Ingesting Data from Kafka to JDBC with Transformation and Enrichment (20)

More from Apache Apex

More from Apache Apex (8)

Recently uploaded

Recently uploaded (20)

Ingesting Data from Kafka to JDBC with Transformation and Enrichment