August Flink Community Update
•Download as PPTX, PDF•
1 like•506 views
This document summarizes updates from the August 2015 Berlin Apache Flink Meetup. It discusses that Apache Flink now has a new committer, discussions have started for the 0.9.1 release, and Flink is gaining popularity with over 1000 Twitter followers and 500 GitHub stars. It also provides information on improvements now in the master version including the Gelly Scala API and a streaming connector for Elastic Search. Upcoming events are noted including Flink meetups in Washington DC, Belgium, and the Flink Talks schedule being announced for ApacheCon in Budapest.
Report
Share
Report
Share
Recommended
Berlin Apache Flink Meetup May 2015, Community Update
This document summarizes the May 2015 community update for Apache Flink. Key updates include a pull request to integrate Flink with Zeppelin, plans to fix issues for the upcoming 0.9 release, and work on the Gelly graph processing API. The document also mentions new meetup groups in Stockholm and Bay Area, frontpage redesign of the Flink website, and that Flink now supports exactly-once streaming processing with Kafka sources in the 0.9 snapshot release.
Flink September 2015 Community Update
This document summarizes the September 2015 community update for Apache Flink. Key highlights include Matthias Sax joining as a new committer, the release of version 0.9.1, and discussions starting around releasing version 0.10. Version 0.10 will include improvements to window operators, memory allocation, and new connectors to HDFS, Elasticsearch, and Kafka. The community held various meetups and presentations around the world in September and Flink was recognized as one of the best open source big data tools.
Flink Cummunity Update July (Berlin Meetup)
This document summarizes an Apache Flink meetup that took place in July 2015. It discusses recent developments with Apache Flink, including the addition of a new JobManager dashboard, integration with Apache SAMOA, and new features page. The document also mentions upcoming Flink meetups and trainings, as well as announcing that registration is open for the Flink Forward conference in Berlin in December 2015.
Apache Flink Community Updates November 2016 @ Berlin Meetup
This document provides a summary of the Flink community update presented at the Berlin Flink Meetup on November 29, 2016. The agenda included a Flink community update discussing developments since May 2016, including the upcoming 1.2 release and work on the 1.3 release. Updates were provided on the Flink developer community growth on GitHub, a new Flink book, and data Artisans' Flink platform launch. Flink adoption by other vendors like Lightbend and on Amazon EMR was highlighted. Details from Flink Forward 2016 like the number of attendees and sessions were shared. The presentation concluded with metrics showing the growing global Flink meetup community and GitHub activity to quantify the expanding Flink community.
Robust Stream Processing with Apache Flink
Slides from my OSCON 2016 talk. Not super useful without the accompanying video and code. I'll post that when it's available.
Kostas Tzoumas_Stephan Ewen - Keynote -The maturing data streaming ecosystem ...
http://flink-forward.org/kb_sessions/keynote-tba-2/
The past 12 months saw the data streaming ecosystem mature and grow tremendously with new open source projects and products being offered in the market, and more large-scale production applications of streaming data. It is now understood that streaming data is not a fad, but a growing industry that is here to stay.
Apache Flink was one of the pioneering communities advocating that stream processing is a great fit for the continuous nature of data production, and that batch processing can be seen and efficiently performed as a special case of stream processing. Flink saw tremendous growth since the last Flink Forward conference, with the project boasting now more than 200 contributors from several companies, several production installations and broad adoption.
In this talk, we discuss several large-scale stream processing use cases that we see at data Artisans. Additionally, we discuss what this accelerated growth means for Flink, how we can sustain this growth moving forward, as well as a vision for the next big directions in Flink.
Riding the Streaming Wave DIY style
Kafka Connect allows developers to easily build plugins that integrate data from various sources and sinks. The document discusses how to develop Kafka Connect plugins using Confluent Open Source tools. It recommends using the Confluent CLI for local development and testing due to features like classloading isolation. Debugging plugins is also made simple by exporting environment variables and attaching a remote debugger. Once developed, plugins can be packaged and published for use in Kafka Connect.
Flink Community Update February 2015
Flink is an open source stream processing framework. The February 2015 Flink community update announced a bugfix release, a new committer, Flink's participation in Google Summer of Code, and new features in development including a graph API, expression API, and access to secured YARN clusters and HDFS. The update also provided links to blog posts about Flink and a call for testing of a new Python API pull request.
Recommended
Berlin Apache Flink Meetup May 2015, Community Update
This document summarizes the May 2015 community update for Apache Flink. Key updates include a pull request to integrate Flink with Zeppelin, plans to fix issues for the upcoming 0.9 release, and work on the Gelly graph processing API. The document also mentions new meetup groups in Stockholm and Bay Area, frontpage redesign of the Flink website, and that Flink now supports exactly-once streaming processing with Kafka sources in the 0.9 snapshot release.
Flink September 2015 Community Update
This document summarizes the September 2015 community update for Apache Flink. Key highlights include Matthias Sax joining as a new committer, the release of version 0.9.1, and discussions starting around releasing version 0.10. Version 0.10 will include improvements to window operators, memory allocation, and new connectors to HDFS, Elasticsearch, and Kafka. The community held various meetups and presentations around the world in September and Flink was recognized as one of the best open source big data tools.
Flink Cummunity Update July (Berlin Meetup)
This document summarizes an Apache Flink meetup that took place in July 2015. It discusses recent developments with Apache Flink, including the addition of a new JobManager dashboard, integration with Apache SAMOA, and new features page. The document also mentions upcoming Flink meetups and trainings, as well as announcing that registration is open for the Flink Forward conference in Berlin in December 2015.
Apache Flink Community Updates November 2016 @ Berlin Meetup
This document provides a summary of the Flink community update presented at the Berlin Flink Meetup on November 29, 2016. The agenda included a Flink community update discussing developments since May 2016, including the upcoming 1.2 release and work on the 1.3 release. Updates were provided on the Flink developer community growth on GitHub, a new Flink book, and data Artisans' Flink platform launch. Flink adoption by other vendors like Lightbend and on Amazon EMR was highlighted. Details from Flink Forward 2016 like the number of attendees and sessions were shared. The presentation concluded with metrics showing the growing global Flink meetup community and GitHub activity to quantify the expanding Flink community.
Robust Stream Processing with Apache Flink
Slides from my OSCON 2016 talk. Not super useful without the accompanying video and code. I'll post that when it's available.
Kostas Tzoumas_Stephan Ewen - Keynote -The maturing data streaming ecosystem ...
http://flink-forward.org/kb_sessions/keynote-tba-2/
The past 12 months saw the data streaming ecosystem mature and grow tremendously with new open source projects and products being offered in the market, and more large-scale production applications of streaming data. It is now understood that streaming data is not a fad, but a growing industry that is here to stay.
Apache Flink was one of the pioneering communities advocating that stream processing is a great fit for the continuous nature of data production, and that batch processing can be seen and efficiently performed as a special case of stream processing. Flink saw tremendous growth since the last Flink Forward conference, with the project boasting now more than 200 contributors from several companies, several production installations and broad adoption.
In this talk, we discuss several large-scale stream processing use cases that we see at data Artisans. Additionally, we discuss what this accelerated growth means for Flink, how we can sustain this growth moving forward, as well as a vision for the next big directions in Flink.
Riding the Streaming Wave DIY style
Kafka Connect allows developers to easily build plugins that integrate data from various sources and sinks. The document discusses how to develop Kafka Connect plugins using Confluent Open Source tools. It recommends using the Confluent CLI for local development and testing due to features like classloading isolation. Debugging plugins is also made simple by exporting environment variables and attaching a remote debugger. Once developed, plugins can be packaged and published for use in Kafka Connect.
Flink Community Update February 2015
Flink is an open source stream processing framework. The February 2015 Flink community update announced a bugfix release, a new committer, Flink's participation in Google Summer of Code, and new features in development including a graph API, expression API, and access to secured YARN clusters and HDFS. The update also provided links to blog posts about Flink and a call for testing of a new Python API pull request.
Counting Elements in Streams
Uses the example of correct, high-througput, grouping and counting of streaming events as a backdrop for exploring the state-of-the art features of Apache Flink
Stateful, Stateless and Serverless - Running Apache Kafka® on Kubernetes
This document provides an overview of a webinar on Kubernetes operators presented by Joe Beda. It discusses:
- Kubernetes allows for developer and operator productivity through improved workflows and better resource efficiency.
- Operators are domain-specific controllers that manage both a software application and related Kubernetes objects, and help automate common operational tasks through code.
- The Kafka operator example shows how an operator can handle tasks like identity, discovery, storage and logging/metrics for Kafka deployments on Kubernetes through a custom resource definition.
Deploying and Operating KSQL
This document provides an overview of deploying and operating KSQL. It introduces Nick Dearden and Hojjat Jafarpour who work on KSQL at Confluent. The agenda includes discussing deployment, configuration, scaling, and monitoring of KSQL. Specific topics covered are getting started with KSQL, starting the KSQL server, connecting to Kafka and Schema Registry, using the KSQL CLI, deployment patterns and log files. The presentation also demonstrates viewing metrics for KSQL servers, queries, input and output topics through JMX.
Flink 1.0-slides
Flink 1.0 includes major features such as out of core state using RocksDB, savepoints for upgrading jobs and versions, and a CEP library for pattern detection. Savepoints allow taking snapshots of streaming jobs for code upgrades, Flink version upgrades, and testing through time travel. Flink 1.0 initiates backwards compatibility and pushes production streaming further through these features.
Getting started with apache flink streaming api
This document provides an overview of Apache Flink's streaming API. It first defines streaming data and common examples. It then discusses typical streaming data architectures involving data sources, collection, ingestion, processing, storage and visualization. The document also summarizes Flink's high-level design and components. Finally, it outlines the typical workflow for a Flink DataStream, including creating an execution environment, adding a source, creating a DataStream, partitioning, defining windows, applying business logic, and sending results to a sink.
Kafka Summit NYC 2017 - The Source of Truth: Why the New York Times Stores Ev...
The document discusses how The New York Times stores all published content in Apache Kafka. It describes how the Times previously used an API-based architecture with different data schemas for each service, which led to inconsistencies. The new approach establishes a single source of truth (Kafka) with a normalized schema. All content is stored as a log in Kafka, then denormalized logs and special-purpose logs are replicated from the main log. This allows content to be efficiently accessed and ensures consistency across services. The implementation uses Google Cloud infrastructure and passive replication across regions. Key lessons are that log-based architectures are new, managed Kafka would help, and it will take time to transition all services.
Apache Flink Online Training
http://www.learntek.org/product/apache-flink/
Apache Flink is an open source stream processing framework developed by the Apache Software Foundation. The core of Apache Flink is a distributed streaming dataflow engine written in Java and Scala. Apache Flink’s dataflow programming model provides event-at-a-time processing on both finite and infinite datasets. At a basic level, Flink programs consist of streams and transformations. Conceptually, a stream is a (potentially never-ending) flow of data records, and a transformation is an operation that takes one or more streams as input, and produces one or more output streams as a result. Programs can be written in Java, Scala, Python, and SQL and are automatically compiled and optimized into dataflow programs that are executed in a cluster or cloud environment.
http://www.learntek.org
Learntek is global online training provider on Big Data Analytics, Hadoop, Machine Learning, Deep Learning, IOT, AI, Cloud Technology, DEVOPS, Digital Marketing and other IT and Management courses. We are dedicated to designing, developing and implementing training programs for students, corporate employees and business professional.
Flink Forward Berlin 2017: Till Rohrmann - From Apache Flink 1.3 to 1.4
The document summarizes updates and new features in Apache Flink versions 1.3 to 1.4, and previews what is planned for 1.5 and beyond. Key points include: improved APIs, side outputs and state handling in 1.3; event-driven I/O, flow control and deployment changes in 1.4; and planned additions of side inputs, state management evolution, and state replication in 1.5. The document encourages attendees to learn more about Flink's internals by attending related talks at the event.
Apache flink
The document outlines the topics that will be covered in an Apache Flink online training, including: what Apache Flink is; why use Apache Flink; its architecture, features, and deployment; its streaming, batch processing, and table APIs; complex event processing; graph processing; and integration with Hadoop. The training will cover Apache Flink's stream processing engine, fault tolerance, state management, and support for stream, batch, and iterative processing using its dataflow model.
Apache flink
https://www.learntek.org/apache-flink/
https://www.learntek.org/
Learntek is global online training provider on Big Data Analytics, Hadoop, Machine Learning, Deep Learning, IOT, AI, Cloud Technology, DEVOPS, Digital Marketing and other IT and Management courses.
Aljoscha Krettek - The Future of Apache Flink
http://flink-forward.org/kb_sessions/the-future-of-apache-flinktm/
In this session we will first have a look at the current state of Apache Flink before diving into some of the upcoming features that are either already in development or still in the design phase. Some of the features currently in development that we are going to cover are: – Dynamic Scaling: Adapting a running program to changing workloads. – Queryable State: External querying of internal Flink state. This has the power to replace key/value stores by turning Flink into a key value store that allows for up to date querying of results. – Side Inputs: Having additional data that evolves over time as input to a stream operation. For the glimpse at the far-off future of Apache Flink™ we dare not make any predictions yet. In the session we will look at the latest whisperings and see what the community is currently thinking up as solutions to existing problems and predicted future challenges in the stream processing space.
OSMC 2021 | Advanced MySQL optimization and troubleshooting using PMM 2
Optimizing MySQL performance and troubleshooting MySQL problems are two of the most critical and challenging tasks for MySQL DBA’s. The databases powering your applications need to be able to handle changing traffic workloads while remaining responsive and stable so that you can deliver an excellent user experience. Further, DBA’s are also expected to find cost-efficient means of solving these issues. In this presentation, we will demonstrate the advanced options of PMM version 2 that enables you to solve these challenges, which is built on free and open-source software. We will look at specific, common MySQL problems and review them.
Javier Lopez_Mihail Vieru - Flink in Zalando's World of Microservices - Flink...
http://flink-forward.org/kb_sessions/flink-in-zalandos-world-of-microservices/
In this talk we present Zalando’s microservices architecture, introduce Saiki – our next generation data integration and distribution platform on AWS and show how we employ stream processing with Apache Flink for near-real time business intelligence.
Zalando is one of the largest online fashion retailers in Europe. In order to secure our future growth and remain competitive in this dynamic market, we are transitioning from a monolithic to a microservices architecture and from a hierarchical to an agile organization.
We first have a look at how business intelligence processes have been working inside Zalando for the last years and present our current approach – Saiki. It is a scalable, cloud-based data integration and distribution infrastructure that makes data from our many microservices readily available for analytical teams.
We no longer live in a world of static data sets, but are instead confronted with endless streams of events that constantly inform us about relevant happenings from all over the enterprise. The processing of these event streams enables us to do near-real time business intelligence. In this context we have evaluated Apache Flink vs. Apache Spark in order to choose the right stream processing framework. Given our requirements, we decided to use Flink as part of our technology stack, alongside with Kafka and Elasticsearch.
With these technologies we are currently working on two use cases: a near real-time business process monitoring solution and streaming ETL.
Monitoring our business processes enables us to check if technically the Zalando platform works. It also helps us analyze data streams on the fly, e.g. order velocities, delivery velocities and to control service level agreements.
On the other hand, streaming ETL is used to relinquish resources from our relational data warehouse, as it struggles with increasingly high loads. In addition to that, it also reduces the latency and facilitates the platform scalability.
Finally, we have an outlook on our future use cases, e.g. near-real time sales and price monitoring. Another aspect to be addressed is to lower the entry barrier of stream processing for our colleagues coming from a relational database background.
Apache Kafka & Kafka Connectを に使ったデータ連携パターン(改めETLの実装)
This document discusses Apache Kafka and Kafka Connect. It provides an overview of Kafka Connect and how it can be used for ETL processes. Kafka Connect allows data to be exported from or imported to Kafka and integrated with other systems through customizable connectors. The document describes how to run Kafka Connect in standalone and distributed modes and highlights some popular connectors available for integrating Kafka with other data sources and sinks.
Elk meetup boston - logz.io
This document summarizes an ELK meetup that took place on March 2nd 2015. It discusses using ELK for log processing, in public clouds like AWS, and activities like kite surfing. The document also provides information on Wind Analytics and their next steps, monitoring large AWS environments, implementing ELK with the right architecture, and Logz.io which provides an ELK as a service solution and insights. It includes demos of Logz.io's architecture and log processing. The meetup concluded with information on job opportunities at Logz.io.
Flink Forward Berlin 2017: Hao Wu - Large Scale User Behavior Analytics by Flink
We are HanSight, a leading security startup based in China. We provide solutions for enterprise cybersecurity with a main focus on User Behavior Analytics(UBA). Typical UBA deployment in large scale enterprise needs to handle 10k+ unique users over 10+ dimensions. Real-time analysis and detection on that scale of data has become a must have functionality yet a challenge for traditional security solutions. Most of the products on the market usually struggles with high throughput(100k TPS) and real-time analysis accuracy. With Flink’s streaming nature, we are able to present a next generation UBA system that tackles the large scale real-time data analysis challenge. Basically, Flink serves as a CEP engine processing data in a streaming fashion. And UBA engine (anomaly detection algorithms, rule engine) runs on top of Flink to achieve dynamic ETL rule configuration and hot deployment. Also we provide a stunning UI design for rule configuration, incident response and system monitoring.
Zentral QueryCon 2018
Learn about core functions and architecture of Zentral. Zentral is a open source hub to process event streams from osquery and other sources into the ElasticStack. Besides support for distinct osquery features like file carving, Zentral provides numerous integrations for inventory acquisition and alerting.
Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...
In Zalando's microservice architecture, each service continuously generates streams of events for the purposes of inter-service communication or data integration. Some of these events describe business processes, e.g. a customer has placed an order or a parcel has been shipped. Out of this, the need to materialize event streams from the central event bus into persistent cloud storage evolved. The temporarily persisted data is then integrated into our relational data warehouse. In this talk we present a materialization engine backed by Apache Flink. We show how we employ Flink’s RESTful API, custom accumulators and stoppable sources to provide another API abstraction layer for deploying, monitoring and controlling our materialization jobs. Our jobs compact event streams depending on event properties and transform their complex JSON structures into flat files for easier integration into the data warehouse.
Apache flink 1.7 and Beyond
This document discusses Apache Flink version 1.7 and beyond. It summarizes key features of Flink 1.7 including contributions from 112 contributors and over 1,000 commits. It also discusses upcoming features in Flink 1.8 such as support for state schema evolution, dynamic scaling, unifying batch and streaming, an extendable scheduler, and end-to-end SQL-only pipelines. The document encourages participation in the Flink community.
Introduction to Streaming Distributed Processing with Storm
Contact:
https://www.linkedin.com/in/brandonjobrien
@hakczar
Introducing streaming data concepts, Storm cluster architecture, Storm topology architecture, and demonstrate working example of a WordCount topology for SIGKDD Seattle chapter meetup.
Presented by Brandon O'Brien
Code example: https://github.com/OpenDataMining/brandonobrien
Meetup: http://www.meetup.com/seattlesigkdd/events/222955114/
Bay Area Apache Flink Meetup Community Update August 2015
This document summarizes updates from Apache Flink community meeting in August 2015. Key points include: new project management committee and committer members joined Flink, discussions started for a new 0.9.1 release, and Flink is gaining popularity with over 1000 Twitter followers and 500 GitHub stars. Updates were provided on new features in development like a new JobManager dashboard, Gelly Scala API, and improvements to YARN integration. Upcoming events were also announced including Flink training sessions and new user group meetups forming in various cities.
Flink Community Update 2015 June
This document summarizes Apache Flink community updates from June 2015. It discusses the 0.9.0 release of Apache Flink, an open source platform for scalable batch and stream data processing. Key points include the addition of two new committers, blog posts and workshops promoting Flink, and various conference and meetup talks about Flink occurring that month. It encourages registration for the Flink Forward conference in October 2015.
More Related Content
What's hot
Counting Elements in Streams
Uses the example of correct, high-througput, grouping and counting of streaming events as a backdrop for exploring the state-of-the art features of Apache Flink
Stateful, Stateless and Serverless - Running Apache Kafka® on Kubernetes
This document provides an overview of a webinar on Kubernetes operators presented by Joe Beda. It discusses:
- Kubernetes allows for developer and operator productivity through improved workflows and better resource efficiency.
- Operators are domain-specific controllers that manage both a software application and related Kubernetes objects, and help automate common operational tasks through code.
- The Kafka operator example shows how an operator can handle tasks like identity, discovery, storage and logging/metrics for Kafka deployments on Kubernetes through a custom resource definition.
Deploying and Operating KSQL
This document provides an overview of deploying and operating KSQL. It introduces Nick Dearden and Hojjat Jafarpour who work on KSQL at Confluent. The agenda includes discussing deployment, configuration, scaling, and monitoring of KSQL. Specific topics covered are getting started with KSQL, starting the KSQL server, connecting to Kafka and Schema Registry, using the KSQL CLI, deployment patterns and log files. The presentation also demonstrates viewing metrics for KSQL servers, queries, input and output topics through JMX.
Flink 1.0-slides
Flink 1.0 includes major features such as out of core state using RocksDB, savepoints for upgrading jobs and versions, and a CEP library for pattern detection. Savepoints allow taking snapshots of streaming jobs for code upgrades, Flink version upgrades, and testing through time travel. Flink 1.0 initiates backwards compatibility and pushes production streaming further through these features.
Getting started with apache flink streaming api
This document provides an overview of Apache Flink's streaming API. It first defines streaming data and common examples. It then discusses typical streaming data architectures involving data sources, collection, ingestion, processing, storage and visualization. The document also summarizes Flink's high-level design and components. Finally, it outlines the typical workflow for a Flink DataStream, including creating an execution environment, adding a source, creating a DataStream, partitioning, defining windows, applying business logic, and sending results to a sink.
Kafka Summit NYC 2017 - The Source of Truth: Why the New York Times Stores Ev...
The document discusses how The New York Times stores all published content in Apache Kafka. It describes how the Times previously used an API-based architecture with different data schemas for each service, which led to inconsistencies. The new approach establishes a single source of truth (Kafka) with a normalized schema. All content is stored as a log in Kafka, then denormalized logs and special-purpose logs are replicated from the main log. This allows content to be efficiently accessed and ensures consistency across services. The implementation uses Google Cloud infrastructure and passive replication across regions. Key lessons are that log-based architectures are new, managed Kafka would help, and it will take time to transition all services.
Apache Flink Online Training
http://www.learntek.org/product/apache-flink/
Apache Flink is an open source stream processing framework developed by the Apache Software Foundation. The core of Apache Flink is a distributed streaming dataflow engine written in Java and Scala. Apache Flink’s dataflow programming model provides event-at-a-time processing on both finite and infinite datasets. At a basic level, Flink programs consist of streams and transformations. Conceptually, a stream is a (potentially never-ending) flow of data records, and a transformation is an operation that takes one or more streams as input, and produces one or more output streams as a result. Programs can be written in Java, Scala, Python, and SQL and are automatically compiled and optimized into dataflow programs that are executed in a cluster or cloud environment.
http://www.learntek.org
Learntek is global online training provider on Big Data Analytics, Hadoop, Machine Learning, Deep Learning, IOT, AI, Cloud Technology, DEVOPS, Digital Marketing and other IT and Management courses. We are dedicated to designing, developing and implementing training programs for students, corporate employees and business professional.
Flink Forward Berlin 2017: Till Rohrmann - From Apache Flink 1.3 to 1.4
The document summarizes updates and new features in Apache Flink versions 1.3 to 1.4, and previews what is planned for 1.5 and beyond. Key points include: improved APIs, side outputs and state handling in 1.3; event-driven I/O, flow control and deployment changes in 1.4; and planned additions of side inputs, state management evolution, and state replication in 1.5. The document encourages attendees to learn more about Flink's internals by attending related talks at the event.
Apache flink
The document outlines the topics that will be covered in an Apache Flink online training, including: what Apache Flink is; why use Apache Flink; its architecture, features, and deployment; its streaming, batch processing, and table APIs; complex event processing; graph processing; and integration with Hadoop. The training will cover Apache Flink's stream processing engine, fault tolerance, state management, and support for stream, batch, and iterative processing using its dataflow model.
Apache flink
https://www.learntek.org/apache-flink/
https://www.learntek.org/
Learntek is global online training provider on Big Data Analytics, Hadoop, Machine Learning, Deep Learning, IOT, AI, Cloud Technology, DEVOPS, Digital Marketing and other IT and Management courses.
Aljoscha Krettek - The Future of Apache Flink
http://flink-forward.org/kb_sessions/the-future-of-apache-flinktm/
In this session we will first have a look at the current state of Apache Flink before diving into some of the upcoming features that are either already in development or still in the design phase. Some of the features currently in development that we are going to cover are: – Dynamic Scaling: Adapting a running program to changing workloads. – Queryable State: External querying of internal Flink state. This has the power to replace key/value stores by turning Flink into a key value store that allows for up to date querying of results. – Side Inputs: Having additional data that evolves over time as input to a stream operation. For the glimpse at the far-off future of Apache Flink™ we dare not make any predictions yet. In the session we will look at the latest whisperings and see what the community is currently thinking up as solutions to existing problems and predicted future challenges in the stream processing space.
OSMC 2021 | Advanced MySQL optimization and troubleshooting using PMM 2
Optimizing MySQL performance and troubleshooting MySQL problems are two of the most critical and challenging tasks for MySQL DBA’s. The databases powering your applications need to be able to handle changing traffic workloads while remaining responsive and stable so that you can deliver an excellent user experience. Further, DBA’s are also expected to find cost-efficient means of solving these issues. In this presentation, we will demonstrate the advanced options of PMM version 2 that enables you to solve these challenges, which is built on free and open-source software. We will look at specific, common MySQL problems and review them.
Javier Lopez_Mihail Vieru - Flink in Zalando's World of Microservices - Flink...
http://flink-forward.org/kb_sessions/flink-in-zalandos-world-of-microservices/
In this talk we present Zalando’s microservices architecture, introduce Saiki – our next generation data integration and distribution platform on AWS and show how we employ stream processing with Apache Flink for near-real time business intelligence.
Zalando is one of the largest online fashion retailers in Europe. In order to secure our future growth and remain competitive in this dynamic market, we are transitioning from a monolithic to a microservices architecture and from a hierarchical to an agile organization.
We first have a look at how business intelligence processes have been working inside Zalando for the last years and present our current approach – Saiki. It is a scalable, cloud-based data integration and distribution infrastructure that makes data from our many microservices readily available for analytical teams.
We no longer live in a world of static data sets, but are instead confronted with endless streams of events that constantly inform us about relevant happenings from all over the enterprise. The processing of these event streams enables us to do near-real time business intelligence. In this context we have evaluated Apache Flink vs. Apache Spark in order to choose the right stream processing framework. Given our requirements, we decided to use Flink as part of our technology stack, alongside with Kafka and Elasticsearch.
With these technologies we are currently working on two use cases: a near real-time business process monitoring solution and streaming ETL.
Monitoring our business processes enables us to check if technically the Zalando platform works. It also helps us analyze data streams on the fly, e.g. order velocities, delivery velocities and to control service level agreements.
On the other hand, streaming ETL is used to relinquish resources from our relational data warehouse, as it struggles with increasingly high loads. In addition to that, it also reduces the latency and facilitates the platform scalability.
Finally, we have an outlook on our future use cases, e.g. near-real time sales and price monitoring. Another aspect to be addressed is to lower the entry barrier of stream processing for our colleagues coming from a relational database background.
Apache Kafka & Kafka Connectを に使ったデータ連携パターン(改めETLの実装)
This document discusses Apache Kafka and Kafka Connect. It provides an overview of Kafka Connect and how it can be used for ETL processes. Kafka Connect allows data to be exported from or imported to Kafka and integrated with other systems through customizable connectors. The document describes how to run Kafka Connect in standalone and distributed modes and highlights some popular connectors available for integrating Kafka with other data sources and sinks.
Elk meetup boston - logz.io
This document summarizes an ELK meetup that took place on March 2nd 2015. It discusses using ELK for log processing, in public clouds like AWS, and activities like kite surfing. The document also provides information on Wind Analytics and their next steps, monitoring large AWS environments, implementing ELK with the right architecture, and Logz.io which provides an ELK as a service solution and insights. It includes demos of Logz.io's architecture and log processing. The meetup concluded with information on job opportunities at Logz.io.
Flink Forward Berlin 2017: Hao Wu - Large Scale User Behavior Analytics by Flink
We are HanSight, a leading security startup based in China. We provide solutions for enterprise cybersecurity with a main focus on User Behavior Analytics(UBA). Typical UBA deployment in large scale enterprise needs to handle 10k+ unique users over 10+ dimensions. Real-time analysis and detection on that scale of data has become a must have functionality yet a challenge for traditional security solutions. Most of the products on the market usually struggles with high throughput(100k TPS) and real-time analysis accuracy. With Flink’s streaming nature, we are able to present a next generation UBA system that tackles the large scale real-time data analysis challenge. Basically, Flink serves as a CEP engine processing data in a streaming fashion. And UBA engine (anomaly detection algorithms, rule engine) runs on top of Flink to achieve dynamic ETL rule configuration and hot deployment. Also we provide a stunning UI design for rule configuration, incident response and system monitoring.
Zentral QueryCon 2018
Learn about core functions and architecture of Zentral. Zentral is a open source hub to process event streams from osquery and other sources into the ElasticStack. Besides support for distinct osquery features like file carving, Zentral provides numerous integrations for inventory acquisition and alerting.
Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...
In Zalando's microservice architecture, each service continuously generates streams of events for the purposes of inter-service communication or data integration. Some of these events describe business processes, e.g. a customer has placed an order or a parcel has been shipped. Out of this, the need to materialize event streams from the central event bus into persistent cloud storage evolved. The temporarily persisted data is then integrated into our relational data warehouse. In this talk we present a materialization engine backed by Apache Flink. We show how we employ Flink’s RESTful API, custom accumulators and stoppable sources to provide another API abstraction layer for deploying, monitoring and controlling our materialization jobs. Our jobs compact event streams depending on event properties and transform their complex JSON structures into flat files for easier integration into the data warehouse.
Apache flink 1.7 and Beyond
This document discusses Apache Flink version 1.7 and beyond. It summarizes key features of Flink 1.7 including contributions from 112 contributors and over 1,000 commits. It also discusses upcoming features in Flink 1.8 such as support for state schema evolution, dynamic scaling, unifying batch and streaming, an extendable scheduler, and end-to-end SQL-only pipelines. The document encourages participation in the Flink community.
Introduction to Streaming Distributed Processing with Storm
Contact:
https://www.linkedin.com/in/brandonjobrien
@hakczar
Introducing streaming data concepts, Storm cluster architecture, Storm topology architecture, and demonstrate working example of a WordCount topology for SIGKDD Seattle chapter meetup.
Presented by Brandon O'Brien
Code example: https://github.com/OpenDataMining/brandonobrien
Meetup: http://www.meetup.com/seattlesigkdd/events/222955114/
What's hot (20)
Stateful, Stateless and Serverless - Running Apache Kafka® on Kubernetes
Stateful, Stateless and Serverless - Running Apache Kafka® on Kubernetes
Kafka Summit NYC 2017 - The Source of Truth: Why the New York Times Stores Ev...
Kafka Summit NYC 2017 - The Source of Truth: Why the New York Times Stores Ev...
Flink Forward Berlin 2017: Till Rohrmann - From Apache Flink 1.3 to 1.4
Flink Forward Berlin 2017: Till Rohrmann - From Apache Flink 1.3 to 1.4
OSMC 2021 | Advanced MySQL optimization and troubleshooting using PMM 2
OSMC 2021 | Advanced MySQL optimization and troubleshooting using PMM 2
Javier Lopez_Mihail Vieru - Flink in Zalando's World of Microservices - Flink...
Javier Lopez_Mihail Vieru - Flink in Zalando's World of Microservices - Flink...
Apache Kafka & Kafka Connectを に使ったデータ連携パターン(改めETLの実装)
Apache Kafka & Kafka Connectを に使ったデータ連携パターン(改めETLの実装)
Flink Forward Berlin 2017: Hao Wu - Large Scale User Behavior Analytics by Flink
Flink Forward Berlin 2017: Hao Wu - Large Scale User Behavior Analytics by Flink
Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...
Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...
Introduction to Streaming Distributed Processing with Storm
Introduction to Streaming Distributed Processing with Storm
Similar to August Flink Community Update
Bay Area Apache Flink Meetup Community Update August 2015
This document summarizes updates from Apache Flink community meeting in August 2015. Key points include: new project management committee and committer members joined Flink, discussions started for a new 0.9.1 release, and Flink is gaining popularity with over 1000 Twitter followers and 500 GitHub stars. Updates were provided on new features in development like a new JobManager dashboard, Gelly Scala API, and improvements to YARN integration. Upcoming events were also announced including Flink training sessions and new user group meetups forming in various cities.
Flink Community Update 2015 June
This document summarizes Apache Flink community updates from June 2015. It discusses the 0.9.0 release of Apache Flink, an open source platform for scalable batch and stream data processing. Key points include the addition of two new committers, blog posts and workshops promoting Flink, and various conference and meetup talks about Flink occurring that month. It encourages registration for the Flink Forward conference in October 2015.
Apache Flink First Half of 2015 Community Update
Flink has graduated from an Apache incubator project to a top-level project, attracting many new contributors. Recent releases have added features like a Table API, Gelly graph processing, and integrations with SAMOA machine learning and Google Dataflow. The talk outlines Flink's history and recent developments from 2014 to mid-2015, including three students working on Flink over the summer, and announces the first Flink Forward conference in October 2015.
Flink Community Update December 2015: Year in Review
This document summarizes the Berlin Apache Flink Meetup #12 that took place in December 2015. It discusses the key releases and improvements to Flink in 2015, including the release of versions 0.10.0 and 0.10.1, and new features that were added to the master branch, such as improvements to the Kafka connector. It also lists pending pull requests, recommended reading, and provides statistics on Flink's growth in 2015 in terms of GitHub activity, meetup groups, organizations at Flink Forward, and articles published.
Flink Community Update April 2015
Monthly Apache Flink Update, featuring the latest highlights from the community.
Presented at the Apache Flink Meetup in Berlin
Overview of Apache Flink: Next-Gen Big Data Analytics Framework
These are the slides of my talk on June 30, 2015 at the first event of the Chicago Apache Flink meetup. Although most of the current buzz is about Apache Spark, the talk shows how Apache Flink offers the only hybrid open source (Real-Time Streaming + Batch) distributed data processing engine supporting many use cases: Real-Time stream processing, machine learning at scale, graph analytics and batch processing.
In these slides, you will find answers to the following questions: What is Apache Flink stack and how it fits into the Big Data ecosystem? How Apache Flink integrates with Apache Hadoop and other open source tools for data input and output as well as deployment? What is the architecture of Apache Flink? What are the different execution modes of Apache Flink? Why Apache Flink is an alternative to Apache Hadoop MapReduce, Apache Storm and Apache Spark? Who is using Apache Flink? Where to learn more about Apache Flink?
Robert Metzger - Apache Flink Community Updates November 2016 @ Berlin Meetup
This document provides a summary of the Flink community update presented at the Berlin Flink Meetup on November 29, 2016. The agenda included a Flink community update by Kostas Tzoumas and Robert Metzger from data Artisans, a look at Apache Flink 1.2 by Stefan Richter from data Artisans, and a presentation on real-time analytics as a service at King by Gyula Fóra. The community update discussed the progress made since May 2016, including the upcoming 1.2 release and work on the 1.3 release through the FLIP process. Metrics were presented showing growth in the Flink community on GitHub and in local meetup groups around the world. Attendees were
Apache flink
Apache Flink Training
https://www.learntek.org/apache-flink/
https://www.learntek.org/
Learntek is global online training provider on Big Data Analytics, Hadoop, Machine Learning, Deep Learning, IOT, AI, Cloud Technology, DEVOPS, Digital Marketing and other IT and Management courses.
Overview of Apache Flink: the 4G of Big Data Analytics Frameworks
This document provides an overview of Apache Flink and discusses why it is suitable for real-world streaming analytics. The document contains an agenda that covers how Flink is a multi-purpose big data analytics framework, why streaming analytics are emerging, why Flink is suitable for real-world streaming analytics, novel use cases enabled by Flink, who is using Flink, and where to go from here. Key points include Flink innovations like custom memory management, its DataSet API, rich windowing semantics, and native iterative processing. Flink's streaming features that make it suitable for real-world use include its pipelined processing engine, stream abstraction, performance, windowing support, fault tolerance, and integration with Hadoop.
Overview of Apache Fink: the 4 G of Big Data Analytics Frameworks
Slides of my talk at the Hadoop Summit Europe in Dublin, Ireland on April 13th, 2016. The talk introduces Apache Flink as both a multi-purpose Big Data analytics framework and real-world streaming analytics framework. It is focusing on Flink's key differentiators and suitability for streaming analytics use cases. It also shows how Flink enables novel use cases such as distributed CEP (Complex Event Processing) and querying the state by behaving like a key value data store.
Overview of Apache Fink: The 4G of Big Data Analytics Frameworks
This document provides an overview of Apache Flink and discusses why it is suitable for real-world streaming analytics. The document contains an agenda that covers how Flink is a multi-purpose big data analytics framework, why streaming analytics are emerging, why Flink is suitable for real-world streaming analytics, novel use cases enabled by Flink, who is using Flink, and where to go from here. Key points include Flink innovations like custom memory management, its DataSet API, rich windowing semantics, and native iterative processing. Flink's streaming features that make it suitable for real-world use include its pipelined processing engine, stream abstraction, performance, windowing support, fault tolerance, and integration with Hadoop.
Real time stock processing with apache nifi, apache flink and apache kafka
Real time stock processing with apache nifi, apache flink and apache kafka with Kafka Connect apps, SMM, NiFi Registry, Scheam Registry, Kafka topics, Flink SQL, NiFi
Apache Flink: Past, Present and Future
These are the slides for the presentation given on the history and vision of Flink for the first Flink meetup in Stockholm.
DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK
Building Real-Time Pipelines With FLaNK
Timothy Spann, Principal Developer Advocate, Streaming - Cloudera Future of Data meetup, startup grind, AI Camp
The combination of Apache Flink, Apache NiFi, and Apache Kafka for building real-time data processing pipelines is extremely powerful, as demonstrated by this case study using the FLaNK-MTA project. The project leverages these technologies to process and analyze real-time data from the New York City Metropolitan Transportation Authority (MTA). FLaNK-MTA demonstrates how to efficiently collect, transform, and analyze high-volume data streams, enabling timely insights and decision-making.
Apache NiFi
Apache Kafka
Apache Flink
Apache Iceberg
LLM
Generative AI
Slack
Postgresql
Apache Flink - Community Update January 2015
The document summarizes updates from the Apache Flink community in January 2015, including Flink graduating to a top-level Apache project, the release of version 0.8.0 with improvements, work on a graph API called "gelly", and an upcoming 0.9 release with changes like intermediate datasets. It also outlines Flink's roadmap for 2015, with work on APIs, runtime improvements, streaming capabilities, machine learning libraries, and integration with other projects.
Apache Fink 1.0: A New Era for Real-World Streaming Analytics
These are the slides of my talk at the Chicago Apache Flink Meetup on April 19, 2016. This talk explains how Apache Flink 1.0 announced on March 8th, 2016 by the Apache Software Foundation, marks a new era of Real-Time and Real-World streaming analytics. The talk will map Flink's capabilities to streaming analytics use cases.
Apache Flink Community Update March 2015
Slides from the monthly Apache Flink Community Update. Presented at the Flink Meetup in Berlin, Germany.
Real time cloud native open source streaming of any data to apache solr
Real time cloud native open source streaming of any data to apache solr
Utilizing Apache Pulsar and Apache NiFi we can parse any document in real-time at scale. We receive a lot of documents via cloud storage, email, social channels and internal document stores. We want to make all the content and metadata to Apache Solr for categorization, full text search, optimization and combination with other datastores. We will not only stream documents, but all REST feeds, logs and IoT data. Once data is produced to Pulsar topics it can instantly be ingested to Solr through Pulsar Solr Sink.
Utilizing a number of open source tools, we have created a real-time scalable any document parsing data flow. We use Apache Tika for Document Processing with real-time language detection, natural language processing with Apache OpenNLP, Sentiment Analysis with Stanford CoreNLP, Spacy and TextBlob. We will walk everyone through creating an open source flow of documents utilizing Apache NiFi as our integration engine. We can convert PDF, Excel and Word to HTML and/or text. We can also extract the text to apply sentiment analysis and NLP categorization to generate additional metadata about our documents. We also will extract and parse images that if they contain text we can extract with TensorFlow and Tesseract.
Robust stream processing with Apache Flink
Aljoscha Krettek offers a very short introduction to stream processing before diving into writing code and demonstrating the features in Apache Flink that make truly robust stream processing possible, with a focus on correctness and robustness in stream processing.
All of this will be done in the context of a real-time analytics application that we’ll be modifying on the fly based on the topics we’re working though, as Aljoscha exercises Flink’s unique features, demonstrates fault recovery, clearly explains why event time is such an important concept in robust, stateful stream processing, and covers the features you need in a stream processor to do robust, stateful stream processing in production.
We’ll also use a real-time analytics dashboard to visualize the results we’re computing in real time, allowing us to easily see the effects of the code we’re developing as we go along.
Topics include:
* Apache Flink
* Stateful stream processing
* Event time versus processing time
* Fault tolerance
* State management in the face of faults
* Savepoints
* Data reprocessing
Apache-Flink-What-How-Why-Who-Where-by-Slim-Baltagi
This introductory level talk is about Apache Flink: a multi-purpose Big Data analytics framework leading a movement towards the unification of batch and stream processing in the open source.
With the many technical innovations it brings along with its unique vision and philosophy, it is considered the 4 G (4th Generation) of Big Data Analytics frameworks providing the only hybrid (Real-Time Streaming + Batch) open source distributed data processing engine supporting many use cases: batch, streaming, relational queries, machine learning and graph processing.
In this talk, you will learn about:
1. What is Apache Flink stack and how it fits into the Big Data ecosystem?
2. How Apache Flink integrates with Hadoop and other open source tools for data input and output as well as deployment?
3. Why Apache Flink is an alternative to Apache Hadoop MapReduce, Apache Storm and Apache Spark.
4. Who is using Apache Flink?
5. Where to learn more about Apache Flink?
Similar to August Flink Community Update (20)
Bay Area Apache Flink Meetup Community Update August 2015
Bay Area Apache Flink Meetup Community Update August 2015
Flink Community Update December 2015: Year in Review
Flink Community Update December 2015: Year in Review
Overview of Apache Flink: Next-Gen Big Data Analytics Framework
Overview of Apache Flink: Next-Gen Big Data Analytics Framework
Robert Metzger - Apache Flink Community Updates November 2016 @ Berlin Meetup
Robert Metzger - Apache Flink Community Updates November 2016 @ Berlin Meetup
Overview of Apache Flink: the 4G of Big Data Analytics Frameworks
Overview of Apache Flink: the 4G of Big Data Analytics Frameworks
Overview of Apache Fink: the 4 G of Big Data Analytics Frameworks
Overview of Apache Fink: the 4 G of Big Data Analytics Frameworks
Overview of Apache Fink: The 4G of Big Data Analytics Frameworks
Overview of Apache Fink: The 4G of Big Data Analytics Frameworks
Real time stock processing with apache nifi, apache flink and apache kafka
Real time stock processing with apache nifi, apache flink and apache kafka
DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK
DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK
Apache Fink 1.0: A New Era for Real-World Streaming Analytics
Apache Fink 1.0: A New Era for Real-World Streaming Analytics
Real time cloud native open source streaming of any data to apache solr
Real time cloud native open source streaming of any data to apache solr
Apache-Flink-What-How-Why-Who-Where-by-Slim-Baltagi
Apache-Flink-What-How-Why-Who-Where-by-Slim-Baltagi
More from Robert Metzger
How to Contribute to Apache Flink (and Flink at the Apache Software Foundation)
This document discusses how to contribute to the Apache Flink project. It provides an overview of the Apache Software Foundation and Flink's role within it. It describes the various roles within Apache projects like Flink, including committers, PMC members, and contributors. It outlines ways to contribute such as through user support, documentation, code contributions, and reviews. The document emphasizes that there are many paths to contribute and all contributions are welcome.
dA Platform Overview
dA Platform is a production-ready platform for stream processing with Apache Flink®. The Platform includes open source Apache Flink, a stateful stream processing and event-driven application framework, and dA Application Manager, a central deployment and management component. dA Platform schedules clusters on Kubernetes, deploys stateful Flink applications, and controls these applications and their state.
Apache Flink @ Tel Aviv / Herzliya Meetup
Slides of my Apache Flink talk "Big Things" meetup at Proofpoint and "Apache Flink" meetup at Clicktale in January 2018
A Data Streaming Architecture with Apache Flink (berlin Buzzwords 2016)
This document discusses Apache Flink, an open source stream processing framework. It describes how Flink enables streaming Extract, Transform, Load (ETL) workflows with low latency and high throughput. The document outlines how streaming ETL can continuously move and transform data as it arrives, rather than in periodic batch jobs. It concludes with an announcement for an upcoming Flink hackathon and questions.
Community Update May 2016 (January - May) | Berlin Apache Flink Meetup
This document provides a community update from Robert Metzger about Apache Flink activities from January to May 2016. Key events include the release of Apache Flink 1.0.0 in March, the announcement of Flink Forward 2016, new connectors being released, and work beginning on Flink 1.1 including documentation improvements and new features. Upcoming talks promoting Flink at various conferences are also listed.
GOTO Night Amsterdam - Stream processing with Apache Flink
This document discusses Apache Flink, an open source stream processing framework. It provides an overview of Flink and how it enables low-latency stream processing compared to traditional batch processing systems. Key aspects covered include windowing, state handling, fault tolerance, and performance benchmarks showing Flink can achieve high throughput. The document demonstrates how Flink addresses challenges like out-of-order events, state management, and exactly-once processing through features like event-time processing, managed state, and distributed snapshots.
QCon London - Stream Processing with Apache Flink
Robert Metzger presented on Apache Flink, an open source stream processing framework. He discussed how streaming data enables real-time analysis with low latency compared to traditional batch processing. Flink provides unique building blocks like windows, state handling, and fault tolerance to process streaming data reliably at high throughput. Benchmark results showed Flink achieving throughputs over 15 million messages/second, outperforming Storm by 35x.
January 2016 Flink Community Update & Roadmap 2016
This presentation from the 13th Flink Meetup in Berlin contains the regular community update for January and a walkthrough of the most important upcoming features in 2016
Apache Flink Meetup Munich (November 2015): Flink Overview, Architecture, Int...
This presentation held in at Inovex GmbH in Munich in November 2015 was about a general introduction of the streaming space, an overview of Flink and use cases of production users as presented at Flink Forward.
Chicago Flink Meetup: Flink's streaming architecture
This document summarizes the architecture of Apache Flink's streaming runtime. Flink is a stream processor that embraces the streaming nature of data with low latency, high throughput, and exactly-once guarantees. It achieves this through pipelining to keep data moving efficiently and distributed snapshots for fault tolerance. Flink also supports batch processing as a special case of streaming by running bounded streams as a single global window.
Architecture of Flink's Streaming Runtime @ ApacheCon EU 2015
This is a presentation I gave at ApacheCon 2015 in Budapest.
It explains Apache Flink, its streaming runtime, APIs and unique features.
Click-Through Example for Flink’s KafkaConsumer Checkpointing
See how Apache Flink's Kafka Consumer is integrating with the checkpointing mechanisms of Flink for exactly once guarantees
Apache Flink Deep-Dive @ Hadoop Summit 2015 in San Jose, CA
Flink is a unified stream and batch processing framework that natively supports streaming topologies, long-running batch jobs, machine learning algorithms, and graph processing through a pipelined dataflow execution engine. It provides high-level APIs, automatic optimization, efficient memory management, and fault tolerance to execute all of these workloads without needing to treat the system as a black box. Flink achieves native support through its ability to execute everything as data streams, support iterative and stateful computation through caching and managed state, and optimize jobs through cost-based planning and local execution strategies like sort merge join.
Apache Flink Hands On
This document provides an overview of how to run, debug, and tune Apache Flink applications. It discusses:
- Writing and testing Flink jobs locally and submitting them to a cluster for execution
- Debugging techniques like logs, accumulators, and remote debugging
- Tuning jobs by configuring parallelism, memory settings, and I/O directories
- Common issues like OutOfMemoryErrors and how to resolve them
Unified batch and stream processing with Flink @ Big Data Beers Berlin May 2015
Robert Metzger presented on the 1 year growth of the Apache Flink community and an overview of Flink's capabilities. Flink can natively support streaming, batch, machine learning, and graph processing workloads by executing everything as data streams, allowing some iterative and stateful operations, and operating on managed memory. Key aspects of Flink streaming include its pipelined processing, expressive APIs, efficient fault tolerance, and flexible windows and state. Batch pipelines in Flink are also executed as streaming programs with some blocking operations. Flink additionally supports SQL-like queries, machine learning algorithms through iterative data flows, and graph analysis through stateful delta iterations.
Compute "Closeness" in Graphs using Apache Giraph.
The document describes validating different implementations for measuring closeness in graphs using the Apache Giraph framework. Statistical tests show the HyperLogLogSketch implementation exhibits the highest correlation with the baseline "bitfield" implementation on two datasets and outperforms other approaches in approximating closeness values while using less memory. Next steps involve benchmarking the implementations on larger datasets stored out-of-core in Giraph to see if the HyperLogLogSketch performance and accuracy results hold for bigger graphs.
Stratosphere System Overview Big Data Beers Berlin. 20.11.2013
Stratosphere is the next generation big data processing engine.
These slides introduce the most important features of Stratosphere by comparing it with Apache Hadoop.
For more information, visit stratosphere.eu
Based on university research, it is now a completely open-source, community driven development with focus on stability and usability.
Stratosphere Intro (Java and Scala Interface)
A quick walk overview of Stratosphere, including our Scala programming interface.
See also bigdataclass.org for two self-paced Stratosphere Big Data exercises.
More information about Stratosphere: stratosphere.eu
More from Robert Metzger (18)
How to Contribute to Apache Flink (and Flink at the Apache Software Foundation)
How to Contribute to Apache Flink (and Flink at the Apache Software Foundation)
A Data Streaming Architecture with Apache Flink (berlin Buzzwords 2016)
A Data Streaming Architecture with Apache Flink (berlin Buzzwords 2016)
Community Update May 2016 (January - May) | Berlin Apache Flink Meetup
Community Update May 2016 (January - May) | Berlin Apache Flink Meetup
GOTO Night Amsterdam - Stream processing with Apache Flink
GOTO Night Amsterdam - Stream processing with Apache Flink
January 2016 Flink Community Update & Roadmap 2016
January 2016 Flink Community Update & Roadmap 2016
Apache Flink Meetup Munich (November 2015): Flink Overview, Architecture, Int...
Apache Flink Meetup Munich (November 2015): Flink Overview, Architecture, Int...
Chicago Flink Meetup: Flink's streaming architecture
Chicago Flink Meetup: Flink's streaming architecture
Architecture of Flink's Streaming Runtime @ ApacheCon EU 2015
Architecture of Flink's Streaming Runtime @ ApacheCon EU 2015
Click-Through Example for Flink’s KafkaConsumer Checkpointing
Click-Through Example for Flink’s KafkaConsumer Checkpointing
Apache Flink Deep-Dive @ Hadoop Summit 2015 in San Jose, CA
Apache Flink Deep-Dive @ Hadoop Summit 2015 in San Jose, CA
Unified batch and stream processing with Flink @ Big Data Beers Berlin May 2015
Unified batch and stream processing with Flink @ Big Data Beers Berlin May 2015
Compute "Closeness" in Graphs using Apache Giraph.
Compute "Closeness" in Graphs using Apache Giraph.
Stratosphere System Overview Big Data Beers Berlin. 20.11.2013
Stratosphere System Overview Big Data Beers Berlin. 20.11.2013
Recently uploaded
AWS Cloud Cost Optimization Presentation.pptx
This presentation provides valuable insights into effective cost-saving techniques on AWS. Learn how to optimize your AWS resources by rightsizing, increasing elasticity, picking the right storage class, and choosing the best pricing model. Additionally, discover essential governance mechanisms to ensure continuous cost efficiency. Whether you are new to AWS or an experienced user, this presentation provides clear and practical tips to help you reduce your cloud costs and get the most out of your budget.
Programming Foundation Models with DSPy - Meetup Slides
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
Taking AI to the Next Level in Manufacturing.pdf
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
Dandelion Hashtable: beyond billion requests per second on a commodity server
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
FREE A4 Cyber Security Awareness Posters-Social Engineering part 3
Free A4 downloadable and printable Cyber Security, Social Engineering Safety and security Training Posters . Promote security awareness in the home or workplace. Lock them Out From training providers datahops.com
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
Trusted Execution Environment for Decentralized Process Mining
Presentation of the paper "Trusted Execution Environment for Decentralized Process Mining" given during the CAiSE 2024 Conference in Cyprus on June 7, 2024.
HCL Notes and Domino License Cost Reduction in the World of DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/temporal-event-neural-networks-a-more-efficient-alternative-to-the-transformer-a-presentation-from-brainchip/
Chris Jones, Director of Product Management at BrainChip , presents the “Temporal Event Neural Networks: A More Efficient Alternative to the Transformer” tutorial at the May 2024 Embedded Vision Summit.
The expansion of AI services necessitates enhanced computational capabilities on edge devices. Temporal Event Neural Networks (TENNs), developed by BrainChip, represent a novel and highly efficient state-space network. TENNs demonstrate exceptional proficiency in handling multi-dimensional streaming data, facilitating advancements in object detection, action recognition, speech enhancement and language model/sequence generation. Through the utilization of polynomial-based continuous convolutions, TENNs streamline models, expedite training processes and significantly diminish memory requirements, achieving notable reductions of up to 50x in parameters and 5,000x in energy consumption compared to prevailing methodologies like transformers.
Integration with BrainChip’s Akida neuromorphic hardware IP further enhances TENNs’ capabilities, enabling the realization of highly capable, portable and passively cooled edge devices. This presentation delves into the technical innovations underlying TENNs, presents real-world benchmarks, and elucidates how this cutting-edge approach is positioned to revolutionize edge AI across diverse applications.Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
TrustArc Webinar - 2024 Global Privacy Survey
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
Predictive maintenance is a proactive approach that anticipates equipment failures before they happen. At the forefront of this innovative strategy is Artificial Intelligence (AI), which brings unprecedented precision and efficiency. AI in predictive maintenance is transforming industries by reducing downtime, minimizing costs, and enhancing productivity.
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
This case study explores designing a scalable e-commerce platform, covering key requirements, system components, and best practices.
Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks creates AI-boosted business software that helps employees work more efficiently and effectively. Managing data across multiple RDBMS and NoSQL databases was already a challenge at their current scale. To prepare for 10X growth, they knew it was time to rethink their database strategy. Learn how they architected a solution that would simplify scaling while keeping costs under control.
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Inconsistent user experience and siloed data, high costs, and changing customer expectations – Citizens Bank was experiencing these challenges while it was attempting to deliver a superior digital banking experience for its clients. Its core banking applications run on the mainframe and Citizens was using legacy utilities to get the critical mainframe data to feed customer-facing channels, like call centers, web, and mobile. Ultimately, this led to higher operating costs (MIPS), delayed response times, and longer time to market.
Ever-changing customer expectations demand more modern digital experiences, and the bank needed to find a solution that could provide real-time data to its customer channels with low latency and operating costs. Join this session to learn how Citizens is leveraging Precisely to replicate mainframe data to its customer channels and deliver on their “modern digital bank” experiences.
Recently uploaded (20)
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
FREE A4 Cyber Security Awareness Posters-Social Engineering part 3
FREE A4 Cyber Security Awareness Posters-Social Engineering part 3
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Trusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process Mining
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
August Flink Community Update
- 1. Berlin Apache Flink Meetup #10 Community Update August 2015 Robert Metzger Committer and PMC Member rmetzger@apache.org @rmetzger_
- 2. Apache Flink is an open source platform for scalable batch and stream data processing. Apache Flink is … flink.apache.org 1 • The core of Flink is a distributed streaming dataflow engine. • Executing dataflows in parallel on clusters • Providing a reliable foundation for various workloads • DataSet and DataStream programming abstractions are the foundation for user programs and higher layers
- 3. One engine for many use cases flink.apache.org 2 Real time streaming topologies Machine Learning at scale Graph Analysis Long batch pipelines
- 4. What happened? • New Committer: Chesnay Schepler • Discussions for a 0.9.1 release started • Flink is becoming more popular: – 1000+ Twitter followers – 500+ GitHub stars – Named as “open source Big Data project” to watch by ZDNet. – Flink Forward schedule with great speakers announced flink.apache.org 3
- 5. Now in master (0.10-SNAPSHOT) flink.apache.org 4 • Gelly Scala API • Flink dropped Java 6 support • Streaming connector for Elastic Search • Sampling operation on DataSet API • A lot of bug fixes: – Streaming: APIs, general stability, kafka connector
- 6. Articles and Meetups • High-throughput, low-latency, and exactly-once stream processing with Apache Flink [1] • Introducing Gelly: Graph Processing with Apache Flink [2] • Apache Flink and the case for stream processing [3] • Crunching Parquet Files with Apache Flink [4] • The morning paper: Asynchronous Distributed Snapshots for Distributed Dataflows [5] • Five open source Big Data projects to watch [6] • Big Data Performance Engineering: Examples from Hadoop, Pig, HBase, Flink and Spark [7] 5 [1] http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/ [2] http://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html [3] http://www.kdnuggets.com/2015/08/apache-flink-stream-processing.html [4] https://medium.com/@istanbul_techie/crunching-parquet-files-with-apache-flink-200bec90d8a7 [5] http://blog.acolyer.org/2015/08/19/asynchronous-distributed-snapshots-for-distributed-dataflows/ [6] http://www.zdnet.com/article/five-open-source-big-data-projects-to-watch/ [7] http://www.bigsynapse.com/addressing-big-data-performance
- 8. Upcoming • Sept 15: Washington DC Area Apache Flink Meetup • Sept 17: StreamProcessing.be meetup • Sept 28-30: Flink Talks at ApacheCon Big Data Budapest New Meetup groups: • New York • Boston flink.apache.org 7
- 9. Flink Forward schedule published flink.apache.org 8 • http://flink-forward.org/?post_type=day • Talks by Huawei, CapitalOne, Bouyges, Ericsson, Amadeus, ResearchGate, RedHat, and many more.
Editor's Notes
- in 3 month