IMPACT Final Conference - Clemens Neudecker
•Download as PPT, PDF•
0 likes•1,096 views
The IMPACT Interoperability Framework provides a way to integrate various OCR and other software components into reusable workflows. It uses a Java-based architecture with web services and the open source Taverna workflow system. Developers can integrate new command line tools as web services with minimal effort, and workflows can then be built, shared, and executed through a web portal. The framework has been evaluated for scalability and is intended to support a community around sharing workflows and experiments.
Report
Share
Report
Share
![Background ,[object Object],[object Object],[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
IMPACT Final Conference - Claus Gravenhorst
Applied IMPACT: Does the new FineReader Engine and Dutch Lexicon increase OCR accuracy and production efficiency? A case study by KB and CCS.
The IMPACT Interoperability Framework - Workflows for OCR and beyond
The IMPACT Interoperability Framework - Workflows for OCR and beyond
Better, faster, cheaper. Solutions of the IMPACT Centre of Competence and future challenges, The British Library, 24-25 October 2011, London, United Kingdom.
I²: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...
We present I², an interactive development environment for real-time analysis pipelines, which is based on Apache Flink and Apache Zeppelin. The sheer amount of available streaming data frequently makes it impossible to visualize all data points at the same time. I² coordinates running Flink jobs and corresponding visualizations such that only the currently depicted data points are processed in Flink and transferred towards the front end. We show how Flink jobs can adapt to changed visualization properties at runtime to allow interactive data exploration on high bandwidth data streams. Moreover, we present a data reduction technique which minimizes data transfer while providing loss free time-series plots. We show I² in a live demonstration in which we replay recorded sensor data from a football match (ca. 12k event/s). I² was first presented at EDBT'17 where it was awarded as best demonstration. The demonstration is available as open source at github.com/TU-Berlin-DIMA/i2.
Building Applications with Streams and Snapshots
Stream processing has been traditionally associated with realtime analytics. Modern stream processors, like Apache Flink, however, go far beyond that and give us a new approach to build applications and services as a whole.
This talk shows how to build applications on *data streams*, *state*, and *snaphots* (point-in-time views of application state) using Apache Flink. Rather than separating computation (application) and state (database), Flink manages the application logic and state as a tight pair and uses snapshots for consistent view onto the application and its state. With features like Flink's queryable state, the stream processor and database effectively become one.
This application pattern has many interesting properties: Aside from having fewer moving parts, it supports very high event rates because of its tight integration between computation and state, and its simple concurrency and recovery model. At the same time, it exposes a powerful consistency model, allows for seamless forking/updating/rollback of online applications, generalizes across historic and real-time data, and easily incorporates event time semantics and handling of late data. Finally, it allows applications to be defined in an easy way via streaming SQL.
Flink Forward Berlin 2018: Xingcan Cui - "Stream Join in Flink: from Discrete...
As a distributed stream processing engine, Flink provides users with convenient operators to manipulate data on the fly. Among all these operators, join could be the most complicated one as it requires the capability to cross-analyze various sources simultaneously. In this talk, we aim to give a comprehensive introduction to the stream join in Flink. Specifically, we'll first provide an overview of the different join types and which of them are currently supported by Flink DataStream and Table & SQL APIs. Then we'll discuss some key points when performing distributed stream join. After that, we'd like to focus the rationale and implementation details of the time-windowed join launched in version 1.4. Since there are still a lot of improvements can be made, we'll end our talk by sharing some proposals for the future work.
Monitoring Flink with Prometheus
Video: https://data-artisans.com/flink-forward-berlin/resources/monitoring-flink-with-prometheus
Live Demo Code: https://github.com/mbode/flink-prometheus-example
Prometheus is a cloud-native monitoring system prioritizing reliability and simplicity – and Flink works really well with it! This session will show you how to leverage the Flink metrics system together with Pronetheus to improve the observability of your jobs. There will be a live demo establishing how everything ties in together. The talk is aimed at people already building and running Flink jobs who would like to gain more insight into them. It is fine if you are not familiar with Prometheus yet as the basic concepts will be introduced. If you have ever wondered how you could use modern monitoring tools to be alerted in the middle of the night in case your Flink job‘s 99th percentile end-to-end latency degraded for some reason, this might just be the talk you are looking for.
Flink Forward Berlin 2018: Viktor Klang - Keynote "The convergence of stream ...
Two of the main software architectural trends in software development this decade has been the move to streaming data processing, and the move to microservice architecture.
Both of these architectures are driven by the needs of managing and mining knowledge from ever increasing volumes of data in a close to real-time fashion—all while being reactive: responsive under failure, and responsive under load. I'm here to tell you that these two trends are converging, and a fusion of the two is both logical and inevitable. In this session we will talk about what a fused approach to stream processing and microservices could look like, what opportunities exist—what software development for business software can look like in the following decade.
Flink Forward Berlin 2018: Shriya Arora - "Taming large-state to join dataset...
Streaming engines like Apache Flink are redefining ETL and data processing. Data can be extracted, transformed, filtered and written out in real-time with an ease matching that of batch processing. However the real challenge of matching the prowess of batch ETL remains in doing joins, in maintaining state and to have the data be paused or rested dynamically. Netflix has a microservices architecture. Different microservices serve and record different kind of user interactions with the product. Some of these live services generate millions of events per second, all carrying meaningful but often partial information. Things start to get exciting when we want to combine the events coming from one high-traffic microservice to another. Joining these raw events generates rich datasets that are used to train the machine learning models that serve Netflix recommendations. Historically we have done this joining of large volume data-sets in batch. However we asked ourselves if the data is being generated in real-time, why must it not be processed downstream in real time? Why wait a full day to get information from an event that was generated a few mins ago? In this talk, we will share how we solved a complex join of two high-volume event streams using Flink. We will talk about maintaining large state, fault tolerance of a stateful application and strategies for failure recovery.
Recommended
IMPACT Final Conference - Claus Gravenhorst
Applied IMPACT: Does the new FineReader Engine and Dutch Lexicon increase OCR accuracy and production efficiency? A case study by KB and CCS.
The IMPACT Interoperability Framework - Workflows for OCR and beyond
The IMPACT Interoperability Framework - Workflows for OCR and beyond
Better, faster, cheaper. Solutions of the IMPACT Centre of Competence and future challenges, The British Library, 24-25 October 2011, London, United Kingdom.
I²: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...
We present I², an interactive development environment for real-time analysis pipelines, which is based on Apache Flink and Apache Zeppelin. The sheer amount of available streaming data frequently makes it impossible to visualize all data points at the same time. I² coordinates running Flink jobs and corresponding visualizations such that only the currently depicted data points are processed in Flink and transferred towards the front end. We show how Flink jobs can adapt to changed visualization properties at runtime to allow interactive data exploration on high bandwidth data streams. Moreover, we present a data reduction technique which minimizes data transfer while providing loss free time-series plots. We show I² in a live demonstration in which we replay recorded sensor data from a football match (ca. 12k event/s). I² was first presented at EDBT'17 where it was awarded as best demonstration. The demonstration is available as open source at github.com/TU-Berlin-DIMA/i2.
Building Applications with Streams and Snapshots
Stream processing has been traditionally associated with realtime analytics. Modern stream processors, like Apache Flink, however, go far beyond that and give us a new approach to build applications and services as a whole.
This talk shows how to build applications on *data streams*, *state*, and *snaphots* (point-in-time views of application state) using Apache Flink. Rather than separating computation (application) and state (database), Flink manages the application logic and state as a tight pair and uses snapshots for consistent view onto the application and its state. With features like Flink's queryable state, the stream processor and database effectively become one.
This application pattern has many interesting properties: Aside from having fewer moving parts, it supports very high event rates because of its tight integration between computation and state, and its simple concurrency and recovery model. At the same time, it exposes a powerful consistency model, allows for seamless forking/updating/rollback of online applications, generalizes across historic and real-time data, and easily incorporates event time semantics and handling of late data. Finally, it allows applications to be defined in an easy way via streaming SQL.
Flink Forward Berlin 2018: Xingcan Cui - "Stream Join in Flink: from Discrete...
As a distributed stream processing engine, Flink provides users with convenient operators to manipulate data on the fly. Among all these operators, join could be the most complicated one as it requires the capability to cross-analyze various sources simultaneously. In this talk, we aim to give a comprehensive introduction to the stream join in Flink. Specifically, we'll first provide an overview of the different join types and which of them are currently supported by Flink DataStream and Table & SQL APIs. Then we'll discuss some key points when performing distributed stream join. After that, we'd like to focus the rationale and implementation details of the time-windowed join launched in version 1.4. Since there are still a lot of improvements can be made, we'll end our talk by sharing some proposals for the future work.
Monitoring Flink with Prometheus
Video: https://data-artisans.com/flink-forward-berlin/resources/monitoring-flink-with-prometheus
Live Demo Code: https://github.com/mbode/flink-prometheus-example
Prometheus is a cloud-native monitoring system prioritizing reliability and simplicity – and Flink works really well with it! This session will show you how to leverage the Flink metrics system together with Pronetheus to improve the observability of your jobs. There will be a live demo establishing how everything ties in together. The talk is aimed at people already building and running Flink jobs who would like to gain more insight into them. It is fine if you are not familiar with Prometheus yet as the basic concepts will be introduced. If you have ever wondered how you could use modern monitoring tools to be alerted in the middle of the night in case your Flink job‘s 99th percentile end-to-end latency degraded for some reason, this might just be the talk you are looking for.
Flink Forward Berlin 2018: Viktor Klang - Keynote "The convergence of stream ...
Two of the main software architectural trends in software development this decade has been the move to streaming data processing, and the move to microservice architecture.
Both of these architectures are driven by the needs of managing and mining knowledge from ever increasing volumes of data in a close to real-time fashion—all while being reactive: responsive under failure, and responsive under load. I'm here to tell you that these two trends are converging, and a fusion of the two is both logical and inevitable. In this session we will talk about what a fused approach to stream processing and microservices could look like, what opportunities exist—what software development for business software can look like in the following decade.
Flink Forward Berlin 2018: Shriya Arora - "Taming large-state to join dataset...
Streaming engines like Apache Flink are redefining ETL and data processing. Data can be extracted, transformed, filtered and written out in real-time with an ease matching that of batch processing. However the real challenge of matching the prowess of batch ETL remains in doing joins, in maintaining state and to have the data be paused or rested dynamically. Netflix has a microservices architecture. Different microservices serve and record different kind of user interactions with the product. Some of these live services generate millions of events per second, all carrying meaningful but often partial information. Things start to get exciting when we want to combine the events coming from one high-traffic microservice to another. Joining these raw events generates rich datasets that are used to train the machine learning models that serve Netflix recommendations. Historically we have done this joining of large volume data-sets in batch. However we asked ourselves if the data is being generated in real-time, why must it not be processed downstream in real time? Why wait a full day to get information from an event that was generated a few mins ago? In this talk, we will share how we solved a complex join of two high-volume event streams using Flink. We will talk about maintaining large state, fault tolerance of a stateful application and strategies for failure recovery.
Flink Forward Berlin 2018: Xiaowei Jiang - Keynote: "Unified Engine for Data ...
Flink started with the mission to unify batch and stream processing. We believe that Flink’s architecture is uniquely positioned to be a great engine for streaming, batch and AI workloads at the same time. We will talk about the work we did in this direction.
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
http://flink-forward.org/kb_sessions/declarative-stream-processing-with-streamsql-and-cep/
Complex event processing (CEP) and stream analytics are commonly treated as distinct classes of stream processing applications. While CEP workloads identify patterns from event streams in near real-time, stream analytics queries ingest and aggregate high-volume streams. Both types of use cases have very different requirements which resulted in diverging system designs. CEP systems excel at low-latency processing whereas engines for stream analytics achieve high throughput. Recent advances in open source stream processing yielded systems that can process several millions of events per second at sub-second latency. Systems like Apache Flink enable applications that include typical CEP features as well as heavy aggregations. In this talk we will show how Apache Flink unifies CEP and stream analytics workloads. Guided by examples, we introduce Flink’s CEP-enriched StreamSQL interface and discuss how queries are compiled, optimized, and executed on Flink.
Flink Forward San Francisco 2018: Gregory Fee - "Bootstrapping State In Apach...
Apache Flink is a popular stream computing framework for real-time stream computing. Many stream compute algorithms require trailing data in order to compute the intended result. One example is computing the number of user logins in the last 7 days. This creates a dilemma where the results of the stream program are incomplete until the runtime of the program exceeds 7 days. The alternative is to bootstrap the program using historic data to seed the state before shifting to use real-time data. This talk will discuss alternatives to bootstrap programs in Flink. Some alternatives rely on technologies exogenous to the stream program, such as enhancements to the pub/sub layer, that are more generally applicable to other stream compute engines. Other alternatives include enhancements to Flink source implementations. Lyft is exploring another alternative using orchestration of multiple Flink programs. The talk will cover why Lyft pursued this alternative and future directions to further enhance bootstrapping support in Flink.
Flink Forward Berlin 2017: Maciek Próchniak - TouK Nussknacker - creating Fli...
TouK Nussknacker is a GUI for Apache Flink that was created to address client needs for a more user-friendly interface. It allows defining data pipelines and business rules through expressions that are accessible for semi-technical users. The architecture uses a model-driven approach where the data, sources, sinks, processors and rules are defined through JSON configuration. It has been used successfully in production for over a year at one of the largest Polish telcos for real-time marketing and fraud detection applications.
Flink Forward Berlin 2018: Oleksandr Nitavskyi - "Data lossless event time st...
One of the main characteristics of the good streaming pipeline is correctness for event time processing. Real challenges become when such pipeline should be resilient to different types of failures. In this talk, we describe how Criteo runs Flink on one of the biggest Yarn clusters in Europe and computes 100k messages per second to acknowledge revenue of our platform within the delay of 5 minutes. Real-time revenue monitoring system calculates data under 1% of discrepancies and minimizes business impact in case of revenue anomalies.
Flink Forward San Francisco 2019: Real-time Processing with Flink for Machine...
eal-time Processing with Flink for Machine Learning at Netflix
Machine learning plays a critical role in providing a great Netflix member experience. It is used to drive many parts of the site including video recommendations, search results ranking, and selection of artwork images. Providing high-fidelity, near real-time data is increasingly important for these machine learning pipelines, especially as multi-armed bandit and reinforcement learning techniques, in addition to more ""traditional"" supervised learning, become more prevalent. With access to this data, models are able to converge more quickly, features can be updated more frequently, and analysis can be done in a more timely manner.
In this talk, we will focus on the practical details of leveraging Flink to process trillions of events per day, work with the time dimension, and manage large and frequently-changing state. We will discuss different processing schemes and dataflows, scalability and resiliency challenges we tackled, operational considerations, and instrumentation we added for monitoring job health in production.
Flink Forward Berlin 2018: Raj Subramani - "A streaming Quantitative Analytic...
The application of Quantitative Analytics to trades for the generation of Risk and P&L metrics has traditionally followed a batch based approach. Regulatory changes impose increasing demand for compute on financial institutions along with a growing demand for real time analytics due to increased volumes in eTrading across all asset classes
The talk is based on a use case for pricing Interest Rate Swaps, using Apache Beam, with a call to an external C++ analytics process. It describes the performance characteristics when operating in a non-cloud environment using Apache Flink as opposed to Google Cloud Dataflow
The talk will touch upon the subtle difference when operating across multiple runners. It will make suggestions on approaches to portability when architecting for a multi-runner operational environment.
Flink Forward Berlin 2018: Brian Wolfe - "Upshot: distributed tracing using F...
Distributed tracing is used to analyze performance and error cases in service oriented architectures. The Observability team at Airbnb recently created Upshot, a data pipeline that uses Flink to analyze over 40 million trace events per minute. Summaries of the resulting data are sent to Druid, Datadog, and other downstream datastores. This talk will focus on how we use Flink and how we analyzed and addressed scaling issues we encountered while building Upshot.
pracfinal
This document discusses Daniel Waters' work on a compiler instrumentation tool. It provides details on optimizing instrumented code, handling distributed computing environments, ensuring compatibility across systems, addressing file handling issues, and verifying changes through testing on a supercomputer. The document analyzes techniques for improving cache usage, function inlining, and variable placement in optimized code. It also covers using process IDs, file names, and flags to aggregate profiling data from multiple nodes.
Nova orchestra overview
1. Nova Orchestra provides process automation capabilities within and between organizations, including automatic document transfer and composing the front ends of two business applications.
2. Orchestra is an open source business process management solution based on the OASIS BPEL standard. It includes an orchestration engine, administration console, and monitoring tools.
3. The vision is for a flexible, extensible, and standard workflow/BPM platform that supports multiple process languages and allows for both descriptive and executable process models.
Tuning Flink For Robustness And Performance
Flink's stateful stream processing engine presents a huge variety of optional features and configuration choices to the user. Figuring out the "optimal" choices for any production environment and use-case can therefore often be challenging. In this talk, we will explore and discuss the universe of Flink configuration with respect to robustness and performance.
We will start with a closer look under the hood, at core data structures and algorithms, to build the foundation for understanding the impact of tuning parameters and the costs-benefit-tradeoffs that come with certain features and options. In particular, we will focus on state backend choices (Heap vs RocksDB), tuning checkpointing (incremental checkpoints, ...) and recovery (local recovery), file systems, TTL state, and considerations for the network stack. This also includes a discussion about estimating memory requirements and memory partitioning.
Stefan Richter
Flink Forward 2018
Berlin
Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...
In Zalando's microservice architecture, each service continuously generates streams of events for the purposes of inter-service communication or data integration. Some of these events describe business processes, e.g. a customer has placed an order or a parcel has been shipped. Out of this, the need to materialize event streams from the central event bus into persistent cloud storage evolved. The temporarily persisted data is then integrated into our relational data warehouse. In this talk we present a materialization engine backed by Apache Flink. We show how we employ Flink’s RESTful API, custom accumulators and stoppable sources to provide another API abstraction layer for deploying, monitoring and controlling our materialization jobs. Our jobs compact event streams depending on event properties and transform their complex JSON structures into flat files for easier integration into the data warehouse.
From Apache Flink® 1.3 to 1.4
With Flink 1.3 being released, the Flink community is already working towards the upcoming release 1.4. Given Flink's high development pace, which manifested in Flink 1.3 being one of the feature-wise biggest releases in its recent history, it becomes more and more difficult to keep track of all development threads. Moreover, it requires more effort to learn about newly added features and which value they provide for your application.
In this talk, I want to present and explain some of Flink's latest features, including incremental checkpointing, fine grained recovery, side outputs and many more. Furthermore, I want to put them in perspective with respect to Flink's future direction by giving some insights into ongoing development threads in the community. Thereby, I intend to give attendees a better picture about Flink's current and future capabilities.
Apache Flink and what it is used for
Aljoscha Krettek is the PMC chair of Apache Flink and Apache Beam, and co-founder of data Artisans. Apache Flink is an open-source platform for distributed stream and batch data processing. It allows for stateful computations over data streams in real-time and historically. Flink supports batch and stream processing using APIs like DataSet and DataStream. Data Artisans originated Flink and provides an application platform powered by Flink and Kubernetes for building stateful stream processing applications.
IMPACT Final Conference - Research Parallel Sessions - 01 impact conference_r...
IMPACT Final Conference - Research Parallel Sessions - 01 impact conference_r...IMPACT Centre of Competence
The document discusses digitization workflows for enhancing and segmenting documents for optical character recognition (OCR). It describes steps for image enhancement including border removal, page curl removal, and correction of arbitrary warping. It then discusses standalone methods for segmenting text lines, words, and characters without relying on character recognition. These include a hybrid text line segmenter and density-based word segmenter that have been evaluated on historical documents with promising results. The techniques allow digitization of documents with non-standard words or layouts.IMPACT Final Conference - Gregory Crane
The document discusses the transformation of humanities research through digital technologies and optical character recognition (OCR). It describes efforts to extract over 2,000 years of Latin text from digitized books and track linguistic changes over time using machine learning techniques. Computational analysis is helping scholars build dynamic digital editions and study underrepresented languages on a massive scale.
IMPACT Final Conference - Aly Conteh
The document introduces the IMPACT Centre of Competence, a not-for-profit organization that aims to advance digitization of historical materials. It provides tools, services, and testing facilities for practitioners in content institutions, researchers, and industry. Membership offers benefits like access to datasets and tools, implementation support, and knowledge sharing. The Centre will be sustained through membership fees and contributions to support continued collaboration in the community.
IMPACT Final Conference - Paul Fogel
Paul Fogel of the California Digital Library examined OCR quality at scale using the corpus from the HathiTrust and its member institutions. The document discusses issues that arise when performing OCR at a massive scale, including the challenges of indexing very large document collections, supporting many different languages, and correcting the inevitable OCR errors produced when scanning and recognizing text from millions of pages.
IMPACT Final Conference - Research Parallel Sessions - 03 typewritten ocr
IMPACT Final Conference - Research Parallel Sessions - 03 typewritten ocrIMPACT Centre of Competence
The document discusses OCR for typewritten documents. It describes the IMPACT project, which is supported by the European Community under the FP7 ICT Work Programme and coordinated by the National Library of the Netherlands. The presentation covers the challenges of typewritten documents for OCR, the specific approaches used in the IMPACT project's TOCR system, and some example results showing its performance.IMACT Final Conference - Language Parallel Sessions - Erjavec
Tomaž Erjavec discusses the development of language resources for historical Slovene, including transcribed texts, an annotated corpus, and a historical lexicon. Over 10 million words of historical Slovene texts have been transcribed. A reference corpus of 300,000 words from the 15th-19th centuries was annotated for part-of-speech and modern equivalents. An initial lexicon of 3,000 entries was expanded to over 20,000 entries incorporating forms from the annotated corpus. The resources aim to support research on and processing of historical Slovene texts.
IMPACT Final Conference - Majlis Bremer Laamanen
Crowdsourcing in the DigitalKoot project with Majlis Bremner-Laamanen from the National Library of Finland.
IMPACT Final Conference - Khalil Rouhana
Khalil Rouhana from the European Commission talks on: The EC Digital agenda & official launch of the IMPACT Centre of Competence
More Related Content
What's hot
Flink Forward Berlin 2018: Xiaowei Jiang - Keynote: "Unified Engine for Data ...
Flink started with the mission to unify batch and stream processing. We believe that Flink’s architecture is uniquely positioned to be a great engine for streaming, batch and AI workloads at the same time. We will talk about the work we did in this direction.
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
http://flink-forward.org/kb_sessions/declarative-stream-processing-with-streamsql-and-cep/
Complex event processing (CEP) and stream analytics are commonly treated as distinct classes of stream processing applications. While CEP workloads identify patterns from event streams in near real-time, stream analytics queries ingest and aggregate high-volume streams. Both types of use cases have very different requirements which resulted in diverging system designs. CEP systems excel at low-latency processing whereas engines for stream analytics achieve high throughput. Recent advances in open source stream processing yielded systems that can process several millions of events per second at sub-second latency. Systems like Apache Flink enable applications that include typical CEP features as well as heavy aggregations. In this talk we will show how Apache Flink unifies CEP and stream analytics workloads. Guided by examples, we introduce Flink’s CEP-enriched StreamSQL interface and discuss how queries are compiled, optimized, and executed on Flink.
Flink Forward San Francisco 2018: Gregory Fee - "Bootstrapping State In Apach...
Apache Flink is a popular stream computing framework for real-time stream computing. Many stream compute algorithms require trailing data in order to compute the intended result. One example is computing the number of user logins in the last 7 days. This creates a dilemma where the results of the stream program are incomplete until the runtime of the program exceeds 7 days. The alternative is to bootstrap the program using historic data to seed the state before shifting to use real-time data. This talk will discuss alternatives to bootstrap programs in Flink. Some alternatives rely on technologies exogenous to the stream program, such as enhancements to the pub/sub layer, that are more generally applicable to other stream compute engines. Other alternatives include enhancements to Flink source implementations. Lyft is exploring another alternative using orchestration of multiple Flink programs. The talk will cover why Lyft pursued this alternative and future directions to further enhance bootstrapping support in Flink.
Flink Forward Berlin 2017: Maciek Próchniak - TouK Nussknacker - creating Fli...
TouK Nussknacker is a GUI for Apache Flink that was created to address client needs for a more user-friendly interface. It allows defining data pipelines and business rules through expressions that are accessible for semi-technical users. The architecture uses a model-driven approach where the data, sources, sinks, processors and rules are defined through JSON configuration. It has been used successfully in production for over a year at one of the largest Polish telcos for real-time marketing and fraud detection applications.
Flink Forward Berlin 2018: Oleksandr Nitavskyi - "Data lossless event time st...
One of the main characteristics of the good streaming pipeline is correctness for event time processing. Real challenges become when such pipeline should be resilient to different types of failures. In this talk, we describe how Criteo runs Flink on one of the biggest Yarn clusters in Europe and computes 100k messages per second to acknowledge revenue of our platform within the delay of 5 minutes. Real-time revenue monitoring system calculates data under 1% of discrepancies and minimizes business impact in case of revenue anomalies.
Flink Forward San Francisco 2019: Real-time Processing with Flink for Machine...
eal-time Processing with Flink for Machine Learning at Netflix
Machine learning plays a critical role in providing a great Netflix member experience. It is used to drive many parts of the site including video recommendations, search results ranking, and selection of artwork images. Providing high-fidelity, near real-time data is increasingly important for these machine learning pipelines, especially as multi-armed bandit and reinforcement learning techniques, in addition to more ""traditional"" supervised learning, become more prevalent. With access to this data, models are able to converge more quickly, features can be updated more frequently, and analysis can be done in a more timely manner.
In this talk, we will focus on the practical details of leveraging Flink to process trillions of events per day, work with the time dimension, and manage large and frequently-changing state. We will discuss different processing schemes and dataflows, scalability and resiliency challenges we tackled, operational considerations, and instrumentation we added for monitoring job health in production.
Flink Forward Berlin 2018: Raj Subramani - "A streaming Quantitative Analytic...
The application of Quantitative Analytics to trades for the generation of Risk and P&L metrics has traditionally followed a batch based approach. Regulatory changes impose increasing demand for compute on financial institutions along with a growing demand for real time analytics due to increased volumes in eTrading across all asset classes
The talk is based on a use case for pricing Interest Rate Swaps, using Apache Beam, with a call to an external C++ analytics process. It describes the performance characteristics when operating in a non-cloud environment using Apache Flink as opposed to Google Cloud Dataflow
The talk will touch upon the subtle difference when operating across multiple runners. It will make suggestions on approaches to portability when architecting for a multi-runner operational environment.
Flink Forward Berlin 2018: Brian Wolfe - "Upshot: distributed tracing using F...
Distributed tracing is used to analyze performance and error cases in service oriented architectures. The Observability team at Airbnb recently created Upshot, a data pipeline that uses Flink to analyze over 40 million trace events per minute. Summaries of the resulting data are sent to Druid, Datadog, and other downstream datastores. This talk will focus on how we use Flink and how we analyzed and addressed scaling issues we encountered while building Upshot.
pracfinal
This document discusses Daniel Waters' work on a compiler instrumentation tool. It provides details on optimizing instrumented code, handling distributed computing environments, ensuring compatibility across systems, addressing file handling issues, and verifying changes through testing on a supercomputer. The document analyzes techniques for improving cache usage, function inlining, and variable placement in optimized code. It also covers using process IDs, file names, and flags to aggregate profiling data from multiple nodes.
Nova orchestra overview
1. Nova Orchestra provides process automation capabilities within and between organizations, including automatic document transfer and composing the front ends of two business applications.
2. Orchestra is an open source business process management solution based on the OASIS BPEL standard. It includes an orchestration engine, administration console, and monitoring tools.
3. The vision is for a flexible, extensible, and standard workflow/BPM platform that supports multiple process languages and allows for both descriptive and executable process models.
Tuning Flink For Robustness And Performance
Flink's stateful stream processing engine presents a huge variety of optional features and configuration choices to the user. Figuring out the "optimal" choices for any production environment and use-case can therefore often be challenging. In this talk, we will explore and discuss the universe of Flink configuration with respect to robustness and performance.
We will start with a closer look under the hood, at core data structures and algorithms, to build the foundation for understanding the impact of tuning parameters and the costs-benefit-tradeoffs that come with certain features and options. In particular, we will focus on state backend choices (Heap vs RocksDB), tuning checkpointing (incremental checkpoints, ...) and recovery (local recovery), file systems, TTL state, and considerations for the network stack. This also includes a discussion about estimating memory requirements and memory partitioning.
Stefan Richter
Flink Forward 2018
Berlin
Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...
In Zalando's microservice architecture, each service continuously generates streams of events for the purposes of inter-service communication or data integration. Some of these events describe business processes, e.g. a customer has placed an order or a parcel has been shipped. Out of this, the need to materialize event streams from the central event bus into persistent cloud storage evolved. The temporarily persisted data is then integrated into our relational data warehouse. In this talk we present a materialization engine backed by Apache Flink. We show how we employ Flink’s RESTful API, custom accumulators and stoppable sources to provide another API abstraction layer for deploying, monitoring and controlling our materialization jobs. Our jobs compact event streams depending on event properties and transform their complex JSON structures into flat files for easier integration into the data warehouse.
From Apache Flink® 1.3 to 1.4
With Flink 1.3 being released, the Flink community is already working towards the upcoming release 1.4. Given Flink's high development pace, which manifested in Flink 1.3 being one of the feature-wise biggest releases in its recent history, it becomes more and more difficult to keep track of all development threads. Moreover, it requires more effort to learn about newly added features and which value they provide for your application.
In this talk, I want to present and explain some of Flink's latest features, including incremental checkpointing, fine grained recovery, side outputs and many more. Furthermore, I want to put them in perspective with respect to Flink's future direction by giving some insights into ongoing development threads in the community. Thereby, I intend to give attendees a better picture about Flink's current and future capabilities.
Apache Flink and what it is used for
Aljoscha Krettek is the PMC chair of Apache Flink and Apache Beam, and co-founder of data Artisans. Apache Flink is an open-source platform for distributed stream and batch data processing. It allows for stateful computations over data streams in real-time and historically. Flink supports batch and stream processing using APIs like DataSet and DataStream. Data Artisans originated Flink and provides an application platform powered by Flink and Kubernetes for building stateful stream processing applications.
What's hot (14)
Flink Forward Berlin 2018: Xiaowei Jiang - Keynote: "Unified Engine for Data ...
Flink Forward Berlin 2018: Xiaowei Jiang - Keynote: "Unified Engine for Data ...
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
Flink Forward San Francisco 2018: Gregory Fee - "Bootstrapping State In Apach...
Flink Forward San Francisco 2018: Gregory Fee - "Bootstrapping State In Apach...
Flink Forward Berlin 2017: Maciek Próchniak - TouK Nussknacker - creating Fli...
Flink Forward Berlin 2017: Maciek Próchniak - TouK Nussknacker - creating Fli...
Flink Forward Berlin 2018: Oleksandr Nitavskyi - "Data lossless event time st...
Flink Forward Berlin 2018: Oleksandr Nitavskyi - "Data lossless event time st...
Flink Forward San Francisco 2019: Real-time Processing with Flink for Machine...
Flink Forward San Francisco 2019: Real-time Processing with Flink for Machine...
Flink Forward Berlin 2018: Raj Subramani - "A streaming Quantitative Analytic...
Flink Forward Berlin 2018: Raj Subramani - "A streaming Quantitative Analytic...
Flink Forward Berlin 2018: Brian Wolfe - "Upshot: distributed tracing using F...
Flink Forward Berlin 2018: Brian Wolfe - "Upshot: distributed tracing using F...
Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...
Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...
Viewers also liked
IMPACT Final Conference - Research Parallel Sessions - 01 impact conference_r...
IMPACT Final Conference - Research Parallel Sessions - 01 impact conference_r...IMPACT Centre of Competence
The document discusses digitization workflows for enhancing and segmenting documents for optical character recognition (OCR). It describes steps for image enhancement including border removal, page curl removal, and correction of arbitrary warping. It then discusses standalone methods for segmenting text lines, words, and characters without relying on character recognition. These include a hybrid text line segmenter and density-based word segmenter that have been evaluated on historical documents with promising results. The techniques allow digitization of documents with non-standard words or layouts.IMPACT Final Conference - Gregory Crane
The document discusses the transformation of humanities research through digital technologies and optical character recognition (OCR). It describes efforts to extract over 2,000 years of Latin text from digitized books and track linguistic changes over time using machine learning techniques. Computational analysis is helping scholars build dynamic digital editions and study underrepresented languages on a massive scale.
IMPACT Final Conference - Aly Conteh
The document introduces the IMPACT Centre of Competence, a not-for-profit organization that aims to advance digitization of historical materials. It provides tools, services, and testing facilities for practitioners in content institutions, researchers, and industry. Membership offers benefits like access to datasets and tools, implementation support, and knowledge sharing. The Centre will be sustained through membership fees and contributions to support continued collaboration in the community.
IMPACT Final Conference - Paul Fogel
Paul Fogel of the California Digital Library examined OCR quality at scale using the corpus from the HathiTrust and its member institutions. The document discusses issues that arise when performing OCR at a massive scale, including the challenges of indexing very large document collections, supporting many different languages, and correcting the inevitable OCR errors produced when scanning and recognizing text from millions of pages.
IMPACT Final Conference - Research Parallel Sessions - 03 typewritten ocr
IMPACT Final Conference - Research Parallel Sessions - 03 typewritten ocrIMPACT Centre of Competence
The document discusses OCR for typewritten documents. It describes the IMPACT project, which is supported by the European Community under the FP7 ICT Work Programme and coordinated by the National Library of the Netherlands. The presentation covers the challenges of typewritten documents for OCR, the specific approaches used in the IMPACT project's TOCR system, and some example results showing its performance.IMACT Final Conference - Language Parallel Sessions - Erjavec
Tomaž Erjavec discusses the development of language resources for historical Slovene, including transcribed texts, an annotated corpus, and a historical lexicon. Over 10 million words of historical Slovene texts have been transcribed. A reference corpus of 300,000 words from the 15th-19th centuries was annotated for part-of-speech and modern equivalents. An initial lexicon of 3,000 entries was expanded to over 20,000 entries incorporating forms from the annotated corpus. The resources aim to support research on and processing of historical Slovene texts.
IMPACT Final Conference - Majlis Bremer Laamanen
Crowdsourcing in the DigitalKoot project with Majlis Bremner-Laamanen from the National Library of Finland.
IMPACT Final Conference - Khalil Rouhana
Khalil Rouhana from the European Commission talks on: The EC Digital agenda & official launch of the IMPACT Centre of Competence
IMPACT Final Conference - Research Parallel Sessions02 research session_ncsr_...
IMPACT Final Conference - Research Parallel Sessions02 research session_ncsr_...IMPACT Centre of Competence
The document summarizes research activities and tools developed by the National Center for Scientific Research "Demokritos" for the IMPACT project. It describes tools for border detection, page curl detection, and character segmentation. Evaluation results for the border detection and page curl detection tools on large datasets are provided.IMPACT Final Conference - Steven Krauwer
- CLARIN aims to create a federated infrastructure providing researchers access to digital language data and tools through a single sign-on. It seeks to integrate existing resources across Europe to advance humanities and social sciences research.
- CLARIN's success requires collaboration with libraries, which hold vast amounts of printed materials indispensable for researchers but face obstacles like copyright and lack of standardization.
- The IMPACT project's work on optical character recognition technology and goal of an OCR center of expertise can help address a key challenge and bring CLARIN and libraries closer through continued collaboration beyond the project.
IMPACT Final Conference - Language Parallel Sessions - Gotscharek
The document discusses linguistic resources created for improving access to 16th century German texts. It describes how the IMPACT project adapted resources like lexicons to account for the differences between historical and modern German. A groundtruth corpus spanning 1500-1950 was created, as well as a hypothetical lexicon of rule-based variants and a manually verified lexicon to map historical words to their modern equivalents. These resources were able to cover 30% of 16th century vocabulary and improve optical character recognition.
IMPACT Final Conference - Michael Fuchs
The document discusses ABBYY's involvement in the IMPACT project. It states that ABBYY is the OCR technology provider for IMPACT members. It also notes that ABBYY improved its core OCR technologies for the recognition of old documents through its work on the IMPACT project, focusing on areas like image pre-processing, segmentation, character recognition, and export formats. The presentation provides examples of how ABBYY's technologies were enhanced between versions 9 and 10 for tasks like binarization, layout analysis, and character recognition of historical documents.
IMPACT Final Conference - Apostolos Antonacopoulos
This document summarizes the results of experiments examining the effect of scanning parameters like color, resolution, and binarization method on OCR accuracy. The experiments found that bitonal images produced the best OCR results on average but the optimal method varied between images. Higher resolution images did not necessarily improve OCR accuracy. The quality of archival images was also found to affect OCR performance. The document concludes different scanning choices may be suitable depending on the document type and quality.
IMPACT/myGrid Hackathon - Taverna Server as a Portal
IMPACT/myGrid Hackathon - Taverna Server as a Portal with Rob Haines from myGrid - Manchester 14,15/11/2011
IMPACT/myGrid Hackathon - Introduction to IMPACT
The document announces an IMPACT-myGrid-Hackathon event scheduled for November 14-15, 2011 at the University of Manchester. The event website provides additional information and is located at http://impact-mygrid-taverna-hackathon.wikispaces.com/. The hackathon focused on myGrid and Taverna tools.
IMPACT Final Conference - Language Parallel Sessions - Landsbergen
The document discusses named entity (NE) recognition in digitized historical texts. It describes how NEs like people, locations and organizations can be identified during optical character recognition (OCR) and retrieved for users. The key steps include building an NE lexicon database by collecting data, tagging and enriching NEs with metadata, and linking variant names. This helps improve OCR quality and allows users to find NEs despite spelling variations in historical texts.
IMPACT Final Conference - Muehlberger - FEP
The Functional Extension Parser: A Document Understanding Platform by Günter Mühlberger (University of Innsbruck)
IMPACT/myGrid Hackathon - Taverna Roadmap
The document outlines the roadmap for updates and new features in the Taverna workflow system, including releasing versions 2.3 and 3.0 with improvements to the user interface, support for new standards, and integration with additional technologies and domains like clouds, semantic web, and biodiversity. It also discusses new plugins and tools being developed to enhance provenance capture, support additional file formats, and provide domain-specific functionality for astronomy, life sciences, and data mining.
IMPACT/myGrid Hackathon - Introduction to Taverna
Katy Wolstencroft gives an introduction to Taverna at the IMPACT/myGrid Taverna Hackathon, 14th November 2011
IMPACT Final Conference - Asaf Tzadok
The document describes CONCERT, an adaptive collaborative correction platform for digitized text. It uses feedback from users to improve optical character recognition and increase productivity of post-correction. Key features include adaptive OCR, quality control tools, productivity tools like games to motivate volunteers, and monitoring of users to prevent data corruption. It has been used successfully in several library digitization projects worldwide.
Viewers also liked (20)
IMPACT Final Conference - Research Parallel Sessions - 01 impact conference_r...
IMPACT Final Conference - Research Parallel Sessions - 01 impact conference_r...
IMPACT Final Conference - Research Parallel Sessions - 03 typewritten ocr
IMPACT Final Conference - Research Parallel Sessions - 03 typewritten ocr
IMACT Final Conference - Language Parallel Sessions - Erjavec
IMACT Final Conference - Language Parallel Sessions - Erjavec
IMPACT Final Conference - Research Parallel Sessions02 research session_ncsr_...
IMPACT Final Conference - Research Parallel Sessions02 research session_ncsr_...
IMPACT Final Conference - Language Parallel Sessions - Gotscharek
IMPACT Final Conference - Language Parallel Sessions - Gotscharek
IMPACT Final Conference - Apostolos Antonacopoulos
IMPACT Final Conference - Apostolos Antonacopoulos
IMPACT/myGrid Hackathon - Taverna Server as a Portal
IMPACT/myGrid Hackathon - Taverna Server as a Portal
IMPACT Final Conference - Language Parallel Sessions - Landsbergen
IMPACT Final Conference - Language Parallel Sessions - Landsbergen
Similar to IMPACT Final Conference - Clemens Neudecker
Vijay Oscon
Microsoft is taking a multifaceted approach to interoperability including collaboration with partners, developing interoperable products/technologies, promoting standards, and providing developer resources. This includes tools like Eclipse for Silverlight which allows Eclipse developers to build applications with Silverlight, and SDKs for Azure that enable PHP, Java and Ruby developers to leverage Microsoft cloud services. Customer feedback has been positive for these cross-platform tools and Microsoft is continuing work on interoperability bridges between technologies like RIA, SOA, identity and web services.
The Taverna Software Suite
The Taverna Suite provides tools for interactive and batch workflow execution. It includes a workbench for graphical workflow construction, various client interfaces, and servers for multi-user workflow execution. The suite utilizes a plug-in framework and supports a variety of domains, infrastructures, and tools through custom plug-ins.
Vijay Mix Presentation
Vijay Rajagopalan Mix 2009 Silverlight Interoperability:- Eclipse Tooling for Silverlight, Microsoft Cloud Computing Vision
Teched India Vijay Interop Track
Vijay Rajagopalan from Microsoft presented on the company's approach to interoperability. Microsoft is working with open source communities and competitors to build bridges between platforms like Eclipse, Java, PHP and Ruby with Microsoft products and services. Rajagopalan demonstrated Eclipse tools for developing Silverlight applications, SDKs for building Java and Ruby apps on Azure, and discussed enabling claims-based authentication across languages and platforms. He emphasized Microsoft's multifaceted approach to interoperability through collaboration, standards, and developer resources.
.NET Cloud-Native Bootcamp- Los Angeles
This document outlines an agenda for a .NET cloud-native bootcamp. The bootcamp will introduce practices, platforms and tools for building modern .NET applications, including microservices, Cloud Foundry, and cloud-native .NET technologies and patterns. The agenda includes sessions on microservices, Cloud Foundry, hands-on exercises, and a wrap up. Break times are scheduled between sessions.
2012 04-18 (educon2012) emadrid uned on design remote laboratories study lab ...
This document compares LabVIEW and MATLAB for use in developing remote laboratory server software. Both LabVIEW and MATLAB are powerful and rich platforms that support data exchange, standard APIs, and database connectivity. While LabVIEW excels at graphical programming and remote operation, MATLAB is stronger for control algorithm development and simulation. Hybrid approaches using both platforms can achieve high efficiency. LabVIEW supports integration with MATLAB through technologies like ActiveX, DLLs, COM, and MathScript modules.
Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière
During the Confluent Streaming event in Paris, Florent Ramière, Technical Account Manager at Confluent, goes beyond brokers, introducing a whole new ecosystem with Kafka Streams, KSQL, Kafka Connect, Rest proxy, Schema Registry, MirrorMaker, etc.
OCP Datacomm RedHat - Kubernetes Launch
This document provides an overview of cloud-native development and Red Hat OpenShift:
- It discusses moving to cloud-native development through optimizing existing applications, developing new applications faster, and automating infrastructure.
- Red Hat OpenShift is positioned as the enterprise solution for running Kubernetes in production, as it addresses limitations of "raw" Kubernetes through features like developer tools, operations automation, and additional services.
- New features are highlighted for OpenShift 4.6, including improved application topology and monitoring, a new log forwarding API, and enhancements to the developer experience.
Ignacio design and building of iaa s clouds
The document discusses the design and building of Infrastructure as a Service (IaaS) clouds. It describes positioning IaaS in the cloud ecosystem and transforming IT infrastructure into a cloud. It then covers designing a cloud infrastructure with OpenNebula, addressing challenges, and sharing experiences building clouds with OpenNebula. Finally, it discusses innovative European projects utilizing clouds and the outlook for OpenNebula development.
Ignacio design and building of iaa s clouds
The document discusses the design and building of Infrastructure as a Service (IaaS) clouds. It describes positioning IaaS in the cloud ecosystem and transforming IT infrastructure into a cloud. It also covers designing a flexible cloud manager, addressing interoperability, and using OpenNebula as an open-source toolkit to build private, public and hybrid clouds. Finally, it discusses experiences deploying OpenNebula and several innovative European projects related to enhancing grids with cloud computing.
What's New in Confluent Platform 5.5
Watch this webcast here: https://www.confluent.io/online-talks/whats-new-in-confluent-platform-55/
Join the Confluent Product Marketing team as we provide an overview of Confluent Platform 5.5, which makes Apache Kafka and event streaming more broadly accessible to developers with enhancements to data compatibility, multi-language development, and ksqlDB.
Building an event-driven architecture with Apache Kafka allows you to transition from traditional silos and monolithic applications to modern microservices and event streaming applications. With these benefits has come an increased demand for Kafka developers from a wide range of industries. The Dice Tech Salary Report recently ranked Kafka as the highest-paid technological skill of 2019, a year removed from ranking it second.
With Confluent Platform 5.5, we are making it even simpler for developers to connect to Kafka and start building event streaming applications, regardless of their preferred programming languages or the underlying data formats used in their applications.
This session will cover the key features of this latest release, including:
-Support for Protobuf and JSON schemas in Confluent Schema Registry and throughout our entire platform
-Exactly once semantics for non-Java clients
-Admin functions in REST Proxy (preview)
-ksqlDB 0.7 and ksqlDB Flow View in Confluent Control Center
Educon 2012- On the Design of Remote Laboratories
This document discusses remote laboratory architectures and technologies for developing lab server software. It compares LabVIEW and MATLAB, the most common technologies used. Both LabVIEW and MATLAB possess rich features for data exchange, instrument control, and database connectivity. LabVIEW is most popular for remote labs due to its graphical interface while MATLAB is powerful for algorithms. Hybrid methods using both are common, with LabVIEW for signals/GUI and MATLAB for calculations. The document was presented by researchers from the Spanish University for Distance Education.
GOLC 2012 - On Standardizing the Management of LabVIEW-based Remote Laborator...
This document discusses standardizing the management of remote laboratories built using LabVIEW through remote laboratory management systems (RLMSs). It outlines the need for a standard application programming interface (API) layer to wrap LabVIEW-based remote labs and make them compatible with different RLMSs. The layer would define a common set of communication tools from LabVIEW, such as VI server and web services, to connect remote labs to RLMSs while addressing factors like simultaneous access, security, and session management. Developing such an API layer could help share and manage the many existing LabVIEW-based remote labs across various university platforms.
Day in the life event-driven workshop
This document provides an agenda and overview for an event-driven architecture workshop. The workshop will cover topics including event-driven architecture, change data capture, cloud-native integration, how cloud architectures have evolved, modern application elements like APIs, events and data, container-based application development, and integration patterns. Hands-on labs will allow attendees to work with technologies like Apache Kafka and Red Hat AMQ Streams.
Aneka
This document provides an overview of Aneka, an open-source platform for building and managing distributed applications in multi-cloud environments. It describes Aneka's architecture, application development process using its SDK, and supported programming models including task, thread, MapReduce, and parameter sweeping. Features of Aneka allow users to build, accelerate, and manage applications across different runtime environments like PC grids, clusters, multicore processors, and public/private clouds. Current and example applications using Aneka include scientific, commercial, educational projects. Ongoing research with Aneka focuses on areas like cooperative scheduling, virtual execution environments, and quality of service.
Reactive Microservices with Spring 5: WebFlux
On November 27 Trayan Iliev from IPT presented “Reactive microservices with Spring 5: WebFlux” @Dev.bg in Betahaus Sofia. IPT – Intellectual Products & Technologies has been organizing Java & JavaScript trainings since 2003.
Spring 5 introduces a new model for end-to-end functional and reactive web service programming with Spring 5 WebFlow, Spring Data & Spring Boot. The main topics include:
– Introduction to reactive programming, Reactive Streams specification, and project Reactor (as WebFlux infrastructure)
– REST services with WebFlux – comparison between annotation-based and functional reactive programming approaches for building.
– Router, handler and filter functions
– Using reactive repositories and reactive database access with Spring Data. Building end-to-end non-blocking reactive web services using Netty-based web runtime
– Reactive WebClients and integration testing. Reactive WebSocket support
– Realtime event streaming to WebClients using JSON Streams, and to JS client using SSE.
Red Hat and kubernetes: awesome stuff coming your way
2019 is the year lots of techs mature. Lets talk about Istio, Knative, Operators, Machine API and CRDs, Federation and lots of other cool stuff
Achieve Sub-Second Analytics on Apache Kafka with Confluent and Imply
Presenters: Rachel Pedreschi, Senior Director, Solutions Engineering, Imply.io + Josh Treichel, Partner Solutions Architect, Confluent
Analytic pipelines running purely on batch processing systems can suffer from hours of data lag, resulting in accuracy issues with analysis and overall decision-making. Join us for a demo to learn how easy it is to integrate your Apache Kafka® streams in Apache Druid (incubating) to provide real-time insights into the data.
In this online talk, you’ll hear about ingesting your Kafka streams into Imply’s scalable analytic engine and gaining real-time insights via a modern user interface.
Register now to learn about:
-The benefits of combining a real-time streaming platform with a comprehensive analytics stack
-Building an analytics pipeline by integrating Confluent Platform and Imply
-How KSQL, streaming SQL for Kafka, can easily transform and filter streams of data in real time
-Querying and visualizing streaming data in Imply
-Practical ways to implement Confluent Platform and Imply to address common use cases such as analyzing network flows, collecting and monitoring IoT data and visualizing clickstream data
Confluent Platform, developed by the creators of Kafka, enables the ingest and processing of massive amounts of real-time event data. Imply, the complete analytics stack built on Druid, can ingest, store, query and visualize streaming data from Confluent Platform, enabling end-to-end real-time analytics. Together, Confluent and Imply can provide low latency data delivery, data transform, and data querying capabilities to power a range of use cases.
Running PHP In The Cloud
Maarten Balliauw presented on running PHP applications in the cloud using Windows Azure. He began with an overview of Windows Azure and its infrastructure and platform as a service capabilities. He then discussed how to develop PHP applications that leverage Windows Azure capabilities like storage, SQL Azure, and AppFabric. Balliauw demonstrated the Eclipse tools for PHP development on Azure and showed a sample image uploading application that uses various Azure services. He concluded with resources for PHP developers to learn more about developing for the cloud on Windows Azure.
Bn1001 demo ppt advance dot net
The document provides an introduction to the .NET framework. It discusses that .NET is a software platform and runtime that supports multiple programming languages and allows development of web and Windows applications. It describes key aspects of .NET including the Common Language Runtime (CLR), Windows Forms, ASP.NET, ADO.NET, and Visual Studio.NET development tools. It also provides comparisons between .NET and Java platforms and architectures.
Similar to IMPACT Final Conference - Clemens Neudecker (20)
2012 04-18 (educon2012) emadrid uned on design remote laboratories study lab ...
2012 04-18 (educon2012) emadrid uned on design remote laboratories study lab ...
Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière
Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière
GOLC 2012 - On Standardizing the Management of LabVIEW-based Remote Laborator...
GOLC 2012 - On Standardizing the Management of LabVIEW-based Remote Laborator...
Red Hat and kubernetes: awesome stuff coming your way
Red Hat and kubernetes: awesome stuff coming your way
Achieve Sub-Second Analytics on Apache Kafka with Confluent and Imply
Achieve Sub-Second Analytics on Apache Kafka with Confluent and Imply
More from IMPACT Centre of Competence
Session6 01.helmut schmid
Slides of the paper Deep Learning-Based Morphological Taggers and Lemmatizers for Annotating Historical Texts by Helmut Schmid at the 3rd Edition of the DATeCH2019 International Conference
Session1 03.hsian-an wang
This document discusses using text models to improve the accuracy of optical character recognition (OCR) on Chinese rare books. It conducted experiments using n-gram, backward/forward n-gram, and LSTM models on OCR data from ancient medicine books. The backward and forward 4-gram model achieved the highest correction rate at 97.57%. Mixing the LSTM 6-gram model with the OCR's top 5 candidates and probability of the top candidate further improved accuracy to 97.71%, demonstrating that combining text models with OCR probabilities can better correct OCR errors than text models alone. In conclusion, text models are effective for increasing OCR accuracy on rare books, with backward/forward 4-gram and LSTM 6-gram
Session7 03.katrien depuydt
Slides of the paper Turning Digitised Material into a Diachronic Corpus: Metadata Challenges in the Nederlab Project by Katrien Depuydt and Hennie Brugman at the 3rd Edition of the DATeCH2019 International Conference
Session7 02.peter kiraly
Slides of the paper Validating 126 million MARC records by Péter Király at the 3rd Edition of the DATeCH2019 International Conference
Session6 04.giuseppe celano
Slides of the paper Standoff Annotation for the Ancient Greek and Latin Dependency Treebank by Giuseppe Celano at the 3rd Edition of the DATeCH2019 International Conference
Session6 03.sandra young
Slides of the paper Using lexicography to characterise relations between species mentions in the biodiversity literature by Sandra Young at the 3rd Edition of the DATeCH2019 International Conference
Session6 02.jeremi ochab
Slides of the paper Stylometry of literary papyri by Jeremi Ochab and Holger Essler at the 3rd Edition of the DATeCH2019 International Conference
Session5 04.evangelos varthis
Slides of the paper Implementation of a Databaseless Web REST API for the Unstructured Texts of Migne's Patrologia Graeca with Searching capabilities and additional Semantic and Syntactic expandability by Evagelos Varthis, Marios Poulos, Ilias Yarenis and Sozon Papavlasopoulos at the 3rd Edition of the DATeCH2019 International Conference
Session5 03.george rehm
Slides of the paper Curation Technologies for a Cultural Heritage Archive: Analysing and transforming a heterogeneous data set into an interactive curation workbench by Georg Rehm, Martin Lee, Julián Moreno Schneider and Peter Bourgonje at the 3rd Edition of the DATeCH2019 International Conference
Session5 02.tom derrick
Slides of the paper Cross-disciplinary collaborations to enrich access to non-Western language material in the Cultural Heritage sector by Tom Derrick and Nora McGregor at the 3rd Edition of the DATeCH2019 International Conference
Session5 01.rutger vankoert
Slides of the paper Tribunal Archives as Digital Research Facility (TRIADO): new ways to make archives accessible and useable by Anne Gorter, Edwin Klijn, Rutger Van Koert, Marielle Scherer and Ismee Tames at the 3rd Edition of the DATeCH2019 International Conference
Session4 04.senka drobac
Slides of the paper Improving OCR of historical newspapers and journals published in Finland by Senka Drobac, Pekka Kauppinen and Krister Lindén at the 3rd Edition of the DATeCH2019 International Conference
Session3 04.arnau baro
Slides of the paper Towards a generic unsupervised method for transcription of encoded manuscripts by Arnau Baró, Jialuo Chen, Alicia Fornés and Beáta Megyesi at the 3rd Edition of the DATeCH2019 International Conference
Session3 03.christian clausner
Slides of the paper Towards the Extraction of Statistical Information from Digitised Numerical Tables - The Medical Officer of Health Reports Scoping Study by Christian Clausner, Apostolos Antonacopoulos, Christy Henshaw and Justin Hayes at the 3rd Edition of the DATeCH2019 International Conference
Session3 02.kimmo ketunnen
Slides of the paper Detecting Articles in a Digitized Finnish Historical Newspaper Collection 1771–1929: Early Results Using the PIVAJ Software by Kimmo Kettunen, Teemu Ruokolainen, Erno Liukkonen, Pierrick Tranouez, Daniel Antelme and Thierry Paquet at the 3rd Edition of the DATeCH2019 International Conference
Session3 01.clemens neudecker
Slides of the paper OCR-D: An end-to-end open-source OCR framework for historical documents by Clemens Neudecker, Konstantin Baierer, Maria Federbusch, Kay-Michael Würzner, Matthias Boenig, Elisa Hermann and Volker Hartmann at the 3rd Edition of the DATeCH2019 International Conference
Session2 04.ashkan ashkpour
- The document describes a project to fill gaps in knowledge about diamond mining, trading, and polishing in Borneo by developing a workflow using various CLARIAH tools and resources.
- The workflow involved digitizing a diamond encyclopedia, extracting concepts and place names, linking the data to external sources to create linked open data, and querying newspaper archives to build a corpus of relevant articles.
- Promising results showed mining, trading, and polishing continued in Borneo for Southeast Asian customers, and described previously unknown diamond fields and polishing locations in Borneo. The project aims to apply the workflow to other commodities like sugar.
Session2 03.juri opitz
Slides of the paper Automatic Reconstruction of Emperor Itineraries from the Regesta Imperii by Juri Opitz, Leo Born, Vivi Nastase and Yannick Pultar at the 3rd Edition of the DATeCH2019 International Conference
Session2 02.christian reul
Slides of the paper Automatic Semantic Text Tagging on Historical Lexica by Combining OCR and Typography Classification by Christian Reul, Sebastian Göttel, Uwe Springmann, Christoph Wick, Kay-Michael Würzner and Frank Puppe at the 3rd Edition of the DATeCH2019 International Conference
Session2 01.emad mohamed
This document describes the SOS system for segmenting, stemming, and standardizing Arabic text. It presents the challenges of processing Arabic cultural heritage texts which contain orthographic variations. The system uses gradient boosting machines and achieves state-of-the-art performance on segmentation and derives stemming as a byproduct. It also standardizes orthography with high accuracy, which further improves segmentation. The system addresses issues like hamza forms and letter confusions that previous systems did not handle well.
More from IMPACT Centre of Competence (20)
Recently uploaded
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
Communications Mining Series - Zero to Hero - Session 1
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
20240609 QFM020 Irresponsible AI Reading List May 2024
Everything I found interesting about the irresponsible use of machine intelligence in May 2024
“I’m still / I’m still / Chaining from the Block”
“An Outlook of the Ongoing and Future Relationship between Blockchain Technologies and Process-aware Information Systems.” Invited talk at the joint workshop on Blockchain for Information Systems (BC4IS) and Blockchain for Trusted Data Sharing (B4TDS), co-located with with the 36th International Conference on Advanced Information Systems Engineering (CAiSE), 3 June 2024, Limassol, Cyprus.
Full-RAG: A modern architecture for hyper-personalization
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
National Security Agency - NSA mobile device best practices
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Video Streaming: Then, Now, and in the Future
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Join us to introduce Milvus Lite, a vector database that can run on notebooks and laptops, share the same API with Milvus, and integrate with every popular GenAI framework. This webinar is perfect for developers seeking easy-to-use, well-integrated vector databases for their GenAI apps.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications.
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.20240605 QFM017 Machine Intelligence Reading List May 2024
Everything I found interesting about machines behaving intelligently during May 2024
Mind map of terminologies used in context of Generative AI
Mind map of common terms used in context of Generative AI.
Recently uploaded (20)
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
IMPACT Final Conference - Clemens Neudecker
- 1. The IMPACT Interoperability Framework: Workflows for OCR and beyond Clemens Neudecker, KB National Library of the Netherlands 2 nd IMPACT Conference, British Library, London 24/25 October 2011