Flink Case Study: OKKAM

•Download as PPTX, PDF•

2 likes•7,853 views

Flink forward 2015

A Semantic Big Data
Companion
Stefano Bortoli
bortoli@okkam.it
Flavio Pompermaier
pompermaier@okkam.it

The company (briefly)
• Okkam is
– a SME based in Trento, Italy.
– Started as spin-off of the
University of Trento and FBK (2010)
• Okkam core business is
– large-scale data integration using
semantic technologies and
an Entity Name System
• Okkam operative sectors
– Services for public administration
– Services for restaurants (and more)
– Research projects
• FP7, H2020, and Local agencies

Who we are
• Stefano Bortoli, PhD
– works as technical director and researcher at Okkam S.R.L.
(Trento, Italy). His research and development interests are in the
area of Information Integration, with special focus in entity-
centric applications exploiting semantic technologies.
• Flavio Pompermaier, MSc.
– works as senior software engineer at Okkam S.R.L. (Trento, Italy).
Flavio is a passionate developer working with state of the art
technologies, combining semantic with big data technologies.

What we do

Why we need Flink
Entiton data model
Database record
RDF statement
Triplestore
NOSQL
& Index
+
Quad
provenance IRI
predicate
object
object Type
Subject
local IRI
Subject
ENS IRI
RDF Type
Expensive
datawearhouse

Why we are here
• We want to build and manage (very) large
entity-centric knowledge bases
• We endorsed Flink since Stratosphere as data
processing framework (during DOPA FP7)
• Our use cases for Apache Flink:
– Domain reasoning (Flink + Parquet + Thrift)
– RDF data lifecycle (Flink + Parquet + Jena/Sesame )
– RDF data intelligence (Flink + ELKiBi)
– Duplicate record detection (Flink + HBase + Solr)
– Entiton Record linkage (Flink + MongoDB + Kryo)
– Telemetry analysis (Flink + MongoDB + Weka)

Come to our session!
• We are the last presenting, don’t let us ALONE!
• We are hiring! (maybe ;-)

More Related Content

What's hot

Intranet show and_tell_2010

Intranet show and_tell_2010

Intranet show and_tell_2010

Amundsen at Brex and Looker integration

Amundsen at Brex and Looker integration

Amundsen at Brex and Looker integration

Building a Knowledge Graph using NLP and Ontologies

Building a Knowledge Graph using NLP and Ontologies

Building a Knowledge Graph using NLP and Ontologies

Turning search upside down with powerful open source search software

Turning search upside down with powerful open source search software

Turning search upside down with powerful open source search software

AnzoGraph DB - SPARQL 101

AnzoGraph DB - SPARQL 101

AnzoGraph DB - SPARQL 101

Cambridge Semantics

TaLend Online Training

TaLend Online Training

TaLend Online Training

Glory IT Technologies

This talk is a case-study on how Apache Spark and the Spark-Solr library is being used at Flipp for driving search relevancy. Flipp is a Toronto based digital flyer and ecommerce company which helps shoppers save money on weekly shopping. Our customers have the option of browsing through our 5+ million products from the brick-and-mortar retailers in North America. This makes Search a very challenging function in our app. How to show the most relevant and personalized search results to users on a query? The talk will focus on using user signals such as Click Through Rate (CTR) and Impressions to increase search relevancy. I will also talk about how PySpark is used to create the Flipp Search ETL platform for collecting user signals and reading product data from Solr. The problem scenario will be explained in which keyword search and basic relevancy algorithms become ineffective when dealing with a large product database. The solutions will cover the following implementations being used at Flipp to drive relevancy: – Utilizing user clicks and popularity data to derive and index normalized item weights to implement the Search Crowd Curation models in Apache Solr – How around 5+ million items are classified into Google Categories in real time using Keras and Apache Spark to power product category curation in Solr. – How to create a crowd sourced query intent categorizer in Solr using the Spark-Solr library. – The use of offline and online metrics at Flipp for evaluating changes in search relevancy. – Future plans for incorporating Kafka-connect in Apache Solr with structured streaming to perform real-time product indexing with Spark-Solr library.

Using Spark-Solr at Scale: Productionizing Spark for Search with Apache Solr...

Using Spark-Solr at Scale: Productionizing Spark for Search with Apache Solr...

Using Spark-Solr at Scale: Productionizing Spark for Search with Apache Solr...

20181019 code.talks graph_analytics_k_patenge

20181019 code.talks graph_analytics_k_patenge

20181019 code.talks graph_analytics_k_patenge

Data Integration Solutions Created By Koneksys

Data Integration Solutions Created By Koneksys

Data Integration Solutions Created By Koneksys

Lider Reference Model ld4lt session March, 3rd, 2015

Lider Reference Model ld4lt session March, 3rd, 2015

Lider Reference Model ld4lt session March, 3rd, 2015

Sebastian Hellmann

Neo4j: What's Under the Hood

Neo4j: What's Under the Hood

Neo4j: What's Under the Hood

Micro-Servicing Linked Data

Micro-Servicing Linked Data

Micro-Servicing Linked Data

CodeOne 2018 - Microservices in action at the Dutch National Police

CodeOne 2018 - Microservices in action at the Dutch National Police

CodeOne 2018 - Microservices in action at the Dutch National Police

Bert Jan Schrijver

With the community working on preparing the next versions of Apache Spark you may be asking yourself ‘how do I get involved in contributing to this?’ With such a large volume of contributions, it can be hard to know how to begin contributing yourself. Holden Karau offers a developer-focused head start, walking you through how to find good issues, formatting code, finding reviewers, and what to expect in the code review process. In addition to looking at how to contribute code we explore some of the other ways you can contribute to to Apache Spark from helping test release candidates, to doing the all important code reviews, bug triage, and many more (like answering questions).

Getting Started Contributing to Apache Spark – From PR, CR, JIRA, and Beyond

Getting Started Contributing to Apache Spark – From PR, CR, JIRA, and Beyond

Getting Started Contributing to Apache Spark – From PR, CR, JIRA, and Beyond

Top 5 Considerations When Evaluating NoSQL

Top 5 Considerations When Evaluating NoSQL

Top 5 Considerations When Evaluating NoSQL

While “software is eating the world”, those who are able to best manage the huge mass of data will emerge out on the top. The batch processing model has been faithfully serving us for decades. However, it might have reached the end of its usefulness for all but some very specific use-cases. As the pace of businesses increases, most of the time, decision-makers prefer slightly wrong data sooner, than 100% accurate data later. Stream processing – or data streaming – exactly matches this usage: instead of managing the entire bulk of data, manage pieces of them as soon as they become available.

Devclub.lv - Introduction to stream processing

Devclub.lv - Introduction to stream processing

Devclub.lv - Introduction to stream processing

Nicolas Fränkel

Visual Ontology Modeling for Domain Experts and Business Users with metaphactory

Visual Ontology Modeling for Domain Experts and Business Users with metaphactory

Visual Ontology Modeling for Domain Experts and Business Users with metaphactory

Taking Cross References to the Next Level: Reltables for Non-Topic Elements

Taking Cross References to the Next Level: Reltables for Non-Topic Elements

Taking Cross References to the Next Level: Reltables for Non-Topic Elements

Contrext Solutions

From discovering to trusting data

From discovering to trusting data

From discovering to trusting data

Although you may not have heard of JavaScript Object Notation Linked Data (JSON-LD), it is already impacting your business. Search engine giants such as Google have mandated JSON-LD as a preferred means of adding structured data to web pages to make them considerably easier to parse for more accurate search engine results. The Google use case is indicative of the larger capacity for JSON-LD to increase web traffic for sites and better guide users to the results they want. Expectations are high for (JSON-LD), and with good reason. JSON-LD effectively delivers the many benefits of JSON, a lightweight data interchange format, into the linked data world. Linked data is the technological approach supporting the World Wide Web and one of the most effective means of sharing data ever devised. In addition, the growing number of enterprise knowledge graphs fully exploit the potential of JSON-LD as it enables organizations to readily access data stored in document formats and a variety of semi-structured and unstructured data as well. By using this technology to link internal and external data, knowledge graphs exemplify the linked data approach underpinning the growing adoption of JSON-LD—and the demonstrable, recurring business value that linked data consistently provides. Join us learn more about optimizing the unique Document and Graph Database capabilities provided by AllegroGraph to develop or enhance your Enterprise Knowledge Graph using JSON-LD.

Why is JSON-LD Important to Businesses - Franz Inc

Why is JSON-LD Important to Businesses - Franz Inc

Why is JSON-LD Important to Businesses - Franz Inc

Franz Inc. - AllegroGraph

What's hot (20)

Intranet show and_tell_2010

Intranet show and_tell_2010

Intranet show and_tell_2010

Amundsen at Brex and Looker integration

Amundsen at Brex and Looker integration

Amundsen at Brex and Looker integration

Building a Knowledge Graph using NLP and Ontologies

Building a Knowledge Graph using NLP and Ontologies

Building a Knowledge Graph using NLP and Ontologies

Turning search upside down with powerful open source search software

Turning search upside down with powerful open source search software

Turning search upside down with powerful open source search software

AnzoGraph DB - SPARQL 101

AnzoGraph DB - SPARQL 101

AnzoGraph DB - SPARQL 101

TaLend Online Training

TaLend Online Training

TaLend Online Training

Using Spark-Solr at Scale: Productionizing Spark for Search with Apache Solr...

Using Spark-Solr at Scale: Productionizing Spark for Search with Apache Solr...

Using Spark-Solr at Scale: Productionizing Spark for Search with Apache Solr...

20181019 code.talks graph_analytics_k_patenge

20181019 code.talks graph_analytics_k_patenge

20181019 code.talks graph_analytics_k_patenge

Data Integration Solutions Created By Koneksys

Data Integration Solutions Created By Koneksys

Data Integration Solutions Created By Koneksys

Lider Reference Model ld4lt session March, 3rd, 2015

Lider Reference Model ld4lt session March, 3rd, 2015

Lider Reference Model ld4lt session March, 3rd, 2015

Neo4j: What's Under the Hood

Neo4j: What's Under the Hood

Neo4j: What's Under the Hood

Micro-Servicing Linked Data

Micro-Servicing Linked Data

Micro-Servicing Linked Data

CodeOne 2018 - Microservices in action at the Dutch National Police

CodeOne 2018 - Microservices in action at the Dutch National Police

CodeOne 2018 - Microservices in action at the Dutch National Police

Getting Started Contributing to Apache Spark – From PR, CR, JIRA, and Beyond

Getting Started Contributing to Apache Spark – From PR, CR, JIRA, and Beyond

Getting Started Contributing to Apache Spark – From PR, CR, JIRA, and Beyond

Top 5 Considerations When Evaluating NoSQL

Top 5 Considerations When Evaluating NoSQL

Top 5 Considerations When Evaluating NoSQL

Devclub.lv - Introduction to stream processing

Devclub.lv - Introduction to stream processing

Devclub.lv - Introduction to stream processing

Visual Ontology Modeling for Domain Experts and Business Users with metaphactory

Visual Ontology Modeling for Domain Experts and Business Users with metaphactory

Visual Ontology Modeling for Domain Experts and Business Users with metaphactory

Taking Cross References to the Next Level: Reltables for Non-Topic Elements

Taking Cross References to the Next Level: Reltables for Non-Topic Elements

Taking Cross References to the Next Level: Reltables for Non-Topic Elements

From discovering to trusting data

From discovering to trusting data

From discovering to trusting data

Why is JSON-LD Important to Businesses - Franz Inc

Why is JSON-LD Important to Businesses - Franz Inc

Why is JSON-LD Important to Businesses - Franz Inc

Viewers also liked

Flink Case Study: Amadeus

Flink Case Study: Amadeus

Flink Case Study: Amadeus

Hadoop or Spark: is it an either-or proposition? An exodus away from Hadoop to Spark is picking up steam in the news headlines and talks! Away from marketing fluff and politics, this talk analyzes such news and claims from a technical perspective. In practical ways, while referring to components and tools from both Hadoop and Spark ecosystems, this talk will show that the relationship between Hadoop and Spark is not of an either-or type but can take different forms such as: evolution, transition, integration, alternation and complementarity.

Hadoop or Spark: is it an either-or proposition? By Slim Baltagi

Hadoop or Spark: is it an either-or proposition? By Slim Baltagi

Hadoop or Spark: is it an either-or proposition? By Slim Baltagi

Building a Modern Data Architecture with Enterprise Hadoop

Building a Modern Data Architecture with Enterprise Hadoop

Building a Modern Data Architecture with Enterprise Hadoop

Apache Beam (unified Batch and strEAM processing!) is a new Apache incubator project. Originally based on years of experience developing Big Data infrastructure within Google (such as MapReduce, FlumeJava, and MillWheel), it has now been donated to the OSS community at large. Come learn about the fundamentals of out-of-order stream processing, and how Beam’s powerful tools for reasoning about time greatly simplify this complex task. Beam provides a model that allows developers to focus on the four important questions that must be answered by any stream processing pipeline: What results are being calculated? Where in event time are they calculated? When in processing time are they materialized? How do refinements of results relate? Furthermore, by cleanly separating these questions from runtime characteristics, Beam programs become portable across multiple runtime environments, both proprietary (e.g., Google Cloud Dataflow) and open-source (e.g., Flink, Spark, et al).

Fundamentals of Stream Processing with Apache Beam, Tyler Akidau, Frances Perry

Fundamentals of Stream Processing with Apache Beam, Tyler Akidau, Frances Perry

Fundamentals of Stream Processing with Apache Beam, Tyler Akidau, Frances Perry

Aljoscha Krettek - Portable stateful big data processing in Apache Beam

Aljoscha Krettek - Portable stateful big data processing in Apache Beam

Aljoscha Krettek - Portable stateful big data processing in Apache Beam

Apache Beam: A unified model for batch and stream processing data

Apache Beam: A unified model for batch and stream processing data

Apache Beam: A unified model for batch and stream processing data

DataWorks Summit/Hadoop Summit

This talk given at the Hadoop Summit in San Jose on June 28, 2016, analyzes a few major trends in Big Data analytics. These are a few takeaways from this talk: - Adopt Apache Beam for easier development and portability between Big Data Execution Engines. - Adopt stream analytics for faster time to insight, competitive advantages and operational efficiency. - Accelerate your Big Data applications with In-Memory open source tools. - Adopt Rapid Application Development of Big Data applications: APIs, Notebooks, GUIs, Microservices… - Have Machine Learning part of your strategy or passively watch your industry completely transformed! - How to advance your strategy for hybrid integration between cloud and on-premise deployments?

Analysis-of-Major-Trends-in-big-data-analytics-slim-baltagi-hadoop-summit

Analysis-of-Major-Trends-in-big-data-analytics-slim-baltagi-hadoop-summit

Analysis-of-Major-Trends-in-big-data-analytics-slim-baltagi-hadoop-summit

Kafka, Apache Kafka evolved from an enterprise messaging system to a fully distributed streaming data platform (Kafka Core + Kafka Connect + Kafka Streams) for building streaming data pipelines and streaming data applications. This talk, that I gave at the Chicago Java Users Group (CJUG) on June 8th 2017, is mainly focusing on Kafka Streams, a lightweight open source Java library for building stream processing applications on top of Kafka using Kafka topics as input/output. You will learn more about the following: 1. Apache Kafka: a Streaming Data Platform 2. Overview of Kafka Streams: Before Kafka Streams? What is Kafka Streams? Why Kafka Streams? What are Kafka Streams key concepts? Kafka Streams APIs and code examples? 3. Writing, deploying and running your first Kafka Streams application 4. Code and Demo of an end-to-end Kafka-based Streaming Data Application 5. Where to go from here?

Kafka Streams for Java enthusiasts

Kafka Streams for Java enthusiasts

Kafka Streams for Java enthusiasts

This introductory level talk is about Apache Flink: a multi-purpose Big Data analytics framework leading a movement towards the unification of batch and stream processing in the open source. With the many technical innovations it brings along with its unique vision and philosophy, it is considered the 4 G (4th Generation) of Big Data Analytics frameworks providing the only hybrid (Real-Time Streaming + Batch) open source distributed data processing engine supporting many use cases: batch, streaming, relational queries, machine learning and graph processing. In this talk, you will learn about: 1. What is Apache Flink stack and how it fits into the Big Data ecosystem? 2. How Apache Flink integrates with Hadoop and other open source tools for data input and output as well as deployment? 3. Why Apache Flink is an alternative to Apache Hadoop MapReduce, Apache Storm and Apache Spark. 4. Who is using Apache Flink? 5. Where to learn more about Apache Flink?

Apache-Flink-What-How-Why-Who-Where-by-Slim-Baltagi

Apache-Flink-What-How-Why-Who-Where-by-Slim-Baltagi

Apache-Flink-What-How-Why-Who-Where-by-Slim-Baltagi

This face to face talk about Apache Flink in Sao Paulo, Brazil is the first event of its kind in Latin America! It explains how Apache Flink 1.0 announced on March 8th, 2016 by the Apache Software Foundation (link), marks a new era of Big Data analytics and in particular Real-Time streaming analytics. The talk maps Flink's capabilities to real-world use cases that span multiples verticals such as: Financial Services, Healthcare, Advertisement, Oil and Gas, Retail and Telecommunications. In this talk, you learn more about: 1. What is Apache Flink Stack? 2. Batch vs. Streaming Analytics 3. Key Differentiators of Apache Flink for Streaming Analytics 4. Real-World Use Cases with Flink for Streaming Analytics 5. Who is using Flink? 6. Where do you go from here?

Apache Flink: Real-World Use Cases for Streaming Analytics

Apache Flink: Real-World Use Cases for Streaming Analytics

Apache Flink: Real-World Use Cases for Streaming Analytics

Apache Kafka evolved from an enterprise messaging system to a fully distributed streaming data platform for building real-time streaming data pipelines and streaming data applications without the need for other tools/clusters for data ingestion, storage and stream processing. In this talk you will learn more about: 1. A quick introduction to Kafka Core, Kafka Connect and Kafka Streams: What is and why? 2. Code and step-by-step instructions to build an end-to-end streaming data application using Apache Kafka

Building Streaming Data Applications Using Apache Kafka

Building Streaming Data Applications Using Apache Kafka

Building Streaming Data Applications Using Apache Kafka

Flink vs. Spark

Flink vs. Spark

Flink vs. Spark

Marton Balassi – Stateful Stream Processing

Marton Balassi – Stateful Stream Processing

Marton Balassi – Stateful Stream Processing

Matthias J. Sax – A Tale of Squirrels and Storms

Matthias J. Sax – A Tale of Squirrels and Storms

Matthias J. Sax – A Tale of Squirrels and Storms

Apache Flink Training: System Overview

Apache Flink Training: System Overview

Apache Flink Training: System Overview

Apache Flink internals

Apache Flink internals

Apache Flink internals

Ufuc Celebi – Stream & Batch Processing in one System

Ufuc Celebi – Stream & Batch Processing in one System

Ufuc Celebi – Stream & Batch Processing in one System

Apache Flink: API, runtime, and project roadmap

Apache Flink: API, runtime, and project roadmap

Apache Flink: API, runtime, and project roadmap

Martin Junghans – Gradoop: Scalable Graph Analytics with Apache Flink

Martin Junghans – Gradoop: Scalable Graph Analytics with Apache Flink

Martin Junghans – Gradoop: Scalable Graph Analytics with Apache Flink

Apache Flink Training: DataStream API Part 2 Advanced

Apache Flink Training: DataStream API Part 2 Advanced

Apache Flink Training: DataStream API Part 2 Advanced

Viewers also liked (20)

Flink Case Study: Amadeus

Flink Case Study: Amadeus

Flink Case Study: Amadeus

Hadoop or Spark: is it an either-or proposition? By Slim Baltagi

Hadoop or Spark: is it an either-or proposition? By Slim Baltagi

Hadoop or Spark: is it an either-or proposition? By Slim Baltagi

Building a Modern Data Architecture with Enterprise Hadoop

Building a Modern Data Architecture with Enterprise Hadoop

Building a Modern Data Architecture with Enterprise Hadoop

Fundamentals of Stream Processing with Apache Beam, Tyler Akidau, Frances Perry

Fundamentals of Stream Processing with Apache Beam, Tyler Akidau, Frances Perry

Fundamentals of Stream Processing with Apache Beam, Tyler Akidau, Frances Perry

Aljoscha Krettek - Portable stateful big data processing in Apache Beam

Aljoscha Krettek - Portable stateful big data processing in Apache Beam

Aljoscha Krettek - Portable stateful big data processing in Apache Beam

Apache Beam: A unified model for batch and stream processing data

Apache Beam: A unified model for batch and stream processing data

Apache Beam: A unified model for batch and stream processing data

Analysis-of-Major-Trends-in-big-data-analytics-slim-baltagi-hadoop-summit

Analysis-of-Major-Trends-in-big-data-analytics-slim-baltagi-hadoop-summit

Analysis-of-Major-Trends-in-big-data-analytics-slim-baltagi-hadoop-summit

Kafka Streams for Java enthusiasts

Kafka Streams for Java enthusiasts

Kafka Streams for Java enthusiasts

Apache-Flink-What-How-Why-Who-Where-by-Slim-Baltagi

Apache-Flink-What-How-Why-Who-Where-by-Slim-Baltagi

Apache-Flink-What-How-Why-Who-Where-by-Slim-Baltagi

Apache Flink: Real-World Use Cases for Streaming Analytics

Apache Flink: Real-World Use Cases for Streaming Analytics

Apache Flink: Real-World Use Cases for Streaming Analytics

Building Streaming Data Applications Using Apache Kafka

Building Streaming Data Applications Using Apache Kafka

Building Streaming Data Applications Using Apache Kafka

Flink vs. Spark

Flink vs. Spark

Flink vs. Spark

Marton Balassi – Stateful Stream Processing

Marton Balassi – Stateful Stream Processing

Marton Balassi – Stateful Stream Processing

Matthias J. Sax – A Tale of Squirrels and Storms

Matthias J. Sax – A Tale of Squirrels and Storms

Matthias J. Sax – A Tale of Squirrels and Storms

Apache Flink Training: System Overview

Apache Flink Training: System Overview

Apache Flink Training: System Overview

Apache Flink internals

Apache Flink internals

Apache Flink internals

Ufuc Celebi – Stream & Batch Processing in one System

Ufuc Celebi – Stream & Batch Processing in one System

Ufuc Celebi – Stream & Batch Processing in one System

Apache Flink: API, runtime, and project roadmap

Apache Flink: API, runtime, and project roadmap

Apache Flink: API, runtime, and project roadmap

Martin Junghans – Gradoop: Scalable Graph Analytics with Apache Flink

Martin Junghans – Gradoop: Scalable Graph Analytics with Apache Flink

Martin Junghans – Gradoop: Scalable Graph Analytics with Apache Flink

Apache Flink Training: DataStream API Part 2 Advanced

Apache Flink Training: DataStream API Part 2 Advanced

Apache Flink Training: DataStream API Part 2 Advanced

Similar to Flink Case Study: OKKAM

Nicola_Mezzetti_CV_en.pdf

Nicola_Mezzetti_CV_en.pdf

Nicola_Mezzetti_CV_en.pdf

Nicola Mezzetti

Infoproject company overview

Infoproject company overview

Infoproject company overview

PiCToR @ Vrijdag VISdag

PiCToR @ Vrijdag VISdag

PiCToR @ Vrijdag VISdag

Interesting Thing about Internet of Things

Interesting Thing about Internet of Things

Interesting Thing about Internet of Things

Dr. Mazlan Abbas

Make share point the heart of your information 3 ways to extend your investment

Make share point the heart of your information 3 ways to extend your investment

Make share point the heart of your information 3 ways to extend your investment

Jim Merrifield, IGP, CIP

Intranet systems beyond SharePoint in Scandinavia

Intranet systems beyond SharePoint in Scandinavia

Intranet systems beyond SharePoint in Scandinavia

Perttu Tolvanen

Python Certification Course In Ahmedabad

Python Certification Course In Ahmedabad

Python Certification Course In Ahmedabad

sushmitasharan1

SharePoint 2016 is just around the corner, and it's time to start to plan for an upgrade, but also more important is to prepare your organization for SharePoint 2016. Is SharePoint finally ready for Hybrid environments, join this webinar, to learn more about the Hybrid capabilities of SharePoint 2016, and why you should consider Hybrid in your environment. Learn about Hybrid capabilities in SharePoint 2016 Learn when to use hybrid and Learn if SharePoint is finally hybrid capable?"

SPSToronto: SharePoint 2016 - Hybrid, right choice for you and your organizat...

SPSToronto: SharePoint 2016 - Hybrid, right choice for you and your organizat...

SPSToronto: SharePoint 2016 - Hybrid, right choice for you and your organizat...

Knut Relbe-Moe [MVP, MCT]

Python's integration in Fintech Software Development for data integration underscores its adaptability. It tackles challenges like diverse data sources, real-time demands, and regulatory requirements. While offering benefits such as enhanced data quality, cost reduction, and increased agility, talent acquisition remains a challenge. Python's relevance in fintech, amid challenges, signals ongoing innovation in the dynamic realm of financial technology. Trust Python with Fintech Software Development and harness its capabilities with our skilled Software Developers in Fintech. Visit Us: https://www.ficode.co.uk/industry/fintech-app-development

How Python is Tackling Data Integration Challenges in Fintech.pdf

How Python is Tackling Data Integration Challenges in Fintech.pdf

How Python is Tackling Data Integration Challenges in Fintech.pdf

Ficode Technologies Limited

The Ultimate Things About IoT

The Ultimate Things About IoT

The Ultimate Things About IoT

Dr. Mazlan Abbas

Intranet systems beyond SharePoint and the future of SharePoint

Intranet systems beyond SharePoint and the future of SharePoint

Intranet systems beyond SharePoint and the future of SharePoint

Perttu Tolvanen

Python Certification Course In Bangalore

Python Certification Course In Bangalore

Python Certification Course In Bangalore

sushmitasharan1

Records Management strategy and compliance in SharePoint

Records Management strategy and compliance in SharePoint

Records Management strategy and compliance in SharePoint

Enterprise Collaboration using SharePoint

Enterprise Collaboration using SharePoint

Enterprise Collaboration using SharePoint

SharePoint 2010 - Mobility, Browser Compatibility, Compliance, and its Contin...

SharePoint 2010 - Mobility, Browser Compatibility, Compliance, and its Contin...

SharePoint 2010 - Mobility, Browser Compatibility, Compliance, and its Contin...

Data Curation @ SpazioDati - NEXA Lunch Seminar

Data Curation @ SpazioDati - NEXA Lunch Seminar

Data Curation @ SpazioDati - NEXA Lunch Seminar

Dutch IT Outsourcing Intelligence Report 2011

Dutch IT Outsourcing Intelligence Report 2011

Dutch IT Outsourcing Intelligence Report 2011

IT Sourcing Europe

Sharepoint webinar

Sharepoint webinar

Sharepoint webinar

Case Study: From Strategy to Large-scale Change Program

Case Study: From Strategy to Large-scale Change Program

Case Study: From Strategy to Large-scale Change Program

Jorma Myyryläinen

Published by the Finnish Information Processing Association, the yearly IT Barometer charts the importance of IT to Finnish organizations. In the IT Barometer, we study Finnish IT and business management’s views on how IT is utilized in their organizations, how IT produces value for their business, and what factors and competences are seen to contribute to future success. This is the fourth IT Barometer and during these four years, we have seen dramatic changes in IT and the role of IT in Finnish companies. During these four years, we have gone through one downturn and we are now potentially entering another. During 2009, 2010 and 2011, we have monitored the effect that the general economic trend has on IT and perceptions on IT. During these four years, we also have seen the rise of consumerization, including social media services and a new class of smart phones and tablet computers. IT has also undergone a process of consumerization – new services and devices now first come to consumer markets and move from there to corporate use – oftentimes after years of delay.

IT Barometer 2011 - Summary

IT Barometer 2011 - Summary

IT Barometer 2011 - Summary

Similar to Flink Case Study: OKKAM (20)

Nicola_Mezzetti_CV_en.pdf

Nicola_Mezzetti_CV_en.pdf

Nicola_Mezzetti_CV_en.pdf

Infoproject company overview

Infoproject company overview

Infoproject company overview

PiCToR @ Vrijdag VISdag

PiCToR @ Vrijdag VISdag

PiCToR @ Vrijdag VISdag

Interesting Thing about Internet of Things

Interesting Thing about Internet of Things

Interesting Thing about Internet of Things

Make share point the heart of your information 3 ways to extend your investment

Make share point the heart of your information 3 ways to extend your investment

Make share point the heart of your information 3 ways to extend your investment

Intranet systems beyond SharePoint in Scandinavia

Intranet systems beyond SharePoint in Scandinavia

Intranet systems beyond SharePoint in Scandinavia

Python Certification Course In Ahmedabad

Python Certification Course In Ahmedabad

Python Certification Course In Ahmedabad

SPSToronto: SharePoint 2016 - Hybrid, right choice for you and your organizat...

SPSToronto: SharePoint 2016 - Hybrid, right choice for you and your organizat...

SPSToronto: SharePoint 2016 - Hybrid, right choice for you and your organizat...

How Python is Tackling Data Integration Challenges in Fintech.pdf

How Python is Tackling Data Integration Challenges in Fintech.pdf

How Python is Tackling Data Integration Challenges in Fintech.pdf

The Ultimate Things About IoT

The Ultimate Things About IoT

The Ultimate Things About IoT

Intranet systems beyond SharePoint and the future of SharePoint

Intranet systems beyond SharePoint and the future of SharePoint

Intranet systems beyond SharePoint and the future of SharePoint

Python Certification Course In Bangalore

Python Certification Course In Bangalore

Python Certification Course In Bangalore

Records Management strategy and compliance in SharePoint

Records Management strategy and compliance in SharePoint

Records Management strategy and compliance in SharePoint

Enterprise Collaboration using SharePoint

Enterprise Collaboration using SharePoint

Enterprise Collaboration using SharePoint

SharePoint 2010 - Mobility, Browser Compatibility, Compliance, and its Contin...

SharePoint 2010 - Mobility, Browser Compatibility, Compliance, and its Contin...

SharePoint 2010 - Mobility, Browser Compatibility, Compliance, and its Contin...

Data Curation @ SpazioDati - NEXA Lunch Seminar

Data Curation @ SpazioDati - NEXA Lunch Seminar

Data Curation @ SpazioDati - NEXA Lunch Seminar

Dutch IT Outsourcing Intelligence Report 2011

Dutch IT Outsourcing Intelligence Report 2011

Dutch IT Outsourcing Intelligence Report 2011

Sharepoint webinar

Sharepoint webinar

Sharepoint webinar

Case Study: From Strategy to Large-scale Change Program

Case Study: From Strategy to Large-scale Change Program

Case Study: From Strategy to Large-scale Change Program

IT Barometer 2011 - Summary

IT Barometer 2011 - Summary

IT Barometer 2011 - Summary

More from Flink Forward

Apache Flink is a distributed stream processing framework that allows users to process and analyze data in real-time. At LinkedIn, we developed a fully managed stream processing platform on Flink running on K8s to power hundreds of stream processing pipelines in production. This platform is the backbone for other infra systems like Search, Espresso (internal document store) and feature management etc. We provide a rich authoring and testing environment which allows users to create, test, and deploy their streaming jobs in a self-serve fashion within minutes. Users can focus on their business logic, leaving the Flink platform to take care of management aspects such as split deployment, resource provisioning, auto-scaling, job monitoring, alerting, failure recovery and much more. In this talk, we will introduce the overall platform architecture, highlight the unique value propositions that it brings to stream processing at LinkedIn and share the experiences and lessons we have learned.

Building a fully managed stream processing platform on Flink at scale for Lin...

Building a fully managed stream processing platform on Flink at scale for Lin...

Building a fully managed stream processing platform on Flink at scale for Lin...

Flink Forward San Francisco 2022. When running Flink jobs, skew is a common problem that results in wasted resources and limited scalability. In the past years, we have helped our customers and users solve various skew-related issues in their Flink jobs or clusters. In this talk, we will present the different types of skew that users often run into: data skew, key skew, event time skew, state skew, and scheduling skew, and discuss solutions for each of them. We hope this will serve as a guideline to help you reduce skew in your Flink environment. by Jun Qin & Karl Friedrich

Evening out the uneven: dealing with skew in Flink

Evening out the uneven: dealing with skew in Flink

Evening out the uneven: dealing with skew in Flink

Flink Forward San Francisco 2022. To improve Amazon Alexa experiences and support machine learning inference at scale, we built an automated end-to-end solution for incremental model building or fine-tuning machine learning models through continuous learning, continual learning, and/or semi-supervised active learning. Customer privacy is our top concern at Alexa, and as we build solutions, we face unique challenges when operating at scale such as supporting multiple applications with tens of thousands of transactions per second with several dependencies including near-real time inference endpoints at low latencies. Apache Flink helps us transform and discover metrics in near-real time in our solution. In this talk, we will cover the challenges that we faced, how we scale the infrastructure to meet the needs of ML teams across Alexa, and go into how we enable specific use cases that use Apache Flink on Amazon Kinesis Data Analytics to improve Alexa experiences to delight our customers while preserving their privacy. by Aansh Shah

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...

Flink Forward San Francisco 2022. Probably everyone who has written stateful Apache Flink applications has used one of the fault-tolerant keyed state primitives ValueState, ListState, and MapState. With RocksDB, however, retrieving and updating items comes at an increased cost that you should be aware of. Sometimes, these may not be avoidable with the current API, e.g., for efficient event-time stream-sorting or streaming joins where you need to iterate one or two buffered streams in the right order. With FLIP-220, we are introducing a new state primitive: BinarySortedMultiMapState. This new form of state offers you to (a) efficiently store lists of values for a user-provided key, and (b) iterate keyed state in a well-defined sort order. Both features can be backed efficiently by RocksDB with a 2x performance improvement over the current workarounds. This talk will go into the details of the new API and its implementation, present how to use it in your application, and talk about the process of getting it into Flink. by Nico Kruber

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Flink Forward San Francisco 2022. The Apache Flink Kubernetes Operator provides a consistent approach to manage Flink applications automatically, without any human interaction, by extending the Kubernetes API. Given the increasing adoption of Kubernetes based Flink deployments the community has been working on a Kubernetes native solution as part of Flink that can benefit from the rich experience of community members and ultimately make Flink easier to adopt. In this talk we give a technical introduction to the Flink Kubernetes Operator and demonstrate the core features and use-cases through in-depth examples." by Thomas Weise

Introducing the Apache Flink Kubernetes Operator

Introducing the Apache Flink Kubernetes Operator

Introducing the Apache Flink Kubernetes Operator

Flink Forward San Francisco 2022. Resource Elasticity is a frequently requested feature in Apache Flink: Users want to be able to easily adjust their clusters to changing workloads for resource efficiency and cost saving reasons. In Flink 1.13, the initial implementation of Reactive Mode was introduced, later releases added more improvements to make the feature production ready. In this talk, we’ll explain scenarios to deploy Reactive Mode to various environments to achieve autoscaling and resource elasticity. We’ll discuss the constraints to consider when planning to use this feature, and also potential improvements from the Flink roadmap. For those interested in the internals of Flink, we’ll also briefly explain how the feature is implemented, and if time permits, conclude with a short demo. by Robert Metzger

Autoscaling Flink with Reactive Mode

Autoscaling Flink with Reactive Mode

Autoscaling Flink with Reactive Mode

Flink Forward San Francisco 2022. Flink consumers read from Kafka as a scalable, high throughput, and low latency data source. However, there are challenges in scaling out data streams where migration and multiple Kafka clusters are required. Thus, we introduced a new Kafka source to read sharded data across multiple Kafka clusters in a way that conforms well with elastic, dynamic, and reliable infrastructure. In this presentation, we will present the source design and how the solution increases application availability while reducing maintenance toil. Furthermore, we will describe how we extended the existing KafkaSource to provide mechanisms to read logical streams located on multiple clusters, to dynamically adapt to infrastructure changes, and to perform transparent cluster migrations and failover. by Mason Chen

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Flink Forward San Francisco 2022. Next time you want to integrate with a new destination for a demo, concept or production application, the Async Sink framework will bootstrap development, allowing you to move quickly without compromise. In Flink 1.15 we introduced the Async Sink base (FLIP-171), with the goal to encapsulate common logic and allow developers to focus on the key integration code. The new framework handles things like request batching, buffering records, applying backpressure, retry strategies, and at least once semantics. It allows you to focus on your business logic, rather than spending time integrating with your downstream consumers. During the session we will dive deep into the internals to uncover how it works, why it was designed this way, and how to use it. We will code up a new sink from scratch and demonstrate how to quickly push data to a destination. At the end of this talk you will be ready to start implementing your own Flink sink using the new Async Sink framework. by Steffen Hausmann & Danny Cranmer

One sink to rule them all: Introducing the new Async Sink

One sink to rule them all: Introducing the new Async Sink

One sink to rule them all: Introducing the new Async Sink

Flink Forward San Francisco 2022. In normal situations, the default Kafka consumer and producer configuration options work well. But we all know life is not all roses and rainbows and in this session we’ll explore a few knobs that can save the day in atypical scenarios. First, we'll take a detailed look at the parameters available when reading from Kafka. We’ll inspect the params helping us to spot quickly an application lock or crash, the ones that can significantly improve the performance and the ones to touch with gloves since they could cause more harm than benefit. Moreover we’ll explore the partitioning options and discuss when diverging from the default strategy is needed. Next, we’ll discuss the Kafka Sink. After browsing the available options we'll then dive deep into understanding how to approach use cases like sinking enormous records, managing spikes, and handling small but frequent updates.. If you want to understand how to make your application survive when the sky is dark, this session is for you! by Olena Babenko

Tuning Apache Kafka Connectors for Flink.pptx

Tuning Apache Kafka Connectors for Flink.pptx

Tuning Apache Kafka Connectors for Flink.pptx

Flink Forward San Francisco 2022. Pinterest is a visual discovery engine that serves over 433MM users. Stream processing allows us to unlock value from realtime data for pinners. At Pinterest, we adopt Flink as the unified streaming processing engine. In this talk, we will share our journey in building a stream processing platform with Flink and how we onboarding critical use cases to the platform. Pinterest has supported 90+near realtime streaming applications. We will cover the problem statement, how we evaluate potential solutions and our decision to build the framework. by Rainie Li & Kanchi Masalia

Flink powered stream processing platform at Pinterest

Flink powered stream processing platform at Pinterest

Flink powered stream processing platform at Pinterest

Flink Forward San Francisco 2022. This talk will take you on the long journey of Apache Flink into the cloud-native era. It started all the way from where Hadoop and YARN were the standard way of deploying and operating data applications. We're going to deep dive into the cloud-native set of principles and how they map to the Apache Flink internals and recent improvements. We'll cover fast checkpointing, fault tolerance, resource elasticity, minimal infrastructure dependencies, industry-standard tooling, ease of deployment and declarative APIs. After this talk you'll get a broader understanding of the operational requirements for a modern streaming application and where the current limits are. by David Moravek

Apache Flink in the Cloud-Native Era

Apache Flink in the Cloud-Native Era

Apache Flink in the Cloud-Native Era

Flinkn Forward San Francisco 2022. In this talk, we will cover various topics around performance issues that can arise when running a Flink job and how to troubleshoot them. We’ll start with the basics, like understanding what the job is doing and what backpressure is. Next, we will see how to identify bottlenecks and which tools or metrics can be helpful in the process. Finally, we will also discuss potential performance issues during the checkpointing or recovery process, as well as and some tips and Flink features that can speed up checkpointing and recovery times. by Piotr Nowojski

Where is my bottleneck? Performance troubleshooting in Flink

Where is my bottleneck? Performance troubleshooting in Flink

Where is my bottleneck? Performance troubleshooting in Flink

Flink Forward San Francisco 2022. Running natively on Kubernetes, using the new Apache Flink Kubernetes Operator is a great way to deploy and manage Flink application and session deployments. In this presentation, we provide: - A brief overview of Kubernetes operators and their benefits. - Introduce the five levels of the operator maturity model. - Introduce the newly released Apache Flink Kubernetes Operator and FlinkDeployment CRs - Dockerfile modifications you can make to swap out UBI images and Java of the underlying Flink Operator container - Enhancements we're making in: - Versioning/Upgradeability/Stability - Security - Demo of the Apache Flink Operator in-action, with a technical preview of an upcoming product using the Flink Kubernetes Operator. - Lessons learned - Q&A by James Busche & Ted Chang

Using the New Apache Flink Kubernetes Operator in a Production Deployment

Using the New Apache Flink Kubernetes Operator in a Production Deployment

Using the New Apache Flink Kubernetes Operator in a Production Deployment

Flink Forward San Francisco 2022. The Table API is one of the most actively developed components of Flink in recent time. Inspired by databases and SQL, it encapsulates concepts many developers are familiar with. It can be used with both bounded and unbounded streams in a unified way. But from afar it can be difficult to keep track of what this API is capable of and how it relates to Flink's other APIs. In this talk, we will explore the current state of Table API. We will show how it can be used as a batch processor, a changelog processor, or a streaming ETL tool with many built-in functions and operators for deduplicating, joining, and aggregating data. By comparing it to the DataStream API we will highlight differences and elaborate on when to use which API. We will demonstrate hybrid pipelines in which both APIs interact with one another and contribute their unique strengths. Finally, we will take a look at some of the most recent additions as a first step to stateful upgrades. by David Andreson

The Current State of Table API in 2022

The Current State of Table API in 2022

The Current State of Table API in 2022

Flink Forward San Francisco 2022. Based on the new Flink-Pulsar connector, we implemented Flink's TableAPI and Catalog to help users to interact with the Pulsar cluster via Flink SQL easily. We would like to go through the design and implementation of the SQL connector in the following aspects: 1. Two different modes of use Pulsar as a metadata store 2. Data format transformation and management 3. SQL semantics support within Pulsar context by Sijie Guo & Neng Lu

Flink SQL on Pulsar made easy

Flink SQL on Pulsar made easy

Flink SQL on Pulsar made easy

Flink Forward San Francisco 2022. At Bloomberg, we deal with high volumes of real-time market data. Our clients expect to be notified of any anomalies in this market data, which may indicate volatile movements in the markets, notable trades, forthcoming events, or system failures. The parameters for these alerts are always evolving and our clients can update them dynamically. In this talk, we'll cover how we utilized the open source Apache Flink and Siddhi SQL projects to build a distributed, scalable, low-latency and dynamic rule-based, real-time alerting system to solve our clients' needs. We'll also cover the lessons we learned along our journey. by Ajay Vyasapeetam & Madhuri Jain

Dynamic Rule-based Real-time Market Data Alerts

Dynamic Rule-based Real-time Market Data Alerts

Dynamic Rule-based Real-time Market Data Alerts

Flink Forward San Francisco 2022. At Stripe we have created a complete end to end exactly-once processing pipeline to process financial data at scale, by combining the exactly-once power from Flink, Kafka, and Pinot together. The pipeline provides exactly-once guarantee, end-to-end latency within a minute, deduplication against hundreds of billions of keys, and sub-second query latency against the whole dataset with trillion level rows. In this session we will discuss the technical challenges of designing, optimizing, and operating the whole pipeline, including Flink, Kafka, and Pinot. We will also share our lessons learned and the benefits gained from exactly-once processing. by Xiang Zhang & Pratyush Sharma & Xiaoman Dong

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Flink Forward San Francisco 2022. What if my data is already in order? Stream Processing has given us an elegant and powerful solution for running analytic queries and logic over high volumes of continuously arriving data. However, in both Apache Flink and Apache Beam, the notion of time-ordering is baked in at a very low level, making it difficult to express computations that are interested in a semantic-, rather than time-ordering of the data. In financial services, what often matters the most about the data moving between systems is not when the data was created, but in what order, to the extent that many institutions engineer a global sequencing over all data entering and produced by their systems to achieve complete determinism. How, then, can financial institutions and others best employ Stream Processing on streams of data that are already ordered? I will cover various techniques that can make this work, as well as seek input from the community on how Flink might be improved to better support these use-cases. by Patrick Lucas

Processing Semantically-Ordered Streams in Financial Services

Processing Semantically-Ordered Streams in Financial Services

Processing Semantically-Ordered Streams in Financial Services

Flink Forward San Francisco 2022. In modern data platform architectures, stream processing engines such as Apache Flink are used to ingest continuous streams of data into data lakes such as Apache Iceberg. Streaming ingestion to iceberg tables can suffer by two problems (1) small files problem that can hurt read performance (2) poor data clustering that can make file pruning less effective. To address those two problems, we propose adding a shuffling stage to the Flink Iceberg streaming writer. The shuffling stage can intelligently group data via bin packing or range partition. This can reduce the number of concurrent files that every task writes. It can also improve data clustering. In this talk, we will explain the motivations in details and dive into the design of the shuffling stage. We will also share the evaluation results that demonstrate the effectiveness of smart shuffling. by Gang Ye & Steven Wu

Tame the small files problem and optimize data layout for streaming ingestion...

Tame the small files problem and optimize data layout for streaming ingestion...

Tame the small files problem and optimize data layout for streaming ingestion...

Flink Forward San Francisco 2022. Goldman Sachs's Data Lake platform serves as the firm's centralized data platform, ingesting 140K (and growing!) batches per day of Datasets of varying shape and size. Powered by Flink and using metadata configured by platform users, ingestion applications are generated dynamically at runtime to extract, transform, and load data into centralized storage where it is then exported to warehousing solutions such as Sybase IQ, Snowflake, and Amazon Redshift. Data Latency is one of many key considerations as producers and consumers have their own commitments to satisfy. Consumers range from people/systems issuing queries, to applications using engines like Spark, Hive, and Presto to transform data into refined Datasets. Apache Iceberg allows our applications to not only benefit from consistency guarantees important when running on eventually consistent storage like S3, but also allows us the opportunity to improve our batch processing patterns with its scalability-focused features. by Andreas Hailu

Batch Processing at Scale with Flink & Iceberg

Batch Processing at Scale with Flink & Iceberg

Batch Processing at Scale with Flink & Iceberg

More from Flink Forward (20)

Building a fully managed stream processing platform on Flink at scale for Lin...

Building a fully managed stream processing platform on Flink at scale for Lin...

Building a fully managed stream processing platform on Flink at scale for Lin...

Evening out the uneven: dealing with skew in Flink

Evening out the uneven: dealing with skew in Flink

Evening out the uneven: dealing with skew in Flink

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Introducing the Apache Flink Kubernetes Operator

Introducing the Apache Flink Kubernetes Operator

Introducing the Apache Flink Kubernetes Operator

Autoscaling Flink with Reactive Mode

Autoscaling Flink with Reactive Mode

Autoscaling Flink with Reactive Mode

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

One sink to rule them all: Introducing the new Async Sink

One sink to rule them all: Introducing the new Async Sink

One sink to rule them all: Introducing the new Async Sink

Tuning Apache Kafka Connectors for Flink.pptx

Tuning Apache Kafka Connectors for Flink.pptx

Tuning Apache Kafka Connectors for Flink.pptx

Flink powered stream processing platform at Pinterest

Flink powered stream processing platform at Pinterest

Flink powered stream processing platform at Pinterest

Apache Flink in the Cloud-Native Era

Apache Flink in the Cloud-Native Era

Apache Flink in the Cloud-Native Era

Where is my bottleneck? Performance troubleshooting in Flink

Where is my bottleneck? Performance troubleshooting in Flink

Where is my bottleneck? Performance troubleshooting in Flink

Using the New Apache Flink Kubernetes Operator in a Production Deployment

Using the New Apache Flink Kubernetes Operator in a Production Deployment

Using the New Apache Flink Kubernetes Operator in a Production Deployment

The Current State of Table API in 2022

The Current State of Table API in 2022

The Current State of Table API in 2022

Flink SQL on Pulsar made easy

Flink SQL on Pulsar made easy

Flink SQL on Pulsar made easy

Dynamic Rule-based Real-time Market Data Alerts

Dynamic Rule-based Real-time Market Data Alerts

Dynamic Rule-based Real-time Market Data Alerts

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Processing Semantically-Ordered Streams in Financial Services

Processing Semantically-Ordered Streams in Financial Services

Processing Semantically-Ordered Streams in Financial Services

Tame the small files problem and optimize data layout for streaming ingestion...

Tame the small files problem and optimize data layout for streaming ingestion...

Tame the small files problem and optimize data layout for streaming ingestion...

Batch Processing at Scale with Flink & Iceberg

Batch Processing at Scale with Flink & Iceberg

Batch Processing at Scale with Flink & Iceberg

Recently uploaded

Hyatt driving innovation and exceptional customer experiences with FIDO passw...

Hyatt driving innovation and exceptional customer experiences with FIDO passw...

Hyatt driving innovation and exceptional customer experiences with FIDO passw...

Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...

Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...

Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...

Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx

Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx

Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx

In the dynamic field of DevOps, the quest for efficiency and productivity is endless. This talk introduces a revolutionary toolkit: Large Language Models (LLMs), including ChatGPT, Gemini, and Claude, extending far beyond traditional coding assistance. We'll explore how LLMs can automate not just code generation, but also transform day-to-day operations such as crafting compelling cover letters for TPS reports, streamlining client communications, and architecting innovative DevOps solutions. Attendees will learn effective prompting strategies and examine real-life use cases, demonstrating LLMs' potential to redefine productivity in the DevOps landscape. Join us to discover how to harness the power of LLMs for a comprehensive productivity boost across your DevOps activities.

ChatGPT and Beyond - Elevating DevOps Productivity

ChatGPT and Beyond - Elevating DevOps Productivity

ChatGPT and Beyond - Elevating DevOps Productivity

VictorSzoltysek

The presentation was made in “Web3 Fusion: Embracing AI and Beyond” is more than a conference; it's a journey into the heart of digital transformation. The conference a provided a platform where the future of technology meets practical application. This three-day hybrid event, set in the heart of innovation, served as a gateway to the latest trends and transformative discussions in AI, Blockchain, IoT, AR/VR, and their collective impact on the information space.

AI in Action: Real World Use Cases by Anitaraj

AI in Action: Real World Use Cases by Anitaraj

AI in Action: Real World Use Cases by Anitaraj

AI mind or machine power point presentation

AI mind or machine power point presentation

AI mind or machine power point presentation

yogeshlabana357357

AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)

AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)

AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)

In today's digital world, trust is key to customer relationships, but keeping it is a huge challenge. Customers are well-informed and empowered, quick to change brands if their trust is broken, even if it costs them more. This puts a lot of pressure on organizations to handle trust and safety issues with great care and transparency. The challenge, however, is real. Fragmented solutions have left privacy, legal, and security teams in a perpetual cycle of catch-up, struggling to update privacy notices, manage customer data rights, and answer lengthy security questionnaires—all while trying to prove ROI to the business. It's a thankless job, filled with repetition, tedious tasks, and constant interdepartmental coordination. Combine this with fast regulatory changes and the quick evolution of AI, and it becomes overwhelming. Join this webinar to learn more about TrustArc's new innovative solution Trust Center, the only unified, no-code online hub for trust and safety information built for privacy, security, compliance, and legal teams. Trust Center streamlines your path to compliance, shortens the pre-sales cycle, and reduces both legal and regulatory risks, saving time, effort, and cost. This webinar will review: - Why companies are building unified Trust Centers for a robust privacy program. - How unified Trust Centers streamline sales cycles, ensure regulatory compliance, and reduce operational bottlenecks. - How compliance, legal, security, GRC, and privacy teams benefit from a unified Trust Center in terms of needs, pains, and outcomes. - How TrustArc Trust Center saves time and work while reducing legal, reputational, and compliance risk by effectively managing policies, notices, terms, and disclosures, and providing real-time updates on subprocessors.

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...

Vector Search @ sw2con for slideshare.pptx

Vector Search @ sw2con for slideshare.pptx

Vector Search @ sw2con for slideshare.pptx

Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf

Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf

Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf

JohnPollard-hybrid-app-RailsConf2024.pptx

JohnPollard-hybrid-app-RailsConf2024.pptx

JohnPollard-hybrid-app-RailsConf2024.pptx

"The Ultimate Prompt Engineering Guide for Generative AI" provides a comprehensive guide to leveraging the power of AI assistants through effective prompt design. It explores fundamental prompt concepts and details strategies for crafting prompts that maximize output quality. Readers learn about iterative refinement, examples, constraints, and advanced techniques like chaining and decomposition. Case studies demonstrate real-world applications in content creation, coding, analysis, and more. Trends in multimodal, automated, and responsible prompting are also examined. This book is a must-read for anyone seeking to optimize generative AI capabilities.

The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...

The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...

The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...

Introduction to use of FHIR Documents in ABDM

Introduction to use of FHIR Documents in ABDM

Introduction to use of FHIR Documents in ABDM

ERP Contender Series: Acumatica vs. Sage Intacct

ERP Contender Series: Acumatica vs. Sage Intacct

ERP Contender Series: Acumatica vs. Sage Intacct

BrainSell Technologies

Generative AI refers to a class of machine learning algorithms that are designed to generate new data samples that are similar to those in the training data. Unlike traditional AI models that are trained to recognize patterns and make predictions, generative AI models have the ability to create entirely new data based on the patterns they have learned. This is achieved through techniques such as generative adversarial networks (GANs), variational autoencoders (VAEs), and transformer architectures, among others.

Generative AI Use Cases and Applications.pdf

Generative AI Use Cases and Applications.pdf

Generative AI Use Cases and Applications.pdf

alexjohnson7307

In the ever-evolving landscape of data management, Zero-ETL is an approach that is reshaping how businesses handle and integrate their data. This webinar explores Zero-ETL, a paradigm shift from the traditional Extract, Transform, Load (ETL) process, offering a more streamlined, efficient, and real-time data integration method. We will begin with an introduction to the concept of Zero-ETL, including how it allows direct access to data in its native environment and real-time data transformation, providing up-to-date information with significantly reduced data redundancy. Next, we'll take you through several demonstrations showing how Zero-ETL can deliver real-time data and enable the free movement of data between systems. We will also discuss the various tools that support all aspects of Zero-ETL, providing attendees with an understanding of how they can adopt this innovative approach in their organizations. Lastly, the session will conclude with an interactive Q&A segment, allowing participants to gain deeper insights into how Zero-ETL can be tailored to their specific business needs and how they can get started today. Join us to discover how Zero-ETL can elevate your organization's data strategy.

The Zero-ETL Approach: Enhancing Data Agility and Insight

The Zero-ETL Approach: Enhancing Data Agility and Insight

The Zero-ETL Approach: Enhancing Data Agility and Insight

ADP Passwordless Journey Case Study.pptx

ADP Passwordless Journey Case Study.pptx

ADP Passwordless Journey Case Study.pptx

Tracing the root cause of a performance issue requires a lot of patience, experience, and focus. It’s so hard that we sometimes attempt to guess by trying out tentative fixes, but that usually results in frustration, messy code, and a considerable waste of time and money. This talk explains how to correctly zoom in on a performance bottleneck using three levels of profiling: distributed tracing, metrics, and method profiling. After we learn to read the JVM profiler output as a flame graph, we explore a series of bottlenecks typical for backend systems, like connection/thread pool starvation, invisible aspects, blocking code, hot CPU methods, lock contention, and Virtual Thread pinning, and we learn to trace them even if they occur in library code you are not familiar with. Attend this talk and prepare for the performance issues that will eventually hit any successful system. About authorWith two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Discover the top CodeIgniter development companies that can elevate your project to new heights. Our blog explores the best firms known for their expertise in CodeIgniter framework development. From robust web applications to scalable solutions, these companies deliver excellence. Whether you're a startup or an enterprise, find the perfect match for your development needs on Top CSS Gallery's blog.

Top 10 CodeIgniter Development Companies

Top 10 CodeIgniter Development Companies

Top 10 CodeIgniter Development Companies

Tales from a Passkey Provider Progress from Awareness to Implementation.pptx

Tales from a Passkey Provider Progress from Awareness to Implementation.pptx

Tales from a Passkey Provider Progress from Awareness to Implementation.pptx

Recently uploaded (20)

Hyatt driving innovation and exceptional customer experiences with FIDO passw...

Hyatt driving innovation and exceptional customer experiences with FIDO passw...

Hyatt driving innovation and exceptional customer experiences with FIDO passw...

Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...

Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...

Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...

Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx

Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx

Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx

ChatGPT and Beyond - Elevating DevOps Productivity

ChatGPT and Beyond - Elevating DevOps Productivity

ChatGPT and Beyond - Elevating DevOps Productivity

AI in Action: Real World Use Cases by Anitaraj

AI in Action: Real World Use Cases by Anitaraj

AI in Action: Real World Use Cases by Anitaraj

AI mind or machine power point presentation

AI mind or machine power point presentation

AI mind or machine power point presentation

AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)

AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)

AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...

Vector Search @ sw2con for slideshare.pptx

Vector Search @ sw2con for slideshare.pptx

Vector Search @ sw2con for slideshare.pptx

Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf

Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf

Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf

JohnPollard-hybrid-app-RailsConf2024.pptx

JohnPollard-hybrid-app-RailsConf2024.pptx

JohnPollard-hybrid-app-RailsConf2024.pptx

The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...

The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...

The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...

Introduction to use of FHIR Documents in ABDM

Introduction to use of FHIR Documents in ABDM

Introduction to use of FHIR Documents in ABDM

ERP Contender Series: Acumatica vs. Sage Intacct

ERP Contender Series: Acumatica vs. Sage Intacct

ERP Contender Series: Acumatica vs. Sage Intacct

Generative AI Use Cases and Applications.pdf

Generative AI Use Cases and Applications.pdf

Generative AI Use Cases and Applications.pdf

The Zero-ETL Approach: Enhancing Data Agility and Insight

The Zero-ETL Approach: Enhancing Data Agility and Insight

The Zero-ETL Approach: Enhancing Data Agility and Insight

ADP Passwordless Journey Case Study.pptx

ADP Passwordless Journey Case Study.pptx

ADP Passwordless Journey Case Study.pptx

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Top 10 CodeIgniter Development Companies

Top 10 CodeIgniter Development Companies

Top 10 CodeIgniter Development Companies

Tales from a Passkey Provider Progress from Awareness to Implementation.pptx

Tales from a Passkey Provider Progress from Awareness to Implementation.pptx

Tales from a Passkey Provider Progress from Awareness to Implementation.pptx

Flink Case Study: OKKAM

1. A Semantic Big Data Companion Stefano Bortoli bortoli@okkam.it Flavio Pompermaier pompermaier@okkam.it

2. The company (briefly) • Okkam is – a SME based in Trento, Italy. – Started as spin-off of the University of Trento and FBK (2010) • Okkam core business is – large-scale data integration using semantic technologies and an Entity Name System • Okkam operative sectors – Services for public administration – Services for restaurants (and more) – Research projects • FP7, H2020, and Local agencies

3. Who we are • Stefano Bortoli, PhD – works as technical director and researcher at Okkam S.R.L. (Trento, Italy). His research and development interests are in the area of Information Integration, with special focus in entity- centric applications exploiting semantic technologies. • Flavio Pompermaier, MSc. – works as senior software engineer at Okkam S.R.L. (Trento, Italy). Flavio is a passionate developer working with state of the art technologies, combining semantic with big data technologies.

5. Why we need Flink Entiton data model Database record RDF statement Triplestore NOSQL & Index + Quad provenance IRI predicate object object Type Subject local IRI Subject ENS IRI RDF Type Expensive datawearhouse

6. Why we are here • We want to build and manage (very) large entity-centric knowledge bases • We endorsed Flink since Stratosphere as data processing framework (during DOPA FP7) • Our use cases for Apache Flink: – Domain reasoning (Flink + Parquet + Thrift) – RDF data lifecycle (Flink + Parquet + Jena/Sesame ) – RDF data intelligence (Flink + ELKiBi) – Duplicate record detection (Flink + HBase + Solr) – Entiton Record linkage (Flink + MongoDB + Kryo) – Telemetry analysis (Flink + MongoDB + Weka)

7. Come to our session! • We are the last presenting, don’t let us ALONE! • We are hiring! (maybe ;-)