Kafka and Kafka Streams in the Global Schibsted Data Platform

•

0 likes•181 views

This document discusses Schibsted's use of Kafka and Kafka Streams in its global data platform. It describes how Schibsted processes streaming data using Kafka Streams to build a new streaming pipeline called Yggdrasil that provides low latency and delivery guarantees. It also discusses how Kafka Connect is used to get data into and out of the platform and how third party analytics tools can connect. The document outlines some challenges experienced in scaling up the platform.

Technology

KafkaandKafkaStreamsinthe
GlobalSchibstedDataPlatform
Fredrik Vraalsen, 16.10.2018

•Data Platform Architect 
City of Oslo
•Former Data Engineer 
Data Platform
FredrikVraalsen
WHOAMI?

SCHIBSTED
22 countries
200 million users/month
20 billion pageviews/month

DATAPIPELINE
Collector
Kinesis
Batch
Storage S3
Piper

STREAMPROCESSING
Collector
Kinesis
Batch
Storage S3
Piper

THEISSUES
• “High” latency (~30 seconds)
• Delivery guarantees
• On-boarding
• Manual configuration
• Homegrown solution

VISION
• Data-driven applications
• Self-serve
• GDPR
• Performance
• State of the art

• Lightweight library
• Streams and Tables
• High-level DSL
• Low level API
KAFKASTREAMS

NEWSTREAMINGPIPELINE
Storage Piper YggdrasilStorage

GETTINGDATAIN&OUT
https://www.conﬂuent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines/

GETTINGDATAIN&OUT
http://kafka.apache.org/documentation.html

GETTINGDATAIN&OUT
DuratroYggdrasilStorage
Event
firehose
Sink
topics

DATADRIVENAPPLICATIONS
https://www.slideshare.net/DataStax/c-for-deep-learning-andrew-jefferson-tracktable-cassandra-summit-2016
https://www.flickr.com/photos/rahulrodriguez/14683524180
https://pixabay.com/en/map-photoshop-geolocation-journey-947471/

BUMPYRIDE
https://pixabay.com/no/veien-pukler-fremover-veiskilt-246/

CHALLENGES&EXPERIENCES
http://www.publicdomainpictures.net/view-image.php?image=6884
https://pixabay.com/en/software-testing-service-762486/
https://pixabay.com/no/brett-l%C3%A6r-note-ferdigheter-597190/
99,5%
99,99976%

SELFSERVE
• Challenge: Transformations and routing
• Multiple configurations
• Who maintains?
• Required Scala knowledge

${ "time": round(parse-time(.published, "yyyy-MM-dd'T'HH:mm:ssX") * 1000), "device_manufacturer": .device.manufacturer, "device_model": .device.model, "language": .device.acceptLanguage, "os_name": .device.osType, "os_version": .device.osVersion, "platform": .device.platformType, "user_properties": { "is_logged_in" : boolean(.actor."spt:userId") } } SELFSERVE • JSLT – DSL for JSON transformation & queries https://github.com/schibsted/jslt$

Oslo Origo
• Digital transformation
• Smarter services
• Data driven

Data Platform – Oslo Origo
https://www.flickr.com/photos/boaski/8079390195

Data Platform – Oslo Origo
• Creating value from our data
• Awareness of opportunities
• Simple and safe data access
• Insights and decision making
• Collaboration and sharing

THANKYOU!
Fredrik Vraalsen
@fredriv
fredrik@vraalsen.no

What's hot

Dr. Michael Stonebraker will share his "one-size-doesn’t-fit-all" perspective when it comes to picking the right tool for the job. He will explain the fast data stack, why traditional RDBMS’s fall short and how a modern in-memory SQL, ACID compliant, database with a scale-out architecture is the right choice for enabling fast data applications. Then John Hugg provides the “proof in the pudding” with a step-by-step review of his Unique Devices application, which performs real-time analytics on fast moving data. It's a representative implementation of the speed layer in the Lambda Architecture with the logic captured in just 30 lines of code.

The Expert Guide to Fast Data

VoltDB

A Walkthrough of InfluxCloud 2.0 by Tim Hall

InfluxData

Red Hat is the provider of enterprise open source solutions. Its portfolio of products includes hybrid cloud infrastructure, middleware, cloud-native apps and automation solutions. Its internal network supports all lines of business — including 60+ sites. Discover how Red Hat uses InfluxDB and Flux for better real-time monitoring of their networks to improve performance and to understand utilization better.

Martin Moucka [Red Hat] | How Red Hat Uses gNMI, Telegraf and InfluxDB to Gai...

InfluxData

Are you looking for a cloud-based architecture that includes the best of breed streaming and database technologies? In this session you will learn how to setup and configure the Confluent Cloud with MongoDB Atlas. We'll start the journey learning about the basic connectivity between the two cloud services and end with a brief discovery of what you can do with data once it is in MongoDB Atlas. By the end of this session you will know how to securely setup and configure the MongoDB Atlas connectors in the Confluent Cloud in both a source and sink configuration.

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

HostedbyConfluent

Will LaForest is the Public Sector CTO for Confluent. In his current position, Will evangelizes the benefits of Apache Kafka, event-driven data in motion architecture, and open-source software is addressing mission challenges in the Government. He has spent 25 years wrangling data at massive scale. His technical career spans diverse areas from software engineering, NoSQL, data science, cloud computing, machine learning, and building statistical visualization software but began with code slinging at DARPA as a teenager. Will holds degrees in mathematics and physics from the University of Virginia. To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.

Scylla Summit 2022: An Odyssey to ScyllaDB and Apache Kafka

ScyllaDB

Webinar (https://www.redhat.com/en/events/webinar/hybrid-cloud-financial-services-series): Balancing immediate business requirements with the ability to provide a consistent enterprise infrastructure can be a challenge for financial services firms. Hybrid cloud infrastructure allows institutions to migrate their existing infrastructure and deploy unified business applications by coordinating tasks running in different environments. This approach provides continuous flexibility, scale, and interoperability. This webinar will cover: - How to apply cloud-native payment architecture solutions to your existing environments. - Best practices for delivering a modernized banking payment architecture for your customers. - A practical review of how payment services can be modernized using hybrid cloud. [ Internal slides link: https://docs.google.com/presentation/d/194oClAU97EaGufMd8pnacqhbEV146EK2DVzoF_S3ZA0/edit?usp=sharing ]

How to use hybrid cloud to migrate and deploy unified business applications i...

Eric D. Schabell

DataXDay - Real-Time Access log analysis

DataXDay Conference by Xebia

"Smooth Operator" [Bay Area NewSQL meetup]

Kevin Xu

Red Hat Summit 2020: Are you all in on the concept of agile integration or just getting your toe in the water? Are you an expert or just getting started with concepts like integration, microservices, message integration, process integration, APIs, and all the things that make your customers experience the best it can be? This session has something for all levels as it walks attendees through the architecture concepts with whiteboard diagrams, easy to grasp images, with a bottom up approach to connecting the dots for the concepts of an integration architecture. Once the groundwork has been laid, the second half of this session take a look at an integration architecture blueprint based on three successful customer integration solutions. Presenting the results of researching these successful solution architectures provides attendees with a clear blueprint for matching to their own architectures, or to help solidify their plans for architecting successful integration solutions.

Concept to reality: An advanced agile integration blueprint

Eric D. Schabell

Data volumes continue to grow, demanding new, more scalable solutions for low-latency data processing. Previously, the default approach to deploying such systems was to throw a ton of hardware at the problem. However, that is no longer necessary, as newer technologies showcase a level of efficiency that enables smaller, more manageable clusters while handling extreme workloads. Processing billions of events per second on Kafka can now be done with a modest investment in compute resources. In this session, you will learn how to architect and build the fastest data processing applications that scale linearly, and combine streaming data and reference data data-in-motion and data-at-rest with machine learning. We will take you through the end-to-end framework and example application, built on the Hazelcast Platform, an open source software engine designed for ultra-fast performance. We will also show how you can leverage SQL to further explore the operational data in the solution including querying Kafka topics and key-value data on the in-memory data store. Attendees will also get access to the Github sample application shown.

Low-latency real-time data processing at giga-scale with Kafka | John DesJard...

HostedbyConfluent

Hermes, Germany's largest post-independent logistics service provider for deliveries, had one main goal—make faster and smarter data-driven business decisions. But with high volumes of diverse and disparate data, how can you effectively leverage it as an asset for real-time insights and business intelligence? During this session, Hermes will share their data challenges and how HVR's high volume data replication capabilities enabled Hermes to securely and seamlessly integrate data into Kafka for real-time decision-making and greater visibility into the entire logistics process.

Achieving Real-Time Analytics at Hermes | Zulf Qureshi, HVR and Dr. Stefan Ro...

HostedbyConfluent

How to Build Real-Time Streaming Analytics with an In-memory, Scale-out SQL D...

VoltDB

Scylla Summit 2022: Scalable and Sustainable Supply Chains with DLT and ScyllaDB

ScyllaDB

(Bob Lehmann, Bayer) Kafka Summit SF 2018 You’ve built your streaming data platform. The early adopters are “all in” and have developed producers, consumers and stream processing apps for a number of use cases. A large percentage of the enterprise, however, has expressed interest but hasn’t made the leap. Why? In 2014, Bayer Crop Science (formerly Monsanto) adopted a cloud first strategy and started a multi-year transition to the cloud. A Kafka-based cross-datacenter DataHub was created to facilitate this migration and to drive the shift to real-time stream processing. The DataHub has seen strong enterprise adoption and supports a myriad of use cases. Data is ingested from a wide variety of sources and the data can move effortlessly between an on premise datacenter, AWS and Google Cloud. The DataHub has evolved continuously over time to meet the current and anticipated needs of our internal customers. The “cost of admission” for the platform has been lowered dramatically over time via our DataHub Portal and technologies such as Kafka Connect, Kubernetes and Presto. Most operations are now self-service, onboarding of new data sources is relatively painless and stream processing via KSQL and other technologies is being incorporated into the core DataHub platform. In this talk, Bob Lehmann will describe the origins and evolution of the Enterprise DataHub with an emphasis on steps that were taken to drive user adoption. Bob will also talk about integrations between the DataHub and other key data platforms at Bayer, lessons learned and the future direction for streaming data and stream processing at Bayer.

Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...

confluent

AI is fundamentally transforming how we live and work. Zalando is a data driven company. We deliver an optimal customer experience that drives engagement. We continue to improve this experience by leveraging the latest technologies and machine learning techniques — such as building a cutting edge cloud based infrastructure to support our operations at scale. We provide our data scientists across Zalando with the means to implement artificial intelligence use cases, leveraging data from all parts of our company and the best machine learning techniques from across the industry. Apache Spark delivered through Databricks is at the core of this strategy. In this keynote, I’ll share our AI journey thus far, and share how we are exploring ways to unify data through A.I. with Spark and Databricks.

The Evolution of the Fashion Retail Industry in the Age of AI with Kshitij Ku...

Databricks

The SQL/NoSQL/NewSQL discussion isn’t a new one, and it continues to be a confused and confusing debate internally at many companies. In this webinar, VoltDB Director of Product Marketing, Dennis Duckworth, and David Rolfe, Director of Solutions Engineering EMEA, at VoltDB will discuss the importance and portability of using a standard, expressive programming language, and how ACID impacts concurrent actions and makes some things most of you are trying to do much easier. They also cover how SQL operational databases deliver correct answers - and why NoSQL can’t without significant cost (like, in performance). Last they will outline how VoltDB customers dramatically reduced the number of servers needed when they switched from open source DIY NoSQL stacks to VoltDB.

Why you really want SQL in a Real-Time Enterprise Environment

VoltDB

How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Using Telegraf, InfluxDB and Mist Mist.io are the creators of Mist, the open source multi-cloud management platform. Their solution helps clients manage the complexities of using a mix of public and private clouds, hypervisors, containers and bare metals. Mist streamlines its customers’ operations and optimizes costs by improving DevOps practices. Discover how Mist achieves that by building on top of Telegraf and InfluxDB. Join this webinar as Chris Psaltis and Dimitris Moraitis dive into: Mist’s approach to improving visibility across environments How a time series database is used to monitor infrastructures Mist’s ability to automate processes and reduce infrastructure costs

How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...

InfluxData

Kafka Summit NYC 2017 - Simplifying Omni-Channel Retail at Scale

confluent

[WSO2Con USA 2018] Microservices, Containers, and Beyond

WSO2

(Dmitry Milman + Ankur Kaneria, Express Scripts) Kafka Summit SF 2018 Building cloud-based microservices can be a challenge when the system of record is a relational database residing on an on-premise mainframe. The challenge lies in the ability to efficiently and cost-effectively access the ever-increasing amount of data. Express Scripts is reimagining its data architecture to bring best-in-class user experience and provide the foundation of next-generation applications. This talk will showcase how Kafka plays a key role within Express Scripts’ transformation from mainframe to a microservice-based ecosystem, ensuring data integrity between two worlds. It will discuss how change data capture (CDC) is leveraged to stream data changes to Kafka, allowing us to build a low-latency data sync pipeline. We will describe how we achieve transactional consistency by collapsing all events that belong together onto a single topic, yet have the ability to scale out to meet the real time SLAs and low-latency requirements through means of partitions. We will share our Kafka Streams configuration to handle the data transformation workload. We will discuss our overall Kafka cluster footprint, configuration and security measures. Express Scripts Holding Company is an American Fortune 100 company. As of 2018, the company is the 25th largest in the U.S. as well as one of the largest pharmacy benefit management organizations in the U.S. Customers rely on 24/7 access to our services, and need the ability to interact with our systems in real time via various channels such as web and mobile. Sharing our mainframe t0 microservices migration journey, our experiences and lessons learned would be beneficial to other companies venturing on a similar path.

Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...

confluent

What's hot (20)