Life is a Stream of Events

•

2 likes•2,855 views

(Bjørn Kvernstuen + Tommy Jocumsen, Norwegian Directorate for Work and Welfare) Kafka Summit SF 2018 NAV (Norwegian Work and Welfare Department) currently distributes more than one third of the national budget to citizens in Norway or abroad. We’re there to assist people through all phases of life within the domains of work, family, health, retirement and social security. Events happening throughout a person’s life determines which services we provide to them, how we provide them and when we provide them. Today, each person has to apply for these services resulting in many tasks that are largely handled manually by various case workers in the organization. Their access to insight and useful information is limited and often hard to find, causing frustration to both our case workers and our users. By streaming a person’s life events through our Kafka pipelines, we can revolutionize the way our users experience our services and the way we work. NAV and the government as a whole have access to vast amounts of data about our citizens, reported by health institutions, employers, various government agencies or the users themselves. Some data is distributed by large batches, while others are available on-demand through APIs. We’re changing these patterns into streams using Kafka, Streams API and Java microservices. We aim to distribute and act on events about birth, death, relationships, employment, income and business processes to vastly improve the user experience, provide real-time insight and reduce the need to apply for services we already know are needed. This talk will touch on the following topics: -How we move from data-on-demand to streams -How streams of life events will free our case workers from mundane tasks -How life and business events make valuable insight -How we protect our users and comply with GDPR -Why we chose Confluent Platform

Technology

Norwegian Labour and Welfare
Administration
Bjørn Frode Kvernstuen
Senior Data Engineer / Tech lead
Bjorn.Frode.Kvernstuen@nav.no
+47 94 05 86 85
Tommy Jocumsen
Senior Data Engineer / Tech lead
Tommy.Lundestad.Jocumsen@nav.no
+47 93 01 36 45
@jocumsen on

Imagine
A government that knows just enough about you to provide services
without you applying for them first

Imagine
A business process with all the data it needs to
start and complete automatically

Imagine
Creating software and insight in an environment where
all the data you need is available at your fingertips

We want
The data to announce their arrival so the various business domains can
trigger processes and update domain-specific data

We want
The data democratized and made available outside of siloes and brick walls

We want
We want to learn more about ourselves and our users in real-time,
and we want the users to learn more about themselves and us, too!

© 2018 Berenice Abbott/Commerce Graphics
Fetch data

Life is a stream of events
Norwegian Labour and Welfare
Administration

CV
Person registry
Employment registry
Income registry
Person registry
NAV.no / CRM client
Person registry Parental benefits
application
Payment processing
system

How streams of life events
will enable innovation and
help eliminate Bureaucracy

Lawmakers use the written word..
..when crafting legislation. Semantics vary a lot.

..and a bureaucratic hierarchy of administrators.
Legislation was turned into procedures, forms..

Digitization has not eliminated bureaucracy..
.. but typically turned paper into digital forms.

Life as a stream of events
A domain system
Other domain systems
No bureaucracy?

Laws:
Data minimization
Purpose limitation
Data portability
Teams of excellence:
Autonomy, agility, CD
Domain driven design
Coupled on data, automation
Users:
Usability
Data once
Outcome oriented

Life as a stream of events
?
?
?
?
! !
!!!
DS1
Data characteristics:
Completeness and Comprehensiveness
Availability and Accessibility
Granularity and Uniqueness
Reliability and Consistency
Timeliness and Relevance
Legitimacy and Validity
Accuracy and Precision
A distributed ecosystem
with useful data
DS2 Ds3
Ds4 Dsn
Ecosystem capabilities:
Catalogs and schemas
Data Quality monitoring
Schema Registry &
Consumer Driven Contracts
Access control & audit
!
Event Sourcing
Governance

Where data was stored
When data was stored
Why data was stored

Personal Events Topic
Employment Events Topic
P
P
C
Does the law permit data usage?
Is the data relevant?
Data lineage events
Delete
Owners consent events
Control
Did the owner consent to usage?

Used by
A welfare state guided by data and insights
Life as a stream of events
! !
!!
!
!
Data about

Project real-life events
as Kafka events
Communicate across
domains strictly using
events
People want to share
information if they get
better services
Build privacy and data
governance into your
architecture early
1 2 3 4

The Enterprise Knowledge Graph is a disruptive platform that combines emerging Big Data and Graph technologies to reinvent knowledge management inside organizations. This platform aims to organize and distribute the organization’s knowledge, and making it centralized and universally accessible to every employee. The Enterprise Knowledge Graph is a central place to structure, simplify and connect the knowledge of an organization. By removing complexity, the knowledge graph brings more transparency, openness and simplicity into organizations. That leads to democratized communications and empowers individuals to share knowledge and to make decisions based on comprehensive knowledge. This platform can change the way we work, challenge the traditional hierarchical approach to get work done and help to unleash human potential!

Semantic AI

Semantic Web Company

Semantic Artificial Intelligence is the fusion of various types of AI, incl. symbolic AI, reasoning, and machine learning techniques like deep learning. At the same time, Semantic AI has a strong focus on data management and data governance. With the 'wedding' of various AI techniques new promises are made, but also fundamental approaches like 'Explainable AI (XAI)', knowledge graphs, or Linked Data are more strongly focused.

Best Practices for Migrating your Data Warehouse to Amazon Redshift

Amazon Web Services

You can gain substantially more business insights and save costs by migrating your existing data warehouse to Amazon Redshift. This session will cover the key benefits of migrating to Amazon Redshift, migration strategies, and tools and resources that can help you in the process. We’ll learn about AWS Database Migration Service and AWS Schema Migration Tool, which were recently enhanced to import data from six common data warehouse platforms.

Intro to new Google cloud technologies: Google Storage, Prediction API, BigQuery

Chris Schalk

RAPIDS Overview

NVIDIA Japan

Knowledge Graphs for Supply Chain Operations.pdf

Vaticle

Agility in supply chain operations has never been so important, especially with today's nonlinear and complex world. That is why companies with supply chains need knowledge graphs. So how do enterprises unleash the power of their own supply chain data to make smarter decisions? This is where bops comes into play. Bops activates supply chain data from existing operating systems (ERPs, Pos, OMS, etc) simplifying how operators optimize working capital in every decision. In this session, bops will showcase a few use cases that portray the power of a knowledge graph to represent a supply chain network composed of an end to end product flow driven by actions among plants, customers and suppliers. Supply chain operations visibility: - Story of a Product and an SKU: from raw material to finished goods track trace & bill of material deviations - Story of a Supplier – risk assessments – “the most influential supplier” - Story of a Process – anomaly detection – “what went wrong?” Join us for a lively discussion to learn how using knowledge graphs is already helping supply chain companies to better collect, unify, and activate their data. Speaker: Jorge Risquez Jorge is the Co-founder and CEO of bops, a headless supply chain intelligence platform helping manufacturers and distributors source, make, and deliver their products, and unlock working capital. Previously, Jorge spent a decade as a Supply Chain Consultant for Deloitte, where he worked with Fortune 500 companies such as Tyson and Cargill. In his spare time, he enjoys going for a run in Central Park and spending time with family and friends.

Architect’s Open-Source Guide for a Data Mesh Architecture

Databricks

Data Mesh is an innovative concept addressing many data challenges from an architectural, cultural, and organizational perspective. But is the world ready to implement Data Mesh? In this session, we will review the importance of core Data Mesh principles, what they can offer, and when it is a good idea to try a Data Mesh architecture. We will discuss common challenges with implementation of Data Mesh systems and focus on the role of open-source projects for it. Projects like Apache Spark can play a key part in standardized infrastructure platform implementation of Data Mesh. We will examine the landscape of useful data engineering open-source projects to utilize in several areas of a Data Mesh system in practice, along with an architectural example. We will touch on what work (culture, tools, mindset) needs to be done to ensure Data Mesh is more accessible for engineers in the industry. The audience will leave with a good understanding of the benefits of Data Mesh architecture, common challenges, and the role of Apache Spark and other open-source projects for its implementation in real systems. This session is targeted for architects, decision-makers, data-engineers, and system designers.

Apache Parquet is among the most commonly used column-oriented data formats in the big data processing space. It leverages various techniques to store data in a CPU- and I/O-efficient way. Furthermore, it has the capabilities to push-down analytical queries on the data to the I/O layer to avoid the loading of nonrelevant data chunks. With various Java and a C++ implementation, Parquet is also the perfect choice to exchange data between different technology stacks. As part of this talk, a general introduction to the format and its techniques will be given. Their benefits and some of the inner workings will be explained to give a better understanding how Parquet achieves its performance. At the end, benchmarks comparing the new C++ & Python implementation with other formats will be shown.

“Powering the Connected Intelligent Edge and the Future of On-Device AI,” a P...

Edge AI and Vision Alliance

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2022/06/powering-the-connected-intelligent-edge-and-the-future-of-on-device-ai-a-presentation-from-qualcomm/ Ziad Asghar, Vice President of Product Management at Qualcomm, presents the “Powering the Connected Intelligent Edge and the Future of On-Device AI” tutorial at the May 2022 Embedded Vision Summit. Qualcomm is leading the realization of the “connected intelligent edge,” where the convergence of wireless connectivity, efficient computing and distributed AI will power the devices and experiences that you deserve. In this talk, Asghar explores some of the key challenges in deploying AI across diverse edge products in markets including mobile, automotive, XR, IoT, robotics and PCs — and some of the important differences in the AI requirements of these applications. Asghar identifies unique AI features that will be needed as physical and digital spaces converge in what is now called the “metaverse”. He highlights key AI technologies offered within Qualcomm products, and how the company connects them to enable the connected intelligent edge. Finally, he shares his vision of the future of on-device AI — including on-device learning, efficient models, state-of-the-art quantization, and how Qualcomm plans to make this vision a reality.

Data pipeline and data lake for autonomous driving

Yu Huang

Oracle Stream Analytics - Developer Introduction

Jeffrey T. Pollock

Best Practices for Building and Deploying Data Pipelines in Apache Spark

Databricks

Many data pipelines share common characteristics and are often built in similar but bespoke ways, even within a single organisation. In this talk, we will outline the key considerations which need to be applied when building data pipelines, such as performance, idempotency, reproducibility, and tackling the small file problem. We’ll work towards describing a common Data Engineering toolkit which separates these concerns from business logic code, allowing non-Data-Engineers (e.g. Business Analysts and Data Scientists) to define data pipelines without worrying about the nitty-gritty production considerations. We’ll then introduce an implementation of such a toolkit in the form of Waimak, our open-source library for Apache Spark (https://github.com/CoxAutomotiveDataSolutions/waimak), which has massively shortened our route from prototype to production. Finally, we’ll define new approaches and best practices about what we believe is the most overlooked aspect of Data Engineering: deploying data pipelines.

[EN] Building modern data pipeline with Snowflake + DBT + Airflow.pdf

Chris Hoyean Song

I'm posting the slide presented at the Snowflake user group meet up. NFT Bank has introduced DBT to rebuild and operate the entire data pipeline from scratch. Data quality control and monitoring are critical as data is at the core of the company. You can manage your numerous data validation tests in organized way. You can add one data validation test with just single line of yaml. You can build the data catalog and data lineage docs if you just implement your data pipeline on top of DBT without big effort. --- Session 1: Data Quality & Productivity Data Quality Data Quality Validation Data Catalog, Lineage Documentation DBT Introduction Session 2: Integrate DBT with Airflow DBT Cloud or Airflow? Astronomer Cosmos dbt deps Session 3: Cost Optimization Query Optimization Cost Monitoring

JupyterHub: Learning at Scale

Carol Willing

Data Modeling & Metadata for Graph Databases

DATAVERSITY

Graph databases are seeing a spike in popularity as their value in leveraging large data sets for key areas such as fraud detection, marketing, and network optimization become increasingly apparent. With graph databases, it’s been said that ‘the data model and the metadata are the database’. What does this mean in a practical application, and how can this technology be optimized for maximum business value?

Cloud-native Semantic Layer on Data Lake

Databricks

Deep Learning Enabled Price Action with Databricks and AWS

Databricks

Enterprise Knowledge Graph

Benjamin Raethlein

by Lukas Masuch, Henning Muszynski and Benjamin Raethlein The Enterprise Knowledge Graph is a disruptive platform that combines emerging Big Data and Graph technologies to reinvent knowledge management inside organizations. This platform aims to organize and distribute the organization’s knowledge, and making it centralized and universally accessible to every employee. The Enterprise Knowledge Graph is a central place to structure, simplify and connect the knowledge of an organization. By removing complexity, the knowledge graph brings more transparency, openness and simplicity into organizations. That leads to democratized communications and empowers individuals to share knowledge and to make decisions based on comprehensive knowledge. This platform can change the way we work, challenge the traditional hierarchical approach to get work done and help to unleash human potential!

Overcoming catastrophic forgetting in neural network

Katy Lee

Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...

Kai Wähner

The Rise of Data in Motion in the Public Sector powered by event streaming with Apache Kafka. Citizen Services: - Health services, e.g. hospital modernization, track & trace - Covid distance control - Public administration - reduce bureaucracy, data democratization across government departments - eGovernment - Efficient and digital citizen engagement, e.g. personal ID application process Smart City - Smart driving, parking, buildings, environment Waste management - Open exchange – e.g. mobility services (1st and 3rd party) Energy - Smart grid and utilities infrastructure (energy distribution, smart home, smart meters, smart water, etc.) - National Security Law enforcement, surveillance, police/interior security data exchange - Defense and military (border control, intelligent solider) Cybersecurity for situational awareness and threat intelligence

Knowledge Graphs Overview

Neo4j

Zipline—Airbnb’s Declarative Feature Engineering Framework

Databricks

Zipline is Airbnb’s data management platform specifically designed for ML use cases. Previously, ML practitioners at Airbnb spent roughly 60% of their time on collecting and writing transformations for machine learning tasks. Zipline reduces this task from months to days – by making the process declarative. It allows data scientists to easily define features in a simple configuration language. The framework then provides access to point-in-time correct features – for both – offline model training and online inference. In this talk we will describe the architecture of our system and the algorithm that makes the problem of efficient point-in-time correct feature generation, tractable. The attendee will learn Importance of point-in-time correct features for achieving better ML model performance Importance of using change data capture for generating feature views An algorithm – to efficiently generate features over change data. We use interval trees to efficiently compress time series features. The algorithm allows generating feature aggregates over this compressed representation. A lambda architecture – that enables using the above algorithm – for online feature generation. A framework, based on category theory, to understand how feature aggregations be distributed, and independently composed. While the talk if fairly technical – we will introduce all the concepts from first principles with examples. Basic understanding of data-parallel distributed computation and machine learning might help, but are not required.

Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...

Kaxil Naik

Machine Learning on Streaming Data using Kafka, Beam, and TensorFlow (Mikhail...

Life is a Stream of Events

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Life is a Stream of Events

Similar to Life is a Stream of Events (20)

More from confluent

More from confluent (20)

Recently uploaded

Recently uploaded (20)

Life is a Stream of Events