SlideShare a Scribd company logo
1 of 20
Download to read offline
Seite 1Munich, 15th of October, Lucas Meyer
BigData and DevOps:
Powering Postbank’s Data-Driven
Strategy
Lucas Meyer
Oct 15 2019, Application Manager - Big Data | DevOps, Postbank Systems
Seite 2Munich, 15th of October, Lucas Meyer
„We are Postbank Systems!“
Zwischenheadline Us doluptas il int, cumquid
eliquos sequas abore, saere volupta tusamenis se maio. As
seque corehenti odisqui deriat min-verem rest, volupta tempora a
ex et magnis mo ima volupta mendic te sae. Ucienistrum rero
con pedis est am, que es aut eiciis eatur, ipsaped eventuris et,
saepero odicae mosam.
Aliquatur millute ctotam quo consequ untio.
nulleniscit ad erit experume nos dolora si ressi re verum elendem
olupta nobit, omnis aut pra con-sed quiatesendia sant. Ta
dolorum ut vendae ipi-tionsequo quat parum am repedio cus int.
Cerciis ut a volutaque co.
Postbank Systems is a full-service IT provider that centrally bundles the IT
expertise of the Postbank. We offer stable, secure and high-quality IT
systems and services and see ourselves as the backbone of modern
banking.
The goal: to offer our customers reliable IT solutions that are convincing overall.
As part of the Postbank Group, we cover the entire value chain in application
development and operation with around 1,500 colleagues. We master the full
range of technological services
Unique full-service IT provider for the Postbank
Seite 3Munich, 15th of October, Lucas Meyer
Who I am?
Lucas Meyer
• 30 years
• happily married
• 2 children aged 1 and 2 ¾
My main hobbys:
• making music with my band
• OpenSource Smart Home Technologies
Main career steps so far:
▪ With the company Postbank Systems AG since 2008
▪ Operation as system integrator with the focus on provision monitoring solutions
▪ Degree in Business Informatics; Thesis in the Context of BigData Architectures
▪ Internal change ~ 4 years ago from IT – Ops to IT – Project.
Start with the strategic mission to build up BigData skill.
▪ Meanwhile: Lead of the "Elastic Squad".
One of the first, 7 member strong DevOps team’s
▪ Active as "DevOps Evangelist" in various transformation initiatives
Seite 4Munich, 15th of October, Lucas Meyer
2015 2016 2017 2018 2019
Getting started What about a Goal? Technology is one aspect... IT4IT
A small team emerges
and launches the first Use
Case
A Data Lake architecture
is adopted and more Use
Cases are added
Collaboration, organization,
and the General Data
Protection Regulation
(GDPR)
Container technology and
the issue with the logs - a
central logging service
takes its away
Summary & Lessons
Learned
Transformation needs
commitment
Seite 5Munich, 15th of October, Lucas Meyer
2015 – Let‘s give it a try: first use cases, small team
Challenges
Challenges with new technologies
Skill
IT – Operation
Acceptance
Governance
Seite 6Munich, 15th of October, Lucas Meyer
Sales data categorization & cash flow analysis
2015 – Let‘s give it a try: first use cases, small team
Use Case: Smart Banking Analytic Workplace
The idea of a digital
household diary was born
Analysis Tools
Revenues and
purpose of use information
Seite 7Munich, 15th of October, Lucas Meyer
Core NetworkDMZ
KBS DB
MergeCoreData
Apache FarmLoadbalancer InfoneaLogstash ElasticSearch
2015 – Let‘s give it a try: first use cases, small team
Use Case: Monitoring Self Service Terminals
Classical monitoring
ATM Cash Box status
Seite 8Munich, 15th of October, Lucas Meyer
2015 2016 2017 2018 2019
Getting started What about a Goal? Technology is one aspect... IT4IT
A small team emerges
and launches the first Use
Case
A Data Lake architecture
is adopted and more Use
Cases are added
Collaboration, organization,
and the General Data
Protection Regulation
(GDPR)
Container technology and
the issue with the logs - a
central logging service
takes its away
Summary & Lessons
Learned
Transformation needs
commitment
Seite 9Munich, 15th of October, Lucas Meyer
100101001001010010010010
010001010110100101010010
101110010100101001100110
Data Lake
Digital Account Book
& Credit – Worthiness
What is a Data Lake?
Data pool for storing raw data for analysis
and evaluation purposes.
2016 – A Data Lake is born
…on which we could build data driven products
Seite 10Munich, 15th of October, Lucas Meyer
2016 – Business Process Monitoring
Apache Farm Loadbalancer Logstash Elasticsearch Kibana
Availability and performance
Monitoring of the branch client
Dashboards for analytical purposes
Seite 11Munich, 15th of October, Lucas Meyer
2015 2016 2017 2018 2019
Getting started What about a Goal? Technology is one aspect... IT4IT
A small team emerges
and launches the first Use
Case
A Data Lake architecture
is adopted and more Use
Cases are added
Collaboration, organization,
and the General Data
Protection Regulation
(GDPR)
Container technology and
the issue with the logs - a
central logging service
takes its away
Summary & Lessons
Learned
Transformation needs
commitment
Seite 12Munich, 15th of October, Lucas Meyer
2017 – Challenges
The team grows and the different perspectives collide
DevelopmentBusiness Operations
We have nearly no
experience with these
technologies, we cannot
guarantee the business a
SLA?
Big Data technology does not fit into
our existing data center design.
We can't just ignore valid backup
policies just because the data is
supposedly replicated in the cluster.
Before the first line of code is
written, extensive concepts are
written, discussed, reviewed,
changed and reviewed again,
again, again …. that's stress.
Seite 13Munich, 15th of October, Lucas Meyer
2017 – Breaking the Status Quo
Previous project organization is supplemented by platform parts
Elastic Squad Team
We have been developing agilely with
Scrum for some time now. DevOps was
now a real step forward.
I receive immediate feedback as to
whether functional and non-functional
requirements have been met.
As guardians of stability, we have
always seen ourselves as the antithesis
of agile software developers.
Meanwhile I know that this is not true.
I need to become an enabler and not
as a barrier. People like me are needed
more than ever, because: I make sure
that infrastructure is provided as a
service.
What the team shows me at
the end of the sprint are not
only features that work on
their screen but they are also
built in such a way that we
can be productive with them
at any time.
Seite 14Munich, 15th of October, Lucas Meyer
2015 2016 2017 2018 2019
Getting started What about a Goal? Technology is one aspect... IT4IT
A small team emerges
and launches the first Use
Case
A Data Lake architecture
is adopted and more Use
Cases are added
Collaboration, organization,
and the General Data
Protection Regulation
(GDPR)
Container technology and
the issue with the logs - a
central logging service
takes its away
Summary & Lessons
Learned
Transformation needs
commitment
Seite 15Munich, 15th of October, Lucas Meyer
2018 – DSGVO / GDPR
Use Case – Read Access Logging
Elastic Stack for compliant reasons:
Read access log with 2-year retention period
Data Protection Officer access only
Data Lake export functionality of the RALs
Data Lake interface acknowledges after
complete receipt of the data to SAP
Seite 16Munich, 15th of October, Lucas Meyer
Alert:
Alerting in case of anomalies in
the data
2018 – Central Logging Service
Collect & Ship Transform & Enrich Store & Analyze Visualize
Archive & Backup:
Archiving of raw data for
subsequent transformation,
reprocessing in other systems
and/or recovery.
Secure:
Role- and attribute-based access
restriction to the data
Seite 17Munich, 15th of October, Lucas Meyer
2015 2016 2017 2018 2019
Getting started What about a Goal? Technology is one aspect... IT4IT
A small team emerges
and launches the first Use
Case
A Data Lake architecture
is adopted and more Use
Cases are added
Collaboration, organization,
and the General Data
Protection Regulation
(GDPR)
Container technology and
the issue with the logs - a
central logging service
takes its away
Summary & Lessons
Learned
Transformation needs
commitment
Seite 18Munich, 15th of October, Lucas Meyer
Reengineering a Cluster that doesn‘t fit the requirements
Today - Anti Fraud Management
Endpoint Malware
Detecion
Filebeat
EMD
Fraud Detection
System
Filebeat
FDS
Key Components:
Fraud Detection System
Endpoint Malware Detection System
Fraud Management System for Analytics
Seite 19Munich, 15th of October, Lucas Meyer
why elastic
Conclusion
Instances: 20
Pipelines: 20
Logstash
Instances: 50
DataVolume: 70 TB
• 00_Admin: 600 GB
• 01_ICFEM: 150 GB
• 02_SBD: 2,5 TB
• 03_FMSA:65 TB
• 04_RAL: 1,7 TB
• 05_CLS: 500 GB
Elasticsearch
Instances: 20
User: 180
Kibana
Instances: 150
• Metricbeat: 60
• Filebeat: 60
• Packetbeat: 30
• Heartbeat: 10
Beats Kafka
Seite 20Munich, 15th of October, Lucas Meyer
„Start anywhere and improve from there“
(Douglas Adams)
THANK YOU

More Related Content

What's hot

Elastic on a Hyper-Converged Infrastructure for Operational Log Analytics
Elastic on a Hyper-Converged Infrastructure for Operational Log AnalyticsElastic on a Hyper-Converged Infrastructure for Operational Log Analytics
Elastic on a Hyper-Converged Infrastructure for Operational Log AnalyticsElasticsearch
 
IoT meets AI in the Clouds
IoT meets AI in the CloudsIoT meets AI in the Clouds
IoT meets AI in the CloudsDr. Mirko Kämpf
 
Security Events Logging at Bell with the Elastic Stack
Security Events Logging at Bell with the Elastic StackSecurity Events Logging at Bell with the Elastic Stack
Security Events Logging at Bell with the Elastic StackElasticsearch
 
Société Générale: Monitoring as a service with the Elastic Stack
Société Générale: Monitoring as a service with the Elastic StackSociété Générale: Monitoring as a service with the Elastic Stack
Société Générale: Monitoring as a service with the Elastic StackElasticsearch
 
Credit Agricole: Powering banking apps with the Elastic Stack
Credit Agricole: Powering banking apps with the Elastic StackCredit Agricole: Powering banking apps with the Elastic Stack
Credit Agricole: Powering banking apps with the Elastic StackElasticsearch
 
Achieving cyber mission assurance with near real-time impact
Achieving cyber mission assurance with near real-time impactAchieving cyber mission assurance with near real-time impact
Achieving cyber mission assurance with near real-time impactElasticsearch
 
Driving the On-Demand Economy with Spark and Predictive Analytics
Driving the On-Demand Economy with Spark and Predictive AnalyticsDriving the On-Demand Economy with Spark and Predictive Analytics
Driving the On-Demand Economy with Spark and Predictive AnalyticsSingleStore
 
Aeris + Cassandra: An IOT Solution Helping Automakers Make the Connected Car ...
Aeris + Cassandra: An IOT Solution Helping Automakers Make the Connected Car ...Aeris + Cassandra: An IOT Solution Helping Automakers Make the Connected Car ...
Aeris + Cassandra: An IOT Solution Helping Automakers Make the Connected Car ...DataStax
 
Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elasti...
Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elasti...Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elasti...
Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elasti...Elasticsearch
 
Memory Database Technology is Driving a New Cycle of Business Innovation
Memory Database Technology is Driving a New Cycle of Business InnovationMemory Database Technology is Driving a New Cycle of Business Innovation
Memory Database Technology is Driving a New Cycle of Business InnovationVoltDB
 
Combining Logs, Metrics, and Traces for Unified Observability
Combining Logs, Metrics, and Traces for Unified ObservabilityCombining Logs, Metrics, and Traces for Unified Observability
Combining Logs, Metrics, and Traces for Unified ObservabilityElasticsearch
 
Building an IoT Kafka Pipeline in Under 5 Minutes
Building an IoT Kafka Pipeline in Under 5 MinutesBuilding an IoT Kafka Pipeline in Under 5 Minutes
Building an IoT Kafka Pipeline in Under 5 MinutesSingleStore
 
In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017SingleStore
 
Kubernetes Jakarta Meetup 010 - Service Mesh Observability with Kiali
Kubernetes Jakarta Meetup 010 - Service Mesh Observability with KialiKubernetes Jakarta Meetup 010 - Service Mesh Observability with Kiali
Kubernetes Jakarta Meetup 010 - Service Mesh Observability with KialiYusuf Hadiwinata Sutandar
 
Artik cloud deview 2016
Artik cloud   deview 2016Artik cloud   deview 2016
Artik cloud deview 2016NAVER D2
 
Elastic community Abidjan #225 meetup 08 May 2021
Elastic community Abidjan #225 meetup 08 May 2021Elastic community Abidjan #225 meetup 08 May 2021
Elastic community Abidjan #225 meetup 08 May 2021Yassine, LASRI
 
Data Warehouse Modernization Webinar Series- Critical Trends, Implementation ...
Data Warehouse Modernization Webinar Series- Critical Trends, Implementation ...Data Warehouse Modernization Webinar Series- Critical Trends, Implementation ...
Data Warehouse Modernization Webinar Series- Critical Trends, Implementation ...Impetus Technologies
 
CTO View: Driving the On-Demand Economy with Predictive Analytics
CTO View: Driving the On-Demand Economy with Predictive AnalyticsCTO View: Driving the On-Demand Economy with Predictive Analytics
CTO View: Driving the On-Demand Economy with Predictive AnalyticsSingleStore
 

What's hot (20)

Elastic on a Hyper-Converged Infrastructure for Operational Log Analytics
Elastic on a Hyper-Converged Infrastructure for Operational Log AnalyticsElastic on a Hyper-Converged Infrastructure for Operational Log Analytics
Elastic on a Hyper-Converged Infrastructure for Operational Log Analytics
 
IoT meets AI in the Clouds
IoT meets AI in the CloudsIoT meets AI in the Clouds
IoT meets AI in the Clouds
 
Security Events Logging at Bell with the Elastic Stack
Security Events Logging at Bell with the Elastic StackSecurity Events Logging at Bell with the Elastic Stack
Security Events Logging at Bell with the Elastic Stack
 
Société Générale: Monitoring as a service with the Elastic Stack
Société Générale: Monitoring as a service with the Elastic StackSociété Générale: Monitoring as a service with the Elastic Stack
Société Générale: Monitoring as a service with the Elastic Stack
 
Credit Agricole: Powering banking apps with the Elastic Stack
Credit Agricole: Powering banking apps with the Elastic StackCredit Agricole: Powering banking apps with the Elastic Stack
Credit Agricole: Powering banking apps with the Elastic Stack
 
Achieving cyber mission assurance with near real-time impact
Achieving cyber mission assurance with near real-time impactAchieving cyber mission assurance with near real-time impact
Achieving cyber mission assurance with near real-time impact
 
Driving the On-Demand Economy with Spark and Predictive Analytics
Driving the On-Demand Economy with Spark and Predictive AnalyticsDriving the On-Demand Economy with Spark and Predictive Analytics
Driving the On-Demand Economy with Spark and Predictive Analytics
 
Aeris + Cassandra: An IOT Solution Helping Automakers Make the Connected Car ...
Aeris + Cassandra: An IOT Solution Helping Automakers Make the Connected Car ...Aeris + Cassandra: An IOT Solution Helping Automakers Make the Connected Car ...
Aeris + Cassandra: An IOT Solution Helping Automakers Make the Connected Car ...
 
Zero Downtime App Deployment using Hadoop
Zero Downtime App Deployment using HadoopZero Downtime App Deployment using Hadoop
Zero Downtime App Deployment using Hadoop
 
Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elasti...
Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elasti...Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elasti...
Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elasti...
 
Memory Database Technology is Driving a New Cycle of Business Innovation
Memory Database Technology is Driving a New Cycle of Business InnovationMemory Database Technology is Driving a New Cycle of Business Innovation
Memory Database Technology is Driving a New Cycle of Business Innovation
 
Combining Logs, Metrics, and Traces for Unified Observability
Combining Logs, Metrics, and Traces for Unified ObservabilityCombining Logs, Metrics, and Traces for Unified Observability
Combining Logs, Metrics, and Traces for Unified Observability
 
Building an IoT Kafka Pipeline in Under 5 Minutes
Building an IoT Kafka Pipeline in Under 5 MinutesBuilding an IoT Kafka Pipeline in Under 5 Minutes
Building an IoT Kafka Pipeline in Under 5 Minutes
 
Ford
FordFord
Ford
 
In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017
 
Kubernetes Jakarta Meetup 010 - Service Mesh Observability with Kiali
Kubernetes Jakarta Meetup 010 - Service Mesh Observability with KialiKubernetes Jakarta Meetup 010 - Service Mesh Observability with Kiali
Kubernetes Jakarta Meetup 010 - Service Mesh Observability with Kiali
 
Artik cloud deview 2016
Artik cloud   deview 2016Artik cloud   deview 2016
Artik cloud deview 2016
 
Elastic community Abidjan #225 meetup 08 May 2021
Elastic community Abidjan #225 meetup 08 May 2021Elastic community Abidjan #225 meetup 08 May 2021
Elastic community Abidjan #225 meetup 08 May 2021
 
Data Warehouse Modernization Webinar Series- Critical Trends, Implementation ...
Data Warehouse Modernization Webinar Series- Critical Trends, Implementation ...Data Warehouse Modernization Webinar Series- Critical Trends, Implementation ...
Data Warehouse Modernization Webinar Series- Critical Trends, Implementation ...
 
CTO View: Driving the On-Demand Economy with Predictive Analytics
CTO View: Driving the On-Demand Economy with Predictive AnalyticsCTO View: Driving the On-Demand Economy with Predictive Analytics
CTO View: Driving the On-Demand Economy with Predictive Analytics
 

Similar to Powering Postbank Group’s Data-driven Strategy

Managing Valuable Ip Assets Owned By Their Clients Essay
Managing Valuable Ip Assets Owned By Their Clients EssayManaging Valuable Ip Assets Owned By Their Clients Essay
Managing Valuable Ip Assets Owned By Their Clients EssayJessica Howard
 
It%40arts update may2013_v6.2
It%40arts update may2013_v6.2It%40arts update may2013_v6.2
It%40arts update may2013_v6.2SarahKante
 
How a Time Series Database Contributes to a Decentralized Cloud Object Storag...
How a Time Series Database Contributes to a Decentralized Cloud Object Storag...How a Time Series Database Contributes to a Decentralized Cloud Object Storag...
How a Time Series Database Contributes to a Decentralized Cloud Object Storag...InfluxData
 
IBM Big Data Analytics Concepts and Use Cases
IBM Big Data Analytics Concepts and Use CasesIBM Big Data Analytics Concepts and Use Cases
IBM Big Data Analytics Concepts and Use CasesTony Pearson
 
Data Culture Keynote and Exec Track Birm Dec 8th
Data Culture Keynote and Exec Track Birm Dec 8thData Culture Keynote and Exec Track Birm Dec 8th
Data Culture Keynote and Exec Track Birm Dec 8thJonathan Woodward
 
OpsStack Overview 20170806.1
OpsStack Overview 20170806.1OpsStack Overview 20170806.1
OpsStack Overview 20170806.1Siglos
 
AIOps Roundtable Munich 2018
AIOps Roundtable Munich 2018AIOps Roundtable Munich 2018
AIOps Roundtable Munich 2018Splunk
 
Fiducia & GAD IT AG: From Fraud Detection to Big Data Platform: Bringing Hado...
Fiducia & GAD IT AG: From Fraud Detection to Big Data Platform: Bringing Hado...Fiducia & GAD IT AG: From Fraud Detection to Big Data Platform: Bringing Hado...
Fiducia & GAD IT AG: From Fraud Detection to Big Data Platform: Bringing Hado...Seeling Cheung
 
BIG DATA, a new way to achieve success in Enterprise Architecture.
BIG DATA, a new way to achieve success in Enterprise Architecture.BIG DATA, a new way to achieve success in Enterprise Architecture.
BIG DATA, a new way to achieve success in Enterprise Architecture.Georges Colin
 
Simply Business' Data Platform
Simply Business' Data PlatformSimply Business' Data Platform
Simply Business' Data PlatformDani Solà Lagares
 
The Growth Of Data Centers
The Growth Of Data CentersThe Growth Of Data Centers
The Growth Of Data CentersGina Buck
 
[Webinar] Measure Twice, Build Once: Real-Time Predictive Analytics
[Webinar] Measure Twice, Build Once: Real-Time Predictive Analytics[Webinar] Measure Twice, Build Once: Real-Time Predictive Analytics
[Webinar] Measure Twice, Build Once: Real-Time Predictive AnalyticsInfochimps, a CSC Big Data Business
 
Eecs6893 big dataanalytics-lecture1
Eecs6893 big dataanalytics-lecture1Eecs6893 big dataanalytics-lecture1
Eecs6893 big dataanalytics-lecture1Aravindharamanan S
 
From IoT to IoTA
From IoT to IoTAFrom IoT to IoTA
From IoT to IoTAStriim
 
Tyler Rutschman- Kansas City
Tyler Rutschman- Kansas CityTyler Rutschman- Kansas City
Tyler Rutschman- Kansas CitySplunk
 
Data Engineer's Lunch #85: Designing a Modern Data Stack
Data Engineer's Lunch #85: Designing a Modern Data StackData Engineer's Lunch #85: Designing a Modern Data Stack
Data Engineer's Lunch #85: Designing a Modern Data StackAnant Corporation
 
Unify Data at Memory Speed
Unify Data at Memory SpeedUnify Data at Memory Speed
Unify Data at Memory SpeedAlluxio, Inc.
 
How to Bring Shadow IT to the Light
How to Bring Shadow IT to the LightHow to Bring Shadow IT to the Light
How to Bring Shadow IT to the LightRackspace
 
Smarter Analytics: Supporting the Enterprise with Automation
Smarter Analytics: Supporting the Enterprise with AutomationSmarter Analytics: Supporting the Enterprise with Automation
Smarter Analytics: Supporting the Enterprise with AutomationInside Analysis
 
How to build and run a big data platform in the 21st century
How to build and run a big data platform in the 21st centuryHow to build and run a big data platform in the 21st century
How to build and run a big data platform in the 21st centuryAli Dasdan
 

Similar to Powering Postbank Group’s Data-driven Strategy (20)

Managing Valuable Ip Assets Owned By Their Clients Essay
Managing Valuable Ip Assets Owned By Their Clients EssayManaging Valuable Ip Assets Owned By Their Clients Essay
Managing Valuable Ip Assets Owned By Their Clients Essay
 
It%40arts update may2013_v6.2
It%40arts update may2013_v6.2It%40arts update may2013_v6.2
It%40arts update may2013_v6.2
 
How a Time Series Database Contributes to a Decentralized Cloud Object Storag...
How a Time Series Database Contributes to a Decentralized Cloud Object Storag...How a Time Series Database Contributes to a Decentralized Cloud Object Storag...
How a Time Series Database Contributes to a Decentralized Cloud Object Storag...
 
IBM Big Data Analytics Concepts and Use Cases
IBM Big Data Analytics Concepts and Use CasesIBM Big Data Analytics Concepts and Use Cases
IBM Big Data Analytics Concepts and Use Cases
 
Data Culture Keynote and Exec Track Birm Dec 8th
Data Culture Keynote and Exec Track Birm Dec 8thData Culture Keynote and Exec Track Birm Dec 8th
Data Culture Keynote and Exec Track Birm Dec 8th
 
OpsStack Overview 20170806.1
OpsStack Overview 20170806.1OpsStack Overview 20170806.1
OpsStack Overview 20170806.1
 
AIOps Roundtable Munich 2018
AIOps Roundtable Munich 2018AIOps Roundtable Munich 2018
AIOps Roundtable Munich 2018
 
Fiducia & GAD IT AG: From Fraud Detection to Big Data Platform: Bringing Hado...
Fiducia & GAD IT AG: From Fraud Detection to Big Data Platform: Bringing Hado...Fiducia & GAD IT AG: From Fraud Detection to Big Data Platform: Bringing Hado...
Fiducia & GAD IT AG: From Fraud Detection to Big Data Platform: Bringing Hado...
 
BIG DATA, a new way to achieve success in Enterprise Architecture.
BIG DATA, a new way to achieve success in Enterprise Architecture.BIG DATA, a new way to achieve success in Enterprise Architecture.
BIG DATA, a new way to achieve success in Enterprise Architecture.
 
Simply Business' Data Platform
Simply Business' Data PlatformSimply Business' Data Platform
Simply Business' Data Platform
 
The Growth Of Data Centers
The Growth Of Data CentersThe Growth Of Data Centers
The Growth Of Data Centers
 
[Webinar] Measure Twice, Build Once: Real-Time Predictive Analytics
[Webinar] Measure Twice, Build Once: Real-Time Predictive Analytics[Webinar] Measure Twice, Build Once: Real-Time Predictive Analytics
[Webinar] Measure Twice, Build Once: Real-Time Predictive Analytics
 
Eecs6893 big dataanalytics-lecture1
Eecs6893 big dataanalytics-lecture1Eecs6893 big dataanalytics-lecture1
Eecs6893 big dataanalytics-lecture1
 
From IoT to IoTA
From IoT to IoTAFrom IoT to IoTA
From IoT to IoTA
 
Tyler Rutschman- Kansas City
Tyler Rutschman- Kansas CityTyler Rutschman- Kansas City
Tyler Rutschman- Kansas City
 
Data Engineer's Lunch #85: Designing a Modern Data Stack
Data Engineer's Lunch #85: Designing a Modern Data StackData Engineer's Lunch #85: Designing a Modern Data Stack
Data Engineer's Lunch #85: Designing a Modern Data Stack
 
Unify Data at Memory Speed
Unify Data at Memory SpeedUnify Data at Memory Speed
Unify Data at Memory Speed
 
How to Bring Shadow IT to the Light
How to Bring Shadow IT to the LightHow to Bring Shadow IT to the Light
How to Bring Shadow IT to the Light
 
Smarter Analytics: Supporting the Enterprise with Automation
Smarter Analytics: Supporting the Enterprise with AutomationSmarter Analytics: Supporting the Enterprise with Automation
Smarter Analytics: Supporting the Enterprise with Automation
 
How to build and run a big data platform in the 21st century
How to build and run a big data platform in the 21st centuryHow to build and run a big data platform in the 21st century
How to build and run a big data platform in the 21st century
 

More from Elasticsearch

An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxAn introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxElasticsearch
 
From MSP to MSSP using Elastic
From MSP to MSSP using ElasticFrom MSP to MSSP using Elastic
From MSP to MSSP using ElasticElasticsearch
 
Cómo crear excelentes experiencias de búsqueda en sitios web
Cómo crear excelentes experiencias de búsqueda en sitios webCómo crear excelentes experiencias de búsqueda en sitios web
Cómo crear excelentes experiencias de búsqueda en sitios webElasticsearch
 
Te damos la bienvenida a una nueva forma de realizar búsquedas
Te damos la bienvenida a una nueva forma de realizar búsquedas Te damos la bienvenida a una nueva forma de realizar búsquedas
Te damos la bienvenida a una nueva forma de realizar búsquedas Elasticsearch
 
Tirez pleinement parti d'Elastic grâce à Elastic Cloud
Tirez pleinement parti d'Elastic grâce à Elastic CloudTirez pleinement parti d'Elastic grâce à Elastic Cloud
Tirez pleinement parti d'Elastic grâce à Elastic CloudElasticsearch
 
Comment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesComment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesElasticsearch
 
Plongez au cœur de la recherche dans tous ses états.
Plongez au cœur de la recherche dans tous ses états.Plongez au cœur de la recherche dans tous ses états.
Plongez au cœur de la recherche dans tous ses états.Elasticsearch
 
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]Elasticsearch
 
An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxAn introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxElasticsearch
 
Welcome to a new state of find
Welcome to a new state of findWelcome to a new state of find
Welcome to a new state of findElasticsearch
 
Building great website search experiences
Building great website search experiencesBuilding great website search experiences
Building great website search experiencesElasticsearch
 
Keynote: Harnessing the power of Elasticsearch for simplified search
Keynote: Harnessing the power of Elasticsearch for simplified searchKeynote: Harnessing the power of Elasticsearch for simplified search
Keynote: Harnessing the power of Elasticsearch for simplified searchElasticsearch
 
Cómo transformar los datos en análisis con los que tomar decisiones
Cómo transformar los datos en análisis con los que tomar decisionesCómo transformar los datos en análisis con los que tomar decisiones
Cómo transformar los datos en análisis con los que tomar decisionesElasticsearch
 
Explore relève les défis Big Data avec Elastic Cloud
Explore relève les défis Big Data avec Elastic Cloud Explore relève les défis Big Data avec Elastic Cloud
Explore relève les défis Big Data avec Elastic Cloud Elasticsearch
 
Comment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesComment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesElasticsearch
 
Transforming data into actionable insights
Transforming data into actionable insightsTransforming data into actionable insights
Transforming data into actionable insightsElasticsearch
 
Opening Keynote: Why Elastic?
Opening Keynote: Why Elastic?Opening Keynote: Why Elastic?
Opening Keynote: Why Elastic?Elasticsearch
 
Empowering agencies using Elastic as a Service inside Government
Empowering agencies using Elastic as a Service inside GovernmentEmpowering agencies using Elastic as a Service inside Government
Empowering agencies using Elastic as a Service inside GovernmentElasticsearch
 
The opportunities and challenges of data for public good
The opportunities and challenges of data for public goodThe opportunities and challenges of data for public good
The opportunities and challenges of data for public goodElasticsearch
 
Enterprise search and unstructured data with CGI and Elastic
Enterprise search and unstructured data with CGI and ElasticEnterprise search and unstructured data with CGI and Elastic
Enterprise search and unstructured data with CGI and ElasticElasticsearch
 

More from Elasticsearch (20)

An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxAn introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolbox
 
From MSP to MSSP using Elastic
From MSP to MSSP using ElasticFrom MSP to MSSP using Elastic
From MSP to MSSP using Elastic
 
Cómo crear excelentes experiencias de búsqueda en sitios web
Cómo crear excelentes experiencias de búsqueda en sitios webCómo crear excelentes experiencias de búsqueda en sitios web
Cómo crear excelentes experiencias de búsqueda en sitios web
 
Te damos la bienvenida a una nueva forma de realizar búsquedas
Te damos la bienvenida a una nueva forma de realizar búsquedas Te damos la bienvenida a una nueva forma de realizar búsquedas
Te damos la bienvenida a una nueva forma de realizar búsquedas
 
Tirez pleinement parti d'Elastic grâce à Elastic Cloud
Tirez pleinement parti d'Elastic grâce à Elastic CloudTirez pleinement parti d'Elastic grâce à Elastic Cloud
Tirez pleinement parti d'Elastic grâce à Elastic Cloud
 
Comment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesComment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitables
 
Plongez au cœur de la recherche dans tous ses états.
Plongez au cœur de la recherche dans tous ses états.Plongez au cœur de la recherche dans tous ses états.
Plongez au cœur de la recherche dans tous ses états.
 
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
 
An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxAn introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolbox
 
Welcome to a new state of find
Welcome to a new state of findWelcome to a new state of find
Welcome to a new state of find
 
Building great website search experiences
Building great website search experiencesBuilding great website search experiences
Building great website search experiences
 
Keynote: Harnessing the power of Elasticsearch for simplified search
Keynote: Harnessing the power of Elasticsearch for simplified searchKeynote: Harnessing the power of Elasticsearch for simplified search
Keynote: Harnessing the power of Elasticsearch for simplified search
 
Cómo transformar los datos en análisis con los que tomar decisiones
Cómo transformar los datos en análisis con los que tomar decisionesCómo transformar los datos en análisis con los que tomar decisiones
Cómo transformar los datos en análisis con los que tomar decisiones
 
Explore relève les défis Big Data avec Elastic Cloud
Explore relève les défis Big Data avec Elastic Cloud Explore relève les défis Big Data avec Elastic Cloud
Explore relève les défis Big Data avec Elastic Cloud
 
Comment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesComment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitables
 
Transforming data into actionable insights
Transforming data into actionable insightsTransforming data into actionable insights
Transforming data into actionable insights
 
Opening Keynote: Why Elastic?
Opening Keynote: Why Elastic?Opening Keynote: Why Elastic?
Opening Keynote: Why Elastic?
 
Empowering agencies using Elastic as a Service inside Government
Empowering agencies using Elastic as a Service inside GovernmentEmpowering agencies using Elastic as a Service inside Government
Empowering agencies using Elastic as a Service inside Government
 
The opportunities and challenges of data for public good
The opportunities and challenges of data for public goodThe opportunities and challenges of data for public good
The opportunities and challenges of data for public good
 
Enterprise search and unstructured data with CGI and Elastic
Enterprise search and unstructured data with CGI and ElasticEnterprise search and unstructured data with CGI and Elastic
Enterprise search and unstructured data with CGI and Elastic
 

Recently uploaded

"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsMiki Katsuragi
 
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks..."LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...Fwdays
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationSlibray Presentation
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Enterprise Knowledge
 
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 3652toLead Limited
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr LapshynFwdays
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time ClashPowerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clashcharlottematthew16
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Wonjun Hwang
 
Understanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitectureUnderstanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitecturePixlogix Infotech
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
costume and set research powerpoint presentation
costume and set research powerpoint presentationcostume and set research powerpoint presentation
costume and set research powerpoint presentationphoebematthew05
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machinePadma Pradeep
 

Recently uploaded (20)

"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering Tips
 
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks..."LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck Presentation
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024
 
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
Powerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time ClashPowerpoint exploring the locations used in television show Time Clash
Powerpoint exploring the locations used in television show Time Clash
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
 
Understanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitectureUnderstanding the Laravel MVC Architecture
Understanding the Laravel MVC Architecture
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
costume and set research powerpoint presentation
costume and set research powerpoint presentationcostume and set research powerpoint presentation
costume and set research powerpoint presentation
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
 

Powering Postbank Group’s Data-driven Strategy

  • 1. Seite 1Munich, 15th of October, Lucas Meyer BigData and DevOps: Powering Postbank’s Data-Driven Strategy Lucas Meyer Oct 15 2019, Application Manager - Big Data | DevOps, Postbank Systems
  • 2. Seite 2Munich, 15th of October, Lucas Meyer „We are Postbank Systems!“ Zwischenheadline Us doluptas il int, cumquid eliquos sequas abore, saere volupta tusamenis se maio. As seque corehenti odisqui deriat min-verem rest, volupta tempora a ex et magnis mo ima volupta mendic te sae. Ucienistrum rero con pedis est am, que es aut eiciis eatur, ipsaped eventuris et, saepero odicae mosam. Aliquatur millute ctotam quo consequ untio. nulleniscit ad erit experume nos dolora si ressi re verum elendem olupta nobit, omnis aut pra con-sed quiatesendia sant. Ta dolorum ut vendae ipi-tionsequo quat parum am repedio cus int. Cerciis ut a volutaque co. Postbank Systems is a full-service IT provider that centrally bundles the IT expertise of the Postbank. We offer stable, secure and high-quality IT systems and services and see ourselves as the backbone of modern banking. The goal: to offer our customers reliable IT solutions that are convincing overall. As part of the Postbank Group, we cover the entire value chain in application development and operation with around 1,500 colleagues. We master the full range of technological services Unique full-service IT provider for the Postbank
  • 3. Seite 3Munich, 15th of October, Lucas Meyer Who I am? Lucas Meyer • 30 years • happily married • 2 children aged 1 and 2 ¾ My main hobbys: • making music with my band • OpenSource Smart Home Technologies Main career steps so far: ▪ With the company Postbank Systems AG since 2008 ▪ Operation as system integrator with the focus on provision monitoring solutions ▪ Degree in Business Informatics; Thesis in the Context of BigData Architectures ▪ Internal change ~ 4 years ago from IT – Ops to IT – Project. Start with the strategic mission to build up BigData skill. ▪ Meanwhile: Lead of the "Elastic Squad". One of the first, 7 member strong DevOps team’s ▪ Active as "DevOps Evangelist" in various transformation initiatives
  • 4. Seite 4Munich, 15th of October, Lucas Meyer 2015 2016 2017 2018 2019 Getting started What about a Goal? Technology is one aspect... IT4IT A small team emerges and launches the first Use Case A Data Lake architecture is adopted and more Use Cases are added Collaboration, organization, and the General Data Protection Regulation (GDPR) Container technology and the issue with the logs - a central logging service takes its away Summary & Lessons Learned Transformation needs commitment
  • 5. Seite 5Munich, 15th of October, Lucas Meyer 2015 – Let‘s give it a try: first use cases, small team Challenges Challenges with new technologies Skill IT – Operation Acceptance Governance
  • 6. Seite 6Munich, 15th of October, Lucas Meyer Sales data categorization & cash flow analysis 2015 – Let‘s give it a try: first use cases, small team Use Case: Smart Banking Analytic Workplace The idea of a digital household diary was born Analysis Tools Revenues and purpose of use information
  • 7. Seite 7Munich, 15th of October, Lucas Meyer Core NetworkDMZ KBS DB MergeCoreData Apache FarmLoadbalancer InfoneaLogstash ElasticSearch 2015 – Let‘s give it a try: first use cases, small team Use Case: Monitoring Self Service Terminals Classical monitoring ATM Cash Box status
  • 8. Seite 8Munich, 15th of October, Lucas Meyer 2015 2016 2017 2018 2019 Getting started What about a Goal? Technology is one aspect... IT4IT A small team emerges and launches the first Use Case A Data Lake architecture is adopted and more Use Cases are added Collaboration, organization, and the General Data Protection Regulation (GDPR) Container technology and the issue with the logs - a central logging service takes its away Summary & Lessons Learned Transformation needs commitment
  • 9. Seite 9Munich, 15th of October, Lucas Meyer 100101001001010010010010 010001010110100101010010 101110010100101001100110 Data Lake Digital Account Book & Credit – Worthiness What is a Data Lake? Data pool for storing raw data for analysis and evaluation purposes. 2016 – A Data Lake is born …on which we could build data driven products
  • 10. Seite 10Munich, 15th of October, Lucas Meyer 2016 – Business Process Monitoring Apache Farm Loadbalancer Logstash Elasticsearch Kibana Availability and performance Monitoring of the branch client Dashboards for analytical purposes
  • 11. Seite 11Munich, 15th of October, Lucas Meyer 2015 2016 2017 2018 2019 Getting started What about a Goal? Technology is one aspect... IT4IT A small team emerges and launches the first Use Case A Data Lake architecture is adopted and more Use Cases are added Collaboration, organization, and the General Data Protection Regulation (GDPR) Container technology and the issue with the logs - a central logging service takes its away Summary & Lessons Learned Transformation needs commitment
  • 12. Seite 12Munich, 15th of October, Lucas Meyer 2017 – Challenges The team grows and the different perspectives collide DevelopmentBusiness Operations We have nearly no experience with these technologies, we cannot guarantee the business a SLA? Big Data technology does not fit into our existing data center design. We can't just ignore valid backup policies just because the data is supposedly replicated in the cluster. Before the first line of code is written, extensive concepts are written, discussed, reviewed, changed and reviewed again, again, again …. that's stress.
  • 13. Seite 13Munich, 15th of October, Lucas Meyer 2017 – Breaking the Status Quo Previous project organization is supplemented by platform parts Elastic Squad Team We have been developing agilely with Scrum for some time now. DevOps was now a real step forward. I receive immediate feedback as to whether functional and non-functional requirements have been met. As guardians of stability, we have always seen ourselves as the antithesis of agile software developers. Meanwhile I know that this is not true. I need to become an enabler and not as a barrier. People like me are needed more than ever, because: I make sure that infrastructure is provided as a service. What the team shows me at the end of the sprint are not only features that work on their screen but they are also built in such a way that we can be productive with them at any time.
  • 14. Seite 14Munich, 15th of October, Lucas Meyer 2015 2016 2017 2018 2019 Getting started What about a Goal? Technology is one aspect... IT4IT A small team emerges and launches the first Use Case A Data Lake architecture is adopted and more Use Cases are added Collaboration, organization, and the General Data Protection Regulation (GDPR) Container technology and the issue with the logs - a central logging service takes its away Summary & Lessons Learned Transformation needs commitment
  • 15. Seite 15Munich, 15th of October, Lucas Meyer 2018 – DSGVO / GDPR Use Case – Read Access Logging Elastic Stack for compliant reasons: Read access log with 2-year retention period Data Protection Officer access only Data Lake export functionality of the RALs Data Lake interface acknowledges after complete receipt of the data to SAP
  • 16. Seite 16Munich, 15th of October, Lucas Meyer Alert: Alerting in case of anomalies in the data 2018 – Central Logging Service Collect & Ship Transform & Enrich Store & Analyze Visualize Archive & Backup: Archiving of raw data for subsequent transformation, reprocessing in other systems and/or recovery. Secure: Role- and attribute-based access restriction to the data
  • 17. Seite 17Munich, 15th of October, Lucas Meyer 2015 2016 2017 2018 2019 Getting started What about a Goal? Technology is one aspect... IT4IT A small team emerges and launches the first Use Case A Data Lake architecture is adopted and more Use Cases are added Collaboration, organization, and the General Data Protection Regulation (GDPR) Container technology and the issue with the logs - a central logging service takes its away Summary & Lessons Learned Transformation needs commitment
  • 18. Seite 18Munich, 15th of October, Lucas Meyer Reengineering a Cluster that doesn‘t fit the requirements Today - Anti Fraud Management Endpoint Malware Detecion Filebeat EMD Fraud Detection System Filebeat FDS Key Components: Fraud Detection System Endpoint Malware Detection System Fraud Management System for Analytics
  • 19. Seite 19Munich, 15th of October, Lucas Meyer why elastic Conclusion Instances: 20 Pipelines: 20 Logstash Instances: 50 DataVolume: 70 TB • 00_Admin: 600 GB • 01_ICFEM: 150 GB • 02_SBD: 2,5 TB • 03_FMSA:65 TB • 04_RAL: 1,7 TB • 05_CLS: 500 GB Elasticsearch Instances: 20 User: 180 Kibana Instances: 150 • Metricbeat: 60 • Filebeat: 60 • Packetbeat: 30 • Heartbeat: 10 Beats Kafka
  • 20. Seite 20Munich, 15th of October, Lucas Meyer „Start anywhere and improve from there“ (Douglas Adams) THANK YOU