OpenNebulaConf2015 1.03 Private, Public, Hybrid: The Real Economics of Open S...OpenNebula Project
With all the debate on public, private and hybrid clouds one of the main missing points is hard data: what is the actual, real economic impact of choosing a specific cloud model. We will present the results of an extensive survey of cost models, what is the impact of choosing an open source cloud platform like OpenNebula, the difference between planning for “cattle or cows” and how to compare different clouds using reliable performance metrics. We will also present a small sample of potentially relevant open source project that may help in deployment and management of ad-hoc cloud platforms.
Author Biography
Carlo Daffara the Technical director of Cloudweavers, a company that developed the first hyperconverged appliance based on OpenNebula; Italian member of the European Working Group on Libre Software and co-coordinator of the working group on SMEs of the EU ICT task force on competitiveness. Since 1999, works as evaluator for IST programme submissions in the field of component-based software engineering, GRIDs and international cooperation. Coordinator of the open source platforms technical area of the IEEE technical committee on scalable computing, co-chair of the SIENA EU cloud initiative roadmap editorial board and part of the editorial review board of the International Journal of Open Source Software & Processes (IJOSSP). Has worked as a researcher in the field of collaborative development and open source business models; working with international entitities to promote the development of economic networks through open source software, recently worked with public authorities like UK JISC and CENATIC on estimating the economic impact of cloud computing and the adoption of open source development models. For OpenForum Europe has developed the first Europe-wide macroeconomic analysis of the economic value introduced by the adoption of open source software.
HIgh Performance Redis- Tague Griffith, GoProRedis Labs
High Performance Redis looks at a wide range of techniques - from programming to system tuning - to deploy and maintain an extremely high performing Redis cluster. From the operational
perspective, the talk lays out multiple techniques for clustering (sharding) Redis systems and examines how the different
approaches impact performance time. The talk further examines system settings (Linux network parameters, Redis
system) and how they impact performance (both good and bad). Finally, for the developer, we look at how different ways of structuring data actually demonstrate different performance characteristics
Walmart & IBM Revisit the Linear Road Benchmark- Roger Rea, IBMRedis Labs
The Linear Road benchmark was devised in 2004 to
compare Stream Data Management Systems. Walmart selected Linear Road to compare performance of streaming analytic
offerings. IBM implemented the benchmark application using Redis to maintain state, and IBM Streams to handle the
incoming events and queries. Walmart had to completely revamp the data drivers and test verification to take advantage
of multicore multithreaded servers available today. Tests were run on Microsoft Azure cloud to ensure fair comparison of
vendors. Redis and IBM Streams handled nearly 1 billion events in a 3 hour test on a single 16 core Azure node, and 3.8 billion
when scaled out to 4 nodes. Come learn about the application and near linear scalability of Redis and IBM Streams.
Redis Networking Nerd Down: For Lovers of Packets and Jumbo Frames- John Bull...Redis Labs
Packets per second (PPS) is an often overlooked value within an environment. Most network concerns are around throughput and interface speed, but what happens when this value becomes the bottleneck due to large-scale hosting providers (AWS, Azure, etc.) with rigid standards? This talk covers what a Redis packet looks like and how the out-of-the-box configuration can drastically affect
packet per second overhead. From here we’ll deep dive into specific configuration values which help lower PPS numbers, as well as different Redis master/slave relationships that can be utilized to keep PPS below inflexible network thresholds.
Common Patterns of Multi Data-Center Architectures with Apache Kafkaconfluent
Whether you know you want to run Apache Kafka in multiple data centers and need practical advice or you are wondering why some organizations even need more than one cluster, this online talk is for you.
In this short session, we’ll discuss the basic patterns of multi-datacenter Kafka architectures, explore some of the use-cases enabled by each architecture and show how Confluent Enterprise products make these patterns easy to implement.
Visit www.confluent.io for more information.
OpenNebulaConf2015 1.03 Private, Public, Hybrid: The Real Economics of Open S...OpenNebula Project
With all the debate on public, private and hybrid clouds one of the main missing points is hard data: what is the actual, real economic impact of choosing a specific cloud model. We will present the results of an extensive survey of cost models, what is the impact of choosing an open source cloud platform like OpenNebula, the difference between planning for “cattle or cows” and how to compare different clouds using reliable performance metrics. We will also present a small sample of potentially relevant open source project that may help in deployment and management of ad-hoc cloud platforms.
Author Biography
Carlo Daffara the Technical director of Cloudweavers, a company that developed the first hyperconverged appliance based on OpenNebula; Italian member of the European Working Group on Libre Software and co-coordinator of the working group on SMEs of the EU ICT task force on competitiveness. Since 1999, works as evaluator for IST programme submissions in the field of component-based software engineering, GRIDs and international cooperation. Coordinator of the open source platforms technical area of the IEEE technical committee on scalable computing, co-chair of the SIENA EU cloud initiative roadmap editorial board and part of the editorial review board of the International Journal of Open Source Software & Processes (IJOSSP). Has worked as a researcher in the field of collaborative development and open source business models; working with international entitities to promote the development of economic networks through open source software, recently worked with public authorities like UK JISC and CENATIC on estimating the economic impact of cloud computing and the adoption of open source development models. For OpenForum Europe has developed the first Europe-wide macroeconomic analysis of the economic value introduced by the adoption of open source software.
HIgh Performance Redis- Tague Griffith, GoProRedis Labs
High Performance Redis looks at a wide range of techniques - from programming to system tuning - to deploy and maintain an extremely high performing Redis cluster. From the operational
perspective, the talk lays out multiple techniques for clustering (sharding) Redis systems and examines how the different
approaches impact performance time. The talk further examines system settings (Linux network parameters, Redis
system) and how they impact performance (both good and bad). Finally, for the developer, we look at how different ways of structuring data actually demonstrate different performance characteristics
Walmart & IBM Revisit the Linear Road Benchmark- Roger Rea, IBMRedis Labs
The Linear Road benchmark was devised in 2004 to
compare Stream Data Management Systems. Walmart selected Linear Road to compare performance of streaming analytic
offerings. IBM implemented the benchmark application using Redis to maintain state, and IBM Streams to handle the
incoming events and queries. Walmart had to completely revamp the data drivers and test verification to take advantage
of multicore multithreaded servers available today. Tests were run on Microsoft Azure cloud to ensure fair comparison of
vendors. Redis and IBM Streams handled nearly 1 billion events in a 3 hour test on a single 16 core Azure node, and 3.8 billion
when scaled out to 4 nodes. Come learn about the application and near linear scalability of Redis and IBM Streams.
Redis Networking Nerd Down: For Lovers of Packets and Jumbo Frames- John Bull...Redis Labs
Packets per second (PPS) is an often overlooked value within an environment. Most network concerns are around throughput and interface speed, but what happens when this value becomes the bottleneck due to large-scale hosting providers (AWS, Azure, etc.) with rigid standards? This talk covers what a Redis packet looks like and how the out-of-the-box configuration can drastically affect
packet per second overhead. From here we’ll deep dive into specific configuration values which help lower PPS numbers, as well as different Redis master/slave relationships that can be utilized to keep PPS below inflexible network thresholds.
Common Patterns of Multi Data-Center Architectures with Apache Kafkaconfluent
Whether you know you want to run Apache Kafka in multiple data centers and need practical advice or you are wondering why some organizations even need more than one cluster, this online talk is for you.
In this short session, we’ll discuss the basic patterns of multi-datacenter Kafka architectures, explore some of the use-cases enabled by each architecture and show how Confluent Enterprise products make these patterns easy to implement.
Visit www.confluent.io for more information.
Securing Kafka At Zendesk (Joy Nag, Zendesk) Kafka Summit 2020confluent
Kafka is one of the most important foundation services at Zendesk. It became even more crucial with the introduction of Global Event Bus which my team built to propagate events between Kafka clusters hosted at different parts of the world and between different products. As part of its rollout, we had to add mTLS support in all of our Kafka Clusters (we have quite a few of them), this was to make propagation of events between clusters hosted at different parts of the world secure. It was quite a journey, but we eventually built a solution that is working well for us.
Things I will be sharing as part of the talk:
1. Establishing the use case/problem we were trying to solve (why we needed mTLS)
2. Building a Certificate Authority with open source tools (with self-signed Root CA)
3. Building helper components to generate certificates automatically and regenerate them before they expire (helps using a shorter TTL (Time To Live) which is good security practice) for both Kafka Clients and Brokers
4. Hot reloading regenerated certificates on Kafka brokers without downtime
5. What we built to rotate the self-signed root CA without downtime as well across the board
6. Monitoring and alerts on TTL of certificates
7. Performance impact of using TLS (along with why TLS affects kafka’s performance)
8. What we are doing to drive adoption of mTLS for existing Kafka clients using PLAINTEXT protocol by making onboarding easier
9. How this will become a base for other features we want, eg ACL, Rate Limiting (by using the principal from the TLS certificate as Identity of clients)
Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...HostedbyConfluent
When choosing an event streaming platform, Kafka shouldn’t be the only technology you look at. There are a plethora of others in the messaging space today, including open source and proprietary software as well as a range of cloud services. So how do you know you are choosing the right one? A great way to deepen our understanding of event streaming and Kafka is exploring the trade-offs in distributed system design and learning about the choices made by the Kafka project. We’ll look at how Kafka stacks up against other technologies in the space, including traditional messaging systems like Apache ActiveMQ and RabbitMQ as well as more contemporary ones, such as BookKeeper derivatives like Apache Pulsar or Pravega. This talk focuses on the technical details such as difference in messaging models, how data is stored locally as well as across machines in a cluster, when (not) to add tiers to your system, and more. By the end of the talk, you should have a good high-level understanding of how these systems compare and which you should choose for different types of use cases.
Multi-Tenancy Kafka cluster for LINE services with 250 billion daily messagesLINE Corporation
Yuto Kawamura
LINE / Z Part Team
At LINE we've been operating Apache Kafka to provide the company-wide shared data pipeline for services using it for storing and distributing data.
Kafka is underlying many of our services in some way, not only the messaging service but also AD, Blockchain, Pay, Timeline, Cryptocurrency trading and more.
Many services feeding many data into our cluster, leading over 250 billion daily messages and 3.5GB incoming bytes in 1 second which is one of the world largest scale.
At the same time, it is required to be stable and performant all the time because many important services uses it as a backend.
In this talk I will introduce the overview of Kafka usage at LINE and how we're operating it.
I'm also going to talk about some engineerings we did for maximizing its performance, solving troubles led particularly by hosting huge data from many services, leveraging advanced techniques like kernel-level dynamic tracing.
Nagios Conference 2014 - Jeremy Rust - Avoiding Downtime Using Linux High Ava...Nagios
Jeremy Rust's presentation on Avoiding Downtime Using Linux High Availability.
The presentation was given during the Nagios World Conference North America held Oct 13th - Oct 16th, 2014 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/conference
How to build 1000 microservices with Kafka and thriveNatan Silnitsky
This talk is about the Wix ecosystem for event driven architecture on top of Kafka.
I share the best practices, SDKs and tools we have created in order to be able to scale our distributed system to more than 1000 microservices.
Mario Molina, Datio, Software Engineer
Kafka Streams is an open source JVM library for building event streaming applications on top of Apache Kafka. Its goal is to allow programmers to create efficient, real-time, streaming applications and perform analysis and operations on the incoming data.
In this presentation we’ll cover the main features of Kafka Streams and do a live demo!
This demo will be partially on Confluent Cloud, if you haven’t already signed up, you can try Confluent Cloud for free. Get $200 every month for your first three months ($600 free usage in total) get more information and claim it here: https://cnfl.io/cloud-meetup-free
https://www.meetup.com/Mexico-Kafka/events/271972045/
How THINQ runs both transactions and analytics at scaleMariaDB plc
THINQ provides a cloud-based Communications-Platform-as-a-Service (CPaaS) that routes tens of millions of phone calls per day for customers in enterprise and telecommunications industries. In this session Sasha Vaniachine, Senior Database Administrator at THINQ, explains how he combined MariaDB Server and MariaDB ColumnStore to support both high-performance transaction processing and scalable analytics. In addition, he shares some of THINQ's best practices and lessons learned from supporting an ever-increasing database workload that currently exceeds 10,000 transactions per second.
Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...HostedbyConfluent
Organizations have a need to protect Personally Identifiable Information (PII). As Event Streaming Architecture (ESA) becomes ubiquitous in the enterprise, the prevalence of PII within data streams will only increase. Data architects must be cognizant of how their data pipelines can allow for potential leaks. In highly distributed systems, zero-trust networking has become an industry best practice. We can do the same with Kafka by introducing message-level security.
A DevSecOps Engineer with some Kafka experience can leverage Kafka Streams to protect PII by enforcing role-based access control using Open Policy Agent. Rather than implementing a REST API to handle message-level security, Kafka Streams can filter, or even transform outgoing messages in order to redact PII data while leveraging the native capabilities of Kafka.
In our proposed presentation, we will provide a live demonstration that consists of two consumers subscribing to the same Kafka topic, but receiving different messages based on the rules specified in Open Policy Agent. At the conclusion of the presentation, we will provide attendees with a GitHub repository, so that they can enjoy a sandbox environment for hands-on experimentation with message-level security.
How Criteo is managing one of the largest Kafka Infrastructure in EuropeRicardo Paiva
In Criteo we manage one of the largest Kafka infrastructure in Europe, with more than 7 million msgs/sec. This talk was first presented on the Kafka Meetup Paris, in January of 2019.
Back your App with MySQL & Redis, the Cloud Foundry Way- Kenny Bastani, PivotalRedis Labs
In this session, we will build a minimum viable Spring Data web service with REST API, add a MySQL backing service as the primary data store, and a Redis Labs backing service for caching. We will demonstrate performance metrics without Redis caching enabled and then with Redis caching enabled. I will also provide an intro-level explanation of the platform capabilities within Pivotal Web Services.
Talk given at ClueCon 2016 that discusses FreeSWITCH and its place in a microservices architecture. Covers a specific deployment case using Docker and Adhearsion, along with certain features that make FreeSWITCH a model use-case for such a technology stack.
Securing Kafka At Zendesk (Joy Nag, Zendesk) Kafka Summit 2020confluent
Kafka is one of the most important foundation services at Zendesk. It became even more crucial with the introduction of Global Event Bus which my team built to propagate events between Kafka clusters hosted at different parts of the world and between different products. As part of its rollout, we had to add mTLS support in all of our Kafka Clusters (we have quite a few of them), this was to make propagation of events between clusters hosted at different parts of the world secure. It was quite a journey, but we eventually built a solution that is working well for us.
Things I will be sharing as part of the talk:
1. Establishing the use case/problem we were trying to solve (why we needed mTLS)
2. Building a Certificate Authority with open source tools (with self-signed Root CA)
3. Building helper components to generate certificates automatically and regenerate them before they expire (helps using a shorter TTL (Time To Live) which is good security practice) for both Kafka Clients and Brokers
4. Hot reloading regenerated certificates on Kafka brokers without downtime
5. What we built to rotate the self-signed root CA without downtime as well across the board
6. Monitoring and alerts on TTL of certificates
7. Performance impact of using TLS (along with why TLS affects kafka’s performance)
8. What we are doing to drive adoption of mTLS for existing Kafka clients using PLAINTEXT protocol by making onboarding easier
9. How this will become a base for other features we want, eg ACL, Rate Limiting (by using the principal from the TLS certificate as Identity of clients)
Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...HostedbyConfluent
When choosing an event streaming platform, Kafka shouldn’t be the only technology you look at. There are a plethora of others in the messaging space today, including open source and proprietary software as well as a range of cloud services. So how do you know you are choosing the right one? A great way to deepen our understanding of event streaming and Kafka is exploring the trade-offs in distributed system design and learning about the choices made by the Kafka project. We’ll look at how Kafka stacks up against other technologies in the space, including traditional messaging systems like Apache ActiveMQ and RabbitMQ as well as more contemporary ones, such as BookKeeper derivatives like Apache Pulsar or Pravega. This talk focuses on the technical details such as difference in messaging models, how data is stored locally as well as across machines in a cluster, when (not) to add tiers to your system, and more. By the end of the talk, you should have a good high-level understanding of how these systems compare and which you should choose for different types of use cases.
Multi-Tenancy Kafka cluster for LINE services with 250 billion daily messagesLINE Corporation
Yuto Kawamura
LINE / Z Part Team
At LINE we've been operating Apache Kafka to provide the company-wide shared data pipeline for services using it for storing and distributing data.
Kafka is underlying many of our services in some way, not only the messaging service but also AD, Blockchain, Pay, Timeline, Cryptocurrency trading and more.
Many services feeding many data into our cluster, leading over 250 billion daily messages and 3.5GB incoming bytes in 1 second which is one of the world largest scale.
At the same time, it is required to be stable and performant all the time because many important services uses it as a backend.
In this talk I will introduce the overview of Kafka usage at LINE and how we're operating it.
I'm also going to talk about some engineerings we did for maximizing its performance, solving troubles led particularly by hosting huge data from many services, leveraging advanced techniques like kernel-level dynamic tracing.
Nagios Conference 2014 - Jeremy Rust - Avoiding Downtime Using Linux High Ava...Nagios
Jeremy Rust's presentation on Avoiding Downtime Using Linux High Availability.
The presentation was given during the Nagios World Conference North America held Oct 13th - Oct 16th, 2014 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/conference
How to build 1000 microservices with Kafka and thriveNatan Silnitsky
This talk is about the Wix ecosystem for event driven architecture on top of Kafka.
I share the best practices, SDKs and tools we have created in order to be able to scale our distributed system to more than 1000 microservices.
Mario Molina, Datio, Software Engineer
Kafka Streams is an open source JVM library for building event streaming applications on top of Apache Kafka. Its goal is to allow programmers to create efficient, real-time, streaming applications and perform analysis and operations on the incoming data.
In this presentation we’ll cover the main features of Kafka Streams and do a live demo!
This demo will be partially on Confluent Cloud, if you haven’t already signed up, you can try Confluent Cloud for free. Get $200 every month for your first three months ($600 free usage in total) get more information and claim it here: https://cnfl.io/cloud-meetup-free
https://www.meetup.com/Mexico-Kafka/events/271972045/
How THINQ runs both transactions and analytics at scaleMariaDB plc
THINQ provides a cloud-based Communications-Platform-as-a-Service (CPaaS) that routes tens of millions of phone calls per day for customers in enterprise and telecommunications industries. In this session Sasha Vaniachine, Senior Database Administrator at THINQ, explains how he combined MariaDB Server and MariaDB ColumnStore to support both high-performance transaction processing and scalable analytics. In addition, he shares some of THINQ's best practices and lessons learned from supporting an ever-increasing database workload that currently exceeds 10,000 transactions per second.
Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...HostedbyConfluent
Organizations have a need to protect Personally Identifiable Information (PII). As Event Streaming Architecture (ESA) becomes ubiquitous in the enterprise, the prevalence of PII within data streams will only increase. Data architects must be cognizant of how their data pipelines can allow for potential leaks. In highly distributed systems, zero-trust networking has become an industry best practice. We can do the same with Kafka by introducing message-level security.
A DevSecOps Engineer with some Kafka experience can leverage Kafka Streams to protect PII by enforcing role-based access control using Open Policy Agent. Rather than implementing a REST API to handle message-level security, Kafka Streams can filter, or even transform outgoing messages in order to redact PII data while leveraging the native capabilities of Kafka.
In our proposed presentation, we will provide a live demonstration that consists of two consumers subscribing to the same Kafka topic, but receiving different messages based on the rules specified in Open Policy Agent. At the conclusion of the presentation, we will provide attendees with a GitHub repository, so that they can enjoy a sandbox environment for hands-on experimentation with message-level security.
How Criteo is managing one of the largest Kafka Infrastructure in EuropeRicardo Paiva
In Criteo we manage one of the largest Kafka infrastructure in Europe, with more than 7 million msgs/sec. This talk was first presented on the Kafka Meetup Paris, in January of 2019.
Back your App with MySQL & Redis, the Cloud Foundry Way- Kenny Bastani, PivotalRedis Labs
In this session, we will build a minimum viable Spring Data web service with REST API, add a MySQL backing service as the primary data store, and a Redis Labs backing service for caching. We will demonstrate performance metrics without Redis caching enabled and then with Redis caching enabled. I will also provide an intro-level explanation of the platform capabilities within Pivotal Web Services.
Talk given at ClueCon 2016 that discusses FreeSWITCH and its place in a microservices architecture. Covers a specific deployment case using Docker and Adhearsion, along with certain features that make FreeSWITCH a model use-case for such a technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.Renzo Tomà
Presentation given by Renzo Tomà as "Tech and Use Case Deep Dive", during the Elastic{ON}Tour 2015 event in Amsterdam on October 29th.
Explanation of how bol.com is using the Elastic ELK stack to power a logsearch platform. Lots of details on the types of sources and number of feeds. Some history and reasoning why the current set of in-process JSON based logshippers are used. Links to the bol.com github account for the logshipper projects. The presentation ends with two special sauces: fun things you can do with lots of data in Elasticsearch. The 1st sauce is 'the call stack' - tagging each request with a unique ID, passing that ID along to all service calls and making sure this ID ends up in all access logging, enables you to group all calls together and get a call stack. The 2nd sauce is a way of generating a service map using access logging and some logstash magic.
I love questions and feedback. My mail address can be found in the presentation.
The Good, the Bad and the Ugly of Migrating Hundreds of Legacy Applications ...Josef Adersberger
Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs, and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud native apps. But what to do if you’ve no shiny new cloud native apps but a whole bunch of JEE legacy systems? No chance to leverage the advantages of Kubernetes? Yes you can!
We’re facing the challenge of migrating hundreds of JEE legacy applications of a major German insurance company onto a Kubernetes cluster within one year. We're now close to the finish line and it worked pretty well so far.
The talk will be about the lessons we've learned - the best practices and pitfalls we've discovered along our way. We'll provide our answers to life, the universe and a cloud native journey like:
- What technical constraints of Kubernetes can be obstacles for applications and how to tackle these?
- How to architect a landscape of hundreds of containerized applications with their surrounding infrastructure like DBs MQs and IAM and heavy requirements on security?
- How to industrialize and govern the migration process?
- How to leverage the possibilities of a cloud native platform like Kubernetes without challenging the tight timeline?
Migrating Hundreds of Legacy Applications to Kubernetes - The Good, the Bad, ...QAware GmbH
CloudNativeCon North America 2017, Austin (Texas, USA): Talk by Josef Adersberger (@adersberger, CTO at QAware)
Abstract:
Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs, and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud native apps. But what to do if you’ve no shiny new cloud native apps but a whole bunch of JEE legacy systems? No chance to leverage the advantages of Kubernetes? Yes you can!
We’re facing the challenge of migrating hundreds of JEE legacy applications of a major German insurance company onto a Kubernetes cluster within one year. We're now close to the finish line and it worked pretty well so far.
The talk will be about the lessons we've learned - the best practices and pitfalls we've discovered along our way. We'll provide our answers to life, the universe and a cloud native journey like:
- What technical constraints of Kubernetes can be obstacles for applications and how to tackle these?
- How to architect a landscape of hundreds of containerized applications with their surrounding infrastructure like DBs MQs and IAM and heavy requirements on security?
- How to industrialize and govern the migration process?
- How to leverage the possibilities of a cloud native platform like Kubernetes without challenging the tight timeline?
Introducing Events and Stream Processing into Nationwide Building Society (Ro...confluent
Facing Open Banking regulation, rapidly increasing transaction volumes and increasing customer expectations, Nationwide took the decision to take load off their back-end systems through real-time streaming of data changes into Kafka. Hear about how Nationwide started their journey with Kafka, from their initial use case of creating a real-time data cache using Change Data Capture, Kafka and Microservices to how Kafka allowed them to build a stream processing backbone used to reengineer the entire banking experience including online banking, payment processing and mortgage applications. See a working demo of the system and what happens to the system when the underlying infrastructure breaks. Technologies covered include: Change Data Capture, Kafka (Avro, partitioning and replication) and using KSQL and Kafka Streams Framework to join topics and process data.
Capital One Delivers Risk Insights in Real Time with Stream Processingconfluent
Speakers: Ravi Dubey, Senior Manager, Software Engineering, Capital One + Jeff Sharpe, Software Engineer, Capital One
Capital One supports interactions with real-time streaming transactional data using Apache Kafka®. Kafka helps deliver information to internal operation teams and bank tellers to assist with assessing risk and protect customers in a myriad of ways.
Inside the bank, Kafka allows Capital One to build a real-time system that takes advantage of modern data and cloud technologies without exposing customers to unnecessary data breaches, or violating privacy regulations. These examples demonstrate how a streaming platform enables Capital One to act on their visions faster and in a more scalable way through the Kafka solution, helping establish Capital One as an innovator in the banking space.
Join us for this online talk on lessons learned, best practices and technical patterns of Capital One’s deployment of Apache Kafka.
-Find out how Kafka delivers on a 5-second service-level agreement (SLA) for inside branch tellers.
-Learn how to combine and host data in-memory and prevent personally identifiable information (PII) violations of in-flight transactions.
-Understand how Capital One manages Kafka Docker containers using Kubernetes.
Watch the recording: https://videos.confluent.io/watch/6e6ukQNnmASwkf9Gkdhh69?.
The presentation explains the reasons we picked Kafka as Streaming Hub and the use of Kafka Streams to avoid common anti-patterns, streamline development experience, improve resilience, enhance performances and enable experimentation. A step-by-step example will be presented to introduce the Kafka Streams DSL and understand what happens under the hood of a stateful streaming application.
Initiative Based Technology Consulting Case Studieschanderdw
Our initiative-based “pay-as-you-go” model empowers you to buy only the services you need without long-term contract obligations, and better optimizes your resources with greater accuracy and efficiency.
An agile, flexible technology partner using this model helps clients secure resources in advance, map them to their initiatives, and enjoy on-demand service availability--which means real-time project control.
You gain improved transparency for your tech spend with predictable cash flow that is consumption-based. The client benefits from utilizing resources only as and when required during the lifecycle of the technology initiative.
Production high-performance networking with Snabb and LuaJIT (Linux.conf.au 2...Igalia
By Andy Wingo.
It used to be that to set up a serious network, you needed to stock racks and racks with specialized proprietary single-purpose boxes. This was because only specialized hardware could handle the hundreds of gigabits per second that might flow through any given box.
Things have changed. With the rise of cheap commodity Xeon-based servers and widespread availability of 10 gigabit network cards, an off-the-shelf server with a few NICs can now handle the workload. The age of open source software-driven routers is fully here -- but it doesn't look like what we thought it would, 10 years ago.
We thought it would be Linux everywhere, but it turns out that Linux's networking stack is just too slow. To get around this problem, modern high-speed software switches bypass the kernel entirely, instead booting network cards and handling traffic entirely from user-space. The up-side of this is that now we have the possibility of using pleasant, hackable, open source, standalone software stacks to deliver network applications that are tailored to specific needs.
This talk presents Snabb, a toolkit for building user-space network functions. Snabb is entirely written in the expressive Lua language, minimizing the amount of code that you have to write to get stuff done. Snabb specifically uses the LuaJIT implementation of Lua, giving us excellent code generation as
well as efficient access to low-level binary data and AVX2 assembly generation.
Snabb's goal is to be "rewritable software": software that's so simple that you could explain it to someone and they could write their own. By the end of the presentation, you too should have this feeling.
We will also describe how Snabb is used in practice in major telecoms and ISPs to provide IPv6 transition technologies to entire countries. Using Snabb allowed a small team of open-source hackers to ship a product that competed favorably
against offerings from traditional network vendors.
(c) linux.conf.au 2017, CC-BY-SA
Hobart, 16-20 January 2017
https://linux.conf.au
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...confluent
Microservices, events, containers, and orchestrators are dominating our vernacular today. As operations teams adapt to support these technologies in production, cloud-native platforms like Pivotal Cloud Foundry and Kubernetes have quickly risen to serve as force multipliers of automation, productivity and value.
Apache Kafka® is providing developers a critically important component as they build and modernize applications to cloud-native architecture.
This talk will explore:
• Why cloud-native platforms and why run Apache Kafka on Kubernetes?
• What kind of workloads are best suited for this combination?
• Tips to determine the path forward for legacy monoliths in your application portfolio
• Demo: Running Apache Kafka as a Streaming Platform on Kubernetes
Service Discovery and Registration in a Microservices ArchitecturePLUMgrid
Microservices, Service Discovery and Registration have been heading towards the peak of inflated expectations on the Gartner Hype cycle for over the last year or so, but there has often been a lack of clarity as to what these are, why are they needed or how to implement them well.
Service discovery and registration are key components of most distributed systems and service oriented architectures. In this session we will talk about what, why and how of service registration and discovery in distributed systems in general and OpenStack in particular.
We will talk about some of the technologies that address this challenge like Zookeeper, Etcd, Consul, Mesos-DNS, Minuteman, SkyDNS, SmartStack or Eureka. We will also address how these technologies as well as existing OpenStack projects can be used to solve this problem inside OpenStack environments.
In the last years we have seen huge changes in IT infrastructures and concepts. VoIP architectures too are evolving towards Software Defined Telecoms. In this talk we'll see how VoIP solutions are being shaped by the Cloud, the open points and share some thoughts about its future.
This is co-authored by Giacomo Vacca and Federico Cabiddu.
In the agile, lean, devops communities people talk about improving security by "shifting left". Patterns and tools are emerging, or re-emerging, that make security less of a pain in the development process while also making applications more secure.
"Shift Lef Security" What the funk does that mean?
In the agile, lean, DevOps communities people talk about improving security by "shifting left". Patterns and tools are emerging, or re-emerging, that make security less of a pain in the development process while also making applications more secure.
Quick introduction to wtf is devops.
Since there is no formal description on purpose it can mean different things to different people yet there is still strong consensus on what it is and what it isn't.
ER(Entity Relationship) Diagram for online shopping - TAEHimani415946
https://bit.ly/3KACoyV
The ER diagram for the project is the foundation for the building of the database of the project. The properties, datatypes, and attributes are defined by the ER diagram.
Multi-cluster Kubernetes Networking- Patterns, Projects and GuidelinesSanjeev Rampal
Talk presented at Kubernetes Community Day, New York, May 2024.
Technical summary of Multi-Cluster Kubernetes Networking architectures with focus on 4 key topics.
1) Key patterns for Multi-cluster architectures
2) Architectural comparison of several OSS/ CNCF projects to address these patterns
3) Evolution trends for the APIs of these projects
4) Some design recommendations & guidelines for adopting/ deploying these solutions.
This 7-second Brain Wave Ritual Attracts Money To You.!nirahealhty
Discover the power of a simple 7-second brain wave ritual that can attract wealth and abundance into your life. By tapping into specific brain frequencies, this technique helps you manifest financial success effortlessly. Ready to transform your financial future? Try this powerful ritual and start attracting money today!
1.Wireless Communication System_Wireless communication is a broad term that i...JeyaPerumal1
Wireless communication involves the transmission of information over a distance without the help of wires, cables or any other forms of electrical conductors.
Wireless communication is a broad term that incorporates all procedures and forms of connecting and communicating between two or more devices using a wireless signal through wireless communication technologies and devices.
Features of Wireless Communication
The evolution of wireless technology has brought many advancements with its effective features.
The transmitted distance can be anywhere between a few meters (for example, a television's remote control) and thousands of kilometers (for example, radio communication).
Wireless communication can be used for cellular telephony, wireless access to the internet, wireless home networking, and so on.
2. Me:
Gérard de Vos
MCE @ Schuberg Philis 2008-current. Previously @ Shell, Ziggo, POIS, TNO, …
Now: “full stack”, *-lead. Then: infrastructure, hardware, HPC, Linux, provisioning, web & such
@gr4rd
!
!
!
“Schuberg Philis is an innovative business technology company. We focus on the mission critical applications that our customers and society rely on 24/7.”
Customers include:
3. What we had
• 2009: new internet savings bank!
• Way-of-working 2009: !
• Dedicated DC space, !
• Dedicated servers, !
• Dedicated network, !
• Dedicated team!
• Growth: 0€, 0 customers -> 4B€, 120k customers!
• “Classic” application stack
4.
5. Trigger
1. Contract to expire in <1 year
2. Evaluated current environment:
• Dev environment(s). Not enough, clashes.
• Data refreshes. Too hard <> not done often enough.
• Different environments are different.
• And the usual suspects: lack of flexibility,
underutilization of resources, huggable snowflake
servers.
3. Time moved on:
• Agile development is reaching the enterprise.
• Agile infrastructure is not just for startups & unicorns
anymore.
• "The Lean Startup" is for everybody.
6. Way-we-work now
• Dedicated team (we kept something the same!)
• Shared infra
• MCC: Apache CloudStack
• Shared services
• Chef, chef cookbooks
• Github enterprise
• SBP is more Lean & Agile & Devopsy
• Contribute
• Software is eating the world
• Focus on the value chain. Reduce waste
9. Public site
http://www.leaseplanbank.nl
Secure site
https://sparen.leaseplanbank.nl
LeasePlan
Infrastructure Services
(LPIS) Dublin - Ireland
email2sms email
WebLogic
lpbpapp1/2
active/standby
lpbpws101/102
active/active
lpbpws1/2
active /standby
lpbpapp101/102
active/active
lpbpsan1/2
FCDB
High available SAN (FCAL)
via synchronous mirroring
Site to Site VPN
Site to Site VPN
Managed by LPIS
Apache
Hippo
container
Tomcat
Back Office Front End Services
x equens get
x KYC put
x and other file
exchange
Oracle Reporting
Content
publication
CMS and Public Web Content
http
File system FC Rep FC UBS
https
Direct Banking
email2sms
Alphen a/d Rijn
http
FC Gateway
(active/active)
FCUBS
(active/standby)
Once a month postcode file
is retrieved
ssmtp
SFTP
Manual reporting
Logius/DigiPort interface tbd
SFTP
Hippo
http
http
BKR FC DB
Site to Site VPN
Back office and Customer Care Center Services
Active
standby
Standby
active
Operations
jms
LeasePlan Infrastructure Services
(LPIS) Dublin - Ireland
Direct Banking
Bank Admin GUI
1. Direct Banking:
- Bank Admin GUI
- Super Admin GUI
2. Core Banking
- UBS Admin
3. CMS
incl preview to content staging web site
4. OBIEE reporting
FTP-S
WebLogic
lpbpmx1/2
active/active
Apache
(s)smtp
ssmtp
Almere mail
Home Office users
Marketing
ICT
Finance Control
lpbprep2/1
active/standby
Apache
Scoring and Business rule
System (SBS)
Verification of new customers
Verificatie Informatie Customer CRM
screening
Postcode Table
Rensageg file transfer
FLEXCUBE Core Banking and Gateway
Oracle database
lpbpd1/2
active/standby
Central Storage Array Network (SAN) for SFTP, application, database and some management servers
Secure
site
Sorry
site
KYC file Equens
files
OBIEE
App Server
VPN
VPN
VPN
FLEXCUBE Direct Banking
MySQL
Hippo CMS
Data upload / KYC download
http:7002
sftp http(s) http(s)
smtp
http
mysql
scp SQL*Net V2 SQL*Net V2 SQL*Net V2
FCAL FCAL FCAL
/ VIS
Other files
equens put
KYC get
smtp smtp
NMUT/betOPD/batch
VerwINF
FTP-S (get + put)
equens
Payment Services
For CMS + staging
and OBIEE
http
Public
site
HTTPS
Upload list of customers
lpbprep1/2
active/standby
Savings calculator XML
smtp
Antivirus + antispam
email
customers
LPB office
Email 2 sms
Multi homed
internet acces
Direct Banking
Bank Admin GUI Direct Banking VPN
x BankAdmin interface for CCC
x BankAdmin + SuperAdmin
interface for LPB BackOffice
Customers DMZ for mail, public and secure web sites
Customer Contact Center
VPN
VPN
10.
11. We came up with this
• Private storage for datastores
• Private hypervisors for transaction processing systems
• Kept existing internet facing network connections kit
• Shared cloud for
• Dev/dev2/../test(UAT) environments with anonymised data
• Admin env. monitoring, deployment, etc.
12.
13. Shopping list
• Shared MCC zone:
• Network: I don’t care,
• Hypervisors: I don’t care
• CloudStack Primary secondary storage: I don’t care
14. Shopping list
• Private customer zone:
• Two pods - 2 datacentres
• Network: Arista 10GbE Top-of-rack,
• Hypervisors: HP DL380G8 8core, 192GB
• CloudStack Primary secondary storage: NetApp
• NFS storage for datavolumes: NetApp metroclustre
• Runs the production and preproduction environments
15. The challenges
• New tech
• CloudStack SDN
• git
• Chef
• Many others
• New thinking
• WayWeWork (highly in flux)
• Shared infra
• Shared svcs
• Design-for-failure vs Enterprisey apps
16.
17.
18. The nice things
• Infra-as-code. We now think things go slow when
it takes 10 minutes to go from nothing to
functioning server.
• Re-re-re-rebuilds. Process maturity, Cookbook
maturity, DR/BCP maturity confidence.
• Infra is almost a non-topic in discussions with the
customer around new applications services.
• SBP cloud HW performance. CPU/mem IOPS/
mbps
EndOfDay 2hr - 45m
• MCC matured a lot.
• WayWeWork is maturing.
19. 20/20 hindsight
• Pushed/pulled the shared services team more. They
are providing a service, not tech.
• Sales/mgt/engineers overestimated what IAAS brings.
• Sales/mgt/engineers underestimated what IAAS brings.
• Put more of the stack into shared cloud.
• DBMS redundancy higher in the stack. (e.g. ASM vs
metroclustre)
20. What do we need help with?
• How do we run in multitenant environments and have everything
secure?
• How do we explain this to auditors so they agree?
