For a long time in order to achieve mutual TLS between Kafka brokers and its clients we had to use long-lived certificates which is a nightmare to manage at large scale. At TransferWise, we have around 300 microservices and most of them use Kafka for the async communication, stream processing, event sourcing, etc. We wanted to implement Kafka security in a way that reduced the maintenance burden on platform teams, while making migration of diverse clients as simple as possible. In this talk we will describe how we have achieved that goal using SPIFFE with SPIRE and Envoy, requiring zero code changes on the client side.
Introduce Brainf*ck, another Turing complete programming language. Then, try to implement the following from scratch: Interpreter, Compiler [x86_64 and ARM], and JIT Compiler.
For a long time in order to achieve mutual TLS between Kafka brokers and its clients we had to use long-lived certificates which is a nightmare to manage at large scale. At TransferWise, we have around 300 microservices and most of them use Kafka for the async communication, stream processing, event sourcing, etc. We wanted to implement Kafka security in a way that reduced the maintenance burden on platform teams, while making migration of diverse clients as simple as possible. In this talk we will describe how we have achieved that goal using SPIFFE with SPIRE and Envoy, requiring zero code changes on the client side.
Introduce Brainf*ck, another Turing complete programming language. Then, try to implement the following from scratch: Interpreter, Compiler [x86_64 and ARM], and JIT Compiler.
Describes how Clear Linux OS is designed, highlighting core features, operating models, and foundational tools that are key to understanding how the distro operates.
It's Time To Stop Using Lambda Architecture | Yaroslav Tkachenko, ShopifyHostedbyConfluent
Lambda Architecture has been a common way to build data pipelines for a long time, despite difficulties in maintaining two complex systems. An alternative, Kappa Architecture, was proposed in 2014, but many companies are still reluctant to switch to Kappa. And there is a reason for that: even though Kappa generally provides a simpler design and similar or lower latency, there are a lot of practical challenges in areas like exactly-once delivery, late-arriving data, historical backfill and reprocessing.
In this talk, I want to show how you can solve those challenges by embracing Apache Kafka as a foundation of your data pipeline and leveraging modern stream-processing frameworks like Apache Kafka Streams and Apache Flink.
PNUTS is a massively parallel and geographically distributed database system for Yahoo!’s web applications. It provides data storage organized as hashed or ordered tables, low latency for large numbers of concurrent requests including updates and queries, and novel per-record consistency guarantees. It is a hosted, centrally managed, and geographically distributed service, and utilizes automated load-balancing and failover to reduce operational complexity. The first version of the system is currently serving in production. This presentation describes the motivation for PNUTS and the design and implementation of its table storage and replication layers, and then presents experimental results.
Talk for SCaLE13x. Video: https://www.youtube.com/watch?v=_Ik8oiQvWgo . Profiling can show what your Linux kernel and appliacations are doing in detail, across all software stack layers. This talk shows how we are using Linux perf_events (aka "perf") and flame graphs at Netflix to understand CPU usage in detail, to optimize our cloud usage, solve performance issues, and identify regressions. This will be more than just an intro: profiling difficult targets, including Java and Node.js, will be covered, which includes ways to resolve JITed symbols and broken stacks. Included are the easy examples, the hard, and the cutting edge.
Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder. Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Flink) and in-house technologies have helped Uber scale.
This slide deck focuses on eBPF JIT compilation infrastructure and how it plays an important role in the entire eBPF life cycle inside the Linux kernel. First, it does quite a number of control flow checks to reject vulnerable programs and then JIT compiles the eBPF program to either host or offloading target instructions which boost performance. However, there is little documentation about this topic which this slide deck will dive into.
C* Summit 2013: How Not to Use Cassandra by Axel LiljencrantzDataStax Academy
At Spotify, we see failure as an opportunity to learn. During the two years we've used Cassandra in our production environment, we have learned a lot. This session touches on some of the exciting design anti-patterns, performance killers and other opportunities to lose a finger that are at your disposal with Cassandra.
Log Management
Log Monitoring
Log Analysis
Need for Log Analysis
Problem with Log Analysis
Some of Log Management Tool
What is ELK Stack
ELK Stack Working
Beats
Different Types of Server Logs
Example of Winlog beat, Packetbeat, Apache2 and Nginx Server log analysis
Mimikatz
Malicious File Detection using ELK
Practical Setup
Conclusion
Reigning in Protobuf with David Navalho and Graham Stirling | Kafka Summit Lo...HostedbyConfluent
With a rich ecosystem and support for multiple languages, it’s no surprise that Protobuf has emerged as a challenger to Avro’s crown as the de-facto serialization format for Kafka. Helped by first class support from Confluent, RedHat and others, Protobuf has finally arrived as viable choice for enterprise wide use cases.
During this talk we will tackle how we have used Protobuf successfully with Kafka: from clients to connectors; streams to schema registry; and gitops to governance. We will go over our learnings, including how we have improved the developer experience through the use of linting and early breaking change detection.
Expect to leave this talk knowing more about Protobuf and how it is supported across the Kafka ecosystem. We will cover thorny topics such as field presence, reusability, the value of a registry and when schema-less is the right answer. Finally, we will share the pitfalls and challenges, how we’ve made Protobuf work seamlessly for us at scale.
Describes how Clear Linux OS is designed, highlighting core features, operating models, and foundational tools that are key to understanding how the distro operates.
It's Time To Stop Using Lambda Architecture | Yaroslav Tkachenko, ShopifyHostedbyConfluent
Lambda Architecture has been a common way to build data pipelines for a long time, despite difficulties in maintaining two complex systems. An alternative, Kappa Architecture, was proposed in 2014, but many companies are still reluctant to switch to Kappa. And there is a reason for that: even though Kappa generally provides a simpler design and similar or lower latency, there are a lot of practical challenges in areas like exactly-once delivery, late-arriving data, historical backfill and reprocessing.
In this talk, I want to show how you can solve those challenges by embracing Apache Kafka as a foundation of your data pipeline and leveraging modern stream-processing frameworks like Apache Kafka Streams and Apache Flink.
PNUTS is a massively parallel and geographically distributed database system for Yahoo!’s web applications. It provides data storage organized as hashed or ordered tables, low latency for large numbers of concurrent requests including updates and queries, and novel per-record consistency guarantees. It is a hosted, centrally managed, and geographically distributed service, and utilizes automated load-balancing and failover to reduce operational complexity. The first version of the system is currently serving in production. This presentation describes the motivation for PNUTS and the design and implementation of its table storage and replication layers, and then presents experimental results.
Talk for SCaLE13x. Video: https://www.youtube.com/watch?v=_Ik8oiQvWgo . Profiling can show what your Linux kernel and appliacations are doing in detail, across all software stack layers. This talk shows how we are using Linux perf_events (aka "perf") and flame graphs at Netflix to understand CPU usage in detail, to optimize our cloud usage, solve performance issues, and identify regressions. This will be more than just an intro: profiling difficult targets, including Java and Node.js, will be covered, which includes ways to resolve JITed symbols and broken stacks. Included are the easy examples, the hard, and the cutting edge.
Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder. Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Flink) and in-house technologies have helped Uber scale.
This slide deck focuses on eBPF JIT compilation infrastructure and how it plays an important role in the entire eBPF life cycle inside the Linux kernel. First, it does quite a number of control flow checks to reject vulnerable programs and then JIT compiles the eBPF program to either host or offloading target instructions which boost performance. However, there is little documentation about this topic which this slide deck will dive into.
C* Summit 2013: How Not to Use Cassandra by Axel LiljencrantzDataStax Academy
At Spotify, we see failure as an opportunity to learn. During the two years we've used Cassandra in our production environment, we have learned a lot. This session touches on some of the exciting design anti-patterns, performance killers and other opportunities to lose a finger that are at your disposal with Cassandra.
Log Management
Log Monitoring
Log Analysis
Need for Log Analysis
Problem with Log Analysis
Some of Log Management Tool
What is ELK Stack
ELK Stack Working
Beats
Different Types of Server Logs
Example of Winlog beat, Packetbeat, Apache2 and Nginx Server log analysis
Mimikatz
Malicious File Detection using ELK
Practical Setup
Conclusion
Reigning in Protobuf with David Navalho and Graham Stirling | Kafka Summit Lo...HostedbyConfluent
With a rich ecosystem and support for multiple languages, it’s no surprise that Protobuf has emerged as a challenger to Avro’s crown as the de-facto serialization format for Kafka. Helped by first class support from Confluent, RedHat and others, Protobuf has finally arrived as viable choice for enterprise wide use cases.
During this talk we will tackle how we have used Protobuf successfully with Kafka: from clients to connectors; streams to schema registry; and gitops to governance. We will go over our learnings, including how we have improved the developer experience through the use of linting and early breaking change detection.
Expect to leave this talk knowing more about Protobuf and how it is supported across the Kafka ecosystem. We will cover thorny topics such as field presence, reusability, the value of a registry and when schema-less is the right answer. Finally, we will share the pitfalls and challenges, how we’ve made Protobuf work seamlessly for us at scale.
33. 銷售管理機制思考 (1/3)
● 銷售規則同步會存在時間差
● 賣場頁, 派卷系統, EC主系統三者落差問題
● 如何控制資料同步的順序與時間點?
rule A賣場頁
派卷系統
EC主系統
rule A
rule A
rule A
rule B
因更新時間差可能造成規則互衝
rule B
34. 銷售管理機制思考 (2/3)
rule A
賣場頁
派卷系統
EC主系統 rule B
rule A
U-A
rule B
rule B
rule A U-A
當銷售規則變化對使用者之影響
User currently with rule A
U-B User currently with rule B
rule A
rule B
rule B
U-A
U-B
U-A