Scaling distributed data systems: A LinkedIn Case study
•
0 likes•134 views
Scaling a stateless system like web servers & application servers are pretty well understood, but scaling a stateful system has its own set of challenges. In this presentation, we hope to present our learnings & challenges faced while scaling a NoSQL database used within LinkedIn. Scaling a data system involves significant movement & replication of data within in a cluster. This can put considerable load on a system that is already running hot, affecting the service experience We will look at some of the challenges & the approaches that we took.
Report
Share
Report
Share
Recommended
Microservices, Monoliths, SOA and How We Got Here
Microservice architecture (MSA) helps address modern software development needs by breaking applications into independently deployable microservices that communicate asynchronously. MSA evolves past SOA approaches by focusing on small, single-purpose services rather than large monolithic applications. Key aspects of MSA include ubiquitous language, well-defined boundaries, independent deployability, asynchronous communication via messaging, and each service owning its own data. MSA also relies on modern cloud-ready infrastructure approaches like service discovery, elastic scaling, and failure resilience at the service level rather than relying on specific hardware.
DBaaS at Scale
This document discusses Database as a Service (DBaaS). It begins by defining DBaaS and describing the tasks involved in managing databases, such as setup, configuration, monitoring, backups and upgrades. It then lists the benefits of DBaaS such as reduced time spent on management tasks and increased scalability. Requirements for DBaaS like security, availability and ease of use are outlined. Different DBaaS architectures including proxy-based and DNS-based models are presented, along with examples. The document concludes by comparing several DBaaS providers and listing useful ScaleChamp features and links.
Kafka Summit SF 2017 - Providing Reliability Guarantees in Kafka at One Trill...
In this presentation, I will talk about my firsthand experience dealing with the unique challenges of running Kafka at a massive scale. If you ever thought that running Kafka is difficult, this talk may change your mind and provide you with valuable insights into how to configure a Kafka cluster efficiently, how to manage Kafka for enterprise customers and how to measure, monitor and maintain the Quality of Kafka Service. Our production Kafka cluster runs over 1500+ VMs, and serves over 10 GBPS data spread across hundreds of topics for multiple teams across Microsoft. We built a self-serve Kafka management service to make the process manageable and scalable across many teams. In this talk, I will also share insights about running Kafka in Private vs multi-tenant mode, supporting failover and disaster recovery requirements, and how to make Kafka Compliant with regulatory certifications such as ISO, SOC, FEDRAMP, etc.
Presented by Nitin Kumar, Microsoft
Systems Track
SQL, Redis and Kubernetes by Paul Stanton of Windocks - Redis Day Seattle 2020
The document discusses common use cases for combining SQL, Redis, and Kubernetes including caching, session management, rate limiting, and data ingestion. It outlines how Kubernetes can be used for scaling microservices while Redis is used for data service scaling. The presentation proposes combining Redis, SQL Server, and Kubernetes with a proxy service, and describes using Redis for caching, session storage, and rate limiting of SQL data. It also discusses running Redis and front-end apps on Kubernetes and deploying SQL as a Kubernetes service through a proxy.
Apache pulsar - storage architecture
Apache Pulsar is a flexible pub-sub messaging system backed by a durable log storage. It uses a segment-centric architecture where messages are stored in independent segments across multiple brokers and bookies for redundancy. This allows for strong durability, high throughput, and seamless expansion without data rebalancing. Pulsar brokers serve client requests and acquire ownership of topics, while bookies provide durable storage with replication for fault tolerance.
Introduction to cloud computing
This document introduces cloud computing, which provides computing resources as a service via virtualization and shared resources. Key aspects include flexible provisioning without large upfront costs, cost-effectiveness through on-demand access without ownership costs, and scalability through elastic scaling of resources via API calls. Popular cloud services from Amazon are described, including compute, storage, content delivery, DNS, databases, messaging, and data processing.
Evaluating Streaming Data Solutions
What are the key considerations people should look at to decide on the right technology to meet their messaging and queuing need? This presentation provides an overview of key requirements and introduces Apache Pulsar, the open source messaging and queuing solution.
Rpsonmongodb
- Rediff News uses MongoDB for its publishing system to manage the lifecycle of articles, store article metadata and roles, acquire external feeds, enable tagging and notifications, and power search and data visualization on maps.
- The system allows users to upload Excel data, match it to map attributes, generate articles using data science insights, and visualize data on interactive maps.
- Rediff's architecture uses POJOs to define schemas, custom collections to store different data types, and a REST layer to expose data resources and abstract storage from applications.
Recommended
Microservices, Monoliths, SOA and How We Got Here
Microservice architecture (MSA) helps address modern software development needs by breaking applications into independently deployable microservices that communicate asynchronously. MSA evolves past SOA approaches by focusing on small, single-purpose services rather than large monolithic applications. Key aspects of MSA include ubiquitous language, well-defined boundaries, independent deployability, asynchronous communication via messaging, and each service owning its own data. MSA also relies on modern cloud-ready infrastructure approaches like service discovery, elastic scaling, and failure resilience at the service level rather than relying on specific hardware.
DBaaS at Scale
This document discusses Database as a Service (DBaaS). It begins by defining DBaaS and describing the tasks involved in managing databases, such as setup, configuration, monitoring, backups and upgrades. It then lists the benefits of DBaaS such as reduced time spent on management tasks and increased scalability. Requirements for DBaaS like security, availability and ease of use are outlined. Different DBaaS architectures including proxy-based and DNS-based models are presented, along with examples. The document concludes by comparing several DBaaS providers and listing useful ScaleChamp features and links.
Kafka Summit SF 2017 - Providing Reliability Guarantees in Kafka at One Trill...
In this presentation, I will talk about my firsthand experience dealing with the unique challenges of running Kafka at a massive scale. If you ever thought that running Kafka is difficult, this talk may change your mind and provide you with valuable insights into how to configure a Kafka cluster efficiently, how to manage Kafka for enterprise customers and how to measure, monitor and maintain the Quality of Kafka Service. Our production Kafka cluster runs over 1500+ VMs, and serves over 10 GBPS data spread across hundreds of topics for multiple teams across Microsoft. We built a self-serve Kafka management service to make the process manageable and scalable across many teams. In this talk, I will also share insights about running Kafka in Private vs multi-tenant mode, supporting failover and disaster recovery requirements, and how to make Kafka Compliant with regulatory certifications such as ISO, SOC, FEDRAMP, etc.
Presented by Nitin Kumar, Microsoft
Systems Track
SQL, Redis and Kubernetes by Paul Stanton of Windocks - Redis Day Seattle 2020
The document discusses common use cases for combining SQL, Redis, and Kubernetes including caching, session management, rate limiting, and data ingestion. It outlines how Kubernetes can be used for scaling microservices while Redis is used for data service scaling. The presentation proposes combining Redis, SQL Server, and Kubernetes with a proxy service, and describes using Redis for caching, session storage, and rate limiting of SQL data. It also discusses running Redis and front-end apps on Kubernetes and deploying SQL as a Kubernetes service through a proxy.
Apache pulsar - storage architecture
Apache Pulsar is a flexible pub-sub messaging system backed by a durable log storage. It uses a segment-centric architecture where messages are stored in independent segments across multiple brokers and bookies for redundancy. This allows for strong durability, high throughput, and seamless expansion without data rebalancing. Pulsar brokers serve client requests and acquire ownership of topics, while bookies provide durable storage with replication for fault tolerance.
Introduction to cloud computing
This document introduces cloud computing, which provides computing resources as a service via virtualization and shared resources. Key aspects include flexible provisioning without large upfront costs, cost-effectiveness through on-demand access without ownership costs, and scalability through elastic scaling of resources via API calls. Popular cloud services from Amazon are described, including compute, storage, content delivery, DNS, databases, messaging, and data processing.
Evaluating Streaming Data Solutions
What are the key considerations people should look at to decide on the right technology to meet their messaging and queuing need? This presentation provides an overview of key requirements and introduces Apache Pulsar, the open source messaging and queuing solution.
Rpsonmongodb
- Rediff News uses MongoDB for its publishing system to manage the lifecycle of articles, store article metadata and roles, acquire external feeds, enable tagging and notifications, and power search and data visualization on maps.
- The system allows users to upload Excel data, match it to map attributes, generate articles using data science insights, and visualize data on interactive maps.
- Rediff's architecture uses POJOs to define schemas, custom collections to store different data types, and a REST layer to expose data resources and abstract storage from applications.
Jay Kreps | Kafka Summit NYC 2019 Keynote (Events Everywhere) | CEO, Confluent
Jay Kreps is the CEO of Confluent, Inc., a company backing the popular Apache Kafka® messaging system. Prior to founding Confluent, he was formerly the lead architect for data infrastructure at LinkedIn. He is among the original authors of several open source projects including Project Voldemort (a key-value store). Apache Kafka (a distributed messaging system) and Apache Samza (a stream processing system).
Troubleshooting as Your Kafka Clusters Grow (Krunal Vora, Tinder) Kafka Summi...
The graphical curve of ‘getting things done with Kafka’ vs. ‘time spent with Kafka’ rises pretty quickly before starting to plateau. Kafka, as we know, is great to get started out of the box. Eventually though, anyone who has used Kafka for a reasonable amount of time would come across problems, both trivial and hardcore. More often than not, these problems feel tedious and repetitive which could be addressed better by the Kafka community. Issues like skewed partitions and brokers, partition reassignment for a large number of topics, working with messed up consumer offsets, distributing workload evenly to partitions, monitor and view the Kafka cluster better, etc. sound familiar among the community. Come, learn from our experiences trying to overcome and automate some of the most common problems surrounding the scaling Kafka clusters at Tinder. Let’s create an environment of sharing the best practices for usage of tools around Kafka.
Choosing the right Cloud Database
Database as a Service (DBaaS) is cloud database hosted and managed by the cloud service providers that can be accessed through public cloud or the hybrid cloud. The cloud provider takes care of provisioning, configuring, setup, maintenance, backups and patching the database. Customers are expected to export the database and start consuming the service through the pay-as-you-go model.
In his session at 5th Big Data Expo, Janakiram MSV will analyze the current market landscape while exploring the available options, strengths and weaknesses of current DBaaS players. He will highlight the key factors that enterprises should consider before adopting a cloud database platform.
Introducing Confluent Cloud: Apache Kafka as a Service
1. The document introduces Confluent Cloud, which provides Apache Kafka as a service and allows organizations to liberate developers from managing Kafka infrastructure.
2. Confluent Cloud is built on open source Apache Kafka and provides access to the rich Kafka ecosystem through pre-built connectors while minimizing operations burden by handling infrastructure setup, cluster tuning, upgrades and more.
3. Key benefits of Confluent Cloud include freedom to deploy Kafka in any public cloud, leveraging the leading streaming platform to develop faster applications, and being relieved of operations management through a fully managed service.
Using galera replication to create geo distributed clusters on the wan
We will show the advantages of having a geo-distributed database cluster and how to create one using Galera Cluster for MySQL. We will also discuss the configuration and status variables that are involved and how to deal with typical situations on the WAN such as slow, untrusted or unreliable links, latency and packet loss. We will demonstrate a multi-region cluster on Amazon EC2 and perform some throughput and latency measurements in real-time.
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
Best Practices for building Hybrid-Cloud Architectures - Hans Jespersen
Afternoon opening presentation during Confluent’s streaming event in Paris, presented by Hans Jespersen, VP WW Systems Engineering at Confluent.
Blr hadoop meetup
Suneet Grover from 24/7 Customer, Inc. presented on building resilient cross-DC data pipelines using Apache Kafka. 24/7 Customer uses Kafka for intent prediction, data analytics, and business intelligence across multiple data centers. Suneet discussed challenges with earlier Kafka versions, solutions implemented in Kafka 0.10, metrics and monitoring best practices, and design considerations for mirroring data between data centers.
CQRS
This document provides an overview of Command Query Responsibility Segregation (CQRS). It discusses some of the problems CQRS aims to solve, such as data-centric applications being CRUD-based and focusing on frameworks over the domain. CQRS separates commands, which capture user intent and mutate data, from queries, which return data without side effects. Commands and events update aggregate states and are handled asynchronously. Queries operate on denormalized read models optimized for specific views. This allows for quick queries without mappings and eventual consistency by working with stale but optimized data.
The 6 Rules for Modernizing Your Legacy Java Monolith with Microservices
We change a monolithic system only when we have no other choice. Traditional enterprise systems are tightly-coupled; all-in-one, all-or-nothing, difficult to scale, difficult to understand and difficult to maintain.
Rather than swiftly capture opportunity, we ponder if it’s really worth it—is it worth upsetting the delicate balance of the house of cards we call our enterprise system? Often the opportunity quickly disappears, captured by a faster company. Some people have started calling this “Getting Ubered”.
So what can you do about it? Talking about Microservices is one thing, but how can your organization start taking action to address this issue?
In this webinar by battle-hardened Enterprise Advocate, Kevin Webber, we walk through the 6 key concepts to understand as a guide for taking action:
1. Domain Driven Design (DDD)
2. Asynchronous messaging
3. API management
4. Dependency management
5. CQRS & event sourcing
6. Transactions & ordering
Reactive Platform has what you need to breath new life into your legacy system with a new Microservices-based approach.
Real World Event Sourcing and CQRS
Event Sourcing, Domain Driven Design, and Command Query Responsibility Segregation – we hear all of these technologies used together frequently, but how do they actually work together? How do you manage complex co-ordination in a CQRS system?
In this talk, we will discuss a real world example of DDD with ES and CQRS written in F# - a functional first language on the .NET Framework. We’ll take a deep dive into the F# algebraic type system that constructs the domain model. Also, the explanation of the abstract notion of a DDD aggregate root compared to the CQRS implementation of an aggregate root. Finishing with how sagas and triggers facilitate poly-aggregate communication.
Building data-driven microservices
Dr. Karthik Ramasamy of Streamlio draws on his experience building data products at companies including Pivotal, Twitter, and Streamlio to discuss technology and best practices for designing and implementing data-driven microservices:
* The key principles of microservices and microservice architecture
* The implications of microservices for data
* The role of messaging and processing technology in connecting microservices
Amazon web services (aws) main developer services
Developing in the cloud should not be different than on-premise development and AWS services aim to facilitate developers and reduce costs. The main AWS developer services that a developer should know include IAM for managing users and credentials, CloudWatch for monitoring and alarms, EC2 for renting compute resources, VPC for network management, S3 for object storage, DynamoDB for NoSQL database, RDS for relational databases, Elastic Beanstalk for deploying applications, and CloudFront for low-latency content delivery. Additional services are CloudFormation for provisioning via templates and Elasticache for caching.
Kafka Deployment to Steel Thread
Technical breakout during Confluent’s streaming event in Munich, presented by Sam Julian, Chief Cloud Engineer at E.On SE. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.
Kafka Summit NYC 2017 - The Real-time Event Driven Bank: A Kafka Story
This document discusses the use of Apache Kafka as the heart of ING's event-driven architecture. It outlines how ING implemented Kafka locally at first for fraud detection, then expanded to clusters for higher availability across data centers and regions. A global event bus structure was created using mirroring to allow events to be streamed and consumed across applications in different countries and regions on a single logical event bus. Governance, metadata and data lineage are important as events are used across multiple regions and drive by regulations.
How Yelp Leapt to Microservices with More than a Message Queue
Without seeing what’s wrong with today’s messaging queues, it can be initially confusing to view Apache Kafka as more. By adding additional functionality, true storage, and guarantees it opens opportunities to take full advantage of a publish/subscribe model.
Joined by Yelp’s Justin Cunningham we’ll see how their infrastructure has quickly evolved. Powered by Kafka, Yelp has made the leap to microservices and is seeing the benefits of efficiency and performance.
Speakers:
Justin Cunningham
Technical Lead, Software Engineer, Yelp
Gehrig Kunz
Technical Product Marketing Manager, Confluent
Microservices and Integration: what's next with Istio service mesh
This document discusses microservices architectures and emerging technologies to support them. It introduces Envoy proxy as a sidecar proxy that implements common microservices patterns like circuit breaking and load balancing. It then introduces Istio as a control plane that manages Envoy proxies and provides higher-level capabilities like traffic management, security, and observability across microservices. The presentation argues that 2018 will be the year of service meshes, with Istio being a prominent example for managing microservices communication using Envoy proxies.
Stream Reasoning/CEP
Kurze Zusammenfassung Über Stream Reasoning/CEP und Überblick zu Apache Storm/Apache Spark im Rahmen des #devspace / OpenSpace 2015 in Leipzig
Relevante "Grundlagen" die ich als Ausgangspunkt u.a. genommen habe:
http://www.kr.tuwien.ac.at/staff/beck/pub/ijcai15-bde.pdf
http://www.kr.tuwien.ac.at/staff/beck/pub/aaai2015.pdf
http://www.kr.tuwien.ac.at/staff/beck/slides/slides-aaai2015.pdf
CouchbasetoHadoop_Matt_Michael_Justin v4
This document discusses LinkedIn's use of Kafka, Hadoop, Storm, and Couchbase in their big data pipeline. It provides an overview of each technology and how LinkedIn uses them together. Specifically, it describes how LinkedIn uses Kafka to stream data to Hadoop for analytics and report generation. It also discusses how LinkedIn uses Hadoop to pre-build and warm Couchbase buckets for improved performance. The presentation includes a use case of streaming member profile and activity data through Kafka to both Hadoop and Couchbase clusters.
Kafka Summit NYC 2017 - Simplifying Omni-Channel Retail at Scale
This document summarizes Aaron Strey's presentation on how Target uses Apache Kafka to support omni-channel retail operations at scale. Some key points:
Target uses Kafka for log aggregation, threat detection, clickstream analysis, and business event messaging. They have over 1,800 stores and 38 distribution centers in the US, serving over 26 million online visitors per month.
Target's large Kafka deployment includes up to 300 topics per cluster, with 10-20 thousand consumer requests per second and compaction widely used. They aim for exactly once semantics across a diverse set of clients.
Strey suggests reinventing log aggregation to allow querying log streams directly from Kafka as easily as current methods using Elastic, to avoid indexing ter
Modeling microservices using DDD
The document discusses modeling microservices using Domain-Driven Design (DDD) for a drone delivery domain. It covers DDD concepts like bounded context, tactical patterns, and identifies microservices from aggregates and domain services. It then models the drone delivery domain applying DDD, identifying bounded contexts, domain model, and refining them into microservices for the shipping bounded context.
Tuning Java Driver for Apache Cassandra by Nenad Bozic at Big Data Spain 2017
Apache Cassandra is distributed masterless column store database which is becoming mainstream for analytics and IoT data.
https://www.bigdataspain.org/2017/talk/tuning-java-driver-for-apache-cassandra
Big Data Spain 2017
November 16th - 17th Kinépolis Madrid
Tuning Java Driver for Apache Cassandra
Apache Cassandra is distributed masterless column store database which is becoming mainstream for analytics and IoT data. Many use cases where Cassandra is natural fit require latency tuning in order to serve requests really fast. DataStax driver has many options, some less familiar, which can greatly influence performance aspect. This talk will focus on Java applications and options at your disposal in DataStax Java driver which became standard when you want to use this database. We will concentrate on both monitoring and tuning aspect of things and we will provide different options for different use cases. There is no silver bullet solution and having many options requires deep dive when you want to figure out right decision. This talk will narrow down options and give you push in the right direction.
More Related Content
What's hot
Jay Kreps | Kafka Summit NYC 2019 Keynote (Events Everywhere) | CEO, Confluent
Jay Kreps is the CEO of Confluent, Inc., a company backing the popular Apache Kafka® messaging system. Prior to founding Confluent, he was formerly the lead architect for data infrastructure at LinkedIn. He is among the original authors of several open source projects including Project Voldemort (a key-value store). Apache Kafka (a distributed messaging system) and Apache Samza (a stream processing system).
Troubleshooting as Your Kafka Clusters Grow (Krunal Vora, Tinder) Kafka Summi...
The graphical curve of ‘getting things done with Kafka’ vs. ‘time spent with Kafka’ rises pretty quickly before starting to plateau. Kafka, as we know, is great to get started out of the box. Eventually though, anyone who has used Kafka for a reasonable amount of time would come across problems, both trivial and hardcore. More often than not, these problems feel tedious and repetitive which could be addressed better by the Kafka community. Issues like skewed partitions and brokers, partition reassignment for a large number of topics, working with messed up consumer offsets, distributing workload evenly to partitions, monitor and view the Kafka cluster better, etc. sound familiar among the community. Come, learn from our experiences trying to overcome and automate some of the most common problems surrounding the scaling Kafka clusters at Tinder. Let’s create an environment of sharing the best practices for usage of tools around Kafka.
Choosing the right Cloud Database
Database as a Service (DBaaS) is cloud database hosted and managed by the cloud service providers that can be accessed through public cloud or the hybrid cloud. The cloud provider takes care of provisioning, configuring, setup, maintenance, backups and patching the database. Customers are expected to export the database and start consuming the service through the pay-as-you-go model.
In his session at 5th Big Data Expo, Janakiram MSV will analyze the current market landscape while exploring the available options, strengths and weaknesses of current DBaaS players. He will highlight the key factors that enterprises should consider before adopting a cloud database platform.
Introducing Confluent Cloud: Apache Kafka as a Service
1. The document introduces Confluent Cloud, which provides Apache Kafka as a service and allows organizations to liberate developers from managing Kafka infrastructure.
2. Confluent Cloud is built on open source Apache Kafka and provides access to the rich Kafka ecosystem through pre-built connectors while minimizing operations burden by handling infrastructure setup, cluster tuning, upgrades and more.
3. Key benefits of Confluent Cloud include freedom to deploy Kafka in any public cloud, leveraging the leading streaming platform to develop faster applications, and being relieved of operations management through a fully managed service.
Using galera replication to create geo distributed clusters on the wan
We will show the advantages of having a geo-distributed database cluster and how to create one using Galera Cluster for MySQL. We will also discuss the configuration and status variables that are involved and how to deal with typical situations on the WAN such as slow, untrusted or unreliable links, latency and packet loss. We will demonstrate a multi-region cluster on Amazon EC2 and perform some throughput and latency measurements in real-time.
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
Best Practices for building Hybrid-Cloud Architectures - Hans Jespersen
Afternoon opening presentation during Confluent’s streaming event in Paris, presented by Hans Jespersen, VP WW Systems Engineering at Confluent.
Blr hadoop meetup
Suneet Grover from 24/7 Customer, Inc. presented on building resilient cross-DC data pipelines using Apache Kafka. 24/7 Customer uses Kafka for intent prediction, data analytics, and business intelligence across multiple data centers. Suneet discussed challenges with earlier Kafka versions, solutions implemented in Kafka 0.10, metrics and monitoring best practices, and design considerations for mirroring data between data centers.
CQRS
This document provides an overview of Command Query Responsibility Segregation (CQRS). It discusses some of the problems CQRS aims to solve, such as data-centric applications being CRUD-based and focusing on frameworks over the domain. CQRS separates commands, which capture user intent and mutate data, from queries, which return data without side effects. Commands and events update aggregate states and are handled asynchronously. Queries operate on denormalized read models optimized for specific views. This allows for quick queries without mappings and eventual consistency by working with stale but optimized data.
The 6 Rules for Modernizing Your Legacy Java Monolith with Microservices
We change a monolithic system only when we have no other choice. Traditional enterprise systems are tightly-coupled; all-in-one, all-or-nothing, difficult to scale, difficult to understand and difficult to maintain.
Rather than swiftly capture opportunity, we ponder if it’s really worth it—is it worth upsetting the delicate balance of the house of cards we call our enterprise system? Often the opportunity quickly disappears, captured by a faster company. Some people have started calling this “Getting Ubered”.
So what can you do about it? Talking about Microservices is one thing, but how can your organization start taking action to address this issue?
In this webinar by battle-hardened Enterprise Advocate, Kevin Webber, we walk through the 6 key concepts to understand as a guide for taking action:
1. Domain Driven Design (DDD)
2. Asynchronous messaging
3. API management
4. Dependency management
5. CQRS & event sourcing
6. Transactions & ordering
Reactive Platform has what you need to breath new life into your legacy system with a new Microservices-based approach.
Real World Event Sourcing and CQRS
Event Sourcing, Domain Driven Design, and Command Query Responsibility Segregation – we hear all of these technologies used together frequently, but how do they actually work together? How do you manage complex co-ordination in a CQRS system?
In this talk, we will discuss a real world example of DDD with ES and CQRS written in F# - a functional first language on the .NET Framework. We’ll take a deep dive into the F# algebraic type system that constructs the domain model. Also, the explanation of the abstract notion of a DDD aggregate root compared to the CQRS implementation of an aggregate root. Finishing with how sagas and triggers facilitate poly-aggregate communication.
Building data-driven microservices
Dr. Karthik Ramasamy of Streamlio draws on his experience building data products at companies including Pivotal, Twitter, and Streamlio to discuss technology and best practices for designing and implementing data-driven microservices:
* The key principles of microservices and microservice architecture
* The implications of microservices for data
* The role of messaging and processing technology in connecting microservices
Amazon web services (aws) main developer services
Developing in the cloud should not be different than on-premise development and AWS services aim to facilitate developers and reduce costs. The main AWS developer services that a developer should know include IAM for managing users and credentials, CloudWatch for monitoring and alarms, EC2 for renting compute resources, VPC for network management, S3 for object storage, DynamoDB for NoSQL database, RDS for relational databases, Elastic Beanstalk for deploying applications, and CloudFront for low-latency content delivery. Additional services are CloudFormation for provisioning via templates and Elasticache for caching.
Kafka Deployment to Steel Thread
Technical breakout during Confluent’s streaming event in Munich, presented by Sam Julian, Chief Cloud Engineer at E.On SE. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.
Kafka Summit NYC 2017 - The Real-time Event Driven Bank: A Kafka Story
This document discusses the use of Apache Kafka as the heart of ING's event-driven architecture. It outlines how ING implemented Kafka locally at first for fraud detection, then expanded to clusters for higher availability across data centers and regions. A global event bus structure was created using mirroring to allow events to be streamed and consumed across applications in different countries and regions on a single logical event bus. Governance, metadata and data lineage are important as events are used across multiple regions and drive by regulations.
How Yelp Leapt to Microservices with More than a Message Queue
Without seeing what’s wrong with today’s messaging queues, it can be initially confusing to view Apache Kafka as more. By adding additional functionality, true storage, and guarantees it opens opportunities to take full advantage of a publish/subscribe model.
Joined by Yelp’s Justin Cunningham we’ll see how their infrastructure has quickly evolved. Powered by Kafka, Yelp has made the leap to microservices and is seeing the benefits of efficiency and performance.
Speakers:
Justin Cunningham
Technical Lead, Software Engineer, Yelp
Gehrig Kunz
Technical Product Marketing Manager, Confluent
Microservices and Integration: what's next with Istio service mesh
This document discusses microservices architectures and emerging technologies to support them. It introduces Envoy proxy as a sidecar proxy that implements common microservices patterns like circuit breaking and load balancing. It then introduces Istio as a control plane that manages Envoy proxies and provides higher-level capabilities like traffic management, security, and observability across microservices. The presentation argues that 2018 will be the year of service meshes, with Istio being a prominent example for managing microservices communication using Envoy proxies.
Stream Reasoning/CEP
Kurze Zusammenfassung Über Stream Reasoning/CEP und Überblick zu Apache Storm/Apache Spark im Rahmen des #devspace / OpenSpace 2015 in Leipzig
Relevante "Grundlagen" die ich als Ausgangspunkt u.a. genommen habe:
http://www.kr.tuwien.ac.at/staff/beck/pub/ijcai15-bde.pdf
http://www.kr.tuwien.ac.at/staff/beck/pub/aaai2015.pdf
http://www.kr.tuwien.ac.at/staff/beck/slides/slides-aaai2015.pdf
CouchbasetoHadoop_Matt_Michael_Justin v4
This document discusses LinkedIn's use of Kafka, Hadoop, Storm, and Couchbase in their big data pipeline. It provides an overview of each technology and how LinkedIn uses them together. Specifically, it describes how LinkedIn uses Kafka to stream data to Hadoop for analytics and report generation. It also discusses how LinkedIn uses Hadoop to pre-build and warm Couchbase buckets for improved performance. The presentation includes a use case of streaming member profile and activity data through Kafka to both Hadoop and Couchbase clusters.
Kafka Summit NYC 2017 - Simplifying Omni-Channel Retail at Scale
This document summarizes Aaron Strey's presentation on how Target uses Apache Kafka to support omni-channel retail operations at scale. Some key points:
Target uses Kafka for log aggregation, threat detection, clickstream analysis, and business event messaging. They have over 1,800 stores and 38 distribution centers in the US, serving over 26 million online visitors per month.
Target's large Kafka deployment includes up to 300 topics per cluster, with 10-20 thousand consumer requests per second and compaction widely used. They aim for exactly once semantics across a diverse set of clients.
Strey suggests reinventing log aggregation to allow querying log streams directly from Kafka as easily as current methods using Elastic, to avoid indexing ter
Modeling microservices using DDD
The document discusses modeling microservices using Domain-Driven Design (DDD) for a drone delivery domain. It covers DDD concepts like bounded context, tactical patterns, and identifies microservices from aggregates and domain services. It then models the drone delivery domain applying DDD, identifying bounded contexts, domain model, and refining them into microservices for the shipping bounded context.
What's hot (20)
Jay Kreps | Kafka Summit NYC 2019 Keynote (Events Everywhere) | CEO, Confluent
Jay Kreps | Kafka Summit NYC 2019 Keynote (Events Everywhere) | CEO, Confluent
Troubleshooting as Your Kafka Clusters Grow (Krunal Vora, Tinder) Kafka Summi...
Troubleshooting as Your Kafka Clusters Grow (Krunal Vora, Tinder) Kafka Summi...
Introducing Confluent Cloud: Apache Kafka as a Service
Introducing Confluent Cloud: Apache Kafka as a Service
Using galera replication to create geo distributed clusters on the wan
Using galera replication to create geo distributed clusters on the wan
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
The 6 Rules for Modernizing Your Legacy Java Monolith with Microservices
The 6 Rules for Modernizing Your Legacy Java Monolith with Microservices
Kafka Summit NYC 2017 - The Real-time Event Driven Bank: A Kafka Story
Kafka Summit NYC 2017 - The Real-time Event Driven Bank: A Kafka Story
How Yelp Leapt to Microservices with More than a Message Queue
How Yelp Leapt to Microservices with More than a Message Queue
Microservices and Integration: what's next with Istio service mesh
Microservices and Integration: what's next with Istio service mesh
Kafka Summit NYC 2017 - Simplifying Omni-Channel Retail at Scale
Kafka Summit NYC 2017 - Simplifying Omni-Channel Retail at Scale
Similar to Scaling distributed data systems: A LinkedIn Case study
Tuning Java Driver for Apache Cassandra by Nenad Bozic at Big Data Spain 2017
Apache Cassandra is distributed masterless column store database which is becoming mainstream for analytics and IoT data.
https://www.bigdataspain.org/2017/talk/tuning-java-driver-for-apache-cassandra
Big Data Spain 2017
November 16th - 17th Kinépolis Madrid
Tuning Java Driver for Apache Cassandra
Apache Cassandra is distributed masterless column store database which is becoming mainstream for analytics and IoT data. Many use cases where Cassandra is natural fit require latency tuning in order to serve requests really fast. DataStax driver has many options, some less familiar, which can greatly influence performance aspect. This talk will focus on Java applications and options at your disposal in DataStax Java driver which became standard when you want to use this database. We will concentrate on both monitoring and tuning aspect of things and we will provide different options for different use cases. There is no silver bullet solution and having many options requires deep dive when you want to figure out right decision. This talk will narrow down options and give you push in the right direction.
Amazon`s Dynamo
Dynamo is a highly available and scalable key-value store designed by Amazon to power core services. It uses an eventually consistent model where replicas may diverge temporarily during network partitions or failures but eventually converge. The system is decentralized and partitions data using consistent hashing across nodes. It provides high availability even during failures through mechanisms like hinted handoff, replica synchronization, and gossip-based membership protocols.
Using Apache Cassandra and Apache Kafka to Scale Next Gen Applications
Adoption of open source software (OSS) at the enterprise level has flourished, as more businesses discover the considerable advantages that open source solutions hold over their proprietary counterparts, and as the enterprise mentality around open source continues to shift. We will discuss how to identify good application candidates for Apache Cassandra and Kafka as well as best practices and common pitfalls.
This presentation will also cover:
The origins of Apache Cassandra and Kafka and how these technologies have come to shape how next-gen applications are built.
Production use cases of Cassandra and Kafka: Real-time payments and buying a house (Lendi and Worldpay)
Core concepts that make the magic; Explaining the technical attributes that make your project a good fit for these technologies and the architectural patterns that make the best use of it’s capability.
Speaker: Adam Zegelin, SVP Engineering and Co-Founder, Instaclustr
As Instaclustr's founding software engineer, Adam provides the foundation knowledge of Instaclustr's capability and engineering environment. Adam is also focused on providing Instaclustr's contribution to the broader open source community on which our products and the services rely, including Apache Cassandra, Apache Spark, and other core technologies such as CoreOS and Docker. Prior to founding Instaclustr, Adam worked on large-scale big data projects with Australian Government agencies.
Building a highly scalable and available cloud application
This document discusses lessons learned from building large, scalable applications on Azure. It emphasizes designing for scale from the start by making applications stateless and partitioning data. It also stresses designing for failure since failures will occur at large scale. Other key lessons include optimizing for density to reduce costs, using telemetry to monitor applications, and handling transient and enduring failures through retries and failover. The presenter concludes by offering to share more detailed guidance and reusable patterns for building scalable Azure applications.
NoSQL – Data Center Centric Application Enablement
The growth of Datacenter infrastructure is trending out of bounds, along with the pace in user activity and data generation in this digital era. However, the nature of the typical application deployment within the data center is changing to accommodate new business needs. Those changes introduce complexities in application deployment architecture and design, which cascade into requirements for a new generation of database technology (NoSQL) destined to ease that complexity. This webcast will discuss the modern data centers data centric application, the complexities that must be dealt with and common architectures found to describe and prescribe new data center aware services. Well look at the practical issues in implementation and overview current state of art in NoSQL database technology solving the problems of data center awareness in application development.
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Discover how to avoid common pitfalls when shifting to an event-driven architecture (EDA) in order to boost system recovery and scalability. We cover Kafka Schema Registry, in-broker transformations, event sourcing, and more.
Select Stars: A SQL DBA's Introduction to Azure Cosmos DB (SQL Saturday Orego...
Select Stars: A SQL DBA's Introduction to Azure Cosmos DB - Presented at SQL Saturday Oregon on 3 November 2018
Azure DocumentDB Overview
DocumentDB is a fast, globally distributed, multi-model NoSQL database service. It provides automatic scaling of storage and throughput, high availability across regions, flexible data models, and developer productivity with support for SQL and JavaScript queries. Customers can use DocumentDB for building scalable applications that need to handle large volumes of data across any number of regions worldwide with low latency and high availability.
Data Pipelines with Spark & DataStax Enterprise
This document discusses building data pipelines for both static and streaming data using Apache Spark and DataStax Enterprise (DSE). For static data, it recommends using optimized data storage formats, distributed and scalable technologies like Spark, interactive analysis tools like notebooks, and DSE for persistent storage. For streaming data, it recommends using scalable distributed technologies, Kafka to decouple producers and consumers, and DSE for real-time analytics and persistent storage across datacenters.
MySQL NDB Cluster 101
This document provides an overview of MySQL Cluster, a distributed, in-memory database that provides high availability, scale-out, and real-time performance. Key points include:
- MySQL Cluster can scale linearly to handle massive workloads through data sharding and replication across nodes. It offers 99.9999% availability.
- It is open source and can be used standalone or with MySQL. Data is partitioned and distributed automatically across nodes with no single point of failure.
- It is used by many large companies and systems that require high throughput, low latency access to large datasets, including for telecom, gaming, and financial applications.
Data Stream Processing for Beginners with Kafka and CDC
This document provides an overview of data stream processing using Apache Kafka and change data capture (CDC). It defines CDC as identifying and capturing changes made to a database. CDC is useful for applications like data replication, microservice architectures, and caching. The document discusses how Kafka and Kafka Connect can be used to capture change data from databases logs and send them as a stream of events. It also introduces Debezium, an open-source CDC connector for databases like MySQL, PostgreSQL and MongoDB. Finally, it mentions that a live demo will be shown of capturing change events from a MySQL database using Debezium and viewing them in a Kafka topic.
Got documents - The Raven Bouns Edition
Slides related to architectural considerations of document databases, with some additional RavenDB information added in.
Ai big dataconference_ml_fastdata_vitalii bondarenko
This document discusses machine learning on fast data. It presents an agenda covering ML on production systems, TensorFlow, Kafka, Docker and Kubernetes. It then describes the machine learning process and shows how an enterprise analytics platform can integrate data sources, a machine learning cluster using Kafka, and data destinations. It provides details on TensorFlow and how it can be used for linear regression and neural networks. It also explains Apache Kafka as a streaming data service bus and how Confluent Platform extends it. Finally, it briefly introduces Docker and Kubernetes.
Vitalii Bondarenko "Machine Learning on Fast Data"
This document discusses machine learning on fast data. It presents an agenda covering ML on production systems, TensorFlow, Kafka, Docker and Kubernetes. It then describes the machine learning process and shows how an enterprise analytics platform can integrate data sources, a machine learning cluster using Kafka, and data destinations. Details are provided on using TensorFlow for linear regression and neural networks. Apache Kafka is explained as a distributed streaming platform using topics, brokers, and consumer groups. The Confluent platform, KStream and KTable APIs are also summarized. Docker and Kubernetes are mentioned for containerization.
Cloud computing
This document discusses various considerations for cloud computing applications including scalability, the cloud environment, cloud native applications, common cloud patterns, and challenges. Specifically, it covers:
- Achieving horizontal and vertical scalability by adding/increasing compute and storage resources
- How the cloud enables on-demand resources and pay-per-use models
- Designing applications to leverage cloud services and scale horizontally across nodes
- Common patterns for queue-based workflows, auto-scaling, data partitioning, and handling failures
- Challenges of network latency, node failures, and eventual consistency in distributed systems
Cloud Computing - Geektalk
This document discusses various considerations for cloud computing applications including scalability, the cloud environment, cloud native applications, common cloud patterns, and challenges. Specifically, it covers:
- Achieving horizontal and vertical scalability by adding/increasing compute and storage resources
- How the cloud enables on-demand resources and pay-per-use models
- Designing applications to leverage cloud services and scale horizontally across nodes
- Common patterns for queue-based workflows, auto-scaling, data partitioning, and handling failures
- Challenges of network latency, node failures, and transient service unavailability
Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...
We will start from understanding how Real-Time Analytics can be implemented on Enterprise Level Infrastructure and will go to details and discover how different cases of business intelligence be used in real-time on streaming data. We will cover different Stream Data Processing Architectures and discus their benefits and disadvantages. I'll show with live demos how to build Fast Data Platform in Azure Cloud using open source projects: Apache Kafka, Apache Cassandra, Mesos. Also I'll show examples and code from real projects.
Got documents?
This document provides an overview of document databases, comparing MongoDB, CouchDB, and RavenDB. It discusses how document databases work by storing related data in documents rather than normalizing across tables. It also covers considerations like schema flexibility, ACID transactions, modeling aggregates, scaling out, indexing, eventual consistency, and administrative requirements. Two case studies demonstrate how document databases were used to model survey and CRM systems.
Got documents Code Mash Revision
This document provides an overview of several document database technologies including MongoDB, CouchDB, and RavenDB. It discusses key architectural considerations for using document databases such as their schema-free model, eventual consistency, ability to model object aggregates, scaling out through sharding and replication, need for queries to use indexes, and ongoing administration requirements. It also presents two case studies where document databases were used for a survey system and a CRM.
Similar to Scaling distributed data systems: A LinkedIn Case study (20)
Tuning Java Driver for Apache Cassandra by Nenad Bozic at Big Data Spain 2017
Tuning Java Driver for Apache Cassandra by Nenad Bozic at Big Data Spain 2017
Using Apache Cassandra and Apache Kafka to Scale Next Gen Applications
Using Apache Cassandra and Apache Kafka to Scale Next Gen Applications
Building a highly scalable and available cloud application
Building a highly scalable and available cloud application
NoSQL – Data Center Centric Application Enablement
NoSQL – Data Center Centric Application Enablement
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Select Stars: A SQL DBA's Introduction to Azure Cosmos DB (SQL Saturday Orego...
Select Stars: A SQL DBA's Introduction to Azure Cosmos DB (SQL Saturday Orego...
Data Stream Processing for Beginners with Kafka and CDC
Data Stream Processing for Beginners with Kafka and CDC
Ai big dataconference_ml_fastdata_vitalii bondarenko
Ai big dataconference_ml_fastdata_vitalii bondarenko
Vitalii Bondarenko "Machine Learning on Fast Data"
Vitalii Bondarenko "Machine Learning on Fast Data"
Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...
Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...
Recently uploaded
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressions
K8sGPT is a tool that analyzes and diagnoses Kubernetes clusters. This presentation was used to share the requirements and dependencies to deploy K8sGPT in a local environment.
Iron and Steel Technology Roadmap - Towards more sustainable steelmaking.pdf
Iron and Steel Technology towards Sustainable Steelmaking
bank management system in java and mysql report1.pdf
truth is high but higher still is truth full living
Recycled Concrete Aggregate in Construction Part III
Using recycled concrete aggregates (RCA) for pavements is crucial to achieving sustainability. Implementing RCA for new pavement can minimize carbon footprint, conserve natural resources, reduce harmful emissions, and lower life cycle costs. Compared to natural aggregate (NA), RCA pavement has fewer comprehensive studies and sustainability assessments.
Embedded machine learning-based road conditions and driving behavior monitoring
Car accident rates have increased in recent years, resulting in losses in human lives, properties, and other financial costs. An embedded machine learning-based system is developed to address this critical issue. The system can monitor road conditions, detect driving patterns, and identify aggressive driving behaviors. The system is based on neural networks trained on a comprehensive dataset of driving events, driving styles, and road conditions. The system effectively detects potential risks and helps mitigate the frequency and impact of accidents. The primary goal is to ensure the safety of drivers and vehicles. Collecting data involved gathering information on three key road events: normal street and normal drive, speed bumps, circular yellow speed bumps, and three aggressive driving actions: sudden start, sudden stop, and sudden entry. The gathered data is processed and analyzed using a machine learning system designed for limited power and memory devices. The developed system resulted in 91.9% accuracy, 93.6% precision, and 92% recall. The achieved inference time on an Arduino Nano 33 BLE Sense with a 32-bit CPU running at 64 MHz is 34 ms and requires 2.6 kB peak RAM and 139.9 kB program flash memory, making it suitable for resource-constrained embedded systems.
A review on techniques and modelling methodologies used for checking electrom...
The proper function of the integrated circuit (IC) in an inhibiting electromagnetic environment has always been a serious concern throughout the decades of revolution in the world of electronics, from disjunct devices to today’s integrated circuit technology, where billions of transistors are combined on a single chip. The automotive industry and smart vehicles in particular, are confronting design issues such as being prone to electromagnetic interference (EMI). Electronic control devices calculate incorrect outputs because of EMI and sensors give misleading values which can prove fatal in case of automotives. In this paper, the authors have non exhaustively tried to review research work concerned with the investigation of EMI in ICs and prediction of this EMI using various modelling methodologies and measurement setups.
Low power architecture of logic gates using adiabatic techniques
The growing significance of portable systems to limit power consumption in ultra-large-scale-integration chips of very high density, has recently led to rapid and inventive progresses in low-power design. The most effective technique is adiabatic logic circuit design in energy-efficient hardware. This paper presents two adiabatic approaches for the design of low power circuits, modified positive feedback adiabatic logic (modified PFAL) and the other is direct current diode based positive feedback adiabatic logic (DC-DB PFAL). Logic gates are the preliminary components in any digital circuit design. By improving the performance of basic gates, one can improvise the whole system performance. In this paper proposed circuit design of the low power architecture of OR/NOR, AND/NAND, and XOR/XNOR gates are presented using the said approaches and their results are analyzed for powerdissipation, delay, power-delay-product and rise time and compared with the other adiabatic techniques along with the conventional complementary metal oxide semiconductor (CMOS) designs reported in the literature. It has been found that the designs with DC-DB PFAL technique outperform with the percentage improvement of 65% for NOR gate and 7% for NAND gate and 34% for XNOR gate over the modified PFAL techniques at 10 MHz respectively.
[JPP-1] - (JEE 3.0) - Kinematics 1D - 14th May..pdf
Kinematics 11th jpp- 01. ( Solved ) unacademy namo kaul on 14th may...
Harnessing WebAssembly for Real-time Stateless Streaming Pipelines
Traditionally, dealing with real-time data pipelines has involved significant overhead, even for straightforward tasks like data transformation or masking. However, in this talk, we’ll venture into the dynamic realm of WebAssembly (WASM) and discover how it can revolutionize the creation of stateless streaming pipelines within a Kafka (Redpanda) broker. These pipelines are adept at managing low-latency, high-data-volume scenarios.
22CYT12-Unit-V-E Waste and its Management.ppt
Introduction- e - waste – definition - sources of e-waste– hazardous substances in e-waste - effects of e-waste on environment and human health- need for e-waste management– e-waste handling rules - waste minimization techniques for managing e-waste – recycling of e-waste - disposal treatment methods of e- waste – mechanism of extraction of precious metal from leaching solution-global Scenario of E-waste – E-waste in India- case studies.
Recently uploaded (20)
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressions
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressions
Iron and Steel Technology Roadmap - Towards more sustainable steelmaking.pdf
Iron and Steel Technology Roadmap - Towards more sustainable steelmaking.pdf
bank management system in java and mysql report1.pdf
bank management system in java and mysql report1.pdf
Recycled Concrete Aggregate in Construction Part III
Recycled Concrete Aggregate in Construction Part III
Embedded machine learning-based road conditions and driving behavior monitoring
Embedded machine learning-based road conditions and driving behavior monitoring
A review on techniques and modelling methodologies used for checking electrom...
A review on techniques and modelling methodologies used for checking electrom...
Low power architecture of logic gates using adiabatic techniques
Low power architecture of logic gates using adiabatic techniques
[JPP-1] - (JEE 3.0) - Kinematics 1D - 14th May..pdf
[JPP-1] - (JEE 3.0) - Kinematics 1D - 14th May..pdf
basic-wireline-operations-course-mahmoud-f-radwan.pdf
basic-wireline-operations-course-mahmoud-f-radwan.pdf
Harnessing WebAssembly for Real-time Stateless Streaming Pipelines
Harnessing WebAssembly for Real-time Stateless Streaming Pipelines
Scaling distributed data systems: A LinkedIn Case study
- 1. Scaling distributed Data Systems: a LinkedIn Case study By Sai Kiran Kanuri
- 2. Agenda • Brief intro to Espresso • Challenges & Approaches • Additional reading
- 3. Espresso • A distributed, fault tolerant NoSQL DB • ~ 7 PB of data footprint in a colo • Multi Master cross colo support • Bridges gap between RDBMS & k-v stores
- 7. Challenges
- 8. Fault Tolerance Challenges • Nodes fail all the time • Failover time • Data replication • Partition movement during failures • Service Discovery
- 9. Approach to Fault Tolerance • Minimize resource sizes • Distribute data movement across cluster • Throttle data migration • Minimize latency in control events • Load balancing with error back-off • Read after write consistency requirements • Enable client re-tries based on response codes
- 10. Multi-Tenancy • Security • Data growth • Hot partitions • Service Discovery
- 11. Approach to Multi-Tenancy • Ensure tenant isolation • Quotas • Have a mechanism to transparently migrate data from one cluster to another • Be able to re-partition existing data • Data movement should be transparent to customer.
- 12. Future Changes • Improving on MTTR with P2P communication • Cross cluster data balancer • Dynamic data re-partitioning • Weighted nodes & partitions
- 13. Additional Reading • Espresso - https://engineering.linkedin.com/espresso/introducing-espresso- linkedins-hot-new-distributed-document-store • R2D2 - https://github.com/linkedin/rest.li • Brooklin - https://engineering.linkedin.com/blog/2017/10/streaming-data- pipelines-with-brooklin • Helix - https://engineering.linkedin.com/apache-helix/apache-helix-framework- distributed-system-development