Introducing Apache Kafka - a visual overview. Presented at the Canberra Big Data Meetup 7 February 2019. We build a Kafka "postal service" to explain the main Kafka concepts, and explain how consumers receive different messages depending on whether there's a key or not.
The document provides an overview of a 5-day API Management training agenda with the following key points:
- Day 1 covers an introduction to architecture, components, concepts, and deployment models.
- Day 2 focuses on installing and setting up the API Gateway, virtualizing an API, and covering the "Hello API" lab.
- Day 3 goes deeper into using the Policy Studio with additional labs.
- Day 4 discusses advanced filters and security topics.
- Day 5 covers administration and advanced setup.
Kafka Intro With Simple Java Producer ConsumersJean-Paul Azar
Introduction to Kafka streaming platform. Covers Kafka Architecture with some small examples from the command line. Then we expand on this with a multi-server example. Lastly, we added some simple Java client examples for a Kafka Producer and a Kafka Consumer.
AWS Directory Service enables you to create a new Active Directory domain in AWS with Simple AD or to connect your existing Active Directory domain with AD Connector. Learn how to use these offerings to domain join and enable single sign-on (SSO) to your Amazon EC2 Windows and Linux instances, set up federated access to the AWS Management Console, and use Amazon WorkSpaces, Amazon WorkDocs, and Amazon WorkMail.
(ENT206) Migrating Thousands of Workloads to AWS at Enterprise Scale | AWS re...Amazon Web Services
Migrating workloads to AWS in an enterprise environment is not easy, but with the right approach, an enterprise-sized organization can migrate thousands of instances to AWS quickly and cost effectively. You can leave this session with a good understanding of the migration framework used to assess an enterprise application portfolio and how to move thousands of instances to AWS in a quick and repeatable fashion.
In this session, we describe the components of Accenture's cloud migration framework, including tools and capabilities provided by Accenture, AWS, and third-party software solutions, and how enterprises can leverage these techniques to migrate efficiently and effectively. The migration framework covers:
- Defining an overall cloud strategy
- Assessing the business requirements, including application and data requirements
- Creating the right AWS architecture and environment
- Moving applications and data using automated migration tools- Services to manage the migrated environment
Introducing Apache Kafka - a visual overview. Presented at the Canberra Big Data Meetup 7 February 2019. We build a Kafka "postal service" to explain the main Kafka concepts, and explain how consumers receive different messages depending on whether there's a key or not.
The document provides an overview of a 5-day API Management training agenda with the following key points:
- Day 1 covers an introduction to architecture, components, concepts, and deployment models.
- Day 2 focuses on installing and setting up the API Gateway, virtualizing an API, and covering the "Hello API" lab.
- Day 3 goes deeper into using the Policy Studio with additional labs.
- Day 4 discusses advanced filters and security topics.
- Day 5 covers administration and advanced setup.
Kafka Intro With Simple Java Producer ConsumersJean-Paul Azar
Introduction to Kafka streaming platform. Covers Kafka Architecture with some small examples from the command line. Then we expand on this with a multi-server example. Lastly, we added some simple Java client examples for a Kafka Producer and a Kafka Consumer.
AWS Directory Service enables you to create a new Active Directory domain in AWS with Simple AD or to connect your existing Active Directory domain with AD Connector. Learn how to use these offerings to domain join and enable single sign-on (SSO) to your Amazon EC2 Windows and Linux instances, set up federated access to the AWS Management Console, and use Amazon WorkSpaces, Amazon WorkDocs, and Amazon WorkMail.
(ENT206) Migrating Thousands of Workloads to AWS at Enterprise Scale | AWS re...Amazon Web Services
Migrating workloads to AWS in an enterprise environment is not easy, but with the right approach, an enterprise-sized organization can migrate thousands of instances to AWS quickly and cost effectively. You can leave this session with a good understanding of the migration framework used to assess an enterprise application portfolio and how to move thousands of instances to AWS in a quick and repeatable fashion.
In this session, we describe the components of Accenture's cloud migration framework, including tools and capabilities provided by Accenture, AWS, and third-party software solutions, and how enterprises can leverage these techniques to migrate efficiently and effectively. The migration framework covers:
- Defining an overall cloud strategy
- Assessing the business requirements, including application and data requirements
- Creating the right AWS architecture and environment
- Moving applications and data using automated migration tools- Services to manage the migrated environment
Achieving a 50% Reduction in Cross-AZ Network Costs from Kafka (Uday Sagar Si...confluent
Cloud providers like AWS allow free data transfers within an Availability Zone (AZ), but bill users when data moves between AZs. When the data volume streamed through Kafka reaches big data scale, (e.g. numeric data points or user activity tracking), the costs incurred by cross-AZ traffic can add significantly to your monthly cloud spend. Since Kafka serves reads and writes only from leader partitions, for a topic with a replication factor of 3, a message sent through Kafka can cross AZs up to 4 times. Once when a producer produces a message onto broker in a different AZ, two times during Kafka replication, and once more during message consumption. With careful design, we can eliminate the first and last part of the cross AZ traffic. We can also use message compression strategies provided by Kafka to reduce costs during replication. In this talk, we will discuss the architectural choices that allow us to ensure a Kafka message is produced and consumed within a single AZ, as well as an algorithm that lets consumers intelligently subscribe to partitions with leaders in the same AZ. We will also cover use cases in which cross-AZ message streaming is unavoidable due to design limitations. Talk outline: 1) A review of Kafka replication, 2) Cross-AZ traffic implications, 3) Architectural choices for AZ-aware message streaming, 4) Algorithms for AZ-aware producers and consumers, 5) Results, 6) Limitations, 7) Takeaways.
Watch this talk here: https://www.confluent.io/online-talks/apache-kafka-architecture-and-fundamentals-explained-on-demand
This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput.
This talk provides a comprehensive overview of Kafka architecture and internal functions, including:
-Topics, partitions and segments
-The commit log and streams
-Brokers and broker replication
-Producer basics
-Consumers, consumer groups and offsets
This session is part 2 of 4 in our Fundamentals for Apache Kafka series.
AWS Glue is a serverless data integration service that allows users to discover, prepare, and transform data for analytics and machine learning. It provides a fully managed extract, transform, and load (ETL) service on AWS. AWS Glue crawls data sources, automatically extracts metadata and stores it in a centralized data catalog. It then executes ETL jobs developed by users to clean, enrich and move data between various data stores.
The document discusses challenges of deploying Kubernetes on-premise, including how load balancers are provisioned without cloud providers, using Nginx and Haproxy for load balancing on bare metal. It also covers how persistent volumes are provisioned with CSI drivers like Ember CSI to interface with storage backends, and tools for deploying and managing on-premise Kubernetes clusters like RKE.
This document discusses cloud-native applications and serverless computing. It begins with an introduction to cloud-native applications and core technologies like containers, orchestrators, and microservices. Examples are then given of how companies like Fujifilm and ASOS have benefited from serverless architectures on Azure. The document concludes with an overview of Azure serverless services like Functions, Event Grid, Cosmos DB, and Logic Apps and a sample serverless application architecture diagram.
No Hassle NoSQL - Amazon DynamoDB & Amazon DocumentDB | AWS Summit Tel Aviv ...Amazon Web Services
NoSQL databases are a great fit for many modern applications such as mobile, web, and gaming that require flexible, scalable, high-performance, and highly functional databases to provide great user experiences but they can be hard to manage and require high proficiency and attention.In this session we will present Amazon DynamoDB, a fully managed, multi-region, multi-master database that provides consistent single-digit millisecond latency in any scale.
This document provides an overview and summary of a presentation on Amazon DynamoDB. The presentation will cover DynamoDB tables, APIs, data types, indexes, scaling, data modeling, scenarios and best practices. It will also discuss using DynamoDB Streams to enable cross-region replication and integration with other AWS services like S3, CloudSearch, ElastiCache and Redshift. The goal is to teach design patterns and best practices for building highly scalable applications with DynamoDB.
Building Real-time Travel Alerts
In this session, we will walk through how to build a complete streaming application to send alerts based on travel advisories from public data. We will also join in other data sources of relevance and push out alerts.
We will show you how to build this streaming application with Apache NiFi, Apache Kafka, and Apache Flink and show you when/why/how, and what to build to maximize performance, productivity, and ease of development.
Let's get streaming.
Apache Flink
Apache Kafka
Apache NiFi
FLaNK Stack
Tim Spann
Big Data Conference Europe 2023
Migrating Single-Tenant Applications to Multi-Tenant SaaS (ARC326-R1) - AWS r...Amazon Web Services
The appeal of SaaS has many ISVs interested in the power and value of delivering their solutions in a SaaS model. However, moving a single-tenant application to a multi-tenant environment can be daunting. In this session, we'll look at the obstacles that many ISVs face as they consider the move to a SaaS delivery model. We'll explore a wide range of transformation patterns that cover everything from lift-and-shift of your monolith to an incremental cutover to multi-tenant aware microservices, data, and infrastructure. Along the way, we'll highlight the challenges and technical considerations that shape your solution and allow you to better align your transformed solution with SaaS best practices. This includes looking at all the new architectural elements you'll need to add to your environment to support SaaS (onboarding, identity, billing, metering, analytics, and so on).
This document summarizes New Relic, a software performance monitoring tool. It provides an overview of New Relic's history and competitors. Key points covered include how New Relic supports various programming languages and platforms, its ease of setup and use, different feature sets across pricing tiers, and examples of customer spend. The document also briefly discusses New Relic's future opportunities around analytics, security, container monitoring, and strategic partnerships.
This document summarizes New Relic's key features including out-of-the-box monitoring of real user performance, server monitoring, and proactive notifications. It also discusses advanced features for deeper performance analytics and complementary tools for monitoring jobs. The presentation concludes with discussing New Relic's backlog including custom tiers, deployment notifications, and custom alerts.
The document discusses DevOps practices at Amazon Web Services (AWS). It begins with an overview of DevOps and how it has helped Amazon deploy code faster and more frequently. It then discusses specific DevOps tools and services offered by AWS, including AWS CodeCommit for source control, AWS CodeBuild for builds, AWS CodeDeploy for deployments, AWS CodePipeline for release orchestration, and AWS CodeStar for application development. The document explains how these services work together to enable continuous integration and continuous delivery workflows. It also discusses how AWS has implemented DevOps practices like infrastructure as code and monitoring within its own systems to deploy millions of times per day while maintaining quality, security and reliability.
This document provides an overview of Apache Kafka. It begins with defining Kafka as a distributed streaming platform and messaging system. It then lists the agenda which includes what Kafka is, why it is used, common use cases, major companies that use it, how it achieves high performance, and core concepts. Core concepts explained include topics, partitions, brokers, replication, leaders, and producers and consumers. The document also provides examples to illustrate these concepts.
Google Cloud Storage | Google Cloud Platform Tutorial | Google Cloud Architec...Edureka!
(Google Cloud Certification Training - Cloud Architect: https://www.edureka.co/google-cloud-architect-certification-training)
This tutorial on Google Cloud Storage will provide you with a detailed introduction to the various Cloud Storage Services provided by Google. You will also get hands-on on each of the storage options.
When dealing with critical Excel workbooks, you want to have proper source control in place. This presentation is about the different challenges we face when using Git with Excel files and how we can solve them by using the right mix of settings, extensions and Git workflow.
Event Grid is a fully managed event routing service in Azure that uses a publisher-subscriber model to route events from sources to subscribers. It handles routing and delivery of events from many sources and subscribers in a reliable, secure, and scalable way. Event Grid is designed for building reactive, event-driven applications and works well for serverless and microservices architectures. Events are delivered from sources through topics to event handlers. Event Grid offers reliable delivery, filtering, authentication, and is agnostic to language or platform.
Achieving a 50% Reduction in Cross-AZ Network Costs from Kafka (Uday Sagar Si...confluent
Cloud providers like AWS allow free data transfers within an Availability Zone (AZ), but bill users when data moves between AZs. When the data volume streamed through Kafka reaches big data scale, (e.g. numeric data points or user activity tracking), the costs incurred by cross-AZ traffic can add significantly to your monthly cloud spend. Since Kafka serves reads and writes only from leader partitions, for a topic with a replication factor of 3, a message sent through Kafka can cross AZs up to 4 times. Once when a producer produces a message onto broker in a different AZ, two times during Kafka replication, and once more during message consumption. With careful design, we can eliminate the first and last part of the cross AZ traffic. We can also use message compression strategies provided by Kafka to reduce costs during replication. In this talk, we will discuss the architectural choices that allow us to ensure a Kafka message is produced and consumed within a single AZ, as well as an algorithm that lets consumers intelligently subscribe to partitions with leaders in the same AZ. We will also cover use cases in which cross-AZ message streaming is unavoidable due to design limitations. Talk outline: 1) A review of Kafka replication, 2) Cross-AZ traffic implications, 3) Architectural choices for AZ-aware message streaming, 4) Algorithms for AZ-aware producers and consumers, 5) Results, 6) Limitations, 7) Takeaways.
Watch this talk here: https://www.confluent.io/online-talks/apache-kafka-architecture-and-fundamentals-explained-on-demand
This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput.
This talk provides a comprehensive overview of Kafka architecture and internal functions, including:
-Topics, partitions and segments
-The commit log and streams
-Brokers and broker replication
-Producer basics
-Consumers, consumer groups and offsets
This session is part 2 of 4 in our Fundamentals for Apache Kafka series.
AWS Glue is a serverless data integration service that allows users to discover, prepare, and transform data for analytics and machine learning. It provides a fully managed extract, transform, and load (ETL) service on AWS. AWS Glue crawls data sources, automatically extracts metadata and stores it in a centralized data catalog. It then executes ETL jobs developed by users to clean, enrich and move data between various data stores.
The document discusses challenges of deploying Kubernetes on-premise, including how load balancers are provisioned without cloud providers, using Nginx and Haproxy for load balancing on bare metal. It also covers how persistent volumes are provisioned with CSI drivers like Ember CSI to interface with storage backends, and tools for deploying and managing on-premise Kubernetes clusters like RKE.
This document discusses cloud-native applications and serverless computing. It begins with an introduction to cloud-native applications and core technologies like containers, orchestrators, and microservices. Examples are then given of how companies like Fujifilm and ASOS have benefited from serverless architectures on Azure. The document concludes with an overview of Azure serverless services like Functions, Event Grid, Cosmos DB, and Logic Apps and a sample serverless application architecture diagram.
No Hassle NoSQL - Amazon DynamoDB & Amazon DocumentDB | AWS Summit Tel Aviv ...Amazon Web Services
NoSQL databases are a great fit for many modern applications such as mobile, web, and gaming that require flexible, scalable, high-performance, and highly functional databases to provide great user experiences but they can be hard to manage and require high proficiency and attention.In this session we will present Amazon DynamoDB, a fully managed, multi-region, multi-master database that provides consistent single-digit millisecond latency in any scale.
This document provides an overview and summary of a presentation on Amazon DynamoDB. The presentation will cover DynamoDB tables, APIs, data types, indexes, scaling, data modeling, scenarios and best practices. It will also discuss using DynamoDB Streams to enable cross-region replication and integration with other AWS services like S3, CloudSearch, ElastiCache and Redshift. The goal is to teach design patterns and best practices for building highly scalable applications with DynamoDB.
Building Real-time Travel Alerts
In this session, we will walk through how to build a complete streaming application to send alerts based on travel advisories from public data. We will also join in other data sources of relevance and push out alerts.
We will show you how to build this streaming application with Apache NiFi, Apache Kafka, and Apache Flink and show you when/why/how, and what to build to maximize performance, productivity, and ease of development.
Let's get streaming.
Apache Flink
Apache Kafka
Apache NiFi
FLaNK Stack
Tim Spann
Big Data Conference Europe 2023
Migrating Single-Tenant Applications to Multi-Tenant SaaS (ARC326-R1) - AWS r...Amazon Web Services
The appeal of SaaS has many ISVs interested in the power and value of delivering their solutions in a SaaS model. However, moving a single-tenant application to a multi-tenant environment can be daunting. In this session, we'll look at the obstacles that many ISVs face as they consider the move to a SaaS delivery model. We'll explore a wide range of transformation patterns that cover everything from lift-and-shift of your monolith to an incremental cutover to multi-tenant aware microservices, data, and infrastructure. Along the way, we'll highlight the challenges and technical considerations that shape your solution and allow you to better align your transformed solution with SaaS best practices. This includes looking at all the new architectural elements you'll need to add to your environment to support SaaS (onboarding, identity, billing, metering, analytics, and so on).
This document summarizes New Relic, a software performance monitoring tool. It provides an overview of New Relic's history and competitors. Key points covered include how New Relic supports various programming languages and platforms, its ease of setup and use, different feature sets across pricing tiers, and examples of customer spend. The document also briefly discusses New Relic's future opportunities around analytics, security, container monitoring, and strategic partnerships.
This document summarizes New Relic's key features including out-of-the-box monitoring of real user performance, server monitoring, and proactive notifications. It also discusses advanced features for deeper performance analytics and complementary tools for monitoring jobs. The presentation concludes with discussing New Relic's backlog including custom tiers, deployment notifications, and custom alerts.
The document discusses DevOps practices at Amazon Web Services (AWS). It begins with an overview of DevOps and how it has helped Amazon deploy code faster and more frequently. It then discusses specific DevOps tools and services offered by AWS, including AWS CodeCommit for source control, AWS CodeBuild for builds, AWS CodeDeploy for deployments, AWS CodePipeline for release orchestration, and AWS CodeStar for application development. The document explains how these services work together to enable continuous integration and continuous delivery workflows. It also discusses how AWS has implemented DevOps practices like infrastructure as code and monitoring within its own systems to deploy millions of times per day while maintaining quality, security and reliability.
This document provides an overview of Apache Kafka. It begins with defining Kafka as a distributed streaming platform and messaging system. It then lists the agenda which includes what Kafka is, why it is used, common use cases, major companies that use it, how it achieves high performance, and core concepts. Core concepts explained include topics, partitions, brokers, replication, leaders, and producers and consumers. The document also provides examples to illustrate these concepts.
Google Cloud Storage | Google Cloud Platform Tutorial | Google Cloud Architec...Edureka!
(Google Cloud Certification Training - Cloud Architect: https://www.edureka.co/google-cloud-architect-certification-training)
This tutorial on Google Cloud Storage will provide you with a detailed introduction to the various Cloud Storage Services provided by Google. You will also get hands-on on each of the storage options.
When dealing with critical Excel workbooks, you want to have proper source control in place. This presentation is about the different challenges we face when using Git with Excel files and how we can solve them by using the right mix of settings, extensions and Git workflow.
Event Grid is a fully managed event routing service in Azure that uses a publisher-subscriber model to route events from sources to subscribers. It handles routing and delivery of events from many sources and subscribers in a reliable, secure, and scalable way. Event Grid is designed for building reactive, event-driven applications and works well for serverless and microservices architectures. Events are delivered from sources through topics to event handlers. Event Grid offers reliable delivery, filtering, authentication, and is agnostic to language or platform.
1. The document discusses the rise of "prosumption", where consumers take on roles in production through activities like user-generated content online.
2. Capitalists have difficulty controlling prosumers and face resistance from them. The exploitation of prosumers is ambiguous since some may benefit from their activities.
3. Web 2.0 companies provide mostly free services and make profits through advertising or premium upgrades rather than charging for content. This model focuses on abundance and effectiveness over efficiency.