Using Apache Kafka from Go
•
2 likes•551 views
At Uber, we use Kafka and Golang a lot. In this talk, I go over the basic concepts in Kafka and the libraries we use to publish and consume messages.
Report
Share
Report
Share
Download to read offline
Recommended
Scaling big with Apache Kafka
Apache Kafka sits at the core of the modern scalable event driven architecture. It’s no longer used only as logging infrastructure, but as a core component in thousands of companies around the world. It has the unique capability to provide low-latency, fault-tolerant pipeline at scale that is very important for today’s world of big data. During this talk we’ll see what makes Apache Kafka perfect for the job. We’ll explore how to optimize it for throughput or for durability. And we’ll also go over the messaging semantics it provides. Last but not least, we’ll see how Apache Kafka can help us solve some everyday problems that we face when we build large scale systems in an elegant way.
Large scale stream processing with Apache Flink
In today’s world it’s no longer enough to build systems that process big volumes of information. We now need applications that can handle large continuous streams of data with very low latency so we can react to the ever-changing environment around us. To efficiently handle such problems we need to deploy a stream processing solution. During the talk we’ll explore one of the most popular frameworks for stream processing – Apache Flink. We’ll see what unique capabilities it provides and how they apply to some real world problems. And we’ll also explore how it works under the hood and how to get the scalable and fault-tolerant stream processing that Flink provides.
ISTA 2019 - Migrating data-intensive microservices from Python to Go
In order for our systems to scale continuously and be resilient, they need to be constantly evolving. In this talk, I’m going to tell the store of how my team migrated a data-intensive microservice from Python to Go. First, we are going to start with the rationale behind the migration. Then we are going to go over the Python and Go tech stacks that we use. Last but not least, I’m also going to share our approach for migrating the service while running in production, adding new features and making sure there are no regressions.
Building a modern SaaS in 2020
Building a modern Software as a Service platform brings a lot of interesting engineering challenges. During this talk, I’m going to share my team’s journey of building a SaaS from scratch in 2020. First, we are going to start with the technologies and the architecture we picked. Then, we’ll go over the interesting challenge of implementing multitenancy. And we'll see how we benchmarked three different options and picked one. And last but not least, we’ll explore how every startup can use open source technologies to build observability infrastructure. And how to run their SaaS in production.
10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA
Kafka is the bedrock of Wix’s distributed Mega Microservices system.
Over the years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 mostly Scala microservices.
In this talk, you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system.
These Include:
* How to increase dev velocity of event-driven style code.
* How to optimize working with Kafka in polyglot setting
* How to migrate from request-reply to event-driven
* How to tackle multiple DCs environment.
Evolving big microservice architectures
Microservices are a well-established architecture applied by many organizations around the world to build scalable and fault-tolerant backend systems. But as these systems grow so does the number of services in them. And this brings many challenges when we want to introduce new functionality. For a simple feature, engineers may need to spend a lot of time designing the end to end flow, changing code in multiple services and setting up complex test scenarios. During this talk, we’ll explore how to evolve a microservice architecture to be easily extensible based on some lessons learned from running 5000 microservices in production. We’ll go over different architectural patterns and open source tools that we can use to make it easy for all engineers to understand, extend and be more and more productive in such big complex systems.
Exactly Once Delivery with Kafka - JOTB2020 Mini Session
In this talk I go over the basic theory of messaging in distributed systems, the different message delivery guarantees in Kafka and the to use them.
I focus on exactly once delivery guarantees and the way Kafka implements it with transaction based messaging protocol.
Including a discussion of the latency/throughput trade-offs, resource utilisation and its overall advantages and shortcomings.
Finally, I show a use-case at Wix where exactly once delivery helped us solve a big problem.
Devoxx fr 2016 - Apache Kafka - Stream Data Platform
Kafka est un système de messagerie distribué, en mode publish-subscribe, persistant les données qu'il reçoit, conçu pour facilement monter en charge et supporter des débits de données très importants.
Originellement développé chez LinkedIn, et maintenu au sein de la fondation Apache depuis 2012, son adoption n'a cessé de croitre pour en faire un quasi de-facto standard dans les pipelines de traitement de données.
Venez découvrir cet outil durant ce Hand's on de 3h où vous installerez un mini cluster Kafka et explorerez ses différentes API. En bonus, vous aurez la possibilité d'analyser vos données en temps réel avec Spark Streaming.
Recommended
Scaling big with Apache Kafka
Apache Kafka sits at the core of the modern scalable event driven architecture. It’s no longer used only as logging infrastructure, but as a core component in thousands of companies around the world. It has the unique capability to provide low-latency, fault-tolerant pipeline at scale that is very important for today’s world of big data. During this talk we’ll see what makes Apache Kafka perfect for the job. We’ll explore how to optimize it for throughput or for durability. And we’ll also go over the messaging semantics it provides. Last but not least, we’ll see how Apache Kafka can help us solve some everyday problems that we face when we build large scale systems in an elegant way.
Large scale stream processing with Apache Flink
In today’s world it’s no longer enough to build systems that process big volumes of information. We now need applications that can handle large continuous streams of data with very low latency so we can react to the ever-changing environment around us. To efficiently handle such problems we need to deploy a stream processing solution. During the talk we’ll explore one of the most popular frameworks for stream processing – Apache Flink. We’ll see what unique capabilities it provides and how they apply to some real world problems. And we’ll also explore how it works under the hood and how to get the scalable and fault-tolerant stream processing that Flink provides.
ISTA 2019 - Migrating data-intensive microservices from Python to Go
In order for our systems to scale continuously and be resilient, they need to be constantly evolving. In this talk, I’m going to tell the store of how my team migrated a data-intensive microservice from Python to Go. First, we are going to start with the rationale behind the migration. Then we are going to go over the Python and Go tech stacks that we use. Last but not least, I’m also going to share our approach for migrating the service while running in production, adding new features and making sure there are no regressions.
Building a modern SaaS in 2020
Building a modern Software as a Service platform brings a lot of interesting engineering challenges. During this talk, I’m going to share my team’s journey of building a SaaS from scratch in 2020. First, we are going to start with the technologies and the architecture we picked. Then, we’ll go over the interesting challenge of implementing multitenancy. And we'll see how we benchmarked three different options and picked one. And last but not least, we’ll explore how every startup can use open source technologies to build observability infrastructure. And how to run their SaaS in production.
10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA
Kafka is the bedrock of Wix’s distributed Mega Microservices system.
Over the years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 mostly Scala microservices.
In this talk, you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system.
These Include:
* How to increase dev velocity of event-driven style code.
* How to optimize working with Kafka in polyglot setting
* How to migrate from request-reply to event-driven
* How to tackle multiple DCs environment.
Evolving big microservice architectures
Microservices are a well-established architecture applied by many organizations around the world to build scalable and fault-tolerant backend systems. But as these systems grow so does the number of services in them. And this brings many challenges when we want to introduce new functionality. For a simple feature, engineers may need to spend a lot of time designing the end to end flow, changing code in multiple services and setting up complex test scenarios. During this talk, we’ll explore how to evolve a microservice architecture to be easily extensible based on some lessons learned from running 5000 microservices in production. We’ll go over different architectural patterns and open source tools that we can use to make it easy for all engineers to understand, extend and be more and more productive in such big complex systems.
Exactly Once Delivery with Kafka - JOTB2020 Mini Session
In this talk I go over the basic theory of messaging in distributed systems, the different message delivery guarantees in Kafka and the to use them.
I focus on exactly once delivery guarantees and the way Kafka implements it with transaction based messaging protocol.
Including a discussion of the latency/throughput trade-offs, resource utilisation and its overall advantages and shortcomings.
Finally, I show a use-case at Wix where exactly once delivery helped us solve a big problem.
Devoxx fr 2016 - Apache Kafka - Stream Data Platform
Kafka est un système de messagerie distribué, en mode publish-subscribe, persistant les données qu'il reçoit, conçu pour facilement monter en charge et supporter des débits de données très importants.
Originellement développé chez LinkedIn, et maintenu au sein de la fondation Apache depuis 2012, son adoption n'a cessé de croitre pour en faire un quasi de-facto standard dans les pipelines de traitement de données.
Venez découvrir cet outil durant ce Hand's on de 3h où vous installerez un mini cluster Kafka et explorerez ses différentes API. En bonus, vous aurez la possibilité d'analyser vos données en temps réel avec Spark Streaming.
ApacheCon Berlin 2019: Kongo:Building a Scalable Streaming IoT Application us...
Join with me in a journey of exploration upriver with "Kongo", a scalable streaming IoT logistics demonstration application using Apache Kafka, the popular open source distributed streaming platform. Along the way you'll discover: an example logistics IoT problem domain (involving the rapid movement of thousands of goods by trucks between warehouses, with real-time checking of complex business and safety rules from sensor data); an overview of the Apache Kafka architecture and components; lessons learned from making critical Kaka application design decisions; an example of Kafka Streams for checking truck load limits; and finish the journey by overcoming final performance challenges and shooting the rapids to scale Kongo on a production Kafka cluster.
https://aceu19.apachecon.com/session/kongo-building-scalable-streaming-iot-application-using-apache-kafka
Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...
At Wix, we have created a universal event-driven programming infrastructure on top of the Kafka message broker.
This infra makes sure messages are eventually successfully consumed and produced no matter what failure it encounters.
In this talk, you will learn about the features we introduced in order to make sure our distributed system can safely handle an ever growing message throughput in a fault tolerant manner.
You will be introduced to such techniques as retry topics, local persistent queues, and cooperative fibers that help make your flows more resilient and performant.
You will also learn how to make this infra work for all programming languages tech stacks with optimal resource manage using the power of Kubernetes and gRPC.
When to use a client library, and when to deploy an external pod (DaemonSet, StatefulSet) or even deploy a sidecar.
Codeless pipelines with pulsar and flink
Codeless pipelines with pulsar and flink
datacon la
apache pulsar, apache flink, apache nifi
streaming data, iot, events, rest, go, python, java
apache bookkeeper json
Securing Your Containerized Applications with NGINX
Kevin Jones, NGNIX -
NGINX is one of the most popular images on Docker Hub and has been at the forefront of the web since the early 2000's. In this talk we will discuss how and why NGINX's lightweight and powerful architecture makes it a very popular choice for securing containerized applications as a sidecar reverse proxy within containers. We will highlight important aspects of application security that NGINX can help with, such as TLS, HTTP, AuthN, AuthZ and traffic control.
Cilium – Kernel Native Security & DDOS Mitigation for Microservices with BPF
We have introduced Cilium at DockerCon US 2017 this year. Cilium provides application-aware network connectivity, security, and load-balancing for containers. This talk will follow up on the introduction and deep dive into recent kernel developments that address two fundamental questions: How can I provide application-aware security and routing efficiently without overhead embedded into every service? How can container hosts protect themselves from internal and external DDoS attacks? The solutions include:
kproxy: a kernel-based socket proxy which allows for application-aware routing and security enforcement with minimal overhead.
XDP: A lightning-fast packet processing datapath using BPF. The technology is intended for DDoS mitigation, load-balancing, and forwarding.
This talk will deep dive into these exciting technologies and show how Cilium makes BPF and these kernel features available on Linux for your Docker containers.
Apache kafka
Apresentação sobre os conceitos básicos envolvendo o sistema de stream distribuído apache kafka.
Practical tips and tricks for Apache Kafka messages integration | Francesco T...
Interacting with Apache Kafka seems straightforward at first, you “just” push and pull messages. Yet it can quickly become a source of frustration as the user encounters timeouts, vague error descriptions and disappearing messages. Experience helps a lot and I’m here to share what I know.
In this talk you will learn the tips & tricks I wish I had known at the beginning of my Apache Kafka journey. We’ll discuss topics like producer acknowledgments, server and consumer parameters (auto_offset_reset anyone?) that are commonly overlooked causing lots of developer’s pain. I’ll share with you how to generate code that works as expected on the first run, making your first integration painless. These tips will kickstart your Apache Kafka experience in Python and save you hours of debugging.
Introduction To Streaming Data and Stream Processing with Apache Kafka
Modern businesses have data at their core, and this data is changing continuously. How can we harness this torrent of continuously changing data in real time? The answer is stream processing, and one system that has become a core hub for streaming data is Apache Kafka.
This presentation will give a brief introduction to Apache Kafka and describe its usage as a platform for streaming data. It will explain how Kafka serves as a foundation for both streaming data pipelines and applications that consume and process real-time data streams. It will introduce some of the newer components of Kafka that help make this possible, including Kafka Connect, a framework for capturing continuous data streams, and Kafka Streams, a lightweight stream processing library.
This is talk 1 out of 6 from the Kafka Talk Series.
http://www.confluent.io/apache-kafka-talk-series/introduction-to-stream-processing-with-apache-kafka
Connect at Twitter-scale | Jordan Bull and Ryanne Dolan, Twitter
Twitter has one of the largest Kafka fleets in the world, handling hundreds of millions of events per second. In order to operate Kafka Connect at this scale, we've had to get creative. In this talk we'll present some of the problems we've run into with Kafka Connect, and how we've engineered around them.
A Kong retrospective: from 0.10 to 0.13
A short introductory talk given as part of the April 2018 Kong meetup "Introducing Kubernetes Ingress Controller for Kong".
This talk covers the new features and improvements made to Kong from 2017 to 2018, including the groundwork conducted by Kong Inc. and open source contributors that allowed for the development of the Kong Ingress Controller for Kubernetes.
The Kong Ingress Controller for Kubernetes was then announced during the meetup:
https://github.com/Kong/kubernetes-ingress-controller
ApacheCon 2021 Apache Deep Learning 302
ApacheCon 2021 Apache Deep Learning 302
Tuesday 18:00 UTC
Apache Deep Learning 302
Timothy Spann
This talk will discuss and show examples of using Apache Hadoop, Apache Kudu, Apache Flink, Apache Hive, Apache MXNet, Apache OpenNLP, Apache NiFi and Apache Spark for deep learning applications. This is the follow up to previous talks on Apache Deep Learning 101 and 201 and 301 at ApacheCon, Dataworks Summit, Strata and other events. As part of this talk, the presenter will walk through using Apache MXNet Pre-Built Models, integrating new open source Deep Learning libraries with Python and Java, as well as running real-time AI streams from edge devices to servers utilizing Apache NiFi and Apache NiFi - MiNiFi. This talk is geared towards Data Engineers interested in the basics of architecting Deep Learning pipelines with open source Apache tools in a Big Data environment. The presenter will also walk through source code examples available in github and run the code live on Apache NiFi and Apache Flink clusters.
Tim Spann is a Developer Advocate @ StreamNative where he works with Apache NiFi, Apache Pulsar, Apache Flink, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal Field Engineer at Cloudera, a senior solutions architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.
* https://github.com/tspannhw/ApacheDeepLearning302/
* https://github.com/tspannhw/nifi-djl-processor
* https://github.com/tspannhw/nifi-djlsentimentanalysis-processor
* https://github.com/tspannhw/nifi-djlqa-processor
* https://www.linkedin.com/pulse/2021-schedule-tim-spann/
Using the flipn stack for edge ai (flink, nifi, pulsar)
Pulsar Summit Asia 2021/2022
Using The FLiPN Stack for EdgeAI (Flink, Pulsar, NiFi)
Tim Spann
Exactly Once Delivery with Kafka - Kafka Tel-Aviv Meetup
In this talk I go over the basic theory of messaging in distributed systems, the different message delivery guarantees in Kafka and the to use them.
I focus on exactly once delivery guarantees and the way Kafka implements it with transaction based messaging protocol.
Including a discussion of the latency/throughput trade-offs, resource utilisation and its overall advantages and shortcomings.
Finally, I show a use-case at Wix where exactly once delivery helped us solve a big problem.
Kafka on Pulsar
In Apache Pulsar Meetup, Jia Zhai from StreamNative presents KoP (Kafka-on-Pulsar) which bring native Kafka protocol support on Pulsar broker. He gave a demo about how to use Kafka clients and Pulsar clients can work seamlessly on same data, and how Kafka Connectors can work on a Pulsar cluster.
Using FLiP with influxdb for edgeai iot at scale 2022
https://adtmag.com/webcasts/2021/12/influxdata-february-10.aspx?tc=page0
FLiP Stack (Apache Flink, Apache Pulsar, Apache NiFi, Apache Spark) with Influx DB for Edge AI and IoT workloads at scale
Tim Spann
Developer Advocate
StreamNative
datainmotion.dev
Robust Stream Processing with Apache Flink
Slides from my OSCON 2016 talk. Not super useful without the accompanying video and code. I'll post that when it's available.
ApacheCon 2021 - Apache NiFi Deep Dive 300
21-September-2021 - ApacheCon - Tuesday 17:10 UTC Apache NIFi Deep Dive 300
* https://github.com/tspannhw/EverythingApacheNiFi
* https://github.com/tspannhw/FLiP-ApacheCon2021
* https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html
* https://github.com/tspannhw/FLiP-IoT
* https://github.com/tspannhw/FLiP-Energy
* https://github.com/tspannhw/FLiP-SOLR
* https://github.com/tspannhw/FLiP-EdgeAI
* https://github.com/tspannhw/FLiP-CloudQueries
* https://github.com/tspannhw/FLiP-Jetson
* https://www.linkedin.com/pulse/2021-schedule-tim-spann/
Tuesday 17:10 UTC
Apache NIFi Deep Dive 300
Timothy Spann
For Data Engineers who have flows already in production, I will dive deep into best practices, advanced use cases, performance optimizations, tips, tricks, edge cases, and interesting examples. This is a master class for those looking to learn quickly things I have picked up after years in the field with Apache NiFi in production.
This will be interactive and I encourage questions and discussions.
You will take away examples and tips in slides, github, and articles.
This talk will cover:
Load Balancing
Parameters and Parameter Contexts
Stateless vs Stateful NiFi
Reporting Tasks
NiFi CLI
NiFi REST Interface
DevOps
Advanced Record Processing
Schemas
RetryFlowFile
Lookup Services
RecordPath
Expression Language
Advanced Error Handling Techniques
Tim Spann is a Developer Advocate @ StreamNative where he works with Apache NiFi, Apache Pulsar, Apache Flink, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal Field Engineer at Cloudera, a senior solutions architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.
More Related Content
What's hot
ApacheCon Berlin 2019: Kongo:Building a Scalable Streaming IoT Application us...
Join with me in a journey of exploration upriver with "Kongo", a scalable streaming IoT logistics demonstration application using Apache Kafka, the popular open source distributed streaming platform. Along the way you'll discover: an example logistics IoT problem domain (involving the rapid movement of thousands of goods by trucks between warehouses, with real-time checking of complex business and safety rules from sensor data); an overview of the Apache Kafka architecture and components; lessons learned from making critical Kaka application design decisions; an example of Kafka Streams for checking truck load limits; and finish the journey by overcoming final performance challenges and shooting the rapids to scale Kongo on a production Kafka cluster.
https://aceu19.apachecon.com/session/kongo-building-scalable-streaming-iot-application-using-apache-kafka
Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...
At Wix, we have created a universal event-driven programming infrastructure on top of the Kafka message broker.
This infra makes sure messages are eventually successfully consumed and produced no matter what failure it encounters.
In this talk, you will learn about the features we introduced in order to make sure our distributed system can safely handle an ever growing message throughput in a fault tolerant manner.
You will be introduced to such techniques as retry topics, local persistent queues, and cooperative fibers that help make your flows more resilient and performant.
You will also learn how to make this infra work for all programming languages tech stacks with optimal resource manage using the power of Kubernetes and gRPC.
When to use a client library, and when to deploy an external pod (DaemonSet, StatefulSet) or even deploy a sidecar.
Codeless pipelines with pulsar and flink
Codeless pipelines with pulsar and flink
datacon la
apache pulsar, apache flink, apache nifi
streaming data, iot, events, rest, go, python, java
apache bookkeeper json
Securing Your Containerized Applications with NGINX
Kevin Jones, NGNIX -
NGINX is one of the most popular images on Docker Hub and has been at the forefront of the web since the early 2000's. In this talk we will discuss how and why NGINX's lightweight and powerful architecture makes it a very popular choice for securing containerized applications as a sidecar reverse proxy within containers. We will highlight important aspects of application security that NGINX can help with, such as TLS, HTTP, AuthN, AuthZ and traffic control.
Cilium – Kernel Native Security & DDOS Mitigation for Microservices with BPF
We have introduced Cilium at DockerCon US 2017 this year. Cilium provides application-aware network connectivity, security, and load-balancing for containers. This talk will follow up on the introduction and deep dive into recent kernel developments that address two fundamental questions: How can I provide application-aware security and routing efficiently without overhead embedded into every service? How can container hosts protect themselves from internal and external DDoS attacks? The solutions include:
kproxy: a kernel-based socket proxy which allows for application-aware routing and security enforcement with minimal overhead.
XDP: A lightning-fast packet processing datapath using BPF. The technology is intended for DDoS mitigation, load-balancing, and forwarding.
This talk will deep dive into these exciting technologies and show how Cilium makes BPF and these kernel features available on Linux for your Docker containers.
Apache kafka
Apresentação sobre os conceitos básicos envolvendo o sistema de stream distribuído apache kafka.
Practical tips and tricks for Apache Kafka messages integration | Francesco T...
Interacting with Apache Kafka seems straightforward at first, you “just” push and pull messages. Yet it can quickly become a source of frustration as the user encounters timeouts, vague error descriptions and disappearing messages. Experience helps a lot and I’m here to share what I know.
In this talk you will learn the tips & tricks I wish I had known at the beginning of my Apache Kafka journey. We’ll discuss topics like producer acknowledgments, server and consumer parameters (auto_offset_reset anyone?) that are commonly overlooked causing lots of developer’s pain. I’ll share with you how to generate code that works as expected on the first run, making your first integration painless. These tips will kickstart your Apache Kafka experience in Python and save you hours of debugging.
Introduction To Streaming Data and Stream Processing with Apache Kafka
Modern businesses have data at their core, and this data is changing continuously. How can we harness this torrent of continuously changing data in real time? The answer is stream processing, and one system that has become a core hub for streaming data is Apache Kafka.
This presentation will give a brief introduction to Apache Kafka and describe its usage as a platform for streaming data. It will explain how Kafka serves as a foundation for both streaming data pipelines and applications that consume and process real-time data streams. It will introduce some of the newer components of Kafka that help make this possible, including Kafka Connect, a framework for capturing continuous data streams, and Kafka Streams, a lightweight stream processing library.
This is talk 1 out of 6 from the Kafka Talk Series.
http://www.confluent.io/apache-kafka-talk-series/introduction-to-stream-processing-with-apache-kafka
Connect at Twitter-scale | Jordan Bull and Ryanne Dolan, Twitter
Twitter has one of the largest Kafka fleets in the world, handling hundreds of millions of events per second. In order to operate Kafka Connect at this scale, we've had to get creative. In this talk we'll present some of the problems we've run into with Kafka Connect, and how we've engineered around them.
A Kong retrospective: from 0.10 to 0.13
A short introductory talk given as part of the April 2018 Kong meetup "Introducing Kubernetes Ingress Controller for Kong".
This talk covers the new features and improvements made to Kong from 2017 to 2018, including the groundwork conducted by Kong Inc. and open source contributors that allowed for the development of the Kong Ingress Controller for Kubernetes.
The Kong Ingress Controller for Kubernetes was then announced during the meetup:
https://github.com/Kong/kubernetes-ingress-controller
ApacheCon 2021 Apache Deep Learning 302
ApacheCon 2021 Apache Deep Learning 302
Tuesday 18:00 UTC
Apache Deep Learning 302
Timothy Spann
This talk will discuss and show examples of using Apache Hadoop, Apache Kudu, Apache Flink, Apache Hive, Apache MXNet, Apache OpenNLP, Apache NiFi and Apache Spark for deep learning applications. This is the follow up to previous talks on Apache Deep Learning 101 and 201 and 301 at ApacheCon, Dataworks Summit, Strata and other events. As part of this talk, the presenter will walk through using Apache MXNet Pre-Built Models, integrating new open source Deep Learning libraries with Python and Java, as well as running real-time AI streams from edge devices to servers utilizing Apache NiFi and Apache NiFi - MiNiFi. This talk is geared towards Data Engineers interested in the basics of architecting Deep Learning pipelines with open source Apache tools in a Big Data environment. The presenter will also walk through source code examples available in github and run the code live on Apache NiFi and Apache Flink clusters.
Tim Spann is a Developer Advocate @ StreamNative where he works with Apache NiFi, Apache Pulsar, Apache Flink, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal Field Engineer at Cloudera, a senior solutions architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.
* https://github.com/tspannhw/ApacheDeepLearning302/
* https://github.com/tspannhw/nifi-djl-processor
* https://github.com/tspannhw/nifi-djlsentimentanalysis-processor
* https://github.com/tspannhw/nifi-djlqa-processor
* https://www.linkedin.com/pulse/2021-schedule-tim-spann/
Using the flipn stack for edge ai (flink, nifi, pulsar)
Pulsar Summit Asia 2021/2022
Using The FLiPN Stack for EdgeAI (Flink, Pulsar, NiFi)
Tim Spann
Exactly Once Delivery with Kafka - Kafka Tel-Aviv Meetup
In this talk I go over the basic theory of messaging in distributed systems, the different message delivery guarantees in Kafka and the to use them.
I focus on exactly once delivery guarantees and the way Kafka implements it with transaction based messaging protocol.
Including a discussion of the latency/throughput trade-offs, resource utilisation and its overall advantages and shortcomings.
Finally, I show a use-case at Wix where exactly once delivery helped us solve a big problem.
Kafka on Pulsar
In Apache Pulsar Meetup, Jia Zhai from StreamNative presents KoP (Kafka-on-Pulsar) which bring native Kafka protocol support on Pulsar broker. He gave a demo about how to use Kafka clients and Pulsar clients can work seamlessly on same data, and how Kafka Connectors can work on a Pulsar cluster.
Using FLiP with influxdb for edgeai iot at scale 2022
https://adtmag.com/webcasts/2021/12/influxdata-february-10.aspx?tc=page0
FLiP Stack (Apache Flink, Apache Pulsar, Apache NiFi, Apache Spark) with Influx DB for Edge AI and IoT workloads at scale
Tim Spann
Developer Advocate
StreamNative
datainmotion.dev
Robust Stream Processing with Apache Flink
Slides from my OSCON 2016 talk. Not super useful without the accompanying video and code. I'll post that when it's available.
ApacheCon 2021 - Apache NiFi Deep Dive 300
21-September-2021 - ApacheCon - Tuesday 17:10 UTC Apache NIFi Deep Dive 300
* https://github.com/tspannhw/EverythingApacheNiFi
* https://github.com/tspannhw/FLiP-ApacheCon2021
* https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html
* https://github.com/tspannhw/FLiP-IoT
* https://github.com/tspannhw/FLiP-Energy
* https://github.com/tspannhw/FLiP-SOLR
* https://github.com/tspannhw/FLiP-EdgeAI
* https://github.com/tspannhw/FLiP-CloudQueries
* https://github.com/tspannhw/FLiP-Jetson
* https://www.linkedin.com/pulse/2021-schedule-tim-spann/
Tuesday 17:10 UTC
Apache NIFi Deep Dive 300
Timothy Spann
For Data Engineers who have flows already in production, I will dive deep into best practices, advanced use cases, performance optimizations, tips, tricks, edge cases, and interesting examples. This is a master class for those looking to learn quickly things I have picked up after years in the field with Apache NiFi in production.
This will be interactive and I encourage questions and discussions.
You will take away examples and tips in slides, github, and articles.
This talk will cover:
Load Balancing
Parameters and Parameter Contexts
Stateless vs Stateful NiFi
Reporting Tasks
NiFi CLI
NiFi REST Interface
DevOps
Advanced Record Processing
Schemas
RetryFlowFile
Lookup Services
RecordPath
Expression Language
Advanced Error Handling Techniques
Tim Spann is a Developer Advocate @ StreamNative where he works with Apache NiFi, Apache Pulsar, Apache Flink, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal Field Engineer at Cloudera, a senior solutions architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.
What's hot (20)
ApacheCon Berlin 2019: Kongo:Building a Scalable Streaming IoT Application us...
ApacheCon Berlin 2019: Kongo:Building a Scalable Streaming IoT Application us...
Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...
Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...
Securing Your Containerized Applications with NGINX
Securing Your Containerized Applications with NGINX
Cilium – Kernel Native Security & DDOS Mitigation for Microservices with BPF
Cilium – Kernel Native Security & DDOS Mitigation for Microservices with BPF
Practical tips and tricks for Apache Kafka messages integration | Francesco T...
Practical tips and tricks for Apache Kafka messages integration | Francesco T...
Introduction To Streaming Data and Stream Processing with Apache Kafka
Introduction To Streaming Data and Stream Processing with Apache Kafka
Connect at Twitter-scale | Jordan Bull and Ryanne Dolan, Twitter
Connect at Twitter-scale | Jordan Bull and Ryanne Dolan, Twitter
Using the flipn stack for edge ai (flink, nifi, pulsar)
Using the flipn stack for edge ai (flink, nifi, pulsar)
Exactly Once Delivery with Kafka - Kafka Tel-Aviv Meetup
Exactly Once Delivery with Kafka - Kafka Tel-Aviv Meetup
Using FLiP with influxdb for edgeai iot at scale 2022
Using FLiP with influxdb for edgeai iot at scale 2022
Similar to Using Apache Kafka from Go
Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning
Apache Kafka for 010php Meetup
Apache Kafka - Scalable Message-Processing and more !
Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. How can me make sure that all these event are accepted and forwarded in an efficient and reliable way? This is where Apache Kafaka comes into play, a distirbuted, highly-scalable messaging broker, build for exchanging huge amount of messages between a source and a target.
This session will start with an introduction into Apache and presents the role of Apache Kafka in a modern data / information architecture and the advantages it brings to the table. Additionally the Kafka ecosystem will be covered as well as the integration of Kafka in the Oracle Stack, with products such as Golden Gate, Service Bus and Oracle Stream Analytics all being able to act as a Kafka consumer or producer.
[March sn meetup] apache pulsar + apache nifi for cloud data lake
https://www.meetup.com/new-york-city-apache-pulsar-meetup/events/283837865/
Learn how to use Apache Pulsar and Apache NiFi to Stream to your Data Lake
Discover how to stream data to and from your data lake or data mart using Apache Pulsar™ and Apache NiFi®. Learn how these cloud-native, scalable open-source projects built for streaming data pipelines work together to enable you to quickly build applications with minimal coding.
|WHAT THE SESSION WILL COVER|
Best Practices for using Pulsar and NiFi
A deep dive on Apache NiFi's Pulsar connector and demos
Building an End-to-End Application in the Hybrid Cloud
Attend for a chance to win a We <3 Pulsar t-shirt! The first 50 registrants who register through here [https://hubs.ly/Q013LTpn0] will be entered in a drawing!
—------------------------
|AGENDA|
6:00 - 7:00 PM EST: Presentation - Tim Spann, StreamNative Developer Advocate
7:00 - 8:00 PM EST: Presentation - John Kuchmek, Cloudera Principal Solutions Engineer
8:00 - 8:30 PM EST: Q&A + Networking
—------------------------
|ABOUT THE SPEAKERS|
John Kuchmek is a Principal Solutions Engineer for Cloudera. Before joining Cloudera, John transitioned to the Autonomous Intelligence team where he was in charge of integrating the platforms to allow data scientists to work with various types of data.
Tim Spann is a Developer Advocate for StreamNative. He works with StreamNative Cloud, Apache Pulsar™, Apache Flink®, Flink® SQL, Big Data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science. He is currently working on a book about the FLiP Stack.
Apache Kafka from 0.7 to 1.0, History and Lesson Learned
What is Kafka, really? Why did we build it? What have been evolved inside Kafka in the past 7 years and and what lessons have we learned from it?
Getting Started with Kafka on k8s
Presented by Rohini Rajaram at the Fedex Cloud-Native Conference in Pittsburgh on July 12th, 2019.
Timothy Spann: Apache Pulsar for ML
Timothy Spann: Apache Pulsar for ML
Data Science Online Camp 2023 Winter
Website: https://dscamp.org
Youtube: https://www.youtube.com/channel/UCeHtPZ_ZLZ-nHFMUCXY81RQ
FB: https://www.facebook.com/people/Data-Science-Camp/100064240830422/
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...
At Wix, we have created a universal event-driven programming infrastructure on top of the Kafka message broker.
This infra makes sure messages are eventually successfully consumed and produced no matter what failure it encounters.
In this talk, you will learn about the features we introduced in order to make sure our distributed system can safely handle an ever growing message throughput in a fault tolerant manner.
You will be introduced to such techniques as retry topics, local persistent queues, and cooperative fibers that help make your flows more resilient and performant.
You will also learn how to make this infra work for all programming languages tech stacks with optimal resource manage using the power of Kubernetes and gRPC.
When to use a client library, and when to deploy an external pod (DaemonSet) or even deploy a sidecar.
Data Pipelines with Apache Kafka
Introduction to Kafka and some of the systems you can build with it.
The Evolution of Trillion-level Real-time Messaging System in BIGO - Puslar ...
BIGO currently has two major video products and services, Live and Likee. At present, BIGO Live's live broadcast business has covered more than 150 countries and regions and Likee short video also has more than 100 million users. Being well received and approved among young people, our products are popular all around the world.
In the past technical architecture, BIGO adopted open-source Kafka as a basic service to support real-time data processing and analyzing. And based on this, BIGO has built a complete recommendation infrastructure to provide users with high-quality recommendation services. However, as the business continues to develop rapidly, the scale of our processing messages has entered a trillion scale and the past architecture has encountered huge challenges.
Regarding these problems, we got in touch with Apache Pulsar. With the gradual deepening of Pulsar, we have benefited a lot from these excellent features brought by Apache Pulsar such as hierarchical persistent storage, low E2E latency and horizontal scalability. These advantages have also helped us overcome many difficulties in the production system.
This talk will introduce the development and construction of BIGO's messaging platform based on Pulsar. We will start with message processing, storage, etc., and deeply analyze the performance bottlenecks that may be encountered in the production environment, and share our production processing practice experience.
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Apache Kafka is a new breed of messaging system built for the "big data" world. Coming out of LinkedIn (and donated to Apache), it is a distributed pub/sub system built in Scala. It has been an Apache TLP now for several months with the first Apache release imminent. Built for speed, scalability, and robustness, Kafka should definitely be one of the data tools you consider when designing distributed data-oriented applications.
The talk will cover a general overview of the project and technology, with some use cases, and a demo.
JConf.dev 2022 - Apache Pulsar Development 101 with Java
JConf.dev 2022 - Apache Pulsar Development 101 with Java
https://2022.jconf.dev/
In this session I will get you started with real-time cloud native streaming programming with Java. We will start off with a gentle introduction to Apache Pulsar and setting up your first easy standalone cluster. We will then l show you how to produce and consume message to Pulsar using several different Java libraries including native Java client, AMQP/RabbitMQ, MQTT and even Kafka. After this session you will building real-time streaming and messaging applications with Java. We will also touch on Apache Spark and Apache Flink.
Timothy Spann
Tim Spann is a Developer Advocate @ StreamNative where he works with Apache Pulsar, Apache Flink, Apache NiFi, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science. https://www.datainmotion.dev/p/about-me.html https://dzone.com/users/297029/bunkertor.html https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/speaker/185963
Apache Kafka - Scalable Message-Processing and more !
Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. How can me make sure that all these event are accepted and forwarded in an efficient and reliable way? This is where Apache Kafaka comes into play, a distirbuted, highly-scalable messaging broker, build for exchanging huge amount of messages between a source and a target.
This session will start with an introduction into Apache and presents the role of Apache Kafka in a modern data / information architecture and the advantages it brings to the table. Additionally the Kafka ecosystem will be covered as well as the integration of Kafka in the Oracle Stack, with products such as Golden Gate, Service Bus and Oracle Stream Analytics all being able to act as a Kafka consumer or producer.
Similar to Using Apache Kafka from Go (20)
Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning
Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning
Apache Kafka - Scalable Message-Processing and more !
Apache Kafka - Scalable Message-Processing and more !
[March sn meetup] apache pulsar + apache nifi for cloud data lake
[March sn meetup] apache pulsar + apache nifi for cloud data lake
Apache Kafka from 0.7 to 1.0, History and Lesson Learned
Apache Kafka from 0.7 to 1.0, History and Lesson Learned
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...
The Evolution of Trillion-level Real-time Messaging System in BIGO - Puslar ...
The Evolution of Trillion-level Real-time Messaging System in BIGO - Puslar ...
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
JConf.dev 2022 - Apache Pulsar Development 101 with Java
JConf.dev 2022 - Apache Pulsar Development 101 with Java
Apache Kafka - Scalable Message-Processing and more !
Apache Kafka - Scalable Message-Processing and more !
More from Nikolay Stoitsev
Building vs Buying Software
As engineers, we like to solve problems by building solutions from scratch. Even though on some occasions it’s better to buy and integrate existing software. But how come? Are engineers who don’t always deliver from scratch real engineers? The goal of this talk is to answer all important questions about making build vs buy decisions. We’ll see how to define a clear strategy for making such decisions. And we’ll explore how to select and integrate existing software efficiently. Even if your company doesn’t have the habit of doing it.
How and why to manage your manager
This talk will help you build a better partnership with your eng manager in 3 ways. First, you’ll understand what your manager does all day. Second, you’ll learn how to manage up. And third, you’ll see that there isn’t a single person that can manage you perfectly and what you can do about it.
A practical introduction to observability
Good observability is essential for modern software. It gives us confidence that our systems are working properly. And it also allows us to debug issues efficiently. In this talk, we’ll explore everything you need to know to start applying good observability to your projects. And we’ll see the most common pitfalls you need to be aware of. We will start with the tools and basic concepts in monitoring. And we’ll go over the 3 most common mistakes people make with it. Then we’ll see how to have automatic alerts to detect issues. And, we’ll touch on the principles for setting up good alerts. As a final step, we’ll see how to build our logging system and how to apply it in the most efficient way to debug issues easily.
Everything You Need to Know About NewSQL in 2020
The database is usually the heart of a software system. And there are many database technologies that we can pick from. In this talk, we’ll explore where RDBMS and NoSQL fall short and how NewSQL fills the gap. We’ll see what types of NewSQL databases exist and how they work. And we’ll also go over different NewSQL solutions that we can pick for our projects. By the end of the talk, we’ll have a good understanding of when and how to apply a NewSQL database in our big scale applications.
3 lessons on effective communication for engineers
Effective communication is one of the most important skills we need. It greatly improves our productivity. And multiplies the positive impact that we have on the products we build and the people we work with. In this talk, we are going to explore three lessons on better communication. First, we’ll start with key principles for building trust and good relationships with the people around us. Then, we’ll see why and how to manage expectations. And we’ll explore how requirements facilitation can make our work easier. We are also going to see how to apply code reviews to our communication and scale it to amplify our impact. And most importantly, we’ll go over some real-world examples of how to apply these lessons in our everyday work to become better engineers.
The career path of software engineers and how to navigate it
During the talk, we explore the career path of a software engineer, what are the expectations at each level and how to acquire new skills to move between levels. We go over a way to decide on switching to management or staying on the individual contributor track. And we explore three strategies for continuous improvements.
Migrating a data intensive microservice from Python to Go
As Uber is hyper-growing as a company so does our need for scalable and resilient systems. In this talk, I’m going to tell the story of how my team migrated from Python to Go, a microservice that processes millions of events every day. First, we are going to start with the rationale behind the migration. Then we are going to go over the Python and Go tech stacks that we use. Last but not least, I’m also going to share our approach for migrating the service while running in production, adding new features and making sure there are no regressions.
NewSQL: what, when and how
The database has always been one of the key components in every architecture. There is a great variety of tradeoffs we should consider and implementation that we can pick from. If we need consistency and correctness in exchange of availability and performance, we should pick a relational database. If we need scale and increased availability by sacrificing transactional and consistency guarantees, we should use a NoSQL database. And if we need both horizontal scalability and transactions, we need to pick a NewSQL database. During this talk we’ll explore what guarantees a NewSQL system provides. We’ll go over the different approaches in building such a system. And we’ll see some open source projects that implements each approach. At the end of the talk we’ll have a good understanding of when and how to apply a NewSQL database in our big scale applications.
How to read the v8 source code?
JavaScript runs on many platforms and in different environments. We often forget about the low level infrastructure that makes it work. During this talk we are going to explore how to read the source code of v8. We'll see how to dig into different features of the language and understand how they work. This can give us a very deep understanding and make us a better JavaScript developers.
Running in multiple data centers
The applications that we build for today's world have a lot of requirements. They need to provide the best user experience and to be always up and running. To achieve this in a massive scale you need a multi data center architecture. When we have more than one data centers, even if one of them goes down, the other can handle the traffic and your users will continue to use your application uninterrupted. Also by having datacenters in different locations around the world you can you take advantage of lower latencies and provide a better usability. But to take advantage of all those benefits you need to architect your application in a special way. During the talk we’ll explore the different multi data center configurations and the tradeoff of each one of them. We’ll also go over the ways to do failover and some useful processes to facilitate it better. Moreover, we’ll see how each layer of the application is affected by such architecture, all the way down to the database and the data model. Finally, I’ll share what technologies help Uber to run in multiple data centers and the lessons we learned by doing so.
Distributed tracing for big systems
As our systems grow, so does our software architecture complexity. As we scale and add more and more components, the interactions occurring between them become very complex and we start to lose visibility into the system. Traditional monitoring tools such as metrics and distributed logging still have their place, but they often fail to provide visibility across services. This is where distributed tracing thrives. During the talk we’ll explore what distributed tracing is, what open tools we can use to facilitate it and some lessons learned while implementing distributed tracing at Uber and how it helps us build big and impactful systems.
Reusable patterns for scalable APIs running on Docker @ Java2Days
The shipping containers were introduced around 1830s and since then they play a crucial role in the modern society by providing efficient packaging, storage and transportation. Today we see the same revolution happening in the DevOps world with the adoption of container technologies like Docker. They allow us to package, distribute and run software in a scalable and efficient way. In this talk we’ll see how we can abstract the common problem we solve everyday when building scalable Java APIs with Docker into design patterns to create reusable solutions. We’ll explore the good practices of writing Java applications with Docker. Then we’ll see how each design pattern can be applied in real scenarios that address different concerns that a large system creates. We’ll see some real life implementations of those patterns and how they help us solve problems in scalable systems. By the end of the talk we’ll have a very powerful abstraction to tackle the everyday problems we face in building big and impactful systems.
Everyday tools and tricks for scaling Node.js
In order to understand how to scale Node.js you need to know how the internals work together and what type of problems are best suited for it. With the right combination of tools you can easily have a scalable and reliable Node.js cluster.
Node.js at Uber
In this talk we'll explore different open source projects developed at Uber that helps you build scalable Node.js applications.
Tracing python applications
As hackers we love to understand how stuff works and how to optimize it. A very good tool to do both is software tracing. During the talk we'll see how tracing tools work and we'll zoom on one particular project called pyflame.
Distributed tracing for Node.js
Distributed tracing is a very useful practice for Node.js because it gives you a good visibility over the way your async code executes and the lifecycle of your external calls as they travels between many services.
Design Patterns for Docker Applications
In the container world you can use design patterns the same way you do in the object oriented world. There are some common problems in building scalable systems and we can use design patterns to solve them efficiently.
From Python to Java
As software engineers we do trade-offs every day. We often need to pick between things like space vs time or budget vs scope. Or sometimes the amount of creative waste we can afford to have. And when we make the decision we need to be in full comprehension of both the upside and downside of a particular choice. In this talk we will discuss why our organization decided to move from Python to Java. We will go over each tradeoff we decided to do and the motivation behind it.
OpenFest 2016 - Open Microservice Architecture
The microservice architecture approach has been very popular in the recent years. There is a big hype around it and a large swarm of open source tools to facilitate each aspect of this architecture. The purpose of this talk is to identify the main components of a microservice architecture. After that we compare different open source tools that fits into each area. At the end we’ll have a good understanding what a microservice architecture based on OSS looks like.
More from Nikolay Stoitsev (20)
3 lessons on effective communication for engineers
3 lessons on effective communication for engineers
The career path of software engineers and how to navigate it
The career path of software engineers and how to navigate it
Migrating a data intensive microservice from Python to Go
Migrating a data intensive microservice from Python to Go
Reusable patterns for scalable APIs running on Docker @ Java2Days
Reusable patterns for scalable APIs running on Docker @ Java2Days
Recently uploaded
Graspan: A Big Data System for Big Code Analysis
We built a disk-based parallel graph system, Graspan, that uses a novel edge-pair centric computation model to compute dynamic transitive closures on very large program graphs.
We implement context-sensitive pointer/alias and dataflow analyses on Graspan. An evaluation of these analyses on large codebases such as Linux shows that their Graspan implementations scale to millions of lines of code and are much simpler than their original implementations.
These analyses were used to augment the existing checkers; these augmented checkers found 132 new NULL pointer bugs and 1308 unnecessary NULL tests in Linux 4.4.0-rc5, PostgreSQL 8.3.9, and Apache httpd 2.2.18.
- Accepted in ASPLOS ‘17, Xi’an, China.
- Featured in the tutorial, Systemized Program Analyses: A Big Data Perspective on Static Analysis Scalability, ASPLOS ‘17.
- Invited for presentation at SoCal PLS ‘16.
- Invited for poster presentation at PLDI SRC ‘16.
Enhancing Research Orchestration Capabilities at ORNL.pdf
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
Cracking the code review at SpringIO 2024
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
Text-Summarization-of-Breaking-News-Using-Fine-tuning-BART-Model.pptx
Text summarization of braking news using fine tuning BART model
Lecture 1 Introduction to games development
Software engineering
Game Development
unity Game Development
E-commerce Application Development Company.pdf
Your business can reach new heights with our assistance as we design solutions that are specifically appropriate for your goals and vision. Our eCommerce application solutions can digitally coordinate all retail operations processes to meet the demands of the marketplace while maintaining business continuity.
Game Development with Unity3D (Game Development lecture 3)
Game Development with Unity3D (Game Development lecture 3)
Empowering Growth with Best Software Development Company in Noida - Deuglo
Do you want Software for your Business? Visit Deuglo
Deuglo has top Software Developers in India. They are experts in software development and help design and create custom Software solutions.
Deuglo follows seven steps methods for delivering their services to their customers. They called it the Software development life cycle process (SDLC).
Requirement — Collecting the Requirements is the first Phase in the SSLC process.
Feasibility Study — after completing the requirement process they move to the design phase.
Design — in this phase, they start designing the software.
Coding — when designing is completed, the developers start coding for the software.
Testing — in this phase when the coding of the software is done the testing team will start testing.
Installation — after completion of testing, the application opens to the live server and launches!
Maintenance — after completing the software development, customers start using the software.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
OpenMetadata Community Meeting - 5th June 2024
The OpenMetadata Community Meeting was held on June 5th, 2024. In this meeting, we discussed about the data quality capabilities that are integrated with the Incident Manager, providing a complete solution to handle your data observability needs. Watch the end-to-end demo of the data quality features.
* How to run your own data quality framework
* What is the performance impact of running data quality frameworks
* How to run the test cases in your own ETL pipelines
* How the Incident Manager is integrated
* Get notified with alerts when test cases fail
Watch the meeting recording here - https://www.youtube.com/watch?v=UbNOje0kf6E
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-fusion-buddy-review
AI Fusion Buddy Review: Key Features
✅Create Stunning AI App Suite Fully Powered By Google's Latest AI technology, Gemini
✅Use Gemini to Build high-converting Converting Sales Video Scripts, ad copies, Trending Articles, blogs, etc.100% unique!
✅Create Ultra-HD graphics with a single keyword or phrase that commands 10x eyeballs!
✅Fully automated AI articles bulk generation!
✅Auto-post or schedule stunning AI content across all your accounts at once—WordPress, Facebook, LinkedIn, Blogger, and more.
✅With one keyword or URL, generate complete websites, landing pages, and more…
✅Automatically create & sell AI content, graphics, websites, landing pages, & all that gets you paid non-stop 24*7.
✅Pre-built High-Converting 100+ website Templates and 2000+ graphic templates logos, banners, and thumbnail images in Trending Niches.
✅Say goodbye to wasting time logging into multiple Chat GPT & AI Apps once & for all!
✅Save over $5000 per year and kick out dependency on third parties completely!
✅Brand New App: Not available anywhere else!
✅ Beginner-friendly!
✅ZERO upfront cost or any extra expenses
✅Risk-Free: 30-Day Money-Back Guarantee!
✅Commercial License included!
See My Other Reviews Article:
(1) AI Genie Review: https://sumonreview.com/ai-genie-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
#AIFusionBuddyReview,
#AIFusionBuddyFeatures,
#AIFusionBuddyPricing,
#AIFusionBuddyProsandCons,
#AIFusionBuddyTutorial,
#AIFusionBuddyUserExperience
#AIFusionBuddyforBeginners,
#AIFusionBuddyBenefits,
#AIFusionBuddyComparison,
#AIFusionBuddyInstallation,
#AIFusionBuddyRefundPolicy,
#AIFusionBuddyDemo,
#AIFusionBuddyMaintenanceFees,
#AIFusionBuddyNewbieFriendly,
#WhatIsAIFusionBuddy?,
#HowDoesAIFusionBuddyWorks
Introduction to Pygame (Lecture 7 Python Game Development)
Introduction to Pygame (Lecture 7 Python Game Development)
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
This session covers GraalVM & Spring Boot, Spring AOT, performance optimizations and using libraries
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
Zoom is a comprehensive platform designed to connect individuals and teams efficiently. With its user-friendly interface and powerful features, Zoom has become a go-to solution for virtual communication and collaboration. It offers a range of tools, including virtual meetings, team chat, VoIP phone systems, online whiteboards, and AI companions, to streamline workflows and enhance productivity.
Mobile App Development Company In Noida | Drona Infotech
Looking for a reliable mobile app development company in Noida? Look no further than Drona Infotech. We specialize in creating customized apps for your business needs.
Visit Us For : https://www.dronainfotech.com/mobile-application-development/
Globus Compute wth IRI Workflows - GlobusWorld 2024
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
A Sighting of filterA in Typelevel Rite of Passage
Slide deck home: https://fpilluminated.com/deck/220
Vitthal Shirke Java Microservices Resume.pdf
Software Engineering, Software Consulting, Tech Lead, Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Transaction, Spring MVC, OpenShift Cloud Platform, Kafka, REST, SOAP, LLD & HLD.
Recently uploaded (20)
Enhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdf
Text-Summarization-of-Breaking-News-Using-Fine-tuning-BART-Model.pptx
Text-Summarization-of-Breaking-News-Using-Fine-tuning-BART-Model.pptx
Game Development with Unity3D (Game Development lecture 3)
Game Development with Unity3D (Game Development lecture 3)
Empowering Growth with Best Software Development Company in Noida - Deuglo
Empowering Growth with Best Software Development Company in Noida - Deuglo
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
Introduction to Pygame (Lecture 7 Python Game Development)
Introduction to Pygame (Lecture 7 Python Game Development)
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024
A Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of Passage
Using Apache Kafka from Go
- 1. Using Apache Kafka from Go Nikolay Stoitsev, Software Engineer
- 2. Trillion+ Messages/Day PBs Data Volume Tens of Thousands Topics
- 3. Message Message Message Message Message Message Topic
- 4. Partition - ordered, immutable sequence Message Message Message Message Message Message Message Message Message Message Message Message Message Message Message Partition 0 Partition 1 Partition 2
- 5. Offset Message Message Message Message MessagePartition 0 Offset: 1
- 6. Offset Message Message Message Message MessagePartition 0 Offset: 2
- 7. Offset Message Message Message Message MessagePartition 0 Offset: 2 Offset: 3 Consumer: 1Consumer: 2
- 15. At-most-once cluster for logging
- 16. At-least-one cluster for message bus
- 17. Publishing to lossy kafka for logging
- 18. log.Printf("failed to fetch URL %s", url)
- 23. SugaredLogger
- 24. Logger
- 30. Publishing to lossless kafka
- 31. Avro
- 33. *net/http.Client
- 42. Heartbeat