The document discusses distributed tracing with Zipkin and Kubernetes. It defines microservices and their benefits and costs. Distributed tracing allows tracing requests across multiple services and systems. This helps identify bottlenecks and failures. The document discusses OpenTracing and Zipkin libraries for instrumenting services. It also covers Zipkin's architecture, performance considerations like sampling, and how Linkerd can help with legacy systems. Finally, it mentions answering questions about slow parts, optimizations, and failures using distributed tracing before cautioning about data volumes and interpretation.
Open Tracing, to order and understand your mess. - ApiConf 2017Gianluca Arbezzano
This about how many api calls your applications were doing 3-4 years ago, and think about how many integration and difference services your requests is crossing before to come back to the final destination. How do you know this step of your pipeline is taking too much time? What is taking 2 seconds to answer? Is it the authentication service? Maybe it's the invoice generation service or the notification platform. Open Tracing is a distributed tracing cross vendor and open source that help you to understand bottleneck and to profile the requests from where they arrive at the final user. In an ecosystem where microservices and as a service concept are growing this can be a real challenge. During this presentation, we will see how it works from a general point of view to land in some real implementation, examples, and demo.
Tracing Micro Services with OpenTracingHemant Kumar
Tracing in the world of micro services has become a standard with people using distributed tracers like Zipkin, Jaeger, Appdash etc. But, with so many different tracers, its confusing to choose one tracer and then painful to replace a tracer. That's where OpenTracing comes in. OT provides a consistent, vendor-neutral API to allow users to choose whatever distributed tracer they need and can change the tracer with just an O(1) operation.
Distributed tracing - get a grasp on your productionnklmish
Slides from my presentation on distributed tracing, explaining what is latency and why it matters. We took a look at openzipkin and its concepts like how the core annotations works, what are tags/logs, etc. Followed by a demo application created using golang and java (spring boot , spring cloud sleuth zipkin) . You can find source code here
https://github.com/nklmish/go-distributed-tracing-demo
https://github.com/nklmish/java-distributed-tracing-demo
Tracing 2000+ polyglot microservices at Uber with Jaeger and OpenTracingYuri Shkuro
Slides from my talk & demo at Go NYC Meeetup 19-Jan-2017.
We present Jaeger, Uber’s open source distributed tracing system, featuring Go backend, React based UI, and OpenTracing API support. We show examples of instrumenting application code for tracing and using distributed context propagation to attribute backend resource usage to top level consumers.
Open Tracing, to order and understand your mess. - ApiConf 2017Gianluca Arbezzano
This about how many api calls your applications were doing 3-4 years ago, and think about how many integration and difference services your requests is crossing before to come back to the final destination. How do you know this step of your pipeline is taking too much time? What is taking 2 seconds to answer? Is it the authentication service? Maybe it's the invoice generation service or the notification platform. Open Tracing is a distributed tracing cross vendor and open source that help you to understand bottleneck and to profile the requests from where they arrive at the final user. In an ecosystem where microservices and as a service concept are growing this can be a real challenge. During this presentation, we will see how it works from a general point of view to land in some real implementation, examples, and demo.
Tracing Micro Services with OpenTracingHemant Kumar
Tracing in the world of micro services has become a standard with people using distributed tracers like Zipkin, Jaeger, Appdash etc. But, with so many different tracers, its confusing to choose one tracer and then painful to replace a tracer. That's where OpenTracing comes in. OT provides a consistent, vendor-neutral API to allow users to choose whatever distributed tracer they need and can change the tracer with just an O(1) operation.
Distributed tracing - get a grasp on your productionnklmish
Slides from my presentation on distributed tracing, explaining what is latency and why it matters. We took a look at openzipkin and its concepts like how the core annotations works, what are tags/logs, etc. Followed by a demo application created using golang and java (spring boot , spring cloud sleuth zipkin) . You can find source code here
https://github.com/nklmish/go-distributed-tracing-demo
https://github.com/nklmish/java-distributed-tracing-demo
Tracing 2000+ polyglot microservices at Uber with Jaeger and OpenTracingYuri Shkuro
Slides from my talk & demo at Go NYC Meeetup 19-Jan-2017.
We present Jaeger, Uber’s open source distributed tracing system, featuring Go backend, React based UI, and OpenTracing API support. We show examples of instrumenting application code for tracing and using distributed context propagation to attribute backend resource usage to top level consumers.
Performance monitoring and call tracing in microservice environmentsMartin Gutenbrunner
Performance analysis can easily be done with on-board tools of nearly any programming language. In microservice environments, the real challenge is not in single, high-performing services, but in resiliently running a complex ecosystem of many services.This talk will introduce open-source tools for analysis and call tracing. Concluding, we will briefly get to know Dynatrace Ruxit - a commercial alternative. After this session, the audience will know about how to get started in performance analysis and call-tracing and some according tools.
OSMC 2018 | Tailored SNMP monitoring – Your own SNMP MIB and sub-agent with P...NETWAYS
SNMP continues to be a essential component in monitoring where the information being made available is structured in so-called Management Information (MIB) modules. The standard net-snmp distribution comes a with a variety of standard MIBs implemented by its snmpd, but sometimes there is the need to make your own information available via SNMP. Luckily snmpd can be dynamically extended by so-called subagents implementing the AgentX protocol (RFC2747). The net-snmp API however pretty much focusses on the C programming language only, laying the entrance barrier especially for non-developers rather high. In this talk Pieter will not only demonstrate the creation of a MIB, which, being a text file, is the easier part, but also how easy it is to implement a simple subagent in Python using his python-netsnmpagent module. python-netsnmpagent is a shim wrapper over the net-snmp C API trying to implement just enough abstraction. Licensed under the GPL it is available at https://github.com/pief/python-netsnmpagent as well as PyPI.
Using the FLiPN Stack for Edge AI (Flink, NiFi, Pulsar) - Pulsar Summit Asia ...StreamNative
Introducing the FLiPN stack which combines Apache Flink, Apache NiFi, Apache Pulsar and other Apache tools to build fast applications for IoT, AI, rapid ingest.
FLiPN provides a quick set of tools to build applications at any scale for any streaming and IoT use cases.
Tools
Apache Flink, Apache Pulsar, Apache NiFi, MiNiFi, Apache MXNet, DJL.AI
References
https://www.datainmotion.dev/2019/08/...
https://www.datainmotion.dev/2019/09/...
https://www.datainmotion.dev/2019/05/...
https://www.datainmotion.dev/2019/03/...
Get the presentation slides: https://www.slideshare.net/streamnati...
Subscribe to the StreamNative Newsletter for Apache Pulsar for more Pulsar content: https://share.hsforms.com/1IS56E-RvSV...
Get started with the on-demand Pulsar training by StreamNative Academy: https://www.academy.streamnative.io/
In this slide, we go through the Google Dapper, OpenTracing, Jaeger to OpenTelemetry. By reading and studying the history of Dapper, we could lean the experience and design theory of a large-scale distributed tracing system and then know how it affects other solutions, like OpenTracing and Jaeger.
We also discuss the difference between the OpenTracing and Jaeger and also demonstrate how Jaeger works and looks like.
After, we talked about the future of OpenTracing, the new organization called OpenTelemetry, what's its goal and how to do that.
OSMC 2018 | Distributed Tracing FAQ by Gianluca ArbezzanoNETWAYS
Microservices, containers and more in general distributed systems have opened a different point of view on our system and applications. We need to understand how a single event or requests cross our app jumping over networks, containers, virtual machines and sometime clod provider. There is a specific practice called distributed tracing to increase observability of systems like that. After this talk, you will have a solid idea around what tracing means, how you can instrument your applications and you will be ready to trace your application across many languages using open source technologies like OpenTracing, OpenCensus, Zipkin, Jaeger, InfluxDB. You will ask yourself how you survived until today!
The Web is broken. HTTP is inefficient and expensive, especially for large files. Webpages are being deleted constantly, with the average lifespan of a web being 100 days. The Web's centralization limits opportunity and innovation. And it causes problems in the developing world, with natural disasters or faulty connections. We can do better. In this talk, I'll explain IPFS, a project intended to replace HTTP and build a better web. IPFS is a peer-to-peer hypermedia protocol to make the web faster, safer, and more open. In addition, IPFS will use Filecoin as a reward mechanism. Filecoin aims to provide a decentralized network for digital storage through which users can effectively rent out their spare capacity, receiving filecoins as payment. Filecoin raised 200M$ last month, breaking all records in blockchain ICOs to date.
In this webinar we covered how to improve search with analytics using the Elastic Stack: ElasticSearch, Logstash, Kibana. Check out our upcoming events: www.mcplusa.com/events
As more and more developers move to distributed architectures such as micro services, distributed actor systems, and so forth it becomes increasingly complex to understand, debug, and diagnose.
In this talk we're going to introduce the emerging OpenTracing standard and talk about how you can instrument your applications to help visualize every operation, even across process and service boundaries. We'll also introduce Zipkin, one of the most popular implementations of the OpenTracing standard.
Lithe: Lightweight Secure CoAP for the Internet of ThingsJoon Young Park
Paper Survey.
Secure CoAP scheme for Internet of Things.
DTLS, 6LoWPAN
constrained environment.
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6576185
Amazon EKS 그리고 Service Mesh
Kubernetes는 컨테이너 서비스를 도입하는 기업들에게 가장 있기있는 Orchestration 플랫폼입니다. 이 세션에서는 아마존에서 6월 정식 출시한 managed Kubenetes서비스인 EKS를 소개해드리며, 오픈소스 버전과의 차이점 및 장점 등에 대해 설명하고, 진보한 마이크로 서비스인 Service Mesh를 구현하는 Linkerd 소개 및 데모를 진행하고자 합니다.
apidays New York 2022 - Leveraging Event Streaming to Super-Charge your Busin...apidays
apidays New York 2022 - Beyond API Regulations for Finance, Insurance, and Healthcare
July 27 & 28, 2022
Leveraging Event Streaming to Super-Charge your Business
Mary Grygleski, Streaming Developer Advocate at DataStax
------------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
Deep dive into the API industry with our reports:
https://www.apidays.global/industry-reports/
Subscribe to our global newsletter:
https://apidays.typeform.com/to/i1MPEW
Apache Kafka - Scalable Message-Processing and more !Guido Schmutz
ndependent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. How can me make sure that all these event are accepted and forwarded in an efficient and reliable way? This is where Apache Kafaka comes into play, a distirbuted, highly-scalable messaging broker, build for exchanging huge amount of messages between a source and a target.
This session will start with an introduction into Apache and presents the role of Apache Kafka in a modern data / information architecture and the advantages it brings to the table. Additionally the Kafka ecosystem will be covered as well as the integration of Kafka in the Oracle Stack, with products such as Golden Gate, Service Bus and Oracle Stream Analytics all being able to act as a Kafka consumer or producer.
Performance monitoring and call tracing in microservice environmentsMartin Gutenbrunner
Performance analysis can easily be done with on-board tools of nearly any programming language. In microservice environments, the real challenge is not in single, high-performing services, but in resiliently running a complex ecosystem of many services.This talk will introduce open-source tools for analysis and call tracing. Concluding, we will briefly get to know Dynatrace Ruxit - a commercial alternative. After this session, the audience will know about how to get started in performance analysis and call-tracing and some according tools.
OSMC 2018 | Tailored SNMP monitoring – Your own SNMP MIB and sub-agent with P...NETWAYS
SNMP continues to be a essential component in monitoring where the information being made available is structured in so-called Management Information (MIB) modules. The standard net-snmp distribution comes a with a variety of standard MIBs implemented by its snmpd, but sometimes there is the need to make your own information available via SNMP. Luckily snmpd can be dynamically extended by so-called subagents implementing the AgentX protocol (RFC2747). The net-snmp API however pretty much focusses on the C programming language only, laying the entrance barrier especially for non-developers rather high. In this talk Pieter will not only demonstrate the creation of a MIB, which, being a text file, is the easier part, but also how easy it is to implement a simple subagent in Python using his python-netsnmpagent module. python-netsnmpagent is a shim wrapper over the net-snmp C API trying to implement just enough abstraction. Licensed under the GPL it is available at https://github.com/pief/python-netsnmpagent as well as PyPI.
Using the FLiPN Stack for Edge AI (Flink, NiFi, Pulsar) - Pulsar Summit Asia ...StreamNative
Introducing the FLiPN stack which combines Apache Flink, Apache NiFi, Apache Pulsar and other Apache tools to build fast applications for IoT, AI, rapid ingest.
FLiPN provides a quick set of tools to build applications at any scale for any streaming and IoT use cases.
Tools
Apache Flink, Apache Pulsar, Apache NiFi, MiNiFi, Apache MXNet, DJL.AI
References
https://www.datainmotion.dev/2019/08/...
https://www.datainmotion.dev/2019/09/...
https://www.datainmotion.dev/2019/05/...
https://www.datainmotion.dev/2019/03/...
Get the presentation slides: https://www.slideshare.net/streamnati...
Subscribe to the StreamNative Newsletter for Apache Pulsar for more Pulsar content: https://share.hsforms.com/1IS56E-RvSV...
Get started with the on-demand Pulsar training by StreamNative Academy: https://www.academy.streamnative.io/
In this slide, we go through the Google Dapper, OpenTracing, Jaeger to OpenTelemetry. By reading and studying the history of Dapper, we could lean the experience and design theory of a large-scale distributed tracing system and then know how it affects other solutions, like OpenTracing and Jaeger.
We also discuss the difference between the OpenTracing and Jaeger and also demonstrate how Jaeger works and looks like.
After, we talked about the future of OpenTracing, the new organization called OpenTelemetry, what's its goal and how to do that.
OSMC 2018 | Distributed Tracing FAQ by Gianluca ArbezzanoNETWAYS
Microservices, containers and more in general distributed systems have opened a different point of view on our system and applications. We need to understand how a single event or requests cross our app jumping over networks, containers, virtual machines and sometime clod provider. There is a specific practice called distributed tracing to increase observability of systems like that. After this talk, you will have a solid idea around what tracing means, how you can instrument your applications and you will be ready to trace your application across many languages using open source technologies like OpenTracing, OpenCensus, Zipkin, Jaeger, InfluxDB. You will ask yourself how you survived until today!
The Web is broken. HTTP is inefficient and expensive, especially for large files. Webpages are being deleted constantly, with the average lifespan of a web being 100 days. The Web's centralization limits opportunity and innovation. And it causes problems in the developing world, with natural disasters or faulty connections. We can do better. In this talk, I'll explain IPFS, a project intended to replace HTTP and build a better web. IPFS is a peer-to-peer hypermedia protocol to make the web faster, safer, and more open. In addition, IPFS will use Filecoin as a reward mechanism. Filecoin aims to provide a decentralized network for digital storage through which users can effectively rent out their spare capacity, receiving filecoins as payment. Filecoin raised 200M$ last month, breaking all records in blockchain ICOs to date.
In this webinar we covered how to improve search with analytics using the Elastic Stack: ElasticSearch, Logstash, Kibana. Check out our upcoming events: www.mcplusa.com/events
As more and more developers move to distributed architectures such as micro services, distributed actor systems, and so forth it becomes increasingly complex to understand, debug, and diagnose.
In this talk we're going to introduce the emerging OpenTracing standard and talk about how you can instrument your applications to help visualize every operation, even across process and service boundaries. We'll also introduce Zipkin, one of the most popular implementations of the OpenTracing standard.
Lithe: Lightweight Secure CoAP for the Internet of ThingsJoon Young Park
Paper Survey.
Secure CoAP scheme for Internet of Things.
DTLS, 6LoWPAN
constrained environment.
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6576185
Amazon EKS 그리고 Service Mesh
Kubernetes는 컨테이너 서비스를 도입하는 기업들에게 가장 있기있는 Orchestration 플랫폼입니다. 이 세션에서는 아마존에서 6월 정식 출시한 managed Kubenetes서비스인 EKS를 소개해드리며, 오픈소스 버전과의 차이점 및 장점 등에 대해 설명하고, 진보한 마이크로 서비스인 Service Mesh를 구현하는 Linkerd 소개 및 데모를 진행하고자 합니다.
apidays New York 2022 - Leveraging Event Streaming to Super-Charge your Busin...apidays
apidays New York 2022 - Beyond API Regulations for Finance, Insurance, and Healthcare
July 27 & 28, 2022
Leveraging Event Streaming to Super-Charge your Business
Mary Grygleski, Streaming Developer Advocate at DataStax
------------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
Deep dive into the API industry with our reports:
https://www.apidays.global/industry-reports/
Subscribe to our global newsletter:
https://apidays.typeform.com/to/i1MPEW
Apache Kafka - Scalable Message-Processing and more !Guido Schmutz
ndependent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. How can me make sure that all these event are accepted and forwarded in an efficient and reliable way? This is where Apache Kafaka comes into play, a distirbuted, highly-scalable messaging broker, build for exchanging huge amount of messages between a source and a target.
This session will start with an introduction into Apache and presents the role of Apache Kafka in a modern data / information architecture and the advantages it brings to the table. Additionally the Kafka ecosystem will be covered as well as the integration of Kafka in the Oracle Stack, with products such as Golden Gate, Service Bus and Oracle Stream Analytics all being able to act as a Kafka consumer or producer.
Osacon 2021 hello hydrate! from stream to clickhouse with apache pulsar and...Timothy Spann
https://altinity.com/osa-con-2021/
An empty real-time SQL data warehouse is not useful to anyone. How can you load data quickly from diverse data sources? Utilizing open source tools from Apache, the FLiP stack enables any data engineer, programmer or analyst to build reusable modules with low or no code. We’ll show how to use them to load CDC, logs, events, XML, images, and many other types of data into ClickHouse and similar data warehouses.
Osacon 2021 hello hydrate! from stream to clickhouse with apache pulsar and friends
Microservices for performance - GOTO Chicago 2016Peter Lawrey
How do Microservices and Trading Systems overlap?
How can one area learn from the other?
How can we test components of microservices?
Is there a library which helps us implement and test these services?
Big Data Streams Architectures. Why? What? How?Anton Nazaruk
With a current zoo of technologies and different ways of their interaction it's a big challenge to architect a system (or adopt existed one) that will conform to low-latency BigData analysis requirements. Apache Kafka and Kappa Architecture in particular take more and more attention over classic Hadoop-centric technologies stack. New Consumer API put significant boost in this direction. Microservices-based streaming processing and new Kafka Streams tend to be a synergy in BigData world.
Let’s Monitor Conditions at the Conference With Timothy Spann & David Kjerrum...HostedbyConfluent
Let’s Monitor Conditions at the Conference With Timothy Spann & David Kjerrumgaard | Current 2022
At home, I monitor the temperature, humidity, gas levels, ozone, air quality, and other features around my desk.
Let's bring this to the different spots around the conference including lunch tables, vendor booths, hotel rooms, and more. I need to know about these readings now, not when I get back home from the conference. We need to get these sensor readings immediately in case we need to turn on a fan or move to another area. We will also see if my talk produces a lot of hot air!?!??
My setup is pretty simple, a raspberry pi, a breakout garden sensor mount, and as many sensors as I am willing to fly to Austin. The software stack is Python and Java, Apache Pulsar, MQTT, HTML, JQuery, and Apache Kafka.
https://dzone.com/articles/five-sensors-real-time-with-pulsar-and-python-on-a
https://www.datainmotion.dev/2022/04/flip-py-pi-enviroplus-using-apache.html
https://dzone.com/articles/pulsar-in-python-on-pi
(Current22) Let's Monitor The Conditions at the ConferenceTimothy Spann
(Current22) Let's Monitor The Conditions at the Conference
Let's Monitor The Conditions at the Conference
Session Time11:15 am - 12:00 pm Session DateWednesday, 5 October 2022 Session Type:In-Person Location:Ballroom G
Session Description:
At home, I monitor the temperature, humidity, gas levels, ozone, air quality, and other features around my desk. Let's bring this to the different spots around the conference including lunch tables, vendor booths, hotel rooms, and more. I need to know about these readings now, not when I get back home from the conference. We need to get these sensor readings immediately in case we need to turn on a fan or move to another area. We will also see if my talk produces a lot of hot air!? My setup is pretty simple, a raspberry pi, a breakout garden sensor mount, and as many sensors as I am willing to fly to Austin. The software stack is Python and Java, Apache Pulsar, MQTT, HTML, JQuery, and Apache Kafka.
Timothy Spann, StreamNative
Developer Advocate
Tim Spann is a Developer Advocate @ StreamNative where he works with Apache Pulsar, Apache Flink, Apache NiFi, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC.
Data Streaming with Apache Kafka & MongoDBconfluent
Explore the use-cases and architecture for Apache Kafka, and how it integrates with MongoDB to build sophisticated data-driven applications that exploit new sources of data.
Data Streaming with Apache Kafka & MongoDB - EMEAAndrew Morgan
A new generation of technologies is needed to consume and exploit today's real time, fast moving data sources. Apache Kafka, originally developed at LinkedIn, has emerged as one of these key new technologies.
This webinar explores the use-cases and architecture for Kafka, and how it integrates with MongoDB to build sophisticated data-driven applications that exploit new sources of data.
Webinar: Data Streaming with Apache Kafka & MongoDBMongoDB
A new generation of technologies is needed to consume and exploit today's real time, fast moving data sources. Apache Kafka, originally developed at LinkedIn, has emerged as one of these key new technologies.
Captial One: Why Stream Data as Part of Data Transformation?ScyllaDB
Event-driven architectures are increasingly part of a complete data transformation solution. Learn how to employ Apache Kafka, Cloud Native Computing Foundation’s NATS, Amazon SQS, or other message queueing technologies. This talks covers the details of each, their advantages and disadvantages and how to select the best for your company’s needs.
Following simple patterns of good application design can allow you to scale your application for your customers easily. This presentation dives into the 12 factor application design and demo how this applies to containers and deployments on Amazon ECS and Fargate. We'll take a look at tooling that can be used to simplify your workflow and help you adopt the principles of the 12 factor application.
Amazon aws big data demystified | Introduction to streaming and messaging flu...Omid Vahdaty
amazon aws big data demystified meetup:
https://www.meetup.com/AWS-Big-Data-Demystified/
Introduction to streaming and messaging flume kafka sqs kinesis
PHP At 5000 Requests Per Second: Hootsuite’s Scaling Storyvanphp
Bill Monkman, Lead Engineer at Hootsuite, presenting on how Hootsuite went from zero to hundreds of millions of requests per day with its PHP codebase, and how dealing with that growth has shaped its future direction. Tips, optimizations, and horror stories from a rapidly-scaling PHP startup.
Video: https://www.youtube.com/watch?v=TZGeBAIMPII
Handling eventual consistency in a transactional world with Matteo Cimini and...HostedbyConfluent
Change data capture (CDC) is a widely used solution to offload data in real time from legacy systems to Kafka in order to make it available to all the other downstream consumer applications. Despite other solutions CDC can in fact guarantee at the same time low latency and a very small footprint on the source system. However when data is moved from a relational database to a distributed stream platform what is gained in terms of throughput and latency is lost in terms of strong consistency and not all consumers are able to manage this loss by themselves. There are different upstream solutions that can be implemented to mitigate this problem preserving different levels of consistency.
In this talk we’ll:
- see what is eventual consistency and where strong consistency is lost while moving data from a database to Kafka
- describe different solutions to preserve consistency working at the source level (i.e. outbox pattern and call back pattern), working on Kafka topology or working on an external storage (i.e. integration hub)
- analyze the pros and cons of all the presented solutions in terms of consistency guarantees and latency loss
Introduction to streaming and messaging flume,kafka,SQS,kinesis Omid Vahdaty
Big data makes you a bit Confused ? messaging? batch processing? data streaming? in flight analytics? Cloud? open source? Flume? kafka? flafka (both)? SQS? kinesis? firehose?
Similar to Distributed Tracing with OpenTracing, ZipKin and Kubernetes (20)
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Monitoring Java Application Security with JDK Tools and JFR Events
Distributed Tracing with OpenTracing, ZipKin and Kubernetes
1. container-solutions.com | @containersoluti | info@container-solutions.com
Distributed Tracing
with ZipKin &
Kubernetes
Maximilian Schöfmann
@schoefmann
Container Solutions AG
@containersoluti
2. container-solutions.com | @containersoluti | info@container-solutions.com
Microservices...
In short, the microservice architectural style is an approach to develop a single application
as a suite of small services, each running in its own process and communicating with
lightweight mechanisms, often an HTTP resource API. These services are built around
business capabilities and independently deployable by fully automated deployment
machinery. There is a bare minimum of centralized management of these services,
which may be written in different programming languages and use different data storage
technologies.
-- James Lewis and Martin Fowler
11. Distributed Tracing | container-solutions.com
Why distributed tracing?
“Per-process logging and metric monitoring have their place, but
neither can reconstruct the elaborate journeys that transactions
take as they propagate across a distributed system.
Distributed traces are these journeys.”
-- Chris Aniszczyk, Cloud Native Computing Foundation
12. Distributed Tracing | container-solutions.com
Fundamental requirements to make it work
● Ubiquitous deployment
● Continuous monitoring
See also: “Dapper, a Large-Scale Distributed Systems Tracing Infrastructure”
http://research.google.com/pubs/pub36356.html (2010)
13. Distributed Tracing | container-solutions.com
Requirements to make is useful
● Low overhead
● Application-level transparency
● Scalability
● (Timely) data availability
14. Distributed Tracing | container-solutions.com
A distributed trace...
“A tracing infrastructure for distributed
services needs to record information
about all the work done in a system, on
behalf of a given initiator”
15. Distributed Tracing | container-solutions.com
Data aggregation
Message record:
Record = Message identifier + timestamped event
Data aggregation classes:
● Black box
● Annotation-based
16. Distributed Tracing | container-solutions.com
● Trace as a tree of nested calls
● Trace trees and spans
Trace data model
17. www.container-solutions.com | info@container-solutions.com
Span
Logged event in a typical span
● Span name
● Span start time
● Span end time
● Trace id
● Span id
● Span parent id
● Any timing information recorded by the instrumentation library (RPC, HTTP)
● Additional custom labels (“foo”)
18. www.container-solutions.com | info@container-solutions.com
OpenTracing & ZipKin
Common libraries for several programming languages
➔ Libraries attach a trace context to the thread local storage
➔ RPC friendly (specially when using gRPC)
➔ The data is language-independent
opentracing.io zipkin.io
26. www.container-solutions.com | info@container-solutions.com
Sampling
➔ 2-stage sampling:
a. Client: Don’t send every trace instrumented
● limits client-side CPU and bandwidth overhead
● adjustable per service, hard to change in one go
b. Server: Don’t persist every trace received
● limits server-side IO and data volume overhead
● adjustable centrally with simple config change
➔ Adaptive sampling to trade off overhead against missing relevant traces
30. www.container-solutions.com | info@container-solutions.com
Some of the answered questions...
...with a distributed tracing system are:
● Which parts of my system are slow?
● Which call pattern can be optimized with parallelization?
● Which calls are redundant?
● Which routes are affected by this failing part?
● Under which circumstances is it failing?
● How often is it failing?
● Detect queries issued to read and write masters,
instead of read only replicas
31. www.container-solutions.com | info@container-solutions.com
A word of caution about distributed tracing
● Documentation is still rather poor
● Yet another moving part
● Can accumulate huge amounts of data
● Metrics need to be interpreted
● Commercial APM solutions might be an easier route for your use case...
32. www.container-solutions.com | info@container-solutions.com
A word of caution about distributed tracing
● Documentation is still rather poor
● Yet another moving part
● Can accumulate huge amounts of data
● Metrics need to be interpreted
● Commercial APM solutions might be an easier route for your use case...
35. www.container-solutions.com | info@container-solutions.com
Questions? Want to learn more?
● Come to our 2 day tinyurl.com/microservice-workshop
(November 8. + 9. or at your company on request)
● Follow us on Twitter: @containersoluti
● Read more on our blog: container-solutions.com/blog
● Or just get in touch: info@container-solutions.com