This document discusses considerations for implementing Kafka in production environments. It covers topics like event sourcing vs event basing, scaling and clustering Kafka, partitioning strategies, and ensuring data resiliency and fault tolerance. It provides guidance on configuration, rollout strategies, retention policies, replication, and addressing consumer lag.
The document discusses Kafka infrastructure monitoring. It describes important Kafka metrics for brokers, producers and consumers, as well as metrics for Zookeeper. It also summarizes two open source Kafka monitoring tools: Yahoo Kafka Manager and LinkedIn Burrow. Both tools monitor Kafka clusters and consumer lag, with Burrow additionally providing consumer status and notifications.
The document discusses running Apache Kafka infrastructure in the cloud. It covers the benefits of using cloud computing such as cost reduction, scalability, and high availability. It also describes various Platform as a Service (PaaS) cloud solutions for running Kafka including offerings from AWS, Google Cloud, Azure, and Confluent Cloud. Finally, it discusses using Kubernetes for orchestrating and managing containers in a cloud-based Kafka deployment.
This document provides information on Kafka infrastructure services including the Schema Registry, Kafka Connect, KSQL, and the HTTP Proxy. It describes the functions and components of each service. The Schema Registry stores and retrieves Avro schemas for Kafka data. Kafka Connect allows ingestion and delivery of data between Kafka and other systems using reusable connectors. KSQL is a streaming SQL engine that enables SQL queries on Kafka streams without coding. The HTTP Proxy provides a REST API for common Kafka operations like production and consumption of messages.
Corporate presentation TJC Group - SAP PartnerThierryJulien
TJC Group supports organizations to overcome complex problems within SAP Data Volume Management (DVM), from Data Archiving to SAP Information Lifecycle Management (ILM), GDPR and SAP data extraction for tax, audit and SAF-T requirements and the decommissioning of legacy systems. TJC Group is an SAP partner with over 20 years’ experience providing DVM consultancy services and SAP-certified software. Our 500 clients span across a wide range of industries and sizes and are located globally.
TJC’s mission is to craft an intelligent Data Management Strategy for our clients, ensuring data consistency and stability from day one to achieve business excellence.
Because your data matters.
To learn more, please email: contact@tjc-group.com or visit our website: www.tjc-group.com.
VEGA Pressure & Level Measurement - Oil and Gas Offshore ApplicationsThorne & Derrick UK
The document discusses VEGA's instrumentation products for the oil and gas industry, focusing on offshore applications. It describes VEGA's modular plics® instrument system, which allows for customized sensor, fitting, electronics, and housing combinations. It also highlights VEGA's level, limit level, and pressure measurement technologies, explosion-proof certifications, and ability to operate in harsh offshore conditions. The document emphasizes the reliability, safety, and ease-of-use of VEGA's instrumentation solutions for various offshore processes.
Los centros de datos son infraestructuras críticas cuyo diseño, certificación, operación y mantenimiento constituye una convergencia de disciplinas: arquitectura (edificación, restauración, impacto en el paisaje), diferentes ingenierías (informática, teleco, eléctrica, mecánica/climatización) y ecología (energías renovables, eficiencia energética). Esta conferencia propone un viaje breve, pero intenso por ese territorio, todavía en parte por explorar. Un terreno que debe conocer el ingeniero informático, asumiendo de forma definitiva que los bits no son entidades espirituales, al contrario, tienen masa, ocupan volumen y su movimiento o transformación requiere energía que inevitablemente se transforma en calor.
PDO Predictive Analytics Share for the Annual Research Forum 2015Faris Al-Kharusi
The Research Council of Oman's Annual Research Forum invited the Real-Time Operations Team of PDO to speak about the progress so far and emerging trends. Author: Faris Al-Kharusi.
This deck has been reviewed by PDO for public domain share however all rights are reserved by the organization.
Get with the system - Rogerio Martins, Schneider Electric disucsses the advantages of modern distributed control systems in coal handling preparation plants.
The document discusses Kafka infrastructure monitoring. It describes important Kafka metrics for brokers, producers and consumers, as well as metrics for Zookeeper. It also summarizes two open source Kafka monitoring tools: Yahoo Kafka Manager and LinkedIn Burrow. Both tools monitor Kafka clusters and consumer lag, with Burrow additionally providing consumer status and notifications.
The document discusses running Apache Kafka infrastructure in the cloud. It covers the benefits of using cloud computing such as cost reduction, scalability, and high availability. It also describes various Platform as a Service (PaaS) cloud solutions for running Kafka including offerings from AWS, Google Cloud, Azure, and Confluent Cloud. Finally, it discusses using Kubernetes for orchestrating and managing containers in a cloud-based Kafka deployment.
This document provides information on Kafka infrastructure services including the Schema Registry, Kafka Connect, KSQL, and the HTTP Proxy. It describes the functions and components of each service. The Schema Registry stores and retrieves Avro schemas for Kafka data. Kafka Connect allows ingestion and delivery of data between Kafka and other systems using reusable connectors. KSQL is a streaming SQL engine that enables SQL queries on Kafka streams without coding. The HTTP Proxy provides a REST API for common Kafka operations like production and consumption of messages.
Corporate presentation TJC Group - SAP PartnerThierryJulien
TJC Group supports organizations to overcome complex problems within SAP Data Volume Management (DVM), from Data Archiving to SAP Information Lifecycle Management (ILM), GDPR and SAP data extraction for tax, audit and SAF-T requirements and the decommissioning of legacy systems. TJC Group is an SAP partner with over 20 years’ experience providing DVM consultancy services and SAP-certified software. Our 500 clients span across a wide range of industries and sizes and are located globally.
TJC’s mission is to craft an intelligent Data Management Strategy for our clients, ensuring data consistency and stability from day one to achieve business excellence.
Because your data matters.
To learn more, please email: contact@tjc-group.com or visit our website: www.tjc-group.com.
VEGA Pressure & Level Measurement - Oil and Gas Offshore ApplicationsThorne & Derrick UK
The document discusses VEGA's instrumentation products for the oil and gas industry, focusing on offshore applications. It describes VEGA's modular plics® instrument system, which allows for customized sensor, fitting, electronics, and housing combinations. It also highlights VEGA's level, limit level, and pressure measurement technologies, explosion-proof certifications, and ability to operate in harsh offshore conditions. The document emphasizes the reliability, safety, and ease-of-use of VEGA's instrumentation solutions for various offshore processes.
Los centros de datos son infraestructuras críticas cuyo diseño, certificación, operación y mantenimiento constituye una convergencia de disciplinas: arquitectura (edificación, restauración, impacto en el paisaje), diferentes ingenierías (informática, teleco, eléctrica, mecánica/climatización) y ecología (energías renovables, eficiencia energética). Esta conferencia propone un viaje breve, pero intenso por ese territorio, todavía en parte por explorar. Un terreno que debe conocer el ingeniero informático, asumiendo de forma definitiva que los bits no son entidades espirituales, al contrario, tienen masa, ocupan volumen y su movimiento o transformación requiere energía que inevitablemente se transforma en calor.
PDO Predictive Analytics Share for the Annual Research Forum 2015Faris Al-Kharusi
The Research Council of Oman's Annual Research Forum invited the Real-Time Operations Team of PDO to speak about the progress so far and emerging trends. Author: Faris Al-Kharusi.
This deck has been reviewed by PDO for public domain share however all rights are reserved by the organization.
Get with the system - Rogerio Martins, Schneider Electric disucsses the advantages of modern distributed control systems in coal handling preparation plants.
Case Study: Increasing Produban's Critical Systems Availability and PerformanceCA Technologies
The Santander Group is a Spanish banking group and the largest bank in the Eurozone by market value. It is also one of the largest banks in the world in terms of market capitalization. Produban is Santander’s group company responsible for Santander's entire IT infrastructure. The Produban challenge was to monitor - proactively and in real time - all transactions running in critical systems and being able to take action before major problems happen. Considering this scenario, Produban adopted CA Core APM (CA Introscope) in order to count with alerts that permit to the technical team to detect problems before they impact business.
For more information on DevOps solutions from CA Technologies, please visit: http://bit.ly/1wbjjqX
Industry 4.0 promises to create new customer value in the market place by unleashing a combination of new technologies, data analytics, new generation cyber-physical production systems and newer methods of human machine interfaces. What does a developing country like India need to do to join the race?
This document discusses the NOGESI project, which used Simware software to create a distributed simulation laboratory for the Spanish Ministry of Defense. Some key points:
- NOGESI used Simware and DDS (Data Distribution Service) to improve runtime interoperability between simulators, addressing long-standing issues with interoperability.
- It demonstrated interoperability between simulated, operational, and legacy systems in a HLA (High Level Architecture) federation.
- The architecture was based on an open, layered approach as defined by SISO's LSA (Layered Simulation Architecture) concept, using DDS as the underlying middleware to improve upon limitations of HLA.
- For under $1 million
VEGA Process Measurement (Level, Limit Level & Pressure) - Oil & Gas OffshoreThorne & Derrick UK
The document discusses instrumentation solutions from VEGA for various applications in offshore oil and gas production, including level, density, and flow measurement of drilling mud. VEGA offers a full portfolio of measurement technologies including radar, ultrasonic, guided microwave, and capacitive sensors. Their plics modular instrument system allows for customized configurations and easy installation. The technologies and products discussed can provide reliable measurement under harsh offshore conditions to ensure safety and optimize production processes.
The document summarizes new autoclave models called the Pro Line and Prime Line from MELAG. Key points include:
- The new autoclaves offer improved performance, functions, and design at affordable prices compared to previous models.
- Features include fast cycle times, large loading capacity, intuitive touchscreen interface, energy saving modes, and connectivity for documentation.
- Accessories like trays, water treatment systems, and software allow users to customize configurations to suit their needs.
The document discusses Tripp Lite's NH modular UPS models and their advantages for data centers. The NH models offer high efficiency and flexibility. They allow for lower installation and operating costs compared to non-modular UPS systems. The modular design also provides increased availability, redundancy, and easy upgradability. The document provides guidance on how to effectively sell the NH UPS models to data centers and what technical specifications to highlight to customers.
There are three levels of process control for thermal profiling that have advanced manufacturing:
1) Traditional manual profiling uses test boards with thermocouples but is unreliable, labor intensive, and damages boards.
2) Oven verification devices like the DeltaProbe provide more accurate and repeatable profiling without test boards.
3) Continuous monitoring systems like the Automatic Profiling System (APS) profile every board and detect process faults in real-time, improving quality and reducing costs. The APS represents the most advanced profiling technology.
VEGA Pressure & Level Measurement - Paper Industry ApplicationsThorne & Derrick UK
VEGA provides measurement technology for the paper industry, including level, pressure, density, and point level devices. Their plics modular system allows customization of sensors, fittings, electronics, and housings. VEGA's devices are robust for the demanding conditions in paper processes like pulp cooking, bleaching towers, stock preparation, and more.
White Paper Mold-ID - Mold Management for Injection Molding with RFIDManfred Muenzl
Tools are subject to wear and tear and must be maintained and serviced on a regular base.
Inspection of the tools is often dependent on the experience values of individual employees, handwritten notes or planned figures in the ERP system that are not available to everyone or that are not monitored.
In many cases the maintenance or inspection is only carried out when the produced parts no longer fulfill the required quality standards or if the mold fails at its service.
That causes unplanned downtime of the machine and is not acceptable.
The document discusses how MOSAICO's Intelligent Platform transforms operations in lime plants through digital solutions. It introduces MOSAICO, an industrial internet platform that collects operational data and provides visualization, analytics, and reporting to optimize asset performance and operations. The platform was implemented on a QualiCal kiln for Calcis Group to reduce downtime, optimize maintenance costs, and introduce smart maintenance through the ZERO App. This allows maintenance managers to monitor plant performance remotely and technicians to access maintenance documentation and provide feedback from mobile devices. The results have been more efficient maintenance and increased plant availability through minimized unplanned stoppages.
With digital solutions transforming and imporving operations in lime plants, the MOSAICO intelligent platform has proven to be a game changer for maintenance and operatios
The article looks at how new technologies will lead to an increasingly integrated approach within the O&G sector, siting specifics such as the IoT and robotics & the radical impact they will have on optimising productions within the sector.
Network performance - skilled craft to hard scienceMartin Geddes
This document describes the technical and business journey for network operators wanting to turn network performance from a skilled craft into hard science.
This document discusses Industry 4.0 and how KEOPS helps manufacturers with digital transformation through their manufacturing intelligence solutions. It summarizes that KEOPS:
1) Helps maximize productivity across the manufacturing process from design to distribution through data collection, analysis, and visualization.
2) Allows manufacturers to transition from mass production to customized production by reducing time-to-market.
3) Integrates external supply chains to connect suppliers and customers.
4) Gathers and analyzes performance data to increase productivity, quality, and profits.
In 2017, the World Economic Forum recognized the potential of advanced manufacturing technologies. In 2018, from among more than 1,000 examined production facilities, 16
companies were recognized as Fourth Industrial Revolution leaders in advanced manufacturing for demonstrating step-change results, both operational and financial, across individual sites. They had succeeded in scaling beyond the pilot phase and their sites were designated advanced manufacturing “Lighthouses”. In 2019, 28 additional facilities were identified and added to the network, which now provides an opportunity for cross-company learning and collaboration, and for setting new benchmarks for the global manufacturing community.
Lighthouses have succeeded by innovating new operating systems, including in how they manage and optimize business and processes, transforming the way people work and use technology. These new operating systems can become the blueprint for modernizing the entire company operating system; therefore, how they prepare for scaling up and engaging the workforce matters.
Virtual Human Brain Simulations with Abaqus in the CloudThe UberCloud
UberCloud, Dassault Systèmes Simulia and Advania Data Centers presentation about the award winning project: HPC Cloud Simulation of Neuromodulation in Schizophrenia. Learn how simulation and high performance computing in the cloud play a key role in accelerating personalized healthcare.
The Brain Neuromodulation project represents a breakthrough in demonstrating the high value of computational modeling and simulation in improving the clinical application of non-invasive electro-stimulation of the human brain in schizophrenia and the potential to apply this technology to the treatment of other neuropsychiatric disorders such as depression and Parkinson’s disease. With the addition of HPC, clinicians can now precisely and non-invasively target regions of the brain without disrupting nearby healthy brain regions.
2018 Hyperion HPC Innovation Excellence Award: UberCloud, the National Institute of Mental Health & Neuro Sciences (NIMHANS) in Bangalore, Dassault Systèmes Simulia, Advania Data Centers, Hewlett Packard Enterprise and Intel won the 2018 Hyperion HPC Innovation Excellence Award for their Neuromodulation Project, based on computer simulations of non-invasive transcranial electro-stimulation of the human brain in schizophrenia.
The document discusses the evolution of the digital oilfield concept from early data gathering to modern integrated operations enabled by technology advances. It provides an overview of key developments like early logging data transmission and downhole sensors. The modern digital oilfield uses real-time data in continuous optimization cycles and allows centralized decision making. Integrated operations aim to improve effectiveness through cross-discipline collaboration. An example is provided in the Kuwait Intelligent Digital Field project.
Gravity White Paper - How to Close the 3rd Party Logistics Technology GapAero Wong
Tactical, transactional-based services are becoming less profitable. In order to compete, 3PLs and freight forwarders must shift to providing technologies that help shippers overcome supply chain visibility obstacles. The question is: should you develop your own solutions in-house, or partner with a third-party vendor?
This brief white paper will help you answer that question.
The document discusses how machine learning in manufacturing has focused on optimizing individual machines, but now needs to take the next step of analyzing data across entire production networks. Analyzing inventory, costs, machine capabilities, and more across all plants unlock opportunities to:
1) Determine optimal workflows and allow machines to run at slower rates when downstream processes are delayed
2) Enable a "gig economy" to deploy specialized labor more precisely as needed
3) Allow facilities to produce different product types more efficiently through a multi-modal model informed by cross-network data
4) Better share excess capacity or workload across all plants to optimize the entire supply network.
Hands-on with Apache Druid: Installation & Data Ingestion StepsservicesNitor
Supercharge your analytics workflow with https://bityl.co/Qcuk Apache Druid's real-time capabilities and seamless Kafka integration. Learn about it in just 14 steps.
Software Test Automation - A Comprehensive Guide on Automated Testing.pdfkalichargn70th171
Moving to a more digitally focused era, the importance of software is rapidly increasing. Software tools are crucial for upgrading life standards, enhancing business prospects, and making a smart world. The smooth and fail-proof functioning of the software is very critical, as a large number of people are dependent on them.
Case Study: Increasing Produban's Critical Systems Availability and PerformanceCA Technologies
The Santander Group is a Spanish banking group and the largest bank in the Eurozone by market value. It is also one of the largest banks in the world in terms of market capitalization. Produban is Santander’s group company responsible for Santander's entire IT infrastructure. The Produban challenge was to monitor - proactively and in real time - all transactions running in critical systems and being able to take action before major problems happen. Considering this scenario, Produban adopted CA Core APM (CA Introscope) in order to count with alerts that permit to the technical team to detect problems before they impact business.
For more information on DevOps solutions from CA Technologies, please visit: http://bit.ly/1wbjjqX
Industry 4.0 promises to create new customer value in the market place by unleashing a combination of new technologies, data analytics, new generation cyber-physical production systems and newer methods of human machine interfaces. What does a developing country like India need to do to join the race?
This document discusses the NOGESI project, which used Simware software to create a distributed simulation laboratory for the Spanish Ministry of Defense. Some key points:
- NOGESI used Simware and DDS (Data Distribution Service) to improve runtime interoperability between simulators, addressing long-standing issues with interoperability.
- It demonstrated interoperability between simulated, operational, and legacy systems in a HLA (High Level Architecture) federation.
- The architecture was based on an open, layered approach as defined by SISO's LSA (Layered Simulation Architecture) concept, using DDS as the underlying middleware to improve upon limitations of HLA.
- For under $1 million
VEGA Process Measurement (Level, Limit Level & Pressure) - Oil & Gas OffshoreThorne & Derrick UK
The document discusses instrumentation solutions from VEGA for various applications in offshore oil and gas production, including level, density, and flow measurement of drilling mud. VEGA offers a full portfolio of measurement technologies including radar, ultrasonic, guided microwave, and capacitive sensors. Their plics modular instrument system allows for customized configurations and easy installation. The technologies and products discussed can provide reliable measurement under harsh offshore conditions to ensure safety and optimize production processes.
The document summarizes new autoclave models called the Pro Line and Prime Line from MELAG. Key points include:
- The new autoclaves offer improved performance, functions, and design at affordable prices compared to previous models.
- Features include fast cycle times, large loading capacity, intuitive touchscreen interface, energy saving modes, and connectivity for documentation.
- Accessories like trays, water treatment systems, and software allow users to customize configurations to suit their needs.
The document discusses Tripp Lite's NH modular UPS models and their advantages for data centers. The NH models offer high efficiency and flexibility. They allow for lower installation and operating costs compared to non-modular UPS systems. The modular design also provides increased availability, redundancy, and easy upgradability. The document provides guidance on how to effectively sell the NH UPS models to data centers and what technical specifications to highlight to customers.
There are three levels of process control for thermal profiling that have advanced manufacturing:
1) Traditional manual profiling uses test boards with thermocouples but is unreliable, labor intensive, and damages boards.
2) Oven verification devices like the DeltaProbe provide more accurate and repeatable profiling without test boards.
3) Continuous monitoring systems like the Automatic Profiling System (APS) profile every board and detect process faults in real-time, improving quality and reducing costs. The APS represents the most advanced profiling technology.
VEGA Pressure & Level Measurement - Paper Industry ApplicationsThorne & Derrick UK
VEGA provides measurement technology for the paper industry, including level, pressure, density, and point level devices. Their plics modular system allows customization of sensors, fittings, electronics, and housings. VEGA's devices are robust for the demanding conditions in paper processes like pulp cooking, bleaching towers, stock preparation, and more.
White Paper Mold-ID - Mold Management for Injection Molding with RFIDManfred Muenzl
Tools are subject to wear and tear and must be maintained and serviced on a regular base.
Inspection of the tools is often dependent on the experience values of individual employees, handwritten notes or planned figures in the ERP system that are not available to everyone or that are not monitored.
In many cases the maintenance or inspection is only carried out when the produced parts no longer fulfill the required quality standards or if the mold fails at its service.
That causes unplanned downtime of the machine and is not acceptable.
The document discusses how MOSAICO's Intelligent Platform transforms operations in lime plants through digital solutions. It introduces MOSAICO, an industrial internet platform that collects operational data and provides visualization, analytics, and reporting to optimize asset performance and operations. The platform was implemented on a QualiCal kiln for Calcis Group to reduce downtime, optimize maintenance costs, and introduce smart maintenance through the ZERO App. This allows maintenance managers to monitor plant performance remotely and technicians to access maintenance documentation and provide feedback from mobile devices. The results have been more efficient maintenance and increased plant availability through minimized unplanned stoppages.
With digital solutions transforming and imporving operations in lime plants, the MOSAICO intelligent platform has proven to be a game changer for maintenance and operatios
The article looks at how new technologies will lead to an increasingly integrated approach within the O&G sector, siting specifics such as the IoT and robotics & the radical impact they will have on optimising productions within the sector.
Network performance - skilled craft to hard scienceMartin Geddes
This document describes the technical and business journey for network operators wanting to turn network performance from a skilled craft into hard science.
This document discusses Industry 4.0 and how KEOPS helps manufacturers with digital transformation through their manufacturing intelligence solutions. It summarizes that KEOPS:
1) Helps maximize productivity across the manufacturing process from design to distribution through data collection, analysis, and visualization.
2) Allows manufacturers to transition from mass production to customized production by reducing time-to-market.
3) Integrates external supply chains to connect suppliers and customers.
4) Gathers and analyzes performance data to increase productivity, quality, and profits.
In 2017, the World Economic Forum recognized the potential of advanced manufacturing technologies. In 2018, from among more than 1,000 examined production facilities, 16
companies were recognized as Fourth Industrial Revolution leaders in advanced manufacturing for demonstrating step-change results, both operational and financial, across individual sites. They had succeeded in scaling beyond the pilot phase and their sites were designated advanced manufacturing “Lighthouses”. In 2019, 28 additional facilities were identified and added to the network, which now provides an opportunity for cross-company learning and collaboration, and for setting new benchmarks for the global manufacturing community.
Lighthouses have succeeded by innovating new operating systems, including in how they manage and optimize business and processes, transforming the way people work and use technology. These new operating systems can become the blueprint for modernizing the entire company operating system; therefore, how they prepare for scaling up and engaging the workforce matters.
Virtual Human Brain Simulations with Abaqus in the CloudThe UberCloud
UberCloud, Dassault Systèmes Simulia and Advania Data Centers presentation about the award winning project: HPC Cloud Simulation of Neuromodulation in Schizophrenia. Learn how simulation and high performance computing in the cloud play a key role in accelerating personalized healthcare.
The Brain Neuromodulation project represents a breakthrough in demonstrating the high value of computational modeling and simulation in improving the clinical application of non-invasive electro-stimulation of the human brain in schizophrenia and the potential to apply this technology to the treatment of other neuropsychiatric disorders such as depression and Parkinson’s disease. With the addition of HPC, clinicians can now precisely and non-invasively target regions of the brain without disrupting nearby healthy brain regions.
2018 Hyperion HPC Innovation Excellence Award: UberCloud, the National Institute of Mental Health & Neuro Sciences (NIMHANS) in Bangalore, Dassault Systèmes Simulia, Advania Data Centers, Hewlett Packard Enterprise and Intel won the 2018 Hyperion HPC Innovation Excellence Award for their Neuromodulation Project, based on computer simulations of non-invasive transcranial electro-stimulation of the human brain in schizophrenia.
The document discusses the evolution of the digital oilfield concept from early data gathering to modern integrated operations enabled by technology advances. It provides an overview of key developments like early logging data transmission and downhole sensors. The modern digital oilfield uses real-time data in continuous optimization cycles and allows centralized decision making. Integrated operations aim to improve effectiveness through cross-discipline collaboration. An example is provided in the Kuwait Intelligent Digital Field project.
Gravity White Paper - How to Close the 3rd Party Logistics Technology GapAero Wong
Tactical, transactional-based services are becoming less profitable. In order to compete, 3PLs and freight forwarders must shift to providing technologies that help shippers overcome supply chain visibility obstacles. The question is: should you develop your own solutions in-house, or partner with a third-party vendor?
This brief white paper will help you answer that question.
The document discusses how machine learning in manufacturing has focused on optimizing individual machines, but now needs to take the next step of analyzing data across entire production networks. Analyzing inventory, costs, machine capabilities, and more across all plants unlock opportunities to:
1) Determine optimal workflows and allow machines to run at slower rates when downstream processes are delayed
2) Enable a "gig economy" to deploy specialized labor more precisely as needed
3) Allow facilities to produce different product types more efficiently through a multi-modal model informed by cross-network data
4) Better share excess capacity or workload across all plants to optimize the entire supply network.
Hands-on with Apache Druid: Installation & Data Ingestion StepsservicesNitor
Supercharge your analytics workflow with https://bityl.co/Qcuk Apache Druid's real-time capabilities and seamless Kafka integration. Learn about it in just 14 steps.
Software Test Automation - A Comprehensive Guide on Automated Testing.pdfkalichargn70th171
Moving to a more digitally focused era, the importance of software is rapidly increasing. Software tools are crucial for upgrading life standards, enhancing business prospects, and making a smart world. The smooth and fail-proof functioning of the software is very critical, as a large number of people are dependent on them.
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSISTier1 app
Are you ready to unlock the secrets hidden within Java thread dumps? Join us for a hands-on session where we'll delve into effective troubleshooting patterns to swiftly identify the root causes of production problems. Discover the right tools, techniques, and best practices while exploring *real-world case studies of major outages* in Fortune 500 enterprises. Engage in interactive lab exercises where you'll have the opportunity to troubleshoot thread dumps and uncover performance issues firsthand. Join us and become a master of Java thread dump analysis!
Nashik's top web development company, Upturn India Technologies, crafts innovative digital solutions for your success. Partner with us and achieve your goals
🏎️Tech Transformation: DevOps Insights from the Experts 👩💻campbellclarkson
Connect with fellow Trailblazers, learn from industry experts Glenda Thomson (Salesforce, Principal Technical Architect) and Will Dinn (Judo Bank, Salesforce Development Lead), and discover how to harness DevOps tools with Salesforce.
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdfBaha Majid
IBM watsonx Code Assistant for Z, our latest Generative AI-assisted mainframe application modernization solution. Mainframe (IBM Z) application modernization is a topic that every mainframe client is addressing to various degrees today, driven largely from digital transformation. With generative AI comes the opportunity to reimagine the mainframe application modernization experience. Infusing generative AI will enable speed and trust, help de-risk, and lower total costs associated with heavy-lifting application modernization initiatives. This document provides an overview of the IBM watsonx Code Assistant for Z which uses the power of generative AI to make it easier for developers to selectively modernize COBOL business services while maintaining mainframe qualities of service.
Flutter vs. React Native: A Detailed Comparison for App Development in 2024dhavalvaghelanectarb
Choosing the right framework for your cross-platform mobile app can be a tough decision. Both Flutter and React Native offer compelling features and have earned their place in the development world. Here is a detailed comparison to help you weigh their strengths and weaknesses. Here are the pros and cons of developing mobile apps in React Native vs Flutter.
What is Continuous Testing in DevOps - A Definitive Guide.pdfkalichargn70th171
Once an overlooked aspect, continuous testing has become indispensable for enterprises striving to accelerate application delivery and reduce business impacts. According to a Statista report, 31.3% of global enterprises have embraced continuous integration and deployment within their DevOps, signaling a pervasive trend toward hastening release cycles.
Just like life, our code must adapt to the ever changing world we live in. From one day coding for the web, to the next for our tablets or APIs or for running serverless applications. Multi-runtime development is the future of coding, the future is to be dynamic. Let us introduce you to BoxLang.
Strengthening Web Development with CommandBox 6: Seamless Transition and Scal...Ortus Solutions, Corp
Join us for a session exploring CommandBox 6’s smooth website transition and efficient deployment. CommandBox revolutionizes web development, simplifying tasks across Linux, Windows, and Mac platforms. Gain insights and practical tips to enhance your development workflow.
Come join us for an enlightening session where we delve into the smooth transition of current websites and the efficient deployment of new ones using CommandBox 6. CommandBox has revolutionized web development, consistently introducing user-friendly enhancements that catalyze progress in the field. During this presentation, we’ll explore CommandBox’s rich history and showcase its unmatched capabilities within the realm of ColdFusion, covering both major variations.
The journey of CommandBox has been one of continuous innovation, constantly pushing boundaries to simplify and optimize development processes. Regardless of whether you’re working on Linux, Windows, or Mac platforms, CommandBox empowers developers to streamline tasks with unparalleled ease.
In our session, we’ll illustrate the simple process of transitioning existing websites to CommandBox 6, highlighting its intuitive features and seamless integration. Moreover, we’ll unveil the potential for effortlessly deploying multiple websites, demonstrating CommandBox’s versatility and adaptability.
Join us on this journey through the evolution of web development, guided by the transformative power of CommandBox 6. Gain invaluable insights, practical tips, and firsthand experiences that will enhance your development workflow and embolden your projects.
How GenAI Can Improve Supplier Performance Management.pdfZycus
Data Collection and Analysis with GenAI enables organizations to gather, analyze, and visualize vast amounts of supplier data, identifying key performance indicators and trends. Predictive analytics forecast future supplier performance, mitigating risks and seizing opportunities. Supplier segmentation allows for tailored management strategies, optimizing resource allocation. Automated scorecards and reporting provide real-time insights, enhancing transparency and tracking progress. Collaboration is fostered through GenAI-powered platforms, driving continuous improvement. NLP analyzes unstructured feedback, uncovering deeper insights into supplier relationships. Simulation and scenario planning tools anticipate supply chain disruptions, supporting informed decision-making. Integration with existing systems enhances data accuracy and consistency. McKinsey estimates GenAI could deliver $2.6 trillion to $4.4 trillion in economic benefits annually across industries, revolutionizing procurement processes and delivering significant ROI.
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...kalichargn70th171
Visual testing plays a vital role in ensuring that software products meet the aesthetic requirements specified by clients in functional and non-functional specifications. In today's highly competitive digital landscape, users expect a seamless and visually appealing online experience. Visual testing, also known as automated UI testing or visual regression testing, verifies the accuracy of the visual elements that users interact with.
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...The Third Creative Media
"Navigating Invideo: A Comprehensive Guide" is an essential resource for anyone looking to master Invideo, an AI-powered video creation tool. This guide provides step-by-step instructions, helpful tips, and comparisons with other AI video creators. Whether you're a beginner or an experienced video editor, you'll find valuable insights to enhance your video projects and bring your creative ideas to life.
These are the slides of the presentation given during the Q2 2024 Virtual VictoriaMetrics Meetup. View the recording here: https://www.youtube.com/watch?v=hzlMA_Ae9_4&t=206s
Topics covered:
1. What is VictoriaLogs
Open source database for logs
● Easy to setup and operate - just a single executable with sane default configs
● Works great with both structured and plaintext logs
● Uses up to 30x less RAM and up to 15x disk space than Elasticsearch
● Provides simple yet powerful query language for logs - LogsQL
2. Improved querying HTTP API
3. Data ingestion via Syslog protocol
* Automatic parsing of Syslog fields
* Supported transports:
○ UDP
○ TCP
○ TCP+TLS
* Gzip and deflate compression support
* Ability to configure distinct TCP and UDP ports with distinct settings
* Automatic log streams with (hostname, app_name, app_id) fields
4. LogsQL improvements
● Filtering shorthands
● week_range and day_range filters
● Limiters
● Log analytics
● Data extraction and transformation
● Additional filtering
● Sorting
5. VictoriaLogs Roadmap
● Accept logs via OpenTelemetry protocol
● VMUI improvements based on HTTP querying API
● Improve Grafana plugin for VictoriaLogs -
https://github.com/VictoriaMetrics/victorialogs-datasource
● Cluster version
○ Try single-node VictoriaLogs - it can replace 30-node Elasticsearch cluster in production
● Transparent historical data migration to object storage
○ Try single-node VictoriaLogs with persistent volumes - it compresses 1TB of production logs from
Kubernetes to 20GB
● See https://docs.victoriametrics.com/victorialogs/roadmap/
Try it out: https://victoriametrics.com/products/victorialogs/
Ensuring Efficiency and Speed with Practical Solutions for Clinical OperationsOnePlan Solutions
Clinical operations professionals encounter unique challenges. Balancing regulatory requirements, tight timelines, and the need for cross-functional collaboration can create significant internal pressures. Our upcoming webinar will introduce key strategies and tools to streamline and enhance clinical development processes, helping you overcome these challenges.
3. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- Configuration & rollout strategy
- Retention
- Replication
- Consumer lag
- Batching & compression
3
KAFKA IN PRODUCTION: CONSIDERATIONS
4. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- Kafka configuration is highly reliant on the use case
and application needs.
- Having a set strategy for rolling out the changes to
the cluster without stopping the service is vital.
- There is no perfect configuration from the get-go,
there are many parameters to fine tune.
- Having a clear performance goal and agile ways to
roll out the changes will make your life a lot easier.
4
KAFKA IN PRODUCTION: CONFIGURATION & ROLLOUT
5. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- It’s important to configure data retention in your
kafka cluster, tailored to the needs of the
application.
- Retention can be configured with a retention time
and a retention volume.
- This space vs time is really important to fine tune
- Have a red alert button! Whenever something goes
wrong invalidate the retention policies in order to fix
the issue without losing data.
- Pro tip: Kafka supports time travel from 0.10.1
onwards!
5
KAFKA IN PRODUCTION: RETENTION
6. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- Have the right amount of replication depending on
the application needs and data sensitivity.
- Too much replication will lead to unnecessary costs
and complexity.
- Too little replication won’t let your SysAdmins sleep.
- Keep the replicated data in separate failure
domains.
- Be careful with the hardware infrastructure, disk
I/Os get exponentially high with replication.
6
KAFKA IN PRODUCTION: REPLICATION
7. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
7
KAFKA IN PRODUCTION: REPLICATION
8. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- Consumer lag is one of the scariest problems when
dealing with kafka infrastructure.
- If not detected and the retention policies kick in,
you’ll start losing your data before having it
processed.
- Monitoring append lag vs commit lag is important to
get an accurate diagnostic of the causes of the lag.
- Append lag is the most sensitive to be monitoring.
8
KAFKA IN PRODUCTION: CONSUMER LAG
9. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
9
KAFKA IN PRODUCTION: CONSUMER LAG
10. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- Some use cases can benefit greatly from batchin
strategies.
- The batching can happen in the producer, the
consumer, or both of them.
- Producer batching stresses the resources of the
kafka machine but lowers the total network and I/O
requirements.
- Consumer batching lowers the resources on the
consumer application at zookeeper peak resource
consumption cost
10
KAFKA IN PRODUCTION: BATCHING
11. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
11
KAFKA IN PRODUCTION: BATCHING
12. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- Kafka implementation varies greatly depending on
whether one wants to keep the events as an
immutable state or delete them periodically.
- When building an application on event sourcing or
using kafka as a data bus to connect microservices,
event consistency is key.
- If, on the other hand, one is dealing with pipelining
of high throughput of data, scaling and order
management is a greater concern.
12
EVENT SOURCING & EVENT BASING
13. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- Event versioning comes into play here. Since the events are persisted
permanently in the queue, one needs to be able to read them at all times. Avro’s
schema registry is the best tool available to handle the versioning of events and
entities present in them.
- Don’t trust the queue, rebuild constantly. This not only ensures the consistency of
one’s schemas and events, but also allows for cool DevOps application
deployment strategies, like zero downtime database clusterization or migrations
of application.
- One needs to be really careful when scaling the cluster and relying on event
sourcing, specially when dealing with partitions of topics.
- Thankfully, this implementation generally has the lowest load on the cluster
compared to pipelining high traffic through it, therefore kafka setup and resource
management becomes easier.
13
EVENT SOURCING
14. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- Infrastructure isn’t the only point of consideration, since event sourcing is an end-
to end effort.
- Clients and APIs need to be tailored for eventual consistency.
- For the client side, local management of the state through a store (specially when
persisting changes) can prevent a large number of fraudulent events.
- On the API side of things, making the required comprobations both before
producing the event and right before persisting the data when consuming it may
prevent undesired exception handling.
- Specially when scaling producers and partitions, it is really important to maintain
bounded context on an entity level on the same pipeline. You don’t want events
being produced on one partition that depend on entities being modified on
another one, leading to data corruption on the consumer level.
- Soft deletes help a lot conserving data integrity when dealing with event-sourced
microservice implementations: The service may appear faulty to the end-user if a
bug is found, but all data is recoverable without needing to completely rebuild a
queue if a flaw is detected.
14
EVENT SOURCING
15. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- When using kafka as a pipeline of data processing, loads can increase greatly.
Such cases are real-time event tracking, monitoring, process pipelines, etc.
- In this cases, a lot of the times one bases the queue structure in batches of
processing, making it easier to handle.
- Scaling partitions is key here: in most use cases one will find a high load of events
of the same type, and handling the scaling of the pipelining of said events and the
consumption is the focus when dealing with performance and optimization.
- Data sensitivity is generally lower in this implementations, therefore the need for
event versioning and producing assurance isn’t as critical compared to event
sourcing.
15
EVENT BASING
17. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- One of the least concerning implementations in the
early stages of kafka.
- Only important for really big clusters or for multi-
cluster support.
- Removes a single point of failure for the application.
- Allows for smaller machines to deploy a Zookeeper
instance
17
SCALING ZOOKEEPER
18. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
Pros:
- Increases the highest throughput of the queue.
- Enables data resiliency with multiples copies of the
same partition spread across brokers.
- Permits fault tolerance to the system, depending on
the amount of nodes said tolerance may vary.
Cons:
- Increases complexity of the system
- Increases operational deployment cost
- Increases system monetary cost
18
SCALING KAFKA
19. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- It’s really important to handle partition assignment
when scaling consumers on your application (more
on this later).
- There are two major strategies when scaling
consumers, competing consumers and
publish/subscribe.
- If handling consumer assignment manually, one
needs to be really careful when mixing both
strategies to avoid data loss.
19
SCALING CONSUMERS
20. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- Consumers subscribed to a same topic in a
consumer group are competing consumers.
- Each of those receives messages from one or more
partitions of the topic
- This allows to scale the number of consumers of a
topic up to the number of partitions for said topic.
- Extra consumers will remain idle until another one
fails or more partitions of the topic are created.
20
SCALING CONSUMERS: Competing consumers
21. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- This patterns separates consumers by consumer
groups, and subscribes each consumer group to all
the messages of a single topic.
- In a single consumer group one will find the same
competing pattern explained before, but all the
messages are being sent to all the groups.
- Especially useful for microservice orchestration and
data sharing, since one can assign a single consumer
group per microservice and handle the needed
events for said service there.
21
SCALING CONSUMERS: publish/subscribe
22. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- There are two ways to connect a consumer to a partition of a
certain topic. The subscribe() and the assign() methods (as
per Kafka API).
- The subscribe() one assigns the consumer group to a topic
and lets handle the consumer assignation and rebalancing by
itself.
- On the other hand, assign()ing a consumer to a specific
partitions makes it manual, therefore increasing the risk of
missing partitions and/or trying to overlap multiple
consumers on the same partition. Be really careful when
going manual on consumer assignation!
22
SCALING CONSUMERS: careful when going manual!
23. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- By default, a kafka producer will write in any of the partitions of the topic where
it’s producing. Depending on the sending strategy the producer has a buffer of
events to be sent before the last one has been validated.
- Producer acknowledgements (acks) present a strategy for confirmation of an
event being persisted to the queue. This can be set to 0 (NONE), 1 (LEADER) or -1
(ALL)
- If the strategy chosen is too restrictive (-1) or the cluster’s brokers have trouble
keeping up with the producer’s throughput, one may experience higher memory
size of the producer’s buffer, leading to unexpected crashes and/or data loss.
- On the other hand if the strategy is too loose (0 or 1), a broker failure my imply
data loss.
- Batching strategies help with restrictive acks and faster processing of the events,
since the producer can keep building the next batch while the previous one is
being acknowledged by the cluster.
- Batch compression is also an option when the producers are overwhelming the
network of the cluster but the processing of events is still being handled properly.
23
SCALING PRODUCERS
24. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- If the load of the application is high enough to warrant
multiple instances, you’ll need partitioning of your data.
- Whenever the application allows it, random data distribution
is the most efficient way to scale partitions
- It’s important to consider whether you’ll need to make
aggregates, guarantee order, shard the data or batching
when deciding the partitioning strategy
24
PARTITIONING STRATEGIES
25. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- Makes no differentiation on which partition handles which
kind of event.
- Makes consumer scaling easier, since any consumer can
consume from any partition.
- Doesn’t ensure any kind of ordering of consumption outside
a single partition, and therefore event type.
25
RANDOM PARTITIONING
26. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- Each partition handles a certain type of event.
- Consumer scaling gets trickier on a single consumer group.
- Ensures order preservation on a single event type, but adds
design complexity to the events that may be interconnected.
- Also adds complexity to the consumers, but ensuring that
every consumer can process any event makes it easier to
handle.
26
AGGREGATE PARTITIONING
27. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- If the aggregate partition strategy isn’t homogeneous (some aggregates have
more load than others) the partitions themselves will face different loads,
makings consumer scaling harder.
- You can then separate the highest load partitions with time windows, spreading
the highest load across different partitions
27
TIME WINDOW PARTITIONING
28. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- Once the chunking of the partitions is in place, one can consume said events on a
time window basis.
- Then, produce in a new sorted topic, partitioning each event in their aggregates.
28
TIME WINDOW PARTITIONING
29. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- When choosing a partitioning strategy it’s important to take into consideration
the possible resource bottlenecks outside of the kafka cluster.
Example: If a consumer of a topic is dependant on a high load database that has been
sharded, it makes sense to set the topic partitions to match said shards of the db. This
allows to scale the consumers per partition and database shard.
- When dealing with multiple partitions and replication, storage considerations are
really important. If a broker fails and there is replication in place, the partition
leader may change and the replication may move to another broker, creating high
traffic and/or disk I/Os.
29
PARTITIONING: BOTTLENECKS AND EFFICIENCY
30. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- When a consumer enters or leaves a consumer group, kafka by default rebalances
the partitions for said consumer group.
- When rebalancing happens, all consumers drop their partitions and are
reassigned new ones. If the consumer has state associated to the data being
consumer, you need to be very careful with the rebalancing strategies of the
cluster.
- Another option is use the native Kafka API instead of a consumer group, and
manually assign consumers to partitions (avoiding automatic load balancing).
30
PARTITION REBALANCING
31. Oficinas en Madrid: C/ Francisco Silvela, 54 Duplicado 1ºD 28028
Telf: 91 080 82 44
Oficinas en Barcelona: C/ Madrazo 27-29 4ª 08006
Telf: 933 68 52 46
- As stated before, data resiliency and replication is one of the biggest strengths of
kafka.
- Although it adds load in both disk I/Os and network it ensures that no data loss
will happen.
- A good starting point for data replication is at 3 replicas per partition. This allows
the cluster to lose one broker without critical alert, and two of them without
losing any data!
- This way if a single broker fails at night a single notification would suffice and you
can fix the problem the next morning. Otherwise, if two of them fail you can still
fix the issue without service downtime and/or data loss.
- We’ll talk about data spreading and levels of fault tolerance depending on the
type of infrastructure when we evaluate different production environments.
31
DATA RESILIENCY & FAULT TOLERANCE