Using galera replication to create geo distributed clusters on the wanSakari Keskitalo
We will show the advantages of having a geo-distributed database cluster and how to create one using Galera Cluster for MySQL. We will also discuss the configuration and status variables that are involved and how to deal with typical situations on the WAN such as slow, untrusted or unreliable links, latency and packet loss. We will demonstrate a multi-region cluster on Amazon EC2 and perform some throughput and latency measurements in real-time.
How Criteo is managing one of the largest Kafka Infrastructure in EuropeRicardo Paiva
In Criteo we manage one of the largest Kafka infrastructure in Europe, with more than 7 million msgs/sec. This talk was first presented on the Kafka Meetup Paris, in January of 2019.
Linked In Stream Processing Meetup - Apache PulsarKarthik Ramasamy
Apache Pulsar is the next generation messaging system that uses a fundamentally different architecture to achieve durability, performance, scalability, efficiency, multi-tenancy and geo replication.
Introducing Confluent labs Parallel Consumer client | Anthony Stubbes, ConfluentHostedbyConfluent
Consuming messages in parallel is what Apache Kafka® is all about, so you may well wonder, why would we want anything else? It turns out that, in practice, there are a number of situations where Kafka’s partition-level parallelism gets in the way of optimal design.
This session will go over some of these types of situations that can benefit from parallel message processing within a single application instance (aka slow consumers or competing consumers), and then introduce the new Parallel Consumer labs project from Confluent, which can improve functionality and massively improve performance in such situations.
It will cover the
- Different ordering modes of the client
- Relative performance improvements
- Usage with other components like Kafka Streams
- An introduction to the internal architecture of the project
- How it can achieve all this in a reassignment friendly manner
Better Kafka Performance Without Changing Any Code | Simon Ritter, AzulHostedbyConfluent
Apache Kafka is the most popular open-source stream-processing software for collecting, processing, storing, and analyzing data at scale. Most known for its excellent performance, low latency, fault tolerance, and high throughput, it's capable of handling thousands of messages per second. For mission-critical applications, how do you ensure that the performance delivered is the performance required? This is especially important as Kafka is written in Java and Scala and runs on the JVM. The JVM is a fantastic platform that delivers on an internet scale. In this session, we'll explore how making changes to the JVM design can eliminate the problems of garbage collection pauses and raise the throughput of applications. For cloud-based Kafka applications, this can deliver both lower latency and reduced infrastructure costs. All without changing a line of code!
The much anticipated release of Galera Cluster 4 for MySQL 8 is now Generally Available. Please join Codership, the developers of Galera Cluster, and learn how we improve MySQL High Availability with the new features in Galera Cluster 4, and how you can benefit from using them. We will also give you an idea of the Galera 4 short term road map, as well as an overview of Galera 4 in MariaDB, MySQL and Percona.
Learn about how you can load data faster with streaming replication, how to use the new system tables in the mysql database, how your application can benefit from the new synchronization functions, and how Galera Cluster is now so much more robust in handling a bad network for Geo-distributed Multi-master MySQL. We will go through a quick install of a 3-node Galera Cluster as well.
Watch this talk here: https://www.confluent.io/online-talks/apache-kafka-architecture-and-fundamentals-explained-on-demand
This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput.
This talk provides a comprehensive overview of Kafka architecture and internal functions, including:
-Topics, partitions and segments
-The commit log and streams
-Brokers and broker replication
-Producer basics
-Consumers, consumer groups and offsets
This session is part 2 of 4 in our Fundamentals for Apache Kafka series.
Using galera replication to create geo distributed clusters on the wanSakari Keskitalo
We will show the advantages of having a geo-distributed database cluster and how to create one using Galera Cluster for MySQL. We will also discuss the configuration and status variables that are involved and how to deal with typical situations on the WAN such as slow, untrusted or unreliable links, latency and packet loss. We will demonstrate a multi-region cluster on Amazon EC2 and perform some throughput and latency measurements in real-time.
How Criteo is managing one of the largest Kafka Infrastructure in EuropeRicardo Paiva
In Criteo we manage one of the largest Kafka infrastructure in Europe, with more than 7 million msgs/sec. This talk was first presented on the Kafka Meetup Paris, in January of 2019.
Linked In Stream Processing Meetup - Apache PulsarKarthik Ramasamy
Apache Pulsar is the next generation messaging system that uses a fundamentally different architecture to achieve durability, performance, scalability, efficiency, multi-tenancy and geo replication.
Introducing Confluent labs Parallel Consumer client | Anthony Stubbes, ConfluentHostedbyConfluent
Consuming messages in parallel is what Apache Kafka® is all about, so you may well wonder, why would we want anything else? It turns out that, in practice, there are a number of situations where Kafka’s partition-level parallelism gets in the way of optimal design.
This session will go over some of these types of situations that can benefit from parallel message processing within a single application instance (aka slow consumers or competing consumers), and then introduce the new Parallel Consumer labs project from Confluent, which can improve functionality and massively improve performance in such situations.
It will cover the
- Different ordering modes of the client
- Relative performance improvements
- Usage with other components like Kafka Streams
- An introduction to the internal architecture of the project
- How it can achieve all this in a reassignment friendly manner
Better Kafka Performance Without Changing Any Code | Simon Ritter, AzulHostedbyConfluent
Apache Kafka is the most popular open-source stream-processing software for collecting, processing, storing, and analyzing data at scale. Most known for its excellent performance, low latency, fault tolerance, and high throughput, it's capable of handling thousands of messages per second. For mission-critical applications, how do you ensure that the performance delivered is the performance required? This is especially important as Kafka is written in Java and Scala and runs on the JVM. The JVM is a fantastic platform that delivers on an internet scale. In this session, we'll explore how making changes to the JVM design can eliminate the problems of garbage collection pauses and raise the throughput of applications. For cloud-based Kafka applications, this can deliver both lower latency and reduced infrastructure costs. All without changing a line of code!
The much anticipated release of Galera Cluster 4 for MySQL 8 is now Generally Available. Please join Codership, the developers of Galera Cluster, and learn how we improve MySQL High Availability with the new features in Galera Cluster 4, and how you can benefit from using them. We will also give you an idea of the Galera 4 short term road map, as well as an overview of Galera 4 in MariaDB, MySQL and Percona.
Learn about how you can load data faster with streaming replication, how to use the new system tables in the mysql database, how your application can benefit from the new synchronization functions, and how Galera Cluster is now so much more robust in handling a bad network for Geo-distributed Multi-master MySQL. We will go through a quick install of a 3-node Galera Cluster as well.
Watch this talk here: https://www.confluent.io/online-talks/apache-kafka-architecture-and-fundamentals-explained-on-demand
This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput.
This talk provides a comprehensive overview of Kafka architecture and internal functions, including:
-Topics, partitions and segments
-The commit log and streams
-Brokers and broker replication
-Producer basics
-Consumers, consumer groups and offsets
This session is part 2 of 4 in our Fundamentals for Apache Kafka series.
Using OVSDB and OpenFlow southbound pluginsOpenDaylight
Southbound plugins are essential for programming your network with OpenDaylight. In this meetup, we will discuss the plugins for OpenFlow and OVSDB, as well as the differences in writing applications with MD-SAL and AD-SAL. We will do bite-sized hands-on exercises to learn how to use the two plugins.
Pulsar - flexible pub-sub for internet scaleMatteo Merli
Pub-Sub messaging is a very convenient abstraction that allows system and application developers to decouple components and let them communicate, by acting as durable buffer for transient data, or as a persistent log from where to recover after crashes. This talk will present an overview of Apache Pulsar, the reasons that led to its development and how it enabled many teams at Yahoo and to build scalable and reliable applications. Apache Pulsar has become the defacto pub-sub messaging at Yahoo serving 100+ applications and processing 100’s of billions of messages for over 3+ years.
In this talk, we will explore in detail different categories of use cases that highlight how Pulsar can be applied to solve a broad range of problems thanks to its flexible messaging model that supports both queuing and streaming semantics with a focus on durability and transaction guarantees.
Asynchronous Transaction Processing With Kafka as a Single Source of Truth - ...HostedbyConfluent
Our backend system (ERP, CRM, Billing) is completely cloud, asynchronous Microservices and Kafka based. No databases at all. In this environment most challenging part is transaction management with error handling and compensation. Especially for bus tickets mobile application backend that has to respond fast. This short talk explains how we overcome this challenges and what are some of the solutions we implemented.
Monitoring Large-scale Cloud Infrastructures with OpenNebulaNETWAYS
Efficient monitoring is crucial when managing your Cloud infrastructure. The metrics collected by OpenNebula can be used to trigger automatic scaling, or quickly detect failures to automatically restart virtual machines. During this talk, I will show how OpenNebula can be used to efficiently monitor thousands of virtual machines at sub-1 minute interval. I will show how OpenNebula can be enhanced and optimized, and how different metrics collection tools such as Ganglia and Host-sFlow can be used with OpenNebula to monitor large-scale Cloud infrastructures.
Load balancing In cloud - In a semi distributed systemAchal Gupta
Load Balancing in Cloud
What is load balancing in Cloud in semi distributed system and why it is better than a centralized system and distributed system
Matteo Merli and Sijie Guo from Streamlio gave a hands-on workshop on Apache Pulsar. #fast #durable #pubsub #messaging system. A low latency alternative to #kafka.
There are many Galera Cluster distributions and sometimes differences are well worth noting. We get a lot of queries about which Galera Cluster to use, or why one should use one distribution over the other.
Learn about Galera Cluster with MySQL 5.7 from Codership, and we’ll compare it with Galera Cluster 4 with MariaDB 10.4, and Percona XtraDB Cluster 5.7 with Galera 3. This is also the webinar where we preview Galera Cluster 4 with MySQL 8.0 as well as compare it with the preview release of Percona XtraDB Cluster 8.0.
Overall, learn why distributions exists, and how you can get the most out of your Galera Cluster experience.
PLNOG 13: Michał Dubiel: OpenContrail software architecturePROIDEA
Michał Dubiel – TBD
Topic of Presentation: OpenContrail software architecture
Language: Polish
Abstract:
OpenContrail is a complete solution for Software Defined Networking (SDN). Its relatively new approach to network virtualization in data centers utilizes the overlay networking technology in order to achieve full decoupling of the physical infrastructure from the tenant’s logical configurations.
This presentation describes the software architecture of the system and its functional partitioning. A special emphasis is put on a compute node components: the vRouter kernel module and the vRouter Agent. Also, selected implementation details are presented in greater details along with an analysis of their impact on an overall system’s exceptional scalability and great performance.
Task Context.
Context Switching between tasks.
Shared Data Problem.
Non Reentrant Function.
Reentrant Function.
Gray area of reentrancy.
How to protect Shared Data?
Blockchain:
------------
Well-ordered collections of block
Consensus driven
Maintains history, order of transactions
Secure, trustless, immutable
Major enabler for DApps
Sequential processing, block size
Can be slow for high volume transactions
Wasteful, costly
Pegged Sidechain:
--------------------
A Blockchain that validates data from other main Blockchain.
Allows tokens from one Blockchain to be securely in a separate Blockchain.
If required blocks can be moved back.
Helps to extend functionality through interoperable networks allowing decentralised processing.
Mainly help address performance and scalability concerns by enabling parallel processing of blocks.
Allows different mining algorithms, different smart contract, block size
Two-way Peg is a mechanism by which coins are transferred between sidechains and back. Two kids of mechanism either a symmetric or asymmetric, depending on if SPV-Proof is used.
Introducción a Stream Processing utilizando Kafka Streamsconfluent
Matías Cascallares, Confluent, Customer Success Architect
Streams es uno de los keywords de moda! En esta presentación, veremos cómo implementar stream processing con Kafka Streams, que consideraciones tenemos que tener en cuenta, y un pequeño tour por ksqlDB como herramienta.
https://www.meetup.com/Mexico-Kafka/events/276717476/
A Practical Guide to Post-EMV Card Not Present FraudForter
Experts predict that EMV adoption in the US will set off a tsunami of card not present fraud. Are you leveraging everything you've got in preparation for the oncoming threat?
Find out in this Practical Guide.
Using OVSDB and OpenFlow southbound pluginsOpenDaylight
Southbound plugins are essential for programming your network with OpenDaylight. In this meetup, we will discuss the plugins for OpenFlow and OVSDB, as well as the differences in writing applications with MD-SAL and AD-SAL. We will do bite-sized hands-on exercises to learn how to use the two plugins.
Pulsar - flexible pub-sub for internet scaleMatteo Merli
Pub-Sub messaging is a very convenient abstraction that allows system and application developers to decouple components and let them communicate, by acting as durable buffer for transient data, or as a persistent log from where to recover after crashes. This talk will present an overview of Apache Pulsar, the reasons that led to its development and how it enabled many teams at Yahoo and to build scalable and reliable applications. Apache Pulsar has become the defacto pub-sub messaging at Yahoo serving 100+ applications and processing 100’s of billions of messages for over 3+ years.
In this talk, we will explore in detail different categories of use cases that highlight how Pulsar can be applied to solve a broad range of problems thanks to its flexible messaging model that supports both queuing and streaming semantics with a focus on durability and transaction guarantees.
Asynchronous Transaction Processing With Kafka as a Single Source of Truth - ...HostedbyConfluent
Our backend system (ERP, CRM, Billing) is completely cloud, asynchronous Microservices and Kafka based. No databases at all. In this environment most challenging part is transaction management with error handling and compensation. Especially for bus tickets mobile application backend that has to respond fast. This short talk explains how we overcome this challenges and what are some of the solutions we implemented.
Monitoring Large-scale Cloud Infrastructures with OpenNebulaNETWAYS
Efficient monitoring is crucial when managing your Cloud infrastructure. The metrics collected by OpenNebula can be used to trigger automatic scaling, or quickly detect failures to automatically restart virtual machines. During this talk, I will show how OpenNebula can be used to efficiently monitor thousands of virtual machines at sub-1 minute interval. I will show how OpenNebula can be enhanced and optimized, and how different metrics collection tools such as Ganglia and Host-sFlow can be used with OpenNebula to monitor large-scale Cloud infrastructures.
Load balancing In cloud - In a semi distributed systemAchal Gupta
Load Balancing in Cloud
What is load balancing in Cloud in semi distributed system and why it is better than a centralized system and distributed system
Matteo Merli and Sijie Guo from Streamlio gave a hands-on workshop on Apache Pulsar. #fast #durable #pubsub #messaging system. A low latency alternative to #kafka.
There are many Galera Cluster distributions and sometimes differences are well worth noting. We get a lot of queries about which Galera Cluster to use, or why one should use one distribution over the other.
Learn about Galera Cluster with MySQL 5.7 from Codership, and we’ll compare it with Galera Cluster 4 with MariaDB 10.4, and Percona XtraDB Cluster 5.7 with Galera 3. This is also the webinar where we preview Galera Cluster 4 with MySQL 8.0 as well as compare it with the preview release of Percona XtraDB Cluster 8.0.
Overall, learn why distributions exists, and how you can get the most out of your Galera Cluster experience.
PLNOG 13: Michał Dubiel: OpenContrail software architecturePROIDEA
Michał Dubiel – TBD
Topic of Presentation: OpenContrail software architecture
Language: Polish
Abstract:
OpenContrail is a complete solution for Software Defined Networking (SDN). Its relatively new approach to network virtualization in data centers utilizes the overlay networking technology in order to achieve full decoupling of the physical infrastructure from the tenant’s logical configurations.
This presentation describes the software architecture of the system and its functional partitioning. A special emphasis is put on a compute node components: the vRouter kernel module and the vRouter Agent. Also, selected implementation details are presented in greater details along with an analysis of their impact on an overall system’s exceptional scalability and great performance.
Task Context.
Context Switching between tasks.
Shared Data Problem.
Non Reentrant Function.
Reentrant Function.
Gray area of reentrancy.
How to protect Shared Data?
Blockchain:
------------
Well-ordered collections of block
Consensus driven
Maintains history, order of transactions
Secure, trustless, immutable
Major enabler for DApps
Sequential processing, block size
Can be slow for high volume transactions
Wasteful, costly
Pegged Sidechain:
--------------------
A Blockchain that validates data from other main Blockchain.
Allows tokens from one Blockchain to be securely in a separate Blockchain.
If required blocks can be moved back.
Helps to extend functionality through interoperable networks allowing decentralised processing.
Mainly help address performance and scalability concerns by enabling parallel processing of blocks.
Allows different mining algorithms, different smart contract, block size
Two-way Peg is a mechanism by which coins are transferred between sidechains and back. Two kids of mechanism either a symmetric or asymmetric, depending on if SPV-Proof is used.
Introducción a Stream Processing utilizando Kafka Streamsconfluent
Matías Cascallares, Confluent, Customer Success Architect
Streams es uno de los keywords de moda! En esta presentación, veremos cómo implementar stream processing con Kafka Streams, que consideraciones tenemos que tener en cuenta, y un pequeño tour por ksqlDB como herramienta.
https://www.meetup.com/Mexico-Kafka/events/276717476/
A Practical Guide to Post-EMV Card Not Present FraudForter
Experts predict that EMV adoption in the US will set off a tsunami of card not present fraud. Are you leveraging everything you've got in preparation for the oncoming threat?
Find out in this Practical Guide.
Slides from Ian Forsey and Ariel Kogan's session at Skill Matter's Scala Exchange 2013.
-------------------------------------------------
In this session we will share our experience at Net-a-porter, creating our first reactive Scala/Akka/Spray service in a company with a long-standing Java codebase and production infrastructure.
We've heard how Twitter and LinkedIn adopted Scala on greenfield initiatives and we're excited to use a more expressive language running on a robust, familiar VM. But is the ecosystem ready to support the demands of a long-established enterprise infrastructure, mission-critical (non-Scala!) middleware and the traditional dev-test-release workflow?
We'll start by exposing what drove our decision to dive into Scala. Next: We'll talk about some of the challenges we faced designing, building, load testing and debugging our service. We will discuss some of the patterns we used moving to a more reactive platform, the availability/maturity of the tooling and some of the framework code we had to write. Finally we'll outline the benefits gained through embarking on this project and any prices we have paid for doing so.
Orchestration tool roundup - OpenStack Israel summit - kubernetes vs. docker...Uri Cohen
It’s no news that containers represent a portable unit of deployment, and OpenStack has proven an ideal environment for running container workloads. However, where it usually becomes more complex is that many times an application is often built out of multiple containers. What’s more, setting up a cluster of container images can be fairly cumbersome because you need to make one container aware of another and expose intimate details that are required for them to communicate which is not trivial especially if they’re not on the same host.
These scenarios have instigated the demand for some kind of orchestrator. The list of container orchestrators is growing fairly fast. This session will compare the different orchestation projects out there - from Heat to Kubernetes to TOSCA - and help you choose the right tool for the job.
A presentation on what's the magic in LinkedIn? LinkedIn's DNA & recipe, tips, tricks and examples.
Presented at Sosa Tel Aviv, on 1/3/16 at the Israeli Marketing Innovation Forum conference - a LinkedIn event, with Tal Shmueli, Linkedin Marketing Solutions Account Manager, Tsur Shraibman, Linkedin Marketing Solutions Account Executive and Efrat Fenigson, Senior Director of Marketing Communications at Viaccess-Orca (Orange group)
Introduction to Scrum development process. Main concepts. The process. Examples. Issues
Slides for course Software engineering seminar @ Afeka College Of Engineering
"The joy of Scala" - Maxim Novak / Wix
Around eight years ago I started my journey as a developer. Since then, I've played around with many languages and thought that C# offers the best developer productivity. After joining Wix two years ago, I was exposed to the amazing world of Scala and Functional Programming and never looked back.
In Scala the code is much more concise, less ceremonious, immutable by default, combines functional with object oriented, seamlessly interoperates with Java, and many software engineering patterns are already baked into the language. Most importantly - Scala is FUN! By the end of the session you too will, hopefully, convert to Scala and never look back.
Recording of the lecture (Hebrew) - https://youtu.be/TcnYTwff2xU
How does Google work? How can a friend user her own computer to enter a Website you developed on your own machine? Where are your Facebook posts saved once you exit the browser?
In this talk we will learn the general ideas behind the Internet, what the main components of a Web application are and what happens from the moment you open the browser, enter an address and until you can see the Website with all the relevant data.
This talk was given at the she codes; Google Campus branch.
The lecture recording is available here: https://youtu.be/qys1rsBRhUs
Lessons we learned while getting Wonderball Heroes on WebGL using Unity 5.
The slides share our challenges, optimizations made and general tips for working with Unity and WebGL.
How fast ist it really? Benchmarking in practiceTobias Pfeiffer
“What’s the fastest way of doing this?” - you might ask yourself during development. Sure, you can guess what’s fastest or how long something will take, but do you know? How long does it take to sort a list of 1 Million elements? Are tail-recursive functions always the fastest?
Benchmarking is here to answer these questions. However, there are many pitfalls around setting up a good benchmark and interpreting the results. This talk will guide you through, introduce best practices and show you some surprising benchmarking results along the way.
Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and ...Gilad Garon
VMware’s Common SaaS Platform (CSP) is a brand new offering designed to enhance the productivity of developers and cloud providers by equipping them with a set of common and configurable capabilities (such as Identity, Telemetry, Account Management, Billing etc.), thus enabling them to focus on their core businesses.
But enough with the product pitch.
CSP is distributed to numerous cloud providers around the globe, used by developers and IT alike to empower their services and better answer the business need of their customers.
Please join us and witness how we take continuous delivery to the next step where sometimes the target environment is not on our control and still seamlessly manage and deliver our unique collection of capabilities, packaged as platform for ease of use, using the best and shiniest tools the frogs can provide.
Optimizing DevOps strategy in a large enterpriseEyal Edri
Large enterprises today are pacing a flood of multiple devops tools to choose from for their infrastructure. The problem intensifies when you have dozens of devops teams across the world, each with his own background of devops tools and knowledge and each with his own agenda of pushing to use his tools. How would you leverage this distributed, disconnected knowledge into a single working devops knowledge source, and common infrastructure to leverage the whole enterprise? Come and hear about Red Hat Global CI initiative to hear on one possible approach for taking on the battle.
Jay Kreps on Project Voldemort Scaling Simple Storage At LinkedInLinkedIn
Jay Kreps on Project Voldemort Scaling Simple Storage At LinkedIn. This was a presentation made at QCon 2009 and is embedded on LinkedIn's blog - http://blog.linkedin.com/
From cars, to thermostats, through media players and embedded controllers, devices are being connected to the Internet at a furious pace. This session will discuss and demonstrate and coding practices from live Azure customers.
Hadoop Ecosystem and Low Latency Streaming ArchitectureInSemble
"Hadoop Ecosystem and Low Latency Streaming Architecture" was presented by Vijay Mandava and Lan Jiang to Detroit Java User Group on 3/23/2015. It covers the basic introduction of Hadoop Ecosystem and then focus on the low latency streaming architecture, including frameworks such as Flume, Kafka and Storm.
The Power of Determinism in Database SystemsDaniel Abadi
Slides for Daniel Abadi talk at UC Berkeley on 10/22/2014. Discusses the problems with traditional database systems, especially around modularity and horizontal scalability, and shows how deterministic database systems can help.
Journey to Blockchain Scalability: A Close Look at Complete Scaling Solutions...Zeeve
In this webinar, Ravi Chamria, the CEO and Co-founder of Zeeve, dives deep into the topic of blockchain scalability. Discover the complete scaling solutions for L1 and L2 chains as Ravi breaks down the challenges and trade-offs associated with scalability, security, and decentralization.
Ravi gives an overview of the topics covered in this insightful webinar. Starting the webinar, he explains the basics of scalability in blockchain and introduces the Scalability Trilemma, which highlights the trade-offs developers face when designing a blockchain.
Explore the three parts of the Blockchain Scalability solutions at the beginning, which include L1 Solutions (On-Chain), L2 Solutions (Off-Chain), and App-Chains.
This tutorial gives out an brief and interesting introduction to modern stream computing technologies. The participants can learn the essential concepts and methodologies for designing and building a advanced stream processing system. The tutorial unveils the key fundamentals behind various kinds of design choices. Some forecast of technology developments in this domain is also introduced at the last section of this tutorial.
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)MdTanvirMahtab2
This presentation is about the working procedure of Shahjalal Fertilizer Company Limited (SFCL). A Govt. owned Company of Bangladesh Chemical Industries Corporation under Ministry of Industries.
Hierarchical Digital Twin of a Naval Power SystemKerry Sado
A hierarchical digital twin of a Naval DC power system has been developed and experimentally verified. Similar to other state-of-the-art digital twins, this technology creates a digital replica of the physical system executed in real-time or faster, which can modify hardware controls. However, its advantage stems from distributing computational efforts by utilizing a hierarchical structure composed of lower-level digital twin blocks and a higher-level system digital twin. Each digital twin block is associated with a physical subsystem of the hardware and communicates with a singular system digital twin, which creates a system-level response. By extracting information from each level of the hierarchy, power system controls of the hardware were reconfigured autonomously. This hierarchical digital twin development offers several advantages over other digital twins, particularly in the field of naval power systems. The hierarchical structure allows for greater computational efficiency and scalability while the ability to autonomously reconfigure hardware controls offers increased flexibility and responsiveness. The hierarchical decomposition and models utilized were well aligned with the physical twin, as indicated by the maximum deviations between the developed digital twin hierarchy and the hardware.
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptxR&R Consult
CFD analysis is incredibly effective at solving mysteries and improving the performance of complex systems!
Here's a great example: At a large natural gas-fired power plant, where they use waste heat to generate steam and energy, they were puzzled that their boiler wasn't producing as much steam as expected.
R&R and Tetra Engineering Group Inc. were asked to solve the issue with reduced steam production.
An inspection had shown that a significant amount of hot flue gas was bypassing the boiler tubes, where the heat was supposed to be transferred.
R&R Consult conducted a CFD analysis, which revealed that 6.3% of the flue gas was bypassing the boiler tubes without transferring heat. The analysis also showed that the flue gas was instead being directed along the sides of the boiler and between the modules that were supposed to capture the heat. This was the cause of the reduced performance.
Based on our results, Tetra Engineering installed covering plates to reduce the bypass flow. This improved the boiler's performance and increased electricity production.
It is always satisfying when we can help solve complex challenges like this. Do your systems also need a check-up or optimization? Give us a call!
Work done in cooperation with James Malloy and David Moelling from Tetra Engineering.
More examples of our work https://www.r-r-consult.dk/en/cases-en/
About
Indigenized remote control interface card suitable for MAFI system CCR equipment. Compatible for IDM8000 CCR. Backplane mounted serial and TCP/Ethernet communication module for CCR remote access. IDM 8000 CCR remote control on serial and TCP protocol.
• Remote control: Parallel or serial interface.
• Compatible with MAFI CCR system.
• Compatible with IDM8000 CCR.
• Compatible with Backplane mount serial communication.
• Compatible with commercial and Defence aviation CCR system.
• Remote control system for accessing CCR and allied system over serial or TCP.
• Indigenized local Support/presence in India.
• Easy in configuration using DIP switches.
Technical Specifications
Indigenized remote control interface card suitable for MAFI system CCR equipment. Compatible for IDM8000 CCR. Backplane mounted serial and TCP/Ethernet communication module for CCR remote access. IDM 8000 CCR remote control on serial and TCP protocol.
Key Features
Indigenized remote control interface card suitable for MAFI system CCR equipment. Compatible for IDM8000 CCR. Backplane mounted serial and TCP/Ethernet communication module for CCR remote access. IDM 8000 CCR remote control on serial and TCP protocol.
• Remote control: Parallel or serial interface
• Compatible with MAFI CCR system
• Copatiable with IDM8000 CCR
• Compatible with Backplane mount serial communication.
• Compatible with commercial and Defence aviation CCR system.
• Remote control system for accessing CCR and allied system over serial or TCP.
• Indigenized local Support/presence in India.
Application
• Remote control: Parallel or serial interface.
• Compatible with MAFI CCR system.
• Compatible with IDM8000 CCR.
• Compatible with Backplane mount serial communication.
• Compatible with commercial and Defence aviation CCR system.
• Remote control system for accessing CCR and allied system over serial or TCP.
• Indigenized local Support/presence in India.
• Easy in configuration using DIP switches.
2. Forter
• We detect fraud
• A lot of data is collected
• New data can introduce new data sources
• At transaction time, we do our magic. Fast.
• We deny less
3. What’s Storm?
• Streaming/data-pipeline infrastructure
• What’s a pipeline?
• “Topology” driven flow, static
• Written over JVM and also supports Python and
Node.js
• Easy clustering
• Apache top level project, large community
4. Storm Lingo
• Tuples
• The basic data transfer object in storm. Basically a dictionary (key->val).
• Spouts
• Entry points into the pipe. This is where data comes from.
• Bolts
• Components that can transform and route tuples
• Joins
• Joins are where async branches of the topology meet and join
• Streams
• Streams allow for flow control in the topology
5. System challenges
• Latency should be determined by business needs -
flexible per customer (300ms - customers who just don’t
care)
• Data dependencies in decision part can get very complex
• Getting data can be slow, especially 3rd party
• Data scientists write in Python
• Should be scaleable, because we’re ever growing
• Should be very granularly monitored
6. Bird’s eye view
• Two systems:
• System 1: data prefetching & preparing
• System 2: decision engine, must have all
available data handy at TX time
7. System 1: high
throughput pipeline
• Stream Batching
• Prefetching / Preparing
• Common use case, lots of competitors
8. System 2: low latency
decision
• Dedicated everything
• Complex dependency graph
• Less common, fewer players
10. Cache and cache layering
• Storm constructs make it easy to tweak caches,
add enrichment steps transparently
• Different enrichment operations may require
different execution power
• Each operation can be replaced by a sub-topology
- layering of cache levels
• Field grouping allows the ability to maintain state in
components - local cache or otherwise
11.
12. Maintain a stored state
• Many events coming in, some cause a state to
change
• State of a working set is saved in memory
• New/old states are fetched from an external data
source
• Sate updates are saved immediately
• State machine is scalable - again, field grouping
13.
14. And the rest…
• Batching content for writing (Storm’s tick tuples)
• Aggregating events in memory
• Throttling/Circuit-breaking external calls
16. Unique Challenges
• Scaling. Resources need to be very dedicated,
parallelizing is bad
• Join logic is much stricter, with short timeouts
• Data validity is crucial for the stream routing
• Error handling
• Component graph is immense and hard to contain
mentally - especially considering the delicate time
window configurations.
17. Scalability
• Each topology is built to handle a fixed number of
parallel TXs. Storm’s max-spout-pending
• Each topology atomically polls a queue
• Trying to keep as much of the logic in the same
process to reduce network and serialization costs
• Latency is the only measure
18. Joining and errors
• Waiting is not an option
• Tick tuples no good, break the single
thread illusion
• Static topologies are easy to analyze and
edit in runtime, and intervene
• Fallback streams are an elegant solution
to the problem, preventing developers
from explicitly defining escape routes
• Also allow for “try->finally” semantics
19. Multilang
• Storm allows running bolt processes (shell-bolt)
with the builtin capability of communicating through
standard i/o
• Not hugely scalable, but works
• Implemented are: Node.js (our contribution) and
Python
• We use for legacy and to keep data scientists
happy
20. Data Validity
• Wrapping the bolts, we implemented contracts for
outputs
• Java POJOs with Hibernate Validator
• Contracts allow us “hard-typing” the links in the
topologies
• Also help minimize data flow, especially to shell-bolts
• Checkout storm-data-contracts on github
21. Managing Complexity
• Complexity of the data dependencies is maintained
by literally drawing it.
• Nimbus REST APIs offer access to the topology
layout
• Timing complexity reduced by synchronizing the
joins to a shared point-in-time. Still pretty complex.
• Proves better than our previous iterative solution
22. Monitoring
• Nimbus metrics give out averages - not
good enough
• Reimann used to efficiently monitor
latencies for every tuple in the system
• Inherent low latency monitoring issue:
CPU utilization monitoring
• More at Itai Frenkel’s lecture