This presentation is about Java performance and the most effective ways to work with Java memory, including memory saving techniques and overcoming of memory barriers. Moreover, it contains debunking of the most popular myths on speed boosting.
This presentation by Andrii Antilikatorov (Consultant, GlobalLogic) was delivered at GlobalLogic Java Conference #2 in Krakow on April 23, 2016.
Low latency microservices in java QCon New York 2016Peter Lawrey
In this talk we explore how Microservices and Trading System overlap and what they can learn from each other. In particular, how can we make microservices easy to test and performant. How can Trading System have shorter time to market and easier to maintain.
This presentation is about Java performance and the most effective ways to work with Java memory, including memory saving techniques and overcoming of memory barriers. Moreover, it contains debunking of the most popular myths on speed boosting.
This presentation by Andrii Antilikatorov (Consultant, GlobalLogic) was delivered at GlobalLogic Java Conference #2 in Krakow on April 23, 2016.
Low latency microservices in java QCon New York 2016Peter Lawrey
In this talk we explore how Microservices and Trading System overlap and what they can learn from each other. In particular, how can we make microservices easy to test and performant. How can Trading System have shorter time to market and easier to maintain.
In computing, ZFS is a combined file system and logical volume manager designed by Sun Microsystems, a subsidiary of Oracle Corporation. The features of ZFS include support for high storage capacities, integration of the concepts of file system and volume management, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, RAID-Z and native NFSv4 ACLs. Unlike traditional file systems, which reside on single devices and thus require a volume manager to use more than one device, ZFS file systems are built on top of virtual storage pools called zpools. A zpool is constructed of virtual devices (vdevs), which are themselves constructed of block devices: files, hard drive partitions, or entire drives, with the last being the recommended usage.[7] Thus, a vdev can be viewed as a group of hard drives. This means a zpool consists of one or more groups of drives.
In addition, pools can have hot spares to compensate for failing disks. In addition, ZFS supports both read and write caching, for which special devices can be used. Solid State Devices can be used for the L2ARC, or Level 2 ARC, speeding up read operations, while NVRAM buffered SLC memory can be boosted with supercapacitors to implement a fast, non-volatile write cache, improving synchronous writes. Finally, when mirroring, block devices can be grouped according to physical chassis, so that the filesystem can continue in the face of the failure of an entire chassis. Storage pool composition is not limited to similar devices but can consist of ad-hoc, heterogeneous collections of devices, which ZFS seamlessly pools together, subsequently doling out space to diverse file systems as needed. Arbitrary storage device types can be added to existing pools to expand their size at any time. The storage capacity of all vdevs is available to all of the file system instances in the zpool. A quota can be set to limit the amount of space a file system instance can occupy, and a reservation can be set to guarantee that space will be available to a file system instance.
Communication between Microservices is inherently unreliable. These integration points may produce cascading failures, slow responses, service outages. We will walk through stability patterns like timeouts, circuit breaker, bulkheads and discuss how they improve stability of Microservices.
We talk a lot about Galera Cluster being great for High Availability, but what about Disaster Recovery (DR)? Database outages can occur when you lose a data centre due to data center power outages or natural disaster, so why not plan appropriately in advance?
In this webinar, we will discuss the business considerations including achieving the highest possible uptime, analysis business impact as well as risk, focus on disaster recovery itself, as well as discussing various scenarios, from having no offsite data to having synchronous replication to another data centre.
This webinar will cover MySQL with Galera Cluster, as well as branches MariaDB Galera Cluster as well as Percona XtraDB Cluster (PXC). We will focus on architecture solutions, DR scenarios and have you on your way to success at the end of it.
P99 Pursuit: 8 Years of Battling P99 LatencyScyllaDB
Performance engineering is a Sisyphean hill climb for perfection. Those who climb the hill are hardly ever satisfied with the results. You should always ask yourself where the bottleneck is today and what’s holding you back. Great performance improves your software. It enables you to run fewer layers, manage 10x less machines, simplifies your stack, and more.
In this keynote session, ScyllaDB CEO Dor Laor will cover the principles for successful creation of projects like ScyllaDB, KVM, the Linux kernel and explain why they spurred his vision for the P99 CONF.
ProxySQL is a popular database proxy for MySQL/MariaDB servers. This focuses on the possible High availability options for ProxySQL and operations of inbuilt clustering feature in ProxySQL. This tech talk was presented at Mydbops Database Meetup on 27-04-2019 by Aakash M, Database Administrator with Mydbops and Vignesh Prabhu, Database Administrator with Mydbops.
Boosting I/O Performance with KVM io_uringShapeBlue
Storage performance is becoming much more important. KVM io_uring attempts to bring the I/O performance of a virtual machine on almost the same level of bare metal. Apache CloudStack has support for io_uring since version 4.16. Wido will show the difference in performance io_uring brings to the table.
Wido den Hollander is the CTO of CLouDinfra, an infrastructure company offering total Webhosting solutions. CLDIN provides datacenter, IP and virtualization services for the companies within TWS. Wido den Hollander is a PMC member of the Apache CloudStack Project and a Ceph expert. He started with CloudStack 9 years ago. What attracted his attention is the simplicity of CloudStack and the fact that it is an open-source solution. During the years Wido became a contributor, a PMC member and he was a VP of the project for a year. He is one of our most active members, who puts a lot of efforts to keep the project active and transform it into a turnkey solution for cloud builders.
-----------------------------------------
The CloudStack European User Group 2022 took place on 7th April. The day saw a virtual get together for the European CloudStack Community, hosting 265 attendees from 25 countries. The event hosted 10 sessions with from leading CloudStack experts, users and skilful engineers from the open-source world, which included: technical talks, user stories, new features and integrations presentations and more.
------------------------------------------
About CloudStack: https://cloudstack.apache.org/
PostgreSQL is one of the most advanced relational databases. It offers superb replication capabilities. The most important features are: Streaming replication, Point-In-Time-Recovery, advanced monitoring, etc.
Crimson: Ceph for the Age of NVMe and Persistent MemoryScyllaDB
Ceph is a mature open source software-defined storage solution that was created over a decade ago.
During that time new faster storage technologies have emerged including NVMe and Persistent memory.
The crimson project aim is to create a better Ceph OSD that is more well suited to those faster devices. The crimson OSD is built on the Seastar C++ framework and can leverage these devices by minimizing latency, cpu overhead, and cross-core communication. This talk will discuss the project design, our current status, and our future plans.
Wars of MySQL Cluster ( InnoDB Cluster VS Galera ) Mydbops
MySQL Clustering over InnoDB engines has grown a lot over the last decade. Galera began working with InnoDB early and then Group Replication came to the environment later, where the features are now rich and robust. This presentation offers a technical comparison of both of them.
Quarkus - a next-generation Kubernetes Native Java frameworkSVDevOps
For years, the client-server architecture has been the de-facto standard to build applications. But a major shift happened. The one model rules them all age is over. A new range of applications and architectures has emerged and impacts how code is written and how applications are deployed and executed. HTTP microservices, reactive applications, message-driven microservices, and serverless are now central players in modern systems.
Quarkus has been designed with this new world in mind and provides first-class support for these different paradigms. Developers using the Red Hat build of Quarkus can now choose between deploying natively compiled code or JVM-based code depending on an application’s needs. Natively compiled Quarkus applications are extremely fast and memory-efficient, making Quarkus a great choice for serverless and high-density cloud deployments.
Speakers
1) Shanna Chan, Senior Solutions Architect at Red Hat
2) Mark Baker, Senior Solutions Architect at Red Hat
Speaker Bios
Shanna Chan - Shanna is passionate about how open source solutions help others in their journey of application modernization and transformation of their business into cloud infrastructures. Her background includes application developments, DevOps, and architecting solutions for large enterprises. More about Shanna at http://linkedin.com/in/shanna-chan
Mark Baker - Mark’s experiences coalesce around solution /business architecture and leadership bringing together people in both post / pre-sales software projects bridging traditional legacy systems (i.e. Jakarta (JEE) MVC) with Cloud tolerant and Cloud native open source in the journey of modernization and transformation. More about Mark at http://linkedin.com/in/markwbaker-tsl
Seastore: Next Generation Backing Store for CephScyllaDB
Ceph is an open source distributed file system addressing file, block, and object storage use cases. Next generation storage devices require a change in strategy, so the community has been developing crimson-osd, an eventual replacement for ceph-osd intended to minimize cpu overhead and improve throughput and latency. Seastore is a new backing store for crimson-osd targeted at emerging storage technologies including persistent memory and ZNS devices.
This presentation provides an overview of the Dell PowerEdge R730xd server performance results with Red Hat Ceph Storage. It covers the advantages of using Red Hat Ceph Storage on Dell servers with their proven hardware components that provide high scalability, enhanced ROI cost benefits, and support of unstructured data.
Kafka on ZFS: Better Living Through Filesystems confluent
(Hugh O'Brien, Jet.com) Kafka Summit SF 2018
You’re doing disk IO wrong, let ZFS show you the way. ZFS on Linux is now stable. Say goodbye to JBOD, to directories in your reassignment plans, to unevenly used disks. Instead, have 8K Cloud IOPS for $25, SSD speed reads on spinning disks, in-kernel LZ4 compression and the smartest page cache on the planet. (Fear compactions no more!)
Learn how Jet’s Kafka clusters squeeze every drop of disk performance out of Azure, all completely transparent to Kafka.
-Striping cheap disks to maximize instance IOPS
-Block compression to reduce disk usage by ~80% (JSON data)
-Instance SSD as the secondary read cache (storing compressed data), eliminating >99% of disk reads and safe across host redeployments
-Upcoming features: Compressed blocks in memory, potentially quadrupling your page cache (RAM) for free
We’ll cover:
-Basic Principles
-Adapting ZFS for cloud instances (gotchas)
-Performance tuning for Kafka
-Benchmarks
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah
Apache Kafka is a new breed of messaging system built for the "big data" world. Coming out of LinkedIn (and donated to Apache), it is a distributed pub/sub system built in Scala. It has been an Apache TLP now for several months with the first Apache release imminent. Built for speed, scalability, and robustness, Kafka should definitely be one of the data tools you consider when designing distributed data-oriented applications.
The talk will cover a general overview of the project and technology, with some use cases, and a demo.
Introduction to memcached, a caching service designed for optimizing performance and scaling in the web stack, seen from perspective of MySQL/PHP users. Given for 2nd year students of professional bachelor in ICT at Kaho St. Lieven, Gent.
This presentation by Think Big principal Matt Cooke and Martin Oberhuber, Senior Data Scientist, discusses high frequency trading, requirements for success, and underlying architectures which may include Apache Spark.
Introduction to OpenHFT for Melbourne Java Users GroupPeter Lawrey
Updated Introduction to Chronicle
Added Introduction to SharedHashMap, an off heap map which is persisted and shared between processes.
http://openhft.net/
In computing, ZFS is a combined file system and logical volume manager designed by Sun Microsystems, a subsidiary of Oracle Corporation. The features of ZFS include support for high storage capacities, integration of the concepts of file system and volume management, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, RAID-Z and native NFSv4 ACLs. Unlike traditional file systems, which reside on single devices and thus require a volume manager to use more than one device, ZFS file systems are built on top of virtual storage pools called zpools. A zpool is constructed of virtual devices (vdevs), which are themselves constructed of block devices: files, hard drive partitions, or entire drives, with the last being the recommended usage.[7] Thus, a vdev can be viewed as a group of hard drives. This means a zpool consists of one or more groups of drives.
In addition, pools can have hot spares to compensate for failing disks. In addition, ZFS supports both read and write caching, for which special devices can be used. Solid State Devices can be used for the L2ARC, or Level 2 ARC, speeding up read operations, while NVRAM buffered SLC memory can be boosted with supercapacitors to implement a fast, non-volatile write cache, improving synchronous writes. Finally, when mirroring, block devices can be grouped according to physical chassis, so that the filesystem can continue in the face of the failure of an entire chassis. Storage pool composition is not limited to similar devices but can consist of ad-hoc, heterogeneous collections of devices, which ZFS seamlessly pools together, subsequently doling out space to diverse file systems as needed. Arbitrary storage device types can be added to existing pools to expand their size at any time. The storage capacity of all vdevs is available to all of the file system instances in the zpool. A quota can be set to limit the amount of space a file system instance can occupy, and a reservation can be set to guarantee that space will be available to a file system instance.
Communication between Microservices is inherently unreliable. These integration points may produce cascading failures, slow responses, service outages. We will walk through stability patterns like timeouts, circuit breaker, bulkheads and discuss how they improve stability of Microservices.
We talk a lot about Galera Cluster being great for High Availability, but what about Disaster Recovery (DR)? Database outages can occur when you lose a data centre due to data center power outages or natural disaster, so why not plan appropriately in advance?
In this webinar, we will discuss the business considerations including achieving the highest possible uptime, analysis business impact as well as risk, focus on disaster recovery itself, as well as discussing various scenarios, from having no offsite data to having synchronous replication to another data centre.
This webinar will cover MySQL with Galera Cluster, as well as branches MariaDB Galera Cluster as well as Percona XtraDB Cluster (PXC). We will focus on architecture solutions, DR scenarios and have you on your way to success at the end of it.
P99 Pursuit: 8 Years of Battling P99 LatencyScyllaDB
Performance engineering is a Sisyphean hill climb for perfection. Those who climb the hill are hardly ever satisfied with the results. You should always ask yourself where the bottleneck is today and what’s holding you back. Great performance improves your software. It enables you to run fewer layers, manage 10x less machines, simplifies your stack, and more.
In this keynote session, ScyllaDB CEO Dor Laor will cover the principles for successful creation of projects like ScyllaDB, KVM, the Linux kernel and explain why they spurred his vision for the P99 CONF.
ProxySQL is a popular database proxy for MySQL/MariaDB servers. This focuses on the possible High availability options for ProxySQL and operations of inbuilt clustering feature in ProxySQL. This tech talk was presented at Mydbops Database Meetup on 27-04-2019 by Aakash M, Database Administrator with Mydbops and Vignesh Prabhu, Database Administrator with Mydbops.
Boosting I/O Performance with KVM io_uringShapeBlue
Storage performance is becoming much more important. KVM io_uring attempts to bring the I/O performance of a virtual machine on almost the same level of bare metal. Apache CloudStack has support for io_uring since version 4.16. Wido will show the difference in performance io_uring brings to the table.
Wido den Hollander is the CTO of CLouDinfra, an infrastructure company offering total Webhosting solutions. CLDIN provides datacenter, IP and virtualization services for the companies within TWS. Wido den Hollander is a PMC member of the Apache CloudStack Project and a Ceph expert. He started with CloudStack 9 years ago. What attracted his attention is the simplicity of CloudStack and the fact that it is an open-source solution. During the years Wido became a contributor, a PMC member and he was a VP of the project for a year. He is one of our most active members, who puts a lot of efforts to keep the project active and transform it into a turnkey solution for cloud builders.
-----------------------------------------
The CloudStack European User Group 2022 took place on 7th April. The day saw a virtual get together for the European CloudStack Community, hosting 265 attendees from 25 countries. The event hosted 10 sessions with from leading CloudStack experts, users and skilful engineers from the open-source world, which included: technical talks, user stories, new features and integrations presentations and more.
------------------------------------------
About CloudStack: https://cloudstack.apache.org/
PostgreSQL is one of the most advanced relational databases. It offers superb replication capabilities. The most important features are: Streaming replication, Point-In-Time-Recovery, advanced monitoring, etc.
Crimson: Ceph for the Age of NVMe and Persistent MemoryScyllaDB
Ceph is a mature open source software-defined storage solution that was created over a decade ago.
During that time new faster storage technologies have emerged including NVMe and Persistent memory.
The crimson project aim is to create a better Ceph OSD that is more well suited to those faster devices. The crimson OSD is built on the Seastar C++ framework and can leverage these devices by minimizing latency, cpu overhead, and cross-core communication. This talk will discuss the project design, our current status, and our future plans.
Wars of MySQL Cluster ( InnoDB Cluster VS Galera ) Mydbops
MySQL Clustering over InnoDB engines has grown a lot over the last decade. Galera began working with InnoDB early and then Group Replication came to the environment later, where the features are now rich and robust. This presentation offers a technical comparison of both of them.
Quarkus - a next-generation Kubernetes Native Java frameworkSVDevOps
For years, the client-server architecture has been the de-facto standard to build applications. But a major shift happened. The one model rules them all age is over. A new range of applications and architectures has emerged and impacts how code is written and how applications are deployed and executed. HTTP microservices, reactive applications, message-driven microservices, and serverless are now central players in modern systems.
Quarkus has been designed with this new world in mind and provides first-class support for these different paradigms. Developers using the Red Hat build of Quarkus can now choose between deploying natively compiled code or JVM-based code depending on an application’s needs. Natively compiled Quarkus applications are extremely fast and memory-efficient, making Quarkus a great choice for serverless and high-density cloud deployments.
Speakers
1) Shanna Chan, Senior Solutions Architect at Red Hat
2) Mark Baker, Senior Solutions Architect at Red Hat
Speaker Bios
Shanna Chan - Shanna is passionate about how open source solutions help others in their journey of application modernization and transformation of their business into cloud infrastructures. Her background includes application developments, DevOps, and architecting solutions for large enterprises. More about Shanna at http://linkedin.com/in/shanna-chan
Mark Baker - Mark’s experiences coalesce around solution /business architecture and leadership bringing together people in both post / pre-sales software projects bridging traditional legacy systems (i.e. Jakarta (JEE) MVC) with Cloud tolerant and Cloud native open source in the journey of modernization and transformation. More about Mark at http://linkedin.com/in/markwbaker-tsl
Seastore: Next Generation Backing Store for CephScyllaDB
Ceph is an open source distributed file system addressing file, block, and object storage use cases. Next generation storage devices require a change in strategy, so the community has been developing crimson-osd, an eventual replacement for ceph-osd intended to minimize cpu overhead and improve throughput and latency. Seastore is a new backing store for crimson-osd targeted at emerging storage technologies including persistent memory and ZNS devices.
This presentation provides an overview of the Dell PowerEdge R730xd server performance results with Red Hat Ceph Storage. It covers the advantages of using Red Hat Ceph Storage on Dell servers with their proven hardware components that provide high scalability, enhanced ROI cost benefits, and support of unstructured data.
Kafka on ZFS: Better Living Through Filesystems confluent
(Hugh O'Brien, Jet.com) Kafka Summit SF 2018
You’re doing disk IO wrong, let ZFS show you the way. ZFS on Linux is now stable. Say goodbye to JBOD, to directories in your reassignment plans, to unevenly used disks. Instead, have 8K Cloud IOPS for $25, SSD speed reads on spinning disks, in-kernel LZ4 compression and the smartest page cache on the planet. (Fear compactions no more!)
Learn how Jet’s Kafka clusters squeeze every drop of disk performance out of Azure, all completely transparent to Kafka.
-Striping cheap disks to maximize instance IOPS
-Block compression to reduce disk usage by ~80% (JSON data)
-Instance SSD as the secondary read cache (storing compressed data), eliminating >99% of disk reads and safe across host redeployments
-Upcoming features: Compressed blocks in memory, potentially quadrupling your page cache (RAM) for free
We’ll cover:
-Basic Principles
-Adapting ZFS for cloud instances (gotchas)
-Performance tuning for Kafka
-Benchmarks
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah
Apache Kafka is a new breed of messaging system built for the "big data" world. Coming out of LinkedIn (and donated to Apache), it is a distributed pub/sub system built in Scala. It has been an Apache TLP now for several months with the first Apache release imminent. Built for speed, scalability, and robustness, Kafka should definitely be one of the data tools you consider when designing distributed data-oriented applications.
The talk will cover a general overview of the project and technology, with some use cases, and a demo.
Introduction to memcached, a caching service designed for optimizing performance and scaling in the web stack, seen from perspective of MySQL/PHP users. Given for 2nd year students of professional bachelor in ICT at Kaho St. Lieven, Gent.
This presentation by Think Big principal Matt Cooke and Martin Oberhuber, Senior Data Scientist, discusses high frequency trading, requirements for success, and underlying architectures which may include Apache Spark.
Introduction to OpenHFT for Melbourne Java Users GroupPeter Lawrey
Updated Introduction to Chronicle
Added Introduction to SharedHashMap, an off heap map which is persisted and shared between processes.
http://openhft.net/
How are systems in finance design for deterministic outcomes, and performance. What are the benefits and what is the performance you can achieve.
Included a demo you can download.
Learn how you can enjoy the developer productivity, low TCO, and unlimited scale of MongoDB as a tick database for capturing, analyzing, and taking advantage of opportunities in tick data. This presentation will illustrates how MongoDB can easily and quickly store variable data formats, like top and depth of book, multiple asset classes, and even news and social networking feeds. It will explore aggregating and analyzing tick data in real-time for automated trading or in batch for research and analysis and how auto-sharding enables MongoDB to scale with commodity hardware to satisfy unlimited storage and performance requirements.
Writing and testing high frequency trading engines in javaPeter Lawrey
JavaOne presentation of Writing and Testing High Frequency Trading Engines in Java. Talk looks at low latency trading, thread affinity, lock free code, ultra low garbage and low latency persistence and IPC.
Reactive Programming, Traits and Principles. What is Reactive, where does it come from, and what is it good for? How does it differ from event driven programming? It only functional?
Unless you have a problem which scales to many independent tasks easily e.g. web services, you may find that the best way to improve throughput is by reducing latency. This talk starts with Little's Law and it's consequences for high performance computing.
Streams and lambdas the good, the bad and the uglyPeter Lawrey
Based on a six month migration of C# code to Java 8, what is legacy lambda code likely to look like and what mistakes can be made.
Good use cases.
Bad use cases with solutions
Ugly use cases.
Responding rapidly when you have 100+ GB data sets in JavaPeter Lawrey
One way to speed up you application is to bring more of your data into memory. But how to do you handle hundreds of GB of data in a JVM and what tools can help you.
Mentions: Speedment, Azul, Terracotta, Hazelcast and Chronicle.
After migrating a three year old C# project to Java we ending up with a significant portion of legacy code using lambdas in Java. What was some of the good use cases, code which could be written better and the problems we had migrating from C#. At the end we look at the performance implications of using Lambdas.
Microservices for performance - GOTO Chicago 2016Peter Lawrey
How do Microservices and Trading Systems overlap?
How can one area learn from the other?
How can we test components of microservices?
Is there a library which helps us implement and test these services?
Real World MongoDB: Use Cases from Financial Services by Daniel RobertsMongoDB
Huge upheaval in the finance industry has led to a major strain on existing IT infrastructure and systems. New finance industry regulation has meant increased volume, velocity and variability of data. This coupled with cost pressures from the business has led these institutions to seek alternatives. In this session learn how FS companies are using MongoDB to solve their problems. The use cases are specific to FS but the patterns of usage - agility, scale, global distribution - will be applicable across many industries.
Движок LMDB — особенный чемпион / Юрьев Леонид (Петер-Сервис R&D)Ontico
Lightning Memory-Mapped Database (LMDB) представляет собой интересный, во многом уникальный движок базы данных класса Berkeley DB и Level DB с ребус-подобным исходным кодом. Будучи относительно малоизвестным, LMDB показывает ЧЕМПИОНСКУЮ производительность по чтению. Однако при интенсивной записи всё не так радужно…
Было ещё несколько проблем и недостатков, которые нам пришлось устранить, разбираясь в ребусах исходного кода. Доклад точно будет интересен разработчикам, интересующимся внутренностями баз данных или характеристиками отдельных движков.
Проект реализуется силами компании Петер-Сервис R&D, резидента Сколково, для применения в телеком-проектах федерального масштаба.
Информация о нашем проекте https://github.com/ReOpen/ReOpenLDAP/wiki, об исходной версии LMDB http://symas.com/mdb/.
MySQL 5.7 clustering: The developer perspectiveUlf Wendel
(Compiled from revised slides of previous presentations - skip if you know the old presentations)
A summary on clustering MySQL 5.7 with focus on the PHP clients view and the PHP driver. Which kinds on MySQL clusters are there, what are their goal, how does wich one scale, what extra work does which clustering technique put at the client and finally, how the PHP driver (PECL/mysqlnd_ms) helps you.
Netflix Open Source Meetup Season 4 Episode 2aspyker
In this episode, we will take a close look at 2 different approaches to high-throughput/low-latency data stores, developed by Netflix.
The first, EVCache, is a battle-tested distributed memcached-backed data store, optimized for the cloud. You will also hear about the road ahead for EVCache it evolves into an L1/L2 cache over RAM and SSDs.
The second, Dynomite, is a framework to make any non-distributed data-store, distributed. Netflix's first implementation of Dynomite is based on Redis.
Come learn about the products' features and hear from Thomson and Reuters, Diego Pacheco from Ilegra and other third party speakers, internal and external to Netflix, on how these products fit in their stack and roadmap.
Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...Виталий Стародубцев
##Что такое Storage Replica
##Архитектура и сценарии
##Синхронная и асинхронная репликация
##Междисковая, межсерверная, внутрикластерная и межкластерная репликация
##Дизайн и проектирование Storage Replica
##Нововведения в Windows Server 2016 TP5
##Графический интерфейс управления, и другие возможности - демонстрация и планы развития
##Интеграция Storage Replica с Storage Spaces Direct
Apache Kafka - Scalable Message-Processing and more !Guido Schmutz
ndependent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. How can me make sure that all these event are accepted and forwarded in an efficient and reliable way? This is where Apache Kafaka comes into play, a distirbuted, highly-scalable messaging broker, build for exchanging huge amount of messages between a source and a target.
This session will start with an introduction into Apache and presents the role of Apache Kafka in a modern data / information architecture and the advantages it brings to the table. Additionally the Kafka ecosystem will be covered as well as the integration of Kafka in the Oracle Stack, with products such as Golden Gate, Service Bus and Oracle Stream Analytics all being able to act as a Kafka consumer or producer.
In search of the perfect IoT Stack - Scalable IoT Architectures with MQTTDominik Obermaier
Web-scale Internet of Things applications have one thing in common: They produce and process massive amounts of data. But how to design the next-generation IoT backend that is able to meet the business requirements and doesn’t explode as soon as the traffic increases? This presentation will cover how to use MQTT to connect millions of devices with commodity servers and process huge amounts of data. Learn all the common design patterns and see the technologies that actually scale. Explore when to use Cassandra, Kafka, Spark, Docker, and other tools and when to stick with your good ol’ SQL database or Enterprise Message Queue.
What are some of the performance implications of using lambdas and what strategies can be used to address these. When might be want an alternative to using a lambda and how can we design our APIs to be flexible in this regard. What are the principles of writing low latency code in Java? How do we tune and optimize our code for low latency? When don’t we optimize our code? Where does the JVM help and where does it get in our way? How does this apply to lambdas? How can we design our APIs to use lambdas and minimize garbage?
Big Data Streams Architectures. Why? What? How?Anton Nazaruk
With a current zoo of technologies and different ways of their interaction it's a big challenge to architect a system (or adopt existed one) that will conform to low-latency BigData analysis requirements. Apache Kafka and Kappa Architecture in particular take more and more attention over classic Hadoop-centric technologies stack. New Consumer API put significant boost in this direction. Microservices-based streaming processing and new Kafka Streams tend to be a synergy in BigData world.
Anton Moldovan "Building an efficient replication system for thousands of ter...Fwdays
For one of our projects, we needed to improve the current content delivery system for terminals. In this talk, I will share our experience in building an efficient data replication system for thousands of terminals. We will touch on architecture decisions and tradeoffs, technologies that we used, and a bit of load testing.
Spoiler: We didn't use Kafka.
Extreme is rethinking the data plane, the control plane, and the management plane. Extreme is a better mouse trap which delivers new features, advanced function, and wire-speed performance. Our switches deliver deterministic performance independent of load or what features are enabled. All Extreme Switches are based on XOS, the industries first and only truly modular operating system. Having a modular OS provides higher availability of critical network resources. By isolating each critical process in its own protected memory space, a single failed process can not take down the entire switch. Application modules can be loaded and unloaded without the need for rebooting the switch. This is the level of functionality that users expect on other technology. Reaching the twenty million port milestone is a significant achievement demonstrating how our highly effective network solutions, with rich features, innovative software and integrated support for secure convergence. VoIP/Unified Communica Fons/Infrastructure/SIP Trunking (SBC) – Because of strong ROI, investment in this segment remains on a very strong growth trajectory.
Enterprises depend on modular switching solutions for all aspects of the enterprise network: in the enterprise core and data center, the distribution layer that lies between the core and wiring closet, and in the wiring closet itself. Modular solutions provide port diversity and density that fixed solutions simply cannot match. There are also high-capacity modular solutions that only the largest of enterprises and institutions use for high-density and high-speed deployments. Modular solutions are generally much more expensive than their fixed cousins, especially in situations where density or flexibility are not required. Fixed-configuration stackable switches are typically cost- optimized, but they offer no real port diversity on an individual switch. Port diversity means the availability of different port types, such as fiber versus copper ports. Stackable switches have gotten better at offering port diversity, but they still cannot match their modular cousins. Many of these products now offer high-end features such as 802.3af PoE, QoS, and multi-layer intelligence that were only found on modular switches in the past. This is due to the proliferation of third-party merchant silicon in the fixed configuration market. Generally, a stack of fixed configuration switches can be managed as a single virtual entity. Fixed configuration switches generally cannot be used to provision an entire large enterprise, but instead are mostly used out at the edge or departmental level as a low-cost alternative to modular products.
Assumptions:
Ethernet is Open
Active/Active in the Fabric
Therefore:
Open at the Edge
Active/Active at the edge
Gruter TECHDAY 2014 Realtime Processing in TelcoGruter
Big Telco, Bigger real-time demands: Real-time processing in Telco
- Presented by Jung-ryong Lee, engineer manager at SK Telecom at Gruter TECHDAY 2014 Oct.29 Seoul, Korea
Have you heard that all in-memory databases are equally fast but unreliable, inconsistent and expensive? This session highlights in-memory technology that busts all those myths.
Redis, the fastest database on the planet, is not a simply in-memory key-value data-store; but rather a rich in-memory data-structure engine that serves the world’s most popular apps. Redis Labs’ unique clustering technology enables Redis to be highly reliable, keeping every data byte intact despite hundreds of cloud instance failures and dozens of complete data-center outages. It delivers full CP system characteristics at high performance. And with the latest Redis on Flash technology, Redis Labs achieves close to in-memory performance at 70% lower operational costs. Learn about the best uses of in-memory computing to accelerate everyday applications such as high volume transactions, real time analytics, IoT data ingestion and more.
Similar to High Frequency Trading and NoSQL database (20)
Software Engineering, Software Consulting, Tech Lead.
Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Security,
Spring Transaction, Spring MVC,
Log4j, REST/SOAP WEB-SERVICES.
Unleash Unlimited Potential with One-Time Purchase
BoxLang is more than just a language; it's a community. By choosing a Visionary License, you're not just investing in your success, you're actively contributing to the ongoing development and support of BoxLang.
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
Large Language Models and the End of ProgrammingMatt Welsh
Talk by Matt Welsh at Craft Conference 2024 on the impact that Large Language Models will have on the future of software development. In this talk, I discuss the ways in which LLMs will impact the software industry, from replacing human software developers with AI, to replacing conventional software with models that perform reasoning, computation, and problem-solving.
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamtakuyayamamoto1800
In this slide, we show the simulation example and the way to compile this solver.
In this solver, the Helmholtz equation can be solved by helmholtzFoam. Also, the Helmholtz equation with uniformly dispersed bubbles can be simulated by helmholtzBubbleFoam.
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Shahin Sheidaei
Games are powerful teaching tools, fostering hands-on engagement and fun. But they require careful consideration to succeed. Join me to explore factors in running and selecting games, ensuring they serve as effective teaching tools. Learn to maintain focus on learning objectives while playing, and how to measure the ROI of gaming in education. Discover strategies for pitching gaming to leadership. This session offers insights, tips, and examples for coaches, team leads, and enterprise leaders seeking to teach from simple to complex concepts.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
How Recreation Management Software Can Streamline Your Operations.pptxwottaspaceseo
Recreation management software streamlines operations by automating key tasks such as scheduling, registration, and payment processing, reducing manual workload and errors. It provides centralized management of facilities, classes, and events, ensuring efficient resource allocation and facility usage. The software offers user-friendly online portals for easy access to bookings and program information, enhancing customer experience. Real-time reporting and data analytics deliver insights into attendance and preferences, aiding in strategic decision-making. Additionally, effective communication tools keep participants and staff informed with timely updates. Overall, recreation management software enhances efficiency, improves service delivery, and boosts customer satisfaction.
Into the Box Keynote Day 2: Unveiling amazing updates and announcements for modern CFML developers! Get ready for exciting releases and updates on Ortus tools and products. Stay tuned for cutting-edge innovations designed to boost your productivity.
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
2. Agenda
Who are we?
Brief introduction to OpenHFT.
What does a typical trading system look like
What requirements do these systems have
OpenHFT performance.
3. Who are we
Higher Frequency Trading is a small consulting
and software development house specialising
in
Low latency, high throughput software
8 developers in Europe and USA.
Sponsor HFT related open source projects
Core Java engineering
4. Who am I?
Peter Lawrey
- CEO and Principal Consultant
- 3rd on Stackoverflow for Java,
most Java Performance answers.
- Founder of the Performance Java User's Group
- An Australian, based in the U.K.
5. What is our OSS
Key OpenHFT projects
Chronicle, low latency logging, event store and
IPC. (record / log everything)
HugeCollections, cross process embedded
persisted data stores. (only need the latest)
Millions of operations per second.
Micro-second latency.
6. With other NoSQL databases
Uses with NoSQL
Off heap cache of data from a DB.
Low latency queue for persisting to a DB
Map which replicates only the latest values.
Supports very high update rates by only
replicating the latest value.
7. What is HFT?
No standard definition.
Trading faster than a human can see.
Being fast can make the difference between
making and losing money.
For different systems this means typical
latencies of between
− 10 micro-seconds and
− 10 milli-second.
(Latencies external to the provider)
8. Time scales every developer
should know.
Operation Latency In human terms
L1 Cache hit 1 ns A blink of an eye (~20 ms)
L2 Cache hit 3 ns Noticeable flicker
L3 Cache hit 10 – 20 ns Time to say “A”
Main memory 70 – 100 ns Time to say a ten word sentence
Signal down a 200m
fibre cable
1 μsec One slide (speaking quickly)
SSD access 5 – 25 μsec Time to reheat a meal (3 mins)
HDD access 8 msec Time to flight around the world. (1.8
days)
Network packet from
Germany to the USA
45 msec Waiting for a 7 working day delivery
10. Event driven processing
Trading system use event driven processing to
minimise latency in a system.
Any data needed should already be loaded in
memory, not go off to a slow SQL database.
Each input event triggers a response, unless
there is a need to limit the output.
11. Critical Path
A trading system is designed around the critical
path. This has to be as short in terms of
latency as possible.
Critical path has a tight latency budget which
excludes many traditional databases.
Even the number of network hops can be
minimised.
Non critical path can use tradition databases
12. Critical Path databases
Time Series databases
− Kdb, kona
− InfluxDB
− OpenTSDB
Designed for millions of writes per second.
Column based database => 100 Million
operations per second e.g. sum a column.
14. Critical Path data store
HFT strategies are;
described using graphs.
handle events in real time ~10 – 100 μsec.
cache state rather than query a database.
all custom written libraries AFAIK.
15. Critical Path data store
Logging is performed by appending to memory
mapped files.
OpenHFT's Java Chronicle makes this easier to
do in Java in a GC-free, off heap, lock less
way.
Such low level coding is relatively easy in C or
C++.
19. Reliable persistence
Trades and Orders are high value data and less
voluminous than Market data or strategy
results.
Typically SQL Database.
Sometimes multiple databases for different
applications.
20. Why use more exotic database?
Mostly for high throughput.
− Million per second in one node.
Often for low latency.
− Latencies well below a milli-second.
21. Why wouldn't you use exotic DB
Not easy to learn, high knowledge investment.
(!R)@&{&/x!/:2_!x}'!R
Often harder to use.
− Less management tools.
− Not designed to work with web applications.
More sensitive to the details of the hardware
and what else is running on the same
machine.
22. Low latency at high throughput
Java Chronicle is designed as a low latency
logger and IPC.
At one million small messages per second
Almost zero garbage
Latency between processes around 1 micro-
second
Concurrent readers and writers
Supports bursts of 10 million messages/sec.
23. Chronicle and replication
Replication is point to point (TCP)
Server A records an event
– replicates to Server B
Server B reads local copy
– B processes the event
Server B stores the result.
– replicates to Server A
Server A replies.
Round trip
25 micro-seconds
99% of the time
GC-free
Lock less
Off heap
Unbounded
25. HugeCollections and throughput
SharedHashMap tested on a machine with 128
GB, 16 cores, 32 threads.
String keys, 64-bit long values.
10 million key-values updated at 37 M/s
500 million key-values updated at 23 M/s
On tmpfs, 2.5 billion key-values at 26 M/s
26. HugeCollections and latency
For a Map of small key-values (both 64-bit longs)
With an update rate of 1 M/s, one thread.
Percentile 100K
entries
1 M entries 10 M entries
50% (typical) 0.1 μsec 0.2 μsec 0.2 μsec
90% (worst 1 in 10) 0.4 μsec 0.5 μsec 0.5 μsec
99% (worst 1 in 100) 4.4 μsec 5.5 μsec 7 μsec
99.9% 9 μsec 10 μsec 10 μsec
99.99% 10 μsec 12 μsec 13 μsec
worst 24 μsec 29 μsec 26 μsec
27. Bonus topic: Units
A peak times an application writes 49 “mb/s” to a
disk which supports 50 “mb/s” and is replicated
over a 100 “mb/s” network.
What units were probably intended and where
would you expect buffering if any?
28. Bonus topic: Units
A peak times an application writes 49 MiB/s to a
disk which supports 50 MB/s and is replicated
over a 100 Mb/s network.
MiB = 1024^2 bytes
MB = 1000^2 bytes
Mb = 125,000 bytes
The 49 MiB/s is the highest rate and 100 Mb/s is
the lowest.
29. Bonus topic: Units
Unit bandwidth Used for
mb - miili-bit mb/s – milli-bits per second ?
mB - milli-byte mB/s – milli-bytes per second ?
kb – kilo-bit (1000) kb/s – kilo-bits (baud) per second Dial up bandwidth
kB – kilo-byte (1000) kB/s – kilo-bytes per second ?
Mb – mega-bit (1000^2) Mb/s – mega-bits (baud) per second Cat 5 ethernet
MB - mega-byte (1000^2) MB/s – mega bytes per second Disk bandwidth
Mib – mibi-bit (1024^2) Mib – Mibi-bits per second ?
MiB – mibi-byte (1024^2) MiB – Mibi-bytes per second Memory bandwidth
Gb – giga-bit (1000^3) Gb/s – giga-bit (baud) per second High speed networks
GB – giga-byte (1000^3) GB/s – giga-byte per second -
Gib – gibi-bit (1024^3) Gib/s – gibi-bit per second -
GiB – gibi-byte (1024^3) GiB/s – gibi-byte per second. Memory Bandwidth