Percona XtraDB Cluster is a high availability and high scalability solution for MySQL clustering. Percona XtraDB Cluster integrates Percona Server with the Galera synchronous replication library in a single product package which enables you to create a cost-effective MySQL cluster.
This tutorial will cover the following topics:
- Migration from standard MySQL Master-Slave Architecture to PXC
- Configuration differences between standard MySQLl and Xtradb Cluster
- How to add a node and what does SST, IST mean ? How to use them ?
- How to backup the cluster
- How to monitor the cluster
- 2 nodes servers- Why this isn't ideal but reasons and steps to setting it up anyway.
- Galera Arbitrator: Defining what it is.
- How to maintain the cluster
- Setting up load balancing for Xtradb cluster
- How to handle the cluster in the cloud
- Tips and tricks
- ... and if available cover PXC 5.6 with Galera 3 !!
Repair & Recovery for your MySQL, MariaDB & MongoDB / TokuMX Clusters - Webin...Severalnines
Slides for the webinar held on January 21st 2014
Repair & Recovery for your MySQL, MariaDB & MongoDB / TokuMX Clusters
Galera Cluster, NDB Cluster, VIP with HAProxy and Keepalived, MongoDB Sharded Cluster, etc. all have their own availability models. We are aware of these availability models and will demonstrate in this webinar how to take corrective action in case of failures via our cluster management tool, ClusterControl.
In this webinar, Severalnines CTO Johan Andersson will show you how to leverage ClusterControl to detect failures in your database cluster and automatically repair them to maximize the availability of your database services. And Codership CEO Seppo Jaakola will be joining Johan to provide a deep-dive into Galera recovery internals.
Agenda:
Redundancy models for Galera, NDB and MongoDB/TokuMX
Failover & Recovery (Automatic vs Manual)
Zooming into Galera recovery procedures
Split brains in multi-datacenter setups
Webinar slides: Introducing Galera 3.0 - Now supporting MySQL 5.6Severalnines
You'll learn how Galera integrates with MySQL 5.6 and Global Transaction IDs to enable cross-datacenter and cloud replication over high latency networks. The benefits are clear; a globally distributed MySQL setup across regions to deliver Severalnines availability and real-time responsiveness.
Galera Cluster for MySQL is a true multi-master MySQL replication plugin, and has been proven in mission-critical infrastructures of companies like Ping Identity, AVG Technologies, KPN and HP Cloud DNS. In this webcast you¹ll learn about the following Galera Cluster capabilities, including the latest innovations in the new 3.0 release:
Galera Cluster features and benefits
Support for MySQL 5.6
Integration with MySQL Global Transaction Identifiers
Mixing Galera synchronous replication and asynchronous MySQL replication
Deploying in WAN and Cloud environments
Handling high-latency networks
Management of Galera
Percona XtraDB Cluster is a high availability and high scalability solution for MySQL clustering. Percona XtraDB Cluster integrates Percona Server with the Galera synchronous replication library in a single product package which enables you to create a cost-effective MySQL cluster.
This tutorial will cover the following topics:
- Migration from standard MySQL Master-Slave Architecture to PXC
- Configuration differences between standard MySQLl and Xtradb Cluster
- How to add a node and what does SST, IST mean ? How to use them ?
- How to backup the cluster
- How to monitor the cluster
- 2 nodes servers- Why this isn't ideal but reasons and steps to setting it up anyway.
- Galera Arbitrator: Defining what it is.
- How to maintain the cluster
- Setting up load balancing for Xtradb cluster
- How to handle the cluster in the cloud
- Tips and tricks
- ... and if available cover PXC 5.6 with Galera 3 !!
Repair & Recovery for your MySQL, MariaDB & MongoDB / TokuMX Clusters - Webin...Severalnines
Slides for the webinar held on January 21st 2014
Repair & Recovery for your MySQL, MariaDB & MongoDB / TokuMX Clusters
Galera Cluster, NDB Cluster, VIP with HAProxy and Keepalived, MongoDB Sharded Cluster, etc. all have their own availability models. We are aware of these availability models and will demonstrate in this webinar how to take corrective action in case of failures via our cluster management tool, ClusterControl.
In this webinar, Severalnines CTO Johan Andersson will show you how to leverage ClusterControl to detect failures in your database cluster and automatically repair them to maximize the availability of your database services. And Codership CEO Seppo Jaakola will be joining Johan to provide a deep-dive into Galera recovery internals.
Agenda:
Redundancy models for Galera, NDB and MongoDB/TokuMX
Failover & Recovery (Automatic vs Manual)
Zooming into Galera recovery procedures
Split brains in multi-datacenter setups
Webinar slides: Introducing Galera 3.0 - Now supporting MySQL 5.6Severalnines
You'll learn how Galera integrates with MySQL 5.6 and Global Transaction IDs to enable cross-datacenter and cloud replication over high latency networks. The benefits are clear; a globally distributed MySQL setup across regions to deliver Severalnines availability and real-time responsiveness.
Galera Cluster for MySQL is a true multi-master MySQL replication plugin, and has been proven in mission-critical infrastructures of companies like Ping Identity, AVG Technologies, KPN and HP Cloud DNS. In this webcast you¹ll learn about the following Galera Cluster capabilities, including the latest innovations in the new 3.0 release:
Galera Cluster features and benefits
Support for MySQL 5.6
Integration with MySQL Global Transaction Identifiers
Mixing Galera synchronous replication and asynchronous MySQL replication
Deploying in WAN and Cloud environments
Handling high-latency networks
Management of Galera
Slides for the webinar held on January 21st 2014
Repair & Recovery for your MySQL, MariaDB & MongoDB / TokuMX Clusters
Galera Cluster, NDB Cluster, VIP with HAProxy and Keepalived, MongoDB Sharded Cluster, etc. all have their own availability models. We are aware of these availability models and will demonstrate in this webinar how to take corrective action in case of failures via our cluster management tool, ClusterControl.
In this webinar, Severalnines CTO Johan Andersson will show you how to leverage ClusterControl to detect failures in your database cluster and automatically repair them to maximize the availability of your database services. And Codership CEO Seppo Jaakola will be joining Johan to provide a deep-dive into Galera recovery internals.
Agenda:
Redundancy models for Galera, NDB and MongoDB/TokuMX
Failover & Recovery (Automatic vs Manual)
Zooming into Galera recovery procedures
Split brains in multi-datacenter setups
Topics covered in this presentation:
1. The difference between traditional (e.g. MySQL) replication and Galera Cluster
2. General Galera Cluster principles
The next release of the Java Standard Edition is scheduled for the beginning of 2010. In this session we'll review the latest feature list of what's in and what's out of the next version of the JDK.
An update on the latest news around JavaFX including the 1.2 release and the general availability of JavaFX Mobile for Windows Mobile devices.
"An introduction to Java 9" presented at the Java Hellenic User Group on 20th October 2017, by Ioannis Kolaxis - Senior Digital Expert / Software Engineer @ Unify, Greece
Up to MySQL 5.5, replication was not crash safe: after an unclean shutdown, it would fail with “duplicate key” or “row not found” error, or might generate silent data corruption. It looks like 5.6 is much better, right ? The short answer is maybe: in the simplest case, it is possible to achieve replication crash safety, but it is not the default setting. MySQL 5.7 is not much better, 8.0 has better defaults, but it is still not replication crash-safe by default, and it is still easy to get things wrong.
Crash safety is impacted by replication positioning (File+Position or GTID), type (single-threaded or MTS), MTS settings (Database or Logical Clock, and with or without slave preserve commit order), the sync-ing of relay logs, the presence of binary logs, log-slave-updates and the sync-ing of binary logs. This is very complicated stuff and even the manual is sometimes confused about it.
In this talk, I will explain the impact of the above and help you find the path to crash safety nirvana. I will also give details about replication internals, so you might learn a thing or two.
Overview of Cassandra architecture. Learn about how data is read and written into a Cassandra cluster. Internal gossip protocol. Some key data structure Cassandra uses like bloom filters, consistent hashing.
This tutorial covers all parallel replication implementation in MariaDB 10.0 and 10.1 and MySQL 5.6, 5.7 and 8.0 (including how it works in Group Replication).
MySQL and MariaDB have different types of parallel replication. In this tutorial, we present the different implementations that allow us to understand their limitations and tuning parameters. We cover how to make parallel replication faster and what to avoid for maximizing its benefits. We also present tests from Booking.com workloads.
Some of the subjects that are covered are group commit and optimistic parallel replication in MariaDB, the parallelism interval of MySQL and its Write Set optimization, and the ?slowing down the master to speed up the slave? optimization.
After this tutorial, you will know everything you need to implement and tune parallel replication in your environment. But more importantly, we will show how you can test parallel replication benefit in a non-disruptive way before deployment.
Have you ever needed to get some additional write throughput from MySQL ? If yes, you probably found that setting sync_binlog to 0 (and trx_commit to 2) gives you an extra performance boost. As all such easy optimisation, it comes at a cost. This talk explains how this tuning works, presents its consequences and makes recommendations to avoid them. This will bring us to the details of how MySQL commits transactions and how those are replicated to slaves. Come to this talk to learn how to get the benefit of this tuning the right way and to learn some replication internals.
Topics covered in this presentation, which was used for user group meetings, conferences & webinars:
1. Galera Cluster for MySQL - overview
2. Release 3 New Features:
* WAN Replication
* 5.6 Global Transaction ID (GTID) Support
* MySQL Replication Support
* and more features
3. The Galera Cluster Project
This session was presented at the JFall 2011 conference in The Netherlands. It introduces the Fork/Join framework that was introduced in Java 7. Example code can be found at http://bit.y/jfall-forkjoin
Slides for the webinar held on January 21st 2014
Repair & Recovery for your MySQL, MariaDB & MongoDB / TokuMX Clusters
Galera Cluster, NDB Cluster, VIP with HAProxy and Keepalived, MongoDB Sharded Cluster, etc. all have their own availability models. We are aware of these availability models and will demonstrate in this webinar how to take corrective action in case of failures via our cluster management tool, ClusterControl.
In this webinar, Severalnines CTO Johan Andersson will show you how to leverage ClusterControl to detect failures in your database cluster and automatically repair them to maximize the availability of your database services. And Codership CEO Seppo Jaakola will be joining Johan to provide a deep-dive into Galera recovery internals.
Agenda:
Redundancy models for Galera, NDB and MongoDB/TokuMX
Failover & Recovery (Automatic vs Manual)
Zooming into Galera recovery procedures
Split brains in multi-datacenter setups
Topics covered in this presentation:
1. The difference between traditional (e.g. MySQL) replication and Galera Cluster
2. General Galera Cluster principles
The next release of the Java Standard Edition is scheduled for the beginning of 2010. In this session we'll review the latest feature list of what's in and what's out of the next version of the JDK.
An update on the latest news around JavaFX including the 1.2 release and the general availability of JavaFX Mobile for Windows Mobile devices.
"An introduction to Java 9" presented at the Java Hellenic User Group on 20th October 2017, by Ioannis Kolaxis - Senior Digital Expert / Software Engineer @ Unify, Greece
Up to MySQL 5.5, replication was not crash safe: after an unclean shutdown, it would fail with “duplicate key” or “row not found” error, or might generate silent data corruption. It looks like 5.6 is much better, right ? The short answer is maybe: in the simplest case, it is possible to achieve replication crash safety, but it is not the default setting. MySQL 5.7 is not much better, 8.0 has better defaults, but it is still not replication crash-safe by default, and it is still easy to get things wrong.
Crash safety is impacted by replication positioning (File+Position or GTID), type (single-threaded or MTS), MTS settings (Database or Logical Clock, and with or without slave preserve commit order), the sync-ing of relay logs, the presence of binary logs, log-slave-updates and the sync-ing of binary logs. This is very complicated stuff and even the manual is sometimes confused about it.
In this talk, I will explain the impact of the above and help you find the path to crash safety nirvana. I will also give details about replication internals, so you might learn a thing or two.
Overview of Cassandra architecture. Learn about how data is read and written into a Cassandra cluster. Internal gossip protocol. Some key data structure Cassandra uses like bloom filters, consistent hashing.
This tutorial covers all parallel replication implementation in MariaDB 10.0 and 10.1 and MySQL 5.6, 5.7 and 8.0 (including how it works in Group Replication).
MySQL and MariaDB have different types of parallel replication. In this tutorial, we present the different implementations that allow us to understand their limitations and tuning parameters. We cover how to make parallel replication faster and what to avoid for maximizing its benefits. We also present tests from Booking.com workloads.
Some of the subjects that are covered are group commit and optimistic parallel replication in MariaDB, the parallelism interval of MySQL and its Write Set optimization, and the ?slowing down the master to speed up the slave? optimization.
After this tutorial, you will know everything you need to implement and tune parallel replication in your environment. But more importantly, we will show how you can test parallel replication benefit in a non-disruptive way before deployment.
Have you ever needed to get some additional write throughput from MySQL ? If yes, you probably found that setting sync_binlog to 0 (and trx_commit to 2) gives you an extra performance boost. As all such easy optimisation, it comes at a cost. This talk explains how this tuning works, presents its consequences and makes recommendations to avoid them. This will bring us to the details of how MySQL commits transactions and how those are replicated to slaves. Come to this talk to learn how to get the benefit of this tuning the right way and to learn some replication internals.
Topics covered in this presentation, which was used for user group meetings, conferences & webinars:
1. Galera Cluster for MySQL - overview
2. Release 3 New Features:
* WAN Replication
* 5.6 Global Transaction ID (GTID) Support
* MySQL Replication Support
* and more features
3. The Galera Cluster Project
This session was presented at the JFall 2011 conference in The Netherlands. It introduces the Fork/Join framework that was introduced in Java 7. Example code can be found at http://bit.y/jfall-forkjoin
In the big data world, our data stores communicate over an asynchronous, unreliable network to provide a facade of consistency. However, to really understand the guarantees of these systems, we must understand the realities of networks and test our data stores against them.
Jepsen is a tool which simulates network partitions in data stores and helps us understand the guarantees of our systems and its failure modes. In this talk, I will help you understand why you should care about network partitions and how can we test datastores against partitions using Jepsen. I will explain what Jepsen is and how it works and the kind of tests it lets you create. We will try to understand the subtleties of distributed consensus, the CAP theorem and demonstrate how different data stores such as MongoDB, Cassandra, Elastic and Solr behave under network partitions. Finally, I will describe the results of the tests I wrote using Jepsen for Apache Solr and discuss the kinds of rare failures which were found by this excellent tool.
Troubleshooting Common Network Related Issues with NetScalerDavid McGeough
Webinar recording: https://www1.gotomeeting.com/register/737119097
As a NetScaler Administrator, you will need to understand how the NetScaler interacts with the network to ensure an optimally running environment for your applications. In this Webinar delivered by NetScaler Escalation Engineers you will learn some of the common network configuration issues, how to avoid them and when necessary how to troubleshoot them.
You will learn how to troubleshoot:
- HA issues
- GARP issues
- LA channel issues
- Layer 2 issues
The Foundations of Multi-DC Kafka (Jakub Korab, Solutions Architect, Confluen...confluent
Kafka is notoriously tricky for multi-dc use cases. The log abstraction and client failover breaks down when you cannot at least guarantee offset consistency. In this talk, we define the current state of Kafka in terms of multi-dc usage, how different approaches provide different guarantees as well as examining the missing gaps, and how the community is addressing them.
FPGA based 10G Performance Tester for HW OpenFlow SwitchYutaka Yasuda
SDN operators need to measure the performance of OF HW switch on their site. Cause there is 1000 times differences in latency, depends on the specified flow entry. ASIC can forward in several μsecs but the software (CPU) may take msec.
To protect yourself from unexpected performance plunge, monitor your switches healthiness on your site.
Using Groovy? Got lots of stuff to do at the same time? Then you need to take a look at GPars (“Jeepers!”), a library providing support for concurrency and parallelism in Groovy. GPars brings powerful concurrency models from other languages to Groovy and makes them easy to use with custom DSLs:
- Actors (Erlang and Scala)
- Dataflow (Io)
- Fork/join (Java)
- Agent (Clojure agents)
In addition to this support, GPars integrates with standard Groovy frameworks like Grails and Griffon.
Background, comparisons to other languages, and motivating examples will be given for the major GPars features.
Looming Marvelous - Virtual Threads in Java Javaland.pdfjexp
Nowadays we have 2 options for concurrency in Java:
* simple, synchronous, blocking code with limited scalability that tracks well linearly at runtime, or.
* complex, asynchronous libraries with high scalability that are harder to handle.
Project Loom aims to bring together the best aspects of these two approaches and make them available to developers.
In the talk, I'll briefly cover the history and challenges of concurrency in Java before we dive into Loom's approaches and do some behind-the-scenes implementation. To manage so many threads reasonably needs some structure - for this there are proposals for "Structured Concurrency" which we will also look at. Some examples and comparisons to test Loom will round up the talk.
Project Loom is included in Java 19 and 20 as a preview feature, it can already be tested how well it works with our applications and libraries.
Spoiler: Pretty good.
Top Mistakes When Writing Reactive Applications - Scala by the Bay 2016Petr Zapletal
Reactive applications are becoming a de-facto industry standard and, if employed correctly, toolkits like Lightbend Reactive Platform make the implementation easier than ever. But the design of these systems might be challenging as it requires particular mindset shift to tackle problems we might not be used to. In this talk, we’re going to discuss the most common things I’ve seen in the field that prevented applications to work as expected. I’d like to talk about typical pitfalls that might cause troubles, about trade-offs that might not be fully understood or important choices that might be overlooked including persistent actors pitfalls, tackling of network partitions, proper implementations of graceful shutdown or distributed transactions, trade-offs of micro-services or actors and more. This talk should be interesting for anyone who is thinking about, implementing, or have already deployed a reactive application. My goal is to provide is to provide a comprehensive explanation of common problems to be sure they won’t be repeated by fellow developers. The talk is a little bit more focused on Lightbend platform but the understanding of the concepts we are going to talk about should be beneficial for everyone interested in this field.
Akka Cluster is a powerful toolkit for building distributed applications. But as always, power comes with a price. Special care needs to be taken during bootstrapping, network partitions and gracefully stopping a cluster. In this talk, I'll introduce Akka Management and Akka Split Brain Resolver modules and show how they can help to manage an Akka Cluster in production. Join us when you want to know how a stateful application can handle rolling deployments without downtime and without losing in-flight requests and how it can safely recover from network partitions.
ZeroMQ is a socket abstraction and concurrency framework that's changing the way we think and reason about distributed systems. In this talk we'll discuss :
The importance of lightweight messaging for applications
Composable messaging patterns
Swappable transports
Refactoring at the transport layer
Resiliency and reliability
Small case study
he recent explosion in products supporting multiple devices require exposing data through lightweight services and data driven APIs. We're shifting data across flaky networks, massaging between different storage backends, encode / decode by protocol and frequently drive
this from Cloud platforms with very variable Quality of Service and availability guarantees (if any). Yet most engineering teams aren't up to speed with POSIX and POSIX.1b(Realtime-extensions), the API's exposed by most modern production systems for I/O.
In this talk I'll look at a few very often misunderstood concepts, as well as challenge others.
* Understanding buffered I/O and how to sustain throughput for fast and slow clients alike
* Blocking VS Non-blocking I/O
* POSIX Lies - compliance isn't always as it should be, in some cases *worse* than CSS and other specifications "supported" by modern browsers
* Asynchronous I/O myths and why I/O concurrency is often an illusion
* Why and how Windows I/O Completion Ports knock the socks off most UNIX systems
* I/O multiplexing gotchas
* The Cloud: managing latency with variable network and storage characteristics
* High throughput system calls
In the Loop - Lone Star Ruby ConferenceLourens Naudé
The Reactor Pattern's present in a lot of production infrastructure (Nginx, Eventmachine, 0mq, Redis), yet not very well understood by developers and systems fellas alike. In this talk we'll have a look at what code is doing at a lower level and also how underlying subsystems affect your event driven services.
Below the surface : system calls, file descriptor behavior, event loop internals and buffer management
Evented Patterns : handler types, deferrables and half sync / half async work
Anti patterns : on CPU time, blocking system calls and protocol gotchas
A whirlwind tour of Event Driven Architecture, extensibility, Domain Driven Design, Command and Query Responsibility Segregation (CQRS) and Complex Event Processing
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
5. A change in state ...
MySQL server sends results down the wire
Connection file descriptor becomes readable
Process results
Render results
Ignorance is bliss
Never be flooded
Only handle events that matter
6. Talk#to_a
Context switches
Scheduling
Programming models
Frameworks
Eventmachine
Neverblock
Operating System
Lessons from massively multiplayer online games
Testing
Q&A?
8. Scenario we all can relate to ...
Quiet weekend day, erratic thoughts from the
week before comes together
Sharing space with your better half ... who's
being chatty
You're being interrupted at random
Let's assume an average quot;in the zonequot; startup
threshold of 2 minutes per interruption
Deadlock's not an option ...
9. Multithreading
Let the initial pattern of thought and the
interruption be distinct Threads
The interruption's effective right away
The context switch is 'expensive' : coding VS
whichever chore's been neglected
Preemptive scheduling == constant interruption
10. Fiber / coroutine pool
Two distinct Fibered contexts
Able to defer the interruption
Cheap switching : being thick skinned buys time
You#resume the interruption if and when there's
time to handle it
11. Thread
Scheduling
Handled by the VM – timesliced @ 10ms by MRI
Executes as soon as CPU resources become available
Availability
MRI 18: Maps to a single native thread
MRI 19: 1 x native thread for each Ruby thread - GIL
JRuby: 1 x Java thread for each Ruby thread
Overheads
Large initial stack size
MRI uses a SIGVTALRM to signal a context switch
12. Fiber
Scheduling
No concurrency – scheduled by application code
Never executes right away
Availability
MRI 18: Patch by Aman Gupta && Joe Damato
MRI 19: Core Fiber class
JRuby: Poor Man's Fibers
Overheads
Small initial stack size @ 4k
Context switches exclusively in user space – very fast
13. Why Fibers matter
Computes a partial result - generator
Yields back to the caller
Saves execution state
Caller resumes
Data exchange
Accept arguments through Fiber#resume
Return values through Fiber.yield
14. Threads as Coroutines
Through a voodo combination of
Thread priorities
Thread.pass
Thread.stop
Thread#run (yields CPU) or Thread#wakeup (no CPU)
Poor Man's Fibers piggy backs off a Queue
MRI 18 Fibers = Threads – preemptive
scheduling, much like continuations, but with a
smaller stack.
Performant, but not as lightweight as Fibers
15. Cooperative scheduling
Request / response with evented DB access
HTTP request wrapped in a Fiber
=> mysql->fd : SELECT * FROM users WHERE id = 1
Execution paused
<= mysql->fd : #<User id: 1 uname: methodmissing>
Execution resumed
=> mysql->fd: SELECT * FROM wishlist WHERE
user_id = 1
Execution paused
<= mysql->fd: #<Wishlist id: 20 user_id 1>
17. Scheduling
active time
Thread 1
idle 10 ms
quanta
Thread 2
idle No scheduling time
Fiber 1
active Run as long as you need
Fiber 2
Cooperative VS Preemptive
18. Event-driven Programming
Program flow determined by events
Characterized by an event handler, main
loop or reactor
The reactor typically 'owns' the process
loop { button.on(:click).dispatch }
Binds event handlers to events
Deep nesting of callbacks and constructs
eg. Twisted for Python
22. … app server
class UsersController < AC::Base
def index
@users = User.paginate( :page => 1 )
end
end
Thin::Server.start('0.0.0.0', 3000) do
Fiber.new{ AC::Dispatcher.call( env ) }
end
24. Eventmachine
Implementation of the Reactor pattern
C++ with Ruby Bindings
JRuby compatible through jeventmachine
High performance event driven IO : epoll
on Linux, kqueue on BSD and derivatives
25. The Reactor
Maintains a global current loop time
Quantum is just under 100ms … adjustable
Verifies descriptors through a heartbeat
Runs timer events
New descriptors dropped onto a different
queue
Modified descriptors processed outside loop
Stopped with special Loop Break Descriptor
27. Eventable File Descriptors
Nonblocking
Notify of read and writability
Closed in 3 ways
hard
immediate ( when possible, next tick )
after writing ( protocol handlers, serve HTTP page,
close connection
28. Deferrables
A callback ... specifically a callback for
:success or :failure
Immediately pushed to the background and
scheduled for future execution
Deferrables should have an associated
timeout
29. Timers
Periodic or oneshot
Set a max timer count to avoid flooding the
reactor
Can be cancelled
30. Files
Very fast transfer of < 32k files with
EM#send_file_data
File watch API
Notified when modified, deleted or moved
fastfilereader extension
Memory <=> disk transfer with mmap
31. Processes
Supports deferrable child processes
through pipes
EM#system
Process watch API
Notified when forked or exits
32. Client / Server
module Challenge
def post_init
send_data 'psw?'
end
def receive_data( data )
( @data ||= '' ) << data
end
end
EM.run{ EM.connect 'localhost', 80, Challenge }
34. Why ?
Most web apps block on IO
Ruby processes has 'high' memory
overheads
60MB RSS blocking 30ms on IO == $$$
Allocated in slabs – 80MB -> 120MB
Significantly reduces capacity for
concurrent requests
eSpace stepped in
35. How ?
Fibered connection pool – typically 8 to 12
Attaches to either EM or Rev as main loop
Patches for Evented Mongrel && Thin to
wrap requests in Fibers
Fibered DB connections for postgres &&
mysql ( with mysqlplus )
Respects transactions - BEGIN ... COMMIT
required to be on the same connection
43. Signals
Software interrupts
Portable
Signal.list supported on all host systems
Unobtrusive
Caught Signal.trap( 'IO' ){ puts 'readable' }
Sent with Process.kill( 'IO', $ )
Can be scoped to a single process or a
process group
Daemons: SIGHUP && SIGTERM
44. Handling Signals
Ignore them – except KILL && STOP
Catch with trap('IO'){}
Let the default action apply
SIGVTALRM is reserved for threading
Signals the process every 10ms
Flags that it's time to schedule
Green threading maps to 1 OS thread, 1
per process, signals scoped per process
45. Processes
Process groups
Leader ? Process#pid == process group identifier
Group persist when leader exists as long as there's
some members
46. Multi-process Quality of Service
Signals to enforce QOS
URG: Slow down
CONT: Resume normal operation
XCPU: Crawl
Monitoring agent sends URG during load
spike
Sends CONT when load drops
XCPU during extremely high load,
effectively pausing the worker(s)
47. POSIX Realtime Extensions
Well supported, but implementations differ
Linux Kernel 2.6
FreeBSD / Darwin
Solaris
Kernel level asynchronous IO
Supports thread, signal based or no-op
callbacks
Reduces syscall overheads in IO bound
applications
48. How it works
Control block
Struct: fd, buffer, notification, offset, number of bytes
aio_read( '/tmp/file' ) # signal notification
with SIGIO
Returns right away, faster than
O_NONBLOCK
Kernel signals with SIGIO on completion
Handler notification contains pointer to the the original
control block
49. Why it matters
Parallel execution of aio_read && aio_write
lio_listio( LIO_WRITE, list_of_cbs, 90, &list_sig )
A single user to kernel space switch
90 operations, a single signal notification
Experimental rb_aio extension
AIO.write( { :filename => 'buffer' } )
51. Gaming @ a Rails conf ?
Rails Rack compatibility
Middleware + ESI
SOA architecture with thin processes +
layers
Russian doll pattern
52. Architecture Overview
Communications
Pubsub && direct channels suported
Execution Kernel
Stateless execution of tasks
Data storage
Transactional storage of serialized objects
53. Task Execution
Spawned and managed by a task manager
Appears monothreaded, executes in parallel
Immediate, delayed or periodic
Parent <=> child relationships
Guaranteed to execute in order
Children inherits priority of parent
54. Task Lifetime
Max execution time is 100ms - configurable
Split long running tasks into multiple
smaller tasks
55. Managed Objects && References
Managed object == a persitable entity
References
Encapsulates all interactions with the data store
Retrieval strategies
Get with the intention of modifying state
Get for read
Object#mark_for_update to promote a read only
instance
56. Listeners
Application listener
Initialize: setup any game play items when first booted
Login callback: setup per player game state
Client listener
Defines methods to repond to client events eg. logout
58. Test strategies
Define environments : test / production
Ability to switch between an evented and
batched programming model depending on
the environment
A desirable side effect of loose coupling
Very well suited for integration testing
59. Block in testing mode
Inject correlation middleware when in the
test environment
extend BlockInTest
A single event may spawn *data
@broker.collateral_report( 'ACCXXXX' )
=> [#<Report:0x202948>, #<Report:0x202234>]
A single event may spawn many others
@broker.collateral_report( 'ACCXXXX' )
=> [:acknowledge_reports, :reports]
60. Event => expectation
Mapping Event X to Reaction *Y : Event
IDXXXXXXX => { IDXXXXXXX => [Y,Y,Y] }
Mapping Event X to Data Elements *Z :
Event IDXXXXXXX => { IDXXXXXXX =>
[Z,Z,Z] }
Backbone of integration tests
61. Existing specs ?
github.com/tmm1/em-spec
Bacon and Rspec backends
A reactor / event loop for context
How ?
Context executes in a Fiber, each expectation yields
Unobtrusive
describe 'This is a context'
Em.spec 'This is an evented context