MySQL Fabric is a clustering solution for MySQL introduced by Oracle after acquiring MySQL. It can provide high availability (HA) capabilities for MySQL through clustering and can also provide data sharding functionality. The HA mode relies on MySQL 5.6's GTID replication capability, while data sharding can be implemented using techniques like range-based or hash-based sharding. Fabric demonstrates Oracle's commitment to advancing MySQL towards mature enterprise-grade applications.
SCALE 15x Minimizing PostgreSQL Major Version Upgrade DowntimeJeff Frost
Let's face it, major version upgrades can be a pain. Most people are familiar with the dump/restore method, but if your database happens to be larger than a few GB, the downtime required for dump/restore is likely going to exceed your maximum maintenance window.
We'll briefly explore upgrade options including dump/restore, pg_upgrade and logical replication tools like Slony and why I think Slony is currently the best option. Then we'll run through a tutorial on how to use slony for a major version upgrade with minimal downtime.
You can find the scripts used in the demos here:
https://github.com/jfrost/scale-15x-talk
PGDay.Amsterdam 2018 - Stefan Fercot - Save your data with pgBackRestPGDay.Amsterdam
Ever heard of Point-in-time recovery? pgBackRest is an awsome tool to handle backups, restores and even helps you build streaming replication ! This talk will introduce the tool, its basic features and how to use it.
In this session we will cover wide area replica sets and using tags for backup. Attendees should be well versed in basic replication and familiar with concepts in the morning's basic replication talk. No beginner topics will be covered in this session
This tutorial will guide you through the many considerations when deploying a sharded cluster. We will cover the services that make up a sharded cluster, configuration recommendations for these services, shard key selection, use cases, and how data is managed within a sharded cluster. Maintaining a sharded cluster also has its challenges. We will review these challenges and how you can prevent them with proper design or ways to resolve them if they exist today. Additional topics like tag aware sharding (Zones), disaster recovery, and data streaming will also be covered.
SCALE 15x Minimizing PostgreSQL Major Version Upgrade DowntimeJeff Frost
Let's face it, major version upgrades can be a pain. Most people are familiar with the dump/restore method, but if your database happens to be larger than a few GB, the downtime required for dump/restore is likely going to exceed your maximum maintenance window.
We'll briefly explore upgrade options including dump/restore, pg_upgrade and logical replication tools like Slony and why I think Slony is currently the best option. Then we'll run through a tutorial on how to use slony for a major version upgrade with minimal downtime.
You can find the scripts used in the demos here:
https://github.com/jfrost/scale-15x-talk
PGDay.Amsterdam 2018 - Stefan Fercot - Save your data with pgBackRestPGDay.Amsterdam
Ever heard of Point-in-time recovery? pgBackRest is an awsome tool to handle backups, restores and even helps you build streaming replication ! This talk will introduce the tool, its basic features and how to use it.
In this session we will cover wide area replica sets and using tags for backup. Attendees should be well versed in basic replication and familiar with concepts in the morning's basic replication talk. No beginner topics will be covered in this session
This tutorial will guide you through the many considerations when deploying a sharded cluster. We will cover the services that make up a sharded cluster, configuration recommendations for these services, shard key selection, use cases, and how data is managed within a sharded cluster. Maintaining a sharded cluster also has its challenges. We will review these challenges and how you can prevent them with proper design or ways to resolve them if they exist today. Additional topics like tag aware sharding (Zones), disaster recovery, and data streaming will also be covered.
Staying Ahead of the Curve with Spring and Cassandra 4VMware Tanzu
SpringOne 2020
Staying Ahead of the Curve with Spring and Cassandra 4
Mark Paluch, Spring Data Project Lead at VMware
Alexandre Dutra, Technical Manager at DataStax
Staying Ahead of the Curve with Spring and Cassandra 4 (SpringOne 2020)Alexandre Dutra
Spring and Cassandra are two of the leading technologies for building cloud native applications. In this talk by the project leads for Spring Data and the Cassandra Java Driver, we’ll cover the recent improvements in the latest and greatest versions of Spring Boot, Spring Data Cassandra, Cassandra 4.0 and the Cassandra Java driver. Whether you’re a novice, intermediate, or expert developer, this content will help you get started or migrate your existing application to the latest innovations. We’ll illustrate these new concepts with code samples and snippets that you can find on GitHub to help you get things done faster with these tools.
This tutorial will guide you through the many considerations when deploying a sharded cluster. We will cover the services that make up a sharded cluster, configuration recommendations for these services, shard key selection, use cases, and how data is managed within a sharded cluster. Maintaining a sharded cluster also has its challenges. We will review these challenges and how you can prevent them with proper design or ways to resolve them if they exist today. There will be lab sessions at the end of some chapters so please have your laptops with you.
Beyond the Query – Bringing Complex Access Patterns to NoSQL with DataStax - ...StampedeCon
Learn how to model beyond traditional direct access in Apache Cassandra. Utilizing the DataStax platform to harness the power of Spark and Solr to perform search, analytics, and complex operations in place on your Cassandra data!
Spark Streaming has quickly established itself as one of the more popular Streaming Engines running on the Hadoop Ecosystem. Not only does it provide integration with many type of message brokers and stream sources, but it also provides the ability to leverage other major modules in Spark like Spark SQL and MLib in conjunction. This allows for businesses and developers to make use out of data in ways they couldn’t hope to do in the past.
However, while building a Spark Streaming pipeline, it’s not sufficient to only know how to express your business logic. Operationalizing these pipelines and running the application with high uptime and continuous monitoring has a lot of operational challenges. Fortunately, Spark Streaming makes all that easy as well. In this talk, we’ll go over some of the main steps you’ll need to take to get your Spark Streaming application ready for production, specifically in conjunction with Kafka. This includes steps to gracefully shutdown your application, steps to perform upgrades, monitoring, various useful spark configurations and more.
Spark and Cassandra with the Datastax Spark Cassandra Connector
How it works and how to use it!
Missed Spark Summit but Still want to see some slides?
This slide deck is for you!
These are the slides from a presentation I gave in 1999 at the Seattle Area System Administrators Guild monthly meeting. I haven't done this in a while, so I can't say how much of this is no longer valid, but it may prove useful to someone as a reference.
A Cassandra + Solr + Spark Love Triangle Using DataStax EnterprisePatrick McFadin
Wait! Back away from the Cassandra 2ndary index. It’s ok for some use cases, but it’s not an easy button. "But I need to search through a bunch of columns to look for the data and I want to do some regression analysis… and I can’t model that in C*, even after watching all of Patrick McFadins videos. What do I do?” The answer, dear developer, is in DSE Search and Analytics. With it’s easy Solr API and Spark integration so you can search and analyze data stored in your Cassandra database until your heart’s content. Take our hand. WE will show you how.
Similar to My sql fabric ha and sharding solutions (20)
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
1. www.vmcd.org
MySQL Fabric 为 Oracle 接手 MySQL 之后推出的一颗重磅炸弹。 它以集群形式管理 MySQL 提供 HA 功能的同时可以提供数据分片的功能
(Sharding)。其中 HA 模式依赖 MySQL5.6+的 GTID 功能,数据分片可以使用多种方式如 Range,Hash 等。随着互联网的发展 MySQL 在 RDBMS
中的地位也越来越高.以目前 Oracle 的态度来说,MySQL 在将来将是一颗重要的棋子,在内部 Oracle 也在积极推动部分的 Cluster 朝 MySQL cluster
的转移。包括最新的 12c RAC Clusterware 套件也将会支持 MySQL。Fabric 的推出表明了 Oracle 推进 MySQL 向成熟化企业级应用发展的决心。
Fabric 总体架构
2. www.vmcd.org
MySQL Fabric 需要安装的组件:
●MySQL Fabric: Prerequisites
●MySQL Servers (version 5.6.10 or later)
●Backing store database server
●Application databaseservers
●Python 2.6 or 2.7
●No support for 3.x yet
●MySQL Utilities 1.4
●Available at https://dev.MySQL.com/downloads/tools/utilities
●MySQL-connect-python-1.2.2-1.e16.noarch
$MySQL --socket=/data/MySQL3334/data/MySQL.sock
Welcome to the MySQL monitor. Commands end with ; or g.
Your MySQL connection id is 57
Server version: 5.6.16-log MySQL Community Server (GPL)
Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.
MySQL>
3. www.vmcd.org
我们的环境为:
MySQL Fabric + DB : 10.0.128.115:3334 MySQL5.6.16
MySQL server : 10.0.128.116-117:3334 10.0.128.254:3334 MySQL5.6.16
先创建 Fabric 用户注意 这个用户所在的 DB 为 localhost 即 Fabric manage 所在主机的 DB:
MySQL> show grants for 'Fabric'@'127.0.0.1';
+---------------------------------------------------------------------------------------------------------------+
| Grants for Fabric@127.0.0.1 |
+---------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'Fabric'@'127.0.0.1' IDENTIFIED BY PASSWORD '*14E65567ABDB5135D0CFD9A70B3032C179A49EE7' |
| GRANT ALL PRIVILEGES ON `Fabric`.* TO 'Fabric'@'127.0.0.1' |
+---------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)
MySQL> show grants for 'Fabric'@'localhost';
+---------------------------------------------------------------------------------------------------------------+
| Grants for Fabric@localhost |
+---------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'Fabric'@'localhost' IDENTIFIED BY PASSWORD '*14E65567ABDB5135D0CFD9A70B3032C179A49EE7' |
| GRANT ALL PRIVILEGES ON `Fabric`.* TO 'Fabric'@'localhost' |
+---------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)
6. www.vmcd.org
threads = 5
user = admin
address = 10.0.128.115:32274
password = admin
[executor]
executors = 5
[Sharding]
MySQLdump_program = /usr/bin/MySQLdump
MySQLclient_program = /usr/bin/MySQL
直接初始化 Fabric 注意 执行命令会后将在 Fabric 节点 MySQL database 建立相关的 table:
MySQLFabric manage setup --param=storage.user=Fabric --param=storage.password=secret
MySQL> use Fabric;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
9. www.vmcd.org
Fabric 的全部命令 (后面利用 add group 等命令可以添加激活 Fabric 集群)
#MySQLFabric help commands
statistics node Retrieve statistics on the Fabric node.
statistics group Retrieve statistics on Procedures.
statistics procedure Retrieve statistics on Procedures.
group activate Activate failure detector for server(s) in a group.
group description Update group's description.
group deactivate Deactivate failure detector for server(s) in a group.
group create Create a group.
group remove Remove a server from a group.
group add Add a server into group.
group health Check if any server within a group has failed and report health
information.
group lookup_servers Return information on existing server(s) in a group.
group destroy Remove a group.
group demote Demote the current master if there is one.
group promote Promote a server into master.
group lookup_groups Return information on existing group(s).
dump Fabric_nodes Return a list of Fabric servers.
dump shard_index Return information about the index for all mappings matching any of
the patterns provided.
dump Sharding_information Return all the Sharding information about the tables passed as
patterns.
dump servers Return information about servers.
10. www.vmcd.org
dump shard_tables Return information about all tables belonging to mappings matching any
of the provided patterns.
dump shard_maps Return information about all shard mappings matching any of the
provided patterns.
manage teardown Teardown Fabric Storage System.
manage stop Stop the Fabric server.
manage setup Setup Fabric Storage System.
manage ping Check whether Fabric server is running or not.
manage start Start the Fabric server.
manage logging_level Set logging level.
server set_weight Set a server's weight.
server clone Clone the objects of a given server into a destination server.
server lookup_uuid Return server's uuid.
server set_mode Set a server's mode.
server set_status Set a server's status.
role list List roles and associated permissions
user roles Change roles for a Fabric user * protocol: Protocol of the user (for
example 'xmlrpc') * roles: Comma separated
list of roles, IDs or names (see `role
list`)
user usercommand Base class for all user commands
user list List users and their roles
user add Add a new Fabric user.
user password Change password of a Fabric user.
user delete Delete a Fabric user.
threat report_error Report a server error.
11. www.vmcd.org
threat report_failure Report with certantity that a server has failed or is unreachable.
Sharding list_definitions Lists all the shard mapping definitions.
Sharding remove_definition Remove the shard mapping definition represented by the Shard Mapping
ID.
Sharding move_shard Move the shard represented by the shard_id to the destination group.
Sharding disable_shard Disable a shard.
Sharding remove_table Remove the shard mapping represented by the Shard Mapping object.
Sharding split_shard Split the shard represented by the shard_id into the destination
group.
Sharding create_definition Define a shard mapping.
Sharding add_shard Add a shard.
Sharding add_table Add a table to a shard mapping.
Sharding lookup_table Fetch the shard specification mapping for the given table
Sharding enable_shard Enable a shard.
Sharding remove_shard Remove a Shard.
Sharding list_tables Returns all the shard mappings of a particular Sharding_type.
Sharding prune_shard Given the table name prune the tables according to the defined
Sharding specification for the table.
Sharding lookup_servers Lookup a shard based on the give Sharding key.
event trigger Trigger an event.
event wait_for_procedures Wait until procedures, which are identified through their uuid in a
list and separated by comma, finish their
execution.
12. www.vmcd.org
Fabric HA 测试:
直接添加 GROUP:
MySQLFabric group create pajk_group
在 MySQL server 中 (data node) 加入 Fabric 用户:
Grant all privileges on *.* to ‘Fabric’@’%’ identified by ‘Fabric’;
在 group 中添加 node:
MySQLFabric group add pajk_group 10.0.128.254:3334
MySQLFabric group add pajk_group 10.0.128.116:3334
MySQLFabric group add pajk_group 10.0.128.117:3334
激活 GROUP 选举出 PRIAMRY 节点
MySQLFabric group promote pajk_group --slave_id=10.0.128.116:3334
MySQLFabric group activate pajk_group
13. www.vmcd.org
查看节点选举情况:
#MySQLFabric group lookup_servers pajk_group
Command :
{ success = True
return = [{'status': 'SECONDARY', 'server_uuid': 'c05867d3-f2cd-11e3-b224-5254009c6d0c', 'mode': 'READ_ONLY', 'weight': 1.0, 'address':
'10.0.128.254:3334'}, {'status': 'PRIMARY', 'server_uuid': 'e2ca75e1-d65f-11e3-b8c2-525400d3350c', 'mode': 'READ_WRITE', 'weight': 1.0, 'address':
'10.0.128.116:3334'}, {'status': 'SECONDARY', 'server_uuid': 'e90d3e07-f2cd-11e3-b225-525400991d01', 'mode': 'READ_ONLY', 'weight': 1.0, 'address':
'10.0.128.117:3334'}]
activities =
}
可以看到 已经成功建立了复制集合 注意需要使用 GTID
Kill 掉主节点 PRIMARY
[WARNING] 1403000877.240037 - FailureDetector(pajk_group) - Server (e2ca75e1-d65f-11e3-b8c2-525400d3350c) in group (pajk_group) is unreachable.
[WARNING] 1403000879.259873 - FailureDetector(pajk_group) - Server (e2ca75e1-d65f-11e3-b8c2-525400d3350c) in group (pajk_group) is unreachable.
[WARNING] 1403000881.279679 - FailureDetector(pajk_group) - Server (e2ca75e1-d65f-11e3-b8c2-525400d3350c) in group (pajk_group) is unreachable.
[WARNING] 1403000881.458935 - Executor-2 - Reported issue (FAULTY) for server (e2ca75e1-d65f-11e3-b8c2-525400d3350c).
[INFO] 1403000881.463634 - Executor-2 - Master (e2ca75e1-d65f-11e3-b8c2-525400d3350c) in group (pajk_group) has been lost.
[WARNING] 1403000882.248376 - Executor-2 - Error (<server(uuid=e90d3e07-f2cd-11e3-b225-525400991d01, address=10.0.128.117:3334, mode=READ_ONLY,
status=SECONDARY>) trying to process transactions in the relay log for candidate (Error waiting for slave to catch up. Binary log (MySQL-bin.000018, 2050964).).
[INFO] 1403000882.990106 - Executor-2 - Master has changed from e2ca75e1-d65f-11e3-b8c2-525400d3350c to e90d3e07-f2cd-11e3-b225-525400991d01.