Ceph can provide storage tiering with different performance levels. It allows combining SSDs, SAS, and SATA disks from multiple nodes into pools to provide tiered storage. Performance testing showed that for reads, Ceph provided good performance across all tiers, while for writes Nvme disks had the best performance compared to SSD, SAS, and SATA disks. FIO, IOmeter, and IOzone were some of the tools used to measure throughput and IOPS.
This presentation provides an overview of the Dell PowerEdge R730xd server performance results with Red Hat Ceph Storage. It covers the advantages of using Red Hat Ceph Storage on Dell servers with their proven hardware components that provide high scalability, enhanced ROI cost benefits, and support of unstructured data.
This presentation provides an overview of the Dell PowerEdge R730xd server performance results with Red Hat Ceph Storage. It covers the advantages of using Red Hat Ceph Storage on Dell servers with their proven hardware components that provide high scalability, enhanced ROI cost benefits, and support of unstructured data.
Study: The Future of VR, AR and Self-Driving CarsLinkedIn
We asked LinkedIn members worldwide about their levels of interest in the latest wave of technology: whether they’re using wearables, and whether they intend to buy self-driving cars and VR headsets as they become available. We asked them too about their attitudes to technology and to the growing role of Artificial Intelligence (AI) in the devices that they use. The answers were fascinating – and in many cases, surprising.
This SlideShare explores the full results of this study, including detailed market-by-market breakdowns of intention levels for each technology – and how attitudes change with age, location and seniority level. If you’re marketing a tech brand – or planning to use VR and wearables to reach a professional audience – then these are insights you won’t want to miss.
The slides from our first webinar on getting started with Ceph. You can watch the full webinar on demand from http://www.inktank.com/news-events/webinars/. Enjoy!
Best Practices & Performance Tuning - OpenStack Cloud Storage with Ceph - In this presentation, we discuss best practices and performance tuning for OpenStack cloud storage with Ceph to achieve high availability, durability, reliability and scalability at any point of time. Also discuss best practices for failure domain, recovery, rebalancing, backfilling, scrubbing, deep-scrubbing and operations
TUT18972: Unleash the power of Ceph across the Data CenterEttore Simone
From SUSECon 2015: Smooth integration of emerging Software Defined Storage technologies into traditional Data Center using Fiber Channel and iSCSI as key values for success.
Ceph at Work in Bloomberg: Object Store, RBD and OpenStackRed_Hat_Storage
Bloomberg's Chris Jones and Chris Morgan joined Red Hat Storage Day New York on 1/19/16 to explain how Red Hat Ceph Storage helps the financial giant tackle its data storage challenges.
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...Odinot Stanislas
Après la petite intro sur le stockage distribué et la description de Ceph, Jian Zhang réalise dans cette présentation quelques benchmarks intéressants : tests séquentiels, tests random et surtout comparaison des résultats avant et après optimisations. Les paramètres de configuration touchés et optimisations (Large page numbers, Omap data sur un disque séparé, ...) apportent au minimum 2x de perf en plus.
Similar to Ceph Day KL - Ceph Tiering with High Performance Archiecture (20)
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
4. Why we need Ceph?
Distributed storage system
- Fault tolerant , no SPoF
X86 Commodity hardware
- Saving you costs, giving you flexibility
Large scale – Incremental expansion
- 10s to 1000s of nodes
Unified storage platform
- Scalable object , Block , File system.
Open source – No vender lock-in
Automatically balance the file system
Data security
- with 2 or more copies in different physical store media.
6. PG and Pools
PG number for Single OSD : 30 ~ 300 (soft limit), usually we suggest 256
PG number for each Pool =
Number of OSD * PG number of 1 OSD / Replica
Example for this Pool : 4 * 256 / 2 = 512
Object Size = 4KB ~ 32MB, default = 4MB
Sometimes, with the increasing of the Object size,
throughput will also be increased
Average distribute PGs on OSD will get better performance
7. CRUSH Maps
CRUSH Map Parameters
1. Settings : Basic settings, usually we don’t need to change
2. Devices : Physical device list (List all osd devices and
define the id and name mapping)
3. Types : Define bucket types (from Root to OSD)
4. Buckets : Define OSD group and tiering structure
5. Rules : crush rule (define object chunk)
12. CRUSH Maps
Rules
ruleset : rule_id
type : object chunk method, replicated or erasure
min_size : if the replica number less than this setting the pool will NOT select this
rule
max_size : if the replica number larger than this setting the pool will NOT select this
rule
step take : set which osd_tree should be mapped to this rule
step chooseleaf : set the mapping method for object chunk’s replica. For example,
“step chooseleaf firstn 0 type host” will set replicas by hosts. (each host will have 1
replica)
20. Performance Measurement Tools
Tool Name Testing Scenario Command line/GUI OS Support Popularity Reference
FIO
(Flexible I/O Tester)
major in Block level storage
ex.SAN、DAS
Command line Linux / Windows High fio github
IOmeter
major in Block level storage
ex.SAN、DAS
GUI / Command line Linux / Windows High
Iometer and
IOzone
iozone File Level Storage ex.NAS GUI / Command line Linux / Windows High
IOzone
Filesystem
Benchmark
dd File Level Storage ex.NAS Command line Linux / Windows High
dd over NFS
testing
rados bench Ceph RADOS Command line Linux only Normal
BENCHMARK A
CEPH STORAGE
CLUSTER
rest-bench CEPH RESTful Gateway Command line Linux only Normal
BENCHMARK A
CEPH OBJECT
GATEWAY
cosbench Cloud Object Storage Service GUI / Command line Linux / Windows High
COSBench -
Cloud Object
Storage
Benchmark
21. IOPS and Throughput formula
IOPS
IOPS = (MBps Throughput / KB per IO) * 1024
Throughput
MBps = (IOPS * KB per IO) / 1024
25. Conclusion
Ceph Storage Tiering System is useful for
different kinds of user scenario aggregated in
one system.
For the Read Scenario, Ceph will provide very
good performance in all kinds of Tiers.
For Write Scenario, Nvme can give us a very
good performance than SSD, SAS and SATA.
Editor's Notes
Why we need Ceph? Here we can see the benefits listed about Ceph.
Ceph is a Distributed storage system with Fault tolerance and no single point of failure architecture.
Ceph is open source and no vender lock-in issue.
Another benefit is cost saving. For Ceph architecture, we only need x86 hardware.
Moreover, Ceph is a scalable storage system which means it can support large scale from 10 nodes to more than one thousand nodes.
When we talk about Ceph, many people may ask what is the difference between Ceph and HDFS? There are some features similar to each other such as replicas, distributed and scalable architecture.
It looks like the same, but the most important about Ceph is that it can support three kinds of protocol (Block, Object, and File) and HDFS can only support File. Besides, the most important thing of Ceph is the Crush Map Algorithm. When we modify the Crush Map, we can do many things that HDFS cannot do, like DR Architecture, guarantees 3 replicas into 3 different clusters; Set weight for each OSD.
About the Crush Map, we will talk about it more detail In the following pages.
This is the Ceph Architecture, from bottom to up means from hardware level to application level.
So, in this architecture, we can see in the hardware level, Ceph monitor and OSD is scalable and on top of them is the API level.
The first API is LIBRADOS and RADOSGW, RBD, CEPHFS protocols are on top of LIBRADOS. As a result, if you want to have a better performance, you can call the LIBRADOS to develop your applications.
On the top is the applications integrate with Ceph. The most common scenarios are OpenStack and File sharing. When Ceph integrate with OpenStack, the Rados gateway can provide Keystone and Swift API; RBD can provide cinder and glance API. For file sharing scenario, before Jewel version, we use RBD and NFS to provide file sharing function because CephFS in Hammer version is not stable. After Jewel version, due to the critical bugs (data loss) about CephFS are fixed, we can provide file sharing functions through CephFS. By using CephFS, the clients need to install the key ring for data access and if you do not want to install key ring for each client, you can also use NFS to simplify the process. When you use NFS you will not need to install key ring for each client because you only need to install key ring on NFS server. But the important thing is that client connect to CephFS directly can get better performance than using NFS. This is a trade-off, depends on the scenario and requirements.
In this page, we will talk about PG and Pools. These are very important concept in Ceph.
In Ceph, each file will be separated into many objects and objects will be stored to different placement group (PG).
A Pool will aggregate placement groups and the formula about the PG Number for each pool is (Number of OS ) Multiply the PG Number for each OSD than Divide the Replica number.
For the PG number of each OSD, we have a soft limit, 30 to 300 and usually we suggest 256.
For example if we have 4 OSD and PG number for each OSD is 256 with 2 replicas, The PG number for each Pool is 4 multiply 256 and divide 2, equal to 512.
When we want to have good performance, we need to average distribute the PGs on OSD.
For the Object size, we can set from 4 KB to 32 MB and by default the size is 4MB.
Sometimes, with the increasing of the object size, the throughput will also be increased but it still depends on the real environment.
After the PG and Pools, we will start to talk about the most important algorithm in the Ceph – Crush Map.
Crush map is the most important thing in Ceph. We can achieve many special functions or get better performance by setting the crush map.
The are five important parameters in Crush map which are settings, devices, types, buckets, and rules.
About settings, which is the basic settings for crush map and usually we don’t need to change it.
Devices is the physical device list which will list all the osd devices and define the device id and device name mappings.
Types will define the bucket types from root to OSD
Buckets will define the OSD group and tiering structures
Rules means the crush rule and will define the object chunk.
In the following pages, I will take a example about set the crush map to achieve tiering architecture.
This is the default OSD tree and we can see we have 3 OSD nodes and each node has 6 OSD (hard drives) with 3 STAT and 3 SSD
We separate the replica of object’s chunk by host.
This is the defaut setting for crush map and usually we don’t need to change it.
Here we can see the device list and the osd id.
For the types, here are 10 types from osd to root
In the bucket settings, we can set the weight for each osd to set the tiering architecture.
There are six imiportant parameters in crush rules.
Ruleset is the rule id and type will define the object chunk method, replicated or erasure coding.
The min size and the max size is the criteria for the pool the select this rule.
The step take will set which osd tree should be mappted to this rule, and in this example will select default osd tree.
The step chooseleaf will set the mapping method for object chunk’s replica. In this example “set firstn 0 type host”will set replicas by host which means each host will have 1 replica.
So, if you want to have a DR architecture to guarantee each site will have 1 replica, this setting is helpful to you.
Then, after the settings, the OSD tree will changed to 2 pools, one is SSD pool and another is SATA pool.
We can also use ceph ose crush rule list to see the ruleset and ceph osd dump to see the pool list.
We set 4 tier with SATA, SAS, SSD and Nvme SSD Tier and test 4 kinds of parameters which are 128 sequential read / write and 4k random read / write.
We can see the nvme get the best write performance and than are SSD, SAS and SATA, but for read, they almost get the similar perfromance.
We also find out that 128k block size would get better performance than 4k block size.
The IOPS get the same reslut as throughput. Nvme get the best performance in write scenario and all the tier get similar performance in read scenario.