Trying and evaluating the new features of GlusterFS 3.5Keisuke Takahashi
My presentation in LinuxCon/CloudOpen Japan 2014.
It has passed few days since GlusterFS 3.5 released so feel free to correct me if you find my mistakes or misunderstandings. Thanks.
Trying and evaluating the new features of GlusterFS 3.5Keisuke Takahashi
My presentation in LinuxCon/CloudOpen Japan 2014.
It has passed few days since GlusterFS 3.5 released so feel free to correct me if you find my mistakes or misunderstandings. Thanks.
Practical advices how to achieve persistence in Redis. Detailed overview of all cons and pros of RDB snapshots and AOF logging. Tips and tricks for proper persistence configuration with Redis pools and master/slave replication.
Presentation from 2016 Austin OpenStack Summit.
The Ceph upstream community is declaring CephFS stable for the first time in the recent Jewel release, but that declaration comes with caveats: while we have filesystem repair tools and a horizontally scalable POSIX filesystem, we have default-disabled exciting features like horizontally-scalable metadata servers and snapshots. This talk will present exactly what features you can expect to see, what's blocking the inclusion of other features, and what you as a user can expect and can contribute by deploying or testing CephFS.
Pulsar is used by a portfolio of products at Splunk for stream processing of different types of data, including metrics and logs. In this talk, Karthik Ramasamy will share how Splunk helped a flagship customer scale a Pulsar deployment to handle 10 PB/day in a single cluster. He will talk about the journey, the challenges faced, and the trade-offs made to scale Pulsar and operate it reliably and stably in Google Cloud Platform (GCP).
A summary of available MongoDB backup options. We manage backup for hundreds of MongoDB servers at scalegrid.io and we have learnt a few things along the way.
PostgreSQL and ZFS were made for each other. This talk dives downstack into the internals and way that PostgreSQL consumes disk resources and tricks that are available if you run PostgreSQL on ZFS (ZFS on Linux, ZFS on FreeBSD, or ZFS on Illumos). Topics covered will include:
* Performance and sizing considerations
* Workload estimation heuristics
* Standard administrative practices that leverage ZFS
* Recovery using ZFS
* Performing database migrations using ZFS
This presentation covers all aspects of PostgreSQL administration, including installation, security, file structure, configuration, reporting, backup, daily maintenance, monitoring activity, disk space computations, and disaster recovery. It shows how to control host connectivity, configure the server, find the query being run by each session, and find the disk space used by each database.
Practical advices how to achieve persistence in Redis. Detailed overview of all cons and pros of RDB snapshots and AOF logging. Tips and tricks for proper persistence configuration with Redis pools and master/slave replication.
Presentation from 2016 Austin OpenStack Summit.
The Ceph upstream community is declaring CephFS stable for the first time in the recent Jewel release, but that declaration comes with caveats: while we have filesystem repair tools and a horizontally scalable POSIX filesystem, we have default-disabled exciting features like horizontally-scalable metadata servers and snapshots. This talk will present exactly what features you can expect to see, what's blocking the inclusion of other features, and what you as a user can expect and can contribute by deploying or testing CephFS.
Pulsar is used by a portfolio of products at Splunk for stream processing of different types of data, including metrics and logs. In this talk, Karthik Ramasamy will share how Splunk helped a flagship customer scale a Pulsar deployment to handle 10 PB/day in a single cluster. He will talk about the journey, the challenges faced, and the trade-offs made to scale Pulsar and operate it reliably and stably in Google Cloud Platform (GCP).
A summary of available MongoDB backup options. We manage backup for hundreds of MongoDB servers at scalegrid.io and we have learnt a few things along the way.
PostgreSQL and ZFS were made for each other. This talk dives downstack into the internals and way that PostgreSQL consumes disk resources and tricks that are available if you run PostgreSQL on ZFS (ZFS on Linux, ZFS on FreeBSD, or ZFS on Illumos). Topics covered will include:
* Performance and sizing considerations
* Workload estimation heuristics
* Standard administrative practices that leverage ZFS
* Recovery using ZFS
* Performing database migrations using ZFS
This presentation covers all aspects of PostgreSQL administration, including installation, security, file structure, configuration, reporting, backup, daily maintenance, monitoring activity, disk space computations, and disaster recovery. It shows how to control host connectivity, configure the server, find the query being run by each session, and find the disk space used by each database.
Slide chia sẻ công nghệ về caching, thông qua slide này bạn sẽ trả lời được những câu hỏi như:
- Caching là gì
- Làm sao sử dụng cũng như xây dựng hệ thống caching
- Tại sao cache giúp tăng tốc ứng dụng lên vài chục, vài trăm lần
- Các hệ thống lớn của Facebook, Twitter, ... đang sử dụng cache thế nào
- ...
Slide chia sẻ về công nghệ về caching, thông qua slide này bạn sẽ trả lời được những câu hỏi như:
- Caching là gì
- Làm sao sử dụng cũng như xây dựng hệ thống caching
- Tại sao cache giúp tăng tốc ứng dụng lên vài chục, vài trăm lần
- Các hệ thống lớn của Facebook, Twitter, ... đang sử dụng cache thế nào
- ...
Find Site Performance from the server to WordPress. A look at how some good performance gains can be made in tuning MySQL and APC and getting the most of out W3 Total Cache.
hbaseconasia2017: Large scale data near-line loading method and architectureHBaseCon
Shuaifeng Zhou
When we do real-time data loading to HBase, we use put/putlist interface. After receiving put request, regionserver will write WAL, write data into memory store, flush memory store to disk-store, then compact files again and again. That precedure occupies too much resource and causing read/write performance decrease. To solve the problem, we provide a kind of near-line loading method and architecture, greatly increase the loading bandwidth, and decrease the influence to read operations.
hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#
Gluster for Geeks: Performance Tuning Tips & TricksGlusterFS
In this Gluster for Geeks technical webinar, Jacob Shucart, Senior Systems Engineer, will provide useful tips and tricks to make a Gluster cluster meet your performance requirements. He will review considerations for all different phases including planning, configuration, implementation, tuning, and benchmarking.
Topics covered will include:
• Protocols (CIFS, NFS, GlusterFS)
• Hardware configuration
• Tuning parameters
• Performance benchmarks
This presentation provides a basic overview of Ceph, upon which SUSE Storage is based. It discusses the various factors and trade-offs that affect the performance and other functional and non-functional properties of a software-defined storage (SDS) environment.
Revisiting CephFS MDS and mClock QoS SchedulerYongseok Oh
This presents the CephFS performance scalability and evaluation results. Specifically, it addresses some technical issues such as multi core scalability, cache size, static pinning, recovery, and QoS.
Windows Server 2012 R2 Software-Defined StorageAidan Finn
In this presentation I taught attendees how to build a Scale-Out File Server (SOFS) using Windows Server 2012 R2, JBODs, Storage Spaces, Failover Clustering, and SMB 3.0 Networking, suitable for storing application data such as Hyper-V and SQL Server.
How to implement PassKeys in your applicationMarian Marinov
PassKeys is relatively new way of authentication. This presentation aims to provide a bit of guidance on how you can implement them in your own application.
Management of system administrators and devops teams is different then managing Developers.
This presentation shows key differences and what to worry about :)
MySQL security is not trivial. This presentation will walk you trough some of the more important decisions you have to take, when configuring a MySQL server instance
Immunizing Image Classifiers Against Localized Adversary Attacksgerogepatton
This paper addresses the vulnerability of deep learning models, particularly convolutional neural networks
(CNN)s, to adversarial attacks and presents a proactive training technique designed to counter them. We
introduce a novel volumization algorithm, which transforms 2D images into 3D volumetric representations.
When combined with 3D convolution and deep curriculum learning optimization (CLO), itsignificantly improves
the immunity of models against localized universal attacks by up to 40%. We evaluate our proposed approach
using contemporary CNN architectures and the modified Canadian Institute for Advanced Research (CIFAR-10
and CIFAR-100) and ImageNet Large Scale Visual Recognition Challenge (ILSVRC12) datasets, showcasing
accuracy improvements over previous techniques. The results indicate that the combination of the volumetric
input and curriculum learning holds significant promise for mitigating adversarial attacks without necessitating
adversary training.
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...Dr.Costas Sachpazis
Terzaghi's soil bearing capacity theory, developed by Karl Terzaghi, is a fundamental principle in geotechnical engineering used to determine the bearing capacity of shallow foundations. This theory provides a method to calculate the ultimate bearing capacity of soil, which is the maximum load per unit area that the soil can support without undergoing shear failure. The Calculation HTML Code included.
About
Indigenized remote control interface card suitable for MAFI system CCR equipment. Compatible for IDM8000 CCR. Backplane mounted serial and TCP/Ethernet communication module for CCR remote access. IDM 8000 CCR remote control on serial and TCP protocol.
• Remote control: Parallel or serial interface.
• Compatible with MAFI CCR system.
• Compatible with IDM8000 CCR.
• Compatible with Backplane mount serial communication.
• Compatible with commercial and Defence aviation CCR system.
• Remote control system for accessing CCR and allied system over serial or TCP.
• Indigenized local Support/presence in India.
• Easy in configuration using DIP switches.
Technical Specifications
Indigenized remote control interface card suitable for MAFI system CCR equipment. Compatible for IDM8000 CCR. Backplane mounted serial and TCP/Ethernet communication module for CCR remote access. IDM 8000 CCR remote control on serial and TCP protocol.
Key Features
Indigenized remote control interface card suitable for MAFI system CCR equipment. Compatible for IDM8000 CCR. Backplane mounted serial and TCP/Ethernet communication module for CCR remote access. IDM 8000 CCR remote control on serial and TCP protocol.
• Remote control: Parallel or serial interface
• Compatible with MAFI CCR system
• Copatiable with IDM8000 CCR
• Compatible with Backplane mount serial communication.
• Compatible with commercial and Defence aviation CCR system.
• Remote control system for accessing CCR and allied system over serial or TCP.
• Indigenized local Support/presence in India.
Application
• Remote control: Parallel or serial interface.
• Compatible with MAFI CCR system.
• Compatible with IDM8000 CCR.
• Compatible with Backplane mount serial communication.
• Compatible with commercial and Defence aviation CCR system.
• Remote control system for accessing CCR and allied system over serial or TCP.
• Indigenized local Support/presence in India.
• Easy in configuration using DIP switches.
Final project report on grocery store management system..pdfKamal Acharya
In today’s fast-changing business environment, it’s extremely important to be able to respond to client needs in the most effective and timely manner. If your customers wish to see your business online and have instant access to your products or services.
Online Grocery Store is an e-commerce website, which retails various grocery products. This project allows viewing various products available enables registered users to purchase desired products instantly using Paytm, UPI payment processor (Instant Pay) and also can place order by using Cash on Delivery (Pay Later) option. This project provides an easy access to Administrators and Managers to view orders placed using Pay Later and Instant Pay options.
In order to develop an e-commerce website, a number of Technologies must be studied and understood. These include multi-tiered architecture, server and client-side scripting techniques, implementation technologies, programming language (such as PHP, HTML, CSS, JavaScript) and MySQL relational databases. This is a project with the objective to develop a basic website where a consumer is provided with a shopping cart website and also to know about the technologies used to develop such a website.
This document will discuss each of the underlying technologies to create and implement an e- commerce website.
Cosmetic shop management system project report.pdfKamal Acharya
Buying new cosmetic products is difficult. It can even be scary for those who have sensitive skin and are prone to skin trouble. The information needed to alleviate this problem is on the back of each product, but it's thought to interpret those ingredient lists unless you have a background in chemistry.
Instead of buying and hoping for the best, we can use data science to help us predict which products may be good fits for us. It includes various function programs to do the above mentioned tasks.
Data file handling has been effectively used in the program.
The automated cosmetic shop management system should deal with the automation of general workflow and administration process of the shop. The main processes of the system focus on customer's request where the system is able to search the most appropriate products and deliver it to the customers. It should help the employees to quickly identify the list of cosmetic product that have reached the minimum quantity and also keep a track of expired date for each cosmetic product. It should help the employees to find the rack number in which the product is placed.It is also Faster and more efficient way.
Welcome to WIPAC Monthly the magazine brought to you by the LinkedIn Group Water Industry Process Automation & Control.
In this month's edition, along with this month's industry news to celebrate the 13 years since the group was created we have articles including
A case study of the used of Advanced Process Control at the Wastewater Treatment works at Lleida in Spain
A look back on an article on smart wastewater networks in order to see how the industry has measured up in the interim around the adoption of Digital Transformation in the Water Industry.
Saudi Arabia stands as a titan in the global energy landscape, renowned for its abundant oil and gas resources. It's the largest exporter of petroleum and holds some of the world's most significant reserves. Let's delve into the top 10 oil and gas projects shaping Saudi Arabia's energy future in 2024.
Hierarchical Digital Twin of a Naval Power SystemKerry Sado
A hierarchical digital twin of a Naval DC power system has been developed and experimentally verified. Similar to other state-of-the-art digital twins, this technology creates a digital replica of the physical system executed in real-time or faster, which can modify hardware controls. However, its advantage stems from distributing computational efforts by utilizing a hierarchical structure composed of lower-level digital twin blocks and a higher-level system digital twin. Each digital twin block is associated with a physical subsystem of the hardware and communicates with a singular system digital twin, which creates a system-level response. By extracting information from each level of the hierarchy, power system controls of the hardware were reconfigured autonomously. This hierarchical digital twin development offers several advantages over other digital twins, particularly in the field of naval power systems. The hierarchical structure allows for greater computational efficiency and scalability while the ability to autonomously reconfigure hardware controls offers increased flexibility and responsiveness. The hierarchical decomposition and models utilized were well aligned with the physical twin, as indicated by the maximum deviations between the developed digital twin hierarchy and the hardware.
Student information management system project report ii.pdfKamal Acharya
Our project explains about the student management. This project mainly explains the various actions related to student details. This project shows some ease in adding, editing and deleting the student details. It also provides a less time consuming process for viewing, adding, editing and deleting the marks of the students.
Student information management system project report ii.pdf
Comparison of foss distributed storage
1. Talk Title Here
Author Name, Company
Comparison of FOSS
distributed filesystems
Marian “HackMan” Marinov, SiteGround
2. Who am I?
• Chief System Architect of SiteGround
• Sysadmin since 96
• Teaching Network Security & Linux System
Administration in Sofia University
• Organizing biggest FOSS conference in Bulgaria
- OpenFest
3. • We are a hosting provider
• We needed shared filesystem between VMs and
containers
• Different size of directories and relatively small
files
• Sometimes milions of files in a single dir
Why were we considering shared FS?
4. • We started with DBRD + OCFS2
• I did a small test of cLVM + ATAoE
• We tried DRBD + OCFS2 for MySQL clusters
• We then switched to GlusterFS
• Later moved to CephFS
• and finally settled on good old NFS :)
but with the storage on Ceph RBD
The story of our storage endeavors
5. • CephFS
• GlusterFS
• MooseFS
• OrangeFS
• BeeGFS - no stats
I haven't played with Lustre or AFS
And because of our history with OCFS2 and GFS2 I skipped them
Which filesystems were tested
7. What did I test?
• Setup
• Fault tollerance
• Resource requirements (client & server)
• Capabilities (Redundancy, RDMA, caching)
• FS performance (creation, deletion, random
read, random write)
8. Complexity of the setup
• CephFS - requires a lot of knowledge and time
• GlusterFS - relatively easy to setup and install
• OrangeFS - extremely easy to do basic setup
• MooseFS - extremely easy to do basic setup
• DRBD + NFS - very complex setup if you want HA
• BeeGFS -
9. CephFS
• Fault tollerant by desing...
– until all MDSes start crashing
● single directory may crash all MDSes
● MDS behind on trimming
● Client failing to respond to cache pressure
● Ceph has redundancy but lacks RDMA support
10. CephFS
• Uses a lot of memory on the MDS nodes
• Not suitable to run on the same machines as the
compute nodes
• Small number of nodes 3-5 is a no go
11. CephFS tunning
• Placement Groups(PG)
• mds_log_max_expiring &
mds_log_max_segments fixes the problem with
trimming
• When you have a lot of inodes, increasing
mds_cache_size works but increases the
memory usage
12. CephFS fixes
• Client XXX failing to respond to cache pressure
● This issue is due to the current working set size of the
inodes.
The fix for this is setting the mds_cache_size in
/etc/ceph/ceph.conf accordingly.
ceph daemon mds.$(hostname) perf dump | grep inode
"inode_max": 100000, - max cache size value
"inodes": 109488, - currently used
13. CephFS fixes
• Client XXX failing to respond to cache pressure
● Another “FIX” for the same issue is to login to current active
MDS and run:
● This way it will change currnet active MDS to another server
and will drop inode usage
/etc/init.d/ceph restart mds
14. GlusterFS
• Fault tollerance
– in case of network hikups sometimes the mount may
become inaccesable and requires manual remount
– in case one storage node dies, you have to manually
remount if you don't have local copy of the data
• Capabilities - local caching, RDMA and data
Redundancy are supported. Offers different
ways for data redundancy and sharding
15. GlusterFS
• High CPU usage for heavily used file systems
– it required a copy of the data on all nodes
– the FUSE driver has a limit of the number of
small operations, that it can perform
• Unfortunatelly this limit was very easy to be
reached by our customers
16. GlusterFS Tunning
● use distributed and replicated volumes
instead of only replicated
– gluster volume create VOL replica 2 stripe 2 ....
● setup performance parameters
17. GlusterFS Tunning
volume set VOLNAME performance.cache-refresh-timeout 3
volume set VOLNAME performance.io-thread-count 8
volume set VOLNAME performance.cache-size 256MB
volume set VOLNAME performance.cache-max-file-size 300KB
volume set VOLNAME performance.cache-min-file-size 0B
volume set VOLNAME performance.readdir-ahead on
volume set VOLNAME performance.write-behind-window-size 100KB
18. GlusterFS Tunning
volume set VOLNAME features.lock-heal on
volume set VOLNAME cluster.self-heal-daemon enable
volume set VOLNAME cluster.metadata-self-heal on
volume set VOLNAME cluster.consistent-metadata on
volume set VOLNAME cluster.stripe-block-size 100KB
volume set VOLNAME nfs.disable on
20. MooseFS
• Reliability with multiple masters
• Multiple metadata servers
• Multiple chunkservers
• Flexible replication (per directory)
• A lot of stats and a web interface
22. OrangeFS
• No native redundancy uses corosync/pacemaker
for HA
• Adding new storage servers requires stopping of
the whole cluster
• It was very easy to break it
23. Ceph RBD + NFS
• the main tunning goes to NFS, by using the
cachefilesd
• it is very important to have cache for both
accessed and missing files
• enable FS-Cache by using "-o fsc" mount option
24. Ceph RBD + NFS
• verify that your mounts are with enabled cache:
•
# cat /proc/fs/nfsfs/volumes
NV SERVER PORT DEV FSID FSC
v4 aaaaaaaa 801 0:35 17454aa0fddaa6a5:96d7706699eb981b yes
v4 aaaaaaaa 801 0:374 7d581aa468faac13:92e653953087a8a4 yes
29. Sysctl tunning
# turn off selective ACK and timestamps
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_low_latency = 1
# scalable or bbr
net.ipv4.tcp_congestion_control = scalable
30. Sysctl tunning
# Increase system IP port range to allow for more
concurrent connections
net.ipv4.ip_local_port_range = 1024 65000
# OS tunning
vm.swappiness = 0
# Increase system file descriptor limit
fs.file-max = 65535
31. Stats
• For small files, network BW is not a problem
• However network latency is a killer :(