Batch processing allows jobs to run without manual intervention by shifting processing to less busy times. It avoids idling computing resources and allows higher overall utilization. Batch processing provides benefits like prioritizing batch and interactive work. The document then discusses different approaches to batch processing like dedicating all resources to it or sharing resources. It outlines challenges like systems being unavailable during batch processing. The rest of the document summarizes Whiptail's flash storage solutions for accelerating workloads and reducing costs and resources compared to HDDs.
Right-Sizing your SQL Server Virtual Machineheraflux
Virtualizing your top-tier production SQL Servers is not as easy as P2V’ing it. Sometimes allocating more resources to the VM is the wrong approach, and getting it wrong will silently hurt performance. What is the most effective method for determining the ‘right’ amount of resources to allocate? What happens if the workload changes a month from now?
The methods for understanding the performance of your mission-critical SQL Servers gathered over the past ten years of SQL Server virtualization will be addressed, and valuable processes for performance statistic collection and analysis will be displayed. Come learn how to properly ‘right-size’ the resources allocated to a VM, improve the performance of your SQL Servers, and keep it maximized well into the future.
Scaling Up and Out your Virtualized SQL Serversheraflux
Scaling up a single SQL Server instance can be tough. Scaling up hundreds or thousands is tougher. Now virtualize them all. Whew! But… does it have to be harder when virtualized? Could it be easier than when physical? This session will explore the use of virtualization technologies to help augment and improve SQL Server’s native capabilities to help you better scale up for a single intense workload and scale out for many such workloads in the same environment. Come learn valuable tips and tricks that you can bring back to your organization on topics such as workload characteristic analysis, horizontal versus vertical scalability, common pitfalls and ways around them, performance optimization, VM sizing, and more!
Session source: IT/Dev Connections conference, 8/2014
This presentation answer a lot of your questions about PostgreSQL and the Red Hat Cluster Suite.
It reviews how you can create failover/standby capabilities with the following activities:
General PostgreSQL clustering options
Overview of Red Hat Cluster Service
Identification of candidate databases for clustering
Identification of hardware for clustering
Analysis of uptime requirements and data latency
Implementation of clustering
Testing of clustering
PostgreSQL installation tips for RHCS
Right-Sizing your SQL Server Virtual Machineheraflux
Virtualizing your top-tier production SQL Servers is not as easy as P2V’ing it. Sometimes allocating more resources to the VM is the wrong approach, and getting it wrong will silently hurt performance. What is the most effective method for determining the ‘right’ amount of resources to allocate? What happens if the workload changes a month from now?
The methods for understanding the performance of your mission-critical SQL Servers gathered over the past ten years of SQL Server virtualization will be addressed, and valuable processes for performance statistic collection and analysis will be displayed. Come learn how to properly ‘right-size’ the resources allocated to a VM, improve the performance of your SQL Servers, and keep it maximized well into the future.
Scaling Up and Out your Virtualized SQL Serversheraflux
Scaling up a single SQL Server instance can be tough. Scaling up hundreds or thousands is tougher. Now virtualize them all. Whew! But… does it have to be harder when virtualized? Could it be easier than when physical? This session will explore the use of virtualization technologies to help augment and improve SQL Server’s native capabilities to help you better scale up for a single intense workload and scale out for many such workloads in the same environment. Come learn valuable tips and tricks that you can bring back to your organization on topics such as workload characteristic analysis, horizontal versus vertical scalability, common pitfalls and ways around them, performance optimization, VM sizing, and more!
Session source: IT/Dev Connections conference, 8/2014
This presentation answer a lot of your questions about PostgreSQL and the Red Hat Cluster Suite.
It reviews how you can create failover/standby capabilities with the following activities:
General PostgreSQL clustering options
Overview of Red Hat Cluster Service
Identification of candidate databases for clustering
Identification of hardware for clustering
Analysis of uptime requirements and data latency
Implementation of clustering
Testing of clustering
PostgreSQL installation tips for RHCS
Hardware planning & sizing for sql serverDavide Mauri
Purchasing a dedicated server to SQL Server is still a necessary operation. The cloud is a great choice but if you need to create a data warehouse of non-trivial size or if you have the need for optimal performance and control of your production database server, the choice of on-premise server is still an optimal choice. So, how not to throw away money on unnecessary hardware? In this session we will see how each component works together to form a balanced hardware (this is the key word!), without bottlenecks, maximizing the investment made. We'll talk about SAN, CPU, HBA, Fibre Channel, Memory and everything you thought you knew well...
OpenStack is rapidly gaining popularity with businesses as they realize the benefits of a private cloud architecture. This presentation was delivered by Dave Page, Chief Architect, Tools & Installers at EnterpriseDB & PostgreSQL Core Team member during PG Open 2014. He addressed some of the common components of OpenStack deployments, how they can affect Postgres servers, and how users might best utilize some of the features they offer when deploying Postgres, including:
• Different configurations for the Nova compute service
• Use of the Cinder block store
• Virtual networking options with Neutron
• WAL archiving with the Swift object store
My experience with embedding PostgreSQLJignesh Shah
At my current company, we embed PostgreSQL based technologies in various applications shipped as shrink-wrapped software. In this session we talk about the experience of embedding PostgreSQL where it is not directly exposed to end-user and the issues encountered on how they were resolved.
We will talk about business reasons,technical architecture of deployments, upgrades, security processes on how to work with embedded PostgreSQL databases.
Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...Виталий Стародубцев
##Что такое Storage Replica
##Архитектура и сценарии
##Синхронная и асинхронная репликация
##Междисковая, межсерверная, внутрикластерная и межкластерная репликация
##Дизайн и проектирование Storage Replica
##Нововведения в Windows Server 2016 TP5
##Графический интерфейс управления, и другие возможности - демонстрация и планы развития
##Интеграция Storage Replica с Storage Spaces Direct
Master VMware Performance and Capacity ManagementIwan Rahabok
12 Sep 2016 update: See this http://virtual-red-dot.info/operationalize-sddc-program-2/ for details.
-------------
Based on the book http://virtual-red-dot.info/performance-and-capacity-management/
Master performance and capacity management of VMware SDDC
C-Drive 2009 presentation by Scott DesBles about how Compellent's Data Instant Replay and Data Progression work together to create an efficient data storage system.
Linux internals for Database administrators at Linux Piter 2016PostgreSQL-Consulting
Input-output performance problems are on every day agenda for DBAs since the databases exist. Volume of data grows rapidly and you need to get your data fast from the disk and moreover - fast to the disk. For most databases there is a more or less easy to find checklist of recommended Linux settings to maximize IO throughput. In most cases that checklist is good enough. But it is always better to understand how it works, especially if you run into some corner-cases. This talk is about how IO in Linux works, how database pages travel from disk level to database own shared memory and back and what kind of mechanisms exist to control this. We will discuss memory structures, swap and page-out daemons, filesystems, schedullers and IO methods. Some fundamental differences in IO approaches between PostgreSQL, Oracle and MySQL will be covered.
Various HA and DR setups for Postgres Plus Advanced Server -
Active – Passive OS HA Clustering
Log Shipping Replication (Hot Standby Mode)
Hot Streaming Replication (Hot Standby Mode)
EDB Postgres Plus Failover Manager
HA with read scaling (with pg-pool)
xDB Single Master Replication (SMR)
xDB Multi Master Replication (MMR)
Use Cases
План вебинара:
##Что такое Storage Spaces Direct?
##Сценарии использования Storage Spaces.
##Описание минимальных требований для Storage Spaces.
##Как настроить Windows Server 2016 Spaces Direct для работы с локальными дисками сервера?
##Что такое Storage Replica?
##Разница подходов синхронной и асинхронной репликации.
##Какие технологии репликации для каких задач использовать (DFS-R, Hyper-V Repica, SQL AlwaysOn, Exchange DAG) - и как это комбинируется с новыми возможностями Windows Server 2016?
##Что такое ReFS и чем она отличается в Server 2016 от предыдущих изданий ОС?
##Что даёт использование ReFS для виртуальных машин Hyper-V. Сценарии и возможности.
##Общие изменения Storage технологий в Windows Server 2016.
Hardware planning & sizing for sql serverDavide Mauri
Purchasing a dedicated server to SQL Server is still a necessary operation. The cloud is a great choice but if you need to create a data warehouse of non-trivial size or if you have the need for optimal performance and control of your production database server, the choice of on-premise server is still an optimal choice. So, how not to throw away money on unnecessary hardware? In this session we will see how each component works together to form a balanced hardware (this is the key word!), without bottlenecks, maximizing the investment made. We'll talk about SAN, CPU, HBA, Fibre Channel, Memory and everything you thought you knew well...
OpenStack is rapidly gaining popularity with businesses as they realize the benefits of a private cloud architecture. This presentation was delivered by Dave Page, Chief Architect, Tools & Installers at EnterpriseDB & PostgreSQL Core Team member during PG Open 2014. He addressed some of the common components of OpenStack deployments, how they can affect Postgres servers, and how users might best utilize some of the features they offer when deploying Postgres, including:
• Different configurations for the Nova compute service
• Use of the Cinder block store
• Virtual networking options with Neutron
• WAL archiving with the Swift object store
My experience with embedding PostgreSQLJignesh Shah
At my current company, we embed PostgreSQL based technologies in various applications shipped as shrink-wrapped software. In this session we talk about the experience of embedding PostgreSQL where it is not directly exposed to end-user and the issues encountered on how they were resolved.
We will talk about business reasons,technical architecture of deployments, upgrades, security processes on how to work with embedded PostgreSQL databases.
Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...Виталий Стародубцев
##Что такое Storage Replica
##Архитектура и сценарии
##Синхронная и асинхронная репликация
##Междисковая, межсерверная, внутрикластерная и межкластерная репликация
##Дизайн и проектирование Storage Replica
##Нововведения в Windows Server 2016 TP5
##Графический интерфейс управления, и другие возможности - демонстрация и планы развития
##Интеграция Storage Replica с Storage Spaces Direct
Master VMware Performance and Capacity ManagementIwan Rahabok
12 Sep 2016 update: See this http://virtual-red-dot.info/operationalize-sddc-program-2/ for details.
-------------
Based on the book http://virtual-red-dot.info/performance-and-capacity-management/
Master performance and capacity management of VMware SDDC
C-Drive 2009 presentation by Scott DesBles about how Compellent's Data Instant Replay and Data Progression work together to create an efficient data storage system.
Linux internals for Database administrators at Linux Piter 2016PostgreSQL-Consulting
Input-output performance problems are on every day agenda for DBAs since the databases exist. Volume of data grows rapidly and you need to get your data fast from the disk and moreover - fast to the disk. For most databases there is a more or less easy to find checklist of recommended Linux settings to maximize IO throughput. In most cases that checklist is good enough. But it is always better to understand how it works, especially if you run into some corner-cases. This talk is about how IO in Linux works, how database pages travel from disk level to database own shared memory and back and what kind of mechanisms exist to control this. We will discuss memory structures, swap and page-out daemons, filesystems, schedullers and IO methods. Some fundamental differences in IO approaches between PostgreSQL, Oracle and MySQL will be covered.
Various HA and DR setups for Postgres Plus Advanced Server -
Active – Passive OS HA Clustering
Log Shipping Replication (Hot Standby Mode)
Hot Streaming Replication (Hot Standby Mode)
EDB Postgres Plus Failover Manager
HA with read scaling (with pg-pool)
xDB Single Master Replication (SMR)
xDB Multi Master Replication (MMR)
Use Cases
План вебинара:
##Что такое Storage Spaces Direct?
##Сценарии использования Storage Spaces.
##Описание минимальных требований для Storage Spaces.
##Как настроить Windows Server 2016 Spaces Direct для работы с локальными дисками сервера?
##Что такое Storage Replica?
##Разница подходов синхронной и асинхронной репликации.
##Какие технологии репликации для каких задач использовать (DFS-R, Hyper-V Repica, SQL AlwaysOn, Exchange DAG) - и как это комбинируется с новыми возможностями Windows Server 2016?
##Что такое ReFS и чем она отличается в Server 2016 от предыдущих изданий ОС?
##Что даёт использование ReFS для виртуальных машин Hyper-V. Сценарии и возможности.
##Общие изменения Storage технологий в Windows Server 2016.
Presentation & discussion around low-level graphics APIs. This was a quickly made presentation that I put together for a discussion with Intel and fellow ISVs, thought it could be worth sharing
Optimizing the Upstreaming Workflow: Flexibly Scale Storage for Seismic Proce...Avere Systems
Of all the applications in the oil and gas industry's upstream workflow, those involved in seismic processing place the greatest demand on storage. Pre-stack and post-stack migration, velocity modeling, and other processing steps are challenging even the highest performance NAS systems. In this Webinar, we discuss meeting these demands with accelerated performance, reduced cost, and a streamlined workflow.
Why new hardware may not make Oracle databases fasterSolarWinds
How can you know if hardware is the right answer to your Oracle database performance issues? How can you know for sure which hardware components will have the biggest impact? As a DBA or database developer, you should know that you can gain significant performance improvements without the time, money and risk associated with providing the latest server or flash storage array.
Learn why new hardware may not make your Oracle database faster and what you can do instead.
Similar to Storage and performance- Batch processing, Whiptail (20)
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!SOFTTECHHUB
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
3. BATCH PROCESSING
Batch processing is execution of a series of programs ("jobs") on a
computer without manual intervention.
Batch processing has these benefits:
• It can shift the time of job processing to when the computing resources
are less busy.
• It avoids idling the computing resources with minute-by-minute manual
intervention and supervision.
• By keeping high overall rate of utilization, it amortizes the computer,
especially an expensive one.
• It allows the system to use different priorities for batch and interactive
work.
3
4. BATCH PROCESSING
• Systems Access Unavailable
– All Resources dedicated to Batch Processing
– Historically this is how people have done
things because of the load on the systems
• Running whilst System is Available
– Shared resources for the Batch as well as
normal usage
– Complex architectures and huge investments
to make the normal usage usable.
4
5. THE PROBLEM WITH PERFORMANCE
Accelerate
Workloads
--------
Decrease
Costs
-Accelerate Productivity
-Scale
-Total Costs
A “More Assets” Problem
Resources
Resources
Storage Decisions
Database
60
drives
3 TB
11k IOPS
0% Write
72
drives
Or
discs
Or
cache
Or
arrays
13k IOPS
25% Write
96
drives
Or
more
discs
Or
more
cache
Or
more
arrays
5
Batch
OLTP
Analytics
VDI
HPC
Email
Video
A Demand Solution
Speed
Productivity
3 TB SQL – 17 k IOPS
Total Costs
And
12
TB
Batch – 20 k IOPS
And
OLTP – 10 k IOPS
And…
17k IOPS
80% Write
Workload
Workload
6. SINCE 1956, HDDS HAVE DEFINED
APPLICATION PERFORMANCE
Speed
Design
6
• 10s of MB/s Data
Transfer Rates
• 100s of Write / Read
operation per second
• .001s Latency (ms)
• Motors
• Spindles
• High Energy
Consumption
7. FLASH ENABLES APPLICATIONS TO
WRITE FASTER
Speed
Design
7
• 100s of MB/s data
transfer rates
• 1000s of Write or Read
operations per second
• .000001 Latency (µs)
• Silicon
• MLC/SLC NAND
• Low energy
consumption
8. USE OF FLASH – HOST SIDE – PCIE /
FLASH DRIVE DAS
• PCIe
–
–
–
–
Very fast and low latency
Expensive per GB
No redundancy
CPU/Memory stolen from host
• Flash SATA/SAS
– More cost effective
– Cant get more than 2 drives per blade
– Unmanaged can have perf / endurance issues
8
8
9. USE OF FLASH – ARRAY BASED
CACHE / TIERING
9
• Array flash cache
– Typically read only
– PVS already caches most reads
– Effectiveness limited by storage array designed for hard disks
• Automated storage tiering
– “Promotes” hot blocks into flash tier
– Only effective for READ
– Cache misses still result in “media” reads
9
10. USE OF FLASH – FLASH IN THE
TRADITIONAL ARRAY
10
• Flash in a traditional array
–
–
–
–
–
Typically uses SLC or eMLC media
High cost per GB
Array is not designed for flash media
Unmanaged will result in poor random write performance
Unmanaged will result in poor endurance
10
11. USE OF FLASH – FLASH IN THE ALL
FLASH ARRAY
•
•
•
•
•
•
Optimized to sustain High Write and Read throughput
High bandwidth and IOPS. Low latency.
Multi-protocol
LUN Tunable performance
Software designed to enhance lower cost NAND MLC
Flash by optimizing High Write throughput while
substantially reducing wear
• RAID protection and replication
11
13. NAND FLASH FUNDAMENTALS:
13
HDD WRITE PROCESS REVIEW
Rewritten data block
4K data blocks
A physical HDD is a bit-addressable medium!
Virtually limitless write and rewrite
capabilities.
14. STANDARD NAND FLASH ARRAY
WRITE I/O
Fabric
ISCSI
FC
SRP
1. Write request from host
passes over fabric through
HBAs.
2. Write request passes
through the transport stack
to RAID.
Unified Transport
RAID
HBA
NAND
Flash x 8
HBA
NAND
Flash x8
HBA
NAND
Flash x8
3. Request is written to
media.
14
15. NAND FLASH FUNDAMENTALS:
FLASH WRITE PROCESS
15
2MB NAND Page
1. NAND Page contents are
read to a buffer.
2. NAND Page is erased
(aka, “flashed”).
3. Buffer is written back
with previous data and any
changed or new blocks –
including zeroes.
16. UNDERSTANDING
ENDURANCE/RANDOM WRITE
PERFORMANCE
16
Endurance
Each cell has physical limits (dielectric breakdown) 2K-5K PE’s
Time to erase a block is non-deterministic (2-6 ms)
Program time is fairly static based on geometry
Failure to control write amplification *will* cause wear out in a
short amount of time
Desktop workload is one of the worst for write amplification
Most writes are 4-8KB
• Random Write Performance
– Write amplification not only causes wear out issues, it also
creates unnecessary delays in small random write workloads.
– What is the point of higher cost flash storage with latency
between 2-5ms?
16
17. RACERUNNER OS:
17
DESIGN AND OPERATION
Fabric
iSCSI
FC
SRP
Unified Transport
RaceRunner
BlockTranslation Layer:
Alignment | Linearization
Enhanced RAID
NAND SSD
x8
HBA
NAND SSD
x8
2. Write request passes
through the transport stack to
BTL.
3. Incoming blocks are
aligned to native NAND page
size.
Data integrity Layer
HBA
1. Write request from host
passes over fabric through
HBAs.
HBA
NAND SSD
x8
4. Request is written to
media.
18. THE DATA WAITING DAYS ARE OVER
ACCELA
1.5TB – 12TB
250,000 IOPS
1.9 GB/s Bandwidth
Scalability Path
INVICTA
2-6 Nodes
6TB-72TB
650,000 IOPS
7GB/s Bandwidth
INVICTA – INFINITY (Q1/13)
7-30 Nodes
21TB-360TB
800,000 – 4 Million IOPS
40GB/s Bandwidth
18
19. THE DATA WAITING DAYS ARE OVER
19
ACCELA
INVICTA
INVICTA INFINITY
Height
2U
6U-14U
16U-64U
Capacity
1.5TB-12TB
6TB-72TB
21TB-360TB
IOPS
Up to 250K
250K – 650K
800K – 4M
Bandwidth
Up to 1.9GB/Sec
Up to 7GB/Sec
Up to 40GB/Sec
Latency
120µs
220µs
250µs
Interfaces
2/4/8 Gbit/Sec FC
1/10 GBE
Infiniband
Protocols
FC, ISCSI, NFS, QDR
Features
RAID Protection & Hot Sparing
Async Replication
VAAI
Write Protection Buffer
Options
vCenter Plugin/INVICTA Node
Kit
RAID Protection and Hot Sparing
LUN Mirroring and LUN Striping
Async Replication
VAAI
Write Protection Buffer
vCenter
Plugin/INFINITY Switch
Kit
vCenter Plugin
20. MULTI-WORKLOAD
REFERENCE ARCHITECTURE
20
Mercury
Workload Engines
Workload Type
Workload Demand
Dell DVD Store
MS SQL Server
1200 Transactions Per
Second (Continuous)
4,000 IOPS
.05 GB/s
VMWare
View
600 Desktops Boot Storm
(2:30)
109,000 IOPS
.153 GB/s
Heavy OLTP Simulation
100% 4K Writes
(Continuous)
86,000 IOPS
.350 GB/s
Batch Report Simulation
100% 64K Reads
(Continuous)
16,000 IOPS
1 GB/s
SQLIO
MS SQL Server
• INVICTA
•
•
•
350,000 IOPS
3.5 GB/s
18 TB
• 8 Servers
In 2012 Mercury traveled to Barcelona, New York, San
Francisco, Santa Clara, and Seattle demonstrating the
ability to accelerate multiple workloads on to Solid State
Storage.
215,000 IOPS
1.553 GB/s
Raid 5 HDD Equivalent = 3,800
RAID 10 HDD Equivalent = 2,000
21. FASTER GPS FLEET TRACKING
21
Tracks trucks 97% faster
Had to turn off Email systems to allow
extra resources to be allocated to
Batch Run which was taking longer
and longer and created massive
queue of messages
Replaced Hard Disk Drives with four
WHIPTAIL 3TB units and reclaimed
substantial datacenter space
Needed to improve workload
performance of write intensive
Oracle database supporting realtime truck fleet management
system
WHIPTAIL’s 1.9 GB/s WRITE
throughput and 250,000 WRITE IOPS
deliver dramatic performance
improvement in truck management
and monitoring
Workloads are now the fastest in the
enterprise. Query response times
decreased from 2:30 seconds to :05
seconds
22. WHAT WHIPTAIL CAN OFFER:
22
Throughput …..
1.9GB/s – 40GB/s
120µs
Power …………….
90% less
Floor Space …….
90% less
Cooling …………..
90% less
Endurance …….
7.5yrs Guaranteed
Making Decision faster ….
• Cost
250K – 4m
Latency ………….
• Performance
IOPS ………………
POA
Highly experienced - 250+ customers since 2009 for VDI, Database , Analytics etc…
Best in class performance at most competitive price
Disk drives were designed around capacity not speed. As a result write performance is poor. This poor performance has had a profound impact on how IT operates as a whole.
1. A NAND page is the minimal addressable write element a NAND page t 25nm geometry is between 4 and 8KB2. An ERASE-BLOCK is a grouping of NAND pages that can range anywhere from 128KB on a single die to 2MB when multiple die are striped3. You can write a NAND page individually, but you cannot RE-WRITE a page without bringing the entire block into a buffer modifying its contents, erasing the block and then re-writing the block
This leads a lot of people down the road of deploying small footprint servers or blades. Physical constraints of these platforms don’t allow for the room to get enough hard disks in a host to deploy enough spindles to handle the load.
Vendors who deploy Flash caching are aware of this and often deploy Flash as a READ only cache layer bypassing these challenges, but introduce two new ones: COST, and the dreaded cache miss.
But, unfortunately, once you start putting Flash drives in a standard array, you end up staring right back in to the eyes of the dragons we mentioned before. Endurance, random write performance, and cost all rear their heads very quickly.
1. A NAND page is the minimal addressable write element a NAND page t 25nm geometry is between 4 and 8KB2. An ERASE-BLOCK is a grouping of NAND pages that can range anywhere from 128KB on a single die to 2MB when multiple die are striped3. You can write a NAND page individually, but you cannot RE-WRITE a page without bringing the entire block into a buffer modifying its contents, erasing the block and then re-writing the block
First and foremost it has a physical endurance limit. You can only write to it X number of times, before error rates to unacceptable levels current MLS technology has a PE rating of 5,000. without managing the write cycle, it is very easy to exceed this limit due to what is called “write amplification.”
In2012 Mercury traveled Barcelona, New York, San Francisco, Santa Clara, And Seattle demonstrating the advantages of consolidating workloads on to Solid State Storage.