Uploaded byBob Roudebush
Availability Considerations for SQL Server
This document discusses various options for providing high availability and disaster recovery for SQL Server deployments. It describes database mirroring and failover clustering which provide redundancy at the database or instance level. It also discusses third party solutions like Neverfail which provide end-to-end protection of multi-tier applications and entire business services across local and remote sites. Neverfail leverages application awareness and automation to minimize downtime from both planned and unplanned outages.
More Related Content
Similar to Availability Considerations for SQL Server
Availability Considerations for SQL Server
- 1. Availability Considerations forSQL Server DeploymentsBob Roudebush, Neverfail1/27/2010
- 2. Neverfail Company OverviewOurFocusDelivering world-class Continuous Availability software for High Availability and Disaster RecoveryEnsuring Business Continuity and IT Services Continuity by reducing both planned and unplanned downtimeLeveraging Application Awareness, Intelligence and Automation to deliver the highest levels of uptimeCompany Background3,000+ customers in over 40 countriesSMB to Global Enterprises across all major industry verticalsWorldwide ReachNorth America, EMEA and Asia-PacificStrategic Relationships with Microsoft, VMware, Cisco, RIM, SolarWinds and others
- 3. Is There ABetter Option for Protecting SQL?Need for Disaster Recovery Strategies is Undeniable.61% of North American enterprises and 64% of European enterprises say that upgrading DR capabilities is either a critical or high priority for the next 12 months.1Many Only Protect Mission-Critical Applications – Why?The cost and complexity of the supporting infrastructure required at the recovery data center.1 Forrester’s Enterprise and SMB Hardware Survey, NA and Europe, Q3 20092 Source: Mid-market Business Continuity Survey (N=230)
- 4. Degrees of protectionfor information systems:So you need to make your SQL Server deployment bulletproof?Data Availability Continuum4
- 5. Business Case forSQL AvailabilityHigh AvailabilityDisaster RecoveryKeep business-critical applications availableSecondary:Server maintenanceProtect against loss of data centerSecondary:Application upgradesInfrastructure upgrades5
- 6. What Should YouProtect?Application Data StoresDatabasesFilesOther data repositoriesDatabase ServicesDBMS availability for applicationsApplication ServicesApplication availability for users and external systemsDatabases are the heart of most information systems;they deserve the highest affordable protection.
- 7. Big “D” orlittle “d” Disasters?Physical Infrastructure FailuresLogical Data FailuresStorage SubsystemDiskControllerNetworkServerPowerSite-wide OutagesOperator ErrorsDBMS interruptionDrops / DeletesApplication DefectsDBMS DefectsData Corruption
- 8. Database Backup /RecoveryTraditional backup typesFull backupDifferential backupTransaction log backupDisk is better than tapeFirst backup to disk (separate physical disk volume)Detect exceptions encountered during backupVerify backup filesCopy backup files to tape or remote diskBackup everything!External data storesSystem databases
- 9. Protecting User DatabasesIsn’t SufficientTechRepublic: “If the master database fails, Microsoft SQL Server can be brought to its knees.”1SQL Server does not protect system databasesmaster, model, temp, msdbMaster db is most criticalInstance-wide meta data, system configuration, location of database files, initialization info for SQL serverSQL Server can not start if master is unavailablemaster, model, temp, msdb1 http://articles.techrepublic.com.com/5100-10878_11-5025441.html
- 10. It’s the Recovery,Stupid. What’s The Process Like?Backup file setsFull baseline, differential, and transaction logsRetrieving backup filesOffsite storageTapeNetwork copyDependency on multiple people to get access to backup filesDoes It Meet Your SLAs?Recovery strategy depends on failure scenarioCreate comprehensive failure matrixDevise recovery strategy for each scenarioDoes worst-case recovery scenario fit within SLA parameters?Recovery Time / Recovery PointInclude future data growth in recovery plan
- 11. Evolution of ContinuousAvailabilityBusinessEmphasisContinuous AvailabilityNo User or Application Downtime, Application-Awareness, Automation, Local & Remote AvailabilityAvailabilityCentricApplicationEmphasisBasic FailoverClustering, Fault Tolerance, High AvailabilityData ReplicationSnapshot, Asynchronous, Synchronous, CDPDataEmphasisRecoveryCentricBackup & RestoreFile, Block, Full, Incremental, On-site, Off-site
- 12. Shift Focus toAvailabilityMinimize/Avoid service downtime – planned or unplanned.On failure, service interruption is brief or non-existentAutomatic failoverEliminate single points of failure (as affordable)Redundant componentsFault-tolerant servers
- 13. Transactional Log ShippingWarmstandby solutionDuplicate user databaseCopy transaction logs to standby server & restoreDatabase available for read-only accessUsers must disconnect for logs to be appliedTwo database licenses required if querying standbyManual application failoverSupported on standard hardwarePossible data loss (unapplied transactions)
- 14. SQL Database MirroringRedundancyat user database levelDuplicate copy of user databaseMirrored over private network channelMirror replays transactions from principalMultiple mirroring modes:High-safety: commit @ log on mirror; automatic failoverHigh-safety: commit @ log on mirror; manual failoverHigh-performance: commit when logged on principalRequires witness serverMirror-aware application client connectionProvided by client libraryDatabase connection string must specify both servers
- 15. SQL Server FailoverClusteringRedundancy at database instance levelAll databases fail over togetherShared copy of system databasesSingle data copy on shared storage deviceNo I/O overhead reducing throughputStorage unit is single point of failure for clusterAll database services are clusteredSQL Agent; Analysis Services; Full-Text engine, MS DTCAutomatic failover (up to minutes)DBMS accessed over virtual IPRequires hardware certified by Microsoft for Microsoft Cluster Service
- 16. Considerations for HA/DRDeploymentsHA complements backup and recovery strategyDoes not replace data recovery planApplication service availability is often determined by a network of interdependent servicesAvailability can be difficult to define (e.g. partial failures)Failure probability difficult to measure or computeAdditional Infrastructure Costs (Storage/WAN/Hardware)Increased system complexity could lead to lower service availability!Operator error a leading cause of availability issuesIncreased number/types of system componentsMore complex to configure and administer
- 17. Business ServiceBusiness-Central Approachto ProtectionTraditionally, servers of like functionality are grouped together for availability and protection, leading to multiple, standalone availability solutions and incomplete protection for the Business ServiceNeverfail’s unique Business-Centric approach manages and protects the completeBusiness Service, including: Load BalancingWeb ServerWeb Server?????Complete Protection for Business ServiceApplication ServerApplication ServerMS ClusteringDatabase MirroringDatabase ServerDatabase Server
- 18. Neverfail High-Level ArchitectureApplicationModulesMonitoringContinuityNeverfailContinuousAvailabilityDirectorApplicationAwareManagement Framework (AMF)ScopeDataWANSmart3rd PartyAdaptorsManage availabilityfrom a single point across a wide variety of heterogeneous environments.CLOUD
- 19. SecondaryPrimaryAppsAppsNeverfail HeartbeatNeverfail HeartbeatNeverfailChannelLAN or WANResync: Once the cause of the failure has been addressed, Neverfail will resynchronize the primary without any additional downtime.How Failover and Failback WorkStep #1: Install 2nd server with 2nd network interface card. This will be known as the “Neverfail Channel” used for Neverfail traffic only.Step #2: Run S.C.O.P.E. (Server Check Optimization & Performance Evaluation) Tool to check required components are available.Step #3: Run Neverfail setup to clone the Windows OS, applications and data to the secondary server.Step #4: Neverfail will begin replicating changes to both protected data and the application configuration in real-time.Failover: Neverfail will detect failure of the application and can automatically failover the application to the secondary server.Switchback: Once the data has been completely resynchronized, Neverfail will perform a switchback, returning the application to the operating state it was in prior to the failure.
- 20. High Availability andDisaster Recovery Deployment with WANSmart™Application ServerApplication ServerApplication ServerWANSmart™Neverfail ChannelNeverfail ChannelFlexible Deployment ScenariosProduction SiteDisaster Recovery SiteFlexible, P2P, P2V , V2P or V2V operationLocal and remote deploymentShared nothing architectureSpecific Microsoft Cluster, Hyper-V and vSphere Options LANLAN
- 21. How WANSmart™ WorksNeverfailWANSmart™ACTIVESERVERPASSIVE SERVERDuplicatesFoundWANRestoreDuplicatesData ChangesInterceptedCompression& De-duplicationTransmissionAcross WANDecompression& Re-constitutionData ChangesAppliedBandwidth optimization built-in Lower cost and latency reductionAdvanced de-duplication Up to 30x improvement
- 22. Comparison: SQL versusNeverfail for SQL1Basic monitoring only i.e. can the witness server connect to the database?2Requires synchronous mirroring and an additional witness server
- 23. Neverfail CustomersRIM BlackBerry®Enterprise Server and Lotus Domino“British American Tobacco’s mobile platforms have come to lie at the heart of our business use of technology. The simple fact is that Neverfail keeps that heart beating.” David Sampson, Head of Headquarters IT - British American TobaccoRIM BlackBerry® Enterprise Server and Microsoft® SQL Server“I have complete confidence that we now have a highly reliable solution in place which will keep our staff connected through any outage.”Darren Unwin IS Services Manager DB Schenker Rail (UK) LtdMicrosoft® Exchange and Microsoft® SQL Server“The Neverfail solution supports our rigorous requirements for business continuity while respecting our technical and budgetary goals.”PärPersson IT Manager, Stena Rederi AB Stena BulkMicrosoft® SharePoint Server“I am 100 percent confident that our entire legal team has continuous access to their business-critical documents anytime, anywhere.”Andy Jurczyk CIO Sonnenschein Nath & Rosenthal LLPRIM BlackBerry® Enterprise Server“Neverfail is unique. Its nearest competitor only delivers 50 percent of the functionality.”Timo Sillober Head of Office IT Vodafone Group Services GmbH
- 24. SummaryProtects End-to-End BusinessServicesNeverfail aggregates & protects any set of multi-tier components, including applications, servers, services, and it maps all their interdependenciesComplete Protection Against the Widest Range of FailuresLeverages Application Awareness, Intelligence and AutomationEliminates both planned and unplanned downtimeWAN Acceleration & DeduplicationEssential feature set to enable DR deploymentsHeterogeneous ProtectionPhysical and Virtual Environments (P2P, P2V, V2P, V2V)Any Combination of Physical Server and Storage Hardware
- 25.
- 26. High Availability DeploymentFlexibleDeployment ScenariosApplication ServerApplication ServerNeverfail ChannelProduction SiteFlexible, P2P, P2V , V2P or V2V operationLocal and remote deploymentShared nothing architectureSpecific Microsoft Cluster, Hyper-V and vSphere Options LAN
- 27. Disaster Recovery Deploymentwith WANSmart™Flexible Deployment ScenariosApplication ServerApplication ServerWANSmart™Neverfail ChannelNeverfail ChannelProduction SiteDisaster Recovery SiteFlexible, P2P, P2V , V2P or V2V operationLocal and remote deploymentShared nothing architectureSpecific Microsoft Cluster, Hyper-V and vSphere Options LANLAN
- 28. ClusterProtector Deployment withWANSmart™Application ServerApplication ServerApplication ServerWANSmart™Neverfail ChannelDisaster Recovery SiteProduction SiteFlexible, P2P, P2V , V2P or V2V operationLocal and remote deploymentShared nothing architectureSpecific Microsoft Cluster, Hyper-V and vSphere Options LANLANSANCluster StorageFlexible Deployment Scenarios
- 29. Virtualization-Based Deployments (P2V,V2V or V2P)WANSmart™Neverfail ChannelDisaster Recovery SiteFlexible, P2P, P2V , V2P or V2V operationLocal and remote deploymentShared nothing architectureSpecific Microsoft Cluster, Hyper-V and vSphere Options LANFlexible Deployment ScenariosApplication ServerApplication ServerApplication ServerNeverfail ChannelProduction SiteLAN
- 30. Continuous Availability DirectorSharePointFarmWeb TierIndex ServerIISIISIISQuery ServerExec Calc ServerCentral AdminApplicationModulesVirtualizedApplication TierNeverfailContinuousAvailabilityDirectorAMFScope3rd PartyAdaptorsDatabase TierPhysicalSQL ServerSQL Server
- 31. SQL ServerSQL ServerCommerceBiztalkCustomBlackberryServerDomino ServerExchange ServerSymantec A/VIIS+MOSSIIS+MOSSIIS+MOSSIIS+MOSSIndex ServerIIS+MOSSIIS+MOSSIIS+MOSSIIS+MOSSFile ServerQuery ServerExec Calc ServerCentral AdminApplicationModulesSharePoint FarmMessaging SystemsWeb ApplicationsMonitoringContinuityWeb Front EndWeb Front EndWeb Front EndNeverfailContinuousAvailabilityDirectorAMFScopeOutlook Web AccessVirtualizedSQL ServerSQL ServerSQL ServerSQL ServerSQL ServerApplication TierApplication TierApplication TierDataWANSmart3rd PartyAdaptorsVirtualizedPhysicalCLOUDDatabase TierDatabase TierDatabase TierPhysicalContinuous Availability Director
- 32. Scope: user DBStandardhardwareOne SQL license (unless querying snapshots on mirror)Very fast failover (seconds)OS flexible (e.g. 32/64)Independent storageIndependent servicesReporting on mirrorGeographic separation OKSQL Server Options for HA ComparisonDatabase MirroringFailover ClusteringScope: DBMS InstanceCertified hardwareOne SQL license (unless querying snapshots on mirror)Automatic Failover (minutes)Enterprise OSShared storageClustered servicesStandby not availableServers usually co-located32
Editor's Notes
- #2 This PowerPoint Template includes a series of slide masters with predefined layouts and color schemes for formatting slides Slide Masters are displayed when you right click on a slide and select Layout from menu
- #3 We are a global software company with operations split into 3 key territories: Asia Pac, EMEA and the Americas.We have sales offices in Sydney-Australia, Frankfurt-Germany, Theale-UK, Austin-Texas-USA and Dubai Internet City-UAE.We have a Development centre in Stirling, Scotland and a support and QA centre in Cluj, Romania. The US is our biggest market representing some 65% of the business.We now have around 2000 customers worldwide.We are a Microsoft Gold Partner in the US and UK and the Neverfail product is Microsoft Certified.The company started as a Disaster Recovery consultancy over 10 years ago and with time developed tools and techniques that it decided to turn into a software product, hiring developers from the Tandem non-stop kernel team to help bring the vision to fruition.
- #4 Colt's Manufacturing Company (CMC, formerly Colt's Patent Firearms Manufacturing Company) is a United Statesfirearms manufacturer originally founded in 1836.One of the first truly modern-style handguns, the Colt revolvers became known as "The Great Equalizer", because they could be loaded and fired by anyone, whereas most previous guns had required sufficient strength and dexterity. In theory, anyone who had a modern-style revolver was equal to anyone else, regardless of their relative physical abilities. This term has since come to be used for firearms in general, as awkward weapons like muzzle-loaded muskets became a thing of the past.
- #9 External Datastore Examples…Identity column seedsFull-text indexes(SQL Server 2000)LDAP entriesFile system objectsOther databases
- #15 Mirror WitnessWith mirroring, more than one server is required to decide on failoverWitness automates failover from primary to mirrorWatches database availabilityReports observations back to principal and mirrorRuns in separate SQL Server instance (Express is OK)Prevents “split brain” scenarioVery low resource consumptionCan be witness for multiple databasesNot a single point of failureWhat They DOSimple data replication (application data only – log shipping)Verify installation pre-requisitesData compression (SQL Server 2008 only)What They DON’T DOImplement easilyRequires a complete manual build of the secondary server, including all apps and patchesRequires a 3rd server (witness) for High Availability modeFailover seamlesslyEnd users will need to restart certain applications, unless they’re coded with ADO.NETMonitor the entire environmentAutomatic failover only supported in synchronous replication mode, Enterprise version onlyOnly protect at individual DB level (not all DBs on the server, nor SQL jobs, DTC, etc.)Application monitoring only checks for running services, so they won’t catch hung servicesNo network or performance monitoringNo protection for auxiliary applicationsReplicate registry changesOnly replicate data on disk, so application config changes won’t be copied acrossData rollbackComplete environment health checkMicrosoft employees (NY MTC) will recommend Neverfail in DR /large data scenarios
- #16 What They DOServer monitoring (heartbeat)Seamless high-availability failover and failbackWhat They DON’T DOImplement easilyRequires expensive SAN storage, which introduces a single point of failureMonitor the entire environmentApplication monitoring only checks if services are running, so they won’t realize if a service is hungNo network or performance monitoringNo protection for auxiliary applicationsProtect against disastersOnly by implementing another cluster and SAN replication technology can DR be achievedData rollback
- #18 Traditionally, servers of like functionality are grouped together for availability/protectionNeverfail takes a Business-Centric approach and manages and protects the completeBusiness application.
- #19 Neverfail’s Continuous Availability Suite ensures that end users remain connected to their business-critical applications, regardless of the type of outage.Neverfail achieves this by protecting the entire application ecosystem, to provide a complete HA and DR solution, that spans physical and virtual systems, across multiple sites.Neverfail’s application-aware management framework (AMF) continuously monitors the health, state and performance of individual applications and services, as well as Windows management instrumentation.In the event of a failure, Neverfail automatically detects the failure and attempts to heal the application, to ensure continuous availability and maximum uptime. If the application cannot be healed for whatever reason, the system can be configured to failover automatically (or manually) to ensure that users remain connected.
- #21 This slide shows that our logical architecture can be deployed to operate in various physical topologies. High Availability is our LAN Topology DUO configuration and we will walk you through its installation and operation shortly. Disaster Recovery is our WAN Topology which applies the same DUO configuration across a Wide Area Network. We can then extended the same powerful core capability of Neverfail to provide Disaster Recovery for Microsoft Active/Passive Clusters running Exchange or SQL. This is our ClusterProtector product which would merit a separate presentation to go into any level of detail. And finally we can now extend the same core architecture to combine the HA and DR capability into a Continuously Available linked TRIO of servers. Along with TRIO we have introduced a “WANSmart” capability that dramatically improves bandwidth utilization between local and remote sites, as well as reducing replication time. Depending on application type, WANSmart achieves up to 30x improved bandwidth utilization by using advanced data de-duplication algorithms prior to normal compression. Using WANSmart offers the potential to avoid costly WAN upgrades. Since our failover process is fully symmetrical, Neverfail’s Active/Passive architecture can be deployed in pretty much any Physical or Virtualised Primary and Secondary server configuration. And in any virtualized VMware environment, if the resilience of your virtual management tool is important, we even protect VMware’s VirtualCentre which is an acknowledged single point of failure. In fact VMware have licenced Neverfail’s technology which is embeded in their vCenter Heartbeat product. An equivalent level of protection for Microsoft’s VMM is in development.
- #25 Our value proposition is about keeping users permanently connected to their applications. Allowing access to what they need when they need it.Our Total Cost Of Ownership equation = No loss of productivity + meeting or exceeding SLA’s + preventing reputational damage. Service interruptions cause a loss of productivity; the cost of one minute of downtime multiplied by the number of business users in an organization can be surprisingly costly. As can the financial penalties associated with not meeting Service Level Agreements. And service interruptions can also lead to a loss of confidence in the IT function and damage the credibility of the CIO with the business. Reputational damage is hard to put a price on.Our technology will enable reputations to be protected and SLA’s to be maintained or even exceeded by maintaining service continuity through interruptions and application maintenance windows that normally require downtimeIn summary Neverfail enables Enterprise Data Centre levels of high availability and disaster recovery based on cost effective “Wintel” commodity technology.
- #27 This slide shows that our logical architecture can be deployed to operate in various physical topologies. High Availability is our LAN Topology DUO configuration and we will walk you through its installation and operation shortly. Disaster Recovery is our WAN Topology which applies the same DUO configuration across a Wide Area Network. We can then extended the same powerful core capability of Neverfail to provide Disaster Recovery for Microsoft Active/Passive Clusters running Exchange or SQL. This is our ClusterProtector product which would merit a separate presentation to go into any level of detail. And finally we can now extend the same core architecture to combine the HA and DR capability into a Continuously Available linked TRIO of servers. Along with TRIO we have introduced a “WANSmart” capability that dramatically improves bandwidth utilization between local and remote sites, as well as reducing replication time. Depending on application type, WANSmart achieves up to 30x improved bandwidth utilization by using advanced data de-duplication algorithms prior to normal compression. Using WANSmart offers the potential to avoid costly WAN upgrades. Since our failover process is fully symmetrical, Neverfail’s Active/Passive architecture can be deployed in pretty much any Physical or Virtualised Primary and Secondary server configuration. And in any virtualized VMware environment, if the resilience of your virtual management tool is important, we even protect VMware’s VirtualCentre which is an acknowledged single point of failure. In fact VMware have licenced Neverfail’s technology which is embeded in their vCenter Heartbeat product. An equivalent level of protection for Microsoft’s VMM is in development.
- #28 This slide shows that our logical architecture can be deployed to operate in various physical topologies. High Availability is our LAN Topology DUO configuration and we will walk you through its installation and operation shortly. Disaster Recovery is our WAN Topology which applies the same DUO configuration across a Wide Area Network. We can then extended the same powerful core capability of Neverfail to provide Disaster Recovery for Microsoft Active/Passive Clusters running Exchange or SQL. This is our ClusterProtector product which would merit a separate presentation to go into any level of detail. And finally we can now extend the same core architecture to combine the HA and DR capability into a Continuously Available linked TRIO of servers. Along with TRIO we have introduced a “WANSmart” capability that dramatically improves bandwidth utilization between local and remote sites, as well as reducing replication time. Depending on application type, WANSmart achieves up to 30x improved bandwidth utilization by using advanced data de-duplication algorithms prior to normal compression. Using WANSmart offers the potential to avoid costly WAN upgrades. Since our failover process is fully symmetrical, Neverfail’s Active/Passive architecture can be deployed in pretty much any Physical or Virtualised Primary and Secondary server configuration. And in any virtualized VMware environment, if the resilience of your virtual management tool is important, we even protect VMware’s VirtualCentre which is an acknowledged single point of failure. In fact VMware have licenced Neverfail’s technology which is embeded in their vCenter Heartbeat product. An equivalent level of protection for Microsoft’s VMM is in development.
- #29 This slide shows that our logical architecture can be deployed to operate in various physical topologies. High Availability is our LAN Topology DUO configuration and we will walk you through its installation and operation shortly. Disaster Recovery is our WAN Topology which applies the same DUO configuration across a Wide Area Network. We can then extended the same powerful core capability of Neverfail to provide Disaster Recovery for Microsoft Active/Passive Clusters running Exchange or SQL. This is our ClusterProtector product which would merit a separate presentation to go into any level of detail. And finally we can now extend the same core architecture to combine the HA and DR capability into a Continuously Available linked TRIO of servers. Along with TRIO we have introduced a “WANSmart” capability that dramatically improves bandwidth utilization between local and remote sites, as well as reducing replication time. Depending on application type, WANSmart achieves up to 30x improved bandwidth utilization by using advanced data de-duplication algorithms prior to normal compression. Using WANSmart offers the potential to avoid costly WAN upgrades. Since our failover process is fully symmetrical, Neverfail’s Active/Passive architecture can be deployed in pretty much any Physical or Virtualised Primary and Secondary server configuration. And in any virtualized VMware environment, if the resilience of your virtual management tool is important, we even protect VMware’s VirtualCentre which is an acknowledged single point of failure. In fact VMware have licenced Neverfail’s technology which is embeded in their vCenter Heartbeat product. An equivalent level of protection for Microsoft’s VMM is in development.
- #30 This slide shows that our logical architecture can be deployed to operate in various physical topologies. High Availability is our LAN Topology DUO configuration and we will walk you through its installation and operation shortly. Disaster Recovery is our WAN Topology which applies the same DUO configuration across a Wide Area Network. We can then extended the same powerful core capability of Neverfail to provide Disaster Recovery for Microsoft Active/Passive Clusters running Exchange or SQL. This is our ClusterProtector product which would merit a separate presentation to go into any level of detail. And finally we can now extend the same core architecture to combine the HA and DR capability into a Continuously Available linked TRIO of servers. Along with TRIO we have introduced a “WANSmart” capability that dramatically improves bandwidth utilization between local and remote sites, as well as reducing replication time. Depending on application type, WANSmart achieves up to 30x improved bandwidth utilization by using advanced data de-duplication algorithms prior to normal compression. Using WANSmart offers the potential to avoid costly WAN upgrades. Since our failover process is fully symmetrical, Neverfail’s Active/Passive architecture can be deployed in pretty much any Physical or Virtualised Primary and Secondary server configuration. And in any virtualized VMware environment, if the resilience of your virtual management tool is important, we even protect VMware’s VirtualCentre which is an acknowledged single point of failure. In fact VMware have licenced Neverfail’s technology which is embeded in their vCenter Heartbeat product. An equivalent level of protection for Microsoft’s VMM is in development.