This document summarizes a solution for deploying 7,500 mailboxes using Exchange Server 2016 on HPE ProLiant BL460cGen9 server blades and D6000 storage. The solution utilizes five servers in a high availability configuration with database availability groups to provide redundancy in case of server failures. Testing showed the solution could support the target workload even if two servers were unavailable. The solution leverages the BladeSystem infrastructure for efficiency and scalability, and can be expanded beyond 7,500 mailboxes by adding more server blocks.

1. HPEReferenceArchitecturefor
Microsoft ExchangeServer2016on
HPE ProLiant BL460cGen9andD6000
Storage
Deploy 7,500 mailboxes with Exchange Server 2016
on RAID-less JBOD storage
Technical whitepaper

2. Technical whitepaper
Contents
Executivesummary 3
Solutionoverview 4
Designprinciples 5
Highavailability (HA) and disasterrecovery(DR) 5
Scaling thesolutionbeyond 7,500 mailboxes 6
Solutioncomponents 6
EnclosuresandI/Omodules 6
Servers 6
Operatingsystemandapplication software 8
Best practicesandconfigurationguidancefor thesolution 8
Exchange2013PerformanceHealth Checker Script 12
Capacity andsizing 13
Performance 13
Capacity 13
Analysisandrecommendations 13
Summary 15
Implementingaproof-of-concept 15
AppendixA: Bill of materials 16
Resourcesandadditional links 17

3. Technical whitepaper Page 3
Executivesummary
Thiswhitepaper isintendedfor customerswhowant tomigratetoMicrosoft®ExchangeServer 2016andwhohavestandardized onHPE
BladeSystemservers.ExchangeServer2016isoneof thelatest releasesinalonglineof ExchangeServerreleases.Overtheyears,Exchangehas
changedinanumber of ways.It movedfroma32bit toa64bit architectureandhaschangedstorageinput/output (I/O) requirementssuchthat
slower,lower cost diskscanbeutilized.Thehighavailability model wasmodifiedsuchthat Exchangenativedataprotectioncanbethedefault high
availability solution. Manyother architectural advanceshavebeenimplementedthat changedsystemrequirements,someof which changethe
focusfromvery highperformancestorage,tohigher performanceserverswith RAMandCPUcapabilitiestoaddressthenewapplication
requirements.
Deploying ExchangeServer2016onolderserverscanrequiremoreserverstobedeployed,dependingontheuserprofileandothersolution
parameters.Inoneexample,usingearliergenerationsof serverscould result inneedingfromtwo tosevenmoreserverstosupport thesame
ExchangeServer2016workloadoperating on5HPEProLiant BL460cGen9servers.Inaddition toserver costs,thiswould alsoincrease
administration, networking andother infrastructurecosts.Deploying ExchangeServer 2016onthemost recent HPEProLiant generationcan help
reducethenumberof serversrequired,while alsoreducing networking,power and administrativecosts.Anin-placeupgradeusingexisting
hardwareusedtohost earlier versionsof ExchangeServer isnot possible,sonewserver hardwareandstoragecapacityisrequiredtodeploy
ExchangeServer2016.
Thiswhitepaperdescribesdeploying Microsoft ExchangeServer 2016onHPEProLiant BL460cGen9serverbladesinanHPEBladeSystem
c7000 enclosurewith direct attachedstorage(DAS),utilizing HPED6000 diskenclosures.Thisimplementationensuresahighly available
solutionwhile minimizing costs.Thistestedsolutionsupportsacustomerusecasewith 7,500 10GBmailboxesat aprofile of 150 total messages
sent andreceived per dayper mailbox.Rather thanusingRAIDtoprovidehighavailability (HA) at thestoragelayer,thissolutionutilizesthe
nativedataprotection featuresof ExchangeServer2016throughimplementationof adatabaseavailability group(DAG) with threecopiesof each
databaseamongfiveservers.Thedatabasesandtransactionlogsarestoredonindividual RAID-lessdisksina“just abunchof disks”,or JBOD
configuration.
Deploying thesolutiononHPEBladeSystemcapitalizesonthepower andcooling savingsof theBladeSysteminfrastructurealong with
simplifying datacenteroperationsand offeringgreaterdensity,sharedresources,scalability and greateroverall efficiency.For moreinformation
about thebenefitsof deploying HPEBladeSystem,pleaserefertotheIDCTop5ReasonstoMovetoBladeSystem,presentation:
http://h20195.www2.hpe.com/v2/GetDocument.aspx?docname=4AA5-7828ENW
TheMicrosoft PreferredArchitecture(PA) for Exchangeisutilized asaguidein deploying this solution.WherepossiblethePAis followed.
Encryption of dataat rest utilizing BitLocker isalsodiscussed,alongwith keysystemcomponents,suchasTrustedPlatformModule(TPM) 1.2,
andCPUsthat support theIntel®AES-NIinstruction settoaccelerateencryption andreducethestorageperformanceimpact of encryption.
Insummarizingtestingwith Microsoft ExchangeJetstressandLoadGen,thiswhitepaper showsthat thissolutioncansupport thetarget 7,500
mailboxeswith a150 messagesperdaypermailboxprofile in several different scenarios.Thesolutionsupportsatwoserver outagescenario,
whereevenwith twoserversunavailable,thesolution cansupport apeakload of 300 messagespermailbox perday.
ExchangeServer2016wasreleasedinSeptemberof 2015.It advancesonExchangeServer 2013with moreefficient catalogingandsearch
capabilities,simplifiedserverarchitectureanddeployment models,fasterand morereliableeDiscovery,and expandeddatalossprevention
features.Sizing forExchange2016isvery similar tothesizing processfor Exchange2013,which isdemonstratedwith thetest resultslater in this
document,ascomparedtothereferencearchitectureat:http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=4AA6-2902ENW

4. Technical whitepaper Page 4
Solutionoverview
Thissolution isbuilt aroundan HPEBladeSystemc7000 enclosure,aD6000 diskenclosure,fiveBL460cGen9servers,andinternal componentsto
bediscussedlater in thisdocument.Thissolutionalsoutilizesthenativedataprotectionfeaturesof Exchange2016with multiple copiesof each
database.Storagecostsarereducedby usingdirect attachedstorage(DAS) inajust abunchof disks(JBOD) configurationfor Exchange
databasesandlogs,which meansthat noRAIDprotection isprovidedby thestoragesubsystem.Eachphysical 6TBSAS7.2K3.5indiskhoststhree
copiesof variousdatabases,andeachdatabasehasthreecopiesspreadamongthefiveservers.
Figure 1. Logical Exchange Server 2016 di agram with DAG, servers and some database copies.
Intheabovefigure,eachExchangeserver ishostedonaProLiant BL460cGen9server inaBladeSystemc7000 enclosure,which isattachedtoa
D6000 diskenclosuretoprovidestoragecapacity for theExchangedatabases.
Inthissolution,eachserver hasapair of boot/systemdiskswhich areprotected in aRAID1pair,andeachserver isprovisionedfourteendisks
fromtheD6000,which areutilized inaJBODconfigurationwhereeachphysical diskisitsownarrayandlogical drive.Withtwelvedisksfor
databases,theextratwodisksperserverarefor recoveryspaceandrepair intheevent of adiskfailure.
Ageneral viewof thesolutionin asingleBladeSystemc7000 enclosureand D6000 disk enclosureis outlinedbelowin Figure2.Thedetailsof
thesolutioncomponentsarediscussedin thenext section.

5. Technical whitepaper Page 5
Figure 2. Front and rear views of BladeSystem c7000 enclosure with fiv e server blades and D6000 disk enclosure
Eachof thefiveBL460cGen9serverscontain96GBof RAM, twoE5-2630 v3CPUs,oneHPEFlexFabric20Gb2-port 650FLBFlexibleLOM,one
Smart ArrayP244br controller andoneSmart ArrayP741mcontroller.TheSmart ArrayP244br controller isusedfor theinternal boot/system
drivesineachserver,while theP741mcontroller isinstalled in mezzanineslot 2inorder tointerfacewith HPE6GbSASSwitchesinI/Obays5and
6of thec7000 enclosure.For Ethernet connectivity thec7000 enclosurecontainsHPEVirtual Connect FlexFabric10Gb/24-port Modulesin I/O
bays1and 2.
Designprinciples
Microsoft ExchangeServerisoneof the,if not the,most widely usedbusinessproductivity applications.It isamissioncritical servicefor many
businesseswherebusinessquicklycomestoahalt if theserviceisnot available.Inordertomeetbusinessrequirements,theExchangeservice
shouldbedesignedaround ServiceLevelAgreements(SLA) which canincludeperformancelevels,uptimerequirements,capacity requirements,
andrecovery time.
Highavailability (HA) anddisaster recovery (DR)
Thissolutionutilizesthenativedataprotectionfeaturesof Exchangeasonelayer of highavailability (HA).Inthissolution,up totwoserverscan
fail or otherwisebeofflineandeachdatabasewill still haveat least oneactivecopytoserveusers.Eachserver usesaRAID1logical drivefor
WindowsBoot/SystemandforExchangetransport databases,but theExchangemailboxdatabasesandlogsareheld onsinglephysical disksona
RAID0 logical drive.TheserversaresizedtoprovidetheExchangeserviceevenwhen twoserversareunavailable.Theload increasesonthethree
remaining servers,but they aresizedtohandlethat increasedloadin thefailurescenario.
Another aspect of highavailability ismanagingsinglepointsof failure.TheHPEBladeSystemc7000 enclosureinthissolution hasredundant
power supplies,fans,OnboardAdministrator (OA) modules,andredundant Virtual Connect FlexFabric10Gb/24-port andSASmodules.Eachof

6. Technical whitepaper Page 6
theseredundant componentsneedstobemanagedaccordingly, suchasthepower suppliesbeingconfiguredin asuitableHAmodeandbeing
suppliedby separateand redundant power sources,and theOAmodulesbeing onseparatenetworks.
Thestorage subsystem also hassome level of HA built-in. The D6000 disk enclosure has two drawers,each with 35LFFdisk bays.Each drawer
has redundant power supplies,redundant fans,and redundant datapaths to the c7000 enclosure.The 6TBSAS7.2K3.5in disk drives each have
twoI/Oports,soalossof asingleSASpathwill not causelossof accesstothediskdrives.
TheMicrosoft PArecommendsnot using NICteaminginorder tosimplifythefailover model.WhilethePAisMicrosoft’srecommendationsbased
ontheir deployment methods,not all installations will beat thesamescaleasthat deployment.Insmaller scale,anExchangedeployment may not
spandatacenters,or evenrackswithin thesamedatacenter,soother deployment optionsshouldbeconsidered.
For network connectivity redundancy of each of the BL460c Gen9 servers,while NICteaming isnot recommended in the Microsoft PA, the most
efficient way to implement network HA in this configuration is to useWindows NICteaming.With NICteaming,the failure of any single NICon a
server will not impact thesolutionandneither will thefailureof either of theVirtual Connect FlexFabric10Gb/24-port Modules.
For further HA,thesolution canbedeployed acrossmultiplegeographicsites.Becauseof thedetailsof suchadeployment,suchasnamespace
design,activation scenarios,quorumplacement,etc.thistype of deployment isoutsidethescopeof thisdocument.
Scaling thesolutionbeyond7,500mailboxes
Thesolutioncanbescaledbeyond 7,500 mailboxessimplybyusingthissolutionasa7,500 mailboxbuilding block,meaningthat for each7,500
mailbox increment,anotherfiveserversandD6000 aredeployed.Asthesolutiongrows,multiple c7000 enclosureswould beutilized and
databasecopiesshouldbedistributed amongthec7000 enclosuressuchthat not all copiesof adatabasewould resideonasingleD6000 disk
enclosureor c7000 enclosure.Sincethissolutionwasnot testedbeyond 7,500 mailboxes,further discussionof thisisoutsidethescopeof this
document.
Solutioncomponents
ThisExchange2016solutionfor 7,500 10GBmailboxesutilizesthefollowing HPEBladeSystemcomponents:c7000 enclosure,Virtual Connect
FlexFabric10Gb/24-port Modulesand BL460cGen9.TheD6000 diskenclosureisoutsideof thec7000 enclosure.Eachof thesearecoveredin
moredetail asfollows.
Enclosures andI/Omodules
Thissolution isdesignedaround theHPEBladeSystemc7000 enclosurewith six platinumpower suppliesandtwoOnboardAdministrator (OA)
modules.For network I/O,twoVirtual Connect FlexFabric10Gb/24-port Modulesareutilized for redundant Ethernet connectivity within the
enclosure.Whileeachserver utilizesoneNICasper theMicrosoft PreferredArchitecture,connectingall of theNICstoasinglenon-redundant
switch or I/Omodulewould createasinglepoint of failurefor all of theservers.Inthissolution,twoEthernet I/Omodulesareusedtoprovide
networkHAwithin theenclosureandtoexternal topof rackswitches.
For connectivity to the D6000 disk enclosureHPE6Gb SASSwitch Dual Pack for HPEBladeSystemc-Classisinstalled in I/Obays 5and 6 of the
c7000 enclosure.TheseI/Omodulesare connected to the D6000 asshown in Figure 2above.Four of the HPEExt Mini SAS1m Cablesareused
tocabletheD6000 totheSASI/Omodules.
Servers
HPEProLiant BL460cGen9server bladesareutilized in this solution.Eachof theserversis configuredwith 96GBof RAM, andwith twoIntel
Xeon®E5-2630 v3CPUs,for 16corestotal.Asnoted in theExchangeTeamBlog(http://blogs.technet.com/b/exchange/archive/2013/05/06/ask-
the-perf-guy-sizing-exchange-2013-deployments.aspx) sizing guidance,theseserversaredeployed with Hyper-Threading turnedoff.Also,as
notedinthesesizing recommendations:https://technet.microsoft.com/en-us/library/dn879075(v=exchg.150).aspx,theserversshouldalsobe
configuredin theBIOStoallowtheOStomanagepower,andWindows®shouldbeset tothehighperformancepower plan.
Eachserver isconfiguredwith apair of internal 2TBSAS7.2K2.5inHDDsfor boot andsystem.Thedisksin theD6000 for Exchangedatabases,
logs,recoveryandautomaticreseedingarediscussedintheStoragesectionbelow.
Aninternal viewof theBL460cGen9server isshowninFigure3.Moreinformationabout theBL460cGen9serversisavailableat:
hpe.com/servers/bl460c

7. Technical whitepaper Page 7
Figure 3. Internal view of BL460c Gen9
Storage
The D6000 disk enclosure is utilized for database and transaction log files,and recovery,maintenance and automatic reseeding capacity.The
D6000 can hold atotal of seventy largeform factor (LFF) 3.5inch drives.Thoseseventy drives are divided between two drawers of thirty-five
driveseach.Thissolutionusesall seventy drivebaysin theD6000 with seventy HPE6TB6GSAS7.2KrpmLFF(3.5-inch) drives.
Toutilize thestoragein theD6000,eachof theExchangeserversisconfiguredwith aSmart ArrayP741mcontroller.TheRAIDfeaturesof the
Smart Arraycontroller arenot usedonthesedatabaseandlogdisksasthey areconfigured asJBOD,whereeachphysical diskisitsown array
andRAID0 logical drive.Eachserverusestwelve disksfor active andpassivedatabasecopies,with another twodisksperserverfor recovery,
auto-reseedor maintenancespace.
TheD6000 isshownbelowin Figure4.Moreinformationabout theD6000 isavailableat:
http://www8.hp.com/us/en/products/disk-enclosures/product-detail.html?oid=7390970
Informationabout deployment andcablingof theD6000 isavailableat:
http://h20565.www2.hpe.com/portal/site/hpsc/template.PAGE/public/psi/manualsResults/?sp4ts.oid=5307027and
http://h20564.www2.hpe.com/portal/site/hpsc/public/kb/docDisplay/?docId=c01956983
TheMicrosoft PreferredArchitectureoutlinesusingWindowsBitLocker for at rest dataprotection.For effectiveuseof BitLocker,eachserver
shouldbeconfiguredwith theTrustedPlatformModule(TPM) 1.2.Thiseasestheuseof BitLocker by storingand securing theencryption keys
local totheserver without requiring aBitLocker passwordeachtimetheserver boots.Toalsoeasetheperformanceimpact of BitLocker,the
CPUsusedin thissolutionincludetheIntel AES-NIinstruction set,which isusedby BitLocker toreduceCPUandperformanceimpact.More
informationabout Intel AES-NI isavailableat:intel.com/content/dam/doc/white-paper/enterprise-security-aes-ni-white-paper.pdf
Thissolution wastestedwith unencryptedstorageandwith storageencryptedbyWindowsBitLocker and theperformancedifferencewas
negligible.Informationondeploying BitLocker is available at:https://technet.microsoft.com/en-us/library/hh831713.aspxand
https://technet.microsoft.com/en-us/library/jj612864.aspx

8. Technical whitepaper Page 8
Figure 4. D6000 f ront external view.
Operating systemandapplicationsoftware
This solutionis built onWindowsServer 2012R2with all applicableupdatesinstalled viaWindowsUpdateasof thedateof testing (December
2015):Version6.3,Build 9600.ExchangeServer 2016(build 15.01.0225.042) wasusedfor thissolution.HPEServicePackforProLiant version
2015.06.0 wasinstalled oneach of theBL460cGen9serversfor driverandfirmwareupdates.
Microsoft ExchangeLoadGenandJetstresswereusedasthetest toolsfor thissolution. LoadGenwasversion15.00.0847.030 andJetstresswas
15.00.0995.000. For Jetstresstherequired.dll andother support fileswereusedfromExchangeServer2016(build 15.01.0225.042).
Bestpracticesandconfigurationguidanceforthesolution
Microsoft recommendsdisablingHyper-Threading whendeploying Exchangeonphysical servers.Thisisnot necessaryinavirtualized
environment aslongastheadditional CPUsarenot usedin capacityplanning.Inthissolution, eachserver hastheBIOSconfiguredtodisable
Hyper-Threading andtoset thepower profileset toOScontrol with theEnergy/PerformanceBIASset to“MaximumPerformance”.Figures5
through7belowshowthosesettings.
Figure 5. Disabling Hyperthreading i n the server BIOS.
Figure 6. Setting Power Pro f ile to “ Custom” and Power Regulator t o “OS Contr o l Mode”.

9. Technical whitepaper Page 9
Figure 7. Setting Energy/Performance Bias to “ Maximum Per f ormance” .
Key point
TheseExchangeserversshould beconfiguredwith Hyperthreading turnedoff andwith thepower profileset toOScontrolled.Misconfiguring
thesesettingscanhaveanegativeimpact onperformance.
TheBladeSystemSASswitchesmust beconfiguredtoprovision 14 drivesfor eachof thefiveExchangeServers.TheHPE6GVirtual SASmanager
applicationislaunched fromtheHPEBladeSystemOnboardAdministrator web console.ZoneGroupsarecreated,populatedwith drivebays,and
thenassignedtoserver bays.Azonegroupiscreatedfor eachExchangeserverand ispopulatedwith drivebays accordingtothefollowingtable.
Table 1. SAS Zone group con f iguration parameters .
TheVirtual SASManager providesasummaryviewof theSAStopology asshownbelowinFigure8.
ZONE NAME ASSIGNED DRIVE BAYS ASSIGNED SERVER BAY
Exch01 Drawer 1: Bays 1-14 Server bay 6
Exch02 Drawer 1: Bays 15-28 Server bay 7
Exch03 Drawer 1: Bays 29-35
Drawer 2: Bays 29-35
Server bay 8
Exch04 Drawer 2: Bays 1-14 Server bay 14
Exch04 Drawer 2: Bays 15-28 Server bay 15

10. Technical whitepaper Page 10
Figure 8. Configured SAS Topology
Afterthezonegroupsareconfiguredand assignedtoaserverbay,eachserverwill see14 newdiskdrivesontheSmart Array P741mcontroller.
Inthissolutioneachoneof thosediskdrivesisconfiguredasaseparatearray, with asingleRAID0 logical drive.Theviewfromoneserveris
shownbelowinFigure9for arraysAthroughC.ThisconfigurationcontinuesthrougharrayN.
Figure 9. HPE Smar t Storage Administrat or configuration .

11. Technical whitepaper Page 11
TheaboveconfigurationcanbeaccomplishedthroughtheGUIshownabove,or it canbedoneat thecommandline.Thedefault pathfor the
commandlineutility is C:ProgramFilesHPhpssaclibin.Oncein that directory,acommandlinesuchasbelow,canbeutilized toconfigureall 14
drivesontheSmart ArrayP741massingledriveRAID0 logical drives:
1..14 | foreach ($_) {$drivestring= "51:1:$_" ; write- host " . hpssacli.exe c t r l slot=2 create type=ld
drives="$drivestring" raid=0 stripsize=256"}
OncethedrivesareconfiguredandexposedtoWindowsasnewdiskdrives,theymust beinitialized, mountedtomount points,andformatted.
This solutionutilized themount point of C:ExchangeVolumeswith eachdiskmountedtoafolder under that asshownin Figure10 below.
Figure 10. Mount point root , mount points, volumes and multiple databases per volume
Either theWindowsGUIor PowerShell canbeusedtoinitialize,mount andformatthevolumes.Thedetailsof thesePowerShell commandsare
outsidethescopeof thisdocument.Extremecaremust betakenwhenautomatingdiskprovisioning inordertoprotect disksalready inusethat
maycontainoperatingsystemor other critical information.OnedifferencebetweenanExchange2016deployment andanExchange2013
deployment isthat theWindowsvolumesshould beformattedwith theResilient Filesystem,or ReFs,for Exchange2016.
Key points
OntheSmart Arraycontroller,thelogical drivesshould haveastrip sizeof at least 256KB.WhenformattingthedisksinWindows,theallocation
unit sizeshould beset to64KB.TheSmart Arraycontroller cachewasleft at thedefault of 10%readand90%writecache.
Thissolutionutilizes60 Exchangedatabaseswith threecopiesof thosedatabasesamongthefiveservers.Figure11showsthedistributionof
thosedatabasesamongtheservers.Thenumbersin thecellsrepresent thepreferred,secondaryandtertiary server designatedfor each
database.Oncethestorageisconfigured,mount pointscreatedanddatabaseandlog folderscreatedwithin thosepaths,thenthedatabase
distribution amongtheserversshouldbeconfiguredasshownbelow.

12. Technical whitepaper Page 12
Figure 11. Database copy di s t ributio n among the servers.
Exchange2013PerformanceHealthChecker Script
Microsoft providesanExchange2013PerformanceHealth CheckerScript that checkscommonconfigurationsettingssuchasproduct versions,
pagefilesettings,power plansettings,NICsettings,andprocessor/memoryinformation.Exampleoutput for theExchangeserversisshownbelowin
Figure12.It isrecommendedtoruntheExchange2013Health CheckerScript periodically toensurethat your environment isoperatingat peak
healthandthat configurationsettingshavenot beeninadvertently changed.Microsoft providestheExchange2013PerformanceHealth Checker
Script at https://gallery.technet.microsoft.com/Exchange-2013-Performance-23bcca58.Whilethis script wasinitially releasedfor Exchange2013,it
still providesvaluableinformationforExchange2016.
Figure 12. Example Health Checker scr i p t output .

13. Technical whitepaper Page 13
Capacity and sizing
Performance
Twotoolsareusedtoplantheperformanceandcapacityrequirementsfor thissolution: theHPESizer for Microsoft ExchangeServer 2013
(hpe.com/solutions/microsoft/exchange2013/sizer) andtheMicrosoft ExchangeServer RoleRequirement Calculator v7.8,which includes
calculations for Exchange2016(https://gallery.technet.microsoft.com/office/Exchange-2013-Server-Role-f8a61780).TheExchange2010
ProcessorQuery tool (https://gallery.technet.microsoft.com/office/Exchange-Processor-Query-b06748a5) isalsousedwith theMicrosoft
calculator tolookup theSPECInt valuesfor prospectiveCPUs.TheHPESizer for Microsoft ExchangeServer2013wasusedassizing for
Exchange2016isnot that differentthansizing for Exchange2013.
In using either the HPESizer or the Microsoft Calculator,some characteristics of the workload are utilized for performance sizing.The number of
messagessent and received per day per mailbox is a primary workload characteristic.This solution issized around aprofile of 150 messagessent
and received per mailbox per day. Using that profile resultsin thefive serverseach sized with two Intel Xeon E5-2630 v3processorswith 96GBof
RAM.
Table2showstheprofile,CPUandRAMcombinationandwhat theexpectedCPUutilization isfor different server failurescenarios.Actual test
resultsarecoveredlater in thispaper.
Table 2. CPU and RAM configuration and expected CPU u t il i z ation.
Capacity
Thissolutionisdesignedsuchthat eachof the7,500 mailboxescanbeup to10GBin capacity.Withthreeactive/passivedatabasecopies,thisis
efficiently doneutilizing fourteen6TBLFF7.2KHDDsfor eachof thefiveservers.Thiscapacity isoptimizedby usingaRAID-lessJBOD
configurationwherethereisnoRAIDoverheadaswith RAID10 or RAID5logical drives.WhileRAID5minimizesthecapacity overheadof RAID
protection,performanceisnot typically adequatewith RAID5and 7.2KRPMdiskdrives,and with lessmeantimebetweenfailure(MTBF),thereis
greater riskof datalosswith 7.2KRPMdrivesin aRAID5arrayduring anarrayrebuild operation.
Analysis andrecommendations
Jetstress testresults
Multipletestswereperformedwith Microsoft ExchangeJetstresstotest variousaspectsof thestoragesubsystem,asoutlinedbelow.Inthese
tests,thetarget IOPSper mailbox is 0.13,or 195IOPS/serverwith fiveserversonline,or 325IOPS/serverwith threeserversonline.
• JetstressTest 1–Normal Load,all fiveserversonline.Goal = Target IOPSunder latency thresholds.
• JetstressTest 2–Normal load,threeserversonline.Goal = Target IOPSunder latency thresholds.
• JetstressTest 3–Very highload.Goal = Demonstratenear upper IOPSlimit perserver.
MESSAGE PROFILE CPU
(2X PER SERVER)
RAM
(PER SERVER)
%CPU WITH FIVE
SERVERS ONLINE
%CPU WITH FOUR
SERVERS ONLINE
%CPU WITH THREE
SERVERS ONLINE
150 msg/day/mbx E5-2630 v3 96GB 48% 54% 64%

14. Technical whitepaper Page 14
Theresultsof thesethreetests areshowninthetablebelow.
Table 3. Summary of tes t r esults f or Jetstress tests.
Table3showsthat thestoragesubsystemeasily meetstherequiredIOPSfor thetarget profileandhasmuchI/Otosparebeforereaching17.6ms
databasereadlatency.Withahardlimit of 20ms,thereisevenmoreheadroomtospare.
TheseJetstresstestshaveshownthat thestoragesubsystemcansatisfythestorageI/Oneedsof this solutionandthat thereisI/Oheadroomin
thesolution.
Thefollowing Microsoft ExchangeLoadGentest resultswill addresstheCPUand memoryrequirementsandperformanceof thissolution.
LoadGentestresults
Microsoft ExchangeLoadGenwasusedtomorefullytest thisExchangeServer 2016solution.WhileJetstresssimulatesExchangeServerstorage
I/Oonaserver that isnot runningExchange,LoadGensimulatesclient loadonserversactually runningExchange2016.Thistypeof testing allows
analysisof CPUandRAMutilization, latencyandresponseof variousExchangesubsystems,andmeasurement of actual messagessent and
receivedby eachmailbox toensurethetarget profileisbeing simulatedasaccurately aspossible.
Theloadof 150 messages/dayrepresentstheaverageload per mailbox,but peaktimescanfrequentlyexceedthat average,soagenerally
acceptedpracticeistotest at twicetheaverageleveltosimulatepeakusage.Testswerethusrun at aneffectiverateof 300 messages/dayto
simulatethispeakimpact andareshownbelow
For thissolution,LoadGenwasusedtosimulatethreescenarios.
• LoadGenTest 1–Normal load,all fiveserversonline.Goal = meet target profileandanalyze CPU,RAMandExchangesubsystems.
• LoadGenTest 2–High stressload,fiveserversonline.Goal = meettarget profileandanalyze CPU,RAMandExchangesubsystems.
• LoadGenTest 3–High stressload,threeserversonline.Goal = meettarget profileand analyze CPU,RAMandExchangesubsystems.
JETSTRESS TEST 1 JETSTRESS TEST 2 JETSTRESS TEST 3
I/O Profile Normal – 150 msg/day Normal – 150 msg/day Very high
Target IOPS / Server 195 325 Near Maximum
Number of
Servers
Online
5 3 5
Achieved IOPS / Server 327 454 1247
Database Read IOPS / Server 226 314 861
Database Write IOPS / Server 101 140 386
Average Database Read Latency
(ms)
5.80 6.6 17.6
Average Database Write Latency
(ms)
0.17 0.29 1.2
Transaction Log Writes / Sec 2.1 1.7 8.4
Transaction Log Write Latency (ms) 0.07 0.08 0.14

15. Technical whitepaper Page 15
Table 4. Summary of tes t r esults f or LoadGen t ests 1 – 3.
Inthepeakload scenarioswheretheCPUisslightly higher thanthedesiredthresholdof 70%,adeeper look at theCPUutilization showsthat
content indexingisutilizing asignificant amount of CPU.Inthescenarioswith lower CPUutilization, thenoderunner processes,which arethe
content indexingprocesses,wereabout 18%of thetotal CPUutilization. Inscenarioswith higher CPUutilization, thenoderunner processes
representedabout 34%of theCPUutilization.
TheLoadGentestingshowedthat thesolutioncansupport thetarget numberof mailboxesat thetarget profile,eveninpeakloadscenarios,and in
failurescenarioswheretwo serversareofflinefor maintenanceor becauseof anunplannedoutage.
Summary
Properly sizing anExchangeServer2016solutioncanbechallenging,andsupport becomesmorechallenging if it isnot sized properly.Thiswhite
paper outlined anExchangesolutiondesignedandsized for 7,500 mailboxesof up to10GBeach.Testingshowedthat thestorage,CPU,RAMand
networkingsubsystemscansupport thissolutionat the150 messagesper dayper mailboxprofilefor which it wasdesignedinnormaloperations
with all fiveserversonlineandin scenarioswhereup totwoserversareoffline.Thesolutionwasalsodesignedandtestedfor peakload scenarios
of 300 messagesperdaypermailbox in order tohandlepeakloadsat thebeginning of thework day,or during other highloadtimes.
With the architectural changes of Exchange Server over the last several generations,Exchange requires more CPUand RAM resources.This
need can best be met by the latest CPUand architecture of theHPEProLiant BL460c Gen9 servers.Based on CPUcomparisons in multiple
sizing scenarios,deploying with the latest generation of ProLiant servers can reduce the number of servers required by two to seven servers
dependingonthespecificsof theworkload.
Implementing aproof-of-concept
As amatter of best practice for all deployments,HPErecommends implementing a proof-of-concept using a test environment that matches as
closely as possible the planned production environment.In this way,appropriate performance and scalability characterizations can be obtained.
For help with aproof-of-concept,contact anHPEServicesrepresentative(hpe.com/us/en/services/consulting.html) or your HPEpartner.
LOADGEN TEST 1 LOADGEN TEST 2 LOADGEN TEST 3
I/O Profile Normal – 150 msg/day Peak – 300 msg/day Peak – 300 msg/day
Measured messages/day 145 295 307
Number of
servers
online
5 5 3
Average CPU utilization % 53 73 79
MS Exchange RPC Client Access Connection
Count
2944 2943 5058
MS Exchange RPC Client Access Active User
Count
1070 1358 2311
MS Exchange RPC Client Access RPC
Operations/sec
468 879 1490
MS Exchange RPC Client Access RPC Averaged
Latency
4.6 6.5 6.1
Network Interface MBytes sent/sec 3.0 5.7 7.8
Network Interface MBytes received/sec 2..81 5.5 5.0
Average database read latency (ms) 8.2 10.6 9.5
Average database write latency (ms) 0.32 0.25 0.38

16. Technical whitepaper Page 16
AppendixA:Billof materials
Note
Part numbersareat time of testing and subject to change.The bill of materials doesnot include completesupport options or other rack and
power requirements.If you have questions regarding ordering,please consult with your HPEReseller or HPESales Representative for more
details.hpe.com/us/en/services/consulting.html
Table 5. B i ll of materials
QTY PART NUMBER DESCRIPTION
1 681844-B21 HPE BLc7000 Platinum CTO with ROHS Trial IC License Single Phase
6 733459-B21 HPE 2650W Plat Ht Plg Pwr Supply Kit
2 571956-B21 HPE Virtual Connect FlexFabric 10Gb/24-port Module for c-Class
BladeSystem
1 456204-B21 HPE BLc7000 Onboard Administrator with KVM Option
1 BK764A HPE 6Gb SAS Switch Dual Pack for HPE BladeSystem c-Class
4 407337-B21 HPE Ext Mini SAS 1m Cable
1 517520-B21 HPE BLc 6X Active Cool 200 FIO Fan Opt
5 488069-B21 HPE TPM Module Kit
5 727021-B21 HPE BL460c Gen9 10Gb/20Gb FLB CTO Blade
5 700764-B21 HPE FlexFabric 20Gb 2-port 650FLB FIO Adapter
5 726782-B21 HPE Smart Array P741m/4GB FBWC 12Gb 4-ports Ext Mezzanine SAS
Controller
5 726994-L21 HPE BL460c Gen9 E5-2630v3 FIO Kit
5 726994-B21 HPE BL460c Gen9 E5-2630v3 Kit
30 726719-B21 HPE 16GB 2Rx4 PC4-2133P-R Kit
5 761871-B21 HPE Smart Array P244br/1G FIO Controller
10 765466-B21 2TB 12G SAS 7.2K 2.5in 512e SC HDD
1 K2Q12A HPE D6000 w/70 6TB 6G SAS 7.2K LFF (3.5in) Dual Port MDL HDD 420TB
Bundle

17. Technical whitepaper Page 17
Signup forupdates
Ratethisdocument
© Copyright 2016 Hewlett Packard EnterpriseDevelopment LP.Theinformation contained herein issubject to change without notice. The
only warranties for HPEproducts and servicesareset forth intheexpress warranty statements accompanying such products and services.
Nothing herein should beconstrued asconstituting an additional warranty. HPEshall not beliablefor technical or editorial errors or
omissionscontained herein.
Microsoft, WindowsServer,and Windowsareeither registered trademarks or trademarks of Microsoft Corporation intheUnitedStates
and/or other countries.Intel and Xeon aretrademarks of Intel Corporation intheU.S.and other countries.
4AA6-3740ENW,January 2016
Resources andadditionallinks
Toreadmoreabout HPEsolutionsfor Exchangepleaserefer tohttp://h17007.www1.hpe.com/us/en/enterprise/converged-
infrastructure/info-library/index.aspx?app=microsoft_exchange
HPESizer for Microsoft ExchangeServer 2013
hpe.com/solutions/microsoft/exchange2013/sizer
HPEBladeSystem
hpe.com/info/bladesystem
HPEServers
hpe.com/servers
HPEStorage
hpe.com/storage
HPENetworking
hpe.com/us/en/networking.html
HPETechnologyConsulting Services
hpe.com/us/en/services/consulting.html
HPEConvergedInfrastructureLibrary
hpe.com/info/convergedinfrastructure
Contact Hewlett PackardEnterprise
http://www8.hp.com/us/en/hpe/contact/contact.html
Tohelpusimproveour documents,pleaseprovidefeedbackat hpe.com/contact/feedback.