- The document discusses various technical considerations for running Exchange in a large environment with over 130,000 mailboxes spread across 132 Exchange servers in multiple data centers. It covers topics like TCP connections, TCP keepalive time, LDAP policies, .NET garbage collection, filtering, and using PowerShell for automation. - It provides recommendations like setting the TCP keepalive time lower to prevent idle connections, modifying LDAP policies to prevent forced disconnections, monitoring .NET garbage collection, and using server-side filtering in PowerShell for better performance. The goal is to optimize the environment to handle the scale while preventing issues like port exhaustion or server overload.

1. RUNNING EXCHANGE
IN LARGE ENVIRONMENT

2. ♡ DIAMOND AND PLATINUM SPONSORS ♡

3. ABOUT ME
Ingo Gegenwarth
IT Principal Consultant @SAP
MCM Exchange 2010
Office Server and Services MVP
Blog:
https://ingogegenwarth.wordpress.com/
Twitter:
@IngoGegenwarth
E-mail:
ingo@thecluelessguy.de

4. Agenda
• TCP connections (concurrent)
• TCP KeepAliveTime
• LDAP policies
• .NET Garbage Collection
• Filtering (LDAP, server-side vs. client-side)
• PowerShell
• Q&A

5. Some background facts (environment)
• 132 Exchange servers
• 4 internet outbreaks (EUR and NAM)
• 5 AD sites
• Geographically dispersed
• Mix of un-/bound model
• Following PA (in most cases )
• Approx. 130k mailboxes

6. Some background facts (sever layout)
• 2U, dual socket servers with 24 cores
• 2 servers in same DC share one DAS (4U, 2 drawer)
• 21 disks (18 prod and 3 spare)
• 2 NIC (one for client-side and one for main traffic)
• 192GB memory

7. Some background facts (DC/DAG layout)
• EUR
• 2 Datacenter
• 4 nodes across 2 DC
• NAM
• 1 Datacenter
• 8 nodes across 3 DC (4 nodes in NAM and 4 in 2 DC in EUR for DisasterRecovery)
• 72 databases per DAG (18 active, 36 highly available, 18 lagged)

8. TCP connections (concurrent)
Loadbalancer

9. TCP connections (concurrent)
What does this really mean?
• Client connections will be established against remaining DC
• Databases will also perform failover to remaining DC
• Clients (e.g.: Outlook, Skype for Business) are impatient and will establish
new connections
Risks:
• TCP port exhausting
• Exceeding limit of load balancer (# of connections, TPS, new connections per
second)
• Server resource exceedance (domino effect!)

10. TCP connections (concurrent)
What can we do?
• Monitor your concurrent connections on load balancer
• Monitor new connections per seconds on load balancer
• Plan for complete Datacenter failover
• Know your limits in terms of concurrent and new connections per second!

11. TCP KeepAliveTime
• Controls time window a remote host is assumed to be alive
• Default is 7,200,000 milliseconds  2 hours (the connection will be kept for
this time!)
• TCP KeepAliveTime ≠ TCP Idle Timeout and TCP Session Timeout
• Lower TCP Idle Timeout on network components
• From external to internal shorter
• Last component needs to have higher value than TCP KeepAliveTime set on
Exchange
Recommendation:
• Set TCP KeepAliveTime to a value between 5 – 15 minutes in a high
frequented environment

12. DEMO

13. LDAP policies
• Exists since Windows 2000
• Similar to throttling policies in Exchange
• Safeguarding Domain Controller against rogue LDAP clients

14. LDAP policies
(Source: Microsoft)

15. LDAP policies
Why it matters?
“MaxConnections - The maximum number of simultaneous LDAP connections that a domain controller
will accept. If a connection comes in after the domain controller reaches this limit, the domain
controller drops another connection.”
Source: Microsoft
• Domain controller will randomly and forcible close connections
• Exchange don’t expect that connections get forcibly closed after previous LDAP check was
successful
Risks:
• Prompt for credentials
• Failed request
• Server unresponsive

16. LDAP policies
What can be done?
• Modify LDAP policy either per server or per site (do you have a dedicated site for
Exchange?)

17. DEMO

18. .NET Garbage Collection
What is garbage collection?
• Exchange was rewritten based on .NET
• .NET is cleaning the house during GC
• There are 2 modus operandi (Workstation and
Server GC)

19. .NET Garbage Collection
Workstation GC
Memory allocation:
256MB
Threads:
Single
Source: https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/fundamentals
Server GC
Memory allocation:
4GB (logical CPU>4=2GB >8=1GB)
Threads:
Multiple

20. .NET Garbage Collection
• Use Performance Monitor with 1 second update to
detect spikes, which might been missed
• Process Explorer is much more convenient for quick
check
Risks:
• Components get unresponsive
• Clients establish new TCP connections
• Server failure due to overload

21. DEMO

22. Filtering (LDAP, server-side vs. client-side)
Be smart and let the server do the job for you!
Why?
• Less traffic over the wire
• Less or no penalty for throttling
• Higher performance (response)

23. Filtering (LDAP, server-side vs. client-side)
Bad example:
Get-Mailbox | Where-Object {$_.RecipienttypeDetails -eq 'RoomMailbox’}
Good example:
Get-Mailbox -Filter {RecipienttypeDetails -eq 'RoomMailbox'}

24. Filtering (LDAP, server-side vs. client-side)
Recent real-life use case:
Tombstoned ACEs on shared mailboxes
• Report across 16.000 mailboxes using LDAP took approx. 16 minutes
• Clearing invalid permissions took over an hour as Exchange Cmdlet have to be used

25. DEMO

26. PowerShell
• Helpful for automation
• Combine multiple technologies e.g.: ADSI, PS modules, C#
• Repeatable steps using functions, scripts
• Desired State Configuration (DSC)
• Parallel vs. sequential processing (multi-threading)

27. DEMO

28. Wrap-Up
• TCP concurrent connections and TCP KeepAliveTime
• LDAP policies
• .NET Garbage Collection
• Server-side vs. client-side filtering
• PowerShell, PowerShell, PowerShell…. (being lazy is good!)
• Scale-out and NOT scale-up

29. QUESTIONS?
Q&A