Your SlideShare is downloading. ×
0
Load Balancing and Intelligent Load Balancing Ivan Lorente Escalation Engineer
Objectives <ul><li>Deep understanding of load balancing </li></ul><ul><li>Understanding Intelligent load balancing </li></...
Agenda Load Balancing on an Application Delivery Scenario Architecture Where can it go wrong? Troubleshooting
Load Balancing on an Application Delivery Scenario Agenda item number 1 Architecture Where can it go wrong? Troubleshootin...
Load Balancing on an Application Delivery Scenario Citrix ®  NetScaler ® Deliver Web Apps Citrix  Presentation  Server ™ D...
Load Balancing on an Application Delivery Scenario Users Apps PS Farm
Architecture Agenda item number 1 Load Balancing on an Application Delivery Scenario Where can it go wrong? Troubleshootin...
Architecture <ul><li>Data Collector Updates </li></ul><ul><li>Understanding Bias </li></ul><ul><li>Load Balancing during l...
Architecture Data Collector DC Dynamic Store DS LmsSS.dll IMA Subsystem LMS20Rules.dll System counters MFRules.dll MetaFra...
Data Collector Updates <ul><li>Connection logon or logoff </li></ul><ul><li>Change in the load evaluator +/- 500 </li></ul...
Understanding Bias <ul><li>Definition of Bias in Science </li></ul><ul><li>Bias is defined as the difference between the m...
Understanding Bias <ul><li>queryds /table:LMS_ServerLoadTable </li></ul>5: Scheduling 6: Page Swaps 7: Disk Data I/O 8: Di...
Load Balancing during logon IMA Server B Ctxnotif.dll LMSSS.dll MFSrvSS.dll Server A WEVENT_LOGON ICA Client Data Collecto...
Performance Data Helper <ul><li>The performance data helper (PDH) interface calls the registry interface to retrieve perfo...
Performance Data Helper <ul><li>Code examples </li></ul><ul><li>http://msdn2.microsoft.com/en-us/library/aa373169.aspx </l...
Performance Data Helper <ul><li>The performance counters information is stored under: </li></ul><ul><ul><li>HKLMSOFTWAREMi...
Performance Data Helper
Performance Data Helper <ul><li>The same information can be found in the following files </li></ul><ul><ul><li>%Systemroot...
Performance Data Helper <ul><li>Perfmon.exe reads the registry keys </li></ul><ul><li>IMA reads the Perfc009.dat </li></ul...
Citrix and the PDH <ul><li>We also provide some performance counters </li></ul><ul><ul><li>%Systemroot%inf�09IMAServiceima...
Citrix and the PDH <ul><li>HKLMSYSTEMCurrentControlSetServicesIMAServicePerformance </li></ul>
Citrix and the PDH <ul><li>List of counters that  LMS20Rules.dll  uses </li></ul><ul><ul><li>rocessor(_Total) Processor Ti...
Citrix and the PDH <ul><li>Same counter with different name depending on the OS version </li></ul><ul><ul><li>English W2K ...
Load balancing - Data Store <ul><li>  </li></ul><ul><li>012C Hex = 300 dec = 5 min </li></ul><ul><li>3C Hex = 60 dec = 1 m...
Load balancing - Data Store
Where can it go wrong? Agenda item number 1 Load Balancing on an Application Delivery Scenario Architecture Troubleshootin...
Where can it go wrong? <ul><li>Black hole effect </li></ul><ul><ul><li>Load Throttling - Intelligent Load Balancing </li><...
Black hole effect <ul><li>Problem </li></ul><ul><ul><li>At Peak logon times a recently boot up server will get all connect...
Black hole effect <ul><li>Solution </li></ul><ul><li>Load Throttling </li></ul><ul><li>??? </li></ul>
Load Throttling <ul><li>Load Throttling uses Intelligent Load Biasing (ILB) </li></ul><ul><ul><li>HKLMSOFTWARECitrixIMALMS...
Load Throttling - Low logon rate <ul><li>2 server farm (server A and server B) using the Default load evaluator </li></ul>
Load Throttling - Low logon rate Server B Server A Server  Load BIAS 0 0 0 0 (10000-0)/2+0 = 5000 100 (10000-0)/2+0 = 5000...
Load Throttling - Low logon rate Data Collector ICA Client Dynamic Store Server A Server B Server B Server A Server  Load ...
Load Throttling - Low logon rate <ul><li>Server load if Low logon rate </li></ul>
Load Throttling -  Black hole effect <ul><li>2 server farm (server A and server B) using the Default load evaluator </li><...
Load Throttling -  Black hole effect Server B Server A Server  Load BIAS 0 6000 0 0 (10000-0)/2+0 = 5000 (10000-5000)/2+50...
Load Throttling -  Black hole effect Data Collector ICA Client Dynamic Store Server A Server B Server B Server A Server  L...
Load Throttling <ul><li>Server load at Peak logon times </li></ul>
Failing to read performance counters <ul><li>Problem </li></ul><ul><ul><li>One server  in the farm gets  all the connectio...
Failing to read performance counters <ul><li>Solution </li></ul><ul><ul><li>From  CPS4.0 HRP02  we have change the way the...
English OS with and MUI <ul><li>Problem </li></ul><ul><li>Advance load evaluators report full load when an MUI is installe...
English OS with and MUI <ul><li>Solution </li></ul><ul><ul><li>%Program Files%/citrix/system32/citrix/ima/lms/lms20rules.d...
Architecture Agenda item number 1 Load Balancing on an Application Delivery Scenario Architecture Where can it go wrong? T...
Troubleshooting <ul><li>Failing to read performance counters => full load </li></ul><ul><ul><li>Use  filemon while rebooti...
Troubleshooting <ul><li>Failing to read performance counters => full load </li></ul><ul><ul><li>CDF traces </li></ul></ul>...
Summary <ul><li>Deep understanding of load balancing </li></ul><ul><ul><li>BIAS, PDH… </li></ul></ul><ul><li>Understanding...
Q&A <ul><li>? </li></ul>
 
Upcoming SlideShare
Loading in...5
×

XenApp Load Balancing

7,012

Published on

0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
7,012
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
264
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
  • Presentation Title Goes Here Insert Version Number Here © 2003 Citrix Systems, Inc.—All rights reserved. This presentation was designed to assist Technical Support in the creation of consistent and effective presentations.
  • Simply copy and paste this slide to use in your presentation. If more than 4 agenda items are required, simply duplicate the bottom blue bar and space evenly
  • © 2007 Citrix Systems, Inc. — All rights reserved In most organizations, someone in IT Infrastructure Operations is the one primarily responsible for the delivery of Windows-based applications. When this person looks down the line-of-sight from the datacenter to the users he/she is supporting, it looks something like this. Presentation Server is the application delivery controller that initiates the delivery process for Windows applications at the point-of-origin, so this product is the center of gravity for this individual. Products like Access Gateway, WANScaler and EdgeSight are common to all applications, so they are also essential to IT infrastructure ops.
  • © 2007 Citrix Systems, Inc. — All rights reserved In most organizations, someone in IT Infrastructure Operations is the one primarily responsible for the delivery of Windows-based applications. When this person looks down the line-of-sight from the datacenter to the users he/she is supporting, it looks something like this. Presentation Server is the application delivery controller that initiates the delivery process for Windows applications at the point-of-origin, so this product is the center of gravity for this individual. Products like Access Gateway, WANScaler and EdgeSight are common to all applications, so they are also essential to IT infrastructure ops.
  • Presentation Title Goes Here Insert Version Number Here © 2003 Citrix Systems, Inc.—All rights reserved. It can often be estimated and/or eliminated by calibration to a reference standard.
  • Presentation Title Goes Here Insert Version Number Here © 2003 Citrix Systems, Inc.—All rights reserved. CDF entry LMS_Subsystem::EventWinStationChangeNotify: WEVENT_LOGON
  • Presentation Title Goes Here Insert Version Number Here © 2003 Citrix Systems, Inc.—All rights reserved.
  • Presentation Title Goes Here Insert Version Number Here © 2003 Citrix Systems, Inc.—All rights reserved.
  • Presentation Title Goes Here Insert Version Number Here © 2003 Citrix Systems, Inc.—All rights reserved.
  • Presentation Title Goes Here Insert Version Number Here © 2003 Citrix Systems, Inc.—All rights reserved.
  • Presentation Title Goes Here Insert Version Number Here © 2003 Citrix Systems, Inc.—All rights reserved.
  • Presentation Title Goes Here Insert Version Number Here © 2003 Citrix Systems, Inc.—All rights reserved.
  • Presentation Title Goes Here Insert Version Number Here © 2003 Citrix Systems, Inc.—All rights reserved. CDF entry LMS_Subsystem::EventWinStationChangeNotify: WEVENT_LOGON
  • Presentation Title Goes Here Insert Version Number Here © 2003 Citrix Systems, Inc.—All rights reserved.
  • Presentation Title Goes Here Insert Version Number Here © 2003 Citrix Systems, Inc.—All rights reserved. CDF entry LMS_Subsystem::EventWinStationChangeNotify: WEVENT_LOGON
  • Presentation Title Goes Here Insert Version Number Here © 2003 Citrix Systems, Inc.—All rights reserved.
  • Presentation Title Goes Here Insert Version Number Here © 2003 Citrix Systems, Inc.—All rights reserved.
  • Transcript of "XenApp Load Balancing"

    1. 1. Load Balancing and Intelligent Load Balancing Ivan Lorente Escalation Engineer
    2. 2. Objectives <ul><li>Deep understanding of load balancing </li></ul><ul><li>Understanding Intelligent load balancing </li></ul><ul><li>Typical scenarios </li></ul><ul><li>Troubleshooting techniques </li></ul>
    3. 3. Agenda Load Balancing on an Application Delivery Scenario Architecture Where can it go wrong? Troubleshooting
    4. 4. Load Balancing on an Application Delivery Scenario Agenda item number 1 Architecture Where can it go wrong? Troubleshooting Load Balancing on an Application Delivery Scenario
    5. 5. Load Balancing on an Application Delivery Scenario Citrix ® NetScaler ® Deliver Web Apps Citrix Presentation Server ™ Deliver Windows Apps Citrix Desktop Server ™ Deliver Windows Desktops Citrix EdgeSight ™ Monitor Real-Time User Experience Citrix WANScaler ™ Accelerate Apps to Branch Offices Citrix Access Gateway ™ Enable Secure App Access Users Apps
    6. 6. Load Balancing on an Application Delivery Scenario Users Apps PS Farm
    7. 7. Architecture Agenda item number 1 Load Balancing on an Application Delivery Scenario Where can it go wrong? Troubleshooting Architecture
    8. 8. Architecture <ul><li>Data Collector Updates </li></ul><ul><li>Understanding Bias </li></ul><ul><li>Load Balancing during logon </li></ul><ul><li>Performance Data Helper </li></ul><ul><li>Citrix and the PDH </li></ul><ul><li>Load balancing in the Data Store </li></ul>
    9. 9. Architecture Data Collector DC Dynamic Store DS LmsSS.dll IMA Subsystem LMS20Rules.dll System counters MFRules.dll MetaFrame counters PDH.dll Performance Data Helper
    10. 10. Data Collector Updates <ul><li>Connection logon or logoff </li></ul><ul><li>Change in the load evaluator +/- 500 </li></ul><ul><li>DC(DS) is updated every 30 s </li></ul><ul><li>If nothing changes in 1 minute the DC(DS) sends 40 byte ping, to make sure they are alive. </li></ul><ul><li>If nothing changes in 5 minutes the DC(DS) request a server load update. </li></ul>
    11. 11. Understanding Bias <ul><li>Definition of Bias in Science </li></ul><ul><li>Bias is defined as the difference between the measurement result and its unknown 'true value' </li></ul>
    12. 12. Understanding Bias <ul><li>queryds /table:LMS_ServerLoadTable </li></ul>5: Scheduling 6: Page Swaps 7: Disk Data I/O 8: Disk operations 9: IP Range a: Application user load b: Server User load 1: CPU Utilization 2: Context switches 3: Memory Usage 4: Page Faults Bias = 10000/(40 USERS) = 250 DEC = fa HEX Load = 3E8/FA HEX = 4 USERS
    13. 13. Load Balancing during logon IMA Server B Ctxnotif.dll LMSSS.dll MFSrvSS.dll Server A WEVENT_LOGON ICA Client Data Collector Server A Server B Dynamic Store
    14. 14. Performance Data Helper <ul><li>The performance data helper (PDH) interface calls the registry interface to retrieve performance data </li></ul><ul><li>Uses the PDH.DLL to access the PDH API </li></ul><ul><ul><li>http://msdn.microsoft.com/archive/default.asp?url=/archive/en-us/dnarperfmo/html/msdn_pdhlib.asp </li></ul></ul><ul><ul><li>http://msdn2.microsoft.com/en-us/library/aa373083.aspx </li></ul></ul>
    15. 15. Performance Data Helper <ul><li>Code examples </li></ul><ul><li>http://msdn2.microsoft.com/en-us/library/aa373169.aspx </li></ul><ul><li>… </li></ul><ul><li>TCHAR szCounterPath[45] = TEXT(&quot;rocessor(0) Processor Time&quot;); </li></ul><ul><li>… </li></ul><ul><li>pdhStatus = PdhAddCounter (hQuery, szCounterPath, 0, &hCounter); </li></ul><ul><li>… </li></ul>
    16. 16. Performance Data Helper <ul><li>The performance counters information is stored under: </li></ul><ul><ul><li>HKLMSOFTWAREMicrosoftWindowsNTCurrentVersionPerflib09 Counter </li></ul></ul><ul><ul><li>HKLMSOFTWAREMicrosoftWindows NTCurrentVersionPerflib09 Help </li></ul></ul><ul><ul><li>… </li></ul></ul><ul><ul><li>6 </li></ul></ul><ul><ul><li>% Processor Time </li></ul></ul><ul><ul><li>8 </li></ul></ul><ul><ul><li>… </li></ul></ul><ul><ul><li>… </li></ul></ul><ul><ul><li>7 </li></ul></ul><ul><ul><li>% Processor Time is the percentage of elapsed time that the processor </li></ul></ul><ul><ul><li>spends to execute a non-Idle thread. It is calculated by measuring the </li></ul></ul><ul><ul><li>duration of the idle thread is active in the sample interval, and </li></ul></ul><ul><ul><li>subtracting that time from interval duration. (Each processor has an </li></ul></ul><ul><ul><li>idle thread that consumes cycles when no other threads are ready </li></ul></ul><ul><ul><li>to run). This counter is the primary indicator of processor activity, and </li></ul></ul><ul><ul><li>displays the average percentage of busy time observed during the </li></ul></ul><ul><ul><li>sample interval. It is calculated by monitoring the time that the service </li></ul></ul><ul><ul><li>is inactive, and subtracting that value from 100%. </li></ul></ul><ul><ul><li>9 </li></ul></ul><ul><ul><li>… </li></ul></ul>
    17. 17. Performance Data Helper
    18. 18. Performance Data Helper <ul><li>The same information can be found in the following files </li></ul><ul><ul><li>%Systemroot%System32 Perfc009.dat </li></ul></ul><ul><ul><li>%Systemroot%System32 Perfh009.dat </li></ul></ul><ul><li>For instance those files will be used to recreate the performance counters again </li></ul><ul><ul><li>http://support.microsoft.com/kb/300956/en-us </li></ul></ul>
    19. 19. Performance Data Helper <ul><li>Perfmon.exe reads the registry keys </li></ul><ul><li>IMA reads the Perfc009.dat </li></ul><ul><ul><li>Only once when IMA is starting </li></ul></ul>
    20. 20. Citrix and the PDH <ul><li>We also provide some performance counters </li></ul><ul><ul><li>%Systemroot%inf09IMAServiceimaperf.ini </li></ul></ul><ul><ul><li>%Systemroot%inf09MetaFrameXPmfxpperf.ini </li></ul></ul>
    21. 21. Citrix and the PDH <ul><li>HKLMSYSTEMCurrentControlSetServicesIMAServicePerformance </li></ul>
    22. 22. Citrix and the PDH <ul><li>List of counters that LMS20Rules.dll uses </li></ul><ul><ul><li>rocessor(_Total) Processor Time </li></ul></ul><ul><ul><li>ystemontext Switches/sec </li></ul></ul><ul><ul><li>emory Committed Bytes In Use </li></ul></ul><ul><ul><li>emoryage Faults/sec </li></ul></ul><ul><ul><li>emoryages/sec </li></ul></ul><ul><ul><li>hysicalDisk(_Total)isk Bytes/sec </li></ul></ul><ul><ul><li>hysicalDisk(_Total)isk Reads/sec </li></ul></ul><ul><ul><li>hysicalDisk(_Total)isk Writes/sec </li></ul></ul>
    23. 23. Citrix and the PDH <ul><li>Same counter with different name depending on the OS version </li></ul><ul><ul><li>English W2K </li></ul></ul><ul><li>ystem Total Processor Time </li></ul><ul><ul><li>English W2K3 </li></ul></ul><ul><ul><ul><li>rocessor(_Total) Processor Time </li></ul></ul></ul>
    24. 24. Load balancing - Data Store <ul><li> </li></ul><ul><li>012C Hex = 300 dec = 5 min </li></ul><ul><li>3C Hex = 60 dec = 1 min </li></ul>
    25. 25. Load balancing - Data Store
    26. 26. Where can it go wrong? Agenda item number 1 Load Balancing on an Application Delivery Scenario Architecture Troubleshooting Where can it go wrong?
    27. 27. Where can it go wrong? <ul><li>Black hole effect </li></ul><ul><ul><li>Load Throttling - Intelligent Load Balancing </li></ul></ul><ul><li>Failing to read performance counters </li></ul><ul><li>English OS with and MUI </li></ul>
    28. 28. Black hole effect <ul><li>Problem </li></ul><ul><ul><li>At Peak logon times a recently boot up server will get all connections and might become unresponsive </li></ul></ul><ul><li>Description </li></ul><ul><ul><li>When a server comes into a farm its load will be 0 and it will be getting all new connections to the farm. At peak logon times this could make the server unresponsive if overloaded </li></ul></ul>
    29. 29. Black hole effect <ul><li>Solution </li></ul><ul><li>Load Throttling </li></ul><ul><li>??? </li></ul>
    30. 30. Load Throttling <ul><li>Load Throttling uses Intelligent Load Biasing (ILB) </li></ul><ul><ul><li>HKLMSOFTWARECitrixIMALMSUseILB </li></ul></ul><ul><li>Default =1 (0,1) </li></ul><ul><ul><li>HKLMSOFTWARECitrixIMALMSILBMultiplier </li></ul></ul><ul><li>Default =2 (2, 3, 4…) </li></ul><ul><li>Current Resolution Load += [(Max Load – Current Resolution Load) / ILBMultiplier] </li></ul>
    31. 31. Load Throttling - Low logon rate <ul><li>2 server farm (server A and server B) using the Default load evaluator </li></ul>
    32. 32. Load Throttling - Low logon rate Server B Server A Server Load BIAS 0 0 0 0 (10000-0)/2+0 = 5000 100 (10000-0)/2+0 = 5000 100 ICA Client Data Collector Server A Server B Dynamic Store
    33. 33. Load Throttling - Low logon rate Data Collector ICA Client Dynamic Store Server A Server B Server B Server A Server Load BIAS 0 0 0 0 (10000-0)/2+0 = 5000 100 (10000-0)/2+0 = 5000 100
    34. 34. Load Throttling - Low logon rate <ul><li>Server load if Low logon rate </li></ul>
    35. 35. Load Throttling - Black hole effect <ul><li>2 server farm (server A and server B) using the Default load evaluator </li></ul><ul><li>Server B is running normally (60 users) while at Peak logon times the server A is booted up </li></ul>
    36. 36. Load Throttling - Black hole effect Server B Server A Server Load BIAS 0 6000 0 0 (10000-0)/2+0 = 5000 (10000-5000)/2+5000 = 7500 (10000-6000)/2+6000 = 8000 (10000-7500)/2+7500 = 8750 x60 ICA Client Data Collector Server A Server B Dynamic Store
    37. 37. Load Throttling - Black hole effect Data Collector ICA Client Dynamic Store Server A Server B Server B Server A Server Load BIAS 0 6000 0 0 (10000-0)/2+0 = 5000 (10000-5000)/2+5000 = 7500 (10000-6000)/2+6000 = 8000 (10000-7500)/2+7500 = 8750
    38. 38. Load Throttling <ul><li>Server load at Peak logon times </li></ul>
    39. 39. Failing to read performance counters <ul><li>Problem </li></ul><ul><ul><li>One server in the farm gets all the connections </li></ul></ul><ul><li>Cause </li></ul><ul><li>If a farm is balanced using advance load evaluators and we fail to read those performance counters in one server, it will report 0 load and so it will get all the connections and will get unresponsive, even crashing eventually </li></ul>
    40. 40. Failing to read performance counters <ul><li>Solution </li></ul><ul><ul><li>From CPS4.0 HRP02 we have change the way the load is calculated </li></ul></ul><ul><ul><li>If we fail to read some performance counters we will report full load </li></ul></ul><ul><li>Note: If you were using custom load evaluators like Advance + Default after HRP02 you might report 10000 load while with HRP01 seemed to be OK </li></ul>
    41. 41. English OS with and MUI <ul><li>Problem </li></ul><ul><li>Advance load evaluators report full load when an MUI is installed </li></ul><ul><li>Cause </li></ul><ul><li>We were expecting English counters only </li></ul><ul><li>Note: To know if your problem is due to the MUI and not to a problem reading performance counters you can swap the files perfc009.dat and perfc007.dat. If that resolves the issue you are in the right direction </li></ul>
    42. 42. English OS with and MUI <ul><li>Solution </li></ul><ul><ul><li>%Program Files%/citrix/system32/citrix/ima/lms/lms20rules.dll </li></ul></ul><ul><ul><li>HKLMSoftwareCitrixIMALMSEnableTranslation=1 (dword) </li></ul></ul><ul><ul><li>To be manually added </li></ul></ul><ul><ul><li>First introduced in PSE400R02W2k3016 </li></ul></ul>
    43. 43. Architecture Agenda item number 1 Load Balancing on an Application Delivery Scenario Architecture Where can it go wrong? Troubleshooting
    44. 44. Troubleshooting <ul><li>Failing to read performance counters => full load </li></ul><ul><ul><li>Use filemon while rebooting IMA to see possible problems accessing perf*.dat files. (leave it running for a while if they don't come up) </li></ul></ul><ul><ul><li>Consider rebuilding performance counters </li></ul></ul><ul><ul><li>Check that no performance counters are disabled </li></ul></ul><ul><ul><li>[HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesPerfDiskPerformance] </li></ul></ul><ul><ul><ul><li>&quot;Disable Performance Counters“ = 1 (DWORD) </li></ul></ul></ul>
    45. 45. Troubleshooting <ul><li>Failing to read performance counters => full load </li></ul><ul><ul><li>CDF traces </li></ul></ul><ul><ul><ul><ul><li>lms20rules ERROR!!!!! _CreatePdhQueryHelper() failed for m_hDiskWriteCounter,error = -1073738823 </li></ul></ul></ul></ul><ul><ul><ul><ul><li>lms20rules LMSRuleDll_Interface::CreatePDHQuery() ERROR!!!!! rc = </li></ul></ul></ul></ul><ul><ul><ul><ul><li>-1073738823 </li></ul></ul></ul></ul><ul><ul><ul><li>-1073738823 in HEX is FFFFFFFFC0000BB9 </li></ul></ul></ul><ul><li>http://msdn2.microsoft.com/en-us/library/aa373046.aspx </li></ul><ul><li>0xC0000BB9 (PDH_CSTATUS_NO_COUNTER) The specified counter could not be found. </li></ul>
    46. 46. Summary <ul><li>Deep understanding of load balancing </li></ul><ul><ul><li>BIAS, PDH… </li></ul></ul><ul><li>Understanding Intelligent load balancing </li></ul><ul><ul><li>Load Throttling </li></ul></ul><ul><li>Typical scenarios </li></ul><ul><ul><li>Full load versus 0 load, MUI… </li></ul></ul><ul><li>Troubleshooting techniques </li></ul><ul><ul><li>So far we are resolving all the problems with those tips </li></ul></ul>
    47. 47. Q&A <ul><li>? </li></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×