Rock Solid Sametime for High Availability


Published on

My Best Practices session from Connect 13

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Rock Solid Sametime for High Availability

  1. 1. Rock Solid Sametime For High Availability BP301: Rock Solid IBM Sametime For High Availability Gabriella Davis The Turtle Partnership© 2013 IBM Corporation
  2. 2. Gabriella Davis§ Proud Nerd Girl –Mathmo / Problem Solver / System Designer / Optimist§ ccMail & Agenda then Lotus & WAS –I’m much older than I look§ Co-Author of Sametime 8.5.2 Admin Guide –Available at all good bookshops but mostly Amazon –Domino & Exchange, Sametime & Lync Server, Sharepoint§ Co-Author (being updated for Connections 4)§ I present a lot globally & blog in fits and starts§ The Turtle Partnership –High Level Support of IBM Lotus products • 20% support, 40% system design and implementation, 40% development • 50% of our customers are in Europe and 50% in the US (nothing against Australasia mind you)2
  3. 3. There’s Nothing Sexy About High Availability§ Believe me I tried to get as many jokes in as I could§ There’s high availability and there’s disaster recovery and there’s load balancing and a slew of options in between§ Your Sametime install could be 3 servers or it could be 50 servers. –I’ve done both ends of the scale§ Build for now but design for the future –Adding new servers or services later on can be a simple process if you’ve designed correctly from the start 3
  4. 4. How This Presentation Works§ There’s no single right way to do this, it depends on what services are critical to you and what demand you expect for each service§ My goal is to take you through each Sametime service and explain how you would design each for clustering, load balancing and failover –I use lots of pictures so we can all visualise what I’m talking about –also hand puppets§ I also want to talk about the cool things that you may not know happen under the hood and some things you need to be careful of§ There’s a lot of information here and I’m going to have to go fast so grab the presentation and feel free to ask questions afterwards ! 4
  5. 5. Sametime Servers and Services§ Instant Messaging§ Web clients, mobile clients, web based awareness§ Meetings§ Audio and Video services§ Audio / Video reflector services (for A/V across networks)§ Audio / Video traffic management 5
  6. 6. Sametime System Console§ The SSC is the management and administration environment for all Sametime servers and services –You can build without a SSC but I don’t recommend it unless you love messing about with XML files§ The SSC is a WebSphere based application and cannot be clustered§ Servers within the same SSC are aware of each other –The Meeting servers know about the Media servers etc§ There can only be one cluster of each type in a SSC –One meeting cluster –One sametime proxy cluster –One proxy registrar cluster etc –THIS IS GOING TO BE CRITICAL LATER ON WHEN WE TALK ABOUT PLANNING§ You can’t cluster the SSC (it was worth repeating) 6
  7. 7. Database Server§ Several of the Sametime servers use a DB2 database for management –Meetings –Gateway –System Console –Bandwidth Manager –Advanced –even the Proxy server has a DB2 database it uses for iPhone traffic§ DB2 has several high availability models but the DB2 9.7 license supplied with your Sametime licensing entitles you to use DB2 in a HADR configuration with a single active and another passive server –The switchover from active to passive is manual –The passive DB2 server cannot be used or accessed but will take updates from the active server and can itself be made the active server at any time§ Without HADR you could lose your DB2 server –Instant Messaging, Audio and Video as well as Web / Mobile clients will continue to work 7
  8. 8. Edge Load Balancer§ Licensed for use with Sametime§ Very easy to deploy and configure on a wide range of platforms with a GUI interface for management§ Always use the new ULB IPV4/IPV6 load balancer and not the older IPV4 one which is being deprecated§ To set up a server for load balancing you need to –Assign a virtual ip address that can be used by each cluster of servers and that points to the ULB machine • So if you’re load balancing Meetings, Sametime Proxy, Conference Manager, Proxy Registrar, Packet Switcher and TURN server you need 6 separate unique virtual ips all pointing to the load balancer –Create a Loopback adapter on your backend servers (ie your Meeting, Media etc servers) configured to use the virtual ip address –Create routing rules in the ULB (Edge Load Balancer) wizard to forward traffic to the backend servers 8
  9. 9. Load Balancer - Tips§ You can use any load balancer to manage traffic to the Meeting, Sametime Proxy, Gateway, Advanced and Community Servers but the Audio / Video components must be configured to use MAC forwarding§ For many load balancer appliances that’s a network wide configuration setting that administrators don’t like to do (I have been told this many times :-)) –In those cases you could deploy the Edge LB to handle the A/V traffic§ To configure a loopback adapter on Windows 2008 you need to go to Device Manager and right mouse click on the server name then choose “Add Legacy Hardware” - you can then go ahead and add a loopback network adapter by choosing Microsoft as the provider –There are specific network configuration settings that need to be run on a Windows 2008 server to set up the loopback adapter correctly • 9
  10. 10. Domino Clustering§ What does Domino clustering give us in a Sametime world? –vpuserinfo (contact list and privacy information) replicated between servers in seconds –directory information replicated between servers, if you set it up correctly • Beware of using a replica of Directory Assistance on multiple servers without proper configuration –don’t cluster stconfig.nsf, its information is server specific§ It can’t replicate Sametime specific information such as policies (which are held in stconfig.nsf and business card configuration which is held in the file userinfoconfig.xml)10
  11. 11. Domino Clustering for Sametime - Tips§ If you’re running the Sametime System Console, it controls Sametime configuration settings in stconfig.nsf including policies, trusted ips and LDAP configuration§ Changing the LDAP deployment in the SSC will cause it to overwrite the LDAP settings in stconfig.nsf when Domino next restarts –but only once, if you then edit it the LDAP document in stconfig.nsf via Notes, it won’t be overwritten –Make sure you edit the LDAP and Policies documents always in the same place - the SSC, don’t be tempted to quickly edit a Notes document in stconfig.nsf as the settings may not hold11
  12. 12. Domino Clustering for Sametime - Weirdness§ If you have multiple LDAP configurations in stconfig.nsf the priority is NOT the “Search Order” as defined on the LDAP document but is the order in which the LDAP documents appear in the view (by modified date). –This priority determines which directory is searched first when finding contacts§ To ensure this doesn’t happen you’ll need to change the design of the “All - by form and date” view in stconfig.nsf to sort LDAP documents by Search Order field The search order says that we want this directory searched 2nd but because it was modified most recently, it will be searched first
  13. 13. Sametime Clustering§ What does Sametime Clustering give us? –The Community Server is central to everything Sametime does. –None of the other components function without a Community server to talk to –This makes clustering and availability of Community servers the most important component in your design –Users can log on to any Community server in the cluster and expect the same experience • Assuming you have your Domino clustering and Sametime customisation set up correctly ✦vpuserinfo.nsf replicating ✦sametime.ini with same settings ✦userinfoconfig.xml the same ✦LDAP configuration in stconfig.nsf the same –Other Sametime components such as the Sametime Proxy or Meeting servers will be able to recognise a cluster mate as a failover alternative13
  14. 14. Sametime Clustering - Tips§ Verify Sametime.ini settings such as security level and allowable clients match on all cluster-mates – VPS_ALLOWED_LOGIN_TYPES – VP_ONLY_SINGLE_LOGIN_ALLOWED – VP_SECURITY_LEVEL§ Don’t have one cluster server running a different version of Sametime than its other cluster-mates§ Users with no Sametime Server defined in their person document have no home Sametime server and can log into any server in the Community (ie in the Domino Directory) –Users with a cluster name in the Sametime Server field in the person document can log into another server in the cluster –Use an LDAP attribute to store the home Sametime server name or cluster name for a user so it’s accessible to other products such as Connections • The sametime server name should be in hierarchical format eg cn=st1,o=turtlehost or as a cluster name cn=stcluster 14
  15. 15. Sametime Clustering - Weirdness§ Trusted ips in the stconfig.nsf determine what servers can connect to this Sametime server –These might be other Sametime servers or Domino servers using iNotes or Sametime Proxy servers or Meeting or Media servers • The list can get pretty long§ The list of ips is updated in the Sametime System Console if you have one and this overwrites the configuration document in stconfig.nsf –Don’t edit the list in stconfig.nsf if you are running a SSC as your edits will be overwritten§ One bug I continually see is when the trusted ips list in the SSC gets very long, it writes out to the Community Trusted IPs field in the Community Connectivity document as a long string instead of a list –This breaks all trusted ip connections –You can open and save the Notes document to fix it, saving recalculates the field correctly as a list –It will break again next time you update the SSC15
  16. 16. Community Multiplexors - Multiple Multiplexors!§ The Community Multiplexor (MUX) is a service running as part of your Sametime Community Server –It’s responsible for authenticating users and connecting them to the back end Sametime services –In a standard install the MUX runs as a service on the Community Server –You can install a Community MUX separately from the Community server to offload the authentication and client network connections to another machine • The MUX maintains a single network connection to the Sametime Community Server increasing the capacity of the server to 100k concurrent users§ A MUX can be used to front-end multiple servers in a cluster, the assumption being that users can connect to any and all of the listed servers§ The sametime.ini file in a MUX contains a list of servers it can connect to [Config] VPMX_CAPACITY=80000 [Connectivity],, stchat3.turtlehost.net16
  17. 17. Community Multiplexors - Tips§ You don’t have to have a separate multiplexor and if you’re going to only have one it itself becomes a single point of failure§ It used to be that that MUX’s were ‘dumb’ in that they would round robin connections to their known servers, trying each one in turn, even if that server is down –That’s no longer the case and the MUX’s are now service aware and will not route traffic to unavailable servers –They have become load balancers for Sametime Community Server traffic§ Everything in Sametime world pivots around DNS and fully qualified hostnames, the hostname your Sametime clients ask for must eventually resolve to a Multiplexor somewhere –If you have multiple multiplexors you will therefore need to use a load balancer to make sure traffic isn’t sent to a non responding multiplexor17
  18. 18. So What Does It All Look Like?18
  19. 19. Instant Messaging (Sametime Clients) Clustering§ Sametime servers clustered, users can access either server but the FQHN defined in their client connections will determine here they end up§ There is no failover in the Sametime client Domino Cluster Domino Server A Domino Server B Sametime Cluster Sametime Server Sametime Server stchat2.connect13.com19
  20. 20. Instant Messaging (Sametime Clients) Failover§ Load Balancer in front of Clustered Domino servers§ Users are directed to either server and get the same experience Load Balancer Domino Cluster Domino Server A Domino Server B Sametime Cluster Sametime Server Sametime Server stchat2.connect13.com20
  21. 21. Instant Messaging (Sametime Clients) Failover§ Use a Multiplexor instead of a Load Balancer in front of your Sametime cluster and take advantage of the failover logic built into it§ Define multiple servers in the sametime.ini file in the Multiplexor Sametime Multiplexor Domino Cluster Domino Server A Domino Server B Sametime Cluster Sametime Server Sametime Server stchat2.connect13.com21
  22. 22. Instant Messaging (Sametime Clients) Failover§ If you don’t want your multiplexor to be a single point of failure then put multiple multiplexors behind a load balancer. The client configuration FQHN points to the load balancer and the multiplexors connect to either Community server Load Balancer Sametime Multiplexor Sametime Multiplexor Domino Cluster Domino Server A Domino Server B Sametime Cluster Sametime Server Sametime Server stchat2.connect13.com22
  23. 23. Instant Messaging (Web and Mobile) Clustering§ Multiple Community Servers§ Single Sametime Proxy Server§ Sametime Proxy Server can be directed to a single Community Server but can and will utilise any Community server it is trusted for in your Domino Directory Sametime Proxy Server IBM WAS HTTP Proxy (80/443) Sametime Community Servers Not Necessarily Clustered stchat4.connect13.com23
  24. 24. Instant Messaging (Web and Mobile) Clustering§ Clustered Community Servers§ Single Sametime Proxy Servers Sametime Proxy Server IBM WAS HTTP Proxy (80/443) Sametime Community Servers CLUSTERED stchat3.connect13.com24
  25. 25. Instant Messaging (Web and Mobile) Failover§ Clustered Community Servers§ Multiple Sametime Proxy Servers (not clustered)§ Load Balancer Load Balancer Sametime Proxy Server Sametime Proxy Server IBM WAS HTTP Proxy IBM WAS HTTP Proxy (80/443) (80/443) Sametime Community Servers CLUSTERED stchat3.connect13.com25
  26. 26. WebSphere Clustering§ Each Sametime server (other than the Community Server) is installed on a WebSphere node.§ Each node must have a single primary instance but can have multiple secondary instances installed as part of a cluster in either –Horizontal (servers installed on different machines) –Vertical (servers installed on the same machine but uses their own dedicated resources)§ WebSphere has built in logic to share resources across Sametime servers in a cluster –You don’t have to configure it to do that§ BUT, the Sametime System Console can only manage 1 cluster of each server type26
  27. 27. WebSphere - Tips§ Attempting to deploy multiple applications on a single server is a very bad idea§ Each application (Meetings, Media etc) must have its own FQHN and dedicated ip address§ To ensure WebSphere binds the correct ip address to the correct application (and avoids port conflicts with other applications already installed) you have to explicitly edit the underlying XML files to creating the hostname bindings§ Even then WebSphere’s will bind the bootstrap port to the first ip address on the box§ We’ve also discovered that all traffic sent from any of the servers has a source ip of the first ip on the box (Windows 2008 R2)§ In general it’s to be avoided. It can be done but Virtual Machines is a far cleaner, easier and more reliable way to go27
  28. 28. WAS Proxy Clustering§ The WebSphere WAS Proxy server can act as both a HTTP or SIP proxy to sit in front of your Sametime servers§ Each Sametime server installs on its own application port (default is 9081 / 9043 secure but that will increment by one for every other server installed on the same machine that conflicts)§ We don’t want to construct URLs that have port numbers in them so for HTTP services we use a WAS HTTP Proxy server in front of the Sametime application to manage the traffic on port 80 /443 secure§ They are cluster aware, able to identify when an application server is unavailable and redirect the traffic to that server’s cluster mate which provides built in failover§ The WAS Proxy is often installed on the same node and server as the application server it is managing but that doesn’t have to be the case, it can install standalone28
  29. 29. WAS HTTP Proxy - Tip§ The WAS HTTP Proxy is much more than a reverse proxy –It is part of the Sametime System Console and will authenticate users before directing the traffic onto a single destination - the application server it is supporting –A reverse proxy doesn’t have the same intelligence to validate that traffic should be allowed through and should only be allowed through to a specific destination§ We primarily use WAS HTTP Proxies with Meeting and Sametime Proxy servers§ You can have multiple WAS HTTP Proxies providing service to the same application server –You can cluster WAS HTTP Proxies and then they will also monitor each other§ Don’t create WAS HTTP Proxies with the SIP checkbox enabled, we use SIP for Audio / Video traffic and it can cause conflicts in the environment with other servers if you leave it enabled29
  30. 30. Meeting Server Clustering§ Meeting Servers can be clustered in Sametime by installing a single server as a primary node and then a series of secondary nodes either on the same machine (vertical clustering) or, more usually, on multiple machines (horizontal clustering)§ All Meeting servers in a cluster share the same DB2 database§ In a Meeting server cluster WebSphere treats all the servers as equal§ When a user creates a meeting WebSphere will decide which of the Meeting servers will host that meeting –This is very important as all users attending the meeting will be directed to the same host meeting server –it is outside of your control to determine which server will be selected30
  31. 31. Meeting Server Clustering - Tips§ If you build a Meeting Server cluster with servers in France, Singapore and Chicago - a meeting of users in Chicago could just as easily be directed to France –WebSphere will choose which cluster-mate hosts each meeting when the meeting is first created§ The WAS HTTP Proxy will be able to detect if the host meeting server is down and redirect all user requests to an alternate server by rewriting the URL§ Don’t create a meeting server cluster with one server inside a firewall and one outside unless you are prepared to open ports to allow public access to your internal servers§ Consider your network architecture when designing your meeting servers, if you want multiple clusters you will need to have multiple System Consoles§ The Community Server a user logs in to determines their environment from SSC to Meeting, and Media Servers –but users can log into different environments depending on where they are31
  32. 32. Meetings Multiple Servers§ Three Meeting Servers, all independent, none clustered, each with their own DB2 database (and different DB2 servers if you want)§ Meetings will be created on a specific server as determined by the user creating the meeting “where do you want to create this meeting”§ The WAS HTTP Proxies are to ensure servers are accessible on 80 /443 Sametime Meeting Servers DB2 Active Server IBM WAS HTTP Proxy STMeet Database(s) IBM WAS HTTP Proxy DB2 Passive Server STMeet Database(s) IBM WAS HTTP Proxy32
  33. 33. Meetings Clustering§ Three Meeting Servers, this time in a cluster§ Meetings are created on any of the three servers, determined by WebSphere and outside of user control§ The WAS HTTP Proxies can proxy traffic for only one server each§ It’s your decision if you want to separate servers and proxies onto their own machines, you don’t have to Sametime Clustered Meeting Servers DB2 Active Server STMeet Database(s) DB2 Passive Server STMeet Database(s) IBM WAS HTTP Proxy IBM WAS HTTP Proxy IBM WAS HTTP Proxy33
  34. 34. Meetings Failover§ With the WAS HTTP Proxies themselves clustered together, each one can provide service for any of the Meeting servers that are clustered Clustered Meeting Servers DB2 Active Server STMeet Database(s) DB2 Passive Server STMeet Database(s) Cluster WAS Proxy Servers IBM WAS HTTP Proxy IBM WAS HTTP Proxy Load Balancer34
  35. 35. Media Manager§ Media Manager - contains all media components (other than Bandwidth Manager) –Conference Manager –Proxy Registrar –Packet Switcher§ Up to two Media Managers can be clustered, but only two and only one cluster in a SSC Sametime Media Servers CLUSTERED stmedia2.connect13.com35
  36. 36. WAS SIP Proxy§ When deploying a Proxy Registrar or Conference Manager as standalone components, each needs to have a dedicated WAS SIP Proxy server to handle traffic between the servers themselves and to / from the clients§ WAS SIP Proxies aren’t clustered and are deployed usually in a 1 - 1 relationship with the application server –However any of the servers can provide Audio / Video service to users so it doesn’t matter which server a request is directed to§ The Packet Switcher component cannot be clustered and does not require a WAS SIP Proxy§ When creating a new WAS Proxy server for Audio / Video components you deselect HTTP and select SIP to tell the newly created server it will be handling SIP traffic only§ WAS SIP Proxies deploy on the standard ports of 5062 (SIP) and 5063 (SIPS). –These may be incremented by 1 if there is conflicting port activity on the same server36
  37. 37. Audio Video Components - Clustered§ Some Individual components can be clustered –Proxy Registrar –Conference Manager§ Packet Switchers can’t be clustered but there can be multiple PS instances§ SIP Proxies must be installed in front of the Proxy Registrar and Conference Managers Sametime Media Components CLUSTERED Conference Manager Cluster IBM WAS SIP Proxy Multiple Packet Switchers (NON CLUSTERED) Proxy Registrar Cluster IBM WAS SIP Proxy stpr2.connect13.com37
  38. 38. Audio Video Components - Failover§ In this design I have clustered the SIP Proxies so they provide failover for each other via a Load Balancer§ This isn’t strictly necessary if you have a Load Balancer in front as it should be able to detect an unavailable SIP Proxy and redirect the user requests accordingly Sametime Media Components CLUSTERED Conference Manager Cluster SIP Cluster IBM WAS SIP Proxy Multiple Packet Switchers IBM WAS SIP Proxy (NON CLUSTERED) SIP Cluster Proxy Registrar Cluster IBM WAS SIP Proxy IBM WAS SIP Proxy stprsip1.connect13.com38 Load Balancer
  39. 39. Audio Video Failover - With Bandwidth Manager§ Bandwidth Manager should be deployed in any environment where there is a significant requirement for Audio / Video or where the user network is not consistent§ The Bandwidth Manager cannot be installed in the same cell as the Media components because of conflicts with their SIP configuration Sametime Media Components Bandwidth Manager CLUSTERED Cluster SIP Cluster Separate Cell Conference Manager Cluster IBM WAS SIP Proxy Multiple Packet Switchers BWM Cluster IBM WAS SIP Proxy (NON CLUSTERED) stbwm1…… stbwm2…… SIP Cluster Proxy Registrar Cluster IBM WAS SIP Proxy SIP Cluster WAS SIP Proxy IBM WAS SIP Proxy stbwmsip1 WAS SIP Proxy stbwmsip2 Load Balancer39
  40. 40. Sametime Advanced§ Sametime Advanced can also be deployed as a cluster of multiple servers§ Each cluster-mate will use the same DB2 database§ We don’t deploy a WAS HTTP Proxy with Sametime Advanced because although many of the applications are on HTTP ports (9080 / 9443), the Broadcast Tools are on ports that cannot be proxied (1883, 8883)§ Since we can’t use a proxy in front of Sametime Advanced servers, if you want to access them from a public network, they must be located in a DMZ or somewhere you are confident exposing to a public interface Public DMZ Internal Clustered Advanced Servers DB2 Active Server STMeet Database(s) Load Balancer40
  41. 41. TURN Servers§ TURN Server handle audio and video traffic being redirected between clients on different networks –Similar to the old reflector service on pre 8.5 Sametime§ The TURN Server is defined by its hostname in the configuration of the Media Components in the SSC –A user is assigned media components based on which Community server and therefore SSC environment they log into –Meetings don’t use the TURN server§ You can have as many TURN servers as you want fronted by a Load Balancer –users involved in a conference don’t need to be on the same TURN server41
  42. 42. TURN Server - Tip§ If you deploy TURN configuration on your Media servers then the FQHN you use for the TURN server has to be resolvable for all clients, both internal and external§ Multiple TURN servers on different networks can cause Audio and Video latency issues if the users involved in a conference are directed to different TURN servers such as one internal and one in a DMZ§ All users must be able to access the TURN server on port 3478 either UDP or TCP (whichever you’ve configured) –UDP is more efficient than TCP for routing this type of traffic42
  43. 43. Public Access, Firewalls and DMZ§ The biggest issue you may find with deploying high availability Sametime is in deciding how access will work§ If you’re prepared to deploy a VPN to all users then you are basically building an internal only environment§ If you want to deploy publicly then there are a few things to consider –In the DMZ we deploy the WAS proxy elements only ie the WAS HTTP Proxy that then accesses the Meeting servers on the internal network –WebSphere has about 20 ports that need to be bi-drectionally open between the SSC and all other server components, even those in the DMZ –In addition each server has to be able to connect to each other on discovery ports, there are about 10 of those that need to be open and accessible between every server in the environment –For public access both the Sametime Gateway and Sametime Advanced applications must be in the DMZ as their traffic can’t be proxied§ Think about how iPad, Android, Smartphone access will work - will have need to have WAS HTTP Proxies to the Sametime Proxy servers in the DMZ to support them? –For notifications to iOS devices you’ll also need to open access to the apple gateway43
  44. 44. Thank You !§ Gabriella Davis – – – – (IM) (IM) (IM) –gabrielladavis (skype) –www.turtlepartnership.com44
  45. 45. Legal Disclaimer© IBM Corporation 2009. All Rights Reserved.The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is providedAS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not beresponsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating anywarranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in thispresentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothingcontained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States,other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.IJava and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.Other company, product, or service names may be trademarks or service marks of others.45