Demystifying Oracle RAC Workload Management by Alex Gorbachev, Pythian | NoCOUG 2010 Spring Conference

6,696 views

Published on

This presentation starts with a brief overview of connection management in Oracle RAC and then focuses on workload management, including exploiting cluster services, connection load balancing, and run-time workload balancing. Finally, it shows how to create your own run-time workload management, using the standard load balancing advisory framework. The presentation goes through demonstrations using a live Oracle RAC cluster, illustrating the internals of workload management and how to monitor it. The live demos simplify understanding of this rather complex topic.

Published in: Technology

Demystifying Oracle RAC Workload Management by Alex Gorbachev, Pythian | NoCOUG 2010 Spring Conference

  1. 1. Demystifying Oracle RAC Workload Management North California Oracle User Group 20-May-2010 Alex Gorbachev
  2. 2. Alex Gorbachev today... • CTO, The Pythian Group • Blogger • OakTable Network member • Oracle ACE Director • BattleAgainstAnyGuess.com • Vice-president, Oracle RAC SIG 2 © 2009/2010 Pythian
  3. 3. Why Companies Trust Pythian • Recognized Leader: • Global industry-leader in remote database administration services and consulting for Oracle, Oracle Applications, MySQL and SQL Server • Work with over 150 multinational companies such as Forbes.com, Fox Interactive media, and MDS Inc. to help manage their complex IT deployments • Expertise: • One of the world’s largest concentrations of dedicated, full-time DBA expertise. • Global Reach & Scalability: • 24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response 3 © 2009/2010 Pythian
  4. 4. Where we are... !Client-side basics • Oracle Cluster Services • Server-side CLB • FAN - HA & LBA events • Thin JDBC RLB • ONS Subscriber • Your own LBA! © 2009/2010 Pythian
  5. 5. Simple example • Establishingone connection • Dedicated server • Several RAC nodes © 2009/2010 Pythian
  6. 6. Client-side CLB (DESCRIPTION= (ADDRESS_LIST= (LOAD_BALANCE=ON) (FAILOVER=ON) (ADDRESS=(PROTOCOL=TCP)(HOST=lh1-vip)(PORT=1521)) (ADDRESS=(PROTOCOL=TCP)(HOST=lh2-vip)(PORT=1521)) (ADDRESS=(PROTOCOL=TCP)(HOST=lh3-vip)(PORT=1521)) (ADDRESS=(PROTOCOL=TCP)(HOST=lh4-vip)(PORT=1521)) ) (CONNECT_DATA=(SERVICE_NAME=service10g)) ) 6 © 2009/2010 Pythian
  7. 7. Client-side connection balancing and connection-time failover lh1-vip lh2-vip lh3-vip lh4-vip LISTENER_G41 LISTENER_G42 LISTENER_G43 LISTENER_G44 lh1 lh2 lh3 lh4 © 2009/2010 Pythian
  8. 8. Client-side connection balancing and connection-time failover lh1-vip lh2-vip lh3-vip lh4-vip LISTENER_G41 LISTENER_G42 LISTENER_G43 LISTENER_G44 lh1 lh2 lh3 lh4 © 2009/2010 Pythian
  9. 9. Client-side connection balancing and connection-time failover LOAD_BALANCE=OFF lh1-vip FAILOVER=OFF lh2-vip lh3-vip lh4-vip LISTENER_G41 LISTENER_G42 LISTENER_G43 LISTENER_G44 lh1 lh2 lh3 lh4 © 2009/2010 Pythian
  10. 10. Client-side connection balancing and connection-time failover LOAD_BALANCE=OFF lh1-vip FAILOVER=OFF lh2-vip lh3-vip lh4-vip LISTENER_G41 LISTENER_G42 LISTENER_G43 LISTENER_G44 lh1 lh2 lh3 lh4 © 2009/2010 Pythian
  11. 11. Client-side connection balancing and connection-time failover lh1-vip lh2-vip lh3-vip lh4-vip LOAD_BALANCE=OFF FAILOVER=ON LISTENER_G41 LISTENER_G42 LISTENER_G43 LISTENER_G44 lh1 lh2 lh3 lh4 © 2009/2010 Pythian
  12. 12. Client-side connection balancing and connection-time failover lh1-vip lh2-vip lh3-vip lh4-vip LOAD_BALANCE=OFF FAILOVER=ON LISTENER_G41 LISTENER_G42 LISTENER_G43 LISTENER_G44 lh1 lh2 lh3 lh4 © 2009/2010 Pythian
  13. 13. Client-side connection balancing and connection-time failover lh1-vip lh2-vip lh3-vip lh4-vip LOAD_BALANCE=ON FAILOVER=ON LISTENER_G41 LISTENER_G42 LISTENER_G43 LISTENER_G44 lh1 lh2 lh3 lh4 © 2009/2010 Pythian
  14. 14. Client-side connection balancing and connection-time failover lh1-vip lh2-vip lh3-vip lh4-vip LOAD_BALANCE=ON FAILOVER=ON LISTENER_G41 LISTENER_G42 LISTENER_G43 LISTENER_G44 lh1 lh2 lh3 lh4 © 2009/2010 Pythian
  15. 15. Summary – client-side business • Client chooses between several addresses • Why • Failure tolerance • Distribution across several listeners • How • Several connection points in connection descriptor • LOAD_BALANCE=ON • FAILOVER=ON © 2009/2010 Pythian
  16. 16. Listener – where to direct request? lsnrctl status …… Service "service10g" has 2 instance(s). Instance "g41", status READY, has 1 handler(s) for this service... Instance "g42", status READY, has 1 handler(s) for this service... …… © 2009/2010 Pythian
  17. 17. Where we are... !Client-side basics !Oracle Cluster Services • Server-side CLB • FAN - HA & LBA events • Thin JDBC RLB • ONS Subscriber • Your own LBA! © 2009/2010 Pythian
  18. 18. Orders processing example • Business areas • Customers’ order (web) • Content display (web cached on app-tier) • Feedback (web) • Orders fulfillment (back-office) • Data extraction batches 11 © 2009/2010 Pythian
  19. 19. Services: preferred & available NEW_ORD NEW_ORD NEW_ORD NEW_ORD CONTENT CONTENT CONTENT CONTENT FEEDBACK FEEDBACK FEEDBACK FEEDBACK PROC_ORD PROC_ORD BATCH BATCH DB1 DB2 DB3 DB4 12 © 2009/2010 Pythian
  20. 20. Node affinity NEW_ORD NEW_ORD CONTENT CONTENT FEEDBACK FEEDBACK PROC_ORD PROC_ORD BATCH DB1 DB2 DB3 DB4 13 © 2009/2010 Pythian
  21. 21. Service failover NEW_ORD NEW_ORD CONTENT CONTENT FEEDBACK FEEDBACK PROC_ORD PROC_ORD BATCH DB1 DB2 DB3 DB4 14 © 2009/2010 Pythian
  22. 22. Service failover NEW_ORD NEW_ORD CONTENT CONTENT FEEDBACK FEEDBACK PROC_ORD PROC_ORD BATCH DB1 DB2 DB3 DB4 14 © 2009/2010 Pythian
  23. 23. Service failover NEW_ORD NEW_ORD CONTENT CONTENT FEEDBACK FEEDBACK PROC_ORD PROC_ORD BATCH DB1 DB2 DB3 DB4 14 © 2009/2010 Pythian
  24. 24. Service failover NEW_ORD NEW_ORD CONTENT CONTENT FEEDBACK FEEDBACK PROC_ORD PROC_ORD BATCH DB1 DB2 DB3 DB4 14 © 2009/2010 Pythian
  25. 25. Service failover NEW_ORD NEW_ORD CONTENT CONTENT FEEDBACK FEEDBACK PROC_ORD PROC_ORD BATCH DB1 DB2 DB3 DB4 14 © 2009/2010 Pythian
  26. 26. Demo 1 - Services Automation • HA events introduction • Logging example • Using server-side FAN HA events callouts 15 © 2009/2010 Pythian
  27. 27. Server-side FAN HA events • Server-side callout can be used for: • Relocating services back to preferred nodes • Smart services handling • Notify and move sessions after service Down • Rebalance sessions on service up • Accommodate workload re-balancing for legacy applications 16 © 2009/2010 Pythian
  28. 28. Services & Resource Manager DBMS_RESOURCE_MANAGER.SET_CONSUMER_GROUP_MAPPING (DBMS_RESOURCE_MANAGER.SERVICE_NAME, 'NEW_ORD', 'NEW_ORD_GROUP'); DBMS_RESOURCE_MANAGER.SET_CONSUMER_GROUP_MAPPING (DBMS_RESOURCE_MANAGER.SERVICE_NAME, 'CONTENT', 'CONTENT_GROUP'); ... • Works on instance level - not RAC-aware • Doesn’t account for >1 instances per node 17 © 2009/2010 Pythian
  29. 29. Services abstraction for apps • Applicationsdon’t need to know current services configuration! • Listeners know services status • Listeners control which instance gets a new connection 18 © 2009/2010 Pythian
  30. 30. Where we are... !Client-side basics !Oracle Cluster Services !Server-side CLB • FAN - HA & LBA events • Thin JDBC RLB • ONS Subscriber • Your own LBA! © 2009/2010 Pythian
  31. 31. Remote listener registration LISTENER_G41 LISTENER_G41 G41 G42 lh1 lh2 © 2009/2010 Pythian
  32. 32. Remote listener registration LISTENER_G41 LISTENER_G41 Local registration G41.local_listener=LISTENER_G41 G42.local_listener=LISTENER_G42 G43.local_listener=LISTENER_G43 G44.local_listener=LISTENER_G44 G41 G42 lh1 lh2 © 2009/2010 Pythian
  33. 33. Remote listener registration LISTENER_G41 LISTENER_G41 Local registration G41.local_listener=LISTENER_G41 G42.local_listener=LISTENER_G42 G43.local_listener=LISTENER_G43 G44.local_listener=LISTENER_G44 G41 G42 Remote registration *.remote_listener=LISTENERS_G4 lh1 lh2 © 2009/2010 Pythian
  34. 34. Correct listener configuration (google for Pythian video VIP) lsnrctl service Service "service10g.oracloid.com" has 2 instance(s). Instance "g41", status READY, has 1 handler(s) for this service... Handler(s): "DEDICATED" established:0 refused:0 state:ready REMOTE SERVER (ADDRESS=(PROTOCOL=TCP)(HOST=lh1-vip)(PORT=1521)) "DEDICATED" established:0 refused:0 state:ready LOCAL SERVER Instance "g42", status READY, has 1 handler(s) for this service... Handler(s): "DEDICATED" established:0 refused:0 state:ready REMOTE SERVER (ADDRESS=(PROTOCOL=TCP)(HOST=lh2-vip)(PORT=1521)) Instance "g43", status READY, has 1 handler(s) for this service... Handler(s): "DEDICATED" established:0 refused:0 state:ready REMOTE SERVER (ADDRESS=(PROTOCOL=TCP)(HOST=lh3-vip)(PORT=1521)) Instance "g44", status READY, has 1 handler(s) for this ... © 2009/2010 Pythian
  35. 35. Server-side CLB LISTENER_G44 G41 G42 G43 G44 22 © 2009/2010 Pythian
  36. 36. Server-side CLB LISTENER_G44 Which instance? G41 G42 G43 G44 22 © 2009/2010 Pythian
  37. 37. Server-side CLB LISTENER_G44 Which instance? • <= 10gR1 • Instance load • Node load G41 G42 G43 G44 22 © 2009/2010 Pythian
  38. 38. Server-side CLB LISTENER_G44 Which instance? • <= 10gR1 • Instance load • Node load • 10gR2 G41 G42 G43 G44 • Service Metrics • AWR 22 © 2009/2010 Pythian
  39. 39. Demo helper Tracing listener lsnrctl trace user nsglgrDoRegister: inst loads: ld1:145 mld1:5120 ld2:5 mld2:170 nsglgrDoRegister: inst loads: ld1:58 mld1:5120 ld2:4 mld2:170 • No notion of instance name in trace ! • ld1 – Load Data 1, mld1 – Max Load Data 1 • ld2 – Load Data 2, mld2 – Max Load Data 2 • Load data is instance load and node load © 2009/2010 Pythian
  40. 40. Demo helper PREFER_LEAST_LOADED_NODE • listener.ora parameter • PREFER_LEAST_LOADED_NODE_% • ON (default) • ld1 => node, ld2 = > instance • OFF • ld1 => instance, ld1 => node © 2009/2010 Pythian
  41. 41. Demo helper Instance and node loads • Instance load • Node load • Number of user sessions • Index based probably • TYPE<>BACKGROUND on load average • Max • Max node load instance load • Max • Max CPU capacity number of sessions • cpu_count*5K • SESSIONS init.ora © 2009/2010 Pythian
  42. 42. Demo helper Uneven connection distribution • PREFER_LEAST_LOADED_NODE_...=ON • Default • Listeneruses CPU load to balance connections to the machine with more CPU capacity left • Can work fine with differently sized nodes • But what if connections are “permanent”? © 2009/2010 Pythian
  43. 43. Demo helper “Even” connection distribution PREFER_LEAST_LOADED_NODE_LISTENER_G41=OFF PREFER_LEAST_LOADED_NODE_LISTENER_G42=OFF • Loadbased on number of sessions left until maximum is reaches • init.ora parameter SESSIONS • Can configure SESSIONS differently for every node and influence distribution © 2009/2010 Pythian
  44. 44. Demo 3 - 9i style CLB • listener trace • PREFER_LEAST_LOADED_NODE • instance load • node load • PMON trace 10257 event 28 © 2009/2010 Pythian
  45. 45. 9i and 10gR1 CLB summary • PMON reports to listener • instance load • node load • PREFER_LEAST_LOADED_NODE • ON - node load (default) • OFF - instance load • lsnrctl trace user • PMON: 10257 trace name context forever, level 16 29 © 2009/2010 Pythian
  46. 46. DB_DOMAIN and bug 5593693 • db_domain init.ora parameter is set • service_name is not FQDN • that’s what srvctl does • Listenerappends default domain • Service metrics updates are lost • on the way from PMON => listener • Result: fallback to instance / node load • Fixed - 10.2.0.4 & 11.1.0.6 30 © 2009/2010 Pythian
  47. 47. 10gR2 and 11g server-side CLB • Service metrics for each service • GOODNESS • Attractiveness of the instance for this service • Higher is worse (should be BADNESS) • DELTA • Estimate of “badness” increase for additional session • FLAGS (bit map) • 0 – all good • 1 – blocked • 2 – crossed threshold • 4 – goodness unknown (usually when no sessions connected) 31 © 2009/2010 Pythian
  48. 48. Connection life-time • Short living sessions • Persistent / Oracle Forms / long connections • Connection pool • Oracle standard connection pool • Custom connection pool • Application server connection pool © 2009/2010 Pythian
  49. 49. Connection classification & CLB • Long connections • idle most of the time (!) • persistent connections (non-pooled) • over-allocated connection pools • Short connections • active most of the time • dynamic connection pools • Batch-like connection • long but always active • under-allocated connection pools (!) 33 © 2009/2010 Pythian
  50. 50. CLB goal • CLB_GOAL_LONG • equal distributions of sessions • CLB_GOAL_SHORT • based on Service Metrics and LBA goal (!) • GOAL_NONE • GOAL_SERVICE_TIME • GOAL_THROUGHPUT 34 © 2009/2010 Pythian
  51. 51. Demo 4 - Service Metrics & CLB • CLB_GOAL_LONG • CLB_GOAL_SHORT • listener trace • PMON trace • V$SERVICEMETRIC 35 © 2009/2010 Pythian
  52. 52. 10g services load balancing • For each service defined: • Connection load balancing goal • CLB_GOAL_SHORT • CLB_GOAL_LONG • Load Balancing Advisory - LBA • GOAL_SERVICE_TIME • GOAL_THROUGHPUT © 2009/2010 Pythian
  53. 53. V$SERVICEMETRIC • For each service and instance contains • Short interval (5 seconds) • Long interval (60 seconds) • INTSIZE_CSEC • CPUPERCALL • DBTIMEPERCALL • CPUPERSEC • DBTIMEPERSEC © 2009/2010 Pythian
  54. 54. V$SERVICEMETRIC (cont’d) • GOODNESS • Attractiveness of the instance for this service • Higher is worse (should be BADNESS) • DELTA • Estimate of “badness” increase for additional session • FLAGS (bit map) • 0 – all good • 1 – blocked • 2 – crossed threshold • 4 – goodness unknown (usually when no sessions connected) © 2009/2010 Pythian
  55. 55. GOODNESS and CLB_GOAL • LONG • GOODNESS=number of connected sessions • DELTA=1 • SHORT • Calculated based on • GOAL_SERVICE_TIME • GOAL_THROUGHPUT • GOAL_NONE © 2009/2010 Pythian
  56. 56. GOODNESS and CLB_GOAL_SHORT • According to my observations • GOAL_SERVICE_TIME • “per call” metrics used • CPUPERCALL • DBTIMEPERCALL • GOAL_THROUGHPUT • “per second” metrics used • CPUPERSEC • DBTIMEPERSEC © 2009/2010 Pythian
  57. 57. Listener trace… again • We can see goodness and delta when instance/service started nsglgrDoRegister: Creating new service: service10g nsglgrDoRegister: service:service10g flag:2 goodness:0 delta:0 • During service updates nsglgrDoRegister: service:service10g what:4 value:1 nsglgrDoRegister: service:service10g what:2 value:47 • what: 2 = goodness • what: 4 = delta © 2009/2010 Pythian
  58. 58. How does listener know about it? • MMON is the “calculator” • V$SERVICEMETRIC • PMON is the “messenger” • 10257 trace name context forever, level 16 © 2009/2010 Pythian
  59. 59. PMON trace example • 11g trace *** 2007-09-19 05:52:24.797 kmmlrl: update for session drop delta: 111 106 10 10 82 kmmlrl: service11g.oracloid.com goodness 8 kmmlrl: update for service goodness kmmlrl: 55 processes kmmlrl: node load 358 kmmlrl: instance load 19 kmmlrl: nsgr update returned 0 © 2009/2010 Pythian
  60. 60. Service Metrics & CLB • CLB_LONG Target - equal number of sessions per instance GOODNESS=number of session, DELTA=1 • CLB_SHORT + GOAL_NONE Target - equalize CPU utilization GOODNESS and DELTA => based on run-queue / load average • CLB_SHORT + GOAL_SERVICE_TIME Target - minimize response time GOODNESS + DELTA <= LBA _PER_CALL (?) • CLB_SHORT + GOAL_THROUGHPUT Target - maximize throughput GOODNESS + DELTA <= LBA _PER_SEC (?) © 2009/2010 Pythian
  61. 61. Nodes with different capacity • Works with instance load • use different SESSIONS in init.ora • Works with node load • based on CPU_COUNT • Service Metrics? • bug 6613950 • CLB_GOAL_SHORT is slower than node load (my observations) • CLB faster than RLB 45 © 2009/2010 Pythian
  62. 62. Where we are... !Client-side basics !Oracle Cluster Services !Server-side CLB !FAN - HA & LBA events • Thin JDBC RLB • ONS Subscriber • Your own LBA! © 2009/2010 Pythian
  63. 63. Clusterware processes © 2009/2010 Pythian Diagram from Barb Lundhild’s presentation
  64. 64. ONS Configuration • $ORA_CRS_HOME/opmn/conf/ons.config localport=6101 #10g #remoteport=6201 loglevel=9 useocr=on • $ORA_CRS_HOME/bin/onsctl start/stop/ping • $ORA_CRS_HOME/bin/racgons add_config/remove_config 48 © 2009/2010 Pythian
  65. 65. Demo 6 - LBA events • ONS log eventType: database/event/servicemetrics/service10g VERSION=1.0 database=g4 { {instance=g42 percent=34 flag=GOOD} {instance=g43 percent=33 flag=GOOD} {instance=g41 percent=33 flag=GOOD} } timestamp=2008-02-11 03:24:18 • SYS.SYS$SERVICE_METRICS_TAB 49 © 2009/2010 Pythian
  66. 66. Using FAN events on app-tier • ONC – Oracle Notification Client • Java ONS API •C ONS API • Canbe used to manage non-pooled connections or home- grown pools • Can be standalone ONC – no JDBC/OCI •– – – local ONS is required (at least with 10g) © 2009/2010 Pythian
  67. 67. Where we are... !Client-side basics !Oracle Cluster Services !Server-side CLB !FAN - HA & LBA events !Thin JDBC RLB • ONS Subscriber • Your own LBA! © 2009/2010 Pythian
  68. 68. Establishing a Connection Java Connection Pool Instance Instance Instance 52 © 2009/2010 Pythian
  69. 69. Establishing a Connection Java Connection Pool Client-side connection load balancing (if not using 11gR2 SCAN listener) Instance Instance Instance 52 © 2009/2010 Pythian
  70. 70. Establishing a Connection Java Connection Pool Instance Instance Instance 53 © 2009/2010 Pythian
  71. 71. Establishing a Connection Java Connection Pool SCAN Listener Instance Instance Instance 53 © 2009/2010 Pythian
  72. 72. Establishing a Connection Java Connection Pool 11gR2 -> hando" to SCAN listener (optionally) SCAN Listener Instance Instance Instance 53 © 2009/2010 Pythian
  73. 73. Establishing a Connection Java Connection Pool SCAN listener is not doing CLB instead of the client itself SCAN Listener Instance Instance Instance 53 © 2009/2010 Pythian
  74. 74. Establishing a Connection Java Connection Pool SCAN Listener Instance Instance Instance Server-side CLB by listeners 53 © 2009/2010 Pythian
  75. 75. RLB - connection gravitation Connection Pool (90 sessions) 54 © 2009/2010 Pythian
  76. 76. RLB - connection gravitation Establishing connections Connection Pool (90 sessions) 54 © 2009/2010 Pythian
  77. 77. RLB - connection gravitation Establishing connections Connection Pool (90 sessions) 30 30 30 instance 1 instance 2 instance 3 54 © 2009/2010 Pythian
  78. 78. RLB - connection gravitation Connection Pool (90 sessions) 30 30 30 LBA event 33% 33% 34% instance 1 instance 2 instance 3 54 © 2009/2010 Pythian
  79. 79. RLB - connection gravitation Connection Pool (90 sessions) 30 30 30 20 20 20 33% 33% 34% DB tra#c instance 1 instance 2 instance 3 54 © 2009/2010 Pythian
  80. 80. RLB - connection gravitation Connection Pool (90 sessions) 30 30 30 20 20 20 33% 33% 34% instance 1 instance 2 instance 3 54 © 2009/2010 Pythian
  81. 81. RLB - connection gravitation Connection Pool (90 sessions) 50% 25% 25% instance 1 instance 2 instance 3 55 © 2009/2010 Pythian
  82. 82. RLB - connection gravitation Connection Pool (90 sessions) 80% 10% 10% instance 1 instance 2 instance 3 56 © 2009/2010 Pythian
  83. 83. RLB - connection gravitation Connection Pool (90 sessions) ? 80% 10% 10% instance 1 instance 2 instance 3 56 © 2009/2010 Pythian
  84. 84. RLB - connection gravitation Connection Pool (90 sessions) ? 80% 10% 10% instance 1 instance 2 instance 3 Connection pool must be over-allocated! 56 © 2009/2010 Pythian
  85. 85. RLB - how is workload gravitated? 12.0 INST1 INST2 INST3 25% Cumulative connections given 9.0 25% 6.0 3.0 50% 0 Time 57 © 2009/2010 Pythian
  86. 86. CLB options for RLB CLB_SHORT Connection Pool instance 1 instance 2 instance 3 58 © 2009/2010 Pythian
  87. 87. CLB options for RLB CLB_SHORT Establishing connections Connection Pool instance 1 instance 2 instance 3 58 © 2009/2010 Pythian
  88. 88. CLB options for RLB CLB_SHORT Establishing connections Connection Pool instance 1 instance 2 instance 3 Connections requests burst - most go to the node with the lowest goodness 58 © 2009/2010 Pythian
  89. 89. CLB options for RLB CLB_SHORT Establishing connections Connection Pool instance 1 instance 2 instance 3 Connections requests burst - most go to the node with the lowest goodness 58 © 2009/2010 Pythian
  90. 90. CLB options for RLB CLB_LONG Connection Pool instance 1 instance 2 instance 3 Connections requests distributed equally across all instances running a service 59 © 2009/2010 Pythian
  91. 91. CLB options for RLB CLB_LONG Connection Pool instance 1 instance 2 instance 3 Connections requests distributed equally across all instances running a service 59 © 2009/2010 Pythian
  92. 92. Rule of thumb use CLB_LONG with connection pools 60 © 2009/2010 Pythian
  93. 93. JDBC Implicit Connection Cache • Standard JDBC drivers + ons.jar ods.setConnectionCachingEnabled(True); ods.setFastConnectionFailoverEnabled(True); ods.setONSConfiguration ("nodes=lh1:6201,lh2:6201"); Properties prop = new Properties(); prop.setProperty("MinLimit", MIN_CONN); prop.setProperty("MaxLimit", MAX_CONN); prop.setProperty("InitialLimit", INIT_CONN); prop.put (oracle.net.ns.SQLnetDef.TCP_CONNTIMEOUT_STR, "1000")); // 1 second ods.setConnectionCacheProperties(prop); © 2009/2010 Pythian
  94. 94. Demo 7 - JDBC RLB • ImplicitConnection Cache • Enabling FCF • Subscribe with ONS • Tracing LBA events in JDBC 62 © 2009/2010 Pythian
  95. 95. ONS JDBC bugs • Bug 6151350 - RESTRICTION ON SETONSCONFIGURATION • Bug 6315760 - SETONSCONFIGURATION ONLY REGISTERS A MAXIMUM OF THREE NODES • -Doracle.ons.maxconnections=5 63 © 2009/2010 Pythian
  96. 96. Are we there yet? !Client-sidebasics !Oracle Cluster Services !Server-side CLB !FAN - HA & LBA events !Thin JDBC RLB !ONS Subscriber • Your own LBA! © 2009/2010 Pythian
  97. 97. Demo 8 - ONC JDBC API • LocalONS is required on client • Subscribing to events • How to filter only required events • Parsing events 65 © 2009/2010 Pythian
  98. 98. Are we there YET??? !Client-sidebasics !Oracle Cluster Services !Server-side CLB !FAN - HA & LBA events !Thin JDBC RLB !ONS Subscriber !Your own LBA! © 2009/2010 Pythian
  99. 99. Demo 9 - your own LBA • Maximum flexibility of LBA • ONS Publisher • Oracle doesn’t know how to balance you workload better than you! • This is not documented anywhere and likely unsupported 67 © 2009/2010 Pythian
  100. 100. Are we there yet? NO :) !Oracle Cluster Services !Establishing a connection !FAN - HA & LBA events ! !Thin JDBC RLB !ONS Subscriber !Your own LBA! © 2009/2010 Pythian
  101. 101. 11g New Features • SCAN Listener • Universal Connection Pool • Connection Affinity in LBA (web sessions, XA) • Reduces Cache Fusion overhead • Requires UCP • Data Guard Broker integration with FAN events • Policy-Based Cluster and Capacity Management 69 © 2009/2010 Pythian
  102. 102. ye t? t he re A re we © 2009/2010 Pythian
  103. 103. Bonus - WebLogic Server Connection Management 71 © 2009/2010 Pythian
  104. 104. Source - WebLogic Server 10.3.1 documentation 72 © 2009/2010 Pythian
  105. 105. WebLogic Configurations Options for Oracle RAC •4 Supported Data Source Configurations MDS MDS RAC Oracle XA no-XA Services native Load balancing Y Y Y N Failover Y Y Y Y XA support Y N Y N JDBC Store N Y Y Y RAC Services N N Y Y 73 © 2009/2010 Pythian
  106. 106. WebLogic Multi Data Source with Oracle RAC WebLogic Server Multi Data Source • App uses MDS Data Data Data • MDS routes connection Source 1 Source 2 Source 3 to one of its DS • failover or load balancing • DS online management • Suspend Instance 1 Instance 2 Instance 3 • Shutdown • Add • Remove Database 74 © 2009/2010 Pythian
  107. 107. WebLogic MDS for Failover with Oracle RAC WebLogic Server • Failover MDS Multi Data Source • DS health checks Data Data Data Source 1 Source 2 Source 3 • failure => DS dead • Can manually disable DS • Dead DS are re-tested • default 120 seconds Instance 1 Instance 2 Instance 3 Database 75 © 2009/2010 Pythian
  108. 108. WebLogic MDS for Failover with Oracle RAC WebLogic Server • Failover MDS Multi Data Source • DS health checks Data Data Data Source 1 Source 2 Source 3 • failure => DS dead • Can manually disable DS • Dead DS are re-tested • default 120 seconds Instance 1 Instance 2 Instance 3 Database 75 © 2009/2010 Pythian
  109. 109. WebLogic MDS for Failover with Oracle RAC WebLogic Server • Failover MDS Multi Data Source • DS health checks Data Data Data Source 1 Source 2 Source 3 • failure => DS dead • Can manually disable DS • Dead DS are re-tested • default 120 seconds Instance 1 Instance 2 Instance 3 Database 75 © 2009/2010 Pythian
  110. 110. WebLogic MDS for Failover with Oracle RAC WebLogic Server • Failover MDS Multi Data Source • DS health checks Data Data Data Source 1 Source 2 Source 3 • failure => DS dead • Can manually disable DS • Dead DS are re-tested • default 120 seconds Instance 1 Instance 2 Instance 3 Database 75 © 2009/2010 Pythian
  111. 111. WebLogic MDS for Failover with Oracle RAC WebLogic Server • Failover MDS Multi Data Source • DS health checks Data Data Data Source 1 Source 2 Source 3 • failure => DS dead • Can manually disable DS • Dead DS are re-tested • default 120 seconds Instance 1 Instance 2 Instance 3 Database 75 © 2009/2010 Pythian
  112. 112. WebLogic MDS for Failover with Oracle RAC WebLogic Server • Failover MDS Multi Data Source • DS health checks Data Data Data Source 1 Source 2 Source 3 • failure => DS dead • Can manually disable DS • Dead DS are re-tested • default 120 seconds Instance 1 Instance 2 Instance 3 Database 75 © 2009/2010 Pythian
  113. 113. WebLogic MDS for Failover with Oracle RAC WebLogic Server • Failover MDS Multi Data Source • DS health checks Data Data Data Source 1 Source 2 Source 3 • failure => DS dead • Can manually disable DS • Dead DS are re-tested • default 120 seconds Instance 1 Instance 2 Instance 3 Database 75 © 2009/2010 Pythian
  114. 114. WebLogic MDS for Failover with Oracle RAC WebLogic Server • Failover MDS Multi Data Source • DS health checks Data Data Data Source 1 Source 2 Source 3 • failure => DS dead • Can manually disable DS • Dead DS are re-tested • default 120 seconds Instance 1 Instance 2 Instance 3 Database 75 © 2009/2010 Pythian
  115. 115. 76 © 2009/2010 Pythian
  116. 116. WebLogic Multi Data Source with Oracle RAC WebLogic Server Multi Data Source • Load Balancing Data Data Data • MDS routes connection Source 1 Source 2 Source 3 to all active DS • round-robin • failover is supported • no performance base load Instance 1 Instance 2 Instance 3 balancing like Oracle RLB • DS size doesn’t affect LB Database 77 © 2009/2010 Pythian
  117. 117. Oracle RAC: JDBC Pool vs WebLogic Multi DS Oracle JDBC WLS Multi-DS Connect failover connect time MDS failover failover Connection load balancing JDBC CLB none Server-side CLB based on Listener none service metrics / performance Control connections layout RAC Services MDS config / members Reconnect on failure FCF MDS managed Run-time load balancing based RLB / LBA MDS LB on service performance round-robin Use new (or recovered) instance not until automated reconnect by MDS 78 © 2009/2010 Pythian
  118. 118. WebLogic Connections to RAC Services WebLogic Server • Initial Capacity = 0 Multi Data Source Data Data Data • DS must exist for every node Source 1 Source 2 Source 3 that can run the service • Services layout change requires WLS reconfiguration • XA supported MY_SRV MY_SRV • branches of the same XA Instance 1 Instance 2 Instance 3 transactions are routed to the same instance • MDS for load balancing • Failover makes little sense Database 79 © 2009/2010 Pythian
  119. 119. WebLogic Connections to RAC Services WebLogic Server Multi Data Source Data Data Data Source 1 Source 2 Source 3 BOB_SRV Instance 1 Instance 2 Instance 3 Database 80 © 2009/2010 Pythian
  120. 120. WebLogic Unmanaged DS with Oracle RAC WebLogic Server Unmanaged Data Source • JDBC native CLB Oracle JDBC “unmanaged” pool • DB Server - CLB_GOAL_LONG • Failover - FCF • Based on HA events • RLB does not work Instance 1 Instance 2 Instance 3 • Connection testing not required Database 81 © 2009/2010 Pythian
  121. 121. What’s next? • Book draw • Leave me your business cards Q&A Email me - gorbachev@pythian.com Read my blog - http://www.pythian.com Follow me on Twitter - @AlexGorbachev Join Pythian fan club on Facebook & LinkedIn 82 © 2009/2010 Pythian

×