Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Demystifying Oracle RAC Workload Management by Alex Gorbachev, Pythian | NoCOUG 2010 Spring Conference

7,092 views

Published on

This presentation starts with a brief overview of connection management in Oracle RAC and then focuses on workload management, including exploiting cluster services, connection load balancing, and run-time workload balancing. Finally, it shows how to create your own run-time workload management, using the standard load balancing advisory framework. The presentation goes through demonstrations using a live Oracle RAC cluster, illustrating the internals of workload management and how to monitor it. The live demos simplify understanding of this rather complex topic.

Published in: Technology
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Demystifying Oracle RAC Workload Management by Alex Gorbachev, Pythian | NoCOUG 2010 Spring Conference

  1. 1. Demystifying Oracle RAC Workload Management North California Oracle User Group 20-May-2010 Alex Gorbachev
  2. 2. Alex Gorbachev today... • CTO, The Pythian Group • Blogger • OakTable Network member • Oracle ACE Director • BattleAgainstAnyGuess.com • Vice-president, Oracle RAC SIG 2 © 2009/2010 Pythian
  3. 3. Why Companies Trust Pythian • Recognized Leader: • Global industry-leader in remote database administration services and consulting for Oracle, Oracle Applications, MySQL and SQL Server • Work with over 150 multinational companies such as Forbes.com, Fox Interactive media, and MDS Inc. to help manage their complex IT deployments • Expertise: • One of the world’s largest concentrations of dedicated, full-time DBA expertise. • Global Reach & Scalability: • 24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response 3 © 2009/2010 Pythian
  4. 4. Where we are... !Client-side basics • Oracle Cluster Services • Server-side CLB • FAN - HA & LBA events • Thin JDBC RLB • ONS Subscriber • Your own LBA! © 2009/2010 Pythian
  5. 5. Simple example • Establishingone connection • Dedicated server • Several RAC nodes © 2009/2010 Pythian
  6. 6. Client-side CLB (DESCRIPTION= (ADDRESS_LIST= (LOAD_BALANCE=ON) (FAILOVER=ON) (ADDRESS=(PROTOCOL=TCP)(HOST=lh1-vip)(PORT=1521)) (ADDRESS=(PROTOCOL=TCP)(HOST=lh2-vip)(PORT=1521)) (ADDRESS=(PROTOCOL=TCP)(HOST=lh3-vip)(PORT=1521)) (ADDRESS=(PROTOCOL=TCP)(HOST=lh4-vip)(PORT=1521)) ) (CONNECT_DATA=(SERVICE_NAME=service10g)) ) 6 © 2009/2010 Pythian
  7. 7. Client-side connection balancing and connection-time failover lh1-vip lh2-vip lh3-vip lh4-vip LISTENER_G41 LISTENER_G42 LISTENER_G43 LISTENER_G44 lh1 lh2 lh3 lh4 © 2009/2010 Pythian
  8. 8. Client-side connection balancing and connection-time failover lh1-vip lh2-vip lh3-vip lh4-vip LISTENER_G41 LISTENER_G42 LISTENER_G43 LISTENER_G44 lh1 lh2 lh3 lh4 © 2009/2010 Pythian
  9. 9. Client-side connection balancing and connection-time failover LOAD_BALANCE=OFF lh1-vip FAILOVER=OFF lh2-vip lh3-vip lh4-vip LISTENER_G41 LISTENER_G42 LISTENER_G43 LISTENER_G44 lh1 lh2 lh3 lh4 © 2009/2010 Pythian
  10. 10. Client-side connection balancing and connection-time failover LOAD_BALANCE=OFF lh1-vip FAILOVER=OFF lh2-vip lh3-vip lh4-vip LISTENER_G41 LISTENER_G42 LISTENER_G43 LISTENER_G44 lh1 lh2 lh3 lh4 © 2009/2010 Pythian
  11. 11. Client-side connection balancing and connection-time failover lh1-vip lh2-vip lh3-vip lh4-vip LOAD_BALANCE=OFF FAILOVER=ON LISTENER_G41 LISTENER_G42 LISTENER_G43 LISTENER_G44 lh1 lh2 lh3 lh4 © 2009/2010 Pythian
  12. 12. Client-side connection balancing and connection-time failover lh1-vip lh2-vip lh3-vip lh4-vip LOAD_BALANCE=OFF FAILOVER=ON LISTENER_G41 LISTENER_G42 LISTENER_G43 LISTENER_G44 lh1 lh2 lh3 lh4 © 2009/2010 Pythian
  13. 13. Client-side connection balancing and connection-time failover lh1-vip lh2-vip lh3-vip lh4-vip LOAD_BALANCE=ON FAILOVER=ON LISTENER_G41 LISTENER_G42 LISTENER_G43 LISTENER_G44 lh1 lh2 lh3 lh4 © 2009/2010 Pythian
  14. 14. Client-side connection balancing and connection-time failover lh1-vip lh2-vip lh3-vip lh4-vip LOAD_BALANCE=ON FAILOVER=ON LISTENER_G41 LISTENER_G42 LISTENER_G43 LISTENER_G44 lh1 lh2 lh3 lh4 © 2009/2010 Pythian
  15. 15. Summary – client-side business • Client chooses between several addresses • Why • Failure tolerance • Distribution across several listeners • How • Several connection points in connection descriptor • LOAD_BALANCE=ON • FAILOVER=ON © 2009/2010 Pythian
  16. 16. Listener – where to direct request? lsnrctl status …… Service "service10g" has 2 instance(s). Instance "g41", status READY, has 1 handler(s) for this service... Instance "g42", status READY, has 1 handler(s) for this service... …… © 2009/2010 Pythian
  17. 17. Where we are... !Client-side basics !Oracle Cluster Services • Server-side CLB • FAN - HA & LBA events • Thin JDBC RLB • ONS Subscriber • Your own LBA! © 2009/2010 Pythian
  18. 18. Orders processing example • Business areas • Customers’ order (web) • Content display (web cached on app-tier) • Feedback (web) • Orders fulfillment (back-office) • Data extraction batches 11 © 2009/2010 Pythian
  19. 19. Services: preferred & available NEW_ORD NEW_ORD NEW_ORD NEW_ORD CONTENT CONTENT CONTENT CONTENT FEEDBACK FEEDBACK FEEDBACK FEEDBACK PROC_ORD PROC_ORD BATCH BATCH DB1 DB2 DB3 DB4 12 © 2009/2010 Pythian
  20. 20. Node affinity NEW_ORD NEW_ORD CONTENT CONTENT FEEDBACK FEEDBACK PROC_ORD PROC_ORD BATCH DB1 DB2 DB3 DB4 13 © 2009/2010 Pythian
  21. 21. Service failover NEW_ORD NEW_ORD CONTENT CONTENT FEEDBACK FEEDBACK PROC_ORD PROC_ORD BATCH DB1 DB2 DB3 DB4 14 © 2009/2010 Pythian
  22. 22. Service failover NEW_ORD NEW_ORD CONTENT CONTENT FEEDBACK FEEDBACK PROC_ORD PROC_ORD BATCH DB1 DB2 DB3 DB4 14 © 2009/2010 Pythian
  23. 23. Service failover NEW_ORD NEW_ORD CONTENT CONTENT FEEDBACK FEEDBACK PROC_ORD PROC_ORD BATCH DB1 DB2 DB3 DB4 14 © 2009/2010 Pythian
  24. 24. Service failover NEW_ORD NEW_ORD CONTENT CONTENT FEEDBACK FEEDBACK PROC_ORD PROC_ORD BATCH DB1 DB2 DB3 DB4 14 © 2009/2010 Pythian
  25. 25. Service failover NEW_ORD NEW_ORD CONTENT CONTENT FEEDBACK FEEDBACK PROC_ORD PROC_ORD BATCH DB1 DB2 DB3 DB4 14 © 2009/2010 Pythian
  26. 26. Demo 1 - Services Automation • HA events introduction • Logging example • Using server-side FAN HA events callouts 15 © 2009/2010 Pythian
  27. 27. Server-side FAN HA events • Server-side callout can be used for: • Relocating services back to preferred nodes • Smart services handling • Notify and move sessions after service Down • Rebalance sessions on service up • Accommodate workload re-balancing for legacy applications 16 © 2009/2010 Pythian
  28. 28. Services & Resource Manager DBMS_RESOURCE_MANAGER.SET_CONSUMER_GROUP_MAPPING (DBMS_RESOURCE_MANAGER.SERVICE_NAME, 'NEW_ORD', 'NEW_ORD_GROUP'); DBMS_RESOURCE_MANAGER.SET_CONSUMER_GROUP_MAPPING (DBMS_RESOURCE_MANAGER.SERVICE_NAME, 'CONTENT', 'CONTENT_GROUP'); ... • Works on instance level - not RAC-aware • Doesn’t account for >1 instances per node 17 © 2009/2010 Pythian
  29. 29. Services abstraction for apps • Applicationsdon’t need to know current services configuration! • Listeners know services status • Listeners control which instance gets a new connection 18 © 2009/2010 Pythian
  30. 30. Where we are... !Client-side basics !Oracle Cluster Services !Server-side CLB • FAN - HA & LBA events • Thin JDBC RLB • ONS Subscriber • Your own LBA! © 2009/2010 Pythian
  31. 31. Remote listener registration LISTENER_G41 LISTENER_G41 G41 G42 lh1 lh2 © 2009/2010 Pythian
  32. 32. Remote listener registration LISTENER_G41 LISTENER_G41 Local registration G41.local_listener=LISTENER_G41 G42.local_listener=LISTENER_G42 G43.local_listener=LISTENER_G43 G44.local_listener=LISTENER_G44 G41 G42 lh1 lh2 © 2009/2010 Pythian
  33. 33. Remote listener registration LISTENER_G41 LISTENER_G41 Local registration G41.local_listener=LISTENER_G41 G42.local_listener=LISTENER_G42 G43.local_listener=LISTENER_G43 G44.local_listener=LISTENER_G44 G41 G42 Remote registration *.remote_listener=LISTENERS_G4 lh1 lh2 © 2009/2010 Pythian
  34. 34. Correct listener configuration (google for Pythian video VIP) lsnrctl service Service "service10g.oracloid.com" has 2 instance(s). Instance "g41", status READY, has 1 handler(s) for this service... Handler(s): "DEDICATED" established:0 refused:0 state:ready REMOTE SERVER (ADDRESS=(PROTOCOL=TCP)(HOST=lh1-vip)(PORT=1521)) "DEDICATED" established:0 refused:0 state:ready LOCAL SERVER Instance "g42", status READY, has 1 handler(s) for this service... Handler(s): "DEDICATED" established:0 refused:0 state:ready REMOTE SERVER (ADDRESS=(PROTOCOL=TCP)(HOST=lh2-vip)(PORT=1521)) Instance "g43", status READY, has 1 handler(s) for this service... Handler(s): "DEDICATED" established:0 refused:0 state:ready REMOTE SERVER (ADDRESS=(PROTOCOL=TCP)(HOST=lh3-vip)(PORT=1521)) Instance "g44", status READY, has 1 handler(s) for this ... © 2009/2010 Pythian
  35. 35. Server-side CLB LISTENER_G44 G41 G42 G43 G44 22 © 2009/2010 Pythian
  36. 36. Server-side CLB LISTENER_G44 Which instance? G41 G42 G43 G44 22 © 2009/2010 Pythian
  37. 37. Server-side CLB LISTENER_G44 Which instance? • <= 10gR1 • Instance load • Node load G41 G42 G43 G44 22 © 2009/2010 Pythian
  38. 38. Server-side CLB LISTENER_G44 Which instance? • <= 10gR1 • Instance load • Node load • 10gR2 G41 G42 G43 G44 • Service Metrics • AWR 22 © 2009/2010 Pythian
  39. 39. Demo helper Tracing listener lsnrctl trace user nsglgrDoRegister: inst loads: ld1:145 mld1:5120 ld2:5 mld2:170 nsglgrDoRegister: inst loads: ld1:58 mld1:5120 ld2:4 mld2:170 • No notion of instance name in trace ! • ld1 – Load Data 1, mld1 – Max Load Data 1 • ld2 – Load Data 2, mld2 – Max Load Data 2 • Load data is instance load and node load © 2009/2010 Pythian
  40. 40. Demo helper PREFER_LEAST_LOADED_NODE • listener.ora parameter • PREFER_LEAST_LOADED_NODE_% • ON (default) • ld1 => node, ld2 = > instance • OFF • ld1 => instance, ld1 => node © 2009/2010 Pythian
  41. 41. Demo helper Instance and node loads • Instance load • Node load • Number of user sessions • Index based probably • TYPE<>BACKGROUND on load average • Max • Max node load instance load • Max • Max CPU capacity number of sessions • cpu_count*5K • SESSIONS init.ora © 2009/2010 Pythian
  42. 42. Demo helper Uneven connection distribution • PREFER_LEAST_LOADED_NODE_...=ON • Default • Listeneruses CPU load to balance connections to the machine with more CPU capacity left • Can work fine with differently sized nodes • But what if connections are “permanent”? © 2009/2010 Pythian
  43. 43. Demo helper “Even” connection distribution PREFER_LEAST_LOADED_NODE_LISTENER_G41=OFF PREFER_LEAST_LOADED_NODE_LISTENER_G42=OFF • Loadbased on number of sessions left until maximum is reaches • init.ora parameter SESSIONS • Can configure SESSIONS differently for every node and influence distribution © 2009/2010 Pythian
  44. 44. Demo 3 - 9i style CLB • listener trace • PREFER_LEAST_LOADED_NODE • instance load • node load • PMON trace 10257 event 28 © 2009/2010 Pythian
  45. 45. 9i and 10gR1 CLB summary • PMON reports to listener • instance load • node load • PREFER_LEAST_LOADED_NODE • ON - node load (default) • OFF - instance load • lsnrctl trace user • PMON: 10257 trace name context forever, level 16 29 © 2009/2010 Pythian
  46. 46. DB_DOMAIN and bug 5593693 • db_domain init.ora parameter is set • service_name is not FQDN • that’s what srvctl does • Listenerappends default domain • Service metrics updates are lost • on the way from PMON => listener • Result: fallback to instance / node load • Fixed - 10.2.0.4 & 11.1.0.6 30 © 2009/2010 Pythian
  47. 47. 10gR2 and 11g server-side CLB • Service metrics for each service • GOODNESS • Attractiveness of the instance for this service • Higher is worse (should be BADNESS) • DELTA • Estimate of “badness” increase for additional session • FLAGS (bit map) • 0 – all good • 1 – blocked • 2 – crossed threshold • 4 – goodness unknown (usually when no sessions connected) 31 © 2009/2010 Pythian
  48. 48. Connection life-time • Short living sessions • Persistent / Oracle Forms / long connections • Connection pool • Oracle standard connection pool • Custom connection pool • Application server connection pool © 2009/2010 Pythian
  49. 49. Connection classification & CLB • Long connections • idle most of the time (!) • persistent connections (non-pooled) • over-allocated connection pools • Short connections • active most of the time • dynamic connection pools • Batch-like connection • long but always active • under-allocated connection pools (!) 33 © 2009/2010 Pythian
  50. 50. CLB goal • CLB_GOAL_LONG • equal distributions of sessions • CLB_GOAL_SHORT • based on Service Metrics and LBA goal (!) • GOAL_NONE • GOAL_SERVICE_TIME • GOAL_THROUGHPUT 34 © 2009/2010 Pythian
  51. 51. Demo 4 - Service Metrics & CLB • CLB_GOAL_LONG • CLB_GOAL_SHORT • listener trace • PMON trace • V$SERVICEMETRIC 35 © 2009/2010 Pythian
  52. 52. 10g services load balancing • For each service defined: • Connection load balancing goal • CLB_GOAL_SHORT • CLB_GOAL_LONG • Load Balancing Advisory - LBA • GOAL_SERVICE_TIME • GOAL_THROUGHPUT © 2009/2010 Pythian
  53. 53. V$SERVICEMETRIC • For each service and instance contains • Short interval (5 seconds) • Long interval (60 seconds) • INTSIZE_CSEC • CPUPERCALL • DBTIMEPERCALL • CPUPERSEC • DBTIMEPERSEC © 2009/2010 Pythian
  54. 54. V$SERVICEMETRIC (cont’d) • GOODNESS • Attractiveness of the instance for this service • Higher is worse (should be BADNESS) • DELTA • Estimate of “badness” increase for additional session • FLAGS (bit map) • 0 – all good • 1 – blocked • 2 – crossed threshold • 4 – goodness unknown (usually when no sessions connected) © 2009/2010 Pythian
  55. 55. GOODNESS and CLB_GOAL • LONG • GOODNESS=number of connected sessions • DELTA=1 • SHORT • Calculated based on • GOAL_SERVICE_TIME • GOAL_THROUGHPUT • GOAL_NONE © 2009/2010 Pythian
  56. 56. GOODNESS and CLB_GOAL_SHORT • According to my observations • GOAL_SERVICE_TIME • “per call” metrics used • CPUPERCALL • DBTIMEPERCALL • GOAL_THROUGHPUT • “per second” metrics used • CPUPERSEC • DBTIMEPERSEC © 2009/2010 Pythian
  57. 57. Listener trace… again • We can see goodness and delta when instance/service started nsglgrDoRegister: Creating new service: service10g nsglgrDoRegister: service:service10g flag:2 goodness:0 delta:0 • During service updates nsglgrDoRegister: service:service10g what:4 value:1 nsglgrDoRegister: service:service10g what:2 value:47 • what: 2 = goodness • what: 4 = delta © 2009/2010 Pythian
  58. 58. How does listener know about it? • MMON is the “calculator” • V$SERVICEMETRIC • PMON is the “messenger” • 10257 trace name context forever, level 16 © 2009/2010 Pythian
  59. 59. PMON trace example • 11g trace *** 2007-09-19 05:52:24.797 kmmlrl: update for session drop delta: 111 106 10 10 82 kmmlrl: service11g.oracloid.com goodness 8 kmmlrl: update for service goodness kmmlrl: 55 processes kmmlrl: node load 358 kmmlrl: instance load 19 kmmlrl: nsgr update returned 0 © 2009/2010 Pythian
  60. 60. Service Metrics & CLB • CLB_LONG Target - equal number of sessions per instance GOODNESS=number of session, DELTA=1 • CLB_SHORT + GOAL_NONE Target - equalize CPU utilization GOODNESS and DELTA => based on run-queue / load average • CLB_SHORT + GOAL_SERVICE_TIME Target - minimize response time GOODNESS + DELTA <= LBA _PER_CALL (?) • CLB_SHORT + GOAL_THROUGHPUT Target - maximize throughput GOODNESS + DELTA <= LBA _PER_SEC (?) © 2009/2010 Pythian
  61. 61. Nodes with different capacity • Works with instance load • use different SESSIONS in init.ora • Works with node load • based on CPU_COUNT • Service Metrics? • bug 6613950 • CLB_GOAL_SHORT is slower than node load (my observations) • CLB faster than RLB 45 © 2009/2010 Pythian
  62. 62. Where we are... !Client-side basics !Oracle Cluster Services !Server-side CLB !FAN - HA & LBA events • Thin JDBC RLB • ONS Subscriber • Your own LBA! © 2009/2010 Pythian
  63. 63. Clusterware processes © 2009/2010 Pythian Diagram from Barb Lundhild’s presentation
  64. 64. ONS Configuration • $ORA_CRS_HOME/opmn/conf/ons.config localport=6101 #10g #remoteport=6201 loglevel=9 useocr=on • $ORA_CRS_HOME/bin/onsctl start/stop/ping • $ORA_CRS_HOME/bin/racgons add_config/remove_config 48 © 2009/2010 Pythian
  65. 65. Demo 6 - LBA events • ONS log eventType: database/event/servicemetrics/service10g VERSION=1.0 database=g4 { {instance=g42 percent=34 flag=GOOD} {instance=g43 percent=33 flag=GOOD} {instance=g41 percent=33 flag=GOOD} } timestamp=2008-02-11 03:24:18 • SYS.SYS$SERVICE_METRICS_TAB 49 © 2009/2010 Pythian
  66. 66. Using FAN events on app-tier • ONC – Oracle Notification Client • Java ONS API •C ONS API • Canbe used to manage non-pooled connections or home- grown pools • Can be standalone ONC – no JDBC/OCI •– – – local ONS is required (at least with 10g) © 2009/2010 Pythian
  67. 67. Where we are... !Client-side basics !Oracle Cluster Services !Server-side CLB !FAN - HA & LBA events !Thin JDBC RLB • ONS Subscriber • Your own LBA! © 2009/2010 Pythian
  68. 68. Establishing a Connection Java Connection Pool Instance Instance Instance 52 © 2009/2010 Pythian
  69. 69. Establishing a Connection Java Connection Pool Client-side connection load balancing (if not using 11gR2 SCAN listener) Instance Instance Instance 52 © 2009/2010 Pythian
  70. 70. Establishing a Connection Java Connection Pool Instance Instance Instance 53 © 2009/2010 Pythian
  71. 71. Establishing a Connection Java Connection Pool SCAN Listener Instance Instance Instance 53 © 2009/2010 Pythian
  72. 72. Establishing a Connection Java Connection Pool 11gR2 -> hando" to SCAN listener (optionally) SCAN Listener Instance Instance Instance 53 © 2009/2010 Pythian
  73. 73. Establishing a Connection Java Connection Pool SCAN listener is not doing CLB instead of the client itself SCAN Listener Instance Instance Instance 53 © 2009/2010 Pythian
  74. 74. Establishing a Connection Java Connection Pool SCAN Listener Instance Instance Instance Server-side CLB by listeners 53 © 2009/2010 Pythian
  75. 75. RLB - connection gravitation Connection Pool (90 sessions) 54 © 2009/2010 Pythian
  76. 76. RLB - connection gravitation Establishing connections Connection Pool (90 sessions) 54 © 2009/2010 Pythian
  77. 77. RLB - connection gravitation Establishing connections Connection Pool (90 sessions) 30 30 30 instance 1 instance 2 instance 3 54 © 2009/2010 Pythian
  78. 78. RLB - connection gravitation Connection Pool (90 sessions) 30 30 30 LBA event 33% 33% 34% instance 1 instance 2 instance 3 54 © 2009/2010 Pythian
  79. 79. RLB - connection gravitation Connection Pool (90 sessions) 30 30 30 20 20 20 33% 33% 34% DB tra#c instance 1 instance 2 instance 3 54 © 2009/2010 Pythian
  80. 80. RLB - connection gravitation Connection Pool (90 sessions) 30 30 30 20 20 20 33% 33% 34% instance 1 instance 2 instance 3 54 © 2009/2010 Pythian
  81. 81. RLB - connection gravitation Connection Pool (90 sessions) 50% 25% 25% instance 1 instance 2 instance 3 55 © 2009/2010 Pythian
  82. 82. RLB - connection gravitation Connection Pool (90 sessions) 80% 10% 10% instance 1 instance 2 instance 3 56 © 2009/2010 Pythian
  83. 83. RLB - connection gravitation Connection Pool (90 sessions) ? 80% 10% 10% instance 1 instance 2 instance 3 56 © 2009/2010 Pythian
  84. 84. RLB - connection gravitation Connection Pool (90 sessions) ? 80% 10% 10% instance 1 instance 2 instance 3 Connection pool must be over-allocated! 56 © 2009/2010 Pythian
  85. 85. RLB - how is workload gravitated? 12.0 INST1 INST2 INST3 25% Cumulative connections given 9.0 25% 6.0 3.0 50% 0 Time 57 © 2009/2010 Pythian
  86. 86. CLB options for RLB CLB_SHORT Connection Pool instance 1 instance 2 instance 3 58 © 2009/2010 Pythian
  87. 87. CLB options for RLB CLB_SHORT Establishing connections Connection Pool instance 1 instance 2 instance 3 58 © 2009/2010 Pythian
  88. 88. CLB options for RLB CLB_SHORT Establishing connections Connection Pool instance 1 instance 2 instance 3 Connections requests burst - most go to the node with the lowest goodness 58 © 2009/2010 Pythian
  89. 89. CLB options for RLB CLB_SHORT Establishing connections Connection Pool instance 1 instance 2 instance 3 Connections requests burst - most go to the node with the lowest goodness 58 © 2009/2010 Pythian
  90. 90. CLB options for RLB CLB_LONG Connection Pool instance 1 instance 2 instance 3 Connections requests distributed equally across all instances running a service 59 © 2009/2010 Pythian
  91. 91. CLB options for RLB CLB_LONG Connection Pool instance 1 instance 2 instance 3 Connections requests distributed equally across all instances running a service 59 © 2009/2010 Pythian
  92. 92. Rule of thumb use CLB_LONG with connection pools 60 © 2009/2010 Pythian
  93. 93. JDBC Implicit Connection Cache • Standard JDBC drivers + ons.jar ods.setConnectionCachingEnabled(True); ods.setFastConnectionFailoverEnabled(True); ods.setONSConfiguration ("nodes=lh1:6201,lh2:6201"); Properties prop = new Properties(); prop.setProperty("MinLimit", MIN_CONN); prop.setProperty("MaxLimit", MAX_CONN); prop.setProperty("InitialLimit", INIT_CONN); prop.put (oracle.net.ns.SQLnetDef.TCP_CONNTIMEOUT_STR, "1000")); // 1 second ods.setConnectionCacheProperties(prop); © 2009/2010 Pythian
  94. 94. Demo 7 - JDBC RLB • ImplicitConnection Cache • Enabling FCF • Subscribe with ONS • Tracing LBA events in JDBC 62 © 2009/2010 Pythian
  95. 95. ONS JDBC bugs • Bug 6151350 - RESTRICTION ON SETONSCONFIGURATION • Bug 6315760 - SETONSCONFIGURATION ONLY REGISTERS A MAXIMUM OF THREE NODES • -Doracle.ons.maxconnections=5 63 © 2009/2010 Pythian
  96. 96. Are we there yet? !Client-sidebasics !Oracle Cluster Services !Server-side CLB !FAN - HA & LBA events !Thin JDBC RLB !ONS Subscriber • Your own LBA! © 2009/2010 Pythian
  97. 97. Demo 8 - ONC JDBC API • LocalONS is required on client • Subscribing to events • How to filter only required events • Parsing events 65 © 2009/2010 Pythian
  98. 98. Are we there YET??? !Client-sidebasics !Oracle Cluster Services !Server-side CLB !FAN - HA & LBA events !Thin JDBC RLB !ONS Subscriber !Your own LBA! © 2009/2010 Pythian
  99. 99. Demo 9 - your own LBA • Maximum flexibility of LBA • ONS Publisher • Oracle doesn’t know how to balance you workload better than you! • This is not documented anywhere and likely unsupported 67 © 2009/2010 Pythian
  100. 100. Are we there yet? NO :) !Oracle Cluster Services !Establishing a connection !FAN - HA & LBA events ! !Thin JDBC RLB !ONS Subscriber !Your own LBA! © 2009/2010 Pythian
  101. 101. 11g New Features • SCAN Listener • Universal Connection Pool • Connection Affinity in LBA (web sessions, XA) • Reduces Cache Fusion overhead • Requires UCP • Data Guard Broker integration with FAN events • Policy-Based Cluster and Capacity Management 69 © 2009/2010 Pythian
  102. 102. ye t? t he re A re we © 2009/2010 Pythian
  103. 103. Bonus - WebLogic Server Connection Management 71 © 2009/2010 Pythian
  104. 104. Source - WebLogic Server 10.3.1 documentation 72 © 2009/2010 Pythian
  105. 105. WebLogic Configurations Options for Oracle RAC •4 Supported Data Source Configurations MDS MDS RAC Oracle XA no-XA Services native Load balancing Y Y Y N Failover Y Y Y Y XA support Y N Y N JDBC Store N Y Y Y RAC Services N N Y Y 73 © 2009/2010 Pythian
  106. 106. WebLogic Multi Data Source with Oracle RAC WebLogic Server Multi Data Source • App uses MDS Data Data Data • MDS routes connection Source 1 Source 2 Source 3 to one of its DS • failover or load balancing • DS online management • Suspend Instance 1 Instance 2 Instance 3 • Shutdown • Add • Remove Database 74 © 2009/2010 Pythian
  107. 107. WebLogic MDS for Failover with Oracle RAC WebLogic Server • Failover MDS Multi Data Source • DS health checks Data Data Data Source 1 Source 2 Source 3 • failure => DS dead • Can manually disable DS • Dead DS are re-tested • default 120 seconds Instance 1 Instance 2 Instance 3 Database 75 © 2009/2010 Pythian
  108. 108. WebLogic MDS for Failover with Oracle RAC WebLogic Server • Failover MDS Multi Data Source • DS health checks Data Data Data Source 1 Source 2 Source 3 • failure => DS dead • Can manually disable DS • Dead DS are re-tested • default 120 seconds Instance 1 Instance 2 Instance 3 Database 75 © 2009/2010 Pythian
  109. 109. WebLogic MDS for Failover with Oracle RAC WebLogic Server • Failover MDS Multi Data Source • DS health checks Data Data Data Source 1 Source 2 Source 3 • failure => DS dead • Can manually disable DS • Dead DS are re-tested • default 120 seconds Instance 1 Instance 2 Instance 3 Database 75 © 2009/2010 Pythian
  110. 110. WebLogic MDS for Failover with Oracle RAC WebLogic Server • Failover MDS Multi Data Source • DS health checks Data Data Data Source 1 Source 2 Source 3 • failure => DS dead • Can manually disable DS • Dead DS are re-tested • default 120 seconds Instance 1 Instance 2 Instance 3 Database 75 © 2009/2010 Pythian
  111. 111. WebLogic MDS for Failover with Oracle RAC WebLogic Server • Failover MDS Multi Data Source • DS health checks Data Data Data Source 1 Source 2 Source 3 • failure => DS dead • Can manually disable DS • Dead DS are re-tested • default 120 seconds Instance 1 Instance 2 Instance 3 Database 75 © 2009/2010 Pythian
  112. 112. WebLogic MDS for Failover with Oracle RAC WebLogic Server • Failover MDS Multi Data Source • DS health checks Data Data Data Source 1 Source 2 Source 3 • failure => DS dead • Can manually disable DS • Dead DS are re-tested • default 120 seconds Instance 1 Instance 2 Instance 3 Database 75 © 2009/2010 Pythian
  113. 113. WebLogic MDS for Failover with Oracle RAC WebLogic Server • Failover MDS Multi Data Source • DS health checks Data Data Data Source 1 Source 2 Source 3 • failure => DS dead • Can manually disable DS • Dead DS are re-tested • default 120 seconds Instance 1 Instance 2 Instance 3 Database 75 © 2009/2010 Pythian
  114. 114. WebLogic MDS for Failover with Oracle RAC WebLogic Server • Failover MDS Multi Data Source • DS health checks Data Data Data Source 1 Source 2 Source 3 • failure => DS dead • Can manually disable DS • Dead DS are re-tested • default 120 seconds Instance 1 Instance 2 Instance 3 Database 75 © 2009/2010 Pythian
  115. 115. 76 © 2009/2010 Pythian
  116. 116. WebLogic Multi Data Source with Oracle RAC WebLogic Server Multi Data Source • Load Balancing Data Data Data • MDS routes connection Source 1 Source 2 Source 3 to all active DS • round-robin • failover is supported • no performance base load Instance 1 Instance 2 Instance 3 balancing like Oracle RLB • DS size doesn’t affect LB Database 77 © 2009/2010 Pythian
  117. 117. Oracle RAC: JDBC Pool vs WebLogic Multi DS Oracle JDBC WLS Multi-DS Connect failover connect time MDS failover failover Connection load balancing JDBC CLB none Server-side CLB based on Listener none service metrics / performance Control connections layout RAC Services MDS config / members Reconnect on failure FCF MDS managed Run-time load balancing based RLB / LBA MDS LB on service performance round-robin Use new (or recovered) instance not until automated reconnect by MDS 78 © 2009/2010 Pythian
  118. 118. WebLogic Connections to RAC Services WebLogic Server • Initial Capacity = 0 Multi Data Source Data Data Data • DS must exist for every node Source 1 Source 2 Source 3 that can run the service • Services layout change requires WLS reconfiguration • XA supported MY_SRV MY_SRV • branches of the same XA Instance 1 Instance 2 Instance 3 transactions are routed to the same instance • MDS for load balancing • Failover makes little sense Database 79 © 2009/2010 Pythian
  119. 119. WebLogic Connections to RAC Services WebLogic Server Multi Data Source Data Data Data Source 1 Source 2 Source 3 BOB_SRV Instance 1 Instance 2 Instance 3 Database 80 © 2009/2010 Pythian
  120. 120. WebLogic Unmanaged DS with Oracle RAC WebLogic Server Unmanaged Data Source • JDBC native CLB Oracle JDBC “unmanaged” pool • DB Server - CLB_GOAL_LONG • Failover - FCF • Based on HA events • RLB does not work Instance 1 Instance 2 Instance 3 • Connection testing not required Database 81 © 2009/2010 Pythian
  121. 121. What’s next? • Book draw • Leave me your business cards Q&A Email me - gorbachev@pythian.com Read my blog - http://www.pythian.com Follow me on Twitter - @AlexGorbachev Join Pythian fan club on Facebook & LinkedIn 82 © 2009/2010 Pythian

×