IO Resource Management on Exadata

1,738 views
1,522 views

Published on

Published in: Technology
1 Comment
2 Likes
Statistics
Notes
No Downloads
Views
Total views
1,738
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
1
Likes
2
Embeds 0
No embeds

No notes for slide

IO Resource Management on Exadata

  1. 1. IO Resource ManagementCONFIDENTIAL |12012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  2. 2. Contents§ What is IO Resource Management?§ IO Resource Management and DBRM§ IORM Architecture§ IORM Monitoring§ IORM in Action§ Business Justification for IORM CONFIDENTIAL |2 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  3. 3. About Me§ John Clarke, Centroid§ Specialized in §  Oracle Exadata §  Performance Tuning and RAC §  Data Warehousing, 11g, Linux §  Oracle Apps DBA§ john.clarke@centroid.com, www.centroid.com§ www.centroid.com§ Blogs at www.centroid.com/knowledgebase§ Sporadically dumps random thoughts at @jccclarke CONFIDENTIAL |3 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  4. 4. What is IORM?§ IORM = IO Resource Management§ IORM is a means to govern and prioritize IO requests to Exadata storage servers§ Often used in conjunction with DBRM§ If consolidating databases on Exadata, use IORM§ If you’re not consolidating, still often makes sense! CONFIDENTIAL |4 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  5. 5. What is IORM?Some basics …§ Exadata “IORM Plan” = Inter-Database plan + Category plan§ A category plan specifies resource allocations per category of resource consumer groups –  Categories are assigned to DBRM resource consumer groups –  Categories (can) span databases –  Consumer groups are within a database§ A Inter-Database plan specifies IO resource allocations per database§ An Intra-Database plan = a DBRM Plan CONFIDENTIAL |5 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  6. 6. What is IORM? (continued)There are 3 IORM methods§ Category IORM –  (implies categories assigned to resource consumer groups)§ Inter-Database IORM –  (i.e., IO resource allocation between databases)§ Intra-Database IORM –  (i.e., allocations within a database)You can use combinations of the above … CONFIDENTIAL |6 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  7. 7. What is IORM? – In Pictures PROD consumer groups CG: FIN_OLTP CAT: NORMAL_PRIORITY DB% for PROD PROD CG: FIN_REPORTS CAT: LOW_PRIORITY CG: SHIPPING_OLTP CAT: HIGH_PRIORITY CG: DW_CRITICAL CAT: HIGH_PRIORITY DB% for DWPRD DWPRD CG: DW_NORMAL CAT: NORMAL_PRIORITY DWPRD consumer groups DEV consumer group DB% for DEV DEV CG: DEV_ALL CAT: LOW_PRIORITY CONFIDENTIAL |7 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  8. 8. Database Resource Management (DBRM)§ Database Resource Management (DBRM) – see http://docs.oracle.com/cd/E14072_01/server.112/e10595/dbrm.htm§ Used within a single database when you’ve got workloads with different performance requirements§ Resource consumer groups allow you to group sessions by workload§ Administrator specifies resource allocations per consumer group§ Administrator maps sessions to consumer groups§ The combination of consumer groups and session mappings = a resource plan§ A DBRM plan is also called an Intra-Database resource plan CONFIDENTIAL |8 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  9. 9. IORM and DBRMIntra-Database IORM§  If you have multiple workloads within a database, you need an Intra-Database resource plan§  If you only have only one database on your Exadata, the Intra-Database plan is all you need for IORM§  Exadata automatically prioritizes IO requests based on this Intra-Database plan CONFIDENTIAL |9 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  10. 10. IORM and DBRMInter-Database IORM§  If you have multiple databases on an Exadata that you wish to control IO allocations amongst, you need an Inter-Database plan§  Rules for an Inter-Database plan specify IO allotments per database, not resource consumer groups CONFIDENTIAL | 10 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  11. 11. IORM and DBRMCategory IORM§  A category plan is used to control IO resource utilization by classification of work being done – i.e., by categories of consumer groups§  Category plans assume Intra-Database plans are in effect CONFIDENTIAL | 11 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  12. 12. IORM Architecture DBRM runs in the database IORM part of CELLSRV CONFIDENTIAL | 12 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  13. 13. IORM Architecture – Rules of Engagement§ IORM only kicks in to prioritize IO when needed§ “When needed” = when IO is saturated to cell disks§ Any disk allocation not fully utilized is made available to other workloads§ Background IO is scheduled based on priority relative to user IO§ Redo and control file writes always have higher priority§ DBWR writes scheduled at same priority as user IO CONFIDENTIAL | 13 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  14. 14. IORM Architecture – IO Queues§ For each cell disk, each database accessing cell has one IO queue per consumer group and three background queues§ Background IO queues are mapped to high, medium, and low priority requests with different IO types mapped to each queue§ If no Intra-Database plans are set, all non-background IO requests are grouped into a single queue called OTHER_GROUPS CONFIDENTIAL | 14 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  15. 15. IORM Architecture – IO Queues No consumer groups No consumer groups All user IO goes to Other Groups queue OTHER_GROUPS OTHER_GROUPS High BG High BG Medium BG Medium BG Low BG Low BG PROD Queues DEV Queues Cell Disk CD_00_cm01cel01 CONFIDENTIAL | 15 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  16. 16. IORM Architecture – IO Queues DBRM consumer groups CG: OLTP PROD configured CG: REPT with three resource consumer groups CG: FINANCE OTHER_GROUPS OTHER_GROUPS High BG High BG Medium BG Medium BG Low BG Low BG PROD Queues DEV Queues Cell Disk CD_00_cm01cel01 CONFIDENTIAL | 16 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  17. 17. IORM ArchitectureiDB message contains databasename, consumer group + category,IO type, extents to get, etc. PROD DWPRD DEV PROD Cell disk 1 DWPRD Cell Disk 7 DEV Cell Disk 11 FIN_OLTP DW_CRITICAL DEV_ALL FIN_REPORTS DW_NORMAL OTHER_GROUPS SHIPPING_OLTP OTHER_GROUPS HIGH OTHER_GROUPS HIGH MEDIUM HIGH MEDIUM LOW MEDIUM LOW LOW PROD Cell disk 2 DWPRD Cell Disk 2 DEV Cell Disk 2 PROD Cell disk 3 DWPRD Cell Disk 3 DEV Cell Disk 3 PROD Cell disk N DWPRD Cell Disk N DEV Cell Disk NIO queues and cell services Disk 1 Disk 3 Disk 5 Disk 7 Disk 9 Disk 11 Exadata Cell Disks Disk 2 Disk 4 Disk 6 Disk 8 Disk 10 Disk 12 CONFIDENTIAL | 17 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  18. 18. IORM Architecture PROD DWPRD DEV PROD Cell disk 1 DWPRD Cell Disk 7 DEV Cell Disk 11 FIN_OLTP DW_CRITICAL DEV_ALL FIN_REPORTS DW_NORMAL OTHER_GROUPS SHIPPING_OLTP OTHER_GROUPS HIGH OTHER_GROUPS HIGH MEDIUM HIGH MEDIUM LOW MEDIUM LOW LOW PROD Cell disk N DWPRD Cell Disk N DEV Cell Disk N IORMIO queues and cell services Disk IO queues ordered after IORM does its logic Disk 1 Disk 3 Disk 5 Disk 7 Disk 9 Disk 11 Exadata Cell Disks Disk 2 Disk 4 Disk 6 Disk 8 Disk 10 Disk 12 CONFIDENTIAL | 18 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  19. 19. IORM Architecture – The MathMethods are evaluated based on a fixed order:§ Category IORM is evaluated first Intra CG% / sum (x) =§ Inter-Database IORM next “Consumer group weight” (my words …)§ Intra-Database IORM next “cat%” = percentage of CG% = (Intra CG% / Σ(x)) * db% * cat% resource allocation for the category in which the consumer group belongs “CG%” = IORM “Intra CG%” = determined resource resource allocation for allocation for consumer group “x” = sum of Intra- db% = percentage of consumer group within an Intra- Database consumer database allocation in sessions Database plan group allocations for all the Inter-Database plan consumer groups in the same category CONFIDENTIAL | 19 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  20. 20. IORM Architecture – The Math All User IONo IORM 100% 9.87% = Cat% * DB% * CG weight Cat% = 65%Category IORM Cat1: 65% Cat2: 25% Cat3: 10%= 40% DB% CG Weight = 60% / (sum over Cat1) = 60% / (80% + 8% + 60% + 10%) = 37.7% 9.87% = 65% * 40% * 37.7%Inter-Database IORM DB1: 60% DB2: 40% DB1: 60% DB2: 40% DB1: 60% DB2: 40% 39% 26% 15% 10% 6% 4% CG1: 80% L1, Category=Cat1 19.97% CG1: 60% L1, Category=Cat1 9.87% CG2: 40% L2, Category=Cat1 1.97% CG2: 10% L1, Category=Cat1 1.65%Intra-Database IORM CG3: 40% L2, Category=Cat3 0.38% CG3: 40% L2, Category=Cat2 5.45% DB1 DB2 CG4: 50% L3, Category=Cat2 6.82% CG4: 40% L2, Category=Cat3 3.75% CONFIDENTIAL | 20 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  21. 21. Or if you like Excel …Enter values in the blue IORM resource calculation https://dl.dropbox.com/u/92079964/IORM%20Planning%20Calculator.xlsx CONFIDENTIAL | 21 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  22. 22. Monitoring IORMHere’s what I’m going to talk about in the next few slides …§ IORM metrics§ Using CellCLI to monitor IORM§ metric_iorm.pl (MOS note 1337265.1)§ cellsrv_statedump§ What I like to do with the data CONFIDENTIAL | 22 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  23. 23. Monitoring IORM: IORM Metrics§ Exadata tracks IORM metrics in the storage cell. Metrics include: –  Counter and wait metrics –  Utilization and throughput metrics§ Exadata metrics can be queried from CellCLI as of: –  Now, using LIST METRICCURRENT –  Historically, using LIST METRICHISTORY§ Exadata groups IORM metrics by: –  Category IORM method §  objectType=‘IORM_CATEGORY’ §  name like ‘CT_.*’ –  Inter-Database IORM method §  objectType=‘IORM_DATABASE’ §  name like ‘DB_.*’ –  Consumer group IORM method §  objectType = ‘IORM_CONSUMER_GROUP’ §  name like ‘CG_.*’ CONFIDENTIAL | 23 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  24. 24. Monitoring IORM: Category IORM Metrics [root@cm01cel01 ~]# cellcli -e list metricdefinition where objectType=IORM_CATEGORY!Name Description Type UnitCT_FC_IO_BY_SEC Number of megabytes of I/O per second for this category to flash cache Instantaneous MB/secCT_FC_IO_RQ Number of IO requests issued by an IORM category to flash cache Cumulative IO requestsCT_FC_IO_RQ_SEC Number of IO requests issued by an IORM category to flash cache per second Rate IO/secCT_FD_IO_BY_SEC Number of megabytes of I/O per second for this category to flash disks Instantaneous MB/secCT_FD_IO_LOAD Average I/O load from this category for flash disks Instantaneous NumberCT_FD_IO_RQ_LG Number of large IO requests issued by an IORM category to flash disks Cumulative IO requestsCT_FD_IO_RQ_LG_SEC Number of large IO requests issued by an IORM category to flash disks per second Rate IO/secCT_FD_IO_RQ_SM Number of small IO requests issued by an IORM category to flash disks Cumulative IO requestsCT_FD_IO_RQ_SM_SEC Number of small IO requests issued by an IORM category to flash disks per second Rate IO/sec Throughput, MB/secCT_IO_BY_SEC Number of megabytes of I/O per second for this category to hard disks Instantaneous MB/sec by categoryCT_IO_LOAD Average I/O load from this category for hard disks Instantaneous NumberCT_IO_RQ_LG Number of large IO requests issued by an IORM category to hard disks Cumulative IO requestsCT_IO_RQ_LG_SEC Number of large IO requests issued by an IORM category to hard disks per second Rate IO/secCT_IO_RQ_SM Number of small IO requests issued by an IORM category to hard disks Cumulative IO requests/sec IO requestsCT_IO_RQ_SM_SEC Number of small IO requests issued by an IORM category to hard disks per second Rate IO/secCT_IO_UTIL_LG Percentage of disk resources utilized by large requests from this category Instantaneous % Disk utilization byCT_IO_UTIL_SM Percentage of disk resources utilized by small requests from this category Instantaneous category %CT_IO_WT_LG IORM wait time for large IO requests issued by an IORM category Cumulative msCT_IO_WT_LG_RQ Average IORM wait time per request for large IO requests issued by an IORM category Rate ms/requestCT_IO_WT_SM IORM wait time for small IO requests issued by an IORM category Cumulative IO wait ms time/secCT_IO_WT_SM_RQ Average IORM wait time per request for small IO requests issued by an IORM category Rate ms/request CONFIDENTIAL | 24 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  25. 25. Monitoring IORM: Inter-Database IORM Metrics [root@cm01cel01 ~]# cellcli -e list metricdefinition where objectType=IORM_DATABASE!Name Description Type UnitDB_FC_IO_BY_SEC Number of megabytes of I/O per second for this database to flash cache Instantaneous MB/secDB_FC_IO_RQ Number of IO requests issued by a database to flash cache Cumulative IO requestsDB_FC_IO_RQ_SEC Number of IO requests issued by a database to flash cache per second Rate IO/secDB_FD_IO_BY_SEC Number of megabytes of I/O per second for this database to flash disks Instantaneous MB/secDB_FD_IO_LOAD Average I/O load from this database for flash disks Instantaneous NumberDB_FD_IO_RQ_LG Number of large IO requests issued by a database to flash disks Cumulative IO requestsDB_FD_IO_RQ_LG_SEC Number of large IO requests issued by a database to flash disks per second Rate IO/secDB_FD_IO_RQ_SM Number of small IO requests issued by a database to flash disks Cumulative IO requestsDB_FD_IO_RQ_SM_SEC Number of small IO requests issued by a database to flash disks per second Rate IO/sec Throughput, MB/secDB_IO_BY_SEC Number of megabytes of I/O per second for this database to hard disks Instantaneous MB/sec by DBDB_IO_LOAD Average I/O load from this database for hard disks Instantaneous NumberDB_IO_RQ_LG Number of large IO requests issued by a database to hard disks Cumulative IO requestsDB_IO_RQ_LG_SEC Number of large IO requests issued by a database to hard disks per second Rate IO/secDB_IO_RQ_SM Number of small IO requests issued by a database to hard disks Cumulative IO requests/sec IO requestsDB_IO_RQ_SM_SEC Number of small IO requests issued by a database to hard disks per second Rate IO/secDB_IO_UTIL_LG Percentage of disk resources utilized by large requests from this database Instantaneous % Disk utilization byDB_IO_UTIL_SM Percentage of disk resources utilized by small requests from this database Instantaneous DB %DB_IO_WT_LG IORM wait time for large IO requests issued by a database Cumulative msDB_IO_WT_LG_RQ Average IORM wait time per request for large IO requests issued by a database Rate ms/requestDB_IO_WT_SM IORM wait time for small IO requests issued by a database Cumulative IO wait ms time/secDB_IO_WT_SM_RQ Average IORM wait time per request for small IO requests issued by a database Rate ms/request CONFIDENTIAL | 25 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  26. 26. Monitoring IORM: Inter-Database IORM Metrics (cont) [root@cm01cel01 ~]# cellcli -e list metricdefinition where objectType=IORM_DATABASE!Name Description Type UnitDB_FL_IO_BY The number of MB written to the Flash Log Cumulative MBDB_FL_IO_BY_SEC The number of MB written per second to the Flash Log Instantaneous MB/secDB_FL_IO_RQ The number of I/O requests issued to the Flash Log Cumulative IO RequestsDB_FL_IO_RQ_SEC The number of I/O requests per second issued to the Flash Log Instantaneous IO/sec CONFIDENTIAL | 26 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  27. 27. Monitoring IORM: Consumer Group IORM Metrics [root@cm01cel01 ~]# cellcli -e list metricdefinition where objectType=IORM_CONSUMER_GROUP!Name Description Type UnitCG_FC_IO_BY_SEC Number of megabytes of I/O per second for this consumer group to flash cache Instantaneous MB/secCG_FC_IO_RQ Number of IO requests issued by a consumer group to flash cache Cumulative IO requestsCG_FC_IO_RQ_SEC Number of IO requests issued by a consumer group to flash cache per second Rate IO/secCG_FD_IO_BY_SEC Number of megabytes of I/O per second for this consumer group to flash disks Instantaneous MB/secCG_FD_IO_LOAD Average I/O load from this consumer group for flash disks Instantaneous NumberCG_FD_IO_RQ_LG Number of large IO requests issued by a consumer group to flash disks Cumulative IO requestsCG_FD_IO_RQ_LG_SEC Number of large IO requests issued by a consumer group to flash disks per second Rate IO/secCG_FD_IO_RQ_SM Number of small IO requests issued by a consumer group to flash disks Cumulative IO requestsCG_FD_IO_RQ_SM_SEC Number of small IO requests issued by a consumer group to flash disks per second Rate IO/sec Throughput, MB/secCG_IO_BY_SEC Number of megabytes of I/O per second for this consumer group to hard disks Instantaneous MB/sec by CGCG_IO_LOAD Average I/O load from this consumer group for hard disks Instantaneous NumberCG_IO_RQ_LG Number of large IO requests issued by a consumer group category to hard disks Cumulative IO requestsCG_IO_RQ_LG_SEC Number of large IO requests issued by a consumer group to hard disks per second Rate IO/secCG_IO_RQ_SM Number of small IO requests issued by a consumer group to hard disks Cumulative IO requests/sec IO requestsCG_IO_RQ_SM_SEC Number of small IO requests issued by a consumer group to hard disks per second Rate IO/secCG_IO_UTIL_LG Percentage of disk resources utilized by large requests from this consumer group Instantaneous % Disk utilization byCG_IO_UTIL_SM Percentage of disk resources utilized by small requests from this consumer group Instantaneous CG %CG_IO_WT_LG IORM wait time for large IO requests issued by a consumer group Cumulative msCG_IO_WT_LG_RQ Average IORM wait time per request for large IO requests issued by a consumer group Rate ms/requestCG_IO_WT_SM IORM wait time for small IO requests issued by a consumer group Cumulative IO wait ms time/secCG_IO_WT_SM_RQ Average IORM wait time per request for small IO requests issued by a consumer group Rate ms/request CONFIDENTIAL | 27 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  28. 28. Monitoring IORM§ Even if you don’t implement IORM plans, database IO is tagged and some database metrics are populated§ These ‘DB_.*’ metrics can be valuable No IORM plan active Did some large IO on DWPRD, metrics displayed CONFIDENTIAL | 28 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  29. 29. Monitoring IORM: Using CellCLI § Use cellcli “list” to display IORM metrics § “list metriccurrent” displays current metrics and “list metrichistory” displays historical metrics § Query with dcli to show metrics on all the nodes § Examples: Display historical metrics for Show historical metrics all cells using dcli based on date rangeDisplay current metrics for MB/sec per database CONFIDENTIAL | 29 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  30. 30. Monitoring IORM: Common challenges with CellCLI1.  Difficult to remember metric names2.  Difficult to remember/type cellcli query syntax, especially with metrichistory and dates3.  Lack of ability to monitor metrics over time interval, real-time (like “sar –u <interval> <iterations>”4.  Difficult to assemble output and make it usableHere’s how I address it:§ celldig.sh => Addresses #1 & #2 above§ cellstats.sh => Addresses #3 above§ celliormbundle.sh => Addresses #4 above§ celliormbundle2.sh => Uses ageInMinutes criteria (got lazy)https://dl.dropbox.com/u/92079964/cellscr.zip CONFIDENTIAL | 30 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  31. 31. Monitoring IORM: celldig.sh Download and try it out … CONFIDENTIAL | 31 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  32. 32. Monitoring IORM: celldig.sh Start from now – 1 dayand display 20 min of data Here’s the cellcli command CONFIDENTIAL | 32 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  33. 33. Monitoring IORM: cellstats.sh Query all cells, 2 second intervals, 5 iterations Query single cell, 2 second intervals, 5 iterations CONFIDENTIAL | 33 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  34. 34. Monitoring IORM: celliormbundle.shWhat it does:1.  Collects cell server metrics for IORM-related objects2.  Accepts a “from” and “to” date range3.  Output directed to temp files and loaded from external tablesPrep work: Run celliormbundle_setup.shcelliormbundle.sh and celliormbundle2.sh will be used to gatherperformance metrics for IORM test scenarios later … CONFIDENTIAL | 34 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  35. 35. Monitoring IORM: celliormbundle.sh Usage Execution Loading IORM_METRICS CONFIDENTIAL | 35 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  36. 36. Monitoring IORM: celliormbundle2.sh Usage Execution Load IORM_METRICS CONFIDENTIAL | 36 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  37. 37. Monitoring IORM: metric_iorm.pl§ metric_iorm.pl nicely summarizes IORM metrics and can be downloaded from MOS note 1337265.1§ Without any input parameters, displays current metrics§ With collectionTime-scoped input parameters, displays historical metrics summarized by minute§ Good way to understand what resources are consuming the most IO, whether disk latencies are reasonable, flash cache hit metrics per database, and overall IORM plan impact on performance§ Works regardless of whether IORM plan is active CONFIDENTIAL | 37 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  38. 38. Monitoring IORM: metric_iorm.plExample: Show current metrics for a cell: CONFIDENTIAL | 38 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  39. 39. Monitoring IORM: metric_iorm.plExample: Display metrics between 9:00am and 9:30am on 7/12/2012 Summarizes cell stats as well CONFIDENTIAL | 39 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  40. 40. Monitoring IORM: metric_iorm.plExample: Display metrics from ½ hour ago for a duration of 10 minutes Use “ageInMinutes” syntax CONFIDENTIAL | 40 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  41. 41. Monitoring IORM: metric_iorm.plExample: Collect across all cells using dcli CONFIDENTIAL | 41 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  42. 42. Monitoring IORM: Inside a CELLSRV statedump§ Event cellsrv.cellsrv_statedump dumps current state of cell to trace file§ Contains some good IORM information§ Invoke like this: CONFIDENTIAL | 42 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  43. 43. Monitoring IORM: Inside a CELLSRV statedumpUnder the “IORM State Dump” section, trace file organized like:§ IORM Enabled heading –  Current IORM Plans §  Category Plan details –  Map of database and consumer groups to categories §  Inter-Database Plan details §  Intra-Database Plan details §  Complete Mappings –  Queued IO Requests for cellDisk /dev/sdX (repeated for each cell disk) §  Category Plan –  Inter-Database Plan »  Intra-Database Plan »  IO queuing details per consumer group, database, category, and cell disk –  Queued IO Requests for cellDisk /dev/sdY (repeat previous section) CONFIDENTIAL | 43 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  44. 44. Monitoring IORM: Inside a CELLSRV statedump Search for this section in trace file Currently, no IORM plan is enabled CONFIDENTIAL | 44 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  45. 45. Monitoring IORM: Inside a CELLSRV statedump Sections arranged by disk device IORM boost = ratio of IOs skipped to IOs scheduled = relative measure of how active IO scheduling is happening Broken down by Category first … then database … then intra db consumer group CONFIDENTIAL | 45 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  46. 46. Monitoring IORM: Inside a CELLSRV statedump Here’s one with IORM enabled Dumps Category plan first … then DB + CG to Category mapping CONFIDENTIAL | 46 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  47. 47. Monitoring IORM: Inside a CELLSRV statedump Next is displays Inter-Database plan Intra-Database plans for each database CONFIDENTIAL | 47 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  48. 48. Monitoring IORM: Inside a CELLSRV statedump For each devices, shows queued IO requests – we have none here Detail section similar to before CONFIDENTIAL | 48 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  49. 49. Monitoring IORM: What to do with the dataI like to do this:§ When analyzing a specific time period: –  Run celliormbundle.sh or celliormbundle2.sh –  Query the data in CELLIORM.IORM_METRICS via SQL –  Dump output to Excel and graph –  Capture test timings§ When analyzing performance for “right now”: –  Run metric_iorm.pl with no arguments and examine output –  Run cellstats.sh and iterate over a period of time for a specific metric for a metric or 2 CONFIDENTIAL | 49 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  50. 50. IORM in Action§ Test #1: Inter-Database IORM plan with 4 databases§ Test #2: Inter-Database IORM plan with 4 databases + DB limits§ Test #3: Category IORM plan§ Test #4: Inter-Database IORM + Category IORM§ Test #5: Inter-Database IORM + Category IORM, give DWPRD all the resources it needs CONFIDENTIAL | 50 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  51. 51. IORM in ActionFor each test, we’ll:§ Plan§ Show expected resource allocations§ Implement§ Validate§ Conduct a load test§ Analyze CONFIDENTIAL | 51 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  52. 52. IORM in Action: Baseline TestFirst I’ll run a baseline without IORM plans active and analyze data …§ I’ll use a modified version of Karl Arao’s IOsaturationtoolkit-v2 ( http://karlarao.wordpress.com/2012/05/14/iosaturationtoolkit-v2-with- iorm-and-awesome-text-graph/) to generate load§ In this, I’ll run 4 concurrent full-scans against a large table with 3 separate schemas (ORACLE,ORACLE2, and ORACLE3) across 4 databases (DWPRD, DWPROD, VISX, and VISY)§ I’ll analyze the following: –  Output from IOsaturationtoolkit-v2 log files –  Data in IORM_METRICS from celliormbundle.sh –  Data from output of metric_iorm.plTest started at 2012-07-17T17:22:43 and ended at 2012-07-17T17:28:45 CONFIDENTIAL | 52 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  53. 53. IORM in Action: Baseline TestIOsaturationtoolkit-v2 Results Statistics relatively random CONFIDENTIAL | 53 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  54. 54. IORM in Action: Test 1 PlanningInter-Database IORM with 4 databasesInter-Database IO allocation:§ VISX: 55%§ DWPRD: 25%§ DWPROD: 15%§ VISY: 5% CONFIDENTIAL | 54 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  55. 55. IORM in Action: Test 1 Resource AllocationsInter-Database IORM with 4 databases Empty the Category cells Enter IORM % per DB per plan Select DB, enter arbitrary consumer groups, select OTHER/ALL Category Inter-Database IO allocations calculated CONFIDENTIAL | 55 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  56. 56. IORM in Action: Test 1 ImplementInter-Database IORM with 4 databases I like to do this first … Set VISX to 55%, level 1 Set DWPRD to 25%, level 1 Set DWPROD to 15%, level 1 Set VISY to 5%, level 1 Set OTHER to 100%, level 2 Activate plan CONFIDENTIAL | 56 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  57. 57. IORM in Action: Test 1 ValidateInter-Database plan using 4 databases “list iormplan detail” CONFIDENTIAL | 57 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  58. 58. IORM in Action: Test 1 Validate (continued)Inter-Database plan using 4 databases Issue cell server statedump IORM plan active Just 2 categories … OTHER = all DBs for now . OTHER_GROUPS since no Intra-Database plans CONFIDENTIAL | 58 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  59. 59. IORM in Action: Test 1 Load TestInter-Database IORM with 4 databasesWe’ll conduct the same load test we did in our Test 0 baseline test Sampled DB_IO_WT_LG_RQ (avg. IO wait time / request / large) Notice wait times per DB align with inter-database plan CONFIDENTIAL | 59 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  60. 60. IORM in Action: Test 1 AnalyzeInter-Database IORM with 4 databasesIOsaturationtoolkit-v2 Results:Test duration times matched expectations IO throughput matched expectations visx < dwprd < dwprod < visy visx > dwprd > dwprod > visy CONFIDENTIAL | 60 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  61. 61. IORM in Action: Test 1 AnalyzeInter-Database IORM with 4 databasesCELLORM.IORM_METRIC data Query or MBPS, IOPS, Utilization, and Waits CONFIDENTIAL | 61 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  62. 62. IORM in Action: Test 1 AnalyzeInter-Database IORM with 4 databasesCELLORM.IORM_METRIC data As expected, VISX had best throughput CONFIDENTIAL | 62 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  63. 63. IORM in Action: Test 1 AnalyzeInter-Database IORM with 4 databasesCELLORM.IORM_METRIC data VISX had highest IOPs and VISY the lowest CONFIDENTIAL | 63 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  64. 64. IORM in Action: Test 1 AnalyzeInter-Database IORM with 4 databasesCELLORM.IORM_METRIC data Utilization numbers match expectations CONFIDENTIAL | 64 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  65. 65. IORM in Action: Test 1 AnalyzeInter-Database IORM with 4 databasesCELLORM.IORM_METRIC data VISY waited the most, on average, per request CONFIDENTIAL | 65 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  66. 66. IORM in Action: Test 1 AnalyzeInter-Database IORM with 4 databasesmetric_iorm.pl output Seeing non-zero queue times CONFIDENTIAL | 66 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  67. 67. IORM in Action: Test 2 PlanningInter-Database IORM plan with 4 databases and IORM database limits§ Inter-Database plan requirements same as Test 1 –  VISX: 55% –  DWPRD: 25% –  DWPROD: 15% –  VISXY: 5%§ IORM database IO limits also imposed on each database: –  VISX: 80% –  DWPRD: 10% –  DWPROD: 5% –  VISXY: 5% CONFIDENTIAL | 67 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  68. 68. IORM in Action: Test 2 Resource AllocationsInter-Database IORM plan with 4 databases and IORM database limits IO limits per database Adjusted resource limits CONFIDENTIAL | 68 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  69. 69. IORM in Action: Test 2 ImplementInter-Database IORM plan with 4 databases and IORM database limits Note limit clauses CONFIDENTIAL | 69 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  70. 70. IORM in Action: Test 2 ValidateInter-Database IORM plan with 4 databases and IORM database limits CONFIDENTIAL | 70 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  71. 71. IORM in Action: Test 2 ValidateInter-Database IORM plan with 4 databases and IORM database limits DB limit CONFIDENTIAL | 71 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  72. 72. IORM in Action: Test 2 Load TestInter-Database IORM plan with 4 databases and IORM database limitsWe’ll conduct the same load test we did previously Note wait distribution CONFIDENTIAL | 72 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  73. 73. IORM in Action: Test 2 AnalyzeInter-Database IORM plan with 4 databases and IORM database limitsIOsaturationtoolkit-v2 Results: VISX and DWPRD took the least amount of time IO throughput numbers significantly lower than Test 1but limits caused overall IO time to be less than in Test 1 Relative MB/s/database as expected CONFIDENTIAL | 73 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  74. 74. IORM in Action: Test 2 AnalyzeInter-Database IORM plan with 4 databases and IORM database limitsCELLIORM.IORM_METRICS data: Throughput less than Test 1, as expected CONFIDENTIAL | 74 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  75. 75. IORM in Action: Test 2 AnalyzeInter-Database IORM plan with 4 databases and IORM database limitsCELLIORM.IORM_METRICS data: IOPs less than Test 1, as expected CONFIDENTIAL | 75 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  76. 76. IORM in Action: Test 2 AnalyzeInter-Database IORM plan with 4 databases and IORM database limitsCELLIORM.IORM_METRICS data: Utilization less – imposing limits yields IO capacity CONFIDENTIAL | 76 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  77. 77. IORM in Action: Test 2 AnalyzeInter-Database IORM plan with 4 databases and IORM database limitsCELLIORM.IORM_METRICS data: Significantly more queue time, a reflection of limits CONFIDENTIAL | 77 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  78. 78. IORM in Action: Test 3 PlanningCategory IORM Plan§  No Inter-Database Plan§  Category Plan –  CAT_HIGH = 70%, Level 1 –  CAT_MEDIUM = 20%, Level 1 –  CAT_LOW = 80%, Level 2§  Consumer Group Mappings –  DWPRD: §  CG_CRITICAL (80% L1 intra) assigned CAT_HIGH category, mapped to user “ORACLE” §  CG_NORMAL (20% L1 intra) assigned CAT_MEDIUM category, mapped to user “ORACLE2” §  CG_OTHER (100% L2 intra) assigned CAT_LOW, mapped to user “ORACLE3” –  DWPROD: §  CG_CRITICAL (70% L1 intra) assigned CAT_HIGH category, mapped to user “ORACLE” §  CG_NORMAL (30% L1 intra) assigned CAT_MEDIUM category, mapped to user “ORACLE2” §  CG_OTHER (100% L2 intra) assigned CAT_LOW , mapped to user “ORACLE3” –  VISX: §  CG_SHIP_OLTP (50% L1 intra) assigned CAT_HIGH category, mapped to user “ORACLE” §  CG_FIN_OLTP (40% L1 intra) assigned CAT_HIGH category, mapped to user “ORACLE2” §  CG_FIN_REPT (10% L1 intra) assigned CAT_MEDIUM , mapped to user “ORACLE3” –  VISY: §  CG_OLTP(100% L1 intra) assigned CAT_LOW category, mapped to user “ORACLE” §  CG_REPT (100% L2 intra) assigned CAT_LOW category, mapped to user “ORACLE2” §  CG_OTHER (100% L3 intra) assigned CAT_LOW category, mapped to user “ORACLE3” CONFIDENTIAL | 78 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  79. 79. IORM in Action: Test 3 Planning Category IORM Plan VISX DWPRD DWPROD VISYCG_SHIP_OLTP: 50% L1 CG_CRITICAL: 80% L1 CG_CRITICAL: 70% L1 CG_OLTP: 100% L1CG_FIN_OLTP: 40% L1 CG_NORMAL: 20% L1 CG_NORMAL: 30% L1 CG_REPT: 100% L2CG_FIN_REPT: 10% L1 CG_OTHER: 100% L2 CG_OTHER: 100% L2 CG_OTHER: 100% L3 CG_OTHER: 100% L2 CATEGORY: HIGH_PRIORITY (70% L1) CATEGORY: MEDIUM_PRIORITY (20% L1) CATEGORY: LOW_PRIORITY (80% L2) CONFIDENTIAL | 79 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  80. 80. IORM in Action: Test 3 Resource AllocationsCategory IORM Plan Adjusted allocation percentages CONFIDENTIAL | 80 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  81. 81. IORM in Action: Test 3 ImplementCategory IORM PlanImplementation steps:§ Creating consumer groups in each database to match scenario and assign consumer groups to categories§ Map sessions to consumer groups and activate plan§ Enable IORM Category plan CONFIDENTIAL | 81 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  82. 82. IORM in Action: Test 3 ImplementCategory IORM Plan (DWPROD Intra-Database plan) Create categories Create CG and assign to category Map users Specify CG allocations Activate plan CONFIDENTIAL | 82 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  83. 83. IORM in Action: Test 3 ImplementCategory IORM Plan (DWPRD Intra-Database plan) CONFIDENTIAL | 83 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  84. 84. IORM in Action: Test 3 ImplementCategory IORM Plan (VISX Intra-Database plan) CONFIDENTIAL | 84 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  85. 85. IORM in Action: Test 3 ImplementCategory IORM Plan (VISY Intra-Database plan) CONFIDENTIAL | 85 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  86. 86. IORM in Action: Test 3 ImplementCategory IORM Plan Null out previous DB plan Category IORM CONFIDENTIAL | 86 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  87. 87. IORM in Action: Test 3 ValidateCategory IORM Plan CONFIDENTIAL | 87 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  88. 88. IORM in Action: Test 3 ValidateCategory IORM Plan Shows my categories Consumer group mapping per DB per Category cellsrv.statedump summarizes this well CONFIDENTIAL | 88 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  89. 89. IORM in Action: Test 3 Load TestCategory IORM Plan IO wait pattern: CAT_HIGH < CAT_MEDIUM < CAT_LOW Consumer group wait data similar during run CONFIDENTIAL | 89 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.
  90. 90. IORM in Action: Test 3 AnalyzeCategory IORM PlanIOsaturationtoolkit-v2 Results VISX had 2 peak MB/sec throughput due to two Relatively smooth distribution because we had a CAT_HIGH mappings mix of category mappings between databases (and corresponding “elapsed” time valleys) CONFIDENTIAL | 90 2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

×