ITEC Grid Control Adoption           Fall Wizard 2009 Update           Dan@itec.suny.edu                               1Mo...
Agenda               Recap of last years presentation               EM Case Studies               Q&A                     ...
Fall Wizard 2008 (Last Year)            0   8         20                          3Monday, August 29, 2011                ...
Fall Wizard 2008 (Last Year)            0   8         20                          4Monday, August 29, 2011                ...
Fall Wizard 2008 (Last Year)            0   8         20                          5Monday, August 29, 2011                ...
Fall Wizard 2008 (Last Year)            0   8         20                          6Monday, August 29, 2011                ...
Fall Wizard 2008 (Last Year)            0   8         20                          7Monday, August 29, 2011                ...
Fall Wizard 2008 (Last Year)            0   8         20                          8Monday, August 29, 2011                ...
Fall Wizard 2008 (Last Year)            0   8         20                          9Monday, August 29, 2011                ...
Fall Wizard 2008 (Last Year)            0   8         20                          10Monday, August 29, 2011               ...
Fall Wizard 2008 (Last Year)            0   8         20                          11Monday, August 29, 2011               ...
Fall Wizard 2008 (Last Year)            0   8         20                          12Monday, August 29, 2011               ...
Fall Wizard 2008 (Last Year)            0   8         20           1.Dictionary Comparisons           2.Database Compariso...
Fall Wizard 2008 (Last Year)            0   8         20                          14Monday, August 29, 2011               ...
On to the new stuff ...                                  15Monday, August 29, 2011                  15
EM Case Studies               We’ll be discussing some specific cases               wherein we used EM to our advantage   ...
EM Case Studies - Groups                          What groups do for you                          How we are using them fo...
EM Groups       With so many (1600+) targets, we       have a need to categorize them       Groups provide a mechanism to ...
EM Groups (W,W,W)             WHO                     WHAT                WHERE      Who uses these           The service ...
W,W,W Group Usage @ ITEC               Every target must exist in at least 3 groups               - one from each of WHO, ...
W,W,W Group Usage @ ITEC               Administrator-to-target assignments are               (beginning to be) assigned vi...
Tailored EM Home Page                                 Administrators                                 now experience       ...
Group Views               Group Home Page               Group Charts Page               Group Admin Page               Gro...
Group Home Page                Lets explore this a bit .                                            24Monday, August 29, 2...
Group Views - Home Page                    (left half)                               Similar to the EM                    ...
Group Views - Home Page                   (right half)                               Alerts, general                      ...
Group Views - Current Critical               Alerts          Drop down for View Data.          Open Alerts are alerts that...
Group Views - Security Policy       Violations (Distinct Rules)                               Same data                   ...
Group Views - Charts                                     Customizable                                     to show ‘many’  ...
Group Views - Administration                          30Monday, August 29, 2011              30
Group Views - Administration                               Create Jobs                               Execute host or      ...
Group Views - Administration                               Available                               targets are auto       ...
Group Views - Members                                 Combination of                                 top level Hosts      ...
Group Views - Dashboards                          34Monday, August 29, 2011                34
EM Groups - Summary       Powerful capabilities that are easy to       implement       Improves organizational capabilitie...
EM Case Studies - Storage                    Woes               What happened               How EM could not help (much) i...
Storage Failure               Catastrophic storage failure (gobs more               detail in other sessions), emails,    ...
Storage Failure               EM was not able to detect the underlying               problem nor did it help much in diagn...
Storage Failure - EM backlog    FYI - It takes time for EM to catch up   (before it can tell you anything terribly        ...
Storage Failure                               4:17 pm            3:41 pm                                 7:23 pm          ...
Storage Failure - Getting                  Back in Business                          http://grid01.itec.suny.edu:7778/em/c...
Storage Failure - Conclusions               Not an essential part of the process but               certainly helpful as a ...
EM Case Studies - Angel                Performance Problems               What happened               How EM helped or hin...
Service Tests confirmed      FYI - These        are MAX       response     times across      all available       beacons. ...
Service Tests - bad data?               We had 26 service tests defined running               from 3 different locations  ...
Service tests -                          inconsistencies                                        Data that makes no        ...
Service Tests inconsistencies               - hindsight               It turns out that all campuses were               ex...
Service Tests - Granularity               By default - we were capturing TOTAL               perceived response time.     ...
Service Tests - granularity                          49Monday, August 29, 2011                   49
Keep in mind that you’re increasing the                               amount of data collected. In this case by           ...
So - we need more data               We quickly worked with campuses to               provision more agents throughout the...
Problem Resolution               Ultimately the issue was directly tied to               network consumption (in part due ...
Service Tests - a resource                      concern               Now that we had some breathing room and             ...
Service tests and network                        load               Not a trivial question to understand how              ...
Service tests and network                        load                          55Monday, August 29, 2011                  55
Service tests and network                        load                                  Total Time         Transfer        ...
Service Test - last bit-o-math               So, lets go with this higher number of 630 KB               of data transferr...
Service Test - last bit-o-math             This represented 14% when we were capped at             33 mb/s !             N...
Service Test - last bit-o-math             FYI - these numbers assume perfectly uniform             test distribution (tim...
Service Tests - research                       continues               We are working an SR with Oracle to               c...
Service Tests - final lessons               We are taking advantage of the SLN               development installation to d...
Also something of value...                          (Web Inspector in Safari)                                           Ca...
Nassau Audit           ITEC is being asked to help auditors           understand our change management           procedure...
Config History - where is it                      at?                          64Monday, August 29, 2011                  ...
Configuration History -                    Narrow the search      Note - you can use GROUPS as well as specific           ...
66Monday, August 29, 2011        66
Configuration Change -                        Details                           67Monday, August 29, 2011                 ...
Configuration Change               Out of box functionality - no configuration               changes are required on your ...
EM Case Studies - Preferred                Credentials               How preferred credentials work (out of box)          ...
70Monday, August 29, 2011        70
71Monday, August 29, 2011        71
72Monday, August 29, 2011        72
Preferred Creds - what makes              it interesting?               We have been able to create scripts which         ...
Preferred Credentials               Basically for all the work we do through EM               we don’t need to know the cr...
Review               Review               EM Groups               Recovery from storage failure               Angel perfor...
Q&A        You can always email me at:               Dan@itec.suny.edu                                   76Monday, August ...
Upcoming SlideShare
Loading in...5
×

A discusion of adoption of Oracle Enterprise Manager

3,656

Published on

A history of EM adoption within my organization

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
3,656
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "A discusion of adoption of Oracle Enterprise Manager"

  1. 1. ITEC Grid Control Adoption Fall Wizard 2009 Update Dan@itec.suny.edu 1Monday, August 29, 2011 1
  2. 2. Agenda Recap of last years presentation EM Case Studies Q&A 2Monday, August 29, 2011 2
  3. 3. Fall Wizard 2008 (Last Year) 0 8 20 3Monday, August 29, 2011 3
  4. 4. Fall Wizard 2008 (Last Year) 0 8 20 4Monday, August 29, 2011 4
  5. 5. Fall Wizard 2008 (Last Year) 0 8 20 5Monday, August 29, 2011 5
  6. 6. Fall Wizard 2008 (Last Year) 0 8 20 6Monday, August 29, 2011 6
  7. 7. Fall Wizard 2008 (Last Year) 0 8 20 7Monday, August 29, 2011 7
  8. 8. Fall Wizard 2008 (Last Year) 0 8 20 8Monday, August 29, 2011 8
  9. 9. Fall Wizard 2008 (Last Year) 0 8 20 9Monday, August 29, 2011 9
  10. 10. Fall Wizard 2008 (Last Year) 0 8 20 10Monday, August 29, 2011 10
  11. 11. Fall Wizard 2008 (Last Year) 0 8 20 11Monday, August 29, 2011 11
  12. 12. Fall Wizard 2008 (Last Year) 0 8 20 12Monday, August 29, 2011 12
  13. 13. Fall Wizard 2008 (Last Year) 0 8 20 1.Dictionary Comparisons 2.Database Comparisons 3.Server Comparisons 4.User Defined Metric (student registration) 5.SLM - Service Level Management 13Monday, August 29, 2011 13
  14. 14. Fall Wizard 2008 (Last Year) 0 8 20 14Monday, August 29, 2011 14
  15. 15. On to the new stuff ... 15Monday, August 29, 2011 15
  16. 16. EM Case Studies We’ll be discussing some specific cases wherein we used EM to our advantage Usage of EM Groups Storage failure in September Angel performance problems in September Audit for Nassau CCC Preferred Credential management 16Monday, August 29, 2011 16
  17. 17. EM Case Studies - Groups What groups do for you How we are using them for Information Organization Management (admins) 17Monday, August 29, 2011 17
  18. 18. EM Groups With so many (1600+) targets, we have a need to categorize them Groups provide a mechanism to organize these into collections of targets that have something in common. All Prod databases All Linux servers All servers for Nassau Targets can of course be assigned to multiple groups P.S we have not yet figured out how to do (mathematical) ‘set’ operations with our groups. 18Monday, August 29, 2011 18
  19. 19. EM Groups (W,W,W) WHO WHAT WHERE Who uses these The service The location of the targets (regardless delivered (regardless targets. of purpose). of who uses them) eg: eg: eg: WHERE_ITEC WHO_NASSAU WHAT_BANNER WHERE_OAS WHO_SLN WHAT_ANGEL WHERE_NOT_ WHO_ITEC WHAT_ATTASK HOSTED 19Monday, August 29, 2011 19
  20. 20. W,W,W Group Usage @ ITEC Every target must exist in at least 3 groups - one from each of WHO, WHAT, WHERE We have EM reports that help us find targets that are not in at least one WHO, WHAT and WHERE groups 20Monday, August 29, 2011 20
  21. 21. W,W,W Group Usage @ ITEC Administrator-to-target assignments are (beginning to be) assigned via groups as opposed to individual targets. This means that our views and alerts in EM are tailor made for each admin Notifications are no longer blasted out to everyone (about everything). 21Monday, August 29, 2011 21
  22. 22. Tailored EM Home Page Administrators now experience a home page that reflects ONLY targets of direct interest to them. 22Monday, August 29, 2011 22
  23. 23. Group Views Group Home Page Group Charts Page Group Admin Page Group Members Page 23Monday, August 29, 2011 23
  24. 24. Group Home Page Lets explore this a bit . 24Monday, August 29, 2011 24
  25. 25. Group Views - Home Page (left half) Similar to the EM home page Has an overall status (with history to back it up). But also shows configuration changes (on the left) Details on that later 25Monday, August 29, 2011 25
  26. 26. Group Views - Home Page (right half) Alerts, general Policy Violations and Security Policy Violations Current and new in last 24 hours Distinct Rules Violated 26Monday, August 29, 2011 26
  27. 27. Group Views - Current Critical Alerts Drop down for View Data. Open Alerts are alerts that are still a concern and have not been dispositioned (automatically or manually) Can also see all alerts raised in the last 24h, 7d, 31d 27Monday, August 29, 2011 27
  28. 28. Group Views - Security Policy Violations (Distinct Rules) Same data available elsewhere in EM but ... Uses the ‘member of’ option to deliver relevent violations only. 28Monday, August 29, 2011 28
  29. 29. Group Views - Charts Customizable to show ‘many’ of the applicable metrics Cuts through the forest by showing ‘top X’ or ‘bottom Y’ targets only 29Monday, August 29, 2011 29
  30. 30. Group Views - Administration 30Monday, August 29, 2011 30
  31. 31. Group Views - Administration Create Jobs Execute host or sql Search configurations Manage / view listeners Create blackouts For each of these fns the group eases 31 mgmt ...Monday, August 29, 2011 31
  32. 32. Group Views - Administration Available targets are auto populated Remove those you don’t want See execution results across all targets in one place. Alternately - switch to single target mode 32Monday, August 29, 2011 32
  33. 33. Group Views - Members Combination of top level Hosts and Database pages in EM Shows key ‘life signs’ of each target in the group Columns are customizable and almost every one has drilldown Total Disk IO repeating column is a bug 33Monday, August 29, 2011 33
  34. 34. Group Views - Dashboards 34Monday, August 29, 2011 34
  35. 35. EM Groups - Summary Powerful capabilities that are easy to implement Improves organizational capabilities of targets Greatly improves manageability for admins Requires vigilence though in keeping group memberships up to date Can use custom EM reports to help with this 35Monday, August 29, 2011 35
  36. 36. EM Case Studies - Storage Woes What happened How EM could not help (much) in detection How EM did help in recovery 2 36Monday, August 29, 2011 36
  37. 37. Storage Failure Catastrophic storage failure (gobs more detail in other sessions), emails, documents, ... For the purpose of this discussion we’re interested in EM usage at the time of crisis 37Monday, August 29, 2011 37
  38. 38. Storage Failure EM was not able to detect the underlying problem nor did it help much in diagnosis after the fact. It WAS however very helpful in getting us back in business 38Monday, August 29, 2011 38
  39. 39. Storage Failure - EM backlog FYI - It takes time for EM to catch up (before it can tell you anything terribly interesting). 39Monday, August 29, 2011 39
  40. 40. Storage Failure 4:17 pm 3:41 pm 7:23 pm Our WHERE ITEC group to the rescue 40Monday, August 29, 2011 40
  41. 41. Storage Failure - Getting Back in Business http://grid01.itec.suny.edu:7778/em/console/monitoring/availCurrent$ctxType=$target=WHERE_20ITEC$type=composite$curStatus=2,3,4,6 Setup Preferences Hosts | Databases | Application Servers | Web Applications | Services | Systems | Groups | All Targets | Collaboration Suites Group: WHERE ITEC > Member Status for WHERE ITEC (Group) Page Refreshed Sep 10, 2009 3:33:15 PM EDT Status Unknown Previous Show All 155 Next Availability Target Type as_dutapp11.dutapp11.dut.itec.suny.edu.dutapp11.dut.itec.suny.edu_BC4J ADF Business Components for Java as_dutapp12.dutapp12.dut.itec.suny.edu.dutapp12.dut.itec.suny.edu_BC4J ADF Business Components for Java buc_10.bucapp10.buc.itec.suny.edu_BC4J ADF Business Components for Java admdevbuc.buc.itec.suny.edu:3872 Agent admoradr3.dr.itec.suny.edu:3872 Agent admorania2.nia.itec.suny.edu:3872 41 AgentMonday, August 29, 2011 41 bucapp10.buc.itec.suny.edu:3872 Agent
  42. 42. Storage Failure - Conclusions Not an essential part of the process but certainly helpful as a triage tool I don’t seem to have saved screenshots but we did again make use of our limited SLM dashboards to see if applications (as opposed to servers) were operational. 42Monday, August 29, 2011 42
  43. 43. EM Case Studies - Angel Performance Problems What happened How EM helped or hindered Not a blow by blow recount - just showing things that are interesting from the EM perspective 43Monday, August 29, 2011 43
  44. 44. Service Tests confirmed FYI - These are MAX response times across all available beacons. 44Monday, August 29, 2011 44
  45. 45. Service Tests - bad data? We had 26 service tests defined running from 3 different locations We doubted what we were seeing because only 5 campuses were reporting problems via the helpdesk - we concluded that the service test results were erroneous Mostly because the tests were showing consistent problems across all 26 campuses (from 3 known locations) 45Monday, August 29, 2011 45
  46. 46. Service tests - inconsistencies Data that makes no sense: The beacon running from OAS was consistently showing NO problems The beacon internal to our network WAS showing performance problems 46Monday, August 29, 2011 46
  47. 47. Service Tests inconsistencies - hindsight It turns out that all campuses were experiencing problems - we just were not aware of them We upgraded the beacon at OAS and it fell back in line with what others were seeing The network topology for our internal beacon was not what we thought it was owing to it’s location 47Monday, August 29, 2011 47
  48. 48. Service Tests - Granularity By default - we were capturing TOTAL perceived response time. A good rough metric But what if there is a single step in the synthetic transaction that is a problem? 48Monday, August 29, 2011 48
  49. 49. Service Tests - granularity 49Monday, August 29, 2011 49
  50. 50. Keep in mind that you’re increasing the amount of data collected. In this case by factor of 11x. 50Monday, August 29, 2011 50
  51. 51. So - we need more data We quickly worked with campuses to provision more agents throughout the state We went from 3 to 11 On a variety of platforms should help identify outlying causes of misleading data i.e. why do beacons on Windows show X while linux ones do not? Data now conclusively showed that the problem was related to our network 51Monday, August 29, 2011 51
  52. 52. Problem Resolution Ultimately the issue was directly tied to network consumption (in part due to what EM was showing us) and we increased the allotment of SLN traffic from 33 mb/s to 66 to 120 mb/s and this particular problem moved onto the next component in the stack. (Application Servers, Database Servers, ...) 52Monday, August 29, 2011 52
  53. 53. Service Tests - a resource concern Now that we had some breathing room and time to digest, SLN brought up the concern that with the increased number of beacons, the synthetic transactions had been contributing to the saturation of the network A real and valid concern How to understand the impact of service tests then from a network perspective? 53Monday, August 29, 2011 53
  54. 54. Service tests and network load Not a trivial question to understand how much bandwidth a test consumes Networking team was able to see single 400 KB/s spikes in bandwidth consumption at the time that a test was executing Surely EM KNOWs how much - it’s running the test and has about a dozen metrics it gathers? 54Monday, August 29, 2011 54
  55. 55. Service tests and network load 55Monday, August 29, 2011 55
  56. 56. Service tests and network load Total Time Transfer Bytes Beacon (ms) Rate (KB/s) Transfered itxpgrid01 3383 186.4 630.5912 OldWestbury 4921 128.1 630.3801 Oneonta 4553 138.5 630.5905 Bytes Transferred = Transfer Rate * (Total Time / 1000) 56Monday, August 29, 2011 56
  57. 57. Service Test - last bit-o-math So, lets go with this higher number of 630 KB of data transferred during a test 11 beacons running 26 service tests at 5 minute intervals = ~3432 tests per hour (I’m starting to worry) 3432 * 630 KB = 2,162,160 KB / Hour Divide that by 3600 seconds per hour Average throughput would be 600 KB/s or 4,804 kb/s 57Monday, August 29, 2011 57
  58. 58. Service Test - last bit-o-math This represented 14% when we were capped at 33 mb/s ! Now that we are at 120mb/s (or more) this drops down to 4% Bottom line is - it is NOT zero and needs to be considered. 58Monday, August 29, 2011 58
  59. 59. Service Test - last bit-o-math FYI - these numbers assume perfectly uniform test distribution (timing) which is not under our direct control More beacons means even more network usage and we would certainly like to go beyond the 11 we have. 59Monday, August 29, 2011 59
  60. 60. Service Tests - research continues We are working an SR with Oracle to confirm the math We are working with Oracle to create an enhancement request to enable us to cache graphics and scripts from the beacon (akin to what a user would do) May ask them to get fancy and let us cache all day and purge the cache at a set time each day (sort of mimicking ‘existing users’ vs ‘first time users’) 60Monday, August 29, 2011 60
  61. 61. Service Tests - final lessons We are taking advantage of the SLN development installation to develop and test out service tests and understand their impact P.S. also be aware that these synthetic tests may show up in your business analytics as real users and need to be filtered out. 61Monday, August 29, 2011 61
  62. 62. Also something of value... (Web Inspector in Safari) Can show both TIME and SIZE 62Monday, August 29, 2011 62
  63. 63. Nassau Audit ITEC is being asked to help auditors understand our change management procedures. As part of this - they wanted to see all system changes between dd/mon/yyyy and dd/mon/ yyyy. 63Monday, August 29, 2011 63
  64. 64. Config History - where is it at? 64Monday, August 29, 2011 64
  65. 65. Configuration History - Narrow the search Note - you can use GROUPS as well as specific target names 65Monday, August 29, 2011 65
  66. 66. 66Monday, August 29, 2011 66
  67. 67. Configuration Change - Details 67Monday, August 29, 2011 67
  68. 68. Configuration Change Out of box functionality - no configuration changes are required on your part Collects and tracks all Oracle software (database and application server) and some database (meta) changes. Startup, redo log rotation, options installation, ... Not changes internal to the database Beware - it does not keep it forever. Seems to age out after a year. 68Monday, August 29, 2011 68
  69. 69. EM Case Studies - Preferred Credentials How preferred credentials work (out of box) Why this is a real concern for us (other than the obvious) How we are automating our way to goodness (thanks Todd) 69Monday, August 29, 2011 69
  70. 70. 70Monday, August 29, 2011 70
  71. 71. 71Monday, August 29, 2011 71
  72. 72. 72Monday, August 29, 2011 72
  73. 73. Preferred Creds - what makes it interesting? We have been able to create scripts which make use of OS commands to create new ‘hardened’ passwords and .. Set new host and database passwords (on the hosts and databases) Update the preferred credentials in EM for appropriate admins Update confluence with pgp encrypted pages that contain this password information 73Monday, August 29, 2011 73
  74. 74. Preferred Credentials Basically for all the work we do through EM we don’t need to know the credentials to do our day to day work And we can refer to Confluence should we need direct access Targets which employ this are once again controlled by EMs group capabilities. 74Monday, August 29, 2011 74
  75. 75. Review Review EM Groups Recovery from storage failure Angel performance problems Nassau change management Credential management 75Monday, August 29, 2011 75
  76. 76. Q&A You can always email me at: Dan@itec.suny.edu 76Monday, August 29, 2011 76

×