• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
A discusion of adoption of Oracle Enterprise Manager

A discusion of adoption of Oracle Enterprise Manager



A history of EM adoption within my organization

A history of EM adoption within my organization



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    A discusion of adoption of Oracle Enterprise Manager A discusion of adoption of Oracle Enterprise Manager Presentation Transcript

    • ITEC Grid Control Adoption Fall Wizard 2009 Update Dan@itec.suny.edu 1Monday, August 29, 2011 1
    • Agenda Recap of last years presentation EM Case Studies Q&A 2Monday, August 29, 2011 2
    • Fall Wizard 2008 (Last Year) 0 8 20 3Monday, August 29, 2011 3
    • Fall Wizard 2008 (Last Year) 0 8 20 4Monday, August 29, 2011 4
    • Fall Wizard 2008 (Last Year) 0 8 20 5Monday, August 29, 2011 5
    • Fall Wizard 2008 (Last Year) 0 8 20 6Monday, August 29, 2011 6
    • Fall Wizard 2008 (Last Year) 0 8 20 7Monday, August 29, 2011 7
    • Fall Wizard 2008 (Last Year) 0 8 20 8Monday, August 29, 2011 8
    • Fall Wizard 2008 (Last Year) 0 8 20 9Monday, August 29, 2011 9
    • Fall Wizard 2008 (Last Year) 0 8 20 10Monday, August 29, 2011 10
    • Fall Wizard 2008 (Last Year) 0 8 20 11Monday, August 29, 2011 11
    • Fall Wizard 2008 (Last Year) 0 8 20 12Monday, August 29, 2011 12
    • Fall Wizard 2008 (Last Year) 0 8 20 1.Dictionary Comparisons 2.Database Comparisons 3.Server Comparisons 4.User Defined Metric (student registration) 5.SLM - Service Level Management 13Monday, August 29, 2011 13
    • Fall Wizard 2008 (Last Year) 0 8 20 14Monday, August 29, 2011 14
    • On to the new stuff ... 15Monday, August 29, 2011 15
    • EM Case Studies We’ll be discussing some specific cases wherein we used EM to our advantage Usage of EM Groups Storage failure in September Angel performance problems in September Audit for Nassau CCC Preferred Credential management 16Monday, August 29, 2011 16
    • EM Case Studies - Groups What groups do for you How we are using them for Information Organization Management (admins) 17Monday, August 29, 2011 17
    • EM Groups With so many (1600+) targets, we have a need to categorize them Groups provide a mechanism to organize these into collections of targets that have something in common. All Prod databases All Linux servers All servers for Nassau Targets can of course be assigned to multiple groups P.S we have not yet figured out how to do (mathematical) ‘set’ operations with our groups. 18Monday, August 29, 2011 18
    • EM Groups (W,W,W) WHO WHAT WHERE Who uses these The service The location of the targets (regardless delivered (regardless targets. of purpose). of who uses them) eg: eg: eg: WHERE_ITEC WHO_NASSAU WHAT_BANNER WHERE_OAS WHO_SLN WHAT_ANGEL WHERE_NOT_ WHO_ITEC WHAT_ATTASK HOSTED 19Monday, August 29, 2011 19
    • W,W,W Group Usage @ ITEC Every target must exist in at least 3 groups - one from each of WHO, WHAT, WHERE We have EM reports that help us find targets that are not in at least one WHO, WHAT and WHERE groups 20Monday, August 29, 2011 20
    • W,W,W Group Usage @ ITEC Administrator-to-target assignments are (beginning to be) assigned via groups as opposed to individual targets. This means that our views and alerts in EM are tailor made for each admin Notifications are no longer blasted out to everyone (about everything). 21Monday, August 29, 2011 21
    • Tailored EM Home Page Administrators now experience a home page that reflects ONLY targets of direct interest to them. 22Monday, August 29, 2011 22
    • Group Views Group Home Page Group Charts Page Group Admin Page Group Members Page 23Monday, August 29, 2011 23
    • Group Home Page Lets explore this a bit . 24Monday, August 29, 2011 24
    • Group Views - Home Page (left half) Similar to the EM home page Has an overall status (with history to back it up). But also shows configuration changes (on the left) Details on that later 25Monday, August 29, 2011 25
    • Group Views - Home Page (right half) Alerts, general Policy Violations and Security Policy Violations Current and new in last 24 hours Distinct Rules Violated 26Monday, August 29, 2011 26
    • Group Views - Current Critical Alerts Drop down for View Data. Open Alerts are alerts that are still a concern and have not been dispositioned (automatically or manually) Can also see all alerts raised in the last 24h, 7d, 31d 27Monday, August 29, 2011 27
    • Group Views - Security Policy Violations (Distinct Rules) Same data available elsewhere in EM but ... Uses the ‘member of’ option to deliver relevent violations only. 28Monday, August 29, 2011 28
    • Group Views - Charts Customizable to show ‘many’ of the applicable metrics Cuts through the forest by showing ‘top X’ or ‘bottom Y’ targets only 29Monday, August 29, 2011 29
    • Group Views - Administration 30Monday, August 29, 2011 30
    • Group Views - Administration Create Jobs Execute host or sql Search configurations Manage / view listeners Create blackouts For each of these fns the group eases 31 mgmt ...Monday, August 29, 2011 31
    • Group Views - Administration Available targets are auto populated Remove those you don’t want See execution results across all targets in one place. Alternately - switch to single target mode 32Monday, August 29, 2011 32
    • Group Views - Members Combination of top level Hosts and Database pages in EM Shows key ‘life signs’ of each target in the group Columns are customizable and almost every one has drilldown Total Disk IO repeating column is a bug 33Monday, August 29, 2011 33
    • Group Views - Dashboards 34Monday, August 29, 2011 34
    • EM Groups - Summary Powerful capabilities that are easy to implement Improves organizational capabilities of targets Greatly improves manageability for admins Requires vigilence though in keeping group memberships up to date Can use custom EM reports to help with this 35Monday, August 29, 2011 35
    • EM Case Studies - Storage Woes What happened How EM could not help (much) in detection How EM did help in recovery 2 36Monday, August 29, 2011 36
    • Storage Failure Catastrophic storage failure (gobs more detail in other sessions), emails, documents, ... For the purpose of this discussion we’re interested in EM usage at the time of crisis 37Monday, August 29, 2011 37
    • Storage Failure EM was not able to detect the underlying problem nor did it help much in diagnosis after the fact. It WAS however very helpful in getting us back in business 38Monday, August 29, 2011 38
    • Storage Failure - EM backlog FYI - It takes time for EM to catch up (before it can tell you anything terribly interesting). 39Monday, August 29, 2011 39
    • Storage Failure 4:17 pm 3:41 pm 7:23 pm Our WHERE ITEC group to the rescue 40Monday, August 29, 2011 40
    • Storage Failure - Getting Back in Business http://grid01.itec.suny.edu:7778/em/console/monitoring/availCurrent$ctxType=$target=WHERE_20ITEC$type=composite$curStatus=2,3,4,6 Setup Preferences Hosts | Databases | Application Servers | Web Applications | Services | Systems | Groups | All Targets | Collaboration Suites Group: WHERE ITEC > Member Status for WHERE ITEC (Group) Page Refreshed Sep 10, 2009 3:33:15 PM EDT Status Unknown Previous Show All 155 Next Availability Target Type as_dutapp11.dutapp11.dut.itec.suny.edu.dutapp11.dut.itec.suny.edu_BC4J ADF Business Components for Java as_dutapp12.dutapp12.dut.itec.suny.edu.dutapp12.dut.itec.suny.edu_BC4J ADF Business Components for Java buc_10.bucapp10.buc.itec.suny.edu_BC4J ADF Business Components for Java admdevbuc.buc.itec.suny.edu:3872 Agent admoradr3.dr.itec.suny.edu:3872 Agent admorania2.nia.itec.suny.edu:3872 41 AgentMonday, August 29, 2011 41 bucapp10.buc.itec.suny.edu:3872 Agent
    • Storage Failure - Conclusions Not an essential part of the process but certainly helpful as a triage tool I don’t seem to have saved screenshots but we did again make use of our limited SLM dashboards to see if applications (as opposed to servers) were operational. 42Monday, August 29, 2011 42
    • EM Case Studies - Angel Performance Problems What happened How EM helped or hindered Not a blow by blow recount - just showing things that are interesting from the EM perspective 43Monday, August 29, 2011 43
    • Service Tests confirmed FYI - These are MAX response times across all available beacons. 44Monday, August 29, 2011 44
    • Service Tests - bad data? We had 26 service tests defined running from 3 different locations We doubted what we were seeing because only 5 campuses were reporting problems via the helpdesk - we concluded that the service test results were erroneous Mostly because the tests were showing consistent problems across all 26 campuses (from 3 known locations) 45Monday, August 29, 2011 45
    • Service tests - inconsistencies Data that makes no sense: The beacon running from OAS was consistently showing NO problems The beacon internal to our network WAS showing performance problems 46Monday, August 29, 2011 46
    • Service Tests inconsistencies - hindsight It turns out that all campuses were experiencing problems - we just were not aware of them We upgraded the beacon at OAS and it fell back in line with what others were seeing The network topology for our internal beacon was not what we thought it was owing to it’s location 47Monday, August 29, 2011 47
    • Service Tests - Granularity By default - we were capturing TOTAL perceived response time. A good rough metric But what if there is a single step in the synthetic transaction that is a problem? 48Monday, August 29, 2011 48
    • Service Tests - granularity 49Monday, August 29, 2011 49
    • Keep in mind that you’re increasing the amount of data collected. In this case by factor of 11x. 50Monday, August 29, 2011 50
    • So - we need more data We quickly worked with campuses to provision more agents throughout the state We went from 3 to 11 On a variety of platforms should help identify outlying causes of misleading data i.e. why do beacons on Windows show X while linux ones do not? Data now conclusively showed that the problem was related to our network 51Monday, August 29, 2011 51
    • Problem Resolution Ultimately the issue was directly tied to network consumption (in part due to what EM was showing us) and we increased the allotment of SLN traffic from 33 mb/s to 66 to 120 mb/s and this particular problem moved onto the next component in the stack. (Application Servers, Database Servers, ...) 52Monday, August 29, 2011 52
    • Service Tests - a resource concern Now that we had some breathing room and time to digest, SLN brought up the concern that with the increased number of beacons, the synthetic transactions had been contributing to the saturation of the network A real and valid concern How to understand the impact of service tests then from a network perspective? 53Monday, August 29, 2011 53
    • Service tests and network load Not a trivial question to understand how much bandwidth a test consumes Networking team was able to see single 400 KB/s spikes in bandwidth consumption at the time that a test was executing Surely EM KNOWs how much - it’s running the test and has about a dozen metrics it gathers? 54Monday, August 29, 2011 54
    • Service tests and network load 55Monday, August 29, 2011 55
    • Service tests and network load Total Time Transfer Bytes Beacon (ms) Rate (KB/s) Transfered itxpgrid01 3383 186.4 630.5912 OldWestbury 4921 128.1 630.3801 Oneonta 4553 138.5 630.5905 Bytes Transferred = Transfer Rate * (Total Time / 1000) 56Monday, August 29, 2011 56
    • Service Test - last bit-o-math So, lets go with this higher number of 630 KB of data transferred during a test 11 beacons running 26 service tests at 5 minute intervals = ~3432 tests per hour (I’m starting to worry) 3432 * 630 KB = 2,162,160 KB / Hour Divide that by 3600 seconds per hour Average throughput would be 600 KB/s or 4,804 kb/s 57Monday, August 29, 2011 57
    • Service Test - last bit-o-math This represented 14% when we were capped at 33 mb/s ! Now that we are at 120mb/s (or more) this drops down to 4% Bottom line is - it is NOT zero and needs to be considered. 58Monday, August 29, 2011 58
    • Service Test - last bit-o-math FYI - these numbers assume perfectly uniform test distribution (timing) which is not under our direct control More beacons means even more network usage and we would certainly like to go beyond the 11 we have. 59Monday, August 29, 2011 59
    • Service Tests - research continues We are working an SR with Oracle to confirm the math We are working with Oracle to create an enhancement request to enable us to cache graphics and scripts from the beacon (akin to what a user would do) May ask them to get fancy and let us cache all day and purge the cache at a set time each day (sort of mimicking ‘existing users’ vs ‘first time users’) 60Monday, August 29, 2011 60
    • Service Tests - final lessons We are taking advantage of the SLN development installation to develop and test out service tests and understand their impact P.S. also be aware that these synthetic tests may show up in your business analytics as real users and need to be filtered out. 61Monday, August 29, 2011 61
    • Also something of value... (Web Inspector in Safari) Can show both TIME and SIZE 62Monday, August 29, 2011 62
    • Nassau Audit ITEC is being asked to help auditors understand our change management procedures. As part of this - they wanted to see all system changes between dd/mon/yyyy and dd/mon/ yyyy. 63Monday, August 29, 2011 63
    • Config History - where is it at? 64Monday, August 29, 2011 64
    • Configuration History - Narrow the search Note - you can use GROUPS as well as specific target names 65Monday, August 29, 2011 65
    • 66Monday, August 29, 2011 66
    • Configuration Change - Details 67Monday, August 29, 2011 67
    • Configuration Change Out of box functionality - no configuration changes are required on your part Collects and tracks all Oracle software (database and application server) and some database (meta) changes. Startup, redo log rotation, options installation, ... Not changes internal to the database Beware - it does not keep it forever. Seems to age out after a year. 68Monday, August 29, 2011 68
    • EM Case Studies - Preferred Credentials How preferred credentials work (out of box) Why this is a real concern for us (other than the obvious) How we are automating our way to goodness (thanks Todd) 69Monday, August 29, 2011 69
    • 70Monday, August 29, 2011 70
    • 71Monday, August 29, 2011 71
    • 72Monday, August 29, 2011 72
    • Preferred Creds - what makes it interesting? We have been able to create scripts which make use of OS commands to create new ‘hardened’ passwords and .. Set new host and database passwords (on the hosts and databases) Update the preferred credentials in EM for appropriate admins Update confluence with pgp encrypted pages that contain this password information 73Monday, August 29, 2011 73
    • Preferred Credentials Basically for all the work we do through EM we don’t need to know the credentials to do our day to day work And we can refer to Confluence should we need direct access Targets which employ this are once again controlled by EMs group capabilities. 74Monday, August 29, 2011 74
    • Review Review EM Groups Recovery from storage failure Angel performance problems Nassau change management Credential management 75Monday, August 29, 2011 75
    • Q&A You can always email me at: Dan@itec.suny.edu 76Monday, August 29, 2011 76