A presentation I did for the APAC OTN Tour in 2013

1. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.1

2. 13-Nov-13
2Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.
StrategiesforScalable,SmarterMonitoring
usingEM12c
PeteSharman
PrincipalProductManager,EMProductSuite

3. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.3
ProgramAgenda
MonitoringBestPractices
CommonQuestionsandTheirSolutions
EnterpriseManager12cRelease3Enhancements

4. 13-Nov-13
4Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.
TheObligatorySlideAboutMe
•18+yearswithOracle
–DBAinstructor/EducationCentreManager
–HAConsultant
–Internalservicescurriculumdeveloper
–NorthAmericaSalesandConsultingdatabaselead
–RACDevelopment
–STCurriculumDBA/EMteam
–ProductManager,EMProductSuite
•OakTablemember
•Presentedatconferencesallovertheplace
•Owner,petewhodoesnottweet.com☺

5. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.5
LaytheGroundworkforMonitoring
SetupAdministrationGroup
hierarchy
Fortheadministrationgroup:
–Define,AssociateTemplate
Collections
–Defineincidentrulesetstocreate
incidents,sendnotifications
–Createrole(s)containing
privilegesonthegroup
–Grantrole(s)toadministrators
SetUpOnce:MonitoringSettings,Privileges,IncidentRules
PRODGroupNon-PROD
Group
AllTargetsGroup
Production
Template
Collection
(Containsmonitoring
settings)
IncidentRuleSets
(Email,CreateIncident,Sendtraps…)
Role
AdministrationGrouphierarchy
Lifecycle
Status

6. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.6
ScalableMonitoring
Whennewtargetisaddedto
EnterpriseManager…
Setitstargetpropertiestojoin
administrationgroup
–Automaticallyappliedwith
monitoringsettings
–Automaticallyincludedinincident
rulesets
–Privilegeontargetincludedinrole
SolutionScalesasEnterpriseGrows
PRODGroup
Non-PROD
Group
AllTargetsGroup
Production
Template
Collection
(Containsmonitoring
settings)
IncidentRuleSets
(Email,CreateIncident,Sendtraps…)
Role
AdministrationGrouphierarchy
Lifecycle
Status=
Production

7. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.7
CommonQuestions:
AdministrationGroups
DynamicGroups
TargetProperties

8. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.8
AllAboutGroups
What’stheDifferenceBetweenDifferentTypesofGroups?
TypesofGroups
Groups/DynamicGroups
Group
Management
Operations:
Jobs
Blackouts
Dashboard,
Reports
etc.
AdministrationGroup
Hierarchy
Monitoring
(Auto
template
apply)

9. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.9
AdministrationGroupsandDynamicGroups
WhenShouldIUseAdministrationGroupsvs.DynamicGroups?
Main
Purpose
Membership
Basedon
Criteria
Additional
Membership
Requirements
Privilege
Propagating
Administration
Groups
Auto-apply
monitoring
templates
Yes,basedon
targetproperties
Targetcanbelongtoat
most1administration
group
Yes(always)
Dynamic
Groups
Anygroup
operation
Yes,basedon
targetproperties
Targetcanbelongto1
ormoregroups
User-
specified
option
AdministrationGroupsvs.DynamicGroups

10. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.10
AdministrationGroupHierarchy
Onehierarchyensuresatargetbelongstoatmost1administration
group
Preventsconflictscenarioswheretargetjoinsmultiplegroupswith
differentassociatedmonitoringtemplates
Willknowexactlyhowatargetobtainsitsmonitoringsettings
WhyCanIHaveOnly1AdministrationGroupHierarchy?

11. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.11
AdministrationGroupsandDynamicGroups
UseAdministrationGroupsformonitoring
UseAdministrationGroupsforothergroupoperationswherever
possible
Ifrequirementsforgroupisslightlydifferentfromadministrationgroup
criteria:
Option1:Changeadministrationgrouphierarchytoaccommodate
requirements
OR
Option2:Usedynamicgroups
WhenShouldIUseAdministrationGroupsvs.DynamicGroups?-2

12. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.12
Option1:Addnewlevel
AdministrationGroupvs.DynamicGroup
Example:NeedtodivideProdGroupintoProd-DBandProd-MW
AllTargets
Group
Prod
Group
Non-Prod
Group
Lifecycle
Status
Option2:UseDynamicGroups
Producti
Lifecycle
Status
Prod
Group
Non-Prod
Group
Administration
grouphierarchy
AllTargets
Group
Administration
grouphierarchy
Dynamic
Group
Prod-DB
LifecycleStatus=Production
AND
TargetType=OracleDatabase
Dynamic
Group
Prod-MW
LifecycleStatus=Production
AND
TargetType=OracleWebLogicServer
Non-Prod
MW
Target
Type
Prod-DBProd-MWNon-Prod
DB

13. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.13
AddingTargetstoAdministrationGroups
Tworequirements
1.Targetpropertiesmustmatchalladministrationgroupcriteria
Targetsinleafnodes
Targetsmatchingpartofcriterianotadded
2.Ifaggregatetarget,mustbeprivilege-propagating
Administrationgroupsareprivilege-propagating
Cannotdisableprivilege-propagationforadministrationgroups
WhyAreMyTargetsNotAddedtoAdministrationGroups?

14. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.14
AddingTargetstoAdministrationGroups-2
Tosettargetproperties
1.TargetAddition/PromotionUI
2.TargetPropertiesUI(targetmenuTargetSetupProperties)
3.CLIset_target_property_value
Bulksettingofproperties:useCLIset_target_property_valuewith–input_file
Forclustertargets,valuesetatclusterparentautoappliestoallmembertargets
Fornon-clusteraggregatetargets
–CLIset_target_property_valuewith–propagate_to_members
Setssamevaluetocurrentmembertargets
Re-runcommandforfuturemembertargets
–Ifmembertargetdoesn’tmatchcriteria,addedtoadministrationgroupas‘indirect
member’
SettingTargetProperties

15. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.15
AdministrationGroups
Directmembers
–Memberswhosetargetpropertiesmatchadministrationgroupcriteria
–Allgroupoperationsincludingautomatictemplateapply
Indirectmembers
–Targetswhosepropertiesdonotmatchcriteriabutaddedtogroupbecauseitsparentwas
added
–Aggregatetargetsautomaticallyincludeallmemberswhenaddedtogroup
–Example:WebLogicServer(direct),ApplicationDeployment(indirect)
–Allgroupoperations*except*automatictemplateapply
Tip:Ifyouwanttemplatestoauto-applytoallmembersofanaggregate,ensure
membersmatchadministrationgroupcriteria.Watchoutfor‘indirect’members.
Directvs.IndirectMembers

16. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.16
AdministrationGroups
Memberspage(Groupmenu->
Members->ShowAll)
UsefiltertoshowDirector
IndirectMembers
Templateapplyoperationsonly
forDirectMembers
Allothergroupoperationsapply
toallmembers
HowDoIKnowIfaMemberTargetisDirectorIndirectMember?

17. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.17
AddingTargetstoAdministrationGroups
Tworequirements
1.Targetpropertiesmustmatchalladministrationgroupcriteria
Targetsinleafnodes
Targetsmatchingpartofcriterianotadded
2.Ifaggregatetarget,mustbeprivilege-propagating
Administrationgroupsareprivilege-propagating
Cannotdisableprivilege-propagationforadministrationgroups
WhyAreMyTargetsNotAddedtoAdministrationGroups?

18. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.18
CheckforNon-PrivilegePropagatingAggregates
CheckUnassignedTargetspagefor
non-privilegepropagatingaggregates
Toconverttoprivilege-propagating
–EMCLImodify_system
-privilege_propagation=true
–Exclude:group,genericsystem,
genericservice
Pagealsoshowstargetsnotmatching
criteria
UnassignedTargets:TargetsNotPartofAnyAdministrationGroup

19. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.19
WhatIfIHavetoChangetheHierarchy?
Addand/ordeletenewlevels
Addnewlevel:Addtargetproperty
criterion
–Example:AddpropertyLineofBusiness
Finance,Sales
–Needtoassignvaluesofnewtarget
propertytomembers
Deletelevel:Removetargetproperty
criterion
ChangeHierarchyWithoutRe-Creation:Add/Removelevels
New
Rel3
AddLevel
AllTargets
Group
Prod
Group
Non-Prod
Group
Lifecycle
Status
Prod-
Finance
Sales-MWNon-Prod
Finance
Non-Prod
Sales
Lineof
Business

20. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.20
Retain
group
WhatIfIHavetoChangetheHierarchy?
Mergetwoormoregroupsatthesamelevel
–Choosegrouptoretain
–Membersofothergroup(s)movedtoretained
group
–Templatesonretainedgroupappliedtonewly
addedmembers
–Othergroup(s)deleted
Retainedgroup(s):
–Optiontokeeporgeneratenewgroupnames
–Operationsonretainedgrouparemaintained
ChangeHierarchyWithoutRe-Creation:MergeGroups
New
Rel3
Templatecollection
MergeGroups

21. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.21
MissionCriticalTest
OtherLinesof
Business
Production
TargetTypeDatabase
WebLogic
Server
DatabaseWebLogic
Server
DatabaseWebLogic
Server
Lifecycle
Status
Targetsmovedtoretainedgroup(s)
Lifecycle
Status=
Mission
Critical
Lifecycle
Status=
Production
Example:MergeMissionCriticalandProduction
•MergeMissionCriticaland
Production
•RetainMissionCriticalGroup
MonitoringTemplateappliedtonewlyadded
targets
AllTargetsin
AdminGroup

22. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.22
MissionCriticalor
Production
Test
OtherLinesof
Business
TargetTypeDatabase
WebLogic
Server
DatabaseWebLogic
Server
Lifecycle
Status
Result:MergedMissionCriticalandProduction
LifecycleStatus=MissionCriticalorProduction
ANDTargetType=Database
LifecycleStatus=
MissionCriticalor
Production
LifecycleStatus=MissionCriticalorProduction
ANDTargetType=WebLogicServer
AllTargetsin
AdminGroup

23. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.23
CommonQuestions:
MonitoringSettings

24. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.24
AreMyTargetsMonitoredUsingOurStandardsfor
Monitoring?
CheckSynchronizationStatusregionofTOPMOSTadministration
group(Showssyncstatusofalltargetsinhierarchy)
SyncStatusColumnWhatToDo
SynchronizedTargetsNothing.Targetsareinsyncwithmonitoringtemplates.
PendingTargetsEnsureyouhaveGlobalSyncScheduledefined.Indicatedby‘Next
Synchronization’date;ifN/Asetschedule.
RunningTargetsNothing.Checklatertoseeifthey’reallsynchronized.
FailedTargetsDrilldowntogetdetails;Fixwherepossible.Willattempttore-synconnext
syncschedule,orondemandbyuser.
ExcludedTargetsTargetsdeliberatelyexcludedfromsync,reviewifexclusionstillapplies.
N/ATargetsTargetshavenoassociatedmonitoringtemplate.Drilldowntogettarget
type,addmonitoringtemplatetotemplatecollection.

25. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.25
WhichTargetsHaveExemptions?
Targetcanhavemonitoringsettingsdifferentfrommonitoringtemplate
–Use‘PreventTemplateOverride’optioninmetricsettings
–Willnotbeflaggedasneededsynchronization
Listofexemptedtargets
–Report:“Targetswithpreventoverridesettingsassociatedwitha
MonitoringTemplatethroughaTemplateCollection”
TargetsWithSettingsDifferentFromMonitoringTemplates
New
Rel3

26. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.26
CommonQuestions:
EventsandIncidents
Notifications

27. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.27
Events
AMetricAlertisatypeofEvent
EventTypesrelatedtomonitoring
–MetricAlert
Examples:CPUUtilization(%),TablespaceSpaceUsed(%)
–TargetAvailability
Examples:TargetDown,AgentUnreachable,BlackoutStarted
BasedonStatusmetric
–MetricEvaluationError
Generatedwhenthereiserrorinevaluatingmetric
AreEventsandAlertsthesame?

28. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.28
EventsandIncidents
Incident~EnterpriseManager“ticket”
–Contains1ormoreevents
–Manage:Acknowledge,AssignOwner,TrackResolutionStatus,Prioritize,
Escalate
–Severity:worstseverityofalleventsitcontains
–Clearswhenalleventsthatitcontainsclears
IncidentRuleSets
–Sendnotifications,automateincidentmanagementoperations
IncidentManager
–Manage,track,diagnose,escalate,prioritizeincidents
HowareEventsandIncidentsRelated?

29. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.29
EventsandIncidents-2
AnIncidentautomaticallyclearswheneventthatitcontainsiscleared
Howisametricalert(event)cleared?
–Mosteventsareclearedbytheagentwhenvalueisbelowthreshold
Troubleshooting:manualre-evaluate,changethresholds,disable/
re-enablecollection
–ManuallyclearedalertsakaStatelessalerts(e.g.log-basedalerts)
ManuallyclearinIncidentManager
CLIclear_stateless_alert
Rulewithaction‘Clearevents’
HowDoIClearanIncident?

30. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.30
HowdoIManageEventswhenTargetGoesDown?
Automaticanalysisoftargetdownevents
Enablesadministratorstofocusonimportantevents
RCAevaluates/re-evaluatesastargetdowneventscomein
CausalAnalysisUpdateattributeinEvent/Incident
–RootCause
–CompositeDown
–Symptom
–N/A
RCAusesDependsOnandCompositeContainsassociations
RootCausalAnalysis(RCA)forTargetAvailabilityEventsNew
Rel3

31. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.31
CausalAnalysisforTargetAvailabilityEvents-2
DependsOnAssociation
WebLogicServer
J2EEapplication
DependsOn
J2EEApp
Down
event
WLSDown
event
Symptom
RootCause
RCAEngine
E
V
E
N
T
S

32. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.32
RAC
Instance1
CausalAnalysisforTargetAvailabilityEvents-3
CompositeContainsAssociation
ClusterDatabase
Composite
Contains
RAC
Instance
down
event
Cluster
Database
Down
event
Symptom
Composite
Down
RCAEngine
RAC
Instance2
E
V
E
N
T
S

33. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.33
UsingRCAinIncidentRuleSets
Createincidentsornotifyonlyfornon-
symptomevents
EventRule:ChooseTargetAvailability
eventtypeANDSeverityinFatal(target
down)
Causalanalysisupdate:
–Eventismarkedcauseorcompositedown
–Eventisnotcausenorsymptom
Action:email,createincident,etc.

34. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.34
UsingRCAinIncidentRuleSets-2
Createincidentsornotifyonlyfornon-
symptomevents
AddconditiontotheAction:
–Eventhasbeenopenforxxminutes
–Allowsforallrelevanteventstocomein,
updatedRCAresults
Addactionasneeded:createincident,
sendemail,etc.
Variation:SmallDelaytoGetUpdatedRCAResults

35. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.35
RCAResultsinIncidentManager
RootCause
ImpactedTargets
IncidentisSymptom

36. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.36
ExcludingSymptoms
FilterinSystemDashboardandIncident
Managertoexcludesymptoms
Shows:
–RootCauseIncidents
–CompositeDownIncidents
–N/AIncidents
ExcludeSymptomsinSystemDashboardandIncidentManager

37. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.37
IncidentRuleSets
RuleSetsexecutedinordercreated(canre-order)
RuleSetorderdoesnotmatterifrulesetsoperateonseparatetargets
Ordermatterswhenmultiplerulesetsoperateoncommonsetof
targets
–Creatingincidents:Firstruletocreateincidentforeventwins(eventcan
onlybepartofoneincident)
–Changingincidentworkflowattributes(owner,priority,status,escalation
level):Lastruleexecutedwins
Notification:Willalwaysbesent,regardlessofrulesetorder
OrderingRuleSets:Doesitreallymatter?

38. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.38
IncidentRuleSets
Out-of-boxrulesetsrecommendedforquickstart
Alwayscreateyourownrulesets(cancreate-likeonout-of-boxrulesets)
anddisableout-of-boxrulesets
Creatingrulesets
–Specifygroupastargetofruleset
–Allrulesrelatedtothegroupshouldbepartofthesameruleset
RuleSetsexecutedusingprivilegeofthecreatorfornon-emailactions
–Ifcreating/updatingincidents,needManageTargetEventsontarget
Toreceiveemailnotifications,recipientneedsViewontarget
Ifemailaddressisspecified,rulesetcreatorprivilegeswillbechecked
BestPractices

39. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.39
IncidentManagement
Periodicreviewandmaintenance
–Onlygenerateeventsthatyoucareabout
Removethresholds,disablecollectionofmetricsyoudon’tcareabout
–AddRulestoauto-clearalertsafterxxdays
–Purgemanuallyclearablealerts/incidents(CLI,UI)
Leverageyourdata
–Out-of-boxIncidentReports:EscalatedIncidents,IncidentHistory,Top
Events
–Repositoryviews:MGMT$INCIDENTS,MGMT$EVENTS_LATEST,etc.
Optimization

40. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.40
Benefits:EnterpriseManagerMonitoring
Scalablesolutionenablesdatacenter
growthwithminimaleffort
FlexibilityinsupportingwidevarietyofIT
operationsandprocedures
EnablesITtoprovidehighquality
serviceandmeetSLAgoals
EnablingValueThroughBestPractices

41. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.41
AdditionalResources
FamiliarizeyourselfwithEnterpriseManagermonitoring
–EnterpriseManagerLearningLibrary(OTN)
StrategiesforScalable,SmarterMonitoringusingOracleEnterprise
ManagerCloudControl12cwhitepaper(OTN)Updated
Rel3

42. 13-Nov-13
42Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.
Q&A

43. 13-Nov-13
Copyright©2013,Oracleand/oritsaffiliates.Allrightsreserved.43