MOUS13: Maintaining and Caring for your EPM Environment


Published on

'Maintaining and Caring for your EPM Environment', presented by Oracle Platinum Partner, Emtec, discusses log files, periodic maintenance activities & common issues, monitoring your environment, backups & patches and more!

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

MOUS13: Maintaining and Caring for your EPM Environment

  1. 1. Maintaining and Caring for your EPM Environment Luis Castillo and Len Romano, November 13, 2013
  2. 2. Presentation Agenda Introductions General Log Files Periodic Maintenance Activities-Common Issues Monitoring your Environment Backups and Patches Questions Emtec, Inc. Proprietary & Confidential. All rights reserved 2013. 2
  3. 3. Emtec… ‘Big enough to do the work, small enough to care’ EMTEC THE COMPANY • Serving clients for over 46 years • 14 locations, including 8 development centers • Voted One of the Top 100 Places to Work by Crain’s Chicago in 2012 • Our consultants have an average of 14 years of experience. Nearly 45% have advanced certifications • 73% of our PMs, Architects & Consultants worked at Tier 1 consultancies in a previous life • $255M IT Services Provider • Regularly compete & win against larger IT service providers US Locations • Radnor, PA (HQ) • Atlanta, GA • Chicago, IL • Dayton, OH • Fremont, CA • Jacksonville, FL • Herndon, VA • Minneapolis, MN • Springfield, NJ • St. Louis, MO THE WORLD Global Offices • Toronto, Canada • Ottawa, Canada • Pune, India • Bangalore, India Emtec, Inc. Proprietary & Confidential. All rights reserved 2012. • Emtec has completed more than 1,100 Package Application engagements, including: - • Over 1,000 full-time professionals EMTEC AROUND EMTEC SERVICES AT A GLANCE 200+ Oracle ERP clients with more than 415 engagements - 206 EPM / Hyperion & OBIEE clients and 323 engagements - 186 Microsoft clients and more than 255 projects • Emtec is a premier provider of SaaS and Emerging technologies solutions, including 51 Salesforce Clients with more than 75 engagements, Including Sales Cloud, Service Cloud and Custom Cloud • Emtec has performed more than 6.3M hours of application development and maintenance work ranging from Business Analysis to Product Development • Emtec has served more than 565 Infrastructure services clients where we have: - Deployed more than 1.85M servers, personal computers, tablets and mobile devices - Installed or provided more than 925 exabytes of storage (That’s a whole lot…) - Triaged more than 30,000 L1 support incidents - Implemented service management solutions for more than 125 clients 3
  4. 4. Emtec’s five practice areas offer focused solutions to help our clients solve complex business & IT challenges From up-front strategy & consulting all the way to application, infrastructure & procurement services, Emtec’s 1,000 professionals are what makes us special, and we invite you to see the difference for yourself. Consulting Services • IT Strategy & Planning • IT Process Improvement OFFERINGS • IT Performance and Governance • Information Management: − Big Data − Business Intelligence − Analytics • Mobility Solutions • Program and Project Management Package Application Services (PAS) Cloud Technologies Application Development & Management Services (ADMS) Infrastructure Services • Oracle − Fusion − EBS − PeopleSoft − EPM : Including Hyperion − OBIEE • Cloud Strategy & Planning • Portfolio Analysis • IT Service Mgmt. • Solutions − Sales Cloud − Service Cloud − Marketing Cloud − • Application Advisory Services • Data Center • Business Application Infrastructure • End User Computing • Microsoft − AX and GP − BI − CRM • Workday • Platforms • Package Application: Development, Maintenance & Support • SAP • Functional: HCM, FM, SCM • Cloud Application Development & Technology • Private Virtual Cloud • Cloud Applications • Product Development • Custom Application Development & Maintenance • Mobile solutions • Quality Assurance & Testing Services PRACTICE FACTS • Managed Services, Capacity Management & Staffing Services • SWAT team of 45 highly skilled Strategists, Architects and Consultants • 10 years of experience and more than 75 projects • 180 Specialists, including Project Managers, Architects, Consultants and Analysts • More than 300 ERP & CRM projects completed • Expert team of Big Data, business intelligence and analytics professionals Emtec, Inc. Proprietary & Confidential. All rights reserved 2012. • Specialized team of 25 Strategists, Project Managers, Architects, and Consultants • More than 90% certified • 62 Customers & 112 engagements 4 • Global workforce of nearly 750 associates (approximately 250 associates based in North America) • Provided more than 6.3M hours of application development and maintenance services. • Special expertise in Business Analysis and Testing (among other things) • More than 100 associates providing data center, compute, network and storage related services • Value Add Reseller providing full device lifecycle management services for more than 30 years • IT Service Mgmt. BU having supported more than 70 distinct clients
  5. 5. Subset of clients that have leveraged Emtec’s EPM services Emtec, Inc. Proprietary & Confidential. All rights reserved 2012. 5
  6. 6. General
  7. 7. General • As with all applications, the EPM Suite of products require care and maintenance to ensure optimal condition and minimize downtime for Business users. • The purpose of this session is to talk about some of the basic, yet important steps to maintain your EPM environment and some of the basic troubleshooting steps. Emtec, Inc. Proprietary & Confidential. All rights reserved 2012. 7
  8. 8. Log Files
  9. 9. Log Files • The first step to troubleshoot issues is to review the application logs • Different type of logs: –Installation logs –Configuration logs –WebLogic Managed Server logs –Application Logs Emtec, Inc. Proprietary & Confidential. All rights reserved 2012. 9
  10. 10. Log Files • Component Log location Log Type Logs Location Installation Logs <DRIVE>:OracleMiddlewareEPMSystem11R1diagnosticslogsinstall EPM System Configurator Logs <DRIVE>:OracleMiddlewareuser_projectsepmsystemxdiagnosticslogscon fig EPM System Diagnostics (also the location for the EPM System Validation Report) <DRIVE>:OracleMiddlewareuser_projectsepmsystemxdiagnosticslogsvali dation Service Startup log for each WebLogic Managed Server (detail level log and error information log) <DRIVE>:OracleMiddlewareuser_projectsepmsystemxdiagnosticslogsserv ices WebLogic Activity Logs <DRIVE>:OracleMiddlewareuser_projectsdomainsEPMSystemservers<MA NAGED_SERVER>logs Essbase Server Logs <DRIVE>:OracleMiddlewareuser_projectsepmsystemxdiagnosticslogsess baseessbase_0 (where 0 is the Essbase instance number) Essbase Application Logs <DRIVE>:OracleMiddlewareuser_projectsepmsystemxdiagnosticslogsess baseessbase_0 <APP> HFM Logs <DRIVE>:OracleMiddlewareuser_projectsepmsystemxdiagnosticslogshfm Planning Logs <DRIVE>:OracleMiddlewareuser_projectsepmsystemxdiagnosticslogsplan ning Emtec, Inc. Proprietary & Confidential. All rights reserved 2012. 10
  11. 11. Log Files • Foundation (Shared Services and Workspace) Logs Default Log Location Log File Name and Contents MIDDLEWARE_HOME/user_projects/domains/EPMSystem/servers/Foun  dationServices0/logs FoundationServices0.log–Server and security activity   SharedServices_Audit.log–Audit server errors while reading/writing audit information to the database or while configuring auditing  SharedServices_Audit_Client.log–Information about the audit client  SharedServices_CMSClient.log–Metadata Service client activity  SharedServices_Hub.log–Shared Services listener and initialization activity  SharedServices_ImportExport.log–  SharedServices_LCM.log–Lifecycle Management activity when it is run from EPM Workspace  SharedServices_Registry.log–Shared Services Registry activity  SharedServices_Security.log–User management, provisioning, authentication, and single sign-on activity  Emtec, Inc. Proprietary & Confidential. All rights reserved 2012. SharedServices_Admin.log–Applications Groups management activity Workspace.log–EPM Workspace error and informational messages 11
  12. 12. Log Files • EPM Application logs: Default Log Location Log File Name and Contents <DRIVE>:OracleMiddlewareuser_projectsepmsystemxdiagnosticslogspla  nning   <DRIVE>:OracleMiddlewareuser_projectsepmsystemxdiagnosticslogshf  m UserProvisionSync.log–Security refresh information, such as provisioning or “user not found” issues Planning utility logs–A log for each Planning utility PlanningAppUpgradeLog_application_name.txt–An upgrade log for each upgraded Planning application EPMWindowsConfig.log—Activity related to Financial Managementspecific configuration tasks  hfm.odl.log—Financial Management core activity  HsvEventLog.log—Financial Management activity  InteropJava.log—Financial Management interop activity <DRIVE>:OracleMiddlewareuser_projectsepmsystemxdiagnosticslogses  sbaseessbase_0, where 0 is an instance number ESSBASE.LOG–Essbase Server activities and errors dataload_ODL.err–Data load and dimension build errors  Emtec, Inc. Proprietary & Confidential. All rights reserved 2012.  log0000x.xcp–Errors that result when Essbase Server stops abnormally 12
  13. 13. Periodic Maintenance Activities-Common Issues
  14. 14. Periodic Maintenance Activities • Periodic maintenance activities can help prevent issues and performance impacts • The activities below should be performed periodically • Common: –Exclude the installation folder from automatic (<DRIVE>:OracleMiddleware) because this impacts performance –Virus scan should be scheduled to happen during maintenance period –Archive log files: Logs can become very large and may affect performance and may cause disk space shortage –Review patches periodically for potential defect resolution • HFM: –HFM Error and Audit Tables can impact performance after 500,000 rows –Archive and truncate the following HFM Tables • Quarterly: HFM_ERRORLOG • Yearly: <HFM_APP>_DATA_AUDIT • Yearly: <HFM_APP>_TASK_AUDIT –Delete HFM Invalid Records on a quarterly basis Emtec, Inc. Proprietary & Confidential. All rights reserved 2012. 14
  15. 15. Common Issues • The table below provides a list of the most common issues and possible solutions Issue Possible Solution Receive “Page Cannot be Displayed” when trying to access Workspace or Shared Services (HTTP Error 404) Ensure that the web server (IIS or Oracle HTTP Server) is running. Also, ensure that the services for the web component is also running External users (Active Directory) cannot login while native users can Ensure the connection information for Active Directory is correct and the password for the AD account has not changed. Engage AD Administrator Web application service crashes and log reports out of memory error Increase the Java Heap Size for the application. This can be done via the Windows Registry or via the startup script in non-Windows environments Essbase Service starts and stops immediately Verify that the drive did not run out of space. It is recommended that there is as much free space available as the largest Essbase cube to accommodate database restructures Essbase does not start The most common reason for Essbase to not start is a corrupted Security file. This happens if Essbase quit unexpectedly or due to ungraceful shutdown. Restore the Security file from its last backup Emtec, Inc. Proprietary & Confidential. All rights reserved 2012. 15
  16. 16. Monitoring your Environment
  17. 17. Monitoring your environment • Constantly monitoring the environment and resources will help avoid issues with the applications • Use Enterprise monitoring tools to alert on service failures, memory shortage, disk space, disk performance and CPU utilization. Below is an example on some statistics that can be monitored: Windows Server Resource Monitoring SAN Monitoring Network Monitoring Memory: % Committed Bytes In Use Memory: Page Faults/sec PhysicalDisk: Current Disk Queue Length PhysicalDisk: Disk Read Bytes/sec PhysicalDisk: Disk Reads/sec PhysicalDisk: Disk Write Bytes/sec PhysicalDisk: Disk Writes/sec Processor: % Idle Time Processor: Interrupts/sec System: Threads Storage Area Network (SAN) devices are now a standard infrastructure. They tend to be shared by many servers and can act as a performance bottleneck if not properly configured, low on internal caches, or many servers are using the same SAN port. Network switches and routers link all the servers and a failure or intermittent errors will cause connectivity issues. Automated alerts should be sent on high utilization and/or if key error conditions occur. Emtec, Inc. Proprietary & Confidential. All rights reserved 2012. 17
  18. 18. Monitoring your environment - Alerts • When setting up alerts on server resources, the following can be used as threshold: –Available Space on Drive: 25% or less - On Essbase servers, as much space as the largest cube –Available Memory: 10% or less – when the server runs out of physical memory, it uses virtual memory which is slower and impacts performance –CPU Utilization: 90% - review processes running and “stuck” applications. Review application logs for possible errors • When receiving alerts of application failures or service crash, archive the logs first, and then restart the service(s). Some logs are deleted during startup. Emtec, Inc. Proprietary & Confidential. All rights reserved 2012. 18
  19. 19. Monitoring your environment • On highly available environments using a load balancer, ensure the load balancer does health checks on the applications so that the load balancer does not route traffic to an unavailable server • On Virtual Environments, monitor performance statistics on the physical host(s) • Do not overcommit physical hosts with VMs. This severely impacts the performance and it is the most common reason for poor performance in virtual environments Emtec, Inc. Proprietary & Confidential. All rights reserved 2012. 19
  20. 20. Backups and Patches
  21. 21. Backups • Relational Database Backups –Incremental backups should occur daily with a full database backup on a weekly basis. –The Reporting and Analysis Database should be backed up in conjunction with its physical file repository: (Default Location) <Drive>OracleMiddlewareuser_projectsepmsystemxReportingAnalysisdat aRM1 • Application Server Backup –Daily incremental backups of the following with a full weekly backup: • System registry • Temporary directories • User profile directories • Application file system Emtec, Inc. Proprietary & Confidential. All rights reserved 2012. 21
  22. 22. Backups • Essbase Backups: –It is recommended to backup Essbase on a nightly basis –Essbase should be shut down prior to backup. If Essbase is running, some files will not be backed up –Below are the Essbase files that should be backed up: ARBORPATH = <DRIVE>:OracleMiddlewareuser_projectsepmsystemxEssbaseServeress baseserver1 File essxxxxx.ind* essxxxxx.pag* dbname.esm dbname.tct dbname.ind dbname.db x.lro essbase.sec essbase.bak essbase.cfg .otl .csc .rul .rep .eqd .sel Description Essbase index file Essbase data file Essbase Kernel file that contains control information used for database recovery Transaction control table Free fragment file for data and index free fragments Application file containing application settings Database file containing database settings Linked reporting objects Essbase security file Backup of the Essbase security file Essbase Server configuration file Database artifact files Emtec, Inc. Proprietary & Confidential. All rights reserved 2012. 22 Location ARBORPATH/app/appname/dbname/ ARBORPATH/app/appname/dbname/ ARBORPATH/app/appname/dbname/ ARBORPATH/app/appname/dbname/ ARBORPATH/app/appname/dbname/ ARBORPATH/app/ ARBORPATH/app/appname/dbname/ ARBORPATH/app/appname/dbname/ ARBORPATH/bin/ ARBORPATH/bin/ ARBORPATH/bin/ ARBORPATH/app/appname/dbname/
  23. 23. Patches • Develop a plan for periodic maintenance and patching • Search for patches on your licensed products periodically –HFM, Planning, Essbase, etc. –When searching for patches, include other components in the EPM suite: Foundation (Shared Services and Workspace), Reporting and Analysis, etc. • Apply patches only applicable to your environment –Patches should be applied to non-production environments first –Review the patch notes for compatible releases, OS, defect(s) fixed –Apply patches to client machines if required by the patch • Review patches for components such as Oracle HTTP Server and Java. OHS is a web server and it can be vulnerable Emtec, Inc. Proprietary & Confidential. All rights reserved 2012. 23
  24. 24. Patches • What to look for in a patch: –Look at the list of fixes and ensure the patch applies to your environment –Review the prerequisites – the most ignored section yet one of the most important –Apply OS Patches or patches to other components in the suite required by the patch you’re installing –Some patches require post installation activities, make sure to review them in the readme file. Emtec, Inc. Proprietary & Confidential. All rights reserved 2012. 24
  25. 25. Questions Emtec, Inc. Proprietary & Confidential. All rights reserved 2012. 25
  26. 26. THANK YOU FOR YOUR TIME Please visit us online at