At SimCorp there are hundreds of databases running various types for workload spanning multiple platforms and RDBMS versions. In such a landscape and during a busy day it is impossible to keep up with all the databases. The AWR is a treasure chest of valuable instrumentation data but it is not very easy to get a good overview and identify issues without digging into the details of every single database. SimCorp decided to build an AWR data warehouse wherein the data could be aggregated and presented to a DBA in a consolidated fashion. Join this session to hear about the experiences gained at SimCorp by looking at AWR data from an consolidated, and aggregated view. Using SQLs and AWR views you will be shown how SimCorp saved many spinning CPUs and unnecessary TBs of I/O.

1. © 2019
USE AWR TO SAVE RESOURCES
DANIEL OVERBY HANSEN, LEAD DEVELOPER
FRANK VIDTFELDT LARSEN, LEAD DEVELOPER

2. © 20192
WELCOME
LET’S HAVE FUN

3. © 20193
BRINGING INTEGRATED SOLUTIONS TO INVESTMENT MANAGERS
THE WORLD’S LEADING PROVIDER
• Global provider of software and services to financial industry
• Established in 1971
• Headquartered in Copenhagen, Denmark
• Offices across Europe, North America and Asia-Pacific
• Flagship product, SimCorp Dimension®, built on-top of Oracle Database
• More than USD 20 trillion managed on SimCorp Dimension®

4. © 20194
A SHORT BIO
THE PRESENTERS
Frank Vidtfeldt Larsen
Lead Developer
frank-v-larsen
Daniel Overby Hansen
Lead Developer
dohdatabase
@dohdatabase

5. © 20195
WHAT’S IN IT FOR YOU
GOAL
1 Inspire you to use AWR proactively
2 Demonstrate how easy to setup an AWR warehouse
3 Show by example unnecessary resource consumption
4 Inspire you to save resources and license fees

6. © 20196
THE WAREHOUSE ”PROJECT”

7. © 20197
TABLE STRUCTURE
• One table to describe all installations, host, sid/service and installation name
• All DBA_HIST_xxx views are created as tables in the AWR warehouse in a compressed tablespace
• One table to store generic sql id and sql_text
• A few indices

8. © 20198
SIZE
• 160 in-house development installation
• 470 GB compressed
• AWR snapshot interval hourly
• 8-month of data
• Largest table 100GB (DBA_HIST_ACTIVE_SESS_HISTORY), however you could consider if the table is
relevant in the warehouse.

9. © 20199
REFRESH DATA PROCESS
• One package controls cleanup and daily apply of data.
• Data are collected via database links
• For DBA_HIST_xxx tables without snap_id columns we use “MERGE INTO”
• Load time is less than an hour for all installations.
• Any errors are logged in a logging table

10. © 201910
GENERIC SQL ID (1)
• Thanks to Carlos Sierra’s for sharing he’s function to create a unique sql_id from any text
https://carlos-sierra.net/2013/09/12/function-to-compute-sql_id-out-of-sql_text/
• Combined with the REPLACE function we can create unique sql_id and sql_text

11. © 201911
GENERIC SQL ID (2)

12. © 2019© 2019
HOW WE BECAME FRIENDS
WITH STORAGE ADMIN
12

13. © 201913
HOW WE BECAME FRIENDS WITH STORAGE ADMIN
CASE #1 – PHYSICAL I/O
1 Finding segments that produces most physical I/O
2 Using DBA_HIST_SEG_STAT as driver
3
Use the sum of PHYSICAL_READS_DELTA
and group by database and day

14. © 201914
HOW WE BECAME FRIENDS WITH STORAGE ADMIN
CASE #1 – PHYSICAL I/O

15. © 201915
HOW WE BECAME FRIENDS WITH STORAGE ADMIN
CASE #1 – PHYSICAL I/O
1 Top sinner: table SYS.SCHEDULER$_EVENT_LOG
2 Used by DBMS_SCHEDULER
3 Average daily I/O produced: 15-20 TB
4 Maximum daily I/O produced: 59 TB

16. © 201916
HOW WE BECAME FRIENDS WITH STORAGE ADMIN
CASE #1 – PHYSICAL I/O
TRUNCATE
TABLE

17. © 201917
HOW WE BECAME FRIENDS WITH STORAGE ADMIN
CASE #1 – PHYSICAL I/O

18. © 2019© 2019
A SPINNING CPU
IS A BEAUTIFUL CPU
18

19. © 201919
SQL TUNING TASK
CASE #2 – CPU
1 Finding SQLs that burn most CPU
2 Using DBA_HIST_SQLSTAT as driver
3
Use the sum of CPU_TIME
and group by day and SQL_ID

20. © 201920
SQL TUNING TASK
CASE #2 – CPU

21. © 201921
SQL TUNING TASK
CASE #2 – CPU

22. © 2019© 201922
Do we really want to use up to five CPUs during the
weekend to run SQL Tuning Task?
Or do we want to use those CPUs to e.g. generate better
statistics?
Or mine bitcoins?

23. © 201923
ENTERPRISE MANAGER
CASE #3 – CPU

24. © 201924
ENTERPRISE MANAGER
CASE #3 – CPU

25. © 201925
ENTERPRISE MANAGER
CASE #3 – CPU

26. © 201926
ENTERPRISE MANAGER
CASE #3 – CPU

27. © 201927
ENTERPRISE MANAGER
CASE #3 – CPU

28. © 2019© 201928
On average these two Enterprise Manager metrics
use one CPU - all day, everyday!
Do we really need Enterprise Manager to verify our
recovery strategy?
Even if we do – do we need to verify it 12 times an hour?

29. © 201929
REGEXP_LIKE BUG
CASE #4 – CPU

30. © 201930
REGEXP_LIKE BUG
CASE #4 – CPU

31. © 201931
REGEXP_LIKE BUG
CASE #4 – CPU

32. © 2019© 201932
On average this SQL was consuming more than two CPUs
all day, everyday!
By regularly applying Release Updates we eventually got
rid of this annoyance

33. © 201933
GOOD THING ORACLE DOESN’T LICENSE PER CPU … OR … WAIT-A-MINUTE
THANK YOU

34. © 2019© 2019
HOW WE BECAME FRIENDS WITH
THE BACKUP GUY
34

35. © 201935
WHAT ON EARTH IS GOING ON
CASE #5 – REDO LOG GENERATION

36. © 2019© 201936
Probably something unsual is going on…
This would (hopefully) have been discovered anyway –
but it can be used to find other anomalies

37. © 2019© 2019
HOT BLOCKS
37

38. © 201938
QUEUE
CASE #6 – HOT BLOCKS

39. © 201939
QUEUE
CASE #6 – HOT BLOCKS

40. © 201940
QUEUE
CASE #6 – HOT BLOCKS

41. © 2019© 201941
This is a development database.
Most likely they have no clue…
Let’s be a rock star DBA and have a friendly, and
educating talk over a cup of coffee

42. © 2019© 2019
FAMOUS LAST WORDS
42

43. © 201943
TIPS AND TRICKS – AND FAMOUS LAST WORDS
LESSONS LEARNED
1
2
3
Always aim for the lowest hanging fruits, i.e. the top consumers
Use your knowledge of the business
to sort out those that can’t be changed anyway
Apply patches regularly

44. © 201944
TIPS AND TRICKS – AND FAMOUS LAST WORDS
LESSONS LEARNED
5
6
For recursive SQLs it is useful to search MOS for the SQL_ID
The level of instrumentation in an Oracle database is incredible
AWR is a real treasure chest waiting to be found
4 Check for issues after configuration changes

45. © 2019© 2019
QUESTIONS?
45

46. © 2019
LEGAL NOTICE
The contents of this publication are for general information and illustrative purposes only and are used at the reader’s own risk. SimCorp uses all reasonable endeavors to ensure the accuracy of the information. However, SimCorp does not guarantee or warrant
the accuracy, completeness, factual correctness, or reliability of any information in this publication and does not accept liability for errors, omissions, inaccuracies, or typographical errors. The views and opinions expressed in this publication are not necessarily
those of SimCorp. © 2019 SimCorp A/S. All rights reserved. Without limiting rights under copyright, no part of this document may be reproduced, stored in, or introduced into a retrieval system, or transmitted in any form, by any means (electronic, mechanical,
photocopying, recording, or otherwise), or for any purpose without the express written permission of SimCorp A/S. SimCorp, the SimCorp logo, SimCorp ®, and SimCorp Services are either registered trademarks or trademarks of SimCorp A/S in Denmark and/or
other countries. Refer to www.simcorp.com/trademarks for a full list of SimCorp A/S trademarks. Other trademarks referred to in this document are the property of their respective owners.