This document summarizes a presentation about leveraging in-memory storage to overcome Oracle PGA memory limits. The presenter is a senior consultant with experience designing and implementing clustered and high availability Oracle solutions. They discuss how data volumes and processing power have increased while database designs have decreased over time. They cover Oracle's PGA memory structure and limits, including how manually and automatically managing work areas. The document also summarizes how using techniques like Linux tmpfs or ZFSSA can dramatically improve temporary I/O performance by 10x to 50x times for large queries that hit PGA limits.

1. Leveraging In-Memory Storage to
Overcome Oracle PGA Memory Limits
March 5, 2014
Presented by: Alex Fatkulin
Senior Consultant

2. Who am I ?
 Senior Technical Consultant at Enkitec
 12 years using Oracle
 Clustered and HA solutions
 Database Development and Design
 Technical Reviewer
 Blog at http://afatkulin.blogspot.com
3

3. Why This Presentation?
4

4. Data Growth and Processing
 Data Volumes UP
 Processing Power UP
 Database Design (in general) DOWN
5
Data Volume
Processing Power
Database Design
BetterWorse

5. 6
How many people have systems with
more than 128G of RAM?

6. “640K” Problems
7
 Processing patterns change
 Things that didn’t matter before matter now
 Legacy RDBMS code vs 64-bit systems
 KIWI (Kill It With Iron)

7. PGA Memory
8

8. PGA Memory (Dedicated Server)
9
Sort AreaHash Area Bitmap Area
SQL Work Areas
Session Memory Private SQL Area
Process Memory

9. Query Execution Work Areas
10
Work Area Size
Manual Management Auto Management
hash_area_size
sort_area_size
bitmap_merge_area_size
create_bitmap_area_size
pga_aggregate_target

10. PGA Memory Limits
11

11. Manual Work Area Management
12
SQL> alter session set workarea_size_policy=manual;
Session altered
SQL> alter session set hash_area_size=4294967296; -- 4GB
alter session set hash_area_size=4294967296
ORA-02017: integer value required
 11.2.0.4 Linux x64
SQL> alter session set hash_area_size=2147483648; -- 2GB
alter session set hash_area_size=2147483648
ORA-02017: integer value required
SQL> alter session set hash_area_size=2147483647; -- 2GB - 1
Session altered
 32-bit Signed Integer Limit

12. Auto Work Area Management
13
SQL> alter system set pga_aggregate_target=1024g; -- 1TB
System altered
 11.2.0.4 Linux x64
 Where is the catch?
 PGA_AGGREGATE_TARGET
 _smm_max_size/_smm_max_size_static
 Maximum work area size per process (px/serial)
 _smm_px_max_size/_smm_px_max_size_static
 Maximum work area size per query (px)
 _pga_max_size
 Maximum PGA size per process (px/serial)

13. Auto Work Area Management
14
 PGA_AGGREGATE_TARGET 16M – 512M
0
100
200
300
400
500
600
16 32 64 128 256 512
pga_aggregate_target _smm_max_size _smm_px_max_size _pga_max_size
 _pga_max_size = 200M
 _smm_max_size[_static] = 20%
 _smm_px_max_size[_static] = 50%

14. Auto Work Area Management
15
 PGA_AGGREGATE_TARGET 1G – 10G
 _pga_max_size = 20%
 _smm_max_size[_static] = 10%
 _smm_px_max_size[_static] = 50%
0
2000
4000
6000
8000
10000
12000
pga_aggregate_target _smm_max_size _smm_px_max_size _pga_max_size

15. Auto Work Area Management
16
 PGA_AGGREGATE_TARGET 10G – 16G
 _pga_max_size = 2G
 _smm_max_size[_static] = 2G
 _smm_px_max_size[_static] = 50%
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
10240 10752 11264 11776 12288 12800 13312 13824 14336 14848 15360 15872 16384
pga_aggregate_target _smm_max_size _smm_px_max_size _pga_max_size

16. Auto Work Area Management
17
 PGA_AGGREGATE_TARGET >= 10G
 _smm_max_size[_static] = 1G (maximum value)
 _pga_max_size = 2G (maximum value)
 If you’re bumping into per process limits further rising
pga_aggregate_target will not help

17. Per Process Limit
18
 PGA_AGGREGATE_TARGET = 192G / DOP = 16
 PGA_AGGREGATE_TARGET = 512G / DOP = 16

18. Per Process Limit
19
 PGA_AGGREGATE_TARGET = 192G / DOP = 16
 PGA_AGGREGATE_TARGET = 512G / DOP = 16
 _pga_max_size * DOP

19. Run With Higher DOP?
20
 Exadata X3-8
 2TB RAM (per node)
 80 CPU cores (per node)
 PGA_AGGREGATE_TARGET = 1536G
 Would require at least DOP 768 (_pga_max_size)
 9.6x core count
 Concurrency issues
 Manageability issues
 PX data distribution/algorithm issues

20. Run With Higher DOP?
21
 DOP vs CPU Cores Used
5 (spill) 8 (spill)
64 (no spill)
0
16
32
48
64
16 32 64
CPUCoresUsed
DOP

21. Run With Higher DOP?
22
 PX Algorithm Issues (ex.: median function)

22. Play With Underscores?
23
 MOS Doc ID 453540.1
 Allows _smm_max_size=2097151
 Allows _pga_max_size > 2GB with patch 17951233
 Can get 4G per process limit
 Can get > 4G per process limit
 /proc/sys/vm/max_map_count (OS page count)
 _realfree_heap_pagesize_hint (allocator page size)
 Weird behavior and likely not fully supported for work
areas (MOS Doc ID 453540.1)

23. Radically Improve TEMP I/O?
24
 TEMP I/O (in general)
 Doesn’t need redundancy
 Doesn’t need persistency
 Doesn’t need recoverability
 In-Memory FS (tmpfs, etc.)
 SAN/NAS LUN with write-back cache

24. Linux tmpfs (via loop device)
25
 10.1B rows / 416G HCC (QUERY HIGH)
SELECT /*+ parallel(t,16) */
CUST_ID, DATE_ID, COUNT(DISTINCT STORE_ID) D
FROM TRANS T
WHERE DATE_ID between to_date('01012012', 'ddmmyyyy') and to_date('31122013', 'ddmmyyyy')
GROUP BY CUST_ID, DATE_ID;
Exadata X3-8 TEMP
tmpfs TEMP
 13.8x faster TEMP I/O (237G)

25. Linux tmpfs (via loop device)
26
 10.1B rows / 416G HCC (QUERY HIGH)
SELECT /*+ parallel(t,16) */
CUST_ID, DATE_ID, COUNT(DISTINCT STORE_ID) D
FROM TRANS T
WHERE DATE_ID between to_date('01012012', 'ddmmyyyy') and to_date('31122013', 'ddmmyyyy')
GROUP BY CUST_ID, DATE_ID;

26. ZFSSA (Infiniband SRP)
27
 7.3B rows / 300G HCC (QUERY HIGH)
SELECT /*+ parallel(t,32) */
CUST_ID, DATE_ID, COUNT(DISTINCT STORE_ID) D
FROM TRANS T
GROUP BY CUST_ID, DATE_ID;
Exadata X3-8 TEMP
ZFSSA TEMP
(write-back cache)
 64.3x faster TEMP I/O (141G)

27. ZFSSA (Infiniband SRP)
28
 7.3B rows / 300G HCC (QUERY HIGH)
SELECT /*+ parallel(t,32) */
CUST_ID, DATE_ID, COUNT(DISTINCT STORE_ID) D
FROM TRANS T
GROUP BY CUST_ID, DATE_ID;

28. Summary
29
 Remember about per process limits
 Larger PGA_AGGREGATE_TARGET may do nothing
 Balance PGA/DOP/CPU
 It’s possible to increase TEMP I/O performance by 10x-
50x and more
 Oracle PGA code does not fully embrace 64-bit systems
at the moment

29. Q & A
Email: afatkulin@enkitec.com
Blog: http://afatkulin.blogspot.com
30