Understanding NonStop SQLMX SDA and its impact on performance

Understanding DP2
SQLMXBuffer and its impact on
SQL/MX performance
Frans Jongma,
Hewlett Packard Enterprise
Advanced Technology Center
NonStop Enterprise Division
Version 2.0
Date: July 2017
Changes from 1.1: Modified to match new HPE report format
Abstract
The SQLMXBUFFER or SQL/MX Session Data Area (SDA) is a memory area in the disk
process that is used to store parts of query execution plans and session-specific data for
NonStop SQL/MX applications.
When adding more volumes to an application does not improve throughput, this may be caused
by a lack of SDA space.
SQL/MX releases before 3.3 used to share the SDA address space with the database data
cache: increasing one would decrease the available memory for the other. With the release of
SQL/MX 3.3 on RVUs J06.19 for NonStop Integrity servers and L15.02 for Integrity NonStop X
servers, the amount of space that can be used for the SDA has been increased significantly to 2
GB per volume and is no longer competing with memory used for data cache. The data cache
can now be as much as approx.1.4 GB per database volume.
This paper describes how the SDA is used and how its configuration affects performance. The
area is reserved on a session basis and query fragments of a session can only be reused by the
same session. It differs in this from the database data cache in DP2, which is shared by all
clients. Because the area is a shared resource and the amount of data is limited, it is advised to
track its usage and where possible, adjust the size of the area to the amount of sessions.

NonStop Advanced Technology Center
Understanding DP2 SQLMXBuffer and its impact on SQL/MX performance -2-
Introduction
Database administrators who need to tune NonStop SQL/MX database applications typically
know how to improve performance by adding more volumes to their database to increase
concurrency and allocate sufficient database cache to the data volumes to reduce physical IO.
The disk processes, called DP2 or Data Access Manager (DAM), are part of the SQL/MX
engine: part of the query processing actually takes place within these disk processes. With the
release of NonStop SQL/MX 3.3 this part of the SQL/MX engine, called the “Executor in DP2”
(EID) supports 64-bit memory address space and this has a big (positive) impact on the of
amount of query fragments that can be stored within DP2 as well as the amount of memory that
can be used for database cache.
This paper explains how SQL/MX in DP2 uses this memory and how this usage can be
monitored and tuned to improve concurrency and performance.
NonStop development has been improving DP2 memory usage in phases across multiple
releases and will continue this in future. This document is based on the features that are present
starting with NonStop OS Release J06.17. It will also explain the improvements that were added
in the J06.19 release for Integrity NonStop and L15.02 for Integrity NonStop X servers.
Intended audience
This paper is intended for DBAs and performance analysts with knowledge of the NonStop OS
and NonStop SQL/MX RDBMS.
Acknowledgement
Thanks to Jim Tate and Brian Vickery of NED Development for sharing their information and
their patience. Also thanks to those HP colleagues who took the time to review and comment.
What is the SQLMXBUFFER or SDA?
The SQLMXBUFFER, also known as SQL/MX Session Data Area (SDA), is a part of DP2 virtual
memory. In SQL/MX releases prior to 3.3, the SDA shares the address space, which is approx.
1.4 GB, with the DP2 cache area. The maximum size of the SDA in releases prior to 3.3 is 768
MB. SQL/MX Release 3.3 will increase the maximum SDA size to 2 GB and, because the SDA
is no longer sharing the cache area, a substantial increase of the data cache is possible1
.
A SQL/MX session is an execution context of a SQL statement. This context is established by
the Master Executor, the SQL library routines in a process that runs on the NonStop server,
which executes the statement.
1 In releases before J06.17, other DP2 data, such as lock tables, were sharing the cache area.

The size of the cache and the size of the SDA are configurable using the Subsystem Control
Facility (SCF). Figure 1 shows the current maximum amounts of memory that can be used per
DP2 volume.
Figure 1: Effect of SQL/MX 3.3 on available memory
The SDA contains the fragments of the SQL execution plans that are executed by the Executor
in DP2 (EID), as well as the memory that the EID needs to execute these fragments. A fragment
contains instructions, memory needed to communicate with the calling executor, an ESP or a
Master Executor, and pointers into the DP2 cache where the actual data resides. The amount of
memory that is needed for a fragment depends on the functions that the fragment needs to
perform, such as the checking of constraints, hashing and sorting results from a selection of
data from a partition and so on.
When a SQL query is compiled, the SQL compiler generates an execution plan which is
composed of multiple fragments. There are plan fragments for each process that is involved in a
query: the Master Executor, the application process that executes the query, any ESP used in
that query, and DP2 to access the data. When the query is executed, specific plan fragments
are sent from the Master Executor to each process involved in the query. For example, when a
table is partitioned, the fragment is sent to all DP2s that are needed to retrieve the result; this
could be a single DP2 for a read by primary key or a set of a DP2s for a full-table scan.
0
500
1000
1500
2000
2500
3000
3500
4000
SQL/MX 3.2.1 SQL/MX 3.3
MemoryMBperDP2
DP2 Usable memory increase for SQL/MX 3.3
SDA DP2 Cache

The first example below shows two fragments, a Master Executor fragment (operators 3 to 6)
and an EID fragment that contains the hash/groupby and the file-scan operators (1 and 2) that
execute within DP2. In a formatted explain listing, as shown below, the DP2 fragments can be
identified by partition_access (PA) operands. The operands below the PA nodes execute in
DP2.
>>explain options 'f' select region, avg(QTY_ORDERED), count (*) from order_example
group by region;
LC RC OP OPERATOR OPT DESCRIPTION CARD
---- ---- ---- -------------------- -------- -------------------- ---------
5 . 6 root 4.00E+000
4 . 5 hash_partial_groupby 4.00E+000
3 . 4 split_top 1:3(range) 4.00E+000
2 . 3 partition_access 4.00E+000
1 . 2 hash_partial_groupby 4.00E+000
. . 1 file_scan ORDER_EXAMPLE (s) 1.10E+001
--- SQL operation complete.
>>explain options 'f' select * from order_example where ordernum = 12345;
LC RC OP OPERATOR OPT DESCRIPTION CARD
---- ---- ---- -------------------- -------- -------------------- ---------
3 . 4 root 1.00E+000
2 . 3 split_top 1:3(range) 1.00E+000
1 . 2 partition_access 1.00E+000
. . 1 file_scan fr ORDER_EXAMPLE (s) 1.00E+000
--- SQL operation complete.
>>
In the first example, an EID fragment will be sent to all three DP2s of the three-way partitioned
ORDER_EXAMPLE table because all regions need to be examined in the query. In the second
example, it will be sent only to the DP2 that manages the partition that contains the primary key
value 12345.
References
The HP NonStop SQL/MX Release 3.x Query Guide contains detailed information about plan
fragments in Chapter 4, Reviewing Execution Plans and Chapter 8, Parallelism.

Re-use of plan fragments
When a query is executed the first time by a session, the plan fragment is sent to DP2, which
applies a fixup to change offsets in the plan to actual memory pointers specific to this query
instance.
If a program executes a compiled plan multiple times, it is not necessary to send the fragments
to DP2 each time, because the fragments are stored in the SDA and will be available for re-use.
A request is sent to DP2 with a unique plan ID. If it exists in the SDA, it will be re-used. If it does
not exist in the SDA, DP2 will request the plan fragment to be sent first. Because of the data
structures involved, re-use of the fragments is only possible by the database session which
originally sent it.
A session can re-use plan fragments only if plan fragments come from statements that are
prepared to be executed multiple times such as:
Embedded SQL from COBOL or C/C++ programs that is preprocessed by the SQL/MX
preprocessor;
SQL statements in Java and ODBC programs that are compiled as the result of the PREPARE
API call, and are executed multiple times. This includes plans fragments that are loaded from
modules in the Module File Cache (MFC);
SQL statements from the mxci command interpreter that use PREPARE and EXECUTE.
Concurrency in the SDA
If an application has been running for a while, all fragments used will be present in the SDA for
re-use and this is the desired state of an application since only user data will flow from
application to DP2 and back.
What determines SDA Usage
Every volume has an SDA. The amount of memory required for the SDA depends on three
things: the number of concurrent sessions that use the SDA, the number of tables or partitions
on a volume and the size of each fragment.
The number of concurrent sessions
Over time, an application may become more popular, and may be attracting more users. This
may lead to increasing response times due to queuing in the application. With NonStop servers,
the system will accommodate the growing number of users by automatically adding more
connections (in remote JDBC and ODBC applications) or more application server instances (in
Webserver, Pathway or TS/MP environments). However, when the application is scaled by
adding more connections or server instances, the SDA memory requirements will increase

proportionally because each session that is serving the same application will execute the same
SQL statements but will require its own set of DP2 plan fragments.
Figure 2: Increasing of sessions requires more SDA memory
When there are more sessions than fit in the SDA, fragments that are not currently in use by
SQL statements may be removed to make room for fragments from sessions that want to
execute a SQL statement. This is similar to how a computer system acts if there is not enough
physical memory to accommodate all processes: memory pages are stolen and need to be
reloaded when a process is rescheduled to use the processor. The reloading of plan fragments
happens when an executor wants to execute a statement and is requested by DP2 to resend
the fragment.
Note that server processes that execute statements at a low rate, still occupy SDA memory. It is
therefore better to have fewer processes that run at a high TPS rate than many processes that
execute only a few transaction per second and constantly need to resend their plan fragments.
The TPS rate per instance can be increased by concentrating the incoming transactions on
fewer server processes or by routing the transactions to specific processes based on key values
such that not every server process has to access every volume and this will increase the re-use
of fragments.
The number of partitions on a volume
DP2 fragments belong to a table, or rather a partition of a table. If a volume holds many tables
or table-partitions and these are frequently accessed by the application, the number of
fragments per session may become so large that fewer concurrent sessions will fit into the SDA.
This can be solved by moving partitions to other volumes, thus leaving fewer table partitions on
one volume.
Figure 3 shows how 10 tables can be evenly partitioned across four volumes and the alternative
where five tables are placed on two volumes and the other five on two other volumes. Note that
this is just an example; in real life systems, the ratio may be 40 vs. 20 volumes.
Session 1
fragment-1
fragment-2
Fragment-3
…
Session 2
fragment-1
fragment-2
Fragment-3
…
Session 3
fragment-1
fragment-2
Fragment-3
…
Session n
fragment-1
fragment-2
Fragment-3
…

Figure 3: Repartitioning tables reduces fragments needed per volume
DP2 fragments for table T1 through T10 will be present in the SDA of all four volumes in the
upper example, but only in the SDA of two volumes in the lower example. This means that more
sessions will fit in the alternative configuration without having to resend plan fragments.
Alternatively, this issue can be resolved if the disk is partitioned into multiple volumes. Disk
partitioning creates multiple logical volumes for a physical disk. Because SDA and DP2 cache
are assigned per DP2 volume, more database and fragment data can be placed in DP2 memory
by disk partitioning. Disk partitioning can be effective if there is enough memory in the system
and the amount of physical IO to the disk does not exceed the device limitations. Solid State
Drives (SSDs) have higher device limitations than traditional hard drives.
The size of the fragments
Obviously, the size of a fragment also determines how many will fit in the available space. The
size depends on the functionality that the fragment provides. This functionality is what the EID is
doing for this statement within the DP2 process. Examples of such functionality are the checking
of column constraints, (many columns in an insert or update statement will require more space)
and grouping, hashing, counting of rows within DP2, all of which requires memory. Looking at
the detailed information of an execution plan, will give an idea of the amount of work an EID
performs.
Monitoring SDA usage
In performance critical situations it may be useful to monitor the SDA usage, especially when
Measure indicates heavy DP2 message traffic.

The SDA is managed using the Subsystem Control Facility (SCF). SCF also has commands to
monitor the usage of the SDA, here called SQLMXBUFFER. SCF shows the current status of
the SDA and it totals some information since the last time the counters were reset.
References
Detailed information on the SCF commands can be found in the SCF Reference Manual for the
Storage Subsystem. A description of the attributes of SQLMXBUFFER can be found in the
SQL/MX Management Guide; Chapter 13, Managing SQL/MX Buffer Space.
This topic was also discussed in the presentation “Demystifying EID and SQLMXBuffer
Memory” by Brian Vickery, NonStop Technical Bootcamp 2014.
The example in Figure 4 shows the output of the “SCF STATS disk <volume>, sqlmx“-
command. The SDA-related information, in bold, includes the following information:
The amount of bytes allocated to the SDA for this volume. This amount is shown in bytes as
well as in blocks of 4 KB. This volume has 512 MB allocated for SDA space.
STORAGE - Stats DISK NSBLDP5.$SAS062
SQL/MX Statistics:
Session Data bytes....... 524288 KB Max Data bytes... 524288 KB
Total Sessions........... 1 Active Sessions.. 0
- 4KB Blocks - - - - - - - - - - - - Reuse - - - - - - - - - -
Max...................... 131072 Attempts......... 2
Number................... 131072 OK............... 2
In Use................... 11 Failed FST....... 0
Failed ID........ 0
Publish/Subscribe bytes.. 54765 KB
Max Subscribers.......... 512 Keys In Use...... 0
Subscribers Active....... 0 Resyncs.......... 0
Maximum Tables........... 512
Tables Active............ 0
Savepoint Buffer Data.... 0 KB Max.............. 32768 KB
- 16KB Blocks - - - - - - - - - - - - Failures - - - - - - - -
Max...................... 0 SavePoint Limit.. 0
In Use................... 0 Buffer Full...... 0
Memory Pressure.. 0
No Memory........ 0
Figure 4: Output of “SCF stats disk $SAS062, sqlmx” command

There are 11 blocks of 4KB in use for one session. In this example, this fragment has been
executed multiple times; two attempts to reuse the fragment and they were both successful
(OK).
The next example shows that the amount of SDA space that has been allocated is not enough
for the amount of fragments and required by the sessions that have been using the volume.
Figure 5: Example of frequent segment misses
At the time this snapshot was taken, there are no open sessions that use the SDA. However,
since the last time the statistics were reset, over 103,000 attempts to reuse fragments were
made, but most of them, 65,446 (shown as “Failed ID”), had to be resent from an Executor to
DP2. Failures to reuse can happen because a fragment is not located in the Free Space Table
(FST), which is not very common in current releases, or they exist in the FST, but cannot be
used, for example, because space has been stolen by DP2 for use by another session.
Using MEASURE
Measure does not explicitly report missing DP2 fragments, but it may show indirectly that
fragment stealing occurs, because resending fragments will show as increased message traffic
to DP2 from the application processes.
STORAGE - Stats DISK NSBLDP5.$SAS062
SQL/MX Statistics:
Session Data bytes....... 524288 KB Max Data bytes... 524288 KB
Total Sessions........... 0 Active Sessions.. 0
- 4KB Blocks - - - - - - - - - - - - Reuse - - - - - - - - - -
Max...................... 131072 Attempts......... 103390
Number................... 131072 OK............... 37944
In Use................... 0 Failed FST....... 0
Failed ID........ 65446
Publish/Subscribe bytes.. 54765 KB
Max Subscribers.......... 512 Keys In Use...... 0
Subscribers Active....... 0 Resyncs.......... 0
Maximum Tables........... 512
Tables Active............ 0
Savepoint Buffer Data.... 0 KB Max.............. 32768 KB
- 16KB Blocks - - - - - - - - - - - - Failures - - - - - - - -
Max...................... 0 SavePoint Limit.. 0
In Use................... 0 Buffer Full...... 0
Memory Pressure.. 0
No Memory........ 0

For example, in a JDBC type-4 benchmark executed at the ATC, we observed a single business
transaction sending 5 messages to the disk processes. In Measure terms: one MXOSRVR
instance shows 5 Messages-Sent for each Begin-Transaction. When the number of connections
of the application was increased, an increase in message traffic of 16% was observed when
SCF was reporting Failed Reuse attempts.
Increasing SDA space
If the SDA turns out not to be big enough and changes to the application or database are not
appropriate, the amount of memory to use for SDA can be increased.
Note:
Altering the size of the SDA is done through SCF and is an offline operation: the volume needs
to be in the stopped state when the value of SQLMXBUFFER is altered. The new value takes
effect after the volume has been restarted.
The amount of memory allocated to the SDA is managed by SCF. The following ALTER
command was used to set the SDA for volume $SAS062 to 512 MB.
SCF - T9082H01 - (23JUN11) (02MAY11) - 01/27/2015 08:33:43 System NSBLDP5
(C) 1986 Tandem (C) 2006 Hewlett Packard Development Company, L.P.
(Invoking NSBLDP5.$DATA06.FRANS.SCFCSTM)
1->ALTER DISK $SAS062, SQLMXBUFFER 512
The value entered as SQLMXBUFFER is the size (in MB) used as the SDA. NonStop OS
releases prior to J06.19 and release J06.19 with NonStop SQL/MX release 3.2.1 limit the size of
the SDA to 768 MB, still allowing for 600MB cache. Releases that use SQL/MX 3.3 or higher,
J06.19 and L15.02, increase this limit to 2048 (2 GB). And the total amount of cache can be up
to 1,400 MB regardless of the amount of SDA.
However, be sure to leave enough system memory for application processes!
SQLMXBUFFER related to Audittrailbuffer
Historically, the SDA and the TMF Audittrailbuffer shared the same space and used the same
configuration parameter for DP2. Starting with J06.19, configuring the SDA has an effect on the
space that is reserved for the Audittrailbuffer segment. Unlike the SDA for SQL/MX 3.3 and
higher, the Audittrrail buffer resides in 32-bit memory and it is present even if the volume is not
an audittrail volume. Previously, this was not an issue, because the space was used as SDA
when a volume was used a data volume. The actual size of this space in 32-bit memory will now
be determined as described in Table 1.

Table 1: Allocation of Audittrailbuffer space
Parameter Value specified in
SCF
Audittrailbuffer
allocated in DP2
Note
AUDITTRAILBUFFER 0 128 MB in 32-bit
memory
AUDITTRAILBUFFER Less than 768 As specified in 32-bit
memory
AUDITTRAILBUFFER Greater than 768 128 MB in 32-bit
memory
SQLMXBUFFER 0 128 MB in 64-bit
memory
128 MB reserved
in 32-bit memory
SQLMXBUFFER Less than 768 As specified in 32-bit
memory
SDA allocated in
64-bit memory
SQLMXBUFFER Greater than 768 128 MB in 32-bit
memory
SDA allocated in
64-bit memory
Unfortunately, this means that a minimum of 128MB cannot be used for DP2 cache which
currently also resides in 32-bit memory. Users should be careful when setting an SDA size
between 128 and 768 MB, because this effectively limits the amount of memory that can be
used for DP2 cache. The example SCF command to increase the SDA to 512 MB that was
shown earlier was a good thing to do in J06.18, but with SQL/MX release 3.3 where it allocates
512 MB in 64-bit memory, it also takes 512 MB out of DP2 cache and so in Release 3.3, it is
better to set it to be greater than 768 MB.
Note:
The first use of the Audittrailbuffer was by RDF, the Remote Database Facility which used this
area to quickly send data from one node to its backup node. As systems became faster and
more volumes were configured to use one audittrail volume, the Audittrailbuffer was used also
to efficiently write large amounts of data to the audittrail disk. As a result, the default size for this
area has been increased over time. Many customers automatically benefitted from this
increased default since they never explicitly configured the Audittrailbuffer size. To prevent
customer from taking a performance hit when they migrate to J06.19 the default (and minimum)
size was set to 128 MB.

Conclusion
NonStop SQL/MX 3.3 allows customers to use memory for their database applications. Placing
the SQL Session Data into 64 bit memory not only allows more concurrent sessions without
associated overhead but also allows more data to be placed in the DP2 cache.
This paper has described how SQL uses the SDA and how customers can configure both
database data and SDA to get optimal performance.

Understanding NonStop SQLMX SDA and its impact on performance

Recommended

Recommended

More Related Content

What's hot

What's hot (9)

Similar to Understanding NonStop SQLMX SDA and its impact on performance

Similar to Understanding NonStop SQLMX SDA and its impact on performance (20)

Recently uploaded

Recently uploaded (20)

Understanding NonStop SQLMX SDA and its impact on performance