Sybase ASE 15.7- Two Case Studies of Successful Migration

(c) 2015 Independent SAP Technical User GroupAnnual Conference, 2015
-ISUG TECH 2015-ISUG TECH 2015
ConferenceConference
ASE 15.7: 2 case studies of successful migrationASE 15.7: 2 case studies of successful migration
(shhhhh… preps floods & sc)(shhhhh… preps floods & sc)
,Andrew Melkonyan Senior DB Architect,Andrew Melkonyan Senior DB Architect

AgendaAgenda
Welcome
Speaker Introduction
 (Session Title add presentation
)title
 &Q A
2 [30]

WelcomeWelcome
 , , , , , -ISUG ASUG UKSUG SAUG TechEd D CODE
& .c
• Independent S ( )AP Technical User G ( . . )…roup www isug com
• A ’mericas SAP U ’sers G ( . . )…roup www asug com
• UK ( )and Europe S ( )AP Database and Technology products User
G ( . . )…roup www uksug com
• SAP Australian User G ( . . . )…roup www saug com au
• & - ( , , –SAP TechED d code technologists engineers developers
. .www sapteched co )…m

 , …Sybase is dead long live SAP Sybase
• …Most user groups are heavily SAP oriented
• / - - …I UK SUG leads the way with ex Sybase technologies

3 [30]

:Ecce homo:Ecce homo .Mr Andrew.Mr Andrew
MelkonyanMelkonyan
Over 15 years working with Sybase
( / )ASE RS
 … … … …Developer DBA Lead DBA Team Leader
…Consultant

 -Ex Ness Employee
 - …Official Sybase Products Re seller in Israel Responsible
( - )…for local case management via case express

Passionate for Sybase
 :// . . …http andrewmeph wordpress com
 :// . . / / . ...http scn sap com people andrew melkonyan
4 [30]

( . . ) . ( . &Safety Zone 12 5 x vs Futuristic Dreams 15 7
)…beyond
Migration to 15.7 – Motivation
15.7
T5-4
PRDR
IMDRRDR15.7 T5-4
3000+ TPS
PROD
REPIMDB
DRP
64 bit
MS2
M
’
’R
M R
ETL32 bit
12.5.4
M5
M51212
12.5.4 M5
1000 TPS
PROD
DRP
PRDR
32 bit
MS2
REPDRREP
MS1
5 [30]

Migration to 15.7 – 2 case studies
In the past 2 years I witnessed two reasonably large production ASE servers failing
to migrate from ASE 12.5.x to the pre ASE 15.7 releases of ASE 15:
One upgrade (Customer#1) was performed on an HP Integrity BL890c i2 host with 8 Intel(R)
Itanium(R) Processor 9340s (1.6 GHz), 32 logical processors (4 per socket), 256 GB RAM –
24 Engine ASE, HPUX.
The other upgrade (Customer#2) was performed on an M5000 server with 8 SPARC VII
Processors (2.4 GHz), 32 logical processors (4 per socket), 128 GB RAM – 16 Engine ASE,
Solaris.
Disproportionally high degree of engine utilization coupled with significant drop in
throughput turned each migration attempt into a failure.
6 [30]

Migration to 15.5 – Cust#1 Profile
ASE 12.5.3: Runs about 300 transactions per second with an average of 25% engine
utilization. The system runs about 600 procedure requests and 1700 statements per
second (Total Rows Affected: ~6.5K, Total Index Scans ~72K, Total Lock Requests
~300K, 1.4 M bytes received/sent per second).
Customer #1: an OLTP server (12.5.3) accessed by a mixture of clients – mostly Power Builder CTLIB software.
7 [30]

Migration to 15.5 – Cust#1
ASE 15.5: Run about 100 transactions per second with an average of 30% Engine
utilization. The system run ~600 procedure and ~500 statement requests per second.
No apparent problems with ASE configuration. No apparent reasons for slowing down.
TF753 did not help. Migration was aborted.
Early during migration ASE started to exhibit higher than expected engine utilization while the throughput sunk.
8 [30]

Migration to 15.5 – Cust#1, TSE
Sybase TS initial direction: procedure cache configured too small (12 GB for 24 Engine ASE) & statement cache configured too
large (2 GB). The two together were thought to result in high spinlock contention on ASE resources (SSQLCACHE and
RPROCMGR spinlocks).
9 [30]

Migration to 15.5 – Cust#2 Profile
# : ( . . ) – ,Customer 2 an OLTP server 12 5 4 accessed by a mixture of clients JDBC BDE and native
.CTLIB software
ASE 12.5.4: Runs about 900 transactions per second with an average of 40% engine utilization.
The system runs approximately 1400 procedure and 800 statement requests per second (Total Rows
Affected: ~15K, Total Index Scans ~65K, Total Lock Requests ~250K, 1 M bytes received/sent per
second).
10
[30]

Migration to 15.5 – Cust#2
Early during migration ASE started to exhibit extremely high engine utilization while the throughput sunk.
ASE 15.[0|5]: Run about 200 transactions per second with an average of 95% Engine utilization.
The system run ~650 procedure requests per second. Again, no apparent problems with ASE
configuration. No apparent reasons to slow down. TF753 did not help. Migration aborted – twice
(massive code review in-between).
11
[30]

Migration to 15.5 – Cust#2,TSE
Sybase TS initial direction: procedure cache fragmentation (4 GB for 16 engine ASE).
In the aftermath of work done for Customer #1 it has become clear that here too the
problem is around RPROCMGR (visible neither in regular sp_sysmon invocation nor
through MDAs – at that time).
12
[30]

Migration to 15.x – Aftermath
Following the migration, new simulators were written by customers teams in order to
reproduce failed migration better. None of the customers succeeded to reproduce
comparable throughput drop.
Only under extreme stress the same degree of spinlock contention or drop in
throughput started to surface.
Narrowing down the checks, it was found that what causes a high degree of
contention is running a high volume of prepared statements simultaneously.
13
[30]

Migration to 15.x – Aftermath
In order to deconstruct migration failures we had to:
1.Learn the difference in impact of prepared statement on ASE 12.5.x versus ASE 15.x correspondingly.
2.See what is so peculiar about these customers ASE environment from the prepared statement perspective.
3.Learn the right way to handle a high influx of prepared statement calls with the tools available at the DBMS side.
4.
14
[30]

To Prep or not to Prep, is it a question?
When application code uses prepared statement API both client host and DBMS server host
prepare internal memory structures designed for subsequent reuse. For ASE’s this structure is
a lightweight procedure – or LWP.
Depending on client connection settings ASE may receive and handle:
A.Fully prepared statements (sent via TDS_DYNP).
B.Partially prepared statements (sent via TDS_LANG).
The footprint of each on ASE is different. When these structures are created without the purpose
of being reused and sent en-masse to ASE they may cause substantial damage.
The distribution of prepared statement types may be inspected either with dbcc cis trace flags
or in monSysSQLText (DYNPs are identified as “DYNAMIC_SQL... CREATE PROC…”).
15
[30]

DYNP impact on ASE 15.x
Consider the table below:
We use three JDBC clients running (fully) prepared statements in a loop. Instead of reusing them in client
code, we just create them and drop right after being used once. All the 15.x versions prior to 15.7 produce
significant spinlock contention and the throughput drops. Only the 15.7 with the “streamline dynamic
SQL” option turned on fixes the issue(the last column: note the change in procedure removal and statement
reuse).
For ASE 12.5.x this was not an issue at all.
16
[30]

DYNPs are better Streamlined
Given the impact of fully prepared statements on ASE 15.x, the following
recommendations apply for application hitting ASE at high rate with fully prepared
statements:
If the application layer mustuse prepared statement semantics and there is a high volume of
fully prepared statements hitting ASE (generated and dropped at once rather than reused),
run ASE 15.7 ESD#1 or later and turn the “streamlined dynamic SQL” option on.
Earlier versions of ASE 15.x cannot handle high rate of fully prepared statements well. For
earlier ASE 15.x versions dynamic prepared option must be turned off on driver/connection
level (e.g. DYNAMIC_PREPARE = false for JDBC/ODBC client).
17
[30]

Non-DYNP impact on ASE 15.x
With the statement cache sized properly ASE 15.x handles load generated by partially
prepared statements much better than ASE 12.5.x.
The problem arises only when client code generates high volume of unique SQL
statements forced on the application layer into prepared statement API – again
generated and dropped rather than being reused. In this case, statement cache becomes
inefficient. Statements come in and out of statement cache at high rate causing high
statement cache turnover. Throughput goes down and engine utilization goes up.
In fact, in this case fully and partially prepared statements as well as regular callable
statements have the same negative impact.
18
[30]

Non-DYNP impact on ASE 15.x
Consider the table below:
We use three JDBC clients running (partially) prepared (or callable) statements in a loop – again with
no reuse: create->execute once->free. We generate unique SQL statements. Statement cache
turnover go up. ASE 15.5 may handle this situation only if the “bad” unique code is executed with
statement cache turned off. ASE 15.7 handles the situation gracefully.
For ASE 12.5.x this was not an issue at all.
19
[30]

Statement Cache Wonders
Statement cache management in 15.7 has been improved to handle reuse of statements much
more effectively. Still, flooding ASE with unique statements is risky:
If the application layer must use prepared statement logic or there is a high volume of unique
statements streamed into ASE (either due to bad coding or third party application components – e.g.
data-window), ASE 15.7 ESD#1 or later will in most cases deal with it gracefully. However: if the
statement cache will get over-flooded by statements contention will arise to the degree of making ASE
inoperable. In this case it is better to cut them off statement cache altogether.
Earlier ASE 15.x versions cannot handle situation unless this stream of unique code hitting the
statement cache is isolated on the connection level and the statement cache is turned off (e.g., set
statement_cache off in login trigger).
20
[30]

Cust#1: Migration Fiasco Revisited
Let’s inspect prepared statement situation in 12.5.3 for Customer#1.
We may see here that there is a medium rate of procedure requests. However, there is almost no procedure
removals. Client application here has little or no fully prepared statement calls (confirmed through
monSysSQLText inspection).
In addition, we see ~2000 statement requests per second.
21
[30]

We have experimented with statement cache size during simulation sessions and found that the greater the cache the lower is engine
utilization – all the way up to 2GB statement cache. So 2GB statement cache by itself did not constitute a problem here (compare
throughput below vs. migration data).
Cust#1: Simulation & SC size
22
[30]

Cust#1: Potential Issue
The simulation gave us a chance to inspect statement cache distribution. Cache buckets were found to contain 1 to ~2000 hashed
statements in very uneven distribution.
Below is a sample statement from a sc bucket:
What we see here is that this is the
same SQL which is slightly modified
by the PB client data-window
component.
23
[30]

Cust#1: Migration to 15.7
Customer#1 used a lot of unique statements. Versions of ASE prior to 15.7 could not handle this
influx with SC enabled.
For 15.7 Migration to succeed, we had to monitor SC influx rate closely. High rate of
statement influx is more detrimental to ASE 15.x than it has been back in the days of 12.5.4.
During the final migration tests it has been discovered that not only PB data-window component
generated a huge number of large unique statements that land in statement cache, but the same
code is run by different logins. Statement residing in SC may be reused ONLY if it is invoked
by the same login – unless a specific TF is applied to ASE.
The combination of sizing SC properly + controlling SC influx through forcing ASE to reuse
statements across different logins made the migration possible.
24
[30]

Cust#2: Migration Fiasco Revisited
Let’s inspect prepared statement situation in 12.5.4 for the Customer#2.
In contrast with Cust#1, Cust#2 has uses fully prepared statement API extensively. It has
also a large number of other statements hitting SC.
25
[30]

Cust#2: Migration to 15.7
15.7 provided solution for the Customer#2 as well. Streamlined dynamic SQL option
allowed fully prepared statements to be reused rather than discarded.
Here too, we had to monitor SC influx rate closely since high rate of SC turnover is
detrimental to ASE 15.x.
During the migration it has been discovered that there are certain applications that wake up
periodically and flood the SC, bringing about high spinlock contention on SC.
The combination of “streamlined” option and controlling SC influx tightly through
turning off statement cache access for particular logins made the migration possible.
26
[30]

Migration to 15.7 – A Happy Twin
Both customers have been successfully migrated to ASE 15.7
Customer#2*:
moved from 12.5.4 to 15.7 in July 2013 with a performance boost of over 100% in TPS. There were initial issues around ASE
Procedure Cache which were eventually solved using a combination of ASE settings, different access patterns to ASE Statement
Cache for different logins and custom PC monitoring scripts.
Customer#1**:
moved from 12.5.3 to 15.7 in March 2014. In addition to the pressure on Statement Cache generated by the high volume of large
statements, it was discovered that the situation was aggravated by the fact that ASE generates separate entries in SC for the
same statement run by different logins. Customer has over 1000 logins running the same code. Luckily there is a TF to loosen
SSQL-suid link.
* Migration orchestrated and performed under my lead.
** Migration has been performed with my personal involvement on-site.
27
[30]

ASE 15.7 Migration – an Aside
In order to prepare/analyze/monitor migration I’ve had to write quite a number of custom
monitoring applications.
You may find some of the tools @ https://andrewmeph.wordpress.com
28
[30]

Annual Conference, 2015 (c) 2015 Independent SAP Technical User Group
Questions and AnswersQuestions and Answers

Annual Conference, 2015 (c) 2015 Independent SAP Technical User Group
Thank You for AttendingThank You for Attending
Please complete yourPlease complete your
session feedback formsession feedback form

Sybase ASE 15.7- Two Case Studies of Successful Migration

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (11)

Similar to Sybase ASE 15.7- Two Case Studies of Successful Migration

Similar to Sybase ASE 15.7- Two Case Studies of Successful Migration (20)

More from SAP Technology

More from SAP Technology (20)

Recently uploaded

Recently uploaded (20)

Sybase ASE 15.7- Two Case Studies of Successful Migration