Training Webinar: Detect Performance Bottlenecks of Applications

Server-side performance bottleneck
Webinar with Paulo Garrudo

Effective Platform Server Monitoring
Paulo Garrudo
Architecture Team Leader
Expert Services
paulo.garrudo@outsystems.com
https://www.linkedin.com/in/paulogarrudo
2

Server-side Performance bottlenecks
Agenda
● Performance Overview
● OutSystems Platform Logs
● Tools
● Performance CSI Live Demo
● TOP 10 Worst Practices
● Application Delivery Cycle
● Q&A

Web application Performance
What is
●Application is consider performant if consistently responds fast and
efficiently
●On web application, users expects a page to load under 2 seconds
●Measures the elapsed time between the beginning of a web request and
the end of the response

Web application Performance
Why is hard to achieve
●Developers have difficulty understanding why an application is slow
●It’s difficult to troubleshoot over web layers:
device → network → application server → database
●On server-side, it’s difficult to drill-down over a slow page
This webinar is going to focus on the Server-side performance

Web Request
Simple overview
●Request sent (network)
●Server-side
●Content download (network)
●Client side

Client side and network performance
Existing webinar
• Details for client side and network

How to measure
Use logs to measure elapsed times
●OutSystems Platform collects a set of logs out of the box
– By default, individual logs are kept for 4 weeks: configurable using the configuration tool
– For each eSpace / Extension, log setting must be enabled
●OutSystems Platform aggregates the collected logs
●On complex application actions, add you own audit actions to capture
server-side execution time

How to measure
OutSystems Platform monitoring
●General Logs
– Application specific logging
– System logs
– Slow queries: queries that take more than 200ms (configurable)
●ViewState size
●Session size
●Timer Logs: each executed Scheduled Job
●Integrations Log: each Web Service / Web Reference invocation

How to measure
OutSystems Platform monitoring
●Screens Log: each Screen access (regardless if Submit or Ajax)
●Extensions Log: keeps track of Extension actions execution
●Processes Log
– Monitors process execution
– Allows to drill down into each activity duration
●Emails Log: Includes an entry for each Email sent or received.
– Doesn’t measure execution time but it could be useful to check the size of the emails

Performance Monitor
● Platform built-in
● Captures End User Performance
● Allows to monitor history trends

Performance Monitor
● End User Experience = Client + Network + Server

Performance Monitor
● Client - time consumed for most used operating systems and browsers

Performance Monitor
● Network - time consumed for network connections and data carriers

Performance Monitor
● Server - slow database queries and slow integration response

Service Center
● Platform built-in
● Allows to check individual logs
● Useful to monitor specific execution times over one element
● During development, allows to evaluate optimizations in real time
● Information on ViewState and Session size is not available
● No easy use, lacks dashboards

Performance CSI
Forge component
• Lets you check the performance of your applications, and investigate the behavior of slow
screens to find the causes of poor performance

Performance CSI
Farm load and performance distribution

Performance CSI
Top screens evaluation
Weekly slow screen list:
● Number of hits (screen & ajax)
● Average Duration (screen & ajax)
● Total time = Hits * Avg. Duration
Allows to
● filter by application
● Sort screen by the metrics more
important for your application
● Drill-down to screen details
Very Slow > 6s
Slow ]4 to 6] s
Acceptable ]2 to 4] s
Fast [1 to 2] s
Very fast < 1s

Performance CSI
Top screens evaluation: Configuration
Configurable to:
● Ignore fast screens, to reduce the scope of the analysis to your performance target
● Ignore outliers: screens that take more than X seconds, so you can focus on the more usual
execution times

Performance CSI
Drill-down to screen performance

Performance CSI
Screen performance: dashboard
● Keep Session below 10 KB
● Keep ViewState below 5KB, to prevent client side latency

Performance CSI
Screen performance: aggregated metrics collapsed
Aggregated metrics by log type:
● Slow SQL
● Extension
● Web Reference
● User Defined Log Type (audit action ModuleName)
● Other: remaining execution time that cannot be aggregated

Screen performance: aggregated metrics expanded
Performance CSI

Performance CSI
Screen performance: Log View

Performance CSI
Tip: your own logs on performance CSI
Performance CSI automatically aggregates OutSystems Platform logs. If you use the
following pattern, your audit logs will also be aggregated:
● Use Ticks extension to measure the execution time
● Use the Audit action available on the system components
● Write audit message in the format: “.. took” + ExecTimeMS + “ms”
● Write execution time in ms, rounded to 0
● Use meaningful ModuleName, so it appears aggregated as a Log Type
Performance CSI adds the prefix User- when displaying user logs

Performance CSI
Tip: your logs on performance CSI

Performance CSI

Top 10 Worst Practices
1.Fetching more records than required
2.Inappropriate record counting
3.High reuse of queries/actions with same inputs
4.Inefficient data preparation
5.Abuse of inline parameters in advanced queries
6.Inefficient query filtering & sort
7.Abuse of scope data
8.Session Variables using large data types
9. Massive string transformations
10.One-shot load of heavy screens

#1 Fetching more records than required
Symptom
●Slow query with no top/maxrow treatment
Cause
●Unnecessary CPU and IO consumptions on the Database

Solution with Aggregates:
●Set the Max. records parameter of the aggregate to the required usage
●If required, the <query>.count will hold the total count of records, without
the Max. records limitation
In this example, Max. Records should be set to 1,
since only the first row is ever used.

Solution with Advanced Queries
1. Limit the number of Rows according to required usage
1. If total count is required, adapt a 2nd query from the 1st
SELECT column_names
FROM table_joins
WHERE conditions
ORDER BY order
SELECT * FROM (
SELECT column_names
FROM table_joins
WHERE conditions
ORDER BY order
) WHERE ROWNUM <= @maxrows
Oracle
SQL Server
SELECT TOP @maxrows column_names
FROM table_joins
WHERE conditions
ORDER BY order

#2 Inappropriate record counting
Symptom
●Slow <Query>.count
Cause
●Query used to fetch data is inefficient for performing a count
Solution
●Implement a simplified query that performs the count without fetching and
joining with unnecessary data
●Avoid execution of the query that collects the number of records
●Avoid testing <Query>.Count = 0 instead, use <Query>.List.Empty

Adapt a 2nd query from the original one
SELECT column_names
FROM table_joins
WHERE conditions
ORDER BY order_by_list
SELECT Count(1)
FROM Simplified_table_joins
WHERE conditions
1. Replace all the column
names by Count(1)
2. Remove any join that is not restricting
the number of rows and not contributing
for the conditions
3. Remove ORDER BY clause

Example
Lots of data fetching and computation,
not required for counting

Example (cont)
Most of this joins are only
necessary to access data
irrelevant for the counting

#3 High reuse of queries/actions with same inputs
Symptom
●High number of hits on same query/action
Cause
●Lack of caching (or bad logic flow)
Solution
●Cache the query or action for the minimum amount of time that will
value the memory used
! Always validate with business before add cache to your elements

Examples
Multiple calls with same inputs:
● During the day
→ cache for 24h
● In the scope of a user session
→ cache for 1h (or the average session duration)
● During the rendering of Web Block
→ cache for 1 minute

Example #1
Executed 15K times per day, in 0.7s per
execution, with the same inputs and retrieving
the same result, per Country in the same day
Caching for 60 minutes, would reduce
to a maximum of 24*Nr of countries
executions per day

Example #2
Repeated operations inside the same screen
Web block executed several times in same screen, causing
repeated calls to actions and queries with same inputs
Cache those calls inside the web block for a couple of minutes
to ensure that data is reused in same screen

Example #3
Cache for about 20 minutes to reduce the number of
slow hits given the contextual repetition of inputs
during that period
Reused in several
Web Blocks

Example #3
CacheVersion
20
Add a dummy input to change the
signature every time it’s necessary to
invalidate the cached value
●Invalidate your Query/Action/Web Block cache with the use of a dummy
input to change the signature

#4 Inefficient data preparation
Symptom
●Too many query calls
Cause
●Queries called inside foreach in the preparation
●Queries in expressions inside table records

Solutions
●Use proper joins and SQL functions
to gather all required data in a single query
●Make decisions in the preparation to get data with different queries
instead of making decisions on the screen to call different expressions with different queries
●Create different specialized screens
instead of a power screen with inefficient data preparation that needs to cope with all situation

Example Same conditions evaluated per column, and according to the result a
similar query is repeated individually per cell
Solution #1
Make the decision in the preparation to perform a different specialized query, joining with the
required tables and returning the same Output structure.
Solution #2:
Implement 2 specialized screens and make the decision to call the correct screen

#5 Abuse of inline parameters in advanced queries
Symptom
●Slow query with inline parameters
Cause
– Inline parameters with values that change too often generate too many
different queries in runtime, preventing DB optimization of the
execution plan and increasing SQL Server plan cache size

Solution
●Create specialized queries for the most used cases and make a decision on
which one to use, storing the result in a local variable
instead of injecting filtering or join clauses dynamically
●Don’t inject the result of a query in another one – create a complex query
instead
although more difficult to implement, it pays off performance wise
● Use advanced database database techniques, such as Materialized Views
(Oracle), indexed views (Sql Server), Global Temporary Tables (Oracle)
Ask your DBA for how to implement these

Example
In this example, GetInConditionsCounters computes two IN conditions that are injected in query
GetCounters to avoid complex sub-queries inside GetCounters
Although more complex to code, in terms of efficiency it is better to implement the sub-queries
and let the DB Engine optimize the entire query

This common pattern always results in performance issues once data
volume increases
where [Id] in @IdList
Solution
● Create a temporary table (using a WITH clause) with the list of id’s
WITH IdsTable as ( select … )
● Change the query to
where [Id] in (select id from IdsTable)
● Another advantage is not having to build the comma separated list
Inline parameter with comma separated Ids

Solution example
Upfront query to generate temporary
table myuserIDS
Sub-queries use the temporary table

#6 Inefficient Query Filtering & Sort
Symptom
●Slow query with filters and/or sort
Cause
●Casting of input parameters inside the query
●Filters with LIKE over text
●No indexes on the columns filtered or ordered
●Missing database maintenance plans

Solution: Optimize filters
●Avoid filters like WHERE name like '%Paul%'
●Make the casts outside the query, on the input expressions
–Casting of the input parameters inside the query causes conversion to
be done for every tuple tested

Solution: Index your entities
●Improves the performance of select (slight overhead in insert and update)
●Creating simple index for the most used entity attribute.
●Create compound indexes even if there is simple indexes on the same
fields – balance benefits between insertions and retrieval
●Indexes that you may consider adding:
– equality_columns: used in queries involving equality (ex: Id = 1 or Id != 1)
– included_columns: columns that should be included in the index (keeps a copy of the data at an
intermediate level of the index)
• Indexes with INCLUDE keyword are not supported in Service Studio… but can be created in the DB

Solution: Involve the resident DBA
●Database maintenance plan
– Periodically run index reorganization and statistics update
● Monitor existing indexes
– Unused indexes also impact performance negatively
– Costly used indexes: indexes hurting insert/update operations

#7 Abuse of scope data
Symptom
●Ajax requests become slower and page load keeps degenerating
Cause
●Any reference to preparation data requires data to be stored in the
viewstate, increasing IO in Screen actions
Solution
●Reload queries, re-execute actions, use screen action input parameters

Think viewstate
●Avoid using preparation data in screen actions:
– Increases network traffic between the server and the browser
#7 Abuse of scope data

#8 Session Variables using large data types
Symptom
●Increased contention on Session loading, with the number of concurrent
users
Cause
●Each web request loads session context which takes longer with large data
types, causing contention in the session data model with the increase of
concurrent users.
This is even more important when you use AJAX as it will increase the number of requests.

#8 Session Variables using large data types
Solution
●Store session information in database tables with the SessionId as the
primary key
Instead of fetching all session information in every request, only required data is fetched

Symptom
● Massive string transformation takes too long
Cause
● Processing logic using the standard string operations
#9 Massive string transformations

Solution
● Use the String Builder actions from Text Extension
Eg. Processing time reduced from 8 minutes to 30 seconds when serializing a list to a file (~9MB of
data)
#9 Massive string transformations

#10 One-shot load of heavy screens
Symptom
●First byte takes to long, lack of responsiveness, no partial information
Cause
●Power pages with too much content blocks, each requiring heavy data
retrieval and calculations
Solution
●Adopt a lazy load strategy

Check Lazy Load component in OutSystems Forge
Page structure is
quickly displayed…
… while heavy content
is progressively loaded
Screen preparation is faster because data preparation is made in each Web Block
Use caching mechanisms when repeating the same action/query on different Web Blocks
#10 One-shot load of heavy screens

How do I ensure my application is performant?
During development
●Follow OutSystems best practices one
https://success.outsystems.com/Support/Enterprise_Customers/Maintenance_and_Operations/Performance_Best_Practices
●Discuss complex business requirement within the team and define which
patterns should be applied
●Recurrently review the code
●Monitor the application
– Look for slow queries, slow screens

How do I ensure my application is performant?
During solution release
●Load & Stress tests
After Go-Live
●Proactively monitor your application
●Pay attention to performance decrease over time
– React before it becomes a big issue

Training Webinar: Detect Performance Bottlenecks of Applications

In this document

More Related Content

What's hot

Viewers also liked

Similar to Training Webinar: Detect Performance Bottlenecks of Applications

More from OutSystems

Recently uploaded

Training Webinar: Detect Performance Bottlenecks of Applications