There are three levels of tracing in Vertica: Select, Session, and System. The Select level traces a single statement, Session traces all statements in a session, and System traces all queries across sessions. The trace output populates tables with query metadata and execution details. A long-running query was identified and traced to a costly GroupByHash operator. Creating a projection to presort the data enabled pipelining between operators, improving performance by 85%.

1. Some Tips about Vertica's Trace
I come from Oracle Performance/Tuning World. Oracle has some great utilities that I use for tracking
down problems and solving them. That's why when I approached Vertica I first asked how could I see
what was going on in the system and what causes the queries to be slower than usual? Well … not for
all the questions I have answers. However, I've studied the trace mechanism of Vertica and I'd like to
share my insights.
There are 3 levels of tracing in Vertica,
1. Select
2. Session
3. System
The trace output populates some of the 3 tables, depends on the level of tracing .
1. session_profiles – aggregate counter for the session and locks stats.
2. Query_profile – info about queries such as query start, projection used and sql text.
3. execution_engine_profiles- detailed info per operator like the clock time of a join operator.
Session & system have three levels of tracing
1. Session - populates ONLY session_profiles table.
2. Query – populates ONLY Query_profile table.
3. ee – populates both Query_profile and execution_engine_profiles.
Let’s start by explaining those levels:
Select - when we put “profile” before our select, it will trace only the select statement .
Then we will be able to analyze the results by querying
○ Query_profile
○ execution_engine_profiles
Example – profile select ‘testing’ from dual;
Session – when using ENABLE_PROFILING command will trace all the session statements.
The output will populate the tables based on the level of tracing ( session, query and ee).
Example - SELECT ENABLE_PROFILING(‘ee’) ;
System – when using the SET_CONFIG_PARAMETER command it will trace all queries from all sessions
the output will populate the tables based on the level of tracing ( session,query and ee).

2. Example - SELECT SET_CONFIG_PARAMETER(‘GlobalEEProfiling’, 1) ;
Some general notes:
● When we trace at a session or a system level
we will get a lot of rows in execution_engine_profiles table.
To clean those trace tables use the clear_profiling command.
Example -select clear_profiling(‘GlobalEEProfiling’, 'local');
local – will clear data from tables only for the current session .
global – will clear all sessions data from all nodes from tables.
● Even when tracing is not enabled, Active Queries will still show up in query_profiles and
execution_engine_profiles tables.
This can be used to monitor long running queries in real time.
● If we want to save the history of queries we can turn on the query repository .
It will save all queries + internal Vertica queries to a persistent table and not in the system
tables shown above.
According to Vertica support this would be deprecated in the future and data collector tables
will replace it.( this is for another article).
● execution_engine_profiles tables is a very detailed table.
It has about 30 distinct operator for each sql (if all were active)
It has 50 counters for each operator
It’s mostly useful for a single statement to see which phase in the plan took the longest time and
used the most resources.
At system level we can see which operator takes most of the system resources.
for example :

3. This query will show the top 10 most time consuming operators on your system.
(Remember when profiling is not enabled this will show only active queries)
select
*
from
(
SELECT
row_number () over (order by counter_valuedesc ) as rnum ,
decode(is_executing,true,'X',null) as run,
decode(is_executing,true,eep.session_id||'/'||eep.transaction_id||'/'||eep.statement_id,null) as "sid/
trx_id/stm_id",
eep.node_name,
ses.client_hostname,
eep.user_name,
operator_name,
path_id ,
counter_name,
counter_value/1000000 as sec
FROM v_monitor.execution_engine_profileseep left outer join v_monitor.sessionsses on ( ses.session_id
= eep.session_id)
WHERE (counter_name='execution time (us)' or counter_name = 'clock time (us)')
)
a
wherernum<= 10
ORDER BY rnum
● The steps that I use to identify a problem are:
○ Get an alert for long running query or identify a problematic query mainly from querying
the query_profile table
○ Drill down to the execution_engine_profiles table to see which step in the plan
consumed the most time and resources.
○ Run explain on the query.
○ Analyze the explain plan and tune the query.
Here is a simple situation describing the steps used to identify and fix a problem.
1. We got an alert for a long running query in the db
Here is a sample output from the query_profile table:
select
is_executing as exe,

4. qpo.transaction_id||' / '||qpo.statement_id as "trx_id/stm_id",
qpo.user_name,
qpo.schema_name||'.'||qpo.table_name as t_name,
substr(qpo.projections_used,1,30) as proj_use,
substr(replace(query,chr(10),' '),1,35) as query,
to_char(query_start,'dd/mm/yy hh24:mi:ss') query_start,
decode(is_executing,true,extract (seconds from (now()-query_start))+(extract (minutes from (now()-query_start))
*60)+(extract (hours from (now()-query_start))*60),query_duration_us/1000000) as Sec,
processed_row_count as rows,
error_code as err
from query_profiles qpo
where qpo.user_name ilike :1
order by is_executing desc ,query_duration_us desc ;
output:
exe | trx_id/stm_id | user_name | t_name | proj_use | query | query_start | Sec | rows | err
-----+-----------------------+-----------+--------+--------------------------------+-------------------------------------+-------------------+------------+------+-----
t | 45035996274377652 / 9 | dbadmin | . | lp_15744040.FACT_VISIT_ROOM_fi | select a11.LP_ACCOUNT_ID A | 16/04/12 09:17:31 | 185.417043 | 0 | 0
t | 45035996274380738 / 1 | dbadmin | . | v_monitor.query_profiles_p | select is_executing as exe, qpo.tra | 16/04/12 09:20:36 | 0.000409 | 0 | 0
2. Let’s drill down to the execution_engine_profiles table to check what consumes the most resources.
select "trx/stm",operator_name,path_id,
"execution time(sec)","clock time (sec)",
"estimated rows produced","rows produced",
"estimated rows produced"-"rows produced" as RowsDiff,
"memory reserved (MB)" ,
"memory allocated (MB)"
from (
select transaction_id||' / '||statement_id as "trx/stm",
operator_name,path_id,
sum(decode(counter_name,'execution time (us)',counter_value,null))/1000000 as "execution time(sec)",
sum(decode(counter_name,'clock time (us)',counter_value,null))/1000000 as "clock time (sec)",
sum(decode(counter_name,'estimated rows produced',counter_value,null)) as "estimated rows produced",
sum(decode(counter_name,'rows produced',counter_value,null)) as "rows produced",
sum(decode(counter_name,'memory reserved (bytes)',counter_value/1024/1024,null)) as "memory reserved
(MB)",
sum(decode(counter_name,'memory allocated (bytes)',counter_value/1024/1024,null)) as "memory allocated
(MB)"
from v_monitor.execution_engine_profiles
where transaction_id = <trx_id>
and statement_id = <stm_id>
and counter_value/1000000 > 0
and counter_name in ('execution time (us)','clock time (us)','estimated rows produced','rows produced','memory
reserved (bytes)','memory allocated (bytes)')
group by transaction_id||' / '||statement_id,operator_name,path_id ) a
order by 4 desc
;
Output:

5. trx/stm | operator_name | path_id | execution time(sec) | clock time (sec) | estimated rows produced | rows produced | RowsDiff | memory reserved (MB) | memory allocated
(MB)
-----------------------+---------------+---------+---------------------+------------------+-------------------------+---------------+-----------+----------------------+-----------------------
45035996274377652 / 2 | GroupByHash | 2| 115 | 57 | 100000000 | | | 1263 | 1123
45035996274377652 / 2 | Join | 3| 18 | 29 | 794821764 | 61278976 | 733542788 | 4| 80
45035996274377652 / 2 | StorageUnion | 2| 2| 6| | 61268471 | | 3| 1
45035996274377652 / 2 | GroupByPipe | 2| | | 400000000 | 61269239 | 338730761 | |
45035996274377652 / 2 | ExprEval | 3| | | 794821764 | 61278976 | 733542788 | |
45035996274377652 / 2 | StorageUnion | -1 | | | | | | 3|
45035996274377652 / 2 | Scan | 5| | | | | | | 1
45035996274377652 / 2 | Scan | 4| | | 794821764 | 61284352 | 733537412 | |
45035996274377652 / 2 | NewEENode | -1 | | | | | | 64 |
45035996274377652 / 2 | ExprEval | 0| | | 100000000 | | | |
45035996274377652 / 2 | GroupByPipe | 1| | | 100000000 | | | |
(11 rows)
sorry for the little fonts.
As we can see GroupByHash is taking the most resources and it’s path_id is 2.
3. Now, let’s run explain against the Bad query
explain select
a11.LP_ACCOUNT_ID AS LP_ACCOUNT_ID,
count(distinct a11.VS_LP_SESSION_ID) AS Visits,
(count(distinct a11.VS_LP_SESSION_ID) * 1.0) AS WJXBFS1
from lp_15744040.FACT_VISIT_ROOM a11
group by
a11.LP_ACCOUNT_ID;
Output:
Access Path:
+-GROUPBY PIPELINED [Cost: 7M, Rows: 10K] (PATH ID: 1)
| Aggregates: count(DISTINCT a11.VS_LP_SESSION_ID)
| Group By: a11.LP_ACCOUNT_ID
| +---> GROUPBY HASH (SORT OUTPUT) [Cost: 7M, Rows: 10K] (PATH ID: 2)
| | Group By: a11.LP_ACCOUNT_ID, a11.VS_LP_SESSION_ID
| | +---> STORAGE ACCESS for a11 [Cost: 5M, Rows: 199M] (PATH ID: 3)
| | | Projection: lp_15744040.FACT_VISIT_ROOM_fix
| | | Materialize: a11.LP_ACCOUNT_ID, a11.VS_LP_SESSION_ID
4. In the execution plan we can see that path_id 2
is the GROUPBY HASH (SORT OUTPUT) step in the plan.
The problem is that Vertica needs to wait for all rows to flow to this step until it can sort them and
grouped them.
As we can see it takes the most time and memory.
One way to speed up this step is to create a projection that it will presort the rows which will enable the
group by operation to pipeline the results to the next step.

6. For example we can create the following projection:
CREATE PROJECTION lp_15744040.FACT_VISIT_ROOM_fix1
(
LP_ACCOUNT_ID,
VS_LP_SESSION_ID,
VS_LP_VISITOR_ID,
VISIT_FROM_DT_TRUNC,
ACCOUNT_ID,
ROOM_ID,
VISIT_FROM_DT_ACTUAL,
VISIT_TO_DT_ACTUAL,
HOT_LEAD_IND
)
AS
SELECT FACT_VISIT_ROOM.LP_ACCOUNT_ID,
FACT_VISIT_ROOM.VS_LP_SESSION_ID,
FACT_VISIT_ROOM.VS_LP_VISITOR_ID,
FACT_VISIT_ROOM.VISIT_FROM_DT_TRUNC,
FACT_VISIT_ROOM.ACCOUNT_ID,
FACT_VISIT_ROOM.ROOM_ID,
FACT_VISIT_ROOM.VISIT_FROM_DT_ACTUAL,
FACT_VISIT_ROOM.VISIT_TO_DT_ACTUAL,
FACT_VISIT_ROOM.HOT_LEAD_IND
FROM lp_15744040.FACT_VISIT_ROOM
ORDER BY FACT_VISIT_ROOM.LP_ACCOUNT_ID,
FACT_VISIT_ROOM.VS_LP_SESSION_ID,
FACT_VISIT_ROOM.VS_LP_VISITOR_ID,
FACT_VISIT_ROOM.VISIT_FROM_DT_TRUNC,
FACT_VISIT_ROOM.ACCOUNT_ID,
FACT_VISIT_ROOM.ROOM_ID,
FACT_VISIT_ROOM.VISIT_FROM_DT_ACTUAL,
FACT_VISIT_ROOM.VISIT_TO_DT_ACTUAL,
FACT_VISIT_ROOM.HOT_LEAD_IND
SEGMENTED BY hash(FACT_VISIT_ROOM.VS_LP_SESSION_ID) ALL NODES ;
Now we run start_refresh() to make this projection valid and gather stats again.
We ran the explain again
explain select
a11.LP_ACCOUNT_ID AS LP_ACCOUNT_ID,
count(distinct a11.VS_LP_SESSION_ID) AS Visits,
(count(distinct a11.VS_LP_SESSION_ID) * 1.0) AS WJXBFS1
from lp_15744040.FACT_VISIT_ROOM a11

7. group by
a11.LP_ACCOUNT_ID;
Access Path:
+-GROUPBY PIPELINED [Cost: 7M, Rows: 10K] (PATH ID: 1)
| Aggregates: count(DISTINCT a11.VS_LP_SESSION_ID)
| Group By: a11.LP_ACCOUNT_ID
| +---> GROUPBY PIPELINED [Cost: 7M, Rows: 10K] (PATH ID: 2)
| | Group By: a11.LP_ACCOUNT_ID, a11.VS_LP_SESSION_ID
| | +---> STORAGE ACCESS for a11 [Cost: 5M, Rows: 199M] (PATH ID: 3)
| | | Projection: lp_15744040.FACT_VISIT_ROOM_fix1
| | | Materialize: a11.LP_ACCOUNT_ID, a11.VS_LP_SESSION_ID
Now path_id 2 is not sorting and it’s doing GROUPBY PIPELINED.
This is what we wanted .
Vertica will group bulk of rows and send them to the next step.
Here are the results:
Before:
Time: First fetch (7 rows): 247037.106 ms. All rows formatted: 247037.177 ms
After:
Time: First fetch (7 rows): 34855.253 ms. All rows formatted: 34855.299 ms
85% decrease in elapsed time.