SlideShare a Scribd company logo
1 of 43
Download to read offline
Konstantine Krutiy
Principal Engineer, Crew Lead
PATH TO EXTRA PERFORMANCE
Eliminate unneeded work
§ Choose data types wisely
Eliminate unneeded waits
§ Reduce number of locks
Make system operate in more efficient way
§ Optimize BIOS settings
§ Stay in same “technology slice”
§ Make sure you have enough RAM
ART OF CHOOSING DATATYPES
WHY DATA TYPE MATTERS ?
WHY DATA TYPE MATTERS ?
Fastest CPU today is 3.7 GHz
It takes
1 / 3,700,000,000 of a second
to do single operation
WHY DATA TYPE MATTERS ?
Fastest CPU today is 3.7 GHz
It takes
1 / 3,700,000,000 of a second
to do single operation
“BIG DATA” record set
starts from
100 billion records
WHY DATA TYPE MATTERS ?
Fastest CPU today is 3.7 GHz
It takes
1 / 3,700,000,000 of a second
to do single operation
“BIG DATA” record set
starts from
100 billion records
Processing time
1 / 3,700,000,000 sec X 100,000,000,000 = 27 sec
DO YOU NEED TO STORE DATA SAME
WAY IT IS PRESENTED ?
DO YOU NEED TO STORE DATA SAME
WAY IT IS PRESENTED ?
Presentation: $395.17
DO YOU NEED TO STORE DATA SAME
WAY IT IS PRESENTED ?
Presentation: $395.17
Data: 395.17
DO YOU NEED TO STORE DATA SAME
WAY IT IS PRESENTED ?
Presentation: $395.17
Data: 395.17
Storage: Store as Money
Data type: MONEY
Internal data type: NUMERIC(18,4)
Storage: Store as numeric
Data type: NUMERIC
Internal data type: NUMERIC(37,15)
Storage: Store as integer
Data type: INT
Internal data type: INT
DO YOU NEED TO STORE DATA SAME
WAY IT IS PRESENTED ?
Presentation: $395.17
Data: 395.17
Storage: Store as Money
Data type: MONEY
Internal data type: NUMERIC(18,4)
Storage: Store as numeric
Data type: NUMERIC
Internal data type: NUMERIC(37,15)
Storage: Store as integer
Data type: INT
Internal data type: INT
DATA TYPE BENCHMARK DATA
DATA TYPE BENCHMARK AVERAGES IN SEC
27.2
29.7
37
0
5
10
15
20
25
30
35
40
INT NUMERIC(18,5) NUMERIC(37,15)
MAKING RIGHT CHOICES
• If you can store data as INTEGER
• Choose INTEGER
• If your data fits into 18 digits of PRECISION
• Choose NUMERIC(18)
• If your data larger then 18 digits of PRECISION
• Choose NUMERIC(your-desired-precision)
Vertica default for NUMERIC is NUMERIC(37,15)
ELIMINATING UNNECESSARY LOCKING
LOCKING BEHAVIOR
AUTOCOMMIT = ON (jdbc driver default)
§ Each statement treated as complete transaction
§ When statement completes changes automatically
committed to database
AUTOCOMMIT = OFF
§ Transaction continue until manually run COMMIT or
ROLLBACK
§ Locks kept on objects for transaction duration
CONTROLLING AUTOCOMMIT STATE
JAVA:
conn = DriverManager.getConnection("jdbc:vertica://DBHost:5433/MyDB", myProperties);
// get the state of the auto commit parameter
System.out.println("Autocommit state: " + conn.getAutoCommit());
// Change the auto commit state to false
conn.setAutoCommit(false);
SQL:
IMPACT ON LOCK COUNTS BY CHANGING
AUTOCOMMIT SETTING TO OFF
HOW TO DISABLE – OBVIOUS METHOD
HOW TO DISABLE – BETTER METHOD
BIOS SETTINGS OPTIMIZATIONS
WHAT IS TUNABLE IN BIOS?
HOW TO TUNE ?
http://h10032.www1.hp.com/ctg/Manual/c01804533.pdf
DOES IT REALLY MATTER ?
0
100
200
300
400
500
600
700
800
900
1000
DSS BIOS settings with 1x
DRAM refresh rate
DSS BIOS settings with 4x
DRAM refresh rate
HPC BIOS settings with 4x
DRAM refresh rate
HPC + HyperThreading BIOS
settings with 4x DRAM
refresh rate
HPC - NO TurboBoost BIOS
settings with 4x DRAM
refresh rate
Sec
DSS	
  BIOS	
  se)ngs	
  with	
  1x	
  DRAM	
  refresh	
  rate	
   738.949439	
  
DSS	
  BIOS	
  se)ngs	
  with	
  4x	
  DRAM	
  refresh	
  rate	
   745.111176	
  
HPC	
  BIOS	
  se)ngs	
  with	
  4x	
  DRAM	
  refresh	
  rate	
   552.148285	
  
HPC	
  +	
  HyperThreading	
  BIOS	
  se)ngs	
  with	
  4x	
  DRAM	
  refresh	
  rate	
   877.838469	
  
HPC	
  -­‐	
  NO	
  TurboBoost	
  BIOS	
  se)ngs	
  with	
  4x	
  DRAM	
  refresh	
  rate	
   561.260084	
  
Performance
increase potential
about 40%
WHAT TUNING DOC SAYS ?
STAYING IN THE SAME “TECHNOLOGY SLICE”
WHAT I WILL BE SLICING THROUGH ???
CPU and chipset
Hardware
Operating System (OS)
Database Management System (DBMS)
WHAT IS “TECHNOLOGY SLICE” ANYWAY ???
CPU Gen3 CPU Gen4
Server Gen-B
OS v. 36
DBMS v. 6
Server Gen-C
OS v. 37
DBMS v. 7
CPU Gen5
Server
Gen-D
CPU Gen6 CPU Gen7
Server Gen-E
Srv
Gen
F
OS v. 38
Server
Gen-A
OS v. 35OS v. 34
DBMS v. 5DBMS v. 4DBMS v. 3
WHAT IS “TECHNOLOGY SLICE” ANYWAY ???
CPU Gen3 CPU Gen4
Server Gen-B
OS v. 36
DBMS v. 6
Server Gen-C
OS v. 37
DBMS v. 7
CPU Gen5
Server
Gen-D
CPU Gen6 CPU Gen7
Server Gen-E
Srv
Gen
F
OS v. 38
Server
Gen-A
OS v. 35OS v. 34
DBMS v. 5DBMS v. 4DBMS v. 3
COMMON “TECHNOLOGY SLICE” TRAP
CPU Gen3 CPU Gen4
✔
Server Gen-B
OS v. 36
DBMS v. 6
Server Gen-C
✔
OS v. 37
✔
DBMS v. 7
✔
CPU Gen5
Server
Gen-D
CPU Gen6 CPU Gen7
Server Gen-E
Srv
Gen
F
OS v. 38
Server
Gen-A
OS v. 35OS v. 34
DBMS v. 5DBMS v. 4DBMS v. 3
COMMON “TECHNOLOGY SLICE” TRAP
CPU Gen3 CPU Gen4
✔
Server Gen-B
OS v. 36
DBMS v. 6
Server Gen-C
✔
OS v. 37
✔
DBMS v. 7
✔
CPU Gen5
Server
Gen-D
CPU Gen6 CPU Gen7
Server Gen-E
Srv
Gen
F
OS v. 38
Server
Gen-A
OS v. 35OS v. 34
DBMS v. 5DBMS v. 4DBMS v. 3
?
?
SYMPTOMS OF “TECHNOLOGY SLICE” ISSUES
System AVG: 57.90
Nice AVG: 46.56
System AVG > Nice AVG
System AVG / Nice AVG = 1.24
System AVG: 11.19
Nice AVG: 57.38
System AVG < Nice AVG
System AVG / Nice AVG = 0.19
“TECHNOLOGY SLICE” PERFORMANCE IMPACT
0
20
40
60
80
100
120
140
different “TECHNOLOGY SLICE” kernel proper “TECHNOLOGY SLICE” kernel
Sec
SUFFICIENT RAM CALCULATIONS
DO I REALLY NEED MORE RAM ?
select event_type, count(1) from query_events group by event_type order by 2 desc;
Spilled events are very good
indication of queries not fitting in
RAM
HOW I CAN QUANTIFY IMPACT ?
select 'event_timestamp' as timestamp_type,
min(event_timestamp) as min_timestamp,
max(event_timestamp) as max_timestamp from query_events
union
select 'query_timestamp' as timestamp_type,
min(start_timestamp) as min_timestamp,
max(start_timestamp) as max_timestamp from query_requests;
System tables in Vertica have
individual rolling window. Make
sure you understand relation of
histories available.
HOW I CAN QUANTIFY IMPACT ? CONT.
select spilled_queries, total_qieries, round( spilled_queries / total_qieries * 100 , 2 ) as spilled_queries_percent
from
(select count(1) as total_qieries from query_requests
where request_type = 'QUERY' and start_timestamp > (select min(event_timestamp) from query_events)) query_data,
(select count(1) as spilled_queries
from (select session_id, transaction_id, statement_id from query_events
where event_type ilike '%SPILLED%' group by session_id, transaction_id, statement_id) spill_data) spill_data2;
Amount of spilled queries in
relation to entire query volume.
CAN MY SPILLED DATA FIT IN TO RAM ?
select min(counter_value) as min_bytes_spilled,
max(counter_value) as max_bytes_spilled,
avg(counter_value) as avg_bytes_spilled
from execution_engine_profiles
where counter_name = 'bytes spilled' and counter_value > 0;
Understanding size of
spillage to disk.
WHO CAUSING SPILLS ?
select user_name, count(1) as spill_event_count
from query_events where event_type ilike '%SPILLED%' group by user_name order by 2 desc;
In Vertica RAM allocated to queries
through resource pools. Resource
pools connected to users. Knowing
user will point us to resource pool,
which needs tuning.
WHAT I SHOULD TUNE ?
select distinct resource_pool from users where user_name in ('peter', 'john');
Identified resource pool with
spilled queries. Now we know
what to tune.
The resource pool parameters of
MEMORYSIZE and
PLANNEDCONCURRENCY provide the
options that let you tune the target
memory allocated to queries.
WHAT I SHOULD CHANGE ?
HP Vertica Analytics Platform Version 7.1.x Documentation
Administrator's Guide
Managing the Database
Managing Workloads
Resource Pool Architecture
Target Memory Determination for Queries in Concurrent Environments
Q & A

More Related Content

What's hot

Profiling the logwriter and database writer
Profiling the logwriter and database writerProfiling the logwriter and database writer
Profiling the logwriter and database writerKyle Hailey
 
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel PoderTroubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel PoderTanel Poder
 
OOUG: Oracle transaction locking
OOUG: Oracle transaction lockingOOUG: Oracle transaction locking
OOUG: Oracle transaction lockingKyle Hailey
 
How Prometheus Store the Data
How Prometheus Store the DataHow Prometheus Store the Data
How Prometheus Store the DataHao Chen
 
Sessionization with Spark streaming
Sessionization with Spark streamingSessionization with Spark streaming
Sessionization with Spark streamingRamūnas Urbonas
 
High Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseHigh Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseAltinity Ltd
 
UKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction LocksUKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction LocksKyle Hailey
 
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEOClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEOAltinity Ltd
 
ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...
ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...
ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...Altinity Ltd
 
How to Avoid Pitfalls in Schema Upgrade with Galera
How to Avoid Pitfalls in Schema Upgrade with GaleraHow to Avoid Pitfalls in Schema Upgrade with Galera
How to Avoid Pitfalls in Schema Upgrade with GaleraSveta Smirnova
 
Your first ClickHouse data warehouse
Your first ClickHouse data warehouseYour first ClickHouse data warehouse
Your first ClickHouse data warehouseAltinity Ltd
 
PostgreSQL Meetup Berlin at Zalando HQ
PostgreSQL Meetup Berlin at Zalando HQPostgreSQL Meetup Berlin at Zalando HQ
PostgreSQL Meetup Berlin at Zalando HQPostgreSQL-Consulting
 
Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020
Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020
Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020Altinity Ltd
 
InfluxDB IOx Tech Talks: The Impossible Dream: Easy-to-Use, Super Fast Softw...
InfluxDB IOx Tech Talks: The Impossible Dream:  Easy-to-Use, Super Fast Softw...InfluxDB IOx Tech Talks: The Impossible Dream:  Easy-to-Use, Super Fast Softw...
InfluxDB IOx Tech Talks: The Impossible Dream: Easy-to-Use, Super Fast Softw...InfluxData
 
Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEO
Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEODangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEO
Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEOAltinity Ltd
 
Oracle: Binding versus caging
Oracle: Binding versus cagingOracle: Binding versus caging
Oracle: Binding versus cagingBertrandDrouvot
 
Advanced Apache Cassandra Operations with JMX
Advanced Apache Cassandra Operations with JMXAdvanced Apache Cassandra Operations with JMX
Advanced Apache Cassandra Operations with JMXzznate
 
Using Apache Spark and MySQL for Data Analysis
Using Apache Spark and MySQL for Data AnalysisUsing Apache Spark and MySQL for Data Analysis
Using Apache Spark and MySQL for Data AnalysisSveta Smirnova
 

What's hot (20)

Profiling the logwriter and database writer
Profiling the logwriter and database writerProfiling the logwriter and database writer
Profiling the logwriter and database writer
 
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel PoderTroubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
 
OOUG: Oracle transaction locking
OOUG: Oracle transaction lockingOOUG: Oracle transaction locking
OOUG: Oracle transaction locking
 
How Prometheus Store the Data
How Prometheus Store the DataHow Prometheus Store the Data
How Prometheus Store the Data
 
PostgreSQL and RAM usage
PostgreSQL and RAM usagePostgreSQL and RAM usage
PostgreSQL and RAM usage
 
Sessionization with Spark streaming
Sessionization with Spark streamingSessionization with Spark streaming
Sessionization with Spark streaming
 
Rac 12c optimization
Rac 12c optimizationRac 12c optimization
Rac 12c optimization
 
High Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseHigh Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouse
 
UKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction LocksUKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction Locks
 
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEOClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
 
ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...
ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...
ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...
 
How to Avoid Pitfalls in Schema Upgrade with Galera
How to Avoid Pitfalls in Schema Upgrade with GaleraHow to Avoid Pitfalls in Schema Upgrade with Galera
How to Avoid Pitfalls in Schema Upgrade with Galera
 
Your first ClickHouse data warehouse
Your first ClickHouse data warehouseYour first ClickHouse data warehouse
Your first ClickHouse data warehouse
 
PostgreSQL Meetup Berlin at Zalando HQ
PostgreSQL Meetup Berlin at Zalando HQPostgreSQL Meetup Berlin at Zalando HQ
PostgreSQL Meetup Berlin at Zalando HQ
 
Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020
Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020
Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020
 
InfluxDB IOx Tech Talks: The Impossible Dream: Easy-to-Use, Super Fast Softw...
InfluxDB IOx Tech Talks: The Impossible Dream:  Easy-to-Use, Super Fast Softw...InfluxDB IOx Tech Talks: The Impossible Dream:  Easy-to-Use, Super Fast Softw...
InfluxDB IOx Tech Talks: The Impossible Dream: Easy-to-Use, Super Fast Softw...
 
Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEO
Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEODangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEO
Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEO
 
Oracle: Binding versus caging
Oracle: Binding versus cagingOracle: Binding versus caging
Oracle: Binding versus caging
 
Advanced Apache Cassandra Operations with JMX
Advanced Apache Cassandra Operations with JMXAdvanced Apache Cassandra Operations with JMX
Advanced Apache Cassandra Operations with JMX
 
Using Apache Spark and MySQL for Data Analysis
Using Apache Spark and MySQL for Data AnalysisUsing Apache Spark and MySQL for Data Analysis
Using Apache Spark and MySQL for Data Analysis
 

Similar to Extra performance out of thin air

The hidden engineering behind machine learning products at Helixa
The hidden engineering behind machine learning products at HelixaThe hidden engineering behind machine learning products at Helixa
The hidden engineering behind machine learning products at HelixaAlluxio, Inc.
 
Oracle Basics and Architecture
Oracle Basics and ArchitectureOracle Basics and Architecture
Oracle Basics and ArchitectureSidney Chen
 
Entenda de onde vem toda a potência do Intel® Xeon Phi™
Entenda de onde vem toda a potência do Intel® Xeon Phi™ Entenda de onde vem toda a potência do Intel® Xeon Phi™
Entenda de onde vem toda a potência do Intel® Xeon Phi™ Intel Software Brasil
 
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New FeaturesAmazon Web Services
 
Building an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open SourceBuilding an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open SourceAltinity Ltd
 
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptxBuilding an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptxAltinity Ltd
 
Barga IC2E & IoTDI'16 Keynote
Barga IC2E & IoTDI'16 KeynoteBarga IC2E & IoTDI'16 Keynote
Barga IC2E & IoTDI'16 KeynoteRoger Barga
 
Cassandra Tuning - above and beyond
Cassandra Tuning - above and beyondCassandra Tuning - above and beyond
Cassandra Tuning - above and beyondMatija Gobec
 
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...DataStax
 
The post release technologies of Crysis 3 (Slides Only) - Stewart Needham
The post release technologies of Crysis 3 (Slides Only) - Stewart NeedhamThe post release technologies of Crysis 3 (Slides Only) - Stewart Needham
The post release technologies of Crysis 3 (Slides Only) - Stewart NeedhamStewart Needham
 
Monitoring InfluxEnterprise
Monitoring InfluxEnterpriseMonitoring InfluxEnterprise
Monitoring InfluxEnterpriseInfluxData
 
Performance and how to measure it - ProgSCon London 2016
Performance and how to measure it - ProgSCon London 2016Performance and how to measure it - ProgSCon London 2016
Performance and how to measure it - ProgSCon London 2016Matt Warren
 
Column store indexes and batch processing mode (nx power lite)
Column store indexes and batch processing mode (nx power lite)Column store indexes and batch processing mode (nx power lite)
Column store indexes and batch processing mode (nx power lite)Chris Adkin
 
CMP301_Deep Dive on Amazon EC2 Instances
CMP301_Deep Dive on Amazon EC2 InstancesCMP301_Deep Dive on Amazon EC2 Instances
CMP301_Deep Dive on Amazon EC2 InstancesAmazon Web Services
 
Migrating To PostgreSQL
Migrating To PostgreSQLMigrating To PostgreSQL
Migrating To PostgreSQLGrant Fritchey
 
Intelligent Monitoring
Intelligent MonitoringIntelligent Monitoring
Intelligent MonitoringIntelie
 
Patterns in the cloud
Patterns in the cloudPatterns in the cloud
Patterns in the cloudDavid Manning
 
Deep learning with kafka
Deep learning with kafkaDeep learning with kafka
Deep learning with kafkaNitin Kumar
 
Scalable Integration with JBoss Fuse
Scalable Integration with JBoss FuseScalable Integration with JBoss Fuse
Scalable Integration with JBoss FuseChristina Lin
 

Similar to Extra performance out of thin air (20)

The hidden engineering behind machine learning products at Helixa
The hidden engineering behind machine learning products at HelixaThe hidden engineering behind machine learning products at Helixa
The hidden engineering behind machine learning products at Helixa
 
Oracle Basics and Architecture
Oracle Basics and ArchitectureOracle Basics and Architecture
Oracle Basics and Architecture
 
Entenda de onde vem toda a potência do Intel® Xeon Phi™
Entenda de onde vem toda a potência do Intel® Xeon Phi™ Entenda de onde vem toda a potência do Intel® Xeon Phi™
Entenda de onde vem toda a potência do Intel® Xeon Phi™
 
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features
 
Building an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open SourceBuilding an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open Source
 
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptxBuilding an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
 
Barga IC2E & IoTDI'16 Keynote
Barga IC2E & IoTDI'16 KeynoteBarga IC2E & IoTDI'16 Keynote
Barga IC2E & IoTDI'16 Keynote
 
Cassandra Tuning - above and beyond
Cassandra Tuning - above and beyondCassandra Tuning - above and beyond
Cassandra Tuning - above and beyond
 
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...
 
The post release technologies of Crysis 3 (Slides Only) - Stewart Needham
The post release technologies of Crysis 3 (Slides Only) - Stewart NeedhamThe post release technologies of Crysis 3 (Slides Only) - Stewart Needham
The post release technologies of Crysis 3 (Slides Only) - Stewart Needham
 
Monitoring InfluxEnterprise
Monitoring InfluxEnterpriseMonitoring InfluxEnterprise
Monitoring InfluxEnterprise
 
Performance and how to measure it - ProgSCon London 2016
Performance and how to measure it - ProgSCon London 2016Performance and how to measure it - ProgSCon London 2016
Performance and how to measure it - ProgSCon London 2016
 
Column store indexes and batch processing mode (nx power lite)
Column store indexes and batch processing mode (nx power lite)Column store indexes and batch processing mode (nx power lite)
Column store indexes and batch processing mode (nx power lite)
 
Osol Pgsql
Osol PgsqlOsol Pgsql
Osol Pgsql
 
CMP301_Deep Dive on Amazon EC2 Instances
CMP301_Deep Dive on Amazon EC2 InstancesCMP301_Deep Dive on Amazon EC2 Instances
CMP301_Deep Dive on Amazon EC2 Instances
 
Migrating To PostgreSQL
Migrating To PostgreSQLMigrating To PostgreSQL
Migrating To PostgreSQL
 
Intelligent Monitoring
Intelligent MonitoringIntelligent Monitoring
Intelligent Monitoring
 
Patterns in the cloud
Patterns in the cloudPatterns in the cloud
Patterns in the cloud
 
Deep learning with kafka
Deep learning with kafkaDeep learning with kafka
Deep learning with kafka
 
Scalable Integration with JBoss Fuse
Scalable Integration with JBoss FuseScalable Integration with JBoss Fuse
Scalable Integration with JBoss Fuse
 

Recently uploaded

WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...WSO2
 
WSO2Con2024 - Unleashing the Financial Potential of 13 Million People
WSO2Con2024 - Unleashing the Financial Potential of 13 Million PeopleWSO2Con2024 - Unleashing the Financial Potential of 13 Million People
WSO2Con2024 - Unleashing the Financial Potential of 13 Million PeopleWSO2
 
WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...
WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...
WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...WSO2
 
WSO2Con204 - Hard Rock Presentation - Keynote
WSO2Con204 - Hard Rock Presentation - KeynoteWSO2Con204 - Hard Rock Presentation - Keynote
WSO2Con204 - Hard Rock Presentation - KeynoteWSO2
 
WSO2Con2024 - Enabling Transactional System's Exponential Growth With Simplicity
WSO2Con2024 - Enabling Transactional System's Exponential Growth With SimplicityWSO2Con2024 - Enabling Transactional System's Exponential Growth With Simplicity
WSO2Con2024 - Enabling Transactional System's Exponential Growth With SimplicityWSO2
 
WSO2Con2024 - Simplified Integration: Unveiling the Latest Features in WSO2 L...
WSO2Con2024 - Simplified Integration: Unveiling the Latest Features in WSO2 L...WSO2Con2024 - Simplified Integration: Unveiling the Latest Features in WSO2 L...
WSO2Con2024 - Simplified Integration: Unveiling the Latest Features in WSO2 L...WSO2
 
WSO2CON 2024 - Does Open Source Still Matter?
WSO2CON 2024 - Does Open Source Still Matter?WSO2CON 2024 - Does Open Source Still Matter?
WSO2CON 2024 - Does Open Source Still Matter?WSO2
 
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open SourceWSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open SourceWSO2
 
WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...
WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...
WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...WSO2
 
Novo Nordisk: When Knowledge Graphs meet LLMs
Novo Nordisk: When Knowledge Graphs meet LLMsNovo Nordisk: When Knowledge Graphs meet LLMs
Novo Nordisk: When Knowledge Graphs meet LLMsNeo4j
 
WSO2Con2024 - Organization Management: The Revolution in B2B CIAM
WSO2Con2024 - Organization Management: The Revolution in B2B CIAMWSO2Con2024 - Organization Management: The Revolution in B2B CIAM
WSO2Con2024 - Organization Management: The Revolution in B2B CIAMWSO2
 
WSO2CON 2024 Slides - Unlocking Value with AI
WSO2CON 2024 Slides - Unlocking Value with AIWSO2CON 2024 Slides - Unlocking Value with AI
WSO2CON 2024 Slides - Unlocking Value with AIWSO2
 
WSO2CON 2024 Slides - Open Source to SaaS
WSO2CON 2024 Slides - Open Source to SaaSWSO2CON 2024 Slides - Open Source to SaaS
WSO2CON 2024 Slides - Open Source to SaaSWSO2
 
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...WSO2
 
WSO2CON 2024 - Software Engineering for Digital Businesses
WSO2CON 2024 - Software Engineering for Digital BusinessesWSO2CON 2024 - Software Engineering for Digital Businesses
WSO2CON 2024 - Software Engineering for Digital BusinessesWSO2
 
Driving Innovation: Scania's API Revolution with WSO2
Driving Innovation: Scania's API Revolution with WSO2Driving Innovation: Scania's API Revolution with WSO2
Driving Innovation: Scania's API Revolution with WSO2WSO2
 
WSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go PlatformlessWSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go PlatformlessWSO2
 
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...WSO2
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AIAGATSoftware
 
Effective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConEffective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConNatan Silnitsky
 

Recently uploaded (20)

WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
 
WSO2Con2024 - Unleashing the Financial Potential of 13 Million People
WSO2Con2024 - Unleashing the Financial Potential of 13 Million PeopleWSO2Con2024 - Unleashing the Financial Potential of 13 Million People
WSO2Con2024 - Unleashing the Financial Potential of 13 Million People
 
WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...
WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...
WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...
 
WSO2Con204 - Hard Rock Presentation - Keynote
WSO2Con204 - Hard Rock Presentation - KeynoteWSO2Con204 - Hard Rock Presentation - Keynote
WSO2Con204 - Hard Rock Presentation - Keynote
 
WSO2Con2024 - Enabling Transactional System's Exponential Growth With Simplicity
WSO2Con2024 - Enabling Transactional System's Exponential Growth With SimplicityWSO2Con2024 - Enabling Transactional System's Exponential Growth With Simplicity
WSO2Con2024 - Enabling Transactional System's Exponential Growth With Simplicity
 
WSO2Con2024 - Simplified Integration: Unveiling the Latest Features in WSO2 L...
WSO2Con2024 - Simplified Integration: Unveiling the Latest Features in WSO2 L...WSO2Con2024 - Simplified Integration: Unveiling the Latest Features in WSO2 L...
WSO2Con2024 - Simplified Integration: Unveiling the Latest Features in WSO2 L...
 
WSO2CON 2024 - Does Open Source Still Matter?
WSO2CON 2024 - Does Open Source Still Matter?WSO2CON 2024 - Does Open Source Still Matter?
WSO2CON 2024 - Does Open Source Still Matter?
 
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open SourceWSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
 
WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...
WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...
WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...
 
Novo Nordisk: When Knowledge Graphs meet LLMs
Novo Nordisk: When Knowledge Graphs meet LLMsNovo Nordisk: When Knowledge Graphs meet LLMs
Novo Nordisk: When Knowledge Graphs meet LLMs
 
WSO2Con2024 - Organization Management: The Revolution in B2B CIAM
WSO2Con2024 - Organization Management: The Revolution in B2B CIAMWSO2Con2024 - Organization Management: The Revolution in B2B CIAM
WSO2Con2024 - Organization Management: The Revolution in B2B CIAM
 
WSO2CON 2024 Slides - Unlocking Value with AI
WSO2CON 2024 Slides - Unlocking Value with AIWSO2CON 2024 Slides - Unlocking Value with AI
WSO2CON 2024 Slides - Unlocking Value with AI
 
WSO2CON 2024 Slides - Open Source to SaaS
WSO2CON 2024 Slides - Open Source to SaaSWSO2CON 2024 Slides - Open Source to SaaS
WSO2CON 2024 Slides - Open Source to SaaS
 
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...
 
WSO2CON 2024 - Software Engineering for Digital Businesses
WSO2CON 2024 - Software Engineering for Digital BusinessesWSO2CON 2024 - Software Engineering for Digital Businesses
WSO2CON 2024 - Software Engineering for Digital Businesses
 
Driving Innovation: Scania's API Revolution with WSO2
Driving Innovation: Scania's API Revolution with WSO2Driving Innovation: Scania's API Revolution with WSO2
Driving Innovation: Scania's API Revolution with WSO2
 
WSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go PlatformlessWSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go Platformless
 
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AI
 
Effective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConEffective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeCon
 

Extra performance out of thin air

  • 2. PATH TO EXTRA PERFORMANCE Eliminate unneeded work § Choose data types wisely Eliminate unneeded waits § Reduce number of locks Make system operate in more efficient way § Optimize BIOS settings § Stay in same “technology slice” § Make sure you have enough RAM
  • 3. ART OF CHOOSING DATATYPES
  • 4. WHY DATA TYPE MATTERS ?
  • 5. WHY DATA TYPE MATTERS ? Fastest CPU today is 3.7 GHz It takes 1 / 3,700,000,000 of a second to do single operation
  • 6. WHY DATA TYPE MATTERS ? Fastest CPU today is 3.7 GHz It takes 1 / 3,700,000,000 of a second to do single operation “BIG DATA” record set starts from 100 billion records
  • 7. WHY DATA TYPE MATTERS ? Fastest CPU today is 3.7 GHz It takes 1 / 3,700,000,000 of a second to do single operation “BIG DATA” record set starts from 100 billion records Processing time 1 / 3,700,000,000 sec X 100,000,000,000 = 27 sec
  • 8. DO YOU NEED TO STORE DATA SAME WAY IT IS PRESENTED ?
  • 9. DO YOU NEED TO STORE DATA SAME WAY IT IS PRESENTED ? Presentation: $395.17
  • 10. DO YOU NEED TO STORE DATA SAME WAY IT IS PRESENTED ? Presentation: $395.17 Data: 395.17
  • 11. DO YOU NEED TO STORE DATA SAME WAY IT IS PRESENTED ? Presentation: $395.17 Data: 395.17 Storage: Store as Money Data type: MONEY Internal data type: NUMERIC(18,4) Storage: Store as numeric Data type: NUMERIC Internal data type: NUMERIC(37,15) Storage: Store as integer Data type: INT Internal data type: INT
  • 12. DO YOU NEED TO STORE DATA SAME WAY IT IS PRESENTED ? Presentation: $395.17 Data: 395.17 Storage: Store as Money Data type: MONEY Internal data type: NUMERIC(18,4) Storage: Store as numeric Data type: NUMERIC Internal data type: NUMERIC(37,15) Storage: Store as integer Data type: INT Internal data type: INT
  • 14. DATA TYPE BENCHMARK AVERAGES IN SEC 27.2 29.7 37 0 5 10 15 20 25 30 35 40 INT NUMERIC(18,5) NUMERIC(37,15)
  • 15. MAKING RIGHT CHOICES • If you can store data as INTEGER • Choose INTEGER • If your data fits into 18 digits of PRECISION • Choose NUMERIC(18) • If your data larger then 18 digits of PRECISION • Choose NUMERIC(your-desired-precision) Vertica default for NUMERIC is NUMERIC(37,15)
  • 17. LOCKING BEHAVIOR AUTOCOMMIT = ON (jdbc driver default) § Each statement treated as complete transaction § When statement completes changes automatically committed to database AUTOCOMMIT = OFF § Transaction continue until manually run COMMIT or ROLLBACK § Locks kept on objects for transaction duration
  • 18. CONTROLLING AUTOCOMMIT STATE JAVA: conn = DriverManager.getConnection("jdbc:vertica://DBHost:5433/MyDB", myProperties); // get the state of the auto commit parameter System.out.println("Autocommit state: " + conn.getAutoCommit()); // Change the auto commit state to false conn.setAutoCommit(false); SQL:
  • 19. IMPACT ON LOCK COUNTS BY CHANGING AUTOCOMMIT SETTING TO OFF
  • 20. HOW TO DISABLE – OBVIOUS METHOD
  • 21. HOW TO DISABLE – BETTER METHOD
  • 23. WHAT IS TUNABLE IN BIOS?
  • 24. HOW TO TUNE ? http://h10032.www1.hp.com/ctg/Manual/c01804533.pdf
  • 25. DOES IT REALLY MATTER ? 0 100 200 300 400 500 600 700 800 900 1000 DSS BIOS settings with 1x DRAM refresh rate DSS BIOS settings with 4x DRAM refresh rate HPC BIOS settings with 4x DRAM refresh rate HPC + HyperThreading BIOS settings with 4x DRAM refresh rate HPC - NO TurboBoost BIOS settings with 4x DRAM refresh rate Sec DSS  BIOS  se)ngs  with  1x  DRAM  refresh  rate   738.949439   DSS  BIOS  se)ngs  with  4x  DRAM  refresh  rate   745.111176   HPC  BIOS  se)ngs  with  4x  DRAM  refresh  rate   552.148285   HPC  +  HyperThreading  BIOS  se)ngs  with  4x  DRAM  refresh  rate   877.838469   HPC  -­‐  NO  TurboBoost  BIOS  se)ngs  with  4x  DRAM  refresh  rate   561.260084   Performance increase potential about 40%
  • 26. WHAT TUNING DOC SAYS ?
  • 27. STAYING IN THE SAME “TECHNOLOGY SLICE”
  • 28. WHAT I WILL BE SLICING THROUGH ??? CPU and chipset Hardware Operating System (OS) Database Management System (DBMS)
  • 29. WHAT IS “TECHNOLOGY SLICE” ANYWAY ??? CPU Gen3 CPU Gen4 Server Gen-B OS v. 36 DBMS v. 6 Server Gen-C OS v. 37 DBMS v. 7 CPU Gen5 Server Gen-D CPU Gen6 CPU Gen7 Server Gen-E Srv Gen F OS v. 38 Server Gen-A OS v. 35OS v. 34 DBMS v. 5DBMS v. 4DBMS v. 3
  • 30. WHAT IS “TECHNOLOGY SLICE” ANYWAY ??? CPU Gen3 CPU Gen4 Server Gen-B OS v. 36 DBMS v. 6 Server Gen-C OS v. 37 DBMS v. 7 CPU Gen5 Server Gen-D CPU Gen6 CPU Gen7 Server Gen-E Srv Gen F OS v. 38 Server Gen-A OS v. 35OS v. 34 DBMS v. 5DBMS v. 4DBMS v. 3
  • 31. COMMON “TECHNOLOGY SLICE” TRAP CPU Gen3 CPU Gen4 ✔ Server Gen-B OS v. 36 DBMS v. 6 Server Gen-C ✔ OS v. 37 ✔ DBMS v. 7 ✔ CPU Gen5 Server Gen-D CPU Gen6 CPU Gen7 Server Gen-E Srv Gen F OS v. 38 Server Gen-A OS v. 35OS v. 34 DBMS v. 5DBMS v. 4DBMS v. 3
  • 32. COMMON “TECHNOLOGY SLICE” TRAP CPU Gen3 CPU Gen4 ✔ Server Gen-B OS v. 36 DBMS v. 6 Server Gen-C ✔ OS v. 37 ✔ DBMS v. 7 ✔ CPU Gen5 Server Gen-D CPU Gen6 CPU Gen7 Server Gen-E Srv Gen F OS v. 38 Server Gen-A OS v. 35OS v. 34 DBMS v. 5DBMS v. 4DBMS v. 3 ? ?
  • 33. SYMPTOMS OF “TECHNOLOGY SLICE” ISSUES System AVG: 57.90 Nice AVG: 46.56 System AVG > Nice AVG System AVG / Nice AVG = 1.24 System AVG: 11.19 Nice AVG: 57.38 System AVG < Nice AVG System AVG / Nice AVG = 0.19
  • 34. “TECHNOLOGY SLICE” PERFORMANCE IMPACT 0 20 40 60 80 100 120 140 different “TECHNOLOGY SLICE” kernel proper “TECHNOLOGY SLICE” kernel Sec
  • 36. DO I REALLY NEED MORE RAM ? select event_type, count(1) from query_events group by event_type order by 2 desc; Spilled events are very good indication of queries not fitting in RAM
  • 37. HOW I CAN QUANTIFY IMPACT ? select 'event_timestamp' as timestamp_type, min(event_timestamp) as min_timestamp, max(event_timestamp) as max_timestamp from query_events union select 'query_timestamp' as timestamp_type, min(start_timestamp) as min_timestamp, max(start_timestamp) as max_timestamp from query_requests; System tables in Vertica have individual rolling window. Make sure you understand relation of histories available.
  • 38. HOW I CAN QUANTIFY IMPACT ? CONT. select spilled_queries, total_qieries, round( spilled_queries / total_qieries * 100 , 2 ) as spilled_queries_percent from (select count(1) as total_qieries from query_requests where request_type = 'QUERY' and start_timestamp > (select min(event_timestamp) from query_events)) query_data, (select count(1) as spilled_queries from (select session_id, transaction_id, statement_id from query_events where event_type ilike '%SPILLED%' group by session_id, transaction_id, statement_id) spill_data) spill_data2; Amount of spilled queries in relation to entire query volume.
  • 39. CAN MY SPILLED DATA FIT IN TO RAM ? select min(counter_value) as min_bytes_spilled, max(counter_value) as max_bytes_spilled, avg(counter_value) as avg_bytes_spilled from execution_engine_profiles where counter_name = 'bytes spilled' and counter_value > 0; Understanding size of spillage to disk.
  • 40. WHO CAUSING SPILLS ? select user_name, count(1) as spill_event_count from query_events where event_type ilike '%SPILLED%' group by user_name order by 2 desc; In Vertica RAM allocated to queries through resource pools. Resource pools connected to users. Knowing user will point us to resource pool, which needs tuning.
  • 41. WHAT I SHOULD TUNE ? select distinct resource_pool from users where user_name in ('peter', 'john'); Identified resource pool with spilled queries. Now we know what to tune.
  • 42. The resource pool parameters of MEMORYSIZE and PLANNEDCONCURRENCY provide the options that let you tune the target memory allocated to queries. WHAT I SHOULD CHANGE ? HP Vertica Analytics Platform Version 7.1.x Documentation Administrator's Guide Managing the Database Managing Workloads Resource Pool Architecture Target Memory Determination for Queries in Concurrent Environments
  • 43. Q & A