1. Advanced Databasing
Bogdan Kecman
MySQL Principal Technical Engineer @ORACLE
CEO, BAD-TEAM
bogdan.kecman@oracle.com
bogdan.kecman@bad-team.net
INIT, Banja Luka, Novembar 2019
2. Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
2
6. MAGIC?
• Education?
• Information volume?
• Information complexity?
• Not sexy enough?
• Money not good?
• Requirement?
Why is so little attention given to monitoring and data storage?
6
7. How to uncover the MAGIC?
●
Understand the problem
●
Check out all available tools
●
Think OUTSIDE THE BOX but KNOW THE BOX!!!
Any sufficiently advanced technology is indistinguishable from magic!
7
8. Problems?
●
Availability & Reliability – robustness
●
Scalability – “all directions”
●
Performance
●
TCO optimization
For every expert, there is an equal and opposite expert!
8
9. Know your tools
Any sufficiently advanced technology is indistinguishable from magic!
9
10. Examples – MySQL
MySQL environment can be very complex or very simple!
10
MySQL Cluster Data Nodes InnoDB SE s1 SE s2
MyISAM
11. Examples – MySQL InnoDB
●
SQL (via MySQL connector/php,java,ruby,python..., odbc, C-API..)
●
MEMCACHED (via memcached plugin on the mysqld)
●
InnoDB API (C-API)
How can you access data stored in InnoDB table?
11
12. Examples – MySQL HTTP plugin
●
SQL over HTTP(s)
– http(s)://host:port/sql/SELECT+%2A+FROM+t1+ORDER+BY+f1
●
CRUD (aka REST) over HTTP(s)
– http(s)://host:port/crud/database/table/primaryKey
●
JSON over HTTP(s)
– http(s)://host:port/doc/database/table
– http(s)://host:port/doc/database/table/key
Other ways to access data?
12
14. Examples – MySQL Cluster
●
SQL (via MySQL connector/php,java,ruby..., odbc, MySQL C-API..)
●
MEMCACHED (add ndbcluster driver to memcached server)
●
ClusterJ, JPA, ClusterJPA, LDAP
●
ndbAPI (C/C++ API)
MySQL Cluster is a system that runs 80% of mobile traffic in the World!
14
ClusterJ
MySQL
JDBC
Apps
JPA
JNI
Python Ruby
ClusterJPA
Apps Apps Apps Apps Apps
Node.js
JSON
Apps
mod-ndb
Apache
Apps
ndb-eng
Memcached
Apps Apps
NDB API (C++)
MySQL Cluster Data Nodes
Apps
PHP PERL
Apps
15. Examples – MySQL Cluster
●
SQL (via MySQL C-API or JDBC..) to create and backup/restore schema (DDL)
●
ClusterJ, LDAP to access and manipulate data (DML)
MySQL Cluster how are Telco’s accessing it
15
Visitor Location Register
Authentication Center
Home Location Register
NDB API (C++)
MySQL Cluster Data Nodes
Management
Apps
C API / JDBC ClusterJ
JNI
MySQL LDAP
16. Examples – MySQL Cluster as persistent memcached
Memcached is fast as very usable, can we get similar performance but keep
data persistent ?
16
hash key
to find data
Memcache
httpd memcached
memcached
memcached
memcache key
PHP/Perl
friends:12389
MySQL
Cluster
Data Node
memcached
Memcache
Client
Application
MySQL
Cluster
Data Node
NDB Engine
Traditional
Persistent with MCCGE
17. Results - MySQL Cluster as persistent memcached
memcachetest -t 2 -M 7000 -c 25000
17
18. Results – MySQL Cluster SQL vs NDB API
Reads / seconds – note logarithmic Y axis
18
23. Server monitoring & profiling
●
MySQL Enterprise Monitor
●
MySQL SYS schema
●
pgBadger
●
pgFouine
●
Nagios and Zabbix plugins to monitor (r)DBMS state
I believe in having each device secured and monitoring each device, rather
than just monitoring holistically on the network, and then responding in
short enough time for damage control.
23
24. DB Model optimization
●
First Normal Form
●
Second Normal Form
●
Third Normal Form
●
Boyce – Codd normal form
●
Fourth Normal Form
●
5th
, Domain Key and 6th
Normal Form.
In order to denormalize db model you first need to achieve it’s NF
24
25. Query profiling and tuning / optimization
●
ORM
●
ORM?
●
ORM REALLY?
●
Profile queries!
●
Choose a right (R)DBMS
●
Use CTE (common table expressions)
Do you really trust ORM to write a query for you?
25
26. Example - Query profiling and tuning / optimization
Original:
DELETE FROM t1 WHERE id NOT IN (SELECT id FROM t1_data);
Query time 3.5s - 5s
Optimized to:
DELETE t1 FROM t1 LEFT OUTER JOIN t1_data
ON msg.id=msg_data.id WHERE t1_data.id IS NULL;
Optimized query time below 1.5s! More then 3 times faster!
Large number of entries inside IN() clause is not smart
26
27. Example - Query profiling and tuning / optimization
Do you really trust ORM to write a query for you?
27
Original query (ORM generated) takes 10 minutes to execute:
SELECT t1.wid,
t2.wname,
Date(t1.t_stamp) AS 'dateOnly',
t1.rpd + ( t1.runminpd / 60 ) AS 'PDRuntime',
Min(t1.t_stamp),
l.name,
fg.fieldgroupname
FROM arch.lfedata plc
LEFT JOIN arch.lfe_wellsites w
ON t1.wid = t2.id
LEFT JOIN arch.lfe_locations l
ON t2.locationid = l.id
LEFT JOIN arch.lfe_fieldgroups fg
ON l.fieldgroupid = fg.fieldgroupid
28. Example - Query profiling and tuning / optimization
Do you really trust ORM to write a query for you?
28
WHERE t1.t_stamp BETWEEN '2016-06-01 00:00:00' AND '2016-08-31 23:00:00'
AND quality_code = 192
AND t2.compreport = 1
GROUP BY t1.wid,
Date(t1.t_stamp)
UNION
(SELECT 2,
'NONE',
'2000-00-00 00:00:00',
0,
'2000-00-00 00:00:00',
'ZZ_NoName',
'NoFieldGroup'
ORDER BY name, wname, dateonly DESC);
29. Example - Query profiling and tuning / optimization
Do you really trust ORM to write a query for you?
29
Final optimized query takes 10 seconds to execute:
SELECT t.wid, t.wname, t. 'dateOnly',
plc.rpd +(plc.RunMinPD/60) as 'PDRuntime', l.name, fg.FieldGroupName
FROM (SELECT plc.wid, DATE(plc.t_stamp) as 'dateOnly', MIN(lfedata_ndx) as 'ndx', t2.wname, t2.locationId
FROM arch.lfedata plc INNER JOIN arch.lfe_wellsites w ON plc.wid = t2.id
WHERE plc.t_stamp BETWEEN '2016-06-01 00:00:00' AND '2016-08-31 23:55:00'
AND plc.quality_code = 192
AND t2.compReport = 1
GROUP BY plc.wid, DATE(plc.t_stamp) ) t
INNER JOIN arch.lfedata plc
ON t.ndx = plc.lfedata_ndx
LEFT JOIN arch.lfe_locations l
ON t.locationId = l.id
LEFT JOIN arch.lfe_fieldgroups fg
ON l.fieldGroupId = fg.FieldGroupId;
30. Sources
●
Voxxed Days – Bogdan Kecman: https://youtu.be/u0A0i9OEohs
●
mDevDay – Bogdan Kecman: https://youtu.be/0vnSchjQHoE
●
Data Science – Nikola Krgović: https://youtu.be/MhE1skm0y7k
●
MySQL Documentation: https://dev.mysql.com/doc/
●
PostgreSQL Documentation: https://www.postgresql.org/docs/
●
The one book to rule them all: https://www.amazon.com/dp/0321197844/
●
TokuDB Fractal Index WP: http://www.miniuri.com/306
The only source of knowledge is experience!
30