The document discusses high performance JDBC and database optimization. It covers topics like persistence frameworks, JDBC drivers and connections, query performance, and tuning database and application design. Specific tips mentioned include using prepared statements, setting fetch sizes, closing resources, and leveraging database features like indexes and materialized views.

1. High Performance JDBC Jorg Janke Consultant and Coach [email_address]

2. Agenda What I do Persistence Performance JDBC Connection Query Result Timing Statistics

3. Shameless Self-Promotion Consultant & Coach (available) I talk business Trained Accountant, MBA VP Product Marketing I talk tech Started programming PL/I, SmallTalk, Oracle5 Director in Oracle Apps Development Certified Oracle DBA, Sun Java Pr/Dev/Arch

4. Compiere The Open Source Business Solution More than 240.000 downloads Usually among Top 10 in SourceForce Targeting Small-Medium Enterprise ASP Solution Accounting, Inventory, … Worldwide (Language, Law, …) J2EE compliant Application Swing or HTML interface

5. Agenda -- Persistence What I do Persistence Performance JDBC Connection Query Result

6. Persistence Object meets Database Put a square in a round JDBC is not a Persistence Framework Object-relational Mapping Container Managed Persistence Java Data Objects

7. Persistence – Entity Bean CMP <enterprise-beans> <entity> <description>Models a music CD</description> <ejb-name>CDBean</ejb-name> <home>com.web_tomorrow.cd.CDHome</home> <remote>com.web_tomorrow.cd.CD</remote> <ejb-class>com.web_tomorrow.cd.CDBean</ejb-class> <persistence-type>Container</persistence-type> <prim-key-class> java.lang.String </prim-key-class> <reentrant>False</reentrant> <cmp-field><field-name> id </field-name></cmp-field> <cmp-field><field-name> title </field-name></cmp-field> <cmp-field><field-name> artist </field-name></cmp-field> <primkey-field> id </primkey-field> </entity> </entity-beans>

8. Persistence – JDO Java Data Objects Transparent database access Code that accesses the underlying data store, without using database-specific code Use of JDO in J2EE Persistent helper classes for Session Beans Delegate classes for Bean or Container Managed Persistence Entity Beans

9. JDBC Based Frameworks Everyone (… and their Grandmother) writes JDBC based Persistence Libraries Free or Commercial Often with IDE integration Examples Borland Oracle Compiere (no IDE) 30 % of typical IT project: DB programming & persistence mgt

10. Agenda -- Performance What I do Persistence Performance JDBC Connection Query Result

11. Performance Is a side effect of good OO design Schema should be ≥ 3 rd Normal Form That is want Optimizers recognize For high-end databases (e.g. Oracle) there is no need for de-normalization Performance Rules #1 Clean design #2 Tweak performance Don’t Design for Performance

12. Performance Rules Tune the Design Tune the Application Tune the OS Tune Memory Structure Tune Disk I/O Usage Detect and Eliminate Resource Contention

13. Performance Design Options Data Intensive Routines in DB Server Within DB Engine - Low Switching Cost No Communication Overhead PL/SQL, Java Stored Procedures, C++, … “Automatic” data consistency (Helps the Optimizer) Explicit Constraints Trigger (in PL/SQL, Java, …)

14. DB Programming Options Set Processing outperforms everything else Not your beginner SQL Complex UPDATE statements PL/SQL Dependency Management (*) Data intensive manipulations Java Stored Procedures Special VMs – Upload, Debugging Special JDBC Driver “always connected”

15. DB Programming Options SQLJ Java Language SQL Pre-compiler High level SQLJ-Language Converted to “normal” JDBC code Supported by IBM, Oracle, …

16. Question: Is SQL a Standard?

17. DB Independence Lowest common denominator Use Access Functionality with Oracle ? You have to design & program differently for the database / camps Oracle, DB2, (PostgreSQL), … Microsoft SQL Server, Sybase, … MS Access, mySQL, …

18. DB Incompatibilities (1) SQL join SELECT .. FROM a JOIN b ON (a.id=b.id); SELECT .. FROM a LEFT OUTER JOIN b ON (a.id=b.id); SELECT .. FROM a, b WHERE a.id=b.id(+); SELECT .. FROM a, b WHERE a.id=*b.id; Sub-Query SELECT .. FROM a, (SELECT .. FROM b) AS b WHERE … Pseudo-Columns ROWNUM, SysDate LIMIT Reserved Words – Optional Syntax – Alias

19. DB Incompatibilities (2) Data Types Not just naming (NUMBER vs. DECIMAL) BLOB, CLOB, … PL/SQL - Syntax/Functionality for Packages, Functions, Procedures, Triggers, … Commit Strategy Exception (raise & catch) Data Dictionary OLAP Support Materialized Views, …

20. DB Independence – cont. DB Management Layer Lowest Common SQL Denominator Alternative Application Dependent Uses Database specific Functionality Examples: SAP’s DB Access Layer (CMP Products) Compiere’s dbPort … Oracle PG dbPort Application I am talking to Oracle DB/2 ?

21. Performance Reality You can’t have everything High functionality & Ability to fine-tune Database independence Automatic Object Mapping You need skill-mix (Traditional) Database Design Database Programming Environment Object Design Java Programming Environment

22. The Test Environment Windows 2000 SP 2 Server Dual 450MHz CPU Oracle 8.1.7 (parallel option not enabled) JDBC driver for 9i RedHat Linux 7.2 Single 500 MHz CPU PostgreSQL 7.1.3 JDBC driver for 7.2 Java 1.4.0 SELECT * FROM AD_Column; 4,800 Rows - 38 Columns – ca. 1,2 MB Testing DB Driver

23. Agenda -- JDBC What I do Persistence Performance JDBC Connection Query Result

24. Connecting Connection Options java.sql.Driver & java.sql.DriverManager javax.sql.DataSource javax.sql.PooledConnection Connection Cache

25. JDBC Driver Types Type 4 Pure Java Type 3 Type 1 Type 2

26. Connecting .. Driver Type 1..4 Driver Type 4: Pure Java Driver DB/2 does not provide Type 4 driver Interface java.sql.Driver Explicitly Class.forName(“org.postgresql.Driver&quot;); new oracle.jdbc.OracleDriver(); System Properties jdbc.drivers= oracle.jdbc.OracleDriver:org.postgresql.Driver

27. Connecting DriverManager Basic service for managing a set of JDBC drivers JDBC Drivers register with the DriverManager Source Connection c = DriverManager.getConnection (&quot;jdbc:oracle:thin:@dev:1521:dev1&quot;); Connection Strings jdbc:oracle:oci8:@ -- (#1) Bequeath jdbc:oracle:oci8:@ora -- (#3) Net8 jdbc:oracle:thin:@dev:1521:ora -- (#2) Type 4 jdbc:postgresql://dev:5432/compiere

28. Connecting .. Data Source Interface javax.sql.DataSource Alternative to DriverManager Based on JNDI Server Source DataSource ds = new OracleDataSource(); Context ctx = new Initial Context(); ctx.bind (“jdbc/myDB”, ds); Client Source Context ctx = new Initial Context(); DataSource ds = (DataSource)ctx.lookup(“jdbc/myDB”); Connection c = ds.getConnection();

29. Connection .. Pool Optional extension One Physical Connection Multiple Logical Connection Connection.close() makes it available Eliminates overhead of creating physical connection: Connection c = pool.getConnection(); Statement stmt = c.createStatement(..); stmt.close(); c.close();

30. Connection .. Cache Optional Extension … no standard (!) Multiple Physical Connections We wrote own Connection Cache Connection not closed in code

31. Multi-Thread Most JDBC implementations synchronize on Connection You need multiple connections for multi-threaded applications Compiere uses 2 “read-only” never closed Cached Connections (w. health check) 1 “update” never closed Connection

32. Test Results (10 Threads)

33. Test Results (10 Threads) Multiple Pre-Created Connections No xxx.getConnection() Yield ‘surprise’ while (rs.next) yield(); (High) switching costs Database has result in buffer just transmitting Connection Cache Optimum Cache Size 2-4

34. Test Results Oracle Use Thin Driver jdbc:oracle:thin:@dev:1521:ora Fastest Driver – Bequeath &quot;jdbc:oracle:oci8:@“ 20-40% faster – Only works if only one instance installed For Server Connections Slowest Driver – Net8 jdbc:oracle:oci8:@ora 10-20% slower Net8 conversion Overhead

35. Connection – Trx Level Transaction Isolation Level TRANSACTION_NONE Not supported TRANSACTION_READ_UNCOMMITTED Get: dirty, non-repeatable, phantom reads TRANSACTION_READ_COMMITTED No: dirty reads (uncommitted data) Get: non-repeatable, phantom reads TRANSACTION_REPEATABLE_READ No: dirty, non-repeatable (data changed) reads Get: phantom reads TRANSACTION_SERIALIZABLE No: dirty reads, non-repeatable, phantom (WHERE changed) reads

36. Connection You probably want to get the latest data conn.setTransactionIsolation (Connection.TRANSACTION_READ_COMMITTED); Data updated by others & Triggers Set explicitly Connection is Interface Default set by Vendor Implementation Consider Read Only – Hint to Driver/Database conn.setReadOnly(true);

37. Querying Statement – conn.createStatement() ResultSet.TYPE_ FORWARD_ONLY, SCROLL_INSENSITIVE, SCROLL_SENSITIVE ResultSet.CONCUR_ READ_ONLY or _UPDATABLE Prepared Statement – conn.prepareStatement() Database dependent statement caching “ UPDATE EMPLOYEES SET SALARY = ? WHERE ID = ?” pstmt.setDataType (pos, value); Callable Statement – conn.prepareCall() “ {?= call myProcedure (?,?)}” Batch [implementation not mandatory] stmt.addBatch(“Insert ..”); stmt.executeBatch();

38. set DataType Conversion

39. Result - ResultSet Created by stmt.executeQuery() Fetch Size rs.setFetchSize(int); stmt.setFetchSize(int); VendorConnection.setFetchSize(int); Driver hint Non updateable RS Forward only & read just once Data Retrieval get DataType (int pos); get DataType (String columnName); wasNull(); No way to Cancel a Query

40. get DataSet Conversion

41. ResultSet Navigate absolute(int), relative(int) – first(), last(), next(), previous() Update updateDataType (pos, data); updateDataType (columnName, data); updateRow(); – or – cancelRowUpdates(); Insert moveToInsertRow(); updateDataType insertRow(); – or – cancelRowUpdates(); moveToCurrentRow(); Delete deleteRow(); Refresh refreshRow();

42. Result - RowSet JavaBean - Extends ResultSet Types – Implementation Optional Cached – serialized - disconnected - X-JVM - Synchronize JDBC Web Code Example RowSet rowset = new OracleCachedRowSet(); rowset.setUrl (CONNECTION); rowset.setUsername (UID); rowset.setPassword (PWD); rowset.setFetchSize (fetchSize); rowset. setCommand (SQL_STATEMENT); rowset. execute (); Client … RMI … no JDBC

43. Meta Data Interfaces DatabaseMetaData Driver JDBC compatibility Data Dictionary Queries ParameterMetaData Prepared Statement ResultSetMetaData Get Column Name, Data Type of “SELECT *” query

44. Connection - Query - Retrieve First Connection, (Query) & Retrieval took significantly longer – 4,800 records

45. Fetch Size

46. Compiere JDBC Use Async worker to load data Select Prepared Statement - Read-Only, Forward Only Statement & ResultSet (cursor) closed after read Insert ResultSet from Prepared Statement (WHERE 1=2) rs.moveToInsertRow(); rs.insertRow(); Re-read from Database using PK (trigger, ..) Update Re-read using ROWID or OID (“for update”) Check, if data* not changed rs.updateXX(); rs.updateRow(); or rs.cancelRowUpdates(); Re-read from Database using PK deterministic, optimistic locking

47. Performance Hints Qualify Columns in SELECT Clause Gets only data needed Use position of Column – not Name rs.getString(1); Give as much info as possible WHERE clause Scrollable/Updateable/… Use Locks carefully … and release them explicitly Close Statements & ResultSets – closes DB cursors rs.close(); stmt.close();

48. Performance Hints (2) Use Batches where possible Connection.setAutoCommit (false) Database Meta-Data Queries are slow Quantify arguments, search patterns Consider Use of Statement.setFetchSize (int no) Statement.setMaxRows (int no) Statement.setQueryTimeout (int seconds) Statement.setMaxFieldSize (int byte) Hidden unique ID: RowID, OID RowSet for complex mass update Use Prepared Statements (*)

49. Resources (Selection) JDBC http://java.sun.com/products/jdbc Several courses and introductions http://www.jguru.com/faq/JDBC http://www.javaskyline.com/learnjdbc.html JDO http://access1.sun.com/jdo http://www.jdocentral.com

50. Thanks You can download the presentation http://www.compiere.org/download/ If you have questions [email_address] (203) 445-8182 Have a look http://www.compiere.com/consulting.html