An overview of some database techniques that can be leverage to simplify development of Applications

1. Web Developer, make the most out
of your Database !
Leveraging Database Techniques to simplify
and performance-optimize your data
intensive web applications

2. Jean-Marc Desvaux
Head of Information Systems
GENERAL CONSTRUCTION CO.LTD
@jmdesvaux

3. How many times have you
changed an Application
underlying Database ?

4. Very rarely
you always end-up relying on database specific behaviours and
on extensions from SQL standards (ANSI-SQL)
Never use Transact-SQL, PL/SQL, PLpgSQL, PL/php,… ?
Never sort out a performance problem using
a database specific feature ?
Don’t you at least rely on constraints (FKs, PKs)
or DB sequences/PK auto-increment ?

5. Commercial :Customer wants to choose
Database desupport protection
Database-agnostic App Advantages

6. Database-agnostic App Disadvantages
Commercial : Less options to optimize Data intensive
app performance
In the real world you will at some point have to face database
specific features (PKs,FKs,DB Sequences,ANSI-SQL non standard
syntax)
Coding in the wrong place (Client/Middleware/Database)
=Poor Architecture of your App.

7. Why don’t we leverage the Database
development capabilities for our Apps ?

8. See the Database as an important piece
of our Application Architecture, a place
where
we can add value to our App

9. Let’s walk through of couple things we can
do at the database(*) level
..other than DML statements
(*)We will take the example of an Oracle 12c Database (it’s the one we use) but you
will find similar capabilities in other database too

10. DATABASE VIEWS
create view MYVIEW as select * from MYTABLE;
To hide complexity
Better reusability of SQL complex queries
Avoid modifying applications in case you need to refactor a table
view definition can be reviewed while the view name and attributes stay unchanged.
As a Security mechanism
To surface only the data needed by the application or the user.
As an example : we want to allow only specific account codes to specific user
Application will use an ACCOUNT_CODES_VIEW view to list account codes:-
Create view ACCOUNT_CODES_VIEW as
select acc_id,… from ACCOUNT_CODES AS T1 ,USERS_AUTHORISED_ACCOUNTS AS T2
where T1.acc_id=T2.acc_id and T2.user_id=T3.user_id and T2.user_ldap_account=<database session application user>
VIEW
TABLE1
SELECT .. FROM
TABLE1,TABLE2,TABLE3 WHERE ….
TABLE2 TABLE3
APP

11. VIRTUAL COLUMNS
Calculated column without de-normalization
No value stored (only in indexes when indexed)
Avoid modifying applications (if you need to refactor calculation)
TABLE
Col1 Number
Col2 Number
VCol Number as (col1*col2)
select remuneration from EMPLOYEES where remuneration = 10000;
You can create an index on a virtual or add constraints to the virtual column.
alter table EMPLOYEES add (Remuneration number as salary+bonus);
alter table EMPLOYEES add (Remuneration number as myfunction(salary,bonus));
select * from EMPLOYEES where salary + bonus = 10000;
select * from EMPLOYEES where myfunction(salary,bonus) = 10000;
where myfunction is a database function (deterministic, not selecting from EMPLOYEES)

12. DATABASE PACKAGES
Stored Procedures and Functions
Wrap business logic in a database stored procedure or function to
mask complexity and centralize complex and data intensive
processes inside the database, close to the data for better
performance.

13. DATABASE PACKAGES
Stored Procedures and Functions
create or replace PACKAGE BODY MYPACKAGE as
function is myfunction1(param1 in varchar2,..) return varchar2 is
begin
….. Do something and return a varchar2…..
end myfunction1;
function is myfunction2(param2 in varchar2,..) return number is
begin
….. Do something and return a number…..
end myfunction2;
………..
end PACKAGE_MYPACKAGE;
Good practice to create packages
(Package Header + Package Body)
create or replace PACKAGE MYPACKAGE as
function myfunction1(param1 in varchar2,..) return varchar2;
function myfunction2(param2 in varchar2,…) return number;
…..
end PACKAGE_MYPACKAGE;

14. DATABASE PACKAGES
Call the package function from :
SQL statements
SELECT column1, mypackage.myfunction1(column2)
FROM mytable
WHERE column3 = mypackage.myfunction2(column4);

15. DATABASE PACKAGES
Call the package function from :
From other procedures or functions :
create or replace PACKAGE BODY MYPACKAGE as
function is myfunction1(param1 in number,..)
return varchar2 is
declare mychar number;
begin
….. mychar:=myfunction2(param1);
….. Do something and return a varchar2…..
end myfunction1;
…
end PACKAGE_MYPACKAGE;

16. DATABASE PACKAGES
Call the package function from :
Your Application (Java in this case) :
public void getEmployeeName(int empID) throws SQLException {
CallableStatement stmt=null;
Connection conn=null;
try {
conn getConnection ();
String myfunct = “begin ? := mypackage.myfunction1(?); end;”;
stmt = conn.prepareCall(myfunct);
stmt.setInt(1, empID);
stmt.registerOutParameter(2,java.sql.Types.VARCHAR);
stmt.execute;
String empName = stmt.getString(2);
stmt.close();
….
}
catch(SQLException se){….
Finally { ..if stmt!=null) stmt.close();….
…

17. SUBQUERY FACTORING
Example 2 (In-line function)
WITH
FUNCTION myfunction(p_id IN VARCHAR2)
RETURN VARCHAR2
IS
BEGIN
RETURN p_id||'- ok';
END;
SELECT myfunction(mytable.mycolummn) FROM
mytable
Example 1 :
WITH T1
AS
select T2.col x
from T2,T3
where …
SELECT x from T1

18. SUBQUERY FACTORING
WITH counter (r)
AS
(select 1 r from dual
union all
select r+1 from counter
where r < 5 )
SELECT r from counter
WITH RECURSIVITY
Output :
r
---
1
2
3
4
5
Which is similar here to : select rownum r from dual connect by rownum <= 5

19. SUBQUERY FACTORING
with x( s, ind ) as
( select sud, instr( sud, ' ' )
from ( select '53 7 6 195 98 6 8 6 34 8 3 17 2 6 6 28 419 5 8 79' sud from dual
)
union all
select substr( s, 1, ind - 1 ) || z || substr( s, ind + 1 )
, instr( s, ' ', ind + 1 )
from x , ( select to_char( rownum ) z from dual connect by rownum <= 9 ) z
where ind > 0
and not exists ( select null
from ( select rownum lp from dual connect by rownum <= 9 )
where z = substr( s, trunc( ( ind - 1 ) / 9 ) * 9 + lp, 1 )
or z = substr( s, mod( ind - 1, 9 ) - 8 + lp * 9, 1 )
or z = substr( s, mod( trunc( ( ind - 1 ) / 3 ), 3 ) * 3
+ trunc( ( ind - 1 ) / 27 ) * 27 + lp
+ trunc( ( lp - 1 ) / 3 ) * 6
, 1 ) ) )
Select s from x where ind = 0;
WITH RECURSIVITY

20. PIPELINED FUNCTIONS
Process data and pipe rows as you go

21. PIPELINED FUNCTIONS
Create an Array of one dimension:
create or replace TYPE " TYP_CHARS_REC" AS OBJECT ( v1 varchar2(200),…, vN varchar2(200))
Create an Array of the one dimension array to define a table array :
create or replace TYPE " TYP_CHARS_REC_TABLE " is table of TYP_CHARS_REC;
Create a pipelined function :
function GET_MYPIPELINED_RECORDS(var1 type1,.., varP typeP)
return TYP_CHARS_REC_TABLE pipelined
as
…
begin
…
pipe row (TYP_CHARS_REC (value1,….,valueN));
…
end;

22. PIPELINED FUNCTIONS
Select from the pipelined function directly as if it was a table :
select * from TABLE(GET_MYPIPELINED_RECORDS(Param1,,,,ParamP));
Wrap it in a database view where possible
(no parameters required or parameters set via session context) :
create view MYPIPELINED_RECORDS_VIEW
as select * from TABLE(GET_MYPIPELINED_RECORDS());
Use the new complex view to
select * from MYPIPELINED_RECORDS_VIEW

23. Use Case examples :
• Stock item qty update when store transactions are committed.
• Audit logs (prev values and new values audit by user and date)
• To enforce complex consistency/compliance checks
Create trigger Store_Issue_Line_Insert_tigger
After insert
on STORE_ISSUE_LINE
For each row
Begin
Update STOCK_ITEMS
set Stock_item_qty= Stock_item_qty - :new.store_issue_line_qty
where Stock_items.stock_item_id=:new.stock_item_id;
End Store_Issue_Line_Insert_tigger;
DATABASE TRIGGERS

24. INSTEAD OF TRIGGERS
Masking CRUD operations complexity
VIEW1
TABLE1
INSERT INTO VIEW / UPDATE VIEW /
DELETE FROM VIEW
TABLE3 TABLEx
APP
INSTEAD OF TRIGGERS
TABLEy
create trigger MY_VIEW1_INSTEAD_OF_TRIGGER
instead of insert or delete or update
on VIEW1
for each row
BEGIN
If INSERTING then
…do your inserts
Elseif UPDATING then
…do your updates
Elseif DELETING then
…do your deletes
End if;
END;

25. URL_MATCH
Comparing Strings similarities
Select Match_strings from
(Select TABLE.Column Matched_strings,
Utl_Match.Jaro_Winkler_Similarity(TABLE.Column,
’String to match’) match_score
From TABLE)
Where match_score>=80;

26. Lightweight Directory Access Protocol (LDAP)
Integration
Use Case :-
Database is a good place to store security data (Employees or Customers records
are already stored in the database)
Your Applications require a standard LDAP directory to work
Use LDAP functions to sync or manage LDAP entries in the database directly, you
can use other data at the same time (check if employee is employed before
granting access to some App for example).

27. DBMS_LDAP
Example: Disabling a user in the LDAP directory
…
retval PLS_INTEGER:=-1;
l_session DBMS_LDAP.SESSION;
emp_array DBMS_LDAP.MOD_ARRAY;
emp_vals DBMS_LDAP.STRING_COLLECTION;
BEGIN
DBMS_LDAP.USE_EXCEPTION := true;
l_session := DBMS_LDAP.INIT ('yourldaphostnameIP', 389 ) ;
retval :=DBMS_LDAP.SIMPLE_BIND_S (l_session, 'ldapaccount', 'ldapaccountpwd') ;
emp_array := DBMS_LDAP.CREATE_MOD_ARRAY(20);
emp_vals(1) := ‘userlogin’;
DBMS_LDAP.POPULATE_MOD_ARRAY(emp_array,DBMS_LDAP.MOD_REPLACE,'uid',emp_vals);
DBMS_LDAP.POPULATE_MOD_ARRAY(emp_array,DBMS_LDAP.MOD_REPLACE,'cn',emp_vals);
emp_vals(1) := 'DISABLED';
DBMS_LDAP.POPULATE_MOD_ARRAY(emp_array,DBMS_LDAP.MOD_REPLACE,'orclisenabled',emp_vals);
retval := DBMS_LDAP.MODIFY_S(l_session, ‘cn=userlogin,cn=users,dc=yourdomain,dc=mu',emp_array);
DBMS_LDAP.FREE_MOD_ARRAY(emp_array);
retval := DBMS_LDAP.UNBIND_S(l_session);
Exception when others then
retval := DBMS_LDAP.UNBIND_S (l_session);
return 'Error : '||sqlerrm;
END;

28. DECLARE
c utl_tcp.connection; -- TCP/IP connection to the Web server
ret_val pls_integer;
TCPresult varchar2(1000);
BEGIN
c := utl_tcp.open_connection -- open TCP connection
(remote_host => 'www.acme.com', remote_port => 80, charset => 'US7ASCII');
ret_val := utl_tcp.write_line(c, 'GET / HTTP/1.0'); -- send HTTP
request ret_val := utl_tcp.write_line(c);
BEGIN LOOP
TCPresult:=utl_tcp.get_line(c, TRUE)); -- read result
…. do something ….
END LOOP;
EXCEPTION WHEN utl_tcp.end_of_input THEN NULL; -- end of input
END;
utl_tcp.close_connection(c);
END;
UTL_TCP
Example: requesting an URL and reading the content

29. UTL_SMTP
DECLARE
c UTL_SMTP.CONNECTION;
PROCEDURE send_header(name IN VARCHAR2, header IN VARCHAR2)
AS
BEGIN
UTL_SMTP.WRITE_DATA(c, name || ': ' || header || UTL_TCP.CRLF);
END;
BEGIN
c := UTL_SMTP.OPEN_CONNECTION(‘mysmtpserver.dom.com');
UTL_SMTP.HELO(c, ‘dom.com');
UTL_SMTP.MAIL(c, 'sender@dom.com');
UTL_SMTP.RCPT(c, 'recipient@some.com');
UTL_SMTP.OPEN_DATA(c);
send_header('From', '"Sender" <sender@xyz.com>');
send_header('To', '"Recipient" <recipient@abc.com>');
send_header('Subject', 'Hello');
UTL_SMTP.WRITE_DATA(c, UTL_TCP.CRLF || 'Hello, world!');
UTL_SMTP.CLOSE_DATA(c);
UTL_SMTP.QUIT(c);
EXCEPTION
WHEN UTL_SMTP.TRANSIENT_ERROR or
UTL_SMTP.PERMANENT_ERROR
THEN
BEGIN
UTL_SMTP.QUIT(c);
EXCEPTION
WHEN UTL_SMTP.TRANSIENT_ERROR or
UTL_SMTP.PERMANENT_ERROR
THEN null;
END;
raise_application_error(-20000,
'Failed to send mail due to the following error: ' ||
sqlerrm);
END;

30. UTL_MAIL
UTL_MAIL.SEND (sender => ‘mymail@mail1.com’,
recipients => ‘yourmail@mail2.com,
cc => ‘hismail1@mail3.com’,
bcc => ‘hismail2@mail4.com’,
subject => ’My Subject’,
message => ’My message text’,
mime_type => ‘text/plain; charset=us-ascii’);

31. UTL_HTTP
Example: getting map info from Google Maps API
l_url VARCHAR2(2000);
l_response VARCHAR2(4000):=null;
myString VARCHAR2(4000);
resp_value VARCHAR2(4000);
BEGIN
l_url := 'http://maps.googleapis.com/maps/api/whatever_api_path_and_param';
begin
req := UTL_HTTP.BEGIN_REQUEST(l_url);
resp := UTL_HTTP.GET_RESPONSE(req);
loop
UTL_HTTP.READ_LINE(resp, resp_value, TRUE);
l_response := l_response || resp_value;
end loop;
UTL_HTTP.END_RESPONSE(resp);
exception
when UTL_HTTP.END_OF_BODY then
UTL_HTTP.END_RESPONSE(resp);
end;
myString:=XMLType(l_response).extract('xmlelement/xmlelementb/xmlelement3()').getStringVal();
END;
XMLType is a system-defined
opaque type for handling XML
data, with predefined member
functions like extract to extract
XML nodes and fragments.

32. HTTPURITYPE
Another Way to easily retrieve HTTP content
select HTTPURITYPE('http://somesite.com/path').getxml()
FROM dual;
Output :
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet href="http://somesite.com ...
SELECT HTTPURITYPE('http://google.com/').getClob()
FROM dual;
Output :
<html><head><meta http-equiv="content-type" content="text/html; charset=ISO-8859 ...

33. <!DOCTYPE html>
<!-- paulirish.com/2008/conditional-stylesheets-vs-css-hacks-answer-neither/ -->
<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7" lang="en"> <![endif]-->
<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8" lang="en"> <![endif]-->
<!--[if IE 8]> <html class="no-js lt-ie9" lang="en"> <![endif]-->
<!-- Consider adding a manifest.appcache: h5bp.com/d/Offline -->
<!--[if gt IE 8]><!-->
<html class="no-js" lang="en">
<!--<![endif]-->
<head>
<link rel="dns-prefetch" href="//ajax.googleapis.com">
<!-- Mobile viewport optimized: h5bp.com/viewport -->
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Developers Conference 2015 - Mauritius</title>
<meta name="description" content="The Developers Conference 2015 is a 3-day event going to take place between the 23rd and the 25th of April 2015.
The Mauritius Software Craftsmanship Community (MSCC) is going to organise this event in corporation with local and international sponsors and partners.">
<meta name="keywords" content="devconmru, developers conference mauritius, gab2015, global azure bootcamp, user group, community, call for papers, internet
of things" />
<!-- Place favicon.ico and mscc-touch-icon.png in the root directory: mathiasbynens.be/notes/touch-icons -->
<link rel="shortcut icon" href="favicon.ico" type="image/x-icon" />
<!-- Chrome Touch Icons -->
<meta name="mobile-web-app-capable" content="yes">
<link rel="icon" sizes="192x192" href="img/mscc-touch-icon-192x192.png">
<link rel="manifest" href="manifest.json">
<!-- Apple Touch Icons -->
SELECT HTTPURITYPE('http://www.devconmru.org/').getClob() FROM dual;

34. JSON (JavaScript Object Notation) support
Example : a google translate response for
http://ajax.googleapis.com/ajax/services/language/translate?v=1.0....
SELECT HTTPURITYPE('http://ajax.googleapis.com/…').getClob()
FROM dual;
You can store the JSON document returned in a table (here my_json_doc) :
CREATE TABLE json_doc ( id number, json_data CLOB, CONSTRAINT
json_doc_json_data_chk CHECK (json_data IS JSON) );
IS JSON will automatically check that the JSON document syntax is compliant.
SELECT T.json_data.FirstName, T.json_data.LastName, JSON_QUERY(T.json_data,
'$.ContactDetails') FROM json_doc T ORDER BY T.json_data.FirstName, T.json_data.Last_name;
Output :
James Bond {"Email":"james.bond@dom.com","Phone ":""}
Lorem Ipsum {"Email": "lorem.ipsum@dom2.com","Phone" :"44 123 123456","Twitter":"@lorem"}

35. JSON (JavaScript Object Notation) support
Another example of a web API call this time with a single SELECT
SELECT JSON_QUERY(T.json_data, '$.ContactDetails') from
(SELECT
HTTPURITYPE('http://ajax.googleapis.com/…').getClob() json_data
FROM mytable)
From dual;
Output :
James Bond {"Email":"james.bond@dom.com","Phone ":""}
Lorem Ipsum {"Email":« lorem.ipsum@dom2.com","Phone" :"44 123 123456","Twitter":"@lorem"}

36. AND MANY MORE ….

37. TO CONCLUDE
Database can do much more than just storing data
Keep data near the database tier as much as possible
Leverage your database to mask complexity and to surface your data
ready for consumption
Use Database Views and Packages with an API approach to encapsulate
database functionality
Use Database functionality to reach out to external data

38. THANK YOU
Enjoy the rest of your day
@ DEVCONMRU