Advanced Web Programming outside of APEX

Advanced Web Programming outside of APEX
Filipe Silva
Silva . Filipe @ gmail . com
Faculdade de Engenharia - Universidade do Porto
Portugal

Advanced Web Programming outside of beyond APEX
Filipe Silva
Portugal

Brief review of PL/SQL Web Toolkit
Web Request processing
Security
Error management
Improve DB use
File upload/ download and generation
Best Practices
PL/SQL Gateways alternatives
Topics

Some Examples

Main Aplication:
Almost 2 million PL/SQL code lines
4 000 Java lines
120 000 calls to HTP
Framework:
240 000 PL/SQL code lines
1 300 Java lines
11 000 calls to HTP
Statistics for SIGARRA©
Documents generated
(at FEUP)
September
17-9-2010 : 1 230 493 (high)
05-09-2010: 196 595 (low)
month average: 555 868 /day

The HTTP Server receives a request from a client browser (1)
…routes the request to the plsql gateway (adapter)
…that ”maps” the request to an Oracle DB call (2)
Web Request Processing
http://<http_server_name>:<port>/<dad>/<procedure>?<var1>=<value_var1>

Web Request Processing (cont.)
Use of PL/SQL Web Toolkit will create an output in a buffer array (3)
If no unhandled exception was raise a COMMIT is made at the end of the PL/SQL call
The plsql Gateway gets the OWA buffer content and sends it to the HTTP Server (some sanitization is made) (4)
…thatdeliverstheresult to theclient. (5)

Allows to generate web content directly from Oracle (since 8.1.6)
Has direct access to the DB (no-round trip)
It’s easy to learn
Keeps all the code in the same place (backups,…)
You can keep all the files in DB
PL/SQL Web Toolkit

PL/SQL Web Toolkit
HTP – write to OWA (Web) output
HTF – construct HTML (mainly)
OWA_CACHE - writescachingheadertags
OWA_UTIL - retrievesenvironment variables, write header,...
OWA_COOKIE – send and receive browser cookies
OWA_OPT_LOCK –handles optimistic lock
OWA_IMAGE – get coordinates from user click in image map
OWA_SEC – security programs
OWA_CUSTOM - authorize function
OWA_PATTERN – pattern matching
OWA_MATCH – pattern matching
OWA_TEXT - stringmanipulation
WPG_DOCLOAD – BLOBs and BFILEs downloads

HTP.p
OWA_UTIL.mime_header
OWA_UTIL.redirect_url
OWA_UTIL.status_line
OWA_UTIL.http_header_close
OWA_UTIL.get_procedure
OWA_UTIL.get_cgi_env
WPG_DOCLOAD.download_file
PL/SQL Web Toolkit
OWA_COOKIE.send
OWA_COOKIE.get
HTF.escape_sc

DECLARE
la_thepage SYS.HTP.htbuf_arr;
li_irows INTEGER; li_version INTEGER;
la_name_arr OWA.vc_arr; la_value_arr OWA.vc_arr; li_num PLS_INTEGER := 0;
BEGIN
li_version := OWA.initialize;
OWA.init_cgi_env (li_num, la_name_arr, la_value_arr);
HTP.p ('My output'); -- or call to a procedure that will generated HTP like calls
irows := 99999999999;
OWA.get_page (thepage => la_thepage, irows => li_irows);
FOR i IN 1 .. li_irows
LOOP
DBMS_OUTPUT.put_line (la_thepage (i));
ENDLOOP;
END;
Test Output

DECLARE
(...)
lv_cookies VARCHAR2 (2000) := 'TESTE=Teste;';
BEGIN
li_version := OWA.initialize;
OWA_COOKIE.init; -- so it will reload the cookies buffer
la_name_arr (1) := 'HTTP_COOKIE';
la_value_arr (1) := lv_cookies;
li_num := li_num +1;
OWA.init_cgi_env (li_num, la_name_arr, la_value_arr);
my_program();
irows := 99999999999;
OWA.get_page (thepage => la_thepage, irows => li_irows);
(...)
END;
Test Output with cookies
OWA_COOKIE.cookie
RECORD (
name varchar2(4096),
vals vc_arr,
num_vals integer
)
PROCEDURE my_program IS
(...)
lrec_cookieOWA_COOKIE.cookie
BEGIN
(...)
lrec_cookie := OWA_COOKIE.get ('TESTE');
IF lrec_cookie.num_vals > 0
THEN
HTP.p ('Cookie value:' || lrec_cookie.vals (1));
ENDIF;
(...)
END;

Needs ACL (access control list) permissions in 11g
Allows HTTP callouts from PL/SQL and SQL
Takes a URL as parameter and returns corresponding Webpage
Request :data is returned in the form of pieces, each piece is varchar2(2000)
Request Pieces: all pieces obtained at once as a table type of Varchar2(2000)
UTL_HTTP

DECLARE
l_request UTL_HTTP.req; l_responseUTL_HTTP.resp;
(…)
BEGIN
UTL_HTTP.set_proxy (proxy => ‘myproxy‘, no_proxy_domains => ‘dom1,dom2');
l_request := UTL_HTTP.begin_request ('http://someurl’ ,method => 'GET');
UTL_HTTP.set_header (l_request, 'User-Agent', 'Mozilla/4.0');
l_response := UTL_HTTP.get_response (l_request);
DBMS_OUTPUT.put_line (' HTTP response: status code: ' || l_response.status_code ||
' reason phrase: ' || l_response.reason_phrase);
read_header ();
read_lines ();
UTL_HTTP.end_response (l_response);
END;
UTL_HTTP
PROCEDURE read_header IS
lv_header_name VARCHAR2 (256);
lv_header_value VARCHAR2 (1024);
BEGIN
FOR i IN 1 .. UTL_HTTP.get_header_count (l_response)
LOOP
UTL_HTTP.get_header (l_response, i,
lv_header_name, lv_header_value);
DBMS_OUTPUT.put_line (lv_header_name || ': ' || lv_header_value);
END LOOP;
END read_header;
PROCEDURE read_lines IS
lv_line VARCHAR2 (32767);
BEGIN
LOOP
UTL_HTTP.read_line (l_response, lv_line);
DBMS_OUTPUT.put_line (lv_line);
END LOOP;
EXCEPTION
WHEN UTL_HTTP.end_of_body
THEN
NULL;
END read_lines;

Read RSS
ReadWebservices (REST, …)
ReadWebpages
Get Documents via URL
Example: crawling sites to index by Oracle Text
…
UTL_HTTP: Uses

HTTP is a “stateless” protocol
Means credentials have to go with every request (i.e. cookies)
Should use SSL for everything requiring authentication (https)
Important answer status
200 – ok
204 – no content
304 – not modified
403 – forbidden
404 – not found
About HTTP

Improve DB use (1)
HTTP Requests Types
HTTP request types supported:
GET – parameters passed by query string (URL)
POST – parameters passed by HTTP Request body
(large amount of data, file uploads, password forms, etc..)
HEAD – like GET but web server only returns header info
So...only return header info and exit the procedure!
(…) –- send header
IF OWA_UTIL.get_cgi_env ('REQUEST_METHOD') = ‘HEAD'
THEN
RETURN;
END IF;
(…) –- send content

DAD configuration
PlsqlErrorStyle – use ApacheStyle in production systems
PlsqlExclusionList - Blacklist
default: sys.*, dbms_*, utl_*, owa_*, owa.*, htp.*, htf.*
PlsqlRequestValidationFunction – blacklist or whitelist
booleanfunction_name (procedure_name IN varchar2)
Security MOD_PLSQL

FUNCTIONmy_validation_check (procedure_nameINVARCHAR2)
RETURNBOOLEAN
IS
li_check PLS_INTEGER;
BEGIN
SELECT 1
INTO li_check
FROM my_pages
WHERE name = UPPER (procedure_name);
RETURNTRUE;
EXCEPTION
WHEN NO_DATA_FOUND
THEN
RETURNFALSE;
END my_validation_check;
PlsqlRequestValidationFunction

Use a whitelist Web procedure table
Best practices (0)

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html>
<head>
<title>Hello world</title>
<link rel="stylesheet" href="mystyle.css">
</head>
<body>
Hello World!
</body>
</html>
HTML

Why?
HTML code in lowercase letters, and NEVER skip closing tags (like </p>)
Parsing errors => stop parse!
Not supported by totally of HTF (and HTP)
XHTML

Make your own HTP and HTF packages
The new HTP must call sys.HTP to send the buffer to the OWA Output and use the same buffer types
Create procedures for new tags, opening and closing tags
Delete/comment obsolete tags
XHTML complain (lower case, close tags, …)
Add ‘class’ attribute to tags (…)
Allow buffer control
Read without changing it (or clean it)
Manipulation
For instance, if you need to get some storage buffer (3 columns divs problem)
Best practices (1)

Text files
CSS
JSON
http://reseau.erasme.org/pl-sql-library-for-JSON
http://sourceforge.net/projects/pljson/
etc..
XML
RSS
http://tylermuth.wordpress.com/2008/01/22/producing-rss-from-plsql/
etc..
What else?

PDF
PL_FPDF - http://www.erasme.org/PL-FPDF,1337
PL/PDF - http://plpdf.com/ €
AS_PDF - http://technology.amis.nl/blog/8650/as_pdf-generating-a-pdf-document-with-some-plsql
Youcanalsomanipulateimage files using Oracle interMedia
To createthumbnails
Putwatermark (11.2)
…
What else?

QrCode
http://sourceforge.net/projects/oracle-jutils/
ZIP (BLOBs)
What else?

owa_util.mime_header('application/vnd.ms-excel');
Use SYLK or even HTML!!
SYLK is a text-based interchange format for spreadsheets; it supports formulas, borders, fonts, point sizes, etc.
Or CSV (Comma Separated File) (mime-type: text/csv)
PL/SQL ExcelDocumentType
http://www.jasonsdevelopercorner.com/?page_id=8
PL/XLS-XML €
http://plpdf.com/
Excel

SVG (XML)
FusionCharts http://www.jasonsdevelopercorner.com/?page_id=22
Needgraphics?

DAD configuration: PlsqlDocumentTablename - the table name for storing documents when file uploads are performed through the DAD
Table definition:
NAME VARCHAR(128) UNIQUE NOT NULL,
MIME_TYPE VARCHAR(128),
DOC_SIZE NUMBER,
DAD_CHARSET VARCHAR(128),
LAST_UPDATED DATE,
CONTENT_TYPE VARCHAR(128),
[[CONTENT LONG RAW], -- files with extensions in PlsqlUploadAsLongRaw
[BLOB_CONTENT BLOB]] – other files
(...) – can have more columns
File upload

HTML:
<form enctype="multipart/form-data" action=“my_pack.my_proc" method="POST">
<input type="file" name=“p_my_file“ />
(...)
In DB:
procedure my_proc (p_my_file IN varchar2, …) is
lrt_docdocuments_table%ROWTYPE;
begin
select *
into lrt_doc
from documents_table
where name=p_my_file;
(…)
File upload

Do not keep documents in the uploaded documents table
Avoid collisions
Provide distinct access privileges
Backup polices
Distinct Business logic
…
Best practice (2)

1) Send Header info
OWA_UTIL.mime_header(prt_doc.mime_type, FALSE);
-- specify mime_type and do not close header
HTP.p ('Last-Modified: ' ||pv_last_updated); -- allow cache
HTP.p ('Content-Length: ' ||prt_doc.doc_size); -- download info
HTP.p ('Content-disposition: attachment; filename=“'||
prt_doc.filename||'” '); -- “save As”
OWA_UTIL.http_header_close; --close header
File download

2) Send Data
2a) If BLOB call:
DECLARE
lb_lob BLOB := prt_doc.content; -- due to need to be an IN OUT
BEGIN
WPG_DOCLOAD.download_file (p_blob => lb_lob);
END;
2b) If CLOB:
DECLARE
lb_lob BLOB := clob_util.clob2blob (p_clob);
BEGIN
WPG_DOCLOAD.download_file (p_blob => lb_lob);
END;
File download

DECLARE
offset PLS_INTEGER;
len PLS_INTEGER;
amount PLS_INTEGER := 32000;
BEGIN
len := NVL (DBMS_LOB.getlength (c_lob), 0);
FOR i IN0 .. len / amount
LOOP
offset := (amount * i) + 1;
HTP.prn(DBMS_LOB.SUBSTR(c_lob, amount , offset));
ENDLOOP;
END;
File download
Or2b) If CLOB

FUNCTION clob2blob (p_clob IN CLOB) RETURN BLOB DETERMINISTICIS
lbl_blob BLOB;
li_dest_offset INTEGER := 1; li_src_offset INTEGER := 1;
li_amount INTEGER := DBMS_LOB.lobmaxsize;
ln_blob_csid NUMBER := DBMS_LOB.default_csid;
li_lang_ctx INTEGER := DBMS_LOB.default_lang_ctx;
li_warning INTEGER;
BEGIN
DBMS_LOB.createtemporary (lob_loc => lbl_blob, cache => TRUE, dur => DBMS_LOB.call);
DBMS_LOB.converttoblob (dest_lob => lbl_blob
,src_clob => p_clob
,amount => li_amount
,dest_offset => li_dest_offset
,src_offset => li_src_offset
,blob_csid => ln_blob_csid
,lang_context => li_lang_ctx
,warning => li_warning);
RETURN lbl_blob;
END clob2blob;
CLOB 2 BLOB

An Example:
OWA_UTIL.mime_header ('text/css', FALSE); -- sends mime-type
-- and doesn’t closes the header
HTP.P ('Last-Modified: ' || lv_updated_date);
HTP.P ('Cache-Control: public, s-maxage=300, max-age=86400');
-- or
-- HTP.p ('Cache-Control: private, max-age=86400');
--or
-- HTP.p (‘Cache-Control: no-cache’); --or even no-store
OWA_UTIL.http_header_close; --close the header
(…)
Cache – Last Modified

-- Thu, 30 Sep 2010 16:03:22 GMT
lv_updated_date :=TO_CHAR (
SYS_EXTRACT_UTC (
CAST (NVL (pd_last_modified, SYSDATE)
AS TIMESTAMP)),
'Dy, DD Mon YYYY HH24:MI:SS',
'NLS_DATE_LANGUAGE=''AMERICAN''')
|| ' GMT';
Last-Modified

Then I can check if the browser has the document in cache
IF OWA_UTIL.get_cgi_env ('HTTP_IF_MODIFIED_SINCE') =
lv_updated_date
THEN
OWA_UTIL.status_line (nstatus => 304,
creason => 'Not Modified',
bclose_header => TRUE);
RETURN;
END IF;
Improve DB use (2)

Can also use ETag:
HTP.p('ETag: ' || lv_file_etag);
-- or owa_cache.set_cache(p_etag, p_level);
-- length(lv_file_etag)<= 55
And to check:
OWA_UTIL.get_cgi_env('HTTP_IF_NONE_MATCH')
--needs PlsqlCGIEnvironmentList HTTP_IF_NONE_MATCH
Cache - ETag

Pseudo-Code
Check if it’s in client cache
-> if so return status 503 -> exit
Output HTTP (Response) header info
If HEAD request
-> exit
Output Content
Best Practices (3)Improve DB use (1+2)

If a page can only be access by a POST form -> check it!
owa_util.get_cgi_env( 'REQUEST_METHOD' ) =‘POST’ and owa_util.get_cgi_env( 'HTTP_REFERER‘) is not null
-- maybe even check that the referer is from the same site or an exact page …
If a page is only to be access by a AJAX -> check it!
OWA_UTIL.get_cgi_env ('X-Requested-With') is not null
-- needs DAD PlsqlCGIEnvironmentList X-Requested-With
But an hacker can avoid that if they know of it 
Disallow Access

begin
(…)
exceptions
(…)
whenothersthen
qem_web.register_error;
end;
Error Manager

Error Manager

Avoid reporting system messages that the normal user cannot understand but an hacker could use (Error message SQL Injection). (i.e SQLERRM, …)
The errors should be reported to a system maintainer even if the user doesn’t report them.
Should have enough information to replicate the error.
We based our solution on Quest Error Manager (http://qem.inside.quest.com/index.jspa)
Error Manager

Quest Error Manager
Log 4 PLSQL
Logger
PL/SQL Log & DBugLibrary
PLJ_LG
…
Error and Log Libraries

Have an error handler for when others in every procedure that will generate your web pages
The ones that are to called by URL
They are in a whitelist table, right?
With PL/Scope (11g) you can have a script that check the use of your error report program inside the procedures
Best Pratices (4)

The OWASP Top 10 Web Application Security Risks for 2010 are:
A1: Injection
A2: Cross-Site Scripting (XSS)
A3: Broken Authentication and Session Management
A4: Insecure Direct Object References
A5: Cross-Site Request Forgery (CSRF)
A6: Security Misconfiguration
A7: Insecure Cryptographic Storage
A8: Failure to Restrict URL Access
A9: Insufficient Transport Layer Protection
A10: Unvalidated Redirects and Forwards
Security

One of the more devastating attacks on a web application
One of the most common attacks
Probably the most popular technique used for database intrusion caused by inefficient input validation
Some real World examples (2005-2010):
http://en.wikipedia.org/wiki/SQL_injection#Real-world_examples
SQL injection: what?

PROCEDUREmy_bad_proc (v inVARCHAR2) IS
TYPE cursortyp ISREFCURSOR;
cursordual cursortyp;
result_v VARCHAR2 (300);
sql_stmt VARCHAR2 (300);
BEGIN
sql_stmt := 'select * from dual where dummy=''' || v || ''';
DBMS_OUTPUT.put_line (sql_stmt);
OPEN cursordual FOR sql_stmt;
LOOP
FETCH cursordual INTO result_v;
EXITWHEN cursordual%NOTFOUND;
DBMS_OUTPUT.put_line (result_v);
ENDLOOP;
END my_bad_proc;
Sql injection: example
BEGIN
my_bad_proc(v=>
'aaa''
union
select login from logins --
END;
select * from dual where dummy='aaa'
union
select login from logins --
……
Mark
Steven
(…)

In Oracle when concatenating non-constants to be used as a statement in
EXECUTE IMMEDIATE v_string;
OPEN v_cursor FOR v_string;
DBMS_SQL.parse (..., v_string, …)
Safe code:
SELECT … INTO …
And above if not concatenating with non- constants
OPEN … USING var1, var2,…
EXECUTE IMMEDIATE …. USING var1, var2,…
DBMS_SQL.bind_variable
SQL Injection: where?

We need to convert the Text value to a SQL text Literal.
If needed by design replace all the quotation marks with two quotations
replace(v,'''','''''')
-- for instance to allow O’Neil
Use DBMS_ASSERT.ENQUOTE_LITERAL:
Enclose the literal with quotation
Check with DBMS_ASSERT.enquote_literal the validy catching the exception VALUE_ERROR it raises if invalid
OR
Use DBMS_ASSERT.enquote_literal to enclose the literal with in quotation marks and it raises VALUE_ERROR if invalid.
Best Pratices (4)

PROCEDUREmy_good_proc1 (v inVARCHAR2) IS
(…)
lv_literal VARCHAR2(100):='''' || REPLACE (v, '''', '''''') || '''';
BEGIN
BEGIN
lv_literal :=sys.DBMS_ASSERT.enquote_literal(lv_literal );
EXCEPTION
WHENVALUE_ERROR
THENDBMS_OUTPUT.put_line ('Value entered invalid');
RETURN;
END;
sql_stmt := 'select * from dual where dummy=' || lv_literal;
OPEN cursordual FOR sql_stmt;
(…)
END my_good_proc1;
DBMS_ASSERT.enquote_literal (1)

PROCEDUREmy_good_proc2(v VARCHAR2)
IS
(…)
BEGIN
BEGIN
sql_stmt := 'select * from dual where dummy='
||sys.DBMS_ASSERT.enquote_literal(v);
EXCEPTION
WHENVALUE_ERROR
THEN
DBMS_OUTPUT.put_line ('Value entered invalid');
RETURN;
END;
OPEN cursordual FORsql_stmt;
(...)
END my_good_proc2;

PROCEDUREmy_good_proc2(v VARCHAR2)
IS
(…)
BEGIN
BEGIN
sql_stmt := 'select * from dual where dummy=‘
||sys.DBMS_ASSERT.enquote_literal(REPLACE (v, '''', '''''') );
EXCEPTION
WHENVALUE_ERROR
THEN
DBMS_OUTPUT.put_line ('Value entered invalid');
RETURN;
END;
OPEN cursordual FORsql_stmt;
(...)
END my_good_proc2;

DBMS_ASSERT
SIMPLE_SQL_NAME
QUALIFIED_SQL_NAME
SCHEMA_NAME
SQL_OBJECT_NAME
ENQUOTE_LITERAL – single quotes check
ENQUOTE_NAME – double quotes check

Protect the database and web server (patches)
Use minimum privileges
AUTHID CURRENT_USER when possible
Use of prepared statement
Data API
Sanitize Input:
from users
URL parameters
Cookies
avoid cursor injection (11g):
DBMS_SQL.open_cursor(security_level => 2);
SQL Injection: Prevention

A1: Injection
Security

XSS is a form of injection where the interpreter is the browser and attacks are “buried” in the HTML document
Cross-Site Scripting (XSS)

Sanitize input
HTML Escapeuntrusted data (htf.escape_sc)
if HTML input allowed…check it
Use the new Content Security Policy header
Examples:
If no external script and no inline-script
X-Content-Security-Policy: allow 'self';
If inline script exists but no external script
X-Content-Security-Policy: allow 'self'; options inline-script;
Cross-Site Scripting (XSS): Prevention

Avoid iframe/frame
X-Frame-Options: [deny | sameOrigin]
X-Content-Security-Policy: frame-src: ...
HTTPOnly cookie flag (OAS version >= 10.1.3.5.0)
...
Cross-Site Scripting (XSS): Prevention (cont.)

A1: Injection
Security

Password Strength
Password storage encrypted
Login attemps restricted (by time/tries)
Avoid Cross-Site Scripting (XSS)
No session Id in URL
Protecting Credentials in Transit: cookies secure (HTTPS)
AUTOCOMPLETE=OFF in authentication forms
…
Broken Authentication and Session Management : Prevention

DBMS_EXPORT_EXTENSION
DBMS_CDC_IMPDP
DBMS_CDC_ISUBSCRIBE
DBMS_CDC_SUBSCRIBE
DRILOAD
SDO_TOPO_DROP_FTBL SQL
LT
DBMS_METADATA
ALTER SESSION SET NLS_DATE_FORMAT (lateral SQL Injection)
Advice: check Oracle Security bugs

Mod_plsql in Oracle HTTP Server (since 8i) (Build on Apache 1.3 or 2.2)
Embedded PL/SQL Gateway (DBMS_EPG)
Apex Listener
Doug McMahon’s Mod_owa
Thoth Gateway in Microsoft Internet Information Server
DBPrism Servlet engine (Tomcat,…)
JOPA Gateway Servlet(Tomcat,…)
Total Knowledge’s Mod_plsql
Alternative PL/SQL Gateways

Web Request processing
File upload/ download and generation
Best Practices
HTP/HTF replacer
Error management
Improve DB use (HEAD, client file caching)
Security
Black listing, white listing
How to avoid SQL Injection
PL/SQL Gateways alternatives
Summary

Questions?
Filipe Silva
Blog: http://oracleblues.blogspot.com Label: Oracle Blues
Portugal
Thanks for listening!

http://oracleblues.blogspot.com	51
http://oracleblues.blogspot.in	1
http://www.symbaloo.com	1

Advanced Web Programming outside of APEX

by Filipe Silva

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 53

Statistics