Ogh Ace Case, Part 1 and 2, Oracle Xml Database, Marco Gralike

OGh Ace Case Oracle XML Database – Part 1 Marco Gralike – AMIS   1

A shortlist…  Started as DBA with Oracle 7 on Windows NT 3.1 (1994)  Experienced with Oracle 6 / 7.x / 8.x / 9.x / 10.x and 11.1  Started with Oracle XMLDB in 2003 (Oracle 10.1.3)  Oracle 11g Beta tester for Oracle XMLDB  Dedicated XMLDB Blog (blog.gralike.com)  Active Oracle OTN XMLDB Forum Member  Oracle ACE Award for XMLDB Community Contributions   2

Overview  What is Oracle XML Database  The Protocol Server  Configuration and Basic Concepts  HTTP API’s  PL/SQL Gateways, ORADB, NDWS  URITypes in the Database  DBUri-, HTTPUri- and XDBUriTypes  Combining Worlds  Selecting and Saving of Files Outside the Database  Repository Events   3

Storing XML in Oracle XML Database  File Storage  XML Database Oracle XML DB  Native  A World within a World  XML Enabled  Complies to Standards  License Free  Core part of (OXJR)DBMS  The (relational) foundation is there and can be used for solving XML “imperfections”   4

Oracle XML Milestones / Importance 11gR1 XDB 10gR2 10gR1 9.2.0.3.0 XDK 9i 8iR1 XMLType Performance XQuery Binary XML SQL/XML Robustness XMLIndex, NDWS   5

XMLType  XML data exchange  SQL, C, C++, Java, PL/SQL, .Net, etc  Via Protocol Listener supported methods  Via Oracle Advanced Queuing methods (Oracle 11gR1)  XML data storage  CLOB, Object Relational, Binary XML (11gR1)  XML data validation  XMLSchema, Programmatically, via Storage definitions  XML data creation  XML Operators, Packages   6

The Protocol Server   7

XMLType storage and more Out-of-the-Box Interfacing Logical Design Level Security, Management, Versioning Physical Design Level   8

Protocol Server - Detail * Listener listener.ora or database parameters * Shared Server * Protocol “Adapter” xdbconfig.xml xdb.xdb$config   9

Configuration of XDBCONFIG.xml file  PL/SQL Control  DBMS_XDB.GetHttpPort(), DBMS_XDB.GetFTPPort()  DBMS_XDB.SetHttpPort(), DBMS_XDB.SetFTPPort()  WebDAV  XMLSpy  JDeveloper  Drag & Drop  Notepad  MS Word   10

Protocol Server Access WebDAV FTP HTTP(s)   11

Enable Protocol Server Functionality demo01.sql   12

HTTP API‟s   13

Protocol Server HTTP API‟s  Your Own…  Java Servlets, PL/SQL Gateway  PL/SQL Gateways  APEX (/apex/*) PL/SQL  “Servlets”  Native Database WebService (/orawsv/*) C  ReportFmwkServlet (/orarep/*) C  TestServlet (/Test) Java  DBURI (/oradb/*) C   14

Native Database Web Services (NDWS) WSDL Structure Architecture   15

Native Database WebService (NDWS) 1. Enable the Protocol Server for HTTP access 2. Enable the orawsv entry point in xdbconfig.xml 3. Create a user and grant access  DBMS_NETWORK_ACL_ADMIN  XDB_WEBSERVICES  XDB_WEBSERVICES_OVER_HTTP  XDB_WEBSERVICES_WITH_PUBLIC 4. Create your procedure  For instance a procedure called OBJECTTYPES 5. Test (demo02.sql)  http://localhost:8080/orawsv/SCOTT/OBJECTTYPES?wsdl   16

DBURI Servlet  Access Any Table or View From a Browser  DBUriType  Security is handled via DB roles  Transformation via XSL  Query options via XPath Expressions http://localhost:8080/public/demo03/   17

URIType„s in the Database   18

URITypes  DBUriType  Represents an URI that targets database data  HTTPUriType  Represents an URL that starts with http://  Uses UTL_HTTP  XDBUriType  Represents an URI that targets a resource in the Repository   19

HTTPUriType - Google API SQL> SELECT extractvalue(value(t),'/Hostip/gml:name' 2 , 'xmlns:gml=quot;http://www.opengis.net/gmlquot; 3 xmlns=quot;http://www.hostip.info/apiquot;') AS quot;Namequot; 4 , extractvalue(value(t),'/Hostip/countryName' 5 , 'xmlns:gml=quot;http://www.opengis.net/gmlquot; 6 xmlns=quot;http://www.hostip.info/apiquot;') AS quot;CNamequot; 7 , extractvalue(value(t),'/Hostip/countryAbbrev' 8 , 'xmlns:gml=quot;http://www.opengis.net/gmlquot; 9 xmlns=quot;http://www.hostip.info/apiquot;') AS quot;CAbbrevquot; 10 , extractvalue(value(t) ,'/Hostip/ipLocation/gml:PointProperty/gml:Point/gml:coordinates' 11 , 'xmlns:gml=quot;http://www.opengis.net/gmlquot; 12 xmlns=quot;http://www.hostip.info/apiquot;') AS quot;Coord.quot; 13 FROM TABLE (XMLSEQUENCE (EXTRACT (HTTPURITYPE ('http://api.hostip.info/?ip=141.146.8.66').getxml () 14 , '/HostipLookupResultSet/gml:featureMember/Hostip' 15 , 'xmlns:gml=quot;http://www.opengis.net/gmlquot; 16 xmlns=quot;http://www.hostip.info/apiquot;' ))) t; Name CName CAbbrev Coordinates -------------------- -------------------- -------------------- -------------------- Redwood City, CA UNITED STATES US -122.206,37.5164   20

HTTPUriType – Accessing RSS Feeds SQL> SELECT * 2 FROM XMLTABLE (XMLNAMESPACES('http://purl.org/dc/elements/1.1/' AS quot;MGRquot;) 3 ,'//item' 4 passing HTTPURITYPE('http://my.blog.com/blog/?feed=rss2').getXML() 5 columns title varchar2(50) path '/item/title/text()', 6 link varchar2(50) path '/item/link/text()', 7 publication_date varchar2(50) path '/item/pubDate/text()', 8 creator varchar2(50) path '/item/MGR:creator/text()', 9 description varchar2(250) path '/item/description/text()', 10 category XMLTYPE path '/item/category/text()');   21

XML/SQL Output TITLE LINK PUBLICATION_DATE CREATOR ----------------------------------------------------------------- DESCRIPTION CATEGORY ----------------------------------------------------------------- Quering RSS Feeds The XMLDB Way http://feeds.feedburner.com/~r/Bloggralikecom/~3/3 Wed, 25 Jun 2008 16:47:19 +0000 Marco Gralike Actually this IS old stuff (2006), but it got lost IN a… < ![CDATA[11g]]>< ![CDATA[Howto]]> demo04.sql   22

XDBUriType  Useful for accessing resources in the XML DB Repository  Resources = files and folders  Helpful during XML Schema and XML Document registration  Helpful while insert XML document content from the XML DB Repository into XMLType Tables, Columns or Views   23

Combining Worlds   24

Combining BFileName with XML  The bfilename function returns a BFILE locator for a physical LOB binary file  Syntax: bfilename( 'directory', 'filename' )  Often used for inserting huge XML documents into the database  But you can do more…   25

Selecting XML Data outside the Database SQL> SELECT extract((XMLTYPE(bfilename('XMLSTORE','data.xml'),NLS_CHARSET_ID('AL32UTF8'))),'*') AS quot;XDATAquot; 2 FROM dual; XDATA ------------------------------------------------------- <root> <id>0</id><info><info_id>0</info_id> <info_content>Text</info_content></info> </root> demo06.sql SQL> SELECT xdata 2 FROM (XMLTABLE('*' 3 PASSING (XMLTYPE(bfilename('XMLSTORE','data.xml'),NLS_CHARSET_ID('AL32UTF8'))) 4 COLUMNS xdata xmltype PATH '/*' 5 ) 6 ); XDATA ------------------------------------------------------------------------------------- <root> <id>0</id><info><info_id>0</info_id><info_content>Text</info_content></info></root>   26

Using dbms_xmldom or dbms_xslprocessor  Ways to write to disk  dbms_xslprocessor.clob2file  dbms_xmldom.writetofile  Syntax: bfilename( 'directory', 'filename' )  Often used for inserting huge XML documents into the database  But you can do more…   27

Saving XML Data outside the Database SQL> declare 2 rc sys_refcursor; 3 BEGIN 4 open rc FOR SELECT * FROM ( SELECT rownum FROM dual connect BY level < 100 ); 5 dbms_xslprocessor.clob2file( xmltype( rc ).getclobval( ) , ‘STORE’,'anton.xml’); 6 END; 7/ SQL> declare 2 rc sys_refcursor; 3 doc DBMS_XMLDOM.DOMDocument; 4 begin 5 open rc for select * from ( select rownum from dual connect by level < 100 ); 6 doc := DBMS_XMLDOM.NewDOMDocument(xmltype( rc )); 7 DBMS_XMLDOM.WRITETOFILE(doc, ‘STORE/marco.xml’); 8 end; 9 / demo07.sql   28

Repository Events   29

Repository Events  Repository Events are events which get triggered via defined methods on the Repository For Example:  creating, deleting, locking, unlocking, placing under version control, checking in, checking out, unchecking out, opening, and/or updating  Repository Events <> Database Triggers  Are specially designed to act on Repository actions  Repository operation can be associated with one or more repository events   30

Repository Events  Event Listener  Java class or a PL/SQL package or an object type  A repository event listener can be configured for a particular resource or for the entire repository  It comprises a set of PL/SQL procedures or Java methods, each of which is called an event handler  Event Handler  Each event handler processes a single event  Resource Configuration File  Associates a repository event listener with a resource by mapping a resource configuration file to the resource (= folder or file)   31

Repository Events demo08.sql   32

End Part 1 marco.gralike@amis.nl http://www.amis.nl   33

OGh Ace Case Oracle XML Database – Part 2 Marco Gralike – AMIS   34

Disclaimer The following are “Rules of Numb” Every environment has its own unique criteria and needs… Pay attention to: “Maintainability” “Testing” “Choice” “Extendibility” “Performance” “Design”   35

Overview  Storage of XML data in the Oracle database  Physical and Logical Design  The General “Issues” with XML  Performance !  An Customer Use Case   36

Storage of XML data in the Database   37

Storage, Logical & Physical Design ?   38

The “XML Relational” Mindset CREATE TABLE MYTABLE xmltype); (data Here is your Design !   39

“Hidden” Oracle Design Features CREATE TABLE “AMISquot;.“OBJECTquot; OF quot;SYSquot;.quot;XMLTYPEquot; XMLSCHEMA quot;http://www.myserver.com/myapp/1.0/xsd/object.xsdquot; ELEMENT “ROOTquot; ID 4733 PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255 NOCOMPRESS LOGGING STORAGE(INITIAL 65536Schema: your container for XML NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645 PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1 Design Logical and Physical BUFFER_POOL DEFAULT) TABLESPACE quot;USERS“ VARRAY quot;XMLEXTRAquot;.quot;NAMESPACESquot; STORE AS BASICFILE LOB quot;NAMESPACES660_Lquot; (ENABLE STORAGE IN ROW CHUNK 8192 PCTVERSION 10 CACHE STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645 PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT)) VARRAY quot;XMLEXTRAquot;.quot;EXTRADATAquot; STORE AS BASICFILE LOB quot;EXTRADATA659_Lquot; (ENABLE STORAGE IN ROW CHUNK 8192 PCTVERSION 10 CACHE STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS Physical Design 2147483645 PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT))   40

Designing Your Building Blocks  Logical Design  XML Instance / Document  XML Schema (XDB Annotations)  XML Evolution, Validation  Physical Design  Oracle Database / Storage Design based on XML Logical Design  Binary Storage (Encrypted, Compressed, Validation, Evolution?)  XML Schema (XDB Annotations)  Indexes (BTree, IOT, XMLIndex, Function Based, Text) Is your XML data CONTENT or DOCUMENT driven ? Think: Access Paths, The Optimizer, Performance   41

Control your design: XML Schemata  Logical, Physical Data Design  Maintained via the Repository  Storage, Evolution, Security  Storage can be a Black Box (without schemata)  CLOB  Binary XML (ALLOW NONSCHEMA)  Namespaces (Recursiveness ?!)  Overhead due to Parsing  XDB Annotations can define:  Storage, Control, Name   42

When to use what kind of Storage Document centric XML  CLOB XML Document No shredding, document “keeps intact” Content centric XML  Object Relational Shredding, rebuilding document info XML Document part   43

Oracle XML DB Data Storage   44

Choices…  XMLType CLOB  Document Driven  Only in 11gR1 and upwards  XMLIndex  XMLType OR  10gRx  Index Organized Tables  11gR1 and upwards  BTree indexes XML Schema “overhead”, Data is “shredded”   Query Re-write  XMLType Binary XML  XMLType OR features kind off behavior PLUS  Post Parse Presentation   45

Physical & Logical Design   46

XDB Annotations Hybrid: CLOB within OR (Binary XML incompatible)   47

Mixing Logical and Physical Design   48

xdb:annotations – XML Schema Declaration  xdb:storeVarrayAsTable (“false” < 11g >= “true”)  If true, the varray is stored as a table (OCT)  If false, the varray is stored in a LOB  xdb:mapUnboundedStringToLob (default=“false”)  If true, then unbounded strings are mapped to CLOB instances by default. Similarly, unbounded binary data gets mapped to a BLOB value, by default.  If false, then unbounded strings are mapped to VARCHAR2(4000) values, and unbounded binary components are mapped to RAW(2000) values. Don’t forget: xmlns:xdb=quot;http://xmlns.oracle.com/xdbquot;   49

xdb:annotations – Global ComplexTypes  xdb:maintainDOM (default=“true”)  If false, then the output is not guaranteed to have the same DOM action as the input.  xdb:SQLSchema  Name of the database user who owns the type specified by SQLType  xdb:SQLType  Name of the SQL type that corresponds to this XML element declaration  Any SQL data type except NCHAR, NVARCHAR, and NCLOB   50

Some xdb:annotations – Elements (SimpleTypes)  xdb:defaultTable  xdb:defaultTableSchema  xdb:maintainDOM  xdb:maintainOrder  xdb:SQLInline  xdb:columnProps  xdb:tableProps   51

Generate or Manual Creation  Manual Creation  DBMS_METADATA.GET_DDL  EVENT setting • [alter session set events='31098 trace name context forever'; ]  Use Oracle 11g generation to apply in 10gR2 (IOT <> BTREE)  Generate via DBMS_XMLSCHEMA  XML Schema Based  XDB Annotations  JDeveloper or XMLSpy   52

Creating Storage Structures (OR) SQL> BEGIN DBMS_XMLSCHEMA.registerSchema (SCHEMAURL => 'http://www.myserver.com/root.xsd', SCHEMADOC => xdbURIType('/public/root.xsd').getClob(), LOCAL => TRUE, -- local GENTYPES => TRUE, -- generate object types GENBEAN => FALSE, -- no java beans GENTABLES => TRUE, -- generate object tables OWNER => USER); END; /   53

Result Object Relational Storage SQL> select * from tab; TNAME TABTYPE ------------------------------ ------- INFO_TABLE TABLE ROOT_TABLE TABLE SQL> select dbms_metadata.get_ddl('TABLE','ROOT_TABLE') from dual; DBMS_METADATA.GET_DDL('TABLE','ROOT_TABLE') ------------------------------------------------------ CREATE TABLE “AMISquot;.quot;ROOT_TABLEquot; OF quot;SYSquot;.quot;XMLTYPEquot; XMLSCHEMA quot;http://www.myserver.com/root.xsdquot; ELEMENT quot;ROOTquot; PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255 NOCOMPRESS LOGGING STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645 PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT) TABLESPACE quot;USERSquot; VARRAY quot;XMLEXTRAquot;.quot;NAMESPACESquot; STORE AS BASICFILE LOB quot;NAMESPACES780_Lquot; ... VARRAY quot;XMLDATAquot;.quot;INFOquot;.quot;SYS_XDBPD$quot; STORE AS BASICFILE LOB quot;SYS_XDBP...quot; DISALLOW NONSCHEMA ...   54

XMLType CLOB Storage (simplistic) SQL> create table XMLCLOB of XMLType XMLTYPE store as CLOB; Table created.   55

XMLType Object Relational Storage (simplistic) SQL> create table XMLOR of XMLType XMLTYPE store as OBJECT RELATIONAL XMLSCHEMA quot;http://www.amis.nl/xsd/root.xsdquot; ELEMENT quot;RootElementquot;; Table created.   56

XMLType Binary XML Storage (simplistic) SQL> create table XMLBIN of XMLType XMLTYPE store as BINARY XML; Table created. SQL> create table XMLBIN_XSD of XMLType XMLTYPE store as BINARY XML XMLSCHEMA quot;http://www.amis.nl/xsd/root.xsdquot; ELEMENT quot;RootElementquot;; Table created.   57

The general “issues” with XML   58

Lets presume it is Relational   59

Structured Relational Data   60

Relational Fundamentals  What goes IN, must come OUT (IN=OUT)  Regarding Data Integrity, Meaning and Context  Why do we DESIGN in the First Place?  ACID: Atomicity, Consistency, Isolation, Durability Edgar F Codd’s 12 Rules defining a “Relational Database”   Database Normalization (eg. 3NF)  Nullology  Imperfections of the (Procedural / Query) Language  Maintainability, Performance, Usability  Decades of Innovation, Methods and Standards   61

Query Optimizer – Optimal Access Paths   62

Structured Data Unstructured (XML) Data   63

Optimal Access Paths for a Tree ? Keep it intact? To much empty space! Access Optimization? Should we shred it? Rebuilding and shredding the tree structure…? There is no solution for   64

Relational contra XML Data ONE…yes ONE XML document can contain a whole database, schema of objects !   65

Impedance Mismatch  Different data models.  E.g. XPath, XQuery models an XML document as a tree, while most general purpose programming languages have no native data types for a tree.  Different programming paradigms.  XSLT is a functional language, while Java is object- oriented, and Perl is a procedural one. Source: Wikipedia SXML   66

So in General  NO PROBLEM  Data Shipping / Transport XML via XML  UI’s that “Understand”, are based on XML  PROBLEM  Accessing Data in XML structures • CPU Intensive • Memory Intensive  PGA, XML Fragments, Document Object Model  Transformation: • XML  Relational (Risk: Data / Information Loss) • Document Object Model   67

Issues you will encounter with XML  The nature of XML:  It is (most of the time) not Structured  It can not be stored in a Universal XML “Container”  It can not be accessed via a standard Access Path Method that “understands” the free format XML structure all the time…  It is all about free format  OK and Not OK  Standards: W3C, Oasis, etc  Performance problem induced by its Nature…  Nature is not structured…(as is Human Nature)   68

Nope…!  XML can handle (package) structured and semi- structured data and sometimes even unstructured data.  The open standard for data exchange  If it is XML, use XML !  Oracle XML Database = Continues Innovation “There is essentially no computer in the world, desktop, handheld, or backroom, that doesn't process XML sometimes...”   69

Performance !   70

Performance  Do it faster  for example  via indexes  Do it smarter  for example  less data  partitioning   71

XMLType Storage (Object Relational) Edith.sql   72

Recap… XMLDB Relational XMLDB Data Storage World World XMLType XMLType v? Views (Tables) Structured Binary XML Hybrid Unstructured (Object) Document Content Mixed? Content Relational na complex[n] complex[y] complex[n] Objects CLOB OR BINARY OR/CLOB XSD [y] XSD [y] XSD [y/n] XSD [y] Relational XMLIndex BTree, IOT XMLIndex BTree, IOT Tables   73

Some XML Storage Issues O.R. CLOB Binary XML Schema Based + - + XSD Flexibility - (in xml) ++ DML ++ - + XPath Performance ++ - + Throughput - + ++ Memory Management ++ - + XMLIndex - + + Function / Text + + + BTree (/ IOT) + - -   74

Indexation of Data  BTree Indexes  (11g Schema based)  Cluster Indexes  Bitmap Indexes   Index Organized Table  (10g Schema Indexes based)  Function Based Indexes   Partition Indexes  >= V 11.1.0.7.0  Domain Indexes  Oracle Text Context Index   Unstructured XMLIndex    75

XML Objects (XOBs)  DOM String Char bookstore Varchar String 2(20) book whitepaper title author author chapter title author id paragraph Number Float content content CLOB   76

Query Re-Write BTre BTre BTre ee e Index Index Index bookstore Function based Index (XPath) book whitepaper title author author chapter title author id paragraph XMLIndex content content Oracle Text Index   77

A Customer Use Case   78

ETL in Memory Processing Memor y / DOM Oracle BLOB CLOB XMLType Advanced Queue Validation Process Shred Elements Store in Oracle XML Schema Checks Via XMLDOM ETL Tables Workflow (JAVA)   79

ETL - XMLType Table (OR) Processing Oracle BLOB CLOB Advanced Queue Validation XMLType Table Store in Oracle Against Checks (O.R) ETL Tables Workflow XML Schema Rewrite on Disk / XOB (Relational)   80

Results XML Relational Approach  100.000 “zaken” shredded, validated in 5 minutes (instead of 1000 “zaken” in 3 minutes)  Avoiding  ORA-31186: Document contains too many nodes  Scalable  Efficient with Memory and CPU  Checked on 9.2.0.5 database version   81

References  Oracle 11g XMLDB Developers Guide  The XMLDB Forum  http://forums.oracle.com/forums/forum.jspa?forumID=34  OTN XML DB FAQ Thread  http://forums.oracle.com/forums/thread.jspa?threadID=410714  Technology Blogs  http://blog.gralike.com  http://technology.amis.nl/blog  Thanks to:  Mark Drake, Tom Kyte  Keith Matthew, Anton Scheffer, Laurent Schneider   82

marco.gralike@amis.nl http://www.amis.nl   83

http://www.liberidu.com	83
http://www.slideshare.net	5
http://webcache.googleusercontent.com	1

Ogh Ace Case, Part 1 and 2, Oracle Xml Database, Marco Gralike

by Marco Gralike

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 89

Statistics

Ogh Ace Case, Part 1 and 2, Oracle Xml Database, Marco Gralike Presentation Transcript