Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Testing Delphix: easy data virtualization


Published on

Testing Delphix: easy data virtualization

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Testing Delphix: easy data virtualization

  1. 1. 19Tips&techniques19 SOUG Newsletter 5/2014 – Sonderausgabe Testing Delphix: easy data virtualization Franck Pachot, dbi services File system technologies such as snapshots, clo- ning, copy-on-write are arising to handle the ever increasing size of databases and the multiplying number of required copies (backups, preprod, test environment). But there are new technologies to learn beyond the basic storage system options. EMC, NetAPP, Oracle ZFS Storage Appliance are hardware appliance and requires specialized skills. But there is a new breed of products that have arisen to tackle both the technical obstacle as well as the specia- lized skills obstacle. The new product is a software appliance: it runs as a VMware virtual machine and exposes the virtual storage through NFS. The software is intimately integrated with the database providing power and ease of use. At Oracle Open World I tested it and discussed with the oracle experts Kyle Hailey (Ace and Oaktable) and Tim Gorman (Ace Director and Oaktable) that are now working at Delphix. Why data virtualization? What is the size of your database? Let’s say you have a small to medium size – 200GB. And you generate 20GB of archivelog every day. Let’s go into production needs first. You requires to keep 30 days of backup. With daily full back- ups that’s more than 5TB of backupsets. If you do weekly full backups and daily incrementals, you may reduce it to 2TB. Ok 2TB for backups. Now you need environments for pre- production (testing upgrades), UAT (testing releases), qualifi- cation (testing performance). Your UAT users dream about having their own database for each testers, so they don’t overlap on data. And now about developers. If you want them to release efficient application you should give them an environment with representative volume of data. And once again each de- veloper team has its own set of data set. How many database do you need to provide? That requirements exists for years. But today the delivery delays have decreased a lot. We must be agile. Developers are committing to continuous integration. They have a bunch of test cases to run on a fixed set of data. And they need to rewind the database each time. I forget one environment: bugfix. What if you find an ap- plication bug on production? You have a patch but want to test it. If you have Active Data Guard (i.e. Enterprise Edition plus ADG option), you can use one of the physical standby to do a snapshot standby. What if you were able to just click on a GUI and provision a new copy instantaneously? Well, it’s a bit more than a click. It is a click and slide: Yes: you can choose the point in time snapshot that you want to copy.
  2. 2. 20 Tipstechniques SOUG Newsletter 5/2014 – Sonderausgabe Source databases So you have your existing databases and they can act as source for cloning. You don’t have to do anything about them except installing the Delphix Toolkit on their server. Here is the screen to define the source server. Toolkit de- ployment is automatic: You just set the host connection and where to install the Toolkit. Then you add the databases. How does it works? Delphix will use RMAN to get the source database. The first time you add the source database, it will do a full backup through RMAN. And only the first time. Then only in- cremental changes are collected (using incremental backups and/or archived logs) to maintain the copy on the Delphix server. The Delphix server is a VMware virtual machine. You al- locate storage as you want and Delphix maps its own file sys- tem onto the storage. Delphix uses a copy-on-write filesys- tem in order to be able to rewind to any state between the first initialization (or the defined window) and the current time. In addition to that the storage is compressed by Delphix. Then the copy is 2 or 3 times smaller than the source database and it is able to flashback in the past. Clone databases Then you have the base: a copy of your production which you can view as of any timestamp. And with a few clicks you can provision a clone immed­ iately: data blocks are not copied until they are changed. I have the ’#cloneattack’ environment on my laptop. Del- phix is running and I can connect with any web browser to get the full interface, The server running in VMware Workstation: And the console, connected from my browser. Here I show the source database properties. Delphix has a nice GUI: you click on the upper right arrow to flip between sum- mary and detail:
  3. 3. 21Tipstechniques21 SOUG Newsletter 5/2014 – Sonderausgabe It will create automatically an instance for it (I need the Delphix toolkit installed on the target server as well) and I can change the default init.ora parameters, file location, listener etc. Ok, after a few screens Delphix is doing everything auto- matically. I can control the running jobs: DB_PROVISION cre- ates my instance and database and DB_SYNC manages the data virtualization to show the data from the source point-in- time I have chosen. Ok, now I want to get a copy of it – as of 08:30 this morn- ing: I have then to define where to put it (storage is virtualized but I need a server to run the instance) and a name for it:
  4. 4. 22 Tipstechniques SOUG Newsletter 5/2014 – Sonderausgabe How long does it take? That’s just a few minutes and the da- tabase is opened in read write, ready to accept connections with the connection string defined by the listener and data- base configuration. You can see it on the target server: [oracle@linuxtarget ~]$ ps -edf | grep pmon oracle 57363 1 0 02:19 ? 00:00:00 ora_pmon_VEmp519 oracle 60795 1 0 03:29 ? 00:00:00 ora_pmon_MyCopy oracle 61180 61147 0 03:33 pts/0 00:00:00 grep pmon and the listener is listening on it: [oracle@linuxtarget ~]$ lsnrctl status LSNRCTL for Linux: Version - Production on 30-SEP-2014 03:34:01 Copyright (c) 1991, 2009, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=linuxtarget.delphix.local)(PORT=1521))) STATUS of the LISTENER ------------------------ Alias LISTENER Version TNSLSNR for Linux: Version - Production Start Date 29-SEP-2014 21:49:14 Uptime 0 days 5 hr. 44 min. 47 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora Listener Log File /u01/app/oracle/diag/tnslsnr/linuxtarget/listener/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=linuxtarget)(PORT=1521))) Services Summary... Service “MyCopy” has 1 instance(s). Instance “MyCopy, status READY, has 1 handler(s) for this service... Service “VEmp_519” has 1 instance(s). Instance “VEmp519, status READY, has 1 handler(s) for this service... The command completed successfully I like to see the structure of the database with RMAN ’re- port schema’: RMAN report schema; using target database control file instead of recovery catalog Report of database schema for database with db_unique_name MYCOPY List of Permanent Datafiles =========================== File Size(MB) Tablespace RB segs Datafile Name ---- -------- --------- ------- ------------------------ 1 680 SYSTEM *** /mnt/provision/MyCopy/datafile/u02/app/orcl/data/u01/app/oracle/oradata/orcl/system01.dbf 2 530 SYSAUX *** /mnt/provision/MyCopy/datafile/u02/app/orcl/data/u01/app/oracle/oradata/orcl/sysaux01.dbf 3 60 UNDOTBS1 *** /mnt/provision/MyCopy/datafile/u02/app/orcl/data/u01/app/oracle/oradata/orcl/undotbs01.dbf 4 38 USERS *** /mnt/provision/MyCopy/datafile/u02/app/orcl/data/u01/app/oracle/oradata/orcl/users01.dbf List of Temporary Files ======================= File Size(MB) Tablespace Maxsize(MB) Tempfile Name ---- -------- ------------------- ----------- -------------------- 1 20 TEMP 32767 /mnt/provision/MyCopy/temp/temp1.dbf
  5. 5. 23Tipstechniques23 SOUG Newsletter 5/2014 – Sonderausgabe We see that the datafiles are on the NFS mountpoint, with the name of the copy and the name of the source as well. And finally I like to see all what happened on the back- ground: [oracle@linuxtarget trace]$ cat alert_MyCopy.log Tue Sep 30 03:26:01 2014 Starting ORACLE instance (restrict) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 … System parameters with non-default values: processes = 150 event = 10052 trace name context forever filesystemio_options = setall _omf = DISABLED control_files = /mnt/provision/MyCopy/datafile/u02/app/orcl/data/u01/app/oracle/oradata/orcl/control01.ctl _controlfile_enqueue_timeout = 4500 db_block_size = 8192 compatible = log_archive_dest_1 = location=/mnt/provision/MyCopy/source-archive MANDATORY log_archive_format = arch_%t_%s_%r.log db_create_file_dest = /mnt/provision/MyCopy/datafile remote_login_passwordfile = EXCLUSIVE audit_sys_operations = FALSE job_queue_processes = 0 _system_trig_enabled = FALSE audit_trail = NONE db_name = orcl db_unique_name = MyCopy open_cursors = 300 _diag_hm_rc_enabled = FALSE … ALTER DATABASE MOUNT … alter database rename file '/u02/app/orcl/data/u01/app/oracle/oradata/orcl/system01.dbf' ,'/u02/app/orcl/data/u01/app/oracle/oradata/orcl/sysaux01.dbf' ,'/u02/app/orcl/data/u01/app/oracle/oradata/orcl/undotbs01.dbf' ,'/u02/app/orcl/data/u01/app/oracle/oradata/orcl/users01.dbf' to '/mnt/provision/MyCopy/datafile/u02/app/orcl/data/u01/app/oracle/oradata/orcl/system01.dbf' ,'/mnt/provision/MyCopy/datafile/u02/app/orcl/data/u01/app/oracle/oradata/orcl/sysaux01.dbf' ,'/mnt/provision/MyCopy/datafile/u02/app/orcl/data/u01/app/oracle/oradata/orcl/undotbs01.dbf' ,'/mnt/provision/MyCopy/datafile/u02/app/orcl/data/u01/app/oracle/oradata/orcl/users01.dbf' What I see here is that this has been designed by people that really knows how oracle works in depth. It’s not just a simple clone script that works only in the common case. It’s real DBA stuff. Then what do you have? You have a database that takes minimal storage and is seen as an exact point-in-time copy of the source database. You can use it, rewind it to another point in time, you can refresh it from source, you can clone it to another copy etc.
  6. 6. 24 Tipstechniques SOUG Newsletter 5/2014 – Sonderausgabe Use Cases So what can you do with virtual databases? ■ You can give a database to each tester, each developer and they have a full volume database without taking too much storage ■ You can give a database for continuous integration that can be rewind before each test (Delphix has an API to control without the GUI) ■ You set up QA environments with real data. Choosing a point-in-time let open a database in an ’end-of-month’ situation for example if you want to test a monthly job. ■ You can have synced databases for protection. If the Delphix VM is set with high availability, then you can get an exact copy of a production database that is not available anymore. ■ You can quickly provision a database for bugfix, showing an exact situation in the past where you had the bug happening. ■ You can use it for backups: you reduce storage and reduce RPO and RTO. Do you prefer RMAN backups as it detects block corruption? Why not automating a daily clone provisioning that you open and run a rman validate database on it? ■ You can avoid lot of the storage you use for backup retention. You can get yearly snapshots to keep 10 years, monthly snapshots to keep 2 years and be able to get a copy from any point-in time from the previous two months. ■ Your BI ODS database can be refreshed with Delphix in a few minutes. Remember: Delphix maintains its copy of the production database (synchronizing with incre- mentals and archivelogs) so you don’t impact production performance even with high activity on the clone. Those are the few Use Cases that are the most common I see in many companies. Conclusion I’ve heard of Delphix for months, mainly because people at Delphix are well known in the Oracle community. And Jon- athan Lewis has presented that solution on several occa- sions. His performance tuning activity often show the need to have an exact copy of production at a specific state in order to reproduce the issue. But I like to learn about a product by testing it instead of reading docs or looking at presentations. And the #cloneat- tack organized at the Oracle Open World was a good occa- sion for that. By playing with it just a few hours, I can see how it addresses some common questions frequently raised by my customers. Both in operation and development teams. This article is only about what I’ve tested. There is a lot more. Delphix is not only for Oracle. It does the same for Microsoft SQL Server. PostgreSQL is also supported. ■ Contact dbi services Franck Pachot E-Mail: Besuchen Sie jetzt die Oracle-Originalkurse bei Digicomp: Neue Kurse zu Oracle Database 12c! 0844 844 822,, A N Z E I G E