Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Jurijs Velikanovs Direct NFS - Why and How?

2,370 views

Published on

source http://www.slideshare.net/yvelikanov/sharing-experience-implementing-direct-nfs

  • Be the first to comment

Jurijs Velikanovs Direct NFS - Why and How?

  1. 1. Sharing experience implementing Direct NFS Yury & Oracle Community
  2. 2. Contributors list From Twitter John Piwowar (@jpiwowar) Leighton L. Nelson (@leight0nn) Kevin Closson (@kevinclosson) Guenadi Jilevski (@gjilevski) Simon Haslam (@simon_haslam) neto from Brazil (@netofrombrazil) Frits Hoogland (@fritshoogland) Martin Bach (@MartinDBA) Gwen (Chen) Shapira (@gwenshap) Avi Miller (@Djelibeybi) Marcin Przepiorowski (@pioro) Arup Nanda (@arupnanda) Martin Berger (@martinberx) https://dl.dropbox.com/u/28277005/DNFS%20 Martin Bach @MartinDBA Look in slides notes for TW discussions © 2012 – Pythian
  3. 3. Why Pythian ● Recognized Leader: ● Global industry leader in data infrastructure managed services and consulting with expertise in Oracle, Oracle Applications, Microsot SQL Server, MySQL, big data and systems administration ● Work with over 200 multinational companies such as Forbes.com, Fox Sports, Nordion and Western Union to help manage their complex IT deployments ● Expertise: ● One of the world’s largest concentrations of dedicated, full-time DBA expertise. Employ 8 Oracle ACEs/ACE Directors ● Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC ● Global Reach & Scalability: ● 24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response © 2012 – Pythian
  4. 4. Why Yury Google: Yury Oracle [phone|email] Twitter, LinkedIn, Blog, Slideshare,YouTube Oracle ACE (RAC SIG international chair, Sydney Oracle Meetups) Oracle Certified Master Oracle DBA with 15+ years experience I like my job, I like what I do, I like to share knowledge, I like to help others to share knowledge, I like to learn, I like my job. © 2012 – Pythian
  5. 5. Why Direct NFS Network IO SPEED Easy to use © 2012 – Pythian +RAC
  6. 6. d NFS k NFS Load Profile ~~~~~~~~~~~~ DB Time(s): DB CPU(s): Logical reads: Per Second --------------19.6 0.8 12,219.1 12,141.3 Physical reads: Load Profile ~~~~~~~~~~~~ DB Time(s): DB CPU(s): Logical reads: Per Second --------------19.6 5.2 38,277.5 38,156.1 Physical reads: 1.7ms 100% 0.5ms 300% Oracle VM, PIO SLOB test, ram disk mounted via 127.0.0.1 backloop device, other bottlenecks but kernel have been excluded, “db file sequential read” aka random reads Direct storage 0.36 ms © 2012 – Pythian
  7. 7. It isn’t about data files only ! File Type Supported Control file YES Data file YES Redo log file YES Archive/Flashback log file YES Backup files YES Temp file YES Datapump dump file YES blog “Direct NFS speeds up Data Pump” OCR files NO spfile YES passwd file YES ASM files YES blog “Reasons for using ASM on NFS” Voting files NO Audit files NO Database trace files NO External tables NO © 2012 – Pythian Direct NFS: FAQ [ID 954425.1] RDBMS file type support matrix for Direct NFS client
  8. 8. Direct NFS Concepts
  9. 9. IO Client Oracle process IO Client Oracle process IO Client Oracle process IO Client Oracle process IO Client Oracle process NFS client NFS client NFS client kernel Kernel mode NFS client NFS service File A File B NFS service File C © 2012 – Pythian File A File B File C kernel User IO Client Oracle process user mode d NFS k NFS Simplified
  10. 10. dNFS Setup simple version mount NFS volumes cd $ORACLE_HOME/rdbms/lib make -f ins_rdbms.mk dnfs_on ls -l $ORACLE_HOME/lib/lib*od*11.so -rw-r--r--. 1 oracle oinstall 60431 Sep 17 2011 $OH/lib/libnfsodm11.so -rw-r--r--. 1 oracle oinstall 60431 Oct 20 19:45 $OH/lib/libodm11.so -rw-r--r--. 1 oracle oinstall 12259 Sep 17 2011 $OH/lib/libodmd11.so << dnfs # Oracle used << no dnfs Oracle processes use /etc/mtab to access NFS settings http://www.pythian.com/news/34425/oracle-direct-nfs-how-to-start/ odm – stands for Oracle Disk Manager © 2012 – Pythian
  11. 11. How can the dNFS usage be verified? Direct NFS: FAQ [ID 954425.1] 1. Just after the initialization parameters are listed in the alert log, you will see the following entry Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 2.0 2. Along with the message from the alert log, this query on v$dnfs_servers ensures that dNFS is truly being used (returns !=0 value): select count(*) from v$dnfs_servers ( sometimes it isn’t true ;) © 2012 – Pythian
  12. 12. How can the dNFS usage be verified? In my experience, the easiest and most reliable way is 1. lsof –p < dbw pid > # DNFS OFF lsof -p 725 | grep data01.dbf oracle 725 oracle 262u REG ... /nfsimp/data01.dbf (192.168.51.21:/u01) # DNFS ON lsof -p 6540 | grep 192.168.51.21 oracle 6540 oracle 32u IPv4 ... TCP dbhost:26171->nfsserver:nfs (ESTABLISHED) 2. alert.log Direct NFS: channel id [0] path [IPnfs] to filer [KUKARACHA] via local [IPdb] is UP Direct NFS: channel id [1] path [IPnfs] to filer [KUKARACHA] via local [IPdb] is UP © 2012 – Pythian
  13. 13. Direct NFS advanced implementation
  14. 14. dNFS setup – is documented Step by Step - Configure Direct NFS Client on Linux [ID 762374.1] Step by Step - Configure Direct NFS Client on Windows [ID 1468114.1] ======================================================= Set filesystemio_options at least to directIO. ALL database files. ● PART A -- SETTING UP THE NFS SERVER ON LINUX ... iv) Make sure the NFS server will get started during boot of this server. ... MOUNTING NFS ON THE CLIENT NODE / CLUSTER NODES ON LINUX ... stgasm:/oraclenfs /oradata1 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600 0 0 # Please contact your NAS vendor for NFS mount option recommendations. ... ● PART B -- Configure Direct NFS Client (DNFS) i) Configure oranfstab file ... Direct NFS Client can use a new configuration file or the mount tab file (/etc/mtab on Linux) to determine the mount point settings for NFS storage devices. ... ● PART C -- DNFS Workshop Oracle® Grid Infrastructure Installation Guide11g Release 2 (11.2) for Linux 3 Configuring Storage for Grid Infrastructure for a Cluster and Oracle RAC © 2012 – Pythian
  15. 15. dNFS oranfstab= Performance, Scalability, High Availability, Cost Savings 1Gbit 1Gbit Switch 1Gbit 1Gbit NFS service NIC 1Gbit 4 Gbit NIC Switch NIC NIC NIC © 2012 – Pythian 1Gbit File B NIC NIC 1Gbit 1Gbit File A NIC NIC 1Gbit 1Gbit NIC` NIC RAC Node 2 IO Client Oracle process NIC IO Client Oracle process RAC Node 1 Round Robin 4 TCP/IP connections File C
  16. 16. dNFS oranfstab (things to remember) ● You can live without it (simple implementations) ● DB Hangs When DNFS is Enabled on UEK kernel [ID 1460787.1] ● You may get confused reading though documentations ● Enabling Direct NFS Client Oracle Disk Manager Control of NFS ● Server, Local, Path, Export, Mount, Mnt_timeout, Dontroute ● Server – any name you like (alias for a channel) Local, Path - You can specify other IPs than in mtab Export, Mount – local and remote mount points Mnt_timeout – sessions drops a connection after the timeout Dontroute – don’t use OS routeing table to send TCP/IP packages ● ● ● ● Oracle Direct NFS configuration file explained http://www.pythian.com/news/37259/oracle-direct-nfs-configuration-file-explained/ © 2012 – Pythian
  17. 17. dNFS oranfstab (things to remember) ● Direct NFS searches for mount entries in the following order 1. 2. 3. $ORACLE_HOME/dbs/oranfstab # an instance specific /etc/oranfstab /etc/mtab ● You must restart an instance after adjusting oranfstab file © 2012 – Pythian
  18. 18. Troubleshooting & War stories
  19. 19. One DNFS disadvantage ●It is relatively difficult to troubleshoot ●You will not find many DNFS messages in alert.log or trace files ●Sometimes Oracle SILINTLY switches from dNFS to kNFS ●This sections gives you useful troubleshooting hints © 2012 – Pythian
  20. 20. Don’t mix old and new methods ●Old method cd $ORACLE_HOME/lib mv libodm11.so libodm11.so_bak ln -s libnfsodm11.so libodm11.so ●New/current method make -f ins_rdbms.mk dnfs_on ●If you enable dNFS using one method disable it the same way © 2012 – Pythian
  21. 21. Magic words to remember Oracle® Grid Infrastructure Installation Guide 11g Release 2 (11.2) for Linux If Oracle Database cannot open an NFS server using Direct NFS, then Oracle Database SILENTLY uses the platform operating system kernel NFS client. 1. lsof –p < dbw pid > # dNFS OFF lsof -p 725 | grep data01.dbf oracle 725 oracle 262u REG ... /nfsimp/data01.dbf (192.168.51.21:/u01) # dNFS ON lsof -p 6540 | grep 192.168.51.21 oracle 6540 oracle 32u IPv4 ... TCP dbhost:26171->nfsserver:nfs (ESTABLISHED) © 2012 – Pythian
  22. 22. NFS server reserved ports restriction Oracle® Grid Infrastructure Installation Guide 11g Release 2 (11.2) for Linux Some NFS file servers require NFS clients to connect using reserved ports. If your filer is running with reserved port checking, then you must disable it for Direct NFS to operate. To disable reserved port checking, consult your NFS file server documentation. In a simple NFS implementation that means: cat /etc/exports < dir > IP(...,insecure,...) © 2012 – Pythian
  23. 23. dNFS: wtmax = /proc/fs/nfsd/max_block_size Oracle® Grid Infrastructure Installation Guide 11g Release 2 (11.2) for Linux Caution: Direct NFS will not serve an NFS server with write size values (wtmax) less than 32768. ... ALTER DATABASE OPEN Direct NFS: attempting to mount /u02 on filer KUKARACHA2 defined in oranfstab Direct NFS: channel config is: channel id [0] local [192.168.51.30] path [192.168.51.21] Direct NFS: mount complete dir /u02 on KUKARACHA2 mntport 963 nfsport 2049 Direct NFS: Invalid filer wtmax 525232 on filer KUKARACHA2 Direct NFS: Filer wtmax 525232 must be an even multiple of 32768 Thread 1 opened at log sequence 26 ... © 2012 – Pythian
  24. 24. dNFS: wtmax = /proc/fs/nfsd/max_block_size Pythian’s dNFS related blog posts root@nfsfiler01 sysconfig# cat /proc/fs/nfsd/max_block_size 524288 root@nfsfiler01 sysconfig# echo 1048576 > /proc/fs/nfsd/max_block_size root@nfsfiler01 sysconfig# cat /proc/fs/nfsd/max_block_size 1048576 ... ALTER DATABASE OPEN Direct NFS: attempting to mount /u02 on filer KUKARACHA2 defined in oranfstab Direct NFS: channel config is: channel id [0] local [192.168.51.30] path [192.168.51.21] Direct NFS: mount complete dir /u02 on KUKARACHA2 mntport 883 nfsport 2049 Direct NFS: channel id [0] path [192.168.51.21] to filer [KUKARACHA2] via local [192.168.51.30] is UP Direct NFS: channel id [1] path [192.168.51.21] to filer [KUKARACHA2] via local [192.168.51.30] is UP Beginning crash recovery of 1 threads ... © 2012 – Pythian
  25. 25. init.ora events to set ●If you are confused and need more information then set the following events and access NFS files event "19392 trace name context forever, level 8" # kgnfs event "19394 trace name context forever, level 8" # skgnfs event "19396 trace name context forever, level 2" # kgodm event="19398 trace name context forever, level 128" alter system set event="19392 trace name context forever, level 8","19394 trace name context forever, level 8","19396 trace name context forever, level 2","19398 trace name context forever, level 128" scope=spfile; ● Read trace file, search MOS/Google, ask others ( Yury ;) © 2012 – Pythian
  26. 26. References Direct NFS: FAQ [ID 954425.1] Example About How To Setup DNFS On Oracle Release 11.2 [ID 1452614.1] How to Setup Direct NFS client multipaths in same subnet [ID 822481.1] DB Hangs When DNFS is Enabled on UEK kernel [ID 1460787.1] DB hang and ORA-600 [2116] errors when enabling DNFS [ID 1484422.1] DNFS CANNOT MOUNT FILESYSTEM AND DATABASE FAILED WITH ORA-600 [2116] AFTER A REBOOT OF SERVER DUE TO POWER FAILURE [ID 1480788.1] DATABASE STARTUP HANGS AT MOUNTING CONTROLFILE WHEN DNFS IS ENABLED. [ID 971406.1] Database Startup Failed with "Direct NFS: please check that oradism is setuid“ [1430654.1] ORA-600 [2116] Using The Veritas Odm Lib Oracle Fails To Mount [ID 418603.1] OERI [2116] [900] during database mount with > 13 instances [ID 9790947.8] TESTCASE Step by Step - Configure Direct NFS Client (DNFS) on Windows [ID 1468114.1] Database Alert Log entries: Direct NFS: Failed to set socket buffer size.wtmax=[1048576] rtmax=[1048576], errno=-1 [ID 1352886.1] @kevinclosson Oracle 11g, Direct NFS Client, An Oracle White Paper + http://bit.ly/QU3w82 @leight0nn http://blogs.griddba.com/2012/02/direct-nfs-speeds-up-data-pump.html @rene_kundersma https://blogs.oracle.com/XPSONHA/entry/using_dnfs_for_test_purposes @yvelikanov http://www.pythian.com/news/tag/dnfs/ © 2012 – Pythian
  27. 27. Your best friends OR things to keep in mind Network IO SPEED efficiency © 2012 – Pythian
  28. 28. Thank you and Q&A To contact us… sales@pythian.com 1-877-PYTHIAN To follow us… http://www.pythian.com/news/ http://www.facebook.com/pages/The-Pythian-Group/163902527671 @pythian @pythianjobs http://www.linkedin.com/company/pythian © 2012 – Pythian

×