Sharing experience implementing Direct NFS
•Download as PPTX, PDF•
16 likes•13,927 views
Direct NFS (dNFS) allows Oracle databases to access NFS-mounted storage directly without going through the kernel. This improves performance by reducing context switches between user and kernel space. The document discusses setting up dNFS including mounting NFS shares, configuring the oranfstab file, and verifying dNFS usage. It also provides an overview of dNFS concepts such as using multiple TCP connections in a round-robin fashion for high throughput and availability.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Sharing experience implementing Direct NFS
Similar to Sharing experience implementing Direct NFS (20)
More from Yury Velikanov
More from Yury Velikanov (8)
Recently uploaded
Recently uploaded (20)
Sharing experience implementing Direct NFS
- 1. Sharing experience implementing Direct NFS Yury & Oracle Community
- 2. Contributors list From Twitter John Piwowar (@jpiwowar) Leighton L. Nelson (@leight0nn) Kevin Closson (@kevinclosson) Guenadi Jilevski (@gjilevski) Simon Haslam (@simon_haslam) neto from Brazil (@netofrombrazil) Frits Hoogland (@fritshoogland) Martin Bach (@MartinDBA) Gwen (Chen) Shapira (@gwenshap) Avi Miller (@Djelibeybi) Marcin Przepiorowski (@pioro) Arup Nanda (@arupnanda) Martin Berger (@martinberx) https://dl.dropbox.com/u/28277005/DN Martin Bach @MartinDBA Look in slides notes for TW discussions FS%20exp%2004.pptx © 2012 – Pythian
- 3. Why Pythian ● Recognized Leader: ● Global industry leader in data infrastructure managed services and consulting with expertise in Oracle, Oracle Applications, Microsot SQL Server, MySQL, big data and systems administration ● Work with over 200 multinational companies such as Forbes.com, Fox Sports, Nordion and Western Union to help manage their complex IT deployments ● Expertise: ● One of the world’s largest concentrations of dedicated, full-time DBA expertise. Employ 8 Oracle ACEs/ACE Directors ● Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC ● Global Reach & Scalability: ● 24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response © 2012 – Pythian
- 4. Why Yury Google: Yury Oracle [phone|email] Twitter, LinkedIn, Blog, Slideshare,YouTube Oracle ACE (RAC SIG international chair, Sydney Oracle Meetups) Oracle Certified Master Oracle DBA with 15+ years experience I like my job, I like what I do, I like to share knowledge, I like to help others to share knowledge, I like to learn, I like my job. © 2012 – Pythian
- 5. Why Direct NFS Network IO SPEED Easy to use +RAC © 2012 – Pythian
- 6. k NFS d NFS Load Profile Per Second Load Profile Per Second ~~~~~~~~~~~~ --------------- ~~~~~~~~~~~~ --------------- DB Time(s): 19.6 DB Time(s): 19.6 DB CPU(s): 0.8 DB CPU(s): 5.2 Logical reads: 12,219.1 Logical reads: 38,277.5 Physical reads: 12,141.3 Physical reads: 38,156.1 1.7ms 0.5ms 100% 300% Oracle VM, PIO SLOB test, ram disk mounted via 127.0.0.1 backloop device, other bottlenecks but kernel have been excluded, “db file sequential read” aka random reads Direct storage 0.36 ms © 2012 – Pythian
- 7. It isn’t about data files only ! File Type Supported Control file YES Direct NFS: FAQ [ID 954425.1] Data file YES Redo log file YES RDBMS file type support matrix for Direct NFS client Archive/Flashback log file YES Backup files YES Temp file YES Datapump dump file YES blog “Direct NFS speeds up Data Pump” OCR files NO spfile YES passwd file YES ASM files YES blog “Reasons for using ASM on NFS” Voting files NO Audit files NO Database trace files NO External tables NO © 2012 – Pythian
- 9. Simplified User k NFS d NFS IO Client IO Client IO Client IO Client IO Client IO Client Oracle Oracle Oracle Oracle Oracle Oracle process process process process process process user mode NFS client NFS client NFS client kernel NFS client Kernel mode NFS service NFS service kernel File A File B File C File A File B File C © 2012 – Pythian
- 10. dNFS Setup simple version mount NFS volumes cd $ORACLE_HOME/rdbms/lib make -f ins_rdbms.mk dnfs_on ls -l $ORACLE_HOME/lib/lib*od*11.so -rw-r--r--. 1 oracle oinstall 60431 Sep 17 2011 $OH/lib/libnfsodm11.so << dnfs -rw-r--r--. 1 oracle oinstall 60431 Oct 20 19:45 $OH/lib/libodm11.so # Oracle used -rw-r--r--. 1 oracle oinstall 12259 Sep 17 2011 $OH/lib/libodmd11.so << no dnfs Oracle processes use /etc/mtab to access NFS settings http://www.pythian.com/news/34425/oracle-direct-nfs-how-to-start/ odm – stands for Oracle Disk Manager © 2012 – Pythian
- 11. How can the dNFS usage be verified? Direct NFS: FAQ [ID 954425.1] 1. Just after the initialization parameters are listed in the alert log, you will see the following entry Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 2.0 2. Along with the message from the alert log, this query on v$dnfs_servers ensures that dNFS is truly being used (returns !=0 value): select count(*) from v$dnfs_servers ( sometimes it isn’t true ;) © 2012 – Pythian
- 12. How can the dNFS usage be verified? In my experience, the easiest and most reliable way is 1. lsof –p < dbw pid > # DNFS OFF lsof -p 725 | grep data01.dbf oracle 725 oracle 262u REG ... /nfsimp/data01.dbf (192.168.51.21:/u01) # DNFS ON lsof -p 6540 | grep 192.168.51.21 oracle 6540 oracle 32u IPv4 ... TCP dbhost:26171->nfsserver:nfs (ESTABLISHED) 2. alert.log Direct NFS: channel id [0] path [IPnfs] to filer [KUKARACHA] via local [IPdb] is UP Direct NFS: channel id [1] path [IPnfs] to filer [KUKARACHA] via local [IPdb] is UP © 2012 – Pythian
- 13. Direct NFS advanced implementation
- 14. dNFS setup – is documented Step by Step - Configure Direct NFS Client on Linux [ID 762374.1] Step by Step - Configure Direct NFS Client on Windows [ID 1468114.1] ======================================================= Set filesystemio_options at least to directIO. ALL database files. ● PART A -- SETTING UP THE NFS SERVER ON LINUX ... iv) Make sure the NFS server will get started during boot of this server. ... MOUNTING NFS ON THE CLIENT NODE / CLUSTER NODES ON LINUX ... stgasm:/oraclenfs /oradata1 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600 0 0 # Please contact your NAS vendor for NFS mount option recommendations. ... ● PART B -- Configure Direct NFS Client (DNFS) i) Configure oranfstab file ... Direct NFS Client can use a new configuration file or the mount tab file (/etc/mtab on Linux) to determine the mount point settings for NFS storage devices. ... ● PART C -- DNFS Workshop Oracle® Grid Infrastructure Installation Guide11g Release 2 (11.2) for Linux 3 Configuring Storage for Grid Infrastructure for a Cluster and Oracle RAC © 2012 – Pythian
- 15. dNFS oranfstab = Performance, Scalability, High Availability, Cost Savings Round Robin 1Gbit NIC 1Gbit NIC` 4 TCP/IP connections RAC Node 1 1Gbit File A IO Client NIC Oracle 1Gbit Switch 1Gbit NIC NIC process 1Gbit 1Gbit NFS service NIC NIC 4 Gbit File B RAC Node 2 1Gbit NIC IO Client NIC Oracle Switch NIC process 1Gbit 1Gbit File C NIC NIC © 2012 – Pythian
- 16. dNFS oranfstab (things to remember) ● You can live without it (simple implementations) ● DB Hangs When DNFS is Enabled on UEK kernel [ID 1460787.1] ● You may get confused reading though documentations ● Enabling Direct NFS Client Oracle Disk Manager Control of NFS ● Server, Local, Path, Export, Mount, Mnt_timeout, Dontroute ● Server – any name you like (alias for a channel) ● Local, Path - You can specify other IPs than in mtab ● Export, Mount – local and remote mount points ● Mnt_timeout – sessions drops a connection after the timeout ● Dontroute – don’t use OS routeing table to send TCP/IP packages Oracle Direct NFS configuration file explained http://www.pythian.com/news/37259/oracle-direct-nfs-configuration-file-explained/ © 2012 – Pythian
- 17. dNFS oranfstab (things to remember) ● Direct NFS searches for mount entries in the following order 1. $ORACLE_HOME/dbs/oranfstab # an instance specific 2. /etc/oranfstab 3. /etc/mtab ● You must restart an instance after adjusting oranfstab file © 2012 – Pythian
- 18. Troubleshooting & War stories
- 19. One DNFS disadvantage ● It is relatively difficult to troubleshoot ● You will not find many DNFS messages in alert.log or trace files ● Sometimes Oracle SILINTLY switches from dNFS to kNFS ● This sections gives you useful troubleshooting hints © 2012 – Pythian
- 20. Don’t mix old and new methods ● Old method cd $ORACLE_HOME/lib mv libodm11.so libodm11.so_bak ln -s libnfsodm11.so libodm11.so ● New/current method make -f ins_rdbms.mk dnfs_on ● If you enable dNFS using one method disable it the same way © 2012 – Pythian
- 21. Magic words to remember Oracle® Grid Infrastructure Installation Guide 11g Release 2 (11.2) for Linux If Oracle Database cannot open an NFS server using Direct NFS, then Oracle Database SILENTLY uses the platform operating system kernel NFS client. 1. lsof –p < dbw pid > # dNFS OFF lsof -p 725 | grep data01.dbf oracle 725 oracle 262u REG ... /nfsimp/data01.dbf (192.168.51.21:/u01) # dNFS ON lsof -p 6540 | grep 192.168.51.21 oracle 6540 oracle 32u IPv4 ... TCP dbhost:26171->nfsserver:nfs (ESTABLISHED) © 2012 – Pythian
- 22. NFS server reserved ports restriction Oracle® Grid Infrastructure Installation Guide 11g Release 2 (11.2) for Linux Some NFS file servers require NFS clients to connect using reserved ports. If your filer is running with reserved port checking, then you must disable it for Direct NFS to operate. To disable reserved port checking, consult your NFS file server documentation. In a simple NFS implementation that means: cat /etc/exports < dir > IP(...,insecure,...) © 2012 – Pythian
- 23. dNFS: wtmax = /proc/fs/nfsd/max_block_size Oracle® Grid Infrastructure Installation Guide 11g Release 2 (11.2) for Linux Caution: Direct NFS will not serve an NFS server with write size values (wtmax) less than 32768. ... ALTER DATABASE OPEN Direct NFS: attempting to mount /u02 on filer KUKARACHA2 defined in oranfstab Direct NFS: channel config is: channel id [0] local [192.168.51.30] path [192.168.51.21] Direct NFS: mount complete dir /u02 on KUKARACHA2 mntport 963 nfsport 2049 Direct NFS: Invalid filer wtmax 525232 on filer KUKARACHA2 Direct NFS: Filer wtmax 525232 must be an even multiple of 32768 Thread 1 opened at log sequence 26 ... © 2012 – Pythian
- 24. dNFS: wtmax = /proc/fs/nfsd/max_block_size Pythian’s dNFS related blog posts root@nfsfiler01 sysconfig# cat /proc/fs/nfsd/max_block_size 524288 root@nfsfiler01 sysconfig# echo 1048576 > /proc/fs/nfsd/max_block_size root@nfsfiler01 sysconfig# cat /proc/fs/nfsd/max_block_size 1048576 ... ALTER DATABASE OPEN Direct NFS: attempting to mount /u02 on filer KUKARACHA2 defined in oranfstab Direct NFS: channel config is: channel id [0] local [192.168.51.30] path [192.168.51.21] Direct NFS: mount complete dir /u02 on KUKARACHA2 mntport 883 nfsport 2049 Direct NFS: channel id [0] path [192.168.51.21] to filer [KUKARACHA2] via local [192.168.51.30] is UP Direct NFS: channel id [1] path [192.168.51.21] to filer [KUKARACHA2] via local [192.168.51.30] is UP Beginning crash recovery of 1 threads ... © 2012 – Pythian
- 25. init.ora events to set ● If you are confused and need more information then set the following events and access NFS files event "19392 trace name context forever, level 8" # kgnfs event "19394 trace name context forever, level 8" # skgnfs event "19396 trace name context forever, level 2" # kgodm event="19398 trace name context forever, level 128" alter system set event="19392 trace name context forever, level 8","19394 trace name context forever, level 8","19396 trace name context forever, level 2","19398 trace name context forever, level 128" scope=spfile; ● Read trace file, search MOS/Google, ask others ( Yury ;) © 2012 – Pythian
- 26. References Direct NFS: FAQ [ID 954425.1] Example About How To Setup DNFS On Oracle Release 11.2 [ID 1452614.1] How to Setup Direct NFS client multipaths in same subnet [ID 822481.1] DB Hangs When DNFS is Enabled on UEK kernel [ID 1460787.1] DB hang and ORA-600 [2116] errors when enabling DNFS [ID 1484422.1] DNFS CANNOT MOUNT FILESYSTEM AND DATABASE FAILED WITH ORA-600 [2116] AFTER A REBOOT OF SERVER DUE TO POWER FAILURE [ID 1480788.1] DATABASE STARTUP HANGS AT MOUNTING CONTROLFILE WHEN DNFS IS ENABLED. [ID 971406.1] Database Startup Failed with "Direct NFS: please check that oradism is setuid“ [1430654.1] ORA-600 [2116] Using The Veritas Odm Lib Oracle Fails To Mount [ID 418603.1] OERI [2116] [900] during database mount with > 13 instances [ID 9790947.8] TESTCASE Step by Step - Configure Direct NFS Client (DNFS) on Windows [ID 1468114.1] Database Alert Log entries: Direct NFS: Failed to set socket buffer size.wtmax=[1048576] rtmax=[1048576], errno=-1 [ID 1352886.1] @kevinclosson Oracle 11g, Direct NFS Client, An Oracle White Paper + http://bit.ly/QU3w82 @leight0nn http://blogs.griddba.com/2012/02/direct-nfs-speeds-up-data-pump.html @rene_kundersma https://blogs.oracle.com/XPSONHA/entry/using_dnfs_for_test_purposes @yvelikanov http://www.pythian.com/news/tag/dnfs/ © 2012 – Pythian
- 27. Your best friends OR things to keep in mind Network IO SPEED efficiency © 2012 – Pythian
- 28. Thank you and Q&A To contact us… sales@pythian.com 1-877-PYTHIAN To follow us… http://www.pythian.com/news/ http://www.facebook.com/pages/The-Pythian-Group/163902527671 @pythian @pythianjobs http://www.linkedin.com/company/pythian © 2012 – Pythian
Editor's Notes
- COMMENTS:-- It is understandable that without my comments it may be challenging to understand the full content of the presentation.-- For some most controversial slides (e.g. performance, dNFS setup) I put comments in the notes. Please try to check if there is a comment from me.-- I am deliverable using dNFS (small cap ‘d’) in the slides to make it looks close to kNFS-- In most available documentation the DNFS is used-- For the AUSOUG conference I will have 45 minutes only.-- AUSOUG conference is relatively small event (can’t compare to OOW most US events). Therefore there are different level participants. I should cover basics and touch some advanced topics at the same time. Can’t talk details as much as I would on OOW or Collaborate.-- I would like to give more details on performance improvement results however I wouldn't have time to cover the rest than-- Some slides have animation and I would advice you to run though it get an educated guess that I am going to talk about on those slides
- NOTES:-- This is true only if there are no other bottlenecks-- We are talking about 1.2ms to 0.5ms difference here-- Most often systems are limited by HW (Network equipment, Storage speed) and therefore we will not see 300% improvement-- You may expect to see performance improvements somewhere between 0% and 300%kNFS Physical reads: 12,042.2 65,923.3db file sequential read 791,093 1,370 2 97.6 User I/ODB CPU 53 3.81.731781219 msavg PIO6.6995916/10000 CPU sec per PIOdNFS Physical reads: 37,298.0 193,878.0db file sequential read 2,326,535 1,229 1 92.5 User I/ODB CPU 312 23.50.52825339 msavg PIO13.410501/10000 CPU sec per PIODirect storage Physical reads: 53,897.5 281,475.3db file sequential read 3,377,685 1,224 0 91.9 User I/ODB CPU 229 17.20.362378375 msavg PIO6.7797915/10000 CPU sec per PIO Elapsed: 1.09 (mins) DB Time(s): 21.4 116.9 0.36 4.94 DB CPU(s): 0.8 4.4 0.01 0.19 Logical reads: 12,140.1 66,458.8 Block changes: 41.8 228.8 Physical reads: 12,042.2 65,923.3db file sequential read 791,093 1,370 2 97.6 User I/ODB CPU 53 3.8awr_0w_22r.20121023_201639.txtTue Oct 23 20:16:40 EDT 2012real 1m13.117suser 0m0.576ssys 0m1.281s Elapsed: 1.04 (mins) DB Time(s): 21.3 110.7 0.13 4.68 DB CPU(s): 5.0 26.0 0.03 1.10 Logical reads: 37,408.2 194,450.9 Block changes: 33.3 173.0 Physical reads: 37,298.0 193,878.0db file sequential read 2,326,535 1,229 1 92.5 User I/ODB CPU 312 23.5awr_0w_22r.20121023_203540.txtTue Oct 23 20:35:40 EDT 2012Elapsed: 1.04 (mins) DB Time(s): 21.3 111.0 0.09 4.69 DB CPU(s): 3.7 19.1 0.02 0.81 Logical reads: 54,761.2 285,985.7 Block changes: 40.5 211.4 Physical reads: 53,897.5 281,475.3db file sequential read 3,377,685 1,224 0 91.9 User I/ODB CPU 229 17.2awr_0w_22r.20121023_183221.txtTue Oct 23 18:32:21 EDT 2012
- Picture – source URL: http://www.bannerblog.com.au/news/picts/thumbs_up.jpgGrid DBA – my friend,Leighton L. Nelson (@leight0nn) blogs about how he speeded up Data Pump using dNFS“Direct NFS speeds up Data Pump”http://blogs.griddba.com/2012/02/direct-nfs-speeds-up-data-pump.htmlAny other references are welcome! Let me know about others good examples.Why ASM in NAS ?https://twitter.com/yvelikanov/status/260674761380749312Yury@yvelikanovPersonally I don't see the point building ASM on (d)NFS. ASM suppose to exclude unnecessary layers. In NAS case it adds an additional layer.Kevin Closson @kevinclosson. @yvelikanov @netofrombrazil @leight0nn Simple, a) Standard Edition 1 and b) ASM striping between filers. "a" is mandatory.Leighton L. Nelson @leight0nn@yvelikanov @netofrombrazil easy storage migration?Yury @yvelikanov@leight0nn @netofrombrazil u can use Incr refreshable data files copies to migrate on a FS (short downtime). But I do agree. ASM no downtimehttp://bit.ly/RTTkxnGuenadiJilevski@gjilevski@netofrombrazil @yvelikanov @leight0nnEx.ASM on additional NFS for quorum of the vote disk in extended RAC clusters.https://twitter.com/simon_haslam/status/260892761102901248Simon Haslam@simon_haslam@netofrombrazil @leight0nn @yvelikanovSo you're saying ASM on dNFS is good? Or just dNFS with anything (say, OMF instead of ASM)?Yury @yvelikanov@netofrombrazil @simon_haslam @leight0nn @NetApp The right question is: would you recommend naked NFS or ASM on top of NFS?Yury @yvelikanov@yvelikanov @netofrombrazil @simon_haslam @leight0nnOn top of @NetApp of cause. Or it doesn't matter for you as far as #NetApp is in useneto from Brazil @netofrombrazil@yvelikanov @simon_haslam @leight0nn @netapp NFS. ASM on top of D or K NFS only in special cases IMHOREF:https://plus.google.com/u/1/107075205411714880234/posts/G7EPReaJGvFDirect NFS does not support Oracle Clusterware files. REF: http://docs.oracle.com/cd/E11882_01/install.112/e22489/storage.htm#CDEBGJAGNFS file system on a certified NAS filer => OCR and Voting Disk Files => YeskNFS is supported => dNFS isn't for OCRsGood catch +Arup Nanda
- Simplified - TCP still involved kernel THX to @fritshooglandfrom Twitter=B===Martin Bach@MartinDBADNFS Mostly in USER Mode (Good for VM solutions)does it make sense to say that one of the main benefits is that using dNFS you stay (mostly?) in user mod, whereas knfs requires transitions into the kernel (want to trace using perf, but haven't had time).According to James MorledNFS is great for vmware where all user mode code is quick, but kernel transitions (requiring interrupts on x86) take longer since the hypervisor has to "trap" the instruction and translate it to be safe for multiple guests.=E===Martin Bach@MartinDBAAdditional comment from@fritshoogland: Personally, I would draw a nfsd square which gets the line from io client, partly in/partly outside of kernel, and remove that with DNFSCAN kNFS work as well as dNFS?https://twitter.com/yvelikanov/status/260893343150653440Yury@yvelikanov@netofrombrazil @kevinclosson Did I get it right? We can tune kNFS to work on dNFS speed if we invest a lot of NFS expert time?Additional comment from @MartinDBA:The only really cool think worth mentioning is that you have less transitions into kernel code which is good for vitrualized environments. I'm sure if you tested dNFS on a virtual machine it would clearly beat kNFS!Additional comment from @kevinclosson:DNFS addresses circa-2004 NFS and bonding weaknesses specific Oracle I/O profile and OSs. Times change.http://bit.ly/RTRLQ5Kevin Closson @kevinclosson.@netofrombrazil @leight0nn @yvelikanov Even though Solaris needs to die really badly, you might find Sol 11 x64 dNFS and kNFS show parityneto from Brazil @netofrombrazil@kevinclosson @leight0nn @yvelikanov :-) knfs well tuned works good :-)neto from Brazil @netofrombrazil@kevinclosson @leight0nn @yvelikanov OL works pretty well too. I've got 2GBytes per second with KNFS :-)@kevinclosson @yvelikanov agreed you can tune well KNFS but DNFS you can have better optimization. But depends IO that you are generating
- =B===Martin Bach@MartinDBAyou could also use ldd oracle to see the libraries compiled in.additionally pmap or /proc/pid/smaps in RHEL 6.x=E===Martin Bach@MartinDBA
- This slide is intentionally made overcrowded. I have 2 goals here:Provide references and state that the setup is documented with reasonable level of detailsMake a joke :)
- dNFS = Speed = High Availability = Scalability (?reduced CPU as we are skipping longer code path?)http://docs.oracle.com/cd/E11882_01/install.112/e22489/storage.htm#CWLIN2743.2.3.4 Specifying Network Paths with the Oranfstab FileDirect NFS can use up to four network paths defined in the oranfstab file for an NFS server. The Direct NFS client performs load balancing across all specified paths. If a specified path fails, then Direct NFS reissues I/O commands over any remaining paths.Could bonding be a good alternative to dNFS?https://twitter.com/gwenshap/statuses/260668780903026688Gwen (Chen) Shapira@gwenshap@yvelikanov you can bond 1Gb nics, but for each client-server pair, you will still only get 1Gb line. Tricky things :)neto from Brazil @netofrombrazil@yvelikanov @gwenshap @jantup 2 x 1Gbit max you can get 240MB/s either k or d [nfs]https://twitter.com/gwenshap/statuses/260958482059112449Gwen (Chen) Shapira@gwenshap@netofrombrazil @kevinclosson@yvelikanov so this is one of the things you need to get right when configuring knfs, but dnfs handles for youGwen (Chen) Shapira @gwenshap@netofrombrazil @kevinclosson @yvelikanov how do you get bonded 2Gb with knfs? Linux uses 802.3ad bonding. One link per conversation = 1Gb.Details6m43neto from Brazil @netofrombrazil@gwenshap @kevinclosson @yvelikanov LACP mode 4neto from Brazil@netofrombrazil@gwenshap @kevinclosson @yvelikanovand other parameters like backlog, sun rpc etc...https://twitter.com/kevinclosson/status/260959183900405760Kevin Closson@kevinclosson.@gwenshap @netofrombrazil @yvelikanovexactly also dnfs is a combined agg+failover. Simple wire. All my writings from 2006 are sexy now?Kevin Closson@kevinclosson. @yvelikanov @gwenshap good lord, gus, read the paper .. I didn't waste ink:http://www.oracle.com/technetwork/articles/directnfsclient-11gr1-twp-129785.pdf … who wrote that?@netofrombrazil @gwenshap @jantup My questions to you is: Can !!! 1 !!! session get 240MB/s out of 2 x 1Gbit ?Kevin Closson @kevinclosson@netofrombrazil @yvelikanov @gwenshap @jantupdepends on your database host (nfs client) kernelhttps://www.google.com/search?num=100&hl=en&site=&source=hp&q=closson+%2Bdirect+NFS&oq=closson+%2Bdirect+NFS&gs_l=hp.3...2255.7134.0.7296.19.18.0.0.0.0.419.2714.4j5j2j3j1.15.0.les%3Bcesh..0.0...1.1.mDyX7Tu3XpI …Kevin Closson @kevinclosson@gwenshap @jantup @yvelikanov depends on the form of bonding. dNFS is really better for aggregating NICs.neto from Brazil @netofrombrazil@yvelikanov @gwenshap @jantup one tcp session or 1 thread?Yury @yvelikanov@netofrombrazil @gwenshap @jantup 1 database session (unix process, foreground process, full table scan)neto from Brazil @netofrombrazil@yvelikanov @gwenshap @jantup of course you can :-)neto from Brazil @netofrombrazil@netofrombrazil @yvelikanov @gwenshap @jantup one full table scan works if you have the right conf for db multi block readKevin Closson @kevinclosson@yvelikanov @netofrombrazil @gwenshap @jantup a single foregound on modern CPU would have not trouble saturating 2x1GbE (240MB/s)Kevin Closson @kevinclosson. @netofrombrazil @yvelikanov @gwenshap @jantup No surprise. But, hold it, Manly Man doesn't use NFS for Oraclehttps://www.google.com/search?num=100&hl=en&biw=1507&bih=707&q=manly+man+NFS&oq=manly+man+NFS&gs_l=serp.3..0l10.12633532.12640635.0.12640847.13.13.0.0.0.0.453.2518.2j1j5j0j2.10.0.les%3Bcesh..0.0...1.1.6bgcFszDVok …neto from Brazil @netofrombrazil@netofrombrazil @kevinclosson @yvelikanov @gwenshap@jantup for TCP the rule is 1 Hz to process 1 bit - 1GHz process 1Gbit got it?https://twitter.com/Djelibeybi/status/260669539027648512Djelibeybi@Djelibeybi@yvelikanov you can get more bandwidth with a LACP (802.3ad) bond of two or more NICs. Not a linear scale, though and needs switch support.Leighton L. Nelson @leight0nn@Djelibeybi @yvelikanov Read somewhere multiple paths on diff subnet recommended over LACP vif. Could be wrong.
- Local: Is IP on the DB server you cant connections to go thoughPath: Is IP on the Filler you can the connections to end upExport, Mount – is a pair that finalise each block of informationYou can have more than one (up to 4) Local/Path parameters specifiedUse Dontroute if you about to use several pairs of IPs from the same networkMnt_timeout is in seconds and defaults to 10 minutes. It seems a bit too high to me. I would set it to 1 minute (disclaimer: didn’t have much experience in failing NFS area)You can specify block per mountI suspect that there is a limit of IP connections each session can have. As of now I didn’t hit this limit. But at the same time I didn’t use too many connections per sesison (4 max as o now). If you have experience please let me know (@yvelikanov)Special thanks to @pioro for comments on UEK
- Local: Is IP on the DB server you cant connections to go thoughPath: Is IP on the Filler you can the connections to end upExport, Mount – is a pair that finalise each block of informationYou can have more than one (up to 4) Local/Path parameters specifiedUse Dontroute if you about to use several pairs of IPs from the same networkMnt_timeout is in seconds and defaults to 10 minutes. It seems a bit too high to me. I would set it to 1 minute (disclaimer: didn’t have much experience in failing NFS area)You can specify block per mountI suspect that there is a limit of IP connections each session can have. As of now I didn’t hit this limit. But at the same time I didn’t use too many connections per sesison (4 max as o now). If you have experience please let me know (@yvelikanov)Special thanks to @pioro for comments on UEK
- Remove limit on the number of diskmon slaves [bug: 9842238]ORACLE DATABASE WILL NOT OPEN [bug: 14383403]LRGIONFS RUN IS FAILING ON WIN2K8 R2 [bug: 13689216]DATABASE STARTUP AND QUERY TAKING HUGE TIME WHEN DNFS IS ENABLED [bug: 13510654]LGWR hangs for long periods using DNFS - CF waits likely [bug: 9556189]