HA group notification from cluster out of inodes error
HA Group Notification from Cluster-02
(OUT OF INODES) ERROR
Error details: This error indicates that the certain 'volume(s)' is out of inodes. Autosupport error
does not list which volume has run out of inodes but only the originating node. However by running
the command: df -ih on the specific node of the cluster mentioned in the error (In this case -02), one
can find out the specific volume(s) that indicates 100% inodes used.
Please note: ONTAP does not do 'decimal' calculations, and therefore any value in decimal is
rounded to next digit. In our case, ONTAP has been complaining on '100%-inode' consumption for
some time now. Running 'df -ih' command reveals that there are still 20K inodes free, but as it was
99.x, it is alerted as 100% usage.
What exactly is inode?
An inode is a meta-data that defines a file, it has an integer value, and it contains all the information
about the file except its filename which is stored in the directory entry. Again, a directory is just
Data ONTAP allocates one inode per 32KB of data in a volume by default. This value can be increased
by reducing the KB size down up to 4KB only.
Bydefault: 1 inode ---> 32 KB
In most cases that I have seen over the years: ONTAP starts complaining about 'out-of-inodes' error
despite enough space on the volume. This basically suggests that the volume may contain lots of
small files which are less than 32KB in size and utilizing the inode. As this is a common knowledge
that each file that is created must be mapped to an inode and inode basically contains the meta-data
about the file such as - size,where it is on the volume, number of bytes, block pointers, permissions,
access,modification,creation time etc.
Let’s see one such scenario:
I create a 500GB volume
I will get = 500x1024x1024KB / 32KB = 16384000 inodes
That's about 1 million inodes I have got, and technically I can have 1 million files [Provided each file
is exactly 32KB]. However, if my volume contains only office shares, then most likely it will have files
less than 32KB.
Let's assume I have files of 16KB size and there are one million plus inodes = 16x16384000 =
262144000/262GB in total space consumed out of 500GB, however all my inodes are consumed and
the NetApp will start complaining 100% inode used.
IMPACT: New writes will fail.
In this case, there is no use of extending the volume space, as this is not a space issue, rather inodes
must be increased.
Recommended practice: 2% of the total inodes
= 2 % 16384000
= 16384000 + 327680
Syntax: volume modify –vserver <vserver name> -volume <volume name> -files <number of files>
cluster::> volume modify -vserver vsan01 -volume abc -files 16711680
Volume modify successful on volume abc of Vserver vsan01.
There are 3 ways to check inodes:
1) volume show -volume <volume-name> -fields files = shows total inodes
2) volume show -volume <volume-name> -instance = shows total & consumed inodes
3) df –ih <volume-name>[@ node level] = shows inode-used, inode-free & inode-%