=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2012.11.13 17:11:43 =~=~=~=~=~=~=~=~=~=~=~=
Tue Nov 13 16:11:22 AKST [akfiler13: cf.fsm.takeoverByPartnerDisabled:notice]:
Cluster monitor: takeover of akfiler13 by akfiler12 disabled (interconnect
error)
Tue Nov 13 16:11:22 AKST [akfiler13: cf.fsm.firmwareStatus:info]: Cluster
monitor: partner rebooting
Tue Nov 13 16:11:22 AKST [akfiler13: cf.fsm.nfo.partnerShutdown:warning]:
Negotiated failover: partner has shutdown
Tue Nov 13 16:11:22 AKST [akfiler13: cf.fsm.takeover.nfo:info]: Cluster monitor:
takeover attempted after 'cf takeover'. command
Tue Nov 13 16:11:22 AKST [akfiler13: cf.fsm.stateTransit:warning]: Cluster
monitor: UP --> TAKEOVER
Tue Nov 13 16:11:22 AKST [akfiler13: cf.fm.takeoverStarted:warning]: Cluster
monitor: takeover started
Tue Nov 13 16:11:22 AKST [akfiler12/akfiler13: coredump.spare.none:info]: No
sparecore disk was found.
Tue Nov 13 16:11:23 AKST [akfiler13: nv.partner.disabled:info]: NVRAM takeover:
Partner NVRAM was disabled.
Tue Nov 13 16:11:24 AKST [akfiler12/akfiler13:
wafl.takeover.nvram.missing:info]: WAFL takeover: No WAFL nvlog records were
found to replay.
Tue Nov 13 16:11:24 AKST [akfiler12/akfiler13: wafl.replay.done:info]: WAFL log
replay completed, 0 seconds
Tue Nov 13 16:11:34 AKST [akfiler12/akfiler13: syslogd:info]: Lookup of
lx9239.ak.providence.org failed with DNS server 170.173.48.15: Can't assign
requested address.
Tue Nov 13 16:11:56 AKST [akfiler12/akfiler13: kern.syslogd.error:error]: syslog
daemon error logerror syslogd: Couldn't find address for host name
"lx9239.ak.providence.org": Can't get order of sources to search .
Tue Nov 13 16:11:56 AKST [akfiler12/akfiler13: kern.syslogd.restarted:info]:
syslogd: Restarted.
add net default: gateway 10.21.20.1
There are 3 spare disks; you may want to use the vol or aggr command
to create new volumes or aggregates or add disks to the existing aggregate.
CIFS partner server is running.
Tue Nov 13 16:11:57 AKST [akfiler13 (takeover): cf.rsrc.transitTime:notice]: Top
Takeover transit times syslog=30370, wafl=1398 {pvvols_to_done=939,
prvol_to_done=261, paggrs_to_done=197, part_vols_mnt_end=102, prvol_mnt_end=3},
wafl_sync=1324, registry_postrc_phase1=508, wafl_restart=372 {vdisk=210,
restarters=162}, rc=267 {always_do_just_after_etc_rc=57, hostname=45,
ifconfig=44, raid_config_prerc=40, always_do_just_before_etc_rc=26, options=21,
options=12, options=11, ems_postrc=1}, registry_prerc=222,
Tue Nov 13 16:11:57 AKST [akfiler13 (takeover): cf.fm.takeoverComplete:warning]:
Cluster monitor: takeover completed
Tue Nov 13 16:11:57 AKST [akfiler13 (takeover): cf.fm.takeoverDuration:warning]:
Cluster monitor: takeover duration time is 35 seconds

akfiler13(takeover)> Tue Nov 13 16:12:01 AKST [akfiler13 (takeover):
monitor.globalStatus.critical:CRITICAL]: This node has taken over akfiler12.
Tue Nov 13 16:12:07 AKST [akfiler12/akfiler13:
nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have
completed for the partner server.
Tue Nov 13 16:12:15 AKST [akfiler13 (takeover): asup.smtp.fail:warning]:
Autosupport mail to "" failed (Protocol error) for message: CLUSTER TAKEOVER
COMPLETE MANUAL
Tue Nov 13 16:12:15 AKST [akfiler13 (takeover): asup.smtp.sent:notice]: Cluster
Notification mail sent: Cluster Alert from akfiler13
Tue Nov 13 16:12:32 AKST [akfiler12/akfiler13: asup.smtp.sent:notice]: Cluster
Notification mail sent: Cluster Notification from akfiler12 (REBOOT (CLUSTER
TAKEOVER)) WARNING
Tue Nov 13 16:12:34 AKST [akfiler13 (takeover):
cf.fsm.releasingReservations:info]: Cluster monitor: Releasing disk reservations
in preparation for giveback
Tue Nov 13 16:12:34 AKST [akfiler13 (takeover): cf.fm.diskRelease:info]: Cluster
monitor: released disk reservations.
Tue Nov 13 16:12:34 AKST [akfiler13 (takeover): asup.smtp.sent:notice]: Cluster
Notification mail sent: Cluster Notification from akfiler13 (CLUSTER TAKEOVER
COMPLETE MANUAL) INFO
Tue Nov 13 16:12:34 AKST [akfiler13 (takeover):
cf.nm.nicTransitionDown:warning]: Cluster Interconnect link 0 is DOWN
Tue Nov 13 16:12:34 AKST [akfiler13 (takeover): cf.rv.notConnected:error]:
Connection for cfo_rv failed
Tue Nov 13 16:12:36 AKST [akfiler13 (takeover): cf.nm.nicTransitionUp:info]:
Interconnect link 0 is UP

akfiler13(takeover)> Tue Nov 13 16:13:03 AKST [akfiler13 (takeover):
asup.smtp.sent:notice]: Cluster Notification mail sent: Cluster Notification
from akfiler13 (PERFORMANCE SNAPSHOT) INFO

akfiler13(takeover)>
akfiler13(takeover)>
akfiler13(takeover)>
akfiler13(takeover)> Tue Nov 13 16:21:12 AKST [akfiler13 (takeover):
cf.hwassist.missedKeepAlive:warning]: Cluster hw_assist: missed keep alive alert
from partner(akfiler12).
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.0
(5000C500:4C629A33:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.1
(5000C500:4C5AB147:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.2
(5000C500:4C57270F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.3
(5000C500:4C57ADEB:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.4
(5000C500:4C5FF017:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.5
(5000C500:4C576C93:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.6
(5000C500:4C5101C7:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.7
(5000C500:4C67D387:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.8
(5000C500:4C5F261B:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.9
(5000C500:4C50E7D7:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.10
(5000C500:4C4E646F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.11
(5000C500:4C5E9B1F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.12
(5000C500:4C65FFB3:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.13
(5000CCA0:1FC2210C:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.14
(5000C500:4C548E33:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.15
(5000C500:4C698D4B:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.16
(5000C500:4C532023:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.17
(5000C500:4C4E02DF:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.18
(5000C500:4C5AB39F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.19
(5000C500:4C581EE3:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.20
(5000C500:4C549B67:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.21
(5000C500:4C514A57:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.22
(5000C500:4C5A8EA3:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.23
(5000C500:4C56B1BF:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.0
(5000C500:4C60EE5F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.1
(5000C500:4C50E917:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.2
(5000C500:4C5485FF:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.3
(5000C500:4C50F503:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.4
(5000C500:4C54A897:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.5
(5000C500:4C5B160F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.6
(5000C500:4C4F0D5F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.7
(5000C500:4C5823CB:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.8
(5000C500:4C359B67:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.9
(5000C500:4C5B3D63:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.10
(5000C500:4C4E7D2F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.11
(5000C500:4C50C46F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.12
(5000C500:4C482ED7:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.13
(5000C500:4C549F9B:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.14
(5000C500:4C4DE207:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.15
(5000C500:4C51C327:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.16
(5000C500:4C5E1AEF:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.17
(5000C500:4C5EF477:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.18
(5000C500:4C56B103:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.19
(5000C500:4C51157F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.20
(5000C500:4C53F92F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.21
(5000C500:4C542753:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.22
(5000C500:4C40C7C3:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:21:59 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.23
(5000C500:4C5E8DEF:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) has recently changed or the node (akfiler12) is missing the disk.
Tue Nov 13 16:22:14 AKST [akfiler13 (takeover): asup.smtp.fail:warning]:
Autosupport mail to "" failed (Protocol error) for message: CLUSTER ERROR:
DISK/SHELF COUNT MISMATCH
Tue Nov 13 16:22:14 AKST [akfiler13 (takeover): asup.smtp.sent:notice]: Cluster
Notification mail sent: Cluster Alert from akfiler13
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.0
(5000C500:4C629A33:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.1
(5000C500:4C5AB147:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.2
(5000C500:4C57270F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.3
(5000C500:4C57ADEB:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.4
(5000C500:4C5FF017:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.5
(5000C500:4C576C93:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.6
(5000C500:4C5101C7:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.7
(5000C500:4C67D387:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.8
(5000C500:4C5F261B:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.9
(5000C500:4C50E7D7:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.10
(5000C500:4C4E646F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.11
(5000C500:4C5E9B1F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.12
(5000C500:4C65FFB3:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.13
(5000CCA0:1FC2210C:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.14
(5000C500:4C548E33:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.15
(5000C500:4C698D4B:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.16
(5000C500:4C532023:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.17
(5000C500:4C4E02DF:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.18
(5000C500:4C5AB39F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.19
(5000C500:4C581EE3:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.20
(5000C500:4C549B67:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.21
(5000C500:4C514A57:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.22
(5000C500:4C5A8EA3:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3a.10.23
(5000C500:4C56B1BF:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.0
(5000C500:4C60EE5F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.1
(5000C500:4C50E917:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.2
(5000C500:4C5485FF:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.3
(5000C500:4C50F503:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.4
(5000C500:4C54A897:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.5
(5000C500:4C5B160F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.6
(5000C500:4C4F0D5F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.7
(5000C500:4C5823CB:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.8
(5000C500:4C359B67:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.9
(5000C500:4C5B3D63:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.10
(5000C500:4C4E7D2F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.11
(5000C500:4C50C46F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.12
(5000C500:4C482ED7:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.13
(5000C500:4C549F9B:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.14
(5000C500:4C4DE207:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.15
(5000C500:4C51C327:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.16
(5000C500:4C5E1AEF:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.17
(5000C500:4C5EF477:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.18
(5000C500:4C56B103:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.19
(5000C500:4C51157F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.20
(5000C500:4C53F92F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.21
(5000C500:4C542753:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.22
(5000C500:4C40C7C3:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
3d.11.23
(5000C500:4C5E8DEF:00000000:00000000:00000000:00000000:00000000:00000000:0000000
0:00000000) in its inventory.
Tue Nov 13 16:22:33 AKST [akfiler13 (takeover): asup.smtp.sent:notice]: Cluster
Notification mail sent: Cluster Notification from akfiler13 (CLUSTER ERROR:
DISK/SHELF COUNT MISMATCH) ERROR
Tue Nov 13 16:24:00 AKST [akfiler13 (takeover):
cf.nm.nicTransitionDown:warning]: Cluster Interconnect link 0 is DOWN
Tue Nov 13 16:24:00 AKST [akfiler13 (takeover): cf.rv.notConnected:error]:
Connection for cfo_rv failed
Tue Nov 13 16:25:00 AKST [akfiler13 (takeover): cf.nm.nicTransitionUp:info]:
Interconnect link 0 is UP
Tue Nov 13 16:25:12 AKST [akfiler13 (takeover): cf.hwassist.recvKeepAlive:info]:
Cluster hw_assist: Received hw_assist KeepAlive alert from partner(akfiler12).
Tue Nov 13 16:25:31 AKST [akfiler13 (takeover): cf.nm.nicReset:warning]:
Initiating soft reset on Cluster Interconnect card 0 due to rendezvous
connection timeout
Tue Nov 13 16:25:33 AKST [akfiler13 (takeover): cf.nm.nicReset:warning]:
Initiating soft reset on Cluster Interconnect card 0 due to rendezvous jammed
Tue Nov 13 16:26:03 AKST last message repeated 10 times
Tue Nov 13 16:26:06 AKST [akfiler13 (takeover): cf.nm.nicReset:warning]:
Initiating soft reset on Cluster Interconnect card 0 due to rendezvous jammed

akfiler13(takeover)>
akfiler13(takeover)>
akfiler13(takeover)> storage show disk -p
PRIMARY PORT SECONDARY PORT SHELF BAY
-------- ---- --------- ---- ---------
0a.16     B    0d.16      A     1    0
0a.17     B    0d.17      A     1    1
0a.18     B    0d.18      A     1    2
0a.19     B    0d.19      A     1    3
0a.20     B    0d.20      A     1    4
0a.21     B    0d.21      A     1    5
0a.22     B    0d.22      A     1    6
0a.23     B    0d.23      A     1    7
0a.24     B    0d.24      A     1    8
0a.25     B    0d.25      A     1    9
0a.26     B    0d.26      A     1   10
0a.27     B    0d.27      A     1   11
0a.28     B    0d.28      A     1   12
0a.29     B    0d.29      A     1   13
0a.32     B    0d.32      A     2    0
0a.33     B    0d.33      A     2    1
0a.34     B    0d.34      A     2    2
0a.35     B    0d.35      A     2    3
0a.36     B    0d.36      A     2    4
0a.37     B    0d.37      A     2    5
0a.38     B    0d.38      A     2    6
0a.39     B    0d.39      A     2    7
0a.40     B    0d.40      A     2    8
0a.41     B    0d.41      A     2    9
0a.42     B    0d.42      A     2   10
0a.43     B    0d.43      A     2   11
0a.44     B    0d.44      A     2   12
0a.45     B    0d.45      A     2   13
0b.16     B    0c.16      A     1    0
0b.17      B   0c.17      A    1    1
0b.18      B   0c.18      A    1    2
0b.19      B   0c.19      A    1    3
0b.20      B   0c.20      A    1    4
0b.21      B   0c.21      A    1    5
0b.22      B   0c.22      A    1    6
0b.23      B   0c.23      A    1    7
0b.24      B   0c.24      A    1    8
0b.25      B   0c.25      A    1    9
0b.26      B   0c.26      A    1   10
0b.27      B   0c.27      A    1   11
0b.28      B   0c.28      A    1   12
0b.29      B   0c.29      A    1   13
0b.32      B   0c.32      A    2    0
0b.33      B   0c.33      A    2    1
0b.34      B   0c.34      A    2    2
0b.35      B   0c.35      A    2    3
0b.36      B   0c.36      A    2    4
0b.37      B   0c.37      A    2    5
0b.38      B   0c.38      A    2    6
0b.39      B   0c.39      A    2    7
0b.40      B   0c.40      A    2    8
0b.41      B   0c.41      A    2    9
0b.42      B   0c.42      A    2   10
0b.43      B   0c.43      A    2   11
0b.44      B   0c.44      A    2   12
0b.45      B   0c.45      A    2   13
3a.10.0    B   3d.10.0    A   10    0
3a.10.1    B   3d.10.1    A   10    1
3a.10.2    B   3d.10.2    A   10    2
3a.10.3    B   3d.10.3    A   10    3
3a.10.4    B   3d.10.4    A   10    4
3a.10.5    B   3d.10.5    A   10    5
3a.10.6    B   3d.10.6    A   10    6
3a.10.7    B   3d.10.7    A   10    7
3a.10.8    B   3d.10.8    A   10    8
3a.10.9    B   3d.10.9    A   10    9
3a.10.10   B   3d.10.10   A   10   10
3a.10.11   B   3d.10.11   A   10   11
3a.10.12   B   3d.10.12   A   10   12
3a.10.13   B   3d.10.13   A   10   13
3a.10.14   B   3d.10.14   A   10   14
3a.10.15   B   3d.10.15   A   10   15
3a.10.16   B   3d.10.16   A   10   16
3a.10.17   B   3d.10.17   A   10   17
3a.10.18   B   3d.10.18   A   10   18
3a.10.19   B   3d.10.19   A   10   19
3a.10.20   B   3d.10.20   A   10   20
3a.10.21   B   3d.10.21   A   10   21
3a.10.22   B   3d.10.22   A   10   22
3a.10.23   B   3d.10.23   A   10   23
3d.11.0    A   3a.11.0    B   11    0
3d.11.1    A   3a.11.1    B   11    1
3d.11.2    A   3a.11.2    B   11    2
3d.11.3    A   3a.11.3    B   11    3
3d.11.4    A   3a.11.4    B   11    4
3d.11.5    A   3a.11.5    B   11    5
3d.11.6    A   3a.11.6    B   11    6
3d.11.7    A   3a.11.7    B   11    7
3d.11.8    A   3a.11.8    B   11    8
3d.11.9    A   3a.11.9    B   11    9
3d.11.10   A   3a.11.10   B   11   10
3d.11.11   A   3a.11.11   B   11   11
3d.11.12   A   3a.11.12   B   11   12
3d.11.13 A     3a.11.13   B    11   13
3d.11.14 A     3a.11.14   B    11   14
3d.11.15 A     3a.11.15   B    11   15
3d.11.16 A     3a.11.16   B    11   16
3d.11.17 A     3a.11.17   B    11   17
3d.11.18 A     3a.11.18   B    11   18
3d.11.19 A     3a.11.19   B    11   19
3d.11.20 A     3a.11.20   B    11   20
3d.11.21 A     3a.11.21   B    11   21
3d.11.22 A     3a.11.22   B    11   22
3d.11.23 A     3a.11.23   B    11   23
akfiler13(takeover)> cf givebacklogin as: p375290
p375290@akfiler13.ak.providence.org's password:

akfiler13> Tue Nov 13 16:30:49 AKST [akfiler13:
cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of akfiler12
disabled (partner booting)
Tue Nov 13 16:30:49 AKST [akfiler13: cf.fsm.takeoverByPartnerDisabled:notice]:
Cluster monitor: takeover of akfiler13 by akfiler12 disabled (unsynchronized
log)

akfiler13> Tue Nov 13 16:31:03 AKST [akfiler13:
cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of akfiler12
disabled (interconnect error)
Tue Nov 13 16:31:03 AKST [akfiler13: cf.fsm.takeoverByPartnerDisabled:notice]:
Cluster monitor: takeover of akfiler13 by akfiler12 disabled (interconnect
error)
Tue Nov 13 16:31:04 AKST [akfiler13: cf.partner.short_uptime:warning]: Partner
up for 13 seconds only
Tue Nov 13 16:31:05 AKST [akfiler13: cf.fsm.takeoverOfPartnerDisabled:notice]:
Cluster monitor: takeover of akfiler12 disabled (unsynchronized log)
Tue Nov 13 16:31:05 AKST [akfiler13: cf.fsm.takeoverByPartnerDisabled:notice]:
Cluster monitor: takeover of akfiler13 by akfiler12 disabled (unsynchronized
log)
Tue Nov 13 16:31:07 AKST [akfiler13: cf.fsm.takeoverOfPartnerEnabled:notice]:
Cluster monitor: takeover of akfiler12 enabled
Tue Nov 13 16:31:10 AKST [akfiler13: cf.fsm.takeoverByPartnerEnabled:notice]:
Cluster monitor: takeover of akfiler13 by akfiler12 enabled
Tue Nov 13 16:32:00 AKST [akfiler13: monitor.globalStatus.ok:info]: The system's
global status is normal.

akfiler13> aggr stastus
aggr: No such command "stastus".
The following commands are available; for more information
type "aggr help <command>"
add                  mirror               rename            split
copy                 offline              restrict          status
create               online               scrub             undestroy
destroy              options              show_space        verify
media_scrub
akfiler13> aggr stastus# # # ## # # ## # # ## # # ## # # #atus -r
Aggregate aggr1 (online, raid_dp) (block checksums)
  Plex /aggr1/plex0 (online, normal, active)
     RAID group /aggr1/plex0/rg0 (normal)

      RAID Disk     Device     HA   SHELF BAY CHAN Pool Type     RPM    Used (MB/blks)
Phys (MB/blks)
      ---------     ------     ------------- ---- ---- ---- ----- --------------
--------------
      dparity       3d.11.9     3d    11   9   SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      parity        3d.11.8     3d    11   8   SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.7     3d    11   7   SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.6     3d      11   6    SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.5     3d      11   5    SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.4     3d      11   4    SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.3     3d      11   3    SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.23    3d      11   23   SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.22    3d      11   22   SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.21    3d      11   21   SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.20    3d      11   20   SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.2     3d      11   2    SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.19    3d      11   19   SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.18    3d      11   18   SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.17    3d      11   17   SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.16    3d      11   16   SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.15    3d      11   15   SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.14    3d      11   14   SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.13    3d      11   13   SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.12    3d      11   12   SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.11    3d      11   11   SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.10    3d      11   10   SA:A   -   SAS   15000
560000/1146880000   560208/1147307688
      data          3d.11.1     3d      11   1    SA:A   -   SAS   15000
560000/1146880000   560208/1147307688

Aggregate aggr0 (online, raid_dp) (block checksums)
  Plex /aggr0/plex0 (online, normal, active)
    RAID group /aggr0/plex0/rg0 (normal)

      RAID Disk     Device     HA   SHELF BAY CHAN Pool Type        RPM    Used (MB/blks)
Phys (MB/blks)
      ---------     ------     ------------- ---- ---- ---- ----- --------------
--------------
      dparity       0b.16       0b      1    0    FC:B   -   FCAL 15000
272000/557056000    274845/562884296
      parity        0c.17       0c      1    1    FC:A   -   FCAL 15000
272000/557056000    274845/562884296
      data          0c.18       0c      1    2    FC:A   -   FCAL 15000
272000/557056000    274845/562884296
      data          0b.19       0b      1    3    FC:B   -   FCAL 15000
272000/557056000    274845/562884296
      data          0c.20       0c      1    4    FC:A   -   FCAL 15000
272000/557056000    274845/562884296
      data          0b.21       0b      1    5    FC:B   -   FCAL 15000
272000/557056000    274845/562884296
      data          0c.22       0c      1    6    FC:A   -   FCAL 15000
272000/557056000    274845/562884296
data         0c.23       0c     1   7    FC:A   -   FCAL 15000
272000/557056000   274845/562884296
      data         0b.24       0b     1   8    FC:B   -   FCAL 15000
272000/557056000   274845/562884296
      data         0b.25       0b     1   9    FC:B   -   FCAL 15000
272000/557056000   274845/562884296
      data         0c.26       0c     1   10   FC:A   -   FCAL 15000
272000/557056000   274845/562884296
      data         0b.27       0b     1   11   FC:B   -   FCAL 15000
272000/557056000   274845/562884296
      data         0b.28       0b     1   12   FC:B   -   FCAL 15000
272000/557056000   274845/562884296
      data         0b.32       0b     2   0    FC:B   -   FCAL 15000
272000/557056000   274845/562884296
      data         0b.29       0b     1   13   FC:B   -   FCAL 15000
272000/557056000   274845/562884296
      data         0b.33       0b     2   1    FC:B   -   FCAL 15000
272000/557056000   274845/562884296

    RAID group /aggr0/plex0/rg1 (normal)

      RAID Disk    Device     HA   SHELF BAY CHAN Pool Type     RPM    Used (MB/blks)
Phys (MB/blks)
      ---------    ------     ------------- ---- ---- ---- ----- --------------
--------------
      dparity      0c.34       0c     2   2    FC:A   -   FCAL 15000
272000/557056000   274845/562884296
      parity       0b.35       0b     2   3    FC:B   -   FCAL 15000
272000/557056000   274845/562884296
      data         0c.36       0c     2   4    FC:A   -   FCAL 15000
272000/557056000   274845/562884296
      data         0b.37       0b     2   5    FC:B   -   FCAL 15000
272000/557056000   274845/562884296
      data         0b.38       0b     2   6    FC:B   -   FCAL 15000
272000/557056000   274845/562884296
      data         0b.39       0b     2   7    FC:B   -   FCAL 15000
272000/557056000   274845/562884296
      data         0c.40       0c     2   8    FC:A   -   FCAL 15000
272000/557056000   274845/562884296
      data         0c.41       0c     2   9    FC:A   -   FCAL 15000
272000/557056000   274845/562884296
      data         0c.42       0c     2   10   FC:A   -   FCAL 15000
272000/557056000   274845/562884296
      data         0c.43       0c     2   11   FC:A   -   FCAL 15000
272000/557056000   274845/562884296


Spare disks

RAID Disk   Device      HA SHELF BAY CHAN Pool Type RPM Used (MB/blks)
Phys (MB/blks)
---------   ------      ------------- ---- ---- ---- ----- --------------
--------------
Spare disks for block or zoned checksum traditional volumes or aggregates
spare       0b.45       0b    2   13 FC:B    - FCAL 15000 272000/557056000
274845/562884296
spare       0c.44       0c    2   12 FC:A    - FCAL 15000 272000/557056000
274845/562884296
spare       3d.11.0     3d    11 0    SA:A   - SAS 15000 560000/1146880000
560208/1147307688

Partner disks

RAID Disk     Device    HA   SHELF BAY CHAN Pool Type     RPM   Used (MB/blks)
Phys (MB/blks)
---------   ------     ------------- ---- ---- ---- ----- --------------
--------------
partner     3a.10.13   3a   10   13   SA:B   -   SAS   15000 0/0
560879/1148681096
partner     3a.10.2    3a   10   2    SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.22   3a   10   22   SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.23   3a   10   23   SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.21   3a   10   21   SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.8    3a   10   8    SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.20   3a   10   20   SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.9    3a   10   9    SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.7    3a   10   7    SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.6    3a   10   6    SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.5    3a   10   5    SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.4    3a   10   4    SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.3    3a   10   3    SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.17   3a   10   17   SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.16   3a   10   16   SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.15   3a   10   15   SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.14   3a   10   14   SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.11   3a   10   11   SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.10   3a   10   10   SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.1    3a   10   1    SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.19   3a   10   19   SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.12   3a   10   12   SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.18   3a   10   18   SA:B   -   SAS   15000 0/0
560208/1147307688
partner     3a.10.0    3a   10   0    SA:B   -   SAS   15000 0/0
560208/1147307688
partner     0d.16      0d   1    0    FC:A   -   FCAL 15000 0/0
274845/562884296
partner     0d.23      0d   1    7    FC:A   -   FCAL 15000 0/0
274845/562884296
partner     0a.27      0a   1    11   FC:B   -   FCAL 15000 0/0
274845/562884296
partner     0d.40      0d   2    8    FC:A   -   FCAL 15000 0/0
274845/562884296
partner     0d.25      0d   1    9    FC:A   -   FCAL 15000 0/0
274845/562884296
partner     0d.43      0d   2    11   FC:A   -   FCAL 15000 0/0
274845/562884296
partner     0a.22      0a   1    6    FC:B   -   FCAL 15000 0/0
274845/562884296
partner     0a.17       0a    1   1   FC:B   - FCAL 15000 0/0
274845/562884296
partner     0d.44       0d    2   12 FC:A    - FCAL 15000 0/0
274845/562884296
partner     0a.32       0a    2   0   FC:B   - FCAL 15000 0/0
274845/562884296
partner     0d.35       0d    2   3   FC:A   - FCAL 15000 0/0
274845/562884296
partner     0d.45       0d    2   13 FC:A    - FCAL 15000 0/0
274845/562884296
partner     0d.41       0d    2   9   FC:A   - FCAL 15000 0/0
274845/562884296
partner     0d.19       0d    1   3   FC:A   - FCAL 15000 0/0
274845/562884296
partner     0d.42       0d    2   10 FC:A    - FCAL 15000 0/0
274845/562884296
partner     0a.36       0a    2   4   FC:B   - FCAL 15000 0/0
274845/562884296
partner     0d.26       0d    1   10 FC:A    - FCAL 15000 0/0
274845/562884296
partner     0a.21       0a    1   5   FC:B   - FCAL 15000 0/0
274845/562884296
partner     0a.39       0a    2   7   FC:B   - FCAL 15000 0/0
274845/562884296
partner     0a.34       0a    2   2   FC:B   - FCAL 15000 0/0
274845/562884296
partner     0d.29       0d    1   13 FC:A    - FCAL 15000 0/0
274845/562884296
partner     0a.20       0a    1   4   FC:B   - FCAL 15000 0/0
274845/562884296
partner     0a.18       0a    1   2   FC:B   - FCAL 15000 0/0
274845/562884296
partner     0a.37       0a    2   5   FC:B   - FCAL 15000 0/0
274845/562884296
partner     0d.28       0d    1   12 FC:A    - FCAL 15000 0/0
274845/562884296
partner     0a.24       0a    1   8   FC:B   - FCAL 15000 0/0
274845/562884296
partner     0a.38       0a    2   6   FC:B   - FCAL 15000 0/0
274845/562884296
partner     0a.33       0a    2   1   FC:B   - FCAL 15000 0/0
274845/562884296
akfiler13> Tue Nov 13 16:35:48 AKST [akfiler13: asup.smtp.fail:warning]:
Autosupport mail to "" failed (Protocol error) for message: CLUSTER GIVEBACK
COMPLETE
Tue Nov 13 16:35:49 AKST [akfiler13: asup.smtp.sent:notice]: Cluster
Notification mail sent: Cluster Alert from akfiler13
Tue Nov 13 16:36:07 AKST [akfiler13: asup.smtp.sent:notice]: Cluster
Notification mail sent: Cluster Notification from akfiler13 (CLUSTER GIVEBACK
COMPLETE) INFO

akfiler13>
akfiler13> Tue Nov 13 17:00:00 AKST [akfiler13: kern.timed.adjust:info]: server
'time.ak.providence.org' reports the appliance date is slow by 0.002 seconds.
Adjusting date.
Tue Nov 13 17:00:00 AKST [akfiler13: kern.uptime.filer:info]:   5:00pm up 201
days, 22:09 27134312584 NFS ops, 3698206 CIFS ops, 59 HTTP ops, 0 FCP ops, 0
iSCSI ops

akfiler13>
akfiler13>

Ak13 pam

  • 1.
    =~=~=~=~=~=~=~=~=~=~=~= PuTTY log2012.11.13 17:11:43 =~=~=~=~=~=~=~=~=~=~=~= Tue Nov 13 16:11:22 AKST [akfiler13: cf.fsm.takeoverByPartnerDisabled:notice]: Cluster monitor: takeover of akfiler13 by akfiler12 disabled (interconnect error) Tue Nov 13 16:11:22 AKST [akfiler13: cf.fsm.firmwareStatus:info]: Cluster monitor: partner rebooting Tue Nov 13 16:11:22 AKST [akfiler13: cf.fsm.nfo.partnerShutdown:warning]: Negotiated failover: partner has shutdown Tue Nov 13 16:11:22 AKST [akfiler13: cf.fsm.takeover.nfo:info]: Cluster monitor: takeover attempted after 'cf takeover'. command Tue Nov 13 16:11:22 AKST [akfiler13: cf.fsm.stateTransit:warning]: Cluster monitor: UP --> TAKEOVER Tue Nov 13 16:11:22 AKST [akfiler13: cf.fm.takeoverStarted:warning]: Cluster monitor: takeover started Tue Nov 13 16:11:22 AKST [akfiler12/akfiler13: coredump.spare.none:info]: No sparecore disk was found. Tue Nov 13 16:11:23 AKST [akfiler13: nv.partner.disabled:info]: NVRAM takeover: Partner NVRAM was disabled. Tue Nov 13 16:11:24 AKST [akfiler12/akfiler13: wafl.takeover.nvram.missing:info]: WAFL takeover: No WAFL nvlog records were found to replay. Tue Nov 13 16:11:24 AKST [akfiler12/akfiler13: wafl.replay.done:info]: WAFL log replay completed, 0 seconds Tue Nov 13 16:11:34 AKST [akfiler12/akfiler13: syslogd:info]: Lookup of lx9239.ak.providence.org failed with DNS server 170.173.48.15: Can't assign requested address. Tue Nov 13 16:11:56 AKST [akfiler12/akfiler13: kern.syslogd.error:error]: syslog daemon error logerror syslogd: Couldn't find address for host name "lx9239.ak.providence.org": Can't get order of sources to search . Tue Nov 13 16:11:56 AKST [akfiler12/akfiler13: kern.syslogd.restarted:info]: syslogd: Restarted. add net default: gateway 10.21.20.1 There are 3 spare disks; you may want to use the vol or aggr command to create new volumes or aggregates or add disks to the existing aggregate. CIFS partner server is running. Tue Nov 13 16:11:57 AKST [akfiler13 (takeover): cf.rsrc.transitTime:notice]: Top Takeover transit times syslog=30370, wafl=1398 {pvvols_to_done=939, prvol_to_done=261, paggrs_to_done=197, part_vols_mnt_end=102, prvol_mnt_end=3}, wafl_sync=1324, registry_postrc_phase1=508, wafl_restart=372 {vdisk=210, restarters=162}, rc=267 {always_do_just_after_etc_rc=57, hostname=45, ifconfig=44, raid_config_prerc=40, always_do_just_before_etc_rc=26, options=21, options=12, options=11, ems_postrc=1}, registry_prerc=222, Tue Nov 13 16:11:57 AKST [akfiler13 (takeover): cf.fm.takeoverComplete:warning]: Cluster monitor: takeover completed Tue Nov 13 16:11:57 AKST [akfiler13 (takeover): cf.fm.takeoverDuration:warning]: Cluster monitor: takeover duration time is 35 seconds akfiler13(takeover)> Tue Nov 13 16:12:01 AKST [akfiler13 (takeover): monitor.globalStatus.critical:CRITICAL]: This node has taken over akfiler12. Tue Nov 13 16:12:07 AKST [akfiler12/akfiler13: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the partner server. Tue Nov 13 16:12:15 AKST [akfiler13 (takeover): asup.smtp.fail:warning]: Autosupport mail to "" failed (Protocol error) for message: CLUSTER TAKEOVER COMPLETE MANUAL Tue Nov 13 16:12:15 AKST [akfiler13 (takeover): asup.smtp.sent:notice]: Cluster Notification mail sent: Cluster Alert from akfiler13 Tue Nov 13 16:12:32 AKST [akfiler12/akfiler13: asup.smtp.sent:notice]: Cluster Notification mail sent: Cluster Notification from akfiler12 (REBOOT (CLUSTER TAKEOVER)) WARNING Tue Nov 13 16:12:34 AKST [akfiler13 (takeover): cf.fsm.releasingReservations:info]: Cluster monitor: Releasing disk reservations in preparation for giveback Tue Nov 13 16:12:34 AKST [akfiler13 (takeover): cf.fm.diskRelease:info]: Cluster
  • 2.
    monitor: released diskreservations. Tue Nov 13 16:12:34 AKST [akfiler13 (takeover): asup.smtp.sent:notice]: Cluster Notification mail sent: Cluster Notification from akfiler13 (CLUSTER TAKEOVER COMPLETE MANUAL) INFO Tue Nov 13 16:12:34 AKST [akfiler13 (takeover): cf.nm.nicTransitionDown:warning]: Cluster Interconnect link 0 is DOWN Tue Nov 13 16:12:34 AKST [akfiler13 (takeover): cf.rv.notConnected:error]: Connection for cfo_rv failed Tue Nov 13 16:12:36 AKST [akfiler13 (takeover): cf.nm.nicTransitionUp:info]: Interconnect link 0 is UP akfiler13(takeover)> Tue Nov 13 16:13:03 AKST [akfiler13 (takeover): asup.smtp.sent:notice]: Cluster Notification mail sent: Cluster Notification from akfiler13 (PERFORMANCE SNAPSHOT) INFO akfiler13(takeover)> akfiler13(takeover)> akfiler13(takeover)> akfiler13(takeover)> Tue Nov 13 16:21:12 AKST [akfiler13 (takeover): cf.hwassist.missedKeepAlive:warning]: Cluster hw_assist: missed keep alive alert from partner(akfiler12). Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.0 (5000C500:4C629A33:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.1 (5000C500:4C5AB147:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.2 (5000C500:4C57270F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.3 (5000C500:4C57ADEB:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.4 (5000C500:4C5FF017:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.5 (5000C500:4C576C93:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.6 (5000C500:4C5101C7:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.7 (5000C500:4C67D387:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.8 (5000C500:4C5F261B:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.9 (5000C500:4C50E7D7:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.10 (5000C500:4C4E646F:00000000:00000000:00000000:00000000:00000000:00000000:0000000
  • 3.
    0:00000000) has recentlychanged or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.11 (5000C500:4C5E9B1F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.12 (5000C500:4C65FFB3:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.13 (5000CCA0:1FC2210C:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.14 (5000C500:4C548E33:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.15 (5000C500:4C698D4B:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.16 (5000C500:4C532023:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.17 (5000C500:4C4E02DF:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.18 (5000C500:4C5AB39F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.19 (5000C500:4C581EE3:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.20 (5000C500:4C549B67:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.21 (5000C500:4C514A57:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.22 (5000C500:4C5A8EA3:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3a.10.23 (5000C500:4C56B1BF:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.0 (5000C500:4C60EE5F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.1 (5000C500:4C50E917:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.2 (5000C500:4C5485FF:00000000:00000000:00000000:00000000:00000000:00000000:0000000
  • 4.
    0:00000000) has recentlychanged or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.3 (5000C500:4C50F503:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.4 (5000C500:4C54A897:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.5 (5000C500:4C5B160F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.6 (5000C500:4C4F0D5F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.7 (5000C500:4C5823CB:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.8 (5000C500:4C359B67:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.9 (5000C500:4C5B3D63:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.10 (5000C500:4C4E7D2F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.11 (5000C500:4C50C46F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.12 (5000C500:4C482ED7:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.13 (5000C500:4C549F9B:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.14 (5000C500:4C4DE207:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.15 (5000C500:4C51C327:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.16 (5000C500:4C5E1AEF:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.17 (5000C500:4C5EF477:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.18 (5000C500:4C56B103:00000000:00000000:00000000:00000000:00000000:00000000:0000000
  • 5.
    0:00000000) has recentlychanged or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.19 (5000C500:4C51157F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.20 (5000C500:4C53F92F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.21 (5000C500:4C542753:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.22 (5000C500:4C40C7C3:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:21:59 AKST [akfiler13 (takeover): cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 3d.11.23 (5000C500:4C5E8DEF:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) has recently changed or the node (akfiler12) is missing the disk. Tue Nov 13 16:22:14 AKST [akfiler13 (takeover): asup.smtp.fail:warning]: Autosupport mail to "" failed (Protocol error) for message: CLUSTER ERROR: DISK/SHELF COUNT MISMATCH Tue Nov 13 16:22:14 AKST [akfiler13 (takeover): asup.smtp.sent:notice]: Cluster Notification mail sent: Cluster Alert from akfiler13 Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.0 (5000C500:4C629A33:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.1 (5000C500:4C5AB147:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.2 (5000C500:4C57270F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.3 (5000C500:4C57ADEB:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.4 (5000C500:4C5FF017:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.5 (5000C500:4C576C93:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.6 (5000C500:4C5101C7:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.7
  • 6.
    (5000C500:4C67D387:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in itsinventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.8 (5000C500:4C5F261B:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.9 (5000C500:4C50E7D7:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.10 (5000C500:4C4E646F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.11 (5000C500:4C5E9B1F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.12 (5000C500:4C65FFB3:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.13 (5000CCA0:1FC2210C:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.14 (5000C500:4C548E33:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.15 (5000C500:4C698D4B:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.16 (5000C500:4C532023:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.17 (5000C500:4C4E02DF:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.18 (5000C500:4C5AB39F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.19 (5000C500:4C581EE3:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk
  • 7.
    3a.10.20 (5000C500:4C549B67:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in itsinventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.21 (5000C500:4C514A57:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.22 (5000C500:4C5A8EA3:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3a.10.23 (5000C500:4C56B1BF:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.0 (5000C500:4C60EE5F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.1 (5000C500:4C50E917:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.2 (5000C500:4C5485FF:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.3 (5000C500:4C50F503:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.4 (5000C500:4C54A897:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.5 (5000C500:4C5B160F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.6 (5000C500:4C4F0D5F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.7 (5000C500:4C5823CB:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.8 (5000C500:4C359B67:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover):
  • 8.
    cf.disk.inventory.mismatchOK:info]: The node(akfiler12) included the disk 3d.11.9 (5000C500:4C5B3D63:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.10 (5000C500:4C4E7D2F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.11 (5000C500:4C50C46F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.12 (5000C500:4C482ED7:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.13 (5000C500:4C549F9B:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.14 (5000C500:4C4DE207:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.15 (5000C500:4C51C327:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.16 (5000C500:4C5E1AEF:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.17 (5000C500:4C5EF477:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.18 (5000C500:4C56B103:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.19 (5000C500:4C51157F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.20 (5000C500:4C53F92F:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.21 (5000C500:4C542753:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory.
  • 9.
    Tue Nov 1316:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.22 (5000C500:4C40C7C3:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:20 AKST [akfiler13 (takeover): cf.disk.inventory.mismatchOK:info]: The node (akfiler12) included the disk 3d.11.23 (5000C500:4C5E8DEF:00000000:00000000:00000000:00000000:00000000:00000000:0000000 0:00000000) in its inventory. Tue Nov 13 16:22:33 AKST [akfiler13 (takeover): asup.smtp.sent:notice]: Cluster Notification mail sent: Cluster Notification from akfiler13 (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH) ERROR Tue Nov 13 16:24:00 AKST [akfiler13 (takeover): cf.nm.nicTransitionDown:warning]: Cluster Interconnect link 0 is DOWN Tue Nov 13 16:24:00 AKST [akfiler13 (takeover): cf.rv.notConnected:error]: Connection for cfo_rv failed Tue Nov 13 16:25:00 AKST [akfiler13 (takeover): cf.nm.nicTransitionUp:info]: Interconnect link 0 is UP Tue Nov 13 16:25:12 AKST [akfiler13 (takeover): cf.hwassist.recvKeepAlive:info]: Cluster hw_assist: Received hw_assist KeepAlive alert from partner(akfiler12). Tue Nov 13 16:25:31 AKST [akfiler13 (takeover): cf.nm.nicReset:warning]: Initiating soft reset on Cluster Interconnect card 0 due to rendezvous connection timeout Tue Nov 13 16:25:33 AKST [akfiler13 (takeover): cf.nm.nicReset:warning]: Initiating soft reset on Cluster Interconnect card 0 due to rendezvous jammed Tue Nov 13 16:26:03 AKST last message repeated 10 times Tue Nov 13 16:26:06 AKST [akfiler13 (takeover): cf.nm.nicReset:warning]: Initiating soft reset on Cluster Interconnect card 0 due to rendezvous jammed akfiler13(takeover)> akfiler13(takeover)> akfiler13(takeover)> storage show disk -p PRIMARY PORT SECONDARY PORT SHELF BAY -------- ---- --------- ---- --------- 0a.16 B 0d.16 A 1 0 0a.17 B 0d.17 A 1 1 0a.18 B 0d.18 A 1 2 0a.19 B 0d.19 A 1 3 0a.20 B 0d.20 A 1 4 0a.21 B 0d.21 A 1 5 0a.22 B 0d.22 A 1 6 0a.23 B 0d.23 A 1 7 0a.24 B 0d.24 A 1 8 0a.25 B 0d.25 A 1 9 0a.26 B 0d.26 A 1 10 0a.27 B 0d.27 A 1 11 0a.28 B 0d.28 A 1 12 0a.29 B 0d.29 A 1 13 0a.32 B 0d.32 A 2 0 0a.33 B 0d.33 A 2 1 0a.34 B 0d.34 A 2 2 0a.35 B 0d.35 A 2 3 0a.36 B 0d.36 A 2 4 0a.37 B 0d.37 A 2 5 0a.38 B 0d.38 A 2 6 0a.39 B 0d.39 A 2 7 0a.40 B 0d.40 A 2 8 0a.41 B 0d.41 A 2 9 0a.42 B 0d.42 A 2 10 0a.43 B 0d.43 A 2 11 0a.44 B 0d.44 A 2 12 0a.45 B 0d.45 A 2 13 0b.16 B 0c.16 A 1 0
  • 10.
    0b.17 B 0c.17 A 1 1 0b.18 B 0c.18 A 1 2 0b.19 B 0c.19 A 1 3 0b.20 B 0c.20 A 1 4 0b.21 B 0c.21 A 1 5 0b.22 B 0c.22 A 1 6 0b.23 B 0c.23 A 1 7 0b.24 B 0c.24 A 1 8 0b.25 B 0c.25 A 1 9 0b.26 B 0c.26 A 1 10 0b.27 B 0c.27 A 1 11 0b.28 B 0c.28 A 1 12 0b.29 B 0c.29 A 1 13 0b.32 B 0c.32 A 2 0 0b.33 B 0c.33 A 2 1 0b.34 B 0c.34 A 2 2 0b.35 B 0c.35 A 2 3 0b.36 B 0c.36 A 2 4 0b.37 B 0c.37 A 2 5 0b.38 B 0c.38 A 2 6 0b.39 B 0c.39 A 2 7 0b.40 B 0c.40 A 2 8 0b.41 B 0c.41 A 2 9 0b.42 B 0c.42 A 2 10 0b.43 B 0c.43 A 2 11 0b.44 B 0c.44 A 2 12 0b.45 B 0c.45 A 2 13 3a.10.0 B 3d.10.0 A 10 0 3a.10.1 B 3d.10.1 A 10 1 3a.10.2 B 3d.10.2 A 10 2 3a.10.3 B 3d.10.3 A 10 3 3a.10.4 B 3d.10.4 A 10 4 3a.10.5 B 3d.10.5 A 10 5 3a.10.6 B 3d.10.6 A 10 6 3a.10.7 B 3d.10.7 A 10 7 3a.10.8 B 3d.10.8 A 10 8 3a.10.9 B 3d.10.9 A 10 9 3a.10.10 B 3d.10.10 A 10 10 3a.10.11 B 3d.10.11 A 10 11 3a.10.12 B 3d.10.12 A 10 12 3a.10.13 B 3d.10.13 A 10 13 3a.10.14 B 3d.10.14 A 10 14 3a.10.15 B 3d.10.15 A 10 15 3a.10.16 B 3d.10.16 A 10 16 3a.10.17 B 3d.10.17 A 10 17 3a.10.18 B 3d.10.18 A 10 18 3a.10.19 B 3d.10.19 A 10 19 3a.10.20 B 3d.10.20 A 10 20 3a.10.21 B 3d.10.21 A 10 21 3a.10.22 B 3d.10.22 A 10 22 3a.10.23 B 3d.10.23 A 10 23 3d.11.0 A 3a.11.0 B 11 0 3d.11.1 A 3a.11.1 B 11 1 3d.11.2 A 3a.11.2 B 11 2 3d.11.3 A 3a.11.3 B 11 3 3d.11.4 A 3a.11.4 B 11 4 3d.11.5 A 3a.11.5 B 11 5 3d.11.6 A 3a.11.6 B 11 6 3d.11.7 A 3a.11.7 B 11 7 3d.11.8 A 3a.11.8 B 11 8 3d.11.9 A 3a.11.9 B 11 9 3d.11.10 A 3a.11.10 B 11 10 3d.11.11 A 3a.11.11 B 11 11 3d.11.12 A 3a.11.12 B 11 12
  • 11.
    3d.11.13 A 3a.11.13 B 11 13 3d.11.14 A 3a.11.14 B 11 14 3d.11.15 A 3a.11.15 B 11 15 3d.11.16 A 3a.11.16 B 11 16 3d.11.17 A 3a.11.17 B 11 17 3d.11.18 A 3a.11.18 B 11 18 3d.11.19 A 3a.11.19 B 11 19 3d.11.20 A 3a.11.20 B 11 20 3d.11.21 A 3a.11.21 B 11 21 3d.11.22 A 3a.11.22 B 11 22 3d.11.23 A 3a.11.23 B 11 23 akfiler13(takeover)> cf givebacklogin as: p375290 p375290@akfiler13.ak.providence.org's password: akfiler13> Tue Nov 13 16:30:49 AKST [akfiler13: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of akfiler12 disabled (partner booting) Tue Nov 13 16:30:49 AKST [akfiler13: cf.fsm.takeoverByPartnerDisabled:notice]: Cluster monitor: takeover of akfiler13 by akfiler12 disabled (unsynchronized log) akfiler13> Tue Nov 13 16:31:03 AKST [akfiler13: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of akfiler12 disabled (interconnect error) Tue Nov 13 16:31:03 AKST [akfiler13: cf.fsm.takeoverByPartnerDisabled:notice]: Cluster monitor: takeover of akfiler13 by akfiler12 disabled (interconnect error) Tue Nov 13 16:31:04 AKST [akfiler13: cf.partner.short_uptime:warning]: Partner up for 13 seconds only Tue Nov 13 16:31:05 AKST [akfiler13: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of akfiler12 disabled (unsynchronized log) Tue Nov 13 16:31:05 AKST [akfiler13: cf.fsm.takeoverByPartnerDisabled:notice]: Cluster monitor: takeover of akfiler13 by akfiler12 disabled (unsynchronized log) Tue Nov 13 16:31:07 AKST [akfiler13: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of akfiler12 enabled Tue Nov 13 16:31:10 AKST [akfiler13: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of akfiler13 by akfiler12 enabled Tue Nov 13 16:32:00 AKST [akfiler13: monitor.globalStatus.ok:info]: The system's global status is normal. akfiler13> aggr stastus aggr: No such command "stastus". The following commands are available; for more information type "aggr help <command>" add mirror rename split copy offline restrict status create online scrub undestroy destroy options show_space verify media_scrub akfiler13> aggr stastus# # # ## # # ## # # ## # # ## # # #atus -r Aggregate aggr1 (online, raid_dp) (block checksums) Plex /aggr1/plex0 (online, normal, active) RAID group /aggr1/plex0/rg0 (normal) RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- dparity 3d.11.9 3d 11 9 SA:A - SAS 15000 560000/1146880000 560208/1147307688 parity 3d.11.8 3d 11 8 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.7 3d 11 7 SA:A - SAS 15000
  • 12.
    560000/1146880000 560208/1147307688 data 3d.11.6 3d 11 6 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.5 3d 11 5 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.4 3d 11 4 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.3 3d 11 3 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.23 3d 11 23 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.22 3d 11 22 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.21 3d 11 21 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.20 3d 11 20 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.2 3d 11 2 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.19 3d 11 19 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.18 3d 11 18 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.17 3d 11 17 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.16 3d 11 16 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.15 3d 11 15 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.14 3d 11 14 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.13 3d 11 13 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.12 3d 11 12 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.11 3d 11 11 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.10 3d 11 10 SA:A - SAS 15000 560000/1146880000 560208/1147307688 data 3d.11.1 3d 11 1 SA:A - SAS 15000 560000/1146880000 560208/1147307688 Aggregate aggr0 (online, raid_dp) (block checksums) Plex /aggr0/plex0 (online, normal, active) RAID group /aggr0/plex0/rg0 (normal) RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- dparity 0b.16 0b 1 0 FC:B - FCAL 15000 272000/557056000 274845/562884296 parity 0c.17 0c 1 1 FC:A - FCAL 15000 272000/557056000 274845/562884296 data 0c.18 0c 1 2 FC:A - FCAL 15000 272000/557056000 274845/562884296 data 0b.19 0b 1 3 FC:B - FCAL 15000 272000/557056000 274845/562884296 data 0c.20 0c 1 4 FC:A - FCAL 15000 272000/557056000 274845/562884296 data 0b.21 0b 1 5 FC:B - FCAL 15000 272000/557056000 274845/562884296 data 0c.22 0c 1 6 FC:A - FCAL 15000 272000/557056000 274845/562884296
  • 13.
    data 0c.23 0c 1 7 FC:A - FCAL 15000 272000/557056000 274845/562884296 data 0b.24 0b 1 8 FC:B - FCAL 15000 272000/557056000 274845/562884296 data 0b.25 0b 1 9 FC:B - FCAL 15000 272000/557056000 274845/562884296 data 0c.26 0c 1 10 FC:A - FCAL 15000 272000/557056000 274845/562884296 data 0b.27 0b 1 11 FC:B - FCAL 15000 272000/557056000 274845/562884296 data 0b.28 0b 1 12 FC:B - FCAL 15000 272000/557056000 274845/562884296 data 0b.32 0b 2 0 FC:B - FCAL 15000 272000/557056000 274845/562884296 data 0b.29 0b 1 13 FC:B - FCAL 15000 272000/557056000 274845/562884296 data 0b.33 0b 2 1 FC:B - FCAL 15000 272000/557056000 274845/562884296 RAID group /aggr0/plex0/rg1 (normal) RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- dparity 0c.34 0c 2 2 FC:A - FCAL 15000 272000/557056000 274845/562884296 parity 0b.35 0b 2 3 FC:B - FCAL 15000 272000/557056000 274845/562884296 data 0c.36 0c 2 4 FC:A - FCAL 15000 272000/557056000 274845/562884296 data 0b.37 0b 2 5 FC:B - FCAL 15000 272000/557056000 274845/562884296 data 0b.38 0b 2 6 FC:B - FCAL 15000 272000/557056000 274845/562884296 data 0b.39 0b 2 7 FC:B - FCAL 15000 272000/557056000 274845/562884296 data 0c.40 0c 2 8 FC:A - FCAL 15000 272000/557056000 274845/562884296 data 0c.41 0c 2 9 FC:A - FCAL 15000 272000/557056000 274845/562884296 data 0c.42 0c 2 10 FC:A - FCAL 15000 272000/557056000 274845/562884296 data 0c.43 0c 2 11 FC:A - FCAL 15000 272000/557056000 274845/562884296 Spare disks RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- Spare disks for block or zoned checksum traditional volumes or aggregates spare 0b.45 0b 2 13 FC:B - FCAL 15000 272000/557056000 274845/562884296 spare 0c.44 0c 2 12 FC:A - FCAL 15000 272000/557056000 274845/562884296 spare 3d.11.0 3d 11 0 SA:A - SAS 15000 560000/1146880000 560208/1147307688 Partner disks RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks)
  • 14.
    Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- partner 3a.10.13 3a 10 13 SA:B - SAS 15000 0/0 560879/1148681096 partner 3a.10.2 3a 10 2 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.22 3a 10 22 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.23 3a 10 23 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.21 3a 10 21 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.8 3a 10 8 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.20 3a 10 20 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.9 3a 10 9 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.7 3a 10 7 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.6 3a 10 6 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.5 3a 10 5 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.4 3a 10 4 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.3 3a 10 3 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.17 3a 10 17 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.16 3a 10 16 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.15 3a 10 15 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.14 3a 10 14 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.11 3a 10 11 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.10 3a 10 10 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.1 3a 10 1 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.19 3a 10 19 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.12 3a 10 12 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.18 3a 10 18 SA:B - SAS 15000 0/0 560208/1147307688 partner 3a.10.0 3a 10 0 SA:B - SAS 15000 0/0 560208/1147307688 partner 0d.16 0d 1 0 FC:A - FCAL 15000 0/0 274845/562884296 partner 0d.23 0d 1 7 FC:A - FCAL 15000 0/0 274845/562884296 partner 0a.27 0a 1 11 FC:B - FCAL 15000 0/0 274845/562884296 partner 0d.40 0d 2 8 FC:A - FCAL 15000 0/0 274845/562884296 partner 0d.25 0d 1 9 FC:A - FCAL 15000 0/0 274845/562884296 partner 0d.43 0d 2 11 FC:A - FCAL 15000 0/0 274845/562884296 partner 0a.22 0a 1 6 FC:B - FCAL 15000 0/0
  • 15.
    274845/562884296 partner 0a.17 0a 1 1 FC:B - FCAL 15000 0/0 274845/562884296 partner 0d.44 0d 2 12 FC:A - FCAL 15000 0/0 274845/562884296 partner 0a.32 0a 2 0 FC:B - FCAL 15000 0/0 274845/562884296 partner 0d.35 0d 2 3 FC:A - FCAL 15000 0/0 274845/562884296 partner 0d.45 0d 2 13 FC:A - FCAL 15000 0/0 274845/562884296 partner 0d.41 0d 2 9 FC:A - FCAL 15000 0/0 274845/562884296 partner 0d.19 0d 1 3 FC:A - FCAL 15000 0/0 274845/562884296 partner 0d.42 0d 2 10 FC:A - FCAL 15000 0/0 274845/562884296 partner 0a.36 0a 2 4 FC:B - FCAL 15000 0/0 274845/562884296 partner 0d.26 0d 1 10 FC:A - FCAL 15000 0/0 274845/562884296 partner 0a.21 0a 1 5 FC:B - FCAL 15000 0/0 274845/562884296 partner 0a.39 0a 2 7 FC:B - FCAL 15000 0/0 274845/562884296 partner 0a.34 0a 2 2 FC:B - FCAL 15000 0/0 274845/562884296 partner 0d.29 0d 1 13 FC:A - FCAL 15000 0/0 274845/562884296 partner 0a.20 0a 1 4 FC:B - FCAL 15000 0/0 274845/562884296 partner 0a.18 0a 1 2 FC:B - FCAL 15000 0/0 274845/562884296 partner 0a.37 0a 2 5 FC:B - FCAL 15000 0/0 274845/562884296 partner 0d.28 0d 1 12 FC:A - FCAL 15000 0/0 274845/562884296 partner 0a.24 0a 1 8 FC:B - FCAL 15000 0/0 274845/562884296 partner 0a.38 0a 2 6 FC:B - FCAL 15000 0/0 274845/562884296 partner 0a.33 0a 2 1 FC:B - FCAL 15000 0/0 274845/562884296 akfiler13> Tue Nov 13 16:35:48 AKST [akfiler13: asup.smtp.fail:warning]: Autosupport mail to "" failed (Protocol error) for message: CLUSTER GIVEBACK COMPLETE Tue Nov 13 16:35:49 AKST [akfiler13: asup.smtp.sent:notice]: Cluster Notification mail sent: Cluster Alert from akfiler13 Tue Nov 13 16:36:07 AKST [akfiler13: asup.smtp.sent:notice]: Cluster Notification mail sent: Cluster Notification from akfiler13 (CLUSTER GIVEBACK COMPLETE) INFO akfiler13> akfiler13> Tue Nov 13 17:00:00 AKST [akfiler13: kern.timed.adjust:info]: server 'time.ak.providence.org' reports the appliance date is slow by 0.002 seconds. Adjusting date. Tue Nov 13 17:00:00 AKST [akfiler13: kern.uptime.filer:info]: 5:00pm up 201 days, 22:09 27134312584 NFS ops, 3698206 CIFS ops, 59 HTTP ops, 0 FCP ops, 0 iSCSI ops akfiler13> akfiler13>