Hi,
I am facing high CPU utilization issue with multiple NetApp filers running on 7- Mode, i tried to identify the cause of high CPU utilization but its not getting clear to me.
Below are the sample out put of sysstat -x 1, sysstat -m 1 and sysstat -M 1, and statit capture and try to identify what process is causing high CPU utilization.
sysstat -x 1
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time ty util in out in out
96% 0 0 0 2569 55126 34452 167972 152950 0 0 7s 92% 99% Bn 96% 0 0 2569 0 0 52513 32921
95% 0 0 0 4858 110991 67370 219428 174513 0 0 9s 88% 100% :s 99% 6 0 4852 0 0 105656 64185
79% 0 0 0 3285 47090 73832 190141 93772 0 0 4s 86% 100% :f 75% 0 0 3285 0 0 44455 72380
91% 0 0 0 1840 24627 29797 210990 293491 0 0 4s 94% 99% Bs 83% 0 0 1840 0 0 23294 29005
99% 0 0 0 3424 147063 17753 159145 260516 0 0 6s 91% 98% Hn 93% 0 0 3424 0 0 140604 14158
99% 0 0 0 3610 95470 17616 167211 188118 0 0 6s 92% 100% :s 89% 0 0 3610 0 0 91313 15192
89% 0 0 0 4446 98371 58813 177714 118398 0 0 8s 92% 100% :v 70% 5 0 4441 0 0 93805 56098
98% 0 0 0 2045 67803 16601 163976 367372 0 0 8s 93% 99% Bs 91% 0 0 2045 0 0 65370 14781
96% 0 0 0 4858 131156 21542 132989 133849 0 0 3s 89% 98% Hn 79% 0 0 4858 0 0 124643 18160
99% 0 0 0 4121 90312 32774 190865 156729 0 0 5s 92% 100% :n 100% 0 0 4121 0 0 86247 30353
83% 0 0 0 3821 93764 20924 128495 137691 0 0 5s 89% 100% :s 84% 0 0 3821 0 0 89439 18351
75% 0 0 0 1755 27946 12668 164329 208325 0 0 7s 81% 99% Bn 62% 5 0 1750 0 0 26564 11839
97% 0 0 0 4205 119386 7600 144720 327178 0 0 7s 93% 100% :f 85% 0 0 4205 0 0 114179 4545
99% 0 0 0 3279 83809 4598 172910 225482 0 0 7s 90% 99% Hs 99% 0 0 3279 0 0 80071 2515
97% 0 0 0 7285 100670 9977 115014 160170 0 0 7s 88% 100% :f 74% 0 0 7285 0 0 96324 7142
95% 0 0 0 2496 83731 4836 143294 155091 0 0 7s 87% 99% Bn 100% 39 0 2457 0 0 80372 2835
88% 0 0 0 2489 102816 7312 106326 146807 0 0 12s 89% 100% :s 95% 318 0 2171 0 0 98129 4764
53% 0 0 0 1652 16819 6639 48480 46076 0 0 12s 86% 100% :s 50% 0 0 1652 0 0 16013 6114
89% 0 0 0 1976 56528 4249 125429 220625 0 0 12s 88% 99% Bn 77% 0 0 1976 0 0 54126 2844
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP
sysstat -m 1
sysstat -m -c 50
ANY AVG CPU0 CPU1 CPU2 CPU3
99% 92% 92% 92% 94% 88%
100% 90% 89% 88% 92% 91%
93% 77% 75% 75% 81% 76%
98% 87% 86% 86% 89% 86%
98% 86% 85% 85% 89% 86%
98% 92% 91% 91% 95% 89%
99% 86% 82% 88% 87% 86%
sysstat -M 1 in diag mode
ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Raid_Ex Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt SSAN_Ex Intr Host Ops/s CP
100% 100% 99% 89% 97% 98% 98% 99% 92% 31% 0% 0% 25% 0% 64% 12% 10% 158%( 88%) 14% 30% 0% 39% 0% 3% 1% 4030 100%
100% 96% 90% 77% 91% 87% 90% 92% 95% 21% 0% 0% 25% 0% 59% 8% 12% 167%( 88%) 10% 20% 0% 36% 0% 3% 1% 2697 99%
100% 99% 96% 84% 95% 96% 96% 96% 91% 40% 0% 0% 18% 0% 52% 11% 14% 150%( 83%) 11% 37% 0% 40% 0% 3% 2% 3636 100%
100% 100% 97% 84% 95% 97% 98% 97% 89% 16% 0% 0% 22% 0% 68% 6% 9% 138%( 86%) 22% 30% 0% 53% 0% 3% 15% 1655 99%
100% 95% 83% 59% 84% 85% 83% 85% 83% 22% 0% 0% 18% 0% 39% 12% 15% 133%( 75%) 6% 69% 0% 18% 0% 3% 1% 3713 99%
100% 97% 89% 74% 90% 90% 90% 91% 90% 14% 0% 0% 25% 1% 54% 10% 10% 163%( 88%) 12% 40% 0% 29% 0% 3% 1% 2291 100%
100% 94% 85% 64% 86% 84% 84% 85% 91% 26% 0% 0% 19% 0% 36% 18% 21% 132%( 78%) 5% 70% 0% 14% 0% 3% 1% 4512 100%
100% 97% 90% 70% 89% 90% 90% 90% 86% 16% 0% 0% 23% 0% 56% 13% 13% 130%( 80%) 13% 55% 0% 33% 0% 3% 2% 3064 99%
100% 97% 85% 59% 85% 89% 90% 85% 77% 24% 0% 0% 15% 0% 38% 17% 21% 92%( 61%) 7% 95% 0% 20% 0% 3% 9% 4432 100%
100% 98% 93% 76% 92% 94% 93% 92% 88% 30% 0% 0% 19% 0% 45% 15% 18% 130%( 76%) 12% 63% 0% 28% 0% 3% 2% 4263 84%
100% 97% 91% 73% 90% 92% 90% 92% 89% 24% 0% 0% 21% 0% 51% 11% 14% 141%( 80%) 13% 50% 0% 31% 0% 3% 1% 3591 99%
100% 98% 94% 79% 93% 93% 93% 94% 90% 30% 0% 0% 23% 0% 51% 13% 15% 143%( 82%) 8% 50% 0% 32% 0% 3% 1% 4161 100%
100% 99% 97% 86% 96% 98% 97% 97% 91% 33% 0% 0% 24% 1% 56% 14% 13% 140%( 83%) 16% 43% 0% 36% 0% 3% 3% 4441 99%
100% 98% 95% 86% 95% 94% 95% 95% 95% 31% 0% 0% 26% 0% 56% 13% 16% 158%( 83%) 13% 29% 0% 31% 0% 3% 1% 4880 99%
100% 98% 97% 89% 96% 96% 96% 97% 96% 30% 0% 0% 29% 0% 59% 12% 15% 170%( 85%) 12% 23% 0% 28% 0% 3% 1% 4341 99%
Statit output -
ID: 2014386083 Memory: 6114 MB
NetApp Release 8.2P6 7-Mode: Fri Jan 17 22:23:50 PST 2014
<1O>
Start time: Fri Dec 23 07:24:07 GMT 2016
CPU Statistics
19.444790 time (seconds) 100 %
69.620104 system time 358 %
0.653313 rupt time 3 % (491410 rupts x 1 usec/rupt)
68.966791 non-rupt system time 355 %
8.159056 idle time 42 %
19.378947 time in CP 100 % 100 %
0.651628 rupt time in CP 3 % (490122 rupts x 1 usec/rupt)
Multiprocessor Statistics (per second)
cpu0 cpu1 cpu2 cpu3 total
sk switches 11072.32 13190.17 15529.20 5615.69 45407.38
hard switches 11010.66 13146.04 15502.40 5323.02 44982.13
domain switches 327.85 286.61 265.37 901.53 1781.35
CP rupts 4931.66 9591.57 5725.90 4956.70 25205.83
nonCP rupts 12.65 24.27 16.66 12.65 66.24
IPI rupts 0.00 0.00 0.00 0.00 0.00
grab kahuna 0.00 0.00 0.00 0.00 0.00
grab kahuna usec 0.00 0.00 0.00 0.00 0.00
suspend domain 0.00 0.00 0.00 0.00 0.00
suspend domain usec 0.00 0.00 0.00 0.00 0.00
CP rupt usec 11355.18 10717.83 7170.97 4267.62 33511.70
nonCP rupt usec 33.22 24.99 19.75 8.59 86.66
idle 121405.43 105709.19 77763.35 114723.17 419601.19
kahuna 0.00 0.00 0.00 93915.49 93915.49
storage 0.00 0.00 276214.71 0.00 276214.71
exempt 145447.29 132218.40 86338.76 60.79 364065.34
raid 1716.35 1295.15 722.87 0.00 3734.42
raid_exempt 237375.20 213198.86 180858.42 0.00 631432.48
target 17350.87 96006.13 12860.26 0.00 126217.31
dnscache 0.00 0.00 0.00 0.00 0.00
cifs 92.83 75.19 94.73 0.00 262.80
wafl_exempt 324797.44 287419.00 250991.14 787024.18 1650231.76
wafl_xcleaner 46336.63 52061.25 31884.79 0.00 130282.66
sm_exempt 28.90 37.54 27.21 0.00 93.70
cluster 0.00 0.00 0.00 0.00 0.00
protocol 443.25 450.76 298.38 0.00 1192.45
nwk_exclusive 3788.52 2920.68 2340.42 0.00 9049.72
nwk_exempt 81534.08 89270.29 64611.45 0.00 235415.91
nwk_legacy 1975.34 1872.12 1464.87 0.00 5312.37
hostOS 6319.12 6722.21 6337.58 0.00 19378.97
ssan_exempt 0.00 0.00 0.00 0.00 0.00
FreeBSD CPU state Statistics (per second)
user 0.10 0.05 0.00 0.00 0.15
nice 0.00 0.00 0.00 0.00 0.00
sys 115.97 118.03 122.29 118.54 474.83
intr 0.05 0.10 0.10 0.00 0.26
idle 16.92 14.86 10.59 14.45 56.83
nonrt-pf-cnt 0.00 0.00 0.00 0.00 0.00
nonrt-pf-usec 0.00 0.00 0.00 0.00 0.00
rt-pf-cnt 0.00 0.00 0.00 0.00 0.00
rt-pf-usec 0.00 0.00 0.00 0.00 0.00
kern-pf-cnt 0.00 0.00 0.00 0.00 0.00
kern-pf-usec 0.00 0.00 0.00 0.00 0.00
19.251686 seconds with one or more CPUs active ( 99%)
18.366020 seconds with 2 or more CPUs active ( 94%)
17.215831 seconds with 3 or more CPUs active ( 89%)
0.885665 seconds with one CPU active ( 5%)
1.150189 seconds with 2 CPUs active ( 6%)
2.500874 seconds with 3 CPUs active ( 13%)
14.714956 seconds with all CPUs active ( 76%)
Domain Utilization of Shared Domains (per second)
0.00 idle 862962.57 kahuna
0.00 storage 0.00 exempt
521829.55 raid 0.00 raid_exempt
0.00 target 0.00 dnscache
0.00 cifs 0.00 wafl_exempt
0.00 wafl_xcleaner 0.00 sm_exempt
0.00 cluster 0.00 protocol
225824.76 nwk_exclusive 0.00 nwk_exempt
0.00 nwk_legacy 0.00 hostOS
0.00 ssan_exempt
switch domain to domain (per second)
0.00 idle 571.36 kahuna
74.88 storage 86.40 exempt
19.23 raid 0.00 raid_exempt
0.26 target 0.00 dnscache
1.03 cifs 373.72 wafl_exempt
0.00 wafl_xcleaner 0.00 sm_exempt
0.00 cluster 0.31 protocol
12.50 nwk_exclusive 296.58 nwk_exempt
345.08 nwk_legacy 0.00 hostOS
0.00 ssan_exempt
Exempt Domain Suspension Stats (per second)
Miscellaneous Statistics (per second)
44982.13 hard context switches 0.00 NFS operations
0.00 CIFS operations 0.00 HTTP operations
81696.95 network KB received 35315.53 network KB transmitted
187898.56 disk KB read 202124.07 disk KB written
78651.25 NVRAM KB written 27675.07 nolog KB written
8542.08 WAFL bufs given to clients 0.00 checksum cache hits ( 0%)
0.00 no checksum - partial buffer 0.00 FCP operations
3913.54 iSCSI operations
WAFL Statistics (per second)
165.60 name cache hits ( 82%) 36.10 name cache misses ( 18%)
306702.62 buf hash hits ( 67%) 149447.44 buf hash misses ( 33%)
10920.35 inode cache hits ( 100%) 0.00 inode cache misses ( 0%)
63701.85 buf cache hits ( 89%) 7497.12 buf cache misses ( 11%)
4732.48 blocks read 16781.20 blocks read-ahead
4832.09 chains read-ahead 199.54 dummy reads
3626.68 blocks speculative read-ahead 31543.00 blocks written
153.77 stripes written 0.00 blocks page flipped
0.00 blocks over-written 0.00 wafl_timer generated CP
0.00 snapshot generated CP 0.00 wafl_avail_bufs generated CP
0.15 dirty_blk_cnt generated CP 0.10 full NV-log generated CP
0.15 back-to-back CP 0.00 flush generated CP
0.00 sync generated CP 0.00 deferred back-to-back CP
0.00 low mbufs generated CP 0.00 low datavecs generated CP
0.00 nvlog replay takeover time limit CP 34380.93 non-restart messages
2321.65 IOWAIT suspends 434.10 next nvlog nearly full msecs
804.48 dirty buffer susp msecs 0.00 nvlog full susp msecs
998538 buffers
RAID Statistics (per second)
13156.79 xors 0.00 long dispatches [0]
0.00 long consumed [0] 0.00 long consumed hipri [0]
0.00 long low priority [0] 0.00 long high priority [0]
99.36 long monitor tics [0] 0.00 long monitor clears [0]
0.00 long dispatches [1] 0.00 long consumed [1]
0.00 long consumed hipri [1] 0.00 long low priority [1]
99.36 long high priority [1] 99.36 long monitor tics [1]
0.00 long monitor clears [1] 18 max batch
164.83 blocked mode xor 129.39 timed mode xor
2.73 fast adjustments 0.57 slow adjustments
0 avg batch start 0 avg stripe/msec
0.00 checksum dispatches 0.00 checksum consumed
157.06 tetrises written 0.00 master tetrises
0.00 slave tetrises 8037.32 stripes written
5203.35 partial stripes 2833.97 full stripes
33879.87 blocks written 19607.51 blocks read
1676.70 1 blocks per stripe size 3 1791.89 2 blocks per stripe size 3
2184.44 3 blocks per stripe size 3 83.67 1 blocks per stripe size 13
76.78 2 blocks per stripe size 13 92.57 3 blocks per stripe size 13
83.88 4 blocks per stripe size 13 115.61 5 blocks per stripe size 13
122.09 6 blocks per stripe size 13 139.93 7 blocks per stripe size 13
159.89 8 blocks per stripe size 13 154.95 9 blocks per stripe size 13
170.28 10 blocks per stripe size 13 228.70 11 blocks per stripe size 13
286.30 12 blocks per stripe size 13 639.04 13 blocks per stripe size 13
2.57 1 blocks per stripe size 14 2.83 2 blocks per stripe size 14
1.44 3 blocks per stripe size 14 0.98 4 blocks per stripe size 14
0.51 5 blocks per stripe size 14 0.87 6 blocks per stripe size 14
0.87 7 blocks per stripe size 14 0.87 8 blocks per stripe size 14
0.62 9 blocks per stripe size 14 1.29 10 blocks per stripe size 14
1.70 11 blocks per stripe size 14 3.29 12 blocks per stripe size 14
2.26 13 blocks per stripe size 14 10.49 14 blocks per stripe size 14
Network Interface Statistics (per second)
iface side bytes packets multicasts errors collisions pkt drops
e0a recv 309.70 4.73 0.00 0.00 0.00
xmit 33.22 0.31 0.05 0.00 0.00
e0b recv 793.12 8.07 0.00 0.00 0.00
xmit 59.04 0.41 0.05 0.00 0.00
e1a recv 3309129.33 1102.76 0.15 0.00 0.00
xmit 2426425.59 1209.06 0.00 0.00 0.00
e1b recv 3223077.90 1099.57 0.15 0.00 0.00
xmit 2350960.64 1156.71 0.00 0.00 0.00
e2a recv 38777662.40 27336.11 1.18 0.00 0.00
xmit 15955724.49 14593.42 0.00 0.00 0.00
e2b recv 38345798.95 27027.55 1.90 0.00 0.00
xmit 15429151.25 14434.15 0.00 0.00 0.00
c0a recv 381.08 12.55 2.06 0.00 0.00
xmit 363.49 1.95 1.95 0.00 0.00
c0b recv 381.08 1.95 2.06 0.00 0.00
xmit 363.49 1.95 1.95 0.00 0.00
e0M recv 92.57 1.54 1.54 0.00 0.00
xmit 0.00 0.00 0.00 0.00 0.00
e0P recv 80.69 0.46 0.15 0.00 0.00
xmit 36.41 0.31 0.00 0.00 0.00
IFGRP_LAN recv 1116.96 12.91 11.73 0.00 0.00
xmit 87.22 0.67 0.10 0.00 0.00
Disk Statistics (per second)
ut% is the percent of time the disk was busy.
xfers is the number of data-transfer commands issued per second.
xfers = ureads + writes + cpreads + greads + gwrites
chain is the average number of 4K blocks per command.
usecs is the average disk round-trip time per 4K block.
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs
/Aggr1_SATA3T_FP/plex0/rg0:
5c.12.8L1 14 25.97 0.00 .... . 11.77 40.54 223 14.19 26.47 221 0.00 .... . 0.00 .... .
3b.11.8L1 15 26.12 0.00 .... . 12.08 39.55 264 14.04 26.97 238 0.00 .... . 0.00 .... .
5c.12.9L1 77 127.57 97.34 2.83 7430 16.76 20.27 863 13.47 6.51 2362 0.00 .... . 0.00 .... .
3b.11.9L1 76 125.57 102.74 2.86 7193 12.39 26.01 812 10.44 9.61 1479 0.00 .... . 0.00 .... .
5c.12.10L1 75 125.15 100.94 2.50 7907 12.49 25.68 1044 11.72 10.13 1292 0.00 .... . 0.00 .... .
3b.11.10L1 76 128.19 101.66 2.63 8569 14.81 22.44 1023 11.72 10.03 1405 0.00 .... . 0.00 .... .
5c.12.11L1 74 126.03 100.11 2.57 6939 13.78 23.25 853 12.13 9.55 1092 0.00 .... . 0.00 .... .
3b.11.11L1 77 128.81 102.99 2.77 7770 14.71 22.75 988 11.11 10.38 1186 0.00 .... . 0.00 .... .
5c.12.12L1 78 128.75 101.91 2.68 8168 14.96 20.99 954 11.88 9.93 1265 0.00 .... . 0.00 .... .
3b.11.12L1 77 132.76 108.60 2.58 7311 13.37 24.64 852 10.80 11.74 1351 0.00 .... . 0.00 .... .
5c.12.13L1 76 124.85 99.91 2.56 8780 13.06 27.61 954 11.88 9.10 1359 0.00 .... . 0.00 .... .
3b.11.13L1 73 126.90 100.11 2.73 7748 14.40 24.33 772 12.39 8.34 1318 0.00 .... . 0.00 .... .
5c.12.14L1 73 121.14 95.64 2.76 8399 13.93 24.37 962 11.57 8.08 1707 0.00 .... . 0.00 .... .
3b.11.14L1 72 120.99 94.97 2.71 6709 14.09 23.53 1017 11.93 8.22 1612 0.00 .... . 0.00 .... .
5c.12.15L1 75 129.88 102.79 2.61 8023 14.35 23.10 896 12.75 9.49 1457 0.00 .... . 0.00 .... .
/Aggr1_SATA3T_FP/plex0/rg1:
3b.11.15L1 13 21.90 0.00 .... . 9.15 49.29 224 12.75 26.54 231 0.00 .... . 0.00 .... .
5c.12.16L1 13 22.01 0.00 .... . 9.26 48.76 210 12.75 26.54 224 0.00 .... . 0.00 .... .
3b.11.16L1 72 120.58 97.85 2.60 7245 12.44 26.86 760 10.28 10.48 1121 0.00 .... . 0.00 .... .
5c.12.17L1 74 118.93 96.87 2.86 7616 12.44 26.00 845 9.62 10.59 1208 0.00 .... . 0.00 .... .
3b.11.17L1 72 118.68 96.82 2.74 7185 11.72 27.21 763 10.13 10.07 1429 0.00 .... . 0.00 .... .
5c.12.18L1 74 121.66 98.88 2.69 7428 12.80 24.19 831 9.98 10.84 1193 0.00 .... . 0.00 .... .
3b.11.18L1 72 116.62 96.36 2.97 5842 11.62 27.73 736 8.64 11.32 1120 0.00 .... . 0.00 .... .
5c.12.19L1 70 113.89 92.91 2.71 7781 12.08 26.20 790 8.90 10.41 1416 0.00 .... . 0.00 .... .
3b.11.19L1 72 116.77 96.26 2.91 8033 11.16 28.06 814 9.36 8.41 1830 0.00 .... . 0.00 .... .
5c.12.20L1 73 118.62 97.34 2.73 6981 11.67 25.65 804 9.62 10.26 1279 0.00 .... . 0.00 .... .
3b.11.20L1 74 121.71 102.22 2.85 7607 10.90 29.59 817 8.59 12.59 1044 0.00 .... . 0.00 .... .
5c.12.21L1 75 121.09 101.14 2.63 7662 11.52 29.55 813 8.43 10.06 1579 0.00 .... . 0.00 .... .
3b.11.21L1 76 119.55 100.37 3.13 6643 10.64 30.10 740 8.54 10.92 1326 0.00 .... . 0.00 .... .
5c.12.22L1 73 114.77 94.10 2.77 7080 11.67 26.58 729 9.00 12.43 1110 0.00 .... . 0.00 .... .
3b.11.22L1 75 118.01 96.93 2.42 7885 11.72 26.61 760 9.36 10.25 1125 0.00 .... . 0.00 .... .
/Aggr1_SATA3T_FP/plex0/rg2:
5c.12.23L1 14 22.98 0.00 .... . 9.67 49.59 249 13.32 26.05 271 0.00 .... . 0.00 .... .
3b.11.23L1 14 23.04 0.00 .... . 9.77 49.08 229 13.27 26.38 250 0.00 .... . 0.00 .... .
5c.12.1L2 76 124.43 102.48 2.65 8441 12.55 28.27 950 9.41 9.63 1646 0.00 .... . 0.00 .... .
3b.11.1L2 76 125.00 103.20 2.51 8328 12.39 28.20 785 9.41 10.21 1287 0.00 .... . 0.00 .... .
5c.12.2L2 77 123.10 101.71 2.97 6477 12.19 29.04 793 9.20 11.33 1060 0.00 .... . 0.00 .... .
3b.11.2L2 74 119.81 98.42 2.74 8025 12.49 29.49 851 8.90 10.77 1297 0.00 .... . 0.00 .... .
5c.12.3L2 74 119.19 95.02 2.62 7376 13.78 25.29 788 10.39 10.19 1212 0.00 .... . 0.00 .... .
3b.11.3L2 73 119.40 97.34 2.48 7677 12.60 27.12 798 9.46 10.59 1193 0.00 .... . 0.00 .... .
5c.12.4L2 75 121.35 99.70 2.74 6796 11.83 29.44 754 9.82 9.75 1480 0.00 .... . 0.00 .... .
3b.11.4L2 76 126.08 103.92 2.67 7461 13.21 27.09 846 8.95 11.41 1101 0.00 .... . 0.00 .... .
5c.12.5L2 75 120.68 98.36 2.64 7517 13.16 28.04 827 9.15 10.63 1339 0.00 .... . 0.00 .... .
3b.11.5L2 75 120.12 98.78 2.75 7045 11.93 30.57 848 9.41 11.25 1023 0.00 .... . 0.00 .... .
5c.12.6L2 75 120.58 96.57 2.29 9169 13.57 24.19 982 10.44 10.58 1452 0.00 .... . 0.00 .... .
3b.11.6L2 76 122.74 100.99 2.85 7564 12.24 30.64 836 9.51 7.86 1739 0.00 .... . 0.00 .... .
5c.12.7L2 73 112.20 92.55 2.55 7464 11.11 32.34 902 8.54 10.27 1474 0.00 .... . 0.00 .... .
/Aggr1_SATA3T_FP/plex0/rg3:
3b.11.7L2 14 23.40 0.00 .... . 9.51 50.12 243 13.88 25.37 221 0.00 .... . 0.00 .... .
5c.12.9L2 14 23.40 0.00 .... . 9.62 49.61 240 13.78 25.48 215 0.00 .... . 0.00 .... .
3b.11.9L2 71 113.94 90.55 2.84 6064 13.21 27.06 639 10.18 8.12 1365 0.00 .... . 0.00 .... .
5c.12.10L2 74 118.78 94.87 2.62 7255 13.63 25.56 822 10.28 7.91 1810 0.00 .... . 0.00 .... .
3b.11.10L2 73 124.49 98.11 2.71 7207 14.86 23.12 716 11.52 8.71 1310 0.00 .... . 0.00 .... .
5c.12.11L2 72 113.89 90.70 2.79 6770 12.85 26.74 774 10.34 7.59 1714 0.00 .... . 0.00 .... .
3b.11.11L2 73 117.80 95.13 2.75 7133 12.55 26.23 827 10.13 9.61 1340 0.00 .... . 0.00 .... .
5c.12.12L2 72 120.68 93.17 2.69 6649 15.37 22.36 799 12.13 9.56 1152 0.00 .... . 0.00 .... .
3b.11.12L2 75 120.22 94.25 2.84 6997 15.12 24.13 791 10.85 9.14 1694 0.00 .... . 0.00 .... .
5c.12.13L2 75 125.87 99.34 2.73 7279 15.84 22.95 816 10.70 8.63 1401 0.00 .... . 0.00 .... .
3b.11.13L2 68 109.78 84.79 3.11 7043 14.71 22.58 742 10.28 9.55 1188 0.00 .... . 0.00 .... .
5c.12.14L2 74 118.37 94.41 2.79 7403 14.60 23.97 816 9.36 9.12 1613 0.00 .... . 0.00 .... .
3b.11.14L2 75 125.93 99.14 2.59 7628 15.68 22.40 870 11.11 9.82 1393 0.00 .... . 0.00 .... .
5c.12.15L2 73 112.45 88.39 2.73 6870 13.78 24.78 859 10.28 9.19 1482 0.00 .... . 0.00 .... .
3b.11.16L2 76 126.29 102.22 2.74 8802 13.57 26.33 813 10.49 8.88 1696 0.00 .... . 0.00 .... .
/Aggr1_SATA3T_FP/plex0/rg4:
5c.12.17L2 14 21.24 0.00 .... . 9.41 50.84 234 11.83 30.77 226 0.00 .... . 0.00 .... .
3b.11.17L2 14 21.44 0.00 .... . 9.51 50.31 241 11.93 30.59 223 0.00 .... . 0.00 .... .
5c.12.18L2 72 120.17 99.50 2.78 6103 11.00 29.16 645 9.67 11.04 971 0.00 .... . 0.00 .... .
3b.11.18L2 72 120.73 101.09 2.53 6831 11.06 29.50 759 8.59 12.83 934 0.00 .... . 0.00 .... .
5c.12.19L2 72 119.86 99.70 2.83 5722 10.80 31.77 726 9.36 11.81 1095 0.00 .... . 0.00 .... .
3b.11.19L2 72 113.64 95.02 2.54 7330 10.34 31.59 706 8.28 12.35 1021 0.00 .... . 0.00 .... .
5c.12.20L2 74 118.06 99.55 2.54 7949 10.18 34.63 764 8.33 11.60 1344 0.00 .... . 0.00 .... .
3b.11.20L2 76 125.67 104.69 2.79 7218 11.57 29.04 778 9.41 12.87 1142 0.00 .... . 0.00 .... .
5c.12.21L2 75 120.53 100.78 2.70 7411 11.31 29.86 802 8.43 12.04 1142 0.00 .... . 0.00 .... .
3b.11.21L2 73 119.34 100.32 2.79 5728 10.64 32.20 674 8.38 12.20 918 0.00 .... . 0.00 .... .
5c.12.22L2 72 125.15 105.46 2.71 7004 10.90 29.75 640 8.79 12.43 903 0.00 .... . 0.00 .... .
3b.11.22L2 72 113.02 93.99 2.81 6758 10.34 34.85 630 8.69 11.70 1141 0.00 .... . 0.00 .... .
5c.12.0L2 74 122.02 101.50 2.59 6933 11.67 28.49 723 8.84 10.31 1561 0.00 .... . 0.00 .... .
3b.11.8L2 69 112.25 93.27 2.73 5918 10.59 31.81 644 8.38 14.35 944 0.00 .... . 0.00 .... .
5c.12.16L2 75 116.00 96.98 2.61 7461 10.28 32.69 779 8.74 12.24 1072 0.00 .... . 0.00 .... .
/Aggr1_SATA3T_FP/plex0/rg5:
5b.21.6 37 1356.75 0.00 .... . 481.13 11.75 89 875.62 3.98 199 0.00 .... . 0.00 .... .
3c.21.7 32 1357.31 0.00 .... . 483.13 11.71 75 874.18 3.98 152 0.00 .... . 0.00 .... .
5b.21.8 79 3378.86 2154.77 1.34 2009 632.97 6.23 141 591.11 1.96 485 0.00 .... . 0.00 .... .
3c.21.9 80 3381.84 2165.52 1.36 1732 627.78 6.26 104 588.54 2.02 393 0.00 .... . 0.00 .... .
5b.21.10 79 3382.30 2169.22 1.35 2014 632.15 6.23 142 580.93 1.96 478 0.00 .... . 0.00 .... .
/Aggr0_SATA3T/plex0/rg0:
3b.11.0L1 6 6.84 0.31 1.00 16167 4.11 8.14 733 2.42 11.23 722 0.00 .... . 0.00 .... .
5c.12.0L1 9 7.71 0.31 1.00 15500 5.04 7.01 834 2.37 9.96 1511 0.00 .... . 0.00 .... .
3b.11.1L1 7 12.13 5.14 1.29 6953 3.14 7.38 969 3.86 2.12 1943 0.00 .... . 0.00 .... .
5c.12.1L1 4 7.04 4.42 1.22 5295 1.23 16.38 578 1.39 6.67 422 0.00 .... . 0.00 .... .
3b.11.2L1 4 6.84 4.42 1.16 6740 1.34 16.50 615 1.08 7.19 517 0.00 .... . 0.00 .... .
5c.12.2L1 4 7.40 4.78 1.17 6294 1.34 16.65 490 1.29 5.36 694 0.00 .... . 0.00 .... .
3b.11.3L1 5 8.43 5.55 1.16 5552 1.23 16.54 562 1.65 5.22 1024 0.00 .... . 0.00 .... .
5c.12.3L1 4 7.61 4.83 1.19 6080 1.23 17.33 550 1.54 4.20 1349 0.00 .... . 0.00 .... .
3b.11.4L1 4 6.89 4.27 1.11 6967 1.23 17.21 547 1.39 4.19 1097 0.00 .... . 0.00 .... .
5c.12.4L1 5 9.05 6.63 1.14 6789 1.34 16.35 569 1.08 3.95 1181 0.00 .... . 0.00 .... .
3b.11.5L1 4 7.35 4.78 1.14 5415 1.34 16.27 515 1.23 5.83 843 0.00 .... . 0.00 .... .
5c.12.5L1 4 5.91 3.14 1.23 7427 1.39 15.37 472 1.39 7.52 1000 0.00 .... . 0.00 .... .
3b.11.6L1 4 6.43 4.01 1.21 6383 1.34 16.38 502 1.08 2.14 889 0.00 .... . 0.00 .... .
5c.12.6L1 4 6.63 3.81 1.31 6216 1.34 16.12 525 1.49 3.69 841 0.00 .... . 0.00 .... .
3b.11.7L1 4 6.53 3.81 1.22 5933 1.23 17.42 577 1.49 4.10 1067 0.00 .... . 0.00 .... .
5c.12.7L1 4 8.38 5.66 1.30 4804 1.34 16.31 580 1.39 5.63 664 0.00 .... . 0.00 .... .
Aggregate statistics:
Minimum 4 5.91 0.00 1.23 1.08 0.00 0.00
Mean 55 219.41 134.77 39.70 44.94 0.00 0.00
Maximum 80 3382.30 2169.22 632.97 875.62 0.00 0.00
Spares and other disks:
3c.21.3 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3c.21.1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3c.21.11 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5b.21.4 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5b.21.0 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5b.21.2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.0L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.1L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.0L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.1L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.2L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.2L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.3L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.3L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.4L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.5L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.4L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.5L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.6L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.6L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.8L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.7L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.8L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.7L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.9L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.9L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.10L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.11L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.10L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.11L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.12L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.13L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.12L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.13L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.14L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.15L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.14L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.16L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.15L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5c.12.8L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.16L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.17L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.17L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.18L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.19L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.18L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.19L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.20L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.20L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.21L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.22L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.21L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.22L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.23L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.23L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.0L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.1L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.0L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.1L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5c.12.23L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3b.11.0L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.2L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.3L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.2L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.3L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.6L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.7L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.6L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.4L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.4L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.7L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.5L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.5L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.8L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.10L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.8L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.9L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.9L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.10L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.11L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.11L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3b.11.15L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.12L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.13L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.12L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.13L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.14L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.15L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.14L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.15L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.16L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.17L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.16L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.17L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3b.11.23L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.18L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.19L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.18L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.19L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.20L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.20L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.21L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.22L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.21L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.22L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.23L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.23L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3c.21.5 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
FCP Statistics (per second)
0.00 FCP Bytes recv 0.00 FCP Bytes sent
0.00 FCP ops
iSCSI Statistics (per second)
79838395.37 iSCSI Bytes recv 34573619.36 iSCSI Bytes xmit
3912.98 iSCSI ops
Interrupt Statistics (per second)
10.13 int_1 1532.18 int_2
1687.29 int_3 3641.59 int_4
2907.05 int_5 2721.81 int_6
1840.08 int_7 2871.51 int_8
5.19 Gigabit Ethernet (IRQ 12) 8.43 Gigabit Ethernet (IRQ 13)
1.54 int_14 0.72 int_15
0.00 RTC 0.00 IPI
996.93 Msec Clock 18224.47 total
'
We are facing this on multiple filers, and on one of our filer we resolved this with a strange solution which was by changing the SNMP string from public to some private string.
Hope for some solution on the basis of provided output and thanks in advance.