Putting your storage to the test – Part 2 NFS on Iomega IX4-200D

After performing the tests with iSCSI and discussing the results with a friend of mine, I was a little disappointed as it seemed that the IX4 didn’t seem to perform as well as his system. Biggest difference between my IX4 and his NAS was the use of iSCSI on the IX4 and NFS on his. The next step might be clear, test the whole set again on NFS and so I did and I can already reveal that performance was much better now and is comparable to his NAS, which made me very happy again :-)

With the same ESX and VM configuration I reran my tests. The IX4-200D was still configured with a 2.7TB volume of which 1.5TB is configured for iSCSI, but the remaining 1.2TB is shared as NFS. I moved the test VM to this NFS volume and started iometer.

NFS Test
Test Description MB/sec IOPS Average IO response  time Maximum IO response time
Nfs 001a Max Throughput 100% read 108.5909 3474.908 17.22548 242.8462
Nfs 001b RealLife-60%Rand-65%Read 0.879284 112.5484 537.8718 7263.568
Nfs 001c Max Throughput-50%Read 21.38742 684.3974 87.72192 1764.932
Nfs 001d Random-8k-70%Read 0.709705 90.84225 659.329 6771.195

For easy comparison find the previous iSCSI test below.

iSCSI Test
Test Description MB/sec IOPS Average IO response  time Maximum IO response time
Test 001a Max Throughput 100% read 55.058866 1761.883723 35.021015 207.740649
Test 001b RealLife-60%Rand-65%Read 0.696917 89.205396 663.790422 11528.93203
Test 001c Max Throughput-50%Read 22.040195 705.286232 83.689648 252.396324
Test 001d Random-8k-70%Read 0.505056 64.647197 913.201061 12127.4405

I was surprised to see so much difference in performance, I had expected some difference, but no difference as big as this. Looking at the data of the first “Super ATTO Clone pattern” test I ran, I can see the biggest difference between NFS and iSCSI being the short peak in read speed where iSCSI remained stable after reaching its peak performance at 41 MB/sec. NFS peaked to 110 MB/sec testing block sizes from 32K to 512K and dropped in speed to 57 MB/sec on blocks of 1M and larger.

Write speeds

I noticed a strange result in the writing section. When enlarging the block sizes the write speed builds to around 11 MB/sec but suddenly spikes to 44MB/sec at 64K blocks and then drops back to 23 MB/sec at 128K and 10 MB/sec at 256K. To be sure this wasn’t a testing error I reran the test a number of times and compared the results. Below you see the table with performance results of the AVERAGE of 4 “Super ATTO Clone” tests for NFS, compared to one iSCSI test.

iSCSI NFS (average)
Block size Read


Write MB/sec Read


Write MB/sec
0.5K 6.950 4.540 4.931 2.931
1K 12.770 5.824 9.736 7.332
2K 17.858 7.154 16.756 10.844
4K 25.980 8.080 31.162 10.508
8K 34.296 9.247 50.480 10.865
16K 34.410 9.652 79.811 10.870
32K 37.686 9.828 108.821 11.102
64K 40.271 9.840 110.614 25.669
128K 41.862 9.712 110.800 18.909
256K 41.918 9.689 110.985 15.503
512K 41.011 9.725 110.976 20.432
1M 41.443 9.713 58.147 17.315
2M 41.093 9.719 57.042 14.480
4M 41.241 9.703 56.709 14.537
8M 41.006 9.687 57.477 12.273

Next is a graph that shows differences in read performance. See how NFS is much faster, even though there is a little drop at block sizes of 1M and more.

When comparing write performance (image below), you can see how NFS is at all points faster than iSCSI, but has a strange pattern above 32K.

Trying to explain the strange results, I ran the same test 4 times, but each run gave different results compared to the other runs and I couldn’t come to one set of constant results, since there is always an unexplained peak somewhere in the larger block sizes. See the graph below in which 4 runs have been compared to each other.

As you can see in Run1 and Run3 I have a peak write performance at 64K block size that is very much out of band compared to Run2 and Run4 at 64K. Later on you see Run4 peeking to 30 MB/secat 512K blocks. I can’t explain why this happens. I’ve read through some NFS whitepapers to see if any saturation is normal for the protocol, but that isn’t the case. This “issue” should be related to my test lab somehow, but I can’t find the exact reason. I checked cpu of the IX4-200D during Run4 test, but cpu usage is well below 80% all of the time. In this run the %buffer that is close to 100% when writing 512K blocks, but that is where throughput is above the average 10MB/sec. If any NFS or storage expert out there can shed a light on this, I would be delighted to learn more about this.

Despite the strange results, I would recommend 64K block size as most optimal block size on NFS connected to the IX4-200D.

Reproducing the tests

To perform all these tests, I used iometer and a set of predefined tests. I have included them in the WinRAR file you can download Storage-Performance-Test.rar. For security reasons I had to rename the .rar file to .doc. So after downloading, please rename it to .rar. When you open the archive you’ll find the following files:

-          Histogram.xls: shows the differences between the iSCSI and NFS “Super ATTO” test.

-          Nfs-001-compare-block-ix4-cpu.xls: Shows the iostat data from the IX4-200D compared to the “Super ATTO “ performance data. Since iostat on the IX4 doesn’t produce any time stamps, there can be a small shift in time where I linked both data sets with each other.

-          Super ATTO Clone Pattern.icf. This is an iometer config file containing the block sizes test. You can open it and use it to run the test with. Before starting check the settings explained belowd.

-          vmware-community-test.icf. This configuration file performs the set of four tests that was also used in the vmware communities.

After starting iometer and loading a configuration set, make sure you also check these settings in the image below. The Maximum Disk size is set to 8,000,000 sectors. This will produce a 4GB test file on D-Drive. Make sure the test file is bigger than the amount of memory in the VM. Also set the “Test Connection Rate” to 500 transactions per second.

At the “Access Specifications” tab select the test(s) you want to run and then hit the green flag to start the tests. You can see live results at the “Results Display” tab.