After performing the tests with iSCSI and discussing the results with a friend of mine, I was a little disappointed as it seemed that the IX4 didn’t seem to perform as well as his system. Biggest difference between my IX4 and his NAS was the use of iSCSI on the IX4 and NFS on his. The next step might be clear, test the whole set again on NFS and so I did and I can already reveal that performance was much better now and is comparable to his NAS, which made me very happy again :-)
With the same ESX and VM configuration I reran my tests. The IX4-200D was still configured with a 2.7TB volume of which 1.5TB is configured for iSCSI, but the remaining 1.2TB is shared as NFS. I moved the test VM to this NFS volume and started iometer.
| NFS Test | |||||
| Test | Description | MB/sec | IOPS | Average IO response time | Maximum IO response time |
| Nfs 001a | Max Throughput 100% read | 108.5909 | 3474.908 | 17.22548 | 242.8462 |
| Nfs 001b | RealLife-60%Rand-65%Read | 0.879284 | 112.5484 | 537.8718 | 7263.568 |
| Nfs 001c | Max Throughput-50%Read | 21.38742 | 684.3974 | 87.72192 | 1764.932 |
| Nfs 001d | Random-8k-70%Read | 0.709705 | 90.84225 | 659.329 | 6771.195 |
For easy comparison find the previous iSCSI test below.
| iSCSI Test | |||||
| Test | Description | MB/sec | IOPS | Average IO response time | Maximum IO response time |
| Test 001a | Max Throughput 100% read | 55.058866 | 1761.883723 | 35.021015 | 207.740649 |
| Test 001b | RealLife-60%Rand-65%Read | 0.696917 | 89.205396 | 663.790422 | 11528.93203 |
| Test 001c | Max Throughput-50%Read | 22.040195 | 705.286232 | 83.689648 | 252.396324 |
| Test 001d | Random-8k-70%Read | 0.505056 | 64.647197 | 913.201061 | 12127.4405 |
I was surprised to see so much difference in performance, I had expected some difference, but no difference as big as this. Looking at the data of the first “Super ATTO Clone pattern†test I ran, I can see the biggest difference between NFS and iSCSI being the short peak in read speed where iSCSI remained stable after reaching its peak performance at 41 MB/sec. NFS peaked to 110 MB/sec testing block sizes from 32K to 512K and dropped in speed to 57 MB/sec on blocks of 1M and larger.
Write speeds
I noticed a strange result in the writing section. When enlarging the block sizes the write speed builds to around 11 MB/sec but suddenly spikes to 44MB/sec at 64K blocks and then drops back to 23 MB/sec at 128K and 10 MB/sec at 256K. To be sure this wasn’t a testing error I reran the test a number of times and compared the results. Below you see the table with performance results of the AVERAGE of 4 “Super ATTO Clone†tests for NFS, compared to one iSCSI test.
| iSCSI | NFS (average) | |||
| Block size | Read
MB/sec |
Write MB/sec | Read
MB/sec |
Write MB/sec |
| 0.5K | 6.950 | 4.540 | 4.931 | 2.931 |
| 1K | 12.770 | 5.824 | 9.736 | 7.332 |
| 2K | 17.858 | 7.154 | 16.756 | 10.844 |
| 4K | 25.980 | 8.080 | 31.162 | 10.508 |
| 8K | 34.296 | 9.247 | 50.480 | 10.865 |
| 16K | 34.410 | 9.652 | 79.811 | 10.870 |
| 32K | 37.686 | 9.828 | 108.821 | 11.102 |
| 64K | 40.271 | 9.840 | 110.614 | 25.669 |
| 128K | 41.862 | 9.712 | 110.800 | 18.909 |
| 256K | 41.918 | 9.689 | 110.985 | 15.503 |
| 512K | 41.011 | 9.725 | 110.976 | 20.432 |
| 1M | 41.443 | 9.713 | 58.147 | 17.315 |
| 2M | 41.093 | 9.719 | 57.042 | 14.480 |
| 4M | 41.241 | 9.703 | 56.709 | 14.537 |
| 8M | 41.006 | 9.687 | 57.477 | 12.273 |
Next is a graph that shows differences in read performance. See how NFS is much faster, even though there is a little drop at block sizes of 1M and more.
When comparing write performance (image below), you can see how NFS is at all points faster than iSCSI, but has a strange pattern above 32K.
Trying to explain the strange results, I ran the same test 4 times, but each run gave different results compared to the other runs and I couldn’t come to one set of constant results, since there is always an unexplained peak somewhere in the larger block sizes. See the graph below in which 4 runs have been compared to each other.
As you can see in Run1 and Run3 I have a peak write performance at 64K block size that is very much out of band compared to Run2 and Run4 at 64K. Later on you see Run4 peeking to 30 MB/secat 512K blocks. I can’t explain why this happens. I’ve read through some NFS whitepapers to see if any saturation is normal for the protocol, but that isn’t the case. This “issue†should be related to my test lab somehow, but I can’t find the exact reason. I checked cpu of the IX4-200D during Run4 test, but cpu usage is well below 80% all of the time. In this run the %buffer that is close to 100% when writing 512K blocks, but that is where throughput is above the average 10MB/sec. If any NFS or storage expert out there can shed a light on this, I would be delighted to learn more about this.
Despite the strange results, I would recommend 64K block size as most optimal block size on NFS connected to the IX4-200D.
Reproducing the tests
To perform all these tests, I used iometer and a set of predefined tests. I have included them in the WinRAR file you can download Storage-Performance-Test.rar. For security reasons I had to rename the .rar file to .doc. So after downloading, please rename it to .rar. When you open the archive you’ll find the following files:
-         Histogram.xls: shows the differences between the iSCSI and NFS “Super ATTO†test.
-         Nfs-001-compare-block-ix4-cpu.xls: Shows the iostat data from the IX4-200D compared to the “Super ATTO “ performance data. Since iostat on the IX4 doesn’t produce any time stamps, there can be a small shift in time where I linked both data sets with each other.
-Â Â Â Â Â Â Â Â Â Super ATTO Clone Pattern.icf. This is an iometer config file containing the block sizes test. You can open it and use it to run the test with. Before starting check the settings explained belowd.
-Â Â Â Â Â Â Â Â Â vmware-community-test.icf. This configuration file performs the set of four tests that was also used in the vmware communities.
After starting iometer and loading a configuration set, make sure you also check these settings in the image below. The Maximum Disk size is set to 8,000,000 sectors. This will produce a 4GB test file on D-Drive. Make sure the test file is bigger than the amount of memory in the VM. Also set the “Test Connection Rate†to 500 transactions per second.
At the “Access Specifications†tab select the test(s) you want to run and then hit the green flag to start the tests. You can see live results at the “Results Display†tab.
