<div class="gmail_quote">On 19 July 2012 12:27, Steve Flynn <span dir="ltr"><<a href="mailto:anothermindbomb@gmail.com" target="_blank">anothermindbomb@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
</div>No surprise there. For reference, we export NFS mounts like so:<br>
<br>
/some_filesystem -public,sec=sys,rw,sec=dh:krb5:krb5i:krb5p,rw<br>
<br>
and we mount with<br>
<br>
/some_filesystem:<br>
dev = "/client_data"<br>
vfs = nfs<br>
nodename = one_of_our_aix_servers<br>
mount = true<br>
options = bg,hard,intr,sec=sys:dh:krb5:krb5i:krb5p<br>
account = false<br>
<div class="im"><br>
<br>
</div>(I'm no expert so take this with the usual grain of salt)<br>
<br>
I'd start with an eyeball of the output from "ifconfig -a" on both<br>
systems. Make sure they both look sensible and that one of them isn't<br>
configured to route all traffic through some flaky old F5 firewall in<br>
the basement, by accident.<br>
<br>
Providing both installations look the same, have the same MTU,<br>
sensible error levels for Tx and Rx after a test session and so forth<br>
it's time to start poking around in /proc to compare tuning values.<br>
This is where I can offer little, as I have no Linux or FreeBSD<br>
systems to hand - just AIX which isn't going to help you much.<br>
<br>
This /might/ help: <a href="http://nfs.sourceforge.net/nfs-howto/ar01s05.html" target="_blank">http://nfs.sourceforge.net/nfs-howto/ar01s05.html</a><br>
<br>
<br>
Out of interest, what are you using to generate the load on the NFS<br>
filesystem and what are you using to measure the number of FSYNCS? Are<br>
you sure that the two are equivalent under both installations (e.g.<br>
bonnie on Linux is running with a 64K chunk of data to write to the<br>
NFS filesystem whereas whatever you're using on FreeBSD is running<br>
with a 2K chunk)... I'm struggling to come up with a plausible reason<br>
for such a massive difference in sync rates between two installations<br>
on the same kit, talking to the same filesystem.<br>
<div class="HOEnZb"><div class="h5"><br>
</div><div class="h5"><br></div></div></blockquote><div>Thanks for your thoughts and input Steve.. Might just stick with FreeBSD for the storage server and Ubuntu/Debian for the KVM hosts..</div><div><br></div><div>Still have to investigate the option of using iSCSI+LVM for VM drives instead of file based virtual disks over NFS so don't want to loose too much more time on the Linus NFS issue..</div>
</div><br><div>I am testing the storage from a Proxmox VE server which includes a utility called pveperf that tests the FSYNC's per second.. Below is the relevant source code for the test..</div><div><br></div><div><div>
sub test_fsync {</div><div> my $basedir = shift;</div><div> drop_cache ();</div><div> my $dir = "$basedir/ptest.$$";</div><div> eval {</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>mkdir $dir;</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>my $data = ('A' x 4000) . "\n";</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>my $starttime = [gettimeofday];</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>my $count;</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>my $elapsed = 0;</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>for ($count=1;;$count++) {</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> my $m = $count % 300;</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> my $filename = "$dir/tf_$m.dat";</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> open (TMP, ">$filename") || die "open failed";</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> print TMP $data;</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> File::Sync::fsync (\*TMP);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> close (TMP);</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> $elapsed = tv_interval ($starttime);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> last if $elapsed > 3;</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>}</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>my $sps = $count /$elapsed; # fsync per second</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>printf "FSYNCS/SECOND: %.2f\n", $sps;</div>
<div> };</div><div> my $err = $@;</div><div> system ("rm -rf $dir");</div><div> die $err if $err;</div><div>}</div></div>