slow network throughput, how to improve?

[SilverNodashi]

Hi all,

I would like some input on this one please.

Two CentOS 5.5 XEN servers, with 1GB NIC's, connected to a 1GB switch
transfer files to each other at about 30MB/s between each other.

Both servers have the following setup:
CentOS 5.5 x64
XEN 3.0 (from xm info: xen_caps : xen-3.0-x86_64
xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64)
kernel 2.6.18-194.11.3.el5xen
1GB NIC's
7200rpm SATA HDD's

The hardware configuration can't change, I need to use these servers
as they are. They are both used in production with a few xen domU's
virtual machines running on them.
I want to connect them both to a SAN, with gigabit connectivity and
would like to know how I can increase network performance a bit, as
is.
The upstream datacentre only supplies 100MB network connection, so in
the internet side of it isn't much of a problem. If I do manage to
reach 100MB that will be my limit in any case.


[email protected]:/vm/xen/template/centos-5-x64-cpanel
[email protected]:/
[email protected]'s password:
centos-5-x64-cpanel.tar.gz
100% 1163MB
29.1MB/s 00:40


iperf indicates that the network throughput is about 930MB though:

root@zaxen01:[~]$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 196.34.x.x port 5001 connected with 196.34.x.x port 45453
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1.09 GBytes 935 Mbits/sec

root@zaxen02:[~]$ iperf -c zaxen01
------------------------------------------------------------
Client connecting to zaxen01, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 196.34.x.x port 45453 connected with 196.34.x.x port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.09 GBytes 936 Mbits/sec


Is iperf really that accurate, or reliable in this instance, since the
packet size is so small that it probably goes straight to memory,
instead of HDD? But at the same time, changing the packet size to
10MB, 100MB and 1000MB respectively doesn't seem to degrade
performance much either:

root@zaxen02:[~]$ iperf -w 10M -c zaxen01
------------------------------------------------------------
Client connecting to zaxen01, TCP port 5001
TCP window size: 256 KByte (WARNING: requested 10.0 MByte)
------------------------------------------------------------
[ 3] local 196.34.x.x port 36756 connected with 196.34.x.x port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.07 GBytes 921 Mbits/sec
root@zaxen02:[~]$ iperf -w 100M -c zaxen01
------------------------------------------------------------
Client connecting to zaxen01, TCP port 5001
TCP window size: 256 KByte (WARNING: requested 100 MByte)
------------------------------------------------------------
[ 3] local 196.34.x.x port 36757 connected with 196.34.x.x9 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.08 GBytes 927 Mbits/sec
root@zaxen02:[~]$ iperf -w 1000M -c zaxen01
------------------------------------------------------------
Client connecting to zaxen01, TCP port 5001
TCP window size: 256 KByte (WARNING: requested 1000 MByte)
------------------------------------------------------------
[ 3] local 196.34.x.x port 36758 connected with 196.34.x.x port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.04 GBytes 895 Mbits/sec

 

[SilverNodashi]

Doesn't anyone have any input on this?

 

[ponder]

http://ltp.sourceforge.net/tooltable.php for benchmark tools.
http://www.robertshady.com/content/measuring-network-throughput-between-linux-servers


30MB/s seems pretty slow.

Check your adapter settings and maybe hard code them for 1Gb/s with Full-Duplex, also try enabling Jumbo frames if supported by all devices.

 

[gregmcc]

Do you have mii-tool on the server? Does it show a gigabit connection. Is it a managed switch - are you able to view and of the port stats. Maybe errors on the port?

 

[syntax]

http://ltp.sourceforge.net/tooltable.php for benchmark tools.
http://www.robertshady.com/content/measuring-network-throughput-between-linux-servers


30MB/s seems pretty slow.

Check your adapter settings and maybe hard code them for 1Gb/s with Full-Duplex, also try enabling Jumbo frames if supported by all devices.

As an FYI, according to RFC Gb should never be forced. (I have before, and not had problems...but IIRC even the RFC recommends against this)
Link for info

 

[ponder]

As an FYI, according to RFC Gb should never be forced. (I have before, and not had problems...but IIRC even the RFC recommends against this)
Link for info

I've never had problems forcing it. I have however had problems before letting it auto negotiate...go figure.

It's just something he can try.

 

[wetkit]

Well, I thinks it looks like a problem with your HD's rather than the network itself.

When there is a lot of users assessing a normal drive, the performance does go way down.
I am not sure how the HD's should be configured on your server itself, but with WIN it does help a lot to change the drives from normal IDE mode to AHCI mode.
This will enable NCQ on those drives, making them perform much better.