10 gigabit LAN file transfers. Theoretical vs actual

Status
This thread has been Locked and is not open to further replies. Please start a New Thread if you're having a similar issue. View our Welcome Guide to learn how to use this site.

Oscis

Thread Starter
Joined
Mar 7, 2020
Messages
2
Hello, I have 3 machines running Windows 10. Computer A has a is being used as a file server of sorts, storing terabytes of backups and network resources. Computers B and C are being used as general purpose machines for gaming, compiling programs, and 3D modeling, and need access to the files on computer A.

In computer A, I have an LSI MegaRAID SAS 9260-8i raid card that has SAS to 4 SATA breakout cable that goes to 4 SSDs. Crystal Disk Mark 7.0.0h, with the settings 1 test count, 1GiB, benchmarks this array at a 2000MB/s SEQ1M Q8T1 (sequential read) speed if I benchmark this on the server using a system linked folder to c:\ssd. If I try to run Crystal Disk Mark on a client machine for a system linked folder c:\ssd "mklink \d "C:\ssd" "\\server\ssd", then I get 285 MB/s.

Running iPerf3, I get a network throughput of 1.7 - 2.0 gb/s. I understand TCP has some overhead, but why is there such a large difference between the read and write speeds I get locally on the server, and the read and write speeds I get over the network?

Computer A has a
QLogic BCM57810 10 Gigabit Ethernet (NDIS VBD Client) #34
and
QLogic BCM57810 10 Gigabit Ethernet (NDIS VBD Client) #35

These are set up in a NetSwitchTeam using powershell, I was hoping so that the two client machines could each get up to 10 gigabits/s for file transfers.

Computer B has a Mellanox ConnectX-2 Ethernet Adapter, and so does computer C.

These are all plugged into a MikroTik 5-Port Desktop Switch, 1 Gigabit Ethernet Port, 4 SFP+ 10Gbps Ports (CRS305-1G-4S+IN).

I do have the option to bond the two interfaces connected to the server in the MikroTik switch, but either way, I seem to get 285MB/s read speed on this array.

What could my bottleneck be? I don't see CPU usage running very high on the server or client machines during benchmarking, nor do I see this SSD array peaking very high. The ethernet interface sends 1-2Gb/s on the server side. I was hoping the real world speed of a 10 gigabit network would be over 9 gigabit per seconds.

Any help you can offer, or ways to log exactly what's happening so I can find the bottleneck for these transfers would be appreciated.
 

zx10guy

Trusted Advisor
Spam Fighter
Joined
Mar 30, 2008
Messages
6,685
I just did a file transfer test. Here is my setup:

Desktop with Intel x520 and Micron 9100 3.84TB NVMe drive connected to a Netgear M4300 10 Gig switch over a single 10Gig connection
Netgear M4300 has two 10Gig links channel bonded to a Dell PowerConnect 3048P.
Dell PowerConnect 3048P has a single 10Gig to a Dell S4810 (need to fix this to add a second 10Gig link)
Dell S4810 connected to an Dell PowerEdge R730 over 40Gig
Dell R730 is running vSphere 6.5 with my fileserver as a VM using the 10Gig virtual NIC and the storage is over dual 8Gig fiber channel to a Compellent SC4020 SAN

With this setup, I tested local disk performance with only the couple of sequential read tests on CrystalMark. The results were about 823 MB/s sequential read. Testing over the network transferring a 40GB file, I got an average of 720 MB/s with an occasional drop to about 699 MB/s and peak at 760 MB/s.

There is some overhead with just basic TCP/IP which will reduce the performance some but not by a lot. I suspect my performance would be a whole lot better if I upgrade my fileserver VM from Windows Server 2008 to Server 2016 with the enhancements done with SMBv3. Not sure why your performance is that low.
 

Oscis

Thread Starter
Joined
Mar 7, 2020
Messages
2
I think it has something to do with my mikrotik 10 gigabit router. I've emailed support to ask what an ideal configuration would be.
 
Status
This thread has been Locked and is not open to further replies. Please start a New Thread if you're having a similar issue. View our Welcome Guide to learn how to use this site.

Users Who Are Viewing This Thread (Users: 0, Guests: 1)

As Seen On
As Seen On...

Welcome to Tech Support Guy!

Are you looking for the solution to your computer problem? Join our site today to ask your question. This site is completely free -- paid for by advertisers and donations.

If you're not already familiar with forums, watch our Welcome Guide to get started.

Join over 807,865 other people just like you!

Latest posts

Members online

Top