10 GIGABYTES a SECOND?? - HOLY $H!T
Linus Tech Tips
·Linus Tech Tips
·2019-05-06
·
2,384 words · ~11 min read
0:00
As many of you probably know,
0:01
the last few years have been a heck of a ride
0:04
for data enthusiasts,
0:06
with high speed storage becoming increasingly accessible.
0:08
I mean, something like Samsung's 960 Pro SSD.
0:12
It's tiny,
0:14
it costs about a third of what I paid per Gigabyte
0:17
for my first large boot SSD,
0:19
and it's over an order of a magnitude faster
0:23
when it comes to real world performance.
0:26
But what some of you might not know
0:28
is that networking has been keeping pace.
0:31
So these bad boys right here
0:35
are Mellanox ConnectX 4 cards.
0:38
And even though they're already two generations old,
0:41
they can reach transfer speeds of 100 gigabit,
0:44
or about 12 gigabytes per second.
0:47
That is fast enough to download Fortnite's install files,
0:51
yes, we went there,
0:52
in literally one second.
0:55
Holy .
0:57
So let's check them out.
0:59
Before we get to the new crazy stuff,
1:08
let's do a quick refresher
1:09
on traditional networking hardware.
1:11
So I've brought along a couple of examples here.
1:14
These two cards run at gigabit
1:16
and 10 gigabit speeds respectively.
1:19
So this guy right here is about 10 times faster
1:21
than this one.
1:22
But other than that, they've got a lot in common.
1:25
So they've both got Intel controllers on board,
1:28
they both plug into a PCIe 2.0 slot,
1:31
and they're both Ethernet.
1:33
So,
1:34
thanks to their use of intercompatible
1:36
communication standards and the ubiquitous RJ45 connector,
1:41
they can talk to each other directly
1:43
or through a network switch like this one,
1:46
albeit only at the speed of the slowest link in the chain,
1:51
be it on this one gig card
1:53
or through this one gig network switch.
1:56
And honestly, either of these, especially this one,
2:00
should be more than enough for the average person
2:02
for quite some time.
2:04
But this isn't average person land,
2:07
which sounds like the world's most boring amusement park.
2:09
So we've decided to go totally overkill
2:13
and take it to the next step,
2:14
to greater than 10 gigabit speeds,
2:17
which brings us then back to our ConnectX 4s.
2:21
So the first difference, the ports.
2:24
This beefy looking thing right here
2:26
is what's called a QSFP plus connector.
2:30
And as you probably figured out on your own,
2:33
you can't just plug in a QSFP plus connector
2:34
and plug a standard network cable into this port.
2:39
And even if you could, well, I guess,
2:42
that brings us to the next difference.
2:45
The fact that out of the box,
2:47
these cards are designed to run not on Ethernet networks,
2:52
but on InfiniBand networks.
2:54
So even if you could plug it into your network switch,
2:58
it wouldn't be able to communicate with it
3:00
without some configuration.
3:02
And then finally, this one's actually
3:04
pretty interesting.
3:05
These cards, yes, my friends,
3:08
these network cards use a full fat PCI Express Gen 3
3:16
times 16 connection.
3:19
That is the same as your graphics card.
3:22
And they actually need it,
3:24
probably more so than your GPU,
3:27
if all you're doing is gaming.
3:29
So then with that in mind,
3:31
we will not be using just your average gaming machine
3:34
for our testing.
3:35
So on one side of our link,
3:37
we've got Intel's flagship 18 core processor
3:39
with a Rampage 6 Extreme motherboard, 128 gigs of RAM.
3:43
Mm, yeah.
3:44
And then in the other corner,
3:45
we had to slum it a little,
3:47
with a 16 core 7960X and an ASUS X299 Deluxe,
3:52
but then with the same amount of RAM.
3:54
And of course it's RGB across the board.
3:55
Now, the reason that we're using the X299 platform
3:59
with Core i9 processors is that we need to make sure
4:02
that we have enough PCI Express lanes coming,
4:04
directly off the CPU.
4:06
So 44 should give us enough for 16 lanes for networking,
4:11
16 lanes for our quad NVMe storage devices,
4:15
and then, you know, some leftovers for the graphics card.
4:19
So then, for our NVMe storage,
4:22
we scraped together four Samsung 960 Pros for our first one,
4:27
and then four Corsair MP500s for the other,
4:31
with both of them running,
4:32
with quad,
4:34
SSDs in RAID 0.
4:37
Today's video is about how to go fast,
4:40
not about how to put on your seatbelt.
4:44
This is like hilarious to me.
4:46
I never thought I would see the day
4:48
when the GPU in a gaming rig is the lowest priority,
4:53
like tier PCI Express device
4:56
sitting at the bottom of the board.
4:58
Anyway, for our OS,
5:00
we had to go with Windows Server 2016,
5:02
because as much as we wanted to try out Windows 10,
5:05
and Pro for Workstation,
5:06
which is supposed to support RDMA,
5:09
the tech that allows for these super high speed transfers,
5:12
it just didn't seem to be working for us for some reason.
5:17
All right, now, at this point, before we go further,
5:19
wanna give a big shout out to the guys over at Mellanox
5:22
for hooking us up with these 100 gigabit cards,
5:24
as well as a pair of 100 gigabit capable,
5:28
passive copper direct attach cables.
5:31
Wow, that just hit the bench.
5:32
Oh!
5:33
So if you wanna try this out at home,
5:35
these cards are actually available on eBay
5:37
for like two, 300 bucks a pop,
5:39
and then you'll pay about $60
5:41
for a three meter cable like this one.
5:43
It is worth mentioning though,
5:44
that if you're planning on running anything
5:45
further than five meters,
5:47
you have to use an active fiber cable,
5:49
which could cost upwards of $2,500 new.
5:53
So...
5:55
Damn!
5:56
All right, moving on.
5:57
Configuration then is our last step.
5:59
So while Jake does that behind me here, thank you.
6:03
Jake.
6:04
Let's talk about some of the technology
6:05
behind this networking magic.
6:07
So these cards are designed for use
6:10
with two different network fabrics,
6:13
InfiniBand and Ethernet.
6:14
And what makes InfiniBand special
6:17
is that compared to even the sub millisecond latency
6:20
of a typical Ethernet network,
6:22
InfiniBand networks can have less than 25% as much,
6:26
making them suitable for use cases
6:28
like over network storage access
6:30
and combining the processing power
6:32
of multiple servers,
6:34
just like in a data center or supercomputer.
6:37
And this is cool.
6:39
When you configure InfiniBand correctly,
6:42
it also forms what's called a lossless network,
6:45
meaning that packet loss should basically never happen.
6:49
For compatibility though,
6:50
we're going to be using them in Ethernet mode
6:52
alongside a technology called RDMA
6:55
or remote direct memory access.
6:58
When you put these together,
6:59
the setup is called ROCE or RDMA over conversion.
7:02
That's a network-converged Ethernet.
7:04
Now regular Ethernet implementations
7:05
require a lot of hoop jumping
7:07
in order to transfer data
7:08
as any information sent must be first moved
7:11
through the transport protocols driver,
7:13
then through sockets
7:15
before it can reach the applications memory,
7:17
eating up CPU cycles and increasing latency in the process.
7:20
However, with RDMA,
7:22
the network adapters are able to access data
7:25
directly from application memory,
7:28
offloading much of the processing from your CPU
7:31
onto the actual process, which is called RDMA,
7:32
and the process is called GPC.
7:32
And that's basically it.
7:32
That's basically it.
7:32
Processor that sits right on your network adapter. So these are known as zero copy transfers and they allow for
7:40
Incredibly fast transfers that are no longer limited by CPU processing power. Pretty dang snazzy. So are we ready to go?
7:48
We should be. Each card has an IP and a 50 Gigabyte RAM disk
7:54
So we should be able to do some pretty quick Windows transfers. So are we going directly from RAM disk to RAM disk right now?
7:59
Uh, so I think what we'll do, we got to see make sure it's working first, right?
8:04
That would be good. I mean that's supposed to be your job, but. Fingers crossed
8:09
You never, the second you try to do a networking demo, like
8:14
We got our 40 Gigabyte text file. What? 36 or 37 gigabytes. Why?
8:21
Why even? It's just a big file, okay?
8:25
Oh, that's not bad. That's not 10 gig, or 100 gig.
8:30
So we've got just over two gigabytes per second. So we're reading from that system's
8:37
NVMe array. Okay. And then we're dumping to the RAM disk on this system. So let's try going the other way.
8:42
So now we can go RAM disk to RAM disk. Is that right? No, that's still the same thing, but just the other way around. Whoa!
8:50
Just shy of four gigabytes per second.
8:54
I mean, let's put what just happened there in context. That is a 40, well,
9:00
just shy of 40 Gigabyte file.
9:02
So like, okay, DOOM. DOOM on PC is like 60 gigs.
9:07
And it takes, you know, probably for your internet connection at home, like what, an hour to download and install? That just happened in real time.
9:14
Okay, what's next? RAM disk to RAM disk? I gotta check. I think one of our RAM disks isn't working. I might have broken something.
9:23
Okay, so check this out. We've got the RAM disks working, but for now, this is just another quick benchmark. So when we aren't limited,
9:30
by the overhead of Windows File Explorer and Windows File Transfers,
9:34
if we're just using a straight disk performance benchmark,
9:38
I wanna show you the kinds of numbers we're looking at.
9:40
So anyone who's familiar with Atto is gonna already know
9:44
that this is freaking nuts.
9:46
At two kilobyte sizes, we are already seeing speeds in excess
9:51
of 100 megabytes per second, because we are actually reading
9:55
and writing off of the four Corsair SSD array that's on the iPad.
10:00
So we're gonna have to figure out how many megabytes per second we're gonna have to write off of the other machine.
10:05
This is stupid.
10:07
It's pretty cool, actually.
10:08
It's really cool. And it's getting stupider as time goes on.
10:12
We are already hitting 600 megabytes a second writes.
10:16
Oh, that's weak, man.
10:17
At 16 kilobytes, though.
10:19
Just wait.
10:20
That's the key. And it just keeps getting crazier.
10:25
We're at one Gigabyte per second already.
10:29
Three!
10:30
Doubled!
10:31
Oh, wow.
10:32
Oh, man.
10:33
We just cracked four gigs a second.
10:35
We just cracked five gigs a second!
10:38
And this is on a remote machine.
10:41
This is not a local array.
10:43
This is over the network.
10:45
This is a slow test.
10:47
Yeah.
10:48
It's a really fast, slow test.
10:49
Yeah, it takes a while.
10:50
A really slow, fast test.
10:51
It's in the billions of bytes.
10:53
Billions of bytes.
10:54
It's, like, not even readable anymore at that point.
10:57
No, it's just like, what is going on?
10:59
How do I maths?
11:00
Oh, meanwhile.
11:01
Are we there?
11:02
Just cracked it.
11:03
10 gigabytes a second.
11:05
Okay.
11:07
So, what's our next test?
11:08
Uh, to RAM disk, I guess?
11:09
Sure.
11:10
Let's do it.
11:11
So, what's interesting here is that we actually hit our peak speed earlier, but then we level
11:20
off to the RAM disk.
11:22
Yeah, but our writes are, like, way better.
11:23
Writes are better.
11:24
And if we look at CPU usage, it will show in here processor 2.
11:29
It will show in here processor time.
11:32
It was almost zero the whole time.
11:34
Crazy.
11:35
So, like, if you're using traditional Ethernet without RDMA, that would be, like, pretty
11:39
high pegged up there.
11:40
So, why don't we do as fast a Windows transfer as we can, then?
11:43
We'll go straight to the RAM disk.
11:45
Sure doesn't take long to move 40 Gigabyte files around like this, eh?
11:48
Even on, like, a relatively slow...
11:50
Relatively slow.
11:51
Oh, 2 gigabytes a second.
11:53
How will we ever manage?
11:55
Painful.
11:57
Yeah.
11:58
So, about the same.
11:59
Yeah.
12:00
I think it's a Windows thing.
12:01
So, pretty much what's going on here is that we have reached...
12:06
Because we've seen faster transfer speeds in Atto.
12:09
So, we have reached pretty much the limit of what Windows can handle for the time being.
12:15
And I don't think it's gonna be in Microsoft's list of high priorities anytime soon to figure
12:22
out how people can use Windows Explorer to copy files at faster than about 4 gigabytes
12:28
a second.
12:29
But that's okay because this was a lot of fun and hopefully you guys enjoyed it.
12:34
If you wanna see what this kind of tech gets used for in the real world, because this is
12:39
just not what it's really for, check out our kind of unboxing of SFU's Cedar Data Center.
12:45
We'll have that linked in the description because that's this kind of technology on
12:50
a whole other level.
12:52
And with that said, I mean, even for someone like us, actually, it could be useful.
12:58
Maybe we could try to rig up like a crazy way to improve the responsiveness of scrubbing
13:04
in Adobe Premiere for the editors or something like this.
13:06
100 gigabit for all the editors?
13:07
Yeah, 100 gig...
13:08
Well, no, but we could have like a 100 gigabit trunk.
13:10
Yeah.
13:11
And then they could all go 10 gig.
13:12
What is it right now?
13:13
It's bonded 10 gigs.
13:15
Oh, like...
13:16
So, you know what?
13:17
Tell you what.
13:18
Let us know in the comments if you'd like to see a video on what we can figure out to
13:22
do with these in an actual deployment.
13:25
So, thanks again for watching, guys.
13:28
If you disliked this video, hit that button.
13:29
But if you liked it, hit like, get subscribed, or maybe consider checking out where to buy
13:30
the stuff we featured at the link below.
13:31
Probably eBay links, I guess.
13:32
Yeah.
13:33
While you're down there, you can check out our merch store, which has cool shirts.
13:34
Like, I don't know how long this one's gonna last, but it's gonna be up there as of the
13:35
time of shooting this pretty soon.
13:36
I think it's funny.
13:37
It's so important.
13:38
And also, we have a link to our community forum, which you should totally join.
13:39
That shirt.
13:40
I know, right?