As many of you probably know,
the last few years have been a heck of a ride
for data enthusiasts,
with high speed storage becoming increasingly accessible.
I mean, something like Samsung's 960 Pro SSD.
It's tiny,
it costs about a third of what I paid per Gigabyte
for my first large boot SSD,
and it's over an order of a magnitude faster
when it comes to real world performance.
But what some of you might not know
is that networking has been keeping pace.
So these bad boys right here
are Mellanox ConnectX 4 cards.
And even though they're already two generations old,
they can reach transfer speeds of 100 gigabit,
or about 12 gigabytes per second.
That is fast enough to download Fortnite's install files,
yes, we went there,
in literally one second.
Holy .
So let's check them out.
Before we get to the new crazy stuff,
let's do a quick refresher
on traditional networking hardware.
So I've brought along a couple of examples here.
These two cards run at gigabit
and 10 gigabit speeds respectively.
So this guy right here is about 10 times faster
than this one.
But other than that, they've got a lot in common.
So they've both got Intel controllers on board,
they both plug into a PCIe 2.0 slot,
and they're both Ethernet.
So,
thanks to their use of intercompatible
communication standards and the ubiquitous RJ45 connector,
they can talk to each other directly
or through a network switch like this one,
albeit only at the speed of the slowest link in the chain,
be it on this one gig card
or through this one gig network switch.
And honestly, either of these, especially this one,
should be more than enough for the average person
for quite some time.
But this isn't average person land,
which sounds like the world's most boring amusement park.
So we've decided to go totally overkill
and take it to the next step,
to greater than 10 gigabit speeds,
which brings us then back to our ConnectX 4s.
So the first difference, the ports.
This beefy looking thing right here
is what's called a QSFP plus connector.
And as you probably figured out on your own,
you can't just plug in a QSFP plus connector
and plug a standard network cable into this port.
And even if you could, well, I guess,
that brings us to the next difference.
The fact that out of the box,
these cards are designed to run not on Ethernet networks,
but on InfiniBand networks.
So even if you could plug it into your network switch,
it wouldn't be able to communicate with it
without some configuration.
And then finally, this one's actually
pretty interesting.
These cards, yes, my friends,
these network cards use a full fat PCI Express Gen 3
times 16 connection.
That is the same as your graphics card.
And they actually need it,
probably more so than your GPU,
if all you're doing is gaming.
So then with that in mind,
we will not be using just your average gaming machine
for our testing.
So on one side of our link,
we've got Intel's flagship 18 core processor
with a Rampage 6 Extreme motherboard, 128 gigs of RAM.
Mm, yeah.
And then in the other corner,
we had to slum it a little,
with a 16 core 7960X and an ASUS X299 Deluxe,
but then with the same amount of RAM.
And of course it's RGB across the board.
Now, the reason that we're using the X299 platform
with Core i9 processors is that we need to make sure
that we have enough PCI Express lanes coming,
directly off the CPU.
So 44 should give us enough for 16 lanes for networking,
16 lanes for our quad NVMe storage devices,
and then, you know, some leftovers for the graphics card.
So then, for our NVMe storage,
we scraped together four Samsung 960 Pros for our first one,
and then four Corsair MP500s for the other,
with both of them running,
with quad,
SSDs in RAID 0.
Today's video is about how to go fast,
not about how to put on your seatbelt.
This is like hilarious to me.
I never thought I would see the day
when the GPU in a gaming rig is the lowest priority,
like tier PCI Express device
sitting at the bottom of the board.
Anyway, for our OS,
we had to go with Windows Server 2016,
because as much as we wanted to try out Windows 10,
and Pro for Workstation,
which is supposed to support RDMA,
the tech that allows for these super high speed transfers,
it just didn't seem to be working for us for some reason.
All right, now, at this point, before we go further,
wanna give a big shout out to the guys over at Mellanox
for hooking us up with these 100 gigabit cards,
as well as a pair of 100 gigabit capable,
passive copper direct attach cables.
Wow, that just hit the bench.
Oh!
So if you wanna try this out at home,
these cards are actually available on eBay
for like two, 300 bucks a pop,
and then you'll pay about $60
for a three meter cable like this one.
It is worth mentioning though,
that if you're planning on running anything
further than five meters,
you have to use an active fiber cable,
which could cost upwards of $2,500 new.
So...
Damn!
All right, moving on.
Configuration then is our last step.
So while Jake does that behind me here, thank you.
Jake.
Let's talk about some of the technology
behind this networking magic.
So these cards are designed for use
with two different network fabrics,
InfiniBand and Ethernet.
And what makes InfiniBand special
is that compared to even the sub millisecond latency
of a typical Ethernet network,
InfiniBand networks can have less than 25% as much,
making them suitable for use cases
like over network storage access
and combining the processing power
of multiple servers,
just like in a data center or supercomputer.
And this is cool.
When you configure InfiniBand correctly,
it also forms what's called a lossless network,
meaning that packet loss should basically never happen.
For compatibility though,
we're going to be using them in Ethernet mode
alongside a technology called RDMA
or remote direct memory access.
When you put these together,
the setup is called ROCE or RDMA over conversion.
That's a network-converged Ethernet.
Now regular Ethernet implementations
require a lot of hoop jumping
in order to transfer data
as any information sent must be first moved
through the transport protocols driver,
then through sockets
before it can reach the applications memory,
eating up CPU cycles and increasing latency in the process.
However, with RDMA,
the network adapters are able to access data
directly from application memory,
offloading much of the processing from your CPU
onto the actual process, which is called RDMA,
and the process is called GPC.
And that's basically it.
That's basically it.
Processor that sits right on your network adapter. So these are known as zero copy transfers and they allow for
Incredibly fast transfers that are no longer limited by CPU processing power. Pretty dang snazzy. So are we ready to go?
We should be. Each card has an IP and a 50 Gigabyte RAM disk
So we should be able to do some pretty quick Windows transfers. So are we going directly from RAM disk to RAM disk right now?
Uh, so I think what we'll do, we got to see make sure it's working first, right?
That would be good. I mean that's supposed to be your job, but. Fingers crossed
You never, the second you try to do a networking demo, like
We got our 40 Gigabyte text file. What? 36 or 37 gigabytes. Why?
Why even? It's just a big file, okay?
Oh, that's not bad. That's not 10 gig, or 100 gig.
So we've got just over two gigabytes per second. So we're reading from that system's
NVMe array. Okay. And then we're dumping to the RAM disk on this system. So let's try going the other way.
So now we can go RAM disk to RAM disk. Is that right? No, that's still the same thing, but just the other way around. Whoa!
Just shy of four gigabytes per second.
I mean, let's put what just happened there in context. That is a 40, well,
just shy of 40 Gigabyte file.
So like, okay, DOOM. DOOM on PC is like 60 gigs.
And it takes, you know, probably for your internet connection at home, like what, an hour to download and install? That just happened in real time.
Okay, what's next? RAM disk to RAM disk? I gotta check. I think one of our RAM disks isn't working. I might have broken something.
Okay, so check this out. We've got the RAM disks working, but for now, this is just another quick benchmark. So when we aren't limited,
by the overhead of Windows File Explorer and Windows File Transfers,
if we're just using a straight disk performance benchmark,
I wanna show you the kinds of numbers we're looking at.
So anyone who's familiar with Atto is gonna already know
that this is freaking nuts.
At two kilobyte sizes, we are already seeing speeds in excess
of 100 megabytes per second, because we are actually reading
and writing off of the four Corsair SSD array that's on the iPad.
So we're gonna have to figure out how many megabytes per second we're gonna have to write off of the other machine.
This is stupid.
It's pretty cool, actually.
It's really cool. And it's getting stupider as time goes on.
We are already hitting 600 megabytes a second writes.
Oh, that's weak, man.
At 16 kilobytes, though.
Just wait.
That's the key. And it just keeps getting crazier.
We're at one Gigabyte per second already.
Three!
Doubled!
Oh, wow.
Oh, man.
We just cracked four gigs a second.
We just cracked five gigs a second!
And this is on a remote machine.
This is not a local array.
This is over the network.
This is a slow test.
Yeah.
It's a really fast, slow test.
Yeah, it takes a while.
A really slow, fast test.
It's in the billions of bytes.
Billions of bytes.
It's, like, not even readable anymore at that point.
No, it's just like, what is going on?
How do I maths?
Oh, meanwhile.
Are we there?
Just cracked it.
10 gigabytes a second.
Okay.
So, what's our next test?
Uh, to RAM disk, I guess?
Sure.
Let's do it.
So, what's interesting here is that we actually hit our peak speed earlier, but then we level
off to the RAM disk.
Yeah, but our writes are, like, way better.
Writes are better.
And if we look at CPU usage, it will show in here processor 2.
It will show in here processor time.
It was almost zero the whole time.
Crazy.
So, like, if you're using traditional Ethernet without RDMA, that would be, like, pretty
high pegged up there.
So, why don't we do as fast a Windows transfer as we can, then?
We'll go straight to the RAM disk.
Sure doesn't take long to move 40 Gigabyte files around like this, eh?
Even on, like, a relatively slow...
Relatively slow.
Oh, 2 gigabytes a second.
How will we ever manage?
Painful.
Yeah.
So, about the same.
Yeah.
I think it's a Windows thing.
So, pretty much what's going on here is that we have reached...
Because we've seen faster transfer speeds in Atto.
So, we have reached pretty much the limit of what Windows can handle for the time being.
And I don't think it's gonna be in Microsoft's list of high priorities anytime soon to figure
out how people can use Windows Explorer to copy files at faster than about 4 gigabytes
a second.
But that's okay because this was a lot of fun and hopefully you guys enjoyed it.
If you wanna see what this kind of tech gets used for in the real world, because this is
just not what it's really for, check out our kind of unboxing of SFU's Cedar Data Center.
We'll have that linked in the description because that's this kind of technology on
a whole other level.
And with that said, I mean, even for someone like us, actually, it could be useful.
Maybe we could try to rig up like a crazy way to improve the responsiveness of scrubbing
in Adobe Premiere for the editors or something like this.
100 gigabit for all the editors?
Yeah, 100 gig...
Well, no, but we could have like a 100 gigabit trunk.
Yeah.
And then they could all go 10 gig.
What is it right now?
It's bonded 10 gigs.
Oh, like...
So, you know what?
Tell you what.
Let us know in the comments if you'd like to see a video on what we can figure out to
do with these in an actual deployment.
So, thanks again for watching, guys.
If you disliked this video, hit that button.
But if you liked it, hit like, get subscribed, or maybe consider checking out where to buy
the stuff we featured at the link below.
Probably eBay links, I guess.
Yeah.
While you're down there, you can check out our merch store, which has cool shirts.
Like, I don't know how long this one's gonna last, but it's gonna be up there as of the
time of shooting this pretty soon.
I think it's funny.
It's so important.
And also, we have a link to our community forum, which you should totally join.
That shirt.
I know, right?