Building the $100,000 PC Pt. 2 - SO MANY PCIe CARDS
Linus Tech Tips
·Linus Tech Tips
·2019-05-06
·
2,969 words · ~14 min read
0:00
so you might have noticed it's been a while since i've talked about the oh
0:05
the hundred thousand dollar pc and i'm approximating the cost because i
0:09
actually don't know exactly how much it's going to end up coming to yet
0:13
because they're still a few things that are to be determined
0:19
about the final configuration
0:22
but there's a good reason for all of that i've been waiting on some really
0:26
important hardware to continue my testing so right now we've got a problem
0:31
with the current configuration this motherboard can accept a measly
0:36
seven pci express 16x devices i mean
0:39
what kind of chump nonsense is that for a hundred thousand
0:44
dollar computer that i aim to have six video editors editing 8k footage off of
0:50
at the same time so i got in touch with uh actually a
0:54
couple of really special companies one of them is thermaltake who
0:59
sent over a bunch of their pci express extenders i'm going to use these
1:04
to test running a bunch of graphics cards off of this thing without water
1:08
blocking them all up beforehand which is what i did last time
1:11
which was super stupid because then when i had issues it was it was a nightmare
1:15
and this this
1:18
is actually a bunch of boxes from a company called one stop systems who if
1:23
you're not in the enterprise space you've probably never heard of but if
1:27
you are even then you might not have heard of them and they make
1:32
pci express extension solutions
1:35
allowing you to take a single slot and adapt it to in this case
1:42
eight slots how does that work
1:46
well we're gonna find out not cheaply though
1:58
thanks to videoblocks for sponsoring this video they're a great
2:01
subscription-based resource for downloading stock footage video and more
2:05
check out the link below to learn more so the funny thing about these guys was
2:10
when i got in touch with them um the bloke that i spoke with was actually
2:15
a fan of the channel he was like super into working with us and he was like
2:19
yeah let me let me carry this up the chain and then i i spoke with his
2:24
manager and she wasn't as familiar personally but then her husband was
2:29
apparently super into Linus tech tips and she was like okay yeah no you know
2:33
this sounds this sounds great we'd love to work with you and then
2:37
i followed up and they basically said yeah everything's shipped
2:41
but i hadn't actually worked out what i needed or really talked to them about it
2:45
much yet so i'm not a hundred percent sure
2:49
what's here yet so we're gonna find out together
2:53
i did know this board was coming so they've actually got a bunch of
2:57
different pci express expansion products even you can get like chassis that allow
3:02
you to plug something from the back of one computer into the back of another
3:07
one that looks like a computer but it's actually just a box full of pci express
3:10
expansion or you can get a bare bones board which
3:16
it would appear is what they've sent to me but i'm not a
3:20
hundred percent sure how or if
3:24
any of it is going to work so this guy
3:27
right here is their hiv 35 x4 this is a
3:30
pci express x4 2
3:34
cable adapter so this is going to go into our
3:38
system and is going to adapt a pci express slot
3:43
to a cable that we can plug into the back of the system this is pci express
3:49
over a cable i'm not talking like you know Thunderbolt i'm talking pure full
3:53
capability pci express trippy stuff
3:57
right look at this connector
4:02
oh that feels good we're turning the one pci express slot
4:06
on the other end here into one two three four four eight more
4:11
eight more pci express slots five eight
4:14
pin power connectors and a 24 pin what
4:18
are they expecting you to plug into this thing honestly what they're doing
4:23
is not that unusual
4:26
in fact your motherboard you watching
4:30
right now your motherboard probably has similar technology because you're
4:34
probably aware already if you've followed our CPU reviews that especially
4:37
mainstream cpus don't have a ton of pci
4:41
express lanes on them so then how exactly is it that you end up with you
4:45
know four 16x slots and a bunch of PCIe 4x
4:50
slots when your CPU only has like 20 to 24 lanes
4:54
well the way that they do that is using what's called a plx chip so these are
4:58
effectively think of them as kind of like a network switch where you take one
5:02
Ethernet connection and you split it out to eight computers for pci express so
5:06
this is going to be a very high end pci express switcher chip and then you're
5:10
going to find those same kinds of adapters on here so that allows all of
5:14
this traffic to share this one interface
5:17
now to be clear you're not going to be able to connect eight 16x
5:21
graphics cards and get the full performance out of all them at exactly
5:25
the same time but i'm planning to use it for usb 3. something where with six
5:29
editors not everyone's going to be slamming it at exactly the same moment
5:33
so it should work fine
5:38
and just knock on wood till you until your knuckles are raw
5:42
right there was another piece of hardware that i needed to continue my testing so we
5:47
only have a couple of optane pci express ssds in the building so what i wanted
5:51
was to swap out my 750s for these and have a look at what vm performance looks
5:56
like running on optane raid1 the other
6:00
thing i was waiting for cable mods sent over some pci express power
6:04
connectors that i'm going to need to extend the cables for these graphics
6:08
cards that i'm going to be running on extensions
6:12
i just found the most perfect spot for it you guys never would have believed
6:17
that i didn't know exactly what i was doing here so
6:20
i've got the the host adapter installed on the other
6:24
side i throw in a second power supply here and then check this out
6:29
oh no
6:33
now that could work man we'd have to like pull this panel
6:37
off and maybe like laser cut them or something but we could have all our usb
6:41
ports right on the back of the system like that anyway for today though this
6:45
is just this is just a dry dry run i'm going to bring it around the other side
6:50
so with this expansion daughter board we're going to have a total of
6:55
six remaining because i'm going to take up one of the PCIe slots with the host
6:58
card plus eight so that is 14 pci
7:02
express slots plus all the ones that i am adapting
7:06
from the u.2 plugs on the motherboard so
7:10
uh
7:13
is 18 pci express slots
7:18
just the right number creep 100 okay
7:21
let's get our PCIe power
7:25
and then let's hook up some graphics cards here
7:28
shall we we're just gonna do four graphics cards for today so some of the
7:32
editors are gonna get full 16x slots for their graphics cards and others are
7:37
going to be stuck with a little bit less but for the purposes of what we're doing
7:41
that's not going to affect performance in any meaningful way
7:45
so we're just going to pick some random slots here
7:48
and install some gpus baby so those will go there
7:53
sure so that'll go there
7:56
sure yeah um
8:00
i mean is it there does that work is that kind of reasonable we never claimed this video
8:06
wasn't gonna be madness you knew what you were clicking on
8:10
okay leaning tower of titans uh right
8:15
yeah what could go wrong
8:27
oh there we go okay that's one
8:31
i mean should we do a quick test to make sure this even works before we go any
8:34
further yeah okay all right
8:38
all right let's do it is this on
8:42
what the heck oh it turned off
8:46
let's give it one more try because if there's one slot that doesn't like a
8:50
graphics card or something like that being in it that's going to be a big
8:53
problem because i need to fill all of them with graphics cards ee
8:59
you know what i think it might be is our one stop systems expansion card
9:03
because i don't have the power supply plugged into the
9:06
target end of it yet so it might not like something about
9:10
what's going on here so let's try it with just the graphics card so we're not
9:13
changing too many variables at once here there it is
9:18
okay all right so while this is booting up
9:22
we need to go get ourselves another power supply and it can't be like some
9:27
wussy power supply it's got to have
9:30
i got to find out what kind of connectors these are these pci express
9:34
or the uh oh
9:37
oh crap they're not PCIe
9:42
oh man it's gonna be ghetto
9:46
so this is interesting of our five ax 1500 eyes
9:51
i apparently have three of them and then we also have a 1600i that Jake
9:55
apparently has checked out long term what does he what does he need a 1600 on
9:59
his test bench for okay let's go let's go
10:03
get one of the three axis that Jake has
10:07
so i have one on my bench i have one we have like five so yeah so i have one in
10:11
my no no i have one in my rig at home i have one in the desk pc
10:16
why would i not because it's so stupid you're stupid why do you have so much
10:19
crap here can you clean this up i mean it's working out for me right now
10:22
but in general okay
10:27
what did he do can you believe what Jake did to this
10:31
power supply you see this is why this is why i rage out at people for
10:36
like the way that they treat the hardware around here sometimes
10:40
what the whoa
10:44
you say this like you've never seen it before you're literally the only person
10:47
who's ever used it
10:51
and naturally the eps connectors aren't even in here thanks Jake
10:56
should i just fire him let me know in the comments
11:00
new plan Jake did find another 1500i
11:05
so we're just gonna use that this is the beauty of modular power
11:08
supplies because like how else would you have ended up being able to
11:13
frankenstein a regular desktop power supply to power this
11:18
weird board all we got to do is uh
11:22
what the hell is this
11:28
Jake just peel it off well i mean it makes it
11:33
look like there's something wrong with it
11:36
why did you put so much tape on it
11:40
it's fine
11:45
now i have no idea
11:48
what triggers this thing to turn on
11:51
here we go yeah let's get a bit of a better look at whoa
12:04
all right that's all looking peachy
12:10
uh editor don't include that these guys were very gracious to send us this stuff
12:14
and i don't want them to see that the bloke that i spoke with was actually a
12:17
fan of the channel now i do want to know if it's working so
12:22
here why don't we grab our octane drives and throw those on there and see what
12:26
happens obviously with something like a high-speed storage device we're not
12:29
expecting to get our full performance out of it in this kind of a situation
12:32
but i just want to know if it's working at all this is terrifying everything about this
12:38
is wrong let's take a picture for instagram
12:41
there they are there's the optane ssds just like that
12:46
basically plug and play you just have to make sure you actually plug all the
12:50
things may not look like much but she's got it where it counts kid right
12:55
gorgeous so our optane drives are being picked up
12:58
by the os nvme0n1 and
13:02
this is what i really need to know this is going to tell me if this whole thing
13:05
is going to work or not because if i can't pass through the controllers
13:09
individually then there's i'm not going to be able to get everything hot
13:14
pluggable look at all of these plx bridges here
13:20
that is madness
13:23
so the update is mostly bad news there is what's called an acs override
13:28
patch that you can enable to try to break
13:32
individual items that are in a single i o mmu group out into their own iomu
13:37
groups but as far as unraid knows the the code for the patch has been broken
13:41
since Intel sky lake and it doesn't seem to be working here
13:46
obviously this is much newer than sky lake but they have suggested that if we
13:50
shut this baby down pull the card out and put it in a
13:54
haswell or older system and see if that
13:58
works that maybe they could try fixing the
14:01
patch to work on sky lake and then it might work in this system so
14:05
that's our that's our first step here uh
14:10
well we're not running this for very long so we're just gonna
14:14
put that there yeah
14:18
okay yeah that seems pretty stable
14:21
all right so this is where we find out if the hack will potentially work with
14:28
our pci express splitter device one thing that's nice is how plug and play
14:32
this thing's been these ssds are both showing up perfectly just plugged into a
14:36
consumer board so that's pretty cool and honestly we
14:40
should still get some use out of it even if it just means that all of our usb
14:44
cards that we have plugged into it can't be passed through the vms it'll still be
14:48
nice to have all of those ports okay so i can't reach my contact at unread so
14:53
what i'm gonna do then is just throw some usb cards in here and see if
14:58
we can pass one through to something everything continues to get more ghetto
15:04
now we have ssds hanging off here
15:07
aren't they beautiful our renaissas usb controller is showing
15:12
up here that's one of the ones that's plugged into the daughter board but we still
15:17
don't know if the override worked so let's go ahead and let this populate
15:22
remember this doesn't tell us if we can get it working this tells us if maybe
15:28
we could get it working oh
15:32
this looks like a very big group so this
15:35
is all the plx style no they're all in the same group
15:41
damn it okay
15:45
it's not all bad we still get the benefits of using
15:50
a pci express switch for all our usb devices so everything that we do hook up
15:56
will be better managed and the whole thing does
15:59
work really seamlessly i had just really hoped for hot plug
16:04
and it looks like that might be out of my grasp however cool this hardware
16:09
might be
16:12
have you ever needed a beautiful shot of like mountains or wild animals or even
16:17
something like an astronaut in space for your video but you didn't have the time
16:21
or budget to get it yourself well videoblocks has you covered with
16:25
unlimited downloads through their member library that contains over 115
16:30
000 hd and 4k clips along with after
16:34
effects templates motion backgrounds and more
16:37
an additional marketplace for other footage is also available so if you're a
16:41
member you say 40 and you get to support the artists
16:44
because videoblocks doesn't take a single cent all the footage on
16:47
videoblocks is royalty free which means you can keep and use the clip as many
16:52
times as you want forever and new footage gets added regularly so you'll
16:55
always have fresh new content to check out and use get all of this with an
16:59
unlimited member planet videoblocks today at the link below
17:03
so thanks for watching guys hopefully you enjoyed coming along for
17:07
what would normally be like a behind the scenes type of
17:11
troubleshooting for a project like this if you guys disliked this video you can
17:15
hit that button but if you liked it hit like get subscribed maybe consider
17:19
checking out where to buy the stuff we featured at the link in the video description as long as you don't plan to
17:23
use pci passthrough for the individual things you have
17:27
plugged into it for each one of your vms
17:32
just had a new thought wait
17:35
what if every single slot got its own daughter board with the GPU
17:42
and the usb on that okay that's probably a project for
17:46
another day let's just get the video editing working off this one system
17:49
anyway uh subscribe and check out our merch
17:53
and uh we have that linked below and also our forum you should join that