When everything goes wrong - 6 Editors 1 CPU Pt. 3
Linus Tech Tips
·Linus Tech Tips
·2019-05-06
·
2,963 words · ~14 min read
0:00
now this is the part where we do six video editors with six
0:05
graphics cards but we don't have six graphics cards
0:09
i don't know what accent that is
0:12
now it's not often around here that we have
0:16
a project that's so big we don't even have enough hardware to validate if it's
0:21
going to work but that's what six video editors one CPU has been like
0:26
but the good news is that even though some of the final hardware isn't quite
0:31
here yet we have enough to know for sure if it's
0:36
gonna work so we've got all six monitors
0:39
and the keyboards and mice that go with them for all of our stations we've got
0:43
our pile of high-end graphics cards over here i don't know if you guys can even
0:47
see this but i want one per system over here we've got oh this
0:53
is really cool where is it ah oh yeah here it is we've got the
0:57
replacement for that PCIe 4x interface
1:01
to our daughter board so i believe this is either 8x or 16x which means we
1:06
shouldn't run into bottlenecks for all of our usb controllers and
1:11
we've got Intel optane along with the cards to go
1:16
with that and better coolers for our 28 core cpus
1:21
so today is the acid test does it actually work
1:28
we're going to find out
1:37
smart deploy allows it departments to re-image unlimited computer models from
1:42
one golden image search their library and grab your exclusive free licenses
1:46
worth over six hundred dollars at smartdeploy.com
1:49
Linus so first order of business is to kind of reconstruct what's going
1:55
on in here because we've had to tear apart a lot of it uh over the last
2:00
little while due to other projects like that crazy thing with the four titan v's
2:04
and all that nonsense so let's go ahead and get back to a
2:09
functioning state and then we will go from there now because a big part of the
2:14
goal today is eliminating all of the hypotheticals
2:18
we're gonna do away with things like our gigantic split
2:24
screen 4-way monitor and we're going to actually set up each workstation
2:29
as it is intended to be used with a single monitor and keyboard and mouse
2:33
although it should be noted that we could easily run dual monitors since
2:37
we're just passing through entire graphics cards i gotta get this passenger here sorry
2:42
david we're not quite going back to the
2:46
previous state though one thing that we are going to change
2:50
is this so now instead of a half height like low
2:56
profile card that passes through pci express
2:59
4x we have got this
3:03
honkin sucka that we're gonna go ahead and uh
3:07
i'm not sure which one's the host side and which one is the target side but uh
3:12
we're gonna go ahead and install one of them
3:16
into here you know what i think it actually has dip switches so that you
3:20
can configure it they're both configured the same way
3:23
maybe it doesn't uh
3:27
so on
3:30
on oh my god like it doesn't indicate
3:33
what for six just one two four
3:37
ah we'll figure it out
3:40
okay so i think i figured it out uh these little switches we just need them
3:45
in on on off for this guy and then
3:48
uh hold on off on on for one two and four
3:54
and uh oh okay
3:59
but then six is labeled backplane type
4:03
what it looks like we are running PCIe 8x
4:08
although if we got our hands on another one of these cables we could run 16x
4:14
by using uh dual host
4:18
i think or something anyway
4:21
8x should be enough to start this thing is just ridiculous like this
4:26
connector look at it so even compared to the 4x connector we had last time which
4:30
looks ba that's the 8x connector this is great i
4:35
found the card from last time so instead of this card now we need this card
4:40
so it's basically exactly
4:44
this doubled up
4:48
sheesh now we got to get the other side
4:51
installed in our system here
4:55
that's the only configurable thing on here let's go ahead and install it
5:00
here these are all in
5:03
i've got my NVMe drives here
5:09
so the next thing i need ah there we go
5:12
is the test cart hooked up to the back of the system here
5:17
and then while i do that Yvonne is setting up all the stations
5:22
with their monitors keyboards and mice though it will be a little bit until
5:26
we're actually quite ready for that can i get that big yep
5:32
exactly that and i think
5:36
we should be ready to fire up the system i hooked up the power button this time
5:39
to make my life a little bit easier so i guess we find out if that works right
5:43
now uh-oh
5:47
oh what the crap
5:50
put the motherboard back in without actually hooking the
5:54
motherboard power back up woo okay
5:58
it's a little better so ridiculous
6:03
you have to sign up for an account to get a user manual
6:07
for this card and it wasn't working so
6:10
fine i'm signed up and here we go
6:14
okay i think i did it entirely wrong but i'm not sure why
6:19
because their diagram here doesn't actually correspond
6:24
to the number of dip switches that's on here
6:28
on for one and two off for three and four that is completely different from
6:32
what it says here as far as i can tell on
6:36
for one and two off for three and four i really hope
6:40
this works
6:44
so i think it's configured right now what we should see is this fan spin when
6:47
i power up and
6:51
we got nothing and i don't know why
6:55
okay there's another dip for black backplane type and that's just labeled
6:59
directly on here so let's try selecting a different backplane type shall we
7:04
nothing no power to it at all maybe we have to do the same thing on the other
7:07
side wait oh no i turned this off while i was
7:11
troubleshooting
7:14
this is what happens when you hurry
7:18
not that that fixed it okay now hold on a minute i think i get
7:23
it oh boy
7:26
okay whatever that is probably isn't good stop stop stop stop
7:32
okay so off on off on
7:41
so whatever mode you're using now you have
7:44
to go from the bottom port out into
7:48
the top port and it yeah i don't know it's on so uh maybe
7:54
it's working there it is they have flashing lights on their slots but they're detected so
7:59
presumably they're working uh one way we can tell for sure is just plugging this
8:03
mouse into into one of them wait no i think i already have them stubbed
8:09
yeah i do okay that that's fine that's fine
8:14
oh snap oh crap
8:19
oh shoot i just unplugged this uh that's really bad probably
8:24
oh shoot oh i hope this still works
8:32
don't smell burning it's turning back on and everything
8:36
looks normal hopefully that's fine what just happened there so the next stage
8:40
assuming that we're still past getting this to work stage
8:44
is 10 gig networking for the whole thing
8:49
still works that's good
8:52
oh there it is we're 10 gig all right now i just just
8:56
need to copy our install images off or
9:00
at least one of them because it's time for us to put optane in
9:08
that's how everyone does it that's industry standard
9:12
so i actually only had five usb cards in here before i found another one that
9:16
gives us the full complement of six
9:20
and it's happening i'm swapping out these peasant
9:26
750 series drives for optane the reason we care about
9:31
running optane for our boot drives is that these handle random i o much better
9:37
than traditional ssds and when we're going to have six operating systems
9:41
running off them at the same time that becomes really important
9:45
oh shoot i totally forgot i got to put all the graphics cards in
9:50
let's see x little p
9:53
titan x okay so titan x titan xp 1060
9:57
1070 and
10:00
maybe we'll put another quadro in there i don't know now we've yet to actually boot
10:06
this motherboard with six graphics cards
10:10
so we're in uncharted territory right now this could be the thing that kills
10:14
the entire project
10:18
let's do that one so jank
10:24
jankness is real
10:31
all right let's throw that on there okay
10:34
this last card is our jankiest
10:39
i think it's gonna have to kind of hang out right there
10:42
okay should be fine
10:46
now we fire it up
10:51
and we have a display out so that's a good sign
10:55
that's not a good sign i know we're asking for a lot here
11:00
motherboard but if you could come through for me
11:07
what the hell i don't think i have ever heard a
11:11
hardcore sounding error code like that
11:14
followed by the system just booting
11:18
oh in order to display this message some
11:21
pci devices were set to a disabled state
11:25
oh maybe above 4g decoding will solve this
11:29
problem okay let's see if above 4g decoding is going
11:33
to help us boom there it is i think we're booting
11:38
now did it just turn off now why didn't it pick up my boot device
11:44
well that's interesting our usb key is just not even in here anymore
11:51
okay so uh the only thing i changed was disabling the boot option rom from all
11:56
of our pci express devices which i can't imagine mattered and then i moved our
12:00
usb boot device into a different usb port i wonder if one of our onboard usb
12:04
controllers isn't working because now it's showing up although maybe it
12:08
was just a fluke last time i i don't know so it's here now
12:13
let's see all right
12:16
so now we're going optane on our cache
12:22
and starting the array there's our titan xp
12:26
there's our titan there's our gtx 1060
12:31
there's our quadro gv100
12:35
there's another one and there's our 1070. all six graphics
12:39
cards are here and
12:43
six usb controllers at least
12:47
beautiful and there's our 10 gig card i think we've got everything we need here
12:52
i should go duban show so update time lancho is now complete
12:57
and we can start to dig in and make sure
13:00
everything's working the way we expected now i still only have my four vms set up
13:06
from the last test but this is as simple as making some
13:12
reconfigurations to the CPU assignments
13:15
the GPU assignments and the usb card assignments and bippity boppity we
13:19
should be off to the races so theoretically
13:23
we are going to just fire this bad boy up
13:27
and and go now once we know everything's
13:30
working and it's time to performance optimize we're going to want to do
13:33
things like make sure that none of the vms have
13:37
CPU cores that cross from one CPU socket
13:41
to another because then we're going to take a performance hit there and we're
13:46
also going to want to make sure that every vm
13:49
has a graphics card that's in a slot that is actually also directly connected
13:54
to the CPU that those cores are physically present on that's going to be
13:59
really important
14:07
i think i might have bumped that stupid cable again
14:10
the whole system we started when i bumped that external pci express cable
14:15
well i don't think it's because i bumped the cable i think it's because i don't
14:18
really have the cards screwed in
14:22
okay well while we wait for that we can do the
14:26
uh totally safe
14:30
um plugging in of display cables
14:34
so oh can i just take a moment to point out
14:38
how good i am at plugging in monitor cables without looking
14:45
that no scope monitor plugging in david can i get an HDMI cable from down
14:50
there sorry it's just hard for me to get down
14:53
it's all good
14:56
do you ever just like picture the engineers at Intel and NVIDIA
15:00
watching something like this oh
15:04
okay let's make sure all of our usb doodads are still in here
15:08
one two three and four are up number five's not
15:12
not doing anything did that one fail
15:17
execution error could not find device all right
15:22
we're just gonna do yield reboot and reseed all that stuff
15:26
crap now let's try
15:30
firing this up again and it should be fine so i think what happened was the
15:33
usb controller for this one uh disconnected from the system
15:38
when i bumped the cable and then the one for this one took the
15:42
fifth slot so neither of them was able to start also
15:46
what we can do now is we can come around here and see if
15:51
any of these monitors are lighting up this one just turned blue
15:56
so i think that means it's about to light up
16:00
hey there we go there's one okay we got
16:04
hey one this is not working
16:10
uh do you want to let me know when you see a a pop-up over there
16:15
on the like i'm going to plug in a mouse
16:20
oh that's it yeah now we should be able to figure out which vm is supposed to be
16:25
on all of the other monitors here theoretically this is ws1 but my display
16:29
isn't lit up so
16:33
what i'm hoping is that i'll like move the mouse and it will wake up
16:41
honestly i don't even know what's going on right now
16:44
also the name of this didn't change what is going on here
16:47
yo hey
16:52
wake up like this one just went from having a
16:57
blue light to having no signal and a yellow light as i'm shutting down vms so
17:02
what we can do in the meantime i guess is uh unraid has a new version or two
17:07
actually since uh i installed this so why don't we update
17:12
that and see if everything magically resolves itself the
17:17
plot thickens now that we're rebooting the system to
17:21
apply our update we have power to this keyboard
17:25
my level of pissed off right now is pretty high
17:28
now i've seen monitors that have a bit of a hard time
17:34
picking up signals i have a hard time believing that
17:39
six monitors here all have the same problem but i just say
17:43
no signal so my theory
17:48
about those cards being in order
17:51
whenever it works we're going to label it as quickly as we
17:54
can yeah
17:57
all right next up
18:01
is this guy that's not what i was expecting
18:06
so this is the 1060. which one is that
18:10
who knows so the little game we're playing is
18:14
called find the patterns and so far the only one that has managed a successful
18:19
reboot is number five i believe
18:23
that's the one that i assigned the sound card to
18:28
wouldn't that be funny okay that's actually booting there's four so
18:33
it lit up but has not actually managed to start
18:36
loading Windows yet oh there it goes now one thing john from unraid suggested
18:42
was that we ended up screwing up where our v disks were some of them are on the
18:46
array and some of them are on the cache and he had said he's run into some weird
18:50
i o bottlenecking nonsense with the way that storage devices are virtualized
18:55
that might be explaining some of the finicky behavior
18:58
so we need to make sure all of those are on the octane drives
19:01
number five came up so this one's now consistent which is good to see
19:06
this is getting very interesting
19:09
so six and two are the ones we haven't seen yet
19:13
so everything that we've had working so
19:16
far is working right now
19:21
is six gonna light up for me so six should be this one theoretically
19:26
all right so let's try moving those v discs and
19:29
see if that solves it yeah neither of those are coming up okay
19:35
okay we have some material to work with here i'm gonna shut her down for now
19:39
pia hides your true ip address and allows you to bypass geo restrictions
19:43
and censorship by making you appear as though you're connecting from somewhere
19:47
else and you can use up to five devices
19:50
at once it also helps prevent attacks by blocking unwanted connections it
19:54
autoblocks all traffic if the vpn disconnects and it keeps your data out
19:58
of the hands of advertisers and other activity tracking snoops so go try it
20:02
out at the link in the video description today it's affordable and it works
20:10
you bet you might as well eat some for a bit
20:15
like don't don't die over this project