I almost cried... - 6 Editors 1 CPU Pt. 4
Linus Tech Tips
·Linus Tech Tips
·2019-05-06
·
3,498 words · ~17 min read
0:00
now we've yet to actually boot this motherboard
0:05
with six graphics cards this could be the thing that kills the
0:09
entire project
0:17
the whole system restarted when i bumped that external pci express cable my level
0:22
of pissed off right now is pretty high that's not what i was expecting
0:30
story blocks video offers you studio quality clips at a fraction of the cost
0:34
check them out today at the link in the video description
0:37
so we found a pattern vm2 and vm6
0:42
are not working and honestly i'm at the point now where all i can really do is
0:47
give up on two and six in their current configurations and we're gonna try
0:50
swapping out the graphics cards because we finally got an error and it was
0:55
something to do with the onboard audio controller for the graphics cards makes
0:59
no sense never seen this before on any pascal card especially a founder's
1:03
edition so here's a titan v which is a good
1:06
choice because this is what we'd like to use for the final build anyway because
1:10
it has single slot i o although we have to convince NVIDIA to send us six of
1:14
them and they're like this is not a supported configuration for the titan v
1:19
i'm like yeah yeah i know people are gonna know that
1:22
anyway our quadro systems are working fine too so we're going to swap out our
1:26
1070 and our titan x which conveniently are right on the top of the pile here
1:30
with these two puppies
1:38
i could do that a thousand times and i'd never stop dropping things and i'd never
1:41
stop wearing sandals okay i think it's booted
1:46
so it picked that up so that's fine and six
1:50
needs a GPU as well and it picked up that one which might be
1:55
fine oh sound card d9
1:58
okay new update we swapped in two new graphics cards and then we did end up
2:03
having to disable their audio controllers which is weird but then one
2:07
of them worked so we've actually got five lit up right now so it's just the
2:11
last one that's not firing up so i'm wondering if maybe it's just like my
2:16
DisplayPort to HDMI cable or something no i don't think so because num lock's
2:20
not working on here so we're gonna throw an rtx 2080 in here
2:25
because nothing's worked yet
2:28
i really doubt it's gonna fire up though no
2:33
yeah it lit up all right let's try the other five no
2:37
i'm not actually doing it i'm not actually doing it i want to see all of them up
2:41
i know it's normal when you're installing display drivers for the screen to go blank but i'm just
2:46
very on edge about this
2:50
num lock's not working you have got to be kidding me
2:54
yeah five didn't fire up why
2:58
what is the symbolism okay well six fired up but it's in
3:04
Windows recovery nonsense mode
3:09
all the usb devices are there and
3:17
and now it's off bastard it's trying so hard
3:26
you're just capturing my sadness over here yeah thanks
3:34
so after sleeping on it i thought maybe we need to get a little bit back to
3:38
basics here i pulled out all the graphics cards and
3:43
re-ran the utility called hw lock
3:47
so something's going on with pci express that's causing some kind of nonsense in
3:52
the system it was causing hw lock to crash and it was causing our vms not to load
3:56
now with our new BIOS which took forever to even start posting now what we can do
4:02
is start to slowly re-add hardware
4:05
until we reach a problem
4:09
everything's in it should be fine
4:13
NVIDIA confirmed the shipment of our six titan v's for this project
4:19
there seems to have been a mistake because i never had the briefing call
4:22
where i you know came up with some real workloads to use them with
4:27
you know like realistically they were going to support this cool project they
4:31
just jumped the gun a little that's all
4:34
now i do want to get in here and turn on above 4g decoding and it's weird because
4:38
i've had a really hard time getting into the BIOS ever since i updated it like i
4:42
just go straight from this pre-BIOS loading thing to
4:47
boom unraid's firing up
4:50
oh there's a ps2 port maybe i should plug into that
4:55
the saga continues
5:00
it's erroring out on probably pci resources allocation
5:05
but instead of letting me go into the BIOS and reconfigure that it's just hard
5:10
resetting you
5:14
bastard it's funny i remember sending ASUS a
5:17
memo about this board saying this is the best behaved dual socket board i've yet
5:21
seen to issue a retraction
5:26
i mean it's not garbage it's like really cool fancy stuff but like
5:31
i need it to work come on
5:35
just output
5:44
new plan we pull out all the gpus again
5:48
and we reconfigure the pci 4g decoding
5:51
thing with only the onboard connected
5:54
go just go
5:59
instead of showing me this whole bar and showing me all the postcodes it just
6:02
shows me the first one it's not working correctly
6:15
turned off
6:18
come on
6:24
yes okay
6:27
so we are on the latest BIOS
6:31
let's just go through here and see what all we need to change
6:34
disable the serial port CPU storage nope
6:38
on board lan disabled
6:41
ASUS engine booster above 4g decoding on
6:46
okay see i don't know what aspm does we don't need any of these operons
6:50
never extract csm i mean maybe if we just start
6:55
disabling devices that'll help
6:58
this is it everything's reconfigured which means we
7:02
need to plug these SATA devices into the other controller
7:07
now we are going to add graphics cards
7:12
let's just throw them all in i mean why not yellow it again that worked out so
7:15
well for us last time guys how many times do you live
7:22
i don't make everything hard i mean just ask the audience i'm sure tons of them
7:26
are flaccid right now
7:30
okay so we're back to square i don't know what square this is but
7:34
we're back to actually being able to boot on raid see
7:40
a display off of the onboard vga and we have all six graphics cards installed in
7:45
the system what square is that
7:48
hip square you know like it's hip to be i'm gonna go get my laptop
7:53
so here's what we're gonna do we're gonna edit
7:57
all of our vms to
8:01
not have oh well they're gone so that's convenient okay they don't have their
8:06
usb controllers because those aren't plugged in
8:09
so can we start them which one's this four let's start number
8:13
four i expect this to turn on
8:18
i'm using the power of positive speech oh
8:22
crap domain id3 is tainted this is not a good
8:27
indicator right now because if it's the graphics cards that are our problem we
8:31
are a lot further away from this project working than i hoped
8:35
okay so it fired up but i don't see anything here yet which is also
8:40
not a good indicator because remember four worked before
8:45
maybe turning my mom's around again yeah we can try that
8:52
i don't think so though oh boy
8:56
okay good work david uh okay
8:59
starting numero 6 internal error yeah once again it is
9:04
flipping out about the audio crap that's
9:07
bad what the hell
9:10
okay i'm really screwing this up right now
9:13
okay let's try two just prefer the lulls
9:17
device eight f zero zero whatever the crap what do you
9:21
mean af whatever what what is this okay did it change the is that the GPU
9:26
it changed my GPU ids oh crap so ah man which one it okay
9:31
forget it we're just going to create a new one however much memory it doesn't
9:35
matter right now let's just get it working primary disk location manual
9:39
graphics card titan v create
9:42
all right let's see if that worked oh maybe actually yes
9:47
so there it is keyboard and mouse obviously aren't working right now
9:51
because i don't have my usb controller plugged in but
9:54
okay well let's keep going then i guess
9:58
uh oh dang it well
10:02
two's not up unless it is when i turn the monitor
10:06
back on new plan
10:10
we're gonna try two and six as the only things plugged into the system
10:19
okay i got
10:24
ls toppo to run so the difference between our new ls toppo run and the old
10:29
one is now i have all six gpus installed
10:34
in the system so what that allows us to do
10:38
if this image viewing program wasn't completely worthless
10:42
sure good enough what this allows us to do is figure out
10:47
which gpus are the problem so we can see here
10:52
six is our titan x and two is our titan
10:56
v so then if we go into system devices
11:00
this tells us that 1d81
11:04
right here and then our titan x
11:08
those are our problem gpus okay
11:12
so this confirms a few things number one is it confirms
11:16
that my map of the pci express slots is probably right because slot two and slot
11:21
three which is where those are one of them should be CPU two i believe that's
11:24
slot two and one of them should be CPU one i believe that's slot three and
11:28
actually i checked that that's right so it confirms that
11:31
and it confirms why i think
11:35
four was fine and six is bad both of them are the only subordinate
11:40
devices on a plx bridge
11:43
and those are the ones i'm having issues with so
11:48
it is my belief that either the solution is UEFI tuning so i've drafted an email
11:53
to ASUS explaining that these are my problems
11:56
or a new UEFI i remember how i started that sentence
12:00
my brain is very tired um we're i'm
12:04
gonna send them an email hopefully it'll be coherent in the meantime though
12:10
now what i haven't been having problems with
12:14
is my u.2 to pci express
12:18
x4 adapters
12:22
like this really is just for testing i intend to populate these
12:27
two now graphics card can draw up to 75
12:32
watts through the pci express slot itself so obviously
12:36
this SATA to power adapter thing like look at these pin or wires on this thing
12:41
but we are not going to be loading this thing anyway i just want to see them
12:45
work i don't need to run doom for 24
12:48
hours straight or anything like that all right so
12:53
here goes one and here goes
12:58
two
13:01
so bad so i'm going to put this
13:05
here so we're only getting four lanes but
13:08
that's four lanes PCIe gen three like that's not bad
13:12
does anyone else have really mixed feelings when they
13:15
you know finally reached the end of a problem like where part of them is relieved and
13:20
happy and the other part of them is just like pissed off that it took so long
13:24
if the cringing was real last time
13:28
now it's reached a level of hyper realism that we thought only possible
13:32
with rtx okay so i need this
13:37
to reach in through here
13:40
ah yeah
13:46
this seems fine so step number one is finding out if my
13:49
gpus are even working in those slots i mean
13:53
theoretically there's no real difference between this
13:57
and this they're just pci express over a cable but you never know
14:01
titan v boom there it is and one b zero zero
14:05
this one's the same so that's our titan x so they're there
14:08
so let's try starting it started right away
14:12
i don't know if we have a display output though
14:15
well would you look at that not only is it working but what's really
14:20
odd is it's running in the correct resolution let's see if it survives a reboot here
14:25
need i remind you guys that this is the GPU that is plugged into an extension
14:30
plugged into a daughter board is vertical in the case and completely
14:34
unsupported by anything like if this works it's a small christmas miracle
14:39
hey are you working or what
14:44
same behavior
14:48
let's try the titan v let's see what happens oh oh oh oh oh
14:53
okay yay obtain it's going fast at least wow
14:56
it's going really fast now it slowed down and of course it's 100 complete and
15:01
yet here we are progress bars man how are they this broken
15:06
i'm gonna see if i can reboot that other one in the meantime
15:12
so i ran ls topo again and let's have a look at what our topology is now so
15:19
here's those two devices we're still having issues why
15:24
why those two specifically we're actually moving backwards we're losing
15:28
vms now four no longer boots i'm gonna try uh
15:32
three and one as well one's up five's up
15:37
three's up i don't know if they're gonna stay up
15:40
five three and one let's try stopping them
15:45
and then we'll try rebooting them that's the real test
15:48
and none of them have a signal
15:58
okay then okay
16:03
i had really hoped these adapters were going to be the solution
16:07
we are going to put our gpus
16:10
in this thing like there's nothing
16:14
that would lead me to believe that this will be better than using those NVMe
16:18
adapters blah all right maybe we can shove this
16:23
over to get that HDMI cable in there yep there we go okay so that's in let's go
16:27
ahead and throw our expansion card doodad
16:30
back in here oh ballsack which slot was i think it's the bottom one that i use
16:35
okay ah
16:38
why is this keyboard powered up oh these cards have power even though
16:42
they don't have a data signal did that just shut down and reboot
16:47
sounds good okay
16:50
that's the pain train it's leaving the station
16:56
so the configuration's a complete cluster at this point the two that i'm
17:00
really worried about are the titan x in vm number two and the titan v and
17:05
number six so why don't we do those first all right so they're both up but
17:09
let's find out if they're going to stay up this has a blinking cursor so that'll
17:13
tell us if this freezes mind you if it behaves like before it'll probably just
17:17
go away did that just go away
17:20
oh whoa whoa whoa whoa whoa whoa whoa whoa
17:24
this one's up i'm gonna shut it down and reboot it
17:29
is it doing a graceful shutdown okay so this is still working i can tell
17:34
because the system clock is still right this one finished its update while i
17:39
wasn't looking we're gonna rerun our command here
17:43
did we just crash oh yeah that bm's gone
17:49
we're not going hail mary anymore we're not going for glory so
17:53
we're gonna go back to basics and we're just gonna start with one then we're
17:58
gonna put in two and three and four we're gonna see where we crap
18:03
out and see if we can find a pattern here so after sleeping on it i thought
18:07
maybe we need to get a little bit back to basics here now what we can do is
18:11
start to slowly re-add hardware
18:14
until we reach a problem the is that
18:20
this is honestly starting to feel a little bit like a journal that i'm
18:23
keeping on a on a desert island where i'm the only survivor or some garbage
18:27
like that like so while the camera was off last night we
18:32
went back to unraid 6.5.3
18:36
just in case the newer Linux kernel with all the spectre and meltdown nonsense
18:40
that's going on was was causing some problems that was a suggestion from the
18:43
unraid guys and i went down to one card and it was working
18:48
then i added a second card and boom already flaky behavior
18:52
i don't even i don't even know if that was one of the slots i was having trouble with before so this is this is
18:57
like a big problem and now that i had all the mapping information i was
19:02
definitely assigning gpus to the correct
19:05
CPU course so we weren't having to cross between the cpus in order to get to the
19:10
pci express lanes so that got me thinking
19:15
what if the problem is hardware now ivan
19:18
is a competent pc technician
19:21
but he doesn't have experience with lga 3647
19:26
and it occurred to me that he was the one who installed my
19:30
cpus so that got me thinking
19:34
what if we just have a bent pin in the socket
19:37
so
19:42
it was a little piece of debris in the socket but
19:46
it seems to be clear now all right so there's one
19:50
now let's try the other one no obvious issues
19:54
i should put them in the opposite sockets just to see if like
19:58
maybe i start having problems with like different slots or something
20:04
you know
20:08
well so much for that theory i mean you never
20:11
know when you know reseeding can just
20:16
magically solve something though so still
20:20
i'm not very hopeful but you never know
20:23
so this is interesting we just got a kernel panic again
20:28
this time it's CPU one
20:31
processor id one it is still possible
20:36
that one of my you know ten thousand dollar cpus is
20:39
defective but i'm really leaning towards like motherboard
20:44
BIOS optimizations that are
20:47
wonky at this point we're not getting away from these hard resets
20:52
so my idea just now was do a fresh install to that vm and see if we can
20:57
avoid whatever devices it's trying to install drivers for
21:02
so that all is behaving completely normally
21:07
wouldn't that be a weird thing
21:11
if it spazzed out over the copied vdisks but then
21:15
getting ready i don't know about you computer but i've
21:19
been ready for a while here
21:22
hey uh
21:28
what what's going on
21:33
hi
21:40
we have a Windows desktop
21:44
on machine number two
21:47
how did this happen so this is how it's going to be for now
21:53
we finally got machine 2
21:56
working we think but Windows updates
22:01
are not having any of it storyblocks video gets you studio
22:05
quality video clips at a fraction of the cost you can download all the stock
22:09
video that your heart desires from their member library including hd and
22:14
4k footage after effects templates motion backgrounds and more we actually
22:19
use it quite a bit here on our tech wiki channel plus you can get exclusive
22:23
discounts on millions of additional marketplace clips you'll save 40 on the
22:28
purchase compared to non-members and what's really cool is the original artist will take a commission on each
22:33
sale all the content is royalty free so you can use it even for commercial use
22:37
to your heart's content and for personal projects like youtube videos for example
22:41
and new clips are added regularly so there's always something new to check
22:44
out so check it out today at the link below so thanks for watching guys
22:49
if you disliked this video then quite frankly you and i are in the same boat
22:52
because i did not have a good time over the last few days working on this and
22:56
i'm not looking forward to continuing to work on it but if you liked it hit like
23:00
get subscribed and maybe consider checking out where to buy the stuff we featured at the link in the video
23:03
description but i wouldn't recommend buying it yet because we haven't quite determined if any of this works also
23:07
down there is our merch store which has cool shirts like this one and our community forum which you should
23:11
definitely join