I bet most of you forgot about the million dooll PC here and honestly I
don't blame you it's kind of been a while since we touched the thing but I
swear it's not our fault right before we were going to do the big demo it broke
then it broke again and again it's just been the worst kind of problem to
troubleshoot sometimes our full petabyte
of NVMe ssds gets detected and other
times some of them don't but I wouldn't make a video just to tell
you it's still broken would I the solution ended up being shockingly
simple and something that I bet you've actually seen before but that's in the
past this is the now and it is finally
time when this video is over we are
going to have the largest and fastest
storage server on YouTube at least until
we have to find all the boxes pack it up and send it back to them did I tell you
they threw away the boxes sorry what oh my God they're not behind me how are
we going to deal with that how are we going to segue to our sponsor gskill G
skills Trident Z5 Neo ddr5 memory is built for AMD ryzen 7,000 series
processors and oneclick memory overclocking learn more at the link down
below the culprit was this guy but not
this guy entirely four of the drives
specifically out of the 12 had this erratic Behavior where we would fire up
the whole system and they wouldn't be there then we'd replug the cables and
they'd come back and we could Benchmark it and test it and validate and then and
then they would be gone we tried everything from moving those drives to
different Bays to different servers even unplugging and replugging the cables and
it just kept freaking happening and in the end it was one of
the most basic troubleshooting steps in the book have you tried unplugging it
and plugging it back in but not the drive not the back plane or the cables
the CPU cuz what it's hard to wrap your brain around is that these are NVMe or
PCI Express drives and through the
connector on the back through the back plane through the motherboard to the CPU
socket they actually connect directly
all the way back to your processor which means that un Lo Loosely seated CPU
could actually cause a broken Link in that chain and here's the thing after
the servers had been running for a few minutes that's just enough thermal
expansion for that CPU pin that's just not quite touching bam now it's touching
boom intermittent problem yeah of course with the system already on it's not
going to pick those drives up again so when we were rebooting it after reting
the cables they'd show up again it makes perfect sense now but oh my God trying
to figure F this out you got to think outside of the box and also also deep
inside the Box you got it you got it
everybody's got it oh this is not screwed on Jake what is not uh this top
cover oh whatever where are we working on this we in there oh sure you have a
screwdriver right I have a highly erect server that's for sure yes I have my
highquality ratcheting magnetic screwdriver LTT Store.com now if we were
smart folks we might have looked at what drives they were and traced it back to
what CPU it was yeah that's CP but you know what I was trying to say is we're
just going to reat both the CPUs damn it sure I don't have a screwdriver yet I
went I put a Phillips bit so you're just going to have to do this all yourself no that's fine crap I don't have a torque
set you know what sometimes hex works oh my God don't do that what don't do that
that's awful you're going to strip this and then we're going to strip it you're
going to make this a very long video I'm not going to strip it you could go grab
another one you can make yourself useful I could the funny thing is oh God oh my
God I'm kidding oh my God his face why did you even take
it out just I'm just no don't clean the thermal
P oh my god really yeah receed it baby
no we should reop it no all right noral pce is expensive in this economy right I
mean left correct left correct yes no uh
no I was never much good at math why are these so finicky damn
it beautiful watch the server not boot
now I know right that is a very real possibility
with AMD's Threadripper and epic CPUs
because they're so large it is pretty
easy to accidentally install them a little bit cockeyed which can cause
these sorts of issues whether it's PCI Express devices having intermittent
problems or RAM this isn't even on yet
this is like the boot sequence this is the the ipmi for those of you who
haven't seen the previous parts of these series or who understandably forget them
at this point what you're looking at here is a one pyte of flash storage
server which is a non-trivial thing to build because aside from just having
enough Bays to put that many SSS in if
you want to get anywhere near the full performance of these kokia drives you
need a ton of computer so inside each of
these six oneu servers here is 12 15
tbte Drive so that's for a total of
72 drives each of these CD 6r Enterprise
drives is capable of a whopping 5.5 gab
a second reads 4 GB a second rights all
of these work together using a file system called W FS that's designed
specifically for NVMe drives to achieve
unbelievable performance of course in
order to measure this performance you actually have to put some kind of load
on the system that's where this guy comes in it has
two 64 core epic CPUs eight of NVIDIA's
a100 gpus those are critical to generate
the load and it has a walking
8200 gabit per second network connections and of course those run
through this 32 Port 200 GB network
switch to each of our servers down below
that's a lot of high-speed connectivity but with great performance comes great
power consumption and to run this thing full bore we needed to plug extension
cords into five separate 15 amp 120 volt
Breakers and a separate 208v 30 amp
breaker and did I mention they're overclocked I mean I feel like I can
tell from all the heat coming off of it it is flipping warm back here it's very
uncomfortable to stand here another thing we ran into last time just to get
the array started these servers were so out of sync just from Shipping like
their time the time on the servers was so out of sync the real time clock you
know could be 5 10 seconds off whatever that the array would not start and the
easiest way to get them to sync up again is ntp Network time protocol and of
course without networking this wasn't set up to actually connect to the
internet all of these have static IPS that are assigned so in our router I had
to create uh a VLAN that has the same
subnet information and routed from the server room all the way over here to
here and plug it into the switch had to go in the switch and tell it oh 10 gig
is fine it doesn't have to be 200 gig and then they turned on no problem but I
want to access the ipmi so we can see the power consumption sure so we've got
to plug this bad boy into all the ipmi ports really quick oh okay so we need
some patch cables patch cables are right in the front all right let's do it oh that's wild yeah the management port for
this one is actually on the front oh man
it's so much nicer in here I want to see all 72 drives is what I want to see you
might be asking for a lot no no bare
minimum when those four drives are missing the array just rebuilds oh God
just give it a second oh sick it's working okay the cool thing though when those
four drives are missing the array like rebuilt itself like it was nothing takes
like 5 minutes cuz they're so fast fast yeah but we want the capacity want the
whole point a petabyte a flash okay I I
hate two burst your bubble for some reason they only configured it with a
500 tbte like drive only 500
terab I think we can expand it later but for the purposes of the first demo I
think let's just leave it I don't want to break it I'm ssing into the the node
oh look at that how many is that 1 2 3 4 5 6 7 8 good that's the correct number
that's a lot of GPU it's a lot of freaking gpus is there is there NVIDIA
SMI each of these is like oh look at that four 00 Watts 3200 watts of just
GPU we're not connecting to these things over anything like SMB cuz that would be
way too slow we're like directly connecting with the WCA interface and
we're going to be using GPU direct storage which is super cool we'll talk a little bit more about that later yeah
this visualization is super cool the green things I'm looking at I guess are
drives and then the gray things are cores or no the gray things are drives
the green things are core that makes way more sense actually and then blue is our
network interfaces that is a pretty
visual it oh yeah you see this look it's they're pinning cores to specific Drive
slots it looks like there's some special sauce going on here I think there is
some manual ass shiz that went on in the configuration of the system seriously
though a lot of tuning because like I said before making a one petabyte flash
server easy making one that performs
near actual we're still not going to be hitting the peak you'll remember when we
did the honey badger server we got like 100 gigs of second that was directly
writing to individual drives there's no file system no no raid parody networking
nothing was going on yeah this is a usable file system across six servers
all right you ready to see some big numbers yes this one's not as exciting as you'd expect you have to you run it
and then it shows you it like gives you a file with the numbers that's not that
fast that's not very fast just's give it a sec okay ooh it's faster you said this
was going to be cool it's trying different sizes this is probably like 4K
right now the script is starting with GDs or GPU direct storage it sounds very
complicated and realistically in practice setting it up was probably very
complicated for somebody but the main difference is rather than taking the
data from those ndme drives putting them into the CPU's memory and then into the
GPU's memory we're skipping that middle step it goes right from NVMe to the
GPU's memory oh there we go 15 gig second was CPU first then no it's doing
GDs but like I said it's doing different block sizes you ready to see some numbers okay so this is GDs says GPU
direct storage only read tests
1006 gibes per second so that's like
114 gigabytes per second over two with a
file system Blu-rays per second like like full quality Big Boy the whole
thing per second holy and again we've
seen numbers like this before but not with a file system loaded on it no we
could take oh you know W you could actually use this you want to like SMB
for a second here I mean well we don't have a client that's anywhere near fast
enough which I guess is a perfect opportunity for us to talk about what this would be for want to show us that
visualization demo Humanity's desire to look beyond the stars is nothing new
it's not and one of the major stepping stones in the space travel is finding a
way to transport people in all seriousness though sending people to
another planet is an incredibly complex challenge but one that modern Computing
can help us with and one of the current proposed plans from NASA involves
sending a six-person manned craft all the way to Mars after the 10-month
journey the crew would transfer into a landing pod that is 16 m in diameter so
roughly the size of a two-story house due to the sheer size of the Lander as
well as the thin atmosphere on Mars it's not possible to use a parachute to slow
down like they have on past Landings like the perseverance Rover so instead
NASA wants to use retropulsion otherwise known as creating thrust in the opposite
direction to slow the Lander down from
12,000 M an hour to you know the ground
in less than 7 minutes simulating this Landing is a very important part of the
research and development for the project and with the power of the summit supercomputers 27,000 NVIDIA gpus over
the course of a week they were able to build a 100 plus terabyte model of the
landing that is over 1 billion Points
each with seven numerical values density vorticity pressure that sort of thing
and I guess we have NASA security clearance now because we have that exact
model loaded up on the million dooll server and here it is and Jake's got it
running right behind me now in the past Engineers would actually take this data
model and render it out frame by frame into a video still super cool and very
useful but if you want to tweak a parameter and see how how it affects the
model guess what you're waiting hours for a new render with new technology
like super fast NVMe storage uh NVMe
speed file systems like w fs and the power of GPU direct well now we can have
the graphics cards render the data directly from the storage bypassing the
CPU allowing us to do this in basically
real time real time manipulation that is
I mean it's a it's a bit cinematic but we're getting we're getting like 5 FP
5 FPS that's I mean a lot better than hold on hold on hold on 5 FPS each of
these frames is over 14 gab of data yeah
that's before it becomes a frame the gpus have to process that so each second
we are streaming between 70 and 90 gab
from that storage directly to those gpus
so that's one to two Blu-rays each
second can you like move it yeah can
here let's yeah move it play it play it trust us this is really neat I wish
there was a way for me to like translate how much is happening in the background
I guess we could look at how much power it's drawing this is what you guys are
going to want to see how many amps are
going through our pdu here and that's just for this top server yeah why does
it need 17 amps of 200 4,000 watts right
now what the hell oh hey this is interesting look so one of our gpus you
can see it's refreshing every every 2 seconds yeah they're kind of drawing a
little more power now but still not anywhere near 400 W yeah that's
interesting I guess this is just more storage dependent than it is actual GPU
rendering dependent which makes sense that's they're pretty op gpus sort of
why they sent this demo it's about the storage like moving it around a bit oh
yeah they got a little Spike here you can see the utilization on the gpus is
like 50 60% probably a lot of this
is more the memory usage and less the actual core itself you can imagine the
type of workflow Improvement you would have from being able to just mess around
with this in real time rather than like rendering it waiting 3 days for it to
finish rendering cuz you're at a university and you got a small budget
works just like this segue to our sponsor vessie do you hate wet socks as
much as I do vessie Footwear makes lightweight breathable and most
importantly waterresistant shoes so no more squelchy socks their diamex
material not only keeps your feet dry but keeps them warm in the winter and
cool in the summer how does that work the stretchy design shows that Comfort
is at the Forefront at times making you forget you're wearing shoes vessie makes
cruelty-free products right down to the glue their shoes are 100% vegan whether
it's a rainy city or a Rocky Trail the Herring bone tread design is there to
help stop you from slipping around your feet deserve a little treat so go ahead
and click the link below and use promo code Linus Tech tips to save $25 bucks
on your first pair today if you guys enjoyed this video maybe check out the
previous parts and uh I don't know maybe subscribe to Floatplane maybe we'll
shoot an exclusive of packing them up no I don't think we'll do that I I like I
got to wonder if there's anything else we could do I thought about video
editing I would love to use these for the Machine Vision uh cat squirter
turret project