it's all here look at this it's been
almost a year since we discovered the major issue that effectively put our
entire video archive server offline
that's right all the full quality footage since i was still working at
ncix gone temporarily after the painstaking
process of recovering nearly all that data love you wendell from level1tx we
finally have all the hardware in place so today's the day we get our vault back
up and running we're talking sprinting we're talking better than before we're
talking an entire 3.6 petabyte array
managed by one single head server instead of the five
servers that we have now and i promise it's gonna be awesome
like the segway to our sponsor i fix it looking to expand your selection of
tools ifixit has you covered learn more about the manta toolkit and its 12 what
112 steel bits what with two premium
drivers learn all about it at the end of this video manta
this is the one computer that is going to run the mother vault and the astute
among you have probably already realized that we are not going to be able to put
3.6 petabytes worth of storage inside it
unless it's from hogwarts it's magic
wingardium leviosa in here no it's one guardian leviosa oh sorry excuse me no
in fact there are no hard drives in here this is
where the magic happens the machine you saw just now that's only the brain so
instead of trying to put 3.6 petabytes of hard drives into there they're gonna
go in separate enclosures like this one called j buds or just a bunch of disks
and you can think of an external jbot enclosure kind of like an external hard
drive dock except instead of one hard drive they hold up to a whopping 90 hard
drives each we'll talk more about these a little bit later
just because the 2024 us-trt from
supermicro is shirking its storage responsibilities doesn't mean that it's
not packing inside here we find a dual
socket AMD epic milan system that honestly has a bit of an unusual
configuration typically a system with this much CPU horsepower so those are
two 24 core AMD epic 74 f3s and the ability
to run up to dual 64 cores would be decked out with a ton of NVMe slots or
some enterprise gpus or something like
that but this guy only has four
NVMe slots and these are an optional accessory the rest of the storage up
here and in fact these four slots by default are i kid you not
six gigabit SATA three and a half inch slots
like that you would use for a hard drive that's because this system is actually
designed with a different purpose in mind say for example uh as a
virtualization server where you might need a ton of CPU that's going to be
split up to different clients or if you're like us a ton of CPU that you're
going to use to drive a boatload of connected storage that
isn't inside the chassis so instead of using up all those express lanes from
those epic cpus which is
buddy 128 lanes each with like NVMe drives in
the front we instead want them allocated to PCIe slots in the back of the server
back there we could install high-end networking adapters what is this dual
100 gigabit this guy right here
or the other piece of our puzzle host bus adapters just like with an external
hard drive our jbods back there which the keynight among you will have noticed
that there are two require some kind of
connection to the actual server with the
actual CPU in it and the actual operating system running on it in a
modern jbod this is usually handled by mini sas hd cables like these ones from
infinite cables we're going to have these linked down below
so each of these ports on the back of our three cards can handle four lanes of
12 gigabit sas that's the enterprise big
brother of SATA for a total of 48 gigabits per second per cable your
average computer even server probably doesn't have any factory mini sas hd
ports so we're going to be adding them to this machine with three of broadcom's
9500 16e host bus adapters that means
we'll have a full dedicated card for each of our j-bonds oh yeah there are
three of these right there's one more j-bud that's actually already in the
server room we will show you guys that one later now each of our jbods allows
us to connect up to three cables for increased throughput so that's exactly
what we're going to do on the subject of throughput to push this thing to the
absolute limit we want to pack this machine with as much RAM as possible
since the file system that we're using called zfs can use RAM as a read caching
layer to significantly improve the responsiveness of the array because
mechanical drives on their own yeah even with this many of them not
that great in terms of access times luckily the folks over at micron just
sent over one terabyte of 3200 megatransfer per second
registered ecc memory which last time i
checked smells like performance but the
performance improvements don't stop there we had some
leftover keoxia 30 terabyte NVMe drives
from the dream server build sorry and we figured they would be the perfect
choice to configure as a special metadata device in our zfs pool this is
a game changer for performance because it allows metadata like where the files
are physically stored on those hundreds of hard drives the allocation tables and
optionally even small complete files to be stored on a discrete higher
performance device like these NVMe drives this can make
navigating through large data structures like the thousands of video projects
that we're archiving much much faster
and that directly results in workflow improvements for our staff i'd say it's
about as much of a quality of life improvement as our flow plane subscribers get with no baked in ads
higher quality streaming and exclusive behind the scenes content we're going to
be setting these up in a mirrored stripe leaving us with a total of 60 terabytes
of capacity which is hilarious given that it's a raid 10 of just four drives
which we think should be an ample amount of fast metadata storage for our 3.6
petabyte array unfortunately because we only have these four NVMe slots we're
not going to be able to expand that storage unless higher capacity drives
come along in the future and the performance compared to some of our
other all NVMe arrays is not going to be
that amazing another problem is that this doesn't leave us any additional
space for NVMe storage drives for a level 2 arc reed cache or an intent log
at least at the front but there's a solution to that huh yes
this cute little half height PCIe card from liquid allows us to install four
m.2 drives in a single slot
inside the server and this will be absolutely perfect for that roll what's
the total capacity of this bad boy i don't actually know what ssds are in
there we don't know what's in it there is one we have that is four or four
terabyte ssds okay so it might be this one it could be this one it could also
not be this one great the problem for later Jake this is
pretty cool Jake did you design this three and a half to two and a half inch adapter thing no we found it on
thingiverse oh cool printed it shut out thingiverse whoa
what well that came out which slot did you want the liquid card in uh you gotta
change this guy and then put it right here oh man can we
find out if it's the right one before we do this because we're going to have to
pull the whole thing off what do you mean what's the right one the the high capacity one oh
no oh come on there's got to be a test bench not far
no there's none over there there's none over there Jake says there is none well
let's just go to my bench upstairs logistics
i think it's important for you guys to know that Jake thinks you guys are
utterly incompetent that's what he said whoa those are his words not mine he
said those guys don't even have a test bench over there at all not one
yeah that's what i thought yeah yeah Jake mere minutes later all right hey we
got a foreign dynamic disc here these are what three terabytes
okay right click on that one let's see if it's
toshiba three terabytes 84. yeah perfect these
are the four terabyte ones thank you fast and competent logistics team i just
want you guys to know that i defended you i was the first to say no i would
never use those words about our logistics team
not like Jake
goodbye get out of here
ah here's what we've got next now i mentioned that this was 100 gigabit but
what i didn't say i don't think is that it has dual 100 gigabit ports it can do
a total of 200 gigabit it's PCIe gen 4
by 16 that is a flipping ton of
bandwidth and it uses these sick qsfp28
connectors while Jake orders his lunch i'm going to go ahead and get our other
two hbas installed here i'm getting mucho burrito editor you better put this
in the goddamn cut okay i'm just gonna put this in because
you're taking too long what do you mean i'm taking too long i'm not the one
sitting here ordering lunch touche this is amazing so we've got our
three hbas we've got our 100 gig network
card we've got our additional SSD which is somewhere in the neighborhood of
around 15 terabytes of storage for our
slog and level 2 arc well they're 3.84 each oh i see so like somewhere in that
15 point something oh
and how do you forget about that i forgot about these what i forget 120
terabytes of NVMe in the front yeah yeah are we ready to turn it on i'm
turned on okay i could be more turned on if i was
chilling for lttstore.com hey check out the RGB diode shirt it's like based on
what the individual sub pixels look like but then it's also just a cool design so
do you want the top yeah do a peel very graceful Jake you're doing a fan oh
oh that's right i'm a monster
this thing is sick do we get to power it on now got a laptop a laptop
uh high this one is easy just wait it's actually
really quiet give it a little bit of time are you
gaslighting me sir no i would never
bmw
oh that is so much better and can we turn off that alarm as well i have a
solution i don't think that's going to work it does pulling the power supply
out right makes the alarm for oh wow it didn't work it worked this is a 1600
watt power supply what i mean to be clear that's only a 200 volt input it's
a thousand watts if you're from 100 227 volts
that's what you wanted to do i want to go fast
you said it was quiet so i thought full fan speed must not be that bad right i
had him log me separately into the ipmi
just so i could do that in other news it's booted we got our terabyte of RAM
we got our oh yeah we got our melanox card in there that's good did you put
the nvmes in the right slot i mean one of them is wrong what do you
mean wrong like oh it's wrong
try the bodies one of those is a boot drive might it
might be good there was an activity flash yeah right
okay it's cool and look because the boot device
is a pool just immediately fine i mean it's
probably mad actually this is going to be kind of complicated because some of the drives we like need to put in the
right spot and not mix up because some of the arrays we're keeping and some of
the drives are in another jbod that wendell's working on right now so we
can't really touch those ones uh-huh it's gonna be fun let's go start
carrying how am i supposed to move this this is a four-man carry it literally
has a label on it that says four people need to keep it actually yeah dude these
are like the best earmuffs put them on oh wow now with 80 percent
less Jake i can endorse these
we don't actually have to take all of these out we could just take this one
and that one and then just move this so the head server will just go here
and then we'll go one so we don't even have to move this one so we just have to take the temp vault out should i let the
editors know temp fault's going away uh i already told them it was going to
happen today how do you tell them it's happening now i'll tell them what's
happening now hello hardworking and truly valued
editing team the vault that you have come to know and love temp vault
is now deadvault press f to pay respects
they don't care at all all right bye kind of raises the question do they
need a vault at all then why don't we just have no vault then forget it
if you guys aren't even going to miss it then
what do you mean you copied the files you guys aren't supposed to keep things on your local machines we copied it to
the active server new new new new monik
hopefully this clears everything up and is not confusing at all i think they're
more confused look somebody is reading files from it
right now like ed was lying look oh somebody's is reading from it or
they're writing to it what are they doing oh my god
at 350 megabytes a second this is a not insignificant data transfer
the chart's like nothing and then somebody's like
you want to shut it off now okay who's messing with the server are you messing
with us ed somebody is vaulting right now
did you copy things to the vault right now
probationary employee number 17. you cut and paste it
i should not i use the tara terra copy
but it's still going it's still kind of taste with territory
dang it guys it's such a good you're done you're good
god damn it ignore the the discs with errors uh what there's um we've lost a
few the original vaults 10 terabyte drives are still like in production and
they're like six years old now and we've had problems too our cooling in that
room didn't used to be super stable there were times when you would walk in
there and it was like a wall of heat yeah because we had this
air conditioning unit that would just whatever when we didn't have an air conditioner
yeah yeah at one point i remember when i started here atlantis was like no we
don't need it we don't need it and then all the ups exploded because it was too
hot and he was like okay fine how are we going to move these
fortunately these aren't really rails they're actually like shelves so all we
really need to do is kind of put it on the ground and then just
tilt one end and then slide it in and tilt the other end that's how we put the
other one in there installation by two persons is recommended if these need two
people that server needs 25.
i think i can handle it i wonder when my burrito's going to arrive all the drives
are going to have to come out of this thing before we lift it out no no it's
fine how are we supposed to lift this we can lift it we've done it okay well how
it's drunk no really though it's not off the ground i just use my
feet like this usually your feet yeah and then you just walk it up walk it out
yeah three two i'm holding the heavy side
don't drop it okay i didn't drop it wow you just put all the weight on my what
was on your foot cool well that's all the hard work yeah now
that we've got lab two over there we might actually have a purpose for all
these additional storinators oh man i haven't gone spelunking in the uh server
rack in ages Jake there's so much treasure in here you're
giving everyone quite the view right now whatever usually you got to pay extra
for that onlyplanes.com how do you like this setup i'm taking a
little server cabinet nap here got my little uh pad on the back of my head
okay bottom of the fifth not bad
this was definitely the best way to do this the front of this shelf is out
yeah that's what it appears uh that's why we screw
how full is this uh it's got 60 drives in it okay what seems to have happened
with this server because we didn't screw the shelf in from the front is this
little tooth here that holds it into place seems to have slipped so it's
sagging down like that which means that a ton of this weight is
being borne by a part that's really not designed for that complicates our
operation here a little bit because we can't just pop it back into place from
here we can't really get at it looks like we need to shut this down
pull it off put it back into place screw it in and
then put it back in which sucks if i take the weight and you slide
it forward or the other way around here i'll use my leg up there we go it's up
just put the put the rail in don't move wait put what rail in
well i can't see it oh my god like this oh i think we got it i think we got it
let go let go yeah this actually went pretty okay
situations like this are when i'm so glad we developed this screwdriver i got
the nice strong magnet holding it in place almost no
back force required to spin the ratchet
like is it in there easy yeah it's in oh my
god wait what how are we supposed to get it from here down to there we got it up
there somehow so it's just like a really heavy storenator why don't we just put
the other one on top that says this cart can take 600 pounds no we're not doing that okay
that's right what are the odds uh that it just goes right in
no the angle is still too much okay i'm getting under it boys put the right side
in a bit more okay good good here oh my god it's it yeah it's on it's on the top
okay go ahead you can push it in
hi all right now for the hard one yeah
this is more of a Linus sized server yolo
well it's easy to yolo when somebody else already did all the work yeah good
work Jake there goes the brain and what's really
wild is that if we want more capacity
more than 90 180 270 drives with the
cards we have in now we could do another one and then i think there's room for
one more card right i would say right now with no hbas we
could probably add two and it would still be pretty good performance
um all right let's uh let's put in the hard one ready one two three no yes
what weak all right still weak now that they're in
there working on these is glorious you just press the buttons
pull them out boom funny story when we first put this in i was like ah it's so
heavy it doesn't really need to be bolted down does it i was told yes in
fact it does need to be bolted down and this
is why because if we had a handful of these hanging off the front of it and it
wasn't bolted into the concrete
we'd have a bit of a problem bit of an update here we shucked all those drives
put them in the servers i plugged a bunch of stuff in some power and we have
blinky lights now do i get to press the button yeah it's on the right side give
them a sec everybody in the next square mile is going to hear this they're just
they're they're spinning up boys wow it's actually not as bad in here
come outside let's uh let's shut the door yeah sure
you can hear how awful it is still through the door
oh yeah we wouldn't be able to shoot with those things going
but they don't go okay that's fine oh you just got to add another layer of
insulation there's no space Jake well this is the
backup plan right because there's a thermostat in here that will start up a
fan that exhausts out the top and will pull air passively through this grill in
the event that the air conditioning completely fails do you remember that
the shroud for this thing i made that in metal shop when i was still in high
school that's hilarious okay that was it it's up so far only 20 drives are
showing up but Jake tried a thing which is a what a
domain change we changed some cables and i changed the jbods from three zone to
one zone are you going to click it yes
okay thank god okay one of these
145 drives god damn brother i mean 45 drives is
nice but 145 drives
now that's walking around hard dress i'm just going to try this it's probably
a bad idea oh i'm just going gonna do it i'm gonna try to import the pools it
should find both of them there's two discreet pools on those existing drives
yeah and there's a bunch of extra drives so let's see what happens what are the odds this is just gonna be like yeah
your ancient ass pool and just well it's not they're not that ancient i think
they've all been recreated now okay but actually the crazy thing is i took like
2015 pools from a version of zfs that was not recommended for production and
had not been updated and i put it in like 2022 ass trunas
it just imported it just upgraded for some reason it was super slow like
scrubs would take like a week which is really weird yeah but there could be
kind of flaky drives it wasn't the drives the drives were fine i moved all
the data off deleted the pool made it and then brought it back and it was fine
it might just be like a fragmentation thing or
oh let's see did something and
oh look at that go fast is the 12 drives i think got it i wish i saved these ids
before i did this let's see what happens
does this mean vault is just going to be like well i'm back baby temp vault will
be back the read-only new vault because we're still recovering a few things
that's why one of the jbods is still mia we'll be back just like that and i can
just recreate new shares i mean the ip will be different but i can just recreate the shares
zfs is cool man don't put that there this is a really nice shirt though oh is
that from ltdstore.com it sure is it will be oh is it not out yet i don't
know i don't think so have you announced it on rancho i don't think so 30 hours
later wow the pool's finally imported holy crap
super micro massive shout out thank you for sending
over the gear it's amazing we did have a little bit of trouble with setting it up
with three domains yeah it's for some reason they're only working in single
zone mode that means all 90 drives are going over one cable i don't know why
because we had it set up on the bench like a couple weeks ago with three zones
anyway it's accessible we're ready to copy some files and hopefully that's
something they can sort out either through helping us set up the configuration or through firmware yeah
it's probably just we're missing something this is one of those solutions
where it's kind of like you just know what you're doing if you're doing it you're an integrator who
sets these things up every day week after week and that's not us but i
copied here i copy the file let's do it look i'm copying a file from it to this
like hundreds of megabytes a second it's not the most impressive scenes ever this
is a SATA SSD and we're screen recording to it now so
it's like ah but hey remember guys this is with
ancient like degraded degraded-ass fragmented v-devs once we strip all the
data off of them rearrange them reshuffle them set them up fresh and
dump all the data back onto them we should be able to be in the neighborhood
of a Gigabyte a second which is all we're going to have for for sure well yeah but that's all we're going to have
yeah that's all we're going to do network interface it's 100 oh that's
right it is but here look at that 147 drives never in actual production that's
amazing and then we'll add 60 more and it'll be
over 200. and then we'll add 50 more and it'll be 250. and then we'll add this
segway to our sponsor i fixed its manta kit has a 112 steel bits and comes
complete with standard bits like phillips and flathead as well as more
niche options like pentalobe hex and tri-point bits it also includes two
aluminum screwdriver handles a weighty quarter inch drive for full size
applications and a smaller four millimeter for precision work both
drivers feature magnetic bit sockets knurled grips and silky smooth spinning
tops you can even use the toolkit's lid as a built-in storage tray best of all
every ifixit tool is backed by a lifetime warranty so if something breaks
they will replace it for free so visit ifixit.com forward slash LTT and pick up
your manta kit today monta if you guys
enjoyed this video maybe you'll enjoy the one where we tried to deploy wanik
no no that was awful okay sure yeah sure
to watch that or the the previous one where we just looked at jbots because they're cool