WEBVTT

00:00:00.320 --> 00:00:06.080
it's all here look at this it's been

00:00:03.439 --> 00:00:12.240
almost a year since we discovered the major issue that effectively put our

00:00:08.639 --> 00:00:14.480
entire video archive server offline

00:00:12.240 --> 00:00:17.600
that's right all the full quality footage since i was still working at

00:00:16.000 --> 00:00:22.800
ncix gone temporarily after the painstaking

00:00:20.720 --> 00:00:28.640
process of recovering nearly all that data love you wendell from level1tx we

00:00:25.680 --> 00:00:33.600
finally have all the hardware in place so today's the day we get our vault back

00:00:31.519 --> 00:00:40.000
up and running we're talking sprinting we're talking better than before we're

00:00:35.600 --> 00:00:41.680
talking an entire 3.6 petabyte array

00:00:40.000 --> 00:00:47.520
managed by one single head server instead of the five

00:00:44.640 --> 00:00:52.480
servers that we have now and i promise it's gonna be awesome

00:00:49.520 --> 00:00:56.480
like the segway to our sponsor i fix it looking to expand your selection of

00:00:54.000 --> 00:01:03.039
tools ifixit has you covered learn more about the manta toolkit and its 12 what

00:00:59.760 --> 00:01:04.960
112 steel bits what with two premium

00:01:03.039 --> 00:01:08.920
drivers learn all about it at the end of this video manta

00:01:16.320 --> 00:01:23.759
this is the one computer that is going to run the mother vault and the astute

00:01:21.439 --> 00:01:30.560
among you have probably already realized that we are not going to be able to put

00:01:26.560 --> 00:01:31.759
3.6 petabytes worth of storage inside it

00:01:30.560 --> 00:01:37.920
unless it's from hogwarts it's magic

00:01:35.360 --> 00:01:42.240
wingardium leviosa in here no it's one guardian leviosa oh sorry excuse me no

00:01:41.200 --> 00:01:46.640
in fact there are no hard drives in here this is

00:01:44.720 --> 00:01:51.840
where the magic happens the machine you saw just now that's only the brain so

00:01:49.520 --> 00:01:56.799
instead of trying to put 3.6 petabytes of hard drives into there they're gonna

00:01:54.000 --> 00:02:02.960
go in separate enclosures like this one called j buds or just a bunch of disks

00:02:00.399 --> 00:02:08.239
and you can think of an external jbot enclosure kind of like an external hard

00:02:05.759 --> 00:02:14.319
drive dock except instead of one hard drive they hold up to a whopping 90 hard

00:02:12.160 --> 00:02:19.040
drives each we'll talk more about these a little bit later

00:02:15.760 --> 00:02:21.120
just because the 2024 us-trt from

00:02:19.040 --> 00:02:26.959
supermicro is shirking its storage responsibilities doesn't mean that it's

00:02:23.360 --> 00:02:29.920
not packing inside here we find a dual

00:02:26.959 --> 00:02:34.239
socket AMD epic milan system that honestly has a bit of an unusual

00:02:31.760 --> 00:02:37.680
configuration typically a system with this much CPU horsepower so those are

00:02:36.800 --> 00:02:44.720
two 24 core AMD epic 74 f3s and the ability

00:02:42.319 --> 00:02:51.599
to run up to dual 64 cores would be decked out with a ton of NVMe slots or

00:02:48.319 --> 00:02:54.480
some enterprise gpus or something like

00:02:51.599 --> 00:02:57.920
that but this guy only has four

00:02:55.280 --> 00:03:02.159
NVMe slots and these are an optional accessory the rest of the storage up

00:03:00.080 --> 00:03:06.239
here and in fact these four slots by default are i kid you not

00:03:04.959 --> 00:03:11.840
six gigabit SATA three and a half inch slots

00:03:09.360 --> 00:03:15.440
like that you would use for a hard drive that's because this system is actually

00:03:13.599 --> 00:03:19.519
designed with a different purpose in mind say for example uh as a

00:03:17.680 --> 00:03:23.599
virtualization server where you might need a ton of CPU that's going to be

00:03:21.360 --> 00:03:28.080
split up to different clients or if you're like us a ton of CPU that you're

00:03:26.400 --> 00:03:32.959
going to use to drive a boatload of connected storage that

00:03:30.000 --> 00:03:38.480
isn't inside the chassis so instead of using up all those express lanes from

00:03:35.360 --> 00:03:40.599
those epic cpus which is

00:03:38.480 --> 00:03:47.680
buddy 128 lanes each with like NVMe drives in

00:03:44.720 --> 00:03:53.360
the front we instead want them allocated to PCIe slots in the back of the server

00:03:50.480 --> 00:03:58.640
back there we could install high-end networking adapters what is this dual

00:03:55.200 --> 00:03:58.640
100 gigabit this guy right here

00:03:58.799 --> 00:04:06.879
or the other piece of our puzzle host bus adapters just like with an external

00:04:04.000 --> 00:04:11.760
hard drive our jbods back there which the keynight among you will have noticed

00:04:08.480 --> 00:04:14.799
that there are two require some kind of

00:04:11.760 --> 00:04:17.120
connection to the actual server with the

00:04:14.799 --> 00:04:20.959
actual CPU in it and the actual operating system running on it in a

00:04:18.720 --> 00:04:25.360
modern jbod this is usually handled by mini sas hd cables like these ones from

00:04:23.919 --> 00:04:29.680
infinite cables we're going to have these linked down below

00:04:27.120 --> 00:04:36.000
so each of these ports on the back of our three cards can handle four lanes of

00:04:32.960 --> 00:04:38.560
12 gigabit sas that's the enterprise big

00:04:36.000 --> 00:04:43.680
brother of SATA for a total of 48 gigabits per second per cable your

00:04:41.280 --> 00:04:48.880
average computer even server probably doesn't have any factory mini sas hd

00:04:46.880 --> 00:04:54.960
ports so we're going to be adding them to this machine with three of broadcom's

00:04:51.600 --> 00:04:57.360
9500 16e host bus adapters that means

00:04:54.960 --> 00:05:01.520
we'll have a full dedicated card for each of our j-bonds oh yeah there are

00:04:59.840 --> 00:05:04.800
three of these right there's one more j-bud that's actually already in the

00:05:03.120 --> 00:05:09.680
server room we will show you guys that one later now each of our jbods allows

00:05:07.280 --> 00:05:13.680
us to connect up to three cables for increased throughput so that's exactly

00:05:11.600 --> 00:05:18.320
what we're going to do on the subject of throughput to push this thing to the

00:05:15.680 --> 00:05:23.280
absolute limit we want to pack this machine with as much RAM as possible

00:05:21.600 --> 00:05:28.479
since the file system that we're using called zfs can use RAM as a read caching

00:05:26.880 --> 00:05:32.639
layer to significantly improve the responsiveness of the array because

00:05:30.560 --> 00:05:38.000
mechanical drives on their own yeah even with this many of them not

00:05:35.440 --> 00:05:42.560
that great in terms of access times luckily the folks over at micron just

00:05:40.960 --> 00:05:48.880
sent over one terabyte of 3200 megatransfer per second

00:05:45.520 --> 00:05:51.199
registered ecc memory which last time i

00:05:48.880 --> 00:05:55.600
checked smells like performance but the

00:05:53.199 --> 00:06:01.680
performance improvements don't stop there we had some

00:05:57.759 --> 00:06:04.160
leftover keoxia 30 terabyte NVMe drives

00:06:01.680 --> 00:06:08.479
from the dream server build sorry and we figured they would be the perfect

00:06:06.400 --> 00:06:14.319
choice to configure as a special metadata device in our zfs pool this is

00:06:12.080 --> 00:06:20.479
a game changer for performance because it allows metadata like where the files

00:06:17.680 --> 00:06:26.400
are physically stored on those hundreds of hard drives the allocation tables and

00:06:23.520 --> 00:06:30.400
optionally even small complete files to be stored on a discrete higher

00:06:28.720 --> 00:06:35.280
performance device like these NVMe drives this can make

00:06:33.120 --> 00:06:40.800
navigating through large data structures like the thousands of video projects

00:06:37.600 --> 00:06:43.360
that we're archiving much much faster

00:06:40.800 --> 00:06:47.199
and that directly results in workflow improvements for our staff i'd say it's

00:06:45.680 --> 00:06:53.280
about as much of a quality of life improvement as our flow plane subscribers get with no baked in ads

00:06:51.120 --> 00:06:57.600
higher quality streaming and exclusive behind the scenes content we're going to

00:06:54.720 --> 00:07:03.199
be setting these up in a mirrored stripe leaving us with a total of 60 terabytes

00:07:01.280 --> 00:07:07.919
of capacity which is hilarious given that it's a raid 10 of just four drives

00:07:05.120 --> 00:07:13.919
which we think should be an ample amount of fast metadata storage for our 3.6

00:07:11.199 --> 00:07:17.520
petabyte array unfortunately because we only have these four NVMe slots we're

00:07:16.160 --> 00:07:20.960
not going to be able to expand that storage unless higher capacity drives

00:07:19.280 --> 00:07:25.840
come along in the future and the performance compared to some of our

00:07:22.639 --> 00:07:28.720
other all NVMe arrays is not going to be

00:07:25.840 --> 00:07:33.280
that amazing another problem is that this doesn't leave us any additional

00:07:30.639 --> 00:07:38.400
space for NVMe storage drives for a level 2 arc reed cache or an intent log

00:07:36.639 --> 00:07:44.080
at least at the front but there's a solution to that huh yes

00:07:41.440 --> 00:07:50.479
this cute little half height PCIe card from liquid allows us to install four

00:07:47.280 --> 00:07:52.639
m.2 drives in a single slot

00:07:50.479 --> 00:07:57.599
inside the server and this will be absolutely perfect for that roll what's

00:07:55.599 --> 00:08:00.960
the total capacity of this bad boy i don't actually know what ssds are in

00:07:59.120 --> 00:08:04.960
there we don't know what's in it there is one we have that is four or four

00:08:02.639 --> 00:08:08.319
terabyte ssds okay so it might be this one it could be this one it could also

00:08:06.800 --> 00:08:11.759
not be this one great the problem for later Jake this is

00:08:10.319 --> 00:08:16.639
pretty cool Jake did you design this three and a half to two and a half inch adapter thing no we found it on

00:08:14.319 --> 00:08:20.560
thingiverse oh cool printed it shut out thingiverse whoa

00:08:18.560 --> 00:08:23.440
what well that came out which slot did you want the liquid card in uh you gotta

00:08:22.240 --> 00:08:27.599
change this guy and then put it right here oh man can we

00:08:25.440 --> 00:08:30.240
find out if it's the right one before we do this because we're going to have to

00:08:29.120 --> 00:08:36.479
pull the whole thing off what do you mean what's the right one the the high capacity one oh

00:08:33.919 --> 00:08:40.360
no oh come on there's got to be a test bench not far

00:08:41.680 --> 00:08:47.200
no there's none over there there's none over there Jake says there is none well

00:08:45.680 --> 00:08:51.839
let's just go to my bench upstairs logistics

00:08:50.080 --> 00:08:55.839
i think it's important for you guys to know that Jake thinks you guys are

00:08:53.680 --> 00:09:00.640
utterly incompetent that's what he said whoa those are his words not mine he

00:08:58.480 --> 00:09:05.760
said those guys don't even have a test bench over there at all not one

00:09:03.120 --> 00:09:11.120
yeah that's what i thought yeah yeah Jake mere minutes later all right hey we

00:09:08.720 --> 00:09:14.640
got a foreign dynamic disc here these are what three terabytes

00:09:15.120 --> 00:09:19.680
okay right click on that one let's see if it's

00:09:18.240 --> 00:09:23.120
toshiba three terabytes 84. yeah perfect these

00:09:21.680 --> 00:09:26.959
are the four terabyte ones thank you fast and competent logistics team i just

00:09:25.279 --> 00:09:31.360
want you guys to know that i defended you i was the first to say no i would

00:09:29.279 --> 00:09:35.600
never use those words about our logistics team

00:09:32.800 --> 00:09:35.600
not like Jake

00:09:37.279 --> 00:09:42.320
goodbye get out of here

00:09:40.399 --> 00:09:46.480
ah here's what we've got next now i mentioned that this was 100 gigabit but

00:09:44.320 --> 00:09:53.120
what i didn't say i don't think is that it has dual 100 gigabit ports it can do

00:09:49.200 --> 00:09:56.399
a total of 200 gigabit it's PCIe gen 4

00:09:53.120 --> 00:10:00.399
by 16 that is a flipping ton of

00:09:56.399 --> 00:10:02.480
bandwidth and it uses these sick qsfp28

00:10:00.399 --> 00:10:06.399
connectors while Jake orders his lunch i'm going to go ahead and get our other

00:10:04.000 --> 00:10:10.800
two hbas installed here i'm getting mucho burrito editor you better put this

00:10:08.480 --> 00:10:13.600
in the goddamn cut okay i'm just gonna put this in because

00:10:12.000 --> 00:10:17.120
you're taking too long what do you mean i'm taking too long i'm not the one

00:10:15.200 --> 00:10:24.079
sitting here ordering lunch touche this is amazing so we've got our

00:10:20.480 --> 00:10:26.800
three hbas we've got our 100 gig network

00:10:24.079 --> 00:10:31.279
card we've got our additional SSD which is somewhere in the neighborhood of

00:10:28.079 --> 00:10:34.240
around 15 terabytes of storage for our

00:10:31.279 --> 00:10:38.160
slog and level 2 arc well they're 3.84 each oh i see so like somewhere in that

00:10:36.160 --> 00:10:40.800
15 point something oh

00:10:39.200 --> 00:10:44.720
and how do you forget about that i forgot about these what i forget 120

00:10:42.720 --> 00:10:48.880
terabytes of NVMe in the front yeah yeah are we ready to turn it on i'm

00:10:46.959 --> 00:10:52.880
turned on okay i could be more turned on if i was

00:10:50.560 --> 00:10:57.519
chilling for lttstore.com hey check out the RGB diode shirt it's like based on

00:10:55.760 --> 00:11:01.760
what the individual sub pixels look like but then it's also just a cool design so

00:10:59.440 --> 00:11:06.240
do you want the top yeah do a peel very graceful Jake you're doing a fan oh

00:11:04.959 --> 00:11:12.399
oh that's right i'm a monster

00:11:09.760 --> 00:11:16.880
this thing is sick do we get to power it on now got a laptop a laptop

00:11:15.120 --> 00:11:20.880
uh high this one is easy just wait it's actually

00:11:18.880 --> 00:11:25.519
really quiet give it a little bit of time are you

00:11:22.000 --> 00:11:28.519
gaslighting me sir no i would never

00:11:25.519 --> 00:11:28.519
bmw

00:11:31.920 --> 00:11:37.680
oh that is so much better and can we turn off that alarm as well i have a

00:11:36.160 --> 00:11:42.000
solution i don't think that's going to work it does pulling the power supply

00:11:39.519 --> 00:11:46.959
out right makes the alarm for oh wow it didn't work it worked this is a 1600

00:11:44.880 --> 00:11:52.000
watt power supply what i mean to be clear that's only a 200 volt input it's

00:11:49.440 --> 00:11:57.320
a thousand watts if you're from 100 227 volts

00:11:57.680 --> 00:12:03.600
that's what you wanted to do i want to go fast

00:12:01.200 --> 00:12:09.839
you said it was quiet so i thought full fan speed must not be that bad right i

00:12:06.560 --> 00:12:12.480
had him log me separately into the ipmi

00:12:09.839 --> 00:12:17.200
just so i could do that in other news it's booted we got our terabyte of RAM

00:12:15.360 --> 00:12:20.560
we got our oh yeah we got our melanox card in there that's good did you put

00:12:18.639 --> 00:12:24.610
the nvmes in the right slot i mean one of them is wrong what do you

00:12:22.079 --> 00:12:27.839
mean wrong like oh it's wrong

00:12:27.839 --> 00:12:33.200
try the bodies one of those is a boot drive might it

00:12:32.079 --> 00:12:37.440
might be good there was an activity flash yeah right

00:12:35.279 --> 00:12:40.880
okay it's cool and look because the boot device

00:12:38.480 --> 00:12:44.000
is a pool just immediately fine i mean it's

00:12:42.560 --> 00:12:49.440
probably mad actually this is going to be kind of complicated because some of the drives we like need to put in the

00:12:47.360 --> 00:12:53.760
right spot and not mix up because some of the arrays we're keeping and some of

00:12:51.600 --> 00:12:57.839
the drives are in another jbod that wendell's working on right now so we

00:12:55.120 --> 00:13:01.040
can't really touch those ones uh-huh it's gonna be fun let's go start

00:12:59.279 --> 00:13:05.600
carrying how am i supposed to move this this is a four-man carry it literally

00:13:03.519 --> 00:13:09.279
has a label on it that says four people need to keep it actually yeah dude these

00:13:07.760 --> 00:13:13.680
are like the best earmuffs put them on oh wow now with 80 percent

00:13:11.839 --> 00:13:16.720
less Jake i can endorse these

00:13:15.120 --> 00:13:20.959
we don't actually have to take all of these out we could just take this one

00:13:19.200 --> 00:13:24.320
and that one and then just move this so the head server will just go here

00:13:23.120 --> 00:13:29.440
and then we'll go one so we don't even have to move this one so we just have to take the temp vault out should i let the

00:13:27.519 --> 00:13:32.639
editors know temp fault's going away uh i already told them it was going to

00:13:30.639 --> 00:13:35.680
happen today how do you tell them it's happening now i'll tell them what's

00:13:33.760 --> 00:13:40.000
happening now hello hardworking and truly valued

00:13:37.839 --> 00:13:47.639
editing team the vault that you have come to know and love temp vault

00:13:42.639 --> 00:13:47.639
is now deadvault press f to pay respects

00:13:48.560 --> 00:13:54.720
they don't care at all all right bye kind of raises the question do they

00:13:53.519 --> 00:13:57.920
need a vault at all then why don't we just have no vault then forget it

00:13:56.639 --> 00:14:00.079
if you guys aren't even going to miss it then

00:14:00.399 --> 00:14:07.920
what do you mean you copied the files you guys aren't supposed to keep things on your local machines we copied it to

00:14:04.720 --> 00:14:09.760
the active server new new new new monik

00:14:07.920 --> 00:14:13.680
hopefully this clears everything up and is not confusing at all i think they're

00:14:11.279 --> 00:14:18.639
more confused look somebody is reading files from it

00:14:16.160 --> 00:14:23.440
right now like ed was lying look oh somebody's is reading from it or

00:14:21.600 --> 00:14:28.320
they're writing to it what are they doing oh my god

00:14:25.360 --> 00:14:33.279
at 350 megabytes a second this is a not insignificant data transfer

00:14:31.279 --> 00:14:36.560
the chart's like nothing and then somebody's like

00:14:35.040 --> 00:14:41.440
you want to shut it off now okay who's messing with the server are you messing

00:14:38.959 --> 00:14:44.240
with us ed somebody is vaulting right now

00:14:42.320 --> 00:14:46.480
did you copy things to the vault right now

00:14:47.440 --> 00:14:55.120
probationary employee number 17. you cut and paste it

00:14:52.639 --> 00:14:58.160
i should not i use the tara terra copy

00:14:56.639 --> 00:15:03.600
but it's still going it's still kind of taste with territory

00:15:00.959 --> 00:15:08.160
dang it guys it's such a good you're done you're good

00:15:05.600 --> 00:15:14.079
god damn it ignore the the discs with errors uh what there's um we've lost a

00:15:11.680 --> 00:15:19.040
few the original vaults 10 terabyte drives are still like in production and

00:15:16.720 --> 00:15:23.360
they're like six years old now and we've had problems too our cooling in that

00:15:21.760 --> 00:15:27.040
room didn't used to be super stable there were times when you would walk in

00:15:24.959 --> 00:15:31.839
there and it was like a wall of heat yeah because we had this

00:15:29.760 --> 00:15:36.639
air conditioning unit that would just whatever when we didn't have an air conditioner

00:15:34.560 --> 00:15:39.440
yeah yeah at one point i remember when i started here atlantis was like no we

00:15:38.000 --> 00:15:44.160
don't need it we don't need it and then all the ups exploded because it was too

00:15:41.920 --> 00:15:46.880
hot and he was like okay fine how are we going to move these

00:15:45.279 --> 00:15:50.399
fortunately these aren't really rails they're actually like shelves so all we

00:15:48.880 --> 00:15:54.800
really need to do is kind of put it on the ground and then just

00:15:52.800 --> 00:15:58.399
tilt one end and then slide it in and tilt the other end that's how we put the

00:15:56.560 --> 00:16:04.800
other one in there installation by two persons is recommended if these need two

00:16:01.279 --> 00:16:06.800
people that server needs 25.

00:16:04.800 --> 00:16:10.000
i think i can handle it i wonder when my burrito's going to arrive all the drives

00:16:08.800 --> 00:16:14.399
are going to have to come out of this thing before we lift it out no no it's

00:16:12.000 --> 00:16:19.600
fine how are we supposed to lift this we can lift it we've done it okay well how

00:16:17.199 --> 00:16:22.720
it's drunk no really though it's not off the ground i just use my

00:16:20.959 --> 00:16:26.560
feet like this usually your feet yeah and then you just walk it up walk it out

00:16:24.800 --> 00:16:30.800
yeah three two i'm holding the heavy side

00:16:27.839 --> 00:16:34.399
don't drop it okay i didn't drop it wow you just put all the weight on my what

00:16:32.320 --> 00:16:38.320
was on your foot cool well that's all the hard work yeah now

00:16:36.079 --> 00:16:42.959
that we've got lab two over there we might actually have a purpose for all

00:16:40.000 --> 00:16:48.079
these additional storinators oh man i haven't gone spelunking in the uh server

00:16:46.000 --> 00:16:51.759
rack in ages Jake there's so much treasure in here you're

00:16:49.759 --> 00:16:55.519
giving everyone quite the view right now whatever usually you got to pay extra

00:16:53.519 --> 00:16:59.680
for that onlyplanes.com how do you like this setup i'm taking a

00:16:57.440 --> 00:17:04.720
little server cabinet nap here got my little uh pad on the back of my head

00:17:01.920 --> 00:17:07.839
okay bottom of the fifth not bad

00:17:05.839 --> 00:17:12.880
this was definitely the best way to do this the front of this shelf is out

00:17:10.640 --> 00:17:17.280
yeah that's what it appears uh that's why we screw

00:17:14.799 --> 00:17:22.000
how full is this uh it's got 60 drives in it okay what seems to have happened

00:17:20.000 --> 00:17:26.160
with this server because we didn't screw the shelf in from the front is this

00:17:24.400 --> 00:17:31.360
little tooth here that holds it into place seems to have slipped so it's

00:17:29.600 --> 00:17:37.679
sagging down like that which means that a ton of this weight is

00:17:34.720 --> 00:17:41.760
being borne by a part that's really not designed for that complicates our

00:17:40.080 --> 00:17:45.600
operation here a little bit because we can't just pop it back into place from

00:17:43.840 --> 00:17:49.039
here we can't really get at it looks like we need to shut this down

00:17:47.600 --> 00:17:53.320
pull it off put it back into place screw it in and

00:17:50.799 --> 00:17:58.960
then put it back in which sucks if i take the weight and you slide

00:17:57.200 --> 00:18:02.880
it forward or the other way around here i'll use my leg up there we go it's up

00:18:01.200 --> 00:18:08.480
just put the put the rail in don't move wait put what rail in

00:18:05.760 --> 00:18:11.600
well i can't see it oh my god like this oh i think we got it i think we got it

00:18:09.760 --> 00:18:15.440
let go let go yeah this actually went pretty okay

00:18:13.280 --> 00:18:19.440
situations like this are when i'm so glad we developed this screwdriver i got

00:18:17.760 --> 00:18:25.520
the nice strong magnet holding it in place almost no

00:18:22.080 --> 00:18:26.559
back force required to spin the ratchet

00:18:25.520 --> 00:18:30.880
like is it in there easy yeah it's in oh my

00:18:28.799 --> 00:18:35.200
god wait what how are we supposed to get it from here down to there we got it up

00:18:32.880 --> 00:18:38.320
there somehow so it's just like a really heavy storenator why don't we just put

00:18:37.039 --> 00:18:43.679
the other one on top that says this cart can take 600 pounds no we're not doing that okay

00:18:41.120 --> 00:18:48.320
that's right what are the odds uh that it just goes right in

00:18:46.160 --> 00:18:52.160
no the angle is still too much okay i'm getting under it boys put the right side

00:18:50.640 --> 00:18:58.000
in a bit more okay good good here oh my god it's it yeah it's on it's on the top

00:18:54.559 --> 00:18:58.720
okay go ahead you can push it in

00:18:58.000 --> 00:19:03.600
hi all right now for the hard one yeah

00:19:01.360 --> 00:19:07.919
this is more of a Linus sized server yolo

00:19:06.000 --> 00:19:11.840
well it's easy to yolo when somebody else already did all the work yeah good

00:19:10.480 --> 00:19:17.520
work Jake there goes the brain and what's really

00:19:14.320 --> 00:19:22.080
wild is that if we want more capacity

00:19:17.520 --> 00:19:24.000
more than 90 180 270 drives with the

00:19:22.080 --> 00:19:26.799
cards we have in now we could do another one and then i think there's room for

00:19:25.360 --> 00:19:30.320
one more card right i would say right now with no hbas we

00:19:28.799 --> 00:19:34.559
could probably add two and it would still be pretty good performance

00:19:32.080 --> 00:19:42.240
um all right let's uh let's put in the hard one ready one two three no yes

00:19:40.320 --> 00:19:46.320
what weak all right still weak now that they're in

00:19:44.480 --> 00:19:52.640
there working on these is glorious you just press the buttons

00:19:49.840 --> 00:19:56.160
pull them out boom funny story when we first put this in i was like ah it's so

00:19:54.880 --> 00:20:00.799
heavy it doesn't really need to be bolted down does it i was told yes in

00:19:58.559 --> 00:20:04.720
fact it does need to be bolted down and this

00:20:01.840 --> 00:20:10.240
is why because if we had a handful of these hanging off the front of it and it

00:20:07.360 --> 00:20:11.919
wasn't bolted into the concrete

00:20:11.919 --> 00:20:16.960
we'd have a bit of a problem bit of an update here we shucked all those drives

00:20:15.360 --> 00:20:20.799
put them in the servers i plugged a bunch of stuff in some power and we have

00:20:19.120 --> 00:20:23.760
blinky lights now do i get to press the button yeah it's on the right side give

00:20:22.160 --> 00:20:28.000
them a sec everybody in the next square mile is going to hear this they're just

00:20:25.600 --> 00:20:31.120
they're they're spinning up boys wow it's actually not as bad in here

00:20:29.840 --> 00:20:35.039
come outside let's uh let's shut the door yeah sure

00:20:33.360 --> 00:20:37.600
you can hear how awful it is still through the door

00:20:38.799 --> 00:20:44.640
oh yeah we wouldn't be able to shoot with those things going

00:20:42.240 --> 00:20:47.440
but they don't go okay that's fine oh you just got to add another layer of

00:20:46.240 --> 00:20:51.760
insulation there's no space Jake well this is the

00:20:50.080 --> 00:20:57.120
backup plan right because there's a thermostat in here that will start up a

00:20:54.799 --> 00:21:00.960
fan that exhausts out the top and will pull air passively through this grill in

00:20:59.200 --> 00:21:05.039
the event that the air conditioning completely fails do you remember that

00:21:02.960 --> 00:21:08.799
the shroud for this thing i made that in metal shop when i was still in high

00:21:06.400 --> 00:21:12.080
school that's hilarious okay that was it it's up so far only 20 drives are

00:21:10.640 --> 00:21:16.880
showing up but Jake tried a thing which is a what a

00:21:14.159 --> 00:21:20.640
domain change we changed some cables and i changed the jbods from three zone to

00:21:19.520 --> 00:21:27.840
one zone are you going to click it yes

00:21:23.679 --> 00:21:27.840
okay thank god okay one of these

00:21:28.280 --> 00:21:34.480
145 drives god damn brother i mean 45 drives is

00:21:33.520 --> 00:21:39.520
nice but 145 drives

00:21:36.720 --> 00:21:43.520
now that's walking around hard dress i'm just going to try this it's probably

00:21:41.039 --> 00:21:46.480
a bad idea oh i'm just going gonna do it i'm gonna try to import the pools it

00:21:44.960 --> 00:21:49.760
should find both of them there's two discreet pools on those existing drives

00:21:48.400 --> 00:21:54.960
yeah and there's a bunch of extra drives so let's see what happens what are the odds this is just gonna be like yeah

00:21:52.640 --> 00:21:58.480
your ancient ass pool and just well it's not they're not that ancient i think

00:21:56.240 --> 00:22:04.159
they've all been recreated now okay but actually the crazy thing is i took like

00:22:01.280 --> 00:22:08.559
2015 pools from a version of zfs that was not recommended for production and

00:22:06.400 --> 00:22:13.919
had not been updated and i put it in like 2022 ass trunas

00:22:11.760 --> 00:22:18.320
it just imported it just upgraded for some reason it was super slow like

00:22:15.360 --> 00:22:21.280
scrubs would take like a week which is really weird yeah but there could be

00:22:19.760 --> 00:22:25.440
kind of flaky drives it wasn't the drives the drives were fine i moved all

00:22:23.120 --> 00:22:28.799
the data off deleted the pool made it and then brought it back and it was fine

00:22:26.799 --> 00:22:33.440
it might just be like a fragmentation thing or

00:22:30.080 --> 00:22:36.320
oh let's see did something and

00:22:33.440 --> 00:22:41.840
oh look at that go fast is the 12 drives i think got it i wish i saved these ids

00:22:38.640 --> 00:22:44.000
before i did this let's see what happens

00:22:41.840 --> 00:22:48.640
does this mean vault is just going to be like well i'm back baby temp vault will

00:22:46.480 --> 00:22:51.679
be back the read-only new vault because we're still recovering a few things

00:22:49.840 --> 00:22:55.440
that's why one of the jbods is still mia we'll be back just like that and i can

00:22:53.679 --> 00:23:01.520
just recreate new shares i mean the ip will be different but i can just recreate the shares

00:22:58.880 --> 00:23:05.120
zfs is cool man don't put that there this is a really nice shirt though oh is

00:23:02.880 --> 00:23:08.559
that from ltdstore.com it sure is it will be oh is it not out yet i don't

00:23:07.039 --> 00:23:14.000
know i don't think so have you announced it on rancho i don't think so 30 hours

00:23:11.039 --> 00:23:17.520
later wow the pool's finally imported holy crap

00:23:16.000 --> 00:23:22.080
super micro massive shout out thank you for sending

00:23:19.360 --> 00:23:26.960
over the gear it's amazing we did have a little bit of trouble with setting it up

00:23:24.480 --> 00:23:31.120
with three domains yeah it's for some reason they're only working in single

00:23:29.120 --> 00:23:34.159
zone mode that means all 90 drives are going over one cable i don't know why

00:23:32.960 --> 00:23:38.400
because we had it set up on the bench like a couple weeks ago with three zones

00:23:36.240 --> 00:23:41.360
anyway it's accessible we're ready to copy some files and hopefully that's

00:23:40.000 --> 00:23:46.080
something they can sort out either through helping us set up the configuration or through firmware yeah

00:23:44.080 --> 00:23:49.200
it's probably just we're missing something this is one of those solutions

00:23:47.919 --> 00:23:54.559
where it's kind of like you just know what you're doing if you're doing it you're an integrator who

00:23:51.840 --> 00:24:00.159
sets these things up every day week after week and that's not us but i

00:23:57.520 --> 00:24:04.320
copied here i copy the file let's do it look i'm copying a file from it to this

00:24:02.400 --> 00:24:08.159
like hundreds of megabytes a second it's not the most impressive scenes ever this

00:24:06.400 --> 00:24:12.159
is a SATA SSD and we're screen recording to it now so

00:24:09.760 --> 00:24:17.919
it's like ah but hey remember guys this is with

00:24:15.200 --> 00:24:24.559
ancient like degraded degraded-ass fragmented v-devs once we strip all the

00:24:22.159 --> 00:24:28.559
data off of them rearrange them reshuffle them set them up fresh and

00:24:26.640 --> 00:24:31.440
dump all the data back onto them we should be able to be in the neighborhood

00:24:30.000 --> 00:24:35.279
of a Gigabyte a second which is all we're going to have for for sure well yeah but that's all we're going to have

00:24:34.000 --> 00:24:39.679
yeah that's all we're going to do network interface it's 100 oh that's

00:24:37.279 --> 00:24:42.799
right it is but here look at that 147 drives never in actual production that's

00:24:41.279 --> 00:24:47.120
amazing and then we'll add 60 more and it'll be

00:24:44.400 --> 00:24:51.360
over 200. and then we'll add 50 more and it'll be 250. and then we'll add this

00:24:49.039 --> 00:24:55.120
segway to our sponsor i fixed its manta kit has a 112 steel bits and comes

00:24:53.679 --> 00:24:59.760
complete with standard bits like phillips and flathead as well as more

00:24:57.120 --> 00:25:04.559
niche options like pentalobe hex and tri-point bits it also includes two

00:25:02.080 --> 00:25:08.400
aluminum screwdriver handles a weighty quarter inch drive for full size

00:25:06.400 --> 00:25:12.080
applications and a smaller four millimeter for precision work both

00:25:10.240 --> 00:25:16.240
drivers feature magnetic bit sockets knurled grips and silky smooth spinning

00:25:14.320 --> 00:25:20.559
tops you can even use the toolkit's lid as a built-in storage tray best of all

00:25:18.480 --> 00:25:24.159
every ifixit tool is backed by a lifetime warranty so if something breaks

00:25:22.080 --> 00:25:30.240
they will replace it for free so visit ifixit.com forward slash LTT and pick up

00:25:26.799 --> 00:25:32.320
your manta kit today monta if you guys

00:25:30.240 --> 00:25:37.679
enjoyed this video maybe you'll enjoy the one where we tried to deploy wanik

00:25:34.559 --> 00:25:38.960
no no that was awful okay sure yeah sure

00:25:37.679 --> 00:25:43.840
to watch that or the the previous one where we just looked at jbots because they're cool