WEBVTT

00:00:00.080 --> 00:00:07.200
yeehaw this is one bucking bronco of a server okay that didn't work but it

00:00:04.880 --> 00:00:12.080
doesn't matter because we have a serious problem and this

00:00:08.880 --> 00:00:13.920
this is one serious solution

00:00:12.080 --> 00:00:18.400
in spite of all the issues we encountered with our archival storage

00:00:15.920 --> 00:00:24.800
system dubbed the vault we've actually been able to recover nearly all of the

00:00:21.119 --> 00:00:27.920
data spread across the 2.4 petabytes of

00:00:24.800 --> 00:00:30.400
raw storage that made it up except that

00:00:27.920 --> 00:00:33.760
because of the recovery process and the sheer amount of capacity required to

00:00:32.640 --> 00:00:39.920
hold it all that data is now scattered to the high

00:00:36.960 --> 00:00:44.719
winds across the five discrete servers that are required to hold it some of

00:00:42.320 --> 00:00:49.200
which are getting close to a decade old we had no desire to return to the

00:00:46.960 --> 00:00:52.079
admittedly imperfect setup we had before which means

00:00:50.320 --> 00:00:58.079
it is time for oh oh mother vault

00:00:54.960 --> 00:00:59.039
so called because this is one giant

00:00:58.079 --> 00:01:05.519
mother the final deployment is gonna be not one

00:01:02.079 --> 00:01:09.200
not two but three of these each capable

00:01:05.519 --> 00:01:11.600
of housing nearly two petabytes of hard

00:01:09.200 --> 00:01:17.280
drive face storage is it time to bring back holy did it ever go away no it

00:01:14.479 --> 00:01:21.200
didn't just like our sponsor crucial don't just work faster work better when

00:01:19.439 --> 00:01:25.119
you need to open transfer or download files faster get some great RAM with

00:01:23.360 --> 00:01:30.880
less lag time and more efficiency with crucial get your crucial ddr5 RAM today

00:01:27.759 --> 00:01:30.880
at the link down below

00:01:39.840 --> 00:01:46.399
before we get to the new stuff we're going to have to get you guys up to

00:01:43.280 --> 00:01:49.439
speed back in 2015 we built the first

00:01:46.399 --> 00:01:52.240
petabyte project or vault cluster it was

00:01:49.439 --> 00:01:56.799
composed of two 45-drive 60-base storinator servers holding 120

00:01:55.200 --> 00:02:02.240
10 terabyte hard drives giving us a total capacity of 1.2 petabytes raw

00:02:00.640 --> 00:02:06.079
these were combined into a single network share using a file system called

00:02:04.479 --> 00:02:12.879
glosterfest a few years later we built new vault or

00:02:09.039 --> 00:02:16.160
petabyte project 2 which used 75

00:02:12.879 --> 00:02:18.480
16 terabyte drives across two more 45

00:02:16.160 --> 00:02:22.879
drive storinators again they were joined together with bluster however since

00:02:21.040 --> 00:02:27.840
those two clusters were built several years apart we ended up setting them up

00:02:25.040 --> 00:02:32.800
as two discrete clusters that meant we had two separate network storage shares

00:02:30.560 --> 00:02:36.480
which you can kind of think of as having two separate drives in your system so if

00:02:35.120 --> 00:02:39.440
you want to find a file and you don't know which drive it's on chances are

00:02:38.319 --> 00:02:44.319
you're going to have to search through them separately this was fine at the

00:02:42.480 --> 00:02:48.640
time because we could usually guess which server a project was on based on

00:02:46.640 --> 00:02:52.720
how old it was and honestly searching just two sources is not that difficult

00:02:50.879 --> 00:02:57.040
however if we went to scale that up again and build a third eventually a

00:02:54.959 --> 00:03:00.080
fourth cluster well would start to get pretty annoying not to mention

00:02:58.480 --> 00:03:04.800
inefficient for the staff who have to use the freaking thing the other issue

00:03:02.080 --> 00:03:09.920
is that managing even a single system takes a good amount of time and every

00:03:07.680 --> 00:03:15.680
time we scale it up in this manner it's another set of components CPU

00:03:12.239 --> 00:03:17.360
motherboard RAM power supplies ah oh my

00:03:15.680 --> 00:03:22.239
god this is really heavy that's the heaviest it's 130 kilograms i think

00:03:19.840 --> 00:03:27.599
that's when it's full holy crap you ready

00:03:24.800 --> 00:03:33.000
oh my god i told you you had the heavy side

00:03:28.560 --> 00:03:33.000
yeah but what the hell

00:03:33.040 --> 00:03:40.080
anyway more hardware means more potential issues more potential service

00:03:38.080 --> 00:03:43.760
so with that in mind we've been looking for an alternative solution for some

00:03:42.159 --> 00:03:47.280
time and while we could have continued to use gloucester fs and expand the

00:03:45.440 --> 00:03:52.480
existing clusters just don't really have anyone on staff

00:03:49.680 --> 00:03:56.879
who is familiar with it and it's not that user friendly i mean heck even

00:03:54.720 --> 00:04:00.400
45 drives the company who provided us with the original petabyte project have

00:03:58.640 --> 00:04:04.560
actually switched primarily to a different cluster solution called seth

00:04:03.439 --> 00:04:09.360
now ceph is cool really cool we use it for

00:04:06.959 --> 00:04:12.959
Floatplane for example and it's a bit better for scaling up but it shares a

00:04:11.439 --> 00:04:16.160
lot of the same problems of our old setup we don't have a ton of people on

00:04:14.560 --> 00:04:22.000
staff who are experienced with it and it would have the same or similar maintenance requirements so instead

00:04:19.919 --> 00:04:25.520
we've opted to switch to a configuration that many home Labs and data centers

00:04:24.639 --> 00:04:28.960
adore one i personally have actually never

00:04:27.680 --> 00:04:32.000
used the jbod

00:04:33.040 --> 00:04:38.000
you can fit so much hard drive in here

00:04:38.479 --> 00:04:51.040
this magnificent beast is the super micro 947 he1c dash 42k 05 jb od it's a

00:04:48.160 --> 00:04:57.919
90 bay chassis that as the name jbot implies is quite literally just a bunch

00:04:54.320 --> 00:05:00.240
of disks there's actually

00:04:57.919 --> 00:05:03.280
no computer in there which raises the question

00:05:01.360 --> 00:05:06.880
what is here in the back where the compute module or

00:05:04.960 --> 00:05:10.880
multiple compute modules would normally be this is it

00:05:08.639 --> 00:05:15.199
this is what passes for the computer in this chassis and there there

00:05:13.440 --> 00:05:19.440
is some computing going on here we've got some kind of asic

00:05:17.440 --> 00:05:24.160
that is i mean is this an fpga even i'm not sure it would be for sas though yeah

00:05:22.000 --> 00:05:28.320
it could be could are they doing enough oh oh gosh uh let's see you don't want

00:05:27.360 --> 00:05:33.759
to break it no i don't want to break it i just want to see it there's a button cell battery

00:05:31.360 --> 00:05:36.720
in there this chassis does have ipmi so that's what the battery would be for

00:05:34.720 --> 00:05:40.479
keeping track of time so there's going to be some compute for that probably on

00:05:38.400 --> 00:05:44.560
the bottom board but the rest of it should just be sas stuff there's that

00:05:42.400 --> 00:05:50.240
ipmi management port and then the only other i o on this machine is these six

00:05:47.680 --> 00:05:55.840
mini sas external ports what are these 88 so each of these is four lanes of 12

00:05:53.039 --> 00:06:00.400
gigabit sas so it's 48 gig per port and you can use up to three at one time for

00:05:58.160 --> 00:06:06.639
connection to the head server which is what is that 144 gigabit okay

00:06:04.000 --> 00:06:11.440
casual perfect then oh you can see the a-speed uh chip for

00:06:09.120 --> 00:06:15.919
the ipmi right there sit in there there'll be some uh non-volatile memory

00:06:13.600 --> 00:06:18.880
on there too somewhere it might actually be this

00:06:17.039 --> 00:06:23.600
that looks like a nand package these are some chungus cooling fans look

00:06:22.000 --> 00:06:28.639
how thick they are there's two in there right 80 millimeter fans i actually

00:06:26.000 --> 00:06:32.639
don't think so it's dual blade but no i think these are individual fans Jake oh

00:06:31.120 --> 00:06:37.919
no it's got to be two it's got to be two there's two discrete connectors either

00:06:34.560 --> 00:06:39.680
way i guarantee you they absolutely rip

00:06:37.919 --> 00:06:44.160
and then aside from i guess these are probably the sas and that would be your

00:06:41.199 --> 00:06:48.479
power that's pretty much it very very simple and that's with intention it's

00:06:46.240 --> 00:06:52.560
designed to be reliable right our other module here is

00:06:50.319 --> 00:06:56.319
well doesn't have a whole lot going on to be perfectly honest with you what on

00:06:54.400 --> 00:07:03.039
earth does this do you've got a small controller board that has a couple SATA

00:06:58.240 --> 00:07:05.680
ports a couple NVMe ports and

00:07:03.039 --> 00:07:09.919
more power and whatever the crap this is this is probably PCIe i think when you

00:07:07.840 --> 00:07:15.039
buy the computer version of this these two nvmes would come over here somewhere

00:07:12.479 --> 00:07:19.919
if you just had a single compute node but yeah i'm not really sure

00:07:17.440 --> 00:07:23.680
okay you could have a jbod that's something called dual path now this is a

00:07:21.599 --> 00:07:26.800
single path which means there's one path for the data to go from the drives to

00:07:25.360 --> 00:07:29.599
whatever computer it's plugged into because remember there's no computer in

00:07:28.400 --> 00:07:33.840
here it's got to plug into a different computer yeah but in a dual path there's

00:07:32.319 --> 00:07:38.160
two connections and you'll actually have redundant sas expanders which is the

00:07:35.599 --> 00:07:41.199
part that splits into multiple drives you'll have redundant connections so

00:07:39.840 --> 00:07:44.080
there'll be six more back here and you'll have another ipmi and the reason

00:07:42.960 --> 00:07:48.720
for that is you can have a high availability controller so you could have controller 1 and controller 2. and

00:07:47.120 --> 00:07:52.639
if controller 1 had problems it would just fail over to controller 2 and you'd

00:07:50.479 --> 00:07:56.800
have better availability the problem is that in order to take advantage of that

00:07:54.319 --> 00:08:01.599
we need to use sas drives and that's a hassle well it's not only

00:07:59.520 --> 00:08:08.319
is it a hassle but we already have 3.6 petabytes of seda drives that's a hassle

00:08:04.000 --> 00:08:08.319
that's a that's like 150 000 hassle

00:08:08.560 --> 00:08:13.520
not doing it now let's take a look at the sas

00:08:11.680 --> 00:08:18.000
expanders that make this magic happen naturally they're hot swappable because

00:08:15.280 --> 00:08:23.199
servers and in a nutshell each of these takes in four of those sas 12 gigabit

00:08:21.039 --> 00:08:30.160
per second lanes and then is able to split that bandwidth to up to 30 flippin

00:08:26.800 --> 00:08:33.760
drives so we only actually need three of

00:08:30.160 --> 00:08:36.640
these to populate the entire 90 bay

00:08:33.760 --> 00:08:40.320
capacity of this j-bond by the way Jake was talking about how

00:08:38.080 --> 00:08:42.800
you could have a dual path well you'd need

00:08:41.360 --> 00:08:48.880
three more expanders then but we don't need them Jake do you have any idea what these connectors are called i have no

00:08:45.839 --> 00:08:51.200
idea shout out them because these things

00:08:48.880 --> 00:08:57.920
are flipping cool look at the way these are joined back onto the board

00:08:54.000 --> 00:09:00.880
that is dense with two s's that's oh my

00:08:57.920 --> 00:09:05.360
god the pins on the bottom right holy crap the next thing i need Jake is for

00:09:03.360 --> 00:09:09.839
you to get me the head it's it's right there right here you can

00:09:07.360 --> 00:09:14.480
think of your jbot kind of like an external hard drive or in this case

00:09:12.320 --> 00:09:19.360
many many external hard drives which means that as Jake alluded to before we

00:09:16.959 --> 00:09:24.560
need to have a computer somewhere well that is where the head or the controller

00:09:22.240 --> 00:09:28.800
server comes into play this particular machine is a placeholder supermicro is

00:09:26.399 --> 00:09:34.000
actually sending us over a dual AMD epic milan server that's a little more

00:09:31.600 --> 00:09:39.360
tailored for this kind of application so instead of being full of NVMe drive bays

00:09:36.880 --> 00:09:45.680
in the front it's got its PCIe lanes allocated to allow for lots of hbas and

00:09:42.399 --> 00:09:47.839
network connectivity in the back because

00:09:45.680 --> 00:09:52.560
hard drives may be slow but when you hook up enough of them you can be

00:09:49.440 --> 00:09:55.360
pushing some serious freaking data and

00:09:52.560 --> 00:10:00.240
this is going to be connected to up to 270 drives

00:09:58.320 --> 00:10:04.240
not 255 i think that's what it works out to 90 times three is 200 and we don't

00:10:02.399 --> 00:10:07.839
have 98. oh but yeah i mean you could you get you could say you could add more

00:10:05.760 --> 00:10:11.440
drives it's just jbod um so cool right

00:10:09.760 --> 00:10:15.279
in the meantime for the purposes of our demo

00:10:13.120 --> 00:10:20.079
we've loaded this machine up with three of these broadcom hbas these things

00:10:18.000 --> 00:10:24.079
actually contain very little logic compared to raid cards which had to

00:10:22.079 --> 00:10:27.760
perform parity calculations on the data running through them

00:10:25.440 --> 00:10:32.959
these do almost nothing they just take your bandwidth from your PCIe slot in

00:10:29.920 --> 00:10:34.720
this case PCIe 8x gen 4 slot run it

00:10:32.959 --> 00:10:40.399
through a sas controller and then that breaks out into for a 12

00:10:37.680 --> 00:10:43.519
16 sas connections do you wanna see my wonderful cardboard yeah do you wanna

00:10:41.920 --> 00:10:47.680
see my wonderful car do you wanna see my wonderful cardboard yeah

00:10:45.600 --> 00:10:51.120
this math actually makes a lot of sense if you have 90 drives and we'll say

00:10:49.279 --> 00:10:55.760
optimistically they're running at 250 megabytes a second that's 22.5 gigabytes

00:10:53.920 --> 00:11:00.240
a second it's pretty fast conveniently three sets of 48 gigabit sas cables is

00:10:58.160 --> 00:11:04.720
around 18 gigabytes a second and even more conveniently PCIe gen four by eight

00:11:03.040 --> 00:11:07.760
it's roughly 16 gigabytes a second in either direction so we're going to be

00:11:06.160 --> 00:11:11.839
giving up some of the theoretical maximum performance of our drives but

00:11:09.440 --> 00:11:15.360
given that this is only under perfect conditions with brand new

00:11:13.519 --> 00:11:18.959
high-performance drives with reading and writing sequential data with nothing on

00:11:17.760 --> 00:11:22.880
them this will be lots

00:11:21.279 --> 00:11:27.040
and that's times three because we have three j bonds

00:11:24.560 --> 00:11:30.320
we're never gonna get anywhere close to this maybe one Gigabyte a second i'd

00:11:29.040 --> 00:11:34.880
like to point out something else that's convenient what check this out

00:11:32.640 --> 00:11:40.160
see that ratchet back pressure wow i can start a screw with it and in spite of

00:11:37.760 --> 00:11:44.240
the silver shaft look at that i used it and my dick didn't fall off

00:11:42.720 --> 00:11:49.680
i actually think the silver shaft looks better wow all right

00:11:48.000 --> 00:11:52.800
he's actually been one of the bigger skeptics internally about this whole

00:11:51.440 --> 00:11:56.959
project i'm just a little bit critical

00:11:54.959 --> 00:12:01.680
but that ratchet i want it to be good okay you can sign up for a notification when

00:11:59.600 --> 00:12:05.839
it comes in stock lttstore.com try to sound less stressed in the meantime we

00:12:03.760 --> 00:12:09.519
also have shirts and stuff cash flow's a little tight now you might look at this

00:12:07.760 --> 00:12:13.040
machine and think that's ludicrous overkill surely the one that super micro

00:12:11.600 --> 00:12:17.920
is going to send is a little more pedestrian but that's not actually the case even

00:12:16.320 --> 00:12:21.519
though we're not running super high speed NVMe drives when you're

00:12:20.480 --> 00:12:26.000
hooking up hundreds of hard drives with the

00:12:23.760 --> 00:12:30.720
potential to expand to hundreds of more you need CPU and RAM galore more than

00:12:29.680 --> 00:12:36.880
you'd think in fact we actually have two 32 core

00:12:33.519 --> 00:12:40.800
epic 75 f3 processors those are the

00:12:36.880 --> 00:12:42.320
frequency oriented 280 watt per cpus and

00:12:40.800 --> 00:12:46.000
we're going to have around a terabyte of RAM with the potential to expand that

00:12:44.160 --> 00:12:50.399
down the road because we will be using zfs and zfs greatly benefits from RAM

00:12:48.880 --> 00:12:53.040
for recaching we'll also have NVMe drives and stuff

00:12:51.920 --> 00:12:58.959
but that's tbd you'll have to get subscribed for the next video when we actually deploy the whole are we doing

00:12:56.000 --> 00:13:03.680
tiering no oh well maybe one day it's possible let you know a secret

00:13:00.959 --> 00:13:07.200
we might build a highly available wanik and set this up as a tiered

00:13:05.680 --> 00:13:13.519
archival layer entirely transparent to the user

00:13:09.760 --> 00:13:15.519
so just one drive yeah letter one drive

00:13:13.519 --> 00:13:19.920
oh my god these are just these are just drive sleds i thought

00:13:17.760 --> 00:13:23.279
these were boxes of hard drives Jake no those are the traits i've been deceived

00:13:21.680 --> 00:13:26.480
do you want to see what they trade like we were talking about how we wanted to

00:13:24.880 --> 00:13:30.399
discuss yeah we also need the high performance CPU for zfs compression i

00:13:28.959 --> 00:13:33.680
was like oh well we'll just dump a bunch of drives in here and then we'll talk

00:13:32.160 --> 00:13:38.560
about okay we've got this much capacity but we're going to use compression and then Jake's like oh we've only got 13

00:13:36.480 --> 00:13:43.360
drives to put in it for the demo come on Jake it's almost like all our hard

00:13:40.800 --> 00:13:49.839
drives are deployed holding data temporarily from the old vault yeah

00:13:45.760 --> 00:13:51.440
almost okay well i guess it's a 13 drive

00:13:49.839 --> 00:13:56.560
demo if you want ed to yell at you we could take apart delta 5. that's got new vault

00:13:55.120 --> 00:13:59.519
on it i think and some of old vault when ed yells at

00:13:58.720 --> 00:14:04.880
me wendell's still working on delta 3 so that's not available

00:14:03.120 --> 00:14:08.240
temp vault we could do temp vault but then ed would also yell at you probably

00:14:06.639 --> 00:14:12.079
it's not fit 13 drive seems fine

00:14:10.399 --> 00:14:18.480
all right we found another solution to some test hard drives this here

00:14:13.760 --> 00:14:20.560
storinator was full of 60 spinnerino's

00:14:18.480 --> 00:14:25.920
and now they're in here it has made the balance of this system

00:14:22.320 --> 00:14:27.199
super sketchy like

00:14:25.920 --> 00:14:30.680
oh it gets worse

00:14:31.040 --> 00:14:36.160
look look at that look it's it

00:14:36.160 --> 00:14:41.519
oh god oh i can't reach okay there we go but

00:14:40.079 --> 00:14:45.440
there you go there's 60 hard drives they're kind of in a weird orientation

00:14:43.839 --> 00:14:49.519
because there's a kind of a poop mix of drives in here there's some tens there's

00:14:47.279 --> 00:14:54.720
some 12s there's some 16s i think there's even 120 so i just wanted to

00:14:51.680 --> 00:14:56.399
kind of separate those but this

00:14:54.720 --> 00:15:00.079
is our test now these 60 drives do already

00:14:58.800 --> 00:15:03.680
have an array on them and i think it's about 70 full this is some of our

00:15:02.079 --> 00:15:08.720
existing archival data but they're going to be set up as four v devs that are

00:15:05.600 --> 00:15:10.240
raid z2 15 drives wide not great for

00:15:08.720 --> 00:15:14.480
performance i don't think we're gonna see anything crazy uh in terms of

00:15:12.320 --> 00:15:18.320
numbers out of this machine but it is kind of indicative of what we're

00:15:16.240 --> 00:15:23.760
gonna have it set up as we're probably gonna switch to 10 drive raid z2s

00:15:21.120 --> 00:15:27.760
just to make it a little bit faster but this should still be pretty dang fast

00:15:25.839 --> 00:15:33.199
now in order to get a connection between our server and our jbod we have these

00:15:30.160 --> 00:15:34.959
these cool mini sas hd external cables

00:15:33.199 --> 00:15:39.600
from infinite cables i think supermicro also sent us some by default this jbod's

00:15:37.360 --> 00:15:43.040
configured as a single zone now you can do kind of cool stuff with zoning in a

00:15:41.519 --> 00:15:46.720
jbot for instance you could split this up into two zones and have two

00:15:44.880 --> 00:15:50.880
controller servers being serviced by one jbot so you'd have 45 drives on either

00:15:49.279 --> 00:15:55.120
you can also do three zones so that'd be 30 30 30 do three controller servers but

00:15:53.440 --> 00:15:58.800
they can't access the other zone's data you'd only get the

00:15:57.360 --> 00:16:03.440
zone that you're actually physically attached to so you'd have

00:16:00.639 --> 00:16:07.519
a c or e on the back of this server now in our case we're going to be using

00:16:05.120 --> 00:16:13.360
three zones but to one controller server and the reason for this is it allows us

00:16:10.320 --> 00:16:16.079
instead of getting one cable which is 48

00:16:13.360 --> 00:16:21.199
gigabit bandwidth we now get two and then three for 144 gigabit which again

00:16:19.199 --> 00:16:25.199
is probably overkill but what do we do that's not overkill

00:16:23.199 --> 00:16:29.519
hi you might need these like like for realsies

00:16:27.680 --> 00:16:35.800
no don't just wait this is just the power stuff that's the power supplies

00:16:31.680 --> 00:16:35.800
that's like full tilt though

00:16:37.920 --> 00:16:44.720
whoa this is ludicrously loud so i put it in

00:16:42.399 --> 00:16:50.000
the server room with the door shut you can hear it in the bedroom

00:16:47.920 --> 00:16:54.320
but this is like they're actually at a hundred percent right now they should

00:16:51.680 --> 00:16:59.360
never run it never and with the 60 drives i already tried it like running a

00:16:56.320 --> 00:17:03.440
load and they barely even spin up okay

00:16:59.360 --> 00:17:05.679
that's good yeah i got a little worried

00:17:03.440 --> 00:17:09.199
is if the air conditioning ever fails in the server room

00:17:07.199 --> 00:17:12.400
we'll know because we'll hear this everywhere

00:17:11.280 --> 00:17:14.640
there we go oh

00:17:15.439 --> 00:17:21.839
give it a sec okay coming back to planet earth

00:17:19.600 --> 00:17:26.160
that's a lot more reasonable i mean i guess that's what fans that

00:17:24.240 --> 00:17:29.760
thick will do for you well i mean when you have 90 hard drives you're talking

00:17:27.919 --> 00:17:34.080
like at least a thousand watts yeah that's a lot of power i mean what are

00:17:31.600 --> 00:17:38.400
these power supplies uh they're 2 000 watt on 208 volt 2 000 watt power

00:17:37.280 --> 00:17:43.280
supplies um so to have like actual power supply

00:17:41.120 --> 00:17:47.039
redundancy you need to be running 208 volt wow otherwise you'll probably use

00:17:45.679 --> 00:17:52.720
enough power that it needs to split across the two so we might actually have to rewire our server room to be 208 volt

00:17:51.200 --> 00:17:57.679
because for some reason it wasn't wired that way in the first place i don't know i don't remember why brian there's

00:17:55.360 --> 00:18:02.400
probably a reason yeah there's a reason and i bet you there's a way to fix it oh

00:17:59.760 --> 00:18:05.760
my god yeah you can fix anything with money mm-hmm you have lots of that

00:18:04.559 --> 00:18:10.160
right now right no oh well that's fine we'll figure it out

00:18:08.559 --> 00:18:14.000
you know what no we should just put these in lab two because that one won't

00:18:12.000 --> 00:18:18.240
be too late right time to put more insulation on the walls

00:18:16.400 --> 00:18:21.520
jesus with nails yeah i'll show you that quick config here this is where you can

00:18:19.760 --> 00:18:25.039
change the zone so there's single zone two zone or three zone again like i

00:18:23.440 --> 00:18:28.480
explained before you could use that for multiple controllers or to increase

00:18:27.039 --> 00:18:32.000
throughput we're doing the throughput route of course that's pretty much all

00:18:30.320 --> 00:18:35.679
you can do in here there's no remote control like there's no there's no

00:18:33.840 --> 00:18:39.840
screen it doesn't have a display output right yeah power it's just a big

00:18:38.559 --> 00:18:44.080
dumb big discs

00:18:42.720 --> 00:18:48.799
system critical i don't know why probably because you pulled the power supply probably plug it back in i'm

00:18:46.559 --> 00:18:52.880
sorry about that buddy here you go yeah there's your medicine right in the butt

00:18:51.120 --> 00:18:57.440
suppository medicine power cable now tell me something if we

00:18:55.440 --> 00:19:01.200
decided to go high availability in the future nothing would actually prevent us

00:18:59.520 --> 00:19:06.559
from adding a second interface unit back there yes a

00:19:03.919 --> 00:19:10.480
second head server yes no problem right we need sas drives sorry not high

00:19:08.799 --> 00:19:14.960
availability at the oh cause dual path oh crap okay yeah the

00:19:13.600 --> 00:19:20.880
thing about this approach though is we've already proven we don't really depend on this yeah it's

00:19:19.360 --> 00:19:26.559
been like it's gotta be like eight months at this point that we haven't had it it's funny how many people were

00:19:24.000 --> 00:19:31.919
listening to us in the video saying yeah it's really optional retaining this data

00:19:29.280 --> 00:19:36.240
we lost it because we don't really care that much and we're like

00:19:33.440 --> 00:19:40.400
wise like no we haven't had it for like almost a year yeah at this point and

00:19:37.760 --> 00:19:44.720
it's been totally fine it is nice to have but here's nice to have if this

00:19:42.240 --> 00:19:48.480
controller server were to poop the bed or to have some problem well everything

00:19:46.320 --> 00:19:53.679
here should still be we can just unplug this we'll have some other server temporarily

00:19:51.600 --> 00:19:56.880
so like a cold spare put the hp yeah a basic spare server i mean you're talking

00:19:55.440 --> 00:20:01.440
maybe an hour of downtime like realistically i was looking at this and

00:19:59.039 --> 00:20:04.799
i thought maybe this was i saw four v devs i thought that was the four drives

00:20:03.280 --> 00:20:09.919
i i just saw it on the corner of my eye and i thought you had put the 30 terabyte nvmes in here

00:20:08.160 --> 00:20:13.039
we might though wendell's been talking about this special metadata device for a

00:20:12.080 --> 00:20:20.160
while and basically he was saying he has like half a petabyte with 24 terabytes of

00:20:17.760 --> 00:20:23.039
special metadata device in zfs and it fills

00:20:21.280 --> 00:20:27.120
so i'm like we might as well just put 430 terabytes in there and that'll give

00:20:25.039 --> 00:20:30.559
us 60 terabytes what does special metadata do so you know like file

00:20:29.039 --> 00:20:35.679
metadata and like the directory structure is that it's like searching really fast it makes searching way

00:20:33.760 --> 00:20:39.280
faster oh that's cool so stores metadata like where files are

00:20:38.000 --> 00:20:44.640
she's talking about it right here it's literally literally Windows this is the first thing that comes up hi wendell

00:20:42.480 --> 00:20:50.480
yeah yeah but here's the thing is we can have a level two arc we can have a log

00:20:47.120 --> 00:20:52.799
device and we can have special data yeah

00:20:50.480 --> 00:20:55.840
so he's saying 172 terabytes of space is five terabytes of metadata okay but

00:20:54.720 --> 00:20:59.280
here's the thing we don't have to have all of it metadata

00:20:57.840 --> 00:21:03.039
in there like it'll pull it when it needs it right okay yeah by default this

00:21:01.360 --> 00:21:08.400
includes all the metadata the indirect blocks of user data and any deduplication tables stored on that

00:21:06.240 --> 00:21:12.080
device so maybe deduplication if you are running deduplication this is probably

00:21:10.640 --> 00:21:16.000
really important to improving performance yeah

00:21:14.159 --> 00:21:21.520
but either way we'll try it as you can see it's about 70 full not ideal for

00:21:18.720 --> 00:21:25.120
performance and on top of that it's an existing pool

00:21:23.120 --> 00:21:29.679
um that it's set up in four v-depths right yeah so they're 15 wide yeah so

00:21:28.559 --> 00:21:34.480
really i'm expecting like maybe three to five

00:21:32.240 --> 00:21:38.159
gigabytes a second okay sequential like i i would not be surprised if that's

00:21:36.240 --> 00:21:40.720
where we end up because when we move to the final deployment we're gonna do 10

00:21:39.760 --> 00:21:45.679
wide which will be better for performance we'll do 10 wide raid z2s

00:21:43.600 --> 00:21:49.440
which is basically the same as 15 wide raid z3s yeah

00:21:47.679 --> 00:21:53.919
and then we'll obviously have a lot more drives if you've got 10 drives of which

00:21:51.840 --> 00:21:58.799
you could lose two of them before you any data loss that's kind of the same as

00:21:56.000 --> 00:22:03.679
having 15 drives of which you could lose three in terms of the ratio of the whole

00:22:01.200 --> 00:22:09.360
pool if you look at it from a wide perspective but it's probably slightly

00:22:06.720 --> 00:22:14.400
better to have more v devs if you run the probability analysis

00:22:12.000 --> 00:22:19.120
anyway so this no no we gotta explain the data set is set up with caching only

00:22:17.120 --> 00:22:22.880
metadata which is what we want the RAM cache is turned off uh let's

00:22:21.679 --> 00:22:27.679
give it a second there we go we're getting like let's just give it give it you know just warm up there bud come on

00:22:26.640 --> 00:22:31.840
okay that's about two two and two gigabytes

00:22:29.840 --> 00:22:35.039
john you can do better than that i didn't even look at it

00:22:33.120 --> 00:22:39.760
come on three three yeah it's not three sustained

00:22:36.720 --> 00:22:41.520
that's for sure 3.6 you can't just read

00:22:39.760 --> 00:22:45.360
the high numbers i'm reading just the high numbers that was 3.7 there for a

00:22:43.600 --> 00:22:50.159
second yeah it really doesn't work like that that's

00:22:47.200 --> 00:22:54.799
579. oh there's four thousand percent there's

00:22:52.159 --> 00:22:57.280
point three probably has something to do with the fact that the drives are pretty

00:22:56.080 --> 00:23:02.080
full already remember we have no NVMe caching we have

00:22:59.440 --> 00:23:08.159
no RAM caching like this is raw dog giant v dev just disc

00:23:06.000 --> 00:23:12.480
now tell me something hold on a second wait here's something i don't fully

00:23:10.080 --> 00:23:16.159
understand this is really cool for maintenance the fact that you can have

00:23:14.080 --> 00:23:20.320
the system running you don't have to pull all your cables

00:23:18.000 --> 00:23:24.720
out at the back in order to see yeah the cable's staying in the same spot super

00:23:22.080 --> 00:23:27.840
nice but how is all of this still connected these drives are blinking

00:23:26.240 --> 00:23:31.360
these are doing things oh it's just like a ribbon yeah giant

00:23:29.679 --> 00:23:38.080
ribbon cable in the back you see it back there oh that's super cool

00:23:34.880 --> 00:23:40.080
man that's amazing for maintenance

00:23:38.080 --> 00:23:44.320
so you can just go be like okay yeah i've got a bad drive and bay whatever

00:23:42.720 --> 00:23:47.760
pull this out you don't have to slide any servers out see you later because

00:23:46.000 --> 00:23:50.720
this is internal right you just pull it out and you grab that one we're gonna

00:23:49.120 --> 00:23:54.400
have to label them oh yeah or i might just do a spreadsheet honestly um yeah

00:23:52.559 --> 00:23:59.120
that would be fine if they were sas drives you have like uh sas

00:23:56.640 --> 00:24:04.240
enclosure service or something like that ses whatever anyways you can go and be

00:24:01.520 --> 00:24:08.080
like what's in that slot or by drive you can say what slot is that driving and

00:24:06.159 --> 00:24:11.440
it'll tell you but uh since they're saying it drives they'll just show us

00:24:09.440 --> 00:24:15.760
not connected let's put this back in this is making me

00:24:12.960 --> 00:24:21.600
anxious let's do that i'm like i just oh boy okay yeah so we're doing 16 jobs in

00:24:18.640 --> 00:24:25.360
the i o depth of 16 block size one mag it's gonna be probably basically the

00:24:23.360 --> 00:24:30.799
same oh two thousand whew

00:24:27.760 --> 00:24:32.400
blow my skirt up

00:24:30.799 --> 00:24:35.760
more consistent though yeah

00:24:34.000 --> 00:24:39.120
like nope there it goes see you later

00:24:37.760 --> 00:24:43.600
in the grand scheme of things we're going to be writing to this thing like

00:24:40.799 --> 00:24:47.520
basically peak one Gigabyte a second ever and it will be better when there is

00:24:46.000 --> 00:24:52.720
more v-divs okay then i think we're good

00:24:50.640 --> 00:24:56.400
good to tell you about our sponsor bessie bessie proves that waterproof

00:24:54.799 --> 00:25:00.320
shoes don't need to be ugly or uncomfortable thanks to their dymatex

00:24:58.240 --> 00:25:04.559
technology you can keep those toes dry while remaining sleek snug and stylish

00:25:02.640 --> 00:25:09.120
need even more comfort and breathability the everyday move line has added padding

00:25:06.640 --> 00:25:13.360
at the midsole and a looser knit to keep up with your busy active lifestyle do

00:25:11.120 --> 00:25:17.440
you hate tying laces tired of chasing the rabbit around the tree and down the

00:25:15.200 --> 00:25:22.400
hole well fear not because vessy's everyday move comes with handy pull tabs

00:25:19.679 --> 00:25:26.159
for easy slip-on action and are made 100 creature-free so you can be walking

00:25:24.240 --> 00:25:31.200
around on fluffy clouds with a clear conscience all day long and when vessy

00:25:28.799 --> 00:25:35.039
says everyday move they mean it hot cold wet dry stay comfortable in any weather

00:25:33.679 --> 00:25:38.799
just think of all the freedom you can get with your new shoes so treat those

00:25:36.960 --> 00:25:44.799
feet of yours to vessy everyday move shoes and save 25 with offer code Linus

00:25:41.440 --> 00:25:46.799
tech tips at vetsy.com linustechtips if

00:25:44.799 --> 00:25:51.039
you guys enjoyed this video you might also enjoy the one where we explain what

00:25:49.120 --> 00:25:56.559
happened to the old vault and how we lost nearly all of the data on it

00:25:54.080 --> 00:26:01.279
what we got it all back well yeah i know but first we lost it well like sort of