back when we installed a petabyte worth of hard drives in our server closet we
were sure that with that much storage we'd be good for a long time and in
fairness i guess two years worth of red footage
is pretty good but it finally happened we are
critically low on space i've got less than five percent available on our main
editing server but with only 20 terabytes available on the vault there
is nowhere to dump it to fortunately
i've got a band-aid solution the very best kind of solution
seagate actually sent over these 15
12 terabyte iron wolf pro drives for a
totally unrelated project that um well
didn't actually go very well so instead of using them for that we're
going to use them to add more capacity to the vault
speaking of the vault i keep all my segways in a vault ridge wallet is the
sleek way to keep wallet bulge down with its compact frame and rfid blocking
inner plates use the offer code ltt september to save 10
and get free worldwide shipping
all right so while i wait for anthony to come down i'm gonna run through how this
whole thing is going to work with some really crude diagrams here our petabyte
cluster uses a file system it's open source and it's called gluster fs that's
what allows these two independent servers here to present to the rest of
the network as a single large share now
we could increase our capacity by adding more server boxes
but that's a project for another day
literally that's a project for another day we've got a video coming with a full
petabyte of storage in a single server rather than two so make sure you're
subscribed so you don't miss that today we're going to take the empty 15 bays in
server delta 2 down here and we're going to expand our storage with another
180 terabytes of raw capacity we don't
get to use all of this capacity though
our gloucester fs implementation is geared towards raw capacity
rather than redundancy so the only fail-safe built into our local network
here is the raid z2v devs in each of our
machines what that means is that out of our 15
drives only 13 of them count towards our
capacity with the rest these two taken
up by parody data that protects us from data loss in the event of a physical
drive failure or a cable failure speaking of which
how about we replace that uh bad cable in delta one oh that's gonna be quite a
project uh guys uh the vault is going offline for
probably about two hours tm just vault so wanaka is up and these
servers are so heavy so we're gonna have to empty all the drives out of them onto
one of these carts then take the server out put it on the
cart and wheel it over to the island where we can work on it i'm still super
proud of this cabinet door holding open innovation we drilled a hole in the side
of the keyboard tray and now we can get the servers out super easily you know
you can just take the door off right no i let the door has a filter
like that's the purpose of the door for us anyway
yes i know i could take the door off i mean with the sides off the back's off
clearly we figured that out thank you youtube comments
it's quick to label a drive it's slow to label 60 drives and this actually has to
be legible that's important why don't you actually work your way from the
front and i'll work my way from the back
okay and we'll do this like uh you know lady and the eating the spaghetti
style uh that's a strange mental image but okay speaking of
terrible segues sure is cold in this server room good
thing i'm wearing my ltt swacket it's a sweater it's a jacket ltt store.com
don't worry about it i don't think most people feel bad for me struggling to
stack all my hard drives that are very time consuming to stack
why can't i stack all these hard drives if you could grab this end though and
help me up onto the cart that would be swell
oh it's hooked on my it's hooked on my fly okay i got it that's fine
one thing we want to do as we're carting this over to the kitchen there is be
really really gentle with the way that we're moving this
this is about what
350 terabytes of our company's valuable
data on here right now and i remember patrick from serve the home telling me
that yahoo had an incident where they moved their data center like across the
parking lot rolling hard drives on carts not unlike this one and all the
vibration killed like a significant portion of their drives so we could lose
up to eight of them depending on oh seven because one of them is already
degraded but but we don't want to do that so i guess this is the part of the
video where we explain what happens when a cable fails in a storenator now most
bulk storage servers with like lots and lots of hard drives use what's called a
backplane so they'll take fewer connections off of your
sata or your sas adapter or your raid
card or whatever the case may be and then they will take those and they will
split that bandwidth across multiple drives 45 drives takes a bit of a
different approach so they wire every single drive up individually across less
of a back plane and more of an underplane the advantage is you get the
full bandwidth another advantage is that in the event that you a connection fails
you're not replacing an entire costly back plane but the disadvantage is that if a cable
fails you are digging this entire apparatus
out to replace one flaky table so there were two drives
that were dead uh one of them it was fine when we replaced it it was it came
back up everything rebuilt and everything was rosy but then there was
the other one we replaced it a couple times we actually replaced the controller cards itself and tried
different ports on different controllers that we knew worked
but it still had the same issue and what's weird is sometimes it would kind
of work and we could start rebuilding the data on it but like what kind of
data speeds were we getting it would start out at like you know 300
400 megabytes per second which is kind of low but fine but then it would go
down to like 10. yeah and the eta was like a year
yeah like
come on by the way evidence that my dust filter works just
great yeah it looks brand new there is one little bit here that i
noticed but that's it okay so let's not hate on my filtered front cabinet door
there okay okay so this guy needs to come out now this
is a little tricky does that entire plane need to come out i hope not yeah
cause like i'm looking at this and this needs to go up under there in order to
get screwed in there and in order to do that this like straight sucks
this is exactly why i haven't had time to do it until now
how long did we tell the editors this was going to be down for two hours
um one thing i did notice though is that
with how tightly integrated it is into the bottom of the case i
like i don't know how i can't really get it very well
okay they're shooting tech link now so we're going to have to do asmr server
upgrade so i pulled these out and i can see where the cable goes so
yes i will in fact have to pull out this and this i don't see anything obviously
defective about it
that's an angry episode of teclint someone must have removed some headphone
jacks i hope it wasn't samsung
okay so this is it we begin the funeral procession again and this way
it's an open casket oh yeah should we close this server
maybe like now is as good a time as any to do
that uh if we're gonna do full yellow and assume
that we have everything right then yes otherwise no
otherwise no let's go now let's put some hard drives in
yeah having to keep track of everything sucks should have just got a jellyfish
so that drive with the rubbed off label that i wasn't 100 sure about i got to
the end and the only thing left is 131 which is clearly not that unless i have
a wicked case of dyslexia and i put 131 over here so i think it's time for a
sanity check here we go ladies and gentlemen
okay uh anthony you got the drives do you want me to turn it on first should
we do it yeah it's not hot swap if it's not on all
right we'll hot swap it we'll hot swap it i'm turning it on
okay here we go uh anthony you want to take the wheel here sure all right
let's see z pool status okay so guys you
can actually see here what was going on with one of our raid z2s so each of the
15 drives is a raid z2 v-dev so this
raid z2 raids 2-0 is online you can see
the whole z-pool is degraded though that's because raid z2 won here drive
117 the one that we just replaced the cable for
is unavailable and then these are all the previous attempts at rebuilding it
with different drives now we're going to try again but with a new cable so we
fixed 117 but now we've got four drives offline oh balls that's like way down
there you want to let me know if anything changed five six seven and
eight disappeared they're unavailable so that's the wrong one then yep
damn it okay so we're back
and all the drives are here but they're not showing up with their um
like their 45 drive storinator friendly ids here and also
five of them are re-silvering that seems bad how do you re-silver five drives
these are re-silvering because they got cut without being offline first uh in
the meantime should we add the other 15 drives to delta ii
what could go wrong what could go wrong so do we need to make a brick
because that's gluster's crap no we don't need to do that with buster it's
all in under slash z pool pretty sure okay
i don't see anything here that looks like just making a v dab
yeah me neither oh okay so we just do z pull create z pool
raid z2 and then the paths to the disks but if our disks don't show up
okay hold on so let's do this and it is online
so we might need to restart oh okay so maybe they're not hot
swappable it should be but maybe not
i don't know if they're configured the server to not find them yeah their special driver
might actually not do that okay well let's see
in the meantime we can check in on delta one and see if it's re-silvering a
little faster now it is not so now what we've replaced literally
everything the drive the cable the controller i
mean do we want to just pop the drive out and pop it in one more
time and see what happens we can try it okay
brand new drive okay so delta ii is rebooted now and
we've got 1.1 okay so i guess
throw them all in yeah and see if we get
them all and if so we'll create the raid z2 then at least
it may be degraded but it's bigger
so all 15 drives for the expansion are in delta ii now and i switched back over
to delta one and i have good news
our re-silvering is going at 1.91 gigabytes a second
which is pretty sweet that means it should only take a few days
so 117 it's not there is it it's there okay but
but when i try to replace it new device is a different optimal sector size what the
crap so i need to figure out what that is well hold on hold on that's not like a
4k sector drive is it
it might be because that would probably i don't i don't know this for a fact
but i suspect mixing 4k sector drives with
512 sector drives in in some kind of an
array is probably super terrible so do you want it to soft or it's offline it's
already offline yet okay did we accidentally buy the wrong drives
anthony oh my god that couldn't be like the problem could
it they're not are these advanced format yeah yeah
wait you have got to be kidding me
is that why these replace operations haven't been working possibly you know
what we can check the original video unboxing the petabyte project it's a
good thing our entire life exists on youtube at least mine does
okay where's some b-roll of a hard drive here oh no they are advanced format so
they're all there but they're not there
there we go new raid z2 raid z23
includes 15 drives all online 117 is now
back online re-silvering and it's doing it at almost 900 megabytes per second
so we're good so
uh i did notice curiously
that the vault is still not any bigger right
i believe the gluster at best service either needs to be restarted or i might
need to adjust the config real quick okay uh so let me just quickly look at
this in delta wait is it one of these uh disc.conf is it this dot config here oh
yeah there it is hey i helped
did i help on a linux thing yes or no
i did usually i'm not nearly as helpful when
it comes to server stuff okay so final update guys it was just a matter of
getting volume four's brick integrated
into gluster so you can see everything's here volume one two three one two three
four four sure whatever doesn't matter no big deal
so now we're gonna fire over to our other server this is wanic server it
runs windows whatever want to fight about it
we've got our z drive which is actually even tighter for storage than it was
before we have less than a terabyte of space left but that's no problem the
vault 163 terabytes ready to freaking go
and as for delta one it is re-silvering at 375 megabytes a second i can push
this back in very ever so slowly and gently and we're done bud by the way if
you guys liked this video we actually did a video rebuilding our water cooled
render server down here uh you can check that out up there
and if you're not into that you can check out our sponsor for today's video
ifixit the ifixit essential electronics toolkit is compact so it can go anywhere
you can and help you fix almost anything it includes their most popular precision
bits and they're held in place with high density foam so you can throw it around
without any of the bits falling out and of course it comes with ifixit's
lifetime warranty it's just 24.99 at ifixit.com forward slash linus so go
check it out today that's it for this video i will see you guys next time when
we're installing the single box petabyte project