this seems safe
okay a few weeks ago I'm browsing Facebook Marketplace and I stumble
across this a once top of the line
NetApp storage appliance that would have cost around a hundred thousand dollars
new for five grand and it gets even better
we low bulb them offering just a thousand dollars and they said yes
that's how desperate they were to get rid of this thing which raises some
interesting questions like why does it even work I didn't follow my
own advice and actually power it on before we carted it away so for all I
know it's completely dead and even if it's not dead is it so slow by modern
standards that it really is only worth one percent of what they originally paid
for it there's some way that we could
D proprietaryify this thing and roll it
as commodity storage Hardware we're going to have the answers to those
questions and more after this message from our sponsor unbounce engage your
audience with beautiful high performing marketing campaigns in just a few
minutes with unbounce's AI powered landing page builder on bounce on over
to the link below and get up to 35 off for those who missed part one of our
adventure let's take a quick look at what we've got here a NetApp storage
Appliance is not that dissimilar to the
kind of Nas that you would build for yourself just everything about it is
Enterprise grade instead of having a single box that contains everything
though it's split out so we've got this box here that contains our brain or our
controller for the appliance and then this particular configuration has four
24 Bay disk shelves which are as the
name implies shelves that hold disks
these ones in particular have 600 Gigabyte 15 000 RPM SAS drives in them
and then just power supplies at the back in order to power them but here's the
thing a hard drive is not very useful without a data connection
but remember how I said these were SAS you can think of SAS kind of like the
Enterprise Big Brother of SATA and each of these drives is connected to a back
plane inside our disc shelf that is then connected to our controller through an
external cable those of you who have been paying very close attention will
know that this is almost exactly like our mother Vault server keyword being
almost unlike the mother vault which only has a
single controller because it doesn't host Mission critical data this is
designed for what's called high availability which means that no matter
what in this configuration fails there
should be something ready to pick up the slack immediately so that there's no
interruption to your business and it starts with of course the controller box
which contains not one but actually two
separate servers Each of which needs to be connected to
every single one of these j-bots see like that then there's the j-bods
themselves which contain redundant power supplies so I could literally rip this
out mid operation and it would keep working contain redundant controllers so
I could have a controller failure and every Drive in there would still work
and even have two separate cables
connected to every Drive with all that said the reason I got this
at E-Waste pricing is because it kinda is
and not just because of its age you see NetApp locks down their systems so that
you can only use them with drives that have customized NetApp firmware and I'm
sure they'll tell you that this is for your own benefit in some way but the
real reason they do this is to ensure that you won't be able to take advantage
of the ever falling pricing of commodity hard drives and you will have to pay the
NetApp tax for ever with that said
though if we abandon the NetApp controller which frankly isn't very
performant or power efficient these days anyway we should be able to use these
disk Shelf jbods with practically any server or even a desktop computer which
makes these things pretty valuable in the home lab Community as a cheap way to
run a bunch of hard drives all we need is some adapter cables a controller
server and an HBA to plug into but we're getting ahead of ourselves a
little bit you see neither Jake nor I has ever gotten a chance to play with
NetApp anything and despite paying a thousand dollars for the entire get up
we haven't powered it on so why don't we
start by taking a look at how it's supposed to run the first thing we'll
need to do is wire this up properly uh what I did earlier was just for
illustrative purposes these both need to come out so we'll start by taking the
leftmost port on each of our servers I
apologize earlier I referred to these as controllers and they are in a sense
because they're the controller for the entire storage Appliance but these are
also SAS controllers so I'm just going to call these servers and these controllers moving forward I plug each
of these into the two controllers on the top jbod just
like that then one by one top to top bottom to
bottom I daisy chain in each of the
additional controllers then once we reach the bottom jbod we actually Loop
that one back up around to the servers and the reason for this is that if
there's a failure at any point in the chain it can just go oh well I'll just
come at it that way there is four power supplies per disc shelf you technically
only need to plug in the top and the bottom so that both controllers are powered wow that's loud I hope that
spins down later yeah she goes
no power buttons on this oh
wait is this one bad maybe it's just not plugged in oh wait hold on is it not
plugged yeah those are the same lights that are bad on the yeah yeah ones that
are unplugged so okay does it push it in maybe oh no it's in there try the other
side yeah I'll try the other side oh did you just not turn the switch on oh you
just didn't turn the switch oh I missed the switch on that one there you go high
defense it took you a minute to see that switch too it's right under the thing
well I mean in my defense you were flipping all the other switches it'd be
kind of weird to miss them I can't really serial into both of these at the same time so I'm hoping one is enough
you think the top one is the primary good thing all right let's try that the
only link lights that are not illuminated are the ones that are coming
straight from the servers if it's not booting up that makes sense I also don't
know does it need boot drives did they take boot drives out in theory we should
have cereal even at the BIOS so let's try that I guess okay it looks like it's
doing things nvrd one I'm assuming that's like the non-volatile dim
interface e5a e5b does not exist were
all those management ports plugged in it wasn't in their server room anymore so
they wouldn't be nothing plugged in here I really don't know if it needs boot
drives oh hey look here's all the instruction manuals they still have them on their website oh my god wow baby's
first storage cluster data for seven mode
10 gig Ethernet ports so that makes sense we have those plugged in between
we have our SAS ports that's right connect gig ports blah blah blah to Gig
Data Network switches okay all power cords must be used for all units yeah
whatever what's the point of even having a manual okay so the whole thing has to
be Daisy chained every single jbot so the farthest drives from the actual SAS
controller it's just because if one connection dies or like something goes
wrong the connection can come back the other way okay well we can fix that uh
let's flip the power switch okay
now we get to do it all over again with Ethernet ah yes the locked wrench Park transport
to the first ones ah good quite I
screwed up one of mine what uh it goes
top to bottom then bottom to top so I just gotta do a quick uh switchy boo
here okay those are Blinky that's good oh wait what how did I end up oh God
we're still not getting a link light back to the server well it's not booted
so ah oh the link lights are up it's doing stuff yeah something is happening
now deliver monitor take over disabled blah blah blah it's doing things but
these controller nodes each have a two Port 10 gig card in there we didn't
account for that in our like money oh yeah those are worth like 50 100 bucks
each yeah I think we did pretty well here in spite of the fact that they
ended up telling us after the fact that the five thousand dollar listing was a
typo it had intended 500 dollars system
was down for 5.3 million seconds what the hell is that not saying that was the
last time it was on well no I oh it's Blinky it's got internet even oh my God
what is the login install and run system setup software oh register and get your
license Keys maybe you can download this part of it we could maybe be able to
download the system setup one but anything further than that I doubt
download system set of 3.1 or later 2015 was the last time this was updated oh no
we wanted to check the power consumption because 60 terabytes or whatever what
does it work out to Yes it really it's only 60 terabytes yeah and you don't
actually get all of it some of the capacity I think minimum three discs
have to be used for each node to host their OS so the reason it wouldn't turn
on after talking to lead eater on the Forum is one of our form admins shout
out red eater the OS is stored on the array if you move the disc shelves
around and change the cables and move the discs around
yeah okay fine 45 terabytes whatever the
capacity ends up being that's not a bad thing you know I
wouldn't mind you know tens of terabytes of storage but at what cost these days
you could get literally two hard drives and that would be a
total of what like 15 20 watts yeah the
cost of running this compared to buying two or three new hard drives this takes
a while to turn off yeah it is off sort of it shuts off the OS and it goes back
to the loader which is like the bootloader menu and I guess that's as off as it goes and then you just click
click the button I guess so yesterday trying to get it set back up cleared the
config and initialized all the discs which is just one command but that means
it has to zero every single disk and since this is on an old operating system
it zeroed like sequentially and did that
overnight sure oh here's the other thing if you update it from the older version
to the newer version faster row doesn't work you have to fresh install in theory
this one can be updated to a version that has pass zero but we can't download
the software because we don't have a license I emailed them and said hey we
bought this from X company but they said oh you didn't purchase it from NetApp so
we can't give you access so it literally is e-waste basically
all I'm saying is it should be illegal to sell Hardware that requires software
that you need an account to access
all right let's go three amps a 208 volt whoa holy crap
it's drawing 15 1600 Watts right now
eight freaking amps for one year it will
cost us two thousand dollars in electricity here where electricity is
like 13 cents a kilowatt Canadian which
is like 10 cents a kilowatt U.S so six
grand a year in California 10 grand a year in Europe oh look at how like
archaic that PCB is on the back well yeah it kind of looks like carbon fiber
oh man look at that that is just like industrial AF we're not going to destroy
all these drives if somebody in the comments you want them for your NetApp
and you live locally and you want to just come pick them up and take them out
of all of these trays maybe you can have them well NetApp is on now do you want
to take a quick look at it I do this is quite uh the journey to get working so
after I installed and zeroed all the drives I booted up the second node which
was still crashing and had to do the same thing luckily it only zero to three
drives and it took like an hour we got old school cool bottom and top
once you set up the cluster on the first node after it's reinstalled itself you
can then reinstall the second node join that to the cluster but wait it's not
high availability yet you have to go in and like type a command that allows it
to be high availability like turn High availability on otherwise it's just two
separate notes So in theory now both nodes can perform takeover on each other
it used to be that you needed a base license to even use the software at all
they've changed that so you can install it but to do basically anything
sifs like SMB NFS iSCSI you need a
feature license and NetApp won't give it to me because we're not the owner yeah
basically right now this is just an expensive storage device that you can't
really put anything on as far as I can tell because here like I made a
aggregate which is like a pool and then I made an SV honestly I was about right
we only have 45 terabytes of actual addressable space which is a storage
virtual machine you go to edit and I want to turn on sifts which is SMB okay
and it's just grayed out you can see the cluster based license is deprecated but
all of these other things you want to use any of these features you need a
license because we didn't buy it from them or a reseller oh hey look a hard drive look how small the platter is
so the reason for that is that a smaller platter actually gives you faster seek
times because the ARM doesn't have to move as far to find the correct track
that's one of the reasons that these high performance SAS drives in
appliances like this used smaller disc platters the bigger the platter gets the
more actual area that is after all of
that work to get this thing to work it still doesn't really work let's do what
we intended to do then and try and repurpose this Hardware okay cool I get
to run that fun really long command again to turn it off classic Jake
see you later and the like condescending attitude of the people who have told me
how stupid I am over the years for not buying one of these yeah nothing we have
is that mission critical especially when the cost is so high that you could
literally hire a person to come in and deal with it for you
after your hand whoa I forgot about the handles
hey hey how you doing yeah this is the
most junk part of the whole thing I think easily worth the thousand bucks
right here even if we just give them away at the Christmas party
and then what did you have in mind for a replacement
what now we're gonna put old wanak in there oh my God what is it like dual
high performance 32 course or is that one dual 64 7702 64 cores okay I don't
know if this is ever what liquid intended this thing to be but um no I
think they really didn't intend for hard drives the first thing you're going to
need to make a disc shelf like this work with a different computer is an HBA or
host bus adapter this is effectively just a storage controller on a PCIe cart
like what would be on your motherboard just separate this one is a 9305 16 I
the eye is standing for internal you can see that these are internal connectors I
see the problem here this is not what you should buy you should buy one that
ends in e for external which has the proper external mini SAS HD connectors
we just had a bunch of these laying around and startech makes these internal to external adapters so this is really
janky it's not that bad okay it's it's not that great
Enterprise grade Hardware that literally looks like someone hand soldered the
wires onto the PCB I mean why have a housing and a connector when you can
just do this and put hot glue over it you'll notice this card has four
connections and we have two of these adapter things that's because we have
four disc shelves you could Cascade them and have one cable that goes to all of
them I don't really have the right cable actually no we could reuse them could
you reuse it it's slower to do that though yes we would rather have more bandwidth this means we have a full
connection to each of them I maintain it won't matter for the use case for the
use case where you're going to be more limited by your random i o re-silver
replacing a disc okay In fairness to you this ended up being less stupid than I
expected yeah it's really not that bad get kick hey hey hey hey hey hey kick
just stop kicking stop the one that we got uses iom 3 controller modules those
are SAS three gigabit luckily you can buy iom 6 modules which are SAS
six gigabit which doubles our throughput uh how much money did we waste on those
20 bucks oh they're 20 bucks each wow that's cheap and we technically only
need one as far as I'm aware the good news is that the upgrade process is as
simple as oop so I'll pull out old ones you put in new ones yeah
upgraded beautiful now we have double the bandwidth double the throughput
these are qsfp which is what the NetApp uses too many SAS HD which is like the
modern external SAS cable I don't know if anything other than these use qsfp
because this is usually a network cable connector like for 40 gig or 100 Gig
networking yeah I mean in theory it's anything that needs a lot of pins
handle feels like it's not strong enough it really does it's flexing so hard not
as hard as I'm flexing man hold on the plastic thing who cares about the
plastic power cable no I just it might have interfered with that oh it would
have been fine it might not have been things like 100 pounds Linus actually
dropped one of these oh my God a foot off the ground onto pavement and all the
drives still work seemingly wow wiggle while you work
look at how mint that is Buddy I mean
from this side it's mint I really hope this just works I don't think it's gonna
just work really why wait for Jake when I can just turn it on myself food failed
booting from Debbie just give it a second oh okay I just didn't change the
boot order oh no my 100 useless capacity
High wattage drives aren't all detected oh no oh I think we only got one shelf
yeah we have one okay how do we know which one it is though oh I have no idea
do an identify thing on the thing and I'll tell you what light flashes
show me an Access let's go hey it's the top one are they all on yep
all good I want to point out that as much as we're mocking the power
efficiency of the original setup only a
small part of the problem was the server itself and now that we're running a new
server that's not a problem at all also the power efficiency of these disc
shelves is only a problem if the capacity is super low if we were to put
22 terabyte drives in this thing obviously these power supplies aren't
quite as efficient as something modern you know 80 plus titanium would be but
your energy usage per capacity would
actually be much more reasonable and could even be worth the cost savings if
you were trying to set up a very low cost bulk storage solution for
a small business honestly if something like this had existed back when we
started just cheap j-bod disc shelves like this I would have totally gone this
route they existed they just weren't low cost hey some of these drives are
different capacity 556.29 and 555.63 that actually makes
sense because some of them are the two and a half inch replacement drives and
they would be ever so slightly different even though they're both 600 gigabytes
failed to wipe disk invalid argument I
mean that's the whole point of their locked off firmware BS but why would
they care if you put these discs into something else because
see you later buddy knock to addition screwdriver coming
soon you can sign up for an in-stock notification on lttstar.com two
colorways this is interesting the two
and a half inch drives have a little adapter magic oh it's just a height
adapter they've got a screw in from the bottom I have a thing for that hey oh
interesting this is an interposer we don't need these a
benefit of using an interposer like this is that you can use sas-ish signaling up
until the interposer and then it converts to say to right at the drive
and this has the added benefit of allowing both controllers to see the
drive which is super cool but the thing is most SAS controllers can read SATA
drives it doesn't work the other way around you cannot run a SAS drive on a
SATA controller well without an interposer at least you cannot but the point is we don't need an interposer to
read these drives necessarily it just is a little better one thing that's a
little bit confusing are there two separate mounting hole there are there's
two separate mounting holes on these sleds because they are interposer ready
wow that's pretty cool we don't have that many spare drives right now but the
drives that we're going to be using 22 terabytes it's still only detecting 25
discs McCall Wendell uh hello hello is this level one tech support well
technically he's level two so we got the death those disc shelves
set up they're all into the same HBA and it only seems to detect one of them
es yeah the one it detected is four
I have a 9305 16i HBA that's what they're plugged into
the IDS maybe it's just mad because the IDS aren't sequential I think
you are hitting the button on the front Wendell suggested that part of our
problem might be these disc shelves complaining about having 15 000 RPM
drives in them with only two of the power supplies plugged in because
I see so much more power it'll force you to use staggered startup and you gotta
have three out of the four power supplies for 15K so we are trying to
plug in a third power supply for each of our shelves to see if this resolves this
just weird issue where the top one is detecting drives but the other three are
not clear SCS util show it still just
shows the one it's only detecting seven drives
all the lights are up on the front there was some kernel messaging happening when
we were unplugging and plugging in enclosures which makes me think that things are happening but it really just
wasn't detecting anything like you run SES util and it still would only show
the one enclosure after talking to Wendell for a while it's become pretty
clear that regardless of how we had this thing hooked up directly to each disc
shelf or into one and cascaded down to the others this should just work there
isn't really any config that needs to happen to make this work and it turns
out it should have been Wendell's setup at home is running Rocky Linux which is
different than FreeBSD which is what trunass core is based on on the off
chance that it might work we decided to upgrade to trunass scale which is
actually based on Linux and as soon as it turned on I immediately saw all the
drives start populating so I guess it's an issue with true Nas core which is a
little bit ironic because I specifically chose that thinking that it would be
more reliable and better suited for this and it turns out it is not the weird
thing is they show as disc size zero I think it's because of the weird sector
size these drives with a special NetApp firmware use a 520 byte sector size
which is not standard for most computers
but it does turn out you can change them I'm running it on this one right now
it's probably been going for like 10 minutes already and it's at 45 or
something like that you can do multiple at a time so it's not going to take like
a week to do all of these drives but I'm also not gonna do all these drives
they're really not that useful for us I just wanted to demo it on one so we'll
see when that's finished and see if it works but in the meantime why don't we
throw in these 22 terabyte drives I grabbed eight of them and set them up on
the trays with the interposer so we can do two per disc shelf the other thing
that's important about this while you can use SATA drives with these disc
shelves when you have SATA drives and
SAS drives on the same backplane it's really not recommended because the
voltage level is different between SAS and SATA so using interposers in a mix
setup like this is very recommended I'm not even going to try and put just to
say to dry in here because I don't want to damage anything
oh luckily I have this like server shelf here when I set up my server at home I
really wanted to get one of these disc shelves but they're like particularly in
Canada it's like oh 300 bucks on eBay plus 700 shipping like what they're
blinking that's a good sign let's go see if it works it takes a sec for them to
populate but I can already see the number it's increasing around 87.88
there's another one popping up this is still going 45 eight percent hey
two three four five six seven eight hey
look at that plus we have a direct connection to each jbod which means we
have like not 48 that would be 12. we have 24 gigabit to each disc shelf which
is two and a half gigabytes a second something like that which is more than
two drives we'll do hey look it works there we go okay manage devices look at
that all that's left to do is create a data set and then a share so we can
access it I'm going to change it also to Z standard compression we have a lot of
CPU in This Server 128 epic cores and
that's just gonna serve to make our discs go faster so there's really not really a lot of downside I heard we
switched to True now's scale yup how
often have you started on core lately for the better reliability and more
maturity and then ended up on scale I don't know what it is core very much
seems to be built for like IX systems which is a company that makes true Nas
yeah the hardware that they sell these days and scale you know it's Linux it
kind of just works with everything and sure enough as soon as we booted it up it was like wow look at all these discs
really that easy right now I'm just plugging in that test Bench way the hell
over there to 10 gig so we can like do a file transfer I called it the pool is
called sort of slow this is Wi-Fi oh and then the data set
is sort of fast we're directly connected to the server over there via 200 foot
cat68 cable now to this workstation which has a 10 gig Nick in it we have
two 22 terabyte hard drives in each it should be able to hit about a Gigabyte
so it should be fine okay wow that's reading from it immediately
1.1 gigabytes a second it just saturated a 10 gigabit link
yeah yeah
okay 600 megabytes a second I mean it
did like five gigs before it slowed down that's probably just a little bit of RAM
caching there is nothing to be too concerned about this is a video file as
well so it's not compressible so even though we have compression enabled over
there it's really not doing anything if we were to copy a bunch of big Word
documents or big spreadsheets yeah gold it's worth noting that even though we
got a great deal on these Drive shelves
if all you need is a hundred terabytes of storage or less you don't need to do
any of this stuff with a basic consumer motherboard and consumer chassis you can
do anywhere from four to eight hard drives pretty easily and spend a
fraction of what we did here and just direct attach them all it'll work just
as well with the free or inexpensive software we already mentioned and you
don't have to deal with the noise of an enterprise-oriented solution that being
said if you do want more capacity like this I'm one of those people getting one
of these disc shelves you can probably find one on eBay if you're lucky for two
three hundred bucks maybe even with the new modules you need an HBA which you
can buy Chinese ones for a couple hundred bucks and then a cable that's
basically it yeah and your drives and you're ready to hang out on our slash
data hoarders yeah assuming that subreddit is still up these days you
know what is up the Segway
to our sponsor delete me your personal information is
out on the web being passed around between companies like hors d'oeuvres at
a fancy party and it's time to stop With a Little Help from delete me delete me
will help you find and remove hundreds of online profiles sharing your personal
info so it can't be used by spammers scammers and even Slammers I don't know
what those are but they sound bad in more extreme cases Bad actors having
your data could lead to identity theft or fraud removing all of this
information by yourself would be a huge task to take many hours but with delete
me software and team of experts it can be done in minutes on average delete me
finds and removes over 2 000 pieces of data for a customer in their first two
years so if you want to get your personal information removed from search
results on the web go to join deleteme.com Linus techtips and use code
LTT for 20 off okay if you guys enjoyed this video I think the best one to go
check out is setting up the mother Vault these are so quiet
you can have that under your bed no