We make a lot of videos, which is why over the years, we've built out over two and a half petabytes of bold storage.
But somehow, it's not enough. It's never enough!
And that's especially true when you consider all of our upcoming projects.
The videos just keep coming. The badminton slash gaming center needs capacity.
And if I am ever going to deliver on that free storage backups as an employee perk concept,
I'm going to need a lot more storage, like at least another petabyte worth.
In this economy? You're right. Storage ain't cheap, but what if I told you that I could get a petabyte for half the price?
I'm listening. Well, we can. In our quest to find the most cost-effective storage upgrade, we stumbled upon these.
They are recertified drives, but the recertification is done by Seagate themselves,
and they come in at just about half the price of the same drive's brand new.
So, like any other sane person would, I immediately handed eBay my credit card information
and ordered $10,000 worth of them. That's 60 drives.
So you've got to be wondering, Linus, what's the catch? Come closer, and I'm going to tell you.
It's our sponsor.
Okay, the real answer to what's the catch is we don't know yet.
That's kind of the point of this adventure. See, the best advice that we've ever gotten and given is to avoid secondhand storage.
So this is kind of untreaded territory for us, but the deal was too sweet to pass up.
Now, it's not like we were worried about getting outright scammed.
eBay has pretty good buyer protections, and the main sellers of these recertified drives
seem to have excellent ratings, with most of the drives even coming with some kind of warranty.
But there's still a lot of questions, like what does recertified even mean?
Are they used? It's hard to say. I mean, the eBay listings for these drives say that they have extremely low power on hours
and no bad sectors, which would seem to suggest that they're basically brand new,
but if they were recertified by Seagate, the manufacturer,
then in theory, they could just reset the power on hours. The next thing the script says is unbox them, but there's not a whole lot of unboxing to do.
Like, is this how they came? Well, it's not like when you buy an individual drive,
it's not like they're all going to ship in, like, Amazon packaging. When you're buying 60, they come in treads.
That makes sense. Okay, so how used do these look?
I would say... They look pretty not used to me. Not at all. There's some fingerprints, but, you know, our inventory people definitely put asset tags on them.
Not on this one. Oh, it's freaking clean. And red globes.
Even the screw holes. I don't think these have ever been mounted to anything.
Look at this. 22. 2022.
Are they all like that? 2020. 2021.
So these were manufactured any time over the last few years.
I mean, these drives didn't come out until, like, the end of 2021. Like, December.
Oh, yeah, that one's 2020. So that must be a very early one.
Where did they come from? Server parts deals.
Yes. On eBay. So instead of buying our drives from Seagate's own store, we chose server part deals.
They sell them either factory sealed, still recertified, but sealed,
or they actually crack them open and pre-burning test them for up to 50 hours.
According to server part deals, the chance of getting a DOA drive is actually much lower
with those burn-in ones. So we opted for that.
When we picked these up last December, sometimes it takes us a while to catch up to our production ideas.
We found that the sweet spot was these 20 terabyte drives for $180 a piece.
And looking on eBay, it seems like we actually got a pretty sweet deal.
They're closer to $220 now, but even that is a huge discount compared to new.
And you could probably save a bit by buying from them directly rather than buying through eBay.
Disclosure, by the way. Server part deals is one of our sponsors now, but not on this video,
and they actually weren't when we bought these drives from them. No, we paid retail for these.
Should we plug them in? Yeah, I want to see if the hours are correct.
Okay. We've got CKTC tools. We've got one of our drives here.
Yeah. Drive details, smart 21.
Okay. That checks out. That's like nothing, essentially.
Yeah. Great. Now we just need a machine to put them in.
Since our goal is to save a buck, we're going to reuse as much hardware as possible,
starting with this 45 drive, Stornator XL60 that we decommissioned a few years ago.
It still fits 60 drives, just like it did when it was part of our original petabyte cluster.
But now it has double the capacity because we're using 20 terabyte drives.
Man, I remember when we built that, so many people were like,
you're crazy, dude. Just use the cloud, bro.
Newsflash, the cloud is f**king expensive, bro.
And even when it isn't, getting your data back from it is really expensive.
It's actually been pretty validating seeing how many folks have been migrating back
to having their servers on-prem. Now, to be clear, we do use backblaze backups,
but only for backup and only for our active projects.
Now, obviously a home or a small business user is unlikely to have one of these kicking around,
but everything that we're doing could be scaled down to just a handful of drives in a NAS chassis
or even a single drive in a recycled Opuplex,
like that one that Jake has right next to him, which is the same one that we showed off in our HexOS announcement video.
There's actually even people selling 3D printed NAS cases that look pretty cool.
The point is, you don't need to buy dozens of drives in a fancy old server to get a great deal,
or at least what we hope is a really great deal.
You sound so confident. Oh, I'm confident, all right.
I'm confident it's time to rip these out of here. Oh, it's almost as old as me.
It is not. On the subject of our server being kind of old,
there are some choice upgrades we want to make to this old girl before we throw her back in the rack.
Oh, yeah. If our inventory records are correct, the CPU in that machine is over eight years old
and performs similarly to a first gen Ryzen 5. I mean, look, don't get me wrong.
There is absolutely no shame in driving the car you got. And for home use, realistically,
your network speed is going to be a bottleneck before your CPU. But we happened to have some newer hardware sitting on the shelf,
specifically this Asrock Rack Rome D8 2T with an Epic 7402P 24 core processor.
According to Passmark, it is nearly five times faster than the old eight core that it's replacing.
Now, one important tip. If you are going to be re-yosing old gear like our old board,
make sure to update the BIOS to get the latest security patches and definitely replace any dried out thermal paste.
This one should be good. I redid that not that long ago. Okay, board's in, but I forgot to check the memory.
It looks like, yeah, we've got 256 gigs of RAM in here,
which means we should probably put in some more. The rule of thumb is about one gig of RAM per terabyte of storage for read caching,
but our workload is actually pretty read light. So I think we're just going to leave this for now.
What we're not going to leave are these HBAs.
These high point 750s were end of life years ago, but more importantly, they just kind of suck.
So the plan is to replace them with some LSI 9200 series cards.
Now these are also very old, like PCI Express Gen 2 era old,
but for us, that's a feature, since they happen to have the right connectors for the cabling in our machine.
So specifically, we're going to be using a pair of genuine LSI 8 drive cards,
and then three of these 16 drive cards, two of which are genuine,
and one of which is an eBay knockoff. Now we had a whole video planned on these things back in the day,
which is actually why we have this.
I think this one's the fake one. Yeah, it doesn't say LSI on the front, whereas the other ones do,
but the thing is, counterfeit silicon is pretty unusual,
and I suspect this is just old chips
from maybe failed boards in some other way that have been resoldered onto new boards.
So should be fine.
I guess that's part of the adventure. I've had some friends who've had these cheap cards
work for years without issues, and others who've had them die in a few months.
We don't have much choice either way, since the legit ones are really hard to find.
I'm almost unwiring up all the drives, then all I gotta do is chuck a little M.2 boot drive in there,
and then in our last slot, that's not it,
I'm going to be using a Melanox, there we go, NVIDIA Connect X6 dual port 100 gig network card.
Oh, you want to put that into? We're also going to use this cool adapter doodad
to connect this two and a half inch Dell Gen4 NVMe SSD
that we had lying around, and then we can use that to help with read caching or level two arc.
It can hang out. Screwed in first. But I don't need to screw nothing in.
You screwed in. Look, it fell out. Well, it's because you didn't screw it in.
Way to go, Jake. I didn't say drop it on the ground. Way to go, Jake. Well, that's from StarTech.
They kindly sent that to us. Thanks, StarTech.
Now, to keep everything in this area cool, I'm going to throw in a Noctua industrial 3000 RPM,
140 mil fan. That way we won't have any issues with overheating
on our network card, HBAs, or the NVMe SSD that's in there.
Now to find out if it boots. I don't actually know.
It, like, it probably should.
Nah, f***. My optimism is steadily decreasing.
If you have a situation like this where you're knocking in a display signal from a computer,
you're trying to troubleshoot, a good way to check if it's actually working
is to use caps lock or num lock, which you can see when I click caps lock,
it is turning on and off. So clearly the system is posting. It's just not outputting the display signal for some reason,
or potentially this cable or the monitor is not working.
I'm going to try the IPMI and see if that works.
Hey, okay, it's working. Sick. With that out of the way, let's talk software.
There are obviously a lot of options, but our go-to for a while has been TrueNAS scale.
It's free, which is always great. And most importantly, the ZFS file system it uses
is perfect for large setups like this and super resilient.
So if we have any issues with these drives, which I hope we don't, our data should be fine.
If you're a home gamer, you might also want to check out HexOS, the user-friendly NAS software line is invested in
once it's out of early access. But if you're not feeling adventurous,
there's always Unrayed, whose main claim to fame is the ability to expand
its array one drive at a time, or there's Open Media Vault.
We've got TrueNAS installed now, everything seems to be working,
and we can finally put some hard drives in. Oh, heavy as a pull train.
It's heavy. So I've recruited help.
It'll take a sec for them all to get detected and imported, but let's see how many we have so far.
Is she working? I mean, it's on, but I haven't looked at the drives yet.
Oh, okay. Let's see.
61. Wow. They're all working. Well, they all show up.
Yeah, I mean, we haven't made an array yet.
Should we run a smart test on all of them first? Can you do that all at once? Oh, yeah.
Nice. Wow, well, that's great. So they all pass a short smart test.
If you're getting new drives, you should do more than that.
You should, usually people recommend zeroing them all out,
not because you need to clear the data, but as a burn-in to make sure that all the sectors are good
and everything is happy. We're not going to do that right now,
but it's not a bad thing to try. And in theory, server-in-part deals has already done that for us.
Cool. So we can make a pool now. Oh boy, what do you want to call it?
Swimming. What? So it's a swimming pool.
I don't like that. I'm going to call it ocean. I'm not salty.
Since this is primarily archival storage, and since we have a lot of disks,
we're going to group our drives into six 10-drive RAID-Z2 VDEVs.
That two in the RAID-Z2 means that we have two drives
worth of data protection, kind of like RAID-6. So for us to lose any actual data,
we would need to have three drives fail, and all three would have to be within the same 10-drive group,
which is fairly unlikely, since multi-drive failures like that usually happen
when you end up with drives from the same bad batch. And since our drives are recertified,
it's much more likely that they are from different time periods
and different batches. And the script says in theory, because we hadn't actually looked at the stamps on the chassis yet,
in practice, they are very different. 2021, 2022, they're all over the place.
Now, for a home setup, you're probably going to go with something like a four, six,
or eight-wide RAID-Z1 or Z2,
depending on how important that data is to you. But with the new RAID-Z expansion feature,
you can start as small as two or even three drives and then expand that existing group as you want to add more drives.
That's pretty convenient. Yeah, well, we can click start now or go.
Yeah, do it. So we should have 873 tibbytes.
What are you waiting for, you coward? 873 tib to tb.
960 terabytes. Damn. Hard drives have just been kind of quietly in the background,
getting like kind of awesome. Kind of thick. Well, I've paid absolutely no attention to them.
It's like children. Our cache SSD is not detected.
Most people, if they're trying to build something like this at home, they're not building with a petabyte of drives.
They're not going to have a cache SSD in it. So let's have a look.
Something about it is not happy. We will have to figure that out later. Yep.
I'm not expecting like crazy numbers because it's really old HBAs
and we have a very wide V-depth size. I mean. But eight gigabytes a second, that's pretty good.
Or gibby bytes. I'll take it. It goes up and down and up and down.
It's hard drives. So we're not really going to get much better than that.
Well, dude, I'm not going to complain about that. That's sick. We're not even going to be able to realistically do that.
So yeah, that's great. But most of what we're doing on this thing is sequential anyway.
Exactly. That is like five to seven times what we would even be doing
over like a 10 gig connection. Yeah, which is most of the accesses we do,
except when we're doing backups and stuff anyways. So great. Making awesome.
Yeah. I mean, the drives, they haven't complained. Wait, this is great.
But remember you said you were going to reach out to Seagate and find out what's the deal with these drives?
I did. They didn't respond. At least they haven't yet. And it's been like four days or something like that.
OK. I did find a press release on their website about this program with eBay.
And it makes it sound almost like they're taking drives that people are going to throw away,
like from a data center recycler and refurbing them.
I don't know. It doesn't say anything conclusively. And they look really new.
They look really new. It seems like our best guess is that they're just customer
returns or something like that. OK. Well. Bad batches they rehabbed.
One way or another, in total, we spent $11,160 on 62 drives,
which compared to the $23,500 we would have paid for new ones,
seems like a pretty good deal as long as they don't all crap out.
For now, things look good. But it's hard to say what the long-term reliability
is going to look like. I'm hopeful though, especially given that none of them
died immediately. So even if they do, the fact that we're using ZFS
means we should be able to re-silver and repair the array
in a pretty safe way. Yeah. I mean, most hard drives either die in the first day
or three, four, five years later, right? Server part deals also warranties the drives themselves,
so should be OK. Yeah. And full disclosure, I took about half of these drives
for another project, and they were running for six months. Oh. They still don't have errors.
OK. Well, that's a good sign. Yeah. And to feed my data hoarder habits in this economy,
I actually switched over to recertified drives at my house.
And I've got two or three more of them, actually, that have been running for multiple years now,
and they've been great. Now, a brand new Exos drive is going to come with a five-year warranty instead of just two.
But given these are around half the price, with half-ish the warranty, it was kind of worth the gamble
for me. But let us know in the comments if you've had any experience with recertified drives, either good or bad.
Like, I don't know. We get flack sometimes when we show a good deal online,
and then it goes out of stock a few hours later, and no one else can get the same deal.
But that shouldn't be the case with these. There's an entire industry around reusing, recertifying,
and refurbishing enterprise IT equipment, and there seems to be no shortage of storage devices.
One of our favorites is those JBod disk shelves.
Oh, yeah. Those allow you to connect a bunch of hard drives to an existing system.
And we did a video on them a while ago. You should check it out. Yeah, we also came across a pretty wild deal
on some NVMe SSDs when we were researching this project. So happy deal hunting out there.
For now, it's time for me to go hunting. It's segue season to our sponsor.
If you guys liked this video, why not check out the Mother Vault build?
Hell, yeah. That was pretty sick. That was like the deluxe version of those drive shelves.
Yeah, it is exactly. Yeah. And I might use some of these drives as spares for that.