WEBVTT 00:00:00.000 --> 00:00:08.240 We make a lot of videos, which is why over the years, we've built out over two and a half petabytes of bold storage. 00:00:08.240 --> 00:00:12.720 But somehow, it's not enough. It's never enough! 00:00:12.720 --> 00:00:16.480 And that's especially true when you consider all of our upcoming projects. 00:00:16.480 --> 00:00:20.320 The videos just keep coming. The badminton slash gaming center needs capacity. 00:00:20.320 --> 00:00:25.520 And if I am ever going to deliver on that free storage backups as an employee perk concept, 00:00:25.600 --> 00:00:30.480 I'm going to need a lot more storage, like at least another petabyte worth. 00:00:31.360 --> 00:00:39.200 In this economy? You're right. Storage ain't cheap, but what if I told you that I could get a petabyte for half the price? 00:00:40.000 --> 00:00:47.520 I'm listening. Well, we can. In our quest to find the most cost-effective storage upgrade, we stumbled upon these. 00:00:48.160 --> 00:00:54.240 They are recertified drives, but the recertification is done by Seagate themselves, 00:00:54.240 --> 00:00:58.240 and they come in at just about half the price of the same drive's brand new. 00:00:58.240 --> 00:01:03.920 So, like any other sane person would, I immediately handed eBay my credit card information 00:01:03.920 --> 00:01:09.040 and ordered $10,000 worth of them. That's 60 drives. 00:01:09.600 --> 00:01:13.920 So you've got to be wondering, Linus, what's the catch? Come closer, and I'm going to tell you. 00:01:15.360 --> 00:01:16.400 It's our sponsor. 00:01:24.880 --> 00:01:29.680 Okay, the real answer to what's the catch is we don't know yet. 00:01:29.680 --> 00:01:37.040 That's kind of the point of this adventure. See, the best advice that we've ever gotten and given is to avoid secondhand storage. 00:01:37.040 --> 00:01:43.280 So this is kind of untreaded territory for us, but the deal was too sweet to pass up. 00:01:44.000 --> 00:01:47.040 Now, it's not like we were worried about getting outright scammed. 00:01:47.040 --> 00:01:52.320 eBay has pretty good buyer protections, and the main sellers of these recertified drives 00:01:52.320 --> 00:01:57.200 seem to have excellent ratings, with most of the drives even coming with some kind of warranty. 00:01:57.760 --> 00:02:02.240 But there's still a lot of questions, like what does recertified even mean? 00:02:02.240 --> 00:02:09.520 Are they used? It's hard to say. I mean, the eBay listings for these drives say that they have extremely low power on hours 00:02:09.520 --> 00:02:13.920 and no bad sectors, which would seem to suggest that they're basically brand new, 00:02:13.920 --> 00:02:17.440 but if they were recertified by Seagate, the manufacturer, 00:02:17.440 --> 00:02:24.640 then in theory, they could just reset the power on hours. The next thing the script says is unbox them, but there's not a whole lot of unboxing to do. 00:02:24.640 --> 00:02:28.000 Like, is this how they came? Well, it's not like when you buy an individual drive, 00:02:28.000 --> 00:02:33.280 it's not like they're all going to ship in, like, Amazon packaging. When you're buying 60, they come in treads. 00:02:34.240 --> 00:02:38.640 That makes sense. Okay, so how used do these look? 00:02:38.640 --> 00:02:45.520 I would say... They look pretty not used to me. Not at all. There's some fingerprints, but, you know, our inventory people definitely put asset tags on them. 00:02:45.520 --> 00:02:48.640 Not on this one. Oh, it's freaking clean. And red globes. 00:02:48.640 --> 00:02:52.080 Even the screw holes. I don't think these have ever been mounted to anything. 00:02:52.080 --> 00:02:56.000 Look at this. 22. 2022. 00:02:56.000 --> 00:02:59.840 Are they all like that? 2020. 2021. 00:03:00.720 --> 00:03:05.360 So these were manufactured any time over the last few years. 00:03:05.360 --> 00:03:08.800 I mean, these drives didn't come out until, like, the end of 2021. Like, December. 00:03:08.800 --> 00:03:11.920 Oh, yeah, that one's 2020. So that must be a very early one. 00:03:11.920 --> 00:03:14.480 Where did they come from? Server parts deals. 00:03:15.520 --> 00:03:21.680 Yes. On eBay. So instead of buying our drives from Seagate's own store, we chose server part deals. 00:03:21.680 --> 00:03:26.320 They sell them either factory sealed, still recertified, but sealed, 00:03:26.320 --> 00:03:31.360 or they actually crack them open and pre-burning test them for up to 50 hours. 00:03:31.360 --> 00:03:35.840 According to server part deals, the chance of getting a DOA drive is actually much lower 00:03:35.840 --> 00:03:38.720 with those burn-in ones. So we opted for that. 00:03:39.360 --> 00:03:45.440 When we picked these up last December, sometimes it takes us a while to catch up to our production ideas. 00:03:46.480 --> 00:03:51.680 We found that the sweet spot was these 20 terabyte drives for $180 a piece. 00:03:51.680 --> 00:03:54.960 And looking on eBay, it seems like we actually got a pretty sweet deal. 00:03:54.960 --> 00:04:01.280 They're closer to $220 now, but even that is a huge discount compared to new. 00:04:01.280 --> 00:04:05.600 And you could probably save a bit by buying from them directly rather than buying through eBay. 00:04:05.600 --> 00:04:10.960 Disclosure, by the way. Server part deals is one of our sponsors now, but not on this video, 00:04:10.960 --> 00:04:14.960 and they actually weren't when we bought these drives from them. No, we paid retail for these. 00:04:14.960 --> 00:04:18.080 Should we plug them in? Yeah, I want to see if the hours are correct. 00:04:18.080 --> 00:04:21.440 Okay. We've got CKTC tools. We've got one of our drives here. 00:04:21.440 --> 00:04:24.800 Yeah. Drive details, smart 21. 00:04:25.440 --> 00:04:28.560 Okay. That checks out. That's like nothing, essentially. 00:04:28.560 --> 00:04:31.600 Yeah. Great. Now we just need a machine to put them in. 00:04:31.600 --> 00:04:35.680 Since our goal is to save a buck, we're going to reuse as much hardware as possible, 00:04:35.680 --> 00:04:40.800 starting with this 45 drive, Stornator XL60 that we decommissioned a few years ago. 00:04:40.800 --> 00:04:46.080 It still fits 60 drives, just like it did when it was part of our original petabyte cluster. 00:04:46.080 --> 00:04:50.560 But now it has double the capacity because we're using 20 terabyte drives. 00:04:51.520 --> 00:04:54.560 Man, I remember when we built that, so many people were like, 00:04:54.560 --> 00:04:57.760 you're crazy, dude. Just use the cloud, bro. 00:04:58.320 --> 00:05:01.600 Newsflash, the cloud is f**king expensive, bro. 00:05:01.600 --> 00:05:05.920 And even when it isn't, getting your data back from it is really expensive. 00:05:05.920 --> 00:05:10.560 It's actually been pretty validating seeing how many folks have been migrating back 00:05:10.560 --> 00:05:15.680 to having their servers on-prem. Now, to be clear, we do use backblaze backups, 00:05:15.680 --> 00:05:19.600 but only for backup and only for our active projects. 00:05:19.600 --> 00:05:25.760 Now, obviously a home or a small business user is unlikely to have one of these kicking around, 00:05:25.760 --> 00:05:31.680 but everything that we're doing could be scaled down to just a handful of drives in a NAS chassis 00:05:31.680 --> 00:05:35.040 or even a single drive in a recycled Opuplex, 00:05:35.040 --> 00:05:40.480 like that one that Jake has right next to him, which is the same one that we showed off in our HexOS announcement video. 00:05:40.480 --> 00:05:44.320 There's actually even people selling 3D printed NAS cases that look pretty cool. 00:05:44.320 --> 00:05:48.800 The point is, you don't need to buy dozens of drives in a fancy old server to get a great deal, 00:05:48.800 --> 00:05:52.480 or at least what we hope is a really great deal. 00:05:54.080 --> 00:05:57.440 You sound so confident. Oh, I'm confident, all right. 00:05:57.440 --> 00:06:01.600 I'm confident it's time to rip these out of here. Oh, it's almost as old as me. 00:06:02.480 --> 00:06:06.160 It is not. On the subject of our server being kind of old, 00:06:06.160 --> 00:06:10.400 there are some choice upgrades we want to make to this old girl before we throw her back in the rack. 00:06:10.400 --> 00:06:16.880 Oh, yeah. If our inventory records are correct, the CPU in that machine is over eight years old 00:06:16.880 --> 00:06:21.680 and performs similarly to a first gen Ryzen 5. I mean, look, don't get me wrong. 00:06:21.680 --> 00:06:26.480 There is absolutely no shame in driving the car you got. And for home use, realistically, 00:06:26.480 --> 00:06:33.600 your network speed is going to be a bottleneck before your CPU. But we happened to have some newer hardware sitting on the shelf, 00:06:33.600 --> 00:06:40.320 specifically this Asrock Rack Rome D8 2T with an Epic 7402P 24 core processor. 00:06:40.400 --> 00:06:46.160 According to Passmark, it is nearly five times faster than the old eight core that it's replacing. 00:06:46.880 --> 00:06:50.960 Now, one important tip. If you are going to be re-yosing old gear like our old board, 00:06:50.960 --> 00:06:56.320 make sure to update the BIOS to get the latest security patches and definitely replace any dried out thermal paste. 00:06:56.320 --> 00:07:01.440 This one should be good. I redid that not that long ago. Okay, board's in, but I forgot to check the memory. 00:07:01.440 --> 00:07:04.960 It looks like, yeah, we've got 256 gigs of RAM in here, 00:07:04.960 --> 00:07:11.920 which means we should probably put in some more. The rule of thumb is about one gig of RAM per terabyte of storage for read caching, 00:07:11.920 --> 00:07:16.480 but our workload is actually pretty read light. So I think we're just going to leave this for now. 00:07:17.040 --> 00:07:20.240 What we're not going to leave are these HBAs. 00:07:20.240 --> 00:07:25.920 These high point 750s were end of life years ago, but more importantly, they just kind of suck. 00:07:25.920 --> 00:07:30.400 So the plan is to replace them with some LSI 9200 series cards. 00:07:30.400 --> 00:07:35.760 Now these are also very old, like PCI Express Gen 2 era old, 00:07:35.760 --> 00:07:41.840 but for us, that's a feature, since they happen to have the right connectors for the cabling in our machine. 00:07:41.840 --> 00:07:46.480 So specifically, we're going to be using a pair of genuine LSI 8 drive cards, 00:07:46.480 --> 00:07:50.800 and then three of these 16 drive cards, two of which are genuine, 00:07:50.800 --> 00:07:56.560 and one of which is an eBay knockoff. Now we had a whole video planned on these things back in the day, 00:07:56.560 --> 00:07:59.600 which is actually why we have this. 00:07:59.600 --> 00:08:04.960 I think this one's the fake one. Yeah, it doesn't say LSI on the front, whereas the other ones do, 00:08:04.960 --> 00:08:08.960 but the thing is, counterfeit silicon is pretty unusual, 00:08:08.960 --> 00:08:12.080 and I suspect this is just old chips 00:08:12.080 --> 00:08:16.720 from maybe failed boards in some other way that have been resoldered onto new boards. 00:08:16.720 --> 00:08:19.920 So should be fine. 00:08:19.920 --> 00:08:23.840 I guess that's part of the adventure. I've had some friends who've had these cheap cards 00:08:23.840 --> 00:08:27.360 work for years without issues, and others who've had them die in a few months. 00:08:27.360 --> 00:08:31.440 We don't have much choice either way, since the legit ones are really hard to find. 00:08:31.440 --> 00:08:35.840 I'm almost unwiring up all the drives, then all I gotta do is chuck a little M.2 boot drive in there, 00:08:35.840 --> 00:08:38.160 and then in our last slot, that's not it, 00:08:39.040 --> 00:08:45.280 I'm going to be using a Melanox, there we go, NVIDIA Connect X6 dual port 100 gig network card. 00:08:45.280 --> 00:08:50.160 Oh, you want to put that into? We're also going to use this cool adapter doodad 00:08:50.160 --> 00:08:53.200 to connect this two and a half inch Dell Gen4 NVMe SSD 00:08:53.200 --> 00:08:57.440 that we had lying around, and then we can use that to help with read caching or level two arc. 00:08:58.080 --> 00:09:01.360 It can hang out. Screwed in first. But I don't need to screw nothing in. 00:09:01.360 --> 00:09:05.200 You screwed in. Look, it fell out. Well, it's because you didn't screw it in. 00:09:05.200 --> 00:09:08.560 Way to go, Jake. I didn't say drop it on the ground. Way to go, Jake. Well, that's from StarTech. 00:09:08.560 --> 00:09:10.480 They kindly sent that to us. Thanks, StarTech. 00:09:16.880 --> 00:09:21.920 Now, to keep everything in this area cool, I'm going to throw in a Noctua industrial 3000 RPM, 00:09:21.920 --> 00:09:25.120 140 mil fan. That way we won't have any issues with overheating 00:09:25.120 --> 00:09:28.400 on our network card, HBAs, or the NVMe SSD that's in there. 00:09:28.400 --> 00:09:32.320 Now to find out if it boots. I don't actually know. 00:09:32.320 --> 00:09:33.360 It, like, it probably should. 00:09:37.680 --> 00:09:41.520 Nah, f***. My optimism is steadily decreasing. 00:09:41.520 --> 00:09:45.280 If you have a situation like this where you're knocking in a display signal from a computer, 00:09:45.280 --> 00:09:48.400 you're trying to troubleshoot, a good way to check if it's actually working 00:09:48.400 --> 00:09:52.400 is to use caps lock or num lock, which you can see when I click caps lock, 00:09:53.760 --> 00:09:58.960 it is turning on and off. So clearly the system is posting. It's just not outputting the display signal for some reason, 00:09:58.960 --> 00:10:01.680 or potentially this cable or the monitor is not working. 00:10:02.400 --> 00:10:05.760 I'm going to try the IPMI and see if that works. 00:10:05.760 --> 00:10:09.760 Hey, okay, it's working. Sick. With that out of the way, let's talk software. 00:10:09.760 --> 00:10:14.560 There are obviously a lot of options, but our go-to for a while has been TrueNAS scale. 00:10:14.560 --> 00:10:18.960 It's free, which is always great. And most importantly, the ZFS file system it uses 00:10:18.960 --> 00:10:22.080 is perfect for large setups like this and super resilient. 00:10:22.080 --> 00:10:27.280 So if we have any issues with these drives, which I hope we don't, our data should be fine. 00:10:27.280 --> 00:10:32.400 If you're a home gamer, you might also want to check out HexOS, the user-friendly NAS software line is invested in 00:10:32.400 --> 00:10:35.600 once it's out of early access. But if you're not feeling adventurous, 00:10:35.600 --> 00:10:39.360 there's always Unrayed, whose main claim to fame is the ability to expand 00:10:39.360 --> 00:10:42.640 its array one drive at a time, or there's Open Media Vault. 00:10:42.640 --> 00:10:45.520 We've got TrueNAS installed now, everything seems to be working, 00:10:46.080 --> 00:10:50.080 and we can finally put some hard drives in. Oh, heavy as a pull train. 00:10:50.080 --> 00:10:51.600 It's heavy. So I've recruited help. 00:10:55.440 --> 00:10:59.840 It'll take a sec for them all to get detected and imported, but let's see how many we have so far. 00:11:00.880 --> 00:11:03.920 Is she working? I mean, it's on, but I haven't looked at the drives yet. 00:11:03.920 --> 00:11:05.600 Oh, okay. Let's see. 00:11:07.280 --> 00:11:11.120 61. Wow. They're all working. Well, they all show up. 00:11:11.760 --> 00:11:14.960 Yeah, I mean, we haven't made an array yet. 00:11:14.960 --> 00:11:18.560 Should we run a smart test on all of them first? Can you do that all at once? Oh, yeah. 00:11:18.560 --> 00:11:21.920 Nice. Wow, well, that's great. So they all pass a short smart test. 00:11:21.920 --> 00:11:25.280 If you're getting new drives, you should do more than that. 00:11:25.280 --> 00:11:28.560 You should, usually people recommend zeroing them all out, 00:11:28.560 --> 00:11:33.040 not because you need to clear the data, but as a burn-in to make sure that all the sectors are good 00:11:33.040 --> 00:11:35.840 and everything is happy. We're not going to do that right now, 00:11:36.480 --> 00:11:41.520 but it's not a bad thing to try. And in theory, server-in-part deals has already done that for us. 00:11:41.520 --> 00:11:44.480 Cool. So we can make a pool now. Oh boy, what do you want to call it? 00:11:45.280 --> 00:11:47.840 Swimming. What? So it's a swimming pool. 00:11:48.400 --> 00:11:52.080 I don't like that. I'm going to call it ocean. I'm not salty. 00:11:52.080 --> 00:11:56.480 Since this is primarily archival storage, and since we have a lot of disks, 00:11:56.480 --> 00:12:01.360 we're going to group our drives into six 10-drive RAID-Z2 VDEVs. 00:12:01.360 --> 00:12:05.040 That two in the RAID-Z2 means that we have two drives 00:12:05.040 --> 00:12:10.960 worth of data protection, kind of like RAID-6. So for us to lose any actual data, 00:12:10.960 --> 00:12:17.200 we would need to have three drives fail, and all three would have to be within the same 10-drive group, 00:12:17.200 --> 00:12:22.000 which is fairly unlikely, since multi-drive failures like that usually happen 00:12:22.000 --> 00:12:27.120 when you end up with drives from the same bad batch. And since our drives are recertified, 00:12:27.120 --> 00:12:30.320 it's much more likely that they are from different time periods 00:12:30.320 --> 00:12:35.840 and different batches. And the script says in theory, because we hadn't actually looked at the stamps on the chassis yet, 00:12:36.800 --> 00:12:40.800 in practice, they are very different. 2021, 2022, they're all over the place. 00:12:40.800 --> 00:12:44.320 Now, for a home setup, you're probably going to go with something like a four, six, 00:12:44.320 --> 00:12:47.360 or eight-wide RAID-Z1 or Z2, 00:12:47.360 --> 00:12:51.360 depending on how important that data is to you. But with the new RAID-Z expansion feature, 00:12:51.360 --> 00:12:57.360 you can start as small as two or even three drives and then expand that existing group as you want to add more drives. 00:12:58.240 --> 00:13:01.280 That's pretty convenient. Yeah, well, we can click start now or go. 00:13:01.280 --> 00:13:04.720 Yeah, do it. So we should have 873 tibbytes. 00:13:04.720 --> 00:13:06.880 What are you waiting for, you coward? 873 tib to tb. 00:13:07.760 --> 00:13:13.600 960 terabytes. Damn. Hard drives have just been kind of quietly in the background, 00:13:13.600 --> 00:13:17.840 getting like kind of awesome. Kind of thick. Well, I've paid absolutely no attention to them. 00:13:18.560 --> 00:13:22.880 It's like children. Our cache SSD is not detected. 00:13:22.880 --> 00:13:27.600 Most people, if they're trying to build something like this at home, they're not building with a petabyte of drives. 00:13:27.600 --> 00:13:31.200 They're not going to have a cache SSD in it. So let's have a look. 00:13:31.200 --> 00:13:34.240 Something about it is not happy. We will have to figure that out later. Yep. 00:13:34.240 --> 00:13:37.680 I'm not expecting like crazy numbers because it's really old HBAs 00:13:37.680 --> 00:13:43.840 and we have a very wide V-depth size. I mean. But eight gigabytes a second, that's pretty good. 00:13:43.840 --> 00:13:47.280 Or gibby bytes. I'll take it. It goes up and down and up and down. 00:13:47.360 --> 00:13:50.720 It's hard drives. So we're not really going to get much better than that. 00:13:50.720 --> 00:13:55.040 Well, dude, I'm not going to complain about that. That's sick. We're not even going to be able to realistically do that. 00:13:55.040 --> 00:13:59.520 So yeah, that's great. But most of what we're doing on this thing is sequential anyway. 00:13:59.520 --> 00:14:04.080 Exactly. That is like five to seven times what we would even be doing 00:14:04.080 --> 00:14:07.920 over like a 10 gig connection. Yeah, which is most of the accesses we do, 00:14:07.920 --> 00:14:11.520 except when we're doing backups and stuff anyways. So great. Making awesome. 00:14:11.520 --> 00:14:15.760 Yeah. I mean, the drives, they haven't complained. Wait, this is great. 00:14:15.760 --> 00:14:19.520 But remember you said you were going to reach out to Seagate and find out what's the deal with these drives? 00:14:19.520 --> 00:14:24.000 I did. They didn't respond. At least they haven't yet. And it's been like four days or something like that. 00:14:24.000 --> 00:14:27.440 OK. I did find a press release on their website about this program with eBay. 00:14:27.440 --> 00:14:31.600 And it makes it sound almost like they're taking drives that people are going to throw away, 00:14:31.600 --> 00:14:34.800 like from a data center recycler and refurbing them. 00:14:36.000 --> 00:14:40.160 I don't know. It doesn't say anything conclusively. And they look really new. 00:14:40.160 --> 00:14:44.080 They look really new. It seems like our best guess is that they're just customer 00:14:44.080 --> 00:14:47.840 returns or something like that. OK. Well. Bad batches they rehabbed. 00:14:47.840 --> 00:14:53.280 One way or another, in total, we spent $11,160 on 62 drives, 00:14:53.280 --> 00:14:58.240 which compared to the $23,500 we would have paid for new ones, 00:14:58.240 --> 00:15:01.360 seems like a pretty good deal as long as they don't all crap out. 00:15:02.160 --> 00:15:05.600 For now, things look good. But it's hard to say what the long-term reliability 00:15:05.600 --> 00:15:10.160 is going to look like. I'm hopeful though, especially given that none of them 00:15:10.160 --> 00:15:13.680 died immediately. So even if they do, the fact that we're using ZFS 00:15:13.680 --> 00:15:17.360 means we should be able to re-silver and repair the array 00:15:17.920 --> 00:15:22.080 in a pretty safe way. Yeah. I mean, most hard drives either die in the first day 00:15:22.080 --> 00:15:28.080 or three, four, five years later, right? Server part deals also warranties the drives themselves, 00:15:28.080 --> 00:15:34.720 so should be OK. Yeah. And full disclosure, I took about half of these drives 00:15:34.720 --> 00:15:38.880 for another project, and they were running for six months. Oh. They still don't have errors. 00:15:38.880 --> 00:15:43.680 OK. Well, that's a good sign. Yeah. And to feed my data hoarder habits in this economy, 00:15:43.680 --> 00:15:46.960 I actually switched over to recertified drives at my house. 00:15:46.960 --> 00:15:51.520 And I've got two or three more of them, actually, that have been running for multiple years now, 00:15:51.520 --> 00:15:56.800 and they've been great. Now, a brand new Exos drive is going to come with a five-year warranty instead of just two. 00:15:56.800 --> 00:16:02.800 But given these are around half the price, with half-ish the warranty, it was kind of worth the gamble 00:16:02.800 --> 00:16:09.120 for me. But let us know in the comments if you've had any experience with recertified drives, either good or bad. 00:16:09.120 --> 00:16:12.880 Like, I don't know. We get flack sometimes when we show a good deal online, 00:16:12.880 --> 00:16:16.000 and then it goes out of stock a few hours later, and no one else can get the same deal. 00:16:16.000 --> 00:16:22.080 But that shouldn't be the case with these. There's an entire industry around reusing, recertifying, 00:16:22.080 --> 00:16:27.280 and refurbishing enterprise IT equipment, and there seems to be no shortage of storage devices. 00:16:27.280 --> 00:16:30.480 One of our favorites is those JBod disk shelves. 00:16:30.480 --> 00:16:34.400 Oh, yeah. Those allow you to connect a bunch of hard drives to an existing system. 00:16:34.400 --> 00:16:38.960 And we did a video on them a while ago. You should check it out. Yeah, we also came across a pretty wild deal 00:16:38.960 --> 00:16:44.640 on some NVMe SSDs when we were researching this project. So happy deal hunting out there. 00:16:44.640 --> 00:16:49.520 For now, it's time for me to go hunting. It's segue season to our sponsor. 00:16:49.520 --> 00:16:52.720 If you guys liked this video, why not check out the Mother Vault build? 00:16:52.720 --> 00:16:57.280 Hell, yeah. That was pretty sick. That was like the deluxe version of those drive shelves. 00:16:57.280 --> 00:17:02.480 Yeah, it is exactly. Yeah. And I might use some of these drives as spares for that.