1 00:00:00,000 --> 00:00:05,759 when we did our last update on the server room I said that our days of 2 00:00:03,840 --> 00:00:10,620 janky Solutions were over it was all proper from here on out but if you 3 00:00:07,440 --> 00:00:13,080 freeze frame you can see that I had my 4 00:00:10,620 --> 00:00:19,380 fingers crossed our current live video editing server new new wanak is filled 5 00:00:15,660 --> 00:00:21,900 up with 32 4 terabyte NVMe ssds split 6 00:00:19,380 --> 00:00:25,320 across four of these liquid honey badger carrier cards all we really cared about 7 00:00:23,699 --> 00:00:29,699 at the time was building something that was fast enough to support our 10 plus 8 00:00:27,359 --> 00:00:33,780 editors all hitting it at the same time and it did in spite of all the hate it's 9 00:00:32,040 --> 00:00:36,960 been running for nearly two years now without missing a beat and that's 10 00:00:35,520 --> 00:00:40,920 despite the fact that our team has nearly doubled in size since then did I 11 00:00:39,180 --> 00:00:45,120 say without missing a beat it missed one beat a couple of months ago conveniently 12 00:00:43,200 --> 00:00:51,000 on the same day the LTT Channel Got Hijacked one of the drives on one of the 13 00:00:48,960 --> 00:00:54,539 carrier cards died dropping it from the array like any other sane person we set 14 00:00:52,980 --> 00:00:59,280 up the server in a raid configuration that protects us from losing drives so 15 00:00:57,000 --> 00:01:03,180 no biggie all we should have had to do is swap out the dead drive with a new 16 00:01:01,199 --> 00:01:06,540 one rebuild the array with the parity data and we should be back to normal 17 00:01:04,680 --> 00:01:11,760 except there was a massive oversight in our setup each SSD is in a carrier card 18 00:01:09,600 --> 00:01:15,840 which is mounted into a PCIe tray which which is inside the server how do we 19 00:01:13,920 --> 00:01:20,400 replace a drive in this thing without taking the whole thing apart well we 20 00:01:18,299 --> 00:01:24,600 never documented which SSD is in which carrier card so even if I tore the 21 00:01:22,380 --> 00:01:28,920 entire server apart I'd have a hell of a time finding the Dead one so the answer 22 00:01:26,520 --> 00:01:33,659 is you don't you just leave it in there and replace it with a mismatch u.2 SSD 23 00:01:31,380 --> 00:01:38,040 in one of the empty front Bays that is an awful solution but at least it's a 24 00:01:36,360 --> 00:01:42,060 good transition into how we're going to fix this now if we weren't crazy we'd 25 00:01:40,380 --> 00:01:46,740 probably reconfigure our current Dell server to have more u.2 slots and then 26 00:01:44,460 --> 00:01:52,560 fill those with Enterprise drives but Enterprise drives are really expensive 27 00:01:48,960 --> 00:01:55,320 and we happen to have 22 of sabrin's 28 00:01:52,560 --> 00:01:59,040 absolutely baller eight terabyte Gen 4 plus drives left over from a previous 29 00:01:56,880 --> 00:02:03,420 video that were pretty much begging to be used for something like this but if 30 00:02:00,899 --> 00:02:07,320 we just swapped these ssds in sure we'd be faster and we'd have double the 31 00:02:05,100 --> 00:02:13,140 capacity but we would have the same maintenance problem enter these These 32 00:02:10,259 --> 00:02:17,520 are m.2 to u.2 adapters from Star Tech and icy Dock and what they allow us to 33 00:02:15,300 --> 00:02:23,040 do is take practically any consumer grade m.2 drive and install it into a 34 00:02:21,120 --> 00:02:26,940 hot swappable Bay in the front of a server just like this it's the best of 35 00:02:25,319 --> 00:02:31,200 every solution because we can use cheap consumer ssds they're easily serviceable 36 00:02:29,580 --> 00:02:36,239 without even powering down the machine and we get the full performance of each 37 00:02:34,080 --> 00:02:41,940 Drive instead of only half like we do with the carrier cards oh and you guys 38 00:02:38,760 --> 00:02:44,459 get this segue to our sponsor AMD get 39 00:02:41,940 --> 00:02:49,140 great deals during the game on AMD event while supplies last on AMD ryzen 40 00:02:46,739 --> 00:02:53,400 processors and AMD Radeon graphics cards including amazing game bundles for a 41 00:02:51,000 --> 00:02:57,660 limited time from now until July 1st Canada's birthday check them out at the 42 00:02:55,680 --> 00:03:02,160 link in the description even if we had all the parts on hand to turn our 43 00:02:59,220 --> 00:03:07,379 current production Dell server into new new wanek wanak 4. it wouldn't really be 44 00:03:05,459 --> 00:03:11,640 a good idea because we'd have to do all of this outside of ours and even fully 45 00:03:09,900 --> 00:03:16,980 populated we would still be eight days short of fitting all 32 of the m.2 46 00:03:14,459 --> 00:03:21,379 drives so instead we're going to use a new server specifically we're going to 47 00:03:18,900 --> 00:03:25,140 be using the Gigabyte r282-z9g and you might remember this 48 00:03:23,459 --> 00:03:30,120 from a previous video that we did on g-raid that weird 49 00:03:27,019 --> 00:03:32,879 GPU-based NVMe raid that was wildly fast 50 00:03:30,120 --> 00:03:36,480 but we ultimately didn't use because it doesn't verify the data at all so if you 51 00:03:34,800 --> 00:03:40,860 like went in and like hex edited a file and then turned it back on it wouldn't 52 00:03:38,879 --> 00:03:44,659 catch that that's okay this machine is still perfectly good for software raid 53 00:03:42,360 --> 00:03:50,040 with plenty of AMD epic PCIe Lanes so what we're going 54 00:03:47,940 --> 00:03:56,099 to do is build it up and then just do a quick IP swap from new new wannick to 55 00:03:52,799 --> 00:03:58,080 new new new wanak no new mnemonic to new 56 00:03:56,099 --> 00:04:01,379 new mnemonic this has been in mothballs for a while though so it's time to talk 57 00:03:59,400 --> 00:04:07,019 about the config for CPUs we've gone with AMD epic 75 f3s these are third gen 58 00:04:04,980 --> 00:04:12,180 epics so that's the Milan family of processors 32 cores each the f is for 59 00:04:10,080 --> 00:04:16,260 high frequency these will boost up to 4 gigahertz meaning that they are perfect 60 00:04:13,860 --> 00:04:21,239 for high-speed storage applications like this the bottleneck is going to be our 61 00:04:18,419 --> 00:04:25,199 network interface and our application is that a fast yeah long before it will be 62 00:04:23,340 --> 00:04:29,580 these CPUs and especially long before it'll be those drives we're also missing 63 00:04:27,660 --> 00:04:33,080 a bunch of other essential gear here we have no network cards ah I got network 64 00:04:32,100 --> 00:04:39,000 cards no RAM Jake catch there's no coolers oh 65 00:04:36,720 --> 00:04:43,520 my God what are you doing 100 these are fancy ah those are connect X4 those are 66 00:04:41,220 --> 00:04:43,520 old now 67 00:04:46,759 --> 00:04:53,820 the good news is that these are actually really really inexpensive eight gig 68 00:04:51,840 --> 00:04:57,960 dimms that we have too many of to know what to do with and the reason we're 69 00:04:55,320 --> 00:05:03,419 using this is that while normally in a ZFS array you want a ton of RAM to use 70 00:05:00,479 --> 00:05:08,940 as an arc cash in a high speed NVMe array your drives are so fast that 71 00:05:06,780 --> 00:05:12,540 you're actually adding more overhead by using that cash than you are benefiting 72 00:05:11,100 --> 00:05:16,979 from it that's not to say that there's no need for RAM you could also use it as 73 00:05:14,639 --> 00:05:21,840 a metadata cache which will help accelerate NVMe storage like this it 74 00:05:19,440 --> 00:05:26,400 just doesn't need a ton of space I suspect it'll probably use between 5 and 75 00:05:23,940 --> 00:05:29,759 10 gigs so we're just putting the absolute smallest dims we have to 76 00:05:28,259 --> 00:05:33,900 populate all the memory channels so we get all the speed just one dim per 77 00:05:31,380 --> 00:05:36,780 Channel there's 16 slots per CPU but that would be two dims per Channel we're 78 00:05:35,460 --> 00:05:39,560 just gonna do one well now they're sucking response 79 00:05:43,820 --> 00:05:50,100 I love how many adapters This Server 80 00:05:47,460 --> 00:05:55,860 comes with by default in order to get full PCIe by four operation to all of 81 00:05:53,759 --> 00:06:00,900 these front Bays here they basically take every slot in here every mezzanine 82 00:05:58,919 --> 00:06:04,560 card spot and then convert them to these Slimline ports and run them up to the 83 00:06:02,940 --> 00:06:09,539 front and that's in spite of the fact that with a two socket configuration you 84 00:06:06,360 --> 00:06:10,630 get a total of 160 usable PCIe Lanes 85 00:06:09,539 --> 00:06:13,730 okay 86 00:06:14,759 --> 00:06:20,940 please stop I wouldn't usually bother doing this on like a little consumer CPU 87 00:06:19,259 --> 00:06:24,900 but these things are so huge because there's multiple dies under that heat 88 00:06:22,800 --> 00:06:27,840 spreader it's not like traditional CPUs where if you put some thermal compound 89 00:06:26,160 --> 00:06:31,979 in the middle realistically you've got the hot spots covered there are lots of 90 00:06:30,060 --> 00:06:36,419 hot spots down there and they're very widely spread have you seen these yeah 91 00:06:34,440 --> 00:06:40,500 they're so yeah the Epic ones are green I know they never gave us one until we 92 00:06:38,039 --> 00:06:43,500 went to Texas for Genoa oh yeah lab use only that's awesome I know oh yeah those 93 00:06:42,300 --> 00:06:48,720 are special ones too those are like actually from the lab wait did you steal 94 00:06:46,139 --> 00:06:54,780 this no no I mean sort of oh I should probably start doing this hey oh 95 00:06:52,160 --> 00:06:58,620 oh why have I just been standing here this whole time oh right I was making 96 00:06:56,160 --> 00:07:03,720 fun of him there is one drawback to using these adapters every one of our 97 00:07:01,139 --> 00:07:07,800 m.2 drives needs to be installed on a sled now you probably noticed we've got 98 00:07:05,639 --> 00:07:11,520 two different options for our sleds this one is from Star Tech and honestly 99 00:07:09,539 --> 00:07:18,180 speaking is probably a little faster to use but doesn't have any additional 100 00:07:13,919 --> 00:07:21,180 cooling for the m.2 drive this one is 101 00:07:18,180 --> 00:07:23,580 from Icy Dock and is a bit more of a 102 00:07:21,180 --> 00:07:28,080 bear to deal with but comes with a large heatsink that will 103 00:07:25,800 --> 00:07:31,740 hopefully help with cooling I tested it it does make a difference if you were to 104 00:07:30,300 --> 00:07:34,680 do this in your personal computer for whatever reason or like a home lab where 105 00:07:33,300 --> 00:07:37,979 there's not a lot of airflow you should probably buy these ones for us this is 106 00:07:36,900 --> 00:07:42,720 going to be right in the front of a server with tens of thousands of RPM of 107 00:07:40,620 --> 00:07:47,580 fan I want to do this one this is way better oh we have to peel like 20 108 00:07:44,819 --> 00:07:52,800 thermal pads but it's toolless Nick he's trying to not promote the screwdriver 109 00:07:50,060 --> 00:07:56,520 I have secret screwdriver things to show you oh secret screwdriver things that 110 00:07:54,780 --> 00:07:59,819 can't go on camera yeah they're in my pockets take your hands out of my pocket 111 00:07:58,139 --> 00:08:05,220 sir I don't appreciate this here you can show that oh sick what the hell is that 112 00:08:02,520 --> 00:08:10,380 don't worry about it oh yeah hey make sure we don't open too many 113 00:08:08,340 --> 00:08:13,979 yeah we can open them it's fine well yeah but like our 114 00:08:12,419 --> 00:08:19,639 um we're gonna run out of drives here at some point oh they're actually we're gonna run out of slots let's make sure 115 00:08:16,500 --> 00:08:19,639 we don't open too many of them 116 00:08:20,039 --> 00:08:26,580 and yeah these are not great sleds nope 117 00:08:23,759 --> 00:08:30,180 I am so glad the industry has moved on from using screws for these sleds though 118 00:08:28,620 --> 00:08:33,659 let's put it in there and it goes a gloop and then if you want to take it 119 00:08:31,680 --> 00:08:37,440 out you just go you have to make the noise when you do 120 00:08:35,099 --> 00:08:43,620 it otherwise it doesn't work see look uh it's not working clue ah see Magic 121 00:08:41,399 --> 00:08:47,160 these are freaking awesome you know what's even better I was looking at this 122 00:08:45,360 --> 00:08:51,420 earlier today and I was like wow I wonder if they make ones that go into 123 00:08:48,420 --> 00:08:53,820 the ruler stick ones like the e1s drives 124 00:08:51,420 --> 00:08:58,140 oh shut up they do the other thing I realized on these Star Tech ones is you 125 00:08:56,040 --> 00:09:01,980 can put a thermal pad between the m.2 and this PCB which obviously has a bunch 126 00:09:00,180 --> 00:09:06,240 of copper and it lowers the temps like five degrees but then the icy dock ones 127 00:09:04,260 --> 00:09:11,100 come down another three to five degrees after that yeah so the heatsink is 128 00:09:08,760 --> 00:09:16,200 definitely more better to be fair that was with the fans not really spinning 129 00:09:13,920 --> 00:09:20,160 that much in the server I suspect if you were to put a really big load on the 130 00:09:18,839 --> 00:09:23,700 server it would make less of a difference that's fair yeah we can put 131 00:09:22,080 --> 00:09:28,680 the network card in now it's a connect x400 gig card which is a bit old now you 132 00:09:26,399 --> 00:09:32,580 can get connect X6 which has a little bit more acceleration you can get it in 133 00:09:30,720 --> 00:09:35,640 200 gig capacity I don't have any more connect X6 cards 134 00:09:34,500 --> 00:09:42,899 so that's what's going in there for now like to be clear the the most our Network throughput on wanna ever gets is 135 00:09:39,660 --> 00:09:44,519 like 30 gig maybe I mean that's kind of 136 00:09:42,899 --> 00:09:47,820 a lie it's a fair bit but three gigabytes a second it's not bad what is 137 00:09:46,019 --> 00:09:51,420 wrong with you what do you mean I don't think you dropped it so much and I just 138 00:09:49,620 --> 00:09:55,260 no did well don't touch it though I wanted to can I just show them what's 139 00:09:52,740 --> 00:09:59,959 going to happen no why it's a framework you can fix it I don't want no walks in 140 00:09:57,540 --> 00:10:03,779 oh that's perfectly fine 141 00:10:02,160 --> 00:10:11,040 we didn't even really like show off the server what kind of power supplies are in here 1600 Watts holy Christ here we 142 00:10:09,120 --> 00:10:15,779 go you're just taking in the noise 143 00:10:13,680 --> 00:10:18,540 this is double useful I don't have to hear the server and I don't have to 144 00:10:17,100 --> 00:10:24,860 listen to him look at all the lights I don't see any 145 00:10:21,000 --> 00:10:24,860 activity on them yet can you not 146 00:10:25,800 --> 00:10:31,560 what are these it's like a whole sled placeholders 147 00:10:30,480 --> 00:10:37,019 everyone this is manufactured garbage see okay so 148 00:10:34,080 --> 00:10:42,240 if we do four five wide raid Z ones we get 160. oh that's so much wasted 149 00:10:39,120 --> 00:10:44,519 capacity it's like 40 terabytes of Boise 150 00:10:42,240 --> 00:10:50,279 just do 10. 10 wide be a man I want the toxically 151 00:10:48,720 --> 00:10:54,660 masculine array I don't even have a Bugatti don't even talk 152 00:10:52,260 --> 00:10:58,440 to him packing the name frame DFS okay so we're going to turn the arc cache to 153 00:10:56,700 --> 00:11:02,880 be metadata only it's running kind of warm that's really hot though it's like 154 00:11:00,899 --> 00:11:08,399 wow oh this is gonna be a long command I'll provide moral support um yeah go 155 00:11:04,860 --> 00:11:09,899 Jake yeah Jake type it type it yeah come 156 00:11:08,399 --> 00:11:18,120 on Jake think about what oh this is awesome Jake you're doing so great that's still 15. 16 gig of 12 12 11. 157 00:11:15,720 --> 00:11:22,519 it's going down that's two that's nine that's eight is it possible we should 158 00:11:19,860 --> 00:11:28,380 have done with concealer ssds what is happening 159 00:11:25,279 --> 00:11:30,060 18. what is happening right now you know 160 00:11:28,380 --> 00:11:36,560 parody calculations are a hell of a drug especially when your thing is 10 wide 161 00:11:32,880 --> 00:11:40,040 wow that is sucking back some CPU there 162 00:11:36,560 --> 00:11:43,920 67 usage of 163 00:11:40,040 --> 00:11:46,380 64. high frequency epic course 18 164 00:11:43,920 --> 00:11:50,100 gigabytes a second is very fast this was 20 that's 20 per second man well this is 165 00:11:48,360 --> 00:11:54,600 awesome and this is only a q depth of four I mean I guess for a sequential 166 00:11:52,260 --> 00:11:58,680 load this is not the most accurate so here let's try two threads per drive it 167 00:11:56,940 --> 00:12:02,760 might go a little faster it's going like 24 and then 18 and then seems like it 168 00:12:00,839 --> 00:12:07,019 needs a second to settle into how the CPUs want to handle this it's really not 169 00:12:04,980 --> 00:12:12,180 that loud I know that sounds like a crazy thing to say while it's gone but 170 00:12:09,720 --> 00:12:16,260 he's not wrong we're probably kind of getting to the limit of ZFS here I'm 171 00:12:14,519 --> 00:12:20,820 impressed though 20 gigs a second is like very respectable for software raid 172 00:12:18,240 --> 00:12:24,180 like our CPU is not at 100 is that like 70. so there's still some Headroom for 173 00:12:22,500 --> 00:12:28,800 other things to run and these are not even Enterprise SSD these I know right 174 00:12:26,579 --> 00:12:32,339 I'm trying to think of editors yeah like what kind of workload could we possibly 175 00:12:30,779 --> 00:12:38,100 be hitting it with we would need to overwhelm that cash yeah you'd have to 176 00:12:34,920 --> 00:12:39,899 be copying from every station Red mags 177 00:12:38,100 --> 00:12:43,380 at their full transfer because remember they're not even that fast and that's 178 00:12:41,640 --> 00:12:47,220 assuming Samba could keep up which it can't although what we're going to use 179 00:12:44,940 --> 00:12:50,639 for high availability wantic weka FS they built their own Samba 180 00:12:48,600 --> 00:12:55,200 implementation that has SMB direct so the CPU usage from writing files to this 181 00:12:53,459 --> 00:13:00,000 file system would be like almost nothing when's that coming the server's here you 182 00:12:57,600 --> 00:13:05,339 know what else is here this message from our sponsor racing are you tired of slow 183 00:13:03,000 --> 00:13:09,899 and inefficient file transfers well have you heard about racing with exceptional 184 00:13:07,440 --> 00:13:14,160 speed and efficiency racing can transfer large files with ease plus with 185 00:13:12,300 --> 00:13:17,760 unlimited transfer volume and file size you'll never have to worry about 186 00:13:15,480 --> 00:13:22,740 limitations again but that's not all racing also offers Rich features like 187 00:13:20,279 --> 00:13:26,639 peer-to-peer transfer auto file sync and real-time data backup and with the 188 00:13:24,660 --> 00:13:30,779 ability to sync files of any size you can rest easy knowing your data is 189 00:13:28,860 --> 00:13:34,500 always safe and secure tucked into its bed like a cute little data that it is 190 00:13:32,639 --> 00:13:38,279 need to collaborate with your team racing has got you covered with group 191 00:13:36,420 --> 00:13:42,120 folders and real-time audit logs of users and transfers and with 192 00:13:40,019 --> 00:13:46,620 cross-platform compatibility you can seamlessly transfer files between 193 00:13:43,500 --> 00:13:48,720 Windows Mac Linux Android and iOS 194 00:13:46,620 --> 00:13:53,040 that almost feels wrong so what are you waiting for upgrade your file transfer 195 00:13:50,820 --> 00:13:57,240 game with racing today at the link below if you guys enjoyed this video you 196 00:13:55,500 --> 00:14:01,260 should probably get some more details on what's changed in our infrastructure in 197 00:13:58,920 --> 00:14:06,620 our last server room tour it's done it made it five minutes staying above 10 198 00:14:03,360 --> 00:14:06,620 gigs a second oh sick