WEBVTT 00:00:00.240 --> 00:00:08.320 i'm not feeling great today but we have a dire situation here at Linus tech tips 00:00:04.960 --> 00:00:11.360 our wanik server our main production 00:00:08.320 --> 00:00:12.400 editing storage server is completely 00:00:11.360 --> 00:00:16.800 full fortunately i have been planning an 00:00:14.799 --> 00:00:20.080 upgrade to it for the last little while that is going to be freaking awesome 00:00:19.119 --> 00:00:24.640 this is new wanik it's gonna have more 00:00:22.720 --> 00:00:31.359 storage space whoa over 100 terabytes of NVMe storage it's 00:00:29.039 --> 00:00:35.440 gonna be faster it is going to have lower power consumption it is going to 00:00:33.280 --> 00:00:40.480 have more processing cores it is going to be epic and perhaps the most epic 00:00:38.239 --> 00:00:45.760 thing about it is going to be how many pci express lanes it has you see pci 00:00:43.920 --> 00:00:50.800 express gets used for all kinds of things graphics cards video capture 00:00:48.480 --> 00:00:56.719 devices peripheral expansion but thing is it's been fast enough for all those 00:00:53.520 --> 00:00:59.199 consumer uses for many years now so 00:00:56.719 --> 00:01:05.360 what's driving the creation of faster and faster pci express is actually the 00:01:02.559 --> 00:01:10.320 server world where it gets used for storage that's right that is a direct 00:01:07.680 --> 00:01:16.400 pci express by four connection at the back of all 24 of these drives and the 00:01:14.000 --> 00:01:20.960 truly unbelievable thing about this server is that unlike our existing wanik 00:01:18.720 --> 00:01:25.680 server which takes a handful of those pci express connections and splits it 00:01:23.600 --> 00:01:30.159 across 48 drives this one has a dedicated connection for 00:01:28.880 --> 00:01:33.920 every single one we are going to be putting up some crazy numbers today 00:01:32.320 --> 00:01:39.119 ladies and gentlemen and it's going to be brought to you by seasonic no 00:01:36.640 --> 00:01:42.880 marketing today just happy holidays from seasonic go check them out at the link 00:01:41.040 --> 00:01:49.479 in the video description really they they paid for a spot for that 00:01:46.159 --> 00:01:49.479 nice guys 00:01:54.640 --> 00:02:00.079 while i'm cracking these open let's talk about the drives that i'm using for the 00:01:58.399 --> 00:02:04.640 server and why reason number one is that wanik is full 00:02:02.960 --> 00:02:10.160 so i needed a capacity upgrade we're going from 1.2 terabyte drives to 4 00:02:08.239 --> 00:02:13.920 terabyte drives because that's what a couple of years of NVMe development does 00:02:12.080 --> 00:02:18.640 for you reason number two is we didn't want to go back to SATA because NVMe was 00:02:16.480 --> 00:02:23.520 such a great upgrade for us in terms of not just performance but stability last 00:02:21.280 --> 00:02:28.160 time thanks to the extremely low access latencies reason number three and really 00:02:26.400 --> 00:02:33.120 the main reason for these drives in particular is that we got a great tip 00:02:30.480 --> 00:02:37.599 from wendell over at level one techs that facebook was apparently flipping a 00:02:35.519 --> 00:02:44.480 bunch of these on ebay because they're upgrading to optane so these were a 00:02:40.879 --> 00:02:47.280 really great deal just a hair over 350 00:02:44.480 --> 00:02:52.560 dollars for Intel data center grade drives that apparently are still under 00:02:49.840 --> 00:02:56.879 warranty on top of everything else now they're not the fastest thing in the 00:02:53.920 --> 00:03:02.720 world by today's standards 3.2 gigabytes per second reads 1.8 gigabytes per 00:02:59.200 --> 00:03:06.000 second rights maximum read iops of 645 00:03:02.720 --> 00:03:08.400 000 and rights of 48 000 but remember 00:03:06.000 --> 00:03:15.760 guys that's almost not going to matter at all because i'm going to have 24 00:03:12.400 --> 00:03:17.280 of them in the same server and it's just 00:03:15.760 --> 00:03:21.879 a naz anyway now let's meet our server this 00:03:20.080 --> 00:03:27.040 is the Gigabyte r272 z32 00:03:24.800 --> 00:03:31.840 and we chose this particular model based on the glowing review that was given to 00:03:29.360 --> 00:03:35.760 it by our buddy patrick kennedy over at serve the home oh 00:03:34.480 --> 00:03:40.720 heavy boy this is a 2u server so that's to say it 00:03:38.239 --> 00:03:46.400 is two rack units in height and it was designed from the ground up for AMD's 00:03:43.360 --> 00:03:48.400 epic 7002 rome platform 00:03:46.400 --> 00:03:55.599 and the advantage we get from that is that it actually has compatibility with 00:03:50.720 --> 00:03:58.319 pci express gen 4 not all epic servers 00:03:55.599 --> 00:04:04.239 are PCIe gen 4 ready what gen 4 compatibility means for us in practical 00:04:00.319 --> 00:04:06.000 terms is not necessarily that much today 00:04:04.239 --> 00:04:09.920 but in the event that we wanted to upgrade it means that we could 00:04:07.280 --> 00:04:14.879 effectively double the bandwidth to almost the whole system so let's take a 00:04:12.080 --> 00:04:18.239 quick tour here we've got dual 1.2 000 watt power supplies here those are 00:04:16.320 --> 00:04:24.479 redundant in case one fails we've got our CPU socket here with support for up 00:04:20.160 --> 00:04:26.960 to 64 cores 128 threads we've got 16 00:04:24.479 --> 00:04:30.720 memory slots that run in eight channel mode freaking incredible 00:04:28.800 --> 00:04:36.639 bandwidth and on the subject of bandwidth most of the internal pci 00:04:33.520 --> 00:04:39.600 express slots are taken up of course by 00:04:36.639 --> 00:04:44.160 the 24 u.2 slots at the front so the way that they're fed is they're using their 00:04:41.759 --> 00:04:48.080 mezzanine card slot here to run four of them you can see that managed over here 00:04:46.400 --> 00:04:53.600 SATA actually goes to these two bays at the back so that's what you would typically boot off of now the CPU that 00:04:52.000 --> 00:05:00.000 we're using is pretty overkill but AMD sent over a 00:04:56.720 --> 00:05:01.919 bunch of cpus and unfortunately 00:05:00.000 --> 00:05:06.160 all the ones that kind of make more sense for this project like lower core 00:05:04.080 --> 00:05:11.919 count ones i sort of have earmarked for other projects so unfortunately i am 00:05:08.560 --> 00:05:14.080 locking away my 132 core 00:05:11.919 --> 00:05:17.360 basically forever that's okay though we really need the upgrade 00:05:16.160 --> 00:05:24.160 it really says a lot about the efficiency of these things that all we need is one of these basic 00:05:20.639 --> 00:05:26.240 passive heat pipe coolers thrown on here 00:05:24.160 --> 00:05:29.840 one of these shrouds and just the airflow from one of the chassis 80 00:05:27.840 --> 00:05:34.960 millimeter fans to keep it cool and remember guys it's validated for up to 00:05:31.759 --> 00:05:36.800 the 64 core one this is cool for my boot 00:05:34.960 --> 00:05:40.400 drive i had intended to use the SATA bays at the back of the chassis 00:05:38.479 --> 00:05:45.759 but then i realized i've actually got two m.2s that share 00:05:43.680 --> 00:05:50.479 their pci express lanes with this 8x slot over here so i figured if i have a 00:05:48.240 --> 00:05:53.360 couple of these old octane m.2s lying around 00:05:51.600 --> 00:05:58.240 what the hey might as well throw them in raid 1 and 00:05:54.880 --> 00:06:00.800 go full pci express on everything all 00:05:58.240 --> 00:06:04.800 right let's turn it around and oh oh that's funny Gigabyte sent over a uh a 00:06:03.520 --> 00:06:09.919 demo unit it seems to be a little bit broken 00:06:06.880 --> 00:06:11.759 that's okay oh they've got the stupid 00:06:09.919 --> 00:06:15.840 cable management things that make it really hard to plug in i mean it also 00:06:14.240 --> 00:06:21.609 makes it hard to come out accidentally which is nice 00:06:22.240 --> 00:06:28.080 now i've actually worked with this motherboard a little bit already for a 00:06:25.680 --> 00:06:31.680 whole home one CPU and it takes 00:06:30.400 --> 00:06:37.120 forever to post so i'm gonna fire it up while we check 00:06:35.280 --> 00:06:41.039 the compatibility of the sleds that our drives were pre-installed on 00:06:39.120 --> 00:06:44.960 and see if we're gonna have to swap all those out i really really hope i don't 00:06:42.800 --> 00:06:49.120 have to it's up 00:06:47.120 --> 00:06:53.360 that's good let's get in the BIOS and see uh if everything's detected 00:06:51.199 --> 00:06:56.560 correctly come on baby everything detector oh would that but 00:06:55.120 --> 00:07:03.840 memory training error what with the what now that is not all of our RAM that is 00:07:00.039 --> 00:07:06.479 458 gigs of RAM uh-oh okay brandon i've 00:07:03.840 --> 00:07:12.400 got good news and bad news um the good news is resetting those two memory 00:07:08.479 --> 00:07:14.400 modules boom 512 gigs of RAM so we're 00:07:12.400 --> 00:07:19.120 ready to rock as far as that goes the bad news is that due to AMD's 00:07:17.039 --> 00:07:24.240 architecture of their epic processors where there's no actual chipset with 00:07:21.680 --> 00:07:30.800 functionality like you know a raid controller or anything like that this 00:07:26.800 --> 00:07:34.080 has no support for hardware raid either 00:07:30.800 --> 00:07:35.599 on the m.2 drives or the SATA ones if 00:07:34.080 --> 00:07:40.000 you want to run raid on your state of drives you have to put an add-in raid 00:07:37.919 --> 00:07:43.680 card in and i ask Gigabyte well why didn't you guys just put 00:07:41.919 --> 00:07:48.639 a raid chip on the motherboard and they were like well it would have used up pci 00:07:45.759 --> 00:07:52.880 express lanes which i mean as part of the design of this board we were trying 00:07:50.319 --> 00:07:57.520 to reserve for all these NVMe drives so i kind of went ah okay fair enough now 00:07:54.720 --> 00:08:02.000 in Linux you can run your os on rated drives but Windows has no easy way to do 00:08:00.319 --> 00:08:06.370 that so that's just something we're going to have to consider as we 00:08:04.800 --> 00:08:11.120 build out this machine i'm going to shut it down for now though 00:08:09.199 --> 00:08:17.199 because the last thing that i really want to know is if i am remounting all 00:08:14.080 --> 00:08:18.879 of these drives to the included sleds 00:08:17.199 --> 00:08:22.800 let's see if they happen to be intercompatible that would be so cool 00:08:20.960 --> 00:08:25.599 that would be so cool oh 00:08:24.000 --> 00:08:31.199 yes yeah they are oh it is so tedious it's so 00:08:28.400 --> 00:08:36.240 tedious well that's not a good sign uh only my two m.2 drives are in here 00:08:34.240 --> 00:08:41.760 which means that maybe the sleds don't go all the way back 00:08:39.760 --> 00:08:44.880 that would blow all right 00:08:42.800 --> 00:08:48.480 i put one drive in each one and lined it up and 00:08:46.480 --> 00:08:52.080 you can see it doesn't go back far enough i gotta swap them all 00:08:52.240 --> 00:08:57.680 damage and i've had people ask me why so many 00:08:55.680 --> 00:09:01.200 people work here you thought i was gonna shock all these drives by myself and put 00:09:00.000 --> 00:09:07.279 them in mounts what one person can do slowly four people can do also slowly but less 00:09:04.880 --> 00:09:10.480 slowly pretty tedious now i just need to install a couple of roles and features 00:09:08.720 --> 00:09:13.279 including file server then 00:09:12.240 --> 00:09:17.680 yes we're going to shut it down and put all the drives in 00:09:16.000 --> 00:09:21.200 and we're going to hope they all work now really the most benefit for a 00:09:19.680 --> 00:09:26.560 solution like this is when you've got like a a storage 00:09:23.600 --> 00:09:32.320 attached network and you're using like banks and banks of these to act as 00:09:29.360 --> 00:09:36.080 the storage for like a whack of compute servers that are connected over fiber 00:09:34.160 --> 00:09:39.279 optics on you know the other end of the data center or whatever the case may be 00:09:38.240 --> 00:09:46.000 but come on guys this line is tech tips so we're using it 00:09:43.440 --> 00:09:49.839 as an 8k video editing nas i really hope these all pick up worst 00:09:48.240 --> 00:09:55.440 case scenario though we've got a couple of bad ones and the key night among you might have 00:09:52.640 --> 00:09:59.839 noticed there's two drives not installed these are either cold spares or to 00:09:57.920 --> 00:10:03.279 account for if a couple of our drives are just doa i mean that's the thing 00:10:01.120 --> 00:10:06.560 with used hardware you get a good deal you might get a couple bad ones all 00:10:04.959 --> 00:10:10.480 right that's a good sign every one of them has its light lit but 00:10:08.959 --> 00:10:14.640 we won't know if everything's picked up 00:10:12.240 --> 00:10:18.720 until we actually see them in here boom okay they're all there man storage 00:10:16.800 --> 00:10:23.839 spaces takes a lot of your capacity by default oh actually that's not that bad 00:10:20.640 --> 00:10:24.720 it's about 70 terabytes 00:10:23.839 --> 00:10:31.120 okay let's try that as long as it works oh you got to be 00:10:27.920 --> 00:10:32.880 kidding me this stupid error like 00:10:31.120 --> 00:10:37.360 five years or whatever it's had this problem and they don't just allow you to choose 00:10:35.760 --> 00:10:40.480 your columns in the gui you have to do it via 00:10:39.120 --> 00:10:44.399 powershell so dumb oh you can hear it ramping up 00:10:42.560 --> 00:10:48.399 now boys 5.5 gigs a second 00:10:46.480 --> 00:10:52.320 well that result is horrible worse across the board than old wanik 00:10:50.640 --> 00:10:56.320 clearly there's a configuration issue here all right so i did a simple 00:10:54.000 --> 00:11:01.200 mirrored space this time and we're gonna try running that we've 00:10:58.320 --> 00:11:04.160 got 44 terabytes here well that's better when it comes to 00:11:02.480 --> 00:11:09.120 sequential reads almost 11 gigs a second not even close 00:11:07.200 --> 00:11:15.360 to what we should be getting though okay that's it we're trying a simple virtual 00:11:11.680 --> 00:11:17.600 disk which is basically just striped 00:11:15.360 --> 00:11:22.959 if this doesn't perform well then i'm at a loss wow that is complete garbage town 00:11:20.800 --> 00:11:26.399 it's even worse how is that even possible 00:11:24.079 --> 00:11:30.560 and crystal disk mark is not the issue there's a microsoft tool called disk 00:11:28.079 --> 00:11:33.839 speed that you can use via command line to run sort of any mixture of loads that 00:11:32.880 --> 00:11:36.800 you want and it seems like there's just this hard 00:11:35.920 --> 00:11:43.279 cap at around 10 gigabytes a second which is like 00:11:41.360 --> 00:11:48.480 i mean obviously it's like good or whatever but for this hardware 00:11:46.000 --> 00:11:52.060 it's terrible so we're going to freenas 00:11:50.160 --> 00:11:54.560 that's it 00:11:54.560 --> 00:12:01.680 and it's been about four days so we did manage to push Windows 00:11:58.720 --> 00:12:05.600 performance a little bit further uh the drives needed firmware updates and they 00:12:03.839 --> 00:12:10.320 were doing some idle garbage collection stuff like that but we capped out at 00:12:07.680 --> 00:12:13.279 about 10 gigabytes a second which to be clear is plenty for anything you're 00:12:11.920 --> 00:12:17.440 going to access over the network i mean we would need a 100 gigabit per second 00:12:15.360 --> 00:12:20.000 network card in order to saturate that 00:12:18.880 --> 00:12:27.519 um but that's not enough for us we know this thing should be capable of so much 00:12:25.120 --> 00:12:33.680 more so we moved on to Linux Jake loaded it up with proxmox and built a pool with 00:12:30.880 --> 00:12:40.079 four v devs each with six drives in them running in raid z1 so raid z1 is kind of 00:12:37.519 --> 00:12:45.279 like raid 5 but without the right hole issue meaning that we should be giving 00:12:42.240 --> 00:12:48.240 up uh let's see four drives worth of 00:12:45.279 --> 00:12:53.040 capacity total but with the resiliency of being able to lose up to four drives 00:12:51.200 --> 00:12:57.040 as long as we don't lose more than one in a single v dev so you'd be well let's 00:12:55.600 --> 00:13:00.720 see six so four chunks like that of drives working 00:12:58.959 --> 00:13:05.279 together now that should have resulted in some rocking 00:13:02.720 --> 00:13:09.600 performance except it didn't it was actually complete 00:13:06.959 --> 00:13:15.760 dog crap like what did we max out at like a hundred giga megabytes a second 00:13:12.560 --> 00:13:20.720 maybe 200 200 megs a second 00:13:15.760 --> 00:13:23.200 terrible so turns out as soon as zfs was 00:13:20.720 --> 00:13:28.560 being loaded up our machine was actually spitting out kernel panics 00:13:26.240 --> 00:13:34.160 that's not a chicken emergency that's like really bad you liked it 00:13:31.040 --> 00:13:36.160 shut up you liked it that makes no sense 00:13:34.160 --> 00:13:41.279 a single one of these drives if i plug it into my computer should be doing 15 00:13:38.480 --> 00:13:45.600 to 20 times that performance and we verified this we checked in with our bud 00:13:43.360 --> 00:13:50.880 patrick over at serve the home he's got one of these boxes and he was getting 00:13:47.600 --> 00:13:53.120 great numbers with just 10 drives 00:13:50.880 --> 00:13:57.519 so what could we do well we called in the tech support for 00:13:55.200 --> 00:14:02.240 tech support wendell from level one tax got on the horn with Jake chatting it up 00:13:59.760 --> 00:14:06.800 on discord and between the two of them they managed to figure out that it looks 00:14:04.240 --> 00:14:11.120 like there is some kind of compatibility issue between the particular Linux 00:14:09.360 --> 00:14:16.880 kernel that the latest version of proxmox is running and zfs in our 00:14:14.320 --> 00:14:21.920 particular system that's causing these problems so we said you know what 00:14:19.519 --> 00:14:26.480 it we're just gonna run a benchmark where we load up each of the drives 00:14:23.839 --> 00:14:32.160 individually and slam them all at the same time Jake hit it 00:14:28.880 --> 00:14:32.160 he wandered away he'll be back 00:14:32.480 --> 00:14:37.839 okay you hear that thing ramping up 00:14:35.760 --> 00:14:42.399 oh my goodness oh my goodness what's going on here is 00:14:39.920 --> 00:14:48.000 that over a Gigabyte per second per drive 28 gigabytes a second 00:14:45.519 --> 00:14:53.680 at those kinds of speeds we are at the limits of AMD's infinity fabric and in 00:14:50.880 --> 00:14:58.079 fact to get to those speeds we actually had to overclock it a little bit thank 00:14:55.839 --> 00:15:00.560 you wendell for the tip there we pushed it too far it's actually spitting out 00:14:59.519 --> 00:15:05.519 some memory errors we're gonna have to dial it back a little bit but whatever you guys want to see impressive numbers 00:15:03.360 --> 00:15:09.199 right well here they are now i'm running a second script that he created that 00:15:07.199 --> 00:15:14.600 writes what was it 15 gigs of data so this is gonna write to the drives this 00:15:11.279 --> 00:15:18.240 time these numbers are nuts there you go 00:15:14.600 --> 00:15:21.600 23.7 gigabytes a second that are right 00:15:18.240 --> 00:15:25.199 up against the theoretical limits of 00:15:21.600 --> 00:15:26.720 what can be done so that's it guys that 00:15:25.199 --> 00:15:32.880 is basically the pinnacle of modern technology nearly 00:15:29.760 --> 00:15:36.160 30 gigabytes a second of reads and just 00:15:32.880 --> 00:15:38.240 over 20 gigabytes a second of rights 00:15:36.160 --> 00:15:42.199 guys to put that in perspective your home network is going to be probably 00:15:40.160 --> 00:15:48.240 gigabit this would be 300 times that to read it you could read 00:15:46.079 --> 00:15:51.440 one full blu-ray disc almost every second 00:15:50.240 --> 00:15:57.360 blu-ray blu-ray blu-ray it's crazy stuff now we just 00:15:55.440 --> 00:16:01.519 need to figure out some minor details like how we're actually gonna get an 00:15:59.680 --> 00:16:04.720 array loaded up on the thing because this is just 00:16:02.720 --> 00:16:07.920 i mean this is just theoretical stuff right now 00:16:05.839 --> 00:16:10.880 and theoretically i could have a good segue for a change but we all knew that 00:16:09.360 --> 00:16:15.120 wasn't going to happen thanks to our sponsor freshbooks for making this video 00:16:13.040 --> 00:16:18.720 possible today freshbooks is the cloud accounting solution that's built for how 00:16:16.880 --> 00:16:22.959 you want to work and you can work not just how you want but anywhere you want 00:16:20.959 --> 00:16:27.120 thanks to the freshbooks mobile app you can create professional looking invoices 00:16:24.959 --> 00:16:31.279 on the go snap pictures of your receipts so you don't lose them stay on top of 00:16:29.199 --> 00:16:35.199 important conversations and never miss an update for example being able to see 00:16:33.600 --> 00:16:39.040 when a client has viewed your invoice for the first time so you can get your 00:16:36.959 --> 00:16:44.320 money you know what i'm talking about yeah you do so visit freshbooks.com 00:16:42.079 --> 00:16:47.680 forward slash tech tips to get your free 30-day trial today we're going to have 00:16:45.920 --> 00:16:50.560 that linked in the video description so thanks again for watching guys if you're 00:16:49.040 --> 00:16:54.320 looking for something else to watch server related maybe check out the video 00:16:52.320 --> 00:16:59.519 where i lost all our data you know i was accused in the comments of that video of 00:16:55.920 --> 00:17:02.639 being a very bad actor uh brandon out of 00:16:59.519 --> 00:17:04.720 10 how much was i acting in that video 00:17:02.639 --> 00:17:07.280 negative 10. that happened that was a real thing