WEBVTT

00:00:00.000 --> 00:00:05.759
when we did our last update on the server room I said that our days of

00:00:03.840 --> 00:00:10.620
janky Solutions were over it was all proper from here on out but if you

00:00:07.440 --> 00:00:13.080
freeze frame you can see that I had my

00:00:10.620 --> 00:00:19.380
fingers crossed our current live video editing server new new wanak is filled

00:00:15.660 --> 00:00:21.900
up with 32 4 terabyte NVMe ssds split

00:00:19.380 --> 00:00:25.320
across four of these liquid honey badger carrier cards all we really cared about

00:00:23.699 --> 00:00:29.699
at the time was building something that was fast enough to support our 10 plus

00:00:27.359 --> 00:00:33.780
editors all hitting it at the same time and it did in spite of all the hate it's

00:00:32.040 --> 00:00:36.960
been running for nearly two years now without missing a beat and that's

00:00:35.520 --> 00:00:40.920
despite the fact that our team has nearly doubled in size since then did I

00:00:39.180 --> 00:00:45.120
say without missing a beat it missed one beat a couple of months ago conveniently

00:00:43.200 --> 00:00:51.000
on the same day the LTT Channel Got Hijacked one of the drives on one of the

00:00:48.960 --> 00:00:54.539
carrier cards died dropping it from the array like any other sane person we set

00:00:52.980 --> 00:00:59.280
up the server in a raid configuration that protects us from losing drives so

00:00:57.000 --> 00:01:03.180
no biggie all we should have had to do is swap out the dead drive with a new

00:01:01.199 --> 00:01:06.540
one rebuild the array with the parity data and we should be back to normal

00:01:04.680 --> 00:01:11.760
except there was a massive oversight in our setup each SSD is in a carrier card

00:01:09.600 --> 00:01:15.840
which is mounted into a PCIe tray which which is inside the server how do we

00:01:13.920 --> 00:01:20.400
replace a drive in this thing without taking the whole thing apart well we

00:01:18.299 --> 00:01:24.600
never documented which SSD is in which carrier card so even if I tore the

00:01:22.380 --> 00:01:28.920
entire server apart I'd have a hell of a time finding the Dead one so the answer

00:01:26.520 --> 00:01:33.659
is you don't you just leave it in there and replace it with a mismatch u.2 SSD

00:01:31.380 --> 00:01:38.040
in one of the empty front Bays that is an awful solution but at least it's a

00:01:36.360 --> 00:01:42.060
good transition into how we're going to fix this now if we weren't crazy we'd

00:01:40.380 --> 00:01:46.740
probably reconfigure our current Dell server to have more u.2 slots and then

00:01:44.460 --> 00:01:52.560
fill those with Enterprise drives but Enterprise drives are really expensive

00:01:48.960 --> 00:01:55.320
and we happen to have 22 of sabrin's

00:01:52.560 --> 00:01:59.040
absolutely baller eight terabyte Gen 4 plus drives left over from a previous

00:01:56.880 --> 00:02:03.420
video that were pretty much begging to be used for something like this but if

00:02:00.899 --> 00:02:07.320
we just swapped these ssds in sure we'd be faster and we'd have double the

00:02:05.100 --> 00:02:13.140
capacity but we would have the same maintenance problem enter these These

00:02:10.259 --> 00:02:17.520
are m.2 to u.2 adapters from Star Tech and icy Dock and what they allow us to

00:02:15.300 --> 00:02:23.040
do is take practically any consumer grade m.2 drive and install it into a

00:02:21.120 --> 00:02:26.940
hot swappable Bay in the front of a server just like this it's the best of

00:02:25.319 --> 00:02:31.200
every solution because we can use cheap consumer ssds they're easily serviceable

00:02:29.580 --> 00:02:36.239
without even powering down the machine and we get the full performance of each

00:02:34.080 --> 00:02:41.940
Drive instead of only half like we do with the carrier cards oh and you guys

00:02:38.760 --> 00:02:44.459
get this segue to our sponsor AMD get

00:02:41.940 --> 00:02:49.140
great deals during the game on AMD event while supplies last on AMD ryzen

00:02:46.739 --> 00:02:53.400
processors and AMD Radeon graphics cards including amazing game bundles for a

00:02:51.000 --> 00:02:57.660
limited time from now until July 1st Canada's birthday check them out at the

00:02:55.680 --> 00:03:02.160
link in the description even if we had all the parts on hand to turn our

00:02:59.220 --> 00:03:07.379
current production Dell server into new new wanek wanak 4. it wouldn't really be

00:03:05.459 --> 00:03:11.640
a good idea because we'd have to do all of this outside of ours and even fully

00:03:09.900 --> 00:03:16.980
populated we would still be eight days short of fitting all 32 of the m.2

00:03:14.459 --> 00:03:21.379
drives so instead we're going to use a new server specifically we're going to

00:03:18.900 --> 00:03:25.140
be using the Gigabyte r282-z9g and you might remember this

00:03:23.459 --> 00:03:30.120
from a previous video that we did on g-raid that weird

00:03:27.019 --> 00:03:32.879
GPU-based NVMe raid that was wildly fast

00:03:30.120 --> 00:03:36.480
but we ultimately didn't use because it doesn't verify the data at all so if you

00:03:34.800 --> 00:03:40.860
like went in and like hex edited a file and then turned it back on it wouldn't

00:03:38.879 --> 00:03:44.659
catch that that's okay this machine is still perfectly good for software raid

00:03:42.360 --> 00:03:50.040
with plenty of AMD epic PCIe Lanes so what we're going

00:03:47.940 --> 00:03:56.099
to do is build it up and then just do a quick IP swap from new new wannick to

00:03:52.799 --> 00:03:58.080
new new new wanak no new mnemonic to new

00:03:56.099 --> 00:04:01.379
new mnemonic this has been in mothballs for a while though so it's time to talk

00:03:59.400 --> 00:04:07.019
about the config for CPUs we've gone with AMD epic 75 f3s these are third gen

00:04:04.980 --> 00:04:12.180
epics so that's the Milan family of processors 32 cores each the f is for

00:04:10.080 --> 00:04:16.260
high frequency these will boost up to 4 gigahertz meaning that they are perfect

00:04:13.860 --> 00:04:21.239
for high-speed storage applications like this the bottleneck is going to be our

00:04:18.419 --> 00:04:25.199
network interface and our application is that a fast yeah long before it will be

00:04:23.340 --> 00:04:29.580
these CPUs and especially long before it'll be those drives we're also missing

00:04:27.660 --> 00:04:33.080
a bunch of other essential gear here we have no network cards ah I got network

00:04:32.100 --> 00:04:39.000
cards no RAM Jake catch there's no coolers oh

00:04:36.720 --> 00:04:43.520
my God what are you doing 100 these are fancy ah those are connect X4 those are

00:04:41.220 --> 00:04:43.520
old now

00:04:46.759 --> 00:04:53.820
the good news is that these are actually really really inexpensive eight gig

00:04:51.840 --> 00:04:57.960
dimms that we have too many of to know what to do with and the reason we're

00:04:55.320 --> 00:05:03.419
using this is that while normally in a ZFS array you want a ton of RAM to use

00:05:00.479 --> 00:05:08.940
as an arc cash in a high speed NVMe array your drives are so fast that

00:05:06.780 --> 00:05:12.540
you're actually adding more overhead by using that cash than you are benefiting

00:05:11.100 --> 00:05:16.979
from it that's not to say that there's no need for RAM you could also use it as

00:05:14.639 --> 00:05:21.840
a metadata cache which will help accelerate NVMe storage like this it

00:05:19.440 --> 00:05:26.400
just doesn't need a ton of space I suspect it'll probably use between 5 and

00:05:23.940 --> 00:05:29.759
10 gigs so we're just putting the absolute smallest dims we have to

00:05:28.259 --> 00:05:33.900
populate all the memory channels so we get all the speed just one dim per

00:05:31.380 --> 00:05:36.780
Channel there's 16 slots per CPU but that would be two dims per Channel we're

00:05:35.460 --> 00:05:39.560
just gonna do one well now they're sucking response

00:05:43.820 --> 00:05:50.100
I love how many adapters This Server

00:05:47.460 --> 00:05:55.860
comes with by default in order to get full PCIe by four operation to all of

00:05:53.759 --> 00:06:00.900
these front Bays here they basically take every slot in here every mezzanine

00:05:58.919 --> 00:06:04.560
card spot and then convert them to these Slimline ports and run them up to the

00:06:02.940 --> 00:06:09.539
front and that's in spite of the fact that with a two socket configuration you

00:06:06.360 --> 00:06:10.630
get a total of 160 usable PCIe Lanes

00:06:09.539 --> 00:06:13.730
okay

00:06:14.759 --> 00:06:20.940
please stop I wouldn't usually bother doing this on like a little consumer CPU

00:06:19.259 --> 00:06:24.900
but these things are so huge because there's multiple dies under that heat

00:06:22.800 --> 00:06:27.840
spreader it's not like traditional CPUs where if you put some thermal compound

00:06:26.160 --> 00:06:31.979
in the middle realistically you've got the hot spots covered there are lots of

00:06:30.060 --> 00:06:36.419
hot spots down there and they're very widely spread have you seen these yeah

00:06:34.440 --> 00:06:40.500
they're so yeah the Epic ones are green I know they never gave us one until we

00:06:38.039 --> 00:06:43.500
went to Texas for Genoa oh yeah lab use only that's awesome I know oh yeah those

00:06:42.300 --> 00:06:48.720
are special ones too those are like actually from the lab wait did you steal

00:06:46.139 --> 00:06:54.780
this no no I mean sort of oh I should probably start doing this hey oh

00:06:52.160 --> 00:06:58.620
oh why have I just been standing here this whole time oh right I was making

00:06:56.160 --> 00:07:03.720
fun of him there is one drawback to using these adapters every one of our

00:07:01.139 --> 00:07:07.800
m.2 drives needs to be installed on a sled now you probably noticed we've got

00:07:05.639 --> 00:07:11.520
two different options for our sleds this one is from Star Tech and honestly

00:07:09.539 --> 00:07:18.180
speaking is probably a little faster to use but doesn't have any additional

00:07:13.919 --> 00:07:21.180
cooling for the m.2 drive this one is

00:07:18.180 --> 00:07:23.580
from Icy Dock and is a bit more of a

00:07:21.180 --> 00:07:28.080
bear to deal with but comes with a large heatsink that will

00:07:25.800 --> 00:07:31.740
hopefully help with cooling I tested it it does make a difference if you were to

00:07:30.300 --> 00:07:34.680
do this in your personal computer for whatever reason or like a home lab where

00:07:33.300 --> 00:07:37.979
there's not a lot of airflow you should probably buy these ones for us this is

00:07:36.900 --> 00:07:42.720
going to be right in the front of a server with tens of thousands of RPM of

00:07:40.620 --> 00:07:47.580
fan I want to do this one this is way better oh we have to peel like 20

00:07:44.819 --> 00:07:52.800
thermal pads but it's toolless Nick he's trying to not promote the screwdriver

00:07:50.060 --> 00:07:56.520
I have secret screwdriver things to show you oh secret screwdriver things that

00:07:54.780 --> 00:07:59.819
can't go on camera yeah they're in my pockets take your hands out of my pocket

00:07:58.139 --> 00:08:05.220
sir I don't appreciate this here you can show that oh sick what the hell is that

00:08:02.520 --> 00:08:10.380
don't worry about it oh yeah hey make sure we don't open too many

00:08:08.340 --> 00:08:13.979
yeah we can open them it's fine well yeah but like our

00:08:12.419 --> 00:08:19.639
um we're gonna run out of drives here at some point oh they're actually we're gonna run out of slots let's make sure

00:08:16.500 --> 00:08:19.639
we don't open too many of them

00:08:20.039 --> 00:08:26.580
and yeah these are not great sleds nope

00:08:23.759 --> 00:08:30.180
I am so glad the industry has moved on from using screws for these sleds though

00:08:28.620 --> 00:08:33.659
let's put it in there and it goes a gloop and then if you want to take it

00:08:31.680 --> 00:08:37.440
out you just go you have to make the noise when you do

00:08:35.099 --> 00:08:43.620
it otherwise it doesn't work see look uh it's not working clue ah see Magic

00:08:41.399 --> 00:08:47.160
these are freaking awesome you know what's even better I was looking at this

00:08:45.360 --> 00:08:51.420
earlier today and I was like wow I wonder if they make ones that go into

00:08:48.420 --> 00:08:53.820
the ruler stick ones like the e1s drives

00:08:51.420 --> 00:08:58.140
oh shut up they do the other thing I realized on these Star Tech ones is you

00:08:56.040 --> 00:09:01.980
can put a thermal pad between the m.2 and this PCB which obviously has a bunch

00:09:00.180 --> 00:09:06.240
of copper and it lowers the temps like five degrees but then the icy dock ones

00:09:04.260 --> 00:09:11.100
come down another three to five degrees after that yeah so the heatsink is

00:09:08.760 --> 00:09:16.200
definitely more better to be fair that was with the fans not really spinning

00:09:13.920 --> 00:09:20.160
that much in the server I suspect if you were to put a really big load on the

00:09:18.839 --> 00:09:23.700
server it would make less of a difference that's fair yeah we can put

00:09:22.080 --> 00:09:28.680
the network card in now it's a connect x400 gig card which is a bit old now you

00:09:26.399 --> 00:09:32.580
can get connect X6 which has a little bit more acceleration you can get it in

00:09:30.720 --> 00:09:35.640
200 gig capacity I don't have any more connect X6 cards

00:09:34.500 --> 00:09:42.899
so that's what's going in there for now like to be clear the the most our Network throughput on wanna ever gets is

00:09:39.660 --> 00:09:44.519
like 30 gig maybe I mean that's kind of

00:09:42.899 --> 00:09:47.820
a lie it's a fair bit but three gigabytes a second it's not bad what is

00:09:46.019 --> 00:09:51.420
wrong with you what do you mean I don't think you dropped it so much and I just

00:09:49.620 --> 00:09:55.260
no did well don't touch it though I wanted to can I just show them what's

00:09:52.740 --> 00:09:59.959
going to happen no why it's a framework you can fix it I don't want no walks in

00:09:57.540 --> 00:10:03.779
oh that's perfectly fine

00:10:02.160 --> 00:10:11.040
we didn't even really like show off the server what kind of power supplies are in here 1600 Watts holy Christ here we

00:10:09.120 --> 00:10:15.779
go you're just taking in the noise

00:10:13.680 --> 00:10:18.540
this is double useful I don't have to hear the server and I don't have to

00:10:17.100 --> 00:10:24.860
listen to him look at all the lights I don't see any

00:10:21.000 --> 00:10:24.860
activity on them yet can you not

00:10:25.800 --> 00:10:31.560
what are these it's like a whole sled placeholders

00:10:30.480 --> 00:10:37.019
everyone this is manufactured garbage see okay so

00:10:34.080 --> 00:10:42.240
if we do four five wide raid Z ones we get 160. oh that's so much wasted

00:10:39.120 --> 00:10:44.519
capacity it's like 40 terabytes of Boise

00:10:42.240 --> 00:10:50.279
just do 10. 10 wide be a man I want the toxically

00:10:48.720 --> 00:10:54.660
masculine array I don't even have a Bugatti don't even talk

00:10:52.260 --> 00:10:58.440
to him packing the name frame DFS okay so we're going to turn the arc cache to

00:10:56.700 --> 00:11:02.880
be metadata only it's running kind of warm that's really hot though it's like

00:11:00.899 --> 00:11:08.399
wow oh this is gonna be a long command I'll provide moral support um yeah go

00:11:04.860 --> 00:11:09.899
Jake yeah Jake type it type it yeah come

00:11:08.399 --> 00:11:18.120
on Jake think about what oh this is awesome Jake you're doing so great that's still 15. 16 gig of 12 12 11.

00:11:15.720 --> 00:11:22.519
it's going down that's two that's nine that's eight is it possible we should

00:11:19.860 --> 00:11:28.380
have done with concealer ssds what is happening

00:11:25.279 --> 00:11:30.060
18. what is happening right now you know

00:11:28.380 --> 00:11:36.560
parody calculations are a hell of a drug especially when your thing is 10 wide

00:11:32.880 --> 00:11:40.040
wow that is sucking back some CPU there

00:11:36.560 --> 00:11:43.920
67 usage of

00:11:40.040 --> 00:11:46.380
64. high frequency epic course 18

00:11:43.920 --> 00:11:50.100
gigabytes a second is very fast this was 20 that's 20 per second man well this is

00:11:48.360 --> 00:11:54.600
awesome and this is only a q depth of four I mean I guess for a sequential

00:11:52.260 --> 00:11:58.680
load this is not the most accurate so here let's try two threads per drive it

00:11:56.940 --> 00:12:02.760
might go a little faster it's going like 24 and then 18 and then seems like it

00:12:00.839 --> 00:12:07.019
needs a second to settle into how the CPUs want to handle this it's really not

00:12:04.980 --> 00:12:12.180
that loud I know that sounds like a crazy thing to say while it's gone but

00:12:09.720 --> 00:12:16.260
he's not wrong we're probably kind of getting to the limit of ZFS here I'm

00:12:14.519 --> 00:12:20.820
impressed though 20 gigs a second is like very respectable for software raid

00:12:18.240 --> 00:12:24.180
like our CPU is not at 100 is that like 70. so there's still some Headroom for

00:12:22.500 --> 00:12:28.800
other things to run and these are not even Enterprise SSD these I know right

00:12:26.579 --> 00:12:32.339
I'm trying to think of editors yeah like what kind of workload could we possibly

00:12:30.779 --> 00:12:38.100
be hitting it with we would need to overwhelm that cash yeah you'd have to

00:12:34.920 --> 00:12:39.899
be copying from every station Red mags

00:12:38.100 --> 00:12:43.380
at their full transfer because remember they're not even that fast and that's

00:12:41.640 --> 00:12:47.220
assuming Samba could keep up which it can't although what we're going to use

00:12:44.940 --> 00:12:50.639
for high availability wantic weka FS they built their own Samba

00:12:48.600 --> 00:12:55.200
implementation that has SMB direct so the CPU usage from writing files to this

00:12:53.459 --> 00:13:00.000
file system would be like almost nothing when's that coming the server's here you

00:12:57.600 --> 00:13:05.339
know what else is here this message from our sponsor racing are you tired of slow

00:13:03.000 --> 00:13:09.899
and inefficient file transfers well have you heard about racing with exceptional

00:13:07.440 --> 00:13:14.160
speed and efficiency racing can transfer large files with ease plus with

00:13:12.300 --> 00:13:17.760
unlimited transfer volume and file size you'll never have to worry about

00:13:15.480 --> 00:13:22.740
limitations again but that's not all racing also offers Rich features like

00:13:20.279 --> 00:13:26.639
peer-to-peer transfer auto file sync and real-time data backup and with the

00:13:24.660 --> 00:13:30.779
ability to sync files of any size you can rest easy knowing your data is

00:13:28.860 --> 00:13:34.500
always safe and secure tucked into its bed like a cute little data that it is

00:13:32.639 --> 00:13:38.279
need to collaborate with your team racing has got you covered with group

00:13:36.420 --> 00:13:42.120
folders and real-time audit logs of users and transfers and with

00:13:40.019 --> 00:13:46.620
cross-platform compatibility you can seamlessly transfer files between

00:13:43.500 --> 00:13:48.720
Windows Mac Linux Android and iOS

00:13:46.620 --> 00:13:53.040
that almost feels wrong so what are you waiting for upgrade your file transfer

00:13:50.820 --> 00:13:57.240
game with racing today at the link below if you guys enjoyed this video you

00:13:55.500 --> 00:14:01.260
should probably get some more details on what's changed in our infrastructure in

00:13:58.920 --> 00:14:06.620
our last server room tour it's done it made it five minutes staying above 10

00:14:03.360 --> 00:14:06.620
gigs a second oh sick