when we did our last update on the server room I said that our days of
janky Solutions were over it was all proper from here on out but if you
freeze frame you can see that I had my
fingers crossed our current live video editing server new new wanak is filled
up with 32 4 terabyte NVMe ssds split
across four of these liquid honey badger carrier cards all we really cared about
at the time was building something that was fast enough to support our 10 plus
editors all hitting it at the same time and it did in spite of all the hate it's
been running for nearly two years now without missing a beat and that's
despite the fact that our team has nearly doubled in size since then did I
say without missing a beat it missed one beat a couple of months ago conveniently
on the same day the LTT Channel Got Hijacked one of the drives on one of the
carrier cards died dropping it from the array like any other sane person we set
up the server in a raid configuration that protects us from losing drives so
no biggie all we should have had to do is swap out the dead drive with a new
one rebuild the array with the parity data and we should be back to normal
except there was a massive oversight in our setup each SSD is in a carrier card
which is mounted into a PCIe tray which which is inside the server how do we
replace a drive in this thing without taking the whole thing apart well we
never documented which SSD is in which carrier card so even if I tore the
entire server apart I'd have a hell of a time finding the Dead one so the answer
is you don't you just leave it in there and replace it with a mismatch u.2 SSD
in one of the empty front Bays that is an awful solution but at least it's a
good transition into how we're going to fix this now if we weren't crazy we'd
probably reconfigure our current Dell server to have more u.2 slots and then
fill those with Enterprise drives but Enterprise drives are really expensive
and we happen to have 22 of sabrin's
absolutely baller eight terabyte Gen 4 plus drives left over from a previous
video that were pretty much begging to be used for something like this but if
we just swapped these ssds in sure we'd be faster and we'd have double the
capacity but we would have the same maintenance problem enter these These
are m.2 to u.2 adapters from Star Tech and icy Dock and what they allow us to
do is take practically any consumer grade m.2 drive and install it into a
hot swappable Bay in the front of a server just like this it's the best of
every solution because we can use cheap consumer ssds they're easily serviceable
without even powering down the machine and we get the full performance of each
Drive instead of only half like we do with the carrier cards oh and you guys
get this segue to our sponsor AMD get
great deals during the game on AMD event while supplies last on AMD ryzen
processors and AMD Radeon graphics cards including amazing game bundles for a
limited time from now until July 1st Canada's birthday check them out at the
link in the description even if we had all the parts on hand to turn our
current production Dell server into new new wanek wanak 4. it wouldn't really be
a good idea because we'd have to do all of this outside of ours and even fully
populated we would still be eight days short of fitting all 32 of the m.2
drives so instead we're going to use a new server specifically we're going to
be using the Gigabyte r282-z9g and you might remember this
from a previous video that we did on g-raid that weird
GPU-based NVMe raid that was wildly fast
but we ultimately didn't use because it doesn't verify the data at all so if you
like went in and like hex edited a file and then turned it back on it wouldn't
catch that that's okay this machine is still perfectly good for software raid
with plenty of AMD epic PCIe Lanes so what we're going
to do is build it up and then just do a quick IP swap from new new wannick to
new new new wanak no new mnemonic to new
new mnemonic this has been in mothballs for a while though so it's time to talk
about the config for CPUs we've gone with AMD epic 75 f3s these are third gen
epics so that's the Milan family of processors 32 cores each the f is for
high frequency these will boost up to 4 gigahertz meaning that they are perfect
for high-speed storage applications like this the bottleneck is going to be our
network interface and our application is that a fast yeah long before it will be
these CPUs and especially long before it'll be those drives we're also missing
a bunch of other essential gear here we have no network cards ah I got network
cards no RAM Jake catch there's no coolers oh
my God what are you doing 100 these are fancy ah those are connect X4 those are
old now
the good news is that these are actually really really inexpensive eight gig
dimms that we have too many of to know what to do with and the reason we're
using this is that while normally in a ZFS array you want a ton of RAM to use
as an arc cash in a high speed NVMe array your drives are so fast that
you're actually adding more overhead by using that cash than you are benefiting
from it that's not to say that there's no need for RAM you could also use it as
a metadata cache which will help accelerate NVMe storage like this it
just doesn't need a ton of space I suspect it'll probably use between 5 and
10 gigs so we're just putting the absolute smallest dims we have to
populate all the memory channels so we get all the speed just one dim per
Channel there's 16 slots per CPU but that would be two dims per Channel we're
just gonna do one well now they're sucking response
I love how many adapters This Server
comes with by default in order to get full PCIe by four operation to all of
these front Bays here they basically take every slot in here every mezzanine
card spot and then convert them to these Slimline ports and run them up to the
front and that's in spite of the fact that with a two socket configuration you
get a total of 160 usable PCIe Lanes
okay
please stop I wouldn't usually bother doing this on like a little consumer CPU
but these things are so huge because there's multiple dies under that heat
spreader it's not like traditional CPUs where if you put some thermal compound
in the middle realistically you've got the hot spots covered there are lots of
hot spots down there and they're very widely spread have you seen these yeah
they're so yeah the Epic ones are green I know they never gave us one until we
went to Texas for Genoa oh yeah lab use only that's awesome I know oh yeah those
are special ones too those are like actually from the lab wait did you steal
this no no I mean sort of oh I should probably start doing this hey oh
oh why have I just been standing here this whole time oh right I was making
fun of him there is one drawback to using these adapters every one of our
m.2 drives needs to be installed on a sled now you probably noticed we've got
two different options for our sleds this one is from Star Tech and honestly
speaking is probably a little faster to use but doesn't have any additional
cooling for the m.2 drive this one is
from Icy Dock and is a bit more of a
bear to deal with but comes with a large heatsink that will
hopefully help with cooling I tested it it does make a difference if you were to
do this in your personal computer for whatever reason or like a home lab where
there's not a lot of airflow you should probably buy these ones for us this is
going to be right in the front of a server with tens of thousands of RPM of
fan I want to do this one this is way better oh we have to peel like 20
thermal pads but it's toolless Nick he's trying to not promote the screwdriver
I have secret screwdriver things to show you oh secret screwdriver things that
can't go on camera yeah they're in my pockets take your hands out of my pocket
sir I don't appreciate this here you can show that oh sick what the hell is that
don't worry about it oh yeah hey make sure we don't open too many
yeah we can open them it's fine well yeah but like our
um we're gonna run out of drives here at some point oh they're actually we're gonna run out of slots let's make sure
we don't open too many of them
and yeah these are not great sleds nope
I am so glad the industry has moved on from using screws for these sleds though
let's put it in there and it goes a gloop and then if you want to take it
out you just go you have to make the noise when you do
it otherwise it doesn't work see look uh it's not working clue ah see Magic
these are freaking awesome you know what's even better I was looking at this
earlier today and I was like wow I wonder if they make ones that go into
the ruler stick ones like the e1s drives
oh shut up they do the other thing I realized on these Star Tech ones is you
can put a thermal pad between the m.2 and this PCB which obviously has a bunch
of copper and it lowers the temps like five degrees but then the icy dock ones
come down another three to five degrees after that yeah so the heatsink is
definitely more better to be fair that was with the fans not really spinning
that much in the server I suspect if you were to put a really big load on the
server it would make less of a difference that's fair yeah we can put
the network card in now it's a connect x400 gig card which is a bit old now you
can get connect X6 which has a little bit more acceleration you can get it in
200 gig capacity I don't have any more connect X6 cards
so that's what's going in there for now like to be clear the the most our Network throughput on wanna ever gets is
like 30 gig maybe I mean that's kind of
a lie it's a fair bit but three gigabytes a second it's not bad what is
wrong with you what do you mean I don't think you dropped it so much and I just
no did well don't touch it though I wanted to can I just show them what's
going to happen no why it's a framework you can fix it I don't want no walks in
oh that's perfectly fine
we didn't even really like show off the server what kind of power supplies are in here 1600 Watts holy Christ here we
go you're just taking in the noise
this is double useful I don't have to hear the server and I don't have to
listen to him look at all the lights I don't see any
activity on them yet can you not
what are these it's like a whole sled placeholders
everyone this is manufactured garbage see okay so
if we do four five wide raid Z ones we get 160. oh that's so much wasted
capacity it's like 40 terabytes of Boise
just do 10. 10 wide be a man I want the toxically
masculine array I don't even have a Bugatti don't even talk
to him packing the name frame DFS okay so we're going to turn the arc cache to
be metadata only it's running kind of warm that's really hot though it's like
wow oh this is gonna be a long command I'll provide moral support um yeah go
Jake yeah Jake type it type it yeah come
on Jake think about what oh this is awesome Jake you're doing so great that's still 15. 16 gig of 12 12 11.
it's going down that's two that's nine that's eight is it possible we should
have done with concealer ssds what is happening
18. what is happening right now you know
parody calculations are a hell of a drug especially when your thing is 10 wide
wow that is sucking back some CPU there
67 usage of
64. high frequency epic course 18
gigabytes a second is very fast this was 20 that's 20 per second man well this is
awesome and this is only a q depth of four I mean I guess for a sequential
load this is not the most accurate so here let's try two threads per drive it
might go a little faster it's going like 24 and then 18 and then seems like it
needs a second to settle into how the CPUs want to handle this it's really not
that loud I know that sounds like a crazy thing to say while it's gone but
he's not wrong we're probably kind of getting to the limit of ZFS here I'm
impressed though 20 gigs a second is like very respectable for software raid
like our CPU is not at 100 is that like 70. so there's still some Headroom for
other things to run and these are not even Enterprise SSD these I know right
I'm trying to think of editors yeah like what kind of workload could we possibly
be hitting it with we would need to overwhelm that cash yeah you'd have to
be copying from every station Red mags
at their full transfer because remember they're not even that fast and that's
assuming Samba could keep up which it can't although what we're going to use
for high availability wantic weka FS they built their own Samba
implementation that has SMB direct so the CPU usage from writing files to this
file system would be like almost nothing when's that coming the server's here you
know what else is here this message from our sponsor racing are you tired of slow
and inefficient file transfers well have you heard about racing with exceptional
speed and efficiency racing can transfer large files with ease plus with
unlimited transfer volume and file size you'll never have to worry about
limitations again but that's not all racing also offers Rich features like
peer-to-peer transfer auto file sync and real-time data backup and with the
ability to sync files of any size you can rest easy knowing your data is
always safe and secure tucked into its bed like a cute little data that it is
need to collaborate with your team racing has got you covered with group
folders and real-time audit logs of users and transfers and with
cross-platform compatibility you can seamlessly transfer files between
Windows Mac Linux Android and iOS
that almost feels wrong so what are you waiting for upgrade your file transfer
game with racing today at the link below if you guys enjoyed this video you
should probably get some more details on what's changed in our infrastructure in
our last server room tour it's done it made it five minutes staying above 10
gigs a second oh sick