after months of performance hiccups several blue screens and us literally
being able to bring down the entire core of our operation
just by ingesting video footage to our storage server i'm finally ready for a
change admittedly maybe
running new new wanik our main video editing server on Windows
was a mistake i had my reasons some of
which were valid but it's clear at this point that you guys were right the
advantages of running Windows are far outweighed by the problems that we've
encountered so it is finally the day so many of you have been waiting for when
we remove the last Windows machine from our server room
we're going Linux baby we're doing it right thanks cablemod for sponsoring
this video cablemod allows you to personalize the look of your pc with
custom colored sleeved cables try out their configurator and build your cables
exactly how you want them with their realistic cable preview we're gonna have
it linked down below
this should be easy i've got my Linux install usb right here all we got to do
is slap it into wanik's server and boom we're off to the races right
you see the thing is unlike most of our videos where the hardware is just going
to get torn down immediately afterward wanik server is our one and only
production network editing server we can't just shut it down format it and
reinstall anytime we want because a we wouldn't have anything to store our
videos on or or edit off of and b we
would lose all of the work we've done on our active projects that means we either
have to do this whole process on a weekend or we have to migrate at least
temporarily to a different server as you
can see it is not the weekend so the plan is to use my home nas machine as a
test bed and then eventually migrate what we've done over to new new wanick
one three new pneumonic my intention is for this to eventually end up at my
house but that may not end up happening for a number of reasons not the least of
which is that Jake has become very enamored with the kind of performance
that we get out of these gen four keoxia NVMe drives he and i are gonna have to
fight that out a little bit later in the meantime
let's uh let's build up a server shall we actually that's a bit of a lie this
server is already built so what we need to do is we need to
upgrade it we'll start by taking out this epic 7402
i got it 24 cores is plenty for what i'll be doing at home but as we learned
the hard way you basically can't have enough CPU performance if you
want to get the most out of these gen 4 NVMe drives oh and if another 8 cores
doesn't end up alleviating the bottlenecks we've found we've got a 7742
64 core that we can play around with later
another thing you can't have enough of in a storage server like this is RAM and
i mean that both in terms of capacity and in terms of speed when we were
deploying this at my house we put the smallest crappiest kit that we possibly
could i mean obviously without compromising on reliability so this is
64 gigs of 2666 ddr4 truthfully the
sticks i'm putting in now aren't really ideal either they're running at 26.66
mega transfers per second compared to epic's maximum of 3200
but we're gonna have 256 gigs rather
than 64 gigs which is going to make a big difference especially for using zfs
which we are as much as there might be some kind of GPU accelerated
storage something uh i've never heard of it so we're gonna pull this 1050 ti out
here that was supposed to be for plex transcoding and we're gonna replace it
with this gorgeous melanox connect x6 100 gigabit
per second network card uh do we really need these one terabyte drives for boot
we have like a dozen of those uh those octane m.2s that are like 32 gigs and
they're perfect there's actually not that many left server boot drives oh
really oh all right well i guess i keep using them for stuff like this where it just
really doesn't matter how much capacity you have and you just want something reliable speaking of our installation
media there was a fair bit of deliberation when it came to choosing an
operating system ubuntu server seemed
good except that the current long-term supported version or lts for short
doesn't have open zfs 2.0 which features a bunch of performance improvements for
zfs which is the file system we intend to use and then db and 11 sounded good
it has open cfs 2.0 and a version of
samba the software used to host Windows compatible smb network shares from this
year which is great except it's from march um which is not great so the
beginning of this video ended up being a bit of a lie because while we could have
still gone with Linux just find a distro that's got a new enough kernel uh
potentially build open zfs 2.0 and samba
from scratch there is an easier option trueness it's got opencfs 2.0 a very
recent samba and a really nice web interface for managing your storage it's
not Linux but it is based on a unix-like
operating system called freebsd so we're
we're back on the used to be free nas now truenas train i don't know what it
is about their product but pretty much every time i go to use it something
blows up in a way that is totally inexplicable i've ended up escalated to
like directors and vps of engineering multiple times with nobody ever able to
explain why the product just won't work properly for me but hey fifth time's a
charm works properly for Jake yeah as long as
it works properly for Jake you know our backup servers are running truenast for
like a year now i i know i know it's a good product i've recommended it to
countless people i've just never been able to get it working the way that i
need it to work did it just turn on do you have it set to just restore power on
ac probably okay let's just give it a sec here that's what servers are
supposed to do okay where's that usb we don't need it oh we're gonna do it
properly i thought you told me we need a usb uh just for just for visuals are we
gonna go ipmi af and use an image and everything yeah look at us doing things
properly today yeah i didn't download truenast but i
will now wait what happened to my laptop i put it away you don't need it oh you
told me to bring it i didn't realize there was a keyboard here ah the problem
with mac ipmi is i can't like press f10 properly like doesn't work press f10 to
pay respect doesn't work i can screen record on this thing for
like an hour and only use like 30 battery and its touch pad is the
equivalent of being hung like a horse it is look at it
look at this let me show you my touch pad okay look oh okay it's a touch pad
actually what i was really going to show you was my underwear from lttstore.com
store lttstore.com it comes with cat hair comes no it doesn't that's from
your cat i just walk out in the hallway and she's like what's up bro
virtual cd-rom enter here we go boys this is the smart way to do it you
shouldn't actually ever have to plug a monitor into a server you ready install
wow wow that was inspirational look at it go okay hold on hold on pick our door
hold on yeah we don't want to install the wrong drive now this is cool even
though AMD epic servers do not have any innate support for raid that means you
can't even install two boot drives like we did and then just configured in the
BIOS to run raid freaking simple one
trueness has a quick and easy way for you to install your operating system to
two drives so in the event of a hardware failure you're just
ready to go again immediately yeah it literally just shows up as a separate boot entry you can just pick which one
to boot from okay time to go to lunch see you later i'm asking if i can turn it on well
that's too bad because i already did ah roasted
yeah what's up now what are you gonna hard power it off yeah
what's up no ha got him now who turned it on
this guy wanna turn it off again
i'm turning it off first
this is bad hardware practices hey wait mother
man it's not plugged in all the way oh my god snapped out must have been when
you unplug the power i'm turning it off first
why is there is there blinking on the port ip
no we have to take it to the server room later anyways let's just take it there now this is the way to do it i mean did
we say we were gonna do everything properly today
uh no okay cool because that's definitely not what's happening hey i
got 100 gig you won't believe how much gigabytes you
can put in this bad boy there's a bunch of hard drives under there stop slapping
it let's go back around i don't want to wait okay let's create a pool
name name it's called lambo so do i click them in order to add the
video no you don't um just click all and then unselect that
oh yeah we don't want this dao i think that's the virtual
thing from ipmi 12 selected just click over okay so i did a little bit of testing
beforehand it doesn't seem like it makes a difference between having one v dev or
two for our workload okay raid z is fine yeah raid z is going to give us one
failed drive before we actually start to lose data i'm not expecting to lose a
ton of these drives and this data is going to be replicated using our snapshot to another server that's over
in unit 101 yeah and then it'll be replicated again off-site to kamloops
once Jake gets around to fixing the new kamloops server yeah there we go oh
that's 72 terabytes that's not bad that's not bad usable okay so we got to
make a oh here first thing go to that yeah we have to turn auto trim on pool
options oh why is that not by default well most people don't use ssds with
trueness that's fair we got to make a
new data set add data set and then let's call this a z drive turn
compression off we don't need that pretty much all the data we're going to
write to this thing is incompressible video footage so there's just no benefit
we could turn it on later because there are other things like word files but for
now because we're going to do some performance testing too let's leave it off turn a time off we don't need that
either that's like the access time is that a festival store the last time a
file is accessed you might use that to see if your kids have been looking at your
share type smb that's gonna set it to be case insensitive okay um we're gonna
leave the record size at 128 kb now you could theoretically change it to one
megabyte that might be better if you're like have a plex server because those
are bigger chunks which means less overhead i mean our chunks are pretty
big but for whatever reason when i was playing with
premiere it seemed like it was a little worse so at least in scrubbing
yeah because it will increase your latency stuff oh we gotta do some zfs
tuning yeah because why would you want things to just like work immediately
look it's not made for NVMe we're kind of doing it dirty right now primary
cache all one word equals
metadata and then we'll do
space lambo slash z drive
okay enter aha okay
okay that just means our RAM now stores only metadata yes
which makes sense because our storage is so fast
that there's pretty much no point using our system RAM as a more conventional
cache for it for most use cases though i haven't actually played with it in
premiere it might actually like the lower latency and when we deploy to
actual wanik we'll have a terabyte of RAM so maybe
it might actually make sense but for now we're just gonna leave it off okay wow it's really poorly formatted but
lambo z drive users metadata on the next line cool it worked all right now we got
a tomb samba so on truenast smb
samba is not really configured for the type of performance that we're going for
we're talking like 20 30 40 gigabit on our 100 gig card not you know gigabit or
10 gig so we have to make a few changes now by default for some reason at least
according to this dude on the internet that has this article about trueness and
the testing that i've done it seems like they set the the default threads for smb
multi-channel to like one read one right whereas like the default in samba is
like a hundred right there's probably a reason for this and maybe they've
changed it back but from testing before changing it to testing after it was a
huge difference i wonder if it's one of those things where for the average home
user repurposing an old machine and just like throwing a couple hard drives in
which closet yeah yeah maybe it like overwhelms you know that the athlon xp
then this is also like the community edition right so if you were to buy a
machine from truenas one of their pre-built i imagine they would already configure this stuff for you anyways
we're just going to copy paste this stuff into our config
just need one more zero in there a lot of bytes yep now the thing is these
drives are capable of like tens of gigabytes a second of raw reads and
writes but as soon as you start trying to run a
file system on them like zfs performance
takes a bit of a kick in the balls especially when you have to calculate
parity which is a whole other thing that's why we upgraded the CPU hopefully
we can get a little more parody in let's see where we end up okay
we're going to do a right test first 50. here we go 12 jobs because we have 12 drives laying
out some files that's pretty good wow
that's more than 100 gig it kind of fluctuates a bit
but that's pretty damn good that is not bad as long as we're above 100 gig yeah
then we can't possibly be bottlenecked by the drives and
i think we're at bottleneck by CPU right now too i bet if we go look at the CPU
oh yeah it's probably getting absolutely crashed 100
41 you know it's probably a little bit slower in terms of writing then i bet
you if we do a read test it'll be 100 oh okay but i think it's just because it's calculating parity and whatnot true nas
is not built for these speeds we're using them for these speeds
but properly yeah all right read eight gigabytes a second you know i was
getting a little higher than this before i'm a little surprised i wonder if the
CPU is slower so far beyond good enough now something to note is that when
you're benchmarking your desktop system changing the queue depth to get like
massive numbers is not representative of the real world because an individual
user sitting in front of a computer could never use it hard enough to reach
q depths of you know eight probably let
alone 16 or 32 which you might do in benchmarking however for a machine like
this where literally dozens of people are accessing it that is conceivable and
that is a reasonable way to test it but we were only testing with two yeah so
that's why i said it's turning it up is not it's not a hack to just show you
guys a bigger number it actually is applicable to this user i wonder if we
try more threads we have more threads so let's try 24 2
give me all the threads 20.
give me more threads i want a bigger number
whoa hey there it is
18 gb bytes per second interestingly CPU usage doesn't really
go up tldr it's very fast very fast it's way
faster than we ever need and it's way faster than one so let's
try fishy you know it's good that we actually left the one terabyte in there
because of those ingest issues so that was another problem that
actually that was what really prompted me to go okay yeah forget it we're done
with this because when we would ingest footage from the stations over there
if it was greater than one terabyte it would fill up less than that is it it
basically like when you write to storage spaces it just goes to RAM and then it
flushes to disk slowly and when we had the ursas those drives could do like one
and a half gigabytes a second you know times a few if you ingested fast enough
you'd fill up the RAM and storage spaces only lets itself use half of your system
memory right and then at that point it would just crash the network share like for
everyone and there's no evident way to turn it off yeah because
our disks are fast enough we could just go straight to the disks but we couldn't
turn off the stupid round caching
and especially when you've got an os that blue screens every once in a while
having 500 gigabytes of data potentially in a
RAM cache yeah one does not simply
so this is kind of dog poo what we need to do is go into one of the editors
stations that is not Windows server running storage spaces and like copy these files
or something okay so i actually oh my god i didn't touch it
i'm sorry it's been a stressful week
what's up can i borrow this of course okay let me drive for a sec here
okay so i want to copy this to my local drive
huh that is not very good
Jake
it's possible that the issue is just Windows file transfer being Windows file
transfer so we're pulling out the big guns here bringing out show easy copy
look at that 20 gigabit oh well that's better
uh where are you going to and from right now i'm going from wanik to
to the other the test one yeah and i'm sure we open this
lots of blinky lights
so there are okay
yeah there's your CPU right there okay
i'm copying over a red mag
boom pinned to 495 megabytes a second which
is the speed of a red meg okay what else we got red map brandon is
what's yours going at
499 okay so we've got a total of a Gigabyte a second heading over there i
need to find some more media to copy here what you'll see is sometimes it'll
slow down when it's going between clips red footage is a bunch of clips so yeah
between each clip Windows file transfer will like dip for a bit yeah we're
definitely running into some kind of bottleneck on this system where it's maxing out at 500 megabytes a second no
matter what we plug into but still overall the ingestations are
performing better than with old wanik which is new 1-3 so this will be new new
new wanik anyway the point is i paused all the transfers and we're
going to have three or four editors now try to edit and then i'm going to start these again and make sure everyone's
smooth our guinea pigs are going to be ed mark
and emily over here and you guys are
pretending to edit right i mean that's basically all everyone in here does
right if everyone a prime pretending to edit what are you
pretending to edit right now i'm not pretending you know oh okay well we've
got at least okay we got at least one editor actually editing fantastic
hoffman that's a beautiful picture of david all right so guys keep pretending
to edit andy you stay here i'm gonna go turn on all those file copies just like
before we got about 10 gig on the one on the right five gig on the one on the
left and how are your projects going it seemed fairly normal or very normal or
better than normal it's not better but it's normal not better but normal yeah
it's pretty much exactly exactly the same all right thanks teddy yeah it's
good it's acceptable is it it's not worse
that's good but it's not better it's
it's hard to say to be honest okay yeah okay it's it's
i could i could have it with this yeah okay
really wasn't the goal today the goal today was hey this is going to be perfect and amazing you guys are going
to love it so uh is it no no
the goal was the same okay ed how's yours doing
um i think this is doing well historically i think it's about on par
with uh what performance was like before but as of late this is a lot better um i
used to have runaway footage pretty pretty often
in the last uh little bit for those of you not familiar that means when you
stop the playhead it like keeps going for a bit or yeah yeah okay like the
last couple weeks right yeah the last couple weeks when i would play back
footage especially uh the footage from the sony's it would
just like run away for me for a few seconds it's hard to make an accurate
car that way yeah okay now it seems fine
okay all right confirmed we're gonna move ahead with it and confirmed we're
gonna tell you about our sponsor thanks to current for sponsoring this video current is a mobile app and debit card
that helps you spend and save better so you have the freedom to do you their app
makes it easy to see where your money is going and set different savings goals to
keep you on track you can withdraw your money without fees across all 40 000 all
point atms in the u.s and there are no overdraft fees up to a hundred dollars
you can send money easily to friends and sign up takes less than two minutes so
head to current.com Linus tech tips and sign up for current today if you guys
enjoyed this video maybe check out of some of our previous server vlogs it's
the last time we tried to deploy this server how about that yeah
someday we'll get it right maybe this is it