when i signed off at the end of the video about our amazing fast new
all SSD storage server i thought it was as simple as okay let's
load the final os on this thing chuck it in the server room we're ready to start
editing off of it but it wasn't so our story begins with
some short video clips that i sent over to wendell from level one text
complaining hey about Windows storage spaces on our new 24 drive NVMe server
machine here because what was happening
was while i was copying files to what should be one of the fastest storage
servers on the freaking planet i was getting great performance sometimes and
then rock bottom performance others we're talking like
10 20 30 megabytes a second so wendell
dug into the system logs and discovered that there was some kind of a problem at
the driver or pci express level where it
was actually resetting individual drives
like they were effectively timing out for seconds at a time while the data was
in flight and then the poor array would be sitting there trying to figure out
what to do while a drive is effectively mia then the drive reset would finish
which is essentially like if you were pulling a drive out for like two seconds
and then popping it back in and then the transfer would roll along at multiple
hundreds of megabytes a second or we even saw at times numbers as high as 20
plus gigabytes a second in crystal disk mark
then it would hitch again rinse and repeat obviously i can't deploy it like
that so i thought it was my knowledge of Windows storage spaces or lack thereof
and that i had configured it wrong but then the mystery deepened so this dropping
out behavior actually happened with a simple Windows software raid with just
four devices in it i mean that's a relatively pedestrian
16 gigabytes a second by the way if guys our sponsor for this
video pulse way with pulseway you can remotely monitor manage and control all
your Windows mac and Linux machines from one app create your free account today
at the link below so we tried all the usual things we tried updating the
drivers it was using the microsoft drivers we put the latest Intel drivers
for these NVMe devices onto the system that didn't work we tried tweaking the
power management to prevent the pci express lanes from switching to lower
speeds when we were accessing all the drives and that could be a desirable
behavior because there's so many drives in here that you're going to run into
other system bottlenecks before you could possibly hope to use all the
bandwidth of even a pci gen 3 link so
gen 2 could be a pretty good bet but when it's happening automatically this
speed switching takes time and that could be part of
what's causing the problems but neither of those things or both of them were
able to solve the problem and we only got a small improvement in the behavior
so wendell suggested gee why don't we go
over to Linux as he tends to do but then get this we got the same dropouts on
Linux that seemed to suggest a hardware issue
of some sort so guys this is why i ultimately made this
video about it because this is pretty dry technical stuff for a lot of people
but i thought it was fascinating NVMe is already so fast that a lot of
stuff particularly software is not engineered for it which is turning out
to be a bit of an industry-wide problem and when you take 24 of these drives
that are capable of multiple gigabytes a second on paper that is now 24 times the
problem think about it this way even with eight channels of memory which is
pretty impressive the theoretical maximum memory bandwidth of our system
here is around 200 gigabytes a second
and real world you're looking at more like 100 to 150 gigabytes a second
now let's talk about this storage array here this
is capable on paper of about a hundred
gigabytes a second in reads so we would
need assuming perfect efficiency which
obviously never happens in the real world nearly half of our memory bandwidth just
to handle shifting data around when we're reading
or writing to our storage array that's ridiculous and even the Linux
kernel is going to be on the struggle bus when you're talking about that much
data as wendell so succinctly put it because
here's the way it's supposed to work the operating system kernel asks for
some chunk of data let's say a loot of your wife to enjoy on your lunch break
all right the disk says yep no problem but nan flash is pretty slow so i'm
going to need a sec to load that into my buffer i'll let you know when it's ready
the disk gets everything ready loaded into the buffer and then it sends what's
called an interrupt to the CPU to say
hey all right it's chill you can swing by and grab that data now
but here's the problem we're running into if the CPU core that the interrupt was
intended for is too busy doing something else or it gets put to sleep or it gets
reassigned to some other task in the middle of this process which can be
quite common on multi-core cpus that interrupt never arrives your processor
never goes and gets the data and the whole train comes to a screeching halt
and that is why we had no issues last
video slamming the individual drives with data but then as soon as we put a
file system you know as soon as we started running a
zfs raid and our CPU was doing parity calculations while we were reading and
writing to the array making the CPU actually do any work we were getting
crippling errors all over the place
so aws just rolled out NVMe and there
are a ton of threads about issues under heavy loads suggesting that this appears
to be an industry-wide problem and the dumbest part of this is that i don't
actually even need my server to be this fast i'm only hitting it with a 40
gigabit connection here that's only four gigabytes a second maximum so wendell
actually even thought of turning down the pci express links to gen 2 and just
leaving them there Gigabyte meanwhile the makers of this server was like sorry
wait you want a speed limiter on this thing but then wendell ended up finding
a software way to do it but then it turned out there was a kernel bug something something something ultimately
it didn't pan out and it didn't work anyway that's okay because Linux already
has kind of a solution to this now very very high
speed devices like RAM based caching devices operate in a
completely different mode called polling where the kernel essentially assumes
that the device is so fast that the data is going to be ready right away and it
would add a lot of overhead to do this on slower drives because there'd be a
lot of pointless hey are you done yet hey are you done yet so a single NVMe
doesn't need to be pulled but 24
oh there's an argument to be made for operating in that mode
so here's the mitigation that wendell implemented when possible the kernel is
going to wait for the interrupt because that's the most efficient thing but if
it waits for too long the queuing algorithm will just have the CPU pull
the drive rapidly and say hey do you have that do you have that okay
great i'm going to take that now
all that tweaking and learning means that our final config ended up being
quite different from the initial intention so we're using the latest version of proxmox a Linux distro that's
designed for virtualization with zfs support out of the box and while we had
actually initially intended to use zfs
we were hitting 100 utilization on a 24
core 48 thread CPU and doing
best case scenario assuming that the bug
didn't surface 10 gigs a second reads 4 gigs of second
lights which would have actually been fine remember we've only got a 40
gigabit network connection except that the access latency was not really
suitable for a multi-user video editing environment it was over 150 microseconds
and the craziest part of that is that we actually hit those numbers even with
some pretty esoteric tweaks like disabling arc compression i mean most
seasoned cfs users would freak out about doing that but the problem is that arc
compression makes three copies of the data in memory while you are writing and
remember how much left over memory bandwidth we have
so yeah tripling the load there ain't gonna fly so new plan
Linux multi-disk ain't perfect it's Linux's own built-in
software raid and the main disadvantage is that in the event of an unexpected
shutdown it'll be really slow for the 30 minutes or so that it takes to resync
four terabytes of data but that should be fine i mean that's what
the seventeen thousand dollar battery
backup in this room is supposed to be for so we settled on four striped
software raid fives and the next experiment was to play
around with the chunk size so that's how the blocks of data are broken up on the
raid as well as the block size which is on the file system level so the default
raid chunk size is 512k and the file system is 64.
but when we were running benchmarks based on an editor usage pattern we
actually found that the 512k chunks were
a little bit higher latency than we'd like to see which is really really
important when you're you know scrubbing through files on a timeline so we
actually ended up using 128k for both which happens to line up with
the buffer size on these devices perfect
now the conventional wisdom for accessing very large files over a
network share would actually be to use a very large chunk size like even one
megabyte but while that would be great for ingesting like big batches of new
footage when we're skipping around rather than reading them sequentially
with many users doing that all the same time it actually makes sense that this
would work well and experimentally so far it seems pretty good i just
realized i forgot to drive upstairs i'm gonna go grab that
with multi-disc we ended up with a maximum throughput of around 16
gigabytes per second reads and eight gigabytes per second rights which
obviously is way less than the maximum this hardware
can theoretically do but there's a lot of overhead to contend
with and besides that doesn't mean that there's no benefit to having all of this
performance in reserve so the latency advantage is something
that we've already talked about we've actually seen that high latency storage
can cause instability in the video editing software that's accessing it but
another benefit counter intuitively is
that because the storage is so fast
no one chiplet on our CPU can keep up with it a disadvantage of a chiplet
design is that it's got huge horsepower but it's hard to harness all of it for
one single task like a file copy from
one user over the network with that said it's great as a multi-user experience
because each discrete user like let's say a camera operator who's dumping red
footage and a video editor who also has to work at the same time
end up having their access spread over multiple chiplets that are individually
kind of limited so we had remember guys 150 gigabytes
per second of memory bandwidth one chiplet can't get at all of it so when
we have one user copying a file over the network that user can only get to one or
two cores so there's no way that user can monopolize all the resources on the
system because of the way the whole thing is architected all of this in
theory so far we haven't actually thrown our editors at it so let's go see if
it's booted up and get them to try it
taran was about to eat but now he has something very important to do what um
you laughed dennis but you need to help too
uh okay we're off to a good is this new yes this is pneumonic
hi alex hi how's it going
hi alex i have a new server to log you into i wouldn't even do that at this
point nope what would you do no no this is not real i'm acting out come on
and i'm not acknowledging it i add you too wait are we supposed to work off of
this or just just just i just want to know if it works
so you're supposed to work but like not important work i'm gonna mirror old
wanik over to this one one more time okay so anything you do here will be
overwritten so we're not supposed to use it look do you want me to do this
so do them but then we're going to just wipe it out okay when are you going to
swipe it what part of test is not clear just open up a project
how's it going oh seems fine
it's you know let's see if we can pump it up to full
res well that's less of a network bottleneck thing and more of a you know
the rest of the system but okay it's playing it though
which is kind of surprising what
well Linus uh you having wanting to do
increasingly ambitious projects i appreciate that we now have more space
for them us running out of space has been a large
large large problem good work it's not broken
so this does this feel any different than it was before it might be a little
snippier snappier better you don't have
to lie to me but i don't know i mean it i don't really see much difference this
is at 1 8 res though what if you crank it a bit um okay thank you
but is it better why are you asking me i'm asking you that's the whole point of
this exercise you can't do anything participating oh
okay fine from what i can tell it's actually
a lot snappier than what i remember the editors say it's good enough and
we're not getting any data corruption and the performance is
fine but every one of these line items is an
NVMe device timing out and we actually
did some troubleshooting that i haven't talked about yet so one of the first
things that we did was we swapped out the 24 core CPU that i originally
configured the server with for a 64 core
one because we found that with the 24 core the CPU during heavy reads and
writes was getting hit with 50 or more buffer flushing tasks that were each
pulling 20 usage of a single core just
choking the poor thing and 64 cores did help significantly
but i also didn't want to allocate a four or five thousand dollar CPU to the
server so we dialed back to 32 and that ended up being a big improvement as well
so bottom line the 32 core so adding just another eight cores and then
tweaking the timing between going from
interrupt base to polling based access to the drives gave us good enough
performance that we've seen three gigabytes a second when we're hitting it
with three different clients at a time in the real world without any
significant jumps in access latency or dips in
transfer speeds so we're rolling with it but there's something to be said for
like a dual socket approach to this with more spare pci express lanes and even
more CPU cores or oh i don't know AMD
working with their oems to make sure that you know when you actually hit
their pci express lanes it doesn't cause a bunch of traffic jams elsewhere in the
CPU a massive shout out to wendell from level one text by the way that guy's
anything but level one i would strongly recommend going and subscribing to him
if you love this kind of deep dive server stuff linode provides virtual
servers that make it easy and affordable to host your own app site service or
whatever in the cloud other entry-level hostings work when you start up but
you'll eventually want to get something powerful customizable and easy to use
for cloud computing they've got a diy option if you want a full custom setup
or you can easily set up your own server with their one-click apps you can deploy
minecraft cs go servers wordpress and
much more and you can even spin up your own vpn and have plenty of space to host
a website app or game server they've got affordable pricing with no hidden fees
that try to sneak onto your monthly bill and they've got 100 human 24 7 365
customer service via phone or support tickets get twenty dollars in free
credit on your new account with code minus 20 or by clicking the link in the
video description so thanks for watching guys if you're looking for another
survey video to check out maybe uh have a look at our petabyte project update
and actually we're going to have another petabyte project coming soon so make
sure you're subscribed so you don't miss it and remember how much memory
no i need to scroll down okay no problem
but give me a second um
[ __ ] off and remember how much leftover memory
bandwidth we have so yeah [ __ ] off
why then your CPU goes [ __ ] off why isn't
this working we're using the latest version of proxmox a Linux distribute
[ __ ] off i need this to work what the
[ __ ] okay