i'm not feeling great today but we have a dire situation here at Linus tech tips
our wanik server our main production
editing storage server is completely
full fortunately i have been planning an
upgrade to it for the last little while that is going to be freaking awesome
this is new wanik it's gonna have more
storage space whoa over 100 terabytes of NVMe storage it's
gonna be faster it is going to have lower power consumption it is going to
have more processing cores it is going to be epic and perhaps the most epic
thing about it is going to be how many pci express lanes it has you see pci
express gets used for all kinds of things graphics cards video capture
devices peripheral expansion but thing is it's been fast enough for all those
consumer uses for many years now so
what's driving the creation of faster and faster pci express is actually the
server world where it gets used for storage that's right that is a direct
pci express by four connection at the back of all 24 of these drives and the
truly unbelievable thing about this server is that unlike our existing wanik
server which takes a handful of those pci express connections and splits it
across 48 drives this one has a dedicated connection for
every single one we are going to be putting up some crazy numbers today
ladies and gentlemen and it's going to be brought to you by seasonic no
marketing today just happy holidays from seasonic go check them out at the link
in the video description really they they paid for a spot for that
nice guys
while i'm cracking these open let's talk about the drives that i'm using for the
server and why reason number one is that wanik is full
so i needed a capacity upgrade we're going from 1.2 terabyte drives to 4
terabyte drives because that's what a couple of years of NVMe development does
for you reason number two is we didn't want to go back to SATA because NVMe was
such a great upgrade for us in terms of not just performance but stability last
time thanks to the extremely low access latencies reason number three and really
the main reason for these drives in particular is that we got a great tip
from wendell over at level one techs that facebook was apparently flipping a
bunch of these on ebay because they're upgrading to optane so these were a
really great deal just a hair over 350
dollars for Intel data center grade drives that apparently are still under
warranty on top of everything else now they're not the fastest thing in the
world by today's standards 3.2 gigabytes per second reads 1.8 gigabytes per
second rights maximum read iops of 645
000 and rights of 48 000 but remember
guys that's almost not going to matter at all because i'm going to have 24
of them in the same server and it's just
a naz anyway now let's meet our server this
is the Gigabyte r272 z32
and we chose this particular model based on the glowing review that was given to
it by our buddy patrick kennedy over at serve the home oh
heavy boy this is a 2u server so that's to say it
is two rack units in height and it was designed from the ground up for AMD's
epic 7002 rome platform
and the advantage we get from that is that it actually has compatibility with
pci express gen 4 not all epic servers
are PCIe gen 4 ready what gen 4 compatibility means for us in practical
terms is not necessarily that much today
but in the event that we wanted to upgrade it means that we could
effectively double the bandwidth to almost the whole system so let's take a
quick tour here we've got dual 1.2 000 watt power supplies here those are
redundant in case one fails we've got our CPU socket here with support for up
to 64 cores 128 threads we've got 16
memory slots that run in eight channel mode freaking incredible
bandwidth and on the subject of bandwidth most of the internal pci
express slots are taken up of course by
the 24 u.2 slots at the front so the way that they're fed is they're using their
mezzanine card slot here to run four of them you can see that managed over here
SATA actually goes to these two bays at the back so that's what you would typically boot off of now the CPU that
we're using is pretty overkill but AMD sent over a
bunch of cpus and unfortunately
all the ones that kind of make more sense for this project like lower core
count ones i sort of have earmarked for other projects so unfortunately i am
locking away my 132 core
basically forever that's okay though we really need the upgrade
it really says a lot about the efficiency of these things that all we need is one of these basic
passive heat pipe coolers thrown on here
one of these shrouds and just the airflow from one of the chassis 80
millimeter fans to keep it cool and remember guys it's validated for up to
the 64 core one this is cool for my boot
drive i had intended to use the SATA bays at the back of the chassis
but then i realized i've actually got two m.2s that share
their pci express lanes with this 8x slot over here so i figured if i have a
couple of these old octane m.2s lying around
what the hey might as well throw them in raid 1 and
go full pci express on everything all
right let's turn it around and oh oh that's funny Gigabyte sent over a uh a
demo unit it seems to be a little bit broken
that's okay oh they've got the stupid
cable management things that make it really hard to plug in i mean it also
makes it hard to come out accidentally which is nice
now i've actually worked with this motherboard a little bit already for a
whole home one CPU and it takes
forever to post so i'm gonna fire it up while we check
the compatibility of the sleds that our drives were pre-installed on
and see if we're gonna have to swap all those out i really really hope i don't
have to it's up
that's good let's get in the BIOS and see uh if everything's detected
correctly come on baby everything detector oh would that but
memory training error what with the what now that is not all of our RAM that is
458 gigs of RAM uh-oh okay brandon i've
got good news and bad news um the good news is resetting those two memory
modules boom 512 gigs of RAM so we're
ready to rock as far as that goes the bad news is that due to AMD's
architecture of their epic processors where there's no actual chipset with
functionality like you know a raid controller or anything like that this
has no support for hardware raid either
on the m.2 drives or the SATA ones if
you want to run raid on your state of drives you have to put an add-in raid
card in and i ask Gigabyte well why didn't you guys just put
a raid chip on the motherboard and they were like well it would have used up pci
express lanes which i mean as part of the design of this board we were trying
to reserve for all these NVMe drives so i kind of went ah okay fair enough now
in Linux you can run your os on rated drives but Windows has no easy way to do
that so that's just something we're going to have to consider as we
build out this machine i'm going to shut it down for now though
because the last thing that i really want to know is if i am remounting all
of these drives to the included sleds
let's see if they happen to be intercompatible that would be so cool
that would be so cool oh
yes yeah they are oh it is so tedious it's so
tedious well that's not a good sign uh only my two m.2 drives are in here
which means that maybe the sleds don't go all the way back
that would blow all right
i put one drive in each one and lined it up and
you can see it doesn't go back far enough i gotta swap them all
damage and i've had people ask me why so many
people work here you thought i was gonna shock all these drives by myself and put
them in mounts what one person can do slowly four people can do also slowly but less
slowly pretty tedious now i just need to install a couple of roles and features
including file server then
yes we're going to shut it down and put all the drives in
and we're going to hope they all work now really the most benefit for a
solution like this is when you've got like a a storage
attached network and you're using like banks and banks of these to act as
the storage for like a whack of compute servers that are connected over fiber
optics on you know the other end of the data center or whatever the case may be
but come on guys this line is tech tips so we're using it
as an 8k video editing nas i really hope these all pick up worst
case scenario though we've got a couple of bad ones and the key night among you might have
noticed there's two drives not installed these are either cold spares or to
account for if a couple of our drives are just doa i mean that's the thing
with used hardware you get a good deal you might get a couple bad ones all
right that's a good sign every one of them has its light lit but
we won't know if everything's picked up
until we actually see them in here boom okay they're all there man storage
spaces takes a lot of your capacity by default oh actually that's not that bad
it's about 70 terabytes
okay let's try that as long as it works oh you got to be
kidding me this stupid error like
five years or whatever it's had this problem and they don't just allow you to choose
your columns in the gui you have to do it via
powershell so dumb oh you can hear it ramping up
now boys 5.5 gigs a second
well that result is horrible worse across the board than old wanik
clearly there's a configuration issue here all right so i did a simple
mirrored space this time and we're gonna try running that we've
got 44 terabytes here well that's better when it comes to
sequential reads almost 11 gigs a second not even close
to what we should be getting though okay that's it we're trying a simple virtual
disk which is basically just striped
if this doesn't perform well then i'm at a loss wow that is complete garbage town
it's even worse how is that even possible
and crystal disk mark is not the issue there's a microsoft tool called disk
speed that you can use via command line to run sort of any mixture of loads that
you want and it seems like there's just this hard
cap at around 10 gigabytes a second which is like
i mean obviously it's like good or whatever but for this hardware
it's terrible so we're going to freenas
that's it
and it's been about four days so we did manage to push Windows
performance a little bit further uh the drives needed firmware updates and they
were doing some idle garbage collection stuff like that but we capped out at
about 10 gigabytes a second which to be clear is plenty for anything you're
going to access over the network i mean we would need a 100 gigabit per second
network card in order to saturate that
um but that's not enough for us we know this thing should be capable of so much
more so we moved on to Linux Jake loaded it up with proxmox and built a pool with
four v devs each with six drives in them running in raid z1 so raid z1 is kind of
like raid 5 but without the right hole issue meaning that we should be giving
up uh let's see four drives worth of
capacity total but with the resiliency of being able to lose up to four drives
as long as we don't lose more than one in a single v dev so you'd be well let's
see six so four chunks like that of drives working
together now that should have resulted in some rocking
performance except it didn't it was actually complete
dog crap like what did we max out at like a hundred giga megabytes a second
maybe 200 200 megs a second
terrible so turns out as soon as zfs was
being loaded up our machine was actually spitting out kernel panics
that's not a chicken emergency that's like really bad you liked it
shut up you liked it that makes no sense
a single one of these drives if i plug it into my computer should be doing 15
to 20 times that performance and we verified this we checked in with our bud
patrick over at serve the home he's got one of these boxes and he was getting
great numbers with just 10 drives
so what could we do well we called in the tech support for
tech support wendell from level one tax got on the horn with Jake chatting it up
on discord and between the two of them they managed to figure out that it looks
like there is some kind of compatibility issue between the particular Linux
kernel that the latest version of proxmox is running and zfs in our
particular system that's causing these problems so we said you know what
it we're just gonna run a benchmark where we load up each of the drives
individually and slam them all at the same time Jake hit it
he wandered away he'll be back
okay you hear that thing ramping up
oh my goodness oh my goodness what's going on here is
that over a Gigabyte per second per drive 28 gigabytes a second
at those kinds of speeds we are at the limits of AMD's infinity fabric and in
fact to get to those speeds we actually had to overclock it a little bit thank
you wendell for the tip there we pushed it too far it's actually spitting out
some memory errors we're gonna have to dial it back a little bit but whatever you guys want to see impressive numbers
right well here they are now i'm running a second script that he created that
writes what was it 15 gigs of data so this is gonna write to the drives this
time these numbers are nuts there you go
23.7 gigabytes a second that are right
up against the theoretical limits of
what can be done so that's it guys that
is basically the pinnacle of modern technology nearly
30 gigabytes a second of reads and just
over 20 gigabytes a second of rights
guys to put that in perspective your home network is going to be probably
gigabit this would be 300 times that to read it you could read
one full blu-ray disc almost every second
blu-ray blu-ray blu-ray it's crazy stuff now we just
need to figure out some minor details like how we're actually gonna get an
array loaded up on the thing because this is just
i mean this is just theoretical stuff right now
and theoretically i could have a good segue for a change but we all knew that
wasn't going to happen thanks to our sponsor freshbooks for making this video
possible today freshbooks is the cloud accounting solution that's built for how
you want to work and you can work not just how you want but anywhere you want
thanks to the freshbooks mobile app you can create professional looking invoices
on the go snap pictures of your receipts so you don't lose them stay on top of
important conversations and never miss an update for example being able to see
when a client has viewed your invoice for the first time so you can get your
money you know what i'm talking about yeah you do so visit freshbooks.com
forward slash tech tips to get your free 30-day trial today we're going to have
that linked in the video description so thanks again for watching guys if you're
looking for something else to watch server related maybe check out the video
where i lost all our data you know i was accused in the comments of that video of
being a very bad actor uh brandon out of
10 how much was i acting in that video
negative 10. that happened that was a real thing