oh damn it what is that a total 128 course
64. but good try and a terabyte of memory that's 256 gigs oh supermicro
told us that we're not allowed to build this server ourselves they have to build
it for us naturally we said no so we are
going to be taking one of the thin 1u
storage servers from the petabyte of flash project and seeing just how fast
we can drag race it with a handful of keyoxia cd6 drives octane acceleration
64 epic cores the fastest we can get and
is this 200 gigabit per second network it's 200 gigabits and 256 gigs of memory
the optin's just for boot this thing's gonna be crazy fast almost as fast as i
can segment to our sponsor glasswire are you having poor quality video meetings
use glasswire and instantly see what apps are wasting your bandwidth during
your meeting and block them get 25 off
today using code Linus at the link below
in a way a server is a lot more like a laptop than it is like a commodity
desktop made of off-the-shelf components because they tend to be way more
tailored to a specific use case and
they're not really as flexible say you want to build a storage server there's a
dozen different ways to skin that cat pardon the expression for example here
at Linus media group our primary concern is getting as much capacity as possible
at the lowest possible price yeah so we're willing to give up some compute in
favor of stuffing more drives into a single chassis that's why our storage
servers tend to be this thick or this thick the reality is that 4k or even 8k
video editing is pretty demanding especially if you've got you know 10 or
a dozen editors working off the server at once compared to enterprise or
scientific applications it's not even close so that is why any
good server deployment starts with the chassis this right here is the super
micro super server 1124 us and it is all
about density not storage density because if we went
with a 2u remember that dual layer old
wanik they absolutely could pack in more drives but they choose not to because
you're going to run into performance bottlenecks if you don't have enough
compute and that's the density that we're increasing here by going with
these 1u's each layer of this 12 drives yes
but 2 cpus so no bottlenecks right that's the idea
super micro only sells these as a complete system these days meaning that
it must leave their warehouse with a minimum of two cpus four sticks of RAM
and at least one storage drive and the intention there is for them to be able
to ensure quality and compatibility and then as a side benefit obviously they
make some money off the parts but because of petabyte of flash project we
were able to get our hands on some bare bones ones so let's take a closer look
wow built in on board you've got dual sfp
ports are those 10 gig or all four of those are 10k all four of these ports
rj45 and sfp are 10 gig dual usb 3s
we've got an ipmi management port serial vga that'll have that vga as well as two
PCIe 16x slots back here and what do we got for power there's three there's
three oh there's a third one oh look at that oh wait there's actually four
there's there's one more like hidden inside we'll see that later oh cool okay let's have a look at our power supply
obviously dual up to 64 core cpus wow
1200 watt power supply huge strictly
speaking this they didn't actually send us a bare bones they sent us a completed
one and we took it apart oh do i ever have the story for you on a
call for this project the super micro guy was like you know taking out a PCIe
card that's easy but you know get to a CPU there's there's pins and thermal
paste i'm like bro i have probably taken out slash
installed at least a thousand cpus yeah is it wrong for me to just love looking
at thermal solutions for super thin systems like this really you know what
i'm looking at the RAM slots it's like
sixty percent of the width of the server is just RAM slots it's a forest of
memory slots why aren't we putting more memory in then um fetch me more memory
no no no the thing with epic is we want to have all of the channels filled out
so that's 8 per CPU but once you add more it can be harder to hit the same
speed and the same latency and speed and latency of your memory is
super super important if you're running software raid which is exactly what
we're going to be doing with zfs we're using dfs right well for now just
to test it but the actual deployment is going to be using weka fs which is a
different thing that costs hundreds of thousands of dollars but seems to be
software raid too so yeah i guess
whoa what oh what the hell
that's cool you dropped something it comes out as one big fat mama of a
module here i love it i don't know where that's right
now this is a fun fact small fans not
great at moving a ton of air because i've got these little tiny tiny
blades but what they are really good at is generating a ton of static pressure
which is really important in a deployment like this see look at the
front of this chassis it's going to be all full of drives in there right and in
order to fill it with drives you've got to have a backplane for them to connect
to well that back plane has a hard pcb and you can see that there's only tiny
little gaps in it wherever they were able to get a little hole to draw air
through the front of this chassis they need to generate enormous static
pressure in order for there to be enough airflow to force over the CPU's memory
power supply and pci express cards did i
say power supply power supplies they're redundant in the event that one fails
and it's also super useful for connecting your server to two
independent power sources in case your power source fails side note Jake i
think this might be the thickest pcb i've ever seen holy sh i mean you want
rigidity obviously especially somewhere where there's going to be mechanical
strains on the device it's like two sticks of RAM yeah thickness here let's
get a shot of this just for context here's a stick of memory i think it's
more like three Jake what that pcb is is
almost an eighth of an inch thick that's crazy oh this is interesting you can see that
in order to avoid recycling any of the hot air back to the other side they've
got these little like rubber curtain things anywhere where cables
have to pass between the front of the chassis and the back and that's not the
only cable management trick it's got up at sleeve power runs up this side but
the front enclosures also need pci express connections for the NVMe drives
and all of those are flat connectors
check this out that run right in between these memory slots to these sick
freaking PCIe connectors that go into the motherboard and do they have any
cards for them no they just all come directly off the board yeah there's the
little ones over here too of course you can add even more NVMe storage if you
wanted to there's the three excuse me four PCIe slots here oh wow
this is see he this little guy he's right here oh there it is that's where
we're gonna put our octane oh wait actually no it doesn't fit ah oh god no
it's fine oh cool okay so this is a dual
riser on this side you've got a simple PCIe 16x to 16x slot and knowing AMD
epic it's probably running at full speed that's right actually all of this is
just going to be full speed PCIe gen4 and then over on this other side we've
got i believe this is a PCIe 32x slot oh
that is crazy Jake it is a 32x slot you
can see they've actually got the pins that correspond to each of the 16x slots
silk screened onto the pcb well if you think that one's crazy look up here
that's amazing i love it there's 1 16
another 16 and then a what is that eight it's an 8x right over here i think some
of the nvmes run off of this oh you know what the 8x is running these sfp ports
at the back oh no bottlenecks and then
one slot and then those are two more 8x NVMe connections running to the front
yep there's so much PCIe and again very
purpose built right yeah you can put a GPU in here though look
GPU power if you wanted like an a100 or something again if you had a very
purpose build specific use case oh my god i mean we did see a storage
deployment recently where GPU acceleration was used for raid
parity data i have a bit of an update on that one wendell
informed us that there could be some issues with that particular solution we
tested it there is an issue
basically we we stopped the array edited one of the drives i think we edited 32
bytes of it to be something different started it up
and it didn't fix it it just has no error handling what is it is depending
on the drive to tell it that there's an error Jake i just realized something i was
trying to figure out why the front of the slot was over here and i was like
right that's where the power pins are yeah so it's got normal size power pins
and then these itty-bitty higher density data pins now the goal today is to see
how the system would perform if you were just to set it up yourself with something like zfs but even with a more
optimized and actually specifically built for NVMe file system like weka fs
you still need a lot of CPU compute to handle things like networking the actual
connection to the NVMe drives themselves and any sort of networking overhead
fortunately AMD stepped up to the plate and provided 12 of their 7543 epic
processors so those are 32 core each for a total of 64 cores in each of our six
servers these are configurable to a max TDP of 240 watts and a max boost clock
of 3.7 gigahertz they're not quite as fast as the 75 f3s we had in the g-raid
server but they're still plenty potent for what we're trying to do here so uh
let's get them installed i got i got to prove super micro right here i know how
to do this david i swear i swear i can put a CPU in
i don't know watch me screw this oh
ah you saw nothing you know my my hands coin aren't what
they used to
be all right david i'm doing the the most dangerous part here thermal paste
don't want to mess this up oh i already messed it up
is there treasure under that look at these bad boys it's crazy to
think that this could handle a 280 watt CPU like there's just underneath here
there's going to be a massive vapor chamber that just spans the entire thing
now it's time for the tedious process of installing all of these sticks of memory
the dims are made by samsung they are 16 gigs each and they run at 3 200 mega
transfer per second but the most important thing about them is that they're qualified by super micro for
this particular server and i get it you know who would want to run unqualified
memory in their mission critical server it's like drinking from a non-LTT Store
qualified water bottle
crazy i think the craziest thing about this memory setup is that it's not even that
crazy 256 gigs of ecc error correcting
memory would be mind-blowing for a desktop but for a server this is
pedestrian this is a storage server we don't actually need to put enormous data
sets in memory for these cpus or gpus to crunch away at these are to make sure
that each of our cpus gets two full fans
worth of dedicated airflow blowing through them in our final deployment as part of our
petabyte of flash storage project oh we're gonna have six of these acting
as NVMe over fabric posts and it's kind of similar to iscsi in that your storage
is in one box over here and then it's connected via networking to your compute
box but NVMe over fabric was designed
specifically with NVMe devices in mind so it's way more performant but to push
that kind of speed you need to make decent use of the drives right and that
requires a lot of networking a hundred gig
huh can i get aha
400 gig is what we're targeting with dual NVIDIA connectx 6 series cards and
i couldn't help noticing that one of these has a half height bracket on it oh
you want it on this side yeah it's for cooling for more cooling get them
separated it's just gonna hang there teamwork in it i'll go at it from one
side you go at it from the other yeah i think they called that
i thought you wanted to put this one on this one it doesn't fit oh yeah it's
fine we can just take too much too much cable oh careful
yeah we really need to not break any of these if we're going to hit our petabyte
of flash storage and also we don't want them to be able to say i told you so
they did tell us not to bother them
this may be the most overkill boot drive of all time
especially like it doesn't it's it's not redundant though so it's like actually
not that great hi in fact many server motherboards most
even have an internal usb port that is exactly for that that is what it's for
really yeah it's for just running an os off of usb but just using a cheap thumb
drive and plugging it in a lot of them also have an internal like little
powered SATA thing and that's what we'll be using for this one
uh we're gonna in the real deployment yeah i took it out oh well where does it
go oh is it the super dom one here yeah
um something about that doesn't look right yeah i did not put this in right
yeah oh my god every time you didn't screw this
in oh i forgot oh i can still access it where'd the screws go just because
they're through like a
no that one i screwed oh i didn't screw that one in either last but not least
storage the actual deployment of this cluster is going to be making use of 12
cd6 15 terabyte drives per server but because those drives already have like
specific demo data assigned to specific
slots we had to be very careful about taking them out did you see my little
diagram oh no i didn't oh oh yeah baby
oh my gosh it's perfect okay i mean i
labeled the drives i didn't want to screw it up that's fair
that's fair because that would be catastrophic so instead we're going to
be using these seven terabyte cd6s that we already had laying around and we're
going to be installing truenast to run zfs on them just to see like if you were
to buy this server and these drives yeah how much could you get without spending
400 000 on a file system yeah that
i mean we're expecting really impressive results even without the fancy file
system because these are PCIe gen4 drives that are capable of in excess of
what is over six gigs a second these are exactly six gigabytes six gigabytes a
second of throughput so put together that's around 75 gigabytes a second the
interesting thing is the 15 terabyte ones are a little bit slower i think
they're five and a half gigabytes a second so it works out to be closer to
like 65 gigabytes a second which is a lot closer to the 50 gigabytes a second
that our network can do shout out super micro by the way for
these tula sleds this was so fast compared to when i built that simply
double server that you had to screw them all in all of them even with one screw
boy does that ever add a lot of time you kind of have to do two
so that's like oh 96 screws he's not
even doing it right and he's complaining are we done that was it that's it
wow it's so cute freaking crazy i mean
cute is like not giving it enough credit 400 gigabit per second okay shall we
plug her in captain it's gonna complain that i only plug in one of these and
then we'll tell it to shut up this one doesn't have a shut up port but it does
have a shut up function called unplugging the power supply oh uh well
we could just plug it in okay we're gonna plug the power supply into our
server oh that was a little rough oh i got the wrong power cable
one moment please when i was living at ivonne's house like
with her parents yeah i had a GPU make
that noise when i forgot to plug in the PCIe power yeah like an 800 and it made her dog
throw up this thing is surprisingly quiet for a
1u i mean i know it's idling but still
i guess if it's not doing anything yeah well it's nice
to not have to hear it just oh some of them the power supplies
no never mind that's still not bad two 32 core processors
all right we see 13 drives looks good we got our boot drive and our 12
7 ish terabyte drives it's time to make our pool should we do realistic
or should we do full send i actually don't think a stripe is going to be that
much faster honestly we might be best to just do like two
raid z ones two raid z1 pools would allow us to have two drive failures
before we actually experienced any data loss and the way Jake's going to
configure it is with two six-drive v-devs that we will then combine into a
single pool all right what do what do you want to call this
i am speed 69 timmy bates you know what it's even
dot 84 like that's that's double 420 you
know we're going to make a couple tweaks here
they've actually updated it so a time is off by default but it's a time it's like
that it records the access time of the data you only need that for like very
specific use cases or diagnostics i would think yeah but uh you don't want
it it's not good for performance unless you actually need it uh we're gonna go
from 128 to one meg because that's kind of closer to our use case of like video
which is big files yeah if you were to host like a database or something where
you have lots of random reads that are small like especially like a text-based
database uh you would probably want a smaller record size but for us one mega
skill 128 is the default for a reason yeah that's excellent for a mixed use
case yeah okay and we want to do one more thing uh we're going to set the arc
that is the RAM cache of zfs to just be metadata only if you use it for files as
well when you have such fast backend storage you can actually lose
performance yeah so setting it to metadata only gives us a little bit of
acceleration from it but not the same kind that arc would for hard drives so
we're running an i o depth of 32 which is somewhat unrealistic but two threads
per NVMe so 24 threads total at a 128k block size you make the server
go fast your way i'll make it go fast my way here we go ready
did you just unplug the network oh uh
maybe maybe but i did it really fast yeah i think you did just unplug the
network cool well it's fine now 15 17 20.
now we've done you know 18
20 gigabytes a second on a zfs pool before
but what you have to consider now is that we've done that on servers that
were generally double the thickness so in a cluster deployment where density
is key you're able to get effectively double
the performance of your drives by having two one use by adding all that compute
that's the point of this and that's what made these ideal for our petabyte of
flash projects we're ramping up baby almost 30 gigs a second oh wow
look at the CPU usage those cores are
going i bet you if we switch our tests to a 128 block size
uh leaving the array at one meg this is going to go even faster yeah there's 22
20. see if it ramps up even higher five threads at a hundred percent
well there's more than that if you look at it like realistically so this is a
write test sequential write we're looking at around 20 gigabytes a second
as well what that tells us is that we are still
CPU limited because in theory these drives don't write as fast as they read
these are a more read optimized data center drive that's actually really
impressive considering that we're dealing with parity data here though and
is that a fast bump the threads up a bit
man i am excited to see what this thing can do when there's another five of them
in a cluster yep okay so this is a random read 4k block
size we're doing four threads per drive and a 64q depth this is not only going
to be a petabyte of flash it's going to be the highest performance setup that i
look at how i'd ever see dog crap that is
oh that's a shame a hundred and fifty thousand iops individually
these drives will do more than millions i'll do a million each
that's that one meg record size kind of hurting us sheesh look at our cores they're just
picked wow
yep 47 threads at 100
right now poor thing and what about a random right
poor drives we're just abusing them oh that's embarrassing 20 000 iobs this
is literally slower than a hard drive but no it's not a hard drive
sequentially if we were doing 4k random writes to a
hard drive way slower than this so that's something
you got to keep in mind about these numbers it's not as simple as just
megabytes a second the kind of data that you're hitting your storage device with
makes an enormous difference really that brings us back to what was kind of the
whole point of this video doesn't it yeah that servers
have to be designed for the application that they're intended for and these are
absolutely perfect for what we will be doing with them but not perfect for what
we do with our regular servers here like we wouldn't replace new nuwanik with one
of these that's the thing with software raid man random reads and writes are just
not it but you know what is it our sponsor
manscaped the new manscaped ultra premium collection is an all-in-one skin
and hair care kit for the everyday man and covers you from head to toe there's
the two in one shampoo and conditioner their body wash with cologne scent
hydrating body spray deodorant and a free gift moisturizing lip balm
your man maintenance just got easier and best of all all manscapes products in
the ultra premium collection are cruelty free paraben free and vegan visit
manscape.com tech or click on the link below for 20 off and free shipping
if you guys enjoyed this video go check out part one where we got into way more
depth about the complete configuration including taking a close look at the
eight GPU server that is going to act as the
the head controller for the six of these
that we're gonna have stacks i wanna do a video like this on that server and
like boot it into Windows just for feeds NVIDIA specifically told us not to mine
on it let's just do it like could they hate us anymore at this point