i called it
called what okay but i did though i mean sort of i'm
pretty sure it was me and i'm also pretty sure it existed before that video
even came out it doesn't matter the point is meet the supreme raid sr 1000
it looks like an NVIDIA t1000
workstation GPU in fact it even has the letters t1000 printed on it and the same
mini displayport ports are in there but but they're blocked
by solid metal
that's because this GPU is not meant for
graphics and before you say you know where this is going no it's not for
cryptocurrency either so what the heck is it through some kind of
software funkery g-raid is using this
GPU to act as a freaking storage
accelerator and if they're to be believed which i'm not sure if i do yet
this thing with the right array of NVMe drives can supposedly sustain transfer
speeds of over 100 gigabytes per second
of sequential throughput holy sh
is what you might say if i didn't segue to our sponsor keoxia their bg5 NVMe SSD
brings PCIe gen 4 performance to an affordable price for systems and
notebooks they even make a 2230 sized
one so your pcs can be lighter and smaller than ever check them out at the
link in the video description i have so many questions about this
but before we can even start to answer them we need a little bit of background
combining multiple storage devices has been a staple of computing for decades
and generally falls under the umbrella of technologies that we call raid or
redundant array of independent disks raid can serve a variety of purposes
improving speed data protection capacity or usually some
combination of all three of those compared to a single drive
now traditionally high performance raid required dedicated co-processors
typically found on hardware raid cards you would slot one of those bad boys
into your motherboard connect all of your drives to it and it would handle
both the high throughput of these many disk arrays as well as the parity
calculations that are required by popular raid configurations like raid 5
and raid 6. if you want to learn more we actually have a Techquickie on this
subject from almost 10 years ago but
raid cards have a problem as we've transitioned from mechanical
drives to solid state and then to NVMe
storage devices have gotten so fast that raid cards haven't been able to keep
pace turning them into a performance bottleneck so the current meta is to
connect your storage devices directly to your CPU via pci express this improves
both the throughput and latency but
requires the CPU to handle those parity calculations and any other overhead this
is called software raid and
in some ways it's actually kind of a big step backward first of all cpus are
freaking expensive and in more ways than you might think a lot of enterprise
software is licensed according to how many cpus or how many cores are present
in your server so you better believe that big businesses are all about
squeezing the absolute most out of every box
also cpus are generalized processors i
mean you can brute force it here's us hitting 20 gigabytes a second in
software raid but the issue is that even with a 32 core epic processor we're
looking at a lot of utilization here
just to manage storage and if you compare that to the theoretical combined
read speed of around 75 gigabytes a second for our 12 kiosks cd6 drives you
can see that we were leaving a lot of performance on that table
if i'm a server vendor that's too many wasted CPU cycles that my customers now
can't allocate to something useful or can't rent out to their customers
that is where g-raid comes into play
the most obvious difference here right out of the gate is that there is no port
to plug a drive into never mind 8 or 16
drives instead the drives connect directly to
the CPU's PCIe lanes just like they would with software raid so this server
from Gigabyte handles all of that through a backplane here in the front of
the chassis then the g-raid card just plugs into any available PCIe gen4 slot
we'll do that a little bit later and all the storage communication happens over
the PCIe bus no direct connection between our
raid card and our drives weird i guess we won't need any of our
new cable ties from lttstore.com
are available in so many colors and you might be thinking gee Linus even at gen
4 speeds a 16x PCIe slot can only push
around 32 gigabytes a second in either direction how could this thing possibly
do over 100 that's the special sauce
none of the storage data actually goes through the card
that's the old way of doing raid cards this card only handles the raid
calculations and directing the system where to read and write from
all of the actual data flow just goes directly between the drives and the
system memory no man in the middle and it does all of
this while using barely any CPU resources
or so they claim what i think this means is that we could
plot this g-raid card into any system and it would just work we are definitely
going to try that later but for now we're going to stick to the validated
server that Gigabyte sent over for performance testing our cpus are a pair
of epic 75 f3 monsters they're only 32
core but they'll hit 280 watts of max
power consumption and will boost to 4 gigahertz and then we paired these with
some equally monstrous memory micron sent over a metric whack ton of 32 mega
transfer per second ecc RAM giving us a total of one terabyte of system memory
shouldn't be a bottleneck right i don't think so it should be fine
then for our drives we're using kyocia cd6rs
they're a well-balanced enterprise gen 4nbma drive and with 12 of them in here
we should be looking at raw sequential performance of around 75 gigabytes a
second before we can set up an array though we need to install the g-raid
supreme raid software and also we need to like finish putting all the RAM in
this system it still blows me away how low profile of a cooler they can use for
these 280 watt cpus but that's the thing is under this giant heat spreader all
the dyes right it's a chiplet design so they're actually like freaking spread
out like it's it's huge it's a lot of surface area to transfer that heat
brother these are these are 80 watt fans consume
80 watts that's uh each 12 volts 7 amp
i think that might be part of the equation
wow that is a heavy vapor chamber i love it
for how small this thing is like
it's a heavy boy oh my god this is a lot of memory
it's gorgeous a freaking terabyte
why are you whispering because it's a terrible memory i don't want to wake it
up as of recording the video this only runs
on Linux server operating systems so we're going to be firing it up with do
we have an SSD in here or something back here
yeah just a little SATA SSD there cool so we're gonna be running ubuntu server
20.04 lts uh Jake has already gone ahead
and installed that onto our boot drive as well as the required NVIDIA drivers
and supreme raid itself this is cool i am liking this like super
over engineered airflow director here pretty sure they just ripped it off of
the dell but oh all right
that goes hey yeah that's not even close to full send yet
either the whole process was surprisingly easy i literally just had to copy paste some commands from their
user guide and it looks to be working so i think we can make it a right now
should we start with raid zero i'd be obviously have to start with rate zero
raid zero is not really a great use case for this because raid zero has no parity
data to calculate it's just taking each
bit writing it to the next drive in the sequence and attempting to multiply your
capacity and your speed you get no extra resiliency whatsoever in fact it's worse
because if any one drive fails all the data is
gone oh hey look at that you can totally use this with satan sas drives too i
don't know if you'd want to because i think the limit is 32 drives per
controller thing i don't know what if you could put multiple i
guess why why yeah good point why why why it's for NVMe yes okay so here let's
see list NVMe drive let's see if they all show up so we got 7.7
it's probably tibi bites or what capacity are those stripes 7.68 oh so
then that is terabytes cool there's a bit of an
it's not interesting because it's very similar to like zfs but there's kind of
a structure so you start with your physical drives you know you got your NVMe drives you
can also connect NVMe over fiber drives which is pretty cool and have the
controller in this and your your drives and some other jbod pretty sick some are
across we're not going to do that yet there is still a limit of 32 drives so
it's not like you're going to connect 200 right but 32 with that done you can
go ahead and make your drive group which is kind of like a zfs z pool or just
right it's like your array you can pick your raid level at this stage uh you can
have multiple drive groups i think you can have four um so you could have like
say you had 16 drives you could have like four groups of four sure those would all be discrete
unlike zfs they're not it's not like having four v devs that you then combine
into a pool yeah it's like having four pools yeah let's go back a bit and
actually create our physical drives
it says create that's the command but really it's like take me take me over
we're unbinding it from the operating system and giving it to the g-rate
controller there's a cool little feature here you can go like dev slash NVMe 0-11 oh
that's cool and it makes them all you don't have to do it one by one there you
go made them all successfully and then we can just check the list to see if
they're all there ah cool yes this is
pretty solid documentation from what i've seen so far compared to the documentation for microsoft storage
spaces oh god compared to the documentation for anything microsoft all
right create drive group uh we're gonna do raid and then
pd id so that's a physical disk so we'll go zero to eleven
it's doing stuff let's see if we can see it now g-raid
list drive underscore group
92 terabytes that's not bad just like that hey yeah that's fast a little bit
of an interesting tidbit here we don't have a usable file system yet this is
just the array it's not like zetta fest where there's a file system built in instead
we actually have to make a virtual disk so you can make
a number of them you could have like a 10 terabyte one you could have like a 50
terabyte one sure we're just gonna make one that's the whole thing this is just block level storage yeah so we'll make
big virtual disks that's the full size and then we'll have to put a file system
on it as well so let's do that create virtual drive
okay so our drive group is zero so we'll say zero and then i'm not gonna specify
a size and i think yeah that'll make a full-size one
and there we go we have our virtual drive it says it's optimal and it's 92
terabytes okay now i got to make a file system on
it i already like like made these commands so i can just copy past them
got the file system working i think it was a little bit angry about how i had
previously had a file system on these disks and just deleted it made a new one
anyways i deleted everything rebooted it
created it again now it's happy i also realized that because we now have two
dimms per channel previously when i was just kind of tinkering with this i just
had 16 sticks in which is eight per one per channel now we have twice that
usually that means your memory speed is going to go down fortunately on the Gigabyte servers you can force it to be
full speed captain we're doing 32q depth one meg sequential
read and that's with 24 threads so two per NVMe drive that's usually pretty
standard oh i hear it
holy it is straight up just immediately okay
it did go down a bit it's it's still twice as fast as what
we've seen with CPU raid though yeah look at why ice is fast the CPU usage is
only like and it's barely touching the CPU three four percent that might even
just be like fio but actually it says it's system stuff so it probably isn't
oh that's crazy it looks like we've looked you can hear
the fans going though yeah it knows it's doing something it looks like we've
leveled off around 40 gibby bites a second which is yeah it's basically twice what
you could get with zetta fest pretty much that's insane what's to be more
interesting is the rights because in like traditional raid 5 it's really CPU
intensive to write you'll get like a 10th the performance of your read
speed so if that's still good that will be this is grade 0 anyway
though yeah so do we even care should we just switch to right five switch to raid five okay okay
to be clear there are potential disadvantages of
going with the solution like this one of the great things about zfs is its
resiliency we have actually made significant progress big shout out
wendell from level one techs by the way on the data restoration project from our
failed zfs pools and uh we're gonna have an update for
you guys we're down from like 169 million file errors to like six thousand
so make sure you're subscribed so you don't miss that and i cannot necessarily
say the same thing about whatever this is doing well the other thing is we're
also locked into like their ecosystem now this is very like proprietary
software we were able to take these zfs vdevs and pools and just like
import them into trunas yeah even the ones i just did delta one and two those
pools are from like 2015 new hardware new software new version of zfs
i imported those 2016 pools it took like literally 30 minutes for it to import
which is a scary 30 minutes but it just it did it do you want to do raid 5 or
raid 10. raid 5 raid 10 is lame i mean
it might be lame but it's fast well no it's not okay i shouldn't say it's lame
there's a time and a place for raid 10. let me walk you through with raid 10 i
get only 6 drives worth of capacity that's it the rest is all redundant
which is great because all these could fail and i'd still have all my data but
it's bad because that's expensive with raid 5 i get 11 drives worth of data
but i can only sustain one failure on these kinds of solid state enterprise
class devices though they usually all fail at the same time
should be okay yeah you're going to want to have a backup you got a backup that's
for i mean what's our backup within about 30 seconds or two minutes or something like that anyway yeah yeah
we're on our raid 5 array now i'm going to be doing basically the same test
wow me 32 32 qs one meg read sequential
world of warcraft my whole face you see that's it's only 18 gigabytes a second
that's not that bad wait what really 18 gigs a second what's
our CPU usage 1.7 1.9 okay wow but wait now it's
getting faster wait what happened did you know that was gonna happen yeah
oh well i watched it happen earlier it does like two steps okay so it's written
you know like 30 gigs and then it starts going
and then there'll be kind of one more bump where it'll go like above 30 gigabytes a second wow
that's freaking crazy with grade 5.
still at 2.5 in CPU usage there we go 35 gigs a
second we're at almost three percent CPU now remember this is a read test so it'll be
interesting to see what the right is like yeah that's that's pretty quick 35.
that's really fast 35 gigabytes of snapchat switching to this we might
switch to this i want to try to put it in the wanix server because then we can
use the same ssds right yeah that would be even faster but i'd have
to do that on a weekend that's Jake asking for overtime on camera
yeah no i don't want to work i have zero desire to do that okay let's
try right now because that's really where we're gonna see the difference wow
CPU usage is like 20 that's pretty hefty yeah
and we're only doing five gigs a second that's not actually that great less
impressive so CPU usage is actually going down
while performance goes up yeah it's more like 12 CPU right now 11
10. it's like it takes a second to like
what am i doing where am i where am i putting stuff yeah like it needs a ramp
up he's over there i'm coming back
okay seems like it's leveled off around nine gigabytes a second with around nine to
ten percent CPU usage so pretty good CPU usage
still very strange i mean it's very acceptable
in terms of performance i mean that's a gibby gibby or maybe bytes
so it's probably closer to about 10 gigabytes a second
should we try like a random test it'd be interesting to see how many iops because that's another thing software raid will
struggle with yeah let's do that look at this guys like those wide stance here man
spreading just trying to be a little more ergonomic here this is gonna be 4k
random read we're doing 48 threads a little bit more and a q depth of 64 this
time okay let's see oh that's that's
usage though this is like the absolute
most punishing test you can do and we're pulling off
six and a half million iops on a raid 5
and actually 25 gigs a second at 4k
block size holy that's insane
oh my god so the theoretical performance of these drives would put us at around
12 million iops like raw to each of them that's
insane pretty good if we were on an Intel based system we might actually be
able to get a little bit more uh or with Intel drives but yeah
dang that CPU usage is staying
high i can tell you just from the temperature of the backplate though that GPU is at
work we can look at it actually so it's at 70 degrees
and considering the kind of airflow going over it right now the interesting
thing is the GPU usage just it stays at 100 even if you're not using the array
it's kind of weird i wonder if that's like uh
look the fan is spinning it's 55
it just has like ah it's like you you have like a little
desk fan inside of like a like a hurricane yeah a wind tunnel just going
past it oh thank you for the cooling yeah
let's try rights same specs everything else let's just
give it a sec to no no sec
no sex no i mean i mean that's a camera
and that's the kind of operation we're in that's one
yeah it's writing though that's way harder so doing one million iops writing
so these drives are only rated for 85k random right well that's really wow
so that's that's actually almost perfect stealing yeah that's freaking incredible
so if we do 85 times 12
it's almost perfect scaling this that's probably the most impressive
test we've seen so far then yeah that's crazy
what's cool about the wrights still
maxing out these drives though is that because people are actively editing off
of these drives while you are dumping copious amounts of
data onto them this could make a huge difference to
footage ingest this is freaking crazy we're still gonna
run into a huge bottleneck that is samba but maybe once they have smb direct on
Linux like rdma support
we kind of have to deploy this yeah i think so
let's find out if we can oh yeah okay so that's why the bench is
here i got a threadripper bench here we're gonna just put that card in it i got us a little NVMe
drive we're kind of clashing brands here this is like wearing adidas and nike in the
same outfit we got our our liquid drive this has got four NVMe drives on it and
then just like a little plx switch so we'll put that in there and then we can
raid those four drives okay it's even less sophisticated than i thought yeah
it's just that's not on a normal card there's no like it's just this piece of
metal so then they just
probably sourced a random pci it looks
the same as the regular NVIDIA one just because like these slots are the same
but there's no cutouts yeah okay sure i'm gonna go get a cooling fan because
that looks awful which one the one in there oh he's fine
no he's not fine Jake he'll be all right
he's a good guy so let's see describe license and see if our license is still
valid license is still valid well that's awesome interesting
okay yeah it's these are nvmes interesting
oh these don't support 4k block size
oh do we do we think this is all we really needed to know it boots it runs
it probably works let's just boot into Windows and we'll get the last the last
answer holy sh it's their Linus what NVIDIA t1000 is device manager it's just
a freaking GPU i just want to see if this BIOS version matches any of the
BIOS versions in the tech powerup database because if it does chances are oh
hello uh yeah i'm pretty sure that just worked
gc display connected no way oh my god it
just works it's a graphics card well i mean we knew it was a graphics
card it's a functional display outputting graphics card
okay i got to see this BIOS thing now it's a pny t-1000 so they must be who
the uh pny is the exclusive manufacturer of
quadro or excuse me NVIDIA rtx
works it's exactly the same in every other metric
GPU accelerated raid what world are we even living in
it's just a regular ass GPU and it's not even a crazy powerful one like this is a
basically what like a 1650 yeah something like that literally a 1650
same same silicon basically so what can we just run can we just run g-rate on
like freaking like a6000
so maybe it just completely doesn't care what GPU should we put a different GPU
in that should i go get a GPU there's got to be some ballad i'm going to get a
gp okay well what i'll be back do we have an a anything i'll just get like a
quadro or something or a t would be better touring excuse me jesus
it took a little while to generate the rendering kernels but uh
it blenders oh i think um i found something it's pretty fast too oh my god
are we going to have g-raid check that license
so let's see let's first see if
the server is okay it's not running on the GPU you see
it would it would show the running process so let's see it might not work
i don't think it's going to bomber that would be so funny
i think the license is done per GPU oh i wonder if it
as part of applying the license that binds it to it
it's relatively unsophisticated it makes you kind of wonder why they would even
bother at that point but money no no i don't mean that like
licensing is a waste of time i just mean like it's relatively unsophisticated if
you wanted to spoof that yeah you can do it yeah yeah
huh oh i'm disappointed i wanted to like
throw eight times the compute at it and see what it would do
oh well to be fair the game is not launching watch g-raid reach out to us
after this and be like yeah we can hook you up with that and there we go continue campaign sure
five percent how do i play this uh with a controller this is a good game
you should totally play it though it's really there's actually no way to
play with the keyboard i don't know it kind of looks like i like wasd or
like shift god damn it all right i'm glad i told rocket league to keep downloading
this doesn't launch either what the hell yeah the t-1000 is behaving kind of
weird but it can game we launched broforce it's fine and you know what else is fine our sponsor
thanks to telus for sponsoring today's video we've done a ton of upgrades to my
house over the past few months and probably the most important one to me is
our new telus pure fiber x connection
telus pure fiber x comes with telus's wi-fi 6 access point and can get you
download and upload speeds of up to two and a half gigabit per second that is
the fastest residential freaking internet speeds that you can get in
western canada and is perfect for multiple devices and simultaneous
streaming with the new consoles out it means that you can download a new 50
Gigabyte game in less than three minutes
or a 5 Gigabyte 1080p movie in just 16
seconds assuming that the connection on the other side can keep up with you
you'll also get to enjoy an upload speed that's up to 25 times faster than
competitors which means that your streams will be crystal clear i mean you
could be streaming in like freaking 8k at that point so get unparalleled speed
on canada's fastest internet technology with telus purefibre x by going to
telus.com purefibrex
if you guys enjoyed this video this little SSD from liquid here is hardly
the most potent we built a server using five i think of their like way bigger
eight SSD honey badgers it's called the badger den and it's freaking amazing
that's crazy