is this just like oh yeah yeah but where are they
really buy an order of magnitude this
will be the single most expensive unboxing we've ever done inside these
boxes this crate this server rack this
additional box and this final box over here this monster
server from super micro is everything that we need to deploy a
server with a petabyte of ssds keyoxia
provided the ssds they sponsored this video it has been a year in the making
and we are about to see performance the likes of which we have never seen and
may well never see again realistically like yeah in our lifetime
they're not even just ssds or NVMe ssds let's bring back holy sh that's a series
we haven't done in a while holy holy
holy think that server's worth like a hundred thousand dollars a hundred
thousand dollar server you said something like that well let's uh move it gently then
there's gonna be a whole follow-up video detailing the exact
architecture there's gonna be multiple videos detailing the setup and
architecture of a petabyte of flash because while keoksia does make such
high capacity NVMe ssds now that you could pretty easily build a server with
a petabyte of flash accessible to it but in order to get the full performance out
of it
oh my goodness you gotta do something
a little special i don't think i'm gonna let you pick this one up
you know you gotta give NVIDIA credit
it looks pretty special does their hardware have sex appeal or does it have
sex appeal i think this is just the module this actually slots into the the
main server so what are we looking at here so each of these
is an NVIDIA a100 80 gig so we've got
eight of them envy linked together that's
640 gigabytes of vram
video memory these cards alone are probably worth
more than any other individual server this thing
this unit is almost 100 000 us dollars just this part of it and what are these
are these just like some kind of i think that's just power delivery
they asked us not to take them off but really we already have it so like we
could just do it anyways i mean NVIDIA can't hate us anymore can they yeah well
no they probably can you know what let's not piss off the enterprise guys this
looks like paradise can i just say that it's all sponsored by keoxia who
provided the flash but NVIDIA has been absolute bros and broads about this
whole thing because we wouldn't have been able to do it without them so i
guess yeah let's pick it up um why don't we move this off of the server so we at
least have somewhere to put this i don't know what's under it the server oh wait
that's it ah yes oh that's why the box was so big
yeah oh look at those rails i know right
these are chunk tasks let's just yeah let's just lift this uh put it on top of
that sure sure that's safe so how did this even come to
be now i've been babysitting this for a year a year but i wasn't there for the
first conversation what happened was keoksi has worked with us a number of
times and they were looking for you know some kind of cool SSD related video to
do cool you say i took an absolute long shot
because this was after we'd already seen the appetite for petabyte project on the
channel right but they're a flash vendor and i was like well it would have to be
ssds and it would have to be and i thought they were gonna come back with
something way smaller don't tell kyocea but they would have gotten away with like a quarter of this
but i was like well it's gonna it can't be smaller than what we've done before
and they were like oh this is actually a very interesting engineering challenge
from our standpoint let's see who we can partner with to try
and make this happen and the answer was super micro AMD weka
freaking everyone you could possibly name in the server space so how does
this friggin work okay let's lift this boy up we're going to start at the back
which is a little unusual this is our head unit
this is PCIe over fiber
because the ssds don't actually go in the control unit this is where all the
compute lives in order to do this kind of throughput you need a very very
exotic very powerful solution the ssds
themselves this is the size of an atx power supply are gonna live in
additional units that we're gonna unbox in a few minutes here
how many watts are these holy sh this is a 3 000 watt power supply
can we even power this i've never actually seen one of these
plugs properly used
i'm just curious what is in here oh oh
wait what is this just like oh yeah yeah this will
be like so you gotta unscrew it then uh no sorry
like i meant this part oh yeah okay no you're right stop though you gotta take this side out
well i thought you were gonna i don't have a screwdriver what are you doing what are you doing over here just
waiting for you to hand me the screwdriver well
oh
so here's what we're looking at we've got our uh PCIe over fiber cards here
these are just NVIDIA connect x6 200 gig
cards wow there's eight of them and then we've got more PCIe so you can see we've got
there you go right there okay we've got a little bit more PCIe to these NVMe
bays at the back here so you got four more of those
and then you've got a little bit more PCIe okay up here and that's going to
this which is probably like a dual 10 gig card or something like that that's
just just gigabit like for management you know what
i know you like that much PCIe i got to tell you this would
there's more PCIe oh are you serious so this unit can take
a total of 10 NVMe drives and then it has another eight PCIe 16x slots then
there were those NVMe connectors i didn't recognize to the sleds at the
back and then there's all the a100s oh my
goodness so do we slide it in uh i think we should put it in the rack
first they have to be epics of some sort oh they are so dual 64 core where are
they oh they're in the top they're in like the very top yeah so there's the PCIe
lanes for the a100s that just sit under it
very cool i don't know how to take this out cause kind of i wonder if it got hit
a little actually in shipping looks a little wobbly a little bowed
oh boy oh boy oh there's a screw right here oh there's your problem
you got this don't drop it i got it dual 64 core
epics i have to assume that that's like is this two terabytes of samsung memory
uh the spec sheet's at 256 but i don't actually know who sent it okay
here let's just pop one out oh boy they're not gonna like this that's a 64
gig dim dual rank 3200 megahertz samsung well it's super micro memory so yeah
this is they're qualified so 64 gigs that would make this yeah 32 2 terabytes
two terabytes of RAM oh look at those frequencies these
things must scream yeah these fans one
of those 40 mils gotta be absolutely ridiculous
this is a lot of hardware in one box on freaking reel love the airflow
management here gotta put basically all of it through
the cpus the rams the rams just like a little bit yeah come on maybe there's a
little bit of cooling it's like a tiny little spot there yeah these vrm heat
sinks are like i don't need much
there's business level hardware and then there's enterprise level
hardware this thing is unbelievably heavy
there is a lot of freaking cooling in it it's like a
solid ass block it's very heavy yes i understand let's put it in the goddamn
thing huh how many watts oh look at that they
got flappers what the hell holy crap the
fan itself is this thick yeah wait no is
there two in there crazy there might be two in there so there's two blades
see this yeah so you've got this blade is swept this way they spin
independently and then this one is swept this way because if you want more static
pressure you can't put two fans that spin the same way there's some kind of
aerodynamics thing that makes it so that doesn't work very well they have to be
spinning counter to each other uh we didn't really consider that when we
assembled this up here we were going to have to bring it down
yeah okay where are we going to put it we're
going to put it on top of the rack for now i'll take the lights off i see we ready
oh my god okay if we drop this like this
game over no no you know we're good we're here's what we're gonna do this thing already got beat up to hell in
shipping so we're gonna bring it down like this
go ahead and put it on yeah there you go little guy
oh i don't know if little guy is the right word for it absolute unit guy this is
when you have all the budget because it's not just
powerful it's dense you got to remember in the data center
density has a ton of value because building a data center is the most
expensive thing about building a data center so each of these 1u servers are
going to be our storage hosts there's six of them six six we gotta put 72
drives in here now you might think hey guys this is austin
you might think hey guys why didn't you just pick servers that could take more
NVMe ssds well here's the thing once you've got high performance enough ssds
and these are all keyoxys cd6 PCIe gen
4x4 NVMe drives you run out of CPU power
as well as PCIe for networking as well so you can't
just put as many drives as you want in each server you gotta add more compute
you gotta add more networking yeah it's a dual processor so these are
oh gosh 32 cores each and a far more reasonable
i'm guessing this is 512 gigs of RAM it
says 256 but i think this is wrong yeah only 256. and then of course we've got
the NVIDIA connect x6 200 gigabit network cards that are going to handle
our NVMe over i guess actually they're not fiber are they because they're so
close i think we're going to be using direct attachments but how do you call it NVMe over fire it's
NVMe over fiber just with fiber you can go farther is that the boot SSD it's
adorable oh my god it just slots right into the it's a little see the thumb
drive come on out i think there's probably a
lot there's probably a lock on it oh yeah there is look at that
these are these are super cool oh look they even included a usb type-a
port so you could run unrate on this thing yeah so the reason that people use these instead
of an internal usb drive is just that the grade of nand flash that gets used
for a device like this tends to be higher and while honestly in a real
enterprise deployment uh a boot drive failing is probably not the end of the
world a lot of the time they're booted over fiber too yeah it's just
why right just use something better now you might be wondering why we went AMD
epic which means you probably haven't been paying close attention over the
last few years but basically these will boost up to 3.7 gigahertz
giving us tons of compute 64 total cores
and they have lightning fast not to mention a ton of PCIe connectivity so
every device in pretty much this whole
rig i think is going to be running at full speed maximum definitely we don't
want any bottlenecks we can't have bottlenecks before we move on a couple more fun
things i noticed about this these are 20 watt fans
80 watts that's 150 watts in fans that
just oh yeah they're like the same type epic has so much PCIe that you can just
put a PCIe 16x slot just chilling in the middle of your server
just hang in there yes this is a 200 gigabit card
one of two each of these is gonna have a 400
gigabit per second connection to our head unit
the best part there's that slot there's another one back here oh wow second one
over there well that one's at least accessible from the outside yeah yeah this is just hanging they're like wow
you want to put some NVMe in there no just you only live once man
now for the next part you guys saw this crate we were using as
a table oh it's very sharp what the heck is all
this wait hold on go down what i think there might be more servers in here
kiosks yeah oh these must be the drives
oh what's this from mel oh excuse me excuse me the box says melanox but i
mean to say NVIDIA NVIDIA acquired melanox somewhere in between this box
being printed and us doing this project
what could be in this box hmm oh wait what
we originally asked micron to send memory for this project but super mario was like oh it's not qualified yet blah
blah blah blah blah wait what i think these are just eight
gigabytes pick 3 200. can i take some of this home
is it all aching i don't know
well thanks micron what's behind this door in
fairness though there are some really cool cables in here there's power cables
those aren't the cool ones and then there's these guys
these will do 200
gigabit per second and these are the direct attached copper cables that we're
going to be running between our head unit and our SSD units
now there's networking gear that's like pretty cool
and then there's this this is a melanox well NVIDIA
3700 series 200
gigabit switch that's right each of these ports 200
gigabit per second and it can do it all at full speed at the same time this thing is bananas
they will all do 200 gigabit per second it's always fun when you start measuring
your network switches and terabits yeah
the fans are 30 watt 31 fans i want to see how big of the
power supply
really i just looked at the spec sheet of that switch
12.8 terabit uh oh they are
oh that's hilarious we were look kioskia
are you guys new to working with the position matters oh the position
matters yeah don't don't switch them around oh i was going to be all like
kyocea come on what's the fun of you know unboxing all the ssds if you don't
put them in a box but okay the position actually matters a lot for
this configuration so we'll just take a look at this drive and carefully put it
back in here each of these is 15.36
terabytes and can deliver a total throughput of six gigabytes per second
which is impressive considering that we're gonna have 12 of them per server
so that's 72 gigabytes a second per server absolutely
mind-boggling numbers but there's even better stuff so it's obviously using a
completely custom solution everything from the controller to the flash is all
keyoxia through and through and this puppy can do a million iops per drive so
by the time we're done with this we are going to be putting everything we have
ever done before to absolute shame
if you don't watch our channel regularly you might not have heard of kyocia but
we've actually worked with them a fair bit they were the first to deliver a u.3
capable drive the first to deliver PCIe gen 4 drives they are formerly toshiba's
flash division so they've got literally decades of experience working
with this kind of technology and i'm really excited to see
what these drives can do at their absolute
peak of course speed's not the entire story and i'm obligated to tell you that
they are of course using the same high quality flash that they use in their
high-end enterprise sas and other NVMe products with the same reliability
features and of course also they've got support for everything from non-encrypted drives to sanitize instant
array self-encrypting drive scd fips to level two certified
if you need it and it's an SSD kyoku has something for you
or something better than you or something better
the final piece of the puzzle is our server rack
sort of
let's talk about how we're powering this
oh my goodness what the heck is going on here
is this the most wack power cord you guys have ever seen here it is
next to a human hand for scale i mean i don't know about a human
normal hand freaking ridiculous these are l630s so that's 240 volt 30 amp
each of them we didn't actually have a place to plug these in uh yeah we're
going to be stealing i think one of the plugs from the welder and then one of
the plugs from the mill i got infinite cables to make these giant extension
cords they're like just as thick maybe thicker
so yeah we can't use we can't use the welder or the router when we run this
rack but uh i think that'll be okay infinite cables also provided a 100
meter 100 gig cable so we can plug it into the switch back there no i guess
we'll have to like run it outside but yeah guys this is going to be a really
fun series we're going to go through the hardware the software
we're going to show you ai benchmarks yeah it's going to be great 4 times 12
384 300 plus another one another 512
cores
that's the same amount of RAM we have in our main uh editing server we're going to have 512
cores and a terabyte of flash it's a
little bit more than that this is going to be a terabyte of flash head like 1.1 sorry my brain broke yeah
that's not possible yeah petabytes 12.8 petabyte of flash that's crazy no
one would be crazy enough final shout out to everyone who helps
keoxia NVIDIA slash melanox whatever you decide you want to be AMD micron super
micro fs.com infinite infinite
cables whoever and Jake tyvee Jake tybee
who's been cool and all of our contacts taking all this home you guys have been
so patient this has been it's going to be a trade show at my house if you guys
are are enjoying this series maybe go check out the original petabyte project
series i actually don't think the panel in my house has enough capacity to run
this i don't think mine does either