this is an SSD
no no seriously this whole thing is a complete SSD and it's designed for
massive data centers like what you might find at the likes of meta so
how come i've never heard of it well pure storage wonders that too and
they've sponsored this video for us to take a tour of their facilities here in
sunny mountain view california and dig deeper into just how these things work
and maybe while they're at it they can explain a little bit about how
they can make the slowest flash used in ssds today run
twice as fast as last year's good stuff
let's take a look
from the beginning pure storage's secret sauce has been in their approach to
software while their solution is proprietary with it they're able to
tightly control each individual nand flash chip in an array potentially the
size of a whole rack but
there's no way that i know of to drill down to that level of detail with any
normal SSD like even enterprise grade sas and u.2 drives have controllers that
make the drive show up as disks like you don't get direct access to the chips so
what's up with that this
is a direct flash module or dfm for
short it's a strange looking module it's a lot larger than any SSD form factor
i've seen before and there's a lot of nand flash on there
up to 48 terabytes in the current generation along with super capacitors
to make sure writes complete properly in the case of a sudden power loss
pretty standard for data center SSD yes but while you'd expect loads of dram
cash to make it fast there's no dram on here that is a
terrible thing for performance under normal circumstances just because of the
way ssds work whenever an individual die is busy performing some action the data
on it is completely inaccessible and you have to wait dram caches collect the
data and delay writing it until there's enough of it that it actually makes
sense to spend the time actually writing it out because in order for an SSD to
write that data any blocks that are totally empty first had to be erased
as you add additional levels of storage to a cell so from single to multi to
triple to quad the time it takes to actually do this
increases exponentially and if you remember from our review of Intel's
early dreamless qlc drives you would know that it can be slower than a hard
drive this could be catastrophic for
real-time applications if your d-ramless array decided that it was time for some
spring cleaning so have pure storage just gone
completely mad if so well
there may be a method to their madness up to four of these dfms can slot into a
blade like this one which itself
is a self-contained server running a socketed xeon processor
there's the dram yes this xeon CPU is
acting as a glorified SSD controller now it does other array management stuff too
but to talk to the storage it actually runs a very low level interface
bootstrap from Linux that talks to a custom controller on each dfm
these controllers do one very simple task
provide an interface to all the flash on the module
it doesn't really do anything that you'd expect from a typical SSD controller all
of that is handled by the CPU while it does use the NVMe protocol and
a u.2 connection it won't show up as a disk in any other system simply because
it doesn't have the brain this arrangement lets them directly
control when where and how new data gets stored like
it's kind of like apple silicon in a way except they can handle multiple vendors
of flash in the same array more importantly there's no pretending
to be a hard drive so while block sizes can vary between vendors you're not
forced to write out a whole four kilobytes each time you need to write a
couple of bytes so because the blade knows what the whole array looks like
data that's changed very frequently can be grouped together to prevent a
situation known as tombstoning where deleted but not erased data can stick
around because it's uneconomical to evict it which means that that cell is
effectively never touched again that's partly why over provision space which is
where a percentage of the raw capacity is reserved for
system use is important for SSD performance a workaround that pure
storage doesn't really need in a nutshell each
of these blades is basically an SSD with extra steps
because of that pure storage is able to control things like
when and where to start wear leveling which is incredibly important to qlc
flash because it's typically only rated for less than a thousand programming
race cycles so obviously there's a lot
of incentive to never have to cycle a cell if you don't have to to help with
that data is compressed wherever possible
there's another component to this too because you can only ever read or write
flash memory not both the more you can avoid writing to
frequently read cells or overwriting frequently written data to those cells
the less often you're stuck waiting for the SSD to come around and actually spit
out the data you need and that's especially important with qlc because of
how long it takes to erase and rewrite data on that type of flash remember at
this kind of scale you're talking databases that are massive enough and
accessed often enough that even a delay of tens of milliseconds could be a major
problem and of course they've thought about that
too what if i told you that in the time it takes for a right cycle to end the
inaccessible bits can be rebuilt from
parity data yeah that's not something you can do
with a traditional SSD the controllers that pretend that they're still spinning
discs for the pc's sake quickly become the bottleneck let's meet a complete
system and see it at work this is a flashblade s chassis pure
storage's newest product each of these can hold 10 blades with 4 dfm's each
which at up to 48 terabytes per dfm is a
staggering 1.9
petabytes of storage in a single 5u
chassis remember the lengths we had to go to in order to get a petabyte of hard drives
and then how many boxes a petabyte of flash took up
this is next level there are four power supplies on this
thing that are all modular each with the capability of drawing 2.4 kilowatts from
the wall and while they don't sell them right now multi-chassis configs with up
to 10 of these things in a rack are in the pipeline a fully loaded
flashblade s has enough redundancy that it can lose a
whole blade and a dfm and another blade and be
totally fine the blades connect to each other via the same backplane that the
blades slot into called fabric i o modules or fioms for short each
flashblade s has two files which automates managing the blades and
chassis units with a total of eight 100 gigabit per second links
only four of which are currently used they say that when they're ready to
enable the others they'll be able to do it for free via a software update
something they're able to do thanks to their evergreen model there's a lot to
this but basically they don't charge for new software features and their modular
approach to hardware means that you can continuously upgrade to the point where
one of their alpha stage clients from 10 years ago has continuously upgraded
their array with no downtime to this very day
it's like the array of theseus none of it is still the same except for the data
and even that might be different now but enough talk
let's see them roar in the data center
this is a rack full of the previous generation versions of what we were just
looking at a little while ago these are all connected together
with a fabric like this i think all fiber and they're all connected via
these backplanes here so this top one here is connected to all
of the pink cables and this bottom one is
connected to mostly glue there's some tape here as well
and all of these are communicating with each other in a maximum layout so this
whole rack is acting as basically a single SSD as
you can see all of these power connectors
are modular the actual power cables themselves have tabs on them so you
can't pull them out so yeah this is all of their last gen stuff this over here
is a rack full of their new gen stuff this is glass plate s
what we're looking at here is more or less a setup of a bunch of
standalone units rather than having everything built
together into one giant array each one of these is a smaller array
compared to spinning disks this is actually pretty good power
density like for the amount of storage you're
actually getting here it's a significant saving but it's something
like 1.3 watts per terabyte or something like that this is one of the first
modules that they actually built for this type of array
it actually has an fpga on it that
takes all of its information from an sd card here and that is a mini usb port
that tells you how long ago this was but this was NVMe before pretty much
anybody was using NVMe which uses a u.2
connector here to connect directly to the back plate
basically i had to create a pci express backplane for this because
actual controllers hpas didn't really
exist at the time for this kind of story and i was just handed one of these this
is a trainless version it's been taken out of
the tray of the dfm that's currently being used
we can see that there are
super capacitors on here in order to make sure that
in case uh say for example one of these were unceremoniously yanked out of the
server any data that was already being written
will be written so it won't be partially written you
won't get corruption in that way also on the back we can see here that we
have extra man flash ships this is a 48 terabyte module
4848 that is a massive amount of storage no
hard drive could possibly even come close to this within the next
i don't i don't know how long it would take because currently at the rate of
growth for hard drives it would probably be another 10 years or
so assuming hard drives are still relevant at that point in time
which is something that beer storage is trying to actually avoid
they feel that flash storage is inherently superior to spinning disks
and quite frankly from what i've seen so far it definitely kind of looks like it
you get better density
better power draw and you get
a much much lower level ability to like just
manage the data on the drive than you would get from a traditional type of
hard drive you may always be asking yourself who
could possibly need this much storage well the answer al there's rails and
stuff here so oh this this is great so the answer is
anybody who's in the deep learning so like this is an NVIDIA dgs
this is one of the things that pure storage is actually partnered with
NVIDIA 4 uh meta recently partnered with uh
your stores and i think also NVIDIA for their ai uh deep learning platform so
the djx is powering the brains whereas pure storage is powering all of
the data sets that they need to crunch through
so you're never going to get
a GPU with petabytes of
video memory at least not in the next uh i don't know several decades at least
so what we're looking at here is an array
that's fast enough to keep up with those deep learning workloads on those gpus to
keep them fed so that they can actually be doing their job more often
otherwise what you would need is a massive amount of memory for the system
to cache that kind of thing which is just impractical so here we have man
flash doing that exact job it is very
responsive and very dense
so for those reasons basically like
if you have a really deep learning workload this is pretty much the
premiere solution right now as far as i can tell both NVIDIA and pure storage need to
think so we're looking at here is an example of an NVIDIA dgx a100 in action
right now this is actually computing deep learning data and it's connected
via 100 meter per second links through these switches here
to the gear storage arrays so there's the
previous generation flashblades and i believe there's also a flashblade s
there as well which is the current generation i think they're doing performance testing on those right now
to see which is faster and how to you know how to basically optimize for these
workloads it's pretty amazing that
we're basically at a point where ssds aren't fast enough and yet
the ssds they're replacing them with are technically slower
because they're using qlc memory instead of tlc which is just mind-boggling it's
supposed to be an order of magnitude less efficient and yet they're making it
work now you might be thinking to yourself these are full computers in
these blades and server chassis that are basically just being used as SSD arrays
like what happens after the SSD array is
retired like when you no longer need it or whether you upgrade to a new one is
the whole chassis that's kind of thrown out well no what your storage is doing
is they have a whole bunch of completely empty chassis here
running virtual machines and other workloads
that don't require that kind of storage so they're repurposing those xeon
processors that would otherwise just be i don't know like if you if you retired
in SSD and put it on yourself what does that make it you know
so in this case these live on even if the flash has died
or it's been upgraded because it was too uh smalling capacity and in fact behind
you there are a whole bunch of processors and RAM and
other stuff that's just sitting there waiting to be tested or
reused now you'll be wondering how you even managed to like talk to these things
well the file system that they use i say file system very loosely it's actually
more like a database that you can actually then create
uh what they call uh authorities on top of it with 128 of them split
across the entire array the larger the array is the faster they go all of those
authorities can be used to do things like create object stores like amazon s3
and from there you can also create smb
so samba Windows file sharing
or nfs for Linux file sharing support
so pretty much it doesn't look any different to the end
user you can use it however you feel like you need to use your data so you
can pull stuff directly down through amazon for your cloud services
deployments or you can just use it as network storage if that's what you
really want and what's really cool they probably wouldn't let me do it
but if you take out one of the dfms
uh and like rearrange it or something within 10 minutes it picks right back up
without having to do anything special just you slot it back in and it's like
nothing happened so you can say for example if you've got a blade that's
misbehaving or you want to upgrade it you can completely migrate over without
having to change anything about your configuration your users basically won't
know what happened because nothing will have happened
it's it's kind of magic it's really really warm back here but
these switches we were looking at earlier with all of these chassis
plugged in this is one of them not only is this a
network switch but it's also basically the same type of
thing you find in the file in the back of a flashblade s
so it's got an x86 processor in here that handles all of the communications
and in fact when you plug in multiple flash play chassis
this thing takes over and actually orchestrates the entire lottery
so they're no longer doing their own individual arrays this thing takes over
automatically also
each one of these ports can do 40 or 100
gigabits per second depending on the five-stage capacity the older models
could always be 40 whereas the newer model flash plate s that can do 100
so it's a massive amount of data that will
flow between this thing and the rest of iraq now that you've seen
the tech let's talk about who these guys even are pure storage was founded in
stealth mode back in 2009 and debuted in 2011 as one of the first companies to
introduce all flash infrastructure solutions in the industry
in the early days they used consumer grade ssds which
if you look back on the state of ssds back then
was pretty ballsy but the solution was always software driven and they quickly
began developing their own flash modules which started shipping in 2015. fast
forward to today and they partnered with companies like cisco and NVIDIA with
clients across the globe so big thanks to pure storage for sponsoring this
video and letting us show off their gear you can learn more and maybe deploy one
of these for yourselves if you're a straight baller or if you're an i.t
manager at the links below thanks for watching guys maybe go check out one of
the Intel design center tours that Linus did a little while ago like those are
really really next level in terms of how like
behind the scenes we're seeing like Intel basically never lets anybody see
that kind of stuff and i'm glad that we got to see this kind of stuff here