when you make as many videos as we do you need a lot of fast reliable storage
and our main editing server wanic has
checked all of those boxes for years it's a great little server it's built
out of high quality components and it even looks cool but as our team is grown
we've reached the point where even a minute one single minute of downtime
costs over $50 and that's just in
payroll now practically speaking the way to mitigate that is by adding redundant
now our drives are already redundant we've got 20 drives in there with data
striping but the problem is they all sit in one single server I'm sure you can
see where this is going it's been over a year in the making but it's finally here
wanic final form and I'm calling it wanic 10 because it's the last W ever
avability W you this like 10 times
nobody even knows what high availability means it means it's lus just go ahead
unplug one do it go for it well okay I should probably tell you the stakes
before you do that each of these two grand twin boxes has four entire servers
inside of them that were provided by super micro who sponsored this whole thing and they're set up with WCA a
redundant NVMe first file system in this
config it should sustain two entire servers dropping out without anyone even
noticing except that we moved the entire team onto it last night without telling
anyone and it's the middle of the work day with a ton of high priority videos
in progress do you really want to test it right now I like I haven't tried that
all right here we go okay what could go
wrong I mean a
lot naturally a huge part of a project like this is the software the stuff
that's going to handle Distributing all of ourish terabytes of video projects
Word documents and Linux isos to the
multiple machines that we just showed you but we can't install any software
until we have some Hardware so why don't we start there meet the super micro
Grand twin A+ server as- 2115 gt-
hntr despite its sort of ordinary looking appearance and unexciting
sounding name it is anything but ordinary and it is very
exciting because inside this 2u is four
independent Compu computers but for what we're doing four nodes please we want
eight inside each of these is a completely independent motherboard 384
gigs of memory an AMD epic Genoa processor with 64 cores dual m.2 slots
for redundant boot drives six PCIe Gen 5
2 and 1/2 in NVMe slots up front and
we've got IO in the rear now this bit
here could be a little confusing at first glance but that is because not
only do we have USB but we have two full
gen 5x6 PCIe connections back here along
with display output and power for the entire server this whole thing slides
into the chassis which holds a really cool modular backplane assembly that
we'll take a look at in a minute and then passes through thank you Jake ah to
the back at the server where you've got a Management Port a single USB port for
each server nope it's two and they're shared what the I was about to ask cuz
we've also got a single VGA you see the button for two servers there no way this
button toggles yeah and okay before we talk about that
a little bit more look at these power
supplies each of these is
2200 Watts 80 plus typ tianium which
sounds like a lot but when you're potentially handling four 400 wat epic
Genoa CPUs along with a bunch of RAM up to 24 NVMe drives and eight network
cards well it seems downright reasonable doesn't it is it 24 drives can't be 6
yes 6 * 4 is 24 and of course that's just one of them
we've got two of those and that means that in the event that one of these dies
the system should be able to continue to operate uninterrupted which is a big
part of the high availability goal that we have for this deployment speaking of
high availability let's move on to our network cards each of those PCIe gen 5x6
slots I showed you guys before terminates in one of these ocp 3.0 small
form factor mezzanine slots and what we're putting in them is these connectx
6 200 gbit cards from melanox excuse me from NVIDIA that okay
these are the older Gen 4 ones so they're going to be limited by the slot
speed of around 250 gabit per second but
if we had newer cards that means that each of these nodes could do 200 plus
another 200 400 up to 800 gigabit which would of course be a
complete waste for us a because our workload can't take advantage of it and
B because our switch is only 100 gbit
sorry of course the two ports are still helpful we do have redundant
switches except there's kind of a problem here that's still a single point
of failure in a perfect world we would have two single port Nicks so if a Nick
were to die it would still be okay but because we have so many nodes we're not
really worried about an individual node you know they could have one boot drive
and it die or one Nick and it die we still have an extra backup how many
nines do you want I mean I don't know like one would would be good 9% which
Jokes Aside is a really good point if we were architecting this properly there
are so many more considerations that we would need to make like the power coming
into the rack would have to come from two independent backed up sources the
connectivity to our clients would have to be redundant as well the connectivity
between all of the systems would have to be architected in such a way that no
matter what fails everything will stay up and realistically for us we're not
going to get that deep into it because our goal is better than we had before
which was a single machine with its own built-in redundancies but other than
that nothing now at least we should be able to lose a full machine out of these
eight we can restart one of our core switches totally fine two machines out
of these eight and we can still be
limping along I mean limping is a bit of a stretch it's going to be very fast now
normally if you buy a super micro machine they're going to pre-build it for you they're going to validate it for
you you can even have them pre-build an entire Rack or racks of these things and
then validate your application on it before it ships to you in fact we've got
a whole video that we did about that that was sponsored by super micro a little while back of course this is LT
my friends so we will be assembling this
one ourselves do you like that spin of the screwdriver above the server don't
worry I won't miss I'll never miss see I could do this a hundred times and I
would never miss why no it's fine it's good it's okay we have seven more any
who for our CPU we've gone with an epic Genova
9534 this is a 64 core
128 thread monster of a CPU it'll do 3.7
GHz Max boost it has A4 Gigabyte of
level three cache a 300 wat TDP it supports ddr5 memory up to 12 channels
and it supports a whopping 128 Lanes of
PCIe Gen 5 originally we were intending
to go with 32 core chips but they were out of stock so free upgrade lucky us
compared to previous generation AMD epic CPUs dooa is a big step up in terms of
IO performance which makes it perfect for this application and in the long
term I mean if we've got all the extra CPU cores and a whole bunch of RAM
anyway why run WCA on the bare metal when we could install prox Mox and then
use the other cores for I don't know High
availability Plex server yeah Linux isos
more realistically it would be something like active directory yeah which we
don't really want to do right now because if you run active directory on
one server and it goes down you're going to have a really really bad time but if
you run it on a bunch of servers yeah it's good great so normally server CPU
coolers would come with their own thermal paste pre-applied but since
we're doing this ourselves and uh if you look carefully it's not the first time
that it's been installed we are going to be using okay thank you for that a piece
of Honeywell PTM 7950 this stuff is
freaking awesome it has great thermal transfer properties and it can handle
varying temperatures like seriously I don't remember many not even just
varying but like a lot of huge cycles for a very very long time now available
LTD store.com is that big enough does that cover all of the ccds and
cxs oh there's a second piece of PL am I stupid is there a second piece of
plastic no there isn't should I put one in the fridge no no no it's totally fine
I've done this like a bunch of times yeah oh she's Min look at that see all
right easy I would recommend putting it in the fridge before you use it all
right to ensure we're making the absolute most of our CPU especially in
this High throughput storage workload we're going to be populating all 12 of
our memory Channels with 32 gig dims of ddr5 ECC running at 4800 megga
transitors per second that's a total
of 384 three terabytes of memory what
across all eight oh each of the cables Jake removing
right now is a PCIe by8 cable that feeds
two of the drive bays in the front but the reason he's taking them out is that
we can install our boot drives these are
consumer grade each system is getting two Sab 512 gig gen 3 rocket drives and
it's not because they're particularly special in any meaningful way they're
not even that fast by modern standards but what they are is from our experience
reliable enough and they are fast enough for what we're going to be doing which
is just booting our operating system off of them movie Magic all of the other
nodes are already built so what do you mean movie Magic super micro built them
Oh I thought you buil them super micro builds them for you I took it apart okay
fine I took that one apart no secrets left anymore yep no Intrigue no mystery
you know what is still mysterious is inside of here I've actually never opened this before Oh okay let's have a
look woo holy oh that's power supplies
yeah this is so cool so the whole computer is cooled by four fans no way
there's the two power supply fans and then these fans in their what do they call this like IO module I think is what
they call it look at the blades on this thing counter rotating you're serious
that's what you're looking at not this the most delicate of spaghet oh my God
there's not even connectors every one of these wires is soldered directly to the
back of the ocp 3.0 what yeah for
storage we're installing ing two of kok's Speedy cd6 Gen 4 and vme drives in
each node so we've got one that is 7
tabt and another one that is 15 terabytes they're kind of placeholders
for now and in the long term we're going to switch to Something in the neighborhood of about 4 15 tab drives
per node but the drives we want to use are currently occupied by oh that
project by a top secret pastry related
project so that's going to have to wait the good news is that when those drives
become available WCA supports live upgrading and downgrading so we can just
pull these drives swap in the new ones pull swap pull swap pull swap as long as
we uh don't do it all at once are we ready to fire these things up okay
there's a lot going on here what is that is that a switch y hey look you can see
the button now oh that's cool what you're hearing so far is just
the NVIDIA SN 3700 32 Port 200 gig
switch oh my God it even says melanox on the front I know maybe it's an old like
review sample demo univ we got it with the $1 million PC and I'm pretty sure
that that was already NVIDIA at that point can you hear that you hear it getting louder yeah
who well that one's just excited to see this is the WKA dashboard maybe if I go
over here cluster servers we can see all of our servers we have two drives per
and then course this is a very interesting part of how wo works it's
not like trass let's say where it just uses the whole CPU for whatever you're
trying to do they dedicate and like fence off specific cores for specific
tasks for instance each Drive gets a core so we've got two Drive containers
that means two a full core per Drive
yeah damn yeah you also have compute
cores which do like the par calculation and intercluster communication and then
there's front end which you don't necessarily always have frontend cores
managed connecting to a file system so if you just had drives and Compu compute
you wouldn't be able to access the files on this machine so you would have your
backend servers right those would run drives and compute which is the cluster
and then on your like GPU box you would run just the front end and that would
allow the GPU box to connect to the backend cluster servers oh the back-end
cluster servers don't need to run a front end unless you want to be able to
access the files on that machine or from
that machine which we want to cuz we're using SMB we're using it as a a file
server stupid NZ for our stupid Windows machines yeah you can also have a
dedicated front end machine yes so if you had like a 100 backend servers but
then that's adding a single point of failure which is what we're trying to avoid you could have multiple of them
okay you thought they thought of that yeah I set it up so every single machine
in the cluster all eight of them are part of our SMB cluster which means it
cannot go down realistically there are a ton of other file systems out there that
you could use for something like this traz has their scale out setup for
clustered ZFS which only requires three
nodes and is something we'd be quite interested in trying out or if you're
looking for object storage there's a million options but the main open-
source one Min iio requires only four nodes though when we saw how nuts WCA
was when we set up the million dooll server cluster I mean we had to try it
out for ourselves and try it out we did
so this is each not no holy
sh look okay the crazy thing is look at the read latency now guys look look hold
on hold on hold on at 70 gabt a second
we've seen numbers like this before but we're talking with in some cases double
the number of drives and no file system without a file system like raw to each
drive this is with a file system with a file system over a network and we're
only using 100 Gig ports like usually
with a WCA setup like this you'd probably use 200 yeah cuz we oh my God
we didn't know cuz we didn't even have networking as a factor last time all the
drives were in one box I know this is networking too and the crazy part is
we're not using RDMA this is like um some fancy uh what's it called dpdk I
think is the library this is wild yeah
look at that so read latency 131 microc
seconds that's 4 million read iops with
a latency of 1 millisecond average are are we able to keep using W FS like this
is a trial okay this software is quite expensive this is unreal 4 million iops
this is like it is unreal it's way more than we could possibly ever need but
it's cool it's so cool don't they support tearing and everything oh yeah
here I'll show you actually what that looks like this is on mother vault which I think right now has 400 Tippy bytes
left so let's say Max Capacity is 400 terab now once we run out of the 100
terab of SSD capacity which you can see here it'll just it'll tear I mean it
automatically tear anyways and you do need to make sure that your object store
is at least the same size as the flash or bigger because they're going to
automatically tear everything to it that makes sense so in theory we
move manually copy everything from Vault
one time to wo one time because it stores in like 64 megabyte chunks and
then it just stays there forever stays there forever and then we just have one
network share and when something needs to get vaed you just you just move it
from allow it to Decay yeah you would probably move it from pending projects
to like done or something like that we make a folder for done yeah sure um and
then it will just do it automatically wow or if it's a video that like
somebody was working on and then you know it's been on hold for 3 months and
we shot you know a ter of footage it will just and then when we're ready to
work on it it'll promote it back up holy we K net boot off of this
followup video yeah I mean why not it's so fast you literally could not we we
couldn't saturate this now a lot of you at this point must be thinking gosh
Mister that's an awful lot of computers for high availability couldn't you do
this with two and you're not that far
off the old school high availability net
app storage appliances like that one we looked at recently did have just two
machines but those were both connected
to the same storage drives if each
system has its own drives when things can get out of sync like let's say if
one machine has downtime you can run into a situation where each system
believes with all the conviction in its heart that it has the correct data and
then if all you have is two how will
they decide who's right this is typically referred to as split brain and
that's why the majority of High availability systems have at bare
minimum three servers this allows the
third system to be a tie breaker of sorts in the case of a disagreement now
in our case WCA that stupid Ultra fast
file system that we're using which unlike anything that we've used before
has been built specifically for NVMe drives not hard drives well it requires
a minimum of six nodes with a
recommendation of eight but running WKA can still be an advantage video editing
with Adobe Premiere like we use is very latency sensitive and even a small delay
when going to access a clip can be enough to make the software crash so any
Improvement there is huge not to mention
that a pair of these Grand twins speced out to the max with 128 car epic Berg
CPUs would get you just four rack units
with 1,000 CPU cores actually actually a
little more 24 terab of ddr5 and up to 3
pedabytes of ndme storage I mean h that
makes our setup seem downright reasonable now the average W customers
are going to be a little more demanding than us visual effect Studios AI
developers genomics Labs all the folks
out there that need Stupid Fast low latency storage and WCA showed us
screenshots of clusters that were reading in excess of 1 terte per second
consistently obviously that was a bigger cluster but it shows you what can be
achieved with this kind of Hardware running on I mean what used to be the
crappier option software raid man I feel
bad even calling it that these days I had a interesting idea with the super
micro folks so you know how we have like two pedabytes of 13 years worth of
footage thousands and thousands of hours of footage thousands it's really cool
that we have it but it's really hard to use unless you just happen to know what
video the thing you were looking for is in well what if you could just like
search for something lonus Sebastian I
want every clip with lonus Sebastian in it wow bam look at that shot up and
let's say you know there's this one that's uh detected that it's you
throughout the entire clip yeah you're in a chair so you could search for clips
of lonus sitting down with a keyboard
yeah like we're going to be able to actually find stuff yeah right now there
is a a finite amount of objects that are trained I mean chihuahua let me scroll
through this it's a lot eventually you'll be able to train it and tell it
hey this is what a computer fan looks like or this is what an SSD looks like
oh my God that is so cool so wait is this running on these extra CPU cores or
okay no not right now faces and logos are running on CPU yeah objects OCR and
scenes run on GPU got it but they're not running on any of those machines they're
running on a GPU workstation that super micro sent that's sitting at my desk um
it was Heavy anyways what is happening on that new server is proxies because if
we were to analyze the original Clips oh AAL formatting is a huge problem when
you go into an AI model it might not necessarily support the Kodak that
you're filming in sure but also Clips are like hundreds of megabytes a second
potentially that would take forever so instead it generates proxies of
everything first first which we're dumping to that new server and then we
can take advantage of the Lightning Fast storage yeah you can we have 2.6 massive
compute and we can basically create like a proxy map of what everything is in the
main archive right that is so cool so far I've generated 2.6 terab of proxies
which might not sound like a lot but they're only 5 megabit so it's actually
like a lot this is going to be a flipping game Cher News
sports can you imagine your CNN you want
that person wearing a red tie yeah but right now we've done 25,000 so 2.6
terabyt is 25,000 Pro okay well let's try and find something oh hold on once
you've generated a proxy you have to then analyze it right ah so the analysis
is not done no not even close I've analyzed 22 Clips okay everything with
Elijah Elijah and this is the every clip that
Elijah's in and you can even see this is so cool this is the actual ma'am as they
call it media asset manager the axle AI
guys built this before it was like AI as far as I'm aware back when you would
have had to make comments like this manually now it's just AI so all of the
data is in here now and we can see here's Adam and Elijah oh that's so cool
here's all the different objects chair flower pot microphone oh let me show you
the scene understanding thing cuz that is so cool this is like brand new thing
they barely even worked it in but it basic it basically takes a snapshot
every seconds two men are working on a project in a room there is a speaker
stereo equipment there's a faucet there's a tripod there's the tripod some
of these are a little less accurate two men are working on a robot in a room it
kind of looks like a robot you I mean yeah sure two men are in a workshop
looking at a laptop computer looking at a machine there is person Alex Clark so
this is just running right now in real time like more stuff is getting processed as see here processing logos 9
there it is processing logos and faces it's going to take a while yeah it's
going to take forever they're still working on making it function on
multiple gpus so once we can get it
running on like four gpus say one GPU is doing face detection one's doing scene
analysis one's doing object detection or something like that we'll be able to go
a lot faster but right now it's just one GPU got it but this is so cool all
that's left is to deploy it lonus had to run away to do some other stuff so I've
hired some backup Cavalry Sean our infrastructure administrator except
we've run into a bit of a problem lonus and me and our Infinite Wisdom while we
were making this rack so much better ran a bunch of cables right where we need to
put the server did we just start unplugging no yeah how are we even going
to do this we have to like part the seas exactly I started to try to move some of
the cables out of the way but they're all twisted together so hopefully the
LTT Cable Management thing which you can finally get at ltp store.com will save
us beautiful cable managed we can slide a server in there now I hope you're in
yeah it's on ow ow ow ow ow ow ow okay
you're good just go that wasn't so bad
like made for it
next hey we're in now we just have to
run a million cables uhoh do you notice anything different well it's loud most
of that's actually just the vent is on one of the air conditioners is broken
again but do you notice anything different I mean the sticker's here that
that sticker's been there for years seriously you haven't noticed anything else well you guys uh screwed something
onto the oh did you put sauna Pan behind it yeah but I thought this is supposed
to be a vented door my original plan was
to get rid of the vent that you put in but that vent was there as a backup in
case the HVAC ever failed so that fan is
the exhaust and that's the intake you see all the gaps F God there gaps but do
you notice the sound difference yeah it's a big difference it's huge but that
server is so loud we basically ended up where we
started yeah but that's okay I was just trying to normalize I just mean I didn't
make it worse it's not that okay look at
that woo cute right God that's a lot of
metal if all goes to plan we could get rid of this and this and just have these
so no more additional rack taken up which is nice
wow it should sustain two entire servers
dropping out without anyone even noticing do you really want to test it
right now I like I haven't tried that all right here we go what could go
wrong u i mean a lot the fact that all
the fans just like turned down a bit is a little scary let's go see if anyone
noticed oh hi Mark hi I'm holding your file server how's your edit going uh
what huh is it working it's working is this on Wi-Fi hey Emily hey how's your
edit going I'm holding your server that's cool is it working are you sure yeah Hoffman
what's up how's your edit going this is your server right here it's amazing look
feel it it's still warm wow yeah it's it's still warm how well how's it
working it's great you know I'm editing the video that we're shooting you are yeah uh we're going to pull another one
wait no l you forgot one yeah here here's another here's another one of
your servers is it working it's great though huhuh for reference you're not
supposed to do this you should power off the system first but we're just trying
to simulate it failing yeah a terrible
catastrophic failure I can't believe how smoothly it handled that see all the
lights they never stopped blinking big thanks to Super Micro for these awesome
servers thanks to WCA for making this crazy software thanks to axle for the
awesome AI detection if you like this video maybe check out the video series
of us building our nearly three pedabytes of archival storage which we
call the mother Vault that thing is awesome and we showed it to you and it's
faster now oh and thanks to you for being an awesome viewer