behind me right now is the biggest
supercomputer in the country it'll be serving researchers across Canada
studying the human genome in bioinformatics particle physics
materials research even humani research
it's called Cedar it cost the federal government through the Canadian
foundation for Innovation over $16
million and we get to be the first to
unbox this Beast
Savage jerky is created without the use of nitrates or preservatives use offer
code LTT to save 10% at the link in the
video description so Cedar is a big data
machine it takes up a quar of the 5,000
ft Data Center it occupies meaning actually that there's room for it to
grow but right now it has
27,000 Intel Zeon processing course
190 terab of RAM 64 petabytes of storage
584 gpus and a total power draw of
560,000 wats though with that said it's
efficiency is a shocking
1.07 on the Pue scale where one would be
perfect and a typical data center would be 1 and 1/2 to two we'll get into how
they did that a little bit later though so our tour starts right here behind me
are what they call the high availability racks so everything back there has dual
power supplies for redundancy with a battery backup for that and a diesel
generator back up that everything back
here is Mission critical things like networking login servers and management
servers are all here and this is also
where you'll find the bulk of Cedar's
storage let's get in for a closer look at Cedar's connection to the outside
world this networking Appliance from
Huawei has a street price of around a
million doll W and right here this is
where it gets really bananas these guys are seedar
dual 100 gabit connections through
Vancouver and then as if that wasn't enough these orange ones here are dual
40 gbit connections through nearby sui
just in case somebody puts a back ho through one of these other fiber lines
and they would have otherwise lost their internet connectivity I mean that's
their Backup backup but Ethernet is not really the
way you want to connect high performance Computing nodes this this right here is
the true networking heart of Cedar these
are 48 Port omn path switches and
they're configured in what's called an island topology so the island is in
almost all cases 32 compute nodes each
of those compute nodes is connected to 32 ports on one of these switches in its
rack then the remaining 16 ports come
back to here that means that every
island gets a dedicated line to each of
the core switches giving you failover
and massive bandwidth each one of these fiber links
right here is capable of 100 gbit per
second so even though between islands we
are let's say bottlenecked by our 16
connections so that's only half the total theoretical speed within an island
we're still talking over 100 gabyt per
second so it's not really an issue okay now let's move on to SFU and
compute Canada's version of petabyte project spoiler alert theirs is better
in every conceivable way so in the five cabinets behind me we've got Cedar 50
petabyte IBM tape Library System they
have a 40 gabit link to the rest of the
supercomputer and each of the 5,000 10
tbte magnetic tapes inside can be grabbed out of storage moved with like a
robotic ARM into a reader and the data
can then be accessed when needed and this is done
automatically cool right okay yeah but due to the slowness
of that swapping process this is still
what we would consider to be cold or
archival storage next up here is general purpose
storage land where any data that's being
used for any current research project
would be housed so here here they're
using offthe shelf five U racks Each of
which contains let's see if we can crack One open here a total of two kind of
trays here and 84 8 terab uh let's have a look here
Enterprise capacity SAS drives from
Seagate but there's actually more to this system than meets the eye every
four of of these storage nodes requires
two nodes of what they're calling object
storage servers these act as a high-speed cache with their SAS 10,000
RPM drives as well as as kind of like a
a a traffic cough for everything behind
it so every single read or write to
these hard drives actually goes through these nodes so right now General Storage
Land is 10 ped byes but in the near to Mid future it will be expanding to 20
20 now that DIY approach to storage is
great for scaling up at a low cost but when it comes to Performance they went
for this data direct network storage Appliance because it has got the real
Goods now in the rack next to this brain
you'll find a mere 4 pedabytes of actual
storage due to its higher cost but thanks to its proprietary Hardware
custom software and solid state burst buffers this thing can handle up to 40
gabt per second of sustained throughput
making it perfect for data intensive
applications that rely on humongous data sets now let's get into
compute there are about half a dozen
different types of compute nodes all
connected to the same high-speed Omni paath Network backbone that are
optimized for different types of
research we'll begin with the base
compute note there are a whopping
576 of these each of these is a computer
so there's actually four in a single toou shell Each of which contains two
Zeon E5 2683 16 core processors 128 gigs
of RAM and about a terabyte of raid zero SSD storage for Scratch so each rack
here contains two islands so that's a
total of 64 compute nodes giving us a
whopping 2,48 compute units per rack so these
nodes are the basic Workhorse of Cedar
handling everything from Monte Carlo simulations for Material Science to sim
ating Dynamic processes in nature with a high degree of Randomness like snowfall
or rainfall they would also be used in
any highly parallelized workload because
if you need you know 10,000 CPU cores for one job there
aren't enough cores in any other class of server to handle that kind of
load moving right on up we've got the big memory nodes there are 48 of these
and half of them are just like the basic nodes except with 512 gigs of RAM while
the other half of them these puppies
have 1 and half terabytes of system
memory these ones take up twice as much
rack space though each of these one use
is a single dual socket system because
you know what there just wasn't enough go darn room for old 24 64 gig memory
modules that are required for that much
RAM first world problem
yes these guys are really special these
are the aptly named 3 tbte nodes there
are only a handful of them but these are quad socket machines with Zeon
4809 v4s four of them but wait a tick
those are only eight core processors these don't even have more processing
cores than those little tiny ones that take up half of you what's the deal here
well it turns out that some
bioinformatics workloads like genome sequencing don't actually scale very
well with more processors they just need
massive amounts of memory to hold the data sets that they need to work on so
while the team here probably isn't super stoked on using up for use just so they
can stuff more memory into the system until until optane reaches a higher
level of maturity this is the only choice they have now finally we're
getting to my favorite nodes the most expensive nodes these are the GPU nodes
and while they're actually quite similar
to the base nodes with respect to their CPU and RAM configurations what's got
the researchers in the fields of molecular Dynamics Ai and machine
learning all amped up about these are the quad NVIDIA Tesla p100 graphics
cards that they have cram into each one
I mean seriously with 1,500 watts of power being consumed by
each one of these is it it is an engineering Marvel that they've crammed
enough power and cool to make this whole thing work so
actually now that you think about it how exactly did they do that so the Keen ey
among you might have already caught a couple of hints earlier in this video
but the secret lies in the rear doors on
the server racks look how thick this is
yes my friends this entire door is a
gigantic heat exchanger so their servers
don't actually have have water blocks that would be more expensive what
they're doing is they've just got the fronts of the racks all sealed up so
there's no Backdraft pressure and they've got normal air cooled servers
that pass the air from the front where they just draw in room temperature air
in here and it comes out hot like 30 plus de and push it through this heat
exchanger where it is actually cool to my skin that's how efficient these are
and that cooling system system is massively expandable too you can
actually see above me I am standing
where we've got a blue and green cooling pipe connected to a whole bunch of quick
release fittings ready to add more racks
right here but to see what they actually do with the heat we're actually going to
have to go upstairs where we'll find the final and
perhaps the uh coolest stop in our tour
here this is the mechanical room where
the pumps and these freaking pipes take
all the water from downstairs and dump
it into three cooling towers outside the
building now right now the weather is
favorable to cooling the ambient temperature is quite low so it's just
operating as gigantic radiators but get
this when the conditions become less favorable in the summer they kick things
into high gear with an automated system
that sprays water onto the fins of the
radiators in the cooling towers and if you watched our bong cooling video which
you can check out right here you'll be familiar with this concept already but
this is called evaporative cooling and by these means even in ambient
temperatures up to 30° C
they can achieve the 17° coolant levels
that they need to without employing the
massive Chiller unit that they have over
on the other side of the
room Squarespace is the way to build a
website whether it's for your small business or for your you know uh local
freaking book club it does doesn't matter if you want a web presence
affordably and quickly Squarespace gets
it done for you it starts at just 12 bucks a month and you get a free domain
if you buy Squarespace for the year you just pick one of their templates and
boom you upload some pictures you fill in some text it's all cloud-based and
your website will simple as that look
great on any device every website comes
with a free online store and their cover pages feature allows you to set up a
beautiful one-page online presence in just minutes so start a trial with no
credit card required and start building your website today then when you decide
to sign up for Squarespace don't forget head over to squarespace.com
LTT and use offer code LTT to get 10%
off your first purchase so a massive thank you to SFU
and compute Canada for allowing us to uh run a muck in their data center thanks
to you guys for watching if you dislike this video you know what to do but if
you liked it hit that like button get subscribed maybe consider checking out
where to buy the stuff we featured at the link in the video description also
down there you'll find a link to our merch store which has cool shirts like this one and our community Forum which
you should totally join