165,000 CPU cores, 20 million of GPU,
and a cool pabyte of RAM. I wouldn't
normally describe myself as a fur bree, but the new fur supercomput has
definitely awakened some feelings that I can't say I've ever felt before.
Feelings like wanting to go deep inside it, to gently remove its panels, and
maybe some light screwing. And thanks to
our friends here at Simon Fraser University in beautiful British
Columbia, we are going to be doing just that. Going deep under the hood of the
CPU and GPU compute deployment that is
going to be serving tens of thousands of scientists and researchers in fields all
the way from AI to zoology all over the country for years to come. This will be
our first upclose look at a realworld deployment that uses direct die liquid
cooling to increase cooling efficiency from about 30% to over 90%. Or at least
it'll be the first data center grade deployment. Mine doesn't count and it
doesn't look nearly as sexy. But what is sexy is this segue to our sponsor MSI.
MSI's MAG B850 Tomahawk Max Wi-Fi motherboard has bells, whistles, and a
bunch of other things that make noise. With support for the latest AMD CPUs,
DDR5, and a slew of unique features, it's a performance monster. Check it out
today using our link down below.
In the row behind me is 640 NVIDIA H180
GB GPUs, each with an estimated cost of
around $31,000.
Even at less than half of the maximum density, just 20 nodes per rack, the
team here had to reroute power from elsewhere in the building and
significantly upgrade the building's cooling system just to accommodate the
incredible power requirements of these NVIDIA hoppers. This is actually a
common theme that I hear from basically anyone in the data center space. I mean,
we tried to build for the future, but we couldn't have possibly seen this coming.
And there's no sign of things slowed down. We'll get to that later, though.
First, the most exciting part of the tour. They pulled one of their spares
out of the rack for us to crack open and get up close and personal with. And oh
my god, look at this thing.
It's heavy. I guess when you got this much copper in you, like, wow.
It's kind of scary handling it. I mean, this one you node alone is worth more
than my car. And a rack of these is worth more than my house.
It's a little sketchy, but I want you guys to be able to see it. The CPUs are
epic genoa. So, last generation Zen
4-based, but Genova still supports up to
12 channel DDR5 memory and 128 lanes of
PCIe Gen 5, which is plenty to keep these GPU cores fed. if more CPU compute
is needed. Clearly, there is support for dual CPU sockets, but the team at SFU
found that eight CPU cores per GPU was
plenty for their purposes, and they opted for a single 48 core CPU and 1.152
TB of RAM in each of their nodes. Now,
for a closer look at the GPUs. Unfortunately, I'm not allowed to take
the coolers off them. But under each of these four cold plates is an NVIDIA H100
SXM5 80 gig GPU, giving us a total of
320 GB of VRAM per node. And guys,
that's not just any VRAM. That is HBM3 running on a 5,120 bit bus for a total
bandwidth per GPU of 3.36 terabytes per
second. For context, a top-of-the-line consumer card, the RTX 5090, achieves
just over half of that bandwidth. This
kind of power does come with drawbacks, however, like for example, heat. Each of
these is rated for 700 W of power
consumption through the SXM socket that's underneath them. And that is
where the incredible cooling solution in this Lenovo Node comes in. As a liquid
cooling nerd, I got to say, guys, this is the coolest part for me. I mean, did
you notice that there isn't a single fan in sight anywhere in this machine? That
is because everything, CPUs, GPUs, VRM,
network interface, SSD caddy, even the
system memory is directly liquid cooled.
All of it. This feels a little bit like doing the maze and highlights magazine.
So, here's our inlet over here, which splits into two main loops that go
through the system. The primary loop, which we can tell because it has a
thicker pipe coming off of it, goes straight to the middle of our four GPUs,
where this manifold splits fresh incoming water out to our four GPUs. Two
of them just spit right back into the outlet here, while the other two run up
to this networking board and then consolidate back to the outlet. That's
our primary loop. Our secondary loop comes through here handling some of the
power delivery and then carries over to
interesting. It splits out doing the RAM
next. I am not 100% sure what to make of that
because I would think RAM would be a tertiary priority in terms of cooling,
but that's what they've done. We go through the RAM splitting into three
different tubes that sit between our DIMs down both rows. Then one side
handles this network caddy here and the other side handles our SSD caddy. Then
each of those come back to one of the CPUs which come out into the middle here
and then run back to the outlet here. Not maybe the way I would have laid it
out. There's a lot of 90° turns in here, meaning a lot of restriction. But I'm
sure the engineers at Lenovo know what they're doing. There's a ton of other
cool stuff to unpack here, too. You probably noticed there's no power
supply. That's because it uses these chunky connectors here at the back to
plug into a back plane in the back of the rack. As for the coolant
connections, well, according to the manufacturer, these do have a little bit
of natural leakage, but uh it's on the order of molecules, which is pretty damn
impressive. There are sensors all over
the motherboard to detect any kind of leakage. Now, the team here wasn't sure
about the exact chemistry of the coolant they're using, but they did tell me that
it has antimicrobial properties to prevent anything from growing in the
loop. There's some other fun stuff. There's a little stylus in here.
Apparently, this is meant to uh assist in removing memory, which is great. I'd
actually love to see more gaming motherboards come with that.
I thought this lone 7.68 TB NVMe drive
was interesting, too. I mean, the networking is 400 GB per second* 2 to
the 2 pabytes of NVMe storage, not to
mention 49 pabytes of spinning Rust that's right over there. But according
to the team here, occasionally they need node local storage to improve GPU
performance a little bit. So, you'd never boot off of this or anything, but
it's nice to have there as a scratch. Also, the button cell in here is mounted
in a vertical caddy because the density is so high in this one you node that
they just couldn't give up the space that it would have taken to mount it
parallel to the board. I also spotted a
micro SD header. If anyone out there works in the data
center and knows what that's for, I haven't seen it before. And Jim and I
just assumed that I typed. Oh, there was one other thing we wanted
to look at. These big power bad boys. We
couldn't see them until we got that shroud off. So these they're just bus
bars. They're going from power supply here, which is a DC toDC power supply,
>> 48 volt it looks like, and they're going over to our GPUs. Damn. What's
interesting to me that I just noticed is that there's a clear delineation between
the NVIDIA engineered parts of this with the black PCB and they're completely
separate from everything else and the Lenovo engineered parts of this. So
Lenovo is acting like more of a system integrator around this compute block
here. Like you can even see the silk screening on the PCB is distinctly
NVIDIA and Lenovo's just doing their DC toDC power. So, it's just power in here
and then PCIe in here in the form of
these four MCIO connectors right here. This is essentially like plugging a GPU
into your Legion gaming PC.
>> The the GPU house >> with extra steps. >> Yeah.
>> Before we poke around in one of the 192
core CPU nodes that they've got, let's take a look at one of the racks that
these boys slide into.
They're still using a very similar rear door chilled liquid rack like we saw
with their air cooled modes when we did a tour oh lordy 8 years ago. Um
anywh who the point is that chilled 16
1/2° cooling comes from the evaporative cooling towers outside. Then hot air
from the power supplies and any of the networking equipment that's in the rack
runs through here and wow is that ever hot. Then it spits out nice comfortable
room temperature air on the other side. Each of these racks is fed by dual
three-phase 60 amp feeds for a total of about 70,000 watts per rack. Now, if SFU
had the power and cooling in this 1960s
bunker, they could juice these up to 180,000 watts per rack. But they don't.
Hence the empty rack space. Since we have this open, oh wow, that is a big
difference between the cold side and the hot side going into the the back of
these back planes for the servers. I don't have to ask which one's the
supply. That's the cold side, which
since we're on the subject, this is a perfect time to look at the cooling
distribution system. This is the Liber XPU from Vertip. It can do 600,000
watts of cooling capacity per one of these cooling distribution units or
CDUs. Water comes in the supply side
here. This thick boy, she's chilly.
That's coming from the cooling towers outside. Then that runs all the way down
to the bottom here to the heat exchanger in the front. This liquid toliquid heat
exchanger does exactly what it says on the tin. Taking that cold water from the
primary loop that goes outside and using it to chill the warm water that is
coming directly off of the blocks that are going to our nose. This unit uses
dual redundant pumps. And if we go back
around to the back, uses these manifolds and valves to control flow to up to six
different racks. And it's very easy to tell which is the cold side that's being
chilled in, which is the hot side here. Wow.
I want one. Von, can I have one? Probably the coolest part is this little
touchcreen display on the front that much more succinctly illustrates what I
just said. Here's your primary loop. Here's your secondary loop. Here's all
your flow rates, all your temperatures, and here's an alarm that they assure me
is totally fine.
This data is super important because if they accidentally had water that was too
cool going into the servers behind me, then they could end up with
condensation, which hopefully I don't have to explain why that's super super
bad. Everything's hooked up using AquaM tubing from Germany. The admin here
spoke with some other facilities that used stainless steel and one of them got
rust in their cooling system. is a big big mess. They've been really really
happy with their offerings. Now, let's go check out a CPU mode. Contrary to
what NVIDIA would like everyone to believe, not everything runs best on a
GPU, even today. And that's where these come in. Each of these one U racks
contains two nodes. And each node
contains 192 Zen 5 epic Turing cores
with 768 gigs of memory. So that's a
total of nearly 400 cores in each of
these one use. Holy freaking
for networking. They actually don't go as heavy on these using 200 gig
connections and NDR to dynamically share
that 200 Gbit link between the two nodes depending on their needs. This approach
does have the drawback of meaning that if the primary node goes down, we lose
network connection to the secondary one. But I have to assume that the cost
savings outweigh the disadvantages in this case. In terms of loop layout, this
one is much simpler coming in to both sides and then out of both sides. But
just like the GPU nodes, the goal here is to get a water tube up against pretty
much anything in the server that generates heat because there are no fans
whatsoever. One cool thing we missed on the GPU node
was we never got to look under the little cooling plates that the SSDs and
network cards sit on. So, here's what it looks like. It pretty much looks like a
heat pipe except instead of being full of what is usually a vapor and sometimes
a liquid that circulates just within itself, it's just full of water or other
coolant that is circulating to an external system. Now, let's take a look
at the racks that these live in. Each of these racks contains 72 of the nodes
that I just showed you guys, top to freaking bottom, with roughly 13,824
cores. Each three racks is an island
with a non-blocking 800 gig connection between islands. So, 41,000 cores can
represent a single job with no blocking. They have some other specialized nodes
like the storage ones, including the ones on the other side of the aisle that
hold data for our local particle collider. Try them. We've got a whole
video about that. Along with some 8 TB RAM nodes, which are, I think, pretty
self-explanatory. They're for jobs that would overflow on a regular node as long
as you don't mind them having a few bugs. And finally, a single AMD MI300X
node to I don't know what, keeping video on the
toes or We could do it. We could buy more than
one of these. You better not charge too much, especially when you factor in
modern security needs. There are six zones of security to get to some of the
cages that actually have biometric locks on them where not only do you need to
know the pin code, but you have to put your hand under it and it will check if
that hand is attached to a living person. There's cameras everywhere in
the data center with full visibility in all directions. And our tour guide today
actually said that there's someone monitoring them so often that it's
become a bit of a game where they'll send non-flattering pictures of him
moving around in the data center just to make sure he knows they're watching.
Cooling everything are these evaporative cooling towers behind me. The three that
are closest to the building, they were there the last time we were here, but I
couldn't show them to you for reasons that involve red tape and approvals. So,
here they are. I still can't get any closer to them for reasons that involve
red tape and approvals, but hey, we can check out the acoustic damping that's on
these ones on the other side. That's impressive. Here I am right at the
intake next to these sound baffles. And for context, here's the untreated ones.
The total cooling capacity is about 4.7 megawatt, which that is way more than
what's needed for the machines inside. But just like power and storage, you
want to have some extra for resiliency in the event of an equipment failure.
Hey, what's one of these worth? Maybe I'll pick one up for the office.
>> $1.2 million each. >> Oh,
never mind. Frankly, I'd rather have one of these anyway. To augment the original
pumps for Cedar, which could do 800 gall
per minute of coolant, they added these two new ones that do 1,500
gallons per minute. They also now have
two mechanical chillers which can be useful during the times of year that we
get outside temperatures above 33 C. So
for maybe 6 hours a day, they'll switch over to mechanical chilling to help out
their evaporative cooling tower. Probably the coolest thing about this gear though is how smart it is. They've
got telemetry capture for things like temperature and flow rates, and it all
feeds into a third party called Kaizen that helps with logging and determining
if something's gone wrong with the system. Fun fact, by the way, the twoft
thick concrete floor that I'm standing on is so burdened by all of this heavy
equipment and coolant that it actually deflects half an inch in the center. Is
that Is that okay? Am I going to break
it? I mean, the place used to be the power distribution center for the
southern half of our province, and it's built like a bunker, but that's not
enough. Who paid for it all? Fur had a total budget of about $82 million which
oh I assume is Canadian rubbledos. Uh so
a little under $60 million US and that came from a combination of the digital
research alliance of Canada, BCKDF and
vendor inind contributions which I just learned are a vendor giving significant
discounts. How how do I get signed up for that program? Is that only for
educational institutions? Anyway, they did ask us to shout out a
couple of companies who helped them out. Lenovo, DDN, and Vertive on the cooling
side. Um, and they didn't ask us to shout these guys out, but we're going to
do it anyway. A shout out for our sponsor, Vessie. We're partnering up
with them to bring you an exclusive giveaway extravaganza all month long. By
entering, you'll get a chance to win one of seven prize packs, each loaded with
Vessie gear, LT Storm merch, and more. We're talking footwear like their
Stormbburst hightops or bags like our very own commuter backpack, Sony XM6
headphones, and some prizes so special we can't even tell you about them yet.
It's 100% free to enter. But if you're looking for some stylish shoes to
protect yourself from the rain right now, well, we've got a special discount
for you that's valid until the end of the month, 20% off. So, head over to
vessie.com/LMG to enter the giveaway today and get an
instant 20% off code sent directly to your inbox. Act fast, though. Everything
closes on October 31st. Once again, that's vessie.com/LMG.
Take part in an exclusive giveaway and secure your 20% off code.
>> If you guys enjoyed this video, why not check out the tour we did of Triumph,
the particle accelerator that is just down the road. Well, down the hill, down
the really long road. Canada's only road. It's pretty long.