165,000 CPU cores, 20 million dollars of GPU, and a cool tetebite of RAM.
I wouldn't normally describe myself as a furry, but the new Fur Supercomputer has definitely
awakened some feelings that I can't say I've ever felt before.
Feelings like wanting to go deep inside it to gently remove its panels and maybe some light
screwing? And thanks to our friends here at Simon Fraser University in beautiful British Columbia,
we're going to be doing just that, going deep under the hood of the CPU and GPU compute deployment
that is going to be serving tens of thousands of scientists and researchers in fields all the
way from AI to zoology all over the country for years to come.
This will be our first up close look at a real world deployment that uses direct dye liquid
cooling to increase cooling efficiency from about 30 percent to over 90 percent.
Or at least it'll be the first data center grade deployment.
Mine doesn't count, and it doesn't look nearly as sexy.
But what is sexy is this segue to our sponsor.
In the row behind me is 640 NVIDIA H180 Gigabyte GPUs each with an estimated cost of around
31,000 US dollars.
Even at less than half of the maximum density, just 20 nodes per rack, the team here had to
reroute power from elsewhere in the building and significantly upgrade the building's cooling system
just to accommodate the incredible power requirements of these NVIDIA hoppers.
This is actually a common theme that I hear from basically anyone in the data center space.
I mean we tried to build for the future, but we couldn't have possibly seen this coming,
and there's no sign of things slowing down. We'll get to that later though.
First, the most exciting part of the tour. They pulled one of their spares out of the rack for us to crack open and get up close and personal
width, and oh my god, look at this thing.
It's heavy. I guess when you got this much hopper in you like, wow, it's kind of scary handling it.
I mean this one you know to loan is worth more than my part,
and a rack of these is worth more than my house.
It's a little sketchy, but I want you guys to be able to see it.
The CPUs are Epic Genoa, so last generation Zen 4 based, but Genoa still supports up to 12
channel DDR5 memory and 128 lanes of PCIe Gen5, which is plenty to keep these GPU cores fed.
If more CPU compute is needed, clearly there is support for dual CPU sockets,
but the team at SFU found that 8 CPU cores per GPU was plenty for their purposes,
and they opted for a single 48 core CPU and 1.152 terabytes of RAM in each of their nodes.
Now for a closer look at the GPUs. Unfortunately, I'm not allowed to take the coolers off them, but under each of these
4 cold plates is an NVIDIA H100 SXM5 80 gig GPU giving us a total of 320 gigabytes of VRAM per node.
And guys, that's not just any VRAM, that is HBM3 running on a 5120 bit bus
for a total bandwidth per GPU of 3.36 terabytes per second.
For context, a top of the line consumer card, the RTX 5090, achieves just over half of that bandwidth.
This kind of power does come with drawbacks however, like for example heat.
Each of these is rated for 700 watts of power consumption through the SXM socket that's
underneath them. And that is where the incredible cooling solution in this Lenovo node comes in.
As a liquid cooling nerd, I gotta say guys, this is the coolest part for me.
I mean, did you notice that there isn't a single fan in sight anywhere in this machine?
That is because everything, CPUs, GPUs, VRMs, network interface, SSD caddy, even the system
memory is directly liquid cooled. All of it. This feels a little bit like doing the maze and
highlights magazine. So here's our inlet over here, which splits into two main loops that go
through the system. The primary loop, which we can tell because it has a thicker pipe coming off of it,
goes straight to the middle of our four GPUs, where this manifold splits fresh incoming water
out to our four GPUs. Two of them just fit right back into the outlet here, while the other two
run up to this networking board and then consolidate back to the outlet. That's our primary loop.
Our secondary loop comes through here, handling some of the power delivery, and then carries over to
interesting. It splits out doing the RAM next. I am not 100% sure what to make of that because I
would think RAM would be a tertiary priority in terms of cooling. But that's what they've done.
We go through the RAM, splitting into three different tubes that sit between our dims down
both rows. Then one side handles this network caddy here and the other side handles our SSD caddy.
Then each of those come back to one of the CPUs, which come out into the middle here and then run
back to the outlet here. Not maybe the way I would have laid it out. There's a lot of 90 degree
turns in here, meaning a lot of restriction, but I'm sure the engineers at Lenovo know what
they're doing. There's a ton of other cool stuff to unpack here too. You probably noticed there's
no power supply. That's because it uses these shonky connectors here at the back to plug into a
backplane in the back of the rack. As for the cooling connections, well, according to the manufacturer,
these do have a little bit of natural leakage, but it's on the order of molecules, which is pretty
damn impressive. There are sensors all over the motherboard to detect any kind of leakage,
and there is grounding throughout the system in the form of these little copper, what look like
solder wick pieces. That's to prevent what's called stray current corrosion, which can be
caused by a current that's accidentally induced in the coolant, which can lead to massive corrosion,
which can lead to leaks, which I know from experience. Now, the team here wasn't sure about
the exact chemistry of the coolant they're using, but they did tell me that it has antimicrobial
properties to prevent anything from growing in the loop. There's some other fun stuff.
There's a little stylus in here. Apparently, this is meant to assist in removing memory,
which is great. I'd actually love to see more gaming motherboards come with that.
I thought this lone 7.68 terabyte NVMe drive was interesting too. I mean, the networking is 400
gigabit per second times two to the two petabytes of NVMe storage, not to mention 49 petabytes of
spinning rust that's right over there, but according to the team here, occasionally they need no local
storage to improve GPU performance a little bit. So you'd never boot off of this or anything,
but it's nice to have there as a scratch. Also, the button cell in here is mounted in a vertical
caddy because the density is so high in this one, you know, that they just couldn't give up the space
that it would have taken to mount it parallel to the board. I also spotted a micro SD header.
If anyone out there works in the data center and knows what that's for, I haven't seen it before,
and Gem and I just assumed that I typoed. Oh, there was one other thing we wanted to look at,
these big power bad boys. We couldn't see them until we got that shroud off. So these,
they're just bus bars. They're going from power supply here, which is a DC to DC power supply.
And they're going over to our GPUs. Damn. What's interesting to me that I just noticed
is that there's a clear delineation between the NVIDIA engineered parts of this with the Black
PCB and they're completely separate from everything else and the Lenovo engineered parts of this.
So Lenovo is acting like more of a system integrator around this compute block here.
Like you can even see the silk screening on the PCB is distinctly NVIDIA and Lenovo is just doing
their DC to DC power. So it's just power in here and then PCIe in here in the form of these four
MCIO connectors right here. This is essentially like plugging a GPU into your Legion gaming PC.
The GPU house. With extra steps. Yeah. Before we poke around in one of the 192 core CPU nodes that
they've got, let's take a look at one of the racks that these boys slide into. They're still using a
very similar rear door chilled liquid rack like we saw with their air cooled nodes when we did a tour
over eight years ago. Anywho, the point is that chilled 16 and a half degree cooling comes from
the evaporative cooling towers outside. Then hot air from the power supplies and any of the
networking equipment that's in the rack runs through here and wow is that ever hot. Then it
spits out nice comfortable room temperature air on the other side. Each of these racks is fed by
dual three phase 60 amp feeds for a total of about 70,000 watts per rack. Now if SFU had the power and
cooling in this 1960s bunker, they could juice these up to 180,000 watts per rack, but they don't.
Hence the empty rack space. Since we have this open. Oh wow. That is a big difference between the
cold side and the hot side going into the back of these back planes for the servers. I don't have
to ask which one's the supply. That's the cold side, which since we're on the subject, this is a
perfect time to look at the cooling distribution system. This is the Lieber XTU from Virtus. It
can do 600,000 watts of cooling capacity per one of these cooling distribution units or CDUs. Water
comes in the supply side here. This thick boy. Ha, she's chilly. That's coming from the cooling
towers outside. Then that runs all the way down to the bottom here to the heat exchanger in the front.
This liquid to liquid heat exchanger does exactly what it says on the tin. Taking that cold water
from the primary leak that goes outside and using it to chill the warm water that is coming directly
off of the blocks that are going to our nodes. This unit uses dual redundant pumps and if we go
back or under the back uses these manifolds and valves to control flow to up to six different racks.
And it's very easy to tell which is the cold side that's being chilled in, which is the hot side here.
Wow. I want one. Vaughn, can I have one? Probably the coolest part is this little touchscreen display
on the front that much more succinctly illustrates what I just said. Here's your primary loop,
here's your secondary loop, here's all your flow rates, all your temperatures, and here's an alarm
that they assure me is totally fine. This data is super important because if they accidentally add
water that is too cool going into the servers behind me, then they could end up a condensation,
which hopefully I don't have to explain why that's super, super bad. Everything's hooked up using
aqua-therm tubing from Germany. The admins here spoke with some other facilities that used stainless
steel and one of them got rust in their cooling system. It was a big, big mess. They've been really,
really happy with their aqua-therm. Now let's go check out the CPU mode. Contrary to what NVIDIA
would like everyone to believe, not everything runs best on a GPU even today and that's where these
come in. Each of these 1U racks contains two nodes and each node contains 192 Zen 5
epic Turing cores with 768 gigs of memory. So that's a total of nearly 400 cores in each of these
1Us. Holy freaking... For networking, they actually don't go as heavy on these using 200 gig connections
and NDR to dynamically share that 200 gigabit link between the two nodes depending on their needs.
This approach does have the drawback of meaning that if the primary node goes down,
we lose network connection to the secondary one, but I have to assume that the cost savings
outweigh the disadvantages in this case. In terms of loop layout, this one is much
simpler coming in to both sides and then out of both sides, but just like the GPU nodes,
the goal here is to get a water tube up against pretty much anything in the server that generates
heat because there are no fans whatsoever. One cool thing we missed on the GPU node was we never
got a look under the little cooling plates that the SSDs and network cards sit on. So here's what
it looks like. It pretty much looks like a heat pipe, except instead of being full of what is usually
a vapor and sometimes a liquid that circulates just within itself, it's just full of water or
other coolant that will circulate into an external system. Now let's take a look at the racks that
these live in. Each of these racks contains 72 of the nodes that I just showed you guys, top to
freak and bottom with roughly 13,824 cores. Each polyrack is an island with a non-blocking 800
gig connection between islands, so 41,000 cores can represent a single job with no blocking.
They have some other specialized nodes like the storage ones, including the ones on the other
side of the aisle that hold data for our local particle collider. Try them. We've got a whole
video about that, along with some eight terabyte RAM nodes, which are, I think, pretty self-explanatory.
They're for jobs that would overflow on a regular node, as long as you don't mind them having a few
bugs. And finally, a single AMD MI300X node, too. I don't know what, keeping video on the toes or
we could do it. We could buy more than one of these. You better not charge too much, especially
when you factor in modern security needs. There are six zones of security to get to some of the cages
that actually have biometric locks on them, where not only do you need to know the pin code,
but you have to put your hand under it and it will check if that hand is attached to a living person.
There's cameras everywhere in the data center with full visibility in all directions,
and our tour guide today actually said that there's someone monitoring them so often
that it's become a bit of a game where they'll send non-flattering pictures of him moving around in
the data center just to make sure he knows they're watching. Cooling everything are these evaporative
cooling towers behind me. The three that are closest to the building, they were there the last
time we were here, but I couldn't show them to you for reasons that involve red tape and approvals.
So here they are. I still can't get any closer to them for reasons that involve red tape and
approvals, but hey, we can check out the acoustic damping that's on these ones on the other side.
That's impressive. Here I am right at the intake next to these sound baffles.
And for context, here's the untreated ones. The total cooling capacity is about 4.7 megawatts,
which that is way more than what's needed for the machines inside. But just like power and
storage, you want to have some extra for resiliency in the event of an equipment failure.
Hey, what's one of these worth? Maybe I'll pick one up for the office.
$1.2 million each. Oh, never mind. Frankly, I'd rather have one of these anyway.
To augment the original pumps for cedar, which could do 800 gallons per minute of cooling,
they added these two new ones that do 1500 gallons per minute. They also now have two
mechanical chillers, which can be useful during the times of year that we get outside temperatures
above 33 Celsius. So for maybe six hours a day, they'll switch over to mechanical chilling to
help out their evaporative cooling tower. Probably the coolest thing about this gear, though,
is how smart it is. They've got telemetry capture for things like temperature and flow rates,
and it all feeds into a third party called Kaizen that helps with logging and determining if
something's gone wrong with the system. Fun fact, by the way, the two-foot thick concrete floor
that I'm standing on is so burdened by all of this heavy equipment and coolant that it actually
deflects half an inch in the center. Is that okay? Am I going to break it? I mean, the place used to
be the power distribution center for the southern half of our province, and it's built like a bunker.
But that's not enough. Who paid for it all? FUR had a total budget of about $82 million, which,
oh, I assume, is Canadian ruble deduce. So a little under 60 million US dollars,
and that came from a combination of the Digital Research Alliance of Canada,
BCKDF, and vendor-in-kind contributions, which I just learned are a vendor giving
significant discounts. How do I get signed up for that program? Is that only for educational
institutions? Anyway, they did ask us to shout out a couple of companies who helped them out.
Lenovo, DDN, and Vertiv on the cooling side. And they didn't ask us to shout these guys out,
but we're going to do it anyway. A shout out for our sponsor. If you guys enjoyed this video,
why not check out the tour we did of Triumph, the particle accelerator that is just down the road.
Down the hill, down the really long road. Canada's only road. It's pretty long.