When I was a young tinkerer, one of my favorite tools was super pi, a benchmark and stability test
that calculates pi. No, not this kind of pi.
But rather, the mathematical constant that describes the ratio between a circle's circumference
and its diameter. As far as we know, pi is an irrational number,
meaning that there is an infinite number of digits
after the decimal place. And with my overclocked athelon,
I could calculate 32 million of those digits
in a half an hour or so. Girls dug it.
But I dreamed of more. I wanted to calculate more digits of pi
than any man had ever calculated before. And why shouldn't I?
Online is motherfucking tech tips. So the current record, set by Jordan Reines last year,
is 202 trillion digits. Trillion?
Yeah, you say. That's a lot of athelons. It's more than that.
See, at a certain point, it's not even just the computation
that becomes the problem. But rather, it's the storage.
Fortunately for us, both have gotten a little beefier.
And thanks to our friends over at Kyoksia, who sponsored this entire project,
providing over two petabytes of Gen 4 NVMe storage,
we were able to smash that record, calculating nearly 100 trillion more digits.
The process of getting this was an absolute cluster.
But hey, that's content, baby. And here it is, our verified Guinness World Record
for a calculating pi to an astonishing 300 trillion digits.
Holy shit. Let's talk about how we got here.
The scale of 300 trillion digits into perspective.
At size four aerial font, which is still readable,
but pushing the limits, you can fit around 25 and a half thousand digits
on a normal sheet of paper. That means that my childhood dreams of 32 million digits
could fit on around 1,250 sheets of paper.
But if we wanted to print out the number of digits that we just calculated.
We should totally do that. No, why not? Because it would be literally billions of pages.
It's only paper, Linus. What could it cost, $10?
No. Let's take a look at the hardware.
Yes, my friends. This is the super secret project that we teased
last time we were working on thermal management for the million dollar PC.
You've seen this six server cluster here before,
but while originally it had about a petabyte
of stupid fast, Keopsia Gen 4 NVMe storage,
it has thrown a little bit since then.
The 72 15 terabyte drives that made up the original pool
would have like almost been enough space to reach my original goal of 200 trillion digits.
That was double the standing record at the time set by Emma at Google.
But it wasn't anywhere near enough to beat the 202 trillion digit record
that popped up a few months into the planning and testing for this project. So I had to call in like just a couple favors.
Specifically from our friends over at Gigabyte, who sent this lovely 1U DualSocket Epic chassis,
our 183 from Keopsia, who went to us an older tie-in server
from their test feed lab. And finally from AMD, who provided this tight mate reference platform
for the launch of Epic Genoa. That gave us a total of nine servers.
Why so many servers? I mean, couldn't we just pack all the storage in one?
I mean, yeah, but here's the thing. We already had the million dollar PC,
and those nodes, they're already full. So if we wanted to expand the storage,
we either needed to throw away that existing petabyte we already had,
or we need to expand the cluster with more machines.
It's definitely not the most power or space-efficient way to get two petabytes of NVMe,
but you work with what you got. That's why we ended up with
kind of a mix of drives as well. See, every one of our servers
needs to have the same amount of storage space.
Otherwise, it's lowest common denominator, and you're gonna waste any of the extra capacity
that's in the higher capacity nodes. But the issue is that some of our machines
are expansion slot-challenged. So Kyoksia had to send over a bucket
of the 30 terabyte drives, allowing us to stuff a whopping 245 terabytes
in each of these nine servers, totaling 2.2 petabytes of raw combined Gen4 storage.
So that takes care of the capacity, but if my count is correct,
there's another server here that we haven't mentioned yet, or at least it looks like a server,
but IDK, half the fronts are missing. So those are speed holes, brother.
It's for performance. All right, let's get it out of here and take a look.
No problem to get this removed. I just got to battle the crack in here.
Do you want to mark any of these? Nope, doesn't matter.
This is the compute node, the machine that actually did the crunching.
This Gigabyte R283 Z96 has been through some modifications, let's go.
It started life as a 24-base storage server, actually.
So most of the 160-inch PCIe Gen5 lanes
from its dual 96-core Epic processors
were allocated to storage upfront, but our machine, it doesn't really need storage anymore.
All that storage is in everything else. What it needs now, or needed anyways,
was networking a lot of it. Specifically, four of these NVIDIA Melanox
200 gigabit Kinect X7 network cards.
These things are wild. Oh yeah. They have two of those 200 gigabit ports per card,
and thanks to the insane bandwidth
of their 16X PCIe Gen5 slot, which is good for, I don't know,
64 gigabytes a second both ways, these can saturate both of those ports at the same time.
So if you put them together, that's... It's 1.6 terabits of throughput.
That's a lot of terabits. Yeah, it's actually around 100 gigabytes per second
to each of our 96-core CPUs, which yes,
can actually handle that, thanks to the 24128 gig sticks
of DDR5 ECC that Micron sent over. That's three terabytes of RAM.
The more the better. This is the moreest I could get. As for the storage nodes,
those are using dual Kinect X6 200 gig cards
from the original setup. So each storage server has 400 gig,
or about a dual layer Blu-ray per second.
But how does the whole thing work together? Well, we gotta put it back in the rack before
I can show you that. Yeah, that'll probably help me. Okay, there we go.
All right, it should be back up. With the magic of Weka FS,
which is the same clustered file system that we've been using ever since we first set up
the million dollar PC, we're able to use, all that network speed we just talked about
to run one combined file system off of all nine of our storage servers,
completely transparently to any application, including Y Cruncher,
the application we're using to calculate Pi. When I say the same though, I don't mean the same, same.
I mean, the same old array did work, but that version of Weka was years out of date
and running on an operating system that is now completely end of life.
Luckily for us, the Weka folks helped us nuke the old installs and install their custom image,
which comes with everything pretty much ready to roll. Massive shout out to Josh and Bob, you guys rock.
Thank you for helping us achieve this silly goal. And the rest of the folks at Weka
for allowing us to use your software in a very, very unsupported unconventional way.
Oh, oh yeah. I basically had to trick Weka into thinking
that each server is actually two servers. That way we could use the most space efficient stripe width,
which for Weka means each chunk of data gets split into 16 pieces
with two pieces of parity data calculated. 16 plus two is 18, which is also what nine servers
times two instances of Weka gets you. For reference, in a supported configuration,
we would have needed 19 discrete servers
in order to accomplish this stripe width, including two for parity and one as a hot spare,
which is fantastic for an enterprise environment like where Weka is meant to be deployed,
but very expensive for us. Enough to ever gather.
How fast is it? We haven't even tuned it yet, what do you buy?
Okay, well we can tune it. First we can tune it. The biggest hurdle on the storage side
was finding a way to limit the amount of data that flowed between the two CPUs,
which may be a bit counterintuitive, but as soon as say the left CPU wants to send data
via the network cards that are connected to the right CPU,
that's a ton of latency. That's a lot of hops. And on top of that, there's a limited amount of bandwidth.
And since this is such a memory intensive calculation, that's why we need so much RAM,
and that's why we need all this storage, we don't wanna waste memory bandwidth.
So we set up two Weka client containers, which is just their application that runs on a computer
and allows you to access the storage. Each of those containers got 12 cores assigned to it,
one per chiplet on our giant CPUs.
So we can maximize the turbo speed? No, actually the reason for that is the cache.
So those are 3DV cache CPUs. That gives us a certain amount of cache per chiplet,
and we didn't want the buffers of Y Cruncher, which is like the amount of space it uses
to like hold stuff in flight to spill out of that cache.
Because as soon as you do, now it's in memory, more memory copies, more wasted bandwidth.
And I tested a lot, which we'll get into a bit. But first, why don't we look at how fast it goes?
Final setup, underscore final, underscore for real. These are just scripts to like make the Weka containers.
Look how the cores, those ones are at 100% usage, those individual ones,
those are all Weka IO cores basically. It's a lot of compute that needs to be reserved.
But when you're talking like 100 plus gigabytes a second,
which theoretically we are, but you haven't actually shown me that yet.
Here's our little script. The interesting thing about those cores being used
is that while you can just run an app and hope that it ignores them,
the Linux scheduler, not always the best for that. So there's this command called task set,
which allows you to like map whatever command or application you're running
to only run on specific cores. Core one is a Weka core, we're skipping that one.
Core nine, we're skipping that one. And then this is running two separate tasks,
one for each of our mounts. And you can see it only has the CPU cores
from CPU one or CPU two, dependent on the map folder we're using.
Let me run over to Weka. Cute little dashboard.
Woo! It's not a hundred, but it is pretty nice.
That's writing. This is a write. That's right. That's just setting.
You said you were doing read. It is a retest, but it's setting up the files.
I was telling Jake as we were working on the review for this script, I was like, man, I've gotten kind of numb to these numbers.
You know, after all the iterations of one, it can all that. You know, a hundred gigabytes a second,
this is over the network. It never gets old.
Actually, you know, you're numb to the numbers until the numbers you're looking at are like 200 gigabytes a second or something.
But like, no, but dude, like the first time we cracked a hundred gigabytes a second.
It was all installed locally. And that was no file system, all local.
This is over a network. With a file system. With a functioning file system.
Real ass actual copying data. Yeah.
That's crazy. It is crazy. And look at the read. The latency is two milliseconds.
It's because I'm like oversaturating this. So down here, you see the front end usage.
That's the cores on this machine. They're being utilized a hundred percent.
I have the system set to have four NUMA nodes per CPU because that made Y Cruncher a little bit happier.
If I turn that off and do one NUMA node per socket, I was able to get this up to like 150 gigabytes a second.
At the time, I actually set the record for the fastest single client usage.
According to the WECA guys, they since have broken that with like GPU direct storage or whatever.
But this isn't even with RDMA. This is just good code. Built for NVMe.
It's also good SSDs. Oh brother. Yeah. Look at this.
The average usage of the drives in the array right now is 23%.
Wait, nothing. Shout out Kyokesia. We're running a mix of their CD
and CM series Gen 4 drives. These things are super fast
with individual drive read speeds that are in excess of five gigabytes per second.
And that's not even the fastest they have. You step up to their Gen 5 drives and you're talking like 12, 13, 14 gigabytes a second.
They're available in self-encrypting SKUs. They have Dyke failure recovery, power loss protection.
They're perfect for your next server or data center deployment.
Yeah. And this entire time running this application, I didn't have a single drive, had a single issue.
You got a spreadsheet for tuning Y Cruncher? Dude, dude.
When he adjust the glasses, you know, getting real. Okay. Here's Y Cruncher.
Let's just do a normal pie run. 32 million, 25.
Let's go. So this would have taken about half an hour on my old past one.
It took 0.2 seconds to compute. Really? Yeah.
What? For the uninitiated, Y Cruncher is the software we use to do this run.
It was developed by a guy named Alexander Yee. Super nice guy, helped us do some messing about.
Also, didn't help me that much. Honestly, I asked a lot of questions he didn't answer,
but I figured it out anyways, I guess. To be clear, the storage, we've talked about,
oh man, we need a lot of storage. It's because Y Cruncher uses the storage like RAM
because the output of digits, like that 300 trillion, is only about 120 terabytes compressed.
Wow, it's not that much. Could we make that available to people? Oh, God.
I don't wanna think about that. We'll try. Maybe we'll do a torrent or something. Oh, God.
But when you're setting up for a run, it actually tells you how much storage you need
for this swap space, which is just basically like RAM plus.
That's slower. That's what it's using it for. For us, it was like, yeah, we're probably gonna use
like a 1.5 petabytes of space at peak.
It's pretty crazy. Okay. But what did you tune, Jake?
You wanna see the tuning? Oh boy. So this is like some of the tests I did.
So why Cruncher was built for direct attached storage?
And in fact, it doesn't even want you to use like a RAID controller or software RAID.
It does its own internal RAID. And then on top of that,
it also has things you can tune like, what multi-threading algorithm do you use?
And like how many threads? And what size are your IO buffers?
How much memory? How much memory and how many bytes can we read per seek?
Ideally, because if you're using hard drives or SSDs, it's different.
Got it. It was built in an older time and the code base is huge.
And Alex just does it in his spare time as far as I'm aware.
So no shade, super cool project. But at some point in the future,
technology has gotten good enough that we can just rely on the operating system to do this.
Like that Weka speed test we just did. Let's hope for that. One day.
Anyway, with everything dialed in on August 1st, 2024.
Yes. It was a while ago. Yes. Jake finally hit enter on his command prompt
and began our glorious journey to nerd glory.
Yeah, for 12 days. And then it stopped thanks to a multi-day power outage
while I was on vacation. And it was so early in the process that I said,
f*** that s***. Let's just start it again. I want to get a clean run with no outages.
But it was smooth sailing from then on.
No, it wasn't. See, even with a cluster this chonk,
calculations like this take a lot of time.
The previous 202 trillion digit record took a hundred days just to compute.
And whether it's bad luck or user error.
I think there's a little bit of user error. Finding a space in our facilities
where a machine like that can operate completely uninterrupted.
I have no idea what I just done plugged.
How's your edit going? I'm holding your server. What's the challenge?
At first things were pretty okay in the lab server room here. We had our air conditioning working to keep things cool.
We had our battery backup to keep the digits flowing
during a short outage or a brownout. It's just that over the course of this run,
we had multiple other power outages and none of them were small.
So each time our calculation had to stop and restart.
And the same goes for when the cooling failed multiple times.
It's pretty mid now though. The AC is fixed and that with our sick water door
that is definitely not going to leak has room at around 22, 23 degrees.
And the cluster is still running. Fortunately, Y Cruncher makes checkpoints
which allows resuming the calculation. But it does mean our record could have been done much faster.
Like based on the log somewhere in the neighborhood of like 30, 40, 50 days faster.
The 300 trillionth digit of pie is five.
Really? Ha ha ha ha ha. It's done baby.
Wow. It only took way longer than it should have.
190 days. Speaking of the logs, Jake's got them here right now.
100 gigabytes a second read and then you're writing for like 30, 40 gigabytes a second.
So that's as it's, what? Pulling in data from the swap space
which is our NVMe drives and bringing it into the three terabytes of RAM
that are in the system. So the crunch, crunch, crunch, crunch, crunch, crunch, crunch
and huck it back over there. Write some data over there, yeah. What I don't see in the logs here
is how much power this consumed. How much did this cost?
I actually haven't done the math on that. I think it roughly draws around 8,000 watts
which means 24 hours a day for a year.
Yeah, it was like 10 grand. That's Canadian.
You know, that's, are you kidding me right now? No. Like just CPUs.
We don't even have GPUs in this thing. It's like 1,500 Watts in SSDs alone.
Okay, but hey, that means our record should be safe for a while then, right?
Well, it's possible, maybe even probable
that someone is already working on a run that would beat this record
and they could probably even do it on a single machine. They totally could.
But that's how it is with computing and they can never take that piece of paper away.
I can, I'm taking this one home. And don't forget about the other pieces of paper.
Okay, real talk though. In school, we're taught that two digits of pi
is enough to approximate most calculations but obviously, depending what you're doing,
you could need a few more. Is there, in your mind, any practical use
for 300 trillion digits? No, I mean other than for this.
But it was fun. It's about the journey, not the destination Linus.
It's about doing something cool with the help of Kyoksia who builds high quality, high performance storage
for the data center and who will have link down below. It's about Weka and their crazy software.
It's about Y Cruncher. It's about because we fucking could.
Because we fucking can't. Just like we could also shout out
some of the other folks who helped us. Yeah, Josh and Bob again. Thank you so much from Weka.
Gigabyte for sending us that server and I haven't made content about it in like four years.
Just, just thank you. AMD, AMD sent the CPUs for the compute node
like three years ago. Finally, thank you. I swore I was gonna make this video
and it happened. It just took longer than I thought. Thank you, James, the writing manager
for being patient with this project. And hey, if you guys wanna check out
more Linus and Jake shenanigans, how about the high availability cheapo computers?
That was fun. Cheapo computers? Yeah, I remember. Oh, that was cool.
Yeah, that was super cool. I don't know if we didn't actually do the cheapo one. We did it, we just did the demo.
Just for the intro. I know, but it was cool. Yeah, yeah, that was cool.
Yeah, this was fun. I don't think I ever wanna do this again.
And cut.