When I was a young tinkerer, one of my favorite tools was Super Pi, a benchmark
and stability test that calculates
pi. Not this kind of pie, but rather the
mathematical constant that describes the ratio between a circle's circumference
and its diameter. As far as we know, pi
is an irrational number, meaning that there is an infinite number of digits
after the decimal place. And with my overclocked Athlon, I could calculate 32
million of those digits in a half an hour or so. Girls dug it. But I dreamed
of more. I wanted to calculate more digits of pi than any man had ever
calculated before. And why shouldn't I? I'm minus motherucking tech tips. Uh so
the current record set by Jordan Reis last year is 202 trillion digits.
Trillion? Yeah, you say that's a lot of athlons. It's more than that. See, at a
certain point, it's not even just the computation that becomes the problem,
but rather it's the storage. Fortunately for us, both have gotten a little
beefier. And thanks to our friends over at Kioxia, who sponsored this entire
project, providing over two pabytes of
Gen 4 NVMe storage, we were able to
smash that record, calculating nearly
100 trillion more digits. The process of
getting this was an absolute cluster. But hey, that's content, baby. And here
it is. Our verified Guinness World Record for a calculating pi to an
astonishing 300 trillion digit.
Holy. Let's talk about how we got here.
To put the scale of 300 trillion digits into perspective, at size 4 aerial font,
which is still readable, but pushing the limits, you can fit around
25,000 digits on a normal sheet of paper. That means that my childhood
dreams of 32 million digits could fit on around
1,250 sheets of paper. But if we wanted
to print out the number of digits that we just calculated, we should totally do
that. No. Why not? because it would be literally billions of pages. It's only
paper, Lionus. What could it cost? $10?
Take. No. Let's take a look at the hardware.
Yes, my friends. This is the super
secret project that we teased last time we were working on thermal management
for the milliondoll PC. You've seen this
six server cluster here before. But while originally it had about a pabyte
of stupid fast Kioxia Gen 4 NVMe
storage, it has uh grown a little bit
since then. The 72 15 TBTE drives that
made up the original pool would have like almost been enough space to reach
my original goal of 200 trillion digits. That was double the standing record at
the time set by Emma at Google. but it wasn't anywhere near enough to beat the
202 trilliondigit record that popped up a few months into the planning and
testing for this project. So, I had to call in like just a couple favors,
specifically from our friends over at Gigabyte, who sent this lovely WU dual
socket epic chassis, a R183 from Kyozia,
who lent us an older Cyan server from their test feed lab, and finally from
AMD, who provided this Titanite reference platform for the launch of
Epic Genoa. That gave us a total of nine servers. Why so many servers? I mean,
couldn't we just pack all the storage in one? I mean, yeah, but here's the thing.
We already had the million-dollar PC, and those nodes, they're already full.
So, if we wanted to expand the storage, we either needed to throw away that
existing pabyte we already had, or we need to expand the cluster with more
machines. It's definitely not the most power or space efficient way to get two
pabytes of NVMe, but you work with what you got. That's why we ended up with
kind of a mix of drives as well. See,
every one of our servers needs to have the same amount of storage space.
Otherwise, it's lowest common denominator, and you're going to waste
any of the extra capacity that's in the higher capacity nodes. But the issue is
that some of our machines are expansion
slot challenged. So, Kiozia had to send over a bucket of 30 terbte drives,
allowing us to stuff a whopping
245 tab in each of these nine servers
totaling 2.2 pabytes of raw combined Gen
4 storage. So, that takes care of the capacity. But um if my counter is
correct, there's another server here that we haven't mentioned yet. Or at
least it looks like a server, but IDK, half the fronts are missing. So those
are speed holes, brother. It's for performance. All right, let's get it out
of here and take a look. No problem to get this removed. I just got to battle
the crack in here. Do you want to mark any of these or No, doesn't matter.
This is the compute node, the machine
that actually did the crunching. This Gigabyte R283 Z96 has been through some
modifications. Let's It started life as a 24-base storage server. Actually,
wait. So, most of the 160in PCIe Gen 5
lanes from its dual 96 core epic processors were allocated to storage up
front. But our machine, it doesn't really need storage anymore. All that
storage is in everything else. What it needs now or needed anyways was
networking. A lot of it specifically four of these NVIDIA Melanox 200 GB
Connect X7 network cards. These things are wild. Oh yeah. They have two of
those 200 GB ports per card. And thanks
to the insane bandwidth of their 16x
PCIe Gen 5 slot, which is good for, I don't know, 64 GB a second both ways,
these can saturate both of those ports at the same time. So, if you put them
together, that's it's 1.6 terabs of
throughput. That's a lot of terabytes. Yeah, it's actually around 100 GB per
second to each of our 96 core CPUs, which yes, can actually handle that
thanks to the 24 128 gig sticks of DDR5
ECC that Micron sent over. That's 3 terb
of RAM. The more the better. This is the morrest I could get. As for the storage
nodes, those are using dual Connect 6
200 gig cards from the original setup. So each storage server has 400 gig or
about a dual layer Blu-ray per second.
But how does the whole thing work together? Well, we got to put it back in the rack before I can show you that.
Yeah, that'll probably help me. Okay, there we go. All right, it should be
back up. With the magic of Wet FS, which is the same clustered file system that
we've been using ever since we first set up the million-dollar PC, we're able to
use all that network speed we just talked about to run one combined file
system off of all nine of our storage servers completely transparently to any
application, including YC Cruncher, the application we're using to calculate pi.
When I say the same though, I don't mean the same same. I mean, the same old
array did work, but uh that version of Weta was years out of date and running
on an operating system that is now completely end of life. Luckily for us,
the Weta folks helped us nuke the old installs and install their custom image,
which comes with everything pretty much ready to roll. Massive shout out to Josh
and Bob. You guys rock. Thank you for helping us achieve this silly goal. And
the rest of the folks at Weta for allowing us to use your software in a
very, very unsupported, unconventional way. Oh. Oh, yeah. I basically had to
trick Weta into thinking that each server is actually two servers. That way
we could use the most space efficient stripe width, which for Weta means each
chunk of data gets split into 16 pieces with two pieces of parody data
calculated. 16 + 2 is 18, which is also what nine servers times two instances of
WA gets you. For reference, in a supported configuration, we would have
needed 19 discrete servers in order to
accomplish this stripe width, including two for par and one as a hot spare,
which is fantastic for an enterprise environment like where Weta is meant to
be deployed, but very expensive for us.
Enough diver. How fast is it? We haven't
even tuned it yet. What do you Oh, okay. Well, we can tune it first. We can tune
it. The biggest hurdle on the storage side was finding a way to limit the amount of data that flowed between the
two CPUs, which may be a bit counterintuitive, but as soon as say the
left CPU wants to send data via the network cards that are connected to the
right CPU, that's a ton of lats and on
top of that there's a limited amount of bandwidth. And since this is such a
memory intensive calculation, that's why we need so much RAM and that's why we
need all this storage. We don't want to waste memory bandwidth. So we set up two
WA client containers which is just their application that runs on a computer and
allows you to access the storage. Each of those containers got 12 cores
assigned to it. One per chip lit on our giant CPUs so we can maximize the turbo
speed. Uh no actually the reason for that is the cache. So those are 3D
vcache CPUs. That gives us a certain amount of cash per chiplet. And we
didn't want the buffers of YC Cruncher, which is like the amount of space it
uses to like hold stuff in flight to spill out of that cache cuz as soon as
you do, now it's in memory, more memory copies, more wasted bandwidth. And I
tested a lot, which we'll get into a bit. But first, why don't we look at how
fast it goes? Sure. Final setup_final real. Yeah. These are just
scripts to like make the WA containers. Look how the cores, those ones are at
100% usage. Those individual ones, those are all Weta IO cores. Basically, it's a
lot of compute that needs to be reserved. But when you're talking like
100 plus GB a second, which theoretically we are, but you haven't
actually shown me that yet. Here's our little script. The interesting thing about those cores being used is that
while you can just run an app and hope that it ignores them, the Linuxer not
always the best for that. So there's this command called task set which
allows you to like map whatever command or application you're running to only
run on specific cores. Core one is a WA core. We're skipping that one. Core 9,
we're skipping that one. And then this is running two separate tasks. One for
each of our mounts. And you can see it only has the CPU cores from CPU 1 or CPU
2 dependent on the map folder we're using. Let me run over to WA. Cute
little dashboard. There we go. Woo. It's
not 100, but it is pretty nice. That's writing. This is a write. That's right.
That's just setting up. I thought you said you were doing read. It is a retest, but it's setting up the files.
I was telling Jake as we were working on the review for this script. I was like,
man, I've gotten kind of numb to these numbers, you know, after all the
iterations of Wic and all that, you know, 100 GB a second. You're like,
whatever. This is over the network. It never gets old. Actually, I, you know,
you're numb to the numbers until the numbers you're looking at are like 200 gigabytes a second or something like But
like, no, but dude, like the first time we cracked 100 gigabytes a second, it
was all installed locally and that was no file system. All local. This is over
a network with a file system with a functioning file system. Like real ass
actual copying data. Yeah, that's crazy.
It is crazy. And look at the read. The latency is 2 milliseconds. It's because
I'm like oversaturating this. So down here you see the front end usage. That's
the cores on this machine. They're being utilized 100%. I have the system set to
have four NUMA nodes per CPU because that made Y cruncher a little bit
happier. If I turn that off and do one NUMA node per socket, I was able to get
this up to like 150 GB a second. At the
time, I actually set the record for the fastest single client usage according to
the Wacka guys. They since have broken that with like GPU direct storage or
whatever, but this isn't even with RDMA. This is just good code built for NVMe.
It's also good SSDs. Oh, brother. Yeah, look at this. The average usage of the
drives in the array right now is 23%. Nothing. Shout out Kyokia. We're running
a mix of their CD and CM series Gen 4
drives. These things are super fast with individual drive read speeds that are in
excess of 5 gigabytes per second. And
that's not even the fastest they have. You step up to their Gen 5 drives and you're talking like 12, 13, 14 GB a
second. They're available in self- encrypting SKs. They have die failure
recovery, power loss protection. They're perfect for your next server or data
center deployment. Yeah. And this entire time running this application, I didn't
have a single drive. Had a single issue. You got a spreadsheet for tuning Y
cruncher. Dude. Dude, when he adjusts the glasses, you know, getting real.
Okay, here's Y Cruncher. Let's just do a normal pie run. 32 million25. Let's go.
So, this would have taken about half an hour on my old It took 2 seconds to
compute. Really? Yeah. What? for the uninitiated. Why Cruncher is the
software we use to do this run? It was developed by a guy named Alexander E.
Super nice guy. Helped us do some some messing about. Also, didn't help me that
much. Honestly, I asked a lot of questions you didn't answer, but I
figured it out. Anyways, I guess to be clear, the storage we've talked about,
oh man, we need a lot of storage. It's because YC Cruncher uses the storage
like RAM because the output of digits like that 300 trillion is only about 120
tab compressed. Oh, it's not that much. Could we make that available to people?
Oh god, I don't want to think about that. We'll try. Maybe we'll do a
torrent or something. Oh god. But when you're setting up for a run, it actually
tells you how much storage you need for this swap space, which is just basically
like RAM plus that's slower. That's what it's using it for. For us, it was like,
yeah, we're probably going to use like 1.5 pabytes of space at peak. It's uh
it's pretty crazy. Okay, but what did you tune with? Did you see the tuning?
Oh boy. So, this is like some of the tests I did. So YC Cruncher was built
for direct attached storage. And in fact, it doesn't even want you to use
like a RAID controller or software RAID. It does its own internal RAID. And then
on top of that, it also has things you can tune like what multi-threading
algorithm do you use and like how many threads and what size are your IO
buffers, how much memory, how much memory and how many bytes can we read
per seek ideally because if you're using hard drives or SSDs, it's different. Got
it. It was built in an older time and the code base is huge and Alex just does
it in his spare time as far as I'm aware. So, no shade. Super cool project.
But at some point in the future, technology has gotten good enough that we can just rely on the operating system
to do this. Like that Wacka speed test we just did. Let's hope for that one
day. Anyway, with everything dialed in on August 1st, 2024. Yes. Which was a
while ago. Yes. Jake finally hit enter on his command prompt and began our
glorious journey to nerd glory. Yeah. For 12 days. And then it stopped thanks
to a multi-day power outage while I was on vacation. And it was so early in the
process that I said, "Fuck that. Let's just start it again. I want to get a
clean run with no outages. But it was
smooth sailing from then on. No, it wasn't. See, even with a cluster this
chunk, calculations like this take a lot of time. The previous 202 trilliondigit
record took a 100 days just to compute. And whether it's bad luck or user error,
I think there's a little bit of user error. Finding a space in our facilities
where a machine like that can operate completely uninterrupted. I have no idea
what I just unplugged. How's your edit going? I'm holding your server. What's a
challenge? At first, things were pretty okay in the lab server room here. We had
our air conditioning working to keep things cool. We had our battery back up
to keep the digits flowing during a short outage or a brown out. It's just
that over the course of this run, we had multiple other power outages and none of
them were small. So each time our calculation had to stop and restart. And
the same goes for when the cooling failed. multiple times. It's pretty mint
now though. The AC is fixed and that with our sick water door that is
definitely not going to leak has the room at around 22 23° and the cluster is
still running. Fortunately, Ycuncher makes checkpoints which allows resuming
the calculation but it does mean our record could have been done much faster
like based on the log somewhere in the neighborhood of like 30 40 50 days
faster. The 300 trillionth digit of pi is five. Really?
It's done, baby. Wow. It only
took way longer than it should have. 190 days. Speaking of the logs,
Jake's got them here right now. 100 gigabytes a second read and then you're
writing for like 30 40 gigabytes a second. So that's as it
pulling in data from the swap space which is our NVMe drives and bringing it
into the 3 TB of RAM that are in the system. So, crunch crunch crunch crunch
crunch crunch crunch and huck it back on. Write some data over there. Yeah.
What I don't see in the logs here is how much power this consumed. How much did
this cost? I actually haven't done the math on that. I think it roughly draws
around 8,000 watts, which means 24 hours
a day for a year. Yeah, it was like 10 grand. That's
Canadian in electric. Are you kidding me right
now? No. Like just CPUs. We don't even have GPUs in this thing. It's like 1,500
watts in SSDs alone. Okay. But hey, that
means our record should be safe for a while then, right? Well, it's possible,
maybe even probable, that someone is already working on a run that would beat
this record. And they could probably even do it on a single machine. They
totally could. But that's how it is with computing. And they can never take that
piece of paper away. I can't. I'm taking this one home. And don't forget about
the other pieces of paper. Okay, real talk though. In school, we're taught
that two digits of pi is enough to approximate most calculations. But
obviously, depending what you're doing, you you could need a few more. Is there
in your mind any practical use for 300 trillion digits? No. I mean, other than
for this, but it was fun. It's about the
journey, not the destination, Lionus. It's about doing something cool with the
help of Kioxia, who builds highquality, high performance storage for the data
center, and who we'll have linked down below. It's about Weta and their crazy
software. It's about YC Cruncher. It's about because we could. Because
we can. Just like we could. Also, shout out some of the other folks
who helped us. Oh yeah. Josh and Bob again. Thank you so much from Wacka.
Gigabyte for sending us that server and I haven't made content about it in like
4 years. Just just thank you AMD. AMD
sent the CPUs for the compute node like 3 years ago. Finally. Thank you. I swore
I was going to make this video and it happened. It just took longer than I thought. Thank you, James. the writing
manager for being patient with this project. And uh hey, if you guys want to
check out more Linus and Jake shenanigans, how about the high availability uh cheapo computers? That
was fun. Cheapo computer. Yeah, remember we Oh, that was cool. I don't know. But
we didn't actually do the cheapo one just for the intro. I know, but it was
cool. Yeah. Yeah, that was cool. It works on it. I um Yeah, this is this was
fun. I don't think I ever want to do this again. And cut.