We have looked at a lot of balling GPUs over the years. Whether it's the six
Titan V's we had for the six editors project, three GV100 Quadras for 12K
ultrawide gaming, or even this unreleased mining GPU, the CMP 170HX.
There are not a lot of cards out there that we have not been able to get our
hands on in one way or another, except for one until now, the NVIDIA A100. This
is their absolute top dog AI enterprise
high performance compute big data analytics monster and they refused to
send it to me. Well, I got one anyway,
NVIDIA. So, deal with it.
The first two questions on your mind are probably why we weren't able to get one
of these and what ultimately changed that resulted in me holding one in my
hands right now. The answer to the first one is that NVIDIA just plain doesn't
seed these things to reviewers. And at a cost of about $10,000,
it's not the sort of thing that I would just, you know, buy because I got that
swagger. You know what I'm saying? As for how we got one, I can't tell you.
And in fact, we even blacked out the serial number to prevent the fan who
reached out offering to get us one from getting identified. This individual
agreed to let us do anything we want with it. So, you can bet your butt we're
going to be taking it apart. And all we had to offer in return was that we would
test Ethereum mining on it, send a shroud that'll allow him to actually
cool the thing, and reassemble it before we return it. So, let's compare it
really quickly to the CMP 170HX, which
is the most similar card that we have. It's this silver metal and it's not
ribbed for my pleasure. Comfortable. And
we actually have one other point of comparison. This isn't a perfect one.
This is an RTX3090. And what would have
been maybe more apt is the Quadro, or rather they dropped the Quadro branding,
but the A6000. Unfortunately, that's
another really expensive card that I don't have a legitimate reason to buy,
and NVIDIA wouldn't send one of those for the comparison either. So, the specs
on this are pretty similar. We're going to use it as a stand-in since we're not really looking at any workstation loads
anyway. So, the A100, then this is a 40
Gigabyte card. I'm going to let that sink in for a second. And the craziest
part is that 40 gigs is not even enough
for the kinds of workloads that these cards are used to crunch through. We're
talking enormous data sets to the point where this 40 gig model is actually
obsolete now replaced by an 80 gig model. And these NVL link bridge uh
connectors on the top here. Let's go ahead and pull these off. These we go
are used to link up multiples of these cards so they can all pull memory and
work on even larger data sets. Now the die at the center of it is a 7nmter TSMC
manufactured GPU called the GA 100. We're going to pop this shroud off.
We're going to take a look at it. It has a base clock of just 765 MHz, but it'll
boost up to 1410. That memory runs at a
whopping 1.5 terabt a second of
bandwidth on a massive 5,120
bit bus. It's got 6,912
CUDA cores and a what is it? Uh 250 watt
TDP. She's packing. Oh, you're just going
right for I'm going right for it. Oh jeez. This is Linus Tech Tips. And
basically every part of this is identical to the CMP card. It kind of
looks that way. I mean, the color is obviously different. Yeah. But it looks
like the clam shell is two pieces in the same manner. There's no display outputs.
The fins look the same. Now, here's something. The CMP card specifically
didn't even contain the hardware for video encode, if I recall correctly.
Yeah. This doesn't have NBN. Okay. So, it's not that it was fused off. It's that it's just plain not on the chip. on
not on GA 100. Yeah. Okay. So, G102,
which is like 3090. Yes. Does have it. God. And A6000. Okay. You ready? Uh oh
god. So, yeah, it's like exactly the same on
the inside. Same jank power connector. Wow, that is super jank. Check this out,
guys. It uses a single 8 pin EPS power
connector, which you might think is a PCIe power connector. See here, look,
I'll show you guys. This is an 8 pin like normal GPU connector. But watch
can no go in. But if we take the connector out of our CPU socket on the
motherboard, there you go. Oh, well, the clips are
interfering a little bit. I mean, what the what the heck is going on here,
ladies and gentlemen? You need more power. Yeah, exactly. So, you can
combine two PCIe connectors into that. Can't remember how to get it out of
here. I see the fingerprint of the technician who assembled the card,
though. Think we have to uncip this part first to just Oh, there's a little
screw, right? Yeah, there's a little screw. Haha. Third type of screws.
Yourself. You didn't see that one, nerd. You're a nerd. Your face is a nerd. Your
butt's a nerd. Whoa. It's not coming off, Jake. What? You got to like tilt it
out, buddy. Whoa, whoa, whoa. Don't pull the cooler off. See, it's like it's caught uh back here. I wouldn't re Hey.
Oh, hey. How you doing? Jesus.
stressful. Look, maybe if we break it, you'll
actually have to buy one. I don't want to buy one. That's not the goal. What? I
thought you put your hand up for a high five. I was like, what are you talking
about? I don't want to buy one. Why not? Wo. What is going on here? You see that?
It looks like there was a thermal pad there or something, but there isn't.
It's like greasy. No, look at it closer. It's not greasy. It's like You see how
this is like brushed almost or like like looks like somebody sand blasted it?
That part's not. Oh, that's I don't remember that on this car. All right, so
the spring loading mechanism is just from the bend of the back plate. That's
kind of cool. So, I checked the CMP thing. Doesn't look like it. I wonder
why they would have like a M. This doesn't look brushed at all. What did we
last time we twisted? No, I don't think we did. Yeah, we did. I just looked. I
just I'm pretty sure I just reamed on it. Oh my god. No, you were against
reaming on it and then we were like, just twist a little. I'm reamer. Oh god.
Ah, it has an IHS. It looks basically the same. Yeah, we're
going to have to clean that off and see. There's not much alcohol. No, I like to
go in dry first. So, yep, that's the
same thing. All right. I mean, this isn't the first time NVIDIA has used the
same silicon in two different products with two different capabilities. We see
the same thing with their Quadro lineup versus their GeForce lineup where things
will just be disabled through drivers or fusing off different functional units on
the chip. What I want to know then is besides the lack of Envy Link connectors
on this one, well, they are in there. They're just not accessible and they
probably don't work, right? What is the actual difference in function between
them? Well, this one doesn't have full PCIe 16X, right? Less memory. It's I
think it has way less transistors, but it is still a G100. Yeah. So, the
transistors are there. Yeah, they're probably just not functional. Let me see
what the chip number is on that one. Yeah, cuz weren't we not even able to
find a proper NVIDIA.com reference to this one anyway? So, we're just relying
on someone else's spec sheet. So, the transistor count could just be wrong.
Okay, so this is So, the CMP card was a
GA Look at this guy. Yeah, what a weirdo. GA00 105F
and this is a GA100 833. If it's a GA, I
guess it could be a different GA. I don't know. Yeah, I mean it used to be
back in the day you would assume that it's just using the same silicon as the GeForce cards because NVIDIA's data
center business hadn't gotten that big yet. But nowadays they can totally justify an individual like new die
design for a particular lineup of enterprise cards. And interestingly
enough, the SXM version doesn't have an IHS. At least it seems that way. But the
SXM version is also like 400 watts and
this is like 250. Yeah. Totally different classes of capabilities. All
right, let's put it back together then, shall we? I got you new goop. Goop me. I
brought two goop. Going for the no look catch.
Oh yeah, baby.
X marks the spot, baby. My finest work.
Maybe it'll perform better now. Probably not.
We're backing it up.
Cool story, bro. Thanks. Thanks, bro. Uh, where's our back plate? Did you take
it? Oh, shoot. Yes, black. I thought it
was gold. I was looking for gold. Aren't we all? I don't know about you,
but I found my gold. What's What's that?
Yvon, shut up. All right. All right. Let's get
going here. Which one do you want to put on the bench first? What do you mean? We're not going to compare to that
thing. Oh, it doesn't do doesn't do anything. Okay, so we don't need this
thing. Here we go, boys. See you later. We can't put this in the first slot because we don't have a display up. But
you like the bottom up? Your bottom?
Sure. This Okay. This is how you flex it
style. Now, you might have noticed at some point that the A100 doesn't have
any sort of cooling fan. It's just one big fat long heat sink with a giant
vapor chamber under it to spread the heat from that massive GPU. So, Jake
actually designed uh what we call the Shroudinator. It allows us to take these
two screws that are on the back of the card for securing it in a server chassis
because that's how it's designed to be used. So, it's passive, but there's lots
of air flow going through the chassis. And then lets us take those screw holes
and mount a fan to the back of the card. It's frankly not amazing.
What? No. That is aerodynamics at its
peak. You should hire me to work on F1 cars. Okay. Yeah, not so much. Yeah, it
it only blows probably more air out this end from the back pressure than it does
out this end, but it's enough to cool it. I swear it is. Yeah. Uh let's go
ahead and turn on the computer, shall we? Okay. So, a couple interesting
points here. It wouldn't boot right off the bat. You have to enable above 4G
decoding. And then I also had to go in and I think it's called like 4G MMIO or
something like that. I had to set that to 42. Okay. The answer to the universe.
Yes. Thank you. And they are both here. A100 PCIe 40 freaking gigabytes.
I installed the like game ready driver for the 3090 and then I installed the
data center driver and I think it overrode it, but the game ready driver
it still showed as like active and you could do stuff with the A100 and vice
versa. So, it's probably fine. Now, interestingly, the A100 doesn't show up
in task manager at all. Did the CMP? I can't remember. No, no, I don't think it
did actually. Anyways, what do you want to do in Blender? Classroom. BMW. BMW is
probably too short. Yeah, let's do classroom. I think BMW on a 3090 is like
15 seconds or something like that. Anyway, let's do classroom. That's also
like the spiciest 3090 that you can get. Yeah, pretty much. It's just so thick.
Why would you ever use it? Yeah, because you want Is it even doing anything?
Like, here's one reason. Cuz you can do classroom renders in a minute and 18
seconds. That's why. Okay. Well, what about the A100? Oh, the You didn't plug
the fan in. Okay. Oh, whoops. How hot is this? Probably warm. Fortunately, it
hasn't been doing anything. Time to beat is a minute and 18 seconds. So, let's go
ahead and see how it does. It feels like
this is the intake. I mean, it's hot, so like Oh, yeah. But Oh, it's it's going.
It's going, Jake. It's going. You did good. It works enough. This should be
like This is It should be way faster. Way huger GPU, right? It's actually
slower. How much? Not by much. It's like a few seconds, but it's slower. So, it's
worse in CUDA. What about optics? So,
the interesting thing is this card doesn't have rateracing cores. The 3090
does. So, you'd think that optics would only work on the 3090, right? Do you
want me to just try the A100? Yeah, sure. Let's Yeah, it's still GPU
compute. I mean, you got to give it to it in terms of efficiency. For real
though, even running two renders to the 3090s one, the average power consumption
here is still lower. Yeah. Well, and looking at while it's running, it's like
150 watts. Yeah. Versus 350 or whatever
it was on the 3090. Yeah. Ready to go again. Yep. Uh Okay. Oh my god, man.
This thing is fast. What's the power cons?
353. The fan is still like just I want one of
these. This looks sick, dude. It's way faster. Yeah, there's no question. We
don't even need to. It's going to be like 30 seconds. Yeah, not even close.
So, do you want to know why? I would love to know why. You said it earlier.
You just weren't really thinking about it. This has half the CUDA cores of a
3090. It's like 7,000ish, I think. So,
it's just full of like machine learning stuff. Yeah. So, it has basically half
the CUDA cores. So, the fact that it is even close is kind of crazy in CUDA
mode. But in optics, what I found out is optics will use the tensor cores for
like AI denoising, but nothing else. You'll see in there. Um, so I I think
it's falling back to CUDA for the other stuff. Got it. But the 3090 has ray
tracing and tensor cores. So, right, it
just demolishes. Uh, where's the thing where you can
select apps and then tell it which GPU to use? Yeah, here we go. No. So, it
will not allow you to select the A100 to run games even if we could pipe it
through our onboard or through a different graphics card like we did with that direct mining card ages ago. No
DirectX support whatsoever. Let's check it in GPUZ. So, way fewer CUDA cores.
You can see that we go from over 10,000 to
a lot less than 10,000. The pixel fill rate is actually higher. I guess that's
your HPM2 memory talking.
1.5 gigabytes per second. What's a 39?
1.5 terabytes per second. It's like
almost%. Yeah. 60% almost.
Holy bananas. But what about the supported tech? Yeah. So, we can do
CUDA, Open CL, Physex.
Sure, we should set it as the PhysX code. Dedicated Physex card. All the rag
dolls everywhere. And OpenGL, but not
Direct Anything or Vulcan even. OpenGL.
Now that's interesting. Go to the advanced tab. Yeah, cuz you can select
like a specific DirectX version at the top under general. Like what about like
DX12? What does it say? Device not found. It's the same as the mining card.
It'll do open seal. So we can mine on it.
All right. I mean, should we try that? Yeah, we could do mining or folding or
Sure. I have a feeling it's going to kind of suck for that, too. Uh, there's
no AI in mining. I don't think so. It's still a big GPU, dude. So, you can't
Well, suck is relative, right? Like, for
the price, you'd never buy. Oh, I think it might be better than the CMP card,
though. Just a little bit. Shut up. I think so. So, the only thing you can
adjust I think this is the same with the CMP card is the core clock and the power
limit. You can't mess with the memory speed. And you can move the power limit
only down, it looks like. Yeah. Top is the 390, bottom is the A100. Wow, that
is a crap ton faster than a 3090. It's
pretty much the same as a CMP, but look at the efficiency. 714 kilahash per
watt. Uh, and I bet you if we lower the power limit to like 80. Uh, it's a
little bit lower speed. Maybe we can go I don't know. We probably don't have to
tinker with this too much. I mean, it doesn't draw that much power to begin
with, I guess. Yeah, I think it's pretty freaking efficient right out of the box.
I mean, the efficiency is better. It's a little bit better. But before it was
doing 175 megaash roughly at 250 watts,
so it's pretty damn good. 3090 you can
probably do like 300 watts with 120
megaash. That's We're running the folding client now. I've had it running
for a few minutes and it's kind of hard to say. The thing with folding is based
on whatever project you're running, which is whatever job the server has
sent you to process, your points per day will be higher or lower. So, it's
possible that the A100 got a job that rewards less points than the 3090 did,
right? It does look like it's a bit higher, but you can see our 39, this is
like a little like comparison app thing, um, is 31% lower than the average. So,
it's probably just that this job doesn't give you that many points. Got it. The
interesting part is the 3090 is drawing
a lot. 400. Holy A 100 is drying.
240, man. That's efficient. And performance
per watt. Maybe gamers don't care that much. Actually, we know for a fact
gamers don't care that much. In the data center, that's everything because the
cost of the card is trivial compared to the cost of power delivery and cooling
on a data center scale. Especially when you have eight of these with a 400 watt
power budget like you would get on the SXM cards in a single chassis times 50
chassis. Like that's a lot of power.
Let's try something machine learning. Unfortunately, for obvious reasons, most
machine learning or deep learning, whatever you want to call it, benchmarks
don't run on Windows. So instead, I've switched over to Ubuntu and we've set up
the CUDA toolkit which is going to include our GPU drivers that we need to
even run the thing as well as Docker and the NVIDIA Docker container which will
allow us to run the benchmark. We're going to be running the ResNet 50
benchmark which runs within TensorFlow 2. This is a really really common
benchmark for big data clusters and stuff except our cluster it's just one
GPU. In a separate window I've got NVIDIA SMI running. It's kind of like
the Linux version of MSI Afterburner, but it's made by NVIDIA, so not quite.
But what it's good for is at least telling us our power and the memory
usage, which we should see spike a lot when we run this benchmark. I took the
liberty of precreating a command to run the benchmark. So, we're going to be
running with XLA on to hopefully bump the numbers a bit. We will do that for
the A100 as well, so no worries there. It should be the same as well as using a
What do you want? Look, he he left cuz he didn't have time for this and now
he's back. This is a the world's most expensive lint roller. I don't even
remember what I was saying. Damn it. Distractions aside, we're going to be
running with XLA on. That'll probably give us a bit higher number than you
would normally. Um, but it is still accurate. And we're going to be running
the same settings on the A100 as well. So, no concerns there. We'll also be
using a batch size of 512 as well as FP16 rather than FP32. So, if you want
to recreate these tests yourself, you totally can. Let's see what our 3090 can
do. Look at that. 24 gigs of VRAM
completely used. God, I don't I don't know if there's any
application aside from like Premiere that will use all that VRAM. I'm sure
Andy can attest to that. Okay. 1,400 images a second. That's
pretty respectable. I think like a V100, which is the predecessor to the A100,
does like less than a thousand. So, the fact that a 3090, which is a consumer
gaming card, can pull off those kind of numbers is huge. Mind you, the wattage,
412 watts, that's that's a lot of power.
It'll be interesting to see how much more efficient the A100 is when we try that after. The test is done now, and
the average total images per second is,435.
It's pretty good. I've gone ahead and added our A100, so we can run the
benchmarks on that instead. And I'm expecting this is going to be
substantially more performant. So, it's the same test. I'm just going to run the
command here. Got to wait a few seconds.
We got NVIDIA SMI up again. You can see that it's just running on the A100. The
RAM on the 3090 is not getting filled. We're just using that as a display output. Yeah. All 40 gigabytes used.
That's crazy.
If we thought the 3090 was fast, look at that, Andy. That's like a full,000
images more. We're getting like 2400 instead of 1,400. And the icing on the
cake, if you look at NVIDIA SMI, we're using like 250 watts instead of 400
while getting like almost double the performance. That is nuts. Probably the
coolest thing about this whole experience though is seeing the Ampear
architecture on a 7nanmter manufacturing process. Cuz you got to remember, while
none of this is applicable to our daily business, what this card does do is
excite me for the next generation of NVIDIA GPUs. Because even though the
word on the street is that the upcoming ADA love lace architecture is not going
to be that different from Aier, consider
this. NVIDIA's gaming lineup is built on Samsung's 8nm node, while the A100 is
built on TSMC's 7 nanometer node. Now, we've talked a fair bit about how
nanometers from one fab to another can't
really be directly compared in that way. But what we can do is say that it is
rumored that NVIDIA will be building the newer ADA love lace gaming GPUs on
TSMC's 5nanmter node, which should
perform even better than their 7nmter node. And if the efficiency improvements
are anything like what we're seeing here, we are expecting those cards to be
absolute freaking monsters. So, good
luck buying one. Hey, at least you can buy one of these.
We've got new pillows. That's right. This is the what are we calling it? The
couch. The couch ripper. It's an AMD
themed version of our CPU pillow with alpaca and regular filling blend. You
guys enjoyed this video? Maybe go check out our previous video looking in more
depth at the CMP 170HX.
I like the silver better. If we were smart, we'd be mining on this, but we're
not that smart. Well, you know, mining is