When you fire up chat GPT, you're connecting to a big silicon brain that lives somewhere and
contains something. So what exactly is that thing? What is the hardware that runs the chat bot that
you've fallen in love with? There's actually a sort of basic building block of the chat GPT
infrastructure, the NVIDIA A100 GPU. And if you thought the graphics card in your computer was
expensive, the A100 goes for around $10,000 a pop, roughly the same as six RTX 4090s.
Artificial intelligence applications often utilize GPUs because GPUs are very good at doing lots of
math calculations at once in parallel. And NVIDIA's newer models have the tensor cores,
which are good at matrix operations that AIs frequently use. So even though the A100 is called
a GPU, it's built specifically for AI and analytical applications. And as such, you can't
realistically game on it. It doesn't even have a display out. Although you can get the A100 in a
PCI Express version, such as the one Linus is holding up here, it's more common in data centers
for them to come in this form factor called SXM4. Unlike a normal graphics card, the SXM4 cards
lie flat and connect to a large motherboard like PCB using a pair of sockets with the connectors
sitting on the underside of the card. Although SXM is just a connector and data is still carried
over a PCI Express interface, SXM4 is preferred over the traditional PCIe slot for data centers
because the socket can handle more electrical power. The PCIe version of the A100 can use a
max of up to 300 watts, but the SXM4 version handles up to 500 watts, leading to higher
performance. An SXM4 A100 has 312 teraflops of FP16 processing power. To put that in context,
that's nearly four times as much as an RTX 4090, the most powerful consumer GPU on the market at
the time of filming. Additionally, these GPUs are linked up with a high-speed NVLink interconnect
so that the GPUs that sit on a single board can act like a single gigantic Chungish GPU.
Now that you know what lies at the heart of the GPU servers, though, exactly how many A100s are
needed to keep the service running for 100 million users. It turns out you can run ChatGPT
just fine on its own on a single NVIDIA HGX A100 unit. These units typically contain 8 A100 GPUs
in one machine powered by a pair of server CPUs that each feature a few dozen cores.
But the issue is that with so many users, you need a lot more processing power to ensure the
chatbot can answer query smoothly for everyone. OpenAI and Microsoft, who are behind the ChatGPT
project, haven't disclosed exact numbers about their hardware, but given the processing capacity
of these HGX A100 systems, ChatGPT likely uses somewhere around 30,000 A100s to keep up with
demand. To put that into context, it's a heck of a lot more than the roughly 4 or 5,000 they likely
needed to train the language model in the first place. Training is the process of feeding the AI
lots of information in order to build it out before it can be used publicly. Intuitively,
it might seem like the training process would need more processing power than actually running
the model. But because of the massive amount of IO ChatGPT has to handle with 100 million users,
it ends up actually requiring roughly 6 times more GPUs to run it. And with as pricey as these
systems are, you can bet this meant a massive investment on Microsoft's part. While the
actual dollar amount hasn't been disclosed, we do know that it was in the hundreds of millions of
dollars in addition to several hundred grand a day just to keep the system running. Unless you
think that Microsoft is going to stop there, the company is also integrating the newer NVIDIA
H100 GPUs into its Azure Cloud AI service, which actually dwarf the A100's FP16 performance
by a factor of 6. In addition to adding FP8 support, which should prove to be very useful for AI
due to how the math calculations involved in running AI models work. Not only will this ensure
that more people can use ChatGPT and other AI services, but will also allow Microsoft to train
more complicated large language models. Maybe soon, you'll be able to completely replace those
pesky real life friends of yours. So thanks for watching guys, if you liked this video hit like,
hit subscribe, and hit us up in the comments section with your suggestions for topics that we
should cover in the future.