AI chips have suddenly become a big selling point for phones,
but that might seem a little surprising that your little smartphone, which already
has serious limitations on power consumption and heat generation, can run something as seemingly complicated as AI.
So how exactly do they pull this off? Well, these neural processing units, or NPUs,
are quite a bit different than your phone's main CPU cores, features like Apple's neural engine
or the machine learning engine on a Google Tensor chip
are highly optimized for AI tasks, but probably suck at pretty much anything else.
It's kind of like how a GPU works. Although they are much better for rendering graphics
than a more general-purpose CPU, you're not going to run your operating system off
of your graphics card. They are embarrassingly parallel.
A relatively small amount of dye area then that is dedicated to AI can effectively run machine learning
based tasks without sucking down too much power.
But that doesn't answer the question of why there's such a push to put these chips in our phones
in the first place. I mean, we hear so much about cloud AI,
where neural networks run on powerful servers. So can't we just offload tasks like image optimization
and voice recognition to the cloud? Well, the answer lies in how large and complex the AI models
are that your device needs to use. Models for common smartphone AI features,
such as voice recognition, facial recognition, and some kinds of image correction
are often relatively small, meaning that they can be run on device
on a limited amount of silicon. And if these functions can be run locally
instead of in the cloud, it's generally better to do so.
For example, if you use an Android phones speech recognition
button, you will wait around for your phone to send your speech over to a server over the internet,
wait for that server to figure out what you're trying to say, and then wait to get the results back to your phone.
If you could get results right now, that would be a big selling point for a modern phone.
So even though cloud hardware might be more powerful,
the latency advantage of having a chip on your device
makes this trade-off worth it. Not to mention that it helps protect your privacy
by keeping as much of your data on your phone as possible.
But when may it not make sense to rely on a phone's NPU?
More advanced forms of generative AI aren't quite at the point
where you can run them on a phone efficiently. And by generative AI, I mean artificial intelligence
that can create new media. Think about the stories that get generated by chat GPT
or the AI art from services like mid-journey.
Now, you probably don't expect to run an entire advanced image generation model on a phone,
at least with NPUs the size they are now. But what about commonly touted features
like Google's Magic Editor on its Pixel lineup? Well, Magic Editor appears to need an internet connection
since the feature uses enough generative AI to the point where the phone has to rely on cloud servers
in order to give you the image you want in a reasonable amount of time.
However, less demanding features, such as live translate, can run on device.
Since the idea of AI-specific hardware on consumer devices is still relatively new,
tech companies are still trying to figure out exactly where the sweet spot is
in terms of which tasks can and should be done on device
versus which ones should be offloaded to the cloud. In fact, lots of AI as a service type products
don't yet have a clear pathway to monetization.
Instead, it's more common for tech firms to roll the features out now,
figure out how they work, and then jam them into their business model
at some point down the line. This is actually part of the reason that the dye areas of NPUs and phones
are still relatively small. Hardware manufacturers would rather have
enough inside the phone to enable AI features,
but then figure out exactly what the use cases are
before they dedicate more hardware to AI.
You're also seeing this on the desktop and laptop side of things,
with both AMD and Intel coming out with consumer processors that include NPUs.
And the ideas that features like Windows Studio Effects
will run on device so your video calls look a little bit nicer.
But as time goes on, both PC and phone manufacturers
are aiming to get more and more AI functions running locally.
You're already seeing the push for this with how both Team Red and Team Blue
have partnered with a number of outside software developers to make applications that can take advantage of their NPUs.
While it remains to be seen what AI features will become mainstays,
it's clear that your gadgets are going to have significantly more brain power going forward.
For better or for worse.
If you guys enjoyed this video, leave a like or dislike depending on how you feel.
Check out our video on the hardware that runs ChatGPT
if you're looking for something else to watch and leave a comment if you have a suggestion
for a future video. And of course, don't forget to subscribe.