Hardware-Aware AI, Not Just Bigger Models

What if the obstacle to fast, reliable AI isn’t your dataset or your optimizer—but the silicon under your model? We dig into why performance collapses when architecture and hardware don’t align, and we lay out a clear path to ship models that actually fly on the devices your users own. Starting with the Ferrari-and-hummingbird metaphor, we show how theoretical efficiency—FLOPs, parameters, even TOPS—often fails to predict real-world latency, power, and user experience.

We walk through a surprising benchmark: MobileNet V2, small and “efficient,” runs slower than an older ResNet18 on GPUs because depthwise, sequential kernels underutilize parallel hardware. Then we zoom out to hardware selection itself, where NPUs can outperform GPUs despite lower TOPS due to operator support, kernel fusion, and memory behavior. The takeaway is simple: architecture matters only in context, and context means the execution engine, compiler stack, and memory hierarchy that will carry your model in production.

From there, we share a four-step framework to become hardware aware: profile on real devices from day one, verify operator compatibility early, automate bottleneck discovery and model selection in CI, and optimize with context using hardware-aware pruning and mixed precision. To show how this works in practice, we unpack our Llama 3.2-1B project on Snapdragon Gen 3, where targeted pruning and precision tuning delivered 31% faster token generation, 25% faster prompt processing, and a 126% faster initialization—all with under 1% accuracy loss.

If you build models for the edge, mobile, GPUs, or NPUs, this conversation will help you avoid dead-ends and design for the hardware you actually ship on. Subscribe for more deep dives, share this episode with your team, and leave a review to tell us which hardware you’re targeting next.

Transcript

Ferraris, Hummingbirds, And Mismatch

SPEAKER_00 0:08

I guess we all know the Ferrari. And when I think about the Ferrari, I think about speed. It made for asphalt, for smooth curves. But putting it in an environment it was not made for, it will sink. All that horsepower lost. Or here. The hummingbird. Meteorism on overdrive at a wing speed ninety wing speeding 90 times a second. It needs to fuel up. It needs to visit flowers, up to a thousand flowers a day, feeding every 10 to 15 minutes. And asking it to cross an ocean, we would never do that. So why are we asking our models in HII to be deployed in an environment that they are not made for? There's nothing quite so useless as doing something with great efficiency that should not be done at all. The Ferrari is made for asphalt. The hummingbird is made where there is fuel, where there are flowers, and where they fail, the Jeep and the Swift Excel. Today we have three goals. And today I want to compress the knowledge we have gained at Embattel over the last couple of years in deep learning optimization. And teach you some of that. And I hope everyone of you get something out of this presentation. My name is Jonah Matizen, and I'm a deep learning researcher at Embattel. And I'm mostly involved in the algorithm development. Have seen many different customer projects, have seen many different issues that arise when it comes to deploying models onto the edge. And today we're going to try to tackle some of them. And we're going to see how we can solve the issue of deploying models onto environments they are not made for. And we're going to look first at some examples, and then at the end, I show you some practical tips on how to avoid all those traps and tricks. So let's start with this. I think most of you have probably seen this. And

Goal: Hardware-Aware Optimization

SPEAKER_00 2:42

maybe today we are we are getting better at at not doing this, but it is still kind of true. When we try to optimize models, when we try to increase the accuracy of models, oftentimes the solution to it, or what we think the solution, is to just stack more lays, making the models larger and larger. Instead of really maybe improving the architectural or the other, uh maybe the training or the data set. Our silly approach may be to just stack more layers. I said in the beginning we're going to look at some of the examples of what not to do, of where models are meant to be run in a different environment. And the first example we're going to look at is uh WestNet 18 and MobileNet V2. So those are even today, even though they are quite old architectures, they are widely used even today in vision models, perhaps as backbones in some object detection models, or uh image uh classification by themselves, or semantic segmentation. So the uh areas we deploy those models are huge. And they are quite distinct. They are quite different. Even though they have roughly the same acquaintance on ImageNet, uh in terms of flops, in terms of parameters, and in terms of architecture, they differ quite a lot. So the uh mobile net v2

Bigger Isn’t Better By Default

SPEAKER_00 4:17

is roughly six times smaller than the ResNet 18. However, if we deploy this model onto a Nvidia GPU, it will be slower. Even though theoretically, just looking at the numbers, just looking at the flops or the number of parameters, this model is tiny. It should run fast. It should be much faster than this ResNet 18 just looking at the numbers. But the architecture has implications. And if we remember the mobile net paper, it is called mobile net for a reason. And mobile in this context means CPU devices. MobileNet V2 mainly consists of death by several convolutions. And those convolutions are many small sequential computations. Great for a CPU, but definitely not for a GPU. And ResNet 18, very old architecture, but mainly consists of the typical dense convolution. Even then, even though it is very old, it outperforms the MobileNet V2 just because its architecture fits the hardware better. Second example. We stick with the ResNet 18 or ResNet 15, in this case, just a larger ResNet 18 that's like like this. We quantized it. And here we see an extract from the Battle Hub, which is essentially a large benchmarking website, among other things. And we deployed this model onto different devices. Mainly we are now going to focus on the first and the last. The first being the Nvidia Jetson AGX Owen, the last being the Qualcomm's Netduring 8 Gen 3. The first is the GPU, the last one an NPU. And on the Owen, the WestNet 50 runs at roughly 0.61 milliseconds. But on the Qualcomm, with much lower TOPS, and TOPS being kind of an approximation of the hardware capabilities, it is faster. So just looking at the tops of a hardware is also not really helping us. We need to look at both, with the model architecture and the hardware capabilities, and then make decisions based on those factors.

ResNet vs MobileNet On GPUs

SPEAKER_00 6:57

I just leave it at that. So how do we become hardware aware? And the background looks a bit strange. It is not my but uh let me make a statement. And even in this room where we are supposed to be experts, I am quite certain that most of us are still using the wrong model, are still using models that are not entirely a good fit for the hardware. Of course, now I'm gonna trigger a couple of you, and uh, and you will hate me for it, but that's the point. And what's the issue and what can we do? On the edge, there is a wide range of hardware vendors. A wide range of hardware I can choose. And I, as a maybe as a naive developer, how should I know which hardware to choose? How should I know which hardware render fits my model, fits my application best? And on top of that, another challenge is on the edge, there's a wide range of hardware architectures, each of them having unique challenges, unique capabilities, and each of them have a perfect use case and a perfect model that fits it very well. But how do I become hardware aware? And I'm gonna present four steps. Very simplified, but maybe or hopefully all of you get something out of it. And maybe some of the steps may be something you already do today, but I hope some other step may be something you can implement already in your existing pipelines as well. And one of the issues I see, or we see at MBATL, is oftentimes when it comes to a model selection, when it comes to optimizing models, when it comes to we have a use case, we optimize for accuracy first. And then oftentimes the second step, maybe as an afterthought, is to optimize for your hardware and take other KPIs, latency, power consumption, into account. And we should change

TOPS Misleads: GPU vs NPU

SPEAKER_00 9:30

that sentiment. And we should change and become more KPI rare, more hardware rare early on in our development cycle. And how do we do that? We do that by doing profiling early on. We need to measure on real hardware from day one. We need to ensure compatibility. No one wants to develop and train a model for years and then find out that half of its layers are not compatible on the hardware we are going to deploy on. This sounds manual. So the next step is to make it automatic. There's of course the term of premature optimization. So depending on your use case, that's perhaps one thing to consider. But long term, you want to do it in an automatic analog an automatic way. You want to uncover latency and accuracy bottlenecks without having the reliance on experts, without maybe having some sentiment from your even deep learning researchers who may be choosing the mobile network because they really like it. You want to take the human out of the loop. And then finally, to take the loop and come up with optimize with context. Not just make the model smaller, but make the model hardware aware. And especially today, there were quite many presentations on hardware aware optimization techniques. And even I've seen some uh demo tables downstairs and some posts on, for example, hardware air nas, pruning, mixed position, quantization, all those techniques we all know about, but now make them part of the pipeline and of the model selection and optimization uh techniques. I hope I have three minutes left, perfect. I want to finish

Four Steps To Hardware Awareness

SPEAKER_00 11:40

with one example on what we did internally. So, since I'm a researcher at MBETL, I want to finish with some uh research work we have done roughly half a year ago, and then we have hopefully about a couple uh questions left at the end. And since it's not that much time left, I'm gonna go through that a bit quicker. And in this project, we optimize the Lama 3.2 for Qualcomm chip. Specifically, we optimized the Lama 3.2 1 billion, which is by itself already mobile friendly. And we optimized it for the Qualcomm Snapdragon Gen 3A chip. And I guess in this room we all know this chip, otherwise uh it's for example employed in the Galaxy S24 Ultra. And as I said, this model is already mobile friendly, but what can we do on top? How can we make it qualcom friendly in this case? And apart from all compatibility issues, maybe making the model run at all and making a quanti uh quantization of the model, our focus in this project was to apply hardware rare compression and specifically pruning. And without much talk, let's just get to the numbers right away. And through do by making this model not only mobile friendly but Qualcomm friendly, having hardware rare optimization that fits the hardware best. Uh, we made the uh token generation rate roughly 31% faster, prompt processing rate 25% faster, initialization time, how long it takes to get started and to get an output from the network, 126% faster. And all those numbers within or below 1% drop in accuracy. Thank you so much.