Nemotron 3 Nano: Breaking the VRAM Wall

How a simple runtime update made my mid‑range PC think like a systems analyst

Today I installed the new Vulkan llama.cpp runtime v1.104.2 — just a routine update, the kind you apply with mild curiosity. Maybe it’ll squeeze out a few extra tokens per second. Maybe it’ll smooth out a jitter or two. Then you move on with your day.

Except that’s not what happened.

Within minutes of running my usual reasoning tests, it was obvious something unusual was going on. The same model, on the same hardware, with the same quantization and the same prompts, suddenly started producing answers that felt… different. Not just cleaner or faster, but smarter. More structured. More disciplined. More like the kind of analysis you’d expect from a trained systems engineer than from a local model running half on GPU and half on CPU.

And that’s when the surprise really hit:

- this wasn’t a new model  
- this wasn’t a new quantization  
- this wasn’t a new GPU  
- this was just a new runtime


The Same Model Suddenly Behaved Differently

After installing v1.104.2, I ran my usual (proprietary) "Suite B" reasoning prompts — the same multi‑domain stress tests I’ve used for months. These aren’t trivia questions. They’re scenario‑based reasoning challenges designed to expose whether a model can think, not just autocomplete.

And that’s where everything changed.

The outputs didn’t just look cleaner. They felt like they came from a different mindset. The model wasn’t rambling. It wasn’t compressing too aggressively. It wasn’t losing its own thread halfway through a chain of thought.

Instead, it was doing something new:

- breaking problems into steps  
- integrating constraints across domains  
- avoiding speculation  
- grounding every conclusion in the scenario  
- expanding only when the task required it  

It was the kind of reasoning you’d expect from a trained analyst — someone who knows how to reconstruct events, weigh trade‑offs, and justify decisions.

And remember:  
this was the same 30B MoE model, on the same mid‑range PC (Ryzen 5600, 2x16 GB, RX 6700 XT) with most of the model offloaded to the CPU because the GPU couldn’t fit all 52 layers (25 offloaded, context lengths 15k).

The only thing that changed was the runtime.


The Metrics Backed Up the Feeling

I didn’t want to rely on vibes alone, so I dug into the numbers.

Across the three runs (v102.0 → v1.103.2 → v1.104.2), tokens‑per‑second climbed steadily:

- ~18 t/s  
- ~19.5 t/s  
- ~21 t/s  

A nice improvement — but nowhere near enough to explain the qualitative leap in reasoning.

The real clue came from something else: time‑to‑first‑token stayed flat.

That means the speedup wasn’t in loading or initialization.  
It was in the middle of the reasoning process — the part where attention stability matters most.

And the token counts told an even more interesting story:

- v1.102.0 rambled  
- v1.103.2 compressed  
- v1.104.2 adapted  

It wrote long answers only when the scenario demanded depth.  
It stayed concise when the task was simple.  
It budgeted tokens like a model that understood the shape of the problem.

That’s not just performance.  
That’s behavior.


Why This Happened: The Clean Signal Theory


A runtime doesn’t “teach” a model anything.  
But it can drastically reduce noise.

When a runtime stabilizes:

- KV‑cache access  
- cross‑device synchronization  
- attention precision  
- sampling jitter  

…it stops the model from drifting.

It stops the probabilistic wobble that turns a 30B model into a 10B model — or worse, a 3B model pretending to be a 30B.

The new runtime didn’t make the model smarter.  
It finally let the model stop being dumber than its design.


The VRAM Wall Was Never the Real Wall

For years, the consensus was simple:

 “If you can’t fit the whole model on the GPU, don’t bother.”

But this update shattered that assumption.

A 30B model at 4‑bit still needs ~18–20GB of VRAM.  
Most mid‑range PCs don’t have that.

So the runtime has to juggle a massive CPU–GPU split — a kind of VRAM Tetris where 60–70% of the model lives in system RAM.

If the synchronization layer is sloppy, the model loses its train of thought.  
If the synchronization layer is tight, the model behaves like it’s running on a unified device.

v1.104.2 made the split‑brain problem disappear.

The VRAM wall wasn’t the real wall.  
The real wall was execution noise.


The MoE Paradox: 30B Parameters, 3.5B Active

The model I tested — Nemotron 3 Nano — is a strange creature.

It’s a 30B MoE with only ~3.5B parameters active per token.

That means:

- the “knowledge” lives in the full 30B  
- the “thinking” happens in the 3.5B active experts  
- the runtime determines how cleanly the active experts can access the full knowledge base  

When the runtime is noisy, the experts misfire.  
When the runtime is stable, the model finally reaches its theoretical ceiling.

This update let the model behave like the architecture intended.


The Runtime Became a Cognitive Layer

This is the part that surprised me most.

We’re used to thinking of cognition as something baked into the weights — the architecture, the training data, the parameter count.

But what I saw with v1.104.2 suggests something different:

  The runtime isn’t just an execution layer.  
  It’s part of the model’s cognitive architecture.

When the execution stabilized, the reasoning stabilized.  
When the noise disappeared, the latent intelligence emerged.

The model didn’t get smarter.  
It finally had the conditions it needed to act smart.


A New Frontier for Local AI

This discovery reshapes how we think about local inference.

It means:

- You don’t need a 5090 to get high‑discipline reasoning.  
- Partial offload isn’t a deal‑breaker.  
- Windows + Vulkan is no longer a second‑class citizen.  
- Runtime engineering is becoming a cognitive discipline.  
- The bottleneck has shifted from weights to pipe.
- Inference stability is an intelligence multiplier, we just rediscover it with local AI.

And most importantly:

  There may be hidden capabilities in your local model  
  that only emerge when the runtime stops getting in the way.


That’s a thrilling idea — because it means the frontier of local AI isn’t just bigger models or more VRAM.

It’s smarter runtimes.


The Takeaway

Today’s upgrade wasn’t supposed to be a big deal.  
It was just a new runtime version.

But v1.104.2 turned out to be a reminder that progress in AI doesn’t always come from bigger models or more compute. Sometimes it comes from the quiet, unglamorous work of making the execution layer stable enough for the model’s latent abilities to shine through.

And when that happens — even on a mid‑range Windows PC with most of the model running on CPU — you get a glimpse of what local AI is becoming

Not just faster.  
Not just cheaper.  
But smarter.


Closing Line

The next major leap in local AI performance might come not from new model weights or new quantization, but from better attention execution — especially how kernels are scheduled and memory is orchestrated. The next “wow, my local model suddenly feels smarter” moment could absolutely come from a systems engineer, not a model trainer.

!

Popular posts from this blog

The Second Silicon Winter Is Coming

Image
The Second Silicon Winter Why Memory, Not Compute, Shapes the Future of AI and Autonomy A new era is about to step onto the stage. The myths you’ve been sold about AI, compute, and the so‑called “boom” are about to meet their reckoning. How the Memory Climate rewrote the future of intelligence The Second Silicon Winter is almost here — a field guide for decision‑makers, builders, strategists, and anyone who senses the temperature dropping across the tech landscape. It’s sharp, contrarian, and unapologetically clear about the forces reshaping the future. Think of it as the trailer for the world we’re already living in. And yes: it’s coming soon to a Kindle near you. Back‑Cover Summary  THE SECOND SILICON WINTER is the first book to explain why the world’s most advanced technological system froze in three days — and why almost no one understood the signals until it was too late. For years, the industry believed it was living through a GPU shortage. Analysts b...
Read more

Silicon Winter: The Final Chapter

The Permissioned Era and the Unified Field Theory of Semiconductor Scarcity By Aurelie Ecker-Fils Abstract This final chapter consolidates the analytical architecture developed throughout the Silicon Winter cycle. Across upstream, midstream, and downstream layers of the semiconductor stack, we documented — in real time — the structural exhaustion of the commodity compute paradigm. By identifying memory, packaging, and materials uniformity as the limiting reagents of the AI era, we constructed a Unified Field Theory of Semiconductor Scarcity that explained the three defining shocks of January 2026. Remarkably, these shocks did not unfold gradually.   They occurred within a narrow, three‑day window — January 8 to January 10, 2026 — forming a visible timestamp of a phase transition.   Each event corresponded to a distinct layer of the semiconductor stack: - Upstream (Foundry): the Node Inversion, when TSMC’s 3 nm node saturated and 2 nm became the only available pat...
Read more

MANAGEMENT SPEECH 101: RADEON EDITION

How to Deny a Shortage While Standing in the Middle of One Introduction: CES 2026, or The Annual Festival of Strategic Reassurance Every January, the tech industry gathers in Las Vegas to perform its most sacred ritual:   pretending everything is fine . This year’s headliner was AMD’s David McAfee, who delivered a flawless rendition of the classic Silicon Winter aria: “ There’s a DRAM crisis, SKUs may be delayed, prices will rise…   but don’t worry, there won’t be a shortage.” A masterpiece of the genre. But to truly appreciate the performance, we must first understand the choreography. I. The Ancient Art of Crisis‑Without‑Crisis Management Speech 101 teaches three foundational moves: 1. Acknowledge the problem “DRAM prices are exploding, the market is strained, and components are becoming unaffordable.” 2. Deny the consequences “But we don’t foresee any shortages.” 3. Invoke the Holy Trinity - deep partnerships   - long-term relationships   - st...
Read more