WIBTD/W/M: A fool's folly with local AI models

When I originally come up with the ‘What I built this ….’ concept, it was about sharing things I built that I thought might be useful with the community make AI and cloud computing things easier.

This week, it’s a bit different of an article. I’ve been really into local and open models. I think they have a lot of potential and may represent the future of where AI workloads run, in particular at the edge. But I also ran into a lot of struggles in a recent folly, and I’ve chosen to document them here for everyone as learnings I’ve had.

The goal: rebuild the Samsung Tiny Recursive Model pattern with Gemma 4

Say what you want about the goal, but I’ve been fascinated by the Samsung SAIL team’s paper about how a 7 million parameter model can outperform an LLM at problem solving tasks.

So I set out to try to build a similar pattern, but using a 'smaller’ Gemma 4 model with it instead of a custom trained model. In this case, Gemma 4 2B.

The code based development for this actually wasn’t too bad, there are a few gotchas that I’ll highlight, but a lot of my pain came in re-discovering things in local and open AI that became large blockers to success. You can find my code here.

The RAMpocalypse is real

If you’re on Mac, LM Studio is by far the easiest app I’ve found to get up and running with local models. It abstracts a lot of the packaging and runtime choices you need to make.

Getting started with Gemma 4 is equally easy. You download the model, then load it into memory to get started.

Here’s the hard part. All of my machines are either 16GB unified memory (aka Macs) or 16GB RAM mini PC’s. I have one gaming computer with a Radeon 9070 we’ll talk about here at the end, but the point was to be able to run models in my office, and everything except said gaming PC is there.

I quickly discovered that any model north of about 8GB size will make all of these computers come to a crawl. At one point, between my coding IDE, loading a model, and Chrome, I had a 48GB swap that MacOS was valiantly trying to manage, but just couldn’t do it.

Development is simply difficult with a 16GB device, and it’s highly limiting to scaling in communities. I haven’t touched fine tuning much, but at one point it took 90 minutes for a single tuning pass.

Insight: Anything serious with models requires an outsized investment. People who are running things locally today have already invested and have capacity to grow. Those who want to learn will be constrained by the physical limitations of hardware.

Learning: I’m planning to do a lot more in Q3 about how to learn with local models given the limitations I’m discovering to make it more accessible for everyone.

AI Runtimes and configurations remind me of the early Hadoop days

Let’s assume we’ve gotten the hardware problem fixed. The next question is how you’re going to run models, and what configurations matter the most.

Ever heard of ROCm? I hadn’t until going down this path recently. The runtime community is improving rapidly, but the experience is incredibly uneven across hardware types and suites. In a non-NVIDIA stack, the variability and compatibility of software to hardware to models is all over the place (see recent results from people trying to use the newest Intel B70, a value 32GB but with a lot of software compatibility issues).

Want to run a model with MLX on MacOS? The model you download is a different type than GGUF used elsewhere. Want to deploy whatever you built? You’ll have to make sure you understand where it’s going to be run in order for it to be compiled properly.

These are just two choices. If you consider iGPUs and CPU model runtimes, you’re probably going to need llama.cpp but need to learn all the settings.

Coming back to the header, this reminds me when Hadoop was young. You had to know the specific version of file types you used, trade offs of file types, the runtime that was used to build them, and what use case you were actually doing to pick the runtime to use for your workload.

Insight: This market is immature, but luckily it’s evolving quickly. Understand where you’re going to run all this in advance, pick common components wherever possible, and be acutely aware of the tradeoffs. Use LLM’s for research.

Learning: Similar to above, there’s just not much out there for someone to get started or go from 101 to 201. That said, the community answers are really good when you know what you’re looking for, but if you want to try this at home, pick the most popular hardware and software to make your life easy (do as I say, not as I did).

Evaluation is still the big question

Hardware and software learnings aside, I’m coming back to the hardest thing when doing this work I’ve found, which is a good eval dataset and training / test data to support it.

In the recursive models example, I’m using HumanEval, but the model tuning is struggling with the iterations required given hardware constraints, and I find myself asking more philosophical questions about what a ‘good eval’ actually looks like.

At one point, when I did a test, the model changes went from 0% to 100% passed on a sample of evals, but I’ve been peeling that back to make sure there’s no overfit on each pass to a question or test set.

Learning: I’m planning to spend a significant amount of my time in the coming weeks getting in depth with some of the leading and popular evaluation frameworks to build my own understanding of how they work, what they are actually measuring, and the input datasets. For people and organizations building agents and models, this is the critical piece.

Meanwhile, DiffusionGemma is released

While I was doing all of this, Google (my employer) just released DiffusionGemma 4, which is aware of leading and trailing tokens in a prediction. This might have been the best approach to start with over my recursive one, but I’m going to keep going and see how far I can do this.

The next steps in my local AI journey

Now you could probably also ask ‘does this person know what they are even doing?’. It’s probably a fair question. I’m a believer in learning by trying, failing, and trying again. I've found my questions and searches are getting better, and I’m getting precise about issues. But I’m less than 1% in on this journey, and while it’s been incredibly frustrating, it’s also surprisingly rewarding.

Remember that gaming PC I mentioned? Well now I’m looking to get a Radeon R9700. 32GB of vRAM seems to be a good buffer size active parameter models and models below about 16GB in size, and I’m curious how it turns out.

Could I do all of this in the cloud? Yeah, but that’s not really living most developers experiences now, is it?

Hit me up with things you’ve found useful learning about all this so we can all upskill in this process.

$ curl -s -X POST https://davenporter.com/api/agent/chat -H 'Content-Type: application/json' -d '{"prompt": "What is cosm?"}'
ENGINE: vertex-fallback (gemini-3.8-flash) | LATENCY: 284ms
MODEL_CONTEXT: document.modelContext grounded 3 tools [search_davenporter_records, inspect_davenporter_record, summarize_active_record]
AUTH: Google Cloud ADC (davenport-boutique) | STATUS: 200 OK
RECORDS_CITED: ['cosm', 'smartrouter']