If you can’t afford the latest, most expensive hardware, local AI is all about trade-offs. And some trade-offs are more dangerous than others.
Using the best model generally recommended for my era of hardware (Qwen3.8 27B 8-bit on a 64GB M1 Max, Hermes Agent, 64K context, thinking off), it functioned, but slowly.
Almost 3 minutes to respond to a simple “GM” when loaded into Hermes. Much of that time was spent bootstrapping Hermes’s system prompt before it wrote a word, but the harness is what makes the model useful.
Once it had loaded its environment, things started to speed up, but it still took approximately five minutes for a basic readme text file. Nowhere near functional enough to get in the *flow*.
I also tried switching from a standard LM Studio server to oMLX. Same model, different engine. Token generation was about 12.5% faster, but that didn’t translate into faster work through Hermes. The default engine actually finished about 9% quicker overall in my limited tests, not the breakthrough I was looking for.
So I started hunting speed with lesser models and more quantised versions (6-bit, then 4-bit). I made modest incremental gains with each step down. Tests still passed, but needed more course correction, and it was still just slow.
To make any real gains meant switching to mixture-of-experts models (Ornith and Qwen3.6, both around 4x faster on the tasks they completed), but at that point things started getting dangerous…
Ornith got file paths wrong, created files in the wrong place, then attempted a recursive deletion to clean up its own mess. Hermes blocked the deletion, but that’s not something you can leave to chance with an agentic AI.
The uncomfortable truth is if you want to get into a flow state with AI, you’re probably going to need to max out on modern hardware or simply accept that you need to use cloud inference.