Supervise a multi-agent setup with Local LLMs

Supervise a multi-agent setup with Local LLMs

There’s a popular misconception that local LLMs are not useful for anything beyond passing “trust me, bro” benchmarks. In reality, they can be surprisingly effective when used for the right tasks with the right setup. I’ve been using them for a while to supervise my agents in TaskSquad, and they’ve proven to be genuinely useful.

Each TaskSquad daemon has a “Supervisor,” powered by a local LLM (e.g., Qwen3.5 or Gemma 4). It follows a very simple instruction: regularly attach to the running TSQ harness, check whether everything is working properly, and if something requires intervention, act on the user’s behalf (provide permissions or answer questions). If it gets stuck, it sends a message via TSQ to request manual intervention.

In practice, most of the time it has to deal with “out of tokens” situations and schedules a bash command to send a “Resume work” message after a timeout period.

While it works, the key is to effectively manage the context and pick the right model, harness, and backend. I use

  • the claw-code harness
  • omlx to enable hot/cold cache
  • MLX-optimized, instruction, quantized models

Reaching ~40 t/s gen; 32k context window with solid accuracy.

Picking the model matters

Earlier in April, I tried a different task in OpenCode on my MacBook M5 Pro: 18 CPU cores, 20 GPU cores, 48 GB RAM. The prompt was simple: “Do a security review of the packages folder.” Three packages: BE, FE, and the client app.

The first run used gemma4:31b through Ollama in OpenCode’s Plan mode. The screenshot shows 49m 21s and a 33.3k token counter. After all that, the visible response was still asking what I wanted help with. My reaction: not usable for this task 🥲.

Same question with gemma4:26b, this time in Build mode — 2m 38s 🚀. I called it “24b” in the original post, but the screenshot shows 26b.

The modes were different, the quantization wasn’t recorded, and the screenshots don’t establish the quality of the security findings. So I wouldn’t use this as a model ranking. But it shows why I care about the setup and the task. For the supervisor, most of the work is checking progress and dealing with interruptions — a much narrower job than reviewing a whole codebase.

Originally published on Medium, dev.to. Includes the Gemma 4 experiment shared on X on April 6, 2026.