325 lines
24 KiB
Markdown
325 lines
24 KiB
Markdown
# ren
|
||
|
||
**A self-optimizing harness for small language models**
|
||
|
||
> ren minimizes LLM calls by learning from its own successful executions, converting repeated tasks into deterministic, zero-inference scripts. So the small model is only invoked for genuinely novel work.
|
||
|
||
|
||
## 1. Vision
|
||
|
||
Every existing agentic coding tool — Claude Code, Hermes, Cursor, etc. — is built around the assumption that the model behind it is large, cloud-hosted, and can absorb a 15–20k token system prompt without blinking. That assumption breaks completely on consumer hardware. A 4GB VRAM laptop running Gemma 4B doesn't have room for both the model weights and a novel's worth of instructions in every request.
|
||
|
||
**ren** is a harness — not a runtime, not a framework — built from the ground up for small models running locally: 4B-class models on low-end GPUs or even CPU-only laptops. The constraint isn't a limitation to work around; it's the design center.
|
||
|
||
### Why "harness"
|
||
|
||
The word is deliberate. A runtime executes code. A framework hands you scaffolding and gets out of the way. A **harness** is neither — it's the thing strapped around something with real but limited power, directing that power so none of it is wasted. That's exactly ren's relationship to the model sitting inside it.
|
||
|
||
ren doesn't try to make the model smarter. It controls three things the model never sees or touches directly:
|
||
|
||
- **When** the model gets invoked at all
|
||
- **What** it sees when it is invoked (a lean system prompt, relevant context only — never the whole history)
|
||
- **What happens** to a successful run afterward (generalized into a script so the model isn't needed for that shape of task again)
|
||
|
||
The model is the engine. ren is the harness deciding when the engine actually needs to turn over, and remembering every route it's already learned to drive on its own.
|
||
|
||
The core insight: **most of the cost in agentic systems isn't the task, it's re-deciding how to do the task every single time.** A model doesn't need to reason from scratch about "how do I check if a URL is reachable" every time a user asks — it needs to reason about it *once*, then remember. A harness is what makes that memory possible without turning into a bigger, slower model itself.
|
||
|
||
|
||
## 2. The Problem
|
||
|
||
| Constraint | Reality on target hardware |
|
||
|---|---|
|
||
| VRAM | 4GB typical (integrated / entry discrete GPU) |
|
||
| System RAM | 4–5GB available for inference workloads |
|
||
| Model class | ~4B parameters (Gemma 4B, Phi, Qwen 4B-class) |
|
||
| Context budget | Every token of system prompt is a token *not* available for task context |Existing tools spend 15–20k tokens on system prompt before the user has typed a single word. On a small model with a small context window, this is not a minor inefficiency — it can be the majority of the usable context. It also means slower time-to-first-token on hardware that is already generation-speed constrained.
|
||
|
||
**ren's answer:** keep the system prompt minimal (under ~1.5k tokens), and instead of trying to make the model smarter, make the *harness* smarter about when to invoke the model at all.
|
||
|
||
|
||
## 3. Core Design Principles
|
||
|
||
1. **Minimize LLM calls, not just tokens per call.** The biggest lever isn't prompt compression — it's not calling the LLM at all when a deterministic path already exists.
|
||
2. **Small system prompt, always.** No matter how the harness grows, the per-request system prompt stays lean. Complexity lives in the harness's architecture, not in prose fed to the model.
|
||
3. **Self-improvement is a byproduct of execution, not a separate training loop.** ren doesn't fine-tune anything. It learns by converting its own successful tool-use traces into reusable, generalized scripts.
|
||
4. **Deterministic execution is always preferred over inference.** If a task can be reduced to a script, it should be — scripts are faster, cheaper, auditable, and don't hallucinate.
|
||
5. **The model is a fallback, not the default.** The steady-state behavior of a mature ren installation should be: most tasks resolve via recipe match, and the LLM is invoked only for genuinely novel requests.
|
||
6. **The harness is disposable; what it learns is not.** Swap the underlying model, restart the process, move to new hardware — the recipe store is what carries the accumulated value forward, not the harness binary itself.
|
||
7. **Everything the harness learns is inspectable.** Traces and recipes are plain JSON/scripts on disk — a user (or the model itself) can read exactly what ren has learned to do and why.
|
||
|
||
|
||
## 4. High-Level Architecture
|
||
|
||
```
|
||
┌──────────────────┐
|
||
│ User Input │
|
||
└────────┬─────────┘
|
||
│
|
||
▼
|
||
┌───────────────────────┐
|
||
│ Intent Embedder │ ← tiny model, CPU, ms-scale
|
||
└──────────┬────────────┘
|
||
│
|
||
▼
|
||
┌───────────────────────┐
|
||
│ Recipe Matcher │
|
||
│ (cosine similarity │
|
||
│ vs. stored triggers)│
|
||
└─────┬──────────┬──────┘
|
||
match ≥ threshold │ no confident match
|
||
│ │
|
||
▼ ▼
|
||
┌────────────────┐ ┌────────────────────────┐
|
||
│ Recipe Runner │ │ Agent Loop │
|
||
│ (zero LLM call)│ │ (LLM + tool calling) │
|
||
└───────┬────────┘ └───────────┬────────────┘
|
||
│ │
|
||
▼ ▼
|
||
Result to user Result to user
|
||
│
|
||
▼
|
||
┌────────────────────────┐
|
||
│ Trace Recorder │
|
||
│ (saves full run to │
|
||
│ traces/*.json) │
|
||
└──────────┬─────────────┘
|
||
│
|
||
success? │
|
||
▼
|
||
┌────────────────────────┐
|
||
│ Recipe Extractor │
|
||
│ (one LLM call: │
|
||
│ generalize trace → │
|
||
│ parameterized script) │
|
||
└──────────┬─────────────┘
|
||
│
|
||
▼
|
||
┌────────────────────────┐
|
||
│ Recipe Store │
|
||
│ recipes/*.json │
|
||
└────────────────────────┘
|
||
```
|
||
|
||
The harness runs two loops around the model, never letting the model see both at once:
|
||
|
||
- **The fast loop** (top path): user input → embed → match → execute script. No LLM involved. This is the steady state ren converges toward — the harness handling everything itself.
|
||
- **The slow loop** (bottom path): user input → LLM agent with tools → trace recorded → if successful, extracted into a new recipe that joins the fast loop's search space. This is the harness *teaching itself* a new fast-loop route.
|
||
|
||
Over time, the ratio of fast-loop to slow-loop invocations should increase for any given user's workflow, since recurring tasks get captured. The harness gets quieter around the model, not louder.
|
||
|
||
|
||
## 5. Component Deep-Dive
|
||
|
||
### 5.1 Agent Loop (`main.rs`)
|
||
|
||
The part of the harness that actually talks to the model. Speaks to any OpenAI-compatible completions endpoint (`llama.cpp` server, Ollama, LM Studio, or a cloud API) — the harness doesn't care where the model lives, only how it's invoked.
|
||
|
||
**Contract:**
|
||
|
||
```
|
||
1. Send messages → LLM
|
||
2. Push the assistant turn into history (always — even if it only contains tool_calls)
|
||
3a. If tool_calls present: execute each, push tool_result messages, go to 1.
|
||
3b. If no tool_calls: content is the final answer, terminate.
|
||
```
|
||
|
||
**Key implementation details:**
|
||
|
||
- `ChatMessage` is a single unified struct across all four roles (`system`, `user`, `assistant`, `tool`), with role-specific fields marked optional and skipped on serialization when absent. This keeps the wire format clean while letting the whole conversation live in one `Vec<ChatMessage>`.
|
||
- `ToolCall` / `FunctionCall` derive both `Serialize` and `Deserialize` — the model emits them, and they must be re-serialized into the next request's message history. Also `Clone`, since the response is consumed by value while the loop needs to retain it in history.
|
||
- The assistant turn is **always pushed before** tool result messages. Most backends validate that every `tool_call_id` on a `tool`-role message matches an `id` on the immediately preceding assistant turn's `tool_calls`. Wrong order is a 400 error.
|
||
- `content` on the response is `Option<String>` — models frequently return `null` content when the turn is tool-calls-only. A bare `String` field would panic on deserialization.
|
||
- **Tool output truncation** (`MAX_TOOL_OUTPUT = 2000` chars). A single verbose `curl` response can consume the entire remaining context budget on a small model. Truncation is a hard ceiling per tool call, tunable per-tool in the future — the harness's job is to protect the model's limited context, not trust every tool to behave.
|
||
- **`AgentOutput` struct** — `run_agent` always returns `Ok(AgentOutput)`, even on non-convergence (hitting `MAX_ITERATIONS`). The outer `Result::Err` is reserved strictly for unrecoverable failures (network down, malformed JSON from the server). This guarantees the message history is *always* available for trace recording, success or failure.
|
||
|
||
**Current tools implemented:**
|
||
- `run_curl` — HTTP GET/POST to a URL
|
||
- `run_shell` — arbitrary shell command execution
|
||
|
||
These are intentionally general-purpose placeholders. The long-term tool model is closer to MCP servers / pluggable scripts than a fixed hardcoded set — the harness's tool registry, not the model's built-in knowledge.
|
||
|
||
|
||
### 5.2 Trace Recorder (`trace.rs`)
|
||
|
||
Every agent run — successful or not — is serialized to `traces/<unix_ms>_<slug>.json`. This is the harness's memory of what the model actually did, separate from the model's own (nonexistent) memory of it.
|
||
|
||
**Why record failures too:** a failed trace is diagnostic material. It's not (yet) fed into recipe extraction, but it's the raw material for a future "error reflection" pass — recognizing recurring failure patterns and adjusting tool descriptions or the system prompt.
|
||
|
||
**`Trace` schema:**
|
||
|
||
```json
|
||
{
|
||
"id": 1718123456789,
|
||
"task": "check if wttr.in/delhi is accessible...",
|
||
"status": "success",
|
||
"iterations": 2,
|
||
"duration_ms": 1840,
|
||
"tools_invoked": ["run_curl"],
|
||
"messages": [ /* full ChatMessage history */ ],
|
||
"final_answer": "wttr.in/Delhi is accessible. Currently 38°C.",
|
||
"error": null
|
||
}
|
||
```
|
||
|
||
- `tools_invoked` is derived automatically by walking the message list for assistant turns containing `tool_calls`, deduplicated in first-seen order. This lets downstream consumers (the extractor) know a trace's tool dependencies without re-parsing the full conversation.
|
||
- `id` doubles as both a unique identifier and a natural sort key (Unix ms timestamp).
|
||
- The full `messages` array is preserved verbatim — this is what makes recipe extraction possible. Nothing is summarized or lost at record time.
|
||
|
||
|
||
### 5.3 Recipe Extractor (`recipe.rs`)
|
||
|
||
Takes a successful `Trace` and makes **one, focused LLM call** to generalize it into a reusable recipe. This is the harness's only "learning" step, and it's deliberately isolated from the main agent loop.
|
||
|
||
**Why this has to be a separate, single-purpose call** rather than something the agent does inline: generalization requires stepping back from the specific instance ("Delhi", "wttr.in") to the abstract pattern ("check URL reachability, parameterized by URL"). Asking the agent to do this mid-task would pollute the tool-execution context and cost tokens on every run, not just successful new-pattern runs. The harness keeps this reflective step entirely out of the model's task-facing context.
|
||
|
||
**Process:**
|
||
|
||
1. `render_trace()` converts the raw `ChatMessage` list into readable plain text — `[SYSTEM]`, `[USER]`, `[ASSISTANT — tool calls]`, `[TOOL RESULT]` sections separated by `---`. This is what the extraction LLM actually reads; it's optimized for the model's comprehension, not wire efficiency.
|
||
2. A single prompt instructs the model to:
|
||
- Identify the **high-level goal**, not the specific instance
|
||
- Replace every concrete value (URLs, filenames, search terms) with `$UPPERCASE_PARAMETER` placeholders
|
||
- Write a script implementing the generalized task
|
||
- Produce a one-sentence `trigger` description plus 2–3 `trigger_examples` (alternate phrasings) — this is what will later be embedded for semantic matching
|
||
3. The call uses **structured output** (`response_format: json_schema`, strict mode) to constrain the model to a fixed shape. This matters a lot for small models — without a schema, a 4B model is much more likely to add prose commentary around the JSON or drift from the expected keys.
|
||
4. `strip_fences()` defensively handles models that wrap JSON in markdown fences despite instructions not to — common failure mode below the ~7B range.
|
||
|
||
**`Recipe` schema:**
|
||
|
||
```json
|
||
{
|
||
"id": 1718123999000,
|
||
"trigger": "Check whether a given URL is reachable via HTTP",
|
||
"trigger_examples": [
|
||
"is this site up",
|
||
"check if example.com is accessible",
|
||
"ping this endpoint for me"
|
||
],
|
||
"parameters": [
|
||
{ "name": "URL", "description": "The full URL to check" }
|
||
],
|
||
"script": "#!/bin/bash\nset -euo pipefail\n: \"${URL:?URL is required}\"\ncurl -s -o /dev/null -w '%{http_code}' \"$URL\"",
|
||
"script_type": "shell",
|
||
"tools_required": ["run_curl"],
|
||
"source_trace_id": 1718123456789,
|
||
"created_at": 1718123999000,
|
||
"run_count": 0,
|
||
"success_count": 0
|
||
}
|
||
```
|
||
|
||
Saved to `recipes/<id>_<slugified-trigger>.json`.
|
||
|
||
`run_count` / `success_count` are pre-wired into the schema for the recipe runner to update — this gives a future confidence-decay mechanism (a recipe that starts failing repeatedly should get deprioritized or flagged for re-extraction) without needing a schema migration later.
|
||
|
||
|
||
### 5.4 Script Language Decision
|
||
|
||
| Language | Verdict | Reasoning |
|
||
|---|---|---|
|
||
| **Rust** |Excluded | Requires compilation — reintroduces the exact latency the recipe system exists to eliminate. |
|
||
| **Shell** |Default (~80% of recipes) | Zero interpreter startup beyond the OS shell, universally available, native parameter substitution via env vars, and matches what the agent's own tools (`run_curl`, `run_shell`) already produce. Small models have seen enormous volumes of bash and generate it reliably for this class of task. |
|
||
| **Python** |Fallback (~20% of recipes) | Used when a recipe needs JSON parsing, structured data manipulation, or conditional logic that would be unreadable in bash. |Both interpreters read parameters identically — as **environment variables**, not string-interpolated arguments. This sidesteps injection risk entirely and keeps the runner interface uniform across script types:
|
||
|
||
```rust
|
||
fn execute_recipe(recipe: &Recipe, params: &HashMap<String, String>) -> Output {
|
||
let mut cmd = match recipe.script_type.as_str() {
|
||
"python" => Command::new("python3"),
|
||
_ => Command::new("sh"),
|
||
};
|
||
let script_file = write_temp_script(&recipe.script, &recipe.script_type);
|
||
cmd.arg(&script_file).envs(params);
|
||
cmd.output()
|
||
}
|
||
```
|
||
|
||
Generated shell scripts are expected to start with a safety header:
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
set -euo pipefail
|
||
: "${URL:?URL is required}"
|
||
```
|
||
|
||
`set -euo pipefail` ensures failures propagate instead of silently continuing; the `:` parameter-check pattern gives a clear error if a required env var wasn't supplied.
|
||
|
||
|
||
## 6. Planned Components (Not Yet Built)
|
||
|
||
### 6.1 Semantic Matcher
|
||
|
||
Embeds incoming user input and compares it against the `trigger` + `trigger_examples` of every stored recipe via cosine similarity. This is the harness's gatekeeper — the single decision point that determines whether the model gets invoked at all for a given request.
|
||
|
||
**Open design questions:**
|
||
- Embedding model: something CPU-fast and small — `all-MiniLM-L6-v2` (~22MB) is the leading candidate. Needs to run in well under 100ms to not undermine the "fast loop" premise.
|
||
- Vector store: given the target hardware profile, a lightweight embedded option makes more sense than a server — likely a flat in-memory index rebuilt on startup (recipe counts are expected to be in the hundreds/low-thousands, not millions) rather than a full vector DB. `lancedb` or a hand-rolled flat cosine scan are both plausible.
|
||
- **Threshold tuning is the highest-risk part of the whole harness.** A false-positive match — running the wrong recipe with the wrong parameters — is worse than falling back to the LLM. The matcher should be tuned for high precision over high recall, and:
|
||
- Recipes involving destructive operations (file deletion, overwrites, network mutations) should require either a higher confidence threshold or explicit user confirmation before running, regardless of match score.
|
||
- Ambiguous matches (multiple recipes above threshold, or a score in a "gray zone") should fall back to the LLM rather than guessing.
|
||
|
||
### 6.2 Recipe Runner
|
||
|
||
Executes a matched recipe: extracts required parameters from the user's phrasing (likely via a lightweight extraction step, not necessarily the main LLM — could be regex/slot-filling for simple cases, small-model call for complex ones), writes the script to a temp file, executes with `envs()`, captures output, updates `run_count` / `success_count` on the recipe file.
|
||
|
||
**Open question:** how does parameter extraction happen without invoking the main LLM? Options being considered:
|
||
- A much smaller/cheaper model dedicated to slot-filling
|
||
- Structured regex/pattern matching keyed off the parameter descriptions
|
||
- A hybrid: try pattern matching first, fall back to a cheap LLM call only if extraction confidence is low
|
||
|
||
This matters because if parameter extraction always requires an LLM call, the "zero-inference execution" claim weakens — it becomes "one call instead of many" rather than "zero calls." Worth deciding early since it affects the recipe schema (parameters may need extraction hints, e.g. regex patterns, alongside their descriptions).
|
||
|
||
### 6.3 Failure Reflection
|
||
|
||
Failed traces are currently recorded but unused. The planned second consumer of the trace store: periodically (or on-demand) analyze failure patterns to identify:
|
||
- Tools that are frequently misused (suggesting a description needs improvement)
|
||
- Recurring task types the agent struggles with (suggesting a manual recipe should be authored, or the system prompt needs a nudge)
|
||
|
||
This is explicitly lower priority than the success path — the core value proposition depends on the success → recipe pipeline working well first.
|
||
|
||
### 6.4 Config
|
||
|
||
Currently hardcoded (endpoint URL, model name, `MAX_ITERATIONS`, `MAX_TOOL_OUTPUT`). Needs to move to a `ren.toml` before the harness grows further — flagged as technical debt, not urgent.
|
||
|
||
|
||
## 7. Technology Choices & Rationale
|
||
|
||
| Choice | Rationale |
|
||
|---|---|
|
||
| **Rust** for the harness | Near-zero process overhead — critical when 4GB RAM is shared between the OS, the model weights, and the harness itself. Easy FFI into `llama.cpp` if/when local inference moves in-process. Deterministic recipe execution with no GC pauses or interpreter startup cost. |
|
||
| **OpenAI-compatible API surface** | Works unmodified against `llama.cpp` server, Ollama, LM Studio, or a real cloud endpoint. The model location (local or cloud) is a deployment detail, not an architectural one — the harness is model-agnostic by design. |
|
||
| **JSON Schema structured outputs** for extraction | Small models are meaningfully less reliable at free-form JSON generation. Constraining the output shape at the API level (rather than hoping the prompt is followed) removes an entire failure class. |
|
||
| **Plain JSON files** for traces/recipes (no database, for now) | Fully inspectable and diffable by a human or the model itself. No schema migration tooling needed yet at expected scale (hundreds of recipes). Revisit if/when recipe counts grow past what a flat directory scan can handle at matcher startup time. |
|
||
| **Environment variables** for recipe parameters | Uniform interface across shell and Python, avoids string-interpolation injection risk entirely, and matches how both languages naturally expect external input. |
|
||
## 8. What "Done" Looks Like (Steady State)
|
||
|
||
A mature ren installation, after some period of use on a given machine/workflow, should exhibit:
|
||
|
||
- The large majority of routine requests resolve via the fast loop (recipe match → script execution), with sub-100ms latency and zero token generation.
|
||
- The LLM is invoked primarily for genuinely novel requests, complex multi-step reasoning, or tasks that don't decompose cleanly into a deterministic script.
|
||
- The recipe store is a legible, growing library that a user can audit, edit, or delete from directly — it's not a black box.
|
||
- System prompt size stays flat regardless of how many recipes exist, since recipes are matched via embedding similarity, not stuffed into context.
|
||
- The harness itself stays thin and swappable — the underlying model can be upgraded or replaced without losing anything the recipe store has already learned.
|
||
|
||
|
||
## 9. Current Implementation Status
|
||
|
||
| Component | Status |
|
||
|---|---|
|
||
| Agentic loop (multi-turn tool calling) |Built |
|
||
| Trace recorder |Built |
|
||
| Recipe extractor (trace → generalized script) |Built |
|
||
| Recipe schema + save/load |Built |
|
||
| Semantic matcher | Not started |
|
||
| Recipe runner (execution + param extraction) | Not started |
|
||
| Failure reflection | Not started |
|
||
| Config file (`ren.toml`) | Not started |
|
||
| Tool plugin system (beyond `run_curl` / `run_shell`) | Not started |
|
||
## 10. Open Questions Tracker
|
||
|
||
These are unresolved decisions flagged during design, kept here so they don't get lost:
|
||
|
||
1. How is parameter extraction done at recipe-run time without reintroducing a full LLM call per invocation?
|
||
2. What embedding model and similarity threshold produce acceptably low false-positive rates for recipe matching?
|
||
3. Should destructive recipes require explicit user confirmation on every run, or only above a certain risk classification — and who assigns that classification (the extractor, a fixed tool allowlist, or the user)?
|
||
4. At what recipe-store size does a flat directory scan become a real bottleneck, and what's the migration path if so?
|
||
5. Should failed traces eventually feed an automated system-prompt/tool-description tuning loop, or stay purely diagnostic for manual review? |