# ren **A self-optimizing agentic runtime for small language models** > ren minimizes LLM calls by learning from its own successful executions, converting repeated tasks into deterministic, zero-inference scripts. So the small model is only invoked for genuinely novel work. ## 1. Vision Every existing agentic coding tool — Claude Code, Hermes, Cursor, etc. — is built around the assumption that the model behind it is large, cloud-hosted, and can absorb a 15–20k token system prompt without blinking. That assumption breaks completely on consumer hardware. A 4GB VRAM laptop running Gemma 4B doesn't have room for both the model weights and a novel's worth of instructions in every request. **ren** is an agentic runtime designed from the ground up for small models running locally — 4B-class models on low-end GPUs or even CPU-only laptops. The constraint isn't a limitation to work around; it's the design center. The core insight: **most of the cost in agentic systems isn't the task, it's re-deciding how to do the task every single time.** A model doesn't need to reason from scratch about "how do I check if a URL is reachable" every time a user asks — it needs to reason about it *once*, then remember. ## 2. The Problem | Constraint | Reality on target hardware | |---|---| | VRAM | 4GB typical (integrated / entry discrete GPU) | | System RAM | 4–5GB available for inference workloads | | Model class | ~4B parameters (Gemma 4B, Phi, Qwen 4B-class) | | Context budget | Every token of system prompt is a token *not* available for task context | Existing tools spend 15–20k tokens on system prompt before the user has typed a single word. On a small model with a small context window, this is not a minor inefficiency — it can be the majority of the usable context. It also means slower time-to-first-token on hardware that is already generation-speed constrained. **ren's answer:** keep the system prompt minimal (under ~1.5k tokens), and instead of trying to make the model smarter, make the *system* smarter about when to invoke the model at all. ## 3. Core Design Principles 1. **Minimize LLM calls, not just tokens per call.** The biggest lever isn't prompt compression — it's not calling the LLM at all when a deterministic path already exists. 2. **Small system prompt, always.** No matter how the system grows, the per-request system prompt stays lean. Complexity lives in the architecture, not in prose fed to the model. 3. **Self-improvement is a byproduct of execution, not a separate training loop.** ren doesn't fine-tune anything. It learns by converting its own successful tool-use traces into reusable, generalized scripts. 4. **Deterministic execution is always preferred over inference.** If a task can be reduced to a script, it should be — scripts are faster, cheaper, auditable, and don't hallucinate. 5. **The model is a fallback, not the default.** The steady-state behavior of a mature ren installation should be: most tasks resolve via recipe match, and the LLM is invoked only for genuinely novel requests. 6. **Everything the system learns is inspectable.** Traces and recipes are plain JSON/scripts on disk — a user (or the model itself) can read exactly what ren has learned to do and why. ## 4. High-Level Architecture ``` ┌─────────────────┐ │ User Input │ └────────┬─────────┘ │ ▼ ┌──────────────────────┐ │ Intent Embedder │ ← tiny model, CPU, ms-scale └──────────┬────────────┘ │ ▼ ┌──────────────────────┐ │ Recipe Matcher │ │ (cosine similarity │ │ vs. stored triggers)│ └─────┬──────────┬──────┘ match ≥ threshold │ no confident match │ │ ▼ ▼ ┌────────────────┐ ┌──────────────────────┐ │ Recipe Runner │ │ Agent Loop │ │ (zero LLM call)│ │ (LLM + tool calling) │ └───────┬────────┘ └──────────┬──────────────┘ │ │ ▼ ▼ Result to user Result to user │ ▼ ┌──────────────────────┐ │ Trace Recorder │ │ (saves full run to │ │ traces/*.json) │ └──────────┬──────────────┘ │ success? │ ▼ ┌──────────────────────┐ │ Recipe Extractor │ │ (one LLM call: │ │ generalize trace → │ │ parameterized script) │ └──────────┬──────────────┘ │ ▼ ┌──────────────────────┐ │ Recipe Store │ │ recipes/*.json │ └──────────────────────┘ ``` The system has two loops: - **The fast loop** (top path): user input → embed → match → execute script. No LLM involved. This is the steady state ren converges toward. - **The slow loop** (bottom path): user input → LLM agent with tools → trace recorded → if successful, extracted into a new recipe that joins the fast loop's search space. Over time, the ratio of fast-loop to slow-loop invocations should increase for any given user's workflow, since recurring tasks get captured. ## 5. Component Deep-Dive ### 5.1 Agent Loop (`main.rs`) The core LLM-driven execution engine. Talks to any OpenAI-compatible completions endpoint (`llama.cpp` server, Ollama, LM Studio, or a cloud API — the client doesn't care). **Contract:** ``` 1. Send messages → LLM 2. Push the assistant turn into history (always — even if it only contains tool_calls) 3a. If tool_calls present: execute each, push tool_result messages, go to 1. 3b. If no tool_calls: content is the final answer, terminate. ``` **Key implementation details:** - `ChatMessage` is a single unified struct across all four roles (`system`, `user`, `assistant`, `tool`), with role-specific fields marked optional and skipped on serialization when absent. This keeps the wire format clean while letting the whole conversation live in one `Vec`. - `ToolCall` / `FunctionCall` derive both `Serialize` and `Deserialize` — the model emits them, and they must be re-serialized into the next request's message history. Also `Clone`, since the response is consumed by value while the loop needs to retain it in history. - The assistant turn is **always pushed before** tool result messages. Most backends validate that every `tool_call_id` on a `tool`-role message matches an `id` on the immediately preceding assistant turn's `tool_calls`. Wrong order is a 400 error. - `content` on the response is `Option` — models frequently return `null` content when the turn is tool-calls-only. A bare `String` field would panic on deserialization. - **Tool output truncation** (`MAX_TOOL_OUTPUT = 2000` chars). A single verbose `curl` response can consume the entire remaining context budget on a small model. Truncation is a hard ceiling per tool call, tunable per-tool in the future. - **`AgentOutput` struct** — `run_agent` always returns `Ok(AgentOutput)`, even on non-convergence (hitting `MAX_ITERATIONS`). The outer `Result::Err` is reserved strictly for unrecoverable failures (network down, malformed JSON from the server). This guarantees the message history is *always* available for trace recording, success or failure. **Current tools implemented:** - `run_curl` — HTTP GET/POST to a URL - `run_shell` — arbitrary shell command execution These are intentionally general-purpose placeholders. The long-term tool model is closer to MCP servers / pluggable scripts than a fixed hardcoded set. ### 5.2 Trace Recorder (`trace.rs`) Every agent run — successful or not — is serialized to `traces/_.json`. **Why record failures too:** a failed trace is diagnostic material. It's not (yet) fed into recipe extraction, but it's the raw material for a future "error reflection" pass — recognizing recurring failure patterns and adjusting tool descriptions or the system prompt. **`Trace` schema:** ```json { "id": 1718123456789, "task": "check if wttr.in/delhi is accessible...", "status": "success", "iterations": 2, "duration_ms": 1840, "tools_invoked": ["run_curl"], "messages": [ /* full ChatMessage history */ ], "final_answer": "wttr.in/Delhi is accessible. Currently 38°C.", "error": null } ``` - `tools_invoked` is derived automatically by walking the message list for assistant turns containing `tool_calls`, deduplicated in first-seen order. This lets downstream consumers (the extractor) know a trace's tool dependencies without re-parsing the full conversation. - `id` doubles as both a unique identifier and a natural sort key (Unix ms timestamp). - The full `messages` array is preserved verbatim — this is what makes recipe extraction possible. Nothing is summarized or lost at record time. ### 5.3 Recipe Extractor (`recipe.rs`) Takes a successful `Trace` and makes **one, focused LLM call** to generalize it into a reusable recipe. **Why this has to be a separate, single-purpose call** rather than something the agent does inline: generalization requires stepping back from the specific instance ("Delhi", "wttr.in") to the abstract pattern ("check URL reachability, parameterized by URL"). Asking the agent to do this mid-task would pollute the tool-execution context and cost tokens on every run, not just successful new-pattern runs. **Process:** 1. `render_trace()` converts the raw `ChatMessage` list into readable plain text — `[SYSTEM]`, `[USER]`, `[ASSISTANT — tool calls]`, `[TOOL RESULT]` sections separated by `---`. This is what the extraction LLM actually reads; it's optimized for the model's comprehension, not wire efficiency. 2. A single prompt instructs the model to: - Identify the **high-level goal**, not the specific instance - Replace every concrete value (URLs, filenames, search terms) with `$UPPERCASE_PARAMETER` placeholders - Write a script implementing the generalized task - Produce a one-sentence `trigger` description plus 2–3 `trigger_examples` (alternate phrasings) — this is what will later be embedded for semantic matching 3. The call uses **structured output** (`response_format: json_schema`, strict mode) to constrain the model to a fixed shape. This matters a lot for small models — without a schema, a 4B model is much more likely to add prose commentary around the JSON or drift from the expected keys. 4. `strip_fences()` defensively handles models that wrap JSON in markdown fences despite instructions not to — common failure mode below the ~7B range. **`Recipe` schema:** ```json { "id": 1718123999000, "trigger": "Check whether a given URL is reachable via HTTP", "trigger_examples": [ "is this site up", "check if example.com is accessible", "ping this endpoint for me" ], "parameters": [ { "name": "URL", "description": "The full URL to check" } ], "script": "#!/bin/bash\nset -euo pipefail\n: \"${URL:?URL is required}\"\ncurl -s -o /dev/null -w '%{http_code}' \"$URL\"", "script_type": "shell", "tools_required": ["run_curl"], "source_trace_id": 1718123456789, "created_at": 1718123999000, "run_count": 0, "success_count": 0 } ``` Saved to `recipes/_.json`. `run_count` / `success_count` are pre-wired into the schema for the recipe runner to update — this gives a future confidence-decay mechanism (a recipe that starts failing repeatedly should get deprioritized or flagged for re-extraction) without needing a schema migration later. ### 5.4 Script Language Decision | Language | Verdict | Reasoning | |---|---|---| | **Rust** | ❌ Excluded | Requires compilation — reintroduces the exact latency the recipe system exists to eliminate. | | **Shell** | ✅ Default (~80% of recipes) | Zero interpreter startup beyond the OS shell, universally available, native parameter substitution via env vars, and matches what the agent's own tools (`run_curl`, `run_shell`) already produce. Small models have seen enormous volumes of bash and generate it reliably for this class of task. | | **Python** | ✅ Fallback (~20% of recipes) | Used when a recipe needs JSON parsing, structured data manipulation, or conditional logic that would be unreadable in bash. | Both interpreters read parameters identically — as **environment variables**, not string-interpolated arguments. This sidesteps injection risk entirely and keeps the runner interface uniform across script types: ```rust fn execute_recipe(recipe: &Recipe, params: &HashMap) -> Output { let mut cmd = match recipe.script_type.as_str() { "python" => Command::new("python3"), _ => Command::new("sh"), }; let script_file = write_temp_script(&recipe.script, &recipe.script_type); cmd.arg(&script_file).envs(params); cmd.output() } ``` Generated shell scripts are expected to start with a safety header: ```bash #!/bin/bash set -euo pipefail : "${URL:?URL is required}" ``` `set -euo pipefail` ensures failures propagate instead of silently continuing; the `:` parameter-check pattern gives a clear error if a required env var wasn't supplied. ## 6. Planned Components (Not Yet Built) ### 6.1 Semantic Matcher Embeds incoming user input and compares it against the `trigger` + `trigger_examples` of every stored recipe via cosine similarity. **Open design questions:** - Embedding model: something CPU-fast and small — `all-MiniLM-L6-v2` (~22MB) is the leading candidate. Needs to run in well under 100ms to not undermine the "fast loop" premise. - Vector store: given the target hardware profile, a lightweight embedded option makes more sense than a server — likely a flat in-memory index rebuilt on startup (recipe counts are expected to be in the hundreds/low-thousands, not millions) rather than a full vector DB. `lancedb` or a hand-rolled flat cosine scan are both plausible. - **Threshold tuning is the highest-risk part of the whole system.** A false-positive match — running the wrong recipe with the wrong parameters — is worse than falling back to the LLM. The matcher should be tuned for high precision over high recall, and: - Recipes involving destructive operations (file deletion, overwrites, network mutations) should require either a higher confidence threshold or explicit user confirmation before running, regardless of match score. - Ambiguous matches (multiple recipes above threshold, or a score in a "gray zone") should fall back to the LLM rather than guessing. ### 6.2 Recipe Runner Executes a matched recipe: extracts required parameters from the user's phrasing (likely via a lightweight extraction step, not necessarily the main LLM — could be regex/slot-filling for simple cases, small-model call for complex ones), writes the script to a temp file, executes with `envs()`, captures output, updates `run_count` / `success_count` on the recipe file. **Open question:** how does parameter extraction happen without invoking the main LLM? Options being considered: - A much smaller/cheaper model dedicated to slot-filling - Structured regex/pattern matching keyed off the parameter descriptions - A hybrid: try pattern matching first, fall back to a cheap LLM call only if extraction confidence is low This matters because if parameter extraction always requires an LLM call, the "zero-inference execution" claim weakens — it becomes "one call instead of many" rather than "zero calls." Worth deciding early since it affects the recipe schema (parameters may need extraction hints, e.g. regex patterns, alongside their descriptions). ### 6.3 Failure Reflection Failed traces are currently recorded but unused. The planned second consumer of the trace store: periodically (or on-demand) analyze failure patterns to identify: - Tools that are frequently misused (suggesting a description needs improvement) - Recurring task types the agent struggles with (suggesting a manual recipe should be authored, or the system prompt needs a nudge) This is explicitly lower priority than the success path — the core value proposition depends on the success → recipe pipeline working well first. ### 6.4 Config Currently hardcoded (endpoint URL, model name, `MAX_ITERATIONS`, `MAX_TOOL_OUTPUT`). Needs to move to a `ren.toml` before the codebase grows further — flagged as technical debt, not urgent. ## 7. Technology Choices & Rationale | Choice | Rationale | |---|---| | **Rust** for the runtime | Near-zero process overhead — critical when 4GB RAM is shared between the OS, the model weights, and the runtime itself. Easy FFI into `llama.cpp` if/when local inference moves in-process. Deterministic recipe execution with no GC pauses or interpreter startup cost. | | **OpenAI-compatible API surface** | Works unmodified against `llama.cpp` server, Ollama, LM Studio, or a real cloud endpoint. The model location (local or cloud) is a deployment detail, not an architectural one. | | **JSON Schema structured outputs** for extraction | Small models are meaningfully less reliable at free-form JSON generation. Constraining the output shape at the API level (rather than hoping the prompt is followed) removes an entire failure class. | | **Plain JSON files** for traces/recipes (no database, for now) | Fully inspectable and diffable by a human or the model itself. No schema migration tooling needed yet at expected scale (hundreds of recipes). Revisit if/when recipe counts grow past what a flat directory scan can handle at matcher startup time. | | **Environment variables** for recipe parameters | Uniform interface across shell and Python, avoids string-interpolation injection risk entirely, and matches how both languages naturally expect external input. | ## 8. What "Done" Looks Like (Steady State) A mature ren installation, after some period of use on a given machine/workflow, should exhibit: - The large majority of routine requests resolve via the fast loop (recipe match → script execution), with sub-100ms latency and zero token generation. - The LLM is invoked primarily for genuinely novel requests, complex multi-step reasoning, or tasks that don't decompose cleanly into a deterministic script. - The recipe store is a legible, growing library that a user can audit, edit, or delete from directly — it's not a black box. - System prompt size stays flat regardless of how many recipes exist, since recipes are matched via embedding similarity, not stuffed into context. ## 9. Current Implementation Status | Component | Status | |---|---| | Agentic loop (multi-turn tool calling) | ✅ Built | | Trace recorder | ✅ Built | | Recipe extractor (trace → generalized script) | ✅ Built | | Recipe schema + save/load | ✅ Built | | Semantic matcher | ⏳ Not started | | Recipe runner (execution + param extraction) | ⏳ Not started | | Failure reflection | ⏳ Not started | | Config file (`ren.toml`) | ⏳ Not started | | Tool plugin system (beyond `run_curl` / `run_shell`) | ⏳ Not started | ## 10. Open Questions Tracker These are unresolved decisions flagged during design, kept here so they don't get lost: 1. How is parameter extraction done at recipe-run time without reintroducing a full LLM call per invocation? 2. What embedding model and similarity threshold produce acceptably low false-positive rates for recipe matching? 3. Should destructive recipes require explicit user confirmation on every run, or only above a certain risk classification — and who assigns that classification (the extractor, a fixed tool allowlist, or the user)? 4. At what recipe-store size does a flat directory scan become a real bottleneck, and what's the migration path if so? 5. Should failed traces eventually feed an automated system-prompt/tool-description tuning loop, or stay purely diagnostic for manual review?