From 755fbe7ef26166eee770df626fa6d6a9a6ba996e Mon Sep 17 00:00:00 2001 From: Aditya Gupta Date: Thu, 11 Jun 2026 22:00:00 +0530 Subject: [PATCH] Update README.md --- README.md | 112 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 59 insertions(+), 53 deletions(-) diff --git a/README.md b/README.md index fac0c28..ef9eb41 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # ren -**A self-optimizing agentic runtime for small language models** +**A self-optimizing harness for small language models** > ren minimizes LLM calls by learning from its own successful executions, converting repeated tasks into deterministic, zero-inference scripts. So the small model is only invoked for genuinely novel work. @@ -9,9 +9,21 @@ Every existing agentic coding tool — Claude Code, Hermes, Cursor, etc. — is built around the assumption that the model behind it is large, cloud-hosted, and can absorb a 15–20k token system prompt without blinking. That assumption breaks completely on consumer hardware. A 4GB VRAM laptop running Gemma 4B doesn't have room for both the model weights and a novel's worth of instructions in every request. -**ren** is an agentic runtime designed from the ground up for small models running locally — 4B-class models on low-end GPUs or even CPU-only laptops. The constraint isn't a limitation to work around; it's the design center. +**ren** is a harness — not a runtime, not a framework — built from the ground up for small models running locally: 4B-class models on low-end GPUs or even CPU-only laptops. The constraint isn't a limitation to work around; it's the design center. -The core insight: **most of the cost in agentic systems isn't the task, it's re-deciding how to do the task every single time.** A model doesn't need to reason from scratch about "how do I check if a URL is reachable" every time a user asks — it needs to reason about it *once*, then remember. +### Why "harness" + +The word is deliberate. A runtime executes code. A framework hands you scaffolding and gets out of the way. A **harness** is neither — it's the thing strapped around something with real but limited power, directing that power so none of it is wasted. That's exactly ren's relationship to the model sitting inside it. + +ren doesn't try to make the model smarter. It controls three things the model never sees or touches directly: + +- **When** the model gets invoked at all +- **What** it sees when it is invoked (a lean system prompt, relevant context only — never the whole history) +- **What happens** to a successful run afterward (generalized into a script so the model isn't needed for that shape of task again) + +The model is the engine. ren is the harness deciding when the engine actually needs to turn over, and remembering every route it's already learned to drive on its own. + +The core insight: **most of the cost in agentic systems isn't the task, it's re-deciding how to do the task every single time.** A model doesn't need to reason from scratch about "how do I check if a URL is reachable" every time a user asks — it needs to reason about it *once*, then remember. A harness is what makes that memory possible without turning into a bigger, slower model itself. ## 2. The Problem @@ -21,37 +33,36 @@ The core insight: **most of the cost in agentic systems isn't the task, it's re- | VRAM | 4GB typical (integrated / entry discrete GPU) | | System RAM | 4–5GB available for inference workloads | | Model class | ~4B parameters (Gemma 4B, Phi, Qwen 4B-class) | -| Context budget | Every token of system prompt is a token *not* available for task context | +| Context budget | Every token of system prompt is a token *not* available for task context |Existing tools spend 15–20k tokens on system prompt before the user has typed a single word. On a small model with a small context window, this is not a minor inefficiency — it can be the majority of the usable context. It also means slower time-to-first-token on hardware that is already generation-speed constrained. -Existing tools spend 15–20k tokens on system prompt before the user has typed a single word. On a small model with a small context window, this is not a minor inefficiency — it can be the majority of the usable context. It also means slower time-to-first-token on hardware that is already generation-speed constrained. - -**ren's answer:** keep the system prompt minimal (under ~1.5k tokens), and instead of trying to make the model smarter, make the *system* smarter about when to invoke the model at all. +**ren's answer:** keep the system prompt minimal (under ~1.5k tokens), and instead of trying to make the model smarter, make the *harness* smarter about when to invoke the model at all. ## 3. Core Design Principles 1. **Minimize LLM calls, not just tokens per call.** The biggest lever isn't prompt compression — it's not calling the LLM at all when a deterministic path already exists. -2. **Small system prompt, always.** No matter how the system grows, the per-request system prompt stays lean. Complexity lives in the architecture, not in prose fed to the model. +2. **Small system prompt, always.** No matter how the harness grows, the per-request system prompt stays lean. Complexity lives in the harness's architecture, not in prose fed to the model. 3. **Self-improvement is a byproduct of execution, not a separate training loop.** ren doesn't fine-tune anything. It learns by converting its own successful tool-use traces into reusable, generalized scripts. 4. **Deterministic execution is always preferred over inference.** If a task can be reduced to a script, it should be — scripts are faster, cheaper, auditable, and don't hallucinate. 5. **The model is a fallback, not the default.** The steady-state behavior of a mature ren installation should be: most tasks resolve via recipe match, and the LLM is invoked only for genuinely novel requests. -6. **Everything the system learns is inspectable.** Traces and recipes are plain JSON/scripts on disk — a user (or the model itself) can read exactly what ren has learned to do and why. +6. **The harness is disposable; what it learns is not.** Swap the underlying model, restart the process, move to new hardware — the recipe store is what carries the accumulated value forward, not the harness binary itself. +7. **Everything the harness learns is inspectable.** Traces and recipes are plain JSON/scripts on disk — a user (or the model itself) can read exactly what ren has learned to do and why. ## 4. High-Level Architecture ``` - ┌─────────────────┐ + ┌──────────────────┐ │ User Input │ └────────┬─────────┘ │ ▼ - ┌──────────────────────┐ + ┌───────────────────────┐ │ Intent Embedder │ ← tiny model, CPU, ms-scale └──────────┬────────────┘ │ ▼ - ┌──────────────────────┐ + ┌───────────────────────┐ │ Recipe Matcher │ │ (cosine similarity │ │ vs. stored triggers)│ @@ -59,50 +70,50 @@ Existing tools spend 15–20k tokens on system prompt before the user has typed match ≥ threshold │ no confident match │ │ ▼ ▼ - ┌────────────────┐ ┌──────────────────────┐ + ┌────────────────┐ ┌────────────────────────┐ │ Recipe Runner │ │ Agent Loop │ │ (zero LLM call)│ │ (LLM + tool calling) │ - └───────┬────────┘ └──────────┬──────────────┘ + └───────┬────────┘ └───────────┬────────────┘ │ │ ▼ ▼ Result to user Result to user │ ▼ - ┌──────────────────────┐ + ┌────────────────────────┐ │ Trace Recorder │ │ (saves full run to │ │ traces/*.json) │ - └──────────┬──────────────┘ + └──────────┬─────────────┘ │ success? │ ▼ - ┌──────────────────────┐ + ┌────────────────────────┐ │ Recipe Extractor │ │ (one LLM call: │ │ generalize trace → │ │ parameterized script) │ - └──────────┬──────────────┘ + └──────────┬─────────────┘ │ ▼ - ┌──────────────────────┐ + ┌────────────────────────┐ │ Recipe Store │ │ recipes/*.json │ - └──────────────────────┘ + └────────────────────────┘ ``` -The system has two loops: +The harness runs two loops around the model, never letting the model see both at once: -- **The fast loop** (top path): user input → embed → match → execute script. No LLM involved. This is the steady state ren converges toward. -- **The slow loop** (bottom path): user input → LLM agent with tools → trace recorded → if successful, extracted into a new recipe that joins the fast loop's search space. +- **The fast loop** (top path): user input → embed → match → execute script. No LLM involved. This is the steady state ren converges toward — the harness handling everything itself. +- **The slow loop** (bottom path): user input → LLM agent with tools → trace recorded → if successful, extracted into a new recipe that joins the fast loop's search space. This is the harness *teaching itself* a new fast-loop route. -Over time, the ratio of fast-loop to slow-loop invocations should increase for any given user's workflow, since recurring tasks get captured. +Over time, the ratio of fast-loop to slow-loop invocations should increase for any given user's workflow, since recurring tasks get captured. The harness gets quieter around the model, not louder. ## 5. Component Deep-Dive ### 5.1 Agent Loop (`main.rs`) -The core LLM-driven execution engine. Talks to any OpenAI-compatible completions endpoint (`llama.cpp` server, Ollama, LM Studio, or a cloud API — the client doesn't care). +The part of the harness that actually talks to the model. Speaks to any OpenAI-compatible completions endpoint (`llama.cpp` server, Ollama, LM Studio, or a cloud API) — the harness doesn't care where the model lives, only how it's invoked. **Contract:** @@ -119,19 +130,19 @@ The core LLM-driven execution engine. Talks to any OpenAI-compatible completions - `ToolCall` / `FunctionCall` derive both `Serialize` and `Deserialize` — the model emits them, and they must be re-serialized into the next request's message history. Also `Clone`, since the response is consumed by value while the loop needs to retain it in history. - The assistant turn is **always pushed before** tool result messages. Most backends validate that every `tool_call_id` on a `tool`-role message matches an `id` on the immediately preceding assistant turn's `tool_calls`. Wrong order is a 400 error. - `content` on the response is `Option` — models frequently return `null` content when the turn is tool-calls-only. A bare `String` field would panic on deserialization. -- **Tool output truncation** (`MAX_TOOL_OUTPUT = 2000` chars). A single verbose `curl` response can consume the entire remaining context budget on a small model. Truncation is a hard ceiling per tool call, tunable per-tool in the future. +- **Tool output truncation** (`MAX_TOOL_OUTPUT = 2000` chars). A single verbose `curl` response can consume the entire remaining context budget on a small model. Truncation is a hard ceiling per tool call, tunable per-tool in the future — the harness's job is to protect the model's limited context, not trust every tool to behave. - **`AgentOutput` struct** — `run_agent` always returns `Ok(AgentOutput)`, even on non-convergence (hitting `MAX_ITERATIONS`). The outer `Result::Err` is reserved strictly for unrecoverable failures (network down, malformed JSON from the server). This guarantees the message history is *always* available for trace recording, success or failure. **Current tools implemented:** - `run_curl` — HTTP GET/POST to a URL - `run_shell` — arbitrary shell command execution -These are intentionally general-purpose placeholders. The long-term tool model is closer to MCP servers / pluggable scripts than a fixed hardcoded set. +These are intentionally general-purpose placeholders. The long-term tool model is closer to MCP servers / pluggable scripts than a fixed hardcoded set — the harness's tool registry, not the model's built-in knowledge. ### 5.2 Trace Recorder (`trace.rs`) -Every agent run — successful or not — is serialized to `traces/_.json`. +Every agent run — successful or not — is serialized to `traces/_.json`. This is the harness's memory of what the model actually did, separate from the model's own (nonexistent) memory of it. **Why record failures too:** a failed trace is diagnostic material. It's not (yet) fed into recipe extraction, but it's the raw material for a future "error reflection" pass — recognizing recurring failure patterns and adjusting tool descriptions or the system prompt. @@ -158,9 +169,9 @@ Every agent run — successful or not — is serialized to `traces/__.json`. | Language | Verdict | Reasoning | |---|---|---| -| **Rust** | ❌ Excluded | Requires compilation — reintroduces the exact latency the recipe system exists to eliminate. | -| **Shell** | ✅ Default (~80% of recipes) | Zero interpreter startup beyond the OS shell, universally available, native parameter substitution via env vars, and matches what the agent's own tools (`run_curl`, `run_shell`) already produce. Small models have seen enormous volumes of bash and generate it reliably for this class of task. | -| **Python** | ✅ Fallback (~20% of recipes) | Used when a recipe needs JSON parsing, structured data manipulation, or conditional logic that would be unreadable in bash. | - -Both interpreters read parameters identically — as **environment variables**, not string-interpolated arguments. This sidesteps injection risk entirely and keeps the runner interface uniform across script types: +| **Rust** |Excluded | Requires compilation — reintroduces the exact latency the recipe system exists to eliminate. | +| **Shell** |Default (~80% of recipes) | Zero interpreter startup beyond the OS shell, universally available, native parameter substitution via env vars, and matches what the agent's own tools (`run_curl`, `run_shell`) already produce. Small models have seen enormous volumes of bash and generate it reliably for this class of task. | +| **Python** |Fallback (~20% of recipes) | Used when a recipe needs JSON parsing, structured data manipulation, or conditional logic that would be unreadable in bash. |Both interpreters read parameters identically — as **environment variables**, not string-interpolated arguments. This sidesteps injection risk entirely and keeps the runner interface uniform across script types: ```rust fn execute_recipe(recipe: &Recipe, params: &HashMap) -> Output { @@ -239,12 +248,12 @@ set -euo pipefail ### 6.1 Semantic Matcher -Embeds incoming user input and compares it against the `trigger` + `trigger_examples` of every stored recipe via cosine similarity. +Embeds incoming user input and compares it against the `trigger` + `trigger_examples` of every stored recipe via cosine similarity. This is the harness's gatekeeper — the single decision point that determines whether the model gets invoked at all for a given request. **Open design questions:** - Embedding model: something CPU-fast and small — `all-MiniLM-L6-v2` (~22MB) is the leading candidate. Needs to run in well under 100ms to not undermine the "fast loop" premise. - Vector store: given the target hardware profile, a lightweight embedded option makes more sense than a server — likely a flat in-memory index rebuilt on startup (recipe counts are expected to be in the hundreds/low-thousands, not millions) rather than a full vector DB. `lancedb` or a hand-rolled flat cosine scan are both plausible. -- **Threshold tuning is the highest-risk part of the whole system.** A false-positive match — running the wrong recipe with the wrong parameters — is worse than falling back to the LLM. The matcher should be tuned for high precision over high recall, and: +- **Threshold tuning is the highest-risk part of the whole harness.** A false-positive match — running the wrong recipe with the wrong parameters — is worse than falling back to the LLM. The matcher should be tuned for high precision over high recall, and: - Recipes involving destructive operations (file deletion, overwrites, network mutations) should require either a higher confidence threshold or explicit user confirmation before running, regardless of match score. - Ambiguous matches (multiple recipes above threshold, or a score in a "gray zone") should fall back to the LLM rather than guessing. @@ -269,20 +278,18 @@ This is explicitly lower priority than the success path — the core value propo ### 6.4 Config -Currently hardcoded (endpoint URL, model name, `MAX_ITERATIONS`, `MAX_TOOL_OUTPUT`). Needs to move to a `ren.toml` before the codebase grows further — flagged as technical debt, not urgent. +Currently hardcoded (endpoint URL, model name, `MAX_ITERATIONS`, `MAX_TOOL_OUTPUT`). Needs to move to a `ren.toml` before the harness grows further — flagged as technical debt, not urgent. ## 7. Technology Choices & Rationale | Choice | Rationale | |---|---| -| **Rust** for the runtime | Near-zero process overhead — critical when 4GB RAM is shared between the OS, the model weights, and the runtime itself. Easy FFI into `llama.cpp` if/when local inference moves in-process. Deterministic recipe execution with no GC pauses or interpreter startup cost. | -| **OpenAI-compatible API surface** | Works unmodified against `llama.cpp` server, Ollama, LM Studio, or a real cloud endpoint. The model location (local or cloud) is a deployment detail, not an architectural one. | +| **Rust** for the harness | Near-zero process overhead — critical when 4GB RAM is shared between the OS, the model weights, and the harness itself. Easy FFI into `llama.cpp` if/when local inference moves in-process. Deterministic recipe execution with no GC pauses or interpreter startup cost. | +| **OpenAI-compatible API surface** | Works unmodified against `llama.cpp` server, Ollama, LM Studio, or a real cloud endpoint. The model location (local or cloud) is a deployment detail, not an architectural one — the harness is model-agnostic by design. | | **JSON Schema structured outputs** for extraction | Small models are meaningfully less reliable at free-form JSON generation. Constraining the output shape at the API level (rather than hoping the prompt is followed) removes an entire failure class. | | **Plain JSON files** for traces/recipes (no database, for now) | Fully inspectable and diffable by a human or the model itself. No schema migration tooling needed yet at expected scale (hundreds of recipes). Revisit if/when recipe counts grow past what a flat directory scan can handle at matcher startup time. | | **Environment variables** for recipe parameters | Uniform interface across shell and Python, avoids string-interpolation injection risk entirely, and matches how both languages naturally expect external input. | - - ## 8. What "Done" Looks Like (Steady State) A mature ren installation, after some period of use on a given machine/workflow, should exhibit: @@ -291,23 +298,22 @@ A mature ren installation, after some period of use on a given machine/workflow, - The LLM is invoked primarily for genuinely novel requests, complex multi-step reasoning, or tasks that don't decompose cleanly into a deterministic script. - The recipe store is a legible, growing library that a user can audit, edit, or delete from directly — it's not a black box. - System prompt size stays flat regardless of how many recipes exist, since recipes are matched via embedding similarity, not stuffed into context. +- The harness itself stays thin and swappable — the underlying model can be upgraded or replaced without losing anything the recipe store has already learned. ## 9. Current Implementation Status | Component | Status | |---|---| -| Agentic loop (multi-turn tool calling) | ✅ Built | -| Trace recorder | ✅ Built | -| Recipe extractor (trace → generalized script) | ✅ Built | -| Recipe schema + save/load | ✅ Built | -| Semantic matcher | ⏳ Not started | -| Recipe runner (execution + param extraction) | ⏳ Not started | -| Failure reflection | ⏳ Not started | -| Config file (`ren.toml`) | ⏳ Not started | -| Tool plugin system (beyond `run_curl` / `run_shell`) | ⏳ Not started | - - +| Agentic loop (multi-turn tool calling) |Built | +| Trace recorder |Built | +| Recipe extractor (trace → generalized script) |Built | +| Recipe schema + save/load |Built | +| Semantic matcher | Not started | +| Recipe runner (execution + param extraction) | Not started | +| Failure reflection | Not started | +| Config file (`ren.toml`) | Not started | +| Tool plugin system (beyond `run_curl` / `run_shell`) | Not started | ## 10. Open Questions Tracker These are unresolved decisions flagged during design, kept here so they don't get lost: