Update README.md
This commit is contained in:
112
README.md
112
README.md
@@ -1,6 +1,6 @@
|
||||
# ren
|
||||
|
||||
**A self-optimizing agentic runtime for small language models**
|
||||
**A self-optimizing harness for small language models**
|
||||
|
||||
> ren minimizes LLM calls by learning from its own successful executions, converting repeated tasks into deterministic, zero-inference scripts. So the small model is only invoked for genuinely novel work.
|
||||
|
||||
@@ -9,9 +9,21 @@
|
||||
|
||||
Every existing agentic coding tool — Claude Code, Hermes, Cursor, etc. — is built around the assumption that the model behind it is large, cloud-hosted, and can absorb a 15–20k token system prompt without blinking. That assumption breaks completely on consumer hardware. A 4GB VRAM laptop running Gemma 4B doesn't have room for both the model weights and a novel's worth of instructions in every request.
|
||||
|
||||
**ren** is an agentic runtime designed from the ground up for small models running locally — 4B-class models on low-end GPUs or even CPU-only laptops. The constraint isn't a limitation to work around; it's the design center.
|
||||
**ren** is a harness — not a runtime, not a framework — built from the ground up for small models running locally: 4B-class models on low-end GPUs or even CPU-only laptops. The constraint isn't a limitation to work around; it's the design center.
|
||||
|
||||
The core insight: **most of the cost in agentic systems isn't the task, it's re-deciding how to do the task every single time.** A model doesn't need to reason from scratch about "how do I check if a URL is reachable" every time a user asks — it needs to reason about it *once*, then remember.
|
||||
### Why "harness"
|
||||
|
||||
The word is deliberate. A runtime executes code. A framework hands you scaffolding and gets out of the way. A **harness** is neither — it's the thing strapped around something with real but limited power, directing that power so none of it is wasted. That's exactly ren's relationship to the model sitting inside it.
|
||||
|
||||
ren doesn't try to make the model smarter. It controls three things the model never sees or touches directly:
|
||||
|
||||
- **When** the model gets invoked at all
|
||||
- **What** it sees when it is invoked (a lean system prompt, relevant context only — never the whole history)
|
||||
- **What happens** to a successful run afterward (generalized into a script so the model isn't needed for that shape of task again)
|
||||
|
||||
The model is the engine. ren is the harness deciding when the engine actually needs to turn over, and remembering every route it's already learned to drive on its own.
|
||||
|
||||
The core insight: **most of the cost in agentic systems isn't the task, it's re-deciding how to do the task every single time.** A model doesn't need to reason from scratch about "how do I check if a URL is reachable" every time a user asks — it needs to reason about it *once*, then remember. A harness is what makes that memory possible without turning into a bigger, slower model itself.
|
||||
|
||||
|
||||
## 2. The Problem
|
||||
@@ -21,37 +33,36 @@ The core insight: **most of the cost in agentic systems isn't the task, it's re-
|
||||
| VRAM | 4GB typical (integrated / entry discrete GPU) |
|
||||
| System RAM | 4–5GB available for inference workloads |
|
||||
| Model class | ~4B parameters (Gemma 4B, Phi, Qwen 4B-class) |
|
||||
| Context budget | Every token of system prompt is a token *not* available for task context |
|
||||
| Context budget | Every token of system prompt is a token *not* available for task context |Existing tools spend 15–20k tokens on system prompt before the user has typed a single word. On a small model with a small context window, this is not a minor inefficiency — it can be the majority of the usable context. It also means slower time-to-first-token on hardware that is already generation-speed constrained.
|
||||
|
||||
Existing tools spend 15–20k tokens on system prompt before the user has typed a single word. On a small model with a small context window, this is not a minor inefficiency — it can be the majority of the usable context. It also means slower time-to-first-token on hardware that is already generation-speed constrained.
|
||||
|
||||
**ren's answer:** keep the system prompt minimal (under ~1.5k tokens), and instead of trying to make the model smarter, make the *system* smarter about when to invoke the model at all.
|
||||
**ren's answer:** keep the system prompt minimal (under ~1.5k tokens), and instead of trying to make the model smarter, make the *harness* smarter about when to invoke the model at all.
|
||||
|
||||
|
||||
## 3. Core Design Principles
|
||||
|
||||
1. **Minimize LLM calls, not just tokens per call.** The biggest lever isn't prompt compression — it's not calling the LLM at all when a deterministic path already exists.
|
||||
2. **Small system prompt, always.** No matter how the system grows, the per-request system prompt stays lean. Complexity lives in the architecture, not in prose fed to the model.
|
||||
2. **Small system prompt, always.** No matter how the harness grows, the per-request system prompt stays lean. Complexity lives in the harness's architecture, not in prose fed to the model.
|
||||
3. **Self-improvement is a byproduct of execution, not a separate training loop.** ren doesn't fine-tune anything. It learns by converting its own successful tool-use traces into reusable, generalized scripts.
|
||||
4. **Deterministic execution is always preferred over inference.** If a task can be reduced to a script, it should be — scripts are faster, cheaper, auditable, and don't hallucinate.
|
||||
5. **The model is a fallback, not the default.** The steady-state behavior of a mature ren installation should be: most tasks resolve via recipe match, and the LLM is invoked only for genuinely novel requests.
|
||||
6. **Everything the system learns is inspectable.** Traces and recipes are plain JSON/scripts on disk — a user (or the model itself) can read exactly what ren has learned to do and why.
|
||||
6. **The harness is disposable; what it learns is not.** Swap the underlying model, restart the process, move to new hardware — the recipe store is what carries the accumulated value forward, not the harness binary itself.
|
||||
7. **Everything the harness learns is inspectable.** Traces and recipes are plain JSON/scripts on disk — a user (or the model itself) can read exactly what ren has learned to do and why.
|
||||
|
||||
|
||||
## 4. High-Level Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
┌──────────────────┐
|
||||
│ User Input │
|
||||
└────────┬─────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
┌───────────────────────┐
|
||||
│ Intent Embedder │ ← tiny model, CPU, ms-scale
|
||||
└──────────┬────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
┌───────────────────────┐
|
||||
│ Recipe Matcher │
|
||||
│ (cosine similarity │
|
||||
│ vs. stored triggers)│
|
||||
@@ -59,50 +70,50 @@ Existing tools spend 15–20k tokens on system prompt before the user has typed
|
||||
match ≥ threshold │ no confident match
|
||||
│ │
|
||||
▼ ▼
|
||||
┌────────────────┐ ┌──────────────────────┐
|
||||
┌────────────────┐ ┌────────────────────────┐
|
||||
│ Recipe Runner │ │ Agent Loop │
|
||||
│ (zero LLM call)│ │ (LLM + tool calling) │
|
||||
└───────┬────────┘ └──────────┬──────────────┘
|
||||
└───────┬────────┘ └───────────┬────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
Result to user Result to user
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
┌────────────────────────┐
|
||||
│ Trace Recorder │
|
||||
│ (saves full run to │
|
||||
│ traces/*.json) │
|
||||
└──────────┬──────────────┘
|
||||
└──────────┬─────────────┘
|
||||
│
|
||||
success? │
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
┌────────────────────────┐
|
||||
│ Recipe Extractor │
|
||||
│ (one LLM call: │
|
||||
│ generalize trace → │
|
||||
│ parameterized script) │
|
||||
└──────────┬──────────────┘
|
||||
└──────────┬─────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
┌────────────────────────┐
|
||||
│ Recipe Store │
|
||||
│ recipes/*.json │
|
||||
└──────────────────────┘
|
||||
└────────────────────────┘
|
||||
```
|
||||
|
||||
The system has two loops:
|
||||
The harness runs two loops around the model, never letting the model see both at once:
|
||||
|
||||
- **The fast loop** (top path): user input → embed → match → execute script. No LLM involved. This is the steady state ren converges toward.
|
||||
- **The slow loop** (bottom path): user input → LLM agent with tools → trace recorded → if successful, extracted into a new recipe that joins the fast loop's search space.
|
||||
- **The fast loop** (top path): user input → embed → match → execute script. No LLM involved. This is the steady state ren converges toward — the harness handling everything itself.
|
||||
- **The slow loop** (bottom path): user input → LLM agent with tools → trace recorded → if successful, extracted into a new recipe that joins the fast loop's search space. This is the harness *teaching itself* a new fast-loop route.
|
||||
|
||||
Over time, the ratio of fast-loop to slow-loop invocations should increase for any given user's workflow, since recurring tasks get captured.
|
||||
Over time, the ratio of fast-loop to slow-loop invocations should increase for any given user's workflow, since recurring tasks get captured. The harness gets quieter around the model, not louder.
|
||||
|
||||
|
||||
## 5. Component Deep-Dive
|
||||
|
||||
### 5.1 Agent Loop (`main.rs`)
|
||||
|
||||
The core LLM-driven execution engine. Talks to any OpenAI-compatible completions endpoint (`llama.cpp` server, Ollama, LM Studio, or a cloud API — the client doesn't care).
|
||||
The part of the harness that actually talks to the model. Speaks to any OpenAI-compatible completions endpoint (`llama.cpp` server, Ollama, LM Studio, or a cloud API) — the harness doesn't care where the model lives, only how it's invoked.
|
||||
|
||||
**Contract:**
|
||||
|
||||
@@ -119,19 +130,19 @@ The core LLM-driven execution engine. Talks to any OpenAI-compatible completions
|
||||
- `ToolCall` / `FunctionCall` derive both `Serialize` and `Deserialize` — the model emits them, and they must be re-serialized into the next request's message history. Also `Clone`, since the response is consumed by value while the loop needs to retain it in history.
|
||||
- The assistant turn is **always pushed before** tool result messages. Most backends validate that every `tool_call_id` on a `tool`-role message matches an `id` on the immediately preceding assistant turn's `tool_calls`. Wrong order is a 400 error.
|
||||
- `content` on the response is `Option<String>` — models frequently return `null` content when the turn is tool-calls-only. A bare `String` field would panic on deserialization.
|
||||
- **Tool output truncation** (`MAX_TOOL_OUTPUT = 2000` chars). A single verbose `curl` response can consume the entire remaining context budget on a small model. Truncation is a hard ceiling per tool call, tunable per-tool in the future.
|
||||
- **Tool output truncation** (`MAX_TOOL_OUTPUT = 2000` chars). A single verbose `curl` response can consume the entire remaining context budget on a small model. Truncation is a hard ceiling per tool call, tunable per-tool in the future — the harness's job is to protect the model's limited context, not trust every tool to behave.
|
||||
- **`AgentOutput` struct** — `run_agent` always returns `Ok(AgentOutput)`, even on non-convergence (hitting `MAX_ITERATIONS`). The outer `Result::Err` is reserved strictly for unrecoverable failures (network down, malformed JSON from the server). This guarantees the message history is *always* available for trace recording, success or failure.
|
||||
|
||||
**Current tools implemented:**
|
||||
- `run_curl` — HTTP GET/POST to a URL
|
||||
- `run_shell` — arbitrary shell command execution
|
||||
|
||||
These are intentionally general-purpose placeholders. The long-term tool model is closer to MCP servers / pluggable scripts than a fixed hardcoded set.
|
||||
These are intentionally general-purpose placeholders. The long-term tool model is closer to MCP servers / pluggable scripts than a fixed hardcoded set — the harness's tool registry, not the model's built-in knowledge.
|
||||
|
||||
|
||||
### 5.2 Trace Recorder (`trace.rs`)
|
||||
|
||||
Every agent run — successful or not — is serialized to `traces/<unix_ms>_<slug>.json`.
|
||||
Every agent run — successful or not — is serialized to `traces/<unix_ms>_<slug>.json`. This is the harness's memory of what the model actually did, separate from the model's own (nonexistent) memory of it.
|
||||
|
||||
**Why record failures too:** a failed trace is diagnostic material. It's not (yet) fed into recipe extraction, but it's the raw material for a future "error reflection" pass — recognizing recurring failure patterns and adjusting tool descriptions or the system prompt.
|
||||
|
||||
@@ -158,9 +169,9 @@ Every agent run — successful or not — is serialized to `traces/<unix_ms>_<sl
|
||||
|
||||
### 5.3 Recipe Extractor (`recipe.rs`)
|
||||
|
||||
Takes a successful `Trace` and makes **one, focused LLM call** to generalize it into a reusable recipe.
|
||||
Takes a successful `Trace` and makes **one, focused LLM call** to generalize it into a reusable recipe. This is the harness's only "learning" step, and it's deliberately isolated from the main agent loop.
|
||||
|
||||
**Why this has to be a separate, single-purpose call** rather than something the agent does inline: generalization requires stepping back from the specific instance ("Delhi", "wttr.in") to the abstract pattern ("check URL reachability, parameterized by URL"). Asking the agent to do this mid-task would pollute the tool-execution context and cost tokens on every run, not just successful new-pattern runs.
|
||||
**Why this has to be a separate, single-purpose call** rather than something the agent does inline: generalization requires stepping back from the specific instance ("Delhi", "wttr.in") to the abstract pattern ("check URL reachability, parameterized by URL"). Asking the agent to do this mid-task would pollute the tool-execution context and cost tokens on every run, not just successful new-pattern runs. The harness keeps this reflective step entirely out of the model's task-facing context.
|
||||
|
||||
**Process:**
|
||||
|
||||
@@ -206,11 +217,9 @@ Saved to `recipes/<id>_<slugified-trigger>.json`.
|
||||
|
||||
| Language | Verdict | Reasoning |
|
||||
|---|---|---|
|
||||
| **Rust** | ❌ Excluded | Requires compilation — reintroduces the exact latency the recipe system exists to eliminate. |
|
||||
| **Shell** | ✅ Default (~80% of recipes) | Zero interpreter startup beyond the OS shell, universally available, native parameter substitution via env vars, and matches what the agent's own tools (`run_curl`, `run_shell`) already produce. Small models have seen enormous volumes of bash and generate it reliably for this class of task. |
|
||||
| **Python** | ✅ Fallback (~20% of recipes) | Used when a recipe needs JSON parsing, structured data manipulation, or conditional logic that would be unreadable in bash. |
|
||||
|
||||
Both interpreters read parameters identically — as **environment variables**, not string-interpolated arguments. This sidesteps injection risk entirely and keeps the runner interface uniform across script types:
|
||||
| **Rust** |Excluded | Requires compilation — reintroduces the exact latency the recipe system exists to eliminate. |
|
||||
| **Shell** |Default (~80% of recipes) | Zero interpreter startup beyond the OS shell, universally available, native parameter substitution via env vars, and matches what the agent's own tools (`run_curl`, `run_shell`) already produce. Small models have seen enormous volumes of bash and generate it reliably for this class of task. |
|
||||
| **Python** |Fallback (~20% of recipes) | Used when a recipe needs JSON parsing, structured data manipulation, or conditional logic that would be unreadable in bash. |Both interpreters read parameters identically — as **environment variables**, not string-interpolated arguments. This sidesteps injection risk entirely and keeps the runner interface uniform across script types:
|
||||
|
||||
```rust
|
||||
fn execute_recipe(recipe: &Recipe, params: &HashMap<String, String>) -> Output {
|
||||
@@ -239,12 +248,12 @@ set -euo pipefail
|
||||
|
||||
### 6.1 Semantic Matcher
|
||||
|
||||
Embeds incoming user input and compares it against the `trigger` + `trigger_examples` of every stored recipe via cosine similarity.
|
||||
Embeds incoming user input and compares it against the `trigger` + `trigger_examples` of every stored recipe via cosine similarity. This is the harness's gatekeeper — the single decision point that determines whether the model gets invoked at all for a given request.
|
||||
|
||||
**Open design questions:**
|
||||
- Embedding model: something CPU-fast and small — `all-MiniLM-L6-v2` (~22MB) is the leading candidate. Needs to run in well under 100ms to not undermine the "fast loop" premise.
|
||||
- Vector store: given the target hardware profile, a lightweight embedded option makes more sense than a server — likely a flat in-memory index rebuilt on startup (recipe counts are expected to be in the hundreds/low-thousands, not millions) rather than a full vector DB. `lancedb` or a hand-rolled flat cosine scan are both plausible.
|
||||
- **Threshold tuning is the highest-risk part of the whole system.** A false-positive match — running the wrong recipe with the wrong parameters — is worse than falling back to the LLM. The matcher should be tuned for high precision over high recall, and:
|
||||
- **Threshold tuning is the highest-risk part of the whole harness.** A false-positive match — running the wrong recipe with the wrong parameters — is worse than falling back to the LLM. The matcher should be tuned for high precision over high recall, and:
|
||||
- Recipes involving destructive operations (file deletion, overwrites, network mutations) should require either a higher confidence threshold or explicit user confirmation before running, regardless of match score.
|
||||
- Ambiguous matches (multiple recipes above threshold, or a score in a "gray zone") should fall back to the LLM rather than guessing.
|
||||
|
||||
@@ -269,20 +278,18 @@ This is explicitly lower priority than the success path — the core value propo
|
||||
|
||||
### 6.4 Config
|
||||
|
||||
Currently hardcoded (endpoint URL, model name, `MAX_ITERATIONS`, `MAX_TOOL_OUTPUT`). Needs to move to a `ren.toml` before the codebase grows further — flagged as technical debt, not urgent.
|
||||
Currently hardcoded (endpoint URL, model name, `MAX_ITERATIONS`, `MAX_TOOL_OUTPUT`). Needs to move to a `ren.toml` before the harness grows further — flagged as technical debt, not urgent.
|
||||
|
||||
|
||||
## 7. Technology Choices & Rationale
|
||||
|
||||
| Choice | Rationale |
|
||||
|---|---|
|
||||
| **Rust** for the runtime | Near-zero process overhead — critical when 4GB RAM is shared between the OS, the model weights, and the runtime itself. Easy FFI into `llama.cpp` if/when local inference moves in-process. Deterministic recipe execution with no GC pauses or interpreter startup cost. |
|
||||
| **OpenAI-compatible API surface** | Works unmodified against `llama.cpp` server, Ollama, LM Studio, or a real cloud endpoint. The model location (local or cloud) is a deployment detail, not an architectural one. |
|
||||
| **Rust** for the harness | Near-zero process overhead — critical when 4GB RAM is shared between the OS, the model weights, and the harness itself. Easy FFI into `llama.cpp` if/when local inference moves in-process. Deterministic recipe execution with no GC pauses or interpreter startup cost. |
|
||||
| **OpenAI-compatible API surface** | Works unmodified against `llama.cpp` server, Ollama, LM Studio, or a real cloud endpoint. The model location (local or cloud) is a deployment detail, not an architectural one — the harness is model-agnostic by design. |
|
||||
| **JSON Schema structured outputs** for extraction | Small models are meaningfully less reliable at free-form JSON generation. Constraining the output shape at the API level (rather than hoping the prompt is followed) removes an entire failure class. |
|
||||
| **Plain JSON files** for traces/recipes (no database, for now) | Fully inspectable and diffable by a human or the model itself. No schema migration tooling needed yet at expected scale (hundreds of recipes). Revisit if/when recipe counts grow past what a flat directory scan can handle at matcher startup time. |
|
||||
| **Environment variables** for recipe parameters | Uniform interface across shell and Python, avoids string-interpolation injection risk entirely, and matches how both languages naturally expect external input. |
|
||||
|
||||
|
||||
## 8. What "Done" Looks Like (Steady State)
|
||||
|
||||
A mature ren installation, after some period of use on a given machine/workflow, should exhibit:
|
||||
@@ -291,23 +298,22 @@ A mature ren installation, after some period of use on a given machine/workflow,
|
||||
- The LLM is invoked primarily for genuinely novel requests, complex multi-step reasoning, or tasks that don't decompose cleanly into a deterministic script.
|
||||
- The recipe store is a legible, growing library that a user can audit, edit, or delete from directly — it's not a black box.
|
||||
- System prompt size stays flat regardless of how many recipes exist, since recipes are matched via embedding similarity, not stuffed into context.
|
||||
- The harness itself stays thin and swappable — the underlying model can be upgraded or replaced without losing anything the recipe store has already learned.
|
||||
|
||||
|
||||
## 9. Current Implementation Status
|
||||
|
||||
| Component | Status |
|
||||
|---|---|
|
||||
| Agentic loop (multi-turn tool calling) | ✅ Built |
|
||||
| Trace recorder | ✅ Built |
|
||||
| Recipe extractor (trace → generalized script) | ✅ Built |
|
||||
| Recipe schema + save/load | ✅ Built |
|
||||
| Semantic matcher | ⏳ Not started |
|
||||
| Recipe runner (execution + param extraction) | ⏳ Not started |
|
||||
| Failure reflection | ⏳ Not started |
|
||||
| Config file (`ren.toml`) | ⏳ Not started |
|
||||
| Tool plugin system (beyond `run_curl` / `run_shell`) | ⏳ Not started |
|
||||
|
||||
|
||||
| Agentic loop (multi-turn tool calling) |Built |
|
||||
| Trace recorder |Built |
|
||||
| Recipe extractor (trace → generalized script) |Built |
|
||||
| Recipe schema + save/load |Built |
|
||||
| Semantic matcher | Not started |
|
||||
| Recipe runner (execution + param extraction) | Not started |
|
||||
| Failure reflection | Not started |
|
||||
| Config file (`ren.toml`) | Not started |
|
||||
| Tool plugin system (beyond `run_curl` / `run_shell`) | Not started |
|
||||
## 10. Open Questions Tracker
|
||||
|
||||
These are unresolved decisions flagged during design, kept here so they don't get lost:
|
||||
|
||||
Reference in New Issue
Block a user