Add README.md
This commit is contained in:
334
README.md
Normal file
334
README.md
Normal file
@@ -0,0 +1,334 @@
|
||||
# ren
|
||||
|
||||
**A self-optimizing agentic runtime for small language models**
|
||||
|
||||
---
|
||||
|
||||
## 1. Vision
|
||||
|
||||
Every existing agentic coding tool — Claude Code, Hermes, Cursor, etc. — is built around the assumption that the model behind it is large, cloud-hosted, and can absorb a 15–20k token system prompt without blinking. That assumption breaks completely on consumer hardware. A 4GB VRAM laptop running Gemma 4B doesn't have room for both the model weights and a novel's worth of instructions in every request.
|
||||
|
||||
**ren** is an agentic runtime designed from the ground up for small models running locally — 4B-class models on low-end GPUs or even CPU-only laptops. The constraint isn't a limitation to work around; it's the design center.
|
||||
|
||||
The core insight: **most of the cost in agentic systems isn't the task, it's re-deciding how to do the task every single time.** A model doesn't need to reason from scratch about "how do I check if a URL is reachable" every time a user asks — it needs to reason about it *once*, then remember.
|
||||
|
||||
### The one-sentence pitch
|
||||
|
||||
> ren minimizes LLM calls by learning from its own successful executions, converting repeated tasks into deterministic, zero-inference scripts — so the small model is only invoked for genuinely novel work.
|
||||
|
||||
---
|
||||
|
||||
## 2. The Problem
|
||||
|
||||
| Constraint | Reality on target hardware |
|
||||
|---|---|
|
||||
| VRAM | 4GB typical (integrated / entry discrete GPU) |
|
||||
| System RAM | 4–5GB available for inference workloads |
|
||||
| Model class | ~4B parameters (Gemma 4B, Phi, Qwen 4B-class) |
|
||||
| Context budget | Every token of system prompt is a token *not* available for task context |
|
||||
|
||||
Existing tools spend 15–20k tokens on system prompt before the user has typed a single word. On a small model with a small context window, this is not a minor inefficiency — it can be the majority of the usable context. It also means slower time-to-first-token on hardware that is already generation-speed constrained.
|
||||
|
||||
**ren's answer:** keep the system prompt minimal (under ~1.5k tokens), and instead of trying to make the model smarter, make the *system* smarter about when to invoke the model at all.
|
||||
|
||||
---
|
||||
|
||||
## 3. Core Design Principles
|
||||
|
||||
1. **Minimize LLM calls, not just tokens per call.** The biggest lever isn't prompt compression — it's not calling the LLM at all when a deterministic path already exists.
|
||||
2. **Small system prompt, always.** No matter how the system grows, the per-request system prompt stays lean. Complexity lives in the architecture, not in prose fed to the model.
|
||||
3. **Self-improvement is a byproduct of execution, not a separate training loop.** ren doesn't fine-tune anything. It learns by converting its own successful tool-use traces into reusable, generalized scripts.
|
||||
4. **Deterministic execution is always preferred over inference.** If a task can be reduced to a script, it should be — scripts are faster, cheaper, auditable, and don't hallucinate.
|
||||
5. **The model is a fallback, not the default.** The steady-state behavior of a mature ren installation should be: most tasks resolve via recipe match, and the LLM is invoked only for genuinely novel requests.
|
||||
6. **Everything the system learns is inspectable.** Traces and recipes are plain JSON/scripts on disk — a user (or the model itself) can read exactly what ren has learned to do and why.
|
||||
|
||||
---
|
||||
|
||||
## 4. High-Level Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ User Input │
|
||||
└────────┬─────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ Intent Embedder │ ← tiny model, CPU, ms-scale
|
||||
└──────────┬────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ Recipe Matcher │
|
||||
│ (cosine similarity │
|
||||
│ vs. stored triggers)│
|
||||
└─────┬──────────┬──────┘
|
||||
match ≥ threshold │ no confident match
|
||||
│ │
|
||||
▼ ▼
|
||||
┌────────────────┐ ┌──────────────────────┐
|
||||
│ Recipe Runner │ │ Agent Loop │
|
||||
│ (zero LLM call)│ │ (LLM + tool calling) │
|
||||
└───────┬────────┘ └──────────┬──────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
Result to user Result to user
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ Trace Recorder │
|
||||
│ (saves full run to │
|
||||
│ traces/*.json) │
|
||||
└──────────┬──────────────┘
|
||||
│
|
||||
success? │
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ Recipe Extractor │
|
||||
│ (one LLM call: │
|
||||
│ generalize trace → │
|
||||
│ parameterized script) │
|
||||
└──────────┬──────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ Recipe Store │
|
||||
│ recipes/*.json │
|
||||
└──────────────────────┘
|
||||
```
|
||||
|
||||
The system has two loops:
|
||||
|
||||
- **The fast loop** (top path): user input → embed → match → execute script. No LLM involved. This is the steady state ren converges toward.
|
||||
- **The slow loop** (bottom path): user input → LLM agent with tools → trace recorded → if successful, extracted into a new recipe that joins the fast loop's search space.
|
||||
|
||||
Over time, the ratio of fast-loop to slow-loop invocations should increase for any given user's workflow, since recurring tasks get captured.
|
||||
|
||||
---
|
||||
|
||||
## 5. Component Deep-Dive
|
||||
|
||||
### 5.1 Agent Loop (`main.rs`)
|
||||
|
||||
The core LLM-driven execution engine. Talks to any OpenAI-compatible completions endpoint (`llama.cpp` server, Ollama, LM Studio, or a cloud API — the client doesn't care).
|
||||
|
||||
**Contract:**
|
||||
|
||||
```
|
||||
1. Send messages → LLM
|
||||
2. Push the assistant turn into history (always — even if it only contains tool_calls)
|
||||
3a. If tool_calls present: execute each, push tool_result messages, go to 1.
|
||||
3b. If no tool_calls: content is the final answer, terminate.
|
||||
```
|
||||
|
||||
**Key implementation details:**
|
||||
|
||||
- `ChatMessage` is a single unified struct across all four roles (`system`, `user`, `assistant`, `tool`), with role-specific fields marked optional and skipped on serialization when absent. This keeps the wire format clean while letting the whole conversation live in one `Vec<ChatMessage>`.
|
||||
- `ToolCall` / `FunctionCall` derive both `Serialize` and `Deserialize` — the model emits them, and they must be re-serialized into the next request's message history. Also `Clone`, since the response is consumed by value while the loop needs to retain it in history.
|
||||
- The assistant turn is **always pushed before** tool result messages. Most backends validate that every `tool_call_id` on a `tool`-role message matches an `id` on the immediately preceding assistant turn's `tool_calls`. Wrong order is a 400 error.
|
||||
- `content` on the response is `Option<String>` — models frequently return `null` content when the turn is tool-calls-only. A bare `String` field would panic on deserialization.
|
||||
- **Tool output truncation** (`MAX_TOOL_OUTPUT = 2000` chars). A single verbose `curl` response can consume the entire remaining context budget on a small model. Truncation is a hard ceiling per tool call, tunable per-tool in the future.
|
||||
- **`AgentOutput` struct** — `run_agent` always returns `Ok(AgentOutput)`, even on non-convergence (hitting `MAX_ITERATIONS`). The outer `Result::Err` is reserved strictly for unrecoverable failures (network down, malformed JSON from the server). This guarantees the message history is *always* available for trace recording, success or failure.
|
||||
|
||||
**Current tools implemented:**
|
||||
- `run_curl` — HTTP GET/POST to a URL
|
||||
- `run_shell` — arbitrary shell command execution
|
||||
|
||||
These are intentionally general-purpose placeholders. The long-term tool model is closer to MCP servers / pluggable scripts than a fixed hardcoded set.
|
||||
|
||||
---
|
||||
|
||||
### 5.2 Trace Recorder (`trace.rs`)
|
||||
|
||||
Every agent run — successful or not — is serialized to `traces/<unix_ms>_<slug>.json`.
|
||||
|
||||
**Why record failures too:** a failed trace is diagnostic material. It's not (yet) fed into recipe extraction, but it's the raw material for a future "error reflection" pass — recognizing recurring failure patterns and adjusting tool descriptions or the system prompt.
|
||||
|
||||
**`Trace` schema:**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 1718123456789,
|
||||
"task": "check if wttr.in/delhi is accessible...",
|
||||
"status": "success",
|
||||
"iterations": 2,
|
||||
"duration_ms": 1840,
|
||||
"tools_invoked": ["run_curl"],
|
||||
"messages": [ /* full ChatMessage history */ ],
|
||||
"final_answer": "wttr.in/Delhi is accessible. Currently 38°C.",
|
||||
"error": null
|
||||
}
|
||||
```
|
||||
|
||||
- `tools_invoked` is derived automatically by walking the message list for assistant turns containing `tool_calls`, deduplicated in first-seen order. This lets downstream consumers (the extractor) know a trace's tool dependencies without re-parsing the full conversation.
|
||||
- `id` doubles as both a unique identifier and a natural sort key (Unix ms timestamp).
|
||||
- The full `messages` array is preserved verbatim — this is what makes recipe extraction possible. Nothing is summarized or lost at record time.
|
||||
|
||||
---
|
||||
|
||||
### 5.3 Recipe Extractor (`recipe.rs`)
|
||||
|
||||
Takes a successful `Trace` and makes **one, focused LLM call** to generalize it into a reusable recipe.
|
||||
|
||||
**Why this has to be a separate, single-purpose call** rather than something the agent does inline: generalization requires stepping back from the specific instance ("Delhi", "wttr.in") to the abstract pattern ("check URL reachability, parameterized by URL"). Asking the agent to do this mid-task would pollute the tool-execution context and cost tokens on every run, not just successful new-pattern runs.
|
||||
|
||||
**Process:**
|
||||
|
||||
1. `render_trace()` converts the raw `ChatMessage` list into readable plain text — `[SYSTEM]`, `[USER]`, `[ASSISTANT — tool calls]`, `[TOOL RESULT]` sections separated by `---`. This is what the extraction LLM actually reads; it's optimized for the model's comprehension, not wire efficiency.
|
||||
2. A single prompt instructs the model to:
|
||||
- Identify the **high-level goal**, not the specific instance
|
||||
- Replace every concrete value (URLs, filenames, search terms) with `$UPPERCASE_PARAMETER` placeholders
|
||||
- Write a script implementing the generalized task
|
||||
- Produce a one-sentence `trigger` description plus 2–3 `trigger_examples` (alternate phrasings) — this is what will later be embedded for semantic matching
|
||||
3. The call uses **structured output** (`response_format: json_schema`, strict mode) to constrain the model to a fixed shape. This matters a lot for small models — without a schema, a 4B model is much more likely to add prose commentary around the JSON or drift from the expected keys.
|
||||
4. `strip_fences()` defensively handles models that wrap JSON in markdown fences despite instructions not to — common failure mode below the ~7B range.
|
||||
|
||||
**`Recipe` schema:**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 1718123999000,
|
||||
"trigger": "Check whether a given URL is reachable via HTTP",
|
||||
"trigger_examples": [
|
||||
"is this site up",
|
||||
"check if example.com is accessible",
|
||||
"ping this endpoint for me"
|
||||
],
|
||||
"parameters": [
|
||||
{ "name": "URL", "description": "The full URL to check" }
|
||||
],
|
||||
"script": "#!/bin/bash\nset -euo pipefail\n: \"${URL:?URL is required}\"\ncurl -s -o /dev/null -w '%{http_code}' \"$URL\"",
|
||||
"script_type": "shell",
|
||||
"tools_required": ["run_curl"],
|
||||
"source_trace_id": 1718123456789,
|
||||
"created_at": 1718123999000,
|
||||
"run_count": 0,
|
||||
"success_count": 0
|
||||
}
|
||||
```
|
||||
|
||||
Saved to `recipes/<id>_<slugified-trigger>.json`.
|
||||
|
||||
`run_count` / `success_count` are pre-wired into the schema for the recipe runner to update — this gives a future confidence-decay mechanism (a recipe that starts failing repeatedly should get deprioritized or flagged for re-extraction) without needing a schema migration later.
|
||||
|
||||
---
|
||||
|
||||
### 5.4 Script Language Decision
|
||||
|
||||
| Language | Verdict | Reasoning |
|
||||
|---|---|---|
|
||||
| **Rust** | ❌ Excluded | Requires compilation — reintroduces the exact latency the recipe system exists to eliminate. |
|
||||
| **Shell** | ✅ Default (~80% of recipes) | Zero interpreter startup beyond the OS shell, universally available, native parameter substitution via env vars, and matches what the agent's own tools (`run_curl`, `run_shell`) already produce. Small models have seen enormous volumes of bash and generate it reliably for this class of task. |
|
||||
| **Python** | ✅ Fallback (~20% of recipes) | Used when a recipe needs JSON parsing, structured data manipulation, or conditional logic that would be unreadable in bash. |
|
||||
|
||||
Both interpreters read parameters identically — as **environment variables**, not string-interpolated arguments. This sidesteps injection risk entirely and keeps the runner interface uniform across script types:
|
||||
|
||||
```rust
|
||||
fn execute_recipe(recipe: &Recipe, params: &HashMap<String, String>) -> Output {
|
||||
let mut cmd = match recipe.script_type.as_str() {
|
||||
"python" => Command::new("python3"),
|
||||
_ => Command::new("sh"),
|
||||
};
|
||||
let script_file = write_temp_script(&recipe.script, &recipe.script_type);
|
||||
cmd.arg(&script_file).envs(params);
|
||||
cmd.output()
|
||||
}
|
||||
```
|
||||
|
||||
Generated shell scripts are expected to start with a safety header:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
: "${URL:?URL is required}"
|
||||
```
|
||||
|
||||
`set -euo pipefail` ensures failures propagate instead of silently continuing; the `:` parameter-check pattern gives a clear error if a required env var wasn't supplied.
|
||||
|
||||
---
|
||||
|
||||
## 6. Planned Components (Not Yet Built)
|
||||
|
||||
### 6.1 Semantic Matcher
|
||||
|
||||
Embeds incoming user input and compares it against the `trigger` + `trigger_examples` of every stored recipe via cosine similarity.
|
||||
|
||||
**Open design questions:**
|
||||
- Embedding model: something CPU-fast and small — `all-MiniLM-L6-v2` (~22MB) is the leading candidate. Needs to run in well under 100ms to not undermine the "fast loop" premise.
|
||||
- Vector store: given the target hardware profile, a lightweight embedded option makes more sense than a server — likely a flat in-memory index rebuilt on startup (recipe counts are expected to be in the hundreds/low-thousands, not millions) rather than a full vector DB. `lancedb` or a hand-rolled flat cosine scan are both plausible.
|
||||
- **Threshold tuning is the highest-risk part of the whole system.** A false-positive match — running the wrong recipe with the wrong parameters — is worse than falling back to the LLM. The matcher should be tuned for high precision over high recall, and:
|
||||
- Recipes involving destructive operations (file deletion, overwrites, network mutations) should require either a higher confidence threshold or explicit user confirmation before running, regardless of match score.
|
||||
- Ambiguous matches (multiple recipes above threshold, or a score in a "gray zone") should fall back to the LLM rather than guessing.
|
||||
|
||||
### 6.2 Recipe Runner
|
||||
|
||||
Executes a matched recipe: extracts required parameters from the user's phrasing (likely via a lightweight extraction step, not necessarily the main LLM — could be regex/slot-filling for simple cases, small-model call for complex ones), writes the script to a temp file, executes with `envs()`, captures output, updates `run_count` / `success_count` on the recipe file.
|
||||
|
||||
**Open question:** how does parameter extraction happen without invoking the main LLM? Options being considered:
|
||||
- A much smaller/cheaper model dedicated to slot-filling
|
||||
- Structured regex/pattern matching keyed off the parameter descriptions
|
||||
- A hybrid: try pattern matching first, fall back to a cheap LLM call only if extraction confidence is low
|
||||
|
||||
This matters because if parameter extraction always requires an LLM call, the "zero-inference execution" claim weakens — it becomes "one call instead of many" rather than "zero calls." Worth deciding early since it affects the recipe schema (parameters may need extraction hints, e.g. regex patterns, alongside their descriptions).
|
||||
|
||||
### 6.3 Failure Reflection
|
||||
|
||||
Failed traces are currently recorded but unused. The planned second consumer of the trace store: periodically (or on-demand) analyze failure patterns to identify:
|
||||
- Tools that are frequently misused (suggesting a description needs improvement)
|
||||
- Recurring task types the agent struggles with (suggesting a manual recipe should be authored, or the system prompt needs a nudge)
|
||||
|
||||
This is explicitly lower priority than the success path — the core value proposition depends on the success → recipe pipeline working well first.
|
||||
|
||||
### 6.4 Config
|
||||
|
||||
Currently hardcoded (endpoint URL, model name, `MAX_ITERATIONS`, `MAX_TOOL_OUTPUT`). Needs to move to a `ren.toml` before the codebase grows further — flagged as technical debt, not urgent.
|
||||
|
||||
---
|
||||
|
||||
## 7. Technology Choices & Rationale
|
||||
|
||||
| Choice | Rationale |
|
||||
|---|---|
|
||||
| **Rust** for the runtime | Near-zero process overhead — critical when 4GB RAM is shared between the OS, the model weights, and the runtime itself. Easy FFI into `llama.cpp` if/when local inference moves in-process. Deterministic recipe execution with no GC pauses or interpreter startup cost. |
|
||||
| **OpenAI-compatible API surface** | Works unmodified against `llama.cpp` server, Ollama, LM Studio, or a real cloud endpoint. The model location (local or cloud) is a deployment detail, not an architectural one. |
|
||||
| **JSON Schema structured outputs** for extraction | Small models are meaningfully less reliable at free-form JSON generation. Constraining the output shape at the API level (rather than hoping the prompt is followed) removes an entire failure class. |
|
||||
| **Plain JSON files** for traces/recipes (no database, for now) | Fully inspectable and diffable by a human or the model itself. No schema migration tooling needed yet at expected scale (hundreds of recipes). Revisit if/when recipe counts grow past what a flat directory scan can handle at matcher startup time. |
|
||||
| **Environment variables** for recipe parameters | Uniform interface across shell and Python, avoids string-interpolation injection risk entirely, and matches how both languages naturally expect external input. |
|
||||
|
||||
---
|
||||
|
||||
## 8. What "Done" Looks Like (Steady State)
|
||||
|
||||
A mature ren installation, after some period of use on a given machine/workflow, should exhibit:
|
||||
|
||||
- The large majority of routine requests resolve via the fast loop (recipe match → script execution), with sub-100ms latency and zero token generation.
|
||||
- The LLM is invoked primarily for genuinely novel requests, complex multi-step reasoning, or tasks that don't decompose cleanly into a deterministic script.
|
||||
- The recipe store is a legible, growing library that a user can audit, edit, or delete from directly — it's not a black box.
|
||||
- System prompt size stays flat regardless of how many recipes exist, since recipes are matched via embedding similarity, not stuffed into context.
|
||||
|
||||
---
|
||||
|
||||
## 9. Current Implementation Status
|
||||
|
||||
| Component | Status |
|
||||
|---|---|
|
||||
| Agentic loop (multi-turn tool calling) | ✅ Built |
|
||||
| Trace recorder | ✅ Built |
|
||||
| Recipe extractor (trace → generalized script) | ✅ Built |
|
||||
| Recipe schema + save/load | ✅ Built |
|
||||
| Semantic matcher | ⏳ Not started |
|
||||
| Recipe runner (execution + param extraction) | ⏳ Not started |
|
||||
| Failure reflection | ⏳ Not started |
|
||||
| Config file (`ren.toml`) | ⏳ Not started |
|
||||
| Tool plugin system (beyond `run_curl` / `run_shell`) | ⏳ Not started |
|
||||
|
||||
---
|
||||
|
||||
## 10. Open Questions Tracker
|
||||
|
||||
These are unresolved decisions flagged during design, kept here so they don't get lost:
|
||||
|
||||
1. How is parameter extraction done at recipe-run time without reintroducing a full LLM call per invocation?
|
||||
2. What embedding model and similarity threshold produce acceptably low false-positive rates for recipe matching?
|
||||
3. Should destructive recipes require explicit user confirmation on every run, or only above a certain risk classification — and who assigns that classification (the extractor, a fixed tool allowlist, or the user)?
|
||||
4. At what recipe-store size does a flat directory scan become a real bottleneck, and what's the migration path if so?
|
||||
5. Should failed traces eventually feed an automated system-prompt/tool-description tuning loop, or stay purely diagnostic for manual review?
|
||||
Reference in New Issue
Block a user