Open beta · v0.1.0

Prompt tests
that catch
regressions.

Run your LLM prompts with flakiness-corrected pass rates. Catch quality regressions before production. Compare providers side-by-side.

Get started →Read the docs
wobble — ~/my-project
Platform

Everything you need
to test LLM prompts.

From flakiness correction to multi-model comparison — the full toolkit, zero cloud required.

Core
Flakiness-corrected
pass rates
Run each case N times. A check passes only when it succeeds ≥ threshold of runs — catching non-deterministic failures that single-shot testing misses entirely.
runs: 10 · threshold: 0.7 · 8/10 ✓ pass
Compare
Side-by-side
model comparison
Run your full suite against multiple providers concurrently. Flakiness-corrected pass rates and cost estimates per model, in one aligned table.
gpt-4o
100%
claude-haiku
88%
gemini-flash
62%
History
Trend tracking
Every run appended to .wobble/history.jsonl. First-regression detection tells you exactly when quality dropped.
10 runs ago100% now
Evaluators
8 built-in evaluators
Contains, regex, JSON schema, LLM-as-judge, rubric scoring, custom JS. Mix and match freely within one test case.
containsmax_lengthregexjson_schemallm_judgerubricstarts_withcustom
CI/CD
Ships to CI on day one
JUnit + JSON output. Non-zero exit on regression. Plug into GitHub Actions, GitLab, or any runner.
# .github/workflows/ci.yml
- run: wobble run --output junit
✓ 8 passedexit 0
wobble watch
Watch mode
Re-run on every file save. 300ms debounce.
max_cost_per_run
Cost guardrails
Abort if projected spend exceeds your limit.
turns: […]
Multi-turn chats
Test full dialogues with the turns: key.
{{variable}}
Variable injection
{{name}} placeholders in prompts and inputs.
Evaluators

Eight ways to assert
model output.

Compose freely — every evaluator works with N-run flakiness correction. Mix and match inside a single test case.

contains/ not_contains
Substring present or absent in output. Optional case-insensitive flag.
type: contains
value: "deadline"
case_sensitive: false
starts_with/ ends_with
Prefix or suffix assertion. Useful for structured response formats.
type: ends_with
value: "."
max_length
Cap output in chars or words. Catches verbosity regressions early.
type: max_length
value: 100
unit: words
regex
Full regex match — ISO dates, IDs, phone numbers, structured patterns.
type: regex
value: '^\d{4}-\d{2}-\d{2}$'
json_schema
Assert valid JSON conforming to a schema. Catches structural regressions.
type: json_schema
schema:
  type: object
  required: [name, id]
llm_judge/ pass/fail
Natural-language criterion evaluated by a model. Supports few-shot examples.
type: llm_judge
criteria: "Is it polite?"
llm_judge/ rubric
Score dimensions 1–10 with weights. Pass if weighted avg ≥ threshold.
type: llm_judge
threshold: 7
rubric:
  - dimension: tone
    weight: 2
custom
Any JS/TS module exporting evaluate(). Fully extensible — no limits.
type: custom
evaluator: ./my-check.js
wobble compare

One command.
Every model,
compared.

Run your full suite against multiple providers concurrently. Flakiness-corrected pass rates, per-check breakdowns, and estimated cost — all in one table.

$ wobble compare --targets gpt-4o,claude-haiku,gemini-flash
Input · Evalgpt-4oclaudegemini
"Scientists…"max_length
100%100%100%
"Scientists…"llm_judge
90% 100% 70%
"Book table…"json_schema
100%100% 0%
"My order…"llm_judge
80% 100% 60%
est. cost$0.0021$0.0018$0.0009
CLI

Complete CLI.
All stable.

Eight commands, all stable. From init to history — everything you need without the extras.

wobble initInteractive setup. Generates wobble.yaml and a starter prompt file.stable
wobble runRun tests, compare to baseline. Terminal / JSON / JUnit output.stable
wobble watchRe-run on every file save. 300ms debounce. Baseline diff after each run.stable
wobble compareRun suite against N models concurrently. Pass-rates + cost table.stable
wobble baseline approveLock current pass rates as baseline. Always exits 0.stable
wobble baseline showPrint stored baseline rates by test. --json for raw output.stable
wobble historyOverall pass-rate trend across all recorded runs. --last N to limit.stable
wobble history showPer-check trend + first-regression detection. --test --input --eval.stable
Roadmap

Open beta.
Shipping fast.

Features ship in order of demand. Star on GitHub to vote with your ⭐

soon
◆
Web dashboard
Browser UI for history, trend charts, baseline diffs. Reads .wobble/ locally.
soon
◆
Dataset support
Load cases from CSV or JSONL. Run against hundreds of production inputs.
soon
◆
Prompt chaining
Wire one prompt's output into another's input. Test pipelines end-to-end.
soon
◆
Live watch TUI
Full-screen terminal with live spinners per test, real-time rows.
soon
◆
VS Code extension
Inline pass/fail next to each YAML case. One-click re-run from editor.
soon
◆
Auto case gen
Generate adversarial test cases from a prompt file. Seed without hand-writing.
Get started

Up and running
in 60 seconds.

No account. No cloud. No telemetry.
Node ≥ 18 and an API key is all you need.

npm
$ npm install -g wobble-cli
pnpm
$ pnpm add -g wobble-cli
Supported providers
OpenAIOPENAI_API_KEYgpt-4o, gpt-4o-mini
AnthropicANTHROPIC_API_KEYclaude-sonnet-4-6
GoogleGOOGLE_API_KEYgemini-2.0-flash
Get started →GitHub
quickstart.sh
# 1. Interactive config
$ wobble init
 
# 2. Set API key
$ export OPENAI_API_KEY=sk-...
 
# 3. Run tests
$ wobble run
 
# 4. Lock baseline
$ wobble baseline approve
 
# 5. Edit prompt → catch regressions
$ wobble run
 
summarise
✓ max_length 3/3
✗ llm_judge 1/3
 
1 regression: 100% → 33%