`PROMPTFOO_PASS_RATE_THRESHOLD` takes a **percentage, not a fraction** — so writing `0.9` when you meant 90% gives you a **0.9%** threshold, and CI goes green with half your tests failing. Nothing warns you. This episode builds the eval gate that makes swapping a model a check instead of a vibe: one YAML file, one `echo` provider, and an exit code you can read. Plus three finds, two tips, and a ten-minute thing to do today. No git this episode — episodes 3, 4 and 5 lived in worktrees and hooks, and this one deliberately leaves them. ## In this episode - **Kimi K3 (Moonshot AI)** — 2.8T-parameter MoE, 104B activated per token, open weights, 1M context, natively multimodal (text, images and video). Weights went public **Monday 27 July 2026**. **The licence is the story.** Plenty of coverage carried "Modified MIT" forward from K2. It isn't. K2's file is titled *Modified MIT License*; K3's is titled *Kimi K3 License*, and **MIT is never named in it**. **Section 2** (MaaS revenue over **$20M across any consecutive 12 months** → separate agreement with Moonshot; and MaaS is defined narrowly, excluding models embedded in a specific product feature and excluding mere relaying to models hosted by others) and **Section 3** (**$20M per month** in revenue or 100M MAU → display "Kimi K3" in your UI) are **different units and must not be conflated**. **Section 4** exempts internal use and use via Moonshot's own products or certified inference partners — but "internal use" is defined as **not making the software, its outputs, or its underlying capabilities available to third parties**, which is narrower than "inside your own company". Weights are free; Moonshot **names no price** for the separate agreement, which is not the same as free. - **DeepSeek-V4-Flash-0731** — the official release, superseding the preview. `0731` is a **build date, not a version**. **304B parameters, read from the repo's own safetensors index**, not from press coverage. Active parameter count and context length are **not stated** on the model card page fetched, so the episode doesn't state them. **Licence: MIT** — verified from the repo's licence tag and the README. Weights free; **DeepSeek API pricing was not checked this run**. - **DwarfStar / `antirez/ds4` (Salvatore Sanfilippo, creator of Redis)** — **name mismatch: the repo and binary are `ds4`, the README calls it "DwarfStar"**. GitHub *repository* search for "DwarfStar" won't surface it (the name is in neither the repo name nor the description) — you'll get other people's wrappers instead. Search the path, `antirez/ds4`. A small native local inference engine tuned first for DeepSeek V4 Flash — and specifically the **0731 build**: the README names it repeatedly and ships `DeepSeek-V4-Flash-DSpark-support-0731.gguf`. Also GLM 5.2 and, on very-high-memory machines, DeepSeek V4 PRO. Metal / CUDA / ROCm. **MIT**, free, no paid tier named anywhere in the README or LICENSE. README credits **Georgi Gerganov** for the llama.cpp / GGML work. **Caveats in the maintainer's own words: "beta quality", DSpark "experimental" and opt-in, and — on 128 GB MacBooks — PRO q2 streaming "experimental but usable for inspection".** Metal's primary target is **Macs with 96 GB+**; smaller machines get `--ssd-streaming`. **Created back in May — ~20k stars and climbing, so trending, not new.** Its README also credits GPT 5.5, 5.6 and Claude Fable with "strong assistance", humans leading ideas, testing and debugging. - **The Recipe — put a pass/fail gate on your prompts before you change models.** promptfoo 0.122.0, **MIT — verified from the installed `package.json` and npm's registry entry.** (Neither OpenAI's post nor promptfoo's names a licence at all; OpenAI's says they'll "continue building the open-source project", which is a claim about openness, not a second confirmation of MIT.) **OpenAI announced its intent to acquire Promptfoo on Monday 9 March 2026 — announced, not closed: the post is titled "OpenAI *to acquire* Promptfoo" and says closing is subject to customary conditions.** Founders **Ian Webster** and **Michael D'Angelo** — worth knowing that an OpenAI-bound tool grades every vendor's models. **Price: Community is "Free Forever" including red teaming capped at 10,000 probes/month; Enterprise and On-Premise both say "Custom", which means promptfoo names no price — not the same as free.** Node floor is **`>=22.22.0`**, read off the installed package; the GitHub Action states the same floor and recommends Node 24 LTS. - **The steps** — `npx promptfoo@latest init --no-interactive` (**plain `init` is a wizard** — with nothing attached to answer it, it exits **130** and writes no files) → get a **zero-cost green run first** using the `echo` provider (seven deterministic assertion types passed, $0 spent) → then swap in the provider you actually ship → gate CI on the exit code. - **The exit-code contract, measured rather than quoted** — all pass: **0**. At ...
Mehr anzeigen
Weniger anzeigen