SurpriseLens — offline token-surprise heatmap (private)

predictable mid surprising

Privacy: LM runs locally (WebAssembly). Heuristic fallback needs no download.

use GPT-2 surprise (~50MB quantized, cached) Heuristic ready (no download)

Text

—

Avg surprise—0 low → 1 high

High-surprise words—≥0.72 (potential AI/cliché flags)

Hover a word for score. Red clusters = unusual phrasing; all-green = highly predictable (possible AI boilerplate).

How it works

LM mode: loads Xenova/distilgpt2 quantized via Transformers.js (WebAssembly). Tokenizes text, runs causal LM, gets logits per position → logprob of actual next token → surprise = (-logprob /9)^0.85 core.js:62.
Merge: BPE tokens (Ġ) grouped to words, averaged. Chunked 1200 chars to avoid O(n²).
Heuristic fallback: no download. Scores by common-word list, length, digits/caps, doc frequency & rare trigrams core.js:14.
Private: model cached by browser; subsequent runs offline. Multilingual heuristic handles Arabic; LM is English-biased (GPT-2) — heuristic used for Arabic segments.
Not a detector: high avg ≠ AI; use as writing aid / calibration.