This is a mirror of thinletter.io/demo: the page runs here, the corpus and the model chunks download from thinletter.io. Hugging Face Spaces cannot send the COOP/COEP headers, so the scalar (llama.cpp) client runs single-threaded and is slow here; the vector-quantised clients (WebGPU) run at full speed. Code: GitHub, Apache-2.0.
Legal CS over the full production index (Elasticsearch, side by side with the original fp32 model) → legal.html
checking WebGPU …
The file is fetched as 24 MiB chunks from this site and kept in the browser's private storage (OPFS), so the next visit starts without downloading. Nothing is sent anywhere.
On this device
- Model file
- …
- Download
- –
- Load (wasm + weights)
- –
- Backend
- –
- Threads
- –
- Cross-origin isolated
- –
- Memory after load
- –
- Encode, last / p50
- –
- Search … docs, last / p50
- –
- Queries this session
- 0
Reference, same file, native llama.cpp on a laptop (8 threads, ctx 512, idle): 151 ms per query, 928 MiB peak RSS. Latency in a busy tab or on a loaded machine is higher.
Try a test query:
Verify on this device
Runs the … … test queries through the model in this tab, one by one, scores each against the index and computes nDCG@10 with the same code path as the search box. Takes … × (your latency); abortable.
| nDCG@10 | vs native | Latency p50 | p95 | |
|---|---|---|---|---|
| This device (browser, this run) | – | – | – | – |
| Native llama.cpp, same file | 0.7440 | – | 151 ms | – |
| Browser, our measurement | 0.7425 | – | under load, see README | |
| Original fp16 model (1 143 MiB) | 0.7559 | – | not in the browser | |
Vector-quantised clients: results
Below llama.cpp's smallest format (IQ2_XS, 177.5 MiB) we run a vector-quantised container (4-d codebooks, 1.8–2.1 bits per weight) with its own WebGPU runtime. The runtime is open source (Apache-2.0) and is what this page runs when you pick a VQ client. Same laptop, integrated Intel GPU, headless Chrome, 300 SciFact test queries, machine idle.
| runtime / file | MiB | load | peak RSS | p50 | p95 | nDCG@10 |
|---|---|---|---|---|---|---|
| VQ runtime, 2.10 bpw | 119.5 | 2.4 s | 1 592 MiB | 103 ms | 113 ms | 0.7375 |
| VQ runtime, 1.83 bpw | 105.3 | 2.5 s | 1 501 MiB | 102 ms | 112 ms | 0.7109 |
| wllama WebGPU, Q3_K (this page) | 235.1 | 4.2 s | 2 080 MiB | 444 ms | 754 ms | 0.7481 |
| wllama WebGPU, IQ2_M | 199.3 | 5.2 s | 1 983 MiB | 508 ms | 1 012 ms | 0.7387 |
| wllama WASM 8 threads, Q2_K | 192.4 | 4.4 s | 1 228 MiB | 1 669 ms | 3 363 ms | 0.7425 |
fp32 model: 0.7559. On the Czech legal index the same format at 2.1 bpw keeps only 83 % (jina) / 64 % (Qwen3-Embedding) of fp32, so a ~3-bit variant is needed there. Full tables, caveats and the paired comparison with BitNet-270m: vq_results.md. To try them here, pick a VQ client in the Client selector of the SciDocs corpus.