Benchmarked on 2026-09-04 against 3138 test files from the chardet test corpus. Methodology mirrors chardet/docs/rewrite_performance.md: each detector runs against the full corpus, equivalence rules (directional supersets, bidirectional groups, decoded-output equivalence) are applied identically across all detectors, and timing benchmarks take the median across 5 runs.
Three detectors are compared:
- jschardet 3.1.4 — the previous JavaScript port
- jschardet 4.0.0 — this package, a TypeScript ground-up port of chardet 7
- chardet 7.6.0 — upstream Python, run via PYTHONPATH against the
chardet/submodule
Reproduce locally:
npm run benchmark:accuracy
npm run benchmark:throughput
npm run benchmark:coldstart
npm run benchmark:memory| Detector | Correct | Accuracy |
|---|---|---|
| jschardet 3.1.4 | 1352/3138 | 43.1% |
| jschardet 4.0.0 | 3120/3138 | 99.4% |
| chardet 7.6.0 | 3122/3138 | 99.5% |
jschardet 4 lifts accuracy by 56.3pp over jschardet 3 — the underlying chardet rewrite ships new bigram models, EBCDIC/DOS/Mac coverage, and magic-number plus markup-charset pipelines that v3 lacked. It trails upstream by two files (two cp932 samples, see the per-encoding table below); every other encoding matches chardet exactly.
| Detector | Correct | Accuracy |
|---|---|---|
| jschardet 3.1.4 | n/a | n/a |
| jschardet 4.0.0 | 2922/3130 | 93.4% |
| chardet 7.6.0 | 2922/3130 | 93.4% |
jschardet 3 does not return a language field. The 208 wrong-language
cases under jschardet 4 are primarily confusable language pairs within
the same script (Danish/Norwegian, Belarusian/Bulgarian for Cyrillic,
etc.).
Per-file detection latency over the full corpus, median of 5 in-process
runs with per-file times aggregated element-wise. Chardet's upstream
_run_timing_with_median isolates each pass in its own subprocess; we
don't, to keep runtime reasonable.
| Detector | Files/s | Mean | Median | p90 | p95 |
|---|---|---|---|---|---|
| jschardet 3.1.4 | 126 | 7.91 ms | 0.76 ms | 9.00 ms | 18.00 ms |
| jschardet 4.0.0 | 780 | 1.28 ms | 0.43 ms | 2.43 ms | 3.69 ms |
| chardet 7.6.0 | 222 | 4.51 ms | 1.85 ms | 10.91 ms | 15.76 ms |
jschardet 4 processes about 6× more files per second than jschardet 3 on this corpus, and the tail latency narrows too — p95 drops from 18.00 ms to 3.69 ms.
This port does not implement upstream's rowmax scoring pruning (see
port-notes.md);
it always scores every candidate, matching chardet's full_ranking=True
path. Probed separately against this corpus pin, that fast path is worth
single-digit percent here rather than the multiple upstream's "~2.9x with
mypyc" changelog entry suggests: pruning skips 22% of model scorings
(263k vs 337k across the corpus), statistical scoring accounts for only
~21% of this port's detection time, and chardet's own pruned and unpruned
paths measure within ~4% of each other when run interpreted. Making
single-byte scoring entirely free — a ceiling no real pruning reaches —
lifts a probe build by ~30%.
Import time and first detect() call latency in a fresh subprocess
(median of 5 runs). Each measurement is isolated to avoid module-cache
effects.
| Detector | Import | First detect | Total |
|---|---|---|---|
| jschardet 3.1.4 | 29.23 ms | 0.68 ms | 29.91 ms |
| jschardet 4.0.0 | 44.74 ms | 64.46 ms | 109.20 ms |
| chardet 7.6.0 | 36.31 ms | 65.44 ms | 101.74 ms |
jschardet 4 trades a heavier first-call cost for a lighter steady state:
the bigram models ship zlib-compressed and decompress lazily on the first
detect() call (see docs/model-compression.md),
so cold-start latency is ~4× higher than jschardet 3 but every
subsequent call is faster (see throughput above). jschardet 3 has no
models to decompress, which is why its first detect is essentially free.
Peak resident-set-size (RSS high-water mark since process start, via
getrusage(RUSAGE_SELF).ru_maxrss — process.resourceUsage().maxRSS
in Node, resource.getrusage().ru_maxrss in Python) sampled in a fresh
subprocess at three points: before the detector is imported, after
import, and after detect() has been called on every file in the
corpus (median of 5 runs). Peak RSS is the apples-to-apples
cross-language metric — V8 has no tracemalloc equivalent, and reading
the same syscall on both sides keeps the numbers comparable.
| Detector | Baseline RSS | Import delta | Peak delta | Final RSS |
|---|---|---|---|---|
| jschardet 3.1.4 | 138.5 MiB | 668.0 KiB | 831.8 MiB | 970.2 MiB |
| jschardet 4.0.0 | 138.2 MiB | 6.0 MiB | 85.5 MiB | 223.6 MiB |
| chardet 7.6.0 | 89.2 MiB | 8.6 MiB | 37.6 MiB | 126.9 MiB |
jschardet 4's peak RSS is ~10× lower than jschardet 3 (85.5 MiB vs 831.8 MiB of growth above baseline). The chardet rewrite's dense bigram model format (one 64 KiB lookup table per language, loaded once and shared across calls) replaces the per-call sparse-map allocations that drive v3's high water mark.
The baseline gap between Node (~138 MiB) and Python (~89 MiB) is the interpreter's own resident footprint plus the corpus bytes — both workers pre-load the full corpus into memory before measuring baseline, so the corpus shows up there rather than under the detector.
| Encoding | N | jschardet 3.1.4 | jschardet 4.0.0 | chardet 7.6.0 |
|---|---|---|---|---|
| (binary) | 8 | 7/8 (87.5%) | 8/8 (100.0%) | 8/8 (100.0%) |
| ascii | 32 | 31/32 (96.9%) | 32/32 (100.0%) | 32/32 (100.0%) |
| big5 | 29 | 29/29 (100.0%) | 29/29 (100.0%) | 29/29 (100.0%) |
| cp037 | 34 | 0/34 (0.0%) | 28/34 (82.4%) | 28/34 (82.4%) |
| cp1006 | 3 | 0/3 (0.0%) | 3/3 (100.0%) | 3/3 (100.0%) |
| cp1026 | 8 | 0/8 (0.0%) | 8/8 (100.0%) | 8/8 (100.0%) |
| cp1125 | 5 | 0/5 (0.0%) | 5/5 (100.0%) | 5/5 (100.0%) |
| cp273 | 7 | 0/7 (0.0%) | 7/7 (100.0%) | 7/7 (100.0%) |
| cp424 | 9 | 0/9 (0.0%) | 9/9 (100.0%) | 9/9 (100.0%) |
| cp437 | 33 | 0/33 (0.0%) | 32/33 (97.0%) | 32/33 (97.0%) |
| cp500 | 29 | 0/29 (0.0%) | 28/29 (96.6%) | 28/29 (96.6%) |
| cp720 | 10 | 0/10 (0.0%) | 10/10 (100.0%) | 10/10 (100.0%) |
| cp737 | 3 | 0/3 (0.0%) | 3/3 (100.0%) | 3/3 (100.0%) |
| cp775 | 12 | 0/12 (0.0%) | 12/12 (100.0%) | 12/12 (100.0%) |
| cp850 | 45 | 0/45 (0.0%) | 42/45 (93.3%) | 42/45 (93.3%) |
| cp852 | 26 | 0/26 (0.0%) | 26/26 (100.0%) | 26/26 (100.0%) |
| cp855 | 39 | 39/39 (100.0%) | 39/39 (100.0%) | 39/39 (100.0%) |
| cp856 | 8 | 0/8 (0.0%) | 8/8 (100.0%) | 8/8 (100.0%) |
| cp857 | 6 | 0/6 (0.0%) | 6/6 (100.0%) | 6/6 (100.0%) |
| cp858 | 35 | 0/35 (0.0%) | 33/35 (94.3%) | 33/35 (94.3%) |
| cp860 | 6 | 0/6 (0.0%) | 6/6 (100.0%) | 6/6 (100.0%) |
| cp861 | 5 | 0/5 (0.0%) | 5/5 (100.0%) | 5/5 (100.0%) |
| cp862 | 7 | 0/7 (0.0%) | 7/7 (100.0%) | 7/7 (100.0%) |
| cp863 | 5 | 0/5 (0.0%) | 5/5 (100.0%) | 5/5 (100.0%) |
| cp864 | 2 | 0/2 (0.0%) | 2/2 (100.0%) | 2/2 (100.0%) |
| cp865 | 6 | 0/6 (0.0%) | 6/6 (100.0%) | 6/6 (100.0%) |
| cp866 | 37 | 37/37 (100.0%) | 37/37 (100.0%) | 37/37 (100.0%) |
| cp869 | 6 | 0/6 (0.0%) | 6/6 (100.0%) | 6/6 (100.0%) |
| cp874 | 8 | 5/8 (62.5%) | 6/8 (75.0%) | 6/8 (75.0%) |
| cp875 | 8 | 0/8 (0.0%) | 8/8 (100.0%) | 8/8 (100.0%) |
| cp932 | 9 | 0/9 (0.0%) | 7/9 (77.8%) | 9/9 (100.0%) |
| cp949 | 8 | 7/8 (87.5%) | 8/8 (100.0%) | 8/8 (100.0%) |
| euc-jp | 32 | 32/32 (100.0%) | 32/32 (100.0%) | 32/32 (100.0%) |
| euc-kr | 37 | 37/37 (100.0%) | 37/37 (100.0%) | 37/37 (100.0%) |
| gb18030 | 9 | 9/9 (100.0%) | 9/9 (100.0%) | 9/9 (100.0%) |
| gb2312 | 25 | 25/25 (100.0%) | 25/25 (100.0%) | 25/25 (100.0%) |
| hp-roman8 | 44 | 12/44 (27.3%) | 44/44 (100.0%) | 44/44 (100.0%) |
| hz-gb-2312 | 8 | 8/8 (100.0%) | 8/8 (100.0%) | 8/8 (100.0%) |
| iso-2022-jp | 8 | 8/8 (100.0%) | 8/8 (100.0%) | 8/8 (100.0%) |
| iso-2022-jp-2004 | 6 | 6/6 (100.0%) | 6/6 (100.0%) | 6/6 (100.0%) |
| iso-2022-jp-ext | 4 | 4/4 (100.0%) | 4/4 (100.0%) | 4/4 (100.0%) |
| iso-2022-kr | 8 | 8/8 (100.0%) | 8/8 (100.0%) | 8/8 (100.0%) |
| iso-8859-1 | 75 | 46/75 (61.3%) | 75/75 (100.0%) | 75/75 (100.0%) |
| iso-8859-10 | 8 | 5/8 (62.5%) | 8/8 (100.0%) | 8/8 (100.0%) |
| iso-8859-11 | 3 | 3/3 (100.0%) | 3/3 (100.0%) | 3/3 (100.0%) |
| iso-8859-13 | 16 | 1/16 (6.3%) | 16/16 (100.0%) | 16/16 (100.0%) |
| iso-8859-14 | 19 | 14/19 (73.7%) | 19/19 (100.0%) | 19/19 (100.0%) |
| iso-8859-15 | 42 | 26/42 (61.9%) | 41/42 (97.6%) | 41/42 (97.6%) |
| iso-8859-16 | 22 | 0/22 (0.0%) | 22/22 (100.0%) | 22/22 (100.0%) |
| iso-8859-2 | 71 | 18/71 (25.4%) | 71/71 (100.0%) | 71/71 (100.0%) |
| iso-8859-3 | 13 | 0/13 (0.0%) | 13/13 (100.0%) | 13/13 (100.0%) |
| iso-8859-4 | 15 | 4/15 (26.7%) | 15/15 (100.0%) | 15/15 (100.0%) |
| iso-8859-5 | 51 | 51/51 (100.0%) | 51/51 (100.0%) | 51/51 (100.0%) |
| iso-8859-6 | 15 | 0/15 (0.0%) | 15/15 (100.0%) | 15/15 (100.0%) |
| iso-8859-7 | 17 | 10/17 (58.8%) | 17/17 (100.0%) | 17/17 (100.0%) |
| iso-8859-8 | 21 | 21/21 (100.0%) | 21/21 (100.0%) | 21/21 (100.0%) |
| iso-8859-9 | 45 | 0/45 (0.0%) | 45/45 (100.0%) | 45/45 (100.0%) |
| johab | 10 | 0/10 (0.0%) | 10/10 (100.0%) | 10/10 (100.0%) |
| koi8-r | 25 | 25/25 (100.0%) | 25/25 (100.0%) | 25/25 (100.0%) |
| koi8-t | 3 | 0/3 (0.0%) | 3/3 (100.0%) | 3/3 (100.0%) |
| koi8-u | 11 | 0/11 (0.0%) | 11/11 (100.0%) | 11/11 (100.0%) |
| kz1048 | 6 | 0/6 (0.0%) | 6/6 (100.0%) | 6/6 (100.0%) |
| maccyrillic | 38 | 34/38 (89.5%) | 38/38 (100.0%) | 38/38 (100.0%) |
| macgreek | 5 | 0/5 (0.0%) | 5/5 (100.0%) | 5/5 (100.0%) |
| maciceland | 5 | 0/5 (0.0%) | 5/5 (100.0%) | 5/5 (100.0%) |
| maclatin2 | 23 | 0/23 (0.0%) | 23/23 (100.0%) | 23/23 (100.0%) |
| macroman | 46 | 0/46 (0.0%) | 46/46 (100.0%) | 46/46 (100.0%) |
| macturkish | 5 | 0/5 (0.0%) | 5/5 (100.0%) | 5/5 (100.0%) |
| ptcp154 | 6 | 0/6 (0.0%) | 6/6 (100.0%) | 6/6 (100.0%) |
| shift-jis | 3 | 3/3 (100.0%) | 3/3 (100.0%) | 3/3 (100.0%) |
| shift_jis | 34 | 34/34 (100.0%) | 34/34 (100.0%) | 34/34 (100.0%) |
| tis-620 | 8 | 8/8 (100.0%) | 8/8 (100.0%) | 8/8 (100.0%) |
| utf-16 | 220 | 220/220 (100.0%) | 220/220 (100.0%) | 220/220 (100.0%) |
| utf-16be | 153 | 0/153 (0.0%) | 153/153 (100.0%) | 153/153 (100.0%) |
| utf-16le | 154 | 0/154 (0.0%) | 154/154 (100.0%) | 154/154 (100.0%) |
| utf-32 | 154 | 154/154 (100.0%) | 154/154 (100.0%) | 154/154 (100.0%) |
| utf-32be | 153 | 0/153 (0.0%) | 153/153 (100.0%) | 153/153 (100.0%) |
| utf-32le | 153 | 0/153 (0.0%) | 153/153 (100.0%) | 153/153 (100.0%) |
| utf-7 | 149 | 0/149 (0.0%) | 149/149 (100.0%) | 149/149 (100.0%) |
| utf-8 | 268 | 268/268 (100.0%) | 268/268 (100.0%) | 268/268 (100.0%) |
| utf-8-sig | 151 | 0/151 (0.0%) | 151/151 (100.0%) | 151/151 (100.0%) |
| windows-1250 | 44 | 3/44 (6.8%) | 44/44 (100.0%) | 44/44 (100.0%) |
| windows-1251 | 63 | 59/63 (93.7%) | 63/63 (100.0%) | 63/63 (100.0%) |
| windows-1252 | 44 | 28/44 (63.6%) | 44/44 (100.0%) | 44/44 (100.0%) |
| windows-1253 | 8 | 3/8 (37.5%) | 8/8 (100.0%) | 8/8 (100.0%) |
| windows-1254 | 9 | 0/9 (0.0%) | 9/9 (100.0%) | 9/9 (100.0%) |
| windows-1255 | 7 | 7/7 (100.0%) | 7/7 (100.0%) | 7/7 (100.0%) |
| windows-1256 | 46 | 0/46 (0.0%) | 46/46 (100.0%) | 46/46 (100.0%) |
| windows-1257 | 18 | 1/18 (5.6%) | 18/18 (100.0%) | 18/18 (100.0%) |
| windows-1258 | 14 | 2/14 (14.3%) | 14/14 (100.0%) | 14/14 (100.0%) |
The full list of known per-file failures is tracked in
tests/accuracy.test.ts.
- Correctness rule — exact match, alias, directional superset (e.g.
windows-1252accepted wheniso-8859-1was expected), bidirectional pair (UTF-16/32 endian variants), or decoded-output equivalence (NFKD-normalized text matches under both encodings). Same rule used bytests/accuracy.test.tsand chardet's own benchmark pipeline. - Raw counts — no
KNOWN_FAILURESallow-list is applied. This matches chardet'srewrite_performance.mdmethodology so the two documents can be read side by side. - Median of 5 runs — applied to throughput, cold start, and memory. For throughput, the median is taken element-wise across per-file times (each file's median is taken across the 5 passes, then the standard mean/median/p90/p95 are computed over those medians).
- Per-detector isolation — chardet 7 runs in a Python subprocess via
tests/benchmark/lib/*-worker-chardet7.py. For cold start and memory, every measurement is in a fresh subprocess to avoid module-cache or RSS-high-water-mark contamination.
Node v22.22.2 on linux/x64
CPU: Intel(R) Xeon(R) Processor @ 2.80GHz (4 cores)
RAM: 15.7 GiB