Hedgeling benchmark results
@hedgeling/i18n 1.3.1
Eight real open-source applications, each extracted with its own tuned
configuration: a zero-config start, then hedgeling tune (its deterministic
tier and its LLM tier), then a hand-tuned pass driven by the diagnostics
report. Every figure below comes from that final configuration. Precision is
measured on a 100-string sample per project; recall is measured by a strip
test on a fully internationalized app.
Headline
| Measure | Result |
|---|---|
Recall, strip test (Keycloak, 2,181 inlined t() messages) |
99.13% (2,162 recovered) |
| Precision, apps with hardcoded English UI (6 apps) | 82% to 100% |
| Strings found for migration (6 apps) | 19,952 |
| Developer time saved finding those strings by hand | about 1,000 hours (3 minutes per string) |
| Real i18n gaps found in already-localized apps (Keycloak, Directus) | about 180 of 241 strings surfaced |
| Code covered | 8,403 files, 1,124,670 lines, extracted in 11.2 s |
Apps that start with hardcoded English (migration)
| Project | Framework | Files | LOC | Strings | Precision |
|---|---|---|---|---|---|
| Bulletproof React | React | 128 | 6,960 | 103 | 100% |
| Appwrite Console | Svelte | 1,445 | 154,639 | 4,129 | 98% |
| HF Chat UI | Svelte | 360 | 39,934 | 528 | 100% |
| PocketBase Admin | vanilla JS | 185 | 34,650 | 992 | 96% |
| Analog | Angular | 219 | 10,080 | 116 | 82% |
| Supabase Studio | React | 4,142 | 578,458 | 14,084 | 97% |
Supabase Studio is the largest app here and was also used to test the extractor before this release, so treat its row as a scale result, not an independent one.
Precision is the share of surfaced strings that are real user-facing copy. Everything else was a sampled miss: a selector, a code sample, a mock name, a template.
Apps that already ship i18n (audit)
| Project | Framework | Files | LOC | Strings surfaced | Real gaps |
|---|---|---|---|---|---|
| Keycloak Admin Console | React | 618 | 95,840 | 122 | 70% |
| Directus | Vue | 1,306 | 204,109 | 119 | 83% |
These apps already externalize their copy, so what Hedgeling surfaces is the residue their own i18n pass missed. "Real gaps" is the share of surfaced strings that turned out to be genuinely untranslated UI copy rather than noise.
Recall
2,181 plain-text react-i18next messages in the Keycloak Admin Console were inlined back into the source as hardcoded English and the tuned configuration was run against the result. It recovered 2,162 of them as catalog strings or diagnostics (99.13%).
Against i18next-cli
Same eight repositories, same pinned commits, i18next-cli 1.87.0. Its side was tuned as far as its documented options allow: per-project file ignores, a custom scorer that skips noise and raises the confidence of everything else, every lint tag accepted, and the community Vue and Svelte lint plugins where its own docs point to them (there is no Angular plugin). Its results were labeled on 100-string samples with the same rubric as ours.
| Measure | Hedgeling, tuned config | i18next-cli, tuned config |
|---|---|---|
| Recall, same Keycloak strip test (2,181 inlined messages) | 99.13% (19 missed) | 91.61% (183 missed) |
| Precision, migration apps (6) | 96% | 84% |
| Precision, audit apps (2) | 77% | 53% |
Recall is measured the same way for both: a message counts as found when it appears in what the tool shows the developer (our catalog and diagnostics; its instrument candidates and lint findings).
Out of the box, i18next-cli finds 72.81% of the same messages, missing about one in four; Hedgeling with no configuration at all finds 98.90% (24 missed). Tuned as far as its options allow it still misses 183, against Hedgeling's 19. About seven in ten of those remaining misses are the same pattern: a success or error message passed straight to a notification helper, which i18next-cli does not treat as user-facing copy, so toast and alert text is invisible to it.
The per-project comparison has every row.
Questions
Want the per-project samples and configurations, or a run against your own codebase? Email support@thehedgeling.com.