Read the linked article.
That’s very impressive! I haven’t heard of Kroko, Banafo, or Zipformer2 before.
Median of less than 2 seconds for final word after its spoken on esp32 is insane, amazing work!
This looks interesting!
What’s the power draw while transcribing?
Hi HN, I’m the author. This runs an open-vocabulary English speech recognizer live on an ESP32-P4: you speak into the board’s mic and the words appear on its screen as you talk. No network, no NPU: just the chip’s two RISC-V cores and their SIMD extension (PIE).
The model is Kroko (by Banafo), a 65.7M-parameter streaming Zipformer2 transducer. As released it is 71 MB in int8, which does not fit the board’s 32 MB of PSRAM, so I quantized the weights to 2 and 4 bits (26.65 MB, groups of 128 with a Hadamard rotation, the bit width chosen per matrix) and wrote my own engine: integer arithmetic throughout, with the hot loops hand-written in PIE assembly.
Numbers, measured on all of LibriSpeech:
test-clean 4.34% WER, test-other 10.26% (the original int8 model on a PC: 3.94% and 9.17%).
1.28 s of audio processed in 0.90 s (0.70x real time), both cores at 400 MHz.
A word is final about 2 s after it is spoken.
One thing I care about: the chip computes exactly the same bits as the engine’s PC build, so every accuracy number above is the chip’s, not a simulation. Getting there meant replacing libm (newlib and glibc disagree in the last bit of log/exp/cos) with deterministic math on both sides.
There are also hotwords (phrases to listen for, from a text file on the SD card) and six voice-control demos driven by simple word rules, no LLM.
Limits: English only; it needs the chip revision v3 (P4X) with 32 MB PSRAM; accuracy is measured on clean read speech, noisy rooms not yet. You can flash it from the browser; the weights are on Hugging Face.
Video: https://www.youtube.com/watch?v=oe-2MowdsSs
Happy to answer questions about the quantization, the kernels or the streaming.
Nice! Thanks for posting. How critical is the 1.28s chunking? Is that to ensure getting a full word? I’d think getting final-word latency down would be a goal. Does the 0.9s processing delay scale down with the sample size?
Kroko ASR team here, great job and thanks for the shoutout! We have some more models coming that might be helpful.