EAI 0.9: INT4 LLM Runtime — 11 tok/s on Cortex-M85
EAI's new quantized inference path squeezes a 1.3B-parameter model into 312 MB of flash and runs at interactive speed on a 480 MHz microcontroller.
The numbers
On a Cortex-M85 running at 480 MHz with 512 KB SRAM and 8 MB PSRAM, the INT4 runtime achieves 11 tokens/second on a 1.3B-parameter GGUF model. Peak memory footprint is 312 MB flash + 4.2 MB SRAM for the KV cache. This is the first time interactive-speed LLM inference has been demonstrated on a microcontroller without an NPU.
Block-streamed inference
The key insight is block-streamed weight loading: instead of mapping the entire model into PSRAM, the runtime streams 4 KB weight blocks from flash into SRAM just-in-time for each attention layer. This reduces peak SRAM usage by 94% compared to a naive full-model load, at the cost of 12% throughput reduction from flash read latency.
Quantization quality
INT4 quantization uses per-channel symmetric quantization with outlier clamping. On the MMLU benchmark, the INT4 model scores 58.2% vs 61.4% for the FP16 reference — a 3.2 percentage point drop. For on-device use cases (intent classification, command parsing, sensor data summarization), this quality level is acceptable.
What this enables
With 11 tok/s on a $4 MCU, natural language device control (eBot) becomes viable without cloud connectivity. A device can parse free-form commands, generate sensor summaries, and respond to queries entirely on-chip. This is the foundation for the eAI Edge stack's offline-first design.
About these articles
How these articles are published
Every article on this site renders through this one page, from the Foundation's shared content registry. The title, date, badge, tags, and body all come from the same data the News page lists — which is why the two can never disagree about what an article says or when it appeared.
The registry replaced eight separate article pages whose metadata had drifted apart: seven of them showed a different publication date than the news listing. One source of truth is less interesting than eight bespoke pages, and considerably more honest.
What an article is
Articles are the long form: explanations of how something works, technical writing about the stack, and reflections on the work. News carries the timely announcement; the article carries the full treatment — the reasoning, the context, and the detail that would not fit in a listing.
Each article states its assumptions and links its evidence where evidence exists. The source material behind the technical claims — the code, the measurements, the filings — is public in the Foundation's repositories.
Dates and addresses
The date on an article is its publication date. The address is /article/ followed by the article's slug, and the older /article-xxx addresses from the previous site still resolve to the same pages — nothing already linked or indexed breaks.
If an article is corrected after publication, the correction is noted rather than silently made — the same standard our press releases are held to. The page you are reading is the current version, and the record of what changed stays with it.