Accurate Decoding of Natural Sentences from Non-Invasive Brain Recordings

3 min read

Brain2Qwerty v2 decodes typed sentences from MEG recordings alone. Nine healthy volunteers heard a sentence, waited, then typed it, for 10 hours each (about 22,000 sentences over 90 sessions). The pipeline trains three parts together: an asynchronous encoder (a BrainModule followed by a Conformer) trained with CTC so it needs no keystroke timings, a word-level alignment, and an LLM fine-tuned with LoRA that turns neural tokens and CTC output into a sentence. A separate experiment hands parts of the pipeline to coding agents (Claude Opus 4.6 through Cursor) and compares them with Optuna under the same compute.

Findings

  • Word error rate 39% on average and 22% for the best subject, against 52% for the best subject in v1 (Lévy et al. 2025).
  • Data volume matters most. Asynchronous CER falls from 0.59 on the earlier small Spanish dataset to 0.25 on the new English one, and scales log-linearly with hours (slope −0.39 CER per decade, r = −0.99) with no plateau at about 90 pooled hours.
  • Per-subject LoRA adapters averaged into one “model soup” keep improving with rank and LLM size. A single jointly trained adapter overfits the roughly 2.7K unique sentences above rank 2.
  • From a default validation WER of 0.45 on one subject, Optuna stops at 0.41 and its gain vanishes across subjects (test WER 0.493). Three agents reach 0.38, 0.36 and 0.37, and their configurations hold up on all nine subjects (test WER 0.42, 0.45, 0.43). Useful changes they found: label smoothing, modality dropout, beam search over LLM outputs, a sentence-level contrastive loss and minimal prompts.
  • Given an open-ended goal (rebuild the v2 pipeline from the v1 code), the agents failed. Large tangled edits crashed most cluster jobs, and when a job did run they tended to idle.

Limits

  • Healthy typists only. Patients who cannot press keys, in training or at test time, are untested, as the authors say.
  • The model is not causal and decodes whole sentences, so real-time latency is poor.
  • Large spread between subjects (N-gram CER 17.1% to 41.0%), and 306-sensor cryogenic MEG is a lab instrument.
  • When the encoder output is badly corrupted, the LLM writes fluent but wrong sentences. Better semantics costs character accuracy.
  • The agents worked in a search space the researchers fenced off, and humans picked the final configuration.

Relevance

Off-topic for robots on the surface, but two lessons carry over. A language model downstream of a noisy sensor writes a confident, fluent answer when the sensor fails, and a monitor has to catch that (robust robotics, runtime-monitoring). And auto-research agents helped inside a fenced space and failed outside it, matching EmboCoach-Bench and AlphaEvolve.