BuzzASR · FFT recipe

Replacing the tokenizer without retraining from zero

Whisper’s tokenizer splits Estonian into many small pieces. FFT gives the model one that fits the language, in three steps: learn a native tokenizer, seed its embeddings from Whisper’s, then fine-tune. The middle step is what lets it keep the pretrained knowledge instead of starting over.

1 Learn a tokenizer

Train BPE on Estonian text. Whole words survive instead of shattering.

Whisper · 8 pieces

↓

Native · 2 pieces

2 Warm-start init

A new token’s embedding starts as the element-wise sum of the old sub-token embeddings.

e(raamatukogu) = Σᵢ e(subᵢ)

3 Fine-tune

Text MTL settles the new vocabulary; ASR maps audio to it.

text onlynext-token LM

audio → textASR

validation CER ↓

Result: native tokenizer, warm-started, fine-tuned. 10 tokens/sentence instead of 28, 2.4× faster to decode, 1.40 CER (SOTA on Estonian).

strategy C (sum warm-start), the configuration deployed across all 102 languages. tokens from frankenstein_v4/estonian.