4 min read
FP8 was twice as fast, so I didn't ship it
When I started s2pro-native, my C rewrite of the Fish Audio S2-Pro speech model, it needed 2.05 seconds of compute for every second of audio. Too slow to talk to. Today it needs 0.51. The fastest version I built along the way never shipped, and that is what this post is about.
The number that matters
Real-time factor, RTF, is compute time divided by audio length. Below 1.0 the voice is generated faster than you hear it, so a stream never stalls. Every frame of audio is 46.4 milliseconds long, so that is the budget a frame has.
Smaller weights
Some of the speed came from plain engineering, like giving the vocoder its own CUDA stream and writing faster kernels for it. But most of the time per frame went into reading the model's weights from memory. So the real way to go faster was smaller weights: from 16-bit to 8-bit, then 4-bit. A frame now reads 4.37 GB instead of 15.5 GB, and a step takes 21.7 ms.
Then FP8
FP8 was twice as fast. On the sentence I tried it on, it sounded fine. But a voice model can be wrong and still sound fine on one sentence, so before every change I compare each layer against the original PyTorch model, on fixed inputs, with a pass mark I set before the run.
Each FP8 matrix multiply on its own looked great: 0.9993 cosine similarity against 16-bit. But the backbone runs about 144 of them in a row, each adding a few percent of noise, and the noise compounds. This is the run, one bar per layer:
By the final norm the output matched the original at 0.33. That is a different model. The first token it picked was wrong, and on the test sentence it stopped before saying anything:
reference FP8 first token 154636 151645 wrong speech tokens 64 0 stopped at onceINT8, same check every layer ≥ 0.99986 passed
It isn't a bug I could fix. The way FP8 scales its blocks just loses too much over a network this deep. So it stayed on my machine, where I still use it to measure the memory system.
What shipped
INT8 passed and carried the project for a while. The 4-bit version only went out after I trained the small decoder to work with its own 4-bit weights, and after it passed the same check, an end-of-speech test over 24 pairs of sentences, and a long round of careful listening.
I write the ideas that failed into the repository too. Next time I want to try one of them, I can read why it didn't work before I spend another week on it.