back

4 min read

FP8 was twice as fast, so I didn't ship it

When I started s2pro-native, my C rewrite of the Fish Audio S2-Pro speech model, it needed 2.05 seconds of compute for every second of audio. Too slow to talk to. Today it needs 0.51. The fastest version I built along the way never shipped, and that is what this post is about.

The number that matters

Real-time factor, RTF, is compute time divided by audio length. Below 1.0 the voice is generated faster than you hear it, so a stream never stalls. Every frame of audio is 46.4 milliseconds long, so that is the budget a frame has.

Smaller weights

Some of the speed came from plain engineering, like giving the vocoder its own CUDA stream and writing faster kernels for it. But most of the time per frame went into reading the model's weights from memory. So the real way to go faster was smaller weights: from 16-bit to 8-bit, then 4-bit. A frame now reads 4.37 GB instead of 15.5 GB, and a step takes 21.7 ms.

the first working version2.05
vocoder on its own stream1.19
packed 4-bit backbone0.95
faster vocoder kernels0.75
everything 4-bit0.51

Then FP8

FP8 was twice as fast. On the sentence I tried it on, it sounded fine. But a voice model can be wrong and still sound fine on one sentence, so before every change I compare each layer against the original PyTorch model, on fixed inputs, with a pass mark I set before the run.

Each FP8 matrix multiply on its own looked great: 0.9993 cosine similarity against 16-bit. But the backbone runs about 144 of them in a row, each adding a few percent of noise, and the noise compounds. This is the run, one bar per layer:

1.000.750.500.25INT8, every layerlayer 018final norm 0.33

By the final norm the output matched the original at 0.33. That is a different model. The first token it picked was wrong, and on the test sentence it stopped before saying anything:

                     reference     FP8
first token          154636        151645      wrong
speech tokens        64            0           stopped at onceINT8, same check     every layer ≥ 0.99986    passed

It isn't a bug I could fix. The way FP8 scales its blocks just loses too much over a network this deep. So it stayed on my machine, where I still use it to measure the memory system.

What shipped

INT8 passed and carried the project for a while. The 4-bit version only went out after I trained the small decoder to work with its own 4-bit weights, and after it passed the same check, an end-of-speech test over 24 pairs of sentences, and a long round of careful listening.

I write the ideas that failed into the repository too. Next time I want to try one of them, I can read why it didn't work before I spend another week on it.