back

s2pro-native

You type a sentence and a voice reads it out. The first sound comes about a seventh of a second later, and from there the speech is made faster than you can listen to it. Any voice, many languages, and you can clone a new voice from a few seconds of a recording.

Most speech models come wrapped in a huge stack of Python that breaks every few months. I removed that stack and rewrote the model itself in C and CUDA, every step of the maths. What ships is one program and the model's weights.

s2pro.lukaloehr.com: speech faster than it plays, no Python

Why I did it

With an open speech model, most of what you deploy is the serving stack: a framework, a runtime, a package manager, and versions that will clash in a year. The model itself is a few gigabytes of weights and about a dozen operations. So I wrote those operations myself against Fish Audio S2-Pro: the prompt building, the transformers, the vocoder, the scheduler and the HTTP server. Python, PyTorch, SGLang and vLLM aren't part of it.

The best part is that nothing is in between anymore. When a frame takes 22 milliseconds, the reason is in code I wrote, and I can go and fix it.

What happens to a sentence

text      tokenizer          byte-level BPE, written by hand          backbone           36 layers, picks one token per frame
          decoder             4 layers, turns it into ten codes
          vocoder            DAC, 2,048 samples = 46.4 ms of sound          again, until the sentence is done → audio

The text goes through a tokenizer I wrote by hand. On 4,000 random test cases it gives exactly the same output as the original. That sounds boring, but if it's wrong, everything after it is wrong too.

Then two transformers run for every frame. A 36-layer backbone picks one token, and a 4-layer decoder turns it into ten codes. A DAC vocoder turns those codes into 2,048 audio samples, which is 46.4 milliseconds of sound. This repeats until the sentence is finished.

The audio streams out while it is still being made. The vocoder windows overlap and are crossfaded, so you can't hear the seams. A client can play the first word before the last one exists.

How fast it is

Measured on one DGX Spark (GB10, sm_121), streamed over HTTP:

request                        wall RTF   first audio
new voice from a clip            0.51       0.14 s
named voice, warm cache          0.52       0.20 s
long text (~143 s), in chunks    0.50       0.20 s

RTF is compute time divided by audio length. Below 1.0, speech is made faster than it plays. It started at 2.05. The first version of this page said realtime was impossible with 16-bit weights, and that was true. The way out was smaller weights that are still correct: 8-bit, then a packed 4-bit backbone, and then I trained the decoder and the output layer to work with 4-bit weights too. A frame now reads 4.37 GB of weights instead of 15.5 GB, and a step takes 21.7 ms of its 46.4 ms budget. Up to five streams can run at once before one falls behind.

the first working version2.05
vocoder on its own CUDA stream1.19
4-bit backbone1.10
packed 4-bit, 4.5 bits per weight0.95
faster vocoder kernels0.75
4-bit decoder and output layer0.60
today0.51

I wrote about the way there, and the fast version I threw away, in FP8 was twice as fast, so I didn't ship it.

Checking that it's right

A speech model can be wrong and still sound fine on the one sentence you test. So every change is compared against the original PyTorch model layer by layer, on fixed inputs, with a pass mark I set before the run.

FP8 was twice as fast and failed that check: the error grew over 36 layers until the output only matched with a cosine similarity of 0.33. So it never shipped. INT8 passed everywhere. The 4-bit version passed the same check, an end-of-speech test over 24 pairs of sentences, and careful listening. The ideas that didn't work are written down in the repository too, like speculative multi-frame decoding and a 4-bit KV cache.

Voices

The server comes with 33 named voices in 7 languages. You put a .wav and its transcript into a folder, and that voice can be used by name. It is encoded once at startup and kept warm in a cache, which is why a named voice answers in 0.20 s instead of the 1.58 s it used to take. You can also send a clip with the request and the voice is cloned on the spot. If a text switches language, the voice stays the same.

Status

v1.0.1, stable. It runs as a container: two commands and the server is up, and it downloads the model on the first start. The HTTP API is a fixed OpenAPI 3.1 contract that only changes in compatible ways within 1.x.

The code is Apache-2.0. The weights aren't mine to give away: Fish Audio publishes S2-Pro under their own research license, so the repository downloads them instead of including them.