Running Qwen3-TTS on an AMD R9700

Recently I got an AMD R9700 GPU. There were a couple of reasons I went for it, but the main one was that I thought it was the best-value GPU (besides an Intel B70) with 32 GB VRAM and decent 640 GB/s memory bandwidth.

This is much cheaper than the nearest NVIDIA 32 GB GPU, the 5090, which is about 2.5x the price and comes with much more memory bandwidth, about 1.8 TB/s. I was okay with less. The other reason it is more expensive is that NVIDIA has CUDA, which is the most mature GPU programming software. With AMD, you’re using ROCm or Vulkan.

That CUDA advantage is real, but for what I wanted to do I did not think it was worth the NVIDIA tax. A lot of ML software still supports CUDA first, with AMD support arriving later or requiring some extra work, but that tradeoff feels different now. NVIDIA’s CUDA moat helped power the LLM boom, and those same LLMs make it easier than ever to tailor software for a specific stack that is not CUDA.

Running Qwen3-TTS

I started playing around with the Qwen3-TTS model and was impressed. I had been playing around with TTS models for a few years, but this was the first open one that I thought had caught up to an ElevenLabs-level model. It is also autoregressive, so it supports streaming nicely, which is cool if you want realtime streaming.

There are a few variants of the model (CustomVoice, VoiceDesign, and Base) in different sizes, 0.6B and 1.7B. I started thinking about what the maximum speed I could run the larger model was. Usually when you measure the speed of an LLM, you are measuring the tokens/second output (tg). For an audio model it’s a little different: we want to know how much audio we can generate per unit time, or what is known as RTF (audio generation time / audio length). So if we can generate 2s of audio per second, we get an RTF of 0.5.

To understand the theoretical maximum speed, it helps to think of Qwen3-TTS as producing audio in small chunks.

The model does not generate waveform samples directly from text. It generates a compressed version of the next chunk of audio, then a codec decoder expands that into waveform samples. I will call one of those compressed chunks an acoustic frame. Each acoustic frame represents about 80 ms of audio.

24,000 samples/sec (24 kHz) / 1,920 samples per frame = 12.5 frames/sec

So for realtime audio, the system must produce and decode at least:

12.5 acoustic frames per second

Each acoustic frame contains 16 codebook tokens:

[semantic token, acoustic token 1, acoustic token 2, ..., acoustic token 15]

The generation loop looks like this:

The Talker is the main text-to-speech transformer. It reads the text, speaker, and language conditioning, then generates the first codebook token for each acoustic frame. This first token is the semantic/control token for that frame.

The CodePredictor is a smaller transformer that fills in the remaining 15 acoustic codebook tokens for that same frame. These tokens describe the detailed acoustic content needed by the codec decoder.

Once the Talker and CodePredictor have produced a complete frame, the codec decoder expands that compressed frame into waveform samples. In this implementation, each decoded frame becomes 1,920 samples, or about 80 ms of audio at 24 kHz.

The model weights are not split evenly across those stages. In fp16, the Talker excluding the CodePredictor is about 3.483 GB, the CodePredictor is about 0.350 GB, and the speech tokenizer / codec is about 0.341 GB, for a total of 4.174 GB.

So how fast can we generate a frame? For simplicity, if we say the time to generate a frame is:

bytes_read_per_frame ~= bytes(Talker once) + 15 * bytes(CodePredictor once)

Then the rough fp16 bandwidth cost per generated frame is:

3.483 GB + 15 * 0.350 GB = 8.733 GB per frame

Given the R9700 memory bandwidth of 640 GB/s, and assuming we are memory bandwidth bound, we get:

640 GB/s / 8.733 GB = 73.3 frames per second

Since realtime is 12.5 frames per second, this is about 5.9x realtime. So in a perfect world where everything is running on the GPU and we are only bound by memory bandwidth, we could get an RTF of ~0.17. These are pretty crude estimates but it gives us a sense of the potential.

So how fast does it run out of the box?

I got GPT 5.5 to write a script to run the 1.7B CustomVoice model with the Qwen3-TTS reference implementation. After a warmup iteration with an RTF of 6.22, we were able to get an RTF of 1.90. There are some kernel optimisations happening in the warmup iteration, which is why it’s so slow and spits out a few warnings:

MIOpen(HIP): Warning [IsEnoughWorkspace] [GetSolutionsFallback WTI] Solver <GemmFwdRest>, workspace required: 105283584, provided ptr: 0 size: 0

This is what the non-streaming baseline reference output sounded like:

For a fairer comparison to the system I want to build, I also ran the reference implementation in streaming mode. This version has a rough start: because streaming feeds the model one text token at a time, it can stumble at the beginning before it has enough context, then it usually gains momentum.

There is an issue open here which goes into some details about the performance issues and potential solutions. There is also another implementation, faster-qwen3-tts, which states that it is able to get a much better RTF using Torch/CUDA graphs to capture repeated kernel launches, but when I tried it, it didn’t seem to make a difference.

At this point I started thinking about how fast we could get this running by cutting out the layers between the model implementation and ROCm. I was inspired by ds4, where antirez targets a specific model and hardware setup and manages to squeeze maximum performance out of it. I think this kind of problem is well suited for LLMs to write the code and optimize it, since you have good reference code, good guard rails and a clear direction to go in.

The next logical step…

Rewrite in Rust

You could do this in a number of languages, but I went with Rust because I am familiar with it and it has good FFI support for calling C/C++ code, which means we can use the HIP API. HIP is AMD’s API for GPU programming and has a very similar interface to CUDA, e.g. hipMalloc vs cudaMalloc. The choice of language here isn’t super important because ideally the CPU is mostly launching kernels, while the GPU does the heavy work.

After some back and forth with GPT 5.5 we had a plan for what the library would look like, how it would be structured and the API it would expose. Then I set it off to work on porting it to Rust using the HIP API. This took about a day before we had an end-to-end working implementation, which produced this:

I’m still pretty amazed that this works, let alone at an RTF of 0.55. It feels like I’m glossing over a bunch of the details from building this, but it was really just GPT 5.5 working through the steps in a plan and waiting for my session limit to refresh. We then implemented streaming support and added a little server that I could use to test everything out. I’m able to stream the same audio as above with a time to first audio of about a second and an RTF of 0.76. You can get the RTF down here by changing the chunk frames (the number of frames before we decode to audio). The tradeoff is that the TTFA goes up, since you’ve got to wait longer before the first decode.

Conclusion

This was a fun project and I learnt a few interesting things. LLMs have changed the economics of software development and created opportunities for very niche things like this to be built. I can’t really think of conditions in the past where building something like this would have made sense. Maybe if you were a very large company and had a team that could build this quickly and cheaply, but the shelf life of models makes it hard to justify that sort of investment. In the past it’s only really been viable to build layers of abstraction which work across a bunch of different hardware, but at each layer you’re kind of losing some precision, which can compound.

This is the first project where I’ve taken my hands off the wheel during the implementation and let the model do its thing. I know the rough structure, but I don’t know the nitty-gritty details, although I am really not too worried about that for this sort of project. We have output parity with the Python implementation with better performance, so do I need to be across all the details? I’m not sure.

Another thing that I think about is why this even works in the first place? The model architecture here seems kind of arbitrary, people have done work on trying to interpret what is happening inside these models but it still just seems like such early days for this sort of thing. There is a massive space of possible architectures out there and we have explored a few, it just seems like there is a lot of room for these things to become faster and cheaper.