Concurrent request latency significantly higher than sequential in vLLM TTS

[odgran]

Great Work! I’m serving the vLLM model for a realtime TTS use-case; and noticed that concurrent requests experience much higher latency compared to sequential requests, even though the model and hardware remain the same. This affects real-time performance.

Environment / Setup:

• Hardware: H100

• Benchmark:

  • Sequential: 5 runs
  • Concurrent: 12 simultaneous requests (in 5 round)

• Average audio duration: ~12.3s

Observed Results:

--- Sequential Benchmark Results ---
TTFB (Mean): 0.130s
Total Time (Mean): 5.807s
RTF (Mean): 0.469

--- Concurrent Benchmark Results ---
TTFB (Mean): 0.667s
Total Time (Mean): 17.228s
RTF (Mean): 1.403

TTFB Difference: +0.537s (+413%)

Clarification:

• Concurrent requests should scale better. Ideally, TTFB and total time should not increase dramatically under load. Am i correct?
• Are there recommended settings for improving concurrent throughput (batch size, async workers, streaming configuration)?
• Are there known limitations in vLLM when handling multiple simultaneous TTS requests?
• Any advice to optimize TTFB and RTF under high concurrency?

Thanks in advance!

[taresh18]

Hey, thanks for trying it out.

• With orpheus, the llama backbone generates the tokens which are then decoded into audio chunks using the snac decoder. vllm handles batching efficiently, the major bottleneck is the snac decoder. The issue becomes much more apparent if you're running the decoding and vllm inference on the same GPU with large number of concurrent requests.
• The easiest fix would be to just increase the workers used to run the app (ideally equal to the number of concurrent requests you're expecting to server). This will load multiple as many snac decoders and will bring down the latency.
• If you wanna serve even higher number of requests, try loading the snac decoding on a different gpu altogether. so you will need a multi-gpu machine where vllm will be on gpu0 and snac decoding on gpu1. This will reduce the GPU contention between vllm inference and snac decoding.

Try these methods and let me know how it goes

[odgran]

Thanks for the earlier insights! I did some further experiments to check whether concurrency overhead is due to GPU contention or worker scaling limits.

• Increasing just the workers didn’t help much - TTFB and RTF still rise with concurrency, which might be due to single-GPU contention.

• I then tried a multi-GPU setup:

Setup:

• vLLM (1st H100 GPU):

  vllm serve model1 \
    --max-model-len 2048 \
    --gpu-memory-utilization 0.9 \
    --host 0.0.0.0 --port 9191 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 48 \
    --enable-chunked-prefill \
    --block-size 16 \
    --enable-prefix-caching
  

• Decoder (2nd H100 GPU):

  uvicorn app:app --host 0.0.0.0 --port 9090 \
    --workers 12 --loop uvloop --http httptools \
    --log-level warning --access-log
  

• Decoder (3rd H100 GPU): same as above

---

Results (Sequential vs Concurrent, 8 requests × 5 rounds, 12 workers of decoder)

Metric	Sequential (2nd, 3rd GPU)	Concurrent (2nd, 3rd GPU)	Δ (Concurrent - Sequential) (2nd, 3rd GPU)
TTFB (mean)	0.109s, 0.087s	0.197s, 0.163s	+0.088, +0.075
Total Time (mean)	1.346s, 1.274s	1.660s , 1.906s	+0.315, +0.632
RTF (mean)	0.380, 0.377	0.473, 0.534	+0.093, +0.157

Results (Sequential vs Concurrent, 16 requests × 5 rounds, 24 workers of decoder)

Metric	Sequential (2nd, 3rd GPU)	Concurrent (2nd, 3rd GPU)	Δ (Concurrent - Sequential) (2nd, 3rd GPU)
TTFB (mean)	0.087s, 0.113s	0.254s, 0.302s	+0.168, +0.189
Total Time (mean)	1.361s, 1.384s	2.313s, 2.439s	+0.952, +1.055
RTF (mean)	0.384, 0.381	0.662, 0.718	+0.278, +0.338

→ At higher concurrency, decoder GPUs show contention as well.

From these stats, it looks like the bottleneck shifts:

• First, vLLM itself shows contention under load (as 2nd GPU only one went fine with one vLLM, stats were normal (8 requests × 5 rounds, 12 workers of decoder)).

• With more decoder workers + requests, contention also appears on the decoder side.

Is my interpretation correct?
And is there any recommended way to balance decoder worker scaling with a single vLLM node? (e.g., is there a hard relation between #decoder workers and the throughput that one vLLM node can realistically sustain, as vRAM was available when more than 50% on 2nd and 3rd GPU?)

[taresh18]

Hey, can you also log the vllm ttft for each request? It will help use know if the issue is with vllm or decoding. If the issue is with vllm, you can give tesnorrt-llm a try. I have integrated trt-llm serve in the main branch. However, I think snac is causing the bottleneck. Nowadays, I am working on kyut ai TTS., found it to have much better voice quality particularly for realtime use-case. You can give that a try as well.

[shekharmeena2896]

Hey I have finetuned the orpheus tts model of hindi on the custom dataset, I am confused as I want to make it realtime how should I do it, as I need to serve over the intelligent ai for telephony, so I have one h100 gpu 80gb, how many concurrent or parallel request I can do, how should I do it, do I need to quantise the model

[taresh18]

@shekharmeena2896 follow the instructions and try to setup this fast api server plus vllm. You should get pretty good latency on h100, more than enough for any real-time use-case

[shekharmeena2896]

Hi I believe tensorRT-llm is more robust for production as it can serve the parallel requests, customised cuda kernels

[taresh18]

@shekharmeena2896 vllm can serve parallel requests as well and its more than production ready. if you're gonna use tensorrt-llm, you will need triton inference server and choose tesnorrt backend for inference. You can go ahead and use that as well, although I would recommend starting with vllm usnleass you have prior experience with triton / tensorrt.

[shekharmeena2896]

I followed your step this is what I am getting :
Voice: tara
Runs: 5
Base URL: http://localhost:9090
Note: Converting raw audio streams to WAV format with proper headers

Running warmup...
Warmup complete: TTFB 0.089s, Total 0.700s

Starting 5 benchmark runs...
Run 1/5... -> Saving 573440 bytes to outputs/run_1.wav TTFB: 0.088s, Total: 5.350s
Run 2/5... -> Saving 548864 bytes to outputs/run_2.wav TTFB: 0.087s, Total: 5.116s
Run 3/5... -> Saving 499712 bytes to outputs/run_3.wav TTFB: 0.088s, Total: 4.652s
Run 4/5... -> Saving 532480 bytes to outputs/run_4.wav TTFB: 0.087s, Total: 4.960s
Run 5/5... -> Saving 581632 bytes to outputs/run_5.wav TTFB: 0.087s, Total: 5.427s

--- Benchmark Results (5/5 successful) ---

Audio Characteristics:

• Sample Rate: 24000Hz, 16-bit, 1 channel(s)
• Average Audio Duration: 11.401s

Performance Metrics:

• TTFB (Time to First Byte):
  • Mean: 0.088s, StdDev: 0.000s, Min: 0.087s, Max: 0.088s
• Total Generation Time:
  • Mean: 5.101s, StdDev: 0.312s, Min: 4.652s, Max: 5.427s
• RTF (Real-Time Factor):
  • Mean: 0.447, StdDev: 0.000, Min: 0.447, Max: 0.448
    (Lower is better; RTF < 1 means faster than real-time generation)

Audio for each run saved in 'outputs/'

I am currently using orignal orpheus model what u have defined not my finetuned one.
as I want to run multiple parallel request what steps should I take and should also quantise the model in fp8 using tensorrt llm

when I changed the script and tried to hit 5 parallel request it gave me this output:
Run 2: TTFB=0.091s, Total=5.185s, Audio=11.52s
Run 3: TTFB=5.248s, Total=10.333s, Audio=11.52s
Run 1: TTFB=10.400s, Total=14.869s, Audio=10.15s
Run 4: TTFB=14.929s, Total=20.095s, Audio=11.69s
Run 5: TTFB=20.311s, Total=25.029s, Audio=10.75s

--- Parallel Benchmark Results ---
Requests: 5/5 successful
TTFB mean=10.196s, min=0.091s, max=20.311s
Total mean=15.102s, min=5.185s, max=25.029s
RTF mean=1.372
Audio saved in 'outputs_parallel/'
(trt) root@snapshots-gpu-h200x1-141g

[taresh18]

@shekharmeena2896 try the other branch - integration/trtllm-serve

[shekharmeena2896]

I tried the other branch , the instructions are pretty vague, can't seem to execute it

[shao-ssq]

It seems the concurrency results are highly inconsistent, which makes it questionable for production use.