Qwen3.6-27B-FP8 on One RTX 6000 Ada: Fast TTFT, 668 tok/s Peak Throughput [Benchmark]

Overview

We benchmarked Qwen/Qwen3.6-27B-FP8 using vLLM 0.19 on a single RTX 6000 Ada 48GB GPU.

The goal was to evaluate realistic chat-serving behavior with:

• Streaming enabled

• OpenAI-compatible /v1/chat/completions

• Multi-turn ShareGPT-style prompts

• Multilingual traffic

• Moderate concurrency levels: 8, 12, and 16

This was not a synthetic one-prompt benchmark. The workload used 128 unique ShareGPT prompts across three concurrency settings, for a total of 384 requests.

Benchmark Configuration

Model

Serving Setup

Engine flags:

--tensor-parallel-size 1
--data-parallel-size 1
--tool-call-parser qwen3_coder
--reasoning-parser qwen3
--gpu-memory-utilization 0.90
--max-model-len 8192
--max-num-seqs 16

Hardware

Workload

Results Summary

The most important high-level result:

TTFT was strong, median decode speed was good, and KV cache usage stayed low. The main area to investigate is p99 decode latency.

1. Time to First Token Latency

Time to First Token, or TTFT, measures how long the user waits before streaming begins. For chat applications, this is one of the most important user-experience metrics.

Deductions

Interpretation

This is one of the strongest parts of the benchmark. The model starts responding quickly, which is exactly what you want for a streaming chat system.

Even though the workload used multi-turn ShareGPT-style prompts, TTFT stayed low on average. That suggests request admission and prefill behavior were healthy under the tested concurrency levels.

2. Time Per Output Token Latency

Time Per Output Token, or TPOT, measures decode speed after the first token has been produced. Lower TPOT means faster generation.

Deductions

Interpretation

Median decode performance is solid. A p50 of roughly 34 tokens/sec is good for a 27B-class model running on a single RTX 6000 Ada in FP8.

The main concern is the gap between p50 and p99. Average p99 TPOT is about 4.36× slower than average p50 TPOT.

That means most requests decode well, but a small number of requests experience much slower generation.

Possible causes include:

3. End-to-End Latency

End-to-end latency measures total request completion time, from request start to final token.

Deductions

Interpretation

The E2E numbers look high if viewed as normal short-chat latency, but they make sense for this workload.

The model had a maximum output budget of 1024 tokens, and the workload was multi-turn. Since TTFT was low, most of the end-to-end latency comes from generating the completion rather than waiting for the model to begin.

In other words:

This is not a first-token problem. It is mostly a decode-duration problem.

4. Request Rate

Equivalent approximate rates:

Deductions

Interpretation

This benchmark is best understood as a moderate-concurrency long-chat serving test, not a maximum QPS test.

The system was handling heavy requests with multi-turn context and large output budgets. For capacity planning, more sweeps would be useful: concurrency 20, 24, 32, and maybe 48, depending on stability and KV usage.

5. Token Throughput

Token throughput measures how many prompt and output tokens the system processed per second.

Deductions

Interpretation

Throughput looks good for a single RTX 6000 Ada running a 27B-class FP8 model.

The difference between average and max throughput suggests that the system had moments where batching was more efficient, but average utilization was limited by request shape, output length, and concurrency.

6. KV Cache Usage

KV cache usage is important because it often becomes the limiting factor for long-context and high-concurrency serving.

Deductions

Interpretation

This is an important result. The system was not close to KV cache saturation.

Since max KV cache usage was only about one-third, there is likely room to push harder with:

• Higher concurrency

• More simultaneous requests

• Longer average prompts

• More aggressive batching

The next bottleneck may appear elsewhere first, such as decode scheduling, compute saturation, or latency tail behavior.

Main Lessons

Key Deductions

1. Qwen3.6-27B-FP8 is practical on a single 48GB GPU

The model served successfully on 1× RTX 6000 Ada 48GB with vLLM, streaming enabled, and concurrency levels up to 16.

The combination of FP8 and vLLM makes this a practical single-GPU deployment profile for 27B-class chat serving.

2. TTFT is excellent for streaming chat

The average p50 TTFT was 0.4802 seconds, and p95 was 0.9444 seconds.

That means users would usually see the response begin quickly. For streaming applications, this matters a lot because perceived latency is often dominated by how quickly the first token appears.

3. Decode tail latency is the main issue

The biggest concern is p99 TPOT.

Average p50 TPOT was 0.0292 sec/token, while average p99 TPOT was 0.1274 sec/token.

That is a large jump.

This suggests that most requests perform well, but a small portion of requests experience much slower generation. For production workloads, that tail behavior should be investigated.

4. End-to-end latency is mostly caused by output length

E2E p50 was 32 seconds, but p50 TTFT was only 0.4802 seconds.

That gap makes the explanation clear:

The model starts quickly, but total request completion time is dominated by generating the response.

Since each request allowed up to 1024 output tokens, this is expected.

5. KV cache usage leaves plenty of headroom

KV cache maxed out at only 32.67%.

This means the tested workload did not saturate memory. There is likely room to increase concurrency beyond 16, increase average prompt length, or run a heavier workload before KV cache becomes the bottleneck.

6. This benchmark was realistic, but not a saturation test

The workload used multilingual, multi-turn ShareGPT-style prompts, which is useful for real serving behavior.

However, average request rate was only 0.2123 req/s, and max request rate was 0.4445 req/s. So this should not be treated as the maximum throughput limit of the deployment.

A proper saturation benchmark should include controlled sweeps across concurrency and context length.

Final Verdict

Qwen/Qwen3.6-27B-FP8 served through vLLM on a single RTX 6000 Ada is a strong and practical configuration for 8K-context chat serving.

The benchmark shows:

• Fast first-token latency

• Good median decode speed

• Decent total token throughput

• Low KV cache pressure

• Practical serving behavior at concurrency 8–16

The main issue to investigate is p99 decode latency.

The deployment looks healthy overall, but tail TPOT should be optimized or understood before pushing this into stricter production latency targets.

For a single 48GB workstation/server GPU, this is a very promising result for 27B-class FP8 inference.