Qwen3.6-27B-FP8 on One RTX 6000 Ada: Fast TTFT, 668 tok/s Peak Throughput [Benchmark]
Qwen/Qwen3.6-27B-FP8 served through vLLM on a single RTX 6000 Ada is a strong and practical configuration for 8K-context chat serving.
Jul 5, 202613 min read171
![Qwen3.6-27B-FP8 on One RTX 6000 Ada: Fast TTFT, 668 tok/s Peak Throughput [Benchmark]](/_next/image?url=https%3A%2F%2Fcdn.hashnode.com%2Fuploads%2Fcovers%2F6a22b1a041d5b05f16273b50%2F8fd36dcb-515c-4f77-8071-9a1aedc1c2ed.png&w=3840&q=75)
Command Palette
Search for a command to run...
#ai
Articles tagged with #ai
Gemma-4 31B + vLLM on RTX 6000 PRO : 1.17k tokens/sec and still asking for more
Throughput, latency, and queue depth for Gemma-4 31B served on vLLM under progressive load, from 12 to 24 concurrency The numbers that matter: 1.17k tok/s peak, ~0.7s median TTFT, and tail latency as the one thing to watch.
Jun 29, 20265 min read116
Gemma-4 31B it, is Now Available on HexGrid.cloud
Open-source AI has crossed an important line. The question is no longer whether open models are good enough to power serious products. The question is how quickly teams can deploy them privately, rel
Jun 21, 20267 min read17
Deploy Gemma 4 31B-it in One Click: 4-bit, 8-bit, 16-bit Production API
Gemma 4 31B is Google’s high-quality 31B-class instruction model — designed to deliver strong reasoning, coding, multilingual understanding, and reliable instruction-following while remaining lighter
Jun 6, 20264 min read12
Deploy Llama-3.3 70B in One Click: 4-bit, 8-bit, 16-bit Production API
Llama 3.3 70B is Meta’s latest high-quality 70B-class instruction model — designed to deliver strong reasoning, coding, multilingual understanding, and tool-use performance while remaining much more c
Jun 6, 20264 min read11
Deploy Qwen 3.5 27B in One Click: 4-bit, 8-bit, 16-bit Production API
Qwen 3.5 27B is Alibaba's latest mid-size model — competitive with models 2–4x larger on coding and reasoning benchmarks. It fits on a single 48GB GPU at AWQ quantization, making it one of the best qu
Jun 6, 20264 min read11