=== Qwen/Qwen3-32B-AWQ, one Spark (TP=1) vs two Sparks (TP=2) 2026-11-12 === hardware: GPUwerk Spark Cluster 2x rental, two GB10 machines (128 GB each), one ConnectX-7 cable (200 Gb/s ports), partner-hosted pair in the United States, DGX OS 7 image software: vLLM 0.33.0 (pip), PyTorch 2.14.0+cu130, NVIDIA driver 580.159.03, FlashInfer enabled server: --max-model-len 8192 --gpu-memory-utilization 0.85; TP=2 with vLLM's multiprocessing backend (--nnodes 2), NCCL over RoCE kv cache: 338,112 tokens on one node (TP=1); 731,392 tokens across two (TP=2) workload: vllm bench serve, random dataset, 1024 input / 256 output tokens, --ignore-eos, --seed 1 sanitised: NCCL lines reduced to the result table (host names and addresses removed); the bench client ran on the node that served the model (127.0.0.1) ### TP=1, max concurrency 1 Namespace(subparser='bench', bench_type='serve', dispatch_function=, trust_remote_code=False, seed=1, num_prompts=8, dataset_name='random', no_stream=False, dataset_path=None, no_oversample=False, skip_chat_template=False, enable_multimodal_chat=False, disable_shuffle=False, custom_output_len=256, custom_ensure_client_side_data=False, spec_bench_output_len=256, spec_bench_category=None, sharegpt_output_len=None, timed_trace_chunk_hash_size=16, timed_trace_sec_multiplier=1, timed_trace_label_timestamp='timestamp', timed_trace_label_input_length='input_length', timed_trace_label_output_length='output_length', timed_trace_label_hash_ids='hash_ids', blazedit_min_distance=0.0, blazedit_max_distance=1.0, asr_max_audio_len_sec=inf, asr_min_audio_len_sec=0.0, random_input_len=1024, random_output_len=256, random_range_ratio='0.0', random_prefix_len=0, random_batch_size=1, no_reranker=False, random_mm_base_items_per_request=1, random_mm_num_mm_items_range_ratio=0.0, random_mm_limit_mm_per_prompt={'image': 255, 'video': 1}, random_mm_bucket_config={(256, 256, 1): 0.5, (720, 1280, 1): 0.5, (720, 1280, 16): 0.0}, hf_subset=None, hf_split=None, hf_name=None, hf_output_len=None, bfcl_categories=None, prefix_repetition_prefix_len=256, prefix_repetition_suffix_len=256, prefix_repetition_num_prefixes=10, prefix_repetition_output_len=128, speed_bench_dataset_subset='qualitative', speed_bench_output_len=4096, speed_bench_category=None, label=None, backend='vllm', base_url='http://127.0.0.1:8000', host='127.0.0.1', port=8000, endpoint='/v1/completions', header=None, max_concurrency=1, model='/workspace/models/qwen3-32b-awq', input_len=None, output_len=None, tokenizer=None, tokenizer_mode='auto', use_beam_search=False, logprobs=None, request_rate=inf, burstiness=1.0, probe_request_rate=0.0, disable_tqdm=False, num_warmups=0, profile=False, save_result=True, save_detailed=False, append_result=False, metadata=None, result_dir='/workspace/bench/results', result_filename='tp1_c1.json', ignore_eos=True, self_timed=None, percentile_metrics='ttft,tpot', metric_percentiles='99', goodput=None, request_id_prefix='bench-bc8ce4eb-', top_p=None, top_k=None, min_p=None, temperature=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, served_model_name=None, lora_modules=None, lora_assignment='random', ramp_up_strategy=None, ramp_up_start_rps=None, ramp_up_end_rps=None, ready_check_timeout_sec=0, chat_template_kwargs=None, extra_body=None, skip_tokenizer_init=False, insecure=False, plot_timeline=False, timeline_itl_thresholds='25,50', plot_dataset_stats=False, log_config_file=None, logging_config=LoggingConfig(log_level='INFO', formatter='text', configure_logging=True, pylogging_config_file=None)) (MainProcess pid=3895) INFO 11-12 12:40:23 [utils.py:88] Sampling input_len from [1024, 1024] and output_len from [256, 256] WARNING: vllm bench serve no longer sets temperature==0 (greedy) in requests by default. The default will be determined on the server side and can be model/API specific. For the old behavior, include --temperature=0. Starting initial single prompt test run... Skipping endpoint ready check. Starting main benchmark run... Traffic request rate: inf Burstiness factor: 1.0 (Poisson process) Maximum request concurrency: 1 tip: install termplotlib and gnuplot to plot the metrics ============ Serving Benchmark Result ============ Successful requests: 8 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 178.49 Total input tokens: 8192 Total generated tokens: 2048 Request throughput (req/s): 0.04 Output token throughput (tok/s): 11.47 Peak output token throughput (tok/s): 13.00 Peak concurrent requests: 2.00 Total token throughput (tok/s): 57.37 ---------------Time to First Token---------------- Mean TTFT (ms): 1188.19 Median TTFT (ms): 1187.48 P99 TTFT (ms): 1192.98 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 82.83 Median TPOT (ms): 82.83 P99 TPOT (ms): 83.03 ================================================== ### TP=1, max concurrency 4 Namespace(subparser='bench', bench_type='serve', dispatch_function=, trust_remote_code=False, seed=1, num_prompts=16, dataset_name='random', no_stream=False, dataset_path=None, no_oversample=False, skip_chat_template=False, enable_multimodal_chat=False, disable_shuffle=False, custom_output_len=256, custom_ensure_client_side_data=False, spec_bench_output_len=256, spec_bench_category=None, sharegpt_output_len=None, timed_trace_chunk_hash_size=16, timed_trace_sec_multiplier=1, timed_trace_label_timestamp='timestamp', timed_trace_label_input_length='input_length', timed_trace_label_output_length='output_length', timed_trace_label_hash_ids='hash_ids', blazedit_min_distance=0.0, blazedit_max_distance=1.0, asr_max_audio_len_sec=inf, asr_min_audio_len_sec=0.0, random_input_len=1024, random_output_len=256, random_range_ratio='0.0', random_prefix_len=0, random_batch_size=1, no_reranker=False, random_mm_base_items_per_request=1, random_mm_num_mm_items_range_ratio=0.0, random_mm_limit_mm_per_prompt={'image': 255, 'video': 1}, random_mm_bucket_config={(256, 256, 1): 0.5, (720, 1280, 1): 0.5, (720, 1280, 16): 0.0}, hf_subset=None, hf_split=None, hf_name=None, hf_output_len=None, bfcl_categories=None, prefix_repetition_prefix_len=256, prefix_repetition_suffix_len=256, prefix_repetition_num_prefixes=10, prefix_repetition_output_len=128, speed_bench_dataset_subset='qualitative', speed_bench_output_len=4096, speed_bench_category=None, label=None, backend='vllm', base_url='http://127.0.0.1:8000', host='127.0.0.1', port=8000, endpoint='/v1/completions', header=None, max_concurrency=4, model='/workspace/models/qwen3-32b-awq', input_len=None, output_len=None, tokenizer=None, tokenizer_mode='auto', use_beam_search=False, logprobs=None, request_rate=inf, burstiness=1.0, probe_request_rate=0.0, disable_tqdm=False, num_warmups=0, profile=False, save_result=True, save_detailed=False, append_result=False, metadata=None, result_dir='/workspace/bench/results', result_filename='tp1_c4.json', ignore_eos=True, self_timed=None, percentile_metrics='ttft,tpot', metric_percentiles='99', goodput=None, request_id_prefix='bench-bb16af32-', top_p=None, top_k=None, min_p=None, temperature=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, served_model_name=None, lora_modules=None, lora_assignment='random', ramp_up_strategy=None, ramp_up_start_rps=None, ramp_up_end_rps=None, ready_check_timeout_sec=0, chat_template_kwargs=None, extra_body=None, skip_tokenizer_init=False, insecure=False, plot_timeline=False, timeline_itl_thresholds='25,50', plot_dataset_stats=False, log_config_file=None, logging_config=LoggingConfig(log_level='INFO', formatter='text', configure_logging=True, pylogging_config_file=None)) (MainProcess pid=3972) INFO 11-12 12:43:26 [utils.py:88] Sampling input_len from [1024, 1024] and output_len from [256, 256] WARNING: vllm bench serve no longer sets temperature==0 (greedy) in requests by default. The default will be determined on the server side and can be model/API specific. For the old behavior, include --temperature=0. Starting initial single prompt test run... Skipping endpoint ready check. Starting main benchmark run... Traffic request rate: inf Burstiness factor: 1.0 (Poisson process) Maximum request concurrency: 4 tip: install termplotlib and gnuplot to plot the metrics ============ Serving Benchmark Result ============ Successful requests: 16 Failed requests: 0 Maximum request concurrency: 4 Benchmark duration (s): 97.24 Total input tokens: 16384 Total generated tokens: 4096 Request throughput (req/s): 0.16 Output token throughput (tok/s): 42.12 Peak output token throughput (tok/s): 48.00 Peak concurrent requests: 8.00 Total token throughput (tok/s): 210.60 ---------------Time to First Token---------------- Mean TTFT (ms): 1758.31 Median TTFT (ms): 813.23 P99 TTFT (ms): 4827.34 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 88.30 Median TPOT (ms): 85.15 P99 TPOT (ms): 98.52 ================================================== ### TP=1, max concurrency 16 Namespace(subparser='bench', bench_type='serve', dispatch_function=, trust_remote_code=False, seed=1, num_prompts=64, dataset_name='random', no_stream=False, dataset_path=None, no_oversample=False, skip_chat_template=False, enable_multimodal_chat=False, disable_shuffle=False, custom_output_len=256, custom_ensure_client_side_data=False, spec_bench_output_len=256, spec_bench_category=None, sharegpt_output_len=None, timed_trace_chunk_hash_size=16, timed_trace_sec_multiplier=1, timed_trace_label_timestamp='timestamp', timed_trace_label_input_length='input_length', timed_trace_label_output_length='output_length', timed_trace_label_hash_ids='hash_ids', blazedit_min_distance=0.0, blazedit_max_distance=1.0, asr_max_audio_len_sec=inf, asr_min_audio_len_sec=0.0, random_input_len=1024, random_output_len=256, random_range_ratio='0.0', random_prefix_len=0, random_batch_size=1, no_reranker=False, random_mm_base_items_per_request=1, random_mm_num_mm_items_range_ratio=0.0, random_mm_limit_mm_per_prompt={'image': 255, 'video': 1}, random_mm_bucket_config={(256, 256, 1): 0.5, (720, 1280, 1): 0.5, (720, 1280, 16): 0.0}, hf_subset=None, hf_split=None, hf_name=None, hf_output_len=None, bfcl_categories=None, prefix_repetition_prefix_len=256, prefix_repetition_suffix_len=256, prefix_repetition_num_prefixes=10, prefix_repetition_output_len=128, speed_bench_dataset_subset='qualitative', speed_bench_output_len=4096, speed_bench_category=None, label=None, backend='vllm', base_url='http://127.0.0.1:8000', host='127.0.0.1', port=8000, endpoint='/v1/completions', header=None, max_concurrency=16, model='/workspace/models/qwen3-32b-awq', input_len=None, output_len=None, tokenizer=None, tokenizer_mode='auto', use_beam_search=False, logprobs=None, request_rate=inf, burstiness=1.0, probe_request_rate=0.0, disable_tqdm=False, num_warmups=0, profile=False, save_result=True, save_detailed=False, append_result=False, metadata=None, result_dir='/workspace/bench/results', result_filename='tp1_c16.json', ignore_eos=True, self_timed=None, percentile_metrics='ttft,tpot', metric_percentiles='99', goodput=None, request_id_prefix='bench-c0aad9b0-', top_p=None, top_k=None, min_p=None, temperature=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, served_model_name=None, lora_modules=None, lora_assignment='random', ramp_up_strategy=None, ramp_up_start_rps=None, ramp_up_end_rps=None, ready_check_timeout_sec=0, chat_template_kwargs=None, extra_body=None, skip_tokenizer_init=False, insecure=False, plot_timeline=False, timeline_itl_thresholds='25,50', plot_dataset_stats=False, log_config_file=None, logging_config=LoggingConfig(log_level='INFO', formatter='text', configure_logging=True, pylogging_config_file=None)) (MainProcess pid=4047) INFO 11-12 12:45:08 [utils.py:88] Sampling input_len from [1024, 1024] and output_len from [256, 256] WARNING: vllm bench serve no longer sets temperature==0 (greedy) in requests by default. The default will be determined on the server side and can be model/API specific. For the old behavior, include --temperature=0. Starting initial single prompt test run... Skipping endpoint ready check. Starting main benchmark run... Traffic request rate: inf Burstiness factor: 1.0 (Poisson process) Maximum request concurrency: 16 tip: install termplotlib and gnuplot to plot the metrics ============ Serving Benchmark Result ============ Successful requests: 64 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 161.71 Total input tokens: 65536 Total generated tokens: 16384 Request throughput (req/s): 0.40 Output token throughput (tok/s): 101.32 Peak output token throughput (tok/s): 160.00 Peak concurrent requests: 32.00 Total token throughput (tok/s): 506.59 ---------------Time to First Token---------------- Mean TTFT (ms): 6364.61 Median TTFT (ms): 7058.68 P99 TTFT (ms): 18843.51 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 133.25 Median TPOT (ms): 143.34 P99 TPOT (ms): 170.09 ================================================== ### TP=2, max concurrency 1 Namespace(subparser='bench', bench_type='serve', dispatch_function=, trust_remote_code=False, seed=1, num_prompts=8, dataset_name='random', no_stream=False, dataset_path=None, no_oversample=False, skip_chat_template=False, enable_multimodal_chat=False, disable_shuffle=False, custom_output_len=256, custom_ensure_client_side_data=False, spec_bench_output_len=256, spec_bench_category=None, sharegpt_output_len=None, timed_trace_chunk_hash_size=16, timed_trace_sec_multiplier=1, timed_trace_label_timestamp='timestamp', timed_trace_label_input_length='input_length', timed_trace_label_output_length='output_length', timed_trace_label_hash_ids='hash_ids', blazedit_min_distance=0.0, blazedit_max_distance=1.0, asr_max_audio_len_sec=inf, asr_min_audio_len_sec=0.0, random_input_len=1024, random_output_len=256, random_range_ratio='0.0', random_prefix_len=0, random_batch_size=1, no_reranker=False, random_mm_base_items_per_request=1, random_mm_num_mm_items_range_ratio=0.0, random_mm_limit_mm_per_prompt={'image': 255, 'video': 1}, random_mm_bucket_config={(256, 256, 1): 0.5, (720, 1280, 1): 0.5, (720, 1280, 16): 0.0}, hf_subset=None, hf_split=None, hf_name=None, hf_output_len=None, bfcl_categories=None, prefix_repetition_prefix_len=256, prefix_repetition_suffix_len=256, prefix_repetition_num_prefixes=10, prefix_repetition_output_len=128, speed_bench_dataset_subset='qualitative', speed_bench_output_len=4096, speed_bench_category=None, label=None, backend='vllm', base_url='http://127.0.0.1:8000', host='127.0.0.1', port=8000, endpoint='/v1/completions', header=None, max_concurrency=1, model='/workspace/models/qwen3-32b-awq', input_len=None, output_len=None, tokenizer=None, tokenizer_mode='auto', use_beam_search=False, logprobs=None, request_rate=inf, burstiness=1.0, probe_request_rate=0.0, disable_tqdm=False, num_warmups=0, profile=False, save_result=True, save_detailed=False, append_result=False, metadata=None, result_dir='/workspace/bench/results', result_filename='tp2_c1.json', ignore_eos=True, self_timed=None, percentile_metrics='ttft,tpot', metric_percentiles='99', goodput=None, request_id_prefix='bench-2f6988c4-', top_p=None, top_k=None, min_p=None, temperature=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, served_model_name=None, lora_modules=None, lora_assignment='random', ramp_up_strategy=None, ramp_up_start_rps=None, ramp_up_end_rps=None, ready_check_timeout_sec=0, chat_template_kwargs=None, extra_body=None, skip_tokenizer_init=False, insecure=False, plot_timeline=False, timeline_itl_thresholds='25,50', plot_dataset_stats=False, log_config_file=None, logging_config=LoggingConfig(log_level='INFO', formatter='text', configure_logging=True, pylogging_config_file=None)) (MainProcess pid=7826) INFO 11-12 13:36:38 [utils.py:88] Sampling input_len from [1024, 1024] and output_len from [256, 256] WARNING: vllm bench serve no longer sets temperature==0 (greedy) in requests by default. The default will be determined on the server side and can be model/API specific. For the old behavior, include --temperature=0. Starting initial single prompt test run... Skipping endpoint ready check. Starting main benchmark run... Traffic request rate: inf Burstiness factor: 1.0 (Poisson process) Maximum request concurrency: 1 tip: install termplotlib and gnuplot to plot the metrics ============ Serving Benchmark Result ============ Successful requests: 8 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 101.89 Total input tokens: 8192 Total generated tokens: 2048 Request throughput (req/s): 0.08 Output token throughput (tok/s): 20.10 Peak output token throughput (tok/s): 22.00 Peak concurrent requests: 2.00 Total token throughput (tok/s): 100.50 ---------------Time to First Token---------------- Mean TTFT (ms): 686.56 Median TTFT (ms): 686.60 P99 TTFT (ms): 692.76 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 47.25 Median TPOT (ms): 47.25 P99 TPOT (ms): 47.46 ================================================== ### TP=2, max concurrency 4 Namespace(subparser='bench', bench_type='serve', dispatch_function=, trust_remote_code=False, seed=1, num_prompts=16, dataset_name='random', no_stream=False, dataset_path=None, no_oversample=False, skip_chat_template=False, enable_multimodal_chat=False, disable_shuffle=False, custom_output_len=256, custom_ensure_client_side_data=False, spec_bench_output_len=256, spec_bench_category=None, sharegpt_output_len=None, timed_trace_chunk_hash_size=16, timed_trace_sec_multiplier=1, timed_trace_label_timestamp='timestamp', timed_trace_label_input_length='input_length', timed_trace_label_output_length='output_length', timed_trace_label_hash_ids='hash_ids', blazedit_min_distance=0.0, blazedit_max_distance=1.0, asr_max_audio_len_sec=inf, asr_min_audio_len_sec=0.0, random_input_len=1024, random_output_len=256, random_range_ratio='0.0', random_prefix_len=0, random_batch_size=1, no_reranker=False, random_mm_base_items_per_request=1, random_mm_num_mm_items_range_ratio=0.0, random_mm_limit_mm_per_prompt={'image': 255, 'video': 1}, random_mm_bucket_config={(256, 256, 1): 0.5, (720, 1280, 1): 0.5, (720, 1280, 16): 0.0}, hf_subset=None, hf_split=None, hf_name=None, hf_output_len=None, bfcl_categories=None, prefix_repetition_prefix_len=256, prefix_repetition_suffix_len=256, prefix_repetition_num_prefixes=10, prefix_repetition_output_len=128, speed_bench_dataset_subset='qualitative', speed_bench_output_len=4096, speed_bench_category=None, label=None, backend='vllm', base_url='http://127.0.0.1:8000', host='127.0.0.1', port=8000, endpoint='/v1/completions', header=None, max_concurrency=4, model='/workspace/models/qwen3-32b-awq', input_len=None, output_len=None, tokenizer=None, tokenizer_mode='auto', use_beam_search=False, logprobs=None, request_rate=inf, burstiness=1.0, probe_request_rate=0.0, disable_tqdm=False, num_warmups=0, profile=False, save_result=True, save_detailed=False, append_result=False, metadata=None, result_dir='/workspace/bench/results', result_filename='tp2_c4.json', ignore_eos=True, self_timed=None, percentile_metrics='ttft,tpot', metric_percentiles='99', goodput=None, request_id_prefix='bench-deaddaa7-', top_p=None, top_k=None, min_p=None, temperature=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, served_model_name=None, lora_modules=None, lora_assignment='random', ramp_up_strategy=None, ramp_up_start_rps=None, ramp_up_end_rps=None, ready_check_timeout_sec=0, chat_template_kwargs=None, extra_body=None, skip_tokenizer_init=False, insecure=False, plot_timeline=False, timeline_itl_thresholds='25,50', plot_dataset_stats=False, log_config_file=None, logging_config=LoggingConfig(log_level='INFO', formatter='text', configure_logging=True, pylogging_config_file=None)) (MainProcess pid=7900) INFO 11-12 13:38:26 [utils.py:88] Sampling input_len from [1024, 1024] and output_len from [256, 256] WARNING: vllm bench serve no longer sets temperature==0 (greedy) in requests by default. The default will be determined on the server side and can be model/API specific. For the old behavior, include --temperature=0. Starting initial single prompt test run... Skipping endpoint ready check. Starting main benchmark run... Traffic request rate: inf Burstiness factor: 1.0 (Poisson process) Maximum request concurrency: 4 tip: install termplotlib and gnuplot to plot the metrics ============ Serving Benchmark Result ============ Successful requests: 16 Failed requests: 0 Maximum request concurrency: 4 Benchmark duration (s): 57.97 Total input tokens: 16384 Total generated tokens: 4096 Request throughput (req/s): 0.28 Output token throughput (tok/s): 70.65 Peak output token throughput (tok/s): 80.00 Peak concurrent requests: 8.00 Total token throughput (tok/s): 353.26 ---------------Time to First Token---------------- Mean TTFT (ms): 1043.44 Median TTFT (ms): 530.78 P99 TTFT (ms): 2748.83 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 52.66 Median TPOT (ms): 50.94 P99 TPOT (ms): 58.72 ================================================== ### TP=2, max concurrency 16 Namespace(subparser='bench', bench_type='serve', dispatch_function=, trust_remote_code=False, seed=1, num_prompts=64, dataset_name='random', no_stream=False, dataset_path=None, no_oversample=False, skip_chat_template=False, enable_multimodal_chat=False, disable_shuffle=False, custom_output_len=256, custom_ensure_client_side_data=False, spec_bench_output_len=256, spec_bench_category=None, sharegpt_output_len=None, timed_trace_chunk_hash_size=16, timed_trace_sec_multiplier=1, timed_trace_label_timestamp='timestamp', timed_trace_label_input_length='input_length', timed_trace_label_output_length='output_length', timed_trace_label_hash_ids='hash_ids', blazedit_min_distance=0.0, blazedit_max_distance=1.0, asr_max_audio_len_sec=inf, asr_min_audio_len_sec=0.0, random_input_len=1024, random_output_len=256, random_range_ratio='0.0', random_prefix_len=0, random_batch_size=1, no_reranker=False, random_mm_base_items_per_request=1, random_mm_num_mm_items_range_ratio=0.0, random_mm_limit_mm_per_prompt={'image': 255, 'video': 1}, random_mm_bucket_config={(256, 256, 1): 0.5, (720, 1280, 1): 0.5, (720, 1280, 16): 0.0}, hf_subset=None, hf_split=None, hf_name=None, hf_output_len=None, bfcl_categories=None, prefix_repetition_prefix_len=256, prefix_repetition_suffix_len=256, prefix_repetition_num_prefixes=10, prefix_repetition_output_len=128, speed_bench_dataset_subset='qualitative', speed_bench_output_len=4096, speed_bench_category=None, label=None, backend='vllm', base_url='http://127.0.0.1:8000', host='127.0.0.1', port=8000, endpoint='/v1/completions', header=None, max_concurrency=16, model='/workspace/models/qwen3-32b-awq', input_len=None, output_len=None, tokenizer=None, tokenizer_mode='auto', use_beam_search=False, logprobs=None, request_rate=inf, burstiness=1.0, probe_request_rate=0.0, disable_tqdm=False, num_warmups=0, profile=False, save_result=True, save_detailed=False, append_result=False, metadata=None, result_dir='/workspace/bench/results', result_filename='tp2_c16.json', ignore_eos=True, self_timed=None, percentile_metrics='ttft,tpot', metric_percentiles='99', goodput=None, request_id_prefix='bench-ac524c04-', top_p=None, top_k=None, min_p=None, temperature=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, served_model_name=None, lora_modules=None, lora_assignment='random', ramp_up_strategy=None, ramp_up_start_rps=None, ramp_up_end_rps=None, ready_check_timeout_sec=0, chat_template_kwargs=None, extra_body=None, skip_tokenizer_init=False, insecure=False, plot_timeline=False, timeline_itl_thresholds='25,50', plot_dataset_stats=False, log_config_file=None, logging_config=LoggingConfig(log_level='INFO', formatter='text', configure_logging=True, pylogging_config_file=None)) (MainProcess pid=7978) INFO 11-12 13:39:29 [utils.py:88] Sampling input_len from [1024, 1024] and output_len from [256, 256] WARNING: vllm bench serve no longer sets temperature==0 (greedy) in requests by default. The default will be determined on the server side and can be model/API specific. For the old behavior, include --temperature=0. Starting initial single prompt test run... Skipping endpoint ready check. Starting main benchmark run... Traffic request rate: inf Burstiness factor: 1.0 (Poisson process) Maximum request concurrency: 16 tip: install termplotlib and gnuplot to plot the metrics ============ Serving Benchmark Result ============ Successful requests: 64 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 99.40 Total input tokens: 65536 Total generated tokens: 16384 Request throughput (req/s): 0.64 Output token throughput (tok/s): 164.84 Peak output token throughput (tok/s): 256.00 Peak concurrent requests: 32.00 Total token throughput (tok/s): 824.18 ---------------Time to First Token---------------- Mean TTFT (ms): 3606.71 Median TTFT (ms): 3951.86 P99 TTFT (ms): 10693.83 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 83.09 Median TPOT (ms): 88.60 P99 TPOT (ms): 103.71 ================================================== ### NCCL all_reduce over RoCE, bf16, 2 ranks (one per node); GPUDirect RDMA disabled on both HCAs (GB10 unified memory) all_reduce 1 MiB: 1.15 ms 0.91 GB/s ( 7.3 Gbit/s) all_reduce 8 MiB: 0.62 ms 13.59 GB/s ( 108.7 Gbit/s) all_reduce 64 MiB: 3.29 ms 20.38 GB/s ( 163.0 Gbit/s) all_reduce 256 MiB: 14.52 ms 18.49 GB/s ( 147.9 Gbit/s) all_reduce 1024 MiB: 48.56 ms 22.11 GB/s ( 176.9 Gbit/s) all_reduce 8 KiB latency: 16.9 us