ComfyUI/benchmarks/README.md

ComfyUI Serving Benchmarks

Measures latency and throughput of a running ComfyUI server by submitting concurrent prompt requests and collecting results from the history API.

Dependencies

pip install aiohttp tqdm gdown

Supported models / tasks

Model	Task	Description
wan22	i2v	Wan 2.2 Image-to-Video — LightX2V 4-step, 720×720, 81 frames

To add a new model/task: drop a workflow JSON in workflows/ (with __INPUT_IMAGE__ as the image placeholder) and add an entry to _MODEL_REGISTRY in benchmark_comfyui_serving.py.

How it works

On each run the script:

1. Downloads model weights into the ComfyUI models/ directory (only if --download-models is passed).
2. Downloads the VBench I2V image dataset via gdown into ComfyUI's input/ folder.
3. Generates one prompt JSON per input image under benchmarks/prompts/<model>_<task>/.
4. Submits --num-requests prompts to the server, cycling through the generated prompt files in round-robin order.
5. Polls /history/{prompt_id} for completion and prints a latency / throughput summary.

Per-node execution times are available when the server is started with --benchmark-server-only.

Usage

Start the server

python main.py --listen 127.0.0.1 --port 8188 --benchmark-server-only

Run the benchmark

# From the ComfyUI root directory:
python3 benchmarks/benchmark_comfyui_serving.py \
  --model wan22 --task i2v \
  --num-requests 50 --max-concurrency 4 \
  --host http://127.0.0.1:8188

Include model weight download on first run:

python3 benchmarks/benchmark_comfyui_serving.py \
  --model wan22 --task i2v \
  --download-models --comfyui-base-dir /path/to/ComfyUI \
  --num-requests 50 --max-concurrency 4 \
  --host http://127.0.0.1:8188

All flags

Flag	Default	Description
--model	(required)	Model name (e.g. wan22)
--task	(required)	Task type (e.g. i2v)
--host	http://127.0.0.1:8188	ComfyUI base URL
--num-requests	50	Total requests to submit
--max-concurrency	8	Max in-flight requests
--request-rate	0	Requests/sec; 0 = fire immediately
--poisson	off	Poisson inter-arrival when --request-rate > 0
--num-images	20	Synthetic images if VBench download unavailable
--prompts-dir	benchmarks/prompts/<model>_<task>/	Prompt JSON output directory
--download-models	off	Download model weights before benchmarking
--comfyui-base-dir	—	ComfyUI root (required with --download-models)
--output-json	—	Write full per-request results to a JSON file

Output

benchmark: 100%|████████████| 50/50 [req, succeeded=50]

=== ComfyUI Serving Benchmark Summary ===
requests_total:   50
requests_success: 50
requests_failed:  0
wall_time_s:      412.341
throughput_req_s: 0.121
latency_p50_s:    38.201
latency_p90_s:    52.110
latency_p95_s:    55.837
latency_p99_s:    60.012
latency_mean_s:   39.445
latency_max_s:    61.203
execution_mean_ms: 35210.44
execution_p95_ms:  51200.11

--- Per-node execution time (mean ms across successful requests) ---
  KSampler (Advanced) (130:110): mean=18200.1  p95=22100.3  n=50
  KSampler (Advanced) (130:111): mean=16900.4  p95=20800.7  n=50
  VAEDecode (130:129):           mean=420.2    p95=510.1    n=50
  ...