mirror of
https://github.com/comfyanonymous/ComfyUI.git
synced 2026-05-13 18:47:29 +08:00
111 lines
3.4 KiB
Markdown
111 lines
3.4 KiB
Markdown
# ComfyUI Serving Benchmarks
|
||
|
||
Measures latency and throughput of a running ComfyUI server by submitting
|
||
concurrent prompt requests and collecting results from the history API.
|
||
|
||
## Dependencies
|
||
|
||
```bash
|
||
pip install aiohttp tqdm gdown
|
||
```
|
||
|
||
## Supported models / tasks
|
||
|
||
| Model | Task | Description |
|
||
|-------|------|-------------|
|
||
| `wan22` | `i2v` | Wan 2.2 Image-to-Video — LightX2V 4-step, 720×720, 81 frames |
|
||
|
||
To add a new model/task: drop a workflow JSON in `workflows/` (with
|
||
`__INPUT_IMAGE__` as the image placeholder) and add an entry to
|
||
`_MODEL_REGISTRY` in `benchmark_comfyui_serving.py`.
|
||
|
||
## How it works
|
||
|
||
On each run the script:
|
||
|
||
1. Downloads model weights into the ComfyUI `models/` directory (only if
|
||
`--download-models` is passed).
|
||
2. Downloads the [VBench I2V](https://github.com/Vchitect/VBench) image
|
||
dataset via `gdown` into ComfyUI's `input/` folder.
|
||
3. Generates one prompt JSON per input image under
|
||
`benchmarks/prompts/<model>_<task>/`.
|
||
4. Submits `--num-requests` prompts to the server, cycling through the
|
||
generated prompt files in round-robin order.
|
||
5. Polls `/history/{prompt_id}` for completion and prints a latency /
|
||
throughput summary.
|
||
|
||
Per-node execution times are available when the server is started with
|
||
`--benchmark-server-only`.
|
||
|
||
## Usage
|
||
|
||
### Start the server
|
||
|
||
```bash
|
||
python main.py --listen 127.0.0.1 --port 8188 --benchmark-server-only
|
||
```
|
||
|
||
### Run the benchmark
|
||
|
||
```bash
|
||
# From the ComfyUI root directory:
|
||
python3 benchmarks/benchmark_comfyui_serving.py \
|
||
--model wan22 --task i2v \
|
||
--num-requests 50 --max-concurrency 4 \
|
||
--host http://127.0.0.1:8188
|
||
```
|
||
|
||
Include model weight download on first run:
|
||
|
||
```bash
|
||
python3 benchmarks/benchmark_comfyui_serving.py \
|
||
--model wan22 --task i2v \
|
||
--download-models --comfyui-base-dir /path/to/ComfyUI \
|
||
--num-requests 50 --max-concurrency 4 \
|
||
--host http://127.0.0.1:8188
|
||
```
|
||
|
||
### All flags
|
||
|
||
| Flag | Default | Description |
|
||
|------|---------|-------------|
|
||
| `--model` | *(required)* | Model name (e.g. `wan22`) |
|
||
| `--task` | *(required)* | Task type (e.g. `i2v`) |
|
||
| `--host` | `http://127.0.0.1:8188` | ComfyUI base URL |
|
||
| `--num-requests` | `50` | Total requests to submit |
|
||
| `--max-concurrency` | `8` | Max in-flight requests |
|
||
| `--request-rate` | `0` | Requests/sec; `0` = fire immediately |
|
||
| `--poisson` | off | Poisson inter-arrival when `--request-rate > 0` |
|
||
| `--num-images` | `20` | Synthetic images if VBench download unavailable |
|
||
| `--prompts-dir` | `benchmarks/prompts/<model>_<task>/` | Prompt JSON output directory |
|
||
| `--download-models` | off | Download model weights before benchmarking |
|
||
| `--comfyui-base-dir` | — | ComfyUI root (required with `--download-models`) |
|
||
| `--output-json` | — | Write full per-request results to a JSON file |
|
||
|
||
## Output
|
||
|
||
```
|
||
benchmark: 100%|████████████| 50/50 [req, succeeded=50]
|
||
|
||
=== ComfyUI Serving Benchmark Summary ===
|
||
requests_total: 50
|
||
requests_success: 50
|
||
requests_failed: 0
|
||
wall_time_s: 412.341
|
||
throughput_req_s: 0.121
|
||
latency_p50_s: 38.201
|
||
latency_p90_s: 52.110
|
||
latency_p95_s: 55.837
|
||
latency_p99_s: 60.012
|
||
latency_mean_s: 39.445
|
||
latency_max_s: 61.203
|
||
execution_mean_ms: 35210.44
|
||
execution_p95_ms: 51200.11
|
||
|
||
--- Per-node execution time (mean ms across successful requests) ---
|
||
KSampler (Advanced) (130:110): mean=18200.1 p95=22100.3 n=50
|
||
KSampler (Advanced) (130:111): mean=16900.4 p95=20800.7 n=50
|
||
VAEDecode (130:129): mean=420.2 p95=510.1 n=50
|
||
...
|
||
```
|