Add benchmark README

benchmarks/README.md

@@ -0,0 +1,110 @@
+# ComfyUI Serving Benchmarks
+
+Measures latency and throughput of a running ComfyUI server by submitting
+concurrent prompt requests and collecting results from the history API.
+
+## Dependencies
+
+```bash
+pip install aiohttp tqdm gdown
+```
+
+## Supported models / tasks
+
+| Model | Task | Description |
+|-------|------|-------------|
+| `wan22` | `i2v` | Wan 2.2 Image-to-Video — LightX2V 4-step, 720×720, 81 frames |
+
+To add a new model/task: drop a workflow JSON in `workflows/` (with
+`__INPUT_IMAGE__` as the image placeholder) and add an entry to
+`_MODEL_REGISTRY` in `benchmark_comfyui_serving.py`.
+
+## How it works
+
+On each run the script:
+
+1. Downloads model weights into the ComfyUI `models/` directory (only if
+   `--download-models` is passed).
+2. Downloads the [VBench I2V](https://github.com/Vchitect/VBench) image
+   dataset via `gdown` into ComfyUI's `input/` folder.
+3. Generates one prompt JSON per input image under
+   `benchmarks/prompts/<model>_<task>/`.
+4. Submits `--num-requests` prompts to the server, cycling through the
+   generated prompt files in round-robin order.
+5. Polls `/history/{prompt_id}` for completion and prints a latency /
+   throughput summary.
+
+Per-node execution times are available when the server is started with
+`--benchmark-server-only`.
+
+## Usage
+
+### Start the server
+
+```bash
+python main.py --listen 127.0.0.1 --port 8188 --benchmark-server-only
+```
+
+### Run the benchmark
+
+```bash
+# From the ComfyUI root directory:
+python3 benchmarks/benchmark_comfyui_serving.py \
+  --model wan22 --task i2v \
+  --num-requests 50 --max-concurrency 4 \
+  --host http://127.0.0.1:8188
+```
+
+Include model weight download on first run:
+
+```bash
+python3 benchmarks/benchmark_comfyui_serving.py \
+  --model wan22 --task i2v \
+  --download-models --comfyui-base-dir /path/to/ComfyUI \
+  --num-requests 50 --max-concurrency 4 \
+  --host http://127.0.0.1:8188
+```
+
+### All flags
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--model` | *(required)* | Model name (e.g. `wan22`) |
+| `--task` | *(required)* | Task type (e.g. `i2v`) |
+| `--host` | `http://127.0.0.1:8188` | ComfyUI base URL |
+| `--num-requests` | `50` | Total requests to submit |
+| `--max-concurrency` | `8` | Max in-flight requests |
+| `--request-rate` | `0` | Requests/sec; `0` = fire immediately |
+| `--poisson` | off | Poisson inter-arrival when `--request-rate > 0` |
+| `--num-images` | `20` | Synthetic images if VBench download unavailable |
+| `--prompts-dir` | `benchmarks/prompts/<model>_<task>/` | Prompt JSON output directory |
+| `--download-models` | off | Download model weights before benchmarking |
+| `--comfyui-base-dir` | — | ComfyUI root (required with `--download-models`) |
+| `--output-json` | — | Write full per-request results to a JSON file |
+
+## Output
+
+```
+benchmark: 100%|████████████| 50/50 [req, succeeded=50]
+
+=== ComfyUI Serving Benchmark Summary ===
+requests_total:   50
+requests_success: 50
+requests_failed:  0
+wall_time_s:      412.341
+throughput_req_s: 0.121
+latency_p50_s:    38.201
+latency_p90_s:    52.110
+latency_p95_s:    55.837
+latency_p99_s:    60.012
+latency_mean_s:   39.445
+latency_max_s:    61.203
+execution_mean_ms: 35210.44
+execution_p95_ms:  51200.11
+
+--- Per-node execution time (mean ms across successful requests) ---
+  KSampler (Advanced) (130:110): mean=18200.1  p95=22100.3  n=50
+  KSampler (Advanced) (130:111): mean=16900.4  p95=20800.7  n=50
+  VAEDecode (130:129):           mean=420.2    p95=510.1    n=50
+  ...
+```