ros/LLaMA-Factory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[infer] support vllm-ascend (#7739)

Browse Source
This commit is contained in:
leo-pony
2025-04-16 20:06:47 +08:00
committed by GitHub
parent ee2ab093a7
commit b9263ff5ac
3 changed files with 25 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
examples/README.md
View File

@@ -181,6 +181,7 @@ llamafactory-cli export examples/merge_lora/llama3_full_sft.yaml
#### Batch Generation using vLLM Tensor Parallel
```
export VLLM_WORKER_MULTIPROC_METHOD=spawn
python scripts/vllm_infer.py --model_name_or_path path_to_merged_model --dataset alpaca_en_demo
```

1
examples/README_zh.md
View File

@@ -181,6 +181,7 @@ llamafactory-cli export examples/merge_lora/llama3_full_sft.yaml
#### 使用 vLLM+TP 批量推理
```
export VLLM_WORKER_MULTIPROC_METHOD=spawn
python scripts/vllm_infer.py --model_name_or_path path_to_merged_model --dataset alpaca_en_demo
```