ros/LLaMA-Factory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update training resuming

Browse Source 
Former-commit-id: 2ec75c31f609e65116ac3b621eeb7d8ccbf69135
This commit is contained in:
hiyouga
2023-08-18 01:41:17 +08:00
parent f11c1ae562
commit c2644f939a
7 changed files with 57 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
README.md
View File

@@ -12,6 +12,8 @@
## Changelog
[23/08/18] Now we support **resuming training**, upgrade `transformers` to `4.31.0` to enjoy this feature.
[23/08/12] Now we support **RoPE scaling** to extend the context length of the LLaMA models. Try `--rope_scaling linear` argument in training and `--rope_scaling dynamic` argument at inference to extrapolate the position embeddings.
[23/08/11] Now we support **[DPO training](https://arxiv.org/abs/2305.18290)** for instruction-tuned models. See [this example](#dpo-training) to train your models (experimental feature).
@@ -158,6 +160,8 @@ pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/downl
CUDA_VISIBLE_DEVICES=0 python src/train_web.py
```
We strongly recommend using the all-in-one Web UI for newcomers since it can also generate training scripts **automatically**.
Currently the web UI only supports training on **a single GPU**.
### Pre-Training