ros/LLaMA-Factory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix bug in galore optimizer

Browse Source 
Former-commit-id: c05ac23261a5a8ba893c2918a43dc7777307407b
This commit is contained in:
hiyouga
2024-04-21 18:53:22 +08:00
parent f8e219dc81
commit d16561e7a4
2 changed files with 7 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
examples/extras/galore/sft.sh
View File

@@ -11,8 +11,8 @@ CUDA_VISIBLE_DEVICES=0 python ../../../src/train_bash.py \
--use_galore \
--galore_layerwise \
--galore_target mlp,self_attn \
--galore_scale 2.0 \
--galore_rank 128 \
--galore_scale 2.0 \
--output_dir ../../../saves/LLaMA2-7B/galore/sft \
--overwrite_cache \
--overwrite_output_dir \
@@ -29,8 +29,8 @@ CUDA_VISIBLE_DEVICES=0 python ../../../src/train_bash.py \
--evaluation_strategy steps \
--load_best_model_at_end \
--learning_rate 5e-5 \
--num_train_epochs 30.0 \
--max_samples 300 \
--num_train_epochs 3.0 \
--max_samples 3000 \
--val_size 0.1 \
--plot_loss \
--pure_bf16