fix #242

Former-commit-id: 80a346e29beb49e8935b786e2af1059fdc4954b2
2023-07-25 17:04:02 +08:00
parent c145bbef3c
commit ac587438f8
4 changed files with 22 additions and 3 deletions
--- a/README.md
+++ b/README.md
@@ -342,6 +342,12 @@ python src/export_model.py \
    --output_dir path_to_export
 ```

+## TODO
+
+- [ ] Supporting flash attention ([torch](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) / [xformers](https://github.com/facebookresearch/xformers) / [flashattn](https://github.com/Dao-AILab/flash-attention)).
+- [ ] Implementing multi-query attention for faster inference.
+- [ ] Supporting full-parameter RLHF training.
+
 ## License

 This repository is licensed under the [Apache-2.0 License](LICENSE).