ros/nanochat
1
0
Fork 0
mirror of https://github.com/karpathy/nanochat.git synced 2026-01-30 04:22:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

delete the inline rustbpe project. it was ugly to have a project within project and rustbpe is now nicely a separate repo on my github karpathy/rustbpe and it's on pypi etc., so we just add it as a depedency to uv. i think it is appropriate that this is a separate repo because 1) it doesn't have too many knobs, other than the ones that are exposed - the regex pattern and vocab size and 2) all of its complexity is not algorithmic (it's equivalent to minbpe), instead it is efficiency-related, so it is ok to hide relatively speaking

Browse Source
This commit is contained in:
Andrej Karpathy
2026-01-03 23:55:28 +00:00
parent 48abd7d85f
commit aa42f40e66
10 changed files with 44 additions and 1029 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
pyproject.toml
View File

@@ -10,6 +10,7 @@ dependencies = [
"files-to-prompt>=0.6",
"psutil>=7.1.0",
"regex>=2025.9.1",
"rustbpe>=0.1.0",
"setuptools>=80.9.0",
"tiktoken>=0.11.0",
"tokenizers>=0.22.0",
@@ -18,19 +19,8 @@ dependencies = [
"wandb>=0.21.3",
]
[build-system]
requires = ["maturin>=1.7,<2.0"]
build-backend = "maturin"
[tool.maturin]
module-name = "rustbpe"
bindings = "pyo3"
python-source = "."
manifest-path = "rustbpe/Cargo.toml"
[dependency-groups]
dev = [
"maturin>=1.9.4",
"pytest>=8.0.0",
]