Really thanks!!
Great job, it’s unique!
Thanks really thanks!
if you like, colibrì always needs to improve so if you have ideas or anything else you are welcome for pull request issues and also benchmarks!
I am curious if it’s possible to adjust this to use more RAM, as i’ve got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU and have GLM-5.2 delegate smaller tasks to it. It might take some retraining of GLM-5.2
I’m also curious if you can speed this up by using many disks in parallel to increase bandwidth.
SSD Wear Warning
Cold starts are heavy on random reads (~11 GB/token). Reads themselves are safe, but the OS page cache can generate writes. Heavy use may accelerate wear on cheaper SSDs. Use with caution and monitor your drive health.
Hmm, maybe a safe way to do this would be to make a separate partition for the model weights, and set them to read-only? Not sure how the page cache works, if it’s like per partition or per disk. If it’s per disk, maybe you could have a read-only data.iso formatted as a partition and mount it as a disk?
That’s possibly a good idea! We can work on it!
OS page cache can generate writes
Is this a hallucination? What am I missing? Why would heavy reads generate writes?
I have a 16-core system with 256GB RAM here I could try it with but regretfully it’s so old the CPUs aren’t AVX2 capable. Otherwise it makes a fairly good llama-server test system for CPU only stuff. Oh well. Time to upgrade (painful to the wallet these days).
Maybe we can see some integration!
I have a small laptop. If you have more disks available, you could really do some testing. When you have some benchmarks, submit a pull request or issue so we can maybe work on them. We are really happy for contribute!
Spilling
Wow, I see you managed to fit in so many models (krea, wan, hunyan, etc.). Did you get to build a common harness to run all of them? Which ones stay under your VRAM budget more consistently?
Is it possible to run this into an agent? pi, claude code, etc..? I’ve only tried it with LM studio, but i’m guessing this is a bit different
Wouldn’t turning off swap fix this issue?
Good catch! Disk reads do generate writes to cache. But the cache itself is in RAM, not on disk. So it shouldn’t cause additional wear of SSD.
Is this a hallucination? What am I missing? Why would heavy reads generate writes?
I take it heavy reads means more stuff goes into RAM, meaning other stuff has to be cached?
I’ve got same question as GP: e.g. is there a way to set moderately fast consumer NVMe SSDs (I’ve got both a Samsung 990 Pro and a WD SN850X) in a complete read-only mode to prevent “wear”?
The page has an SSD wear warning [0] I use desktop PCs that I build from components so I can replace the SSD, but what do users with soldered SSD do? Just avoid these applications or forge ahead disregarding the possible early burnout of their storage? They must use external storage as the burner SSD.
We’re working on it right now with a pull request that will also arrive for opencode!
Another recent project that runs a huge model on a 48gb Mac is https://github.com/danveloper/flash-moe - it gets over 5 tokens/sec on an M3 Max compared to this projects very impressive 1 token/sec on an M5 Max. So for anyone wanting to tackle a Mac only version that targets lower spec machines this looks like a good candidate with plenty of room for speedups [edit: because it doesn’t use the gpu].
Not hijacking anything as this project is amazing.
Yes, avoid.
Laptops with soldered in SSDs should definitely monitor their usage and take care with this.
This project seems more of an experiment than something everyone should run, but pretty cool nonetheless