Why It Matters
The model, because of the depth now allowed by 2048 d_model allows me to push 40 total SWA/Global blocks to take advantage of attention based residual streams (Moonshot Kimi K3 style) in a dense model.
Overview
I posted something about a 9b model a few days ago. The real problem was 2 fold. Someone suggested the size was too big to prove it all out.
Why It Matters
The model, because of the depth now allowed by 2048 d_model allows me to push 40 total SWA/Global blocks to take advantage of attention based residual streams (Moonshot Kimi K3 style) in a dense model.
Key Details
It's a fair argument.
I needed the model depth though.
The second problem was the "ability" of these models and the fact that labs (with money) produce these models still.
No real usefulness in my mind.
Further, the Llama licensing at the core of the model was highly problematic and had to be thrown in the trash.
--- *Original report published by [Reddit LocalLLaMA (Model Leaks & Local AI)](https://www.reddit.com/r/LocalLLaMA/comments/1wis23s/update_small_model_engram/).*
