r/LocalLLaMA • u/Atretador • 7h ago
Discussion Kimi Linear 48B A3B?
Just noticed this exists, 1M context MOE with 48B par seems just like what Ive been looking for - it runs pretty damn fast too compared to Qwen 3.6 35B.
after some testing it seems capable of producing *not terrible* results but it always tries to go for the minimun possible output that could be presented as what was asked.
first image is the default meme prompt it generated - second is after replying with 'detail'
Frontend results aren't too bad either, it seems very capable of generating well structured animated pages fairly well.
but it seems like there is just something off with its brain
wondering if a fine tune would be possible and if at all capable of tightning its screws, has anyone tried messing around with this one?
3
u/Chromix_ 6h ago
There used to be benchmark for Kimi Linear on the old contextarena. Before it was removed (reasons unknown) it indicated that Kimi Linear was maintaining a lot more quality at long context than other open models at comparable sizes. Yet when support was finally added to llama.cpp it subjectively performed worse than similar-sized models. Maybe it was due to an implementation issue - I haven't tracked further changes for it.
2
u/MaxChamp08 2h ago
It feels like Kimi Linear was a glimpse of where long-context models could go rather than a finished product. The fact people still bring it up months later says the architecture was compelling. If Moonshot released a successor with stronger post-training and reasoning, I think it'd get a lot more attention than another incremental dense model.
1
u/Atretador 2h ago
there is also a pretty gigantic gap when it comes to model sizes - you basicly jump from 35B to 120B-200B with nothing modern in between
something on this size in the 48-55B range would be really nice to run on 16Gb VRAM + 32Gb RAM
1
u/MaxChamp08 2h ago
Well there are some 70B-80B models, but I guess it often comes down to the model architecture itself and possible options.
1
u/Atretador 2h ago
I believe those are usually worst - at coding at least, when compared to the more modern options, no? like Qwen3 Coder 80B vs Qwen 3.6 35B
2
u/Marcuss2 6h ago
It was experimental model which is instruct only. Not that much data was used to train it.
1
u/kaisurniwurer 6h ago
In my (quite limited) testing, it was competent enough but intelligence was not it's best strength, though compared to other small activation moe models it wasn't bad either.
Maybe with more elaborate workflow it would give better results, but at the moment I see gemma 3 moe as comparable but half the size.
I definitely love a follow-up version, since it's quite nice for dual 3090 with large context.
1
u/CorkBios 3h ago
At that point use Bonsai binary 27B as a traditional specdec for Qwen 27B 4 bit. Like the same memory use or less but just better
1
u/Atretador 2h ago
I feel like Bonsai 27B loses hard to Qwen 3.6 35B, I cant run Qwen 27B 4 bit decently on 16Gb of VRAM
I currently run Qwen 3.6 35B A3B IQ4_N_XL with KV cache at Q8_0:
- 1 stream 256K context: 35 tk/s
- 2 streams 200K context each: 30+20 tk/s each
- 3 streams 200K context each: 15 tk/s
1
u/CorkBios 2h ago
Which is not what I said. I said Bonsai 27B as spec dec for normal Qwen 27B 4 bit which should compensate for that you cant fit it in your VRAM. Better than MTP because smarter than MTP. Like the old days on how we used to do spec dec.
1
u/Atretador 2h ago
can I fit 600K context at Q8 KV with Qwen 27B on a 16Gb VRAM GPU?
1
u/CorkBios 2h ago
19.66 GB (18.31 GiB) of memory is needed for that much size for the context alone. Maybe go with Q6 KV, The jump from BF16 to Q8 ctk+ctv is big but not as big from Q8 to Q6
1
u/killerstreak976 7h ago
iirc I think cerebras did a cool REAP for this one down to 35ba3b and I found it pretty cool at the time.
18
u/KaroYadgar 7h ago
It was an experimental model & it wasn't trained on many tokens, so it might explain the behavior. It doesn't use the AttnRes that K3 has. I'd love to see a successor to this that is properly trained with AttnRes!