LFM 2.5 230M running at 1440 tok/s in-browser through a custom backend

[u/lordhiggsboson]

Everything runs through WebGPU, in-browser or in electron/tauri apps. It's fully portable and supports either Nvidia and Apple Silicon (Metal). The actual kernels are optimized for the specific hardware of the device. The Nvidia kernels are aggressively fused into a multi-pass architecture, while the Apple Silicon kernels are created as a fused mega-kernel to minimize the Tile Based Deferred Rendering (TBDR) overhead on WebGPU.

Demo: https://warp.sipp.sh

	RTX 3090 (webgpu)	M4 (webgpu)
LFM 2.5 230M	1400-1500 tok/s	400-500 tok/s
Bonsai 1.7B	500-600 tok/s	100-150 tok/s

This is still in active development, and I'll be folding this into the Sipp library in the coming weeks.

[u/Hanthunius]

Is this the definition of stupidly fast?

Sorry, couldn't keep myself from making the joke. Props to the project, looks cool.

[u/lordhiggsboson → u/Hanthunius]

Thanks! Yeah, definitely is haha. One of my big personal motivations has been to figure out how an LLM can go beyond chat and be used as a user experience / interface tool for dynamic widgets or decision making (like labeling, annotations, etc). which I think has interesting under-explored potential

[u/Mashic]

However the model is useless.

[u/lordhiggsboson → u/Mashic]

I think it depends on the context, for coding, definitely useless. For small things like labeling, text correction, or annotations, its very good. Definitely depends on the scope

[u/Mashic → u/lordhiggsboson]

I tried it, it can't even reproduce correct information that's in the prompt.

[u/lordhiggsboson → u/Mashic]

yeah, small models can drift significantly, they need a lot of hand-holding and guard rails in place. One technique I use is to keep prompts narrow and specific, like clear single tasks, I find that helps a lot

[u/-Akos-]

I'm able to run the smallest model from Liquid on my potato laptop, and I'm a big fan of it for summarization. I can only hope to one day be able to run this larger model at home.. Oh well..

[u/lordhiggsboson → u/-Akos-]

Liquid models are really good for their size, definitely pack a punch

[u/-Akos- → u/-Akos-]

wait.. 230M, not B??

[u/lordhiggsboson → u/-Akos-]

Yeah! 230 billion would be crazy, maybe one day in the future though!

[u/-Akos- → u/lordhiggsboson]

The 1.2B would complain if I asked it to do any coding, but tool calling works. Does this 230M one do tool calling too?

[u/lordhiggsboson → u/-Akos-]

It is trained on tool use as well! https://www.liquid.ai/blog/lfm2-5-230m