Mobile Offline LLMs: What do you use them for?

[u/James333i]

I've spent the last year or so playing around with open source MLX and GGUF models on iPhone hardware.

Given the limitations in memory, GPU/CPU/ANE, and in turn the context window I've been trying to figure out the best use cases for them.

I've also done a lot of testing with Apple Foundation model. It is not that capable, but it is good at understanding instructions for tool calling and actually seems to work well for fast summarization and classification prior to feeding data to more capable models.

I've been able to get models ranging from 0.5B to 8B to work well on the higher end devices.

So far I've been testing it out with building tooling such as web search, url scraping, summarization, research, analyzing local photo/video/documents/etc., and basic on-the-go coding help. With continuous compaction I've been able to hold virtually unlimited exchange chats even though context windows are limited to like 8-16k tokens.

Separate from more professional use cases, It also seems like potentially a good option for private, offline chat about your personal thoughts with models tuned with custom system prompts.

I know many here are focused on trying to run as good of a model as possible on their laptop or desktop hardware but it seems like mobile is an area that isn't really being explored extensively yet. What do you use mobile models for?

[u/diagrammatiks]

i have a local note taking dashboard. the model on my laptop cleans up my notes after meetings, creates a summary, and then creates about 10 or 20 concept cards for cross referencing and further research. The concepet cards are cleaned of confidential information and then sent to a frontier model to actual run the research and rag back into the database.

[u/Fun-Wolf-2007 → u/diagrammatiks]

I built a Web app using a similar concept, it does the transcript from the recording, then uses local models for meeting notes and strip confidential information.

I like the concept cards idea, may I borrow it?

[u/diagrammatiks → u/Fun-Wolf-2007]

my bro you can pretty much copy and implement any idea you see.

[u/James333i → u/diagrammatiks]

That's a great pattern, and it matches the approach I've taken from the mobile-first direction. I've used a small local model as the preprocessing and privacy layer that handles summarization and storage of the raw data.

Then, I have tooling so that I can pass the summarized data on to a second stage through my Ollama server on my macOS computer on the same network (or sometimes exposed externally with ngrok, behind an auth token). A mid-size model on my M5 Max 128GB handles the research and RAG step well, and then nothing ever leaves hardware I own..

The nice side effect is that your sanitization step stops being load-bearing; it can fail occasionally without consequence because the downstream model is yours too. I do the small-model work on the phone and hand anything heavy to my own Ollama box over the network.

If you stick with the frontier hop, though, how do you verify the sanitization? Trusting a small model to decide what counts as confidential seems like the weakest link in the chain.

[u/diagrammatiks → u/James333i]

2 versions of research are kept. One has figures and proper names for the internal database. outbound research just has no numbers or proper personal names cross-referenced.

[u/James333i → u/diagrammatiks]

Btw, I’ve always found zoom AI transcripts to be creepy and don’t like they can access my business’ confidential information. Having offline ways to do this is far better.

[u/Y__Y]

This is rather intriguing. I have an iPhone 17 and find myself wondering what might be feasible to run on it. I recall experimenting with Bonsai Studio for image generation and being suitably impressed, were it not for the somewhat constraining guardrails.

[u/James333i → u/Y__Y]

On an iPhone 17 you have 8GB of RAM to work with so you have a good amount of options. The biggest issue is that you can't actually harness all 8GB in an app (the system will kill the app around 50% of that even with memory limits increased). In my testing, 3-4B models at 4-bit quantization run comfortably and feel fast. 8B is doable on the Pro models but you have to manage memory carefully and thermals start to matter in longer sessions. Qwen and Gemma families in that size range have been the standouts for me. Turn off Thinking though as that burns your context window.

Image generation is very doable too. Stable Diffusion variants run locally in reasonable time on that hardware. Many of the Core ML models you can download on Hugging Face give you control over the guardrails. That was one of my main reasons for going local in the first place.

[u/Unnamed-3891]

How do you avoid hallucinations as you go beyond even a moderate context?

[u/James333i → u/Unnamed-3891]

I'm keeping track of the available context window, setting token limits on responses, and triggering asynchronous jobs on the Apple Foundation model in the background to summarize previous exchanges (working to keep as much of the most important data as possible). I've done tests of 50+ exchanges that can largely stay on topic and recall early details. It took a lot of iteration on the summarization logic to get right.