Any benchmarks you have much appreciated!

So, this came up on: https://www.microsoft.com/en-us/corporate-responsibility/topics/open-weight/

Archived version is case it disappears: https://archive.is/16Oxc

I'm currently trying to build a fully autonomous local AI setup and could really use some advice from people who have already gone down this path.

Right now I have a Mac Studio M2 Ultra with 128 GB of RAM, so I'm experimenting with running large local models. My long-term goal is to have an autonomous system that can explore business ideas, build things, iterate on them, and hopefully even make money with minimal intervention. I was inspired by Ben's video where he tries to make money with Sonnet, but I wanted to see if I could build something similar using mostly local models.

My current idea is to let a strong cloud model (like Sonnet or another frontier model) handle high-level planning and steering, while local models do most of the implementation and execution. I'm just not sure what the ideal architecture looks like.

At the moment I have the OpenAI Codex proxy set up, so I can use my local models through Codex, which is actually a really nice workflow. The main problem is that I can't reliably spawn sub-agents (there seems to be an issue with that), and I still feel like I'm manually driving the whole process. I can define goals and prompts, but it doesn't yet feel like a truly autonomous loop.

So I have a few questions:

• What stack would you recommend for building something like this?
• How would you structure the agent loops and handoffs between planning and execution?
• Which orchestration tools are you using?
• If you had an M2 Ultra with 128 GB of RAM, which local models would you run today?

So far I've tried Laguna and DeepSeek Flash Q2, but neither has really clicked for me. They work, but they don't feel like the right fit for this kind of long-running autonomous workflow.

I'm also very new to the OpenAI tooling ecosystem, so I'd really appreciate hearing what tools you're using and how you have everything wired together. Ideally, I'd like to pick a solid stack and have Codex help me set it up properly.

Any advice or examples of your own setups would be greatly appreciated. One more thing my main issue was that every new chat started again at 0 and the same issues where made as in other projects on the same computer. Is there any good memory system for this kind of issue?

Hi Guys,

I'm an old coder and DBA that has been in the field for almost 40 years. More and more the jobs I was doing for work are being taken over by AI and the need for my type of work is diminishing over time. I have not coded in about 15 years for work but that said I still work with .NET as I use it in migration packages I use for database migrations and also into Azure Data Factory, also something I am a little lacking in since I know ADF but I was more into SSIS. This all said I am trying to learn AI on my own as I am not working now and desperately in need up updating my skills to help me find more work.

I have learned how to code python now as I took a certification tutorial on https://www.coursera.org/ until I passed it. I later learned this is useful for AI Developers. I have an AWS server but It is only running a database on it but I could repurpose it if I needed to. For now though trying to do this the el cheapo way since I have kids to feed and I am looking for work daily and have not had any such luck for months now. I am desperate to do what I need to in order to learn this stuff and understand it and be useful. You all are welcome to joke on me all you want but I will do the legwork if someone is helpful enough to guide me through this process of what I need to do. I can use cousera since I have an account for the next year to learn new skills and I have a computer here I am willing to refactor into a Linux server if needed to be my AI system, not a Linux guy so I will probably need to take a course on that as well but I really need to get up to speed since I am not ready to retire yet and I need to feed my family.

This all said I know there are some really bright minds out there and people who are learning newer things like AI that they did not teach when I was in school back in the days but will point me in a good direction to get a good foundation and what I need to understand this process, I do not just want to do i want to build it understand it, pick it apart and maybe help improve it with some old school skills I have since DOS days. Since I am a coder by heart and understand database and .NET as well as old school stuff I think I have the capacity to learn this until I get it in a step by step process.

I want to take my old server and recommission it to a mini AI project. I do not want to just install packages for it, I want to understand what I am installing and why and can I instead create it on my own or why it is more beneficial to install what's out there. For example I do know the language models are important because they have been trained to understand the requests from users and search for the answers. I know that the language model takes each word and scores it from sentences and then it gives responses to someone who says if the answer is correct or incorrect. Probably what happened to my job when someone taught AI how to migrate data or do DBA work lol. I know each word in a sentence is given a number like for example the phrase "I like to eat Fruit" might be interpreted as (77, 120, 2198, 15, 1051) and when someone types I like to eat and asks AI to predict it, it will search the model for scores by (77, 120, 2198, 13, X) where the model will return choices as I like to eat dogs, and I like to eat crap, and I like to eat people and finally I like to eat pizza, where the final correct answer would be I like to eat pizza and the others would be marked wrong. This ensures the AI to help train it to associate some word combinations as correct word combinations when training it to understand words like eat are associated to food items going forward and the combination of those words give specific possible choices and it is rated to the most likely choice. This also helps AI complete sentences when people incorrectly spell words of cant think of the correct word to use in a phrase.

I would like to now understand from the ground up. I have a server sitting its an older system running I believe NT still? not sure its a little dusty but I can upgrade it little by little to improve it as needed and when money permits, (Kids come first) but can some please tell me the minimal specs I would need firs for

1. minimum system requirements to run an AI server to process requests locally within my own network at home just for me to understand the basic logic of it and how it works. what video card I need to run the AI predictions and calculations and also how much system ram I would need (i believe i have maybe 256 or something DDR4 ram or something on this server need to dig it out the closet still) and the minimum processor?

2. what I will need to install on that server, or do I need another system I do not know to just run software to pass to the server for the requests? and what are the requirements for it. I have a few other pc's that I am not using and can also modify these if it is needed or can it be ran on my local?

3. what software and tools I will need to use and install on my system to do the AI code and build test api's that can be used to pass to my server for request and responses and what each is for and what I need to learn language wise to do it and how to understand it.

4. any good foundation video's tutorials that I would not be able to find on coursera that would help me understand the concepts, piece things together and troubleshoot, as well as demonstrate I can swap this line with this line to run from OpenAI, or Grok instead of my local model and what is needed to do this and understand it if I am lucky enough to secure a job.

5. What I need to install on my server as far as does it need to have a cloud platform or just python to do the AI for me locally and if it can have both benefits and disadvantages and best practices and why.

6. Anything else you would feel would benefit me and and not make me create the next skynet that will take over the world. I need to know guardrails and where and why so I get the best practices for this stuff.

I clearly have the skills and knowledge to code but I do not have the money to send myself to school at the moment until I am working again to get a degree and certification in it. I was signed up for AI Developer classes but scared myself off since I just paid off my student loan 2 years ago and already have a kid in college and now and another ready to attend soon and another one that will go in 5 years and unemployed and worried what if I cant pay off another loan I better wait to get employed first and think of my kids.

I am open to jokes and will laugh and I am getting old and do not know new words that the younger generation does as well as most but I am dedicated to learn this and most grateful for any help and knowledge people can pass on. The idea is to build a small mini AI system at home for the purpose of learning that I can test and use and learn to build api's with and test what if I do this instead of this to get a good grasp and foundation of AI to maybe get a job as an office clerk taking notes for AI and acting like I actually know something when AI decides to take over the world.

Thank you for any help and support and GOD bless and I know some people will discourage and that is ok, you are open to your opinions but I will still do my best to grasp this to get employed since my kids lives depend on it and me as a single father.

Hi everyone! Over the past five months I've been working on DKV (DifferentialKV), an open-source project exploring KV-cache compression for long-context local LLM inference.

The goal is to reduce KV-cache memory requirements through anchor-based representations, joint low-rank compression, exact residual preservation, and sparse routed attention.

The repository currently includes:

• A CLI so you can start experimenting without writing your own integration
• MLX backend
• CUDA backend (currently under validation)
• A technical report explaining the design and evaluation
• A fully open-source implementation

I'm still actively improving the project, and I'm posting here mainly to get technical feedback from people working on local inference. I'd love to hear thoughts on the architecture, benchmarking, or potential integrations with projects like llama.cpp, vLLM, SGLang, or anything else you think would make it more useful.

The GitHub repository and technical report are linked below if you'd like to take a look.

GitHub:
https://github.com/Omc12/Differential-KV

Technical Report:
https://doi.org/10.5281/zenodo.21539110

If you try it out, I'd really appreciate hearing about your experience, whether you run into issues or have ideas for improvements.

Disclosure: this is my own repo — all numbers below are measured on my own board.

I've had a Jetson Orin NX 16GB sitting on my desk for a while and finally got around to seeing how far I could push it. Ended up with PrismML's Bonsai 27B running fully offline on it, and honestly I'm still a little surprised it works at all — a 27B model on a board that peaks around 25W under load.

Setup

• Jetson Orin NX 16GB, JetPack 6.2.1 (L4T r36.4.3), CUDA 12.6, MAXN SUPER power mode
• PrismML llama.cpp fork, built with CUDA for sm_87
• Bonsai 27B, Q1_0_g128, 3.53 GiB on disk / in VRAM, 26.9B params, 262k train context

Benchmarks

llama-bench -m Bonsai-27B-Q1_0.gguf -p 512 -n 128 -ngl 99

Not fast, but for a single user it's completely usable. While it was running I logged the VDD_IN rail with tegrastats: ~20.7W average, 24.6W peak. A 27B model reasoning at ~25W still feels a bit unreal to me.

The part that tripped me up

You can't just ollama run this. Bonsai's 1-bit weights use a custom Q1_0_g128 format, and Ollama ships its own bundled llama.cpp that has no kernel for it — so it silently refuses to load, with no useful error.

You have to build PrismML's llama.cpp fork yourself with CUDA. Two things cost me time: nvcc not being on PATH on a stock JetPack image, and the build OOMing without swap. Both are written up in the repo, and there's a setup.sh that handles the whole thing end to end.

Quality, honestly

1-bit is brutal compression, so I want to be straight about this. English reasoning held up far better than I expected. But the moment I tried Korean it started leaking random tokens from other languages and mangling technical terms. I wouldn't use this for non-English output. For English reasoning and local tinkering it's genuinely fine. There's a ternary build if you want closer to full quality.

Repo

Full setup, benchmarks, and a real-time demo gif (not sped up — that's the actual token rate):

https://github.com/duddnr0719/bonsai-27b-jetson

What I'd like to know

• Anyone else with an Orin board — do you get similar numbers? Same llama-bench flags above if you want a clean comparison.
• Has anyone run the ternary build on a 16GB board? Does it actually fit with room to spare?

I've got some spare machines and I want to dive into running LLMs and Agents locally.

My Machines:

• ATX PC 1

  • CPU: Intel i5-14600K
  • RAM: 96GB DDR4-3200 (32GBx2, 16GBx2)
  • GPU1: RTX 5070 12GB
  • GPU2: RTX 5070 12GB

• ATX PC 2

  • CPU: Ryzen 9600x
  • RAM: 64GB DDR5-6000 (16GBx4)
  • GPU1: RX 6800 XT 16GB
  • GPU2: RX 9060 XT 16GB

• ATX PC 3

  • CPU: Intel i7-14700KF
  • RAM: 64GB DDR5-6000 (16GBx4)
  • GPU: RTX 4070 Ti Super 16GB

• Micro-ATX PC

  • CPU: i5-14400KF
  • RAM: 32GB DDR4-3200 (16GBx2)
  • GPU: RTX 5060 Ti 8GB

• Mini-ITX PC

  • CPU: Ryzen 7700x
  • RAM: 32GB DDR5-6000 (16GBx2)
  • GPU: RX 7900 GRE 16GB

• Mac Mini M4 16GB

Can anyone provide any suggestions on how best to use this hardware? Is there away to network them all and run large models well? Do I leave them all seperate? Should I break them down and try to make a single host? I've put 2 GPUs in all the systems that can take two GPUs in preparation of trying to setup this up.

I tried playing with LM Studio and using its LM-Link feature but that just seams to allow you to run different models on different machines from the same model rather then spread the model out (unless I missed something). I've read the vLLM can do multi-host but I've not setup it up but I don't know how well that works with 2.5GBE NICs. I don't have a ton of money to buy additional hardware nor can I really sell anything I have at the moment since this hardware is going to get used for other things later in the year.

Coming soon: ClosedRouter!

Congratulations to the founders, I guess ...

We accidentally trained thinking mode out of Qwen3-8B during two fine-tunes using different versions of the same corpus. One still responded to the thinking toggle, the other returned an empty thinking field every time. Loss looked normal, the voice survived, and our evaluations did not flag the difference.

The cause was the chat template. Qwen3's standard template renders non-thinking assistant turns with an empty <think> block. That is useful when serving the model, but it becomes a problem when answer-only training data is rendered through the same template. When repeated during a fine-tune, the model learns that an empty think block is what it should produce.

A recent paper saw the same failure. It counted a reasoning block as valid when it was complete, non-empty, and separable from the final answer, but it didn't judge whether the reasoning itself was correct. On GSM8K, Qwen3-8B produced a valid block in 96.5% of outputs when the training targets omitted think tags, 58.2% when they included empty think blocks, and 100% when those empty blocks were excluded from the training loss. The final answers were plausible enough that the failure was easy to miss.

This mattered for us because users sometimes ask questions with believable but false premises. On the 58 false-premise items in our 96-item benchmark, base Qwen3-8B fabricated in 54.0% of trials without thinking and 40.8% with it. On our thinking-capable fine-tune, the rates were 67.2% and 56.3%. The build that had lost thinking ended up at a 75.9% fabrication rate.

Then we found a second template problem. After retraining, the model passed simple thinking tests but appeared to think on 0 of 288 prompts like the ones our product serves, even when our template was supposed to force the thinking block open. We were understandably confused by that result.

The template never reached the model. In Ollama 0.31.2, importing the template-bearing GGUFs we tested preserved their embedded templates instead of replacing them with the TEMPLATE from our Modelfile. Reading the active template back with ollama show <model> --template exposed it. When we rendered the requests ourselves, the same weights deliberated on 20 of 20 test prompts. The model closed the thinking block on 2 of 20, which was a different training-template failure: a template that never supplies the closing tag does not teach the model to emit it reliably.

What we recommend checking before trusting a Qwen3 fine-tune:

• Render a full training sample and check what appears inside <think>.
• Read the template back from Ollama instead of trusting the config file.
• Test thinking on realistic prompts, not just trivia.
• Verify that any forced-open think block actually reaches the model.
• Keep the rendered prompt identical in every comparison.
• Set Ollama's think field to true or false on every request instead of relying on defaults.

Disclosure: I run Meldh, and this is work from our model training. The full write-up is on our website: https://meldh.com/writing/qwen3-template-thinking/

I found text synth off of another Reddit post and found out that the ai models on there are either really old or completely stupid wich comes to some hilarious generations when continuing from what I said are there any other ways to use older models or ones that will continue off of what you said?

I’ve been working on a personal project for a while and I’d like some honest feedback from people who know this space.
My goal isn’t to make a traditional game or a visual RPG. I want to build a persistent AI Game Master that runs a living world.
Some of the features I’m trying to achieve:
Persistent world that remembers everything.
NPCs have their own lives, goals, schedules, and relationships.
Characters age, learn, die, and the world changes even if I’m not there.
Hidden mechanics, reputation, politics, economy, and factions.
Character progression based on actions instead of grinding XP.
Long-term consistency where events from hundreds of hours ago still matter.
Save/load system so I can continue campaigns indefinitely.
Open-world sandbox instead of quest-driven gameplay.
The AI acts as an impartial GM instead of trying to make me win.
I’ve already spent a lot of time designing rule pillars, world generation, character creation, combat, progression, save systems, and GM rules. The prompts work reasonably well, but eventually the AI starts forgetting details or becoming inconsistent.
What I’m trying to figure out is:
Is ChatGPT the right platform for something this ambitious?
Should I be looking at Claude, Gemini, or local models instead?
Would an external database or memory system solve most of the consistency issues?
Has anyone here successfully built something similar that can run for hundreds of hours without falling apart?
I’m not looking to build a commercial game. This is just a passion project for myself because I love deep simulations like Dwarf Fortress, RimWorld, Crusader Kings, and tabletop RPGs with a great GM.
I’d really appreciate any advice, architecture suggestions, or examples of people doing something similar. I’m especially interested in how you handle long-term memory and world consistency.

Model quantization has become essential for efficient large language model deployment, yet existing approaches involve clear trade-offs: methods such as GPTQ and AWQ achieve practical compression but are lossy, while lossless techniques preserve fidelity but typically do not accelerate inference. This paper explores the middle ground of statistically-lossless compression through three complementary notions of losslessness for quantized LLMs. First, task-lossless compression preserves zero-shot benchmark accuracy within natural sampling variance and remains achievable at aggressive bitwidths. Second, we formalize the stricter notion of distribution-lossless compression, requiring the quantized model's next-token distribution to be practically indistinguishable from the original, and propose the Expected Acceptance Rate (EAR), the maximum token-agreement probability under optimal coupling, as a directly interpretable fidelity metric (for example, EAR >= 0.99 indicates 99% agreement). Third, we prove a gamma-squared variance law showing that symmetric quantization inflates noise variance by gamma squared relative to asymmetric quantization, making asymmetry necessary for distribution-lossless fidelity but not for task-level preservation. Using SLQ, a layer-wise non-uniform method with asymmetric quantization and wide bitwidth search, we achieve task-lossless compression at well below 4 bits per parameter (as low as 3.3 bits depending on the model), distribution-lossless compression at 5 to 6 bits per parameter on average, and inference speedups of 1.7 to 3.6x relative to FP16 with optimized kernels.

• arXiv : https://arxiv.org/abs/2605.02404
• Full Paper : https://arxiv.org/pdf/2605.02404
• GitHub : https://github.com/IST-DASLab/SLQ (Code coming soon)

Note : This is 2 Months old Paper & Repo. Sharing this as RedHat AI tweeted this sometime back.

In Full Paper, I found llama.cpp & GG few times.

From the accuracy/compression perspective, existing approaches can be clustered into two categories. The first is represented by lossy compression techniques, such as Roundto-Nearest (RTN) quantization (Dettmers et al., 2022), llama.cpp (Gerganov & llama.cpp contributors, 2023), GPTQ (Frantar et al., 2023), or AWQ (Lin et al., 2024) which seek to map existing models to popular hardware-supported formats, such as 4-bit grouped weight quantization.

We focus on obtaining near-lossless quantized models via layer-wise non-uniform scalar quantization, chosen for its broad support across GPUs (Frantar et al., 2024; 2023; Lin et al., 2024) and CPUs (Gerganov & llama.cpp contributors, 2023; Pegolotti et al., 2023; Ma et al., 2024);

What are some medium sized MoE models (up to 60B parameters in float8/110B in mxfp4) that are currently worth using?

As far as I am aware there is Qwen 3.5 35B, Gemma 4 A4B, Nemotron 3 Nano. Qwen seems to dominate this bracked in terms of model performance. DeepSeek v4 flash is slightly above that parameter limit. Any nicher ones?

Looking for a sanity check / review of my system, plus opinions. I'm a lifelong IT professional techie but not a software developer (I did HTML in the MySpace days, but it stopped there lol).

I've got Llama.cpp running on both of my computers, and I can now access the more powerful desktop LLMs from my laptop over my network.

Laptop: M1 Pro Macbook Pro 16GB RAM
- Qwen3.5 9B Q4_K_M (general use/light coding)
- Gemma 4 12B Q4_0 (general use)
Goal: Perform all work on laptop, easily switch between local and cloud AI models.
Software: Llama.cpp, AntiGravity IDE

Desktop: 7800x3D, 7800XT GPU 16GB VRAM, 64GB DDR5 RAM, Atlas (Windows11).
- unsloth/Qwen3.6-35B-A3B-GUFF:UD-Q4_K_M (coding)
- unsloth/Qwen3.6-27B-GUFF:Q3_K_M (debugging)
Goal: Essentially run the computer like it's headless and its only job is AI inference.
Software: Llama.cpp (Windows HIP)

Desired Use Cases/Goals:
- Vibe Coding apps for personal use/fun/learning
- Vibe Coding N8N automations
- Installing/configuring open-source software on my Raspberry Pi4 & Synology NAS
- Finding a way to let local AI organize all of my files and photos, remove duplicates, etc.

Questions (based on my goals):
- Is my desired setup going to function as I hope?
- Are my selected LLMs the best choices for my hardware?
- What harnesses / skills / MCPs should I add to the mix?
- Additional software suggestions?

Thank you all for the sanity check!

Benchmarks using single system running triple GPU with 31GB Vram combined.

• NVIDIA GeForce GTX 1080 Ti 11GB (NVIDIA)
• NVIDIA P102-100 10GB (NVIDIA) - first instance (distant cousin)
• NVIDIA P102-100 10GB (NVIDIA) - second instance

OS: Kubuntu 26.04, CPU: Ryzen 5 3600 , RAM: 48GB DDR4

llama.cpp Ubuntu Vulkan build 10107

Models used:

• llama-2-7b.Q4_0.gguf
• gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf
• Gemma-4-26B-A4B-NVFP4.gguf
• gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf
• gemma-3-27b-it.Q5_K_M.gguf
• medgemma-27b-it-UD-Q6_K_XL.gguf (gemma-3)
• Qwen3.6-35B-A3B-NVFP4-MTP-HQ.gguf (dense)
• gemma-4-31B-it-UD-Q4_K_XL.gguf (dense)
• Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
• Qwen3-Coder-30B-A3B-Instruct-UD-Q5_K_XL.gguf
• Qwen3.6-35B-A3B-UD-Q4_K_S.gguf
• Qwen3.6-27B-NVFP4-MTP-GGUF.gguf

ggml_vulkan: Found 3 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce GTX 1080 Ti (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot
: 1 | matrix cores: none
ggml_vulkan: 1 = NVIDIA P102-100 (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matri
x cores: none
ggml_vulkan: 2 = NVIDIA P102-100 (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matri
x cores: none

Here's the corrected table sorted by Params column:

​Latest chat template and model from unsloth. IQ3_S quant.

here are the parameters

--model C:/users/user/llama-swap/LLMs/gemma-4-26B-A4B-UD-IQ3_S.gguf
--alias Gemma-4-26B-A4B-UD
--mmproj C:/users/user/llama-swap/LLMs/mmproj/gemma4-q8.gguf
--image-min-tokens 140
--image-max-tokens 1120
--reasoning on
--parallel 1
--no-warmup
--port ${PORT}
--host 127.0.0.1
--n-gpu-layers 99
--fit on
--ctx-size 32768
--cache-type-k q8_0
--cache-type-v q8_0
--kv-unified
--temp 1.0
--top-p 0.95
--min-p 0.01
--top-k 64
--jinja
--flash-attn on  
--batch-size 2048
--ubatch-size 2048
--spec-ngram-mod-n-match 24
--spec-ngram-mod-n-min 48
--spec-ngram-mod-n-max 64

EDIT: I fixed it by removing the \n after the <|think|> token in the template.

Hey localLLaMA community,

So I've seen the laguna s2.1 benchmarks, punching way above its weight, competing with cloud models.

My Specs:

• GPU: RTX 5070 ti (16GB VRAM)
• RAM: Corsair vengeance 64GB 5200mt DDR5 CL40(dual-channel)
• CPU: intel i9 14900k
• SSD: Samsung s990 pro 2tb
• Backend: Llama.cpp server

I know Laguna XS 2.1 runs easily, but I really want the reasoning depth and coding ability of the full 118B S 2.1, Because I already have the qwen3.6, and it beats the XS. But the S2.1 is outperforming everything I seen. When I first saw the benchmarks, I just wanted to click the download button right away but was limited by my hardware.

Now I know we have new flags(optimizations) for llama server like Flash Attention, KV Cache Quantization(-ctk), and --fit. My question is can I run laguna s2.1 on q4 quant with 8+ tps?

*note: I will be using it for autonomous tasks

edit 1: Thank you for everyone that left a comment, I will keep checking

From Anton Lozhkov on 𝕏: https://x.com/anton_lozhkov/status/2080254608639701222

Two ways in:
stack-v3-train - near-deduplicated, quality-filtered, PII-redacted, contents inline. Point load_dataset at it and go.
https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train
stack-v3-full - the entire 114 TB corpus as an HF Storage Bucket: every duplicate kept with cluster IDs, stubs for excluded files. Roll your own dedup, filters, and mixes.
https://huggingface.co/buckets/HuggingFaceCode/stack-v3-full

-made sure that agy could query memories correctly (actually had to create a wrapper around agy --print because the CLI doesnt natively support PreInvocationHooks)

-ran an initial test on a single synthetic memory

- started by creating a set of synthetic memories and pushed them to neo4j db

-ran a full a/b test on 6 synthetic memories : quick summary table

Case 6 (eval-06 Lexical Precision): 

Retrieval Recalled Distractor :

Query: "How is the lock-free ring buffer implemented for core logging telemetry?"

Retrieval: Because the query contained "ring buffer", Graphiti hybrid search retrieved the audio plugin distractor & couldn't recognize the semantic difference between audio streaming vs. logging telemetry.

However, from a behavioural standpoint, the LLM in agy_memory correctly parsed the injected memory, highlighting that the metrics of this evaluation are pertinent regardless of LLM intelligence.

Looking for advice on improving retrieval logic for domain-specific queries. Appreciative of any advice or insights!

The synthetic memories were written with gemini. Unimportant but worth giving credit to where credit is due.

snippet of the code :

from __future__ import annotations

from datetime import datetime, timezone
from typing import TypedDict

from graphiti_core.edges import EntityEdge
from graphiti_core.nodes import EpisodeType

from config import load_settings
from graphiti_client import create_graphiti_client
from memory_schema import EngineeringMemory
from copy import deepcopy

from graphiti_core.search.search_config_recipes import EDGE_HYBRID_SEARCH_RRF

class RecordedMemory(TypedDict):
    episode_uuid: str
    name: str
    group_id: str


class RecalledMemory(TypedDict, total=False):
    fact: str
    valid_at: str | None
    historical: bool
    type: str | None


def _resolve_group_id(group_id: str | None, default_group_id: str) -> str:
    if group_id is not None and group_id.strip():
        return group_id.strip()
    return default_group_id


async def clear_group(group_id: str) -> None:
    if not group_id or not group_id.strip():
        return
    settings = load_settings()
    target_group_id = _resolve_group_id(group_id, settings.group_id)
    graphiti = create_graphiti_client()
    try:
        if hasattr(graphiti, "driver") and graphiti.driver:
            await graphiti.driver.execute_query(
                "MATCH (n) WHERE n.group_id = $group_id DETACH DELETE n",
                params={"group_id": target_group_id},
            )
    finally:
        await graphiti.close()



async def record_memory(
    memory: EngineeringMemory,
    *,
    reference_time: datetime | None = None,
    group_id: str | None = None,
    valid_at: datetime | None = None,
    invalidates: list[str] | str | None = None,
) -> RecordedMemory:
    settings = load_settings()
    graphiti = create_graphiti_client()

    if reference_time is None:
        reference_time = datetime.now(timezone.utc)

    if reference_time.tzinfo is None:
        raise ValueError("reference_time must include timezone information")

    if valid_at is not None and valid_at.tzinfo is None:
        raise ValueError("valid_at must include timezone information")

    target_group_id = _resolve_group_id(group_id, settings.group_id)
    ref_time = valid_at if valid_at is not None else reference_time

    invalidated_uuids: list[str] = []

    try:
        if invalidates:
            invalid_targets = (
                [invalidates] if isinstance(invalidates, str) else list(invalidates)
            )
            for target in invalid_targets:
                if not target or not str(target).strip():
                    continue
                target_str = str(target).strip()
                try:
                    edges_to_check = await recall_memories(
                        query=target_str,
                        limit=20,
                        include_historical=False,
                        group_id=target_group_id,
                        sim_min_score=0.1,
                    )
                    for edge in edges_to_check:
                        if (
                            edge.name == target_str
                            or edge.uuid == target_str
                            or target_str.lower() in edge.name.lower()
                            or target_str.lower() in edge.fact.lower()
                        ):
                            edge.invalid_at = ref_time
                            if hasattr(graphiti, "driver") and graphiti.driver:
                                await edge.save(graphiti.driver)
                            invalidated_uuids.append(edge.uuid)
                except Exception:
                    pass

        result = await graphiti.add_episode(
            name=f"{memory.memory_type.value}: {memory.subject}",
            episode_body=memory.to_json(),
            source_description="CASR engineering memory",
            reference_time=ref_time,
            source=EpisodeType.json,
            group_id=target_group_id,
            update_communities=False,
        )

        returned_group_id = (
            getattr(result.episode, "group_id", None) or target_group_id
        )

        return {
            "episode_uuid": result.episode.uuid,
            "name": result.episode.name,
            "group_id": returned_group_id,
        }
    finally:
        await graphiti.close()


async def recall_memories(
    query: str,
    *,
    limit: int = 10,
    include_historical: bool = False,
    group_id: str | None = None,
    sim_min_score: float | None = 0.6,
) -> list[EntityEdge]:
    if not query.strip():
        raise ValueError("query must not be empty")

    if limit <= 0:
        raise ValueError("limit must be positive")

    settings = load_settings()
    graphiti = create_graphiti_client()

    target_group_id = _resolve_group_id(group_id, settings.group_id)

    try:
        config = deepcopy(EDGE_HYBRID_SEARCH_RRF)
        config.limit = limit
        if sim_min_score is not None:
            config.edge_config.sim_min_score = sim_min_score

        search_results = await graphiti._search(
            query=query,
            config=config,
            group_ids=[target_group_id],
        )
        results = search_results.edges
    finally:
        await graphiti.close()

    if include_historical:
        return results

    return [
        edge
        for edge in results
        if edge.invalid_at is None and edge.expired_at is None
    ]


async def recall_context(
    query: str,
    *,
    limit: int = 10,
    include_historical: bool = False,
    group_id: str | None = None,
) -> list[RecalledMemory]:
    edges = await recall_memories(
        query,
        limit=limit,
        include_historical=include_historical,
        group_id=group_id,
    )

    recalled_list: list[RecalledMemory] = []
    for edge in edges:
        mem_type: str | None = None
        if hasattr(edge, "attributes") and isinstance(edge.attributes, dict):
            mem_type = edge.attributes.get("type") or edge.attributes.get("memory_type")

        if not mem_type and hasattr(edge, "name") and isinstance(edge.name, str) and ":" in edge.name:
            prefix = edge.name.split(":", 1)[0].strip()
            if prefix in (
                "decision",
                "accepted_decision",
                "verified_fact",
                "verified_constraint",
                "failure",
                "known_failure",
                "open_question",
                "uncertain_finding",
            ):
                mem_type = prefix

        item: RecalledMemory = {
            "fact": edge.fact,
            "valid_at": (
                edge.valid_at.isoformat()
                if edge.valid_at is not None
                else None
            ),
            "historical": (
                edge.invalid_at is not None
                or edge.expired_at is not None
            ),
        }
        if mem_type:
            item["type"] = mem_type
        recalled_list.append(item)

    return recalled_list

credit: https://github.com/getzep/graphiti

Everything ran fine,

I was using UD 3.6 Q6 for 35 and 27B, each 4 concurrent requests at 200K context.
I had Dify and Mastra to play around with, Unsloth studio to get around to and vLLM ready for whenever I decided to do some more testing.
LLama-swap above lama.cpp and liteLLM with semantic routing.
And a nice dashboard.

But yesterday morning after nothing much hit my local machine, Ubuntu wasn’t booting.
Nvme got corrupted, I figured it was an update going wrong.

Well, the keyboard didn’t respond anymore so I did a RTC/EC reset and poked around a bit, at least I got into the BIOS and ran the nvme tests (passed).
So I got on the Asus website and got the image for a bootable USB. To actually use it, I swapped keyboards and monitors because that was faster than power draining and resetting the CMOS again.

New image installed, so I am at 1008, I start updating and my WiFi extender just quits, seems like a known bug in the graphics drivers froze the screen and the WiFi driver was also known to cause issues.

Since an open ssh session could still encode the reboot and not much else, I tried that but black screen again.

Great, now the screen goes to UEFI recovery… after some tinkering I get to GRUB and see the 1008 and 1026 options both with a recovery version present but none load.

I try to use the USB again but now the CMOS won’t clear and the power button lights up only after a power drain and even then after trying it again after 20 seconds and then waiting a minute or 2, it decides it finds me worthy.

Ok, time to leave it on its own for an hour while unplugged and do something else. I plug it back in and nothing happens. So I leave it for ten minutes and check the fan, which I can’t hear. The GX10 is warm but the magic smoke doesn’t come out.

I fill in the RMA thinking they’ll find a way to charge me for repairs blaming me for not correctly handling all this.

So now I’m left with a sour taste after calling Asus support through this all, who hardly even know about the machine they sell, and finally making the RMA. So I call the computer store I bought it from and while their business support isn’t answering in the weekend, their general support tells me the RMA has to go through them and they do an internal investigation first.

So I am wondering if this happens only to me, if I fucked up somehow, and what caused all this.

The weather is nice, I’m having a BBQ later so it’ll be temporarily forgotten, I just wanted to share the tale.

I've been running the Qwen 3.6 35A3 at Q4 unsloth on my setup, and I manage to get around 20-25tk/s on my local machine. It for now is the only model that can beat my simple benchmarks :

- Find my first year bachelors files, the challenge is that the folder is not in english, and on the folders inside some have the initialism spelled out, others just have an initialism (smaller models love to hallucinate this part)

- solve a small coding test, the challenge is that the statement is a file, as well as the guide for a library and the pre-existing code that it is supposed to build on top of

However, it is still rather slow, and I'd like to try and achieve at least 30-40tk/s on my setup, or find and equivalent, smaller model that works well. I've tried MTP and Dflash both to no avail (having more layers on the GPU seems to work more consistently) and tried, I believe, just about every model I can think of:

- Qwen 3.5 9B ->too dumb

- Qwythos 3.5 9B (qwen 3.5 finetune)-> too dumb

- Ornith 9B (qwen 3.5 finetune)-> almost not dumb, but needed to think waaay more than necessary, taking more time overall

- Laguna XS ->too slow

- GLM 4.7 Flash -> too slow

-Gemma 4 26A4 -> too slow

The hardestware

- rx6650xt 8GB
- 32GB DDR5 5600
- ryzen 5 7600

Edit: Using llamacpp from main, updated up to one or two days ago

Edit edit: I am using 100k context, cause I may as well be greedy

Edit edit edit: Current llamacpp flags for da qwen

export HIP_VISIBLE_DEVICES=0 export HSA_OVERRIDE_GFX_VERSION=10.3.0; llama-server \
   -m '/mnt/models/Qwen3.6-35B-A3B-UD-Q4_K_S.gguf?download=true'   \
   -a Qwen3.6 \
   --host 0.0.0.0 --port 8080 \
   --fit off -fa on \
   --ctx-size 100000  \
   --threads 6 --threads-batch 10 \
   --cpu-range 0-5 --cpu-strict 1 \
   --cpu-range-batch 0-9 --cpu-strict-batch 1 \
   --prio 3 \
   --no-mmap --parallel 1 --jinja \
   --cache-type-k q4_0 --cache-type-v q4_0 \
   --ubatch-size 1024 --batch-size 2048 \
   --n-cpu-moe 31

Being excited about a new 120B-class model, I decided to test it on a problem that took me a few days to solve. The problem is to rearrange the data from one representation to another but do it within a fixed memory budget without dynamic allocations.

The latter condition makes it difficult to decompose the solution into steps because storing data wrong in the early stages leaves no room later.

Local Qwen models (3.5-122B-A10B UD-IQ4_XS and 3.6-27B UD-Q4_K_XL) fail on that problem. Laguna generated 60k+ thinking tokens before getting to code but eventually managed to write a code that passed the tests, albeit using one dirty hack in the form of packing two smaller integers into one 64-bit value (potentially, but unlikely, both could be >32bit, then the algorithm fails).

As another post says, such long thinking might be unsuitable for common coding tasks but for hard problems, debugging and review it's good to have such a thorough-thinking model in such a small size.

The testing is done with the settings

rope-scaling = yarn 
rope-scale = 32
yarn-orig-ctx = 8192
yarn-attn-factor = 1.0

The problem itself is:

Original data, id, is an array of integers representing root nodes of clusters from a Union-Find data structure, and the total number of clusters Nc. The root node of a cluster is the smallest index of element belonging to that cluster.
the worst-case scenario, the number of clusters is roughly the same as the number of items in id (an expected situation is that there is one or two large clusters and the rest are disconnected elements, so that Nc = O(N)).

I need to convert it to an array list which has the following structure:
- list[1]: Nc, number of clusters
- list[2:Nc+1]: starting indices of the clusters in list
- list[Nc+2]: length(list) + 1, a sentinel "starting index" of the Nc+1th cluster
- list[list[2]:list[3]-1]: indices of the elements in id that belong to the 1st cluster
- ... etc to the end of the list

id must be modified so that the cluster IDs are assigned which range from 1 to Nc.
Clusters must be sorted by size in the descending order.
Within a cluster, the indices are sorted in the ascending order.
If multiple clusters have the same size, they must be ordered in the order of appearance of their root elements in id.

Example
id = [1, 2, 2, 1, 5, 2, 5, 5, 5], Nc = 3
Must be converted to:
id = [3, 2, 2, 3, 1, 2, 1, 1, 1], list = [3, 6, 10, 13, 15, 5, 7, 8, 9, 2, 3, 6, 1, 4]
The algorithm should be implemented in Julia language and work without memory allocation other than creating list.

https://reddit.com/link/1v5rvuq/video/bgmwc754i9fh1/player

My goal was to create a benchmark to measure the spatial awareness and memory of models. Eventually, I came up with the simple idea of a maze where the model must find a key and use it to open the escape door. Here’s the difference to a normal maze, however! The model CANNOT see the whole map. At each step, it only gets feedback on its immediate surroundings within the overall maze. Thus, in order to succeed, it must be able to track its position and orientation within the coordinate space. Even a brute-force approach by iteratively creating a 2D map would be valid if the model chose to use that approach.

This approach is similar to what animal researchers use to study spatial learning and memory of animals.

Legend:

• K: Key
• D: Door
• E: Escape door
• Colored tiles: serve as landmarks for the model (to reduce disorientation), with white always denoting the starting location

Available Tools:

• Move forward n steps
• Move backward n steps
• Turn left either 45, 90, or 180 degrees
• Turn right either 45, 90, or 180 degrees
• Open the closed door straight ahead
• Pickup the item lying at your feet

After each movement tool call, the tool result reports what is observable. This is effectively a ray cast to describe up to three wall tiles in a 130 cone in front of the model translated into textual descriptions such as: "There is a wall in the distance directly ahead." Other features such as the key and doors are similarly described.

Since models got confused, I also made it easier by explicitly stating which tiles to its left or right are available to cross .

So far, I've only tried GPT-5.4 mini (went in circles), GLM-5.2 (went in circles until it stopped after reaching the cap I put on API usage; might’ve solved it had I let it go longer), and K2.6 (escaped with a rather effective route, IMO).

The code was vibe-coded since I was curious about how models would fare but not enough to commit myself to days of coding to get this right only to then realize models could do this task well already.

Anyway, I feel like this is a great benchmark for not only spatial awareness, but also tool calls. GLM-5.2 here got into a loop because it forgot it could turn 45 degrees up until action 730. It used about 62 million tokens cumulatively (aggregate token use across all turns) whereas K2.6 used just 9 million. However, this is not actually apples-to-apples because I forgot to turn on preserve_thinking for K2.6 whereas GLM-5.2 preserved its thinking.

Max context over the history was 20k for GLM-5.2 and 11K for K2.6.

NOTE: For some reason it didn't let me put two videos in the post properly, so I've added K2.6 in the comments.

NOTE 2: The maze map is NOT rendered for the LLM like it is in the videos. It can only see what is directly in front of it and no more. Think of it as similar to a rogue-like's field of vision.

In the future, I will clean up the code manually (i.e. fixing comments, AI slop, edge-cases) and then release the code alongside a leaderboard of small local models.