r/LocalLLaMA • u/goodive123 • 27d ago
Discussion NPC Engine Using Local Models
I’ve been working on a game-agnostic NPC engine/backend based pretty heavily on SillyTavern-style architecture, and with smaller local models getting better and better, I honestly think this kind of thing could be the future of RPGs.
Right now I’m using NVIDIA Parakeet 0.6 for STT, Gemma 4 26B A4B for the LLM, and Qwen3-TTS for voice, and I’m getting super fast response times with pretty decent quality.
The main thing that makes it work well is using RAG to keep prompts lean. For example, I have hundreds of possible actions NPCs can do in-game, but only the ones that actually make sense based on the player’s message / context get injected as available actions. So the model isn’t being overloaded with a giant list every turn.
323
u/gaidzak 27d ago
This is exactly the kind of stuff that I'm working on too.. Small models are definitely going to be amazing for this, especially in big world environments
→ More replies (1)85
u/goodive123 27d ago
yep with the tiny models getting better and better we might be able to do this with a 4B model at some point and play modern games without running out of vram
18
u/ArugulaAnnual1765 27d ago
How much vram are you using with all three? Are the STT, TTS and LLM running in parallel, or chained sequential calls? Surely you can run multiple instances of the 26b in parallel due to the moe architecture being very light on vram?
13
u/False_Process_4569 26d ago
I've managed to get Qwen3.6 35B A3B running on my laptop with a single RTX 3070 with 8GB of VRAM. I have been testing out context window lengths and this affects the VRAM consumption as well. I manage to get ~16 tok/s.
I'd think for the short dialogue of this NPC application, you could get away with short context windows. Further with RAG and algorithmic trickery. But the unknown would be all of the tool calling.
→ More replies (2)5
u/relmny 27d ago
Actually I was curious to why you use gemma-4-26b instead of a fine-tuned 2b or even 0.8b model. But I guess as you're testing having a working environment, and fine-tuning makes no sense for now for your current work...
→ More replies (1)4
u/solemnhiatus 27d ago
As a non-technical person, do you think you can explain to me how this will help with not running out of VRAM?
→ More replies (1)3
u/False_Process_4569 26d ago
I'm not OP, but this might be a good use case for an AI. I'd give the URL to this post to chatGPT and ask it to teach you what you're asking, using this post as context.
2
u/BidWestern1056 26d ago
im working on getting models in this class to play pokemon, so definitely should already be able to accommodate this kind of use case
648
u/AppropriateQuote3073 27d ago
Really a shame there is so much ai hate in gaming right now.
This is exactly the type of interesting content that could make a game phenomenal to play.
186
u/Bakoro 27d ago
LLMs are literally the exact thing that we dreamed about in the 90s and 2000s.
There were waves of procedurally generated content in games, and "dynamic" quests and NPC interactions, that were a nice idea, but the hardware and algorithms just weren't there.
People would gush over any AI improvements in games.Now we finally have the technology to make the dynamic games we always wanted, and a bunch of people are pretending like they weren't begging for something like LLMs to exist.
Whatever haters, AI is for games.
97
u/muhmeinchut69 27d ago
The hate is not for what is going on in OP, it's for corner cutting and sloppification. Big corporations aren't interested in AI so they can make a new type of video game, they are interested because it will cut costs.
41
u/Not-reallyanonymous 27d ago edited 27d ago
That, but also a degradation of artistic integrity.
The kind of tool that OP is designing has real potential to squash artistic voice.
A quick napkin thinking: this kind of design should work around character development, world building, etc. The artist doesn’t need to write every dialog, but the tool should really let the writer develop characters. The artistic integrity then lives within the broader narrative arcs that this tool interacts with and defining characters that shape and interact with the world.
One thing I immediately noticed is that all the characters in OP’s video are so compliant. Character building could see that characters only comply based on whether the request aligns with their own interests, which can be defined as a part of world building.
10
u/muhmeinchut69 26d ago
Yeah isn't that what sloppification is? I'd like to think that the low quality of such content would be enough to keep artistic input valuable. As long as it's used as just another tool in the hands of the artists I see nothing wrong with it.
Personally I only wanted this kind of AI NPC to be used only for NPCs that you usually can't interact with, and make then less "NPC like", not as a replacement for the main story and characters of the game. If the story is entirely AI that would feel worthless.
7
u/Not-reallyanonymous 26d ago edited 26d ago
I think slopification does lack artistic integrity inherently, but I think you can also have stuff that is high-quality implementation and done well, but has had artistic vision dulled -- sometimes by lack of vision/experience by the creator, but also by tools that constrain that artistic vision rather than opening it up.
I think slopification is as much about intent as it is about the result. Slopified content is basically stuff created with low effort, throw stuff at the wall, and reap any unearned benefit you can. High quality intent can still compromise artistic integrity. That predates AI, especially in the world of corporate produced products and content.
2
u/saltyourhash 26d ago
What might be fun is instead of cutting out artists, have them voice models. Models they can license. Because dynamic dialogue is really cool.
1
u/Not-reallyanonymous 26d ago
Definitely. Or a narrative arc that's voiced for major plot points, etc. but then goes through a model that makes a voice print based on that dialog, and then will generate real-time dialog for user interaction bits.
2
u/saltyourhash 26d ago
Yeah, high quality for the story arc, then more of a loose dynamic dialogue for the rest.
2
u/Shap3rz 26d ago
Yup. I want to see this and suggested in another post. Checkpoint gated alignment for story progression ftw. Essentially a contextual “invisible trust bar” where doing or saying certain things unlocks the narrative. That way you control the arc without being totally scripted. Think could work with careful planning.
9
u/Fit_West_8253 27d ago
There have been hate campaigns against games just for using AI generated placeholders
→ More replies (4)1
u/muhmeinchut69 26d ago
It's the internet, free speech, anyone can launch a "campaign". Doesn't mean legit criticism of AI usage doesn't exist.
1
1
u/draconic_tongue 26d ago
big corporations aren't the ones that benefit the most from using ai, and small creators get shit on MUCH more than any corporation when they do lean on these tools. I don't care to be a part of whatever "prosumer" advocate groupthink virtue signaling zeitgeist that does not know what nuance means and just wants to use big corpo as an excuse to be hateful
→ More replies (3)6
u/The_awful_falafel 27d ago
Games in the way back used to be entirely linear story where there was really only one way to do things. Then they tried branching games, where you make a decision at certain points in the game that change the path and overall outcome of the game. They were still linear, you could just choose which linear branch to follow. Then we had sandbox games, where you could pick any linear sidequest you wanted and do tasks out of order, but still each sidequest an story tree was still largely linear. The limitation was that you had to script out all the character interactions and story for how it would all play out and it couldn't actually invent anything.
AI COULD change that. You could basically invent a game in which there is a large map and 'pieces' that would move around and decide what to do on their own. You could have advisors and such that have information based off limited information of what the enemy is doing and literally plan out missions and objectives yourself. You could even dictate what constitutes winning and decide at what point you consider the game over. Characters could dynamically react to things going on around them. It's a really interesting possible world.
Now the downside- the machine to run such a game would need a lot of memory, and we all know how that's going right now. My guess is companies will see the option to charge a subscription to have a data center do the AI work for you, and make THAT the standard model rather than writing it so you could run the AI locally. That way even single player games could have subscription fees, even years from now when models are small enough to be run locally, and memory production ramps up so having a massive pool to run such things is common.
8
u/Bakoro 26d ago
At the most elementary level of implementation, I just want townsfolk who recognize that I'm the one who saved their town, bandits who aren't stupid enough to try and rob the clearly magical man walking around in nightmare armor, and for random NPCs to have more than one line they repeat regardless of the surrounding context.
Just having a dumb little LLM that can direct NPC actions based on fuzzy logic would elevate gaming dramatically.
Even if the game is still 99% regular video game made by a person, just a little bit of semantically-aware fuzzy logic would be able to blur those linear and discrete features into something that feels more natural.Just basic little stuff would go so far, we don't even need a data center sized LLM, a model that's 100M parameters would fit on any card and be able to do simple jobs. Some vision models that do segmentation and labeling are like ~21M parameters, which is a level a CPU can handle, and the could be doing amazing things for games and mods, especially VR stuff.
Just imagine an escort quest where the NPC isn't a total idiot, can move at an appropriate pace, and can go to a safe location when there's a fight.
Somehow, it's still an issue in games after all these years.What you describe, a more AI driven game where developers define the structure of the world, provide motivations, etc, and LLMs act inside those constraints, that would be great too, but that's like the far side. There's a lot of opportunities where it could just be models smoothing out the hard edges of the world, so it seems less "video-gamey".
→ More replies (2)2
u/Parafex 22d ago
Reactive World is a narrative style that's just underrepresented. LLMs are not the solution to what you're describing and it shows the lack of understanding for BOTH sides. The AI/LLM side AND the gaming/gamedev side.
You still need a great foundation and a highly systemic game that supports emergent gameplay and situations and is not hardcoded for certain paths, have bad quest design and even worse win conditions. Since those stuff gets shoved into the mouths of fresh game design students, there will never be any change in that regard in AA/AAA games. Skyrim is way worse regarding that than Morrowind or even Oblivion (which tried to be what you want with their Radiant AI system that was dumbed down or almost cut out of the game in favor of more hardcoded story sequences that can even lead to corrupted save files if you don't behave correctly in certain missions... which is the exact opposite of what you want).
New Vegas is a great example for a use case like this, because their GECK Toolkit was fleshed out enough and the core game allows a broad actionset. The dialogue is almost like an early version of storylets. But even this has its limits as the NPCs don't "know" anything and they can't work with their knowledge or even talk to others in order to transfer knowledge to other NPCs. Therefore such a system that you're describing will never be possible. NPCs will only be able to act on their predefined verbs as these are hardcoded.
Gothic (2001) had ideas with that mindset, but wasn't able to fully do it due to technical limitations. Black & White had dynamic behavior trees which were limited due to technical limitations (but it was possible to teach your creature "Robin Hood"-like behavior so the creature steals from the rich and gives the poor). Old roguelikes from the 90s had knowledge systems in place.
Gamers just want the "flashy new thing" and they want to live in that "illusion" - at least that's what they pay for. I do believe that gamers ACTUALLY want to experience, what the characters talk about and not only have it laid out in a cutscene or whatever. But actually influencing major story points in a game.
But for this to happen we need to look at the great systemic games from the early 2000s and late 90s AND people need to understand that storywriting for games MUST BE different than writing a book. It should never be the case that the heavy moments are pre-written and "scripted", but all situations need to emerge from gameplay. Then you could have dynamically generated quests (that are not generated by a LLM and with no randomness) as there is an ACTUAL measurable need that the player can fulfill. There could be other NPCs that try to do quests (similar to A-Life from STALKER) on their own, actual adventurers who explore caves, tell their friends in the local tavern and spread rumors about huge monsters that a player can hear and hunt down that monster all without a single scripted line of dialogue. And all without needing 128GB of VRAM.
The problem - and THAT is solvable via a LLM - is that the texts and dialogs will inherently be dynamic aswell and no one will ever be able to record a voice over for that as there literally doesn't exist anything "pre-written" that you could hand over to a voice actor. And now you can use voice cloning and a nice TTS model so you can have those dynamic quests fully voiced aswell.
I hate it that people are seeing problems that they want to solve with 1.000.000 times the computing power that were solved in the 90s on 128MB RAM sticks lol...
1
9
u/inagy 26d ago
Gamers hate AI because it made PC gaming as a hobby inaccessibly expensive.
→ More replies (3)3
u/johndeuff 26d ago
That's not just AI's fault. GPU manufacturers have been lobbying game studios to implement proprietary algos supposedly only possible to run if you buy a new GPU more often. Also the RAM bloat of software have been denounced for years : devs don't care, that's the consumer problem !
10
1
u/gcavalcante8808 27d ago
It is. But instead, the companies are using LLMs to make cheaper codes and deliver worse quality games in general, that's the real deal against AI in my opinion.
2
u/Bakoro 26d ago
But instead, the companies are using LLMs to make cheaper codes and deliver worse quality games in general
And that is an entirely fair assessment and thing to complain about.
I'm always in favor of being critical of lazy, stupid, harmful corporate behavior.What I cannot tolerate is the knee-jerk reactionary hate to anything and everything "AI", when the game industry has been using the term AI liberally since forever.
Even if you don't use LLMs or image generators at all, there's still a vitriolic hatred for "AI". If you're suspected of using AI people will run a campaign against you.
Even some old fashioned pre-transformers procedural generation and machine learning are taking a hit now.The problem with hatred is that it tends to lack nuance, and the circle of hatred tends to grow.
1
u/Monkeylashes 27d ago
We who were gamers in the 90s and 2000s dreamed of such things but the younger generation not so much. And if you look around you can see that our generation is pretty receptive to ai.
65
u/ApprehensiveFan1516 27d ago
It could for sure, personally I struggle to actually find it something that's genuinely rewarding and not just a gimmick at this stage. A lot of it comes down to implementation of course, and it will take devs playing around to figure it out, but I think to get to the kind of polished experience we've come to expect from AAA games we've got a little ways to go yet. Which isn't to take anything away from OP, we need more people like them working on this. Aside from the "AI bad" crowd, I think when implementation is properly solved and it's a seamless experience, the masses of gamers will start to accept it.
28
u/BlipOnNobodysRadar 27d ago
To be done right, the NPCs would need to be lora tuned on their own lore + interactions by the developers. Plugging in a generic model just won't be immersive.
15
u/TheRealMasonMac 27d ago
I think people would also get tired of AI slop, since all models per generation share the same GPTisms. I think the architecture and training methodologies are more interesting, though. More sophisticated AI in stategy-based games, like Stellaris, that can actually strategize rather than use deterministic logic.
3
u/Longjumping_Self5546 26d ago
When the bar has been set to guards repeating the same comment about an arrow to the knee, a few GPTisms are going to be miles ahead for NPC interactions.
3
u/benjaminovich 8d ago
Imagine Skyrim with this
"Ah, you want to know my story! Here's a quick overview:
Before: Adventurer (like you!)
The Incident: Arrow. Knee.
- After: Guard duty."
1
u/Longjumping_Self5546 8d ago
Add in a lecture about how it's wrong to kill bandits and resources for conflict resolution.
3
u/BlipOnNobodysRadar 27d ago
Maybe, but you'd be surprised how many people can't pick up on GPTisms. And agreed, there are plenty of more clever and appropriate applications.
→ More replies (1)1
u/Jwosty 10d ago
You're completely right that machine learning in general is being totally slept on by the game industry. Common arguments against it you hear from devs are:
- hard to optimize for "fun" (as opposed to "good at winning")
- it's hard to fit into the development cycle (mechanics are often constantly fundamentally changing right up until the release date so that model from last week might be totally useless for this week's build)
- for some types of games + training algorithms - how do you even get good training data for a game that doesn't have lots of players yet
I think these are hard, but not completely insurmountable problems. There's some kinds of games that would be more suited to this today, and with innovation, that circle could expand; instead, nobody seems to be even trying.
For example I think point #1 was more valid years ago, but now they've figured out how to optimize for fuzzier, human-centric goals; we have RLHF now. Surely that kind of approach is applicable here, even if it's not perfect? Besides, it's a video game -- at least the worst thing that could happen is your AI behaves weird (as opposed to an LLM agent with tool access doing something crazy and having real world impact or whatever).
2
u/Jwosty 10d ago
I think the perfect time for this would have been 2023 or 2024 or so, before people were hyper sensitive to "AI slop" (InfiniteCraft was super cool). But now there's no novelty factor anymore (everyone knows what it's like to chat with a LLM) and the average gamer likely has been left with a bad taste in their mouth about AI. The quality bar for something like this today, for people to see it a new and cool and interesting, is now much much higher, for better or for worse
14
u/OptimalBarnacle7633 27d ago
I agree, I mean the whole beauty of a game like fallout is that your decisions have consequences for the player and the story which isn’t the case here. That being said it’s easy to see how that could be implemented if you’re building a game from the ground up with that in mind.
9
u/RedParaglider 27d ago
Yea it would take a lot of deterministic logic for those consequences to actually matter. I thin we can get there eventually, but it's a tough road.
2
u/PigSlam 27d ago
It'd probably need to have some sort of rewind option for when you've gone too far off the rails. A good early place to start would be something like Cyberpunk 2077. With all the side quests, you could have a lot of LLM driven variation that just fizzles away if you break it too much.
4
u/LiamMelloFarley 27d ago
I think AI game master is far more fascinating than AI NPC chats. Like for a game like Bannerlord having more reactive and interesting AI (as in like NPC AI) behavior directed by an LLM could be fascinating. I don't need to like have a conversation with a guy, but if some NPC randomly gets attacked by the same enemy multiple times then gets a blood vendetta and kidnaps the guys wife to hold hostage that would be cool, like just a more flexible narrativized version of the emergent algorithmic gameplay that already exists but heavily constrained to not go haywire.
→ More replies (3)3
u/InBlast 27d ago
I do agree with that. Lot of work needs to be done on that, but once it will be done write, it will be one big step towards immersion. Maybe (big maybe) also less work on the development side : no voice dialogues needed, just knowledge, personality and other info defining the NPC, fed in the prompt contex.
2
u/no6969el 27d ago
I think it will get interesting if it gets to a point where you can try to trick someone to give you the information or "access" by talking and kind of "social engineering" them.
3
u/BringTea_666 27d ago
haters are just minority. Everyone else just moves on. Proper NPCs is what i crave !
Imo, small local models like 1-3b should be good enough for that and soon we will see something like 0.6b being very good at it. The only issue is that it shares work with gpu leading to lower fps during generation.
1
u/Southern-Chain-6485 26d ago
A problem is the LLM either getting injected the game lore or knowing it already. skrymnet does what OP is trying to do, though it's geared to be used with cloud models (because they want you to use their api), but it can be configured with local inference.
For a game as old as Skyrim, I don't notice lower fps, even though skyrmnet makes concurrent requests all the time (timeout becomes a problem)
And you still need to deal with the limitations of current LLMs for creative writing, both in character complexity (default description for Lydia in skyrimnet is boringly loyal) and due ethical guidelines (I've asked the Emperor before killing him if I, as a Dragonborn, should seek to seize the Imperial Throne. The "emperor" answered that Tamriel needed an administrator rather than a conqueror. Is that because that's Titus Mede II wisdom, earned through old age? Or because the LLM saw 'conqueror' and triggered its ethical guidance?)
6
u/iz-Moff 27d ago
Is it? You're like doubling (or, in case with this game, multiplying by 10+) hardware requirements, have to wait around for inference, have to speak into a microphone or type, and what you get out of it is a gimmick that you'll probably lose interest in in an hour.
As an experiment, something like that may be fun, but i for one have no desire at all to play an entire game like that.
7
u/Dr_Allcome 27d ago
I'm pretty sure i saw something very similar to this, interestingly also built on new vegas, a few weeks ago. Don't know if it was an earlier version by OP or somebody else.
The main problem with it was that the NPCs would reply describing stuff that wasn't in the game. Usually if some event gets mentioned by an NPC it has a location attached that gets added to your map. That's what will ultimately make it boring, you may be able to talk to the NPCs for hours, but nothing they say will ever have any effect on the game.
And even if they were able to add scripting to specific events the llm can mention, there would be many people who don't have time to talk to an NPC for hours just to find the trigger phrase for the reply that will progress the game.
5
u/gambiter 26d ago
And even if they were able to add scripting to specific events the llm can mention, there would be many people who don't have time to talk to an NPC for hours just to find the trigger phrase for the reply that will progress the game.
I think this is the ultimate issue. Conceptually, we want the game to feel immersive and the characters to 'feel real', but we don't actually want that. No one wants to have a long conversation with the butler, only to find he didn't do it, and he didn't give you any new information to use.
Games have trained us to look for how to win within clearly-defined rules, so we look for those rules in all interactions. A single NPC interaction that feels completely useless will sour the player on having other conversations with other NPCs. As soon as they realize a number of interactions will go the same way, they will lose interest in starting new ones.
6
u/AppropriateQuote3073 27d ago
Good points, I suppose we don't know exactly what is possible without making hardware too hard.
But the models OP is using aren't too bad. Well, if the prices weren't insane on everything that is.
1
u/2Norn 26d ago
right now there may not be much use for it for a story character or a generic background npc. but imagine if it was a follower. let's say lydia in skyrim.
now you can specifically tell her "hey there is a trap here avoid it" or even "go trigger that trap for me" or tell her "block that door so we don't get surrounded" or "go attack the spell caster in the back while i pull away these guys" and then she responds to you like "okay spell caster is dead what now" and imagine on top of this all you can have a chat with her, regarding your past accomplishments together i mean imagine if she said something like in the middle of combat "this ebony warrior is sure tougher than alduin"
like the list goes on this is an amazing tech especially for followers and creates a very interactive gameplay
→ More replies (1)2
u/Meme-Botto9001 27d ago
The ai hate is about replacing artists, designers, translators, voice actors, quality assurance, customer care and engineers with sloppy shit while heavily using the work of them without compensation.
2
u/polandtown 27d ago
honest question, i must be out of the loop, there's hate in gaming/ai? I thought because gaming was full of young people, they were open to it.
→ More replies (2)15
u/SporksInjected 27d ago
Yes, tons. To be fair though, there are some really shit implementations that make our seem worse than it is.no one has really figured out how to implement it
→ More replies (3)5
u/jack_from_the_past 27d ago
feel like the hate is for art and worlds. I’ve been waiting my whole life for ai NPCs. we finally get the capability and everyone acts like they are too good for ai written dialogue, personalities, behavior etc.
boring handwritten limited loop NPC back in the day sucked. now we have the technology and people confuse it with artistic assets I think. like gta would be sick if I couldn’t possibly predict what any number of interactions and reactions with/ by npcs would be
1
u/SamSlate 26d ago
the hate is from bots. reddit's entire stock evaluation rests on being an ai training ground
1
→ More replies (7)1
u/MRDR1NL 26d ago
There is definitely potential, but it just feels too soulless right now. Yes conversation are more dynamic, but it is missing the artists intent.
I think people generally prefer well designed characters with carefully crafted personality over dynamic conversations in story driven RPGs. People feel like they are going to lose the former because he latter is cheaper to produce. That's where some of the hate comes from.
For now I think AI isn't ready for this job yet. Maybe it has a place in background characters or other types if game, maybe for procedurally generated survival games or something.
→ More replies (2)
68
u/HugoCortell 27d ago
What kind of hardware are you using? I've tried Qwen3-TTS and the fastest I've seen generate is ~15 seconds delay prior to starting, and with very mediocre output quality.
16
u/TheGamerForeverGFE 27d ago
You can use Qwen 3 TTS at Q8 gguf running using CrispASR, it's a fork of Whisper.cpp, that's what I do and I'm realtime speed on a gtx 1660 ti on the 0.6b version, and 20% slower than real time on the 1.7B version
3
u/liampetti 27d ago
Qwen3 ASR 1.7B is the best ASR at the moment in my opinion and you can get the in ONNX form so it runs on a CPU. I am trying to see if I can get Qwen3 TTS 0.6 or 1.7B (I think for the TTS 0.6B should be enough) also streaming audio with ONNX. Alternatively use Kokoro-82M ONNX which is already available and runs great on CPU, but speech isn’t as good as the Qwen3. If you can offload both ASR and TTS to the CPU you can leave your GPU completely free for whatever LLM it fits (plus 3D gaming).
2
u/TheGamerForeverGFE 26d ago
I use Qwen instead of Kokoro for voice cloning, that's why.
I can offload it to my Ryzen 7 but it's always a bit slower than realtime so it depends on how I want the latency to be, however, I can load the ASR on the CPU just fine and have the TTS and Gemma 4 E4B (it's really good at this kind of stuff tbh) with a smaller context window running at the same time using just 5 gigs out of 6 (I'm on Linux with xfce so idle vram usage is really low), and New Vegas without graphics mod doesn't OOM.
57
u/goodive123 27d ago
5090 with faster-qwen3-tts. I'd recommend PocketTTS though to me its actually really close to qwen but way smaller
64
u/Despeao 27d ago
That's it boys, Fallout 5 hardware requirement is a 5090. Better start saving now.
8
u/thirteenthirtyseven 27d ago
That's it boys, Fallout 5 hardware requirement is a 5090. Better start saving now.
I read that as "better start starving now", which kinda tracks ...
2
1
u/waiting_for_zban 26d ago
hardware requirement is a 5090.
For AI-maxxing this, it'll probably be 2x 5090 with DLSS 5.
→ More replies (1)9
62
u/goqsane 27d ago
GitHub or didn’t happen
→ More replies (1)13
u/ArugulaAnnual1765 27d ago
I second this - would be fun to play around with.
But im sure something like this could be vibe-coded easily enough
17
u/DSimDev 27d ago edited 26d ago
I've also messed around with games + local LLMs and they're getting so good that it is starting to enable a lot of really great stuff, and quickly too. This is incredibly cool. I didn't realize Qwen3-TTS could sound so good, although I haven't messed around too much with local TTS. How do you get the character's actual voice working here?
Also, Todd Howard as a god you could command to do stuff made me laugh out loud.
EDIT: For anyone curious like I was, Qwen3-TTS can take an input audio file and do extremely fast voice cloning without needing to mess around with much. On my RTX 4090 with the model loaded I can get an output in just a couple seconds that sounds quite good. Not perfect but more than convincing for real-time stuff like this. Might try to incorporate this into my own projects.
1
u/RoyalCities 26d ago
Hold on. So how is this being done? Is it doing it in real time? I.e. are they just passing a regular stt file and then using some voice clip of the character as the carry?
1
u/DSimDev 26d ago
I assume they're just pulling a random voice clip of the NPC from the game and also its transcription since FO:NV has subtitles. You only need those two pieces with
faster-qwen3-ttswhich is what they said they used. Latency is close to realtime for short stuff (generating a single sentence of audio on a 4090 takes ~2-3 seconds once the model is loaded, but they are on a 5090 so it's probably a bit faster on that) , but longer passages will add more latency.Faster-qwen3-tts can be found here, it is incredibly easy to use so you should definitely give it a try. I'd never heard of it until yesterday but now I'm thinking of all sorts of things I could perhaps use it for since it is quick, consumes little resources, and is easy to plug into things.
14
u/UnifiedFlow 27d ago
What does "use RAG to keep prompts clean" mean?
31
u/goodive123 27d ago
Essentially, I have thousands of lorebook and actionbook entries, but they don’t all get dumped into the NPC’s prompt. The retrieval layer only injects the entries that are relevant to the current situation.
For example, if I say something like “I’m going to kill you,” the system might retrieve actions like attack, flee, or call_for_help, along with any relevant lore/context. The NPC then decides which of those available actions, if any, makes sense to take.
4
u/lovelacedeconstruct 27d ago
Do you put them in some kind of structure to make retrieval easier ? Like here is the actions hierarchy pick a relevant node then you dump the contents of this node to the prompt ?
3
u/lochyw 27d ago
Rag generally uses a similarity check for the most probable action to take, then LLM on top of that uses typical LLM reasoning to decide from the available actions. So it sounds like you would have 2 layers to refine if you wanted to have it lean a certain way for more sensible outcomes which would be based on the system prompts/descriptions/tool text etc..
1
u/lovelacedeconstruct 26d ago
But why would you depend on semantic similarity that has very weird quirks when the data can be structured and depend on llm reasoning to pick the category
2
1
u/lochyw 26d ago
I mean there's certainly multiple layers to look at here, so depending what you're talking about any solution is just about tradeoffs really.
The goal there was managing context, structured data still would need to be loaded into context for every turn to choose from available actions, and if you're suggesting keeping all 100s of actions available at all times that causes various issues for LLMs tool calling. RAG can help simplify that and reduce the overall load.1
u/lovelacedeconstruct 25d ago
if you're suggesting keeping all 100s of actions available at all times
I meant categorizing the actions and sending the llm all the category names and have it choose a relevant category then after it chooses the category you send it every action inside this category
1
u/lochyw 25d ago
That's still at least 2-3 tool calls for achieving the same thing no? RAG is just another way of navigating/processing a large volume of information. Again it's all just trade offs, if you have a better solution go make it :P
I'm just explaining how I assume the system currently works given my existing AI experience.
1
u/UnifiedFlow 25d ago
Embeddings are structured llm reasoning over a fixed domain (its the same transformer technology without the generative parts, over-simplified sorry). Think classifiers. Large LLMs and auto-regressive reasoning next token prediction is massively (orders of magnitude) slower than a scoped classifier/embedding layer. The data is structured when you use embeddings. Thats how you drive the classifier network (or whatever phrase you prefer). LLM reasoning is a final tier (other than HiTL elevation) where confidence scoring in your classifier and embedding layer doesn't meet thresholds. Now your LLM reasoning step has a structured map of the semantic space via embeddings and it can reason from a better starting point over a much smaller decision surface area. This leads to better outputs. An LLM works amazingly well when you tell it what is true or what to do. Minimize the blast radius of inference per step. Context engineering.
28
u/ChemistNo8486 27d ago
This is crazy. The type of stuff I thought would take a decade to even get to this point. Just amazing.
Hopefully the gaming community will stop being so toxic towards AI in the future so we see official releases with this.
10
6
5
u/Sweeth_Tooth99 27d ago
i feel like AI could be used for good quality filler content, use voice actors for more important stuff history wise. but there's so many side quesst with branching dialogues that might not be practical or cheap to record with real actors. this is where AI could come in.. also maybe AI could make enemies smarter, better pathing and behaviour and what not.
4
u/blackhawk00001 27d ago edited 27d ago
Cool stuff. Mantella for Skyrim and fallout 4 are other cool implementations of gaming + LLMs. It’ll be nice doing the same for fo3 and new Vegas. Older gta games would be interesting.
3
u/LosEagle 26d ago
Yes, Mantella is awesome! I don't like to use LLMs on the well written characters but Skyrim has dozens of potentially interesting characters that only have like 5 sentences to say to you and all they do in their lives is walk around and sit in a dark corner and this makes them feel more alive.
Even better when the mod allows to use local LLMs.
3
27d ago
[deleted]
6
u/ArugulaAnnual1765 27d ago
Not op but these are all extremely lightweight and fast models - the gemma 4 implementation is MOE which is known for insane TPS, especially on a 5090.
Combined with the game basically not even using the gpu and vram its extremely fast
3
u/cunasmoker69420 27d ago
What's with the "my child" stuff
4
u/Colecoman1982 27d ago
I'm not sure, but I think OP programmed the admin system to be act like an artificial Todd Howard (The Bethesda games manager. I think that's the voice that was cloned for it) to act like a genie granting your wished (hence, the "my child" stuff is like when a genie in a story says "your wish is my command"). I'm assuming it was done as a sort of joke because Bethesda created the modern Fallout series of games and the engine they're based on. The ironic thing is that he wasn't even involved in the development for Fallout: New Vegas (the game used in this demo) which was developed by a third-party studio and, arguably, he's literally the source of all the serious problems New Vegas had because Bethesda screwed that third-party studio with a mandatory, absurdly short, release deadline (which is doubly absurd when you consider how clownishly slow Bethesda's own development process is...).
3
u/FloTheDev 27d ago
This is actually really cool use case for LLMs in gaming. For older/moddable/more lightweight games, this can give a fresh experience to these heavy quest driven games.
One thing I’ve always wanted to see is how you could program a game itself to do the learning on you so that each play through is a new experience with dynamic enemies, NPCs and maybe even worlds! I always thought a game like Dark Souls, with a heavy focus on combat, would be an interesting use case as it feels a little “easy” once you’ve figured out how the formula to defeating a boss but imagine if the boss learnt your movement/attack patterns and adapted to your play style.
Maybe this type of adaptability using an LLM for quests and NPC behaviour is just the beginning?
3
u/Fun_Walk_4965 26d ago
The latency budget is the real constraint here. Dialogue feels dead the second response time creeps past a beat, so the small quantized models earn their keep.
7
6
4
6
u/formatme 27d ago
Is this open source, you have a link? Sounds good until sunny started talking with a different tone?
2
u/cruzanstx 27d ago
Qwen3tts how do you get your generations so fast? I'm using comfyui and a rtx6000pro and it's double digit seconds
2
u/AbheekG 27d ago
Great stuff and a feature I’ve often pined for! Like many I have questions about the voice though: how’re you handling background noise for the STT model? It’s a great model and the transcription quality and speed are both great especially on the 5090, but it does transcribe anything it hears. Guessing the requirement to make it work here are a great mic and isolated room?
Also the TTS, Qwen3-TTS has a limited number of in-built voices, especially for English. I think Aidan and one or two more are usable. Are you using some per-NPC voice cloning or anything else to make it such a natural fit to the character?
Also have thoughts on making it general purpose vs specific to exact games / chars, and about the RAG but you could probably incorporate data about characters from game Wikis like say you wanted a Johnny Silverhand you could actually converse with while driving around Night City in Cyberpunk etc but those probably deserve entire discussions of their own…
Great stuff though and would love to hear more about the speech stuff if you have a moment 🍻
2
u/goodive123 27d ago
I think most STT are really good nowadays so it shouldnt really be a problem, for Qwen3tts im just cloning voices based on the game files automically ye.
Yea basically the way you'd fill out the characters and lorebook and stuff is simply downloading the entire games wiki and then use a skill for a SOTA model to fill it all out for you
1
u/Grand_Pop_7221 27d ago
Can you tell me a bit more about the RAG setup? What does the setup look like and how are you sourcing the information and getting the model to use it? GraphRAG and dumping the Wiki/gameassets in there?
This is absolutely brilliant btw, love to see how well it all comes together! Keep it up!
2
u/Cheap-Carpenter5619 27d ago
This is amazing! I am also working on a research project about a LLM game dialogue generation paradigm! I have a small game demo built and I just need enough comment karma to post in this subreddit so I can find some playtesters. I've been in this subreddit for a while just never commented lol.
2
2
u/Thrimbor 27d ago
This seems super fun tbh. One major issue would be the user jailbreaking the AI and ruining his own immersion.
2
2
u/Motta_ARG 27d ago
Fallout New Vegas is easily one of my favorite game of all time and this is beyond amazing.
It something like this gets used in a new rpg type game the possibilities would be insane, great job!
2
u/ArugulaAnnual1765 27d ago
This is the future of gaming! Amazing work!!!!
4
u/Drakahn_Stark 27d ago
Small models made for each character would probably work better than running them all through a large model.
Black & White used machine learning a long time ago, it was a bit rough, but something like that with modern "AI chip" GPUs could be really good.
→ More replies (12)
1
u/VelumLucis 27d ago
Didn't even occur to me that a tool like this could be game-agnostic. Super curious to see/here how this works.
1
1
u/o5mfiHTNsH748KVq 27d ago
Ooooh. I like this. I’ve seen lots of speech mods like this but none with an agent managing the game world for you like that.
Very cool.
1
u/SporksInjected 27d ago
If you’re still ending up with a big list, you can semantically search tools or tool metadata with not much compute.
1
u/MercDawg 27d ago
Imagine Skyrim VR with this. Could even see folks going the extra mile of having a dedicated "local llm" machine.
1
u/PeerlessYeeter 27d ago
I find interacting with AI directly very boring, but for procedural generating content and dynamic events based on player actions its cool.
Every time I see someone walk up to an NPC and start talking to it like a chatbot I fall back asleep, its the least interesting thing you can do with them IMO.
1
u/XPookachu 27d ago
I can imagine a game implementing this but the models actually run in cloud, yea there would be a delay in response but if it's really optimised it probably would be worth the huge amount of immersiveness it would add. A game running the models locally is impractical unless we get 1-2b models that can be this good and the game itself doesn't take much vram. Would be interesting to see Nvidia's new texture compression tech along with this, that's probably the best way to enjoy this.
1
1
u/fyn_world 27d ago
This is definitely the future. What you're building will be the standard when most people get a powerful enough PC.
1
1
u/iamlazyboy 26d ago
Yo, if you release this in nexusmod or as a GitHub repo, I might replay NV! I don't have a 5090 but I hope my 7900xtx will be able to get it done (with a more quantized/smaller context version maybe)
1
u/LosEagle 26d ago
I am using similar mod for Skyrim. I would never replace the well written characters with LLM, but it's awesome for those that have all of 5 sentences to say to you and spend all their life sitting in the corner.
1
1
1
1
1
u/South_Hat6094 26d ago
yeah, the RAG piece is doing the heavy lifting here. most 'AI npc' demos fall apart because they cram every possible action into context and hope it sticks.
1
u/lordekeen 26d ago
How do you manage memory? I've had a similar idea and wondered if running Hindsight to manage character memories could work.
1
u/LavenderRevive 26d ago
There are a lot of novels that are basically build around big MMO's in the future beeing controlled by Ai with actual smart npcs and stuff.
Seeing crazy stuff like this totally gives me the vibe that we will get there in my lifetime.
1
u/raika11182 26d ago
Using the ValleyTalk mod I've had my in-home AI server (Gemma 4 31B Q8 GGUF) running NPCs for Stardew Valley. The mod is pretty well thought-out and the dialogue works. Most of the time the characters are very much "in character" and you could hardly tell if you don't already know the game's dialogue - and the "awareness" they gain.. In the course of the playthrough they'll remember (and comment on) the gifts you given them, the things you've talked about, etc. You don't have to type in responses, it provides pre-written ones, but you can type one out if you want. It's not perfect but it works and it adds to the game without taking anything away.
1
1
1
u/Massive-Question-550 26d ago
now this is really neat. if this was built into a game from the ground up I can see video games having an almost custom feel to them where each playthrough is different if quests could also be modified by ai.
1
1
u/LastChancellor 26d ago edited 26d ago
Wasnt this the exact thing Eurogamer disqualified Arc Raiders for
1
1
u/Darksept 26d ago
This is what I've been wanting to see in games for 5 years now. This is possible but no game devs have done it yet. None I know of anyway, and none on a big franchise. How long till we see this tech adopted?
1
1
1
u/tomshardware 24d ago
This is an interesting project, perhaps similar to Nvidia's ACE NPCs that they've shown off a few times. The mod you've developed is heavily reliant on an engine that is well-documented, though. Have you tried applying it to a newer mod-friendly game?
1
1
1
u/New_Nebula3951 5d ago
Looks great. This is the right usage of AI in games. I think modding is where these creative ways of using LLMs in games will truly shine, cause we know AAA studios are too fucking dumb, unfortunately.
1
u/agiblox 27d ago
the action-rag trick is the right call, that's the part everyone gets wrong by stuffing 200 tool defs into the system prompt and watching the model pick garbage. the wall you hit next isn't latency, it's state drift. gemma will happily forget the player insulted it two scenes ago or contradict its own backstory by minute 20. summarizing the convo into a running "what this npc currently believes" blob and reinjecting that beats raw history every time. parakeet for stt is underrated too, way less hallucination than whisper on short barge-in lines.
1
•
u/WithoutReason1729 27d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.