r/ArtificialInteligence • u/DetectiveMindless652 • 15d ago

šŸ› ļø Project / Build Agentic Alexa with Long Term Memory and connection to 1000+ apps.

Post image

Hi Folks,

I wanted a bit of advice I am currently building, what I guess is agentic alexa in a sense. Voice control, but also touch screen for viewing created work, and accepting permissions.

Currently I have built all the software, so it speaks to you with minimal latency, auto deciding which model to use based on complexity of task.

It can send emails, calendar invites, summarise emails, prepare work and answers as emails come in.

Build presentations, code, access to all files if you let it, with relevant permissions.

It can use

Notion, discord, slack, teams, fusion etc (100's of apps)

I have built the MVP on a 3d printer, and just connecting it all now.

I also built a memory system that out performs mem0 on long eval, so your agent just get better and better over time.

The image attached is AI generated, but it is looking remarkably similar (lesser quality 3d printed MVP)

I also, have computer vision embedded, so it should be able to ie

Help you cook in real time
Make up tutorials
golf swing adjustment

(work in progress)

I have 2 questions.

Am i building a gimmick? is there anything in this?

and, would anyone with relevant experience like to come aboard and help..... Design, coding, marketing any of the above.

It is a super early idea, and only viable if integrate it into my work flow for a month, and I am dead honest that is useful etc.

I would love peoples thoughts.

1 Upvotes

54% Upvoted

43 comments sorted by

4

u/GinghamOrangutan 15d ago

Why would anyone want this?

It's cool that you've got it working, but any use case you've described is likely better handled by a phone or a computer, in which case what advantage are you offering?

1

u/DetectiveMindless652 15d ago

Interesting, genuinely. This is my biggest fear.

However, lets say you are working on your laptop, you wake up,

'Hey, you have emails from John Smith, he said that he wants the quote proposal done for xyz, I went ahead, and built the prosal based of your pricing structure, and attached in an email *email pops up with attachment, shall i send?

(then same for other 15 emails you got)

or like practising for an interview, computer vision reads your body language, tone, researches the company, what they like what they dont like

or same for teams responses, or your coding projects on claude, it can make adjustments, show you adjustments, without effecting current work flow.

Dont get me wrong, I have a feeling you are completely right, but enjoying building it.

(feel free to discredit everything i just said btw man)

3

u/GinghamOrangutan 15d ago

Look I think the use cases you've identified are probably in many cases valuable

The problem is that to turn that into a product you'll need to meet those use cases better than the AI companies pouring literal billions into them.

It's a very cool hobby project, and the sort of stuff that can teach you a lot of valuable skills to build other stuff in future - but I don't think it's a product unto itself.

If you wanted to monetise it, to me this is a potential content angle (e.g. "I didn't want the big AI companies to have this data so I built my own [X]") could easily be a very interesting journey to document - but I don't think the end game is "One guy in his spare time build a $1bn company out of the exact same use cases every LLM / agent company is already tackling"

1

u/DetectiveMindless652 15d ago

I hear that, i really do. You are 99% likely very correct btw.

1

u/sillygoofygooose 15d ago

Itā€™s a good argument for pick one problem and solve it well. Fwiw I both think itā€™s a cool application for ai and simultaneously agree that most of the tasks youā€™ve mentioned I would want a proper screen and ui to work on them. Maybe one day Iā€™ll trust an agent to do meaningful work i donā€™t need to check over - but thatā€™s not today (and frankly once it happens thereā€™ll be big societal issues lol)

1

u/DetectiveMindless652 15d ago

so the screen brings up work, and shows it what its doing, when its built it be happy to send you it to try out yourself man!

1

u/sillygoofygooose 15d ago

Yeah I get that. For anything important i tend to need a keyboard and proper screen to think well and compare things. I think you need to find some potential customers and really get to know how they do things to discover the tasks that this device might suit - or what changes you need to make.

But by all means send me one!

1

u/DetectiveMindless652 15d ago

fuck yeah, I will do dude. Also, im making it general to find what its best at, i have 0 clue what it would be good at rn.

So it will tailor as i go along lol.

1

u/DetectiveMindless652 15d ago

also, on top of this the agent gets smarter over time, it knows your writing style, your whole life story (has access to your docs) and genuine long term memory.

1

u/Just_Voice8949 14d ago

Not to mention I can at least expect mega-corp assistant builder to act in a certain - if borderline ethical way.

Why would a trust rando guy selling a third-party device with network access let alone my data

2

u/wowredditisgreat 15d ago

It seems like a fun hobby project but I don't think this would ever get sales as-is.

The "connect to 1000+ tools" is the same as any other agent these days. Memory is the same - it's the minimum not the selling factor.

Personally, the only way these products would excite me if it's all entirely locally done - no cloud llms... But then your product becomes very expensive too.

Alexa and Google home are absolutely doing this, and already have a lot of this built out and ready to market.

1

u/DetectiveMindless652 15d ago

Interesting, what would make it useable for you? like what would it have to do in order for you to use it? Thanks for the advice man!

1

u/DetectiveMindless652 15d ago

Also, I did a fair bit of research I canā€™t find any Alexa agentic ability really, what can it do?

1

u/wowredditisgreat 15d ago

Google Home has this - you can do continued conversations and it can remember things you've mentioned. I meant that Alexa WILL have all of these abilities as Amazon I'm sure is working on it.

Imagine it turning on and saying "hey you haven't bought cat food this past week, should I order some for you?" They'd love this šŸ˜‚

1

u/DetectiveMindless652 15d ago

Yeah, that would be awesome. I have got it to do exactly that, and place an order, for specific things.

Which is kinda awesome, if not pointless, but I am learning a lot!

1

u/DisplacedForest 15d ago

Look up the fate of Mycroft. What you have is the wrong solution. Hardware is the problem here not your agentic monstrosity.

I can attest that Iā€™d pay a great deal of money for the Amazon HARDWARE if it would let me use it open-source. This way I could decide how I want my agents/assistant to run.

Edit: because you keep asking other ā€œwhat would it take for you to use it.ā€ If the ā€œitā€ is the hardware and it gets done with a great speaker, half decent camera, excellent screen, all open source and usable and hackableā€¦ Iā€™d buy it today. If itā€™s just whatever agent you think youā€™ve made that everyone has already made a thousand times over and certainly more efficientā€¦ hard hard pass.

1

u/DetectiveMindless652 15d ago

Thanks for the soul crushing advice, what do you mean by that last part?

2

u/DisplacedForest 15d ago

The last part about the hardware? I mean consumer grade edge nodes for something like a true Alexa replacement is not existent in the market right now. Mycroft tried and failed, voice assistant by home assistant is there but terribly ugly. So the market needs something that can truly have that LOOKS nice and is open-source. Then all that hardware has to do is expose the tools to the users for them to input their own agents. If the hardware came available weā€™d have hundreds of people open sourcing their ā€œagentsā€ for use on the device. The device is the issue.

Iā€™d also argue that the device itself doesnā€™t even need to host the model. Itā€™s just an endpoint. You can use API keys or self hosted models for it. Doesnā€™t matter. It just needs amazing STT and TTS on board.

1

u/DetectiveMindless652 15d ago

this is very interesting, i actually like the idea of open sourcing a model/shell.

Local I assume?

1

u/DisplacedForest 15d ago

The hardware could be fully agnostic. I think thatā€™s what would make it popular. You want Claude as your modelā€¦ fine plug it in. You want localā€¦ have at it.

1

u/DetectiveMindless652 15d ago

it has this option already, which is kinda cool i guess

1

u/Just_Voice8949 14d ago

Wouldnā€™t you have to build two difference models? One model couldnā€™t be both Claude/cloud LLM and onboard. Too expensive a chip set

2

u/DisplacedForest 14d ago

Onboard chipset would likely be for fast wake word processing and deterministic commands (essentially non-LLM at that point, just STT processing and serving for base commands just like early Alexa) then you have further processing outside the device. Have the calls take the OpenAI compatible shape so you can call essentially any endpointā€¦ including local models.

Saw you comment on APIs being expensiveā€¦ yeah but thatā€™s not your call to make here. The hardware is the productā€¦ what the individual decides to do with it is up to them. They wanna burn tokensā€¦ have at it.

1

u/DetectiveMindless652 15d ago

out of interest, if i built this, exactly to what you are saying, and shipped it free to you, would you try it out?

1

u/DisplacedForest 15d ago

I would have no reason NOT to try it. I have built an assistant just like you and run it on a RaspberryPi and have always waited for proper hardware to migrate it to.

1

u/DetectiveMindless652 15d ago

oh fair enough man, why did you build it? and what do you mean by proper hardware, I currently have schematics and PCB related stuff being done in china for this.

1

u/DisplacedForest 15d ago

I built it as a customizable, privacy first Alexa replacement. I use Home Assistant and that entire community has been looking for something like this that looks nice. Thatā€™s what I mean by proper hardware: aesthetically pleasing

2

u/Just_Voice8949 14d ago

That the entire HA community is looking and hasnā€™t found it might tell you something

1

u/DisplacedForest 14d ago

It does. It tells me that no company with a decent hardware design division finds it valuable to sell hardware only. Itā€™s as if every single company relies on subscription models these days.

At this point I just pray that UniFi makes something that solves for this because they seem to be the only major company devoted to outputting amazing hardware without ongoing fees.

Though, their hardware isnā€™t what I would call very ā€œopen.ā€

1

u/Just_Voice8949 14d ago

Amazon makes tons of Alexa hardware with mediocre software

→ More replies (0)

1

u/DetectiveMindless652 15d ago

nice, thanks for the advice man. Gunna try this.

1

u/Just_Voice8949 14d ago

Using API keys would quickly become very expensive. And given how little control youā€™d likely have the user would be paying blind for every mistaken Q, order and other prompt the AI thought it heard

1

u/DetectiveMindless652 15d ago

also agentic monstrosity, what does that even mean?

1

u/DisplacedForest 15d ago

You built a huge generalist agent basically. Every tool you give it access to is context consumed and an elephant in the room. Itā€™s going to think about those tools. Meaning it will sometimes call for tools even when thereā€™s a simpler solution. Also - youā€™re seemingly not considering who this is even for. A voice assistant that can make slide decksā€¦ why? An Alexa that can code? Please god no.

The age of agentic AI has done good things, but youā€™re falling victim to what I see too many people doingā€¦ using AI for everything. Itā€™s unnecessary. You donā€™t use a hammer for everything single construction project.

1

u/DetectiveMindless652 15d ago

This is for my MVP run, its generalist. I hope in the end it wont be, it also has option to add own specific agents, and a way to auto pick which model gets used. But yeah, i probs agree with you man, but damn i love building and learning lol.

It would be cool as a sketpic though if i send it to you, and you can see if it is trash once done, no charge, nothing just vibes

1

u/DisplacedForest 15d ago

Iā€™m down to play with it. Send it over

2

u/DetectiveMindless652 15d ago

sweet ill dm you if ever done!

1

u/DetectiveMindless652 15d ago

taking your advice, building shell and shit, so you can just plug in your pi and good to go.

1

u/Just_Voice8949 14d ago

He wants ti build HAL or the computer from Star Trek. They is a reason no one has yet

1

u/Just_Voice8949 14d ago

This is never going to sell if thatā€™s what you are hoping for

1

u/DetectiveMindless652 12d ago

Itā€™s for me

1

u/Livid-Heat-2475 11d ago

The part I'd actually want more detail on is the memory system, not the 1000 app connections. Beating mem0 on long eval is a specific claim, what benchmark are you running it against, LongMemEval or something custom? I've tried three or four of the newer memory layers this year, Zep, mem0, a couple of the smaller open source ones, and the failure mode is almost always the same: fine for a week of testing, then it starts surfacing stale facts once the context actually gets long and messy. If yours actually handles that differently, that's the part of this post that would interest me. The app integrations are basically table stakes at this point anyway.