UIP — Unified Integration Platform

A personal assistant for one user on one machine (Fedora 44 KDE Wayland, laptop + BenQ monitor): an always-on 3D avatar that listens, talks, calls tools, runs routines, and acts for you within guardrails. It works with any LLM (OpenRouter), local models (Ollama), or OpenAI Live.

Docs

File	What's in it
01-recon.md	What your machine already has (a lot) and what's missing
02-research.md	Options and recommendation for each layer, with sources
03-architecture.md	Proposed design, language, guardrails, routines, packaging
04-testing-and-sandbox.md	Nested KWin, dry-run, test VM, test plan, and pass criteria
05-cross-desktop.md	How it could work on any Linux desktop
sudo-later.md	Commands for you to run (sudo, recordings, VM, accounts)

Key findings

1. The 3D, click-through, always-on-top avatar is feasible on KDE Wayland with what's already installed: QtQuick3D + LayerShellQt's QML module + system PySide6. Layer-shell "overlay" with an empty input region means clicks pass through and focus is never stolen.
2. No sudo needed for desktop control. KWin D-Bus handles windows, AT-SPI handles UI elements, and the RemoteDesktop portal (v1.22, persisted approval) handles keyboard and mouse. ydotool is only a fallback.
3. Everything is an MCP tool. Your existing router_mcp.py plugs in as is. computer-use-linux (an MCP server covering KDE, GNOME, Hyprland, etc.) may give us desktop control for free, so it gets evaluated first.
4. Two voice modes: Cascade (wake word → local STT → any OpenRouter model → local TTS, can run offline) and Live (OpenAI Realtime, about 1.6–5¢/min). Both use the same tools and guardrails.
5. LinkedIn has no personal feed API. The safe route is summarising LinkedIn's emails. YouTube (subscriptions + RSS), Gmail, and Reddit all have free personal access.
6. Guardrails are in code, not the prompt: tool risk tiers, a policy gate, confirmation, no delete tools at all, untrusted-content isolation, dry-run, audit log, and spend caps.

Roadmap (each phase ends with a demo you can test)

Phase	Deliverable	Test
0: Spikes (1–2 days)	(a) a glowing orb in QML on layer-shell, click-through, both screens; (b) wake word + STT + TTS loop printing text; (c) computer-use-linux evaluated in nested KWin or the VM	The criteria in 04
1: Talk	core loop + OpenRouter + avatar states + push-to-talk + dry-run + audit log. Tools: open app, window focus, notes, router MCP	Voice commands work in dry-run for a week
2: Morning routine	YouTube subscriptions summary, Gmail summary (incl. LinkedIn emails), Reddit, daily note in Obsidian, systemd timer	The brief lands every morning with links
3: Act	Leave dry-run for reversible tools; confirm flow for sensitive; AT-SPI + portal input	Injection test suite passes; VM end-to-end tests
4: Live mode + polish	OpenAI Realtime backend, barge-in, custom 3D character, per-screen behaviour	Latency targets
5: Appliance	Rebuildable image (kickstart/bootc) + other desktop adapters	Fresh VM → working UIP

Decisions I need from you

1. Language: Python (core/voice/MCP) + QML (avatar), as proposed? Or do you want Rust/Go somewhere?
2. Wake word name: "UIP"? "Jarvis"? Something else? (Short 2–3 syllable words work best.) Also: wake word, hotkey only, or both?
3. Default voice mode: Cascade (cheaper, any model) or Live (OpenAI, most natural)?
4. Avatar look: start with a JARVIS-style orb (no model needed) and swap in a character later? Please send the Pinterest image as a screenshot. I couldn't load the page.
5. LinkedIn: email digests only (safe) or browser scraping (ToS risk)?
6. Privacy: can email bodies go to cloud models via OpenRouter, or local-only?
7. Git hosting: where should this repo live? (1) git.linhex.com (2) github.com personal (3) github.com/TTS-FZLLC
8. Notes location: ~/Projects/Notes/UIP/ in your Obsidian vault OK?

Not done / couldn't do

• The Pinterest page needs a login, so I didn't see the reference image.
• Nothing installed, no VM created, no git init, and router.env wasn't read.
• Pricing and model numbers are from September 2026 web sources. Check them again before relying on costs.