A personal AI assistant that can do things — not just chat — is usually three layers: a UI, an LLM backend, and a set of tool servers the model can call.
Architecture
UI (web / mobile) ─┬─ chat stream → local or hosted LLM
├─ tool calls → tool servers (MCP or HTTP)
└─ shared database (conversations, data)- Tool servers give the model real capabilities (control devices, query data, run tasks). The agent loop ties it together.
- Expose tools over MCP so any compatible client can use them — see Building MCP Servers as systemd User Services.
- Route models by need — cheap/local for everyday turns, a bigger model when required.
Design tips
| Concern | Approach |
|---|---|
| Persistence | One shared DB so every surface sees the same data |
| Latency | Stream tokens; keep tool calls fast |
| Reliability | Run tool servers as supervised services that auto-restart |
| Privacy | Prefer local models for personal data |
Keep tools small and well-described. Smaller models do multi-step tool use poorly; tight schemas and short tasks make or break it.
Related: Agentic Tool-Use Loops · Building MCP Servers as systemd User Services · Meeting Transcription with Local ASR