# The Core Control ecosystem

A complete reference to what this system is, what each part does, and where the
boundaries are. It is written for three readers at once: someone deciding
whether to run it, someone who has just been handed a deployment and needs to
know what they own, and an AI assistant that will be asked to extend it years
from now.

This page is **public and deliberately unclassified**: it describes capabilities,
interfaces and boundaries, and contains no credential, no private host and no
operator-specific value. Anything you would not put on a marketing page is in
the internal docs, behind the admin surface.

---

## 1. What it is

A self-hosted control plane for a fleet of devices you own, with an AI assistant
that can operate them.

One server, one database, one set of client apps. The server is the single
authority for identity, devices, tunnels, chat, plugins and billing; the clients
are thin, self-updating agents that register, hold a tunnel, and do what they are
told.

Nothing about it is a hosted multi-tenant SaaS you must trust: a deployment is
yours, on your host, with your keys.

| | |
|---|---|
| **Server** | Node.js / Express, MariaDB. API, admin console, tunnel relay, chat proxy, plugin runtime, mesh coordinator. |
| **Clients** | Windows, macOS, Linux desktop apps and an Android app. Each exposes a tool catalog and a control surface. |
| **Browser companion** | An optional companion browser that a plugin can drive over CDP. |
| **Plugin runtime** | A separate, versioned engine shipped to devices and sandboxed per plugin. |

---

## 2. Architecture

**Core and fork.** The product is built as a *core* (product-neutral) plus a
*brand seam*. Everything a user sees — the name, the domain, the application ids,
the FRP tunnel prefix, the installer identity — comes from one `brand.json` and
from validated fork input. Everything internal — source namespaces, binary
names, HTTP routes, the wire protocol, the plugin schema — is shared and
deliberately not rewritten, so a patch or a tool written for one deployment works
on every other.

**One authority, many surfaces.** Clients never talk to each other directly for
control. They hold a tunnel to the server; the server routes. That is what makes
device revocation instant and what makes "which devices are online" answerable
in one place.

**Heartbeat is the source of truth for presence.** A device is online because its
tunnel is registered and it is heartbeating — not because a page says so. Every
list you see is derived from that.

---

## 3. Clients

| Platform | Notes |
|---|---|
| **Windows** | Full desktop service + UI. Installer identity (AppId, upgrade code) is permanent per product — it is the upgrade path. |
| **macOS** | App bundle and a background service, each with their own bundle identifier. |
| **Linux** | GUI plus a root service that keeps a logged-out box reachable. The service installs unattended and owns its own copy of the code the GUI cannot write. |
| **Android** | Runs as an installed app with an accessibility grant; supports headless operation. |

All four are built from the shared core and stamped with the deployment's brand
at build time. Client builds are **build-time identity, not runtime settings**:
the values have to be *inside* the binary.

---

## 4. Accounts and sign-in

Three ways in, and no self-registration without one of them.

| Path | How | Notes |
|---|---|---|
| **Email + password** | Created by an administrator (`POST /admin/create-user`). | There is no public sign-up form. |
| **GitHub OAuth** | `GET /auth/github` → callback. | Optional. Register the OAuth app per deployment. |
| **Google OAuth** | `GET /auth/google` → callback. | Optional. |

- **Access keys.** Each account has a bearer access key used by the APIs and
  embedded in client chat URLs. It is a credential: it can be rotated and it can
  be revoked.
- **Sessions.** The admin console uses an authenticated session, protected by a
  password and optional TOTP 2FA.
- **Admin identity** is *not* a user row. Machine callers use a header key;
  humans use the admin password. On a fresh database those are the only way in.
- **2FA (TOTP)** is available for the admin console and enforced where the
  deployment configures it.

---

## 5. Chat — OonaCode

Every chat model goes through **OonaCode**, which is the only chat upstream:
device chat, the plugin-authoring assistant, and the text half of a voice turn.

**How the operator configures it.** Register your own OonaCode account, get an
API key, and put it in the **server environment** (`OONACODE_API_KEY`). It is not
stored in the database — a key in the database is a second copy of a credential a
fork export must never carry, and it sits in a table a backup cheerfully
includes. `/setup` reports only *whether* a key is set and a short prefix, so you
can tell which key is live.

**What happens without one.** Chat and the plugin authoring assistant are
unavailable and `/config` advertises no chat model. Everything else works —
devices, remote desktop, plugins, mesh, updates.

**Model selection.** The client asks for a model id. A rewrite map folds retired
or provider-prefixed ids onto the current OonaCode model, so an old client does
not break when a model name is retired. Billing is computed on the model actually
sent upstream.

**Which models a client may pick** is a catalogue in the database, filled on
first boot from the account's own model list. A deployment therefore advertises
exactly what its key can call, with no list in source to go stale, and prices
start at what the upstream charges. An administrator edits both; seeding never
overwrites an edit.

**Request shape.** Anthropic-compatible Messages API over HTTPS with an API-key
header. Streaming is forwarded token-for-token; per-request usage is recorded and
deducted from the caller's balance.

---

## 6. Voice — OpenAI

Voice is a **separate provider on purpose**, and leaving it empty disables voice
only.

| Capability | Endpoint / surface |
|---|---|
| Speech-to-text (dictation) | `POST /transcribe` |
| Speech-to-text (voice turn) | voice pipeline |
| Text-to-speech | streaming TTS |
| Realtime conversation | Realtime WebSocket |
| Mic test / TTS preview | admin voice-test surfaces |

Its key **is** stored in the database and edited at **Admin → AI Providers**,
because it is a per-deployment credential the admin panel is designed to manage,
and voice is not chat: they fail independently and are billed independently.

---

## 7. Devices

**Registration.** A client registers and then heartbeats
(`POST /devices/status`). The heartbeat carries the device identity, platform,
binary revision, channel, screen geometry, and a block of state digests that
replaces several separate periodic polls.

**Presence.** FRP tunnel presence is the sole online indicator. There is no
optimistic "online" flag.

**Revocation.** Revoking a device is enforced on several independent paths —
immediate FRP push, key invalidation, tunnel teardown, and the next heartbeat —
so a revoked device does not linger.

**Staleness.** Devices inactive past the stale window drop out of remote-desktop
listings automatically, without being deleted.

**Emergency commands.** An administrator can push commands to a specific device:
tool execution, app/browser update, diagnostics.

**Two ways to reach a device:**

1. **Tool execution** — `POST /devices/:deviceId/execute`, relayed through the
   tunnel. The AI can run any tool in that device's catalog on any remote device.
2. **Remote desktop** — streaming and input, below.

---

## 8. Remote desktop and streaming

Proxied from the device through its tunnel:

| Surface | What it is |
|---|---|
| `/remote/:id/stream` | MJPEG video |
| `/remote/:id/h264` | H.264 video |
| `/remote/:id/scrcpy` | Scrcpy video |
| `/remote/:id/snapshot` | Single JPEG frame |
| `/remote/:id/input` | Mouse / keyboard input |
| `/remote/:id/audio` | Audio |
| `/remote/:id/input-ws` | Input relay over WebSocket |

Streams are logged end to end: upstream status, bytes, client disconnect,
upstream end or error. A device that is unreachable answers with a descriptive
error rather than a hung connection.

---

## 9. MCP and tool execution

Each client exposes an **MCP tool catalog** — a large, platform-specific set of
system tools (window and app control, capture, input, files, shell, device
settings, messaging, and more). The server can invoke any of them on any device
the caller owns, and the AI assistant can invoke them during a chat turn.

- **Tool catalogs are canonical in each platform repository**, because only the
  platform knows its own surface.
- **Cross-platform tool names are normalized** so one command means the same
  thing on every platform; the comparison and the normalization rules are
  documented with the control surfaces.
- **The response envelope is shared** across platforms, so a caller does not have
  to special-case each one.
- **New tools follow one schema** — including annotations such as voice-mode
  eligibility, destructiveness and category — so the catalogs stay comparable.

The transport differs per platform (a Windows executable, an in-process Android
server, a macOS Node child process, a Linux stdio child). That is a deliberate
coexistence, not drift: each platform runs what its own process model allows.

---

## 10. Plugins

A plugin is a small, sandboxed program that runs on a device and can be
dispatched to from the server and from chat.

| | |
|---|---|
| **Format** | A manifest plus a handful of source files. |
| **Sandbox** | A fixed set of operations. A plugin cannot ask for a capability the manifest does not declare, and the sandbox is structurally enforced — not a convention. |
| **Editing** | Always through the API, which records a content-addressed version on every change and invalidates the device's cached archive. Direct file writes lose history and leave devices running stale code. |
| **History** | Per-file change log, whole-plugin version log, and single-file or whole-plugin restore. |
| **Distribution** | The plugin engine is a separate versioned package delivered to devices over the heartbeat channel. |

### Plugin Builder AI

An assistant that authors plugins for you, in a **six-operation sandbox**:

1. Reads the live tool catalog as a reference.
2. Writes manifest and files.
3. Every tool it invokes during authoring is audited against the allowed set —
   a regression test suite enforces that the surface does not widen.

Its system prompt lives in the database, is seeded on first boot, and is editable
by an administrator. **Seeding never overwrites an edit** — the insert runs only
when the row is absent.

### Browser control (CDP)

Where a deployment ships the companion browser, a plugin and the authoring
assistant can drive it over Chrome DevTools Protocol through a per-device TCP
tunnel: open a page, send raw CDP commands, capture a screenshot as multimodal
input.

---

## 11. Mesh, exit nodes and IP rotation

A WireGuard peer-to-peer mesh between a user's devices, coordinated by the
server:

- **Device registry and signaling** — peers find each other without a public
  rendezvous of their own.
- **Exit nodes** — a device can route through another device or a designated
  exit node.
- **Kill switch** — traffic does not leak when the mesh drops.
- **IP rotation** — exit-node rotation is a supported flow, coordinated between
  the server, the device and the administrator.
- **UDP relay** — a fallback relay for NAT hole-punching when direct
  connectivity fails.

---

## 12. Update channels

Clients self-update. There are two channels and both are real, signed artifact
paths — not "download the ZIP from a web page".

| Channel | Who it is for |
|---|---|
| **Dev** | Maintainers and testers. Rolling artifacts. |
| **Release** | Everyone else. |

- Artifacts are uploaded to a relay, manifests are published, and clients compare
  their installed revision to the manifest before offering an update.
- The server can **push** an update to a specific device (app update, browser
  update), which opens the platform's install UI pre-confirmed — or, on a headless
  service, installs unattended and verifies the checksum before swapping.
- A deployment can gate the dev channel to an allowlist of accounts.

---

## 13. Billing and credits

Every account has a credit balance. Chat and plugin-AI turns cost credits.

| | |
|---|---|
| **Pricing** | Per model, stored in the database and editable by an administrator. Seeded at the upstream's own rate — the only default that is not an invented number. |
| **Metering** | Token usage is recorded per request with the model actually sent upstream. |
| **Enforcement** | A caller with no balance cannot start a paid turn; the failure message is branded and explicit. |
| **Top-up** | Optional crypto top-up rails. |
| **Signup bonus** | Configurable credit granted on account creation. |
| **Usage API** | Per-account usage and cost history. |

---

## 14. Admin and public API

**Admin console** (`/admin`) is a single-page application: dashboard, users,
devices, plugins, AI providers and usage, settings, logs, updates, mesh, remote
desktop tests, and the fork/export tooling.

**Public API** is the same server, authenticated by access key or admin key. The
shipped OpenAPI description is the contract. Two shapes that cost people time:

- The device list is `GET /admin/devices/all`. There is no bare `/admin/devices`.
- A tool call puts the tool name under `name` inside `data`, not `tool`.

---

## 15. Observability

- **Proxy request log** — every proxied chat request, with duration and result.
- **Usage log** — per-request token and cost accounting.
- **Client error reporting** — clients report errors to the server.
- **Device logs** — device-side logs are collected and viewable separately from
  server logs.
- **Crash logging** — uncaught exceptions and unhandled rejections are recorded
  before exit.
- **Live requests** — an admin view of in-flight proxy traffic.
- **Readiness** — `GET /admin/readiness` machine-checks the deployment and names
  the fix for anything missing. `/setup` is the human-facing version of the same
  list.

---

## 16. Branding, forking and export

A fork is a branded build of the shared core.

**What you provide** — validated before anything is generated, because a value
that is not a legal application id, hostname or port is better refused in a form
than discovered at build time:

- product name, domain
- Android application id (reverse-DNS, permanent)
- FRP host (literal IP) and FRP port (1024–65535)
- contact and support email
- repository URLs for the platforms you will build
- platforms, headless preference, accent colour, logo

**What is derived and renamed at runtime:**

| Platform | Renamed |
|---|---|
| Android | `applicationId`, provider authority, action prefix, deep-link scheme, FRP host/port, tunnel prefix, server URL |
| macOS | app bundle id, service bundle id |
| Windows | product / install dir, service name and named pipe, installer AppId, MSI upgrade code, FRP host/port, tunnel prefix |
| Linux | service name, install dir, FRP host/port, tunnel prefix |
| Server | brand identity, per-product tunnel prefix, deep-link scheme, compose project, database name |

**What is deliberately NOT renamed:** source namespaces and wire identity.
Rewriting those would break every patch, tool and document that refers to the
core, and no user ever sees them.

**What an export never contains.** Secrets are refused by a content scan over
the whole staged tree — API keys of any provider's shape, private keys, cloud
credentials, GitHub tokens, and filled `.env` files. A fork operator generates
their own secrets and supplies their own AI keys. **A fork never inherits the
upstream project's credentials.**

---

## 17. First-run setup

A new deployment comes up complete where it can, and asks for what only you can
supply.

**Seeded automatically:** the database schema, every system prompt (chat
prompts per platform, voice prompts and TTS instructions, speech-to-text
context, and the captcha answer contracts), and the chat model catalogue. All of
it is first-boot only and never overwrites an administrator's edit.

**You supply:** host and DNS, generated secrets, the first user, OAuth apps if
you want social sign-in, your own OonaCode key for chat, and an OpenAI key if you
want voice.

`/setup` is the checklist that ties these together: blockers first, then
optional capabilities that are off, then what is configured — each with a link to
the screen that fixes it. It never shows a secret and never accepts one.

---

## 18. Security and secret boundaries

| Rule | Why |
|---|---|
| **Secrets live in the deployment environment or the admin store — never in the repository.** | A repository is copied, forked and published. |
| **The chat key is environment-only.** | It would otherwise appear in database dumps and in fork exports. |
| **A fork export is content-scanned and refuses secret-shaped material anywhere in the tree.** | The `.env` deny-list alone misses a key pasted into a script or a generated profile. |
| **The export only packages files git tracks.** | Untracked local litter cannot ride along. |
| **Device revocation has several independent enforcement paths.** | Any single check can be missed. |
| **Plugins run in a structurally enforced sandbox.** | A convention is not a boundary. |
| **The public documentation set is an explicit allowlist.** | Serving a directory publishes everything anyone drops into it. |
| **Admin identity is not a user row.** | Compromising a user account does not grant administration. |

---

## 19. System prompts

The wording the model speaks with is **data, not code**, so it can be tuned
per deployment without a release.

| Key group | Purpose |
|---|---|
| Platform chat prompts (Windows, Android, macOS, Linux) | Injected into proxied chat requests for that platform |
| Plugin authoring prompt | Governs the plugin-building assistant |
| Voice prompts and TTS instructions | Voice turn behaviour and speech style |
| Speech-to-text context | Improves transcription for the deployment's domain |
| Captcha answer contracts | Exact output formats for grid, canvas-drag and canvas-click tasks |

All are seeded on first boot from source and editable in the admin panel. The
product name inside a prompt follows the deployment's brand; the text is yours.

---

## 20. Repository ownership

| Repository | Owns |
|---|---|
| **server** | API, admin console, chat proxy, tunnel relay, plugins, billing, export tooling. **Canonical docs for cross-platform topics.** |
| **windows** | Windows client, its tool catalog, its control surfaces. |
| **android** | Android client, accessibility and ADB behaviour, mesh policy. |
| **macos** | macOS client and AppKit specifics. |
| **linux** | Linux client, capture/input/docking backends, self-update behaviour. |
| **plugin-runtime** | The plugin engine and its schema — the schema is canonical here and mirrored to the server. |
| **netstack** | Networking components shared by clients. |
| **browser** | The companion browser, and the browser variant of the plugin runtime. |

Platform repositories defer to the server repository for protocol shape and
server-side behaviour. Platform tool catalogs are canonical in the platform
repository, because only the platform knows its own surface.

---

## 21. Where to go next

- **Standing up a deployment** — `standing-up-a-fork.md`
- **Public API reference** — `user-api.md`
- **Plugin authoring assistant prompt** — `plugin-author-prompt.md`
- **Full internal documentation index** — `SYSTEM-MAP.md`, available
  authenticated at the admin docs surface