🕶️ Mentra Live
A voice companion with memory on your glasses. Talk to your Companion through Mentra smart glasses. It recalls what you told it before from Companion Memory, answers with your Hub’s local model, and speaks the reply back.
The site has the setup guide, a builder for the bridge’s settings file, and a demo of the companion UI with canned answers.
Open-source repository
CI-Mentra is open source under the PolyForm Noncommercial 1.0.0 license:
- Source code:
github.com/companionintelligence/CI-Mentra - Container registry:
ghcr.io/companionintelligence/ci-mentra:latest - Marketplace manifest: App ID
ci-mentra
Install & availability
Available for developers (developer preview). Install from the Companion Hub App Store (developer catalog) or run locally via Deno or Docker. Connects your smart glasses to your Hub’s local inference engine and Companion Memory.
| Hub App Store | Search Mentra Live (ci-mentra) → Install |
| Bridge port | 7010 inside the container (host port 17010 on Hub; Cloud SDK WebSocket & webhook) |
| Web UI & API port | 3000 (Companion web UI & OpenAI-compatible /v1 facade) |
| Minimum Hub version | 0.2.27 |
| Runtime | Deno 2.9+ (preferred, used by container) or Node.js 20+ |
| Architecture | linux/amd64 and linux/arm64 (multi-arch) |
SDK and ecosystem status: Mentra’s developer platform is transitioning
from the Cloud SDK (used in the MentraOS
Legacy app) to phone-side Miniapps in
MentraOS 3.0+ (see Mentra’s roadmap
update ).
CI-Mentra implements the Cloud SDK WebSocket server today, with a direct
mobile-bridge mode in development for the Miniapps SDK. While the Mentra 3.0
developer console at console.mentra.glass displays a “coming soon”
placeholder, new Cloud SDK app registrations cannot be completed there.
Overview
Companion for Mentra (CI-Mentra) is a MentraOS Cloud SDK app, a bridge that runs on your Companion Hub or on a server you control. You speak to the glasses, MentraOS sends the audio to the bridge, and the bridge turns it into text, asks your local model for an answer, and speaks the answer back through the glasses. With Companion Memory turned on, each exchange is saved to your own Companion Memory and relevant memories are recalled into the next answer.
Architecture and subsystems
CI-Mentra functions as a Device-layer bridge between wearable hardware and the Companion Intelligence ecosystem. It translates streaming audio from smart glasses into text, performs contextual memory retrieval, runs local model inference, and streams synthesized speech back to the wearer.
flowchart TD
subgraph WEARABLE["Mentra Smart Glasses"]
GLASSES["Mentra Live Glasses<br/>(Microphone · Camera · LED)"]
APP["MentraOS Phone Client<br/>(Legacy / Miniapps Bridge)"]
GLASSES <-->|"BLE Opus Audio"| APP
end
subgraph BRIDGE["CI-Mentra Bridge (Deno 2.9+ / Node 20+)"]
WS["Cloud SDK WebSocket<br/>(Port 7010)"]
FACADE["OpenAI-Compatible Facade<br/>(/v1/chat/completions)"]
AUTH["Portal OIDC Auth<br/>(JWKS Bearer + PKCE Flow)"]
subgraph PIPELINE["Voice Processing Pipeline"]
VAD["Voice Activity Detector (VAD)"]
STT["Speech-to-Text Adapter<br/>(Whisper / OpenAI STT)"]
CTX["Memory Context Injector"]
TTS["Text-to-Speech Engine<br/>(Local Piper / Mentra TTS)"]
end
end
subgraph HUB["Companion Hub & Core Services"]
PORTAL["CI-Portal (Identity Provider)<br/>EdDSA JWKS / Token Verification"]
MEM["Companion Memory (CI-Server)<br/>GraphQL Context Search"]
LLM["Inference Engine<br/>Local Ollama · Hermes · OpenClaw"]
end
APP <-->|"HTTPS / WSS"| WS
WS --> VAD
VAD --> STT
STT --> CTX
CTX <-->|"hybrid recall"| MEM
CTX -->|"prompt + memory context"| LLM
LLM -->|"response text"| TTS
TTS -->|"synthesized audio"| WS
FACADE --> AUTH
AUTH -->|"verify credentials"| PORTALDual ESM runtime (Deno + Node.js)
CI-Mentra is authored as ECMAScript Modules ("type": "module", module: NodeNext, verbatimModuleSyntax):
- Relative imports carry explicit
.tsextensions, and type-only imports are annotated. - Runs natively under Deno 2.9+ without bundling or transpile steps (
deno run -A --watch src/index.ts). Deno resolves npm dependencies automatically viadeno.json. - Runs under Node.js 22.6+ via type-stripping (
node --experimental-strip-types src/index.ts) or compiled todist/viatscon Node.js 20+.
Dual OIDC pattern via CI-Portal
CI-Mentra connects to CI-Portal (better-auth, EdDSA) supporting two authentication patterns:
- Resource server mode (
PORTAL_OIDC_ENABLED=true): Validates presented JWT bearer tokens against Portal’s JWKS (/api/auth/jwks), verifyingiss,aud, andexpclaims. - Interactive PKCE login (
MENTRA_OIDC_ENABLED=true): Provides a browser-based Authorization Code flow with PKCE under/api/auth/loginand/api/auth/callback. Tokens remain on the server; the browser receives an encrypted, signedmentra_sessioncookie.
Standardized inference and Hub bootstrap
CI-Mentra routes inference requests across multiple backends:
- Ollama resolution: Resolves endpoints in order:
OLLAMA_BASE_URL→OLLAMA_HOST→LLM_API_BASE→http://localhost:11434. In Hub environments,http://host.docker.internal:11434is injected automatically. - Hermes / OpenClaw auto-discovery: When
HUB_INFERENCE_BOOTSTRAP=true, the service queriesGET ${hub}/api/inference/apps/ci-mentra/bootstrap.envat boot. Hub’sAppContainerOriginGuardverifies the container identity and hands out configured model endpoints and credentials. - OpenAI-compatible facade: Serves
/v1/modelsand/v1/chat/completions, allowing external voice clients and terminal tools to use CI-Mentra as an inference proxy.
The bridge is a voice assistant. It does not record continuous video, index frames, or run a Bluetooth pairing screen in the Hub. Photos are taken only when you ask for one by voice, and only when you have turned that setting on.
Key features
- Voice loop — the bridge detects when you stop speaking, transcribes the utterance with a speech-to-text endpoint you provide, gets an answer from your model, and speaks it. A display, if the glasses have one, shows what you said and the answer.
- Companion Memory — recalls relevant memories into each answer and saves each exchange. It is off until you turn on Remember our conversations in the app’s settings.
- Voice tools — “remember this”, “search my memory”, “what’s on my calendar”, and “take a photo”. See Voice tools.
- Privacy light — on glasses with an LED, the bridge lights it red while the microphone is capturing your speech.
- Companion UI — a phone and R1 ring web app at the bridge’s address for talking, searching memory, and viewing your profile.
- OpenAI-compatible API — the bridge serves
/v1for other clients, protected by an API key.
What runs where
flowchart TD
G["Mentra Live glasses"] -->|"audio"| M["MentraOS Cloud"]
M -->|"HTTPS to your Public URL<br/>(port 7010)"| B["Companion for Mentra bridge<br/>on your Hub or server"]
B -->|"speech-to-text"| S["OpenAI-compatible<br/>STT endpoint<br/>(you provide)"]
B -->|"chat"| L["Your Hub's local model"]
B -->|"recall · save"| CM["Companion Memory"]
B -->|"spoken reply"| M
M --> GAudio passes through MentraOS. The glasses talk to MentraOS Cloud, and MentraOS reaches the bridge over the internet. Answers are generated on your Hub, and your conversations and saved memories are stored only in your own Companion Memory, but the audio and transcripts do travel through Mentra’s service on the way. The bridge speaks replies with the MentraOS text-to-speech voice.
Supported hardware
| Device | Status |
|---|---|
| Mentra Live | The target device. It has a microphone, camera, and LED. |
| Other MentraOS glasses | The bridge checks each device’s capabilities. It skips the display when there is none, ignores audio when there is no microphone, and refuses photos when there is no camera. Only Mentra Live is the documented target. |
You need the MentraOS phone app paired with your glasses, and an app registered in the MentraOS developer console. Because the bridge is a Cloud SDK app, the phone app must be the MentraOS Legacy build (see the status notes above).
Setup
These steps are the Cloud SDK flow. They need the MentraOS Legacy build and a console where you can register an app, which MentraOS 3.0 doesn’t offer yet.
You need:
- A Companion Hub, or a server that can run the bridge with Docker
- A Companion Memory instance, if you want memory
- A speech-to-text server that speaks the OpenAI
/v1/audio/transcriptionsAPI, such as a local Whisper server. The bridge doesn’t include one. - A way to give the bridge a public HTTPS address, such as Tailscale Funnel or Cloudflare Tunnel
- Mentra Live glasses paired with the MentraOS phone app
Register the app in MentraOS
In the MentraOS developer console, create an app. Copy its package name (for example com.yourname.companion) and its API key. You enter both in the next steps. (On 3 October 2026, console.mentra.glass shows a “coming soon” page, so you can’t complete this step there yet.)
Grant the app these permissions in the console. The app’s manifest can’t declare them for you.
| Permission | Needed for |
|---|---|
| Microphone | Required. The voice loop doesn’t work without it. |
| Camera | Optional. Only for the “take a photo” tool. |
| Calendar | Optional. Only for the “what’s on my calendar” tool. |
| Notifications, Location | Optional. Reserved for later features. |
Every optional permission degrades quietly: without it, the voice loop keeps working.
Install the bridge from the Hub app store
In Companion Hub, install Companion for Mentra and fill in the form:
- MentraOS package name and MentraOS API key — the values from the previous step.
- Sign in with Companion Intelligence — on by default. The companion UI needs it.
- Companion Memory URL and Companion Memory API key — leave both blank to connect to your Companion Memory app on the same Hub. Fill them in only for an external, self-managed Companion Memory server; use a memory-scoped key.
At install, the Hub gives the bridge your local model endpoint and, through the Companion Memory connection, a memory-only key. The bridge publishes two things on the Hub: host port 17010 for the MentraOS AppServer, and the app URL for the companion UI. Open the companion UI from the Local access entry on the app’s page in the Hub dashboard. See Finding your app’s URL.
The install form has no field for speech-to-text, and the Hub doesn’t provide one. Point the bridge at your speech server with OPENAI_BASE_URL and OPENAI_API_KEY (or VLLM_BASE_URL for a server that needs no key). Without one, the bridge can’t transcribe what you say, and the glasses answer with an error. See Provide speech-to-text.
Give MentraOS a public address for the bridge
MentraOS calls the bridge over HTTPS, so it must be able to reach host port 17010 (container port 7010) from the internet. Expose that port through a tunnel, then set the tunnel’s HTTPS address as the app’s Public URL in the MentraOS developer console.
Start the app on your glasses
Install and start your app from the MentraOS phone app, then speak to the glasses. If the glasses have a display, it shows a status such as Listening..., Transcribing..., Thinking..., or Speaking..., followed by what you said and the answer.
Turn on memory
In the app’s settings in MentraOS, turn on Remember our conversations. It is off by default, and while it is off the bridge neither recalls nor stores anything from your conversations.
To check the bridge, open <bridge address>/health. It reports the active sign-in mode and whether memory is enabled.
Provide speech-to-text
The bridge sends each utterance to /v1/audio/transcriptions on an OpenAI-compatible server, using the model name whisper-1. It tries the server at OPENAI_BASE_URL (with OPENAI_API_KEY) and then the one at VLLM_BASE_URL. Ollama and the Hub’s model endpoint don’t serve transcription. For a local setup, run a Whisper-compatible server and set OPENAI_BASE_URL and VLLM_BASE_URL to its address. The Language you speak setting is passed along as the transcription language.
Run the bridge without the Hub
If you have access to the bridge’s source repository, you can run it without the Hub. It runs on Deno 2.9 or newer, or Node.js 20 or newer (npm run dev needs Node.js 22.6 or newer). For a standalone deployment, docker compose up -d --build starts the bridge with Postgres and Redis. Copy .env.example to .env first and set at least MENTRA_PACKAGE_NAME, MENTRA_API_KEY, and a model endpoint. To enable memory, also set MENTRA_MEMORY_ENABLED=true, CI_SERVER_URL (the address of your Companion Memory), and a memory-scoped CI_SERVER_MEMORY_KEY. The repository’s README lists every variable.
Settings
MentraOS shows these settings for the app:
| Setting | Default | What it does |
|---|---|---|
| Language you speak | Detect automatically | Sets the language used to transcribe your voice: English, Spanish, French, German, Portuguese, or Japanese. |
| Voice ID | Blank | Chooses a voice for spoken replies. Blank uses the default voice. |
| Speak answers out loud | On | When off, answers appear only on the display. |
| Remember our conversations | Off | Turns on Companion Memory recall and saving. |
| Allow “take a photo” to capture and save to memory | Off | Lets the photo tool use the camera. |
Voice tools
MentraOS matches what you say against these phrases and calls the bridge.
| Tool | Say something like | What happens |
|---|---|---|
| Remember this | ”Note this”, “make a note”, “don’t let me forget”, “save this to memory” | Saves a fact or note to Companion Memory. |
| Search memory | ”Search my memory”, “what do you know about”, “what did I say about”, “do you remember” | Searches Companion Memory and speaks what is relevant. |
| Calendar | ”What’s on my calendar”, “what’s my next meeting” | Reads the next events MentraOS has synced. Needs the Calendar permission. |
| Take a photo | ”Take a photo”, “capture this”, “remember what I’m looking at” | Takes a photo, saves it to the glasses’ gallery, and saves a memory note about when it was taken. |
If memory is off, “remember” and “search” tell you so and name the setting to turn on. The photo tool saves a description of the capture, such as when it happened and any note you gave, not what is in the picture, because the bridge has no vision model. The image stays in the glasses’ gallery. Photo capture also needs Remember our conversations to be on.
The companion UI
Open the bridge’s address (the app URL in Hub) on your phone or R1 ring. It shows Sign in until you sign in with Companion Intelligence, which doesn’t work on a Hub install yet (see Troubleshooting). Once you are signed in, it has three tabs:
- Talk — hold the mic button to speak and release to send, or type a message. Chips under an answer show which memories it used. Voice on toggles spoken replies, and New topic clears the short-term conversation.
- Memory — search your Companion Memory.
- Profile — the profile context the bridge gives the model.
On an R1 ring, the arrow keys move focus, a tap activates the focused control, holding the button is push-to-talk, and Escape goes back.
Troubleshooting
The app never starts on the glasses, or MentraOS says it can’t reach the app
MentraOS must reach the bridge over HTTPS. Check that the tunnel is up, that it points at host port 17010 (container port 7010), and that the same address is the Public URL in the developer console. Confirm the package name and API key in Hub match the console.
The glasses hear you but nothing is spoken back
Check Speak answers out loud, and open <bridge address>/health to confirm the bridge is running. If the display shows Voice pipeline error, the transcription or model call failed. The most common cause is that no speech-to-text endpoint is configured: see Provide speech-to-text. Also check that your Hub’s model is running.
It answers, but doesn’t remember anything
Turn on Remember our conversations in the app’s settings in MentraOS. The bridge also needs a Companion Memory address and key. /health shows memory.enabled. If Companion Memory is unreachable, the assistant still answers but without memory.
“Sign in” fails in the companion UI
The bridge asks Companion Portal for a ci-mentra sign-in client, and Portal doesn’t register one yet, so the Sign in step can’t finish on a Hub install. The voice loop on the glasses doesn’t use it, so spoken questions and answers still work.
“Take a photo” says it’s turned off Turn on Allow “take a photo” and Remember our conversations, and grant the Camera permission in the developer console.