How your personal AI runs
Your personal AI is two products working together on your own hardware. Companion Hub runs the containers and the AI models. Companion Memory is an app on the Hub that keeps your photos, messages, notes and calendar, answers your questions about them, and works through them in the background so the answers stay current.
This page starts with the big picture and gets more detailed as it goes. The numbers come from the Companion Memory and Companion Hub code. Each part carries one of the four status labels used across the Companion Memory pages, as defined on Personas:
- Works today: the capability is on by default.
- Behind a setting: the capability is in the code, but it is off until an owner or operator turns it on.
- Not yet: the capability is not built, or it is built but not connected to the product.
- Planned: the design is documented, and the code does not exist yet.
The page describes Companion Memory main and Companion Hub dev as of 12 November 2026. The latest releases, Memory v2026.11.11 and Hub v0.3.11, predate some of it. On Memory v2026.11.11 the keyword labeler is still on by default, tasks get no activity type, and the scoring endpoints described under Training are missing. Hub v0.3.11 has none of the Hub training code.
- Containers in the Hub stack
- 6
- Two of them, the Cloudflare tunnel and Tailscale, start only when you turn those features on.
- from what runs
- Scheduled jobs in Memory
- 62
- 17 run in the worker container; 12 are switched off by default.
- from background jobs
- Models Memory starts with
- 2
- A chat model and an embedding model. On a Hub, the Hub picks the chat model for your hardware.
- from models
The short version
- The Hub is the part that talks to Docker. It installs Memory as a set of containers, gives Memory the address of its AI endpoint, and restarts Memory when you change AI settings.
- The AI models run in an inference engine on the same machine, usually Ollama. Memory never loads a language model itself, apart from one small fallback labeling model described below. Your forecasting model is small and runs inside
hbpe-service. For everything else, Memory sends requests to the Hub, and the Hub decides which engine, and which paired Hub, answers. - Memory does its work in queues. Your chat turn has its own queue and goes first. Photo descriptions, summaries, embeddings and training wait their turn, and the heaviest cleanup work runs in a night window.
- Your data, the search index and the job queues all live in Memory’s Postgres database on your hardware.
This page describes Memory installed from the Hub app store, which is the usual setup. Memory can also run on its own with its own compose files. It then talks to Ollama on the host directly, and the Hub sections do not apply.
What runs on your hardware
The Hub runs a small stack of its own: a database, a message queue for long-running work, the Hub itself, and Traefik, which receives web traffic and passes it to the right app. Each app you install, Memory included, runs as its own group of containers next to that stack. The AI engines run on the host machine, and the Hub reaches them over the host’s network.
flowchart TB
accTitle: How Companion Hub, Companion Memory and the AI engine talk to each other
you(("You<br/>phone or browser"))
subgraph hub["Companion Hub"]
traefik["Traefik<br/>web entry"]
cihub["ci-hub<br/>apps and AI routing"]
hubdb[("Hub database")]
mq[["RabbitMQ<br/>Hub job queue"]]
end
engines["Ollama on the host<br/>chat, vision, embeddings"]
subgraph mem["Companion Memory"]
gw["gateway"]
api["api"]
worker["worker"]
db[("Postgres + pgvector<br/>your data, vectors, job queue")]
side["hbpe-service<br/>summary-service"]
end
you e1@--> traefik
traefik e2@--> gw
gw e3@--> api
api e4@--> cihub
cihub e5@--> engines
cihub --- hubdb
cihub --- mq
cihub -.->|"docker compose"| mem
api --> db
worker --> db
api --> side
side --> cihub
worker --> cihub
e1@{ animate: true }
e2@{ animate: true }
e3@{ animate: true }
e4@{ animate: true }
e5@{ animate: true }The moving edges are the path a chat question takes. The dotted edge is the Hub creating Memory’s containers through Docker. Memory’s sidecars and worker send their own model requests to the same Hub endpoint as the API.
| Container | What it does | AI models it uses | Status |
|---|---|---|---|
ci-hub | The Hub API and dashboard. Installs, starts and updates apps through Docker, and routes AI requests to an engine or a paired Hub. | None of its own. It forwards requests to engines. | Works today |
ci-hub-db | The Hub’s Postgres 14 database: settings, installed apps, job state. | None | Works today |
ci-hub-queue | RabbitMQ. Carries long-running Hub work such as installs, updates and catalog syncs. | None | Works today |
traefik | Receives web traffic and sends it to the Hub or to an app, after the Hub checks who you are. | None | Works today |
cloudflared | Optional. Public access through a Cloudflare tunnel. | None | Behind a setting |
hub-tailscale | Optional. Private access over Tailscale. | None | Behind a setting |
| Ollama on the host | Runs the models. The Hub can also use vLLM, Lemonade, oMLX or LocalAI if you run them. | Chat, vision and embedding models | Works today |
gateway (Memory) | Memory’s single entry point, an nginx server. Sends /api to the API, sync traffic to PowerSync and file reads to the object store. | None | Works today |
api (Memory) | Answers the apps, runs chat and search, and runs 45 of the 62 scheduled jobs. | Chat, vision and embedding models, through the Hub | Works today |
worker (Memory) | The same code in a second container. Runs imports and 17 maintenance jobs so a large import does not slow down your requests. | Embedding and background models, through the Hub | Works today |
database (Memory) | Postgres 18 with extensions for vector search (pgvector), time series (TimescaleDB) and places (PostGIS). Holds your data, the search vectors and Memory’s job queue. | None | Works today |
object-store (Memory) | RustFS. Holds your original files, such as photos and documents. | None | Works today |
summary-service (Memory) | Writes hourly, daily, weekly and monthly summaries, insights and suggested actions. Also extracts text from documents with MarkItDown. | The chat model and nomic-embed-text, through the Hub | Works today |
hbpe-service (Memory) | Learns your daily rhythm: labels activity, trains a small forecasting model per person, and serves forecasts. | Your own forecasting model, and the chat model or a small local fallback model for labels. A keyword labeler is off by default. | Works today |
powersync (Memory) | Keeps the data that apps hold locally in sync with the server. | None | Works today |
| Setup containers (Memory) | Five containers run once at install or start: they create secrets, fix data-folder permissions, prepare and migrate the database, and set up sync, then exit. | None | Works today |
Memory’s API and worker are one program started twice. The API container serves your requests, and the worker container takes the import and maintenance queues. A long import parser then never shares a process with the request you are waiting on.
Which AI models do what
Memory asks for a model by job, and the Hub decides what serves it. When Memory runs from the Hub, the Hub writes Memory’s model settings into its environment: the endpoint address, the chat model, the background model, the embedding model and the context size. By default the endpoint is the Hub’s pool proxy, so a second Hub can serve requests later without Memory changing anything. You can turn that routing off, and when no local engine is ready the Hub gives Memory a cloud provider’s address instead.
| Model | Task | Where it runs | When it loads | Status |
|---|---|---|---|---|
Chat model, gemma4:e4b unless you or the Hub pick another | Chat answers, agent tools, summaries, planning scans, answer drafts | The engine the Hub picks, on this Hub or a paired one | On the first request after it was unloaded | Works today |
Background model (set as LLM_UTILITY_MODEL) | Extraction and classification in the background | Same as chat | On a Hub, a smaller model you already have, if one fits next to the chat model; otherwise the chat model | Works today |
| Vision (the chat model, when it can read images) | Describes each new photo | Same as chat. Photos are sent at 1280 px on the long edge. | When a photo’s preview is ready and no chat turn is waiting | Works today |
nomic-embed-text v1.5 | Turns text into 768 numbers so search can find things by meaning | Ollama, or the chat engine when it serves this model | When there is text to embed | Works today |
| Your forecasting model | Forecasts your day in 96 quarter-hour slots | hbpe-service, on the CPU unless a GPU is passed through | Trained once you have enough history, then updated nightly | Works today |
| Keyword labeler | Picks the broad activity class (work, exercise and so on) from keyword matches, so the chat model is asked only for the finer label. Other pages call it the keyword fast path or the keyword shortcut. | hbpe-service | Off by default. If you turn it on with HBPE_ENABLE_DECISION_MODELS, a label scoring at least 0.75 is used as is. | Behind a setting |
| Qwen3.5-0.8B (a model file in the GGUF format) | Labels activity when no chat endpoint is configured | Inside hbpe-service | Downloaded from Hugging Face the first time it is needed. It checks for an endpoint every 300 s and switches back when one appears. | Works today |
Memory runs no speech-to-text or OCR model. Document text comes from MarkItDown, which reads the file format and does not use a model.
How the Hub picks a chat model for your hardware
The Hub sorts your machine into a tier and recommends models from its catalog that fit that tier. Until you choose a chat model in Settings → AI, apps get the highest-ranked recommended model that is already downloaded. A model may use only part of your memory, so the operating system and your apps keep room to work.
| Tier | Hardware | Share of memory a model may use |
|---|---|---|
| High | A GPU, or unified memory the GPU can use, of at least 16 GB | 90% of VRAM on a separate GPU; 70% of RAM with unified memory |
| Medium | A GPU, or unified memory the GPU can use, of at least 8 GB | Same as above |
| Low | A GPU, or unified memory the GPU can use, of at least 4 GB | Same as above |
| CPU only | No usable GPU and at least 16 GB of RAM | 70% of system RAM |
| Insufficient | No usable GPU and less than 16 GB of RAM | None recommended |
On an Apple Silicon Mac the GPU can use all of RAM, so a 16 GB Mac is High. If the Hub finds unified memory but no usable GPU runtime, it falls back to 32, 16 and 8 GB of RAM for High, Medium and Low.
Two walkthroughs
Asking a question in chat
- You askThe Memory app sends your questionYour phone
- GatewayTraefik and Memory's gateway pass it to the APIHub, Memory
- Chat queueOne turn at a time; background model work stands asideMemory
- RecallLooks up memories that match, for up to 3 sMemory
- Pool proxyPicks the engine with the shortest queueHub
- ModelLoads the chat model if needed, then answersEngine
- Stream backWords appear as they are written; the turn is savedYour phone
The chat queue takes one turn at a time and nothing else. If the search for matching memories takes longer than 3 seconds, the turn goes ahead without it, and the model can still search your memories with its own tools while it answers.
A new photo arriving
- UploadThe app sends the photoYour phone
- StoreThe original goes to the object storeMemory
- PreviewA preview job makes a smaller copyMemory
- DescribeWaits for chat, then the vision model writes a title and descriptionMemory, Hub
- EmbedTitle, description and file name become a 768-number vectorMemory, Hub
- SearchableSearch finds it by what is in the pictureMemory
A new file is added to Memory’s search index as soon as it is created, so you can find it by name straight away. Finding it by what the picture shows has to wait for the description and the embedding. Memory checks for due pipeline work every 5 minutes. After an import, it starts the embedding pass directly instead of waiting for the hourly slot.
What runs in the background, and when
Memory’s scheduled jobs are timers inside the API and worker containers. A timer does not do the work itself. It puts jobs on a queue in Memory’s Postgres database, one per person where that applies, and the queue workers pick them up. The Hub uses RabbitMQ for its own long-running work, and it schedules its own repeating jobs, such as app status and catalog syncs, through RabbitMQ too.
Memory’s containers set no time zone, so its schedules run in UTC unless you set TZ for the app.
| Job | Runs on | Schedule | Runs a day |
|---|---|---|---|
| Due reminders | Memory | Every minute | 1,440 runs |
| Pipeline tick | Memory | Every 5 minutes | 288 runs |
| Inbox scan | Memory | Every 30 minutes | 48 runs |
| Hourly summary | Memory | Every 60 minutes (at :05) | 24 runs, at :05 |
| Insights | Memory | At 00:30, 03:30, 06:30, 09:30, 12:30, 15:30, 18:30, 21:30 | 8 runs |
| Todo processing | Memory | At 00:00, 06:00, 12:00, 18:00 | 4 runs |
| Daily summary | Memory | At 00:10 | 1 run |
| HBPE nightly training | Memory | At 02:30 | 1 run |
| HBPE bootstrap sweep | Memory | At 03:00 | 1 run |
| Image description sweep | Memory | At 03:20 | 1 run |
| Todo scan | Memory | At 06:00 | 1 run |
| App status sync | Hub | Every 5 minutes | 288 runs |
| Portal check-in | Hub | Every 15 minutes | 96 runs |
| App catalog sync | Hub | Every 15 minutes (production) | 96 runs, production |
| Orphan network cleanup | Hub | Every 30 minutes | 48 runs |
| Job | When | What it does | Runs in | Status |
|---|---|---|---|---|
| Connector sync | Pending syncs every 30 s; new sync jobs every minute | Pulls new data from the cloud accounts you connected | Memory API | Works today |
| Answer drafts | Checked every minute; each person at most once every 20 minutes | Drafts answers to open questions from your own data, on the background model queue | Memory API | Works today |
| Enrichment window | Opens 01:00 UTC for 5 hours, with at most 90 minutes of work a night | Catch-up previews, place lookups, link previews and duplicate checks | Memory API | Works today |
| Weekly and monthly summary | Mondays 00:15 UTC; the 1st of the month at 00:20 UTC | Longer summaries from the daily ones | Memory summary-service | Works today |
| Retrieval checks | Daily at 03:40 UTC; a recall probe on Sundays at 05:20 UTC | Checks that search still finds what it should | Memory worker | Works today |
| Smart rotation | 10 alternative schedules | Would replace the feed, answer, insight and summary timers with a rotation across people. Off by default. | Memory API | Behind a setting |
| Host telemetry | Every 5 s | Records CPU, memory, disk and Docker state, kept for 48 hours | Hub | Works today |
| Memory key rotation | 60 s after start, then every 12 hours | Replaces Memory connection keys older than 60 days | Hub | Works today |
| Hub update check | Every 24 hours, starting 24 hours after the Hub starts (production) | Updates the Hub image when automatic updates are on, which is the default | Hub | Works today |
| Backup retention | Every 7 days, if the Hub stays up that long | Trims each app’s backups to its limit, or to the newest 5 when no limit is set | Hub | Works today |
For the full list of Memory’s queues and jobs, see AI training, inference & jobs. For every Hub timer, see How Hub runs apps and models.
How it stays out of your way
Chat goes first
Memory keeps one queue for chat turns and nothing else. Background model work goes on other queues: thread titles, answer drafts, photo descriptions, object recognition in photos and browsing insights. Those queues share a single lane, so at most one background request from the API reaches the model at a time. They also stop taking new jobs while any chat turn is waiting or running.
The activity labeler in hbpe-service follows the same rule. It sends at most 6 label requests a minute and waits up to 600 seconds for a chat turn to finish before it sends the next one.
Heavy work waits, and queues have limits
Embedding runs at any hour, because search depends on it. A new upload gets its preview right away. The catch-up sweeps for previews, plus place lookups, link previews and duplicate checks only start inside the enrichment window, 01:00 to 06:00 UTC by default, and spend at most 90 minutes of work a night. When several people share one Memory, each gets a 20-minute turn, starting with whoever was served longest ago.
Memory runs at most 10 jobs at once per container and 6 from any one queue. The embedding queue holds at most 25 ready jobs, because the model process it uses also serves chat, the activity labeler and the summary sidecar. A job that fails is retried 3 times with growing delays and then set aside on a failed-jobs list (a dead-letter queue), where you can replay it.
Models load when needed and make room
When the Hub loads a model, it first checks that the model fits a memory budget. The budget keeps 2048 MB of system RAM and 512 MB of graphics memory free for the machine. If the model does not fit, the Hub unloads idle models that it loaded itself and that you have not pinned, least recently used first, and waits up to 10 seconds for the memory to free. Loads happen one at a time, and so do downloads.
A model the Hub loads stays loaded until the Hub unloads it or needs the room. Pinning a model keeps it there, and pins survive a restart. When you unpin, the model goes back to Ollama’s idle timeout, which is 5 minutes unless you changed it.
If an engine keeps failing for one model, the Hub stops sending it that model for 60 seconds, doubling up to 15 minutes while the failures continue. The Hub also sets each model’s context window, starting at 65,536 tokens and stepping down to as little as 4,096 until the model fits.
A second Hub can take some of the load
By default, Memory sends its model requests to the Hub’s pool proxy. With one Hub, the proxy serves everything locally. Once you pair a second Hub with Hub Pool, the proxy ranks every engine that has the requested model, on both Hubs, by how many requests each is already handling. Your own Hub gets a head start of one request. If an engine fails before it starts answering, the proxy tries the next one. Each Hub checks its paired Hubs every 30 seconds by default.
The Hub can also measure how busy its GPU is, but by default it does not use that to route requests.
Settings changes restart Memory once
When you change the AI engine, the chat or embedding model, or pool settings in Settings → AI, the Hub waits 1.5 seconds for further changes, writes Memory’s new settings and restarts it. Memory describes photos with the chat model, so a separate vision model setting does not change it. A change in which Hubs are paired must hold for two checks first. After an automatic restart, the Hub does not restart the same app again for 10 minutes.
What the resource checks do
The Hub records CPU, memory and disk every 5 seconds. It marks an app as struggling when the app’s CPU use stays at 90% or more for 3 samples over 2 minutes and the app stops answering, and writes a warning to the Hub’s log. Every 15 minutes it alerts connected agents when disk or memory use is above 90%. Neither check changes how an app runs. The limits that do apply are the queue limits above, the Hub’s 3, 3 and 1 workers on its three job queues, and a default CPU limit for apps if you set one.
The Performance tab on the Hub’s Resources page shows this history, from the last 30 minutes up to 48 hours.
Training your own models
The models trained on your data today are the routine models in hbpe-service that forecast your day. How Companion learns explains the learning loop, what the nightly update does and which forecast you see. This section covers when that work runs and where the rest of the training code stands.
hbpe-service trains a first model for anyone who has none, in a sweep at 03:00 UTC and once 30 s after Memory starts. The nightly update starts at 02:30 UTC. It labels new activity within a 4-hour budget, then fine-tunes a small add-on to your model on yesterday’s activity, and the queue allows the whole job up to 8 hours. Forecasts normally read an exported copy of the model, which Memory writes only when it trains from scratch, so a nightly fine-tune reaches your forecasts after the next full retrain. Most forecasts come from a small model refit for each forecast anyway. A nightly add-on is saved only when the run trained cleanly, the three before it are kept, and there is no way yet to put an older one back from the app. The nightly update also retrains from scratch when your activities no longer fit the saved model, and an owner can queue a run at any time with ci train run.
| Piece | What it does today | Where | Status |
|---|---|---|---|
| Fixed labels for structured events | Workouts, sleep, trips, place visits, music, videos and social posts get an activity from a lookup table without a model call. Calendar entries and other free text go to the chat model, or to the small fallback model. | hbpe-service | Works today |
| Keyword labeler (fast activity labels) | The keyword labeler described above. If you turn it on, it scores keywords and uses the broad activity class when it scores 0.75 or more; otherwise the chat model picks it. The chat model still writes the finer label either way. This is a rule set, not a trained model. | hbpe-service | Behind a setting |
| Activity types for your tasks | When you create, edit or complete a task, Memory picks an activity type, such as work or exercise, from words in the task. A completed task is added only on a day that already has other activity, such as an event, and reaches the forecasting model with that type. This is a rule set, not a trained model. Older tasks get a type only when you next edit or complete them, and the forecasting service starts using task types only after it restarts once at least one task has one. | Memory API | Works today |
| Decision model trainer | Saves a file of made-up scores; it trains nothing. | hbpe-service | Not yet |
| Batch trainer for a second machine | Can train the forecasting model from a data file, but nothing creates that file or runs the trainer. | hbpe-service | Not yet |
| Keyword scoring for new memories and search | Ways for Memory to ask hbpe-service to score a new memory, reorder search results or label an event, using the keyword engine. Memory’s API gained calls for them in PR #3434 , but nothing in Memory uses them yet. This is separate from the quick-decision service that compares meanings, which How Companion learns describes. | hbpe-service | Not yet |
| Hub training jobs | The Hub has routes for training jobs, but nothing in the Hub or Memory calls them, and the Hub as shipped cannot run the trainer. | Hub | Not yet |
| Hub training schedules and placement | Functions for job manifests, schedules and choosing a machine, with tests. No part of the Hub uses them. | Hub | Not yet |
| Training on a paired Hub | A Hub can send a training job to a paired Hub. The paired Hub accepts it only when the training data is already stored on it, and no Hub can run the trainer yet. | Hub | Not yet |
| Fine-tuning a chat model on your data | A design proposal in the Memory repository. | Memory | Planned |
Personal AI on your own hardware lists what is being built next, including training on another device you own through the Hub.
What stays on your hardware
Your data, the search vectors, the job queues, your files, the summaries and your forecasting model all stay in Memory’s containers on your machine. Model requests stay on your network when the engine is Ollama on this machine or on a Hub you paired. A request sent to a paired Hub is read by that Hub’s engine, so pair only Hubs you control.
Some features do reach the internet:
- Connectors fetch your data from the cloud accounts you connect.
hbpe-servicedownloads its small fallback model from Hugging Face the first time it needs it.- Place lookups, link previews, crash reports and Portal sign-in are on by default, and you can switch each one off in Memory’s settings. Crash reports are only sent if a reporting address is configured, and a default install has none.
- If no local engine is ready and you have added a cloud provider in Settings → AI, the Hub gives Memory that provider instead.
Routing hard questions to a cloud model and sharing with other people’s Memory servers are both off by default.