Skip to Content
DocumentationCompanion MemoryHow your personal AI runs

How your personal AI runs

Your personal AI is two products working together on your own hardware. Companion Hub runs the containers and the AI models. Companion Memory is an app on the Hub that keeps your photos, messages, notes and calendar, answers your questions about them, and works through them in the background so the answers stay current.

This page starts with the big picture and gets more detailed as it goes. The numbers come from the Companion Memory and Companion Hub code. Each part carries one of the four status labels used across the Companion Memory pages, as defined on Personas:

  • Works today: the capability is on by default.
  • Behind a setting: the capability is in the code, but it is off until an owner or operator turns it on.
  • Not yet: the capability is not built, or it is built but not connected to the product.
  • Planned: the design is documented, and the code does not exist yet.

The page describes Companion Memory main and Companion Hub dev as of 12 November 2026. The latest releases, Memory v2026.11.11 and Hub v0.3.11, predate some of it. On Memory v2026.11.11 the keyword labeler is still on by default, tasks get no activity type, and the scoring endpoints described under Training are missing. Hub v0.3.11 has none of the Hub training code.

Containers in the Hub stack
6
Two of them, the Cloudflare tunnel and Tailscale, start only when you turn those features on.
from what runs
Memory containers that stay running
8
Plus 5 setup containers that run once and exit.
from what runs
Scheduled jobs in Memory
62
17 run in the worker container; 12 are switched off by default.
from background jobs
Models Memory starts with
2
A chat model and an embedding model. On a Hub, the Hub picks the chat model for your hardware.
from models
Companion Memory connects your sources, makes them searchable, and lets the apps you choose read from them.

The short version

  • The Hub is the part that talks to Docker. It installs Memory as a set of containers, gives Memory the address of its AI endpoint, and restarts Memory when you change AI settings.
  • The AI models run in an inference engine on the same machine, usually Ollama. Memory never loads a language model itself, apart from one small fallback labeling model described below. Your forecasting model is small and runs inside hbpe-service. For everything else, Memory sends requests to the Hub, and the Hub decides which engine, and which paired Hub, answers.
  • Memory does its work in queues. Your chat turn has its own queue and goes first. Photo descriptions, summaries, embeddings and training wait their turn, and the heaviest cleanup work runs in a night window.
  • Your data, the search index and the job queues all live in Memory’s Postgres database on your hardware.

This page describes Memory installed from the Hub app store, which is the usual setup. Memory can also run on its own with its own compose files. It then talks to Ollama on the host directly, and the Hub sections do not apply.

What runs on your hardware

The Hub runs a small stack of its own: a database, a message queue for long-running work, the Hub itself, and Traefik, which receives web traffic and passes it to the right app. Each app you install, Memory included, runs as its own group of containers next to that stack. The AI engines run on the host machine, and the Hub reaches them over the host’s network.

The moving edges are the path a chat question takes. The dotted edge is the Hub creating Memory’s containers through Docker. Memory’s sidecars and worker send their own model requests to the same Hub endpoint as the API.

ContainerWhat it doesAI models it usesStatus
ci-hubThe Hub API and dashboard. Installs, starts and updates apps through Docker, and routes AI requests to an engine or a paired Hub.None of its own. It forwards requests to engines.Works today
ci-hub-dbThe Hub’s Postgres 14 database: settings, installed apps, job state.NoneWorks today
ci-hub-queueRabbitMQ. Carries long-running Hub work such as installs, updates and catalog syncs.NoneWorks today
traefikReceives web traffic and sends it to the Hub or to an app, after the Hub checks who you are.NoneWorks today
cloudflaredOptional. Public access through a Cloudflare tunnel.NoneBehind a setting
hub-tailscaleOptional. Private access over Tailscale.NoneBehind a setting
Ollama on the hostRuns the models. The Hub can also use vLLM, Lemonade, oMLX or LocalAI if you run them.Chat, vision and embedding modelsWorks today
gateway (Memory)Memory’s single entry point, an nginx server. Sends /api to the API, sync traffic to PowerSync and file reads to the object store.NoneWorks today
api (Memory)Answers the apps, runs chat and search, and runs 45 of the 62 scheduled jobs.Chat, vision and embedding models, through the HubWorks today
worker (Memory)The same code in a second container. Runs imports and 17 maintenance jobs so a large import does not slow down your requests.Embedding and background models, through the HubWorks today
database (Memory)Postgres 18 with extensions for vector search (pgvector), time series (TimescaleDB) and places (PostGIS). Holds your data, the search vectors and Memory’s job queue.NoneWorks today
object-store (Memory)RustFS. Holds your original files, such as photos and documents.NoneWorks today
summary-service (Memory)Writes hourly, daily, weekly and monthly summaries, insights and suggested actions. Also extracts text from documents with MarkItDown.The chat model and nomic-embed-text, through the HubWorks today
hbpe-service (Memory)Learns your daily rhythm: labels activity, trains a small forecasting model per person, and serves forecasts.Your own forecasting model, and the chat model or a small local fallback model for labels. A keyword labeler is off by default.Works today
powersync (Memory)Keeps the data that apps hold locally in sync with the server.NoneWorks today
Setup containers (Memory)Five containers run once at install or start: they create secrets, fix data-folder permissions, prepare and migrate the database, and set up sync, then exit.NoneWorks today

Memory’s API and worker are one program started twice. The API container serves your requests, and the worker container takes the import and maintenance queues. A long import parser then never shares a process with the request you are waiting on.

Which AI models do what

Memory asks for a model by job, and the Hub decides what serves it. When Memory runs from the Hub, the Hub writes Memory’s model settings into its environment: the endpoint address, the chat model, the background model, the embedding model and the context size. By default the endpoint is the Hub’s pool proxy, so a second Hub can serve requests later without Memory changing anything. You can turn that routing off, and when no local engine is ready the Hub gives Memory a cloud provider’s address instead.

ModelTaskWhere it runsWhen it loadsStatus
Chat model, gemma4:e4b unless you or the Hub pick anotherChat answers, agent tools, summaries, planning scans, answer draftsThe engine the Hub picks, on this Hub or a paired oneOn the first request after it was unloadedWorks today
Background model (set as LLM_UTILITY_MODEL)Extraction and classification in the backgroundSame as chatOn a Hub, a smaller model you already have, if one fits next to the chat model; otherwise the chat modelWorks today
Vision (the chat model, when it can read images)Describes each new photoSame as chat. Photos are sent at 1280 px on the long edge.When a photo’s preview is ready and no chat turn is waitingWorks today
nomic-embed-text v1.5Turns text into 768 numbers so search can find things by meaningOllama, or the chat engine when it serves this modelWhen there is text to embedWorks today
Your forecasting modelForecasts your day in 96 quarter-hour slotshbpe-service, on the CPU unless a GPU is passed throughTrained once you have enough history, then updated nightlyWorks today
Keyword labelerPicks the broad activity class (work, exercise and so on) from keyword matches, so the chat model is asked only for the finer label. Other pages call it the keyword fast path or the keyword shortcut.hbpe-serviceOff by default. If you turn it on with HBPE_ENABLE_DECISION_MODELS, a label scoring at least 0.75 is used as is.Behind a setting
Qwen3.5-0.8B (a model file in the GGUF format)Labels activity when no chat endpoint is configuredInside hbpe-serviceDownloaded from Hugging Face the first time it is needed. It checks for an endpoint every 300 s and switches back when one appears.Works today

Memory runs no speech-to-text or OCR model. Document text comes from MarkItDown, which reads the file format and does not use a model.

How the Hub picks a chat model for your hardware

The Hub sorts your machine into a tier and recommends models from its catalog that fit that tier. Until you choose a chat model in Settings → AI, apps get the highest-ranked recommended model that is already downloaded. A model may use only part of your memory, so the operating system and your apps keep room to work.

TierHardwareShare of memory a model may use
HighA GPU, or unified memory the GPU can use, of at least 16 GB90% of VRAM on a separate GPU; 70% of RAM with unified memory
MediumA GPU, or unified memory the GPU can use, of at least 8 GBSame as above
LowA GPU, or unified memory the GPU can use, of at least 4 GBSame as above
CPU onlyNo usable GPU and at least 16 GB of RAM70% of system RAM
InsufficientNo usable GPU and less than 16 GB of RAMNone recommended

On an Apple Silicon Mac the GPU can use all of RAM, so a 16 GB Mac is High. If the Hub finds unified memory but no usable GPU runtime, it falls back to 32, 16 and 8 GB of RAM for High, Medium and Low.

Nomic Embed Text: 0.3 GB — embeddings, every tierNomic Embed Text0.3 GB · embeddings, every tierQwen 3.5 4B: 3.4 GB — cpu-onlyQwen 3.5 4B3.4 GB · cpu-onlyQwen 3.5 9B: 6.6 GB — lowQwen 3.5 9B6.6 GB · lowGemma 4 E4B: 9.6 GB — cpu-onlyGemma 4 E4B9.6 GB · cpu-onlyGPT-OSS 20B: 14 GB — mediumGPT-OSS 20B14 GB · mediumQwen 3.8 27B: 18 GB — mediumQwen 3.8 27B18 GB · mediumQwen 3.5 122B: 81 GB — highQwen 3.5 122B81 GB · high
Download size of some chat models in the Hub catalog, with the lowest tier each is recommended forChat model sizes are the catalog's q4_K_M download sizes (q4_K_M is a common way to compress a model). Gemma 4 E4B is Memory's own default chat model.

Two walkthroughs

Asking a question in chat

  1. You askThe Memory app sends your questionYour phone
  2. GatewayTraefik and Memory's gateway pass it to the APIHub, Memory
  3. Chat queueOne turn at a time; background model work stands asideMemory
  4. RecallLooks up memories that match, for up to 3 sMemory
  5. Pool proxyPicks the engine with the shortest queueHub
  6. ModelLoads the chat model if needed, then answersEngine
  7. Stream backWords appear as they are written; the turn is savedYour phone
What happens when you ask Memory a questionShown for Memory installed from the Hub. On its own, Memory sends the request to Ollama directly.

The chat queue takes one turn at a time and nothing else. If the search for matching memories takes longer than 3 seconds, the turn goes ahead without it, and the model can still search your memories with its own tools while it answers.

A new photo arriving

  1. UploadThe app sends the photoYour phone
  2. StoreThe original goes to the object storeMemory
  3. PreviewA preview job makes a smaller copyMemory
  4. DescribeWaits for chat, then the vision model writes a title and descriptionMemory, Hub
  5. EmbedTitle, description and file name become a 768-number vectorMemory, Hub
  6. SearchableSearch finds it by what is in the pictureMemory
From a new photo to something search can findFile embeddings are due at most once an hour and are never left more than 6 hours behind. A nightly sweep at 03:20 UTC retries photos that did not get a description.

A new file is added to Memory’s search index as soon as it is created, so you can find it by name straight away. Finding it by what the picture shows has to wait for the description and the embedding. Memory checks for due pipeline work every 5 minutes. After an import, it starts the embedding pass directly instead of waiting for the hourly slot.

What runs in the background, and when

Memory’s scheduled jobs are timers inside the API and worker containers. A timer does not do the work itself. It puts jobs on a queue in Memory’s Postgres database, one per person where that applies, and the queue workers pick them up. The Hub uses RabbitMQ for its own long-running work, and it schedules its own repeating jobs, such as app status and catalog syncs, through RabbitMQ too.

Memory’s containers set no time zone, so its schedules run in UTC unless you set TZ for the app.

HubMemory00:0006:0012:0018:0024:00Runs a dayDue reminders — Every minute · 1,440 runs · MemoryDue reminders1,440 runsPipeline tick — Every 5 minutes · 288 runs · MemoryPipeline tick288 runsInbox scan — Every 30 minutes · 48 runs · MemoryInbox scan48 runsHourly summary — Every 60 minutes (at :05) · 24 runs, at :05 · MemoryHourly summary24 runs, at :05Insights — At 00:30, 03:30, 06:30, 09:30, 12:30, 15:30, 18:30, 21:30 · 8 runs · MemoryInsights8 runsTodo processing — At 00:00, 06:00, 12:00, 18:00 · 4 runs · MemoryTodo processing4 runsDaily summary — At 00:10 · 1 run · MemoryDaily summary1 runHBPE nightly training — At 02:30 · 1 run · MemoryHBPE nightly training1 runHBPE bootstrap sweep — At 03:00 · 1 run · MemoryHBPE bootstrap sweep1 runImage description sweep — At 03:20 · 1 run · MemoryImage description sweep1 runTodo scan — At 06:00 · 1 run · MemoryTodo scan1 runApp status sync — Every 5 minutes · 288 runs · HubApp status sync288 runsPortal check-in — Every 15 minutes · 96 runs · HubPortal check-in96 runsApp catalog sync — Every 15 minutes (production) · 96 runs, production · HubApp catalog sync96 runs, productionOrphan network cleanup — Every 30 minutes · 48 runs · HubOrphan network cleanup48 runs
JobRuns onScheduleRuns a day
Due remindersMemoryEvery minute1,440 runs
Pipeline tickMemoryEvery 5 minutes288 runs
Inbox scanMemoryEvery 30 minutes48 runs
Hourly summaryMemoryEvery 60 minutes (at :05)24 runs, at :05
InsightsMemoryAt 00:30, 03:30, 06:30, 09:30, 12:30, 15:30, 18:30, 21:308 runs
Todo processingMemoryAt 00:00, 06:00, 12:00, 18:004 runs
Daily summaryMemoryAt 00:101 run
HBPE nightly trainingMemoryAt 02:301 run
HBPE bootstrap sweepMemoryAt 03:001 run
Image description sweepMemoryAt 03:201 run
Todo scanMemoryAt 06:001 run
App status syncHubEvery 5 minutes288 runs
Portal check-inHubEvery 15 minutes96 runs
App catalog syncHubEvery 15 minutes (production)96 runs, production
Orphan network cleanupHubEvery 30 minutes48 runs
When the main background jobs run in a dayMemory times are UTC. Hub app status, catalog and network jobs run on the clock every 5, 15 and 30 minutes; the Portal check-in counts 15 minutes from start.
JobWhenWhat it doesRuns inStatus
Connector syncPending syncs every 30 s; new sync jobs every minutePulls new data from the cloud accounts you connectedMemory APIWorks today
Answer draftsChecked every minute; each person at most once every 20 minutesDrafts answers to open questions from your own data, on the background model queueMemory APIWorks today
Enrichment windowOpens 01:00 UTC for 5 hours, with at most 90 minutes of work a nightCatch-up previews, place lookups, link previews and duplicate checksMemory APIWorks today
Weekly and monthly summaryMondays 00:15 UTC; the 1st of the month at 00:20 UTCLonger summaries from the daily onesMemory summary-serviceWorks today
Retrieval checksDaily at 03:40 UTC; a recall probe on Sundays at 05:20 UTCChecks that search still finds what it shouldMemory workerWorks today
Smart rotation10 alternative schedulesWould replace the feed, answer, insight and summary timers with a rotation across people. Off by default.Memory APIBehind a setting
Host telemetryEvery 5 sRecords CPU, memory, disk and Docker state, kept for 48 hoursHubWorks today
Memory key rotation60 s after start, then every 12 hoursReplaces Memory connection keys older than 60 daysHubWorks today
Hub update checkEvery 24 hours, starting 24 hours after the Hub starts (production)Updates the Hub image when automatic updates are on, which is the defaultHubWorks today
Backup retentionEvery 7 days, if the Hub stays up that longTrims each app’s backups to its limit, or to the newest 5 when no limit is setHubWorks today

For the full list of Memory’s queues and jobs, see AI training, inference & jobs. For every Hub timer, see How Hub runs apps and models.

How it stays out of your way

Chat goes first

Memory keeps one queue for chat turns and nothing else. Background model work goes on other queues: thread titles, answer drafts, photo descriptions, object recognition in photos and browsing insights. Those queues share a single lane, so at most one background request from the API reaches the model at a time. They also stop taking new jobs while any chat turn is waiting or running.

The activity labeler in hbpe-service follows the same rule. It sends at most 6 label requests a minute and waits up to 600 seconds for a chat turn to finish before it sends the next one.

Heavy work waits, and queues have limits

Embedding runs at any hour, because search depends on it. A new upload gets its preview right away. The catch-up sweeps for previews, plus place lookups, link previews and duplicate checks only start inside the enrichment window, 01:00 to 06:00 UTC by default, and spend at most 90 minutes of work a night. When several people share one Memory, each gets a 20-minute turn, starting with whoever was served longest ago.

Memory runs at most 10 jobs at once per container and 6 from any one queue. The embedding queue holds at most 25 ready jobs, because the model process it uses also serves chat, the activity labeler and the summary sidecar. A job that fails is retried 3 times with growing delays and then set aside on a failed-jobs list (a dead-letter queue), where you can replay it.

Models load when needed and make room

When the Hub loads a model, it first checks that the model fits a memory budget. The budget keeps 2048 MB of system RAM and 512 MB of graphics memory free for the machine. If the model does not fit, the Hub unloads idle models that it loaded itself and that you have not pinned, least recently used first, and waits up to 10 seconds for the memory to free. Loads happen one at a time, and so do downloads.

A model the Hub loads stays loaded until the Hub unloads it or needs the room. Pinning a model keeps it there, and pins survive a restart. When you unpin, the model goes back to Ollama’s idle timeout, which is 5 minutes unless you changed it.

If an engine keeps failing for one model, the Hub stops sending it that model for 60 seconds, doubling up to 15 minutes while the failures continue. The Hub also sets each model’s context window, starting at 65,536 tokens and stepping down to as little as 4,096 until the model fits.

A second Hub can take some of the load

By default, Memory sends its model requests to the Hub’s pool proxy. With one Hub, the proxy serves everything locally. Once you pair a second Hub with Hub Pool, the proxy ranks every engine that has the requested model, on both Hubs, by how many requests each is already handling. Your own Hub gets a head start of one request. If an engine fails before it starts answering, the proxy tries the next one. Each Hub checks its paired Hubs every 30 seconds by default.

The Hub can also measure how busy its GPU is, but by default it does not use that to route requests.

Settings changes restart Memory once

When you change the AI engine, the chat or embedding model, or pool settings in Settings → AI, the Hub waits 1.5 seconds for further changes, writes Memory’s new settings and restarts it. Memory describes photos with the chat model, so a separate vision model setting does not change it. A change in which Hubs are paired must hold for two checks first. After an automatic restart, the Hub does not restart the same app again for 10 minutes.

What the resource checks do

The Hub records CPU, memory and disk every 5 seconds. It marks an app as struggling when the app’s CPU use stays at 90% or more for 3 samples over 2 minutes and the app stops answering, and writes a warning to the Hub’s log. Every 15 minutes it alerts connected agents when disk or memory use is above 90%. Neither check changes how an app runs. The limits that do apply are the queue limits above, the Hub’s 3, 3 and 1 workers on its three job queues, and a default CPU limit for apps if you set one.

The Performance tab on the Hub’s Resources page shows this history, from the last 30 minutes up to 48 hours.

Training your own models

The models trained on your data today are the routine models in hbpe-service that forecast your day. How Companion learns explains the learning loop, what the nightly update does and which forecast you see. This section covers when that work runs and where the rest of the training code stands.

hbpe-service trains a first model for anyone who has none, in a sweep at 03:00 UTC and once 30 s after Memory starts. The nightly update starts at 02:30 UTC. It labels new activity within a 4-hour budget, then fine-tunes a small add-on to your model on yesterday’s activity, and the queue allows the whole job up to 8 hours. Forecasts normally read an exported copy of the model, which Memory writes only when it trains from scratch, so a nightly fine-tune reaches your forecasts after the next full retrain. Most forecasts come from a small model refit for each forecast anyway. A nightly add-on is saved only when the run trained cleanly, the three before it are kept, and there is no way yet to put an older one back from the app. The nightly update also retrains from scratch when your activities no longer fit the saved model, and an owner can queue a run at any time with ci train run.

PieceWhat it does todayWhereStatus
Fixed labels for structured eventsWorkouts, sleep, trips, place visits, music, videos and social posts get an activity from a lookup table without a model call. Calendar entries and other free text go to the chat model, or to the small fallback model.hbpe-serviceWorks today
Keyword labeler (fast activity labels)The keyword labeler described above. If you turn it on, it scores keywords and uses the broad activity class when it scores 0.75 or more; otherwise the chat model picks it. The chat model still writes the finer label either way. This is a rule set, not a trained model.hbpe-serviceBehind a setting
Activity types for your tasksWhen you create, edit or complete a task, Memory picks an activity type, such as work or exercise, from words in the task. A completed task is added only on a day that already has other activity, such as an event, and reaches the forecasting model with that type. This is a rule set, not a trained model. Older tasks get a type only when you next edit or complete them, and the forecasting service starts using task types only after it restarts once at least one task has one.Memory APIWorks today
Decision model trainerSaves a file of made-up scores; it trains nothing.hbpe-serviceNot yet
Batch trainer for a second machineCan train the forecasting model from a data file, but nothing creates that file or runs the trainer.hbpe-serviceNot yet
Keyword scoring for new memories and searchWays for Memory to ask hbpe-service to score a new memory, reorder search results or label an event, using the keyword engine. Memory’s API gained calls for them in PR #3434 , but nothing in Memory uses them yet. This is separate from the quick-decision service that compares meanings, which How Companion learns describes.hbpe-serviceNot yet
Hub training jobsThe Hub has routes for training jobs, but nothing in the Hub or Memory calls them, and the Hub as shipped cannot run the trainer.HubNot yet
Hub training schedules and placementFunctions for job manifests, schedules and choosing a machine, with tests. No part of the Hub uses them.HubNot yet
Training on a paired HubA Hub can send a training job to a paired Hub. The paired Hub accepts it only when the training data is already stored on it, and no Hub can run the trainer yet.HubNot yet
Fine-tuning a chat model on your dataA design proposal  in the Memory repository.MemoryPlanned

Personal AI on your own hardware lists what is being built next, including training on another device you own through the Hub.

What stays on your hardware

Your data, the search vectors, the job queues, your files, the summaries and your forecasting model all stay in Memory’s containers on your machine. Model requests stay on your network when the engine is Ollama on this machine or on a Hub you paired. A request sent to a paired Hub is read by that Hub’s engine, so pair only Hubs you control.

Some features do reach the internet:

  • Connectors fetch your data from the cloud accounts you connect.
  • hbpe-service downloads its small fallback model from Hugging Face the first time it needs it.
  • Place lookups, link previews, crash reports and Portal sign-in are on by default, and you can switch each one off in Memory’s settings. Crash reports are only sent if a reporting address is configured, and a default install has none.
  • If no local engine is ready and you have added a cloud provider in Settings → AI, the Hub gives Memory that provider instead.

Routing hard questions to a cloud model and sharing with other people’s Memory servers are both off by default.

Last updated on