System architecture

Companion Memory is an open-source, local-first personal memory engine. It ingests life archives, communication streams, and sensory inputs, normalizes them into a unified temporal knowledge graph, indexes them with multimodal embeddings, and answers queries using local machine learning models.

The entire system is designed to run self-hosted on user-owned hardware—such as Companion Core or any standard server running Companion Hub—with zero cloud dependencies for core search, retrieval, and chat functions.

High-level topology

Companion Memory operates as a containerized multi-tier service managed by Companion Hub or run standalone via Docker Compose.

flowchart TB
    subgraph Clients["Clients and entry surfaces"]
        Web["Web application<br/>React 19 SPA, Tailwind, Vite"]
        Mobile["Mobile shells<br/>iOS and Android via Capacitor"]
        Capture["Companion Capture<br/>Tray app, screen, files, audio"]
        Ext["Browser extension<br/>WXT, history, DOM, bookmarks"]
        MCPClient["AI agents<br/>Hermes, OpenClaw, IDEs via MCP"]
    end

    subgraph Appliance["Appliance runtime (Companion Hub / Core)"]
        Traefik["Traefik reverse proxy<br/>SSL termination, forward-auth"]

        subgraph CoreApp["Companion Memory backend (@ci/be-api)"]
            Gateway["NestJS API gateway<br/>REST, GraphQL, MCP server"]
            Ingest["Ingestion pipeline<br/>77+ archive parsers, 10 /v1 bulk lanes"]
            Search["Search and RAG engine<br/>Hybrid vector + BM25, graph ranker"]
            AskCore["Asks and consent engine<br/>Grants, ceilings, KERI verification"]
            Worker["Background job workers<br/>pg-boss: unpack, 37-stage enrich, embed"]
        end

        subgraph Sidecars["Companion analysis services"]
            SummarySvc["ci-summary (:8001)<br/>Summarization and MarkItDown /parse"]
            HBPESvc["ci-hbpe (:8002)<br/>Behavioral pattern engine"]
        end

        subgraph Storage["Persistence tier"]
            PG[("PostgreSQL 18<br/>pgvector (768-d), TimescaleDB, PostGIS")]
            Blobs[("Blob storage<br/>Encrypted raw assets, media, files")]
            PowerSync["PowerSync service<br/>SQLite change-data-capture stream"]
        end

        subgraph Inference["Local inference engines"]
            Ollama["Ollama / vLLM host engine<br/>Chat (gemma4:e4b), extraction, summary"]
            EmbedEng["Embedding and vision models<br/>nomic-embed-text (768-d), SigLIP, CLIP"]
        end
    end

    Web -->|HTTP / WebSocket| Traefik
    Mobile -->|PowerSync SQLite sync| PowerSync
    Capture -->|Authenticated REST| Traefik
    Ext -->|Authenticated REST| Traefik
    MCPClient -->|STDIO / SSE / Streamable HTTP| Gateway

    Traefik --> Gateway
    Gateway --> Ingest
    Gateway --> Search
    Gateway --> AskCore
    Gateway --> PG
    Gateway --> Blobs
    Worker --> PG
    Worker --> Blobs
    Worker --> SummarySvc
    Worker --> HBPESvc
    Worker --> Ollama
    Worker --> EmbedEng
    PowerSync --> PG
    Search --> PG
    Search --> Ollama
    Search --> EmbedEng

Monorepo codebase structure

Companion Memory is organized as a modular TypeScript and Python monorepo using Yarn workspaces. The primary NestJS application (@ci/be-api) runs as either an HTTP/GraphQL/MCP server (CI_ROLE=web) or a background queue consumer (CI_ROLE=worker) from the same container image.

• backend
  • apps
    • api
      • src
    • cli
    • summary
    • hbpe
  • packages
• frontend
  • apps
    • web
    • desktop
    • ios
    • android

Ingestion and normalization pipeline

The ingestion pipeline transforms unstructured, heterogeneous exports and live device feeds into a structured, relational knowledge graph. pg-boss is the single durable job scheduler across the appliance, backed by PostgreSQL.

Pipeline execution stages

Companion Memory ingests data through three coordinated paths:

1. Archive import queues (ci_import): Uploaded archives (.zip, .tgz, .json, .mbox) run through four durable pg-boss stages:
   • import-unpack: Verifies free disk headroom, expands nested archives, and catalogs files.
   • import-parse: Dispatches extracted files to matching parsers in the 77-importer registry.
   • import-media: Extracts binary assets, generates thumbnails and previews, and computes SHA-256 content hashes.
   • import-push: Deduplicates and commits normalized entities into the canonical ci schema.
2. Bulk /v1 ingest lanes: Live connectors, mobile apps, and Companion Capture stream records into ten typed REST/JSONL lanes under /api/v1/ (events, communications, contents, things/inventory, browser-activity, positions, notes, receipts, product-models, and observations), finalized by POST /api/v1/ingest/finish.
3. Document parsing and the 37-stage enrichment pipeline: Uploaded documents emit FileUploadedEvent, route through MarkItDownExtractor to the ci-summary /parse endpoint, and split into searchable passages in ci.file_chunks via DocumentChunkerService. Newly written entities then progress through a 37-stage versioned enrichment pipeline (ci.pipeline_state) across pipeline-index, pipeline-enrich, and pipeline-embed.

sequenceDiagram
    autonumber
    actor User as User / Client
    participant Inbox as Upload Inbox / /v1 Lanes
    participant Unpack as import-unpack
    participant Parser as import-parse (77+ Registry)
    participant Media as import-media & import-push
    participant Storage as PostgreSQL 18 & Blob Storage
    participant Enrich as 37-Stage Enrichment & Embed

    User->>Inbox: Upload archive (.zip, .tgz) or POST /api/v1/*
    Inbox->>Storage: Stage upload payload
    Inbox->>Unpack: Queue unpack job (pg-boss)
    Unpack->>Unpack: Check disk threshold, expand nested archives
    Unpack->>Parser: Dispatch extracted files to matching parser
    Parser->>Parser: Validate schemas & extract timestamps, metadata
    Parser->>Media: Emit normalized records and media refs
    Media->>Storage: Write media blobs and deduplicate canonical entities
    Media->>Enrich: Track stage versions in ci.pipeline_state
    Enrich->>Storage: Write file chunks (ci.file_chunks) and 768-d vectors (ci_search.embeddings)

Supported data sources (77+ importers)

The ingestion engine includes 77 purpose-built parsers across seven functional categories:

Deduplication and idempotency

To prevent data duplication across repeated imports or overlapping exports, Companion Memory enforces three layers of identity deduplication:

1. Content-addressable hashing: Media files and binary documents receive a SHA-256 fingerprint. Identical files are stored once and referenced by multiple events.
2. Deterministic source URIs: Communication events generate a deterministic external key: urn:<source>:<account_hash>:<native_message_id>.
3. Temporal fuzzy matching: Calendar events and location entries with matching time intervals and normalized coordinates merge into existing graph nodes rather than creating duplicates.

The canonical data graph

Rather than keeping records in siloed application tables, Companion Memory stores all imported records in PostgreSQL 18 (pgvector, TimescaleDB, PostGIS, btree_gist, and pg_stat_statements) using a single-table and class-table inheritance root at ci.entities. Nodes (Person, Place/Location, Concept, File, Item in ci.objects, Event in ci.events, Communication in ci.communications, and Content in ci.content) connect through typed edges in ci.relationships, while 768-dimensional vectors and lexical search indexes live in the dedicated ci_search schema (ci_search.embeddings and ci_search.vectors):

erDiagram
    EVENT ||--o{ MESSAGE : contains
    EVENT }o--o{ PERSON : involves
    EVENT }o--o{ LOCATION : occurs_at
    EVENT }o--o{ DOCUMENT : attaches
    EVENT }o--o{ THING : references
    PERSON ||--o{ MESSAGE : sends_or_receives
    DOCUMENT ||--o{ LOCATION : tagged_at
    DOCUMENT ||--o{ EMBEDDING : indexed_in

    EVENT {
        uuid id PK
        string title
        timestamp starts_at
        timestamp ends_at
        string event_type
        jsonb metadata
    }
    PERSON {
        uuid id PK
        string display_name
        string[] emails
        string[] phone_numbers
        jsonb handles
    }
    LOCATION {
        uuid id PK
        string label
        geometry coordinates
        string address
        jsonb bounds
    }
    DOCUMENT {
        uuid id PK
        string file_name
        string mime_type
        string storage_path
        string sha256_hash
    }
    EMBEDDING {
        uuid entity_id FK
        string model
        vector_768 embedding
    }
    MESSAGE {
        uuid id PK
        string content
        string sender_id FK
        string recipient_id FK
        timestamp sent_at
    }
    THING {
        uuid id PK
        string name
        string category
        jsonb attributes
    }

Multimodal embedding and search engine

Companion Memory implements a hybrid search architecture combining lexical search, 768-dimensional vector similarity, and relational graph traversal. By default, text embeddings use nomic-embed-text (768-d) and local chat synthesis uses gemma4:e4b on Ollama (or a configured vLLM / OpenAI-compatible host).

flowchart LR
    Query["Incoming user query"] --> Parse["Query parser"]
    Parse --> TextEmbed["Text embedder<br/>nomic-embed-text (768-d)"]
    Parse --> Lexical["Lexical parser<br/>PostgreSQL tsquery"]
    Parse --> GraphFilter["Entity filter<br/>Person, place, time bounds"]

    TextEmbed --> VectorSearch["pgvector cosine similarity<br/>HNSW index on ci_search.embeddings"]
    Lexical --> BM25Search["Full-text BM25 search<br/>GIN index on ci_search.vectors"]

    VectorSearch --> RRF["Reciprocal Rank Fusion (RRF)<br/>Combines dense and sparse results"]
    BM25Search --> RRF
    GraphFilter --> RRF

    RRF --> Rerank["Graph reranker<br/>Social distance and temporal decay"]
    Rerank --> Context["Synthesized LLM context"]
    Context --> LocalLLM["Local LLM (gemma4:e4b)<br/>Answers with grounded citations"]

Search retrieval stages

1. Query expansion: Identifies temporal markers (for example, “last summer” or “Friday”), locations, and named individuals.
2. Parallel retrieval:
   • Dense vector search: Queries the 768-dimensional embedding column in ci_search.embeddings using cosine distance (<=>) backed by an HNSW index.
   • Sparse text search: Evaluates PostgreSQL full-text search dictionaries in ci_search.vectors with BM25-like scoring across content, ci.file_chunks, and metadata.
3. Reciprocal Rank Fusion (RRF): Merges disparate score scales using rank-based weighting:

   RRF(d) = Σ [ 1 / (k + r_m(d)) ]

   where k = 60 and r_m(d) is the document’s rank in retrieval system m.
4. Relational reranking: Multiplies fused scores by temporal decay (recency bias) and social proximity (direct contacts outrank distant mentions).
5. Context assembly and citation grounding: Formats matching snippets with precise source identifiers before passing them to the local LLM.

Native agent intents and MCP tool surface

External AI agents and local assistants interact with the search and memory graph through a curated intent catalog mounted at GET /api/agent/intents (with legacy alias GET /api/intents) and bridged directly into the Model Context Protocol (MCP) server at /mcp. The catalog exposes 26 read-level native intents (and 36 total across read, write, and full capability tiers) spanning the core schema domains (memory, knowledge, people, places, calendar, media, planning, brief, insights, anomalies, forecast, suggestions, messaging, import, and persona). See Connect Companion Memory for the full tool and endpoint breakdown.

Edge replication and local-first storage

Companion Memory features a local-first architecture powered by PowerSync. Clients (web SPA, mobile apps, and desktop wrappers) replicate a read-optimized subset of the database to an embedded SQLite instance:

• Offline availability: The entire memory timeline, contacts graph, and search indices remain accessible on mobile devices with zero network connectivity.
• Bi-directional change data capture: Edits, tags, and reviews made offline queue locally and sync automatically when connectivity returns.
• Selective replication rules: High-resolution video and raw archives remain on the appliance; metadata, thumbnails, embeddings, and text streams replicate to edge devices.

Authentication and isolation model

Companion Memory secures API endpoints and user sessions through a multi-layer defense:

flowchart TB
    Req["Incoming request"] --> Traefik["Traefik reverse proxy"]
    Traefik -->|Forward-auth check| HubAuth["Companion Hub forward-auth endpoint"]
    HubAuth -->|Signs headers with HMAC / app secret| Traefik
    Traefik -->|Injects headers| Guard["HubForwardAuthGuard"]

    subgraph MemoryGuard["Companion Memory verification"]
        Guard --> CheckHeaders["Verify HMAC signature & timestamp skew (≤30s)"]
        CheckHeaders --> Replay["SpentHubAssertions ledger<br/>Prevents assertion reuse"]
        Replay --> Resolve["HubForwardAuthBridgeService"]
        Resolve --> StableID["urn:ci-hub:&lt;directory&gt;#&lt;user_id&gt;"]
        Resolve --> KeriAID["did:keri:&lt;user_aid&gt;"]
    end

    Resolve --> Session["Mint / align Memory sessionToken"]

Forward-auth header contract

When running behind Companion Hub, incoming requests receive signed headers verified by HubForwardAuthGuard:

• X-CI-Hub-User: The verified user’s current username (email).
• X-CI-Hub-User-Timestamp and X-CI-Hub-User-Signature (referred to as X-CI-Hub-Timestamp and X-CI-Hub-Signature at the Hub proxy boundary): Epoch millisecond timestamp and HMAC-SHA256 signature over ${username}\n${timestampMs}, rejected if clock skew exceeds 30 seconds (DEFAULT_FORWARD_AUTH_MAX_SKEW_MS = 30000).
• X-CI-Hub-User-Issuer: The unique Hub directory identifier (urn:ci-hub:<directory-uuid>).
• X-CI-Hub-User-Id and X-CI-Hub-User-Id-Signature: The immutable Hub person UUID and its HMAC-SHA256 binding (id\n${issuer}\n${userId}\n${username}\n${timestampMs}).
• X-CI-Hub-User-Assertion: Versioned one-time cryptographic assertion (2.<nonce>.<signature>) bound to Memory’s unique audience (ci-memory:ci-marketplace) and recorded in SpentHubAssertions to prevent replay.
• X-CI-Hub-User-AID: The person’s decentralized KERI Autonomic Identifier (did:keri:<user-aid>).
• X-CI-Hub-AID and X-CI-Hub-KEL-Digest: The appliance’s decentralized KERI Autonomic Identifier (did:keri:<hub-aid>) and optional Key Event Log digest.

This design prevents account takeover if an email address changes and isolates the personal memory database from neighboring applications on the same server.

Related

• Asking and sharing architecture — the ToIP dual-stack consent model, disclosure ladder, and ADR invariants
• Decentralized trust (KERI and ToIP) — autonomic identifiers, Key Event Logs, pre-rotation, and SAID digests
• Use cases and workflows — end-to-end wire payloads and audit records for common coordination scenarios
• Connect Companion Memory — MCP server, REST endpoints, and GraphQL API for external agents