Skip to Content
DocumentationCompanion Scan

🥽 Companion Scan

Scan a room with your phone, then explore it in 3D. Walk a space with your phone’s camera, and Companion Scan builds a 3D map that you can open, label, and measure in a desktop editor, all stored on your own Hub.

This page covers Companion Scan, not Companion Scan Quest. Companion Scan is the phone scanner and desktop world editor that you install from the Companion Hub App Store (source: CI-Spatial). Companion Scan Quest is a separate app for the Meta Quest 3 that runs in the Meta Quest Browser (source: CI-Spatial-Web-Meta).

Install & availability

Available now, as a beta. Install from the Companion Hub App Store. The store lists it as Companion Scan.

Hub App StoreSearch Companion Scan → Install
Minimum Hub version0.2.41
Port8000 inside the container, on the Hub’s own port for the app (the Hub app card shows it). One address serves the phone scanner, the editor, and the API
Architecturelinux/amd64 and linux/arm64 (multi-arch)
StorageScans, worlds, and summaries live in the app’s data folder on your Hub

This page describes the Companion Scan image that the Hub store installs: the latest tag, which is the v0.1.0 prerelease built on 29 October 2026 from source commit f3e9a1e. Source changes on main made after that commit aren’t in the image. The screenshots on this page were added on 3 October 2026, before v0.1.0.

Limits of the store build.

  • Scanning needs an AR-capable phone and an HTTPS address. See Open the scanner on your phone.
  • Object labels need an Ollama vision model that the app can reach. Without one, the server can’t label scans. See Give the app a vision model.
  • Upload a video and photo-set upload failed in the July 2026 build, whose image lacked ingest-quality.mjs, a file the upload script imports. The v0.1.0 image includes that file. Its release notes say the upload routes weren’t tested, so whether uploads work is to confirm.
  • Splat training and dense reconstruction need GPU tools that the image doesn’t include.
  • Companion Scan sends scans to Companion Memory only after Memory is connected and you queue a Memory push job. In the store build, anyone who can reach the app’s address can open the editor, and Companion Account sign-in doesn’t change that. See Choose who can reach it.

Overview

Companion Scan is a browser-based AR room scanner and a desktop world editor. On the phone it uses the free 8th Wall  SLAM engine binary, with a native WebXR fallback, to track the space and build a live coloured point cloud while a vision-language model labels the objects it sees. On the desktop, the editor places one or more scans in a shared, metric, geo-anchored 3D world that you can inspect, align, label, measure, and share by link.

Scan files, camera frames, and detections stay on your Hub and the Ollama it uses. Two optional features send data further: a Companion Memory push, and sign-in with Companion Account. See What leaves your network for those and for the few third-party requests the pages make.

Companion Scan start screen on a phone with a Scan a room card and an Upload a video card
The phone start screen as captured on 3 October 2026, before v0.1.0. Scan a room starts live AR.

What a world is

A world is a metric frame, in meters, anchored to a GPS origin. It holds one or more scans, and each scan is split into typed layers: point cloud, meshes, floors, detections, labels, and the camera’s path. The fourth dimension is time: every frame is timestamped, and edits are an append-only history, so one world can hold several scans of the same place captured at different moments. The editor opens the world named default unless you ask for another with ?world=<name>.

Key features

  • Web AR scanning — 8th Wall SLAM tracks the room and builds a live point cloud. A native WebXR scanner at /webxr.html runs on browsers that support it.
  • Object detection — each capture sends a frame to your Hub, where an Ollama vision model returns labelled boxes. The labels become 3D pins at their positions in the room.
  • Scan quality guidance — a coverage bar, warnings for fast movement, low light, and reflective surfaces, and a verdict chip (Ready, Review, or Sparse). Done opens the export panel, and export asks you to confirm when the scan is sparse.
  • Three ways to capture — tap to capture, auto-capture as you walk, or keyframe streaming that sends posed frames to the Hub for reconstruction jobs.
  • Live view on the desktop — Start Live Stream shows the scan growing in the editor in real time.
  • Autosave and history — the first capture uploads silently. Sessions stay on the phone, and the history drawer (Scan Memory) lists them with a map of where you scanned.
  • Desktop world editor — scans in one shared frame, with render modes, layer toggles, a measure tool, a map, semantic tags, notes, and move, rotate, and scale controls.
  • Server processing — from the phone’s panel, ask the Hub for a floor plan, a scene description, or an object report. Uploads are compressed and get lower-detail previews that the editor loads first.
  • Share links by role — owner, editor, viewer, and scanner links for a world.
  • Installable and offline-tolerant — a PWA with an offline queue for uploads.

The scanning experience

On your phone: scan a room

Open the app’s HTTPS address, allow camera and motion access, choose Scan a room, and sweep the space slowly. The HUD in the top-left corner shows tracking status, the live point count, the coverage bar, and the verdict chip. The buttons on the right open the desktop editor, the Review and Export panel, keyframe streaming, auto-capture, and the history drawer.

Companion Scan scanning HUD on a phone with a Collect more tracking points warning, a Sparse verdict chip, a coverage bar reading Low keep scanning, a Done button beside the capture button, and an action rail on the right
The scanning HUD in the app's test mode, which uses a simulated camera (the colour blocks) instead of a real room.

Tap the capture button to identify objects. Each detection appears as a card and as a 3D pin in the scene. When you’re done, select Done to review and export.

Review and Export panel with detected objects as cards, an Export GLB button, an Export Summary button, server processing actions for a floor plan, a scene description, and an object report, and a Server or WebGPU engine switch
Review and Export, in test mode. The detection text comes from the app's built-in test engine, not a real vision model.

On the desktop: structure the world

Open the editor at /viewer.html?world=default on your app’s address, or tap the desktop button on the phone’s action rail. The Scans tab lists every scan in the world. Select one to focus the view on it.

Companion Scan world editor with a list of six scans on the left, a focused 3D point cloud of a one-bedroom home with labelled detections and a transform gizmo, render mode buttons along the top, and a compass and scale bar
The world editor with example data: six generated test homes, with one selected.

The World tab shows the world’s totals, the share links, and the processing queue.

World tab of the editor with stats for six scans, eleven layers, and zero jobs, owner, editor, viewer, and scanner share links with Copy buttons, and a processing queue with a job type picker and a Queue Job button
The World tab: totals, share links by role, and the processing queue. Example data.

How it works

From a phone sweep to a shareable world, the data moves through these stages:

Reconstruction methods

MethodWhat it producesHardwareIn the Hub install
Live SLAM point cloudColoured point cloud and object pinsPhone (no GPU)Included
Vision-language object detectionLabelled 3D detectionsAn Ollama vision model, or ONNX or WebGPU in the browserIncluded. The app needs an Ollama vision model it can reach
Scan optimizationCompressed .opt.glb files and lower-detail previewsCPUIncluded
Video and photo-set ingestKeyframes from an uploadCPU (ffmpeg)Failed in the July 2026 build. Whether it works on v0.1.0 is to confirm. The keyframes also need LingBot-Map to get poses
Gaussian splatting (Brush)Photorealistic splat filesGPUNot included. The job is skipped with a reason
Dense reconstruction (LingBot-Map)Fused point cloud and camera posesGPUNot included. The job is skipped with a reason

Setup

Install from Hub

Search for Companion Scan in the Hub app store and install. It needs Companion Hub 0.2.41 or newer. The install form has these choices:

  • Sign in with Companion Intelligence is off by default. When it’s on, an allowed CI Account that signs in lands in the editor with owner rights. It doesn’t lock the app by itself.
  • Allowed Companion accounts lists the emails that may sign in. An empty list rejects everyone.
  • Companion Memory URL and API key are advanced. Leave both blank and the Hub connects the app to your Companion Memory. Fill them in only for an external, self-managed CI-Server. See Settings from Companion Hub.

Choose who can reach it. Companion Scan stores scans of your home, and by default anyone who can reach its address can open the editor and upload scans. When the install form offers Public Web, leave Enable authentication on, or choose Private VPN or This Machine Only. See Expose your apps.

Give the app a vision model

Detection uses an Ollama vision model. The app reads its name from OLLAMA_MODEL, which the Hub sets from its vision model setting. If that setting is empty, the Hub passes the vision model it recommends for the machine. If the Hub passes no model, the app uses qwen3-vl:4b. The app checks Ollama once, when it starts. Pull the model in use on the computer that runs Ollama. For the default, run:

ollama pull qwen3-vl:4b

If the named model isn’t pulled, the app first uses another tag of the same model, if one is pulled. Otherwise it uses the first of qwen3-vl:4b, qwen3.5:4b, qwen3.5:2b, llava:7b, or moondream:latest that is pulled, and then the first model Ollama lists. Open <app address>/health to check: "modelReady": true means the engine loaded, and "engineLoadState": "failed" means it couldn’t use Ollama at startup, either because Ollama couldn’t be reached or because no models were pulled. If the Hub’s inference engine isn’t Ollama, the Hub doesn’t set OLLAMA_HOST, so the app tries http://localhost:11434 inside its own container, and detection stays unavailable.

Open the scanner on your phone

Your phone needs an HTTPS address for the app: camera and motion APIs require a secure origin, and a plain http://<hub-address>:<port> address isn’t one. Expose the app or reach the Hub over Tailscale. Open the link and grant camera and motion access when prompted. A two-step onboarding overlay explains camera and motion access and how to walk, and you can replay it from the ? button.

Live AR needs a recent iPhone with Safari, or an Android phone with Chrome and ARCore. The project’s device checklist covers both, but its results table is empty, so we couldn’t confirm which phones work. Native WebXR (/webxr.html) doesn’t run in iOS Safari. On iOS, open it in a WebXR-capable browser such as HelloXR instead.

Capture footage

For best results:

  • Move slowly and steadily, and let the coverage bar reach a good level before you finish.
  • Keep even lighting, and avoid direct sunlight and harsh shadows.
  • Sweep surfaces from multiple heights and angles.

Select Done when the verdict chip reads Ready, then Export GLB.

Open the world editor

On a computer, open /viewer.html?world=default on the app’s address. Select a scan to focus it, then use the World tab to copy a share link or queue a job.

Settings from Companion Hub

The Hub fills in most of the scanner’s settings. The app’s store manifest declares which environment variables the Hub writes, and the values come from the Hub’s own inference settings and from its Companion Memory and Companion Account connections. The install form has the fields you can change directly.

  • Inference. The Hub writes OLLAMA_HOST, its Ollama address (only when its engine is Ollama), and OLLAMA_MODEL, the vision model it chose. The app reads both when it starts. See Give the app a vision model for how the model is chosen.
  • Companion Memory. Leave the URL and API key blank for auto-connect. Start the Companion Memory connection from the Hub. When it finishes, the Hub writes CI_SERVER_URL (the API base) and CI_SERVER_TOKEN (a memory-scoped key), and restarts the app. A brokered connection takes precedence over values you type, so type them only for an external, self-managed CI-Server. Until both variables are set, a memory push job reports that Companion Memory is not connected.
  • Sign-in with Companion Account (optional). Sign-in is off by default. When Sign in with Companion Intelligence is on, the app reads PORTAL_OIDC_ENABLED and lets only the accounts listed in PORTAL_OIDC_ALLOWED_EMAILS sign in. A signed-in account gets the owner link, and sign-in doesn’t block other visitors. The Hub writes OIDC_ISSUER_URL as the Companion Portal it is paired with, so sign-in uses the same Companion Account as the Hub, over OIDC through the Portal. If the Hub isn’t paired with the Portal, it doesn’t write that variable, and sign-in stays off.

Usage

Viewing a completed scan

Orbit, pan, and zoom the scene. The render buttons along the top switch between Solid, Points, Wire, and Semantic (recoloured by tag). The Layers tab toggles each layer on its own: the point cloud, meshes, floors, detections, labels, and camera path. Map pins every GPS-tagged scan on a map.

Measuring

Select Measure, then click two points. Because the world is metric, the readout is in real meters.

Tagging and editing

Select a scan in the Scans tab to open the Edit tab. There you can move, rotate, or scale the scan, assign a semantic tag, add notes, or hide it. Every edit is saved as an append-only patch.

Sharing

The World tab has owner, editor, viewer, and scanner links. Each carries a token in its address. The store build doesn’t enforce those tokens by default, so treat the app’s address itself as the thing to protect.

Exporting

From the phone’s Review and Export panel, Export GLB saves the scan as a .glb file and uploads it to your Hub, and Export Summary saves a summary of the scan.

Technical architecture

Companion Scan is open source under the PolyForm Noncommercial 1.0.0 license . It is designed as a mobile-first Augmented Reality (AR) spatial indexing application that digitizes physical rooms into structured, queryable 4D metric maps.

The spatial data model hierarchy

  1. Place: The top-level geographic root anchored to a physical GPS origin and metric ENU (East-North-Up) Cartesian coordinate frame.
  2. Area: Named semantic zones or rooms (for example, “Living Room”, “Kitchen”, or “Office”) situated within a Place.
  3. Scan session: A temporal continuous capture pass consisting of timestamped camera poses and raw frames.
  4. Scan layers:
    • points: Colored 3D point vertices and normals.
    • mesh: Reconstructed triangle geometry and collision boundaries.
    • cameraPath: 6DoF camera trajectory spline over time.
    • detections: 3D oriented bounding boxes (OBBs) representing physical objects.
  5. Things and observations: Persistent physical entities identified across multiple scans.

Engine-agnostic inference architecture

The backend isolates computer vision models behind a unified /infer contract controlled by the INFERENCE_MODE environment variable:

  • ollama (default): Connects to the Ollama at OLLAMA_HOST and uses the model in OLLAMA_MODEL, which defaults to qwen3-vl:4b. If that model isn’t pulled, it falls back as described in Give the app a vision model. It provides local object classification and spatial scene captions.
  • onnx: Runs the LiquidAI/LFM2.5-VL-1.6B-ONNX model on the CPU with onnxruntime-node, without an external daemon. Its model files are read from public/models and are not in the store image.
  • mock: A lightweight, deterministic color-blob detector used for headless CI test suites and low-spec development.

What leaves your network

Scan files, camera frames, and detections stay on your Hub. Camera frames go to the app on your Hub, which sends them to the Ollama it uses for labelling. Two optional features send data elsewhere:

  • Companion Memory. A memory-push job sends a scan to Companion Memory: the one on your Hub once the Hub has connected it, or an external CI-Server whose address and key you entered. Scans go there only when you queue that job.
  • Companion Account sign-in. When sign-in is on, the phone goes to the Companion Portal to sign in. Sign-in is off by default.

The pages themselves make a few third-party requests:

  • Fonts load from Google Fonts. The map library ships with the app, so it isn’t loaded from a CDN.
  • The maps draw tiles from the CARTO basemap, which uses OpenStreetMap data.
  • When location access is on, the phone asks OpenStreetMap’s Nominatim service to turn your coordinates into an address.

Troubleshooting

Tracking never starts, or “Move camera to initialise tracking…” never goes away Camera or motion permission was denied. Re-enable it in your browser or OS settings (iOS: Settings → Safari → Motion & Orientation Access, and camera access for the site) and reload over the HTTPS link.

“Needs a secure (HTTPS) link” on the start button Camera and AR APIs only work on a secure origin. Open the app’s HTTPS link, not a plain-HTTP address.

The scan has no labels, or capturing reports an error The app couldn’t use an Ollama vision model. Check <app address>/health as described in Give the app a vision model, pull a vision model, and restart the app, because it checks Ollama only when it starts.

Uploading a video or photos fails The July 2026 build failed here because its image lacked ingest-quality.mjs, which the upload scripts import. The v0.1.0 image includes that file, but whether uploads work on it is to confirm. Use live AR scanning if they don’t.

Splat training or dense reconstruction is skipped These jobs need GPU tools that the store image doesn’t include. The job reports a skipped reason.

The reconstructed model has holes Coverage was too low. Capture more angles of the problem areas and scan again. A world can hold several scans of the same space.

Last updated on